F1: cap parsed terms at MAX_PARSED_TERMS (64) in parseSearchQuery so a huge
pasted query no longer nests the combined tsquery deep enough to blow Postgres'
stack depth limit (HTTP 500); +@MaxLength(10000) on SearchDTO.query as
defense-in-depth. F2/F3: restore titleOnly text_content leak-guard and
parentPageId subtree-scoping coverage in the lexical int-spec. F4: positive
ancestor-path ordering + non-empty snippet asserts. F5: remove dead
SearchLookupTier enum, unused buildTsQuery + pg-tsquery require (and its tests),
and the discarded parentPageId select in fetchDetails. F6: rewrite the stale
#443 DTO comment (parentPageId/titleOnly read by the engine; substring ignored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rebase pulled develop's #494 REVERSE prose drift-guard, which flags every
camelCase token in ROUTING_PROSE that is not a registered MCP tool. #529's search
routing prose documents the search RESPONSE fields matchedTerms/matchedFields/
hasMore/truncatedAtCap — genuine non-tool terms — so add them to
PROSE_NON_TOOL_TERMS (develop's designed allowlist). Reconciles both invariants:
#529's search-prose enforcement and #494's dead-reference guard both stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Develop introduced two migrations at the exact 20260707T120000 stamp
(ai-chat-metadata, page-history-kind). Three files sharing one timestamp
is fragile for deploy ordering, so move #529's ru_en-config migration to a
strictly-later, unique 20260707T130000 so it orders cleanly last. It touches
only the ru_en text-search config (different tables), so running it after the
develop migrations is order-independent and safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
S1: fetchDetails now builds rank/highlight/matchedFields from the same
[...positive, ...required] set matchedTerms uses, so a required-only query
(`+кофейня`) no longer reports matchedFields:[] / rank:null.
W1: documented that CANDIDATE_CAP bounds pagination reachability (JS-side), not
query compute — a SQL LIMIT would corrupt the exact permission-filtered total.
S2: one-line note that literal-`%`/`_` search was intentionally dropped (total:0).
W3: restored the trgm-index EXPLAIN guard (search-lookup-explain.int-spec) that
asserts the coalesce-free substring predicates use idx_pages_*_trgm, not Seq Scan.
W4: added an integration ordering test proving title-exact > title-substring >
text-only tier dominance under RRF.
S3: added a test that an exact-title hit survives a tiny CANDIDATE_CAP window.
S5: server pretest now also builds @docmost/mcp so the ToolWriteClass import
resolves from clean CI (mirrors editor-ext/prosemirror-markdown).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BLOCKER B1: the fts config swap no longer blindly DROP+ADDs the generated
column. swapEmbeddingsFtsConfig now (1) reads the column's actual generation
expression from pg_catalog and TRUE-no-ops when it already references the target
config (real out-of-band escape hatch), and (2) gates the inline ACCESS
EXCLUSIVE rewrite behind SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE (default 'true';
only a literal 'false' opts out), warning that the operator owns the swap.
up() now creates ru_en only when missing (drop-recreate would fail on the fts
hard dependency on a re-run), so up() is genuinely idempotent. down() guards the
config DROP when fts still references ru_en (gated-off path), staying non-fatal.
W2: corrected the header — Kysely runs EACH migration in its own transaction
(not the whole set in ONE); the single-UPDATE atomicity conclusion is unchanged.
Rewrote the runbook: inline path is a full-table ACCESS EXCLUSIVE rewrite of
page_embeddings; removed the false "IF-EXISTS no-op" claim.
Extends the roundtrip test with idempotent-skip (2nd up() no-op) and env-gate
=false (embeddings rewrite skipped, pages.tsv still swapped) assertions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Asserts the ru_en migration is reversible in the correct order: down()
reverts pages.tsv trigger + page_embeddings.fts to english BEFORE dropping
the ru_en config, and up() re-applies it idempotently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- search.types.ts: IPageSearch is the #529 superset (pageId/snippet/score/
path/matchedFields/matchedTerms; rank/highlight nullable); add the
IPageSearchResponse pagination envelope and match/limit/offset params.
- search-spotlight.tsx: operator hint ("exact phrase", +required, -excluded).
The web list already reads response.items, so it picks up the new OR order +
superset with no code change; icon/space/highlight remain (acceptance #12).
Full spotlight pagination UI is deferred within this PR — the API + MCP fully
support total/hasMore/offset.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- client/read.ts search(): forward match + offset; surface the pagination
envelope (total/hasMore/truncatedAtCap/offset) when the #529 server
returns it (a stock upstream leaves them undefined).
- index.ts search tool: document OR-default + RU/EN morphology, the
"phrase"/+/- operators, match modes, limit+offset pagination and the
relevance-CAP unreachable-tail caveat; add match + offset params.
- server-instructions.ts: READ prose + inventory line updated to the new
contract, enforced by a new tool-inventory.test.mjs assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite SearchService.searchPage into ONE engine for web-UI, MCP agent and
share:
- A2 server-side query parser (search-query-parser.ts): quote-aware
tokenizer, leading +/- operators (internal -,.,: literal), "phrase",
metachar-stripped bare terms; tsquery built as a parameterized AST via
SQL ||/&&/!! (never string-concat). only-negation/empty short-circuit.
- A3 match=auto routes identifier-like terms (10.31.41, esp32,
WB-MGE-30D86B) to the substring/trigram branch, words to FTS; word/
prefix/substring overrides.
- A4 RRF (k=60) fuses the FTS branch (ts_rank_cd) and substring branch
(title-exact>title-sub>text tier) by RANK; ORDER BY rrf DESC, id.
- A5 exact permission-filtered total (fail-closed via filterAccessiblePageIds
with the #348 hasRestricted fast-path), CANDIDATE_CAP fusion window,
offset/limit, hasMore, truncatedAtCap.
- A6 single path (spaceId/share/creatorId/titleOnly/parentPageId/match);
share uses getPageAndDescendantsExcludingRestricted.
- A7 response superset per hit (id/pageId/slugId/icon/title/space/…/rank/
highlight/snippet/path/score/matchedFields/matchedTerms).
- A8 buildAncestorPaths now skips deleted + cross-space ancestors.
- A9 DTOs: match, offset, total/hasMore/truncatedAtCap/query/matchedFields.
SEARCH_MODE=or|and toggles the parser; SEARCH_CANDIDATE_CAP tunes the
window. Legacy lookup unit/int specs replaced by parser unit tests + a
13-criteria integration spec on real pg (incl. a permission mutation guard
and a fail-closed propagation test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce a `ru_en` FTS configuration (english_stem over ascii token
classes, russian_stem over Cyrillic ones) and flip every stored + query
side to it IN LOCKSTEP (acceptance #13):
- new migration creates ru_en, swaps the pages.tsv trigger + reindexes
existing rows (row-lock UPDATE, no ACCESS EXCLUSIVE), and swaps the
page_embeddings.fts generated column (documented rewrite/lock trade-off
for large tenants, mirroring the #443 trgm migration). down() reverts
tsv/fts to english BEFORE dropping the config (dependency order).
- page-embedding.repo.ts hybridSearch query config english -> ru_en, so
the RAG lexical leg's query config matches its fts column config.
- search.service.ts current query literals english -> ru_en so the column
and query configs stay paired (this commit is independently revertable;
the engine itself is rewritten in the A2-A9 commit).
The reindex is atomic within the single migration transaction (this repo's
Migrator wraps all pending migrations in one tx), so no morphology-desync
window exists and no dual-config read path is needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agent_coder