Compare commits

...

59 Commits

Author SHA1 Message Date
agent_coder 2c4fc565b6 fix(prosemirror-markdown): escape literal <br> in text so it round-trips as text, not a hardBreak (#554)
A literal inline-HTML break tag typed as prose text (`<br>`, `<br/>`,
`<br />`) was emitted verbatim into markdown, so on re-import marked
parsed it as an inline-HTML line break and silently turned the user's
text into a hardBreak node, dropping the text. HTML-entity-encode only
the angle brackets of a break-tag sequence in `case "text"` so it lands
as `&lt;br&gt;` and the importer decodes it back to literal `<br>`.

Scoped strictly to the `<br…>` pattern (not every `<`/`>`), so stray
angle brackets in prose (`a < b > c`) are untouched, and to the
text-content path only — a real hardBreak serializes from its own case
(`  \n`, or `<br>` via inlineToHtml), so the serializer's own emitted
breaks are never escaped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:58:21 +03:00
vvzvlad a990ebd604 Merge pull request 'feat(search) Фаза A: лексический оверхол — ru_en, OR/RRF, операторы, пагинация (#529)' (#538) from feat/529-search-lexical into develop
Reviewed-on: #538
2026-07-12 17:40:43 +03:00
vvzvlad 94ca907476 Merge pull request 'fix(prosemirror-markdown): hardBreak переживает round-trip (trailing/consecutive/table-cell) (#549)' (#553) from fix/549-hardbreak-md-canon into develop
Reviewed-on: #553
2026-07-12 17:40:18 +03:00
agent_coder fff772cbe2 fix(search): F1 term-cap stack-depth guard + restore/extend coverage, drop dead code (#529)
F1: cap parsed terms at MAX_PARSED_TERMS (64) in parseSearchQuery so a huge
pasted query no longer nests the combined tsquery deep enough to blow Postgres'
stack depth limit (HTTP 500); +@MaxLength(10000) on SearchDTO.query as
defense-in-depth. F2/F3: restore titleOnly text_content leak-guard and
parentPageId subtree-scoping coverage in the lexical int-spec. F4: positive
ancestor-path ordering + non-empty snippet asserts. F5: remove dead
SearchLookupTier enum, unused buildTsQuery + pg-tsquery require (and its tests),
and the discarded parentPageId select in fetchDetails. F6: rewrite the stale
#443 DTO comment (parentPageId/titleOnly read by the engine; substring ignored).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 16c2a4b623 fix(search): whitelist #529 search response-fields in the #494 prose drift-guard
The rebase pulled develop's #494 REVERSE prose drift-guard, which flags every
camelCase token in ROUTING_PROSE that is not a registered MCP tool. #529's search
routing prose documents the search RESPONSE fields matchedTerms/matchedFields/
hasMore/truncatedAtCap — genuine non-tool terms — so add them to
PROSE_NON_TOOL_TERMS (develop's designed allowlist). Reconciles both invariants:
#529's search-prose enforcement and #494's dead-reference guard both stay green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 58f13a7efd chore(search): retimestamp ru_en migration to 20260707T130000 to avoid collision (#529)
Develop introduced two migrations at the exact 20260707T120000 stamp
(ai-chat-metadata, page-history-kind). Three files sharing one timestamp
is fragile for deploy ordering, so move #529's ru_en-config migration to a
strictly-later, unique 20260707T130000 so it orders cleanly last. It touches
only the ru_en text-search config (different tables), so running it after the
develop migrations is order-independent and safe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 2bb48f841f fix(search): S1/W1/W3/W4/S3/S5/S2 — required-term details, CAP note, restored guards (#529)
S1: fetchDetails now builds rank/highlight/matchedFields from the same
[...positive, ...required] set matchedTerms uses, so a required-only query
(`+кофейня`) no longer reports matchedFields:[] / rank:null.

W1: documented that CANDIDATE_CAP bounds pagination reachability (JS-side), not
query compute — a SQL LIMIT would corrupt the exact permission-filtered total.

S2: one-line note that literal-`%`/`_` search was intentionally dropped (total:0).

W3: restored the trgm-index EXPLAIN guard (search-lookup-explain.int-spec) that
asserts the coalesce-free substring predicates use idx_pages_*_trgm, not Seq Scan.

W4: added an integration ordering test proving title-exact > title-substring >
text-only tier dominance under RRF.

S3: added a test that an exact-title hit survives a tiny CANDIDATE_CAP window.

S5: server pretest now also builds @docmost/mcp so the ToolWriteClass import
resolves from clean CI (mirrors editor-ext/prosemirror-markdown).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder daa96eb132 fix(search): B1/W2 — honest page_embeddings.fts swap + true idempotency (#529)
BLOCKER B1: the fts config swap no longer blindly DROP+ADDs the generated
column. swapEmbeddingsFtsConfig now (1) reads the column's actual generation
expression from pg_catalog and TRUE-no-ops when it already references the target
config (real out-of-band escape hatch), and (2) gates the inline ACCESS
EXCLUSIVE rewrite behind SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE (default 'true';
only a literal 'false' opts out), warning that the operator owns the swap.
up() now creates ru_en only when missing (drop-recreate would fail on the fts
hard dependency on a re-run), so up() is genuinely idempotent. down() guards the
config DROP when fts still references ru_en (gated-off path), staying non-fatal.

W2: corrected the header — Kysely runs EACH migration in its own transaction
(not the whole set in ONE); the single-UPDATE atomicity conclusion is unchanged.
Rewrote the runbook: inline path is a full-table ACCESS EXCLUSIVE rewrite of
page_embeddings; removed the false "IF-EXISTS no-op" claim.

Extends the roundtrip test with idempotent-skip (2nd up() no-op) and env-gate
=false (embeddings rewrite skipped, pages.tsv still swapped) assertions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 4c5230d677 test(search): A1 migration down()/up() reversal roundtrip (#529)
Asserts the ru_en migration is reversible in the correct order: down()
reverts pages.tsv trigger + page_embeddings.fts to english BEFORE dropping
the ru_en config, and up() re-applies it idempotently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 50ca27c5d4 feat(search): A9 web-UI — superset types, match/pagination params, operator hint (#529)
- search.types.ts: IPageSearch is the #529 superset (pageId/snippet/score/
  path/matchedFields/matchedTerms; rank/highlight nullable); add the
  IPageSearchResponse pagination envelope and match/limit/offset params.
- search-spotlight.tsx: operator hint ("exact phrase", +required, -excluded).

The web list already reads response.items, so it picks up the new OR order +
superset with no code change; icon/space/highlight remain (acceptance #12).
Full spotlight pagination UI is deferred within this PR — the API + MCP fully
support total/hasMore/offset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder de25c258f9 feat(search): A9 MCP — operators, match, pagination in the search tool (#529)
- client/read.ts search(): forward match + offset; surface the pagination
  envelope (total/hasMore/truncatedAtCap/offset) when the #529 server
  returns it (a stock upstream leaves them undefined).
- index.ts search tool: document OR-default + RU/EN morphology, the
  "phrase"/+/- operators, match modes, limit+offset pagination and the
  relevance-CAP unreachable-tail caveat; add match + offset params.
- server-instructions.ts: READ prose + inventory line updated to the new
  contract, enforced by a new tool-inventory.test.mjs assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
vvzvlad afc50ead38 Merge pull request 'feat(deploy): version-coherence — WS-анонс версии + guarded reload устаревших вкладок (#481)' (#499) from feat/481-version-coherence into develop
Reviewed-on: #499
2026-07-12 17:37:04 +03:00
vvzvlad ec932e89a3 Merge pull request 'feat(page-history): время работы над статьёй — число + суточный punch-card (#395)' (#551) from feat/395-work-time into develop
Reviewed-on: #551
2026-07-12 17:36:27 +03:00
vvzvlad 490d7965a1 Merge pull request 'feat(comment): Undo в тосте резолва (reopen) + сортировка Resolved по времени резолва (#542)' (#548) from feat/542-resolve-undo into develop
Reviewed-on: #548
2026-07-12 17:36:08 +03:00
vvzvlad 32c9361969 Merge pull request 'feat(mcp): буквальная запись markdown (без math/autolink) + getPage format:text (#502)' (#552) from feat/502-mcp-md-modes into develop
Reviewed-on: #552
2026-07-12 17:35:36 +03:00
vvzvlad c30f910d66 Merge pull request 'fix(prosemirror-markdown): markdown-импорт помечает внутренние ссылки на страницы как internal (#522)' (#524) from fix/522-internal-links into develop
Reviewed-on: #524
2026-07-12 17:35:28 +03:00
agent_coder 7007f6bcf9 feat(search): A2-A9 — unified OR-relevance engine, RRF, pagination (#529)
Rewrite SearchService.searchPage into ONE engine for web-UI, MCP agent and
share:

- A2 server-side query parser (search-query-parser.ts): quote-aware
  tokenizer, leading +/- operators (internal -,.,: literal), "phrase",
  metachar-stripped bare terms; tsquery built as a parameterized AST via
  SQL ||/&&/!! (never string-concat). only-negation/empty short-circuit.
- A3 match=auto routes identifier-like terms (10.31.41, esp32,
  WB-MGE-30D86B) to the substring/trigram branch, words to FTS; word/
  prefix/substring overrides.
- A4 RRF (k=60) fuses the FTS branch (ts_rank_cd) and substring branch
  (title-exact>title-sub>text tier) by RANK; ORDER BY rrf DESC, id.
- A5 exact permission-filtered total (fail-closed via filterAccessiblePageIds
  with the #348 hasRestricted fast-path), CANDIDATE_CAP fusion window,
  offset/limit, hasMore, truncatedAtCap.
- A6 single path (spaceId/share/creatorId/titleOnly/parentPageId/match);
  share uses getPageAndDescendantsExcludingRestricted.
- A7 response superset per hit (id/pageId/slugId/icon/title/space/…/rank/
  highlight/snippet/path/score/matchedFields/matchedTerms).
- A8 buildAncestorPaths now skips deleted + cross-space ancestors.
- A9 DTOs: match, offset, total/hasMore/truncatedAtCap/query/matchedFields.

SEARCH_MODE=or|and toggles the parser; SEARCH_CANDIDATE_CAP tunes the
window. Legacy lookup unit/int specs replaced by parser unit tests + a
13-criteria integration spec on real pg (incl. a permission mutation guard
and a fail-closed propagation test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:35:24 +03:00
agent_coder 4bf06c68cf feat(search): A1 — ru_en text-search configuration + config swap (#529)
Introduce a `ru_en` FTS configuration (english_stem over ascii token
classes, russian_stem over Cyrillic ones) and flip every stored + query
side to it IN LOCKSTEP (acceptance #13):

- new migration creates ru_en, swaps the pages.tsv trigger + reindexes
  existing rows (row-lock UPDATE, no ACCESS EXCLUSIVE), and swaps the
  page_embeddings.fts generated column (documented rewrite/lock trade-off
  for large tenants, mirroring the #443 trgm migration). down() reverts
  tsv/fts to english BEFORE dropping the config (dependency order).
- page-embedding.repo.ts hybridSearch query config english -> ru_en, so
  the RAG lexical leg's query config matches its fts column config.
- search.service.ts current query literals english -> ru_en so the column
  and query configs stay paired (this commit is independently revertable;
  the engine itself is rewritten in the A2-A9 commit).

The reindex is atomic within the single migration transaction (this repo's
Migrator wraps all pending migrations in one tx), so no morphology-desync
window exists and no dual-config read path is needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:35:24 +03:00
vvzvlad af44736fb9 Merge pull request 'fix(mcp): drawio — авто-стрип XML-комментариев (гейт на well-formedness) вместо lint-ошибки (#505)' (#545) from fix/505-drawio-strip-comments into develop
Reviewed-on: #545
2026-07-12 17:35:01 +03:00
vvzvlad 2a4d1acfd7 Merge pull request 'fix(prosemirror-markdown): insertNode дописывает пункт в существующий список вместо второго списка (#535)' (#537) from fix/535-list-seam-coalesce into develop
Reviewed-on: #537
2026-07-12 17:34:41 +03:00
vvzvlad 11eb87d58a Merge pull request 'docs(server): getPageBreadCrumbs — предки не утекают (нисходящее наследование ограничений), задокументировано (#471)' (#550) from docs/471-breadcrumb-permission into develop
Reviewed-on: #550
2026-07-12 17:33:55 +03:00
vvzvlad f2c8aa70f3 Merge pull request 'fix(mcp): внятная ошибка на недоступный spaceId вместо «Space permissions not found» (#534)' (#536) from fix/534-mcp-spaceid into develop
Reviewed-on: #536
2026-07-12 17:33:39 +03:00
vvzvlad 661ea8ba07 Merge pull request 'fix(client): reconnect-вход — таймаут-гонка вокруг getRun, hang не залипает FSM в streaming (#541)' (#543) from fix/541-reconnect-timeout into develop
Reviewed-on: #543
2026-07-12 17:32:07 +03:00
vvzvlad 059edccb64 Merge pull request 'test(mcp): hardBreak уже представим в md-каноне (#293) — регресс-гард, закрытие #503' (#544) from fix/503-hardbreak-canon into develop
Reviewed-on: #544
2026-07-12 17:31:55 +03:00
vvzvlad 693a9a350b Merge pull request 'perf(ai-chat) волна C: инкрементальный рендер стрима + append-персист таблица шагов (#492)' (#547) from feat/492-incremental-render into develop
Reviewed-on: #547
2026-07-12 17:31:40 +03:00
agent_coder 79b2da686b fix(work-time): clip cross-class padding; gate & payload fixes (#395)
F1: pad-clip adjacent different-class sessions at the raw-gap midpoint so
workMs and agentOnlyMs can never double-count the same wall-clock (default
agentTGap 7m < pIn+pOut 10m made a work-session-ending-in-agent overlap a
nearby agent_only run). Reword the config guard/comment: cross-class
disjointness is now structural, tGap≥pIn+pOut is a kept sanity bound. Add a
cross-class no-double-count test and a real seeded property/fuzz test (250
random timelines × 4 tz) asserting per-class union, cross-class disjointness,
and Σ per-day activeMs == workMs.

F2: page.controller.spec — add a ForbiddenException view-gate reject test that
asserts the rejection propagates and computeWorkTime is NOT reached, locking
validateCanView before compute.

F3: WorkTimeStat renders for agent-only pages (workMs==0, agentOnlyMs>0) with
an `agent:` headline so the punch-card stays reachable.

F4: drop the dead un-bucketed `sessions` list from the PageWorkTime response
and the client work-time types (no client reads it; bucketByDay still consumes
the core sessions internally).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 08:31:42 +03:00
agent_coder ed3a8f8174 docs(mcp): sync write-tool specs with #502 literal-markdown behavior (#502 review)
The four MCP markdown-write tools (updatePageMarkdown/createPage/patchNode/
insertNode) still described the old parse-everything behavior after #502 turned
TeX math and fuzzy autolink OFF on the write paths. Add a concise contract hint
to each: $...$ / $$...$$ stay literal (not a math formula) and schemeless
www.host / bare emails are not auto-linked (explicit https:// still links); for
a real formula use updatePageJson (or a mathInline/mathBlock node via `node` on
patch/insert). Also note getPage format:"text" in README.md/README.ru.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:57:26 +03:00
agent_coder 8503ff1f3d fix(ai-chat): hydrate crashed mid-run steps for model replay + cover write/read seams (#492)
F1: the server model-replay loaded history via findAllByChat().map(rowToUiMessage)
WITHOUT hydrating parts from ai_chat_run_steps. A HARD crash mid-run (SIGKILL/OOM)
fires no terminal callback, so the assistant row stays parts:[] and its partial
tool-calls/results/text (durable in the steps table) dropped out of the model's
next-turn context. Hydrate needy assistant rows (role==='assistant' &&
!rowHasInlineParts) via findByMessageIds + hydrateAssistantParts before the replay
map — mirroring the controller's withReconstructedParts exactly — guarded on the
optional repo. Fix the now-false interrupt-resume comment.

F2: add a service int-spec that drives the REAL onStep append-persist WRITE branch
through AiChatService.stream with a real AiChatRunStepRepo injected, asserting the
per-step rows' stepIndex + parts slice and the step-marker metadata match a
single-row flush (catches an stepsPersisted-1 off-by-one).

F3: add a controller int-spec that drives withReconstructedParts through getMessages
WITH the repo present (a mid-run marker-only row + its step rows), asserting the
reconstructed metadata.parts and workspace-scoping.

F4: remove the dead countByMessage (zero prod callers; reconstructRunParts derives
stepsPersisted inline) + its now-unused sql import and the redundant test assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:50:50 +03:00
agent_coder c18b4c132c fix(md-canon): preserve hardBreak in trailing/consecutive/table-cell round-trips (#549)
The markdown canon lost `hardBreak` nodes in three pm->md->pm sub-cases where
the two-space `  \n` form does not survive re-import via marked:

1. Trailing at end of a paragraph — the top-level `.trim()` strips the trailing
   `  \n`, and even a non-last paragraph's trailing break sits before the `\n\n`
   block gap, so marked drops it.
2. Consecutive breaks (`a<hardBreak><hardBreak>b`) — the whitespace-only middle
   line reads as a paragraph separator, splitting the run into two paragraphs.
3. Inside a GFM table cell — the single-line cell collapses the break to a space.

Emit `<br>` (inline HTML break) in exactly those positions: marked passes it
through and generateJSON rebuilds a hardBreak in every context, including table
cells. renderInlineChildren now picks `<br>` for a break that is the last inline
child or is followed by another break; a safe mid-paragraph break keeps the
byte-stable `  \n` form (golden output unchanged). Footnote bodies are skipped
(their `^[…]` form already collapses breaks to spaces). The table-cell case
converts the `  \n` marker to `<br>` before the newline collapse.

Adds pm->md->pm count/position round-trip tests for all three sub-cases plus a
mid-paragraph regression guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:10:40 +03:00
agent_coder 3163f50c98 fix(mcp): route #502 extension flags per write tool; add server import flag (#502 review)
Internal review found the two layered-extension flags landed in the SHARED
markdownToProseMirrorCanonical wrapper, which mis-covered two tools. Move the
decision to each caller by the tool's semantics.

BLOCKER 1 — createPage was NOT covered. createPage POSTs to the server
/pages/import endpoint, which imported with DEFAULTS (math + fuzzy autolink ON),
so an agent's `$x=1$` still became a formula and `www.host` still autolinked.
Add an optional `disableMarkdownExtensions` multipart field to /pages/import
(default OFF, so human file uploads keep math ON); import.controller reads it,
import.service.importPage/processMarkdown thread it into markdownToProseMirror.
MCP createPage sends it true.

BLOCKER 2 — import_page_markdown was wrongly disabled. It goes through the same
wrapper, so hardcoding parseMath:false degraded an exported `$x^2$` to literal
text on re-import, breaking the #328 lossless export→import pair. The wrapper no
longer hardcodes the flags: it takes them from the caller and DEFAULTS to the
package importer defaults (extensions ON). Callers now set them explicitly:
  - updatePageMarkdown (updatePageContentRealtime) -> OFF
  - patch_node/insert_node (importMarkdownFragment) -> OFF (unchanged)
  - import_page_markdown -> DEFAULTS (math ON) — #328 round-trip restored

Tests: rewrite the mcp write test around the corrected per-caller semantics
(agent-write OFF, import_page_markdown DEFAULTS incl. the REAL #328 round-trip);
add a collab-backed wiring test driving client.updatePage (OFF) and
client.importPageMarkdown (DEFAULTS) end-to-end; add a createPage multipart-flag
test; add a server import.service.processMarkdown spec (flag OFF -> literal / no
autolink, default -> math ON for human uploads); reword the prosemirror-markdown
round-trip test to its package-default scope. Mutation-verified both caller
wirings (flip updatePageMarkdown -> defaults reddens the OFF wiring test; make
the wrapper default OFF reddens the #328 round-trip).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:58:48 +03:00
agent_coder 3e81f64415 fix(work-time): user idle breaks agent burst; cap tz length (#395)
Only an agent-sourced idle pulse extends an agent burst; a user idle
(human supervision) now falls through to the human branch so the session
is classified `work`, not `agent_only`. Add @MaxLength(64) to tz to match
its comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:53:59 +03:00
agent_coder d36021a111 fix(comment): clear resolved mark only after reopen confirmed
Move the inline ProseMirror comment-mark clear out of the Undo toast's
onClick and into the reopen mutation's onSuccess branch. Clearing it
eagerly flipped the collab mark to unresolved before the reopen result
was known: on a non-404 failure the RQ cache rolls back to resolved but
the doc kept an active highlight the panel treats as resolved, and a
comment refetch can't heal a divergence that lives in the collab doc.
Mirrors the 404 branch's editor-liveness guard/try-catch and the safe
await-then-mark pattern in comment-list-item. The button-triggered
reopen already sets the mark, so the onSuccess call is an idempotent
no-op there.

Tests: reopen failure (500) leaves the mark untouched; null editorRef
on the success path degrades gracefully.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:51:38 +03:00
agent_coder 1f2999e5ad fix(mcp): drawio — авто-стрип XML-комментариев вместо lint-ошибки (#505)
Модель органически вставляет `<!-- ... -->` в mxGraph-XML вопреки запрету
в описании тула (природа LLM, промптингом не лечится). Жёсткая ошибка
линтера [no-comments] заставляла её ПОЛНОСТЬЮ перегенерировать диаграмму —
впустую потраченный tool-call на почти каждой первой генерации.

Теперь в общем prepare-пути (`prepareModel`, через который идут
drawioCreate/drawioUpdate/drawioEditCells) комментарии срезаются ДО линта:
`xml.replace(/<!--[\s\S]*?-->/g, "")`. Комментарии не несут семантики, так
что стрип безопасен. Число срезанных уходит в `warnings[]` как
`stripped N XML comment(s)` (только при N > 0) — модель это видит, но НЕ
ретраит.

Стрип ГЕЙТИТСЯ двумя условиями, чтобы регэксп бил только по настоящим
comment-нодам:
  1. well-formedness — сначала парсим модель; на malformed-входе (сырой
     неэкранированный `<!--` внутри значения атрибута на пути create/update,
     где normalizeInput отдаёт строку без парсинга) стрип НЕ выполняется,
     иначе он молча вырезал бы текст автора. Вход отклоняют правила
     value-escaping / well-formed-xml, автор видит реальную ошибку.
  2. отсутствие CDATA — CDATA-секция well-formed, но её текст может содержать
     литеральный `<!-- ... -->`, который НЕ является comment-нодой; стрип
     молча удалил бы контент автора. Настоящий drawio держит подписи в
     атрибутах `value=` (CDATA невозможен), так что исключение CDATA бесплатно
     на легитимных моделях; CDATA-модель с реальным комментарием падает на
     сохранённый backstop no-comments (явная ошибка), а не тихо портится.

Правило `no-comments` в линтере оставлено как defense-in-depth backstop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:43:03 +03:00
agent_coder c674db2b2f feat(page-history): «время работы над статьёй» — число в UI + суточный punch-card (#395)
Оценка времени работы над страницей как сессионизация истории page_history по
паузам бездействия (WakaTime-подобно), а не span между крайними правками, плюс
drill-down в суточный таймлайн 24ч×дни. Зависит от #374 (kind + idle-пульс).

Server:
- Чистая computeWorkTime(rows, config) (§5): нормализация+дедуп → коллапс
  агентских всплесков по aiChatId → ОДИН проход сессионизации (порог зависит от
  пары: оба agent → agentTGap, иначе tGap; последняя сессия обязательно
  закрывается) → класс готовой сессии (все agent → agent_only, иначе work) →
  добивка (много-сэмпл → [first−P_in, last+P_out], одиночный скаляр →
  [t−P_single, t]) → метрики = union wall-clock по классу. Детерминированная,
  без БД.
- Чистая bucketByDay(sessions, tz) (§6.3): union work/agent_only РАЗДЕЛЬНО, разрез
  по полуночи tz через нативный Intl (DST-корректно, без «+24ч»). Инвариант
  Σ activeMs == workMs держится по построению.
- PageHistoryRepo.findTimelineByPageId — лёгкая проекция без content, ASC.
- PageHistoryService.computeWorkTime + POST /pages/history/time, гейт
  validateCanView как у /history; атрибуция человек/агент по lastUpdatedSource.

Client:
- usePageWorkTime(pageId) шлёт tz зрителя (Intl локаль).
- Кликабельное число «≈ 4 ч 30 мин» в шапке страницы (порог в тултипе).
- Модалка-punch-card: 24-часовые дорожки по дням, окна work/agent разным цветом,
  одиночные (P_single) приглушены, сумма за день, пустой день «—», сворачивание
  длинных серий пустых дней, подпись tz + T_gap. i18n ru-RU + en-US.

Tests: 22 юнита на computeWorkTime/bucketByDay (§7-фикстура ≈1ч32м vs ≈60h наив,
закрытие последней сессии, коллапс/разрыв всплеска, idle-пульс, DST 23/25ч,
полуночный разрез, инвариант, union-без-задвоения); 5 юнитов гейта контроллера;
int-spec на реальном pg (проекция + совпадение с ядром); 6 клиентских на формат.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:32:12 +03:00
agent_coder 44cdb5a4c3 feat(mcp): disable math+autolink in MCP markdown-write; getPage format:text for machine diff (#502)
WRITE: parameterize the canonical importer markdownToProseMirror with
{parseMath, fuzzyLinkify} (defaults true — editor/file-import/git-sync unchanged);
MCP write paths (createPage/updatePageMarkdown/patchNode/insertNode) call with both
false so $...$ stays literal text and bare domains don't autolink (explicit
https:// still links). READ: getPage format:'text' reuses the server jsonToText path
(deterministic flag) for flat machine-diffable text, [image]/[table RxC] placeholders.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:18:31 +03:00
agent_coder 03af65dd06 Merge remote-tracking branch 'gitea/develop' into fix/503-hardbreak-canon 2026-07-12 06:12:53 +03:00
agent_coder dead5fa8a8 test(#503): guard hardBreak representability through the MCP canon
Issue #503 reported a hardBreak sent via updatePageJson being «silently
cut» — no error, no line break — forcing the agent to split lines into
separate paragraphs.

A faithful repro shows the bug does NOT reproduce against develop:
hardBreak is fully representable in the markdown canon in BOTH directions
and survives the real Yjs write path.

- pm -> md: the serializer (now in @docmost/prosemirror-markdown after the
  #293 STEP 5 consolidation) emits the CommonMark hard break `  \n`, not a
  bare `\n`.
- md -> pm: marked tokenizes `  \n` to `<br>`, which generateJSON parses
  back to a hardBreak node in ONE paragraph (not split, not dropped).
- updatePageJson (PM JSON -> applyDocToFragment = PMNode.fromJSON +
  updateYFragment, the real collab-session write encoder -> read back)
  keeps text + hardBreak + text in one paragraph.

The issue's diagnosis points at packages/mcp/.../markdown-converter.ts as
the serializer, but that file is a 15-line re-export shim since #293 STEP 5
— the diagnosis predates the converter consolidation that already closed
both sides. P1 (semantic round-trip) + P2 (byte-fixpoint) hold, and the
node already has broad property/corpus/golden coverage in the package.

No converter change is warranted (switching the break form to `\`+newline
would churn the whole #351 byte-fixpoint corpus against the established
`  \n` repo convention for zero functional gain). This adds one
integration guard at the MCP canon seam covering all three seams of the
reported updatePageJson path. Mutation-verified: neutering the serializer
arm or the importer `<br>` handling reddens seams 1-2; neutering the real
applyDocToFragment write path reddens seam 3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:04:18 +03:00
agent_coder 46bb55dbd1 Merge remote-tracking branch 'gitea/develop' into fix/534-mcp-spaceid 2026-07-12 05:58:05 +03:00
agent_coder 8b34a428f4 fix(mcp): unconditional final ERROR_MESSAGE_CAP backstop in formatSpaceNotAccessible (#536 review)
The list-only cap assumed a well-formed prefix; a pathologically long
agent-supplied spaceId (unvalidated for length) lives in the prefix and bypassed
the cap. Add the same unconditional final slice formatDocmostAxiosError uses, so
the WHOLE message is bounded regardless of spaceId length. No-op for normal
inputs (36-char UUID) where the list cap already keeps it <= 300.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:58:04 +03:00
agent_coder e236782260 fix(mcp): cap #534 space-not-accessible message + cover fail-open guards
Review follow-up (#536).

F1 (test coverage): add two tests that pin the wrapper's fail-open guards, which
previously survived mutation:
- a non-404 error (500) on a wrapped call with a spaceId propagates unchanged and
  triggers NO /spaces sweep (a real 5xx/403 must never be swallowed/reformatted);
- a 404 when the spaceId IS in the accessible index fails open (the 404 is about
  another resource), so it is not falsely rewritten to "not found among spaces".
Both mutation-verified: forcing the non-404 condition to false reddens the first;
removing the id-present guard reddens the second.

F2 (conventions): formatSpaceNotAccessible now honours ERROR_MESSAGE_CAP (300),
the same budget formatDocmostAxiosError enforces. Only the interpolated space
list is truncated (with an ellipsis); the fixed prefix (bad spaceId) and suffix
(listSpaces pointer) are always kept, so the actionable parts survive. Test: 10
long space names -> message <= 300 and still contains the spaceId + listSpaces.
Adjusted the existing list-cap test to short ids/names so it exercises the 10-item
cap + "(+N ещё)" tail below the length cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:49:31 +03:00
agent_coder 12ed3a0332 feat(comment): undo button in resolve toast + sort resolved tab by resolve time
Part 1 (#542): add an inline Undo (reopen) button to the resolve success
toast in useResolveCommentMutation. Undo re-invokes the same mutation via a
self-ref (declared before useMutation, read at click time — no cycle) and
clears the inline comment mark through an editor ref (survives the originating
CommentListItem unmounting). A closured `done` flag guards a fast double-click
(notifications.hide is async). Reopen keeps the plain toast (no Undo).

onError gains a terminal 404 branch: drop the comment from the cache and
unsetComment its orphaned mark (no rollback → no phantom in Resolved, neutral
"Comment no longer exists"). The generic-failure copy is now directional
(resolve vs. re-open). Autoclose policy const RESOLVE_UNDO_AUTOCLOSE_MS=10000.

Part 2: sort the Resolved tab by resolvedAt DESC via an exported
sortResolvedByResolvedAt helper; coerce with new Date(...) because resolvedAt
is an ISO string at runtime (Date only in the optimistic window).

i18n: add "Failed to re-open comment" and "Comment no longer exists" to
en-US/ru-RU. Tests: 6 resolve-undo cases + 4 sort cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:45:42 +03:00
agent_coder b3cc6a7ff8 Merge remote-tracking branch 'gitea/develop' into feat/481-version-coherence 2026-07-12 05:36:59 +03:00
agent_coder c42cb42413 docs(reload-guard): align header/docstring wording to the window model (#481 review F2 consistency)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:36:58 +03:00
agent_coder ae3dfd8de6 Merge remote-tracking branch 'gitea/develop' into feat/492-incremental-render 2026-07-12 05:34:07 +03:00
agent_coder 41030b2c78 test+docs(deploy): покрыть reactive fail-closed guard + выровнять формулировки под оконный бюджет (#499 ревью)
F1: интеграционный тест на reactive путь при storage, который ЧИТАЕТ, но не
ПИШЕТ (quota / Safari private): getItem→null (бюджет доступен, проходим
hasAutoReloaded), setItem→throw → markAutoReloaded()===false → guard на
chunk-load-error-boundary.tsx:43 обязан отбить reload, иначе реактивный путь
зациклил бы reload без сохранённого бюджета. Симметричный proactive случай уже
покрыт guarded-reload.test.tsx. Мутация (снять guard :43) → тест краснеет.

F2: формулировки "per session / this session" в version-coherence.ts и
guarded-reload.tsx приведены к оконной модели ("per RELOAD_WINDOW_MS window" /
"window budget available") — reload-guard.ts и тест восстановления после окна
уже описывают именно окно, а не постоянный one-shot на сессию.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:32:33 +03:00
agent_coder 515a1aaf1d Merge remote-tracking branch 'gitea/develop' into fix/535-list-seam-coalesce 2026-07-12 05:22:26 +03:00
agent_coder 74387bb047 test(#522): make the internal-link subset invariant a real cross-package drift guard
Reviewer follow-up on the #522 internal-link import fix. The fix itself is
confirmed correct; these three changes harden its load-bearing subset invariant
and fix a stale doc.

Finding 1 (drift guard): the in-package test only compared isInternalPagePath
against a HAND-COPIED server regex, so a later narrowing of the real server
INTERNAL_LINK_REGEX would leave the test green while the client silently became
WIDER than the server. Add apps/server/.../export/internal-link-parity.spec.ts:
the top layer already depends on @docmost/prosemirror-markdown and exports
isInternalPagePath, so this spec imports the LIVE server INTERNAL_LINK_REGEX AND
the LIVE isInternalPagePath and asserts subset over an accept-corpus — a server
narrowing now reddens CI. The in-package manual copy is demoted from its
"drift guard" role (comment updated to say so; it stays as local documentation).

Finding 2 (charset non-vacuity): the REJECT list was purely structural, so the
mutation widening the slug charset [a-zA-Z0-9-] -> [a-zA-Z0-9-.] survived the
whole suite. Add REJECT cases whose ONLY defect is a forbidden slug character
(abc.def / abc_def / abc%20 / abc~x); assert both isInternalPagePath and the
server regex reject them. The mutation now reddens the new reject test.

Finding 3 (stale JSDoc): canonicalize.ts KNOWN_DEFAULTS module blurb claimed
every entry was read from docmost-schema and import-materialized. The new
link.internal default breaks both claims (source is editor-ext/src/lib/link.ts;
external links leave internal absent/null, never false). Add a bullet documenting
the one editor-sourced, non-materialized default (false ≡ absent/null).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:59 +03:00
agent_coder 6ec2981743 fix(prosemirror-markdown): помечать внутренние ссылки на страницы как internal при импорте markdown
При импорте markdown ссылка вида `[t](/s/<space>/p/<slug>)` сохранялась как
внешняя (link-mark получал `internal:null`, `target:"_blank"`,
`rel:"noopener noreferrer nofollow"`): открывалась новой вкладкой, без
hover-превью и без участия в backlinks. Единственный путь к нативной внутренней
ссылке был ручной JSON-патч.

Чиню в общем конвертере пакета, через который проходят ВСЕ markdown-пути (MCP
updatePageMarkdown/importPageMarkdown, patch/insertNode, тела комментариев,
серверный REST create/update, single/zip-импорт, ai-chat, git-sync pull,
вставка markdown в редактор) — все они получают исправление разом.

- Новый модуль `internal-links.ts`: чистый `isInternalPagePath(href)` —
  якорный `^/s/<space>/p/<slug>/?$`, СТРОГОЕ подмножество серверного
  `INTERNAL_LINK_REGEX` (apps/server/.../export/utils.ts), поэтому всё
  помеченное гарантированно бэклинкуется и переписывается при экспорте.
  Fail-toward-external: любая неоднозначность (scheme/host, `#`, `?`,
  бесспейсовый `/p/<slug>`, лишний сегмент, не-строка) остаётся внешней.
- `markInternalLinks(doc)`: пост-обход готового ProseMirror-документа,
  помечает КАЖДУЮ text-ноду, покрытую внутренней ссылкой (включая случай
  вложенных bold/italic внутри ссылки) → `{internal:true, target:null,
  rel:null}`. Чистая и идемпотентная.
- Разводка в `markdownToProseMirrorSync` после stripEmptyParagraphs.
- §11 (обязательная сопутствующая правка): `internal:false` добавлен в
  `KNOWN_DEFAULTS.link` канонизатора — редакторные внешние ссылки хранят
  `internal:false`, теперь он канонизируется как absent/null/external, а
  load-bearing `internal:true` (не дефолт) переживает канонизацию.

Тесты: accept/reject-пиннинг матчера (edge: trailing slash, `#`, `?`,
`/p/x` без спейса, `https://h/p/x`, `/api/...`, uppercase, empty-space),
subset-инвариант против серверного регэкспа, мультинодовая ссылка со вложенными
марками, идемпотентность, полный конвертер (internal помечен / external нетронут /
бэклинк-извлекаемость), комментарий-тело через общий путь, canonicalize §11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:58 +03:00
agent_coder dd1fe90515 fix(prosemirror-markdown): guard three-way list coalesce against transitive style loss (#535)
Review follow-up on the list-seam coalescing:

1. The three-way merge only checked each PRE-EXISTING list against the
   inserted one. A default-typed inserted orderedList between two lists with
   explicit DIFFERENT numbering styles passed both pairwise checks through the
   null-typed middle, collapsing all three and silently losing the right
   list's style. Add a `listsMergeable(left, right)` guard to the three-way
   condition. On failure fall through to the single-seam path: a single
   inserted block can be absorbed by at most one neighbour, so prefer the LEFT
   seam (consistent with the three-way survivor choice) and leave the
   incompatible right list separate with its own style.

2. Lock the footnotesList exclusion with a test — inserting a footnotesList
   next to a footnotesList must leave TWO separate blocks (a refactor of the
   allow-list to endsWith("List") would otherwise silently merge them and
   corrupt footnotes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:54 +03:00
agent_coder c96fafc4ad perf(ai-chat): append-персист шагов — per-step INSERT вместо переписи строки (#492)
Раньше каждый onStepFinish переписывал ВСЮ строку ассистента (растущий
metadata.parts jsonb со всеми выводами инструментов) → O(n²) объёма записи
на прогон: под MVCC/TOAST апдейт jsonb переписывает всю версию строки, так
что шаг k пишет ~k×вывод. Прогон из 50 шагов по ~100 КБ = сотни МБ WAL и
мёртвых кортежей за ход, что молотит autovacuum. (#490 убрал только ВТОРУЮ
копию в tool_calls; сам metadata.parts всё ещё рос и переписывался.)

Теперь каждый завершённый шаг ДОПИСЫВАЕТСЯ отдельной строкой в лёгкую
таблицу ai_chat_run_steps (только парты этого шага), а строка сообщения
получает дешёвый маркер (stepsPersisted + toolTraceVersion, без растущего
блоба parts). Полный metadata.parts собирается ОДИН раз на финализации.
НЕ jsonb-append (||): апдейт всё равно переписывает всю TOAST-версию —
экономится только сетевой payload, а WAL/мёртвые кортежи остаются; поэтому
именно ОТДЕЛЬНАЯ таблица + INSERT.

Три обязательные интеграции:
- reconstructRunParts(row, stepRows) → { parts, stepsPersisted }: единый
  шов переключения бэкенда. Читает парты из СТРОКИ, если она уже несёт
  inline-parts (старые записи + ЛЮБАЯ финализированная), иначе из ТАБЛИЦЫ
  ШАГОВ (mid-run запись #492). Дискриминатор — наличие непустого
  metadata.parts (флаг схемы не нужен). Потребители (attach-seed,
  delta-poll, export, reconnect) прогоняют строки через hydrateAssistantParts
  на границе чтения — их контракт/вывод не меняется, старые и новые записи
  восстанавливаются идентично.
- сигнал ротации кольца реестра #491 (confirmPersistedStep) теперь стреляет
  на подтверждённый INSERT шага, под тем же контрактом (updateStreaming
  возвращает stepsPersisted / null).
- era-marker toolTraceVersion (#490) больше не ставится полной переписью —
  ставится в маркере шага и на финализации (flushAssistant), остаётся
  консистентным.

Полная обратная совместимость: прогон, записанный по-старому (полная строка,
без строк шагов), восстанавливается/attach/export идентично. При отсутствии
репозитория шагов (позиционные тест-конструкции) — фолбэк на прежний
полнострочный flush (без регрессии, только без выигрыша WAL).

Тесты (реальный pg, int-lane):
- WAL-дельта (pg_current_wal_lsn) на прогоне 40×100КБ: new=4.3МБ vs
  old=90.3МБ (20.8x) — O(Σ шагов) против O(n²); старый путь в тесте И есть
  ревертнутое поведение (мутация-проверка).
- reconstruct-контракт: new-style (таблица шагов) и old-style (inline) прогоны
  восстанавливаются в идентичные parts; hydrate заполняет строку.
- миграция up/down roundtrip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:02:25 +03:00
agent_coder 64566e9327 test(deploy): согласовать reload-guard/chunk-load тесты с общим оконным бюджетом (#481, rebase reconcile)
После ребейза на develop версии-когерентность (#481, общий one-shot флаг)
и chunk-load boundary (#495, оконный guard) были сведены к ОДНОМУ общему
оконному бюджету в @/lib/reload-guard: не более одного авто-reload за
RELOAD_WINDOW_MS суммарно по обоим путям, при этом второй деплой в той же
вкладке восстанавливается.

- reload-guard.test.ts: переписаны под оконную семантику (ключ chunk-reload-at
  хранит таймстамп, а не флаг); сюда же перенесены чистые тесты shouldAutoReload.
- chunk-load-error-boundary.test.ts: убран импорт shouldAutoReload (переехал в
  reload-guard).
- chunk-load-error-boundary.tsx: handleError экспортирован для теста инварианта.
- reload-budget.integration.test.tsx: инвариант (a) на РЕАЛЬНОМ guard — reload
  на одном пути тратит общий бюджет для другого в пределах окна, после окна
  восстановление, при недоступном sessionStorage ни один путь не перезагружает.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:01:36 +03:00
agent_coder 10a4326fbf docs(deploy): поправить устаревший коммент про auto-reload скрытой вкладки под вариант C (#481, ревью F1)
r1-коммент утверждал «auto-reload on a hidden tab», но вариант C (r2) убрал
авто-reload скрытой вкладки ради защиты несохранённого ввода. Переписал под
фактическое поведение: баннер + отложенный reload на следующей in-app навигации.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:55:22 +03:00
agent_coder 68409a8ae9 fix(deploy): variant C — reload по in-app навигации вместо visibilitychange + лог-след + CHANGELOG (#481, ревью)
Правки по ревью #499. Владелец выбрал вариант C по вопросу потери несохранённого
ввода.

F1 (reload по навигации): убрал ветку visibilitychange→reload И немедленный reload
скрытой-при-получении вкладки — visibility больше не влияет на авто-reload вообще.
На реальном mismatch: баннер (кнопка «Обновить» — немедленный reload под тем же
one-shot гардом) + модульный флаг pendingNavReload. Хук useVersionReloadOnNavigation
(useLocation + useEffect по location.key, пропуск первого рендера через useRef;
react-router v7 BrowserRouter компонентный, объекта роутера нет — подписка изнутри
дерева) на СЛЕДУЮЩЕЙ навигации зовёт consumeNavigationReload → performAutoReload
(mark-перед-reload, при spent/непишущемся storage — только баннер). Скрытая вкладка
идёт тем же путём (C единообразно). chunk-load-error-boundary reload не тронут —
остаётся бэкапом для не-навигирующей вкладки. Так reload в безопасной точке (юзер
и так уходит со страницы), а не когда свернул с недописанным комментарием.

F2 (лог-след): общий recordReloadBreadcrumb/takeReloadBreadcrumb (sessionStorage,
переживает reload). performAutoReload пишет breadcrumb {path:proactive,server,
client} + console.warn перед reload; chunk-boundary пишет {path:chunk-boundary};
surfacePreviousReloadBreadcrumb на маунте UserProvider логирует прошлый след.

F3 (CHANGELOG + AGENTS.md): user-facing запись про version-coherence + строка про
артефакт client/dist/version.json.

Тесты: vitest version-coherence+reload-guard+guarded-reload 23 (переписан на
MemoryRouter+useNavigate: reload РОВНО раз на первой навигации после mismatch, НЕ
на 2-й навигации, НЕ на 2-м mismatch one-shot, НЕ от ухода в фон; скрытая вкладка
— только на навигации; Update — сразу; banner-only/write-fail — никогда). Шире:
features/user 34.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:55:22 +03:00
agent_coder fc624f5a4b feat(deploy): version-coherence — WS-анонс версии сервера + guarded reload устаревших вкладок (#481)
SPA — долгоживущий клиент: вкладку держат часами, сервер за это время
передеплоивают. Старый index.html ссылается на content-hashed чанки, которых в
новом образе нет → ленивый import() → catch-all отдаёт index.html как text/html →
WebKit «not a valid JavaScript MIME type» (наблюдали в проде). Реактивный
предохранитель (chunk-error-boundary) ловит УЖЕ случившуюся ошибку. Делаем сигнал
ПРОАКТИВНЫМ: сервер сообщает версию по существующему WS, клиент сравнивает и
делает guarded reload ДО битого чанка. closes #481

- Единый источник версии: vite.config.ts вычисляет resolveAppVersion РОВНО раз →
  и в define.APP_VERSION, и в плагине, пишущем dist/version.json. Бандл и файл
  тождественны by construction. Сервер читает client/dist/version.json при старте
  (client-version.ts: resolveClientDistPath вынесен из static.module — единый
  источник пути; readClientBuildVersion → версия или '' на любой ошибке,
  fail-safe). НИКАКИХ Docker/ENV-изменений.
- WS: отдельное событие app-version {version} (НЕ в message/WebSocketEvent —
  член без operation сломал бы union). ws.gateway читает версию в onModuleInit +
  boot-лог ACTIVE/DISABLED; emit в handleConnection (success-ветка, после join).
  ТОЛЬКО per-connect анонс, БЕЗ broadcast (broadcast в кластере → thundering-herd
  reload флота; естественный реконнект покрывает и single-container, и кластер).
- Клиент: listener навешан СИНХРОННО в том же useEffect, что создаёт сокет (до
  connect — иначе гонка с немедленным emit сервера). Чистая decideVersionAction
  (reload|banner|noop; пустая версия → noop fail-safe). Грязная оболочка
  triggerGuardedReload: модульный latch, hidden→немедленный reload, visible→баннер
  + reload-on-hidden {once}, фикс-id закрываемый баннер с кнопкой «Обновить».
- Общий one-shot флаг: reload-guard.ts (hasAutoReloaded/markAutoReloaded над
  существующим chunk-reload-attempted); chunk-load-error-boundary переведён на
  него. Проактивный + реактивный reload делят ОДИН счётчик → максимум одна
  авто-перезагрузка за сессию суммарно; permanent skew / oscillation / disabled
  storage → после первого reload только баннер, петли нет (проверено трассировкой).

Проверка: server jest client-version 7/7; client vitest version-coherence+
reload-guard+guarded-reload 16/16 (coverage 97.9%). Build-proof единого источника:
APP_VERSION=test-A → dist/version.json {"version":"test-A"} + вшито в бандл.
Полный staging-приём (docker build-arg, WS-кадры в DevTools, мульти-таб) — вне
автостенда, шаги приёмки #481 (крит.1-4) на ручную проверку.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:54:35 +03:00
agent_coder be70bd2e8e Merge remote-tracking branch 'gitea/develop' into fix/541-reconnect-timeout 2026-07-12 04:37:47 +03:00
agent_coder 2f8c5d9a98 perf(client): инкрементальный рендер стрима ответа (#492)
Путь ответа ассистента теперь рендерится инкрементально, по образцу
StreamingPlainText из ветки reasoning (#492, волна C эпика #497).

Раньше MarkdownPart прогонял ВЕСЬ накопленный ответ через канонический
конвейер (markdownToProseMirrorSync → PMNode.fromJSON → DOMSerializer →
DOMPurify) на КАЖДОМ throttled-тике (~20 Гц). На синтетическом потоке
~100 КБ это 394 вызова renderChatMarkdown — O(числа тиков), причём каждый
вызов заново парсит всю растущую строку.

Теперь:
- StreamingMarkdownText делит текст на блоки по безопасному срезу
  (splitPlainChunks — та же append-only-инвариантность, что у reasoning):
  СТАБИЛИЗИРОВАННЫЕ блоки идут через канонический конвейер и мемоизируются
  (каждый блок парсится РОВНО ОДИН раз), живой ХВОСТ — дешёвый plain-text
  (React-escaped, без парсера/санитайзера/innerHTML) до стабилизации.
- На финализации (флип state → done или конец хода) — ОДИН полный
  канонический рендер всего текста: побайтовая визуальная паритетность с
  прежним выводом (включая <li><p>-обёртки схемы и scoped-CSS из #498).
- Гейт liveness тот же, что у ReasoningBlock: streaming =
  turnStreaming && part.state === "streaming".

Также цикл рендера частей переведён на ИСЧЕРПЫВАЮЩИЙ switch по видам частей
с never-проверкой в default (вместо прежнего WARNING-комментария): новый
закрытый вид части в UIMessagePart теперь ошибка компиляции.

Тесты:
- perf-smoke: на ~100 КБ потоке число вызовов renderChatMarkdown ≤ blocks+2
  и ≪ ticks (O(блоков), не O(тиков)); мутация (снять memo с MarkdownChunk)
  краснит ассерт (78631 вызовов).
- visual-regression: финальный рендер побайтово равен renderChatMarkdown
  всего текста (в т.ч. <li><p>), инкрементальный вид сходится к нему на
  финализации; учтён neutralizeInternalLinks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:35:06 +03:00
agent_coder 69e04349a0 fix(client): вход в reconnect ограничивает ожидание getRun таймаутом (#541)
При живом обрыве SSE вход в лестницу reconnect происходил только
после резолва getRun(cid) (ре-сид из персиста). Путь REJECT
обрабатывался через .catch, но ЗАВИСШИЙ getRun (соединение есть,
ответа нет) был не ограничен: axios-клиент (lib/api-client.ts) не
имеет timeout, а stalled-idle-cap взводится только ПОСЛЕ входа в
reconnecting/polling. Итог — FSM залипала в `streaming` без баннера
и без поллинга до сокет-таймаута браузера (минуты). Это тот самый
класс тихого зависания, который устраняет эпик #497.

Оборачиваю ожидание getRun гонкой с таймаутом
(RECONNECT_RESEED_TIMEOUT_MS = 4s): по таймауту берётся ТОТ ЖЕ
фолбэк, что и на reject — dropLivePartialAndReplayFromStart() +
enterReconnect(runId), так что лестница/поллинг стартуют и
stalled-idle-cap взводится. Локальный флаг `settled` делает ветви
resolve/reject/timeout взаимоисключающими: поздний резолв getRun
после сработавшего таймаута полностью игнорируется (не входит в
reconnect повторно, не перетирает replay-from-start устаревшим
ре-сидом, ничего не перевзводит). Таймер живёт в reseedTimerRef и
очищается при размонтировании (никаких висящих setTimeout).

Тесты: hang-кейс (getRun не резолвится -> по таймауту FSM в
reconnecting, live-partial сброшен, replay-from-start) и
late-resolve safety (поздний резолв — no-op). Мутация: замена гонки
на голый getRun краснит оба #541-теста.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:29:25 +03:00
agent_coder 5bd5995ef0 fix(prosemirror-markdown): coalesce list seams on insert (#535)
insertNode with a markdown/JSON list now appends/prepends its items into
an adjacent same-type sibling list instead of creating a separate sibling
list that the serializer splits with `<!-- -->`.

Adds a single shared merge rule (isCoalescibleList/listsMergeable) used by
both the markdown path (insertNodesRelative) and the raw-node path
(insertNodeRelative). Coalescing is strictly local to the two seams of the
active insertion (each merged at most once — no greedy loop, no global
normalization). Survivor is chosen POSITIONALLY (the pre-existing neighbour
outside the inserted [i,j) range), so it keeps its block id and list-level
attrs (e.g. orderedList start); the inserted wrapper's re-minted id is
discarded. Handles the three-way case (list inserted between two same-type
lists → left survives) and refuses to coalesce an empty inserted list.

before/after now resolve via findAnchorChain and splice into the anchor's
immediate parent, so coalescing runs against the actual parent array
(incl. lists nested in callouts / table cells).

Serializer / LIST_MARKER_SEPARATOR untouched — two genuinely-separate
lists still emit `<!-- -->`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 02:43:53 +03:00
agent_coder 575125a5dc fix(mcp): enrich bad/inaccessible spaceId 404 into an actionable error (#534)
A well-formed but non-existent/inaccessible spaceId made the space-permissions
check answer with an opaque 404 ("Space permissions not found"), which the agent
could not self-correct from. Add an enrich-on-404 wrapper at the client-method
level (Variant A — no backend change) that, ONLY on that 404, replaces the server
text with a factual message naming the bad spaceId and the spaces the token can
actually see, pointing at listSpaces.

Mechanism (context.ts):
- getAccessibleSpaceIndex(): the token's accessible spaces from the single source
  of truth (/spaces), with a per-instance short-TTL cache (MCP_SPACES_CACHE_TTL_MS,
  default 60s; 0 disables) and single-flight dedup. Only a COMPLETE (untruncated)
  result is cached and only a complete result may drive the rewrite. The in-flight
  promise is nulled on BOTH resolve and reject so a transient /spaces blip is never
  memoized. Cache invalidated on every identity change, mirroring collabTokenCache
  (login() + the 401/403 reauth interceptor).
- paginateAllWithMeta(): surfaces the `truncated` flag paginateAll swallows;
  paginateAll now delegates to it (unchanged contract, getSpaces untouched).
- withSpaceAccessDiagnostics(spaceId, mcpName, fn): abort/cap wins FIRST (by the
  toolAbortSignal flag, NOT e.name — a cap may be TimeoutError/custom); only a 404
  is enrichable; FAILS OPEN (rethrows the original server error) on every source of
  uncertainty (fetch failed / !complete / spaceId present / aborted).

Wrapped only the paths where the sole 404 cause is the space membership check:
getTree, listPages (tree + recent-with-spaceId), search (with spaceId),
checkNewComments. createPage/getPageContext are deliberately NOT wrapped.
formatSpaceNotAccessible (errors.ts) composes the message (<=10 spaces + "(+N ещё)"
tail; distinct zero-spaces variant).

Tests: 11 new unit tests (stub client) covering all 9 acceptance criteria +
message shape; full mcp unit suite green (701 tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 02:20:12 +03:00
121 changed files with 11329 additions and 1584 deletions
+1
View File
@@ -463,6 +463,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, `apps/server` (#345), and `apps/client` (#347) — do NOT reintroduce a per-package copy. The client uses the package's `browser` entry (`@docmost/prosemirror-markdown/browser`): markdown paste (`markdown-clipboard.ts`), copy-as-markdown, and AI-chat rendering now all go through the canonical converter, so the hand-written `marked`/`turndown` markdown layer that used to live in `editor-ext` was deleted (#347). The browser entry runs the HTML→DOM stage on the native `DOMParser`, so jsdom stays out of the client bundle. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
- The build also emits `client/dist/version.json` (`{"version": …}`) from a small `vite.config.ts` plugin using the **same** `appVersion` that feeds `define.APP_VERSION`, so the file and the baked-in bundle version are identical by construction. The server reads it at startup (`ws.gateway.ts` via `readClientBuildVersion`/`resolveClientDistPath`) and announces it to each socket on connect (`app-version` event) so a tab left open across a redeploy can guard-reload before hitting a stale chunk (version-coherence). No runtime env / Dockerfile change — the file already ships in `client/dist`; missing/empty file ⇒ feature inert.
## Conventions
+12
View File
@@ -142,6 +142,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
snapshots switched from a fixed interval to a trailing idle-flush with a
max-wait ceiling, and a boundary snapshot is pinned whenever the editing source
changes (e.g. a person's edits followed by the AI agent). (#370)
- **Open tabs pick up a new deploy on their own.** After the server is
redeployed while a tab is left open for hours, the tab now learns the new
build version over the existing WebSocket (announced per-connect, so a natural
reconnect delivers it) and shows a "A new version is available" banner with an
Update button. To avoid dropping a half-written comment or form, the tab is
not reloaded when you merely switch away from it; instead it auto-reloads at
the next safe point — the next in-app navigation (or immediately if you click
Update) — before it can hit a stale lazy-loaded chunk. At most one automatic
reload happens per 5-minute window, shared with the existing chunk-load
recovery, so a permanent version skew degrades to the banner rather than a
reload loop while a second deploy in the same tab still recovers. When the
build carries no version info the feature stays inert. (#481)
- **Place several images side by side in a row.** A new "Inline (side by
side)" alignment mode in the image bubble menu renders consecutive inline
@@ -1,4 +1,5 @@
{
"A new version is available": "A new version is available",
"Account": "Account",
"Active": "Active",
"Add": "Add",
@@ -239,6 +240,8 @@
"Comment re-opened successfully": "Comment re-opened successfully",
"Comment unresolved successfully": "Comment unresolved successfully",
"Failed to resolve comment": "Failed to resolve comment",
"Failed to re-open comment": "Failed to re-open comment",
"Comment no longer exists": "Comment no longer exists",
"Resolve comment": "Resolve comment",
"Unresolve comment": "Unresolve comment",
"Resolve Comment Thread": "Resolve Comment Thread",
@@ -1427,5 +1430,20 @@
"Boundary": "Boundary",
"Autosave": "Autosave",
"Only versions": "Only versions",
"No saved versions yet.": "No saved versions yet."
"No saved versions yet.": "No saved versions yet.",
"Time worked on this article": "Time worked on this article",
"Show time worked on this page": "Show time worked on this page",
"Estimated time worked (inactivity gap {{gap}} min)": "Estimated time worked (inactivity gap {{gap}} min)",
"Estimate · timezone {{tz}} · inactivity gap {{gap}} min": "Estimate · timezone {{tz}} · inactivity gap {{gap}} min",
"No editing activity recorded yet.": "No editing activity recorded yet.",
"× {{count}} days without edits": "× {{count}} days without edits",
"agent: {{value}}": "agent: {{value}}",
"Work": "Work",
"Agent": "Agent",
"≈ {{hours}}h {{minutes}}m": "≈ {{hours}}h {{minutes}}m",
"≈ {{hours}}h": "≈ {{hours}}h",
"≈ {{minutes}}m": "≈ {{minutes}}m",
"{{hours}}h {{minutes}}m": "{{hours}}h {{minutes}}m",
"{{hours}}h": "{{hours}}h",
"{{minutes}}m": "{{minutes}}m"
}
@@ -1,4 +1,5 @@
{
"A new version is available": "Доступна новая версия",
"Account": "Аккаунт",
"Active": "Активный",
"Add": "Добавить",
@@ -239,6 +240,8 @@
"Comment re-opened successfully": "Комментарий успешно открыт повторно",
"Comment unresolved successfully": "Комментарий успешно переведён в нерешённые",
"Failed to resolve comment": "Не удалось разрешить комментарий",
"Failed to re-open comment": "Не удалось переоткрыть комментарий",
"Comment no longer exists": "Комментарий больше не существует",
"Resolve comment": "Решить комментарий",
"Unresolve comment": "Снять статус решённого с комментария",
"Resolve Comment Thread": "Решить ветку комментариев",
@@ -1442,5 +1445,20 @@
"Boundary": "Граница",
"Autosave": "Автосейв",
"Only versions": "Только версии",
"No saved versions yet.": "Пока нет сохранённых версий."
"No saved versions yet.": "Пока нет сохранённых версий.",
"Time worked on this article": "Время работы над статьёй",
"Show time worked on this page": "Показать время работы над страницей",
"Estimated time worked (inactivity gap {{gap}} min)": "Оценка времени работы (порог паузы {{gap}} мин)",
"Estimate · timezone {{tz}} · inactivity gap {{gap}} min": "Оценка · таймзона {{tz}} · порог паузы {{gap}} мин",
"No editing activity recorded yet.": "Правок пока нет.",
"× {{count}} days without edits": "× {{count}} дн. без правок",
"agent: {{value}}": "агент: {{value}}",
"Work": "Работа",
"Agent": "Агент",
"≈ {{hours}}h {{minutes}}m": "≈ {{hours}} ч {{minutes}} мин",
"≈ {{hours}}h": "≈ {{hours}} ч",
"≈ {{minutes}}m": "≈ {{minutes}} мин",
"{{hours}}h {{minutes}}m": "{{hours}} ч {{minutes}} м",
"{{hours}}h": "{{hours}} ч",
"{{minutes}}m": "{{minutes}} м"
}
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { isChunkLoadError, shouldAutoReload } from "./chunk-load-error-boundary";
import { isChunkLoadError } from "./chunk-load-error-boundary";
// The detector decides whether a caught render error is a stale-deploy chunk-404
// (→ auto-reload to fetch the new manifest) vs a genuine app error (→ generic
@@ -35,31 +35,3 @@ describe("isChunkLoadError", () => {
expect(isChunkLoadError(err)).toBe(false);
});
});
// The window gate replaces the old one-shot flag: it must permit recovery across
// several deploys in one tab (each > window apart) while still stopping an infinite
// reload loop when a lazy chunk is permanently broken (a second failure < window).
describe("shouldAutoReload", () => {
const WINDOW = 5 * 60 * 1000;
const NOW = 1_000_000_000_000;
it("allows a reload when we have never auto-reloaded", () => {
expect(shouldAutoReload(NOW, null, WINDOW)).toBe(true);
});
it("allows a reload when the last one was 6 minutes ago (outside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 6 * 60 * 1000, WINDOW)).toBe(true);
});
it("blocks a reload when the last one was 1 minute ago (inside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 1 * 60 * 1000, WINDOW)).toBe(false);
});
it("blocks a reload exactly at the window boundary (not strictly older)", () => {
expect(shouldAutoReload(NOW, NOW - WINDOW, WINDOW)).toBe(false);
});
it("allows a reload when the stored timestamp is unparseable (NaN)", () => {
expect(shouldAutoReload(NOW, NaN, WINDOW)).toBe(true);
});
});
@@ -1,26 +1,11 @@
import { ReactNode } from "react";
import { ErrorBoundary } from "react-error-boundary";
import { Button, Center, Stack, Text } from "@mantine/core";
// sessionStorage key holding the epoch-ms timestamp of the last automatic reload.
const RELOAD_AT_KEY = "chunk-reload-at";
// Allow at most one automatic reload per this window. A stale-deploy 404 is cured
// by a single reload, so anything inside the window is treated as a reload loop
// (permanently-broken chunk) and falls through to the manual UI. A window (rather
// than a one-shot flag) lets a SECOND deploy in the same tab's lifetime recover too.
const RELOAD_WINDOW_MS = 5 * 60 * 1000;
// Pure window decision, unit-tested in isolation: auto-reload only if we have never
// auto-reloaded (lastReloadAt null/NaN) or the last one was strictly older than the
// window. Anything inside the window is suppressed to break an infinite reload loop.
export function shouldAutoReload(
now: number,
lastReloadAt: number | null,
windowMs: number,
): boolean {
if (lastReloadAt === null || Number.isNaN(lastReloadAt)) return true;
return now - lastReloadAt > windowMs;
}
import {
hasAutoReloaded,
markAutoReloaded,
recordReloadBreadcrumb,
} from "@/lib/reload-guard";
// Heuristic detection of a failed dynamic import. Since the code-splitting work,
// every route (plus Aside / AiChatWindow) is React.lazy: when a new deploy
@@ -39,24 +24,26 @@ export function isChunkLoadError(error: unknown): boolean {
);
}
function handleError(error: unknown) {
// Exported for tests: the reactive chunk-load reload decision, so the shared
// window budget (invariant: ≤1 auto-reload per window across this path AND the
// proactive version-coherence path) can be exercised against the real guard.
export function handleError(error: unknown) {
if (!isChunkLoadError(error)) return;
// A stale-chunk 404 is cured by a full reload that re-fetches index.html and
// the new chunk manifest. Auto-reload at most once per RELOAD_WINDOW_MS: this
// recovers across multiple deploys in a single tab's lifetime, yet a
// permanently-broken lazy chunk (which would loop) is stopped after the first
// reload and falls through to the manual recovery UI below.
try {
const raw = sessionStorage.getItem(RELOAD_AT_KEY);
const lastReloadAt = raw === null ? null : Number.parseInt(raw, 10);
const now = Date.now();
if (!shouldAutoReload(now, lastReloadAt, RELOAD_WINDOW_MS)) return;
sessionStorage.setItem(RELOAD_AT_KEY, String(now));
} catch {
// sessionStorage unavailable (private mode / disabled): skip the automatic
// reload rather than risk an unguarded loop; the fallback UI still recovers.
return;
}
// the new chunk manifest. Auto-reload at most once per window via the SHARED
// window-based reload guard (see @/lib/reload-guard — the same budget the
// proactive version-coherence path consumes, so a mismatch that arrives on
// both paths reloads at most once per window across BOTH). This recovers
// across multiple deploys in a single tab's lifetime, yet a permanently-broken
// lazy chunk (which would loop) is stopped after the first reload and falls
// through to the manual recovery UI below. If the shared budget is already
// spent this window, or the stamp write fails (storage unavailable), we return
// without reloading rather than risk a loop.
if (hasAutoReloaded()) return;
if (!markAutoReloaded()) return;
// Trace before the reload clears the console (same diagnostic breadcrumb the
// proactive version-coherence path writes, tagged with this path).
recordReloadBreadcrumb({ path: "chunk-boundary" });
window.location.reload();
}
@@ -1254,4 +1254,88 @@ describe("ChatThread — live reconnect + stalled", () => {
});
expect(onResumeFallback).toHaveBeenCalledWith(false); // POLL_IDLE_CAP -> idle -> disarm
});
it("#541: getRun HANGS on a live disconnect — the timeout race still enters reconnect (no silent freeze in `streaming`)", async () => {
// MUTATION-VERIFY: revert the race to a bare `getRun(cid).then/.catch` and this
// reddens — a HUNG (never-settling, NOT rejected) getRun leaves the FSM stuck in
// `streaming` with no reconnect banner and no poll (the axios client sets no
// request timeout, and the stalled-idle cap only arms AFTER reconnecting/polling).
renderLive();
h.state.getRun.mockReset();
h.state.getRun.mockReturnValue(new Promise(() => {})); // getRun HANGS forever
await disconnect(); // live partial = liveMsg (id "a2")
expect(h.state.getRun).toHaveBeenCalledWith("c1");
// BEFORE the bound fires the FSM is still in the live turn — the very freeze bug.
expect(screen.queryByText(/reconnecting/i)).toBeNull();
// The recovery-start bound fires -> the SAME fallback as the reject path.
act(() => {
vi.advanceTimersByTime(4_000);
});
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
// The live partial (a2) was DROPPED from the store (replay-from-start, no stale
// tail-apply base). Mirrors the getRun-REJECT test's inspection.
const removedLivePartial = (
h.state.setMessages as unknown as {
mock: { calls: [unknown][] };
}
).mock.calls.some(([updater]) => {
if (typeof updater !== "function") return false;
const out = (updater as (p: { id: string }[]) => { id: string }[])([
{ id: "a2" },
{ id: "u1" },
]);
return !out.some((m) => m.id === "a2");
});
expect(removedLivePartial).toBe(true);
// ...and the anchor was NULLED -> replay-from-start (no ?anchor=&n= over the partial).
advanceToAttempt(1);
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
});
it("#541: a getRun resolve AFTER the timeout already fired is IGNORED (no double reconnect, no stale re-seed)", async () => {
// The timeout wins first and enters the ladder via replay-from-start. When the
// hung getRun FINALLY answers, its late `.then` must be a full no-op: it must not
// re-seed the store from the (now stale) persisted row, must not re-set the
// anchor, and must not re-enter reconnect. The local `settled` flag makes the
// resolve/reject/timeout branches mutually exclusive.
// MUTATION-VERIFY: drop the `settled` guard (let the late `.then` run) and the
// late re-seed re-sets the anchor (URL regains ?anchor=a2&n=3) + calls setMessages.
renderLive();
let resolveGetRun!: (v: unknown) => void;
h.state.getRun.mockReset();
h.state.getRun.mockReturnValue(
new Promise((r) => {
resolveGetRun = r;
}),
);
await disconnect();
act(() => {
vi.advanceTimersByTime(4_000); // bound fires -> replay-from-start, reconnecting
});
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
advanceToAttempt(1);
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
// Anchor is null -> replay-from-start URL (the pre-condition the late resolve must
// not undo).
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
const setMessagesCallsBefore = h.state.setMessages.mock.calls.length;
// NOW the hung getRun finally resolves with a persisted anchor (id a2, steps 3).
await act(async () => {
resolveGetRun(persistedAnchor());
await Promise.resolve();
});
// The late resolve did NOT re-seed the store...
expect(h.state.setMessages.mock.calls.length).toBe(setMessagesCallsBefore);
// ...did NOT re-set the anchor (URL stays replay-from-start, no ?anchor=&n=)...
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
// ...and did NOT trigger a fresh reconnect attach.
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
});
});
@@ -81,6 +81,17 @@ const STREAM_THROTTLE_MS = 50;
// THREAD now (the FSM owns polling->stalled); the window just polls while armed.
const DEGRADED_POLL_IDLE_MAX_MS = 10 * 60_000;
// #541: the bound on the persist re-seed (getRun) wait when ENTERING the reconnect
// ladder after a live SSE drop. The axios client (lib/api-client.ts) sets NO request
// timeout, and the stalled-idle cap only arms AFTER the FSM enters reconnecting/
// polling — which never happens if getRun HANGS (connection established, no response).
// Without this bound a live drop + a hung getRun sticks the FSM in `streaming` with
// no banner and no poll until the browser socket timeout (minutes) — the silent-freeze
// class #497 eliminates. This is a deliberate RECOVERY-START bound (drop the live
// partial + enter the ladder so the poll/idle-cap arms), intentionally much shorter
// than any network socket timeout — not a network read timeout.
const RECONNECT_RESEED_TIMEOUT_MS = 4_000;
/** The #487 active (non-terminal) run statuses — mirrors the server's
* ACTIVE_RUN_STATUSES. A run-fact is "active" only for these. */
function isActiveRunStatus(status: string | null | undefined): boolean {
@@ -286,6 +297,9 @@ export default function ChatThread({
// stalled inactivity cap. Cleared by the cancelReconnect effect / the cap effect.
const reconnectTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const idleCapTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// #541: the persist re-seed (getRun) timeout race on the disconnect->reconnect
// entry. Held in a ref so unmount (DISPOSE cleanup) clears it — no dangling timer.
const reseedTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// #491 tail-only: seed EVERY persisted row unchanged (no strip). The streaming
// tail holds steps 0..N-1; the run-stream registry's tail (steps >= N) is APPENDED
@@ -748,37 +762,68 @@ export default function ChatThread({
anchorRef.current = null;
};
if (cid) {
void getRun(cid)
.then((res) => {
if (!mountedRef.current) return;
const persisted = res.message;
if (persisted && persisted.role === "assistant") {
anchorRef.current = {
id: persisted.id,
stepsPersisted: stepsPersistedOf(persisted),
};
// Replace the live partial with the persisted row IN PLACE by id —
// the re-seed from persist. The attach's tail (steps >= N) then
// appends to a store holding EXACTLY steps 0..N-1: no duplication.
setMessages((prev) => mergeById(prev, rowToUiMessage(persisted)));
} else {
// No persisted assistant row (pre-first-frame break): drop the live
// partial + replay from start (no anchor/n) so nothing is duplicated.
dropLivePartialAndReplayFromStart();
}
enterReconnect(res.run?.id ?? runId);
})
.catch(() => {
if (!mountedRef.current) return;
// Persist read FAILED: we cannot re-seed from fresh persist, and a
// stale mount-time anchor over the live partial would tail-apply
// already-present steps -> duplication (a flaky-network blip:
// SSE + getRun both fail, network recovers in ~1s, the registry still
// covers from the mount frontier). Restore the removed-filter guarantee
// instead: drop the live partial + replay from start / 204 -> poll.
// #541: bound the persist re-seed wait with a timeout race. getRun goes
// through the axios client, which has NO request timeout; a HUNG getRun
// (connection open, no response) — distinct from a REJECT, which the
// `.catch` already handles — would otherwise never let us enter the ladder,
// leaving the FSM stuck in `streaming` with no banner and no poll until the
// browser socket timeout. `settled` makes the three branches (resolve /
// reject / timeout) mutually exclusive: whichever fires FIRST wins and the
// others become no-ops. So a LATE getRun resolve AFTER the timeout is fully
// ignored — it cannot (i) re-enter reconnect a second time, (ii) overwrite
// the timeout's replay-from-start with a stale re-seed, or (iii) re-arm any
// timer. On timeout we take the SAME fallback as the reject path (drop the
// live partial + enter the ladder, so the poll and the stalled-idle cap arm).
let settled = false;
const finishReseed = (apply: () => void): void => {
if (settled) return;
settled = true;
if (reseedTimerRef.current) {
clearTimeout(reseedTimerRef.current);
reseedTimerRef.current = null;
}
if (!mountedRef.current) return;
apply();
};
reseedTimerRef.current = setTimeout(() => {
finishReseed(() => {
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
});
}, RECONNECT_RESEED_TIMEOUT_MS);
void getRun(cid)
.then((res) => {
finishReseed(() => {
const persisted = res.message;
if (persisted && persisted.role === "assistant") {
anchorRef.current = {
id: persisted.id,
stepsPersisted: stepsPersistedOf(persisted),
};
// Replace the live partial with the persisted row IN PLACE by id —
// the re-seed from persist. The attach's tail (steps >= N) then
// appends to a store holding EXACTLY steps 0..N-1: no duplication.
setMessages((prev) => mergeById(prev, rowToUiMessage(persisted)));
} else {
// No persisted assistant row (pre-first-frame break): drop the live
// partial + replay from start (no anchor/n) so nothing is duplicated.
dropLivePartialAndReplayFromStart();
}
enterReconnect(res.run?.id ?? runId);
});
})
.catch(() => {
finishReseed(() => {
// Persist read FAILED: we cannot re-seed from fresh persist, and a
// stale mount-time anchor over the live partial would tail-apply
// already-present steps -> duplication (a flaky-network blip:
// SSE + getRun both fail, network recovers in ~1s, the registry still
// covers from the mount frontier). Restore the removed-filter guarantee
// instead: drop the live partial + replay from start / 204 -> poll.
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
});
});
} else {
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
@@ -903,6 +948,12 @@ export default function ChatThread({
}
return () => {
mountedRef.current = false;
// #541: clear the in-flight persist re-seed timeout (not an FSM effect timer,
// so DISPOSE does not touch it) — no dangling setTimeout after unmount.
if (reseedTimerRef.current) {
clearTimeout(reseedTimerRef.current);
reseedTimerRef.current = null;
}
dispatch({ type: "DISPOSE" }); // aborts attach + timers, bumps epoch (I5)
};
// Mount-only by design; the parent remounts per chat via `key`.
@@ -27,6 +27,7 @@ vi.mock("@/features/ai-chat/utils/markdown.ts", async () => {
import MessageItem from "./message-item";
import { messageSignature } from "@/features/ai-chat/utils/message-signature.ts";
import { splitPlainChunks } from "./streaming-plain-text";
// matchMedia (read by MantineProvider) is stubbed globally in vitest.setup.ts.
@@ -114,3 +115,89 @@ describe("MessageItem markdown memoization", () => {
expect(queryByText("streamed answer")).not.toBeNull();
});
});
// PERF SMOKE (#492): the whole point of the incremental streaming render is that
// the ANSWER path costs O(number of markdown blocks), NOT O(number of throttled
// ~20Hz ticks). Pre-#492 the finalized MarkdownPart re-parsed the WHOLE growing
// answer on every delta — a synthetic ~100 KB stream measured 394 renderChatMarkdown
// calls (one per tick). With the incremental render each STABILIZED block is parsed
// exactly once (memoized in MarkdownChunk) and the live tail is cheap plain text, so
// the call count collapses to ~= the block count regardless of tick granularity.
describe("MessageItem streaming answer render is O(blocks), not O(ticks)", () => {
// ~100 KB answer. Each section is a heading + a paragraph — TWO blank-line
// delimited markdown blocks — so the safe-cut block count is ~2× the section
// count. The perf claim is about the BLOCK count (the memoization granularity),
// measured directly with splitPlainChunks below, not the section count.
const buildAnswer = () => {
const SECTIONS = 100;
const paragraphs: string[] = [];
for (let i = 0; i < SECTIONS; i++) {
paragraphs.push(`## Section ${i}\n\n` + "lorem ipsum dolor ".repeat(55));
}
const full = paragraphs.join("\n\n");
// The number of memoized markdown blocks the incremental render splits into
// (all but the live tail are parsed once each).
return { full, blocks: splitPlainChunks(full).length };
};
const streamMsg = (text: string, state: "streaming" | "done"): UIMessage =>
({
id: "m1",
role: "assistant",
parts: [{ type: "text", text, state }],
}) as UIMessage;
it("parses each block ~once over a 100KB stream (≈blocks, ≪ ticks)", () => {
renderChatMarkdownSpy.mockClear();
const { full, blocks } = buildAnswer();
const CHUNK = 128; // a realistic ~20Hz throttled delta size
const ticks = Math.ceil(full.length / CHUNK);
let msg = streamMsg(full.slice(0, CHUNK), "streaming");
const { rerender } = render(
<MantineProvider>
<MessageItem
message={msg}
signature={messageSignature(msg)}
turnStreaming
/>
</MantineProvider>,
);
for (let end = 2 * CHUNK; end < full.length; end += CHUNK) {
msg = streamMsg(full.slice(0, end), "streaming");
rerender(
<MantineProvider>
<MessageItem
message={msg}
signature={messageSignature(msg)}
turnStreaming
/>
</MantineProvider>,
);
}
// Finalize: the streaming→done flip renders the whole answer through ONE
// canonical pass (visual parity), so the finished DOM matches the pre-#492
// output. This is the single extra parse on top of the per-block ones.
const done = streamMsg(full, "done");
rerender(
<MantineProvider>
<MessageItem message={done} signature={messageSignature(done)} />
</MantineProvider>,
);
const calls = renderChatMarkdownSpy.mock.calls.length;
// Sanity: the stream really had far more ticks than blocks (else the test is
// vacuous — the point is that calls scale with blocks, not ticks).
expect(ticks).toBeGreaterThan(blocks * 3);
// O(blocks): each stabilized block parsed once + the single final whole-text
// parse. A small constant absorbs the finalize render and the live-tail block;
// the load-bearing claim is the bound below.
expect(calls).toBeLessThanOrEqual(blocks + 2);
// ≪ ticks — and, non-vacuously, the blocks WERE parsed (not skipped entirely).
expect(calls).toBeLessThan(ticks / 3);
expect(calls).toBeGreaterThan(blocks / 2);
// MUTATION-VERIFY (documented, not run here): dropping the `memo()` wrapper on
// MarkdownChunk (so every stable block re-parses each tick) drives `calls`
// toward `ticks` (~394), reddening both upper-bound assertions above.
});
});
@@ -0,0 +1,112 @@
import { describe, expect, it, vi } from "vitest";
import { render } from "@testing-library/react";
import { MantineProvider } from "@mantine/core";
import type { UIMessage } from "@ai-sdk/react";
// Stub react-i18next (the component reads `useTranslation`). Mirrors the other
// message-item specs.
vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (key: string) => key }),
}));
import MessageItem from "./message-item";
import { messageSignature } from "@/features/ai-chat/utils/message-signature.ts";
// The REAL canonical renderer (NOT the spy the memo test installs): this file
// exercises the actual markdown output so the visual-regression assertions below
// compare against genuine HTML (incl. the schema's `<li><p>` wrappers).
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
import classes from "./ai-chat.module.css";
const msg = (
parts: UIMessage["parts"],
extra?: Partial<UIMessage>,
): UIMessage =>
({ id: "m1", role: "assistant", parts, ...extra }) as UIMessage;
const renderRow = (message: UIMessage, turnStreaming = false) =>
render(
<MantineProvider>
<MessageItem
message={message}
signature={messageSignature(message)}
turnStreaming={turnStreaming}
/>
</MantineProvider>,
);
// A rich multi-block answer that exercises headings, a list (the `<li><p>` case
// the scoped CSS tightens), inline emphasis, and multiple paragraphs.
const ANSWER = [
"# Заголовок",
"",
"Первый абзац с **жирным** и `кодом`.",
"",
"- пункт один",
"- пункт два",
"",
"Второй абзац.",
].join("\n");
describe("MessageItem final render — visual parity with the canonical pipeline", () => {
it("a finalized text part renders exactly renderChatMarkdown(text)", () => {
const { container } = renderRow(
msg([{ type: "text", text: ANSWER, state: "done" }]),
);
const block = container.querySelector(`.${classes.markdown}`);
expect(block).not.toBeNull();
// Byte-for-byte the canonical output (the SAME whole-text pass the pre-#492
// MarkdownPart produced), including `<li><p>…</p></li>` wrappers.
expect(block!.innerHTML).toBe(renderChatMarkdown(ANSWER, {}));
// The list wrapper is really present (guards against a vacuous empty render).
expect(container.querySelectorAll("li p").length).toBe(2);
});
it("the streaming incremental view CONVERGES to the canonical render on finish", () => {
// Mount mid-stream (live tail) — the DOM here is the incremental view.
const { container, rerender } = render(
<MantineProvider>
<MessageItem
message={msg([{ type: "text", text: ANSWER, state: "streaming" }])}
signature={messageSignature(
msg([{ type: "text", text: ANSWER, state: "streaming" }]),
)}
turnStreaming
/>
</MantineProvider>,
);
// Finish the turn: state flips to done AND the turn is no longer streaming.
const done = msg([{ type: "text", text: ANSWER, state: "done" }]);
rerender(
<MantineProvider>
<MessageItem message={done} signature={messageSignature(done)} />
</MantineProvider>,
);
// After finish there is exactly ONE canonical markdown container whose HTML is
// the whole-text render — identical to the non-streaming path above.
const blocks = container.querySelectorAll(`.${classes.markdown}`);
expect(blocks.length).toBe(1);
expect(blocks[0].innerHTML).toBe(renderChatMarkdown(ANSWER, {}));
});
it("neutralizeInternalLinks is honored on the finalized render", () => {
const linkAnswer = "См. [страницу](/p/abc).";
const { container } = render(
<MantineProvider>
<MessageItem
message={msg([{ type: "text", text: linkAnswer, state: "done" }])}
signature={messageSignature(
msg([{ type: "text", text: linkAnswer, state: "done" }]),
)}
neutralizeInternalLinks
/>
</MantineProvider>,
);
const block = container.querySelector(`.${classes.markdown}`);
expect(block!.innerHTML).toBe(
renderChatMarkdown(linkAnswer, { neutralizeInternalLinks: true }),
);
// The internal link was made inert (no href) by the neutralization flag.
const a = container.querySelector("a");
expect(a?.hasAttribute("href")).toBe(false);
});
});
@@ -4,6 +4,7 @@ import { useTranslation } from "react-i18next";
import type { UIMessage } from "@ai-sdk/react";
import ToolCallCard from "@/features/ai-chat/components/tool-call-card.tsx";
import ReasoningBlock from "@/features/ai-chat/components/reasoning-block.tsx";
import { StreamingMarkdownText } from "@/features/ai-chat/components/streaming-markdown-text.tsx";
import ChatErrorAlert from "@/features/ai-chat/components/chat-error-alert.tsx";
import ChatStoppedNotice from "@/features/ai-chat/components/chat-stopped-notice.tsx";
import { ToolUiPart, isToolPart } from "@/features/ai-chat/utils/tool-parts.tsx";
@@ -86,17 +87,39 @@ interface MessageItemProps {
* One assistant text part rendered as sanitized markdown. Memoized on its inputs
* so a finalized text part is NOT re-parsed on every streamed delta: during a
* turn only the actively-growing tail part changes its `text`, so every earlier
* part hits the memo and skips the expensive marked + DOMPurify pass. Props are
* primitives, so React.memo's default shallow compare is exactly right (the
* `text` string is compared by value).
* part hits the memo and skips the expensive canonical parse + DOMPurify pass.
* Props are primitives, so React.memo's default shallow compare is exactly right
* (the `text` string is compared by value).
*
* Streaming gate (#492) — mirrors ReasoningBlock:
* - `streaming` (this is the live, actively-growing tail part of an in-flight
* turn): render incrementally via StreamingMarkdownText — the stabilized blocks
* go through the canonical pipeline (each parsed ONCE, memoized) and only the
* live tail is cheap plain text. This makes the per-tick cost O(new blocks),
* not the pre-#492 O(ticks) whole-answer re-parse on every ~20Hz delta.
* - finalized (the common case, and the turn-end flip): render the WHOLE text
* through ONE canonical pass — byte-identical to the pre-#492 output (visual
* parity). The row re-renders on the streaming→done flip because
* `messageSignature` tracks each part's `state` (and `turnStreaming` flips at
* turn end), so the incremental view always converges to this single render.
*/
const MarkdownPart = memo(function MarkdownPart({
text,
neutralizeInternalLinks,
streaming,
}: {
text: string;
neutralizeInternalLinks: boolean;
streaming: boolean;
}) {
if (streaming) {
return (
<StreamingMarkdownText
text={text}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
);
}
const html = renderChatMarkdown(text, { neutralizeInternalLinks });
if (html) {
return (
@@ -179,47 +202,10 @@ function MessageItem({
{resolveAssistantName(assistantName) ?? t("AI agent")}
</Text>
{message.parts.map((part, index) => {
if (part.type === "reasoning") {
// Reasoning ("thinking") -> a collapsible block with its own token
// count. Empty/whitespace reasoning with no authoritative count carries
// nothing to show, so skip it (avoids an empty 0-token block).
const text = (part as { text?: string }).text ?? "";
if (!text.trim() && !(reasoningTokens && reasoningTokens > 0))
return null;
// Absent state (persisted rows) and "done" both mean finalized.
// `messageSignature` already includes each part's `state`, so the
// streaming→done flip changes the row signature and re-renders this
// row — which is what lets ReasoningBlock switch from chunked plain
// text to its one-time markdown parse (see reasoning-block.tsx).
// ALSO require the turn to be live: a part stranded at
// `state:"streaming"` after the turn ended (no `reasoning-end` — see
// the `turnStreaming` prop doc) must still finalize and parse.
const streaming =
turnStreaming && (part as { state?: string }).state === "streaming";
return (
<ReasoningBlock
key={index}
text={text}
tokens={reasoningTokens}
streaming={streaming}
/>
);
}
if (part.type === "text") {
// Skip empty/whitespace-only text parts (a streaming message often
// starts with an empty text part before the first token arrives); the
// typing indicator covers that gap until real content streams in.
if (!part.text.trim()) return null;
return (
<MarkdownPart
key={index}
text={part.text}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
);
}
// Tool parts (`tool-*` / `dynamic-tool`) are template-literal kinds, so
// they cannot be a `switch` case; the runtime guard handles them, and the
// switch below covers every CLOSED (literal-typed) part kind with a
// compile-time exhaustiveness check in its default.
if (isToolPart(part.type)) {
return (
<ToolCallCard
@@ -232,7 +218,76 @@ function MessageItem({
);
}
return null;
switch (part.type) {
case "reasoning": {
// Reasoning ("thinking") -> a collapsible block with its own token
// count. Empty/whitespace reasoning with no authoritative count
// carries nothing to show, so skip it (avoids an empty 0-token block).
const text = part.text ?? "";
if (!text.trim() && !(reasoningTokens && reasoningTokens > 0))
return null;
// Absent state (persisted rows) and "done" both mean finalized.
// `messageSignature` already includes each part's `state`, so the
// streaming→done flip changes the row signature and re-renders this
// row — which is what lets ReasoningBlock switch from chunked plain
// text to its one-time markdown parse (see reasoning-block.tsx).
// ALSO require the turn to be live: a part stranded at
// `state:"streaming"` after the turn ended (no `reasoning-end` — see
// the `turnStreaming` prop doc) must still finalize and parse.
const streaming = turnStreaming && part.state === "streaming";
return (
<ReasoningBlock
key={index}
text={text}
tokens={reasoningTokens}
streaming={streaming}
/>
);
}
case "text": {
// Skip empty/whitespace-only text parts (a streaming message often
// starts with an empty text part before the first token arrives); the
// typing indicator covers that gap until real content streams in.
if (!part.text.trim()) return null;
// The live, actively-growing tail part of the in-flight turn renders
// incrementally (see MarkdownPart); a finalized part (persisted, or
// the turn-end flip) renders the whole text through one canonical
// pass. Same liveness rule as the reasoning branch above.
const streaming = turnStreaming && part.state === "streaming";
return (
<MarkdownPart
key={index}
text={part.text}
neutralizeInternalLinks={neutralizeInternalLinks}
streaming={streaming}
/>
);
}
case "source-url":
case "source-document":
case "file":
case "step-start":
// Not surfaced in the chat bubble (v1) — same as the pre-#492 default.
return null;
default: {
// Compile-time exhaustiveness over the CLOSED union members: every
// literal-typed part kind is handled above, so the only kinds that
// can reach here are the OPEN template-literal ones (`tool-*` — caught
// by the guard at runtime — and `data-*`) plus `dynamic-tool`. Adding
// a NEW closed part kind to UIMessagePart makes this assignment fail
// to compile, forcing it to be handled instead of silently ignored
// (this replaces the pre-#492 fall-through `return null` + WARNING).
const _exhaustive:
| `tool-${string}`
| "dynamic-tool"
| `data-${string}` = part.type;
void _exhaustive;
return null;
}
}
})}
{/* A persisted turn error (server stored it in metadata.error). Rendered
here so it survives a thread remount and shows in reopened history. */}
@@ -0,0 +1,96 @@
import { memo, useMemo } from "react";
import { splitPlainChunks } from "@/features/ai-chat/components/streaming-plain-text.tsx";
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
import classes from "@/features/ai-chat/components/ai-chat.module.css";
/**
* One STABILIZED markdown block, rendered through the canonical pipeline and
* memoized on its string prop. During streaming only the TAIL chunk grows (the
* `splitPlainChunks` append-only invariant guarantees every earlier chunk is
* byte-identical across deltas), so React skips every stable block and each one
* is parsed by `renderChatMarkdown` EXACTLY ONCE — turning the pre-#492
* "re-parse the whole accumulated answer on every ~20Hz tick" (O(ticks)) into
* O(number of blocks). The markup is DOMPurify-sanitized inside renderChatMarkdown
* before it reaches `dangerouslySetInnerHTML`.
*
* NOTE (transient streaming-only artifact): a safe cut is a blank-line boundary,
* so a construct that legitimately contains a blank line (e.g. a fenced code block
* with an empty line) can be split across chunks and render oddly WHILE it is still
* streaming. This is cosmetic and self-heals: the moment the part finalizes,
* MarkdownPart renders the WHOLE text through one canonical pass (visual parity
* with the pre-#492 output). The reasoning path makes the same trade (plain text
* while streaming, one markdown parse at the end).
*/
const MarkdownChunk = memo(function MarkdownChunk({
text,
neutralizeInternalLinks,
}: {
text: string;
neutralizeInternalLinks: boolean;
}) {
const html = renderChatMarkdown(text, { neutralizeInternalLinks });
if (html) {
return (
<div
className={classes.markdown}
// Sanitized by renderChatMarkdown (DOMPurify) before insertion.
dangerouslySetInnerHTML={{ __html: html }}
/>
);
}
// Malformed/unsupported markdown could not render synchronously: raw text.
return (
<div className={classes.markdown} style={{ whiteSpace: "pre-wrap" }}>
{text}
</div>
);
});
/**
* The cheap streaming-time stand-in for the finalized answer's one-time markdown
* parse (see MarkdownPart in message-item.tsx). Mirrors StreamingPlainText's
* chunked-memo pattern but renders the STABILIZED prefix as real markdown (each
* block parsed once, memoized) and only the LIVE tail as flat plain text — so the
* user sees formatted output for everything up to the last safe cut, and the not-
* yet-stable tail (which markdown-parsing every tick would make O(ticks)) stays a
* single cheap escaped text node until it stabilizes into a new block.
*
* `splitPlainChunks` yields chunks where, under append-only growth, every chunk
* except the LAST is immutable; the last chunk is the live tail. Index keys are
* therefore stable (a given index never changes to a different chunk's content).
*/
export function StreamingMarkdownText({
text,
neutralizeInternalLinks,
}: {
text: string;
neutralizeInternalLinks: boolean;
}) {
const chunks = useMemo(() => splitPlainChunks(text), [text]);
return (
<>
{chunks.map((chunk, index) =>
index < chunks.length - 1 ? (
<MarkdownChunk
key={index}
text={chunk}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
) : (
// The live tail: flat, React-escaped plain text (no markdown parse, no
// sanitizer, no innerHTML). `pre-wrap` preserves its newlines; trailing
// separator newlines are dropped at display time so the block gap comes
// from the markdown margins, not a doubled empty line (mirrors
// PlainChunk in streaming-plain-text.tsx).
<div
key={index}
className={classes.markdown}
style={{ whiteSpace: "pre-wrap" }}
>
{chunk.replace(/\n+$/, "")}
</div>
),
)}
</>
);
}
@@ -27,11 +27,15 @@ vi.mock("@/features/space/queries/space-query.ts", () => ({
import {
buildChildrenByParent,
CommentEditorWithActions,
sortResolvedByResolvedAt,
} from "./comment-list-with-tabs";
const c = (id: string, parentCommentId: string | null = null): IComment =>
({ id, parentCommentId }) as IComment;
const resolvedAtComment = (id: string, resolvedAt: unknown): IComment =>
({ id, resolvedAt }) as unknown as IComment;
describe("buildChildrenByParent (childrenByParent grouping)", () => {
it("returns an empty map for undefined or empty input", () => {
expect(buildChildrenByParent(undefined).size).toBe(0);
@@ -71,6 +75,48 @@ describe("buildChildrenByParent (childrenByParent grouping)", () => {
});
});
describe("sortResolvedByResolvedAt (Resolved tab order, #542)", () => {
it("orders by resolvedAt DESC — newest resolve first — with ISO-STRING values", () => {
// At runtime resolvedAt is an ISO string (axios JSON / WS subscription), so
// the sort must coerce with new Date(...) before .getTime().
const older = resolvedAtComment("older", "2026-07-10T10:00:00.000Z");
const newest = resolvedAtComment("newest", "2026-07-12T10:00:00.000Z");
const middle = resolvedAtComment("middle", "2026-07-11T10:00:00.000Z");
const out = sortResolvedByResolvedAt([older, newest, middle]);
expect(out.map((x) => x.id)).toEqual(["newest", "middle", "older"]);
});
it("also handles Date instances (optimistic onMutate window)", () => {
const older = resolvedAtComment("older", new Date("2026-01-01T00:00:00Z"));
const newer = resolvedAtComment("newer", new Date("2026-06-01T00:00:00Z"));
expect(sortResolvedByResolvedAt([older, newer]).map((x) => x.id)).toEqual([
"newer",
"older",
]);
});
it("does not mutate the input array", () => {
const a = resolvedAtComment("a", "2026-01-01T00:00:00.000Z");
const b = resolvedAtComment("b", "2026-02-01T00:00:00.000Z");
const input = [a, b];
sortResolvedByResolvedAt(input);
expect(input.map((x) => x.id)).toEqual(["a", "b"]);
});
it("keeps stable order for equal resolvedAt timestamps", () => {
const ts = "2026-03-03T03:03:03.000Z";
const x = resolvedAtComment("x", ts);
const y = resolvedAtComment("y", ts);
const z = resolvedAtComment("z", ts);
expect(sortResolvedByResolvedAt([x, y, z]).map((c) => c.id)).toEqual([
"x",
"y",
"z",
]);
});
});
function renderReplyEditor() {
return render(
<MantineProvider>
@@ -53,6 +53,22 @@ export function buildChildrenByParent(
return m;
}
// Sort the Resolved tab by resolve time, newest first, on a COPY (never mutate
// the react-query cache array). `resolvedAt` is typed `Date` but at runtime it
// is an ISO STRING (from the axios-JSON onSuccess and the WS subscription) — a
// real Date only during the optimistic onMutate window — so it MUST be coerced
// with `new Date(...)` before `.getTime()`, or a raw `.getTime()` on the string
// throws / yields NaN. ES2019's stable sort preserves order for equal
// timestamps. Callers pass a list already filtered to a truthy `resolvedAt`, so
// the non-null assertion is safe.
// Exported for unit testing.
export function sortResolvedByResolvedAt(resolved: IComment[]): IComment[] {
return [...resolved].sort(
(a, b) =>
new Date(b.resolvedAt!).getTime() - new Date(a.resolvedAt!).getTime(),
);
}
function CommentListWithTabs({ onClose }: CommentListWithTabsProps) {
const { t } = useTranslation();
const { pageSlug } = useParams();
@@ -91,7 +107,10 @@ function CommentListWithTabs({ onClose }: CommentListWithTabsProps) {
(comment: IComment) => comment.resolvedAt,
);
return { activeComments: active, resolvedComments: resolved };
return {
activeComments: active,
resolvedComments: sortResolvedByResolvedAt(resolved),
};
}, [comments]);
// Index replies by their parent once, instead of an O(n^2) filter per thread.
@@ -0,0 +1,353 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import React from "react";
import { renderHook, waitFor } from "@testing-library/react";
import {
QueryClient,
QueryClientProvider,
InfiniteData,
} from "@tanstack/react-query";
/**
* Coverage for the resolve/reopen mutation (#542): the Undo-in-toast reopen and
* its double-click guard, the terminal 404 branch (drop from cache + clear the
* inline mark, no rollback), and the directional error copy.
*/
// A fake TipTap editor injected via the mocked pageEditorAtom, so we can assert
// the mutation clears the inline comment mark (unsetComment / setCommentResolved).
const editorMock = vi.hoisted(() => ({
current: {
isDestroyed: false,
commands: { unsetComment: vi.fn(), setCommentResolved: vi.fn() },
} as {
isDestroyed: boolean;
commands: {
unsetComment: (id: string) => void;
setCommentResolved: (id: string, v: boolean) => void;
};
} | null,
}));
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn(), hide: vi.fn() },
}));
vi.mock("jotai", () => ({
atom: (v: unknown) => v,
useAtomValue: () => editorMock.current,
}));
vi.mock("@/features/comment/services/comment-service", () => ({
applySuggestion: vi.fn(),
dismissSuggestion: vi.fn(),
createComment: vi.fn(),
updateComment: vi.fn(),
deleteComment: vi.fn(),
resolveComment: vi.fn(),
getPageComments: vi.fn(),
}));
import { notifications } from "@mantine/notifications";
import { resolveComment } from "@/features/comment/services/comment-service";
import {
useResolveCommentMutation,
RESOLVE_UNDO_AUTOCLOSE_MS,
RQ_KEY,
} from "@/features/comment/queries/comment-query";
import { IComment } from "@/features/comment/types/comment.types";
const PAGE_ID = "page-1";
function seededClient(comment: IComment) {
const queryClient = new QueryClient({
defaultOptions: { mutations: { retry: false } },
});
const seed: InfiniteData<any> = {
pageParams: [undefined],
pages: [
{ items: [comment], meta: { hasNextPage: false, nextCursor: null } },
],
};
queryClient.setQueryData(RQ_KEY(PAGE_ID), seed);
const wrapper = ({ children }: { children: React.ReactNode }) => (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
);
return { queryClient, wrapper };
}
function items(queryClient: QueryClient): IComment[] {
const cache = queryClient.getQueryData(RQ_KEY(PAGE_ID)) as
| InfiniteData<any>
| undefined;
return cache?.pages.flatMap((p) => p.items) ?? [];
}
const comment = (over?: Partial<IComment>): IComment =>
({
id: "c-1",
pageId: PAGE_ID,
content: "{}",
creatorId: "u-1",
workspaceId: "ws-1",
createdAt: new Date(),
resolvedAt: null,
...over,
}) as IComment;
// Pull the inline Undo button's onClick out of the success toast's message tree.
function undoOnClickFromToast(): () => void {
const call = vi
.mocked(notifications.show)
.mock.calls.map((c) => c[0])
.find((arg: any) => arg?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS);
expect(call).toBeTruthy();
const message: any = (call as any).message;
// message = Group( Text, Button ); grab the Button element's onClick.
const children = message.props.children as any[];
const button = children[1];
return button.props.onClick;
}
describe("useResolveCommentMutation — Undo toast (#542)", () => {
beforeEach(() => {
vi.clearAllMocks();
editorMock.current = {
isDestroyed: false,
commands: { unsetComment: vi.fn(), setCommentResolved: vi.fn() },
};
});
it("resolve shows an Undo toast with autoClose=10000ms; reopen shows NO Undo", async () => {
vi.mocked(resolveComment).mockImplementation(async (data) =>
comment({
resolvedAt: data.resolved ? (new Date() as any) : null,
}),
);
const { wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const resolveToast = vi
.mocked(notifications.show)
.mock.calls.map((c) => c[0])
.find((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS);
expect(resolveToast).toBeTruthy();
expect((resolveToast as any).id).toBe("resolve-undo-c-1");
expect((resolveToast as any).autoClose).toBe(10000);
// Now a reopen → plain toast, no autoClose/Undo, no id.
vi.clearAllMocks();
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: false,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const calls = vi.mocked(notifications.show).mock.calls.map((c) => c[0]);
expect(
calls.some((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS),
).toBe(false);
expect(calls).toContainEqual({ message: "Comment re-opened successfully" });
});
it("double/fast Undo click fires reopen EXACTLY once (guard)", async () => {
vi.mocked(resolveComment).mockImplementation(async (data) =>
comment({ resolvedAt: data.resolved ? (new Date() as any) : null }),
);
const { wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const onClick = undoOnClickFromToast();
// Fire twice synchronously (notifications.hide is not synchronous).
onClick();
onClick();
await waitFor(() => {
const reopenCalls = vi
.mocked(resolveComment)
.mock.calls.filter(([d]) => d.resolved === false);
expect(reopenCalls).toHaveLength(1);
});
// The mark was cleared once via setCommentResolved(id, false).
expect(editorMock.current!.commands.setCommentResolved).toHaveBeenCalledWith(
"c-1",
false,
);
// The toast was hidden.
expect(notifications.hide).toHaveBeenCalledWith("resolve-undo-c-1");
});
it("404 → drops the comment from cache, clears the inline mark, no rollback, no Undo", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 404 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
// Removed from cache (NOT rolled back to a phantom).
expect(items(queryClient)).toHaveLength(0);
// Inline mark cleared via unsetComment (mandatory — no panel row left to do it).
expect(editorMock.current!.commands.unsetComment).toHaveBeenCalledWith(
"c-1",
);
// Neutral message, red, and crucially NOT the success copy and NO Undo toast.
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment no longer exists",
color: "red",
});
const calls = vi.mocked(notifications.show).mock.calls.map((c) => c[0]);
expect(
calls.some((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS),
).toBe(false);
});
it("404 does not crash when the editor is gone (read-only / panel closed)", async () => {
editorMock.current = null;
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 404 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(items(queryClient)).toHaveLength(0);
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment no longer exists",
color: "red",
});
});
it("non-404 error on REOPEN shows 'Failed to re-open comment' and rolls back", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
// Seed a RESOLVED comment (the reopen target).
const resolved = comment({ resolvedAt: new Date() as any });
const { queryClient, wrapper } = seededClient(resolved);
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: false })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(notifications.show).toHaveBeenCalledWith({
message: "Failed to re-open comment",
color: "red",
});
expect(notifications.show).not.toHaveBeenCalledWith(
expect.objectContaining({ message: "Failed to resolve comment" }),
);
// Rolled back: the comment is still present and still resolved.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeTruthy();
});
it("reopen via Undo FAILS (non-404) → inline mark is NOT left cleared (doc↔panel stay consistent)", async () => {
// First resolve succeeds → produces the Undo toast (no mark change on resolve).
vi.mocked(resolveComment).mockResolvedValueOnce(
comment({ resolvedAt: new Date() as any }),
);
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
// Now the reopen fired by Undo fails with a 500.
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
const onClick = undoOnClickFromToast();
onClick();
await waitFor(() => expect(result.current.isError).toBe(true));
// Core F1 guarantee: the mark-clear now lives in the reopen onSuccess, so a
// FAILED reopen must never flip the inline mark to unresolved — otherwise the
// doc would show an active highlight the panel still treats as resolved and
// the collab mark would diverge with nothing committed on the server.
expect(
editorMock.current!.commands.setCommentResolved,
).not.toHaveBeenCalledWith("c-1", false);
// Cache rolled back: the comment stays resolved and present.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeTruthy();
});
it("reopen success with a null editorRef degrades gracefully (no throw, no-op)", async () => {
// Read-only view / panel closed: pageEditorAtom is null on the success path.
editorMock.current = null;
vi.mocked(resolveComment).mockResolvedValue(comment({ resolvedAt: null }));
const resolved = comment({ resolvedAt: new Date() as any });
const { queryClient, wrapper } = seededClient(resolved);
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: false,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
// No crash from the reopen mark-clear; the plain reopen toast is still shown.
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment re-opened successfully",
});
// Cache updated to reopened (resolvedAt cleared by the server payload).
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeFalsy();
});
it("non-404 error on RESOLVE shows 'Failed to resolve comment' and rolls back", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(notifications.show).toHaveBeenCalledWith({
message: "Failed to resolve comment",
color: "red",
});
// Rolled back to open (previousCache), still present.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeFalsy();
});
});
@@ -20,12 +20,19 @@ import {
ISuggestionOutcome,
} from "@/features/comment/types/comment.types";
import { notifications } from "@mantine/notifications";
import { Button, Group, Text } from "@mantine/core";
import { IPagination } from "@/lib/types.ts";
import { useTranslation } from "react-i18next";
import { useEffect, useMemo } from "react";
import React, { useEffect, useMemo, useRef } from "react";
import { useAtomValue } from "jotai";
import { pageEditorAtom } from "@/features/editor/atoms/editor-atoms";
export const RQ_KEY = (pageId: string) => ["comments", pageId];
// How long the resolve success toast (with its inline Undo) stays up before it
// auto-closes. Policy constant — no env override.
export const RESOLVE_UNDO_AUTOCLOSE_MS = 10000;
export function useCommentsQuery(params: ICommentParams) {
const query = useInfiniteQuery({
queryKey: RQ_KEY(params.pageId),
@@ -376,7 +383,25 @@ export function useResolveCommentMutation() {
const queryClient = useQueryClient();
const { t } = useTranslation();
return useMutation({
// Keep the live editor in a ref: the toast's Undo (and the 404 branch) must
// clear the inline comment mark AFTER the originating CommentListItem has
// unmounted (resolving pulls the comment out of the Open list, so its item is
// already gone by the time the 10s toast is clicked). In read-only view
// pageEditorAtom is null and the mark converges via the server's
// COMMENT_MARK_UPDATE job instead.
const editor = useAtomValue(pageEditorAtom);
const editorRef = useRef(editor);
editorRef.current = editor;
// Self-reference the mutation so the toast's Undo can re-invoke it (reopen)
// long after the triggering component unmounted. Declared BEFORE useMutation
// and assigned AFTER; the onClick reads mutationRef.current at CALL time, not
// definition time, so there is no initialization cycle.
const mutationRef = useRef<{
mutate: (vars: IResolveComment) => void;
} | null>(null);
const mutation = useMutation({
mutationFn: (data: IResolveComment) => resolveComment(data),
onMutate: async (variables) => {
await queryClient.cancelQueries({ queryKey: RQ_KEY(variables.pageId) });
@@ -401,7 +426,39 @@ export function useResolveCommentMutation() {
return { previousCache };
},
onError: (_err, variables, context) => {
onError: (err: any, variables, context) => {
// Terminal 404: the comment was really deleted (missing comment or deleted
// page — access denial is 403, resolve is idempotent so no 400). Do NOT
// roll back (that would resurrect a phantom row in Resolved); instead drop
// it from the cache and clear its now-orphaned inline mark. Mirrors
// handleDeleteComment and the dismiss-mutation 404 branch.
if (err?.response?.status === 404) {
const cache = queryClient.getQueryData(RQ_KEY(variables.pageId)) as
| InfiniteData<IPagination<IComment>>
| undefined;
if (cache) {
queryClient.setQueryData(
RQ_KEY(variables.pageId),
removeCommentFromCache(cache, variables.commentId),
);
}
const ed = editorRef.current;
if (ed && !ed.isDestroyed) {
try {
ed.commands.unsetComment(variables.commentId);
} catch {
/* editor gone / mark already removed */
}
}
notifications.show({
message: t("Comment no longer exists"),
color: "red",
});
return;
}
// Generic failure: roll back the optimistic update and show a DIRECTIONAL
// error (resolve vs. reopen), not always "resolve".
if (context?.previousCache) {
queryClient.setQueryData(
RQ_KEY(variables.pageId),
@@ -409,7 +466,9 @@ export function useResolveCommentMutation() {
);
}
notifications.show({
message: t("Failed to resolve comment"),
message: variables.resolved
? t("Failed to resolve comment")
: t("Failed to re-open comment"),
color: "red",
});
},
@@ -430,11 +489,72 @@ export function useResolveCommentMutation() {
);
}
// Reopen keeps the plain toast without an Undo.
if (!variables.resolved) {
// Clear the inline mark ONLY after the server confirms the reopen, so a
// failed reopen never leaves an active highlight the panel still treats
// as resolved. Mirrors the 404 branch's editor-liveness guard/try-catch.
// The button-triggered reopen already set the mark, so this is an
// idempotent no-op there.
const ed = editorRef.current;
if (ed && !ed.isDestroyed) {
try {
ed.commands.setCommentResolved(variables.commentId, false);
} catch {
/* editor gone — server COMMENT_MARK_UPDATE converges it */
}
}
notifications.show({ message: t("Comment re-opened successfully") });
return;
}
// Resolve: attach an inline Undo (reopen) to the success toast. Built with
// React.createElement because this is a .ts module (no JSX).
const { commentId, pageId } = variables;
const notificationId = `resolve-undo-${commentId}`;
// Double-click guard: notifications.hide is NOT synchronous, so the button
// stays clickable for a frame or two — without this a fast double-click
// would fire reopen twice.
let done = false;
notifications.show({
message: variables.resolved
? t("Comment resolved successfully")
: t("Comment re-opened successfully"),
id: notificationId,
autoClose: RESOLVE_UNDO_AUTOCLOSE_MS,
message: React.createElement(
Group,
{ justify: "space-between", wrap: "nowrap", gap: "md" },
React.createElement(
Text,
{ size: "sm" },
t("Comment resolved successfully"),
),
React.createElement(
Button,
{
variant: "subtle",
size: "compact-sm",
onClick: () => {
if (done) return;
done = true;
// Reopen via the SAME mutation (read at click time — the
// originating item is already unmounted).
mutationRef.current?.mutate({
commentId,
pageId,
resolved: false,
});
// The inline mark is cleared in the reopen mutation's onSuccess
// (bound to server confirmation), NOT here — clearing it eagerly
// would desync the doc from the panel if reopen then fails.
notifications.hide(notificationId);
},
},
t("Undo"),
),
),
});
},
});
mutationRef.current = mutation;
return mutation;
}
@@ -0,0 +1,45 @@
import { describe, it, expect } from "vitest";
import {
formatHeadline,
formatDayTotal,
formatGapMinutes,
} from "./format-work-time";
const MIN = 60 * 1000;
// Fake translator: renders the key with {{tokens}} substituted, so the tests
// assert the rounding + branch selection without depending on the i18n catalogue.
const t = (key: string, opts?: Record<string, unknown>) =>
key.replace(/\{\{(\w+)\}\}/g, (_, k) => String(opts?.[k] ?? ""));
describe("formatHeadline", () => {
it("prefixes ≈ and rounds to a 5-minute step", () => {
expect(formatHeadline(4 * 60 * MIN + 27 * MIN, t)).toBe("≈ 4h 25m");
expect(formatHeadline(90 * MIN, t)).toBe("≈ 1h 30m");
});
it("shows hours only / minutes only cleanly", () => {
expect(formatHeadline(120 * MIN, t)).toBe("≈ 2h");
expect(formatHeadline(35 * MIN, t)).toBe("≈ 35m");
});
it("floors a tiny non-zero estimate to 5m, never 0", () => {
expect(formatHeadline(2 * MIN, t)).toBe("≈ 5m");
});
it("empty string for zero (widget hidden)", () => {
expect(formatHeadline(0, t)).toBe("");
});
});
describe("formatDayTotal", () => {
it('renders "h m" and shows — for empty days', () => {
expect(formatDayTotal(3 * 60 * MIN + 17 * MIN, t)).toBe("3h 17m");
expect(formatDayTotal(0, t)).toBe("—");
});
});
describe("formatGapMinutes", () => {
it("converts the tGap ms threshold to whole minutes", () => {
expect(formatGapMinutes(15 * MIN)).toBe(15);
});
});
@@ -0,0 +1,45 @@
// #395 — display formatting for the work-time estimate. Pure functions that take
// a translator so ru-RU / en-US wording lives in the i18n catalogue and the
// rounding logic stays unit-testable.
type Translate = (key: string, opts?: Record<string, unknown>) => string;
const MIN = 60 * 1000;
function hm(totalMinutes: number): { hours: number; minutes: number } {
return {
hours: Math.floor(totalMinutes / 60),
minutes: totalMinutes % 60,
};
}
/**
* Headline number (§6.1): an ESTIMATE, so rounded to a coarse 5-minute step and
* prefixed with "≈". A non-zero-but-tiny estimate floors to 5m rather than
* rounding down to "0" (which would read as "no work"). Zero → empty string
* (the caller hides the widget).
*/
export function formatHeadline(workMs: number, t: Translate): string {
if (workMs <= 0) return "";
let minutes = Math.round(workMs / MIN / 5) * 5;
if (minutes === 0) minutes = 5;
const { hours, minutes: m } = hm(minutes);
if (hours > 0 && m > 0) return t("≈ {{hours}}h {{minutes}}m", { hours, minutes: m });
if (hours > 0) return t("≈ {{hours}}h", { hours });
return t("≈ {{minutes}}m", { minutes: m });
}
/** Per-day sum (§6.2), rounded to the minute. Zero → "—". */
export function formatDayTotal(activeMs: number, t: Translate): string {
if (activeMs <= 0) return "—";
const minutes = Math.max(1, Math.round(activeMs / MIN));
const { hours, minutes: m } = hm(minutes);
if (hours > 0 && m > 0) return t("{{hours}}h {{minutes}}m", { hours, minutes: m });
if (hours > 0) return t("{{hours}}h", { hours });
return t("{{minutes}}m", { minutes: m });
}
/** The inactivity threshold, for the "estimate · gap = N min" caption. */
export function formatGapMinutes(tGapMs: number): number {
return Math.round(tGapMs / MIN);
}
@@ -0,0 +1,25 @@
import { useQuery, UseQueryResult } from "@tanstack/react-query";
import { IPageWorkTime } from "./work-time.types";
import { getPageWorkTime, viewerTimezone } from "./work-time-service";
const WORK_TIME_STALE_TIME = 5 * 60 * 1000;
/**
* #395 — the "time worked on this article" estimate + per-day punch-card
* buckets. The buckets are computed server-side in the viewer's timezone (so a
* midnight-crossing session lands on the right calendar day for the reader).
* `enabled` is opt-in so the (cheap but non-trivial) projection query only fires
* when the number is actually shown.
*/
export function usePageWorkTime(
pageId: string,
enabled = true,
): UseQueryResult<IPageWorkTime, Error> {
const tz = viewerTimezone();
return useQuery({
queryKey: ["page-work-time", pageId, tz],
queryFn: () => getPageWorkTime(pageId, tz),
enabled: enabled && !!pageId,
staleTime: WORK_TIME_STALE_TIME,
});
}
@@ -0,0 +1,171 @@
import { Group, Stack, Text } from "@mantine/core";
import { useTranslation } from "react-i18next";
import { useMemo } from "react";
import { IPageWorkTime, IPerDay, IDayWindow } from "./work-time.types";
import {
formatDayTotal,
formatGapMinutes,
formatHeadline,
} from "./format-work-time";
import classes from "./work-time.module.css";
const DAY_MS = 24 * 60 * 60 * 1000;
// Collapse a run of this many (or more) consecutive edit-free days into a single
// "× N days" separator (§6.2 long-range) — the row is still always one day.
const EMPTY_RUN_COLLAPSE = 8;
type Row =
| { type: "day"; day: IPerDay }
| { type: "gap"; count: number };
function collapseEmptyRuns(perDay: IPerDay[]): Row[] {
const rows: Row[] = [];
let emptyRun: IPerDay[] = [];
const flush = () => {
if (emptyRun.length >= EMPTY_RUN_COLLAPSE) {
rows.push({ type: "gap", count: emptyRun.length });
} else {
for (const d of emptyRun) rows.push({ type: "day", day: d });
}
emptyRun = [];
};
for (const d of perDay) {
if (d.activeMs === 0 && d.agentMs === 0) {
emptyRun.push(d);
} else {
flush();
rows.push({ type: "day", day: d });
}
}
flush();
return rows;
}
function dayHeading(day: number): string {
return new Date(day).toLocaleDateString(undefined, {
weekday: "short",
day: "numeric",
month: "short",
});
}
function DayTrack({
day,
pSingle,
}: {
day: IPerDay;
pSingle: number;
}) {
const { t } = useTranslation();
const ticks = [6, 12, 18];
return (
<div className={classes.row}>
<span className={classes.dayLabel}>{dayHeading(day.day)}</span>
<div className={classes.track}>
{ticks.map((h) => (
<div
key={h}
className={classes.hourTick}
style={{ left: `${(h / 24) * 100}%` }}
/>
))}
{day.windows.map((w: IDayWindow, i) => {
const leftPct = ((w.start - day.day) / DAY_MS) * 100;
const widthPct = ((w.end - w.start) / DAY_MS) * 100;
const isSingle = w.end - w.start <= pSingle;
const cls = [
classes.window,
w.class === "work" ? classes.windowWork : classes.windowAgent,
isSingle ? classes.windowSingle : "",
].join(" ");
return (
<div
key={i}
className={cls}
style={{
left: `${Math.max(0, Math.min(100, leftPct))}%`,
width: `${Math.max(0, Math.min(100, widthPct))}%`,
}}
/>
);
})}
</div>
<span className={classes.daySum}>
{formatDayTotal(day.activeMs, t)}
</span>
</div>
);
}
interface Props {
data: IPageWorkTime;
}
export default function WorkTimePunchCard({ data }: Props) {
const { t } = useTranslation();
const rows = useMemo(() => collapseEmptyRuns(data.perDay), [data.perDay]);
const gapMin = formatGapMinutes(data.config.tGap);
if (data.workMs <= 0 && data.agentOnlyMs <= 0) {
return (
<Text size="sm" c="dimmed" py="md">
{t("No editing activity recorded yet.")}
</Text>
);
}
return (
<Stack gap="xs">
<Group gap="lg">
<Text size="sm" fw={500}>
{formatHeadline(data.workMs, t)}
</Text>
{data.agentOnlyMs > 0 && (
<Text size="xs" c="dimmed">
{t("agent: {{value}}", { value: formatHeadline(data.agentOnlyMs, t) })}
</Text>
)}
</Group>
<Group gap="md">
<Text size="xs" c="dimmed">
<span
className={`${classes.legendSwatch} ${classes.windowWork}`}
style={{ marginRight: 4 }}
/>
{t("Work")}
</Text>
<Text size="xs" c="dimmed">
<span
className={`${classes.legendSwatch} ${classes.windowAgent}`}
style={{ marginRight: 4 }}
/>
{t("Agent")}
</Text>
</Group>
<div>
{rows.map((row, i) =>
row.type === "day" ? (
<DayTrack
key={row.day.dayISO}
day={row.day}
pSingle={data.config.pSingle}
/>
) : (
<div key={`gap-${i}`} className={classes.gapRow}>
{t("× {{count}} days without edits", { count: row.count })}
</div>
),
)}
</div>
<Text size="xs" c="dimmed" mt="xs">
{t("Estimate · timezone {{tz}} · inactivity gap {{gap}} min", {
tz: data.tz,
gap: gapMin,
})}
</Text>
</Stack>
);
}
@@ -0,0 +1,23 @@
import api from "@/lib/api-client";
import { IPageWorkTime } from "./work-time.types";
/** The viewer's IANA timezone (browser locale) — the punch-card lays days out
* in "my evenings", per §6.3/§10. Falls back to UTC if the runtime hides it. */
export function viewerTimezone(): string {
try {
return Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC";
} catch {
return "UTC";
}
}
export async function getPageWorkTime(
pageId: string,
tz: string,
): Promise<IPageWorkTime> {
const req = await api.post<IPageWorkTime>("/pages/history/time", {
pageId,
tz,
});
return req.data;
}
@@ -0,0 +1,69 @@
import { Modal, Text, Tooltip, UnstyledButton } from "@mantine/core";
import { useDisclosure } from "@mantine/hooks";
import { IconClockHour4 } from "@tabler/icons-react";
import { useTranslation } from "react-i18next";
import { usePageWorkTime } from "./use-page-work-time";
import { formatGapMinutes, formatHeadline } from "./format-work-time";
import WorkTimePunchCard from "./work-time-punch-card";
interface Props {
pageId: string;
}
/**
* #395 — the clickable "time worked on this article" headline (§6.1). Renders
* the `work` estimate with a "≈" sign and the inactivity threshold in a tooltip
* (it is an estimate, not a stopwatch). Clicking opens the daily punch-card
* (§6.2). Renders nothing until there is a non-zero human OR agent estimate, so a
* brand-new / never-edited page shows no widget. For an agent-only-edited page
* (workMs===0, agentOnlyMs>0) the headline shows the agent estimate (labelled
* `agent:`, matching the punch-card) so the punch-card stays reachable (#395:
* "how much a HUMAN and separately the AGENT").
*/
export default function WorkTimeStat({ pageId }: Props) {
const { t } = useTranslation();
const [opened, { open, close }] = useDisclosure(false);
const { data } = usePageWorkTime(pageId);
if (!data || (data.workMs <= 0 && data.agentOnlyMs <= 0)) return null;
const agentOnly = data.workMs <= 0;
const label = agentOnly
? t("agent: {{value}}", { value: formatHeadline(data.agentOnlyMs, t) })
: formatHeadline(data.workMs, t);
const gapMin = formatGapMinutes(data.config.tGap);
return (
<>
<Tooltip
label={t("Estimated time worked (inactivity gap {{gap}} min)", {
gap: gapMin,
})}
position="bottom"
>
<UnstyledButton
onClick={open}
aria-label={t("Show time worked on this page")}
>
<Text
size="xs"
c="dimmed"
style={{ display: "inline-flex", alignItems: "center", gap: 4 }}
>
<IconClockHour4 size={14} />
{label}
</Text>
</UnstyledButton>
</Tooltip>
<Modal
opened={opened}
onClose={close}
title={t("Time worked on this article")}
size="lg"
>
<WorkTimePunchCard data={data} />
</Modal>
</>
);
}
@@ -0,0 +1,82 @@
/* #395 — 24h × days punch-card. Custom CSS segments on a fixed 24-hour track
(position = offset-in-day / 24h, width = duration / 24h), no chart library. */
.row {
display: grid;
grid-template-columns: 96px 1fr 64px;
align-items: center;
gap: 12px;
padding: 3px 0;
}
.dayLabel {
font-size: var(--mantine-font-size-xs);
color: var(--mantine-color-dimmed);
white-space: nowrap;
}
.track {
position: relative;
height: 16px;
border-radius: 4px;
background-color: light-dark(
var(--mantine-color-gray-1),
var(--mantine-color-dark-6)
);
overflow: hidden;
}
/* Faint hour grid so the eye can read "morning vs evening". */
.hourTick {
position: absolute;
top: 0;
bottom: 0;
width: 1px;
background-color: light-dark(
var(--mantine-color-gray-3),
var(--mantine-color-dark-4)
);
}
.window {
position: absolute;
top: 2px;
bottom: 2px;
border-radius: 3px;
min-width: 3px;
}
.windowWork {
background-color: var(--mantine-color-blue-5);
}
.windowAgent {
background-color: var(--mantine-color-grape-5);
}
/* A lone single-sample (P_single) window: minimal + dimmed, so it neither
vanishes nor fakes dense work (§6.2). */
.windowSingle {
opacity: 0.5;
}
.daySum {
font-size: var(--mantine-font-size-xs);
text-align: right;
white-space: nowrap;
}
.gapRow {
padding: 6px 0 6px 108px;
font-size: var(--mantine-font-size-xs);
color: var(--mantine-color-dimmed);
font-style: italic;
}
.legendSwatch {
display: inline-block;
width: 12px;
height: 12px;
border-radius: 3px;
vertical-align: middle;
}
@@ -0,0 +1,37 @@
// #395 — client-side mirror of the server work-time payload
// (apps/server/src/core/page/work-time). Shapes returned by POST /pages/history/time.
export type WorkSessionClass = "work" | "agent_only";
export interface IDayWindow {
start: number;
end: number;
class: WorkSessionClass;
}
export interface IPerDay {
day: number;
dayISO: string;
activeMs: number;
agentMs: number;
windows: IDayWindow[];
}
export interface IWorkTimeConfig {
tGap: number;
agentTGap: number;
pIn: number;
pOut: number;
pSingle: number;
excludeGit: boolean;
burstCapMs?: number;
dedupRoundMs: number;
}
export interface IPageWorkTime {
workMs: number;
agentOnlyMs: number;
perDay: IPerDay[];
config: IWorkTimeConfig;
tz: string;
}
@@ -51,6 +51,7 @@ import {
import { formattedDate } from "@/lib/time.ts";
import { PageEditModeToggle } from "@/features/user/components/page-state-pref.tsx";
import MovePageModal from "@/features/page/components/move-page-modal.tsx";
import WorkTimeStat from "@/features/page-history/work-time/work-time-stat.tsx";
import { useTimeAgo } from "@/hooks/use-time-ago.tsx";
import {
useFavoriteIds,
@@ -265,6 +266,8 @@ function PageActionMenu({ readOnly, onSaveVersion }: PageActionMenuProps) {
return (
<>
{page?.id && <WorkTimeStat pageId={page.id} />}
<Menu
shadow="xl"
position="bottom-end"
@@ -1,6 +1,6 @@
import { Spotlight } from "@mantine/spotlight";
import { IconSearch } from "@tabler/icons-react";
import { Group, VisuallyHidden } from "@mantine/core";
import { Group, Text, VisuallyHidden } from "@mantine/core";
import { useState, useMemo } from "react";
import { useDebouncedValue } from "@mantine/hooks";
import { useTranslation } from "react-i18next";
@@ -84,6 +84,11 @@ export function SearchSpotlight({ spaceId }: SearchSpotlightProps) {
onFiltersChange={handleFiltersChange}
spaceId={spaceId}
/>
{/* #529: operator hint — matches ANY word by default; "…" for an exact
phrase, +term to require, -term to exclude. */}
<Text size="xs" c="dimmed" mt={4}>
{t('Tip: "exact phrase", +required, -excluded')}
</Text>
</div>
<VisuallyHidden role="status" aria-live="polite">
@@ -5,6 +5,9 @@ import { IPage } from "@/features/page/types/page.types.ts";
export interface IPageSearch {
id: string;
// #529 A7 superset: `pageId` aliases `id`; `rank`/`highlight` are null for
// substring-only hits (the UI already falls back to the title/snippet).
pageId?: string;
title: string;
icon: string;
parentPageId: string;
@@ -12,9 +15,36 @@ export interface IPageSearch {
creatorId: string;
createdAt: Date;
updatedAt: Date;
rank: string;
highlight: string;
rank: string | number | null;
highlight: string | null;
space: Partial<ISpace>;
// New #529 fields (present from the native Postgres search driver).
snippet?: string;
score?: number;
path?: string[];
matchedFields?: string[];
matchedTerms?: string[];
}
// #529 A5 pagination envelope returned by POST /search (native driver). The web
// list helpers read `items`; these travel alongside for pagination + diagnostics.
export interface IPageSearchResponse {
items: IPageSearch[];
total: number;
hasMore: boolean;
truncatedAtCap: boolean;
offset: number;
query?: {
raw: string;
parsed: {
positive: string[];
required: string[];
excluded: string[];
reason?: string;
};
mode: "or" | "and";
match: string;
};
}
export interface SearchSuggestionParams {
@@ -37,6 +67,10 @@ export interface IPageSearchParams {
query: string;
spaceId?: string;
shareId?: string;
// #529 A9: match mode (auto default) + pagination.
match?: "auto" | "word" | "prefix" | "substring";
limit?: number;
offset?: number;
}
export interface IAttachmentSearch {
@@ -0,0 +1,195 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { render, act, cleanup } from "@testing-library/react";
import { MemoryRouter, useNavigate } from "react-router-dom";
// Mocks for the dirty shell's side-effecting collaborators.
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
vi.mock("@/i18n.ts", () => ({ default: { t: (k: string) => k } }));
vi.mock("@/lib/reload-guard", () => ({
hasAutoReloaded: vi.fn(() => false),
markAutoReloaded: vi.fn(() => true),
recordReloadBreadcrumb: vi.fn(),
takeReloadBreadcrumb: vi.fn(() => null),
}));
import { notifications } from "@mantine/notifications";
import { hasAutoReloaded, markAutoReloaded } from "@/lib/reload-guard";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
__resetGuardedReloadForTests,
} from "./guarded-reload";
const show = notifications.show as unknown as ReturnType<typeof vi.fn>;
const mockHasAutoReloaded = hasAutoReloaded as unknown as ReturnType<
typeof vi.fn
>;
const mockMarkAutoReloaded = markAutoReloaded as unknown as ReturnType<
typeof vi.fn
>;
let reload: ReturnType<typeof vi.fn>;
let visibility: DocumentVisibilityState;
// Test harness mounted inside a router: it installs the navigation hook and
// exposes `navigate` so a test can drive an in-app router navigation.
let doNavigate: (to: string) => void;
function Harness() {
useVersionReloadOnNavigation();
const navigate = useNavigate();
doNavigate = navigate;
return null;
}
function mountHarness() {
render(
<MemoryRouter initialEntries={["/start"]}>
<Harness />
</MemoryRouter>,
);
}
function navigateTo(path: string) {
act(() => {
doNavigate(path);
});
}
beforeEach(() => {
__resetGuardedReloadForTests();
vi.clearAllMocks();
mockHasAutoReloaded.mockReturnValue(false);
mockMarkAutoReloaded.mockReturnValue(true);
vi.stubGlobal("APP_VERSION", "test-A");
reload = vi.fn();
Object.defineProperty(window, "location", {
configurable: true,
value: { reload },
});
visibility = "visible";
Object.defineProperty(document, "visibilityState", {
configurable: true,
get: () => visibility,
});
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
});
describe("triggerGuardedReload (variant C)", () => {
it("noop when versions match: no banner, no reload", () => {
triggerGuardedReload("test-A");
expect(show).not.toHaveBeenCalled();
expect(reload).not.toHaveBeenCalled();
});
it("noop when the server version is empty (fail-safe)", () => {
triggerGuardedReload("");
triggerGuardedReload(undefined);
expect(show).not.toHaveBeenCalled();
expect(reload).not.toHaveBeenCalled();
});
it("real mismatch shows the banner but does NOT reload immediately", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(show).toHaveBeenCalledTimes(1);
expect(show.mock.calls[0][0]).toMatchObject({
id: "app-version-reload",
autoClose: false,
withCloseButton: true,
});
expect(reload).not.toHaveBeenCalled();
});
it("reloads EXACTLY ONCE on the first in-app navigation after a mismatch", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
// A second navigation must NOT reload again (one-shot was consumed).
navigateTo("/again");
expect(reload).toHaveBeenCalledTimes(1);
});
it("does NOT reload on a 2nd mismatch after the first was armed/consumed (one-shot)", () => {
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
// Another app-version mismatch arrives (reconnect): must not re-arm.
triggerGuardedReload("test-C");
navigateTo("/again");
expect(reload).toHaveBeenCalledTimes(1);
});
it("does NOT reload merely from the tab going to the background", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
visibility = "hidden";
act(() => {
document.dispatchEvent(new Event("visibilitychange"));
});
expect(reload).not.toHaveBeenCalled();
});
it("a hidden-at-receipt tab is also NOT reloaded immediately (variant C uniform); reloads on next navigation", () => {
visibility = "hidden";
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
expect(show).toHaveBeenCalledTimes(1);
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
});
it("the banner's Update button reloads immediately", () => {
triggerGuardedReload("test-B");
const message = show.mock.calls[0][0].message as {
props: { onClick: () => void };
};
message.props.onClick();
expect(reload).toHaveBeenCalledTimes(1);
});
it("banner-only (auto-reload already spent): banner, never auto-reload on navigation", () => {
mockHasAutoReloaded.mockReturnValue(true);
mountHarness();
triggerGuardedReload("test-B");
expect(show).toHaveBeenCalledTimes(1);
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
});
it("does NOT reload when the flag write fails; falls back to the banner", () => {
mockMarkAutoReloaded.mockReturnValue(false);
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
// performAutoReload falls back to showing the banner (initial + fallback).
expect(show).toHaveBeenCalled();
});
it("is idempotent within a tab-load: repeated emits do not stack banners", () => {
triggerGuardedReload("test-B");
triggerGuardedReload("test-B");
triggerGuardedReload("test-C");
expect(show).toHaveBeenCalledTimes(1);
});
});
@@ -0,0 +1,188 @@
import { useEffect, useRef } from "react";
import { useLocation } from "react-router-dom";
import { Button } from "@mantine/core";
import { notifications } from "@mantine/notifications";
import i18n from "@/i18n.ts";
import {
hasAutoReloaded,
markAutoReloaded,
recordReloadBreadcrumb,
takeReloadBreadcrumb,
} from "@/lib/reload-guard";
import { decideVersionAction } from "@/features/user/version-coherence";
// Dirty shell around the pure `decideVersionAction`: it reads globals
// (APP_VERSION), touches sessionStorage via the shared reload-guard, drives the
// Mantine notification, and arms the router-navigation reload hook. Kept
// separate from the pure module so the decision stays unit-testable without a
// DOM.
// One fixed id so repeated app-version signals (e.g. every reconnect) update a
// single banner instead of stacking a new one each time.
const BANNER_ID = "app-version-reload";
// Module-level idempotency for the current tab-load: once a mismatch has been
// handled we don't re-arm the navigation reload or re-show the banner on
// subsequent app-version emits.
let handled = false;
// Variant C: on a real mismatch we do NOT reload the tab when it merely goes to
// the background (that would silently drop a half-written comment/form). Instead
// we arm a one-shot reload for the NEXT in-app router navigation — a point where
// the user is already leaving the current page, so an in-app navigation would
// discard that unsaved component-state anyway and the reload adds no extra loss.
let pendingNavReload = false;
// Remembered from the last detected mismatch for the pre-reload breadcrumb and
// the (already-visible) banner.
let lastServerVersion = "";
let lastClientVersion = "";
// Read the build version baked into THIS bundle. The `typeof` guard avoids a
// ReferenceError where the `APP_VERSION` global is absent (e.g. under vitest,
// where Vite's `define` did not run) — an unknown client version makes the
// pure decision no-op (fail-safe).
function readClientVersion(): string {
return (typeof APP_VERSION !== "undefined" ? APP_VERSION : "").trim();
}
// Perform the actual reload — but only after the shared one-shot flag is
// persisted. If the write fails (storage unavailable) we must NOT reload
// (mirrors the reactive chunk-load boundary's `catch → return`), and fall back
// to the manual banner so the user can still recover.
function performAutoReload(): void {
if (!markAutoReloaded()) {
showReloadBanner();
return;
}
// Trace right before the reload (which clears the console): a persistent
// breadcrumb + a log line so the auto-reload is observable in a field report.
recordReloadBreadcrumb({
path: "proactive",
serverVersion: lastServerVersion,
clientVersion: lastClientVersion,
});
console.warn(
`[version-coherence] auto-reloading: client=${lastClientVersion} -> server=${lastServerVersion}`,
);
window.location.reload();
}
function showReloadBanner(): void {
notifications.show({
id: BANNER_ID,
title: i18n.t("A new version is available"),
message: (
<Button size="xs" mt="xs" onClick={() => performAutoReload()}>
{i18n.t("Update")}
</Button>
),
autoClose: false,
withCloseButton: true,
});
}
/**
* Handle a server `app-version` announcement: compare it to this bundle's
* version and, on a real mismatch, show the banner and arm a guarded reload for
* the next in-app navigation (variant C).
*
* - real mismatch (window budget available) → banner + arm navigation reload.
* The banner's "Update" button reloads immediately (same shared window guard).
* The tab is NOT reloaded on visibility change.
* - auto-reload already used this window / storage error → banner only (no arm),
* so there is at most one automatic reload per RELOAD_WINDOW_MS window (loop
* safety).
* - in sync / unknown version → noop (fail-safe).
*/
export function triggerGuardedReload(
rawServerVersion: string | undefined | null,
): void {
const serverVersion = (rawServerVersion ?? "").trim();
const clientVersion = readClientVersion();
// A storage read error surfaces as autoReloadUsed=true → fail toward NOT
// reloading (banner only).
const autoReloadUsed = hasAutoReloaded();
const action = decideVersionAction({
serverVersion,
clientVersion,
autoReloadUsed,
});
if (action === "noop") return;
// Idempotent per tab-load: don't re-arm or re-stack the banner across repeated
// emits (reconnects) once we've already acted.
if (handled) return;
handled = true;
lastServerVersion = serverVersion;
lastClientVersion = clientVersion;
if (action === "banner") {
// Entered banner-only (permanent skew, node oscillation, or the window's
// auto-reload budget already spent). Log for diagnosability; show the banner.
console.warn(
`[version-coherence] server=${serverVersion} client=${clientVersion}: ` +
"auto-reload budget already spent this window — showing manual banner",
);
showReloadBanner();
return;
}
// action === "reload" (variant C): show the banner and defer the auto-reload
// to the next in-app navigation instead of reloading now / on visibility.
showReloadBanner();
pendingNavReload = true;
}
/**
* Consume the armed one-shot navigation reload, if any. Called by
* `useVersionReloadOnNavigation` on each in-app router navigation.
*/
export function consumeNavigationReload(): void {
if (!pendingNavReload) return;
pendingNavReload = false;
performAutoReload();
}
/**
* Hook (mounted inside the Router) that fires the armed one-shot reload on the
* NEXT in-app router navigation after a version mismatch. Skips the initial
* render so it only reacts to real navigations, not the first location.
*/
export function useVersionReloadOnNavigation(): void {
const location = useLocation();
const firstRender = useRef(true);
useEffect(() => {
if (firstRender.current) {
firstRender.current = false;
return;
}
consumeNavigationReload();
}, [location.key]);
}
/**
* Surface (log once) the breadcrumb left by an auto-reload in the previous page
* load — the reload cleared the console, so this makes a "tab reloaded itself"
* report diagnosable. Call once on app startup.
*/
export function surfacePreviousReloadBreadcrumb(): void {
const crumb = takeReloadBreadcrumb();
if (!crumb) return;
console.info(
`[version-coherence] previous auto-reload: path=${crumb.path} ` +
`client=${crumb.clientVersion ?? ""} -> server=${crumb.serverVersion ?? ""} ` +
`at=${new Date(crumb.at).toISOString()}`,
);
}
// Test-only: reset module-level latches between cases.
export function __resetGuardedReloadForTests(): void {
handled = false;
pendingNavReload = false;
lastServerVersion = "";
lastClientVersion = "";
}
@@ -0,0 +1,171 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { render, act, cleanup } from "@testing-library/react";
import { MemoryRouter, useNavigate } from "react-router-dom";
// Integration test for the SHARED, window-based auto-reload budget (invariant a):
// the reactive chunk-load boundary and the proactive version-coherence path both
// route through the REAL @/lib/reload-guard, so at most one automatic reload
// happens per RELOAD_WINDOW_MS across BOTH paths combined. Only the two paths'
// side-effecting collaborators are mocked — the reload guard is intentionally
// REAL so this exercises the actual shared sessionStorage budget.
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
vi.mock("@/i18n.ts", () => ({ default: { t: (k: string) => k } }));
import { handleError } from "@/components/chunk-load-error-boundary";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
__resetGuardedReloadForTests,
} from "./guarded-reload";
import { RELOAD_WINDOW_MS } from "@/lib/reload-guard";
const CHUNK_ERROR = { name: "ChunkLoadError", message: "boom" };
const T0 = 1_000_000_000_000;
let reload: ReturnType<typeof vi.fn>;
let nowMock: ReturnType<typeof vi.spyOn>;
// Harness mounted inside a router: installs the navigation hook and exposes
// `navigate` so a test can drive an in-app router navigation (the point where the
// proactive path fires its armed reload).
let doNavigate: (to: string) => void;
function Harness() {
useVersionReloadOnNavigation();
doNavigate = useNavigate();
return null;
}
function mountHarness() {
render(
<MemoryRouter initialEntries={["/start"]}>
<Harness />
</MemoryRouter>,
);
}
function navigateTo(path: string) {
act(() => {
doNavigate(path);
});
}
function setNow(t: number) {
nowMock.mockReturnValue(t);
}
beforeEach(() => {
sessionStorage.clear();
__resetGuardedReloadForTests();
vi.clearAllMocks();
nowMock = vi.spyOn(Date, "now").mockReturnValue(T0);
vi.stubGlobal("APP_VERSION", "test-A");
reload = vi.fn();
Object.defineProperty(window, "location", {
configurable: true,
value: { reload },
});
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
vi.restoreAllMocks();
sessionStorage.clear();
});
describe("shared window-based reload budget (invariant a)", () => {
it("a chunk-load reload spends the budget: a version-coherence mismatch within the window shows the banner but does NOT reload", () => {
// Path 1 (reactive): a stale-chunk 404 auto-reloads once and stamps the window.
handleError(CHUNK_ERROR);
expect(reload).toHaveBeenCalledTimes(1);
reload.mockClear();
// Path 2 (proactive), 1 min later — still inside the window. The shared budget
// is spent, so the version mismatch degrades to the banner and never reloads.
setNow(T0 + 60_000);
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
});
it("a version-coherence reload spends the SAME budget: a chunk-load error within the window does NOT reload", () => {
// Path 2 (proactive) first: real mismatch → arm → fire on navigation.
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
reload.mockClear();
// Path 1 (reactive), 2 min later — inside the window. Budget already spent by
// the proactive path, so the stale-chunk error must NOT trigger a second reload.
setNow(T0 + 2 * 60_000);
handleError(CHUNK_ERROR);
expect(reload).not.toHaveBeenCalled();
});
it("recovers after the window: a reload strictly older than the window is allowed again (second deploy)", () => {
// First auto-reload (reactive) stamps the window.
handleError(CHUNK_ERROR);
expect(reload).toHaveBeenCalledTimes(1);
reload.mockClear();
// A second deploy arrives after the window has fully elapsed → the proactive
// path is allowed to reload again (window, not a permanent one-shot).
__resetGuardedReloadForTests();
setNow(T0 + RELOAD_WINDOW_MS + 1);
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
});
it("reactive path fails closed when storage READS but cannot WRITE (quota / Safari private): no unguarded reload", () => {
// getItem→null makes hasAutoReloaded() report the budget as available, so
// handleError passes the first guard and reaches `if (!markAutoReloaded())
// return;`. setItem throws → the stamp cannot stick, so markAutoReloaded()
// returns false and that guard MUST bail — otherwise the reactive path would
// reload on every stale-chunk error with no persisted budget (an unguarded
// loop). This is the asymmetric gap: the proactive path's equivalent is
// covered by guarded-reload.test.tsx "does NOT reload when the flag write
// fails".
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("quota exceeded");
},
removeItem: () => {},
clear: () => {},
});
try {
handleError(CHUNK_ERROR);
expect(reload).not.toHaveBeenCalled();
} finally {
vi.unstubAllGlobals();
}
});
it("sessionStorage unavailable: neither path performs an unguarded reload", () => {
// The real guard fails toward NOT reloading when storage throws.
vi.stubGlobal("sessionStorage", {
getItem: () => {
throw new Error("storage disabled");
},
setItem: () => {
throw new Error("storage disabled");
},
removeItem: () => {},
clear: () => {},
});
try {
handleError(CHUNK_ERROR);
expect(reload).not.toHaveBeenCalled();
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
} finally {
vi.unstubAllGlobals();
}
});
});
@@ -13,6 +13,12 @@ import { useCollabToken } from "@/features/auth/queries/auth-query.tsx";
import { Error404 } from "@/components/ui/error-404.tsx";
import { queryClient } from "@/main.tsx";
import { makeConnectHandler } from "@/features/user/connect-resync.ts";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
surfacePreviousReloadBreadcrumb,
} from "@/features/user/guarded-reload.tsx";
import type { AppVersionSocketPayload } from "@/features/user/version-coherence.ts";
export function UserProvider({ children }: React.PropsWithChildren) {
const [, setCurrentUser] = useAtom(currentUserAtom);
@@ -22,6 +28,16 @@ export function UserProvider({ children }: React.PropsWithChildren) {
// fetch collab token on load
const { data: collab } = useCollabToken();
// version-coherence: fire the armed one-shot reload on the next in-app
// navigation (variant C — a safe point, not on tab backgrounding).
useVersionReloadOnNavigation();
// Surface any breadcrumb left by an auto-reload in the previous page load
// (the reload cleared the console) so a field report stays diagnosable.
useEffect(() => {
surfacePreviousReloadBreadcrumb();
}, []);
useEffect(() => {
if (isLoading || isError) {
return;
@@ -47,6 +63,16 @@ export function UserProvider({ children }: React.PropsWithChildren) {
handleConnect();
});
// Register the version-coherence listener SYNCHRONOUSLY, before the socket
// connects: the server emits `app-version` immediately in handleConnection,
// so a listener attached after connect would miss it on a fast localhost
// connect. On a version mismatch the client shows a banner and defers the
// auto-reload to the next in-app navigation (variant C — avoids reloading a
// backgrounded tab that may hold unsaved input) before it hits a stale chunk.
newSocket.on("app-version", (payload?: AppVersionSocketPayload) => {
triggerGuardedReload(payload?.version);
});
return () => {
console.log("ws disconnected");
newSocket.disconnect();
@@ -0,0 +1,64 @@
import { describe, it, expect } from "vitest";
import { decideVersionAction } from "./version-coherence";
describe("decideVersionAction", () => {
it("noop when the server version is empty (fail-safe)", () => {
expect(
decideVersionAction({
serverVersion: "",
clientVersion: "v1",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("noop when the client version is empty (fail-safe)", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("noop when versions are equal (in sync)", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "v1",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("reload on a real mismatch the first time this session", () => {
expect(
decideVersionAction({
serverVersion: "test-B",
clientVersion: "test-A",
autoReloadUsed: false,
}),
).toBe("reload");
});
it("banner on a mismatch once the session auto-reload is spent", () => {
expect(
decideVersionAction({
serverVersion: "test-B",
clientVersion: "test-A",
autoReloadUsed: true,
}),
).toBe("banner");
});
it("equal versions stay noop even if auto-reload was already used", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "v1",
autoReloadUsed: true,
}),
).toBe("noop");
});
});
@@ -0,0 +1,32 @@
// Payload of the per-connect `app-version` socket.io event announced by the
// server (ws.gateway.ts) after a successful auth. A dedicated event — NOT a
// member of the room-scoped `WebSocketEvent` union (which is discriminated by
// `operation`), so it never touches use-query-subscription.
export type AppVersionSocketPayload = { version: string };
/**
* Pure decision for the version-coherence guard.
*
* All inputs are injected (no globals, no side effects) so it is unit-testable
* without a DOM or the build-time `APP_VERSION` global (undefined under vitest).
*
* - `autoReloadUsed` = an automatic reload has already happened within the
* current ~5-min window, so we must not auto-reload again (loop safety,
* shared window budget with the reactive chunk-load boundary).
*
* Returns:
* - "noop" — do nothing (unknown version on either side, or already in sync).
* - "banner" — show the manual "update available" banner only (no auto-reload).
* - "reload" — real first-time mismatch: eligible for a guarded auto-reload.
*/
export function decideVersionAction(args: {
serverVersion: string;
clientVersion: string;
autoReloadUsed: boolean;
}): "reload" | "banner" | "noop" {
const { serverVersion, clientVersion, autoReloadUsed } = args;
if (!serverVersion || !clientVersion) return "noop"; // fail-safe: unknown version → never act
if (serverVersion === clientVersion) return "noop"; // in sync
if (autoReloadUsed) return "banner"; // one auto-reload per RELOAD_WINDOW_MS window already spent
return "reload"; // real mismatch, window budget available
}
+145
View File
@@ -0,0 +1,145 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import {
hasAutoReloaded,
markAutoReloaded,
shouldAutoReload,
recordReloadBreadcrumb,
takeReloadBreadcrumb,
RELOAD_WINDOW_MS,
} from "./reload-guard";
// The shared budget is a single sessionStorage timestamp keyed here; both the
// reactive chunk-load boundary and the proactive version-coherence path read and
// stamp it, so at most one auto-reload happens per RELOAD_WINDOW_MS across BOTH.
const RELOAD_AT_KEY = "chunk-reload-at";
const NOW = 1_000_000_000_000;
describe("reload-guard", () => {
beforeEach(() => {
sessionStorage.clear();
vi.restoreAllMocks();
});
afterEach(() => {
sessionStorage.clear();
vi.restoreAllMocks();
});
it("hasAutoReloaded is false before any reload, true within the window after mark", () => {
expect(hasAutoReloaded(NOW)).toBe(false);
expect(markAutoReloaded(NOW)).toBe(true);
// Same key both paths share; stores the reload timestamp, not a flag.
expect(sessionStorage.getItem(RELOAD_AT_KEY)).toBe(String(NOW));
// Inside the window → budget spent → true (fall through to manual UI).
expect(hasAutoReloaded(NOW)).toBe(true);
expect(hasAutoReloaded(NOW + RELOAD_WINDOW_MS)).toBe(true);
});
it("hasAutoReloaded is false again once the window has elapsed (a later deploy recovers)", () => {
markAutoReloaded(NOW);
// Strictly older than the window → a new deploy's mismatch may reload again.
expect(hasAutoReloaded(NOW + RELOAD_WINDOW_MS + 1)).toBe(false);
});
it("hasAutoReloaded returns true when reading storage throws (fail toward not reloading)", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => {
throw new Error("storage disabled");
},
setItem: () => {
throw new Error("storage disabled");
},
});
try {
expect(hasAutoReloaded()).toBe(true);
} finally {
vi.unstubAllGlobals();
}
});
it("hasAutoReloaded treats an unparseable stored timestamp as never-reloaded", () => {
sessionStorage.setItem(RELOAD_AT_KEY, "not-a-number");
expect(hasAutoReloaded(NOW)).toBe(false);
});
it("markAutoReloaded returns false when writing storage throws", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("storage disabled");
},
});
try {
expect(markAutoReloaded()).toBe(false);
} finally {
vi.unstubAllGlobals();
}
});
it("records and then takes a breadcrumb once (cleared on read)", () => {
recordReloadBreadcrumb({
path: "proactive",
serverVersion: "test-B",
clientVersion: "test-A",
});
const crumb = takeReloadBreadcrumb();
expect(crumb).toMatchObject({
path: "proactive",
serverVersion: "test-B",
clientVersion: "test-A",
});
expect(typeof crumb?.at).toBe("number");
// Cleared on read → a second take returns null.
expect(takeReloadBreadcrumb()).toBeNull();
});
it("takeReloadBreadcrumb returns null when nothing was recorded", () => {
expect(takeReloadBreadcrumb()).toBeNull();
});
it("recordReloadBreadcrumb swallows a storage-write error (diagnostics only)", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("storage disabled");
},
removeItem: () => {},
});
try {
expect(() =>
recordReloadBreadcrumb({ path: "chunk-boundary" }),
).not.toThrow();
} finally {
vi.unstubAllGlobals();
}
});
});
// The pure window gate replaces the old one-shot flag: it must permit recovery
// across several deploys in one tab (each > window apart) while still stopping an
// infinite reload loop when a lazy chunk is permanently broken (a second failure
// < window). Moved here from the chunk-load boundary now that it is the shared
// guard both paths route through.
describe("shouldAutoReload", () => {
const WINDOW = RELOAD_WINDOW_MS;
it("allows a reload when we have never auto-reloaded", () => {
expect(shouldAutoReload(NOW, null, WINDOW)).toBe(true);
});
it("allows a reload when the last one was 6 minutes ago (outside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 6 * 60 * 1000, WINDOW)).toBe(true);
});
it("blocks a reload when the last one was 1 minute ago (inside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 1 * 60 * 1000, WINDOW)).toBe(false);
});
it("blocks a reload exactly at the window boundary (not strictly older)", () => {
expect(shouldAutoReload(NOW, NOW - WINDOW, WINDOW)).toBe(false);
});
it("allows a reload when the stored timestamp is unparseable (NaN)", () => {
expect(shouldAutoReload(NOW, NaN, WINDOW)).toBe(true);
});
});
+121
View File
@@ -0,0 +1,121 @@
// Shared, window-based auto-reload budget.
//
// Both auto-reload paths — the reactive chunk-load-error-boundary (recovers
// AFTER a stale lazy chunk 404s) and the proactive version-coherence feature
// (reloads BEFORE the tab hits a stale chunk) — go through these functions so
// they share ONE window-scoped reload budget: at most a single automatic
// reload per RELOAD_WINDOW_MS across BOTH paths. A window (rather than a
// permanent one-shot flag) lets a SECOND deploy in the same tab's lifetime
// recover too, while a permanent skew, node oscillation, or a genuinely-missing
// chunk still degrades to a manual banner/UI after the first reload instead of
// looping. When sessionStorage is unavailable every mismatch degrades to the
// manual UI — no unguarded reload.
// sessionStorage key holding the epoch-ms timestamp of the last automatic reload
// (shared by both paths).
const RELOAD_AT_KEY = "chunk-reload-at";
// Allow at most one automatic reload per this window. A stale-deploy 404 is cured
// by a single reload, so anything inside the window is treated as a reload loop
// (permanently-broken chunk / permanent skew) and falls through to the manual UI.
export const RELOAD_WINDOW_MS = 5 * 60 * 1000;
/**
* Pure window decision, unit-tested in isolation: auto-reload only if we have
* never auto-reloaded (lastReloadAt null/NaN) or the last one was strictly older
* than the window. Anything inside the window is suppressed to break an infinite
* reload loop.
*/
export function shouldAutoReload(
now: number,
lastReloadAt: number | null,
windowMs: number,
): boolean {
if (lastReloadAt === null || Number.isNaN(lastReloadAt)) return true;
return now - lastReloadAt > windowMs;
}
/**
* Has an automatic reload already happened within the current window (so the
* shared budget is spent right now)? Both paths check this before reloading; a
* `true` return means fall through to the manual banner/UI instead of reloading.
*
* A storage read error (private mode / disabled) is reported as `true` so the
* caller fails toward NOT reloading — an unguarded loop is worse than a stale
* tab the user can reload manually. Note a window (not a permanent flag): once
* the window elapses a later deploy's mismatch is allowed to reload again.
*/
export function hasAutoReloaded(now: number = Date.now()): boolean {
try {
const raw = sessionStorage.getItem(RELOAD_AT_KEY);
const lastReloadAt = raw === null ? null : Number.parseInt(raw, 10);
return !shouldAutoReload(now, lastReloadAt, RELOAD_WINDOW_MS);
} catch {
return true;
}
}
/**
* Stamp the shared window as consumed now — record that an automatic reload is
* being performed within the current RELOAD_WINDOW_MS window.
*
* Returns whether the write succeeded. A `false` return (storage unavailable)
* means the caller MUST NOT reload — otherwise the stamp would never stick and
* the reload could loop.
*/
export function markAutoReloaded(now: number = Date.now()): boolean {
try {
sessionStorage.setItem(RELOAD_AT_KEY, String(now));
return true;
} catch {
return false;
}
}
// Diagnostic breadcrumb for an automatic reload. Written right before
// window.location.reload() (which clears the console) and read back on the next
// page load, so a "the tab reloaded itself / it's looping" field report is
// diagnosable: which path fired (proactive version-coherence vs the reactive
// chunk-load boundary) and which version pair triggered it. sessionStorage
// survives a same-tab reload, unlike the console.
const RELOAD_BREADCRUMB_KEY = "reload-breadcrumb";
export type ReloadBreadcrumb = {
path: "proactive" | "chunk-boundary";
serverVersion?: string;
clientVersion?: string;
at: number;
};
/**
* Persist a best-effort breadcrumb just before an automatic reload. Failures
* (storage unavailable) are swallowed — this is diagnostics only and must never
* block or alter the reload decision.
*/
export function recordReloadBreadcrumb(
entry: Omit<ReloadBreadcrumb, "at">,
): void {
try {
sessionStorage.setItem(
RELOAD_BREADCRUMB_KEY,
JSON.stringify({ ...entry, at: Date.now() }),
);
} catch {
// best-effort diagnostics only
}
}
/**
* Read and clear the breadcrumb left by an auto-reload in the previous page
* load. Cleared on read so it surfaces exactly once per reload.
*/
export function takeReloadBreadcrumb(): ReloadBreadcrumb | null {
try {
const raw = sessionStorage.getItem(RELOAD_BREADCRUMB_KEY);
if (!raw) return null;
sessionStorage.removeItem(RELOAD_BREADCRUMB_KEY);
return JSON.parse(raw) as ReloadBreadcrumb;
} catch {
return null;
}
}
+29 -2
View File
@@ -1,7 +1,8 @@
import { defineConfig, loadEnv } from "vite";
import { defineConfig, loadEnv, type Plugin } from "vite";
import react from "@vitejs/plugin-react";
import { compression } from "vite-plugin-compression2";
import * as path from "path";
import * as fs from "node:fs";
import { execSync } from "node:child_process";
const envPath = path.resolve(process.cwd(), "..", "..");
@@ -24,7 +25,32 @@ function resolveAppVersion(cwd: string): string {
}
}
// Emit <outDir>/version.json = { "version": appVersion } so the server can read
// the exact same build id the bundle was compiled with. The value is the SAME
// `appVersion` fed into `define.APP_VERSION`, so version.json and the baked-in
// global are identical by construction — the single source of truth (no
// runtime-env second copy that could drift and cause a false version mismatch).
function versionJsonPlugin(version: string): Plugin {
let outDir = "dist";
return {
name: "emit-version-json",
apply: "build",
configResolved(config) {
outDir = config.build.outDir;
},
writeBundle() {
const root = path.resolve(process.cwd(), outDir);
fs.mkdirSync(root, { recursive: true });
fs.writeFileSync(
path.join(root, "version.json"),
JSON.stringify({ version }),
);
},
};
}
export default defineConfig(({ mode }) => {
const appVersion = resolveAppVersion(envPath);
const {
APP_URL,
FILE_UPLOAD_SIZE_LIMIT,
@@ -52,10 +78,11 @@ export default defineConfig(({ mode }) => {
POSTHOG_HOST,
POSTHOG_KEY,
},
APP_VERSION: JSON.stringify(resolveAppVersion(envPath)),
APP_VERSION: JSON.stringify(appVersion),
},
plugins: [
react(),
versionJsonPlugin(appVersion),
// Emit .br and .gz next to every built asset so the server can serve the
// precompressed copy (see @fastify/static preCompressed in static.module.ts).
compression({
+1 -1
View File
@@ -23,7 +23,7 @@
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build && pnpm --filter @docmost/mcp build",
"test": "jest",
"test:int": "jest --config test/jest-integration.json",
"test:watch": "jest --watch",
@@ -141,7 +141,57 @@ export function htmlToJson(html: string) {
}
}
export function jsonToText(tiptapJson: JSONContent) {
/**
* Deterministic text-serializer overrides for the `format:"text"` page read
* (#502). Non-text nodes render to a STABLE placeholder instead of their
* (structure-dependent) inner text, so a machine diff of two text reads is
* driven only by the page's actual prose — output stability across package
* versions IS the contract (pinned by a snapshot test). Returning a string from
* a `textSerializer` also stops `generateText` descending into the node, so a
* table renders as ONE token rather than its flattened cell text.
*
* Only nodes with no meaningful flat-text form are overridden; every other node
* (paragraph/heading/list/code/blockquote/callout/…) keeps its natural text so
* a config written as markdown reads back byte-identical.
*/
const TEXT_READ_SERIALIZERS: Record<string, (props: { node: any }) => string> =
{
// Image atom: no inner text -> a fixed placeholder.
image: () => '[image]',
// Table: `[table RxC]` where R = row count, C = the first row's cell count
// (a table's columns are uniform per the schema). Computed from the PM node,
// so it is independent of cell contents.
table: ({ node }) => {
const rows = node?.childCount ?? 0;
const cols = rows > 0 ? (node.child(0)?.childCount ?? 0) : 0;
return `[table ${rows}x${cols}]`;
},
};
/**
* Serialize a ProseMirror/TipTap document to plain text.
*
* Default (no options): the long-standing search-index behavior — bare
* concatenated node text with `generateText`'s default `\n\n` block separator.
* This feeds the page `textContent` tsvector and MUST NOT change.
*
* `deterministic:true` (#502 `format:"text"` page read): a flat, machine-diffable
* rendering — one line per block (`\n` block separator; `hardBreak` already
* serializes to `\n`), inline marks/anchors/autoformat dropped, and non-text
* nodes replaced by the stable placeholders above (`[image]`, `[table RxC]`).
*/
export function jsonToText(
// `any` (like jsonToHtml/jsonToMarkdown) so a loosely-typed DB `page.content`
// (JsonValue) can be passed straight through, as the controller does.
tiptapJson: any,
options?: { deterministic?: boolean },
) {
if (options?.deterministic) {
return generateText(tiptapJson, tiptapExtensions, {
blockSeparator: '\n',
textSerializers: TEXT_READ_SERIALIZERS,
});
}
return generateText(tiptapJson, tiptapExtensions);
}
@@ -0,0 +1,116 @@
import { jsonToText } from './collaboration.util';
// #502 READ contract: `jsonToText(json, { deterministic: true })` — the flat,
// machine-diffable text rendering behind getPage `format:"text"`. Block-per-line
// (`\n` separator), inline marks/anchors dropped, hardBreak -> `\n`, and non-text
// nodes replaced by STABLE placeholders (`[image]`, `[table RxC]`). Output
// stability across package versions IS the contract, so it is pinned by a
// snapshot below. The DEFAULT (no options) path is the search-index serializer
// and MUST be unchanged — asserted separately.
const doc = (...content: any[]) => ({ type: 'doc', content });
const para = (...content: any[]) => ({ type: 'paragraph', content });
const text = (t: string, marks?: any[]) =>
marks ? { type: 'text', text: t, marks } : { type: 'text', text: t };
describe('jsonToText — default (search index) behavior is unchanged', () => {
it('uses the `\\n\\n` block separator and drops non-text nodes to empty', () => {
const d = doc(para(text('alpha')), para(text('beta')));
expect(jsonToText(d)).toBe('alpha\n\nbeta');
});
it('an image contributes no text in the default (tsvector) mode', () => {
const d = doc(para(text('cap')), { type: 'image', attrs: { src: 's' } });
// No `[image]` placeholder leaks into the search index.
expect(jsonToText(d)).not.toContain('[image]');
});
});
describe('jsonToText — deterministic:true (getPage format:"text")', () => {
it('renders one line per block with `\\n` separators, marks dropped', () => {
const d = doc(
{ type: 'heading', attrs: { level: 2 }, content: [text('Title')] },
para(
text('hello ', [{ type: 'bold' }]),
text('world', [{ type: 'italic' }]),
),
);
expect(jsonToText(d, { deterministic: true })).toBe('Title\nhello world');
});
it('a hardBreak renders as a newline', () => {
const d = doc(para(text('a'), { type: 'hardBreak' }, text('b')));
expect(jsonToText(d, { deterministic: true })).toBe('a\nb');
});
it('an image node -> the stable `[image]` placeholder', () => {
const d = doc(para(text('before')), { type: 'image', attrs: { src: 's' } });
expect(jsonToText(d, { deterministic: true })).toBe('before\n[image]');
});
it('a table -> `[table RxC]` (rows x columns) and its cell text is NOT flattened in', () => {
const cell = (t: string) => ({
type: 'tableCell',
content: [para(text(t))],
});
const row = (...cells: any[]) => ({ type: 'tableRow', content: cells });
const table = {
type: 'table',
content: [
row(cell('CELLONE'), cell('CELLTWO'), cell('CELLTHREE')),
row(cell('CELLFOUR'), cell('CELLFIVE'), cell('CELLSIX')),
],
};
const d = doc(para(text('grid:')), table);
const out = jsonToText(d, { deterministic: true });
expect(out).toBe('grid:\n[table 2x3]');
expect(out).not.toContain('CELL'); // cell text is not flattened into the read
});
it('SNAPSHOT: a mixed document renders to a stable, deterministic string', () => {
const cell = (t: string) => ({
type: 'tableCell',
content: [para(text(t))],
});
const d = doc(
{ type: 'heading', attrs: { level: 1 }, content: [text('Config')] },
para(text('key = ', [{ type: 'bold' }]), text('value')),
{
type: 'bulletList',
content: [
{ type: 'listItem', content: [para(text('one'))] },
{ type: 'listItem', content: [para(text('two'))] },
],
},
{ type: 'image', attrs: { src: 's' } },
{
type: 'table',
content: [{ type: 'tableRow', content: [cell('x'), cell('y')] }],
},
);
expect(jsonToText(d, { deterministic: true })).toMatchInlineSnapshot(`
"Config
key = value
one
two
[image]
[table 1x2]"
`);
});
it('SCENARIO: a config written as a code block reads back byte-identical (diff empty)', () => {
const config =
'export TICKET_LIFETIME=$2592000\nservers:\n - www.internal.host';
const d = doc({
type: 'codeBlock',
attrs: { language: 'yaml' },
content: [text(config)],
});
// The code block is one block; its text (dollars, bare domain, newlines) is
// preserved verbatim, so a read-as-text of a stored config diffs empty.
expect(jsonToText(d, { deterministic: true })).toBe(config);
});
});
@@ -0,0 +1,52 @@
import { join } from 'path';
import * as fs from 'node:fs';
import * as os from 'node:os';
import { readClientBuildVersion } from './client-version';
describe('readClientBuildVersion', () => {
let dir: string;
beforeEach(() => {
dir = fs.mkdtempSync(join(os.tmpdir(), 'client-version-'));
});
afterEach(() => {
fs.rmSync(dir, { recursive: true, force: true });
});
const writeVersionJson = (content: string) =>
fs.writeFileSync(join(dir, 'version.json'), content);
it('returns the version from a valid version.json', () => {
writeVersionJson(JSON.stringify({ version: 'test-A' }));
expect(readClientBuildVersion(dir)).toBe('test-A');
});
it('trims surrounding whitespace in the version', () => {
writeVersionJson(JSON.stringify({ version: ' v1.2.3 ' }));
expect(readClientBuildVersion(dir)).toBe('v1.2.3');
});
it('returns "" when version.json is missing', () => {
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" on malformed JSON', () => {
writeVersionJson('{ not json');
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the version field is absent', () => {
writeVersionJson(JSON.stringify({ notVersion: 'x' }));
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the version field is not a string', () => {
writeVersionJson(JSON.stringify({ version: 123 }));
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the path does not exist at all', () => {
expect(readClientBuildVersion(join(dir, 'nope'))).toBe('');
});
});
@@ -0,0 +1,36 @@
import { join } from 'path';
import * as fs from 'node:fs';
/**
* Resolve the absolute path to the built client bundle directory
* (`apps/client/dist`) shipped into the runtime image.
*
* The `../` depth is anchored on THIS module's compiled location
* (`dist/common/helpers`). `integrations/static` sits at the same depth under
* the compiled root, so both callers (StaticModule and readClientBuildVersion)
* MUST share this single helper rather than duplicating the depth — a copy in a
* module at a different depth would silently resolve to the wrong directory.
*/
export function resolveClientDistPath(): string {
return join(__dirname, '..', '..', '..', '..', 'client/dist');
}
/**
* Read the build version the client bundle was compiled with, from
* `<clientDistPath>/version.json` (written by the Vite build — the single
* source of truth shared by the baked-in `APP_VERSION` global and this file).
*
* Fail-safe: any error (missing file, unreadable, bad JSON, non-string
* version) yields `''`. The caller treats an empty version as "unknown" and
* the whole version-coherence feature stays silently inert — existing deploys
* without the file keep working unchanged.
*/
export function readClientBuildVersion(clientDistPath: string): string {
try {
const raw = fs.readFileSync(join(clientDistPath, 'version.json'), 'utf8');
const version = (JSON.parse(raw) as { version?: unknown }).version;
return typeof version === 'string' ? version.trim() : '';
} catch {
return '';
}
}
+1
View File
@@ -3,3 +3,4 @@ export * from './nanoid.utils';
export * from './file.helper';
export * from './constants';
export * from './security-headers';
export * from './client-version';
@@ -35,6 +35,7 @@ import {
import { PaginationOptions } from '@docmost/db/pagination/pagination-options';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { UserThrottlerGuard } from '../../integrations/throttle/user-throttler.guard';
import { AI_CHAT_THROTTLER } from '../../integrations/throttle/throttler-names';
@@ -43,6 +44,8 @@ import {
AiChatRunHooks,
AiChatService,
AiChatStreamBody,
rowHasInlineParts,
hydrateAssistantParts,
} from './ai-chat.service';
import { AiChatRunService } from './ai-chat-run.service';
import { AiTranscriptionService } from './ai-transcription.service';
@@ -129,8 +132,39 @@ export class AiChatController {
// production. Only touched on the resumable-stream (flag-on) path.
private readonly streamRegistry?: AiChatStreamRegistryService,
private readonly environment?: EnvironmentService,
// #492: reconstruct a #492 mid-run record's parts from the steps table before
// returning rows to the client / export. OPTIONAL so positional controller
// specs compile unchanged; when absent, hydration is skipped (old-era rows
// already carry inline parts, so nothing to reconstruct).
private readonly aiChatRunStepRepo?: AiChatRunStepRepo,
) {}
/**
* Reconstruct parts for any assistant rows that don't carry them INLINE — a
* #492 mid-run record whose per-step parts live in `ai_chat_run_steps` (the
* append-persist backend). Every FINISHED row (old-era + #492) and every old-era
* streaming snapshot already has inline `metadata.parts`, so the common path
* fetches NOTHING and returns the rows untouched; only an actively-streaming
* new-style row triggers the batch step fetch. Consumers (seed/poll/export) read
* `metadata.parts` off the returned rows exactly as before — the era switch is
* invisible to them (reconstructRunParts contract).
*/
private async withReconstructedParts(
rows: AiChatMessage[],
workspaceId: string,
): Promise<AiChatMessage[]> {
if (!this.aiChatRunStepRepo) return rows;
const needy = rows.filter(
(r) => r.role === 'assistant' && !rowHasInlineParts(r),
);
if (needy.length === 0) return rows;
const stepsByMessage = await this.aiChatRunStepRepo.findByMessageIds(
needy.map((r) => r.id),
workspaceId,
);
return hydrateAssistantParts(rows, stepsByMessage);
}
/** List the requesting user's chats in this workspace (paginated). */
@HttpCode(HttpStatus.OK)
@Post('chats')
@@ -184,11 +218,17 @@ export class AiChatController {
@AuthWorkspace() workspace: Workspace,
) {
await this.assertOwnedChat(dto.chatId, user, workspace);
return this.aiChatMessageRepo.findByChat(
const page = await this.aiChatMessageRepo.findByChat(
dto.chatId,
workspace.id,
pagination,
);
// #492: reconstruct parts for any active new-style row so the client seed sees
// `metadata.parts` unchanged (a no-op for the finished rows that fill a page).
return {
...page,
items: await this.withReconstructedParts(page.items, workspace.id),
};
}
/**
@@ -225,7 +265,10 @@ export class AiChatController {
workspace.id,
);
return {
rows,
// #492: the delta of an actively-streaming new-style row carries its parts
// reconstructed from the steps table, so the degraded poll shows persisted
// progress exactly as the pre-#492 full-row snapshot did.
rows: await this.withReconstructedParts(rows, workspace.id),
cursor,
run: run ? { id: run.id, status: run.status } : null,
};
@@ -247,8 +290,10 @@ export class AiChatController {
@AuthWorkspace() workspace: Workspace,
): Promise<{ markdown: string }> {
const chat = await this.assertOwnedChat(dto.chatId, user, workspace);
const rows = await this.aiChatMessageRepo.findAllByChat(
dto.chatId,
const rows = await this.withReconstructedParts(
await this.aiChatMessageRepo.findAllByChat(dto.chatId, workspace.id),
// #492: an interrupted-but-still-active turn exports its persisted steps
// (reconstructed from the steps table) just like the pre-#492 full row did.
workspace.id,
);
const markdown = buildChatMarkdown({
@@ -288,7 +333,13 @@ export class AiChatController {
workspace.id,
)
: undefined;
return { run, message: message ?? null };
// #492: reconnect to an IN-FLIGHT run reconstructs the projection row's parts
// from the steps table (the row itself carries only the step marker mid-run);
// a finished run's row already has inline parts, so this is a no-op.
const [hydrated] = message
? await this.withReconstructedParts([message], workspace.id)
: [undefined];
return { run, message: hydrated ?? null };
}
/**
+203 -12
View File
@@ -22,6 +22,7 @@ import { AiSettingsService } from '../../integrations/ai/ai-settings.service';
import { describeProviderError } from '../../integrations/ai/ai-error.util';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo';
import { AiAgentRoleRepo } from '@docmost/db/repos/ai-agent-roles/ai-agent-roles.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
@@ -518,6 +519,12 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// constructions compile unchanged; Nest always injects the real singleton, so
// reconcile sees the SAME in-memory active/zombie maps the runner mutates.
private readonly aiChatRunService?: AiChatRunService,
// #492 append-persist: per-step INSERT into the lightweight steps table (the
// O(Σ steps) replacement for the O(n²) full-row `metadata.parts` rewrite).
// OPTIONAL so existing positional constructions (int-specs) compile unchanged;
// Nest injects the real singleton. When ABSENT the per-step path falls back to
// the pre-#492 full-row flush (no regression, only no WAL win).
private readonly aiChatRunStepRepo?: AiChatRunStepRepo,
) {}
// #487: periodic reconcile timer (single-process phase 1). Started in
@@ -1114,8 +1121,34 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
chatId,
workspace.id,
);
// #492: HYDRATE needy assistant rows from the steps table BEFORE the replay
// map. A #492 mid-run assistant row carries only a step marker
// (metadata.parts:[]); its real per-step parts live in `ai_chat_run_steps`.
// The graceful terminal callbacks (onFinish/onError/onAbort -> flushAssistant)
// assemble the full inline parts, so a normally-ended turn already has them.
// But a HARD crash mid-run (SIGKILL/OOM) fires NO terminal callback, so the
// row stays parts:[]; without this, rowToUiMessage falls back to an empty
// text part and the partial tool-calls/results/text — durable in the steps
// table — would DROP OUT of the model's replay context (regressing #183
// step-granular durability for the model consumer). Mirrors the controller's
// withReconstructedParts EXACTLY (same needy predicate + hydration helper).
// Guarded on the optional repo: absent (positional test builds) degrades to
// the current behavior rather than crashing.
let replayHistory = oldHistory;
if (this.aiChatRunStepRepo) {
const needy = oldHistory.filter(
(r) => r.role === 'assistant' && !rowHasInlineParts(r),
);
if (needy.length > 0) {
const stepsByMessage = await this.aiChatRunStepRepo.findByMessageIds(
needy.map((r) => r.id),
workspace.id,
);
replayHistory = hydrateAssistantParts(oldHistory, stepsByMessage);
}
}
const uiMessages: Array<Omit<UIMessage, 'id'> & { id: string }> = [
...oldHistory.map(rowToUiMessage),
...replayHistory.map(rowToUiMessage),
{
id: 'pending-user',
role: 'user',
@@ -1154,7 +1187,9 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// hint — confirm it against the persisted history (the preceding assistant
// turn must really be aborted/streaming) so a spoofed flag cannot inject the
// interrupt note onto an ordinary turn. The partial output the model needs is
// already in `messages` (the aborted assistant row replays via findRecent).
// already in `messages`: a #492 mid-run row's per-step parts live only in the
// `ai_chat_run_steps` table and were hydrated into the replay history above,
// so the aborted assistant turn replays WITH its partial parts intact.
// Append the new user turn (shape-only) so index -2 is the prior assistant.
const interrupted = isInterruptResume(
[...oldHistory, { role: 'user', status: null, metadata: null }],
@@ -1559,17 +1594,57 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// connection when finalize runs, so the SQL `WHERE status='streaming'`
// (not this flag) is what prevents it clobbering the terminal row.
if (finalized) return null;
// Build the flush ONCE so the returned count is EXACTLY the persisted
// `stepsPersisted` (both derive from capturedSteps.length at this instant).
const flushed = flushAssistant(capturedSteps, '', 'streaming', {
pageChanged,
partsCache,
});
const stepsPersisted = flushed.metadata.stepsPersisted as number;
// The count derives from capturedSteps.length at THIS instant, so the
// returned value is EXACTLY the persisted `stepsPersisted` the ring rotates
// on (whether we take the append-persist path or the legacy fallback).
const stepsPersisted = capturedSteps.length;
try {
await this.aiChatMessageRepo.update(assistantId, workspace.id, flushed, {
onlyIfStreaming: true,
});
if (this.aiChatRunStepRepo) {
// #492 APPEND-PERSIST: write only THIS finished step's parts to the
// steps table (O(step) WAL), then bump the row's CHEAP step marker —
// NO growing `metadata.parts` blob (that O(n²) full-row rewrite is
// exactly what this removes). The full `metadata.parts` is assembled
// once at finalize; a mid-run resume seed is reconstructed from the
// step rows (reconstructRunParts). The INSERT is idempotent
// (ON CONFLICT DO NOTHING), so a re-fired step never doubles the parts.
const index = stepsPersisted - 1;
if (index >= 0) {
const stepParts = assistantParts(
[capturedSteps[index]],
'',
partsCache,
);
await this.aiChatRunStepRepo.insertStep(
assistantId,
workspace.id,
index,
stepParts,
);
}
// Marker UPDATE: advance stepsPersisted + keep the toolTrace era marker
// (bumps updatedAt so the delta poll observes the step, and carries the
// frontier a resuming client attaches from). Scoped onlyIfStreaming so a
// late marker never clobbers the terminal finalize.
await this.aiChatMessageRepo.update(
assistantId,
workspace.id,
{ metadata: stepMarkerMetadata(stepsPersisted) },
{ onlyIfStreaming: true },
);
} else {
// Legacy fallback (no steps table wired — positional test builds): the
// pre-#492 full-row flush, so parts still land inline on the row.
const flushed = flushAssistant(capturedSteps, '', 'streaming', {
pageChanged,
partsCache,
});
await this.aiChatMessageRepo.update(
assistantId,
workspace.id,
flushed,
{ onlyIfStreaming: true },
);
}
return stepsPersisted;
} catch (err) {
this.logger.warn(
@@ -2749,6 +2824,122 @@ export function rowToUiMessage(row: AiChatMessage): Omit<UIMessage, 'id'> & {
return { id: row.id, role, parts: parts as UIMessage['parts'] };
}
/**
* Cheap step-marker metadata for the #492 per-step UPDATE. Advances
* `stepsPersisted` (the resume attach frontier) and keeps the `toolTraceVersion`
* era marker, WITHOUT the growing `parts` blob (those live in the steps table
* now; the full `metadata.parts` is assembled once at finalize by flushAssistant).
* `parts: []` is kept for shape stability — it reads as an empty inline-parts row,
* which is exactly the discriminator that routes reconstruction to the steps table.
*/
export function stepMarkerMetadata(
stepsPersisted: number,
): Record<string, unknown> {
return { parts: [], toolTraceVersion: 2, stepsPersisted };
}
/**
* Whether an assistant row already carries its full UI parts INLINE on the row
* (`metadata.parts`). TRUE for every FINISHED row — old-era rows AND #492 rows,
* whose full parts are assembled once at finalize — and for old-era streaming
* snapshots (the pre-#492 per-step full-row flush). FALSE for a #492 MID-RUN
* record, whose per-step parts live in the `ai_chat_run_steps` table. This is the
* era discriminator the reconstruct seam branches on — no schema flag needed.
*/
export function rowHasInlineParts(row: { metadata?: unknown }): boolean {
const meta = (row.metadata ?? {}) as { parts?: unknown };
return Array.isArray(meta.parts) && meta.parts.length > 0;
}
/**
* Concatenate persisted per-step parts (in `stepIndex` order) into the turn's UI
* parts (#492). Reproduces EXACTLY what flushAssistant → assistantParts would have
* written to `metadata.parts` for those finished steps, since each step row stored
* `assistantParts([step])` at persist time.
*/
export function assembleStepParts(
stepRows: ReadonlyArray<{ stepIndex: number; parts: unknown }>,
): UIMessage['parts'] {
const parts: Array<Record<string, unknown>> = [];
for (const step of [...stepRows].sort((a, b) => a.stepIndex - b.stepIndex)) {
if (Array.isArray(step.parts)) {
parts.push(...(step.parts as Array<Record<string, unknown>>));
}
}
return parts as UIMessage['parts'];
}
/**
* reconstructRunParts (#492) — the single backend-switch seam. Given an assistant
* ROW and its persisted step rows, return the turn's UI `parts` + the persisted
* step count, reading from the ROW when it already carries inline parts (old-era
* records AND every finished record) and from the STEPS TABLE otherwise (a #492
* mid-run record). The higher-level consumers (attach seed, delta poll, export)
* route their row→parts through this / {@link hydrateAssistantParts}, so old and
* new records reconstruct identically WITHOUT the consumers branching on the era.
*/
export function reconstructRunParts(
row: { metadata?: unknown; content?: string | null },
stepRows: ReadonlyArray<{ stepIndex: number; parts: unknown }>,
): { parts: UIMessage['parts']; stepsPersisted: number } {
if (rowHasInlineParts(row)) {
const meta = row.metadata as {
parts: UIMessage['parts'];
stepsPersisted?: number;
};
return {
parts: meta.parts,
stepsPersisted:
typeof meta.stepsPersisted === 'number'
? meta.stepsPersisted
: stepRows.length,
};
}
if (stepRows.length > 0) {
return {
parts: assembleStepParts(stepRows),
stepsPersisted: stepRows.length,
};
}
// No inline parts and no step rows: an old-era seed / empty streaming row. Fall
// back to a single text part from `content` (mirrors rowToUiMessage).
return {
parts: textPart(row.content ?? '') as UIMessage['parts'],
stepsPersisted: 0,
};
}
/**
* Fill each assistant row's `metadata.parts` from its step rows when the row does
* not already carry them inline (a #492 mid-run record), so a consumer that reads
* `metadata.parts` off the RAW row (the client seed/poll, the Markdown export)
* sees the reconstructed parts with NO change to itself. Rows that already have
* inline parts (old-era + finished) and non-assistant rows pass through untouched.
* Pure: returns new row objects, never mutates the inputs.
*/
export function hydrateAssistantParts<
T extends { id: string; role?: string; metadata?: unknown },
>(
rows: ReadonlyArray<T>,
stepsByMessage: Map<
string,
ReadonlyArray<{ stepIndex: number; parts: unknown }>
>,
): T[] {
return rows.map((row) => {
if (row.role !== 'assistant' || rowHasInlineParts(row)) return row;
const steps = stepsByMessage.get(row.id);
if (!steps || steps.length === 0) return row;
return {
...row,
metadata: {
...((row.metadata ?? {}) as Record<string, unknown>),
parts: assembleStepParts(steps),
},
};
});
}
/**
* The persisted-row patch shape produced by {@link flushAssistant}. It is the
* SAME shape the assistant repo insert/update consume (content + toolCalls +
@@ -10,6 +10,12 @@ import { Transform } from 'class-transformer';
export type ContentFormat = 'json' | 'markdown' | 'html';
// READ-only rendering formats for `getPage` (#502). A superset of the writable
// `ContentFormat` with `text` — a flat, deterministic, machine-diffable text
// rendering — added. Kept SEPARATE from `ContentFormat` so the write path
// (createPage/updatePage `parseProsemirrorContent`) can never be handed `text`.
export type PageReadFormat = ContentFormat | 'text';
export class CreatePageDto {
@IsOptional()
@IsString()
+15 -3
View File
@@ -5,10 +5,11 @@ import {
IsOptional,
IsString,
IsUUID,
MaxLength,
} from 'class-validator';
import { Transform } from 'class-transformer';
import { ContentFormat } from './create-page.dto';
import { PageReadFormat } from './create-page.dto';
import { IsPageIdOrSlugId } from './page-identity.validator';
export class PageIdDto {
@@ -43,8 +44,19 @@ export class PageInfoDto extends PageIdDto {
@IsOptional()
@Transform(({ value }) => value?.toLowerCase())
@IsIn(['json', 'markdown', 'html'])
format?: ContentFormat;
@IsIn(['json', 'markdown', 'html', 'text'])
format?: PageReadFormat;
}
export class PageWorkTimeDto extends PageIdDto {
// Viewer IANA timezone for the per-day punch-card buckets (§6.3). Optional —
// falls back to UTC server-side. Length-capped so a bogus value cannot bloat
// the request; the value is only ever handed to Intl.DateTimeFormat, which
// throws on an unknown zone (caught by the controller → 400).
@IsOptional()
@IsString()
@MaxLength(64)
tz?: string;
}
export class DeletePageDto extends PageIdDto {
@@ -0,0 +1,76 @@
import { NotFoundException } from '@nestjs/common';
import { PageController } from './page.controller';
import { jsonToText, jsonToMarkdown } from '../../collaboration/collaboration.util';
// #502 READ: getPage `format:"text"` routes through the deterministic jsonToText
// path (placeholders for non-text nodes, block-per-line), while `format:"json"`
// (or none) returns the raw content and `format:"markdown"` still converts. This
// pins the CONTROLLER wiring with lightweight mocks (no DB needed — the jsonToText
// output contract itself is pinned in json-to-text-deterministic.spec.ts).
const CONTENT = {
type: 'doc',
content: [
{ type: 'heading', attrs: { level: 1 }, content: [{ type: 'text', text: 'Config' }] },
{
type: 'paragraph',
content: [
{ type: 'text', text: 'key=', marks: [{ type: 'bold' }] },
{ type: 'text', text: 'value' },
],
},
{ type: 'image', attrs: { src: 's' } },
],
};
function makeController(page: any): PageController {
const pageRepo = { findById: jest.fn().mockResolvedValue(page) } as any;
const pageAccessService = {
validateCanViewWithPermissions: jest
.fn()
.mockResolvedValue({ canEdit: true, hasRestriction: false }),
} as any;
// Only pageRepo + pageAccessService are exercised by getPage; the rest are
// never touched on this path, so undefined placeholders are fine.
return new PageController(
undefined as any, // pageService
pageRepo,
undefined as any, // pageHistoryService
undefined as any, // spaceAbility
pageAccessService,
undefined as any, // backlinkService
undefined as any, // labelService
undefined as any, // auditService
);
}
const user = { id: 'u1' } as any;
describe('PageController.getPage — format:"text" (#502)', () => {
it('returns deterministic flat text with placeholders for non-text nodes', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1', format: 'text' } as any, user);
expect(res.content).toBe(jsonToText(CONTENT, { deterministic: true }));
expect(res.content).toBe('Config\nkey=value\n[image]');
expect(res.permissions).toEqual({ canEdit: true, hasRestriction: false });
});
it('markdown format still converts to markdown (unchanged)', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1', format: 'markdown' } as any, user);
expect(res.content).toBe(jsonToMarkdown(CONTENT));
});
it('json / no format returns the raw ProseMirror content object', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1' } as any, user);
expect(res.content).toEqual(CONTENT); // untouched object, not a string
});
it('missing page -> NotFoundException', async () => {
const controller = makeController(null);
await expect(
controller.getPage({ pageId: 'nope', format: 'text' } as any, user),
).rejects.toBeInstanceOf(NotFoundException);
});
});
@@ -1,3 +1,8 @@
import {
BadRequestException,
ForbiddenException,
NotFoundException,
} from '@nestjs/common';
import { PageController } from './page.controller';
// Direct instantiation with stub deps. The Test.createTestingModule form failed
@@ -22,4 +27,88 @@ describe('PageController', () => {
it('should be defined', () => {
expect(controller).toBeDefined();
});
// #395 — the work-time endpoint must be gated exactly like /history.
describe('getPageWorkTime', () => {
const user = { id: 'u1' } as any;
function build(overrides: {
page?: any;
validate?: jest.Mock;
compute?: jest.Mock;
}) {
const pageRepo = { findById: jest.fn().mockResolvedValue(overrides.page) };
const pageAccessService = {
validateCanView: overrides.validate ?? jest.fn().mockResolvedValue(undefined),
};
const pageHistoryService = {
computeWorkTime:
overrides.compute ?? jest.fn().mockResolvedValue({ workMs: 0 }),
};
const c = new PageController(
{} as any,
pageRepo as any,
pageHistoryService as any,
{} as any,
pageAccessService as any,
{} as any,
{} as any,
{} as any,
);
return { c, pageRepo, pageAccessService, pageHistoryService };
}
it('404s when the page does not exist', async () => {
const { c } = build({ page: null });
await expect(
c.getPageWorkTime({ pageId: 'p1' } as any, user),
).rejects.toBeInstanceOf(NotFoundException);
});
it('enforces validateCanView before computing, then delegates with tz', async () => {
const validate = jest.fn().mockResolvedValue(undefined);
const compute = jest.fn().mockResolvedValue({ workMs: 42 });
const { c } = build({ page: { id: 'pg' }, validate, compute });
const out = await c.getPageWorkTime(
{ pageId: 'pg', tz: 'Europe/Moscow' } as any,
user,
);
expect(validate).toHaveBeenCalledWith({ id: 'pg' }, user);
expect(compute).toHaveBeenCalledWith('pg', 'Europe/Moscow');
expect(out).toEqual({ workMs: 42 });
});
it('propagates a denied view gate and does NOT reach compute (security)', async () => {
// If validateCanView is moved AFTER computeWorkTime, the timeline of a page
// the caller may not see would be read/estimated before the gate — this
// locks the order: a rejecting gate must short-circuit before any compute.
const validate = jest.fn().mockRejectedValue(new ForbiddenException());
const compute = jest.fn().mockResolvedValue({ workMs: 1 });
const { c, pageHistoryService } = build({
page: { id: 'pg' },
validate,
compute,
});
await expect(
c.getPageWorkTime({ pageId: 'pg' } as any, user),
).rejects.toBeInstanceOf(ForbiddenException);
expect(pageHistoryService.computeWorkTime).not.toHaveBeenCalled();
});
it('maps an unknown-timezone RangeError to a 400', async () => {
const compute = jest.fn().mockRejectedValue(new RangeError('bad tz'));
const { c } = build({ page: { id: 'pg' }, compute });
await expect(
c.getPageWorkTime({ pageId: 'pg', tz: 'X/Y' } as any, user),
).rejects.toBeInstanceOf(BadRequestException);
});
it('does not swallow a non-RangeError from the service', async () => {
const compute = jest.fn().mockRejectedValue(new Error('db down'));
const { c } = build({ page: { id: 'pg' }, compute });
await expect(
c.getPageWorkTime({ pageId: 'pg' } as any, user),
).rejects.toThrow('db down');
});
});
});
+38 -4
View File
@@ -21,6 +21,7 @@ import {
PageHistoryIdDto,
PageIdDto,
PageInfoDto,
PageWorkTimeDto,
} from './dto/page.dto';
import { PageHistoryService } from './services/page-history.service';
import { AuthUser } from '../../common/decorators/auth-user.decorator';
@@ -49,6 +50,7 @@ import { AddLabelsDto, RemoveLabelDto } from '../label/dto/label.dto';
import {
jsonToHtml,
jsonToMarkdown,
jsonToText,
} from '../../collaboration/collaboration.util';
import { AuditEvent, AuditResource } from '../../common/events/audit-events';
import {
@@ -93,10 +95,16 @@ export class PageController {
const permissions = { canEdit, hasRestriction };
if (dto.format && dto.format !== 'json' && page.content) {
const contentOutput =
dto.format === 'markdown'
? jsonToMarkdown(page.content)
: jsonToHtml(page.content);
let contentOutput: string;
if (dto.format === 'markdown') {
contentOutput = jsonToMarkdown(page.content);
} else if (dto.format === 'text') {
// #502: flat, deterministic, machine-diffable text (block-per-line,
// inline marks/anchors dropped, non-text nodes -> stable placeholders).
contentOutput = jsonToText(page.content, { deterministic: true });
} else {
contentOutput = jsonToHtml(page.content);
}
return {
...page,
content: contentOutput,
@@ -524,6 +532,32 @@ export class PageController {
return this.pageHistoryService.findHistoryByPageId(page.id, pagination);
}
@HttpCode(HttpStatus.OK)
@Post('/history/time')
async getPageWorkTime(
@Body() dto: PageWorkTimeDto,
@AuthUser() user: User,
) {
const page = await this.pageRepo.findById(dto.pageId);
if (!page) {
throw new NotFoundException('Page not found');
}
// Same view gate as /history and /history/info.
await this.pageAccessService.validateCanView(page, user);
try {
return await this.pageHistoryService.computeWorkTime(page.id, dto.tz);
} catch (e) {
// Intl.DateTimeFormat throws RangeError on an unknown IANA zone; surface
// it as a 400 rather than a 500.
if (e instanceof RangeError) {
throw new BadRequestException('Invalid timezone');
}
throw e;
}
}
@HttpCode(HttpStatus.OK)
@Post('/history/info')
async getPageHistoryInfo(
@@ -3,6 +3,23 @@ import { PageHistoryRepo } from '@docmost/db/repos/page/page-history.repo';
import { PageHistory } from '@docmost/db/types/entity.types';
import { PaginationOptions } from '@docmost/db/pagination/pagination-options';
import { CursorPaginationResult } from '@docmost/db/pagination/cursor-pagination';
import {
computeWorkTime,
bucketByDay,
DEFAULT_WORK_TIME_CONFIG,
WorkTimeConfig,
PerDay,
} from '../work-time';
export interface PageWorkTime {
workMs: number;
agentOnlyMs: number;
perDay: PerDay[];
/** the config actually used, so the UI can show "≈" + the T_gap threshold. */
config: WorkTimeConfig;
/** the tz the per-day buckets were computed in (echoed back for the label). */
tz: string;
}
@Injectable()
export class PageHistoryService {
@@ -23,4 +40,33 @@ export class PageHistoryService {
paginationOptions,
);
}
/**
* #395 — estimate time worked on a page (§5) and bucket it into the viewer's
* calendar days for the punch-card (§6.3). Reads only the cheap history
* projection (no `content`); the estimate itself is a pure, deterministic
* function so it is unit-tested exhaustively without a DB.
*
* `tz` is the viewer's IANA zone (browser locale) — it moves which day a
* session lands in and where its windows sit, but never the total (§10).
*/
async computeWorkTime(
pageId: string,
tz = 'UTC',
config?: Partial<WorkTimeConfig>,
): Promise<PageWorkTime> {
const rows = await this.pageHistoryRepo.findTimelineByPageId(pageId);
const result = computeWorkTime(rows, config);
const usedConfig: WorkTimeConfig = { ...DEFAULT_WORK_TIME_CONFIG, ...config };
// `bucketByDay` consumes the pure core's un-bucketed sessions here; the
// full session list is NOT shipped on the response (no client reads it).
const perDay = bucketByDay(result.sessions, tz);
return {
workMs: result.workMs,
agentOnlyMs: result.agentOnlyMs,
perDay,
config: usedConfig,
tz,
};
}
}
@@ -0,0 +1,129 @@
import { bucketByDay, zonedDayStart } from './bucket-by-day';
import { computeWorkTime } from './compute-work-time';
import { WorkSession, TimelineSample } from './work-time.types';
const MIN = 60 * 1000;
const HOUR = 60 * MIN;
function work(start: number, end: number): WorkSession {
return { start, end, class: 'work' };
}
function agent(start: number, end: number): WorkSession {
return { start, end, class: 'agent_only' };
}
function sumActive(perDay: ReturnType<typeof bucketByDay>): number {
return perDay.reduce((a, d) => a + d.activeMs, 0);
}
describe('bucketByDay', () => {
it('Σ activeMs == workMs — the §6.3 consistency invariant', () => {
const rows: TimelineSample[] = [
{ createdAt: '2026-07-04T03:40:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-04T03:49:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-04T18:11:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-06T15:34:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
];
const r = computeWorkTime(rows);
const perDay = bucketByDay(r.sessions, 'UTC');
expect(sumActive(perDay)).toBe(r.workMs);
});
it('empty input → no days', () => {
expect(bucketByDay([], 'UTC')).toEqual([]);
});
it('midnight-crossing session splits across two days, sum preserved (§9#9)', () => {
const start = Date.UTC(2026, 0, 10, 23, 14);
const end = Date.UTC(2026, 0, 11, 0, 40);
const perDay = bucketByDay([work(start, end)], 'UTC');
expect(perDay).toHaveLength(2);
expect(perDay[0].dayISO).toBe('2026-01-10');
expect(perDay[1].dayISO).toBe('2026-01-11');
expect(perDay[0].activeMs).toBe(46 * MIN); // 23:14 → 24:00
expect(perDay[1].activeMs).toBe(40 * MIN); // 00:00 → 00:40
expect(sumActive(perDay)).toBe(end - start);
});
it('empty days between active days are emitted, not skipped (§9#12)', () => {
const d1 = work(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 11, 0));
const d3 = work(Date.UTC(2026, 0, 12, 10, 0), Date.UTC(2026, 0, 12, 11, 0));
const perDay = bucketByDay([d1, d3], 'UTC');
expect(perDay.map((d) => d.dayISO)).toEqual([
'2026-01-10',
'2026-01-11',
'2026-01-12',
]);
expect(perDay[1].activeMs).toBe(0);
expect(perDay[1].windows).toEqual([]);
});
it('agent_only windows are drawn but excluded from activeMs', () => {
const w = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 10, 0));
const a = agent(Date.UTC(2026, 0, 10, 14, 0), Date.UTC(2026, 0, 10, 14, 30));
const perDay = bucketByDay([w, a], 'UTC');
expect(perDay).toHaveLength(1);
expect(perDay[0].activeMs).toBe(1 * HOUR);
expect(perDay[0].agentMs).toBe(30 * MIN);
expect(perDay[0].windows.map((x) => x.class)).toEqual(['work', 'agent_only']);
});
it('work and agent_only are unioned SEPARATELY (agent does not swallow work)', () => {
// Overlapping work + agent windows on the same day.
const w = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 11, 0));
const a = agent(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 12, 0));
const perDay = bucketByDay([w, a], 'UTC');
expect(perDay[0].activeMs).toBe(2 * HOUR);
expect(perDay[0].agentMs).toBe(2 * HOUR);
});
it('overlapping same-class sessions are UNIONed, not summed (no double-count)', () => {
// Two work sessions that overlap 10:00–10:30 on one day.
const a = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 10, 30));
const b = work(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 11, 0));
const perDay = bucketByDay([a, b], 'UTC');
expect(perDay).toHaveLength(1);
// Union 09:00–11:00 = 2h, NOT 90m + 60m = 150m.
expect(perDay[0].activeMs).toBe(2 * HOUR);
// The drawn windows are also merged to one, so the punch-card cannot render
// an overlapping double bar.
expect(perDay[0].windows).toHaveLength(1);
expect(perDay[0].windows[0].start).toBe(a.start);
expect(perDay[0].windows[0].end).toBe(b.end);
});
it('DST fall-back: a full 25-hour day still balances (§9#14)', () => {
// America/New_York ends DST 2026-11-01 (25h day).
const tz = 'America/New_York';
const dayStart = zonedDayStart(Date.UTC(2026, 10, 1, 12, 0), tz);
const nextStart = zonedDayStart(dayStart + 26 * HOUR, tz);
expect(nextStart - dayStart).toBe(25 * HOUR);
const perDay = bucketByDay([work(dayStart, nextStart)], tz);
expect(perDay).toHaveLength(1);
expect(perDay[0].dayISO).toBe('2026-11-01');
expect(perDay[0].activeMs).toBe(25 * HOUR);
expect(sumActive(perDay)).toBe(nextStart - dayStart);
});
it('DST spring-forward: a full 23-hour day still balances (§9#14)', () => {
// America/New_York starts DST 2026-03-08 (23h day).
const tz = 'America/New_York';
const dayStart = zonedDayStart(Date.UTC(2026, 2, 8, 12, 0), tz);
const nextStart = zonedDayStart(dayStart + 26 * HOUR, tz);
expect(nextStart - dayStart).toBe(23 * HOUR);
const perDay = bucketByDay([work(dayStart, nextStart)], tz);
expect(perDay).toHaveLength(1);
expect(perDay[0].activeMs).toBe(23 * HOUR);
expect(sumActive(perDay)).toBe(nextStart - dayStart);
});
it('tz changes the day a session lands in but not the total', () => {
const start = Date.UTC(2026, 0, 10, 2, 0); // 02:00 UTC
const end = Date.UTC(2026, 0, 10, 3, 0);
const utc = bucketByDay([work(start, end)], 'UTC');
const ny = bucketByDay([work(start, end)], 'America/New_York'); // 21:00 prev day
expect(utc[0].dayISO).toBe('2026-01-10');
expect(ny[0].dayISO).toBe('2026-01-09');
expect(sumActive(utc)).toBe(sumActive(ny));
});
});
@@ -0,0 +1,180 @@
import { WorkSession, PerDay, DayWindow } from './work-time.types';
/**
* Merge intervals into a disjoint, sorted union. Overlapping OR touching
* intervals are joined. Empty input → [].
*/
function union(intervals: Array<[number, number]>): Array<[number, number]> {
if (intervals.length === 0) return [];
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
const out: Array<[number, number]> = [];
let [curStart, curEnd] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= curEnd) {
if (e > curEnd) curEnd = e;
} else {
out.push([curStart, curEnd]);
curStart = s;
curEnd = e;
}
}
out.push([curStart, curEnd]);
return out;
}
// Cache one Intl formatter per tz — constructing them is comparatively costly.
const fmtCache = new Map<string, Intl.DateTimeFormat>();
function partsFmt(tz: string): Intl.DateTimeFormat {
let fmt = fmtCache.get(tz);
if (!fmt) {
fmt = new Intl.DateTimeFormat('en-US', {
timeZone: tz,
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit',
second: '2-digit',
hour12: false,
});
fmtCache.set(tz, fmt);
}
return fmt;
}
interface WallParts {
year: number;
month: number;
day: number;
hour: number;
minute: number;
second: number;
}
/** Wall-clock parts of an instant in `tz` (DST-correct, via Intl). */
function wallParts(ms: number, tz: string): WallParts {
const parts = partsFmt(tz).formatToParts(new Date(ms));
const get = (type: string) =>
Number(parts.find((p) => p.type === type)?.value ?? '0');
let hour = get('hour');
// Intl emits "24" for midnight under some engines/locales; normalize to 0.
if (hour === 24) hour = 0;
return {
year: get('year'),
month: get('month'),
day: get('day'),
hour,
minute: get('minute'),
second: get('second'),
};
}
/** tz offset (wall − real) at an instant, in ms. */
function offset(ms: number, tz: string): number {
const p = wallParts(ms, tz);
const asUTC = Date.UTC(p.year, p.month - 1, p.day, p.hour, p.minute, p.second);
return asUTC - ms;
}
/**
* Epoch-ms of the local-midnight day start of `ms` in `tz`. DST-correct: takes
* the calendar day of the instant, its wall-midnight, then converts back with
* the offset that actually applies AT that midnight (refined once). The rare
* tz-with-a-DST-transition-exactly-at-midnight case is a documented edge (§9#14).
*/
export function zonedDayStart(ms: number, tz: string): number {
const p = wallParts(ms, tz);
const wallMidnightAsUTC = Date.UTC(p.year, p.month - 1, p.day, 0, 0, 0);
let start = wallMidnightAsUTC - offset(ms, tz);
// Refine with the offset at the computed midnight (DST may differ from `ms`).
start = wallMidnightAsUTC - offset(start, tz);
return start;
}
/** The next local midnight after `dayStart` (handles 23/25h DST days). */
function nextDayStart(dayStart: number, tz: string): number {
// +26h always lands inside the NEXT calendar day (day length ∈ [23h,25h]),
// never two days ahead; startOf('day') of it is the next midnight.
return zonedDayStart(dayStart + 26 * 60 * 60 * 1000, tz);
}
function isoDay(dayStart: number, tz: string): string {
const p = wallParts(dayStart, tz);
const pad = (n: number) => String(n).padStart(2, '0');
return `${p.year}-${pad(p.month)}-${pad(p.day)}`;
}
/** Clip a union to [lo, hi) and emit windows of `class`. */
function clip(
merged: Array<[number, number]>,
lo: number,
hi: number,
cls: DayWindow['class'],
): DayWindow[] {
const out: DayWindow[] = [];
for (const [s, e] of merged) {
const start = Math.max(s, lo);
const end = Math.min(e, hi);
if (end > start) out.push({ start, end, class: cls });
}
return out;
}
/**
* #395 §6.3 — bucket sessions into calendar days of `tz` for the punch-card.
* Pure and deterministic. `work` and `agent_only` are unioned SEPARATELY (else
* agent windows would swallow work windows on overlap), then each union is split
* at tz midnight boundaries (`startOf('day')` in tz, NOT "+24h" — DST-safe §9#14)
* and clipped to each day.
*
* By construction Σ perDay.activeMs == workMs: the days are a partition of the
* `work` union — no loss, no dup, even on 23/25h DST days. `agent_only` windows
* are drawn but NOT in activeMs. Empty days between the first and last active day
* are emitted (empty track + "—") so the rhythm/pauses stay visible.
*/
export function bucketByDay(sessions: WorkSession[], tz: string): PerDay[] {
const uWork = union(
sessions.filter((s) => s.class === 'work').map((s) => [s.start, s.end]),
);
const uAgent = union(
sessions
.filter((s) => s.class === 'agent_only')
.map((s) => [s.start, s.end]),
);
if (uWork.length === 0 && uAgent.length === 0) return [];
const minStart = Math.min(
uWork.length ? uWork[0][0] : Infinity,
uAgent.length ? uAgent[0][0] : Infinity,
);
const maxEnd = Math.max(
uWork.length ? uWork[uWork.length - 1][1] : -Infinity,
uAgent.length ? uAgent[uAgent.length - 1][1] : -Infinity,
);
const perDay: PerDay[] = [];
let dayStart = zonedDayStart(minStart, tz);
// Guard against a pathological non-advancing boundary.
let guard = 0;
while (dayStart < maxEnd && guard < 100000) {
guard++;
const dayEnd = nextDayStart(dayStart, tz);
const workWin = clip(uWork, dayStart, dayEnd, 'work');
const agentWin = clip(uAgent, dayStart, dayEnd, 'agent_only');
const activeMs = workWin.reduce((a, w) => a + (w.end - w.start), 0);
const agentMs = agentWin.reduce((a, w) => a + (w.end - w.start), 0);
const windows = [...workWin, ...agentWin].sort((a, b) => a.start - b.start);
perDay.push({
day: dayStart,
dayISO: isoDay(dayStart, tz),
activeMs,
agentMs,
windows,
});
dayStart = dayEnd;
}
return perDay;
}
@@ -0,0 +1,358 @@
import { computeWorkTime } from './compute-work-time';
import { bucketByDay } from './bucket-by-day';
import { TimelineSample, WorkSession } from './work-time.types';
const MIN = 60 * 1000;
/** Union wall-clock of a set of intervals (touching intervals merge). */
function unionMs(intervals: Array<[number, number]>): number {
if (intervals.length === 0) return 0;
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
let total = 0;
let [cs, ce] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= ce) {
if (e > ce) ce = e;
} else {
total += ce - cs;
cs = s;
ce = e;
}
}
return total + (ce - cs);
}
const ivsOf = (sessions: WorkSession[], cls?: string): Array<[number, number]> =>
sessions
.filter((x) => cls == null || x.class === cls)
.map((x) => [x.start, x.end] as [number, number]);
function s(
iso: string,
opts: {
source?: string | null;
chat?: string | null;
kind?: string | null;
by?: string | null;
} = {},
): TimelineSample {
return {
createdAt: `${iso}Z`,
lastUpdatedById: opts.by ?? 'human-1',
lastUpdatedSource: opts.source === undefined ? 'user' : opts.source,
lastUpdatedAiChatId: opts.chat ?? null,
kind: opts.kind ?? null,
};
}
// §7 config: T_gap=30m, P_in+P_out=10m, P_single=2m.
const S7 = { tGap: 30 * MIN, agentTGap: 30 * MIN, pIn: 5 * MIN, pOut: 5 * MIN, pSingle: 2 * MIN };
describe('computeWorkTime', () => {
it('§7 fixture — sessionizes 20-ish samples to ≈1h32m, not the ≈60h naive span', () => {
const rows: TimelineSample[] = [
// S1: multi-sample morning session
s('2026-07-04T03:40:00'),
s('2026-07-04T03:45:00'),
s('2026-07-04T03:49:00'),
// S2: agent burst (one run) then human supervising → class work
s('2026-07-04T15:43:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T15:47:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T15:50:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T16:13:00'),
// S3: single
s('2026-07-04T18:11:00'),
// S4: multi-sample evening session
s('2026-07-04T19:38:00'),
s('2026-07-04T19:44:00'),
s('2026-07-04T19:54:00'),
// S5 / S6: two singles two days later, 44m apart → two sessions at T_gap=30
s('2026-07-06T15:34:00'),
s('2026-07-06T16:18:00'),
];
const r = computeWorkTime(rows, S7);
// 19 + 40 + 2 + 26 + 2 + 2 = 91 minutes.
expect(r.workMs).toBe(91 * MIN);
expect(r.agentOnlyMs).toBe(0);
expect(r.sessions).toHaveLength(6);
expect(r.sessions.every((x) => x.class === 'work')).toBe(true);
const naiveSpan =
new Date('2026-07-06T16:18:00Z').getTime() -
new Date('2026-07-04T03:40:00Z').getTime();
expect(naiveSpan).toBeGreaterThan(60 * 60 * MIN); // ≈60h
expect(r.workMs).toBeLessThan(naiveSpan / 30); // dramatically smaller
});
it('n=0 → zero, no sessions', () => {
const r = computeWorkTime([]);
expect(r).toEqual({ workMs: 0, agentOnlyMs: 0, sessions: [] });
});
it('n=1 human → one P_single work session', () => {
const r = computeWorkTime([s('2026-07-04T10:00:00')], S7);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.workMs).toBe(2 * MIN);
expect(r.agentOnlyMs).toBe(0);
// pre-roll only: [t − P_single, t]
expect(r.sessions[0].end).toBe(new Date('2026-07-04T10:00:00Z').getTime());
});
it('n=1 agent → one P_single agent_only session, work=0 (§9#2)', () => {
const r = computeWorkTime(
[s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' })],
S7,
);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('agent_only');
expect(r.workMs).toBe(0);
expect(r.agentOnlyMs).toBe(2 * MIN);
});
it('MUST close the last session — the newest session is not lost (§9#1)', () => {
// Two singles a day apart: without the post-loop close, the 2nd is dropped.
const rows = [s('2026-07-04T10:00:00'), s('2026-07-05T10:00:00')];
const r = computeWorkTime(rows, S7);
expect(r.sessions).toHaveLength(2);
const lastStart = Math.max(...r.sessions.map((x) => x.start));
expect(lastStart).toBe(
new Date('2026-07-05T10:00:00Z').getTime() - 2 * MIN,
);
expect(r.workMs).toBe(4 * MIN);
});
it('agent-burst collapse: density does not inflate — length = wall-clock', () => {
const span = ['00', '01', '02', '03', '04', '05', '06'];
const dense: TimelineSample[] = span.map((sec) =>
s(`2026-07-04T10:00:${sec}`, { source: 'agent', chat: 'c1', kind: 'agent' }),
);
const sparse: TimelineSample[] = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:00:06', { source: 'agent', chat: 'c1', kind: 'agent' }),
];
const rDense = computeWorkTime(dense, S7);
const rSparse = computeWorkTime(sparse, S7);
// Same 6-second wall-clock span → same estimate regardless of snapshot count.
expect(rDense.agentOnlyMs).toBe(rSparse.agentOnlyMs);
expect(rDense.sessions).toHaveLength(1);
expect(rDense.sessions[0].class).toBe('agent_only');
});
it('supervisory agent time inside a human session counts as work, not agent', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:05:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:12:00'), // human within T_gap
];
const r = computeWorkTime(rows, S7);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.agentOnlyMs).toBe(0);
expect(r.workMs).toBeGreaterThan(0);
});
it('a DIFFERENT aiChatId breaks the burst — two agent runs, idle gap excluded', () => {
// Run c1 ends 10:05, run c2 starts 10:20 (15m > agentTGap 7m) → two sessions.
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:05:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:20:00', { source: 'agent', chat: 'c2', kind: 'agent' }),
s('2026-07-04T10:25:00', { source: 'agent', chat: 'c2', kind: 'agent' }),
];
const r = computeWorkTime(rows); // default agentTGap = 7m
expect(r.sessions).toHaveLength(2);
expect(r.sessions.every((x) => x.class === 'agent_only')).toBe(true);
// The 15m idle gap between the two runs is NOT counted.
const run1 = 5 * MIN + 5 * MIN + 5 * MIN; // pIn + span + pOut
expect(r.agentOnlyMs).toBe(2 * run1);
});
it('idle pulse (same/null run) is a full activity sample that continues a burst', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
// idle flush 4m later, null run id → continues the burst, not a new one
s('2026-07-04T10:04:00', { source: 'agent', chat: null, kind: 'idle' }),
s('2026-07-04T10:08:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
// burst span 10:00→10:08 (+pIn/pOut) = 8 + 10 = 18m
expect(r.agentOnlyMs).toBe(18 * MIN);
});
it('a USER-sourced idle breaks an agent burst → session is work, not agent_only', () => {
// A human supervision idle inherits source=user (aiChatId:null) and must NOT
// be swallowed into the agent burst. Δ=3m is within the default agentTGap so
// the two samples stay one session — but its class flips to `work`.
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:03:00', { source: 'user', chat: null, kind: 'idle' }),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.workMs).toBeGreaterThan(0);
// The human idle is NOT captured as agent_only time.
expect(r.agentOnlyMs).toBe(0);
// Σ over `work` sessions == workMs and Σ over `agent_only` == agentOnlyMs.
const sum = (cls: string) =>
r.sessions
.filter((x) => x.class === cls)
.reduce((acc, x) => acc + (x.end - x.start), 0);
expect(sum('work')).toBe(r.workMs);
expect(sum('agent_only')).toBe(r.agentOnlyMs);
});
it('idle pulse keeps a human writing session visible (not excluded)', () => {
const rows = [
s('2026-07-04T10:00:00'),
s('2026-07-04T10:08:00', { kind: 'idle' }), // pulse within T_gap
s('2026-07-04T10:15:00'),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
// span 10:00→10:15 + pIn/pOut = 15 + 10 = 25m
expect(r.workMs).toBe(25 * MIN);
});
it('git-source samples are excluded (§10 excludeGit)', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'git', kind: 'boundary' }),
s('2026-07-04T10:01:00', { source: 'git', kind: 'boundary' }),
];
expect(computeWorkTime(rows).workMs).toBe(0);
// ...but honoured off:
expect(
computeWorkTime(rows, { excludeGit: false }).workMs,
).toBeGreaterThan(0);
});
it('rejects an invalid config (tGap < pIn + pOut)', () => {
expect(() =>
computeWorkTime([s('2026-07-04T10:00:00')], {
tGap: 5 * MIN,
pIn: 5 * MIN,
pOut: 5 * MIN,
}),
).toThrow(/tGap/);
});
it('rejects an invalid config (2·agentTGap < pIn + pOut)', () => {
// tGap (default 15m) still ≥ pIn+pOut, so only the 2·agentTGap guard trips.
// Without it a short session of one class between two of the other could
// produce a NON-adjacent cross-class overlap the adjacent-only clip misses.
expect(() =>
computeWorkTime([s('2026-07-04T10:00:00')], {
agentTGap: 2 * MIN,
pIn: 5 * MIN,
pOut: 5 * MIN,
}),
).toThrow(/agentTGap/);
});
// F1 — cross-class double-count. On the DEFAULT config agentTGap (7m) < pIn+pOut
// (10m), so a `work` session ending in an agent segment and a nearby separate
// `agent_only` run (gap in (7m,10m]) used to produce OVERLAPPING padded
// intervals — the same wall-clock counted into BOTH workMs and agentOnlyMs. The
// cross-class padding clip must make the two per-class unions disjoint.
it('does NOT double-count wall-clock across work/agent_only (§F1)', () => {
// user@0s ; agent(chatX)@60s (breaks into a work session with the human) ;
// agent(chatY)@560s,590s (a separate agent_only run). Raw gap between the work
// session (ends 60s) and the agent run (starts 560s) is 500s ∈ (agentTGap,
// pIn+pOut] once padded — the classic overlap window.
const rows: TimelineSample[] = [
s('2026-07-04T00:00:00'), // user @ 0s
s('2026-07-04T00:01:00', { source: 'agent', chat: 'cX', kind: 'agent' }), // @ 60s
s('2026-07-04T00:09:20', { source: 'agent', chat: 'cY', kind: 'agent' }), // @ 560s
s('2026-07-04T00:09:50', { source: 'agent', chat: 'cY', kind: 'agent' }), // @ 590s
];
const r = computeWorkTime(rows); // DEFAULT config
// Both classes present.
expect(r.workMs).toBeGreaterThan(0);
expect(r.agentOnlyMs).toBeGreaterThan(0);
// Per-class metrics are exactly their own union (union, not Σ).
expect(r.workMs).toBe(unionMs(ivsOf(r.sessions, 'work')));
expect(r.agentOnlyMs).toBe(unionMs(ivsOf(r.sessions, 'agent_only')));
// The F1 invariant: work-union and agent-union are cross-class-disjoint, so
// the union of ALL padded intervals equals workMs + agentOnlyMs (no overlap).
// With the clip disabled this fails (union < sum by the 100s overlap).
expect(unionMs(ivsOf(r.sessions))).toBe(r.workMs + r.agentOnlyMs);
});
// F1 property/fuzz — random timelines across several timezones must uphold the
// work-time invariants. Backs the (corrected) PR claim of a real fuzz test.
it('property: random timelines uphold union & cross-class-disjoint invariants', () => {
// Deterministic LCG (numerical-recipes constants) so a failure is reproducible.
let seed = 0x9e3779b9 >>> 0;
const rand = () => {
seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0;
return seed / 0x100000000;
};
const pick = <T>(arr: T[]): T => arr[Math.floor(rand() * arr.length)];
const tzs = [
'UTC',
'America/New_York',
'Europe/Moscow',
'Australia/Lord_Howe', // 30-min DST offset — a nasty bucket stress
];
const base = Date.UTC(2026, 5, 1, 0, 0, 0); // 2026-06-01Z
const chats = ['c1', 'c2', 'c3'];
for (let iter = 0; iter < 250; iter++) {
const tz = pick(tzs);
const n = 2 + Math.floor(rand() * 18); // 2..19 rows
const rows: TimelineSample[] = [];
// Walk time forward by a random inter-sample gap. The gap distribution is
// centred on the DANGEROUS band — a bit under to a bit over pIn+pOut (10m)
// AND straddling agentTGap (7m) — so adjacent samples routinely split into
// separate sessions whose ±P padding would overlap if a class boundary sits
// there. Mixing user/agent classes at these gaps reliably manufactures the
// work-ending-in-agent → agent_only cross-class boundary F1 is about, plus
// dense within-class runs (occasional 0–2m gaps) that exercise the union.
let t = base + Math.floor(rand() * 60 * MIN);
for (let i = 0; i < n; i++) {
const roll = rand();
const gap =
roll < 0.25
? Math.floor(rand() * 2 * MIN) // dense burst (same-class union)
: roll < 0.85
? 5 * MIN + Math.floor(rand() * 8 * MIN) // 5–13m: the split band
: 20 * MIN + Math.floor(rand() * 40 * MIN); // long idle → new day-ish
t += gap;
const iso = new Date(t).toISOString().slice(0, 19); // 'YYYY-MM-DDTHH:MM:SS'
const isAgent = rand() < 0.5;
rows.push(
isAgent
? s(iso, { source: 'agent', chat: pick(chats), kind: 'agent' })
: s(iso, { source: 'user', kind: rand() < 0.3 ? 'idle' : 'manual' }),
);
}
const r = computeWorkTime(rows); // DEFAULT config
const workIvs = ivsOf(r.sessions, 'work');
const agentIvs = ivsOf(r.sessions, 'agent_only');
// (1) each metric is exactly its per-class union (catches a union→Σ regress).
expect(r.workMs).toBe(unionMs(workIvs));
expect(r.agentOnlyMs).toBe(unionMs(agentIvs));
// (2) NO cross-class overlap: union(all) == workMs + agentOnlyMs (F1).
expect(unionMs(ivsOf(r.sessions))).toBe(r.workMs + r.agentOnlyMs);
// (3) bucket invariant: Σ per-day activeMs == workMs (§6.3).
const perDay = bucketByDay(r.sessions, tz);
const sumActive = perDay.reduce((a, d) => a + d.activeMs, 0);
expect(sumActive).toBe(r.workMs);
}
});
});
@@ -0,0 +1,274 @@
import {
TimelineSample,
WorkSession,
WorkTimeResult,
} from './work-time.types';
import { WorkTimeConfig, resolveWorkTimeConfig } from './work-time.config';
/** A normalized activity sample (one history row), createdAt as epoch-ms. */
interface NormSample {
t: number;
isAgent: boolean;
aiChatId: string | null;
kind: string | null;
}
/**
* A collapsed segment: either a scalar sample (t_start == t_end) or an
* agent-burst spanning several agent samples of one run (§5.1). It participates
* in sessionization as a single "sample".
*/
interface Segment {
tStart: number;
tEnd: number;
isAgent: boolean;
}
function toMs(v: Date | string | number): number {
if (v instanceof Date) return v.getTime();
if (typeof v === 'number') return v;
return new Date(v).getTime();
}
/**
* Normalize raw rows → sorted, deduped activity samples. `git` is dropped when
* configured; every other kind (incl. `idle` — the continuous-work pulse §3) is
* a real activity sample. Sort is by createdAt ASC; samples whose timestamps
* fall in the same `dedupRoundMs` bucket collapse to one (§9#7: a synchronous
* boundary row + the immediate agent snapshot can share a createdAt). A merged
* sample is human unless EVERY member is an agent, so supervision never gets
* mis-attributed to the agent.
*/
function normalize(
rows: TimelineSample[],
config: WorkTimeConfig,
): NormSample[] {
const samples: NormSample[] = [];
for (const row of rows) {
const source = row.lastUpdatedSource;
if (config.excludeGit && source === 'git') continue;
samples.push({
t: toMs(row.createdAt),
isAgent: source === 'agent',
aiChatId: row.lastUpdatedAiChatId ?? null,
kind: row.kind ?? null,
});
}
samples.sort((a, b) => a.t - b.t);
if (config.dedupRoundMs <= 0 || samples.length < 2) return samples;
const deduped: NormSample[] = [];
for (const s of samples) {
const prev = deduped[deduped.length - 1];
if (prev && s.t - prev.t < config.dedupRoundMs) {
// Merge into the previous sample. Human wins the class; keep the earliest
// t; keep a non-null aiChatId if either has one (so a bare boundary row
// does not erase the run id).
prev.isAgent = prev.isAgent && s.isAgent;
prev.aiChatId = prev.aiChatId ?? s.aiChatId;
// Prefer the more specific kind (a real kind over a null/boundary) only
// matters for burst continuation; keep prev.kind (earliest) as-is.
continue;
}
deduped.push({ ...s });
}
return deduped;
}
/**
* Collapse consecutive same-run agent samples into one burst segment (§5.1) so a
* dense burst (8 snapshots in 7 minutes) contributes its wall-clock, not a count
* × block. A burst is broken by any sample NOT continuing the same aiChatId
* agent run: a non-agent sample, a `boundary` (actor transition), or a DIFFERENT
* aiChatId. Only an AGENT-sourced `idle` pulse with the SAME or a null aiChatId
* continues the burst (its label lags the real edit ≤ maxWait, well within
* rounding); a user-sourced `idle` (a human supervision pulse) breaks it.
*/
function collapse(samples: NormSample[], config: WorkTimeConfig): Segment[] {
const segments: Segment[] = [];
let burst: { chatId: string | null; tStart: number; tEnd: number } | null =
null;
const flush = () => {
if (!burst) return;
let tEnd = burst.tEnd;
if (config.burstCapMs != null && tEnd - burst.tStart > config.burstCapMs) {
tEnd = burst.tStart + config.burstCapMs;
}
segments.push({ tStart: burst.tStart, tEnd, isAgent: true });
burst = null;
};
for (const s of samples) {
// An agent-sourced idle pulse continues the current agent burst (same or
// null run id). A user-sourced idle (human supervision) must NOT be swallowed
// here — it falls through to the human branch so the session flips to `work`.
if (
burst &&
s.kind === 'idle' &&
s.isAgent &&
(s.aiChatId === burst.chatId || s.aiChatId == null)
) {
burst.tEnd = s.t;
continue;
}
if (s.isAgent && s.kind !== 'boundary') {
if (burst && burst.chatId === s.aiChatId) {
burst.tEnd = s.t;
} else {
flush();
burst = { chatId: s.aiChatId, tStart: s.t, tEnd: s.t };
}
continue;
}
// A human sample, a boundary, or an agent-boundary: breaks the burst and is
// itself a zero-width segment (its class follows its own source).
flush();
segments.push({ tStart: s.t, tEnd: s.t, isAgent: s.isAgent });
}
flush();
return segments;
}
function gapThreshold(
a: Segment,
b: Segment,
config: WorkTimeConfig,
): number {
return a.isAgent && b.isAgent ? config.agentTGap : config.tGap;
}
/** Merge intervals; overlapping OR touching intervals are unioned. */
function unionDuration(intervals: Array<[number, number]>): number {
if (intervals.length === 0) return 0;
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
let total = 0;
let [curStart, curEnd] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= curEnd) {
if (e > curEnd) curEnd = e;
} else {
total += curEnd - curStart;
curStart = s;
curEnd = e;
}
}
total += curEnd - curStart;
return total;
}
/**
* #395 core — estimate time worked on a page from its history timeline (§5).
* Pure and deterministic: no DB, no clock, no I/O.
*
* Pipeline: normalize+dedup → collapse agent bursts → ONE sessionization pass
* over all segments (threshold depends on the pair: both-agent → agentTGap, else
* tGap; the last session is ALWAYS closed after the loop) → class per finished
* session (all-agent → agent_only, else work) → pad each session (multi-sample
* → [first−P_in, last+P_out]; lone scalar → [t−P_single, t]) → clip padding of
* adjacent DIFFERENT-class sessions at the raw-gap midpoint (so work/agent_only
* never overlap) → metrics are the union wall-clock within each class (union, not
* Σ, so overlaps never double, and cross-class-disjoint by the clip above).
*/
export function computeWorkTime(
rows: TimelineSample[],
config?: Partial<WorkTimeConfig>,
): WorkTimeResult {
const cfg = resolveWorkTimeConfig(config);
const samples = normalize(rows, cfg);
const segments = collapse(samples, cfg);
// Sessionize — one pass over ALL segments.
const rawSessions: Segment[][] = [];
let cur: Segment[] | null = null;
for (const seg of segments) {
if (cur == null) {
cur = [seg];
} else {
const last = cur[cur.length - 1];
if (seg.tStart - last.tEnd <= gapThreshold(last, seg, cfg)) {
cur.push(seg);
} else {
rawSessions.push(cur);
cur = [seg];
}
}
}
if (cur != null) rawSessions.push(cur); // MUST close the last session (§5, §9#1)
// A finished session with BOTH its raw (unpadded) span and its padded bounds.
// `rawSessions` are already in ascending time order, so `built` is too.
interface BuiltSession {
rawStart: number;
rawEnd: number;
padStart: number;
padEnd: number;
cls: WorkSession['class'];
}
const built: BuiltSession[] = [];
for (const segs of rawSessions) {
const first = segs[0];
const last = segs[segs.length - 1];
const cls = segs.every((s) => s.isAgent) ? 'agent_only' : 'work';
let padStart: number;
let padEnd: number;
if (segs.length === 1 && first.tStart === first.tEnd) {
// Lone single-instant session (one scalar, or a one-snapshot agent run):
// pre-roll only, no invented "future" work (§5).
padStart = first.tStart - cfg.pSingle;
padEnd = first.tStart;
} else {
padStart = first.tStart - cfg.pIn;
padEnd = last.tEnd + cfg.pOut;
}
built.push({
rawStart: first.tStart,
rawEnd: last.tEnd,
padStart,
padEnd,
cls,
});
}
// Clip cross-class padding so a `work` and an `agent_only` session that abut
// never claim the same wall-clock. For each ADJACENT pair of DIFFERENT classes,
// cap the earlier session's trailing pad and the later session's leading pad at
// the MIDPOINT of the raw (unpadded) inactivity gap between them: the earlier
// padded interval then ends ≤ midpoint and the later one starts ≥ midpoint, so
// the two are disjoint (they touch at most at the midpoint). This makes the
// per-class unions (workMs / agentOnlyMs) cross-class-disjoint BY CONSTRUCTION
// — closing the double-count where a work session ending in an agent segment
// and a nearby agent_only session (gap in (agentTGap, pIn+pOut]) overlapped and
// were counted into both metrics (§5, §9). Within-class adjacency is left
// untouched: `unionDuration` already dedups it, and clipping there could perturb
// the per-class metric value.
for (let i = 1; i < built.length; i++) {
const a = built[i - 1];
const b = built[i];
if (a.cls === b.cls) continue;
const midpoint = (a.rawEnd + b.rawStart) / 2;
if (a.padEnd > midpoint) a.padEnd = midpoint;
if (b.padStart < midpoint) b.padStart = midpoint;
}
const sessions: WorkSession[] = [];
const workIvs: Array<[number, number]> = [];
const agentIvs: Array<[number, number]> = [];
for (const s of built) {
sessions.push({ start: s.padStart, end: s.padEnd, class: s.cls });
(s.cls === 'work' ? workIvs : agentIvs).push([s.padStart, s.padEnd]);
}
sessions.sort((a, b) => a.start - b.start);
return {
workMs: unionDuration(workIvs),
agentOnlyMs: unionDuration(agentIvs),
sessions,
};
}
@@ -0,0 +1,15 @@
export { computeWorkTime } from './compute-work-time';
export { bucketByDay, zonedDayStart } from './bucket-by-day';
export {
DEFAULT_WORK_TIME_CONFIG,
resolveWorkTimeConfig,
} from './work-time.config';
export type { WorkTimeConfig } from './work-time.config';
export type {
TimelineSample,
WorkSession,
WorkTimeResult,
SessionClass,
DayWindow,
PerDay,
} from './work-time.types';
@@ -0,0 +1,102 @@
import {
IDLE_MAX_WAIT_USER,
IDLE_MAX_WAIT_AGENT,
} from '../../../collaboration/constants';
/**
* #395 — tunables for the work-time estimate (§10). Defaults are calibrated off
* #374's idle-pulse ceilings: after #374 a continuous editing session leaves a
* history row at least every ~IDLE_MAX_WAIT (10m user / 5m agent), so a gap
* WIDER than that ceiling contains un-pulsed idle time = (partial) inactivity.
* `tGap` therefore sits a little above the user ceiling, `agentTGap` a little
* above the agent ceiling — a gap within the threshold is pulse-backed and
* counts as work.
*/
export interface WorkTimeConfig {
/** user inactivity timeout: gap ≤ tGap between samples = continuous work. */
tGap: number;
/** timeout for a pair of consecutive agent samples (tighter than tGap). */
agentTGap: number;
/** pre-roll padding for a multi-sample session (work began before sample 1). */
pIn: number;
/** post-roll padding for a multi-sample session (work continued after last). */
pOut: number;
/** block for a lone single-sample session (pre-roll only, no invented future). */
pSingle: number;
/** drop `git`-source samples (they are not human/agent article work). */
excludeGit: boolean;
/** optional cap on one collapsed agent-burst segment's wall-clock (§9#3). */
burstCapMs?: number;
/** samples whose createdAt round to the same bucket dedup to one (§9#7). */
dedupRoundMs: number;
}
export const DEFAULT_WORK_TIME_CONFIG: WorkTimeConfig = {
// ~15m: IDLE_MAX_WAIT_USER (10m) + headroom. Empirically backcast on a real
// 307-snapshot article (≈24h at 15m matched the owner's estimate; 30/45m
// over-counted). See #395 §10.
tGap: 15 * 60 * 1000,
// ~7m: IDLE_MAX_WAIT_AGENT (5m) + headroom.
agentTGap: 7 * 60 * 1000,
pIn: 5 * 60 * 1000,
pOut: 5 * 60 * 1000,
pSingle: 2 * 60 * 1000,
excludeGit: true,
burstCapMs: undefined,
dedupRoundMs: 1000,
};
// Compile-time cross-check that the defaults really are pulse-anchored — if a
// future edit moves the #374 ceilings, this reminds us to re-calibrate.
void IDLE_MAX_WAIT_USER;
void IDLE_MAX_WAIT_AGENT;
/**
* Fill a partial config with defaults and validate it. Cross-class metric
* disjointness is guaranteed jointly by `computeWorkTime`'s adjacent-pair padding
* clip (it caps the padding of adjacent DIFFERENT-class sessions at the raw-gap
* midpoint) AND the two bounds enforced below (§5):
* - `tGap ≥ pIn + pOut`: a session's own padding never exceeds its inactivity
* window.
* - `2·agentTGap ≥ pIn + pOut`: makes the adjacent-only clip provably COMPLETE.
* A NON-adjacent (i, i+2) cross-class overlap could only arise from two
* same-class sessions separated by a full intervening session of the other
* class; that separation spans at least two inter-session gaps, each strictly
* `> agentTGap`, so it is `> 2·agentTGap`. Requiring `2·agentTGap ≥ pIn + pOut`
* means even the widest padded reach (pIn + pOut) cannot bridge it — so the
* only cross-class overlaps possible are between ADJACENT sessions, which the
* clip handles. `workMs`/`agentOnlyMs` are therefore disjoint by construction.
*/
export function resolveWorkTimeConfig(
partial?: Partial<WorkTimeConfig>,
): WorkTimeConfig {
const config = { ...DEFAULT_WORK_TIME_CONFIG, ...(partial ?? {}) };
for (const key of [
'tGap',
'agentTGap',
'pIn',
'pOut',
'pSingle',
'dedupRoundMs',
] as const) {
const value = config[key];
if (!Number.isFinite(value) || value < 0) {
throw new Error(`work-time config: ${key} must be a non-negative number`);
}
}
if (config.burstCapMs != null && config.burstCapMs <= 0) {
throw new Error('work-time config: burstCapMs must be > 0 when set');
}
if (config.tGap < config.pIn + config.pOut) {
throw new Error(
"work-time config: tGap must be ≥ pIn + pOut (a session's padding may not exceed its inactivity window)",
);
}
if (2 * config.agentTGap < config.pIn + config.pOut) {
throw new Error(
'work-time config: 2·agentTGap must be ≥ pIn + pOut (so non-adjacent cross-class padding cannot overlap)',
);
}
return config;
}
@@ -0,0 +1,73 @@
/**
* #395 — "time worked on an article" domain types.
*
* The estimate is built by sessionizing a page's `page_history` timeline on
* inactivity gaps (WakaTime-style), NOT by taking the span between the first and
* last edit (which over-counts sleep / lunch / idle days). See the design doc in
* issue #395 §5–§6.3 for the normative algorithm.
*/
/**
* A single `page_history` row projected for the work-time computation — the
* cheap columns only (no `content`). Produced by
* `PageHistoryRepo.findTimelineByPageId`. `createdAt` is whatever the DB driver
* hands back (Date); the pure core normalizes it to epoch-ms itself so it stays
* deterministic and DB-free.
*/
export interface TimelineSample {
createdAt: Date | string | number;
lastUpdatedById: string | null;
/** 'user' | 'agent' | 'git' | null (legacy autosave = human). */
lastUpdatedSource: string | null;
lastUpdatedAiChatId: string | null;
/** #370 tier: 'manual' | 'agent' | 'idle' | 'boundary' | null (legacy). */
kind: string | null;
}
/** A finished session's class (§5.1). */
export type SessionClass = 'work' | 'agent_only';
/**
* A finished session: absolute wall-clock bounds already padded with P_in/P_out
* (multi-sample) or P_single (single scalar), plus its class. This is enough for
* both the metrics and the per-day punch-card colouring.
*/
export interface WorkSession {
/** epoch-ms, inclusive lower bound (already P-padded). */
start: number;
/** epoch-ms, exclusive upper bound (already P-padded). */
end: number;
class: SessionClass;
}
/** Output of {@link computeWorkTime}. */
export interface WorkTimeResult {
/** union wall-clock of `work` sessions, ms (the headline metric). */
workMs: number;
/** union wall-clock of `agent_only` sessions, ms (secondary). */
agentOnlyMs: number;
sessions: WorkSession[];
}
/** One activity window inside a calendar day (already clipped to the day). */
export interface DayWindow {
/** epoch-ms. */
start: number;
/** epoch-ms. */
end: number;
class: SessionClass;
}
/** One calendar day of the punch-card (§6.3). */
export interface PerDay {
/** epoch-ms of the local-midnight day start in the requested tz. */
day: number;
/** 'YYYY-MM-DD' in the requested tz — stable, tz-independent label. */
dayISO: string;
/** Σ of `work` windows this day, ms. Σ over days == workMs (invariant §6.3). */
activeMs: number;
/** Σ of `agent_only` windows this day, ms (drawn, NOT in activeMs). */
agentMs: number;
/** both classes, clipped to the day, sorted by start (for drawing). */
windows: DayWindow[];
}
@@ -1,33 +1,56 @@
import { Space } from '@docmost/db/types/entity.types';
export class SearchResponseDto {
// #529 A7 — the single per-hit SUPERSET returned by the unified search engine.
// The web-UI reads id/highlight/icon/space/title/…; the MCP agent maps id→pageId
// and reads snippet/score/path. `rank`/`highlight` are null for substring-only
// hits (the web already falls back). Nothing the legacy web response carried is
// dropped.
export class SearchResultDto {
id: string;
title: string;
// Alias of `id` for the MCP layer (it addresses pages by pageId).
pageId: string;
slugId: string;
icon: string;
parentPageId: string;
title: string;
space?: Partial<Space>;
creatorId: string;
rank: number;
highlight: string;
createdAt: Date;
updatedAt: Date;
space: Partial<Space>;
// ts_rank_cd of the FTS branch; null for substring-only hits.
rank: number | null;
// ts_headline marked HTML; null for substring-only hits.
highlight: string | null;
// Plain windowed snippet around the match (empty for titleOnly).
snippet: string;
// Ancestor titles root → direct parent ([] for a root page).
path: string[];
// Per-response ordering proxy (falls back to rank).
score: number;
// Which fields matched: 'title' and/or 'text'.
matchedFields: string[];
// Which parsed positive/required terms this hit matched.
matchedTerms: string[];
}
// Response shape for the opt-in agent-lookup mode (#443, `substring: true`).
// Additive to the FTS response: carries the location (`path`), a windowed
// `snippet` around the first match and a per-response sort `score`. The MCP
// layer maps `id → pageId`; `slugId` is never exposed.
export class SearchLookupResponseDto {
id: string;
slugId: string;
title: string;
parentPageId: string | null;
// Ancestor titles from the space root down to the direct parent; [] for a
// root page.
path: string[];
// ~300–500 chars around the first match (or a leading text window / extended
// ts_headline fallback).
snippet: string;
// 0..1 float, meaningful ONLY for sorting within one response.
score: number;
// The paginated envelope (A5). `total` is the EXACT permission-filtered count of
// pages matching the positive lexical query (fail-closed). `hasMore` is true when
// more results exist WITHIN the fusion window; `truncatedAtCap` signals the match
// set exceeded CANDIDATE_CAP and the tail is unreachable by pagination.
export class SearchResponseDto {
items: SearchResultDto[];
total: number;
hasMore: boolean;
truncatedAtCap: boolean;
offset: number;
query: {
raw: string;
parsed: {
positive: string[];
required: string[];
excluded: string[];
reason?: string;
};
mode: 'or' | 'and';
match: string;
};
}
+22 -4
View File
@@ -1,16 +1,30 @@
import {
IsBoolean,
IsIn,
IsNotEmpty,
IsNumber,
IsOptional,
IsString,
MaxLength,
} from 'class-validator';
export class SearchDTO {
// Defense-in-depth cap on the raw query length. The real stack-depth bound is
// the parser's MAX_PARSED_TERMS term cap (see search-query-parser.ts); this
// just rejects absurd payloads early. 10k chars still comfortably holds any
// legitimate query.
@IsNotEmpty()
@IsString()
@MaxLength(10000)
query: string;
// #529 A3 — match mode. `auto` (default) routes identifier-like terms
// (10.31.41, esp32, WB-MGE-30D86B) to the substring/trigram branch and words
// to full-text; `word`/`prefix`/`substring` are explicit overrides.
@IsOptional()
@IsIn(['auto', 'word', 'prefix', 'substring'])
match?: 'auto' | 'word' | 'prefix' | 'substring';
@IsOptional()
@IsString()
spaceId: string;
@@ -33,15 +47,19 @@ export class SearchDTO {
// --- Opt-in agent-lookup mode (#443). ------------------------------------
// These fields are ADDITIVE and default-off: a web client that sends none of
// them gets byte-identical FTS behaviour and result shape. They are only read
// by the substring/path/snippet code path in SearchService.searchPage.
// them gets byte-identical FTS behaviour and result shape. In the unified #529
// engine, `parentPageId` and `titleOnly` are read by SearchService.searchPage
// (subtree scoping and title-only matching, respectively). `substring` is NOT
// read by the native driver — it is accepted-but-ignored, kept only for
// back-compat with the upstream lookup request shape.
//
// NOTE (standalone stdio vs stock upstream): stock upstream validates this DTO
// with `whitelist: true`, so an older server silently strips these unknown
// fields and the request degrades gracefully to the plain FTS behaviour.
// Enables the hybrid substring branch (title + text_content LIKE) merged with
// the existing FTS branch, plus tiered ranking, path and windowed snippet.
// Accepted-but-ignored by the #529 native driver (kept for upstream lookup
// back-compat). The unified engine ALWAYS runs the hybrid FTS + substring/
// trigram branches with tiered ranking, so this flag no longer toggles anything.
@IsOptional()
@IsBoolean()
substring?: boolean;
@@ -0,0 +1,168 @@
import {
parseSearchQuery,
hasPositiveRecall,
MAX_PARSED_TERMS,
} from './search-query-parser';
describe('parseSearchQuery — tokenization & operators (A2)', () => {
it('splits on whitespace into positive terms (OR recall)', () => {
const p = parseSearchQuery('стамбул роснефть');
expect(p.positive.map((t) => t.text)).toEqual(['стамбул', 'роснефть']);
expect(p.required).toEqual([]);
expect(p.excluded).toEqual([]);
expect(p.mode).toBe('or');
});
it('treats +term as required and -term as excluded', () => {
const p = parseSearchQuery('+кофейня -архив');
expect(p.required.map((t) => t.text)).toEqual(['кофейня']);
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive).toEqual([]);
});
it('keeps a leading-operator-free hyphen/dot/colon token as ONE literal term', () => {
// WB-MGE-30D86B, 10.0.12.5, a:b — internal -,.,: are literal, one term each.
expect(parseSearchQuery('WB-MGE-30D86B').positive[0].text).toBe(
'WB-MGE-30D86B',
);
expect(parseSearchQuery('10.0.12.5').positive[0].text).toBe('10.0.12.5');
expect(parseSearchQuery('host:8080').positive[0].text).toBe('host:8080');
});
it('only a LEADING +/- is an operator; -архив excludes архив', () => {
const p = parseSearchQuery('-архив');
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive).toEqual([]);
});
it('drops a bare "-" / "+" and an all-operator remainder', () => {
const p = parseSearchQuery('- + foo -- ++');
expect(p.positive.map((t) => t.text)).toEqual(['foo']);
expect(p.required).toEqual([]);
expect(p.excluded).toEqual([]);
});
it('parses a quoted phrase as one adjacency term', () => {
const p = parseSearchQuery('"воздушный шар" кофе');
expect(p.positive[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
expect(p.positive[1].text).toBe('кофе');
});
it('applies +/- to a phrase', () => {
const req = parseSearchQuery('+"воздушный шар"');
expect(req.required[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
const exc = parseSearchQuery('-"воздушный шар"');
expect(exc.excluded[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
});
it('drops an unbalanced quote token', () => {
const p = parseSearchQuery('kafka "unclosed here');
expect(p.positive.map((t) => t.text)).toEqual(['kafka']);
});
it('strips tsquery metacharacters from a bare FTS term (no 500)', () => {
const p = parseSearchQuery('foo|bar');
// `|` is not an identifier signal → FTS branch; the metachar is stripped so
// the term becomes the two words that survive.
expect(p.positive[0].branch === 'fts' || p.positive[0].branch === 'ftsPrefix').toBe(true);
expect(p.positive[0].text).toBe('foo bar');
});
});
describe('parseSearchQuery — match=auto classification (A3)', () => {
it('routes identifier-like terms to the substring branch', () => {
expect(parseSearchQuery('10.31.41').positive[0].branch).toBe('substring');
expect(parseSearchQuery('esp32').positive[0].branch).toBe('substring');
expect(parseSearchQuery('WB-MGE-30D86B').positive[0].branch).toBe('substring');
});
it('routes purely-alphabetic words to the FTS (prefix) branch', () => {
expect(parseSearchQuery('печат').positive[0].branch).toBe('ftsPrefix');
expect(parseSearchQuery('ресторан').positive[0].branch).toBe('ftsPrefix');
});
it('explicit match overrides: word / prefix / substring', () => {
expect(parseSearchQuery('печат', { match: 'word' }).positive[0].branch).toBe('fts');
expect(parseSearchQuery('печат', { match: 'prefix' }).positive[0].branch).toBe(
'ftsPrefix',
);
expect(parseSearchQuery('печат', { match: 'substring' }).positive[0].branch).toBe(
'substring',
);
});
});
describe('parseSearchQuery — reasons & recall', () => {
it('only-negation yields reason only-negation and no positive recall', () => {
const p = parseSearchQuery('-архив');
expect(p.reason).toBe('only-negation');
expect(hasPositiveRecall(p)).toBe(false);
});
it('empty / whitespace / garbage yields reason empty', () => {
expect(parseSearchQuery('').reason).toBe('empty');
expect(parseSearchQuery(' ').reason).toBe('empty');
// A bare operator drops to nothing → empty (no exclusion survived).
expect(parseSearchQuery('+ -').reason).toBe('empty');
});
it('a required term alone IS positive recall (no reason)', () => {
const p = parseSearchQuery('+кофейня');
expect(p.reason).toBeUndefined();
expect(hasPositiveRecall(p)).toBe(true);
});
it('mode flag flows through', () => {
expect(parseSearchQuery('a b', { mode: 'and' }).mode).toBe('and');
expect(parseSearchQuery('a b').mode).toBe('or');
});
});
describe('parseSearchQuery — term cap (stack-depth guard)', () => {
it('caps the total parsed terms at MAX_PARSED_TERMS without throwing', () => {
// A pasted text block: far more words than the cap. The parser must bound the
// SQL tsquery nesting depth (else Postgres blows its stack → HTTP 500).
const words = Array.from({ length: 5000 }, (_, i) => `w${i}`);
let p!: ReturnType<typeof parseSearchQuery>;
expect(() => {
p = parseSearchQuery(words.join(' '));
}).not.toThrow();
const total = p.positive.length + p.required.length + p.excluded.length;
expect(total).toBe(MAX_PARSED_TERMS);
// Stable order: the FIRST cap terms are kept.
expect(p.positive.slice(0, 3).map((t) => t.text)).toEqual(['w0', 'w1', 'w2']);
expect(p.positive[MAX_PARSED_TERMS - 1].text).toBe(`w${MAX_PARSED_TERMS - 1}`);
});
it('counts positive + required + excluded together toward the cap', () => {
// Interleave operators so overflow can fall on any bucket. Total must still
// never exceed the cap, and the first cap terms (in stable order) win.
const tokens: string[] = [];
for (let i = 0; i < 200; i++) {
const op = i % 3 === 0 ? '+' : i % 3 === 1 ? '-' : '';
tokens.push(`${op}t${i}`);
}
const p = parseSearchQuery(tokens.join(' '));
const total = p.positive.length + p.required.length + p.excluded.length;
expect(total).toBe(MAX_PARSED_TERMS);
// t0 (+, required) and t1 (-, excluded) and t2 (bare, positive) are all within
// the first cap tokens → each bucket got its leading terms.
expect(p.required[0].text).toBe('t0');
expect(p.excluded[0].text).toBe('t1');
expect(p.positive[0].text).toBe('t2');
});
it('leaves a normal (<= cap) query completely unchanged', () => {
const p = parseSearchQuery('+кофейня -архив "воздушный шар" ресторан 10.31.41');
expect(p.required.map((t) => t.text)).toEqual(['кофейня']);
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive.map((t) => t.text)).toEqual([
'воздушный шар',
'ресторан',
'10.31.41',
]);
// Phrase / substring branch handling survives under the cap.
expect(p.positive[0].branch).toBe('phrase');
expect(p.positive[2].branch).toBe('substring');
});
});
@@ -0,0 +1,242 @@
// #529 Phase A — server-side query parser (the SINGLE source of query semantics).
//
// Clients (web-UI, MCP agent, public share) send a RAW query string plus flags
// (`match`, `mode`); ALL query interpretation happens HERE, so every consumer
// gets identical operator/phrase/morphology behaviour. The parser is PURE (no
// SQL, no DB) so it is exhaustively unit-testable; SearchService turns the parsed
// AST into a parameterized tsquery/predicate tree (never string-concatenated SQL).
//
// Grammar (A2):
// - Whitespace splits tokens, but a double-quoted run is ONE token ("a b" is a
// phrase). An unbalanced quote is dropped.
// - A token is an OPERATOR token only when it STARTS with `+` or `-`, the
// remainder is non-empty, and the remainder is not itself only operators.
// `+`/`-` inside a token (`WB-MGE-30D86B`, `10.0.12.5`, `a:b`) is a LITERAL —
// the token is a single term. A bare `-`/`+` is dropped.
// - `"phrase"` → phrase term; `+"phrase"`/`-"phrase"` apply the operator to it.
// - Positive terms (bare + phrase, no operator) form the OR recall set.
// `+term` is a REQUIRED predicate, `-term` an EXCLUDED predicate (A2): both
// are applied in SQL WHERE against the whole candidate set, not folded into
// the positive tsquery.
// - Only-negation (no positive term) short-circuits to an empty result with
// reason `only-negation` (never runs a costly NOT-scan).
export type SearchMatchMode = 'auto' | 'word' | 'prefix' | 'substring';
export type SearchBooleanMode = 'or' | 'and';
// Hard cap on the total number of parsed terms (positive + required + excluded).
// SearchService folds each FTS term into a LEFT-NESTED SQL tsquery expression
// `(((t1)||(t2))||(t3))…`, embedded several times per query — so nesting depth
// grows with the term count. A pasted text block (thousands of words) would nest
// deep enough to blow Postgres' `stack depth limit` → an ERROR → HTTP 500 for the
// caller. Capping in the parser bounds that depth for EVERY consumer (web / MCP /
// share), since they all route through here. 64 comfortably covers any real query
// while keeping the SQL nesting shallow. Overflow terms (beyond the first 64, in
// stable order) are dropped rather than throwing.
export const MAX_PARSED_TERMS = 64;
// How a single term is matched against the index.
// - 'fts' : full-text lexeme, exact (no trailing prefix).
// - 'ftsPrefix' : full-text lexeme with a `:*` prefix match.
// - 'phrase' : an adjacency phrase (phraseto_tsquery).
// - 'substring' : a literal LOWER(f_unaccent(col)) LIKE '%needle%' branch
// (identifiers the tokenizer mangles: IPs, hostnames, IDs).
export type SearchTermBranch = 'fts' | 'ftsPrefix' | 'phrase' | 'substring';
export interface ParsedTerm {
// The user-visible term text, operator stripped, quotes removed. This is what
// `matchedTerms` echoes back per hit.
text: string;
branch: SearchTermBranch;
}
export interface ParsedQuery {
raw: string;
// OR-recall set (bare + phrase terms with no operator).
positive: ParsedTerm[];
// AND predicates (`+term`) — the candidate MUST match each of these.
required: ParsedTerm[];
// NOT predicates (`-term`) — the candidate must match NONE of these.
excluded: ParsedTerm[];
mode: SearchBooleanMode;
// Set only when the query yields no positive recall: 'empty' (nothing usable)
// or 'only-negation' (there were exclusions but no positive term).
reason?: 'empty' | 'only-negation';
}
interface RawToken {
op: '' | '+' | '-';
kind: 'word' | 'phrase';
text: string;
}
// tsquery metacharacters that must never reach to_tsquery from a bare term — they
// are what turned adversarial input into a 500 before (#139). Stripped for the FTS
// branch; the substring branch keeps them (they are literal there).
const TSQUERY_META = /[:&|!()*<>\\]+/g;
// A term is "identifier-like" when it carries a digit or one of . _ : / - AND is
// not purely alphabetic (letters only). Such tokens (10.31.41, esp32,
// WB-MGE-30D86B) are mangled by the FTS tokenizer, so `match: auto` routes them
// to the substring branch. A purely-alphabetic word (печат, ресторан) stays FTS.
const IDENTIFIER_SIGNAL = /[0-9._:/\\-]/;
const PURELY_ALPHA = /^\p{L}+$/u;
function isIdentifierLike(text: string): boolean {
return IDENTIFIER_SIGNAL.test(text) && !PURELY_ALPHA.test(text);
}
// Clean a bare term for the FTS branch: NFC-normalize, drop tsquery metacharacters
// and collapse whitespace. Returns '' when nothing usable remains.
export function cleanFtsLexeme(raw: string): string {
return (raw ?? '')
.normalize('NFC')
.replace(TSQUERY_META, ' ')
.replace(/\s+/g, ' ')
.trim();
}
// Split a raw query into tokens, honouring double quotes and a single leading
// +/- operator. Unbalanced quotes and bare operators are dropped.
function tokenize(raw: string): RawToken[] {
const tokens: RawToken[] = [];
const s = raw ?? '';
let i = 0;
const n = s.length;
const isSpace = (c: string) => /\s/.test(c);
while (i < n) {
// Skip leading whitespace.
while (i < n && isSpace(s[i])) i++;
if (i >= n) break;
let op: '' | '+' | '-' = '';
// A single leading +/- is a tentative operator. Only ONE leading operator is
// consumed; a second (`--x`) leaves `-x` as the remainder (a literal dash).
if (s[i] === '+' || s[i] === '-') {
op = s[i] as '+' | '-';
i++;
}
if (i < n && s[i] === '"') {
// Quoted phrase: read until the closing quote.
const close = s.indexOf('"', i + 1);
if (close === -1) {
// Unbalanced quote → drop this token and everything the open quote would
// have consumed (the rest of the string).
break;
}
const phrase = s.slice(i + 1, close);
i = close + 1;
if (phrase.trim().length > 0) {
tokens.push({ op, kind: 'phrase', text: phrase.trim() });
}
continue;
}
// Bare word: read until the next whitespace.
let j = i;
while (j < n && !isSpace(s[j])) j++;
const word = s.slice(i, j);
i = j;
// A bare operator (`-`/`+` with no remainder) or an all-operator remainder is
// dropped.
if (word.length === 0) continue;
if (op && /^[+-]+$/.test(word)) continue;
tokens.push({ op, kind: 'word', text: word });
}
return tokens;
}
// Resolve the match branch for a single term given the global match mode.
function branchForTerm(
text: string,
kind: 'word' | 'phrase',
mode: SearchMatchMode,
): SearchTermBranch {
if (kind === 'phrase') return 'phrase';
switch (mode) {
case 'word':
return 'fts';
case 'prefix':
return 'ftsPrefix';
case 'substring':
return 'substring';
case 'auto':
default:
// Identifiers the tokenizer mangles go to substring; words get a prefix
// FTS match (so `печат` still finds `печатать`, but `печат` no longer drags
// in `впечатления` because the russian stemmer anchors the stem).
return isIdentifierLike(text) ? 'substring' : 'ftsPrefix';
}
}
/**
* Parse a raw user query + flags into a structured, SQL-agnostic AST.
* Pure and total: never throws, always returns a ParsedQuery.
*/
export function parseSearchQuery(
raw: string,
opts: { match?: SearchMatchMode; mode?: SearchBooleanMode } = {},
): ParsedQuery {
const match: SearchMatchMode = opts.match ?? 'auto';
const mode: SearchBooleanMode = opts.mode ?? 'or';
const positive: ParsedTerm[] = [];
const required: ParsedTerm[] = [];
const excluded: ParsedTerm[] = [];
for (const tok of tokenize(raw)) {
// For an FTS branch, the token must survive metacharacter cleaning; for the
// substring/phrase branch the literal text is used. A term that cleans to
// nothing AND is not usable as a substring is dropped.
const branch = branchForTerm(tok.text, tok.kind, match);
let usableText: string;
if (branch === 'fts' || branch === 'ftsPrefix') {
usableText = cleanFtsLexeme(tok.text);
} else {
// phrase / substring keep the literal (trimmed) text.
usableText = tok.text.trim();
}
if (!usableText) continue;
// Stack-depth guard: stop after MAX_PARSED_TERMS surviving terms (positive +
// required + excluded, combined) so the SQL tsquery nesting stays shallow.
// The first 64 terms are kept in stable order; the rest are dropped.
if (positive.length + required.length + excluded.length >= MAX_PARSED_TERMS) {
break;
}
const term: ParsedTerm = { text: usableText, branch };
if (tok.op === '+') required.push(term);
else if (tok.op === '-') excluded.push(term);
else positive.push(term);
}
const parsed: ParsedQuery = { raw: raw ?? '', positive, required, excluded, mode };
if (positive.length === 0) {
// Required terms with no positive recall still form a valid positive set (the
// required predicates ARE the recall). Only when there is neither a positive
// nor a required term is the query empty / only-negation.
if (required.length === 0) {
parsed.reason = excluded.length > 0 ? 'only-negation' : 'empty';
}
}
return parsed;
}
/**
* Does this parsed query have any positive recall to run? False means we must
* short-circuit to an empty result (with `reason`), never a costly NOT-only scan.
*/
export function hasPositiveRecall(parsed: ParsedQuery): boolean {
return parsed.positive.length > 0 || parsed.required.length > 0;
}
@@ -1,19 +1,14 @@
import {
computeLookupScore,
escapeLikePattern,
SearchLookupTier,
} from './search.service';
import { escapeLikePattern } from './search.service';
/**
* Pure-function coverage for the #443 agent-lookup helpers:
* - escapeLikePattern: LIKE-metacharacter escaping so `%`/`_`/`\` are literals
* (the acceptance-table requirement that a query of `%` or `_` does NOT match
* everything);
* - computeLookupScore: the tiered 0..1 ranking score, where a stronger tier
* always outranks a weaker one regardless of the in-tier secondary signal.
* Pure-function coverage for `escapeLikePattern` — LIKE-metacharacter escaping so
* `%`/`_`/`\` are matched literally (the acceptance requirement that a query of
* `%` or `_` does NOT match everything, #529 acceptance #10). The substring
* branch's DB behaviour is covered by the integration spec.
*
* The DB-touching branch (substring UNION FTS, path CTE, snippet window) is
* covered by the integration spec against the real schema.
* NOTE (#529): the old tiered `computeLookupScore` was replaced by RRF rank
* fusion in the unified engine, so its unit coverage moved to the integration
* ordering tests; only the escaping helper remains a pure unit here.
*/
describe('escapeLikePattern', () => {
it('escapes the LIKE metacharacters % _ and \\', () => {
@@ -43,53 +38,3 @@ describe('escapeLikePattern', () => {
expect(escapeLikePattern(null as any)).toBe('');
});
});
describe('computeLookupScore', () => {
it('keeps every score within (0, 1]', () => {
for (const tier of [
SearchLookupTier.TITLE_EXACT,
SearchLookupTier.TITLE_SUBSTRING,
SearchLookupTier.TEXT,
]) {
for (const secondary of [0, 0.001, 1, 100, 1e6]) {
const s = computeLookupScore({ tier, secondary });
expect(s).toBeGreaterThan(0);
expect(s).toBeLessThanOrEqual(1);
}
}
});
it('a stronger tier ALWAYS outranks a weaker tier, whatever the secondary', () => {
// Weak tier with a huge secondary must still lose to a strong tier with a
// tiny secondary — tiers dominate.
const strongLowSecondary = computeLookupScore({
tier: SearchLookupTier.TITLE_EXACT,
secondary: 0,
});
const weakHighSecondary = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 1e9,
});
expect(strongLowSecondary).toBeGreaterThan(weakHighSecondary);
});
it('within a tier a larger secondary sorts higher', () => {
const lo = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 0.1,
});
const hi = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 5,
});
expect(hi).toBeGreaterThan(lo);
});
it('treats a negative/absent secondary as 0', () => {
const zero = computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: 0 });
expect(computeLookupScore({ tier: SearchLookupTier.TEXT })).toBe(zero);
expect(
computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: -5 }),
).toBe(zero);
});
});
@@ -1,74 +1,45 @@
import { SearchService } from './search.service';
/**
* Coverage for SearchService.searchPage query-mode selection (search.service.ts
* @25). searchPage chooses HOW the result set is scoped — by explicit space, by
* the authenticated user's member spaces, or by a share — and must return an
* empty set (without leaking data) for every disallowed combination.
* Unit coverage for SearchService.searchPage SCOPE-SECURITY early returns — the
* branches that must yield an empty result WITHOUT ever touching the DB, so they
* can leak nothing. The happy-path scope SQL (explicit space / member spaces /
* share id set) is covered against the real schema in the integration spec.
*
* The kysely query builder is mocked with the same chainable pattern as the
* existing search.service.spec.ts: every builder method returns the same builder
* and `.execute()` resolves the supplied rows. Each `.where(...)` call is
* recorded so we can assert exactly which scope clause was applied — that is the
* mutation-resistant signal that distinguishes one query mode from another.
*
* These specs catch cross-space / cross-workspace search leakage and
* share-scope bypass (data exposure).
* Every case here returns BEFORE the raw-SQL candidate query runs, so a bare `db`
* stub (never called) is enough — a call to it would itself be a failure signal.
*/
describe('SearchService.searchPage — query-mode selection', () => {
// Build a chainable selectFrom('pages') builder that records its calls. The
// builder is returned from `db.selectFrom` and is the single object every
// chained call mutates/returns, mirroring the existing spec's pattern.
function makeBuilder(rows: Array<{ id: string; highlight?: string }>) {
const builder: any = {};
builder.select = jest.fn(() => builder);
builder.where = jest.fn(() => builder);
builder.$if = jest.fn(() => builder);
builder.orderBy = jest.fn(() => builder);
builder.limit = jest.fn(() => builder);
builder.offset = jest.fn(() => builder);
builder.execute = jest.fn(async () => rows);
return builder;
}
describe('SearchService.searchPage — scope-security early returns', () => {
function makeService(opts?: {
rows?: Array<{ id: string; highlight?: string }>;
share?: any;
isRestricted?: boolean;
descendants?: Array<{ id: string }>;
memberSpaceIds?: string[];
}) {
const builder = makeBuilder(opts?.rows ?? []);
const db: any = {
selectFrom: jest.fn(() => builder),
};
// `getUserSpaceIdsQuery` returns a sub-query object that searchPage passes
// straight into `.where('spaceId', 'in', <subquery>)`. A sentinel is enough
// to assert the user-scoped branch was taken.
const userSpaceIdsQuery = { __userSpaceIdsQuery: true };
// A db that THROWS if touched — these branches must not reach SQL.
const db: any = new Proxy(
{},
{
get() {
throw new Error('db must not be touched on an empty-scope branch');
},
},
);
const pageRepo = {
// `.select((eb) => this.pageRepo.withSpace(eb))` — value ignored by stub.
withSpace: jest.fn(() => ({ __withSpace: true })),
getPageAndDescendantsExcludingRestricted: jest
.fn()
.mockResolvedValue(opts?.descendants ?? []),
getPageAndDescendantsExcludingRestricted: jest.fn(),
getPageAndDescendants: jest.fn(),
};
const shareRepo = {
findById: jest.fn().mockResolvedValue(opts?.share ?? null),
};
const spaceMemberRepo = {
getUserSpaceIdsQuery: jest.fn(() => userSpaceIdsQuery),
getUserSpaceIds: jest.fn().mockResolvedValue(opts?.memberSpaceIds ?? []),
};
const pagePermissionRepo = {
hasRestrictedAncestor: jest
.fn()
.mockResolvedValue(opts?.isRestricted ?? false),
// Let everything through page-level permission filtering by default.
filterAccessiblePageIds: jest
.fn()
.mockImplementation(async ({ pageIds }: { pageIds: string[] }) => pageIds),
filterAccessiblePageIds: jest.fn(),
};
const service = new SearchService(
@@ -78,145 +49,81 @@ describe('SearchService.searchPage — query-mode selection', () => {
spaceMemberRepo as any,
pagePermissionRepo as any,
);
return {
service,
db,
builder,
pageRepo,
shareRepo,
spaceMemberRepo,
pagePermissionRepo,
userSpaceIdsQuery,
};
return { service, pageRepo, shareRepo, spaceMemberRepo, pagePermissionRepo };
}
const whereCallFor = (builder: any, column: any) =>
builder.where.mock.calls.find((c: any[]) => c[0] === column);
it('returns {items:[]} for a blank query WITHOUT touching the DB', async () => {
const { service, db } = makeService();
it('returns total:0 for a blank query WITHOUT touching the DB or any repo', async () => {
const { service, shareRepo, spaceMemberRepo } = makeService();
const result = await service.searchPage(
{ query: '' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(result).toEqual({ items: [] });
// Blank query is rejected before any query builder is constructed.
expect(db.selectFrom).not.toHaveBeenCalled();
expect(result.items).toEqual([]);
expect(result.total).toBe(0);
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(spaceMemberRepo.getUserSpaceIds).not.toHaveBeenCalled();
});
it('scopes to the explicit spaceId branch', async () => {
const { service, builder, db, spaceMemberRepo, shareRepo } = makeService({
rows: [{ id: 'p-1' }],
});
it('only-negation short-circuits with reason "only-negation", never scanning', async () => {
const { service, spaceMemberRepo } = makeService();
const result = await service.searchPage(
{ query: 'plan', spaceId: 'space-42' } as any,
{ query: '-архив' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(db.selectFrom).toHaveBeenCalledWith('pages');
// The explicit-space branch adds exactly `.where('spaceId', '=', 'space-42')`.
expect(whereCallFor(builder, 'spaceId')).toEqual([
'spaceId',
'=',
'space-42',
]);
// It must NOT fall through to the user-member-spaces or share branch.
expect(spaceMemberRepo.getUserSpaceIdsQuery).not.toHaveBeenCalled();
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(result.items.map((i: any) => i.id)).toEqual(['p-1']);
expect(result.total).toBe(0);
expect(result.query.parsed.reason).toBe('only-negation');
// Never resolves scope (returns before) — no expensive NOT-only scan.
expect(spaceMemberRepo.getUserSpaceIds).not.toHaveBeenCalled();
});
it('scopes an authenticated user WITHOUT spaceId to their member spaces', async () => {
const { service, builder, spaceMemberRepo, userSpaceIdsQuery, shareRepo } =
makeService({ rows: [{ id: 'p-9' }] });
await service.searchPage(
{ query: 'plan' } as any,
{ userId: 'user-7', workspaceId: 'ws-1' },
);
// The user-scoped branch resolves the member-spaces sub-query for that user
// and restricts both spaceId (to that sub-query) and workspaceId.
expect(spaceMemberRepo.getUserSpaceIdsQuery).toHaveBeenCalledWith('user-7');
expect(whereCallFor(builder, 'spaceId')).toEqual([
'spaceId',
'in',
userSpaceIdsQuery,
]);
expect(whereCallFor(builder, 'workspaceId')).toEqual([
'workspaceId',
'=',
'ws-1',
]);
// Authenticated user path must not consult shares.
expect(shareRepo.findById).not.toHaveBeenCalled();
});
it('returns {items:[]} when the share belongs to a DIFFERENT workspace', async () => {
const { service, builder, shareRepo, pagePermissionRepo } = makeService({
share: {
id: 'share-1',
pageId: 'page-1',
workspaceId: 'OTHER-ws',
includeSubPages: false,
},
it('returns empty when the share belongs to a DIFFERENT workspace (no leak)', async () => {
const { service, shareRepo, pagePermissionRepo } = makeService({
share: { id: 's1', pageId: 'p1', workspaceId: 'OTHER', includeSubPages: false },
});
const result = await service.searchPage(
{ query: 'plan', shareId: 'share-1' } as any,
{ query: 'plan', shareId: 's1' } as any,
{ workspaceId: 'ws-1' },
);
expect(shareRepo.findById).toHaveBeenCalledWith('share-1');
expect(result).toEqual({ items: [] });
// Workspace mismatch short-circuits before any restricted-ancestor / id
// scoping or DB execution: no leak across workspaces.
expect(shareRepo.findById).toHaveBeenCalledWith('s1');
expect(result.items).toEqual([]);
// Workspace mismatch short-circuits before restricted-ancestor / enumeration.
expect(pagePermissionRepo.hasRestrictedAncestor).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('returns {items:[]} when the shared page has a restricted ancestor', async () => {
const { service, builder, pagePermissionRepo, pageRepo } = makeService({
share: {
id: 'share-1',
pageId: 'page-1',
workspaceId: 'ws-1',
includeSubPages: true,
},
it('returns empty when the shared page has a restricted ancestor', async () => {
const { service, pagePermissionRepo, pageRepo } = makeService({
share: { id: 's1', pageId: 'p1', workspaceId: 'ws-1', includeSubPages: true },
isRestricted: true,
});
const result = await service.searchPage(
{ query: 'plan', shareId: 'share-1' } as any,
{ query: 'plan', shareId: 's1' } as any,
{ workspaceId: 'ws-1' },
);
expect(pagePermissionRepo.hasRestrictedAncestor).toHaveBeenCalledWith(
'page-1',
);
expect(result).toEqual({ items: [] });
// Restricted ancestor must block before page enumeration and DB execution.
expect(pagePermissionRepo.hasRestrictedAncestor).toHaveBeenCalledWith('p1');
expect(result.items).toEqual([]);
expect(
pageRepo.getPageAndDescendantsExcludingRestricted,
).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('returns {items:[]} with no userId, no spaceId and no shareId', async () => {
const { service, builder, shareRepo } = makeService();
it('returns empty with no userId, no spaceId and no shareId', async () => {
const { service, shareRepo } = makeService();
const result = await service.searchPage(
{ query: 'plan' } as any,
{ workspaceId: 'ws-1' },
);
expect(result).toEqual({ items: [] });
// The catch-all else returns empty without scoping/executing or hitting shares.
expect(result.items).toEqual([]);
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('an authenticated user with NO member spaces gets an empty result', async () => {
const { service, spaceMemberRepo } = makeService({ memberSpaceIds: [] });
const result = await service.searchPage(
{ query: 'plan' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(spaceMemberRepo.getUserSpaceIds).toHaveBeenCalledWith('user-1');
expect(result.items).toEqual([]);
expect(result.total).toBe(0);
});
});
@@ -1,4 +1,4 @@
import { SearchService, buildTsQuery } from './search.service';
import { SearchService } from './search.service';
describe('SearchService', () => {
it('should be defined', () => {
@@ -99,59 +99,3 @@ describe('SearchService.searchSuggestions — onlyTemplates filter', () => {
expect(isTemplateWhereCall(pageBuilder)).toBeUndefined();
});
});
// Unit tests for `buildTsQuery` (extracted from search.service.ts). It turns a raw
// user query into a prefix tsquery string fed to `to_tsquery('english', ...)`.
//
// REAL BUG (Gitea #139, item 10): the previous inline `tsquery(query.trim() + '*')`
// let to_tsquery operator characters through, so adversarial inputs could produce a
// fragment that to_tsquery rejects -> 500. The extraction sanitizes the input
// (strip everything but letters/numbers/whitespace) so these inputs degrade to a
// safe, neutral query with NO throw, while normal queries keep working.
describe('buildTsQuery', () => {
it('builds a prefix query for a normal single word', () => {
expect(buildTsQuery('hello')).toBe('hello:*');
});
it('joins multiple words with AND and a trailing prefix match', () => {
expect(buildTsQuery('foo bar')).toBe('foo&bar:*');
});
it('preserves accented and non-Latin words', () => {
expect(buildTsQuery('héllo café')).toBe('héllo&café:*');
expect(buildTsQuery('日本語')).toBe('日本語:*');
});
it('neutralizes to_tsquery operator inputs without throwing', () => {
// Each of these previously risked an invalid to_tsquery -> 500. They must now
// produce a safe (here empty) query and never throw.
for (const input of ['&', '!', '*', '<->', '\\']) {
expect(() => buildTsQuery(input)).not.toThrow();
expect(buildTsQuery(input)).toBe('');
}
});
it('handles stopword-only input safely', () => {
// pg-tsquery still tokenizes stopwords; to_tsquery reduces them to nothing.
// The important contract is: no throw, and a deterministic string.
expect(() => buildTsQuery('the a of')).not.toThrow();
expect(buildTsQuery('the a of')).toBe('the&a&of:*');
});
it('returns empty string for empty / whitespace-only / null-ish input', () => {
expect(buildTsQuery('')).toBe('');
expect(buildTsQuery(' ')).toBe('');
expect(buildTsQuery(undefined as unknown as string)).toBe('');
});
it('handles a very long input without throwing', () => {
const long = 'a'.repeat(10000);
expect(() => buildTsQuery(long)).not.toThrow();
expect(buildTsQuery(long)).toBe(`${long}:*`);
});
it('strips punctuation embedded in otherwise valid words', () => {
expect(buildTsQuery('c++ code')).toBe('c&code:*');
expect(buildTsQuery('a-b-c')).toBe('a&b&c:*');
});
});
File diff suppressed because it is too large Load Diff
@@ -32,6 +32,7 @@ import { TemplateRepo } from '@docmost/db/repos/template/template.repo';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo';
import { AiProviderCredentialsRepo } from '@docmost/db/repos/ai-chat/ai-provider-credentials.repo';
import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo';
@@ -125,6 +126,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
AiChatRepo,
AiChatMessageRepo,
AiChatRunRepo,
AiChatRunStepRepo,
AiChatPageSnapshotRepo,
AiProviderCredentialsRepo,
AiMcpServerRepo,
@@ -161,6 +163,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
AiChatRepo,
AiChatMessageRepo,
AiChatRunRepo,
AiChatRunStepRepo,
AiChatPageSnapshotRepo,
AiProviderCredentialsRepo,
AiMcpServerRepo,
@@ -0,0 +1,242 @@
import { type Kysely, sql } from 'kysely';
/**
* #529 Phase A1 the `ru_en` text-search configuration + the config swap.
*
* WHY: the search stack was pinned to the `english` FTS config, which stems only
* Latin words. On a Russian-language wiki that is a morphology black hole:
* «ресторанов москвы» never matched a page titled «ресторан в москве». `ru_en`
* layers the russian_stem over the Cyrillic token classes and english_stem over
* the ascii ones, so BOTH languages get proper morphology from one config.
*
* CREATE TEXT SEARCH CONFIGURATION ru_en (COPY = simple);
* ALTER ... asciiword/asciihword/hword_asciipart WITH english_stem;
* ALTER ... word/hword/hword_part WITH russian_stem;
*
* `to_tsvector('ru_en', …)` with the LITERAL config name is IMMUTABLE, so it is
* valid inside a trigger, a generated column and an index expression.
*
* THE INVARIANT (acceptance #13): the config of the STORED column and the config
* of the QUERY must change together. This migration flips BOTH stored sides
* (pages.tsv via its trigger + a reindex; page_embeddings.fts, the RAG lexical
* leg, via its generated expression); the matching QUERY-side flips
* (search.service.ts and page-embedding.repo.ts `hybridSearch`) ship in the SAME
* commit. Trigram indexes are LOWER(f_unaccent(...)) and do NOT depend on the FTS
* config, so they are untouched.
*
* REINDEX / LOCK MODEL (deploy-critical). Kysely runs EACH migration in its OWN
* transaction (see sibling 20260706T120000 "Kysely runs each migration in a
* transaction"; migrate.ts / migration.service.ts set no `disableTransactions`,
* and PostgresJSDialect has transactional DDL). A single migration file therefore
* cannot commit between batches, so the issue's "procedural batch job OUTSIDE the
* transaction + dual-config read window + migration_complete gate" is not
* expressible in-migration here. That machinery bridges a reindex spread over
* MANY committed batches, during which some rows are still `english` while others
* are already `ru_en`. Our pages.tsv reindex is a SINGLE `UPDATE pages SET tsv`,
* ATOMIC within THIS migration's own transaction: at COMMIT every row is `ru_en`
* at once, so no morphology-desync window exists and no dual-config read path is
* required the query config flips to `ru_en` in the very same release. This is
* the deliberate, correct adaptation to this framework (see the PR notes).
*
* - pages.tsv: swapping the trigger is a cheap catalog change (no table lock),
* and the reindex is a single `UPDATE pages SET tsv = <ru_en expr>` a
* ROW-level-lock (RowExclusiveLock) backfill, NOT an ACCESS EXCLUSIVE rewrite
* (mirrors the existing space_id backfill in 20250725T052004). On a LARGE
* tenant this still writes every row + its WAL and leaves dead tuples, so it
* can take MINUTES and blocks the startup migrator for that time but it
* never blocks concurrent reads. It stays inline unconditionally.
*
* - page_embeddings.fts is a GENERATED STORED column; Postgres cannot change a
* generated expression without DROP+ADD, which is a full-table ACCESS
* EXCLUSIVE REWRITE of page_embeddings it blocks ALL reads AND writes on
* that table (including the RAG agent) for the rewrite's duration. That inline
* rewrite is appropriate for small/typical tenants (this fork's target) and
* is the DEFAULT.
*
* LARGE TENANTS have two documented escape hatches, either of which makes the
* migration genuinely no-op the rewrite (it is NOT a blind DROP+ADD):
* (a) Set `SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false`. The migration then
* SKIPS the embeddings rewrite entirely and logs a WARNING. The operator
* MUST perform the ru_en fts swap out-of-band; until they do, the RAG
* lexical leg stays on `english` while the query config is `ru_en` a
* documented, operator-owned desync window. (pages.tsv still swaps
* inline the gate is ONLY the embeddings rewrite.)
* (b) Perform the swap out-of-band BEFORE deploy add a plain column
* batched backfill brief-lock swap CREATE INDEX CONCURRENTLY so
* the `fts` column's generated expression already references the TARGET
* config when the migration runs. The migration detects this (it reads
* the column's actual generation expression from pg_catalog) and does a
* TRUE no-op no DROP, no ADD, no rewrite. This is real idempotency,
* not the old (false) "IF-EXISTS guards no-op" claim: `DROP COLUMN IF
* EXISTS` guards against ABSENCE, not presence, so it would have dropped
* and recreated an existing `fts` regardless. The at-target check is the
* only honest no-op path.
*
* Same documented trade-off family as the #443 trgm GIN migration (20260706T120000).
*/
// pages.tsv trigger body for a given FTS config — mirrors the latest form
// (20250729T213756): f_unaccent + a 1MB text cap on text_content, weights A/B.
function pagesTriggerSql(config: 'ru_en' | 'english') {
return sql`
CREATE OR REPLACE FUNCTION pages_tsvector_trigger() RETURNS trigger AS $$
begin
new.tsv :=
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(coalesce(new.title, ''))), 'A') ||
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(substring(coalesce(new.text_content, ''), 1, 1000000))), 'B');
return new;
end;
$$ LANGUAGE plpgsql;
`;
}
async function swapPagesConfig(db: Kysely<any>, config: 'ru_en' | 'english') {
// 1. Point the trigger at the target config (new/edited rows use it going
// forward). CREATE OR REPLACE FUNCTION takes only a brief catalog lock.
await pagesTriggerSql(config).execute(db);
// 2. Reindex existing rows: recompute tsv directly with the target config. A
// plain UPDATE — row locks, no ACCESS EXCLUSIVE. Equivalent to firing the
// trigger but cheaper (no self-update round trip).
await sql`
UPDATE pages
SET tsv =
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(coalesce(title, ''))), 'A') ||
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(substring(coalesce(text_content, ''), 1, 1000000))), 'B')
`.execute(db);
}
// The default in-migration ACCESS EXCLUSIVE rewrite of page_embeddings.fts is
// ON unless the operator explicitly opts out with the env flag. Parsed strictly
// (mirrors CLIENT_TELEMETRY_ENABLED / DEBUG_MODE in common/): only a literal
// (case-insensitive) 'false' disables it; anything else — unset included —
// keeps the default true.
function inlineEmbeddingsRewriteEnabled(): boolean {
return (
(process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE ?? 'true').toLowerCase() !==
'false'
);
}
// Read page_embeddings.fts's ACTUAL generated-column expression from pg_catalog
// (the generation expression is stored as a column default marked generated).
// Returns '' when the column is absent.
async function embeddingsFtsExpr(db: Kysely<any>): Promise<string> {
const r = await sql<{ def: string }>`
SELECT pg_get_expr(d.adbin, d.adrelid) AS def
FROM pg_attrdef d
JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attname = 'fts'
AND a.attrelid = 'page_embeddings'::regclass
AND NOT a.attisdropped
`.execute(db);
return r.rows[0]?.def ?? '';
}
async function swapEmbeddingsFtsConfig(
db: Kysely<any>,
config: 'ru_en' | 'english',
) {
// 1. TRUE no-op path (real out-of-band escape hatch): if the column's current
// generation expression already references the TARGET config, there is
// nothing to do. An operator who pre-swapped the column out-of-band lands
// here and the migration does NOT rewrite the table. ('ru_en' and 'english'
// are disjoint tokens, neither a substring of the other or of the rest of
// the expression, so a plain contains-check is unambiguous.)
const currentExpr = await embeddingsFtsExpr(db);
if (currentExpr.includes(config)) return;
// 2. Env-gated opt-out: large tenants set SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE
// =false to skip the ACCESS EXCLUSIVE rewrite in-migration and own the swap
// out-of-band. Warn loudly so the desync window is not silent.
if (!inlineEmbeddingsRewriteEnabled()) {
// eslint-disable-next-line no-console
console.warn(
`[migration 20260707T130000] SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false: ` +
`SKIPPING the page_embeddings.fts rewrite to '${config}'. The operator MUST ` +
`perform this fts swap out-of-band. Until then the RAG lexical leg stays on ` +
`its current config while the query config is '${config}' (documented, ` +
`operator-owned desync window).`,
);
return;
}
// 3. Default inline path: the generated `fts` expression can only change via
// DROP+ADD (a full-table ACCESS EXCLUSIVE rewrite; see the lock note in the
// header). The GIN index depends on the column, so it is dropped with it and
// recreated.
await sql`DROP INDEX IF EXISTS idx_page_embeddings_fts`.execute(db);
await sql`ALTER TABLE page_embeddings DROP COLUMN IF EXISTS fts`.execute(db);
await sql`
ALTER TABLE page_embeddings
ADD COLUMN fts tsvector
GENERATED ALWAYS AS (to_tsvector('${sql.raw(config)}', f_unaccent(content))) STORED
`.execute(db);
await sql`
CREATE INDEX IF NOT EXISTS idx_page_embeddings_fts
ON page_embeddings USING gin(fts)
`.execute(db);
}
async function ruEnConfigExists(db: Kysely<any>): Promise<boolean> {
const r = await sql<{ n: number }>`
SELECT count(*)::int AS n FROM pg_ts_config WHERE cfgname = 'ru_en'
`.execute(db);
return (r.rows[0]?.n ?? 0) > 0;
}
async function ensureRuEnConfig(db: Kysely<any>): Promise<void> {
// Idempotent by EXISTENCE, not by drop-recreate. The old `DROP ... IF EXISTS;
// CREATE` was safe only on a first run: on a re-run the page_embeddings.fts
// generated column already has a hard dependency on ru_en, so dropping the
// config would fail. Create only when it is genuinely missing.
if (await ruEnConfigExists(db)) return;
await sql`CREATE TEXT SEARCH CONFIGURATION ru_en (COPY = simple)`.execute(db);
// Latin token classes → english_stem.
await sql`
ALTER TEXT SEARCH CONFIGURATION ru_en
ALTER MAPPING FOR asciiword, asciihword, hword_asciipart
WITH english_stem
`.execute(db);
// Cyrillic / non-ascii token classes → russian_stem.
await sql`
ALTER TEXT SEARCH CONFIGURATION ru_en
ALTER MAPPING FOR word, hword, hword_part
WITH russian_stem
`.execute(db);
}
export async function up(db: Kysely<any>): Promise<void> {
await ensureRuEnConfig(db);
// Flip both stored sides to ru_en (query-side flips in the same commit).
// swapEmbeddingsFtsConfig no-ops when fts already references ru_en, so a
// re-run of up() is idempotent and does NOT re-rewrite the embeddings table.
await swapPagesConfig(db, 'ru_en');
await swapEmbeddingsFtsConfig(db, 'ru_en');
}
export async function down(db: Kysely<any>): Promise<void> {
// Reverse ORDER matters: the trigger and the generated column reference the
// `ru_en` config by name, so they must be moved back to `english` BEFORE the
// config can be dropped (a generated column that still depends on `ru_en` would
// block the DROP with a dependency error).
await swapEmbeddingsFtsConfig(db, 'english');
await swapPagesConfig(db, 'english');
// Drop the config ONLY if nothing still references it. When the embeddings
// rewrite was gated off (SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false), the fts
// column can still reference ru_en — dropping the config would then fail with a
// dependency error. Skip + warn so down() stays non-fatal; the operator drops
// ru_en after completing the out-of-band english swap.
if ((await embeddingsFtsExpr(db)).includes('ru_en')) {
// eslint-disable-next-line no-console
console.warn(
`[migration 20260707T130000] down(): page_embeddings.fts still references ` +
`ru_en (inline rewrite was gated off) — leaving the ru_en text-search ` +
`configuration in place. Drop it out-of-band once fts is back on 'english'.`,
);
return;
}
await sql`DROP TEXT SEARCH CONFIGURATION IF EXISTS ru_en`.execute(db);
}
@@ -0,0 +1,70 @@
import { type Kysely, sql } from 'kysely';
/**
* `ai_chat_run_steps` append-only per-step persistence for an assistant turn
* (#492 wave C). Each finished agent step's UI `parts` (its text part + a part
* per tool call, WITH the tool output) is INSERTed as its own lightweight row the
* moment the step ends, instead of REWRITING the whole assistant row's growing
* `metadata.parts` jsonb on every `onStepFinish`.
*
* WHY a separate table + INSERT (not a jsonb `||` append on the message row): a
* Postgres jsonb UPDATE rewrites the ENTIRE TOASTed row version under MVCC, so
* re-persisting a growing `metadata.parts` on every step is O(n²) write volume
* (a 50-step run with ~100 KB tool outputs wrote hundreds of MB of WAL / dead
* tuples per turn, hammering autovacuum). `||` would only shave the network
* payload the WAL/TOAST rewrite harm remains. An INSERT into a per-step table
* writes ONLY that step's bytes, so the per-turn write volume is O(Σ steps).
*
* The full `metadata.parts` on the message row is assembled ONCE at finalize (the
* terminal completed/error/aborted write). Mid-run, a resuming client's seed is
* reconstructed by concatenating these step rows in `step_index` order which
* reproduces exactly what the old per-step full-row rewrite persisted. Records
* written the OLD way (full `metadata.parts` on the row, no step rows) still
* reconstruct from the row unchanged; the two eras are distinguished by whether
* the row already carries non-empty `metadata.parts` (see reconstructRunParts /
* assembleStepParts in ai-chat.service.ts).
*
* ON DELETE CASCADE on `message_id`: the step rows are a derived projection of the
* assistant message; they must vanish with it (or with its workspace).
*/
export async function up(db: Kysely<any>): Promise<void> {
await db.schema
.createTable('ai_chat_run_steps')
.ifNotExists()
.addColumn('id', 'uuid', (col) =>
col.primaryKey().defaultTo(sql`gen_uuid_v7()`),
)
// The assistant message row this step belongs to (the #183 projection). The
// step rows are a derived, per-step slice of that message, so they cascade.
.addColumn('message_id', 'uuid', (col) =>
col.references('ai_chat_messages.id').onDelete('cascade').notNull(),
)
.addColumn('workspace_id', 'uuid', (col) =>
col.references('workspaces.id').onDelete('cascade').notNull(),
)
// 0-based index of the finished step within the turn. Ordering key for
// reconstruction; unique per message (idempotent step re-persist).
.addColumn('step_index', 'integer', (col) => col.notNull())
// The step's UI parts (text part + a `tool-*` part per call, WITH output).
// Concatenated in step order to rebuild the turn's `metadata.parts`.
.addColumn('parts', 'jsonb', (col) => col.notNull())
.addColumn('created_at', 'timestamptz', (col) =>
col.notNull().defaultTo(sql`now()`),
)
.execute();
// Idempotent per-step persist: a retried INSERT of the same (message, step)
// is a no-op (the service uses ON CONFLICT DO NOTHING). This also serves the
// reconstruction read (WHERE message_id ORDER BY step_index).
await db.schema
.createIndex('ai_chat_run_steps_message_step_uidx')
.ifNotExists()
.on('ai_chat_run_steps')
.columns(['message_id', 'step_index'])
.unique()
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema.dropTable('ai_chat_run_steps').ifExists().execute();
}
@@ -0,0 +1,95 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import { AiChatRunStep } from '@docmost/db/types/entity.types';
/**
* Append-only per-step persistence for an assistant turn (#492). Each finished
* agent step's UI `parts` (its text part + a `tool-*` part per call, WITH the
* tool output) is INSERTed as its own lightweight row the moment the step ends
* instead of REWRITING the assistant row's growing `metadata.parts` jsonb on every
* `onStepFinish` (a Postgres jsonb UPDATE rewrites the whole TOASTed row version
* under MVCC, so that was O(n²) WAL/dead-tuple churn per turn).
*
* The full `metadata.parts` on the message row is assembled ONCE at finalize;
* mid-run, a resuming client's seed is rebuilt from these rows in `stepIndex`
* order (see `assembleStepParts` / the reconstruct seam in ai-chat.service.ts).
* Every method is workspace-scoped as defense-in-depth.
*/
@Injectable()
export class AiChatRunStepRepo {
constructor(@InjectKysely() private readonly db: KyselyDB) {}
/**
* Append one finished step's parts. Idempotent: a retried persist of the SAME
* (message, stepIndex) is a no-op via ON CONFLICT DO NOTHING the per-step
* writes are fired fire-and-forget + serialized, and a duplicate must never
* throw into the stream or double the parts. Returns whether a NEW row landed
* (false = the step was already persisted).
*/
async insertStep(
messageId: string,
workspaceId: string,
stepIndex: number,
parts: unknown,
trx?: KyselyTransaction,
): Promise<boolean> {
const db = dbOrTx(this.db, trx);
const inserted = await db
.insertInto('aiChatRunSteps')
.values({
messageId,
workspaceId,
stepIndex,
// jsonb column: cast through never (same pattern as the message repo).
parts: parts as never,
})
.onConflict((oc) => oc.columns(['messageId', 'stepIndex']).doNothing())
.returning('id')
.executeTakeFirst();
return inserted !== undefined;
}
/** All persisted steps for ONE assistant message, in step order. */
async findByMessage(
messageId: string,
workspaceId: string,
): Promise<AiChatRunStep[]> {
return this.db
.selectFrom('aiChatRunSteps')
.selectAll('aiChatRunSteps')
.where('messageId', '=', messageId)
.where('workspaceId', '=', workspaceId)
.orderBy('stepIndex', 'asc')
.execute();
}
/**
* All persisted steps for a SET of assistant messages, grouped by messageId
* (each group in step order). One query for the batch the hydration seam
* (getMessages / delta / export) calls this only for the rows that actually
* need reconstruction (an active new-style row whose `metadata.parts` is still
* empty), which is usually none, so this is skipped on the common path.
*/
async findByMessageIds(
messageIds: string[],
workspaceId: string,
): Promise<Map<string, AiChatRunStep[]>> {
const byMessage = new Map<string, AiChatRunStep[]>();
if (messageIds.length === 0) return byMessage;
const rows = await this.db
.selectFrom('aiChatRunSteps')
.selectAll('aiChatRunSteps')
.where('messageId', 'in', messageIds)
.where('workspaceId', '=', workspaceId)
.orderBy('stepIndex', 'asc')
.execute();
for (const row of rows) {
const list = byMessage.get(row.messageId);
if (list) list.push(row);
else byMessage.set(row.messageId, [row]);
}
return byMessage;
}
}
@@ -200,8 +200,11 @@ export class PageEmbeddingRepo {
*
* The `model_dimensions = $dim` filter applies ONLY on the semantic side
* (cosine compares same-dimension vectors; pgvector errors otherwise). The
* lexical side (`fts`) is dimension-independent. If `websearch_to_tsquery`
* yields an EMPTY query (e.g. the text is all stopwords) the `@@` matches
* lexical side (`fts`) is dimension-independent. Its query config is `ru_en`,
* matched IN LOCKSTEP with the `page_embeddings.fts` generated column's config
* (#529 acceptance #13): a mismatch silently breaks Cyrillic RAG retrieval. If
* `websearch_to_tsquery` yields an EMPTY query (e.g. the text is all stopwords)
* the `@@` matches
* nothing and the lexical CTE is empty, so results degrade to pure-semantic
* which is correct behaviour, not an error.
*
@@ -249,7 +252,7 @@ export class PageEmbeddingRepo {
row_number() OVER (ORDER BY ts_rank(pe.fts, q.query) DESC) AS rank_ix
FROM page_embeddings pe
JOIN pages p ON p.id = pe.page_id,
websearch_to_tsquery('english', f_unaccent(${queryText})) AS q(query)
websearch_to_tsquery('ru_en', f_unaccent(${queryText})) AS q(query)
WHERE pe.workspace_id = ${workspaceId}
AND pe.space_id IN (${spaceList})
AND p.deleted_at IS NULL
@@ -157,6 +157,44 @@ export class PageHistoryRepo {
return { ...result, items: result.items.map(attachPageHistoryAgent) };
}
/**
* #395 cheap projection of a page's FULL history timeline for the work-time
* estimate: only the columns the sessionizer needs, no heavy `content`, sorted
* oldestnewest. The secondary `id` tie-break keeps rows sharing a `createdAt`
* (e.g. a synchronous pre-agent boundary row + the immediate agent snapshot)
* in a deterministic order.
*/
async findTimelineByPageId(
pageId: string,
trx?: KyselyTransaction,
): Promise<
Array<
Pick<
PageHistory,
| 'createdAt'
| 'lastUpdatedById'
| 'lastUpdatedSource'
| 'lastUpdatedAiChatId'
| 'kind'
>
>
> {
const db = dbOrTx(this.db, trx);
return db
.selectFrom('pageHistory')
.select([
'createdAt',
'lastUpdatedById',
'lastUpdatedSource',
'lastUpdatedAiChatId',
'kind',
])
.where('pageId', '=', pageId)
.orderBy('createdAt', 'asc')
.orderBy('id', 'asc')
.execute();
}
async findPageLastHistory(
pageId: string,
opts?: {
+17
View File
@@ -692,6 +692,22 @@ export interface AiChatRuns {
updatedAt: Generated<Timestamp>;
}
// Append-only per-step persistence for an assistant turn (#492). Mirrors
// migration 20260708T120000-ai-chat-run-steps.ts. Each finished agent step's UI
// `parts` are INSERTed as their own row (instead of rewriting the message row's
// growing `metadata.parts` jsonb every step — an O(n²) WAL/TOAST churn). The full
// `metadata.parts` is assembled once at finalize; mid-run a resuming client's seed
// is rebuilt by concatenating these rows in `stepIndex` order. Cascades with the
// assistant message row it projects.
export interface AiChatRunSteps {
id: Generated<string>;
messageId: string;
workspaceId: string;
stepIndex: number;
parts: Json;
createdAt: Generated<Timestamp>;
}
// Per-(chat,page) snapshot of the open page's Markdown at the END of the agent's
// previous turn (#274). Mirrors migration 20260702T120000-ai-chat-page-snapshot.ts.
// The next turn diffs the CURRENT Markdown against `contentMd` to surface edits a
@@ -729,6 +745,7 @@ export interface DB {
aiChats: AiChats;
aiChatMessages: AiChatMessages;
aiChatRuns: AiChatRuns;
aiChatRunSteps: AiChatRunSteps;
aiChatPageSnapshots: AiChatPageSnapshots;
apiKeys: ApiKeys;
attachments: Attachments;
@@ -4,6 +4,7 @@ import {
AiChats,
AiChatMessages,
AiChatRuns,
AiChatRunSteps,
AiChatPageSnapshots,
Attachments,
Comments,
@@ -64,6 +65,12 @@ export type InsertableAiChatMessage = Omit<Insertable<AiChatMessages>, 'tsv'>;
export type AiChatRun = Selectable<AiChatRuns>;
export type InsertableAiChatRun = Insertable<AiChatRuns>;
// AI Chat Run Step (#492): append-only per-step parts persistence. Each finished
// agent step's UI parts are stored as their own row; the full turn's parts are
// assembled from these (in stepIndex order) for a mid-run resume seed.
export type AiChatRunStep = Selectable<AiChatRunSteps>;
export type InsertableAiChatRunStep = Insertable<AiChatRunSteps>;
// AI Chat Page Snapshot (#274): per-(chat,page) Markdown snapshot taken at the
// end of the agent's previous turn, diffed against the current page next turn to
// detect human edits made between turns.
@@ -0,0 +1,70 @@
import { INTERNAL_LINK_REGEX } from './utils';
import { isInternalPagePath } from '@docmost/prosemirror-markdown';
/**
* Cross-package DRIFT GUARD for the internal-link subset invariant (#522).
*
* The client-side `isInternalPagePath`
* (`packages/prosemirror-markdown/src/lib/internal-links.ts`) promotes a markdown
* link to `internal: true` on import. Every link it marks internal MUST be one
* the server would backlink and export-rewrite i.e. the client matcher MUST be
* a STRICT SUBSET of the server's canonical `INTERNAL_LINK_REGEX`
* (`./utils.ts`). If the client ever accepts a path the server rejects, that link
* is stored internal but silently dropped from the backlink graph and broken on
* export the exact bug #522 fixed.
*
* This spec is the load-bearing guard: it imports the LIVE server regex AND the
* LIVE `isInternalPagePath` (no hand-copied regex on either side). A narrowing of
* EITHER most dangerously the server regex reddens here. The in-package
* accept/reject test documents the client's behaviour but cannot see the server
* regex; this top-layer spec is what makes the subset relation mechanical
* (AGENTS.md rule #7: a CI test that fails on drift of the source of truth).
*/
describe('internal-link subset parity (client isInternalPagePath ⊆ server INTERNAL_LINK_REGEX)', () => {
// Every path the CLIENT accepts. Kept deliberately broad across the risky
// dimensions — the slug charset (digits, hyphens, mixed case, leading/trailing
// hyphen), the space charset, and the optional trailing slash — so a narrowing
// of the server regex on any of them reddens the subset assertion below.
const CLIENT_ACCEPTS = [
'/s/eng/p/abc123',
'/s/eng/p/abc123/', // trailing slash
'/s/My-Space/p/A-b-C-9', // hyphens + mixed case in both segments
'/s/x/p/z', // shortest
'/s/eng/p/my-page-abc123', // slug with hyphens (extractPageSlugId shape)
'/s/eng/p/-lead', // leading hyphen in slug
'/s/eng/p/trail-', // trailing hyphen in slug
'/s/eng/p/0123456789', // all-digit slug
'/s/a.b/p/abc', // dot in the SPACE segment (space charset is [^/]+)
'/s/space with space/p/abc', // space char in the SPACE segment
];
it('every corpus path is actually client-accepted (guards the corpus itself)', () => {
// If a path here stopped being client-accepted the subset test would pass
// vacuously; assert acceptance up front so the corpus stays meaningful. The
// filter-to-empty form names the offending paths on failure.
const notAccepted = CLIENT_ACCEPTS.filter((h) => !isInternalPagePath(h));
expect(notAccepted).toEqual([]);
});
it('every client-accepted path also matches the LIVE server regex (subset)', () => {
// The mechanical drift guard: narrow the server INTERNAL_LINK_REGEX and at
// least one hyphen/charset/structure case appears here.
const notInServer = CLIENT_ACCEPTS.filter((h) => !INTERNAL_LINK_REGEX.test(h));
expect(notInServer).toEqual([]);
});
it('the client rejects forbidden-slug-char paths the server also rejects', () => {
// Documents the subset BOUNDARY: correct shape, forbidden slug char. The
// client and the LIVE server regex must agree on rejection.
const forbidden = [
'/s/eng/p/abc.def',
'/s/eng/p/abc_def',
'/s/eng/p/abc%20',
'/s/eng/p/abc~x',
];
const clientAccepts = forbidden.filter((h) => isInternalPagePath(h));
const serverAccepts = forbidden.filter((h) => INTERNAL_LINK_REGEX.test(h));
expect(clientAccepts).toEqual([]);
expect(serverAccepts).toEqual([]);
});
});
@@ -85,6 +85,14 @@ export class ImportController {
throw new BadRequestException('spaceId is required');
}
// #502: optional multipart field. Only the MCP agent `createPage` path sends
// `disableMarkdownExtensions=true` (its body is agent-authored plain prose /
// config, so a `$…$` span must stay literal and a bare `www.host` must not
// autolink). A HUMAN file upload omits the field, so it stays false and math
// + autolink remain ON for human imports. Settable ONLY via this API param.
const disableMarkdownExtensions =
file.fields?.disableMarkdownExtensions?.value === 'true';
const ability = await this.spaceAbility.createForUser(user, spaceId);
if (ability.cannot(SpaceCaslAction.Edit, SpaceCaslSubject.Page)) {
throw new ForbiddenException();
@@ -95,6 +103,7 @@ export class ImportController {
user.id,
spaceId,
workspace.id,
disableMarkdownExtensions,
);
const ext = path.extname(file.filename).toLowerCase();
@@ -0,0 +1,67 @@
// Importing ImportService transitively loads import-formatter.ts, which imports
// the ESM-only @sindresorhus/slugify (not in jest's transform allowlist). It is
// irrelevant to this path, so mock it to keep the module graph loadable (mirrors
// the sibling import.service specs).
jest.mock('@sindresorhus/slugify', () => ({
__esModule: true,
default: (input: string) => String(input),
}));
import { ImportService } from './import.service';
// #502 BLOCKER 1: the server markdown import path (`/pages/import`) now accepts an
// optional `disableMarkdownExtensions` flag threaded into `processMarkdown`.
// - MCP agent `createPage` sends it TRUE -> extensions OFF (a `$…$` config span
// stays literal, a schemeless `www.host` is not autolinked).
// - a HUMAN file upload omits it (default FALSE) -> extensions ON, so a real
// `$x^2$` still becomes a formula (human imports unaffected).
// `processMarkdown` only uses the imported converter (no injected deps on this
// path), so the service is constructed with null deps for this focused unit test.
function makeService(): ImportService {
return new ImportService(null as any, null as any, null as any, null as any);
}
function findAll(node: any, type: string, acc: any[] = []): any[] {
if (!node || typeof node !== 'object') return acc;
if (node.type === type) acc.push(node);
if (Array.isArray(node.content)) for (const c of node.content) findAll(c, type, acc);
return acc;
}
function hasLink(node: any): boolean {
return findAll(node, 'text').some((t: any) =>
t.marks?.some((m: any) => m.type === 'link'),
);
}
describe('ImportService.processMarkdown — #502 disableMarkdownExtensions', () => {
it('disableMarkdownExtensions=true (MCP createPage): `$…$` stays literal, no math', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('export A=$FOO and B=$BAR done', true);
expect(findAll(doc, 'mathInline')).toHaveLength(0);
});
it('disableMarkdownExtensions=true: a schemeless www is NOT autolinked', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see www.example.com here', true);
expect(hasLink(doc)).toBe(false);
});
it('disableMarkdownExtensions=true: an explicit https:// STILL links', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see https://example.com here', true);
expect(hasLink(doc)).toBe(true);
});
it('DEFAULT (human upload): a real `$x^2$` DOES become a math node', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('$x^2$');
expect(findAll(doc, 'mathInline')).toHaveLength(1);
});
it('DEFAULT (human upload): a schemeless www IS autolinked', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see www.example.com here');
expect(hasLink(doc)).toBe(true);
});
});
@@ -52,6 +52,12 @@ export class ImportService {
userId: string,
spaceId: string,
workspaceId: string,
// #502: when true, the markdown importer runs with the two layered
// extensions OFF (a `$…$` span stays literal text; a schemeless `www.host` is
// NOT autolinked). ONLY the MCP agent `createPage` path sets this; a HUMAN
// file upload never passes it, so it defaults false and math/autolink stay ON
// for human imports (their `$x^2$` still becomes a formula).
disableMarkdownExtensions = false,
) {
const file = await filePromise;
const fileBuffer = await file.toBuffer();
@@ -66,7 +72,10 @@ export class ImportService {
try {
if (fileExtension.endsWith('.md')) {
prosemirrorState = await this.processMarkdown(fileContent);
prosemirrorState = await this.processMarkdown(
fileContent,
disableMarkdownExtensions,
);
} else if (fileExtension.endsWith('.html')) {
prosemirrorState = await this.processHTML(fileContent);
}
@@ -138,7 +147,12 @@ export class ImportService {
return createdPage;
}
async processMarkdown(markdownInput: string): Promise<any> {
async processMarkdown(
markdownInput: string,
// #502: forwarded to the importer. DEFAULT false keeps math + fuzzy autolink
// ON (human uploads unaffected); the MCP agent `createPage` path passes true.
disableMarkdownExtensions = false,
): Promise<any> {
// Canonical markdown -> ProseMirror JSON directly via
// `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate and no
// second editor-ext markdown layer. Foreign markdown surfaces the strict
@@ -147,7 +161,12 @@ export class ImportService {
// The HTML-cleanup pass (`normalizeImportHtml`) is intentionally skipped here:
// it targets foreign *HTML* (Notion/XWiki), which only ever arrives on the
// `.html` path (`processHTML`), never as canonical markdown.
return markdownToProseMirror(normalizeForeignMarkdown(markdownInput));
return markdownToProseMirror(
normalizeForeignMarkdown(markdownInput),
disableMarkdownExtensions
? { parseMath: false, fuzzyLinkify: false }
: undefined,
);
}
async processHTML(htmlInput: string): Promise<any> {
@@ -4,6 +4,7 @@ import { join } from 'path';
import * as fs from 'node:fs';
import fastifyStatic from '@fastify/static';
import { EnvironmentService } from '../environment/environment.service';
import { resolveClientDistPath } from '../../common/helpers/client-version';
/**
* Resolve the response headers for a statically served client asset.
@@ -56,14 +57,7 @@ export class StaticModule implements OnModuleInit {
const httpAdapter = this.httpAdapterHost.httpAdapter;
const app = httpAdapter.getInstance();
const clientDistPath = join(
__dirname,
'..',
'..',
'..',
'..',
'client/dist',
);
const clientDistPath = resolveClientDistPath();
const indexFilePath = join(clientDistPath, 'index.html');
+37 -2
View File
@@ -9,10 +9,14 @@ import {
import { Server, Socket } from 'socket.io';
import { TokenService } from '../core/auth/services/token.service';
import { JwtPayload, JwtType } from '../core/auth/dto/jwt-payload';
import { OnModuleDestroy } from '@nestjs/common';
import { Logger, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
import { SpaceMemberRepo } from '@docmost/db/repos/space/space-member.repo';
import { WsService } from './ws.service';
import { getSpaceRoomName, getUserRoomName } from './ws.utils';
import {
readClientBuildVersion,
resolveClientDistPath,
} from '../common/helpers/client-version';
import * as cookie from 'cookie';
@WebSocketGateway({
@@ -20,17 +24,40 @@ import * as cookie from 'cookie';
transports: ['websocket'],
})
export class WsGateway
implements OnGatewayConnection, OnGatewayInit, OnModuleDestroy
implements
OnGatewayConnection,
OnGatewayInit,
OnModuleInit,
OnModuleDestroy
{
@WebSocketServer()
server: Server;
private readonly logger = new Logger(WsGateway.name);
// The build version of the client bundle shipped in this image, read once at
// startup from client/dist/version.json (single source of truth, same value
// baked into the client's APP_VERSION). Empty string => version.json missing
// or empty => the proactive version-coherence reload feature stays inert.
private appVersion = '';
constructor(
private tokenService: TokenService,
private spaceMemberRepo: SpaceMemberRepo,
private wsService: WsService,
) {}
onModuleInit(): void {
this.appVersion = readClientBuildVersion(resolveClientDistPath());
if (this.appVersion) {
this.logger.log(`app-version reload: ACTIVE (v=${this.appVersion})`);
} else {
this.logger.log(
'app-version reload: DISABLED (version.json missing/empty)',
);
}
}
afterInit(server: Server): void {
this.wsService.setServer(server);
}
@@ -55,6 +82,14 @@ export class WsGateway
const spaceRooms = userSpaceIds.map((id) => getSpaceRoomName(id));
client.join([userRoom, workspaceRoom, ...spaceRooms]);
// Announce this container's client build version to the freshly
// authenticated socket. On a redeploy the client reconnects to the new
// container and receives the new version here, letting it guard-reload
// before it hits a stale lazy chunk. Per-connect only (no broadcast):
// natural reconnect covers both single-container and cluster without a
// thundering-herd fleet reload.
client.emit('app-version', { version: this.appVersion });
} catch (err) {
client.emit('Unauthorized');
client.disconnect();
@@ -0,0 +1,412 @@
import * as http from 'node:http';
import { Kysely } from 'kysely';
import { tool } from 'ai';
import { z } from 'zod';
import { MockLanguageModelV3, convertArrayToReadableStream } from 'ai/test';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import {
AiChatService,
assembleStepParts,
assistantParts,
rowHasInlineParts,
stepMarkerMetadata,
} from 'src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #492 append-persist the REAL onStep WRITE path (F2) and the model-REPLAY
* hydration path (F1), driven through `AiChatService.stream` against a LIVE
* Postgres with a REAL `AiChatRunStepRepo` INJECTED. The existing append-persist
* int-specs hand-roll the insert+marker cycle via the repos directly and build
* the service with `aiChatRunStepRepo: undefined` (only the legacy-fallback branch
* is covered), so an off-by-one on `stepsPersisted-1`, a wrong `capturedSteps`
* slice, or a broken marker payload would pass all of them. These tests exercise
* the actual `updateStreaming` append-persist branch end to end.
*
* The seam is the injected `model` (a seeded `MockLanguageModelV3` from `ai/test`)
* plus a REAL Node `ServerResponse` as the hijacked socket mirrors
* ai-chat-stream.int-spec.ts.
*/
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
async function waitFor(
cond: () => Promise<boolean> | boolean,
{ timeoutMs = 15_000, stepMs = 25 } = {},
): Promise<void> {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
if (await cond()) return;
await sleep(stepMs);
}
throw new Error('waitFor: condition not met within timeout');
}
// A real Node ServerResponse wired to a live socket (identical helper to the
// stream int-spec) so the SDK's pipe/heartbeat writes behave as in prod.
function makeRealResponse(): Promise<{
res: http.ServerResponse;
cleanup: () => Promise<void>;
}> {
return new Promise((resolve) => {
const server = http.createServer((_req, res) => {
resolve({
res,
cleanup: () =>
new Promise<void>((done) => {
try {
if (!res.writableEnded) res.end();
} catch {
/* socket already gone */
}
server.close(() => done());
}),
});
});
server.listen(0, () => {
const port = (server.address() as any).port;
const creq = http.request({ port, method: 'GET' }, (cres) => {
cres.resume();
});
creq.on('error', () => undefined);
creq.end();
});
});
}
// Stream parts for a normal, successful single-step turn.
function successStream() {
return convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: 't1' },
{ type: 'text-delta', id: 't1', delta: 'Hello' },
{ type: 'text-delta', id: 't1', delta: ' there' },
{ type: 'text-end', id: 't1' },
{
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 10, outputTokens: 5, totalTokens: 15 },
},
] as any);
}
// A THREE-step turn: steps 0 and 1 each emit text + an `echo` tool call (the SDK
// runs the tool and continues); step 2 answers and stops. Three steps is
// deliberate: the LAST finished step's append-persist write races the terminal
// finalize (which writes the full inline parts anyway, so a lost last-step row is
// by design), but the NON-final steps 0 and 1 always drain to the steps table
// before finalize — so those are what the test asserts on deterministically.
function threeStepModel(): MockLanguageModelV3 {
let step = 0;
const toolStep = (i: number) => ({
stream: convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: `s${i}` },
{ type: 'text-delta', id: `s${i}`, delta: `step ${i} ` },
{ type: 'text-end', id: `s${i}` },
{
type: 'tool-call',
toolCallId: `c${i}`,
toolName: 'echo',
input: JSON.stringify({ msg: `m${i}` }),
},
{
type: 'finish',
finishReason: 'tool-calls',
usage: { inputTokens: 5, outputTokens: 3, totalTokens: 8 },
},
] as any),
});
return new MockLanguageModelV3({
doStream: async () => {
const n = step++;
// Realistic inter-step latency. A real model spends seconds per step, so the
// fire-and-forget per-step write chain drains to the steps table BETWEEN
// steps; the mock otherwise collapses all steps into microseconds and the
// terminal finalize wins the race before any but the first step persists.
if (n > 0) await sleep(200);
if (n < 2) return toolStep(n);
return {
stream: convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: 's2' },
{ type: 'text-delta', id: 's2', delta: 'final answer' },
{ type: 'text-end', id: 's2' },
{
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 6, outputTokens: 4, totalTokens: 10 },
},
] as any),
};
},
} as any);
}
describe('#492 append-persist service paths [integration]', () => {
let db: Kysely<any>;
let aiChatRepo: AiChatRepo;
let msgRepo: AiChatMessageRepo;
let stepRepo: AiChatRunStepRepo;
let workspaceId: string;
let userId: string;
let closeCalls: number;
const mcpClients = {
toolsFor: async () => ({
tools: {},
clients: [
{
close: async () => {
closeCalls += 1;
},
},
],
outcomes: [],
instructions: [],
}),
};
// Build the service WITH a REAL AiChatRunStepRepo injected (the property under
// test) — unlike the legacy-fallback harness that passes it as undefined.
const echoTool = tool({
description: 'echo the message back',
inputSchema: z.object({ msg: z.string() }),
execute: async ({ msg }) => ({ echoed: msg }),
});
function buildService(): AiChatService {
return new AiChatService(
{ getChatModel: async () => null } as any,
aiChatRepo,
msgRepo,
{} as any, // aiChatPageSnapshotRepo
{ resolve: async () => null } as any, // aiSettings
{ forUser: async () => ({ echo: echoTool }) } as any, // tools
mcpClients as any,
{} as any, // aiAgentRoleRepo
{} as any, // pageRepo
{} as any, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatFinalStepLockdownEnabled: () => false,
} as any, // environment (deferred OFF -> all tools active every step)
undefined, // streamRegistry
undefined, // aiChatRunService
stepRepo, // #492 aiChatRunStepRepo — the append-persist backend
);
}
function userUiMessage(text: string) {
return {
id: `u-${Math.random()}`,
role: 'user',
parts: [{ type: 'text', text }],
};
}
async function runStream(opts: {
model: MockLanguageModelV3;
chatId: string;
body: any;
}): Promise<void> {
closeCalls = 0;
const service = buildService();
const { res, cleanup } = await makeRealResponse();
try {
await service.stream({
user: { id: userId, workspaceId } as any,
workspace: { id: workspaceId, name: 'WS' } as any,
sessionId: 'sess-1',
body: opts.body,
res: { raw: res } as any,
signal: new AbortController().signal,
model: opts.model as any,
role: null,
} as any);
await waitFor(async () => {
const rows = await msgRepo.findAllByChat(opts.chatId, workspaceId);
return rows.some(
(r) =>
r.role === 'assistant' &&
['completed', 'error', 'aborted'].includes(r.status as string),
);
});
await waitFor(() => closeCalls > 0, { timeoutMs: 5_000 });
} finally {
await cleanup();
}
}
beforeAll(async () => {
db = getTestDb();
aiChatRepo = new AiChatRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
stepRepo = new AiChatRunStepRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
// --- F2: the real onStep append-persist WRITE branch -----------------------
it('drives steps through the real onStep path: per-step rows + marker match a single-row flush', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const model = threeStepModel();
// Capture the mid-run step-marker UPDATEs the append-persist branch writes on
// the assistant row (a { parts: [], toolTraceVersion, stepsPersisted } patch).
const updateSpy = jest.spyOn(msgRepo, 'update');
try {
await runStream({
model,
chatId,
body: { chatId, messages: [userUiMessage('call the tool then answer')] },
});
const rows = await msgRepo.findAllByChat(chatId, workspaceId);
const assistant = rows.find((r) => r.role === 'assistant')!;
expect(assistant).toBeDefined();
expect(assistant.status).toBe('completed');
// The turn finalizes with the FULL inline parts assembled by a single-row
// flush (assistantParts over every step) — the baseline the per-step slices
// must reproduce.
expect(rowHasInlineParts(assistant)).toBe(true);
const finalParts = (assistant.metadata as { parts: any[] }).parts;
// The two NON-final finished steps each landed their own row, in stepIndex
// order. (The fire-and-forget write chain drains before the next step, so
// poll until both are on disk; the LAST step's write may lose the finalize
// race, which is by design — its parts are already in `finalParts`.)
await waitFor(async () => {
const s = await stepRepo.findByMessage(assistant.id, workspaceId);
return s.length >= 2;
});
const steps = await stepRepo.findByMessage(assistant.id, workspaceId);
expect(steps[0].stepIndex).toBe(0);
expect(steps[1].stepIndex).toBe(1);
// Each per-step row carries a NON-trivial slice: this step's text part + its
// paired tool part (guards a mutation that persists empty/whole-turn parts).
const s0 = steps[0].parts as any[];
expect(s0).toContainEqual({ type: 'text', text: 'step 0 ' });
expect(s0.some((p) => p.type === 'tool-echo')).toBe(true);
// The per-step slices are EXACTLY the corresponding prefix of the single-row
// flush: assembleStepParts([step0, step1]) === finalParts[0 .. len0+len1].
// This is what an off-by-one on `stepsPersisted-1` (a wrong `capturedSteps`
// slice) or a shifted stepIndex breaks — the prefix no longer aligns.
const prefixLen =
(steps[0].parts as any[]).length + (steps[1].parts as any[]).length;
expect(assembleStepParts([steps[0], steps[1]] as any)).toEqual(
finalParts.slice(0, prefixLen),
);
// The mid-run step markers advanced 1 -> 2 -> ... (the resume frontier), each
// a shape-stable empty-parts marker equal to a single-row flush's marker.
const markerCounts = updateSpy.mock.calls
.map((c) => (c[2] as any)?.metadata)
.filter(
(m) =>
m &&
Array.isArray(m.parts) &&
m.parts.length === 0 &&
typeof m.stepsPersisted === 'number',
)
.map((m) => m.stepsPersisted);
// Monotonic from 1, covering at least the two non-final steps.
expect(markerCounts.slice(0, 2)).toEqual([1, 2]);
expect(
updateSpy.mock.calls
.map((c) => (c[2] as any)?.metadata)
.find((m) => m && m.stepsPersisted === 2),
).toEqual(stepMarkerMetadata(2));
} finally {
updateSpy.mockRestore();
}
}, 60_000);
// --- F1: model-REPLAY hydrates a hard-crashed mid-run turn from the steps table
it('replays a hard-crashed mid-run turn WITH its partial steps hydrated from the steps table', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
// Prior turn: a genuine user question...
await createMessage(db, {
workspaceId,
chatId,
userId,
role: 'user',
content: 'What is in the design doc?',
createdAt: new Date(Date.now() - 3000),
});
// ...and an assistant row that a HARD crash (SIGKILL/OOM) left mid-run: only a
// step marker on the row (metadata.parts:[] , content:''), NO terminal
// callback ever fired, so its real parts live ONLY in ai_chat_run_steps.
const crashed = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: '',
status: 'aborted',
metadata: stepMarkerMetadata(1),
createdAt: new Date(Date.now() - 2000),
});
// The durable partial step: some reasoning text + a completed getPage tool
// call (input + output), exactly what #183 step-granular durability preserves.
await stepRepo.insertStep(
crashed.id,
workspaceId,
0,
assistantParts(
[
{
text: 'HYDRATED_PARTIAL_STEP the doc says',
toolCalls: [
{ toolCallId: 'g1', toolName: 'getPage', input: { id: 'p1' } },
],
toolResults: [
{
toolCallId: 'g1',
toolName: 'getPage',
output: { id: 'p1', body: 'PARTIAL_TOOL_OUTPUT budget section' },
},
],
} as any,
],
'',
),
);
// The NEXT turn: the model just answers. The service must REPLAY the crashed
// assistant turn with its partial parts hydrated from the steps table.
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: successStream() }),
} as any);
await runStream({
model,
chatId,
body: { chatId, messages: [userUiMessage('Continue please')] },
});
expect(model.doStreamCalls.length).toBeGreaterThan(0);
const prompt = JSON.stringify(model.doStreamCalls[0].prompt);
// The partial step's TEXT reached the model context (it would be an empty text
// part without hydration — rowToUiMessage falls back to `content:''`).
expect(prompt).toContain('HYDRATED_PARTIAL_STEP');
// The partial TOOL RESULT survived too (durable in the steps table, replayed).
expect(prompt).toContain('PARTIAL_TOOL_OUTPUT');
// The genuine prior user turn is present as well (sanity: real history replay).
expect(prompt).toContain('What is in the design doc?');
}, 60_000);
});
@@ -0,0 +1,173 @@
import { randomBytes } from 'crypto';
import { Kysely, sql } from 'kysely';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import {
assistantParts,
flushAssistant,
stepMarkerMetadata,
} from '../../src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
} from './db';
/**
* #492 append-persist WRITE-VOLUME regression on a LIVE Postgres, measured via
* the `pg_current_wal_lsn()` delta around a realistic multi-step run driven through
* the REAL repos (not a mock a mock cannot observe MVCC/TOAST rewrite volume, the
* whole point). Proves the core claim:
*
* NEW (per-step INSERT into ai_chat_run_steps + a CHEAP step-marker UPDATE on the
* message row) writes O(Σ steps) of WAL each step writes only its own bytes.
*
* OLD (the pre-#492 full-row rewrite: re-persist the GROWING metadata.parts on
* every onStepFinish) writes O(n²) step k rewrites the whole TOASTed jsonb of
* all k prior outputs.
*
* The OLD path here IS the reverted behavior, so this doubles as the mutation
* check: swapping the new path back to `flushAssistant` full-row UPDATEs reddens
* the assertion (OLD is many times larger).
*/
type Step = {
text: string;
toolCalls: Array<{ toolCallId: string; toolName: string; input: unknown }>;
toolResults: Array<{ toolCallId: string; toolName: string; output: unknown }>;
};
// ~100 KB INCOMPRESSIBLE output per step (a page read). Random base64 so TOAST
// cannot compress it away and hide the real write volume.
function makeStep(i: number, outputBytes = 100_000): Step {
const body = randomBytes(Math.ceil(outputBytes * 0.75)).toString('base64');
return {
text: `step ${i} reasoning`,
toolCalls: [
{ toolCallId: `c${i}`, toolName: 'getPage', input: { id: `p${i}` } },
],
toolResults: [
{
toolCallId: `c${i}`,
toolName: 'getPage',
output: { id: `p${i}`, title: `Page ${i}`, body },
},
],
};
}
async function walDelta(
db: Kysely<any>,
fn: () => Promise<void>,
): Promise<number> {
const before = (
await sql<{ l: string }>`select pg_current_wal_lsn() as l`.execute(db)
).rows[0].l;
await fn();
// NOTE: no pg_switch_wal() — a segment switch pads the LSN to the next 16 MB
// boundary and would swamp the delta. The raw LSN advances by the WAL bytes.
const after = (
await sql<{ l: string }>`select pg_current_wal_lsn() as l`.execute(db)
).rows[0].l;
return Number(
(
await sql<{
d: string;
}>`select pg_wal_lsn_diff(${after}::pg_lsn, ${before}::pg_lsn) as d`.execute(
db,
)
).rows[0].d,
);
}
describe('#492 append-persist write volume (pg_current_wal_lsn delta) [integration]', () => {
let db: Kysely<any>;
let stepRepo: AiChatRunStepRepo;
let msgRepo: AiChatMessageRepo;
let workspaceId: string;
let userId: string;
let chatId: string;
beforeAll(async () => {
db = getTestDb();
stepRepo = new AiChatRunStepRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
const seedRow = () =>
msgRepo.insert({
chatId,
workspaceId,
userId,
role: 'assistant',
content: '',
status: 'streaming',
metadata: stepMarkerMetadata(0) as never,
});
const STEPS = 40;
it('NEW per-step INSERT is O(Σ steps); OLD full-row rewrite is O(n²)', async () => {
const steps: Step[] = [];
for (let i = 0; i < STEPS; i++) steps.push(makeStep(i));
// NEW: per-step INSERT of THIS step's parts + a cheap marker UPDATE.
const newRow = await seedRow();
const newWal = await walDelta(db, async () => {
for (let i = 0; i < STEPS; i++) {
await stepRepo.insertStep(
newRow.id,
workspaceId,
i,
assistantParts([steps[i]], ''),
);
await msgRepo.update(
newRow.id,
workspaceId,
{ metadata: stepMarkerMetadata(i + 1) },
{ onlyIfStreaming: true },
);
}
});
// OLD (the pre-#492 revert): re-persist the GROWING metadata.parts on the
// message row on every step.
const oldRow = await seedRow();
const oldWal = await walDelta(db, async () => {
const acc: Step[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(steps[i]);
await msgRepo.update(
oldRow.id,
workspaceId,
flushAssistant(acc as never, '', 'streaming'),
{ onlyIfStreaming: true },
);
}
});
// eslint-disable-next-line no-console
console.log(
`[#492 WAL] ${STEPS} steps ×100KB: new=${(newWal / 1e6).toFixed(1)}MB ` +
`old=${(oldWal / 1e6).toFixed(1)}MB (${(oldWal / newWal).toFixed(
1,
)}x smaller)`,
);
// O(Σ steps): ~STEPS × (100KB output + marker) of WAL. 40 × ~100KB parts plus
// 40 tiny markers is a few tens of MB at most — bounded, linear in step count.
expect(newWal).toBeLessThan(30_000_000);
// O(n²): step k rewrites ~k × 100KB. Σ over 40 steps ≈ 80+ MB — far larger.
expect(oldWal).toBeGreaterThan(30_000_000);
// The load-bearing claim: the new path writes a small FRACTION of the old.
expect(newWal).toBeLessThan(oldWal * 0.35);
}, 120_000);
});
@@ -0,0 +1,169 @@
import { Kysely } from 'kysely';
import { AiChatController } from 'src/core/ai-chat/ai-chat.controller';
import {
assembleStepParts,
assistantParts,
stepMarkerMetadata,
} from 'src/core/ai-chat/ai-chat.service';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import type { User, Workspace } from '@docmost/db/types/entity.types';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #492 controller hydration (crash-before-finalize RESUME) on a LIVE Postgres.
* `AiChatController.withReconstructedParts` is wired into getMessages/delta/export/
* run, but `aiChatRunStepRepo` is OPTIONAL and every controller unit spec passes it
* as `undefined`, so the hydration branch early-returns and NEVER executes in those
* tests. This drives the real read path a mid-run streaming row (marker only,
* empty inline parts) PLUS its `ai_chat_run_steps` rows through getMessages WITH
* the repo present, exercising the `role==='assistant' && !rowHasInlineParts`
* needy predicate, the workspace-scoped batch step fetch, and the endpoint binding.
*/
describe('#492 controller hydration read path [integration]', () => {
let db: Kysely<any>;
let aiChatRepo: AiChatRepo;
let msgRepo: AiChatMessageRepo;
let stepRepo: AiChatRunStepRepo;
let workspaceId: string;
let otherWorkspaceId: string;
let userId: string;
// Build the controller WITH a real AiChatRunStepRepo injected (position 9), the
// seam the unit specs leave undefined. Only the read-path deps are real.
function buildController(): AiChatController {
return new AiChatController(
{} as any, // aiChatService
{} as any, // aiChatRunService
aiChatRepo,
msgRepo,
{} as any, // aiTranscription
{} as any, // pageRepo
undefined, // streamRegistry
undefined, // environment
stepRepo, // #492 aiChatRunStepRepo
);
}
beforeAll(async () => {
db = getTestDb();
aiChatRepo = new AiChatRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
stepRepo = new AiChatRunStepRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
otherWorkspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
it('getMessages reconstructs a mid-run row from the steps table (finished rows untouched)', async () => {
const chatId = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const user = { id: userId } as User;
const workspace = { id: workspaceId } as Workspace;
// A prior FINISHED assistant row that already carries inline parts — the needy
// predicate must SKIP it (no step fetch), returned untouched.
const finishedParts = assistantParts(
[{ text: 'done earlier', toolCalls: [], toolResults: [] } as any],
'',
);
await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: 'done earlier',
status: 'completed',
metadata: { parts: finishedParts, toolTraceVersion: 2, stepsPersisted: 1 },
createdAt: new Date(Date.now() - 3000),
});
// The mid-run row a crash-before-finalize left behind: a step marker only
// (parts:[] , content:''), status 'streaming'. Its real parts live ONLY in the
// steps table.
const midRun = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: '',
status: 'streaming',
metadata: stepMarkerMetadata(2),
createdAt: new Date(Date.now() - 1000),
});
const step0 = assistantParts(
[
{
text: 'reasoning about the page',
toolCalls: [
{ toolCallId: 'g1', toolName: 'getPage', input: { id: 'p1' } },
],
toolResults: [
{ toolCallId: 'g1', toolName: 'getPage', output: { id: 'p1', body: 'B' } },
],
} as any,
],
'',
);
const step1 = assistantParts(
[{ text: 'partial synthesis so far', toolCalls: [], toolResults: [] } as any],
'',
);
await stepRepo.insertStep(midRun.id, workspaceId, 0, step0);
await stepRepo.insertStep(midRun.id, workspaceId, 1, step1);
// Workspace-scoping guard: a step row for the SAME message id under a DIFFERENT
// workspace must NEVER leak into this workspace's reconstruction.
await stepRepo.insertStep(midRun.id, otherWorkspaceId, 99, [
{ type: 'text', text: 'FOREIGN_WORKSPACE_LEAK' },
]);
const res = await buildController().getMessages(
{ chatId } as any,
{ limit: 50 } as any,
user,
workspace,
);
const items = res.items as any[];
const finished = items.find((r) => r.status === 'completed');
const reconstructed = items.find((r) => r.id === midRun.id);
// The finished row passed through with its inline parts unchanged.
expect(finished.metadata.parts).toEqual(finishedParts);
// The mid-run row's parts were reconstructed from the two step rows, in order,
// exactly as assembleStepParts concatenates them — the client seed sees the
// persisted progress with no change to itself.
const expected = assembleStepParts([
{ stepIndex: 0, parts: step0 },
{ stepIndex: 1, parts: step1 },
] as any);
expect(reconstructed.metadata.parts).toEqual(expected);
// The foreign-workspace step row did NOT leak in.
expect(JSON.stringify(reconstructed.metadata.parts)).not.toContain(
'FOREIGN_WORKSPACE_LEAK',
);
// Sanity: reconstruction produced real content (text + the paired tool part +
// the second step's text), not an empty fallback.
expect(reconstructed.metadata.parts).toContainEqual({
type: 'text',
text: 'reasoning about the page',
});
expect(
(reconstructed.metadata.parts as any[]).some((p) => p.type === 'tool-getPage'),
).toBe(true);
}, 60_000);
});
@@ -0,0 +1,163 @@
import { randomBytes } from 'crypto';
import { Kysely } from 'kysely';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import {
assistantParts,
reconstructRunParts,
hydrateAssistantParts,
stepMarkerMetadata,
rowHasInlineParts,
} from '../../src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
} from './db';
/**
* #492 append-persist the reconstruct CONTRACT on a live Postgres. Proves that a
* turn persisted the NEW way (per-step rows in `ai_chat_run_steps`, only a step
* marker on the message row) reconstructs to the SAME UI parts as a turn persisted
* the OLD way (full `metadata.parts` inline on the row, no step rows) so the
* era-switch is invisible to attach / delta-poll / export. Real repos + real jsonb
* roundtrip, not a mock (a mock cannot prove the parts survive the jsonb column
* byte-identical).
*/
type Step = {
text: string;
toolCalls: Array<{ toolCallId: string; toolName: string; input: unknown }>;
toolResults: Array<{ toolCallId: string; toolName: string; output: unknown }>;
};
// A realistic step: some text + a getPage tool call whose ~100 KB body is
// INCOMPRESSIBLE random base64 (a 'x'.repeat filler would TOAST away and hide the
// real bytes). Under MAX_TOOL_OUTPUT_BYTES (200 KB) it is stored uncompacted.
function makeStep(i: number, outputBytes = 4_000): Step {
const body = randomBytes(Math.ceil(outputBytes * 0.75)).toString('base64');
return {
text: `step ${i} text`,
toolCalls: [
{ toolCallId: `c${i}`, toolName: 'getPage', input: { id: `p${i}` } },
],
toolResults: [
{
toolCallId: `c${i}`,
toolName: 'getPage',
output: { id: `p${i}`, title: `Page ${i}`, body },
},
],
};
}
describe('AiChatRunStepRepo + reconstruct contract [integration]', () => {
let db: Kysely<any>;
let stepRepo: AiChatRunStepRepo;
let msgRepo: AiChatMessageRepo;
let workspaceId: string;
let userId: string;
let chatId: string;
beforeAll(async () => {
db = getTestDb();
stepRepo = new AiChatRunStepRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
const seedRow = (metadata: unknown, status: string) =>
msgRepo.insert({
chatId,
workspaceId,
userId,
role: 'assistant',
content: '',
status,
metadata: metadata as never,
});
it('insertStep is idempotent per (message, stepIndex) and reads back in order', async () => {
const row = await seedRow(stepMarkerMetadata(0), 'streaming');
const parts0 = assistantParts([makeStep(0)], '');
const parts1 = assistantParts([makeStep(1)], '');
expect(await stepRepo.insertStep(row.id, workspaceId, 0, parts0)).toBe(true);
expect(await stepRepo.insertStep(row.id, workspaceId, 1, parts1)).toBe(true);
// A retried persist of the SAME step is a no-op (ON CONFLICT DO NOTHING).
expect(await stepRepo.insertStep(row.id, workspaceId, 0, parts0)).toBe(
false,
);
const steps = await stepRepo.findByMessage(row.id, workspaceId);
expect(steps.map((s) => s.stepIndex)).toEqual([0, 1]);
// Batch fetch groups by message id in step order.
const map = await stepRepo.findByMessageIds([row.id], workspaceId);
expect(map.get(row.id)!.map((s) => s.stepIndex)).toEqual([0, 1]);
});
it('a NEW-style (step-table) run reconstructs identically to an OLD-style (inline) run', async () => {
const steps = [makeStep(10), makeStep(11)];
// The inline parts the OLD full-row flush would have written.
const fullParts = assistantParts(steps, '');
// OLD-style record: full parts inline on the row, NO step rows.
const oldRow = await seedRow(
{ parts: fullParts, toolTraceVersion: 2, stepsPersisted: 2 },
'completed',
);
// NEW-style record: only a step marker on the row + per-step rows.
const newRow = await seedRow(stepMarkerMetadata(2), 'streaming');
for (let i = 0; i < steps.length; i++) {
await stepRepo.insertStep(
newRow.id,
workspaceId,
i,
assistantParts([steps[i]], ''),
);
}
// Re-read both from the DB (proves the jsonb roundtrip).
const oldFetched = await msgRepo.findById(oldRow.id, workspaceId);
const newFetched = await msgRepo.findById(newRow.id, workspaceId);
const oldSteps = await stepRepo.findByMessage(oldRow.id, workspaceId);
const newSteps = await stepRepo.findByMessage(newRow.id, workspaceId);
// The discriminator: the old row carries inline parts, the new one does not.
expect(rowHasInlineParts(oldFetched!)).toBe(true);
expect(rowHasInlineParts(newFetched!)).toBe(false);
expect(oldSteps).toHaveLength(0);
expect(newSteps).toHaveLength(2);
const oldRecon = reconstructRunParts(oldFetched!, oldSteps);
const newRecon = reconstructRunParts(newFetched!, newSteps);
// Both reconstruct to the SAME parts + step count — the era is invisible.
expect(newRecon.parts).toEqual(fullParts);
expect(oldRecon.parts).toEqual(fullParts);
expect(newRecon.parts).toEqual(oldRecon.parts);
expect(newRecon.stepsPersisted).toBe(2);
expect(oldRecon.stepsPersisted).toBe(2);
// hydrateAssistantParts fills the new row's metadata.parts to match the old
// row's inline parts — so a consumer reading `metadata.parts` off the raw row
// (the client seed/poll, export) is unchanged across the era.
const map = await stepRepo.findByMessageIds([newRow.id], workspaceId);
const [hydrated] = hydrateAssistantParts([newFetched!], map);
expect((hydrated.metadata as { parts: unknown }).parts).toEqual(fullParts);
// A row that already has inline parts passes through untouched (same ref-shape).
const [oldPassThrough] = hydrateAssistantParts([oldFetched!], map);
expect((oldPassThrough.metadata as { parts: unknown }).parts).toEqual(
fullParts,
);
});
});
@@ -0,0 +1,63 @@
import { Kysely, sql } from 'kysely';
import {
up,
down,
} from '../../src/database/migrations/20260708T120000-ai-chat-run-steps';
import { getTestDb, destroyTestDb } from './db';
/**
* #492 migration up/down roundtrip on a LIVE Postgres. global-setup already
* migrated docmost_test to latest (so the table exists at start); this drives the
* migration's own down()/up() and asserts the table presence toggles, then leaves
* it PRESENT (up) so the shared test DB is intact for any spec that runs after.
*/
async function tableExists(db: Kysely<any>): Promise<boolean> {
const row = (
await sql<{ t: string | null }>`select to_regclass('ai_chat_run_steps') as t`.execute(
db,
)
).rows[0];
return row.t !== null;
}
async function uniqueIndexExists(db: Kysely<any>): Promise<boolean> {
const row = (
await sql<{
t: string | null;
}>`select to_regclass('ai_chat_run_steps_message_step_uidx') as t`.execute(db)
).rows[0];
return row.t !== null;
}
describe('20260708 ai_chat_run_steps migration roundtrip [integration]', () => {
let db: Kysely<any>;
beforeAll(() => {
db = getTestDb();
});
afterAll(async () => {
// Belt-and-suspenders: guarantee the table is present for later specs even if
// an assertion threw mid-roundtrip.
if (!(await tableExists(db))) await up(db);
await destroyTestDb();
});
it('down() drops the table+index and up() recreates them (idempotent)', async () => {
// Starts applied (global-setup migrated to latest).
expect(await tableExists(db)).toBe(true);
expect(await uniqueIndexExists(db)).toBe(true);
await down(db);
expect(await tableExists(db)).toBe(false);
expect(await uniqueIndexExists(db)).toBe(false);
await up(db);
expect(await tableExists(db)).toBe(true);
expect(await uniqueIndexExists(db)).toBe(true);
// up() is idempotent (ifNotExists) — a second run is a harmless no-op.
await expect(up(db)).resolves.not.toThrow();
expect(await tableExists(db)).toBe(true);
});
});
@@ -0,0 +1,137 @@
import { randomUUID } from 'node:crypto';
import { Kysely } from 'kysely';
import { PageHistoryRepo } from '../../src/database/repos/page/page-history.repo';
import { PageHistoryService } from '../../src/core/page/services/page-history.service';
import { computeWorkTime } from '../../src/core/page/work-time';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
createPage,
createUser,
createChat,
} from './db';
/**
* #395 real-Postgres coverage for the work-time timeline projection and the
* service that computes the estimate. The pure sessionizer is unit-tested
* exhaustively (compute-work-time.spec.ts); this asserts the SQL projection
* (right rows, ASC, no `content`) and that the service's numbers agree with the
* pure core over the exact rows the DB returns.
*/
describe('PageHistory work-time [integration]', () => {
let db: Kysely<any>;
let repo: PageHistoryRepo;
let service: PageHistoryService;
let workspaceId: string;
let spaceId: string;
let pageId: string;
let userId: string;
let chatId: string;
const MIN = 60 * 1000;
beforeAll(async () => {
db = getTestDb();
repo = new PageHistoryRepo(db as any);
service = new PageHistoryService(repo);
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
pageId = (await createPage(db, { workspaceId, spaceId })).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
async function insertHistory(rows: Array<{
createdAt: string;
source: string | null;
chat?: string | null;
kind?: string | null;
content?: unknown;
}>) {
for (const r of rows) {
await db
.insertInto('pageHistory')
.values({
id: randomUUID(),
pageId,
spaceId,
workspaceId,
title: 'x',
content: r.content ?? { type: 'doc', content: [] },
lastUpdatedById: userId,
lastUpdatedSource: r.source,
lastUpdatedAiChatId: r.chat ?? null,
kind: r.kind ?? null,
createdAt: new Date(r.createdAt),
})
.execute();
}
}
it('findTimelineByPageId projects the cheap columns, ASC, without content', async () => {
await insertHistory([
{ createdAt: '2026-07-04T19:54:00Z', source: 'user', kind: 'manual' },
{ createdAt: '2026-07-04T03:40:00Z', source: 'user', kind: null },
{ createdAt: '2026-07-04T15:43:00Z', source: 'agent', chat: chatId, kind: 'agent' },
]);
const timeline = await repo.findTimelineByPageId(pageId);
expect(timeline).toHaveLength(3);
// Sorted oldest → newest.
const times = timeline.map((r) => new Date(r.createdAt).getTime());
expect(times).toEqual([...times].sort((a, b) => a - b));
// Projection carries exactly the sessionizer's inputs, and NO content.
for (const row of timeline) {
expect(row).toHaveProperty('createdAt');
expect(row).toHaveProperty('lastUpdatedById');
expect(row).toHaveProperty('lastUpdatedSource');
expect(row).toHaveProperty('lastUpdatedAiChatId');
expect(row).toHaveProperty('kind');
expect(row).not.toHaveProperty('content');
}
// Agent row keeps its provenance.
const agent = timeline.find((r) => r.lastUpdatedSource === 'agent');
expect(agent?.lastUpdatedAiChatId).toBe(chatId);
});
it('service estimate matches the pure core and satisfies Σ perDay == workMs', async () => {
const rows = await repo.findTimelineByPageId(pageId);
const pure = computeWorkTime(rows);
const result = await service.computeWorkTime(pageId, 'UTC');
expect(result.workMs).toBe(pure.workMs);
expect(result.agentOnlyMs).toBe(pure.agentOnlyMs);
expect(result.tz).toBe('UTC');
expect(result.config.tGap).toBe(15 * MIN);
const sumActive = result.perDay.reduce((a, d) => a + d.activeMs, 0);
expect(sumActive).toBe(result.workMs);
// The 3 seeded rows sessionize into ≤ their span; not the naive span.
const naive =
new Date('2026-07-04T19:54:00Z').getTime() -
new Date('2026-07-04T03:40:00Z').getTime();
expect(result.workMs).toBeGreaterThan(0);
expect(result.workMs).toBeLessThan(naive);
});
it('an unknown timezone surfaces as a RangeError (controller maps to 400)', async () => {
await expect(
service.computeWorkTime(pageId, 'Not/AZone'),
).rejects.toBeInstanceOf(RangeError);
});
it('a page with no history → zeros, no days', async () => {
const emptyPage = (await createPage(db, { workspaceId, spaceId })).id;
const result = await service.computeWorkTime(emptyPage, 'UTC');
expect(result.workMs).toBe(0);
expect(result.agentOnlyMs).toBe(0);
expect(result.perDay).toEqual([]);
});
});
@@ -0,0 +1,612 @@
import { randomUUID } from 'node:crypto';
import { Kysely, sql } from 'kysely';
import { SearchService } from 'src/core/search/search.service';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
} from './db';
/**
* #529 Phase A the lexical overhaul, on the REAL migrated schema (ru_en config).
*
* Covers every acceptance criterion of the issue: RU+EN morphology + OR default,
* match=auto identifier routing, "phrase"/+/- operators, RRF ordering, exact
* permission-filtered total (fail-closed) + pagination, only-negation / garbage
* short-circuits, the A8 path fix, the response superset, and the RAG lockstep
* config (acceptance #13).
*
* The tsv column is populated by the pages_tsvector_trigger (now ru_en), so the
* FTS branch is exercised end to end.
*/
describe('SearchService #529 lexical overhaul [integration]', () => {
let db: Kysely<any>;
let workspaceId: string;
let spaceId: string;
async function insertPage(args: {
title: string;
textContent?: string;
parentPageId?: string | null;
spaceId?: string;
deletedAt?: Date | null;
}): Promise<string> {
const id = randomUUID();
await db
.insertInto('pages')
.values({
id,
slugId: `slug-${id.slice(0, 12)}`,
title: args.title,
textContent: args.textContent ?? null,
parentPageId: args.parentPageId ?? null,
spaceId: args.spaceId ?? spaceId,
workspaceId,
deletedAt: args.deletedAt ?? null,
})
.execute();
return id;
}
// Service wired to the real DB + real PageRepo (recursive descendants) with
// stubbed space-membership + permission repos so a test controls scope and the
// permission filter explicitly. `accessibleIds` (when set) is the KEEP list.
function buildService(opts?: {
userSpaceIds?: string[];
accessibleIds?: string[] | null;
filterThrows?: boolean;
}): SearchService {
const pageRepo = new PageRepo(db as any, null as any, null as any);
const spaceMemberRepo = {
getUserSpaceIds: async () => opts?.userSpaceIds ?? [spaceId],
};
const pagePermissionRepo = {
hasRestrictedAncestor: async () => false,
filterAccessiblePageIds: async ({ pageIds }: { pageIds: string[] }) => {
if (opts?.filterThrows) throw new Error('permission query failed');
return opts?.accessibleIds
? pageIds.filter((id) => opts.accessibleIds!.includes(id))
: pageIds;
},
};
return new SearchService(
db as any,
pageRepo as any,
{} as any,
spaceMemberRepo as any,
pagePermissionRepo as any,
);
}
const search = (service: SearchService, params: any) =>
service.searchPage(params, { userId: 'u-1', workspaceId }) as any;
beforeAll(async () => {
db = getTestDb();
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
// 1. RU morphology + OR: «ресторанов москвы» finds «ресторан в москве».
it('#1 russian morphology + OR: finds «ресторан в москве» for «ресторанов москвы»', async () => {
const page = await insertPage({
title: 'ресторан в москве',
textContent: 'Лучший ресторан столицы.',
});
const res = await search(buildService(), {
query: 'ресторанов москвы',
spaceId,
});
expect(res.items.map((i: any) => i.id)).toContain(page);
expect(res.total).toBeGreaterThanOrEqual(1);
});
// 2. OR non-empty: «Стамбул Роснефть».
it('#2 OR yields a hit when only one term matches', async () => {
const page = await insertPage({ title: 'Роснефть отчёт', textContent: 'x' });
const res = await search(buildService(), {
query: 'Стамбул Роснефть',
spaceId,
});
expect(res.items.map((i: any) => i.id)).toContain(page);
});
// 3. Multi-word OR: a page matching >=1 term is returned.
it('#3 «3D принтер» returns pages that matched at least one term', async () => {
const models = await insertPage({
title: 'Модели для печати 3D',
textContent: 'коллекция моделей',
});
const wish = await insertPage({
title: 'Хотеть напечатать на принтере',
textContent: 'очередь печати',
});
const res = await search(buildService(), { query: '3D принтер', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(models); // matched "3D"
expect(ids).toContain(wish); // matched "принтер"
// matchedTerms is populated per hit.
const hit = res.items.find((i: any) => i.id === wish);
expect(hit.matchedTerms).toContain('принтер');
});
// 4. match=auto FTS stemming: «печат» must NOT drag in «впечатления».
it('#4 «печат» (auto) matches «печать» but NOT «впечатления»', async () => {
const good = await insertPage({
title: 'Печать документов',
textContent: 'настройка печати',
});
const bad = await insertPage({
title: 'Впечатления от поездки',
textContent: 'много впечатлений',
});
const res = await search(buildService(), { query: 'печат', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(good);
expect(ids).not.toContain(bad);
});
// 5. Identifier → substring branch.
it('#5 `10.31.41` (auto→substring) finds the page with that IP', async () => {
const page = await insertPage({
title: 'Сетевой узел',
textContent: 'Адрес устройства: 10.31.41.7 в сети.',
});
const res = await search(buildService(), { query: '10.31.41', spaceId });
const hit = res.items.find((i: any) => i.id === page);
expect(hit).toBeDefined();
expect(hit.matchedFields).toContain('text');
});
// 6. +required / -excluded.
it('#6 `+кофейня -архив`: keeps «кофейня», drops pages with «архив»', async () => {
const keep = await insertPage({ title: 'Кофейня в центре', textContent: 'уют' });
const drop = await insertPage({
title: 'Кофейня старый архив',
textContent: 'архивные записи',
});
const res = await search(buildService(), { query: '+кофейня -архив', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(keep);
expect(ids).not.toContain(drop);
});
// 7. Phrase operator: only adjacent phrase hits survive.
it('#7 `+"воздушный шар" кофе`: every hit contains the adjacent phrase', async () => {
const adjacent = await insertPage({
title: 'Воздушный шар и кофе',
textContent: 'воздушный шар над городом, чашка кофе',
});
const nonAdjacent = await insertPage({
title: 'Красный воздушный большой шар',
textContent: 'воздушный красный шар и кофе рядом',
});
const res = await search(buildService(), {
query: '+"воздушный шар" кофе',
spaceId,
});
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(adjacent);
// The non-adjacent page (words separated) must NOT match the phrase.
expect(ids).not.toContain(nonAdjacent);
});
// 8. Pagination determinism + exact total.
it('#8 >50 matches: total>50, hasMore, and offset paginates without dupes', async () => {
const svc = buildService();
const created: string[] = [];
for (let i = 0; i < 60; i++) {
created.push(
await insertPage({
title: `паджинация запись ${i}`,
textContent: 'общий паджинационный маркер',
}),
);
}
const p1 = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 0,
});
expect(p1.total).toBeGreaterThanOrEqual(60);
expect(p1.hasMore).toBe(true);
const p2 = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 25,
});
const p3 = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 50,
});
const ids = [
...p1.items.map((i: any) => i.id),
...p2.items.map((i: any) => i.id),
...p3.items.map((i: any) => i.id),
];
// No duplicates across the three pages.
expect(new Set(ids).size).toBe(ids.length);
// Deterministic: same query twice → identical order.
const p1b = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 0,
});
expect(p1b.items.map((i: any) => i.id)).toEqual(p1.items.map((i: any) => i.id));
});
// 9. Only-negation.
it('#9 `-архив` only-negation: total 0, reason only-negation, no throw', async () => {
const res = await search(buildService(), { query: '-архив', spaceId });
expect(res.total).toBe(0);
expect(res.items).toEqual([]);
expect(res.query.parsed.reason).toBe('only-negation');
});
// 10. Garbage input.
it('#10 garbage `%` / `_` / empty: total 0, does not match everything', async () => {
await insertPage({ title: 'какая-то страница', textContent: 'текст' });
for (const q of ['%', '_', ' ', '%%__']) {
const res = await search(buildService(), { query: q, spaceId });
expect(res.total).toBe(0);
expect(res.items).toEqual([]);
}
});
// 11. Permission-filtered total (fail-closed) + mutation guard.
it('#11 a permission-hidden page is absent from items AND total', async () => {
const visible = await insertPage({
title: 'разрешённая пермишен-страница',
textContent: 'пермишенмаркер',
});
const hidden = await insertPage({
title: 'скрытая пермишен-страница',
textContent: 'пермишенмаркер',
});
// Filter keeps only the visible page.
const filtered = buildService({ accessibleIds: [visible] });
const res = await search(filtered, { query: 'пермишенмаркер', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(visible);
expect(ids).not.toContain(hidden);
// total is the POST-permission count — the hidden page does not leak into it.
expect(res.total).toBe(1);
// MUTATION: disable the guard (passthrough) → the hidden page reappears in
// BOTH items and total. If this did NOT change, the guard is not load-bearing.
const open = buildService({ accessibleIds: null });
const res2 = await search(open, { query: 'пермишенмаркер', spaceId });
expect(res2.items.map((i: any) => i.id)).toContain(hidden);
expect(res2.total).toBe(2);
});
it('#11b a permission-query error PROPAGATES (fail-closed, never empty)', async () => {
await insertPage({ title: 'failclosed маркер', textContent: 'failclosedmarker' });
const svc = buildService({ filterThrows: true });
await expect(
search(svc, { query: 'failclosedmarker', spaceId }),
).rejects.toThrow(/permission query failed/);
});
// A8 path fix.
it('#11c path: a soft-deleted / cross-space ancestor title does not leak', async () => {
const otherSpace = (await createSpace(db, workspaceId)).id;
const root = await insertPage({ title: 'Живой корень' });
const deletedMid = await insertPage({
title: 'УдалённыйПредок',
parentPageId: root,
deletedAt: new Date(),
});
const leaf = await insertPage({
title: 'a8leaf уникальный',
parentPageId: deletedMid,
textContent: 'a8leafmarker',
});
const res = await search(buildService(), { query: 'a8leafmarker', spaceId });
const hit = res.items.find((i: any) => i.id === leaf);
expect(hit).toBeDefined();
// The walk stops at the deleted ancestor — no deleted title in the path.
expect(hit.path).not.toContain('УдалённыйПредок');
expect(hit.path).not.toContain('Живой корень');
// Cross-space parent must also not leak.
const foreignParent = await insertPage({
title: 'ЧужойСпейс',
spaceId: otherSpace,
});
const crossLeaf = await insertPage({
title: 'crossleaf узел',
parentPageId: foreignParent,
textContent: 'crossleafmarker',
});
const res2 = await search(buildService(), {
query: 'crossleafmarker',
spaceId,
});
const hit2 = res2.items.find((i: any) => i.id === crossLeaf);
expect(hit2).toBeDefined();
expect(hit2.path).not.toContain('ЧужойСпейс');
});
// 12. Web-UI superset.
it('#12 web path (no flags) returns the OR result with the icon/space/highlight superset', async () => {
const page = await insertPage({
title: 'веб суперсет страница',
textContent: 'суперсетмаркер контент',
});
const res = await search(buildService(), { query: 'суперсетмаркер', spaceId });
const hit = res.items.find((i: any) => i.id === page);
expect(hit).toBeDefined();
// Superset fields the web-UI relies on.
expect('icon' in hit).toBe(true);
expect('space' in hit).toBe(true);
expect('highlight' in hit).toBe(true);
expect('rank' in hit).toBe(true);
// Plus the new fields.
expect('path' in hit).toBe(true);
expect('snippet' in hit).toBe(true);
expect('score' in hit).toBe(true);
// FTS hit carries a non-null rank + highlight.
expect(hit.rank).not.toBeNull();
});
// 13. RAG lockstep: the page_embeddings.fts generated column is ru_en.
it('#13 page_embeddings.fts uses ru_en (cyrillic stemming) — RAG lockstep', async () => {
const pageId = await insertPage({
title: 'rag страница',
textContent: 'ресторанов много',
});
// Insert a chunk row; the generated fts column is computed by Postgres.
await sql`
INSERT INTO page_embeddings
(id, page_id, workspace_id, space_id, attachment_id, chunk_index,
chunk_start, chunk_length, content, model_name, model_dimensions, embedding)
VALUES
(${randomUUID()}, ${pageId}, ${workspaceId}, ${spaceId}, NULL, 0,
0, 20, ${'ресторанов москвы много'}, 'test-model', 3, '[0.1,0.2,0.3]'::vector)
`.execute(db);
// A ru_en query stems «москва» → «москв», matching the stored «москвы».
// Under the old `english` config the cyrillic word would not stem and this
// inflected-form query would miss — so this asserts the ru_en lockstep.
const row = await sql<{ m: boolean }>`
SELECT fts @@ to_tsquery('ru_en', f_unaccent('москва')) AS m
FROM page_embeddings WHERE page_id = ${pageId}
`.execute(db);
expect(row.rows[0].m).toBe(true);
});
// W4 — substring-tier dominance under RRF: title-exact (tier 3) > title-
// substring (tier 2) > text-only (tier 1). The engine encodes this via
// sub_tier DESC → rn_sub → RRF; no test asserted the end-to-end ordering, so
// this restores that guarantee. match:'substring' routes the term to the
// substring branch (no FTS leg), so sub_tier alone drives the order.
it('W4 tier dominance: title-exact > title-substring > text-only', async () => {
const exact = await insertPage({ title: 'tierdomxyz' }); // tier 3
const titleSub = await insertPage({
title: 'prefix tierdomxyz suffix', // tier 2
});
const textOnly = await insertPage({
title: 'w4 unrelated heading',
textContent: 'body has tierdomxyz here', // tier 1
});
const res = await search(buildService(), {
query: 'tierdomxyz',
match: 'substring',
spaceId,
});
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(exact);
expect(ids).toContain(titleSub);
expect(ids).toContain(textOnly);
// Strict tier order.
expect(ids.indexOf(exact)).toBeLessThan(ids.indexOf(titleSub));
expect(ids.indexOf(titleSub)).toBeLessThan(ids.indexOf(textOnly));
});
// S3 — an exact-title hit must NOT be lost when the match set exceeds
// CANDIDATE_CAP: it ranks first under RRF (tier 3) so it lands in the reachable
// window even with a tiny cap. Restores a guarantee the old lookup suite gave.
it('S3 exact-title survives the CANDIDATE_CAP window', async () => {
const exact = await insertPage({ title: 'capmarkerxyz' }); // tier 3
for (let i = 0; i < 6; i++) {
await insertPage({
title: `cap filler ${i}`,
textContent: 'noise capmarkerxyz noise', // tier 1
});
}
process.env.SEARCH_CANDIDATE_CAP = '2';
try {
const res = await search(buildService(), {
query: 'capmarkerxyz',
match: 'substring',
spaceId,
limit: 25,
});
// More matches than the cap → truncated, but the exact-title survives.
expect(res.total).toBeGreaterThan(2);
expect(res.truncatedAtCap).toBe(true);
expect(res.items.length).toBe(2); // the reachable window
expect(res.items.map((i: any) => i.id)).toContain(exact);
} finally {
delete process.env.SEARCH_CANDIDATE_CAP;
}
});
// S1 — a required-only query (`+term`, no bare positive) really matches, so its
// hits must report matchedFields / rank / highlight, not [] / null. Before the
// fix these detail exprs were built from parsed.positive only.
it('S1 required-only query populates matchedFields + rank on a title match', async () => {
const page = await insertPage({
title: 'Кофейня s1маркер центр',
textContent: 'обычный текст без ключевого слова',
});
const res = await search(buildService(), { query: '+кофейня', spaceId });
const hit = res.items.find((i: any) => i.id === page);
expect(hit).toBeDefined();
// The title matches the required term → matchedFields includes 'title'.
expect(hit.matchedFields).toContain('title');
// FTS rank is populated (was null before the S1 fix).
expect(hit.rank).not.toBeNull();
// matchedTerms already echoed the required term; still true.
expect(hit.matchedTerms).toContain('кофейня');
});
// F1 — stack-depth guard: a pasted text block (thousands of FTS terms) used to
// nest the combined tsquery so deep that Postgres raised `stack depth limit
// exceeded` → HTTP 500 for the caller. The parser now caps terms at
// MAX_PARSED_TERMS, so a huge query returns 200 and the leading (in-cap) term
// still matches. Mutation: remove the cap → this reddens (500 / stack depth).
it('F1 a huge multi-term query returns 200, not a stack-depth 500', async () => {
const page = await insertPage({
title: 'qfonemarker заголовок',
textContent: 'тело страницы',
});
// Purely-alphabetic filler words → FTS branch (the branch that nests in
// combineTsq). A digit-bearing token would route to substring and not nest.
const alphaWord = (i: number): string => {
let s = '';
let n = i + 1;
while (n > 0) {
s = String.fromCharCode(97 + (n % 26)) + s;
n = Math.floor(n / 26);
}
return 'q' + s;
};
const words = [
'qfonemarker',
...Array.from({ length: 5000 }, (_, i) => alphaWord(i)),
];
const res = await search(buildService(), {
query: words.join(' '),
spaceId,
});
// Bounded nesting → no 500; the leading in-cap term still recalls the page.
expect(res.items.map((i: any) => i.id)).toContain(page);
});
// F2 — titleOnly leak-guard (restores coverage the deleted lookup spec gave).
// A term present ONLY in text_content must NOT match under titleOnly, and a
// title hit must carry NO body snippet. Uses match:'substring' because titleOnly
// gates the substring branch (the FTS `pages.tsv` already spans title+body).
it('F2 titleOnly does not match text_content and yields no body snippet', async () => {
// Marker lives only in the body, never in the title.
const bodyOnly = await insertPage({
title: 'нейтральный заголовок f2',
textContent: 'секрет titleonlybody конец',
});
const res = await search(buildService(), {
query: 'titleonlybody',
match: 'substring',
spaceId,
titleOnly: true,
});
// titleOnly must NOT leak a body-substring match.
expect(res.items.map((i: any) => i.id)).not.toContain(bodyOnly);
// Sanity: WITHOUT titleOnly the same term DOES find it via the body.
const resOpen = await search(buildService(), {
query: 'titleonlybody',
match: 'substring',
spaceId,
});
expect(resOpen.items.map((i: any) => i.id)).toContain(bodyOnly);
// A page that hits on its TITLE: the snippet must be empty (no body leaks in).
const titleHit = await insertPage({
title: 'titleonlytitle страница',
textContent: 'какой-то текст тела здесь для сниппета',
});
const res2 = await search(buildService(), {
query: 'titleonlytitle',
match: 'substring',
spaceId,
titleOnly: true,
});
const hit = res2.items.find((i: any) => i.id === titleHit);
expect(hit).toBeDefined();
expect(hit.snippet).toBe('');
});
// F3 — parentPageId subtree scoping (restores coverage the deleted lookup spec
// gave). Two sibling subtrees share one term; scoping to ONE root returns only
// that subtree's hits INCLUDING the root/parent page itself, and NONE of the
// sibling subtree. Mutation: drop the ANY(descendantIds) filter → this reddens.
it('F3 parentPageId scopes to one subtree incl. the parent, excludes siblings', async () => {
const rootA = await insertPage({
title: 'subtreeA корень',
textContent: 'f3marker в корне A',
});
const childA = await insertPage({
title: 'subtreeA потомок',
textContent: 'f3marker в потомке A',
parentPageId: rootA,
});
const rootB = await insertPage({
title: 'subtreeB корень',
textContent: 'f3marker в корне B',
});
const childB = await insertPage({
title: 'subtreeB потомок',
textContent: 'f3marker в потомке B',
parentPageId: rootB,
});
const res = await search(buildService(), {
query: 'f3marker',
spaceId,
parentPageId: rootA,
});
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(rootA); // the parent/root page itself
expect(ids).toContain(childA);
expect(ids).not.toContain(rootB); // sibling subtree cut off
expect(ids).not.toContain(childB);
});
// F4 — positive path + snippet content asserts (the #11c test only asserts what
// must NOT be in path). (a) a nested hit's path is root→parent ordered and a
// root-level hit's path is []; (b) a text-body hit returns a NON-empty snippet.
it('F4 path is root→parent ordered ([] at root) and body hits carry a snippet', async () => {
const root = await insertPage({ title: 'F4Root' });
const parent = await insertPage({ title: 'F4Parent', parentPageId: root });
const leaf = await insertPage({
title: 'f4leaf лист',
parentPageId: parent,
textContent: 'f4leafmarker тело с содержимым для сниппета',
});
const res = await search(buildService(), {
query: 'f4leafmarker',
spaceId,
});
const hit = res.items.find((i: any) => i.id === leaf);
expect(hit).toBeDefined();
// Positive ancestry: root → direct parent, hit's own title excluded.
expect(hit.path).toEqual(['F4Root', 'F4Parent']);
// Snippet content: a text-body hit returns a non-empty snippet with the term.
expect(hit.snippet.length).toBeGreaterThan(0);
expect(hit.snippet).toContain('f4leafmarker');
// A root-level hit (no parent) → empty path.
const rootHit = await insertPage({
title: 'f4rootlevel уникальный',
textContent: 'f4rootmarker в теле',
});
const res2 = await search(buildService(), {
query: 'f4rootmarker',
spaceId,
});
const hit2 = res2.items.find((i: any) => i.id === rootHit);
expect(hit2).toBeDefined();
expect(hit2.path).toEqual([]);
});
});
@@ -1,462 +0,0 @@
import { randomUUID } from 'node:crypto';
import { Kysely } from 'kysely';
import { SearchService } from 'src/core/search/search.service';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
} from './db';
/**
* #443 agent-lookup search mode, acceptance on the REAL DB schema.
*
* Exercises SearchService.searchPage(..., { substring: true }) against a
* migrated Postgres: substring matching of technical tokens the FTS tokenizer
* mangles (backup-srv.local, 10.0.12.5, WB-MGE-30D86B, "Теги: Docker"), the
* populated path + snippet, parentPageId subtree scoping, titleOnly, the empty
* result, LIKE-metacharacter escaping (`%`/`_` must NOT match everything), the
* permission post-filter applied BEFORE the limit, and the web-UI path staying
* on the legacy FTS shape when `substring` is absent.
*
* The tsv column is populated by the pages_tsvector_trigger on insert, so the
* FTS branch is exercised too.
*/
describe('SearchService agent-lookup mode [integration]', () => {
let db: Kysely<any>;
let service: SearchService;
let workspaceId: string;
let spaceId: string;
// Direct page insert (the shared createPage seeder omits text_content /
// parent_page_id, both of which this mode depends on). Returns the id.
async function insertPage(args: {
title: string;
textContent?: string;
parentPageId?: string | null;
spaceId?: string;
}): Promise<string> {
const id = randomUUID();
await db
.insertInto('pages')
.values({
id,
slugId: `slug-${id.slice(0, 12)}`,
title: args.title,
textContent: args.textContent ?? null,
parentPageId: args.parentPageId ?? null,
spaceId: args.spaceId ?? spaceId,
workspaceId,
})
.execute();
return id;
}
// Build a SearchService wired to the real DB + a real PageRepo (only its
// recursive-descendants method is used by this mode, and it needs only `db`),
// with lightweight stubs for the space-membership and permission repos so a
// test can drive scope + the permission post-filter explicitly.
function buildService(opts?: {
userSpaceIds?: string[];
// ids to KEEP after the permission post-filter; undefined = keep all.
accessibleIds?: string[];
}): SearchService {
const pageRepo = new PageRepo(db as any, null as any, null as any);
const spaceMemberRepo = {
getUserSpaceIds: async () => opts?.userSpaceIds ?? [spaceId],
};
const pagePermissionRepo = {
filterAccessiblePageIds: async ({ pageIds }: { pageIds: string[] }) =>
opts?.accessibleIds
? pageIds.filter((id) => opts.accessibleIds!.includes(id))
: pageIds,
};
return new SearchService(
db as any,
pageRepo as any,
{} as any, // shareRepo — unused by the lookup path
spaceMemberRepo as any,
pagePermissionRepo as any,
);
}
beforeAll(async () => {
db = getTestDb();
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
service = buildService();
});
afterAll(async () => {
await destroyTestDb();
});
it('finds `backup-srv.local` by the fragment `srv.local`', async () => {
const pageId = await insertPage({
title: 'backup-srv.local',
textContent: 'A backup server node.',
});
const { items } = (await service.searchPage(
{ query: 'srv.local', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
expect(items.map((i: any) => i.id)).toContain(pageId);
const hit = items.find((i: any) => i.id === pageId);
expect(hit.title).toBe('backup-srv.local');
// slugId must never be part of the server response shape.
expect('slugId' in hit).toBe(true); // server carries it; MCP strips it
});
it('finds a page whose TEXT contains `10.0.12.5` by the fragment `10.0.12` (empty-tsquery case)', async () => {
const pageId = await insertPage({
title: 'Server inventory',
textContent: 'The backup box lives at IP: 10.0.12.5. Debian 12, backups.',
});
const { items } = (await service.searchPage(
{ query: '10.0.12', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// The windowed snippet must include the matched text.
expect(hit.snippet).toContain('10.0.12.5');
});
it('finds `WB-MGE-30D86B` (alphanumeric token with dashes) by title', async () => {
const pageId = await insertPage({
title: 'WB-MGE-30D86B',
textContent: 'Device page.',
});
const { items } = (await service.searchPage(
{ query: 'WB-MGE-30D86B', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// Exact title match → top tier (TITLE_EXACT=3) → score in [0.75, 1].
expect(hit.score).toBeGreaterThanOrEqual(0.75);
// And it is the top-ranked hit of its own result set.
expect(items[0].id).toBe(pageId);
});
it('finds every page whose text literally contains `Теги: Docker`', async () => {
const a = await insertPage({
title: 'Container host A',
textContent: 'Some notes.\nТеги: Docker, compose\nmore.',
});
const b = await insertPage({
title: 'Container host B',
textContent: 'Prelude.\nТеги: Docker\nepilogue.',
});
const noise = await insertPage({
title: 'Unrelated',
textContent: 'Теги: Kubernetes',
});
const { items } = (await service.searchPage(
{ query: 'Теги: Docker', spaceId, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(a);
expect(ids).toContain(b);
expect(ids).not.toContain(noise);
});
it('populates a non-empty `path` for a nested hit and `[]` for a root hit', async () => {
const root = await insertPage({ title: 'Infrastructure' });
const mid = await insertPage({ title: 'Datacenter A', parentPageId: root });
const leaf = await insertPage({
title: 'unique-nested-host',
parentPageId: mid,
});
const { items } = (await service.searchPage(
{ query: 'unique-nested-host', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === leaf);
expect(hit.path).toEqual(['Infrastructure', 'Datacenter A']);
const rootHits = (await service.searchPage(
{ query: 'Infrastructure', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const rootHit = rootHits.items.find((i: any) => i.id === root);
expect(rootHit.path).toEqual([]);
});
it('scopes to a subtree with parentPageId (cutting off sibling branches)', async () => {
const branchA = await insertPage({ title: 'BranchA-root' });
const inA = await insertPage({
title: 'scoped-target-xyz',
parentPageId: branchA,
});
const branchB = await insertPage({ title: 'BranchB-root' });
const inB = await insertPage({
title: 'scoped-target-xyz',
parentPageId: branchB,
});
const { items } = (await service.searchPage(
{
query: 'scoped-target-xyz',
spaceId,
substring: true,
parentPageId: branchA,
} as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(inA);
expect(ids).not.toContain(inB);
});
it('includes the parent page itself in the parentPageId subtree', async () => {
const parent = await insertPage({ title: 'self-included-parent' });
await insertPage({ title: 'child-of-self', parentPageId: parent });
const { items } = (await service.searchPage(
{
query: 'self-included-parent',
spaceId,
substring: true,
parentPageId: parent,
} as any,
{ workspaceId },
)) as any;
expect(items.map((i: any) => i.id)).toContain(parent);
});
it('titleOnly does NOT match on text_content', async () => {
const pageId = await insertPage({
title: 'Plain title',
textContent: 'body mentions the-secret-token here',
});
const withText = (await service.searchPage(
{ query: 'the-secret-token', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
expect(withText.items.map((i: any) => i.id)).toContain(pageId);
const titleOnly = (await service.searchPage(
{
query: 'the-secret-token',
spaceId,
substring: true,
titleOnly: true,
} as any,
{ workspaceId },
)) as any;
expect(titleOnly.items.map((i: any) => i.id)).not.toContain(pageId);
});
// #443 Fix #1 regression: f_unaccent is NOT length-preserving, so an
// expanding char (ß→ss, …→...) BEFORE the match shifted the strpos position
// relative to the ORIGINAL text and the snippet slice ran past end → empty.
// The position and the slice now share the LOWER(f_unaccent(...)) space, so
// the window is aligned and always contains the matched (unaccented) token.
it('returns a populated snippet when an unaccent-EXPANDING char precedes the match', async () => {
// 300 × `ß` (each f_unaccent-expands to `ss`) before the needle. Under the
// old code strpos returned a position ~593 in the expanded space but the
// slice ran over the ORIGINAL (~360 char) text → empty snippet, match lost.
const prefix = 'ß'.repeat(300);
const pageId = await insertPage({
title: 'Expanding-unaccent page',
textContent: `${prefix} needle-token-xyz trailing.`,
});
const { items } = (await service.searchPage(
{ query: 'needle-token-xyz', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// Snippet must be non-empty AND contain the matched token (unaccented form).
expect(hit.snippet.length).toBeGreaterThan(0);
expect(hit.snippet).toContain('needle-token-xyz');
});
// #443 Fix #2 regression: >200 matching pages for a broad substring, with
// exactly ONE exact-title hit. Without an ORDER BY on the 200-cap the exact
// hit could be among the arbitrarily-dropped rows; the ORDER BY keeps the
// strongest candidates so it must survive the cap and rank at the top.
it('keeps an exact-title hit through the 200-cap on a >200-row match set', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
// 250 low-tier TEXT hits: the shared substring `capword` appears only in the
// body, never the title, so each is a TEXT-tier match (weakest tier).
for (let i = 0; i < 250; i++) {
await insertPage({
title: `filler-page-${i}`,
textContent: `body contains capword here #${i}`,
spaceId: isoSpace,
});
}
// Exactly one EXACT-title hit for the same query token.
const exact = await insertPage({
title: 'capword',
textContent: 'unrelated body text',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: 'capword', spaceId: isoSpace, substring: true, limit: 10 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
// The exact-title hit must survive the 200-cap and appear in the top `limit`.
expect(ids).toContain(exact);
// And, being TITLE_EXACT, it must be the single strongest hit.
expect(items[0].id).toBe(exact);
});
// #443 Fix #3: titleOnly matches only the title, so it must not leak the page
// body as the snippet (the old "first 300 chars of text_content" fallback).
it('titleOnly does NOT return a text-body snippet', async () => {
const pageId = await insertPage({
title: 'titleonly-snippet-page',
textContent: 'SECRET-BODY-CONTENT-NOT-IN-TITLE that must not leak.',
});
const { items } = (await service.searchPage(
{
query: 'titleonly-snippet-page',
spaceId,
substring: true,
titleOnly: true,
} as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// The body text must not appear in the snippet; titleOnly → empty snippet.
expect(hit.snippet).not.toContain('SECRET-BODY-CONTENT-NOT-IN-TITLE');
expect(hit.snippet).toBe('');
});
it('returns [] (not an error) for a query that matches nothing', async () => {
const { items } = (await service.searchPage(
{
query: 'zzz-no-such-string-anywhere-42',
spaceId,
substring: true,
} as any,
{ workspaceId },
)) as any;
expect(items).toEqual([]);
});
it('a `%` query does NOT match everything (LIKE metacharacter escaped)', async () => {
// Fresh space so we can assert on total counts without cross-test noise.
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
await insertPage({ title: 'alpha', spaceId: isoSpace });
await insertPage({ title: 'beta', spaceId: isoSpace });
const literal = await insertPage({
title: '100%-coverage',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: '%', spaceId: isoSpace, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
// `%` is a literal → matches only the page that actually contains '%'.
expect(ids).toContain(literal);
expect(ids).not.toContain(
items.find((i: any) => i.title === 'alpha')?.id,
);
expect(items.length).toBe(1);
});
it('an `_` query does NOT match everything (LIKE metacharacter escaped)', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
await insertPage({ title: 'gamma', spaceId: isoSpace });
const literal = await insertPage({
title: 'snake_case_name',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: '_', spaceId: isoSpace, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(literal);
expect(items.length).toBe(1);
});
it('applies the permission post-filter to the MERGED set BEFORE the limit', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const keep = await insertPage({
title: 'perm-visible-target',
spaceId: isoSpace,
});
const hidden = await insertPage({
title: 'perm-hidden-target',
spaceId: isoSpace,
});
// Authenticated (userId set) so the permission filter runs; only `keep` is
// accessible. limit 1 must NOT be able to select `hidden`.
const svc = buildService({
userSpaceIds: [isoSpace],
accessibleIds: [keep],
});
const { items } = (await svc.searchPage(
{
query: 'perm-',
spaceId: isoSpace,
substring: true,
limit: 1,
} as any,
{ userId: 'user-1', workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(keep);
expect(ids).not.toContain(hidden);
});
it('web-UI path (no `substring` flag) keeps the legacy FTS response shape', async () => {
await insertPage({
title: 'legacy shape page',
textContent: 'searchable legacyword content',
});
const { items } = (await service.searchPage(
{ query: 'legacyword', spaceId } as any,
{ userId: 'user-1', workspaceId },
)) as any;
// Legacy hits carry rank + highlight + space, and NO path/snippet/score.
const hit = items[0];
expect(hit).toBeDefined();
expect('rank' in hit).toBe(true);
expect('highlight' in hit).toBe(true);
expect('path' in hit).toBe(false);
expect('snippet' in hit).toBe(false);
expect('score' in hit).toBe(false);
});
});
@@ -0,0 +1,111 @@
import { Kysely, sql } from 'kysely';
import { getTestDb, destroyTestDb } from './db';
import * as migration from '../../src/database/migrations/20260707T130000-search-ru-en-config';
/**
* #529 A1 the ru_en config migration must be REVERSIBLE in the correct order:
* down() moves pages.tsv + page_embeddings.fts back to `english` BEFORE dropping
* the ru_en config (a generated column still depending on ru_en would block the
* DROP). This roundtrips down()up() on the already-migrated test DB and asserts
* the config, the pages trigger and the fts generated expression each flip and
* flip back the deploy-critical property.
*/
describe('search ru_en config migration [integration]', () => {
let db: Kysely<any>;
const configExists = async () => {
const r = await sql<{ n: number }>`
SELECT count(*)::int AS n FROM pg_ts_config WHERE cfgname = 'ru_en'
`.execute(db);
return r.rows[0].n > 0;
};
const ftsDef = async () => {
const r = await sql<{ def: string }>`
SELECT pg_get_expr(adbin, adrelid) AS def
FROM pg_attrdef d
JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attname = 'fts' AND a.attrelid = 'page_embeddings'::regclass
`.execute(db);
return r.rows[0]?.def ?? '';
};
const triggerSrc = async () => {
const r = await sql<{ src: string }>`
SELECT prosrc AS src FROM pg_proc WHERE proname = 'pages_tsvector_trigger'
`.execute(db);
return r.rows[0]?.src ?? '';
};
beforeAll(() => {
db = getTestDb();
});
afterAll(async () => {
// Restore the canonical ru_en state for any later suite regardless of where
// a test left off. up() is now safe on an existing config (ensureRuEnConfig
// no-ops) and re-asserts both stored sides to ru_en.
delete process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE;
await migration.up(db);
await destroyTestDb();
});
it('starts at ru_en (config present, trigger + fts on ru_en)', async () => {
expect(await configExists()).toBe(true);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
});
it('down() reverts tsv + fts to english THEN drops the config', async () => {
await migration.down(db);
expect(await configExists()).toBe(false);
expect(await ftsDef()).toContain('english');
expect(await ftsDef()).not.toContain('ru_en');
expect(await triggerSrc()).toContain('english');
});
it('up() re-applies ru_en cleanly (idempotent config create)', async () => {
await migration.up(db);
expect(await configExists()).toBe(true);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
});
// B1.1 — running up() a SECOND time on an already-migrated DB is a true no-op:
// it must NOT throw (the old drop-recreate-config would fail on the fts hard
// dependency) and must NOT re-rewrite the embeddings table — the fts column
// already references ru_en, so the at-target check short-circuits.
it('up() is idempotent: a 2nd run does not error and leaves ru_en intact', async () => {
await expect(migration.up(db)).resolves.toBeUndefined();
expect(await configExists()).toBe(true);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
});
// B1.2 — env-gate opt-out: with SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false the
// embeddings rewrite is SKIPPED (fts stays where it was) but pages.tsv still
// swaps inline, and the ru_en config is left in place (fts still depends on it).
// Runs down() as the vehicle: english is NOT the current fts config, so the
// skip path — not the at-target no-op — is exercised.
it('env-gate=false: skips the fts rewrite but still swaps pages.tsv', async () => {
// Precondition: fts + trigger on ru_en (from the prior test).
expect(await ftsDef()).toContain('ru_en');
process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE = 'false';
try {
await migration.down(db);
// fts rewrite skipped → still ru_en (the ACCESS EXCLUSIVE rewrite avoided).
expect(await ftsDef()).toContain('ru_en');
expect(await ftsDef()).not.toContain('english');
// pages.tsv trigger still swapped inline to english (gate is fts-only).
expect(await triggerSrc()).toContain('english');
// Config left in place because fts still references it (guarded drop).
expect(await configExists()).toBe(true);
} finally {
// Restore ru_en fully for the afterAll / later suites.
delete process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE;
await migration.up(db);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
}
});
});
+4 -1
View File
@@ -118,7 +118,10 @@ All 41 tools, grouped by what you'd reach for them.
- **`getPage`** — A page's content as clean **Markdown** (canonical for text; drops only
block ids, resolved-comment anchors, and a fixed no-Markdown-representation attr set —
table spans/colwidth/background, indent, `callout.icon`, `orderedList.type`, and link
`internal`/`target`/`rel`/`class`; use `getPageJson` when you need those).
`internal`/`target`/`rel`/`class`; use `getPageJson` when you need those). Pass
`format:"text"` for a flat, deterministic plain-text rendering (one line per block,
marks dropped, stable `[image]`/`[table RxC]` placeholders) to machine-diff what you
wrote against what was stored.
- **`getPageJson`** — A page's **lossless ProseMirror/TipTap JSON**, including every
block's `attrs.id` and the `slugId` used in URLs. This is what the per-block editing
tools consume.
+4 -1
View File
@@ -123,7 +123,10 @@ Docmost-MCP не сочетают:
лишь id блоков, якоря разрешённых комментариев и фиксированный набор атрибутов без
markdown-представления — спаны/colwidth/фон ячеек таблиц, отступы (indent),
`callout.icon`, `orderedList.type` и `internal`/`target`/`rel`/`class` у ссылок;
используйте `getPageJson`, когда они нужны).
используйте `getPageJson`, когда они нужны). Передайте `format:"text"` для плоского
детерминированного plain-text рендера (одна строка на блок, маркировка убрана,
стабильные плейсхолдеры `[image]`/`[table RxC]`) — чтобы machine-diff'ом сверить то,
что вы записали, с тем, что сохранилось.
- **`getPageJson`** — **Lossless ProseMirror/TipTap JSON** страницы, включая `attrs.id`
каждого блока и `slugId`, используемый в URL. Именно его потребляют инструменты
поблочного редактирования.
+5 -1
View File
@@ -31,7 +31,11 @@ import { TransformsMixin, type ITransformsMixin } from "./client/transforms.js";
// existing importer (index.ts, http.ts, stdio.ts, the in-app host) keeps working
// with ZERO changes.
export type { DocmostMcpConfig, SandboxPut } from "./client/context.js";
export { formatDocmostAxiosError, assertFullUuid } from "./client/errors.js";
export {
formatDocmostAxiosError,
assertFullUuid,
formatSpaceNotAccessible,
} from "./client/errors.js";
// Branded canonical page-identity type (#435): the internal page UUID is a
// distinct nominal type so an unresolved raw/slug string can't be swapped into
+9 -4
View File
@@ -725,10 +725,15 @@ export function CommentsMixin<TBase extends GConstructor<DocmostClientContext>>(
// The subtree scope (parentPageId given) already INCLUDES the root node
// itself: /pages/tree seeds getPageAndDescendants with id = parentPageId, so
// no separate getPageRaw fetch for the parent is needed.
const { pages: pagesInScope, truncated } = await this.enumerateSpacePages(
spaceId,
parentPageId,
);
// #534: the enumerateSpacePages seed (`/pages/tree`) 404s for a bad or
// inaccessible spaceId; wrap it so that 404 becomes an actionable "spaceId
// not accessible" hint instead of the opaque "Space permissions not found".
// Only the whole enumeration is wrapped (the only 404 source here) — see the
// wrap-allowlist invariant on withSpaceAccessDiagnostics.
const { pages: pagesInScope, truncated } =
await this.withSpaceAccessDiagnostics(spaceId, "checkNewComments", () =>
this.enumerateSpacePages(spaceId, parentPageId),
);
// 2. Fetch comments for each page, keep ones created after since. Runs with
// bounded concurrency (#490) instead of one-at-a-time — the per-page reads are
+226 -6
View File
@@ -26,7 +26,10 @@ import {
import { withPageLock, isUuid } from "../lib/page-lock.js";
import type { PageId } from "../lib/page-id.js";
import { getCollabToken, performLogin } from "../lib/auth-utils.js";
import { formatDocmostAxiosError } from "./errors.js";
import {
formatDocmostAxiosError,
formatSpaceNotAccessible,
} from "./errors.js";
import { GetPageConversionCache } from "./getpage-cache.js";
// A generic mixin base constructor (issue #450). Each domain mixin is a factory
@@ -117,6 +120,36 @@ function readCollabTokenTtlMs(): number {
return Number.isFinite(raw) ? Math.max(0, raw) : 5 * 60 * 1000;
}
/**
* Accessible-space index cache TTL in milliseconds (issue #534). Read fresh from
* the environment on every access mirroring readCollabTokenTtlMs above so a
* test or a live rollback can change it without reloading the module.
*
* The index (see getAccessibleSpaceIndex) is fetched ONLY on the enrich-on-404
* slow path to turn an opaque "Space permissions not found" 404 into a factual
* "spaceId X is not among your accessible spaces" hint; a short TTL keeps a burst
* of failing tool calls from re-sweeping /spaces each time while never widening
* the permission-staleness window meaningfully. Default 60s. An EXPLICIT 0 (or
* negative) DISABLES the cache (exact fetch-per-enrichment). Unset/unparseable
* (NaN) falls back to the 60s default with the cache ON.
*/
function readSpacesCacheTtlMs(): number {
const raw = parseInt(process.env.MCP_SPACES_CACHE_TTL_MS ?? "", 10);
return Number.isFinite(raw) ? Math.max(0, raw) : 60000;
}
/**
* The set of spaces the current token can see, plus a `complete` flag that is
* false when the /spaces listing was truncated at the pagination ceiling. Used
* by the enrich-on-404 diagnostics: an authoritative membership test is only
* possible when `complete` is true (see withSpaceAccessDiagnostics).
*/
export type AccessibleSpaceIndex = {
ids: Set<string>;
spaces: { id: string; name: string }[];
complete: boolean;
};
export abstract class DocmostClientContext {
protected client: AxiosInstance;
protected token: string | null = null;
@@ -164,6 +197,27 @@ export abstract class DocmostClientContext {
// bypassed on a forced refresh (the 401/403 reauth path). null = no token yet.
protected collabTokenCache: { token: string; mintedAt: number } | null = null;
// Accessible-space index cache + single-flight (issue #534). TWO separate
// fields, mirroring loginPromise (in-flight dedup) vs collabTokenCache
// (persistent value):
// - spaceIndexCache: the last SUCCESSFULLY-FETCHED, COMPLETE index plus the
// wall-clock time it was fetched. Written ONLY from a resolved /spaces
// sweep whose result was complete (a truncated list is never cached, since
// it cannot answer "is this spaceId missing?"). Per-instance (a
// DocmostClient is built per user / per chat) so it can never leak across
// identities; invalidated on every identity change exactly like
// collabTokenCache (login() + the 401/403 reauth interceptor).
// - spaceIndexInFlight: dedups concurrent enrich-on-404 fetches into ONE
// /spaces sweep. CRITICAL INVARIANT: this promise is nulled in `.finally`
// on BOTH resolve AND reject — a rejected/settled promise is NEVER
// memoized, so a transient /spaces blip during one failed tool call cannot
// poison the diagnostics for the rest of the session.
protected spaceIndexCache: {
index: AccessibleSpaceIndex;
fetchedAt: number;
} | null = null;
protected spaceIndexInFlight: Promise<AccessibleSpaceIndex> | null = null;
// Content-addressed conversion cache for getPage (issue #479). Keyed on
// (canonical pageId, updatedAt, optionsHash) -> the converted Markdown, so a
// re-read of an UNCHANGED page skips the expensive convertProseMirrorToMarkdown
@@ -281,6 +335,9 @@ export abstract class DocmostClientContext {
// keep serving a collab token minted under the old one.
this.token = null;
this.collabTokenCache = null;
// #534: a new identity/login must not keep serving a space index
// computed under the old token (same reasoning as collabTokenCache).
this.spaceIndexCache = null;
delete this.client.defaults.headers.common["Authorization"];
try {
await this.login();
@@ -413,6 +470,8 @@ export abstract class DocmostClientContext {
// Identity (re)established: drop any collab token minted under a
// previous identity so the #435 cache can never outlive it.
this.collabTokenCache = null;
// #534: likewise drop the accessible-space index of the old identity.
this.spaceIndexCache = null;
this.client.defaults.headers.common["Authorization"] =
`Bearer ${token}`;
})
@@ -630,13 +689,34 @@ export abstract class DocmostClientContext {
}
/**
* Generic pagination handler for Docmost API endpoints
* Generic pagination handler for Docmost API endpoints. Thin wrapper over
* paginateAllWithMeta that discards the `truncated` flag the historical
* contract every caller (getSpaces, etc.) relies on. Callers that need to KNOW
* whether the result set was complete (e.g. #534's getAccessibleSpaceIndex,
* which must not assert "spaceId missing" against a truncated list) call
* paginateAllWithMeta directly.
*/
async paginateAll<T = any>(
endpoint: string,
basePayload: Record<string, any> = {},
limit: number = 100,
): Promise<T[]> {
return (await this.paginateAllWithMeta<T>(endpoint, basePayload, limit))
.items;
}
/**
* Generic pagination handler that ALSO surfaces whether the result was
* truncated at the MAX_PAGES ceiling. `paginateAll` swallows this flag (it only
* warns); callers that must distinguish "complete listing" from "gave up at the
* cap" use this overload. `truncated` is true iff the loop stopped at the
* ceiling while the server still reported more pages.
*/
async paginateAllWithMeta<T = any>(
endpoint: string,
basePayload: Record<string, any> = {},
limit: number = 100,
): Promise<{ items: T[]; truncated: boolean }> {
await this.ensureAuthenticated();
const clampedLimit = Math.max(1, Math.min(100, limit));
@@ -697,14 +777,154 @@ export abstract class DocmostClientContext {
);
}
return allItems;
return { items: allItems, truncated };
}
/**
* The set of spaces the current token can access (issue #534), fetched from the
* single source of truth the `/spaces` listing with a per-instance
* short-TTL cache and single-flight dedup. Used ONLY by the enrich-on-404 slow
* path (withSpaceAccessDiagnostics), so the happy path incurs ZERO extra
* requests.
*
* `complete` is `!truncated`: it is false when the /spaces listing was cut at
* the pagination ceiling. A truncated index can never authoritatively answer
* "is this spaceId missing?", so only a complete result is cached AND only a
* complete result is allowed to drive the "not accessible" rewrite.
*
* Cache/single-flight discipline (see the spaceIndexCache / spaceIndexInFlight
* field docs):
* - serve a fresh, complete cached index without any request;
* - otherwise collapse concurrent callers onto ONE in-flight /spaces sweep;
* - write the persistent cache ONLY from a resolved, complete fetch;
* - null the in-flight promise on BOTH resolve and reject (never memoize a
* rejected promise a transient /spaces failure must be retried fresh).
*/
async getAccessibleSpaceIndex(): Promise<AccessibleSpaceIndex> {
const ttl = readSpacesCacheTtlMs();
// Fast path: a still-fresh, complete cached index needs no request at all.
if (
ttl > 0 &&
this.spaceIndexCache &&
Date.now() - this.spaceIndexCache.fetchedAt < ttl
) {
return this.spaceIndexCache.index;
}
// Single-flight: a concurrent enrichment joins the in-flight sweep instead of
// issuing its own. (A settled/rejected promise is never left here — see the
// `.finally` below — so this only ever joins a genuinely in-progress fetch.)
if (this.spaceIndexInFlight) return this.spaceIndexInFlight;
const fetchPromise = (async (): Promise<AccessibleSpaceIndex> => {
const { items, truncated } = await this.paginateAllWithMeta("/spaces", {});
const spaces = items.map((s: any) => ({
id: s?.id,
name: s?.name,
}));
return {
ids: new Set(spaces.map((s) => s.id)),
spaces,
complete: !truncated,
};
})();
this.spaceIndexInFlight = fetchPromise
.then((index) => {
// Cache ONLY a complete result, and only while the cache is enabled.
if (ttl > 0 && index.complete) {
this.spaceIndexCache = { index, fetchedAt: Date.now() };
}
return index;
})
.finally(() => {
// CRITICAL (#534): clear the in-flight slot on BOTH resolve and reject.
// Nulling on reject too means a transient /spaces error is retried by the
// NEXT enrichment with a fresh fetch, never re-serving the rejection.
this.spaceIndexInFlight = null;
});
return this.spaceIndexInFlight;
}
/**
* Wrap a client method whose 404 means "the supplied spaceId is not accessible"
* and, ONLY on that 404, replace the opaque server text ("Space permissions not
* found") with a factual, actionable message naming the spaceId and the spaces
* the token can actually see (issue #534). A HINT layered on top of the
* backend, which stays authoritative so it FAILS OPEN on ANY uncertainty:
* every branch below that is not a confident "this spaceId is genuinely
* missing" rethrows the ORIGINAL server error unchanged. The happy path returns
* fn()'s value with zero extra requests.
*
* WRAP-ALLOWLIST INVARIANT (load-bearing read before wrapping a new method):
* among the currently wrapped tools a 404 comes ONLY from the spaceId
* membership / space-permissions check their pageId / rootPageId /
* parentPageId branches resolve to 403 or 200, NEVER 404. If a future change
* adds a `NotFoundException` to `/pages/tree`, `/pages/recent`,
* `/pages/sidebar-pages` or `/search` (e.g. "page not found"), this enrichment
* would MISATTRIBUTE that 404 to the spaceId. Re-audit the wrapped call before
* relying on this, and only wrap paths where the sole 404 cause is the space.
*/
protected async withSpaceAccessDiagnostics<T>(
spaceId: string,
mcpName: string,
fn: () => Promise<T>,
): Promise<T> {
try {
return await fn();
} catch (e) {
// Abort/cap wins FIRST and is detected by the SIGNAL FLAG, not e.name: a
// per-call cap may be an AbortSignal.timeout() (reason name "TimeoutError")
// or a custom reason, so `e.name === 'AbortError'` is NOT reliable (#534
// hole B). A stopped/capped turn must propagate its reason, never trigger a
// /spaces sweep or a rewrite.
if (this.toolAbortSignal?.aborted) throw e;
// Only a 404 is enrichable; any other status/shape is a different failure.
if (!(axios.isAxiosError(e) && e.response?.status === 404)) throw e;
let idx: AccessibleSpaceIndex;
try {
idx = await this.getAccessibleSpaceIndex();
} catch (fetchErr) {
// The /spaces sweep itself failed. If we were aborted mid-sweep,
// propagate the abort reason; otherwise FAIL OPEN with the ORIGINAL
// server error rather than a misleading "not found".
if (this.toolAbortSignal?.aborted) throw fetchErr;
if (process.env.DEBUG) {
console.error("space-diag: /spaces fetch failed:", fetchErr);
}
throw e;
}
// Fail open when the listing is incomplete (can't assert "missing") or when
// the spaceId IS present (the 404 is about something else, not the space).
if (!idx.complete) throw e;
if (idx.ids.has(spaceId)) throw e;
// Confident: the spaceId is well-formed but not among the accessible
// spaces. Replace the opaque server text with the actionable fact.
throw new Error(formatSpaceNotAccessible(mcpName, spaceId, idx.spaces));
}
}
/** Raw page info including the ProseMirror JSON content and slugId. */
async getPageRaw(pageId: string) {
/**
* Raw page info including the ProseMirror JSON content and slugId.
*
* With `format:"text"` (#502) the server instead renders `content` as a flat,
* deterministic text string (its `jsonToText` path the SAME serializer that
* feeds search), so the MCP text read reuses the server's ONE serializer
* rather than shipping a second one. Every other caller omits `format` and
* gets the JSON content unchanged.
*/
async getPageRaw(pageId: string, format?: "text") {
await this.ensureAuthenticated();
const response = await this.client.post("/pages/info", { pageId });
const body: Record<string, unknown> = { pageId };
if (format) body.format = format;
const response = await this.client.post("/pages/info", body);
return response.data?.data ?? response.data;
}
+54
View File
@@ -46,6 +46,60 @@ export function assertFullUuid(
}
}
// Max number of accessible spaces to enumerate inline in the "space not
// accessible" message (issue #534) before collapsing the rest into a "(+N ещё)"
// tail, so a workspace with many spaces cannot blow up the model context.
const SPACE_LIST_CAP = 10;
/**
* Compose the model-facing "spaceId is not accessible" message (issue #534).
* This is FACT text about the supplied spaceId that REPLACES the opaque server
* string ("Space permissions not found") on the enrich-on-404 path it names
* the exact bad id, lists the spaces the token can actually see (id + name, so
* the agent can copy the right id verbatim), and points at `listSpaces`.
*
* Deliberately Russian: like the other agent-facing tool guidance in this repo,
* this is the message the acting agent reads to self-correct.
*/
export function formatSpaceNotAccessible(
mcpName: string,
spaceId: string,
spaces: { id: string; name: string }[],
): string {
// No accessible spaces at all — a distinct diagnosis (token has no space
// access), not "you picked the wrong one from this list".
if (!Array.isArray(spaces) || spaces.length === 0) {
return `${mcpName}: spaceId "${spaceId}" недоступен, и доступных тебе спейсов нет — проверь доступ токена / вызови listSpaces.`;
}
const shown = spaces.slice(0, SPACE_LIST_CAP);
const remaining = spaces.length - shown.length;
const tail =
remaining > 0 ? ` (+${remaining} ещё, см. listSpaces)` : "";
// Cap the whole message at ERROR_MESSAGE_CAP (same budget as
// formatDocmostAxiosError) so ~10 long space names cannot blow up the model
// context. Truncate ONLY the interpolated space LIST — the fixed prefix (which
// carries the bad spaceId) and the fixed suffix (the "…из listSpaces"
// instruction) are always kept intact, so the actionable parts survive even
// when the list is trimmed.
const prefix = `${mcpName}: spaceId "${spaceId}" не найден среди доступных тебе спейсов. Доступные: `;
const suffix = ` — скопируй нужный id дословно из listSpaces.`;
let listed = shown.map((s) => `${s.id} (${s.name})`).join(", ") + tail;
const budget = ERROR_MESSAGE_CAP - prefix.length - suffix.length;
if (listed.length > budget) {
listed = listed.slice(0, Math.max(0, budget - 1)) + "…";
}
const message = `${prefix}${listed}${suffix}`;
// Unconditional final backstop (mirrors formatDocmostAxiosError): the list
// cap above assumes a well-formed prefix, but a pathologically long
// agent-supplied spaceId lives in the prefix and would otherwise blow past the
// budget. Cap the WHOLE message so nothing bloats the model context.
return message.length > ERROR_MESSAGE_CAP
? message.slice(0, ERROR_MESSAGE_CAP - 1) + "…"
: message;
}
// Keep ONLY the pathname of a request (no host, no query string, no fragment)
// so the message never leaks a host or query params. Resolves a relative
// config.url against config.baseURL, then discards everything but the path.
+7
View File
@@ -93,6 +93,13 @@ export function PagesMixin<TBase extends GConstructor<DocmostClientContext>>(Bas
const buildForm = () => {
const form = new FormData();
form.append("spaceId", spaceId);
// #502: this is an AGENT-authored body (plain prose / config), so tell the
// server import path to run the markdown importer with the two layered
// extensions OFF — a `$…$` span stays literal text (real math via
// `update_page_json`) and a schemeless `www.host`/email is not autolinked
// (an explicit `https://…` still links). A human file upload never sends
// this field, so human imports keep math + autolink ON.
form.append("disableMarkdownExtensions", "true");
form.append("file", fileContent, {
filename: `${title || "import"}.md`,
contentType: "text/markdown",
+98 -24
View File
@@ -72,13 +72,13 @@ export interface IReadMixin {
getTree(spaceId: string, rootPageId?: string, maxDepth?: number): any;
getPageContext(pageId: string): any;
listSidebarPages(spaceId: string, pageId?: string): any;
getPage(pageId: string): any;
getPage(pageId: string, format?: "markdown" | "text"): any;
getPageJson(pageId: string): any;
getOutline(pageId: string): any;
getNode(pageId: string, nodeId: string, format?: "markdown" | "json"): any;
searchInPage(pageId: string, query: string, opts?: SearchOptions): any;
getTable(pageId: string, tableRef: string): any;
search(query: string, spaceId?: string, limit?: number, opts?: { parentPageId?: string; titleOnly?: boolean }): any;
search(query: string, spaceId?: string, limit?: number, opts?: { parentPageId?: string; titleOnly?: boolean; offset?: number; match?: "auto" | "word" | "prefix" | "substring" }): any;
}
export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IReadMixin> & TBase {
@@ -132,17 +132,29 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
// BFS hit its node cap) had no way to know pages were missing. Return the
// tree alongside the flag; the primary /pages/tree path is uncapped so this
// is false there.
const { pages, truncated } = await this.enumerateSpacePages(spaceId);
return { tree: buildPageTree(pages), truncated };
// #534: spaceId is required here; wrap so a bad-spaceId 404 (from the
// /pages/tree seed inside enumerateSpacePages) becomes an actionable hint.
return this.withSpaceAccessDiagnostics(spaceId, "listPages", async () => {
const { pages, truncated } = await this.enumerateSpacePages(spaceId);
return { tree: buildPageTree(pages), truncated };
});
}
const clampedLimit = Math.max(1, Math.min(100, limit));
const payload: Record<string, any> = { limit: clampedLimit, page: 1 };
if (spaceId) payload.spaceId = spaceId;
const response = await this.client.post("/pages/recent", payload);
const data = response.data;
const items = data.data?.items || data.items || [];
return items.map((page: any) => filterPage(page));
// #534: only the WITH-spaceId recent path can 404 on space access; wrap it so
// that 404 is rewritten. Without a spaceId there is no space to diagnose, so
// the wrapper is inert (run the request directly).
const runRecent = async () => {
const response = await this.client.post("/pages/recent", payload);
const data = response.data;
const items = data.data?.items || data.items || [];
return items.map((page: any) => filterPage(page));
};
return spaceId
? this.withSpaceAccessDiagnostics(spaceId, "listPages", runRecent)
: runRecent();
}
/**
@@ -174,8 +186,16 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
"getTree: spaceId is required (a page tree is scoped to one space).",
);
}
const { pages } = await this.enumerateSpacePages(spaceId, rootPageId);
return buildPageTree(pages, { shape: "getTree", maxDepth });
// #534: the 404 for a bad/inaccessible spaceId surfaces from the
// `/pages/tree` seeding step inside enumerateSpacePages (which is NOT in a
// try/catch of its own for that case) — wrap the whole body so it is caught
// and rewritten into an actionable "spaceId not accessible" message. Only the
// space membership check can 404 here (rootPageId 403/200), see the
// wrap-allowlist invariant on withSpaceAccessDiagnostics.
return this.withSpaceAccessDiagnostics(spaceId, "getTree", async () => {
const { pages } = await this.enumerateSpacePages(spaceId, rootPageId);
return buildPageTree(pages, { shape: "getTree", maxDepth });
});
}
/**
@@ -420,8 +440,34 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
return convertProseMirrorToMarkdown(content, options);
}
async getPage(pageId: string) {
async getPage(pageId: string, format: "markdown" | "text" = "markdown") {
await this.ensureAuthenticated();
// #502 `format:"text"`: a flat, deterministic, machine-diffable rendering
// (block-per-line, inline marks/comment anchors dropped, non-text nodes ->
// stable placeholders like `[image]` / `[table RxC]`). The server produces
// it via its `jsonToText` path (the SAME serializer that feeds search), so
// there is no second serializer here — we just request it and pass the
// string through. Distinct from the markdown default: no PM->markdown walk,
// no markdown conversion cache, and no `{{SUBPAGES}}` substitution (the text
// renderer emits no such placeholder). Use it to diff a page you wrote as a
// config/prose against what was stored.
if (format === "text") {
const textData = await this.getPageRaw(pageId, "text");
let subpages: any[] = [];
try {
subpages = await this.listSidebarPages(textData.spaceId, textData.id);
} catch (e: any) {
console.warn("Failed to fetch subpages:", e);
}
const textContent =
typeof textData.content === "string" ? textData.content : "";
return {
data: filterPage(textData, textContent, subpages),
success: true,
};
}
const resultData = await this.getPageRaw(pageId);
// Agent read: hide resolved-comment anchors so the agent sees only active
@@ -681,13 +727,19 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
query: string,
spaceId?: string,
limit?: number,
opts: { parentPageId?: string; titleOnly?: boolean } = {},
opts: {
parentPageId?: string;
titleOnly?: boolean;
offset?: number;
match?: "auto" | "word" | "prefix" | "substring";
} = {},
) {
await this.ensureAuthenticated();
// Opt into the #443 agent-lookup mode: `substring: true` turns on the hybrid
// substring + FTS branch that returns path + snippet + score. A stock
// upstream server strips these unknown DTO fields (whitelist:true) and
// silently degrades to plain FTS — see the tool-registration comment.
// #529 unified engine: the query is parsed SERVER-SIDE (operators
// "phrase"/+/-, OR default, RU+EN morphology, match=auto). We forward the RAW
// query plus flags. `substring: true` is kept as a back-compat hint for a
// stock upstream server (whitelist:true strips the unknown #529 fields and it
// degrades to plain FTS — see the tool-registration comment).
const payload: Record<string, any> = {
query,
spaceId,
@@ -695,22 +747,44 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
};
if (opts.parentPageId) payload.parentPageId = opts.parentPageId;
if (opts.titleOnly) payload.titleOnly = true;
if (opts.match) payload.match = opts.match;
// Clamp an optional caller-supplied limit into the lookup range (1..50)
// before forwarding; omit it when not provided so the server default applies.
if (limit !== undefined) {
payload.limit = Math.max(1, Math.min(50, limit));
}
const response = await this.client.post("/search", payload);
// Normalize both response shapes: bare array and paginated { items: [...] }
const data = response.data?.data;
const items = Array.isArray(data) ? data : data?.items || [];
const filteredItems = items.map((item: any) => filterSearchResult(item));
if (opts.offset !== undefined) {
payload.offset = Math.max(0, Math.floor(opts.offset));
}
return {
items: filteredItems,
success: response.data?.success || false,
const runSearch = async () => {
const response = await this.client.post("/search", payload);
// Normalize both response shapes: bare array and paginated { items: [...] }.
const data = response.data?.data;
const items = Array.isArray(data) ? data : data?.items || [];
const filteredItems = items.map((item: any) => filterSearchResult(item));
// Surface the #529 pagination envelope when present (a stock upstream has
// none — the fields are simply undefined and the caller sees just `items`).
const envelope = Array.isArray(data) ? undefined : data;
return {
items: filteredItems,
total: envelope?.total,
hasMore: envelope?.hasMore,
truncatedAtCap: envelope?.truncatedAtCap,
offset: envelope?.offset,
success: response.data?.success || false,
};
};
// #534: a search scoped to a spaceId 404s when that space is inaccessible;
// wrap only that case so the 404 becomes an actionable hint. A workspace-wide
// search (no spaceId) has no space to diagnose — run it directly (inert).
return spaceId
? this.withSpaceAccessDiagnostics(spaceId, "search", runSearch)
: runSearch();
}
}
+31 -9
View File
@@ -450,15 +450,22 @@ server.registerTool(
"search",
{
description:
"Find pages by a fragment of a technical string (hostnames, IPs, IDs " +
"like `srv.local`, `10.0.12`, `WB-MGE-30D86B`) — one call returns each " +
"hit's location (`path`: ancestor titles root→parent) and a `snippet` " +
"around the first match, so you rarely need a follow-up get_page. " +
"Matches substrings literally (dots/dashes/digits are not tokenized) as " +
"well as full-text. Returns `{ pageId, title, path, snippet, score }` " +
"sorted by `score` (a per-response relevance float).",
"Search pages across the wiki. OR by default with relevance ranking " +
"(RU+EN morphology): multi-word queries match ANY term, not all. " +
"Operators: \"exact phrase\" (adjacent words), +term (require), -term " +
"(exclude) — e.g. `+кофейня -архив`, `+\"воздушный шар\" кофе`. A leading " +
"-/+ is the operator; -,.,: INSIDE a token are literal (`WB-MGE-30D86B`, " +
"`10.0.12.5` stay one term). Technical fragments (hostnames, IPs, IDs) " +
"auto-match as substrings; words use full-text. Each hit returns its " +
"location (`path`: ancestor titles root→parent), a `snippet`, `score`, " +
"`matchedTerms` and `matchedFields`, so you rarely need a follow-up " +
"getPage. Paginate with limit + offset; the response carries " +
"`total` (exact, permission-filtered), `hasMore` and `truncatedAtCap`. " +
"NOTE: results past the relevance cap (~500) are unreachable by " +
"pagination — narrow the query (add terms / +required / a spaceId) " +
"instead when `truncatedAtCap` is true.",
inputSchema: {
query: z.string().min(1).describe("Search query"),
query: z.string().min(1).describe("Search query (supports \"phrase\", +require, -exclude)"),
spaceId: z
.string()
.optional()
@@ -473,6 +480,13 @@ server.registerTool(
.boolean()
.optional()
.describe("Match page titles only; skip page text"),
match: z
.enum(["auto", "word", "prefix", "substring"])
.optional()
.describe(
"Match mode (default auto: identifiers→substring, words→full-text). " +
"Override with word/prefix/substring.",
),
limit: z
.number()
.int()
@@ -480,12 +494,20 @@ server.registerTool(
.max(50)
.optional()
.describe("Max results to return (1-50, default 10)"),
offset: z
.number()
.int()
.min(0)
.optional()
.describe("Pagination offset (default 0); use with total/hasMore"),
},
},
async ({ query, spaceId, parentPageId, titleOnly, limit }) => {
async ({ query, spaceId, parentPageId, titleOnly, match, limit, offset }) => {
const result = await docmostClient.search(query, spaceId, limit, {
parentPageId,
titleOnly,
match,
offset,
});
return jsonContent(result);
},
+28 -2
View File
@@ -14,6 +14,7 @@ import {
markdownToProseMirror,
normalizeAgentMarkdown,
} from "@docmost/prosemirror-markdown";
import type { MarkdownImportOptions } from "@docmost/prosemirror-markdown";
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import type { PageId } from "./page-id.js";
@@ -111,16 +112,31 @@ global.WebSocket = WebSocket;
* horizontalRule the serializer emitted, and stripping it would silently drop the
* page's leading content (#493 review). The front-matter strip stays on the
* server FILE-import boundary only (`normalizeForeignMarkdown`).
*
* #502 IMPORTANT the two layered markdown extensions (`$$` math, schemeless
* fuzzy autolink) are NOT decided here; they are the CALLER's choice via
* `options`, because the two callers of this wrapper need OPPOSITE behavior:
* - AGENT-authored plain markdown (`updatePageMarkdown`) both OFF, so a
* `$$` config span stays literal and a bare `www.host` is not autolinked
* (real math is authored via `update_page_json`).
* - FULL-FILE round-trip import (`import_page_markdown`, #328 lossless)
* DEFAULTS (both ON), because the exporter serializes a math node as readable
* `$x^2$`; re-importing with math OFF would degrade it to literal text and
* BREAK the lossless exportimport pair.
* So `options` DEFAULTS to `undefined` the package importer's defaults (ON),
* which is the safe round-trip behavior; the agent-write caller opts OUT
* explicitly. (The fragment path `importMarkdownFragment` opts out on its own.)
*/
export async function markdownToProseMirrorCanonical(
markdownContent: string,
options?: MarkdownImportOptions,
): Promise<any> {
// #419: normalize + merge glyph-forked footnote definitions BEFORE
// canonicalizing, so the canonicalizer re-hangs references and drops the
// now-orphaned duplicate definitions.
return canonicalizeFootnotes(
normalizeAndMergeFootnotes(
await markdownToProseMirror(normalizeAgentMarkdown(markdownContent)),
await markdownToProseMirror(normalizeAgentMarkdown(markdownContent), options),
),
);
}
@@ -358,7 +374,17 @@ export async function updatePageContentRealtime(
): Promise<MutationResult> {
// PAGE write: canonicalize footnotes (markdown import builds the bottom list in
// definition order; numbering is reference-ordered).
const tiptapJson = await markdownToProseMirrorCanonical(markdownContent);
//
// #502: this is the AGENT-authored `updatePageMarkdown` body — plain prose /
// config — so the two layered markdown extensions are turned OFF: a `$…$` span
// stays literal text (real math via `update_page_json`) and a SCHEMELESS
// `www.host`/email is not autolinked (an explicit `https://…` still links).
// Contrast `import_page_markdown`, which keeps DEFAULTS for the #328 lossless
// round-trip.
const tiptapJson = await markdownToProseMirrorCanonical(markdownContent, {
parseMath: false,
fuzzyLinkify: false,
});
return await mutatePageContent(
pageId,
collabToken,
+38 -3
View File
@@ -1080,8 +1080,43 @@ export interface PreparedModel {
*/
export function prepareModel(inputXml: string): PreparedModel {
const rawModel = normalizeInput(inputXml);
const { cells, warnings } = lintModel(rawModel);
const modelXml = normalizeXml(rawModel);
// Auto-strip XML comments before linting. The model routinely inserts
// `<!-- ... -->` into the diagram XML despite the prohibition in the tool
// description; a hard [no-comments] lint failure would force it to regenerate
// the whole diagram (a wasted tool call). Comments carry no diagram semantics,
// so stripping them is always safe. The linter's no-comments rule stays as a
// defense-in-depth backstop for any path that reaches it without this strip.
//
// The strip is GATED on two conditions so the regex only ever targets real
// comment nodes:
// 1. Well-formedness. Well-formed XML forbids a bare `<` inside an attribute
// value (it must be entity-escaped, e.g. `&lt;!--`), so a literal `<!--`
// cannot hide in a value. On MALFORMED input (e.g. a raw unescaped `<!--`
// inside a value on the <mxGraphModel> create/update path, which
// normalizeInput passes through unparsed) we must NOT strip: truncating
// that value would silently corrupt the author's label. We leave it intact
// and let lintModel's value-escaping / well-formed-xml rules reject it so
// the author sees the real error.
// 2. No CDATA. A CDATA section is well-formed yet its text may contain a
// literal `<!-- ... -->` that is NOT a comment node; stripping it would
// silently delete author content. Real drawio labels live in `value=`
// attributes (CDATA impossible), so excluding CDATA is free on legitimate
// models; a CDATA-bearing model with a genuine comment then falls to the
// retained no-comments lint backstop (an explicit error) rather than being
// silently corrupted.
const { error: parseError } = parseXml(rawModel);
const hasCdata = rawModel.includes("<![CDATA[");
const commentMatches =
parseError === null && !hasCdata
? (rawModel.match(/<!--[\s\S]*?-->/g) ?? [])
: [];
const commentCount = commentMatches.length;
const strippedModel =
commentCount > 0 ? rawModel.replace(/<!--[\s\S]*?-->/g, "") : rawModel;
const stripWarnings =
commentCount > 0 ? [`stripped ${commentCount} XML comment(s)`] : [];
const { cells, warnings } = lintModel(strippedModel);
const modelXml = normalizeXml(strippedModel);
const bbox = computeBBox(cells);
const cellCount = cells.filter((c) => c.id !== "0" && c.id !== "1").length;
// Geometry quality warnings (non-blocking) are appended to any structural
@@ -1092,7 +1127,7 @@ export function prepareModel(inputXml: string): PreparedModel {
cells,
bbox,
cellCount,
warnings: [...warnings, ...quality],
warnings: [...stripWarnings, ...warnings, ...quality],
hash: mxHash(modelXml),
};
}
+11 -1
View File
@@ -135,7 +135,17 @@ export interface MarkdownFragment {
export async function importMarkdownFragment(
markdown: string,
): Promise<MarkdownFragment> {
const doc = await markdownToProseMirror(markdown);
// #502: the fragment path is an MCP agent WRITE (patch_node/insert_node
// markdown), so it uses the SAME extensions-OFF importer options as the
// full-page write (markdownToProseMirrorCanonical): a `$…$` span stays literal
// and a schemeless domain/email is not autolinked. This keeps a block written
// via markdown canonically identical to the same content in a full-page write
// (no "second canon"). Explicit `https://…` links and block structure are
// unaffected.
const doc = await markdownToProseMirror(markdown, {
parseMath: false,
fuzzyLinkify: false,
});
const content: any[] = Array.isArray(doc?.content) ? doc.content : [];
const blocks: any[] = [];

Some files were not shown because too many files have changed in this diff Show More