Compare commits

...

61 Commits

Author SHA1 Message Date
agent_coder fbac8e6976 feat(share): Stage B — "approved" publish-only-the-saved-version mode (#370)
A share can now be published in 'approved' mode: public readers by URL see
the last manually-saved version (page_history.kind='manual') instead of the
live draft. The content swap lives in the single canonical resolver
resolveReadableSharePage, so EVERY public read path is frozen consistently —
the shared-page view, the SEO controller, AND the share assistant's page-read
tool (owner decision: a visitor and the assistant see the same bytes). Swaps
only content+title; page.id/workspaceId stay LIVE so attachment tokens keep
minting for the right owner. MVP residue kept live: the assistant's page
LIST/TREE tools and transclusions.

- migration 20260712T130000: shares.published_mode varchar(20) NOT NULL
  DEFAULT 'live' (additive, standalone; existing shares stay live)
- db.d.ts: publishedMode on Shares (Share/Insertable/Updatable derive)
- page-history.repo: findLatestByPageIdAndKind(pageId, kind, {includeContent})
- share.repo: thread publishedMode through baseFields/insert/update
- share.service.getShareForPage: 4-point CTE threading (anchor + union +
  returned object + repo allowlist) — type-guarded, so dropping any point
  fails the build/tests
- createShare/updateShare: accept publishedMode; reject approved+includeSubPages
  (BadRequestException); auto-mint the first manual baseline ON ENABLE
- share.dto: publishedMode @IsOptional @IsIn(['live','approved'])
- client: "Publish only the saved version" toggle (XOR with sub-pages),
  version caption + "unsaved changes newer than the published version" hint;
  share types/api carry publishedMode
- tests: resolve-swap (content+title swapped, id+workspaceId preserved),
  assistant getSharePage sees frozen content, getShareForPage threading
  (direct + descendant), baseline auto-create, XOR reject, migration up/down

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:08:50 +03:00
vvzvlad ec8dd7d110 Merge pull request 'fix(prosemirror-markdown): литеральный <br> в тексте экранируется, не становится hardBreak (#554)' (#560) from fix/554-literal-br into develop
Reviewed-on: #560
2026-07-12 18:57:02 +03:00
vvzvlad 4423b19850 Merge pull request 'fix(ai-chat): реактивная рекавери — итеративная эскалация 0.5^k + пол вместо фикс. 0.5× (#520)' (#546) from fix/520-recovery-escalation into develop
Reviewed-on: #546
2026-07-12 18:56:49 +03:00
agent_coder 688cb54f26 fix(ai-chat): рекавери режет реплей НИЖЕ настроенного бюджета (#520 Опция B)
Решение владельца по #520 (эпик #497 итерация 5): при повторном overflow реактивная
рекавери ДОЛЖНА иметь право резать бюджет реплея ниже явно настроенного окна —
«чат не должен кирпичиться на overflow» это ИНВАРИАНТ реактивной ветки, а
настроенное окно — заявление о ёмкости модели, не обещание о размере реплея; когда
провайдер 400-ит, реальность бьёт конфиг. (Не конфликтует с #510 Опция A: та про
верхнюю границу НОРМАЛЬНОГО пути — уважать конфиг, пока он влезает.)

- Пол эскалации: max(scaled, min(REPLAY_MIN_FLOOR_TOKENS, floor(0.5×threshold))).
  Было min(REPLAY_MIN_FLOOR_TOKENS, threshold) — жёсткий пол на настроенном бюджете.
  Теперь БОЛЬШОЕ окно эскалирует НИЖЕ настроенного бюджета до фикс. пола 8k; МАЛОЕ
  окно (0.5×threshold < 8k) падает до floor(0.5×threshold) — минимум старый 0.5× cut
  (никогда не хуже, чем раньше), но и не поднимается выше самого бюджета.
  REPLAY_MIN_FLOOR_TOKENS=8k без изменений (константа уже была). Сброс k на чистом
  ходу сохранён (счётчик replayOverflowCount не пишется на чистом финализе).
- Наблюдаемость: warn-лог (настроенный бюджет не влезает, реплей ниже него, уровень
  k) + метка турна metadata.replayBelowConfiguredBudget=true — только когда реплей
  реально ниже настроенного бюджета (не на нормальном in-budget реплее).
- Тесты: перепиновка «не раздувает малый бюджет выше себя» (теперь МОЖЕТ резать ниже,
  Опция B) в обоих spec; новый тест эскалации ниже бюджета (окно 8000→бюджет 5600,
  k=1→2800, сходимость к полу, сброс на чистом ходу) + большое окно (140k→8k
  ступенями); тест метки метадаты. Мутации: пол=бюджет краснит below-budget тесты,
  снятие метки краснит observability-тест.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 18:01:07 +03:00
agent_coder ec5416068b fix(ai-chat): итеративная эскалация реактивной рекавери при overflow (#520)
Остаточный кирпич (#490/#510): при незаданном chatContextWindow база = плоский
дефолт 100k, а реальное окно модели маленькое (<50k). Фиксированный одинарный
cut 0.5×100k=50k всё равно превышал реальное окно → провайдер снова 400,
строка переставлялась replayOverflow, но булев priorOverflowed уже был true →
второго ужатия не происходило. Чат навсегда застревал на 50k и не восстанавливался.

Фикс (без парсинга тел 400):
- Сигнал из булева переведён в счётчик `metadata.replayOverflowCount` = число
  ПОДРЯД идущих overflow-ходов: инкремент (prior+1) на каждом overflow, сброс в 0
  на любом чистом финализе (чистая строка не пишет поле → читается как 0).
  BACK-COMPAT: старая строка с булевым `replayOverflow:true` читается как k=1.
- resolveEffectiveReplayThreshold(threshold, k) = max(floor(threshold·0.5**k),
  min(REPLAY_MIN_FLOOR_TOKENS, threshold)). k=0 → база; k=1 → 0.5×; k=2 → 0.25×;
  большой k → упирается в пол 8k (сходимость). null-база (trimming OFF) не трогается;
  пол никогда не поднимает легитимно малый настроенный бюджет выше него самого.
- Пол REPLAY_MIN_FLOOR_TOKENS=8k: ниже него чат не несёт осмысленный недавний
  контекст, и даже малое реальное окно его вмещает; keep-recent-turns сверху.

Тесты: таблица эскалации (k=0/1/2/большой/null), регрессия остаточного кирпича
(база 100k, окно ~40k → сходится ниже 40k, чего фиксированный 0.5× никогда не мог),
жизненный цикл счётчика на реальном pg (инкремент/сброс/back-compat через jsonb),
мутация **k→**1 краснит convergence-тест.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 18:01:07 +03:00
agent_coder 2c4fc565b6 fix(prosemirror-markdown): escape literal <br> in text so it round-trips as text, not a hardBreak (#554)
A literal inline-HTML break tag typed as prose text (`<br>`, `<br/>`,
`<br />`) was emitted verbatim into markdown, so on re-import marked
parsed it as an inline-HTML line break and silently turned the user's
text into a hardBreak node, dropping the text. HTML-entity-encode only
the angle brackets of a break-tag sequence in `case "text"` so it lands
as `&lt;br&gt;` and the importer decodes it back to literal `<br>`.

Scoped strictly to the `<br…>` pattern (not every `<`/`>`), so stray
angle brackets in prose (`a < b > c`) are untouched, and to the
text-content path only — a real hardBreak serializes from its own case
(`  \n`, or `<br>` via inlineToHtml), so the serializer's own emitted
breaks are never escaped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:58:21 +03:00
vvzvlad a990ebd604 Merge pull request 'feat(search) Фаза A: лексический оверхол — ru_en, OR/RRF, операторы, пагинация (#529)' (#538) from feat/529-search-lexical into develop
Reviewed-on: #538
2026-07-12 17:40:43 +03:00
vvzvlad 94ca907476 Merge pull request 'fix(prosemirror-markdown): hardBreak переживает round-trip (trailing/consecutive/table-cell) (#549)' (#553) from fix/549-hardbreak-md-canon into develop
Reviewed-on: #553
2026-07-12 17:40:18 +03:00
agent_coder fff772cbe2 fix(search): F1 term-cap stack-depth guard + restore/extend coverage, drop dead code (#529)
F1: cap parsed terms at MAX_PARSED_TERMS (64) in parseSearchQuery so a huge
pasted query no longer nests the combined tsquery deep enough to blow Postgres'
stack depth limit (HTTP 500); +@MaxLength(10000) on SearchDTO.query as
defense-in-depth. F2/F3: restore titleOnly text_content leak-guard and
parentPageId subtree-scoping coverage in the lexical int-spec. F4: positive
ancestor-path ordering + non-empty snippet asserts. F5: remove dead
SearchLookupTier enum, unused buildTsQuery + pg-tsquery require (and its tests),
and the discarded parentPageId select in fetchDetails. F6: rewrite the stale
#443 DTO comment (parentPageId/titleOnly read by the engine; substring ignored).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 16c2a4b623 fix(search): whitelist #529 search response-fields in the #494 prose drift-guard
The rebase pulled develop's #494 REVERSE prose drift-guard, which flags every
camelCase token in ROUTING_PROSE that is not a registered MCP tool. #529's search
routing prose documents the search RESPONSE fields matchedTerms/matchedFields/
hasMore/truncatedAtCap — genuine non-tool terms — so add them to
PROSE_NON_TOOL_TERMS (develop's designed allowlist). Reconciles both invariants:
#529's search-prose enforcement and #494's dead-reference guard both stay green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 58f13a7efd chore(search): retimestamp ru_en migration to 20260707T130000 to avoid collision (#529)
Develop introduced two migrations at the exact 20260707T120000 stamp
(ai-chat-metadata, page-history-kind). Three files sharing one timestamp
is fragile for deploy ordering, so move #529's ru_en-config migration to a
strictly-later, unique 20260707T130000 so it orders cleanly last. It touches
only the ru_en text-search config (different tables), so running it after the
develop migrations is order-independent and safe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 2bb48f841f fix(search): S1/W1/W3/W4/S3/S5/S2 — required-term details, CAP note, restored guards (#529)
S1: fetchDetails now builds rank/highlight/matchedFields from the same
[...positive, ...required] set matchedTerms uses, so a required-only query
(`+кофейня`) no longer reports matchedFields:[] / rank:null.

W1: documented that CANDIDATE_CAP bounds pagination reachability (JS-side), not
query compute — a SQL LIMIT would corrupt the exact permission-filtered total.

S2: one-line note that literal-`%`/`_` search was intentionally dropped (total:0).

W3: restored the trgm-index EXPLAIN guard (search-lookup-explain.int-spec) that
asserts the coalesce-free substring predicates use idx_pages_*_trgm, not Seq Scan.

W4: added an integration ordering test proving title-exact > title-substring >
text-only tier dominance under RRF.

S3: added a test that an exact-title hit survives a tiny CANDIDATE_CAP window.

S5: server pretest now also builds @docmost/mcp so the ToolWriteClass import
resolves from clean CI (mirrors editor-ext/prosemirror-markdown).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder daa96eb132 fix(search): B1/W2 — honest page_embeddings.fts swap + true idempotency (#529)
BLOCKER B1: the fts config swap no longer blindly DROP+ADDs the generated
column. swapEmbeddingsFtsConfig now (1) reads the column's actual generation
expression from pg_catalog and TRUE-no-ops when it already references the target
config (real out-of-band escape hatch), and (2) gates the inline ACCESS
EXCLUSIVE rewrite behind SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE (default 'true';
only a literal 'false' opts out), warning that the operator owns the swap.
up() now creates ru_en only when missing (drop-recreate would fail on the fts
hard dependency on a re-run), so up() is genuinely idempotent. down() guards the
config DROP when fts still references ru_en (gated-off path), staying non-fatal.

W2: corrected the header — Kysely runs EACH migration in its own transaction
(not the whole set in ONE); the single-UPDATE atomicity conclusion is unchanged.
Rewrote the runbook: inline path is a full-table ACCESS EXCLUSIVE rewrite of
page_embeddings; removed the false "IF-EXISTS no-op" claim.

Extends the roundtrip test with idempotent-skip (2nd up() no-op) and env-gate
=false (embeddings rewrite skipped, pages.tsv still swapped) assertions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 4c5230d677 test(search): A1 migration down()/up() reversal roundtrip (#529)
Asserts the ru_en migration is reversible in the correct order: down()
reverts pages.tsv trigger + page_embeddings.fts to english BEFORE dropping
the ru_en config, and up() re-applies it idempotently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 50ca27c5d4 feat(search): A9 web-UI — superset types, match/pagination params, operator hint (#529)
- search.types.ts: IPageSearch is the #529 superset (pageId/snippet/score/
  path/matchedFields/matchedTerms; rank/highlight nullable); add the
  IPageSearchResponse pagination envelope and match/limit/offset params.
- search-spotlight.tsx: operator hint ("exact phrase", +required, -excluded).

The web list already reads response.items, so it picks up the new OR order +
superset with no code change; icon/space/highlight remain (acceptance #12).
Full spotlight pagination UI is deferred within this PR — the API + MCP fully
support total/hasMore/offset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder de25c258f9 feat(search): A9 MCP — operators, match, pagination in the search tool (#529)
- client/read.ts search(): forward match + offset; surface the pagination
  envelope (total/hasMore/truncatedAtCap/offset) when the #529 server
  returns it (a stock upstream leaves them undefined).
- index.ts search tool: document OR-default + RU/EN morphology, the
  "phrase"/+/- operators, match modes, limit+offset pagination and the
  relevance-CAP unreachable-tail caveat; add match + offset params.
- server-instructions.ts: READ prose + inventory line updated to the new
  contract, enforced by a new tool-inventory.test.mjs assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
vvzvlad afc50ead38 Merge pull request 'feat(deploy): version-coherence — WS-анонс версии + guarded reload устаревших вкладок (#481)' (#499) from feat/481-version-coherence into develop
Reviewed-on: #499
2026-07-12 17:37:04 +03:00
vvzvlad ec932e89a3 Merge pull request 'feat(page-history): время работы над статьёй — число + суточный punch-card (#395)' (#551) from feat/395-work-time into develop
Reviewed-on: #551
2026-07-12 17:36:27 +03:00
vvzvlad 490d7965a1 Merge pull request 'feat(comment): Undo в тосте резолва (reopen) + сортировка Resolved по времени резолва (#542)' (#548) from feat/542-resolve-undo into develop
Reviewed-on: #548
2026-07-12 17:36:08 +03:00
vvzvlad 32c9361969 Merge pull request 'feat(mcp): буквальная запись markdown (без math/autolink) + getPage format:text (#502)' (#552) from feat/502-mcp-md-modes into develop
Reviewed-on: #552
2026-07-12 17:35:36 +03:00
vvzvlad c30f910d66 Merge pull request 'fix(prosemirror-markdown): markdown-импорт помечает внутренние ссылки на страницы как internal (#522)' (#524) from fix/522-internal-links into develop
Reviewed-on: #524
2026-07-12 17:35:28 +03:00
agent_coder 7007f6bcf9 feat(search): A2-A9 — unified OR-relevance engine, RRF, pagination (#529)
Rewrite SearchService.searchPage into ONE engine for web-UI, MCP agent and
share:

- A2 server-side query parser (search-query-parser.ts): quote-aware
  tokenizer, leading +/- operators (internal -,.,: literal), "phrase",
  metachar-stripped bare terms; tsquery built as a parameterized AST via
  SQL ||/&&/!! (never string-concat). only-negation/empty short-circuit.
- A3 match=auto routes identifier-like terms (10.31.41, esp32,
  WB-MGE-30D86B) to the substring/trigram branch, words to FTS; word/
  prefix/substring overrides.
- A4 RRF (k=60) fuses the FTS branch (ts_rank_cd) and substring branch
  (title-exact>title-sub>text tier) by RANK; ORDER BY rrf DESC, id.
- A5 exact permission-filtered total (fail-closed via filterAccessiblePageIds
  with the #348 hasRestricted fast-path), CANDIDATE_CAP fusion window,
  offset/limit, hasMore, truncatedAtCap.
- A6 single path (spaceId/share/creatorId/titleOnly/parentPageId/match);
  share uses getPageAndDescendantsExcludingRestricted.
- A7 response superset per hit (id/pageId/slugId/icon/title/space/…/rank/
  highlight/snippet/path/score/matchedFields/matchedTerms).
- A8 buildAncestorPaths now skips deleted + cross-space ancestors.
- A9 DTOs: match, offset, total/hasMore/truncatedAtCap/query/matchedFields.

SEARCH_MODE=or|and toggles the parser; SEARCH_CANDIDATE_CAP tunes the
window. Legacy lookup unit/int specs replaced by parser unit tests + a
13-criteria integration spec on real pg (incl. a permission mutation guard
and a fail-closed propagation test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:35:24 +03:00
agent_coder 4bf06c68cf feat(search): A1 — ru_en text-search configuration + config swap (#529)
Introduce a `ru_en` FTS configuration (english_stem over ascii token
classes, russian_stem over Cyrillic ones) and flip every stored + query
side to it IN LOCKSTEP (acceptance #13):

- new migration creates ru_en, swaps the pages.tsv trigger + reindexes
  existing rows (row-lock UPDATE, no ACCESS EXCLUSIVE), and swaps the
  page_embeddings.fts generated column (documented rewrite/lock trade-off
  for large tenants, mirroring the #443 trgm migration). down() reverts
  tsv/fts to english BEFORE dropping the config (dependency order).
- page-embedding.repo.ts hybridSearch query config english -> ru_en, so
  the RAG lexical leg's query config matches its fts column config.
- search.service.ts current query literals english -> ru_en so the column
  and query configs stay paired (this commit is independently revertable;
  the engine itself is rewritten in the A2-A9 commit).

The reindex is atomic within the single migration transaction (this repo's
Migrator wraps all pending migrations in one tx), so no morphology-desync
window exists and no dual-config read path is needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:35:24 +03:00
vvzvlad af44736fb9 Merge pull request 'fix(mcp): drawio — авто-стрип XML-комментариев (гейт на well-formedness) вместо lint-ошибки (#505)' (#545) from fix/505-drawio-strip-comments into develop
Reviewed-on: #545
2026-07-12 17:35:01 +03:00
vvzvlad 2a4d1acfd7 Merge pull request 'fix(prosemirror-markdown): insertNode дописывает пункт в существующий список вместо второго списка (#535)' (#537) from fix/535-list-seam-coalesce into develop
Reviewed-on: #537
2026-07-12 17:34:41 +03:00
vvzvlad 11eb87d58a Merge pull request 'docs(server): getPageBreadCrumbs — предки не утекают (нисходящее наследование ограничений), задокументировано (#471)' (#550) from docs/471-breadcrumb-permission into develop
Reviewed-on: #550
2026-07-12 17:33:55 +03:00
vvzvlad f2c8aa70f3 Merge pull request 'fix(mcp): внятная ошибка на недоступный spaceId вместо «Space permissions not found» (#534)' (#536) from fix/534-mcp-spaceid into develop
Reviewed-on: #536
2026-07-12 17:33:39 +03:00
vvzvlad 661ea8ba07 Merge pull request 'fix(client): reconnect-вход — таймаут-гонка вокруг getRun, hang не залипает FSM в streaming (#541)' (#543) from fix/541-reconnect-timeout into develop
Reviewed-on: #543
2026-07-12 17:32:07 +03:00
vvzvlad 059edccb64 Merge pull request 'test(mcp): hardBreak уже представим в md-каноне (#293) — регресс-гард, закрытие #503' (#544) from fix/503-hardbreak-canon into develop
Reviewed-on: #544
2026-07-12 17:31:55 +03:00
vvzvlad 693a9a350b Merge pull request 'perf(ai-chat) волна C: инкрементальный рендер стрима + append-персист таблица шагов (#492)' (#547) from feat/492-incremental-render into develop
Reviewed-on: #547
2026-07-12 17:31:40 +03:00
agent_coder 79b2da686b fix(work-time): clip cross-class padding; gate & payload fixes (#395)
F1: pad-clip adjacent different-class sessions at the raw-gap midpoint so
workMs and agentOnlyMs can never double-count the same wall-clock (default
agentTGap 7m < pIn+pOut 10m made a work-session-ending-in-agent overlap a
nearby agent_only run). Reword the config guard/comment: cross-class
disjointness is now structural, tGap≥pIn+pOut is a kept sanity bound. Add a
cross-class no-double-count test and a real seeded property/fuzz test (250
random timelines × 4 tz) asserting per-class union, cross-class disjointness,
and Σ per-day activeMs == workMs.

F2: page.controller.spec — add a ForbiddenException view-gate reject test that
asserts the rejection propagates and computeWorkTime is NOT reached, locking
validateCanView before compute.

F3: WorkTimeStat renders for agent-only pages (workMs==0, agentOnlyMs>0) with
an `agent:` headline so the punch-card stays reachable.

F4: drop the dead un-bucketed `sessions` list from the PageWorkTime response
and the client work-time types (no client reads it; bucketByDay still consumes
the core sessions internally).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 08:31:42 +03:00
agent_coder ed3a8f8174 docs(mcp): sync write-tool specs with #502 literal-markdown behavior (#502 review)
The four MCP markdown-write tools (updatePageMarkdown/createPage/patchNode/
insertNode) still described the old parse-everything behavior after #502 turned
TeX math and fuzzy autolink OFF on the write paths. Add a concise contract hint
to each: $...$ / $$...$$ stay literal (not a math formula) and schemeless
www.host / bare emails are not auto-linked (explicit https:// still links); for
a real formula use updatePageJson (or a mathInline/mathBlock node via `node` on
patch/insert). Also note getPage format:"text" in README.md/README.ru.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:57:26 +03:00
agent_coder 8503ff1f3d fix(ai-chat): hydrate crashed mid-run steps for model replay + cover write/read seams (#492)
F1: the server model-replay loaded history via findAllByChat().map(rowToUiMessage)
WITHOUT hydrating parts from ai_chat_run_steps. A HARD crash mid-run (SIGKILL/OOM)
fires no terminal callback, so the assistant row stays parts:[] and its partial
tool-calls/results/text (durable in the steps table) dropped out of the model's
next-turn context. Hydrate needy assistant rows (role==='assistant' &&
!rowHasInlineParts) via findByMessageIds + hydrateAssistantParts before the replay
map — mirroring the controller's withReconstructedParts exactly — guarded on the
optional repo. Fix the now-false interrupt-resume comment.

F2: add a service int-spec that drives the REAL onStep append-persist WRITE branch
through AiChatService.stream with a real AiChatRunStepRepo injected, asserting the
per-step rows' stepIndex + parts slice and the step-marker metadata match a
single-row flush (catches an stepsPersisted-1 off-by-one).

F3: add a controller int-spec that drives withReconstructedParts through getMessages
WITH the repo present (a mid-run marker-only row + its step rows), asserting the
reconstructed metadata.parts and workspace-scoping.

F4: remove the dead countByMessage (zero prod callers; reconstructRunParts derives
stepsPersisted inline) + its now-unused sql import and the redundant test assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:50:50 +03:00
agent_coder c18b4c132c fix(md-canon): preserve hardBreak in trailing/consecutive/table-cell round-trips (#549)
The markdown canon lost `hardBreak` nodes in three pm->md->pm sub-cases where
the two-space `  \n` form does not survive re-import via marked:

1. Trailing at end of a paragraph — the top-level `.trim()` strips the trailing
   `  \n`, and even a non-last paragraph's trailing break sits before the `\n\n`
   block gap, so marked drops it.
2. Consecutive breaks (`a<hardBreak><hardBreak>b`) — the whitespace-only middle
   line reads as a paragraph separator, splitting the run into two paragraphs.
3. Inside a GFM table cell — the single-line cell collapses the break to a space.

Emit `<br>` (inline HTML break) in exactly those positions: marked passes it
through and generateJSON rebuilds a hardBreak in every context, including table
cells. renderInlineChildren now picks `<br>` for a break that is the last inline
child or is followed by another break; a safe mid-paragraph break keeps the
byte-stable `  \n` form (golden output unchanged). Footnote bodies are skipped
(their `^[…]` form already collapses breaks to spaces). The table-cell case
converts the `  \n` marker to `<br>` before the newline collapse.

Adds pm->md->pm count/position round-trip tests for all three sub-cases plus a
mid-paragraph regression guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:10:40 +03:00
agent_coder 3163f50c98 fix(mcp): route #502 extension flags per write tool; add server import flag (#502 review)
Internal review found the two layered-extension flags landed in the SHARED
markdownToProseMirrorCanonical wrapper, which mis-covered two tools. Move the
decision to each caller by the tool's semantics.

BLOCKER 1 — createPage was NOT covered. createPage POSTs to the server
/pages/import endpoint, which imported with DEFAULTS (math + fuzzy autolink ON),
so an agent's `$x=1$` still became a formula and `www.host` still autolinked.
Add an optional `disableMarkdownExtensions` multipart field to /pages/import
(default OFF, so human file uploads keep math ON); import.controller reads it,
import.service.importPage/processMarkdown thread it into markdownToProseMirror.
MCP createPage sends it true.

BLOCKER 2 — import_page_markdown was wrongly disabled. It goes through the same
wrapper, so hardcoding parseMath:false degraded an exported `$x^2$` to literal
text on re-import, breaking the #328 lossless export→import pair. The wrapper no
longer hardcodes the flags: it takes them from the caller and DEFAULTS to the
package importer defaults (extensions ON). Callers now set them explicitly:
  - updatePageMarkdown (updatePageContentRealtime) -> OFF
  - patch_node/insert_node (importMarkdownFragment) -> OFF (unchanged)
  - import_page_markdown -> DEFAULTS (math ON) — #328 round-trip restored

Tests: rewrite the mcp write test around the corrected per-caller semantics
(agent-write OFF, import_page_markdown DEFAULTS incl. the REAL #328 round-trip);
add a collab-backed wiring test driving client.updatePage (OFF) and
client.importPageMarkdown (DEFAULTS) end-to-end; add a createPage multipart-flag
test; add a server import.service.processMarkdown spec (flag OFF -> literal / no
autolink, default -> math ON for human uploads); reword the prosemirror-markdown
round-trip test to its package-default scope. Mutation-verified both caller
wirings (flip updatePageMarkdown -> defaults reddens the OFF wiring test; make
the wrapper default OFF reddens the #328 round-trip).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:58:48 +03:00
agent_coder 3e81f64415 fix(work-time): user idle breaks agent burst; cap tz length (#395)
Only an agent-sourced idle pulse extends an agent burst; a user idle
(human supervision) now falls through to the human branch so the session
is classified `work`, not `agent_only`. Add @MaxLength(64) to tz to match
its comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:53:59 +03:00
agent_coder d36021a111 fix(comment): clear resolved mark only after reopen confirmed
Move the inline ProseMirror comment-mark clear out of the Undo toast's
onClick and into the reopen mutation's onSuccess branch. Clearing it
eagerly flipped the collab mark to unresolved before the reopen result
was known: on a non-404 failure the RQ cache rolls back to resolved but
the doc kept an active highlight the panel treats as resolved, and a
comment refetch can't heal a divergence that lives in the collab doc.
Mirrors the 404 branch's editor-liveness guard/try-catch and the safe
await-then-mark pattern in comment-list-item. The button-triggered
reopen already sets the mark, so the onSuccess call is an idempotent
no-op there.

Tests: reopen failure (500) leaves the mark untouched; null editorRef
on the success path degrades gracefully.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:51:38 +03:00
agent_coder 1f2999e5ad fix(mcp): drawio — авто-стрип XML-комментариев вместо lint-ошибки (#505)
Модель органически вставляет `<!-- ... -->` в mxGraph-XML вопреки запрету
в описании тула (природа LLM, промптингом не лечится). Жёсткая ошибка
линтера [no-comments] заставляла её ПОЛНОСТЬЮ перегенерировать диаграмму —
впустую потраченный tool-call на почти каждой первой генерации.

Теперь в общем prepare-пути (`prepareModel`, через который идут
drawioCreate/drawioUpdate/drawioEditCells) комментарии срезаются ДО линта:
`xml.replace(/<!--[\s\S]*?-->/g, "")`. Комментарии не несут семантики, так
что стрип безопасен. Число срезанных уходит в `warnings[]` как
`stripped N XML comment(s)` (только при N > 0) — модель это видит, но НЕ
ретраит.

Стрип ГЕЙТИТСЯ двумя условиями, чтобы регэксп бил только по настоящим
comment-нодам:
  1. well-formedness — сначала парсим модель; на malformed-входе (сырой
     неэкранированный `<!--` внутри значения атрибута на пути create/update,
     где normalizeInput отдаёт строку без парсинга) стрип НЕ выполняется,
     иначе он молча вырезал бы текст автора. Вход отклоняют правила
     value-escaping / well-formed-xml, автор видит реальную ошибку.
  2. отсутствие CDATA — CDATA-секция well-formed, но её текст может содержать
     литеральный `<!-- ... -->`, который НЕ является comment-нодой; стрип
     молча удалил бы контент автора. Настоящий drawio держит подписи в
     атрибутах `value=` (CDATA невозможен), так что исключение CDATA бесплатно
     на легитимных моделях; CDATA-модель с реальным комментарием падает на
     сохранённый backstop no-comments (явная ошибка), а не тихо портится.

Правило `no-comments` в линтере оставлено как defense-in-depth backstop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:43:03 +03:00
agent_coder c674db2b2f feat(page-history): «время работы над статьёй» — число в UI + суточный punch-card (#395)
Оценка времени работы над страницей как сессионизация истории page_history по
паузам бездействия (WakaTime-подобно), а не span между крайними правками, плюс
drill-down в суточный таймлайн 24ч×дни. Зависит от #374 (kind + idle-пульс).

Server:
- Чистая computeWorkTime(rows, config) (§5): нормализация+дедуп → коллапс
  агентских всплесков по aiChatId → ОДИН проход сессионизации (порог зависит от
  пары: оба agent → agentTGap, иначе tGap; последняя сессия обязательно
  закрывается) → класс готовой сессии (все agent → agent_only, иначе work) →
  добивка (много-сэмпл → [first−P_in, last+P_out], одиночный скаляр →
  [t−P_single, t]) → метрики = union wall-clock по классу. Детерминированная,
  без БД.
- Чистая bucketByDay(sessions, tz) (§6.3): union work/agent_only РАЗДЕЛЬНО, разрез
  по полуночи tz через нативный Intl (DST-корректно, без «+24ч»). Инвариант
  Σ activeMs == workMs держится по построению.
- PageHistoryRepo.findTimelineByPageId — лёгкая проекция без content, ASC.
- PageHistoryService.computeWorkTime + POST /pages/history/time, гейт
  validateCanView как у /history; атрибуция человек/агент по lastUpdatedSource.

Client:
- usePageWorkTime(pageId) шлёт tz зрителя (Intl локаль).
- Кликабельное число «≈ 4 ч 30 мин» в шапке страницы (порог в тултипе).
- Модалка-punch-card: 24-часовые дорожки по дням, окна work/agent разным цветом,
  одиночные (P_single) приглушены, сумма за день, пустой день «—», сворачивание
  длинных серий пустых дней, подпись tz + T_gap. i18n ru-RU + en-US.

Tests: 22 юнита на computeWorkTime/bucketByDay (§7-фикстура ≈1ч32м vs ≈60h наив,
закрытие последней сессии, коллапс/разрыв всплеска, idle-пульс, DST 23/25ч,
полуночный разрез, инвариант, union-без-задвоения); 5 юнитов гейта контроллера;
int-spec на реальном pg (проекция + совпадение с ядром); 6 клиентских на формат.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:32:12 +03:00
agent_coder 44cdb5a4c3 feat(mcp): disable math+autolink in MCP markdown-write; getPage format:text for machine diff (#502)
WRITE: parameterize the canonical importer markdownToProseMirror with
{parseMath, fuzzyLinkify} (defaults true — editor/file-import/git-sync unchanged);
MCP write paths (createPage/updatePageMarkdown/patchNode/insertNode) call with both
false so $...$ stays literal text and bare domains don't autolink (explicit
https:// still links). READ: getPage format:'text' reuses the server jsonToText path
(deterministic flag) for flat machine-diffable text, [image]/[table RxC] placeholders.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:18:31 +03:00
agent_coder 03af65dd06 Merge remote-tracking branch 'gitea/develop' into fix/503-hardbreak-canon 2026-07-12 06:12:53 +03:00
agent_coder dead5fa8a8 test(#503): guard hardBreak representability through the MCP canon
Issue #503 reported a hardBreak sent via updatePageJson being «silently
cut» — no error, no line break — forcing the agent to split lines into
separate paragraphs.

A faithful repro shows the bug does NOT reproduce against develop:
hardBreak is fully representable in the markdown canon in BOTH directions
and survives the real Yjs write path.

- pm -> md: the serializer (now in @docmost/prosemirror-markdown after the
  #293 STEP 5 consolidation) emits the CommonMark hard break `  \n`, not a
  bare `\n`.
- md -> pm: marked tokenizes `  \n` to `<br>`, which generateJSON parses
  back to a hardBreak node in ONE paragraph (not split, not dropped).
- updatePageJson (PM JSON -> applyDocToFragment = PMNode.fromJSON +
  updateYFragment, the real collab-session write encoder -> read back)
  keeps text + hardBreak + text in one paragraph.

The issue's diagnosis points at packages/mcp/.../markdown-converter.ts as
the serializer, but that file is a 15-line re-export shim since #293 STEP 5
— the diagnosis predates the converter consolidation that already closed
both sides. P1 (semantic round-trip) + P2 (byte-fixpoint) hold, and the
node already has broad property/corpus/golden coverage in the package.

No converter change is warranted (switching the break form to `\`+newline
would churn the whole #351 byte-fixpoint corpus against the established
`  \n` repo convention for zero functional gain). This adds one
integration guard at the MCP canon seam covering all three seams of the
reported updatePageJson path. Mutation-verified: neutering the serializer
arm or the importer `<br>` handling reddens seams 1-2; neutering the real
applyDocToFragment write path reddens seam 3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:04:18 +03:00
agent_coder 5b17503df7 docs(page): explain breadcrumb ancestors need no per-ancestor permission filter (#471)
getPageBreadCrumbs returns the full ancestor chain filtered only by
deletedAt; the /breadcrumbs endpoint validates validateCanView on the
target page only. Document why this is safe rather than an accepted leak:
page restrictions inherit down the tree, so viewing a page implies the
right to view every ancestor (canUserAccessPage checks the full ancestor
chain). A hidden ancestor would already hide the target itself, making
per-ancestor filtering redundant. Owner decision: document, no logic change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:58:48 +03:00
agent_coder 12ed3a0332 feat(comment): undo button in resolve toast + sort resolved tab by resolve time
Part 1 (#542): add an inline Undo (reopen) button to the resolve success
toast in useResolveCommentMutation. Undo re-invokes the same mutation via a
self-ref (declared before useMutation, read at click time — no cycle) and
clears the inline comment mark through an editor ref (survives the originating
CommentListItem unmounting). A closured `done` flag guards a fast double-click
(notifications.hide is async). Reopen keeps the plain toast (no Undo).

onError gains a terminal 404 branch: drop the comment from the cache and
unsetComment its orphaned mark (no rollback → no phantom in Resolved, neutral
"Comment no longer exists"). The generic-failure copy is now directional
(resolve vs. re-open). Autoclose policy const RESOLVE_UNDO_AUTOCLOSE_MS=10000.

Part 2: sort the Resolved tab by resolvedAt DESC via an exported
sortResolvedByResolvedAt helper; coerce with new Date(...) because resolvedAt
is an ISO string at runtime (Date only in the optimistic window).

i18n: add "Failed to re-open comment" and "Comment no longer exists" to
en-US/ru-RU. Tests: 6 resolve-undo cases + 4 sort cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:45:42 +03:00
agent_coder b3cc6a7ff8 Merge remote-tracking branch 'gitea/develop' into feat/481-version-coherence 2026-07-12 05:36:59 +03:00
agent_coder c42cb42413 docs(reload-guard): align header/docstring wording to the window model (#481 review F2 consistency)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:36:58 +03:00
agent_coder ae3dfd8de6 Merge remote-tracking branch 'gitea/develop' into feat/492-incremental-render 2026-07-12 05:34:07 +03:00
agent_coder 41030b2c78 test+docs(deploy): покрыть reactive fail-closed guard + выровнять формулировки под оконный бюджет (#499 ревью)
F1: интеграционный тест на reactive путь при storage, который ЧИТАЕТ, но не
ПИШЕТ (quota / Safari private): getItem→null (бюджет доступен, проходим
hasAutoReloaded), setItem→throw → markAutoReloaded()===false → guard на
chunk-load-error-boundary.tsx:43 обязан отбить reload, иначе реактивный путь
зациклил бы reload без сохранённого бюджета. Симметричный proactive случай уже
покрыт guarded-reload.test.tsx. Мутация (снять guard :43) → тест краснеет.

F2: формулировки "per session / this session" в version-coherence.ts и
guarded-reload.tsx приведены к оконной модели ("per RELOAD_WINDOW_MS window" /
"window budget available") — reload-guard.ts и тест восстановления после окна
уже описывают именно окно, а не постоянный one-shot на сессию.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:32:33 +03:00
agent_coder 515a1aaf1d Merge remote-tracking branch 'gitea/develop' into fix/535-list-seam-coalesce 2026-07-12 05:22:26 +03:00
agent_coder 74387bb047 test(#522): make the internal-link subset invariant a real cross-package drift guard
Reviewer follow-up on the #522 internal-link import fix. The fix itself is
confirmed correct; these three changes harden its load-bearing subset invariant
and fix a stale doc.

Finding 1 (drift guard): the in-package test only compared isInternalPagePath
against a HAND-COPIED server regex, so a later narrowing of the real server
INTERNAL_LINK_REGEX would leave the test green while the client silently became
WIDER than the server. Add apps/server/.../export/internal-link-parity.spec.ts:
the top layer already depends on @docmost/prosemirror-markdown and exports
isInternalPagePath, so this spec imports the LIVE server INTERNAL_LINK_REGEX AND
the LIVE isInternalPagePath and asserts subset over an accept-corpus — a server
narrowing now reddens CI. The in-package manual copy is demoted from its
"drift guard" role (comment updated to say so; it stays as local documentation).

Finding 2 (charset non-vacuity): the REJECT list was purely structural, so the
mutation widening the slug charset [a-zA-Z0-9-] -> [a-zA-Z0-9-.] survived the
whole suite. Add REJECT cases whose ONLY defect is a forbidden slug character
(abc.def / abc_def / abc%20 / abc~x); assert both isInternalPagePath and the
server regex reject them. The mutation now reddens the new reject test.

Finding 3 (stale JSDoc): canonicalize.ts KNOWN_DEFAULTS module blurb claimed
every entry was read from docmost-schema and import-materialized. The new
link.internal default breaks both claims (source is editor-ext/src/lib/link.ts;
external links leave internal absent/null, never false). Add a bullet documenting
the one editor-sourced, non-materialized default (false ≡ absent/null).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:59 +03:00
agent_coder 6ec2981743 fix(prosemirror-markdown): помечать внутренние ссылки на страницы как internal при импорте markdown
При импорте markdown ссылка вида `[t](/s/<space>/p/<slug>)` сохранялась как
внешняя (link-mark получал `internal:null`, `target:"_blank"`,
`rel:"noopener noreferrer nofollow"`): открывалась новой вкладкой, без
hover-превью и без участия в backlinks. Единственный путь к нативной внутренней
ссылке был ручной JSON-патч.

Чиню в общем конвертере пакета, через который проходят ВСЕ markdown-пути (MCP
updatePageMarkdown/importPageMarkdown, patch/insertNode, тела комментариев,
серверный REST create/update, single/zip-импорт, ai-chat, git-sync pull,
вставка markdown в редактор) — все они получают исправление разом.

- Новый модуль `internal-links.ts`: чистый `isInternalPagePath(href)` —
  якорный `^/s/<space>/p/<slug>/?$`, СТРОГОЕ подмножество серверного
  `INTERNAL_LINK_REGEX` (apps/server/.../export/utils.ts), поэтому всё
  помеченное гарантированно бэклинкуется и переписывается при экспорте.
  Fail-toward-external: любая неоднозначность (scheme/host, `#`, `?`,
  бесспейсовый `/p/<slug>`, лишний сегмент, не-строка) остаётся внешней.
- `markInternalLinks(doc)`: пост-обход готового ProseMirror-документа,
  помечает КАЖДУЮ text-ноду, покрытую внутренней ссылкой (включая случай
  вложенных bold/italic внутри ссылки) → `{internal:true, target:null,
  rel:null}`. Чистая и идемпотентная.
- Разводка в `markdownToProseMirrorSync` после stripEmptyParagraphs.
- §11 (обязательная сопутствующая правка): `internal:false` добавлен в
  `KNOWN_DEFAULTS.link` канонизатора — редакторные внешние ссылки хранят
  `internal:false`, теперь он канонизируется как absent/null/external, а
  load-bearing `internal:true` (не дефолт) переживает канонизацию.

Тесты: accept/reject-пиннинг матчера (edge: trailing slash, `#`, `?`,
`/p/x` без спейса, `https://h/p/x`, `/api/...`, uppercase, empty-space),
subset-инвариант против серверного регэкспа, мультинодовая ссылка со вложенными
марками, идемпотентность, полный конвертер (internal помечен / external нетронут /
бэклинк-извлекаемость), комментарий-тело через общий путь, canonicalize §11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:58 +03:00
agent_coder dd1fe90515 fix(prosemirror-markdown): guard three-way list coalesce against transitive style loss (#535)
Review follow-up on the list-seam coalescing:

1. The three-way merge only checked each PRE-EXISTING list against the
   inserted one. A default-typed inserted orderedList between two lists with
   explicit DIFFERENT numbering styles passed both pairwise checks through the
   null-typed middle, collapsing all three and silently losing the right
   list's style. Add a `listsMergeable(left, right)` guard to the three-way
   condition. On failure fall through to the single-seam path: a single
   inserted block can be absorbed by at most one neighbour, so prefer the LEFT
   seam (consistent with the three-way survivor choice) and leave the
   incompatible right list separate with its own style.

2. Lock the footnotesList exclusion with a test — inserting a footnotesList
   next to a footnotesList must leave TWO separate blocks (a refactor of the
   allow-list to endsWith("List") would otherwise silently merge them and
   corrupt footnotes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:54 +03:00
agent_coder c96fafc4ad perf(ai-chat): append-персист шагов — per-step INSERT вместо переписи строки (#492)
Раньше каждый onStepFinish переписывал ВСЮ строку ассистента (растущий
metadata.parts jsonb со всеми выводами инструментов) → O(n²) объёма записи
на прогон: под MVCC/TOAST апдейт jsonb переписывает всю версию строки, так
что шаг k пишет ~k×вывод. Прогон из 50 шагов по ~100 КБ = сотни МБ WAL и
мёртвых кортежей за ход, что молотит autovacuum. (#490 убрал только ВТОРУЮ
копию в tool_calls; сам metadata.parts всё ещё рос и переписывался.)

Теперь каждый завершённый шаг ДОПИСЫВАЕТСЯ отдельной строкой в лёгкую
таблицу ai_chat_run_steps (только парты этого шага), а строка сообщения
получает дешёвый маркер (stepsPersisted + toolTraceVersion, без растущего
блоба parts). Полный metadata.parts собирается ОДИН раз на финализации.
НЕ jsonb-append (||): апдейт всё равно переписывает всю TOAST-версию —
экономится только сетевой payload, а WAL/мёртвые кортежи остаются; поэтому
именно ОТДЕЛЬНАЯ таблица + INSERT.

Три обязательные интеграции:
- reconstructRunParts(row, stepRows) → { parts, stepsPersisted }: единый
  шов переключения бэкенда. Читает парты из СТРОКИ, если она уже несёт
  inline-parts (старые записи + ЛЮБАЯ финализированная), иначе из ТАБЛИЦЫ
  ШАГОВ (mid-run запись #492). Дискриминатор — наличие непустого
  metadata.parts (флаг схемы не нужен). Потребители (attach-seed,
  delta-poll, export, reconnect) прогоняют строки через hydrateAssistantParts
  на границе чтения — их контракт/вывод не меняется, старые и новые записи
  восстанавливаются идентично.
- сигнал ротации кольца реестра #491 (confirmPersistedStep) теперь стреляет
  на подтверждённый INSERT шага, под тем же контрактом (updateStreaming
  возвращает stepsPersisted / null).
- era-marker toolTraceVersion (#490) больше не ставится полной переписью —
  ставится в маркере шага и на финализации (flushAssistant), остаётся
  консистентным.

Полная обратная совместимость: прогон, записанный по-старому (полная строка,
без строк шагов), восстанавливается/attach/export идентично. При отсутствии
репозитория шагов (позиционные тест-конструкции) — фолбэк на прежний
полнострочный flush (без регрессии, только без выигрыша WAL).

Тесты (реальный pg, int-lane):
- WAL-дельта (pg_current_wal_lsn) на прогоне 40×100КБ: new=4.3МБ vs
  old=90.3МБ (20.8x) — O(Σ шагов) против O(n²); старый путь в тесте И есть
  ревертнутое поведение (мутация-проверка).
- reconstruct-контракт: new-style (таблица шагов) и old-style (inline) прогоны
  восстанавливаются в идентичные parts; hydrate заполняет строку.
- миграция up/down roundtrip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:02:25 +03:00
agent_coder 64566e9327 test(deploy): согласовать reload-guard/chunk-load тесты с общим оконным бюджетом (#481, rebase reconcile)
После ребейза на develop версии-когерентность (#481, общий one-shot флаг)
и chunk-load boundary (#495, оконный guard) были сведены к ОДНОМУ общему
оконному бюджету в @/lib/reload-guard: не более одного авто-reload за
RELOAD_WINDOW_MS суммарно по обоим путям, при этом второй деплой в той же
вкладке восстанавливается.

- reload-guard.test.ts: переписаны под оконную семантику (ключ chunk-reload-at
  хранит таймстамп, а не флаг); сюда же перенесены чистые тесты shouldAutoReload.
- chunk-load-error-boundary.test.ts: убран импорт shouldAutoReload (переехал в
  reload-guard).
- chunk-load-error-boundary.tsx: handleError экспортирован для теста инварианта.
- reload-budget.integration.test.tsx: инвариант (a) на РЕАЛЬНОМ guard — reload
  на одном пути тратит общий бюджет для другого в пределах окна, после окна
  восстановление, при недоступном sessionStorage ни один путь не перезагружает.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:01:36 +03:00
agent_coder 10a4326fbf docs(deploy): поправить устаревший коммент про auto-reload скрытой вкладки под вариант C (#481, ревью F1)
r1-коммент утверждал «auto-reload on a hidden tab», но вариант C (r2) убрал
авто-reload скрытой вкладки ради защиты несохранённого ввода. Переписал под
фактическое поведение: баннер + отложенный reload на следующей in-app навигации.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:55:22 +03:00
agent_coder 68409a8ae9 fix(deploy): variant C — reload по in-app навигации вместо visibilitychange + лог-след + CHANGELOG (#481, ревью)
Правки по ревью #499. Владелец выбрал вариант C по вопросу потери несохранённого
ввода.

F1 (reload по навигации): убрал ветку visibilitychange→reload И немедленный reload
скрытой-при-получении вкладки — visibility больше не влияет на авто-reload вообще.
На реальном mismatch: баннер (кнопка «Обновить» — немедленный reload под тем же
one-shot гардом) + модульный флаг pendingNavReload. Хук useVersionReloadOnNavigation
(useLocation + useEffect по location.key, пропуск первого рендера через useRef;
react-router v7 BrowserRouter компонентный, объекта роутера нет — подписка изнутри
дерева) на СЛЕДУЮЩЕЙ навигации зовёт consumeNavigationReload → performAutoReload
(mark-перед-reload, при spent/непишущемся storage — только баннер). Скрытая вкладка
идёт тем же путём (C единообразно). chunk-load-error-boundary reload не тронут —
остаётся бэкапом для не-навигирующей вкладки. Так reload в безопасной точке (юзер
и так уходит со страницы), а не когда свернул с недописанным комментарием.

F2 (лог-след): общий recordReloadBreadcrumb/takeReloadBreadcrumb (sessionStorage,
переживает reload). performAutoReload пишет breadcrumb {path:proactive,server,
client} + console.warn перед reload; chunk-boundary пишет {path:chunk-boundary};
surfacePreviousReloadBreadcrumb на маунте UserProvider логирует прошлый след.

F3 (CHANGELOG + AGENTS.md): user-facing запись про version-coherence + строка про
артефакт client/dist/version.json.

Тесты: vitest version-coherence+reload-guard+guarded-reload 23 (переписан на
MemoryRouter+useNavigate: reload РОВНО раз на первой навигации после mismatch, НЕ
на 2-й навигации, НЕ на 2-м mismatch one-shot, НЕ от ухода в фон; скрытая вкладка
— только на навигации; Update — сразу; banner-only/write-fail — никогда). Шире:
features/user 34.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:55:22 +03:00
agent_coder fc624f5a4b feat(deploy): version-coherence — WS-анонс версии сервера + guarded reload устаревших вкладок (#481)
SPA — долгоживущий клиент: вкладку держат часами, сервер за это время
передеплоивают. Старый index.html ссылается на content-hashed чанки, которых в
новом образе нет → ленивый import() → catch-all отдаёт index.html как text/html →
WebKit «not a valid JavaScript MIME type» (наблюдали в проде). Реактивный
предохранитель (chunk-error-boundary) ловит УЖЕ случившуюся ошибку. Делаем сигнал
ПРОАКТИВНЫМ: сервер сообщает версию по существующему WS, клиент сравнивает и
делает guarded reload ДО битого чанка. closes #481

- Единый источник версии: vite.config.ts вычисляет resolveAppVersion РОВНО раз →
  и в define.APP_VERSION, и в плагине, пишущем dist/version.json. Бандл и файл
  тождественны by construction. Сервер читает client/dist/version.json при старте
  (client-version.ts: resolveClientDistPath вынесен из static.module — единый
  источник пути; readClientBuildVersion → версия или '' на любой ошибке,
  fail-safe). НИКАКИХ Docker/ENV-изменений.
- WS: отдельное событие app-version {version} (НЕ в message/WebSocketEvent —
  член без operation сломал бы union). ws.gateway читает версию в onModuleInit +
  boot-лог ACTIVE/DISABLED; emit в handleConnection (success-ветка, после join).
  ТОЛЬКО per-connect анонс, БЕЗ broadcast (broadcast в кластере → thundering-herd
  reload флота; естественный реконнект покрывает и single-container, и кластер).
- Клиент: listener навешан СИНХРОННО в том же useEffect, что создаёт сокет (до
  connect — иначе гонка с немедленным emit сервера). Чистая decideVersionAction
  (reload|banner|noop; пустая версия → noop fail-safe). Грязная оболочка
  triggerGuardedReload: модульный latch, hidden→немедленный reload, visible→баннер
  + reload-on-hidden {once}, фикс-id закрываемый баннер с кнопкой «Обновить».
- Общий one-shot флаг: reload-guard.ts (hasAutoReloaded/markAutoReloaded над
  существующим chunk-reload-attempted); chunk-load-error-boundary переведён на
  него. Проактивный + реактивный reload делят ОДИН счётчик → максимум одна
  авто-перезагрузка за сессию суммарно; permanent skew / oscillation / disabled
  storage → после первого reload только баннер, петли нет (проверено трассировкой).

Проверка: server jest client-version 7/7; client vitest version-coherence+
reload-guard+guarded-reload 16/16 (coverage 97.9%). Build-proof единого источника:
APP_VERSION=test-A → dist/version.json {"version":"test-A"} + вшито в бандл.
Полный staging-приём (docker build-arg, WS-кадры в DevTools, мульти-таб) — вне
автостенда, шаги приёмки #481 (крит.1-4) на ручную проверку.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:54:35 +03:00
agent_coder be70bd2e8e Merge remote-tracking branch 'gitea/develop' into fix/541-reconnect-timeout 2026-07-12 04:37:47 +03:00
agent_coder 2f8c5d9a98 perf(client): инкрементальный рендер стрима ответа (#492)
Путь ответа ассистента теперь рендерится инкрементально, по образцу
StreamingPlainText из ветки reasoning (#492, волна C эпика #497).

Раньше MarkdownPart прогонял ВЕСЬ накопленный ответ через канонический
конвейер (markdownToProseMirrorSync → PMNode.fromJSON → DOMSerializer →
DOMPurify) на КАЖДОМ throttled-тике (~20 Гц). На синтетическом потоке
~100 КБ это 394 вызова renderChatMarkdown — O(числа тиков), причём каждый
вызов заново парсит всю растущую строку.

Теперь:
- StreamingMarkdownText делит текст на блоки по безопасному срезу
  (splitPlainChunks — та же append-only-инвариантность, что у reasoning):
  СТАБИЛИЗИРОВАННЫЕ блоки идут через канонический конвейер и мемоизируются
  (каждый блок парсится РОВНО ОДИН раз), живой ХВОСТ — дешёвый plain-text
  (React-escaped, без парсера/санитайзера/innerHTML) до стабилизации.
- На финализации (флип state → done или конец хода) — ОДИН полный
  канонический рендер всего текста: побайтовая визуальная паритетность с
  прежним выводом (включая <li><p>-обёртки схемы и scoped-CSS из #498).
- Гейт liveness тот же, что у ReasoningBlock: streaming =
  turnStreaming && part.state === "streaming".

Также цикл рендера частей переведён на ИСЧЕРПЫВАЮЩИЙ switch по видам частей
с never-проверкой в default (вместо прежнего WARNING-комментария): новый
закрытый вид части в UIMessagePart теперь ошибка компиляции.

Тесты:
- perf-smoke: на ~100 КБ потоке число вызовов renderChatMarkdown ≤ blocks+2
  и ≪ ticks (O(блоков), не O(тиков)); мутация (снять memo с MarkdownChunk)
  краснит ассерт (78631 вызовов).
- visual-regression: финальный рендер побайтово равен renderChatMarkdown
  всего текста (в т.ч. <li><p>), инкрементальный вид сходится к нему на
  финализации; учтён neutralizeInternalLinks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:35:06 +03:00
agent_coder 69e04349a0 fix(client): вход в reconnect ограничивает ожидание getRun таймаутом (#541)
При живом обрыве SSE вход в лестницу reconnect происходил только
после резолва getRun(cid) (ре-сид из персиста). Путь REJECT
обрабатывался через .catch, но ЗАВИСШИЙ getRun (соединение есть,
ответа нет) был не ограничен: axios-клиент (lib/api-client.ts) не
имеет timeout, а stalled-idle-cap взводится только ПОСЛЕ входа в
reconnecting/polling. Итог — FSM залипала в `streaming` без баннера
и без поллинга до сокет-таймаута браузера (минуты). Это тот самый
класс тихого зависания, который устраняет эпик #497.

Оборачиваю ожидание getRun гонкой с таймаутом
(RECONNECT_RESEED_TIMEOUT_MS = 4s): по таймауту берётся ТОТ ЖЕ
фолбэк, что и на reject — dropLivePartialAndReplayFromStart() +
enterReconnect(runId), так что лестница/поллинг стартуют и
stalled-idle-cap взводится. Локальный флаг `settled` делает ветви
resolve/reject/timeout взаимоисключающими: поздний резолв getRun
после сработавшего таймаута полностью игнорируется (не входит в
reconnect повторно, не перетирает replay-from-start устаревшим
ре-сидом, ничего не перевзводит). Таймер живёт в reseedTimerRef и
очищается при размонтировании (никаких висящих setTimeout).

Тесты: hang-кейс (getRun не резолвится -> по таймауту FSM в
reconnecting, live-partial сброшен, replay-from-start) и
late-resolve safety (поздний резолв — no-op). Мутация: замена гонки
на голый getRun краснит оба #541-теста.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:29:25 +03:00
agent_coder 5bd5995ef0 fix(prosemirror-markdown): coalesce list seams on insert (#535)
insertNode with a markdown/JSON list now appends/prepends its items into
an adjacent same-type sibling list instead of creating a separate sibling
list that the serializer splits with `<!-- -->`.

Adds a single shared merge rule (isCoalescibleList/listsMergeable) used by
both the markdown path (insertNodesRelative) and the raw-node path
(insertNodeRelative). Coalescing is strictly local to the two seams of the
active insertion (each merged at most once — no greedy loop, no global
normalization). Survivor is chosen POSITIONALLY (the pre-existing neighbour
outside the inserted [i,j) range), so it keeps its block id and list-level
attrs (e.g. orderedList start); the inserted wrapper's re-minted id is
discarded. Handles the three-way case (list inserted between two same-type
lists → left survives) and refuses to coalesce an empty inserted list.

before/after now resolve via findAnchorChain and splice into the anchor's
immediate parent, so coalescing runs against the actual parent array
(incl. lists nested in callouts / table cells).

Serializer / LIST_MARKER_SEPARATOR untouched — two genuinely-separate
lists still emit `<!-- -->`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 02:43:53 +03:00
136 changed files with 11857 additions and 1647 deletions
+1
View File
@@ -463,6 +463,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, `apps/server` (#345), and `apps/client` (#347) — do NOT reintroduce a per-package copy. The client uses the package's `browser` entry (`@docmost/prosemirror-markdown/browser`): markdown paste (`markdown-clipboard.ts`), copy-as-markdown, and AI-chat rendering now all go through the canonical converter, so the hand-written `marked`/`turndown` markdown layer that used to live in `editor-ext` was deleted (#347). The browser entry runs the HTML→DOM stage on the native `DOMParser`, so jsdom stays out of the client bundle. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
- The build also emits `client/dist/version.json` (`{"version": …}`) from a small `vite.config.ts` plugin using the **same** `appVersion` that feeds `define.APP_VERSION`, so the file and the baked-in bundle version are identical by construction. The server reads it at startup (`ws.gateway.ts` via `readClientBuildVersion`/`resolveClientDistPath`) and announces it to each socket on connect (`app-version` event) so a tab left open across a redeploy can guard-reload before hitting a stale chunk (version-coherence). No runtime env / Dockerfile change — the file already ships in `client/dist`; missing/empty file ⇒ feature inert.
## Conventions
+12
View File
@@ -142,6 +142,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
snapshots switched from a fixed interval to a trailing idle-flush with a
max-wait ceiling, and a boundary snapshot is pinned whenever the editing source
changes (e.g. a person's edits followed by the AI agent). (#370)
- **Open tabs pick up a new deploy on their own.** After the server is
redeployed while a tab is left open for hours, the tab now learns the new
build version over the existing WebSocket (announced per-connect, so a natural
reconnect delivers it) and shows a "A new version is available" banner with an
Update button. To avoid dropping a half-written comment or form, the tab is
not reloaded when you merely switch away from it; instead it auto-reloads at
the next safe point — the next in-app navigation (or immediately if you click
Update) — before it can hit a stale lazy-loaded chunk. At most one automatic
reload happens per 5-minute window, shared with the existing chunk-load
recovery, so a permanent version skew degrades to the banner rather than a
reload loop while a second deploy in the same tab still recovers. When the
build carries no version info the feature stays inert. (#481)
- **Place several images side by side in a row.** A new "Inline (side by
side)" alignment mode in the image bubble menu renders consecutive inline
@@ -1,4 +1,5 @@
{
"A new version is available": "A new version is available",
"Account": "Account",
"Active": "Active",
"Add": "Add",
@@ -239,6 +240,8 @@
"Comment re-opened successfully": "Comment re-opened successfully",
"Comment unresolved successfully": "Comment unresolved successfully",
"Failed to resolve comment": "Failed to resolve comment",
"Failed to re-open comment": "Failed to re-open comment",
"Comment no longer exists": "Comment no longer exists",
"Resolve comment": "Resolve comment",
"Unresolve comment": "Unresolve comment",
"Resolve Comment Thread": "Resolve Comment Thread",
@@ -1427,5 +1430,20 @@
"Boundary": "Boundary",
"Autosave": "Autosave",
"Only versions": "Only versions",
"No saved versions yet.": "No saved versions yet."
"No saved versions yet.": "No saved versions yet.",
"Time worked on this article": "Time worked on this article",
"Show time worked on this page": "Show time worked on this page",
"Estimated time worked (inactivity gap {{gap}} min)": "Estimated time worked (inactivity gap {{gap}} min)",
"Estimate · timezone {{tz}} · inactivity gap {{gap}} min": "Estimate · timezone {{tz}} · inactivity gap {{gap}} min",
"No editing activity recorded yet.": "No editing activity recorded yet.",
"× {{count}} days without edits": "× {{count}} days without edits",
"agent: {{value}}": "agent: {{value}}",
"Work": "Work",
"Agent": "Agent",
"≈ {{hours}}h {{minutes}}m": "≈ {{hours}}h {{minutes}}m",
"≈ {{hours}}h": "≈ {{hours}}h",
"≈ {{minutes}}m": "≈ {{minutes}}m",
"{{hours}}h {{minutes}}m": "{{hours}}h {{minutes}}m",
"{{hours}}h": "{{hours}}h",
"{{minutes}}m": "{{minutes}}m"
}
@@ -1,4 +1,5 @@
{
"A new version is available": "Доступна новая версия",
"Account": "Аккаунт",
"Active": "Активный",
"Add": "Добавить",
@@ -239,6 +240,8 @@
"Comment re-opened successfully": "Комментарий успешно открыт повторно",
"Comment unresolved successfully": "Комментарий успешно переведён в нерешённые",
"Failed to resolve comment": "Не удалось разрешить комментарий",
"Failed to re-open comment": "Не удалось переоткрыть комментарий",
"Comment no longer exists": "Комментарий больше не существует",
"Resolve comment": "Решить комментарий",
"Unresolve comment": "Снять статус решённого с комментария",
"Resolve Comment Thread": "Решить ветку комментариев",
@@ -1442,5 +1445,20 @@
"Boundary": "Граница",
"Autosave": "Автосейв",
"Only versions": "Только версии",
"No saved versions yet.": "Пока нет сохранённых версий."
"No saved versions yet.": "Пока нет сохранённых версий.",
"Time worked on this article": "Время работы над статьёй",
"Show time worked on this page": "Показать время работы над страницей",
"Estimated time worked (inactivity gap {{gap}} min)": "Оценка времени работы (порог паузы {{gap}} мин)",
"Estimate · timezone {{tz}} · inactivity gap {{gap}} min": "Оценка · таймзона {{tz}} · порог паузы {{gap}} мин",
"No editing activity recorded yet.": "Правок пока нет.",
"× {{count}} days without edits": "× {{count}} дн. без правок",
"agent: {{value}}": "агент: {{value}}",
"Work": "Работа",
"Agent": "Агент",
"≈ {{hours}}h {{minutes}}m": "≈ {{hours}} ч {{minutes}} мин",
"≈ {{hours}}h": "≈ {{hours}} ч",
"≈ {{minutes}}m": "≈ {{minutes}} мин",
"{{hours}}h {{minutes}}m": "{{hours}} ч {{minutes}} м",
"{{hours}}h": "{{hours}} ч",
"{{minutes}}m": "{{minutes}} м"
}
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { isChunkLoadError, shouldAutoReload } from "./chunk-load-error-boundary";
import { isChunkLoadError } from "./chunk-load-error-boundary";
// The detector decides whether a caught render error is a stale-deploy chunk-404
// (→ auto-reload to fetch the new manifest) vs a genuine app error (→ generic
@@ -35,31 +35,3 @@ describe("isChunkLoadError", () => {
expect(isChunkLoadError(err)).toBe(false);
});
});
// The window gate replaces the old one-shot flag: it must permit recovery across
// several deploys in one tab (each > window apart) while still stopping an infinite
// reload loop when a lazy chunk is permanently broken (a second failure < window).
describe("shouldAutoReload", () => {
const WINDOW = 5 * 60 * 1000;
const NOW = 1_000_000_000_000;
it("allows a reload when we have never auto-reloaded", () => {
expect(shouldAutoReload(NOW, null, WINDOW)).toBe(true);
});
it("allows a reload when the last one was 6 minutes ago (outside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 6 * 60 * 1000, WINDOW)).toBe(true);
});
it("blocks a reload when the last one was 1 minute ago (inside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 1 * 60 * 1000, WINDOW)).toBe(false);
});
it("blocks a reload exactly at the window boundary (not strictly older)", () => {
expect(shouldAutoReload(NOW, NOW - WINDOW, WINDOW)).toBe(false);
});
it("allows a reload when the stored timestamp is unparseable (NaN)", () => {
expect(shouldAutoReload(NOW, NaN, WINDOW)).toBe(true);
});
});
@@ -1,26 +1,11 @@
import { ReactNode } from "react";
import { ErrorBoundary } from "react-error-boundary";
import { Button, Center, Stack, Text } from "@mantine/core";
// sessionStorage key holding the epoch-ms timestamp of the last automatic reload.
const RELOAD_AT_KEY = "chunk-reload-at";
// Allow at most one automatic reload per this window. A stale-deploy 404 is cured
// by a single reload, so anything inside the window is treated as a reload loop
// (permanently-broken chunk) and falls through to the manual UI. A window (rather
// than a one-shot flag) lets a SECOND deploy in the same tab's lifetime recover too.
const RELOAD_WINDOW_MS = 5 * 60 * 1000;
// Pure window decision, unit-tested in isolation: auto-reload only if we have never
// auto-reloaded (lastReloadAt null/NaN) or the last one was strictly older than the
// window. Anything inside the window is suppressed to break an infinite reload loop.
export function shouldAutoReload(
now: number,
lastReloadAt: number | null,
windowMs: number,
): boolean {
if (lastReloadAt === null || Number.isNaN(lastReloadAt)) return true;
return now - lastReloadAt > windowMs;
}
import {
hasAutoReloaded,
markAutoReloaded,
recordReloadBreadcrumb,
} from "@/lib/reload-guard";
// Heuristic detection of a failed dynamic import. Since the code-splitting work,
// every route (plus Aside / AiChatWindow) is React.lazy: when a new deploy
@@ -39,24 +24,26 @@ export function isChunkLoadError(error: unknown): boolean {
);
}
function handleError(error: unknown) {
// Exported for tests: the reactive chunk-load reload decision, so the shared
// window budget (invariant: ≤1 auto-reload per window across this path AND the
// proactive version-coherence path) can be exercised against the real guard.
export function handleError(error: unknown) {
if (!isChunkLoadError(error)) return;
// A stale-chunk 404 is cured by a full reload that re-fetches index.html and
// the new chunk manifest. Auto-reload at most once per RELOAD_WINDOW_MS: this
// recovers across multiple deploys in a single tab's lifetime, yet a
// permanently-broken lazy chunk (which would loop) is stopped after the first
// reload and falls through to the manual recovery UI below.
try {
const raw = sessionStorage.getItem(RELOAD_AT_KEY);
const lastReloadAt = raw === null ? null : Number.parseInt(raw, 10);
const now = Date.now();
if (!shouldAutoReload(now, lastReloadAt, RELOAD_WINDOW_MS)) return;
sessionStorage.setItem(RELOAD_AT_KEY, String(now));
} catch {
// sessionStorage unavailable (private mode / disabled): skip the automatic
// reload rather than risk an unguarded loop; the fallback UI still recovers.
return;
}
// the new chunk manifest. Auto-reload at most once per window via the SHARED
// window-based reload guard (see @/lib/reload-guard — the same budget the
// proactive version-coherence path consumes, so a mismatch that arrives on
// both paths reloads at most once per window across BOTH). This recovers
// across multiple deploys in a single tab's lifetime, yet a permanently-broken
// lazy chunk (which would loop) is stopped after the first reload and falls
// through to the manual recovery UI below. If the shared budget is already
// spent this window, or the stamp write fails (storage unavailable), we return
// without reloading rather than risk a loop.
if (hasAutoReloaded()) return;
if (!markAutoReloaded()) return;
// Trace before the reload clears the console (same diagnostic breadcrumb the
// proactive version-coherence path writes, tagged with this path).
recordReloadBreadcrumb({ path: "chunk-boundary" });
window.location.reload();
}
@@ -1254,4 +1254,88 @@ describe("ChatThread — live reconnect + stalled", () => {
});
expect(onResumeFallback).toHaveBeenCalledWith(false); // POLL_IDLE_CAP -> idle -> disarm
});
it("#541: getRun HANGS on a live disconnect — the timeout race still enters reconnect (no silent freeze in `streaming`)", async () => {
// MUTATION-VERIFY: revert the race to a bare `getRun(cid).then/.catch` and this
// reddens — a HUNG (never-settling, NOT rejected) getRun leaves the FSM stuck in
// `streaming` with no reconnect banner and no poll (the axios client sets no
// request timeout, and the stalled-idle cap only arms AFTER reconnecting/polling).
renderLive();
h.state.getRun.mockReset();
h.state.getRun.mockReturnValue(new Promise(() => {})); // getRun HANGS forever
await disconnect(); // live partial = liveMsg (id "a2")
expect(h.state.getRun).toHaveBeenCalledWith("c1");
// BEFORE the bound fires the FSM is still in the live turn — the very freeze bug.
expect(screen.queryByText(/reconnecting/i)).toBeNull();
// The recovery-start bound fires -> the SAME fallback as the reject path.
act(() => {
vi.advanceTimersByTime(4_000);
});
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
// The live partial (a2) was DROPPED from the store (replay-from-start, no stale
// tail-apply base). Mirrors the getRun-REJECT test's inspection.
const removedLivePartial = (
h.state.setMessages as unknown as {
mock: { calls: [unknown][] };
}
).mock.calls.some(([updater]) => {
if (typeof updater !== "function") return false;
const out = (updater as (p: { id: string }[]) => { id: string }[])([
{ id: "a2" },
{ id: "u1" },
]);
return !out.some((m) => m.id === "a2");
});
expect(removedLivePartial).toBe(true);
// ...and the anchor was NULLED -> replay-from-start (no ?anchor=&n= over the partial).
advanceToAttempt(1);
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
});
it("#541: a getRun resolve AFTER the timeout already fired is IGNORED (no double reconnect, no stale re-seed)", async () => {
// The timeout wins first and enters the ladder via replay-from-start. When the
// hung getRun FINALLY answers, its late `.then` must be a full no-op: it must not
// re-seed the store from the (now stale) persisted row, must not re-set the
// anchor, and must not re-enter reconnect. The local `settled` flag makes the
// resolve/reject/timeout branches mutually exclusive.
// MUTATION-VERIFY: drop the `settled` guard (let the late `.then` run) and the
// late re-seed re-sets the anchor (URL regains ?anchor=a2&n=3) + calls setMessages.
renderLive();
let resolveGetRun!: (v: unknown) => void;
h.state.getRun.mockReset();
h.state.getRun.mockReturnValue(
new Promise((r) => {
resolveGetRun = r;
}),
);
await disconnect();
act(() => {
vi.advanceTimersByTime(4_000); // bound fires -> replay-from-start, reconnecting
});
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
advanceToAttempt(1);
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
// Anchor is null -> replay-from-start URL (the pre-condition the late resolve must
// not undo).
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
const setMessagesCallsBefore = h.state.setMessages.mock.calls.length;
// NOW the hung getRun finally resolves with a persisted anchor (id a2, steps 3).
await act(async () => {
resolveGetRun(persistedAnchor());
await Promise.resolve();
});
// The late resolve did NOT re-seed the store...
expect(h.state.setMessages.mock.calls.length).toBe(setMessagesCallsBefore);
// ...did NOT re-set the anchor (URL stays replay-from-start, no ?anchor=&n=)...
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
// ...and did NOT trigger a fresh reconnect attach.
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
});
});
@@ -81,6 +81,17 @@ const STREAM_THROTTLE_MS = 50;
// THREAD now (the FSM owns polling->stalled); the window just polls while armed.
const DEGRADED_POLL_IDLE_MAX_MS = 10 * 60_000;
// #541: the bound on the persist re-seed (getRun) wait when ENTERING the reconnect
// ladder after a live SSE drop. The axios client (lib/api-client.ts) sets NO request
// timeout, and the stalled-idle cap only arms AFTER the FSM enters reconnecting/
// polling — which never happens if getRun HANGS (connection established, no response).
// Without this bound a live drop + a hung getRun sticks the FSM in `streaming` with
// no banner and no poll until the browser socket timeout (minutes) — the silent-freeze
// class #497 eliminates. This is a deliberate RECOVERY-START bound (drop the live
// partial + enter the ladder so the poll/idle-cap arms), intentionally much shorter
// than any network socket timeout — not a network read timeout.
const RECONNECT_RESEED_TIMEOUT_MS = 4_000;
/** The #487 active (non-terminal) run statuses — mirrors the server's
* ACTIVE_RUN_STATUSES. A run-fact is "active" only for these. */
function isActiveRunStatus(status: string | null | undefined): boolean {
@@ -286,6 +297,9 @@ export default function ChatThread({
// stalled inactivity cap. Cleared by the cancelReconnect effect / the cap effect.
const reconnectTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const idleCapTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// #541: the persist re-seed (getRun) timeout race on the disconnect->reconnect
// entry. Held in a ref so unmount (DISPOSE cleanup) clears it — no dangling timer.
const reseedTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// #491 tail-only: seed EVERY persisted row unchanged (no strip). The streaming
// tail holds steps 0..N-1; the run-stream registry's tail (steps >= N) is APPENDED
@@ -748,37 +762,68 @@ export default function ChatThread({
anchorRef.current = null;
};
if (cid) {
void getRun(cid)
.then((res) => {
if (!mountedRef.current) return;
const persisted = res.message;
if (persisted && persisted.role === "assistant") {
anchorRef.current = {
id: persisted.id,
stepsPersisted: stepsPersistedOf(persisted),
};
// Replace the live partial with the persisted row IN PLACE by id —
// the re-seed from persist. The attach's tail (steps >= N) then
// appends to a store holding EXACTLY steps 0..N-1: no duplication.
setMessages((prev) => mergeById(prev, rowToUiMessage(persisted)));
} else {
// No persisted assistant row (pre-first-frame break): drop the live
// partial + replay from start (no anchor/n) so nothing is duplicated.
dropLivePartialAndReplayFromStart();
}
enterReconnect(res.run?.id ?? runId);
})
.catch(() => {
if (!mountedRef.current) return;
// Persist read FAILED: we cannot re-seed from fresh persist, and a
// stale mount-time anchor over the live partial would tail-apply
// already-present steps -> duplication (a flaky-network blip:
// SSE + getRun both fail, network recovers in ~1s, the registry still
// covers from the mount frontier). Restore the removed-filter guarantee
// instead: drop the live partial + replay from start / 204 -> poll.
// #541: bound the persist re-seed wait with a timeout race. getRun goes
// through the axios client, which has NO request timeout; a HUNG getRun
// (connection open, no response) — distinct from a REJECT, which the
// `.catch` already handles — would otherwise never let us enter the ladder,
// leaving the FSM stuck in `streaming` with no banner and no poll until the
// browser socket timeout. `settled` makes the three branches (resolve /
// reject / timeout) mutually exclusive: whichever fires FIRST wins and the
// others become no-ops. So a LATE getRun resolve AFTER the timeout is fully
// ignored — it cannot (i) re-enter reconnect a second time, (ii) overwrite
// the timeout's replay-from-start with a stale re-seed, or (iii) re-arm any
// timer. On timeout we take the SAME fallback as the reject path (drop the
// live partial + enter the ladder, so the poll and the stalled-idle cap arm).
let settled = false;
const finishReseed = (apply: () => void): void => {
if (settled) return;
settled = true;
if (reseedTimerRef.current) {
clearTimeout(reseedTimerRef.current);
reseedTimerRef.current = null;
}
if (!mountedRef.current) return;
apply();
};
reseedTimerRef.current = setTimeout(() => {
finishReseed(() => {
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
});
}, RECONNECT_RESEED_TIMEOUT_MS);
void getRun(cid)
.then((res) => {
finishReseed(() => {
const persisted = res.message;
if (persisted && persisted.role === "assistant") {
anchorRef.current = {
id: persisted.id,
stepsPersisted: stepsPersistedOf(persisted),
};
// Replace the live partial with the persisted row IN PLACE by id —
// the re-seed from persist. The attach's tail (steps >= N) then
// appends to a store holding EXACTLY steps 0..N-1: no duplication.
setMessages((prev) => mergeById(prev, rowToUiMessage(persisted)));
} else {
// No persisted assistant row (pre-first-frame break): drop the live
// partial + replay from start (no anchor/n) so nothing is duplicated.
dropLivePartialAndReplayFromStart();
}
enterReconnect(res.run?.id ?? runId);
});
})
.catch(() => {
finishReseed(() => {
// Persist read FAILED: we cannot re-seed from fresh persist, and a
// stale mount-time anchor over the live partial would tail-apply
// already-present steps -> duplication (a flaky-network blip:
// SSE + getRun both fail, network recovers in ~1s, the registry still
// covers from the mount frontier). Restore the removed-filter guarantee
// instead: drop the live partial + replay from start / 204 -> poll.
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
});
});
} else {
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
@@ -903,6 +948,12 @@ export default function ChatThread({
}
return () => {
mountedRef.current = false;
// #541: clear the in-flight persist re-seed timeout (not an FSM effect timer,
// so DISPOSE does not touch it) — no dangling setTimeout after unmount.
if (reseedTimerRef.current) {
clearTimeout(reseedTimerRef.current);
reseedTimerRef.current = null;
}
dispatch({ type: "DISPOSE" }); // aborts attach + timers, bumps epoch (I5)
};
// Mount-only by design; the parent remounts per chat via `key`.
@@ -27,6 +27,7 @@ vi.mock("@/features/ai-chat/utils/markdown.ts", async () => {
import MessageItem from "./message-item";
import { messageSignature } from "@/features/ai-chat/utils/message-signature.ts";
import { splitPlainChunks } from "./streaming-plain-text";
// matchMedia (read by MantineProvider) is stubbed globally in vitest.setup.ts.
@@ -114,3 +115,89 @@ describe("MessageItem markdown memoization", () => {
expect(queryByText("streamed answer")).not.toBeNull();
});
});
// PERF SMOKE (#492): the whole point of the incremental streaming render is that
// the ANSWER path costs O(number of markdown blocks), NOT O(number of throttled
// ~20Hz ticks). Pre-#492 the finalized MarkdownPart re-parsed the WHOLE growing
// answer on every delta — a synthetic ~100 KB stream measured 394 renderChatMarkdown
// calls (one per tick). With the incremental render each STABILIZED block is parsed
// exactly once (memoized in MarkdownChunk) and the live tail is cheap plain text, so
// the call count collapses to ~= the block count regardless of tick granularity.
describe("MessageItem streaming answer render is O(blocks), not O(ticks)", () => {
// ~100 KB answer. Each section is a heading + a paragraph — TWO blank-line
// delimited markdown blocks — so the safe-cut block count is ~2× the section
// count. The perf claim is about the BLOCK count (the memoization granularity),
// measured directly with splitPlainChunks below, not the section count.
const buildAnswer = () => {
const SECTIONS = 100;
const paragraphs: string[] = [];
for (let i = 0; i < SECTIONS; i++) {
paragraphs.push(`## Section ${i}\n\n` + "lorem ipsum dolor ".repeat(55));
}
const full = paragraphs.join("\n\n");
// The number of memoized markdown blocks the incremental render splits into
// (all but the live tail are parsed once each).
return { full, blocks: splitPlainChunks(full).length };
};
const streamMsg = (text: string, state: "streaming" | "done"): UIMessage =>
({
id: "m1",
role: "assistant",
parts: [{ type: "text", text, state }],
}) as UIMessage;
it("parses each block ~once over a 100KB stream (≈blocks, ≪ ticks)", () => {
renderChatMarkdownSpy.mockClear();
const { full, blocks } = buildAnswer();
const CHUNK = 128; // a realistic ~20Hz throttled delta size
const ticks = Math.ceil(full.length / CHUNK);
let msg = streamMsg(full.slice(0, CHUNK), "streaming");
const { rerender } = render(
<MantineProvider>
<MessageItem
message={msg}
signature={messageSignature(msg)}
turnStreaming
/>
</MantineProvider>,
);
for (let end = 2 * CHUNK; end < full.length; end += CHUNK) {
msg = streamMsg(full.slice(0, end), "streaming");
rerender(
<MantineProvider>
<MessageItem
message={msg}
signature={messageSignature(msg)}
turnStreaming
/>
</MantineProvider>,
);
}
// Finalize: the streaming→done flip renders the whole answer through ONE
// canonical pass (visual parity), so the finished DOM matches the pre-#492
// output. This is the single extra parse on top of the per-block ones.
const done = streamMsg(full, "done");
rerender(
<MantineProvider>
<MessageItem message={done} signature={messageSignature(done)} />
</MantineProvider>,
);
const calls = renderChatMarkdownSpy.mock.calls.length;
// Sanity: the stream really had far more ticks than blocks (else the test is
// vacuous — the point is that calls scale with blocks, not ticks).
expect(ticks).toBeGreaterThan(blocks * 3);
// O(blocks): each stabilized block parsed once + the single final whole-text
// parse. A small constant absorbs the finalize render and the live-tail block;
// the load-bearing claim is the bound below.
expect(calls).toBeLessThanOrEqual(blocks + 2);
// ≪ ticks — and, non-vacuously, the blocks WERE parsed (not skipped entirely).
expect(calls).toBeLessThan(ticks / 3);
expect(calls).toBeGreaterThan(blocks / 2);
// MUTATION-VERIFY (documented, not run here): dropping the `memo()` wrapper on
// MarkdownChunk (so every stable block re-parses each tick) drives `calls`
// toward `ticks` (~394), reddening both upper-bound assertions above.
});
});
@@ -0,0 +1,112 @@
import { describe, expect, it, vi } from "vitest";
import { render } from "@testing-library/react";
import { MantineProvider } from "@mantine/core";
import type { UIMessage } from "@ai-sdk/react";
// Stub react-i18next (the component reads `useTranslation`). Mirrors the other
// message-item specs.
vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (key: string) => key }),
}));
import MessageItem from "./message-item";
import { messageSignature } from "@/features/ai-chat/utils/message-signature.ts";
// The REAL canonical renderer (NOT the spy the memo test installs): this file
// exercises the actual markdown output so the visual-regression assertions below
// compare against genuine HTML (incl. the schema's `<li><p>` wrappers).
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
import classes from "./ai-chat.module.css";
const msg = (
parts: UIMessage["parts"],
extra?: Partial<UIMessage>,
): UIMessage =>
({ id: "m1", role: "assistant", parts, ...extra }) as UIMessage;
const renderRow = (message: UIMessage, turnStreaming = false) =>
render(
<MantineProvider>
<MessageItem
message={message}
signature={messageSignature(message)}
turnStreaming={turnStreaming}
/>
</MantineProvider>,
);
// A rich multi-block answer that exercises headings, a list (the `<li><p>` case
// the scoped CSS tightens), inline emphasis, and multiple paragraphs.
const ANSWER = [
"# Заголовок",
"",
"Первый абзац с **жирным** и `кодом`.",
"",
"- пункт один",
"- пункт два",
"",
"Второй абзац.",
].join("\n");
describe("MessageItem final render — visual parity with the canonical pipeline", () => {
it("a finalized text part renders exactly renderChatMarkdown(text)", () => {
const { container } = renderRow(
msg([{ type: "text", text: ANSWER, state: "done" }]),
);
const block = container.querySelector(`.${classes.markdown}`);
expect(block).not.toBeNull();
// Byte-for-byte the canonical output (the SAME whole-text pass the pre-#492
// MarkdownPart produced), including `<li><p>…</p></li>` wrappers.
expect(block!.innerHTML).toBe(renderChatMarkdown(ANSWER, {}));
// The list wrapper is really present (guards against a vacuous empty render).
expect(container.querySelectorAll("li p").length).toBe(2);
});
it("the streaming incremental view CONVERGES to the canonical render on finish", () => {
// Mount mid-stream (live tail) — the DOM here is the incremental view.
const { container, rerender } = render(
<MantineProvider>
<MessageItem
message={msg([{ type: "text", text: ANSWER, state: "streaming" }])}
signature={messageSignature(
msg([{ type: "text", text: ANSWER, state: "streaming" }]),
)}
turnStreaming
/>
</MantineProvider>,
);
// Finish the turn: state flips to done AND the turn is no longer streaming.
const done = msg([{ type: "text", text: ANSWER, state: "done" }]);
rerender(
<MantineProvider>
<MessageItem message={done} signature={messageSignature(done)} />
</MantineProvider>,
);
// After finish there is exactly ONE canonical markdown container whose HTML is
// the whole-text render — identical to the non-streaming path above.
const blocks = container.querySelectorAll(`.${classes.markdown}`);
expect(blocks.length).toBe(1);
expect(blocks[0].innerHTML).toBe(renderChatMarkdown(ANSWER, {}));
});
it("neutralizeInternalLinks is honored on the finalized render", () => {
const linkAnswer = "См. [страницу](/p/abc).";
const { container } = render(
<MantineProvider>
<MessageItem
message={msg([{ type: "text", text: linkAnswer, state: "done" }])}
signature={messageSignature(
msg([{ type: "text", text: linkAnswer, state: "done" }]),
)}
neutralizeInternalLinks
/>
</MantineProvider>,
);
const block = container.querySelector(`.${classes.markdown}`);
expect(block!.innerHTML).toBe(
renderChatMarkdown(linkAnswer, { neutralizeInternalLinks: true }),
);
// The internal link was made inert (no href) by the neutralization flag.
const a = container.querySelector("a");
expect(a?.hasAttribute("href")).toBe(false);
});
});
@@ -4,6 +4,7 @@ import { useTranslation } from "react-i18next";
import type { UIMessage } from "@ai-sdk/react";
import ToolCallCard from "@/features/ai-chat/components/tool-call-card.tsx";
import ReasoningBlock from "@/features/ai-chat/components/reasoning-block.tsx";
import { StreamingMarkdownText } from "@/features/ai-chat/components/streaming-markdown-text.tsx";
import ChatErrorAlert from "@/features/ai-chat/components/chat-error-alert.tsx";
import ChatStoppedNotice from "@/features/ai-chat/components/chat-stopped-notice.tsx";
import { ToolUiPart, isToolPart } from "@/features/ai-chat/utils/tool-parts.tsx";
@@ -86,17 +87,39 @@ interface MessageItemProps {
* One assistant text part rendered as sanitized markdown. Memoized on its inputs
* so a finalized text part is NOT re-parsed on every streamed delta: during a
* turn only the actively-growing tail part changes its `text`, so every earlier
* part hits the memo and skips the expensive marked + DOMPurify pass. Props are
* primitives, so React.memo's default shallow compare is exactly right (the
* `text` string is compared by value).
* part hits the memo and skips the expensive canonical parse + DOMPurify pass.
* Props are primitives, so React.memo's default shallow compare is exactly right
* (the `text` string is compared by value).
*
* Streaming gate (#492) — mirrors ReasoningBlock:
* - `streaming` (this is the live, actively-growing tail part of an in-flight
* turn): render incrementally via StreamingMarkdownText — the stabilized blocks
* go through the canonical pipeline (each parsed ONCE, memoized) and only the
* live tail is cheap plain text. This makes the per-tick cost O(new blocks),
* not the pre-#492 O(ticks) whole-answer re-parse on every ~20Hz delta.
* - finalized (the common case, and the turn-end flip): render the WHOLE text
* through ONE canonical pass — byte-identical to the pre-#492 output (visual
* parity). The row re-renders on the streaming→done flip because
* `messageSignature` tracks each part's `state` (and `turnStreaming` flips at
* turn end), so the incremental view always converges to this single render.
*/
const MarkdownPart = memo(function MarkdownPart({
text,
neutralizeInternalLinks,
streaming,
}: {
text: string;
neutralizeInternalLinks: boolean;
streaming: boolean;
}) {
if (streaming) {
return (
<StreamingMarkdownText
text={text}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
);
}
const html = renderChatMarkdown(text, { neutralizeInternalLinks });
if (html) {
return (
@@ -179,47 +202,10 @@ function MessageItem({
{resolveAssistantName(assistantName) ?? t("AI agent")}
</Text>
{message.parts.map((part, index) => {
if (part.type === "reasoning") {
// Reasoning ("thinking") -> a collapsible block with its own token
// count. Empty/whitespace reasoning with no authoritative count carries
// nothing to show, so skip it (avoids an empty 0-token block).
const text = (part as { text?: string }).text ?? "";
if (!text.trim() && !(reasoningTokens && reasoningTokens > 0))
return null;
// Absent state (persisted rows) and "done" both mean finalized.
// `messageSignature` already includes each part's `state`, so the
// streaming→done flip changes the row signature and re-renders this
// row — which is what lets ReasoningBlock switch from chunked plain
// text to its one-time markdown parse (see reasoning-block.tsx).
// ALSO require the turn to be live: a part stranded at
// `state:"streaming"` after the turn ended (no `reasoning-end` — see
// the `turnStreaming` prop doc) must still finalize and parse.
const streaming =
turnStreaming && (part as { state?: string }).state === "streaming";
return (
<ReasoningBlock
key={index}
text={text}
tokens={reasoningTokens}
streaming={streaming}
/>
);
}
if (part.type === "text") {
// Skip empty/whitespace-only text parts (a streaming message often
// starts with an empty text part before the first token arrives); the
// typing indicator covers that gap until real content streams in.
if (!part.text.trim()) return null;
return (
<MarkdownPart
key={index}
text={part.text}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
);
}
// Tool parts (`tool-*` / `dynamic-tool`) are template-literal kinds, so
// they cannot be a `switch` case; the runtime guard handles them, and the
// switch below covers every CLOSED (literal-typed) part kind with a
// compile-time exhaustiveness check in its default.
if (isToolPart(part.type)) {
return (
<ToolCallCard
@@ -232,7 +218,76 @@ function MessageItem({
);
}
return null;
switch (part.type) {
case "reasoning": {
// Reasoning ("thinking") -> a collapsible block with its own token
// count. Empty/whitespace reasoning with no authoritative count
// carries nothing to show, so skip it (avoids an empty 0-token block).
const text = part.text ?? "";
if (!text.trim() && !(reasoningTokens && reasoningTokens > 0))
return null;
// Absent state (persisted rows) and "done" both mean finalized.
// `messageSignature` already includes each part's `state`, so the
// streaming→done flip changes the row signature and re-renders this
// row — which is what lets ReasoningBlock switch from chunked plain
// text to its one-time markdown parse (see reasoning-block.tsx).
// ALSO require the turn to be live: a part stranded at
// `state:"streaming"` after the turn ended (no `reasoning-end` — see
// the `turnStreaming` prop doc) must still finalize and parse.
const streaming = turnStreaming && part.state === "streaming";
return (
<ReasoningBlock
key={index}
text={text}
tokens={reasoningTokens}
streaming={streaming}
/>
);
}
case "text": {
// Skip empty/whitespace-only text parts (a streaming message often
// starts with an empty text part before the first token arrives); the
// typing indicator covers that gap until real content streams in.
if (!part.text.trim()) return null;
// The live, actively-growing tail part of the in-flight turn renders
// incrementally (see MarkdownPart); a finalized part (persisted, or
// the turn-end flip) renders the whole text through one canonical
// pass. Same liveness rule as the reasoning branch above.
const streaming = turnStreaming && part.state === "streaming";
return (
<MarkdownPart
key={index}
text={part.text}
neutralizeInternalLinks={neutralizeInternalLinks}
streaming={streaming}
/>
);
}
case "source-url":
case "source-document":
case "file":
case "step-start":
// Not surfaced in the chat bubble (v1) — same as the pre-#492 default.
return null;
default: {
// Compile-time exhaustiveness over the CLOSED union members: every
// literal-typed part kind is handled above, so the only kinds that
// can reach here are the OPEN template-literal ones (`tool-*` — caught
// by the guard at runtime — and `data-*`) plus `dynamic-tool`. Adding
// a NEW closed part kind to UIMessagePart makes this assignment fail
// to compile, forcing it to be handled instead of silently ignored
// (this replaces the pre-#492 fall-through `return null` + WARNING).
const _exhaustive:
| `tool-${string}`
| "dynamic-tool"
| `data-${string}` = part.type;
void _exhaustive;
return null;
}
}
})}
{/* A persisted turn error (server stored it in metadata.error). Rendered
here so it survives a thread remount and shows in reopened history. */}
@@ -0,0 +1,96 @@
import { memo, useMemo } from "react";
import { splitPlainChunks } from "@/features/ai-chat/components/streaming-plain-text.tsx";
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
import classes from "@/features/ai-chat/components/ai-chat.module.css";
/**
* One STABILIZED markdown block, rendered through the canonical pipeline and
* memoized on its string prop. During streaming only the TAIL chunk grows (the
* `splitPlainChunks` append-only invariant guarantees every earlier chunk is
* byte-identical across deltas), so React skips every stable block and each one
* is parsed by `renderChatMarkdown` EXACTLY ONCE — turning the pre-#492
* "re-parse the whole accumulated answer on every ~20Hz tick" (O(ticks)) into
* O(number of blocks). The markup is DOMPurify-sanitized inside renderChatMarkdown
* before it reaches `dangerouslySetInnerHTML`.
*
* NOTE (transient streaming-only artifact): a safe cut is a blank-line boundary,
* so a construct that legitimately contains a blank line (e.g. a fenced code block
* with an empty line) can be split across chunks and render oddly WHILE it is still
* streaming. This is cosmetic and self-heals: the moment the part finalizes,
* MarkdownPart renders the WHOLE text through one canonical pass (visual parity
* with the pre-#492 output). The reasoning path makes the same trade (plain text
* while streaming, one markdown parse at the end).
*/
const MarkdownChunk = memo(function MarkdownChunk({
text,
neutralizeInternalLinks,
}: {
text: string;
neutralizeInternalLinks: boolean;
}) {
const html = renderChatMarkdown(text, { neutralizeInternalLinks });
if (html) {
return (
<div
className={classes.markdown}
// Sanitized by renderChatMarkdown (DOMPurify) before insertion.
dangerouslySetInnerHTML={{ __html: html }}
/>
);
}
// Malformed/unsupported markdown could not render synchronously: raw text.
return (
<div className={classes.markdown} style={{ whiteSpace: "pre-wrap" }}>
{text}
</div>
);
});
/**
* The cheap streaming-time stand-in for the finalized answer's one-time markdown
* parse (see MarkdownPart in message-item.tsx). Mirrors StreamingPlainText's
* chunked-memo pattern but renders the STABILIZED prefix as real markdown (each
* block parsed once, memoized) and only the LIVE tail as flat plain text — so the
* user sees formatted output for everything up to the last safe cut, and the not-
* yet-stable tail (which markdown-parsing every tick would make O(ticks)) stays a
* single cheap escaped text node until it stabilizes into a new block.
*
* `splitPlainChunks` yields chunks where, under append-only growth, every chunk
* except the LAST is immutable; the last chunk is the live tail. Index keys are
* therefore stable (a given index never changes to a different chunk's content).
*/
export function StreamingMarkdownText({
text,
neutralizeInternalLinks,
}: {
text: string;
neutralizeInternalLinks: boolean;
}) {
const chunks = useMemo(() => splitPlainChunks(text), [text]);
return (
<>
{chunks.map((chunk, index) =>
index < chunks.length - 1 ? (
<MarkdownChunk
key={index}
text={chunk}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
) : (
// The live tail: flat, React-escaped plain text (no markdown parse, no
// sanitizer, no innerHTML). `pre-wrap` preserves its newlines; trailing
// separator newlines are dropped at display time so the block gap comes
// from the markdown margins, not a doubled empty line (mirrors
// PlainChunk in streaming-plain-text.tsx).
<div
key={index}
className={classes.markdown}
style={{ whiteSpace: "pre-wrap" }}
>
{chunk.replace(/\n+$/, "")}
</div>
),
)}
</>
);
}
@@ -27,11 +27,15 @@ vi.mock("@/features/space/queries/space-query.ts", () => ({
import {
buildChildrenByParent,
CommentEditorWithActions,
sortResolvedByResolvedAt,
} from "./comment-list-with-tabs";
const c = (id: string, parentCommentId: string | null = null): IComment =>
({ id, parentCommentId }) as IComment;
const resolvedAtComment = (id: string, resolvedAt: unknown): IComment =>
({ id, resolvedAt }) as unknown as IComment;
describe("buildChildrenByParent (childrenByParent grouping)", () => {
it("returns an empty map for undefined or empty input", () => {
expect(buildChildrenByParent(undefined).size).toBe(0);
@@ -71,6 +75,48 @@ describe("buildChildrenByParent (childrenByParent grouping)", () => {
});
});
describe("sortResolvedByResolvedAt (Resolved tab order, #542)", () => {
it("orders by resolvedAt DESC — newest resolve first — with ISO-STRING values", () => {
// At runtime resolvedAt is an ISO string (axios JSON / WS subscription), so
// the sort must coerce with new Date(...) before .getTime().
const older = resolvedAtComment("older", "2026-07-10T10:00:00.000Z");
const newest = resolvedAtComment("newest", "2026-07-12T10:00:00.000Z");
const middle = resolvedAtComment("middle", "2026-07-11T10:00:00.000Z");
const out = sortResolvedByResolvedAt([older, newest, middle]);
expect(out.map((x) => x.id)).toEqual(["newest", "middle", "older"]);
});
it("also handles Date instances (optimistic onMutate window)", () => {
const older = resolvedAtComment("older", new Date("2026-01-01T00:00:00Z"));
const newer = resolvedAtComment("newer", new Date("2026-06-01T00:00:00Z"));
expect(sortResolvedByResolvedAt([older, newer]).map((x) => x.id)).toEqual([
"newer",
"older",
]);
});
it("does not mutate the input array", () => {
const a = resolvedAtComment("a", "2026-01-01T00:00:00.000Z");
const b = resolvedAtComment("b", "2026-02-01T00:00:00.000Z");
const input = [a, b];
sortResolvedByResolvedAt(input);
expect(input.map((x) => x.id)).toEqual(["a", "b"]);
});
it("keeps stable order for equal resolvedAt timestamps", () => {
const ts = "2026-03-03T03:03:03.000Z";
const x = resolvedAtComment("x", ts);
const y = resolvedAtComment("y", ts);
const z = resolvedAtComment("z", ts);
expect(sortResolvedByResolvedAt([x, y, z]).map((c) => c.id)).toEqual([
"x",
"y",
"z",
]);
});
});
function renderReplyEditor() {
return render(
<MantineProvider>
@@ -53,6 +53,22 @@ export function buildChildrenByParent(
return m;
}
// Sort the Resolved tab by resolve time, newest first, on a COPY (never mutate
// the react-query cache array). `resolvedAt` is typed `Date` but at runtime it
// is an ISO STRING (from the axios-JSON onSuccess and the WS subscription) — a
// real Date only during the optimistic onMutate window — so it MUST be coerced
// with `new Date(...)` before `.getTime()`, or a raw `.getTime()` on the string
// throws / yields NaN. ES2019's stable sort preserves order for equal
// timestamps. Callers pass a list already filtered to a truthy `resolvedAt`, so
// the non-null assertion is safe.
// Exported for unit testing.
export function sortResolvedByResolvedAt(resolved: IComment[]): IComment[] {
return [...resolved].sort(
(a, b) =>
new Date(b.resolvedAt!).getTime() - new Date(a.resolvedAt!).getTime(),
);
}
function CommentListWithTabs({ onClose }: CommentListWithTabsProps) {
const { t } = useTranslation();
const { pageSlug } = useParams();
@@ -91,7 +107,10 @@ function CommentListWithTabs({ onClose }: CommentListWithTabsProps) {
(comment: IComment) => comment.resolvedAt,
);
return { activeComments: active, resolvedComments: resolved };
return {
activeComments: active,
resolvedComments: sortResolvedByResolvedAt(resolved),
};
}, [comments]);
// Index replies by their parent once, instead of an O(n^2) filter per thread.
@@ -0,0 +1,353 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import React from "react";
import { renderHook, waitFor } from "@testing-library/react";
import {
QueryClient,
QueryClientProvider,
InfiniteData,
} from "@tanstack/react-query";
/**
* Coverage for the resolve/reopen mutation (#542): the Undo-in-toast reopen and
* its double-click guard, the terminal 404 branch (drop from cache + clear the
* inline mark, no rollback), and the directional error copy.
*/
// A fake TipTap editor injected via the mocked pageEditorAtom, so we can assert
// the mutation clears the inline comment mark (unsetComment / setCommentResolved).
const editorMock = vi.hoisted(() => ({
current: {
isDestroyed: false,
commands: { unsetComment: vi.fn(), setCommentResolved: vi.fn() },
} as {
isDestroyed: boolean;
commands: {
unsetComment: (id: string) => void;
setCommentResolved: (id: string, v: boolean) => void;
};
} | null,
}));
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn(), hide: vi.fn() },
}));
vi.mock("jotai", () => ({
atom: (v: unknown) => v,
useAtomValue: () => editorMock.current,
}));
vi.mock("@/features/comment/services/comment-service", () => ({
applySuggestion: vi.fn(),
dismissSuggestion: vi.fn(),
createComment: vi.fn(),
updateComment: vi.fn(),
deleteComment: vi.fn(),
resolveComment: vi.fn(),
getPageComments: vi.fn(),
}));
import { notifications } from "@mantine/notifications";
import { resolveComment } from "@/features/comment/services/comment-service";
import {
useResolveCommentMutation,
RESOLVE_UNDO_AUTOCLOSE_MS,
RQ_KEY,
} from "@/features/comment/queries/comment-query";
import { IComment } from "@/features/comment/types/comment.types";
const PAGE_ID = "page-1";
function seededClient(comment: IComment) {
const queryClient = new QueryClient({
defaultOptions: { mutations: { retry: false } },
});
const seed: InfiniteData<any> = {
pageParams: [undefined],
pages: [
{ items: [comment], meta: { hasNextPage: false, nextCursor: null } },
],
};
queryClient.setQueryData(RQ_KEY(PAGE_ID), seed);
const wrapper = ({ children }: { children: React.ReactNode }) => (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
);
return { queryClient, wrapper };
}
function items(queryClient: QueryClient): IComment[] {
const cache = queryClient.getQueryData(RQ_KEY(PAGE_ID)) as
| InfiniteData<any>
| undefined;
return cache?.pages.flatMap((p) => p.items) ?? [];
}
const comment = (over?: Partial<IComment>): IComment =>
({
id: "c-1",
pageId: PAGE_ID,
content: "{}",
creatorId: "u-1",
workspaceId: "ws-1",
createdAt: new Date(),
resolvedAt: null,
...over,
}) as IComment;
// Pull the inline Undo button's onClick out of the success toast's message tree.
function undoOnClickFromToast(): () => void {
const call = vi
.mocked(notifications.show)
.mock.calls.map((c) => c[0])
.find((arg: any) => arg?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS);
expect(call).toBeTruthy();
const message: any = (call as any).message;
// message = Group( Text, Button ); grab the Button element's onClick.
const children = message.props.children as any[];
const button = children[1];
return button.props.onClick;
}
describe("useResolveCommentMutation — Undo toast (#542)", () => {
beforeEach(() => {
vi.clearAllMocks();
editorMock.current = {
isDestroyed: false,
commands: { unsetComment: vi.fn(), setCommentResolved: vi.fn() },
};
});
it("resolve shows an Undo toast with autoClose=10000ms; reopen shows NO Undo", async () => {
vi.mocked(resolveComment).mockImplementation(async (data) =>
comment({
resolvedAt: data.resolved ? (new Date() as any) : null,
}),
);
const { wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const resolveToast = vi
.mocked(notifications.show)
.mock.calls.map((c) => c[0])
.find((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS);
expect(resolveToast).toBeTruthy();
expect((resolveToast as any).id).toBe("resolve-undo-c-1");
expect((resolveToast as any).autoClose).toBe(10000);
// Now a reopen → plain toast, no autoClose/Undo, no id.
vi.clearAllMocks();
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: false,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const calls = vi.mocked(notifications.show).mock.calls.map((c) => c[0]);
expect(
calls.some((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS),
).toBe(false);
expect(calls).toContainEqual({ message: "Comment re-opened successfully" });
});
it("double/fast Undo click fires reopen EXACTLY once (guard)", async () => {
vi.mocked(resolveComment).mockImplementation(async (data) =>
comment({ resolvedAt: data.resolved ? (new Date() as any) : null }),
);
const { wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const onClick = undoOnClickFromToast();
// Fire twice synchronously (notifications.hide is not synchronous).
onClick();
onClick();
await waitFor(() => {
const reopenCalls = vi
.mocked(resolveComment)
.mock.calls.filter(([d]) => d.resolved === false);
expect(reopenCalls).toHaveLength(1);
});
// The mark was cleared once via setCommentResolved(id, false).
expect(editorMock.current!.commands.setCommentResolved).toHaveBeenCalledWith(
"c-1",
false,
);
// The toast was hidden.
expect(notifications.hide).toHaveBeenCalledWith("resolve-undo-c-1");
});
it("404 → drops the comment from cache, clears the inline mark, no rollback, no Undo", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 404 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
// Removed from cache (NOT rolled back to a phantom).
expect(items(queryClient)).toHaveLength(0);
// Inline mark cleared via unsetComment (mandatory — no panel row left to do it).
expect(editorMock.current!.commands.unsetComment).toHaveBeenCalledWith(
"c-1",
);
// Neutral message, red, and crucially NOT the success copy and NO Undo toast.
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment no longer exists",
color: "red",
});
const calls = vi.mocked(notifications.show).mock.calls.map((c) => c[0]);
expect(
calls.some((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS),
).toBe(false);
});
it("404 does not crash when the editor is gone (read-only / panel closed)", async () => {
editorMock.current = null;
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 404 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(items(queryClient)).toHaveLength(0);
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment no longer exists",
color: "red",
});
});
it("non-404 error on REOPEN shows 'Failed to re-open comment' and rolls back", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
// Seed a RESOLVED comment (the reopen target).
const resolved = comment({ resolvedAt: new Date() as any });
const { queryClient, wrapper } = seededClient(resolved);
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: false })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(notifications.show).toHaveBeenCalledWith({
message: "Failed to re-open comment",
color: "red",
});
expect(notifications.show).not.toHaveBeenCalledWith(
expect.objectContaining({ message: "Failed to resolve comment" }),
);
// Rolled back: the comment is still present and still resolved.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeTruthy();
});
it("reopen via Undo FAILS (non-404) → inline mark is NOT left cleared (doc↔panel stay consistent)", async () => {
// First resolve succeeds → produces the Undo toast (no mark change on resolve).
vi.mocked(resolveComment).mockResolvedValueOnce(
comment({ resolvedAt: new Date() as any }),
);
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
// Now the reopen fired by Undo fails with a 500.
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
const onClick = undoOnClickFromToast();
onClick();
await waitFor(() => expect(result.current.isError).toBe(true));
// Core F1 guarantee: the mark-clear now lives in the reopen onSuccess, so a
// FAILED reopen must never flip the inline mark to unresolved — otherwise the
// doc would show an active highlight the panel still treats as resolved and
// the collab mark would diverge with nothing committed on the server.
expect(
editorMock.current!.commands.setCommentResolved,
).not.toHaveBeenCalledWith("c-1", false);
// Cache rolled back: the comment stays resolved and present.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeTruthy();
});
it("reopen success with a null editorRef degrades gracefully (no throw, no-op)", async () => {
// Read-only view / panel closed: pageEditorAtom is null on the success path.
editorMock.current = null;
vi.mocked(resolveComment).mockResolvedValue(comment({ resolvedAt: null }));
const resolved = comment({ resolvedAt: new Date() as any });
const { queryClient, wrapper } = seededClient(resolved);
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: false,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
// No crash from the reopen mark-clear; the plain reopen toast is still shown.
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment re-opened successfully",
});
// Cache updated to reopened (resolvedAt cleared by the server payload).
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeFalsy();
});
it("non-404 error on RESOLVE shows 'Failed to resolve comment' and rolls back", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(notifications.show).toHaveBeenCalledWith({
message: "Failed to resolve comment",
color: "red",
});
// Rolled back to open (previousCache), still present.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeFalsy();
});
});
@@ -20,12 +20,19 @@ import {
ISuggestionOutcome,
} from "@/features/comment/types/comment.types";
import { notifications } from "@mantine/notifications";
import { Button, Group, Text } from "@mantine/core";
import { IPagination } from "@/lib/types.ts";
import { useTranslation } from "react-i18next";
import { useEffect, useMemo } from "react";
import React, { useEffect, useMemo, useRef } from "react";
import { useAtomValue } from "jotai";
import { pageEditorAtom } from "@/features/editor/atoms/editor-atoms";
export const RQ_KEY = (pageId: string) => ["comments", pageId];
// How long the resolve success toast (with its inline Undo) stays up before it
// auto-closes. Policy constant — no env override.
export const RESOLVE_UNDO_AUTOCLOSE_MS = 10000;
export function useCommentsQuery(params: ICommentParams) {
const query = useInfiniteQuery({
queryKey: RQ_KEY(params.pageId),
@@ -376,7 +383,25 @@ export function useResolveCommentMutation() {
const queryClient = useQueryClient();
const { t } = useTranslation();
return useMutation({
// Keep the live editor in a ref: the toast's Undo (and the 404 branch) must
// clear the inline comment mark AFTER the originating CommentListItem has
// unmounted (resolving pulls the comment out of the Open list, so its item is
// already gone by the time the 10s toast is clicked). In read-only view
// pageEditorAtom is null and the mark converges via the server's
// COMMENT_MARK_UPDATE job instead.
const editor = useAtomValue(pageEditorAtom);
const editorRef = useRef(editor);
editorRef.current = editor;
// Self-reference the mutation so the toast's Undo can re-invoke it (reopen)
// long after the triggering component unmounted. Declared BEFORE useMutation
// and assigned AFTER; the onClick reads mutationRef.current at CALL time, not
// definition time, so there is no initialization cycle.
const mutationRef = useRef<{
mutate: (vars: IResolveComment) => void;
} | null>(null);
const mutation = useMutation({
mutationFn: (data: IResolveComment) => resolveComment(data),
onMutate: async (variables) => {
await queryClient.cancelQueries({ queryKey: RQ_KEY(variables.pageId) });
@@ -401,7 +426,39 @@ export function useResolveCommentMutation() {
return { previousCache };
},
onError: (_err, variables, context) => {
onError: (err: any, variables, context) => {
// Terminal 404: the comment was really deleted (missing comment or deleted
// page — access denial is 403, resolve is idempotent so no 400). Do NOT
// roll back (that would resurrect a phantom row in Resolved); instead drop
// it from the cache and clear its now-orphaned inline mark. Mirrors
// handleDeleteComment and the dismiss-mutation 404 branch.
if (err?.response?.status === 404) {
const cache = queryClient.getQueryData(RQ_KEY(variables.pageId)) as
| InfiniteData<IPagination<IComment>>
| undefined;
if (cache) {
queryClient.setQueryData(
RQ_KEY(variables.pageId),
removeCommentFromCache(cache, variables.commentId),
);
}
const ed = editorRef.current;
if (ed && !ed.isDestroyed) {
try {
ed.commands.unsetComment(variables.commentId);
} catch {
/* editor gone / mark already removed */
}
}
notifications.show({
message: t("Comment no longer exists"),
color: "red",
});
return;
}
// Generic failure: roll back the optimistic update and show a DIRECTIONAL
// error (resolve vs. reopen), not always "resolve".
if (context?.previousCache) {
queryClient.setQueryData(
RQ_KEY(variables.pageId),
@@ -409,7 +466,9 @@ export function useResolveCommentMutation() {
);
}
notifications.show({
message: t("Failed to resolve comment"),
message: variables.resolved
? t("Failed to resolve comment")
: t("Failed to re-open comment"),
color: "red",
});
},
@@ -430,11 +489,72 @@ export function useResolveCommentMutation() {
);
}
// Reopen keeps the plain toast without an Undo.
if (!variables.resolved) {
// Clear the inline mark ONLY after the server confirms the reopen, so a
// failed reopen never leaves an active highlight the panel still treats
// as resolved. Mirrors the 404 branch's editor-liveness guard/try-catch.
// The button-triggered reopen already set the mark, so this is an
// idempotent no-op there.
const ed = editorRef.current;
if (ed && !ed.isDestroyed) {
try {
ed.commands.setCommentResolved(variables.commentId, false);
} catch {
/* editor gone — server COMMENT_MARK_UPDATE converges it */
}
}
notifications.show({ message: t("Comment re-opened successfully") });
return;
}
// Resolve: attach an inline Undo (reopen) to the success toast. Built with
// React.createElement because this is a .ts module (no JSX).
const { commentId, pageId } = variables;
const notificationId = `resolve-undo-${commentId}`;
// Double-click guard: notifications.hide is NOT synchronous, so the button
// stays clickable for a frame or two — without this a fast double-click
// would fire reopen twice.
let done = false;
notifications.show({
message: variables.resolved
? t("Comment resolved successfully")
: t("Comment re-opened successfully"),
id: notificationId,
autoClose: RESOLVE_UNDO_AUTOCLOSE_MS,
message: React.createElement(
Group,
{ justify: "space-between", wrap: "nowrap", gap: "md" },
React.createElement(
Text,
{ size: "sm" },
t("Comment resolved successfully"),
),
React.createElement(
Button,
{
variant: "subtle",
size: "compact-sm",
onClick: () => {
if (done) return;
done = true;
// Reopen via the SAME mutation (read at click time — the
// originating item is already unmounted).
mutationRef.current?.mutate({
commentId,
pageId,
resolved: false,
});
// The inline mark is cleared in the reopen mutation's onSuccess
// (bound to server confirmation), NOT here — clearing it eagerly
// would desync the doc from the panel if reopen then fails.
notifications.hide(notificationId);
},
},
t("Undo"),
),
),
});
},
});
mutationRef.current = mutation;
return mutation;
}
@@ -0,0 +1,45 @@
import { describe, it, expect } from "vitest";
import {
formatHeadline,
formatDayTotal,
formatGapMinutes,
} from "./format-work-time";
const MIN = 60 * 1000;
// Fake translator: renders the key with {{tokens}} substituted, so the tests
// assert the rounding + branch selection without depending on the i18n catalogue.
const t = (key: string, opts?: Record<string, unknown>) =>
key.replace(/\{\{(\w+)\}\}/g, (_, k) => String(opts?.[k] ?? ""));
describe("formatHeadline", () => {
it("prefixes ≈ and rounds to a 5-minute step", () => {
expect(formatHeadline(4 * 60 * MIN + 27 * MIN, t)).toBe("≈ 4h 25m");
expect(formatHeadline(90 * MIN, t)).toBe("≈ 1h 30m");
});
it("shows hours only / minutes only cleanly", () => {
expect(formatHeadline(120 * MIN, t)).toBe("≈ 2h");
expect(formatHeadline(35 * MIN, t)).toBe("≈ 35m");
});
it("floors a tiny non-zero estimate to 5m, never 0", () => {
expect(formatHeadline(2 * MIN, t)).toBe("≈ 5m");
});
it("empty string for zero (widget hidden)", () => {
expect(formatHeadline(0, t)).toBe("");
});
});
describe("formatDayTotal", () => {
it('renders "h m" and shows — for empty days', () => {
expect(formatDayTotal(3 * 60 * MIN + 17 * MIN, t)).toBe("3h 17m");
expect(formatDayTotal(0, t)).toBe("—");
});
});
describe("formatGapMinutes", () => {
it("converts the tGap ms threshold to whole minutes", () => {
expect(formatGapMinutes(15 * MIN)).toBe(15);
});
});
@@ -0,0 +1,45 @@
// #395 — display formatting for the work-time estimate. Pure functions that take
// a translator so ru-RU / en-US wording lives in the i18n catalogue and the
// rounding logic stays unit-testable.
type Translate = (key: string, opts?: Record<string, unknown>) => string;
const MIN = 60 * 1000;
function hm(totalMinutes: number): { hours: number; minutes: number } {
return {
hours: Math.floor(totalMinutes / 60),
minutes: totalMinutes % 60,
};
}
/**
* Headline number (§6.1): an ESTIMATE, so rounded to a coarse 5-minute step and
* prefixed with "≈". A non-zero-but-tiny estimate floors to 5m rather than
* rounding down to "0" (which would read as "no work"). Zero → empty string
* (the caller hides the widget).
*/
export function formatHeadline(workMs: number, t: Translate): string {
if (workMs <= 0) return "";
let minutes = Math.round(workMs / MIN / 5) * 5;
if (minutes === 0) minutes = 5;
const { hours, minutes: m } = hm(minutes);
if (hours > 0 && m > 0) return t("≈ {{hours}}h {{minutes}}m", { hours, minutes: m });
if (hours > 0) return t("≈ {{hours}}h", { hours });
return t("≈ {{minutes}}m", { minutes: m });
}
/** Per-day sum (§6.2), rounded to the minute. Zero → "—". */
export function formatDayTotal(activeMs: number, t: Translate): string {
if (activeMs <= 0) return "—";
const minutes = Math.max(1, Math.round(activeMs / MIN));
const { hours, minutes: m } = hm(minutes);
if (hours > 0 && m > 0) return t("{{hours}}h {{minutes}}m", { hours, minutes: m });
if (hours > 0) return t("{{hours}}h", { hours });
return t("{{minutes}}m", { minutes: m });
}
/** The inactivity threshold, for the "estimate · gap = N min" caption. */
export function formatGapMinutes(tGapMs: number): number {
return Math.round(tGapMs / MIN);
}
@@ -0,0 +1,25 @@
import { useQuery, UseQueryResult } from "@tanstack/react-query";
import { IPageWorkTime } from "./work-time.types";
import { getPageWorkTime, viewerTimezone } from "./work-time-service";
const WORK_TIME_STALE_TIME = 5 * 60 * 1000;
/**
* #395 — the "time worked on this article" estimate + per-day punch-card
* buckets. The buckets are computed server-side in the viewer's timezone (so a
* midnight-crossing session lands on the right calendar day for the reader).
* `enabled` is opt-in so the (cheap but non-trivial) projection query only fires
* when the number is actually shown.
*/
export function usePageWorkTime(
pageId: string,
enabled = true,
): UseQueryResult<IPageWorkTime, Error> {
const tz = viewerTimezone();
return useQuery({
queryKey: ["page-work-time", pageId, tz],
queryFn: () => getPageWorkTime(pageId, tz),
enabled: enabled && !!pageId,
staleTime: WORK_TIME_STALE_TIME,
});
}
@@ -0,0 +1,171 @@
import { Group, Stack, Text } from "@mantine/core";
import { useTranslation } from "react-i18next";
import { useMemo } from "react";
import { IPageWorkTime, IPerDay, IDayWindow } from "./work-time.types";
import {
formatDayTotal,
formatGapMinutes,
formatHeadline,
} from "./format-work-time";
import classes from "./work-time.module.css";
const DAY_MS = 24 * 60 * 60 * 1000;
// Collapse a run of this many (or more) consecutive edit-free days into a single
// "× N days" separator (§6.2 long-range) — the row is still always one day.
const EMPTY_RUN_COLLAPSE = 8;
type Row =
| { type: "day"; day: IPerDay }
| { type: "gap"; count: number };
function collapseEmptyRuns(perDay: IPerDay[]): Row[] {
const rows: Row[] = [];
let emptyRun: IPerDay[] = [];
const flush = () => {
if (emptyRun.length >= EMPTY_RUN_COLLAPSE) {
rows.push({ type: "gap", count: emptyRun.length });
} else {
for (const d of emptyRun) rows.push({ type: "day", day: d });
}
emptyRun = [];
};
for (const d of perDay) {
if (d.activeMs === 0 && d.agentMs === 0) {
emptyRun.push(d);
} else {
flush();
rows.push({ type: "day", day: d });
}
}
flush();
return rows;
}
function dayHeading(day: number): string {
return new Date(day).toLocaleDateString(undefined, {
weekday: "short",
day: "numeric",
month: "short",
});
}
function DayTrack({
day,
pSingle,
}: {
day: IPerDay;
pSingle: number;
}) {
const { t } = useTranslation();
const ticks = [6, 12, 18];
return (
<div className={classes.row}>
<span className={classes.dayLabel}>{dayHeading(day.day)}</span>
<div className={classes.track}>
{ticks.map((h) => (
<div
key={h}
className={classes.hourTick}
style={{ left: `${(h / 24) * 100}%` }}
/>
))}
{day.windows.map((w: IDayWindow, i) => {
const leftPct = ((w.start - day.day) / DAY_MS) * 100;
const widthPct = ((w.end - w.start) / DAY_MS) * 100;
const isSingle = w.end - w.start <= pSingle;
const cls = [
classes.window,
w.class === "work" ? classes.windowWork : classes.windowAgent,
isSingle ? classes.windowSingle : "",
].join(" ");
return (
<div
key={i}
className={cls}
style={{
left: `${Math.max(0, Math.min(100, leftPct))}%`,
width: `${Math.max(0, Math.min(100, widthPct))}%`,
}}
/>
);
})}
</div>
<span className={classes.daySum}>
{formatDayTotal(day.activeMs, t)}
</span>
</div>
);
}
interface Props {
data: IPageWorkTime;
}
export default function WorkTimePunchCard({ data }: Props) {
const { t } = useTranslation();
const rows = useMemo(() => collapseEmptyRuns(data.perDay), [data.perDay]);
const gapMin = formatGapMinutes(data.config.tGap);
if (data.workMs <= 0 && data.agentOnlyMs <= 0) {
return (
<Text size="sm" c="dimmed" py="md">
{t("No editing activity recorded yet.")}
</Text>
);
}
return (
<Stack gap="xs">
<Group gap="lg">
<Text size="sm" fw={500}>
{formatHeadline(data.workMs, t)}
</Text>
{data.agentOnlyMs > 0 && (
<Text size="xs" c="dimmed">
{t("agent: {{value}}", { value: formatHeadline(data.agentOnlyMs, t) })}
</Text>
)}
</Group>
<Group gap="md">
<Text size="xs" c="dimmed">
<span
className={`${classes.legendSwatch} ${classes.windowWork}`}
style={{ marginRight: 4 }}
/>
{t("Work")}
</Text>
<Text size="xs" c="dimmed">
<span
className={`${classes.legendSwatch} ${classes.windowAgent}`}
style={{ marginRight: 4 }}
/>
{t("Agent")}
</Text>
</Group>
<div>
{rows.map((row, i) =>
row.type === "day" ? (
<DayTrack
key={row.day.dayISO}
day={row.day}
pSingle={data.config.pSingle}
/>
) : (
<div key={`gap-${i}`} className={classes.gapRow}>
{t("× {{count}} days without edits", { count: row.count })}
</div>
),
)}
</div>
<Text size="xs" c="dimmed" mt="xs">
{t("Estimate · timezone {{tz}} · inactivity gap {{gap}} min", {
tz: data.tz,
gap: gapMin,
})}
</Text>
</Stack>
);
}
@@ -0,0 +1,23 @@
import api from "@/lib/api-client";
import { IPageWorkTime } from "./work-time.types";
/** The viewer's IANA timezone (browser locale) — the punch-card lays days out
* in "my evenings", per §6.3/§10. Falls back to UTC if the runtime hides it. */
export function viewerTimezone(): string {
try {
return Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC";
} catch {
return "UTC";
}
}
export async function getPageWorkTime(
pageId: string,
tz: string,
): Promise<IPageWorkTime> {
const req = await api.post<IPageWorkTime>("/pages/history/time", {
pageId,
tz,
});
return req.data;
}
@@ -0,0 +1,69 @@
import { Modal, Text, Tooltip, UnstyledButton } from "@mantine/core";
import { useDisclosure } from "@mantine/hooks";
import { IconClockHour4 } from "@tabler/icons-react";
import { useTranslation } from "react-i18next";
import { usePageWorkTime } from "./use-page-work-time";
import { formatGapMinutes, formatHeadline } from "./format-work-time";
import WorkTimePunchCard from "./work-time-punch-card";
interface Props {
pageId: string;
}
/**
* #395 — the clickable "time worked on this article" headline (§6.1). Renders
* the `work` estimate with a "≈" sign and the inactivity threshold in a tooltip
* (it is an estimate, not a stopwatch). Clicking opens the daily punch-card
* (§6.2). Renders nothing until there is a non-zero human OR agent estimate, so a
* brand-new / never-edited page shows no widget. For an agent-only-edited page
* (workMs===0, agentOnlyMs>0) the headline shows the agent estimate (labelled
* `agent:`, matching the punch-card) so the punch-card stays reachable (#395:
* "how much a HUMAN and separately the AGENT").
*/
export default function WorkTimeStat({ pageId }: Props) {
const { t } = useTranslation();
const [opened, { open, close }] = useDisclosure(false);
const { data } = usePageWorkTime(pageId);
if (!data || (data.workMs <= 0 && data.agentOnlyMs <= 0)) return null;
const agentOnly = data.workMs <= 0;
const label = agentOnly
? t("agent: {{value}}", { value: formatHeadline(data.agentOnlyMs, t) })
: formatHeadline(data.workMs, t);
const gapMin = formatGapMinutes(data.config.tGap);
return (
<>
<Tooltip
label={t("Estimated time worked (inactivity gap {{gap}} min)", {
gap: gapMin,
})}
position="bottom"
>
<UnstyledButton
onClick={open}
aria-label={t("Show time worked on this page")}
>
<Text
size="xs"
c="dimmed"
style={{ display: "inline-flex", alignItems: "center", gap: 4 }}
>
<IconClockHour4 size={14} />
{label}
</Text>
</UnstyledButton>
</Tooltip>
<Modal
opened={opened}
onClose={close}
title={t("Time worked on this article")}
size="lg"
>
<WorkTimePunchCard data={data} />
</Modal>
</>
);
}
@@ -0,0 +1,82 @@
/* #395 — 24h × days punch-card. Custom CSS segments on a fixed 24-hour track
(position = offset-in-day / 24h, width = duration / 24h), no chart library. */
.row {
display: grid;
grid-template-columns: 96px 1fr 64px;
align-items: center;
gap: 12px;
padding: 3px 0;
}
.dayLabel {
font-size: var(--mantine-font-size-xs);
color: var(--mantine-color-dimmed);
white-space: nowrap;
}
.track {
position: relative;
height: 16px;
border-radius: 4px;
background-color: light-dark(
var(--mantine-color-gray-1),
var(--mantine-color-dark-6)
);
overflow: hidden;
}
/* Faint hour grid so the eye can read "morning vs evening". */
.hourTick {
position: absolute;
top: 0;
bottom: 0;
width: 1px;
background-color: light-dark(
var(--mantine-color-gray-3),
var(--mantine-color-dark-4)
);
}
.window {
position: absolute;
top: 2px;
bottom: 2px;
border-radius: 3px;
min-width: 3px;
}
.windowWork {
background-color: var(--mantine-color-blue-5);
}
.windowAgent {
background-color: var(--mantine-color-grape-5);
}
/* A lone single-sample (P_single) window: minimal + dimmed, so it neither
vanishes nor fakes dense work (§6.2). */
.windowSingle {
opacity: 0.5;
}
.daySum {
font-size: var(--mantine-font-size-xs);
text-align: right;
white-space: nowrap;
}
.gapRow {
padding: 6px 0 6px 108px;
font-size: var(--mantine-font-size-xs);
color: var(--mantine-color-dimmed);
font-style: italic;
}
.legendSwatch {
display: inline-block;
width: 12px;
height: 12px;
border-radius: 3px;
vertical-align: middle;
}
@@ -0,0 +1,37 @@
// #395 — client-side mirror of the server work-time payload
// (apps/server/src/core/page/work-time). Shapes returned by POST /pages/history/time.
export type WorkSessionClass = "work" | "agent_only";
export interface IDayWindow {
start: number;
end: number;
class: WorkSessionClass;
}
export interface IPerDay {
day: number;
dayISO: string;
activeMs: number;
agentMs: number;
windows: IDayWindow[];
}
export interface IWorkTimeConfig {
tGap: number;
agentTGap: number;
pIn: number;
pOut: number;
pSingle: number;
excludeGit: boolean;
burstCapMs?: number;
dedupRoundMs: number;
}
export interface IPageWorkTime {
workMs: number;
agentOnlyMs: number;
perDay: IPerDay[];
config: IWorkTimeConfig;
tz: string;
}
@@ -51,6 +51,7 @@ import {
import { formattedDate } from "@/lib/time.ts";
import { PageEditModeToggle } from "@/features/user/components/page-state-pref.tsx";
import MovePageModal from "@/features/page/components/move-page-modal.tsx";
import WorkTimeStat from "@/features/page-history/work-time/work-time-stat.tsx";
import { useTimeAgo } from "@/hooks/use-time-ago.tsx";
import {
useFavoriteIds,
@@ -265,6 +266,8 @@ function PageActionMenu({ readOnly, onSaveVersion }: PageActionMenuProps) {
return (
<>
{page?.id && <WorkTimeStat pageId={page.id} />}
<Menu
shadow="xl"
position="bottom-end"
@@ -1,6 +1,6 @@
import { Spotlight } from "@mantine/spotlight";
import { IconSearch } from "@tabler/icons-react";
import { Group, VisuallyHidden } from "@mantine/core";
import { Group, Text, VisuallyHidden } from "@mantine/core";
import { useState, useMemo } from "react";
import { useDebouncedValue } from "@mantine/hooks";
import { useTranslation } from "react-i18next";
@@ -84,6 +84,11 @@ export function SearchSpotlight({ spaceId }: SearchSpotlightProps) {
onFiltersChange={handleFiltersChange}
spaceId={spaceId}
/>
{/* #529: operator hint — matches ANY word by default; "…" for an exact
phrase, +term to require, -term to exclude. */}
<Text size="xs" c="dimmed" mt={4}>
{t('Tip: "exact phrase", +required, -excluded')}
</Text>
</div>
<VisuallyHidden role="status" aria-live="polite">
@@ -5,6 +5,9 @@ import { IPage } from "@/features/page/types/page.types.ts";
export interface IPageSearch {
id: string;
// #529 A7 superset: `pageId` aliases `id`; `rank`/`highlight` are null for
// substring-only hits (the UI already falls back to the title/snippet).
pageId?: string;
title: string;
icon: string;
parentPageId: string;
@@ -12,9 +15,36 @@ export interface IPageSearch {
creatorId: string;
createdAt: Date;
updatedAt: Date;
rank: string;
highlight: string;
rank: string | number | null;
highlight: string | null;
space: Partial<ISpace>;
// New #529 fields (present from the native Postgres search driver).
snippet?: string;
score?: number;
path?: string[];
matchedFields?: string[];
matchedTerms?: string[];
}
// #529 A5 pagination envelope returned by POST /search (native driver). The web
// list helpers read `items`; these travel alongside for pagination + diagnostics.
export interface IPageSearchResponse {
items: IPageSearch[];
total: number;
hasMore: boolean;
truncatedAtCap: boolean;
offset: number;
query?: {
raw: string;
parsed: {
positive: string[];
required: string[];
excluded: string[];
reason?: string;
};
mode: "or" | "and";
match: string;
};
}
export interface SearchSuggestionParams {
@@ -37,6 +67,10 @@ export interface IPageSearchParams {
query: string;
spaceId?: string;
shareId?: string;
// #529 A9: match mode (auto default) + pagination.
match?: "auto" | "word" | "prefix" | "substring";
limit?: number;
offset?: number;
}
export interface IAttachmentSearch {
@@ -29,6 +29,7 @@ import ShareAliasSection from "@/features/share/components/share-alias-section.t
import { useAtom } from "jotai";
import { workspaceAtom } from "@/features/user/atoms/current-user-atom.ts";
import { useSpaceQuery } from "@/features/space/queries/space-query.ts";
import { usePageHistoryListQuery } from "@/features/page-history/queries/page-history-query.ts";
interface ShareModalProps {
readOnly: boolean;
@@ -54,6 +55,33 @@ export default function ShareModal({ readOnly }: ShareModalProps) {
// if level is greater than zero, then it is a descendant page from a shared page
const isDescendantShared = share && share.level > 0;
// #370 Stage B — "publish only the saved version". Mirrors the server XOR:
// approved and includeSubPages are mutually exclusive.
const isApproved = share?.publishedMode === "approved";
const includeSubPages = share?.includeSubPages ?? false;
// Resolve the latest MANUAL version (the one an approved share publishes) so
// the modal can caption it and warn when the live draft has drifted ahead.
// Only fetched for a directly-shared page whose modal is actually open.
const { data: historyData } = usePageHistoryListQuery(
pageIsShared ? pageId : "",
);
const latestManual = useMemo(() => {
const items = historyData?.pages?.flatMap((p) => p.items) ?? [];
// The list is newest-first; the first manual row is the published version.
return items.find((h) => h.kind === "manual") ?? null;
}, [historyData]);
// The live draft has edits newer than the published saved version when the
// page was updated after that manual snapshot was taken.
const hasUnpublishedEdits = useMemo(() => {
if (!isApproved || !latestManual || !page?.updatedAt) return false;
return (
new Date(page.updatedAt).getTime() >
new Date(latestManual.createdAt).getTime()
);
}, [isApproved, latestManual, page?.updatedAt]);
const publicLink = `${getAppUrl()}/share/${share?.key}/p/${pageSlug}`;
const [isPagePublic, setIsPagePublic] = useState<boolean>(false);
@@ -115,6 +143,23 @@ export default function ShareModal({ readOnly }: ShareModalProps) {
}
};
// #370 Stage B — toggle between publishing the live draft ('live') and the
// last saved version ('approved'). The server rejects approved + sub-pages, so
// the UI keeps the two toggles mutually exclusive (see the disabled props).
const handlePublishedModeChange = async (
event: React.ChangeEvent<HTMLInputElement>,
) => {
const value = event.currentTarget.checked;
try {
await updateShareMutation.mutateAsync({
shareId: share.id,
publishedMode: value ? "approved" : "live",
});
} catch {
// query invalidation will revert the UI
}
};
const shareLink = useMemo(
() => (
<Group my="sm" gap={4} wrap="nowrap">
@@ -238,11 +283,42 @@ export default function ShareModal({ readOnly }: ShareModalProps) {
<Switch
onChange={handleSubPagesChange}
checked={share.includeSubPages}
checked={includeSubPages}
size="xs"
disabled={readOnly}
// XOR with approved mode: a version-frozen share cannot also
// publish a whole sub-tree.
disabled={readOnly || isApproved}
/>
</Group>
<Group justify="space-between" wrap="nowrap" gap="xl" mt="sm">
<div>
<Text size="sm">
{t("Publish only the saved version")}
</Text>
<Text size="xs" c="dimmed">
{isApproved
? t(
"Visitors see the last saved version, not live edits",
)
: t("Visitors see live edits as you make them")}
</Text>
</div>
<Switch
onChange={handlePublishedModeChange}
checked={isApproved}
size="xs"
// XOR with sub-pages: hidden intent is enforced by disabling
// this toggle whenever sub-pages are shared.
disabled={readOnly || includeSubPages}
/>
</Group>
{isApproved && hasUnpublishedEdits && (
<Text size="xs" c="orange" mt={4}>
{t(
"You have unsaved changes newer than the published version",
)}
</Text>
)}
<Group justify="space-between" wrap="nowrap" gap="xl" mt="sm">
<div>
<Text size="sm">{t("Search engine indexing")}</Text>
@@ -6,6 +6,9 @@ export interface IShare {
pageId: string;
includeSubPages: boolean;
searchIndexing: boolean;
// #370 Stage B — 'live' serves the current draft; 'approved' serves the last
// manually-saved version. Mutually exclusive with includeSubPages.
publishedMode: "live" | "approved";
creatorId: string;
spaceId: string;
workspaceId: string;
@@ -75,6 +78,7 @@ export interface ICreateShare {
pageId?: string;
includeSubPages?: boolean;
searchIndexing?: boolean;
publishedMode?: "live" | "approved";
}
export type IUpdateShare = ICreateShare & { shareId: string; pageId?: string };
@@ -0,0 +1,195 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { render, act, cleanup } from "@testing-library/react";
import { MemoryRouter, useNavigate } from "react-router-dom";
// Mocks for the dirty shell's side-effecting collaborators.
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
vi.mock("@/i18n.ts", () => ({ default: { t: (k: string) => k } }));
vi.mock("@/lib/reload-guard", () => ({
hasAutoReloaded: vi.fn(() => false),
markAutoReloaded: vi.fn(() => true),
recordReloadBreadcrumb: vi.fn(),
takeReloadBreadcrumb: vi.fn(() => null),
}));
import { notifications } from "@mantine/notifications";
import { hasAutoReloaded, markAutoReloaded } from "@/lib/reload-guard";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
__resetGuardedReloadForTests,
} from "./guarded-reload";
const show = notifications.show as unknown as ReturnType<typeof vi.fn>;
const mockHasAutoReloaded = hasAutoReloaded as unknown as ReturnType<
typeof vi.fn
>;
const mockMarkAutoReloaded = markAutoReloaded as unknown as ReturnType<
typeof vi.fn
>;
let reload: ReturnType<typeof vi.fn>;
let visibility: DocumentVisibilityState;
// Test harness mounted inside a router: it installs the navigation hook and
// exposes `navigate` so a test can drive an in-app router navigation.
let doNavigate: (to: string) => void;
function Harness() {
useVersionReloadOnNavigation();
const navigate = useNavigate();
doNavigate = navigate;
return null;
}
function mountHarness() {
render(
<MemoryRouter initialEntries={["/start"]}>
<Harness />
</MemoryRouter>,
);
}
function navigateTo(path: string) {
act(() => {
doNavigate(path);
});
}
beforeEach(() => {
__resetGuardedReloadForTests();
vi.clearAllMocks();
mockHasAutoReloaded.mockReturnValue(false);
mockMarkAutoReloaded.mockReturnValue(true);
vi.stubGlobal("APP_VERSION", "test-A");
reload = vi.fn();
Object.defineProperty(window, "location", {
configurable: true,
value: { reload },
});
visibility = "visible";
Object.defineProperty(document, "visibilityState", {
configurable: true,
get: () => visibility,
});
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
});
describe("triggerGuardedReload (variant C)", () => {
it("noop when versions match: no banner, no reload", () => {
triggerGuardedReload("test-A");
expect(show).not.toHaveBeenCalled();
expect(reload).not.toHaveBeenCalled();
});
it("noop when the server version is empty (fail-safe)", () => {
triggerGuardedReload("");
triggerGuardedReload(undefined);
expect(show).not.toHaveBeenCalled();
expect(reload).not.toHaveBeenCalled();
});
it("real mismatch shows the banner but does NOT reload immediately", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(show).toHaveBeenCalledTimes(1);
expect(show.mock.calls[0][0]).toMatchObject({
id: "app-version-reload",
autoClose: false,
withCloseButton: true,
});
expect(reload).not.toHaveBeenCalled();
});
it("reloads EXACTLY ONCE on the first in-app navigation after a mismatch", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
// A second navigation must NOT reload again (one-shot was consumed).
navigateTo("/again");
expect(reload).toHaveBeenCalledTimes(1);
});
it("does NOT reload on a 2nd mismatch after the first was armed/consumed (one-shot)", () => {
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
// Another app-version mismatch arrives (reconnect): must not re-arm.
triggerGuardedReload("test-C");
navigateTo("/again");
expect(reload).toHaveBeenCalledTimes(1);
});
it("does NOT reload merely from the tab going to the background", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
visibility = "hidden";
act(() => {
document.dispatchEvent(new Event("visibilitychange"));
});
expect(reload).not.toHaveBeenCalled();
});
it("a hidden-at-receipt tab is also NOT reloaded immediately (variant C uniform); reloads on next navigation", () => {
visibility = "hidden";
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
expect(show).toHaveBeenCalledTimes(1);
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
});
it("the banner's Update button reloads immediately", () => {
triggerGuardedReload("test-B");
const message = show.mock.calls[0][0].message as {
props: { onClick: () => void };
};
message.props.onClick();
expect(reload).toHaveBeenCalledTimes(1);
});
it("banner-only (auto-reload already spent): banner, never auto-reload on navigation", () => {
mockHasAutoReloaded.mockReturnValue(true);
mountHarness();
triggerGuardedReload("test-B");
expect(show).toHaveBeenCalledTimes(1);
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
});
it("does NOT reload when the flag write fails; falls back to the banner", () => {
mockMarkAutoReloaded.mockReturnValue(false);
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
// performAutoReload falls back to showing the banner (initial + fallback).
expect(show).toHaveBeenCalled();
});
it("is idempotent within a tab-load: repeated emits do not stack banners", () => {
triggerGuardedReload("test-B");
triggerGuardedReload("test-B");
triggerGuardedReload("test-C");
expect(show).toHaveBeenCalledTimes(1);
});
});
@@ -0,0 +1,188 @@
import { useEffect, useRef } from "react";
import { useLocation } from "react-router-dom";
import { Button } from "@mantine/core";
import { notifications } from "@mantine/notifications";
import i18n from "@/i18n.ts";
import {
hasAutoReloaded,
markAutoReloaded,
recordReloadBreadcrumb,
takeReloadBreadcrumb,
} from "@/lib/reload-guard";
import { decideVersionAction } from "@/features/user/version-coherence";
// Dirty shell around the pure `decideVersionAction`: it reads globals
// (APP_VERSION), touches sessionStorage via the shared reload-guard, drives the
// Mantine notification, and arms the router-navigation reload hook. Kept
// separate from the pure module so the decision stays unit-testable without a
// DOM.
// One fixed id so repeated app-version signals (e.g. every reconnect) update a
// single banner instead of stacking a new one each time.
const BANNER_ID = "app-version-reload";
// Module-level idempotency for the current tab-load: once a mismatch has been
// handled we don't re-arm the navigation reload or re-show the banner on
// subsequent app-version emits.
let handled = false;
// Variant C: on a real mismatch we do NOT reload the tab when it merely goes to
// the background (that would silently drop a half-written comment/form). Instead
// we arm a one-shot reload for the NEXT in-app router navigation — a point where
// the user is already leaving the current page, so an in-app navigation would
// discard that unsaved component-state anyway and the reload adds no extra loss.
let pendingNavReload = false;
// Remembered from the last detected mismatch for the pre-reload breadcrumb and
// the (already-visible) banner.
let lastServerVersion = "";
let lastClientVersion = "";
// Read the build version baked into THIS bundle. The `typeof` guard avoids a
// ReferenceError where the `APP_VERSION` global is absent (e.g. under vitest,
// where Vite's `define` did not run) — an unknown client version makes the
// pure decision no-op (fail-safe).
function readClientVersion(): string {
return (typeof APP_VERSION !== "undefined" ? APP_VERSION : "").trim();
}
// Perform the actual reload — but only after the shared one-shot flag is
// persisted. If the write fails (storage unavailable) we must NOT reload
// (mirrors the reactive chunk-load boundary's `catch → return`), and fall back
// to the manual banner so the user can still recover.
function performAutoReload(): void {
if (!markAutoReloaded()) {
showReloadBanner();
return;
}
// Trace right before the reload (which clears the console): a persistent
// breadcrumb + a log line so the auto-reload is observable in a field report.
recordReloadBreadcrumb({
path: "proactive",
serverVersion: lastServerVersion,
clientVersion: lastClientVersion,
});
console.warn(
`[version-coherence] auto-reloading: client=${lastClientVersion} -> server=${lastServerVersion}`,
);
window.location.reload();
}
function showReloadBanner(): void {
notifications.show({
id: BANNER_ID,
title: i18n.t("A new version is available"),
message: (
<Button size="xs" mt="xs" onClick={() => performAutoReload()}>
{i18n.t("Update")}
</Button>
),
autoClose: false,
withCloseButton: true,
});
}
/**
* Handle a server `app-version` announcement: compare it to this bundle's
* version and, on a real mismatch, show the banner and arm a guarded reload for
* the next in-app navigation (variant C).
*
* - real mismatch (window budget available) → banner + arm navigation reload.
* The banner's "Update" button reloads immediately (same shared window guard).
* The tab is NOT reloaded on visibility change.
* - auto-reload already used this window / storage error → banner only (no arm),
* so there is at most one automatic reload per RELOAD_WINDOW_MS window (loop
* safety).
* - in sync / unknown version → noop (fail-safe).
*/
export function triggerGuardedReload(
rawServerVersion: string | undefined | null,
): void {
const serverVersion = (rawServerVersion ?? "").trim();
const clientVersion = readClientVersion();
// A storage read error surfaces as autoReloadUsed=true → fail toward NOT
// reloading (banner only).
const autoReloadUsed = hasAutoReloaded();
const action = decideVersionAction({
serverVersion,
clientVersion,
autoReloadUsed,
});
if (action === "noop") return;
// Idempotent per tab-load: don't re-arm or re-stack the banner across repeated
// emits (reconnects) once we've already acted.
if (handled) return;
handled = true;
lastServerVersion = serverVersion;
lastClientVersion = clientVersion;
if (action === "banner") {
// Entered banner-only (permanent skew, node oscillation, or the window's
// auto-reload budget already spent). Log for diagnosability; show the banner.
console.warn(
`[version-coherence] server=${serverVersion} client=${clientVersion}: ` +
"auto-reload budget already spent this window — showing manual banner",
);
showReloadBanner();
return;
}
// action === "reload" (variant C): show the banner and defer the auto-reload
// to the next in-app navigation instead of reloading now / on visibility.
showReloadBanner();
pendingNavReload = true;
}
/**
* Consume the armed one-shot navigation reload, if any. Called by
* `useVersionReloadOnNavigation` on each in-app router navigation.
*/
export function consumeNavigationReload(): void {
if (!pendingNavReload) return;
pendingNavReload = false;
performAutoReload();
}
/**
* Hook (mounted inside the Router) that fires the armed one-shot reload on the
* NEXT in-app router navigation after a version mismatch. Skips the initial
* render so it only reacts to real navigations, not the first location.
*/
export function useVersionReloadOnNavigation(): void {
const location = useLocation();
const firstRender = useRef(true);
useEffect(() => {
if (firstRender.current) {
firstRender.current = false;
return;
}
consumeNavigationReload();
}, [location.key]);
}
/**
* Surface (log once) the breadcrumb left by an auto-reload in the previous page
* load — the reload cleared the console, so this makes a "tab reloaded itself"
* report diagnosable. Call once on app startup.
*/
export function surfacePreviousReloadBreadcrumb(): void {
const crumb = takeReloadBreadcrumb();
if (!crumb) return;
console.info(
`[version-coherence] previous auto-reload: path=${crumb.path} ` +
`client=${crumb.clientVersion ?? ""} -> server=${crumb.serverVersion ?? ""} ` +
`at=${new Date(crumb.at).toISOString()}`,
);
}
// Test-only: reset module-level latches between cases.
export function __resetGuardedReloadForTests(): void {
handled = false;
pendingNavReload = false;
lastServerVersion = "";
lastClientVersion = "";
}
@@ -0,0 +1,171 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { render, act, cleanup } from "@testing-library/react";
import { MemoryRouter, useNavigate } from "react-router-dom";
// Integration test for the SHARED, window-based auto-reload budget (invariant a):
// the reactive chunk-load boundary and the proactive version-coherence path both
// route through the REAL @/lib/reload-guard, so at most one automatic reload
// happens per RELOAD_WINDOW_MS across BOTH paths combined. Only the two paths'
// side-effecting collaborators are mocked — the reload guard is intentionally
// REAL so this exercises the actual shared sessionStorage budget.
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
vi.mock("@/i18n.ts", () => ({ default: { t: (k: string) => k } }));
import { handleError } from "@/components/chunk-load-error-boundary";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
__resetGuardedReloadForTests,
} from "./guarded-reload";
import { RELOAD_WINDOW_MS } from "@/lib/reload-guard";
const CHUNK_ERROR = { name: "ChunkLoadError", message: "boom" };
const T0 = 1_000_000_000_000;
let reload: ReturnType<typeof vi.fn>;
let nowMock: ReturnType<typeof vi.spyOn>;
// Harness mounted inside a router: installs the navigation hook and exposes
// `navigate` so a test can drive an in-app router navigation (the point where the
// proactive path fires its armed reload).
let doNavigate: (to: string) => void;
function Harness() {
useVersionReloadOnNavigation();
doNavigate = useNavigate();
return null;
}
function mountHarness() {
render(
<MemoryRouter initialEntries={["/start"]}>
<Harness />
</MemoryRouter>,
);
}
function navigateTo(path: string) {
act(() => {
doNavigate(path);
});
}
function setNow(t: number) {
nowMock.mockReturnValue(t);
}
beforeEach(() => {
sessionStorage.clear();
__resetGuardedReloadForTests();
vi.clearAllMocks();
nowMock = vi.spyOn(Date, "now").mockReturnValue(T0);
vi.stubGlobal("APP_VERSION", "test-A");
reload = vi.fn();
Object.defineProperty(window, "location", {
configurable: true,
value: { reload },
});
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
vi.restoreAllMocks();
sessionStorage.clear();
});
describe("shared window-based reload budget (invariant a)", () => {
it("a chunk-load reload spends the budget: a version-coherence mismatch within the window shows the banner but does NOT reload", () => {
// Path 1 (reactive): a stale-chunk 404 auto-reloads once and stamps the window.
handleError(CHUNK_ERROR);
expect(reload).toHaveBeenCalledTimes(1);
reload.mockClear();
// Path 2 (proactive), 1 min later — still inside the window. The shared budget
// is spent, so the version mismatch degrades to the banner and never reloads.
setNow(T0 + 60_000);
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
});
it("a version-coherence reload spends the SAME budget: a chunk-load error within the window does NOT reload", () => {
// Path 2 (proactive) first: real mismatch → arm → fire on navigation.
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
reload.mockClear();
// Path 1 (reactive), 2 min later — inside the window. Budget already spent by
// the proactive path, so the stale-chunk error must NOT trigger a second reload.
setNow(T0 + 2 * 60_000);
handleError(CHUNK_ERROR);
expect(reload).not.toHaveBeenCalled();
});
it("recovers after the window: a reload strictly older than the window is allowed again (second deploy)", () => {
// First auto-reload (reactive) stamps the window.
handleError(CHUNK_ERROR);
expect(reload).toHaveBeenCalledTimes(1);
reload.mockClear();
// A second deploy arrives after the window has fully elapsed → the proactive
// path is allowed to reload again (window, not a permanent one-shot).
__resetGuardedReloadForTests();
setNow(T0 + RELOAD_WINDOW_MS + 1);
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
});
it("reactive path fails closed when storage READS but cannot WRITE (quota / Safari private): no unguarded reload", () => {
// getItem→null makes hasAutoReloaded() report the budget as available, so
// handleError passes the first guard and reaches `if (!markAutoReloaded())
// return;`. setItem throws → the stamp cannot stick, so markAutoReloaded()
// returns false and that guard MUST bail — otherwise the reactive path would
// reload on every stale-chunk error with no persisted budget (an unguarded
// loop). This is the asymmetric gap: the proactive path's equivalent is
// covered by guarded-reload.test.tsx "does NOT reload when the flag write
// fails".
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("quota exceeded");
},
removeItem: () => {},
clear: () => {},
});
try {
handleError(CHUNK_ERROR);
expect(reload).not.toHaveBeenCalled();
} finally {
vi.unstubAllGlobals();
}
});
it("sessionStorage unavailable: neither path performs an unguarded reload", () => {
// The real guard fails toward NOT reloading when storage throws.
vi.stubGlobal("sessionStorage", {
getItem: () => {
throw new Error("storage disabled");
},
setItem: () => {
throw new Error("storage disabled");
},
removeItem: () => {},
clear: () => {},
});
try {
handleError(CHUNK_ERROR);
expect(reload).not.toHaveBeenCalled();
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
} finally {
vi.unstubAllGlobals();
}
});
});
@@ -13,6 +13,12 @@ import { useCollabToken } from "@/features/auth/queries/auth-query.tsx";
import { Error404 } from "@/components/ui/error-404.tsx";
import { queryClient } from "@/main.tsx";
import { makeConnectHandler } from "@/features/user/connect-resync.ts";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
surfacePreviousReloadBreadcrumb,
} from "@/features/user/guarded-reload.tsx";
import type { AppVersionSocketPayload } from "@/features/user/version-coherence.ts";
export function UserProvider({ children }: React.PropsWithChildren) {
const [, setCurrentUser] = useAtom(currentUserAtom);
@@ -22,6 +28,16 @@ export function UserProvider({ children }: React.PropsWithChildren) {
// fetch collab token on load
const { data: collab } = useCollabToken();
// version-coherence: fire the armed one-shot reload on the next in-app
// navigation (variant C — a safe point, not on tab backgrounding).
useVersionReloadOnNavigation();
// Surface any breadcrumb left by an auto-reload in the previous page load
// (the reload cleared the console) so a field report stays diagnosable.
useEffect(() => {
surfacePreviousReloadBreadcrumb();
}, []);
useEffect(() => {
if (isLoading || isError) {
return;
@@ -47,6 +63,16 @@ export function UserProvider({ children }: React.PropsWithChildren) {
handleConnect();
});
// Register the version-coherence listener SYNCHRONOUSLY, before the socket
// connects: the server emits `app-version` immediately in handleConnection,
// so a listener attached after connect would miss it on a fast localhost
// connect. On a version mismatch the client shows a banner and defers the
// auto-reload to the next in-app navigation (variant C — avoids reloading a
// backgrounded tab that may hold unsaved input) before it hits a stale chunk.
newSocket.on("app-version", (payload?: AppVersionSocketPayload) => {
triggerGuardedReload(payload?.version);
});
return () => {
console.log("ws disconnected");
newSocket.disconnect();
@@ -0,0 +1,64 @@
import { describe, it, expect } from "vitest";
import { decideVersionAction } from "./version-coherence";
describe("decideVersionAction", () => {
it("noop when the server version is empty (fail-safe)", () => {
expect(
decideVersionAction({
serverVersion: "",
clientVersion: "v1",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("noop when the client version is empty (fail-safe)", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("noop when versions are equal (in sync)", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "v1",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("reload on a real mismatch the first time this session", () => {
expect(
decideVersionAction({
serverVersion: "test-B",
clientVersion: "test-A",
autoReloadUsed: false,
}),
).toBe("reload");
});
it("banner on a mismatch once the session auto-reload is spent", () => {
expect(
decideVersionAction({
serverVersion: "test-B",
clientVersion: "test-A",
autoReloadUsed: true,
}),
).toBe("banner");
});
it("equal versions stay noop even if auto-reload was already used", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "v1",
autoReloadUsed: true,
}),
).toBe("noop");
});
});
@@ -0,0 +1,32 @@
// Payload of the per-connect `app-version` socket.io event announced by the
// server (ws.gateway.ts) after a successful auth. A dedicated event — NOT a
// member of the room-scoped `WebSocketEvent` union (which is discriminated by
// `operation`), so it never touches use-query-subscription.
export type AppVersionSocketPayload = { version: string };
/**
* Pure decision for the version-coherence guard.
*
* All inputs are injected (no globals, no side effects) so it is unit-testable
* without a DOM or the build-time `APP_VERSION` global (undefined under vitest).
*
* - `autoReloadUsed` = an automatic reload has already happened within the
* current ~5-min window, so we must not auto-reload again (loop safety,
* shared window budget with the reactive chunk-load boundary).
*
* Returns:
* - "noop" — do nothing (unknown version on either side, or already in sync).
* - "banner" — show the manual "update available" banner only (no auto-reload).
* - "reload" — real first-time mismatch: eligible for a guarded auto-reload.
*/
export function decideVersionAction(args: {
serverVersion: string;
clientVersion: string;
autoReloadUsed: boolean;
}): "reload" | "banner" | "noop" {
const { serverVersion, clientVersion, autoReloadUsed } = args;
if (!serverVersion || !clientVersion) return "noop"; // fail-safe: unknown version → never act
if (serverVersion === clientVersion) return "noop"; // in sync
if (autoReloadUsed) return "banner"; // one auto-reload per RELOAD_WINDOW_MS window already spent
return "reload"; // real mismatch, window budget available
}
+145
View File
@@ -0,0 +1,145 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import {
hasAutoReloaded,
markAutoReloaded,
shouldAutoReload,
recordReloadBreadcrumb,
takeReloadBreadcrumb,
RELOAD_WINDOW_MS,
} from "./reload-guard";
// The shared budget is a single sessionStorage timestamp keyed here; both the
// reactive chunk-load boundary and the proactive version-coherence path read and
// stamp it, so at most one auto-reload happens per RELOAD_WINDOW_MS across BOTH.
const RELOAD_AT_KEY = "chunk-reload-at";
const NOW = 1_000_000_000_000;
describe("reload-guard", () => {
beforeEach(() => {
sessionStorage.clear();
vi.restoreAllMocks();
});
afterEach(() => {
sessionStorage.clear();
vi.restoreAllMocks();
});
it("hasAutoReloaded is false before any reload, true within the window after mark", () => {
expect(hasAutoReloaded(NOW)).toBe(false);
expect(markAutoReloaded(NOW)).toBe(true);
// Same key both paths share; stores the reload timestamp, not a flag.
expect(sessionStorage.getItem(RELOAD_AT_KEY)).toBe(String(NOW));
// Inside the window → budget spent → true (fall through to manual UI).
expect(hasAutoReloaded(NOW)).toBe(true);
expect(hasAutoReloaded(NOW + RELOAD_WINDOW_MS)).toBe(true);
});
it("hasAutoReloaded is false again once the window has elapsed (a later deploy recovers)", () => {
markAutoReloaded(NOW);
// Strictly older than the window → a new deploy's mismatch may reload again.
expect(hasAutoReloaded(NOW + RELOAD_WINDOW_MS + 1)).toBe(false);
});
it("hasAutoReloaded returns true when reading storage throws (fail toward not reloading)", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => {
throw new Error("storage disabled");
},
setItem: () => {
throw new Error("storage disabled");
},
});
try {
expect(hasAutoReloaded()).toBe(true);
} finally {
vi.unstubAllGlobals();
}
});
it("hasAutoReloaded treats an unparseable stored timestamp as never-reloaded", () => {
sessionStorage.setItem(RELOAD_AT_KEY, "not-a-number");
expect(hasAutoReloaded(NOW)).toBe(false);
});
it("markAutoReloaded returns false when writing storage throws", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("storage disabled");
},
});
try {
expect(markAutoReloaded()).toBe(false);
} finally {
vi.unstubAllGlobals();
}
});
it("records and then takes a breadcrumb once (cleared on read)", () => {
recordReloadBreadcrumb({
path: "proactive",
serverVersion: "test-B",
clientVersion: "test-A",
});
const crumb = takeReloadBreadcrumb();
expect(crumb).toMatchObject({
path: "proactive",
serverVersion: "test-B",
clientVersion: "test-A",
});
expect(typeof crumb?.at).toBe("number");
// Cleared on read → a second take returns null.
expect(takeReloadBreadcrumb()).toBeNull();
});
it("takeReloadBreadcrumb returns null when nothing was recorded", () => {
expect(takeReloadBreadcrumb()).toBeNull();
});
it("recordReloadBreadcrumb swallows a storage-write error (diagnostics only)", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("storage disabled");
},
removeItem: () => {},
});
try {
expect(() =>
recordReloadBreadcrumb({ path: "chunk-boundary" }),
).not.toThrow();
} finally {
vi.unstubAllGlobals();
}
});
});
// The pure window gate replaces the old one-shot flag: it must permit recovery
// across several deploys in one tab (each > window apart) while still stopping an
// infinite reload loop when a lazy chunk is permanently broken (a second failure
// < window). Moved here from the chunk-load boundary now that it is the shared
// guard both paths route through.
describe("shouldAutoReload", () => {
const WINDOW = RELOAD_WINDOW_MS;
it("allows a reload when we have never auto-reloaded", () => {
expect(shouldAutoReload(NOW, null, WINDOW)).toBe(true);
});
it("allows a reload when the last one was 6 minutes ago (outside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 6 * 60 * 1000, WINDOW)).toBe(true);
});
it("blocks a reload when the last one was 1 minute ago (inside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 1 * 60 * 1000, WINDOW)).toBe(false);
});
it("blocks a reload exactly at the window boundary (not strictly older)", () => {
expect(shouldAutoReload(NOW, NOW - WINDOW, WINDOW)).toBe(false);
});
it("allows a reload when the stored timestamp is unparseable (NaN)", () => {
expect(shouldAutoReload(NOW, NaN, WINDOW)).toBe(true);
});
});
+121
View File
@@ -0,0 +1,121 @@
// Shared, window-based auto-reload budget.
//
// Both auto-reload paths — the reactive chunk-load-error-boundary (recovers
// AFTER a stale lazy chunk 404s) and the proactive version-coherence feature
// (reloads BEFORE the tab hits a stale chunk) — go through these functions so
// they share ONE window-scoped reload budget: at most a single automatic
// reload per RELOAD_WINDOW_MS across BOTH paths. A window (rather than a
// permanent one-shot flag) lets a SECOND deploy in the same tab's lifetime
// recover too, while a permanent skew, node oscillation, or a genuinely-missing
// chunk still degrades to a manual banner/UI after the first reload instead of
// looping. When sessionStorage is unavailable every mismatch degrades to the
// manual UI — no unguarded reload.
// sessionStorage key holding the epoch-ms timestamp of the last automatic reload
// (shared by both paths).
const RELOAD_AT_KEY = "chunk-reload-at";
// Allow at most one automatic reload per this window. A stale-deploy 404 is cured
// by a single reload, so anything inside the window is treated as a reload loop
// (permanently-broken chunk / permanent skew) and falls through to the manual UI.
export const RELOAD_WINDOW_MS = 5 * 60 * 1000;
/**
* Pure window decision, unit-tested in isolation: auto-reload only if we have
* never auto-reloaded (lastReloadAt null/NaN) or the last one was strictly older
* than the window. Anything inside the window is suppressed to break an infinite
* reload loop.
*/
export function shouldAutoReload(
now: number,
lastReloadAt: number | null,
windowMs: number,
): boolean {
if (lastReloadAt === null || Number.isNaN(lastReloadAt)) return true;
return now - lastReloadAt > windowMs;
}
/**
* Has an automatic reload already happened within the current window (so the
* shared budget is spent right now)? Both paths check this before reloading; a
* `true` return means fall through to the manual banner/UI instead of reloading.
*
* A storage read error (private mode / disabled) is reported as `true` so the
* caller fails toward NOT reloading — an unguarded loop is worse than a stale
* tab the user can reload manually. Note a window (not a permanent flag): once
* the window elapses a later deploy's mismatch is allowed to reload again.
*/
export function hasAutoReloaded(now: number = Date.now()): boolean {
try {
const raw = sessionStorage.getItem(RELOAD_AT_KEY);
const lastReloadAt = raw === null ? null : Number.parseInt(raw, 10);
return !shouldAutoReload(now, lastReloadAt, RELOAD_WINDOW_MS);
} catch {
return true;
}
}
/**
* Stamp the shared window as consumed now — record that an automatic reload is
* being performed within the current RELOAD_WINDOW_MS window.
*
* Returns whether the write succeeded. A `false` return (storage unavailable)
* means the caller MUST NOT reload — otherwise the stamp would never stick and
* the reload could loop.
*/
export function markAutoReloaded(now: number = Date.now()): boolean {
try {
sessionStorage.setItem(RELOAD_AT_KEY, String(now));
return true;
} catch {
return false;
}
}
// Diagnostic breadcrumb for an automatic reload. Written right before
// window.location.reload() (which clears the console) and read back on the next
// page load, so a "the tab reloaded itself / it's looping" field report is
// diagnosable: which path fired (proactive version-coherence vs the reactive
// chunk-load boundary) and which version pair triggered it. sessionStorage
// survives a same-tab reload, unlike the console.
const RELOAD_BREADCRUMB_KEY = "reload-breadcrumb";
export type ReloadBreadcrumb = {
path: "proactive" | "chunk-boundary";
serverVersion?: string;
clientVersion?: string;
at: number;
};
/**
* Persist a best-effort breadcrumb just before an automatic reload. Failures
* (storage unavailable) are swallowed — this is diagnostics only and must never
* block or alter the reload decision.
*/
export function recordReloadBreadcrumb(
entry: Omit<ReloadBreadcrumb, "at">,
): void {
try {
sessionStorage.setItem(
RELOAD_BREADCRUMB_KEY,
JSON.stringify({ ...entry, at: Date.now() }),
);
} catch {
// best-effort diagnostics only
}
}
/**
* Read and clear the breadcrumb left by an auto-reload in the previous page
* load. Cleared on read so it surfaces exactly once per reload.
*/
export function takeReloadBreadcrumb(): ReloadBreadcrumb | null {
try {
const raw = sessionStorage.getItem(RELOAD_BREADCRUMB_KEY);
if (!raw) return null;
sessionStorage.removeItem(RELOAD_BREADCRUMB_KEY);
return JSON.parse(raw) as ReloadBreadcrumb;
} catch {
return null;
}
}
+29 -2
View File
@@ -1,7 +1,8 @@
import { defineConfig, loadEnv } from "vite";
import { defineConfig, loadEnv, type Plugin } from "vite";
import react from "@vitejs/plugin-react";
import { compression } from "vite-plugin-compression2";
import * as path from "path";
import * as fs from "node:fs";
import { execSync } from "node:child_process";
const envPath = path.resolve(process.cwd(), "..", "..");
@@ -24,7 +25,32 @@ function resolveAppVersion(cwd: string): string {
}
}
// Emit <outDir>/version.json = { "version": appVersion } so the server can read
// the exact same build id the bundle was compiled with. The value is the SAME
// `appVersion` fed into `define.APP_VERSION`, so version.json and the baked-in
// global are identical by construction — the single source of truth (no
// runtime-env second copy that could drift and cause a false version mismatch).
function versionJsonPlugin(version: string): Plugin {
let outDir = "dist";
return {
name: "emit-version-json",
apply: "build",
configResolved(config) {
outDir = config.build.outDir;
},
writeBundle() {
const root = path.resolve(process.cwd(), outDir);
fs.mkdirSync(root, { recursive: true });
fs.writeFileSync(
path.join(root, "version.json"),
JSON.stringify({ version }),
);
},
};
}
export default defineConfig(({ mode }) => {
const appVersion = resolveAppVersion(envPath);
const {
APP_URL,
FILE_UPLOAD_SIZE_LIMIT,
@@ -52,10 +78,11 @@ export default defineConfig(({ mode }) => {
POSTHOG_HOST,
POSTHOG_KEY,
},
APP_VERSION: JSON.stringify(resolveAppVersion(envPath)),
APP_VERSION: JSON.stringify(appVersion),
},
plugins: [
react(),
versionJsonPlugin(appVersion),
// Emit .br and .gz next to every built asset so the server can serve the
// precompressed copy (see @fastify/static preCompressed in static.module.ts).
compression({
+1 -1
View File
@@ -23,7 +23,7 @@
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build && pnpm --filter @docmost/mcp build",
"test": "jest",
"test:int": "jest --config test/jest-integration.json",
"test:watch": "jest --watch",
@@ -141,7 +141,57 @@ export function htmlToJson(html: string) {
}
}
export function jsonToText(tiptapJson: JSONContent) {
/**
* Deterministic text-serializer overrides for the `format:"text"` page read
* (#502). Non-text nodes render to a STABLE placeholder instead of their
* (structure-dependent) inner text, so a machine diff of two text reads is
* driven only by the page's actual prose — output stability across package
* versions IS the contract (pinned by a snapshot test). Returning a string from
* a `textSerializer` also stops `generateText` descending into the node, so a
* table renders as ONE token rather than its flattened cell text.
*
* Only nodes with no meaningful flat-text form are overridden; every other node
* (paragraph/heading/list/code/blockquote/callout/…) keeps its natural text so
* a config written as markdown reads back byte-identical.
*/
const TEXT_READ_SERIALIZERS: Record<string, (props: { node: any }) => string> =
{
// Image atom: no inner text -> a fixed placeholder.
image: () => '[image]',
// Table: `[table RxC]` where R = row count, C = the first row's cell count
// (a table's columns are uniform per the schema). Computed from the PM node,
// so it is independent of cell contents.
table: ({ node }) => {
const rows = node?.childCount ?? 0;
const cols = rows > 0 ? (node.child(0)?.childCount ?? 0) : 0;
return `[table ${rows}x${cols}]`;
},
};
/**
* Serialize a ProseMirror/TipTap document to plain text.
*
* Default (no options): the long-standing search-index behavior — bare
* concatenated node text with `generateText`'s default `\n\n` block separator.
* This feeds the page `textContent` tsvector and MUST NOT change.
*
* `deterministic:true` (#502 `format:"text"` page read): a flat, machine-diffable
* rendering — one line per block (`\n` block separator; `hardBreak` already
* serializes to `\n`), inline marks/anchors/autoformat dropped, and non-text
* nodes replaced by the stable placeholders above (`[image]`, `[table RxC]`).
*/
export function jsonToText(
// `any` (like jsonToHtml/jsonToMarkdown) so a loosely-typed DB `page.content`
// (JsonValue) can be passed straight through, as the controller does.
tiptapJson: any,
options?: { deterministic?: boolean },
) {
if (options?.deterministic) {
return generateText(tiptapJson, tiptapExtensions, {
blockSeparator: '\n',
textSerializers: TEXT_READ_SERIALIZERS,
});
}
return generateText(tiptapJson, tiptapExtensions);
}
@@ -0,0 +1,116 @@
import { jsonToText } from './collaboration.util';
// #502 READ contract: `jsonToText(json, { deterministic: true })` — the flat,
// machine-diffable text rendering behind getPage `format:"text"`. Block-per-line
// (`\n` separator), inline marks/anchors dropped, hardBreak -> `\n`, and non-text
// nodes replaced by STABLE placeholders (`[image]`, `[table RxC]`). Output
// stability across package versions IS the contract, so it is pinned by a
// snapshot below. The DEFAULT (no options) path is the search-index serializer
// and MUST be unchanged — asserted separately.
const doc = (...content: any[]) => ({ type: 'doc', content });
const para = (...content: any[]) => ({ type: 'paragraph', content });
const text = (t: string, marks?: any[]) =>
marks ? { type: 'text', text: t, marks } : { type: 'text', text: t };
describe('jsonToText — default (search index) behavior is unchanged', () => {
it('uses the `\\n\\n` block separator and drops non-text nodes to empty', () => {
const d = doc(para(text('alpha')), para(text('beta')));
expect(jsonToText(d)).toBe('alpha\n\nbeta');
});
it('an image contributes no text in the default (tsvector) mode', () => {
const d = doc(para(text('cap')), { type: 'image', attrs: { src: 's' } });
// No `[image]` placeholder leaks into the search index.
expect(jsonToText(d)).not.toContain('[image]');
});
});
describe('jsonToText — deterministic:true (getPage format:"text")', () => {
it('renders one line per block with `\\n` separators, marks dropped', () => {
const d = doc(
{ type: 'heading', attrs: { level: 2 }, content: [text('Title')] },
para(
text('hello ', [{ type: 'bold' }]),
text('world', [{ type: 'italic' }]),
),
);
expect(jsonToText(d, { deterministic: true })).toBe('Title\nhello world');
});
it('a hardBreak renders as a newline', () => {
const d = doc(para(text('a'), { type: 'hardBreak' }, text('b')));
expect(jsonToText(d, { deterministic: true })).toBe('a\nb');
});
it('an image node -> the stable `[image]` placeholder', () => {
const d = doc(para(text('before')), { type: 'image', attrs: { src: 's' } });
expect(jsonToText(d, { deterministic: true })).toBe('before\n[image]');
});
it('a table -> `[table RxC]` (rows x columns) and its cell text is NOT flattened in', () => {
const cell = (t: string) => ({
type: 'tableCell',
content: [para(text(t))],
});
const row = (...cells: any[]) => ({ type: 'tableRow', content: cells });
const table = {
type: 'table',
content: [
row(cell('CELLONE'), cell('CELLTWO'), cell('CELLTHREE')),
row(cell('CELLFOUR'), cell('CELLFIVE'), cell('CELLSIX')),
],
};
const d = doc(para(text('grid:')), table);
const out = jsonToText(d, { deterministic: true });
expect(out).toBe('grid:\n[table 2x3]');
expect(out).not.toContain('CELL'); // cell text is not flattened into the read
});
it('SNAPSHOT: a mixed document renders to a stable, deterministic string', () => {
const cell = (t: string) => ({
type: 'tableCell',
content: [para(text(t))],
});
const d = doc(
{ type: 'heading', attrs: { level: 1 }, content: [text('Config')] },
para(text('key = ', [{ type: 'bold' }]), text('value')),
{
type: 'bulletList',
content: [
{ type: 'listItem', content: [para(text('one'))] },
{ type: 'listItem', content: [para(text('two'))] },
],
},
{ type: 'image', attrs: { src: 's' } },
{
type: 'table',
content: [{ type: 'tableRow', content: [cell('x'), cell('y')] }],
},
);
expect(jsonToText(d, { deterministic: true })).toMatchInlineSnapshot(`
"Config
key = value
one
two
[image]
[table 1x2]"
`);
});
it('SCENARIO: a config written as a code block reads back byte-identical (diff empty)', () => {
const config =
'export TICKET_LIFETIME=$2592000\nservers:\n - www.internal.host';
const d = doc({
type: 'codeBlock',
attrs: { language: 'yaml' },
content: [text(config)],
});
// The code block is one block; its text (dollars, bare domain, newlines) is
// preserved verbatim, so a read-as-text of a stored config diffs empty.
expect(jsonToText(d, { deterministic: true })).toBe(config);
});
});
@@ -0,0 +1,52 @@
import { join } from 'path';
import * as fs from 'node:fs';
import * as os from 'node:os';
import { readClientBuildVersion } from './client-version';
describe('readClientBuildVersion', () => {
let dir: string;
beforeEach(() => {
dir = fs.mkdtempSync(join(os.tmpdir(), 'client-version-'));
});
afterEach(() => {
fs.rmSync(dir, { recursive: true, force: true });
});
const writeVersionJson = (content: string) =>
fs.writeFileSync(join(dir, 'version.json'), content);
it('returns the version from a valid version.json', () => {
writeVersionJson(JSON.stringify({ version: 'test-A' }));
expect(readClientBuildVersion(dir)).toBe('test-A');
});
it('trims surrounding whitespace in the version', () => {
writeVersionJson(JSON.stringify({ version: ' v1.2.3 ' }));
expect(readClientBuildVersion(dir)).toBe('v1.2.3');
});
it('returns "" when version.json is missing', () => {
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" on malformed JSON', () => {
writeVersionJson('{ not json');
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the version field is absent', () => {
writeVersionJson(JSON.stringify({ notVersion: 'x' }));
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the version field is not a string', () => {
writeVersionJson(JSON.stringify({ version: 123 }));
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the path does not exist at all', () => {
expect(readClientBuildVersion(join(dir, 'nope'))).toBe('');
});
});
@@ -0,0 +1,36 @@
import { join } from 'path';
import * as fs from 'node:fs';
/**
* Resolve the absolute path to the built client bundle directory
* (`apps/client/dist`) shipped into the runtime image.
*
* The `../` depth is anchored on THIS module's compiled location
* (`dist/common/helpers`). `integrations/static` sits at the same depth under
* the compiled root, so both callers (StaticModule and readClientBuildVersion)
* MUST share this single helper rather than duplicating the depth — a copy in a
* module at a different depth would silently resolve to the wrong directory.
*/
export function resolveClientDistPath(): string {
return join(__dirname, '..', '..', '..', '..', 'client/dist');
}
/**
* Read the build version the client bundle was compiled with, from
* `<clientDistPath>/version.json` (written by the Vite build — the single
* source of truth shared by the baked-in `APP_VERSION` global and this file).
*
* Fail-safe: any error (missing file, unreadable, bad JSON, non-string
* version) yields `''`. The caller treats an empty version as "unknown" and
* the whole version-coherence feature stays silently inert — existing deploys
* without the file keep working unchanged.
*/
export function readClientBuildVersion(clientDistPath: string): string {
try {
const raw = fs.readFileSync(join(clientDistPath, 'version.json'), 'utf8');
const version = (JSON.parse(raw) as { version?: unknown }).version;
return typeof version === 'string' ? version.trim() : '';
} catch {
return '';
}
}
+1
View File
@@ -3,3 +3,4 @@ export * from './nanoid.utils';
export * from './file.helper';
export * from './constants';
export * from './security-headers';
export * from './client-version';
@@ -35,6 +35,7 @@ import {
import { PaginationOptions } from '@docmost/db/pagination/pagination-options';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { UserThrottlerGuard } from '../../integrations/throttle/user-throttler.guard';
import { AI_CHAT_THROTTLER } from '../../integrations/throttle/throttler-names';
@@ -43,6 +44,8 @@ import {
AiChatRunHooks,
AiChatService,
AiChatStreamBody,
rowHasInlineParts,
hydrateAssistantParts,
} from './ai-chat.service';
import { AiChatRunService } from './ai-chat-run.service';
import { AiTranscriptionService } from './ai-transcription.service';
@@ -129,8 +132,39 @@ export class AiChatController {
// production. Only touched on the resumable-stream (flag-on) path.
private readonly streamRegistry?: AiChatStreamRegistryService,
private readonly environment?: EnvironmentService,
// #492: reconstruct a #492 mid-run record's parts from the steps table before
// returning rows to the client / export. OPTIONAL so positional controller
// specs compile unchanged; when absent, hydration is skipped (old-era rows
// already carry inline parts, so nothing to reconstruct).
private readonly aiChatRunStepRepo?: AiChatRunStepRepo,
) {}
/**
* Reconstruct parts for any assistant rows that don't carry them INLINE a
* #492 mid-run record whose per-step parts live in `ai_chat_run_steps` (the
* append-persist backend). Every FINISHED row (old-era + #492) and every old-era
* streaming snapshot already has inline `metadata.parts`, so the common path
* fetches NOTHING and returns the rows untouched; only an actively-streaming
* new-style row triggers the batch step fetch. Consumers (seed/poll/export) read
* `metadata.parts` off the returned rows exactly as before the era switch is
* invisible to them (reconstructRunParts contract).
*/
private async withReconstructedParts(
rows: AiChatMessage[],
workspaceId: string,
): Promise<AiChatMessage[]> {
if (!this.aiChatRunStepRepo) return rows;
const needy = rows.filter(
(r) => r.role === 'assistant' && !rowHasInlineParts(r),
);
if (needy.length === 0) return rows;
const stepsByMessage = await this.aiChatRunStepRepo.findByMessageIds(
needy.map((r) => r.id),
workspaceId,
);
return hydrateAssistantParts(rows, stepsByMessage);
}
/** List the requesting user's chats in this workspace (paginated). */
@HttpCode(HttpStatus.OK)
@Post('chats')
@@ -184,11 +218,17 @@ export class AiChatController {
@AuthWorkspace() workspace: Workspace,
) {
await this.assertOwnedChat(dto.chatId, user, workspace);
return this.aiChatMessageRepo.findByChat(
const page = await this.aiChatMessageRepo.findByChat(
dto.chatId,
workspace.id,
pagination,
);
// #492: reconstruct parts for any active new-style row so the client seed sees
// `metadata.parts` unchanged (a no-op for the finished rows that fill a page).
return {
...page,
items: await this.withReconstructedParts(page.items, workspace.id),
};
}
/**
@@ -225,7 +265,10 @@ export class AiChatController {
workspace.id,
);
return {
rows,
// #492: the delta of an actively-streaming new-style row carries its parts
// reconstructed from the steps table, so the degraded poll shows persisted
// progress exactly as the pre-#492 full-row snapshot did.
rows: await this.withReconstructedParts(rows, workspace.id),
cursor,
run: run ? { id: run.id, status: run.status } : null,
};
@@ -247,8 +290,10 @@ export class AiChatController {
@AuthWorkspace() workspace: Workspace,
): Promise<{ markdown: string }> {
const chat = await this.assertOwnedChat(dto.chatId, user, workspace);
const rows = await this.aiChatMessageRepo.findAllByChat(
dto.chatId,
const rows = await this.withReconstructedParts(
await this.aiChatMessageRepo.findAllByChat(dto.chatId, workspace.id),
// #492: an interrupted-but-still-active turn exports its persisted steps
// (reconstructed from the steps table) just like the pre-#492 full row did.
workspace.id,
);
const markdown = buildChatMarkdown({
@@ -288,7 +333,13 @@ export class AiChatController {
workspace.id,
)
: undefined;
return { run, message: message ?? null };
// #492: reconnect to an IN-FLIGHT run reconstructs the projection row's parts
// from the steps table (the row itself carries only the step marker mid-run);
// a finished run's row already has inline parts, so this is a no-op.
const [hydrated] = message
? await this.withReconstructedParts([message], workspace.id)
: [undefined];
return { run, message: hydrated ?? null };
}
/**
@@ -370,10 +370,12 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
);
});
// #490 reactive branch: a provider CONTEXT-OVERFLOW 400 in onError is classified,
// records a distinguishable cause, and stamps metadata.replayOverflow so the NEXT
// turn's budgeter trims aggressively (the recovery that un-bricks the chat).
it('#490: a context-overflow 400 stamps replayOverflow on the finalized row', async () => {
// #490/#520 reactive branch: a provider CONTEXT-OVERFLOW 400 in onError is
// classified, records a distinguishable cause, and stamps the consecutive-overflow
// COUNTER (metadata.replayOverflowCount) so the NEXT turn's budgeter trims with
// escalating aggression (the recovery that un-bricks the chat). This is a fresh
// chat (empty history -> prior streak 0), so the first overflow stamps count 1.
it('#490/#520: a context-overflow 400 stamps replayOverflowCount=1 on the finalized row', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
@@ -397,11 +399,14 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
metadata: Record<string, unknown>;
};
expect(patch.status).toBe('error');
expect(patch.metadata.replayOverflow).toBe(true);
// First overflow on a fresh chat -> k = prior(0) + 1 = 1.
expect(patch.metadata.replayOverflowCount).toBe(1);
// The legacy boolean is no longer written (the counter supersedes it).
expect('replayOverflow' in patch.metadata).toBe(false);
expect(patch.metadata.error).toContain('контекстное окно');
});
it('#490: a non-overflow error does NOT stamp replayOverflow', async () => {
it('#490/#520: a non-overflow error does NOT stamp the overflow counter', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
@@ -412,6 +417,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
status: string;
metadata: Record<string, unknown>;
};
expect('replayOverflowCount' in patch.metadata).toBe(false);
expect('replayOverflow' in patch.metadata).toBe(false);
});
});
@@ -30,13 +30,16 @@ import {
STEP_LIMIT_NO_ANSWER_MARKER,
OUTPUT_DEGENERATION_ERROR,
lastAssistantContextTokens,
lastAssistantReplayOverflow,
lastAssistantReplayOverflowCount,
seedActivatedTools,
} from './ai-chat.service';
import type { AiChatMessage, Workspace } from '@docmost/db/types/entity.types';
import { buildSystemPrompt } from './ai-chat.prompt';
import type { McpClientsService } from './external-mcp/mcp-clients.service';
import { resolveEffectiveReplayThreshold } from './history-budget';
import {
resolveEffectiveReplayThreshold,
REPLAY_MIN_FLOOR_TOKENS,
} from './history-budget';
/**
* Unit tests for compactToolOutput: the pure helper that shrinks tool outputs
@@ -554,49 +557,129 @@ describe('seedActivatedTools', () => {
});
});
describe('lastAssistantReplayOverflow', () => {
describe('lastAssistantReplayOverflowCount', () => {
const row = (
role: string,
metadata: Record<string, unknown> | null,
): AiChatMessage => ({ role, metadata }) as unknown as AiChatMessage;
it('is true only when the LAST assistant turn overflowed', () => {
it('reads the consecutive-overflow count from the LAST assistant turn', () => {
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflowCount: 3 }),
row('user', null),
]),
).toBe(true);
// A recovered (later, non-overflow) assistant turn clears it.
).toBe(3);
// A recovered (later, non-overflow) assistant turn resets it to 0 — the read
// stops at the most recent assistant row, which carries no count.
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflowCount: 3 }),
row('user', null),
row('assistant', { contextTokens: 5 }),
]),
).toBe(false);
expect(lastAssistantReplayOverflow([])).toBe(false);
).toBe(0);
expect(lastAssistantReplayOverflowCount([])).toBe(0);
});
// #490 reactive recovery: a prior turn stamped `replayOverflow` must make the
// NEXT turn's effective budget the AGGRESSIVE 0.5x cut — that harder trim is
// what un-bricks a chat that just 400'd on the context window. This exercises
// the exact wiring the service uses: read the stamp, then scale the threshold.
it('#490: a prior replayOverflow drives the next turn to the 0.5x aggressive budget', () => {
// BACK-COMPAT (#520): an in-flight row written by the pre-#520 boolean stamp
// (`replayOverflow: true`, no count) reads as k=1 — the old single 0.5× behavior —
// so a chat mid-recovery across the deploy does not regress.
it('#520 back-compat: a legacy boolean replayOverflow reads as k=1', () => {
expect(
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflow: true }),
row('user', null),
]),
).toBe(1);
// A legacy row with the flag absent/false is k=0.
expect(
lastAssistantReplayOverflowCount([row('assistant', { contextTokens: 5 })]),
).toBe(0);
});
// A corrupt/negative persisted count never yields a negative k.
it('clamps a corrupt negative count to 0', () => {
expect(
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflowCount: -4 }),
]),
).toBe(0);
});
// #490/#520 reactive recovery: the prior consecutive-overflow count `k` drives
// the next turn's effective budget to an ESCALATING cut (0.5**k) — each further
// consecutive 400 tightens it, which is what un-bricks a chat that keeps
// overflowing. This exercises the exact wiring the service uses: read the count,
// then scale the threshold.
it('#490/#520: the prior count drives the next turn to the escalating aggressive budget', () => {
const history = [
row('assistant', { replayOverflow: true }),
row('assistant', { replayOverflowCount: 1 }),
row('user', null),
];
const priorOverflowed = lastAssistantReplayOverflow(history);
expect(priorOverflowed).toBe(true);
// Base budget 100k -> aggressive recovery halves it to 50k this turn.
expect(resolveEffectiveReplayThreshold(100_000, priorOverflowed)).toBe(50_000);
const k = lastAssistantReplayOverflowCount(history);
expect(k).toBe(1);
// Base budget 100k -> first-overflow recovery halves it to 50k this turn.
expect(resolveEffectiveReplayThreshold(100_000, k)).toBe(50_000);
// A second consecutive overflow (k=2) quarters it.
expect(resolveEffectiveReplayThreshold(100_000, 2)).toBe(25_000);
// Odd base floors, not rounds.
expect(resolveEffectiveReplayThreshold(99_999, true)).toBe(49_999);
// No prior overflow -> the base budget is used verbatim (no aggressive cut).
expect(resolveEffectiveReplayThreshold(100_000, false)).toBe(100_000);
expect(resolveEffectiveReplayThreshold(99_999, 1)).toBe(49_999);
// No prior overflow (k=0) -> the base budget is used verbatim (no cut).
expect(resolveEffectiveReplayThreshold(100_000, 0)).toBe(100_000);
// An explicit off-switch (null) is never overridden, even on recovery.
expect(resolveEffectiveReplayThreshold(null, true)).toBeNull();
expect(resolveEffectiveReplayThreshold(null, 3)).toBeNull();
});
// #520 escalation table + convergence: the cut deepens each consecutive overflow
// and is CLAMPED at the floor so it converges (un-bricks even against a small
// real window), instead of the old fixed single 0.5× that stuck at 50k forever.
it('#520: escalates and converges to the floor, un-bricking a small real window', () => {
const base = 100_000;
expect(resolveEffectiveReplayThreshold(base, 0)).toBe(base);
expect(resolveEffectiveReplayThreshold(base, 1)).toBe(50_000);
expect(resolveEffectiveReplayThreshold(base, 2)).toBe(25_000);
expect(resolveEffectiveReplayThreshold(base, 3)).toBe(12_500);
// Residual-brick regression (#520): with the flat-default base (100k) and a real
// model window of ~40k, the OLD fixed 0.5× stuck at 50k forever (> 40k -> 400s
// again, never recovers). The escalating cut drops BELOW 40k after enough
// consecutive overflows -> the history finally fits -> the chat un-bricks.
const realWindow = 40_000;
// k=1 (50k) still exceeds the window — the old behavior's terminal state.
expect(resolveEffectiveReplayThreshold(base, 1)).toBeGreaterThan(realWindow);
// But escalation converges under the window within a couple more turns.
const converged = [2, 3, 4, 5].some(
(k) => (resolveEffectiveReplayThreshold(base, k) as number) < realWindow,
);
expect(converged).toBe(true);
// Convergence is bounded BELOW by the floor: a large k never trims below it.
for (const k of [4, 8, 20, 100]) {
expect(resolveEffectiveReplayThreshold(base, k)).toBe(REPLAY_MIN_FLOOR_TOKENS);
expect(
resolveEffectiveReplayThreshold(base, k) as number,
).toBeGreaterThanOrEqual(REPLAY_MIN_FLOOR_TOKENS);
}
});
// The floor never RAISES a legitimately small configured budget above itself —
// that would re-overflow the very window it was configured for. Under #520 Option B
// recovery MAY cut it BELOW itself (down to floor(0.5×base) = the old 0.5× cut).
it('#520: never inflates a small configured budget above itself (may cut below)', () => {
const small = 5_000; // below the floor
const floor = Math.min(REPLAY_MIN_FLOOR_TOKENS, Math.floor(small * 0.5)); // 2500
expect(resolveEffectiveReplayThreshold(small, 0)).toBe(small);
// Even under escalation the effective threshold never exceeds the base, and never
// drops below the absolute floor.
for (const k of [1, 2, 3, 10]) {
const t = resolveEffectiveReplayThreshold(small, k) as number;
expect(t).toBeLessThanOrEqual(small);
expect(t).toBeGreaterThanOrEqual(floor);
}
// Option B: on overflow it DOES cut below the configured budget (old floor==budget
// behavior would keep this at `small`).
expect(resolveEffectiveReplayThreshold(small, 1)).toBe(floor);
});
});
@@ -930,21 +1013,50 @@ describe('flushAssistant', () => {
expect(flushed.metadata.error).toBe('boom');
});
// #490 observability: the replay budgeter's decision is stamped on the turn.
it('records replayTrimmedToTokens + replayOverflow when provided', () => {
// #490/#520 observability: the replay budgeter's decision is stamped on the turn,
// now including the consecutive-overflow COUNTER (#520) the next turn escalates on.
it('records replayTrimmedToTokens + replayOverflowCount when provided', () => {
const f = flushAssistant([], '', 'error', {
error: 'ctx',
replayTrimmedToTokens: 42_000,
replayOverflow: true,
replayOverflowCount: 2,
});
expect(f.metadata.replayTrimmedToTokens).toBe(42_000);
expect(f.metadata.replayOverflow).toBe(true);
expect(f.metadata.replayOverflowCount).toBe(2);
});
it('omits the replay metadata when not provided', () => {
const f = flushAssistant([], '', 'completed', { finishReason: 'stop' });
expect('replayTrimmedToTokens' in f.metadata).toBe(false);
expect('replayOverflow' in f.metadata).toBe(false);
expect('replayOverflowCount' in f.metadata).toBe(false);
expect('replayBelowConfiguredBudget' in f.metadata).toBe(false);
});
// #520 Option B observability: the turn is marked when recovery replayed BELOW the
// admin-configured budget, so the UI/telemetry can surface "config too large".
it('stamps replayBelowConfiguredBudget when recovery cut below the configured budget', () => {
const f = flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: 2,
replayBelowConfiguredBudget: true,
});
// MUTATION SENTINEL: dropping the metadata stamp in flushAssistant reddens this.
expect(f.metadata.replayBelowConfiguredBudget).toBe(true);
});
it('omits replayBelowConfiguredBudget on a normal in-budget replay', () => {
// false/omitted must NOT leave the flag on the turn (only the below-budget case).
const off = flushAssistant([], '', 'completed', {
replayBelowConfiguredBudget: false,
});
expect('replayBelowConfiguredBudget' in off.metadata).toBe(false);
});
// A clean finalize (no overflow -> count 0/omitted) leaves NO counter, which the
// next turn reads as k=0 — the reset that ends a recovery streak.
it('omits replayOverflowCount for a zero/absent count (reset semantics)', () => {
const zero = flushAssistant([], '', 'completed', { replayOverflowCount: 0 });
expect('replayOverflowCount' in zero.metadata).toBe(false);
});
// #274 observability: the page-change diff the agent saw this turn is persisted
+299 -42
View File
@@ -22,6 +22,7 @@ import { AiSettingsService } from '../../integrations/ai/ai-settings.service';
import { describeProviderError } from '../../integrations/ai/ai-error.util';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo';
import { AiAgentRoleRepo } from '@docmost/db/repos/ai-agent-roles/ai-agent-roles.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
@@ -140,9 +141,10 @@ const OUTPUT_DEGENERATION_ERROR =
// Prefix recorded on the assistant row when the provider rejected the turn for
// CONTEXT OVERFLOW (#490): the replayed history exceeded the model's window. The
// row is ALSO stamped `metadata.replayOverflow` so the NEXT turn's budgeter trims
// aggressively (the reactive recovery — the overflowing turn had no usage signal
// to trigger preventive trimming, so the classified 400 is what un-bricks it).
// row is ALSO stamped `metadata.replayOverflowCount` (the consecutive-overflow
// counter, #520) so the NEXT turn's budgeter trims with escalating aggression (the
// reactive recovery — the overflowing turn had no usage signal to trigger
// preventive trimming, so the classified 400 is what un-bricks it).
export const CONTEXT_OVERFLOW_ERROR_PREFIX =
'Диалог превысил контекстное окно модели; история будет агрессивно ' +
'сокращена на следующем ходу.';
@@ -518,6 +520,12 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// constructions compile unchanged; Nest always injects the real singleton, so
// reconcile sees the SAME in-memory active/zombie maps the runner mutates.
private readonly aiChatRunService?: AiChatRunService,
// #492 append-persist: per-step INSERT into the lightweight steps table (the
// O(Σ steps) replacement for the O(n²) full-row `metadata.parts` rewrite).
// OPTIONAL so existing positional constructions (int-specs) compile unchanged;
// Nest injects the real singleton. When ABSENT the per-step path falls back to
// the pre-#492 full-row flush (no regression, only no WAL win).
private readonly aiChatRunStepRepo?: AiChatRunStepRepo,
) {}
// #487: periodic reconcile timer (single-process phase 1). Started in
@@ -1114,8 +1122,34 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
chatId,
workspace.id,
);
// #492: HYDRATE needy assistant rows from the steps table BEFORE the replay
// map. A #492 mid-run assistant row carries only a step marker
// (metadata.parts:[]); its real per-step parts live in `ai_chat_run_steps`.
// The graceful terminal callbacks (onFinish/onError/onAbort -> flushAssistant)
// assemble the full inline parts, so a normally-ended turn already has them.
// But a HARD crash mid-run (SIGKILL/OOM) fires NO terminal callback, so the
// row stays parts:[]; without this, rowToUiMessage falls back to an empty
// text part and the partial tool-calls/results/text — durable in the steps
// table — would DROP OUT of the model's replay context (regressing #183
// step-granular durability for the model consumer). Mirrors the controller's
// withReconstructedParts EXACTLY (same needy predicate + hydration helper).
// Guarded on the optional repo: absent (positional test builds) degrades to
// the current behavior rather than crashing.
let replayHistory = oldHistory;
if (this.aiChatRunStepRepo) {
const needy = oldHistory.filter(
(r) => r.role === 'assistant' && !rowHasInlineParts(r),
);
if (needy.length > 0) {
const stepsByMessage = await this.aiChatRunStepRepo.findByMessageIds(
needy.map((r) => r.id),
workspace.id,
);
replayHistory = hydrateAssistantParts(oldHistory, stepsByMessage);
}
}
const uiMessages: Array<Omit<UIMessage, 'id'> & { id: string }> = [
...oldHistory.map(rowToUiMessage),
...replayHistory.map(rowToUiMessage),
{
id: 'pending-user',
role: 'user',
@@ -1154,7 +1188,9 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// hint — confirm it against the persisted history (the preceding assistant
// turn must really be aborted/streaming) so a spoofed flag cannot inject the
// interrupt note onto an ordinary turn. The partial output the model needs is
// already in `messages` (the aborted assistant row replays via findRecent).
// already in `messages`: a #492 mid-run row's per-step parts live only in the
// `ai_chat_run_steps` table and were hydrated into the replay history above,
// so the aborted assistant turn replays WITH its partial parts intact.
// Append the new user turn (shape-only) so index -2 is the prior assistant.
const interrupted = isInterruptResume(
[...oldHistory, { role: 'user', status: null, metadata: null }],
@@ -1193,27 +1229,52 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
}
// Last turn's provider-reported context size (authoritative when present).
const priorContextTokens = lastAssistantContextTokens(oldHistory);
// Reactive recovery (#490): if the LAST turn was rejected for context
// overflow (stamped by onError), trim AGGRESSIVELY this turn — the
// overflowing turn produced no usage signal, so a normal-threshold trim may
// not shrink enough to fit. This is what un-bricks a chat that just 400'd.
const priorOverflowed = lastAssistantReplayOverflow(oldHistory);
// Reactive recovery (#490/#520): `k` = how many CONSECUTIVE preceding turns
// were rejected for context overflow (stamped by onError). Each consecutive
// overflow trims MORE aggressively (resolveEffectiveReplayThreshold scales the
// budget by 0.5**k, clamped at the floor) so recovery ESCALATES until the
// history fits — the overflowing turn produced no usage signal, so a single
// fixed cut may not shrink enough when the real model window is small. This is
// what un-bricks a chat that keeps 400'ing on the context window.
const priorOverflowCount = lastAssistantReplayOverflowCount(oldHistory);
const effectiveThreshold = resolveEffectiveReplayThreshold(
replayBudget.thresholdTokens,
priorOverflowed,
priorOverflowCount,
);
if (priorOverflowed) {
this.logger.warn(
`AI chat (chat ${chatId}): previous turn hit context overflow; ` +
`applying aggressive replay budget (${effectiveThreshold} tokens).`,
);
// #520 Option B: recovery MAY cut the replay budget BELOW the admin-configured
// window when the provider keeps proving non-fit (the anti-brick invariant of
// the reactive branch beats a config that a 400 has disproved). Surface it so
// the admin sees their window is factually too large, and mark the turn so the
// UI/telemetry can show it. Only true when actually below (not on in-budget
// escalation edge cases); k>0 implies below for any budget >= 2 tokens.
const replayBelowConfiguredBudget =
replayBudget.thresholdTokens != null &&
effectiveThreshold != null &&
effectiveThreshold < replayBudget.thresholdTokens;
if (priorOverflowCount > 0) {
if (replayBelowConfiguredBudget) {
this.logger.warn(
`AI chat (chat ${chatId}): configured replay budget ` +
`(${replayBudget.thresholdTokens} tokens) does not fit the model — ` +
`replaying BELOW it at ${effectiveThreshold} tokens after ` +
`${priorOverflowCount} consecutive context overflow(s) ` +
`(escalation level ${priorOverflowCount}). Lower chatContextWindow ` +
`to match the model's real context window.`,
);
} else {
this.logger.warn(
`AI chat (chat ${chatId}): ${priorOverflowCount} consecutive context ` +
`overflow(s); applying escalated aggressive replay budget ` +
`(${effectiveThreshold} tokens).`,
);
}
}
const preTrim = trimHistoryForReplay(
messages,
effectiveThreshold,
// A prior OVERFLOW means the provider count is stale/absent — force the
// char-estimate path by ignoring priorContextTokens on recovery.
priorOverflowed ? undefined : priorContextTokens,
priorOverflowCount > 0 ? undefined : priorContextTokens,
);
messages = preTrim.messages;
// Observability (#490): record the budgeter's decision on the turn so the UI
@@ -1559,17 +1620,57 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// connection when finalize runs, so the SQL `WHERE status='streaming'`
// (not this flag) is what prevents it clobbering the terminal row.
if (finalized) return null;
// Build the flush ONCE so the returned count is EXACTLY the persisted
// `stepsPersisted` (both derive from capturedSteps.length at this instant).
const flushed = flushAssistant(capturedSteps, '', 'streaming', {
pageChanged,
partsCache,
});
const stepsPersisted = flushed.metadata.stepsPersisted as number;
// The count derives from capturedSteps.length at THIS instant, so the
// returned value is EXACTLY the persisted `stepsPersisted` the ring rotates
// on (whether we take the append-persist path or the legacy fallback).
const stepsPersisted = capturedSteps.length;
try {
await this.aiChatMessageRepo.update(assistantId, workspace.id, flushed, {
onlyIfStreaming: true,
});
if (this.aiChatRunStepRepo) {
// #492 APPEND-PERSIST: write only THIS finished step's parts to the
// steps table (O(step) WAL), then bump the row's CHEAP step marker —
// NO growing `metadata.parts` blob (that O(n²) full-row rewrite is
// exactly what this removes). The full `metadata.parts` is assembled
// once at finalize; a mid-run resume seed is reconstructed from the
// step rows (reconstructRunParts). The INSERT is idempotent
// (ON CONFLICT DO NOTHING), so a re-fired step never doubles the parts.
const index = stepsPersisted - 1;
if (index >= 0) {
const stepParts = assistantParts(
[capturedSteps[index]],
'',
partsCache,
);
await this.aiChatRunStepRepo.insertStep(
assistantId,
workspace.id,
index,
stepParts,
);
}
// Marker UPDATE: advance stepsPersisted + keep the toolTrace era marker
// (bumps updatedAt so the delta poll observes the step, and carries the
// frontier a resuming client attaches from). Scoped onlyIfStreaming so a
// late marker never clobbers the terminal finalize.
await this.aiChatMessageRepo.update(
assistantId,
workspace.id,
{ metadata: stepMarkerMetadata(stepsPersisted) },
{ onlyIfStreaming: true },
);
} else {
// Legacy fallback (no steps table wired — positional test builds): the
// pre-#492 full-row flush, so parts still land inline on the row.
const flushed = flushAssistant(capturedSteps, '', 'streaming', {
pageChanged,
partsCache,
});
await this.aiChatMessageRepo.update(
assistantId,
workspace.id,
flushed,
{ onlyIfStreaming: true },
);
}
return stepsPersisted;
} catch (err) {
this.logger.warn(
@@ -1828,6 +1929,10 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
pageChanged,
partsCache,
replayTrimmedToTokens,
// #520 Option B: mark the turn when recovery replayed BELOW the
// configured budget (only when it actually did).
replayBelowConfiguredBudget:
replayBelowConfiguredBudget || undefined,
}),
);
// #184/#487: the RUN is finalized ALWAYS (never gated on the message).
@@ -1906,7 +2011,16 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
pageChanged,
partsCache,
replayTrimmedToTokens,
replayOverflow: overflow || undefined,
// #520: escalate the consecutive-overflow counter so the NEXT turn
// trims MORE aggressively (0.5**k). k grows by 1 each consecutive
// overflow; a clean finalize omits the field, resetting it to 0.
replayOverflowCount: overflow
? priorOverflowCount + 1
: undefined,
// #520 Option B: mark the turn when THIS replay was already below the
// configured budget (only when it actually was).
replayBelowConfiguredBudget:
replayBelowConfiguredBudget || undefined,
}),
);
// #184: settle the RUN as failed, carrying the provider/transport cause.
@@ -2338,22 +2452,39 @@ export function seedActivatedTools(
}
/**
* Whether the most recent assistant turn was rejected for CONTEXT OVERFLOW
* (#490): its row carries `metadata.replayOverflow` (stamped by the stream's
* onError). The next turn's budgeter reads this to trim aggressively the
* reactive recovery. Only the LAST assistant turn matters (an older overflow was
* already recovered), so we stop at the first assistant row scanning backwards.
* How many CONSECUTIVE recent turns were rejected for CONTEXT OVERFLOW (#490/#520):
* `k`, read from the most recent assistant row's `metadata.replayOverflowCount`
* (stamped by the stream's onError, incremented each consecutive overflow and reset
* to 0 on any clean finalize). The next turn's budgeter feeds this to
* {@link resolveEffectiveReplayThreshold} to trim with ESCALATING aggression the
* reactive recovery. Only the LAST assistant turn matters (its count already carries
* the consecutive streak; an older overflow followed by a clean turn was recovered),
* so we stop at the first assistant row scanning backwards.
*
* BACK-COMPAT: a row written by the pre-#520 boolean stamp (`replayOverflow: true`,
* no count) is read as k=1 the old single 0.5× behavior so in-flight chats do
* not regress across the deploy.
*/
export function lastAssistantReplayOverflow(
export function lastAssistantReplayOverflowCount(
history: ReadonlyArray<AiChatMessage>,
): boolean {
): number {
for (let i = history.length - 1; i >= 0; i--) {
const row = history[i];
if (row.role !== 'assistant') continue;
const meta = (row.metadata ?? {}) as { replayOverflow?: unknown };
return meta.replayOverflow === true;
const meta = (row.metadata ?? {}) as {
replayOverflowCount?: unknown;
replayOverflow?: unknown;
};
if (typeof meta.replayOverflowCount === 'number') {
// Guard against a corrupt/negative persisted value.
return meta.replayOverflowCount > 0
? Math.floor(meta.replayOverflowCount)
: 0;
}
// Back-compat: legacy boolean stamp -> one overflow (0.5× cut).
return meta.replayOverflow === true ? 1 : 0;
}
return false;
return 0;
}
/** The last message with role 'user' from a useChat payload, if any. */
@@ -2749,6 +2880,122 @@ export function rowToUiMessage(row: AiChatMessage): Omit<UIMessage, 'id'> & {
return { id: row.id, role, parts: parts as UIMessage['parts'] };
}
/**
* Cheap step-marker metadata for the #492 per-step UPDATE. Advances
* `stepsPersisted` (the resume attach frontier) and keeps the `toolTraceVersion`
* era marker, WITHOUT the growing `parts` blob (those live in the steps table
* now; the full `metadata.parts` is assembled once at finalize by flushAssistant).
* `parts: []` is kept for shape stability it reads as an empty inline-parts row,
* which is exactly the discriminator that routes reconstruction to the steps table.
*/
export function stepMarkerMetadata(
stepsPersisted: number,
): Record<string, unknown> {
return { parts: [], toolTraceVersion: 2, stepsPersisted };
}
/**
* Whether an assistant row already carries its full UI parts INLINE on the row
* (`metadata.parts`). TRUE for every FINISHED row old-era rows AND #492 rows,
* whose full parts are assembled once at finalize and for old-era streaming
* snapshots (the pre-#492 per-step full-row flush). FALSE for a #492 MID-RUN
* record, whose per-step parts live in the `ai_chat_run_steps` table. This is the
* era discriminator the reconstruct seam branches on no schema flag needed.
*/
export function rowHasInlineParts(row: { metadata?: unknown }): boolean {
const meta = (row.metadata ?? {}) as { parts?: unknown };
return Array.isArray(meta.parts) && meta.parts.length > 0;
}
/**
* Concatenate persisted per-step parts (in `stepIndex` order) into the turn's UI
* parts (#492). Reproduces EXACTLY what flushAssistant assistantParts would have
* written to `metadata.parts` for those finished steps, since each step row stored
* `assistantParts([step])` at persist time.
*/
export function assembleStepParts(
stepRows: ReadonlyArray<{ stepIndex: number; parts: unknown }>,
): UIMessage['parts'] {
const parts: Array<Record<string, unknown>> = [];
for (const step of [...stepRows].sort((a, b) => a.stepIndex - b.stepIndex)) {
if (Array.isArray(step.parts)) {
parts.push(...(step.parts as Array<Record<string, unknown>>));
}
}
return parts as UIMessage['parts'];
}
/**
* reconstructRunParts (#492) the single backend-switch seam. Given an assistant
* ROW and its persisted step rows, return the turn's UI `parts` + the persisted
* step count, reading from the ROW when it already carries inline parts (old-era
* records AND every finished record) and from the STEPS TABLE otherwise (a #492
* mid-run record). The higher-level consumers (attach seed, delta poll, export)
* route their rowparts through this / {@link hydrateAssistantParts}, so old and
* new records reconstruct identically WITHOUT the consumers branching on the era.
*/
export function reconstructRunParts(
row: { metadata?: unknown; content?: string | null },
stepRows: ReadonlyArray<{ stepIndex: number; parts: unknown }>,
): { parts: UIMessage['parts']; stepsPersisted: number } {
if (rowHasInlineParts(row)) {
const meta = row.metadata as {
parts: UIMessage['parts'];
stepsPersisted?: number;
};
return {
parts: meta.parts,
stepsPersisted:
typeof meta.stepsPersisted === 'number'
? meta.stepsPersisted
: stepRows.length,
};
}
if (stepRows.length > 0) {
return {
parts: assembleStepParts(stepRows),
stepsPersisted: stepRows.length,
};
}
// No inline parts and no step rows: an old-era seed / empty streaming row. Fall
// back to a single text part from `content` (mirrors rowToUiMessage).
return {
parts: textPart(row.content ?? '') as UIMessage['parts'],
stepsPersisted: 0,
};
}
/**
* Fill each assistant row's `metadata.parts` from its step rows when the row does
* not already carry them inline (a #492 mid-run record), so a consumer that reads
* `metadata.parts` off the RAW row (the client seed/poll, the Markdown export)
* sees the reconstructed parts with NO change to itself. Rows that already have
* inline parts (old-era + finished) and non-assistant rows pass through untouched.
* Pure: returns new row objects, never mutates the inputs.
*/
export function hydrateAssistantParts<
T extends { id: string; role?: string; metadata?: unknown },
>(
rows: ReadonlyArray<T>,
stepsByMessage: Map<
string,
ReadonlyArray<{ stepIndex: number; parts: unknown }>
>,
): T[] {
return rows.map((row) => {
if (row.role !== 'assistant' || rowHasInlineParts(row)) return row;
const steps = stepsByMessage.get(row.id);
if (!steps || steps.length === 0) return row;
return {
...row,
metadata: {
...((row.metadata ?? {}) as Record<string, unknown>),
parts: assembleStepParts(steps),
},
};
});
}
/**
* The persisted-row patch shape produced by {@link flushAssistant}. It is the
* SAME shape the assistant repo insert/update consume (content + toolCalls +
@@ -2893,9 +3140,16 @@ export function flushAssistant(
// the (estimated) token size it trimmed to — the UI can show "replay truncated
// at N tokens". Omitted when nothing was trimmed.
replayTrimmedToTokens?: number;
// #490 reactive branch: set when the provider rejected this turn for context
// overflow. Stamped into metadata so the NEXT turn's budgeter trims aggressively.
replayOverflow?: boolean;
// #490/#520 reactive branch: the consecutive context-overflow count for THIS
// turn (prior streak + 1) when the provider rejected it for context overflow.
// Stamped into metadata so the NEXT turn's budgeter trims with escalating
// aggression (0.5**k). Omitted (undefined) on a clean turn, which resets k to 0.
replayOverflowCount?: number;
// #520 Option B observability: true when recovery replayed this turn's history
// BELOW the admin-configured budget (the configured window did not fit and the
// anti-brick invariant cut past it). Omitted on a normal in-budget replay so the
// flag marks ONLY the "config is factually too large" case.
replayBelowConfiguredBudget?: boolean;
},
): AssistantFlush {
const finished = capturedSteps ?? [];
@@ -2948,7 +3202,10 @@ export function flushAssistant(
metadata.maxContextTokens = extra.maxContextTokens;
if (extra?.replayTrimmedToTokens)
metadata.replayTrimmedToTokens = extra.replayTrimmedToTokens;
if (extra?.replayOverflow) metadata.replayOverflow = true;
if (extra?.replayOverflowCount && extra.replayOverflowCount > 0)
metadata.replayOverflowCount = extra.replayOverflowCount;
if (extra?.replayBelowConfiguredBudget)
metadata.replayBelowConfiguredBudget = true;
if (extra?.error) metadata.error = extra.error;
// Persist the page-change diff the agent saw this turn (#274 observability),
// so history / the Markdown export can show what the user changed. Only when
@@ -1,6 +1,11 @@
import { randomBytes } from 'crypto';
import { Client } from 'pg';
import { flushAssistant, serializeSteps } from './ai-chat.service';
import {
flushAssistant,
serializeSteps,
lastAssistantReplayOverflowCount,
} from './ai-chat.service';
import type { AiChatMessage } from '@docmost/db/types/entity.types';
/**
* #490 write-volume regression an OBSERVABLE-PROPERTY test on a LIVE Postgres,
@@ -207,3 +212,111 @@ describe('#490 write-volume on a live Postgres (pg_current_wal_lsn delta)', () =
expect(v2).toBeLessThan(v1 * 0.75);
}, 120_000);
});
/**
* #520 reactive-recovery COUNTER lifecycle on a LIVE Postgres proves the
* consecutive-overflow count survives a real jsonb metadata round-trip (the persist
* path), not just an in-memory object. flushAssistant BUILDS the row metadata, we
* WRITE it to a jsonb column, READ it back, then reconstruct the assistant row and
* run lastAssistantReplayOverflowCount over it exactly the read the next turn does.
*
* The lifecycle proven end-to-end through pg:
* - consecutive overflows INCREMENT k (1 -> 2 -> 3);
* - a CLEAN finalize omits the field, which the reader treats as a RESET to 0;
* - a legacy boolean row (`replayOverflow: true`) reads back as k=1 (back-compat).
*/
describe('#520 overflow-counter lifecycle on a live Postgres (jsonb round-trip)', () => {
let client: Client | undefined;
let available = false;
beforeAll(async () => {
try {
client = new Client(CONN);
await client.connect();
await client.query('SELECT 1');
available = true;
} catch {
available = false;
client = undefined;
}
});
afterAll(async () => {
await client?.end().catch(() => undefined);
});
// Round-trip an arbitrary metadata object through a real jsonb column and read it
// back as the reconstructed assistant row the next turn would load.
async function roundTrip(
c: Client,
metadata: unknown,
): Promise<AiChatMessage> {
await c.query('UPDATE _wal_counter SET metadata=$1 WHERE id=1', [
JSON.stringify(metadata),
]);
const back = (await c.query('SELECT metadata FROM _wal_counter WHERE id=1'))
.rows[0].metadata as Record<string, unknown>;
return { role: 'assistant', metadata: back } as unknown as AiChatMessage;
}
it('increments across consecutive overflows, resets on a clean turn, and honors the legacy boolean', async () => {
if (!available || !client) {
console.warn('SKIP: gitmost-test-pg not reachable; skipping counter test.');
return;
}
const c = client;
await c.query('DROP TABLE IF EXISTS _wal_counter');
await c.query('CREATE TABLE _wal_counter(id int primary key, metadata jsonb)');
await c.query("INSERT INTO _wal_counter VALUES (1, '{}'::jsonb)");
// Turn 1 overflow: prior streak 0 -> stamp k=1 (as the service does: prior+1).
let prior = lastAssistantReplayOverflowCount([]); // fresh chat
expect(prior).toBe(0);
let row = await roundTrip(
c,
flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: prior + 1,
}).metadata,
);
prior = lastAssistantReplayOverflowCount([row]);
expect(prior).toBe(1);
// Turn 2 overflow: prior 1 -> stamp k=2.
row = await roundTrip(
c,
flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: prior + 1,
}).metadata,
);
prior = lastAssistantReplayOverflowCount([row]);
expect(prior).toBe(2);
// Turn 3 overflow: prior 2 -> stamp k=3.
row = await roundTrip(
c,
flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: prior + 1,
}).metadata,
);
prior = lastAssistantReplayOverflowCount([row]);
expect(prior).toBe(3);
// Turn 4 CLEAN finalize: no overflow -> the field is omitted -> reset to 0.
row = await roundTrip(
c,
flushAssistant([], 'all good', 'completed', { finishReason: 'stop' })
.metadata,
);
expect('replayOverflowCount' in (row.metadata as object)).toBe(false);
expect(lastAssistantReplayOverflowCount([row])).toBe(0);
// Back-compat: a row persisted by the pre-#520 boolean stamp reads back as k=1.
row = await roundTrip(c, { replayOverflow: true });
expect(lastAssistantReplayOverflowCount([row])).toBe(1);
await c.query('DROP TABLE IF EXISTS _wal_counter');
}, 60_000);
});
@@ -1,14 +1,146 @@
import type { ModelMessage } from 'ai';
import {
resolveReplayBudget,
resolveEffectiveReplayThreshold,
isContextOverflowError,
estimateMessagesTokens,
trimHistoryForReplay,
REPLAY_BUDGET_DEFAULT_TOKENS,
REPLAY_BUDGET_WINDOW_FRACTION,
REPLAY_MIN_FLOOR_TOKENS,
REPLAY_TRUNCATION_MARKER,
REPLAY_TURN_COLLAPSED_MARKER,
} from './history-budget';
describe('resolveEffectiveReplayThreshold (#520 iterative escalation)', () => {
// The escalation table: each consecutive overflow (k) deepens the cut by 0.5×.
it('scales the base by 0.5**k, flooring (not rounding) fractional tokens', () => {
const base = 100_000;
expect(resolveEffectiveReplayThreshold(base, 0)).toBe(base); // k=0: unchanged
expect(resolveEffectiveReplayThreshold(base, 1)).toBe(50_000); // 0.5×
expect(resolveEffectiveReplayThreshold(base, 2)).toBe(25_000); // 0.25×
expect(resolveEffectiveReplayThreshold(base, 3)).toBe(12_500); // 0.125×
// Floors, not rounds.
expect(resolveEffectiveReplayThreshold(99_999, 1)).toBe(49_999);
});
it('passes a null base (trimming OFF) through unchanged for any k', () => {
for (const k of [0, 1, 2, 5, 100]) {
expect(resolveEffectiveReplayThreshold(null, k)).toBeNull();
}
});
// The crux of #520: convergence. A large k is clamped at REPLAY_MIN_FLOOR_TOKENS,
// so the escalation CONVERGES to a small-but-usable budget instead of trimming to
// zero — and, unlike the old fixed 0.5× that stuck at 50k, it drops far enough to
// fit a small real model window.
it('clamps a large k at the floor (converges, never below)', () => {
const base = 100_000;
for (const k of [4, 6, 10, 50, 200]) {
const t = resolveEffectiveReplayThreshold(base, k) as number;
expect(t).toBe(REPLAY_MIN_FLOOR_TOKENS);
expect(t).toBeGreaterThanOrEqual(REPLAY_MIN_FLOOR_TOKENS);
}
});
// Residual-brick regression (#520): flat-default base 100k, real window ~40k. The
// OLD fixed single 0.5× stuck at 50k > 40k forever (re-overflows every turn — the
// brick). The iterative cut drops BELOW 40k after a couple more consecutive
// overflows, so the history finally fits and the chat un-bricks.
it('un-bricks: escalation drops below a small real window the fixed 0.5× never could', () => {
const base = 100_000;
const realWindow = 40_000;
// The old terminal state: 0.5× = 50k, still above the window.
expect(resolveEffectiveReplayThreshold(base, 1)).toBeGreaterThan(realWindow);
// Escalation converges under the window.
const converged = [2, 3, 4, 5].some(
(k) => (resolveEffectiveReplayThreshold(base, k) as number) < realWindow,
);
expect(converged).toBe(true);
// MUTATION SENTINEL: reverting `** k` to `** 1` (fixed 0.5×) makes every k yield
// 50k, so `converged` above would be FALSE and this test reddens. Removing the
// floor reddens the clamp test instead.
});
// "Don't INFLATE above itself" is still an invariant: the floor never RAISES a
// legitimately small configured budget above itself (that would re-overflow the
// very window it was set for). Under #520 Option B the floor is min(FLOOR,
// floor(0.5×base)), so for a base BELOW the floor recovery MAY cut it further —
// down to floor(0.5×base), i.e. AT LEAST the old single 0.5× cut — but never above.
it('never inflates a small configured budget above itself (may cut below it, #520 Option B)', () => {
const small = 5_000; // below REPLAY_MIN_FLOOR_TOKENS
const floor = Math.min(REPLAY_MIN_FLOOR_TOKENS, Math.floor(small * 0.5)); // 2500
expect(resolveEffectiveReplayThreshold(small, 0)).toBe(small); // k=0 unchanged
for (const k of [1, 2, 3, 10]) {
const t = resolveEffectiveReplayThreshold(small, k) as number;
// Invariant: never RAISED above the configured budget.
expect(t).toBeLessThanOrEqual(small);
// Option B: never trimmed below the absolute floor (converges).
expect(t).toBeGreaterThanOrEqual(floor);
}
// The old single 0.5× cut is reached (and pinned) — recovery is never WORSE than
// before, and DOES cut below the configured budget on overflow (Option B). Under
// the old floor==budget behavior this would be `small` (5000), not `floor`.
expect(resolveEffectiveReplayThreshold(small, 1)).toBe(floor);
});
// #520 Option B: with a configured window (NOT the flat default), repeated overflow
// escalates the replay budget BELOW the configured budget, down to the absolute
// floor — the chat must not brick even when the configured window is too large for
// the model. A CLEAN turn (k back to 0) restores the full configured budget.
it('escalates below the configured budget down to the floor, and resets on a clean turn', () => {
// Configured context window 8000 -> budget = floor(0.7 × 8000) = 5600 (the code's
// window->budget ratio; compute it, do not hardcode).
const window = 8_000;
const budget = resolveReplayBudget(window).thresholdTokens as number;
expect(budget).toBe(Math.floor(REPLAY_BUDGET_WINDOW_FRACTION * window)); // 5600
// The absolute floor for this budget = min(FLOOR, floor(0.5 × budget)) = 2800
// (a "small window": 0.5×budget < REPLAY_MIN_FLOOR_TOKENS), which is BELOW the
// 5600 configured budget.
const floor = Math.min(
REPLAY_MIN_FLOOR_TOKENS,
Math.floor(budget * 0.5),
); // 2800
expect(floor).toBe(2_800);
expect(floor).toBeLessThan(budget); // below the configured budget
// k=0 -> full configured budget (no overflow yet).
expect(resolveEffectiveReplayThreshold(budget, 0)).toBe(budget);
// k=1 -> cut BELOW the configured budget to the 0.5× floor (2800). This is the
// MUTATION SENTINEL: with the OLD floor==configured-budget behavior this would be
// max(2800, 5600) = 5600 (never below budget), so the assertion reddens.
expect(resolveEffectiveReplayThreshold(budget, 1)).toBe(floor);
expect(resolveEffectiveReplayThreshold(budget, 1)).toBeLessThan(budget);
// k>=2 stays at the floor (converges, never below it).
for (const k of [2, 3, 5, 20]) {
expect(resolveEffectiveReplayThreshold(budget, k)).toBe(floor);
}
// A CLEAN turn resets k to 0 -> the full configured budget is restored (recovery
// is not sticky; it only bites while the provider keeps proving non-fit).
expect(resolveEffectiveReplayThreshold(budget, 0)).toBe(budget);
});
// #520 Option B, LARGE window: a big configured budget escalates in DISTINCT steps
// below itself, converging to the fixed absolute floor REPLAY_MIN_FLOOR_TOKENS.
it('a large configured budget escalates below itself down to REPLAY_MIN_FLOOR_TOKENS', () => {
const budget = resolveReplayBudget(200_000).thresholdTokens as number; // 140000
// The floor for a large window is the fixed REPLAY_MIN_FLOOR_TOKENS (8k), well
// BELOW the configured budget.
expect(
Math.min(REPLAY_MIN_FLOOR_TOKENS, Math.floor(budget * 0.5)),
).toBe(REPLAY_MIN_FLOOR_TOKENS);
const seq = [0, 1, 2, 3, 4, 5, 6].map(
(k) => resolveEffectiveReplayThreshold(budget, k) as number,
);
expect(seq).toEqual([140_000, 70_000, 35_000, 17_500, 8_750, 8_000, 8_000]);
// Every escalated step (k>=1) is below the configured budget; convergence floor.
for (const t of seq.slice(1)) expect(t).toBeLessThan(budget);
expect(seq[seq.length - 1]).toBe(REPLAY_MIN_FLOOR_TOKENS);
});
});
describe('resolveReplayBudget', () => {
it('uses floor(0.7 x window) for a configured window (no cap)', () => {
// 0.7 x 60k = 42k
+62 -11
View File
@@ -22,10 +22,35 @@ export const REPLAY_BUDGET_DEFAULT_TOKENS = 100_000;
/** Fraction of a configured context window used as the budget. */
export const REPLAY_BUDGET_WINDOW_FRACTION = 0.7;
/**
* Fraction of the normal budget used for the REACTIVE re-trim after a provider
* context-overflow 400 the preventive estimate under-counted, so cut harder.
* Per-step fraction of the normal budget applied on the REACTIVE re-trim after a
* provider context-overflow 400 the preventive estimate under-counted, so cut
* harder. This is now applied ITERATIVELY: with `k` consecutive overflow turns the
* budget is scaled by `fraction ** k` (k=1 -> 0.5×, k=2 -> 0.25×, ), so recovery
* ESCALATES turn over turn until the replayed history finally fits, instead of the
* old single fixed 0.5× cut that could never un-brick a chat whose real model
* window is smaller than 0.5 × the (unconfigured, flat-default) base budget (#520).
*/
export const REPLAY_AGGRESSIVE_FRACTION = 0.5;
/**
* Absolute lower bound (tokens) on the escalating reactive budget: the iterative
* cut is clamped here so it CONVERGES (a fixed floor, not an ever-shrinking value
* that would eventually trim everything). Rationale: below ~8k tokens a chat cannot
* carry meaningful recent context, and even a small real model window comfortably
* fits this much keep-recent-turns still applies on top, so a handful of recent
* turns survive.
*
* #520 Option B: for a LARGE configured window this floor sits BELOW the configured
* replay budget, and recovery is allowed to escalate PAST the configured budget down
* to it "the chat must not brick on overflow" is an INVARIANT of the reactive
* branch, and a configured window is a claim about model capacity, not a promise
* about replay size; once the provider proves non-fit with a 400, reality beats the
* config. This does NOT change the NORMAL path's upper bound (#510 Option A still
* respects the configured budget while it fits) it is survival once non-fit is
* proven. For a SMALL configured window the effective floor is instead the old
* single 0.5× cut (never worse than before, and never RAISED above the budget); see
* {@link resolveEffectiveReplayThreshold}.
*/
export const REPLAY_MIN_FLOOR_TOKENS = 8_000;
/**
* Turns (a user message + its assistant/tool replies) kept FULL at the tail,
* including the current one never trimmed. Older turns are compacted first.
@@ -85,22 +110,48 @@ export function resolveReplayBudget(rawContextWindow: unknown): ReplayBudget {
}
/**
* The effective replay threshold for THIS turn, given the base budget and whether
* the PREVIOUS turn hit a context-overflow 400 (the reactive-recovery signal,
* `metadata.replayOverflow`). On recovery the base budget is scaled down by
* {@link REPLAY_AGGRESSIVE_FRACTION}: the overflowing turn produced no usage
* signal, so the preventive estimate under-counted and a normal-threshold trim may
* not shrink enough to fit this harder cut is what un-bricks the chat.
* The effective replay threshold for THIS turn, given the base budget and `k` the
* number of CONSECUTIVE preceding turns that hit a context-overflow 400 (the
* reactive-recovery signal `metadata.replayOverflowCount`, read from the last
* assistant row). On recovery the base budget is scaled down ITERATIVELY by
* {@link REPLAY_AGGRESSIVE_FRACTION} ** k and clamped at {@link REPLAY_MIN_FLOOR_TOKENS}:
* - k=0 -> base unchanged (no overflow: nothing to recover from).
* - k=1 -> floor(0.5 × base); k=2 -> floor(0.25 × base); each further consecutive
* overflow tightens the cut, so recovery ESCALATES until the history fits.
* - the escalation is clamped at the floor so it CONVERGES this is what un-bricks
* a chat whose real model window is smaller than a single 0.5× cut of the base
* (e.g. an unconfigured window: flat-default base 100k, real window <50k) (#520).
*
* The overflowing turn produced no usage signal, so the preventive estimate
* under-counted and a normal-threshold (or single fixed 0.5×) trim may not shrink
* enough to fit; the escalating cut is what recovers such a chat.
*
* A `null` base budget (trimming OFF) is passed through unchanged: an explicit
* off-switch is never overridden by the recovery path.
*
* The absolute floor is `min(REPLAY_MIN_FLOOR_TOKENS, floor(0.5 × base))` (#520
* Option B):
* - LARGE window (floor(0.5 × base) >= REPLAY_MIN_FLOOR_TOKENS) -> the fixed
* REPLAY_MIN_FLOOR_TOKENS, which is BELOW the configured budget: escalation is
* ALLOWED to cut past the configured budget down to this absolute minimum, so a
* chat whose real model window is far smaller than the configured window still
* un-bricks (the invariant beats a config reality disproved with a 400).
* - SMALL window (floor(0.5 × base) < REPLAY_MIN_FLOOR_TOKENS) -> floor(0.5 ×
* base), i.e. AT LEAST the old single 0.5× cut: recovery is never WORSE than the
* pre-#520 behavior, and the floor is still never RAISED above the configured
* budget itself (that would re-overflow the very window it was set for).
*/
export function resolveEffectiveReplayThreshold(
thresholdTokens: number | null,
priorOverflowed: boolean,
k: number,
): number | null {
if (!priorOverflowed || thresholdTokens == null) return thresholdTokens;
return Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION);
if (thresholdTokens == null || k <= 0) return thresholdTokens;
const scaled = Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION ** k);
const floor = Math.min(
REPLAY_MIN_FLOOR_TOKENS,
Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION),
);
return Math.max(scaled, floor);
}
/**
@@ -210,6 +210,55 @@ describe('PublicShareChatToolsService.forShare', () => {
});
});
describe('getSharePage under approved publication (owner decision: assistant sees the frozen saved version)', () => {
it('reads the FROZEN saved content/title returned by the boundary, not a live re-fetch', async () => {
// #370 Stage B owner decision: ALL public reads — including the share
// assistant's page-read tool — see the published saved version, so a
// visitor and the assistant see the same bytes. The content swap lives in
// resolveReadableSharePage (the single canonical boundary), so the tool
// simply consumes the frozen { page } it returns: the assistant must never
// re-read the live draft behind the boundary's back.
const frozenContent = {
type: 'doc',
content: [
{
type: 'paragraph',
content: [{ type: 'text', text: 'FROZEN saved version body' }],
},
],
};
// The boundary already swapped content+title to the manual version while
// preserving the live id (its unit test pins that); the tool sees this.
const frozenPage = {
id: 'page-1',
title: 'Saved Version Title',
deletedAt: null,
content: frozenContent,
};
const { svc, shareService } = makeService({
resolveReadableSharePage: jest
.fn()
.mockResolvedValue({ share: { id: 'SHARE-A' }, page: frozenPage }),
});
// updatePublicAttachments sanitizes the FROZEN page it was handed.
shareService.updatePublicAttachments.mockResolvedValue(frozenContent);
const tools = svc.forShare('SHARE-A', 'ws-1');
const out = (await (tools.getSharePage as unknown as ToolExec).execute({
pageId: 'page-1',
})) as { title: string; markdown: string };
// Sanitizer is handed the frozen page (never a separate live fetch).
expect(shareService.updatePublicAttachments).toHaveBeenCalledWith(
frozenPage,
);
// Title + body come from the saved (frozen) version.
expect(out.title).toBe('Saved Version Title');
expect(out.markdown).toContain('FROZEN saved version body');
});
});
describe('getSharePage non-resolving page (deleted / restricted / out-of-share)', () => {
it('resolveReadableSharePage returns null (e.g. soft-deleted page) => generic error, NO content sanitized/returned', async () => {
// The canonical boundary 404s a soft-deleted / restricted / out-of-tree
@@ -10,6 +10,12 @@ import { Transform } from 'class-transformer';
export type ContentFormat = 'json' | 'markdown' | 'html';
// READ-only rendering formats for `getPage` (#502). A superset of the writable
// `ContentFormat` with `text` — a flat, deterministic, machine-diffable text
// rendering — added. Kept SEPARATE from `ContentFormat` so the write path
// (createPage/updatePage `parseProsemirrorContent`) can never be handed `text`.
export type PageReadFormat = ContentFormat | 'text';
export class CreatePageDto {
@IsOptional()
@IsString()
+15 -3
View File
@@ -5,10 +5,11 @@ import {
IsOptional,
IsString,
IsUUID,
MaxLength,
} from 'class-validator';
import { Transform } from 'class-transformer';
import { ContentFormat } from './create-page.dto';
import { PageReadFormat } from './create-page.dto';
import { IsPageIdOrSlugId } from './page-identity.validator';
export class PageIdDto {
@@ -43,8 +44,19 @@ export class PageInfoDto extends PageIdDto {
@IsOptional()
@Transform(({ value }) => value?.toLowerCase())
@IsIn(['json', 'markdown', 'html'])
format?: ContentFormat;
@IsIn(['json', 'markdown', 'html', 'text'])
format?: PageReadFormat;
}
export class PageWorkTimeDto extends PageIdDto {
// Viewer IANA timezone for the per-day punch-card buckets (§6.3). Optional —
// falls back to UTC server-side. Length-capped so a bogus value cannot bloat
// the request; the value is only ever handed to Intl.DateTimeFormat, which
// throws on an unknown zone (caught by the controller → 400).
@IsOptional()
@IsString()
@MaxLength(64)
tz?: string;
}
export class DeletePageDto extends PageIdDto {
@@ -0,0 +1,76 @@
import { NotFoundException } from '@nestjs/common';
import { PageController } from './page.controller';
import { jsonToText, jsonToMarkdown } from '../../collaboration/collaboration.util';
// #502 READ: getPage `format:"text"` routes through the deterministic jsonToText
// path (placeholders for non-text nodes, block-per-line), while `format:"json"`
// (or none) returns the raw content and `format:"markdown"` still converts. This
// pins the CONTROLLER wiring with lightweight mocks (no DB needed — the jsonToText
// output contract itself is pinned in json-to-text-deterministic.spec.ts).
const CONTENT = {
type: 'doc',
content: [
{ type: 'heading', attrs: { level: 1 }, content: [{ type: 'text', text: 'Config' }] },
{
type: 'paragraph',
content: [
{ type: 'text', text: 'key=', marks: [{ type: 'bold' }] },
{ type: 'text', text: 'value' },
],
},
{ type: 'image', attrs: { src: 's' } },
],
};
function makeController(page: any): PageController {
const pageRepo = { findById: jest.fn().mockResolvedValue(page) } as any;
const pageAccessService = {
validateCanViewWithPermissions: jest
.fn()
.mockResolvedValue({ canEdit: true, hasRestriction: false }),
} as any;
// Only pageRepo + pageAccessService are exercised by getPage; the rest are
// never touched on this path, so undefined placeholders are fine.
return new PageController(
undefined as any, // pageService
pageRepo,
undefined as any, // pageHistoryService
undefined as any, // spaceAbility
pageAccessService,
undefined as any, // backlinkService
undefined as any, // labelService
undefined as any, // auditService
);
}
const user = { id: 'u1' } as any;
describe('PageController.getPage — format:"text" (#502)', () => {
it('returns deterministic flat text with placeholders for non-text nodes', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1', format: 'text' } as any, user);
expect(res.content).toBe(jsonToText(CONTENT, { deterministic: true }));
expect(res.content).toBe('Config\nkey=value\n[image]');
expect(res.permissions).toEqual({ canEdit: true, hasRestriction: false });
});
it('markdown format still converts to markdown (unchanged)', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1', format: 'markdown' } as any, user);
expect(res.content).toBe(jsonToMarkdown(CONTENT));
});
it('json / no format returns the raw ProseMirror content object', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1' } as any, user);
expect(res.content).toEqual(CONTENT); // untouched object, not a string
});
it('missing page -> NotFoundException', async () => {
const controller = makeController(null);
await expect(
controller.getPage({ pageId: 'nope', format: 'text' } as any, user),
).rejects.toBeInstanceOf(NotFoundException);
});
});
@@ -1,3 +1,8 @@
import {
BadRequestException,
ForbiddenException,
NotFoundException,
} from '@nestjs/common';
import { PageController } from './page.controller';
// Direct instantiation with stub deps. The Test.createTestingModule form failed
@@ -22,4 +27,88 @@ describe('PageController', () => {
it('should be defined', () => {
expect(controller).toBeDefined();
});
// #395 — the work-time endpoint must be gated exactly like /history.
describe('getPageWorkTime', () => {
const user = { id: 'u1' } as any;
function build(overrides: {
page?: any;
validate?: jest.Mock;
compute?: jest.Mock;
}) {
const pageRepo = { findById: jest.fn().mockResolvedValue(overrides.page) };
const pageAccessService = {
validateCanView: overrides.validate ?? jest.fn().mockResolvedValue(undefined),
};
const pageHistoryService = {
computeWorkTime:
overrides.compute ?? jest.fn().mockResolvedValue({ workMs: 0 }),
};
const c = new PageController(
{} as any,
pageRepo as any,
pageHistoryService as any,
{} as any,
pageAccessService as any,
{} as any,
{} as any,
{} as any,
);
return { c, pageRepo, pageAccessService, pageHistoryService };
}
it('404s when the page does not exist', async () => {
const { c } = build({ page: null });
await expect(
c.getPageWorkTime({ pageId: 'p1' } as any, user),
).rejects.toBeInstanceOf(NotFoundException);
});
it('enforces validateCanView before computing, then delegates with tz', async () => {
const validate = jest.fn().mockResolvedValue(undefined);
const compute = jest.fn().mockResolvedValue({ workMs: 42 });
const { c } = build({ page: { id: 'pg' }, validate, compute });
const out = await c.getPageWorkTime(
{ pageId: 'pg', tz: 'Europe/Moscow' } as any,
user,
);
expect(validate).toHaveBeenCalledWith({ id: 'pg' }, user);
expect(compute).toHaveBeenCalledWith('pg', 'Europe/Moscow');
expect(out).toEqual({ workMs: 42 });
});
it('propagates a denied view gate and does NOT reach compute (security)', async () => {
// If validateCanView is moved AFTER computeWorkTime, the timeline of a page
// the caller may not see would be read/estimated before the gate — this
// locks the order: a rejecting gate must short-circuit before any compute.
const validate = jest.fn().mockRejectedValue(new ForbiddenException());
const compute = jest.fn().mockResolvedValue({ workMs: 1 });
const { c, pageHistoryService } = build({
page: { id: 'pg' },
validate,
compute,
});
await expect(
c.getPageWorkTime({ pageId: 'pg' } as any, user),
).rejects.toBeInstanceOf(ForbiddenException);
expect(pageHistoryService.computeWorkTime).not.toHaveBeenCalled();
});
it('maps an unknown-timezone RangeError to a 400', async () => {
const compute = jest.fn().mockRejectedValue(new RangeError('bad tz'));
const { c } = build({ page: { id: 'pg' }, compute });
await expect(
c.getPageWorkTime({ pageId: 'pg', tz: 'X/Y' } as any, user),
).rejects.toBeInstanceOf(BadRequestException);
});
it('does not swallow a non-RangeError from the service', async () => {
const compute = jest.fn().mockRejectedValue(new Error('db down'));
const { c } = build({ page: { id: 'pg' }, compute });
await expect(
c.getPageWorkTime({ pageId: 'pg' } as any, user),
).rejects.toThrow('db down');
});
});
});
+43 -4
View File
@@ -21,6 +21,7 @@ import {
PageHistoryIdDto,
PageIdDto,
PageInfoDto,
PageWorkTimeDto,
} from './dto/page.dto';
import { PageHistoryService } from './services/page-history.service';
import { AuthUser } from '../../common/decorators/auth-user.decorator';
@@ -49,6 +50,7 @@ import { AddLabelsDto, RemoveLabelDto } from '../label/dto/label.dto';
import {
jsonToHtml,
jsonToMarkdown,
jsonToText,
} from '../../collaboration/collaboration.util';
import { AuditEvent, AuditResource } from '../../common/events/audit-events';
import {
@@ -93,10 +95,16 @@ export class PageController {
const permissions = { canEdit, hasRestriction };
if (dto.format && dto.format !== 'json' && page.content) {
const contentOutput =
dto.format === 'markdown'
? jsonToMarkdown(page.content)
: jsonToHtml(page.content);
let contentOutput: string;
if (dto.format === 'markdown') {
contentOutput = jsonToMarkdown(page.content);
} else if (dto.format === 'text') {
// #502: flat, deterministic, machine-diffable text (block-per-line,
// inline marks/anchors dropped, non-text nodes -> stable placeholders).
contentOutput = jsonToText(page.content, { deterministic: true });
} else {
contentOutput = jsonToHtml(page.content);
}
return {
...page,
content: contentOutput,
@@ -524,6 +532,32 @@ export class PageController {
return this.pageHistoryService.findHistoryByPageId(page.id, pagination);
}
@HttpCode(HttpStatus.OK)
@Post('/history/time')
async getPageWorkTime(
@Body() dto: PageWorkTimeDto,
@AuthUser() user: User,
) {
const page = await this.pageRepo.findById(dto.pageId);
if (!page) {
throw new NotFoundException('Page not found');
}
// Same view gate as /history and /history/info.
await this.pageAccessService.validateCanView(page, user);
try {
return await this.pageHistoryService.computeWorkTime(page.id, dto.tz);
} catch (e) {
// Intl.DateTimeFormat throws RangeError on an unknown IANA zone; surface
// it as a 400 rather than a 500.
if (e instanceof RangeError) {
throw new BadRequestException('Invalid timezone');
}
throw e;
}
}
@HttpCode(HttpStatus.OK)
@Post('/history/info')
async getPageHistoryInfo(
@@ -818,6 +852,11 @@ export class PageController {
throw new NotFoundException('Page not found');
}
// Target-only validateCanView is intentional: getPageBreadCrumbs returns
// the full ancestor chain WITHOUT per-ancestor permission filtering. Safe
// because page restrictions inherit down the tree, so any ancestor the
// caller could not view would already hide the target here — see the
// getPageBreadCrumbs docstring / #471.
await this.pageAccessService.validateCanView(page, user);
return this.pageService.getPageBreadCrumbs(page.id);
@@ -3,6 +3,23 @@ import { PageHistoryRepo } from '@docmost/db/repos/page/page-history.repo';
import { PageHistory } from '@docmost/db/types/entity.types';
import { PaginationOptions } from '@docmost/db/pagination/pagination-options';
import { CursorPaginationResult } from '@docmost/db/pagination/cursor-pagination';
import {
computeWorkTime,
bucketByDay,
DEFAULT_WORK_TIME_CONFIG,
WorkTimeConfig,
PerDay,
} from '../work-time';
export interface PageWorkTime {
workMs: number;
agentOnlyMs: number;
perDay: PerDay[];
/** the config actually used, so the UI can show "≈" + the T_gap threshold. */
config: WorkTimeConfig;
/** the tz the per-day buckets were computed in (echoed back for the label). */
tz: string;
}
@Injectable()
export class PageHistoryService {
@@ -23,4 +40,33 @@ export class PageHistoryService {
paginationOptions,
);
}
/**
* #395 estimate time worked on a page (§5) and bucket it into the viewer's
* calendar days for the punch-card (§6.3). Reads only the cheap history
* projection (no `content`); the estimate itself is a pure, deterministic
* function so it is unit-tested exhaustively without a DB.
*
* `tz` is the viewer's IANA zone (browser locale) it moves which day a
* session lands in and where its windows sit, but never the total (§10).
*/
async computeWorkTime(
pageId: string,
tz = 'UTC',
config?: Partial<WorkTimeConfig>,
): Promise<PageWorkTime> {
const rows = await this.pageHistoryRepo.findTimelineByPageId(pageId);
const result = computeWorkTime(rows, config);
const usedConfig: WorkTimeConfig = { ...DEFAULT_WORK_TIME_CONFIG, ...config };
// `bucketByDay` consumes the pure core's un-bucketed sessions here; the
// full session list is NOT shipped on the response (no client reads it).
const perDay = bucketByDay(result.sessions, tz);
return {
workMs: result.workMs,
agentOnlyMs: result.agentOnlyMs,
perDay,
config: usedConfig,
tz,
};
}
}
@@ -1071,6 +1071,29 @@ export class PageService {
});
}
/**
* Walk the ancestor chain of `childPageId` up to the space root, filtered
* ONLY by `deletedAt` (+ MAX_PAGE_TREE_DEPTH) WITHOUT per-ancestor
* permission filtering. Callers that expose this to a user (the
* `/breadcrumbs` endpoint) validate `validateCanView` on the TARGET page
* only, then return the whole chain of ancestor titles (#471).
*
* This is safe NOT a title leak because page restrictions inherit DOWN
* the tree: to view a page the caller must hold permission on EVERY
* restricted ancestor (`validateCanView` -> `canUserAccessPage` checks the
* full ancestor chain see page-access.service.ts / page-permission.repo.ts
* `canUserEditPage`). A restricted ancestor the caller may not see would
* therefore already hide the TARGET page itself, so every ancestor reachable
* here is one the caller is already entitled to view (content stays gated
* regardless getPage/getNode re-check permissions).
*
* Note the guarantee is the narrow "may view a descendant => may view its
* ancestors", NOT "space membership sees every page" restricted subtrees do
* hide pages from members. Per-ancestor permission filtering here was
* considered and declined as redundant given the inheritance invariant above
* (#471). The same chain feeds the web-UI breadcrumb bar under identical CASL
* scope.
*/
async getPageBreadCrumbs(childPageId: string, trx?: KyselyTransaction) {
const ancestors = await dbOrTx(this.db, trx)
.withRecursive('page_ancestors', (db) =>
@@ -0,0 +1,129 @@
import { bucketByDay, zonedDayStart } from './bucket-by-day';
import { computeWorkTime } from './compute-work-time';
import { WorkSession, TimelineSample } from './work-time.types';
const MIN = 60 * 1000;
const HOUR = 60 * MIN;
function work(start: number, end: number): WorkSession {
return { start, end, class: 'work' };
}
function agent(start: number, end: number): WorkSession {
return { start, end, class: 'agent_only' };
}
function sumActive(perDay: ReturnType<typeof bucketByDay>): number {
return perDay.reduce((a, d) => a + d.activeMs, 0);
}
describe('bucketByDay', () => {
it('Σ activeMs == workMs — the §6.3 consistency invariant', () => {
const rows: TimelineSample[] = [
{ createdAt: '2026-07-04T03:40:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-04T03:49:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-04T18:11:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-06T15:34:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
];
const r = computeWorkTime(rows);
const perDay = bucketByDay(r.sessions, 'UTC');
expect(sumActive(perDay)).toBe(r.workMs);
});
it('empty input → no days', () => {
expect(bucketByDay([], 'UTC')).toEqual([]);
});
it('midnight-crossing session splits across two days, sum preserved (§9#9)', () => {
const start = Date.UTC(2026, 0, 10, 23, 14);
const end = Date.UTC(2026, 0, 11, 0, 40);
const perDay = bucketByDay([work(start, end)], 'UTC');
expect(perDay).toHaveLength(2);
expect(perDay[0].dayISO).toBe('2026-01-10');
expect(perDay[1].dayISO).toBe('2026-01-11');
expect(perDay[0].activeMs).toBe(46 * MIN); // 23:14 → 24:00
expect(perDay[1].activeMs).toBe(40 * MIN); // 00:00 → 00:40
expect(sumActive(perDay)).toBe(end - start);
});
it('empty days between active days are emitted, not skipped (§9#12)', () => {
const d1 = work(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 11, 0));
const d3 = work(Date.UTC(2026, 0, 12, 10, 0), Date.UTC(2026, 0, 12, 11, 0));
const perDay = bucketByDay([d1, d3], 'UTC');
expect(perDay.map((d) => d.dayISO)).toEqual([
'2026-01-10',
'2026-01-11',
'2026-01-12',
]);
expect(perDay[1].activeMs).toBe(0);
expect(perDay[1].windows).toEqual([]);
});
it('agent_only windows are drawn but excluded from activeMs', () => {
const w = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 10, 0));
const a = agent(Date.UTC(2026, 0, 10, 14, 0), Date.UTC(2026, 0, 10, 14, 30));
const perDay = bucketByDay([w, a], 'UTC');
expect(perDay).toHaveLength(1);
expect(perDay[0].activeMs).toBe(1 * HOUR);
expect(perDay[0].agentMs).toBe(30 * MIN);
expect(perDay[0].windows.map((x) => x.class)).toEqual(['work', 'agent_only']);
});
it('work and agent_only are unioned SEPARATELY (agent does not swallow work)', () => {
// Overlapping work + agent windows on the same day.
const w = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 11, 0));
const a = agent(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 12, 0));
const perDay = bucketByDay([w, a], 'UTC');
expect(perDay[0].activeMs).toBe(2 * HOUR);
expect(perDay[0].agentMs).toBe(2 * HOUR);
});
it('overlapping same-class sessions are UNIONed, not summed (no double-count)', () => {
// Two work sessions that overlap 10:00–10:30 on one day.
const a = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 10, 30));
const b = work(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 11, 0));
const perDay = bucketByDay([a, b], 'UTC');
expect(perDay).toHaveLength(1);
// Union 09:00–11:00 = 2h, NOT 90m + 60m = 150m.
expect(perDay[0].activeMs).toBe(2 * HOUR);
// The drawn windows are also merged to one, so the punch-card cannot render
// an overlapping double bar.
expect(perDay[0].windows).toHaveLength(1);
expect(perDay[0].windows[0].start).toBe(a.start);
expect(perDay[0].windows[0].end).toBe(b.end);
});
it('DST fall-back: a full 25-hour day still balances (§9#14)', () => {
// America/New_York ends DST 2026-11-01 (25h day).
const tz = 'America/New_York';
const dayStart = zonedDayStart(Date.UTC(2026, 10, 1, 12, 0), tz);
const nextStart = zonedDayStart(dayStart + 26 * HOUR, tz);
expect(nextStart - dayStart).toBe(25 * HOUR);
const perDay = bucketByDay([work(dayStart, nextStart)], tz);
expect(perDay).toHaveLength(1);
expect(perDay[0].dayISO).toBe('2026-11-01');
expect(perDay[0].activeMs).toBe(25 * HOUR);
expect(sumActive(perDay)).toBe(nextStart - dayStart);
});
it('DST spring-forward: a full 23-hour day still balances (§9#14)', () => {
// America/New_York starts DST 2026-03-08 (23h day).
const tz = 'America/New_York';
const dayStart = zonedDayStart(Date.UTC(2026, 2, 8, 12, 0), tz);
const nextStart = zonedDayStart(dayStart + 26 * HOUR, tz);
expect(nextStart - dayStart).toBe(23 * HOUR);
const perDay = bucketByDay([work(dayStart, nextStart)], tz);
expect(perDay).toHaveLength(1);
expect(perDay[0].activeMs).toBe(23 * HOUR);
expect(sumActive(perDay)).toBe(nextStart - dayStart);
});
it('tz changes the day a session lands in but not the total', () => {
const start = Date.UTC(2026, 0, 10, 2, 0); // 02:00 UTC
const end = Date.UTC(2026, 0, 10, 3, 0);
const utc = bucketByDay([work(start, end)], 'UTC');
const ny = bucketByDay([work(start, end)], 'America/New_York'); // 21:00 prev day
expect(utc[0].dayISO).toBe('2026-01-10');
expect(ny[0].dayISO).toBe('2026-01-09');
expect(sumActive(utc)).toBe(sumActive(ny));
});
});
@@ -0,0 +1,180 @@
import { WorkSession, PerDay, DayWindow } from './work-time.types';
/**
* Merge intervals into a disjoint, sorted union. Overlapping OR touching
* intervals are joined. Empty input [].
*/
function union(intervals: Array<[number, number]>): Array<[number, number]> {
if (intervals.length === 0) return [];
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
const out: Array<[number, number]> = [];
let [curStart, curEnd] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= curEnd) {
if (e > curEnd) curEnd = e;
} else {
out.push([curStart, curEnd]);
curStart = s;
curEnd = e;
}
}
out.push([curStart, curEnd]);
return out;
}
// Cache one Intl formatter per tz — constructing them is comparatively costly.
const fmtCache = new Map<string, Intl.DateTimeFormat>();
function partsFmt(tz: string): Intl.DateTimeFormat {
let fmt = fmtCache.get(tz);
if (!fmt) {
fmt = new Intl.DateTimeFormat('en-US', {
timeZone: tz,
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit',
second: '2-digit',
hour12: false,
});
fmtCache.set(tz, fmt);
}
return fmt;
}
interface WallParts {
year: number;
month: number;
day: number;
hour: number;
minute: number;
second: number;
}
/** Wall-clock parts of an instant in `tz` (DST-correct, via Intl). */
function wallParts(ms: number, tz: string): WallParts {
const parts = partsFmt(tz).formatToParts(new Date(ms));
const get = (type: string) =>
Number(parts.find((p) => p.type === type)?.value ?? '0');
let hour = get('hour');
// Intl emits "24" for midnight under some engines/locales; normalize to 0.
if (hour === 24) hour = 0;
return {
year: get('year'),
month: get('month'),
day: get('day'),
hour,
minute: get('minute'),
second: get('second'),
};
}
/** tz offset (wall − real) at an instant, in ms. */
function offset(ms: number, tz: string): number {
const p = wallParts(ms, tz);
const asUTC = Date.UTC(p.year, p.month - 1, p.day, p.hour, p.minute, p.second);
return asUTC - ms;
}
/**
* Epoch-ms of the local-midnight day start of `ms` in `tz`. DST-correct: takes
* the calendar day of the instant, its wall-midnight, then converts back with
* the offset that actually applies AT that midnight (refined once). The rare
* tz-with-a-DST-transition-exactly-at-midnight case is a documented edge (§9#14).
*/
export function zonedDayStart(ms: number, tz: string): number {
const p = wallParts(ms, tz);
const wallMidnightAsUTC = Date.UTC(p.year, p.month - 1, p.day, 0, 0, 0);
let start = wallMidnightAsUTC - offset(ms, tz);
// Refine with the offset at the computed midnight (DST may differ from `ms`).
start = wallMidnightAsUTC - offset(start, tz);
return start;
}
/** The next local midnight after `dayStart` (handles 23/25h DST days). */
function nextDayStart(dayStart: number, tz: string): number {
// +26h always lands inside the NEXT calendar day (day length ∈ [23h,25h]),
// never two days ahead; startOf('day') of it is the next midnight.
return zonedDayStart(dayStart + 26 * 60 * 60 * 1000, tz);
}
function isoDay(dayStart: number, tz: string): string {
const p = wallParts(dayStart, tz);
const pad = (n: number) => String(n).padStart(2, '0');
return `${p.year}-${pad(p.month)}-${pad(p.day)}`;
}
/** Clip a union to [lo, hi) and emit windows of `class`. */
function clip(
merged: Array<[number, number]>,
lo: number,
hi: number,
cls: DayWindow['class'],
): DayWindow[] {
const out: DayWindow[] = [];
for (const [s, e] of merged) {
const start = Math.max(s, lo);
const end = Math.min(e, hi);
if (end > start) out.push({ start, end, class: cls });
}
return out;
}
/**
* #395 §6.3 bucket sessions into calendar days of `tz` for the punch-card.
* Pure and deterministic. `work` and `agent_only` are unioned SEPARATELY (else
* agent windows would swallow work windows on overlap), then each union is split
* at tz midnight boundaries (`startOf('day')` in tz, NOT "+24h" DST-safe §9#14)
* and clipped to each day.
*
* By construction Σ perDay.activeMs == workMs: the days are a partition of the
* `work` union no loss, no dup, even on 23/25h DST days. `agent_only` windows
* are drawn but NOT in activeMs. Empty days between the first and last active day
* are emitted (empty track + "—") so the rhythm/pauses stay visible.
*/
export function bucketByDay(sessions: WorkSession[], tz: string): PerDay[] {
const uWork = union(
sessions.filter((s) => s.class === 'work').map((s) => [s.start, s.end]),
);
const uAgent = union(
sessions
.filter((s) => s.class === 'agent_only')
.map((s) => [s.start, s.end]),
);
if (uWork.length === 0 && uAgent.length === 0) return [];
const minStart = Math.min(
uWork.length ? uWork[0][0] : Infinity,
uAgent.length ? uAgent[0][0] : Infinity,
);
const maxEnd = Math.max(
uWork.length ? uWork[uWork.length - 1][1] : -Infinity,
uAgent.length ? uAgent[uAgent.length - 1][1] : -Infinity,
);
const perDay: PerDay[] = [];
let dayStart = zonedDayStart(minStart, tz);
// Guard against a pathological non-advancing boundary.
let guard = 0;
while (dayStart < maxEnd && guard < 100000) {
guard++;
const dayEnd = nextDayStart(dayStart, tz);
const workWin = clip(uWork, dayStart, dayEnd, 'work');
const agentWin = clip(uAgent, dayStart, dayEnd, 'agent_only');
const activeMs = workWin.reduce((a, w) => a + (w.end - w.start), 0);
const agentMs = agentWin.reduce((a, w) => a + (w.end - w.start), 0);
const windows = [...workWin, ...agentWin].sort((a, b) => a.start - b.start);
perDay.push({
day: dayStart,
dayISO: isoDay(dayStart, tz),
activeMs,
agentMs,
windows,
});
dayStart = dayEnd;
}
return perDay;
}
@@ -0,0 +1,358 @@
import { computeWorkTime } from './compute-work-time';
import { bucketByDay } from './bucket-by-day';
import { TimelineSample, WorkSession } from './work-time.types';
const MIN = 60 * 1000;
/** Union wall-clock of a set of intervals (touching intervals merge). */
function unionMs(intervals: Array<[number, number]>): number {
if (intervals.length === 0) return 0;
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
let total = 0;
let [cs, ce] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= ce) {
if (e > ce) ce = e;
} else {
total += ce - cs;
cs = s;
ce = e;
}
}
return total + (ce - cs);
}
const ivsOf = (sessions: WorkSession[], cls?: string): Array<[number, number]> =>
sessions
.filter((x) => cls == null || x.class === cls)
.map((x) => [x.start, x.end] as [number, number]);
function s(
iso: string,
opts: {
source?: string | null;
chat?: string | null;
kind?: string | null;
by?: string | null;
} = {},
): TimelineSample {
return {
createdAt: `${iso}Z`,
lastUpdatedById: opts.by ?? 'human-1',
lastUpdatedSource: opts.source === undefined ? 'user' : opts.source,
lastUpdatedAiChatId: opts.chat ?? null,
kind: opts.kind ?? null,
};
}
// §7 config: T_gap=30m, P_in+P_out=10m, P_single=2m.
const S7 = { tGap: 30 * MIN, agentTGap: 30 * MIN, pIn: 5 * MIN, pOut: 5 * MIN, pSingle: 2 * MIN };
describe('computeWorkTime', () => {
it('§7 fixture — sessionizes 20-ish samples to ≈1h32m, not the ≈60h naive span', () => {
const rows: TimelineSample[] = [
// S1: multi-sample morning session
s('2026-07-04T03:40:00'),
s('2026-07-04T03:45:00'),
s('2026-07-04T03:49:00'),
// S2: agent burst (one run) then human supervising → class work
s('2026-07-04T15:43:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T15:47:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T15:50:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T16:13:00'),
// S3: single
s('2026-07-04T18:11:00'),
// S4: multi-sample evening session
s('2026-07-04T19:38:00'),
s('2026-07-04T19:44:00'),
s('2026-07-04T19:54:00'),
// S5 / S6: two singles two days later, 44m apart → two sessions at T_gap=30
s('2026-07-06T15:34:00'),
s('2026-07-06T16:18:00'),
];
const r = computeWorkTime(rows, S7);
// 19 + 40 + 2 + 26 + 2 + 2 = 91 minutes.
expect(r.workMs).toBe(91 * MIN);
expect(r.agentOnlyMs).toBe(0);
expect(r.sessions).toHaveLength(6);
expect(r.sessions.every((x) => x.class === 'work')).toBe(true);
const naiveSpan =
new Date('2026-07-06T16:18:00Z').getTime() -
new Date('2026-07-04T03:40:00Z').getTime();
expect(naiveSpan).toBeGreaterThan(60 * 60 * MIN); // ≈60h
expect(r.workMs).toBeLessThan(naiveSpan / 30); // dramatically smaller
});
it('n=0 → zero, no sessions', () => {
const r = computeWorkTime([]);
expect(r).toEqual({ workMs: 0, agentOnlyMs: 0, sessions: [] });
});
it('n=1 human → one P_single work session', () => {
const r = computeWorkTime([s('2026-07-04T10:00:00')], S7);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.workMs).toBe(2 * MIN);
expect(r.agentOnlyMs).toBe(0);
// pre-roll only: [t − P_single, t]
expect(r.sessions[0].end).toBe(new Date('2026-07-04T10:00:00Z').getTime());
});
it('n=1 agent → one P_single agent_only session, work=0 (§9#2)', () => {
const r = computeWorkTime(
[s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' })],
S7,
);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('agent_only');
expect(r.workMs).toBe(0);
expect(r.agentOnlyMs).toBe(2 * MIN);
});
it('MUST close the last session — the newest session is not lost (§9#1)', () => {
// Two singles a day apart: without the post-loop close, the 2nd is dropped.
const rows = [s('2026-07-04T10:00:00'), s('2026-07-05T10:00:00')];
const r = computeWorkTime(rows, S7);
expect(r.sessions).toHaveLength(2);
const lastStart = Math.max(...r.sessions.map((x) => x.start));
expect(lastStart).toBe(
new Date('2026-07-05T10:00:00Z').getTime() - 2 * MIN,
);
expect(r.workMs).toBe(4 * MIN);
});
it('agent-burst collapse: density does not inflate — length = wall-clock', () => {
const span = ['00', '01', '02', '03', '04', '05', '06'];
const dense: TimelineSample[] = span.map((sec) =>
s(`2026-07-04T10:00:${sec}`, { source: 'agent', chat: 'c1', kind: 'agent' }),
);
const sparse: TimelineSample[] = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:00:06', { source: 'agent', chat: 'c1', kind: 'agent' }),
];
const rDense = computeWorkTime(dense, S7);
const rSparse = computeWorkTime(sparse, S7);
// Same 6-second wall-clock span → same estimate regardless of snapshot count.
expect(rDense.agentOnlyMs).toBe(rSparse.agentOnlyMs);
expect(rDense.sessions).toHaveLength(1);
expect(rDense.sessions[0].class).toBe('agent_only');
});
it('supervisory agent time inside a human session counts as work, not agent', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:05:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:12:00'), // human within T_gap
];
const r = computeWorkTime(rows, S7);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.agentOnlyMs).toBe(0);
expect(r.workMs).toBeGreaterThan(0);
});
it('a DIFFERENT aiChatId breaks the burst — two agent runs, idle gap excluded', () => {
// Run c1 ends 10:05, run c2 starts 10:20 (15m > agentTGap 7m) → two sessions.
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:05:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:20:00', { source: 'agent', chat: 'c2', kind: 'agent' }),
s('2026-07-04T10:25:00', { source: 'agent', chat: 'c2', kind: 'agent' }),
];
const r = computeWorkTime(rows); // default agentTGap = 7m
expect(r.sessions).toHaveLength(2);
expect(r.sessions.every((x) => x.class === 'agent_only')).toBe(true);
// The 15m idle gap between the two runs is NOT counted.
const run1 = 5 * MIN + 5 * MIN + 5 * MIN; // pIn + span + pOut
expect(r.agentOnlyMs).toBe(2 * run1);
});
it('idle pulse (same/null run) is a full activity sample that continues a burst', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
// idle flush 4m later, null run id → continues the burst, not a new one
s('2026-07-04T10:04:00', { source: 'agent', chat: null, kind: 'idle' }),
s('2026-07-04T10:08:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
// burst span 10:00→10:08 (+pIn/pOut) = 8 + 10 = 18m
expect(r.agentOnlyMs).toBe(18 * MIN);
});
it('a USER-sourced idle breaks an agent burst → session is work, not agent_only', () => {
// A human supervision idle inherits source=user (aiChatId:null) and must NOT
// be swallowed into the agent burst. Δ=3m is within the default agentTGap so
// the two samples stay one session — but its class flips to `work`.
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:03:00', { source: 'user', chat: null, kind: 'idle' }),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.workMs).toBeGreaterThan(0);
// The human idle is NOT captured as agent_only time.
expect(r.agentOnlyMs).toBe(0);
// Σ over `work` sessions == workMs and Σ over `agent_only` == agentOnlyMs.
const sum = (cls: string) =>
r.sessions
.filter((x) => x.class === cls)
.reduce((acc, x) => acc + (x.end - x.start), 0);
expect(sum('work')).toBe(r.workMs);
expect(sum('agent_only')).toBe(r.agentOnlyMs);
});
it('idle pulse keeps a human writing session visible (not excluded)', () => {
const rows = [
s('2026-07-04T10:00:00'),
s('2026-07-04T10:08:00', { kind: 'idle' }), // pulse within T_gap
s('2026-07-04T10:15:00'),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
// span 10:00→10:15 + pIn/pOut = 15 + 10 = 25m
expect(r.workMs).toBe(25 * MIN);
});
it('git-source samples are excluded (§10 excludeGit)', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'git', kind: 'boundary' }),
s('2026-07-04T10:01:00', { source: 'git', kind: 'boundary' }),
];
expect(computeWorkTime(rows).workMs).toBe(0);
// ...but honoured off:
expect(
computeWorkTime(rows, { excludeGit: false }).workMs,
).toBeGreaterThan(0);
});
it('rejects an invalid config (tGap < pIn + pOut)', () => {
expect(() =>
computeWorkTime([s('2026-07-04T10:00:00')], {
tGap: 5 * MIN,
pIn: 5 * MIN,
pOut: 5 * MIN,
}),
).toThrow(/tGap/);
});
it('rejects an invalid config (2·agentTGap < pIn + pOut)', () => {
// tGap (default 15m) still ≥ pIn+pOut, so only the 2·agentTGap guard trips.
// Without it a short session of one class between two of the other could
// produce a NON-adjacent cross-class overlap the adjacent-only clip misses.
expect(() =>
computeWorkTime([s('2026-07-04T10:00:00')], {
agentTGap: 2 * MIN,
pIn: 5 * MIN,
pOut: 5 * MIN,
}),
).toThrow(/agentTGap/);
});
// F1 — cross-class double-count. On the DEFAULT config agentTGap (7m) < pIn+pOut
// (10m), so a `work` session ending in an agent segment and a nearby separate
// `agent_only` run (gap in (7m,10m]) used to produce OVERLAPPING padded
// intervals — the same wall-clock counted into BOTH workMs and agentOnlyMs. The
// cross-class padding clip must make the two per-class unions disjoint.
it('does NOT double-count wall-clock across work/agent_only (§F1)', () => {
// user@0s ; agent(chatX)@60s (breaks into a work session with the human) ;
// agent(chatY)@560s,590s (a separate agent_only run). Raw gap between the work
// session (ends 60s) and the agent run (starts 560s) is 500s ∈ (agentTGap,
// pIn+pOut] once padded — the classic overlap window.
const rows: TimelineSample[] = [
s('2026-07-04T00:00:00'), // user @ 0s
s('2026-07-04T00:01:00', { source: 'agent', chat: 'cX', kind: 'agent' }), // @ 60s
s('2026-07-04T00:09:20', { source: 'agent', chat: 'cY', kind: 'agent' }), // @ 560s
s('2026-07-04T00:09:50', { source: 'agent', chat: 'cY', kind: 'agent' }), // @ 590s
];
const r = computeWorkTime(rows); // DEFAULT config
// Both classes present.
expect(r.workMs).toBeGreaterThan(0);
expect(r.agentOnlyMs).toBeGreaterThan(0);
// Per-class metrics are exactly their own union (union, not Σ).
expect(r.workMs).toBe(unionMs(ivsOf(r.sessions, 'work')));
expect(r.agentOnlyMs).toBe(unionMs(ivsOf(r.sessions, 'agent_only')));
// The F1 invariant: work-union and agent-union are cross-class-disjoint, so
// the union of ALL padded intervals equals workMs + agentOnlyMs (no overlap).
// With the clip disabled this fails (union < sum by the 100s overlap).
expect(unionMs(ivsOf(r.sessions))).toBe(r.workMs + r.agentOnlyMs);
});
// F1 property/fuzz — random timelines across several timezones must uphold the
// work-time invariants. Backs the (corrected) PR claim of a real fuzz test.
it('property: random timelines uphold union & cross-class-disjoint invariants', () => {
// Deterministic LCG (numerical-recipes constants) so a failure is reproducible.
let seed = 0x9e3779b9 >>> 0;
const rand = () => {
seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0;
return seed / 0x100000000;
};
const pick = <T>(arr: T[]): T => arr[Math.floor(rand() * arr.length)];
const tzs = [
'UTC',
'America/New_York',
'Europe/Moscow',
'Australia/Lord_Howe', // 30-min DST offset — a nasty bucket stress
];
const base = Date.UTC(2026, 5, 1, 0, 0, 0); // 2026-06-01Z
const chats = ['c1', 'c2', 'c3'];
for (let iter = 0; iter < 250; iter++) {
const tz = pick(tzs);
const n = 2 + Math.floor(rand() * 18); // 2..19 rows
const rows: TimelineSample[] = [];
// Walk time forward by a random inter-sample gap. The gap distribution is
// centred on the DANGEROUS band — a bit under to a bit over pIn+pOut (10m)
// AND straddling agentTGap (7m) — so adjacent samples routinely split into
// separate sessions whose ±P padding would overlap if a class boundary sits
// there. Mixing user/agent classes at these gaps reliably manufactures the
// work-ending-in-agent → agent_only cross-class boundary F1 is about, plus
// dense within-class runs (occasional 0–2m gaps) that exercise the union.
let t = base + Math.floor(rand() * 60 * MIN);
for (let i = 0; i < n; i++) {
const roll = rand();
const gap =
roll < 0.25
? Math.floor(rand() * 2 * MIN) // dense burst (same-class union)
: roll < 0.85
? 5 * MIN + Math.floor(rand() * 8 * MIN) // 5–13m: the split band
: 20 * MIN + Math.floor(rand() * 40 * MIN); // long idle → new day-ish
t += gap;
const iso = new Date(t).toISOString().slice(0, 19); // 'YYYY-MM-DDTHH:MM:SS'
const isAgent = rand() < 0.5;
rows.push(
isAgent
? s(iso, { source: 'agent', chat: pick(chats), kind: 'agent' })
: s(iso, { source: 'user', kind: rand() < 0.3 ? 'idle' : 'manual' }),
);
}
const r = computeWorkTime(rows); // DEFAULT config
const workIvs = ivsOf(r.sessions, 'work');
const agentIvs = ivsOf(r.sessions, 'agent_only');
// (1) each metric is exactly its per-class union (catches a union→Σ regress).
expect(r.workMs).toBe(unionMs(workIvs));
expect(r.agentOnlyMs).toBe(unionMs(agentIvs));
// (2) NO cross-class overlap: union(all) == workMs + agentOnlyMs (F1).
expect(unionMs(ivsOf(r.sessions))).toBe(r.workMs + r.agentOnlyMs);
// (3) bucket invariant: Σ per-day activeMs == workMs (§6.3).
const perDay = bucketByDay(r.sessions, tz);
const sumActive = perDay.reduce((a, d) => a + d.activeMs, 0);
expect(sumActive).toBe(r.workMs);
}
});
});
@@ -0,0 +1,274 @@
import {
TimelineSample,
WorkSession,
WorkTimeResult,
} from './work-time.types';
import { WorkTimeConfig, resolveWorkTimeConfig } from './work-time.config';
/** A normalized activity sample (one history row), createdAt as epoch-ms. */
interface NormSample {
t: number;
isAgent: boolean;
aiChatId: string | null;
kind: string | null;
}
/**
* A collapsed segment: either a scalar sample (t_start == t_end) or an
* agent-burst spanning several agent samples of one run (§5.1). It participates
* in sessionization as a single "sample".
*/
interface Segment {
tStart: number;
tEnd: number;
isAgent: boolean;
}
function toMs(v: Date | string | number): number {
if (v instanceof Date) return v.getTime();
if (typeof v === 'number') return v;
return new Date(v).getTime();
}
/**
* Normalize raw rows sorted, deduped activity samples. `git` is dropped when
* configured; every other kind (incl. `idle` the continuous-work pulse §3) is
* a real activity sample. Sort is by createdAt ASC; samples whose timestamps
* fall in the same `dedupRoundMs` bucket collapse to one (§9#7: a synchronous
* boundary row + the immediate agent snapshot can share a createdAt). A merged
* sample is human unless EVERY member is an agent, so supervision never gets
* mis-attributed to the agent.
*/
function normalize(
rows: TimelineSample[],
config: WorkTimeConfig,
): NormSample[] {
const samples: NormSample[] = [];
for (const row of rows) {
const source = row.lastUpdatedSource;
if (config.excludeGit && source === 'git') continue;
samples.push({
t: toMs(row.createdAt),
isAgent: source === 'agent',
aiChatId: row.lastUpdatedAiChatId ?? null,
kind: row.kind ?? null,
});
}
samples.sort((a, b) => a.t - b.t);
if (config.dedupRoundMs <= 0 || samples.length < 2) return samples;
const deduped: NormSample[] = [];
for (const s of samples) {
const prev = deduped[deduped.length - 1];
if (prev && s.t - prev.t < config.dedupRoundMs) {
// Merge into the previous sample. Human wins the class; keep the earliest
// t; keep a non-null aiChatId if either has one (so a bare boundary row
// does not erase the run id).
prev.isAgent = prev.isAgent && s.isAgent;
prev.aiChatId = prev.aiChatId ?? s.aiChatId;
// Prefer the more specific kind (a real kind over a null/boundary) only
// matters for burst continuation; keep prev.kind (earliest) as-is.
continue;
}
deduped.push({ ...s });
}
return deduped;
}
/**
* Collapse consecutive same-run agent samples into one burst segment (§5.1) so a
* dense burst (8 snapshots in 7 minutes) contributes its wall-clock, not a count
* × block. A burst is broken by any sample NOT continuing the same aiChatId
* agent run: a non-agent sample, a `boundary` (actor transition), or a DIFFERENT
* aiChatId. Only an AGENT-sourced `idle` pulse with the SAME or a null aiChatId
* continues the burst (its label lags the real edit maxWait, well within
* rounding); a user-sourced `idle` (a human supervision pulse) breaks it.
*/
function collapse(samples: NormSample[], config: WorkTimeConfig): Segment[] {
const segments: Segment[] = [];
let burst: { chatId: string | null; tStart: number; tEnd: number } | null =
null;
const flush = () => {
if (!burst) return;
let tEnd = burst.tEnd;
if (config.burstCapMs != null && tEnd - burst.tStart > config.burstCapMs) {
tEnd = burst.tStart + config.burstCapMs;
}
segments.push({ tStart: burst.tStart, tEnd, isAgent: true });
burst = null;
};
for (const s of samples) {
// An agent-sourced idle pulse continues the current agent burst (same or
// null run id). A user-sourced idle (human supervision) must NOT be swallowed
// here — it falls through to the human branch so the session flips to `work`.
if (
burst &&
s.kind === 'idle' &&
s.isAgent &&
(s.aiChatId === burst.chatId || s.aiChatId == null)
) {
burst.tEnd = s.t;
continue;
}
if (s.isAgent && s.kind !== 'boundary') {
if (burst && burst.chatId === s.aiChatId) {
burst.tEnd = s.t;
} else {
flush();
burst = { chatId: s.aiChatId, tStart: s.t, tEnd: s.t };
}
continue;
}
// A human sample, a boundary, or an agent-boundary: breaks the burst and is
// itself a zero-width segment (its class follows its own source).
flush();
segments.push({ tStart: s.t, tEnd: s.t, isAgent: s.isAgent });
}
flush();
return segments;
}
function gapThreshold(
a: Segment,
b: Segment,
config: WorkTimeConfig,
): number {
return a.isAgent && b.isAgent ? config.agentTGap : config.tGap;
}
/** Merge intervals; overlapping OR touching intervals are unioned. */
function unionDuration(intervals: Array<[number, number]>): number {
if (intervals.length === 0) return 0;
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
let total = 0;
let [curStart, curEnd] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= curEnd) {
if (e > curEnd) curEnd = e;
} else {
total += curEnd - curStart;
curStart = s;
curEnd = e;
}
}
total += curEnd - curStart;
return total;
}
/**
* #395 core estimate time worked on a page from its history timeline (§5).
* Pure and deterministic: no DB, no clock, no I/O.
*
* Pipeline: normalize+dedup collapse agent bursts ONE sessionization pass
* over all segments (threshold depends on the pair: both-agent agentTGap, else
* tGap; the last session is ALWAYS closed after the loop) class per finished
* session (all-agent agent_only, else work) pad each session (multi-sample
* [firstP_in, last+P_out]; lone scalar [tP_single, t]) clip padding of
* adjacent DIFFERENT-class sessions at the raw-gap midpoint (so work/agent_only
* never overlap) metrics are the union wall-clock within each class (union, not
* Σ, so overlaps never double, and cross-class-disjoint by the clip above).
*/
export function computeWorkTime(
rows: TimelineSample[],
config?: Partial<WorkTimeConfig>,
): WorkTimeResult {
const cfg = resolveWorkTimeConfig(config);
const samples = normalize(rows, cfg);
const segments = collapse(samples, cfg);
// Sessionize — one pass over ALL segments.
const rawSessions: Segment[][] = [];
let cur: Segment[] | null = null;
for (const seg of segments) {
if (cur == null) {
cur = [seg];
} else {
const last = cur[cur.length - 1];
if (seg.tStart - last.tEnd <= gapThreshold(last, seg, cfg)) {
cur.push(seg);
} else {
rawSessions.push(cur);
cur = [seg];
}
}
}
if (cur != null) rawSessions.push(cur); // MUST close the last session (§5, §9#1)
// A finished session with BOTH its raw (unpadded) span and its padded bounds.
// `rawSessions` are already in ascending time order, so `built` is too.
interface BuiltSession {
rawStart: number;
rawEnd: number;
padStart: number;
padEnd: number;
cls: WorkSession['class'];
}
const built: BuiltSession[] = [];
for (const segs of rawSessions) {
const first = segs[0];
const last = segs[segs.length - 1];
const cls = segs.every((s) => s.isAgent) ? 'agent_only' : 'work';
let padStart: number;
let padEnd: number;
if (segs.length === 1 && first.tStart === first.tEnd) {
// Lone single-instant session (one scalar, or a one-snapshot agent run):
// pre-roll only, no invented "future" work (§5).
padStart = first.tStart - cfg.pSingle;
padEnd = first.tStart;
} else {
padStart = first.tStart - cfg.pIn;
padEnd = last.tEnd + cfg.pOut;
}
built.push({
rawStart: first.tStart,
rawEnd: last.tEnd,
padStart,
padEnd,
cls,
});
}
// Clip cross-class padding so a `work` and an `agent_only` session that abut
// never claim the same wall-clock. For each ADJACENT pair of DIFFERENT classes,
// cap the earlier session's trailing pad and the later session's leading pad at
// the MIDPOINT of the raw (unpadded) inactivity gap between them: the earlier
// padded interval then ends ≤ midpoint and the later one starts ≥ midpoint, so
// the two are disjoint (they touch at most at the midpoint). This makes the
// per-class unions (workMs / agentOnlyMs) cross-class-disjoint BY CONSTRUCTION
// — closing the double-count where a work session ending in an agent segment
// and a nearby agent_only session (gap in (agentTGap, pIn+pOut]) overlapped and
// were counted into both metrics (§5, §9). Within-class adjacency is left
// untouched: `unionDuration` already dedups it, and clipping there could perturb
// the per-class metric value.
for (let i = 1; i < built.length; i++) {
const a = built[i - 1];
const b = built[i];
if (a.cls === b.cls) continue;
const midpoint = (a.rawEnd + b.rawStart) / 2;
if (a.padEnd > midpoint) a.padEnd = midpoint;
if (b.padStart < midpoint) b.padStart = midpoint;
}
const sessions: WorkSession[] = [];
const workIvs: Array<[number, number]> = [];
const agentIvs: Array<[number, number]> = [];
for (const s of built) {
sessions.push({ start: s.padStart, end: s.padEnd, class: s.cls });
(s.cls === 'work' ? workIvs : agentIvs).push([s.padStart, s.padEnd]);
}
sessions.sort((a, b) => a.start - b.start);
return {
workMs: unionDuration(workIvs),
agentOnlyMs: unionDuration(agentIvs),
sessions,
};
}
@@ -0,0 +1,15 @@
export { computeWorkTime } from './compute-work-time';
export { bucketByDay, zonedDayStart } from './bucket-by-day';
export {
DEFAULT_WORK_TIME_CONFIG,
resolveWorkTimeConfig,
} from './work-time.config';
export type { WorkTimeConfig } from './work-time.config';
export type {
TimelineSample,
WorkSession,
WorkTimeResult,
SessionClass,
DayWindow,
PerDay,
} from './work-time.types';
@@ -0,0 +1,102 @@
import {
IDLE_MAX_WAIT_USER,
IDLE_MAX_WAIT_AGENT,
} from '../../../collaboration/constants';
/**
* #395 tunables for the work-time estimate (§10). Defaults are calibrated off
* #374's idle-pulse ceilings: after #374 a continuous editing session leaves a
* history row at least every ~IDLE_MAX_WAIT (10m user / 5m agent), so a gap
* WIDER than that ceiling contains un-pulsed idle time = (partial) inactivity.
* `tGap` therefore sits a little above the user ceiling, `agentTGap` a little
* above the agent ceiling a gap within the threshold is pulse-backed and
* counts as work.
*/
export interface WorkTimeConfig {
/** user inactivity timeout: gap ≤ tGap between samples = continuous work. */
tGap: number;
/** timeout for a pair of consecutive agent samples (tighter than tGap). */
agentTGap: number;
/** pre-roll padding for a multi-sample session (work began before sample 1). */
pIn: number;
/** post-roll padding for a multi-sample session (work continued after last). */
pOut: number;
/** block for a lone single-sample session (pre-roll only, no invented future). */
pSingle: number;
/** drop `git`-source samples (they are not human/agent article work). */
excludeGit: boolean;
/** optional cap on one collapsed agent-burst segment's wall-clock (§9#3). */
burstCapMs?: number;
/** samples whose createdAt round to the same bucket dedup to one (§9#7). */
dedupRoundMs: number;
}
export const DEFAULT_WORK_TIME_CONFIG: WorkTimeConfig = {
// ~15m: IDLE_MAX_WAIT_USER (10m) + headroom. Empirically backcast on a real
// 307-snapshot article (≈24h at 15m matched the owner's estimate; 30/45m
// over-counted). See #395 §10.
tGap: 15 * 60 * 1000,
// ~7m: IDLE_MAX_WAIT_AGENT (5m) + headroom.
agentTGap: 7 * 60 * 1000,
pIn: 5 * 60 * 1000,
pOut: 5 * 60 * 1000,
pSingle: 2 * 60 * 1000,
excludeGit: true,
burstCapMs: undefined,
dedupRoundMs: 1000,
};
// Compile-time cross-check that the defaults really are pulse-anchored — if a
// future edit moves the #374 ceilings, this reminds us to re-calibrate.
void IDLE_MAX_WAIT_USER;
void IDLE_MAX_WAIT_AGENT;
/**
* Fill a partial config with defaults and validate it. Cross-class metric
* disjointness is guaranteed jointly by `computeWorkTime`'s adjacent-pair padding
* clip (it caps the padding of adjacent DIFFERENT-class sessions at the raw-gap
* midpoint) AND the two bounds enforced below (§5):
* - `tGap ≥ pIn + pOut`: a session's own padding never exceeds its inactivity
* window.
* - `2·agentTGap ≥ pIn + pOut`: makes the adjacent-only clip provably COMPLETE.
* A NON-adjacent (i, i+2) cross-class overlap could only arise from two
* same-class sessions separated by a full intervening session of the other
* class; that separation spans at least two inter-session gaps, each strictly
* `> agentTGap`, so it is `> 2·agentTGap`. Requiring `2·agentTGap ≥ pIn + pOut`
* means even the widest padded reach (pIn + pOut) cannot bridge it so the
* only cross-class overlaps possible are between ADJACENT sessions, which the
* clip handles. `workMs`/`agentOnlyMs` are therefore disjoint by construction.
*/
export function resolveWorkTimeConfig(
partial?: Partial<WorkTimeConfig>,
): WorkTimeConfig {
const config = { ...DEFAULT_WORK_TIME_CONFIG, ...(partial ?? {}) };
for (const key of [
'tGap',
'agentTGap',
'pIn',
'pOut',
'pSingle',
'dedupRoundMs',
] as const) {
const value = config[key];
if (!Number.isFinite(value) || value < 0) {
throw new Error(`work-time config: ${key} must be a non-negative number`);
}
}
if (config.burstCapMs != null && config.burstCapMs <= 0) {
throw new Error('work-time config: burstCapMs must be > 0 when set');
}
if (config.tGap < config.pIn + config.pOut) {
throw new Error(
"work-time config: tGap must be ≥ pIn + pOut (a session's padding may not exceed its inactivity window)",
);
}
if (2 * config.agentTGap < config.pIn + config.pOut) {
throw new Error(
'work-time config: 2·agentTGap must be ≥ pIn + pOut (so non-adjacent cross-class padding cannot overlap)',
);
}
return config;
}
@@ -0,0 +1,73 @@
/**
* #395 "time worked on an article" domain types.
*
* The estimate is built by sessionizing a page's `page_history` timeline on
* inactivity gaps (WakaTime-style), NOT by taking the span between the first and
* last edit (which over-counts sleep / lunch / idle days). See the design doc in
* issue #395 §5§6.3 for the normative algorithm.
*/
/**
* A single `page_history` row projected for the work-time computation the
* cheap columns only (no `content`). Produced by
* `PageHistoryRepo.findTimelineByPageId`. `createdAt` is whatever the DB driver
* hands back (Date); the pure core normalizes it to epoch-ms itself so it stays
* deterministic and DB-free.
*/
export interface TimelineSample {
createdAt: Date | string | number;
lastUpdatedById: string | null;
/** 'user' | 'agent' | 'git' | null (legacy autosave = human). */
lastUpdatedSource: string | null;
lastUpdatedAiChatId: string | null;
/** #370 tier: 'manual' | 'agent' | 'idle' | 'boundary' | null (legacy). */
kind: string | null;
}
/** A finished session's class (§5.1). */
export type SessionClass = 'work' | 'agent_only';
/**
* A finished session: absolute wall-clock bounds already padded with P_in/P_out
* (multi-sample) or P_single (single scalar), plus its class. This is enough for
* both the metrics and the per-day punch-card colouring.
*/
export interface WorkSession {
/** epoch-ms, inclusive lower bound (already P-padded). */
start: number;
/** epoch-ms, exclusive upper bound (already P-padded). */
end: number;
class: SessionClass;
}
/** Output of {@link computeWorkTime}. */
export interface WorkTimeResult {
/** union wall-clock of `work` sessions, ms (the headline metric). */
workMs: number;
/** union wall-clock of `agent_only` sessions, ms (secondary). */
agentOnlyMs: number;
sessions: WorkSession[];
}
/** One activity window inside a calendar day (already clipped to the day). */
export interface DayWindow {
/** epoch-ms. */
start: number;
/** epoch-ms. */
end: number;
class: SessionClass;
}
/** One calendar day of the punch-card (§6.3). */
export interface PerDay {
/** epoch-ms of the local-midnight day start in the requested tz. */
day: number;
/** 'YYYY-MM-DD' in the requested tz — stable, tz-independent label. */
dayISO: string;
/** Σ of `work` windows this day, ms. Σ over days == workMs (invariant §6.3). */
activeMs: number;
/** Σ of `agent_only` windows this day, ms (drawn, NOT in activeMs). */
agentMs: number;
/** both classes, clipped to the day, sorted by start (for drawing). */
windows: DayWindow[];
}
@@ -1,33 +1,56 @@
import { Space } from '@docmost/db/types/entity.types';
export class SearchResponseDto {
// #529 A7 — the single per-hit SUPERSET returned by the unified search engine.
// The web-UI reads id/highlight/icon/space/title/…; the MCP agent maps id→pageId
// and reads snippet/score/path. `rank`/`highlight` are null for substring-only
// hits (the web already falls back). Nothing the legacy web response carried is
// dropped.
export class SearchResultDto {
id: string;
title: string;
// Alias of `id` for the MCP layer (it addresses pages by pageId).
pageId: string;
slugId: string;
icon: string;
parentPageId: string;
title: string;
space?: Partial<Space>;
creatorId: string;
rank: number;
highlight: string;
createdAt: Date;
updatedAt: Date;
space: Partial<Space>;
// ts_rank_cd of the FTS branch; null for substring-only hits.
rank: number | null;
// ts_headline marked HTML; null for substring-only hits.
highlight: string | null;
// Plain windowed snippet around the match (empty for titleOnly).
snippet: string;
// Ancestor titles root → direct parent ([] for a root page).
path: string[];
// Per-response ordering proxy (falls back to rank).
score: number;
// Which fields matched: 'title' and/or 'text'.
matchedFields: string[];
// Which parsed positive/required terms this hit matched.
matchedTerms: string[];
}
// Response shape for the opt-in agent-lookup mode (#443, `substring: true`).
// Additive to the FTS response: carries the location (`path`), a windowed
// `snippet` around the first match and a per-response sort `score`. The MCP
// layer maps `id → pageId`; `slugId` is never exposed.
export class SearchLookupResponseDto {
id: string;
slugId: string;
title: string;
parentPageId: string | null;
// Ancestor titles from the space root down to the direct parent; [] for a
// root page.
path: string[];
// ~300–500 chars around the first match (or a leading text window / extended
// ts_headline fallback).
snippet: string;
// 0..1 float, meaningful ONLY for sorting within one response.
score: number;
// The paginated envelope (A5). `total` is the EXACT permission-filtered count of
// pages matching the positive lexical query (fail-closed). `hasMore` is true when
// more results exist WITHIN the fusion window; `truncatedAtCap` signals the match
// set exceeded CANDIDATE_CAP and the tail is unreachable by pagination.
export class SearchResponseDto {
items: SearchResultDto[];
total: number;
hasMore: boolean;
truncatedAtCap: boolean;
offset: number;
query: {
raw: string;
parsed: {
positive: string[];
required: string[];
excluded: string[];
reason?: string;
};
mode: 'or' | 'and';
match: string;
};
}
+22 -4
View File
@@ -1,16 +1,30 @@
import {
IsBoolean,
IsIn,
IsNotEmpty,
IsNumber,
IsOptional,
IsString,
MaxLength,
} from 'class-validator';
export class SearchDTO {
// Defense-in-depth cap on the raw query length. The real stack-depth bound is
// the parser's MAX_PARSED_TERMS term cap (see search-query-parser.ts); this
// just rejects absurd payloads early. 10k chars still comfortably holds any
// legitimate query.
@IsNotEmpty()
@IsString()
@MaxLength(10000)
query: string;
// #529 A3 — match mode. `auto` (default) routes identifier-like terms
// (10.31.41, esp32, WB-MGE-30D86B) to the substring/trigram branch and words
// to full-text; `word`/`prefix`/`substring` are explicit overrides.
@IsOptional()
@IsIn(['auto', 'word', 'prefix', 'substring'])
match?: 'auto' | 'word' | 'prefix' | 'substring';
@IsOptional()
@IsString()
spaceId: string;
@@ -33,15 +47,19 @@ export class SearchDTO {
// --- Opt-in agent-lookup mode (#443). ------------------------------------
// These fields are ADDITIVE and default-off: a web client that sends none of
// them gets byte-identical FTS behaviour and result shape. They are only read
// by the substring/path/snippet code path in SearchService.searchPage.
// them gets byte-identical FTS behaviour and result shape. In the unified #529
// engine, `parentPageId` and `titleOnly` are read by SearchService.searchPage
// (subtree scoping and title-only matching, respectively). `substring` is NOT
// read by the native driver — it is accepted-but-ignored, kept only for
// back-compat with the upstream lookup request shape.
//
// NOTE (standalone stdio vs stock upstream): stock upstream validates this DTO
// with `whitelist: true`, so an older server silently strips these unknown
// fields and the request degrades gracefully to the plain FTS behaviour.
// Enables the hybrid substring branch (title + text_content LIKE) merged with
// the existing FTS branch, plus tiered ranking, path and windowed snippet.
// Accepted-but-ignored by the #529 native driver (kept for upstream lookup
// back-compat). The unified engine ALWAYS runs the hybrid FTS + substring/
// trigram branches with tiered ranking, so this flag no longer toggles anything.
@IsOptional()
@IsBoolean()
substring?: boolean;
@@ -0,0 +1,168 @@
import {
parseSearchQuery,
hasPositiveRecall,
MAX_PARSED_TERMS,
} from './search-query-parser';
describe('parseSearchQuery — tokenization & operators (A2)', () => {
it('splits on whitespace into positive terms (OR recall)', () => {
const p = parseSearchQuery('стамбул роснефть');
expect(p.positive.map((t) => t.text)).toEqual(['стамбул', 'роснефть']);
expect(p.required).toEqual([]);
expect(p.excluded).toEqual([]);
expect(p.mode).toBe('or');
});
it('treats +term as required and -term as excluded', () => {
const p = parseSearchQuery('+кофейня -архив');
expect(p.required.map((t) => t.text)).toEqual(['кофейня']);
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive).toEqual([]);
});
it('keeps a leading-operator-free hyphen/dot/colon token as ONE literal term', () => {
// WB-MGE-30D86B, 10.0.12.5, a:b — internal -,.,: are literal, one term each.
expect(parseSearchQuery('WB-MGE-30D86B').positive[0].text).toBe(
'WB-MGE-30D86B',
);
expect(parseSearchQuery('10.0.12.5').positive[0].text).toBe('10.0.12.5');
expect(parseSearchQuery('host:8080').positive[0].text).toBe('host:8080');
});
it('only a LEADING +/- is an operator; -архив excludes архив', () => {
const p = parseSearchQuery('-архив');
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive).toEqual([]);
});
it('drops a bare "-" / "+" and an all-operator remainder', () => {
const p = parseSearchQuery('- + foo -- ++');
expect(p.positive.map((t) => t.text)).toEqual(['foo']);
expect(p.required).toEqual([]);
expect(p.excluded).toEqual([]);
});
it('parses a quoted phrase as one adjacency term', () => {
const p = parseSearchQuery('"воздушный шар" кофе');
expect(p.positive[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
expect(p.positive[1].text).toBe('кофе');
});
it('applies +/- to a phrase', () => {
const req = parseSearchQuery('+"воздушный шар"');
expect(req.required[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
const exc = parseSearchQuery('-"воздушный шар"');
expect(exc.excluded[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
});
it('drops an unbalanced quote token', () => {
const p = parseSearchQuery('kafka "unclosed here');
expect(p.positive.map((t) => t.text)).toEqual(['kafka']);
});
it('strips tsquery metacharacters from a bare FTS term (no 500)', () => {
const p = parseSearchQuery('foo|bar');
// `|` is not an identifier signal → FTS branch; the metachar is stripped so
// the term becomes the two words that survive.
expect(p.positive[0].branch === 'fts' || p.positive[0].branch === 'ftsPrefix').toBe(true);
expect(p.positive[0].text).toBe('foo bar');
});
});
describe('parseSearchQuery — match=auto classification (A3)', () => {
it('routes identifier-like terms to the substring branch', () => {
expect(parseSearchQuery('10.31.41').positive[0].branch).toBe('substring');
expect(parseSearchQuery('esp32').positive[0].branch).toBe('substring');
expect(parseSearchQuery('WB-MGE-30D86B').positive[0].branch).toBe('substring');
});
it('routes purely-alphabetic words to the FTS (prefix) branch', () => {
expect(parseSearchQuery('печат').positive[0].branch).toBe('ftsPrefix');
expect(parseSearchQuery('ресторан').positive[0].branch).toBe('ftsPrefix');
});
it('explicit match overrides: word / prefix / substring', () => {
expect(parseSearchQuery('печат', { match: 'word' }).positive[0].branch).toBe('fts');
expect(parseSearchQuery('печат', { match: 'prefix' }).positive[0].branch).toBe(
'ftsPrefix',
);
expect(parseSearchQuery('печат', { match: 'substring' }).positive[0].branch).toBe(
'substring',
);
});
});
describe('parseSearchQuery — reasons & recall', () => {
it('only-negation yields reason only-negation and no positive recall', () => {
const p = parseSearchQuery('-архив');
expect(p.reason).toBe('only-negation');
expect(hasPositiveRecall(p)).toBe(false);
});
it('empty / whitespace / garbage yields reason empty', () => {
expect(parseSearchQuery('').reason).toBe('empty');
expect(parseSearchQuery(' ').reason).toBe('empty');
// A bare operator drops to nothing → empty (no exclusion survived).
expect(parseSearchQuery('+ -').reason).toBe('empty');
});
it('a required term alone IS positive recall (no reason)', () => {
const p = parseSearchQuery('+кофейня');
expect(p.reason).toBeUndefined();
expect(hasPositiveRecall(p)).toBe(true);
});
it('mode flag flows through', () => {
expect(parseSearchQuery('a b', { mode: 'and' }).mode).toBe('and');
expect(parseSearchQuery('a b').mode).toBe('or');
});
});
describe('parseSearchQuery — term cap (stack-depth guard)', () => {
it('caps the total parsed terms at MAX_PARSED_TERMS without throwing', () => {
// A pasted text block: far more words than the cap. The parser must bound the
// SQL tsquery nesting depth (else Postgres blows its stack → HTTP 500).
const words = Array.from({ length: 5000 }, (_, i) => `w${i}`);
let p!: ReturnType<typeof parseSearchQuery>;
expect(() => {
p = parseSearchQuery(words.join(' '));
}).not.toThrow();
const total = p.positive.length + p.required.length + p.excluded.length;
expect(total).toBe(MAX_PARSED_TERMS);
// Stable order: the FIRST cap terms are kept.
expect(p.positive.slice(0, 3).map((t) => t.text)).toEqual(['w0', 'w1', 'w2']);
expect(p.positive[MAX_PARSED_TERMS - 1].text).toBe(`w${MAX_PARSED_TERMS - 1}`);
});
it('counts positive + required + excluded together toward the cap', () => {
// Interleave operators so overflow can fall on any bucket. Total must still
// never exceed the cap, and the first cap terms (in stable order) win.
const tokens: string[] = [];
for (let i = 0; i < 200; i++) {
const op = i % 3 === 0 ? '+' : i % 3 === 1 ? '-' : '';
tokens.push(`${op}t${i}`);
}
const p = parseSearchQuery(tokens.join(' '));
const total = p.positive.length + p.required.length + p.excluded.length;
expect(total).toBe(MAX_PARSED_TERMS);
// t0 (+, required) and t1 (-, excluded) and t2 (bare, positive) are all within
// the first cap tokens → each bucket got its leading terms.
expect(p.required[0].text).toBe('t0');
expect(p.excluded[0].text).toBe('t1');
expect(p.positive[0].text).toBe('t2');
});
it('leaves a normal (<= cap) query completely unchanged', () => {
const p = parseSearchQuery('+кофейня -архив "воздушный шар" ресторан 10.31.41');
expect(p.required.map((t) => t.text)).toEqual(['кофейня']);
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive.map((t) => t.text)).toEqual([
'воздушный шар',
'ресторан',
'10.31.41',
]);
// Phrase / substring branch handling survives under the cap.
expect(p.positive[0].branch).toBe('phrase');
expect(p.positive[2].branch).toBe('substring');
});
});
@@ -0,0 +1,242 @@
// #529 Phase A — server-side query parser (the SINGLE source of query semantics).
//
// Clients (web-UI, MCP agent, public share) send a RAW query string plus flags
// (`match`, `mode`); ALL query interpretation happens HERE, so every consumer
// gets identical operator/phrase/morphology behaviour. The parser is PURE (no
// SQL, no DB) so it is exhaustively unit-testable; SearchService turns the parsed
// AST into a parameterized tsquery/predicate tree (never string-concatenated SQL).
//
// Grammar (A2):
// - Whitespace splits tokens, but a double-quoted run is ONE token ("a b" is a
// phrase). An unbalanced quote is dropped.
// - A token is an OPERATOR token only when it STARTS with `+` or `-`, the
// remainder is non-empty, and the remainder is not itself only operators.
// `+`/`-` inside a token (`WB-MGE-30D86B`, `10.0.12.5`, `a:b`) is a LITERAL —
// the token is a single term. A bare `-`/`+` is dropped.
// - `"phrase"` → phrase term; `+"phrase"`/`-"phrase"` apply the operator to it.
// - Positive terms (bare + phrase, no operator) form the OR recall set.
// `+term` is a REQUIRED predicate, `-term` an EXCLUDED predicate (A2): both
// are applied in SQL WHERE against the whole candidate set, not folded into
// the positive tsquery.
// - Only-negation (no positive term) short-circuits to an empty result with
// reason `only-negation` (never runs a costly NOT-scan).
export type SearchMatchMode = 'auto' | 'word' | 'prefix' | 'substring';
export type SearchBooleanMode = 'or' | 'and';
// Hard cap on the total number of parsed terms (positive + required + excluded).
// SearchService folds each FTS term into a LEFT-NESTED SQL tsquery expression
// `(((t1)||(t2))||(t3))…`, embedded several times per query — so nesting depth
// grows with the term count. A pasted text block (thousands of words) would nest
// deep enough to blow Postgres' `stack depth limit` → an ERROR → HTTP 500 for the
// caller. Capping in the parser bounds that depth for EVERY consumer (web / MCP /
// share), since they all route through here. 64 comfortably covers any real query
// while keeping the SQL nesting shallow. Overflow terms (beyond the first 64, in
// stable order) are dropped rather than throwing.
export const MAX_PARSED_TERMS = 64;
// How a single term is matched against the index.
// - 'fts' : full-text lexeme, exact (no trailing prefix).
// - 'ftsPrefix' : full-text lexeme with a `:*` prefix match.
// - 'phrase' : an adjacency phrase (phraseto_tsquery).
// - 'substring' : a literal LOWER(f_unaccent(col)) LIKE '%needle%' branch
// (identifiers the tokenizer mangles: IPs, hostnames, IDs).
export type SearchTermBranch = 'fts' | 'ftsPrefix' | 'phrase' | 'substring';
export interface ParsedTerm {
// The user-visible term text, operator stripped, quotes removed. This is what
// `matchedTerms` echoes back per hit.
text: string;
branch: SearchTermBranch;
}
export interface ParsedQuery {
raw: string;
// OR-recall set (bare + phrase terms with no operator).
positive: ParsedTerm[];
// AND predicates (`+term`) — the candidate MUST match each of these.
required: ParsedTerm[];
// NOT predicates (`-term`) — the candidate must match NONE of these.
excluded: ParsedTerm[];
mode: SearchBooleanMode;
// Set only when the query yields no positive recall: 'empty' (nothing usable)
// or 'only-negation' (there were exclusions but no positive term).
reason?: 'empty' | 'only-negation';
}
interface RawToken {
op: '' | '+' | '-';
kind: 'word' | 'phrase';
text: string;
}
// tsquery metacharacters that must never reach to_tsquery from a bare term — they
// are what turned adversarial input into a 500 before (#139). Stripped for the FTS
// branch; the substring branch keeps them (they are literal there).
const TSQUERY_META = /[:&|!()*<>\\]+/g;
// A term is "identifier-like" when it carries a digit or one of . _ : / - AND is
// not purely alphabetic (letters only). Such tokens (10.31.41, esp32,
// WB-MGE-30D86B) are mangled by the FTS tokenizer, so `match: auto` routes them
// to the substring branch. A purely-alphabetic word (печат, ресторан) stays FTS.
const IDENTIFIER_SIGNAL = /[0-9._:/\\-]/;
const PURELY_ALPHA = /^\p{L}+$/u;
function isIdentifierLike(text: string): boolean {
return IDENTIFIER_SIGNAL.test(text) && !PURELY_ALPHA.test(text);
}
// Clean a bare term for the FTS branch: NFC-normalize, drop tsquery metacharacters
// and collapse whitespace. Returns '' when nothing usable remains.
export function cleanFtsLexeme(raw: string): string {
return (raw ?? '')
.normalize('NFC')
.replace(TSQUERY_META, ' ')
.replace(/\s+/g, ' ')
.trim();
}
// Split a raw query into tokens, honouring double quotes and a single leading
// +/- operator. Unbalanced quotes and bare operators are dropped.
function tokenize(raw: string): RawToken[] {
const tokens: RawToken[] = [];
const s = raw ?? '';
let i = 0;
const n = s.length;
const isSpace = (c: string) => /\s/.test(c);
while (i < n) {
// Skip leading whitespace.
while (i < n && isSpace(s[i])) i++;
if (i >= n) break;
let op: '' | '+' | '-' = '';
// A single leading +/- is a tentative operator. Only ONE leading operator is
// consumed; a second (`--x`) leaves `-x` as the remainder (a literal dash).
if (s[i] === '+' || s[i] === '-') {
op = s[i] as '+' | '-';
i++;
}
if (i < n && s[i] === '"') {
// Quoted phrase: read until the closing quote.
const close = s.indexOf('"', i + 1);
if (close === -1) {
// Unbalanced quote → drop this token and everything the open quote would
// have consumed (the rest of the string).
break;
}
const phrase = s.slice(i + 1, close);
i = close + 1;
if (phrase.trim().length > 0) {
tokens.push({ op, kind: 'phrase', text: phrase.trim() });
}
continue;
}
// Bare word: read until the next whitespace.
let j = i;
while (j < n && !isSpace(s[j])) j++;
const word = s.slice(i, j);
i = j;
// A bare operator (`-`/`+` with no remainder) or an all-operator remainder is
// dropped.
if (word.length === 0) continue;
if (op && /^[+-]+$/.test(word)) continue;
tokens.push({ op, kind: 'word', text: word });
}
return tokens;
}
// Resolve the match branch for a single term given the global match mode.
function branchForTerm(
text: string,
kind: 'word' | 'phrase',
mode: SearchMatchMode,
): SearchTermBranch {
if (kind === 'phrase') return 'phrase';
switch (mode) {
case 'word':
return 'fts';
case 'prefix':
return 'ftsPrefix';
case 'substring':
return 'substring';
case 'auto':
default:
// Identifiers the tokenizer mangles go to substring; words get a prefix
// FTS match (so `печат` still finds `печатать`, but `печат` no longer drags
// in `впечатления` because the russian stemmer anchors the stem).
return isIdentifierLike(text) ? 'substring' : 'ftsPrefix';
}
}
/**
* Parse a raw user query + flags into a structured, SQL-agnostic AST.
* Pure and total: never throws, always returns a ParsedQuery.
*/
export function parseSearchQuery(
raw: string,
opts: { match?: SearchMatchMode; mode?: SearchBooleanMode } = {},
): ParsedQuery {
const match: SearchMatchMode = opts.match ?? 'auto';
const mode: SearchBooleanMode = opts.mode ?? 'or';
const positive: ParsedTerm[] = [];
const required: ParsedTerm[] = [];
const excluded: ParsedTerm[] = [];
for (const tok of tokenize(raw)) {
// For an FTS branch, the token must survive metacharacter cleaning; for the
// substring/phrase branch the literal text is used. A term that cleans to
// nothing AND is not usable as a substring is dropped.
const branch = branchForTerm(tok.text, tok.kind, match);
let usableText: string;
if (branch === 'fts' || branch === 'ftsPrefix') {
usableText = cleanFtsLexeme(tok.text);
} else {
// phrase / substring keep the literal (trimmed) text.
usableText = tok.text.trim();
}
if (!usableText) continue;
// Stack-depth guard: stop after MAX_PARSED_TERMS surviving terms (positive +
// required + excluded, combined) so the SQL tsquery nesting stays shallow.
// The first 64 terms are kept in stable order; the rest are dropped.
if (positive.length + required.length + excluded.length >= MAX_PARSED_TERMS) {
break;
}
const term: ParsedTerm = { text: usableText, branch };
if (tok.op === '+') required.push(term);
else if (tok.op === '-') excluded.push(term);
else positive.push(term);
}
const parsed: ParsedQuery = { raw: raw ?? '', positive, required, excluded, mode };
if (positive.length === 0) {
// Required terms with no positive recall still form a valid positive set (the
// required predicates ARE the recall). Only when there is neither a positive
// nor a required term is the query empty / only-negation.
if (required.length === 0) {
parsed.reason = excluded.length > 0 ? 'only-negation' : 'empty';
}
}
return parsed;
}
/**
* Does this parsed query have any positive recall to run? False means we must
* short-circuit to an empty result (with `reason`), never a costly NOT-only scan.
*/
export function hasPositiveRecall(parsed: ParsedQuery): boolean {
return parsed.positive.length > 0 || parsed.required.length > 0;
}
@@ -1,19 +1,14 @@
import {
computeLookupScore,
escapeLikePattern,
SearchLookupTier,
} from './search.service';
import { escapeLikePattern } from './search.service';
/**
* Pure-function coverage for the #443 agent-lookup helpers:
* - escapeLikePattern: LIKE-metacharacter escaping so `%`/`_`/`\` are literals
* (the acceptance-table requirement that a query of `%` or `_` does NOT match
* everything);
* - computeLookupScore: the tiered 0..1 ranking score, where a stronger tier
* always outranks a weaker one regardless of the in-tier secondary signal.
* Pure-function coverage for `escapeLikePattern` LIKE-metacharacter escaping so
* `%`/`_`/`\` are matched literally (the acceptance requirement that a query of
* `%` or `_` does NOT match everything, #529 acceptance #10). The substring
* branch's DB behaviour is covered by the integration spec.
*
* The DB-touching branch (substring UNION FTS, path CTE, snippet window) is
* covered by the integration spec against the real schema.
* NOTE (#529): the old tiered `computeLookupScore` was replaced by RRF rank
* fusion in the unified engine, so its unit coverage moved to the integration
* ordering tests; only the escaping helper remains a pure unit here.
*/
describe('escapeLikePattern', () => {
it('escapes the LIKE metacharacters % _ and \\', () => {
@@ -43,53 +38,3 @@ describe('escapeLikePattern', () => {
expect(escapeLikePattern(null as any)).toBe('');
});
});
describe('computeLookupScore', () => {
it('keeps every score within (0, 1]', () => {
for (const tier of [
SearchLookupTier.TITLE_EXACT,
SearchLookupTier.TITLE_SUBSTRING,
SearchLookupTier.TEXT,
]) {
for (const secondary of [0, 0.001, 1, 100, 1e6]) {
const s = computeLookupScore({ tier, secondary });
expect(s).toBeGreaterThan(0);
expect(s).toBeLessThanOrEqual(1);
}
}
});
it('a stronger tier ALWAYS outranks a weaker tier, whatever the secondary', () => {
// Weak tier with a huge secondary must still lose to a strong tier with a
// tiny secondary — tiers dominate.
const strongLowSecondary = computeLookupScore({
tier: SearchLookupTier.TITLE_EXACT,
secondary: 0,
});
const weakHighSecondary = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 1e9,
});
expect(strongLowSecondary).toBeGreaterThan(weakHighSecondary);
});
it('within a tier a larger secondary sorts higher', () => {
const lo = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 0.1,
});
const hi = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 5,
});
expect(hi).toBeGreaterThan(lo);
});
it('treats a negative/absent secondary as 0', () => {
const zero = computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: 0 });
expect(computeLookupScore({ tier: SearchLookupTier.TEXT })).toBe(zero);
expect(
computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: -5 }),
).toBe(zero);
});
});
@@ -1,74 +1,45 @@
import { SearchService } from './search.service';
/**
* Coverage for SearchService.searchPage query-mode selection (search.service.ts
* @25). searchPage chooses HOW the result set is scoped by explicit space, by
* the authenticated user's member spaces, or by a share and must return an
* empty set (without leaking data) for every disallowed combination.
* Unit coverage for SearchService.searchPage SCOPE-SECURITY early returns the
* branches that must yield an empty result WITHOUT ever touching the DB, so they
* can leak nothing. The happy-path scope SQL (explicit space / member spaces /
* share id set) is covered against the real schema in the integration spec.
*
* The kysely query builder is mocked with the same chainable pattern as the
* existing search.service.spec.ts: every builder method returns the same builder
* and `.execute()` resolves the supplied rows. Each `.where(...)` call is
* recorded so we can assert exactly which scope clause was applied that is the
* mutation-resistant signal that distinguishes one query mode from another.
*
* These specs catch cross-space / cross-workspace search leakage and
* share-scope bypass (data exposure).
* Every case here returns BEFORE the raw-SQL candidate query runs, so a bare `db`
* stub (never called) is enough a call to it would itself be a failure signal.
*/
describe('SearchService.searchPage — query-mode selection', () => {
// Build a chainable selectFrom('pages') builder that records its calls. The
// builder is returned from `db.selectFrom` and is the single object every
// chained call mutates/returns, mirroring the existing spec's pattern.
function makeBuilder(rows: Array<{ id: string; highlight?: string }>) {
const builder: any = {};
builder.select = jest.fn(() => builder);
builder.where = jest.fn(() => builder);
builder.$if = jest.fn(() => builder);
builder.orderBy = jest.fn(() => builder);
builder.limit = jest.fn(() => builder);
builder.offset = jest.fn(() => builder);
builder.execute = jest.fn(async () => rows);
return builder;
}
describe('SearchService.searchPage — scope-security early returns', () => {
function makeService(opts?: {
rows?: Array<{ id: string; highlight?: string }>;
share?: any;
isRestricted?: boolean;
descendants?: Array<{ id: string }>;
memberSpaceIds?: string[];
}) {
const builder = makeBuilder(opts?.rows ?? []);
const db: any = {
selectFrom: jest.fn(() => builder),
};
// `getUserSpaceIdsQuery` returns a sub-query object that searchPage passes
// straight into `.where('spaceId', 'in', <subquery>)`. A sentinel is enough
// to assert the user-scoped branch was taken.
const userSpaceIdsQuery = { __userSpaceIdsQuery: true };
// A db that THROWS if touched — these branches must not reach SQL.
const db: any = new Proxy(
{},
{
get() {
throw new Error('db must not be touched on an empty-scope branch');
},
},
);
const pageRepo = {
// `.select((eb) => this.pageRepo.withSpace(eb))` — value ignored by stub.
withSpace: jest.fn(() => ({ __withSpace: true })),
getPageAndDescendantsExcludingRestricted: jest
.fn()
.mockResolvedValue(opts?.descendants ?? []),
getPageAndDescendantsExcludingRestricted: jest.fn(),
getPageAndDescendants: jest.fn(),
};
const shareRepo = {
findById: jest.fn().mockResolvedValue(opts?.share ?? null),
};
const spaceMemberRepo = {
getUserSpaceIdsQuery: jest.fn(() => userSpaceIdsQuery),
getUserSpaceIds: jest.fn().mockResolvedValue(opts?.memberSpaceIds ?? []),
};
const pagePermissionRepo = {
hasRestrictedAncestor: jest
.fn()
.mockResolvedValue(opts?.isRestricted ?? false),
// Let everything through page-level permission filtering by default.
filterAccessiblePageIds: jest
.fn()
.mockImplementation(async ({ pageIds }: { pageIds: string[] }) => pageIds),
filterAccessiblePageIds: jest.fn(),
};
const service = new SearchService(
@@ -78,145 +49,81 @@ describe('SearchService.searchPage — query-mode selection', () => {
spaceMemberRepo as any,
pagePermissionRepo as any,
);
return {
service,
db,
builder,
pageRepo,
shareRepo,
spaceMemberRepo,
pagePermissionRepo,
userSpaceIdsQuery,
};
return { service, pageRepo, shareRepo, spaceMemberRepo, pagePermissionRepo };
}
const whereCallFor = (builder: any, column: any) =>
builder.where.mock.calls.find((c: any[]) => c[0] === column);
it('returns {items:[]} for a blank query WITHOUT touching the DB', async () => {
const { service, db } = makeService();
it('returns total:0 for a blank query WITHOUT touching the DB or any repo', async () => {
const { service, shareRepo, spaceMemberRepo } = makeService();
const result = await service.searchPage(
{ query: '' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(result).toEqual({ items: [] });
// Blank query is rejected before any query builder is constructed.
expect(db.selectFrom).not.toHaveBeenCalled();
expect(result.items).toEqual([]);
expect(result.total).toBe(0);
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(spaceMemberRepo.getUserSpaceIds).not.toHaveBeenCalled();
});
it('scopes to the explicit spaceId branch', async () => {
const { service, builder, db, spaceMemberRepo, shareRepo } = makeService({
rows: [{ id: 'p-1' }],
});
it('only-negation short-circuits with reason "only-negation", never scanning', async () => {
const { service, spaceMemberRepo } = makeService();
const result = await service.searchPage(
{ query: 'plan', spaceId: 'space-42' } as any,
{ query: '-архив' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(db.selectFrom).toHaveBeenCalledWith('pages');
// The explicit-space branch adds exactly `.where('spaceId', '=', 'space-42')`.
expect(whereCallFor(builder, 'spaceId')).toEqual([
'spaceId',
'=',
'space-42',
]);
// It must NOT fall through to the user-member-spaces or share branch.
expect(spaceMemberRepo.getUserSpaceIdsQuery).not.toHaveBeenCalled();
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(result.items.map((i: any) => i.id)).toEqual(['p-1']);
expect(result.total).toBe(0);
expect(result.query.parsed.reason).toBe('only-negation');
// Never resolves scope (returns before) — no expensive NOT-only scan.
expect(spaceMemberRepo.getUserSpaceIds).not.toHaveBeenCalled();
});
it('scopes an authenticated user WITHOUT spaceId to their member spaces', async () => {
const { service, builder, spaceMemberRepo, userSpaceIdsQuery, shareRepo } =
makeService({ rows: [{ id: 'p-9' }] });
await service.searchPage(
{ query: 'plan' } as any,
{ userId: 'user-7', workspaceId: 'ws-1' },
);
// The user-scoped branch resolves the member-spaces sub-query for that user
// and restricts both spaceId (to that sub-query) and workspaceId.
expect(spaceMemberRepo.getUserSpaceIdsQuery).toHaveBeenCalledWith('user-7');
expect(whereCallFor(builder, 'spaceId')).toEqual([
'spaceId',
'in',
userSpaceIdsQuery,
]);
expect(whereCallFor(builder, 'workspaceId')).toEqual([
'workspaceId',
'=',
'ws-1',
]);
// Authenticated user path must not consult shares.
expect(shareRepo.findById).not.toHaveBeenCalled();
});
it('returns {items:[]} when the share belongs to a DIFFERENT workspace', async () => {
const { service, builder, shareRepo, pagePermissionRepo } = makeService({
share: {
id: 'share-1',
pageId: 'page-1',
workspaceId: 'OTHER-ws',
includeSubPages: false,
},
it('returns empty when the share belongs to a DIFFERENT workspace (no leak)', async () => {
const { service, shareRepo, pagePermissionRepo } = makeService({
share: { id: 's1', pageId: 'p1', workspaceId: 'OTHER', includeSubPages: false },
});
const result = await service.searchPage(
{ query: 'plan', shareId: 'share-1' } as any,
{ query: 'plan', shareId: 's1' } as any,
{ workspaceId: 'ws-1' },
);
expect(shareRepo.findById).toHaveBeenCalledWith('share-1');
expect(result).toEqual({ items: [] });
// Workspace mismatch short-circuits before any restricted-ancestor / id
// scoping or DB execution: no leak across workspaces.
expect(shareRepo.findById).toHaveBeenCalledWith('s1');
expect(result.items).toEqual([]);
// Workspace mismatch short-circuits before restricted-ancestor / enumeration.
expect(pagePermissionRepo.hasRestrictedAncestor).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('returns {items:[]} when the shared page has a restricted ancestor', async () => {
const { service, builder, pagePermissionRepo, pageRepo } = makeService({
share: {
id: 'share-1',
pageId: 'page-1',
workspaceId: 'ws-1',
includeSubPages: true,
},
it('returns empty when the shared page has a restricted ancestor', async () => {
const { service, pagePermissionRepo, pageRepo } = makeService({
share: { id: 's1', pageId: 'p1', workspaceId: 'ws-1', includeSubPages: true },
isRestricted: true,
});
const result = await service.searchPage(
{ query: 'plan', shareId: 'share-1' } as any,
{ query: 'plan', shareId: 's1' } as any,
{ workspaceId: 'ws-1' },
);
expect(pagePermissionRepo.hasRestrictedAncestor).toHaveBeenCalledWith(
'page-1',
);
expect(result).toEqual({ items: [] });
// Restricted ancestor must block before page enumeration and DB execution.
expect(pagePermissionRepo.hasRestrictedAncestor).toHaveBeenCalledWith('p1');
expect(result.items).toEqual([]);
expect(
pageRepo.getPageAndDescendantsExcludingRestricted,
).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('returns {items:[]} with no userId, no spaceId and no shareId', async () => {
const { service, builder, shareRepo } = makeService();
it('returns empty with no userId, no spaceId and no shareId', async () => {
const { service, shareRepo } = makeService();
const result = await service.searchPage(
{ query: 'plan' } as any,
{ workspaceId: 'ws-1' },
);
expect(result).toEqual({ items: [] });
// The catch-all else returns empty without scoping/executing or hitting shares.
expect(result.items).toEqual([]);
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('an authenticated user with NO member spaces gets an empty result', async () => {
const { service, spaceMemberRepo } = makeService({ memberSpaceIds: [] });
const result = await service.searchPage(
{ query: 'plan' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(spaceMemberRepo.getUserSpaceIds).toHaveBeenCalledWith('user-1');
expect(result.items).toEqual([]);
expect(result.total).toBe(0);
});
});
@@ -1,4 +1,4 @@
import { SearchService, buildTsQuery } from './search.service';
import { SearchService } from './search.service';
describe('SearchService', () => {
it('should be defined', () => {
@@ -99,59 +99,3 @@ describe('SearchService.searchSuggestions — onlyTemplates filter', () => {
expect(isTemplateWhereCall(pageBuilder)).toBeUndefined();
});
});
// Unit tests for `buildTsQuery` (extracted from search.service.ts). It turns a raw
// user query into a prefix tsquery string fed to `to_tsquery('english', ...)`.
//
// REAL BUG (Gitea #139, item 10): the previous inline `tsquery(query.trim() + '*')`
// let to_tsquery operator characters through, so adversarial inputs could produce a
// fragment that to_tsquery rejects -> 500. The extraction sanitizes the input
// (strip everything but letters/numbers/whitespace) so these inputs degrade to a
// safe, neutral query with NO throw, while normal queries keep working.
describe('buildTsQuery', () => {
it('builds a prefix query for a normal single word', () => {
expect(buildTsQuery('hello')).toBe('hello:*');
});
it('joins multiple words with AND and a trailing prefix match', () => {
expect(buildTsQuery('foo bar')).toBe('foo&bar:*');
});
it('preserves accented and non-Latin words', () => {
expect(buildTsQuery('héllo café')).toBe('héllo&café:*');
expect(buildTsQuery('日本語')).toBe('日本語:*');
});
it('neutralizes to_tsquery operator inputs without throwing', () => {
// Each of these previously risked an invalid to_tsquery -> 500. They must now
// produce a safe (here empty) query and never throw.
for (const input of ['&', '!', '*', '<->', '\\']) {
expect(() => buildTsQuery(input)).not.toThrow();
expect(buildTsQuery(input)).toBe('');
}
});
it('handles stopword-only input safely', () => {
// pg-tsquery still tokenizes stopwords; to_tsquery reduces them to nothing.
// The important contract is: no throw, and a deterministic string.
expect(() => buildTsQuery('the a of')).not.toThrow();
expect(buildTsQuery('the a of')).toBe('the&a&of:*');
});
it('returns empty string for empty / whitespace-only / null-ish input', () => {
expect(buildTsQuery('')).toBe('');
expect(buildTsQuery(' ')).toBe('');
expect(buildTsQuery(undefined as unknown as string)).toBe('');
});
it('handles a very long input without throwing', () => {
const long = 'a'.repeat(10000);
expect(() => buildTsQuery(long)).not.toThrow();
expect(buildTsQuery(long)).toBe(`${long}:*`);
});
it('strips punctuation embedded in otherwise valid words', () => {
expect(buildTsQuery('c++ code')).toBe('c&code:*');
expect(buildTsQuery('a-b-c')).toBe('a&b&c:*');
});
});
File diff suppressed because it is too large Load Diff
@@ -1,5 +1,6 @@
import {
IsBoolean,
IsIn,
IsNotEmpty,
IsOptional,
IsString,
@@ -18,6 +19,13 @@ export class CreateShareDto {
@IsOptional()
@IsBoolean()
searchIndexing: boolean;
// #370 Stage B — publication mode. 'live' (default) serves the current draft;
// 'approved' serves the last manually-saved version (page_history.kind
// ='manual'). Mutually exclusive with includeSubPages (enforced server-side).
@IsOptional()
@IsIn(['live', 'approved'])
publishedMode?: 'live' | 'approved';
}
export class UpdateShareDto extends CreateShareDto {
@@ -39,6 +39,7 @@ function buildService() {
tokenService as any,
{} as any, // transclusionService (unused)
workspaceRepo as any,
{} as any, // pageHistoryRepo (unused on this path)
);
}
@@ -39,6 +39,7 @@ function buildService(over: {
{} as any, // tokenService
{} as any, // transclusionService
{} as any, // workspaceRepo
{} as any, // pageHistoryRepo
);
jest
@@ -62,6 +62,7 @@ function buildService(opts: {
tokenService as any,
{} as any, // transclusionService (unused)
workspaceRepo as any,
{} as any, // pageHistoryRepo (unused on this path)
);
// getSharedPage resolves the share via getShareForPage (a raw db query).
@@ -181,6 +182,7 @@ describe('ShareService.lookupTransclusionForShare htmlEmbed kill-switch (real co
tokenService as any,
transclusionService as any,
workspaceRepo as any,
{} as any, // pageHistoryRepo (unused on this path)
);
// isSharingAllowed and getShareForPage hit the raw db; stub them so the
@@ -23,6 +23,9 @@ function buildService(over: {
resolvedShare?: unknown;
page?: unknown;
restricted?: boolean;
// #370 Stage B — the latest manual page-history row returned for the approved
// swap. `undefined` (the default) means the repo finds no manual version.
manualHistory?: unknown;
} = {}) {
const pageRepo = {
findById: jest.fn(async () =>
@@ -34,6 +37,9 @@ function buildService(over: {
const pagePermissionRepo = {
hasRestrictedAncestor: jest.fn(async () => over.restricted ?? false),
};
const pageHistoryRepo = {
findLatestByPageIdAndKind: jest.fn(async () => over.manualHistory ?? undefined),
};
const service = new ShareService(
{} as any, // shareRepo (unused on this path)
@@ -43,6 +49,7 @@ function buildService(over: {
{} as any, // tokenService (unused)
{} as any, // transclusionService (unused)
{} as any, // workspaceRepo (unused)
pageHistoryRepo as any,
);
jest
@@ -53,7 +60,7 @@ function buildService(over: {
: { id: SHARE, pageId: PAGE, spaceId: 'space-1' }) as any,
);
return { service, pageRepo, pagePermissionRepo };
return { service, pageRepo, pagePermissionRepo, pageHistoryRepo };
}
describe('ShareService.resolveReadableSharePage (the share-access boundary)', () => {
@@ -133,4 +140,90 @@ describe('ShareService.resolveReadableSharePage (the share-access boundary)', ()
includeCreator: true,
});
});
// #370 Stage B — "approved" publication freezes the served content/title to
// the latest manual page-history version, WITHOUT touching page.id or
// page.workspaceId (downstream attachment tokens are minted off those, so
// freezing them would mint tokens for the wrong owner).
describe('approved publication mode (content freeze)', () => {
const LIVE_PAGE = {
id: PAGE,
workspaceId: WS,
deletedAt: null,
title: 'Live draft title',
content: { type: 'doc', live: true },
};
const MANUAL = {
title: 'Saved version title',
content: { type: 'doc', saved: true },
};
it('swaps content + title to the latest manual version but PRESERVES id + workspaceId', async () => {
const { service, pageHistoryRepo } = buildService({
resolvedShare: {
id: SHARE,
pageId: PAGE,
spaceId: 'space-1',
publishedMode: 'approved',
},
page: { ...LIVE_PAGE },
manualHistory: MANUAL,
});
const out = await service.resolveReadableSharePage(SHARE, PAGE, WS);
expect(out).not.toBeNull();
// Content + title come from the saved version.
expect(out!.page.content).toEqual(MANUAL.content);
expect(out!.page.title).toBe(MANUAL.title);
// id + workspaceId stay LIVE (attachment-token ownership must not shift).
expect(out!.page.id).toBe(LIVE_PAGE.id);
expect(out!.page.workspaceId).toBe(LIVE_PAGE.workspaceId);
// Resolved against the LIVE page id and the manual tier, with content.
expect(pageHistoryRepo.findLatestByPageIdAndKind).toHaveBeenCalledWith(
LIVE_PAGE.id,
'manual',
{ includeContent: true },
);
});
it('falls back to live content when no manual version exists yet', async () => {
const { service, pageHistoryRepo } = buildService({
resolvedShare: {
id: SHARE,
pageId: PAGE,
spaceId: 'space-1',
publishedMode: 'approved',
},
page: { ...LIVE_PAGE },
// manualHistory omitted → repo returns undefined
});
const out = await service.resolveReadableSharePage(SHARE, PAGE, WS);
expect(out).not.toBeNull();
expect(out!.page.content).toEqual(LIVE_PAGE.content);
expect(out!.page.title).toBe(LIVE_PAGE.title);
expect(pageHistoryRepo.findLatestByPageIdAndKind).toHaveBeenCalled();
});
it('does NOT consult page-history for a live share', async () => {
const { service, pageHistoryRepo } = buildService({
resolvedShare: {
id: SHARE,
pageId: PAGE,
spaceId: 'space-1',
publishedMode: 'live',
},
page: { ...LIVE_PAGE },
manualHistory: MANUAL,
});
const out = await service.resolveReadableSharePage(SHARE, PAGE, WS);
expect(out).not.toBeNull();
expect(out!.page.content).toEqual(LIVE_PAGE.content);
expect(pageHistoryRepo.findLatestByPageIdAndKind).not.toHaveBeenCalled();
});
});
});
@@ -33,6 +33,7 @@ function buildService() {
tokenService as any,
{} as any, // transclusionService (unused)
workspaceRepo as any,
{} as any, // pageHistoryRepo (unused on this path)
);
}
+115 -4
View File
@@ -19,6 +19,7 @@ import {
} from '../../common/helpers/prosemirror/utils';
import { Node } from '@tiptap/pm/model';
import { ShareRepo } from '@docmost/db/repos/share/share.repo';
import { PageHistoryRepo } from '@docmost/db/repos/page/page-history.repo';
import { PagePermissionRepo } from '@docmost/db/repos/page/page-permission.repo';
import { updateAttachmentAttr } from './share.util';
import { Page } from '@docmost/db/types/entity.types';
@@ -44,6 +45,7 @@ export class ShareService {
private readonly tokenService: TokenService,
private readonly transclusionService: TransclusionService,
private readonly workspaceRepo: WorkspaceRepo,
private readonly pageHistoryRepo: PageHistoryRepo,
) {}
/**
@@ -92,21 +94,39 @@ export class ShareService {
}) {
const { authUserId, workspaceId, page, createShareDto } = opts;
const includeSubPages = createShareDto.includeSubPages ?? false;
const publishedMode = createShareDto.publishedMode ?? 'live';
// #370 Stage B — 'approved' freezes a SINGLE page to its saved version; a
// sub-tree cannot be version-frozen in the MVP, so the two are mutually
// exclusive. Thrown before the try so the specific 400 reaches the client
// instead of being flattened to the generic "Failed to share page".
this.assertPublishedModeCompatible(publishedMode, includeSubPages);
try {
const shares = await this.shareRepo.findByPageId(page.id);
if (shares) {
return shares;
}
return await this.shareRepo.insertShare({
const share = await this.shareRepo.insertShare({
key: nanoIdGen().toLowerCase(),
pageId: page.id,
includeSubPages: createShareDto.includeSubPages ?? false,
includeSubPages,
searchIndexing: createShareDto.searchIndexing ?? false,
publishedMode,
creatorId: authUserId,
spaceId: page.spaceId,
workspaceId,
});
// Guarantee an approved share always has a saved version to serve: mint
// the first manual baseline from the page's current content on enable.
if (publishedMode === 'approved') {
await this.ensureApprovedBaseline(page.id);
}
return share;
} catch (err) {
this.logger.error(err);
throw new BadRequestException('Failed to share page');
@@ -114,20 +134,84 @@ export class ShareService {
}
async updateShare(shareId: string, updateShareDto: UpdateShareDto) {
// Resolve the current share so the XOR gate can be evaluated against the
// EFFECTIVE post-update state (either field may be absent from the DTO).
const current = await this.shareRepo.findById(shareId);
if (!current) {
throw new NotFoundException('Share not found');
}
const effectiveIncludeSubPages =
updateShareDto.includeSubPages ?? current.includeSubPages ?? false;
const effectivePublishedMode = (updateShareDto.publishedMode ??
current.publishedMode ??
'live') as 'live' | 'approved';
// #370 Stage B — enforce the approved/includeSubPages XOR on the merged
// state. Thrown before the try so the specific 400 reaches the client.
this.assertPublishedModeCompatible(
effectivePublishedMode,
effectiveIncludeSubPages,
);
try {
return this.shareRepo.updateShare(
const updated = await this.shareRepo.updateShare(
{
includeSubPages: updateShareDto.includeSubPages,
searchIndexing: updateShareDto.searchIndexing,
publishedMode: updateShareDto.publishedMode,
},
shareId,
);
// On (or while) enabling approved mode, ensure a manual baseline exists so
// public readers always have a saved version to serve. Idempotent: a no-op
// when the page already has a manual history row.
if (effectivePublishedMode === 'approved') {
await this.ensureApprovedBaseline(current.pageId);
}
return updated;
} catch (err) {
this.logger.error(err);
throw new BadRequestException('Failed to update share');
}
}
/**
* #370 Stage B 'approved' publication is mutually exclusive with
* includeSubPages: a whole sub-tree cannot be version-frozen in the MVP.
*/
private assertPublishedModeCompatible(
publishedMode: 'live' | 'approved',
includeSubPages: boolean,
): void {
if (publishedMode === 'approved' && includeSubPages) {
throw new BadRequestException(
'Approved publication cannot be combined with including sub-pages',
);
}
}
/**
* #370 Stage B guarantee an approved share has a saved version to serve.
* When the page has NO manual history row yet, snapshot its CURRENT content as
* the first manual version. Idempotent: returns early when a manual version
* already exists, so it is safe to call on every approved update.
*/
private async ensureApprovedBaseline(pageId: string): Promise<void> {
const existing = await this.pageHistoryRepo.findLatestByPageIdAndKind(
pageId,
'manual',
);
if (existing) return;
const page = await this.pageRepo.findById(pageId, { includeContent: true });
if (!page) return;
await this.pageHistoryRepo.saveHistory(page, { kind: 'manual' });
}
/**
* THE share access boundary in ONE place.
*
@@ -173,7 +257,7 @@ export class ShareService {
// passes no shareId (it resolved the share from the page itself).
if (shareId != null && share.id !== shareId) return null;
const page = await this.pageRepo.findById(pageId, {
let page = await this.pageRepo.findById(pageId, {
includeContent: true,
includeCreator: opts?.includeCreator ?? false,
});
@@ -185,6 +269,30 @@ export class ShareService {
return null;
}
// #370 Stage B — "approved" publication freezes what public readers see to
// the LAST manually-saved version (page_history.kind='manual'), not the live
// draft. This is the SINGLE canonical resolver every public read funnels
// through (the shared-page view, the SEO controller, AND the share
// assistant's page-read tool), so the swap here freezes ALL of them
// consistently — a public visitor and the assistant see the same bytes.
//
// Swap ONLY content + title. page.id and page.workspaceId stay LIVE on
// purpose: downstream updatePublicAttachments mints per-attachment tokens
// off page.id/workspaceId, so freezing those would mint tokens for the
// wrong owner. The SEO controller reads resolved.page.title, so the frozen
// title flows through automatically. If no manual version exists yet (rare —
// enabling an approved share auto-creates a baseline), fall back to live.
if (share.publishedMode === 'approved') {
const hist = await this.pageHistoryRepo.findLatestByPageIdAndKind(
page.id,
'manual',
{ includeContent: true },
);
if (hist) {
page = { ...page, content: hist.content, title: hist.title };
}
}
return { share, page };
}
@@ -302,6 +410,7 @@ export class ShareService {
'shares.key as shareKey',
'shares.includeSubPages',
'shares.searchIndexing',
'shares.publishedMode',
'shares.creatorId',
'shares.spaceId',
'shares.workspaceId',
@@ -326,6 +435,7 @@ export class ShareService {
's.key as shareKey',
's.includeSubPages',
's.searchIndexing',
's.publishedMode',
's.creatorId',
's.spaceId',
's.workspaceId',
@@ -355,6 +465,7 @@ export class ShareService {
key: share.shareKey,
includeSubPages: share.includeSubPages,
searchIndexing: share.searchIndexing,
publishedMode: share.publishedMode,
pageId: share.id,
creatorId: share.creatorId,
spaceId: share.spaceId,
@@ -32,6 +32,7 @@ import { TemplateRepo } from '@docmost/db/repos/template/template.repo';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo';
import { AiProviderCredentialsRepo } from '@docmost/db/repos/ai-chat/ai-provider-credentials.repo';
import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo';
@@ -125,6 +126,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
AiChatRepo,
AiChatMessageRepo,
AiChatRunRepo,
AiChatRunStepRepo,
AiChatPageSnapshotRepo,
AiProviderCredentialsRepo,
AiMcpServerRepo,
@@ -161,6 +163,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
AiChatRepo,
AiChatMessageRepo,
AiChatRunRepo,
AiChatRunStepRepo,
AiChatPageSnapshotRepo,
AiProviderCredentialsRepo,
AiMcpServerRepo,
@@ -0,0 +1,242 @@
import { type Kysely, sql } from 'kysely';
/**
* #529 Phase A1 the `ru_en` text-search configuration + the config swap.
*
* WHY: the search stack was pinned to the `english` FTS config, which stems only
* Latin words. On a Russian-language wiki that is a morphology black hole:
* «ресторанов москвы» never matched a page titled «ресторан в москве». `ru_en`
* layers the russian_stem over the Cyrillic token classes and english_stem over
* the ascii ones, so BOTH languages get proper morphology from one config.
*
* CREATE TEXT SEARCH CONFIGURATION ru_en (COPY = simple);
* ALTER ... asciiword/asciihword/hword_asciipart WITH english_stem;
* ALTER ... word/hword/hword_part WITH russian_stem;
*
* `to_tsvector('ru_en', …)` with the LITERAL config name is IMMUTABLE, so it is
* valid inside a trigger, a generated column and an index expression.
*
* THE INVARIANT (acceptance #13): the config of the STORED column and the config
* of the QUERY must change together. This migration flips BOTH stored sides
* (pages.tsv via its trigger + a reindex; page_embeddings.fts, the RAG lexical
* leg, via its generated expression); the matching QUERY-side flips
* (search.service.ts and page-embedding.repo.ts `hybridSearch`) ship in the SAME
* commit. Trigram indexes are LOWER(f_unaccent(...)) and do NOT depend on the FTS
* config, so they are untouched.
*
* REINDEX / LOCK MODEL (deploy-critical). Kysely runs EACH migration in its OWN
* transaction (see sibling 20260706T120000 "Kysely runs each migration in a
* transaction"; migrate.ts / migration.service.ts set no `disableTransactions`,
* and PostgresJSDialect has transactional DDL). A single migration file therefore
* cannot commit between batches, so the issue's "procedural batch job OUTSIDE the
* transaction + dual-config read window + migration_complete gate" is not
* expressible in-migration here. That machinery bridges a reindex spread over
* MANY committed batches, during which some rows are still `english` while others
* are already `ru_en`. Our pages.tsv reindex is a SINGLE `UPDATE pages SET tsv`,
* ATOMIC within THIS migration's own transaction: at COMMIT every row is `ru_en`
* at once, so no morphology-desync window exists and no dual-config read path is
* required the query config flips to `ru_en` in the very same release. This is
* the deliberate, correct adaptation to this framework (see the PR notes).
*
* - pages.tsv: swapping the trigger is a cheap catalog change (no table lock),
* and the reindex is a single `UPDATE pages SET tsv = <ru_en expr>` a
* ROW-level-lock (RowExclusiveLock) backfill, NOT an ACCESS EXCLUSIVE rewrite
* (mirrors the existing space_id backfill in 20250725T052004). On a LARGE
* tenant this still writes every row + its WAL and leaves dead tuples, so it
* can take MINUTES and blocks the startup migrator for that time but it
* never blocks concurrent reads. It stays inline unconditionally.
*
* - page_embeddings.fts is a GENERATED STORED column; Postgres cannot change a
* generated expression without DROP+ADD, which is a full-table ACCESS
* EXCLUSIVE REWRITE of page_embeddings it blocks ALL reads AND writes on
* that table (including the RAG agent) for the rewrite's duration. That inline
* rewrite is appropriate for small/typical tenants (this fork's target) and
* is the DEFAULT.
*
* LARGE TENANTS have two documented escape hatches, either of which makes the
* migration genuinely no-op the rewrite (it is NOT a blind DROP+ADD):
* (a) Set `SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false`. The migration then
* SKIPS the embeddings rewrite entirely and logs a WARNING. The operator
* MUST perform the ru_en fts swap out-of-band; until they do, the RAG
* lexical leg stays on `english` while the query config is `ru_en` a
* documented, operator-owned desync window. (pages.tsv still swaps
* inline the gate is ONLY the embeddings rewrite.)
* (b) Perform the swap out-of-band BEFORE deploy add a plain column
* batched backfill brief-lock swap CREATE INDEX CONCURRENTLY so
* the `fts` column's generated expression already references the TARGET
* config when the migration runs. The migration detects this (it reads
* the column's actual generation expression from pg_catalog) and does a
* TRUE no-op no DROP, no ADD, no rewrite. This is real idempotency,
* not the old (false) "IF-EXISTS guards no-op" claim: `DROP COLUMN IF
* EXISTS` guards against ABSENCE, not presence, so it would have dropped
* and recreated an existing `fts` regardless. The at-target check is the
* only honest no-op path.
*
* Same documented trade-off family as the #443 trgm GIN migration (20260706T120000).
*/
// pages.tsv trigger body for a given FTS config — mirrors the latest form
// (20250729T213756): f_unaccent + a 1MB text cap on text_content, weights A/B.
function pagesTriggerSql(config: 'ru_en' | 'english') {
return sql`
CREATE OR REPLACE FUNCTION pages_tsvector_trigger() RETURNS trigger AS $$
begin
new.tsv :=
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(coalesce(new.title, ''))), 'A') ||
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(substring(coalesce(new.text_content, ''), 1, 1000000))), 'B');
return new;
end;
$$ LANGUAGE plpgsql;
`;
}
async function swapPagesConfig(db: Kysely<any>, config: 'ru_en' | 'english') {
// 1. Point the trigger at the target config (new/edited rows use it going
// forward). CREATE OR REPLACE FUNCTION takes only a brief catalog lock.
await pagesTriggerSql(config).execute(db);
// 2. Reindex existing rows: recompute tsv directly with the target config. A
// plain UPDATE — row locks, no ACCESS EXCLUSIVE. Equivalent to firing the
// trigger but cheaper (no self-update round trip).
await sql`
UPDATE pages
SET tsv =
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(coalesce(title, ''))), 'A') ||
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(substring(coalesce(text_content, ''), 1, 1000000))), 'B')
`.execute(db);
}
// The default in-migration ACCESS EXCLUSIVE rewrite of page_embeddings.fts is
// ON unless the operator explicitly opts out with the env flag. Parsed strictly
// (mirrors CLIENT_TELEMETRY_ENABLED / DEBUG_MODE in common/): only a literal
// (case-insensitive) 'false' disables it; anything else — unset included —
// keeps the default true.
function inlineEmbeddingsRewriteEnabled(): boolean {
return (
(process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE ?? 'true').toLowerCase() !==
'false'
);
}
// Read page_embeddings.fts's ACTUAL generated-column expression from pg_catalog
// (the generation expression is stored as a column default marked generated).
// Returns '' when the column is absent.
async function embeddingsFtsExpr(db: Kysely<any>): Promise<string> {
const r = await sql<{ def: string }>`
SELECT pg_get_expr(d.adbin, d.adrelid) AS def
FROM pg_attrdef d
JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attname = 'fts'
AND a.attrelid = 'page_embeddings'::regclass
AND NOT a.attisdropped
`.execute(db);
return r.rows[0]?.def ?? '';
}
async function swapEmbeddingsFtsConfig(
db: Kysely<any>,
config: 'ru_en' | 'english',
) {
// 1. TRUE no-op path (real out-of-band escape hatch): if the column's current
// generation expression already references the TARGET config, there is
// nothing to do. An operator who pre-swapped the column out-of-band lands
// here and the migration does NOT rewrite the table. ('ru_en' and 'english'
// are disjoint tokens, neither a substring of the other or of the rest of
// the expression, so a plain contains-check is unambiguous.)
const currentExpr = await embeddingsFtsExpr(db);
if (currentExpr.includes(config)) return;
// 2. Env-gated opt-out: large tenants set SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE
// =false to skip the ACCESS EXCLUSIVE rewrite in-migration and own the swap
// out-of-band. Warn loudly so the desync window is not silent.
if (!inlineEmbeddingsRewriteEnabled()) {
// eslint-disable-next-line no-console
console.warn(
`[migration 20260707T130000] SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false: ` +
`SKIPPING the page_embeddings.fts rewrite to '${config}'. The operator MUST ` +
`perform this fts swap out-of-band. Until then the RAG lexical leg stays on ` +
`its current config while the query config is '${config}' (documented, ` +
`operator-owned desync window).`,
);
return;
}
// 3. Default inline path: the generated `fts` expression can only change via
// DROP+ADD (a full-table ACCESS EXCLUSIVE rewrite; see the lock note in the
// header). The GIN index depends on the column, so it is dropped with it and
// recreated.
await sql`DROP INDEX IF EXISTS idx_page_embeddings_fts`.execute(db);
await sql`ALTER TABLE page_embeddings DROP COLUMN IF EXISTS fts`.execute(db);
await sql`
ALTER TABLE page_embeddings
ADD COLUMN fts tsvector
GENERATED ALWAYS AS (to_tsvector('${sql.raw(config)}', f_unaccent(content))) STORED
`.execute(db);
await sql`
CREATE INDEX IF NOT EXISTS idx_page_embeddings_fts
ON page_embeddings USING gin(fts)
`.execute(db);
}
async function ruEnConfigExists(db: Kysely<any>): Promise<boolean> {
const r = await sql<{ n: number }>`
SELECT count(*)::int AS n FROM pg_ts_config WHERE cfgname = 'ru_en'
`.execute(db);
return (r.rows[0]?.n ?? 0) > 0;
}
async function ensureRuEnConfig(db: Kysely<any>): Promise<void> {
// Idempotent by EXISTENCE, not by drop-recreate. The old `DROP ... IF EXISTS;
// CREATE` was safe only on a first run: on a re-run the page_embeddings.fts
// generated column already has a hard dependency on ru_en, so dropping the
// config would fail. Create only when it is genuinely missing.
if (await ruEnConfigExists(db)) return;
await sql`CREATE TEXT SEARCH CONFIGURATION ru_en (COPY = simple)`.execute(db);
// Latin token classes → english_stem.
await sql`
ALTER TEXT SEARCH CONFIGURATION ru_en
ALTER MAPPING FOR asciiword, asciihword, hword_asciipart
WITH english_stem
`.execute(db);
// Cyrillic / non-ascii token classes → russian_stem.
await sql`
ALTER TEXT SEARCH CONFIGURATION ru_en
ALTER MAPPING FOR word, hword, hword_part
WITH russian_stem
`.execute(db);
}
export async function up(db: Kysely<any>): Promise<void> {
await ensureRuEnConfig(db);
// Flip both stored sides to ru_en (query-side flips in the same commit).
// swapEmbeddingsFtsConfig no-ops when fts already references ru_en, so a
// re-run of up() is idempotent and does NOT re-rewrite the embeddings table.
await swapPagesConfig(db, 'ru_en');
await swapEmbeddingsFtsConfig(db, 'ru_en');
}
export async function down(db: Kysely<any>): Promise<void> {
// Reverse ORDER matters: the trigger and the generated column reference the
// `ru_en` config by name, so they must be moved back to `english` BEFORE the
// config can be dropped (a generated column that still depends on `ru_en` would
// block the DROP with a dependency error).
await swapEmbeddingsFtsConfig(db, 'english');
await swapPagesConfig(db, 'english');
// Drop the config ONLY if nothing still references it. When the embeddings
// rewrite was gated off (SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false), the fts
// column can still reference ru_en — dropping the config would then fail with a
// dependency error. Skip + warn so down() stays non-fatal; the operator drops
// ru_en after completing the out-of-band english swap.
if ((await embeddingsFtsExpr(db)).includes('ru_en')) {
// eslint-disable-next-line no-console
console.warn(
`[migration 20260707T130000] down(): page_embeddings.fts still references ` +
`ru_en (inline rewrite was gated off) — leaving the ru_en text-search ` +
`configuration in place. Drop it out-of-band once fts is back on 'english'.`,
);
return;
}
await sql`DROP TEXT SEARCH CONFIGURATION IF EXISTS ru_en`.execute(db);
}
@@ -0,0 +1,70 @@
import { type Kysely, sql } from 'kysely';
/**
* `ai_chat_run_steps` append-only per-step persistence for an assistant turn
* (#492 wave C). Each finished agent step's UI `parts` (its text part + a part
* per tool call, WITH the tool output) is INSERTed as its own lightweight row the
* moment the step ends, instead of REWRITING the whole assistant row's growing
* `metadata.parts` jsonb on every `onStepFinish`.
*
* WHY a separate table + INSERT (not a jsonb `||` append on the message row): a
* Postgres jsonb UPDATE rewrites the ENTIRE TOASTed row version under MVCC, so
* re-persisting a growing `metadata.parts` on every step is O(n²) write volume
* (a 50-step run with ~100 KB tool outputs wrote hundreds of MB of WAL / dead
* tuples per turn, hammering autovacuum). `||` would only shave the network
* payload the WAL/TOAST rewrite harm remains. An INSERT into a per-step table
* writes ONLY that step's bytes, so the per-turn write volume is O(Σ steps).
*
* The full `metadata.parts` on the message row is assembled ONCE at finalize (the
* terminal completed/error/aborted write). Mid-run, a resuming client's seed is
* reconstructed by concatenating these step rows in `step_index` order which
* reproduces exactly what the old per-step full-row rewrite persisted. Records
* written the OLD way (full `metadata.parts` on the row, no step rows) still
* reconstruct from the row unchanged; the two eras are distinguished by whether
* the row already carries non-empty `metadata.parts` (see reconstructRunParts /
* assembleStepParts in ai-chat.service.ts).
*
* ON DELETE CASCADE on `message_id`: the step rows are a derived projection of the
* assistant message; they must vanish with it (or with its workspace).
*/
export async function up(db: Kysely<any>): Promise<void> {
await db.schema
.createTable('ai_chat_run_steps')
.ifNotExists()
.addColumn('id', 'uuid', (col) =>
col.primaryKey().defaultTo(sql`gen_uuid_v7()`),
)
// The assistant message row this step belongs to (the #183 projection). The
// step rows are a derived, per-step slice of that message, so they cascade.
.addColumn('message_id', 'uuid', (col) =>
col.references('ai_chat_messages.id').onDelete('cascade').notNull(),
)
.addColumn('workspace_id', 'uuid', (col) =>
col.references('workspaces.id').onDelete('cascade').notNull(),
)
// 0-based index of the finished step within the turn. Ordering key for
// reconstruction; unique per message (idempotent step re-persist).
.addColumn('step_index', 'integer', (col) => col.notNull())
// The step's UI parts (text part + a `tool-*` part per call, WITH output).
// Concatenated in step order to rebuild the turn's `metadata.parts`.
.addColumn('parts', 'jsonb', (col) => col.notNull())
.addColumn('created_at', 'timestamptz', (col) =>
col.notNull().defaultTo(sql`now()`),
)
.execute();
// Idempotent per-step persist: a retried INSERT of the same (message, step)
// is a no-op (the service uses ON CONFLICT DO NOTHING). This also serves the
// reconstruction read (WHERE message_id ORDER BY step_index).
await db.schema
.createIndex('ai_chat_run_steps_message_step_uidx')
.ifNotExists()
.on('ai_chat_run_steps')
.columns(['message_id', 'step_index'])
.unique()
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema.dropTable('ai_chat_run_steps').ifExists().execute();
}
@@ -0,0 +1,29 @@
import { type Kysely, sql } from 'kysely';
/**
* #370 Stage B share "approved" (publish-only-the-saved-version) mode.
*
* Adds `shares.published_mode`, the per-share switch between:
* - 'live' public readers see the current draft (legacy behavior)
* - 'approved' public readers see the LAST manually-saved version
* (page_history.kind='manual'), not the live draft
*
* NOT NULL with a 'live' default so every existing share keeps its current
* (live) semantics with no data backfill. Standalone + single-concern +
* additive per the #363 crash-loop rule no coupled baseline write here;
* the first manual baseline is minted by the service on enable, not by this
* migration. Stored as a short varchar to stay forward-compatible without an
* enum migration.
*/
export async function up(db: Kysely<any>): Promise<void> {
await db.schema
.alterTable('shares')
.addColumn('published_mode', 'varchar(20)', (col) =>
col.notNull().defaultTo(sql`'live'`),
)
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema.alterTable('shares').dropColumn('published_mode').execute();
}
@@ -0,0 +1,95 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import { AiChatRunStep } from '@docmost/db/types/entity.types';
/**
* Append-only per-step persistence for an assistant turn (#492). Each finished
* agent step's UI `parts` (its text part + a `tool-*` part per call, WITH the
* tool output) is INSERTed as its own lightweight row the moment the step ends
* instead of REWRITING the assistant row's growing `metadata.parts` jsonb on every
* `onStepFinish` (a Postgres jsonb UPDATE rewrites the whole TOASTed row version
* under MVCC, so that was O(n²) WAL/dead-tuple churn per turn).
*
* The full `metadata.parts` on the message row is assembled ONCE at finalize;
* mid-run, a resuming client's seed is rebuilt from these rows in `stepIndex`
* order (see `assembleStepParts` / the reconstruct seam in ai-chat.service.ts).
* Every method is workspace-scoped as defense-in-depth.
*/
@Injectable()
export class AiChatRunStepRepo {
constructor(@InjectKysely() private readonly db: KyselyDB) {}
/**
* Append one finished step's parts. Idempotent: a retried persist of the SAME
* (message, stepIndex) is a no-op via ON CONFLICT DO NOTHING the per-step
* writes are fired fire-and-forget + serialized, and a duplicate must never
* throw into the stream or double the parts. Returns whether a NEW row landed
* (false = the step was already persisted).
*/
async insertStep(
messageId: string,
workspaceId: string,
stepIndex: number,
parts: unknown,
trx?: KyselyTransaction,
): Promise<boolean> {
const db = dbOrTx(this.db, trx);
const inserted = await db
.insertInto('aiChatRunSteps')
.values({
messageId,
workspaceId,
stepIndex,
// jsonb column: cast through never (same pattern as the message repo).
parts: parts as never,
})
.onConflict((oc) => oc.columns(['messageId', 'stepIndex']).doNothing())
.returning('id')
.executeTakeFirst();
return inserted !== undefined;
}
/** All persisted steps for ONE assistant message, in step order. */
async findByMessage(
messageId: string,
workspaceId: string,
): Promise<AiChatRunStep[]> {
return this.db
.selectFrom('aiChatRunSteps')
.selectAll('aiChatRunSteps')
.where('messageId', '=', messageId)
.where('workspaceId', '=', workspaceId)
.orderBy('stepIndex', 'asc')
.execute();
}
/**
* All persisted steps for a SET of assistant messages, grouped by messageId
* (each group in step order). One query for the batch the hydration seam
* (getMessages / delta / export) calls this only for the rows that actually
* need reconstruction (an active new-style row whose `metadata.parts` is still
* empty), which is usually none, so this is skipped on the common path.
*/
async findByMessageIds(
messageIds: string[],
workspaceId: string,
): Promise<Map<string, AiChatRunStep[]>> {
const byMessage = new Map<string, AiChatRunStep[]>();
if (messageIds.length === 0) return byMessage;
const rows = await this.db
.selectFrom('aiChatRunSteps')
.selectAll('aiChatRunSteps')
.where('messageId', 'in', messageIds)
.where('workspaceId', '=', workspaceId)
.orderBy('stepIndex', 'asc')
.execute();
for (const row of rows) {
const list = byMessage.get(row.messageId);
if (list) list.push(row);
else byMessage.set(row.messageId, [row]);
}
return byMessage;
}
}
@@ -200,8 +200,11 @@ export class PageEmbeddingRepo {
*
* The `model_dimensions = $dim` filter applies ONLY on the semantic side
* (cosine compares same-dimension vectors; pgvector errors otherwise). The
* lexical side (`fts`) is dimension-independent. If `websearch_to_tsquery`
* yields an EMPTY query (e.g. the text is all stopwords) the `@@` matches
* lexical side (`fts`) is dimension-independent. Its query config is `ru_en`,
* matched IN LOCKSTEP with the `page_embeddings.fts` generated column's config
* (#529 acceptance #13): a mismatch silently breaks Cyrillic RAG retrieval. If
* `websearch_to_tsquery` yields an EMPTY query (e.g. the text is all stopwords)
* the `@@` matches
* nothing and the lexical CTE is empty, so results degrade to pure-semantic
* which is correct behaviour, not an error.
*
@@ -249,7 +252,7 @@ export class PageEmbeddingRepo {
row_number() OVER (ORDER BY ts_rank(pe.fts, q.query) DESC) AS rank_ix
FROM page_embeddings pe
JOIN pages p ON p.id = pe.page_id,
websearch_to_tsquery('english', f_unaccent(${queryText})) AS q(query)
websearch_to_tsquery('ru_en', f_unaccent(${queryText})) AS q(query)
WHERE pe.workspace_id = ${workspaceId}
AND pe.space_id IN (${spaceList})
AND p.deleted_at IS NULL
@@ -157,6 +157,44 @@ export class PageHistoryRepo {
return { ...result, items: result.items.map(attachPageHistoryAgent) };
}
/**
* #395 cheap projection of a page's FULL history timeline for the work-time
* estimate: only the columns the sessionizer needs, no heavy `content`, sorted
* oldestnewest. The secondary `id` tie-break keeps rows sharing a `createdAt`
* (e.g. a synchronous pre-agent boundary row + the immediate agent snapshot)
* in a deterministic order.
*/
async findTimelineByPageId(
pageId: string,
trx?: KyselyTransaction,
): Promise<
Array<
Pick<
PageHistory,
| 'createdAt'
| 'lastUpdatedById'
| 'lastUpdatedSource'
| 'lastUpdatedAiChatId'
| 'kind'
>
>
> {
const db = dbOrTx(this.db, trx);
return db
.selectFrom('pageHistory')
.select([
'createdAt',
'lastUpdatedById',
'lastUpdatedSource',
'lastUpdatedAiChatId',
'kind',
])
.where('pageId', '=', pageId)
.orderBy('createdAt', 'asc')
.orderBy('id', 'asc')
.execute();
}
async findPageLastHistory(
pageId: string,
opts?: {
@@ -181,6 +219,36 @@ export class PageHistoryRepo {
.executeTakeFirst();
}
/**
* #370 Stage B resolve the latest snapshot of a page at a given
* intentionality tier. Used by the "approved" share mode to serve the last
* manually-saved version (`kind='manual'`) to public readers instead of the
* live draft. Modeled on `findPageLastHistory` with an added `kind` filter and
* the same deterministic `(createdAt desc, id desc)` tie-break, so two rows
* sharing a createdAt still resolve to a single stable "latest".
*/
async findLatestByPageIdAndKind(
pageId: string,
kind: PageHistoryKind,
opts?: {
includeContent?: boolean;
trx?: KyselyTransaction;
},
) {
const db = dbOrTx(this.db, opts?.trx);
return await db
.selectFrom('pageHistory')
.select(this.baseFields)
.$if(opts?.includeContent, (qb) => qb.select('content'))
.where('pageId', '=', pageId)
.where('kind', '=', kind)
.limit(1)
.orderBy('createdAt', 'desc')
.orderBy('id', 'desc')
.executeTakeFirst();
}
withLastUpdatedBy(eb: ExpressionBuilder<DB, 'pageHistory'>) {
return jsonObjectFrom(
eb
@@ -28,6 +28,7 @@ export class ShareRepo {
'pageId',
'includeSubPages',
'searchIndexing',
'publishedMode',
'creatorId',
'spaceId',
'workspaceId',
+18
View File
@@ -340,6 +340,7 @@ export interface Shares {
includeSubPages: Generated<boolean | null>;
key: string;
pageId: string | null;
publishedMode: Generated<string>;
searchIndexing: Generated<boolean | null>;
spaceId: string;
updatedAt: Generated<Timestamp>;
@@ -692,6 +693,22 @@ export interface AiChatRuns {
updatedAt: Generated<Timestamp>;
}
// Append-only per-step persistence for an assistant turn (#492). Mirrors
// migration 20260708T120000-ai-chat-run-steps.ts. Each finished agent step's UI
// `parts` are INSERTed as their own row (instead of rewriting the message row's
// growing `metadata.parts` jsonb every step — an O(n²) WAL/TOAST churn). The full
// `metadata.parts` is assembled once at finalize; mid-run a resuming client's seed
// is rebuilt by concatenating these rows in `stepIndex` order. Cascades with the
// assistant message row it projects.
export interface AiChatRunSteps {
id: Generated<string>;
messageId: string;
workspaceId: string;
stepIndex: number;
parts: Json;
createdAt: Generated<Timestamp>;
}
// Per-(chat,page) snapshot of the open page's Markdown at the END of the agent's
// previous turn (#274). Mirrors migration 20260702T120000-ai-chat-page-snapshot.ts.
// The next turn diffs the CURRENT Markdown against `contentMd` to surface edits a
@@ -729,6 +746,7 @@ export interface DB {
aiChats: AiChats;
aiChatMessages: AiChatMessages;
aiChatRuns: AiChatRuns;
aiChatRunSteps: AiChatRunSteps;
aiChatPageSnapshots: AiChatPageSnapshots;
apiKeys: ApiKeys;
attachments: Attachments;
@@ -4,6 +4,7 @@ import {
AiChats,
AiChatMessages,
AiChatRuns,
AiChatRunSteps,
AiChatPageSnapshots,
Attachments,
Comments,
@@ -64,6 +65,12 @@ export type InsertableAiChatMessage = Omit<Insertable<AiChatMessages>, 'tsv'>;
export type AiChatRun = Selectable<AiChatRuns>;
export type InsertableAiChatRun = Insertable<AiChatRuns>;
// AI Chat Run Step (#492): append-only per-step parts persistence. Each finished
// agent step's UI parts are stored as their own row; the full turn's parts are
// assembled from these (in stepIndex order) for a mid-run resume seed.
export type AiChatRunStep = Selectable<AiChatRunSteps>;
export type InsertableAiChatRunStep = Insertable<AiChatRunSteps>;
// AI Chat Page Snapshot (#274): per-(chat,page) Markdown snapshot taken at the
// end of the agent's previous turn, diffed against the current page next turn to
// detect human edits made between turns.
@@ -0,0 +1,70 @@
import { INTERNAL_LINK_REGEX } from './utils';
import { isInternalPagePath } from '@docmost/prosemirror-markdown';
/**
* Cross-package DRIFT GUARD for the internal-link subset invariant (#522).
*
* The client-side `isInternalPagePath`
* (`packages/prosemirror-markdown/src/lib/internal-links.ts`) promotes a markdown
* link to `internal: true` on import. Every link it marks internal MUST be one
* the server would backlink and export-rewrite i.e. the client matcher MUST be
* a STRICT SUBSET of the server's canonical `INTERNAL_LINK_REGEX`
* (`./utils.ts`). If the client ever accepts a path the server rejects, that link
* is stored internal but silently dropped from the backlink graph and broken on
* export the exact bug #522 fixed.
*
* This spec is the load-bearing guard: it imports the LIVE server regex AND the
* LIVE `isInternalPagePath` (no hand-copied regex on either side). A narrowing of
* EITHER most dangerously the server regex reddens here. The in-package
* accept/reject test documents the client's behaviour but cannot see the server
* regex; this top-layer spec is what makes the subset relation mechanical
* (AGENTS.md rule #7: a CI test that fails on drift of the source of truth).
*/
describe('internal-link subset parity (client isInternalPagePath ⊆ server INTERNAL_LINK_REGEX)', () => {
// Every path the CLIENT accepts. Kept deliberately broad across the risky
// dimensions — the slug charset (digits, hyphens, mixed case, leading/trailing
// hyphen), the space charset, and the optional trailing slash — so a narrowing
// of the server regex on any of them reddens the subset assertion below.
const CLIENT_ACCEPTS = [
'/s/eng/p/abc123',
'/s/eng/p/abc123/', // trailing slash
'/s/My-Space/p/A-b-C-9', // hyphens + mixed case in both segments
'/s/x/p/z', // shortest
'/s/eng/p/my-page-abc123', // slug with hyphens (extractPageSlugId shape)
'/s/eng/p/-lead', // leading hyphen in slug
'/s/eng/p/trail-', // trailing hyphen in slug
'/s/eng/p/0123456789', // all-digit slug
'/s/a.b/p/abc', // dot in the SPACE segment (space charset is [^/]+)
'/s/space with space/p/abc', // space char in the SPACE segment
];
it('every corpus path is actually client-accepted (guards the corpus itself)', () => {
// If a path here stopped being client-accepted the subset test would pass
// vacuously; assert acceptance up front so the corpus stays meaningful. The
// filter-to-empty form names the offending paths on failure.
const notAccepted = CLIENT_ACCEPTS.filter((h) => !isInternalPagePath(h));
expect(notAccepted).toEqual([]);
});
it('every client-accepted path also matches the LIVE server regex (subset)', () => {
// The mechanical drift guard: narrow the server INTERNAL_LINK_REGEX and at
// least one hyphen/charset/structure case appears here.
const notInServer = CLIENT_ACCEPTS.filter((h) => !INTERNAL_LINK_REGEX.test(h));
expect(notInServer).toEqual([]);
});
it('the client rejects forbidden-slug-char paths the server also rejects', () => {
// Documents the subset BOUNDARY: correct shape, forbidden slug char. The
// client and the LIVE server regex must agree on rejection.
const forbidden = [
'/s/eng/p/abc.def',
'/s/eng/p/abc_def',
'/s/eng/p/abc%20',
'/s/eng/p/abc~x',
];
const clientAccepts = forbidden.filter((h) => isInternalPagePath(h));
const serverAccepts = forbidden.filter((h) => INTERNAL_LINK_REGEX.test(h));
expect(clientAccepts).toEqual([]);
expect(serverAccepts).toEqual([]);
});
});
@@ -85,6 +85,14 @@ export class ImportController {
throw new BadRequestException('spaceId is required');
}
// #502: optional multipart field. Only the MCP agent `createPage` path sends
// `disableMarkdownExtensions=true` (its body is agent-authored plain prose /
// config, so a `$…$` span must stay literal and a bare `www.host` must not
// autolink). A HUMAN file upload omits the field, so it stays false and math
// + autolink remain ON for human imports. Settable ONLY via this API param.
const disableMarkdownExtensions =
file.fields?.disableMarkdownExtensions?.value === 'true';
const ability = await this.spaceAbility.createForUser(user, spaceId);
if (ability.cannot(SpaceCaslAction.Edit, SpaceCaslSubject.Page)) {
throw new ForbiddenException();
@@ -95,6 +103,7 @@ export class ImportController {
user.id,
spaceId,
workspace.id,
disableMarkdownExtensions,
);
const ext = path.extname(file.filename).toLowerCase();
@@ -0,0 +1,67 @@
// Importing ImportService transitively loads import-formatter.ts, which imports
// the ESM-only @sindresorhus/slugify (not in jest's transform allowlist). It is
// irrelevant to this path, so mock it to keep the module graph loadable (mirrors
// the sibling import.service specs).
jest.mock('@sindresorhus/slugify', () => ({
__esModule: true,
default: (input: string) => String(input),
}));
import { ImportService } from './import.service';
// #502 BLOCKER 1: the server markdown import path (`/pages/import`) now accepts an
// optional `disableMarkdownExtensions` flag threaded into `processMarkdown`.
// - MCP agent `createPage` sends it TRUE -> extensions OFF (a `$…$` config span
// stays literal, a schemeless `www.host` is not autolinked).
// - a HUMAN file upload omits it (default FALSE) -> extensions ON, so a real
// `$x^2$` still becomes a formula (human imports unaffected).
// `processMarkdown` only uses the imported converter (no injected deps on this
// path), so the service is constructed with null deps for this focused unit test.
function makeService(): ImportService {
return new ImportService(null as any, null as any, null as any, null as any);
}
function findAll(node: any, type: string, acc: any[] = []): any[] {
if (!node || typeof node !== 'object') return acc;
if (node.type === type) acc.push(node);
if (Array.isArray(node.content)) for (const c of node.content) findAll(c, type, acc);
return acc;
}
function hasLink(node: any): boolean {
return findAll(node, 'text').some((t: any) =>
t.marks?.some((m: any) => m.type === 'link'),
);
}
describe('ImportService.processMarkdown — #502 disableMarkdownExtensions', () => {
it('disableMarkdownExtensions=true (MCP createPage): `$…$` stays literal, no math', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('export A=$FOO and B=$BAR done', true);
expect(findAll(doc, 'mathInline')).toHaveLength(0);
});
it('disableMarkdownExtensions=true: a schemeless www is NOT autolinked', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see www.example.com here', true);
expect(hasLink(doc)).toBe(false);
});
it('disableMarkdownExtensions=true: an explicit https:// STILL links', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see https://example.com here', true);
expect(hasLink(doc)).toBe(true);
});
it('DEFAULT (human upload): a real `$x^2$` DOES become a math node', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('$x^2$');
expect(findAll(doc, 'mathInline')).toHaveLength(1);
});
it('DEFAULT (human upload): a schemeless www IS autolinked', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see www.example.com here');
expect(hasLink(doc)).toBe(true);
});
});
@@ -52,6 +52,12 @@ export class ImportService {
userId: string,
spaceId: string,
workspaceId: string,
// #502: when true, the markdown importer runs with the two layered
// extensions OFF (a `$…$` span stays literal text; a schemeless `www.host` is
// NOT autolinked). ONLY the MCP agent `createPage` path sets this; a HUMAN
// file upload never passes it, so it defaults false and math/autolink stay ON
// for human imports (their `$x^2$` still becomes a formula).
disableMarkdownExtensions = false,
) {
const file = await filePromise;
const fileBuffer = await file.toBuffer();
@@ -66,7 +72,10 @@ export class ImportService {
try {
if (fileExtension.endsWith('.md')) {
prosemirrorState = await this.processMarkdown(fileContent);
prosemirrorState = await this.processMarkdown(
fileContent,
disableMarkdownExtensions,
);
} else if (fileExtension.endsWith('.html')) {
prosemirrorState = await this.processHTML(fileContent);
}
@@ -138,7 +147,12 @@ export class ImportService {
return createdPage;
}
async processMarkdown(markdownInput: string): Promise<any> {
async processMarkdown(
markdownInput: string,
// #502: forwarded to the importer. DEFAULT false keeps math + fuzzy autolink
// ON (human uploads unaffected); the MCP agent `createPage` path passes true.
disableMarkdownExtensions = false,
): Promise<any> {
// Canonical markdown -> ProseMirror JSON directly via
// `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate and no
// second editor-ext markdown layer. Foreign markdown surfaces the strict
@@ -147,7 +161,12 @@ export class ImportService {
// The HTML-cleanup pass (`normalizeImportHtml`) is intentionally skipped here:
// it targets foreign *HTML* (Notion/XWiki), which only ever arrives on the
// `.html` path (`processHTML`), never as canonical markdown.
return markdownToProseMirror(normalizeForeignMarkdown(markdownInput));
return markdownToProseMirror(
normalizeForeignMarkdown(markdownInput),
disableMarkdownExtensions
? { parseMath: false, fuzzyLinkify: false }
: undefined,
);
}
async processHTML(htmlInput: string): Promise<any> {
@@ -4,6 +4,7 @@ import { join } from 'path';
import * as fs from 'node:fs';
import fastifyStatic from '@fastify/static';
import { EnvironmentService } from '../environment/environment.service';
import { resolveClientDistPath } from '../../common/helpers/client-version';
/**
* Resolve the response headers for a statically served client asset.
@@ -56,14 +57,7 @@ export class StaticModule implements OnModuleInit {
const httpAdapter = this.httpAdapterHost.httpAdapter;
const app = httpAdapter.getInstance();
const clientDistPath = join(
__dirname,
'..',
'..',
'..',
'..',
'client/dist',
);
const clientDistPath = resolveClientDistPath();
const indexFilePath = join(clientDistPath, 'index.html');
+37 -2
View File
@@ -9,10 +9,14 @@ import {
import { Server, Socket } from 'socket.io';
import { TokenService } from '../core/auth/services/token.service';
import { JwtPayload, JwtType } from '../core/auth/dto/jwt-payload';
import { OnModuleDestroy } from '@nestjs/common';
import { Logger, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
import { SpaceMemberRepo } from '@docmost/db/repos/space/space-member.repo';
import { WsService } from './ws.service';
import { getSpaceRoomName, getUserRoomName } from './ws.utils';
import {
readClientBuildVersion,
resolveClientDistPath,
} from '../common/helpers/client-version';
import * as cookie from 'cookie';
@WebSocketGateway({
@@ -20,17 +24,40 @@ import * as cookie from 'cookie';
transports: ['websocket'],
})
export class WsGateway
implements OnGatewayConnection, OnGatewayInit, OnModuleDestroy
implements
OnGatewayConnection,
OnGatewayInit,
OnModuleInit,
OnModuleDestroy
{
@WebSocketServer()
server: Server;
private readonly logger = new Logger(WsGateway.name);
// The build version of the client bundle shipped in this image, read once at
// startup from client/dist/version.json (single source of truth, same value
// baked into the client's APP_VERSION). Empty string => version.json missing
// or empty => the proactive version-coherence reload feature stays inert.
private appVersion = '';
constructor(
private tokenService: TokenService,
private spaceMemberRepo: SpaceMemberRepo,
private wsService: WsService,
) {}
onModuleInit(): void {
this.appVersion = readClientBuildVersion(resolveClientDistPath());
if (this.appVersion) {
this.logger.log(`app-version reload: ACTIVE (v=${this.appVersion})`);
} else {
this.logger.log(
'app-version reload: DISABLED (version.json missing/empty)',
);
}
}
afterInit(server: Server): void {
this.wsService.setServer(server);
}
@@ -55,6 +82,14 @@ export class WsGateway
const spaceRooms = userSpaceIds.map((id) => getSpaceRoomName(id));
client.join([userRoom, workspaceRoom, ...spaceRooms]);
// Announce this container's client build version to the freshly
// authenticated socket. On a redeploy the client reconnects to the new
// container and receives the new version here, letting it guard-reload
// before it hits a stale lazy chunk. Per-connect only (no broadcast):
// natural reconnect covers both single-container and cluster without a
// thundering-herd fleet reload.
client.emit('app-version', { version: this.appVersion });
} catch (err) {
client.emit('Unauthorized');
client.disconnect();
@@ -0,0 +1,412 @@
import * as http from 'node:http';
import { Kysely } from 'kysely';
import { tool } from 'ai';
import { z } from 'zod';
import { MockLanguageModelV3, convertArrayToReadableStream } from 'ai/test';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import {
AiChatService,
assembleStepParts,
assistantParts,
rowHasInlineParts,
stepMarkerMetadata,
} from 'src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #492 append-persist the REAL onStep WRITE path (F2) and the model-REPLAY
* hydration path (F1), driven through `AiChatService.stream` against a LIVE
* Postgres with a REAL `AiChatRunStepRepo` INJECTED. The existing append-persist
* int-specs hand-roll the insert+marker cycle via the repos directly and build
* the service with `aiChatRunStepRepo: undefined` (only the legacy-fallback branch
* is covered), so an off-by-one on `stepsPersisted-1`, a wrong `capturedSteps`
* slice, or a broken marker payload would pass all of them. These tests exercise
* the actual `updateStreaming` append-persist branch end to end.
*
* The seam is the injected `model` (a seeded `MockLanguageModelV3` from `ai/test`)
* plus a REAL Node `ServerResponse` as the hijacked socket mirrors
* ai-chat-stream.int-spec.ts.
*/
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
async function waitFor(
cond: () => Promise<boolean> | boolean,
{ timeoutMs = 15_000, stepMs = 25 } = {},
): Promise<void> {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
if (await cond()) return;
await sleep(stepMs);
}
throw new Error('waitFor: condition not met within timeout');
}
// A real Node ServerResponse wired to a live socket (identical helper to the
// stream int-spec) so the SDK's pipe/heartbeat writes behave as in prod.
function makeRealResponse(): Promise<{
res: http.ServerResponse;
cleanup: () => Promise<void>;
}> {
return new Promise((resolve) => {
const server = http.createServer((_req, res) => {
resolve({
res,
cleanup: () =>
new Promise<void>((done) => {
try {
if (!res.writableEnded) res.end();
} catch {
/* socket already gone */
}
server.close(() => done());
}),
});
});
server.listen(0, () => {
const port = (server.address() as any).port;
const creq = http.request({ port, method: 'GET' }, (cres) => {
cres.resume();
});
creq.on('error', () => undefined);
creq.end();
});
});
}
// Stream parts for a normal, successful single-step turn.
function successStream() {
return convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: 't1' },
{ type: 'text-delta', id: 't1', delta: 'Hello' },
{ type: 'text-delta', id: 't1', delta: ' there' },
{ type: 'text-end', id: 't1' },
{
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 10, outputTokens: 5, totalTokens: 15 },
},
] as any);
}
// A THREE-step turn: steps 0 and 1 each emit text + an `echo` tool call (the SDK
// runs the tool and continues); step 2 answers and stops. Three steps is
// deliberate: the LAST finished step's append-persist write races the terminal
// finalize (which writes the full inline parts anyway, so a lost last-step row is
// by design), but the NON-final steps 0 and 1 always drain to the steps table
// before finalize — so those are what the test asserts on deterministically.
function threeStepModel(): MockLanguageModelV3 {
let step = 0;
const toolStep = (i: number) => ({
stream: convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: `s${i}` },
{ type: 'text-delta', id: `s${i}`, delta: `step ${i} ` },
{ type: 'text-end', id: `s${i}` },
{
type: 'tool-call',
toolCallId: `c${i}`,
toolName: 'echo',
input: JSON.stringify({ msg: `m${i}` }),
},
{
type: 'finish',
finishReason: 'tool-calls',
usage: { inputTokens: 5, outputTokens: 3, totalTokens: 8 },
},
] as any),
});
return new MockLanguageModelV3({
doStream: async () => {
const n = step++;
// Realistic inter-step latency. A real model spends seconds per step, so the
// fire-and-forget per-step write chain drains to the steps table BETWEEN
// steps; the mock otherwise collapses all steps into microseconds and the
// terminal finalize wins the race before any but the first step persists.
if (n > 0) await sleep(200);
if (n < 2) return toolStep(n);
return {
stream: convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: 's2' },
{ type: 'text-delta', id: 's2', delta: 'final answer' },
{ type: 'text-end', id: 's2' },
{
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 6, outputTokens: 4, totalTokens: 10 },
},
] as any),
};
},
} as any);
}
describe('#492 append-persist service paths [integration]', () => {
let db: Kysely<any>;
let aiChatRepo: AiChatRepo;
let msgRepo: AiChatMessageRepo;
let stepRepo: AiChatRunStepRepo;
let workspaceId: string;
let userId: string;
let closeCalls: number;
const mcpClients = {
toolsFor: async () => ({
tools: {},
clients: [
{
close: async () => {
closeCalls += 1;
},
},
],
outcomes: [],
instructions: [],
}),
};
// Build the service WITH a REAL AiChatRunStepRepo injected (the property under
// test) — unlike the legacy-fallback harness that passes it as undefined.
const echoTool = tool({
description: 'echo the message back',
inputSchema: z.object({ msg: z.string() }),
execute: async ({ msg }) => ({ echoed: msg }),
});
function buildService(): AiChatService {
return new AiChatService(
{ getChatModel: async () => null } as any,
aiChatRepo,
msgRepo,
{} as any, // aiChatPageSnapshotRepo
{ resolve: async () => null } as any, // aiSettings
{ forUser: async () => ({ echo: echoTool }) } as any, // tools
mcpClients as any,
{} as any, // aiAgentRoleRepo
{} as any, // pageRepo
{} as any, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatFinalStepLockdownEnabled: () => false,
} as any, // environment (deferred OFF -> all tools active every step)
undefined, // streamRegistry
undefined, // aiChatRunService
stepRepo, // #492 aiChatRunStepRepo — the append-persist backend
);
}
function userUiMessage(text: string) {
return {
id: `u-${Math.random()}`,
role: 'user',
parts: [{ type: 'text', text }],
};
}
async function runStream(opts: {
model: MockLanguageModelV3;
chatId: string;
body: any;
}): Promise<void> {
closeCalls = 0;
const service = buildService();
const { res, cleanup } = await makeRealResponse();
try {
await service.stream({
user: { id: userId, workspaceId } as any,
workspace: { id: workspaceId, name: 'WS' } as any,
sessionId: 'sess-1',
body: opts.body,
res: { raw: res } as any,
signal: new AbortController().signal,
model: opts.model as any,
role: null,
} as any);
await waitFor(async () => {
const rows = await msgRepo.findAllByChat(opts.chatId, workspaceId);
return rows.some(
(r) =>
r.role === 'assistant' &&
['completed', 'error', 'aborted'].includes(r.status as string),
);
});
await waitFor(() => closeCalls > 0, { timeoutMs: 5_000 });
} finally {
await cleanup();
}
}
beforeAll(async () => {
db = getTestDb();
aiChatRepo = new AiChatRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
stepRepo = new AiChatRunStepRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
// --- F2: the real onStep append-persist WRITE branch -----------------------
it('drives steps through the real onStep path: per-step rows + marker match a single-row flush', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const model = threeStepModel();
// Capture the mid-run step-marker UPDATEs the append-persist branch writes on
// the assistant row (a { parts: [], toolTraceVersion, stepsPersisted } patch).
const updateSpy = jest.spyOn(msgRepo, 'update');
try {
await runStream({
model,
chatId,
body: { chatId, messages: [userUiMessage('call the tool then answer')] },
});
const rows = await msgRepo.findAllByChat(chatId, workspaceId);
const assistant = rows.find((r) => r.role === 'assistant')!;
expect(assistant).toBeDefined();
expect(assistant.status).toBe('completed');
// The turn finalizes with the FULL inline parts assembled by a single-row
// flush (assistantParts over every step) — the baseline the per-step slices
// must reproduce.
expect(rowHasInlineParts(assistant)).toBe(true);
const finalParts = (assistant.metadata as { parts: any[] }).parts;
// The two NON-final finished steps each landed their own row, in stepIndex
// order. (The fire-and-forget write chain drains before the next step, so
// poll until both are on disk; the LAST step's write may lose the finalize
// race, which is by design — its parts are already in `finalParts`.)
await waitFor(async () => {
const s = await stepRepo.findByMessage(assistant.id, workspaceId);
return s.length >= 2;
});
const steps = await stepRepo.findByMessage(assistant.id, workspaceId);
expect(steps[0].stepIndex).toBe(0);
expect(steps[1].stepIndex).toBe(1);
// Each per-step row carries a NON-trivial slice: this step's text part + its
// paired tool part (guards a mutation that persists empty/whole-turn parts).
const s0 = steps[0].parts as any[];
expect(s0).toContainEqual({ type: 'text', text: 'step 0 ' });
expect(s0.some((p) => p.type === 'tool-echo')).toBe(true);
// The per-step slices are EXACTLY the corresponding prefix of the single-row
// flush: assembleStepParts([step0, step1]) === finalParts[0 .. len0+len1].
// This is what an off-by-one on `stepsPersisted-1` (a wrong `capturedSteps`
// slice) or a shifted stepIndex breaks — the prefix no longer aligns.
const prefixLen =
(steps[0].parts as any[]).length + (steps[1].parts as any[]).length;
expect(assembleStepParts([steps[0], steps[1]] as any)).toEqual(
finalParts.slice(0, prefixLen),
);
// The mid-run step markers advanced 1 -> 2 -> ... (the resume frontier), each
// a shape-stable empty-parts marker equal to a single-row flush's marker.
const markerCounts = updateSpy.mock.calls
.map((c) => (c[2] as any)?.metadata)
.filter(
(m) =>
m &&
Array.isArray(m.parts) &&
m.parts.length === 0 &&
typeof m.stepsPersisted === 'number',
)
.map((m) => m.stepsPersisted);
// Monotonic from 1, covering at least the two non-final steps.
expect(markerCounts.slice(0, 2)).toEqual([1, 2]);
expect(
updateSpy.mock.calls
.map((c) => (c[2] as any)?.metadata)
.find((m) => m && m.stepsPersisted === 2),
).toEqual(stepMarkerMetadata(2));
} finally {
updateSpy.mockRestore();
}
}, 60_000);
// --- F1: model-REPLAY hydrates a hard-crashed mid-run turn from the steps table
it('replays a hard-crashed mid-run turn WITH its partial steps hydrated from the steps table', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
// Prior turn: a genuine user question...
await createMessage(db, {
workspaceId,
chatId,
userId,
role: 'user',
content: 'What is in the design doc?',
createdAt: new Date(Date.now() - 3000),
});
// ...and an assistant row that a HARD crash (SIGKILL/OOM) left mid-run: only a
// step marker on the row (metadata.parts:[] , content:''), NO terminal
// callback ever fired, so its real parts live ONLY in ai_chat_run_steps.
const crashed = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: '',
status: 'aborted',
metadata: stepMarkerMetadata(1),
createdAt: new Date(Date.now() - 2000),
});
// The durable partial step: some reasoning text + a completed getPage tool
// call (input + output), exactly what #183 step-granular durability preserves.
await stepRepo.insertStep(
crashed.id,
workspaceId,
0,
assistantParts(
[
{
text: 'HYDRATED_PARTIAL_STEP the doc says',
toolCalls: [
{ toolCallId: 'g1', toolName: 'getPage', input: { id: 'p1' } },
],
toolResults: [
{
toolCallId: 'g1',
toolName: 'getPage',
output: { id: 'p1', body: 'PARTIAL_TOOL_OUTPUT budget section' },
},
],
} as any,
],
'',
),
);
// The NEXT turn: the model just answers. The service must REPLAY the crashed
// assistant turn with its partial parts hydrated from the steps table.
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: successStream() }),
} as any);
await runStream({
model,
chatId,
body: { chatId, messages: [userUiMessage('Continue please')] },
});
expect(model.doStreamCalls.length).toBeGreaterThan(0);
const prompt = JSON.stringify(model.doStreamCalls[0].prompt);
// The partial step's TEXT reached the model context (it would be an empty text
// part without hydration — rowToUiMessage falls back to `content:''`).
expect(prompt).toContain('HYDRATED_PARTIAL_STEP');
// The partial TOOL RESULT survived too (durable in the steps table, replayed).
expect(prompt).toContain('PARTIAL_TOOL_OUTPUT');
// The genuine prior user turn is present as well (sanity: real history replay).
expect(prompt).toContain('What is in the design doc?');
}, 60_000);
});
@@ -0,0 +1,173 @@
import { randomBytes } from 'crypto';
import { Kysely, sql } from 'kysely';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import {
assistantParts,
flushAssistant,
stepMarkerMetadata,
} from '../../src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
} from './db';
/**
* #492 append-persist WRITE-VOLUME regression on a LIVE Postgres, measured via
* the `pg_current_wal_lsn()` delta around a realistic multi-step run driven through
* the REAL repos (not a mock a mock cannot observe MVCC/TOAST rewrite volume, the
* whole point). Proves the core claim:
*
* NEW (per-step INSERT into ai_chat_run_steps + a CHEAP step-marker UPDATE on the
* message row) writes O(Σ steps) of WAL each step writes only its own bytes.
*
* OLD (the pre-#492 full-row rewrite: re-persist the GROWING metadata.parts on
* every onStepFinish) writes O(n²) step k rewrites the whole TOASTed jsonb of
* all k prior outputs.
*
* The OLD path here IS the reverted behavior, so this doubles as the mutation
* check: swapping the new path back to `flushAssistant` full-row UPDATEs reddens
* the assertion (OLD is many times larger).
*/
type Step = {
text: string;
toolCalls: Array<{ toolCallId: string; toolName: string; input: unknown }>;
toolResults: Array<{ toolCallId: string; toolName: string; output: unknown }>;
};
// ~100 KB INCOMPRESSIBLE output per step (a page read). Random base64 so TOAST
// cannot compress it away and hide the real write volume.
function makeStep(i: number, outputBytes = 100_000): Step {
const body = randomBytes(Math.ceil(outputBytes * 0.75)).toString('base64');
return {
text: `step ${i} reasoning`,
toolCalls: [
{ toolCallId: `c${i}`, toolName: 'getPage', input: { id: `p${i}` } },
],
toolResults: [
{
toolCallId: `c${i}`,
toolName: 'getPage',
output: { id: `p${i}`, title: `Page ${i}`, body },
},
],
};
}
async function walDelta(
db: Kysely<any>,
fn: () => Promise<void>,
): Promise<number> {
const before = (
await sql<{ l: string }>`select pg_current_wal_lsn() as l`.execute(db)
).rows[0].l;
await fn();
// NOTE: no pg_switch_wal() — a segment switch pads the LSN to the next 16 MB
// boundary and would swamp the delta. The raw LSN advances by the WAL bytes.
const after = (
await sql<{ l: string }>`select pg_current_wal_lsn() as l`.execute(db)
).rows[0].l;
return Number(
(
await sql<{
d: string;
}>`select pg_wal_lsn_diff(${after}::pg_lsn, ${before}::pg_lsn) as d`.execute(
db,
)
).rows[0].d,
);
}
describe('#492 append-persist write volume (pg_current_wal_lsn delta) [integration]', () => {
let db: Kysely<any>;
let stepRepo: AiChatRunStepRepo;
let msgRepo: AiChatMessageRepo;
let workspaceId: string;
let userId: string;
let chatId: string;
beforeAll(async () => {
db = getTestDb();
stepRepo = new AiChatRunStepRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
const seedRow = () =>
msgRepo.insert({
chatId,
workspaceId,
userId,
role: 'assistant',
content: '',
status: 'streaming',
metadata: stepMarkerMetadata(0) as never,
});
const STEPS = 40;
it('NEW per-step INSERT is O(Σ steps); OLD full-row rewrite is O(n²)', async () => {
const steps: Step[] = [];
for (let i = 0; i < STEPS; i++) steps.push(makeStep(i));
// NEW: per-step INSERT of THIS step's parts + a cheap marker UPDATE.
const newRow = await seedRow();
const newWal = await walDelta(db, async () => {
for (let i = 0; i < STEPS; i++) {
await stepRepo.insertStep(
newRow.id,
workspaceId,
i,
assistantParts([steps[i]], ''),
);
await msgRepo.update(
newRow.id,
workspaceId,
{ metadata: stepMarkerMetadata(i + 1) },
{ onlyIfStreaming: true },
);
}
});
// OLD (the pre-#492 revert): re-persist the GROWING metadata.parts on the
// message row on every step.
const oldRow = await seedRow();
const oldWal = await walDelta(db, async () => {
const acc: Step[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(steps[i]);
await msgRepo.update(
oldRow.id,
workspaceId,
flushAssistant(acc as never, '', 'streaming'),
{ onlyIfStreaming: true },
);
}
});
// eslint-disable-next-line no-console
console.log(
`[#492 WAL] ${STEPS} steps ×100KB: new=${(newWal / 1e6).toFixed(1)}MB ` +
`old=${(oldWal / 1e6).toFixed(1)}MB (${(oldWal / newWal).toFixed(
1,
)}x smaller)`,
);
// O(Σ steps): ~STEPS × (100KB output + marker) of WAL. 40 × ~100KB parts plus
// 40 tiny markers is a few tens of MB at most — bounded, linear in step count.
expect(newWal).toBeLessThan(30_000_000);
// O(n²): step k rewrites ~k × 100KB. Σ over 40 steps ≈ 80+ MB — far larger.
expect(oldWal).toBeGreaterThan(30_000_000);
// The load-bearing claim: the new path writes a small FRACTION of the old.
expect(newWal).toBeLessThan(oldWal * 0.35);
}, 120_000);
});
@@ -0,0 +1,169 @@
import { Kysely } from 'kysely';
import { AiChatController } from 'src/core/ai-chat/ai-chat.controller';
import {
assembleStepParts,
assistantParts,
stepMarkerMetadata,
} from 'src/core/ai-chat/ai-chat.service';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import type { User, Workspace } from '@docmost/db/types/entity.types';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #492 controller hydration (crash-before-finalize RESUME) on a LIVE Postgres.
* `AiChatController.withReconstructedParts` is wired into getMessages/delta/export/
* run, but `aiChatRunStepRepo` is OPTIONAL and every controller unit spec passes it
* as `undefined`, so the hydration branch early-returns and NEVER executes in those
* tests. This drives the real read path a mid-run streaming row (marker only,
* empty inline parts) PLUS its `ai_chat_run_steps` rows through getMessages WITH
* the repo present, exercising the `role==='assistant' && !rowHasInlineParts`
* needy predicate, the workspace-scoped batch step fetch, and the endpoint binding.
*/
describe('#492 controller hydration read path [integration]', () => {
let db: Kysely<any>;
let aiChatRepo: AiChatRepo;
let msgRepo: AiChatMessageRepo;
let stepRepo: AiChatRunStepRepo;
let workspaceId: string;
let otherWorkspaceId: string;
let userId: string;
// Build the controller WITH a real AiChatRunStepRepo injected (position 9), the
// seam the unit specs leave undefined. Only the read-path deps are real.
function buildController(): AiChatController {
return new AiChatController(
{} as any, // aiChatService
{} as any, // aiChatRunService
aiChatRepo,
msgRepo,
{} as any, // aiTranscription
{} as any, // pageRepo
undefined, // streamRegistry
undefined, // environment
stepRepo, // #492 aiChatRunStepRepo
);
}
beforeAll(async () => {
db = getTestDb();
aiChatRepo = new AiChatRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
stepRepo = new AiChatRunStepRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
otherWorkspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
it('getMessages reconstructs a mid-run row from the steps table (finished rows untouched)', async () => {
const chatId = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const user = { id: userId } as User;
const workspace = { id: workspaceId } as Workspace;
// A prior FINISHED assistant row that already carries inline parts — the needy
// predicate must SKIP it (no step fetch), returned untouched.
const finishedParts = assistantParts(
[{ text: 'done earlier', toolCalls: [], toolResults: [] } as any],
'',
);
await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: 'done earlier',
status: 'completed',
metadata: { parts: finishedParts, toolTraceVersion: 2, stepsPersisted: 1 },
createdAt: new Date(Date.now() - 3000),
});
// The mid-run row a crash-before-finalize left behind: a step marker only
// (parts:[] , content:''), status 'streaming'. Its real parts live ONLY in the
// steps table.
const midRun = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: '',
status: 'streaming',
metadata: stepMarkerMetadata(2),
createdAt: new Date(Date.now() - 1000),
});
const step0 = assistantParts(
[
{
text: 'reasoning about the page',
toolCalls: [
{ toolCallId: 'g1', toolName: 'getPage', input: { id: 'p1' } },
],
toolResults: [
{ toolCallId: 'g1', toolName: 'getPage', output: { id: 'p1', body: 'B' } },
],
} as any,
],
'',
);
const step1 = assistantParts(
[{ text: 'partial synthesis so far', toolCalls: [], toolResults: [] } as any],
'',
);
await stepRepo.insertStep(midRun.id, workspaceId, 0, step0);
await stepRepo.insertStep(midRun.id, workspaceId, 1, step1);
// Workspace-scoping guard: a step row for the SAME message id under a DIFFERENT
// workspace must NEVER leak into this workspace's reconstruction.
await stepRepo.insertStep(midRun.id, otherWorkspaceId, 99, [
{ type: 'text', text: 'FOREIGN_WORKSPACE_LEAK' },
]);
const res = await buildController().getMessages(
{ chatId } as any,
{ limit: 50 } as any,
user,
workspace,
);
const items = res.items as any[];
const finished = items.find((r) => r.status === 'completed');
const reconstructed = items.find((r) => r.id === midRun.id);
// The finished row passed through with its inline parts unchanged.
expect(finished.metadata.parts).toEqual(finishedParts);
// The mid-run row's parts were reconstructed from the two step rows, in order,
// exactly as assembleStepParts concatenates them — the client seed sees the
// persisted progress with no change to itself.
const expected = assembleStepParts([
{ stepIndex: 0, parts: step0 },
{ stepIndex: 1, parts: step1 },
] as any);
expect(reconstructed.metadata.parts).toEqual(expected);
// The foreign-workspace step row did NOT leak in.
expect(JSON.stringify(reconstructed.metadata.parts)).not.toContain(
'FOREIGN_WORKSPACE_LEAK',
);
// Sanity: reconstruction produced real content (text + the paired tool part +
// the second step's text), not an empty fallback.
expect(reconstructed.metadata.parts).toContainEqual({
type: 'text',
text: 'reasoning about the page',
});
expect(
(reconstructed.metadata.parts as any[]).some((p) => p.type === 'tool-getPage'),
).toBe(true);
}, 60_000);
});

Some files were not shown because too many files have changed in this diff Show More