Compare commits

...

63 Commits

Author SHA1 Message Date
agent_coder 1644de87f7 fix(#370): F2 terminal skip reply for empty/missing page + F1 lifecycle tests
Reviewer changes-requested on #565.

F2 (real bug): a save on an empty page (or a missing page row) hung the client
for the full 20s ack window and then falsely reported the collab server as
unreachable + advised retry. Root cause: handleSaveVersion gated its
broadcastStateless on `if (result)`, and `result` stays undefined on the two
reachable early returns (isEmptyParagraphDoc and !page), so the server sent
NOTHING and the client waited out its timeout.
- SERVER (persistence.extension.ts): both early-return branches now record a
  skip reason and the tail broadcasts a terminal `version.skipped` reply
  (reason 'empty' | 'page-not-found'). Exactly one terminal reply per handled
  save. Added VERSION_SAVED/VERSION_SKIPPED message consts.
- CLIENT (collaboration.ts): the predicate now matches both version.saved and
  version.skipped; SaveVersionResult gains {saved, skipped, reason}. An empty
  skip resolves to a clean {saved:false, skipped:true, reason:'empty'} (no
  stall); page-not-found throws an immediate, truthful error; the genuine
  no-reply timeout path is unchanged (that is the real "unreachable" case). A
  terminal reply leaves the session cached (healthy connection); only a
  transport failure destroys it. Kept the message-literal in-sync comments.
- tool-specs.ts / pages.ts: description + docstring reflect the new result shape.

F1 (coverage): added tests for the 4 previously-uncovered sendStatelessAndAwait
branches — unrelated-then-real (predicate filters noise), teardown-mid-wait
(rejects + removes the stateless listener, no leak), concurrent-in-flight
guard — plus the F2 empty-skip / page-not-found client outcomes. Server spec
gains empty-page and page-not-found terminal-reply tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:23:07 +03:00
agent_coder 498a87f3c4 fix(#370): add savePageVersion to DocmostClientMethod union (server in-app tool parity)
The Stage-A save_page_version tool's client method must be listed in the
DocmostClientMethod union so Pick<DocmostClient, DocmostClientMethod> exposes it
to the in-app tool adapter; the shared-tool-specs contract spec asserts parity.
Mirrors the existing restorePageVersion entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:23:07 +03:00
agent_coder 67c94de6ef feat(mcp): agent save_page_version tool — collab stateless transport + role prompts (#370)
Stage A of #370: wire the MCP client transport for the already-shipped server
save-version handler (PR-1 #374). An agent can now pin an intentional named
version (kind='agent', derived server-side from the signed actor) of a page's
CURRENT live collaboration content.

- collab-session.ts: add CollabSession.sendStatelessAndAwait(payload, predicate,
  timeoutMs) — sends a stateless message over the live provider and resolves on
  the first matching reply, with a bounded timeout; lifecycle (inflightReject,
  ready-guard, concurrent fail-fast, idle re-arm) mirrors mutate(). Extend
  CollabProviderLike with sendStateless + the stateless on/off overloads.
- collaboration.ts: add savePageVersionRealtime() — under withPageLock, reuse the
  cached agent-authenticated CollabSession (#400), send {type:'save-version'},
  await {type:'version.saved', …}; no REST read (would race the stale page row).
- client/pages.ts: add savePageVersion(pageId) — resolvePageId +
  getCollabTokenWithReauth + writeWithCollabAuthRetry (#486 self-heal).
- tool-specs.ts: add the deferred savePageVersion spec (+ DocmostClientLike Pick);
  auto-registered on both hosts by the shared loops.
- server-instructions.ts: HISTORY family + routing-prose mention.
- ai-chat-tools.service.ts: contract type-assert for the new client method.
- agent-roles-catalog: tell the content-authoring roles (researcher,
  call-summarizer, ru+en) to save a version when the document is done; bump
  their catalog versions.
- test: unit coverage for savePageVersionRealtime (ack + bounded-timeout paths).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:23:07 +03:00
agent_vscode 6b825ad440 fix(client): anchor toasts to the top edge of the viewport
Move the top-anchored Mantine notification containers from top:96px to
top:8px so success toasts (e.g. "Comment resolved successfully") appear
at the very top of the page, above the header/search, instead of
covering page content. Update the accompanying comment to reflect the
new intent (toast renders over the header chrome while visible).
2026-07-12 19:09:30 +03:00
agent_vscode 0cee2cc9dc Merge remote-tracking branch 'gitea/develop' into develop 2026-07-12 19:09:13 +03:00
agent_vscode e4af672c2c feat(ui): add UI components for comments, history, and time
Introduce redesigned UI elements for the new design system:
- **CommentsPanel** – displays agent‑driven edits with diff view, batch actions, and filtering.
- **PageHistoryModal** – full‑screen modal for browsing page revisions, includes a mini‑calendar and version rendering.
- **TimeWorkedModal** – visualizes work and agent sessions per day with grid or phase axes.

These components replace legacy panels, use Mantine v7, support dark/light themes, and provide richer interactions.
2026-07-12 19:09:03 +03:00
vvzvlad ec8dd7d110 Merge pull request 'fix(prosemirror-markdown): литеральный <br> в тексте экранируется, не становится hardBreak (#554)' (#560) from fix/554-literal-br into develop
Reviewed-on: #560
2026-07-12 18:57:02 +03:00
vvzvlad 4423b19850 Merge pull request 'fix(ai-chat): реактивная рекавери — итеративная эскалация 0.5^k + пол вместо фикс. 0.5× (#520)' (#546) from fix/520-recovery-escalation into develop
Reviewed-on: #546
2026-07-12 18:56:49 +03:00
agent_coder 688cb54f26 fix(ai-chat): рекавери режет реплей НИЖЕ настроенного бюджета (#520 Опция B)
Решение владельца по #520 (эпик #497 итерация 5): при повторном overflow реактивная
рекавери ДОЛЖНА иметь право резать бюджет реплея ниже явно настроенного окна —
«чат не должен кирпичиться на overflow» это ИНВАРИАНТ реактивной ветки, а
настроенное окно — заявление о ёмкости модели, не обещание о размере реплея; когда
провайдер 400-ит, реальность бьёт конфиг. (Не конфликтует с #510 Опция A: та про
верхнюю границу НОРМАЛЬНОГО пути — уважать конфиг, пока он влезает.)

- Пол эскалации: max(scaled, min(REPLAY_MIN_FLOOR_TOKENS, floor(0.5×threshold))).
  Было min(REPLAY_MIN_FLOOR_TOKENS, threshold) — жёсткий пол на настроенном бюджете.
  Теперь БОЛЬШОЕ окно эскалирует НИЖЕ настроенного бюджета до фикс. пола 8k; МАЛОЕ
  окно (0.5×threshold < 8k) падает до floor(0.5×threshold) — минимум старый 0.5× cut
  (никогда не хуже, чем раньше), но и не поднимается выше самого бюджета.
  REPLAY_MIN_FLOOR_TOKENS=8k без изменений (константа уже была). Сброс k на чистом
  ходу сохранён (счётчик replayOverflowCount не пишется на чистом финализе).
- Наблюдаемость: warn-лог (настроенный бюджет не влезает, реплей ниже него, уровень
  k) + метка турна metadata.replayBelowConfiguredBudget=true — только когда реплей
  реально ниже настроенного бюджета (не на нормальном in-budget реплее).
- Тесты: перепиновка «не раздувает малый бюджет выше себя» (теперь МОЖЕТ резать ниже,
  Опция B) в обоих spec; новый тест эскалации ниже бюджета (окно 8000→бюджет 5600,
  k=1→2800, сходимость к полу, сброс на чистом ходу) + большое окно (140k→8k
  ступенями); тест метки метадаты. Мутации: пол=бюджет краснит below-budget тесты,
  снятие метки краснит observability-тест.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 18:01:07 +03:00
agent_coder ec5416068b fix(ai-chat): итеративная эскалация реактивной рекавери при overflow (#520)
Остаточный кирпич (#490/#510): при незаданном chatContextWindow база = плоский
дефолт 100k, а реальное окно модели маленькое (<50k). Фиксированный одинарный
cut 0.5×100k=50k всё равно превышал реальное окно → провайдер снова 400,
строка переставлялась replayOverflow, но булев priorOverflowed уже был true →
второго ужатия не происходило. Чат навсегда застревал на 50k и не восстанавливался.

Фикс (без парсинга тел 400):
- Сигнал из булева переведён в счётчик `metadata.replayOverflowCount` = число
  ПОДРЯД идущих overflow-ходов: инкремент (prior+1) на каждом overflow, сброс в 0
  на любом чистом финализе (чистая строка не пишет поле → читается как 0).
  BACK-COMPAT: старая строка с булевым `replayOverflow:true` читается как k=1.
- resolveEffectiveReplayThreshold(threshold, k) = max(floor(threshold·0.5**k),
  min(REPLAY_MIN_FLOOR_TOKENS, threshold)). k=0 → база; k=1 → 0.5×; k=2 → 0.25×;
  большой k → упирается в пол 8k (сходимость). null-база (trimming OFF) не трогается;
  пол никогда не поднимает легитимно малый настроенный бюджет выше него самого.
- Пол REPLAY_MIN_FLOOR_TOKENS=8k: ниже него чат не несёт осмысленный недавний
  контекст, и даже малое реальное окно его вмещает; keep-recent-turns сверху.

Тесты: таблица эскалации (k=0/1/2/большой/null), регрессия остаточного кирпича
(база 100k, окно ~40k → сходится ниже 40k, чего фиксированный 0.5× никогда не мог),
жизненный цикл счётчика на реальном pg (инкремент/сброс/back-compat через jsonb),
мутация **k→**1 краснит convergence-тест.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 18:01:07 +03:00
agent_coder 2c4fc565b6 fix(prosemirror-markdown): escape literal <br> in text so it round-trips as text, not a hardBreak (#554)
A literal inline-HTML break tag typed as prose text (`<br>`, `<br/>`,
`<br />`) was emitted verbatim into markdown, so on re-import marked
parsed it as an inline-HTML line break and silently turned the user's
text into a hardBreak node, dropping the text. HTML-entity-encode only
the angle brackets of a break-tag sequence in `case "text"` so it lands
as `&lt;br&gt;` and the importer decodes it back to literal `<br>`.

Scoped strictly to the `<br…>` pattern (not every `<`/`>`), so stray
angle brackets in prose (`a < b > c`) are untouched, and to the
text-content path only — a real hardBreak serializes from its own case
(`  \n`, or `<br>` via inlineToHtml), so the serializer's own emitted
breaks are never escaped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:58:21 +03:00
vvzvlad a990ebd604 Merge pull request 'feat(search) Фаза A: лексический оверхол — ru_en, OR/RRF, операторы, пагинация (#529)' (#538) from feat/529-search-lexical into develop
Reviewed-on: #538
2026-07-12 17:40:43 +03:00
vvzvlad 94ca907476 Merge pull request 'fix(prosemirror-markdown): hardBreak переживает round-trip (trailing/consecutive/table-cell) (#549)' (#553) from fix/549-hardbreak-md-canon into develop
Reviewed-on: #553
2026-07-12 17:40:18 +03:00
agent_coder fff772cbe2 fix(search): F1 term-cap stack-depth guard + restore/extend coverage, drop dead code (#529)
F1: cap parsed terms at MAX_PARSED_TERMS (64) in parseSearchQuery so a huge
pasted query no longer nests the combined tsquery deep enough to blow Postgres'
stack depth limit (HTTP 500); +@MaxLength(10000) on SearchDTO.query as
defense-in-depth. F2/F3: restore titleOnly text_content leak-guard and
parentPageId subtree-scoping coverage in the lexical int-spec. F4: positive
ancestor-path ordering + non-empty snippet asserts. F5: remove dead
SearchLookupTier enum, unused buildTsQuery + pg-tsquery require (and its tests),
and the discarded parentPageId select in fetchDetails. F6: rewrite the stale
#443 DTO comment (parentPageId/titleOnly read by the engine; substring ignored).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 16c2a4b623 fix(search): whitelist #529 search response-fields in the #494 prose drift-guard
The rebase pulled develop's #494 REVERSE prose drift-guard, which flags every
camelCase token in ROUTING_PROSE that is not a registered MCP tool. #529's search
routing prose documents the search RESPONSE fields matchedTerms/matchedFields/
hasMore/truncatedAtCap — genuine non-tool terms — so add them to
PROSE_NON_TOOL_TERMS (develop's designed allowlist). Reconciles both invariants:
#529's search-prose enforcement and #494's dead-reference guard both stay green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 58f13a7efd chore(search): retimestamp ru_en migration to 20260707T130000 to avoid collision (#529)
Develop introduced two migrations at the exact 20260707T120000 stamp
(ai-chat-metadata, page-history-kind). Three files sharing one timestamp
is fragile for deploy ordering, so move #529's ru_en-config migration to a
strictly-later, unique 20260707T130000 so it orders cleanly last. It touches
only the ru_en text-search config (different tables), so running it after the
develop migrations is order-independent and safe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 2bb48f841f fix(search): S1/W1/W3/W4/S3/S5/S2 — required-term details, CAP note, restored guards (#529)
S1: fetchDetails now builds rank/highlight/matchedFields from the same
[...positive, ...required] set matchedTerms uses, so a required-only query
(`+кофейня`) no longer reports matchedFields:[] / rank:null.

W1: documented that CANDIDATE_CAP bounds pagination reachability (JS-side), not
query compute — a SQL LIMIT would corrupt the exact permission-filtered total.

S2: one-line note that literal-`%`/`_` search was intentionally dropped (total:0).

W3: restored the trgm-index EXPLAIN guard (search-lookup-explain.int-spec) that
asserts the coalesce-free substring predicates use idx_pages_*_trgm, not Seq Scan.

W4: added an integration ordering test proving title-exact > title-substring >
text-only tier dominance under RRF.

S3: added a test that an exact-title hit survives a tiny CANDIDATE_CAP window.

S5: server pretest now also builds @docmost/mcp so the ToolWriteClass import
resolves from clean CI (mirrors editor-ext/prosemirror-markdown).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder daa96eb132 fix(search): B1/W2 — honest page_embeddings.fts swap + true idempotency (#529)
BLOCKER B1: the fts config swap no longer blindly DROP+ADDs the generated
column. swapEmbeddingsFtsConfig now (1) reads the column's actual generation
expression from pg_catalog and TRUE-no-ops when it already references the target
config (real out-of-band escape hatch), and (2) gates the inline ACCESS
EXCLUSIVE rewrite behind SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE (default 'true';
only a literal 'false' opts out), warning that the operator owns the swap.
up() now creates ru_en only when missing (drop-recreate would fail on the fts
hard dependency on a re-run), so up() is genuinely idempotent. down() guards the
config DROP when fts still references ru_en (gated-off path), staying non-fatal.

W2: corrected the header — Kysely runs EACH migration in its own transaction
(not the whole set in ONE); the single-UPDATE atomicity conclusion is unchanged.
Rewrote the runbook: inline path is a full-table ACCESS EXCLUSIVE rewrite of
page_embeddings; removed the false "IF-EXISTS no-op" claim.

Extends the roundtrip test with idempotent-skip (2nd up() no-op) and env-gate
=false (embeddings rewrite skipped, pages.tsv still swapped) assertions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 4c5230d677 test(search): A1 migration down()/up() reversal roundtrip (#529)
Asserts the ru_en migration is reversible in the correct order: down()
reverts pages.tsv trigger + page_embeddings.fts to english BEFORE dropping
the ru_en config, and up() re-applies it idempotently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder 50ca27c5d4 feat(search): A9 web-UI — superset types, match/pagination params, operator hint (#529)
- search.types.ts: IPageSearch is the #529 superset (pageId/snippet/score/
  path/matchedFields/matchedTerms; rank/highlight nullable); add the
  IPageSearchResponse pagination envelope and match/limit/offset params.
- search-spotlight.tsx: operator hint ("exact phrase", +required, -excluded).

The web list already reads response.items, so it picks up the new OR order +
superset with no code change; icon/space/highlight remain (acceptance #12).
Full spotlight pagination UI is deferred within this PR — the API + MCP fully
support total/hasMore/offset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
agent_coder de25c258f9 feat(search): A9 MCP — operators, match, pagination in the search tool (#529)
- client/read.ts search(): forward match + offset; surface the pagination
  envelope (total/hasMore/truncatedAtCap/offset) when the #529 server
  returns it (a stock upstream leaves them undefined).
- index.ts search tool: document OR-default + RU/EN morphology, the
  "phrase"/+/- operators, match modes, limit+offset pagination and the
  relevance-CAP unreachable-tail caveat; add match + offset params.
- server-instructions.ts: READ prose + inventory line updated to the new
  contract, enforced by a new tool-inventory.test.mjs assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:37:07 +03:00
vvzvlad afc50ead38 Merge pull request 'feat(deploy): version-coherence — WS-анонс версии + guarded reload устаревших вкладок (#481)' (#499) from feat/481-version-coherence into develop
Reviewed-on: #499
2026-07-12 17:37:04 +03:00
vvzvlad ec932e89a3 Merge pull request 'feat(page-history): время работы над статьёй — число + суточный punch-card (#395)' (#551) from feat/395-work-time into develop
Reviewed-on: #551
2026-07-12 17:36:27 +03:00
vvzvlad 490d7965a1 Merge pull request 'feat(comment): Undo в тосте резолва (reopen) + сортировка Resolved по времени резолва (#542)' (#548) from feat/542-resolve-undo into develop
Reviewed-on: #548
2026-07-12 17:36:08 +03:00
vvzvlad 32c9361969 Merge pull request 'feat(mcp): буквальная запись markdown (без math/autolink) + getPage format:text (#502)' (#552) from feat/502-mcp-md-modes into develop
Reviewed-on: #552
2026-07-12 17:35:36 +03:00
vvzvlad c30f910d66 Merge pull request 'fix(prosemirror-markdown): markdown-импорт помечает внутренние ссылки на страницы как internal (#522)' (#524) from fix/522-internal-links into develop
Reviewed-on: #524
2026-07-12 17:35:28 +03:00
agent_coder 7007f6bcf9 feat(search): A2-A9 — unified OR-relevance engine, RRF, pagination (#529)
Rewrite SearchService.searchPage into ONE engine for web-UI, MCP agent and
share:

- A2 server-side query parser (search-query-parser.ts): quote-aware
  tokenizer, leading +/- operators (internal -,.,: literal), "phrase",
  metachar-stripped bare terms; tsquery built as a parameterized AST via
  SQL ||/&&/!! (never string-concat). only-negation/empty short-circuit.
- A3 match=auto routes identifier-like terms (10.31.41, esp32,
  WB-MGE-30D86B) to the substring/trigram branch, words to FTS; word/
  prefix/substring overrides.
- A4 RRF (k=60) fuses the FTS branch (ts_rank_cd) and substring branch
  (title-exact>title-sub>text tier) by RANK; ORDER BY rrf DESC, id.
- A5 exact permission-filtered total (fail-closed via filterAccessiblePageIds
  with the #348 hasRestricted fast-path), CANDIDATE_CAP fusion window,
  offset/limit, hasMore, truncatedAtCap.
- A6 single path (spaceId/share/creatorId/titleOnly/parentPageId/match);
  share uses getPageAndDescendantsExcludingRestricted.
- A7 response superset per hit (id/pageId/slugId/icon/title/space/…/rank/
  highlight/snippet/path/score/matchedFields/matchedTerms).
- A8 buildAncestorPaths now skips deleted + cross-space ancestors.
- A9 DTOs: match, offset, total/hasMore/truncatedAtCap/query/matchedFields.

SEARCH_MODE=or|and toggles the parser; SEARCH_CANDIDATE_CAP tunes the
window. Legacy lookup unit/int specs replaced by parser unit tests + a
13-criteria integration spec on real pg (incl. a permission mutation guard
and a fail-closed propagation test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:35:24 +03:00
agent_coder 4bf06c68cf feat(search): A1 — ru_en text-search configuration + config swap (#529)
Introduce a `ru_en` FTS configuration (english_stem over ascii token
classes, russian_stem over Cyrillic ones) and flip every stored + query
side to it IN LOCKSTEP (acceptance #13):

- new migration creates ru_en, swaps the pages.tsv trigger + reindexes
  existing rows (row-lock UPDATE, no ACCESS EXCLUSIVE), and swaps the
  page_embeddings.fts generated column (documented rewrite/lock trade-off
  for large tenants, mirroring the #443 trgm migration). down() reverts
  tsv/fts to english BEFORE dropping the config (dependency order).
- page-embedding.repo.ts hybridSearch query config english -> ru_en, so
  the RAG lexical leg's query config matches its fts column config.
- search.service.ts current query literals english -> ru_en so the column
  and query configs stay paired (this commit is independently revertable;
  the engine itself is rewritten in the A2-A9 commit).

The reindex is atomic within the single migration transaction (this repo's
Migrator wraps all pending migrations in one tx), so no morphology-desync
window exists and no dual-config read path is needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 17:35:24 +03:00
vvzvlad af44736fb9 Merge pull request 'fix(mcp): drawio — авто-стрип XML-комментариев (гейт на well-formedness) вместо lint-ошибки (#505)' (#545) from fix/505-drawio-strip-comments into develop
Reviewed-on: #545
2026-07-12 17:35:01 +03:00
vvzvlad 2a4d1acfd7 Merge pull request 'fix(prosemirror-markdown): insertNode дописывает пункт в существующий список вместо второго списка (#535)' (#537) from fix/535-list-seam-coalesce into develop
Reviewed-on: #537
2026-07-12 17:34:41 +03:00
vvzvlad 11eb87d58a Merge pull request 'docs(server): getPageBreadCrumbs — предки не утекают (нисходящее наследование ограничений), задокументировано (#471)' (#550) from docs/471-breadcrumb-permission into develop
Reviewed-on: #550
2026-07-12 17:33:55 +03:00
vvzvlad f2c8aa70f3 Merge pull request 'fix(mcp): внятная ошибка на недоступный spaceId вместо «Space permissions not found» (#534)' (#536) from fix/534-mcp-spaceid into develop
Reviewed-on: #536
2026-07-12 17:33:39 +03:00
vvzvlad 661ea8ba07 Merge pull request 'fix(client): reconnect-вход — таймаут-гонка вокруг getRun, hang не залипает FSM в streaming (#541)' (#543) from fix/541-reconnect-timeout into develop
Reviewed-on: #543
2026-07-12 17:32:07 +03:00
vvzvlad 059edccb64 Merge pull request 'test(mcp): hardBreak уже представим в md-каноне (#293) — регресс-гард, закрытие #503' (#544) from fix/503-hardbreak-canon into develop
Reviewed-on: #544
2026-07-12 17:31:55 +03:00
vvzvlad 693a9a350b Merge pull request 'perf(ai-chat) волна C: инкрементальный рендер стрима + append-персист таблица шагов (#492)' (#547) from feat/492-incremental-render into develop
Reviewed-on: #547
2026-07-12 17:31:40 +03:00
agent_coder 79b2da686b fix(work-time): clip cross-class padding; gate & payload fixes (#395)
F1: pad-clip adjacent different-class sessions at the raw-gap midpoint so
workMs and agentOnlyMs can never double-count the same wall-clock (default
agentTGap 7m < pIn+pOut 10m made a work-session-ending-in-agent overlap a
nearby agent_only run). Reword the config guard/comment: cross-class
disjointness is now structural, tGap≥pIn+pOut is a kept sanity bound. Add a
cross-class no-double-count test and a real seeded property/fuzz test (250
random timelines × 4 tz) asserting per-class union, cross-class disjointness,
and Σ per-day activeMs == workMs.

F2: page.controller.spec — add a ForbiddenException view-gate reject test that
asserts the rejection propagates and computeWorkTime is NOT reached, locking
validateCanView before compute.

F3: WorkTimeStat renders for agent-only pages (workMs==0, agentOnlyMs>0) with
an `agent:` headline so the punch-card stays reachable.

F4: drop the dead un-bucketed `sessions` list from the PageWorkTime response
and the client work-time types (no client reads it; bucketByDay still consumes
the core sessions internally).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 08:31:42 +03:00
agent_coder ed3a8f8174 docs(mcp): sync write-tool specs with #502 literal-markdown behavior (#502 review)
The four MCP markdown-write tools (updatePageMarkdown/createPage/patchNode/
insertNode) still described the old parse-everything behavior after #502 turned
TeX math and fuzzy autolink OFF on the write paths. Add a concise contract hint
to each: $...$ / $$...$$ stay literal (not a math formula) and schemeless
www.host / bare emails are not auto-linked (explicit https:// still links); for
a real formula use updatePageJson (or a mathInline/mathBlock node via `node` on
patch/insert). Also note getPage format:"text" in README.md/README.ru.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:57:26 +03:00
agent_coder 8503ff1f3d fix(ai-chat): hydrate crashed mid-run steps for model replay + cover write/read seams (#492)
F1: the server model-replay loaded history via findAllByChat().map(rowToUiMessage)
WITHOUT hydrating parts from ai_chat_run_steps. A HARD crash mid-run (SIGKILL/OOM)
fires no terminal callback, so the assistant row stays parts:[] and its partial
tool-calls/results/text (durable in the steps table) dropped out of the model's
next-turn context. Hydrate needy assistant rows (role==='assistant' &&
!rowHasInlineParts) via findByMessageIds + hydrateAssistantParts before the replay
map — mirroring the controller's withReconstructedParts exactly — guarded on the
optional repo. Fix the now-false interrupt-resume comment.

F2: add a service int-spec that drives the REAL onStep append-persist WRITE branch
through AiChatService.stream with a real AiChatRunStepRepo injected, asserting the
per-step rows' stepIndex + parts slice and the step-marker metadata match a
single-row flush (catches an stepsPersisted-1 off-by-one).

F3: add a controller int-spec that drives withReconstructedParts through getMessages
WITH the repo present (a mid-run marker-only row + its step rows), asserting the
reconstructed metadata.parts and workspace-scoping.

F4: remove the dead countByMessage (zero prod callers; reconstructRunParts derives
stepsPersisted inline) + its now-unused sql import and the redundant test assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:50:50 +03:00
agent_coder c18b4c132c fix(md-canon): preserve hardBreak in trailing/consecutive/table-cell round-trips (#549)
The markdown canon lost `hardBreak` nodes in three pm->md->pm sub-cases where
the two-space `  \n` form does not survive re-import via marked:

1. Trailing at end of a paragraph — the top-level `.trim()` strips the trailing
   `  \n`, and even a non-last paragraph's trailing break sits before the `\n\n`
   block gap, so marked drops it.
2. Consecutive breaks (`a<hardBreak><hardBreak>b`) — the whitespace-only middle
   line reads as a paragraph separator, splitting the run into two paragraphs.
3. Inside a GFM table cell — the single-line cell collapses the break to a space.

Emit `<br>` (inline HTML break) in exactly those positions: marked passes it
through and generateJSON rebuilds a hardBreak in every context, including table
cells. renderInlineChildren now picks `<br>` for a break that is the last inline
child or is followed by another break; a safe mid-paragraph break keeps the
byte-stable `  \n` form (golden output unchanged). Footnote bodies are skipped
(their `^[…]` form already collapses breaks to spaces). The table-cell case
converts the `  \n` marker to `<br>` before the newline collapse.

Adds pm->md->pm count/position round-trip tests for all three sub-cases plus a
mid-paragraph regression guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 07:10:40 +03:00
agent_coder 3163f50c98 fix(mcp): route #502 extension flags per write tool; add server import flag (#502 review)
Internal review found the two layered-extension flags landed in the SHARED
markdownToProseMirrorCanonical wrapper, which mis-covered two tools. Move the
decision to each caller by the tool's semantics.

BLOCKER 1 — createPage was NOT covered. createPage POSTs to the server
/pages/import endpoint, which imported with DEFAULTS (math + fuzzy autolink ON),
so an agent's `$x=1$` still became a formula and `www.host` still autolinked.
Add an optional `disableMarkdownExtensions` multipart field to /pages/import
(default OFF, so human file uploads keep math ON); import.controller reads it,
import.service.importPage/processMarkdown thread it into markdownToProseMirror.
MCP createPage sends it true.

BLOCKER 2 — import_page_markdown was wrongly disabled. It goes through the same
wrapper, so hardcoding parseMath:false degraded an exported `$x^2$` to literal
text on re-import, breaking the #328 lossless export→import pair. The wrapper no
longer hardcodes the flags: it takes them from the caller and DEFAULTS to the
package importer defaults (extensions ON). Callers now set them explicitly:
  - updatePageMarkdown (updatePageContentRealtime) -> OFF
  - patch_node/insert_node (importMarkdownFragment) -> OFF (unchanged)
  - import_page_markdown -> DEFAULTS (math ON) — #328 round-trip restored

Tests: rewrite the mcp write test around the corrected per-caller semantics
(agent-write OFF, import_page_markdown DEFAULTS incl. the REAL #328 round-trip);
add a collab-backed wiring test driving client.updatePage (OFF) and
client.importPageMarkdown (DEFAULTS) end-to-end; add a createPage multipart-flag
test; add a server import.service.processMarkdown spec (flag OFF -> literal / no
autolink, default -> math ON for human uploads); reword the prosemirror-markdown
round-trip test to its package-default scope. Mutation-verified both caller
wirings (flip updatePageMarkdown -> defaults reddens the OFF wiring test; make
the wrapper default OFF reddens the #328 round-trip).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:58:48 +03:00
agent_coder 3e81f64415 fix(work-time): user idle breaks agent burst; cap tz length (#395)
Only an agent-sourced idle pulse extends an agent burst; a user idle
(human supervision) now falls through to the human branch so the session
is classified `work`, not `agent_only`. Add @MaxLength(64) to tz to match
its comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:53:59 +03:00
agent_coder d36021a111 fix(comment): clear resolved mark only after reopen confirmed
Move the inline ProseMirror comment-mark clear out of the Undo toast's
onClick and into the reopen mutation's onSuccess branch. Clearing it
eagerly flipped the collab mark to unresolved before the reopen result
was known: on a non-404 failure the RQ cache rolls back to resolved but
the doc kept an active highlight the panel treats as resolved, and a
comment refetch can't heal a divergence that lives in the collab doc.
Mirrors the 404 branch's editor-liveness guard/try-catch and the safe
await-then-mark pattern in comment-list-item. The button-triggered
reopen already sets the mark, so the onSuccess call is an idempotent
no-op there.

Tests: reopen failure (500) leaves the mark untouched; null editorRef
on the success path degrades gracefully.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:51:38 +03:00
agent_coder 1f2999e5ad fix(mcp): drawio — авто-стрип XML-комментариев вместо lint-ошибки (#505)
Модель органически вставляет `<!-- ... -->` в mxGraph-XML вопреки запрету
в описании тула (природа LLM, промптингом не лечится). Жёсткая ошибка
линтера [no-comments] заставляла её ПОЛНОСТЬЮ перегенерировать диаграмму —
впустую потраченный tool-call на почти каждой первой генерации.

Теперь в общем prepare-пути (`prepareModel`, через который идут
drawioCreate/drawioUpdate/drawioEditCells) комментарии срезаются ДО линта:
`xml.replace(/<!--[\s\S]*?-->/g, "")`. Комментарии не несут семантики, так
что стрип безопасен. Число срезанных уходит в `warnings[]` как
`stripped N XML comment(s)` (только при N > 0) — модель это видит, но НЕ
ретраит.

Стрип ГЕЙТИТСЯ двумя условиями, чтобы регэксп бил только по настоящим
comment-нодам:
  1. well-formedness — сначала парсим модель; на malformed-входе (сырой
     неэкранированный `<!--` внутри значения атрибута на пути create/update,
     где normalizeInput отдаёт строку без парсинга) стрип НЕ выполняется,
     иначе он молча вырезал бы текст автора. Вход отклоняют правила
     value-escaping / well-formed-xml, автор видит реальную ошибку.
  2. отсутствие CDATA — CDATA-секция well-formed, но её текст может содержать
     литеральный `<!-- ... -->`, который НЕ является comment-нодой; стрип
     молча удалил бы контент автора. Настоящий drawio держит подписи в
     атрибутах `value=` (CDATA невозможен), так что исключение CDATA бесплатно
     на легитимных моделях; CDATA-модель с реальным комментарием падает на
     сохранённый backstop no-comments (явная ошибка), а не тихо портится.

Правило `no-comments` в линтере оставлено как defense-in-depth backstop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:43:03 +03:00
agent_coder c674db2b2f feat(page-history): «время работы над статьёй» — число в UI + суточный punch-card (#395)
Оценка времени работы над страницей как сессионизация истории page_history по
паузам бездействия (WakaTime-подобно), а не span между крайними правками, плюс
drill-down в суточный таймлайн 24ч×дни. Зависит от #374 (kind + idle-пульс).

Server:
- Чистая computeWorkTime(rows, config) (§5): нормализация+дедуп → коллапс
  агентских всплесков по aiChatId → ОДИН проход сессионизации (порог зависит от
  пары: оба agent → agentTGap, иначе tGap; последняя сессия обязательно
  закрывается) → класс готовой сессии (все agent → agent_only, иначе work) →
  добивка (много-сэмпл → [first−P_in, last+P_out], одиночный скаляр →
  [t−P_single, t]) → метрики = union wall-clock по классу. Детерминированная,
  без БД.
- Чистая bucketByDay(sessions, tz) (§6.3): union work/agent_only РАЗДЕЛЬНО, разрез
  по полуночи tz через нативный Intl (DST-корректно, без «+24ч»). Инвариант
  Σ activeMs == workMs держится по построению.
- PageHistoryRepo.findTimelineByPageId — лёгкая проекция без content, ASC.
- PageHistoryService.computeWorkTime + POST /pages/history/time, гейт
  validateCanView как у /history; атрибуция человек/агент по lastUpdatedSource.

Client:
- usePageWorkTime(pageId) шлёт tz зрителя (Intl локаль).
- Кликабельное число «≈ 4 ч 30 мин» в шапке страницы (порог в тултипе).
- Модалка-punch-card: 24-часовые дорожки по дням, окна work/agent разным цветом,
  одиночные (P_single) приглушены, сумма за день, пустой день «—», сворачивание
  длинных серий пустых дней, подпись tz + T_gap. i18n ru-RU + en-US.

Tests: 22 юнита на computeWorkTime/bucketByDay (§7-фикстура ≈1ч32м vs ≈60h наив,
закрытие последней сессии, коллапс/разрыв всплеска, idle-пульс, DST 23/25ч,
полуночный разрез, инвариант, union-без-задвоения); 5 юнитов гейта контроллера;
int-spec на реальном pg (проекция + совпадение с ядром); 6 клиентских на формат.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:32:12 +03:00
agent_coder 44cdb5a4c3 feat(mcp): disable math+autolink in MCP markdown-write; getPage format:text for machine diff (#502)
WRITE: parameterize the canonical importer markdownToProseMirror with
{parseMath, fuzzyLinkify} (defaults true — editor/file-import/git-sync unchanged);
MCP write paths (createPage/updatePageMarkdown/patchNode/insertNode) call with both
false so $...$ stays literal text and bare domains don't autolink (explicit
https:// still links). READ: getPage format:'text' reuses the server jsonToText path
(deterministic flag) for flat machine-diffable text, [image]/[table RxC] placeholders.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:18:31 +03:00
agent_coder 03af65dd06 Merge remote-tracking branch 'gitea/develop' into fix/503-hardbreak-canon 2026-07-12 06:12:53 +03:00
agent_coder dead5fa8a8 test(#503): guard hardBreak representability through the MCP canon
Issue #503 reported a hardBreak sent via updatePageJson being «silently
cut» — no error, no line break — forcing the agent to split lines into
separate paragraphs.

A faithful repro shows the bug does NOT reproduce against develop:
hardBreak is fully representable in the markdown canon in BOTH directions
and survives the real Yjs write path.

- pm -> md: the serializer (now in @docmost/prosemirror-markdown after the
  #293 STEP 5 consolidation) emits the CommonMark hard break `  \n`, not a
  bare `\n`.
- md -> pm: marked tokenizes `  \n` to `<br>`, which generateJSON parses
  back to a hardBreak node in ONE paragraph (not split, not dropped).
- updatePageJson (PM JSON -> applyDocToFragment = PMNode.fromJSON +
  updateYFragment, the real collab-session write encoder -> read back)
  keeps text + hardBreak + text in one paragraph.

The issue's diagnosis points at packages/mcp/.../markdown-converter.ts as
the serializer, but that file is a 15-line re-export shim since #293 STEP 5
— the diagnosis predates the converter consolidation that already closed
both sides. P1 (semantic round-trip) + P2 (byte-fixpoint) hold, and the
node already has broad property/corpus/golden coverage in the package.

No converter change is warranted (switching the break form to `\`+newline
would churn the whole #351 byte-fixpoint corpus against the established
`  \n` repo convention for zero functional gain). This adds one
integration guard at the MCP canon seam covering all three seams of the
reported updatePageJson path. Mutation-verified: neutering the serializer
arm or the importer `<br>` handling reddens seams 1-2; neutering the real
applyDocToFragment write path reddens seam 3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 06:04:18 +03:00
agent_coder 5b17503df7 docs(page): explain breadcrumb ancestors need no per-ancestor permission filter (#471)
getPageBreadCrumbs returns the full ancestor chain filtered only by
deletedAt; the /breadcrumbs endpoint validates validateCanView on the
target page only. Document why this is safe rather than an accepted leak:
page restrictions inherit down the tree, so viewing a page implies the
right to view every ancestor (canUserAccessPage checks the full ancestor
chain). A hidden ancestor would already hide the target itself, making
per-ancestor filtering redundant. Owner decision: document, no logic change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:58:48 +03:00
agent_coder 46bb55dbd1 Merge remote-tracking branch 'gitea/develop' into fix/534-mcp-spaceid 2026-07-12 05:58:05 +03:00
agent_coder 8b34a428f4 fix(mcp): unconditional final ERROR_MESSAGE_CAP backstop in formatSpaceNotAccessible (#536 review)
The list-only cap assumed a well-formed prefix; a pathologically long
agent-supplied spaceId (unvalidated for length) lives in the prefix and bypassed
the cap. Add the same unconditional final slice formatDocmostAxiosError uses, so
the WHOLE message is bounded regardless of spaceId length. No-op for normal
inputs (36-char UUID) where the list cap already keeps it <= 300.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:58:04 +03:00
agent_coder e236782260 fix(mcp): cap #534 space-not-accessible message + cover fail-open guards
Review follow-up (#536).

F1 (test coverage): add two tests that pin the wrapper's fail-open guards, which
previously survived mutation:
- a non-404 error (500) on a wrapped call with a spaceId propagates unchanged and
  triggers NO /spaces sweep (a real 5xx/403 must never be swallowed/reformatted);
- a 404 when the spaceId IS in the accessible index fails open (the 404 is about
  another resource), so it is not falsely rewritten to "not found among spaces".
Both mutation-verified: forcing the non-404 condition to false reddens the first;
removing the id-present guard reddens the second.

F2 (conventions): formatSpaceNotAccessible now honours ERROR_MESSAGE_CAP (300),
the same budget formatDocmostAxiosError enforces. Only the interpolated space
list is truncated (with an ellipsis); the fixed prefix (bad spaceId) and suffix
(listSpaces pointer) are always kept, so the actionable parts survive. Test: 10
long space names -> message <= 300 and still contains the spaceId + listSpaces.
Adjusted the existing list-cap test to short ids/names so it exercises the 10-item
cap + "(+N ещё)" tail below the length cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:49:31 +03:00
agent_coder 12ed3a0332 feat(comment): undo button in resolve toast + sort resolved tab by resolve time
Part 1 (#542): add an inline Undo (reopen) button to the resolve success
toast in useResolveCommentMutation. Undo re-invokes the same mutation via a
self-ref (declared before useMutation, read at click time — no cycle) and
clears the inline comment mark through an editor ref (survives the originating
CommentListItem unmounting). A closured `done` flag guards a fast double-click
(notifications.hide is async). Reopen keeps the plain toast (no Undo).

onError gains a terminal 404 branch: drop the comment from the cache and
unsetComment its orphaned mark (no rollback → no phantom in Resolved, neutral
"Comment no longer exists"). The generic-failure copy is now directional
(resolve vs. re-open). Autoclose policy const RESOLVE_UNDO_AUTOCLOSE_MS=10000.

Part 2: sort the Resolved tab by resolvedAt DESC via an exported
sortResolvedByResolvedAt helper; coerce with new Date(...) because resolvedAt
is an ISO string at runtime (Date only in the optimistic window).

i18n: add "Failed to re-open comment" and "Comment no longer exists" to
en-US/ru-RU. Tests: 6 resolve-undo cases + 4 sort cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:45:42 +03:00
agent_coder ae3dfd8de6 Merge remote-tracking branch 'gitea/develop' into feat/492-incremental-render 2026-07-12 05:34:07 +03:00
agent_coder 515a1aaf1d Merge remote-tracking branch 'gitea/develop' into fix/535-list-seam-coalesce 2026-07-12 05:22:26 +03:00
agent_coder 74387bb047 test(#522): make the internal-link subset invariant a real cross-package drift guard
Reviewer follow-up on the #522 internal-link import fix. The fix itself is
confirmed correct; these three changes harden its load-bearing subset invariant
and fix a stale doc.

Finding 1 (drift guard): the in-package test only compared isInternalPagePath
against a HAND-COPIED server regex, so a later narrowing of the real server
INTERNAL_LINK_REGEX would leave the test green while the client silently became
WIDER than the server. Add apps/server/.../export/internal-link-parity.spec.ts:
the top layer already depends on @docmost/prosemirror-markdown and exports
isInternalPagePath, so this spec imports the LIVE server INTERNAL_LINK_REGEX AND
the LIVE isInternalPagePath and asserts subset over an accept-corpus — a server
narrowing now reddens CI. The in-package manual copy is demoted from its
"drift guard" role (comment updated to say so; it stays as local documentation).

Finding 2 (charset non-vacuity): the REJECT list was purely structural, so the
mutation widening the slug charset [a-zA-Z0-9-] -> [a-zA-Z0-9-.] survived the
whole suite. Add REJECT cases whose ONLY defect is a forbidden slug character
(abc.def / abc_def / abc%20 / abc~x); assert both isInternalPagePath and the
server regex reject them. The mutation now reddens the new reject test.

Finding 3 (stale JSDoc): canonicalize.ts KNOWN_DEFAULTS module blurb claimed
every entry was read from docmost-schema and import-materialized. The new
link.internal default breaks both claims (source is editor-ext/src/lib/link.ts;
external links leave internal absent/null, never false). Add a bullet documenting
the one editor-sourced, non-materialized default (false ≡ absent/null).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:59 +03:00
agent_coder 6ec2981743 fix(prosemirror-markdown): помечать внутренние ссылки на страницы как internal при импорте markdown
При импорте markdown ссылка вида `[t](/s/<space>/p/<slug>)` сохранялась как
внешняя (link-mark получал `internal:null`, `target:"_blank"`,
`rel:"noopener noreferrer nofollow"`): открывалась новой вкладкой, без
hover-превью и без участия в backlinks. Единственный путь к нативной внутренней
ссылке был ручной JSON-патч.

Чиню в общем конвертере пакета, через который проходят ВСЕ markdown-пути (MCP
updatePageMarkdown/importPageMarkdown, patch/insertNode, тела комментариев,
серверный REST create/update, single/zip-импорт, ai-chat, git-sync pull,
вставка markdown в редактор) — все они получают исправление разом.

- Новый модуль `internal-links.ts`: чистый `isInternalPagePath(href)` —
  якорный `^/s/<space>/p/<slug>/?$`, СТРОГОЕ подмножество серверного
  `INTERNAL_LINK_REGEX` (apps/server/.../export/utils.ts), поэтому всё
  помеченное гарантированно бэклинкуется и переписывается при экспорте.
  Fail-toward-external: любая неоднозначность (scheme/host, `#`, `?`,
  бесспейсовый `/p/<slug>`, лишний сегмент, не-строка) остаётся внешней.
- `markInternalLinks(doc)`: пост-обход готового ProseMirror-документа,
  помечает КАЖДУЮ text-ноду, покрытую внутренней ссылкой (включая случай
  вложенных bold/italic внутри ссылки) → `{internal:true, target:null,
  rel:null}`. Чистая и идемпотентная.
- Разводка в `markdownToProseMirrorSync` после stripEmptyParagraphs.
- §11 (обязательная сопутствующая правка): `internal:false` добавлен в
  `KNOWN_DEFAULTS.link` канонизатора — редакторные внешние ссылки хранят
  `internal:false`, теперь он канонизируется как absent/null/external, а
  load-bearing `internal:true` (не дефолт) переживает канонизацию.

Тесты: accept/reject-пиннинг матчера (edge: trailing slash, `#`, `?`,
`/p/x` без спейса, `https://h/p/x`, `/api/...`, uppercase, empty-space),
subset-инвариант против серверного регэкспа, мультинодовая ссылка со вложенными
марками, идемпотентность, полный конвертер (internal помечен / external нетронут /
бэклинк-извлекаемость), комментарий-тело через общий путь, canonicalize §11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:58 +03:00
agent_coder dd1fe90515 fix(prosemirror-markdown): guard three-way list coalesce against transitive style loss (#535)
Review follow-up on the list-seam coalescing:

1. The three-way merge only checked each PRE-EXISTING list against the
   inserted one. A default-typed inserted orderedList between two lists with
   explicit DIFFERENT numbering styles passed both pairwise checks through the
   null-typed middle, collapsing all three and silently losing the right
   list's style. Add a `listsMergeable(left, right)` guard to the three-way
   condition. On failure fall through to the single-seam path: a single
   inserted block can be absorbed by at most one neighbour, so prefer the LEFT
   seam (consistent with the three-way survivor choice) and leave the
   incompatible right list separate with its own style.

2. Lock the footnotesList exclusion with a test — inserting a footnotesList
   next to a footnotesList must leave TWO separate blocks (a refactor of the
   allow-list to endsWith("List") would otherwise silently merge them and
   corrupt footnotes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:06:54 +03:00
agent_coder c96fafc4ad perf(ai-chat): append-персист шагов — per-step INSERT вместо переписи строки (#492)
Раньше каждый onStepFinish переписывал ВСЮ строку ассистента (растущий
metadata.parts jsonb со всеми выводами инструментов) → O(n²) объёма записи
на прогон: под MVCC/TOAST апдейт jsonb переписывает всю версию строки, так
что шаг k пишет ~k×вывод. Прогон из 50 шагов по ~100 КБ = сотни МБ WAL и
мёртвых кортежей за ход, что молотит autovacuum. (#490 убрал только ВТОРУЮ
копию в tool_calls; сам metadata.parts всё ещё рос и переписывался.)

Теперь каждый завершённый шаг ДОПИСЫВАЕТСЯ отдельной строкой в лёгкую
таблицу ai_chat_run_steps (только парты этого шага), а строка сообщения
получает дешёвый маркер (stepsPersisted + toolTraceVersion, без растущего
блоба parts). Полный metadata.parts собирается ОДИН раз на финализации.
НЕ jsonb-append (||): апдейт всё равно переписывает всю TOAST-версию —
экономится только сетевой payload, а WAL/мёртвые кортежи остаются; поэтому
именно ОТДЕЛЬНАЯ таблица + INSERT.

Три обязательные интеграции:
- reconstructRunParts(row, stepRows) → { parts, stepsPersisted }: единый
  шов переключения бэкенда. Читает парты из СТРОКИ, если она уже несёт
  inline-parts (старые записи + ЛЮБАЯ финализированная), иначе из ТАБЛИЦЫ
  ШАГОВ (mid-run запись #492). Дискриминатор — наличие непустого
  metadata.parts (флаг схемы не нужен). Потребители (attach-seed,
  delta-poll, export, reconnect) прогоняют строки через hydrateAssistantParts
  на границе чтения — их контракт/вывод не меняется, старые и новые записи
  восстанавливаются идентично.
- сигнал ротации кольца реестра #491 (confirmPersistedStep) теперь стреляет
  на подтверждённый INSERT шага, под тем же контрактом (updateStreaming
  возвращает stepsPersisted / null).
- era-marker toolTraceVersion (#490) больше не ставится полной переписью —
  ставится в маркере шага и на финализации (flushAssistant), остаётся
  консистентным.

Полная обратная совместимость: прогон, записанный по-старому (полная строка,
без строк шагов), восстанавливается/attach/export идентично. При отсутствии
репозитория шагов (позиционные тест-конструкции) — фолбэк на прежний
полнострочный flush (без регрессии, только без выигрыша WAL).

Тесты (реальный pg, int-lane):
- WAL-дельта (pg_current_wal_lsn) на прогоне 40×100КБ: new=4.3МБ vs
  old=90.3МБ (20.8x) — O(Σ шагов) против O(n²); старый путь в тесте И есть
  ревертнутое поведение (мутация-проверка).
- reconstruct-контракт: new-style (таблица шагов) и old-style (inline) прогоны
  восстанавливаются в идентичные parts; hydrate заполняет строку.
- миграция up/down roundtrip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 05:02:25 +03:00
agent_coder be70bd2e8e Merge remote-tracking branch 'gitea/develop' into fix/541-reconnect-timeout 2026-07-12 04:37:47 +03:00
agent_coder 2f8c5d9a98 perf(client): инкрементальный рендер стрима ответа (#492)
Путь ответа ассистента теперь рендерится инкрементально, по образцу
StreamingPlainText из ветки reasoning (#492, волна C эпика #497).

Раньше MarkdownPart прогонял ВЕСЬ накопленный ответ через канонический
конвейер (markdownToProseMirrorSync → PMNode.fromJSON → DOMSerializer →
DOMPurify) на КАЖДОМ throttled-тике (~20 Гц). На синтетическом потоке
~100 КБ это 394 вызова renderChatMarkdown — O(числа тиков), причём каждый
вызов заново парсит всю растущую строку.

Теперь:
- StreamingMarkdownText делит текст на блоки по безопасному срезу
  (splitPlainChunks — та же append-only-инвариантность, что у reasoning):
  СТАБИЛИЗИРОВАННЫЕ блоки идут через канонический конвейер и мемоизируются
  (каждый блок парсится РОВНО ОДИН раз), живой ХВОСТ — дешёвый plain-text
  (React-escaped, без парсера/санитайзера/innerHTML) до стабилизации.
- На финализации (флип state → done или конец хода) — ОДИН полный
  канонический рендер всего текста: побайтовая визуальная паритетность с
  прежним выводом (включая <li><p>-обёртки схемы и scoped-CSS из #498).
- Гейт liveness тот же, что у ReasoningBlock: streaming =
  turnStreaming && part.state === "streaming".

Также цикл рендера частей переведён на ИСЧЕРПЫВАЮЩИЙ switch по видам частей
с never-проверкой в default (вместо прежнего WARNING-комментария): новый
закрытый вид части в UIMessagePart теперь ошибка компиляции.

Тесты:
- perf-smoke: на ~100 КБ потоке число вызовов renderChatMarkdown ≤ blocks+2
  и ≪ ticks (O(блоков), не O(тиков)); мутация (снять memo с MarkdownChunk)
  краснит ассерт (78631 вызовов).
- visual-regression: финальный рендер побайтово равен renderChatMarkdown
  всего текста (в т.ч. <li><p>), инкрементальный вид сходится к нему на
  финализации; учтён neutralizeInternalLinks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:35:06 +03:00
agent_coder 69e04349a0 fix(client): вход в reconnect ограничивает ожидание getRun таймаутом (#541)
При живом обрыве SSE вход в лестницу reconnect происходил только
после резолва getRun(cid) (ре-сид из персиста). Путь REJECT
обрабатывался через .catch, но ЗАВИСШИЙ getRun (соединение есть,
ответа нет) был не ограничен: axios-клиент (lib/api-client.ts) не
имеет timeout, а stalled-idle-cap взводится только ПОСЛЕ входа в
reconnecting/polling. Итог — FSM залипала в `streaming` без баннера
и без поллинга до сокет-таймаута браузера (минуты). Это тот самый
класс тихого зависания, который устраняет эпик #497.

Оборачиваю ожидание getRun гонкой с таймаутом
(RECONNECT_RESEED_TIMEOUT_MS = 4s): по таймауту берётся ТОТ ЖЕ
фолбэк, что и на reject — dropLivePartialAndReplayFromStart() +
enterReconnect(runId), так что лестница/поллинг стартуют и
stalled-idle-cap взводится. Локальный флаг `settled` делает ветви
resolve/reject/timeout взаимоисключающими: поздний резолв getRun
после сработавшего таймаута полностью игнорируется (не входит в
reconnect повторно, не перетирает replay-from-start устаревшим
ре-сидом, ничего не перевзводит). Таймер живёт в reseedTimerRef и
очищается при размонтировании (никаких висящих setTimeout).

Тесты: hang-кейс (getRun не резолвится -> по таймауту FSM в
reconnecting, live-partial сброшен, replay-from-start) и
late-resolve safety (поздний резолв — no-op). Мутация: замена гонки
на голый getRun краснит оба #541-теста.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 04:29:25 +03:00
agent_coder 5bd5995ef0 fix(prosemirror-markdown): coalesce list seams on insert (#535)
insertNode with a markdown/JSON list now appends/prepends its items into
an adjacent same-type sibling list instead of creating a separate sibling
list that the serializer splits with `<!-- -->`.

Adds a single shared merge rule (isCoalescibleList/listsMergeable) used by
both the markdown path (insertNodesRelative) and the raw-node path
(insertNodeRelative). Coalescing is strictly local to the two seams of the
active insertion (each merged at most once — no greedy loop, no global
normalization). Survivor is chosen POSITIONALLY (the pre-existing neighbour
outside the inserted [i,j) range), so it keeps its block id and list-level
attrs (e.g. orderedList start); the inserted wrapper's re-minted id is
discarded. Handles the three-way case (list inserted between two same-type
lists → left survives) and refuses to coalesce an empty inserted list.

before/after now resolve via findAnchorChain and splice into the anchor's
immediate parent, so coalescing runs against the actual parent array
(incl. lists nested in callouts / table cells).

Serializer / LIST_MARKER_SEPARATOR untouched — two genuinely-separate
lists still emit `<!-- -->`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 02:43:53 +03:00
agent_coder 575125a5dc fix(mcp): enrich bad/inaccessible spaceId 404 into an actionable error (#534)
A well-formed but non-existent/inaccessible spaceId made the space-permissions
check answer with an opaque 404 ("Space permissions not found"), which the agent
could not self-correct from. Add an enrich-on-404 wrapper at the client-method
level (Variant A — no backend change) that, ONLY on that 404, replaces the server
text with a factual message naming the bad spaceId and the spaces the token can
actually see, pointing at listSpaces.

Mechanism (context.ts):
- getAccessibleSpaceIndex(): the token's accessible spaces from the single source
  of truth (/spaces), with a per-instance short-TTL cache (MCP_SPACES_CACHE_TTL_MS,
  default 60s; 0 disables) and single-flight dedup. Only a COMPLETE (untruncated)
  result is cached and only a complete result may drive the rewrite. The in-flight
  promise is nulled on BOTH resolve and reject so a transient /spaces blip is never
  memoized. Cache invalidated on every identity change, mirroring collabTokenCache
  (login() + the 401/403 reauth interceptor).
- paginateAllWithMeta(): surfaces the `truncated` flag paginateAll swallows;
  paginateAll now delegates to it (unchanged contract, getSpaces untouched).
- withSpaceAccessDiagnostics(spaceId, mcpName, fn): abort/cap wins FIRST (by the
  toolAbortSignal flag, NOT e.name — a cap may be TimeoutError/custom); only a 404
  is enrichable; FAILS OPEN (rethrows the original server error) on every source of
  uncertainty (fetch failed / !complete / spaceId present / aborted).

Wrapped only the paths where the sole 404 cause is the space membership check:
getTree, listPages (tree + recent-with-spaceId), search (with spaceId),
checkNewComments. createPage/getPageContext are deliberately NOT wrapped.
formatSpaceNotAccessible (errors.ts) composes the message (<=10 spaces + "(+N ещё)"
tail; distinct zero-spaces variant).

Tests: 11 new unit tests (stub client) covering all 9 acceptance criteria +
message shape; full mcp unit suite green (701 tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 02:20:12 +03:00
122 changed files with 12552 additions and 1597 deletions
+460
View File
@@ -0,0 +1,460 @@
/**
* CommentsPanel — редизайн панели комментариев с фокусом на агентские правки.
* Mantine v7. Узкая колонка ~340–400px, светлая/тёмная темы.
*
* Реализованные решения (см. README.md):
* - Дифф-первая карточка: цитата = старая строка диффа (без тройного дублирования)
* - Группировка серии прогона под одной шапкой ("Корректор · 28 правок")
* - Пакетное "Accept all" на фронте (по одной под капотом) с полоской прогресса
* - Метки важности/категории парсятся клиентом из тегов "[Корректура][Существенно]"
* - Фильтры по важности/категории
* - Безопасный Dismiss через undo-тост (удаление откладывается на клиенте)
* - Состояния: pending / applied / dismissed / conflict (потерян якорь)
* - Крайние диффы: вставка (␣), удаление, одна буква, дефис→тире, длинный абзац
* - Человеческий тред не деградирует (та же система, другое наполнение)
* - Вкладки Open/Resolved сохранены
*/
import { useMemo, useState, useCallback, useRef } from 'react';
import {
Box, Group, Stack, Text, Badge, Button, ActionIcon, Tabs, Progress,
Avatar, Tooltip, ScrollArea, useMantineColorScheme, useMantineTheme,
} from '@mantine/core';
import { notifications } from '@mantine/notifications';
/* ─────────────────────────── Типы данных ─────────────────────────── */
export type Severity = 'critical' | 'major' | 'minor';
export type Category = string; // "Корректура" | "Факт" | "Стиль" | …
/** Один сегмент интра-диффа: изменённый фрагмент подсвечивается. */
export interface DiffSegment {
text: string;
changed: boolean;
}
/** Правка: сервер уже отдаёт посегментную разметку обеих строк. */
export interface SuggestedEdit {
before: DiffSegment[]; // "было" (пустой массив ⇒ чистая вставка)
after: DiffSegment[]; // "стало" (пустой массив ⇒ чистое удаление)
}
export type CommentStatus = 'pending' | 'applied' | 'dismissed' | 'conflict';
export interface Comment {
id: string;
runId?: string; // id прогона агента — для группировки серии
authorName: string; // "Корректор" | "Нарратор" | имя человека
authorKind: 'agent' | 'human';
triggeredBy?: string; // кто запустил агента ("vvzvlad")
createdAtLabel: string; // "15 ч", "2 дн" — форматируется вызывающим кодом
/** Сырой текст комментария от агента, метки в квадратных скобках. */
bodyRaw: string;
edit?: SuggestedEdit; // есть ⇒ агентская правка; нет ⇒ обычный тред
status: CommentStatus;
replyCount?: number;
anchorLost?: boolean; // сервер ответил конфликтом якоря
}
export interface CommentsPanelProps {
comments: Comment[];
/** Применить одну правку. Резолвит тред на сервере. */
onApply: (id: string) => Promise<void>;
/** Жёсткое удаление на сервере (для треда без ответов). Вызывается ПОСЛЕ окна undo. */
onDismiss: (id: string) => Promise<void>;
/** Резолв обычного треда. */
onResolve?: (id: string) => Promise<void>;
/** Клик по карточке/цитате — скролл документа к якорю и подсветка. */
onNavigateToAnchor: (id: string) => void;
onClose?: () => void;
/** Мс до фактического удаления после Dismiss (окно undo). По умолчанию 5000. */
dismissUndoMs?: number;
}
/* ───────────────────── Клиентский парсинг меток ───────────────────── */
const SEVERITY_WORDS: Record<string, Severity> = {
'существенно': 'major', 'критично': 'critical', 'критическая': 'critical',
'незначительно': 'minor', 'мелко': 'minor', 'major': 'major',
'minor': 'minor', 'critical': 'critical',
};
interface ParsedBody {
category?: Category;
severity: Severity;
text: string; // тело без скобочных тегов
}
/** "[Корректура] [Существенно] Пробел…" → {category, severity, text}. */
export function parseBody(raw: string): ParsedBody {
const tags: string[] = [];
const text = raw.replace(/\[([^\]]+)\]/g, (_, t) => { tags.push(t.trim()); return ''; }).trim();
let severity: Severity = 'minor';
let category: Category | undefined;
for (const t of tags) {
const sv = SEVERITY_WORDS[t.toLowerCase()];
if (sv) severity = sv; else if (!category) category = t;
}
return { category, severity, text };
}
const SEV_COLOR: Record<Severity, string> = {
critical: 'red', major: 'orange', minor: 'gray',
};
/* ──────────────────────────── Дифф ──────────────────────────── */
/** Делает невидимые символы видимыми в подсветке (пробел, таб). */
function visibleWhitespace(s: string): string {
return s.replace(/ /g, '␣').replace(/\t/g, '⇥');
}
function DiffLine({ segments, kind }: { segments: DiffSegment[]; kind: 'del' | 'ins' }) {
const theme = useMantineTheme();
const { colorScheme } = useMantineColorScheme();
const dark = colorScheme === 'dark';
const isDel = kind === 'del';
const sign = isDel ? '−' : '+';
const signColor = isDel ? theme.colors.red[dark ? 5 : 6] : theme.colors.green[dark ? 5 : 6];
const baseColor = isDel
? (dark ? theme.colors.gray[5] : theme.colors.gray[6])
: (dark ? theme.colors.gray[1] : theme.colors.dark[9]);
const markBg = isDel
? (dark ? 'rgba(224,49,49,.22)' : '#ffe3e3')
: (dark ? 'rgba(47,158,68,.22)' : '#d3f9d8');
const markFg = isDel
? (dark ? theme.colors.red[3] : theme.colors.red[8])
: (dark ? theme.colors.green[3] : theme.colors.green[9]);
return (
<Group gap={7} wrap="nowrap" align="flex-start">
<Text ff="monospace" fw={600} fz={12} c={signColor} w={11} ta="center" style={{ flex: 'none', lineHeight: 1.5 }}>{sign}</Text>
<Text fz={13.5} c={baseColor} style={{ lineHeight: 1.45 }}>
{segments.map((seg, i) =>
seg.changed ? (
<Box key={i} component="mark" px={3} fw={600}
style={{ background: markBg, color: markFg, borderRadius: 3,
textDecoration: isDel ? 'line-through' : 'none' }}>
{visibleWhitespace(seg.text)}
</Box>
) : (
<Box key={i} component="span" style={{ textDecoration: isDel ? 'line-through' : 'none' }}>{seg.text}</Box>
)
)}
</Text>
</Group>
);
}
function DiffBlock({ edit }: { edit: SuggestedEdit }) {
const pureInsert = edit.before.length === 0;
const pureDelete = edit.after.length === 0;
return (
<Stack gap={1}>
{!pureInsert && <DiffLine segments={edit.before} kind="del" />}
{!pureDelete && <DiffLine segments={edit.after} kind="ins" />}
</Stack>
);
}
/* ──────────────────────── Карточка правки ──────────────────────── */
function EditCard({ c, onApply, onDismiss, onNavigateToAnchor, dismissUndoMs = 5000 }: {
c: Comment;
onApply: CommentsPanelProps['onApply'];
onDismiss: CommentsPanelProps['onDismiss'];
onNavigateToAnchor: CommentsPanelProps['onNavigateToAnchor'];
dismissUndoMs?: number;
}) {
const parsed = useMemo(() => parseBody(c.bodyRaw), [c.bodyRaw]);
const [busy, setBusy] = useState(false);
const timer = useRef<number>();
const apply = useCallback(async () => {
setBusy(true);
try { await onApply(c.id); } finally { setBusy(false); }
}, [c.id, onApply]);
// Безопасный Dismiss: прячем сразу, удаляем на сервере после окна undo.
const dismiss = useCallback(() => {
let undone = false;
const nid = notifications.show({
message: 'Edit dismissed',
color: 'gray',
autoClose: dismissUndoMs,
withCloseButton: false,
// Кнопка "Вернуть" — см. README про кастомный рендер экшена.
});
timer.current = window.setTimeout(() => {
if (!undone) onDismiss(c.id);
}, dismissUndoMs);
// undo вызывается из UI: clearTimeout(timer.current); undone = true;
return { nid, cancel: () => { undone = true; clearTimeout(timer.current); } };
}, [c.id, onDismiss, dismissUndoMs]);
const conflict = c.status === 'conflict' || c.anchorLost;
return (
<Box
p="10px 12px"
onClick={() => onNavigateToAnchor(c.id)}
style={(t) => ({
cursor: 'pointer',
borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}`,
})}
>
<Stack gap={8}>
{c.edit && <DiffBlock edit={c.edit} />}
{conflict && (
<Group gap={7} wrap="nowrap" p="6px 8px"
style={(t) => ({ background: t.colorScheme === 'dark' ? 'rgba(230,180,20,.12)' : '#fff9db', borderRadius: 7 })}>
<Text fz={12}></Text>
<Text fz={12} c="yellow.8" style={{ lineHeight: 1.4 }}>Text changed this edit cant be applied</Text>
</Group>
)}
{parsed.text && !conflict && (
<Text fz={12.5} c="dimmed" style={{ lineHeight: 1.45 }}>{parsed.text}</Text>
)}
<Group gap={8} wrap="nowrap" onClick={(e) => e.stopPropagation()}>
<Box w={8} h={8} style={(t) => ({ flex: 'none', borderRadius: '50%', background: t.colors[SEV_COLOR[parsed.severity]][6] })} />
<Text fz={10} fw={600} tt="uppercase" c="dimmed" style={{ letterSpacing: '.06em', flex: 1, whiteSpace: 'nowrap', overflow: 'hidden', textOverflow: 'ellipsis' }}>
{[parsed.category, parsed.severity === 'major' ? 'Major' : parsed.severity === 'critical' ? 'Critical' : null].filter(Boolean).join(' · ')}
</Text>
{c.status === 'applied' ? (
<Badge color="green" variant="light" radius="xl" size="sm"> Applied</Badge>
) : c.status === 'dismissed' ? (
<Badge color="gray" variant="light" radius="xl" size="sm">Dismissed</Badge>
) : conflict ? (
<Button size="compact-sm" variant="default" onClick={() => onNavigateToAnchor(c.id)}>Go to text</Button>
) : (
<>
<Button size="compact-sm" variant="default" color="gray" onClick={dismiss}>Dismiss</Button>
<Button size="compact-sm" color="green" loading={busy} onClick={apply}>Apply</Button>
</>
)}
</Group>
</Stack>
</Box>
);
}
/* ──────────────────────── Человеческий тред ──────────────────────── */
function HumanThread({ c, onResolve, onNavigateToAnchor }: {
c: Comment; onResolve?: CommentsPanelProps['onResolve']; onNavigateToAnchor: CommentsPanelProps['onNavigateToAnchor'];
}) {
const parsed = useMemo(() => parseBody(c.bodyRaw), [c.bodyRaw]);
return (
<Box p="12px 14px" style={(t) => ({ borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}` })}>
<Stack gap={9}>
<Group gap={8} wrap="nowrap">
<Avatar size={26} radius="xl" color="orange">{c.authorName[0]}</Avatar>
<Text fz={12.5} fw={600}>{c.authorName}
<Text span c="dimmed" fw={400}> · {c.triggeredBy ? `${c.triggeredBy} · ` : ''}{c.createdAtLabel}</Text>
</Text>
<Box style={{ flex: 1 }} />
<Tooltip label="Resolve"><ActionIcon variant="default" size="md" onClick={() => onResolve?.(c.id)}></ActionIcon></Tooltip>
<ActionIcon variant="default" size="md"></ActionIcon>
</Group>
<Text fz={13.5} style={{ lineHeight: 1.5 }}>{parsed.text}</Text>
<Group gap={8}>
{c.replyCount ? <Text fz={12} c="dimmed">{c.replyCount} replies</Text> : null}
<Box style={{ flex: 1 }} />
<Button variant="subtle" size="compact-sm">Reply</Button>
</Group>
</Stack>
</Box>
);
}
/* ──────────────────── Шапка серии + прогресс ──────────────────── */
function RunHeader({ runComments, onAcceptAll, progress }: {
runComments: Comment[];
onAcceptAll: () => void;
progress?: { done: number; total: number } | null;
}) {
const head = runComments[0];
const majors = runComments.filter((c) => parseBody(c.bodyRaw).severity !== 'minor').length;
const applied = runComments.filter((c) => c.status === 'applied').length;
return (
<Box>
<Group gap={10} wrap="nowrap" p="11px 13px" style={(t) => ({ borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}` })}>
<Box style={{ position: 'relative', width: 30, height: 30, flex: 'none' }}>
<Avatar size={30} radius={8} color="teal">{head.authorName[0]}</Avatar>
{head.triggeredBy && (
<Avatar size={14} radius="xl" color="gray"
style={{ position: 'absolute', right: -4, bottom: -4, border: '2px solid var(--mantine-color-body)' }} />
)}
</Box>
<Stack gap={1} style={{ minWidth: 0 }}>
<Text fz={13} fw={600}>{head.authorName}
<Text span c="dimmed" fw={400}> · {head.triggeredBy} · {head.createdAtLabel}</Text>
</Text>
<Text fz={11.5} c="dimmed">
{runComments.length} edits · <Text span c="orange.7" fw={600}>{majors} major</Text> · {applied} applied
</Text>
</Stack>
<Box style={{ flex: 1 }} />
{!progress && <Button size="compact-sm" color="green" onClick={onAcceptAll}>Accept all</Button>}
</Group>
{progress && (
<Box p="9px 13px 11px" style={(t) => ({ background: t.colorScheme === 'dark' ? 'rgba(47,158,68,.08)' : '#f8fbf9', borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[2]}` })}>
<Group gap={8} mb={6}>
<Text fz={12} fw={600} c="green.7">Applying {progress.done} of {progress.total}</Text>
<Box style={{ flex: 1 }} />
<Button variant="subtle" color="gray" size="compact-xs">Stop</Button>
</Group>
<Progress value={(progress.done / progress.total) * 100} color="green" size="sm" radius="xl" />
</Box>
)}
</Box>
);
}
/* ──────────────────────────── Панель ──────────────────────────── */
/** Роль-автор для фильтра: у агентов — имя роли, у людей — «Пользователь». */
function roleOf(c: Comment): string {
return c.authorKind === 'human' ? 'Пользователь' : c.authorName;
}
export function CommentsPanel(props: CommentsPanelProps) {
const { comments, onApply, onDismiss, onResolve, onNavigateToAnchor, onClose, dismissUndoMs } = props;
const [tab, setTab] = useState<'open' | 'resolved'>('open');
const [roleFilter, setRoleFilter] = useState<string | null>(null);
const [progress, setProgress] = useState<Record<string, { done: number; total: number }>>({});
const open = comments.filter((c) => c.status === 'pending' || c.status === 'conflict');
const resolved = comments.filter((c) => c.status === 'applied' || c.status === 'dismissed');
const list = tab === 'open' ? open : resolved;
const filtered = list.filter((c) => !roleFilter || roleOf(c) === roleFilter);
// Роли и счётчики для чипов-фильтров (только роли: Корректор / Фактчекер / Пользователь …).
const roles = useMemo(() => {
const order: string[] = [];
const count: Record<string, number> = {};
for (const c of list) {
const r = roleOf(c);
if (!(r in count)) { count[r] = 0; order.push(r); }
count[r]++;
}
return order.map((r) => ({ role: r, count: count[r] }));
}, [list]);
// Группировка агентских правок по runId; человеческие треды — по одному.
const groups = useMemo(() => {
const map = new Map<string, Comment[]>();
const singles: Comment[] = [];
for (const c of filtered) {
if (c.authorKind === 'agent' && c.runId && c.edit) {
if (!map.has(c.runId)) map.set(c.runId, []);
map.get(c.runId)!.push(c);
} else singles.push(c);
}
return { runs: [...map.entries()], singles };
}, [filtered]);
// Пакетное "Accept all" — по одной на фронте, обновляем прогресс.
const acceptAll = useCallback(async (runId: string, items: Comment[]) => {
const minor = items.filter((c) => parseBody(c.bodyRaw).severity === 'minor' && c.status === 'pending');
for (let i = 0; i < minor.length; i++) {
setProgress((p) => ({ ...p, [runId]: { done: i, total: minor.length } }));
try { await onApply(minor[i].id); } catch { /* конфликт — пропускаем, копим для сводки */ }
}
setProgress((p) => { const n = { ...p }; delete n[runId]; return n; });
notifications.show({ color: 'green', message: `${minor.length} applied` });
}, [onApply]);
return (
<Stack gap={0} h="100%" style={{ width: 380, maxWidth: '100%', borderLeft: '1px solid var(--mantine-color-default-border)' }}>
{/* Вкладки — статусная ось. Open/Resolved сохранены. */}
<Group gap={4} p="10px 14px" style={(t) => ({ borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[2]}` })}>
<Tabs value={tab} onChange={(v) => setTab(v as 'open' | 'resolved')} variant="pills">
<Tabs.List>
<Tabs.Tab value="open" rightSection={<Badge size="sm" variant="light" color="blue">{open.length}</Badge>}>Open</Tabs.Tab>
<Tabs.Tab value="resolved" rightSection={<Badge size="sm" variant="light" color="gray">{resolved.length}</Badge>}>Resolved</Tabs.Tab>
</Tabs.List>
</Tabs>
<Box style={{ flex: 1 }} />
{onClose && <ActionIcon variant="subtle" color="gray" onClick={onClose}></ActionIcon>}
</Group>
{/* Фильтры-чипы: только по ролям (Корректор / Фактчекер / Пользователь). */}
{tab === 'open' && roles.length > 1 && (
<ScrollArea type="never" p="8px 14px" style={(t) => ({ borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}` })}>
<Group gap={6} wrap="nowrap">
<FilterChip active={!roleFilter} onClick={() => setRoleFilter(null)} label={`All ${list.length}`} solid />
{roles.map(({ role, count }) => (
<FilterChip key={role} active={roleFilter === role} onClick={() => setRoleFilter(roleFilter === role ? null : role)} label={`${role} ${count}`} />
))}
</Group>
</ScrollArea>
)}
{/* Лента */}
<ScrollArea style={{ flex: 1 }} bg="var(--mantine-color-default-hover)">
{filtered.length === 0 ? (
<EmptyState tab={tab} />
) : (
<>
{groups.runs.map(([runId, items]) => (
<Box key={runId} m="6px 10px" bg="var(--mantine-color-body)" style={{ border: '1px solid var(--mantine-color-default-border)', borderRadius: 10, overflow: 'hidden' }}>
<RunHeader runComments={items} progress={progress[runId] ?? null} onAcceptAll={() => acceptAll(runId, items)} />
{items.map((c) => (
<EditCard key={c.id} c={c} onApply={onApply} onDismiss={onDismiss} onNavigateToAnchor={onNavigateToAnchor} dismissUndoMs={dismissUndoMs} />
))}
</Box>
))}
{groups.singles.map((c) => (
<Box key={c.id} m="6px 10px" bg="var(--mantine-color-body)" style={{ border: '1px solid var(--mantine-color-default-border)', borderRadius: 10, overflow: 'hidden' }}>
{c.edit
? <EditCard c={c} onApply={onApply} onDismiss={onDismiss} onNavigateToAnchor={onNavigateToAnchor} dismissUndoMs={dismissUndoMs} />
: <HumanThread c={c} onResolve={onResolve} onNavigateToAnchor={onNavigateToAnchor} />}
</Box>
))}
</>
)}
</ScrollArea>
</Stack>
);
}
/* ─────────────────────── Мелкие компоненты ─────────────────────── */
function FilterChip({ label, active, onClick, dot, solid }: {
label: string; active: boolean; onClick: () => void; dot?: string; solid?: boolean;
}) {
return (
<Button
onClick={onClick}
size="compact-sm"
radius="xl"
variant={active ? 'filled' : 'default'}
color={active ? (solid ? 'dark' : 'blue') : 'gray'}
leftSection={dot ? <Box w={7} h={7} style={(t) => ({ borderRadius: '50%', background: t.colors[dot][6] })} /> : undefined}
styles={{ root: { flex: 'none' }, label: { fontWeight: 600, fontSize: 12 } }}
>
{label}
</Button>
);
}
function EmptyState({ tab }: { tab: 'open' | 'resolved' }) {
const isOpen = tab === 'open';
return (
<Stack align="center" gap={6} p="40px 20px">
<Avatar size={40} radius="xl" color={isOpen ? 'green' : 'gray'}>{isOpen ? '✓' : '◌'}</Avatar>
<Text fw={600} fz={13}>{isOpen ? 'All caught up' : 'Nothing here yet'}</Text>
<Text fz={12} c="dimmed" ta="center" style={{ lineHeight: 1.45 }}>
{isOpen ? 'No edits waiting on you.' : 'Applied and closed items will appear here.'}
</Text>
</Stack>
);
}
export default CommentsPanel;
+362
View File
@@ -0,0 +1,362 @@
/**
* PageHistoryModal — редизайн окна «Page history».
* Mantine v7. Light/dark. Полноразмерное модальное окно.
*
* ─────────────────────────────────────────────────────────────────────────
* ЧТО ЭТО
* ─────────────────────────────────────────────────────────────────────────
* Окно истории версий страницы. Слева — панель навигации: мини-календарь
* (heatmap: яркость дня = число ревизий, обводка = выбранный день) + плотный
* список ревизий (одна ревизия = одна строка). Справа — реально отрендеренная
* версия страницы с подсветкой изменений. Все контролы — в одну строку шапки.
*
* Ключевые решения:
* - Плотный список: одна ревизия на строку (аватар · время · автор · [SAVED]).
* Бейдж показывается ТОЛЬКО у сохранённых версий; всё остальное — autosave.
* - Агентские ревизии визуально отличаются: квадратный глиф роли (К/Ф) вместо
* круглого аватара пользователя + «via <кто запустил>».
* - Календарь объединён со списком в одной панели (мини-календарь = date jumper).
* - Highlight changes + навигация по изменениям (N of M, ↑/↓) вынесены в шапку.
* - Текущая версия помечена; Restore для неё недоступен.
*
* ─────────────────────────────────────────────────────────────────────────
* УСТАНОВКА / ИСПОЛЬЗОВАНИЕ
* ─────────────────────────────────────────────────────────────────────────
* import { PageHistoryModal, Revision } from './PageHistoryModal';
*
* <PageHistoryModal
* opened={open}
* onClose={() => setOpen(false)}
* revisions={revisions} // Revision[]
* selectedId={selId}
* onSelect={setSelId} // клик по ревизии → рендер справа
* renderVersion={(rev) => <ArticleView versionId={rev.id} />} // ваш рендер страницы
* highlightChanges={hl}
* onToggleHighlight={setHl}
* changeNav={{ index: 1, total: 3, onPrev, onNext }} // навигация по диффу
* onlySaved={onlySaved}
* onToggleOnlySaved={setOnlySaved}
* onRestore={(rev) => api.restore(rev.id)} // async; текущая версия → disabled
* />
*
* Требует MantineProvider на корне приложения (defaultColorScheme="auto" для тем).
*
* ─────────────────────────────────────────────────────────────────────────
* ФОРМАТ ДАННЫХ
* ─────────────────────────────────────────────────────────────────────────
* interface Revision {
* id: string;
* at: Date | string; // время ревизии; форматируется вызывающим/util-ом
* dayGroup: string; // 'Today' | 'Yesterday' | 'Mon 12 Jul' — заголовок группы
* saved: boolean; // true → бейдж SAVED; false → autosave (без бейджа)
* author: { name: string } & (
* | { kind: 'human' }
* | { kind: 'agent'; role: string; triggeredBy: string } // роль + кто запустил
* );
* isCurrent?: boolean; // текущая (последняя) версия — Restore недоступен
* }
*
* Ревизии приходят плоским массивом; группировка по dayGroup — на клиенте.
* Никаких изменений API не требуется: агентское авторство берётся из author.kind.
*
* ─────────────────────────────────────────────────────────────────────────
* СОСТОЯНИЯ (нарисованы/поддержаны)
* ─────────────────────────────────────────────────────────────────────────
* - Ревизия: обычная / hover / выбранная / текущая / агентская / человеческая.
* - Restore: default / disabled (выбрана текущая) / loading (спиннер после клика).
* - Highlight changes: on/off; при off навигация по изменениям неактивна.
* - Пустая история: одна версия / нет ревизий → EmptyState.
* - Тёмная тема: через токены Mantine (useMantineColorScheme, var(--mantine-*)).
*/
import { useMemo, useState, useCallback } from 'react';
import {
Modal, Box, Group, Stack, Text, Switch, Avatar, Badge, Button, ActionIcon,
ScrollArea, useMantineTheme, useMantineColorScheme,
} from '@mantine/core';
/* ─────────────────────────── Типы ─────────────────────────── */
export type Author =
| { name: string; kind: 'human' }
| { name: string; kind: 'agent'; role: string; triggeredBy: string };
export interface Revision {
id: string;
at: Date | string;
atLabel: string; // предформатированное «5:35AM»
dayGroup: string; // «Today» | «Yesterday» | «Mon 12 Jul»
saved: boolean;
author: Author;
isCurrent?: boolean;
}
export interface CalendarDay {
label: string; // «12»
inMonth: boolean;
count: number; // число ревизий за день (для heatmap)
selected?: boolean;
date: Date | string;
}
export interface PageHistoryModalProps {
opened: boolean;
onClose: () => void;
revisions: Revision[];
selectedId: string | null;
onSelect: (id: string) => void;
renderVersion: (rev: Revision | null) => React.ReactNode;
highlightChanges: boolean;
onToggleHighlight: (v: boolean) => void;
changeNav?: { index: number; total: number; onPrev: () => void; onNext: () => void };
onlySaved: boolean;
onToggleOnlySaved: (v: boolean) => void;
onRestore: (rev: Revision) => Promise<void>;
/** Ячейки текущего месяца календаря + управление. Необязательно — без него панель = только список. */
calendar?: {
monthLabel: string; // «July 2025»
weekdays: string[]; // ['Mon',…,'Sun']
days: CalendarDay[]; // обычно 35/42 ячейки
onPrevMonth: () => void;
onNextMonth: () => void;
onToday: () => void;
onPickDay: (d: CalendarDay) => void;
};
}
/* ─────────────────────────── Утилиты представления ─────────────────────────── */
const USER_PALETTE = ['#495057', '#e8590c', '#1c7ed6', '#7048e8', '#0ca678'];
function userColor(name: string) {
let h = 0;
for (let i = 0; i < name.length; i++) h = (h * 31 + name.charCodeAt(i)) >>> 0;
return USER_PALETTE[h % USER_PALETTE.length];
}
const AGENT_GRAD: Record<string, string> = {
'Корректор': 'linear-gradient(135deg,#20c997,#12b886)',
'Фактчекер': 'linear-gradient(135deg,#4c6ef5,#7048e8)',
};
function agentGrad(role: string) { return AGENT_GRAD[role] ?? 'linear-gradient(135deg,#868e96,#495057)'; }
/** heatmap: число ревизий → фон/текст ячейки календаря. */
function heat(n: number, dark: boolean) {
if (n === 0) return { bg: 'transparent', fg: dark ? '#5c5f66' : '#adb5bd' };
if (n <= 2) return { bg: dark ? 'rgba(34,139,230,.20)' : '#e7f0ff', fg: dark ? '#74c0fc' : '#1971c2' };
if (n <= 4) return { bg: dark ? 'rgba(34,139,230,.45)' : '#a5c8ff', fg: dark ? '#dbeafe' : '#0b3d91' };
return { bg: '#4c8dff', fg: '#ffffff' };
}
/* ─────────────────────────── Строка ревизии ─────────────────────────── */
function RevisionRow({ rev, selected, onSelect }: { rev: Revision; selected: boolean; onSelect: () => void }) {
const theme = useMantineTheme();
const { colorScheme } = useMantineColorScheme();
const dark = colorScheme === 'dark';
const a = rev.author;
const isAgent = a.kind === 'agent';
return (
<Group
gap={8} wrap="nowrap" h={28} px="12px 14px" m="1px 6px"
onClick={onSelect}
style={{
cursor: 'pointer', borderRadius: 7,
background: selected ? (dark ? 'rgba(34,139,230,.14)' : '#eef4ff')
: isAgent ? (dark ? 'rgba(112,72,232,.06)' : '#fbfaff') : undefined,
}}
>
{/* аватар/глиф */}
{isAgent ? (
<Box style={{ flex: 'none', width: 16, height: 16, borderRadius: 4, background: agentGrad(a.role), display: 'flex', alignItems: 'center', justifyContent: 'center', color: '#fff', font: '700 8px system-ui' }}>
{a.role[0]}
</Box>
) : (
<Box style={{ flex: 'none', width: 16, height: 16, borderRadius: '50%', background: userColor(a.name), display: 'flex', alignItems: 'center', justifyContent: 'center', color: '#fff', font: '700 8px system-ui' }}>
{a.name[0].toUpperCase()}
</Box>
)}
{/* время */}
<Text fz={12.5} fw={rev.isCurrent ? 600 : 500} c={rev.isCurrent ? undefined : 'dimmed'} style={{ flex: 'none', minWidth: 58 }}>
{rev.atLabel}
</Text>
{/* автор + via */}
<Group gap={4} wrap="nowrap" style={{ flex: 1, minWidth: 0, alignItems: 'baseline', overflow: 'hidden' }}>
<Text fz={12} fw={isAgent ? 600 : 400} c={isAgent ? undefined : 'dimmed'} truncate style={{ flex: 'none', maxWidth: 100 }}>
{isAgent ? a.role : a.name}
</Text>
{isAgent && <Text fz={11} c="dimmed" truncate>· via {a.triggeredBy}</Text>}
</Group>
{/* бейдж — только SAVED */}
{rev.saved && <Badge size="sm" radius="sm" variant="light" color="blue">SAVED</Badge>}
</Group>
);
}
/* ─────────────────────────── Мини-календарь ─────────────────────────── */
function MiniCalendar({ cal }: { cal: NonNullable<PageHistoryModalProps['calendar']> }) {
const { colorScheme } = useMantineColorScheme();
const dark = colorScheme === 'dark';
return (
<Box p="10px 12px 8px" style={(t) => ({ borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}` })}>
<Group gap={6} mb={6}>
<ActionIcon variant="subtle" color="gray" size="sm" onClick={cal.onPrevMonth}></ActionIcon>
<Text fz={12} fw={600}>{cal.monthLabel}</Text>
<ActionIcon variant="subtle" color="gray" size="sm" onClick={cal.onNextMonth}></ActionIcon>
<Box style={{ flex: 1 }} />
<Button variant="subtle" size="compact-xs" onClick={cal.onToday}>Today</Button>
</Group>
<Box style={{ display: 'grid', gridTemplateColumns: 'repeat(7,1fr)', gap: 2, marginBottom: 2 }}>
{cal.weekdays.map((w) => (
<Text key={w} ta="center" fz={8.5} fw={600} c="dimmed">{w}</Text>
))}
</Box>
<Box style={{ display: 'grid', gridTemplateColumns: 'repeat(7,1fr)', gap: 2 }}>
{cal.days.map((d, i) => {
const h = heat(d.count, dark);
return (
<Box
key={i} onClick={() => cal.onPickDay(d)}
style={{
position: 'relative', height: 26, borderRadius: 6, cursor: 'pointer',
display: 'flex', alignItems: 'center', justifyContent: 'center',
background: d.inMonth ? h.bg : 'transparent',
boxShadow: d.selected ? `inset 0 0 0 2px ${dark ? '#f1f3f5' : '#1a1b1e'}` : undefined,
}}
>
<Text fz={10.5} fw={d.selected ? 700 : 500} style={{ color: d.inMonth ? h.fg : (dark ? '#3a3d42' : '#dee2e6') }}>
{d.label}
</Text>
</Box>
);
})}
</Box>
{/* легенда heatmap */}
<Group gap={6} mt={8} align="center">
<Text fz={9.5} c="dimmed">fewer</Text>
{['#e7f0ff', '#a5c8ff', '#4c8dff'].map((c) => (
<Box key={c} style={{ width: 14, height: 10, borderRadius: 2, background: c }} />
))}
<Text fz={9.5} c="dimmed">more revisions</Text>
</Group>
</Box>
);
}
/* ─────────────────────────── Окно ─────────────────────────── */
export function PageHistoryModal(props: PageHistoryModalProps) {
const {
opened, onClose, revisions, selectedId, onSelect, renderVersion,
highlightChanges, onToggleHighlight, changeNav, onlySaved, onToggleOnlySaved,
onRestore, calendar,
} = props;
const [restoring, setRestoring] = useState(false);
const selected = revisions.find((r) => r.id === selectedId) ?? null;
const isEmpty = revisions.length <= 1;
// группировка по dayGroup, с фильтром Only saved
const groups = useMemo(() => {
const list = onlySaved ? revisions.filter((r) => r.saved) : revisions;
const map: { head: string; items: Revision[] }[] = [];
for (const r of list) {
let g = map.find((x) => x.head === r.dayGroup);
if (!g) { g = { head: r.dayGroup, items: [] }; map.push(g); }
g.items.push(r);
}
return map;
}, [revisions, onlySaved]);
const restore = useCallback(async () => {
if (!selected || selected.isCurrent) return;
setRestoring(true);
try { await onRestore(selected); } finally { setRestoring(false); }
}, [selected, onRestore]);
return (
<Modal
opened={opened} onClose={onClose} withCloseButton={false}
size="80rem" radius="lg" padding={0}
styles={{ body: { height: '80vh', maxHeight: 760, display: 'flex', flexDirection: 'column' } }}
overlayProps={{ backgroundOpacity: 0.5, blur: 1 }}
>
{/* ── single-row toolbar ── */}
<Group gap={14} h={60} px={16} wrap="nowrap" style={(t) => ({ flex: 'none', borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}` })}>
<Text fz={16} fw={600}>Page history</Text>
<Box style={(t) => ({ width: 1, height: 22, background: t.colorScheme === 'dark' ? t.colors.dark[4] : t.colors.gray[2] })} />
<Stack gap={1} style={{ minWidth: 0 }}>
<Text fz={12} fw={600} truncate>{selected ? `${selected.dayGroup} · ${selected.atLabel}` : '—'}</Text>
<Text fz={10.5} c="dimmed">selected version</Text>
</Stack>
<Box style={{ flex: 1 }} />
{/* diff cluster */}
<Group gap={2} p={3} wrap="nowrap" style={(t) => ({ background: t.colorScheme === 'dark' ? t.colors.dark[6] : '#f4f6f8', borderRadius: 10 })}>
<Button variant="white" size="compact-sm" onClick={() => onToggleHighlight(!highlightChanges)}
leftSection={<Switch checked={highlightChanges} onChange={() => {}} size="xs" tabIndex={-1} styles={{ track: { cursor: 'pointer' } }} />}
styles={{ root: { boxShadow: '0 1px 2px rgba(0,0,0,.08)' } }}>
Highlight changes
</Button>
{changeNav && (
<>
<Text fz={12} fw={600} c="dimmed" px={6}>{changeNav.index} / {changeNav.total}</Text>
<Box style={{ display: 'flex' }}>
<ActionIcon variant="subtle" color="gray" size="lg" disabled={!highlightChanges} onClick={changeNav.onPrev} style={{ width: 26 }}></ActionIcon>
<ActionIcon variant="subtle" color="gray" size="lg" disabled={!highlightChanges} onClick={changeNav.onNext} style={{ width: 26 }}></ActionIcon>
</Box>
</>
)}
</Group>
<Box style={(t) => ({ width: 1, height: 22, background: t.colorScheme === 'dark' ? t.colors.dark[4] : t.colors.gray[2] })} />
<Button color="blue" onClick={restore} loading={restoring} disabled={!selected || selected.isCurrent}>
Restore this version
</Button>
<ActionIcon variant="subtle" color="gray" size="lg" onClick={onClose}></ActionIcon>
</Group>
{/* ── body ── */}
<Box style={{ flex: 1, display: 'flex', minHeight: 0 }}>
{/* left: nav panel */}
<Stack gap={0} style={(t) => ({ flex: 'none', width: 280, minHeight: 0, borderRight: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}`, background: t.colorScheme === 'dark' ? t.colors.dark[7] : '#fcfcfd' })}>
<Group h={44} px={16} style={(t) => ({ flex: 'none', borderBottom: `1px solid ${t.colorScheme === 'dark' ? t.colors.dark[5] : t.colors.gray[1]}` })}>
<Switch checked={onlySaved} onChange={(e) => onToggleOnlySaved(e.currentTarget.checked)} size="sm" label="Only saved" labelPosition="right" styles={{ label: { fontSize: 13, fontWeight: 500 } }} />
</Group>
{calendar && <MiniCalendar cal={calendar} />}
<ScrollArea style={{ flex: 1 }}>
{groups.map((g) => (
<Box key={g.head}>
<Text px={16} pt={9} pb={5} fz={10.5} fw={600} tt="uppercase" c="dimmed" style={{ letterSpacing: '.05em', position: 'sticky', top: 0, zIndex: 2, background: 'var(--mantine-color-body)' }}>
{g.head}
</Text>
{g.items.map((r) => (
<RevisionRow key={r.id} rev={r} selected={r.id === selectedId} onSelect={() => onSelect(r.id)} />
))}
</Box>
))}
</ScrollArea>
</Stack>
{/* right: rendered version */}
<ScrollArea style={{ flex: 1, minWidth: 0 }} bg="var(--mantine-color-default-hover)">
{isEmpty ? (
<Stack align="center" justify="center" h="100%" gap={6} p={40}>
<Text fw={600} fz={14}>No earlier versions</Text>
<Text fz={12.5} c="dimmed" ta="center">У страницы пока одна версия сравнивать не с чем.</Text>
</Stack>
) : (
<Box p="26px 0" maw={660} mx="auto">
{renderVersion(selected)}
</Box>
)}
</ScrollArea>
</Box>
</Modal>
);
}
export default PageHistoryModal;
+234
View File
@@ -0,0 +1,234 @@
/**
* TimeWorkedModal — редизайн окна «Time worked on this article».
* Mantine v7. Light/dark.
*
* ─────────────────────────────────────────────────────────────────────────
* ЧТО ЭТО
* ─────────────────────────────────────────────────────────────────────────
* Сводка трудозатрат по дням в виде суточных таймлайнов. Главное отличие от
* старой версии: на барах ВИДНО время суток. Реализовано двумя способами
* (проп `axis`):
* - 'grid' — ось часов 00–06–12–18–24 сверху + вертикальные деления,
* ночные часы (0–6, 21–24) слегка затемнены. По умолчанию.
* - 'phases' — цветные полосы фаз дня за блоками (Ночь/Утро/День/Вечер),
* период суток читается сразу, без счёта делений.
*
* Блоки позиционируются по времени: left = start/24, width = dur/24.
* Интенсивность (opacity) блока = длительность/плотность работы, чтобы
* очень короткие сессии не терялись (минимальная видимая ширина задана).
* Hover-тултип на блоке: начало–конец · длительность.
*
* ─────────────────────────────────────────────────────────────────────────
* УСТАНОВКА / ИСПОЛЬЗОВАНИЕ
* ─────────────────────────────────────────────────────────────────────────
* import { TimeWorkedModal, DaySummary } from './TimeWorkedModal';
*
* <TimeWorkedModal
* opened={open}
* onClose={() => setOpen(false)}
* totalLabel="≈ 34h"
* agentLabel="≈ 1h 20m" // undefined → строку agent не показываем
* days={days} // DaySummary[]
* axis="grid" // 'grid' | 'phases'
* tz="Europe/Moscow"
* inactivityGapMin={15}
* />
*
* Требует MantineProvider на корне.
*
* ─────────────────────────────────────────────────────────────────────────
* ФОРМАТ ДАННЫХ
* ─────────────────────────────────────────────────────────────────────────
* interface Block {
* start: number; // час начала в сутках, 0..24 (напр. 13.7 = 13:42)
* end: number; // час конца
* kind: 'work' | 'agent';
* }
* interface DaySummary {
* label: string; // «Mon 29 Jun»
* totalLabel: string; // «1h 24m» | «—» для пустого дня
* blocks: Block[]; // [] → пустой день (полупрозрачная дорожка, итог «—»)
* isToday?: boolean; // сегодня — неполные сутки (рисуем границу «сейчас»)
* nowFraction?: number;// 0..1 позиция «сейчас» для isToday
* }
*
* Данные уже есть в бэкенде трудозатрат: сессии с началом/концом + тип
* (work/agent). Часы = локальные к tz. Изменений API не требуется.
*
* ─────────────────────────────────────────────────────────────────────────
* СОСТОЯНИЯ (нарисованы/поддержаны)
* ─────────────────────────────────────────────────────────────────────────
* - День: с активностью (work+agent) / только work / только agent / пустой (—).
* - Сегодня: граница «сейчас» на дорожке (nowFraction).
* - Короткий блок: минимальная ширина, не исчезает.
* - Пустая панель целиком: нет трудозатрат → EmptyState.
* - Длинный период: вертикальный скролл списка дней, шапка/легенда липкие.
* - Тёмная тема: токены Mantine.
*/
import { Modal, Box, Group, Stack, Text, ActionIcon, ScrollArea, Tooltip, useMantineColorScheme } from '@mantine/core';
/* ─────────────────────────── Типы ─────────────────────────── */
export interface Block { start: number; end: number; kind: 'work' | 'agent'; }
export interface DaySummary {
label: string;
totalLabel: string;
blocks: Block[];
isToday?: boolean;
nowFraction?: number;
}
export interface TimeWorkedModalProps {
opened: boolean;
onClose: () => void;
totalLabel: string;
agentLabel?: string;
days: DaySummary[];
axis?: 'grid' | 'phases';
tz?: string;
inactivityGapMin?: number;
}
const WORK = '#3b82f6';
const AGENT = '#c026d3';
const PHASES = [
{ name: 'Ночь', s: 0, e: 6, bg: 'rgba(99,102,241,.10)', lg: '#e8eaff', fg: '#5b60c9' },
{ name: 'Утро', s: 6, e: 12, bg: 'rgba(245,159,0,.10)', lg: '#fff2dc', fg: '#b5820e' },
{ name: 'День', s: 12, e: 18, bg: 'rgba(56,178,172,.10)', lg: '#dcf5f2', fg: '#1a857d' },
{ name: 'Вечер', s: 18, e: 24, bg: 'rgba(139,92,246,.11)', lg: '#efe7ff', fg: '#6d43c0' },
];
/* ─────────────────────────── Блок активности ─────────────────────────── */
function fmtHour(h: number) {
const hh = Math.floor(h);
const mm = Math.round((h - hh) * 60);
return `${String(hh).padStart(2, '0')}:${String(mm).padStart(2, '0')}`;
}
function fmtDur(h: number) {
const total = Math.round(h * 60);
const hh = Math.floor(total / 60), mm = total % 60;
return hh ? `${hh}h ${mm}m` : `${mm}m`;
}
function ActivityBlock({ b }: { b: Block }) {
const left = (b.start / 24) * 100;
const width = Math.max(((b.end - b.start) / 24) * 100, 0.6);
const dur = b.end - b.start;
const opacity = dur < 0.16 ? 0.5 : dur < 0.35 ? 0.8 : 1;
return (
<Tooltip label={`${fmtHour(b.start)}${fmtHour(b.end)} · ${fmtDur(dur)}`} withArrow openDelay={120} fz={11}>
<Box style={{
position: 'absolute', top: 5, height: 14, left: `${left}%`, width: `${width}%`,
borderRadius: 2, background: b.kind === 'agent' ? AGENT : WORK, opacity, cursor: 'default',
}} />
</Tooltip>
);
}
/* ─────────────────────────── Дорожка дня ─────────────────────────── */
function DayTrack({ d, axis, dark }: { d: DaySummary; axis: 'grid' | 'phases'; dark: boolean }) {
const trackBg = axis === 'grid'
? 'linear-gradient(90deg, rgba(90,100,130,.10) 0 25%, rgba(90,100,130,.02) 25% 87.5%, rgba(90,100,130,.10) 87.5% 100%)'
: (dark ? 'rgba(255,255,255,.04)' : '#f6f8fa');
return (
<Group gap={0} h={30} wrap="nowrap">
<Text style={{ flex: 'none', width: 92 }} fz={13} c="dimmed">{d.label}</Text>
<Box style={{ position: 'relative', flex: 1, height: 24, margin: '0 4px', borderRadius: 5, overflow: 'hidden', background: trackBg }}>
{/* фон: полосы фаз или деления сетки */}
{axis === 'phases'
? PHASES.map((ph) => (
<Box key={ph.name} style={{ position: 'absolute', top: 0, bottom: 0, left: `${(ph.s / 24) * 100}%`, width: `${((ph.e - ph.s) / 24) * 100}%`, background: ph.bg }} />
))
: [25, 50, 75].map((p) => (
<Box key={p} style={{ position: 'absolute', top: 0, bottom: 0, left: `${p}%`, width: 1, background: 'rgba(120,130,150,.16)' }} />
))}
{/* блоки */}
{d.blocks.map((b, i) => <ActivityBlock key={i} b={b} />)}
{/* граница «сейчас» для сегодняшнего дня */}
{d.isToday && d.nowFraction != null && (
<Box style={{ position: 'absolute', top: 0, bottom: 0, left: `${d.nowFraction * 100}%`, width: 2, background: '#fa5252' }} />
)}
</Box>
<Text style={{ flex: 'none', width: 64, textAlign: 'right' }} fz={12.5} fw={500} c={d.totalLabel === '—' ? 'dimmed' : undefined}>
{d.totalLabel}
</Text>
</Group>
);
}
/* ─────────────────────────── Окно ─────────────────────────── */
export function TimeWorkedModal(props: TimeWorkedModalProps) {
const { opened, onClose, totalLabel, agentLabel, days, axis = 'grid', tz = 'Europe/Moscow', inactivityGapMin = 15 } = props;
const { colorScheme } = useMantineColorScheme();
const dark = colorScheme === 'dark';
const isEmpty = days.length === 0 || days.every((d) => d.blocks.length === 0);
return (
<Modal opened={opened} onClose={onClose} withCloseButton={false} size="46rem" radius="lg" padding={0}
overlayProps={{ backgroundOpacity: 0.5, blur: 1 }}>
<Box p="22px 24px 20px">
{/* header */}
<Group mb={14}>
<Text fz={17} fw={600}>Time worked on this article</Text>
<Box style={{ flex: 1 }} />
<ActionIcon variant="subtle" color="gray" size="lg" onClick={onClose}></ActionIcon>
</Group>
{/* summary */}
<Group align="baseline" gap={16} mb={axis === 'grid' ? 8 : 16}>
<Text fz={22} fw={700}>{totalLabel}</Text>
{agentLabel && <Text fz={13} c="dimmed">agent: {agentLabel}</Text>}
</Group>
{/* legend (grid only) */}
{axis === 'grid' && (
<Group gap={16} mb={16}>
<Group gap={6}><Box w={12} h={12} style={{ borderRadius: 3, background: WORK }} /><Text fz={12} fw={500} c="dimmed">Work</Text></Group>
<Group gap={6}><Box w={12} h={12} style={{ borderRadius: 3, background: AGENT }} /><Text fz={12} fw={500} c="dimmed">Agent</Text></Group>
</Group>
)}
{isEmpty ? (
<Stack align="center" gap={6} p="48px 20px">
<Text fw={600} fz={14}>No time tracked yet</Text>
<Text fz={12.5} c="dimmed" ta="center">По этой статье ещё нет трудозатрат.</Text>
</Stack>
) : (
<>
{/* axis header (sticky) */}
<Group gap={0} mb={5} wrap="nowrap" style={{ position: 'sticky', top: 0, zIndex: 2, background: 'var(--mantine-color-body)' }}>
<Box style={{ flex: 'none', width: 92 }} />
{axis === 'grid' ? (
<Box style={{ flex: 1, position: 'relative', height: 14, margin: '0 4px' }}>
{[['0%', '00', 'flex-start'], ['25%', '06', 'center'], ['50%', '12', 'center'], ['75%', '18', 'center'], ['100%', '24', 'flex-end']].map(([l, t, al]) => (
<Text key={t as string} fz={10} fw={500} c="dimmed" style={{ position: 'absolute', left: l as string, transform: al === 'center' ? 'translateX(-50%)' : al === 'flex-end' ? 'translateX(-100%)' : undefined }}>{t}</Text>
))}
</Box>
) : (
<Box style={{ flex: 1, display: 'flex', height: 16, margin: '0 4px', borderRadius: 4, overflow: 'hidden' }}>
{PHASES.map((ph) => (
<Box key={ph.name} style={{ flex: ph.e - ph.s, display: 'flex', alignItems: 'center', justifyContent: 'center', background: ph.lg, color: ph.fg, font: '600 9.5px system-ui' }}>{ph.name}</Box>
))}
</Box>
)}
<Box style={{ flex: 'none', width: 64 }} />
</Group>
{/* day rows */}
<ScrollArea.Autosize mah="60vh" type="hover">
{days.map((d, i) => <DayTrack key={i} d={d} axis={axis} dark={dark} />)}
</ScrollArea.Autosize>
</>
)}
<Text mt={16} fz={11.5} c="dimmed">Estimate · timezone {tz} · inactivity gap {inactivityGapMin} min</Text>
</Box>
</Modal>
);
}
export default TimeWorkedModal;
@@ -346,6 +346,9 @@ roles:
□ Working sections ("Log", "Open Questions", "Revision") are moved to an
appendix at the end of the document or clearly separated from the report
body.
□ Once the report is complete, call save_page_version to pin the finished
document as a restorable named version (a checkpoint the reader can
return to). A repeat call after no further edits is a harmless no-op.
Be honest about gaps. If you couldn't find something, say so — don't disguise
a guess as a fact.
autoStart: false
@@ -442,6 +445,7 @@ roles:
- **The language of the notes = the main language of the call.** Technical terms — in their canonical spelling (usually Latin).
- **Don't evaluate the participants** and don't comment on the quality of the discussion.
- The output is the notes only, with no preambles or meta-comments, apart from targeted uncertainty marks.
- **Pin the result.** Once the notes are complete on the page, call save_page_version to save them as a restorable named version (a checkpoint). Calling it again after no further edits is a harmless no-op.
## Style example (excerpt)
@@ -345,6 +345,9 @@ roles:
□ Working sections («Журнал», «Открытые вопросы», «Ревизия») are moved to
an appendix at the end of the document or clearly separated from the
report body.
□ Once the report is complete, call save_page_version to pin the finished
document as a restorable named version (a checkpoint the reader can
return to). A repeat call after no further edits is a harmless no-op.
Be honest about gaps. If you couldn't find something, say so — don't disguise
a guess as a fact.
autoStart: false
@@ -441,6 +444,7 @@ roles:
- **Язык конспекта = основной язык созвона.** Технические термины — в каноническом написании (обычно латиницей).
- **Не оценивай участников** и не комментируй качество обсуждения.
- На выходе — только конспект, без преамбул и мета-комментариев, кроме точечных пометок неуверенности.
- **Зафиксируй результат.** Когда конспект готов на странице, вызови save_page_version, чтобы сохранить его как именованную версию (точку восстановления). Повторный вызов без новых правок — безвредный no-op.
## Пример стиля (фрагмент)
+2 -2
View File
@@ -33,6 +33,6 @@ bundles:
- en
roles:
- slug: researcher
version: 9
version: 10
- slug: call-summarizer
version: 1
version: 2
@@ -240,6 +240,8 @@
"Comment re-opened successfully": "Comment re-opened successfully",
"Comment unresolved successfully": "Comment unresolved successfully",
"Failed to resolve comment": "Failed to resolve comment",
"Failed to re-open comment": "Failed to re-open comment",
"Comment no longer exists": "Comment no longer exists",
"Resolve comment": "Resolve comment",
"Unresolve comment": "Unresolve comment",
"Resolve Comment Thread": "Resolve Comment Thread",
@@ -1428,5 +1430,20 @@
"Boundary": "Boundary",
"Autosave": "Autosave",
"Only versions": "Only versions",
"No saved versions yet.": "No saved versions yet."
"No saved versions yet.": "No saved versions yet.",
"Time worked on this article": "Time worked on this article",
"Show time worked on this page": "Show time worked on this page",
"Estimated time worked (inactivity gap {{gap}} min)": "Estimated time worked (inactivity gap {{gap}} min)",
"Estimate · timezone {{tz}} · inactivity gap {{gap}} min": "Estimate · timezone {{tz}} · inactivity gap {{gap}} min",
"No editing activity recorded yet.": "No editing activity recorded yet.",
"× {{count}} days without edits": "× {{count}} days without edits",
"agent: {{value}}": "agent: {{value}}",
"Work": "Work",
"Agent": "Agent",
"≈ {{hours}}h {{minutes}}m": "≈ {{hours}}h {{minutes}}m",
"≈ {{hours}}h": "≈ {{hours}}h",
"≈ {{minutes}}m": "≈ {{minutes}}m",
"{{hours}}h {{minutes}}m": "{{hours}}h {{minutes}}m",
"{{hours}}h": "{{hours}}h",
"{{minutes}}m": "{{minutes}}m"
}
@@ -240,6 +240,8 @@
"Comment re-opened successfully": "Комментарий успешно открыт повторно",
"Comment unresolved successfully": "Комментарий успешно переведён в нерешённые",
"Failed to resolve comment": "Не удалось разрешить комментарий",
"Failed to re-open comment": "Не удалось переоткрыть комментарий",
"Comment no longer exists": "Комментарий больше не существует",
"Resolve comment": "Решить комментарий",
"Unresolve comment": "Снять статус решённого с комментария",
"Resolve Comment Thread": "Решить ветку комментариев",
@@ -1443,5 +1445,20 @@
"Boundary": "Граница",
"Autosave": "Автосейв",
"Only versions": "Только версии",
"No saved versions yet.": "Пока нет сохранённых версий."
"No saved versions yet.": "Пока нет сохранённых версий.",
"Time worked on this article": "Время работы над статьёй",
"Show time worked on this page": "Показать время работы над страницей",
"Estimated time worked (inactivity gap {{gap}} min)": "Оценка времени работы (порог паузы {{gap}} мин)",
"Estimate · timezone {{tz}} · inactivity gap {{gap}} min": "Оценка · таймзона {{tz}} · порог паузы {{gap}} мин",
"No editing activity recorded yet.": "Правок пока нет.",
"× {{count}} days without edits": "× {{count}} дн. без правок",
"agent: {{value}}": "агент: {{value}}",
"Work": "Работа",
"Agent": "Агент",
"≈ {{hours}}h {{minutes}}m": "≈ {{hours}} ч {{minutes}} мин",
"≈ {{hours}}h": "≈ {{hours}} ч",
"≈ {{minutes}}m": "≈ {{minutes}} мин",
"{{hours}}h {{minutes}}m": "{{hours}} ч {{minutes}} м",
"{{hours}}h": "{{hours}} ч",
"{{minutes}}m": "{{minutes}} м"
}
@@ -1254,4 +1254,88 @@ describe("ChatThread — live reconnect + stalled", () => {
});
expect(onResumeFallback).toHaveBeenCalledWith(false); // POLL_IDLE_CAP -> idle -> disarm
});
it("#541: getRun HANGS on a live disconnect — the timeout race still enters reconnect (no silent freeze in `streaming`)", async () => {
// MUTATION-VERIFY: revert the race to a bare `getRun(cid).then/.catch` and this
// reddens — a HUNG (never-settling, NOT rejected) getRun leaves the FSM stuck in
// `streaming` with no reconnect banner and no poll (the axios client sets no
// request timeout, and the stalled-idle cap only arms AFTER reconnecting/polling).
renderLive();
h.state.getRun.mockReset();
h.state.getRun.mockReturnValue(new Promise(() => {})); // getRun HANGS forever
await disconnect(); // live partial = liveMsg (id "a2")
expect(h.state.getRun).toHaveBeenCalledWith("c1");
// BEFORE the bound fires the FSM is still in the live turn — the very freeze bug.
expect(screen.queryByText(/reconnecting/i)).toBeNull();
// The recovery-start bound fires -> the SAME fallback as the reject path.
act(() => {
vi.advanceTimersByTime(4_000);
});
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
// The live partial (a2) was DROPPED from the store (replay-from-start, no stale
// tail-apply base). Mirrors the getRun-REJECT test's inspection.
const removedLivePartial = (
h.state.setMessages as unknown as {
mock: { calls: [unknown][] };
}
).mock.calls.some(([updater]) => {
if (typeof updater !== "function") return false;
const out = (updater as (p: { id: string }[]) => { id: string }[])([
{ id: "a2" },
{ id: "u1" },
]);
return !out.some((m) => m.id === "a2");
});
expect(removedLivePartial).toBe(true);
// ...and the anchor was NULLED -> replay-from-start (no ?anchor=&n= over the partial).
advanceToAttempt(1);
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
});
it("#541: a getRun resolve AFTER the timeout already fired is IGNORED (no double reconnect, no stale re-seed)", async () => {
// The timeout wins first and enters the ladder via replay-from-start. When the
// hung getRun FINALLY answers, its late `.then` must be a full no-op: it must not
// re-seed the store from the (now stale) persisted row, must not re-set the
// anchor, and must not re-enter reconnect. The local `settled` flag makes the
// resolve/reject/timeout branches mutually exclusive.
// MUTATION-VERIFY: drop the `settled` guard (let the late `.then` run) and the
// late re-seed re-sets the anchor (URL regains ?anchor=a2&n=3) + calls setMessages.
renderLive();
let resolveGetRun!: (v: unknown) => void;
h.state.getRun.mockReset();
h.state.getRun.mockReturnValue(
new Promise((r) => {
resolveGetRun = r;
}),
);
await disconnect();
act(() => {
vi.advanceTimersByTime(4_000); // bound fires -> replay-from-start, reconnecting
});
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
advanceToAttempt(1);
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
// Anchor is null -> replay-from-start URL (the pre-condition the late resolve must
// not undo).
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
const setMessagesCallsBefore = h.state.setMessages.mock.calls.length;
// NOW the hung getRun finally resolves with a persisted anchor (id a2, steps 3).
await act(async () => {
resolveGetRun(persistedAnchor());
await Promise.resolve();
});
// The late resolve did NOT re-seed the store...
expect(h.state.setMessages.mock.calls.length).toBe(setMessagesCallsBefore);
// ...did NOT re-set the anchor (URL stays replay-from-start, no ?anchor=&n=)...
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
// ...and did NOT trigger a fresh reconnect attach.
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
});
});
@@ -81,6 +81,17 @@ const STREAM_THROTTLE_MS = 50;
// THREAD now (the FSM owns polling->stalled); the window just polls while armed.
const DEGRADED_POLL_IDLE_MAX_MS = 10 * 60_000;
// #541: the bound on the persist re-seed (getRun) wait when ENTERING the reconnect
// ladder after a live SSE drop. The axios client (lib/api-client.ts) sets NO request
// timeout, and the stalled-idle cap only arms AFTER the FSM enters reconnecting/
// polling — which never happens if getRun HANGS (connection established, no response).
// Without this bound a live drop + a hung getRun sticks the FSM in `streaming` with
// no banner and no poll until the browser socket timeout (minutes) — the silent-freeze
// class #497 eliminates. This is a deliberate RECOVERY-START bound (drop the live
// partial + enter the ladder so the poll/idle-cap arms), intentionally much shorter
// than any network socket timeout — not a network read timeout.
const RECONNECT_RESEED_TIMEOUT_MS = 4_000;
/** The #487 active (non-terminal) run statuses — mirrors the server's
* ACTIVE_RUN_STATUSES. A run-fact is "active" only for these. */
function isActiveRunStatus(status: string | null | undefined): boolean {
@@ -286,6 +297,9 @@ export default function ChatThread({
// stalled inactivity cap. Cleared by the cancelReconnect effect / the cap effect.
const reconnectTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const idleCapTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// #541: the persist re-seed (getRun) timeout race on the disconnect->reconnect
// entry. Held in a ref so unmount (DISPOSE cleanup) clears it — no dangling timer.
const reseedTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// #491 tail-only: seed EVERY persisted row unchanged (no strip). The streaming
// tail holds steps 0..N-1; the run-stream registry's tail (steps >= N) is APPENDED
@@ -748,37 +762,68 @@ export default function ChatThread({
anchorRef.current = null;
};
if (cid) {
void getRun(cid)
.then((res) => {
if (!mountedRef.current) return;
const persisted = res.message;
if (persisted && persisted.role === "assistant") {
anchorRef.current = {
id: persisted.id,
stepsPersisted: stepsPersistedOf(persisted),
};
// Replace the live partial with the persisted row IN PLACE by id —
// the re-seed from persist. The attach's tail (steps >= N) then
// appends to a store holding EXACTLY steps 0..N-1: no duplication.
setMessages((prev) => mergeById(prev, rowToUiMessage(persisted)));
} else {
// No persisted assistant row (pre-first-frame break): drop the live
// partial + replay from start (no anchor/n) so nothing is duplicated.
dropLivePartialAndReplayFromStart();
}
enterReconnect(res.run?.id ?? runId);
})
.catch(() => {
if (!mountedRef.current) return;
// Persist read FAILED: we cannot re-seed from fresh persist, and a
// stale mount-time anchor over the live partial would tail-apply
// already-present steps -> duplication (a flaky-network blip:
// SSE + getRun both fail, network recovers in ~1s, the registry still
// covers from the mount frontier). Restore the removed-filter guarantee
// instead: drop the live partial + replay from start / 204 -> poll.
// #541: bound the persist re-seed wait with a timeout race. getRun goes
// through the axios client, which has NO request timeout; a HUNG getRun
// (connection open, no response) — distinct from a REJECT, which the
// `.catch` already handles — would otherwise never let us enter the ladder,
// leaving the FSM stuck in `streaming` with no banner and no poll until the
// browser socket timeout. `settled` makes the three branches (resolve /
// reject / timeout) mutually exclusive: whichever fires FIRST wins and the
// others become no-ops. So a LATE getRun resolve AFTER the timeout is fully
// ignored — it cannot (i) re-enter reconnect a second time, (ii) overwrite
// the timeout's replay-from-start with a stale re-seed, or (iii) re-arm any
// timer. On timeout we take the SAME fallback as the reject path (drop the
// live partial + enter the ladder, so the poll and the stalled-idle cap arm).
let settled = false;
const finishReseed = (apply: () => void): void => {
if (settled) return;
settled = true;
if (reseedTimerRef.current) {
clearTimeout(reseedTimerRef.current);
reseedTimerRef.current = null;
}
if (!mountedRef.current) return;
apply();
};
reseedTimerRef.current = setTimeout(() => {
finishReseed(() => {
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
});
}, RECONNECT_RESEED_TIMEOUT_MS);
void getRun(cid)
.then((res) => {
finishReseed(() => {
const persisted = res.message;
if (persisted && persisted.role === "assistant") {
anchorRef.current = {
id: persisted.id,
stepsPersisted: stepsPersistedOf(persisted),
};
// Replace the live partial with the persisted row IN PLACE by id —
// the re-seed from persist. The attach's tail (steps >= N) then
// appends to a store holding EXACTLY steps 0..N-1: no duplication.
setMessages((prev) => mergeById(prev, rowToUiMessage(persisted)));
} else {
// No persisted assistant row (pre-first-frame break): drop the live
// partial + replay from start (no anchor/n) so nothing is duplicated.
dropLivePartialAndReplayFromStart();
}
enterReconnect(res.run?.id ?? runId);
});
})
.catch(() => {
finishReseed(() => {
// Persist read FAILED: we cannot re-seed from fresh persist, and a
// stale mount-time anchor over the live partial would tail-apply
// already-present steps -> duplication (a flaky-network blip:
// SSE + getRun both fail, network recovers in ~1s, the registry still
// covers from the mount frontier). Restore the removed-filter guarantee
// instead: drop the live partial + replay from start / 204 -> poll.
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
});
});
} else {
dropLivePartialAndReplayFromStart();
enterReconnect(runId);
@@ -903,6 +948,12 @@ export default function ChatThread({
}
return () => {
mountedRef.current = false;
// #541: clear the in-flight persist re-seed timeout (not an FSM effect timer,
// so DISPOSE does not touch it) — no dangling setTimeout after unmount.
if (reseedTimerRef.current) {
clearTimeout(reseedTimerRef.current);
reseedTimerRef.current = null;
}
dispatch({ type: "DISPOSE" }); // aborts attach + timers, bumps epoch (I5)
};
// Mount-only by design; the parent remounts per chat via `key`.
@@ -27,6 +27,7 @@ vi.mock("@/features/ai-chat/utils/markdown.ts", async () => {
import MessageItem from "./message-item";
import { messageSignature } from "@/features/ai-chat/utils/message-signature.ts";
import { splitPlainChunks } from "./streaming-plain-text";
// matchMedia (read by MantineProvider) is stubbed globally in vitest.setup.ts.
@@ -114,3 +115,89 @@ describe("MessageItem markdown memoization", () => {
expect(queryByText("streamed answer")).not.toBeNull();
});
});
// PERF SMOKE (#492): the whole point of the incremental streaming render is that
// the ANSWER path costs O(number of markdown blocks), NOT O(number of throttled
// ~20Hz ticks). Pre-#492 the finalized MarkdownPart re-parsed the WHOLE growing
// answer on every delta — a synthetic ~100 KB stream measured 394 renderChatMarkdown
// calls (one per tick). With the incremental render each STABILIZED block is parsed
// exactly once (memoized in MarkdownChunk) and the live tail is cheap plain text, so
// the call count collapses to ~= the block count regardless of tick granularity.
describe("MessageItem streaming answer render is O(blocks), not O(ticks)", () => {
// ~100 KB answer. Each section is a heading + a paragraph — TWO blank-line
// delimited markdown blocks — so the safe-cut block count is ~2× the section
// count. The perf claim is about the BLOCK count (the memoization granularity),
// measured directly with splitPlainChunks below, not the section count.
const buildAnswer = () => {
const SECTIONS = 100;
const paragraphs: string[] = [];
for (let i = 0; i < SECTIONS; i++) {
paragraphs.push(`## Section ${i}\n\n` + "lorem ipsum dolor ".repeat(55));
}
const full = paragraphs.join("\n\n");
// The number of memoized markdown blocks the incremental render splits into
// (all but the live tail are parsed once each).
return { full, blocks: splitPlainChunks(full).length };
};
const streamMsg = (text: string, state: "streaming" | "done"): UIMessage =>
({
id: "m1",
role: "assistant",
parts: [{ type: "text", text, state }],
}) as UIMessage;
it("parses each block ~once over a 100KB stream (≈blocks, ≪ ticks)", () => {
renderChatMarkdownSpy.mockClear();
const { full, blocks } = buildAnswer();
const CHUNK = 128; // a realistic ~20Hz throttled delta size
const ticks = Math.ceil(full.length / CHUNK);
let msg = streamMsg(full.slice(0, CHUNK), "streaming");
const { rerender } = render(
<MantineProvider>
<MessageItem
message={msg}
signature={messageSignature(msg)}
turnStreaming
/>
</MantineProvider>,
);
for (let end = 2 * CHUNK; end < full.length; end += CHUNK) {
msg = streamMsg(full.slice(0, end), "streaming");
rerender(
<MantineProvider>
<MessageItem
message={msg}
signature={messageSignature(msg)}
turnStreaming
/>
</MantineProvider>,
);
}
// Finalize: the streaming→done flip renders the whole answer through ONE
// canonical pass (visual parity), so the finished DOM matches the pre-#492
// output. This is the single extra parse on top of the per-block ones.
const done = streamMsg(full, "done");
rerender(
<MantineProvider>
<MessageItem message={done} signature={messageSignature(done)} />
</MantineProvider>,
);
const calls = renderChatMarkdownSpy.mock.calls.length;
// Sanity: the stream really had far more ticks than blocks (else the test is
// vacuous — the point is that calls scale with blocks, not ticks).
expect(ticks).toBeGreaterThan(blocks * 3);
// O(blocks): each stabilized block parsed once + the single final whole-text
// parse. A small constant absorbs the finalize render and the live-tail block;
// the load-bearing claim is the bound below.
expect(calls).toBeLessThanOrEqual(blocks + 2);
// ≪ ticks — and, non-vacuously, the blocks WERE parsed (not skipped entirely).
expect(calls).toBeLessThan(ticks / 3);
expect(calls).toBeGreaterThan(blocks / 2);
// MUTATION-VERIFY (documented, not run here): dropping the `memo()` wrapper on
// MarkdownChunk (so every stable block re-parses each tick) drives `calls`
// toward `ticks` (~394), reddening both upper-bound assertions above.
});
});
@@ -0,0 +1,112 @@
import { describe, expect, it, vi } from "vitest";
import { render } from "@testing-library/react";
import { MantineProvider } from "@mantine/core";
import type { UIMessage } from "@ai-sdk/react";
// Stub react-i18next (the component reads `useTranslation`). Mirrors the other
// message-item specs.
vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (key: string) => key }),
}));
import MessageItem from "./message-item";
import { messageSignature } from "@/features/ai-chat/utils/message-signature.ts";
// The REAL canonical renderer (NOT the spy the memo test installs): this file
// exercises the actual markdown output so the visual-regression assertions below
// compare against genuine HTML (incl. the schema's `<li><p>` wrappers).
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
import classes from "./ai-chat.module.css";
const msg = (
parts: UIMessage["parts"],
extra?: Partial<UIMessage>,
): UIMessage =>
({ id: "m1", role: "assistant", parts, ...extra }) as UIMessage;
const renderRow = (message: UIMessage, turnStreaming = false) =>
render(
<MantineProvider>
<MessageItem
message={message}
signature={messageSignature(message)}
turnStreaming={turnStreaming}
/>
</MantineProvider>,
);
// A rich multi-block answer that exercises headings, a list (the `<li><p>` case
// the scoped CSS tightens), inline emphasis, and multiple paragraphs.
const ANSWER = [
"# Заголовок",
"",
"Первый абзац с **жирным** и `кодом`.",
"",
"- пункт один",
"- пункт два",
"",
"Второй абзац.",
].join("\n");
describe("MessageItem final render — visual parity with the canonical pipeline", () => {
it("a finalized text part renders exactly renderChatMarkdown(text)", () => {
const { container } = renderRow(
msg([{ type: "text", text: ANSWER, state: "done" }]),
);
const block = container.querySelector(`.${classes.markdown}`);
expect(block).not.toBeNull();
// Byte-for-byte the canonical output (the SAME whole-text pass the pre-#492
// MarkdownPart produced), including `<li><p>…</p></li>` wrappers.
expect(block!.innerHTML).toBe(renderChatMarkdown(ANSWER, {}));
// The list wrapper is really present (guards against a vacuous empty render).
expect(container.querySelectorAll("li p").length).toBe(2);
});
it("the streaming incremental view CONVERGES to the canonical render on finish", () => {
// Mount mid-stream (live tail) — the DOM here is the incremental view.
const { container, rerender } = render(
<MantineProvider>
<MessageItem
message={msg([{ type: "text", text: ANSWER, state: "streaming" }])}
signature={messageSignature(
msg([{ type: "text", text: ANSWER, state: "streaming" }]),
)}
turnStreaming
/>
</MantineProvider>,
);
// Finish the turn: state flips to done AND the turn is no longer streaming.
const done = msg([{ type: "text", text: ANSWER, state: "done" }]);
rerender(
<MantineProvider>
<MessageItem message={done} signature={messageSignature(done)} />
</MantineProvider>,
);
// After finish there is exactly ONE canonical markdown container whose HTML is
// the whole-text render — identical to the non-streaming path above.
const blocks = container.querySelectorAll(`.${classes.markdown}`);
expect(blocks.length).toBe(1);
expect(blocks[0].innerHTML).toBe(renderChatMarkdown(ANSWER, {}));
});
it("neutralizeInternalLinks is honored on the finalized render", () => {
const linkAnswer = "См. [страницу](/p/abc).";
const { container } = render(
<MantineProvider>
<MessageItem
message={msg([{ type: "text", text: linkAnswer, state: "done" }])}
signature={messageSignature(
msg([{ type: "text", text: linkAnswer, state: "done" }]),
)}
neutralizeInternalLinks
/>
</MantineProvider>,
);
const block = container.querySelector(`.${classes.markdown}`);
expect(block!.innerHTML).toBe(
renderChatMarkdown(linkAnswer, { neutralizeInternalLinks: true }),
);
// The internal link was made inert (no href) by the neutralization flag.
const a = container.querySelector("a");
expect(a?.hasAttribute("href")).toBe(false);
});
});
@@ -4,6 +4,7 @@ import { useTranslation } from "react-i18next";
import type { UIMessage } from "@ai-sdk/react";
import ToolCallCard from "@/features/ai-chat/components/tool-call-card.tsx";
import ReasoningBlock from "@/features/ai-chat/components/reasoning-block.tsx";
import { StreamingMarkdownText } from "@/features/ai-chat/components/streaming-markdown-text.tsx";
import ChatErrorAlert from "@/features/ai-chat/components/chat-error-alert.tsx";
import ChatStoppedNotice from "@/features/ai-chat/components/chat-stopped-notice.tsx";
import { ToolUiPart, isToolPart } from "@/features/ai-chat/utils/tool-parts.tsx";
@@ -86,17 +87,39 @@ interface MessageItemProps {
* One assistant text part rendered as sanitized markdown. Memoized on its inputs
* so a finalized text part is NOT re-parsed on every streamed delta: during a
* turn only the actively-growing tail part changes its `text`, so every earlier
* part hits the memo and skips the expensive marked + DOMPurify pass. Props are
* primitives, so React.memo's default shallow compare is exactly right (the
* `text` string is compared by value).
* part hits the memo and skips the expensive canonical parse + DOMPurify pass.
* Props are primitives, so React.memo's default shallow compare is exactly right
* (the `text` string is compared by value).
*
* Streaming gate (#492) mirrors ReasoningBlock:
* - `streaming` (this is the live, actively-growing tail part of an in-flight
* turn): render incrementally via StreamingMarkdownText the stabilized blocks
* go through the canonical pipeline (each parsed ONCE, memoized) and only the
* live tail is cheap plain text. This makes the per-tick cost O(new blocks),
* not the pre-#492 O(ticks) whole-answer re-parse on every ~20Hz delta.
* - finalized (the common case, and the turn-end flip): render the WHOLE text
* through ONE canonical pass byte-identical to the pre-#492 output (visual
* parity). The row re-renders on the streamingdone flip because
* `messageSignature` tracks each part's `state` (and `turnStreaming` flips at
* turn end), so the incremental view always converges to this single render.
*/
const MarkdownPart = memo(function MarkdownPart({
text,
neutralizeInternalLinks,
streaming,
}: {
text: string;
neutralizeInternalLinks: boolean;
streaming: boolean;
}) {
if (streaming) {
return (
<StreamingMarkdownText
text={text}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
);
}
const html = renderChatMarkdown(text, { neutralizeInternalLinks });
if (html) {
return (
@@ -179,47 +202,10 @@ function MessageItem({
{resolveAssistantName(assistantName) ?? t("AI agent")}
</Text>
{message.parts.map((part, index) => {
if (part.type === "reasoning") {
// Reasoning ("thinking") -> a collapsible block with its own token
// count. Empty/whitespace reasoning with no authoritative count carries
// nothing to show, so skip it (avoids an empty 0-token block).
const text = (part as { text?: string }).text ?? "";
if (!text.trim() && !(reasoningTokens && reasoningTokens > 0))
return null;
// Absent state (persisted rows) and "done" both mean finalized.
// `messageSignature` already includes each part's `state`, so the
// streaming→done flip changes the row signature and re-renders this
// row — which is what lets ReasoningBlock switch from chunked plain
// text to its one-time markdown parse (see reasoning-block.tsx).
// ALSO require the turn to be live: a part stranded at
// `state:"streaming"` after the turn ended (no `reasoning-end` — see
// the `turnStreaming` prop doc) must still finalize and parse.
const streaming =
turnStreaming && (part as { state?: string }).state === "streaming";
return (
<ReasoningBlock
key={index}
text={text}
tokens={reasoningTokens}
streaming={streaming}
/>
);
}
if (part.type === "text") {
// Skip empty/whitespace-only text parts (a streaming message often
// starts with an empty text part before the first token arrives); the
// typing indicator covers that gap until real content streams in.
if (!part.text.trim()) return null;
return (
<MarkdownPart
key={index}
text={part.text}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
);
}
// Tool parts (`tool-*` / `dynamic-tool`) are template-literal kinds, so
// they cannot be a `switch` case; the runtime guard handles them, and the
// switch below covers every CLOSED (literal-typed) part kind with a
// compile-time exhaustiveness check in its default.
if (isToolPart(part.type)) {
return (
<ToolCallCard
@@ -232,7 +218,76 @@ function MessageItem({
);
}
return null;
switch (part.type) {
case "reasoning": {
// Reasoning ("thinking") -> a collapsible block with its own token
// count. Empty/whitespace reasoning with no authoritative count
// carries nothing to show, so skip it (avoids an empty 0-token block).
const text = part.text ?? "";
if (!text.trim() && !(reasoningTokens && reasoningTokens > 0))
return null;
// Absent state (persisted rows) and "done" both mean finalized.
// `messageSignature` already includes each part's `state`, so the
// streaming→done flip changes the row signature and re-renders this
// row — which is what lets ReasoningBlock switch from chunked plain
// text to its one-time markdown parse (see reasoning-block.tsx).
// ALSO require the turn to be live: a part stranded at
// `state:"streaming"` after the turn ended (no `reasoning-end` — see
// the `turnStreaming` prop doc) must still finalize and parse.
const streaming = turnStreaming && part.state === "streaming";
return (
<ReasoningBlock
key={index}
text={text}
tokens={reasoningTokens}
streaming={streaming}
/>
);
}
case "text": {
// Skip empty/whitespace-only text parts (a streaming message often
// starts with an empty text part before the first token arrives); the
// typing indicator covers that gap until real content streams in.
if (!part.text.trim()) return null;
// The live, actively-growing tail part of the in-flight turn renders
// incrementally (see MarkdownPart); a finalized part (persisted, or
// the turn-end flip) renders the whole text through one canonical
// pass. Same liveness rule as the reasoning branch above.
const streaming = turnStreaming && part.state === "streaming";
return (
<MarkdownPart
key={index}
text={part.text}
neutralizeInternalLinks={neutralizeInternalLinks}
streaming={streaming}
/>
);
}
case "source-url":
case "source-document":
case "file":
case "step-start":
// Not surfaced in the chat bubble (v1) — same as the pre-#492 default.
return null;
default: {
// Compile-time exhaustiveness over the CLOSED union members: every
// literal-typed part kind is handled above, so the only kinds that
// can reach here are the OPEN template-literal ones (`tool-*` — caught
// by the guard at runtime — and `data-*`) plus `dynamic-tool`. Adding
// a NEW closed part kind to UIMessagePart makes this assignment fail
// to compile, forcing it to be handled instead of silently ignored
// (this replaces the pre-#492 fall-through `return null` + WARNING).
const _exhaustive:
| `tool-${string}`
| "dynamic-tool"
| `data-${string}` = part.type;
void _exhaustive;
return null;
}
}
})}
{/* A persisted turn error (server stored it in metadata.error). Rendered
here so it survives a thread remount and shows in reopened history. */}
@@ -0,0 +1,96 @@
import { memo, useMemo } from "react";
import { splitPlainChunks } from "@/features/ai-chat/components/streaming-plain-text.tsx";
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
import classes from "@/features/ai-chat/components/ai-chat.module.css";
/**
* One STABILIZED markdown block, rendered through the canonical pipeline and
* memoized on its string prop. During streaming only the TAIL chunk grows (the
* `splitPlainChunks` append-only invariant guarantees every earlier chunk is
* byte-identical across deltas), so React skips every stable block and each one
* is parsed by `renderChatMarkdown` EXACTLY ONCE turning the pre-#492
* "re-parse the whole accumulated answer on every ~20Hz tick" (O(ticks)) into
* O(number of blocks). The markup is DOMPurify-sanitized inside renderChatMarkdown
* before it reaches `dangerouslySetInnerHTML`.
*
* NOTE (transient streaming-only artifact): a safe cut is a blank-line boundary,
* so a construct that legitimately contains a blank line (e.g. a fenced code block
* with an empty line) can be split across chunks and render oddly WHILE it is still
* streaming. This is cosmetic and self-heals: the moment the part finalizes,
* MarkdownPart renders the WHOLE text through one canonical pass (visual parity
* with the pre-#492 output). The reasoning path makes the same trade (plain text
* while streaming, one markdown parse at the end).
*/
const MarkdownChunk = memo(function MarkdownChunk({
text,
neutralizeInternalLinks,
}: {
text: string;
neutralizeInternalLinks: boolean;
}) {
const html = renderChatMarkdown(text, { neutralizeInternalLinks });
if (html) {
return (
<div
className={classes.markdown}
// Sanitized by renderChatMarkdown (DOMPurify) before insertion.
dangerouslySetInnerHTML={{ __html: html }}
/>
);
}
// Malformed/unsupported markdown could not render synchronously: raw text.
return (
<div className={classes.markdown} style={{ whiteSpace: "pre-wrap" }}>
{text}
</div>
);
});
/**
* The cheap streaming-time stand-in for the finalized answer's one-time markdown
* parse (see MarkdownPart in message-item.tsx). Mirrors StreamingPlainText's
* chunked-memo pattern but renders the STABILIZED prefix as real markdown (each
* block parsed once, memoized) and only the LIVE tail as flat plain text so the
* user sees formatted output for everything up to the last safe cut, and the not-
* yet-stable tail (which markdown-parsing every tick would make O(ticks)) stays a
* single cheap escaped text node until it stabilizes into a new block.
*
* `splitPlainChunks` yields chunks where, under append-only growth, every chunk
* except the LAST is immutable; the last chunk is the live tail. Index keys are
* therefore stable (a given index never changes to a different chunk's content).
*/
export function StreamingMarkdownText({
text,
neutralizeInternalLinks,
}: {
text: string;
neutralizeInternalLinks: boolean;
}) {
const chunks = useMemo(() => splitPlainChunks(text), [text]);
return (
<>
{chunks.map((chunk, index) =>
index < chunks.length - 1 ? (
<MarkdownChunk
key={index}
text={chunk}
neutralizeInternalLinks={neutralizeInternalLinks}
/>
) : (
// The live tail: flat, React-escaped plain text (no markdown parse, no
// sanitizer, no innerHTML). `pre-wrap` preserves its newlines; trailing
// separator newlines are dropped at display time so the block gap comes
// from the markdown margins, not a doubled empty line (mirrors
// PlainChunk in streaming-plain-text.tsx).
<div
key={index}
className={classes.markdown}
style={{ whiteSpace: "pre-wrap" }}
>
{chunk.replace(/\n+$/, "")}
</div>
),
)}
</>
);
}
@@ -27,11 +27,15 @@ vi.mock("@/features/space/queries/space-query.ts", () => ({
import {
buildChildrenByParent,
CommentEditorWithActions,
sortResolvedByResolvedAt,
} from "./comment-list-with-tabs";
const c = (id: string, parentCommentId: string | null = null): IComment =>
({ id, parentCommentId }) as IComment;
const resolvedAtComment = (id: string, resolvedAt: unknown): IComment =>
({ id, resolvedAt }) as unknown as IComment;
describe("buildChildrenByParent (childrenByParent grouping)", () => {
it("returns an empty map for undefined or empty input", () => {
expect(buildChildrenByParent(undefined).size).toBe(0);
@@ -71,6 +75,48 @@ describe("buildChildrenByParent (childrenByParent grouping)", () => {
});
});
describe("sortResolvedByResolvedAt (Resolved tab order, #542)", () => {
it("orders by resolvedAt DESC — newest resolve first — with ISO-STRING values", () => {
// At runtime resolvedAt is an ISO string (axios JSON / WS subscription), so
// the sort must coerce with new Date(...) before .getTime().
const older = resolvedAtComment("older", "2026-07-10T10:00:00.000Z");
const newest = resolvedAtComment("newest", "2026-07-12T10:00:00.000Z");
const middle = resolvedAtComment("middle", "2026-07-11T10:00:00.000Z");
const out = sortResolvedByResolvedAt([older, newest, middle]);
expect(out.map((x) => x.id)).toEqual(["newest", "middle", "older"]);
});
it("also handles Date instances (optimistic onMutate window)", () => {
const older = resolvedAtComment("older", new Date("2026-01-01T00:00:00Z"));
const newer = resolvedAtComment("newer", new Date("2026-06-01T00:00:00Z"));
expect(sortResolvedByResolvedAt([older, newer]).map((x) => x.id)).toEqual([
"newer",
"older",
]);
});
it("does not mutate the input array", () => {
const a = resolvedAtComment("a", "2026-01-01T00:00:00.000Z");
const b = resolvedAtComment("b", "2026-02-01T00:00:00.000Z");
const input = [a, b];
sortResolvedByResolvedAt(input);
expect(input.map((x) => x.id)).toEqual(["a", "b"]);
});
it("keeps stable order for equal resolvedAt timestamps", () => {
const ts = "2026-03-03T03:03:03.000Z";
const x = resolvedAtComment("x", ts);
const y = resolvedAtComment("y", ts);
const z = resolvedAtComment("z", ts);
expect(sortResolvedByResolvedAt([x, y, z]).map((c) => c.id)).toEqual([
"x",
"y",
"z",
]);
});
});
function renderReplyEditor() {
return render(
<MantineProvider>
@@ -53,6 +53,22 @@ export function buildChildrenByParent(
return m;
}
// Sort the Resolved tab by resolve time, newest first, on a COPY (never mutate
// the react-query cache array). `resolvedAt` is typed `Date` but at runtime it
// is an ISO STRING (from the axios-JSON onSuccess and the WS subscription) — a
// real Date only during the optimistic onMutate window — so it MUST be coerced
// with `new Date(...)` before `.getTime()`, or a raw `.getTime()` on the string
// throws / yields NaN. ES2019's stable sort preserves order for equal
// timestamps. Callers pass a list already filtered to a truthy `resolvedAt`, so
// the non-null assertion is safe.
// Exported for unit testing.
export function sortResolvedByResolvedAt(resolved: IComment[]): IComment[] {
return [...resolved].sort(
(a, b) =>
new Date(b.resolvedAt!).getTime() - new Date(a.resolvedAt!).getTime(),
);
}
function CommentListWithTabs({ onClose }: CommentListWithTabsProps) {
const { t } = useTranslation();
const { pageSlug } = useParams();
@@ -91,7 +107,10 @@ function CommentListWithTabs({ onClose }: CommentListWithTabsProps) {
(comment: IComment) => comment.resolvedAt,
);
return { activeComments: active, resolvedComments: resolved };
return {
activeComments: active,
resolvedComments: sortResolvedByResolvedAt(resolved),
};
}, [comments]);
// Index replies by their parent once, instead of an O(n^2) filter per thread.
@@ -0,0 +1,353 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import React from "react";
import { renderHook, waitFor } from "@testing-library/react";
import {
QueryClient,
QueryClientProvider,
InfiniteData,
} from "@tanstack/react-query";
/**
* Coverage for the resolve/reopen mutation (#542): the Undo-in-toast reopen and
* its double-click guard, the terminal 404 branch (drop from cache + clear the
* inline mark, no rollback), and the directional error copy.
*/
// A fake TipTap editor injected via the mocked pageEditorAtom, so we can assert
// the mutation clears the inline comment mark (unsetComment / setCommentResolved).
const editorMock = vi.hoisted(() => ({
current: {
isDestroyed: false,
commands: { unsetComment: vi.fn(), setCommentResolved: vi.fn() },
} as {
isDestroyed: boolean;
commands: {
unsetComment: (id: string) => void;
setCommentResolved: (id: string, v: boolean) => void;
};
} | null,
}));
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn(), hide: vi.fn() },
}));
vi.mock("jotai", () => ({
atom: (v: unknown) => v,
useAtomValue: () => editorMock.current,
}));
vi.mock("@/features/comment/services/comment-service", () => ({
applySuggestion: vi.fn(),
dismissSuggestion: vi.fn(),
createComment: vi.fn(),
updateComment: vi.fn(),
deleteComment: vi.fn(),
resolveComment: vi.fn(),
getPageComments: vi.fn(),
}));
import { notifications } from "@mantine/notifications";
import { resolveComment } from "@/features/comment/services/comment-service";
import {
useResolveCommentMutation,
RESOLVE_UNDO_AUTOCLOSE_MS,
RQ_KEY,
} from "@/features/comment/queries/comment-query";
import { IComment } from "@/features/comment/types/comment.types";
const PAGE_ID = "page-1";
function seededClient(comment: IComment) {
const queryClient = new QueryClient({
defaultOptions: { mutations: { retry: false } },
});
const seed: InfiniteData<any> = {
pageParams: [undefined],
pages: [
{ items: [comment], meta: { hasNextPage: false, nextCursor: null } },
],
};
queryClient.setQueryData(RQ_KEY(PAGE_ID), seed);
const wrapper = ({ children }: { children: React.ReactNode }) => (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
);
return { queryClient, wrapper };
}
function items(queryClient: QueryClient): IComment[] {
const cache = queryClient.getQueryData(RQ_KEY(PAGE_ID)) as
| InfiniteData<any>
| undefined;
return cache?.pages.flatMap((p) => p.items) ?? [];
}
const comment = (over?: Partial<IComment>): IComment =>
({
id: "c-1",
pageId: PAGE_ID,
content: "{}",
creatorId: "u-1",
workspaceId: "ws-1",
createdAt: new Date(),
resolvedAt: null,
...over,
}) as IComment;
// Pull the inline Undo button's onClick out of the success toast's message tree.
function undoOnClickFromToast(): () => void {
const call = vi
.mocked(notifications.show)
.mock.calls.map((c) => c[0])
.find((arg: any) => arg?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS);
expect(call).toBeTruthy();
const message: any = (call as any).message;
// message = Group( Text, Button ); grab the Button element's onClick.
const children = message.props.children as any[];
const button = children[1];
return button.props.onClick;
}
describe("useResolveCommentMutation — Undo toast (#542)", () => {
beforeEach(() => {
vi.clearAllMocks();
editorMock.current = {
isDestroyed: false,
commands: { unsetComment: vi.fn(), setCommentResolved: vi.fn() },
};
});
it("resolve shows an Undo toast with autoClose=10000ms; reopen shows NO Undo", async () => {
vi.mocked(resolveComment).mockImplementation(async (data) =>
comment({
resolvedAt: data.resolved ? (new Date() as any) : null,
}),
);
const { wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const resolveToast = vi
.mocked(notifications.show)
.mock.calls.map((c) => c[0])
.find((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS);
expect(resolveToast).toBeTruthy();
expect((resolveToast as any).id).toBe("resolve-undo-c-1");
expect((resolveToast as any).autoClose).toBe(10000);
// Now a reopen → plain toast, no autoClose/Undo, no id.
vi.clearAllMocks();
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: false,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const calls = vi.mocked(notifications.show).mock.calls.map((c) => c[0]);
expect(
calls.some((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS),
).toBe(false);
expect(calls).toContainEqual({ message: "Comment re-opened successfully" });
});
it("double/fast Undo click fires reopen EXACTLY once (guard)", async () => {
vi.mocked(resolveComment).mockImplementation(async (data) =>
comment({ resolvedAt: data.resolved ? (new Date() as any) : null }),
);
const { wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
const onClick = undoOnClickFromToast();
// Fire twice synchronously (notifications.hide is not synchronous).
onClick();
onClick();
await waitFor(() => {
const reopenCalls = vi
.mocked(resolveComment)
.mock.calls.filter(([d]) => d.resolved === false);
expect(reopenCalls).toHaveLength(1);
});
// The mark was cleared once via setCommentResolved(id, false).
expect(editorMock.current!.commands.setCommentResolved).toHaveBeenCalledWith(
"c-1",
false,
);
// The toast was hidden.
expect(notifications.hide).toHaveBeenCalledWith("resolve-undo-c-1");
});
it("404 → drops the comment from cache, clears the inline mark, no rollback, no Undo", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 404 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
// Removed from cache (NOT rolled back to a phantom).
expect(items(queryClient)).toHaveLength(0);
// Inline mark cleared via unsetComment (mandatory — no panel row left to do it).
expect(editorMock.current!.commands.unsetComment).toHaveBeenCalledWith(
"c-1",
);
// Neutral message, red, and crucially NOT the success copy and NO Undo toast.
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment no longer exists",
color: "red",
});
const calls = vi.mocked(notifications.show).mock.calls.map((c) => c[0]);
expect(
calls.some((a: any) => a?.autoClose === RESOLVE_UNDO_AUTOCLOSE_MS),
).toBe(false);
});
it("404 does not crash when the editor is gone (read-only / panel closed)", async () => {
editorMock.current = null;
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 404 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(items(queryClient)).toHaveLength(0);
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment no longer exists",
color: "red",
});
});
it("non-404 error on REOPEN shows 'Failed to re-open comment' and rolls back", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
// Seed a RESOLVED comment (the reopen target).
const resolved = comment({ resolvedAt: new Date() as any });
const { queryClient, wrapper } = seededClient(resolved);
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: false })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(notifications.show).toHaveBeenCalledWith({
message: "Failed to re-open comment",
color: "red",
});
expect(notifications.show).not.toHaveBeenCalledWith(
expect.objectContaining({ message: "Failed to resolve comment" }),
);
// Rolled back: the comment is still present and still resolved.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeTruthy();
});
it("reopen via Undo FAILS (non-404) → inline mark is NOT left cleared (doc↔panel stay consistent)", async () => {
// First resolve succeeds → produces the Undo toast (no mark change on resolve).
vi.mocked(resolveComment).mockResolvedValueOnce(
comment({ resolvedAt: new Date() as any }),
);
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: true,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
// Now the reopen fired by Undo fails with a 500.
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
const onClick = undoOnClickFromToast();
onClick();
await waitFor(() => expect(result.current.isError).toBe(true));
// Core F1 guarantee: the mark-clear now lives in the reopen onSuccess, so a
// FAILED reopen must never flip the inline mark to unresolved — otherwise the
// doc would show an active highlight the panel still treats as resolved and
// the collab mark would diverge with nothing committed on the server.
expect(
editorMock.current!.commands.setCommentResolved,
).not.toHaveBeenCalledWith("c-1", false);
// Cache rolled back: the comment stays resolved and present.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeTruthy();
});
it("reopen success with a null editorRef degrades gracefully (no throw, no-op)", async () => {
// Read-only view / panel closed: pageEditorAtom is null on the success path.
editorMock.current = null;
vi.mocked(resolveComment).mockResolvedValue(comment({ resolvedAt: null }));
const resolved = comment({ resolvedAt: new Date() as any });
const { queryClient, wrapper } = seededClient(resolved);
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current.mutateAsync({
commentId: "c-1",
pageId: PAGE_ID,
resolved: false,
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
// No crash from the reopen mark-clear; the plain reopen toast is still shown.
expect(notifications.show).toHaveBeenCalledWith({
message: "Comment re-opened successfully",
});
// Cache updated to reopened (resolvedAt cleared by the server payload).
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeFalsy();
});
it("non-404 error on RESOLVE shows 'Failed to resolve comment' and rolls back", async () => {
vi.mocked(resolveComment).mockRejectedValue({ response: { status: 500 } });
const { queryClient, wrapper } = seededClient(comment());
const { result } = renderHook(() => useResolveCommentMutation(), {
wrapper,
});
await result.current
.mutateAsync({ commentId: "c-1", pageId: PAGE_ID, resolved: true })
.catch(() => undefined);
await waitFor(() => expect(result.current.isError).toBe(true));
expect(notifications.show).toHaveBeenCalledWith({
message: "Failed to resolve comment",
color: "red",
});
// Rolled back to open (previousCache), still present.
expect(items(queryClient)).toHaveLength(1);
expect(items(queryClient)[0].resolvedAt).toBeFalsy();
});
});
@@ -20,12 +20,19 @@ import {
ISuggestionOutcome,
} from "@/features/comment/types/comment.types";
import { notifications } from "@mantine/notifications";
import { Button, Group, Text } from "@mantine/core";
import { IPagination } from "@/lib/types.ts";
import { useTranslation } from "react-i18next";
import { useEffect, useMemo } from "react";
import React, { useEffect, useMemo, useRef } from "react";
import { useAtomValue } from "jotai";
import { pageEditorAtom } from "@/features/editor/atoms/editor-atoms";
export const RQ_KEY = (pageId: string) => ["comments", pageId];
// How long the resolve success toast (with its inline Undo) stays up before it
// auto-closes. Policy constant — no env override.
export const RESOLVE_UNDO_AUTOCLOSE_MS = 10000;
export function useCommentsQuery(params: ICommentParams) {
const query = useInfiniteQuery({
queryKey: RQ_KEY(params.pageId),
@@ -376,7 +383,25 @@ export function useResolveCommentMutation() {
const queryClient = useQueryClient();
const { t } = useTranslation();
return useMutation({
// Keep the live editor in a ref: the toast's Undo (and the 404 branch) must
// clear the inline comment mark AFTER the originating CommentListItem has
// unmounted (resolving pulls the comment out of the Open list, so its item is
// already gone by the time the 10s toast is clicked). In read-only view
// pageEditorAtom is null and the mark converges via the server's
// COMMENT_MARK_UPDATE job instead.
const editor = useAtomValue(pageEditorAtom);
const editorRef = useRef(editor);
editorRef.current = editor;
// Self-reference the mutation so the toast's Undo can re-invoke it (reopen)
// long after the triggering component unmounted. Declared BEFORE useMutation
// and assigned AFTER; the onClick reads mutationRef.current at CALL time, not
// definition time, so there is no initialization cycle.
const mutationRef = useRef<{
mutate: (vars: IResolveComment) => void;
} | null>(null);
const mutation = useMutation({
mutationFn: (data: IResolveComment) => resolveComment(data),
onMutate: async (variables) => {
await queryClient.cancelQueries({ queryKey: RQ_KEY(variables.pageId) });
@@ -401,7 +426,39 @@ export function useResolveCommentMutation() {
return { previousCache };
},
onError: (_err, variables, context) => {
onError: (err: any, variables, context) => {
// Terminal 404: the comment was really deleted (missing comment or deleted
// page — access denial is 403, resolve is idempotent so no 400). Do NOT
// roll back (that would resurrect a phantom row in Resolved); instead drop
// it from the cache and clear its now-orphaned inline mark. Mirrors
// handleDeleteComment and the dismiss-mutation 404 branch.
if (err?.response?.status === 404) {
const cache = queryClient.getQueryData(RQ_KEY(variables.pageId)) as
| InfiniteData<IPagination<IComment>>
| undefined;
if (cache) {
queryClient.setQueryData(
RQ_KEY(variables.pageId),
removeCommentFromCache(cache, variables.commentId),
);
}
const ed = editorRef.current;
if (ed && !ed.isDestroyed) {
try {
ed.commands.unsetComment(variables.commentId);
} catch {
/* editor gone / mark already removed */
}
}
notifications.show({
message: t("Comment no longer exists"),
color: "red",
});
return;
}
// Generic failure: roll back the optimistic update and show a DIRECTIONAL
// error (resolve vs. reopen), not always "resolve".
if (context?.previousCache) {
queryClient.setQueryData(
RQ_KEY(variables.pageId),
@@ -409,7 +466,9 @@ export function useResolveCommentMutation() {
);
}
notifications.show({
message: t("Failed to resolve comment"),
message: variables.resolved
? t("Failed to resolve comment")
: t("Failed to re-open comment"),
color: "red",
});
},
@@ -430,11 +489,72 @@ export function useResolveCommentMutation() {
);
}
// Reopen keeps the plain toast without an Undo.
if (!variables.resolved) {
// Clear the inline mark ONLY after the server confirms the reopen, so a
// failed reopen never leaves an active highlight the panel still treats
// as resolved. Mirrors the 404 branch's editor-liveness guard/try-catch.
// The button-triggered reopen already set the mark, so this is an
// idempotent no-op there.
const ed = editorRef.current;
if (ed && !ed.isDestroyed) {
try {
ed.commands.setCommentResolved(variables.commentId, false);
} catch {
/* editor gone — server COMMENT_MARK_UPDATE converges it */
}
}
notifications.show({ message: t("Comment re-opened successfully") });
return;
}
// Resolve: attach an inline Undo (reopen) to the success toast. Built with
// React.createElement because this is a .ts module (no JSX).
const { commentId, pageId } = variables;
const notificationId = `resolve-undo-${commentId}`;
// Double-click guard: notifications.hide is NOT synchronous, so the button
// stays clickable for a frame or two — without this a fast double-click
// would fire reopen twice.
let done = false;
notifications.show({
message: variables.resolved
? t("Comment resolved successfully")
: t("Comment re-opened successfully"),
id: notificationId,
autoClose: RESOLVE_UNDO_AUTOCLOSE_MS,
message: React.createElement(
Group,
{ justify: "space-between", wrap: "nowrap", gap: "md" },
React.createElement(
Text,
{ size: "sm" },
t("Comment resolved successfully"),
),
React.createElement(
Button,
{
variant: "subtle",
size: "compact-sm",
onClick: () => {
if (done) return;
done = true;
// Reopen via the SAME mutation (read at click time — the
// originating item is already unmounted).
mutationRef.current?.mutate({
commentId,
pageId,
resolved: false,
});
// The inline mark is cleared in the reopen mutation's onSuccess
// (bound to server confirmation), NOT here — clearing it eagerly
// would desync the doc from the panel if reopen then fails.
notifications.hide(notificationId);
},
},
t("Undo"),
),
),
});
},
});
mutationRef.current = mutation;
return mutation;
}
@@ -0,0 +1,45 @@
import { describe, it, expect } from "vitest";
import {
formatHeadline,
formatDayTotal,
formatGapMinutes,
} from "./format-work-time";
const MIN = 60 * 1000;
// Fake translator: renders the key with {{tokens}} substituted, so the tests
// assert the rounding + branch selection without depending on the i18n catalogue.
const t = (key: string, opts?: Record<string, unknown>) =>
key.replace(/\{\{(\w+)\}\}/g, (_, k) => String(opts?.[k] ?? ""));
describe("formatHeadline", () => {
it("prefixes ≈ and rounds to a 5-minute step", () => {
expect(formatHeadline(4 * 60 * MIN + 27 * MIN, t)).toBe("≈ 4h 25m");
expect(formatHeadline(90 * MIN, t)).toBe("≈ 1h 30m");
});
it("shows hours only / minutes only cleanly", () => {
expect(formatHeadline(120 * MIN, t)).toBe("≈ 2h");
expect(formatHeadline(35 * MIN, t)).toBe("≈ 35m");
});
it("floors a tiny non-zero estimate to 5m, never 0", () => {
expect(formatHeadline(2 * MIN, t)).toBe("≈ 5m");
});
it("empty string for zero (widget hidden)", () => {
expect(formatHeadline(0, t)).toBe("");
});
});
describe("formatDayTotal", () => {
it('renders "h m" and shows — for empty days', () => {
expect(formatDayTotal(3 * 60 * MIN + 17 * MIN, t)).toBe("3h 17m");
expect(formatDayTotal(0, t)).toBe("—");
});
});
describe("formatGapMinutes", () => {
it("converts the tGap ms threshold to whole minutes", () => {
expect(formatGapMinutes(15 * MIN)).toBe(15);
});
});
@@ -0,0 +1,45 @@
// #395 — display formatting for the work-time estimate. Pure functions that take
// a translator so ru-RU / en-US wording lives in the i18n catalogue and the
// rounding logic stays unit-testable.
type Translate = (key: string, opts?: Record<string, unknown>) => string;
const MIN = 60 * 1000;
function hm(totalMinutes: number): { hours: number; minutes: number } {
return {
hours: Math.floor(totalMinutes / 60),
minutes: totalMinutes % 60,
};
}
/**
* Headline number (§6.1): an ESTIMATE, so rounded to a coarse 5-minute step and
* prefixed with "≈". A non-zero-but-tiny estimate floors to 5m rather than
* rounding down to "0" (which would read as "no work"). Zero empty string
* (the caller hides the widget).
*/
export function formatHeadline(workMs: number, t: Translate): string {
if (workMs <= 0) return "";
let minutes = Math.round(workMs / MIN / 5) * 5;
if (minutes === 0) minutes = 5;
const { hours, minutes: m } = hm(minutes);
if (hours > 0 && m > 0) return t("≈ {{hours}}h {{minutes}}m", { hours, minutes: m });
if (hours > 0) return t("≈ {{hours}}h", { hours });
return t("≈ {{minutes}}m", { minutes: m });
}
/** Per-day sum (§6.2), rounded to the minute. Zero → "—". */
export function formatDayTotal(activeMs: number, t: Translate): string {
if (activeMs <= 0) return "—";
const minutes = Math.max(1, Math.round(activeMs / MIN));
const { hours, minutes: m } = hm(minutes);
if (hours > 0 && m > 0) return t("{{hours}}h {{minutes}}m", { hours, minutes: m });
if (hours > 0) return t("{{hours}}h", { hours });
return t("{{minutes}}m", { minutes: m });
}
/** The inactivity threshold, for the "estimate · gap = N min" caption. */
export function formatGapMinutes(tGapMs: number): number {
return Math.round(tGapMs / MIN);
}
@@ -0,0 +1,25 @@
import { useQuery, UseQueryResult } from "@tanstack/react-query";
import { IPageWorkTime } from "./work-time.types";
import { getPageWorkTime, viewerTimezone } from "./work-time-service";
const WORK_TIME_STALE_TIME = 5 * 60 * 1000;
/**
* #395 the "time worked on this article" estimate + per-day punch-card
* buckets. The buckets are computed server-side in the viewer's timezone (so a
* midnight-crossing session lands on the right calendar day for the reader).
* `enabled` is opt-in so the (cheap but non-trivial) projection query only fires
* when the number is actually shown.
*/
export function usePageWorkTime(
pageId: string,
enabled = true,
): UseQueryResult<IPageWorkTime, Error> {
const tz = viewerTimezone();
return useQuery({
queryKey: ["page-work-time", pageId, tz],
queryFn: () => getPageWorkTime(pageId, tz),
enabled: enabled && !!pageId,
staleTime: WORK_TIME_STALE_TIME,
});
}
@@ -0,0 +1,171 @@
import { Group, Stack, Text } from "@mantine/core";
import { useTranslation } from "react-i18next";
import { useMemo } from "react";
import { IPageWorkTime, IPerDay, IDayWindow } from "./work-time.types";
import {
formatDayTotal,
formatGapMinutes,
formatHeadline,
} from "./format-work-time";
import classes from "./work-time.module.css";
const DAY_MS = 24 * 60 * 60 * 1000;
// Collapse a run of this many (or more) consecutive edit-free days into a single
// "× N days" separator (§6.2 long-range) — the row is still always one day.
const EMPTY_RUN_COLLAPSE = 8;
type Row =
| { type: "day"; day: IPerDay }
| { type: "gap"; count: number };
function collapseEmptyRuns(perDay: IPerDay[]): Row[] {
const rows: Row[] = [];
let emptyRun: IPerDay[] = [];
const flush = () => {
if (emptyRun.length >= EMPTY_RUN_COLLAPSE) {
rows.push({ type: "gap", count: emptyRun.length });
} else {
for (const d of emptyRun) rows.push({ type: "day", day: d });
}
emptyRun = [];
};
for (const d of perDay) {
if (d.activeMs === 0 && d.agentMs === 0) {
emptyRun.push(d);
} else {
flush();
rows.push({ type: "day", day: d });
}
}
flush();
return rows;
}
function dayHeading(day: number): string {
return new Date(day).toLocaleDateString(undefined, {
weekday: "short",
day: "numeric",
month: "short",
});
}
function DayTrack({
day,
pSingle,
}: {
day: IPerDay;
pSingle: number;
}) {
const { t } = useTranslation();
const ticks = [6, 12, 18];
return (
<div className={classes.row}>
<span className={classes.dayLabel}>{dayHeading(day.day)}</span>
<div className={classes.track}>
{ticks.map((h) => (
<div
key={h}
className={classes.hourTick}
style={{ left: `${(h / 24) * 100}%` }}
/>
))}
{day.windows.map((w: IDayWindow, i) => {
const leftPct = ((w.start - day.day) / DAY_MS) * 100;
const widthPct = ((w.end - w.start) / DAY_MS) * 100;
const isSingle = w.end - w.start <= pSingle;
const cls = [
classes.window,
w.class === "work" ? classes.windowWork : classes.windowAgent,
isSingle ? classes.windowSingle : "",
].join(" ");
return (
<div
key={i}
className={cls}
style={{
left: `${Math.max(0, Math.min(100, leftPct))}%`,
width: `${Math.max(0, Math.min(100, widthPct))}%`,
}}
/>
);
})}
</div>
<span className={classes.daySum}>
{formatDayTotal(day.activeMs, t)}
</span>
</div>
);
}
interface Props {
data: IPageWorkTime;
}
export default function WorkTimePunchCard({ data }: Props) {
const { t } = useTranslation();
const rows = useMemo(() => collapseEmptyRuns(data.perDay), [data.perDay]);
const gapMin = formatGapMinutes(data.config.tGap);
if (data.workMs <= 0 && data.agentOnlyMs <= 0) {
return (
<Text size="sm" c="dimmed" py="md">
{t("No editing activity recorded yet.")}
</Text>
);
}
return (
<Stack gap="xs">
<Group gap="lg">
<Text size="sm" fw={500}>
{formatHeadline(data.workMs, t)}
</Text>
{data.agentOnlyMs > 0 && (
<Text size="xs" c="dimmed">
{t("agent: {{value}}", { value: formatHeadline(data.agentOnlyMs, t) })}
</Text>
)}
</Group>
<Group gap="md">
<Text size="xs" c="dimmed">
<span
className={`${classes.legendSwatch} ${classes.windowWork}`}
style={{ marginRight: 4 }}
/>
{t("Work")}
</Text>
<Text size="xs" c="dimmed">
<span
className={`${classes.legendSwatch} ${classes.windowAgent}`}
style={{ marginRight: 4 }}
/>
{t("Agent")}
</Text>
</Group>
<div>
{rows.map((row, i) =>
row.type === "day" ? (
<DayTrack
key={row.day.dayISO}
day={row.day}
pSingle={data.config.pSingle}
/>
) : (
<div key={`gap-${i}`} className={classes.gapRow}>
{t("× {{count}} days without edits", { count: row.count })}
</div>
),
)}
</div>
<Text size="xs" c="dimmed" mt="xs">
{t("Estimate · timezone {{tz}} · inactivity gap {{gap}} min", {
tz: data.tz,
gap: gapMin,
})}
</Text>
</Stack>
);
}
@@ -0,0 +1,23 @@
import api from "@/lib/api-client";
import { IPageWorkTime } from "./work-time.types";
/** The viewer's IANA timezone (browser locale) the punch-card lays days out
* in "my evenings", per §6.3/§10. Falls back to UTC if the runtime hides it. */
export function viewerTimezone(): string {
try {
return Intl.DateTimeFormat().resolvedOptions().timeZone || "UTC";
} catch {
return "UTC";
}
}
export async function getPageWorkTime(
pageId: string,
tz: string,
): Promise<IPageWorkTime> {
const req = await api.post<IPageWorkTime>("/pages/history/time", {
pageId,
tz,
});
return req.data;
}
@@ -0,0 +1,69 @@
import { Modal, Text, Tooltip, UnstyledButton } from "@mantine/core";
import { useDisclosure } from "@mantine/hooks";
import { IconClockHour4 } from "@tabler/icons-react";
import { useTranslation } from "react-i18next";
import { usePageWorkTime } from "./use-page-work-time";
import { formatGapMinutes, formatHeadline } from "./format-work-time";
import WorkTimePunchCard from "./work-time-punch-card";
interface Props {
pageId: string;
}
/**
* #395 the clickable "time worked on this article" headline (§6.1). Renders
* the `work` estimate with a "≈" sign and the inactivity threshold in a tooltip
* (it is an estimate, not a stopwatch). Clicking opens the daily punch-card
* (§6.2). Renders nothing until there is a non-zero human OR agent estimate, so a
* brand-new / never-edited page shows no widget. For an agent-only-edited page
* (workMs===0, agentOnlyMs>0) the headline shows the agent estimate (labelled
* `agent:`, matching the punch-card) so the punch-card stays reachable (#395:
* "how much a HUMAN and separately the AGENT").
*/
export default function WorkTimeStat({ pageId }: Props) {
const { t } = useTranslation();
const [opened, { open, close }] = useDisclosure(false);
const { data } = usePageWorkTime(pageId);
if (!data || (data.workMs <= 0 && data.agentOnlyMs <= 0)) return null;
const agentOnly = data.workMs <= 0;
const label = agentOnly
? t("agent: {{value}}", { value: formatHeadline(data.agentOnlyMs, t) })
: formatHeadline(data.workMs, t);
const gapMin = formatGapMinutes(data.config.tGap);
return (
<>
<Tooltip
label={t("Estimated time worked (inactivity gap {{gap}} min)", {
gap: gapMin,
})}
position="bottom"
>
<UnstyledButton
onClick={open}
aria-label={t("Show time worked on this page")}
>
<Text
size="xs"
c="dimmed"
style={{ display: "inline-flex", alignItems: "center", gap: 4 }}
>
<IconClockHour4 size={14} />
{label}
</Text>
</UnstyledButton>
</Tooltip>
<Modal
opened={opened}
onClose={close}
title={t("Time worked on this article")}
size="lg"
>
<WorkTimePunchCard data={data} />
</Modal>
</>
);
}
@@ -0,0 +1,82 @@
/* #395 24h × days punch-card. Custom CSS segments on a fixed 24-hour track
(position = offset-in-day / 24h, width = duration / 24h), no chart library. */
.row {
display: grid;
grid-template-columns: 96px 1fr 64px;
align-items: center;
gap: 12px;
padding: 3px 0;
}
.dayLabel {
font-size: var(--mantine-font-size-xs);
color: var(--mantine-color-dimmed);
white-space: nowrap;
}
.track {
position: relative;
height: 16px;
border-radius: 4px;
background-color: light-dark(
var(--mantine-color-gray-1),
var(--mantine-color-dark-6)
);
overflow: hidden;
}
/* Faint hour grid so the eye can read "morning vs evening". */
.hourTick {
position: absolute;
top: 0;
bottom: 0;
width: 1px;
background-color: light-dark(
var(--mantine-color-gray-3),
var(--mantine-color-dark-4)
);
}
.window {
position: absolute;
top: 2px;
bottom: 2px;
border-radius: 3px;
min-width: 3px;
}
.windowWork {
background-color: var(--mantine-color-blue-5);
}
.windowAgent {
background-color: var(--mantine-color-grape-5);
}
/* A lone single-sample (P_single) window: minimal + dimmed, so it neither
vanishes nor fakes dense work (§6.2). */
.windowSingle {
opacity: 0.5;
}
.daySum {
font-size: var(--mantine-font-size-xs);
text-align: right;
white-space: nowrap;
}
.gapRow {
padding: 6px 0 6px 108px;
font-size: var(--mantine-font-size-xs);
color: var(--mantine-color-dimmed);
font-style: italic;
}
.legendSwatch {
display: inline-block;
width: 12px;
height: 12px;
border-radius: 3px;
vertical-align: middle;
}
@@ -0,0 +1,37 @@
// #395 — client-side mirror of the server work-time payload
// (apps/server/src/core/page/work-time). Shapes returned by POST /pages/history/time.
export type WorkSessionClass = "work" | "agent_only";
export interface IDayWindow {
start: number;
end: number;
class: WorkSessionClass;
}
export interface IPerDay {
day: number;
dayISO: string;
activeMs: number;
agentMs: number;
windows: IDayWindow[];
}
export interface IWorkTimeConfig {
tGap: number;
agentTGap: number;
pIn: number;
pOut: number;
pSingle: number;
excludeGit: boolean;
burstCapMs?: number;
dedupRoundMs: number;
}
export interface IPageWorkTime {
workMs: number;
agentOnlyMs: number;
perDay: IPerDay[];
config: IWorkTimeConfig;
tz: string;
}
@@ -51,6 +51,7 @@ import {
import { formattedDate } from "@/lib/time.ts";
import { PageEditModeToggle } from "@/features/user/components/page-state-pref.tsx";
import MovePageModal from "@/features/page/components/move-page-modal.tsx";
import WorkTimeStat from "@/features/page-history/work-time/work-time-stat.tsx";
import { useTimeAgo } from "@/hooks/use-time-ago.tsx";
import {
useFavoriteIds,
@@ -265,6 +266,8 @@ function PageActionMenu({ readOnly, onSaveVersion }: PageActionMenuProps) {
return (
<>
{page?.id && <WorkTimeStat pageId={page.id} />}
<Menu
shadow="xl"
position="bottom-end"
@@ -1,6 +1,6 @@
import { Spotlight } from "@mantine/spotlight";
import { IconSearch } from "@tabler/icons-react";
import { Group, VisuallyHidden } from "@mantine/core";
import { Group, Text, VisuallyHidden } from "@mantine/core";
import { useState, useMemo } from "react";
import { useDebouncedValue } from "@mantine/hooks";
import { useTranslation } from "react-i18next";
@@ -84,6 +84,11 @@ export function SearchSpotlight({ spaceId }: SearchSpotlightProps) {
onFiltersChange={handleFiltersChange}
spaceId={spaceId}
/>
{/* #529: operator hint matches ANY word by default; "" for an exact
phrase, +term to require, -term to exclude. */}
<Text size="xs" c="dimmed" mt={4}>
{t('Tip: "exact phrase", +required, -excluded')}
</Text>
</div>
<VisuallyHidden role="status" aria-live="polite">
@@ -5,6 +5,9 @@ import { IPage } from "@/features/page/types/page.types.ts";
export interface IPageSearch {
id: string;
// #529 A7 superset: `pageId` aliases `id`; `rank`/`highlight` are null for
// substring-only hits (the UI already falls back to the title/snippet).
pageId?: string;
title: string;
icon: string;
parentPageId: string;
@@ -12,9 +15,36 @@ export interface IPageSearch {
creatorId: string;
createdAt: Date;
updatedAt: Date;
rank: string;
highlight: string;
rank: string | number | null;
highlight: string | null;
space: Partial<ISpace>;
// New #529 fields (present from the native Postgres search driver).
snippet?: string;
score?: number;
path?: string[];
matchedFields?: string[];
matchedTerms?: string[];
}
// #529 A5 pagination envelope returned by POST /search (native driver). The web
// list helpers read `items`; these travel alongside for pagination + diagnostics.
export interface IPageSearchResponse {
items: IPageSearch[];
total: number;
hasMore: boolean;
truncatedAtCap: boolean;
offset: number;
query?: {
raw: string;
parsed: {
positive: string[];
required: string[];
excluded: string[];
reason?: string;
};
mode: "or" | "and";
match: string;
};
}
export interface SearchSuggestionParams {
@@ -37,6 +67,10 @@ export interface IPageSearchParams {
query: string;
spaceId?: string;
shareId?: string;
// #529 A9: match mode (auto default) + pagination.
match?: "auto" | "word" | "prefix" | "substring";
limit?: number;
offset?: number;
}
export interface IAttachmentSearch {
@@ -14,9 +14,11 @@
*/
/*
* Push the top-anchored toast containers below the top chrome (fixed 45px
* header + optional 45px format toolbar + ~6px gap) so a toast (z-index 10000)
* neither covers nor intercepts clicks on the header/toolbar (both z-index 99).
* Anchor the top toast containers to the very top edge of the viewport (a small
* 8px gap), above the header/search chrome, per product request. The toast
* (z-index 10000) therefore renders over the header/toolbar (both z-index 99)
* for the few seconds it is visible intentional, since it is the top-most,
* in-the-line-of-sight surface.
*
* Scoped to [data-position^='top'] on purpose. Mantine renders ALL SIX position
* containers simultaneously (`position` only routes toasts into one via the
@@ -34,7 +36,7 @@
* (the :where() contributes 0), regardless of stylesheet order.
*/
.mantine-Notifications-root[data-position^='top'] {
top: 96px;
top: 8px;
}
[data-mantine-color-scheme='light'] .mantine-Notification-root {
+1 -1
View File
@@ -23,7 +23,7 @@
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build && pnpm --filter @docmost/mcp build",
"test": "jest",
"test:int": "jest --config test/jest-integration.json",
"test:watch": "jest --watch",
@@ -141,7 +141,57 @@ export function htmlToJson(html: string) {
}
}
export function jsonToText(tiptapJson: JSONContent) {
/**
* Deterministic text-serializer overrides for the `format:"text"` page read
* (#502). Non-text nodes render to a STABLE placeholder instead of their
* (structure-dependent) inner text, so a machine diff of two text reads is
* driven only by the page's actual prose output stability across package
* versions IS the contract (pinned by a snapshot test). Returning a string from
* a `textSerializer` also stops `generateText` descending into the node, so a
* table renders as ONE token rather than its flattened cell text.
*
* Only nodes with no meaningful flat-text form are overridden; every other node
* (paragraph/heading/list/code/blockquote/callout/) keeps its natural text so
* a config written as markdown reads back byte-identical.
*/
const TEXT_READ_SERIALIZERS: Record<string, (props: { node: any }) => string> =
{
// Image atom: no inner text -> a fixed placeholder.
image: () => '[image]',
// Table: `[table RxC]` where R = row count, C = the first row's cell count
// (a table's columns are uniform per the schema). Computed from the PM node,
// so it is independent of cell contents.
table: ({ node }) => {
const rows = node?.childCount ?? 0;
const cols = rows > 0 ? (node.child(0)?.childCount ?? 0) : 0;
return `[table ${rows}x${cols}]`;
},
};
/**
* Serialize a ProseMirror/TipTap document to plain text.
*
* Default (no options): the long-standing search-index behavior bare
* concatenated node text with `generateText`'s default `\n\n` block separator.
* This feeds the page `textContent` tsvector and MUST NOT change.
*
* `deterministic:true` (#502 `format:"text"` page read): a flat, machine-diffable
* rendering one line per block (`\n` block separator; `hardBreak` already
* serializes to `\n`), inline marks/anchors/autoformat dropped, and non-text
* nodes replaced by the stable placeholders above (`[image]`, `[table RxC]`).
*/
export function jsonToText(
// `any` (like jsonToHtml/jsonToMarkdown) so a loosely-typed DB `page.content`
// (JsonValue) can be passed straight through, as the controller does.
tiptapJson: any,
options?: { deterministic?: boolean },
) {
if (options?.deterministic) {
return generateText(tiptapJson, tiptapExtensions, {
blockSeparator: '\n',
textSerializers: TEXT_READ_SERIALIZERS,
});
}
return generateText(tiptapJson, tiptapExtensions);
}
@@ -736,6 +736,53 @@ describe('PersistenceExtension.onStoreDocument — Approach-A boundary snapshot'
expect(historyQueue.remove).toHaveBeenCalledWith(PAGE_ID);
expect(historyQueue.remove).not.toHaveBeenCalledWith(SLUG);
});
// #370 F2 — an effectively-empty page is a REACHABLE no-op (agent calls
// save_page_version on a blank page): the version tx short-circuits with
// nothing to pin. The handler MUST still broadcast a TERMINAL reply
// (version.skipped, reason:'empty') so the client resolves at once instead of
// waiting out its 20s ack timeout and misreporting a healthy server as
// unreachable. MUTATION: drop the `else if (skipped)` broadcast → no terminal
// reply is sent → this reddens.
it('empty page → no version written, broadcasts a terminal version.skipped(empty)', async () => {
const emptyDoc = { type: 'doc', content: [{ type: 'paragraph' }] };
const document = ydocFor(emptyDoc);
pageRepo.findById.mockResolvedValue({
...persistedHumanPage('IGNORED'),
content: emptyDoc,
});
await emitSave(document, 'agent');
// Nothing pinned.
expect(pageHistoryRepo.saveHistory).not.toHaveBeenCalled();
expect(pageHistoryRepo.updateHistoryKind).not.toHaveBeenCalled();
// But a terminal reply WAS sent so the client never times out. The flush
// (onStoreDocument) emits its own `page.updated`; the version.skipped is the
// LAST broadcast (dropping the skip branch leaves page.updated last → reds).
const calls = (document as any).broadcastStateless.mock.calls;
const msg = JSON.parse(calls[calls.length - 1][0]);
expect(msg).toEqual({ type: 'version.skipped', reason: 'empty' });
});
// #370 F2 — the page row is gone (deleted / never persisted). Same rule: a
// terminal reply MUST be sent (version.skipped, reason:'page-not-found') so the
// client surfaces a truthful "not found" immediately rather than a health
// timeout. onStoreDocument's own `!page` guard returns early without throwing,
// so the handler reaches the version tx and its `!page` skip branch.
it('page not found → broadcasts a terminal version.skipped(page-not-found)', async () => {
const document = ydocFor(doc('GONE'));
pageRepo.findById.mockResolvedValue(null);
await emitSave(document, 'agent');
expect(pageHistoryRepo.saveHistory).not.toHaveBeenCalled();
expect((document as any).broadcastStateless).toHaveBeenCalledTimes(1);
const msg = JSON.parse(
(document as any).broadcastStateless.mock.calls[0][0],
);
expect(msg).toEqual({ type: 'version.skipped', reason: 'page-not-found' });
});
});
// #370 — the in-memory idle-burst marker must be dropped on doc unload (like
@@ -68,6 +68,19 @@ export const INTENTIONAL_CLEAR_MESSAGE_TYPE = 'intentional-clear';
*/
export const SAVE_VERSION_MESSAGE_TYPE = 'save-version';
/**
* #370 F2 wire format of the serverclient REPLY to a save-version signal, sent
* over the same stateless channel. `version.saved` means a version was created or
* promoted; `version.skipped` is a TERMINAL "nothing was pinned" reply for the two
* reachable no-op cases (an effectively-empty page, or the page row is gone) so the
* client resolves at once instead of waiting out its ack timeout and misreporting a
* healthy server as unreachable. EXACTLY ONE of these is broadcast per handled
* save. The MCP client duplicates these literals (it cannot import server code)
* keep the two in sync (see packages/mcp/src/lib/collaboration.ts).
*/
export const VERSION_SAVED_MESSAGE_TYPE = 'version.saved';
export const VERSION_SKIPPED_MESSAGE_TYPE = 'version.skipped';
/**
* #251 how long an intentional-clear signal stays "pending" before it is
* ignored. The signal is set on the clearing keystroke but consumed by the
@@ -643,6 +656,10 @@ export class PersistenceExtension implements Extension {
let result:
| { historyId: string; kind: PageHistoryKind; alreadySaved: boolean }
| undefined;
// #370 F2 — set when there is nothing to version (empty page / page gone), so
// the tail broadcasts a terminal `version.skipped` instead of staying silent
// and forcing the client to time out. Mutually exclusive with `result`.
let skipped: 'empty' | 'page-not-found' | undefined;
// #370 F8-twin — the contributor set popped from Redis (destructive SPOP)
// must be restored if the version row does not durably land. The inner
@@ -668,11 +685,20 @@ export class PersistenceExtension implements Extension {
includeContent: true,
trx,
});
if (!page) return;
if (!page) {
// The page row is gone (deleted/never persisted). Nothing to version —
// record it so the tail sends a terminal skip reply (#370 F2).
skipped = 'page-not-found';
return;
}
versionedPageId = page.id;
// Never version an effectively-empty page (mirrors the processor's
// first-history guard); there is nothing intentional to pin.
if (isEmptyParagraphDoc(page.content as any)) return;
// first-history guard); there is nothing intentional to pin. Record the
// skip so the client gets a terminal reply rather than a timeout (#370 F2).
if (isEmptyParagraphDoc(page.content as any)) {
skipped = 'empty';
return;
}
const lastHistory = await this.pageHistoryRepo.findPageLastHistory(
page.id,
@@ -747,15 +773,26 @@ export class PersistenceExtension implements Extension {
}
this.idleBurstStart.delete(documentName);
// EXACTLY ONE terminal reply per handled save (#370 F2): a real save/promote
// broadcasts `version.saved`; the two no-op early returns (empty page / page
// gone) broadcast `version.skipped` so the client never waits out its ack
// timeout. A genuine failure threw above and rejected before reaching here.
if (result) {
document.broadcastStateless(
JSON.stringify({
type: 'version.saved',
type: VERSION_SAVED_MESSAGE_TYPE,
historyId: result.historyId,
kind: result.kind,
alreadySaved: result.alreadySaved,
}),
);
} else if (skipped) {
document.broadcastStateless(
JSON.stringify({
type: VERSION_SKIPPED_MESSAGE_TYPE,
reason: skipped,
}),
);
}
}
@@ -0,0 +1,116 @@
import { jsonToText } from './collaboration.util';
// #502 READ contract: `jsonToText(json, { deterministic: true })` — the flat,
// machine-diffable text rendering behind getPage `format:"text"`. Block-per-line
// (`\n` separator), inline marks/anchors dropped, hardBreak -> `\n`, and non-text
// nodes replaced by STABLE placeholders (`[image]`, `[table RxC]`). Output
// stability across package versions IS the contract, so it is pinned by a
// snapshot below. The DEFAULT (no options) path is the search-index serializer
// and MUST be unchanged — asserted separately.
const doc = (...content: any[]) => ({ type: 'doc', content });
const para = (...content: any[]) => ({ type: 'paragraph', content });
const text = (t: string, marks?: any[]) =>
marks ? { type: 'text', text: t, marks } : { type: 'text', text: t };
describe('jsonToText — default (search index) behavior is unchanged', () => {
it('uses the `\\n\\n` block separator and drops non-text nodes to empty', () => {
const d = doc(para(text('alpha')), para(text('beta')));
expect(jsonToText(d)).toBe('alpha\n\nbeta');
});
it('an image contributes no text in the default (tsvector) mode', () => {
const d = doc(para(text('cap')), { type: 'image', attrs: { src: 's' } });
// No `[image]` placeholder leaks into the search index.
expect(jsonToText(d)).not.toContain('[image]');
});
});
describe('jsonToText — deterministic:true (getPage format:"text")', () => {
it('renders one line per block with `\\n` separators, marks dropped', () => {
const d = doc(
{ type: 'heading', attrs: { level: 2 }, content: [text('Title')] },
para(
text('hello ', [{ type: 'bold' }]),
text('world', [{ type: 'italic' }]),
),
);
expect(jsonToText(d, { deterministic: true })).toBe('Title\nhello world');
});
it('a hardBreak renders as a newline', () => {
const d = doc(para(text('a'), { type: 'hardBreak' }, text('b')));
expect(jsonToText(d, { deterministic: true })).toBe('a\nb');
});
it('an image node -> the stable `[image]` placeholder', () => {
const d = doc(para(text('before')), { type: 'image', attrs: { src: 's' } });
expect(jsonToText(d, { deterministic: true })).toBe('before\n[image]');
});
it('a table -> `[table RxC]` (rows x columns) and its cell text is NOT flattened in', () => {
const cell = (t: string) => ({
type: 'tableCell',
content: [para(text(t))],
});
const row = (...cells: any[]) => ({ type: 'tableRow', content: cells });
const table = {
type: 'table',
content: [
row(cell('CELLONE'), cell('CELLTWO'), cell('CELLTHREE')),
row(cell('CELLFOUR'), cell('CELLFIVE'), cell('CELLSIX')),
],
};
const d = doc(para(text('grid:')), table);
const out = jsonToText(d, { deterministic: true });
expect(out).toBe('grid:\n[table 2x3]');
expect(out).not.toContain('CELL'); // cell text is not flattened into the read
});
it('SNAPSHOT: a mixed document renders to a stable, deterministic string', () => {
const cell = (t: string) => ({
type: 'tableCell',
content: [para(text(t))],
});
const d = doc(
{ type: 'heading', attrs: { level: 1 }, content: [text('Config')] },
para(text('key = ', [{ type: 'bold' }]), text('value')),
{
type: 'bulletList',
content: [
{ type: 'listItem', content: [para(text('one'))] },
{ type: 'listItem', content: [para(text('two'))] },
],
},
{ type: 'image', attrs: { src: 's' } },
{
type: 'table',
content: [{ type: 'tableRow', content: [cell('x'), cell('y')] }],
},
);
expect(jsonToText(d, { deterministic: true })).toMatchInlineSnapshot(`
"Config
key = value
one
two
[image]
[table 1x2]"
`);
});
it('SCENARIO: a config written as a code block reads back byte-identical (diff empty)', () => {
const config =
'export TICKET_LIFETIME=$2592000\nservers:\n - www.internal.host';
const d = doc({
type: 'codeBlock',
attrs: { language: 'yaml' },
content: [text(config)],
});
// The code block is one block; its text (dollars, bare domain, newlines) is
// preserved verbatim, so a read-as-text of a stored config diffs empty.
expect(jsonToText(d, { deterministic: true })).toBe(config);
});
});
@@ -35,6 +35,7 @@ import {
import { PaginationOptions } from '@docmost/db/pagination/pagination-options';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { UserThrottlerGuard } from '../../integrations/throttle/user-throttler.guard';
import { AI_CHAT_THROTTLER } from '../../integrations/throttle/throttler-names';
@@ -43,6 +44,8 @@ import {
AiChatRunHooks,
AiChatService,
AiChatStreamBody,
rowHasInlineParts,
hydrateAssistantParts,
} from './ai-chat.service';
import { AiChatRunService } from './ai-chat-run.service';
import { AiTranscriptionService } from './ai-transcription.service';
@@ -129,8 +132,39 @@ export class AiChatController {
// production. Only touched on the resumable-stream (flag-on) path.
private readonly streamRegistry?: AiChatStreamRegistryService,
private readonly environment?: EnvironmentService,
// #492: reconstruct a #492 mid-run record's parts from the steps table before
// returning rows to the client / export. OPTIONAL so positional controller
// specs compile unchanged; when absent, hydration is skipped (old-era rows
// already carry inline parts, so nothing to reconstruct).
private readonly aiChatRunStepRepo?: AiChatRunStepRepo,
) {}
/**
* Reconstruct parts for any assistant rows that don't carry them INLINE a
* #492 mid-run record whose per-step parts live in `ai_chat_run_steps` (the
* append-persist backend). Every FINISHED row (old-era + #492) and every old-era
* streaming snapshot already has inline `metadata.parts`, so the common path
* fetches NOTHING and returns the rows untouched; only an actively-streaming
* new-style row triggers the batch step fetch. Consumers (seed/poll/export) read
* `metadata.parts` off the returned rows exactly as before the era switch is
* invisible to them (reconstructRunParts contract).
*/
private async withReconstructedParts(
rows: AiChatMessage[],
workspaceId: string,
): Promise<AiChatMessage[]> {
if (!this.aiChatRunStepRepo) return rows;
const needy = rows.filter(
(r) => r.role === 'assistant' && !rowHasInlineParts(r),
);
if (needy.length === 0) return rows;
const stepsByMessage = await this.aiChatRunStepRepo.findByMessageIds(
needy.map((r) => r.id),
workspaceId,
);
return hydrateAssistantParts(rows, stepsByMessage);
}
/** List the requesting user's chats in this workspace (paginated). */
@HttpCode(HttpStatus.OK)
@Post('chats')
@@ -184,11 +218,17 @@ export class AiChatController {
@AuthWorkspace() workspace: Workspace,
) {
await this.assertOwnedChat(dto.chatId, user, workspace);
return this.aiChatMessageRepo.findByChat(
const page = await this.aiChatMessageRepo.findByChat(
dto.chatId,
workspace.id,
pagination,
);
// #492: reconstruct parts for any active new-style row so the client seed sees
// `metadata.parts` unchanged (a no-op for the finished rows that fill a page).
return {
...page,
items: await this.withReconstructedParts(page.items, workspace.id),
};
}
/**
@@ -225,7 +265,10 @@ export class AiChatController {
workspace.id,
);
return {
rows,
// #492: the delta of an actively-streaming new-style row carries its parts
// reconstructed from the steps table, so the degraded poll shows persisted
// progress exactly as the pre-#492 full-row snapshot did.
rows: await this.withReconstructedParts(rows, workspace.id),
cursor,
run: run ? { id: run.id, status: run.status } : null,
};
@@ -247,8 +290,10 @@ export class AiChatController {
@AuthWorkspace() workspace: Workspace,
): Promise<{ markdown: string }> {
const chat = await this.assertOwnedChat(dto.chatId, user, workspace);
const rows = await this.aiChatMessageRepo.findAllByChat(
dto.chatId,
const rows = await this.withReconstructedParts(
await this.aiChatMessageRepo.findAllByChat(dto.chatId, workspace.id),
// #492: an interrupted-but-still-active turn exports its persisted steps
// (reconstructed from the steps table) just like the pre-#492 full row did.
workspace.id,
);
const markdown = buildChatMarkdown({
@@ -288,7 +333,13 @@ export class AiChatController {
workspace.id,
)
: undefined;
return { run, message: message ?? null };
// #492: reconnect to an IN-FLIGHT run reconstructs the projection row's parts
// from the steps table (the row itself carries only the step marker mid-run);
// a finished run's row already has inline parts, so this is a no-op.
const [hydrated] = message
? await this.withReconstructedParts([message], workspace.id)
: [undefined];
return { run, message: hydrated ?? null };
}
/**
@@ -370,10 +370,12 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
);
});
// #490 reactive branch: a provider CONTEXT-OVERFLOW 400 in onError is classified,
// records a distinguishable cause, and stamps metadata.replayOverflow so the NEXT
// turn's budgeter trims aggressively (the recovery that un-bricks the chat).
it('#490: a context-overflow 400 stamps replayOverflow on the finalized row', async () => {
// #490/#520 reactive branch: a provider CONTEXT-OVERFLOW 400 in onError is
// classified, records a distinguishable cause, and stamps the consecutive-overflow
// COUNTER (metadata.replayOverflowCount) so the NEXT turn's budgeter trims with
// escalating aggression (the recovery that un-bricks the chat). This is a fresh
// chat (empty history -> prior streak 0), so the first overflow stamps count 1.
it('#490/#520: a context-overflow 400 stamps replayOverflowCount=1 on the finalized row', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
@@ -397,11 +399,14 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
metadata: Record<string, unknown>;
};
expect(patch.status).toBe('error');
expect(patch.metadata.replayOverflow).toBe(true);
// First overflow on a fresh chat -> k = prior(0) + 1 = 1.
expect(patch.metadata.replayOverflowCount).toBe(1);
// The legacy boolean is no longer written (the counter supersedes it).
expect('replayOverflow' in patch.metadata).toBe(false);
expect(patch.metadata.error).toContain('контекстное окно');
});
it('#490: a non-overflow error does NOT stamp replayOverflow', async () => {
it('#490/#520: a non-overflow error does NOT stamp the overflow counter', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
@@ -412,6 +417,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
status: string;
metadata: Record<string, unknown>;
};
expect('replayOverflowCount' in patch.metadata).toBe(false);
expect('replayOverflow' in patch.metadata).toBe(false);
});
});
@@ -30,13 +30,16 @@ import {
STEP_LIMIT_NO_ANSWER_MARKER,
OUTPUT_DEGENERATION_ERROR,
lastAssistantContextTokens,
lastAssistantReplayOverflow,
lastAssistantReplayOverflowCount,
seedActivatedTools,
} from './ai-chat.service';
import type { AiChatMessage, Workspace } from '@docmost/db/types/entity.types';
import { buildSystemPrompt } from './ai-chat.prompt';
import type { McpClientsService } from './external-mcp/mcp-clients.service';
import { resolveEffectiveReplayThreshold } from './history-budget';
import {
resolveEffectiveReplayThreshold,
REPLAY_MIN_FLOOR_TOKENS,
} from './history-budget';
/**
* Unit tests for compactToolOutput: the pure helper that shrinks tool outputs
@@ -554,49 +557,129 @@ describe('seedActivatedTools', () => {
});
});
describe('lastAssistantReplayOverflow', () => {
describe('lastAssistantReplayOverflowCount', () => {
const row = (
role: string,
metadata: Record<string, unknown> | null,
): AiChatMessage => ({ role, metadata }) as unknown as AiChatMessage;
it('is true only when the LAST assistant turn overflowed', () => {
it('reads the consecutive-overflow count from the LAST assistant turn', () => {
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflowCount: 3 }),
row('user', null),
]),
).toBe(true);
// A recovered (later, non-overflow) assistant turn clears it.
).toBe(3);
// A recovered (later, non-overflow) assistant turn resets it to 0 — the read
// stops at the most recent assistant row, which carries no count.
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflowCount: 3 }),
row('user', null),
row('assistant', { contextTokens: 5 }),
]),
).toBe(false);
expect(lastAssistantReplayOverflow([])).toBe(false);
).toBe(0);
expect(lastAssistantReplayOverflowCount([])).toBe(0);
});
// #490 reactive recovery: a prior turn stamped `replayOverflow` must make the
// NEXT turn's effective budget the AGGRESSIVE 0.5x cut — that harder trim is
// what un-bricks a chat that just 400'd on the context window. This exercises
// the exact wiring the service uses: read the stamp, then scale the threshold.
it('#490: a prior replayOverflow drives the next turn to the 0.5x aggressive budget', () => {
// BACK-COMPAT (#520): an in-flight row written by the pre-#520 boolean stamp
// (`replayOverflow: true`, no count) reads as k=1 — the old single 0.5× behavior —
// so a chat mid-recovery across the deploy does not regress.
it('#520 back-compat: a legacy boolean replayOverflow reads as k=1', () => {
expect(
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflow: true }),
row('user', null),
]),
).toBe(1);
// A legacy row with the flag absent/false is k=0.
expect(
lastAssistantReplayOverflowCount([row('assistant', { contextTokens: 5 })]),
).toBe(0);
});
// A corrupt/negative persisted count never yields a negative k.
it('clamps a corrupt negative count to 0', () => {
expect(
lastAssistantReplayOverflowCount([
row('assistant', { replayOverflowCount: -4 }),
]),
).toBe(0);
});
// #490/#520 reactive recovery: the prior consecutive-overflow count `k` drives
// the next turn's effective budget to an ESCALATING cut (0.5**k) — each further
// consecutive 400 tightens it, which is what un-bricks a chat that keeps
// overflowing. This exercises the exact wiring the service uses: read the count,
// then scale the threshold.
it('#490/#520: the prior count drives the next turn to the escalating aggressive budget', () => {
const history = [
row('assistant', { replayOverflow: true }),
row('assistant', { replayOverflowCount: 1 }),
row('user', null),
];
const priorOverflowed = lastAssistantReplayOverflow(history);
expect(priorOverflowed).toBe(true);
// Base budget 100k -> aggressive recovery halves it to 50k this turn.
expect(resolveEffectiveReplayThreshold(100_000, priorOverflowed)).toBe(50_000);
const k = lastAssistantReplayOverflowCount(history);
expect(k).toBe(1);
// Base budget 100k -> first-overflow recovery halves it to 50k this turn.
expect(resolveEffectiveReplayThreshold(100_000, k)).toBe(50_000);
// A second consecutive overflow (k=2) quarters it.
expect(resolveEffectiveReplayThreshold(100_000, 2)).toBe(25_000);
// Odd base floors, not rounds.
expect(resolveEffectiveReplayThreshold(99_999, true)).toBe(49_999);
// No prior overflow -> the base budget is used verbatim (no aggressive cut).
expect(resolveEffectiveReplayThreshold(100_000, false)).toBe(100_000);
expect(resolveEffectiveReplayThreshold(99_999, 1)).toBe(49_999);
// No prior overflow (k=0) -> the base budget is used verbatim (no cut).
expect(resolveEffectiveReplayThreshold(100_000, 0)).toBe(100_000);
// An explicit off-switch (null) is never overridden, even on recovery.
expect(resolveEffectiveReplayThreshold(null, true)).toBeNull();
expect(resolveEffectiveReplayThreshold(null, 3)).toBeNull();
});
// #520 escalation table + convergence: the cut deepens each consecutive overflow
// and is CLAMPED at the floor so it converges (un-bricks even against a small
// real window), instead of the old fixed single 0.5× that stuck at 50k forever.
it('#520: escalates and converges to the floor, un-bricking a small real window', () => {
const base = 100_000;
expect(resolveEffectiveReplayThreshold(base, 0)).toBe(base);
expect(resolveEffectiveReplayThreshold(base, 1)).toBe(50_000);
expect(resolveEffectiveReplayThreshold(base, 2)).toBe(25_000);
expect(resolveEffectiveReplayThreshold(base, 3)).toBe(12_500);
// Residual-brick regression (#520): with the flat-default base (100k) and a real
// model window of ~40k, the OLD fixed 0.5× stuck at 50k forever (> 40k -> 400s
// again, never recovers). The escalating cut drops BELOW 40k after enough
// consecutive overflows -> the history finally fits -> the chat un-bricks.
const realWindow = 40_000;
// k=1 (50k) still exceeds the window — the old behavior's terminal state.
expect(resolveEffectiveReplayThreshold(base, 1)).toBeGreaterThan(realWindow);
// But escalation converges under the window within a couple more turns.
const converged = [2, 3, 4, 5].some(
(k) => (resolveEffectiveReplayThreshold(base, k) as number) < realWindow,
);
expect(converged).toBe(true);
// Convergence is bounded BELOW by the floor: a large k never trims below it.
for (const k of [4, 8, 20, 100]) {
expect(resolveEffectiveReplayThreshold(base, k)).toBe(REPLAY_MIN_FLOOR_TOKENS);
expect(
resolveEffectiveReplayThreshold(base, k) as number,
).toBeGreaterThanOrEqual(REPLAY_MIN_FLOOR_TOKENS);
}
});
// The floor never RAISES a legitimately small configured budget above itself —
// that would re-overflow the very window it was configured for. Under #520 Option B
// recovery MAY cut it BELOW itself (down to floor(0.5×base) = the old 0.5× cut).
it('#520: never inflates a small configured budget above itself (may cut below)', () => {
const small = 5_000; // below the floor
const floor = Math.min(REPLAY_MIN_FLOOR_TOKENS, Math.floor(small * 0.5)); // 2500
expect(resolveEffectiveReplayThreshold(small, 0)).toBe(small);
// Even under escalation the effective threshold never exceeds the base, and never
// drops below the absolute floor.
for (const k of [1, 2, 3, 10]) {
const t = resolveEffectiveReplayThreshold(small, k) as number;
expect(t).toBeLessThanOrEqual(small);
expect(t).toBeGreaterThanOrEqual(floor);
}
// Option B: on overflow it DOES cut below the configured budget (old floor==budget
// behavior would keep this at `small`).
expect(resolveEffectiveReplayThreshold(small, 1)).toBe(floor);
});
});
@@ -930,21 +1013,50 @@ describe('flushAssistant', () => {
expect(flushed.metadata.error).toBe('boom');
});
// #490 observability: the replay budgeter's decision is stamped on the turn.
it('records replayTrimmedToTokens + replayOverflow when provided', () => {
// #490/#520 observability: the replay budgeter's decision is stamped on the turn,
// now including the consecutive-overflow COUNTER (#520) the next turn escalates on.
it('records replayTrimmedToTokens + replayOverflowCount when provided', () => {
const f = flushAssistant([], '', 'error', {
error: 'ctx',
replayTrimmedToTokens: 42_000,
replayOverflow: true,
replayOverflowCount: 2,
});
expect(f.metadata.replayTrimmedToTokens).toBe(42_000);
expect(f.metadata.replayOverflow).toBe(true);
expect(f.metadata.replayOverflowCount).toBe(2);
});
it('omits the replay metadata when not provided', () => {
const f = flushAssistant([], '', 'completed', { finishReason: 'stop' });
expect('replayTrimmedToTokens' in f.metadata).toBe(false);
expect('replayOverflow' in f.metadata).toBe(false);
expect('replayOverflowCount' in f.metadata).toBe(false);
expect('replayBelowConfiguredBudget' in f.metadata).toBe(false);
});
// #520 Option B observability: the turn is marked when recovery replayed BELOW the
// admin-configured budget, so the UI/telemetry can surface "config too large".
it('stamps replayBelowConfiguredBudget when recovery cut below the configured budget', () => {
const f = flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: 2,
replayBelowConfiguredBudget: true,
});
// MUTATION SENTINEL: dropping the metadata stamp in flushAssistant reddens this.
expect(f.metadata.replayBelowConfiguredBudget).toBe(true);
});
it('omits replayBelowConfiguredBudget on a normal in-budget replay', () => {
// false/omitted must NOT leave the flag on the turn (only the below-budget case).
const off = flushAssistant([], '', 'completed', {
replayBelowConfiguredBudget: false,
});
expect('replayBelowConfiguredBudget' in off.metadata).toBe(false);
});
// A clean finalize (no overflow -> count 0/omitted) leaves NO counter, which the
// next turn reads as k=0 — the reset that ends a recovery streak.
it('omits replayOverflowCount for a zero/absent count (reset semantics)', () => {
const zero = flushAssistant([], '', 'completed', { replayOverflowCount: 0 });
expect('replayOverflowCount' in zero.metadata).toBe(false);
});
// #274 observability: the page-change diff the agent saw this turn is persisted
+299 -42
View File
@@ -22,6 +22,7 @@ import { AiSettingsService } from '../../integrations/ai/ai-settings.service';
import { describeProviderError } from '../../integrations/ai/ai-error.util';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo';
import { AiAgentRoleRepo } from '@docmost/db/repos/ai-agent-roles/ai-agent-roles.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
@@ -140,9 +141,10 @@ const OUTPUT_DEGENERATION_ERROR =
// Prefix recorded on the assistant row when the provider rejected the turn for
// CONTEXT OVERFLOW (#490): the replayed history exceeded the model's window. The
// row is ALSO stamped `metadata.replayOverflow` so the NEXT turn's budgeter trims
// aggressively (the reactive recovery — the overflowing turn had no usage signal
// to trigger preventive trimming, so the classified 400 is what un-bricks it).
// row is ALSO stamped `metadata.replayOverflowCount` (the consecutive-overflow
// counter, #520) so the NEXT turn's budgeter trims with escalating aggression (the
// reactive recovery — the overflowing turn had no usage signal to trigger
// preventive trimming, so the classified 400 is what un-bricks it).
export const CONTEXT_OVERFLOW_ERROR_PREFIX =
'Диалог превысил контекстное окно модели; история будет агрессивно ' +
'сокращена на следующем ходу.';
@@ -518,6 +520,12 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// constructions compile unchanged; Nest always injects the real singleton, so
// reconcile sees the SAME in-memory active/zombie maps the runner mutates.
private readonly aiChatRunService?: AiChatRunService,
// #492 append-persist: per-step INSERT into the lightweight steps table (the
// O(Σ steps) replacement for the O(n²) full-row `metadata.parts` rewrite).
// OPTIONAL so existing positional constructions (int-specs) compile unchanged;
// Nest injects the real singleton. When ABSENT the per-step path falls back to
// the pre-#492 full-row flush (no regression, only no WAL win).
private readonly aiChatRunStepRepo?: AiChatRunStepRepo,
) {}
// #487: periodic reconcile timer (single-process phase 1). Started in
@@ -1114,8 +1122,34 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
chatId,
workspace.id,
);
// #492: HYDRATE needy assistant rows from the steps table BEFORE the replay
// map. A #492 mid-run assistant row carries only a step marker
// (metadata.parts:[]); its real per-step parts live in `ai_chat_run_steps`.
// The graceful terminal callbacks (onFinish/onError/onAbort -> flushAssistant)
// assemble the full inline parts, so a normally-ended turn already has them.
// But a HARD crash mid-run (SIGKILL/OOM) fires NO terminal callback, so the
// row stays parts:[]; without this, rowToUiMessage falls back to an empty
// text part and the partial tool-calls/results/text — durable in the steps
// table — would DROP OUT of the model's replay context (regressing #183
// step-granular durability for the model consumer). Mirrors the controller's
// withReconstructedParts EXACTLY (same needy predicate + hydration helper).
// Guarded on the optional repo: absent (positional test builds) degrades to
// the current behavior rather than crashing.
let replayHistory = oldHistory;
if (this.aiChatRunStepRepo) {
const needy = oldHistory.filter(
(r) => r.role === 'assistant' && !rowHasInlineParts(r),
);
if (needy.length > 0) {
const stepsByMessage = await this.aiChatRunStepRepo.findByMessageIds(
needy.map((r) => r.id),
workspace.id,
);
replayHistory = hydrateAssistantParts(oldHistory, stepsByMessage);
}
}
const uiMessages: Array<Omit<UIMessage, 'id'> & { id: string }> = [
...oldHistory.map(rowToUiMessage),
...replayHistory.map(rowToUiMessage),
{
id: 'pending-user',
role: 'user',
@@ -1154,7 +1188,9 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// hint — confirm it against the persisted history (the preceding assistant
// turn must really be aborted/streaming) so a spoofed flag cannot inject the
// interrupt note onto an ordinary turn. The partial output the model needs is
// already in `messages` (the aborted assistant row replays via findRecent).
// already in `messages`: a #492 mid-run row's per-step parts live only in the
// `ai_chat_run_steps` table and were hydrated into the replay history above,
// so the aborted assistant turn replays WITH its partial parts intact.
// Append the new user turn (shape-only) so index -2 is the prior assistant.
const interrupted = isInterruptResume(
[...oldHistory, { role: 'user', status: null, metadata: null }],
@@ -1193,27 +1229,52 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
}
// Last turn's provider-reported context size (authoritative when present).
const priorContextTokens = lastAssistantContextTokens(oldHistory);
// Reactive recovery (#490): if the LAST turn was rejected for context
// overflow (stamped by onError), trim AGGRESSIVELY this turn — the
// overflowing turn produced no usage signal, so a normal-threshold trim may
// not shrink enough to fit. This is what un-bricks a chat that just 400'd.
const priorOverflowed = lastAssistantReplayOverflow(oldHistory);
// Reactive recovery (#490/#520): `k` = how many CONSECUTIVE preceding turns
// were rejected for context overflow (stamped by onError). Each consecutive
// overflow trims MORE aggressively (resolveEffectiveReplayThreshold scales the
// budget by 0.5**k, clamped at the floor) so recovery ESCALATES until the
// history fits — the overflowing turn produced no usage signal, so a single
// fixed cut may not shrink enough when the real model window is small. This is
// what un-bricks a chat that keeps 400'ing on the context window.
const priorOverflowCount = lastAssistantReplayOverflowCount(oldHistory);
const effectiveThreshold = resolveEffectiveReplayThreshold(
replayBudget.thresholdTokens,
priorOverflowed,
priorOverflowCount,
);
if (priorOverflowed) {
this.logger.warn(
`AI chat (chat ${chatId}): previous turn hit context overflow; ` +
`applying aggressive replay budget (${effectiveThreshold} tokens).`,
);
// #520 Option B: recovery MAY cut the replay budget BELOW the admin-configured
// window when the provider keeps proving non-fit (the anti-brick invariant of
// the reactive branch beats a config that a 400 has disproved). Surface it so
// the admin sees their window is factually too large, and mark the turn so the
// UI/telemetry can show it. Only true when actually below (not on in-budget
// escalation edge cases); k>0 implies below for any budget >= 2 tokens.
const replayBelowConfiguredBudget =
replayBudget.thresholdTokens != null &&
effectiveThreshold != null &&
effectiveThreshold < replayBudget.thresholdTokens;
if (priorOverflowCount > 0) {
if (replayBelowConfiguredBudget) {
this.logger.warn(
`AI chat (chat ${chatId}): configured replay budget ` +
`(${replayBudget.thresholdTokens} tokens) does not fit the model — ` +
`replaying BELOW it at ${effectiveThreshold} tokens after ` +
`${priorOverflowCount} consecutive context overflow(s) ` +
`(escalation level ${priorOverflowCount}). Lower chatContextWindow ` +
`to match the model's real context window.`,
);
} else {
this.logger.warn(
`AI chat (chat ${chatId}): ${priorOverflowCount} consecutive context ` +
`overflow(s); applying escalated aggressive replay budget ` +
`(${effectiveThreshold} tokens).`,
);
}
}
const preTrim = trimHistoryForReplay(
messages,
effectiveThreshold,
// A prior OVERFLOW means the provider count is stale/absent — force the
// char-estimate path by ignoring priorContextTokens on recovery.
priorOverflowed ? undefined : priorContextTokens,
priorOverflowCount > 0 ? undefined : priorContextTokens,
);
messages = preTrim.messages;
// Observability (#490): record the budgeter's decision on the turn so the UI
@@ -1559,17 +1620,57 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// connection when finalize runs, so the SQL `WHERE status='streaming'`
// (not this flag) is what prevents it clobbering the terminal row.
if (finalized) return null;
// Build the flush ONCE so the returned count is EXACTLY the persisted
// `stepsPersisted` (both derive from capturedSteps.length at this instant).
const flushed = flushAssistant(capturedSteps, '', 'streaming', {
pageChanged,
partsCache,
});
const stepsPersisted = flushed.metadata.stepsPersisted as number;
// The count derives from capturedSteps.length at THIS instant, so the
// returned value is EXACTLY the persisted `stepsPersisted` the ring rotates
// on (whether we take the append-persist path or the legacy fallback).
const stepsPersisted = capturedSteps.length;
try {
await this.aiChatMessageRepo.update(assistantId, workspace.id, flushed, {
onlyIfStreaming: true,
});
if (this.aiChatRunStepRepo) {
// #492 APPEND-PERSIST: write only THIS finished step's parts to the
// steps table (O(step) WAL), then bump the row's CHEAP step marker —
// NO growing `metadata.parts` blob (that O(n²) full-row rewrite is
// exactly what this removes). The full `metadata.parts` is assembled
// once at finalize; a mid-run resume seed is reconstructed from the
// step rows (reconstructRunParts). The INSERT is idempotent
// (ON CONFLICT DO NOTHING), so a re-fired step never doubles the parts.
const index = stepsPersisted - 1;
if (index >= 0) {
const stepParts = assistantParts(
[capturedSteps[index]],
'',
partsCache,
);
await this.aiChatRunStepRepo.insertStep(
assistantId,
workspace.id,
index,
stepParts,
);
}
// Marker UPDATE: advance stepsPersisted + keep the toolTrace era marker
// (bumps updatedAt so the delta poll observes the step, and carries the
// frontier a resuming client attaches from). Scoped onlyIfStreaming so a
// late marker never clobbers the terminal finalize.
await this.aiChatMessageRepo.update(
assistantId,
workspace.id,
{ metadata: stepMarkerMetadata(stepsPersisted) },
{ onlyIfStreaming: true },
);
} else {
// Legacy fallback (no steps table wired — positional test builds): the
// pre-#492 full-row flush, so parts still land inline on the row.
const flushed = flushAssistant(capturedSteps, '', 'streaming', {
pageChanged,
partsCache,
});
await this.aiChatMessageRepo.update(
assistantId,
workspace.id,
flushed,
{ onlyIfStreaming: true },
);
}
return stepsPersisted;
} catch (err) {
this.logger.warn(
@@ -1828,6 +1929,10 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
pageChanged,
partsCache,
replayTrimmedToTokens,
// #520 Option B: mark the turn when recovery replayed BELOW the
// configured budget (only when it actually did).
replayBelowConfiguredBudget:
replayBelowConfiguredBudget || undefined,
}),
);
// #184/#487: the RUN is finalized ALWAYS (never gated on the message).
@@ -1906,7 +2011,16 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
pageChanged,
partsCache,
replayTrimmedToTokens,
replayOverflow: overflow || undefined,
// #520: escalate the consecutive-overflow counter so the NEXT turn
// trims MORE aggressively (0.5**k). k grows by 1 each consecutive
// overflow; a clean finalize omits the field, resetting it to 0.
replayOverflowCount: overflow
? priorOverflowCount + 1
: undefined,
// #520 Option B: mark the turn when THIS replay was already below the
// configured budget (only when it actually was).
replayBelowConfiguredBudget:
replayBelowConfiguredBudget || undefined,
}),
);
// #184: settle the RUN as failed, carrying the provider/transport cause.
@@ -2338,22 +2452,39 @@ export function seedActivatedTools(
}
/**
* Whether the most recent assistant turn was rejected for CONTEXT OVERFLOW
* (#490): its row carries `metadata.replayOverflow` (stamped by the stream's
* onError). The next turn's budgeter reads this to trim aggressively the
* reactive recovery. Only the LAST assistant turn matters (an older overflow was
* already recovered), so we stop at the first assistant row scanning backwards.
* How many CONSECUTIVE recent turns were rejected for CONTEXT OVERFLOW (#490/#520):
* `k`, read from the most recent assistant row's `metadata.replayOverflowCount`
* (stamped by the stream's onError, incremented each consecutive overflow and reset
* to 0 on any clean finalize). The next turn's budgeter feeds this to
* {@link resolveEffectiveReplayThreshold} to trim with ESCALATING aggression the
* reactive recovery. Only the LAST assistant turn matters (its count already carries
* the consecutive streak; an older overflow followed by a clean turn was recovered),
* so we stop at the first assistant row scanning backwards.
*
* BACK-COMPAT: a row written by the pre-#520 boolean stamp (`replayOverflow: true`,
* no count) is read as k=1 the old single 0.5× behavior so in-flight chats do
* not regress across the deploy.
*/
export function lastAssistantReplayOverflow(
export function lastAssistantReplayOverflowCount(
history: ReadonlyArray<AiChatMessage>,
): boolean {
): number {
for (let i = history.length - 1; i >= 0; i--) {
const row = history[i];
if (row.role !== 'assistant') continue;
const meta = (row.metadata ?? {}) as { replayOverflow?: unknown };
return meta.replayOverflow === true;
const meta = (row.metadata ?? {}) as {
replayOverflowCount?: unknown;
replayOverflow?: unknown;
};
if (typeof meta.replayOverflowCount === 'number') {
// Guard against a corrupt/negative persisted value.
return meta.replayOverflowCount > 0
? Math.floor(meta.replayOverflowCount)
: 0;
}
// Back-compat: legacy boolean stamp -> one overflow (0.5× cut).
return meta.replayOverflow === true ? 1 : 0;
}
return false;
return 0;
}
/** The last message with role 'user' from a useChat payload, if any. */
@@ -2749,6 +2880,122 @@ export function rowToUiMessage(row: AiChatMessage): Omit<UIMessage, 'id'> & {
return { id: row.id, role, parts: parts as UIMessage['parts'] };
}
/**
* Cheap step-marker metadata for the #492 per-step UPDATE. Advances
* `stepsPersisted` (the resume attach frontier) and keeps the `toolTraceVersion`
* era marker, WITHOUT the growing `parts` blob (those live in the steps table
* now; the full `metadata.parts` is assembled once at finalize by flushAssistant).
* `parts: []` is kept for shape stability it reads as an empty inline-parts row,
* which is exactly the discriminator that routes reconstruction to the steps table.
*/
export function stepMarkerMetadata(
stepsPersisted: number,
): Record<string, unknown> {
return { parts: [], toolTraceVersion: 2, stepsPersisted };
}
/**
* Whether an assistant row already carries its full UI parts INLINE on the row
* (`metadata.parts`). TRUE for every FINISHED row old-era rows AND #492 rows,
* whose full parts are assembled once at finalize and for old-era streaming
* snapshots (the pre-#492 per-step full-row flush). FALSE for a #492 MID-RUN
* record, whose per-step parts live in the `ai_chat_run_steps` table. This is the
* era discriminator the reconstruct seam branches on no schema flag needed.
*/
export function rowHasInlineParts(row: { metadata?: unknown }): boolean {
const meta = (row.metadata ?? {}) as { parts?: unknown };
return Array.isArray(meta.parts) && meta.parts.length > 0;
}
/**
* Concatenate persisted per-step parts (in `stepIndex` order) into the turn's UI
* parts (#492). Reproduces EXACTLY what flushAssistant assistantParts would have
* written to `metadata.parts` for those finished steps, since each step row stored
* `assistantParts([step])` at persist time.
*/
export function assembleStepParts(
stepRows: ReadonlyArray<{ stepIndex: number; parts: unknown }>,
): UIMessage['parts'] {
const parts: Array<Record<string, unknown>> = [];
for (const step of [...stepRows].sort((a, b) => a.stepIndex - b.stepIndex)) {
if (Array.isArray(step.parts)) {
parts.push(...(step.parts as Array<Record<string, unknown>>));
}
}
return parts as UIMessage['parts'];
}
/**
* reconstructRunParts (#492) the single backend-switch seam. Given an assistant
* ROW and its persisted step rows, return the turn's UI `parts` + the persisted
* step count, reading from the ROW when it already carries inline parts (old-era
* records AND every finished record) and from the STEPS TABLE otherwise (a #492
* mid-run record). The higher-level consumers (attach seed, delta poll, export)
* route their rowparts through this / {@link hydrateAssistantParts}, so old and
* new records reconstruct identically WITHOUT the consumers branching on the era.
*/
export function reconstructRunParts(
row: { metadata?: unknown; content?: string | null },
stepRows: ReadonlyArray<{ stepIndex: number; parts: unknown }>,
): { parts: UIMessage['parts']; stepsPersisted: number } {
if (rowHasInlineParts(row)) {
const meta = row.metadata as {
parts: UIMessage['parts'];
stepsPersisted?: number;
};
return {
parts: meta.parts,
stepsPersisted:
typeof meta.stepsPersisted === 'number'
? meta.stepsPersisted
: stepRows.length,
};
}
if (stepRows.length > 0) {
return {
parts: assembleStepParts(stepRows),
stepsPersisted: stepRows.length,
};
}
// No inline parts and no step rows: an old-era seed / empty streaming row. Fall
// back to a single text part from `content` (mirrors rowToUiMessage).
return {
parts: textPart(row.content ?? '') as UIMessage['parts'],
stepsPersisted: 0,
};
}
/**
* Fill each assistant row's `metadata.parts` from its step rows when the row does
* not already carry them inline (a #492 mid-run record), so a consumer that reads
* `metadata.parts` off the RAW row (the client seed/poll, the Markdown export)
* sees the reconstructed parts with NO change to itself. Rows that already have
* inline parts (old-era + finished) and non-assistant rows pass through untouched.
* Pure: returns new row objects, never mutates the inputs.
*/
export function hydrateAssistantParts<
T extends { id: string; role?: string; metadata?: unknown },
>(
rows: ReadonlyArray<T>,
stepsByMessage: Map<
string,
ReadonlyArray<{ stepIndex: number; parts: unknown }>
>,
): T[] {
return rows.map((row) => {
if (row.role !== 'assistant' || rowHasInlineParts(row)) return row;
const steps = stepsByMessage.get(row.id);
if (!steps || steps.length === 0) return row;
return {
...row,
metadata: {
...((row.metadata ?? {}) as Record<string, unknown>),
parts: assembleStepParts(steps),
},
};
});
}
/**
* The persisted-row patch shape produced by {@link flushAssistant}. It is the
* SAME shape the assistant repo insert/update consume (content + toolCalls +
@@ -2893,9 +3140,16 @@ export function flushAssistant(
// the (estimated) token size it trimmed to — the UI can show "replay truncated
// at N tokens". Omitted when nothing was trimmed.
replayTrimmedToTokens?: number;
// #490 reactive branch: set when the provider rejected this turn for context
// overflow. Stamped into metadata so the NEXT turn's budgeter trims aggressively.
replayOverflow?: boolean;
// #490/#520 reactive branch: the consecutive context-overflow count for THIS
// turn (prior streak + 1) when the provider rejected it for context overflow.
// Stamped into metadata so the NEXT turn's budgeter trims with escalating
// aggression (0.5**k). Omitted (undefined) on a clean turn, which resets k to 0.
replayOverflowCount?: number;
// #520 Option B observability: true when recovery replayed this turn's history
// BELOW the admin-configured budget (the configured window did not fit and the
// anti-brick invariant cut past it). Omitted on a normal in-budget replay so the
// flag marks ONLY the "config is factually too large" case.
replayBelowConfiguredBudget?: boolean;
},
): AssistantFlush {
const finished = capturedSteps ?? [];
@@ -2948,7 +3202,10 @@ export function flushAssistant(
metadata.maxContextTokens = extra.maxContextTokens;
if (extra?.replayTrimmedToTokens)
metadata.replayTrimmedToTokens = extra.replayTrimmedToTokens;
if (extra?.replayOverflow) metadata.replayOverflow = true;
if (extra?.replayOverflowCount && extra.replayOverflowCount > 0)
metadata.replayOverflowCount = extra.replayOverflowCount;
if (extra?.replayBelowConfiguredBudget)
metadata.replayBelowConfiguredBudget = true;
if (extra?.error) metadata.error = extra.error;
// Persist the page-change diff the agent saw this turn (#274 observability),
// so history / the Markdown export can show what the user changed. Only when
@@ -1,6 +1,11 @@
import { randomBytes } from 'crypto';
import { Client } from 'pg';
import { flushAssistant, serializeSteps } from './ai-chat.service';
import {
flushAssistant,
serializeSteps,
lastAssistantReplayOverflowCount,
} from './ai-chat.service';
import type { AiChatMessage } from '@docmost/db/types/entity.types';
/**
* #490 write-volume regression an OBSERVABLE-PROPERTY test on a LIVE Postgres,
@@ -207,3 +212,111 @@ describe('#490 write-volume on a live Postgres (pg_current_wal_lsn delta)', () =
expect(v2).toBeLessThan(v1 * 0.75);
}, 120_000);
});
/**
* #520 reactive-recovery COUNTER lifecycle on a LIVE Postgres proves the
* consecutive-overflow count survives a real jsonb metadata round-trip (the persist
* path), not just an in-memory object. flushAssistant BUILDS the row metadata, we
* WRITE it to a jsonb column, READ it back, then reconstruct the assistant row and
* run lastAssistantReplayOverflowCount over it exactly the read the next turn does.
*
* The lifecycle proven end-to-end through pg:
* - consecutive overflows INCREMENT k (1 -> 2 -> 3);
* - a CLEAN finalize omits the field, which the reader treats as a RESET to 0;
* - a legacy boolean row (`replayOverflow: true`) reads back as k=1 (back-compat).
*/
describe('#520 overflow-counter lifecycle on a live Postgres (jsonb round-trip)', () => {
let client: Client | undefined;
let available = false;
beforeAll(async () => {
try {
client = new Client(CONN);
await client.connect();
await client.query('SELECT 1');
available = true;
} catch {
available = false;
client = undefined;
}
});
afterAll(async () => {
await client?.end().catch(() => undefined);
});
// Round-trip an arbitrary metadata object through a real jsonb column and read it
// back as the reconstructed assistant row the next turn would load.
async function roundTrip(
c: Client,
metadata: unknown,
): Promise<AiChatMessage> {
await c.query('UPDATE _wal_counter SET metadata=$1 WHERE id=1', [
JSON.stringify(metadata),
]);
const back = (await c.query('SELECT metadata FROM _wal_counter WHERE id=1'))
.rows[0].metadata as Record<string, unknown>;
return { role: 'assistant', metadata: back } as unknown as AiChatMessage;
}
it('increments across consecutive overflows, resets on a clean turn, and honors the legacy boolean', async () => {
if (!available || !client) {
console.warn('SKIP: gitmost-test-pg not reachable; skipping counter test.');
return;
}
const c = client;
await c.query('DROP TABLE IF EXISTS _wal_counter');
await c.query('CREATE TABLE _wal_counter(id int primary key, metadata jsonb)');
await c.query("INSERT INTO _wal_counter VALUES (1, '{}'::jsonb)");
// Turn 1 overflow: prior streak 0 -> stamp k=1 (as the service does: prior+1).
let prior = lastAssistantReplayOverflowCount([]); // fresh chat
expect(prior).toBe(0);
let row = await roundTrip(
c,
flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: prior + 1,
}).metadata,
);
prior = lastAssistantReplayOverflowCount([row]);
expect(prior).toBe(1);
// Turn 2 overflow: prior 1 -> stamp k=2.
row = await roundTrip(
c,
flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: prior + 1,
}).metadata,
);
prior = lastAssistantReplayOverflowCount([row]);
expect(prior).toBe(2);
// Turn 3 overflow: prior 2 -> stamp k=3.
row = await roundTrip(
c,
flushAssistant([], '', 'error', {
error: 'ctx',
replayOverflowCount: prior + 1,
}).metadata,
);
prior = lastAssistantReplayOverflowCount([row]);
expect(prior).toBe(3);
// Turn 4 CLEAN finalize: no overflow -> the field is omitted -> reset to 0.
row = await roundTrip(
c,
flushAssistant([], 'all good', 'completed', { finishReason: 'stop' })
.metadata,
);
expect('replayOverflowCount' in (row.metadata as object)).toBe(false);
expect(lastAssistantReplayOverflowCount([row])).toBe(0);
// Back-compat: a row persisted by the pre-#520 boolean stamp reads back as k=1.
row = await roundTrip(c, { replayOverflow: true });
expect(lastAssistantReplayOverflowCount([row])).toBe(1);
await c.query('DROP TABLE IF EXISTS _wal_counter');
}, 60_000);
});
@@ -1,14 +1,146 @@
import type { ModelMessage } from 'ai';
import {
resolveReplayBudget,
resolveEffectiveReplayThreshold,
isContextOverflowError,
estimateMessagesTokens,
trimHistoryForReplay,
REPLAY_BUDGET_DEFAULT_TOKENS,
REPLAY_BUDGET_WINDOW_FRACTION,
REPLAY_MIN_FLOOR_TOKENS,
REPLAY_TRUNCATION_MARKER,
REPLAY_TURN_COLLAPSED_MARKER,
} from './history-budget';
describe('resolveEffectiveReplayThreshold (#520 iterative escalation)', () => {
// The escalation table: each consecutive overflow (k) deepens the cut by 0.5×.
it('scales the base by 0.5**k, flooring (not rounding) fractional tokens', () => {
const base = 100_000;
expect(resolveEffectiveReplayThreshold(base, 0)).toBe(base); // k=0: unchanged
expect(resolveEffectiveReplayThreshold(base, 1)).toBe(50_000); // 0.5×
expect(resolveEffectiveReplayThreshold(base, 2)).toBe(25_000); // 0.25×
expect(resolveEffectiveReplayThreshold(base, 3)).toBe(12_500); // 0.125×
// Floors, not rounds.
expect(resolveEffectiveReplayThreshold(99_999, 1)).toBe(49_999);
});
it('passes a null base (trimming OFF) through unchanged for any k', () => {
for (const k of [0, 1, 2, 5, 100]) {
expect(resolveEffectiveReplayThreshold(null, k)).toBeNull();
}
});
// The crux of #520: convergence. A large k is clamped at REPLAY_MIN_FLOOR_TOKENS,
// so the escalation CONVERGES to a small-but-usable budget instead of trimming to
// zero — and, unlike the old fixed 0.5× that stuck at 50k, it drops far enough to
// fit a small real model window.
it('clamps a large k at the floor (converges, never below)', () => {
const base = 100_000;
for (const k of [4, 6, 10, 50, 200]) {
const t = resolveEffectiveReplayThreshold(base, k) as number;
expect(t).toBe(REPLAY_MIN_FLOOR_TOKENS);
expect(t).toBeGreaterThanOrEqual(REPLAY_MIN_FLOOR_TOKENS);
}
});
// Residual-brick regression (#520): flat-default base 100k, real window ~40k. The
// OLD fixed single 0.5× stuck at 50k > 40k forever (re-overflows every turn — the
// brick). The iterative cut drops BELOW 40k after a couple more consecutive
// overflows, so the history finally fits and the chat un-bricks.
it('un-bricks: escalation drops below a small real window the fixed 0.5× never could', () => {
const base = 100_000;
const realWindow = 40_000;
// The old terminal state: 0.5× = 50k, still above the window.
expect(resolveEffectiveReplayThreshold(base, 1)).toBeGreaterThan(realWindow);
// Escalation converges under the window.
const converged = [2, 3, 4, 5].some(
(k) => (resolveEffectiveReplayThreshold(base, k) as number) < realWindow,
);
expect(converged).toBe(true);
// MUTATION SENTINEL: reverting `** k` to `** 1` (fixed 0.5×) makes every k yield
// 50k, so `converged` above would be FALSE and this test reddens. Removing the
// floor reddens the clamp test instead.
});
// "Don't INFLATE above itself" is still an invariant: the floor never RAISES a
// legitimately small configured budget above itself (that would re-overflow the
// very window it was set for). Under #520 Option B the floor is min(FLOOR,
// floor(0.5×base)), so for a base BELOW the floor recovery MAY cut it further —
// down to floor(0.5×base), i.e. AT LEAST the old single 0.5× cut — but never above.
it('never inflates a small configured budget above itself (may cut below it, #520 Option B)', () => {
const small = 5_000; // below REPLAY_MIN_FLOOR_TOKENS
const floor = Math.min(REPLAY_MIN_FLOOR_TOKENS, Math.floor(small * 0.5)); // 2500
expect(resolveEffectiveReplayThreshold(small, 0)).toBe(small); // k=0 unchanged
for (const k of [1, 2, 3, 10]) {
const t = resolveEffectiveReplayThreshold(small, k) as number;
// Invariant: never RAISED above the configured budget.
expect(t).toBeLessThanOrEqual(small);
// Option B: never trimmed below the absolute floor (converges).
expect(t).toBeGreaterThanOrEqual(floor);
}
// The old single 0.5× cut is reached (and pinned) — recovery is never WORSE than
// before, and DOES cut below the configured budget on overflow (Option B). Under
// the old floor==budget behavior this would be `small` (5000), not `floor`.
expect(resolveEffectiveReplayThreshold(small, 1)).toBe(floor);
});
// #520 Option B: with a configured window (NOT the flat default), repeated overflow
// escalates the replay budget BELOW the configured budget, down to the absolute
// floor — the chat must not brick even when the configured window is too large for
// the model. A CLEAN turn (k back to 0) restores the full configured budget.
it('escalates below the configured budget down to the floor, and resets on a clean turn', () => {
// Configured context window 8000 -> budget = floor(0.7 × 8000) = 5600 (the code's
// window->budget ratio; compute it, do not hardcode).
const window = 8_000;
const budget = resolveReplayBudget(window).thresholdTokens as number;
expect(budget).toBe(Math.floor(REPLAY_BUDGET_WINDOW_FRACTION * window)); // 5600
// The absolute floor for this budget = min(FLOOR, floor(0.5 × budget)) = 2800
// (a "small window": 0.5×budget < REPLAY_MIN_FLOOR_TOKENS), which is BELOW the
// 5600 configured budget.
const floor = Math.min(
REPLAY_MIN_FLOOR_TOKENS,
Math.floor(budget * 0.5),
); // 2800
expect(floor).toBe(2_800);
expect(floor).toBeLessThan(budget); // below the configured budget
// k=0 -> full configured budget (no overflow yet).
expect(resolveEffectiveReplayThreshold(budget, 0)).toBe(budget);
// k=1 -> cut BELOW the configured budget to the 0.5× floor (2800). This is the
// MUTATION SENTINEL: with the OLD floor==configured-budget behavior this would be
// max(2800, 5600) = 5600 (never below budget), so the assertion reddens.
expect(resolveEffectiveReplayThreshold(budget, 1)).toBe(floor);
expect(resolveEffectiveReplayThreshold(budget, 1)).toBeLessThan(budget);
// k>=2 stays at the floor (converges, never below it).
for (const k of [2, 3, 5, 20]) {
expect(resolveEffectiveReplayThreshold(budget, k)).toBe(floor);
}
// A CLEAN turn resets k to 0 -> the full configured budget is restored (recovery
// is not sticky; it only bites while the provider keeps proving non-fit).
expect(resolveEffectiveReplayThreshold(budget, 0)).toBe(budget);
});
// #520 Option B, LARGE window: a big configured budget escalates in DISTINCT steps
// below itself, converging to the fixed absolute floor REPLAY_MIN_FLOOR_TOKENS.
it('a large configured budget escalates below itself down to REPLAY_MIN_FLOOR_TOKENS', () => {
const budget = resolveReplayBudget(200_000).thresholdTokens as number; // 140000
// The floor for a large window is the fixed REPLAY_MIN_FLOOR_TOKENS (8k), well
// BELOW the configured budget.
expect(
Math.min(REPLAY_MIN_FLOOR_TOKENS, Math.floor(budget * 0.5)),
).toBe(REPLAY_MIN_FLOOR_TOKENS);
const seq = [0, 1, 2, 3, 4, 5, 6].map(
(k) => resolveEffectiveReplayThreshold(budget, k) as number,
);
expect(seq).toEqual([140_000, 70_000, 35_000, 17_500, 8_750, 8_000, 8_000]);
// Every escalated step (k>=1) is below the configured budget; convergence floor.
for (const t of seq.slice(1)) expect(t).toBeLessThan(budget);
expect(seq[seq.length - 1]).toBe(REPLAY_MIN_FLOOR_TOKENS);
});
});
describe('resolveReplayBudget', () => {
it('uses floor(0.7 x window) for a configured window (no cap)', () => {
// 0.7 x 60k = 42k
+62 -11
View File
@@ -22,10 +22,35 @@ export const REPLAY_BUDGET_DEFAULT_TOKENS = 100_000;
/** Fraction of a configured context window used as the budget. */
export const REPLAY_BUDGET_WINDOW_FRACTION = 0.7;
/**
* Fraction of the normal budget used for the REACTIVE re-trim after a provider
* context-overflow 400 the preventive estimate under-counted, so cut harder.
* Per-step fraction of the normal budget applied on the REACTIVE re-trim after a
* provider context-overflow 400 the preventive estimate under-counted, so cut
* harder. This is now applied ITERATIVELY: with `k` consecutive overflow turns the
* budget is scaled by `fraction ** k` (k=1 -> 0.5×, k=2 -> 0.25×, ), so recovery
* ESCALATES turn over turn until the replayed history finally fits, instead of the
* old single fixed 0.5× cut that could never un-brick a chat whose real model
* window is smaller than 0.5 × the (unconfigured, flat-default) base budget (#520).
*/
export const REPLAY_AGGRESSIVE_FRACTION = 0.5;
/**
* Absolute lower bound (tokens) on the escalating reactive budget: the iterative
* cut is clamped here so it CONVERGES (a fixed floor, not an ever-shrinking value
* that would eventually trim everything). Rationale: below ~8k tokens a chat cannot
* carry meaningful recent context, and even a small real model window comfortably
* fits this much keep-recent-turns still applies on top, so a handful of recent
* turns survive.
*
* #520 Option B: for a LARGE configured window this floor sits BELOW the configured
* replay budget, and recovery is allowed to escalate PAST the configured budget down
* to it "the chat must not brick on overflow" is an INVARIANT of the reactive
* branch, and a configured window is a claim about model capacity, not a promise
* about replay size; once the provider proves non-fit with a 400, reality beats the
* config. This does NOT change the NORMAL path's upper bound (#510 Option A still
* respects the configured budget while it fits) it is survival once non-fit is
* proven. For a SMALL configured window the effective floor is instead the old
* single 0.5× cut (never worse than before, and never RAISED above the budget); see
* {@link resolveEffectiveReplayThreshold}.
*/
export const REPLAY_MIN_FLOOR_TOKENS = 8_000;
/**
* Turns (a user message + its assistant/tool replies) kept FULL at the tail,
* including the current one never trimmed. Older turns are compacted first.
@@ -85,22 +110,48 @@ export function resolveReplayBudget(rawContextWindow: unknown): ReplayBudget {
}
/**
* The effective replay threshold for THIS turn, given the base budget and whether
* the PREVIOUS turn hit a context-overflow 400 (the reactive-recovery signal,
* `metadata.replayOverflow`). On recovery the base budget is scaled down by
* {@link REPLAY_AGGRESSIVE_FRACTION}: the overflowing turn produced no usage
* signal, so the preventive estimate under-counted and a normal-threshold trim may
* not shrink enough to fit this harder cut is what un-bricks the chat.
* The effective replay threshold for THIS turn, given the base budget and `k` the
* number of CONSECUTIVE preceding turns that hit a context-overflow 400 (the
* reactive-recovery signal `metadata.replayOverflowCount`, read from the last
* assistant row). On recovery the base budget is scaled down ITERATIVELY by
* {@link REPLAY_AGGRESSIVE_FRACTION} ** k and clamped at {@link REPLAY_MIN_FLOOR_TOKENS}:
* - k=0 -> base unchanged (no overflow: nothing to recover from).
* - k=1 -> floor(0.5 × base); k=2 -> floor(0.25 × base); each further consecutive
* overflow tightens the cut, so recovery ESCALATES until the history fits.
* - the escalation is clamped at the floor so it CONVERGES this is what un-bricks
* a chat whose real model window is smaller than a single 0.5× cut of the base
* (e.g. an unconfigured window: flat-default base 100k, real window <50k) (#520).
*
* The overflowing turn produced no usage signal, so the preventive estimate
* under-counted and a normal-threshold (or single fixed 0.5×) trim may not shrink
* enough to fit; the escalating cut is what recovers such a chat.
*
* A `null` base budget (trimming OFF) is passed through unchanged: an explicit
* off-switch is never overridden by the recovery path.
*
* The absolute floor is `min(REPLAY_MIN_FLOOR_TOKENS, floor(0.5 × base))` (#520
* Option B):
* - LARGE window (floor(0.5 × base) >= REPLAY_MIN_FLOOR_TOKENS) -> the fixed
* REPLAY_MIN_FLOOR_TOKENS, which is BELOW the configured budget: escalation is
* ALLOWED to cut past the configured budget down to this absolute minimum, so a
* chat whose real model window is far smaller than the configured window still
* un-bricks (the invariant beats a config reality disproved with a 400).
* - SMALL window (floor(0.5 × base) < REPLAY_MIN_FLOOR_TOKENS) -> floor(0.5 ×
* base), i.e. AT LEAST the old single 0.5× cut: recovery is never WORSE than the
* pre-#520 behavior, and the floor is still never RAISED above the configured
* budget itself (that would re-overflow the very window it was set for).
*/
export function resolveEffectiveReplayThreshold(
thresholdTokens: number | null,
priorOverflowed: boolean,
k: number,
): number | null {
if (!priorOverflowed || thresholdTokens == null) return thresholdTokens;
return Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION);
if (thresholdTokens == null || k <= 0) return thresholdTokens;
const scaled = Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION ** k);
const floor = Math.min(
REPLAY_MIN_FLOOR_TOKENS,
Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION),
);
return Math.max(scaled, floor);
}
/**
@@ -106,6 +106,7 @@ function __assertClientCallContract(client: DocmostClientLike): void {
void client.sharePage(s, true);
void client.unsharePage(s);
void client.restorePageVersion(s);
void client.savePageVersion(s);
void client.transformPage(s, s, { dryRun: true });
void client.stashPage(s);
// --- write (image / footnote), in-app since #410 ---
@@ -58,6 +58,7 @@ type DocmostClientMethod =
| 'sharePage'
| 'unsharePage'
| 'restorePageVersion'
| 'savePageVersion'
| 'transformPage'
| 'stashPage'
// --- write (image / footnote), in-app since #410 ---
@@ -10,6 +10,12 @@ import { Transform } from 'class-transformer';
export type ContentFormat = 'json' | 'markdown' | 'html';
// READ-only rendering formats for `getPage` (#502). A superset of the writable
// `ContentFormat` with `text` — a flat, deterministic, machine-diffable text
// rendering — added. Kept SEPARATE from `ContentFormat` so the write path
// (createPage/updatePage `parseProsemirrorContent`) can never be handed `text`.
export type PageReadFormat = ContentFormat | 'text';
export class CreatePageDto {
@IsOptional()
@IsString()
+15 -3
View File
@@ -5,10 +5,11 @@ import {
IsOptional,
IsString,
IsUUID,
MaxLength,
} from 'class-validator';
import { Transform } from 'class-transformer';
import { ContentFormat } from './create-page.dto';
import { PageReadFormat } from './create-page.dto';
import { IsPageIdOrSlugId } from './page-identity.validator';
export class PageIdDto {
@@ -43,8 +44,19 @@ export class PageInfoDto extends PageIdDto {
@IsOptional()
@Transform(({ value }) => value?.toLowerCase())
@IsIn(['json', 'markdown', 'html'])
format?: ContentFormat;
@IsIn(['json', 'markdown', 'html', 'text'])
format?: PageReadFormat;
}
export class PageWorkTimeDto extends PageIdDto {
// Viewer IANA timezone for the per-day punch-card buckets (§6.3). Optional —
// falls back to UTC server-side. Length-capped so a bogus value cannot bloat
// the request; the value is only ever handed to Intl.DateTimeFormat, which
// throws on an unknown zone (caught by the controller → 400).
@IsOptional()
@IsString()
@MaxLength(64)
tz?: string;
}
export class DeletePageDto extends PageIdDto {
@@ -0,0 +1,76 @@
import { NotFoundException } from '@nestjs/common';
import { PageController } from './page.controller';
import { jsonToText, jsonToMarkdown } from '../../collaboration/collaboration.util';
// #502 READ: getPage `format:"text"` routes through the deterministic jsonToText
// path (placeholders for non-text nodes, block-per-line), while `format:"json"`
// (or none) returns the raw content and `format:"markdown"` still converts. This
// pins the CONTROLLER wiring with lightweight mocks (no DB needed — the jsonToText
// output contract itself is pinned in json-to-text-deterministic.spec.ts).
const CONTENT = {
type: 'doc',
content: [
{ type: 'heading', attrs: { level: 1 }, content: [{ type: 'text', text: 'Config' }] },
{
type: 'paragraph',
content: [
{ type: 'text', text: 'key=', marks: [{ type: 'bold' }] },
{ type: 'text', text: 'value' },
],
},
{ type: 'image', attrs: { src: 's' } },
],
};
function makeController(page: any): PageController {
const pageRepo = { findById: jest.fn().mockResolvedValue(page) } as any;
const pageAccessService = {
validateCanViewWithPermissions: jest
.fn()
.mockResolvedValue({ canEdit: true, hasRestriction: false }),
} as any;
// Only pageRepo + pageAccessService are exercised by getPage; the rest are
// never touched on this path, so undefined placeholders are fine.
return new PageController(
undefined as any, // pageService
pageRepo,
undefined as any, // pageHistoryService
undefined as any, // spaceAbility
pageAccessService,
undefined as any, // backlinkService
undefined as any, // labelService
undefined as any, // auditService
);
}
const user = { id: 'u1' } as any;
describe('PageController.getPage — format:"text" (#502)', () => {
it('returns deterministic flat text with placeholders for non-text nodes', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1', format: 'text' } as any, user);
expect(res.content).toBe(jsonToText(CONTENT, { deterministic: true }));
expect(res.content).toBe('Config\nkey=value\n[image]');
expect(res.permissions).toEqual({ canEdit: true, hasRestriction: false });
});
it('markdown format still converts to markdown (unchanged)', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1', format: 'markdown' } as any, user);
expect(res.content).toBe(jsonToMarkdown(CONTENT));
});
it('json / no format returns the raw ProseMirror content object', async () => {
const controller = makeController({ id: 'p1', content: CONTENT });
const res: any = await controller.getPage({ pageId: 'p1' } as any, user);
expect(res.content).toEqual(CONTENT); // untouched object, not a string
});
it('missing page -> NotFoundException', async () => {
const controller = makeController(null);
await expect(
controller.getPage({ pageId: 'nope', format: 'text' } as any, user),
).rejects.toBeInstanceOf(NotFoundException);
});
});
@@ -1,3 +1,8 @@
import {
BadRequestException,
ForbiddenException,
NotFoundException,
} from '@nestjs/common';
import { PageController } from './page.controller';
// Direct instantiation with stub deps. The Test.createTestingModule form failed
@@ -22,4 +27,88 @@ describe('PageController', () => {
it('should be defined', () => {
expect(controller).toBeDefined();
});
// #395 — the work-time endpoint must be gated exactly like /history.
describe('getPageWorkTime', () => {
const user = { id: 'u1' } as any;
function build(overrides: {
page?: any;
validate?: jest.Mock;
compute?: jest.Mock;
}) {
const pageRepo = { findById: jest.fn().mockResolvedValue(overrides.page) };
const pageAccessService = {
validateCanView: overrides.validate ?? jest.fn().mockResolvedValue(undefined),
};
const pageHistoryService = {
computeWorkTime:
overrides.compute ?? jest.fn().mockResolvedValue({ workMs: 0 }),
};
const c = new PageController(
{} as any,
pageRepo as any,
pageHistoryService as any,
{} as any,
pageAccessService as any,
{} as any,
{} as any,
{} as any,
);
return { c, pageRepo, pageAccessService, pageHistoryService };
}
it('404s when the page does not exist', async () => {
const { c } = build({ page: null });
await expect(
c.getPageWorkTime({ pageId: 'p1' } as any, user),
).rejects.toBeInstanceOf(NotFoundException);
});
it('enforces validateCanView before computing, then delegates with tz', async () => {
const validate = jest.fn().mockResolvedValue(undefined);
const compute = jest.fn().mockResolvedValue({ workMs: 42 });
const { c } = build({ page: { id: 'pg' }, validate, compute });
const out = await c.getPageWorkTime(
{ pageId: 'pg', tz: 'Europe/Moscow' } as any,
user,
);
expect(validate).toHaveBeenCalledWith({ id: 'pg' }, user);
expect(compute).toHaveBeenCalledWith('pg', 'Europe/Moscow');
expect(out).toEqual({ workMs: 42 });
});
it('propagates a denied view gate and does NOT reach compute (security)', async () => {
// If validateCanView is moved AFTER computeWorkTime, the timeline of a page
// the caller may not see would be read/estimated before the gate — this
// locks the order: a rejecting gate must short-circuit before any compute.
const validate = jest.fn().mockRejectedValue(new ForbiddenException());
const compute = jest.fn().mockResolvedValue({ workMs: 1 });
const { c, pageHistoryService } = build({
page: { id: 'pg' },
validate,
compute,
});
await expect(
c.getPageWorkTime({ pageId: 'pg' } as any, user),
).rejects.toBeInstanceOf(ForbiddenException);
expect(pageHistoryService.computeWorkTime).not.toHaveBeenCalled();
});
it('maps an unknown-timezone RangeError to a 400', async () => {
const compute = jest.fn().mockRejectedValue(new RangeError('bad tz'));
const { c } = build({ page: { id: 'pg' }, compute });
await expect(
c.getPageWorkTime({ pageId: 'pg', tz: 'X/Y' } as any, user),
).rejects.toBeInstanceOf(BadRequestException);
});
it('does not swallow a non-RangeError from the service', async () => {
const compute = jest.fn().mockRejectedValue(new Error('db down'));
const { c } = build({ page: { id: 'pg' }, compute });
await expect(
c.getPageWorkTime({ pageId: 'pg' } as any, user),
).rejects.toThrow('db down');
});
});
});
+43 -4
View File
@@ -21,6 +21,7 @@ import {
PageHistoryIdDto,
PageIdDto,
PageInfoDto,
PageWorkTimeDto,
} from './dto/page.dto';
import { PageHistoryService } from './services/page-history.service';
import { AuthUser } from '../../common/decorators/auth-user.decorator';
@@ -49,6 +50,7 @@ import { AddLabelsDto, RemoveLabelDto } from '../label/dto/label.dto';
import {
jsonToHtml,
jsonToMarkdown,
jsonToText,
} from '../../collaboration/collaboration.util';
import { AuditEvent, AuditResource } from '../../common/events/audit-events';
import {
@@ -93,10 +95,16 @@ export class PageController {
const permissions = { canEdit, hasRestriction };
if (dto.format && dto.format !== 'json' && page.content) {
const contentOutput =
dto.format === 'markdown'
? jsonToMarkdown(page.content)
: jsonToHtml(page.content);
let contentOutput: string;
if (dto.format === 'markdown') {
contentOutput = jsonToMarkdown(page.content);
} else if (dto.format === 'text') {
// #502: flat, deterministic, machine-diffable text (block-per-line,
// inline marks/anchors dropped, non-text nodes -> stable placeholders).
contentOutput = jsonToText(page.content, { deterministic: true });
} else {
contentOutput = jsonToHtml(page.content);
}
return {
...page,
content: contentOutput,
@@ -524,6 +532,32 @@ export class PageController {
return this.pageHistoryService.findHistoryByPageId(page.id, pagination);
}
@HttpCode(HttpStatus.OK)
@Post('/history/time')
async getPageWorkTime(
@Body() dto: PageWorkTimeDto,
@AuthUser() user: User,
) {
const page = await this.pageRepo.findById(dto.pageId);
if (!page) {
throw new NotFoundException('Page not found');
}
// Same view gate as /history and /history/info.
await this.pageAccessService.validateCanView(page, user);
try {
return await this.pageHistoryService.computeWorkTime(page.id, dto.tz);
} catch (e) {
// Intl.DateTimeFormat throws RangeError on an unknown IANA zone; surface
// it as a 400 rather than a 500.
if (e instanceof RangeError) {
throw new BadRequestException('Invalid timezone');
}
throw e;
}
}
@HttpCode(HttpStatus.OK)
@Post('/history/info')
async getPageHistoryInfo(
@@ -818,6 +852,11 @@ export class PageController {
throw new NotFoundException('Page not found');
}
// Target-only validateCanView is intentional: getPageBreadCrumbs returns
// the full ancestor chain WITHOUT per-ancestor permission filtering. Safe
// because page restrictions inherit down the tree, so any ancestor the
// caller could not view would already hide the target here — see the
// getPageBreadCrumbs docstring / #471.
await this.pageAccessService.validateCanView(page, user);
return this.pageService.getPageBreadCrumbs(page.id);
@@ -3,6 +3,23 @@ import { PageHistoryRepo } from '@docmost/db/repos/page/page-history.repo';
import { PageHistory } from '@docmost/db/types/entity.types';
import { PaginationOptions } from '@docmost/db/pagination/pagination-options';
import { CursorPaginationResult } from '@docmost/db/pagination/cursor-pagination';
import {
computeWorkTime,
bucketByDay,
DEFAULT_WORK_TIME_CONFIG,
WorkTimeConfig,
PerDay,
} from '../work-time';
export interface PageWorkTime {
workMs: number;
agentOnlyMs: number;
perDay: PerDay[];
/** the config actually used, so the UI can show "≈" + the T_gap threshold. */
config: WorkTimeConfig;
/** the tz the per-day buckets were computed in (echoed back for the label). */
tz: string;
}
@Injectable()
export class PageHistoryService {
@@ -23,4 +40,33 @@ export class PageHistoryService {
paginationOptions,
);
}
/**
* #395 estimate time worked on a page (§5) and bucket it into the viewer's
* calendar days for the punch-card (§6.3). Reads only the cheap history
* projection (no `content`); the estimate itself is a pure, deterministic
* function so it is unit-tested exhaustively without a DB.
*
* `tz` is the viewer's IANA zone (browser locale) it moves which day a
* session lands in and where its windows sit, but never the total (§10).
*/
async computeWorkTime(
pageId: string,
tz = 'UTC',
config?: Partial<WorkTimeConfig>,
): Promise<PageWorkTime> {
const rows = await this.pageHistoryRepo.findTimelineByPageId(pageId);
const result = computeWorkTime(rows, config);
const usedConfig: WorkTimeConfig = { ...DEFAULT_WORK_TIME_CONFIG, ...config };
// `bucketByDay` consumes the pure core's un-bucketed sessions here; the
// full session list is NOT shipped on the response (no client reads it).
const perDay = bucketByDay(result.sessions, tz);
return {
workMs: result.workMs,
agentOnlyMs: result.agentOnlyMs,
perDay,
config: usedConfig,
tz,
};
}
}
@@ -1071,6 +1071,29 @@ export class PageService {
});
}
/**
* Walk the ancestor chain of `childPageId` up to the space root, filtered
* ONLY by `deletedAt` (+ MAX_PAGE_TREE_DEPTH) WITHOUT per-ancestor
* permission filtering. Callers that expose this to a user (the
* `/breadcrumbs` endpoint) validate `validateCanView` on the TARGET page
* only, then return the whole chain of ancestor titles (#471).
*
* This is safe NOT a title leak because page restrictions inherit DOWN
* the tree: to view a page the caller must hold permission on EVERY
* restricted ancestor (`validateCanView` -> `canUserAccessPage` checks the
* full ancestor chain see page-access.service.ts / page-permission.repo.ts
* `canUserEditPage`). A restricted ancestor the caller may not see would
* therefore already hide the TARGET page itself, so every ancestor reachable
* here is one the caller is already entitled to view (content stays gated
* regardless getPage/getNode re-check permissions).
*
* Note the guarantee is the narrow "may view a descendant => may view its
* ancestors", NOT "space membership sees every page" restricted subtrees do
* hide pages from members. Per-ancestor permission filtering here was
* considered and declined as redundant given the inheritance invariant above
* (#471). The same chain feeds the web-UI breadcrumb bar under identical CASL
* scope.
*/
async getPageBreadCrumbs(childPageId: string, trx?: KyselyTransaction) {
const ancestors = await dbOrTx(this.db, trx)
.withRecursive('page_ancestors', (db) =>
@@ -0,0 +1,129 @@
import { bucketByDay, zonedDayStart } from './bucket-by-day';
import { computeWorkTime } from './compute-work-time';
import { WorkSession, TimelineSample } from './work-time.types';
const MIN = 60 * 1000;
const HOUR = 60 * MIN;
function work(start: number, end: number): WorkSession {
return { start, end, class: 'work' };
}
function agent(start: number, end: number): WorkSession {
return { start, end, class: 'agent_only' };
}
function sumActive(perDay: ReturnType<typeof bucketByDay>): number {
return perDay.reduce((a, d) => a + d.activeMs, 0);
}
describe('bucketByDay', () => {
it('Σ activeMs == workMs — the §6.3 consistency invariant', () => {
const rows: TimelineSample[] = [
{ createdAt: '2026-07-04T03:40:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-04T03:49:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-04T18:11:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
{ createdAt: '2026-07-06T15:34:00Z', lastUpdatedById: 'h', lastUpdatedSource: 'user', lastUpdatedAiChatId: null, kind: null },
];
const r = computeWorkTime(rows);
const perDay = bucketByDay(r.sessions, 'UTC');
expect(sumActive(perDay)).toBe(r.workMs);
});
it('empty input → no days', () => {
expect(bucketByDay([], 'UTC')).toEqual([]);
});
it('midnight-crossing session splits across two days, sum preserved (§9#9)', () => {
const start = Date.UTC(2026, 0, 10, 23, 14);
const end = Date.UTC(2026, 0, 11, 0, 40);
const perDay = bucketByDay([work(start, end)], 'UTC');
expect(perDay).toHaveLength(2);
expect(perDay[0].dayISO).toBe('2026-01-10');
expect(perDay[1].dayISO).toBe('2026-01-11');
expect(perDay[0].activeMs).toBe(46 * MIN); // 23:14 → 24:00
expect(perDay[1].activeMs).toBe(40 * MIN); // 00:00 → 00:40
expect(sumActive(perDay)).toBe(end - start);
});
it('empty days between active days are emitted, not skipped (§9#12)', () => {
const d1 = work(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 11, 0));
const d3 = work(Date.UTC(2026, 0, 12, 10, 0), Date.UTC(2026, 0, 12, 11, 0));
const perDay = bucketByDay([d1, d3], 'UTC');
expect(perDay.map((d) => d.dayISO)).toEqual([
'2026-01-10',
'2026-01-11',
'2026-01-12',
]);
expect(perDay[1].activeMs).toBe(0);
expect(perDay[1].windows).toEqual([]);
});
it('agent_only windows are drawn but excluded from activeMs', () => {
const w = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 10, 0));
const a = agent(Date.UTC(2026, 0, 10, 14, 0), Date.UTC(2026, 0, 10, 14, 30));
const perDay = bucketByDay([w, a], 'UTC');
expect(perDay).toHaveLength(1);
expect(perDay[0].activeMs).toBe(1 * HOUR);
expect(perDay[0].agentMs).toBe(30 * MIN);
expect(perDay[0].windows.map((x) => x.class)).toEqual(['work', 'agent_only']);
});
it('work and agent_only are unioned SEPARATELY (agent does not swallow work)', () => {
// Overlapping work + agent windows on the same day.
const w = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 11, 0));
const a = agent(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 12, 0));
const perDay = bucketByDay([w, a], 'UTC');
expect(perDay[0].activeMs).toBe(2 * HOUR);
expect(perDay[0].agentMs).toBe(2 * HOUR);
});
it('overlapping same-class sessions are UNIONed, not summed (no double-count)', () => {
// Two work sessions that overlap 10:00–10:30 on one day.
const a = work(Date.UTC(2026, 0, 10, 9, 0), Date.UTC(2026, 0, 10, 10, 30));
const b = work(Date.UTC(2026, 0, 10, 10, 0), Date.UTC(2026, 0, 10, 11, 0));
const perDay = bucketByDay([a, b], 'UTC');
expect(perDay).toHaveLength(1);
// Union 09:00–11:00 = 2h, NOT 90m + 60m = 150m.
expect(perDay[0].activeMs).toBe(2 * HOUR);
// The drawn windows are also merged to one, so the punch-card cannot render
// an overlapping double bar.
expect(perDay[0].windows).toHaveLength(1);
expect(perDay[0].windows[0].start).toBe(a.start);
expect(perDay[0].windows[0].end).toBe(b.end);
});
it('DST fall-back: a full 25-hour day still balances (§9#14)', () => {
// America/New_York ends DST 2026-11-01 (25h day).
const tz = 'America/New_York';
const dayStart = zonedDayStart(Date.UTC(2026, 10, 1, 12, 0), tz);
const nextStart = zonedDayStart(dayStart + 26 * HOUR, tz);
expect(nextStart - dayStart).toBe(25 * HOUR);
const perDay = bucketByDay([work(dayStart, nextStart)], tz);
expect(perDay).toHaveLength(1);
expect(perDay[0].dayISO).toBe('2026-11-01');
expect(perDay[0].activeMs).toBe(25 * HOUR);
expect(sumActive(perDay)).toBe(nextStart - dayStart);
});
it('DST spring-forward: a full 23-hour day still balances (§9#14)', () => {
// America/New_York starts DST 2026-03-08 (23h day).
const tz = 'America/New_York';
const dayStart = zonedDayStart(Date.UTC(2026, 2, 8, 12, 0), tz);
const nextStart = zonedDayStart(dayStart + 26 * HOUR, tz);
expect(nextStart - dayStart).toBe(23 * HOUR);
const perDay = bucketByDay([work(dayStart, nextStart)], tz);
expect(perDay).toHaveLength(1);
expect(perDay[0].activeMs).toBe(23 * HOUR);
expect(sumActive(perDay)).toBe(nextStart - dayStart);
});
it('tz changes the day a session lands in but not the total', () => {
const start = Date.UTC(2026, 0, 10, 2, 0); // 02:00 UTC
const end = Date.UTC(2026, 0, 10, 3, 0);
const utc = bucketByDay([work(start, end)], 'UTC');
const ny = bucketByDay([work(start, end)], 'America/New_York'); // 21:00 prev day
expect(utc[0].dayISO).toBe('2026-01-10');
expect(ny[0].dayISO).toBe('2026-01-09');
expect(sumActive(utc)).toBe(sumActive(ny));
});
});
@@ -0,0 +1,180 @@
import { WorkSession, PerDay, DayWindow } from './work-time.types';
/**
* Merge intervals into a disjoint, sorted union. Overlapping OR touching
* intervals are joined. Empty input [].
*/
function union(intervals: Array<[number, number]>): Array<[number, number]> {
if (intervals.length === 0) return [];
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
const out: Array<[number, number]> = [];
let [curStart, curEnd] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= curEnd) {
if (e > curEnd) curEnd = e;
} else {
out.push([curStart, curEnd]);
curStart = s;
curEnd = e;
}
}
out.push([curStart, curEnd]);
return out;
}
// Cache one Intl formatter per tz — constructing them is comparatively costly.
const fmtCache = new Map<string, Intl.DateTimeFormat>();
function partsFmt(tz: string): Intl.DateTimeFormat {
let fmt = fmtCache.get(tz);
if (!fmt) {
fmt = new Intl.DateTimeFormat('en-US', {
timeZone: tz,
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit',
second: '2-digit',
hour12: false,
});
fmtCache.set(tz, fmt);
}
return fmt;
}
interface WallParts {
year: number;
month: number;
day: number;
hour: number;
minute: number;
second: number;
}
/** Wall-clock parts of an instant in `tz` (DST-correct, via Intl). */
function wallParts(ms: number, tz: string): WallParts {
const parts = partsFmt(tz).formatToParts(new Date(ms));
const get = (type: string) =>
Number(parts.find((p) => p.type === type)?.value ?? '0');
let hour = get('hour');
// Intl emits "24" for midnight under some engines/locales; normalize to 0.
if (hour === 24) hour = 0;
return {
year: get('year'),
month: get('month'),
day: get('day'),
hour,
minute: get('minute'),
second: get('second'),
};
}
/** tz offset (wall − real) at an instant, in ms. */
function offset(ms: number, tz: string): number {
const p = wallParts(ms, tz);
const asUTC = Date.UTC(p.year, p.month - 1, p.day, p.hour, p.minute, p.second);
return asUTC - ms;
}
/**
* Epoch-ms of the local-midnight day start of `ms` in `tz`. DST-correct: takes
* the calendar day of the instant, its wall-midnight, then converts back with
* the offset that actually applies AT that midnight (refined once). The rare
* tz-with-a-DST-transition-exactly-at-midnight case is a documented edge (§9#14).
*/
export function zonedDayStart(ms: number, tz: string): number {
const p = wallParts(ms, tz);
const wallMidnightAsUTC = Date.UTC(p.year, p.month - 1, p.day, 0, 0, 0);
let start = wallMidnightAsUTC - offset(ms, tz);
// Refine with the offset at the computed midnight (DST may differ from `ms`).
start = wallMidnightAsUTC - offset(start, tz);
return start;
}
/** The next local midnight after `dayStart` (handles 23/25h DST days). */
function nextDayStart(dayStart: number, tz: string): number {
// +26h always lands inside the NEXT calendar day (day length ∈ [23h,25h]),
// never two days ahead; startOf('day') of it is the next midnight.
return zonedDayStart(dayStart + 26 * 60 * 60 * 1000, tz);
}
function isoDay(dayStart: number, tz: string): string {
const p = wallParts(dayStart, tz);
const pad = (n: number) => String(n).padStart(2, '0');
return `${p.year}-${pad(p.month)}-${pad(p.day)}`;
}
/** Clip a union to [lo, hi) and emit windows of `class`. */
function clip(
merged: Array<[number, number]>,
lo: number,
hi: number,
cls: DayWindow['class'],
): DayWindow[] {
const out: DayWindow[] = [];
for (const [s, e] of merged) {
const start = Math.max(s, lo);
const end = Math.min(e, hi);
if (end > start) out.push({ start, end, class: cls });
}
return out;
}
/**
* #395 §6.3 bucket sessions into calendar days of `tz` for the punch-card.
* Pure and deterministic. `work` and `agent_only` are unioned SEPARATELY (else
* agent windows would swallow work windows on overlap), then each union is split
* at tz midnight boundaries (`startOf('day')` in tz, NOT "+24h" DST-safe §9#14)
* and clipped to each day.
*
* By construction Σ perDay.activeMs == workMs: the days are a partition of the
* `work` union no loss, no dup, even on 23/25h DST days. `agent_only` windows
* are drawn but NOT in activeMs. Empty days between the first and last active day
* are emitted (empty track + "—") so the rhythm/pauses stay visible.
*/
export function bucketByDay(sessions: WorkSession[], tz: string): PerDay[] {
const uWork = union(
sessions.filter((s) => s.class === 'work').map((s) => [s.start, s.end]),
);
const uAgent = union(
sessions
.filter((s) => s.class === 'agent_only')
.map((s) => [s.start, s.end]),
);
if (uWork.length === 0 && uAgent.length === 0) return [];
const minStart = Math.min(
uWork.length ? uWork[0][0] : Infinity,
uAgent.length ? uAgent[0][0] : Infinity,
);
const maxEnd = Math.max(
uWork.length ? uWork[uWork.length - 1][1] : -Infinity,
uAgent.length ? uAgent[uAgent.length - 1][1] : -Infinity,
);
const perDay: PerDay[] = [];
let dayStart = zonedDayStart(minStart, tz);
// Guard against a pathological non-advancing boundary.
let guard = 0;
while (dayStart < maxEnd && guard < 100000) {
guard++;
const dayEnd = nextDayStart(dayStart, tz);
const workWin = clip(uWork, dayStart, dayEnd, 'work');
const agentWin = clip(uAgent, dayStart, dayEnd, 'agent_only');
const activeMs = workWin.reduce((a, w) => a + (w.end - w.start), 0);
const agentMs = agentWin.reduce((a, w) => a + (w.end - w.start), 0);
const windows = [...workWin, ...agentWin].sort((a, b) => a.start - b.start);
perDay.push({
day: dayStart,
dayISO: isoDay(dayStart, tz),
activeMs,
agentMs,
windows,
});
dayStart = dayEnd;
}
return perDay;
}
@@ -0,0 +1,358 @@
import { computeWorkTime } from './compute-work-time';
import { bucketByDay } from './bucket-by-day';
import { TimelineSample, WorkSession } from './work-time.types';
const MIN = 60 * 1000;
/** Union wall-clock of a set of intervals (touching intervals merge). */
function unionMs(intervals: Array<[number, number]>): number {
if (intervals.length === 0) return 0;
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
let total = 0;
let [cs, ce] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= ce) {
if (e > ce) ce = e;
} else {
total += ce - cs;
cs = s;
ce = e;
}
}
return total + (ce - cs);
}
const ivsOf = (sessions: WorkSession[], cls?: string): Array<[number, number]> =>
sessions
.filter((x) => cls == null || x.class === cls)
.map((x) => [x.start, x.end] as [number, number]);
function s(
iso: string,
opts: {
source?: string | null;
chat?: string | null;
kind?: string | null;
by?: string | null;
} = {},
): TimelineSample {
return {
createdAt: `${iso}Z`,
lastUpdatedById: opts.by ?? 'human-1',
lastUpdatedSource: opts.source === undefined ? 'user' : opts.source,
lastUpdatedAiChatId: opts.chat ?? null,
kind: opts.kind ?? null,
};
}
// §7 config: T_gap=30m, P_in+P_out=10m, P_single=2m.
const S7 = { tGap: 30 * MIN, agentTGap: 30 * MIN, pIn: 5 * MIN, pOut: 5 * MIN, pSingle: 2 * MIN };
describe('computeWorkTime', () => {
it('§7 fixture — sessionizes 20-ish samples to ≈1h32m, not the ≈60h naive span', () => {
const rows: TimelineSample[] = [
// S1: multi-sample morning session
s('2026-07-04T03:40:00'),
s('2026-07-04T03:45:00'),
s('2026-07-04T03:49:00'),
// S2: agent burst (one run) then human supervising → class work
s('2026-07-04T15:43:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T15:47:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T15:50:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T16:13:00'),
// S3: single
s('2026-07-04T18:11:00'),
// S4: multi-sample evening session
s('2026-07-04T19:38:00'),
s('2026-07-04T19:44:00'),
s('2026-07-04T19:54:00'),
// S5 / S6: two singles two days later, 44m apart → two sessions at T_gap=30
s('2026-07-06T15:34:00'),
s('2026-07-06T16:18:00'),
];
const r = computeWorkTime(rows, S7);
// 19 + 40 + 2 + 26 + 2 + 2 = 91 minutes.
expect(r.workMs).toBe(91 * MIN);
expect(r.agentOnlyMs).toBe(0);
expect(r.sessions).toHaveLength(6);
expect(r.sessions.every((x) => x.class === 'work')).toBe(true);
const naiveSpan =
new Date('2026-07-06T16:18:00Z').getTime() -
new Date('2026-07-04T03:40:00Z').getTime();
expect(naiveSpan).toBeGreaterThan(60 * 60 * MIN); // ≈60h
expect(r.workMs).toBeLessThan(naiveSpan / 30); // dramatically smaller
});
it('n=0 → zero, no sessions', () => {
const r = computeWorkTime([]);
expect(r).toEqual({ workMs: 0, agentOnlyMs: 0, sessions: [] });
});
it('n=1 human → one P_single work session', () => {
const r = computeWorkTime([s('2026-07-04T10:00:00')], S7);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.workMs).toBe(2 * MIN);
expect(r.agentOnlyMs).toBe(0);
// pre-roll only: [t − P_single, t]
expect(r.sessions[0].end).toBe(new Date('2026-07-04T10:00:00Z').getTime());
});
it('n=1 agent → one P_single agent_only session, work=0 (§9#2)', () => {
const r = computeWorkTime(
[s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' })],
S7,
);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('agent_only');
expect(r.workMs).toBe(0);
expect(r.agentOnlyMs).toBe(2 * MIN);
});
it('MUST close the last session — the newest session is not lost (§9#1)', () => {
// Two singles a day apart: without the post-loop close, the 2nd is dropped.
const rows = [s('2026-07-04T10:00:00'), s('2026-07-05T10:00:00')];
const r = computeWorkTime(rows, S7);
expect(r.sessions).toHaveLength(2);
const lastStart = Math.max(...r.sessions.map((x) => x.start));
expect(lastStart).toBe(
new Date('2026-07-05T10:00:00Z').getTime() - 2 * MIN,
);
expect(r.workMs).toBe(4 * MIN);
});
it('agent-burst collapse: density does not inflate — length = wall-clock', () => {
const span = ['00', '01', '02', '03', '04', '05', '06'];
const dense: TimelineSample[] = span.map((sec) =>
s(`2026-07-04T10:00:${sec}`, { source: 'agent', chat: 'c1', kind: 'agent' }),
);
const sparse: TimelineSample[] = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:00:06', { source: 'agent', chat: 'c1', kind: 'agent' }),
];
const rDense = computeWorkTime(dense, S7);
const rSparse = computeWorkTime(sparse, S7);
// Same 6-second wall-clock span → same estimate regardless of snapshot count.
expect(rDense.agentOnlyMs).toBe(rSparse.agentOnlyMs);
expect(rDense.sessions).toHaveLength(1);
expect(rDense.sessions[0].class).toBe('agent_only');
});
it('supervisory agent time inside a human session counts as work, not agent', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:05:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:12:00'), // human within T_gap
];
const r = computeWorkTime(rows, S7);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.agentOnlyMs).toBe(0);
expect(r.workMs).toBeGreaterThan(0);
});
it('a DIFFERENT aiChatId breaks the burst — two agent runs, idle gap excluded', () => {
// Run c1 ends 10:05, run c2 starts 10:20 (15m > agentTGap 7m) → two sessions.
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:05:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:20:00', { source: 'agent', chat: 'c2', kind: 'agent' }),
s('2026-07-04T10:25:00', { source: 'agent', chat: 'c2', kind: 'agent' }),
];
const r = computeWorkTime(rows); // default agentTGap = 7m
expect(r.sessions).toHaveLength(2);
expect(r.sessions.every((x) => x.class === 'agent_only')).toBe(true);
// The 15m idle gap between the two runs is NOT counted.
const run1 = 5 * MIN + 5 * MIN + 5 * MIN; // pIn + span + pOut
expect(r.agentOnlyMs).toBe(2 * run1);
});
it('idle pulse (same/null run) is a full activity sample that continues a burst', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
// idle flush 4m later, null run id → continues the burst, not a new one
s('2026-07-04T10:04:00', { source: 'agent', chat: null, kind: 'idle' }),
s('2026-07-04T10:08:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
// burst span 10:00→10:08 (+pIn/pOut) = 8 + 10 = 18m
expect(r.agentOnlyMs).toBe(18 * MIN);
});
it('a USER-sourced idle breaks an agent burst → session is work, not agent_only', () => {
// A human supervision idle inherits source=user (aiChatId:null) and must NOT
// be swallowed into the agent burst. Δ=3m is within the default agentTGap so
// the two samples stay one session — but its class flips to `work`.
const rows = [
s('2026-07-04T10:00:00', { source: 'agent', chat: 'c1', kind: 'agent' }),
s('2026-07-04T10:03:00', { source: 'user', chat: null, kind: 'idle' }),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
expect(r.workMs).toBeGreaterThan(0);
// The human idle is NOT captured as agent_only time.
expect(r.agentOnlyMs).toBe(0);
// Σ over `work` sessions == workMs and Σ over `agent_only` == agentOnlyMs.
const sum = (cls: string) =>
r.sessions
.filter((x) => x.class === cls)
.reduce((acc, x) => acc + (x.end - x.start), 0);
expect(sum('work')).toBe(r.workMs);
expect(sum('agent_only')).toBe(r.agentOnlyMs);
});
it('idle pulse keeps a human writing session visible (not excluded)', () => {
const rows = [
s('2026-07-04T10:00:00'),
s('2026-07-04T10:08:00', { kind: 'idle' }), // pulse within T_gap
s('2026-07-04T10:15:00'),
];
const r = computeWorkTime(rows);
expect(r.sessions).toHaveLength(1);
expect(r.sessions[0].class).toBe('work');
// span 10:00→10:15 + pIn/pOut = 15 + 10 = 25m
expect(r.workMs).toBe(25 * MIN);
});
it('git-source samples are excluded (§10 excludeGit)', () => {
const rows = [
s('2026-07-04T10:00:00', { source: 'git', kind: 'boundary' }),
s('2026-07-04T10:01:00', { source: 'git', kind: 'boundary' }),
];
expect(computeWorkTime(rows).workMs).toBe(0);
// ...but honoured off:
expect(
computeWorkTime(rows, { excludeGit: false }).workMs,
).toBeGreaterThan(0);
});
it('rejects an invalid config (tGap < pIn + pOut)', () => {
expect(() =>
computeWorkTime([s('2026-07-04T10:00:00')], {
tGap: 5 * MIN,
pIn: 5 * MIN,
pOut: 5 * MIN,
}),
).toThrow(/tGap/);
});
it('rejects an invalid config (2·agentTGap < pIn + pOut)', () => {
// tGap (default 15m) still ≥ pIn+pOut, so only the 2·agentTGap guard trips.
// Without it a short session of one class between two of the other could
// produce a NON-adjacent cross-class overlap the adjacent-only clip misses.
expect(() =>
computeWorkTime([s('2026-07-04T10:00:00')], {
agentTGap: 2 * MIN,
pIn: 5 * MIN,
pOut: 5 * MIN,
}),
).toThrow(/agentTGap/);
});
// F1 — cross-class double-count. On the DEFAULT config agentTGap (7m) < pIn+pOut
// (10m), so a `work` session ending in an agent segment and a nearby separate
// `agent_only` run (gap in (7m,10m]) used to produce OVERLAPPING padded
// intervals — the same wall-clock counted into BOTH workMs and agentOnlyMs. The
// cross-class padding clip must make the two per-class unions disjoint.
it('does NOT double-count wall-clock across work/agent_only (§F1)', () => {
// user@0s ; agent(chatX)@60s (breaks into a work session with the human) ;
// agent(chatY)@560s,590s (a separate agent_only run). Raw gap between the work
// session (ends 60s) and the agent run (starts 560s) is 500s ∈ (agentTGap,
// pIn+pOut] once padded — the classic overlap window.
const rows: TimelineSample[] = [
s('2026-07-04T00:00:00'), // user @ 0s
s('2026-07-04T00:01:00', { source: 'agent', chat: 'cX', kind: 'agent' }), // @ 60s
s('2026-07-04T00:09:20', { source: 'agent', chat: 'cY', kind: 'agent' }), // @ 560s
s('2026-07-04T00:09:50', { source: 'agent', chat: 'cY', kind: 'agent' }), // @ 590s
];
const r = computeWorkTime(rows); // DEFAULT config
// Both classes present.
expect(r.workMs).toBeGreaterThan(0);
expect(r.agentOnlyMs).toBeGreaterThan(0);
// Per-class metrics are exactly their own union (union, not Σ).
expect(r.workMs).toBe(unionMs(ivsOf(r.sessions, 'work')));
expect(r.agentOnlyMs).toBe(unionMs(ivsOf(r.sessions, 'agent_only')));
// The F1 invariant: work-union and agent-union are cross-class-disjoint, so
// the union of ALL padded intervals equals workMs + agentOnlyMs (no overlap).
// With the clip disabled this fails (union < sum by the 100s overlap).
expect(unionMs(ivsOf(r.sessions))).toBe(r.workMs + r.agentOnlyMs);
});
// F1 property/fuzz — random timelines across several timezones must uphold the
// work-time invariants. Backs the (corrected) PR claim of a real fuzz test.
it('property: random timelines uphold union & cross-class-disjoint invariants', () => {
// Deterministic LCG (numerical-recipes constants) so a failure is reproducible.
let seed = 0x9e3779b9 >>> 0;
const rand = () => {
seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0;
return seed / 0x100000000;
};
const pick = <T>(arr: T[]): T => arr[Math.floor(rand() * arr.length)];
const tzs = [
'UTC',
'America/New_York',
'Europe/Moscow',
'Australia/Lord_Howe', // 30-min DST offset — a nasty bucket stress
];
const base = Date.UTC(2026, 5, 1, 0, 0, 0); // 2026-06-01Z
const chats = ['c1', 'c2', 'c3'];
for (let iter = 0; iter < 250; iter++) {
const tz = pick(tzs);
const n = 2 + Math.floor(rand() * 18); // 2..19 rows
const rows: TimelineSample[] = [];
// Walk time forward by a random inter-sample gap. The gap distribution is
// centred on the DANGEROUS band — a bit under to a bit over pIn+pOut (10m)
// AND straddling agentTGap (7m) — so adjacent samples routinely split into
// separate sessions whose ±P padding would overlap if a class boundary sits
// there. Mixing user/agent classes at these gaps reliably manufactures the
// work-ending-in-agent → agent_only cross-class boundary F1 is about, plus
// dense within-class runs (occasional 0–2m gaps) that exercise the union.
let t = base + Math.floor(rand() * 60 * MIN);
for (let i = 0; i < n; i++) {
const roll = rand();
const gap =
roll < 0.25
? Math.floor(rand() * 2 * MIN) // dense burst (same-class union)
: roll < 0.85
? 5 * MIN + Math.floor(rand() * 8 * MIN) // 5–13m: the split band
: 20 * MIN + Math.floor(rand() * 40 * MIN); // long idle → new day-ish
t += gap;
const iso = new Date(t).toISOString().slice(0, 19); // 'YYYY-MM-DDTHH:MM:SS'
const isAgent = rand() < 0.5;
rows.push(
isAgent
? s(iso, { source: 'agent', chat: pick(chats), kind: 'agent' })
: s(iso, { source: 'user', kind: rand() < 0.3 ? 'idle' : 'manual' }),
);
}
const r = computeWorkTime(rows); // DEFAULT config
const workIvs = ivsOf(r.sessions, 'work');
const agentIvs = ivsOf(r.sessions, 'agent_only');
// (1) each metric is exactly its per-class union (catches a union→Σ regress).
expect(r.workMs).toBe(unionMs(workIvs));
expect(r.agentOnlyMs).toBe(unionMs(agentIvs));
// (2) NO cross-class overlap: union(all) == workMs + agentOnlyMs (F1).
expect(unionMs(ivsOf(r.sessions))).toBe(r.workMs + r.agentOnlyMs);
// (3) bucket invariant: Σ per-day activeMs == workMs (§6.3).
const perDay = bucketByDay(r.sessions, tz);
const sumActive = perDay.reduce((a, d) => a + d.activeMs, 0);
expect(sumActive).toBe(r.workMs);
}
});
});
@@ -0,0 +1,274 @@
import {
TimelineSample,
WorkSession,
WorkTimeResult,
} from './work-time.types';
import { WorkTimeConfig, resolveWorkTimeConfig } from './work-time.config';
/** A normalized activity sample (one history row), createdAt as epoch-ms. */
interface NormSample {
t: number;
isAgent: boolean;
aiChatId: string | null;
kind: string | null;
}
/**
* A collapsed segment: either a scalar sample (t_start == t_end) or an
* agent-burst spanning several agent samples of one run (§5.1). It participates
* in sessionization as a single "sample".
*/
interface Segment {
tStart: number;
tEnd: number;
isAgent: boolean;
}
function toMs(v: Date | string | number): number {
if (v instanceof Date) return v.getTime();
if (typeof v === 'number') return v;
return new Date(v).getTime();
}
/**
* Normalize raw rows sorted, deduped activity samples. `git` is dropped when
* configured; every other kind (incl. `idle` the continuous-work pulse §3) is
* a real activity sample. Sort is by createdAt ASC; samples whose timestamps
* fall in the same `dedupRoundMs` bucket collapse to one (§9#7: a synchronous
* boundary row + the immediate agent snapshot can share a createdAt). A merged
* sample is human unless EVERY member is an agent, so supervision never gets
* mis-attributed to the agent.
*/
function normalize(
rows: TimelineSample[],
config: WorkTimeConfig,
): NormSample[] {
const samples: NormSample[] = [];
for (const row of rows) {
const source = row.lastUpdatedSource;
if (config.excludeGit && source === 'git') continue;
samples.push({
t: toMs(row.createdAt),
isAgent: source === 'agent',
aiChatId: row.lastUpdatedAiChatId ?? null,
kind: row.kind ?? null,
});
}
samples.sort((a, b) => a.t - b.t);
if (config.dedupRoundMs <= 0 || samples.length < 2) return samples;
const deduped: NormSample[] = [];
for (const s of samples) {
const prev = deduped[deduped.length - 1];
if (prev && s.t - prev.t < config.dedupRoundMs) {
// Merge into the previous sample. Human wins the class; keep the earliest
// t; keep a non-null aiChatId if either has one (so a bare boundary row
// does not erase the run id).
prev.isAgent = prev.isAgent && s.isAgent;
prev.aiChatId = prev.aiChatId ?? s.aiChatId;
// Prefer the more specific kind (a real kind over a null/boundary) only
// matters for burst continuation; keep prev.kind (earliest) as-is.
continue;
}
deduped.push({ ...s });
}
return deduped;
}
/**
* Collapse consecutive same-run agent samples into one burst segment (§5.1) so a
* dense burst (8 snapshots in 7 minutes) contributes its wall-clock, not a count
* × block. A burst is broken by any sample NOT continuing the same aiChatId
* agent run: a non-agent sample, a `boundary` (actor transition), or a DIFFERENT
* aiChatId. Only an AGENT-sourced `idle` pulse with the SAME or a null aiChatId
* continues the burst (its label lags the real edit maxWait, well within
* rounding); a user-sourced `idle` (a human supervision pulse) breaks it.
*/
function collapse(samples: NormSample[], config: WorkTimeConfig): Segment[] {
const segments: Segment[] = [];
let burst: { chatId: string | null; tStart: number; tEnd: number } | null =
null;
const flush = () => {
if (!burst) return;
let tEnd = burst.tEnd;
if (config.burstCapMs != null && tEnd - burst.tStart > config.burstCapMs) {
tEnd = burst.tStart + config.burstCapMs;
}
segments.push({ tStart: burst.tStart, tEnd, isAgent: true });
burst = null;
};
for (const s of samples) {
// An agent-sourced idle pulse continues the current agent burst (same or
// null run id). A user-sourced idle (human supervision) must NOT be swallowed
// here — it falls through to the human branch so the session flips to `work`.
if (
burst &&
s.kind === 'idle' &&
s.isAgent &&
(s.aiChatId === burst.chatId || s.aiChatId == null)
) {
burst.tEnd = s.t;
continue;
}
if (s.isAgent && s.kind !== 'boundary') {
if (burst && burst.chatId === s.aiChatId) {
burst.tEnd = s.t;
} else {
flush();
burst = { chatId: s.aiChatId, tStart: s.t, tEnd: s.t };
}
continue;
}
// A human sample, a boundary, or an agent-boundary: breaks the burst and is
// itself a zero-width segment (its class follows its own source).
flush();
segments.push({ tStart: s.t, tEnd: s.t, isAgent: s.isAgent });
}
flush();
return segments;
}
function gapThreshold(
a: Segment,
b: Segment,
config: WorkTimeConfig,
): number {
return a.isAgent && b.isAgent ? config.agentTGap : config.tGap;
}
/** Merge intervals; overlapping OR touching intervals are unioned. */
function unionDuration(intervals: Array<[number, number]>): number {
if (intervals.length === 0) return 0;
const sorted = [...intervals].sort((a, b) => a[0] - b[0]);
let total = 0;
let [curStart, curEnd] = sorted[0];
for (let i = 1; i < sorted.length; i++) {
const [s, e] = sorted[i];
if (s <= curEnd) {
if (e > curEnd) curEnd = e;
} else {
total += curEnd - curStart;
curStart = s;
curEnd = e;
}
}
total += curEnd - curStart;
return total;
}
/**
* #395 core estimate time worked on a page from its history timeline (§5).
* Pure and deterministic: no DB, no clock, no I/O.
*
* Pipeline: normalize+dedup collapse agent bursts ONE sessionization pass
* over all segments (threshold depends on the pair: both-agent agentTGap, else
* tGap; the last session is ALWAYS closed after the loop) class per finished
* session (all-agent agent_only, else work) pad each session (multi-sample
* [firstP_in, last+P_out]; lone scalar [tP_single, t]) clip padding of
* adjacent DIFFERENT-class sessions at the raw-gap midpoint (so work/agent_only
* never overlap) metrics are the union wall-clock within each class (union, not
* Σ, so overlaps never double, and cross-class-disjoint by the clip above).
*/
export function computeWorkTime(
rows: TimelineSample[],
config?: Partial<WorkTimeConfig>,
): WorkTimeResult {
const cfg = resolveWorkTimeConfig(config);
const samples = normalize(rows, cfg);
const segments = collapse(samples, cfg);
// Sessionize — one pass over ALL segments.
const rawSessions: Segment[][] = [];
let cur: Segment[] | null = null;
for (const seg of segments) {
if (cur == null) {
cur = [seg];
} else {
const last = cur[cur.length - 1];
if (seg.tStart - last.tEnd <= gapThreshold(last, seg, cfg)) {
cur.push(seg);
} else {
rawSessions.push(cur);
cur = [seg];
}
}
}
if (cur != null) rawSessions.push(cur); // MUST close the last session (§5, §9#1)
// A finished session with BOTH its raw (unpadded) span and its padded bounds.
// `rawSessions` are already in ascending time order, so `built` is too.
interface BuiltSession {
rawStart: number;
rawEnd: number;
padStart: number;
padEnd: number;
cls: WorkSession['class'];
}
const built: BuiltSession[] = [];
for (const segs of rawSessions) {
const first = segs[0];
const last = segs[segs.length - 1];
const cls = segs.every((s) => s.isAgent) ? 'agent_only' : 'work';
let padStart: number;
let padEnd: number;
if (segs.length === 1 && first.tStart === first.tEnd) {
// Lone single-instant session (one scalar, or a one-snapshot agent run):
// pre-roll only, no invented "future" work (§5).
padStart = first.tStart - cfg.pSingle;
padEnd = first.tStart;
} else {
padStart = first.tStart - cfg.pIn;
padEnd = last.tEnd + cfg.pOut;
}
built.push({
rawStart: first.tStart,
rawEnd: last.tEnd,
padStart,
padEnd,
cls,
});
}
// Clip cross-class padding so a `work` and an `agent_only` session that abut
// never claim the same wall-clock. For each ADJACENT pair of DIFFERENT classes,
// cap the earlier session's trailing pad and the later session's leading pad at
// the MIDPOINT of the raw (unpadded) inactivity gap between them: the earlier
// padded interval then ends ≤ midpoint and the later one starts ≥ midpoint, so
// the two are disjoint (they touch at most at the midpoint). This makes the
// per-class unions (workMs / agentOnlyMs) cross-class-disjoint BY CONSTRUCTION
// — closing the double-count where a work session ending in an agent segment
// and a nearby agent_only session (gap in (agentTGap, pIn+pOut]) overlapped and
// were counted into both metrics (§5, §9). Within-class adjacency is left
// untouched: `unionDuration` already dedups it, and clipping there could perturb
// the per-class metric value.
for (let i = 1; i < built.length; i++) {
const a = built[i - 1];
const b = built[i];
if (a.cls === b.cls) continue;
const midpoint = (a.rawEnd + b.rawStart) / 2;
if (a.padEnd > midpoint) a.padEnd = midpoint;
if (b.padStart < midpoint) b.padStart = midpoint;
}
const sessions: WorkSession[] = [];
const workIvs: Array<[number, number]> = [];
const agentIvs: Array<[number, number]> = [];
for (const s of built) {
sessions.push({ start: s.padStart, end: s.padEnd, class: s.cls });
(s.cls === 'work' ? workIvs : agentIvs).push([s.padStart, s.padEnd]);
}
sessions.sort((a, b) => a.start - b.start);
return {
workMs: unionDuration(workIvs),
agentOnlyMs: unionDuration(agentIvs),
sessions,
};
}
@@ -0,0 +1,15 @@
export { computeWorkTime } from './compute-work-time';
export { bucketByDay, zonedDayStart } from './bucket-by-day';
export {
DEFAULT_WORK_TIME_CONFIG,
resolveWorkTimeConfig,
} from './work-time.config';
export type { WorkTimeConfig } from './work-time.config';
export type {
TimelineSample,
WorkSession,
WorkTimeResult,
SessionClass,
DayWindow,
PerDay,
} from './work-time.types';
@@ -0,0 +1,102 @@
import {
IDLE_MAX_WAIT_USER,
IDLE_MAX_WAIT_AGENT,
} from '../../../collaboration/constants';
/**
* #395 tunables for the work-time estimate (§10). Defaults are calibrated off
* #374's idle-pulse ceilings: after #374 a continuous editing session leaves a
* history row at least every ~IDLE_MAX_WAIT (10m user / 5m agent), so a gap
* WIDER than that ceiling contains un-pulsed idle time = (partial) inactivity.
* `tGap` therefore sits a little above the user ceiling, `agentTGap` a little
* above the agent ceiling a gap within the threshold is pulse-backed and
* counts as work.
*/
export interface WorkTimeConfig {
/** user inactivity timeout: gap ≤ tGap between samples = continuous work. */
tGap: number;
/** timeout for a pair of consecutive agent samples (tighter than tGap). */
agentTGap: number;
/** pre-roll padding for a multi-sample session (work began before sample 1). */
pIn: number;
/** post-roll padding for a multi-sample session (work continued after last). */
pOut: number;
/** block for a lone single-sample session (pre-roll only, no invented future). */
pSingle: number;
/** drop `git`-source samples (they are not human/agent article work). */
excludeGit: boolean;
/** optional cap on one collapsed agent-burst segment's wall-clock (§9#3). */
burstCapMs?: number;
/** samples whose createdAt round to the same bucket dedup to one (§9#7). */
dedupRoundMs: number;
}
export const DEFAULT_WORK_TIME_CONFIG: WorkTimeConfig = {
// ~15m: IDLE_MAX_WAIT_USER (10m) + headroom. Empirically backcast on a real
// 307-snapshot article (≈24h at 15m matched the owner's estimate; 30/45m
// over-counted). See #395 §10.
tGap: 15 * 60 * 1000,
// ~7m: IDLE_MAX_WAIT_AGENT (5m) + headroom.
agentTGap: 7 * 60 * 1000,
pIn: 5 * 60 * 1000,
pOut: 5 * 60 * 1000,
pSingle: 2 * 60 * 1000,
excludeGit: true,
burstCapMs: undefined,
dedupRoundMs: 1000,
};
// Compile-time cross-check that the defaults really are pulse-anchored — if a
// future edit moves the #374 ceilings, this reminds us to re-calibrate.
void IDLE_MAX_WAIT_USER;
void IDLE_MAX_WAIT_AGENT;
/**
* Fill a partial config with defaults and validate it. Cross-class metric
* disjointness is guaranteed jointly by `computeWorkTime`'s adjacent-pair padding
* clip (it caps the padding of adjacent DIFFERENT-class sessions at the raw-gap
* midpoint) AND the two bounds enforced below (§5):
* - `tGap ≥ pIn + pOut`: a session's own padding never exceeds its inactivity
* window.
* - `2·agentTGap ≥ pIn + pOut`: makes the adjacent-only clip provably COMPLETE.
* A NON-adjacent (i, i+2) cross-class overlap could only arise from two
* same-class sessions separated by a full intervening session of the other
* class; that separation spans at least two inter-session gaps, each strictly
* `> agentTGap`, so it is `> 2·agentTGap`. Requiring `2·agentTGap ≥ pIn + pOut`
* means even the widest padded reach (pIn + pOut) cannot bridge it so the
* only cross-class overlaps possible are between ADJACENT sessions, which the
* clip handles. `workMs`/`agentOnlyMs` are therefore disjoint by construction.
*/
export function resolveWorkTimeConfig(
partial?: Partial<WorkTimeConfig>,
): WorkTimeConfig {
const config = { ...DEFAULT_WORK_TIME_CONFIG, ...(partial ?? {}) };
for (const key of [
'tGap',
'agentTGap',
'pIn',
'pOut',
'pSingle',
'dedupRoundMs',
] as const) {
const value = config[key];
if (!Number.isFinite(value) || value < 0) {
throw new Error(`work-time config: ${key} must be a non-negative number`);
}
}
if (config.burstCapMs != null && config.burstCapMs <= 0) {
throw new Error('work-time config: burstCapMs must be > 0 when set');
}
if (config.tGap < config.pIn + config.pOut) {
throw new Error(
"work-time config: tGap must be ≥ pIn + pOut (a session's padding may not exceed its inactivity window)",
);
}
if (2 * config.agentTGap < config.pIn + config.pOut) {
throw new Error(
'work-time config: 2·agentTGap must be ≥ pIn + pOut (so non-adjacent cross-class padding cannot overlap)',
);
}
return config;
}
@@ -0,0 +1,73 @@
/**
* #395 "time worked on an article" domain types.
*
* The estimate is built by sessionizing a page's `page_history` timeline on
* inactivity gaps (WakaTime-style), NOT by taking the span between the first and
* last edit (which over-counts sleep / lunch / idle days). See the design doc in
* issue #395 §5§6.3 for the normative algorithm.
*/
/**
* A single `page_history` row projected for the work-time computation the
* cheap columns only (no `content`). Produced by
* `PageHistoryRepo.findTimelineByPageId`. `createdAt` is whatever the DB driver
* hands back (Date); the pure core normalizes it to epoch-ms itself so it stays
* deterministic and DB-free.
*/
export interface TimelineSample {
createdAt: Date | string | number;
lastUpdatedById: string | null;
/** 'user' | 'agent' | 'git' | null (legacy autosave = human). */
lastUpdatedSource: string | null;
lastUpdatedAiChatId: string | null;
/** #370 tier: 'manual' | 'agent' | 'idle' | 'boundary' | null (legacy). */
kind: string | null;
}
/** A finished session's class (§5.1). */
export type SessionClass = 'work' | 'agent_only';
/**
* A finished session: absolute wall-clock bounds already padded with P_in/P_out
* (multi-sample) or P_single (single scalar), plus its class. This is enough for
* both the metrics and the per-day punch-card colouring.
*/
export interface WorkSession {
/** epoch-ms, inclusive lower bound (already P-padded). */
start: number;
/** epoch-ms, exclusive upper bound (already P-padded). */
end: number;
class: SessionClass;
}
/** Output of {@link computeWorkTime}. */
export interface WorkTimeResult {
/** union wall-clock of `work` sessions, ms (the headline metric). */
workMs: number;
/** union wall-clock of `agent_only` sessions, ms (secondary). */
agentOnlyMs: number;
sessions: WorkSession[];
}
/** One activity window inside a calendar day (already clipped to the day). */
export interface DayWindow {
/** epoch-ms. */
start: number;
/** epoch-ms. */
end: number;
class: SessionClass;
}
/** One calendar day of the punch-card (§6.3). */
export interface PerDay {
/** epoch-ms of the local-midnight day start in the requested tz. */
day: number;
/** 'YYYY-MM-DD' in the requested tz — stable, tz-independent label. */
dayISO: string;
/** Σ of `work` windows this day, ms. Σ over days == workMs (invariant §6.3). */
activeMs: number;
/** Σ of `agent_only` windows this day, ms (drawn, NOT in activeMs). */
agentMs: number;
/** both classes, clipped to the day, sorted by start (for drawing). */
windows: DayWindow[];
}
@@ -1,33 +1,56 @@
import { Space } from '@docmost/db/types/entity.types';
export class SearchResponseDto {
// #529 A7 — the single per-hit SUPERSET returned by the unified search engine.
// The web-UI reads id/highlight/icon/space/title/…; the MCP agent maps id→pageId
// and reads snippet/score/path. `rank`/`highlight` are null for substring-only
// hits (the web already falls back). Nothing the legacy web response carried is
// dropped.
export class SearchResultDto {
id: string;
title: string;
// Alias of `id` for the MCP layer (it addresses pages by pageId).
pageId: string;
slugId: string;
icon: string;
parentPageId: string;
title: string;
space?: Partial<Space>;
creatorId: string;
rank: number;
highlight: string;
createdAt: Date;
updatedAt: Date;
space: Partial<Space>;
// ts_rank_cd of the FTS branch; null for substring-only hits.
rank: number | null;
// ts_headline marked HTML; null for substring-only hits.
highlight: string | null;
// Plain windowed snippet around the match (empty for titleOnly).
snippet: string;
// Ancestor titles root → direct parent ([] for a root page).
path: string[];
// Per-response ordering proxy (falls back to rank).
score: number;
// Which fields matched: 'title' and/or 'text'.
matchedFields: string[];
// Which parsed positive/required terms this hit matched.
matchedTerms: string[];
}
// Response shape for the opt-in agent-lookup mode (#443, `substring: true`).
// Additive to the FTS response: carries the location (`path`), a windowed
// `snippet` around the first match and a per-response sort `score`. The MCP
// layer maps `id → pageId`; `slugId` is never exposed.
export class SearchLookupResponseDto {
id: string;
slugId: string;
title: string;
parentPageId: string | null;
// Ancestor titles from the space root down to the direct parent; [] for a
// root page.
path: string[];
// ~300–500 chars around the first match (or a leading text window / extended
// ts_headline fallback).
snippet: string;
// 0..1 float, meaningful ONLY for sorting within one response.
score: number;
// The paginated envelope (A5). `total` is the EXACT permission-filtered count of
// pages matching the positive lexical query (fail-closed). `hasMore` is true when
// more results exist WITHIN the fusion window; `truncatedAtCap` signals the match
// set exceeded CANDIDATE_CAP and the tail is unreachable by pagination.
export class SearchResponseDto {
items: SearchResultDto[];
total: number;
hasMore: boolean;
truncatedAtCap: boolean;
offset: number;
query: {
raw: string;
parsed: {
positive: string[];
required: string[];
excluded: string[];
reason?: string;
};
mode: 'or' | 'and';
match: string;
};
}
+22 -4
View File
@@ -1,16 +1,30 @@
import {
IsBoolean,
IsIn,
IsNotEmpty,
IsNumber,
IsOptional,
IsString,
MaxLength,
} from 'class-validator';
export class SearchDTO {
// Defense-in-depth cap on the raw query length. The real stack-depth bound is
// the parser's MAX_PARSED_TERMS term cap (see search-query-parser.ts); this
// just rejects absurd payloads early. 10k chars still comfortably holds any
// legitimate query.
@IsNotEmpty()
@IsString()
@MaxLength(10000)
query: string;
// #529 A3 — match mode. `auto` (default) routes identifier-like terms
// (10.31.41, esp32, WB-MGE-30D86B) to the substring/trigram branch and words
// to full-text; `word`/`prefix`/`substring` are explicit overrides.
@IsOptional()
@IsIn(['auto', 'word', 'prefix', 'substring'])
match?: 'auto' | 'word' | 'prefix' | 'substring';
@IsOptional()
@IsString()
spaceId: string;
@@ -33,15 +47,19 @@ export class SearchDTO {
// --- Opt-in agent-lookup mode (#443). ------------------------------------
// These fields are ADDITIVE and default-off: a web client that sends none of
// them gets byte-identical FTS behaviour and result shape. They are only read
// by the substring/path/snippet code path in SearchService.searchPage.
// them gets byte-identical FTS behaviour and result shape. In the unified #529
// engine, `parentPageId` and `titleOnly` are read by SearchService.searchPage
// (subtree scoping and title-only matching, respectively). `substring` is NOT
// read by the native driver — it is accepted-but-ignored, kept only for
// back-compat with the upstream lookup request shape.
//
// NOTE (standalone stdio vs stock upstream): stock upstream validates this DTO
// with `whitelist: true`, so an older server silently strips these unknown
// fields and the request degrades gracefully to the plain FTS behaviour.
// Enables the hybrid substring branch (title + text_content LIKE) merged with
// the existing FTS branch, plus tiered ranking, path and windowed snippet.
// Accepted-but-ignored by the #529 native driver (kept for upstream lookup
// back-compat). The unified engine ALWAYS runs the hybrid FTS + substring/
// trigram branches with tiered ranking, so this flag no longer toggles anything.
@IsOptional()
@IsBoolean()
substring?: boolean;
@@ -0,0 +1,168 @@
import {
parseSearchQuery,
hasPositiveRecall,
MAX_PARSED_TERMS,
} from './search-query-parser';
describe('parseSearchQuery — tokenization & operators (A2)', () => {
it('splits on whitespace into positive terms (OR recall)', () => {
const p = parseSearchQuery('стамбул роснефть');
expect(p.positive.map((t) => t.text)).toEqual(['стамбул', 'роснефть']);
expect(p.required).toEqual([]);
expect(p.excluded).toEqual([]);
expect(p.mode).toBe('or');
});
it('treats +term as required and -term as excluded', () => {
const p = parseSearchQuery('+кофейня -архив');
expect(p.required.map((t) => t.text)).toEqual(['кофейня']);
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive).toEqual([]);
});
it('keeps a leading-operator-free hyphen/dot/colon token as ONE literal term', () => {
// WB-MGE-30D86B, 10.0.12.5, a:b — internal -,.,: are literal, one term each.
expect(parseSearchQuery('WB-MGE-30D86B').positive[0].text).toBe(
'WB-MGE-30D86B',
);
expect(parseSearchQuery('10.0.12.5').positive[0].text).toBe('10.0.12.5');
expect(parseSearchQuery('host:8080').positive[0].text).toBe('host:8080');
});
it('only a LEADING +/- is an operator; -архив excludes архив', () => {
const p = parseSearchQuery('-архив');
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive).toEqual([]);
});
it('drops a bare "-" / "+" and an all-operator remainder', () => {
const p = parseSearchQuery('- + foo -- ++');
expect(p.positive.map((t) => t.text)).toEqual(['foo']);
expect(p.required).toEqual([]);
expect(p.excluded).toEqual([]);
});
it('parses a quoted phrase as one adjacency term', () => {
const p = parseSearchQuery('"воздушный шар" кофе');
expect(p.positive[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
expect(p.positive[1].text).toBe('кофе');
});
it('applies +/- to a phrase', () => {
const req = parseSearchQuery('+"воздушный шар"');
expect(req.required[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
const exc = parseSearchQuery('-"воздушный шар"');
expect(exc.excluded[0]).toMatchObject({ text: 'воздушный шар', branch: 'phrase' });
});
it('drops an unbalanced quote token', () => {
const p = parseSearchQuery('kafka "unclosed here');
expect(p.positive.map((t) => t.text)).toEqual(['kafka']);
});
it('strips tsquery metacharacters from a bare FTS term (no 500)', () => {
const p = parseSearchQuery('foo|bar');
// `|` is not an identifier signal → FTS branch; the metachar is stripped so
// the term becomes the two words that survive.
expect(p.positive[0].branch === 'fts' || p.positive[0].branch === 'ftsPrefix').toBe(true);
expect(p.positive[0].text).toBe('foo bar');
});
});
describe('parseSearchQuery — match=auto classification (A3)', () => {
it('routes identifier-like terms to the substring branch', () => {
expect(parseSearchQuery('10.31.41').positive[0].branch).toBe('substring');
expect(parseSearchQuery('esp32').positive[0].branch).toBe('substring');
expect(parseSearchQuery('WB-MGE-30D86B').positive[0].branch).toBe('substring');
});
it('routes purely-alphabetic words to the FTS (prefix) branch', () => {
expect(parseSearchQuery('печат').positive[0].branch).toBe('ftsPrefix');
expect(parseSearchQuery('ресторан').positive[0].branch).toBe('ftsPrefix');
});
it('explicit match overrides: word / prefix / substring', () => {
expect(parseSearchQuery('печат', { match: 'word' }).positive[0].branch).toBe('fts');
expect(parseSearchQuery('печат', { match: 'prefix' }).positive[0].branch).toBe(
'ftsPrefix',
);
expect(parseSearchQuery('печат', { match: 'substring' }).positive[0].branch).toBe(
'substring',
);
});
});
describe('parseSearchQuery — reasons & recall', () => {
it('only-negation yields reason only-negation and no positive recall', () => {
const p = parseSearchQuery('-архив');
expect(p.reason).toBe('only-negation');
expect(hasPositiveRecall(p)).toBe(false);
});
it('empty / whitespace / garbage yields reason empty', () => {
expect(parseSearchQuery('').reason).toBe('empty');
expect(parseSearchQuery(' ').reason).toBe('empty');
// A bare operator drops to nothing → empty (no exclusion survived).
expect(parseSearchQuery('+ -').reason).toBe('empty');
});
it('a required term alone IS positive recall (no reason)', () => {
const p = parseSearchQuery('+кофейня');
expect(p.reason).toBeUndefined();
expect(hasPositiveRecall(p)).toBe(true);
});
it('mode flag flows through', () => {
expect(parseSearchQuery('a b', { mode: 'and' }).mode).toBe('and');
expect(parseSearchQuery('a b').mode).toBe('or');
});
});
describe('parseSearchQuery — term cap (stack-depth guard)', () => {
it('caps the total parsed terms at MAX_PARSED_TERMS without throwing', () => {
// A pasted text block: far more words than the cap. The parser must bound the
// SQL tsquery nesting depth (else Postgres blows its stack → HTTP 500).
const words = Array.from({ length: 5000 }, (_, i) => `w${i}`);
let p!: ReturnType<typeof parseSearchQuery>;
expect(() => {
p = parseSearchQuery(words.join(' '));
}).not.toThrow();
const total = p.positive.length + p.required.length + p.excluded.length;
expect(total).toBe(MAX_PARSED_TERMS);
// Stable order: the FIRST cap terms are kept.
expect(p.positive.slice(0, 3).map((t) => t.text)).toEqual(['w0', 'w1', 'w2']);
expect(p.positive[MAX_PARSED_TERMS - 1].text).toBe(`w${MAX_PARSED_TERMS - 1}`);
});
it('counts positive + required + excluded together toward the cap', () => {
// Interleave operators so overflow can fall on any bucket. Total must still
// never exceed the cap, and the first cap terms (in stable order) win.
const tokens: string[] = [];
for (let i = 0; i < 200; i++) {
const op = i % 3 === 0 ? '+' : i % 3 === 1 ? '-' : '';
tokens.push(`${op}t${i}`);
}
const p = parseSearchQuery(tokens.join(' '));
const total = p.positive.length + p.required.length + p.excluded.length;
expect(total).toBe(MAX_PARSED_TERMS);
// t0 (+, required) and t1 (-, excluded) and t2 (bare, positive) are all within
// the first cap tokens → each bucket got its leading terms.
expect(p.required[0].text).toBe('t0');
expect(p.excluded[0].text).toBe('t1');
expect(p.positive[0].text).toBe('t2');
});
it('leaves a normal (<= cap) query completely unchanged', () => {
const p = parseSearchQuery('+кофейня -архив "воздушный шар" ресторан 10.31.41');
expect(p.required.map((t) => t.text)).toEqual(['кофейня']);
expect(p.excluded.map((t) => t.text)).toEqual(['архив']);
expect(p.positive.map((t) => t.text)).toEqual([
'воздушный шар',
'ресторан',
'10.31.41',
]);
// Phrase / substring branch handling survives under the cap.
expect(p.positive[0].branch).toBe('phrase');
expect(p.positive[2].branch).toBe('substring');
});
});
@@ -0,0 +1,242 @@
// #529 Phase A — server-side query parser (the SINGLE source of query semantics).
//
// Clients (web-UI, MCP agent, public share) send a RAW query string plus flags
// (`match`, `mode`); ALL query interpretation happens HERE, so every consumer
// gets identical operator/phrase/morphology behaviour. The parser is PURE (no
// SQL, no DB) so it is exhaustively unit-testable; SearchService turns the parsed
// AST into a parameterized tsquery/predicate tree (never string-concatenated SQL).
//
// Grammar (A2):
// - Whitespace splits tokens, but a double-quoted run is ONE token ("a b" is a
// phrase). An unbalanced quote is dropped.
// - A token is an OPERATOR token only when it STARTS with `+` or `-`, the
// remainder is non-empty, and the remainder is not itself only operators.
// `+`/`-` inside a token (`WB-MGE-30D86B`, `10.0.12.5`, `a:b`) is a LITERAL —
// the token is a single term. A bare `-`/`+` is dropped.
// - `"phrase"` → phrase term; `+"phrase"`/`-"phrase"` apply the operator to it.
// - Positive terms (bare + phrase, no operator) form the OR recall set.
// `+term` is a REQUIRED predicate, `-term` an EXCLUDED predicate (A2): both
// are applied in SQL WHERE against the whole candidate set, not folded into
// the positive tsquery.
// - Only-negation (no positive term) short-circuits to an empty result with
// reason `only-negation` (never runs a costly NOT-scan).
export type SearchMatchMode = 'auto' | 'word' | 'prefix' | 'substring';
export type SearchBooleanMode = 'or' | 'and';
// Hard cap on the total number of parsed terms (positive + required + excluded).
// SearchService folds each FTS term into a LEFT-NESTED SQL tsquery expression
// `(((t1)||(t2))||(t3))…`, embedded several times per query — so nesting depth
// grows with the term count. A pasted text block (thousands of words) would nest
// deep enough to blow Postgres' `stack depth limit` → an ERROR → HTTP 500 for the
// caller. Capping in the parser bounds that depth for EVERY consumer (web / MCP /
// share), since they all route through here. 64 comfortably covers any real query
// while keeping the SQL nesting shallow. Overflow terms (beyond the first 64, in
// stable order) are dropped rather than throwing.
export const MAX_PARSED_TERMS = 64;
// How a single term is matched against the index.
// - 'fts' : full-text lexeme, exact (no trailing prefix).
// - 'ftsPrefix' : full-text lexeme with a `:*` prefix match.
// - 'phrase' : an adjacency phrase (phraseto_tsquery).
// - 'substring' : a literal LOWER(f_unaccent(col)) LIKE '%needle%' branch
// (identifiers the tokenizer mangles: IPs, hostnames, IDs).
export type SearchTermBranch = 'fts' | 'ftsPrefix' | 'phrase' | 'substring';
export interface ParsedTerm {
// The user-visible term text, operator stripped, quotes removed. This is what
// `matchedTerms` echoes back per hit.
text: string;
branch: SearchTermBranch;
}
export interface ParsedQuery {
raw: string;
// OR-recall set (bare + phrase terms with no operator).
positive: ParsedTerm[];
// AND predicates (`+term`) — the candidate MUST match each of these.
required: ParsedTerm[];
// NOT predicates (`-term`) — the candidate must match NONE of these.
excluded: ParsedTerm[];
mode: SearchBooleanMode;
// Set only when the query yields no positive recall: 'empty' (nothing usable)
// or 'only-negation' (there were exclusions but no positive term).
reason?: 'empty' | 'only-negation';
}
interface RawToken {
op: '' | '+' | '-';
kind: 'word' | 'phrase';
text: string;
}
// tsquery metacharacters that must never reach to_tsquery from a bare term — they
// are what turned adversarial input into a 500 before (#139). Stripped for the FTS
// branch; the substring branch keeps them (they are literal there).
const TSQUERY_META = /[:&|!()*<>\\]+/g;
// A term is "identifier-like" when it carries a digit or one of . _ : / - AND is
// not purely alphabetic (letters only). Such tokens (10.31.41, esp32,
// WB-MGE-30D86B) are mangled by the FTS tokenizer, so `match: auto` routes them
// to the substring branch. A purely-alphabetic word (печат, ресторан) stays FTS.
const IDENTIFIER_SIGNAL = /[0-9._:/\\-]/;
const PURELY_ALPHA = /^\p{L}+$/u;
function isIdentifierLike(text: string): boolean {
return IDENTIFIER_SIGNAL.test(text) && !PURELY_ALPHA.test(text);
}
// Clean a bare term for the FTS branch: NFC-normalize, drop tsquery metacharacters
// and collapse whitespace. Returns '' when nothing usable remains.
export function cleanFtsLexeme(raw: string): string {
return (raw ?? '')
.normalize('NFC')
.replace(TSQUERY_META, ' ')
.replace(/\s+/g, ' ')
.trim();
}
// Split a raw query into tokens, honouring double quotes and a single leading
// +/- operator. Unbalanced quotes and bare operators are dropped.
function tokenize(raw: string): RawToken[] {
const tokens: RawToken[] = [];
const s = raw ?? '';
let i = 0;
const n = s.length;
const isSpace = (c: string) => /\s/.test(c);
while (i < n) {
// Skip leading whitespace.
while (i < n && isSpace(s[i])) i++;
if (i >= n) break;
let op: '' | '+' | '-' = '';
// A single leading +/- is a tentative operator. Only ONE leading operator is
// consumed; a second (`--x`) leaves `-x` as the remainder (a literal dash).
if (s[i] === '+' || s[i] === '-') {
op = s[i] as '+' | '-';
i++;
}
if (i < n && s[i] === '"') {
// Quoted phrase: read until the closing quote.
const close = s.indexOf('"', i + 1);
if (close === -1) {
// Unbalanced quote → drop this token and everything the open quote would
// have consumed (the rest of the string).
break;
}
const phrase = s.slice(i + 1, close);
i = close + 1;
if (phrase.trim().length > 0) {
tokens.push({ op, kind: 'phrase', text: phrase.trim() });
}
continue;
}
// Bare word: read until the next whitespace.
let j = i;
while (j < n && !isSpace(s[j])) j++;
const word = s.slice(i, j);
i = j;
// A bare operator (`-`/`+` with no remainder) or an all-operator remainder is
// dropped.
if (word.length === 0) continue;
if (op && /^[+-]+$/.test(word)) continue;
tokens.push({ op, kind: 'word', text: word });
}
return tokens;
}
// Resolve the match branch for a single term given the global match mode.
function branchForTerm(
text: string,
kind: 'word' | 'phrase',
mode: SearchMatchMode,
): SearchTermBranch {
if (kind === 'phrase') return 'phrase';
switch (mode) {
case 'word':
return 'fts';
case 'prefix':
return 'ftsPrefix';
case 'substring':
return 'substring';
case 'auto':
default:
// Identifiers the tokenizer mangles go to substring; words get a prefix
// FTS match (so `печат` still finds `печатать`, but `печат` no longer drags
// in `впечатления` because the russian stemmer anchors the stem).
return isIdentifierLike(text) ? 'substring' : 'ftsPrefix';
}
}
/**
* Parse a raw user query + flags into a structured, SQL-agnostic AST.
* Pure and total: never throws, always returns a ParsedQuery.
*/
export function parseSearchQuery(
raw: string,
opts: { match?: SearchMatchMode; mode?: SearchBooleanMode } = {},
): ParsedQuery {
const match: SearchMatchMode = opts.match ?? 'auto';
const mode: SearchBooleanMode = opts.mode ?? 'or';
const positive: ParsedTerm[] = [];
const required: ParsedTerm[] = [];
const excluded: ParsedTerm[] = [];
for (const tok of tokenize(raw)) {
// For an FTS branch, the token must survive metacharacter cleaning; for the
// substring/phrase branch the literal text is used. A term that cleans to
// nothing AND is not usable as a substring is dropped.
const branch = branchForTerm(tok.text, tok.kind, match);
let usableText: string;
if (branch === 'fts' || branch === 'ftsPrefix') {
usableText = cleanFtsLexeme(tok.text);
} else {
// phrase / substring keep the literal (trimmed) text.
usableText = tok.text.trim();
}
if (!usableText) continue;
// Stack-depth guard: stop after MAX_PARSED_TERMS surviving terms (positive +
// required + excluded, combined) so the SQL tsquery nesting stays shallow.
// The first 64 terms are kept in stable order; the rest are dropped.
if (positive.length + required.length + excluded.length >= MAX_PARSED_TERMS) {
break;
}
const term: ParsedTerm = { text: usableText, branch };
if (tok.op === '+') required.push(term);
else if (tok.op === '-') excluded.push(term);
else positive.push(term);
}
const parsed: ParsedQuery = { raw: raw ?? '', positive, required, excluded, mode };
if (positive.length === 0) {
// Required terms with no positive recall still form a valid positive set (the
// required predicates ARE the recall). Only when there is neither a positive
// nor a required term is the query empty / only-negation.
if (required.length === 0) {
parsed.reason = excluded.length > 0 ? 'only-negation' : 'empty';
}
}
return parsed;
}
/**
* Does this parsed query have any positive recall to run? False means we must
* short-circuit to an empty result (with `reason`), never a costly NOT-only scan.
*/
export function hasPositiveRecall(parsed: ParsedQuery): boolean {
return parsed.positive.length > 0 || parsed.required.length > 0;
}
@@ -1,19 +1,14 @@
import {
computeLookupScore,
escapeLikePattern,
SearchLookupTier,
} from './search.service';
import { escapeLikePattern } from './search.service';
/**
* Pure-function coverage for the #443 agent-lookup helpers:
* - escapeLikePattern: LIKE-metacharacter escaping so `%`/`_`/`\` are literals
* (the acceptance-table requirement that a query of `%` or `_` does NOT match
* everything);
* - computeLookupScore: the tiered 0..1 ranking score, where a stronger tier
* always outranks a weaker one regardless of the in-tier secondary signal.
* Pure-function coverage for `escapeLikePattern` LIKE-metacharacter escaping so
* `%`/`_`/`\` are matched literally (the acceptance requirement that a query of
* `%` or `_` does NOT match everything, #529 acceptance #10). The substring
* branch's DB behaviour is covered by the integration spec.
*
* The DB-touching branch (substring UNION FTS, path CTE, snippet window) is
* covered by the integration spec against the real schema.
* NOTE (#529): the old tiered `computeLookupScore` was replaced by RRF rank
* fusion in the unified engine, so its unit coverage moved to the integration
* ordering tests; only the escaping helper remains a pure unit here.
*/
describe('escapeLikePattern', () => {
it('escapes the LIKE metacharacters % _ and \\', () => {
@@ -43,53 +38,3 @@ describe('escapeLikePattern', () => {
expect(escapeLikePattern(null as any)).toBe('');
});
});
describe('computeLookupScore', () => {
it('keeps every score within (0, 1]', () => {
for (const tier of [
SearchLookupTier.TITLE_EXACT,
SearchLookupTier.TITLE_SUBSTRING,
SearchLookupTier.TEXT,
]) {
for (const secondary of [0, 0.001, 1, 100, 1e6]) {
const s = computeLookupScore({ tier, secondary });
expect(s).toBeGreaterThan(0);
expect(s).toBeLessThanOrEqual(1);
}
}
});
it('a stronger tier ALWAYS outranks a weaker tier, whatever the secondary', () => {
// Weak tier with a huge secondary must still lose to a strong tier with a
// tiny secondary — tiers dominate.
const strongLowSecondary = computeLookupScore({
tier: SearchLookupTier.TITLE_EXACT,
secondary: 0,
});
const weakHighSecondary = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 1e9,
});
expect(strongLowSecondary).toBeGreaterThan(weakHighSecondary);
});
it('within a tier a larger secondary sorts higher', () => {
const lo = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 0.1,
});
const hi = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 5,
});
expect(hi).toBeGreaterThan(lo);
});
it('treats a negative/absent secondary as 0', () => {
const zero = computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: 0 });
expect(computeLookupScore({ tier: SearchLookupTier.TEXT })).toBe(zero);
expect(
computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: -5 }),
).toBe(zero);
});
});
@@ -1,74 +1,45 @@
import { SearchService } from './search.service';
/**
* Coverage for SearchService.searchPage query-mode selection (search.service.ts
* @25). searchPage chooses HOW the result set is scoped by explicit space, by
* the authenticated user's member spaces, or by a share and must return an
* empty set (without leaking data) for every disallowed combination.
* Unit coverage for SearchService.searchPage SCOPE-SECURITY early returns the
* branches that must yield an empty result WITHOUT ever touching the DB, so they
* can leak nothing. The happy-path scope SQL (explicit space / member spaces /
* share id set) is covered against the real schema in the integration spec.
*
* The kysely query builder is mocked with the same chainable pattern as the
* existing search.service.spec.ts: every builder method returns the same builder
* and `.execute()` resolves the supplied rows. Each `.where(...)` call is
* recorded so we can assert exactly which scope clause was applied that is the
* mutation-resistant signal that distinguishes one query mode from another.
*
* These specs catch cross-space / cross-workspace search leakage and
* share-scope bypass (data exposure).
* Every case here returns BEFORE the raw-SQL candidate query runs, so a bare `db`
* stub (never called) is enough a call to it would itself be a failure signal.
*/
describe('SearchService.searchPage — query-mode selection', () => {
// Build a chainable selectFrom('pages') builder that records its calls. The
// builder is returned from `db.selectFrom` and is the single object every
// chained call mutates/returns, mirroring the existing spec's pattern.
function makeBuilder(rows: Array<{ id: string; highlight?: string }>) {
const builder: any = {};
builder.select = jest.fn(() => builder);
builder.where = jest.fn(() => builder);
builder.$if = jest.fn(() => builder);
builder.orderBy = jest.fn(() => builder);
builder.limit = jest.fn(() => builder);
builder.offset = jest.fn(() => builder);
builder.execute = jest.fn(async () => rows);
return builder;
}
describe('SearchService.searchPage — scope-security early returns', () => {
function makeService(opts?: {
rows?: Array<{ id: string; highlight?: string }>;
share?: any;
isRestricted?: boolean;
descendants?: Array<{ id: string }>;
memberSpaceIds?: string[];
}) {
const builder = makeBuilder(opts?.rows ?? []);
const db: any = {
selectFrom: jest.fn(() => builder),
};
// `getUserSpaceIdsQuery` returns a sub-query object that searchPage passes
// straight into `.where('spaceId', 'in', <subquery>)`. A sentinel is enough
// to assert the user-scoped branch was taken.
const userSpaceIdsQuery = { __userSpaceIdsQuery: true };
// A db that THROWS if touched — these branches must not reach SQL.
const db: any = new Proxy(
{},
{
get() {
throw new Error('db must not be touched on an empty-scope branch');
},
},
);
const pageRepo = {
// `.select((eb) => this.pageRepo.withSpace(eb))` — value ignored by stub.
withSpace: jest.fn(() => ({ __withSpace: true })),
getPageAndDescendantsExcludingRestricted: jest
.fn()
.mockResolvedValue(opts?.descendants ?? []),
getPageAndDescendantsExcludingRestricted: jest.fn(),
getPageAndDescendants: jest.fn(),
};
const shareRepo = {
findById: jest.fn().mockResolvedValue(opts?.share ?? null),
};
const spaceMemberRepo = {
getUserSpaceIdsQuery: jest.fn(() => userSpaceIdsQuery),
getUserSpaceIds: jest.fn().mockResolvedValue(opts?.memberSpaceIds ?? []),
};
const pagePermissionRepo = {
hasRestrictedAncestor: jest
.fn()
.mockResolvedValue(opts?.isRestricted ?? false),
// Let everything through page-level permission filtering by default.
filterAccessiblePageIds: jest
.fn()
.mockImplementation(async ({ pageIds }: { pageIds: string[] }) => pageIds),
filterAccessiblePageIds: jest.fn(),
};
const service = new SearchService(
@@ -78,145 +49,81 @@ describe('SearchService.searchPage — query-mode selection', () => {
spaceMemberRepo as any,
pagePermissionRepo as any,
);
return {
service,
db,
builder,
pageRepo,
shareRepo,
spaceMemberRepo,
pagePermissionRepo,
userSpaceIdsQuery,
};
return { service, pageRepo, shareRepo, spaceMemberRepo, pagePermissionRepo };
}
const whereCallFor = (builder: any, column: any) =>
builder.where.mock.calls.find((c: any[]) => c[0] === column);
it('returns {items:[]} for a blank query WITHOUT touching the DB', async () => {
const { service, db } = makeService();
it('returns total:0 for a blank query WITHOUT touching the DB or any repo', async () => {
const { service, shareRepo, spaceMemberRepo } = makeService();
const result = await service.searchPage(
{ query: '' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(result).toEqual({ items: [] });
// Blank query is rejected before any query builder is constructed.
expect(db.selectFrom).not.toHaveBeenCalled();
expect(result.items).toEqual([]);
expect(result.total).toBe(0);
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(spaceMemberRepo.getUserSpaceIds).not.toHaveBeenCalled();
});
it('scopes to the explicit spaceId branch', async () => {
const { service, builder, db, spaceMemberRepo, shareRepo } = makeService({
rows: [{ id: 'p-1' }],
});
it('only-negation short-circuits with reason "only-negation", never scanning', async () => {
const { service, spaceMemberRepo } = makeService();
const result = await service.searchPage(
{ query: 'plan', spaceId: 'space-42' } as any,
{ query: '-архив' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(db.selectFrom).toHaveBeenCalledWith('pages');
// The explicit-space branch adds exactly `.where('spaceId', '=', 'space-42')`.
expect(whereCallFor(builder, 'spaceId')).toEqual([
'spaceId',
'=',
'space-42',
]);
// It must NOT fall through to the user-member-spaces or share branch.
expect(spaceMemberRepo.getUserSpaceIdsQuery).not.toHaveBeenCalled();
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(result.items.map((i: any) => i.id)).toEqual(['p-1']);
expect(result.total).toBe(0);
expect(result.query.parsed.reason).toBe('only-negation');
// Never resolves scope (returns before) — no expensive NOT-only scan.
expect(spaceMemberRepo.getUserSpaceIds).not.toHaveBeenCalled();
});
it('scopes an authenticated user WITHOUT spaceId to their member spaces', async () => {
const { service, builder, spaceMemberRepo, userSpaceIdsQuery, shareRepo } =
makeService({ rows: [{ id: 'p-9' }] });
await service.searchPage(
{ query: 'plan' } as any,
{ userId: 'user-7', workspaceId: 'ws-1' },
);
// The user-scoped branch resolves the member-spaces sub-query for that user
// and restricts both spaceId (to that sub-query) and workspaceId.
expect(spaceMemberRepo.getUserSpaceIdsQuery).toHaveBeenCalledWith('user-7');
expect(whereCallFor(builder, 'spaceId')).toEqual([
'spaceId',
'in',
userSpaceIdsQuery,
]);
expect(whereCallFor(builder, 'workspaceId')).toEqual([
'workspaceId',
'=',
'ws-1',
]);
// Authenticated user path must not consult shares.
expect(shareRepo.findById).not.toHaveBeenCalled();
});
it('returns {items:[]} when the share belongs to a DIFFERENT workspace', async () => {
const { service, builder, shareRepo, pagePermissionRepo } = makeService({
share: {
id: 'share-1',
pageId: 'page-1',
workspaceId: 'OTHER-ws',
includeSubPages: false,
},
it('returns empty when the share belongs to a DIFFERENT workspace (no leak)', async () => {
const { service, shareRepo, pagePermissionRepo } = makeService({
share: { id: 's1', pageId: 'p1', workspaceId: 'OTHER', includeSubPages: false },
});
const result = await service.searchPage(
{ query: 'plan', shareId: 'share-1' } as any,
{ query: 'plan', shareId: 's1' } as any,
{ workspaceId: 'ws-1' },
);
expect(shareRepo.findById).toHaveBeenCalledWith('share-1');
expect(result).toEqual({ items: [] });
// Workspace mismatch short-circuits before any restricted-ancestor / id
// scoping or DB execution: no leak across workspaces.
expect(shareRepo.findById).toHaveBeenCalledWith('s1');
expect(result.items).toEqual([]);
// Workspace mismatch short-circuits before restricted-ancestor / enumeration.
expect(pagePermissionRepo.hasRestrictedAncestor).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('returns {items:[]} when the shared page has a restricted ancestor', async () => {
const { service, builder, pagePermissionRepo, pageRepo } = makeService({
share: {
id: 'share-1',
pageId: 'page-1',
workspaceId: 'ws-1',
includeSubPages: true,
},
it('returns empty when the shared page has a restricted ancestor', async () => {
const { service, pagePermissionRepo, pageRepo } = makeService({
share: { id: 's1', pageId: 'p1', workspaceId: 'ws-1', includeSubPages: true },
isRestricted: true,
});
const result = await service.searchPage(
{ query: 'plan', shareId: 'share-1' } as any,
{ query: 'plan', shareId: 's1' } as any,
{ workspaceId: 'ws-1' },
);
expect(pagePermissionRepo.hasRestrictedAncestor).toHaveBeenCalledWith(
'page-1',
);
expect(result).toEqual({ items: [] });
// Restricted ancestor must block before page enumeration and DB execution.
expect(pagePermissionRepo.hasRestrictedAncestor).toHaveBeenCalledWith('p1');
expect(result.items).toEqual([]);
expect(
pageRepo.getPageAndDescendantsExcludingRestricted,
).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('returns {items:[]} with no userId, no spaceId and no shareId', async () => {
const { service, builder, shareRepo } = makeService();
it('returns empty with no userId, no spaceId and no shareId', async () => {
const { service, shareRepo } = makeService();
const result = await service.searchPage(
{ query: 'plan' } as any,
{ workspaceId: 'ws-1' },
);
expect(result).toEqual({ items: [] });
// The catch-all else returns empty without scoping/executing or hitting shares.
expect(result.items).toEqual([]);
expect(shareRepo.findById).not.toHaveBeenCalled();
expect(builder.execute).not.toHaveBeenCalled();
});
it('an authenticated user with NO member spaces gets an empty result', async () => {
const { service, spaceMemberRepo } = makeService({ memberSpaceIds: [] });
const result = await service.searchPage(
{ query: 'plan' } as any,
{ userId: 'user-1', workspaceId: 'ws-1' },
);
expect(spaceMemberRepo.getUserSpaceIds).toHaveBeenCalledWith('user-1');
expect(result.items).toEqual([]);
expect(result.total).toBe(0);
});
});
@@ -1,4 +1,4 @@
import { SearchService, buildTsQuery } from './search.service';
import { SearchService } from './search.service';
describe('SearchService', () => {
it('should be defined', () => {
@@ -99,59 +99,3 @@ describe('SearchService.searchSuggestions — onlyTemplates filter', () => {
expect(isTemplateWhereCall(pageBuilder)).toBeUndefined();
});
});
// Unit tests for `buildTsQuery` (extracted from search.service.ts). It turns a raw
// user query into a prefix tsquery string fed to `to_tsquery('english', ...)`.
//
// REAL BUG (Gitea #139, item 10): the previous inline `tsquery(query.trim() + '*')`
// let to_tsquery operator characters through, so adversarial inputs could produce a
// fragment that to_tsquery rejects -> 500. The extraction sanitizes the input
// (strip everything but letters/numbers/whitespace) so these inputs degrade to a
// safe, neutral query with NO throw, while normal queries keep working.
describe('buildTsQuery', () => {
it('builds a prefix query for a normal single word', () => {
expect(buildTsQuery('hello')).toBe('hello:*');
});
it('joins multiple words with AND and a trailing prefix match', () => {
expect(buildTsQuery('foo bar')).toBe('foo&bar:*');
});
it('preserves accented and non-Latin words', () => {
expect(buildTsQuery('héllo café')).toBe('héllo&café:*');
expect(buildTsQuery('日本語')).toBe('日本語:*');
});
it('neutralizes to_tsquery operator inputs without throwing', () => {
// Each of these previously risked an invalid to_tsquery -> 500. They must now
// produce a safe (here empty) query and never throw.
for (const input of ['&', '!', '*', '<->', '\\']) {
expect(() => buildTsQuery(input)).not.toThrow();
expect(buildTsQuery(input)).toBe('');
}
});
it('handles stopword-only input safely', () => {
// pg-tsquery still tokenizes stopwords; to_tsquery reduces them to nothing.
// The important contract is: no throw, and a deterministic string.
expect(() => buildTsQuery('the a of')).not.toThrow();
expect(buildTsQuery('the a of')).toBe('the&a&of:*');
});
it('returns empty string for empty / whitespace-only / null-ish input', () => {
expect(buildTsQuery('')).toBe('');
expect(buildTsQuery(' ')).toBe('');
expect(buildTsQuery(undefined as unknown as string)).toBe('');
});
it('handles a very long input without throwing', () => {
const long = 'a'.repeat(10000);
expect(() => buildTsQuery(long)).not.toThrow();
expect(buildTsQuery(long)).toBe(`${long}:*`);
});
it('strips punctuation embedded in otherwise valid words', () => {
expect(buildTsQuery('c++ code')).toBe('c&code:*');
expect(buildTsQuery('a-b-c')).toBe('a&b&c:*');
});
});
File diff suppressed because it is too large Load Diff
@@ -32,6 +32,7 @@ import { TemplateRepo } from '@docmost/db/repos/template/template.repo';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo';
import { AiProviderCredentialsRepo } from '@docmost/db/repos/ai-chat/ai-provider-credentials.repo';
import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo';
@@ -125,6 +126,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
AiChatRepo,
AiChatMessageRepo,
AiChatRunRepo,
AiChatRunStepRepo,
AiChatPageSnapshotRepo,
AiProviderCredentialsRepo,
AiMcpServerRepo,
@@ -161,6 +163,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
AiChatRepo,
AiChatMessageRepo,
AiChatRunRepo,
AiChatRunStepRepo,
AiChatPageSnapshotRepo,
AiProviderCredentialsRepo,
AiMcpServerRepo,
@@ -0,0 +1,242 @@
import { type Kysely, sql } from 'kysely';
/**
* #529 Phase A1 the `ru_en` text-search configuration + the config swap.
*
* WHY: the search stack was pinned to the `english` FTS config, which stems only
* Latin words. On a Russian-language wiki that is a morphology black hole:
* «ресторанов москвы» never matched a page titled «ресторан в москве». `ru_en`
* layers the russian_stem over the Cyrillic token classes and english_stem over
* the ascii ones, so BOTH languages get proper morphology from one config.
*
* CREATE TEXT SEARCH CONFIGURATION ru_en (COPY = simple);
* ALTER ... asciiword/asciihword/hword_asciipart WITH english_stem;
* ALTER ... word/hword/hword_part WITH russian_stem;
*
* `to_tsvector('ru_en', …)` with the LITERAL config name is IMMUTABLE, so it is
* valid inside a trigger, a generated column and an index expression.
*
* THE INVARIANT (acceptance #13): the config of the STORED column and the config
* of the QUERY must change together. This migration flips BOTH stored sides
* (pages.tsv via its trigger + a reindex; page_embeddings.fts, the RAG lexical
* leg, via its generated expression); the matching QUERY-side flips
* (search.service.ts and page-embedding.repo.ts `hybridSearch`) ship in the SAME
* commit. Trigram indexes are LOWER(f_unaccent(...)) and do NOT depend on the FTS
* config, so they are untouched.
*
* REINDEX / LOCK MODEL (deploy-critical). Kysely runs EACH migration in its OWN
* transaction (see sibling 20260706T120000 "Kysely runs each migration in a
* transaction"; migrate.ts / migration.service.ts set no `disableTransactions`,
* and PostgresJSDialect has transactional DDL). A single migration file therefore
* cannot commit between batches, so the issue's "procedural batch job OUTSIDE the
* transaction + dual-config read window + migration_complete gate" is not
* expressible in-migration here. That machinery bridges a reindex spread over
* MANY committed batches, during which some rows are still `english` while others
* are already `ru_en`. Our pages.tsv reindex is a SINGLE `UPDATE pages SET tsv`,
* ATOMIC within THIS migration's own transaction: at COMMIT every row is `ru_en`
* at once, so no morphology-desync window exists and no dual-config read path is
* required the query config flips to `ru_en` in the very same release. This is
* the deliberate, correct adaptation to this framework (see the PR notes).
*
* - pages.tsv: swapping the trigger is a cheap catalog change (no table lock),
* and the reindex is a single `UPDATE pages SET tsv = <ru_en expr>` a
* ROW-level-lock (RowExclusiveLock) backfill, NOT an ACCESS EXCLUSIVE rewrite
* (mirrors the existing space_id backfill in 20250725T052004). On a LARGE
* tenant this still writes every row + its WAL and leaves dead tuples, so it
* can take MINUTES and blocks the startup migrator for that time but it
* never blocks concurrent reads. It stays inline unconditionally.
*
* - page_embeddings.fts is a GENERATED STORED column; Postgres cannot change a
* generated expression without DROP+ADD, which is a full-table ACCESS
* EXCLUSIVE REWRITE of page_embeddings it blocks ALL reads AND writes on
* that table (including the RAG agent) for the rewrite's duration. That inline
* rewrite is appropriate for small/typical tenants (this fork's target) and
* is the DEFAULT.
*
* LARGE TENANTS have two documented escape hatches, either of which makes the
* migration genuinely no-op the rewrite (it is NOT a blind DROP+ADD):
* (a) Set `SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false`. The migration then
* SKIPS the embeddings rewrite entirely and logs a WARNING. The operator
* MUST perform the ru_en fts swap out-of-band; until they do, the RAG
* lexical leg stays on `english` while the query config is `ru_en` a
* documented, operator-owned desync window. (pages.tsv still swaps
* inline the gate is ONLY the embeddings rewrite.)
* (b) Perform the swap out-of-band BEFORE deploy add a plain column
* batched backfill brief-lock swap CREATE INDEX CONCURRENTLY so
* the `fts` column's generated expression already references the TARGET
* config when the migration runs. The migration detects this (it reads
* the column's actual generation expression from pg_catalog) and does a
* TRUE no-op no DROP, no ADD, no rewrite. This is real idempotency,
* not the old (false) "IF-EXISTS guards no-op" claim: `DROP COLUMN IF
* EXISTS` guards against ABSENCE, not presence, so it would have dropped
* and recreated an existing `fts` regardless. The at-target check is the
* only honest no-op path.
*
* Same documented trade-off family as the #443 trgm GIN migration (20260706T120000).
*/
// pages.tsv trigger body for a given FTS config — mirrors the latest form
// (20250729T213756): f_unaccent + a 1MB text cap on text_content, weights A/B.
function pagesTriggerSql(config: 'ru_en' | 'english') {
return sql`
CREATE OR REPLACE FUNCTION pages_tsvector_trigger() RETURNS trigger AS $$
begin
new.tsv :=
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(coalesce(new.title, ''))), 'A') ||
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(substring(coalesce(new.text_content, ''), 1, 1000000))), 'B');
return new;
end;
$$ LANGUAGE plpgsql;
`;
}
async function swapPagesConfig(db: Kysely<any>, config: 'ru_en' | 'english') {
// 1. Point the trigger at the target config (new/edited rows use it going
// forward). CREATE OR REPLACE FUNCTION takes only a brief catalog lock.
await pagesTriggerSql(config).execute(db);
// 2. Reindex existing rows: recompute tsv directly with the target config. A
// plain UPDATE — row locks, no ACCESS EXCLUSIVE. Equivalent to firing the
// trigger but cheaper (no self-update round trip).
await sql`
UPDATE pages
SET tsv =
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(coalesce(title, ''))), 'A') ||
setweight(to_tsvector('${sql.raw(config)}', f_unaccent(substring(coalesce(text_content, ''), 1, 1000000))), 'B')
`.execute(db);
}
// The default in-migration ACCESS EXCLUSIVE rewrite of page_embeddings.fts is
// ON unless the operator explicitly opts out with the env flag. Parsed strictly
// (mirrors CLIENT_TELEMETRY_ENABLED / DEBUG_MODE in common/): only a literal
// (case-insensitive) 'false' disables it; anything else — unset included —
// keeps the default true.
function inlineEmbeddingsRewriteEnabled(): boolean {
return (
(process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE ?? 'true').toLowerCase() !==
'false'
);
}
// Read page_embeddings.fts's ACTUAL generated-column expression from pg_catalog
// (the generation expression is stored as a column default marked generated).
// Returns '' when the column is absent.
async function embeddingsFtsExpr(db: Kysely<any>): Promise<string> {
const r = await sql<{ def: string }>`
SELECT pg_get_expr(d.adbin, d.adrelid) AS def
FROM pg_attrdef d
JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attname = 'fts'
AND a.attrelid = 'page_embeddings'::regclass
AND NOT a.attisdropped
`.execute(db);
return r.rows[0]?.def ?? '';
}
async function swapEmbeddingsFtsConfig(
db: Kysely<any>,
config: 'ru_en' | 'english',
) {
// 1. TRUE no-op path (real out-of-band escape hatch): if the column's current
// generation expression already references the TARGET config, there is
// nothing to do. An operator who pre-swapped the column out-of-band lands
// here and the migration does NOT rewrite the table. ('ru_en' and 'english'
// are disjoint tokens, neither a substring of the other or of the rest of
// the expression, so a plain contains-check is unambiguous.)
const currentExpr = await embeddingsFtsExpr(db);
if (currentExpr.includes(config)) return;
// 2. Env-gated opt-out: large tenants set SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE
// =false to skip the ACCESS EXCLUSIVE rewrite in-migration and own the swap
// out-of-band. Warn loudly so the desync window is not silent.
if (!inlineEmbeddingsRewriteEnabled()) {
// eslint-disable-next-line no-console
console.warn(
`[migration 20260707T130000] SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false: ` +
`SKIPPING the page_embeddings.fts rewrite to '${config}'. The operator MUST ` +
`perform this fts swap out-of-band. Until then the RAG lexical leg stays on ` +
`its current config while the query config is '${config}' (documented, ` +
`operator-owned desync window).`,
);
return;
}
// 3. Default inline path: the generated `fts` expression can only change via
// DROP+ADD (a full-table ACCESS EXCLUSIVE rewrite; see the lock note in the
// header). The GIN index depends on the column, so it is dropped with it and
// recreated.
await sql`DROP INDEX IF EXISTS idx_page_embeddings_fts`.execute(db);
await sql`ALTER TABLE page_embeddings DROP COLUMN IF EXISTS fts`.execute(db);
await sql`
ALTER TABLE page_embeddings
ADD COLUMN fts tsvector
GENERATED ALWAYS AS (to_tsvector('${sql.raw(config)}', f_unaccent(content))) STORED
`.execute(db);
await sql`
CREATE INDEX IF NOT EXISTS idx_page_embeddings_fts
ON page_embeddings USING gin(fts)
`.execute(db);
}
async function ruEnConfigExists(db: Kysely<any>): Promise<boolean> {
const r = await sql<{ n: number }>`
SELECT count(*)::int AS n FROM pg_ts_config WHERE cfgname = 'ru_en'
`.execute(db);
return (r.rows[0]?.n ?? 0) > 0;
}
async function ensureRuEnConfig(db: Kysely<any>): Promise<void> {
// Idempotent by EXISTENCE, not by drop-recreate. The old `DROP ... IF EXISTS;
// CREATE` was safe only on a first run: on a re-run the page_embeddings.fts
// generated column already has a hard dependency on ru_en, so dropping the
// config would fail. Create only when it is genuinely missing.
if (await ruEnConfigExists(db)) return;
await sql`CREATE TEXT SEARCH CONFIGURATION ru_en (COPY = simple)`.execute(db);
// Latin token classes → english_stem.
await sql`
ALTER TEXT SEARCH CONFIGURATION ru_en
ALTER MAPPING FOR asciiword, asciihword, hword_asciipart
WITH english_stem
`.execute(db);
// Cyrillic / non-ascii token classes → russian_stem.
await sql`
ALTER TEXT SEARCH CONFIGURATION ru_en
ALTER MAPPING FOR word, hword, hword_part
WITH russian_stem
`.execute(db);
}
export async function up(db: Kysely<any>): Promise<void> {
await ensureRuEnConfig(db);
// Flip both stored sides to ru_en (query-side flips in the same commit).
// swapEmbeddingsFtsConfig no-ops when fts already references ru_en, so a
// re-run of up() is idempotent and does NOT re-rewrite the embeddings table.
await swapPagesConfig(db, 'ru_en');
await swapEmbeddingsFtsConfig(db, 'ru_en');
}
export async function down(db: Kysely<any>): Promise<void> {
// Reverse ORDER matters: the trigger and the generated column reference the
// `ru_en` config by name, so they must be moved back to `english` BEFORE the
// config can be dropped (a generated column that still depends on `ru_en` would
// block the DROP with a dependency error).
await swapEmbeddingsFtsConfig(db, 'english');
await swapPagesConfig(db, 'english');
// Drop the config ONLY if nothing still references it. When the embeddings
// rewrite was gated off (SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false), the fts
// column can still reference ru_en — dropping the config would then fail with a
// dependency error. Skip + warn so down() stays non-fatal; the operator drops
// ru_en after completing the out-of-band english swap.
if ((await embeddingsFtsExpr(db)).includes('ru_en')) {
// eslint-disable-next-line no-console
console.warn(
`[migration 20260707T130000] down(): page_embeddings.fts still references ` +
`ru_en (inline rewrite was gated off) — leaving the ru_en text-search ` +
`configuration in place. Drop it out-of-band once fts is back on 'english'.`,
);
return;
}
await sql`DROP TEXT SEARCH CONFIGURATION IF EXISTS ru_en`.execute(db);
}
@@ -0,0 +1,70 @@
import { type Kysely, sql } from 'kysely';
/**
* `ai_chat_run_steps` append-only per-step persistence for an assistant turn
* (#492 wave C). Each finished agent step's UI `parts` (its text part + a part
* per tool call, WITH the tool output) is INSERTed as its own lightweight row the
* moment the step ends, instead of REWRITING the whole assistant row's growing
* `metadata.parts` jsonb on every `onStepFinish`.
*
* WHY a separate table + INSERT (not a jsonb `||` append on the message row): a
* Postgres jsonb UPDATE rewrites the ENTIRE TOASTed row version under MVCC, so
* re-persisting a growing `metadata.parts` on every step is O(n²) write volume
* (a 50-step run with ~100 KB tool outputs wrote hundreds of MB of WAL / dead
* tuples per turn, hammering autovacuum). `||` would only shave the network
* payload the WAL/TOAST rewrite harm remains. An INSERT into a per-step table
* writes ONLY that step's bytes, so the per-turn write volume is O(Σ steps).
*
* The full `metadata.parts` on the message row is assembled ONCE at finalize (the
* terminal completed/error/aborted write). Mid-run, a resuming client's seed is
* reconstructed by concatenating these step rows in `step_index` order which
* reproduces exactly what the old per-step full-row rewrite persisted. Records
* written the OLD way (full `metadata.parts` on the row, no step rows) still
* reconstruct from the row unchanged; the two eras are distinguished by whether
* the row already carries non-empty `metadata.parts` (see reconstructRunParts /
* assembleStepParts in ai-chat.service.ts).
*
* ON DELETE CASCADE on `message_id`: the step rows are a derived projection of the
* assistant message; they must vanish with it (or with its workspace).
*/
export async function up(db: Kysely<any>): Promise<void> {
await db.schema
.createTable('ai_chat_run_steps')
.ifNotExists()
.addColumn('id', 'uuid', (col) =>
col.primaryKey().defaultTo(sql`gen_uuid_v7()`),
)
// The assistant message row this step belongs to (the #183 projection). The
// step rows are a derived, per-step slice of that message, so they cascade.
.addColumn('message_id', 'uuid', (col) =>
col.references('ai_chat_messages.id').onDelete('cascade').notNull(),
)
.addColumn('workspace_id', 'uuid', (col) =>
col.references('workspaces.id').onDelete('cascade').notNull(),
)
// 0-based index of the finished step within the turn. Ordering key for
// reconstruction; unique per message (idempotent step re-persist).
.addColumn('step_index', 'integer', (col) => col.notNull())
// The step's UI parts (text part + a `tool-*` part per call, WITH output).
// Concatenated in step order to rebuild the turn's `metadata.parts`.
.addColumn('parts', 'jsonb', (col) => col.notNull())
.addColumn('created_at', 'timestamptz', (col) =>
col.notNull().defaultTo(sql`now()`),
)
.execute();
// Idempotent per-step persist: a retried INSERT of the same (message, step)
// is a no-op (the service uses ON CONFLICT DO NOTHING). This also serves the
// reconstruction read (WHERE message_id ORDER BY step_index).
await db.schema
.createIndex('ai_chat_run_steps_message_step_uidx')
.ifNotExists()
.on('ai_chat_run_steps')
.columns(['message_id', 'step_index'])
.unique()
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema.dropTable('ai_chat_run_steps').ifExists().execute();
}
@@ -0,0 +1,95 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import { AiChatRunStep } from '@docmost/db/types/entity.types';
/**
* Append-only per-step persistence for an assistant turn (#492). Each finished
* agent step's UI `parts` (its text part + a `tool-*` part per call, WITH the
* tool output) is INSERTed as its own lightweight row the moment the step ends
* instead of REWRITING the assistant row's growing `metadata.parts` jsonb on every
* `onStepFinish` (a Postgres jsonb UPDATE rewrites the whole TOASTed row version
* under MVCC, so that was O(n²) WAL/dead-tuple churn per turn).
*
* The full `metadata.parts` on the message row is assembled ONCE at finalize;
* mid-run, a resuming client's seed is rebuilt from these rows in `stepIndex`
* order (see `assembleStepParts` / the reconstruct seam in ai-chat.service.ts).
* Every method is workspace-scoped as defense-in-depth.
*/
@Injectable()
export class AiChatRunStepRepo {
constructor(@InjectKysely() private readonly db: KyselyDB) {}
/**
* Append one finished step's parts. Idempotent: a retried persist of the SAME
* (message, stepIndex) is a no-op via ON CONFLICT DO NOTHING the per-step
* writes are fired fire-and-forget + serialized, and a duplicate must never
* throw into the stream or double the parts. Returns whether a NEW row landed
* (false = the step was already persisted).
*/
async insertStep(
messageId: string,
workspaceId: string,
stepIndex: number,
parts: unknown,
trx?: KyselyTransaction,
): Promise<boolean> {
const db = dbOrTx(this.db, trx);
const inserted = await db
.insertInto('aiChatRunSteps')
.values({
messageId,
workspaceId,
stepIndex,
// jsonb column: cast through never (same pattern as the message repo).
parts: parts as never,
})
.onConflict((oc) => oc.columns(['messageId', 'stepIndex']).doNothing())
.returning('id')
.executeTakeFirst();
return inserted !== undefined;
}
/** All persisted steps for ONE assistant message, in step order. */
async findByMessage(
messageId: string,
workspaceId: string,
): Promise<AiChatRunStep[]> {
return this.db
.selectFrom('aiChatRunSteps')
.selectAll('aiChatRunSteps')
.where('messageId', '=', messageId)
.where('workspaceId', '=', workspaceId)
.orderBy('stepIndex', 'asc')
.execute();
}
/**
* All persisted steps for a SET of assistant messages, grouped by messageId
* (each group in step order). One query for the batch the hydration seam
* (getMessages / delta / export) calls this only for the rows that actually
* need reconstruction (an active new-style row whose `metadata.parts` is still
* empty), which is usually none, so this is skipped on the common path.
*/
async findByMessageIds(
messageIds: string[],
workspaceId: string,
): Promise<Map<string, AiChatRunStep[]>> {
const byMessage = new Map<string, AiChatRunStep[]>();
if (messageIds.length === 0) return byMessage;
const rows = await this.db
.selectFrom('aiChatRunSteps')
.selectAll('aiChatRunSteps')
.where('messageId', 'in', messageIds)
.where('workspaceId', '=', workspaceId)
.orderBy('stepIndex', 'asc')
.execute();
for (const row of rows) {
const list = byMessage.get(row.messageId);
if (list) list.push(row);
else byMessage.set(row.messageId, [row]);
}
return byMessage;
}
}
@@ -200,8 +200,11 @@ export class PageEmbeddingRepo {
*
* The `model_dimensions = $dim` filter applies ONLY on the semantic side
* (cosine compares same-dimension vectors; pgvector errors otherwise). The
* lexical side (`fts`) is dimension-independent. If `websearch_to_tsquery`
* yields an EMPTY query (e.g. the text is all stopwords) the `@@` matches
* lexical side (`fts`) is dimension-independent. Its query config is `ru_en`,
* matched IN LOCKSTEP with the `page_embeddings.fts` generated column's config
* (#529 acceptance #13): a mismatch silently breaks Cyrillic RAG retrieval. If
* `websearch_to_tsquery` yields an EMPTY query (e.g. the text is all stopwords)
* the `@@` matches
* nothing and the lexical CTE is empty, so results degrade to pure-semantic
* which is correct behaviour, not an error.
*
@@ -249,7 +252,7 @@ export class PageEmbeddingRepo {
row_number() OVER (ORDER BY ts_rank(pe.fts, q.query) DESC) AS rank_ix
FROM page_embeddings pe
JOIN pages p ON p.id = pe.page_id,
websearch_to_tsquery('english', f_unaccent(${queryText})) AS q(query)
websearch_to_tsquery('ru_en', f_unaccent(${queryText})) AS q(query)
WHERE pe.workspace_id = ${workspaceId}
AND pe.space_id IN (${spaceList})
AND p.deleted_at IS NULL
@@ -157,6 +157,44 @@ export class PageHistoryRepo {
return { ...result, items: result.items.map(attachPageHistoryAgent) };
}
/**
* #395 cheap projection of a page's FULL history timeline for the work-time
* estimate: only the columns the sessionizer needs, no heavy `content`, sorted
* oldestnewest. The secondary `id` tie-break keeps rows sharing a `createdAt`
* (e.g. a synchronous pre-agent boundary row + the immediate agent snapshot)
* in a deterministic order.
*/
async findTimelineByPageId(
pageId: string,
trx?: KyselyTransaction,
): Promise<
Array<
Pick<
PageHistory,
| 'createdAt'
| 'lastUpdatedById'
| 'lastUpdatedSource'
| 'lastUpdatedAiChatId'
| 'kind'
>
>
> {
const db = dbOrTx(this.db, trx);
return db
.selectFrom('pageHistory')
.select([
'createdAt',
'lastUpdatedById',
'lastUpdatedSource',
'lastUpdatedAiChatId',
'kind',
])
.where('pageId', '=', pageId)
.orderBy('createdAt', 'asc')
.orderBy('id', 'asc')
.execute();
}
async findPageLastHistory(
pageId: string,
opts?: {
+17
View File
@@ -692,6 +692,22 @@ export interface AiChatRuns {
updatedAt: Generated<Timestamp>;
}
// Append-only per-step persistence for an assistant turn (#492). Mirrors
// migration 20260708T120000-ai-chat-run-steps.ts. Each finished agent step's UI
// `parts` are INSERTed as their own row (instead of rewriting the message row's
// growing `metadata.parts` jsonb every step — an O(n²) WAL/TOAST churn). The full
// `metadata.parts` is assembled once at finalize; mid-run a resuming client's seed
// is rebuilt by concatenating these rows in `stepIndex` order. Cascades with the
// assistant message row it projects.
export interface AiChatRunSteps {
id: Generated<string>;
messageId: string;
workspaceId: string;
stepIndex: number;
parts: Json;
createdAt: Generated<Timestamp>;
}
// Per-(chat,page) snapshot of the open page's Markdown at the END of the agent's
// previous turn (#274). Mirrors migration 20260702T120000-ai-chat-page-snapshot.ts.
// The next turn diffs the CURRENT Markdown against `contentMd` to surface edits a
@@ -729,6 +745,7 @@ export interface DB {
aiChats: AiChats;
aiChatMessages: AiChatMessages;
aiChatRuns: AiChatRuns;
aiChatRunSteps: AiChatRunSteps;
aiChatPageSnapshots: AiChatPageSnapshots;
apiKeys: ApiKeys;
attachments: Attachments;
@@ -4,6 +4,7 @@ import {
AiChats,
AiChatMessages,
AiChatRuns,
AiChatRunSteps,
AiChatPageSnapshots,
Attachments,
Comments,
@@ -64,6 +65,12 @@ export type InsertableAiChatMessage = Omit<Insertable<AiChatMessages>, 'tsv'>;
export type AiChatRun = Selectable<AiChatRuns>;
export type InsertableAiChatRun = Insertable<AiChatRuns>;
// AI Chat Run Step (#492): append-only per-step parts persistence. Each finished
// agent step's UI parts are stored as their own row; the full turn's parts are
// assembled from these (in stepIndex order) for a mid-run resume seed.
export type AiChatRunStep = Selectable<AiChatRunSteps>;
export type InsertableAiChatRunStep = Insertable<AiChatRunSteps>;
// AI Chat Page Snapshot (#274): per-(chat,page) Markdown snapshot taken at the
// end of the agent's previous turn, diffed against the current page next turn to
// detect human edits made between turns.
@@ -0,0 +1,70 @@
import { INTERNAL_LINK_REGEX } from './utils';
import { isInternalPagePath } from '@docmost/prosemirror-markdown';
/**
* Cross-package DRIFT GUARD for the internal-link subset invariant (#522).
*
* The client-side `isInternalPagePath`
* (`packages/prosemirror-markdown/src/lib/internal-links.ts`) promotes a markdown
* link to `internal: true` on import. Every link it marks internal MUST be one
* the server would backlink and export-rewrite i.e. the client matcher MUST be
* a STRICT SUBSET of the server's canonical `INTERNAL_LINK_REGEX`
* (`./utils.ts`). If the client ever accepts a path the server rejects, that link
* is stored internal but silently dropped from the backlink graph and broken on
* export the exact bug #522 fixed.
*
* This spec is the load-bearing guard: it imports the LIVE server regex AND the
* LIVE `isInternalPagePath` (no hand-copied regex on either side). A narrowing of
* EITHER most dangerously the server regex reddens here. The in-package
* accept/reject test documents the client's behaviour but cannot see the server
* regex; this top-layer spec is what makes the subset relation mechanical
* (AGENTS.md rule #7: a CI test that fails on drift of the source of truth).
*/
describe('internal-link subset parity (client isInternalPagePath ⊆ server INTERNAL_LINK_REGEX)', () => {
// Every path the CLIENT accepts. Kept deliberately broad across the risky
// dimensions — the slug charset (digits, hyphens, mixed case, leading/trailing
// hyphen), the space charset, and the optional trailing slash — so a narrowing
// of the server regex on any of them reddens the subset assertion below.
const CLIENT_ACCEPTS = [
'/s/eng/p/abc123',
'/s/eng/p/abc123/', // trailing slash
'/s/My-Space/p/A-b-C-9', // hyphens + mixed case in both segments
'/s/x/p/z', // shortest
'/s/eng/p/my-page-abc123', // slug with hyphens (extractPageSlugId shape)
'/s/eng/p/-lead', // leading hyphen in slug
'/s/eng/p/trail-', // trailing hyphen in slug
'/s/eng/p/0123456789', // all-digit slug
'/s/a.b/p/abc', // dot in the SPACE segment (space charset is [^/]+)
'/s/space with space/p/abc', // space char in the SPACE segment
];
it('every corpus path is actually client-accepted (guards the corpus itself)', () => {
// If a path here stopped being client-accepted the subset test would pass
// vacuously; assert acceptance up front so the corpus stays meaningful. The
// filter-to-empty form names the offending paths on failure.
const notAccepted = CLIENT_ACCEPTS.filter((h) => !isInternalPagePath(h));
expect(notAccepted).toEqual([]);
});
it('every client-accepted path also matches the LIVE server regex (subset)', () => {
// The mechanical drift guard: narrow the server INTERNAL_LINK_REGEX and at
// least one hyphen/charset/structure case appears here.
const notInServer = CLIENT_ACCEPTS.filter((h) => !INTERNAL_LINK_REGEX.test(h));
expect(notInServer).toEqual([]);
});
it('the client rejects forbidden-slug-char paths the server also rejects', () => {
// Documents the subset BOUNDARY: correct shape, forbidden slug char. The
// client and the LIVE server regex must agree on rejection.
const forbidden = [
'/s/eng/p/abc.def',
'/s/eng/p/abc_def',
'/s/eng/p/abc%20',
'/s/eng/p/abc~x',
];
const clientAccepts = forbidden.filter((h) => isInternalPagePath(h));
const serverAccepts = forbidden.filter((h) => INTERNAL_LINK_REGEX.test(h));
expect(clientAccepts).toEqual([]);
expect(serverAccepts).toEqual([]);
});
});
@@ -85,6 +85,14 @@ export class ImportController {
throw new BadRequestException('spaceId is required');
}
// #502: optional multipart field. Only the MCP agent `createPage` path sends
// `disableMarkdownExtensions=true` (its body is agent-authored plain prose /
// config, so a `$…$` span must stay literal and a bare `www.host` must not
// autolink). A HUMAN file upload omits the field, so it stays false and math
// + autolink remain ON for human imports. Settable ONLY via this API param.
const disableMarkdownExtensions =
file.fields?.disableMarkdownExtensions?.value === 'true';
const ability = await this.spaceAbility.createForUser(user, spaceId);
if (ability.cannot(SpaceCaslAction.Edit, SpaceCaslSubject.Page)) {
throw new ForbiddenException();
@@ -95,6 +103,7 @@ export class ImportController {
user.id,
spaceId,
workspace.id,
disableMarkdownExtensions,
);
const ext = path.extname(file.filename).toLowerCase();
@@ -0,0 +1,67 @@
// Importing ImportService transitively loads import-formatter.ts, which imports
// the ESM-only @sindresorhus/slugify (not in jest's transform allowlist). It is
// irrelevant to this path, so mock it to keep the module graph loadable (mirrors
// the sibling import.service specs).
jest.mock('@sindresorhus/slugify', () => ({
__esModule: true,
default: (input: string) => String(input),
}));
import { ImportService } from './import.service';
// #502 BLOCKER 1: the server markdown import path (`/pages/import`) now accepts an
// optional `disableMarkdownExtensions` flag threaded into `processMarkdown`.
// - MCP agent `createPage` sends it TRUE -> extensions OFF (a `$…$` config span
// stays literal, a schemeless `www.host` is not autolinked).
// - a HUMAN file upload omits it (default FALSE) -> extensions ON, so a real
// `$x^2$` still becomes a formula (human imports unaffected).
// `processMarkdown` only uses the imported converter (no injected deps on this
// path), so the service is constructed with null deps for this focused unit test.
function makeService(): ImportService {
return new ImportService(null as any, null as any, null as any, null as any);
}
function findAll(node: any, type: string, acc: any[] = []): any[] {
if (!node || typeof node !== 'object') return acc;
if (node.type === type) acc.push(node);
if (Array.isArray(node.content)) for (const c of node.content) findAll(c, type, acc);
return acc;
}
function hasLink(node: any): boolean {
return findAll(node, 'text').some((t: any) =>
t.marks?.some((m: any) => m.type === 'link'),
);
}
describe('ImportService.processMarkdown — #502 disableMarkdownExtensions', () => {
it('disableMarkdownExtensions=true (MCP createPage): `$…$` stays literal, no math', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('export A=$FOO and B=$BAR done', true);
expect(findAll(doc, 'mathInline')).toHaveLength(0);
});
it('disableMarkdownExtensions=true: a schemeless www is NOT autolinked', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see www.example.com here', true);
expect(hasLink(doc)).toBe(false);
});
it('disableMarkdownExtensions=true: an explicit https:// STILL links', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see https://example.com here', true);
expect(hasLink(doc)).toBe(true);
});
it('DEFAULT (human upload): a real `$x^2$` DOES become a math node', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('$x^2$');
expect(findAll(doc, 'mathInline')).toHaveLength(1);
});
it('DEFAULT (human upload): a schemeless www IS autolinked', async () => {
const svc = makeService();
const doc = await svc.processMarkdown('see www.example.com here');
expect(hasLink(doc)).toBe(true);
});
});
@@ -52,6 +52,12 @@ export class ImportService {
userId: string,
spaceId: string,
workspaceId: string,
// #502: when true, the markdown importer runs with the two layered
// extensions OFF (a `$…$` span stays literal text; a schemeless `www.host` is
// NOT autolinked). ONLY the MCP agent `createPage` path sets this; a HUMAN
// file upload never passes it, so it defaults false and math/autolink stay ON
// for human imports (their `$x^2$` still becomes a formula).
disableMarkdownExtensions = false,
) {
const file = await filePromise;
const fileBuffer = await file.toBuffer();
@@ -66,7 +72,10 @@ export class ImportService {
try {
if (fileExtension.endsWith('.md')) {
prosemirrorState = await this.processMarkdown(fileContent);
prosemirrorState = await this.processMarkdown(
fileContent,
disableMarkdownExtensions,
);
} else if (fileExtension.endsWith('.html')) {
prosemirrorState = await this.processHTML(fileContent);
}
@@ -138,7 +147,12 @@ export class ImportService {
return createdPage;
}
async processMarkdown(markdownInput: string): Promise<any> {
async processMarkdown(
markdownInput: string,
// #502: forwarded to the importer. DEFAULT false keeps math + fuzzy autolink
// ON (human uploads unaffected); the MCP agent `createPage` path passes true.
disableMarkdownExtensions = false,
): Promise<any> {
// Canonical markdown -> ProseMirror JSON directly via
// `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate and no
// second editor-ext markdown layer. Foreign markdown surfaces the strict
@@ -147,7 +161,12 @@ export class ImportService {
// The HTML-cleanup pass (`normalizeImportHtml`) is intentionally skipped here:
// it targets foreign *HTML* (Notion/XWiki), which only ever arrives on the
// `.html` path (`processHTML`), never as canonical markdown.
return markdownToProseMirror(normalizeForeignMarkdown(markdownInput));
return markdownToProseMirror(
normalizeForeignMarkdown(markdownInput),
disableMarkdownExtensions
? { parseMath: false, fuzzyLinkify: false }
: undefined,
);
}
async processHTML(htmlInput: string): Promise<any> {
@@ -0,0 +1,412 @@
import * as http from 'node:http';
import { Kysely } from 'kysely';
import { tool } from 'ai';
import { z } from 'zod';
import { MockLanguageModelV3, convertArrayToReadableStream } from 'ai/test';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import {
AiChatService,
assembleStepParts,
assistantParts,
rowHasInlineParts,
stepMarkerMetadata,
} from 'src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #492 append-persist the REAL onStep WRITE path (F2) and the model-REPLAY
* hydration path (F1), driven through `AiChatService.stream` against a LIVE
* Postgres with a REAL `AiChatRunStepRepo` INJECTED. The existing append-persist
* int-specs hand-roll the insert+marker cycle via the repos directly and build
* the service with `aiChatRunStepRepo: undefined` (only the legacy-fallback branch
* is covered), so an off-by-one on `stepsPersisted-1`, a wrong `capturedSteps`
* slice, or a broken marker payload would pass all of them. These tests exercise
* the actual `updateStreaming` append-persist branch end to end.
*
* The seam is the injected `model` (a seeded `MockLanguageModelV3` from `ai/test`)
* plus a REAL Node `ServerResponse` as the hijacked socket mirrors
* ai-chat-stream.int-spec.ts.
*/
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
async function waitFor(
cond: () => Promise<boolean> | boolean,
{ timeoutMs = 15_000, stepMs = 25 } = {},
): Promise<void> {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
if (await cond()) return;
await sleep(stepMs);
}
throw new Error('waitFor: condition not met within timeout');
}
// A real Node ServerResponse wired to a live socket (identical helper to the
// stream int-spec) so the SDK's pipe/heartbeat writes behave as in prod.
function makeRealResponse(): Promise<{
res: http.ServerResponse;
cleanup: () => Promise<void>;
}> {
return new Promise((resolve) => {
const server = http.createServer((_req, res) => {
resolve({
res,
cleanup: () =>
new Promise<void>((done) => {
try {
if (!res.writableEnded) res.end();
} catch {
/* socket already gone */
}
server.close(() => done());
}),
});
});
server.listen(0, () => {
const port = (server.address() as any).port;
const creq = http.request({ port, method: 'GET' }, (cres) => {
cres.resume();
});
creq.on('error', () => undefined);
creq.end();
});
});
}
// Stream parts for a normal, successful single-step turn.
function successStream() {
return convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: 't1' },
{ type: 'text-delta', id: 't1', delta: 'Hello' },
{ type: 'text-delta', id: 't1', delta: ' there' },
{ type: 'text-end', id: 't1' },
{
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 10, outputTokens: 5, totalTokens: 15 },
},
] as any);
}
// A THREE-step turn: steps 0 and 1 each emit text + an `echo` tool call (the SDK
// runs the tool and continues); step 2 answers and stops. Three steps is
// deliberate: the LAST finished step's append-persist write races the terminal
// finalize (which writes the full inline parts anyway, so a lost last-step row is
// by design), but the NON-final steps 0 and 1 always drain to the steps table
// before finalize — so those are what the test asserts on deterministically.
function threeStepModel(): MockLanguageModelV3 {
let step = 0;
const toolStep = (i: number) => ({
stream: convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: `s${i}` },
{ type: 'text-delta', id: `s${i}`, delta: `step ${i} ` },
{ type: 'text-end', id: `s${i}` },
{
type: 'tool-call',
toolCallId: `c${i}`,
toolName: 'echo',
input: JSON.stringify({ msg: `m${i}` }),
},
{
type: 'finish',
finishReason: 'tool-calls',
usage: { inputTokens: 5, outputTokens: 3, totalTokens: 8 },
},
] as any),
});
return new MockLanguageModelV3({
doStream: async () => {
const n = step++;
// Realistic inter-step latency. A real model spends seconds per step, so the
// fire-and-forget per-step write chain drains to the steps table BETWEEN
// steps; the mock otherwise collapses all steps into microseconds and the
// terminal finalize wins the race before any but the first step persists.
if (n > 0) await sleep(200);
if (n < 2) return toolStep(n);
return {
stream: convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: 's2' },
{ type: 'text-delta', id: 's2', delta: 'final answer' },
{ type: 'text-end', id: 's2' },
{
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 6, outputTokens: 4, totalTokens: 10 },
},
] as any),
};
},
} as any);
}
describe('#492 append-persist service paths [integration]', () => {
let db: Kysely<any>;
let aiChatRepo: AiChatRepo;
let msgRepo: AiChatMessageRepo;
let stepRepo: AiChatRunStepRepo;
let workspaceId: string;
let userId: string;
let closeCalls: number;
const mcpClients = {
toolsFor: async () => ({
tools: {},
clients: [
{
close: async () => {
closeCalls += 1;
},
},
],
outcomes: [],
instructions: [],
}),
};
// Build the service WITH a REAL AiChatRunStepRepo injected (the property under
// test) — unlike the legacy-fallback harness that passes it as undefined.
const echoTool = tool({
description: 'echo the message back',
inputSchema: z.object({ msg: z.string() }),
execute: async ({ msg }) => ({ echoed: msg }),
});
function buildService(): AiChatService {
return new AiChatService(
{ getChatModel: async () => null } as any,
aiChatRepo,
msgRepo,
{} as any, // aiChatPageSnapshotRepo
{ resolve: async () => null } as any, // aiSettings
{ forUser: async () => ({ echo: echoTool }) } as any, // tools
mcpClients as any,
{} as any, // aiAgentRoleRepo
{} as any, // pageRepo
{} as any, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatFinalStepLockdownEnabled: () => false,
} as any, // environment (deferred OFF -> all tools active every step)
undefined, // streamRegistry
undefined, // aiChatRunService
stepRepo, // #492 aiChatRunStepRepo — the append-persist backend
);
}
function userUiMessage(text: string) {
return {
id: `u-${Math.random()}`,
role: 'user',
parts: [{ type: 'text', text }],
};
}
async function runStream(opts: {
model: MockLanguageModelV3;
chatId: string;
body: any;
}): Promise<void> {
closeCalls = 0;
const service = buildService();
const { res, cleanup } = await makeRealResponse();
try {
await service.stream({
user: { id: userId, workspaceId } as any,
workspace: { id: workspaceId, name: 'WS' } as any,
sessionId: 'sess-1',
body: opts.body,
res: { raw: res } as any,
signal: new AbortController().signal,
model: opts.model as any,
role: null,
} as any);
await waitFor(async () => {
const rows = await msgRepo.findAllByChat(opts.chatId, workspaceId);
return rows.some(
(r) =>
r.role === 'assistant' &&
['completed', 'error', 'aborted'].includes(r.status as string),
);
});
await waitFor(() => closeCalls > 0, { timeoutMs: 5_000 });
} finally {
await cleanup();
}
}
beforeAll(async () => {
db = getTestDb();
aiChatRepo = new AiChatRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
stepRepo = new AiChatRunStepRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
// --- F2: the real onStep append-persist WRITE branch -----------------------
it('drives steps through the real onStep path: per-step rows + marker match a single-row flush', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const model = threeStepModel();
// Capture the mid-run step-marker UPDATEs the append-persist branch writes on
// the assistant row (a { parts: [], toolTraceVersion, stepsPersisted } patch).
const updateSpy = jest.spyOn(msgRepo, 'update');
try {
await runStream({
model,
chatId,
body: { chatId, messages: [userUiMessage('call the tool then answer')] },
});
const rows = await msgRepo.findAllByChat(chatId, workspaceId);
const assistant = rows.find((r) => r.role === 'assistant')!;
expect(assistant).toBeDefined();
expect(assistant.status).toBe('completed');
// The turn finalizes with the FULL inline parts assembled by a single-row
// flush (assistantParts over every step) — the baseline the per-step slices
// must reproduce.
expect(rowHasInlineParts(assistant)).toBe(true);
const finalParts = (assistant.metadata as { parts: any[] }).parts;
// The two NON-final finished steps each landed their own row, in stepIndex
// order. (The fire-and-forget write chain drains before the next step, so
// poll until both are on disk; the LAST step's write may lose the finalize
// race, which is by design — its parts are already in `finalParts`.)
await waitFor(async () => {
const s = await stepRepo.findByMessage(assistant.id, workspaceId);
return s.length >= 2;
});
const steps = await stepRepo.findByMessage(assistant.id, workspaceId);
expect(steps[0].stepIndex).toBe(0);
expect(steps[1].stepIndex).toBe(1);
// Each per-step row carries a NON-trivial slice: this step's text part + its
// paired tool part (guards a mutation that persists empty/whole-turn parts).
const s0 = steps[0].parts as any[];
expect(s0).toContainEqual({ type: 'text', text: 'step 0 ' });
expect(s0.some((p) => p.type === 'tool-echo')).toBe(true);
// The per-step slices are EXACTLY the corresponding prefix of the single-row
// flush: assembleStepParts([step0, step1]) === finalParts[0 .. len0+len1].
// This is what an off-by-one on `stepsPersisted-1` (a wrong `capturedSteps`
// slice) or a shifted stepIndex breaks — the prefix no longer aligns.
const prefixLen =
(steps[0].parts as any[]).length + (steps[1].parts as any[]).length;
expect(assembleStepParts([steps[0], steps[1]] as any)).toEqual(
finalParts.slice(0, prefixLen),
);
// The mid-run step markers advanced 1 -> 2 -> ... (the resume frontier), each
// a shape-stable empty-parts marker equal to a single-row flush's marker.
const markerCounts = updateSpy.mock.calls
.map((c) => (c[2] as any)?.metadata)
.filter(
(m) =>
m &&
Array.isArray(m.parts) &&
m.parts.length === 0 &&
typeof m.stepsPersisted === 'number',
)
.map((m) => m.stepsPersisted);
// Monotonic from 1, covering at least the two non-final steps.
expect(markerCounts.slice(0, 2)).toEqual([1, 2]);
expect(
updateSpy.mock.calls
.map((c) => (c[2] as any)?.metadata)
.find((m) => m && m.stepsPersisted === 2),
).toEqual(stepMarkerMetadata(2));
} finally {
updateSpy.mockRestore();
}
}, 60_000);
// --- F1: model-REPLAY hydrates a hard-crashed mid-run turn from the steps table
it('replays a hard-crashed mid-run turn WITH its partial steps hydrated from the steps table', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
// Prior turn: a genuine user question...
await createMessage(db, {
workspaceId,
chatId,
userId,
role: 'user',
content: 'What is in the design doc?',
createdAt: new Date(Date.now() - 3000),
});
// ...and an assistant row that a HARD crash (SIGKILL/OOM) left mid-run: only a
// step marker on the row (metadata.parts:[] , content:''), NO terminal
// callback ever fired, so its real parts live ONLY in ai_chat_run_steps.
const crashed = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: '',
status: 'aborted',
metadata: stepMarkerMetadata(1),
createdAt: new Date(Date.now() - 2000),
});
// The durable partial step: some reasoning text + a completed getPage tool
// call (input + output), exactly what #183 step-granular durability preserves.
await stepRepo.insertStep(
crashed.id,
workspaceId,
0,
assistantParts(
[
{
text: 'HYDRATED_PARTIAL_STEP the doc says',
toolCalls: [
{ toolCallId: 'g1', toolName: 'getPage', input: { id: 'p1' } },
],
toolResults: [
{
toolCallId: 'g1',
toolName: 'getPage',
output: { id: 'p1', body: 'PARTIAL_TOOL_OUTPUT budget section' },
},
],
} as any,
],
'',
),
);
// The NEXT turn: the model just answers. The service must REPLAY the crashed
// assistant turn with its partial parts hydrated from the steps table.
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: successStream() }),
} as any);
await runStream({
model,
chatId,
body: { chatId, messages: [userUiMessage('Continue please')] },
});
expect(model.doStreamCalls.length).toBeGreaterThan(0);
const prompt = JSON.stringify(model.doStreamCalls[0].prompt);
// The partial step's TEXT reached the model context (it would be an empty text
// part without hydration — rowToUiMessage falls back to `content:''`).
expect(prompt).toContain('HYDRATED_PARTIAL_STEP');
// The partial TOOL RESULT survived too (durable in the steps table, replayed).
expect(prompt).toContain('PARTIAL_TOOL_OUTPUT');
// The genuine prior user turn is present as well (sanity: real history replay).
expect(prompt).toContain('What is in the design doc?');
}, 60_000);
});
@@ -0,0 +1,173 @@
import { randomBytes } from 'crypto';
import { Kysely, sql } from 'kysely';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import {
assistantParts,
flushAssistant,
stepMarkerMetadata,
} from '../../src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
} from './db';
/**
* #492 append-persist WRITE-VOLUME regression on a LIVE Postgres, measured via
* the `pg_current_wal_lsn()` delta around a realistic multi-step run driven through
* the REAL repos (not a mock a mock cannot observe MVCC/TOAST rewrite volume, the
* whole point). Proves the core claim:
*
* NEW (per-step INSERT into ai_chat_run_steps + a CHEAP step-marker UPDATE on the
* message row) writes O(Σ steps) of WAL each step writes only its own bytes.
*
* OLD (the pre-#492 full-row rewrite: re-persist the GROWING metadata.parts on
* every onStepFinish) writes O(n²) step k rewrites the whole TOASTed jsonb of
* all k prior outputs.
*
* The OLD path here IS the reverted behavior, so this doubles as the mutation
* check: swapping the new path back to `flushAssistant` full-row UPDATEs reddens
* the assertion (OLD is many times larger).
*/
type Step = {
text: string;
toolCalls: Array<{ toolCallId: string; toolName: string; input: unknown }>;
toolResults: Array<{ toolCallId: string; toolName: string; output: unknown }>;
};
// ~100 KB INCOMPRESSIBLE output per step (a page read). Random base64 so TOAST
// cannot compress it away and hide the real write volume.
function makeStep(i: number, outputBytes = 100_000): Step {
const body = randomBytes(Math.ceil(outputBytes * 0.75)).toString('base64');
return {
text: `step ${i} reasoning`,
toolCalls: [
{ toolCallId: `c${i}`, toolName: 'getPage', input: { id: `p${i}` } },
],
toolResults: [
{
toolCallId: `c${i}`,
toolName: 'getPage',
output: { id: `p${i}`, title: `Page ${i}`, body },
},
],
};
}
async function walDelta(
db: Kysely<any>,
fn: () => Promise<void>,
): Promise<number> {
const before = (
await sql<{ l: string }>`select pg_current_wal_lsn() as l`.execute(db)
).rows[0].l;
await fn();
// NOTE: no pg_switch_wal() — a segment switch pads the LSN to the next 16 MB
// boundary and would swamp the delta. The raw LSN advances by the WAL bytes.
const after = (
await sql<{ l: string }>`select pg_current_wal_lsn() as l`.execute(db)
).rows[0].l;
return Number(
(
await sql<{
d: string;
}>`select pg_wal_lsn_diff(${after}::pg_lsn, ${before}::pg_lsn) as d`.execute(
db,
)
).rows[0].d,
);
}
describe('#492 append-persist write volume (pg_current_wal_lsn delta) [integration]', () => {
let db: Kysely<any>;
let stepRepo: AiChatRunStepRepo;
let msgRepo: AiChatMessageRepo;
let workspaceId: string;
let userId: string;
let chatId: string;
beforeAll(async () => {
db = getTestDb();
stepRepo = new AiChatRunStepRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
const seedRow = () =>
msgRepo.insert({
chatId,
workspaceId,
userId,
role: 'assistant',
content: '',
status: 'streaming',
metadata: stepMarkerMetadata(0) as never,
});
const STEPS = 40;
it('NEW per-step INSERT is O(Σ steps); OLD full-row rewrite is O(n²)', async () => {
const steps: Step[] = [];
for (let i = 0; i < STEPS; i++) steps.push(makeStep(i));
// NEW: per-step INSERT of THIS step's parts + a cheap marker UPDATE.
const newRow = await seedRow();
const newWal = await walDelta(db, async () => {
for (let i = 0; i < STEPS; i++) {
await stepRepo.insertStep(
newRow.id,
workspaceId,
i,
assistantParts([steps[i]], ''),
);
await msgRepo.update(
newRow.id,
workspaceId,
{ metadata: stepMarkerMetadata(i + 1) },
{ onlyIfStreaming: true },
);
}
});
// OLD (the pre-#492 revert): re-persist the GROWING metadata.parts on the
// message row on every step.
const oldRow = await seedRow();
const oldWal = await walDelta(db, async () => {
const acc: Step[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(steps[i]);
await msgRepo.update(
oldRow.id,
workspaceId,
flushAssistant(acc as never, '', 'streaming'),
{ onlyIfStreaming: true },
);
}
});
// eslint-disable-next-line no-console
console.log(
`[#492 WAL] ${STEPS} steps ×100KB: new=${(newWal / 1e6).toFixed(1)}MB ` +
`old=${(oldWal / 1e6).toFixed(1)}MB (${(oldWal / newWal).toFixed(
1,
)}x smaller)`,
);
// O(Σ steps): ~STEPS × (100KB output + marker) of WAL. 40 × ~100KB parts plus
// 40 tiny markers is a few tens of MB at most — bounded, linear in step count.
expect(newWal).toBeLessThan(30_000_000);
// O(n²): step k rewrites ~k × 100KB. Σ over 40 steps ≈ 80+ MB — far larger.
expect(oldWal).toBeGreaterThan(30_000_000);
// The load-bearing claim: the new path writes a small FRACTION of the old.
expect(newWal).toBeLessThan(oldWal * 0.35);
}, 120_000);
});
@@ -0,0 +1,169 @@
import { Kysely } from 'kysely';
import { AiChatController } from 'src/core/ai-chat/ai-chat.controller';
import {
assembleStepParts,
assistantParts,
stepMarkerMetadata,
} from 'src/core/ai-chat/ai-chat.service';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import type { User, Workspace } from '@docmost/db/types/entity.types';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #492 controller hydration (crash-before-finalize RESUME) on a LIVE Postgres.
* `AiChatController.withReconstructedParts` is wired into getMessages/delta/export/
* run, but `aiChatRunStepRepo` is OPTIONAL and every controller unit spec passes it
* as `undefined`, so the hydration branch early-returns and NEVER executes in those
* tests. This drives the real read path a mid-run streaming row (marker only,
* empty inline parts) PLUS its `ai_chat_run_steps` rows through getMessages WITH
* the repo present, exercising the `role==='assistant' && !rowHasInlineParts`
* needy predicate, the workspace-scoped batch step fetch, and the endpoint binding.
*/
describe('#492 controller hydration read path [integration]', () => {
let db: Kysely<any>;
let aiChatRepo: AiChatRepo;
let msgRepo: AiChatMessageRepo;
let stepRepo: AiChatRunStepRepo;
let workspaceId: string;
let otherWorkspaceId: string;
let userId: string;
// Build the controller WITH a real AiChatRunStepRepo injected (position 9), the
// seam the unit specs leave undefined. Only the read-path deps are real.
function buildController(): AiChatController {
return new AiChatController(
{} as any, // aiChatService
{} as any, // aiChatRunService
aiChatRepo,
msgRepo,
{} as any, // aiTranscription
{} as any, // pageRepo
undefined, // streamRegistry
undefined, // environment
stepRepo, // #492 aiChatRunStepRepo
);
}
beforeAll(async () => {
db = getTestDb();
aiChatRepo = new AiChatRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
stepRepo = new AiChatRunStepRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
otherWorkspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
it('getMessages reconstructs a mid-run row from the steps table (finished rows untouched)', async () => {
const chatId = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const user = { id: userId } as User;
const workspace = { id: workspaceId } as Workspace;
// A prior FINISHED assistant row that already carries inline parts — the needy
// predicate must SKIP it (no step fetch), returned untouched.
const finishedParts = assistantParts(
[{ text: 'done earlier', toolCalls: [], toolResults: [] } as any],
'',
);
await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: 'done earlier',
status: 'completed',
metadata: { parts: finishedParts, toolTraceVersion: 2, stepsPersisted: 1 },
createdAt: new Date(Date.now() - 3000),
});
// The mid-run row a crash-before-finalize left behind: a step marker only
// (parts:[] , content:''), status 'streaming'. Its real parts live ONLY in the
// steps table.
const midRun = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
content: '',
status: 'streaming',
metadata: stepMarkerMetadata(2),
createdAt: new Date(Date.now() - 1000),
});
const step0 = assistantParts(
[
{
text: 'reasoning about the page',
toolCalls: [
{ toolCallId: 'g1', toolName: 'getPage', input: { id: 'p1' } },
],
toolResults: [
{ toolCallId: 'g1', toolName: 'getPage', output: { id: 'p1', body: 'B' } },
],
} as any,
],
'',
);
const step1 = assistantParts(
[{ text: 'partial synthesis so far', toolCalls: [], toolResults: [] } as any],
'',
);
await stepRepo.insertStep(midRun.id, workspaceId, 0, step0);
await stepRepo.insertStep(midRun.id, workspaceId, 1, step1);
// Workspace-scoping guard: a step row for the SAME message id under a DIFFERENT
// workspace must NEVER leak into this workspace's reconstruction.
await stepRepo.insertStep(midRun.id, otherWorkspaceId, 99, [
{ type: 'text', text: 'FOREIGN_WORKSPACE_LEAK' },
]);
const res = await buildController().getMessages(
{ chatId } as any,
{ limit: 50 } as any,
user,
workspace,
);
const items = res.items as any[];
const finished = items.find((r) => r.status === 'completed');
const reconstructed = items.find((r) => r.id === midRun.id);
// The finished row passed through with its inline parts unchanged.
expect(finished.metadata.parts).toEqual(finishedParts);
// The mid-run row's parts were reconstructed from the two step rows, in order,
// exactly as assembleStepParts concatenates them — the client seed sees the
// persisted progress with no change to itself.
const expected = assembleStepParts([
{ stepIndex: 0, parts: step0 },
{ stepIndex: 1, parts: step1 },
] as any);
expect(reconstructed.metadata.parts).toEqual(expected);
// The foreign-workspace step row did NOT leak in.
expect(JSON.stringify(reconstructed.metadata.parts)).not.toContain(
'FOREIGN_WORKSPACE_LEAK',
);
// Sanity: reconstruction produced real content (text + the paired tool part +
// the second step's text), not an empty fallback.
expect(reconstructed.metadata.parts).toContainEqual({
type: 'text',
text: 'reasoning about the page',
});
expect(
(reconstructed.metadata.parts as any[]).some((p) => p.type === 'tool-getPage'),
).toBe(true);
}, 60_000);
});
@@ -0,0 +1,163 @@
import { randomBytes } from 'crypto';
import { Kysely } from 'kysely';
import { AiChatRunStepRepo } from '@docmost/db/repos/ai-chat/ai-chat-run-step.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import {
assistantParts,
reconstructRunParts,
hydrateAssistantParts,
stepMarkerMetadata,
rowHasInlineParts,
} from '../../src/core/ai-chat/ai-chat.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
} from './db';
/**
* #492 append-persist the reconstruct CONTRACT on a live Postgres. Proves that a
* turn persisted the NEW way (per-step rows in `ai_chat_run_steps`, only a step
* marker on the message row) reconstructs to the SAME UI parts as a turn persisted
* the OLD way (full `metadata.parts` inline on the row, no step rows) so the
* era-switch is invisible to attach / delta-poll / export. Real repos + real jsonb
* roundtrip, not a mock (a mock cannot prove the parts survive the jsonb column
* byte-identical).
*/
type Step = {
text: string;
toolCalls: Array<{ toolCallId: string; toolName: string; input: unknown }>;
toolResults: Array<{ toolCallId: string; toolName: string; output: unknown }>;
};
// A realistic step: some text + a getPage tool call whose ~100 KB body is
// INCOMPRESSIBLE random base64 (a 'x'.repeat filler would TOAST away and hide the
// real bytes). Under MAX_TOOL_OUTPUT_BYTES (200 KB) it is stored uncompacted.
function makeStep(i: number, outputBytes = 4_000): Step {
const body = randomBytes(Math.ceil(outputBytes * 0.75)).toString('base64');
return {
text: `step ${i} text`,
toolCalls: [
{ toolCallId: `c${i}`, toolName: 'getPage', input: { id: `p${i}` } },
],
toolResults: [
{
toolCallId: `c${i}`,
toolName: 'getPage',
output: { id: `p${i}`, title: `Page ${i}`, body },
},
],
};
}
describe('AiChatRunStepRepo + reconstruct contract [integration]', () => {
let db: Kysely<any>;
let stepRepo: AiChatRunStepRepo;
let msgRepo: AiChatMessageRepo;
let workspaceId: string;
let userId: string;
let chatId: string;
beforeAll(async () => {
db = getTestDb();
stepRepo = new AiChatRunStepRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
const seedRow = (metadata: unknown, status: string) =>
msgRepo.insert({
chatId,
workspaceId,
userId,
role: 'assistant',
content: '',
status,
metadata: metadata as never,
});
it('insertStep is idempotent per (message, stepIndex) and reads back in order', async () => {
const row = await seedRow(stepMarkerMetadata(0), 'streaming');
const parts0 = assistantParts([makeStep(0)], '');
const parts1 = assistantParts([makeStep(1)], '');
expect(await stepRepo.insertStep(row.id, workspaceId, 0, parts0)).toBe(true);
expect(await stepRepo.insertStep(row.id, workspaceId, 1, parts1)).toBe(true);
// A retried persist of the SAME step is a no-op (ON CONFLICT DO NOTHING).
expect(await stepRepo.insertStep(row.id, workspaceId, 0, parts0)).toBe(
false,
);
const steps = await stepRepo.findByMessage(row.id, workspaceId);
expect(steps.map((s) => s.stepIndex)).toEqual([0, 1]);
// Batch fetch groups by message id in step order.
const map = await stepRepo.findByMessageIds([row.id], workspaceId);
expect(map.get(row.id)!.map((s) => s.stepIndex)).toEqual([0, 1]);
});
it('a NEW-style (step-table) run reconstructs identically to an OLD-style (inline) run', async () => {
const steps = [makeStep(10), makeStep(11)];
// The inline parts the OLD full-row flush would have written.
const fullParts = assistantParts(steps, '');
// OLD-style record: full parts inline on the row, NO step rows.
const oldRow = await seedRow(
{ parts: fullParts, toolTraceVersion: 2, stepsPersisted: 2 },
'completed',
);
// NEW-style record: only a step marker on the row + per-step rows.
const newRow = await seedRow(stepMarkerMetadata(2), 'streaming');
for (let i = 0; i < steps.length; i++) {
await stepRepo.insertStep(
newRow.id,
workspaceId,
i,
assistantParts([steps[i]], ''),
);
}
// Re-read both from the DB (proves the jsonb roundtrip).
const oldFetched = await msgRepo.findById(oldRow.id, workspaceId);
const newFetched = await msgRepo.findById(newRow.id, workspaceId);
const oldSteps = await stepRepo.findByMessage(oldRow.id, workspaceId);
const newSteps = await stepRepo.findByMessage(newRow.id, workspaceId);
// The discriminator: the old row carries inline parts, the new one does not.
expect(rowHasInlineParts(oldFetched!)).toBe(true);
expect(rowHasInlineParts(newFetched!)).toBe(false);
expect(oldSteps).toHaveLength(0);
expect(newSteps).toHaveLength(2);
const oldRecon = reconstructRunParts(oldFetched!, oldSteps);
const newRecon = reconstructRunParts(newFetched!, newSteps);
// Both reconstruct to the SAME parts + step count — the era is invisible.
expect(newRecon.parts).toEqual(fullParts);
expect(oldRecon.parts).toEqual(fullParts);
expect(newRecon.parts).toEqual(oldRecon.parts);
expect(newRecon.stepsPersisted).toBe(2);
expect(oldRecon.stepsPersisted).toBe(2);
// hydrateAssistantParts fills the new row's metadata.parts to match the old
// row's inline parts — so a consumer reading `metadata.parts` off the raw row
// (the client seed/poll, export) is unchanged across the era.
const map = await stepRepo.findByMessageIds([newRow.id], workspaceId);
const [hydrated] = hydrateAssistantParts([newFetched!], map);
expect((hydrated.metadata as { parts: unknown }).parts).toEqual(fullParts);
// A row that already has inline parts passes through untouched (same ref-shape).
const [oldPassThrough] = hydrateAssistantParts([oldFetched!], map);
expect((oldPassThrough.metadata as { parts: unknown }).parts).toEqual(
fullParts,
);
});
});
@@ -0,0 +1,63 @@
import { Kysely, sql } from 'kysely';
import {
up,
down,
} from '../../src/database/migrations/20260708T120000-ai-chat-run-steps';
import { getTestDb, destroyTestDb } from './db';
/**
* #492 migration up/down roundtrip on a LIVE Postgres. global-setup already
* migrated docmost_test to latest (so the table exists at start); this drives the
* migration's own down()/up() and asserts the table presence toggles, then leaves
* it PRESENT (up) so the shared test DB is intact for any spec that runs after.
*/
async function tableExists(db: Kysely<any>): Promise<boolean> {
const row = (
await sql<{ t: string | null }>`select to_regclass('ai_chat_run_steps') as t`.execute(
db,
)
).rows[0];
return row.t !== null;
}
async function uniqueIndexExists(db: Kysely<any>): Promise<boolean> {
const row = (
await sql<{
t: string | null;
}>`select to_regclass('ai_chat_run_steps_message_step_uidx') as t`.execute(db)
).rows[0];
return row.t !== null;
}
describe('20260708 ai_chat_run_steps migration roundtrip [integration]', () => {
let db: Kysely<any>;
beforeAll(() => {
db = getTestDb();
});
afterAll(async () => {
// Belt-and-suspenders: guarantee the table is present for later specs even if
// an assertion threw mid-roundtrip.
if (!(await tableExists(db))) await up(db);
await destroyTestDb();
});
it('down() drops the table+index and up() recreates them (idempotent)', async () => {
// Starts applied (global-setup migrated to latest).
expect(await tableExists(db)).toBe(true);
expect(await uniqueIndexExists(db)).toBe(true);
await down(db);
expect(await tableExists(db)).toBe(false);
expect(await uniqueIndexExists(db)).toBe(false);
await up(db);
expect(await tableExists(db)).toBe(true);
expect(await uniqueIndexExists(db)).toBe(true);
// up() is idempotent (ifNotExists) — a second run is a harmless no-op.
await expect(up(db)).resolves.not.toThrow();
expect(await tableExists(db)).toBe(true);
});
});
@@ -0,0 +1,137 @@
import { randomUUID } from 'node:crypto';
import { Kysely } from 'kysely';
import { PageHistoryRepo } from '../../src/database/repos/page/page-history.repo';
import { PageHistoryService } from '../../src/core/page/services/page-history.service';
import { computeWorkTime } from '../../src/core/page/work-time';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
createPage,
createUser,
createChat,
} from './db';
/**
* #395 real-Postgres coverage for the work-time timeline projection and the
* service that computes the estimate. The pure sessionizer is unit-tested
* exhaustively (compute-work-time.spec.ts); this asserts the SQL projection
* (right rows, ASC, no `content`) and that the service's numbers agree with the
* pure core over the exact rows the DB returns.
*/
describe('PageHistory work-time [integration]', () => {
let db: Kysely<any>;
let repo: PageHistoryRepo;
let service: PageHistoryService;
let workspaceId: string;
let spaceId: string;
let pageId: string;
let userId: string;
let chatId: string;
const MIN = 60 * 1000;
beforeAll(async () => {
db = getTestDb();
repo = new PageHistoryRepo(db as any);
service = new PageHistoryService(repo);
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
pageId = (await createPage(db, { workspaceId, spaceId })).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
async function insertHistory(rows: Array<{
createdAt: string;
source: string | null;
chat?: string | null;
kind?: string | null;
content?: unknown;
}>) {
for (const r of rows) {
await db
.insertInto('pageHistory')
.values({
id: randomUUID(),
pageId,
spaceId,
workspaceId,
title: 'x',
content: r.content ?? { type: 'doc', content: [] },
lastUpdatedById: userId,
lastUpdatedSource: r.source,
lastUpdatedAiChatId: r.chat ?? null,
kind: r.kind ?? null,
createdAt: new Date(r.createdAt),
})
.execute();
}
}
it('findTimelineByPageId projects the cheap columns, ASC, without content', async () => {
await insertHistory([
{ createdAt: '2026-07-04T19:54:00Z', source: 'user', kind: 'manual' },
{ createdAt: '2026-07-04T03:40:00Z', source: 'user', kind: null },
{ createdAt: '2026-07-04T15:43:00Z', source: 'agent', chat: chatId, kind: 'agent' },
]);
const timeline = await repo.findTimelineByPageId(pageId);
expect(timeline).toHaveLength(3);
// Sorted oldest → newest.
const times = timeline.map((r) => new Date(r.createdAt).getTime());
expect(times).toEqual([...times].sort((a, b) => a - b));
// Projection carries exactly the sessionizer's inputs, and NO content.
for (const row of timeline) {
expect(row).toHaveProperty('createdAt');
expect(row).toHaveProperty('lastUpdatedById');
expect(row).toHaveProperty('lastUpdatedSource');
expect(row).toHaveProperty('lastUpdatedAiChatId');
expect(row).toHaveProperty('kind');
expect(row).not.toHaveProperty('content');
}
// Agent row keeps its provenance.
const agent = timeline.find((r) => r.lastUpdatedSource === 'agent');
expect(agent?.lastUpdatedAiChatId).toBe(chatId);
});
it('service estimate matches the pure core and satisfies Σ perDay == workMs', async () => {
const rows = await repo.findTimelineByPageId(pageId);
const pure = computeWorkTime(rows);
const result = await service.computeWorkTime(pageId, 'UTC');
expect(result.workMs).toBe(pure.workMs);
expect(result.agentOnlyMs).toBe(pure.agentOnlyMs);
expect(result.tz).toBe('UTC');
expect(result.config.tGap).toBe(15 * MIN);
const sumActive = result.perDay.reduce((a, d) => a + d.activeMs, 0);
expect(sumActive).toBe(result.workMs);
// The 3 seeded rows sessionize into ≤ their span; not the naive span.
const naive =
new Date('2026-07-04T19:54:00Z').getTime() -
new Date('2026-07-04T03:40:00Z').getTime();
expect(result.workMs).toBeGreaterThan(0);
expect(result.workMs).toBeLessThan(naive);
});
it('an unknown timezone surfaces as a RangeError (controller maps to 400)', async () => {
await expect(
service.computeWorkTime(pageId, 'Not/AZone'),
).rejects.toBeInstanceOf(RangeError);
});
it('a page with no history → zeros, no days', async () => {
const emptyPage = (await createPage(db, { workspaceId, spaceId })).id;
const result = await service.computeWorkTime(emptyPage, 'UTC');
expect(result.workMs).toBe(0);
expect(result.agentOnlyMs).toBe(0);
expect(result.perDay).toEqual([]);
});
});
@@ -0,0 +1,612 @@
import { randomUUID } from 'node:crypto';
import { Kysely, sql } from 'kysely';
import { SearchService } from 'src/core/search/search.service';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
} from './db';
/**
* #529 Phase A the lexical overhaul, on the REAL migrated schema (ru_en config).
*
* Covers every acceptance criterion of the issue: RU+EN morphology + OR default,
* match=auto identifier routing, "phrase"/+/- operators, RRF ordering, exact
* permission-filtered total (fail-closed) + pagination, only-negation / garbage
* short-circuits, the A8 path fix, the response superset, and the RAG lockstep
* config (acceptance #13).
*
* The tsv column is populated by the pages_tsvector_trigger (now ru_en), so the
* FTS branch is exercised end to end.
*/
describe('SearchService #529 lexical overhaul [integration]', () => {
let db: Kysely<any>;
let workspaceId: string;
let spaceId: string;
async function insertPage(args: {
title: string;
textContent?: string;
parentPageId?: string | null;
spaceId?: string;
deletedAt?: Date | null;
}): Promise<string> {
const id = randomUUID();
await db
.insertInto('pages')
.values({
id,
slugId: `slug-${id.slice(0, 12)}`,
title: args.title,
textContent: args.textContent ?? null,
parentPageId: args.parentPageId ?? null,
spaceId: args.spaceId ?? spaceId,
workspaceId,
deletedAt: args.deletedAt ?? null,
})
.execute();
return id;
}
// Service wired to the real DB + real PageRepo (recursive descendants) with
// stubbed space-membership + permission repos so a test controls scope and the
// permission filter explicitly. `accessibleIds` (when set) is the KEEP list.
function buildService(opts?: {
userSpaceIds?: string[];
accessibleIds?: string[] | null;
filterThrows?: boolean;
}): SearchService {
const pageRepo = new PageRepo(db as any, null as any, null as any);
const spaceMemberRepo = {
getUserSpaceIds: async () => opts?.userSpaceIds ?? [spaceId],
};
const pagePermissionRepo = {
hasRestrictedAncestor: async () => false,
filterAccessiblePageIds: async ({ pageIds }: { pageIds: string[] }) => {
if (opts?.filterThrows) throw new Error('permission query failed');
return opts?.accessibleIds
? pageIds.filter((id) => opts.accessibleIds!.includes(id))
: pageIds;
},
};
return new SearchService(
db as any,
pageRepo as any,
{} as any,
spaceMemberRepo as any,
pagePermissionRepo as any,
);
}
const search = (service: SearchService, params: any) =>
service.searchPage(params, { userId: 'u-1', workspaceId }) as any;
beforeAll(async () => {
db = getTestDb();
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
// 1. RU morphology + OR: «ресторанов москвы» finds «ресторан в москве».
it('#1 russian morphology + OR: finds «ресторан в москве» for «ресторанов москвы»', async () => {
const page = await insertPage({
title: 'ресторан в москве',
textContent: 'Лучший ресторан столицы.',
});
const res = await search(buildService(), {
query: 'ресторанов москвы',
spaceId,
});
expect(res.items.map((i: any) => i.id)).toContain(page);
expect(res.total).toBeGreaterThanOrEqual(1);
});
// 2. OR non-empty: «Стамбул Роснефть».
it('#2 OR yields a hit when only one term matches', async () => {
const page = await insertPage({ title: 'Роснефть отчёт', textContent: 'x' });
const res = await search(buildService(), {
query: 'Стамбул Роснефть',
spaceId,
});
expect(res.items.map((i: any) => i.id)).toContain(page);
});
// 3. Multi-word OR: a page matching >=1 term is returned.
it('#3 «3D принтер» returns pages that matched at least one term', async () => {
const models = await insertPage({
title: 'Модели для печати 3D',
textContent: 'коллекция моделей',
});
const wish = await insertPage({
title: 'Хотеть напечатать на принтере',
textContent: 'очередь печати',
});
const res = await search(buildService(), { query: '3D принтер', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(models); // matched "3D"
expect(ids).toContain(wish); // matched "принтер"
// matchedTerms is populated per hit.
const hit = res.items.find((i: any) => i.id === wish);
expect(hit.matchedTerms).toContain('принтер');
});
// 4. match=auto FTS stemming: «печат» must NOT drag in «впечатления».
it('#4 «печат» (auto) matches «печать» but NOT «впечатления»', async () => {
const good = await insertPage({
title: 'Печать документов',
textContent: 'настройка печати',
});
const bad = await insertPage({
title: 'Впечатления от поездки',
textContent: 'много впечатлений',
});
const res = await search(buildService(), { query: 'печат', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(good);
expect(ids).not.toContain(bad);
});
// 5. Identifier → substring branch.
it('#5 `10.31.41` (auto→substring) finds the page with that IP', async () => {
const page = await insertPage({
title: 'Сетевой узел',
textContent: 'Адрес устройства: 10.31.41.7 в сети.',
});
const res = await search(buildService(), { query: '10.31.41', spaceId });
const hit = res.items.find((i: any) => i.id === page);
expect(hit).toBeDefined();
expect(hit.matchedFields).toContain('text');
});
// 6. +required / -excluded.
it('#6 `+кофейня -архив`: keeps «кофейня», drops pages with «архив»', async () => {
const keep = await insertPage({ title: 'Кофейня в центре', textContent: 'уют' });
const drop = await insertPage({
title: 'Кофейня старый архив',
textContent: 'архивные записи',
});
const res = await search(buildService(), { query: '+кофейня -архив', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(keep);
expect(ids).not.toContain(drop);
});
// 7. Phrase operator: only adjacent phrase hits survive.
it('#7 `+"воздушный шар" кофе`: every hit contains the adjacent phrase', async () => {
const adjacent = await insertPage({
title: 'Воздушный шар и кофе',
textContent: 'воздушный шар над городом, чашка кофе',
});
const nonAdjacent = await insertPage({
title: 'Красный воздушный большой шар',
textContent: 'воздушный красный шар и кофе рядом',
});
const res = await search(buildService(), {
query: '+"воздушный шар" кофе',
spaceId,
});
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(adjacent);
// The non-adjacent page (words separated) must NOT match the phrase.
expect(ids).not.toContain(nonAdjacent);
});
// 8. Pagination determinism + exact total.
it('#8 >50 matches: total>50, hasMore, and offset paginates without dupes', async () => {
const svc = buildService();
const created: string[] = [];
for (let i = 0; i < 60; i++) {
created.push(
await insertPage({
title: `паджинация запись ${i}`,
textContent: 'общий паджинационный маркер',
}),
);
}
const p1 = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 0,
});
expect(p1.total).toBeGreaterThanOrEqual(60);
expect(p1.hasMore).toBe(true);
const p2 = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 25,
});
const p3 = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 50,
});
const ids = [
...p1.items.map((i: any) => i.id),
...p2.items.map((i: any) => i.id),
...p3.items.map((i: any) => i.id),
];
// No duplicates across the three pages.
expect(new Set(ids).size).toBe(ids.length);
// Deterministic: same query twice → identical order.
const p1b = await search(svc, {
query: 'паджинационный',
spaceId,
limit: 25,
offset: 0,
});
expect(p1b.items.map((i: any) => i.id)).toEqual(p1.items.map((i: any) => i.id));
});
// 9. Only-negation.
it('#9 `-архив` only-negation: total 0, reason only-negation, no throw', async () => {
const res = await search(buildService(), { query: '-архив', spaceId });
expect(res.total).toBe(0);
expect(res.items).toEqual([]);
expect(res.query.parsed.reason).toBe('only-negation');
});
// 10. Garbage input.
it('#10 garbage `%` / `_` / empty: total 0, does not match everything', async () => {
await insertPage({ title: 'какая-то страница', textContent: 'текст' });
for (const q of ['%', '_', ' ', '%%__']) {
const res = await search(buildService(), { query: q, spaceId });
expect(res.total).toBe(0);
expect(res.items).toEqual([]);
}
});
// 11. Permission-filtered total (fail-closed) + mutation guard.
it('#11 a permission-hidden page is absent from items AND total', async () => {
const visible = await insertPage({
title: 'разрешённая пермишен-страница',
textContent: 'пермишенмаркер',
});
const hidden = await insertPage({
title: 'скрытая пермишен-страница',
textContent: 'пермишенмаркер',
});
// Filter keeps only the visible page.
const filtered = buildService({ accessibleIds: [visible] });
const res = await search(filtered, { query: 'пермишенмаркер', spaceId });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(visible);
expect(ids).not.toContain(hidden);
// total is the POST-permission count — the hidden page does not leak into it.
expect(res.total).toBe(1);
// MUTATION: disable the guard (passthrough) → the hidden page reappears in
// BOTH items and total. If this did NOT change, the guard is not load-bearing.
const open = buildService({ accessibleIds: null });
const res2 = await search(open, { query: 'пермишенмаркер', spaceId });
expect(res2.items.map((i: any) => i.id)).toContain(hidden);
expect(res2.total).toBe(2);
});
it('#11b a permission-query error PROPAGATES (fail-closed, never empty)', async () => {
await insertPage({ title: 'failclosed маркер', textContent: 'failclosedmarker' });
const svc = buildService({ filterThrows: true });
await expect(
search(svc, { query: 'failclosedmarker', spaceId }),
).rejects.toThrow(/permission query failed/);
});
// A8 path fix.
it('#11c path: a soft-deleted / cross-space ancestor title does not leak', async () => {
const otherSpace = (await createSpace(db, workspaceId)).id;
const root = await insertPage({ title: 'Живой корень' });
const deletedMid = await insertPage({
title: 'УдалённыйПредок',
parentPageId: root,
deletedAt: new Date(),
});
const leaf = await insertPage({
title: 'a8leaf уникальный',
parentPageId: deletedMid,
textContent: 'a8leafmarker',
});
const res = await search(buildService(), { query: 'a8leafmarker', spaceId });
const hit = res.items.find((i: any) => i.id === leaf);
expect(hit).toBeDefined();
// The walk stops at the deleted ancestor — no deleted title in the path.
expect(hit.path).not.toContain('УдалённыйПредок');
expect(hit.path).not.toContain('Живой корень');
// Cross-space parent must also not leak.
const foreignParent = await insertPage({
title: 'ЧужойСпейс',
spaceId: otherSpace,
});
const crossLeaf = await insertPage({
title: 'crossleaf узел',
parentPageId: foreignParent,
textContent: 'crossleafmarker',
});
const res2 = await search(buildService(), {
query: 'crossleafmarker',
spaceId,
});
const hit2 = res2.items.find((i: any) => i.id === crossLeaf);
expect(hit2).toBeDefined();
expect(hit2.path).not.toContain('ЧужойСпейс');
});
// 12. Web-UI superset.
it('#12 web path (no flags) returns the OR result with the icon/space/highlight superset', async () => {
const page = await insertPage({
title: 'веб суперсет страница',
textContent: 'суперсетмаркер контент',
});
const res = await search(buildService(), { query: 'суперсетмаркер', spaceId });
const hit = res.items.find((i: any) => i.id === page);
expect(hit).toBeDefined();
// Superset fields the web-UI relies on.
expect('icon' in hit).toBe(true);
expect('space' in hit).toBe(true);
expect('highlight' in hit).toBe(true);
expect('rank' in hit).toBe(true);
// Plus the new fields.
expect('path' in hit).toBe(true);
expect('snippet' in hit).toBe(true);
expect('score' in hit).toBe(true);
// FTS hit carries a non-null rank + highlight.
expect(hit.rank).not.toBeNull();
});
// 13. RAG lockstep: the page_embeddings.fts generated column is ru_en.
it('#13 page_embeddings.fts uses ru_en (cyrillic stemming) — RAG lockstep', async () => {
const pageId = await insertPage({
title: 'rag страница',
textContent: 'ресторанов много',
});
// Insert a chunk row; the generated fts column is computed by Postgres.
await sql`
INSERT INTO page_embeddings
(id, page_id, workspace_id, space_id, attachment_id, chunk_index,
chunk_start, chunk_length, content, model_name, model_dimensions, embedding)
VALUES
(${randomUUID()}, ${pageId}, ${workspaceId}, ${spaceId}, NULL, 0,
0, 20, ${'ресторанов москвы много'}, 'test-model', 3, '[0.1,0.2,0.3]'::vector)
`.execute(db);
// A ru_en query stems «москва» → «москв», matching the stored «москвы».
// Under the old `english` config the cyrillic word would not stem and this
// inflected-form query would miss — so this asserts the ru_en lockstep.
const row = await sql<{ m: boolean }>`
SELECT fts @@ to_tsquery('ru_en', f_unaccent('москва')) AS m
FROM page_embeddings WHERE page_id = ${pageId}
`.execute(db);
expect(row.rows[0].m).toBe(true);
});
// W4 — substring-tier dominance under RRF: title-exact (tier 3) > title-
// substring (tier 2) > text-only (tier 1). The engine encodes this via
// sub_tier DESC → rn_sub → RRF; no test asserted the end-to-end ordering, so
// this restores that guarantee. match:'substring' routes the term to the
// substring branch (no FTS leg), so sub_tier alone drives the order.
it('W4 tier dominance: title-exact > title-substring > text-only', async () => {
const exact = await insertPage({ title: 'tierdomxyz' }); // tier 3
const titleSub = await insertPage({
title: 'prefix tierdomxyz suffix', // tier 2
});
const textOnly = await insertPage({
title: 'w4 unrelated heading',
textContent: 'body has tierdomxyz here', // tier 1
});
const res = await search(buildService(), {
query: 'tierdomxyz',
match: 'substring',
spaceId,
});
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(exact);
expect(ids).toContain(titleSub);
expect(ids).toContain(textOnly);
// Strict tier order.
expect(ids.indexOf(exact)).toBeLessThan(ids.indexOf(titleSub));
expect(ids.indexOf(titleSub)).toBeLessThan(ids.indexOf(textOnly));
});
// S3 — an exact-title hit must NOT be lost when the match set exceeds
// CANDIDATE_CAP: it ranks first under RRF (tier 3) so it lands in the reachable
// window even with a tiny cap. Restores a guarantee the old lookup suite gave.
it('S3 exact-title survives the CANDIDATE_CAP window', async () => {
const exact = await insertPage({ title: 'capmarkerxyz' }); // tier 3
for (let i = 0; i < 6; i++) {
await insertPage({
title: `cap filler ${i}`,
textContent: 'noise capmarkerxyz noise', // tier 1
});
}
process.env.SEARCH_CANDIDATE_CAP = '2';
try {
const res = await search(buildService(), {
query: 'capmarkerxyz',
match: 'substring',
spaceId,
limit: 25,
});
// More matches than the cap → truncated, but the exact-title survives.
expect(res.total).toBeGreaterThan(2);
expect(res.truncatedAtCap).toBe(true);
expect(res.items.length).toBe(2); // the reachable window
expect(res.items.map((i: any) => i.id)).toContain(exact);
} finally {
delete process.env.SEARCH_CANDIDATE_CAP;
}
});
// S1 — a required-only query (`+term`, no bare positive) really matches, so its
// hits must report matchedFields / rank / highlight, not [] / null. Before the
// fix these detail exprs were built from parsed.positive only.
it('S1 required-only query populates matchedFields + rank on a title match', async () => {
const page = await insertPage({
title: 'Кофейня s1маркер центр',
textContent: 'обычный текст без ключевого слова',
});
const res = await search(buildService(), { query: '+кофейня', spaceId });
const hit = res.items.find((i: any) => i.id === page);
expect(hit).toBeDefined();
// The title matches the required term → matchedFields includes 'title'.
expect(hit.matchedFields).toContain('title');
// FTS rank is populated (was null before the S1 fix).
expect(hit.rank).not.toBeNull();
// matchedTerms already echoed the required term; still true.
expect(hit.matchedTerms).toContain('кофейня');
});
// F1 — stack-depth guard: a pasted text block (thousands of FTS terms) used to
// nest the combined tsquery so deep that Postgres raised `stack depth limit
// exceeded` → HTTP 500 for the caller. The parser now caps terms at
// MAX_PARSED_TERMS, so a huge query returns 200 and the leading (in-cap) term
// still matches. Mutation: remove the cap → this reddens (500 / stack depth).
it('F1 a huge multi-term query returns 200, not a stack-depth 500', async () => {
const page = await insertPage({
title: 'qfonemarker заголовок',
textContent: 'тело страницы',
});
// Purely-alphabetic filler words → FTS branch (the branch that nests in
// combineTsq). A digit-bearing token would route to substring and not nest.
const alphaWord = (i: number): string => {
let s = '';
let n = i + 1;
while (n > 0) {
s = String.fromCharCode(97 + (n % 26)) + s;
n = Math.floor(n / 26);
}
return 'q' + s;
};
const words = [
'qfonemarker',
...Array.from({ length: 5000 }, (_, i) => alphaWord(i)),
];
const res = await search(buildService(), {
query: words.join(' '),
spaceId,
});
// Bounded nesting → no 500; the leading in-cap term still recalls the page.
expect(res.items.map((i: any) => i.id)).toContain(page);
});
// F2 — titleOnly leak-guard (restores coverage the deleted lookup spec gave).
// A term present ONLY in text_content must NOT match under titleOnly, and a
// title hit must carry NO body snippet. Uses match:'substring' because titleOnly
// gates the substring branch (the FTS `pages.tsv` already spans title+body).
it('F2 titleOnly does not match text_content and yields no body snippet', async () => {
// Marker lives only in the body, never in the title.
const bodyOnly = await insertPage({
title: 'нейтральный заголовок f2',
textContent: 'секрет titleonlybody конец',
});
const res = await search(buildService(), {
query: 'titleonlybody',
match: 'substring',
spaceId,
titleOnly: true,
});
// titleOnly must NOT leak a body-substring match.
expect(res.items.map((i: any) => i.id)).not.toContain(bodyOnly);
// Sanity: WITHOUT titleOnly the same term DOES find it via the body.
const resOpen = await search(buildService(), {
query: 'titleonlybody',
match: 'substring',
spaceId,
});
expect(resOpen.items.map((i: any) => i.id)).toContain(bodyOnly);
// A page that hits on its TITLE: the snippet must be empty (no body leaks in).
const titleHit = await insertPage({
title: 'titleonlytitle страница',
textContent: 'какой-то текст тела здесь для сниппета',
});
const res2 = await search(buildService(), {
query: 'titleonlytitle',
match: 'substring',
spaceId,
titleOnly: true,
});
const hit = res2.items.find((i: any) => i.id === titleHit);
expect(hit).toBeDefined();
expect(hit.snippet).toBe('');
});
// F3 — parentPageId subtree scoping (restores coverage the deleted lookup spec
// gave). Two sibling subtrees share one term; scoping to ONE root returns only
// that subtree's hits INCLUDING the root/parent page itself, and NONE of the
// sibling subtree. Mutation: drop the ANY(descendantIds) filter → this reddens.
it('F3 parentPageId scopes to one subtree incl. the parent, excludes siblings', async () => {
const rootA = await insertPage({
title: 'subtreeA корень',
textContent: 'f3marker в корне A',
});
const childA = await insertPage({
title: 'subtreeA потомок',
textContent: 'f3marker в потомке A',
parentPageId: rootA,
});
const rootB = await insertPage({
title: 'subtreeB корень',
textContent: 'f3marker в корне B',
});
const childB = await insertPage({
title: 'subtreeB потомок',
textContent: 'f3marker в потомке B',
parentPageId: rootB,
});
const res = await search(buildService(), {
query: 'f3marker',
spaceId,
parentPageId: rootA,
});
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(rootA); // the parent/root page itself
expect(ids).toContain(childA);
expect(ids).not.toContain(rootB); // sibling subtree cut off
expect(ids).not.toContain(childB);
});
// F4 — positive path + snippet content asserts (the #11c test only asserts what
// must NOT be in path). (a) a nested hit's path is root→parent ordered and a
// root-level hit's path is []; (b) a text-body hit returns a NON-empty snippet.
it('F4 path is root→parent ordered ([] at root) and body hits carry a snippet', async () => {
const root = await insertPage({ title: 'F4Root' });
const parent = await insertPage({ title: 'F4Parent', parentPageId: root });
const leaf = await insertPage({
title: 'f4leaf лист',
parentPageId: parent,
textContent: 'f4leafmarker тело с содержимым для сниппета',
});
const res = await search(buildService(), {
query: 'f4leafmarker',
spaceId,
});
const hit = res.items.find((i: any) => i.id === leaf);
expect(hit).toBeDefined();
// Positive ancestry: root → direct parent, hit's own title excluded.
expect(hit.path).toEqual(['F4Root', 'F4Parent']);
// Snippet content: a text-body hit returns a non-empty snippet with the term.
expect(hit.snippet.length).toBeGreaterThan(0);
expect(hit.snippet).toContain('f4leafmarker');
// A root-level hit (no parent) → empty path.
const rootHit = await insertPage({
title: 'f4rootlevel уникальный',
textContent: 'f4rootmarker в теле',
});
const res2 = await search(buildService(), {
query: 'f4rootmarker',
spaceId,
});
const hit2 = res2.items.find((i: any) => i.id === rootHit);
expect(hit2).toBeDefined();
expect(hit2.path).toEqual([]);
});
});
@@ -1,462 +0,0 @@
import { randomUUID } from 'node:crypto';
import { Kysely } from 'kysely';
import { SearchService } from 'src/core/search/search.service';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
} from './db';
/**
* #443 agent-lookup search mode, acceptance on the REAL DB schema.
*
* Exercises SearchService.searchPage(..., { substring: true }) against a
* migrated Postgres: substring matching of technical tokens the FTS tokenizer
* mangles (backup-srv.local, 10.0.12.5, WB-MGE-30D86B, "Теги: Docker"), the
* populated path + snippet, parentPageId subtree scoping, titleOnly, the empty
* result, LIKE-metacharacter escaping (`%`/`_` must NOT match everything), the
* permission post-filter applied BEFORE the limit, and the web-UI path staying
* on the legacy FTS shape when `substring` is absent.
*
* The tsv column is populated by the pages_tsvector_trigger on insert, so the
* FTS branch is exercised too.
*/
describe('SearchService agent-lookup mode [integration]', () => {
let db: Kysely<any>;
let service: SearchService;
let workspaceId: string;
let spaceId: string;
// Direct page insert (the shared createPage seeder omits text_content /
// parent_page_id, both of which this mode depends on). Returns the id.
async function insertPage(args: {
title: string;
textContent?: string;
parentPageId?: string | null;
spaceId?: string;
}): Promise<string> {
const id = randomUUID();
await db
.insertInto('pages')
.values({
id,
slugId: `slug-${id.slice(0, 12)}`,
title: args.title,
textContent: args.textContent ?? null,
parentPageId: args.parentPageId ?? null,
spaceId: args.spaceId ?? spaceId,
workspaceId,
})
.execute();
return id;
}
// Build a SearchService wired to the real DB + a real PageRepo (only its
// recursive-descendants method is used by this mode, and it needs only `db`),
// with lightweight stubs for the space-membership and permission repos so a
// test can drive scope + the permission post-filter explicitly.
function buildService(opts?: {
userSpaceIds?: string[];
// ids to KEEP after the permission post-filter; undefined = keep all.
accessibleIds?: string[];
}): SearchService {
const pageRepo = new PageRepo(db as any, null as any, null as any);
const spaceMemberRepo = {
getUserSpaceIds: async () => opts?.userSpaceIds ?? [spaceId],
};
const pagePermissionRepo = {
filterAccessiblePageIds: async ({ pageIds }: { pageIds: string[] }) =>
opts?.accessibleIds
? pageIds.filter((id) => opts.accessibleIds!.includes(id))
: pageIds,
};
return new SearchService(
db as any,
pageRepo as any,
{} as any, // shareRepo — unused by the lookup path
spaceMemberRepo as any,
pagePermissionRepo as any,
);
}
beforeAll(async () => {
db = getTestDb();
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
service = buildService();
});
afterAll(async () => {
await destroyTestDb();
});
it('finds `backup-srv.local` by the fragment `srv.local`', async () => {
const pageId = await insertPage({
title: 'backup-srv.local',
textContent: 'A backup server node.',
});
const { items } = (await service.searchPage(
{ query: 'srv.local', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
expect(items.map((i: any) => i.id)).toContain(pageId);
const hit = items.find((i: any) => i.id === pageId);
expect(hit.title).toBe('backup-srv.local');
// slugId must never be part of the server response shape.
expect('slugId' in hit).toBe(true); // server carries it; MCP strips it
});
it('finds a page whose TEXT contains `10.0.12.5` by the fragment `10.0.12` (empty-tsquery case)', async () => {
const pageId = await insertPage({
title: 'Server inventory',
textContent: 'The backup box lives at IP: 10.0.12.5. Debian 12, backups.',
});
const { items } = (await service.searchPage(
{ query: '10.0.12', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// The windowed snippet must include the matched text.
expect(hit.snippet).toContain('10.0.12.5');
});
it('finds `WB-MGE-30D86B` (alphanumeric token with dashes) by title', async () => {
const pageId = await insertPage({
title: 'WB-MGE-30D86B',
textContent: 'Device page.',
});
const { items } = (await service.searchPage(
{ query: 'WB-MGE-30D86B', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// Exact title match → top tier (TITLE_EXACT=3) → score in [0.75, 1].
expect(hit.score).toBeGreaterThanOrEqual(0.75);
// And it is the top-ranked hit of its own result set.
expect(items[0].id).toBe(pageId);
});
it('finds every page whose text literally contains `Теги: Docker`', async () => {
const a = await insertPage({
title: 'Container host A',
textContent: 'Some notes.\nТеги: Docker, compose\nmore.',
});
const b = await insertPage({
title: 'Container host B',
textContent: 'Prelude.\nТеги: Docker\nepilogue.',
});
const noise = await insertPage({
title: 'Unrelated',
textContent: 'Теги: Kubernetes',
});
const { items } = (await service.searchPage(
{ query: 'Теги: Docker', spaceId, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(a);
expect(ids).toContain(b);
expect(ids).not.toContain(noise);
});
it('populates a non-empty `path` for a nested hit and `[]` for a root hit', async () => {
const root = await insertPage({ title: 'Infrastructure' });
const mid = await insertPage({ title: 'Datacenter A', parentPageId: root });
const leaf = await insertPage({
title: 'unique-nested-host',
parentPageId: mid,
});
const { items } = (await service.searchPage(
{ query: 'unique-nested-host', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === leaf);
expect(hit.path).toEqual(['Infrastructure', 'Datacenter A']);
const rootHits = (await service.searchPage(
{ query: 'Infrastructure', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const rootHit = rootHits.items.find((i: any) => i.id === root);
expect(rootHit.path).toEqual([]);
});
it('scopes to a subtree with parentPageId (cutting off sibling branches)', async () => {
const branchA = await insertPage({ title: 'BranchA-root' });
const inA = await insertPage({
title: 'scoped-target-xyz',
parentPageId: branchA,
});
const branchB = await insertPage({ title: 'BranchB-root' });
const inB = await insertPage({
title: 'scoped-target-xyz',
parentPageId: branchB,
});
const { items } = (await service.searchPage(
{
query: 'scoped-target-xyz',
spaceId,
substring: true,
parentPageId: branchA,
} as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(inA);
expect(ids).not.toContain(inB);
});
it('includes the parent page itself in the parentPageId subtree', async () => {
const parent = await insertPage({ title: 'self-included-parent' });
await insertPage({ title: 'child-of-self', parentPageId: parent });
const { items } = (await service.searchPage(
{
query: 'self-included-parent',
spaceId,
substring: true,
parentPageId: parent,
} as any,
{ workspaceId },
)) as any;
expect(items.map((i: any) => i.id)).toContain(parent);
});
it('titleOnly does NOT match on text_content', async () => {
const pageId = await insertPage({
title: 'Plain title',
textContent: 'body mentions the-secret-token here',
});
const withText = (await service.searchPage(
{ query: 'the-secret-token', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
expect(withText.items.map((i: any) => i.id)).toContain(pageId);
const titleOnly = (await service.searchPage(
{
query: 'the-secret-token',
spaceId,
substring: true,
titleOnly: true,
} as any,
{ workspaceId },
)) as any;
expect(titleOnly.items.map((i: any) => i.id)).not.toContain(pageId);
});
// #443 Fix #1 regression: f_unaccent is NOT length-preserving, so an
// expanding char (ß→ss, …→...) BEFORE the match shifted the strpos position
// relative to the ORIGINAL text and the snippet slice ran past end → empty.
// The position and the slice now share the LOWER(f_unaccent(...)) space, so
// the window is aligned and always contains the matched (unaccented) token.
it('returns a populated snippet when an unaccent-EXPANDING char precedes the match', async () => {
// 300 × `ß` (each f_unaccent-expands to `ss`) before the needle. Under the
// old code strpos returned a position ~593 in the expanded space but the
// slice ran over the ORIGINAL (~360 char) text → empty snippet, match lost.
const prefix = 'ß'.repeat(300);
const pageId = await insertPage({
title: 'Expanding-unaccent page',
textContent: `${prefix} needle-token-xyz trailing.`,
});
const { items } = (await service.searchPage(
{ query: 'needle-token-xyz', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// Snippet must be non-empty AND contain the matched token (unaccented form).
expect(hit.snippet.length).toBeGreaterThan(0);
expect(hit.snippet).toContain('needle-token-xyz');
});
// #443 Fix #2 regression: >200 matching pages for a broad substring, with
// exactly ONE exact-title hit. Without an ORDER BY on the 200-cap the exact
// hit could be among the arbitrarily-dropped rows; the ORDER BY keeps the
// strongest candidates so it must survive the cap and rank at the top.
it('keeps an exact-title hit through the 200-cap on a >200-row match set', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
// 250 low-tier TEXT hits: the shared substring `capword` appears only in the
// body, never the title, so each is a TEXT-tier match (weakest tier).
for (let i = 0; i < 250; i++) {
await insertPage({
title: `filler-page-${i}`,
textContent: `body contains capword here #${i}`,
spaceId: isoSpace,
});
}
// Exactly one EXACT-title hit for the same query token.
const exact = await insertPage({
title: 'capword',
textContent: 'unrelated body text',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: 'capword', spaceId: isoSpace, substring: true, limit: 10 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
// The exact-title hit must survive the 200-cap and appear in the top `limit`.
expect(ids).toContain(exact);
// And, being TITLE_EXACT, it must be the single strongest hit.
expect(items[0].id).toBe(exact);
});
// #443 Fix #3: titleOnly matches only the title, so it must not leak the page
// body as the snippet (the old "first 300 chars of text_content" fallback).
it('titleOnly does NOT return a text-body snippet', async () => {
const pageId = await insertPage({
title: 'titleonly-snippet-page',
textContent: 'SECRET-BODY-CONTENT-NOT-IN-TITLE that must not leak.',
});
const { items } = (await service.searchPage(
{
query: 'titleonly-snippet-page',
spaceId,
substring: true,
titleOnly: true,
} as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// The body text must not appear in the snippet; titleOnly → empty snippet.
expect(hit.snippet).not.toContain('SECRET-BODY-CONTENT-NOT-IN-TITLE');
expect(hit.snippet).toBe('');
});
it('returns [] (not an error) for a query that matches nothing', async () => {
const { items } = (await service.searchPage(
{
query: 'zzz-no-such-string-anywhere-42',
spaceId,
substring: true,
} as any,
{ workspaceId },
)) as any;
expect(items).toEqual([]);
});
it('a `%` query does NOT match everything (LIKE metacharacter escaped)', async () => {
// Fresh space so we can assert on total counts without cross-test noise.
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
await insertPage({ title: 'alpha', spaceId: isoSpace });
await insertPage({ title: 'beta', spaceId: isoSpace });
const literal = await insertPage({
title: '100%-coverage',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: '%', spaceId: isoSpace, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
// `%` is a literal → matches only the page that actually contains '%'.
expect(ids).toContain(literal);
expect(ids).not.toContain(
items.find((i: any) => i.title === 'alpha')?.id,
);
expect(items.length).toBe(1);
});
it('an `_` query does NOT match everything (LIKE metacharacter escaped)', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
await insertPage({ title: 'gamma', spaceId: isoSpace });
const literal = await insertPage({
title: 'snake_case_name',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: '_', spaceId: isoSpace, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(literal);
expect(items.length).toBe(1);
});
it('applies the permission post-filter to the MERGED set BEFORE the limit', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const keep = await insertPage({
title: 'perm-visible-target',
spaceId: isoSpace,
});
const hidden = await insertPage({
title: 'perm-hidden-target',
spaceId: isoSpace,
});
// Authenticated (userId set) so the permission filter runs; only `keep` is
// accessible. limit 1 must NOT be able to select `hidden`.
const svc = buildService({
userSpaceIds: [isoSpace],
accessibleIds: [keep],
});
const { items } = (await svc.searchPage(
{
query: 'perm-',
spaceId: isoSpace,
substring: true,
limit: 1,
} as any,
{ userId: 'user-1', workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(keep);
expect(ids).not.toContain(hidden);
});
it('web-UI path (no `substring` flag) keeps the legacy FTS response shape', async () => {
await insertPage({
title: 'legacy shape page',
textContent: 'searchable legacyword content',
});
const { items } = (await service.searchPage(
{ query: 'legacyword', spaceId } as any,
{ userId: 'user-1', workspaceId },
)) as any;
// Legacy hits carry rank + highlight + space, and NO path/snippet/score.
const hit = items[0];
expect(hit).toBeDefined();
expect('rank' in hit).toBe(true);
expect('highlight' in hit).toBe(true);
expect('path' in hit).toBe(false);
expect('snippet' in hit).toBe(false);
expect('score' in hit).toBe(false);
});
});
@@ -0,0 +1,111 @@
import { Kysely, sql } from 'kysely';
import { getTestDb, destroyTestDb } from './db';
import * as migration from '../../src/database/migrations/20260707T130000-search-ru-en-config';
/**
* #529 A1 the ru_en config migration must be REVERSIBLE in the correct order:
* down() moves pages.tsv + page_embeddings.fts back to `english` BEFORE dropping
* the ru_en config (a generated column still depending on ru_en would block the
* DROP). This roundtrips down()up() on the already-migrated test DB and asserts
* the config, the pages trigger and the fts generated expression each flip and
* flip back the deploy-critical property.
*/
describe('search ru_en config migration [integration]', () => {
let db: Kysely<any>;
const configExists = async () => {
const r = await sql<{ n: number }>`
SELECT count(*)::int AS n FROM pg_ts_config WHERE cfgname = 'ru_en'
`.execute(db);
return r.rows[0].n > 0;
};
const ftsDef = async () => {
const r = await sql<{ def: string }>`
SELECT pg_get_expr(adbin, adrelid) AS def
FROM pg_attrdef d
JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attname = 'fts' AND a.attrelid = 'page_embeddings'::regclass
`.execute(db);
return r.rows[0]?.def ?? '';
};
const triggerSrc = async () => {
const r = await sql<{ src: string }>`
SELECT prosrc AS src FROM pg_proc WHERE proname = 'pages_tsvector_trigger'
`.execute(db);
return r.rows[0]?.src ?? '';
};
beforeAll(() => {
db = getTestDb();
});
afterAll(async () => {
// Restore the canonical ru_en state for any later suite regardless of where
// a test left off. up() is now safe on an existing config (ensureRuEnConfig
// no-ops) and re-asserts both stored sides to ru_en.
delete process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE;
await migration.up(db);
await destroyTestDb();
});
it('starts at ru_en (config present, trigger + fts on ru_en)', async () => {
expect(await configExists()).toBe(true);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
});
it('down() reverts tsv + fts to english THEN drops the config', async () => {
await migration.down(db);
expect(await configExists()).toBe(false);
expect(await ftsDef()).toContain('english');
expect(await ftsDef()).not.toContain('ru_en');
expect(await triggerSrc()).toContain('english');
});
it('up() re-applies ru_en cleanly (idempotent config create)', async () => {
await migration.up(db);
expect(await configExists()).toBe(true);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
});
// B1.1 — running up() a SECOND time on an already-migrated DB is a true no-op:
// it must NOT throw (the old drop-recreate-config would fail on the fts hard
// dependency) and must NOT re-rewrite the embeddings table — the fts column
// already references ru_en, so the at-target check short-circuits.
it('up() is idempotent: a 2nd run does not error and leaves ru_en intact', async () => {
await expect(migration.up(db)).resolves.toBeUndefined();
expect(await configExists()).toBe(true);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
});
// B1.2 — env-gate opt-out: with SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE=false the
// embeddings rewrite is SKIPPED (fts stays where it was) but pages.tsv still
// swaps inline, and the ru_en config is left in place (fts still depends on it).
// Runs down() as the vehicle: english is NOT the current fts config, so the
// skip path — not the at-target no-op — is exercised.
it('env-gate=false: skips the fts rewrite but still swaps pages.tsv', async () => {
// Precondition: fts + trigger on ru_en (from the prior test).
expect(await ftsDef()).toContain('ru_en');
process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE = 'false';
try {
await migration.down(db);
// fts rewrite skipped → still ru_en (the ACCESS EXCLUSIVE rewrite avoided).
expect(await ftsDef()).toContain('ru_en');
expect(await ftsDef()).not.toContain('english');
// pages.tsv trigger still swapped inline to english (gate is fts-only).
expect(await triggerSrc()).toContain('english');
// Config left in place because fts still references it (guarded drop).
expect(await configExists()).toBe(true);
} finally {
// Restore ru_en fully for the afterAll / later suites.
delete process.env.SEARCH_EMBEDDINGS_FTS_INLINE_REWRITE;
await migration.up(db);
expect(await ftsDef()).toContain('ru_en');
expect(await triggerSrc()).toContain('ru_en');
}
});
});
+4 -1
View File
@@ -118,7 +118,10 @@ All 41 tools, grouped by what you'd reach for them.
- **`getPage`** — A page's content as clean **Markdown** (canonical for text; drops only
block ids, resolved-comment anchors, and a fixed no-Markdown-representation attr set —
table spans/colwidth/background, indent, `callout.icon`, `orderedList.type`, and link
`internal`/`target`/`rel`/`class`; use `getPageJson` when you need those).
`internal`/`target`/`rel`/`class`; use `getPageJson` when you need those). Pass
`format:"text"` for a flat, deterministic plain-text rendering (one line per block,
marks dropped, stable `[image]`/`[table RxC]` placeholders) to machine-diff what you
wrote against what was stored.
- **`getPageJson`** — A page's **lossless ProseMirror/TipTap JSON**, including every
block's `attrs.id` and the `slugId` used in URLs. This is what the per-block editing
tools consume.
+4 -1
View File
@@ -123,7 +123,10 @@ Docmost-MCP не сочетают:
лишь id блоков, якоря разрешённых комментариев и фиксированный набор атрибутов без
markdown-представления — спаны/colwidth/фон ячеек таблиц, отступы (indent),
`callout.icon`, `orderedList.type` и `internal`/`target`/`rel`/`class` у ссылок;
используйте `getPageJson`, когда они нужны).
используйте `getPageJson`, когда они нужны). Передайте `format:"text"` для плоского
детерминированного plain-text рендера (одна строка на блок, маркировка убрана,
стабильные плейсхолдеры `[image]`/`[table RxC]`) — чтобы machine-diff'ом сверить то,
что вы записали, с тем, что сохранилось.
- **`getPageJson`** — **Lossless ProseMirror/TipTap JSON** страницы, включая `attrs.id`
каждого блока и `slugId`, используемый в URL. Именно его потребляют инструменты
поблочного редактирования.
+5 -1
View File
@@ -31,7 +31,11 @@ import { TransformsMixin, type ITransformsMixin } from "./client/transforms.js";
// existing importer (index.ts, http.ts, stdio.ts, the in-app host) keeps working
// with ZERO changes.
export type { DocmostMcpConfig, SandboxPut } from "./client/context.js";
export { formatDocmostAxiosError, assertFullUuid } from "./client/errors.js";
export {
formatDocmostAxiosError,
assertFullUuid,
formatSpaceNotAccessible,
} from "./client/errors.js";
// Branded canonical page-identity type (#435): the internal page UUID is a
// distinct nominal type so an unresolved raw/slug string can't be swapped into
+9 -4
View File
@@ -725,10 +725,15 @@ export function CommentsMixin<TBase extends GConstructor<DocmostClientContext>>(
// The subtree scope (parentPageId given) already INCLUDES the root node
// itself: /pages/tree seeds getPageAndDescendants with id = parentPageId, so
// no separate getPageRaw fetch for the parent is needed.
const { pages: pagesInScope, truncated } = await this.enumerateSpacePages(
spaceId,
parentPageId,
);
// #534: the enumerateSpacePages seed (`/pages/tree`) 404s for a bad or
// inaccessible spaceId; wrap it so that 404 becomes an actionable "spaceId
// not accessible" hint instead of the opaque "Space permissions not found".
// Only the whole enumeration is wrapped (the only 404 source here) — see the
// wrap-allowlist invariant on withSpaceAccessDiagnostics.
const { pages: pagesInScope, truncated } =
await this.withSpaceAccessDiagnostics(spaceId, "checkNewComments", () =>
this.enumerateSpacePages(spaceId, parentPageId),
);
// 2. Fetch comments for each page, keep ones created after since. Runs with
// bounded concurrency (#490) instead of one-at-a-time — the per-page reads are
+226 -6
View File
@@ -26,7 +26,10 @@ import {
import { withPageLock, isUuid } from "../lib/page-lock.js";
import type { PageId } from "../lib/page-id.js";
import { getCollabToken, performLogin } from "../lib/auth-utils.js";
import { formatDocmostAxiosError } from "./errors.js";
import {
formatDocmostAxiosError,
formatSpaceNotAccessible,
} from "./errors.js";
import { GetPageConversionCache } from "./getpage-cache.js";
// A generic mixin base constructor (issue #450). Each domain mixin is a factory
@@ -117,6 +120,36 @@ function readCollabTokenTtlMs(): number {
return Number.isFinite(raw) ? Math.max(0, raw) : 5 * 60 * 1000;
}
/**
* Accessible-space index cache TTL in milliseconds (issue #534). Read fresh from
* the environment on every access mirroring readCollabTokenTtlMs above so a
* test or a live rollback can change it without reloading the module.
*
* The index (see getAccessibleSpaceIndex) is fetched ONLY on the enrich-on-404
* slow path to turn an opaque "Space permissions not found" 404 into a factual
* "spaceId X is not among your accessible spaces" hint; a short TTL keeps a burst
* of failing tool calls from re-sweeping /spaces each time while never widening
* the permission-staleness window meaningfully. Default 60s. An EXPLICIT 0 (or
* negative) DISABLES the cache (exact fetch-per-enrichment). Unset/unparseable
* (NaN) falls back to the 60s default with the cache ON.
*/
function readSpacesCacheTtlMs(): number {
const raw = parseInt(process.env.MCP_SPACES_CACHE_TTL_MS ?? "", 10);
return Number.isFinite(raw) ? Math.max(0, raw) : 60000;
}
/**
* The set of spaces the current token can see, plus a `complete` flag that is
* false when the /spaces listing was truncated at the pagination ceiling. Used
* by the enrich-on-404 diagnostics: an authoritative membership test is only
* possible when `complete` is true (see withSpaceAccessDiagnostics).
*/
export type AccessibleSpaceIndex = {
ids: Set<string>;
spaces: { id: string; name: string }[];
complete: boolean;
};
export abstract class DocmostClientContext {
protected client: AxiosInstance;
protected token: string | null = null;
@@ -164,6 +197,27 @@ export abstract class DocmostClientContext {
// bypassed on a forced refresh (the 401/403 reauth path). null = no token yet.
protected collabTokenCache: { token: string; mintedAt: number } | null = null;
// Accessible-space index cache + single-flight (issue #534). TWO separate
// fields, mirroring loginPromise (in-flight dedup) vs collabTokenCache
// (persistent value):
// - spaceIndexCache: the last SUCCESSFULLY-FETCHED, COMPLETE index plus the
// wall-clock time it was fetched. Written ONLY from a resolved /spaces
// sweep whose result was complete (a truncated list is never cached, since
// it cannot answer "is this spaceId missing?"). Per-instance (a
// DocmostClient is built per user / per chat) so it can never leak across
// identities; invalidated on every identity change exactly like
// collabTokenCache (login() + the 401/403 reauth interceptor).
// - spaceIndexInFlight: dedups concurrent enrich-on-404 fetches into ONE
// /spaces sweep. CRITICAL INVARIANT: this promise is nulled in `.finally`
// on BOTH resolve AND reject — a rejected/settled promise is NEVER
// memoized, so a transient /spaces blip during one failed tool call cannot
// poison the diagnostics for the rest of the session.
protected spaceIndexCache: {
index: AccessibleSpaceIndex;
fetchedAt: number;
} | null = null;
protected spaceIndexInFlight: Promise<AccessibleSpaceIndex> | null = null;
// Content-addressed conversion cache for getPage (issue #479). Keyed on
// (canonical pageId, updatedAt, optionsHash) -> the converted Markdown, so a
// re-read of an UNCHANGED page skips the expensive convertProseMirrorToMarkdown
@@ -281,6 +335,9 @@ export abstract class DocmostClientContext {
// keep serving a collab token minted under the old one.
this.token = null;
this.collabTokenCache = null;
// #534: a new identity/login must not keep serving a space index
// computed under the old token (same reasoning as collabTokenCache).
this.spaceIndexCache = null;
delete this.client.defaults.headers.common["Authorization"];
try {
await this.login();
@@ -413,6 +470,8 @@ export abstract class DocmostClientContext {
// Identity (re)established: drop any collab token minted under a
// previous identity so the #435 cache can never outlive it.
this.collabTokenCache = null;
// #534: likewise drop the accessible-space index of the old identity.
this.spaceIndexCache = null;
this.client.defaults.headers.common["Authorization"] =
`Bearer ${token}`;
})
@@ -630,13 +689,34 @@ export abstract class DocmostClientContext {
}
/**
* Generic pagination handler for Docmost API endpoints
* Generic pagination handler for Docmost API endpoints. Thin wrapper over
* paginateAllWithMeta that discards the `truncated` flag the historical
* contract every caller (getSpaces, etc.) relies on. Callers that need to KNOW
* whether the result set was complete (e.g. #534's getAccessibleSpaceIndex,
* which must not assert "spaceId missing" against a truncated list) call
* paginateAllWithMeta directly.
*/
async paginateAll<T = any>(
endpoint: string,
basePayload: Record<string, any> = {},
limit: number = 100,
): Promise<T[]> {
return (await this.paginateAllWithMeta<T>(endpoint, basePayload, limit))
.items;
}
/**
* Generic pagination handler that ALSO surfaces whether the result was
* truncated at the MAX_PAGES ceiling. `paginateAll` swallows this flag (it only
* warns); callers that must distinguish "complete listing" from "gave up at the
* cap" use this overload. `truncated` is true iff the loop stopped at the
* ceiling while the server still reported more pages.
*/
async paginateAllWithMeta<T = any>(
endpoint: string,
basePayload: Record<string, any> = {},
limit: number = 100,
): Promise<{ items: T[]; truncated: boolean }> {
await this.ensureAuthenticated();
const clampedLimit = Math.max(1, Math.min(100, limit));
@@ -697,14 +777,154 @@ export abstract class DocmostClientContext {
);
}
return allItems;
return { items: allItems, truncated };
}
/**
* The set of spaces the current token can access (issue #534), fetched from the
* single source of truth the `/spaces` listing with a per-instance
* short-TTL cache and single-flight dedup. Used ONLY by the enrich-on-404 slow
* path (withSpaceAccessDiagnostics), so the happy path incurs ZERO extra
* requests.
*
* `complete` is `!truncated`: it is false when the /spaces listing was cut at
* the pagination ceiling. A truncated index can never authoritatively answer
* "is this spaceId missing?", so only a complete result is cached AND only a
* complete result is allowed to drive the "not accessible" rewrite.
*
* Cache/single-flight discipline (see the spaceIndexCache / spaceIndexInFlight
* field docs):
* - serve a fresh, complete cached index without any request;
* - otherwise collapse concurrent callers onto ONE in-flight /spaces sweep;
* - write the persistent cache ONLY from a resolved, complete fetch;
* - null the in-flight promise on BOTH resolve and reject (never memoize a
* rejected promise a transient /spaces failure must be retried fresh).
*/
async getAccessibleSpaceIndex(): Promise<AccessibleSpaceIndex> {
const ttl = readSpacesCacheTtlMs();
// Fast path: a still-fresh, complete cached index needs no request at all.
if (
ttl > 0 &&
this.spaceIndexCache &&
Date.now() - this.spaceIndexCache.fetchedAt < ttl
) {
return this.spaceIndexCache.index;
}
// Single-flight: a concurrent enrichment joins the in-flight sweep instead of
// issuing its own. (A settled/rejected promise is never left here — see the
// `.finally` below — so this only ever joins a genuinely in-progress fetch.)
if (this.spaceIndexInFlight) return this.spaceIndexInFlight;
const fetchPromise = (async (): Promise<AccessibleSpaceIndex> => {
const { items, truncated } = await this.paginateAllWithMeta("/spaces", {});
const spaces = items.map((s: any) => ({
id: s?.id,
name: s?.name,
}));
return {
ids: new Set(spaces.map((s) => s.id)),
spaces,
complete: !truncated,
};
})();
this.spaceIndexInFlight = fetchPromise
.then((index) => {
// Cache ONLY a complete result, and only while the cache is enabled.
if (ttl > 0 && index.complete) {
this.spaceIndexCache = { index, fetchedAt: Date.now() };
}
return index;
})
.finally(() => {
// CRITICAL (#534): clear the in-flight slot on BOTH resolve and reject.
// Nulling on reject too means a transient /spaces error is retried by the
// NEXT enrichment with a fresh fetch, never re-serving the rejection.
this.spaceIndexInFlight = null;
});
return this.spaceIndexInFlight;
}
/**
* Wrap a client method whose 404 means "the supplied spaceId is not accessible"
* and, ONLY on that 404, replace the opaque server text ("Space permissions not
* found") with a factual, actionable message naming the spaceId and the spaces
* the token can actually see (issue #534). A HINT layered on top of the
* backend, which stays authoritative so it FAILS OPEN on ANY uncertainty:
* every branch below that is not a confident "this spaceId is genuinely
* missing" rethrows the ORIGINAL server error unchanged. The happy path returns
* fn()'s value with zero extra requests.
*
* WRAP-ALLOWLIST INVARIANT (load-bearing read before wrapping a new method):
* among the currently wrapped tools a 404 comes ONLY from the spaceId
* membership / space-permissions check their pageId / rootPageId /
* parentPageId branches resolve to 403 or 200, NEVER 404. If a future change
* adds a `NotFoundException` to `/pages/tree`, `/pages/recent`,
* `/pages/sidebar-pages` or `/search` (e.g. "page not found"), this enrichment
* would MISATTRIBUTE that 404 to the spaceId. Re-audit the wrapped call before
* relying on this, and only wrap paths where the sole 404 cause is the space.
*/
protected async withSpaceAccessDiagnostics<T>(
spaceId: string,
mcpName: string,
fn: () => Promise<T>,
): Promise<T> {
try {
return await fn();
} catch (e) {
// Abort/cap wins FIRST and is detected by the SIGNAL FLAG, not e.name: a
// per-call cap may be an AbortSignal.timeout() (reason name "TimeoutError")
// or a custom reason, so `e.name === 'AbortError'` is NOT reliable (#534
// hole B). A stopped/capped turn must propagate its reason, never trigger a
// /spaces sweep or a rewrite.
if (this.toolAbortSignal?.aborted) throw e;
// Only a 404 is enrichable; any other status/shape is a different failure.
if (!(axios.isAxiosError(e) && e.response?.status === 404)) throw e;
let idx: AccessibleSpaceIndex;
try {
idx = await this.getAccessibleSpaceIndex();
} catch (fetchErr) {
// The /spaces sweep itself failed. If we were aborted mid-sweep,
// propagate the abort reason; otherwise FAIL OPEN with the ORIGINAL
// server error rather than a misleading "not found".
if (this.toolAbortSignal?.aborted) throw fetchErr;
if (process.env.DEBUG) {
console.error("space-diag: /spaces fetch failed:", fetchErr);
}
throw e;
}
// Fail open when the listing is incomplete (can't assert "missing") or when
// the spaceId IS present (the 404 is about something else, not the space).
if (!idx.complete) throw e;
if (idx.ids.has(spaceId)) throw e;
// Confident: the spaceId is well-formed but not among the accessible
// spaces. Replace the opaque server text with the actionable fact.
throw new Error(formatSpaceNotAccessible(mcpName, spaceId, idx.spaces));
}
}
/** Raw page info including the ProseMirror JSON content and slugId. */
async getPageRaw(pageId: string) {
/**
* Raw page info including the ProseMirror JSON content and slugId.
*
* With `format:"text"` (#502) the server instead renders `content` as a flat,
* deterministic text string (its `jsonToText` path the SAME serializer that
* feeds search), so the MCP text read reuses the server's ONE serializer
* rather than shipping a second one. Every other caller omits `format` and
* gets the JSON content unchanged.
*/
async getPageRaw(pageId: string, format?: "text") {
await this.ensureAuthenticated();
const response = await this.client.post("/pages/info", { pageId });
const body: Record<string, unknown> = { pageId };
if (format) body.format = format;
const response = await this.client.post("/pages/info", body);
return response.data?.data ?? response.data;
}
+54
View File
@@ -46,6 +46,60 @@ export function assertFullUuid(
}
}
// Max number of accessible spaces to enumerate inline in the "space not
// accessible" message (issue #534) before collapsing the rest into a "(+N ещё)"
// tail, so a workspace with many spaces cannot blow up the model context.
const SPACE_LIST_CAP = 10;
/**
* Compose the model-facing "spaceId is not accessible" message (issue #534).
* This is FACT text about the supplied spaceId that REPLACES the opaque server
* string ("Space permissions not found") on the enrich-on-404 path it names
* the exact bad id, lists the spaces the token can actually see (id + name, so
* the agent can copy the right id verbatim), and points at `listSpaces`.
*
* Deliberately Russian: like the other agent-facing tool guidance in this repo,
* this is the message the acting agent reads to self-correct.
*/
export function formatSpaceNotAccessible(
mcpName: string,
spaceId: string,
spaces: { id: string; name: string }[],
): string {
// No accessible spaces at all — a distinct diagnosis (token has no space
// access), not "you picked the wrong one from this list".
if (!Array.isArray(spaces) || spaces.length === 0) {
return `${mcpName}: spaceId "${spaceId}" недоступен, и доступных тебе спейсов нет — проверь доступ токена / вызови listSpaces.`;
}
const shown = spaces.slice(0, SPACE_LIST_CAP);
const remaining = spaces.length - shown.length;
const tail =
remaining > 0 ? ` (+${remaining} ещё, см. listSpaces)` : "";
// Cap the whole message at ERROR_MESSAGE_CAP (same budget as
// formatDocmostAxiosError) so ~10 long space names cannot blow up the model
// context. Truncate ONLY the interpolated space LIST — the fixed prefix (which
// carries the bad spaceId) and the fixed suffix (the "…из listSpaces"
// instruction) are always kept intact, so the actionable parts survive even
// when the list is trimmed.
const prefix = `${mcpName}: spaceId "${spaceId}" не найден среди доступных тебе спейсов. Доступные: `;
const suffix = ` — скопируй нужный id дословно из listSpaces.`;
let listed = shown.map((s) => `${s.id} (${s.name})`).join(", ") + tail;
const budget = ERROR_MESSAGE_CAP - prefix.length - suffix.length;
if (listed.length > budget) {
listed = listed.slice(0, Math.max(0, budget - 1)) + "…";
}
const message = `${prefix}${listed}${suffix}`;
// Unconditional final backstop (mirrors formatDocmostAxiosError): the list
// cap above assumes a well-formed prefix, but a pathologically long
// agent-supplied spaceId lives in the prefix and would otherwise blow past the
// budget. Cap the WHOLE message so nothing bloats the model context.
return message.length > ERROR_MESSAGE_CAP
? message.slice(0, ERROR_MESSAGE_CAP - 1) + "…"
: message;
}
// Keep ONLY the pathname of a request (no host, no query string, no fragment)
// so the message never leaks a host or query params. Resolves a relative
// config.url against config.baseURL, then discards everything but the path.
+27
View File
@@ -11,6 +11,7 @@ import {
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
savePageVersionRealtime,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
@@ -55,6 +56,7 @@ export interface IPagesMixin {
listPageHistory(pageId: string, cursor?: string): any;
getPageHistory(historyId: string): any;
restorePageVersion(historyId: string): any;
savePageVersion(pageId: string): any;
diffPageVersions(pageId: string, from?: string, to?: string): any;
}
@@ -93,6 +95,13 @@ export function PagesMixin<TBase extends GConstructor<DocmostClientContext>>(Bas
const buildForm = () => {
const form = new FormData();
form.append("spaceId", spaceId);
// #502: this is an AGENT-authored body (plain prose / config), so tell the
// server import path to run the markdown importer with the two layered
// extensions OFF — a `$…$` span stays literal text (real math via
// `update_page_json`) and a schemeless `www.host`/email is not autolinked
// (an explicit `https://…` still links). A human file upload never sends
// this field, so human imports keep math + autolink ON.
form.append("disableMarkdownExtensions", "true");
form.append("file", fileContent, {
filename: `${title || "import"}.md`,
contentType: "text/markdown",
@@ -580,6 +589,24 @@ export function PagesMixin<TBase extends GConstructor<DocmostClientContext>>(Bas
};
}
/**
* Save an intentional NAMED version of a page's CURRENT live content (#370).
* The write goes over the same agent-authenticated collab session content edits
* use, so the server derives kind='agent' from the signed actor. Resolves a
* SaveVersionResult: `{ saved:true, historyId, kind, alreadySaved }` on success,
* or `{ saved:false, skipped:true, reason }` when the server had nothing to pin
* (e.g. an empty page). A stale/missing pageId throws (not a benign skip).
*/
async savePageVersion(pageId: string) {
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
// Self-heal a rejected WS handshake once (#486): re-mint the collab token and
// retry, symmetric to the content-write path.
return this.writeWithCollabAuthRetry(collabToken, (token) =>
savePageVersionRealtime(pageUuid, token, this.apiUrl),
);
}
/**
* Diff two versions of a page and return a Docmost-equivalent change set.
* `from`/`to` each resolve to a ProseMirror doc:
+98 -24
View File
@@ -72,13 +72,13 @@ export interface IReadMixin {
getTree(spaceId: string, rootPageId?: string, maxDepth?: number): any;
getPageContext(pageId: string): any;
listSidebarPages(spaceId: string, pageId?: string): any;
getPage(pageId: string): any;
getPage(pageId: string, format?: "markdown" | "text"): any;
getPageJson(pageId: string): any;
getOutline(pageId: string): any;
getNode(pageId: string, nodeId: string, format?: "markdown" | "json"): any;
searchInPage(pageId: string, query: string, opts?: SearchOptions): any;
getTable(pageId: string, tableRef: string): any;
search(query: string, spaceId?: string, limit?: number, opts?: { parentPageId?: string; titleOnly?: boolean }): any;
search(query: string, spaceId?: string, limit?: number, opts?: { parentPageId?: string; titleOnly?: boolean; offset?: number; match?: "auto" | "word" | "prefix" | "substring" }): any;
}
export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IReadMixin> & TBase {
@@ -132,17 +132,29 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
// BFS hit its node cap) had no way to know pages were missing. Return the
// tree alongside the flag; the primary /pages/tree path is uncapped so this
// is false there.
const { pages, truncated } = await this.enumerateSpacePages(spaceId);
return { tree: buildPageTree(pages), truncated };
// #534: spaceId is required here; wrap so a bad-spaceId 404 (from the
// /pages/tree seed inside enumerateSpacePages) becomes an actionable hint.
return this.withSpaceAccessDiagnostics(spaceId, "listPages", async () => {
const { pages, truncated } = await this.enumerateSpacePages(spaceId);
return { tree: buildPageTree(pages), truncated };
});
}
const clampedLimit = Math.max(1, Math.min(100, limit));
const payload: Record<string, any> = { limit: clampedLimit, page: 1 };
if (spaceId) payload.spaceId = spaceId;
const response = await this.client.post("/pages/recent", payload);
const data = response.data;
const items = data.data?.items || data.items || [];
return items.map((page: any) => filterPage(page));
// #534: only the WITH-spaceId recent path can 404 on space access; wrap it so
// that 404 is rewritten. Without a spaceId there is no space to diagnose, so
// the wrapper is inert (run the request directly).
const runRecent = async () => {
const response = await this.client.post("/pages/recent", payload);
const data = response.data;
const items = data.data?.items || data.items || [];
return items.map((page: any) => filterPage(page));
};
return spaceId
? this.withSpaceAccessDiagnostics(spaceId, "listPages", runRecent)
: runRecent();
}
/**
@@ -174,8 +186,16 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
"getTree: spaceId is required (a page tree is scoped to one space).",
);
}
const { pages } = await this.enumerateSpacePages(spaceId, rootPageId);
return buildPageTree(pages, { shape: "getTree", maxDepth });
// #534: the 404 for a bad/inaccessible spaceId surfaces from the
// `/pages/tree` seeding step inside enumerateSpacePages (which is NOT in a
// try/catch of its own for that case) — wrap the whole body so it is caught
// and rewritten into an actionable "spaceId not accessible" message. Only the
// space membership check can 404 here (rootPageId 403/200), see the
// wrap-allowlist invariant on withSpaceAccessDiagnostics.
return this.withSpaceAccessDiagnostics(spaceId, "getTree", async () => {
const { pages } = await this.enumerateSpacePages(spaceId, rootPageId);
return buildPageTree(pages, { shape: "getTree", maxDepth });
});
}
/**
@@ -420,8 +440,34 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
return convertProseMirrorToMarkdown(content, options);
}
async getPage(pageId: string) {
async getPage(pageId: string, format: "markdown" | "text" = "markdown") {
await this.ensureAuthenticated();
// #502 `format:"text"`: a flat, deterministic, machine-diffable rendering
// (block-per-line, inline marks/comment anchors dropped, non-text nodes ->
// stable placeholders like `[image]` / `[table RxC]`). The server produces
// it via its `jsonToText` path (the SAME serializer that feeds search), so
// there is no second serializer here — we just request it and pass the
// string through. Distinct from the markdown default: no PM->markdown walk,
// no markdown conversion cache, and no `{{SUBPAGES}}` substitution (the text
// renderer emits no such placeholder). Use it to diff a page you wrote as a
// config/prose against what was stored.
if (format === "text") {
const textData = await this.getPageRaw(pageId, "text");
let subpages: any[] = [];
try {
subpages = await this.listSidebarPages(textData.spaceId, textData.id);
} catch (e: any) {
console.warn("Failed to fetch subpages:", e);
}
const textContent =
typeof textData.content === "string" ? textData.content : "";
return {
data: filterPage(textData, textContent, subpages),
success: true,
};
}
const resultData = await this.getPageRaw(pageId);
// Agent read: hide resolved-comment anchors so the agent sees only active
@@ -681,13 +727,19 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
query: string,
spaceId?: string,
limit?: number,
opts: { parentPageId?: string; titleOnly?: boolean } = {},
opts: {
parentPageId?: string;
titleOnly?: boolean;
offset?: number;
match?: "auto" | "word" | "prefix" | "substring";
} = {},
) {
await this.ensureAuthenticated();
// Opt into the #443 agent-lookup mode: `substring: true` turns on the hybrid
// substring + FTS branch that returns path + snippet + score. A stock
// upstream server strips these unknown DTO fields (whitelist:true) and
// silently degrades to plain FTS — see the tool-registration comment.
// #529 unified engine: the query is parsed SERVER-SIDE (operators
// "phrase"/+/-, OR default, RU+EN morphology, match=auto). We forward the RAW
// query plus flags. `substring: true` is kept as a back-compat hint for a
// stock upstream server (whitelist:true strips the unknown #529 fields and it
// degrades to plain FTS — see the tool-registration comment).
const payload: Record<string, any> = {
query,
spaceId,
@@ -695,22 +747,44 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
};
if (opts.parentPageId) payload.parentPageId = opts.parentPageId;
if (opts.titleOnly) payload.titleOnly = true;
if (opts.match) payload.match = opts.match;
// Clamp an optional caller-supplied limit into the lookup range (1..50)
// before forwarding; omit it when not provided so the server default applies.
if (limit !== undefined) {
payload.limit = Math.max(1, Math.min(50, limit));
}
const response = await this.client.post("/search", payload);
// Normalize both response shapes: bare array and paginated { items: [...] }
const data = response.data?.data;
const items = Array.isArray(data) ? data : data?.items || [];
const filteredItems = items.map((item: any) => filterSearchResult(item));
if (opts.offset !== undefined) {
payload.offset = Math.max(0, Math.floor(opts.offset));
}
return {
items: filteredItems,
success: response.data?.success || false,
const runSearch = async () => {
const response = await this.client.post("/search", payload);
// Normalize both response shapes: bare array and paginated { items: [...] }.
const data = response.data?.data;
const items = Array.isArray(data) ? data : data?.items || [];
const filteredItems = items.map((item: any) => filterSearchResult(item));
// Surface the #529 pagination envelope when present (a stock upstream has
// none — the fields are simply undefined and the caller sees just `items`).
const envelope = Array.isArray(data) ? undefined : data;
return {
items: filteredItems,
total: envelope?.total,
hasMore: envelope?.hasMore,
truncatedAtCap: envelope?.truncatedAtCap,
offset: envelope?.offset,
success: response.data?.success || false,
};
};
// #534: a search scoped to a spaceId 404s when that space is inaccessible;
// wrap only that case so the 404 becomes an actionable hint. A workspace-wide
// search (no spaceId) has no space to diagnose — run it directly (inert).
return spaceId
? this.withSpaceAccessDiagnostics(spaceId, "search", runSearch)
: runSearch();
}
}
+31 -9
View File
@@ -450,15 +450,22 @@ server.registerTool(
"search",
{
description:
"Find pages by a fragment of a technical string (hostnames, IPs, IDs " +
"like `srv.local`, `10.0.12`, `WB-MGE-30D86B`) — one call returns each " +
"hit's location (`path`: ancestor titles root→parent) and a `snippet` " +
"around the first match, so you rarely need a follow-up get_page. " +
"Matches substrings literally (dots/dashes/digits are not tokenized) as " +
"well as full-text. Returns `{ pageId, title, path, snippet, score }` " +
"sorted by `score` (a per-response relevance float).",
"Search pages across the wiki. OR by default with relevance ranking " +
"(RU+EN morphology): multi-word queries match ANY term, not all. " +
"Operators: \"exact phrase\" (adjacent words), +term (require), -term " +
"(exclude) — e.g. `+кофейня -архив`, `+\"воздушный шар\" кофе`. A leading " +
"-/+ is the operator; -,.,: INSIDE a token are literal (`WB-MGE-30D86B`, " +
"`10.0.12.5` stay one term). Technical fragments (hostnames, IPs, IDs) " +
"auto-match as substrings; words use full-text. Each hit returns its " +
"location (`path`: ancestor titles root→parent), a `snippet`, `score`, " +
"`matchedTerms` and `matchedFields`, so you rarely need a follow-up " +
"getPage. Paginate with limit + offset; the response carries " +
"`total` (exact, permission-filtered), `hasMore` and `truncatedAtCap`. " +
"NOTE: results past the relevance cap (~500) are unreachable by " +
"pagination — narrow the query (add terms / +required / a spaceId) " +
"instead when `truncatedAtCap` is true.",
inputSchema: {
query: z.string().min(1).describe("Search query"),
query: z.string().min(1).describe("Search query (supports \"phrase\", +require, -exclude)"),
spaceId: z
.string()
.optional()
@@ -473,6 +480,13 @@ server.registerTool(
.boolean()
.optional()
.describe("Match page titles only; skip page text"),
match: z
.enum(["auto", "word", "prefix", "substring"])
.optional()
.describe(
"Match mode (default auto: identifiers→substring, words→full-text). " +
"Override with word/prefix/substring.",
),
limit: z
.number()
.int()
@@ -480,12 +494,20 @@ server.registerTool(
.max(50)
.optional()
.describe("Max results to return (1-50, default 10)"),
offset: z
.number()
.int()
.min(0)
.optional()
.describe("Pagination offset (default 0); use with total/hasMore"),
},
},
async ({ query, spaceId, parentPageId, titleOnly, limit }) => {
async ({ query, spaceId, parentPageId, titleOnly, match, limit, offset }) => {
const result = await docmostClient.search(query, spaceId, limit, {
parentPageId,
titleOnly,
match,
offset,
});
return jsonContent(result);
},
+125
View File
@@ -139,7 +139,13 @@ export interface CollabProviderLike {
unsyncedChanges: number;
destroy(): void;
on(event: "unsyncedChanges", handler: (data: { number: number }) => void): void;
on(event: "stateless", handler: (data: { payload: string }) => void): void;
off(event: "unsyncedChanges", handler: (data: { number: number }) => void): void;
off(event: "stateless", handler: (data: { payload: string }) => void): void;
// Send a stateless (out-of-band, non-doc) message to the server. Used by the
// #370 explicit save-version path (payload `{type:'save-version'}`); the server
// replies with a broadcast `{type:'version.saved', …}` on the same channel.
sendStateless(payload: string): void;
}
/** The configuration object passed to the provider factory. */
@@ -564,6 +570,125 @@ export class CollabSession {
});
}
/**
* Send a stateless message over the live collaboration connection and resolve
* with the FIRST reply whose parsed payload satisfies `predicate`, rejecting on
* a bounded `timeoutMs`.
*
* Used by the #370 explicit save-version path: the client sends
* `{type:'save-version'}` and awaits the server's broadcast
* `{type:'version.saved', historyId, kind, alreadySaved}`. The stateless channel
* (not a REST read) is deliberate the server versions the LIVE in-memory ydoc,
* which the up-to-10s-stale page row would not yet reflect.
*
* `predicate` receives each incoming stateless message (already JSON-parsed) and
* returns the resolved value for a match or `undefined` to keep waiting; a
* malformed / unrelated payload is ignored. The listener is registered BEFORE
* the send so a fast reply is never missed.
*
* Lifecycle mirrors mutate(): it registers through `inflightReject` so a
* disconnect/close/auth-failure/teardown rejects THIS wait with the same
* connection-loss error, refuses to run on a non-ready session, fails fast on a
* concurrent in-flight op (the caller MUST hold the per-page lock), and re-arms
* the idle TTL (or self-destroys an ephemeral session) on completion.
*
* NOTE: the server broadcast carries no request-correlation id, so under a
* genuinely concurrent save on the SAME page (e.g. a human Cmd+S racing the
* agent) this resolves on whichever `version.saved` arrives first. The per-page
* lock serializes THIS process's saves; a cross-client race is inherent to the
* broadcast design and is benign (the result still describes a real save of this
* page's current content, and the server's save is promote-not-duplicate).
*/
sendStatelessAndAwait<T>(
payload: string,
predicate: (message: any) => T | undefined,
timeoutMs: number,
): Promise<T> {
// Belt-and-suspenders (acquire already validated): refuse to send on a
// session that is not in a live, synced, ready state.
if (
this.dead ||
this.state !== "ready" ||
this.connectionLost ||
!this.provider ||
this.provider.synced !== true
) {
return Promise.reject(
new Error("Collaboration session is not in a ready state"),
);
}
// Fail-fast on concurrent use: a second overlapping op would overwrite the
// first's inflightReject and hang it on disconnect (same guard as mutate).
if (this.inflightReject) {
return Promise.reject(
new Error(
"stateless op already in-flight; caller must serialize (hold the page lock)",
),
);
}
return new Promise<T>((resolve, reject) => {
let settled = false;
let timer: ReturnType<typeof setTimeout> | undefined;
let statelessHandler: ((data: { payload: string }) => void) | undefined;
const localFinish = (err: Error | null, value?: T) => {
if (settled) return;
settled = true;
if (timer) clearTimeout(timer);
if (statelessHandler && this.provider) {
try {
this.provider.off("stateless", statelessHandler);
} catch (e) {}
}
this.inflightReject = undefined;
if (err) reject(err);
else resolve(value as T);
// Post-settle lifecycle, identical to mutate's localFinish.
if (this.ephemeral) {
this.destroy("ephemeral op complete");
} else if (!this.dead) {
this.armIdle();
}
};
// Register so a disconnect/close/auth-failure/teardown rejects THIS op with
// the connection-loss error text (the `settled` guard makes a racing
// teardown + normal resolve safe — first one wins).
this.inflightReject = (e: Error) => localFinish(e);
// Register the reply listener BEFORE sending so a fast server broadcast is
// never missed.
statelessHandler = (data: { payload: string }) => {
if (settled) return;
let message: any;
try {
message = JSON.parse(data.payload);
} catch {
return; // unrelated / malformed stateless message — keep waiting
}
const matched = predicate(message);
if (matched !== undefined) localFinish(null, matched);
};
this.provider!.on("stateless", statelessHandler);
timer = setTimeout(() => {
localFinish(
new Error(
`Timeout waiting for a stateless reply from the collaboration server ${this.hint()}`,
),
);
}, timeoutMs);
try {
this.provider!.sendStateless(payload);
} catch (e) {
localFinish(e instanceof Error ? e : new Error(String(e)));
}
});
}
/** (Re)arm the idle TTL so the clock starts from the most recent activity. */
armIdle(): void {
if (this.dead || this.ephemeral) return;
+168 -2
View File
@@ -14,6 +14,7 @@ import {
markdownToProseMirror,
normalizeAgentMarkdown,
} from "@docmost/prosemirror-markdown";
import type { MarkdownImportOptions } from "@docmost/prosemirror-markdown";
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import type { PageId } from "./page-id.js";
@@ -111,16 +112,31 @@ global.WebSocket = WebSocket;
* horizontalRule the serializer emitted, and stripping it would silently drop the
* page's leading content (#493 review). The front-matter strip stays on the
* server FILE-import boundary only (`normalizeForeignMarkdown`).
*
* #502 IMPORTANT the two layered markdown extensions (`$$` math, schemeless
* fuzzy autolink) are NOT decided here; they are the CALLER's choice via
* `options`, because the two callers of this wrapper need OPPOSITE behavior:
* - AGENT-authored plain markdown (`updatePageMarkdown`) both OFF, so a
* `$$` config span stays literal and a bare `www.host` is not autolinked
* (real math is authored via `update_page_json`).
* - FULL-FILE round-trip import (`import_page_markdown`, #328 lossless)
* DEFAULTS (both ON), because the exporter serializes a math node as readable
* `$x^2$`; re-importing with math OFF would degrade it to literal text and
* BREAK the lossless exportimport pair.
* So `options` DEFAULTS to `undefined` the package importer's defaults (ON),
* which is the safe round-trip behavior; the agent-write caller opts OUT
* explicitly. (The fragment path `importMarkdownFragment` opts out on its own.)
*/
export async function markdownToProseMirrorCanonical(
markdownContent: string,
options?: MarkdownImportOptions,
): Promise<any> {
// #419: normalize + merge glyph-forked footnote definitions BEFORE
// canonicalizing, so the canonicalizer re-hangs references and drops the
// now-orphaned duplicate definitions.
return canonicalizeFootnotes(
normalizeAndMergeFootnotes(
await markdownToProseMirror(normalizeAgentMarkdown(markdownContent)),
await markdownToProseMirror(normalizeAgentMarkdown(markdownContent), options),
),
);
}
@@ -308,6 +324,146 @@ export async function mutatePageContent(
});
}
/**
* Stateless-channel message types for the #370 explicit save-version handshake.
* These MUST match the server constants in
* apps/server/src/collaboration/extensions/persistence.extension.ts
* (SAVE_VERSION_MESSAGE_TYPE and the VERSION_SAVED / VERSION_SKIPPED replies)
* the mcp package cannot import server code, so the literals are duplicated here.
* KEEP THE TWO SIDES IN SYNC: the server broadcasts exactly ONE terminal reply per
* handled save (`version.saved` for a real save/promote, `version.skipped` for a
* reachable no-op an empty page or a missing page row); the client must match
* BOTH or it waits out the ack timeout on the skip case and misreports a healthy
* server as unreachable (#370 F2).
*/
const SAVE_VERSION_MESSAGE_TYPE = "save-version";
const VERSION_SAVED_MESSAGE_TYPE = "version.saved";
const VERSION_SKIPPED_MESSAGE_TYPE = "version.skipped";
/**
* Bounded wait for the server's terminal reply after we ask it to save a version.
* The server flushes the live ydoc through its store path and writes a history row
* inside a DB transaction before broadcasting, so allow the same headroom as a
* persistence ack; reject (do NOT hang) past it. A timeout here means NO reply at
* all arrived the genuine "collab server unreachable/overloaded" case as
* distinct from a `version.skipped` reply, which resolves immediately.
*/
const SAVE_VERSION_ACK_TIMEOUT_MS = 20000;
/**
* The resolved outcome of an explicit save-version, surfaced to the tool caller.
* `saved:true` a version was created or promoted (historyId/kind/alreadySaved
* are present). `saved:false` the server had nothing to pin (a reachable no-op,
* e.g. an empty page); `skipped` + `reason` explain why. A page-not-found reply is
* NOT represented here it is surfaced as a thrown error (a bad/stale pageId).
*/
export interface SaveVersionResult {
saved: boolean;
historyId?: string;
kind?: string;
alreadySaved?: boolean;
skipped?: boolean;
reason?: string;
}
/**
* Parsed terminal reply the predicate hands back to savePageVersionRealtime. Kept
* internal: it carries the page-not-found case (mapped to a thrown error, never a
* returned result) that SaveVersionResult deliberately does not model.
*/
type SaveVersionReply =
| { outcome: "saved"; historyId: string; kind: string; alreadySaved: boolean }
| { outcome: "skipped"; reason: string };
/** Match the server's terminal reply (version.saved / version.skipped), ignoring
* any unrelated / cross-page stateless message so the wait is not resolved by
* noise. Returns `undefined` to keep waiting (#370 F1 predicate coverage). */
function matchSaveVersionReply(message: any): SaveVersionReply | undefined {
if (!message || typeof message !== "object") return undefined;
if (message.type === VERSION_SAVED_MESSAGE_TYPE) {
return {
outcome: "saved",
historyId: String(message.historyId),
kind: String(message.kind),
alreadySaved: !!message.alreadySaved,
};
}
if (message.type === VERSION_SKIPPED_MESSAGE_TYPE) {
return { outcome: "skipped", reason: String(message.reason ?? "unknown") };
}
return undefined;
}
/**
* Save an intentional version of a page's CURRENT live collaboration content
* (#370). Runs under the per-page lock, acquires the SAME cached CollabSession the
* content writes use (#400) so it authenticates with the caller's agent collab
* token, and the server derives kind='agent' from that signed actor then sends a
* `{type:'save-version'}` stateless message and awaits the server's terminal reply
* (`version.saved` or `version.skipped`) on the same channel.
*
* Deliberately does NOT read `pages.content` over REST: the versioned content is
* the live in-memory ydoc, which the debounced (up-to-10s-stale) page row would
* not yet reflect. The stateless round-trip is what makes the save exact.
*
* Outcomes:
* - a real save/promote `{ saved:true, historyId, kind, alreadySaved }`;
* - the server had nothing to pin (empty page) `{ saved:false, skipped:true,
* reason:'empty' }` — a clean, immediate no-op, NOT a stall;
* - the page row is gone (a stale/bad pageId) a thrown error, immediate and
* truthful (not the health-timeout path);
* - no reply within the timeout a thrown timeout error (the genuine "server
* unreachable/overloaded" case).
*
* On a TRANSPORT failure (timeout / disconnect) the session is destroyed so the
* next call reconnects fresh; a page-not-found is a healthy-connection terminal
* reply, so the session is left cached. A save is safe to retry the server
* promotes-not-duplicates an identical latest version so the caller (and its
* agent) may re-issue it without risking a duplicate heavy history row.
*/
export async function savePageVersionRealtime(
pageId: PageId,
collabToken: string,
baseUrl: string,
): Promise<SaveVersionResult> {
return withPageLock(pageId, async () => {
const session = await acquireCollabSession(pageId, collabToken, baseUrl);
let reply: SaveVersionReply;
try {
reply = await session.sendStatelessAndAwait<SaveVersionReply>(
JSON.stringify({ type: SAVE_VERSION_MESSAGE_TYPE }),
matchSaveVersionReply,
SAVE_VERSION_ACK_TIMEOUT_MS,
);
} catch (e) {
// TRANSPORT failure (no reply within the timeout, or a disconnect): drop the
// session so the next call reconnects fresh.
session.destroy("save-version failed");
throw e;
}
// A terminal reply arrived over a healthy connection — do NOT destroy the
// session; interpret the outcome.
if (reply.outcome === "skipped") {
if (reply.reason === "page-not-found") {
// A resolved pageId that the collab server no longer holds (deleted, or a
// stale id). Surface it immediately and truthfully, not as a health error.
throw new Error(
`savePageVersion: page ${pageId} was not found on the collaboration ` +
`server (it may have been deleted) — nothing was saved.`,
);
}
// Reachable benign no-op (e.g. an empty page): a clean result, not a throw.
return { saved: false, skipped: true, reason: reply.reason };
}
return {
saved: true,
historyId: reply.historyId,
kind: reply.kind,
alreadySaved: reply.alreadySaved,
};
});
}
/**
* Replace the live content of a page over the collaboration websocket.
* Accepts a ready ProseMirror JSON document; the caller controls whether
@@ -358,7 +514,17 @@ export async function updatePageContentRealtime(
): Promise<MutationResult> {
// PAGE write: canonicalize footnotes (markdown import builds the bottom list in
// definition order; numbering is reference-ordered).
const tiptapJson = await markdownToProseMirrorCanonical(markdownContent);
//
// #502: this is the AGENT-authored `updatePageMarkdown` body — plain prose /
// config — so the two layered markdown extensions are turned OFF: a `$…$` span
// stays literal text (real math via `update_page_json`) and a SCHEMELESS
// `www.host`/email is not autolinked (an explicit `https://…` still links).
// Contrast `import_page_markdown`, which keeps DEFAULTS for the #328 lossless
// round-trip.
const tiptapJson = await markdownToProseMirrorCanonical(markdownContent, {
parseMath: false,
fuzzyLinkify: false,
});
return await mutatePageContent(
pageId,
collabToken,
+38 -3
View File
@@ -1080,8 +1080,43 @@ export interface PreparedModel {
*/
export function prepareModel(inputXml: string): PreparedModel {
const rawModel = normalizeInput(inputXml);
const { cells, warnings } = lintModel(rawModel);
const modelXml = normalizeXml(rawModel);
// Auto-strip XML comments before linting. The model routinely inserts
// `<!-- ... -->` into the diagram XML despite the prohibition in the tool
// description; a hard [no-comments] lint failure would force it to regenerate
// the whole diagram (a wasted tool call). Comments carry no diagram semantics,
// so stripping them is always safe. The linter's no-comments rule stays as a
// defense-in-depth backstop for any path that reaches it without this strip.
//
// The strip is GATED on two conditions so the regex only ever targets real
// comment nodes:
// 1. Well-formedness. Well-formed XML forbids a bare `<` inside an attribute
// value (it must be entity-escaped, e.g. `&lt;!--`), so a literal `<!--`
// cannot hide in a value. On MALFORMED input (e.g. a raw unescaped `<!--`
// inside a value on the <mxGraphModel> create/update path, which
// normalizeInput passes through unparsed) we must NOT strip: truncating
// that value would silently corrupt the author's label. We leave it intact
// and let lintModel's value-escaping / well-formed-xml rules reject it so
// the author sees the real error.
// 2. No CDATA. A CDATA section is well-formed yet its text may contain a
// literal `<!-- ... -->` that is NOT a comment node; stripping it would
// silently delete author content. Real drawio labels live in `value=`
// attributes (CDATA impossible), so excluding CDATA is free on legitimate
// models; a CDATA-bearing model with a genuine comment then falls to the
// retained no-comments lint backstop (an explicit error) rather than being
// silently corrupted.
const { error: parseError } = parseXml(rawModel);
const hasCdata = rawModel.includes("<![CDATA[");
const commentMatches =
parseError === null && !hasCdata
? (rawModel.match(/<!--[\s\S]*?-->/g) ?? [])
: [];
const commentCount = commentMatches.length;
const strippedModel =
commentCount > 0 ? rawModel.replace(/<!--[\s\S]*?-->/g, "") : rawModel;
const stripWarnings =
commentCount > 0 ? [`stripped ${commentCount} XML comment(s)`] : [];
const { cells, warnings } = lintModel(strippedModel);
const modelXml = normalizeXml(strippedModel);
const bbox = computeBBox(cells);
const cellCount = cells.filter((c) => c.id !== "0" && c.id !== "1").length;
// Geometry quality warnings (non-blocking) are appended to any structural
@@ -1092,7 +1127,7 @@ export function prepareModel(inputXml: string): PreparedModel {
cells,
bbox,
cellCount,
warnings: [...warnings, ...quality],
warnings: [...stripWarnings, ...warnings, ...quality],
hash: mxHash(modelXml),
};
}
+11 -1
View File
@@ -135,7 +135,17 @@ export interface MarkdownFragment {
export async function importMarkdownFragment(
markdown: string,
): Promise<MarkdownFragment> {
const doc = await markdownToProseMirror(markdown);
// #502: the fragment path is an MCP agent WRITE (patch_node/insert_node
// markdown), so it uses the SAME extensions-OFF importer options as the
// full-page write (markdownToProseMirrorCanonical): a `$…$` span stays literal
// and a schemeless domain/email is not autolinked. This keeps a block written
// via markdown canonically identical to the same content in a full-page write
// (no "second canon"). Explicit `https://…` links and block structure are
// unaffected.
const doc = await markdownToProseMirror(markdown, {
parseMath: false,
fuzzyLinkify: false,
});
const content: any[] = Array.isArray(doc?.content) ? doc.content : [];
const blocks: any[] = [];

Some files were not shown because too many files have changed in this diff Show More