Compare commits

...

65 Commits

Author SHA1 Message Date
agent_vscode 834684c37a Merge branch 'develop' of https://gitea.vvzvlad.xyz/vvzvlad/gitmost into develop 2026-07-06 21:43:41 +03:00
vvzvlad 080d1b6051 Merge pull request 'feat(ai-chat): показывать текст запроса/аргументы в карточке вызова инструмента (#392)' (#393) from feat/392-tool-input-summary into develop
Reviewed-on: #393
2026-07-06 21:43:23 +03:00
agent_coder 7f88b0b441 test(ai-chat): pin toolInputSummary field priority + clamp boundary (#392)
Review round: add the two missing test-locks the reviewer asked for —
(1) priority order of PRIMARY_INPUT_FIELDS is now pinned (`{query,title}`
-> "Q", so a reordering breaks the test); (2) the clamp boundary is pinned
exactly (140 chars -> unchanged, no ellipsis; 141 -> 140 + "…"), catching
an off-by-one / `>` vs `>=` regression. Test-only, no production change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:40:03 +03:00
agent_vscode 8f7664eb04 feat(agent-roles): restrict when the researcher reuses the current page
The researcher now reuses the current/already-open document only when the
user explicitly asked for it, OR the page is empty/near-empty AND its title
matches the research topic; otherwise it creates a new document.

- Rewrite the reuse rule in the WHERE TO WRITE THE RESULT section
- Reword "Create this document" -> "Set up this document" so it fits both
  the create-new and reuse-current-empty cases
- Apply identically to bundles/research/ru.yaml and en.yaml
- Bump researcher version 3 -> 4 in index.yaml; refresh content-hashes.json

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:29:03 +03:00
agent_coder cb445f0966 feat(ai-chat): show tool-call arguments in the action-log card (#392)
For tools without a friendly label (esp. external MCP tools like
Search_web_search) the card showed a generic "Ran tool <name>" with no
sense of what was searched. Add a compact one-line, dimmed summary of the
call's arguments under the label, pulled from part.input.

New pure toolInputSummary(part): picks the first present "primary" field
(query/q/searchQuery/url/urls/title/name/text/prompt), collapses
whitespace, clamps to ~140 chars; arrays render "first (+N)". It returns
undefined during input-streaming (the input grows while state is fixed and
messageSignature doesn't track input, so a live summary would freeze) —
the state flip to input-available re-renders the row with the final value,
so message-signature.ts is left untouched. The value renders ONLY through
Mantine <Text> (React-escaped) — no markdown/HTML, no XSS.

A showInput prop (default true) is threaded MessageList -> MessageItem
(+memo) -> ToolCallCard; the public share widget passes showInput={false}
so an anonymous reader never sees the agent's raw query text (mirrors
showCitations). No JSON fallback when no primary field is present.

closes #392

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:28:24 +03:00
vvzvlad cafe29b153 Merge pull request 'test(converter): хвост #351 — nightly property cron + README + запиненные баги + media-фазз' (#391) from feat/351-remainder into develop
Reviewed-on: #391
2026-07-06 20:46:59 +03:00
agent_coder 35f2c06f42 test(converter): #391 review round — orderedList start guard + nightly hardening (#351)
DO-1 (regression): the orderedList start hardening `Number(x)||1` let a
negative/fractional start through into the marker ("-3.", "2.5.") — marked
can't tokenize it, so the whole list re-imported as a paragraph (structure
corruption); start=0 churned. Both the markdown and raw-HTML export paths
now use `Number.isInteger(raw) && raw > 1 ? raw : 1`, so any degenerate
start collapses to the default "1." markers / bare <ol> (list always valid).
New ordered-list-start-normalization test pins {0,-3,2.5} → valid start=1
list, byte-stable; the round-trip fuzz keeps to integers >=2 (num 2,3,5,42).

DO-2 (nightly was non-functional): NUM_RUNS=5000 OOM'd the worker and the
crash was misreported as a "counterexample" issue whose prefix-dedup then
locked out all future issues. Reworked to shard 8 fresh vitest processes
(600 runs each, distinct seeds, --max-old-space-size) so deep fuzzing
never OOMs; a failing shard's output is preserved, and issue creation
discriminates a real fast-check counterexample from an infra/OOM failure
(distinct titles + scoped dedup). The two issue steps use `always() &&`
so they actually run on the failure path.

DO-3: envInt extracted to test/generative/env-int.ts + unit-tested.
DO-4: nightly dispatch inputs go through env: (no ${{ }} in run:).
DO-5: attr-arbitraries.ts docblock synced (column.width/orderedList.start
are fixed+fuzzed, not pinned it.fails).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 20:32:29 +03:00
agent_coder ce9f1a8980 test(converter): nightly property cron + env knobs + counterexample README (#351)
Items 1 & 2 of the #351 remainder.

Item 1 — the flat/nested generative property tests now read SEED and
NUM_RUNS from PROPERTY_SEED / PROPERTY_NUM_RUNS (via an envInt helper that
honors an explicit 0 and falls back to the current defaults —
20250705/300 flat, /100 nested — on unset/empty/non-numeric). New
.github/workflows/nightly-property.yml runs the generative suite daily
(and on workflow_dispatch) with a random seed and NUM_RUNS≈5000; on
failure it files a Gitea issue containing fast-check's shrunk
counterexample (jq-escaped, dedup'd by title). No build step — the suite
imports the converter from src/, so a tsc error can't masquerade as a
property failure.

Item 2 — new packages/prosemirror-markdown/README.md documents the
counterexample process (surface -> shrink -> permanent fixture in
test/fixtures/counterexamples/ + counterexamples.test.ts -> fix the
converter, never weaken a property; maintainer-approved ACCEPTED/allowlist
entries carry a reason) plus the two golden layers, the coverage
allowlist, and how to run with the env knobs. Linked from AGENTS.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:51:53 +03:00
agent_coder 15fe998d3d test(converter): value-fuzz the deferred media attributes (#351)
Item 4 of the #351 remainder — implement value-fuzz for the media
dimension/family attributes previously parked in the flat property
suite's ATTR_VALUE_FUZZ_ALLOWLIST as "round-trip candidates deferred to
a later PR": width/height/align/aspectRatio/size/caption/title/alt across
image, video, youtube, pdf, drawio, excalidraw, embed.

Each gets a non-default-value arbitrary in attr-arbitraries.ts and is
removed from the allowlist; the P1 (semantic) + P2 (byte-stability)
property gate now exercises them. All round-trip green — the comment-JSON
serialization (stable key order, String()-stringified) carries every one
faithfully, so no new pins or accepted-limitations were needed.

embed.width/height are fuzzed as numeric STRINGS (not numbers): a
non-default embed dimension round-trips through the comment JSON as a
string (the numeric 800/600 default is still omitted/re-materialized),
so authoring a number diverged under P1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:51:53 +03:00
agent_coder 54f0ba681e fix(converter): preserve orderedList start; fuzz column.width numerically (#351)
Item 3 of the #351 remainder — resolve the two pinned converter
counterexamples, fixtures-first.

orderedList.start (genuine P1 loss): the converter always emitted "1."
and dropped attrs.start. Now the markdown path emits `${start + index}.`
(Number-coerced, guards a stray non-numeric start) and the raw-HTML path
emits `<ol start="N">` when start>1; a default (start=1) list is
byte-unchanged. tiptap StarterKit reads both forms back. Un-pinned as a
passing regression test; orderedList.start is now value-fuzzed.

column.width: the "50% churn" counterexample rested on a false premise —
the canonical editor (editor-ext column.ts) stores width as a unitless
flex-grow NUMBER (parseFloat, `flex:${width}`), never a "%" string, and
docmost-schema.ts is a vendored mirror that MUST match it. The original
parseFloat was correct parity and already byte-stable for numeric widths.
Reverted the mirror to parseFloat, deleted the fabricated counterexample
fixture, and now value-fuzz column.width as a number (real type). No src
behaviour change for column.width — parity with editor-ext preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:51:53 +03:00
vvzvlad 27f7791a0e Merge pull request 'feat(ai-chat): getCurrentPage отдаёт текущее выделение пользователя (#388)' (#390) from feat/388-getcurrentpage-selection into develop
Reviewed-on: #390
2026-07-06 19:08:26 +03:00
agent_coder 15859e3b9f feat(ai-chat): getCurrentPage returns the user's editor selection (#388)
Snapshot the editor selection at send time (same live-ref pattern as
openPageRef in prepareSendMessagesRequest), carry it nested inside
openPage so it dies with the page on a fail-closed resolve, and surface
it to the model only through the existing core getCurrentPage tool.

The selection TEXT is returned exclusively in the tool result (untrusted
collaborative-page content, treated as data by SAFETY_FRAMEWORK); the
system prompt gets only a fixed one-line flag, never the text/before/
after. sanitizeSelection caps text (4000), before/after (200), blockIds
(<=64 chars, <=20). Selection is a hint, not ground truth — the tool
description tells the agent to localize the fragment before editing.

closes #388

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 18:50:55 +03:00
agent_vscode ebf132d5ac feat(agent-roles): make the researcher's search budget mandatory to spend
A user-provided "research budget" now defines the required search volume:
it is binding and must be spent in full, overriding the default "stop at
saturation" rule.

- STEP 0: split the budget into user-set (binding, must be used up) vs
  self-estimated (when the user gave no number)
- VOLUME: limit "stop at saturation" to the no-budget case; add a
  MANDATORY BUDGET block requiring the full budget be spent on genuine
  broadening/lateral/primary-source/verification searches, not padding
- Apply identically to bundles/research/ru.yaml and en.yaml
- Bump researcher version 2 -> 3 in index.yaml; refresh content-hashes.json

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 18:29:57 +03:00
agent_vscode 05d3456f5c feat(agent-roles): researcher builds the report doc live from the start
Rework the "WHERE TO WRITE THE RESULT" section of the researcher role so
the report document is created at the very beginning of the run (right
after the plan, before any searches) and filled dynamically after each
finding, instead of being dumped in one pass at the end.

- Rewrite the section identically in bundles/research/ru.yaml and en.yaml
- Bump researcher version 1 -> 2 in index.yaml
- Refresh scripts/content-hashes.json via check.mjs --update-hashes

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 18:09:40 +03:00
agent_vscode d910180772 docs(readme): reverse-proxy requirements for SSE streaming paths
The AI chat SSE endpoints (POST /api/ai-chat/stream, GET
/api/ai-chat/runs/<chatId>/stream, POST /api/shares/ai/stream) must
bypass response buffering AND compression at every proxy in front of
the app — a compressing proxy silently buffers SSE frames until the
response closes (pending request, tokens arriving in one burst,
reloaded tabs degrading to polling). Document the affected paths, the
DevTools tell (Content-Encoding on text/event-stream), and concrete
nginx/Traefik configuration in both READMEs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:02:37 +03:00
agent_vscode 89a4bf28ce test(auth): de-flake verify-user-credentials.live.spec — hoist bcrypt hash + 30s timeout
Under a fully parallel `pnpm -r test` run the suite computed five separate
bcrypt cost-12 hashes (one per test, ~300ms idle, multi-second with all cores
saturated) and tripped jest's default 5s per-test timeout ("DISABLED user"
test, suite 31s under load vs 6s isolated).

- compute the hash ONCE in a top-level beforeAll (covers both describe
  blocks) and share the read-only string across the five former call sites
- jest.setTimeout(30_000) at module scope so the per-test bcrypt compares
  inside verifyUserCredentials get headroom under load too

No change to test names, assertions, order, or the CREDENTIALS_MISMATCH
contract semantics. Suite: 8/8 green, 4.8s isolated (was ~6s).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 17:24:24 +03:00
agent_vscode da94b1589c test(mcp): align nested-list goldens with the #351 loose-container converter fix
The #351 converter fix (79a461f7) made multi-block list items emit a blank-line
separator between block children (loose list) to stop silent content merging on
re-parse. The prosemirror-markdown goldens were updated in that PR, but the two
mcp unit tests asserting the old tight output were missed — packages/mcp
re-exports the shared converter, so they broke the develop CI run (test job,
`pnpm -r test`, 2/480 failures).

- expect "- Parent\n\n  - A..." instead of "- Parent\n  - A..."
- expect "1. Parent\n\n   - Child" instead of "1. Parent\n   - Child"

Full mcp suite: 480/480 green; editor-ext / prosemirror-markdown / client /
git-sync suites green as well.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 17:13:34 +03:00
vvzvlad 40227bbf51 Merge pull request 'feat(ai-chat): resumable SSE — клиент (#381 PR 2, переоткрыт на develop после стек-мёржа)' (#389) from feat/381-resumable-sse-pr2 into develop
Reviewed-on: #389
2026-07-06 16:29:04 +03:00
agent_vscode a26803a1bc docs(env): document AI_CHAT_RESUMABLE_STREAM staged-rollout flag (#381)
The resumable-SSE run-stream registry (PR #386/#387) ships behind the
server-side AI_CHAT_RESUMABLE_STREAM env flag, OFF by default: with the
flag off attach always answers 204 and reopened tabs of an active run
fall back to degraded history polling. Document the flag, its default,
its relation to the per-workspace autonomousRuns setting, and the
single-instance constraint in .env.example next to the autonomous-runs
section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 16:27:06 +03:00
agent_coder 18eee98b7f fix(ai-chat): #381 PR2 review round 1 — F7 restart-survival + unmount abort + anchor-orphan
Do 1 [F7 regression]: транзиентный сбой attach больше не роняет строку и не
теряет ран. В transport fetch-wrapper `204 || !response.ok` оба зовут
onNoActiveStream (восстановить stripped-строку + invalidate + арм poll), и catch
зовёт его перед rethrow — раньше !ok/throw только сбрасывали флаг, и на 5xx/502/
network-blip in-progress ассистент-турн исчезал, durable-ран не отслеживался.
onNoActiveStream — суперсет (его часть-г всё ещё чистит флаг), идемпотентен.
Расширяет литеральный block-3 спеки (там был только сброс флага) — по ревью и
в согласии с интенцией окна «poll must survive a server restart».

Do 2 [stability]: attach-GET абортится при unmount + mount-гейтинг сайд-эффектов.
mountedRef: mount-эффект ре-армит true и в cleanup ставит false + abort
attachAbortRef; onNoActiveStream рано выходит на !mounted, onFinish-recovery
гейтится `wasResumed && mountedRef.current`. Снимает до-10-мин спурьёзный поллинг
+ чужую invalidateQueries + утёкший fetch на новооткрытом чате (и StrictMode
double-resume).

Do 3 [coherence]: anchor-mismatch не оставляет вечную dots-строку. Reconcile
после мержа хвоста мержит fresh-history версию stripped-строки, если её id !=
id хвоста — settl'ит осиротевшую streaming-A над раном B.

Тесты: F7 500 → restore+арм; F7 network-throw → restore+арм; unmount при pending
attach → abort + поздние колбэки не летят. vitest src/features/ai-chat 304
зелёных, grep-guard пуст.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 16:26:27 +03:00
agent_coder 067fc46170 feat(ai-chat): единый resumable SSE-транспорт — клиент + удаление поллинга/латчей (#381 PR 2)
PR 2 из 2 (#381, фаза 1.5 #184). Все вкладки теперь на ОДНОМ транспорте:
любая подключается к рану через GET-attach (реплей кадров + живой хвост,
реестр из PR 1). Наблюдатель — обычный стример; Stop, точки, инвалидации
работают штатно. Двухпутёвый поллинг снапшотов и вся latch-механика F4/F5/F7
удалены.

- utils/resume-helpers.ts (новый): isStreamingTail / isSettledAssistantTail /
  seedRows / mergeById (дословный перенос mergeObservedMessage из run-polling).
- components/chat-thread.tsx: resume-машинерия (гейтинг по не-settled хвосту,
  strip streaming-хвоста + attach ?expect=live&anchor=<row id>, транспорт
  prepareReconnectToStreamRequest+fetch, 204-обработчик из 4 частей,
  reconcile+degraded-merge, recovery с АСИММЕТРИЕЙ arm-vs-restore — при
  isDisconnect с видимым контентом только arm, без restore-клоббера живого
  стрима (инв. 9), строгий порядок onFinish с ранним return до обеих веток
  отправки (инв. 7), «Send now» скрыт на resumed-ходе, Stop абортит attach).
- components/ai-chat-window.tsx: degraded-poll фолбэк вместо латчей — тупой
  таймер (2500ms, 10-мин кап, без проверок ошибок/хвоста; переживает рестарт
  сервера), гасится тредом через onResumeFallback(false).
- Удалено: run-polling.ts(+test), useAiChatRunQuery/AI_CHAT_RUN_RQ_KEY,
  getAiChatRun (stopRun оставлен), IAiChatRun/IAiChatRunResponse, латчи
  stoppingRun/localStreaming/observedRow/onStreamingChange, F7-эффект,
  observer-merge. Серверный POST /ai-chat/run не тронут.

Проверка: tsc (мои файлы чисты), vitest src/features/ai-chat 34 файла/301 тест
зелёные, grep-guard по удалённым символам пуст. Отдельное внутреннее ревью на
инварианты 7/8/9 + 204-null-safety — чисто.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 16:26:27 +03:00
vvzvlad ab1da408e5 Merge pull request 'feat(ai-chat): resumable SSE run-stream registry — сервер, спящий (#381 PR 1)' (#386) from feat/381-resumable-sse-pr1 into develop
Reviewed-on: #386
2026-07-06 16:24:38 +03:00
vvzvlad da7bb95d4f Merge pull request 'test(int): убрать избыточный forceExit — bounded teardown уже расхангивает test:int (#382)' (#383) from fix/382-int-open-handles into develop
Reviewed-on: #383
2026-07-06 16:08:22 +03:00
vvzvlad 6ab2e989b9 Merge pull request 'test(converter): вложенный variant-A генератор + P4 + фикс всех round-trip багов (#351)' (#385) from test/351-nested-generator into develop
Reviewed-on: #385
2026-07-06 16:07:48 +03:00
vvzvlad 169e34d766 Merge pull request 'docs(agents): build shared packages before a consumer's tsc/tests in isolation' (#384) from docs/agents-workspace-build-order into develop
Reviewed-on: #384
2026-07-06 16:06:34 +03:00
agent_coder 10d5220f5e fix(ai-chat): #381 PR1 review round 1 — open()-gate test + paused-pending byte-cap
Do 1 [test-coverage]: контроллерный тест на флаг-гейт begin-hook open() — при
OFF beginRun зовётся (durable-ран независим), а streamRegistry.open НЕ зовётся
(закрывает регресс: пустая entry → non-null paused attach → зависший SSE вместо
204); при ON — open зовётся с (chatId, runId).

Do 2 [stability]: байт-кап очереди pending paused-подписчика. pendingBytes +
overflowed на Subscriber; в paused-ветке ingestFrame при превышении
SUBSCRIBER_MAX_BUFFERED_BYTES (8MB) подписчик помечается overflowed, pending
чистится, он выбрасывается из entry.subscribers (как overflowed-entry). start()
на overflowed → onEnd (чистый 204-эквивалент, без частичного реплея). Контракт
«start() в том же тике, что attach()» задокументирован в коде — кап это
структурный бэкстоп для phase-2 Redis-await шва. Юнит-тест: paused A + live B,
9×1MB > cap → A выброшен (0 доставок), B получает все 9 живьём, поздний start(A)
→ один onEnd без реплея.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 07:42:36 +03:00
agent_coder 52ee3c1f3e feat(ai-chat): resumable SSE run-stream registry — server, dormant (#381 PR 1)
PR 1 of 2 (#381, фаза 1.5 #184): серверный реестр SSE-стримов агентских ранов,
чтобы любая вкладка могла подключиться к живому рану с реплеем кадров + живым
хвостом. «Спящий» — весь провод за флагом AI_CHAT_RESUMABLE_STREAM (off по
умолчанию); клиент (PR 2) ещё не написан.

- ai-chat-stream-registry.service.ts: in-memory реестр (open/bind/abortEntry/
  attach). attach — снапшот+подписка в ОДНОМ синхронном блоке (инвариант 4: нет
  await между `subscribers.add` и `frames.slice()`), paused-подписчик, overflow,
  retention с identity-guard (инвариант 2), open поверх live entry даёт ровно
  один onEnd (инвариант 3), anchor против кросс-ранового реплея (инвариант 6).
- ai-chat.controller.ts: begin-хук open(chatId, runId) + GET-attach эндпоинт
  (403 чужой чат; 204 нет-entry/finished/anchor-мисматч; cleanup до первой
  записи + recheck req.raw.destroyed; cap→destroy).
- ai-chat.service.ts: tee SSE-кадров в реестр (consumeSseStream + generateMessageId,
  гейт на runId && flag) + abortEntry из внешнего catch.
- environment.service.ts: флаг isAiChatResumableStreamEnabled().

Флаг OFF ⇒ байт-в-байт legacy И #184-фаза-1 (нет start.messageId, нет tee).
Инжектируемые провайдеры НЕ @Optional() → поломка вайринга роняет старт, а не
тихо выключает фичу.

Тесты: registry unit (16), controller.attach (9), service pipe-options (4, вкл.
flag-off-with-runId негатив), int-spec ai-chat-attach (6, реальный MockLanguageModelV3).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 07:22:20 +03:00
agent_coder ccee32cb0b test(git-sync): assert stable drawio markers, not the omittable center default (#351 review)
stabilize.test.ts used `data-align="center"` as proof the convergence pass
materialized the drawio node. Since this PR correctly stops emitting the
schema-default center align in the media builders, that marker is no longer a
reliable convergence proof. Assert on the stable canonical markers instead —
`data-type="drawio"` + `data-src="/d.drawio"` — which are always materialized
regardless of the align default. The fixpoint assertion (file2 === file1) is
unchanged; round-trip stays byte-stable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 06:44:58 +03:00
agent_coder 79a461f79d test(converter): вложенный variant-A генератор + P4-фазз, и фикс всех найденных round-trip багов (#351)
Финальная ступень #351: генератор ЦЕЛЫХ вложенных документов (random walk
по ContentMatch схемы, min-depth fixpoint для терминирования, глубина/размер
ограничены) + инвариант P4 (фазз парсера: для ЛЮБОЙ строки markdownToProseMirror
не бросает и результат валиден по схеме). Инварианты P1 (семантический
round-trip), P2 (байтовый fixpoint со 2-го прохода), P3 (тотальность) — строгие.

Генератор сразу нашёл классы реальных багов конвертера; все починены (инварианты
НЕ ослаблялись — чинился конвертер):

- loose (много-блочные) контейнеры (listItem/taskItem/callout/detailsContent)
  склеивали блоки при реимпорте — ТИХАЯ ПОТЕРЯ ДАННЫХ; теперь blank-line
  разделитель между блок-детьми (по образцу blockquote).
- соседние sibling-списки одного marker-family (task+bullet и т.п.) сливались
  в один список с ПОТЕРЕЙ чекбокса — теперь между ними эмитится инертный
  `<!-- -->` разделитель (byte-stable round-trip).
- paragraph textAlign терялся во вложенных li/td/th.
- вложенный codeBlock терял хвостовой перевод строки.
- pageBreak/pageEmbed/subpages/transclusion дропались во вложении
  (blockquote/callout/details/li).
- медиа в columns: number→string ширины/высоты и лишний data-align (churn).
- callout `> [!type]`, вложенный в список/цитату, парсился неверно
  (prefix-aware regex).

Golden-обновления (6) — прямые следствия loose-container фикса, каждое
round-trip'ится. 2100+ сгенерированных документов (3 seed) — 0 падений P1/P2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 06:29:58 +03:00
claude_code 24946ad820 docs(agents): build shared packages before a consumer's tsc/tests in isolation
Document the TS2307 trap: the gitignored build/ of @docmost/prosemirror-markdown,
@docmost/git-sync and @docmost/mcp is not honoured by a single-package
pnpm --filter <pkg> test/tsc or a bare pnpm -r test (Nx dependsOn ^build is only
applied by nx run-many), so a consumer's typecheck fails with
Cannot find module '@docmost/...' until those packages are built first.
Mirrors the order .github/workflows/test.yml already uses.
2026-07-06 05:50:34 +03:00
agent_coder 84334a1f34 test(int): drop redundant forceExit — bounded teardown already un-hangs test:int (#382)
The int suite could not self-exit after the ESM fix (8e125799) unmasked 4
specs, and was patched with two things: forceExit:true AND a bounded
destroyTestDb (sql.end({ timeout: 5 })). The bounded teardown is the real
fix — postgres.js .end() without a timeout blocks indefinitely on a stuck
pooled connection (the CI-observed "Jest did not exit"); the { timeout: 5 }
grace drains then force-closes sockets so teardown always completes.

forceExit was redundant belt-and-suspenders that also HID whether the
process truly exits on its own. Removing it: every handle-creating spec is
verified to close its handle — ai-chat-stream closes its http.createServer
in a finally, public-share-workspace-limiter closes its ioredis via
redis.quit() in afterAll, and the shared DB pools close via the bounded
destroyTestDb. --detectOpenHandles is clean and the suite self-exits.

Kept: the bounded destroyTestDb (defense for a genuinely stuck connection).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 04:07:44 +03:00
vvzvlad 3085ec1b50 Merge pull request 'fix(metrics): статика в bounded «static»-лейбл — кардинальность route (#362)' (#366) from fix/362-metrics-route-cardinality into develop
Reviewed-on: #366
2026-07-06 03:55:02 +03:00
agent_vscode 05ec9feaf9 test(int): unhang test:int — forceExit + bounded destroyTestDb (#382)
Once the ESM transform fix (8e125799) let the four previously-unparseable
int-specs run, the server integration suite stopped exiting: all 66 tests pass,
then jest prints "Jest did not exit one second after the test run has
completed." and idles until the 20-minute job timeout kills it. Separately,
ai-chat-stream.int-spec.ts failed because its afterAll (destroyTestDb) hit the
60s hook timeout — postgres.js .end() waits for in-flight queries forever, so a
leaked/stuck pooled connection hung teardown.

Pragmatic unblock (the underlying open-handle leak is tracked in #382):

- jest-integration.json: add forceExit so jest always exits after the run even
  if a suite leaves an open handle.
- db.ts: capture the singleton's raw postgres sql instance and bound the pool
  shutdown with sql.end({ timeout: 5 }) (the same bounded-end pattern already
  used in global-setup.ts) instead of Kysely.destroy(), so destroyTestDb can no
  longer hang the afterAll hook.

forceExit masks residual handle leaks rather than fixing them; the proper
investigation (run with --detectOpenHandles on a pg+redis stand, close the
leaking timers/sockets, then drop forceExit) is filed as #382.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 01:28:31 +03:00
agent_vscode 8e12579925 fix(ci): transform ESM @docmost/prosemirror-markdown in server int/e2e jest
The develop build failed in two jobs, both rooted in the ESM-only workspace
package @docmost/prosemirror-markdown (type: module, built to build/index.js
with `export * from`), which the server imports at runtime (collaboration.util,
page.service). Refactor #345 taught only the unit jest config (package.json
"jest" key) to consume it, leaving the integration and e2e configs — and the
e2e-server CI job — broken:

- `pnpm --filter server test:int` -> SyntaxError: Unexpected token 'export'
  (jest did not transform prosemirror-markdown/build/*.js).
- e2e-server job -> TS2307 Cannot find module '@docmost/prosemirror-markdown'
  (the package was never built in that job).

Mirror the proven unit config into the two failing jest configs and add the
missing build step:

- jest-integration.json / jest-e2e.json: add a babel-jest transform rule for
  `prosemirror-markdown/build/.+\.js$` (before the ts-jest rule so it wins) and
  add @docmost/prosemirror-markdown to the transformIgnorePatterns allowlist so
  the pnpm-symlinked package is transformed instead of ignored.
- develop.yml: build @docmost/prosemirror-markdown in the e2e-server job (after
  editor-ext, before migrations), like the test.yml job already does.

Verified locally: an isolated spec importing the package fails with the exact
SyntaxError under the old config and passes under the new one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 00:36:31 +03:00
vvzvlad 20703d06c2 Merge pull request 'feat(gitmost-bridge): вставка transcript в страницу записи (#377)' (#378) from fix/377-bridge-transcript into develop
Reviewed-on: #378
2026-07-06 00:06:59 +03:00
agent_coder dab2660999 fix(gitmost-bridge): нейтрализовать сплошные thematic breaks в транскрипте
Round-1 нейтрализация ловила только spaced-разделители (`- - -`),
но пропускала сплошные `---`/`***`/`___`: на git-sync round-trip такая
строка становится horizontalRule, а он не несёт текста — строка терялась
целиком (хуже list/quote-порчи). Символ `_` вообще отсутствовал в regexе.

GITMOST_MD_BLOCK_TRIGGER_RE дополнен альтернативой на целую строку-
thematic-break `([-*_])(?:\s*\1){2,}\s*$` (3+ одинаковых `-`/`*`/`_`,
solid или через пробел); прежние группы переведены в non-capturing,
чтобы `\1` ссылался на единственную capture-группу. Обе копии regexа
(bridge и pm-markdown тест) синхронны.

Тесты: bare `---`/`***`/`___` документированы как text-losing
horizontalRule; ZWSP-нейтрализованная форма round-trip'ится параграфом
с сохранённым текстом. Кейсы падают на старом regexе.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 00:02:29 +03:00
agent_coder 751d55e9db fix(gitmost-bridge): neutralize col-0 block triggers in transcript lines + lock text-not-HTML (#378 review round 1)
Round-1 review found two issues:
- [regressions] Inserting a transcript line VERBATIM as a paragraph is unsafe on the
  git-sync doc->markdown->doc round-trip: the paragraph serializer emits text with no
  block-escape, so a line starting at col 0 with `- `/`> `/`# `/`1. `/```` ``` ````/`|`/
  `> [!info]` silently re-parses into a list/quote/heading/code/table/callout (the last
  hits the #359 callout machinery). Safe under the host contract (every line prefixed
  `You:`/`Speaker N:`, which starts with a letter) but the helper didn't enforce it, and
  leading-whitespace / stray lines exist. Fix: trim each kept line (drops the indent leak),
  and if it STILL begins with a col-0 markdown block trigger, prepend an invisible
  zero-width space (U+200B) so the trigger isn't at col 0 — the round-trip keeps it a
  paragraph. (Backslash-escape was rejected: `marked` consumes the `\` on re-import, so it
  would render visibly then vanish. ZWSP is invisible and never markdown-escaped.) The
  serializer's missing block-escape is the pre-existing root cause; this is the boundary
  defense. Prefixed transcript lines never match the trigger regex → left byte-exact.
- [test-coverage] The "inserted as TEXT, not HTML/markdown" contract wasn't locked (all
  test strings were plain alphabet). Added a test that inserts `<b>bold</b>
  <script>alert(1)</script> and *stars* and [link](x)` (with Bold/Italic/Link marks
  registered so an insertContent(html) regression would parse them) and asserts one
  verbatim text node, no marks, and getHTML() has no live tags.

Verified: apps/client tsc --noEmit 0 errors; gitmost bridge tests 4 passed; a NEW
prosemirror-markdown round-trip test (real convertProseMirrorToMarkdown ->
markdownToProseMirror) proves bare triggers corrupt while the ZWSP-neutralized form stays
a single byte-preserved paragraph and normal You:/Speaker N: lines round-trip byte-exact —
3 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 00:01:25 +03:00
agent_coder 02308012a6 feat(gitmost-bridge): insert transcript into the recording page (#377)
The native gitmost.app (stage 2) now sends a `transcript` field to
window.gitmost.createPageWithRecording, but the web bridge ignored it — the page
was created with audio only. Insert it.

- gitmost-recording.ts: add `transcript?: string` to GitmostCreatePagePayload
  (plain text, \n-separated `You:` / `Speaker N:` lines, ready to insert). Add a
  gitmostInsertTranscriptIntoEditor(editor, transcript) helper: a "Transcript"
  heading + one paragraph per non-empty line, each line inserted VERBATIM as a
  text node (never HTML → no injection), appended at doc end (below the audio).
  No-op when transcript is undefined/empty/whitespace-only/non-string.
- gitmost-global-bridge.tsx (createPageWithRecording): after the audio insert
  succeeds, call the helper inside a try/catch — best-effort, so a transcript
  failure can never turn the already-successful recording into an error (logs +
  still returns ok). Absent transcript → audio-only page, exactly as today.

DoD: recording with speech → page has audio AND a labeled transcript block;
without transcript → unchanged. Closes the web-side dependency of gitmost-app
stage 2. Verified: apps/client tsc --noEmit 0 errors; 2 unit tests
(transcript present → heading+paragraphs; undefined/""/whitespace/number/object
/null → no-op, doc unchanged).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 00:01:25 +03:00
vvzvlad 0665fcb630 Merge pull request 'fix(queue): убрать мёртвую очередь {search-queue} (#379)' (#380) from fix/379-remove-dead-search-queue into develop
Reviewed-on: #380
2026-07-05 23:44:54 +03:00
agent_coder 97bd554cb5 fix(queue): remove the dead {search-queue} — producers with no consumer (#379)
The {search-queue} BullMQ queue had producers on every page/space/workspace change
but NO consumer/@Processor — it lives in Docmost's EE edition (external search
drivers), never in this fork. Search works independently via the pages tsvector DB
trigger; the queue was pure dead weight, growing forever (1902 stuck jobs in Redis,
the first real hit of the #355 queue-growing alert — which fired correctly).

Remove the plumbing:
- the 3 producers: drop @InjectQueue(SEARCH_QUEUE) + every searchQueue.add(...) from
  page/space/workspace.listener.ts (and the isTypesense() gate that only wrapped the
  search enqueue). page.listener.handlePageUpdated did ONLY the search enqueue, so its
  @OnEvent(PAGE_UPDATED) handler is removed; the other handlers keep their aiQueue.add.
- the registration (queue.module.ts) and the metrics injection + 'search' depth-metric
  entry (metrics-bull.service.ts).
- the constants: QueueName.SEARCH_QUEUE + the 6 unused SEARCH_INDEX_* QueueJob members
  (grep-confirmed unreferenced). Shared QueueJob members (PAGE_*, SPACE_DELETED,
  WORKSPACE_DELETED — used by the AI queue / embedding/history/notification processors)
  are kept.

Search (tsvector trigger, search.service) and the PAGE_UPDATED websocket listener are
untouched. Verified: apps/server tsc --noEmit 0 errors; 6 spec suites / 52 tests green.

Ops (maintainer, out of code scope): one-time Redis cleanup
`redis-cli --scan --pattern "bull:{search-queue}:*" | xargs -r redis-cli del`, then
confirm bullmq_queue_depth{queue="search"} stays 0 (the metric no longer injects it).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 23:35:04 +03:00
vvzvlad f77a6b42de Merge pull request 'docs: how to test the application (browser E2E + out-of-band)' (#376) from docs/how-to-test into develop
Reviewed-on: #376
2026-07-05 22:40:12 +03:00
C9 Tester 134b627806 docs: add how-to-test.md (browser E2E + out-of-band) and link from AGENTS.md
Adds a testing guide covering how to verify features against a running stand:
drive the behaviour under test through the browser (not the API), verify
out-of-band in the DB/git, and the non-obvious traps. Notably the page has two
ProseMirror editors — [aria-label='Page title'] (non-collab) and
[aria-label='Page content'] (the collab body); querySelector('.ProseMirror')
returns the title, so tests must target the body editor and wait ~10s for the
hocuspocus store debounce. Links the new doc from AGENTS.md next to dev-stand.md
and adds a matching gotcha #8 to dev-stand.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 22:39:01 +03:00
vvzvlad 3267512ed9 Merge pull request 'refactor(#345): серверный экспорт/импорт markdown через @docmost/prosemirror-markdown' (#369) from refactor/345-server-converter into develop
Reviewed-on: #369
2026-07-05 20:41:30 +03:00
vvzvlad 48bd27b83c Merge pull request 'test(#351 PR 1): генеративное round-trip-тестирование конвертера — атрибутный уровень' (#373) from test/351-generative-converter into develop
Reviewed-on: #373
2026-07-05 20:40:40 +03:00
vvzvlad 265b81c93d Merge pull request 'fix(db): миграции «задним числом» из долгоживущих веток не роняют старт — CI-гейт + allowUnorderedMigrations (#363, инцидент #361)' (#365) from fix/363-migration-order into develop
Reviewed-on: #365
2026-07-05 20:40:06 +03:00
vvzvlad ed808876be Merge pull request 'fix(ai): patch ai@6.0.134 — drop O(n²) partialOutput accumulation causing heap OOM on long agent runs (#184)' (#368) from fix/ai-sdk-partial-output-oom into develop
Reviewed-on: #368
2026-07-05 20:39:51 +03:00
vvzvlad a72ddbbe86 Merge pull request 'refactor(ai-chat): единый реестр спеков инструментов — унификация tables/pages/misc/comments (#294)' (#367) from refactor/294-spec-registry-cont into develop
Reviewed-on: #367
2026-07-05 20:39:41 +03:00
agent_coder d8fc724d90 test(ai): cover the partialOutput PRESERVE branch of the ai@6.0.134 patch (#184, review F1)
The patch forks createOutputTransformStream: output==null skips partialOutput
(the OOM fix, already tested), output!=null preserves the original cumulative
accumulation. Only the skip branch was tested; the preserve branch — on which the
patch's "byte-identical when an output strategy is set" safety claim rests — had no
coverage, so a future re-port (patches are re-created via `pnpm patch` on every ai
bump) could silently route output-set calls into the skip branch and leave
partialOutput empty for object/text-output consumers, uncaught.

Add a 4th test: streamText({ ..., experimental_output: Output.text() }), drain
textStream, collect experimental_partialOutputStream, and assert it is non-empty and
cumulative (last partial == full text "Hello, world!"). Reuses the existing
makeModel() harness. Verified on the patched dist: partials are
["Hello","Hello, ","Hello, world!"]. `npx jest ai-sdk-partial-output.patch.spec.ts`
→ 4 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 19:35:14 +03:00
vvzvlad e4bfbcabaa Merge pull request 'feat(#371): редизайн модалки каталога ролей — карточки-наборы + per-role результаты импорта' (#375) from feat/371-roles-catalog into develop
Reviewed-on: #375
2026-07-05 16:43:19 +03:00
agent_coder 4c1ee50dc9 test(#351): close the mark-attr coverage hole + reclassify table spans (review round 1)
F1 [WARNING] The 'no invisible coverage hole' guard enumerated only
schema.nodes, so MARK attributes silently escaped the value-fuzz completeness
check — link.internal/target/rel/class are never fuzzed and nothing flagged it,
and a new attributed mark would slip through. Added allSchemaMarkAttrKeys() plus a
MARK_ATTR_FUZZED / MARK_ATTR_ALLOWLIST registry and two tests: every schema mark
attr must be in exactly one set (a new one turns it red), and neither set may hold
a stale row.

F2 [WARNING] The ACCEPTED annotation misclassified table colspan/rowspan as
having 'no md representation'. They DO round-trip — a spanned cell makes the
converter emit the whole table as a raw <table> with colspan/rowspan, which the
tiptap parser reads back. They are frozen only because generating a
geometrically-valid spanned table is deferred PR-2 structural work (the flat
generator hardcodes span = 1), not a markdown limit. Reclassified them as
DEFERRED-BUG (distinct from ACCEPTED) so a maintainer does not read them as an
inherent limitation; colwidth / backgroundColor(Name) stay ACCEPTED (the
raw-<table> fallback drops them).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 06:40:13 +03:00
agent_coder b8cce4f814 fix(#371): skipped role is not 'allInstalled', test the reason->action branch (review round 1)
F1 [WARNING] bundlePhase returned 'allInstalled' when a bundle's only
non-installed role was skipped (0 installed for it), so the collapsed green 'All
installed · up to date' header contradicted the open 'Installed 0 · 1 skipped'
plaque. It now returns 'mixed' whenever a skipped role is present. Fixed the test
that encoded the wrong behavior.

F2 [WARNING] The reason->action branch (name-conflict -> transient overlay +
'Rename & install'; already-installed -> informational, no button) lived only in
the component, untested. Extracted the two decisions into pure, unit-tested
helpers nameConflictSlugs() and partialOffersRename() and wired them into the
modal; both reason values are now covered.

F3 [low] Removed the unused useRef import (client eslint no-unused-vars is off, so
it shipped silently).

F4 [low] Extracted bundleCounts() as the single tally pass; bundlePhase and the
panel both derive from it instead of rescanning the roles array ~5x per render
(the same model<->component consolidation this PR is about).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 06:09:58 +03:00
agent_coder a325ddbabd feat(#371): roles catalog modal redesign — bundle cards + per-role import results
Integrates the designer-handoff Roles Catalog modal, wired to the real API; the
parent ai-agent-roles.tsx and the { opened, onClose, roles } contract are
unchanged.

- Server importFromCatalog now returns per-role lists (createdRoles /
  skippedRoles with a reason) alongside the existing counters (compat-preserving),
  so the UI can name the conflicting/installed roles.
- New pure view-model (catalog-bundle-model.ts): bundlePhase (empty | allNew |
  allInstalled | updates | mixed, ignoring the transient 'skipped'),
  installedLangForRole (same-slug-different-language hint), mapCatalogRoleToView —
  all unit-tested without mounting.
- Bundle cards with a summary status in the collapsed header (eager useQueries
  fan-out over all bundles, sharing the existing per-bundle cache keys), a single
  primary action per bundle, checkboxes + select/deselect-all, an inline result
  plaque that keeps the modal open, per-bundle and global 'Update all' request
  series with progress, and the other-language hint.
- The partial-result plaque distinguishes the skip reason: only a name-conflict
  offers 'Rename & install'; an already-installed race is informational (a rename
  re-import would just skip again and self-heal into a false success).
- All strings i18n'd (en/ru); mock handoff code (SEED/mockImport/delay) removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 05:35:58 +03:00
agent_coder bfcee6dddc test(prosemirror-markdown): generative round-trip testing — attribute level, flat docs (#351 PR 1)
Schema-derived, property-based (fast-check) round-trip tests over flat
single-node ProseMirror documents. One test PR — src/ is untouched; the two
real bugs found are pinned as loud it.fails counterexamples, not fixed here.

- attr-arbitraries.ts: per-attribute four-state arbitraries (absent/default/
  nonDefault/degenerate), attribute list sourced from schema.nodes[t].spec.attrs;
  a documented override table supplies legal domains for constrained attrs and
  distinguishes two frozen classes explicitly — ACCEPTED limitations (no md
  representation) vs PINNED bugs (representable but dropped, tracked as
  counterexamples).
- text-arbitraries.ts: hostile text corpus (ported from the existing property
  test's supported-space guarantees).
- node-generators.ts: flat single-node generators + a completeness contract —
  every one of the schema's 45 nodes / 12 marks is either generated or listed in
  KNOWN_UNCOVERED with a reason.
- flat-roundtrip.property.test.ts: P1 (semantic round-trip via
  docsCanonicallyEqual), P2 (second-pass byte fixpoint — anti GS-EDIT-REVERT),
  P3 (totality), generator validity via schema.check(), and an explicit
  attribute-value-coverage snapshot so the not-fuzzed set can never grow silently.
- counterexamples: column.width (% dropped on parseFloat -> P2 churn) and
  orderedList.start (non-1 start renders as '1.' -> P1 loss) pinned as it.fails.

SEED=20250705, NUM_RUNS=300 per property; ~17s, no OOM (union arbitraries).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 04:38:40 +03:00
agent_coder 36b940fdb8 fix(#294 review F1-F2): test the changed execute wirings + transport-neutral descriptions
- F1: added in-app execute tests for the two wirings that ACTUALLY changed in the
  migration (the contract-parity test only checks advertised schema keys, not
  execute bodies): movePage forwards the newly-added optional `position` to
  client.movePage (and passes undefined position + null parent when omitted); the
  table trio (insert/delete/updateCell) forwards the unified `table` param
  positionally. A field destructured under the wrong name would have silently
  passed undefined to the client (execute is any-cast, tsc won't catch it).
- F2: rewrote the three migrated descriptions that hardcoded snake_case sibling
  tool names (which the in-app camelCase layer exposes under different ids,
  violating the registry's own transport-neutral-prose convention) into neutral
  prose: getPage "use get_page_json" -> "use the lossless page-JSON read tool";
  updatePageJson "get_page_json -> ... -> update_page_json" -> "read the page-JSON
  view -> modify -> write it back", "prefer rename_page" -> "prefer the rename-page
  tool"; exportPageMarkdown "import_page_markdown round-trip" -> "page-Markdown
  import round-trip" (the last was a direct regress — the in-app base said the
  camelCase importPageMarkdown). (stashPage's pre-existing get_page_json mention is
  out of scope, per the reviewer.)

Gate: mcp build 0; ai-chat-tools.service + tool-tiers (catalog-partition) pass,
incl. the 5 new execute-wiring tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 03:10:42 +03:00
agent_coder 0050ad7ebb docs(#363 review F2): update AGENTS.md migration-ordering to the new tolerant behavior
The "Migration ordering" section still described the OLD crash-loop-at-boot
behavior this PR removes ("Kysely refuses to start … rejected at boot"). Rewrote
it to the new two-layer model: the CI migration-order gate is the primary defense
(rename to a current timestamp), and the runtime now sets allowUnorderedMigrations
so the app applies a back-dated migration instead of crash-looping (with the note
that #ensureNoMissingMigrations still guards a removed applied migration, and that
migrations must stay independent since apply order can differ across instances).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:57:11 +03:00
agent_coder 43b11d92ab fix(#362 review F1-F3): add /icons/ prefix + honest comment + real boundary test
- F1: added '/icons/' to STATIC_PATH_PREFIXES — public/icons/ is copied verbatim
  to client/dist (a sibling of the already-included brand/vad/locales), and
  index.html references /icons/favicon-*.png on every page load, so those requests
  were getting their own route labels instead of collapsing to `static`.
- F2: corrected the comment — only /assets/ is content-hashed (unbounded per
  deploy); /vad//brand//locales//icons/ have stable names (repetitive, not
  unbounded). Either way none belong in the API-route histogram.
- F3: the negative test now exercises the trailing-slash boundary (the actual
  anti-false-collapse guard): '/assets' (no slash), '/assetsx/foo.js',
  '/iconset/x.png' must NOT collapse to `static` — cases that a buggy
  includes()/slashless-prefix impl would wrongly collapse. Plus '/icons/*' added
  to the positive it.each.

Gate: server tsc 0; metrics.spec passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:38:47 +03:00
agent_coder ce70fab1df refactor(ai-chat): unify share_page into SHARED_TOOL_SPECS (#294, misc family)
Migrates share_page / sharePage into the transport-agnostic spec registry
(schema + description declared once; each transport keeps only its execute/auth):
- sharePage (deferred) -> SHARED_TOOL_SPECS; index.ts uses registerShared(),
  ai-chat uses sharedTool(); removed from INLINE_TOOL_TIERS.

Drift reconciled (documented inline): both inline copies already carried the
"only share when the user explicitly asked" security framing, so the old
"per-transport divergence" note in BOTH layers was STALE — there was no real
behavioral divergence, only wording drift. The canonical description merges the
MCP copy's URL-format + idempotency detail with the in-app copy's reversibility
note and keeps the shared security framing. pageId keeps the MCP copy's stricter
.min(1). The MCP execute keeps its own `searchIndexing ?? true` default
(per-layer, not part of the shared schema).

Intentionally NOT migrated (kept inline — genuinely divergent, as their existing
notes state):
- search / searchPages: the in-app tool is a semantic+keyword hybrid (RRF) with
  in-process access control and a tuned schema (limit 1-20); the MCP `search` is
  a plain REST full-text search (limit up to 100). Different behavior AND schema.
- docmost_transform / transformPage: the in-app tool deliberately omits the
  `deleteComments` schema field (a comment-deletion guardrail) and carries a
  shorter description. Different schema.

Gate: mcp build 0 + node --test 458/458 (page-search excluded — hangs only under
the local re2->RegExp type-shim, its source untouched), server jest 775 incl.
tool-tiers catalog-partition + shared-spec contract parity, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:27:50 +03:00
agent_coder 7b4617db70 fix(#363 review F1): make the migration-order gate fail CLOSED (not open)
The CI gate — whose whole job is to BLOCK a back-dated migration — could pass
open in exactly the scenario it guards (a long branch vs a moving base, i.e. #361):

- Dropped the redundant `git fetch --depth=1`: the checkout already did
  fetch-depth:0 (full history), and the shallow graft truncated the BASE history,
  so `merge-base` (thus the three-dot `origin/base...HEAD` diff) failed when the
  base had moved ahead of the PR merge commit.
- Removed `|| true` on the diff: it swallowed that failure → `added` empty → loop
  skipped → bad=0 → gate PASS. Now `set -e` aborts the job (fail CLOSED) on any
  diff error — a gate must never pass on error.

Verified: yaml parses (jobs migration-order, test); a broken-ref diff with set -e
and no `|| true` aborts before bad=0 (fail-closed) instead of passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:27:16 +03:00
agent_coder b51dae16a6 docs(mcp): mark media tools MCP-only in index.ts (#294, media family)
The media tools — insert_image, replace_image, insert_footnote — are MCP-only
by design: the in-app AI-chat agent exposes no image or footnote tools, so there
is no second layer to unify into SHARED_TOOL_SPECS. A registry spec's
tier/catalogLine are in-app metadata and the catalog-partition test forbids a
spec without a live in-app tool, so forcing them into the registry would break
the invariant. They stay per-transport (inline in index.ts).

No behavior change — documentation only (adds the rationale above each tool so a
future migrator does not re-investigate why these are not shared).

Gate: mcp tsc 0 (comment-only change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:14:39 +03:00
agent_coder 39735afd73 refactor(ai-chat): unify page tools into SHARED_TOOL_SPECS (#294, pages family)
Migrates the three-layer page tools into the transport-agnostic spec registry
(schema + description declared once; each transport keeps only its execute/auth):
- getPage, listPages (core), createPage, movePage, renamePage, deletePage,
  updatePageJson, exportPageMarkdown (deferred) -> SHARED_TOOL_SPECS; index.ts
  uses registerShared(), ai-chat uses sharedTool(); removed from
  INLINE_TOOL_TIERS. Tiers preserved from CORE_TOOL_KEYS (getPage/listPages =
  core, the rest deferred).

delete_page is genuinely three-layer (in-app deletePage exists), so it IS
migrated — not MCP-only. Its H4 guardrail is preserved: the shared schema
exposes ONLY pageId, so no permanentlyDelete/forceDelete flag can reach the
client (still asserted by ai-chat-tools.service.spec.ts).

Descriptions merged (documented inline): each canonical text takes the MCP
copy's richer structural notes plus the in-app copy's reversibility framing.

Schema DRIFT reconciled (documented inline):
- createPage.content: MCP pinned .min(1) but the in-app copy left it unbounded
  and DOCUMENTS an empty body as valid ("may be empty" — creating an empty page
  to fill later is a real use). Kept the looser no-min form: create_page now also
  accepts an empty body (harmless) and no previously-valid in-app input is
  rejected. title/spaceId keep the MCP .min(1) (empty is never valid).
- movePage: MCP exposed an optional `position` (fractional-index) field the
  in-app copy lacked. Unified by KEEPING position — the in-app client already
  accepts an optional position arg, so the in-app execute now forwards it;
  optional, so no previously-valid call is rejected. `parentPageId` is nullable
  on both (real JSON null -> root); the MCP execute keeps its 'null'/'' string
  coercion as a per-layer robustness fallback.
- getPage/renamePage/updatePageJson/exportPageMarkdown/listPages: kept the MCP
  copy's stricter .min(1) on ids where the in-app copy was unbounded.

Per-transport execute logic preserved: getPage's {title,markdown} projection,
updatePageJson's JSON-string normalization, list_pages' default limit/tree, and
move_page's cycle guard + positive-confirmation check all stay in their execute
bodies.

Intentionally NOT touched: updatePageContent (Markdown-based body update; no MCP
equivalent) and getTable (name-convention divergence, see tables family) stay
inline.

Gate: mcp build 0 + node --test 458/458 (page-search excluded — hangs only under
the local re2->RegExp type-shim, its source untouched), server jest 770 incl.
tool-tiers catalog-partition + shared-spec contract parity + deletePage H4
guardrail, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:13:41 +03:00
agent_vscode 9b4b38a611 fix(ai): patch ai@6.0.134 — drop O(n²) partialOutput accumulation causing heap OOM on long agent runs (#184)
Production OOM'd (JS heap 1.85 GB / 2 GB limit) during a ~20-step,
~28k-chunk autonomous agent turn. Heap snapshot analysis (memlab) showed a
single DefaultStreamTextResult retaining ~1.7 GB via the never-consumed
leftover tee() branch of its internal baseStream.

Root cause in ai@6.0.134: streamText substitutes the default text() output
strategy even when the caller passes NO `output` option. Its
createOutputTransformStream then accumulates the ENTIRE turn text and, on
EVERY text-delta, enqueues `{ part, partialOutput }` where partialOutput is
a flat snapshot of all text so far (JSON.stringify flattens the
cons-string) — O(n²) memory across the turn. Every consumer accessor tees
baseStream and keeps the second branch as the new baseStream; the final
leftover branch is never read, so its controller queue holds every chunk
(28,225 x ~164 KB in the OOM'd run) for the life of the turn.

Fix (pnpm patch on both dist/index.js and dist/index.mjs):
- pass the raw, possibly-undefined `output` option into
  createOutputTransformStream instead of defaulting to text()
- when output == null, publish each text-delta immediately without
  accumulating turn text or producing partialOutput snapshots; streaming
  granularity is unchanged, and callers that DO request an output strategy
  keep the original behavior

Our server never uses partialOutputStream / experimental_output / the
output option, so no behavior changes for us beyond memory.

Regression spec ai-sdk-partial-output.patch.spec.ts drives the real
patched SDK with MockLanguageModelV3: asserts per-delta textStream
granularity, an EMPTY experimental_partialOutputStream (tripwire — yields
one cumulative partial per delta when unpatched), and the PATCH(docmost
marker in both installed dist bundles. Also documents the patch in
AGENTS.md (must be re-created when bumping `ai`) and CHANGELOG.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 02:13:17 +03:00
agent_coder eebbe6717c refactor(ai-chat): unify table row/cell tools into SHARED_TOOL_SPECS (#294, tables family)
Migrates the three-layer table WRITE tools into the transport-agnostic spec
registry (schema + description declared once; each transport keeps only its
execute/auth):
- tableInsertRow, tableDeleteRow, tableUpdateCell -> SHARED_TOOL_SPECS;
  index.ts uses registerShared(), ai-chat uses sharedTool(); removed from
  INLINE_TOOL_TIERS (all three are deferred; not in CORE_TOOL_KEYS).

Drift reconciled (documented inline): the four table tools previously carried a
"NOT shared" note in both layers over a single parameter-NAME drift — the MCP
layer named the table reference `table`, the in-app layer `tableRef`. Unified on
the MCP name `table` (renaming the public MCP parameter would break external MCP
clients; the in-app parameter is model-facing/prompt-only and safe to rename).
The in-app execute bodies now destructure `table`. Descriptions took the MCP
copy's richer wording (documents `#<index>`, padding, header-row behavior) plus
the in-app copy's "Reversible via page history" note; both fields keep the MCP
copy's stricter .min(1) (in-app left them unbounded); sibling tool references
phrased transport-neutrally.

Intentionally NOT migrated (kept inline): table_get / getTable. Its MCP tool
name is noun-first (`table_get`) while the in-app key is verb-first (`getTable`),
which breaks the snake_case(inAppKey) naming convention the registry enforces
(shared-tool-specs.contract.spec.ts). Renaming the public MCP tool would break
external clients, so it stays per-transport — but its in-app reference param was
still aligned to `table` (was `tableRef`) for consistency with the migrated trio.

Gate: mcp tsc 0 + node --test 458/458 (page-search excluded — hangs only under
the local re2->RegExp type-shim, its source is untouched), server jest 730 incl.
tool-tiers catalog-partition + shared-spec contract parity, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:06:52 +03:00
agent_coder e348433a39 refactor(ai-chat): unify comment tools into SHARED_TOOL_SPECS (#294, comments family)
Migrates the three-layer comment tools into the single transport-agnostic spec
registry (schema + model-facing description declared once; each transport keeps
only its execute/auth):
- createComment, listComments, resolveComment, checkNewComments — moved to
  SHARED_TOOL_SPECS; index.ts uses registerShared(), ai-chat uses sharedTool();
  removed from INLINE_TOOL_TIERS (tier/catalogLine now on the spec). Tiers
  preserved from CORE_TOOL_KEYS (create/list/resolve = core, check = deferred).

Intentionally NOT migrated (kept MCP-inline): update_comment / delete_comment —
they are MCP-only by design; the in-app AI-chat layer deliberately has no
updateComment/deleteComment (comment edits are irreversible / not
version-tracked), asserted by ai-chat-tools.service.spec.ts. A registry spec's
tier/catalogLine are in-app metadata and the catalog-partition test forbids a
deferred spec without a live in-app tool, so these stay per-transport.

Drift reconciled (documented inline): createComment/listComments/checkNewComments
took the more-maintained/superset description + stricter .min(1) guards.
resolveComment: `resolved` drifted (MCP optional+default(true) vs in-app
required) — kept the MCP superset, so in-app resolveComment now accepts an
omitted `resolved` (defaults to resolve) — a deliberate, backward-compatible
unification (never rejects a previously-valid input).

Gate: mcp build 0 + node --test 480/480, ai-chat 654, tool-tiers (incl. F3
catalog-partition) 16/16, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 01:45:43 +03:00
agent_coder f759084f41 fix(metrics): collapse static-asset routes to bounded "static" label (#362)
Follow-up to #355: http_request_duration_seconds's `route` label captured raw
content-hashed asset filenames (route="/assets/index-CAbxDtto.js",
"/assets/chunk-*.js"). @fastify/static serves each file through a route whose
matched routeOptions.url IS the raw hashed path, so the label was unbounded — a
new set of names every deploy, growing the series forever (the exact cardinality
leak the API routes were protected against).

resolveRouteLabel now detects a static request by its path prefix (/assets/,
/vad/, /brand/, /locales/) FIRST and collapses it to a single `static` label
(query string stripped before the check); API routes still use the template and
404s still collapse to `unknown`. Static edge latency is already measured by
Traefik's traefik_router_request_duration_*.

Gate: server tsc 0; metrics.spec passes (added static-collapse + query-strip +
"real API route mentioning assets is NOT collapsed" cases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 01:44:16 +03:00
agent_coder 459d636ffb fix(db): prevent the migration-order crash-loop from long-lived branches (#363, incident #361)
A long-lived branch can add a migration whose timestamped filename sorts BEFORE
migrations already applied in prod (#234's 20260627T130000-ai-chat-runs merged
after 20260704T120000-client-metrics was live). Kysely's migrator with the
default ordered setting then rejects the applied set as "corrupted migrations"
(no longer a prefix of the sorted list), throws, and the app crash-loops on boot
— exactly incident #361 (502s for ~11 min after a develop deploy). #119 and #120
(June branches) are the next such threats.

Two levels, both:
1. CI migration-order gate (a new `migration-order` job in test.yml, PR-only):
   fails the PR when an added migration sorts at/before the newest migration on
   the base branch, with an actionable message to rename it to a current
   timestamp before merge. This is the primary defense — makes back-dating
   impossible to merge accidentally.
2. `allowUnorderedMigrations: true` on BOTH Migrators (migration.service.ts
   startup auto-migrate + migrate.ts CLI): the runtime safety net — Kysely applies
   a not-yet-applied older migration instead of bricking startup, so a back-dated
   migration that bypasses the gate (manual push / hotfix branch) still boots.
   Trade-off documented inline: apply order across instances may differ from
   lexicographic, so migrations must stay independent (ours each create their own
   objects); the CI gate remains the primary line.

Verified: allowUnorderedMigrations is a valid Kysely 0.28.17 Migrator option;
server tsc clean; the gate script rejects a back-dated filename and passes a
current one. No new deps, no migration, no runtime behavior change beyond the
migrator resilience.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 01:36:57 +03:00
105 changed files with 10883 additions and 2247 deletions
+12
View File
@@ -222,6 +222,18 @@ MCP_DOCMOST_PASSWORD=
# CLOUD=true) — run a single instance instead. The server logs a startup WARNING
# when it detects a multi-instance deployment (CLOUD=true) so the constraint is
# visible, and a startup sweep settles any run left dangling by a restart.
#
# Resumable run streams (#184 phase 1.5, #381). With the flag ON, an active
# durable run tees its SSE frames into an in-memory registry, and a
# reloaded/second tab attaches via GET /ai-chat/runs/:chatId/stream to follow the
# run LIVE (replay of the buffered frames + the live tail). With the flag OFF
# (default) the registry is never populated and attach always answers 204, so a
# reopened tab of an active run silently falls back to degraded 2.5s history
# polling — every wire path stays byte-for-byte identical to a build without the
# feature. Staged-rollout switch: only meaningful when autonomousRuns (above) is
# enabled for a workspace, and the same single-instance constraint applies (the
# registry is process-local).
# AI_CHAT_RESUMABLE_STREAM=false
# --- Anonymous public-share AI assistant ---
# Opt-in per workspace (AI settings -> "public share assistant"; off by default).
+6
View File
@@ -151,6 +151,12 @@ jobs:
- name: Build editor-ext
run: pnpm --filter @docmost/editor-ext build
# @docmost/prosemirror-markdown is an ESM workspace package the server
# imports at runtime; its build/ is gitignored and test:e2e has no pretest
# hook, so build it before the e2e run (mirrors the test.yml job).
- name: Build prosemirror-markdown
run: pnpm --filter @docmost/prosemirror-markdown build
- name: Run migrations
run: pnpm --filter ./apps/server migration:latest
+224
View File
@@ -0,0 +1,224 @@
name: Nightly property fuzz
# The daily heavy property run for the ProseMirror<->Markdown converter
# (packages/prosemirror-markdown). The PR/CI test run keeps NUM_RUNS modest to
# stay under budget; this cron cranks up total coverage with random seeds to hunt
# for deeper round-trip counterexamples than a fixed-seed PR run can reach.
#
# WHY SHARDING: a single mega-run (~10000 fast-check runs) OOMs the vitest worker
# (empirically ~1625 runs -> "JS heap out of memory", ~2GB) because heap
# accumulates across the whole property run in one process. Instead this job runs
# SHARDS fresh vitest processes, each a MODERATE per-shard count with a DISTINCT
# derived seed, so total coverage ~= SHARDS x PER_SHARD_NUM_RUNS across processes
# that never accumulate heap. On the first failing shard we stop and keep that
# shard's output for triage.
#
# Counterexample -> fixture workflow: when a shard fails, fast-check prints the
# SHRUNK minimal counterexample plus the reproducing seed. This job files a Gitea
# issue containing that seed + counterexample ONLY when the output actually holds
# a fast-check counterexample; an infra failure (OOM/tsc/install, no
# counterexample) is filed under a DISTINCT title so it can never poison the
# counterexample dedup. A human then commits the shrunk doc as a PERMANENT fixture
# under packages/prosemirror-markdown/test/fixtures/counterexamples/ with a case in
# counterexamples.test.ts, and FIXES the converter (never weakens a property to
# hide the bug). See packages/prosemirror-markdown/README.md.
on:
schedule:
# 03:00 UTC daily.
- cron: '0 3 * * *'
workflow_dispatch:
inputs:
num_runs:
description: 'fast-check runs PER SHARD (8 shards run in sequence)'
required: false
default: '600'
seed:
description: 'base fast-check seed (empty = random); shard i uses base+i'
required: false
default: ''
permissions:
contents: read
issues: write
jobs:
property-fuzz:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up pnpm
uses: pnpm/action-setup@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
# No build step: the generative suite imports the converter from src/
# directly (e.g. `from '../../src/lib/markdown-converter.js'`), so it runs
# against source without the package's build/. Skipping the build also
# keeps a tsc build error from masquerading as a property-test failure and
# filing a bogus counterexample issue.
- name: Resolve base seed and per-shard run count
id: params
# Dispatch inputs are read via env (NOT interpolated into the shell body)
# to avoid script injection through a crafted input value.
env:
SEED_INPUT: ${{ inputs.seed }}
NUM_RUNS_INPUT: ${{ inputs.num_runs }}
run: |
set -euo pipefail
SEED="${SEED_INPUT:-}"
# Empty seed (cron, or a dispatch that left it blank) -> random. Combine
# two RANDOMs so the seed spans more than RANDOM's 0..32767 range.
[ -z "$SEED" ] && SEED=$(( (RANDOM << 15) | RANDOM ))
NUM_RUNS="${NUM_RUNS_INPUT:-}"
[ -z "$NUM_RUNS" ] && NUM_RUNS=600
echo "seed=$SEED" >> "$GITHUB_OUTPUT"
echo "num_runs=$NUM_RUNS" >> "$GITHUB_OUTPUT"
echo "Sharded property fuzz: BASE_SEED=$SEED PER_SHARD_NUM_RUNS=$NUM_RUNS SHARDS=8"
- name: Run generative property suite (sharded)
id: fuzz
env:
BASE_SEED: ${{ steps.params.outputs.seed }}
PER_SHARD_NUM_RUNS: ${{ steps.params.outputs.num_runs }}
SHARDS: '8'
run: |
set -uo pipefail
# Give each fresh process headroom, but rely on SHARDING (not a big heap)
# to avoid OOM: a moderate per-shard count in a process that starts clean.
export NODE_OPTIONS=--max-old-space-size=4096
: > property-output.txt
FAILED=0
FAIL_SEED=""
i=0
while [ "$i" -lt "$SHARDS" ]; do
SHARD_SEED=$(( BASE_SEED + i ))
echo "=== shard $((i + 1))/$SHARDS: PROPERTY_SEED=$SHARD_SEED PROPERTY_NUM_RUNS=$PER_SHARD_NUM_RUNS ==="
# tee OVERWRITES property-output.txt each shard; since we break on the
# first failure, the file ends up holding exactly the failing shard's
# output (which carries the shrunk counterexample + reproducing seed).
if PROPERTY_SEED="$SHARD_SEED" PROPERTY_NUM_RUNS="$PER_SHARD_NUM_RUNS" \
pnpm --filter @docmost/prosemirror-markdown exec \
vitest run test/generative/ 2>&1 | tee property-output.txt; then
echo "shard $((i + 1)) passed"
else
echo "shard $((i + 1)) FAILED (seed=$SHARD_SEED) — stopping; keeping its output"
FAILED=1
FAIL_SEED="$SHARD_SEED"
break
fi
i=$(( i + 1 ))
done
echo "failed=$FAILED" >> "$GITHUB_OUTPUT"
echo "fail_seed=$FAIL_SEED" >> "$GITHUB_OUTPUT"
exit "$FAILED"
# A GENUINE counterexample: fast-check printed a shrunk minimal case and its
# reproducing seed into property-output.txt. File a dedup-guarded issue whose
# title prefix is UNIQUE to counterexamples, so an infra failure (handled by
# the next step under a different title) can never poison this dedup.
- name: File counterexample issue
# always() is REQUIRED: the fuzz step exits nonzero on a failing shard,
# so a bare `if:` (implicitly success() && ...) would skip this step
# exactly when it must run. always() lets it run on the failure path.
if: always() && steps.fuzz.outputs.failed == '1'
env:
FAIL_SEED: ${{ steps.fuzz.outputs.fail_seed }}
NUM_RUNS: ${{ steps.params.outputs.num_runs }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TITLE_PREFIX: 'Nightly property counterexample'
run: |
set -uo pipefail
# Discriminate counterexample vs infra failure by the fast-check
# signature. No signature -> leave it to the infra-failure step.
if ! grep -Eq 'Property failed after|Counterexample' property-output.txt; then
echo "No fast-check counterexample signature — infra failure, handled by the next step."
exit 0
fi
TITLE="${TITLE_PREFIX} (seed=${FAIL_SEED})"
# Best-effort dedup: skip if an open issue with the counterexample title
# prefix already exists. A failure of this check must NOT block creation.
EXISTING=""
if EXISTING=$(curl -sS \
-H "Authorization: token ${GITHUB_TOKEN}" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues?state=open&limit=100"); then
if printf '%s' "$EXISTING" \
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const p=process.env.TITLE_PREFIX;process.exit(a.some(i=>typeof i.title==="string"&&i.title.startsWith(p))?0:1)})'; then
echo "An open '${TITLE_PREFIX}' issue already exists — skipping creation."
exit 0
fi
fi
# Build the JSON body with the test output SAFELY escaped (never hand-
# interpolate the counterexample into JSON).
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed with a fast-check counterexample.\n\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nReproduce locally:\n\n```\nPROPERTY_SEED=%s PROPERTY_NUM_RUNS=%s pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/\n```\n\nfast-check shrinks the failure to a minimal counterexample. Commit it as a permanent fixture under `packages/prosemirror-markdown/test/fixtures/counterexamples/` + a case in `counterexamples.test.ts`, then FIX the converter (do not weaken a property). See `packages/prosemirror-markdown/README.md`.\n\nTail of the test output (contains the shrunk counterexample):\n\n```\n%s\n```\n' \
"$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$FAIL_SEED" "$NUM_RUNS" "$(tail -n 120 property-output.txt)")
jq -n --arg title "$TITLE" --arg body "$BODY_TEXT" \
'{title: $title, body: $body}' > payload.json
curl -sS -X POST \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues" \
-H "Authorization: token ${GITHUB_TOKEN}" \
-H 'Content-Type: application/json' \
-d @payload.json
# An INFRA failure (OOM, tsc, install) has NO counterexample signature. File
# it under a DISTINCT title so it is visible but keeps the counterexample
# dedup (above) uncontaminated — a real counterexample can still file even
# while an infra issue is open.
- name: File infra failure issue
# always() is REQUIRED: the fuzz step exits nonzero on a failing shard,
# so a bare `if:` (implicitly success() && ...) would skip this step
# exactly when it must run. always() lets it run on the failure path.
if: always() && steps.fuzz.outputs.failed == '1'
env:
FAIL_SEED: ${{ steps.fuzz.outputs.fail_seed }}
NUM_RUNS: ${{ steps.params.outputs.num_runs }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TITLE_PREFIX: 'Nightly property run infra failure'
run: |
set -uo pipefail
# Only file when there is NO counterexample signature (else the
# counterexample step owns it).
if grep -Eq 'Property failed after|Counterexample' property-output.txt; then
echo "Counterexample present — owned by the counterexample step."
exit 0
fi
TITLE="${TITLE_PREFIX} (seed=${FAIL_SEED})"
EXISTING=""
if EXISTING=$(curl -sS \
-H "Authorization: token ${GITHUB_TOKEN}" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues?state=open&limit=100"); then
if printf '%s' "$EXISTING" \
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const p=process.env.TITLE_PREFIX;process.exit(a.some(i=>typeof i.title==="string"&&i.title.startsWith(p))?0:1)})'; then
echo "An open '${TITLE_PREFIX}' issue already exists — skipping creation."
exit 0
fi
fi
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed WITHOUT a fast-check counterexample (infra failure: OOM / build / install). This is NOT a converter round-trip bug.\n\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nInvestigate the run log (memory, dependency install, or a tsc/import error). The nightly counterexample dedup is intentionally separate from this issue.\n\nTail of the test output:\n\n```\n%s\n```\n' \
"$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$(tail -n 120 property-output.txt)")
jq -n --arg title "$TITLE" --arg body "$BODY_TEXT" \
'{title: $title, body: $body}' > payload.json
curl -sS -X POST \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues" \
-H "Authorization: token ${GITHUB_TOKEN}" \
-H 'Content-Type: application/json' \
-d @payload.json
+43
View File
@@ -13,6 +13,49 @@ permissions:
contents: read
jobs:
# Guard against a long-lived branch adding a migration whose timestamped
# filename sorts BEFORE migrations already applied on the target branch (and
# thus in prod). The Kysely startup migrator rejects that as "corrupted
# migrations" and crash-loops the app on boot (incident #361). This gate fails
# the PR so the migration is renamed to a current timestamp before merge. Only
# runs for pull_request events (needs a base branch to diff against).
migration-order:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Checkout (full history for the base-branch diff)
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Added migrations must sort after the newest on the base branch
env:
TARGET_BRANCH: ${{ github.base_ref }}
run: |
set -euo pipefail
MIG_DIR="apps/server/src/database/migrations"
# checkout above already did fetch-depth:0 (full history). Fetch the base
# WITHOUT --depth (a shallow graft would truncate the base history and
# break the merge-base when the base has moved ahead of the PR merge —
# exactly the long-branch-vs-moving-base case this gate guards, #361).
git fetch --no-tags origin "$TARGET_BRANCH"
newest_on_target=$(git ls-tree -r --name-only "origin/${TARGET_BRANCH}" "$MIG_DIR" | sort | tail -1)
# NO `|| true`: a diff failure (e.g. an unresolved merge-base) must fail
# the job CLOSED — a gate whose job is to BLOCK must never pass on error.
# `set -e` above already aborts on a non-zero diff exit.
added=$(git diff --diff-filter=A --name-only "origin/${TARGET_BRANCH}...HEAD" -- "$MIG_DIR")
bad=0
for f in $added; do
if [[ "$f" < "$newest_on_target" || "$f" == "$newest_on_target" ]]; then
echo "::error::Migration $f sorts at or before the newest on ${TARGET_BRANCH} ($newest_on_target) — rename it with a CURRENT timestamp before merge (do not change its contents). See incident #361."
bad=1
fi
done
if [ "$bad" -eq 0 ]; then
echo "Migration order OK (added migrations all sort after $newest_on_target)."
fi
exit $bad
test:
runs-on: ubuntu-latest
timeout-minutes: 20
+30 -3
View File
@@ -214,6 +214,12 @@ Run from the repo root unless noted. The dev workflow needs **Postgres (with the
> server, `APP_SECRET` mismatch between processes, a stale `editor-ext` white-
> screening the client, LAN exposure. See **[docs/dev-stand.md](docs/dev-stand.md)**
> for the step-by-step and the traps.
>
> **Testing the app against a stand** (browser E2E + out-of-band verification) has
> its own non-obvious traps — the page has two ProseMirror editors (only the body is
> collab-bound), a ~10s store debounce, and API-seeding the thing under test is a
> silent no-test. See **[docs/how-to-test.md](docs/how-to-test.md)** before writing
> UI tests.
```bash
pnpm install # install all workspaces (uses pnpm patches; see package.json `pnpm.patchedDependencies`)
@@ -224,6 +230,24 @@ pnpm build # nx run-many -t build (all packages)
pnpm collab:dev # run the collaboration server process standalone (see "Two server processes")
```
> **Build the shared packages before running a consumer's `tsc`/tests in
> isolation.** The `build/` dirs of `@docmost/prosemirror-markdown`,
> `@docmost/git-sync`, and `@docmost/mcp` are **gitignored** (not committed), and
> a single-package `pnpm --filter <pkg> test` / `tsc` or a bare `pnpm -r test`
> does **NOT** honour the Nx `dependsOn: ["^build"]` ordering. So a consumer — the
> server's `tsc`, `git-sync`'s vitest typecheck, `mcp`'s `pretest: tsc` — fails
> with `error TS2307: Cannot find module '@docmost/…'` until those packages are
> built first:
> ```bash
> pnpm --filter @docmost/prosemirror-markdown build
> pnpm --filter @docmost/editor-ext build
> pnpm --filter @docmost/git-sync build && pnpm --filter @docmost/mcp build
> ```
> `pnpm build` (nx run-many) does this for you; CI does it explicitly in
> `.github/workflows/test.yml` (prosemirror-markdown → git-sync/mcp → server, in
> that order). Reach for it whenever you run a consumer package's checks on their
> own rather than through the full `pnpm build`.
**Lint** (per package — there is no root lint script):
```bash
pnpm --filter server lint # eslint --fix on server .ts
@@ -250,7 +274,10 @@ pnpm --filter server migration:codegen # regenerate src/databa
```
Migration files live in `apps/server/src/database/migrations/` and are named `YYYYMMDDThhmmss-description.ts`. Fork-specific migrations only **add** tables (`page_embeddings`, `ai_chats`, `ai_chat_messages`, `ai_provider_credentials`, `ai_mcp_servers`, `page_template_references`) and columns (e.g. `pages.is_template`, a `NOT NULL DEFAULT false` boolean) — never drop/rewrite Docmost data.
**Migration ordering — always check when merging branches/features.** Kysely runs migrations in **alphabetical (= timestamp) order** and refuses to start if a *new* migration sorts **before** one already applied to the DB (`corrupted migrations: ... must always have a name that comes alphabetically after the last executed migration`). When you merge a branch or land a feature, verify your migration's timestamp still sorts **after every migration that may already be applied on the target** (`/bin/ls -1 apps/server/src/database/migrations | sort | tail`). Branches developed in parallel routinely break this: a feature branch adds `…T130000-…`, `main` meanwhile ships and deploys `…T150000-…`, and after the merge the older-timestamped file is rejected at boot. **Fix = rename your migration to a timestamp after the latest one already in the target** (content unchanged — the filename is the ordering key), then rebuild so the compiled `dist/database/migrations/` picks up the new name.
**Migration ordering — always check when merging branches/features.** Kysely runs migrations in **alphabetical (= timestamp) order**. A *new* migration that sorts **before** one already applied to the DB is a "back-dated" migration, which branches developed in parallel routinely produce: a feature branch adds `…T130000-…`, `develop` meanwhile ships and deploys `…T150000-…`, and after the merge the older-timestamped file has been skipped. Two layers guard this (both added for incident #361, where a back-dated migration crash-looped prod for ~11 min):
- **CI gate (primary):** the `migration-order` job in `.github/workflows/test.yml` fails a PR whose added migration sorts at/before the newest on the base branch. **So the fix is to rename your migration to a timestamp after the latest one already in the target** (`/bin/ls -1 apps/server/src/database/migrations | sort | tail`; content unchanged — the filename is the ordering key), then rebuild so the compiled `dist/database/migrations/` picks up the new name.
- **Runtime safety net:** both Migrators (`migration.service.ts` startup auto-migrate + `migrate.ts` CLI) set `allowUnorderedMigrations: true`, so the app does **not** refuse to start on an out-of-order migration — it applies the skipped older one instead of crash-looping. Kysely's `#ensureNoMissingMigrations` guard is still on (a *removed* applied migration is still an error). Because apply order can then differ from lexicographic across instances, migrations must stay **independent** (each creates its own objects) — the CI gate remains the primary line; this net only covers a gate bypass (manual push / hotfix branch).
## Architecture — the big picture
@@ -284,7 +311,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
@@ -294,7 +321,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- **Errors must never be swallowed or shown as generic messages.** Every caught error MUST (1) be logged in full to the console/logger — error name, message, stack, `cause`, and (for HTTP/provider failures) the status code and response body — and (2) be surfaced to the user with a *specific, human-readable explanation of what actually went wrong*, never a bare generic string like "Something went wrong" / "Could not start recording" / "Transcription failed". Include the real reason (the underlying error/provider message) in the user-facing text. On the server, wrap third-party/provider failures with `describeProviderError` (or equivalent) and rethrow as a meaningful HTTP status + message — never let them collapse into an opaque 500. On the client, `console.error(<context>, err)` the raw error AND show the extracted reason (e.g. `err.response?.data?.message`, or the error `name: message`) in the notification.
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire test: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`) — it MUST be re-created via `pnpm patch` when bumping `ai`.
- **Adding/renaming/removing an MCP tool requires updating `SERVER_INSTRUCTIONS`** in `packages/mcp/src/index.ts` — the intent-routing guide MCP clients receive on initialize. This applies both to inline `server.registerTool(...)` calls in `index.ts` and to specs in `packages/mcp/src/tool-specs.ts`. Enforced by `packages/mcp/test/unit/server-instructions.test.mjs`, which fails when a registered tool is not mentioned in the guide (deliberate opt-outs go into its `EXCEPTIONS` list). `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
+8
View File
@@ -169,6 +169,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **The server no longer runs out of heap during long autonomous agent runs.** A
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
snapshot of the ENTIRE turn text on every streamed text-delta when no output
strategy was requested (our server never requests one). Unpatched, those
O(n²) `partialOutput` snapshots piled up in a never-consumed internal
`tee()` branch of the stream result — a ~20-step, ~28k-chunk agent run
retained ~1.7 GB and OOM'd the 2 GB JS heap. Streaming granularity is
unchanged; the patch must be re-created if `ai` is ever bumped. (#184)
- **Internal links in exported Markdown no longer lose their visible text.** A
link whose target page name had no file extension (e.g. a bare title) was
collapsed to empty text during export, producing an unclickable, label-less
+26
View File
@@ -125,6 +125,32 @@ Gitmost follows the upstream Docmost setup. See the Docmost
[documentation](https://docmost.com/docs) for self-hosting and development instructions; replace the
`docmost/docmost` image with `ghcr.io/vvzvlad/gitmost` where applicable.
### Reverse proxy: SSE streaming paths
The AI agent streams its answers over Server-Sent Events. These endpoints produce a
long-lived `text/event-stream` response and **must bypass response buffering AND response
compression** at every proxy in front of the app:
- `POST /api/ai-chat/stream` — the live agent turn stream
- `GET /api/ai-chat/runs/<chatId>/stream` — attach/resume of a detached agent run
(`AI_CHAT_RESUMABLE_STREAM`)
- `POST /api/shares/ai/stream` — the anonymous public-share assistant
A buffering or compressing proxy does not break these with an error — it silently ruins them:
the request hangs in `pending`, tokens stop streaming and arrive in one burst when the turn
ends, or a reloaded tab falls back to coarse polling. The tell in DevTools is a
`Content-Encoding: gzip/zstd` response header on a `text/event-stream` response.
The server already sends `X-Accel-Buffering: no` (honored by nginx unless ignored), but
compression middleware is applied by proxy configuration, not headers:
- **nginx** — `proxy_buffering off; proxy_cache off; gzip off;` for these locations, e.g.
`location ~ ^/api/(ai-chat/(stream$|runs/.+/stream$)|shares/ai/) { ... }`
- **Traefik** — route these paths through a dedicated router **without** the `compress`
middleware (a `compress` middleware buffers SSE frames until the response closes), e.g.
``PathPrefix(`/api/ai-chat/stream`) || PathPrefix(`/api/ai-chat/runs/`)``. Belt-and-braces:
`traefik.http.middlewares.<name>.compress.excludedcontenttypes: text/event-stream`.
## Migration from Docmost
Gitmost's database schema is a **strict superset** of Docmost's. Every Gitmost-specific migration
+26
View File
@@ -126,6 +126,32 @@ Gitmost повторяет процесс установки upstream-Docmost.
смотрите в [документации](https://docmost.com/docs) Docmost; где это применимо, заменяйте образ
`docmost/docmost` на `ghcr.io/vvzvlad/gitmost`.
### Reverse proxy: SSE-стриминговые пути
AI-агент стримит ответы через Server-Sent Events. Эти эндпоинты отдают долгоживущий
`text/event-stream`-ответ и **обязаны обходить буферизацию И сжатие ответов** на каждом
прокси перед приложением:
- `POST /api/ai-chat/stream` — живой стрим хода агента
- `GET /api/ai-chat/runs/<chatId>/stream` — подключение/резюм detached-рана
(`AI_CHAT_RESUMABLE_STREAM`)
- `POST /api/shares/ai/stream` — анонимный ассистент публичных шар
Буферизующий или сжимающий прокси не ломает эти пути с ошибкой — он тихо их портит:
запрос висит в `pending`, токены не стримятся и вываливаются одним куском в конце хода,
а перезагруженная вкладка падает в грубый поллинг. Диагностический признак в DevTools —
заголовок `Content-Encoding: gzip/zstd` на ответе с `text/event-stream`.
Сервер уже шлёт `X-Accel-Buffering: no` (nginx учитывает его по умолчанию), но
compression-мидлвари управляются конфигом прокси, а не заголовками:
- **nginx** — `proxy_buffering off; proxy_cache off; gzip off;` для этих location,
например `location ~ ^/api/(ai-chat/(stream$|runs/.+/stream$)|shares/ai/) { ... }`
- **Traefik** — вести эти пути через отдельный роутер **без** `compress`-мидлвари
(compress буферизует SSE-кадры до закрытия ответа), например
``PathPrefix(`/api/ai-chat/stream`) || PathPrefix(`/api/ai-chat/runs/`)``. Для надёжности:
`traefik.http.middlewares.<name>.compress.excludedcontenttypes: text/event-stream`.
## Миграция с Docmost
Схема БД Gitmost — это **строгий superset** схемы Docmost. Все Gitmost-специфичные миграции только
+35 -10
View File
@@ -23,18 +23,33 @@ roles:
inside it, which terms are ambiguous or have synonyms/jargon.
- Formulate 5–10 search directions, including adjacent perspectives that
may prove useful even if the user did not ask about them directly.
- Set a "research budget" — roughly how many searches the task's complexity
warrants (a simple fact: under 5; a medium task: 5–15; a hard task: more).
- Fix the "research budget" — how many searches to run. If the USER named a
budget (e.g. "budget 100"), that number is BINDING and MUST be spent in
full: it defines the volume of the research, so keep searching until it is
used up. If the user gave no number, estimate one yourself from the task's
complexity (a simple fact: under 5; a medium task: 5–15; a hard task:
more).
- Decide which languages it makes sense to search in (see below).
═══════════════════════════════════════════════
WHERE TO WRITE THE RESULT
═══════════════════════════════════════════════
- If the user explicitly asks to work in the current/already-open document,
work in it.
- If this is not specified, create a NEW document for the report.
- Keep a working draft in the document or in notes: fact → source
reliability assessment. Update the structure as you go.
- Reuse the current/already-open document ONLY if either (a) the user
explicitly asked to work in it, or (b) it is empty or has very little on
it AND its title matches the topic of the research. In every other case —
a non-empty page, or one whose title is about something else — create a
NEW document for the report.
- Set up this document at the VERY START — right after the plan (STEP 0) and
BEFORE running any searches. Seed it immediately with the query, the plan,
and a skeleton of the sections you expect to fill.
- Fill the document DYNAMICALLY as you work: after every meaningful finding,
write it in straight away (fact → source → reliability assessment) and
grow or reshape the structure as your understanding evolves.
- Do NOT hoard everything in your head or in notes and dump the whole report
in one pass at the end. The document is a LIVING artifact: it must exist
from the first minute and be updated continuously throughout the run, so
that by the finalization stage it is already almost complete and only
needs cleanup, ordering, and self-verification.
═══════════════════════════════════════════════
WORK LOOP (repeat until saturation)
@@ -53,9 +68,19 @@ roles:
HOW TO SEARCH
═══════════════════════════════════════════════
VOLUME. Execute a MINIMUM of 15 distinct searches, more for complex tasks.
Do not stop at the first plausible answer. Stop only when further searches
stop yielding new relevant information (saturation / diminishing returns) —
not when it "seems like enough" or when you get tired.
Do not stop at the first plausible answer. Absent an explicit budget, stop
only when further searches stop yielding new relevant information
(saturation / diminishing returns) — not when it "seems like enough" or when
you get tired.
MANDATORY BUDGET. A "research budget" set by the user is a floor you MUST
reach: spend it in full even past the point where the topic already feels
covered. Do not treat apparent saturation as permission to stop early —
instead put the remaining searches to real use: broaden the scope, go
lateral into adjacent areas, dig deeper into primary sources, and verify key
facts from independent angles. Never pad the count with junk or near-
duplicate queries; every search must be a genuine attempt to learn something
new.
WIDE → NARROW. Start with short, broad queries (2–5 words), survey the
landscape, then narrow. If results are scarce, broaden the phrasing; if
+35 -10
View File
@@ -23,18 +23,33 @@ roles:
inside it, which terms are ambiguous or have synonyms/jargon.
- Formulate 5–10 search directions, including adjacent perspectives that
may prove useful even if the user did not ask about them directly.
- Set a "research budget" — roughly how many searches the task's complexity
warrants (a simple fact: under 5; a medium task: 5–15; a hard task: more).
- Fix the "research budget" — how many searches to run. If the USER named a
budget (e.g. "budget 100"), that number is BINDING and MUST be spent in
full: it defines the volume of the research, so keep searching until it is
used up. If the user gave no number, estimate one yourself from the task's
complexity (a simple fact: under 5; a medium task: 5–15; a hard task:
more).
- Decide which languages it makes sense to search in (see below).
═══════════════════════════════════════════════
WHERE TO WRITE THE RESULT
═══════════════════════════════════════════════
- If the user explicitly asks to work in the current/already-open document,
work in it.
- If this is not specified, create a NEW document for the report.
- Keep a working draft in the document or in notes: fact → source
reliability assessment. Update the structure as you go.
- Reuse the current/already-open document ONLY if either (a) the user
explicitly asked to work in it, or (b) it is empty or has very little on
it AND its title matches the topic of the research. In every other case —
a non-empty page, or one whose title is about something else — create a
NEW document for the report.
- Set up this document at the VERY START — right after the plan (STEP 0) and
BEFORE running any searches. Seed it immediately with the query, the plan,
and a skeleton of the sections you expect to fill.
- Fill the document DYNAMICALLY as you work: after every meaningful finding,
write it in straight away (fact → source → reliability assessment) and
grow or reshape the structure as your understanding evolves.
- Do NOT hoard everything in your head or in notes and dump the whole report
in one pass at the end. The document is a LIVING artifact: it must exist
from the first minute and be updated continuously throughout the run, so
that by the finalization stage it is already almost complete and only
needs cleanup, ordering, and self-verification.
═══════════════════════════════════════════════
WORK LOOP (repeat until saturation)
@@ -53,9 +68,19 @@ roles:
HOW TO SEARCH
═══════════════════════════════════════════════
VOLUME. Execute a MINIMUM of 15 distinct searches, more for complex tasks.
Do not stop at the first plausible answer. Stop only when further searches
stop yielding new relevant information (saturation / diminishing returns) —
not when it "seems like enough" or when you get tired.
Do not stop at the first plausible answer. Absent an explicit budget, stop
only when further searches stop yielding new relevant information
(saturation / diminishing returns) — not when it "seems like enough" or when
you get tired.
MANDATORY BUDGET. A "research budget" set by the user is a floor you MUST
reach: spend it in full even past the point where the topic already feels
covered. Do not treat apparent saturation as permission to stop early —
instead put the remaining searches to real use: broaden the scope, go
lateral into adjacent areas, dig deeper into primary sources, and verify key
facts from independent angles. Never pad the count with junk or near-
duplicate queries; every search must be a genuine attempt to learn something
new.
WIDE → NARROW. Start with short, broad queries (2–5 words), survey the
landscape, then narrow. If results are scarce, broaden the phrasing; if
+1 -1
View File
@@ -33,4 +33,4 @@ bundles:
- en
roles:
- slug: researcher
version: 1
version: 4
@@ -16,8 +16,8 @@
"hash": "cef39fed321779631ddd1077fcba53399adf0e48b301df281c71eb042610900d"
},
"researcher": {
"version": 1,
"hash": "853658fda43ddbe0a4d08f2c6e50b5116d29a2e9ccd7f46e173e65920d8f6ace"
"version": 4,
"hash": "9446ec6d2c8a6ec548358537ac392b8bf9b4d2a832ebb105d5514eac2c76da74"
},
"structural-editor": {
"version": 4,
@@ -1373,6 +1373,39 @@
"The role catalog is unavailable": "The role catalog is unavailable",
"Please try again later.": "Please try again later.",
"No bundles available": "No bundles available",
"Content": "Content",
"Content language of the roles": "Content language of the roles",
"{{count}} updates available in {{bundles}} bundles": "{{count}} updates available in {{bundles}} bundles",
"Update all ({{count}})": "Update all ({{count}})",
"Updating {{current}}/{{total}}…": "Updating {{current}}/{{total}}…",
"{{count}} roles are installed in another language. A different language installs separately and appears as new.": "{{count}} roles are installed in another language. A different language installs separately and appears as new.",
"{{count}} roles": "{{count}} roles",
"{{count}} new — none installed": "{{count}} new — none installed",
"All installed · up to date": "All installed · up to date",
"{{count}} updates · {{installed}} up to date": "{{count}} updates · {{installed}} up to date",
"{{count}} new": "{{count}} new",
"{{count}} installed": "{{count}} installed",
"{{count}} updates": "{{count}} updates",
"Install bundle": "Install bundle",
"Install {{count}} selected": "Install {{count}} selected",
"Install bundle ({{count}})": "Install bundle ({{count}})",
"{{selected}} of {{total}} selected": "{{selected}} of {{total}} selected",
"Select all": "Select all",
"Deselect all": "Deselect all",
"Skipped": "Skipped",
"v{{version}}": "v{{version}}",
"{{count}} roles installed": "{{count}} roles installed",
"{{count}} roles installed · {{renamed}} renamed": "{{count}} roles installed · {{renamed}} renamed",
"{{count}} roles updated": "{{count}} roles updated",
"Installed {{installed}} · {{skipped}} skipped": "Installed {{installed}} · {{skipped}} skipped",
"A role named \"{{name}}\" already exists in this workspace.": "A role named \"{{name}}\" already exists in this workspace.",
"\"{{name}}\" is already installed.": "\"{{name}}\" is already installed.",
"Rename & install": "Rename & install",
"Couldn’t load the catalog": "Couldn’t load the catalog",
"Check your connection and try again. Installed roles are not affected.": "Check your connection and try again. Installed roles are not affected.",
"Retry": "Retry",
"The catalog is empty": "The catalog is empty",
"No role bundles are published for this language yet. Try switching the content language.": "No role bundles are published for this language yet. Try switching the content language.",
"Already up to date": "Already up to date",
"Updated to the latest version": "Updated to the latest version",
"This role is no longer in the catalog": "This role is no longer in the catalog",
@@ -1235,6 +1235,39 @@
"The role catalog is unavailable": "Каталог ролей недоступен",
"Please try again later.": "Попробуйте позже.",
"No bundles available": "Наборы недоступны",
"Content": "Язык контента",
"Content language of the roles": "Язык контента ролей",
"{{count}} updates available in {{bundles}} bundles": "Доступно обновлений: {{count}} в наборах: {{bundles}}",
"Update all ({{count}})": "Обновить все ({{count}})",
"Updating {{current}}/{{total}}…": "Обновление {{current}}/{{total}}…",
"{{count}} roles are installed in another language. A different language installs separately and appears as new.": "Ролей установлено на другом языке: {{count}}. Другой язык устанавливается отдельно и отображается как новый.",
"{{count}} roles": "ролей: {{count}}",
"{{count}} new — none installed": "новых: {{count}} — ничего не установлено",
"All installed · up to date": "Все установлены · актуальны",
"{{count}} updates · {{installed}} up to date": "обновлений: {{count}} · актуальны: {{installed}}",
"{{count}} new": "новых: {{count}}",
"{{count}} installed": "установлено: {{count}}",
"{{count}} updates": "обновлений: {{count}}",
"Install bundle": "Установить набор",
"Install {{count}} selected": "Установить выбранные ({{count}})",
"Install bundle ({{count}})": "Установить набор ({{count}})",
"{{selected}} of {{total}} selected": "выбрано {{selected}} из {{total}}",
"Select all": "Выбрать все",
"Deselect all": "Снять выбор",
"Skipped": "Пропущено",
"v{{version}}": "v{{version}}",
"{{count}} roles installed": "Установлено ролей: {{count}}",
"{{count}} roles installed · {{renamed}} renamed": "Установлено ролей: {{count}} · переименовано: {{renamed}}",
"{{count}} roles updated": "Обновлено ролей: {{count}}",
"Installed {{installed}} · {{skipped}} skipped": "Установлено: {{installed}} · пропущено: {{skipped}}",
"A role named \"{{name}}\" already exists in this workspace.": "Роль с именем «{{name}}» уже существует в этом рабочем пространстве.",
"\"{{name}}\" is already installed.": "«{{name}}» уже установлена.",
"Rename & install": "Переименовать и установить",
"Couldn’t load the catalog": "Не удалось загрузить каталог",
"Check your connection and try again. Installed roles are not affected.": "Проверьте подключение и попробуйте снова. Установленные роли не затронуты.",
"Retry": "Повторить",
"The catalog is empty": "Каталог пуст",
"No role bundles are published for this language yet. Try switching the content language.": "Для этого языка ещё не опубликовано ни одного набора ролей. Попробуйте сменить язык контента.",
"No roles configured": "Роли не настроены",
"Already up to date": "Уже актуальна",
"Updated to the latest version": "Обновлено до последней версии",
@@ -37,21 +37,22 @@ import {
mobileSidebarAtom,
} from "@/components/layouts/global/hooks/atoms/sidebar-atom.ts";
import { usePageQuery } from "@/features/page/queries/page-query.ts";
import {
pageEditorAtom,
readOnlyEditorAtom,
} from "@/features/editor/atoms/editor-atoms.ts";
import {
getEditorSelectionContext,
type EditorSelectionContext,
} from "@/features/editor/utils/get-editor-selection.ts";
import { extractPageSlugId } from "@/lib";
import {
AI_CHATS_RQ_KEY,
AI_CHAT_MESSAGES_RQ_KEY,
AI_CHAT_RUN_RQ_KEY,
useAiChatMessagesQuery,
useAiChatRunQuery,
useAiChatsQuery,
useAiRolesQuery,
} from "@/features/ai-chat/queries/ai-chat-query.ts";
import {
shouldClearLatchOnQueryError,
shouldClearStoppingLatch,
shouldObserveRun,
} from "@/features/ai-chat/utils/run-polling.ts";
import { workspaceAtom } from "@/features/user/atoms/current-user-atom";
import ConversationList from "@/features/ai-chat/components/conversation-list.tsx";
import ChatThread from "@/features/ai-chat/components/chat-thread.tsx";
@@ -85,6 +86,12 @@ const MIN_HEIGHT = 400;
// Margin kept between the window and the viewport edges while dragging.
const EDGE_MARGIN = 8;
// #184 phase 1.5: hard cap on the degraded-poll fallback. The poll is armed when
// a resume attempt could not attach to the live run and disarmed by the thread on
// settle / local stream; this cap is the ONLY backstop against an endless tick
// (a stuck 'streaming' row before the boot-sweep, or a user-tail 204 with no run).
const DEGRADED_POLL_MAX_MS = 10 * 60_000;
/** Compact token formatter: 1.2M / 3.4k / 950. */
function formatTokens(n: number): string {
if (n >= 1_000_000) return `${(n / 1_000_000).toFixed(1)}M`;
@@ -242,150 +249,62 @@ export default function AiChatWindow() {
[roles],
);
// #184 phase 1.5: degraded-poll fallback (replaces the F4/F5/F7 latches). When
// ChatThread could not attach to a still-running run it arms this via
// onResumeFallback(true); the thread disarms it on settle / local stream. The
// window only OWNS the timer (armedAtRef stamps when it was armed for the cap).
const [degradedPoll, setDegradedPoll] = useState(false);
const armedAtRef = useRef(0);
const onResumeFallback = useCallback((active: boolean): void => {
if (active) armedAtRef.current = Date.now();
setDegradedPoll(active);
}, []);
// Reset the degraded poll whenever the open chat changes: it is scoped to the
// resume attempt of the previously-open chat (invariant 8).
useEffect(() => {
setDegradedPoll(false);
}, [activeChatId]);
const { data: messageRows, isLoading: messagesLoading } =
useAiChatMessagesQuery(activeChatId ?? undefined);
useAiChatMessagesQuery(
activeChatId ?? undefined,
// DELIBERATELY DUMB (invariant 8 / task 2.4): poll every 2.5s while armed
// and under the 10-min cap; otherwise off. NO error checks (TanStack v5
// resets fetchFailureCount each fetch, so consecutive errors are not
// expressible — and the poll must survive a server restart) and NO tail
// checks (the settled/local-stream semantics live in ChatThread, which
// disarms via onResumeFallback(false)). The time cap is the only backstop.
() =>
degradedPoll === true &&
Date.now() - armedAtRef.current < DEGRADED_POLL_MAX_MS
? 2500
: false,
);
// #184 reconnect-and-live-follow. Whether detached agent runs are enabled for
// this workspace. The reconnect endpoint itself is NOT flag-gated server-side
// (it is only owner-gated and returns `{ run: null }` when the chat has no
// run); but when the feature is off no runs are ever created, so polling it
// would always come back empty — we gate it off here to avoid pointless polls.
// this workspace. When the feature is off no runs are ever created, so the
// resume attempt would only ever 204; gating ChatThread's resume on it avoids a
// pointless attach round-trip.
const workspace = useAtomValue(workspaceAtom);
const autonomousRunsEnabled =
workspace?.settings?.ai?.autonomousRuns === true;
// Whether THIS tab is the one actively streaming the open chat's run locally
// (it started the run here and holds the SSE). Reported up from ChatThread. We
// are the STREAMER while true and a passive OBSERVER while false — the basis of
// the observer-vs-streamer detection. Reset to false by the fresh ChatThread's
// mount effect on every chat switch.
const [localStreaming, setLocalStreaming] = useState(false);
const onStreamingChange = useCallback((streaming: boolean) => {
setLocalStreaming(streaming);
}, []);
// #184 Stop wiring. While a detached run is being stopped we SUPPRESS the
// observer merge so the stopping run's still-persisting output does not
// re-stream back into view between the moment the user pressed Stop and the run
// actually settling as 'aborted' server-side. Polling itself keeps running (so
// the terminal transition is still detected) — only the visual merge is gated.
// Cleared when the run is observed terminal (below) or the chat is switched.
const [stoppingRun, setStoppingRun] = useState(false);
// Reset the stopping latch whenever the open chat changes: it is scoped to the
// run of the previously-open chat.
useEffect(() => {
setStoppingRun(false);
}, [activeChatId]);
// Authoritative stop of the open chat's detached run (the Stop button in
// autonomous mode). Latch "stopping" first (suppresses the re-stream flash),
// then request the server stop — the ONLY thing that ends a detached run; a mere
// local SSE abort is a client disconnect the server ignores. On failure we
// release the latch so the observer resumes (better to show the live run than to
// freeze the view) and surface the error.
// autonomous mode). Request the server stop — the ONLY thing that ends a
// detached run; a mere local SSE abort is a client disconnect the server
// ignores. On failure surface the error.
const handleServerStop = useCallback(
(chatId: string): void => {
setStoppingRun(true);
// #234 F4: drop the PREVIOUS turn's run from the cache so `run` becomes null
// until the CURRENT turn's run is fetched fresh. Without this, once the local
// stream aborts (localStreaming -> false) the run query re-enables and
// react-query SYNCHRONOUSLY returns the still-cached prior terminal run; the
// terminal effect would then clear the stopping latch against that STALE run
// before the current turn's (still-running, detached, growing) run is ever
// observed — re-opening the observer merge and flashing the growing output
// over the frozen row. With the cache cleared the terminal effect's
// `if (!run) return` holds the latch until the current run itself is observed
// terminal (see shouldClearStoppingLatch).
queryClient.removeQueries({ queryKey: AI_CHAT_RUN_RQ_KEY(chatId) });
void stopRun(chatId).catch(() => {
setStoppingRun(false);
notifications.show({
message: t("Failed to stop the run"),
color: "red",
});
});
},
[t, queryClient],
[t],
);
// Poll the latest run of the open chat ONLY when we are a passive observer:
// feature on, a chat is open, and we are NOT the local streamer (the streamer
// already has the live SSE — polling/merging too would double-render). The
// query's own status-keyed refetchInterval stops once the run is terminal.
const { data: runData, isError: runQueryFailed } = useAiChatRunQuery(
activeChatId ?? undefined,
autonomousRunsEnabled && !localStreaming,
);
const run = runData?.run ?? null;
// Safety net (#234 F4 review): after handleServerStop clears the run cache,
// `run` is null until the current turn's run is fetched fresh, and the terminal
// effect below holds the latch via `if (!run) return`. If that refetch instead
// ERRORS PERMANENTLY (the GET-run keeps failing) while we are no longer the
// streamer, the run stays null, its status-keyed refetchInterval is off, and
// nothing would ever observe a terminal run — freezing the view with the
// observer merge suppressed. Release the latch on that error so the live view
// resumes rather than stays stuck (the local stopRun may already have succeeded
// independently).
//
// #234 F7: this must NOT fire on a TRANSIENT error while `run` is still an
// ACTIVE held run. In TanStack Query v5 (retry:false) the query's `data` is
// RETAINED on error, so `runQueryFailed` can be true while `run` is still
// pending/running — releasing then would re-open the observer merge and flash
// the growing detached run over the frozen row (the very flash F4 prevents). The
// decision is the pure, unit-tested `shouldClearLatchOnQueryError`, which gates
// on the run NOT being active: it cures only the genuine permanent-null-freeze
// (`run === null`) and never releases against an active run.
useEffect(() => {
if (
shouldClearLatchOnQueryError({
stoppingRun,
isLocalStreaming: localStreaming,
runQueryFailed,
run,
})
)
setStoppingRun(false);
}, [stoppingRun, localStreaming, runQueryFailed, run]);
// The run's incrementally-persisted assistant message to merge into the thread,
// but only while we are an observer (never when we are the streamer — guards
// against a stale poll fighting the live stream). Includes a terminal run so the
// final persisted output is shown on reopen.
const observedRow =
shouldObserveRun(run, localStreaming) && !stoppingRun
? (runData?.message ?? null)
: null;
// When the observed run reaches a terminal status, do a final messages refetch
// so the persisted final state (token/context badge, export source) is shown,
// then the query's refetchInterval has already stopped polling. Deduped per run
// id so it fires exactly once per run, not on every subsequent poll-less render.
const finalizedRunIdRef = useRef<string | null>(null);
useEffect(() => {
if (!run || !activeChatId) return;
if (run.status === "pending" || run.status === "running") {
// Active again (a new run) — re-arm so its terminal transition fires once.
finalizedRunIdRef.current = null;
return;
}
// Terminal: a stop we requested has landed (or the run finished on its own),
// so release the stopping latch — the observer merge can now show the final
// persisted (aborted/finished) output without any live re-stream. The decision
// is the pure, unit-tested `shouldClearStoppingLatch` (run-polling.ts): release
// ONLY when we requested a stop, this tab is no longer the streamer, AND the
// CURRENT run is terminal. The #234 F4 cache removal in handleServerStop makes
// `run` null (this branch's `if (!run) return` above holds) until the current
// turn's run is fetched fresh, so the latch can never clear against a stale
// cached run.
if (shouldClearStoppingLatch({ stoppingRun, run, isLocalStreaming: localStreaming }))
setStoppingRun(false);
if (finalizedRunIdRef.current === run.id) return;
finalizedRunIdRef.current = run.id;
queryClient.invalidateQueries({
queryKey: AI_CHAT_MESSAGES_RQ_KEY(activeChatId),
});
}, [run, activeChatId, queryClient, stoppingRun, localStreaming]);
// The page the user is currently viewing. AiChatWindow lives in a pathless
// parent layout route, so useParams() can't see :pageSlug. Match the full
// pathname against the authenticated page route instead so "the current page"
@@ -403,6 +322,27 @@ export default function AiChatWindow() {
? { id: openPageData.id, title: openPageData.title }
: null;
// Live editor handles for the selection snapshot (#388). Both are published by
// the page editor; the read-only editor is used in read mode. Reading the
// selection off `editor.state` stays valid after the editor blurs (ProseMirror
// keeps state.selection), mirroring the comment button (comment-dialog.tsx).
const pageEditor = useAtomValue(pageEditorAtom);
const readOnlyEditor = useAtomValue(readOnlyEditorAtom);
// Snapshot the user's current editor selection at send time. Edit-mode editor
// wins; the read-only editor is the fallback (read mode). Null when neither
// holds a non-empty selection. Passed to <ChatThread>, which reads it live
// from a ref inside prepareSendMessagesRequest — so each turn ships a fresh
// snapshot and multi-turn works without recreating the transport.
const getEditorSelection = useCallback((): EditorSelectionContext | null => {
for (const editor of [pageEditor, readOnlyEditor]) {
if (!editor || editor.isDestroyed) continue;
const sel = getEditorSelectionContext(editor.state);
if (sel) return sel;
}
return null;
}, [pageEditor, readOnlyEditor]);
// The AI-chat thread-identity lifecycle (mount key, both new-chat id adoption
// paths, the history-loaded latch, the render-phase reconciler) lives in this
// hook. See adopt-chat-id.ts for the canonical #137 two-tab race explanation.
@@ -1025,6 +965,9 @@ export default function AiChatWindow() {
chatId={activeChatId}
initialRows={activeChatId ? messageRows : []}
openPage={openPage}
// #388: live snapshotter for the user's editor selection, read at
// send time and nested inside openPage on the wire.
getEditorSelection={getEditorSelection}
// Honoured only for a new chat; null = universal assistant.
roleId={activeChatId === null ? selectedRoleId : null}
// Role cards are the new-chat empty-state; offered only when this
@@ -1034,16 +977,13 @@ export default function AiChatWindow() {
assistantName={currentRole?.name}
onTurnFinished={onTurnFinished}
onServerChatId={onServerChatId}
// #184: live-follow a still-running run when we reopened the chat as
// a passive observer; null when there is nothing to observe or this
// tab is the streamer. onStreamingChange lets the window stop polling
// while we are the streamer.
observedRow={observedRow}
onStreamingChange={onStreamingChange}
// #184 phase 1.5: arm/disarm the degraded-poll fallback when a
// resume attempt could not attach to the live run; the thread
// disarms it on settle / local stream.
onResumeFallback={onResumeFallback}
// #184: in autonomous mode the Stop button must hit the authoritative
// server stop (a local SSE abort is a client disconnect the server
// ignores). onServerStop also arms the "stopping" latch above so the
// stopped run's output does not re-stream via the observer merge.
// ignores).
autonomousRunsEnabled={autonomousRunsEnabled}
onServerStop={handleServerStop}
/>
@@ -1,6 +1,13 @@
import { describe, it, expect, beforeEach, vi } from "vitest";
import { render, screen, fireEvent, act, cleanup } from "@testing-library/react";
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import {
render,
screen,
fireEvent,
act,
cleanup,
} from "@testing-library/react";
import { MantineProvider } from "@mantine/core";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
// Shared, hoisted mock state so the @ai-sdk/react and "ai" module mocks (hoisted
// above the imports) can expose the captured useChat callbacks / transport and
@@ -12,50 +19,61 @@ const h = vi.hoisted(() => ({
sendMessage: vi.fn(),
stop: vi.fn(),
setMessages: vi.fn(),
resumeStream: vi.fn(),
// The messages array useChat was seeded with (to assert strip/seed behavior).
seededMessages: null as null | unknown[],
transport: null as null | {
prepareSendMessagesRequest: (arg: {
prepareSendMessagesRequest?: (arg: {
messages: unknown[];
body: Record<string, unknown>;
}) => { body: Record<string, unknown> };
prepareReconnectToStreamRequest?: () => { api?: string };
fetch?: (input: unknown, init?: { method?: string }) => Promise<unknown>;
},
},
}));
// Mock useChat: capture onFinish, return the spies and the controllable status.
// Mock useChat: capture onFinish + seeded messages, return the spies and the
// controllable status.
vi.mock("@ai-sdk/react", () => ({
useChat: (opts: { onFinish?: (arg: Record<string, unknown>) => void }) => {
useChat: (opts: {
messages?: unknown[];
onFinish?: (arg: Record<string, unknown>) => void;
}) => {
h.state.onFinish = opts.onFinish ?? null;
h.state.seededMessages = opts.messages ?? null;
return {
messages: [],
sendMessage: h.state.sendMessage,
status: h.state.status,
stop: h.state.stop,
error: null,
// #184: ChatThread reads setMessages to merge a polled observer run.
setMessages: h.state.setMessages,
resumeStream: h.state.resumeStream,
};
},
}));
// Mock "ai": deterministic ids + a transport that records its options so the test
// can invoke prepareSendMessagesRequest and assert the `interrupted` flag.
// can invoke prepareSendMessagesRequest / prepareReconnectToStreamRequest / fetch.
vi.mock("ai", () => {
let counter = 0;
return {
generateId: () => `gid-${counter++}`,
DefaultChatTransport: class {
constructor(opts: {
prepareSendMessagesRequest: (arg: {
messages: unknown[];
body: Record<string, unknown>;
}) => { body: Record<string, unknown> };
}) {
h.state.transport = opts;
constructor(opts: Record<string, unknown>) {
h.state.transport = opts as never;
}
},
};
});
// Keep the ai-chat-query import light: ChatThread only needs the messages RQ key,
// so stub the module to avoid pulling axios / i18n transitively.
vi.mock("@/features/ai-chat/queries/ai-chat-query.ts", () => ({
AI_CHAT_MESSAGES_RQ_KEY: (chatId: string) => ["ai-chat-messages", chatId],
}));
// Stub the heavy children: MessageList (markdown/render) and ChatInput (the
// composer). The ChatInput stub exposes a button that queues a message, the only
// interaction this test needs to populate the queue while "streaming".
@@ -63,49 +81,90 @@ vi.mock("@/features/ai-chat/components/message-list.tsx", () => ({
default: () => <div data-testid="message-list" />,
}));
vi.mock("@/features/ai-chat/components/chat-input.tsx", () => ({
default: ({ onQueue }: { onQueue: (text: string) => void }) => (
<button data-testid="queue-btn" onClick={() => onQueue("queued text")}>
queue
</button>
default: ({
onQueue,
onStop,
}: {
onQueue: (text: string) => void;
onStop: () => void;
}) => (
<>
<button data-testid="queue-btn" onClick={() => onQueue("queued text")}>
queue
</button>
<button aria-label="Stop" onClick={() => onStop()}>
stop
</button>
</>
),
}));
import ChatThread from "./chat-thread";
import type { IAiChatMessageRow } from "@/features/ai-chat/types/ai-chat.types.ts";
function renderThread() {
function row(
id: string,
role: string,
status?: string,
text = "",
): IAiChatMessageRow {
return { id, role, content: text, status, createdAt: "2026-01-01T00:00:00Z" };
}
function renderThread(props?: {
chatId?: string | null;
initialRows?: IAiChatMessageRow[];
autonomousRunsEnabled?: boolean;
}) {
const onTurnFinished = vi.fn();
render(
<MantineProvider>
<ChatThread chatId="c1" initialRows={[]} onTurnFinished={onTurnFinished} />
</MantineProvider>,
const onResumeFallback = vi.fn();
const onServerStop = vi.fn();
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
const invalidateSpy = vi.spyOn(queryClient, "invalidateQueries");
const { unmount } = render(
<QueryClientProvider client={queryClient}>
<MantineProvider>
<ChatThread
chatId={props?.chatId === undefined ? "c1" : props.chatId}
initialRows={props?.initialRows ?? []}
autonomousRunsEnabled={props?.autonomousRunsEnabled}
onTurnFinished={onTurnFinished}
onResumeFallback={onResumeFallback}
onServerStop={onServerStop}
/>
</MantineProvider>
</QueryClientProvider>,
);
return { onTurnFinished };
return { onTurnFinished, onResumeFallback, onServerStop, invalidateSpy, unmount };
}
function resetState() {
h.state.status = "streaming";
h.state.onFinish = null;
h.state.seededMessages = null;
h.state.transport = null;
h.state.sendMessage.mockClear();
h.state.stop.mockClear();
h.state.setMessages.mockClear();
h.state.resumeStream.mockClear();
}
describe("ChatThread — send now (#198)", () => {
beforeEach(() => {
h.state.status = "streaming";
h.state.onFinish = null;
h.state.sendMessage.mockClear();
h.state.stop.mockClear();
h.state.transport = null;
});
beforeEach(resetState);
it("aborts the current turn and resends the queued message on the abort", () => {
renderThread();
// Queue a message while the turn is streaming.
fireEvent.click(screen.getByTestId("queue-btn"));
const sendNowBtn = screen.getByLabelText("Send now");
expect(sendNowBtn).toBeTruthy();
// "Send now" interrupts the current turn (stop), but does NOT send yet —
// the resend happens once the abort lands in onFinish.
fireEvent.click(sendNowBtn);
expect(h.state.stop).toHaveBeenCalledTimes(1);
expect(h.state.sendMessage).not.toHaveBeenCalled();
// The abort we triggered reaches onFinish: the promoted head is flushed.
act(() => {
h.state.onFinish?.({
message: { id: "a", role: "assistant", parts: [] },
@@ -122,10 +181,8 @@ describe("ChatThread — send now (#198)", () => {
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
const prep = h.state.transport!.prepareSendMessagesRequest;
// The send right after "send now" carries interrupted: true...
const prep = h.state.transport!.prepareSendMessagesRequest!;
expect(prep({ messages: [], body: {} }).body.interrupted).toBe(true);
// ...and only that one (the flag is read-and-cleared).
expect(prep({ messages: [], body: {} }).body.interrupted).toBe(false);
});
@@ -136,42 +193,92 @@ describe("ChatThread — send now (#198)", () => {
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
// No turn to interrupt: sent straight away, no abort, not flagged.
expect(h.state.stop).not.toHaveBeenCalled();
expect(h.state.sendMessage).toHaveBeenCalledWith({ text: "queued text" });
const prep = h.state.transport!.prepareSendMessagesRequest;
const prep = h.state.transport!.prepareSendMessagesRequest!;
expect(prep({ messages: [], body: {} }).body.interrupted).toBe(false);
});
});
// The turn-end decision lives in the `onFinish` handler: given the terminal
// outcome of a turn (`isAbort` / `isDisconnect` / `isError`, or none = clean),
// it decides whether to CONTINUE (flush the next queued message) or END (leave
// the queue intact for the user), and which stop notice — if any — to show.
// `sendNow` is exercised above; these tests pin down the plain outcomes.
describe("ChatThread — turn-end decision (onFinish)", () => {
beforeEach(() => {
h.state.status = "streaming";
h.state.onFinish = null;
h.state.sendMessage.mockClear();
h.state.stop.mockClear();
h.state.transport = null;
// #388: the editor selection is snapshotted at send time and nested inside
// openPage on the wire. The getter is read live from a ref, so each send ships a
// fresh snapshot.
describe("ChatThread — editor selection wiring (#388)", () => {
beforeEach(resetState);
afterEach(cleanup);
function renderWithSelection(props: {
openPage?: { id: string; title: string } | null;
getEditorSelection?: () => unknown;
}) {
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
render(
<QueryClientProvider client={queryClient}>
<MantineProvider>
<ChatThread
chatId="c1"
initialRows={[]}
openPage={props.openPage as never}
getEditorSelection={props.getEditorSelection as never}
onTurnFinished={vi.fn()}
onResumeFallback={vi.fn()}
onServerStop={vi.fn()}
/>
</MantineProvider>
</QueryClientProvider>,
);
}
it("nests the snapshot from the getter into openPage.selection at send time", () => {
const selection = { text: "fix this", blockIds: ["b1"], before: "a " };
renderWithSelection({
openPage: { id: "p1", title: "Doc" },
getEditorSelection: () => selection,
});
const prep = h.state.transport!.prepareSendMessagesRequest!;
const openPage = prep({ messages: [], body: {} }).body.openPage as Record<
string,
unknown
>;
expect(openPage).toEqual({ id: "p1", title: "Doc", selection });
});
// Drive a fresh onFinish with the given terminal flags after queueing a
// message, and report both what the parent was told and whether the queue was
// flushed (a resend to the sendMessage spy).
it("sends selection: null when the getter returns null", () => {
renderWithSelection({
openPage: { id: "p1", title: "Doc" },
getEditorSelection: () => null,
});
const prep = h.state.transport!.prepareSendMessagesRequest!;
const openPage = prep({ messages: [], body: {} }).body.openPage as Record<
string,
unknown
>;
expect(openPage).toEqual({ id: "p1", title: "Doc", selection: null });
});
it("does not send selection at all on a non-page route (openPage null)", () => {
const getter = vi.fn(() => ({ text: "sel" }));
renderWithSelection({ openPage: null, getEditorSelection: getter });
const prep = h.state.transport!.prepareSendMessagesRequest!;
expect(prep({ messages: [], body: {} }).body.openPage).toBeNull();
// The getter must not even be consulted when there is no page.
expect(getter).not.toHaveBeenCalled();
});
});
describe("ChatThread — turn-end decision (onFinish)", () => {
beforeEach(resetState);
function finishWith(flags: {
isAbort?: boolean;
isDisconnect?: boolean;
isError?: boolean;
}) {
// Tear down any prior render so the loop-driven "every outcome" case does
// not leave duplicate queue buttons in the DOM.
cleanup();
h.state.sendMessage.mockClear();
const { onTurnFinished } = renderThread();
// Populate the queue while the turn is streaming.
fireEvent.click(screen.getByTestId("queue-btn"));
act(() => {
h.state.onFinish?.({
@@ -187,16 +294,12 @@ describe("ChatThread — turn-end decision (onFinish)", () => {
it("CONTINUES — flushes the next queued message on a clean finish", () => {
finishWith({});
// Clean finish (no terminal flag): the queued message is auto-sent.
expect(h.state.sendMessage).toHaveBeenCalledWith({ text: "queued text" });
// A clean finish shows no stop notice.
expect(screen.queryByText("Response stopped.")).toBeNull();
});
it("ENDS — keeps the queue intact on a user abort and shows the stopped notice", () => {
finishWith({ isAbort: true });
// A plain Stop (not the sendNow interrupt path) must NOT auto-resend: the
// queue is preserved for the user to decide.
expect(h.state.sendMessage).not.toHaveBeenCalled();
expect(screen.getByText("Response stopped.")).toBeTruthy();
});
@@ -211,15 +314,11 @@ describe("ChatThread — turn-end decision (onFinish)", () => {
it("ENDS — keeps the queue intact on a stream error (no auto-retry, no stopped notice)", () => {
finishWith({ isError: true });
// Blindly retrying after a failure would be wrong; the queue is left alone.
expect(h.state.sendMessage).not.toHaveBeenCalled();
// isError clears the neutral notice (the error banner covers this case).
expect(screen.queryByText("Response stopped.")).toBeNull();
});
it("notifies the parent on EVERY terminal outcome", () => {
// The chat-list refresh / new-chat id adoption must run on success and on
// every failure path alike.
for (const flags of [
{},
{ isAbort: true },
@@ -232,55 +331,411 @@ describe("ChatThread — turn-end decision (onFinish)", () => {
});
});
// #184 passive-observer merge: when reconnecting to a still-running run, the
// parent feeds the polled run message via `observedRow`; ChatThread merges it via
// setMessages — but ONLY when this tab is NOT itself streaming (the streamer's
// SSE owns the view, so a stale observedRow must never overwrite it).
describe("ChatThread — observer run merge (#184)", () => {
beforeEach(() => {
h.state.onFinish = null;
h.state.setMessages.mockReset();
// #184 phase 1.5: the resumable-SSE client. A reopened tab resumes the live run
// via the SDK's reconnect transport (attach: replay + tail) instead of polling.
describe("ChatThread — resume (attach) machinery (#184)", () => {
const streamingTail = () => [
row("u1", "user", undefined, "hi"),
row("a1", "assistant", "streaming", "partial"),
];
const settledTail = () => [
row("u1", "user", undefined, "hi"),
row("a1", "assistant", "succeeded", "done"),
];
const userTail = () => [row("u1", "user", undefined, "hi")];
const visibleMsg = {
id: "a1",
role: "assistant",
parts: [{ type: "text", text: "streamed answer" }],
};
const emptyMsg = { id: "a1", role: "assistant", parts: [] };
beforeEach(resetState);
// NOTE: do NOT vi.unstubAllGlobals() here — vitest.setup.ts installs
// matchMedia/localStorage via vi.stubGlobal and unstubbing wipes them for the
// rest of the file. Fetch is re-stubbed per test that needs it.
afterEach(cleanup);
it("resumes on mount only when the flag is on, chatId is set, and the tail is not a settled assistant", () => {
// streaming tail -> resume
renderThread({ autonomousRunsEnabled: true, initialRows: streamingTail() });
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
// user tail -> resume (the assistant row may not be seeded yet)
cleanup();
h.state.resumeStream.mockClear();
renderThread({ autonomousRunsEnabled: true, initialRows: userTail() });
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
// settled assistant tail -> NO resume
cleanup();
h.state.resumeStream.mockClear();
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
expect(h.state.resumeStream).not.toHaveBeenCalled();
// flag off -> NO resume
cleanup();
h.state.resumeStream.mockClear();
renderThread({ autonomousRunsEnabled: false, initialRows: streamingTail() });
expect(h.state.resumeStream).not.toHaveBeenCalled();
// no chatId -> NO resume
cleanup();
h.state.resumeStream.mockClear();
renderThread({
autonomousRunsEnabled: true,
chatId: null,
initialRows: streamingTail(),
});
expect(h.state.resumeStream).not.toHaveBeenCalled();
});
const observedRow = {
id: "a-run",
role: "assistant",
content: "step 1\nstep 2",
metadata: {
parts: [{ type: "text", text: "step 1\nstep 2" }],
},
createdAt: "2026-01-01T00:00:00Z",
} as const;
it("strips the streaming tail from the seed, but keeps a user tail whole", () => {
renderThread({ autonomousRunsEnabled: true, initialRows: streamingTail() });
// 2 rows in, streaming tail stripped -> 1 seeded message.
expect(h.state.seededMessages).toHaveLength(1);
function renderObserver(status: string) {
h.state.status = status;
render(
cleanup();
renderThread({ autonomousRunsEnabled: true, initialRows: userTail() });
// user tail is not stripped.
expect(h.state.seededMessages).toHaveLength(1);
});
it("builds the attach URL with expect=live&anchor only when the streaming tail was stripped", () => {
renderThread({ autonomousRunsEnabled: true, initialRows: streamingTail() });
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream?expect=live&anchor=a1",
);
cleanup();
renderThread({ autonomousRunsEnabled: true, initialRows: userTail() });
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream",
);
});
async function fetch204() {
vi.stubGlobal(
"fetch",
vi.fn().mockResolvedValue({ status: 204, ok: false }),
);
await act(async () => {
await h.state.transport!.fetch!("http://x", { method: "GET" });
});
}
it("204 on a user tail: no crash, no restore, reconcile+invalidate, onResumeFallback(true)", async () => {
const { onResumeFallback, invalidateSpy } = renderThread({
autonomousRunsEnabled: true,
initialRows: userTail(),
});
await fetch204();
// No stripped row -> no restore merge.
expect(h.state.setMessages).not.toHaveBeenCalled();
expect(invalidateSpy).toHaveBeenCalledWith({
queryKey: ["ai-chat-messages", "c1"],
});
expect(onResumeFallback).toHaveBeenCalledWith(true);
});
it("204 on a streaming tail: restore + invalidate + onResumeFallback(true)", async () => {
const { onResumeFallback, invalidateSpy } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
await fetch204();
// Stripped row is restored to the store.
expect(h.state.setMessages).toHaveBeenCalledTimes(1);
expect(invalidateSpy).toHaveBeenCalledWith({
queryKey: ["ai-chat-messages", "c1"],
});
expect(onResumeFallback).toHaveBeenCalledWith(true);
});
it("F7 restart-survival: a 500 attach failure restores the stripped row AND arms the poll (not lost)", async () => {
const { onResumeFallback, invalidateSpy } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
vi.stubGlobal(
"fetch",
vi.fn().mockResolvedValue({ status: 500, ok: false }),
);
await act(async () => {
await h.state.transport!.fetch!("http://x", { method: "GET" });
});
expect(h.state.setMessages).toHaveBeenCalledTimes(1); // stripped row restored
expect(invalidateSpy).toHaveBeenCalledWith({
queryKey: ["ai-chat-messages", "c1"],
});
expect(onResumeFallback).toHaveBeenCalledWith(true); // degraded poll armed
});
it("F7 restart-survival: a network throw restores the stripped row AND arms the poll", async () => {
const { onResumeFallback, invalidateSpy } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
vi.stubGlobal(
"fetch",
vi.fn().mockRejectedValue(new Error("network down")),
);
await act(async () => {
await h.state
.transport!.fetch!("http://x", { method: "GET" })
.catch(() => undefined); // the wrapper rethrows; swallow here
});
expect(h.state.setMessages).toHaveBeenCalledTimes(1);
expect(invalidateSpy).toHaveBeenCalledWith({
queryKey: ["ai-chat-messages", "c1"],
});
expect(onResumeFallback).toHaveBeenCalledWith(true);
});
it("unmount during a pending attach aborts the controller and gates late callbacks", async () => {
const { onResumeFallback, invalidateSpy, unmount } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
let abortSeen = false;
let resolveFetch!: (v: unknown) => void;
vi.stubGlobal(
"fetch",
vi.fn().mockImplementation((_input: unknown, init: RequestInit) => {
init.signal?.addEventListener("abort", () => {
abortSeen = true;
});
return new Promise((res) => {
resolveFetch = res;
});
}),
);
// Kick a reconnect GET (stays pending).
let pending!: Promise<unknown>;
act(() => {
pending = h.state.transport!.fetch!("http://x", { method: "GET" });
});
// Unmount: the cleanup aborts the in-flight attach.
unmount();
expect(abortSeen).toBe(true);
// A late 204 landing after unmount must NOT arm a poll / invalidate the (now
// different) chat.
onResumeFallback.mockClear();
invalidateSpy.mockClear();
await act(async () => {
resolveFetch({ status: 204, ok: false });
await pending;
});
expect(onResumeFallback).not.toHaveBeenCalledWith(true);
expect(invalidateSpy).not.toHaveBeenCalled();
});
it("a resume fetch error clears resumedTurn so the next local turn flushes the queue", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: streamingTail() });
h.state.status = "ready";
vi.stubGlobal(
"fetch",
vi.fn().mockResolvedValue({ status: 500, ok: false }),
);
await act(async () => {
await h.state.transport!.fetch!("http://x", { method: "GET" });
});
// Queue then clean-finish: suppression was cleared, so the queue flushes.
fireEvent.click(screen.getByTestId("queue-btn"));
act(() => {
h.state.onFinish?.({
message: visibleMsg,
isAbort: false,
isDisconnect: false,
isError: false,
});
});
expect(h.state.sendMessage).toHaveBeenCalledWith({ text: "queued text" });
});
it("a resumed turn's onFinish does NOT flush the queue", () => {
renderThread({ autonomousRunsEnabled: true, initialRows: streamingTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
act(() => {
h.state.onFinish?.({
message: visibleMsg,
isAbort: false,
isDisconnect: false,
isError: false,
});
});
expect(h.state.sendMessage).not.toHaveBeenCalled();
});
it("a healthy resumed finish (visible content) arms nothing and keeps the store", () => {
h.state.status = "ready";
const { onResumeFallback } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
h.state.setMessages.mockClear();
onResumeFallback.mockClear();
act(() => {
h.state.onFinish?.({
message: visibleMsg,
isAbort: false,
isDisconnect: false,
isError: false,
});
});
// No restore (would clobber the fuller streamed message), no poll arm.
expect(h.state.setMessages).not.toHaveBeenCalled();
expect(onResumeFallback).not.toHaveBeenCalledWith(true);
});
it("isDisconnect WITH visible content arms the poll but does NOT restore", () => {
h.state.status = "ready";
const { onResumeFallback, invalidateSpy } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
h.state.setMessages.mockClear();
onResumeFallback.mockClear();
invalidateSpy.mockClear();
act(() => {
h.state.onFinish?.({
message: visibleMsg,
isAbort: false,
isDisconnect: true,
isError: false,
});
});
expect(onResumeFallback).toHaveBeenCalledWith(true);
expect(invalidateSpy).toHaveBeenCalledWith({
queryKey: ["ai-chat-messages", "c1"],
});
// Restore forbidden: the on-screen partial must not roll back.
expect(h.state.setMessages).not.toHaveBeenCalled();
});
it("an empty resumed message (starved replay) restores the stripped row AND arms the poll", () => {
h.state.status = "ready";
const { onResumeFallback } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
h.state.setMessages.mockClear();
onResumeFallback.mockClear();
act(() => {
h.state.onFinish?.({
message: emptyMsg,
isAbort: false,
isDisconnect: false,
isError: false,
});
});
expect(h.state.setMessages).toHaveBeenCalledTimes(1); // restore
expect(onResumeFallback).toHaveBeenCalledWith(true); // arm
});
it("degraded-merge: merges the tail per initialRows update, and settles disarm the poll", async () => {
h.state.status = "ready";
const { rerender, onResumeFallback } = renderResumable(streamingTail());
// Arm reconcile via a 204.
vi.stubGlobal(
"fetch",
vi.fn().mockResolvedValue({ status: 204, ok: false }),
);
await act(async () => {
await h.state.transport!.fetch!("http://x", { method: "GET" });
});
h.state.setMessages.mockClear();
onResumeFallback.mockClear();
// A streaming-tail update: merge, poll stays armed.
rerender([
row("u1", "user", undefined, "hi"),
row("a1", "assistant", "streaming", "step 1\nstep 2"),
]);
expect(h.state.setMessages).toHaveBeenCalledTimes(1);
expect(onResumeFallback).not.toHaveBeenCalledWith(false);
// A settled-tail update: merge + disarm.
h.state.setMessages.mockClear();
rerender([
row("u1", "user", undefined, "hi"),
row("a1", "assistant", "succeeded", "final"),
]);
expect(h.state.setMessages).toHaveBeenCalledTimes(1);
expect(onResumeFallback).toHaveBeenCalledWith(false);
});
it("a local stream disarms both the merge and the poll", () => {
h.state.status = "streaming";
const { rerender, onResumeFallback } = renderResumable(streamingTail());
onResumeFallback.mockClear();
// A re-render while streaming: the reconciliation effect disarms.
rerender(streamingTail());
expect(onResumeFallback).toHaveBeenCalledWith(false);
});
it("Send now is hidden on a resumed turn but visible on a local stream", () => {
// Resumed turn: hidden.
renderThread({ autonomousRunsEnabled: true, initialRows: streamingTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
expect(screen.queryByLabelText("Send now")).toBeNull();
// Local streaming turn (no resume): visible.
cleanup();
resetState();
renderThread({ initialRows: [] });
fireEvent.click(screen.getByTestId("queue-btn"));
expect(screen.getByLabelText("Send now")).toBeTruthy();
});
it("handleStop aborts the attach controller and calls onServerStop", async () => {
const { onServerStop } = renderThread({
autonomousRunsEnabled: true,
initialRows: streamingTail(),
});
// Establish an attach controller via a (pending) reconnect GET.
let abortSeen = false;
vi.stubGlobal(
"fetch",
vi.fn().mockImplementation((_input: unknown, init: RequestInit) => {
init.signal?.addEventListener("abort", () => {
abortSeen = true;
});
return new Promise(() => undefined); // never resolves
}),
);
act(() => {
void h.state.transport!.fetch!("http://x", { method: "GET" });
});
fireEvent.click(screen.getByLabelText("Stop"));
expect(abortSeen).toBe(true);
expect(onServerStop).toHaveBeenCalledWith("c1");
});
});
// Helper: render a resumable thread and expose a rerender that only swaps
// initialRows (the degraded-merge effect depends on it).
function renderResumable(initialRows: IAiChatMessageRow[]) {
const onResumeFallback = vi.fn();
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
const Wrapper = ({ rows }: { rows: IAiChatMessageRow[] }) => (
<QueryClientProvider client={queryClient}>
<MantineProvider>
<ChatThread
chatId="c1"
initialRows={[]}
initialRows={rows}
autonomousRunsEnabled
onTurnFinished={vi.fn()}
observedRow={observedRow as never}
onResumeFallback={onResumeFallback}
/>
</MantineProvider>,
);
}
it("merges the polled run message when this tab is a passive observer", () => {
renderObserver("ready");
expect(h.state.setMessages).toHaveBeenCalledTimes(1);
// The updater replaces/append the observed assistant row by id.
const updater = h.state.setMessages.mock.calls[0][0] as (
prev: { id: string; parts: { text: string }[] }[],
) => { id: string; parts: { text: string }[] }[];
const merged = updater([{ id: "u1", parts: [{ text: "hi" }] }]);
expect(merged).toHaveLength(2);
expect(merged[1].id).toBe("a-run");
expect(merged[1].parts[0].text).toBe("step 1\nstep 2");
});
it("does NOT merge while THIS tab is the streamer (no double-render)", () => {
renderObserver("streaming");
expect(h.state.setMessages).not.toHaveBeenCalled();
});
});
</MantineProvider>
</QueryClientProvider>
);
const view = render(<Wrapper rows={initialRows} />);
const rerender = (rows: IAiChatMessageRow[]) =>
act(() => view.rerender(<Wrapper rows={rows} />));
return { rerender, onResumeFallback };
}
@@ -1,4 +1,5 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useQueryClient } from "@tanstack/react-query";
import { generateId } from "ai";
import { ActionIcon, Box, Group, Stack, Text, Tooltip } from "@mantine/core";
import {
@@ -24,7 +25,15 @@ import {
} from "@/features/ai-chat/utils/role-launch.ts";
import { describeChatError } from "@/features/ai-chat/utils/error-message.ts";
import { extractServerChatId } from "@/features/ai-chat/utils/adopt-chat-id.ts";
import { mergeObservedMessage } from "@/features/ai-chat/utils/run-polling.ts";
import { assistantMessageHasVisibleContent } from "@/features/ai-chat/utils/message-content.ts";
import {
isStreamingTail,
isSettledAssistantTail,
seedRows,
mergeById,
} from "@/features/ai-chat/utils/resume-helpers.ts";
import { AI_CHAT_MESSAGES_RQ_KEY } from "@/features/ai-chat/queries/ai-chat-query.ts";
import type { EditorSelectionContext } from "@/features/editor/utils/get-editor-selection.ts";
import {
dequeue,
enqueueMessage,
@@ -61,6 +70,10 @@ interface ChatThreadProps {
/** The page currently open in the workspace, or null on a non-page route.
* Sent with each turn so the agent knows what "this page" refers to. */
openPage?: OpenPageContext | null;
/** #388: snapshot the user's current editor selection at SEND time. Invoked
* inside prepareSendMessagesRequest and nested into openPage on the wire, so a
* fresh snapshot ships each turn. Null/absent => nothing selected. */
getEditorSelection?: () => EditorSelectionContext | null;
/** The agent role selected for a NEW chat (null = universal assistant). Sent
* in the request body so the server persists it on chat creation; ignored by
* the server for existing chats (the role is read from the chat row). */
@@ -87,19 +100,13 @@ interface ChatThreadProps {
* Copy/export button available mid-stream). Distinct from onTurnFinished,
* which fires only at the terminal outcome. */
onServerChatId?: (serverChatId?: string) => void;
/** #184 reconnect-and-live-follow. When THIS tab reopened a chat whose agent
* run is still going (it is a PASSIVE OBSERVER — it did not start the run here),
* the parent polls the reconnect endpoint and feeds the run's incrementally-
* persisted assistant message here; we merge it into the live list so new
* steps/tool-calls appear as they are persisted. Null when there is nothing to
* observe (no run, feature off, or this tab IS the streamer). The merge is
* ADDITIONALLY guarded by our own `isStreaming`, so a stale value can never
* fight the local stream when we are the streamer. */
observedRow?: IAiChatMessageRow | null;
/** Report this tab's live streaming status up to the parent, so it can stop
* polling the run while WE are the active streamer (the SSE owns the view) and
* resume once we go idle. Called from an effect on every transition. */
onStreamingChange?: (streaming: boolean) => void;
/** #184 phase 1.5: arm/disarm the parent's degraded-poll fallback for THIS
* chat's window. Called `true` when a resume attempt could not attach to the
* live run (attach 204 / starved-or-torn resumed finish), so the window starts
* a dumb timed poll of the message history to follow the detached run to settle;
* called `false` the moment a local stream starts or the terminal settled row is
* merged (invariant 8). The window owns the timer + its 10-min cap. */
onResumeFallback?: (active: boolean) => void;
/** #184: whether detached/autonomous agent runs are enabled for this workspace.
* When true the Stop button must additionally hit the AUTHORITATIVE server stop
* (via onServerStop) — aborting only the local SSE is just a client disconnect,
@@ -149,21 +156,65 @@ export default function ChatThread({
threadKey,
initialRows,
openPage,
getEditorSelection,
roleId,
roles,
onRolePicked,
assistantName,
onTurnFinished,
onServerChatId,
observedRow,
onStreamingChange,
onResumeFallback,
autonomousRunsEnabled,
onServerStop,
}: ChatThreadProps) {
const { t } = useTranslation();
const queryClient = useQueryClient();
// resume machinery refs (#184 phase 1.5)
const attachAbortRef = useRef<AbortController | null>(null);
const reconcileTailRef = useRef(false);
const noStreamHandledRef = useRef(false);
const onNoActiveStreamRef = useRef<(() => void) | null>(null);
// Live mount flag. The attach GET and the resumed `onFinish` are async and can
// land AFTER this thread unmounts (the parent remounts per chat via `key`); with
// chatIdRef then pointing at the NEW chat, an ungated late callback would arm a
// spurious poll + foreign invalidation on the newly-opened chat. Every parent-
// facing resume side-effect is gated on this.
const mountedRef = useRef(true);
const [resumedTurn, setResumedTurn] = useState(false);
const resumedTurnRef = useRef(false);
// Identity-stable pair setter (bare useState setter + ref write): it is closed
// over by the transport useMemo([]), so it MUST NOT capture state.
const setResumedTurnPair = useCallback((v: boolean) => {
resumedTurnRef.current = v;
setResumedTurn(v);
}, []);
// Mount-time resume gating (in refs — computed once for this mount; the parent
// remounts per chat via `key`).
//
// Attempt resume for any non-settled tail: a streaming tail (strip + expect
// live replay) or a user tail (the run may exist but its assistant row is not
// seeded yet — attach to the pre-opened registry entry and wait for frames).
// A settled assistant tail must NEVER resume: replaying a finished run into a
// store that already contains its message duplicates parts (SDK text-start
// always pushes a new part).
const stripRef = useRef(chatId !== null && isStreamingTail(initialRows ?? []));
const attemptResumeRef = useRef(
autonomousRunsEnabled === true &&
chatId !== null &&
!isSettledAssistantTail(initialRows ?? []),
);
const strippedRowRef = useRef<IAiChatMessageRow | null>(
stripRef.current ? (initialRows ?? [])[initialRows!.length - 1] : null,
);
const initialMessages = useMemo<UIMessage[]>(
() => (initialRows ?? []).map(rowToUiMessage),
() =>
seedRows(
initialRows ?? [],
attemptResumeRef.current && stripRef.current,
).map(rowToUiMessage),
[initialRows],
);
@@ -181,6 +232,14 @@ export default function ChatThread({
const openPageRef = useRef<OpenPageContext | null>(openPage ?? null);
openPageRef.current = openPage ?? null;
// Keep the selection snapshotter in a ref, same rationale as openPageRef: the
// transport useMemo([]) closes it over, so prop-identity churn must not matter.
// Called at send time inside prepareSendMessagesRequest (#388).
const getEditorSelectionRef = useRef<
(() => EditorSelectionContext | null) | undefined
>(getEditorSelection);
getEditorSelectionRef.current = getEditorSelection;
// Keep the selected role id in a ref, same rationale as openPageRef. Only the
// FIRST request of a brand-new chat uses it (the server persists it then and
// ignores it for existing chats), but sending it on every send is harmless.
@@ -261,9 +320,12 @@ export default function ChatThread({
const { head, rest } = dequeue(queuedRef.current);
if (!head) return false;
setQueue(rest);
// Local send: clear any resume-suppression flag so this genuine local turn's
// onFinish flushes normally (invariant 8).
setResumedTurnPair(false);
sendMessageRef.current?.({ text: head.text });
return true;
}, [setQueue]);
}, [setQueue, setResumedTurnPair]);
const enqueue = useCallback(
(text: string) => {
@@ -283,6 +345,47 @@ export default function ChatThread({
new DefaultChatTransport<UIMessage>({
api: "/api/ai-chat/stream",
credentials: "include",
prepareReconnectToStreamRequest: () => ({
// SDK default URL uses the useChat STORE id — always build from the real chat id.
// ?expect=live&anchor=<row id> ONLY when we stripped a streaming tail: expect=live
// is the only case where a finished-retained replay is safe (the row is stripped,
// replay rebuilds it), and the anchor pins the replay to OUR run — a mismatching
// (newer) run must 204 into the restore+poll path instead of replaying a foreign
// transcript into this store.
api: `/api/ai-chat/runs/${chatIdRef.current}/stream${
stripRef.current
? `?expect=live&anchor=${strippedRowRef.current!.id}`
: ""
}`,
}),
fetch: async (input: RequestInfo | URL, init: RequestInit = {}) => {
if ((init.method ?? "GET") !== "GET") return fetch(input, init); // send path untouched
// Reconnect GET: the SDK passes no AbortSignal, so wire our own controller
// for observer Stop / unmount abort.
const controller = new AbortController();
attachAbortRef.current = controller;
try {
const response = await fetch(input, {
...init,
signal: controller.signal,
});
// No onFinish will come for a 204 (silent no-op) OR any non-2xx
// (5xx/502 — a server restart mid-attach). Both run the same
// no-active-stream recovery: restore the stripped row, invalidate, and
// arm the degraded poll (idempotent via noStreamHandledRef; its part-d
// also clears the resumedTurn flag). This is the restart-survival path
// the removed F7 latch used to guard — a transient attach failure must
// NOT drop the in-progress row or stop tracking the durable run.
if (response.status === 204 || !response.ok)
onNoActiveStreamRef.current?.();
return response;
} catch (err) {
// Network throw: same no-onFinish recovery, then rethrow so the SDK
// still surfaces the error to its own machinery.
onNoActiveStreamRef.current?.();
throw err;
}
},
// Inject the chat id and the currently-open page alongside the useChat
// messages so the server can resolve an existing chat (or create one
// when null) and tell the agent which page "this page" refers to. Both
@@ -299,7 +402,16 @@ export default function ChatThread({
body: {
...body,
chatId: chatIdRef.current,
openPage: openPageRef.current,
// Attach the live editor selection to the open-page context at send
// time — "this"/"here" in the user's message means THIS selection.
// Nested inside openPage so it dies with the page when the server
// rejects the page id (#388). Null when nothing is selected.
openPage: openPageRef.current
? {
...openPageRef.current,
selection: getEditorSelectionRef.current?.() ?? null,
}
: null,
// Honoured by the server only when creating a new chat; null =>
// universal assistant.
roleId: roleIdRef.current,
@@ -312,7 +424,15 @@ export default function ChatThread({
[],
);
const { messages, sendMessage, status, stop, error, setMessages } = useChat({
const {
messages,
sendMessage,
status,
stop,
error,
setMessages,
resumeStream,
} = useChat({
// Stable per-mount key. Existing chats use their real id; new chats use a
// generated client id (never `undefined`) so the store is NOT re-created on
// every render mid-stream (see `chatStoreId` above).
@@ -330,6 +450,38 @@ export default function ChatThread({
// would be wrong, so on Stop/disconnect/error the queue is left intact for
// the user to decide.
onFinish: ({ message, isAbort, isDisconnect, isError }) => {
// (1) Capture whether THIS finish belongs to a resumed (attach) turn and
// immediately clear the flag so it can never suppress a LATER local turn.
const wasResumed = resumedTurnRef.current;
setResumedTurnPair(false);
// (2) Recovery after a starved/torn resumed finish (invariant 9). The arm
// and the stripped-row restore are gated DIFFERENTLY. Skip entirely once
// unmounted (an abort-triggered onFinish landing after a chat switch must
// not arm a poll / invalidate on the new chat).
if (wasResumed && mountedRef.current) {
const hasVisibleContent = assistantMessageHasVisibleContent(message);
// ARM the reconcile + degraded poll when the resumed message carries no
// visible content (starved replay) OR the connection dropped mid-run — in
// both cases the poll must drive the row to its real terminal state.
if (isDisconnect || !hasVisibleContent) {
reconcileTailRef.current = true;
queryClient.invalidateQueries({
queryKey: AI_CHAT_MESSAGES_RQ_KEY(chatIdRef.current),
});
onResumeFallback?.(true);
}
// RESTORE the stripped streaming row ONLY when the resumed message has no
// visible content. On isDisconnect WITH visible content restore is
// FORBIDDEN: the live stream may have advanced far past the mount-time
// snapshot, so restoring would clobber on-screen content (invariant 9) —
// the arm above suffices, the poll reaches the true terminal.
if (!hasVisibleContent && strippedRowRef.current) {
setMessages((prev) =>
mergeById(prev, rowToUiMessage(strippedRowRef.current!)),
);
}
}
// (3) Standard branches.
// Forward the authoritative server chatId (streamed on the assistant
// message metadata) so the parent adopts the REAL created chat id for a new
// chat — see adopt-chat-id.ts for the full #137 design. `threadKey` lets the
@@ -342,6 +494,10 @@ export default function ChatThread({
else if (isAbort) setStopNotice("manual");
else if (isDisconnect) setStopNotice("disconnect");
else setStopNotice(null);
// A resumed turn NEVER flushes the queue (invariant 7): skip BOTH the
// flush-on-abort branch and the plain flush. The local streamer is the only
// tab that owns the queue.
if (wasResumed) return;
// "Send now": WE triggered this abort to interrupt the current turn and
// immediately send the promoted head. Flush it even though the turn was
// aborted (the normal abort path below keeps the queue intact). The
@@ -423,26 +579,98 @@ export default function ChatThread({
const isStreaming = status === "submitted" || status === "streaming";
// #184: report our live streaming status up so the parent stops polling the run
// while WE are the streamer (the SSE owns the view) and resumes once we go idle.
// Effect (not render) so it never updates parent state during our own render;
// fires on mount with `false`, which also re-syncs the parent after a chat
// switch remounts this thread (a fresh mount is idle until the user sends).
useEffect(() => {
onStreamingChange?.(isStreaming);
}, [isStreaming, onStreamingChange]);
// 204-handler (`onNoActiveStream`): the attach returned 204 — nothing live to
// resume (overflow / begin-failure / after retention / anchor-mismatch). One-
// shot via noStreamHandledRef (we do NOT null onNoActiveStreamRef). Exactly four
// parts. Kept in a ref (read by the transport's fetch closure) and refreshed
// each render below.
const onNoActiveStream = useCallback(() => {
// A late attach outcome after unmount must not arm a poll / invalidate on the
// now-different chat this thread's refs were reused for.
if (!mountedRef.current) return;
if (noStreamHandledRef.current) return;
noStreamHandledRef.current = true;
// (a) Restore the stripped streaming row to the store — ONLY when we actually
// stripped one (a user-tail 204 does NOT reach here with a stripped row, so do
// not dereference null).
if (strippedRowRef.current) {
setMessages((prev) =>
mergeById(prev, rowToUiMessage(strippedRowRef.current!)),
);
}
// (b) Reconcile the tail from the message history + invalidate it so the
// degraded poll starts from a fresh fetch.
reconcileTailRef.current = true;
queryClient.invalidateQueries({
queryKey: AI_CHAT_MESSAGES_RQ_KEY(chatIdRef.current),
});
// (c) Arm the degraded poll (a dumb timer with a 10-min cap in the window);
// the thread disarms it via onResumeFallback(false) on settle / local stream.
onResumeFallback?.(true);
// (d) 204 means onFinish will NOT fire — clear the suppression flag so it
// cannot swallow the NEXT local turn's queue flush.
setResumedTurnPair(false);
}, [setMessages, queryClient, onResumeFallback, setResumedTurnPair]);
onNoActiveStreamRef.current = onNoActiveStream;
// #184 passive-observer merge: when the parent feeds a polled run message (we
// reopened a chat whose run is still going and did NOT start it here), merge it
// into the live list so new steps/tool-calls appear as they are persisted. Hard-
// gated by `!isStreaming`: if THIS tab is actually the streamer, the local SSE
// owns the view and a stale observedRow must never overwrite it. `observedRow`
// is a stable per-poll object, so this runs once per poll, not per render.
// Mount effect: kick off the resume attempt for a non-settled tail. Marking the
// turn as resumed BEFORE resumeStream so onFinish (invariant 7/8) sees it.
useEffect(() => {
if (isStreaming || !observedRow) return;
const observed = rowToUiMessage(observedRow);
setMessages((prev) => mergeObservedMessage(prev, observed));
}, [observedRow, isStreaming, setMessages]);
// Re-arm on (re)mount — StrictMode dev-mounts twice, and the cleanup below
// flips this false between the two.
mountedRef.current = true;
if (attemptResumeRef.current) {
setResumedTurnPair(true);
void resumeStream();
}
// Unmount: mark unmounted (gates late attach/onFinish side-effects) and abort
// the in-flight attach GET so its callbacks don't fire against the next chat.
return () => {
mountedRef.current = false;
attachAbortRef.current?.abort();
};
// Mount-only by design; the parent remounts per chat via `key`.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
// Reconciliation + degraded-merge (invariant 8). Deps are EXACTLY
// [initialRows, isStreaming, setMessages].
useEffect(() => {
// A local stream owns the view: disarm BOTH the merge and the window poll.
if (isStreaming) {
reconcileTailRef.current = false;
onResumeFallback?.(false);
return;
}
if (!reconcileTailRef.current) return;
const rows = initialRows ?? [];
const tail = rows[rows.length - 1];
if (!tail || tail.role !== "assistant") return;
// Merge the polled assistant tail on EVERY initialRows update — while the
// degraded poll is active this IS the live per-step progress.
setMessages((prev) => mergeById(prev, rowToUiMessage(tail)));
// Anchor-mismatch coherence: when we restored a stripped streaming row A but a
// DIFFERENT run's row B is now the tail (A finished, B replaced the registry
// entry, so the attach 204'd), A would otherwise linger forever as an orphan
// jumping-dots row over the real run. Settle it from fresh history (where A is
// now persisted) so no phantom row survives. No-op in the common case where A
// IS the tail (id match).
const stripped = strippedRowRef.current;
if (stripped && stripped.id !== tail.id) {
const historical = rows.find((r) => r.id === stripped.id);
if (historical)
setMessages((prev) => mergeById(prev, rowToUiMessage(historical)));
}
// Settled: the terminal merge is done — disarm the flag AND the window poll
// explicitly (the window only has a time cap, it will not disarm itself).
if (tail.status !== "streaming") {
reconcileTailRef.current = false;
onResumeFallback?.(false);
}
// onResumeFallback intentionally omitted (parent-stable callback); deps are
// fixed by the resume design.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [initialRows, isStreaming, setMessages]);
// "Send now" on a queued message: interrupt the current turn and immediately
// send THIS message, keeping the agent's partial output. Other queued messages
@@ -469,10 +697,12 @@ export default function ChatThread({
const msg = queuedRef.current.find((m) => m.id === id);
if (!msg) return;
setQueue(removeQueuedById(queuedRef.current, id));
// Local send: clear any resume-suppression flag (invariant 8).
setResumedTurnPair(false);
sendMessageRef.current?.({ text: msg.text });
}
},
[setQueue, stop],
[setQueue, stop, setResumedTurnPair],
);
// Stop the current turn. ALWAYS abort the local SSE (`stop()`) so the composer
@@ -485,6 +715,9 @@ export default function ChatThread({
// is not known yet — a brand-new chat in the first moment of its first turn —
// only the local abort happens (there is no server-side run handle to stop yet).
const handleStop = useCallback(() => {
// Abort the resume/attach GET first: the SDK does not pass it a signal, so an
// observer's Stop would otherwise leave the attach fetch running.
attachAbortRef.current?.abort();
stop();
if (!autonomousRunsEnabled) return;
if (chatIdRef.current) {
@@ -617,17 +850,23 @@ export default function ChatThread({
<Text size="xs" lineClamp={2} className={classes.queuedText}>
{m.text}
</Text>
<Tooltip label={t("Interrupt and send now")} withArrow>
<ActionIcon
size="xs"
variant="subtle"
color="blue"
onClick={() => sendNow(m.id)}
aria-label={t("Send now")}
>
<IconPlayerPlayFilled size={12} />
</ActionIcon>
</Tooltip>
{/* "Send now" (interrupt) is hidden on a RESUMED turn: a local
stop() does not abort the resumed attach fetch, so the click
would be swallowed while flushOnAbortRef would fire minutes
later on the natural finish. Only the remove affordance stays. */}
{!resumedTurn && (
<Tooltip label={t("Interrupt and send now")} withArrow>
<ActionIcon
size="xs"
variant="subtle"
color="blue"
onClick={() => sendNow(m.id)}
aria-label={t("Send now")}
>
<IconPlayerPlayFilled size={12} />
</ActionIcon>
</Tooltip>
)}
<ActionIcon
size="xs"
variant="subtle"
@@ -642,7 +881,11 @@ export default function ChatThread({
</Stack>
)}
<ChatInput
onSend={(text) => sendMessage({ text })}
onSend={(text) => {
// Local send: clear any resume-suppression flag (invariant 8).
setResumedTurnPair(false);
sendMessage({ text });
}}
onQueue={enqueue}
onStop={handleStop}
isStreaming={isStreaming}
@@ -40,6 +40,13 @@ interface MessageItemProps {
* Defaults to true (internal chat). The public share passes false.
*/
showCitations?: boolean;
/**
* Forwarded to ToolCallCard: whether tool cards render the one-line summary of
* a call's arguments (e.g. the search query). Defaults to true (internal
* chat). The public share passes false so an anonymous reader doesn't see the
* agent's raw query/argument text.
*/
showInput?: boolean;
/**
* Neutralize internal/relative markdown links in the rendered answer (drop
* their href so they become inert text). Defaults to false (internal chat,
@@ -117,6 +124,7 @@ const MarkdownPart = memo(function MarkdownPart({
function MessageItem({
message,
showCitations = true,
showInput = true,
neutralizeInternalLinks = false,
assistantName,
turnStreaming = false,
@@ -210,6 +218,7 @@ function MessageItem({
key={index}
part={part as unknown as ToolUiPart}
showCitations={showCitations}
showInput={showInput}
/>
);
}
@@ -274,6 +283,7 @@ export function arePropsEqual(
return (
prev.signature === next.signature &&
prev.showCitations === next.showCitations &&
prev.showInput === next.showInput &&
prev.neutralizeInternalLinks === next.neutralizeInternalLinks &&
prev.assistantName === next.assistantName &&
// The turn-end flip re-renders every row once (cheap, terminal event) —
@@ -25,6 +25,13 @@ interface MessageListProps {
* false because an anonymous reader cannot open the linked internal pages.
*/
showCitations?: boolean;
/**
* Forwarded to MessageItem -> ToolCallCard: whether tool cards render the
* one-line summary of a call's arguments (e.g. the search query). Defaults to
* true (internal chat). The public share passes false so an anonymous reader
* doesn't see the agent's raw query/argument text.
*/
showInput?: boolean;
/**
* Forwarded to MessageItem: neutralize internal/relative markdown links in
* the rendered answers (drop their href so they render as inert text).
@@ -119,6 +126,7 @@ export default function MessageList({
isStreaming,
emptyState,
showCitations = true,
showInput = true,
neutralizeInternalLinks = false,
assistantName,
}: MessageListProps) {
@@ -208,6 +216,7 @@ export default function MessageList({
message={message}
signature={messageSignature(message)}
showCitations={showCitations}
showInput={showInput}
neutralizeInternalLinks={neutralizeInternalLinks}
assistantName={assistantName}
// Turn-level liveness, gated to the TAIL row: only the tail message
@@ -5,6 +5,7 @@ import { useTranslation } from "react-i18next";
import {
getToolName,
toolCitations,
toolInputSummary,
toolLabelKey,
toolRunState,
ToolUiPart,
@@ -21,6 +22,14 @@ interface ToolCallCardProps {
* (the action log itself) while dropping the unusable links.
*/
showCitations?: boolean;
/**
* Whether to render the one-line summary of the call's arguments (e.g. the
* search query) under the label. Defaults to true (the internal chat). The
* public share passes false: an anonymous reader should not see the agent's
* raw query/argument text. Conservative and reversible — it only suppresses
* the extra summary line, leaving the card (the action log) intact.
*/
showInput?: boolean;
}
/**
@@ -31,12 +40,14 @@ interface ToolCallCardProps {
export default function ToolCallCard({
part,
showCitations = true,
showInput = true,
}: ToolCallCardProps) {
const { t } = useTranslation();
const toolName = getToolName(part);
const state = toolRunState(part.state);
const { key, values } = toolLabelKey(toolName);
const citations = showCitations ? toolCitations(part) : [];
const inputSummary = showInput ? toolInputSummary(part) : undefined;
return (
<div className={classes.toolCard}>
@@ -57,6 +68,12 @@ export default function ToolCallCard({
</Text>
</Group>
{inputSummary && (
<Text size="xs" c="dimmed" mt={2} lineClamp={2}>
{inputSummary}
</Text>
)}
{state === "error" && part.errorText && (
<Text size="xs" c="red" mt={2}>
{part.errorText}
@@ -0,0 +1,90 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import React from "react";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
// react-i18next / notifications are pulled in transitively by ai-chat-query.ts
// (the mutation hooks use them); stub so the module imports cleanly.
vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (key: string) => key }),
}));
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
// Mock the service module; only getAiChatMessages is exercised, but the other
// named exports must exist so ai-chat-query.ts imports resolve.
vi.mock("@/features/ai-chat/services/ai-chat-service.ts", () => ({
getAiChatMessages: vi.fn(),
getAiChats: vi.fn(),
getAiRoleCatalog: vi.fn(),
getAiRoleCatalogBundle: vi.fn(),
getAiRoles: vi.fn(),
importAiRolesFromCatalog: vi.fn(),
createAiRole: vi.fn(),
deleteAiChat: vi.fn(),
deleteAiRole: vi.fn(),
renameAiChat: vi.fn(),
updateAiRole: vi.fn(),
updateAiRoleFromCatalog: vi.fn(),
}));
import { getAiChatMessages } from "@/features/ai-chat/services/ai-chat-service.ts";
import { useAiChatMessagesQuery } from "@/features/ai-chat/queries/ai-chat-query.ts";
const emptyPage = { items: [], meta: { hasNextPage: false, nextCursor: null } };
function createWrapper() {
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
return function Wrapper({ children }: { children: React.ReactNode }) {
return (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
);
};
}
// The degraded-poll fallback (#184 phase 1.5) is threaded into this query as a
// `refetchInterval`; AiChatWindow supplies the deliberately-dumb callback. These
// pin the plumbing the window depends on: the interval polls the message history,
// and — critically — fetch ERRORS do NOT stop the tick (TanStack v5 resets the
// failure count each fetch, so the poll must survive a server restart).
describe("useAiChatMessagesQuery — degraded refetchInterval", () => {
beforeEach(() => {
vi.clearAllMocks();
});
it("re-polls at the interval while the callback returns a duration", async () => {
vi.mocked(getAiChatMessages).mockResolvedValue(emptyPage as never);
renderHook(() => useAiChatMessagesQuery("c1", () => 30), {
wrapper: createWrapper(),
});
await waitFor(() =>
expect(vi.mocked(getAiChatMessages).mock.calls.length).toBeGreaterThan(2),
);
});
it("does NOT re-poll when the callback returns false", async () => {
vi.mocked(getAiChatMessages).mockResolvedValue(emptyPage as never);
renderHook(() => useAiChatMessagesQuery("c1", () => false), {
wrapper: createWrapper(),
});
await waitFor(() =>
expect(vi.mocked(getAiChatMessages)).toHaveBeenCalledTimes(1),
);
// Give any errant interval a chance to fire, then assert it did not.
await new Promise((r) => setTimeout(r, 60));
expect(vi.mocked(getAiChatMessages)).toHaveBeenCalledTimes(1);
});
it("keeps ticking through fetch errors (errors do not gate the poll)", async () => {
vi.mocked(getAiChatMessages).mockRejectedValue(new Error("server down"));
renderHook(() => useAiChatMessagesQuery("c1", () => 30), {
wrapper: createWrapper(),
});
await waitFor(() =>
expect(vi.mocked(getAiChatMessages).mock.calls.length).toBeGreaterThan(2),
);
});
});
@@ -1,6 +1,7 @@
import {
useInfiniteQuery,
useMutation,
useQueries,
useQuery,
useQueryClient,
} from "@tanstack/react-query";
@@ -12,7 +13,6 @@ import {
deleteAiChat,
deleteAiRole,
getAiChatMessages,
getAiChatRun,
getAiChats,
getAiRoleCatalog,
getAiRoleCatalogBundle,
@@ -25,7 +25,6 @@ import {
import {
IAiChat,
IAiChatMessageRow,
IAiChatRunResponse,
IAiRole,
IAiRoleCatalog,
IAiRoleCatalogBundle,
@@ -36,7 +35,6 @@ import {
IAiRoleUpdateFromCatalogResult,
} from "@/features/ai-chat/types/ai-chat.types.ts";
import { IPagination } from "@/lib/types.ts";
import { runPollInterval } from "@/features/ai-chat/utils/run-polling.ts";
export const AI_CHATS_RQ_KEY = ["ai-chats"];
export const AI_ROLES_RQ_KEY = ["ai-roles"];
@@ -54,7 +52,6 @@ export const AI_CHAT_MESSAGES_RQ_KEY = (chatId: string) => [
"ai-chat-messages",
chatId,
];
export const AI_CHAT_RUN_RQ_KEY = (chatId: string) => ["ai-chat-run", chatId];
/** Paginated list of the current user's chats (auto-loads further pages). */
export function useAiChatsQuery() {
@@ -88,7 +85,15 @@ export function useAiChatsQuery() {
* Load all persisted messages of a chat (oldest first), flattening the
* paginated server response. Used to seed `useChat` initial messages.
*/
export function useAiChatMessagesQuery(chatId: string | undefined) {
export function useAiChatMessagesQuery(
chatId: string | undefined,
// #184 phase 1.5: the degraded-poll fallback. When a tab could not attach to a
// still-running run (the attach returned 204 / the resumed stream ended with no
// terminal row), the window arms a dumb timed poll of the message history to
// follow the detached run to settle. The callback form lives in AiChatWindow;
// threaded here verbatim so this query owns the polling. Undefined => no poll.
refetchInterval?: number | false | (() => number | false),
) {
const query = useInfiniteQuery({
queryKey: AI_CHAT_MESSAGES_RQ_KEY(chatId ?? ""),
queryFn: ({ pageParam }) =>
@@ -99,6 +104,7 @@ export function useAiChatMessagesQuery(chatId: string | undefined) {
? (lastPage.meta.nextCursor ?? undefined)
: undefined,
enabled: !!chatId,
refetchInterval,
});
// useInfiniteQuery only fetches the first page on its own. The hook's contract
@@ -138,34 +144,6 @@ export function useAiChatMessagesQuery(chatId: string | undefined) {
};
}
/**
* Reconnect to a chat's latest agent run and LIVE-FOLLOW it (#184). While the run
* is active the query re-polls every {@link runPollInterval} ms (driven off the
* fetched `run.status`, the same status-keyed refetchInterval pattern as the
* embeddings reindex polling); once the run reaches a terminal status — or there
* is no run — the interval returns `false` and polling stops on its own. Polling
* is thus naturally bounded by the run terminating; no separate timeout cap.
*
* `enabled` gates the whole thing: callers pass `false` when the autonomous-runs
* feature is off (the endpoint is NOT flag-gated server-side, but with the feature
* off the chat has no runs, so polling would only ever return `{ run: null }`) OR
* when THIS tab is the one actively streaming the run (the live SSE owns the view,
* so we must not also poll/merge). The global `retry: false` means a failed fetch
* leaves `data` undefined, so refetchInterval(undefined run) returns false — a
* failed fetch can never spin a tight loop.
*/
export function useAiChatRunQuery(
chatId: string | undefined,
enabled: boolean,
) {
return useQuery<IAiChatRunResponse, Error>({
queryKey: AI_CHAT_RUN_RQ_KEY(chatId ?? ""),
queryFn: () => getAiChatRun(chatId as string),
enabled: !!chatId && enabled,
refetchInterval: (query) => runPollInterval(query.state.data?.run),
});
}
export function useRenameAiChatMutation() {
const queryClient = useQueryClient();
const { t } = useTranslation();
@@ -307,6 +285,29 @@ export function useAiRoleCatalogBundleQuery(
});
}
/**
* Eagerly open EVERY listed bundle's content in parallel for one language. The
* redesigned catalog shows each bundle's status summary in its COLLAPSED header,
* which needs every role's install state up front — so contents can no longer be
* lazy-loaded on expand. The catalog is small, so a fan-out of `useQueries` (one
* cached read per bundle, sharing the same cache keys as
* `useAiRoleCatalogBundleQuery`) is cheap. Gated by `enabled` (modal open + a
* resolved language) so nothing fetches while the modal is closed.
*/
export function useAiRoleCatalogBundlesQueries(
bundleIds: string[],
language: string,
enabled: boolean,
) {
return useQueries({
queries: bundleIds.map((bundleId) => ({
queryKey: AI_ROLE_CATALOG_BUNDLE_RQ_KEY(bundleId, language),
queryFn: () => getAiRoleCatalogBundle(bundleId, language),
enabled: enabled && !!language,
})),
});
}
export function useImportAiRolesFromCatalogMutation() {
const queryClient = useQueryClient();
const { t } = useTranslation();
@@ -1,92 +0,0 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import React from "react";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import type { IAiChatRunResponse } from "@/features/ai-chat/types/ai-chat.types.ts";
// react-i18next is pulled in transitively by ai-chat-query.ts (the mutation hooks
// use it); stub it so the module imports cleanly in this hook test.
vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (key: string) => key }),
}));
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
// Mock the whole service module; only getAiChatRun is exercised here, but the
// other named exports must exist so ai-chat-query.ts imports resolve.
vi.mock("@/features/ai-chat/services/ai-chat-service.ts", () => ({
getAiChatRun: vi.fn(),
getAiChatMessages: vi.fn(),
getAiChats: vi.fn(),
getAiRoleCatalog: vi.fn(),
getAiRoleCatalogBundle: vi.fn(),
getAiRoles: vi.fn(),
importAiRolesFromCatalog: vi.fn(),
createAiRole: vi.fn(),
deleteAiChat: vi.fn(),
deleteAiRole: vi.fn(),
renameAiChat: vi.fn(),
updateAiRole: vi.fn(),
updateAiRoleFromCatalog: vi.fn(),
}));
import { getAiChatRun } from "@/features/ai-chat/services/ai-chat-service.ts";
import { useAiChatRunQuery } from "@/features/ai-chat/queries/ai-chat-query.ts";
function createWrapper() {
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
return function Wrapper({ children }: { children: React.ReactNode }) {
return (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
);
};
}
const runningResponse: IAiChatRunResponse = {
run: { id: "run-1", chatId: "c1", status: "running" },
message: {
id: "a1",
role: "assistant",
content: "working...",
createdAt: "2026-01-01T00:00:00Z",
},
};
describe("useAiChatRunQuery — enable gating", () => {
beforeEach(() => {
vi.clearAllMocks();
});
it("fetches the run when enabled (passive observer, feature on)", async () => {
vi.mocked(getAiChatRun).mockResolvedValue(runningResponse);
const { result } = renderHook(() => useAiChatRunQuery("c1", true), {
wrapper: createWrapper(),
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
expect(getAiChatRun).toHaveBeenCalledWith("c1");
expect(result.current.data?.run?.status).toBe("running");
});
it("does NOT fetch when disabled (this tab is the streamer / feature off)", async () => {
vi.mocked(getAiChatRun).mockResolvedValue(runningResponse);
renderHook(() => useAiChatRunQuery("c1", false), {
wrapper: createWrapper(),
});
// Give any errant fetch a chance to fire, then assert none did.
await new Promise((r) => setTimeout(r, 20));
expect(getAiChatRun).not.toHaveBeenCalled();
});
it("does NOT fetch when there is no chat id", async () => {
vi.mocked(getAiChatRun).mockResolvedValue(runningResponse);
renderHook(() => useAiChatRunQuery(undefined, true), {
wrapper: createWrapper(),
});
await new Promise((r) => setTimeout(r, 20));
expect(getAiChatRun).not.toHaveBeenCalled();
});
});
@@ -77,7 +77,14 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
});
it("errors:[] -> only the summary notification (counts interpolated)", async () => {
await runMutation({ created: 3, renamed: 1, skipped: 2, errors: [] });
await runMutation({
created: 3,
renamed: 1,
skipped: 2,
errors: [],
createdRoles: [],
skippedRoles: [],
});
expect(notificationsShowMock).toHaveBeenCalledTimes(1);
expect(notificationsShowMock).toHaveBeenCalledWith({
message: "Imported 3, renamed 1, skipped 2",
@@ -93,6 +100,8 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
{ slug: "a", message: "name taken" },
{ slug: "b", message: "name taken" },
],
createdRoles: [{ slug: "ok", name: "Ok" }],
skippedRoles: [],
});
expect(notificationsShowMock).toHaveBeenCalledTimes(2);
expect(notificationsShowMock).toHaveBeenNthCalledWith(1, {
@@ -5,7 +5,6 @@ import {
IAiChatListParams,
IAiChatMessageRow,
IAiChatMessagesParams,
IAiChatRunResponse,
IAiRole,
IAiRoleCatalog,
IAiRoleCatalogBundle,
@@ -43,23 +42,6 @@ export async function getAiChatMessages(
return req.data;
}
/**
* Reconnect to the latest agent run of a chat (#184). Returns the run's
* persisted lifecycle state and the assistant message it materializes (the
* partial output while the run is in-flight, the final output once it finished).
* The DB is the source of truth, so this works for an in-flight run (the browser
* dropped, the run kept going) and a finished one alike; `{ run: null }` when the
* chat has never had a run. Owner-gated server-side (the requesting user must own
* the chat); it is NOT flag-gated — when the feature is off the chat simply has no
* runs, so the endpoint returns `{ run: null }`.
*/
export async function getAiChatRun(
chatId: string,
): Promise<IAiChatRunResponse> {
const req = await api.post<IAiChatRunResponse>("/ai-chat/run", { chatId });
return req.data;
}
/**
* Explicitly STOP the active agent run of a chat (#184). This is the ONLY thing
* that ends a DETACHED run — a mere browser disconnect (aborting the local SSE)
@@ -108,12 +108,25 @@ export interface IAiRoleImportPayload {
conflict: "skip" | "rename";
}
/** Import result counts (mirrors `importFromCatalog()`). */
/**
* Import result (mirrors `importFromCatalog()`). The counters (`created`,
* `skipped`, `renamed`) drive the summary notification; the per-role lists
* (`createdRoles`, `skippedRoles`) drive the redesigned catalog modal's inline
* result plaque — which roles were installed (and any rename) and which were
* skipped and why (so the plaque can name the conflicting role and offer
* "Rename & install").
*/
export interface IAiRoleImportResult {
created: number;
skipped: number;
renamed: number;
errors: { slug: string; message: string }[];
createdRoles: { slug: string; name: string; renamedTo?: string }[];
skippedRoles: {
slug: string;
name: string;
reason: "name-conflict" | "already-installed";
}[];
}
/**
@@ -197,41 +210,14 @@ export interface IAiChatMessageRow {
// renders a "stopped" marker on interrupted turns.
finishReason?: string;
} | null;
// Persisted lifecycle status of the row's turn, carried on the wire by
// `baseFields`. 'streaming' marks a still-in-progress assistant row (used by
// the resume machinery to decide whether a tail is a live stream to attach to
// or a settled row that must not be replayed).
status?: string;
createdAt: string;
}
/**
* A persisted agent-run row (#184), mirroring the `ai_chat_runs` fields the
* client reads from `POST /ai-chat/run`. Only `status` is load-bearing for the
* reconnect-and-live-update UX (it drives the poll cadence); the rest are carried
* for display/diagnostics. The DB is the source of truth, so this resolves for an
* in-flight run (the browser dropped, the run kept going) and a finished one.
*/
export interface IAiChatRun {
id: string;
chatId: string;
// 'pending' | 'running' | 'succeeded' | 'failed' | 'aborted'. The first two are
// ACTIVE (keep polling); the rest are TERMINAL (stop polling).
status: "pending" | "running" | "succeeded" | "failed" | "aborted" | string;
error?: string | null;
stepCount?: number;
assistantMessageId?: string | null;
startedAt?: string | null;
finishedAt?: string | null;
createdAt?: string;
updatedAt?: string;
}
/**
* Response of `POST /ai-chat/run` (#184): the latest run of a chat and the
* assistant message it materializes (the partial/final output, projected from the
* persisted rows). Both are `null` when the chat has never had a run.
*/
export interface IAiChatRunResponse {
run: IAiChatRun | null;
message: IAiChatMessageRow | null;
}
export interface IAiChatListParams extends QueryParams {}
export interface IAiChatMessagesParams {
@@ -0,0 +1,234 @@
import { describe, it, expect } from "vitest";
import {
bundleCounts,
bundlePhase,
installedLangForRole,
mapBundleRolesToView,
mapCatalogRoleToView,
nameConflictSlugs,
partialOffersRename,
type CatalogViewRole,
} from "./catalog-bundle-model.ts";
import type {
IAiRole,
IAiRoleCatalogRole,
} from "@/features/ai-chat/types/ai-chat.types.ts";
function installedRole(
source: { slug: string; language: string; version: number },
overrides: Partial<IAiRole> = {},
): IAiRole {
return {
id: `role-${source.slug}-${source.language}`,
name: source.slug,
emoji: null,
description: null,
enabled: true,
autoStart: true,
launchMessage: null,
source,
...overrides,
};
}
function catalogRole(
overrides: Partial<IAiRoleCatalogRole> = {},
): IAiRoleCatalogRole {
return {
slug: "writer",
emoji: "✍️",
name: "Writer",
description: "Drafts copy.",
instructions: "be a writer",
autoStart: true,
launchMessage: null,
version: 3,
...overrides,
};
}
// Build a minimal view role for bundlePhase tests.
function viewRole(status: CatalogViewRole["status"]): CatalogViewRole {
return { slug: `s-${status}`, name: status, description: "", version: 1, status };
}
describe("bundlePhase", () => {
it("empty bundle -> empty", () => {
expect(bundlePhase([])).toBe("empty");
});
it("all importable, none installed -> allNew", () => {
expect(bundlePhase([viewRole("import"), viewRole("import")])).toBe(
"allNew",
);
});
it("nothing to import or update -> allInstalled", () => {
expect(bundlePhase([viewRole("installed"), viewRole("installed")])).toBe(
"allInstalled",
);
});
it("updates present, nothing to import -> updates", () => {
expect(bundlePhase([viewRole("update"), viewRole("installed")])).toBe(
"updates",
);
});
it("import + installed (no updates) -> mixed", () => {
expect(bundlePhase([viewRole("import"), viewRole("installed")])).toBe(
"mixed",
);
});
it("import + update -> mixed", () => {
expect(bundlePhase([viewRole("import"), viewRole("update")])).toBe("mixed");
});
it("a skipped role with nothing installed -> mixed (NOT allInstalled)", () => {
// F1: a bundle whose only non-installed role was skipped has 0 installed for
// it, so the collapsed 'All installed · up to date' header would contradict
// the open 'Installed 0 · 1 skipped' plaque. It must be mixed until resolved.
expect(bundlePhase([viewRole("skipped")])).toBe("mixed");
});
it("installed + a skipped role -> mixed (partial success is not allInstalled)", () => {
expect(bundlePhase([viewRole("installed"), viewRole("skipped")])).toBe(
"mixed",
);
});
});
describe("bundleCounts", () => {
it("tallies each status once", () => {
expect(
bundleCounts([
viewRole("import"),
viewRole("import"),
viewRole("installed"),
viewRole("update"),
viewRole("skipped"),
]),
).toEqual({ importable: 2, installed: 1, update: 1, skipped: 1 });
});
});
describe("nameConflictSlugs / partialOffersRename (reason -> action)", () => {
it("only name-conflict skips become the transient overlay / offer rename", () => {
const skipped = [
{ slug: "writer", name: "Writer", reason: "name-conflict" as const },
{ slug: "editor", name: "Editor", reason: "already-installed" as const },
];
expect(nameConflictSlugs(skipped)).toEqual(["writer"]);
expect(partialOffersRename(skipped)).toBe(true);
});
it("an already-installed-only skip is informational: no overlay, no rename", () => {
const skipped = [
{ slug: "editor", name: "Editor", reason: "already-installed" as const },
];
expect(nameConflictSlugs(skipped)).toEqual([]);
expect(partialOffersRename(skipped)).toBe(false);
});
});
describe("installedLangForRole", () => {
it("returns the other language when the same slug is installed elsewhere", () => {
const roles = [installedRole({ slug: "writer", language: "ru", version: 2 })];
expect(installedLangForRole("writer", roles, "en")).toBe("ru");
});
it("returns undefined when the same slug is installed in the SAME language", () => {
const roles = [installedRole({ slug: "writer", language: "en", version: 2 })];
expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
});
it("returns undefined when no install of the slug exists", () => {
expect(installedLangForRole("writer", [], "en")).toBeUndefined();
});
it("ignores manually-created roles (no source)", () => {
const roles = [
installedRole({ slug: "writer", language: "ru", version: 2 }, {
source: null,
}),
];
expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
});
});
describe("mapCatalogRoleToView", () => {
it("no install -> import status, catalog version, emoji preserved", () => {
const view = mapCatalogRoleToView(catalogRole(), [], "en");
expect(view).toMatchObject({
slug: "writer",
emoji: "✍️",
name: "Writer",
description: "Drafts copy.",
status: "import",
version: 3,
});
expect(view.installedRoleId).toBeUndefined();
expect(view.installedLang).toBeUndefined();
});
it("import with the slug installed in another language -> installedLang set", () => {
const roles = [installedRole({ slug: "writer", language: "ru", version: 9 })];
const view = mapCatalogRoleToView(catalogRole(), roles, "en");
expect(view.status).toBe("import");
expect(view.installedLang).toBe("ru");
});
it("installed (up to date) -> installed status, catalog version, installedRoleId", () => {
const installed = installedRole({
slug: "writer",
language: "en",
version: 3,
});
const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
expect(view).toMatchObject({
status: "installed",
version: 3,
installedRoleId: installed.id,
});
});
it("update -> version=from, newVersion=to, installedRoleId", () => {
const installed = installedRole({
slug: "writer",
language: "en",
version: 1,
});
const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
expect(view).toMatchObject({
status: "update",
version: 1,
newVersion: 3,
installedRoleId: installed.id,
});
});
it("missing emoji -> emoji undefined; null description -> empty string", () => {
const view = mapCatalogRoleToView(
catalogRole({ emoji: null, description: null }),
[],
"en",
);
expect(view.emoji).toBeUndefined();
expect(view.description).toBe("");
});
});
describe("mapBundleRolesToView", () => {
it("maps a bundle's roles preserving order", () => {
const roles = [
catalogRole({ slug: "a", name: "A", version: 1 }),
catalogRole({ slug: "b", name: "B", version: 1 }),
];
const installed = [installedRole({ slug: "a", language: "en", version: 1 })];
const view = mapBundleRolesToView(roles, installed, "en");
expect(view.map((r) => r.slug)).toEqual(["a", "b"]);
expect(view[0].status).toBe("installed");
expect(view[1].status).toBe("import");
});
});
@@ -0,0 +1,206 @@
import type {
IAiRole,
IAiRoleCatalogRole,
} from "@/features/ai-chat/types/ai-chat.types.ts";
import { catalogRoleInstallState } from "@/features/ai-chat/utils/catalog-role-install-state.ts";
/**
* The redesigned catalog modal renders bundles as cards with a summary status
* (readable without expanding) and a single primary action. The per-role and
* per-bundle view model that drives that UI is derived here as PURE functions so
* the mapping, the "installed in another language" hint, and the bundle-phase
* computation are unit-testable without mounting the component (mirrors the
* `catalogRoleInstallState` precedent).
*/
/**
* A role's status in the catalog view model.
* - `import` — not installed in the current content language.
* - `installed` — installed and up to date.
* - `update` — installed, but the catalog ships a newer version.
* - `skipped` — TRANSIENT client-only status set after a conflicted import
* (a name collision under `conflict:'skip'`); never from the
* backend.
*/
export type RoleStatus = "import" | "installed" | "update" | "skipped";
/** A catalog role mapped into the modal's view model. */
export interface CatalogViewRole {
// Slug is the stable identity within a bundle; used as the row key and as the
// `slugs[]` payload for import.
slug: string;
// Optional in the catalog — the row reserves space and renders nothing when
// absent.
emoji?: string;
name: string;
description: string;
// For `installed`/`import`: the catalog version. For `update`: the installed
// (from) version, with `newVersion` holding the catalog (to) version.
version: number;
newVersion?: number;
status: RoleStatus;
// The language a same-slug role is installed under, when it differs from the
// current content language (drives the Р5 hint). Only set for `import` roles.
installedLang?: string;
// The workspace role id, present for `installed`/`update` — needed to call the
// update-from-catalog mutation.
installedRoleId?: string;
}
/**
* The summary phase of a bundle, derived from its roles' statuses. Determines
* the collapsed-header summary and the bundle's single primary action.
* - `empty` — the bundle has no roles.
* - `allNew` — everything is importable, nothing installed.
* - `allInstalled` — everything installed & up to date; nothing else pending.
* - `updates` — updates available and nothing left to import.
* - `mixed` — any other combination.
*/
export type BundlePhase =
| "empty"
| "allNew"
| "allInstalled"
| "updates"
| "mixed";
/** Per-status tallies for a bundle's roles (the single source of truth). */
export interface BundleCounts {
importable: number;
installed: number;
update: number;
skipped: number;
}
/**
* Count a bundle's roles by status ONCE. Both `bundlePhase` and the panel derive
* from this, so the tally logic lives in exactly one place (no rescans / drift).
*/
export function bundleCounts(roles: CatalogViewRole[]): BundleCounts {
const counts: BundleCounts = {
importable: 0,
installed: 0,
update: 0,
skipped: 0,
};
for (const r of roles) {
if (r.status === "import") counts.importable += 1;
else if (r.status === "installed") counts.installed += 1;
else if (r.status === "update") counts.update += 1;
else if (r.status === "skipped") counts.skipped += 1;
}
return counts;
}
export function bundlePhase(roles: CatalogViewRole[]): BundlePhase {
if (roles.length === 0) return "empty";
const { importable, installed, update, skipped } = bundleCounts(roles);
// A `skipped` role is a pending post-import conflict (0 installed for it), so a
// bundle that has ANY skipped role is NOT "all installed & up to date" — that
// would make the collapsed green "up to date" header contradict the open
// panel's "Installed 0 · 1 skipped" plaque. It is `mixed` until resolved.
if (importable === 0 && update === 0 && skipped === 0) return "allInstalled";
if (update > 0 && importable === 0 && skipped === 0) return "updates";
if (importable > 0 && installed === 0 && update === 0 && skipped === 0)
return "allNew";
return "mixed";
}
/**
* The subset of a skip result that should be shown as a TRANSIENT `skipped`
* overlay in the bundle (so the row offers a re-import path). Only NAME-CONFLICT
* skips qualify: an `already-installed` skip (a concurrent-import race) has
* nothing to act on — re-importing the same slug would just skip again — so it
* must NOT be overlaid (else the row shows a misleading "Rename & install" that
* self-heals into a false "installed"). Pure so both reason branches are tested.
*/
export function nameConflictSlugs(
skipped: { slug: string; reason: "name-conflict" | "already-installed" }[],
): string[] {
return skipped
.filter((s) => s.reason === "name-conflict")
.map((s) => s.slug);
}
/**
* Whether a partial-import result should offer the "Rename & install" action:
* only when at least one skip is a name conflict (renameable). An
* `already-installed`-only partial is informational.
*/
export function partialOffersRename(
skipped: { reason: "name-conflict" | "already-installed" }[],
): boolean {
return skipped.some((s) => s.reason === "name-conflict");
}
/**
* For a role NOT installed in the current `language`, find a workspace role with
* the same catalog `slug` installed under a DIFFERENT language, and return that
* language. Drives the "installed in another language" hint (Р5): a different
* language of the same slug is a separate install and appears as `import`.
*/
export function installedLangForRole(
slug: string,
workspaceRoles: IAiRole[],
language: string,
): string | undefined {
const other = workspaceRoles.find(
(r) =>
r.source?.slug === slug &&
!!r.source?.language &&
r.source.language !== language,
);
return other?.source?.language;
}
/**
* Map one catalog role to the view model, computing its install status against
* the workspace roles (via `catalogRoleInstallState`) and, for importable roles,
* the other-language hint.
*/
export function mapCatalogRoleToView(
role: IAiRoleCatalogRole,
workspaceRoles: IAiRole[],
language: string,
): CatalogViewRole {
const state = catalogRoleInstallState(role, workspaceRoles, language);
const base = {
slug: role.slug,
emoji: role.emoji ?? undefined,
name: role.name,
description: role.description ?? "",
};
if (state.state === "update") {
return {
...base,
status: "update",
version: state.fromVersion,
newVersion: state.toVersion,
installedRoleId: state.installed.id,
};
}
if (state.state === "installed") {
return {
...base,
status: "installed",
version: role.version,
installedRoleId: state.installed.id,
};
}
return {
...base,
status: "import",
version: role.version,
installedLang: installedLangForRole(role.slug, workspaceRoles, language),
};
}
/**
* Map a whole bundle's catalog roles to the view model, preserving order.
*/
export function mapBundleRolesToView(
roles: IAiRoleCatalogRole[],
workspaceRoles: IAiRole[],
language: string,
): CatalogViewRole[] {
return roles.map((r) => mapCatalogRoleToView(r, workspaceRoles, language));
}
@@ -0,0 +1,112 @@
import { describe, it, expect } from "vitest";
import type { UIMessage } from "@ai-sdk/react";
import type { IAiChatMessageRow } from "@/features/ai-chat/types/ai-chat.types.ts";
import {
isStreamingTail,
isSettledAssistantTail,
seedRows,
mergeById,
} from "./resume-helpers.ts";
function row(
id: string,
role: string,
status?: string,
): IAiChatMessageRow {
return { id, role, content: "", status, createdAt: "2026-01-01T00:00:00Z" };
}
function makeMsg(id: string, text: string): UIMessage {
return {
id,
role: "assistant",
parts: [{ type: "text", text }],
} as UIMessage;
}
describe("isStreamingTail", () => {
it("is true when the last row is a streaming assistant row", () => {
expect(
isStreamingTail([row("u1", "user"), row("a1", "assistant", "streaming")]),
).toBe(true);
});
it("is false for a settled assistant tail", () => {
expect(isStreamingTail([row("a1", "assistant", "succeeded")])).toBe(false);
expect(isStreamingTail([row("a1", "assistant")])).toBe(false);
});
it("is false when the tail is a user row or the list is empty", () => {
expect(isStreamingTail([row("u1", "user")])).toBe(false);
expect(isStreamingTail([])).toBe(false);
});
});
describe("isSettledAssistantTail", () => {
it("is true for an assistant tail whose status is not streaming", () => {
expect(isSettledAssistantTail([row("a1", "assistant", "succeeded")])).toBe(
true,
);
expect(isSettledAssistantTail([row("a1", "assistant")])).toBe(true);
expect(isSettledAssistantTail([row("a1", "assistant", "aborted")])).toBe(
true,
);
});
it("is false for a streaming assistant tail", () => {
expect(isSettledAssistantTail([row("a1", "assistant", "streaming")])).toBe(
false,
);
});
it("is false when the tail is a user row or the list is empty", () => {
expect(isSettledAssistantTail([row("u1", "user")])).toBe(false);
expect(isSettledAssistantTail([])).toBe(false);
});
});
describe("seedRows", () => {
const rows = [row("u1", "user"), row("a1", "assistant", "streaming")];
it("returns the rows unchanged when not stripping", () => {
expect(seedRows(rows, false)).toBe(rows);
});
it("drops the last row when stripping", () => {
const seeded = seedRows(rows, true);
expect(seeded).toHaveLength(1);
expect(seeded[0].id).toBe("u1");
});
it("returns an empty list when stripping a single-row list", () => {
expect(seedRows([row("a1", "assistant", "streaming")], true)).toHaveLength(
0,
);
});
});
describe("mergeById", () => {
it("replaces the message with the same id in place (per-step growth)", () => {
const prev = [makeMsg("u1", "hi"), makeMsg("a1", "step 1")];
const incoming = makeMsg("a1", "step 1\nstep 2");
const next = mergeById(prev, incoming);
expect(next).toHaveLength(2);
expect(next[1]).toBe(incoming);
expect(next[0]).toBe(prev[0]); // untouched
expect(next).not.toBe(prev); // new array (never mutates input)
});
it("appends when the incoming message is not yet present", () => {
const prev = [makeMsg("u1", "hi")];
const incoming = makeMsg("a1", "first token");
const next = mergeById(prev, incoming);
expect(next).toHaveLength(2);
expect(next[1]).toBe(incoming);
});
it("returns the original list unchanged when there is nothing to merge", () => {
const prev = [makeMsg("u1", "hi")];
expect(mergeById(prev, null)).toBe(prev);
expect(mergeById(prev, undefined)).toBe(prev);
});
});
@@ -0,0 +1,62 @@
import type { UIMessage } from "@ai-sdk/react";
import type { IAiChatMessageRow } from "@/features/ai-chat/types/ai-chat.types.ts";
/**
* Pure decisions for the resumable-SSE resume machinery (#184 phase 1.5). A tab
* that reopens a chat whose agent run is still going attaches to the server's
* run-stream registry (replay + live tail) instead of polling snapshots; these
* small predicates decide WHICH tail is safe to resume and how to seed the store,
* extracted so they can be unit-tested in isolation.
*/
/**
* A STREAMING tail: the last persisted row is an assistant row still marked
* `status === 'streaming'`. Such a tail is stripped from the seed and rebuilt by
* the replay (`expect=live`), since the SDK's `text-start` always pushes a new
* part and replaying over a seeded in-progress row would duplicate its text.
*/
export function isStreamingTail(rows: IAiChatMessageRow[]): boolean {
const tail = rows[rows.length - 1];
return !!tail && tail.role === "assistant" && tail.status === "streaming";
}
/**
* A SETTLED assistant tail: the last row is an assistant row whose status is
* anything OTHER than 'streaming'. A settled assistant tail must NEVER resume —
* replaying a finished run into a store that already holds its message duplicates
* parts (`text-start` always pushes a new part).
*/
export function isSettledAssistantTail(rows: IAiChatMessageRow[]): boolean {
const tail = rows[rows.length - 1];
return !!tail && tail.role === "assistant" && tail.status !== "streaming";
}
/**
* Seed rows for `useChat`: return the rows unchanged, or without the last row when
* `strip` is set (the streaming tail is stripped so the live replay rebuilds it
* without duplicating parts).
*/
export function seedRows(
rows: IAiChatMessageRow[],
strip: boolean,
): IAiChatMessageRow[] {
return strip ? rows.slice(0, -1) : rows;
}
/**
* Merge an assistant message into the rendered list by id: replace the message
* with the same id in place (the in-progress assistant row is already seeded from
* history, so per-step growth replaces it), or append it when absent. Returns a
* new array; the input is never mutated.
*/
export function mergeById(
messages: UIMessage[],
incoming: UIMessage | null | undefined,
): UIMessage[] {
if (!incoming) return messages;
const idx = messages.findIndex((m) => m.id === incoming.id);
if (idx === -1) return [...messages, incoming];
const next = messages.slice();
next[idx] = incoming;
return next;
}
@@ -1,303 +0,0 @@
import { describe, it, expect } from "vitest";
import type { UIMessage } from "@ai-sdk/react";
import type { IAiChatRun } from "@/features/ai-chat/types/ai-chat.types.ts";
import {
RUN_POLL_INTERVAL_MS,
isRunActive,
runPollInterval,
shouldObserveRun,
shouldClearStoppingLatch,
shouldClearLatchOnQueryError,
mergeObservedMessage,
} from "./run-polling.ts";
function makeRun(status: string): IAiChatRun {
return { id: "run-1", chatId: "c1", status };
}
function makeMsg(id: string, text: string): UIMessage {
return {
id,
role: "assistant",
parts: [{ type: "text", text }],
} as UIMessage;
}
describe("isRunActive", () => {
it("treats pending and running as active", () => {
expect(isRunActive(makeRun("pending"))).toBe(true);
expect(isRunActive(makeRun("running"))).toBe(true);
});
it("treats terminal / unknown / nullish as not active", () => {
expect(isRunActive(makeRun("succeeded"))).toBe(false);
expect(isRunActive(makeRun("failed"))).toBe(false);
expect(isRunActive(makeRun("aborted"))).toBe(false);
expect(isRunActive(makeRun("weird-future-status"))).toBe(false);
expect(isRunActive(null)).toBe(false);
expect(isRunActive(undefined)).toBe(false);
});
});
describe("runPollInterval (the refetchInterval helper)", () => {
it("returns 2000ms while the run is pending/running", () => {
expect(runPollInterval(makeRun("pending"))).toBe(RUN_POLL_INTERVAL_MS);
expect(runPollInterval(makeRun("running"))).toBe(RUN_POLL_INTERVAL_MS);
expect(RUN_POLL_INTERVAL_MS).toBe(2000);
});
it("returns false (stop polling) once the run is terminal", () => {
expect(runPollInterval(makeRun("succeeded"))).toBe(false);
expect(runPollInterval(makeRun("failed"))).toBe(false);
expect(runPollInterval(makeRun("aborted"))).toBe(false);
});
it("returns false (no polling) when there is no run", () => {
expect(runPollInterval(null)).toBe(false);
expect(runPollInterval(undefined)).toBe(false);
});
});
describe("shouldObserveRun (observer-vs-streamer decision)", () => {
it("observes an active run when this tab is NOT the local streamer", () => {
expect(shouldObserveRun(makeRun("running"), false)).toBe(true);
expect(shouldObserveRun(makeRun("pending"), false)).toBe(true);
});
it("observes a terminal run too (so the final output shows on reopen)", () => {
expect(shouldObserveRun(makeRun("succeeded"), false)).toBe(true);
});
it("does NOT observe when this tab IS the streamer (no double-render)", () => {
expect(shouldObserveRun(makeRun("running"), true)).toBe(false);
expect(shouldObserveRun(makeRun("succeeded"), true)).toBe(false);
});
it("does NOT observe when there is no run", () => {
expect(shouldObserveRun(null, false)).toBe(false);
expect(shouldObserveRun(undefined, false)).toBe(false);
});
});
describe("shouldClearStoppingLatch (#234 latch-release decision)", () => {
// The one case the latch SHOULD clear: we requested a stop, we are the passive
// observer (not streaming), and the CURRENT run is terminal.
it("clears only when stopping, observing, and the run is terminal", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("aborted"),
isLocalStreaming: false,
}),
).toBe(true);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("succeeded"),
isLocalStreaming: false,
}),
).toBe(true);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("failed"),
isLocalStreaming: false,
}),
).toBe(true);
});
// Round-3 regression: clearing while THIS tab is still the local streamer would
// re-open the flash for the current turn the moment we switch to observer role.
// A predicate lacking the streaming gate would (wrongly) return true here.
it("does NOT clear while this tab is the local streamer", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("aborted"),
isLocalStreaming: true,
}),
).toBe(false);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("succeeded"),
isLocalStreaming: true,
}),
).toBe(false);
});
// The detached run keeps growing after a local abort — while it is still
// active the latch MUST hold so the observer merge stays suppressed.
it("does NOT clear while the run is still active", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("running"),
isLocalStreaming: false,
}),
).toBe(false);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("pending"),
isLocalStreaming: false,
}),
).toBe(false);
});
// #234 F4: on Stop the stale PREVIOUS-turn run is removed from the cache, so the
// observed `run` is null until the current turn's run is fetched fresh. A null
// run HOLDS the latch — it can never clear against the just-removed stale run,
// only against the current turn's own terminal run once observed.
it("does NOT clear against a removed/absent run (F4 stale-run guard)", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: null,
isLocalStreaming: false,
}),
).toBe(false);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: undefined,
isLocalStreaming: false,
}),
).toBe(false);
});
it("does NOT clear when no stop was requested", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: false,
run: makeRun("aborted"),
isLocalStreaming: false,
}),
).toBe(false);
});
});
describe("shouldClearLatchOnQueryError (#234 F7 error-safety-net decision)", () => {
// This guards the REAL anti-flash decision the component's run-query-error
// safety-net effect uses (ai-chat-window.tsx wires the effect to THIS helper,
// not a copy — so the test is non-vacuous vs the live code).
// (b) The F7 hole: a TRANSIENT run-query error while `run` is STILL ACTIVE must
// NOT clear the latch. TanStack Query v5 retains `data` on error, so
// runQueryFailed can be true while the held run is still pending/running.
// Against the PRE-F7 condition (without `!isRunActive(run)`) this would return
// true — so this assertion fails on the buggy code (non-vacuous).
it("does NOT clear on a transient error while the run is still ACTIVE (F7)", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: makeRun("running"),
}),
).toBe(false);
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: makeRun("pending"),
}),
).toBe(false);
});
// (a) The genuine permanent-null-freeze: run cache cleared by removeQueries +
// the refetch keeps ERRORING, so `run === null`. This is the ONLY case the
// safety-net exists to cure — it MUST clear so the frozen view resumes.
it("clears on a permanent error when the run is null (permanent-null-freeze)", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: null,
}),
).toBe(true);
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: undefined,
}),
).toBe(true);
});
// A TERMINAL run also satisfies `!isRunActive`; clearing then is harmless — the
// terminal effect (shouldClearStoppingLatch) already clears for a terminal run,
// so this only ever agrees with it. Asserted so the (c) reasoning is pinned.
it("clears on an error when the run is terminal (harmless, agrees with terminal effect)", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: makeRun("aborted"),
}),
).toBe(true);
});
it("does NOT clear without an actual query error", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: false,
run: null,
}),
).toBe(false);
});
it("does NOT clear while this tab is the local streamer", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: true,
runQueryFailed: true,
run: null,
}),
).toBe(false);
});
it("does NOT clear when no stop was requested", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: false,
isLocalStreaming: false,
runQueryFailed: true,
run: null,
}),
).toBe(false);
});
});
describe("mergeObservedMessage", () => {
it("replaces the message with the same id in place (per-step growth)", () => {
const prev = [makeMsg("u1", "hi"), makeMsg("a1", "step 1")];
const observed = makeMsg("a1", "step 1\nstep 2");
const next = mergeObservedMessage(prev, observed);
expect(next).toHaveLength(2);
expect(next[1]).toBe(observed);
expect(next[0]).toBe(prev[0]); // untouched
expect(next).not.toBe(prev); // new array (never mutates input)
});
it("appends when the observed message is not yet present", () => {
const prev = [makeMsg("u1", "hi")];
const observed = makeMsg("a1", "first token");
const next = mergeObservedMessage(prev, observed);
expect(next).toHaveLength(2);
expect(next[1]).toBe(observed);
});
it("returns the original list unchanged when there is nothing to merge", () => {
const prev = [makeMsg("u1", "hi")];
expect(mergeObservedMessage(prev, null)).toBe(prev);
expect(mergeObservedMessage(prev, undefined)).toBe(prev);
});
});
@@ -1,151 +0,0 @@
import type { UIMessage } from "@ai-sdk/react";
import type { IAiChatRun } from "@/features/ai-chat/types/ai-chat.types.ts";
/**
* Reconnect-and-live-follow helpers (#184). When a chat is reopened while its
* agent run is STILL going, this tab is a PASSIVE OBSERVER: it did not start the
* run here (no local SSE stream), so it catches up by POLLING the reconnect
* endpoint (`POST /ai-chat/run`) and merging the run's incrementally-persisted
* assistant message into the rendered thread. These are the small pure decisions
* that machinery hangs off, extracted so they can be unit-tested in isolation
* (mirrors how reindex polling / editor-sync-state are tested).
*/
/** How often to re-poll the reconnect endpoint while a run is ACTIVE. */
export const RUN_POLL_INTERVAL_MS = 2000;
// 'pending' and 'running' are the two ACTIVE statuses; 'succeeded' | 'failed' |
// 'aborted' are TERMINAL (and any unknown future status is treated as terminal,
// so a stale/odd value never polls forever).
const ACTIVE_STATUSES = new Set(["pending", "running"]);
/** Whether a run is still going (worth polling / merging live updates from). */
export function isRunActive(run: IAiChatRun | null | undefined): boolean {
return !!run && ACTIVE_STATUSES.has(run.status);
}
/**
* The TanStack Query `refetchInterval` value for the run query: poll every
* {@link RUN_POLL_INTERVAL_MS} while the run is active, and `false` (stop) once
* it is terminal or there is no run. Polling is thus naturally bounded by the run
* reaching a terminal status — no separate timeout cap is needed.
*/
export function runPollInterval(
run: IAiChatRun | null | undefined,
): number | false {
return isRunActive(run) ? RUN_POLL_INTERVAL_MS : false;
}
/**
* Observer-vs-streamer decision. We render the polled run message (catch up +
* keep advancing) ONLY when this tab is a passive observer: there IS a run AND
* this tab is NOT the one locally streaming it (we reconnected, we didn't start
* it here). When this tab is the streamer, the live SSE stream owns the view, so
* we neither poll nor merge — avoiding a double-render fight. Terminal runs still
* merge (so the final persisted output is shown on reopen); the poll itself is
* stopped separately by {@link runPollInterval}.
*/
export function shouldObserveRun(
run: IAiChatRun | null | undefined,
localStreaming: boolean,
): boolean {
return !!run && !localStreaming;
}
/**
* Should the "stopping" latch — which suppresses the observer re-stream flash
* after the user pressed Stop — be RELEASED now? All three must hold:
* - `stoppingRun`: we actually requested a stop (otherwise nothing to release);
* - `!isLocalStreaming`: this tab is NOT the local streamer. While we are the
* streamer the run query is disabled, so the observed `run` is not the run we
* are following — releasing the latch then would re-open the flash for the
* current turn the instant we switch to observer role;
* - the observed `run` EXISTS and has reached a TERMINAL status.
*
* The null / still-active `run` case is the #234 F4 invariant. On Stop the stale
* PREVIOUS-turn run is removed from the query cache (`removeQueries`), so `run`
* is null until the CURRENT turn's run is re-fetched fresh; a null or active run
* therefore HOLDS the latch, so it can only ever clear against the current turn's
* OWN terminal run — never a stale cached one. (The cache removal itself is
* integration-level in AiChatWindow; this predicate encodes the decision given
* whatever run is currently observed, and a stale terminal run is
* indistinguishable from a current terminal run at the predicate level — hence
* the cache removal is what guarantees only the current run is ever passed here.)
*/
export function shouldClearStoppingLatch(args: {
stoppingRun: boolean;
run: IAiChatRun | null | undefined;
isLocalStreaming: boolean;
}): boolean {
const { stoppingRun, run, isLocalStreaming } = args;
if (!stoppingRun || isLocalStreaming) return false;
return !!run && !isRunActive(run);
}
/**
* Should the "stopping" latch be RELEASED by the run-query ERROR safety-net?
* (#234 F7 — a NEW path of the same re-stream flash the F4 latch exists to
* prevent.) After Stop, `handleServerStop` clears the run cache; the terminal
* effect then holds the latch via `if (!run) return` until the CURRENT turn's run
* is fetched fresh. If that refetch instead ERRORS permanently, `run` stays null,
* its status-keyed refetchInterval is off, and nothing would ever observe a
* terminal run — freezing the view with the observer merge suppressed. This
* safety-net cures ONLY that genuine permanent-null-freeze.
*
* All four must hold:
* - `stoppingRun`: we actually requested a stop (otherwise nothing to release);
* - `!isLocalStreaming`: this tab is NOT the local streamer (same reason as
* {@link shouldClearStoppingLatch});
* - `runQueryFailed`: the run query is in its error state (TanStack Query v5 with
* retry:false — isError);
* - `!isRunActive(run)`: the observed `run` is NOT an active (pending/running)
* held run. This is the F7 gate. In TanStack Query v5 the query's `data` is
* RETAINED on error, so `runQueryFailed` can be true while `run` is STILL an
* ACTIVE run (a single transient GET-run failure in the window between Stop and
* settle). Without this gate a transient error would release the latch early —
* re-opening the observer merge and flashing the growing detached run over the
* frozen row (exactly the F4 flash). Gating on the run NOT being active means we
* only ever cure the permanent-null-freeze (`run === null`, so
* `isRunActive(null)` is false), never release against an active run.
*
* (A terminal `run` also satisfies `!isRunActive(run)`; clearing then is harmless
* — the terminal effect's {@link shouldClearStoppingLatch} already clears the
* latch for a terminal run, so this only ever agrees with it, never conflicts.)
*
* INVARIANT (do not break): clearing the latch on the `run === null` branch is safe
* ONLY because the run query's `refetchInterval` (see {@link runPollInterval}) stops
* polling when the data is empty — so after we clear on null+error there is no
* subsequent auto-poll that could return a still-active detached run and re-open the
* merge. If `refetchInterval` is ever changed to keep polling on `run === null`/on
* error, this null-branch clear would re-open the F7 flash through the null path.
* Do not change the run query's refetchInterval without re-checking this path.
*/
export function shouldClearLatchOnQueryError(args: {
stoppingRun: boolean;
isLocalStreaming: boolean;
runQueryFailed: boolean;
run: IAiChatRun | null | undefined;
}): boolean {
const { stoppingRun, isLocalStreaming, runQueryFailed, run } = args;
return (
stoppingRun && !isLocalStreaming && runQueryFailed && !isRunActive(run)
);
}
/**
* Merge an observed assistant message into the rendered list: replace the message
* with the same id in place (the in-progress assistant row is already seeded from
* history, so per-step growth replaces it), or append it when absent. Returns a
* new array; the input is never mutated.
*/
export function mergeObservedMessage(
messages: UIMessage[],
observed: UIMessage | null | undefined,
): UIMessage[] {
if (!observed) return messages;
const idx = messages.findIndex((m) => m.id === observed.id);
if (idx === -1) return [...messages, observed];
const next = messages.slice();
next[idx] = observed;
return next;
}
@@ -1,6 +1,7 @@
import { describe, it, expect } from "vitest";
import {
toolCitations,
toolInputSummary,
toolRunState,
type ToolUiPart,
} from "./tool-parts";
@@ -77,6 +78,138 @@ describe("toolCitations", () => {
});
});
describe("toolInputSummary", () => {
it("returns the primary `query` string", () => {
const part: ToolUiPart = {
type: "tool-Search_web_search",
state: "input-available",
input: { query: "hello world" },
};
expect(toolInputSummary(part)).toBe("hello world");
});
it("summarizes a primary array field with a (+N) suffix", () => {
// `urls` is an external MCP read_pages-style list; the first element plus a
// count of the rest.
const part: ToolUiPart = {
type: "tool-read_pages",
state: "input-available",
input: { urls: ["a", "b", "c"] },
};
expect(toolInputSummary(part)).toBe("a (+2)");
});
it("omits the (+N) suffix for a single-element array", () => {
const part: ToolUiPart = {
type: "tool-read_pages",
state: "input-available",
input: { urls: ["only"] },
};
expect(toolInputSummary(part)).toBe("only");
});
it("falls back to `title` for a page op with no query", () => {
const part: ToolUiPart = {
type: "tool-createPage",
state: "input-available",
input: { pageId: "x", title: "My Page" },
};
expect(toolInputSummary(part)).toBe("My Page");
});
it("prefers the earlier primary field when several are present", () => {
const part: ToolUiPart = {
type: "tool-x",
state: "input-available",
// `query` outranks `title` in PRIMARY_INPUT_FIELDS — the ordered list is
// the contract, so a reordering must break this test.
input: { query: "Q", title: "T" },
};
expect(toolInputSummary(part)).toBe("Q");
});
it("does not clamp a value exactly at the 140-char limit", () => {
const exact = "a".repeat(140);
const part: ToolUiPart = {
type: "tool-Search_web_search",
state: "input-available",
input: { query: exact },
};
const out = toolInputSummary(part)!;
expect(out).toBe(exact);
expect(out.endsWith("…")).toBe(false);
expect(out.length).toBe(140);
});
it("clamps one char over the limit (141 -> 140 + ellipsis)", () => {
const part: ToolUiPart = {
type: "tool-Search_web_search",
state: "input-available",
input: { query: "a".repeat(141) },
};
const out = toolInputSummary(part)!;
expect(out.endsWith("…")).toBe(true);
expect(out.length).toBe(141);
expect(out).toBe("a".repeat(140) + "…");
});
it("clamps a long value to ~140 chars with an ellipsis", () => {
const long = "a".repeat(300);
const part: ToolUiPart = {
type: "tool-Search_web_search",
state: "input-available",
input: { query: long },
};
const out = toolInputSummary(part)!;
expect(out.endsWith("…")).toBe(true);
expect(out.length).toBeLessThanOrEqual(141);
});
it("collapses newlines and repeated spaces to single spaces", () => {
const part: ToolUiPart = {
type: "tool-Search_web_search",
state: "input-available",
input: { query: " foo\n\n bar baz " },
};
expect(toolInputSummary(part)).toBe("foo bar baz");
});
it("returns undefined with no input", () => {
expect(
toolInputSummary({ type: "tool-x", state: "input-available" }),
).toBeUndefined();
});
it("returns undefined for an empty object input", () => {
expect(
toolInputSummary({
type: "tool-x",
state: "input-available",
input: {},
}),
).toBeUndefined();
});
it("returns undefined for a non-object input", () => {
expect(
toolInputSummary({
type: "tool-x",
state: "input-available",
input: "just a string",
}),
).toBeUndefined();
});
it("returns undefined while the input is still streaming (even with a full input)", () => {
const part: ToolUiPart = {
type: "tool-Search_web_search",
state: "input-streaming",
input: { query: "hello world" },
};
expect(toolInputSummary(part)).toBeUndefined();
});
});
describe("toolRunState", () => {
it('maps "output-error" to error', () => {
expect(toolRunState("output-error")).toBe("error");
@@ -97,6 +97,69 @@ function asString(value: unknown): string | undefined {
return typeof value === "string" && value.length > 0 ? value : undefined;
}
/** Collapse runs of whitespace/newlines to a single space and trim. */
function collapse(s: string): string {
return s.replace(/\s+/g, " ").trim();
}
/** Truncate to ~140 chars, appending an ellipsis when it overflows. */
function clamp(s: string): string {
const MAX = 140;
return s.length > MAX ? s.slice(0, MAX).trimEnd() + "…" : s;
}
/**
* Priority "primary" argument fields, in order. The first present one supplies
* the summary. `urls` is included (external MCP `read_pages`-style tools take a
* list of URLs) and is handled as an array; `url` covers the single-URL form.
*/
const PRIMARY_INPUT_FIELDS = [
"query",
"q",
"searchQuery",
"url",
"urls",
"title",
"name",
"text",
"prompt",
] as const;
/**
* A short, PLAIN-TEXT one-line summary of a tool call's arguments (e.g. the
* search query), or undefined when no recognizable primary field is present.
* Rendered under the tool label so tools without a friendly name (external MCP
* tools like `Search_web_search`) still show WHAT was requested, not just a
* generic "Ran tool {{name}}". The returned string is plain text and MUST be
* rendered React-escaped (Mantine `<Text>`), never as markdown/HTML.
*
* Streaming gate: while `state === "input-streaming"` the `input` object grows
* chunk by chunk but `messageSignature` deliberately does NOT track `input`, so
* a live summary computed here would freeze at its first captured value and go
* stale. We therefore return undefined until the state flips to
* `input-available` (input finalized) — that state change IS tracked by the
* signature, so the row re-renders and shows the complete summary. Do NOT add
* `input` to `message-signature.ts` to work around this.
*/
export function toolInputSummary(part: ToolUiPart): string | undefined {
if (part.state === "input-streaming") return undefined;
if (!part.input || typeof part.input !== "object") return undefined;
const input = part.input as Record<string, unknown>;
for (const field of PRIMARY_INPUT_FIELDS) {
const value = input[field];
if (typeof value === "string" && value.length > 0) {
return clamp(collapse(value));
}
if (Array.isArray(value) && value.length > 0) {
const first = collapse(String(value[0]));
if (first.length === 0) continue;
return clamp(first + (value.length > 1 ? ` (+${value.length - 1})` : ""));
}
}
return undefined;
}
/**
* Resolve the page citation(s) a tool part references, from its input/output.
* Only output-available parts (the tool returned) yield citations. Search
@@ -24,6 +24,7 @@ import {
GitmostListPagesResult,
GitmostListSpacesResult,
gitmostDecodePayloadToFile,
gitmostInsertTranscriptIntoEditor,
gitmostUploadFileToEditor,
} from "@/features/editor/gitmost/gitmost-recording.ts";
@@ -281,6 +282,18 @@ export default function GitmostGlobalBridge() {
pageId: page.id,
};
}
// Best-effort: append the transcript (heading + one paragraph per line)
// below the just-inserted audio node. The audio insert already
// succeeded, so a transcript failure must NOT turn this into an error —
// wrap it and, on any throw, log and still return ok. A missing/empty/
// non-string transcript is a no-op inside the helper (audio only).
try {
gitmostInsertTranscriptIntoEditor(editor, payload?.transcript);
} catch (err) {
console.error("[gitmost] transcript insert failed", err);
}
return { ok: true, pageId: page.id };
} catch (err: any) {
console.error("[gitmost] createPageWithRecording failed", err);
@@ -0,0 +1,150 @@
import { describe, it, expect } from "vitest";
import { Editor } from "@tiptap/core";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Heading } from "@tiptap/extension-heading";
import { Bold } from "@tiptap/extension-bold";
import { Italic } from "@tiptap/extension-italic";
import { Link } from "@tiptap/extension-link";
import { gitmostInsertTranscriptIntoEditor } from "./gitmost-recording.ts";
const ZWSP = "​"; // U+200B, the helper's block-trigger neutralizer
/**
* #377 — the web-side bridge must append the native host's transcript below the
* recording. These exercise the pure insert helper through a REAL Tiptap editor
* (Document/Paragraph/Text/Heading + Bold/Italic/Link marks so an HTML-parsing
* regression would be caught), asserting the resulting document rather than
* mocking the editor: transcript present -> "Transcript" heading + one paragraph
* per non-empty line; content is inserted as LITERAL TEXT (no HTML/markdown
* parsing); col-0 markdown block triggers are neutralized so git-sync keeps them
* paragraphs; absent/empty/non-string -> no-op.
*/
describe("gitmostInsertTranscriptIntoEditor", () => {
const makeEditor = () =>
new Editor({
// Bold/Italic/Link are registered specifically so that IF the helper ever
// regressed to inserting an HTML/markdown string (instead of a text node),
// TipTap would parse `<b>`/`*..*`/`[..](..)` into marks and the literal-
// text assertions below would fail.
extensions: [Document, Paragraph, Text, Heading, Bold, Italic, Link],
// Start from a single empty paragraph (a fresh page's baseline). The
// helper appends at the end of the doc, i.e. below existing content.
content: { type: "doc", content: [{ type: "paragraph" }] },
});
it("inserts a Transcript heading + one paragraph per non-empty line, verbatim", () => {
const editor = makeEditor();
const inserted = gitmostInsertTranscriptIntoEditor(
editor,
"You: hello there\nSpeaker 1: hi\n\nYou: bye",
);
expect(inserted).toBe(true);
const nodes = (editor.getJSON().content ?? []) as any[];
// A level-2 "Transcript" heading is present.
const heading = nodes.find((n) => n.type === "heading");
expect(heading?.attrs?.level).toBe(2);
expect(heading?.content?.[0]?.text).toBe("Transcript");
// Every non-empty transcript line becomes a paragraph, in order, verbatim;
// the blank line between them is dropped.
const texts = nodes
.filter((n) => n.type === "paragraph")
.map((n) => n.content?.[0]?.text)
.filter((t) => typeof t === "string");
expect(texts).toEqual(["You: hello there", "Speaker 1: hi", "You: bye"]);
editor.destroy();
});
it("inserts HTML + markdown metacharacters as LITERAL text (no injection / no mark parsing)", () => {
const editor = makeEditor();
const line =
"You: <b>bold</b> <script>alert(1)</script> and *stars* and [link](x)";
const inserted = gitmostInsertTranscriptIntoEditor(editor, line);
expect(inserted).toBe(true);
const paras = (editor.getJSON().content ?? []).filter(
(n: any) => n.type === "paragraph",
) as any[];
// The transcript line is exactly ONE paragraph holding a SINGLE text node
// whose text is the verbatim string — not split into bold/link/other nodes,
// not carrying any marks, not raw HTML. This FAILS if the helper switched to
// insertContent(htmlString): TipTap would then parse <b>/[link](x)/*stars*.
const content = paras[paras.length - 1].content;
expect(content).toHaveLength(1);
expect(content[0].type).toBe("text");
expect(content[0].marks ?? []).toEqual([]);
expect(content[0].text).toBe(line);
// And no bold/italic/link mark exists anywhere in the document.
const html = editor.getHTML();
expect(html).not.toMatch(/<(strong|b|em|i|a)\b/);
// The angle brackets survived as escaped entities (literal text), not a live
// <script>/<b> element.
expect(html).not.toMatch(/<script/i);
editor.destroy();
});
it("neutralizes col-0 markdown block triggers with a leading ZWSP (git-sync safety)", () => {
const editor = makeEditor();
// Trigger lines (some with a leaked indent) + a normal prefixed line.
const inserted = gitmostInsertTranscriptIntoEditor(
editor,
[
"- dash",
" > quote", // leading indent must be trimmed then neutralized
"# hash",
"1. one",
"> [!info] note",
"```js",
"---", // solid thematic break -> horizontalRule (text-losing) if unneutralized
"***",
"___",
"You: normal line",
].join("\n"),
);
expect(inserted).toBe(true);
const texts = (editor.getJSON().content ?? [])
.filter((n: any) => n.type === "paragraph")
.map((n: any) => n.content?.[0]?.text)
.filter((t: any) => typeof t === "string") as string[];
// Every block-trigger line is prefixed with the invisible ZWSP (indent
// trimmed first); the normal `You:` line is left byte-exact.
expect(texts).toEqual([
ZWSP + "- dash",
ZWSP + "> quote",
ZWSP + "# hash",
ZWSP + "1. one",
ZWSP + "> [!info] note",
ZWSP + "```js",
ZWSP + "---",
ZWSP + "***",
ZWSP + "___",
"You: normal line",
]);
editor.destroy();
});
it("is a no-op for undefined / empty / whitespace-only / non-string transcripts", () => {
for (const value of [undefined, "", " \n \n", 42, {}, null]) {
const editor = makeEditor();
const before = JSON.stringify(editor.getJSON());
const inserted = gitmostInsertTranscriptIntoEditor(editor, value as any);
expect(inserted).toBe(false);
// Document is untouched (audio-only behavior preserved).
expect(JSON.stringify(editor.getJSON())).toBe(before);
editor.destroy();
}
});
});
@@ -65,6 +65,11 @@ export interface GitmostCreatePagePayload {
base64: string;
filename: string;
mimeType: string;
// Optional transcript for the recording: plain text, `\n`-separated, each
// line already formatted as `You: ...` / `Speaker N: ...` by the native host
// (ready to insert, no parsing needed). Omitted (no speech / no models) ->
// audio only.
transcript?: string;
}
export interface GitmostCreatePageResult {
@@ -235,6 +240,83 @@ export async function gitmostUploadFileToEditor(
}
}
// Zero-width space (U+200B). Prepended to a transcript line that begins with a
// markdown BLOCK trigger: it is invisible in the rendered doc but shifts the
// trigger off column 0, so the git-sync doc->markdown->doc round-trip keeps the
// line a plain paragraph (see GITMOST_MD_BLOCK_TRIGGER_RE).
const GITMOST_ZWSP = "​";
// A markdown BLOCK-level construct that, sitting at column 0 of a paragraph
// line, the git-sync markdown serializer (packages/prosemirror-markdown
// markdown-converter.ts, `case "paragraph"`) would re-parse into a NON-paragraph
// block on the doc->markdown->doc cycle. That serializer emits paragraph text
// verbatim with NO block-escape (the pre-existing root cause), so a leading
// `#`/`-`/`*`/`+`/`>`, an ordered-list `N.`/`N)`, a code fence ```/~~~, a table
// `|`, or a `> [!info]` callout opener would silently become a heading / list /
// quote / code block / table / callout. The final alternative matches a WHOLE-
// LINE thematic break — solid `---`/`***`/`___` or spaced `- - -`/`_ _ _` (3+ of
// the same `-`/`*`/`_`) — which round-trips into a `horizontalRule`; because
// that node carries NO text, an un-neutralized separator line would LOSE its
// text entirely (worse than the list/quote case). This matches a TRIMMED line's
// start; the transcript's own `You:` / `Speaker N:` prefix begins with a letter
// and never matches, so prefixed lines are left byte-exact.
const GITMOST_MD_BLOCK_TRIGGER_RE =
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
// Append a transcript block BELOW the recording's audio node in a live editor:
// a "Transcript" heading followed by one paragraph per non-empty transcript
// line. The transcript is plain text, `\n`-separated, each line already
// formatted as `You: ...` / `Speaker N: ...` by the native host — line text is
// inserted as a TEXT node (never HTML/markdown), so there is no injection or
// mark-parsing surface. Each kept line is trimmed (drops an indent that would
// both leak into the display and, at col 0, form a markdown block trigger) and,
// if it still begins with a col-0 markdown block trigger, gets an invisible
// zero-width space prepended so the git-sync round-trip cannot turn it into a
// list/quote/heading/callout/code/table (defensive boundary against the
// serializer's missing block-escape). This is best-effort and meant to run
// AFTER the audio has already been inserted; the caller must guard against a
// throw so a transcript failure never fails the (already successful) recording.
// Returns true when a block was inserted, false when there was nothing to
// insert (transcript undefined/empty/not-a-string). A non-string value is a
// no-op, not an error.
export function gitmostInsertTranscriptIntoEditor(
editor: Editor,
transcript: unknown,
): boolean {
if (typeof transcript !== "string") return false;
const lines = transcript
.split("\n")
// Trim each line and drop blank (whitespace-only) ones.
.map((line) => line.trim())
.filter((line) => line.length > 0)
// Neutralize a col-0 markdown block trigger with an invisible ZWSP so the
// git-sync round-trip keeps the line a paragraph. Host lines (`You:` /
// `Speaker N:`) never match and stay byte-exact.
.map((line) =>
GITMOST_MD_BLOCK_TRIGGER_RE.test(line) ? GITMOST_ZWSP + line : line,
);
if (lines.length === 0) return false;
const content = [
{
type: "heading",
attrs: { level: 2 },
content: [{ type: "text", text: "Transcript" }],
},
...lines.map((line) => ({
type: "paragraph",
content: [{ type: "text", text: line }],
})),
];
// Append at the end of the document. On a freshly-created recording page the
// audio node is the last block, so the end position places the transcript
// directly below it.
const endPos = editor.state.doc.content.size;
editor.chain().focus().insertContentAt(endPos, content).run();
return true;
}
// Full insert path used by the open-page bridge (insertRecording): guard the
// editor, validate/decode the payload, then upload. Never throws — resolves to
// a result code.
@@ -0,0 +1,113 @@
import { describe, it, expect } from "vitest";
import { Editor } from "@tiptap/core";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { EditorState, TextSelection } from "@tiptap/pm/state";
import type { Node as PMNode } from "@tiptap/pm/model";
import { UniqueID } from "@docmost/editor-ext";
import { getEditorSelectionContext } from "./get-editor-selection";
/**
* Unit tests for getEditorSelectionContext (#388). Built on a headless
* ProseMirror schema (Document + Paragraph + Text + the block-id UniqueID
* extension), mirroring the editor-ext test style. We assemble docs with
* explicit block ids so the covered-blockIds assertions are deterministic.
*/
// A schema that carries the `id` block attribute (UniqueID) on paragraphs, just
// like the real editor.
const { schema } = new Editor({
extensions: [
Document,
Paragraph,
Text,
UniqueID.configure({ types: ["paragraph"] }),
],
content: "",
});
function docOf(blocks: { id: string; text: string }[]): PMNode {
return schema.node(
"doc",
null,
blocks.map((b) =>
schema.node("paragraph", { id: b.id }, b.text ? schema.text(b.text) : []),
),
);
}
function stateWith(doc: PMNode, from: number, to: number): EditorState {
const base = EditorState.create({ schema, doc });
return base.apply(base.tr.setSelection(TextSelection.create(doc, from, to)));
}
// Select every text position of the doc (pos 1 .. content.size - 1).
function selectAll(doc: PMNode): EditorState {
return stateWith(doc, 1, doc.content.size - 1);
}
describe("getEditorSelectionContext", () => {
it("returns null for an empty (collapsed) selection", () => {
const doc = docOf([{ id: "b1", text: "Hello world" }]);
const state = stateWith(doc, 3, 3); // caret, from === to
expect(getEditorSelectionContext(state)).toBeNull();
});
it("returns null for the default caret-at-start of a fresh editor", () => {
const editor = new Editor({
extensions: [
Document,
Paragraph,
Text,
UniqueID.configure({ types: ["paragraph"] }),
],
content: "<p>fresh</p>",
});
expect(getEditorSelectionContext(editor.state)).toBeNull();
editor.destroy();
});
it("reads a single-paragraph selection with no block-separator artifacts", () => {
const doc = docOf([{ id: "b1", text: "Hello world" }]);
const sel = getEditorSelectionContext(selectAll(doc))!;
expect(sel.text).toBe("Hello world");
expect(sel.blockIds).toEqual(["b1"]);
expect(sel.truncated).toBeUndefined();
});
it("joins multiple blocks with a newline and collects all covered blockIds", () => {
const doc = docOf([
{ id: "b1", text: "First" },
{ id: "b2", text: "Second" },
]);
const sel = getEditorSelectionContext(selectAll(doc))!;
expect(sel.text).toBe("First\nSecond");
expect(sel.blockIds).toEqual(["b1", "b2"]);
});
it("caps the text at 2000 chars and flags truncated", () => {
const doc = docOf([{ id: "b1", text: "x".repeat(2500) }]);
const sel = getEditorSelectionContext(selectAll(doc))!;
expect(sel.text).toHaveLength(2000);
expect(sel.truncated).toBe(true);
});
it("computes before/after context and clamps it to the doc bounds", () => {
// One paragraph "0123456789abcdefghij"; select the middle "56789".
const doc = docOf([{ id: "b1", text: "0123456789abcdefghij" }]);
// text char i lives at pos (1 + i); select chars index 5..9 -> pos 6..11.
const sel = getEditorSelectionContext(stateWith(doc, 6, 11))!;
expect(sel.text).toBe("56789");
expect(sel.before).toBe("01234");
expect(sel.after).toBe("abcdefghij");
});
it("omits before/after at the document boundaries (never reads past 0/size)", () => {
const doc = docOf([{ id: "b1", text: "Edge" }]);
const sel = getEditorSelectionContext(selectAll(doc))!;
// Selection spans the whole single block: nothing before or after it.
expect(sel.before).toBeUndefined();
expect(sel.after).toBeUndefined();
});
});
@@ -0,0 +1,71 @@
import type { EditorState } from "@tiptap/pm/state";
export interface EditorSelectionContext {
text: string;
truncated?: boolean;
blockIds?: string[];
before?: string;
after?: string;
}
// Client-side caps. The server re-caps every field independently (defence in
// depth — the payload is attacker-controllable), so these only keep the wire
// small for the common case.
const TEXT_CAP = 2000;
const CONTEXT_CHARS = 160;
const MAX_BLOCK_IDS = 20;
// Pure: takes an EditorState so it is unit-testable with a headless editor.
// Snapshots the user's current selection into the wire shape carried inside
// openPage — plain text + the ids of the blocks it covers + a little surrounding
// context. Returns null when nothing meaningful is selected.
//
// Deliberately does NOT emit the ProseMirror positions (from/to): they rot the
// instant the document changes and the server tools address content by block id
// + text (getNode / editPageText find-replace), never by position.
export function getEditorSelectionContext(
state: EditorState,
): EditorSelectionContext | null {
const { selection, doc } = state;
// An empty selection (incl. the default caret-at-start of a fresh editor) is
// never a "this"/"here" — bail before reading any text.
if (selection.empty) return null;
const { from, to } = selection;
let text = doc.textBetween(from, to, "\n");
let truncated = false;
if (text.length > TEXT_CAP) {
text = text.slice(0, TEXT_CAP);
truncated = true;
}
// A selection spanning only non-text nodes (e.g. an image) trims to empty ->
// treat as no selection.
if (text.trim().length === 0) return null;
// Ids of every block the selection covers, deduped and capped. These bridge
// the plain-text selection to the server tools (getNode / editPageText).
const blockIds: string[] = [];
doc.nodesBetween(from, to, (node) => {
const id = node.isBlock ? node.attrs?.id : undefined;
if (typeof id === "string" && id.length > 0 && !blockIds.includes(id)) {
blockIds.push(id);
}
});
// ~160 chars of plain text on each side, clamped to the document bounds, so
// editPageText can disambiguate a duplicate of the selected text.
const before = doc.textBetween(Math.max(0, from - CONTEXT_CHARS), from, "\n");
const after = doc.textBetween(
to,
Math.min(doc.content.size, to + CONTEXT_CHARS),
"\n",
);
const result: EditorSelectionContext = { text };
if (truncated) result.truncated = true;
if (blockIds.length > 0) result.blockIds = blockIds.slice(0, MAX_BLOCK_IDS);
if (before.length > 0) result.before = before;
if (after.length > 0) result.after = after;
return result;
}
@@ -165,6 +165,9 @@ export default function ShareAiWidget({
isStreaming={isStreaming}
assistantName={assistantName}
showCitations={false}
// Anonymous reader: suppress the tool-argument summary line so the
// agent's raw query/argument text isn't shown on the public share.
showInput={false}
// Anonymous reader: neutralize internal/relative links in the
// assistant's markdown so internal UUIDs/auth-gated routes don't
// leak as clickable links (external http(s) links are kept).
@@ -0,0 +1,329 @@
import { Injectable, Logger, OnModuleDestroy } from '@nestjs/common';
/**
* In-memory run-stream registry (#184 phase 1.5). A durable agent run tees its
* SSE frames here (via `pipeUIMessageStreamToResponse({ consumeSseStream })`)
* so a LATE tab one that reloaded, or opened after the starter dropped can
* attach through `GET /ai-chat/runs/:chatId/stream`, replay the frames buffered
* so far, and then follow the live tail as a normal streamer.
*
* This is deliberately single-process and best-effort: it holds nothing the DB
* does not (the run + assistant row are the source of truth), so a process
* restart simply drops in-flight entries and the client falls back to its
* restore + degraded-poll path. The async `attach` return type is the seam for a
* future phase-2 cross-process backend (Redis) the interface does not change.
*/
/** How long a finished entry is retained for late attach (replay + immediate end). */
export const RUN_STREAM_RETAIN_FINISHED_MS = 30_000;
/** Per-run replay buffer cap. Past this the buffer is dropped (attach -> 204). */
export const RUN_STREAM_MAX_BUFFER_BYTES = 4 * 1024 * 1024;
// 2x the replay cap: a just-written 4MB replay burst alone can never trip the
// per-subscriber cap (see controller); only a genuinely stalled socket can.
export const SUBSCRIBER_MAX_BUFFERED_BYTES = 2 * RUN_STREAM_MAX_BUFFER_BYTES;
export interface RunStreamCallbacks {
onFrame: (frame: string) => void;
onEnd: () => void;
}
export interface RunStreamAttachment {
replay: string[];
finished: boolean;
start(): void; // drain pending frames (order preserved) and go live
unsubscribe(): void; // safe to call at any point, idempotent
}
interface Subscriber extends RunStreamCallbacks {
started: boolean;
pending: string[];
// Byte size of `pending`, capped at SUBSCRIBER_MAX_BUFFERED_BYTES. `start()` is
// called in the SAME tick as `attach()` today (see attach), so `pending` never
// holds more than one microtask of frames — but the async `attach` signature is
// a phase-2 seam: an await between attach and start would let a stalled paused
// subscriber buffer the WHOLE run here. The cap is the structural backstop.
pendingBytes: number;
overflowed: boolean;
pendingEnd: boolean;
}
interface Entry {
runId: string;
// The persisted assistant row id of this run (set at bind; undefined if the
// seed failed). Used by the attach anchor check (invariant 6).
assistantMessageId?: string;
frames: string[];
bytes: number;
overflowed: boolean;
finished: boolean;
subscribers: Set<Subscriber>;
retainTimer?: NodeJS.Timeout;
}
@Injectable()
export class AiChatStreamRegistryService implements OnModuleDestroy {
private readonly logger = new Logger(AiChatStreamRegistryService.name);
private readonly entries = new Map<string, Entry>(); // key: chatId
/**
* Register a fresh entry at the START of a run (before any frame), so a tab
* that attaches in the begin->seed window finds an entry to wait on. If an
* entry already exists for this chat (a previous, possibly still-live run whose
* tee loop is draining), it is terminated MIRRORING the done-path (invariant 3)
* so its subscribers are released and its retention timer is cleared; a late
* `done` from that old tee then fires against the closed-over old reference and,
* thanks to identity checks, never touches this new entry.
*/
open(chatId: string, runId: string): void {
const existing = this.entries.get(chatId);
if (existing) {
if (existing.retainTimer) {
clearTimeout(existing.retainTimer);
existing.retainTimer = undefined;
}
// Started subscribers get exactly one onEnd() and are removed; paused ones
// are marked pendingEnd (their start() will end them). finished=true guards
// any later done from the old tee loop from double-notifying.
this.terminateSubscribers(existing);
}
this.entries.set(chatId, {
runId,
frames: [],
bytes: 0,
overflowed: false,
finished: false,
subscribers: new Set<Subscriber>(),
});
}
/**
* Tee a run's SSE frame stream into its entry (called from consumeSseStream).
* No-op with a warning when there is no entry or the entry belongs to a
* different run (invariant 1). The reader loop is fire-and-forget: the tee
* branch outlives the client socket by design.
*/
bind(
chatId: string,
runId: string,
assistantMessageId: string | undefined,
stream: ReadableStream<string>,
): void {
const entry = this.entries.get(chatId);
if (!entry || entry.runId !== runId) {
// Invariant 1: only the matching run may mutate the entry.
this.logger.warn(
`bind: no matching run-stream entry for chat=${chatId} run=${runId}`,
);
return;
}
entry.assistantMessageId = assistantMessageId;
const reader = stream.getReader();
const pump = async (): Promise<void> => {
try {
for (;;) {
const { done, value } = await reader.read();
if (done) break;
this.ingestFrame(entry, value);
}
this.finalizeEntry(chatId, entry);
} catch {
// A read error is a terminal event too — release subscribers.
this.finalizeEntry(chatId, entry);
}
};
void pump();
}
/**
* Terminate a run's entry from the OUTER catch of the stream method (a failure
* before/while wiring the pipe, so `done` will never arrive). Identity-checked
* on runId (invariant 1); the shared terminal path is idempotent.
*/
abortEntry(chatId: string, runId: string): void {
const entry = this.entries.get(chatId);
if (!entry || entry.runId !== runId) return;
this.finalizeEntry(chatId, entry);
}
/**
* Attach to a run's stream. Async only for the phase-2 Redis seam the body
* runs synchronously so the replay snapshot and the subscriber registration
* happen in ONE tick with no await between them (invariant 4): a frame ingested
* concurrently cannot slip into the gap and be lost or duplicated.
*
* Returns null (-> the caller answers 204) when:
* - there is no entry, or it overflowed (replay is gone);
* - expect=live with an anchor that does not match this run's assistant id
* (invariant 6: a stripped tab must never replay a FOREIGN run's transcript);
* - the run finished and the caller did not expect a live tail.
* A finished run with expect=live yields a replay-only attachment (no
* subscriber registered). Otherwise a paused subscriber is registered and the
* caller replays `replay`, then calls start() to drain and go live.
*/
async attach(
chatId: string,
expectLive: boolean,
anchor: string | undefined,
cb: RunStreamCallbacks,
): Promise<RunStreamAttachment | null> {
const entry = this.entries.get(chatId);
if (!entry || entry.overflowed) return null;
// Invariant 6: cross-run replay is forbidden. Before bind, assistantMessageId
// is undefined and mismatches any anchor -> 204 -> client restore+poll path.
if (expectLive && anchor && entry.assistantMessageId !== anchor) return null;
if (entry.finished && !expectLive) return null;
if (entry.finished && expectLive) {
// Replay-only: the run is done, no subscriber is registered.
return {
replay: entry.frames.slice(),
finished: true,
start: () => undefined,
unsubscribe: () => undefined,
};
}
const sub: Subscriber = {
onFrame: cb.onFrame,
onEnd: cb.onEnd,
started: false,
pending: [],
pendingBytes: 0,
overflowed: false,
pendingEnd: false,
};
entry.subscribers.add(sub);
// Snapshot in the SAME synchronous block as the registration (invariant 4).
const replay = entry.frames.slice();
// CONTRACT: the caller MUST call start() in the SAME tick as this attach()
// returns — no await between them. While a subscriber is paused, every frame
// is buffered in sub.pending; a delayed start() lets a whole run accumulate
// there. The pendingBytes cap (see ingestFrame) is the structural backstop if
// that contract is ever broken (e.g. the phase-2 Redis await seam).
return {
replay,
finished: false,
start: () => {
if (sub.overflowed) {
// The pending buffer overflowed while paused: end the stream instead of
// replaying a partial (a 204-equivalent post-attach degrade).
try {
sub.onEnd();
} catch {
// The socket is gone; nothing to end.
}
entry.subscribers.delete(sub);
return;
}
// Deliver frames buffered while paused, in order, then go live.
for (const frame of sub.pending) {
try {
sub.onFrame(frame);
} catch {
entry.subscribers.delete(sub);
return;
}
}
sub.pending = [];
sub.started = true;
if (sub.pendingEnd) {
try {
sub.onEnd();
} catch {
// The socket is gone; nothing to end.
}
entry.subscribers.delete(sub);
}
},
unsubscribe: () => {
entry.subscribers.delete(sub);
},
};
}
onModuleDestroy(): void {
for (const entry of this.entries.values()) {
if (entry.retainTimer) clearTimeout(entry.retainTimer);
}
this.entries.clear();
}
/** Buffer + fan-out a single frame. See invariant/overflow semantics inline. */
private ingestFrame(entry: Entry, frame: string): void {
entry.bytes += Buffer.byteLength(frame);
if (!entry.overflowed) {
entry.frames.push(frame);
if (entry.bytes > RUN_STREAM_MAX_BUFFER_BYTES) {
// The crossing frame was already counted AND (below) fanned out; only the
// replay buffer is dropped. After overflow no more frames are buffered,
// but live fan-out continues.
entry.overflowed = true;
entry.frames = [];
this.logger.warn(
`run-stream buffer overflow for run=${entry.runId}; ` +
`late attach will 204 until the run ends`,
);
}
}
for (const sub of entry.subscribers) {
if (sub.started) {
try {
sub.onFrame(frame);
} catch {
entry.subscribers.delete(sub);
}
} else {
sub.pending.push(frame);
sub.pendingBytes += Buffer.byteLength(frame);
if (sub.pendingBytes > SUBSCRIBER_MAX_BUFFERED_BYTES) {
// The paused subscriber's buffer overflowed — only possible if start()
// was delayed past the same-tick contract (the phase-2 await seam).
// Drop it rather than buffer the whole run; on start() it degrades to an
// immediate end (a 204-equivalent) instead of replaying a partial.
sub.overflowed = true;
sub.pending = [];
entry.subscribers.delete(sub);
}
}
}
}
/**
* Shared terminal path for done / read-error / external-abort. Idempotent: a
* second call (already finished) is a no-op, so an open()-replaced or
* abort-then-done entry is never double-armed or double-ended.
*/
private finalizeEntry(chatId: string, entry: Entry): void {
if (entry.finished) return;
this.terminateSubscribers(entry);
const timer = setTimeout(() => {
// Invariant 2: only delete OUR entry (a replacement may already own the key).
if (this.entries.get(chatId) === entry) this.entries.delete(chatId);
}, RUN_STREAM_RETAIN_FINISHED_MS);
timer.unref?.();
entry.retainTimer = timer;
}
/**
* Mark the entry finished and release its subscribers, mirroring the done-path:
* started subscribers get exactly one onEnd() and are removed; paused ones are
* flagged pendingEnd so their start() ends them. Deleting the current element
* during Set iteration is safe.
*/
private terminateSubscribers(entry: Entry): void {
entry.finished = true;
for (const sub of entry.subscribers) {
if (sub.started) {
try {
sub.onEnd();
} catch {
// The socket is gone; nothing to end.
}
entry.subscribers.delete(sub);
} else {
sub.pendingEnd = true;
}
}
}
}
@@ -0,0 +1,388 @@
import {
AiChatStreamRegistryService,
RUN_STREAM_MAX_BUFFER_BYTES,
RUN_STREAM_RETAIN_FINISHED_MS,
RunStreamCallbacks,
} from './ai-chat-stream-registry.service';
/**
* Unit tests for the in-memory run-stream registry (#184 phase 1.5). The registry
* is the whole of the resumable-transport contract: replay ordering, paused ->
* live hand-off, overflow, retention, the anchor check (invariant 6), and the
* mirror-the-done-path replace semantics (invariant 3). Every enumerated case in
* the issue's task 1.5 has a test here.
*/
// A ReadableStream whose frames the test pushes explicitly, plus close/error.
function makePushStream(): {
stream: ReadableStream<string>;
push: (f: string) => void;
close: () => void;
error: (e?: unknown) => void;
} {
let controller!: ReadableStreamDefaultController<string>;
const stream = new ReadableStream<string>({
start(c) {
controller = c;
},
});
return {
stream,
push: (f) => controller.enqueue(f),
close: () => controller.close(),
error: (e) => controller.error(e ?? new Error('read error')),
};
}
// Let the fire-and-forget pump drain queued frames (reader.read() resolves on a
// macrotask boundary for an already-enqueued value).
const flush = (): Promise<void> => new Promise((r) => setTimeout(r, 0));
function collector(): {
cb: RunStreamCallbacks;
frames: string[];
ended: () => number;
} {
const frames: string[] = [];
let ends = 0;
return {
frames,
ended: () => ends,
cb: {
onFrame: (f) => frames.push(f),
onEnd: () => {
ends += 1;
},
},
};
}
describe('AiChatStreamRegistryService', () => {
const CHAT = 'chat-1';
let registry: AiChatStreamRegistryService;
beforeEach(() => {
registry = new AiChatStreamRegistryService();
jest.spyOn((registry as any).logger, 'warn').mockImplementation(() => {});
});
afterEach(() => {
registry.onModuleDestroy();
});
it('replays frames in arrival order (live attach)', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
src.push('b');
src.push('c');
await flush();
const c = collector();
const att = await registry.attach(CHAT, false, undefined, c.cb);
expect(att).not.toBeNull();
expect(att!.replay).toEqual(['a', 'b', 'c']);
expect(att!.finished).toBe(false);
});
it('late attach gets the full prefix as replay plus the live tail', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
src.push('b');
await flush();
const c = collector();
const att = (await registry.attach(CHAT, false, undefined, c.cb))!;
expect(att.replay).toEqual(['a', 'b']);
att.start();
// Live tail arrives after start().
src.push('c');
src.push('d');
await flush();
expect(c.frames).toEqual(['c', 'd']);
});
it('a paused subscriber receives frames buffered during pause in order, then live (no loss/reorder)', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
await flush();
const c = collector();
// Attach (paused). Frames that arrive BEFORE start() must queue, not drop.
const att = (await registry.attach(CHAT, false, undefined, c.cb))!;
expect(att.replay).toEqual(['a']);
src.push('b'); // arrives while paused -> pending
src.push('c');
await flush();
expect(c.frames).toEqual([]); // nothing delivered yet (paused)
att.start(); // drains pending in order
expect(c.frames).toEqual(['b', 'c']);
src.push('d'); // now live
await flush();
expect(c.frames).toEqual(['b', 'c', 'd']);
});
it('a run that finishes while a subscriber is paused ends it on start()', async () => {
registry.open(CHAT, 'run-1');
const c = collector();
const att = (await registry.attach(CHAT, false, undefined, c.cb))!;
// Terminate the run while the subscriber is still paused.
registry.abortEntry(CHAT, 'run-1');
expect(c.ended()).toBe(0); // paused: not ended yet
att.start();
expect(c.ended()).toBe(1); // start() drains + ends
});
it('finished + expect=live returns a replay WITHOUT registering a subscriber', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
src.push('b');
src.close();
await flush();
const c = collector();
const att = (await registry.attach(CHAT, true, undefined, c.cb))!;
expect(att.finished).toBe(true);
expect(att.replay).toEqual(['a', 'b']);
// No subscriber registered: start()/unsubscribe are no-ops and the entry has
// zero subscribers.
const entry = (registry as any).entries.get(CHAT);
expect(entry.subscribers.size).toBe(0);
att.start();
expect(c.frames).toEqual([]);
});
it('finished WITHOUT expect=live returns null', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
src.close();
await flush();
const c = collector();
expect(await registry.attach(CHAT, false, undefined, c.cb)).toBeNull();
});
it('anchor mismatch with expect=live returns null (and null before bind sets assistantMessageId)', async () => {
registry.open(CHAT, 'run-1');
const c = collector();
// Before bind: assistantMessageId is undefined -> mismatches any anchor.
expect(
await registry.attach(CHAT, true, 'assist-1', c.cb),
).toBeNull();
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
await flush();
// Wrong anchor -> null (cross-run replay forbidden, invariant 6).
expect(await registry.attach(CHAT, true, 'other-id', c.cb)).toBeNull();
});
it('matching anchor with expect=live attaches', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
await flush();
const c = collector();
const att = await registry.attach(CHAT, true, 'assist-1', c.cb);
expect(att).not.toBeNull();
expect(att!.replay).toEqual(['a']);
});
it('overflow: attach returns null, but the LIVE subscriber keeps receiving (incl. the crossing frame)', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
// A live (started) subscriber attached before the flood.
const c = collector();
const att = (await registry.attach(CHAT, false, undefined, c.cb))!;
att.start();
const oneMb = 'x'.repeat(1024 * 1024);
// 5 x 1MB = 5MB > 4MB cap; the 5th frame is the one that crosses.
for (let i = 0; i < 5; i++) src.push(oneMb + i);
await flush();
const entry = (registry as any).entries.get(CHAT);
expect(entry.overflowed).toBe(true);
expect(entry.bytes).toBeGreaterThan(RUN_STREAM_MAX_BUFFER_BYTES);
// The live subscriber received ALL 5 frames, including the crossing one.
expect(c.frames).toHaveLength(5);
expect(c.frames[4]).toBe(oneMb + 4);
// A NEW attach after overflow gets null (replay buffer is gone).
const c2 = collector();
expect(await registry.attach(CHAT, false, undefined, c2.cb)).toBeNull();
});
it('a paused subscriber whose pending buffer overflows is dropped and ends on start(); other subscribers keep receiving', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
// A: paused (start() deliberately delayed to simulate the phase-2 await seam).
const a = collector();
const attA = (await registry.attach(CHAT, false, undefined, a.cb))!;
// B: live (started) — its delivery must be unaffected by A's overflow.
const b = collector();
const attB = (await registry.attach(CHAT, false, undefined, b.cb))!;
attB.start();
const oneMb = 'x'.repeat(1024 * 1024);
// 9 x 1MB = 9MB > 8MB per-subscriber cap; A's pending overflows, B streams live.
for (let i = 0; i < 9; i++) src.push(oneMb + i);
await flush();
const entry = (registry as any).entries.get(CHAT);
// A was dropped from the subscriber set on overflow; B (started) remains.
expect(entry.subscribers.size).toBe(1);
expect(a.frames).toEqual([]); // paused + overflowed: nothing was delivered
// B received every frame live (delivery unaffected by A's overflow).
expect(b.frames).toHaveLength(9);
// A's start() (arriving late) degrades to an immediate end, not a partial replay.
attA.start();
expect(a.frames).toEqual([]);
expect(a.ended()).toBe(1);
});
it('open() over a LIVE entry ends started subscribers exactly once and a late done does not touch the new entry (invariant 3)', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
src.push('a');
await flush();
const c = collector();
const att = (await registry.attach(CHAT, false, undefined, c.cb))!;
att.start(); // started subscriber on run-1
// run-2 starts on the same chat while run-1's tee is still reading.
registry.open(CHAT, 'run-2');
expect(c.ended()).toBe(1); // exactly one onEnd from the replace
const newEntry = (registry as any).entries.get(CHAT);
expect(newEntry.runId).toBe('run-2');
expect(newEntry.finished).toBe(false);
// The old tee now completes: its late done must NOT double-end nor delete the
// new entry.
src.push('b');
src.close();
await flush();
expect(c.ended()).toBe(1); // still exactly one
const still = (registry as any).entries.get(CHAT);
expect(still).toBe(newEntry);
expect(still.runId).toBe('run-2');
});
it('bind with a foreign runId is a no-op (invariant 1)', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'WRONG-run', 'assist-x', src.stream);
src.push('a');
await flush();
const entry = (registry as any).entries.get(CHAT);
// Frames were NOT ingested (bind bailed), assistantMessageId untouched.
expect(entry.frames).toEqual([]);
expect(entry.assistantMessageId).toBeUndefined();
});
it('abortEntry with a foreign runId is a no-op (invariant 1)', async () => {
registry.open(CHAT, 'run-1');
registry.abortEntry(CHAT, 'WRONG-run');
const entry = (registry as any).entries.get(CHAT);
expect(entry.finished).toBe(false);
});
it('a throwing onFrame ejects only that subscriber; the ingest loop stays alive', async () => {
registry.open(CHAT, 'run-1');
const src = makePushStream();
registry.bind(CHAT, 'run-1', 'assist-1', src.stream);
const bad = collector();
const badAtt = (await registry.attach(CHAT, false, undefined, {
onFrame: () => {
throw new Error('boom');
},
onEnd: bad.cb.onEnd,
}))!;
badAtt.start();
const good = collector();
const goodAtt = (await registry.attach(CHAT, false, undefined, good.cb))!;
goodAtt.start();
src.push('a'); // bad throws on this frame -> ejected
src.push('b'); // good still receives both
await flush();
const entry = (registry as any).entries.get(CHAT);
expect(entry.subscribers.size).toBe(1); // bad ejected, good remains
expect(good.frames).toEqual(['a', 'b']);
});
});
/**
* Retention + replace timer behavior. Fake timers, and entries are finalized via
* the synchronous abortEntry() path so no stream pump / microtask juggling is
* needed.
*/
describe('AiChatStreamRegistryService retention timers', () => {
const CHAT = 'chat-r';
let registry: AiChatStreamRegistryService;
beforeEach(() => {
jest.useFakeTimers();
registry = new AiChatStreamRegistryService();
jest.spyOn((registry as any).logger, 'warn').mockImplementation(() => {});
});
afterEach(() => {
registry.onModuleDestroy();
jest.useRealTimers();
});
it('a finished entry is removed after the retention window', () => {
registry.open(CHAT, 'run-1');
registry.abortEntry(CHAT, 'run-1'); // finalize -> retention armed
expect((registry as any).entries.get(CHAT)).toBeDefined();
jest.advanceTimersByTime(RUN_STREAM_RETAIN_FINISHED_MS + 1);
expect((registry as any).entries.get(CHAT)).toBeUndefined();
});
it('retention deletes ONLY its own entry (invariant 2)', () => {
registry.open(CHAT, 'run-1');
registry.abortEntry(CHAT, 'run-1'); // arm retention for entry A
// Simulate the race where the key was replaced without clearing A's timer.
const sentinel = { marker: true };
(registry as any).entries.set(CHAT, sentinel);
jest.advanceTimersByTime(RUN_STREAM_RETAIN_FINISHED_MS + 1);
// A's timer saw entries.get(CHAT) !== A, so it did NOT delete the successor.
expect((registry as any).entries.get(CHAT)).toBe(sentinel);
});
it('open() over a retained entry clears its timer and the successor survives', () => {
registry.open(CHAT, 'run-1');
registry.abortEntry(CHAT, 'run-1'); // retained, timer armed
const clearSpy = jest.spyOn(global, 'clearTimeout');
registry.open(CHAT, 'run-2'); // must clear run-1's retain timer
expect(clearSpy).toHaveBeenCalled();
jest.advanceTimersByTime(RUN_STREAM_RETAIN_FINISHED_MS + 1);
const entry = (registry as any).entries.get(CHAT);
expect(entry).toBeDefined();
expect(entry.runId).toBe('run-2');
});
});
@@ -0,0 +1,423 @@
import { ForbiddenException } from '@nestjs/common';
import { AiChatController } from './ai-chat.controller';
import type {
RunStreamAttachment,
RunStreamCallbacks,
} from './ai-chat-stream-registry.service';
import { SUBSCRIBER_MAX_BUFFERED_BYTES } from './ai-chat-stream-registry.service';
import type { User, Workspace } from '@docmost/db/types/entity.types';
/**
* Wiring spec for the #184 phase 1.5 attach endpoint
* (`GET /ai-chat/runs/:chatId/stream`). Owner-gated via assertOwnedChat; the
* registry is mocked so this exercises ONLY the controller's replay/live/204/
* cleanup wiring against a fake raw socket. Constructor order is (aiChatService,
* aiChatRunService, aiChatRepo, aiChatMessageRepo, aiTranscription, pageRepo,
* streamRegistry, environment).
*/
describe('AiChatController attach endpoint (#184 phase 1.5)', () => {
const user = { id: 'u1' } as User;
const workspace = { id: 'ws1' } as Workspace;
function makeRawRes() {
const raw: any = {
writableEnded: false,
writableLength: 0,
destroyed: false,
written: [] as string[],
head: null as any,
write: jest.fn((f: string) => {
raw.written.push(f);
return true;
}),
writeHead: jest.fn((code: number, headers: any) => {
raw.head = { code, headers };
return raw;
}),
flushHeaders: jest.fn(),
end: jest.fn(() => {
raw.writableEnded = true;
}),
destroy: jest.fn(() => {
raw.destroyed = true;
}),
on: jest.fn(),
once: jest.fn(),
};
const res: any = {
raw,
status: jest.fn(() => res),
send: jest.fn(),
hijack: jest.fn(),
};
return { res, raw };
}
function makeReq(destroyed = false) {
const handlers: Record<string, () => void> = {};
const raw: any = {
destroyed,
once: jest.fn((ev: string, fn: () => void) => {
handlers[ev] = fn;
}),
};
return { req: { raw } as any, raw, fireClose: () => handlers['close']?.() };
}
function makeAttachment(
over: Partial<RunStreamAttachment> = {},
): RunStreamAttachment {
return {
replay: [],
finished: false,
start: jest.fn(),
unsubscribe: jest.fn(),
...over,
};
}
function makeController(opts: {
chat?: unknown;
attachment?: RunStreamAttachment | null;
}) {
const aiChatRepo = { findById: jest.fn().mockResolvedValue(opts.chat) };
let capturedCb: RunStreamCallbacks | undefined;
const streamRegistry = {
attach: jest.fn(
(
_chatId: string,
_live: boolean,
_anchor: string | undefined,
cb: RunStreamCallbacks,
) => {
capturedCb = cb;
return Promise.resolve(
opts.attachment === undefined ? makeAttachment() : opts.attachment,
);
},
),
};
const environment = { isAiChatResumableStreamEnabled: () => true };
const controller = new AiChatController(
{} as never, // aiChatService
{} as never, // aiChatRunService
aiChatRepo as never,
{} as never, // aiChatMessageRepo
{} as never, // aiTranscription
{} as never, // pageRepo
streamRegistry as never,
environment as never,
);
return {
controller,
aiChatRepo,
streamRegistry,
getCb: () => capturedCb!,
};
}
const owned = { id: 'c1', creatorId: 'u1' };
it('owner-gates: a foreign chat throws ForbiddenException and never attaches', async () => {
const { controller, streamRegistry } = makeController({
chat: { id: 'c1', creatorId: 'someone-else' },
});
const { res } = makeRawRes();
const { req } = makeReq();
await expect(
controller.attachRunStream(
'c1',
undefined,
undefined,
req,
res,
user,
workspace,
),
).rejects.toBeInstanceOf(ForbiddenException);
expect(streamRegistry.attach).not.toHaveBeenCalled();
});
it('answers 204 when the registry has nothing to resume (no entry / finished / anchor-mismatch)', async () => {
const { controller } = makeController({ chat: owned, attachment: null });
const { res } = makeRawRes();
const { req } = makeReq();
await controller.attachRunStream(
'c1',
undefined,
undefined,
req,
res,
user,
workspace,
);
expect(res.status).toHaveBeenCalledWith(204);
expect(res.send).toHaveBeenCalled();
expect(res.hijack).not.toHaveBeenCalled();
});
it('threads expect=live and anchor through to the registry', async () => {
const { controller, streamRegistry } = makeController({
chat: owned,
attachment: null,
});
const { res } = makeRawRes();
const { req } = makeReq();
await controller.attachRunStream(
'c1',
'live',
'anchor-1',
req,
res,
user,
workspace,
);
expect(streamRegistry.attach).toHaveBeenCalledWith(
'c1',
true,
'anchor-1',
expect.anything(),
);
});
it('passes expect=false when the query is absent', async () => {
const { controller, streamRegistry } = makeController({
chat: owned,
attachment: null,
});
const { res } = makeRawRes();
const { req } = makeReq();
await controller.attachRunStream(
'c1',
undefined,
undefined,
req,
res,
user,
workspace,
);
expect(streamRegistry.attach).toHaveBeenCalledWith(
'c1',
false,
undefined,
expect.anything(),
);
});
it('hijacks, writes headers + replay, registers a close cleanup, then goes live', async () => {
const start = jest.fn();
const attachment = makeAttachment({
replay: ['f1', 'f2'],
finished: false,
start,
});
const { controller } = makeController({ chat: owned, attachment });
const { res, raw } = makeRawRes();
const { req } = makeReq();
await controller.attachRunStream(
'c1',
undefined,
undefined,
req,
res,
user,
workspace,
);
expect(res.hijack).toHaveBeenCalled();
expect(raw.writeHead).toHaveBeenCalledWith(
200,
expect.objectContaining({ 'content-type': 'text/event-stream' }),
);
expect(raw.written).toEqual(['f1', 'f2']); // replay
expect(start).toHaveBeenCalled(); // go live after replay
expect(req.raw.once).toHaveBeenCalledWith('close', expect.any(Function));
});
it('finished replay ends the response immediately without going live', async () => {
const start = jest.fn();
const attachment = makeAttachment({
replay: ['f1'],
finished: true,
start,
});
const { controller } = makeController({ chat: owned, attachment });
const { res, raw } = makeRawRes();
const { req } = makeReq();
await controller.attachRunStream(
'c1',
'live',
'a1',
req,
res,
user,
workspace,
);
expect(raw.written).toEqual(['f1']);
expect(raw.end).toHaveBeenCalled();
expect(start).not.toHaveBeenCalled(); // finished -> returns before start()
});
it('a close during the awaits (req.raw.destroyed) unsubscribes and writes nothing', async () => {
const attachment = makeAttachment({ replay: ['f1'] });
const { controller } = makeController({ chat: owned, attachment });
const { res, raw } = makeRawRes();
const { req } = makeReq(true); // destroyed already at registration time
await controller.attachRunStream(
'c1',
undefined,
undefined,
req,
res,
user,
workspace,
);
expect(attachment.unsubscribe).toHaveBeenCalled();
expect(raw.writeHead).not.toHaveBeenCalled();
expect(raw.written).toEqual([]);
});
it('the registered close handler unsubscribes the attachment', async () => {
const attachment = makeAttachment({ replay: [] });
const { controller } = makeController({ chat: owned, attachment });
const { res } = makeRawRes();
const { req, fireClose } = makeReq();
await controller.attachRunStream(
'c1',
undefined,
undefined,
req,
res,
user,
workspace,
);
expect(attachment.unsubscribe).not.toHaveBeenCalled();
fireClose(); // socket closed
expect(attachment.unsubscribe).toHaveBeenCalled();
});
it('onFrame destroys the socket when the buffered length exceeds the cap', async () => {
const attachment = makeAttachment({ replay: [] });
const { controller, getCb } = makeController({ chat: owned, attachment });
const { res, raw } = makeRawRes();
const { req } = makeReq();
await controller.attachRunStream(
'c1',
undefined,
undefined,
req,
res,
user,
workspace,
);
const cb = getCb();
// Normal live frame writes.
raw.writableLength = 0;
cb.onFrame('live-1');
expect(raw.written).toContain('live-1');
// A stalled socket over the cap is destroyed instead of buffering.
raw.writableLength = SUBSCRIBER_MAX_BUFFERED_BYTES + 1;
raw.write.mockClear();
cb.onFrame('too-much');
expect(raw.destroy).toHaveBeenCalled();
expect(raw.write).not.toHaveBeenCalled();
});
});
/**
* The begin-hook `open()` flag gate (#184 phase 1.5). `open()` lives ONLY in the
* stream() begin-hook, gated on the resumable flag. If it regressed, a flag-off
* turn would create an EMPTY registry entry (never bound, never finished) and a
* later attach would find a non-null paused attachment -> a hung SSE that never
* gets a frame and never ends, instead of a clean 204. These drive stream() only
* far enough to capture the runHooks it hands to the service, then invoke the
* begin-hook and assert whether the registry was opened.
*/
describe('AiChatController begin-hook open() flag gate (#184 phase 1.5)', () => {
const user = { id: 'u1' } as User;
const workspace = {
id: 'ws1',
settings: { ai: { chat: true, autonomousRuns: true } },
} as unknown as Workspace;
function makeController(opts: { resumable: boolean }) {
let capturedArgs: any;
const aiChatService = {
resolveRoleForRequest: jest.fn(async () => null),
getChatModel: jest.fn(async () => ({})),
stream: jest.fn(async (args: any) => {
capturedArgs = args;
}),
};
const aiChatRunService = {
getActiveForChat: jest.fn(async () => undefined),
beginRun: jest.fn(async () => ({
runId: 'run-1',
signal: new AbortController().signal,
})),
};
const streamRegistry = { open: jest.fn(), attach: jest.fn() };
const environment = {
isAiChatResumableStreamEnabled: () => opts.resumable,
};
const controller = new AiChatController(
aiChatService as never,
aiChatRunService as never,
{} as never, // aiChatRepo
{} as never, // aiChatMessageRepo
{} as never, // aiTranscription
{} as never, // pageRepo
streamRegistry as never,
environment as never,
);
return {
controller,
streamRegistry,
aiChatRunService,
getRunHooks: () => capturedArgs?.runHooks,
};
}
function makeReqRes() {
const req: any = {
raw: { sessionId: 'sess-1', once: jest.fn() },
body: {
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
},
};
const res: any = {
raw: { once: jest.fn(), on: jest.fn(), headersSent: false, writableEnded: false },
hijack: jest.fn(),
};
return { req, res };
}
it('flag OFF: the begin-hook does NOT open a registry entry (no hung empty entry)', async () => {
const { controller, streamRegistry, aiChatRunService, getRunHooks } =
makeController({ resumable: false });
const { req, res } = makeReqRes();
await controller.stream(req, res, user, workspace);
const runHooks = getRunHooks();
expect(runHooks).toBeDefined();
const handle = await runHooks.begin('chat-1');
// The run still begins (the durable-run feature is independent of resume)...
expect(aiChatRunService.beginRun).toHaveBeenCalled();
expect(handle).toEqual({ runId: 'run-1', signal: expect.anything() });
// ...but with the flag off the registry entry is NEVER opened.
expect(streamRegistry.open).not.toHaveBeenCalled();
});
it('flag ON: the begin-hook opens the registry entry with (chatId, runId)', async () => {
const { controller, streamRegistry, getRunHooks } = makeController({
resumable: true,
});
const { req, res } = makeReqRes();
await controller.stream(req, res, user, workspace);
const runHooks = getRunHooks();
await runHooks.begin('chat-1');
expect(streamRegistry.open).toHaveBeenCalledWith('chat-1', 'run-1');
});
});
@@ -4,11 +4,15 @@ import {
ConflictException,
Controller,
ForbiddenException,
Get,
HttpCode,
HttpException,
HttpStatus,
Logger,
Param,
ParseUUIDPipe,
Post,
Query,
Req,
Res,
ServiceUnavailableException,
@@ -54,6 +58,12 @@ import {
} from './dto/ai-chat.dto';
import { describeProviderError } from '../../integrations/ai/ai-error.util';
import { buildChatMarkdown } from './chat-markdown.util';
import {
AiChatStreamRegistryService,
SUBSCRIBER_MAX_BUFFERED_BYTES,
} from './ai-chat-stream-registry.service';
import { startSseHeartbeat } from './sse-resilience';
import { EnvironmentService } from '../../integrations/environment/environment.service';
/**
* Per-user AI chat API (§6.1). Routes are POST to match this codebase's
@@ -72,6 +82,11 @@ export class AiChatController {
private readonly aiChatMessageRepo: AiChatMessageRepo,
private readonly aiTranscription: AiTranscriptionService,
private readonly pageRepo: PageRepo,
// #184 phase 1.5. OPTIONAL so existing positional constructions (controller
// specs) compile unchanged; Nest always injects the real providers in
// production. Only touched on the resumable-stream (flag-on) path.
private readonly streamRegistry?: AiChatStreamRegistryService,
private readonly environment?: EnvironmentService,
) {}
/** List the requesting user's chats in this workspace (paginated). */
@@ -233,6 +248,102 @@ export class AiChatController {
return { stopped };
}
/**
* Attach to a chat's live run stream (#184 phase 1.5). A late/reloaded tab
* replays the frames buffered so far and then follows the live tail as a normal
* streamer. Owner-gated via assertOwnedChat (same gate as getRun). When there is
* nothing to resume no entry, a finished run without expect=live, an
* overflowed buffer, or an anchor that pins a DIFFERENT run the endpoint
* answers 204, the ONLY "nothing to resume" signal the AI SDK's reconnect
* accepts (it maps 204 to a silent no-op). With AI_CHAT_RESUMABLE_STREAM off the
* registry is never populated, so attach always 204s.
*
* `expect=live` opts into replaying a finished-but-retained run (safe only when
* the client stripped the streaming tail); `anchor` is the client's assistant
* row id, which must match this run's (invariant 6) or a foreign run's
* transcript would be replayed into the store.
*/
@SkipTransform()
@UseGuards(JwtAuthGuard, UserThrottlerGuard)
@Throttle({ [AI_CHAT_THROTTLER]: { limit: 60, ttl: 60000 } })
@Get('runs/:chatId/stream')
async attachRunStream(
@Param('chatId', new ParseUUIDPipe()) chatId: string,
@Query('expect') expect: string | undefined,
@Query('anchor') anchor: string | undefined,
@Req() req: FastifyRequest,
@Res() res: FastifyReply,
@AuthUser() user: User,
@AuthWorkspace() workspace: Workspace,
): Promise<void> {
await this.assertOwnedChat(chatId, user, workspace); // same gate as getRun
let stopHeartbeat: () => void = () => undefined;
const attachment = await this.streamRegistry?.attach(
chatId,
expect === 'live',
anchor,
{
onFrame: (frame) => {
// Backpressure guard: 2x the replay cap, so the initial replay burst
// alone can never trip it; only a genuinely stalled socket can.
try {
if (res.raw.writableLength > SUBSCRIBER_MAX_BUFFERED_BYTES) {
res.raw.destroy(); // 'close' fires -> unsubscribe below
return;
}
if (!res.raw.writableEnded) res.raw.write(frame);
} catch {
res.raw.destroy();
}
},
onEnd: () => {
stopHeartbeat();
if (!res.raw.writableEnded) res.raw.end();
},
},
);
if (!attachment) {
res.status(204).send(); // the ONLY "nothing to resume" signal the SDK accepts
return;
}
res.hijack();
// Cleanup BEFORE any write (invariant 5): a torn-down socket must not orphan
// a paused subscriber whose pending queue would buffer the whole run.
req.raw.once('close', () => {
attachment.unsubscribe();
stopHeartbeat();
});
// A close emitted DURING the awaits above was missed by the listener — check.
// (Healthy pending GETs have req.raw.destroyed === false, so no false
// positives; returning without end() is fine — the socket is gone.)
if (req.raw.destroyed) {
attachment.unsubscribe();
return;
}
res.raw.on('error', () => undefined);
try {
res.raw.writeHead(200, {
'content-type': 'text/event-stream',
'cache-control': 'no-cache',
'x-vercel-ai-ui-message-stream': 'v1',
'x-accel-buffering': 'no',
// deliberately NO Connection/Keep-Alive (hop-by-hop; Safari/HTTP2)
});
res.raw.flushHeaders?.();
for (const frame of attachment.replay) res.raw.write(frame);
if (attachment.finished) {
res.raw.end();
return;
}
stopHeartbeat = startSseHeartbeat(res.raw, 15_000);
attachment.start(); // drain pending accumulated during replay, go live
} catch {
attachment.unsubscribe();
stopHeartbeat();
res.raw.destroy();
}
}
/** Rename a chat. */
@HttpCode(HttpStatus.OK)
@Post('rename')
@@ -344,13 +455,25 @@ export class AiChatController {
// its progress, and settle its terminal status — see AiChatRunService.
const runHooks: AiChatRunHooks | undefined = autonomousRuns
? {
begin: (chatId) =>
this.aiChatRunService.beginRun({
begin: async (chatId) => {
const handle = await this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
}),
});
// #184 phase 1.5: register the run-stream entry at BEGIN (before any
// frame) so a tab that attaches in the begin->seed window finds an
// entry to wait on. Gated on AI_CHAT_RESUMABLE_STREAM: with the flag
// off nothing is registered and attach always 204s.
if (
handle?.runId &&
this.environment?.isAiChatResumableStreamEnabled?.()
) {
this.streamRegistry?.open(chatId, handle.runId);
}
return handle;
},
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
@@ -4,6 +4,7 @@ import { TokenModule } from '../auth/token.module';
import { AiChatController } from './ai-chat.controller';
import { AiChatService } from './ai-chat.service';
import { AiChatRunService } from './ai-chat-run.service';
import { AiChatStreamRegistryService } from './ai-chat-stream-registry.service';
import { AiTranscriptionService } from './ai-transcription.service';
import { AiChatToolsService } from './tools/ai-chat-tools.service';
import { EmbeddingModule } from './embedding/embedding.module';
@@ -44,6 +45,7 @@ import { PublicShareChatToolsService } from './tools/public-share-chat-tools.ser
providers: [
AiChatService,
AiChatRunService,
AiChatStreamRegistryService,
AiTranscriptionService,
AiChatToolsService,
PublicShareChatService,
@@ -153,6 +153,41 @@ describe('buildSystemPrompt current-page context', () => {
expect(prompt).not.toContain('pageId:');
});
// #388: editor-selection flag. Only a FIXED one-liner is added — the selection
// TEXT (untrusted page content) must never reach the prompt.
const SELECTION_FLAG = 'currently has text SELECTED on this page';
it('adds the selection flag when a selection is present with a page', () => {
const prompt = buildSystemPrompt({
workspace,
openedPage: {
id: 'pg-123',
title: 'Doc',
selection: { text: 'SECRET-SELECTED-TEXT', blockIds: ['b1'] },
},
});
expect(prompt).toContain(SELECTION_FLAG);
// The selection TEXT itself is NEVER in the prompt.
expect(prompt).not.toContain('SECRET-SELECTED-TEXT');
expect(prompt).not.toContain('b1');
});
it('omits the selection flag when there is no selection', () => {
const prompt = buildSystemPrompt({
workspace,
openedPage: { id: 'pg-123', title: 'Doc' },
});
expect(prompt).not.toContain(SELECTION_FLAG);
});
it('omits the selection flag when selection is null', () => {
const prompt = buildSystemPrompt({
workspace,
openedPage: { id: 'pg-123', title: 'Doc', selection: null },
});
expect(prompt).not.toContain(SELECTION_FLAG);
});
it('escapes a malicious opened-page title so it cannot inject tags (F1)', () => {
const prompt = buildSystemPrompt({
workspace,
+14 -1
View File
@@ -156,8 +156,13 @@ export interface BuildSystemPromptInput {
* has an id, a CONTEXT line is added so the agent can resolve "this page" /
* "the current page" to that pageId. The page is NOT fetched here the agent
* uses its CASL-enforced read/write page tools with the id when needed.
*
* `selection` (#388) is present only when the user has a non-empty editor
* selection; the prompt adds ONLY a fixed one-line flag from it the
* selection TEXT is untrusted page content and stays out of the prompt (it is
* surfaced solely via the getCurrentPage tool result).
*/
openedPage?: { id?: string; title?: string } | null;
openedPage?: { id?: string; title?: string; selection?: object | null } | null;
/**
* Admin-authored, per-EXTERNAL-MCP-server guidance ("how/when to use this
* server's tools"), built by `McpClientsService.toolsFor` for servers that
@@ -309,6 +314,14 @@ export function buildSystemPrompt({
? escapeAttr(openedPage.title)
: 'Untitled';
context += `\nThe user is currently viewing the page "${title}" (pageId: ${pageId.trim()}). When they refer to "this page", "the current page", or similar, operate on that pageId — use the read/write page tools with it.`;
// Editor-selection flag (#388). A FIXED one-liner only — the selection TEXT
// is untrusted collaborative-page content and must never enter the prompt; it
// is surfaced solely through the getCurrentPage tool result (SAFETY_FRAMEWORK
// treats a tool result as data). Nested under the page block so it is added
// only alongside a resolved page (a selection cannot outlive its page).
if (openedPage?.selection) {
context += `\nThe user currently has text SELECTED on this page — call getCurrentPage to see the selection. When they say "this", "here", "the selected text" or similar, they mean that selection.`;
}
}
// Interrupt-resume marker (#198). Added to the context section (inside the
@@ -1,4 +1,13 @@
import { ForbiddenException } from '@nestjs/common';
import { ForbiddenException, Logger } from '@nestjs/common';
// Mock ONLY streamText so a driven stream() call can capture the pipe-options
// object (consumeSseStream / generateMessageId). Everything else in the AI SDK
// stays REAL (requireActual), so the pure-helper suites in this file are
// unaffected — none of them call stream()/streamText.
jest.mock('ai', () => ({
...jest.requireActual('ai'),
streamText: jest.fn(),
}));
import { streamText } from 'ai';
import {
AiChatService,
compactToolOutput,
@@ -689,6 +698,7 @@ describe('AiChatService.resolveOpenPageContext (#159 current-page validation)',
id: string;
title: string;
updatedAt: Date;
selection: unknown;
} | null>;
it('returns null when no page is open (no id)', async () => {
@@ -734,8 +744,14 @@ describe('AiChatService.resolveOpenPageContext (#159 current-page validation)',
});
// The client claims it is on "Page A" but the id points at page B.
const result = await call(svc, { id: 'p-1', title: 'Page A' });
// updatedAt (#274 page-change fast path) is carried through from the DB row.
expect(result).toEqual({ id: 'p-1', title: 'Real Title B', updatedAt });
// updatedAt (#274 page-change fast path) is carried through from the DB row;
// selection is null when the client sent none (#388).
expect(result).toEqual({
id: 'p-1',
title: 'Real Title B',
updatedAt,
selection: null,
});
});
it('coerces a null DB title to an empty string', async () => {
@@ -748,8 +764,55 @@ describe('AiChatService.resolveOpenPageContext (#159 current-page validation)',
id: 'p-1',
title: '',
updatedAt,
selection: null,
});
});
// #388: the selection rides ONLY on a successful page resolve, and is
// sanitized on the way through.
it('attaches the SANITIZED selection on a successful resolve', async () => {
const updatedAt = new Date('2026-07-02T10:00:00Z');
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-1', title: 'Doc', updatedAt },
canView: true,
});
const result = await call(svc, {
id: 'p-1',
selection: {
text: 'fix this',
blockIds: ['b1', 123, 'y'.repeat(65)], // garbage stripped by sanitize
before: 'please ',
},
});
expect(result).toEqual({
id: 'p-1',
title: 'Doc',
updatedAt,
selection: { text: 'fix this', blockIds: ['b1'], before: 'please ' },
});
});
it('drops a blank/garbage selection to null on a successful resolve', async () => {
const updatedAt = new Date('2026-07-02T10:00:00Z');
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-1', title: 'Doc', updatedAt },
canView: true,
});
expect(
await call(svc, { id: 'p-1', selection: { text: ' ' } }),
).toEqual({ id: 'p-1', title: 'Doc', updatedAt, selection: null });
});
it('selection does NOT survive a foreign/inaccessible page (dies with the page)', async () => {
// Forbidden page => the WHOLE context is null, so the selection is gone too.
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-1', title: 'Restricted' },
canView: false,
});
expect(
await call(svc, { id: 'p-1', selection: { text: 'secret sel' } }),
).toBeNull();
});
});
/**
@@ -1059,3 +1122,181 @@ describe('isInterruptResume', () => {
expect(isInterruptResume(withPrev(null), true)).toBe(false);
});
});
/**
* #184 phase 1.5 the run-wrapped pipe options (unit). Drives stream() to the
* pipe call with streamText mocked, capturing the options object, and asserts:
* - flag OFF while a runId IS present -> the LEGACY option shape (no
* consumeSseStream, no generateMessageId), and the registry is never touched.
* This is the exact dormancy guarantee this PR rests on.
* - flag ON + runId -> consumeSseStream tees into the registry and
* generateMessageId returns the seeded assistant DB row id.
* - flag ON but no runHooks (runId undefined) -> legacy (the runId gate).
* - flag ON + runId -> the outer catch releases the entry via abortEntry.
*/
describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
let pipeMock: jest.Mock;
beforeEach(() => {
streamTextMock.mockReset();
pipeMock = jest.fn();
streamTextMock.mockReturnValue({
consumeStream: jest.fn(),
pipeUIMessageStreamToResponse: pipeMock,
});
// Silence the service's diagnostic logging for a clean test run.
jest.spyOn(Logger.prototype, 'log').mockImplementation(() => undefined as never);
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as never);
});
afterEach(() => jest.restoreAllMocks());
// A raw-response stub sufficient for the post-streamText wiring.
function makeRes() {
return {
raw: {
writeHead: jest.fn(),
write: jest.fn(),
once: jest.fn(),
on: jest.fn(),
flushHeaders: jest.fn(),
writableEnded: false,
destroyed: false,
},
};
}
// Wire only the deps reached on the way to the pipe call, plus a spy registry.
function makeService(opts: { resumable: boolean }) {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const aiChatMessageRepo = {
// Both the user insert and the assistant seed return the same row id.
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
const mcpClients = {
toolsFor: jest.fn(async () => ({
tools: {},
clients: [],
outcomes: [],
instructions: [],
})),
};
const streamRegistry = {
open: jest.fn(),
bind: jest.fn(),
abortEntry: jest.fn(),
};
const svc = new AiChatService(
{} as never, // ai (model is injected)
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
aiSettings as never,
tools as never,
mcpClients as never,
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo (openPage undefined -> never touched)
{} as never, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatResumableStreamEnabled: () => opts.resumable,
} as never,
streamRegistry as never,
);
return { svc, streamRegistry };
}
const body = {
chatId: 'chat-1',
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
};
const makeRunHooks = () => ({
begin: jest.fn(async () => ({
runId: 'run-1',
signal: new AbortController().signal,
})),
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled: jest.fn(),
});
async function drive(svc: AiChatService, hooks: unknown): Promise<void> {
await svc.stream({
user: { id: 'u1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 's1',
body: body as never,
res: makeRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: hooks as never,
});
}
it('flag OFF + runId present: LEGACY option shape (no consumeSseStream / generateMessageId); registry untouched', async () => {
const { svc, streamRegistry } = makeService({ resumable: false });
await drive(svc, makeRunHooks());
expect(pipeMock).toHaveBeenCalledTimes(1);
const options = pipeMock.mock.calls[0][1];
// The dormancy guarantee: a live run with the flag off tees NOTHING and does
// not stamp a message id — byte-for-byte the pre-1.5 wire.
expect(options.consumeSseStream).toBeUndefined();
expect(options.generateMessageId).toBeUndefined();
expect(streamRegistry.bind).not.toHaveBeenCalled();
expect(streamRegistry.abortEntry).not.toHaveBeenCalled();
});
it('flag ON + runId: consumeSseStream tees into the registry; generateMessageId returns the seeded row id', async () => {
const { svc, streamRegistry } = makeService({ resumable: true });
await drive(svc, makeRunHooks());
const options = pipeMock.mock.calls[0][1];
expect(typeof options.consumeSseStream).toBe('function');
expect(typeof options.generateMessageId).toBe('function');
// generateMessageId stamps the seeded assistant DB row id.
expect(options.generateMessageId()).toBe('msg-1');
// consumeSseStream binds the tee: (chatId, runId, assistantId, stream).
const fakeStream = {} as ReadableStream<string>;
options.consumeSseStream({ stream: fakeStream });
expect(streamRegistry.bind).toHaveBeenCalledWith(
'chat-1',
'run-1',
'msg-1',
fakeStream,
);
});
it('flag ON but NO runHooks (runId undefined): pipe options stay legacy (the runId gate)', async () => {
const { svc, streamRegistry } = makeService({ resumable: true });
await drive(svc, undefined);
const options = pipeMock.mock.calls[0][1];
expect(options.consumeSseStream).toBeUndefined();
expect(options.generateMessageId).toBeUndefined();
expect(streamRegistry.bind).not.toHaveBeenCalled();
});
it('flag ON + runId: the outer catch calls abortEntry when the stream throws', async () => {
const { svc, streamRegistry } = makeService({ resumable: true });
streamTextMock.mockImplementation(() => {
throw new Error('boom');
});
await expect(drive(svc, makeRunHooks())).rejects.toThrow('boom');
expect(streamRegistry.abortEntry).toHaveBeenCalledWith('chat-1', 'run-1');
});
});
@@ -32,6 +32,7 @@ import {
import { AiChatToolsService } from './tools/ai-chat-tools.service';
import { McpClientsService } from './external-mcp/mcp-clients.service';
import { EnvironmentService } from '../../integrations/environment/environment.service';
import { AiChatStreamRegistryService } from './ai-chat-stream-registry.service';
import { buildSystemPrompt } from './ai-chat.prompt';
import {
CORE_TOOL_KEYS,
@@ -42,6 +43,10 @@ import {
} from './tools/tool-tiers';
import { RunAlreadyActiveError } from './ai-chat-run.service';
import { computePageChange } from './page-change/page-change.util';
import {
sanitizeSelection,
type SelectionContext,
} from './tools/current-page.util';
import { roleModelOverride } from './roles/role-model-config';
import {
startSseHeartbeat,
@@ -188,7 +193,12 @@ export interface AiChatStreamBody {
// page" refers to; the page itself is never fetched server-side here. The id
// is attacker-controllable but harmless: the agent reads/writes via its
// CASL-enforced page tools, which 403 on a page the user cannot access.
openPage?: { id?: string; title?: string } | null;
//
// `selection` is the user's editor selection snapshotted client-side at send
// time (#388). It is CLIENT-controlled and UNTRUSTED — a loose `unknown` here
// (the body is parsed off req.body without a DTO) that is type-checked and
// capped by `sanitizeSelection` before it is ever surfaced to the model.
openPage?: { id?: string; title?: string; selection?: unknown } | null;
// Set by the client "send now" action (#198): this turn immediately follows a
// user interruption of the previous turn. A hint only — the server re-confirms
// it against persisted history (`isInterruptResume`) before injecting the
@@ -276,6 +286,10 @@ export class AiChatService implements OnModuleInit {
// Reads the AI_CHAT_DEFERRED_TOOLS toggle (#332). Injected last so existing
// positional constructor callers (tests) only append one stub.
private readonly environment: EnvironmentService,
// #184 phase 1.5 run-stream registry. OPTIONAL so existing positional
// constructions (int-specs) compile unchanged; Nest always injects the real
// provider in production. Only ever touched on the run-wrapped + flag-on path.
private readonly streamRegistry?: AiChatStreamRegistryService,
) {}
/**
@@ -360,10 +374,18 @@ export class AiChatService implements OnModuleInit {
* page, or any non-Forbidden access-check fault, returns null.
*/
private async resolveOpenPageContext(
openPage: { id?: string; title?: string } | null | undefined,
openPage:
| { id?: string; title?: string; selection?: unknown }
| null
| undefined,
workspace: Workspace,
user: User,
): Promise<{ id: string; title: string; updatedAt: Date } | null> {
): Promise<{
id: string;
title: string;
updatedAt: Date;
selection: SelectionContext | null;
} | null> {
const candidatePageId = openPage?.id;
if (!candidatePageId) return null;
const page = await this.pageRepo.findById(candidatePageId);
@@ -385,7 +407,18 @@ export class AiChatService implements OnModuleInit {
// updatedAt is the page's last-modified instant, used by the #274 per-turn
// page-change detection as a cheap fast path (unchanged instant => skip the
// render + diff). The system-prompt / tool consumers ignore the extra field.
return { id: page.id, title: page.title ?? '', updatedAt: page.updatedAt };
//
// The sanitized editor selection (#388) is attached ONLY here, on a
// successful page resolve: the fail-closed branches above return null for the
// WHOLE context, so a selection can never outlive a foreign/missing/deleted
// page (decision 3). Downstream consumers that don't care (detectPageChange,
// snapshotOpenPage) ignore the extra field, same as updatedAt.
return {
id: page.id,
title: page.title ?? '',
updatedAt: page.updatedAt,
selection: sanitizeSelection(openPage?.selection),
};
}
/**
@@ -1174,6 +1207,35 @@ export class AiChatService implements OnModuleInit {
// as the cumulative authoritative usage so the client never jumps DOWN.
let cumulativeStepUsage: ChatStreamUsage | undefined;
result.pipeUIMessageStreamToResponse(res.raw, {
// #184 phase 1.5: run-wrapped mode only — the legacy path (flag off) stays
// byte-for-byte identical, including the absence of start.messageId. Both
// fields are gated on `runId` (present only for a durable run) AND the
// AI_CHAT_RESUMABLE_STREAM flag; the seed `assistantId` is unconditional,
// so gating on `assistantId` alone would change the legacy wire.
...(runId && this.environment?.isAiChatResumableStreamEnabled?.()
? {
// Tee the SSE frames into the run-stream registry so late tabs can
// attach (replay + live tail).
consumeSseStream: ({
stream,
}: {
stream: ReadableStream<string>;
}) =>
this.streamRegistry?.bind(
chatId,
runId!,
assistantId,
stream,
),
// Stamp the persisted assistant row's DB id onto the streamed
// message so every tab renders the SAME id as the DB row (id-based
// reconciliation). Seeding is best-effort: when it failed, let the
// client generate the id.
...(assistantId
? { generateMessageId: () => assistantId }
: {}),
}
: {}),
headers: { 'X-Accel-Buffering': 'no' },
// Surface the authoritative chatId on the streamed assistant UI message so
// the client adopts the REAL id of the row we created, instead of guessing
@@ -1239,6 +1301,12 @@ export class AiChatService implements OnModuleInit {
// finalizeRun (onSettled) is idempotent — a settle here and a settle from a
// streamText callback collapse to a single terminal write.
if (runId) {
// #184 phase 1.5: a failure here means the tee `done` will never arrive,
// so release the registry entry's subscribers explicitly — otherwise an
// attached tab hangs forever. Same flag gate as the tee wiring above.
if (this.environment?.isAiChatResumableStreamEnabled?.()) {
this.streamRegistry?.abortEntry(chatId, runId);
}
await runHooks?.onSettled?.(
runId,
'error',
@@ -610,6 +610,63 @@ describe('AiAgentRolesService guards', () => {
expect(repo.insert.mock.calls[0][0].name).toBe('Researcher (2)');
});
it('createdRoles lists the installed role (no renamedTo when not renamed)', async () => {
const { service } = makeImportService({});
const res = await service.importFromCatalog('ws-1', 'u1', dto());
expect(res.createdRoles).toEqual([
{ slug: 'researcher', name: 'Researcher' },
]);
expect(res.skippedRoles).toEqual([]);
});
it('createdRoles carries renamedTo on a rename', async () => {
const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
const { service } = makeImportService({ existing });
const res = await service.importFromCatalog(
'ws-1',
'u1',
dto({ conflict: 'rename' }),
);
expect(res.createdRoles).toEqual([
{ slug: 'researcher', name: 'Researcher', renamedTo: 'Researcher (2)' },
]);
expect(res.skippedRoles).toEqual([]);
});
it('skippedRoles: already-installed slug carries reason "already-installed"', async () => {
const existing = [
makeRow({
id: 'r-existing',
name: 'Old researcher',
source: { slug: 'researcher', language: 'en', version: 1 } as never,
}),
];
const { service } = makeImportService({ existing });
const res = await service.importFromCatalog('ws-1', 'u1', dto());
expect(res.skippedRoles).toEqual([
{
slug: 'researcher',
name: 'Researcher',
reason: 'already-installed',
},
]);
expect(res.createdRoles).toEqual([]);
});
it('skippedRoles: a name collision under conflict:skip carries reason "name-conflict"', async () => {
const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
const { service } = makeImportService({ existing });
const res = await service.importFromCatalog(
'ws-1',
'u1',
dto({ conflict: 'skip' }),
);
expect(res.skippedRoles).toEqual([
{ slug: 'researcher', name: 'Researcher', reason: 'name-conflict' },
]);
expect(res.createdRoles).toEqual([]);
});
it('dto.slugs filters; an unknown slug becomes an error entry', async () => {
const { service, repo } = makeImportService({
bundleRoles: [catalogRole()],
@@ -677,6 +734,15 @@ describe('AiAgentRolesService guards', () => {
// 'a' converged on the concurrent install (skip); 'b' imported; no errors.
expect(res).toMatchObject({ created: 1, skipped: 1, renamed: 0 });
expect(res.errors).toEqual([]);
// The per-role list records 'a' as an already-installed skip (the UI reads
// skippedRoles, not the counter, to render its plaque — assert the array,
// not just the count).
expect(res.skippedRoles).toContainEqual({
slug: 'a',
name: 'A',
reason: 'already-installed',
});
expect(res.createdRoles.map((r) => r.slug)).toEqual(['b']);
// Both inserts were attempted (the batch did not abort on the 23505).
expect(repo.insert).toHaveBeenCalledTimes(2);
});
@@ -305,6 +305,16 @@ export class AiAgentRolesService {
skipped: number;
renamed: number;
errors: { slug: string; message: string }[];
// Per-role lists alongside the counters (kept for back-compat). The redesigned
// catalog UI needs the actual roles — which were created (and any rename) and
// which were skipped and why — to render an inline result plaque with the
// conflicting role's name and a "Rename & install" affordance.
createdRoles: { slug: string; name: string; renamedTo?: string }[];
skippedRoles: {
slug: string;
name: string;
reason: 'name-conflict' | 'already-installed';
}[];
}> {
const { file, versions } = await this.loadBundleById(
dto.bundleId,
@@ -312,6 +322,13 @@ export class AiAgentRolesService {
);
const errors: { slug: string; message: string }[] = [];
const createdRoles: { slug: string; name: string; renamedTo?: string }[] =
[];
const skippedRoles: {
slug: string;
name: string;
reason: 'name-conflict' | 'already-installed';
}[] = [];
// Resolve the selected catalog roles (honor dto.slugs; flag unknown ones).
let selected = file.roles;
@@ -351,16 +368,27 @@ export class AiAgentRolesService {
// Already installed from the catalog in THIS language => skip (use
// update-from-catalog). A different language of the same slug still imports.
const installKey = `${role.slug}:${dto.language}`;
const originalName = role.name.trim();
if (installedKeys.has(installKey)) {
skipped++;
skippedRoles.push({
slug: role.slug,
name: originalName,
reason: 'already-installed',
});
continue;
}
let name = role.name.trim();
let name = originalName;
let didRename = false;
if (takenNames.has(name.toLowerCase())) {
if (dto.conflict === 'skip') {
skipped++;
skippedRoles.push({
slug: role.slug,
name: originalName,
reason: 'name-conflict',
});
continue;
}
// conflict === 'rename': find a free " (N)" suffix.
@@ -380,6 +408,11 @@ export class AiAgentRolesService {
});
created++;
if (didRename) renamed++;
createdRoles.push({
slug: role.slug,
name: originalName,
...(didRename ? { renamedTo: name } : {}),
});
takenNames.add(name.toLowerCase());
installedKeys.add(installKey);
} catch (err) {
@@ -391,6 +424,11 @@ export class AiAgentRolesService {
// skipped (already installed) and continue; do NOT abort or error.
if (isSourceUniqueViolation(err)) {
skipped++;
skippedRoles.push({
slug: role.slug,
name: originalName,
reason: 'already-installed',
});
installedKeys.add(installKey);
continue;
}
@@ -407,7 +445,7 @@ export class AiAgentRolesService {
}
}
return { created, skipped, renamed, errors };
return { created, skipped, renamed, errors, createdRoles, skippedRoles };
}
/**
@@ -539,3 +539,181 @@ describe('AiChatToolsService model-friendly input validation (#190)', () => {
expect(result.error?.message).toContain('parameter "pageId": missing (required)');
});
});
/**
* #294 F1 the contract-parity test introspects only the ADVERTISED schema keys
* (buildShape), not the execute bodies. Most execs are unchanged pass-throughs,
* but two wirings actually CHANGED in the migration and are otherwise untested:
* - movePage now forwards the newly-added optional `position` field to the
* client (client.movePage(pageId, parentPageId, position));
* - the table trio unified its `tableRef` param to `table` and must forward it
* positionally. A field destructured under the wrong name would silently pass
* `undefined` to the client (execute is `any`-cast, so tsc won't catch it).
*/
describe('AiChatToolsService #294 changed execute wirings', () => {
const calls: Record<string, unknown[][]> = {
movePage: [],
tableInsertRow: [],
tableDeleteRow: [],
tableUpdateCell: [],
};
const fakeClient: Partial<DocmostClientLike> = {
movePage: (...args: unknown[]) => {
calls.movePage.push(args);
return Promise.resolve({ success: true });
},
tableInsertRow: (...args: unknown[]) => {
calls.tableInsertRow.push(args);
return Promise.resolve({ ok: true });
},
tableDeleteRow: (...args: unknown[]) => {
calls.tableDeleteRow.push(args);
return Promise.resolve({ ok: true });
},
tableUpdateCell: (...args: unknown[]) => {
calls.tableUpdateCell.push(args);
return Promise.resolve({ ok: true });
},
};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
generateCollabToken: jest.fn().mockResolvedValue('collab-token'),
};
let service: AiChatToolsService;
beforeEach(() => {
for (const k of Object.keys(calls)) calls[k].length = 0;
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue(
mockLoaded(function () {
return fakeClient as DocmostClientLike;
} as unknown as loader.DocmostClientCtor),
);
service = new AiChatToolsService(
tokenServiceStub as never,
{} as never,
{} as never,
{} as never,
{} as never,
{
asSink: () => ({ put: jest.fn(), has: jest.fn(), evict: jest.fn() }),
} as never,
);
});
afterEach(() => jest.restoreAllMocks());
const buildTools = () =>
service.forUser(
{ id: 'user-1', email: 'u@example.com', workspaceId: 'ws-1' } as never,
'session-1',
'ws-1',
'chat-1',
);
it('movePage forwards the optional position to the client', async () => {
const tools = await buildTools();
await tools.movePage.execute(
{ pageId: 'p1', parentPageId: 'parent1', position: 'a5' } as never,
{} as never,
);
expect(calls.movePage).toEqual([['p1', 'parent1', 'a5']]);
});
it('movePage passes undefined position and null parent when omitted (unchanged behavior)', async () => {
const tools = await buildTools();
await tools.movePage.execute({ pageId: 'p2' } as never, {} as never);
expect(calls.movePage).toEqual([['p2', null, undefined]]);
});
it('tableInsertRow forwards the unified `table` param positionally', async () => {
const tools = await buildTools();
await tools.tableInsertRow.execute(
{ pageId: 'p1', table: '#0', cells: ['a', 'b'], index: 2 } as never,
{} as never,
);
expect(calls.tableInsertRow).toEqual([['p1', '#0', ['a', 'b'], 2]]);
});
it('tableDeleteRow forwards `table` positionally', async () => {
const tools = await buildTools();
await tools.tableDeleteRow.execute(
{ pageId: 'p1', table: '#0', index: 1 } as never,
{} as never,
);
expect(calls.tableDeleteRow).toEqual([['p1', '#0', 1]]);
});
it('tableUpdateCell forwards `table` positionally', async () => {
const tools = await buildTools();
await tools.tableUpdateCell.execute(
{ pageId: 'p1', table: '#0', row: 1, col: 2, text: 'x' } as never,
{} as never,
);
expect(calls.tableUpdateCell).toEqual([['p1', '#0', 1, 2, 'x']]);
});
});
/**
* getCurrentPage selection contract (#388): the tool surfaces the selection that
* was sanitized + nested onto the resolved open-page context (last forUser arg).
* No page => selection is null. The tool never fetches or verifies anything it
* just projects the resolved context.
*/
describe('AiChatToolsService getCurrentPage selection (#388)', () => {
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
generateCollabToken: jest.fn().mockResolvedValue('collab-token'),
};
let service: AiChatToolsService;
beforeEach(() => {
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue(
mockLoaded(function () {
return {} as DocmostClientLike;
} as unknown as loader.DocmostClientCtor),
);
service = new AiChatToolsService(
tokenServiceStub as never,
{} as never,
{} as never,
{} as never,
{} as never,
{
asSink: () => ({ put: jest.fn(), has: jest.fn(), evict: jest.fn() }),
} as never,
);
});
afterEach(() => jest.restoreAllMocks());
const buildTools = (openedPage: unknown) =>
service.forUser(
{ id: 'user-1', email: 'u@example.com', workspaceId: 'ws-1' } as never,
'session-1',
'ws-1',
'chat-1',
openedPage as never,
);
it('returns the nested selection from the resolved context', async () => {
const selection = { text: 'fix this', blockIds: ['b1'], before: 'a ' };
const tools = await buildTools({ id: 'p1', title: 'Doc', selection });
expect(await tools.getCurrentPage.execute({} as never, {} as never)).toEqual(
{ page: { id: 'p1', title: 'Doc' }, selection },
);
});
it('returns selection: null when the context has no selection', async () => {
const tools = await buildTools({ id: 'p1', title: 'Doc' });
expect(await tools.getCurrentPage.execute({} as never, {} as never)).toEqual(
{ page: { id: 'p1', title: 'Doc' }, selection: null },
);
});
it('returns { page: null, selection: null } when no page is open', async () => {
const tools = await buildTools(null);
expect(await tools.getCurrentPage.execute({} as never, {} as never)).toEqual(
{ page: null, selection: null },
);
});
});
@@ -13,7 +13,10 @@ import {
type DocmostClientLike,
type SharedToolSpec,
} from './docmost-client.loader';
import { resolveCurrentPageResult } from './current-page.util';
import {
resolveCurrentPageResult,
type SelectionContext,
} from './current-page.util';
import { parseNodeArg } from './parse-node-arg';
import { modelFriendlyInput } from './model-friendly-input';
import { SandboxStore } from '../../../integrations/sandbox/sandbox.store';
@@ -153,8 +156,13 @@ export class AiChatToolsService {
// The page the user currently has open (from the request context), exposed
// to the model via getCurrentPage. Optional and last so existing callers
// keep compiling. Kept proxy-robust: the model can CALL for the current
// page instead of relying on it surviving in the system prompt text.
openedPage?: { id?: string; title?: string } | null,
// page instead of relying on it surviving in the system prompt text. The
// `selection` (#388) is already sanitized + nested by resolveOpenPageContext.
openedPage?: {
id?: string;
title?: string;
selection?: SelectionContext | null;
} | null,
): Promise<Record<string, Tool>> {
// Build the per-user loopback client (carrying the access + collab
// provenance tokens) and load the shared tool-spec registry. Client
@@ -309,57 +317,40 @@ export class AiChatToolsService {
getCurrentPage: tool({
description:
'Return the page the user is currently viewing — i.e. what "this page", ' +
'"the current page", or "here" refers to. Returns the page id and title, ' +
'or null if the user is not currently on a page. Call this first whenever ' +
'the user refers to the current page without giving an explicit id.',
'"the current page", or "here" refers to — plus the text the user ' +
'currently has SELECTED on that page (what "this", "here", "the selected ' +
'fragment" refers to), or selection: null when nothing is selected. The ' +
'selection is a client-side snapshot taken when the user sent the message ' +
'and includes the ids of the blocks it covers plus surrounding context; ' +
'it is NOT verified server-side — locate it in the page (searchInPage / ' +
'getNode) before editing. Returns page: null if the user is not currently ' +
'on a page. Call this first whenever the user refers to the current page ' +
'or a selected fragment without giving an explicit id.',
inputSchema: modelFriendlyInput({}),
execute: async () => resolveCurrentPageResult(openedPage),
}),
getPage: tool({
description:
'Fetch a single page as Markdown by its page id. Returns the page ' +
'title and its Markdown content. Inline <span data-comment-id> tags ' +
'in the markdown are comment highlight anchors (also present for ' +
'RESOLVED threads) — treat them as markup, not page text.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id (or slugId) of the page.'),
}),
execute: async ({ pageId }) => {
// getPage(pageId) -> { data: filterPage(page, markdown), success }.
const result = await client.getPage(pageId);
const data = (result?.data ?? {}) as {
title?: string;
content?: string;
};
return {
title: data.title ?? '',
markdown: typeof data.content === 'string' ? data.content : '',
};
},
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The execute body keeps this layer's { title, markdown } projection.
getPage: sharedTool(sharedToolSpecs.getPage, async ({ pageId }) => {
// getPage(pageId) -> { data: filterPage(page, markdown), success }.
const result = await client.getPage(pageId);
const data = (result?.data ?? {}) as {
title?: string;
content?: string;
};
return {
title: data.title ?? '',
markdown: typeof data.content === 'string' ? data.content : '',
};
}),
// --- WRITE tools (all reversible — history/trash; §6.5 / D3) ---
createPage: tool({
description:
'Create a new page with a Markdown body in a space, optionally under ' +
'a parent page. Returns the new page id and title. Reversible: a page ' +
'can be moved to trash later.',
inputSchema: modelFriendlyInput({
title: z.string().describe('The title of the new page.'),
content: z
.string()
.describe('The page body as Markdown (may be empty).'),
spaceId: z
.string()
.describe('The id of the space to create the page in.'),
parentPageId: z
.string()
.optional()
.describe('Optional parent page id to nest the new page under.'),
}),
execute: async ({ title, content, spaceId, parentPageId }) => {
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
createPage: sharedTool(
sharedToolSpecs.createPage,
async ({ title, content, spaceId, parentPageId }) => {
// createPage(title, content, spaceId, parentPageId?) ->
// { data: filterPage(page, markdown), success }.
const result = await client.createPage(
@@ -375,7 +366,7 @@ export class AiChatToolsService {
};
return { id: data.id ?? data.slugId, title: data.title ?? title };
},
}),
),
updatePageContent: tool({
description:
@@ -399,115 +390,46 @@ export class AiChatToolsService {
},
}),
renamePage: tool({
description:
"Rename a page (change its title only; the body is untouched). " +
'Reversible: rename back at any time.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to rename.'),
title: z.string().describe('The new title.'),
}),
execute: async ({ pageId, title }) => {
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
renamePage: sharedTool(
sharedToolSpecs.renamePage,
async ({ pageId, title }) => {
// renamePage(pageId, title) -> { success, pageId, title }.
await client.renamePage(pageId, title);
return { pageId, title };
},
}),
),
movePage: tool({
description:
'Move a page under a new parent page, or to the space root when no ' +
'parent is given. Reversible: move it back at any time.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to move.'),
parentPageId: z
.string()
.nullable()
.optional()
.describe(
'Target parent page id. Null/omitted moves the page to the ' +
'space root.',
),
}),
execute: async ({ pageId, parentPageId }) => {
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The shared schema adds the optional `position` field this layer lacked
// before; the execute now forwards it (the client already accepted it).
movePage: sharedTool(
sharedToolSpecs.movePage,
async ({ pageId, parentPageId, position }) => {
// movePage(pageId, parentPageId, position?) -> raw move response.
await client.movePage(pageId, parentPageId ?? null);
await client.movePage(pageId, parentPageId ?? null, position);
return { pageId, parentPageId: parentPageId ?? null, moved: true };
},
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// GUARDRAIL (§14 H4) preserved: the shared schema exposes ONLY pageId, so
// permanentlyDelete/forceDelete are never part of the input and can never
// be forwarded — the agent physically cannot permanently delete a page.
deletePage: sharedTool(sharedToolSpecs.deletePage, async ({ pageId }) => {
// deletePage(pageId) hits POST /pages/delete with { pageId } only,
// which is the soft-delete (trash) path on the server.
await client.deletePage(pageId);
return { pageId, trashed: true };
}),
deletePage: tool({
description:
'Move a page to the trash (SOFT delete only — fully reversible; the ' +
'page can be restored from trash). This NEVER permanently deletes.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to move to trash.'),
}),
// GUARDRAIL (§14 H4): the only field ever passed to the client is
// pageId. permanentlyDelete/forceDelete are not part of the schema and
// are never forwarded, so the agent physically cannot permanently
// delete a page through this tool.
execute: async ({ pageId }) => {
// deletePage(pageId) hits POST /pages/delete with { pageId } only,
// which is the soft-delete (trash) path on the server.
await client.deletePage(pageId);
return { pageId, trashed: true };
},
}),
// INTENTIONAL per-transport divergence (not shared): the description is
// tuned for the in-app agent (e.g. "retry with a corrected EXACT selection"
// and "Reversible via the comment UI"); the standalone MCP `create_comment`
// keeps its own wording. Kept per-layer.
createComment: tool({
description:
'Add an INLINE comment to a page, or reply to an existing top-level ' +
'comment (one level only — the backend rejects replies to replies). ' +
'The comment is anchored inline to the given exact `selection` text ' +
'(which gets highlighted); page-level comments are NOT supported. A ' +
"new top-level comment REQUIRES a `selection`. Replies inherit the " +
"parent's anchor and take no selection. If the call fails with a " +
'"selection not found" error, retry with a corrected EXACT selection ' +
'copied verbatim from a single paragraph/block. You may also attach a ' +
'`suggestedText` proposing a replacement for the `selection` (a human ' +
'applies it from the UI); when set, the `selection` must occur exactly ' +
'once in the page. Reversible via the comment UI.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to comment on.'),
content: z.string().describe('The comment body as Markdown.'),
selection: z
.string()
.min(1)
.max(250)
.optional()
.describe(
'EXACT contiguous text from a SINGLE paragraph/block to anchor ' +
'(highlight) the comment on (<=250 chars, avoid spanning across ' +
'formatting boundaries). Required for a new top-level comment; ' +
'omit only when replying via parentCommentId.',
),
parentCommentId: z
.string()
.optional()
.describe(
'Optional id of a TOP-LEVEL comment to reply to (one level ' +
'of replies only).',
),
suggestedText: z
.string()
.min(1)
.max(2000)
.optional()
.describe(
'Optional proposed replacement (PLAIN TEXT) for the `selection`, ' +
'applied by a human via the UI (never auto-applied). REQUIRES a ' +
'`selection`; NOT allowed on a reply. When set, the `selection` ' +
'must be UNIQUE in the page — expand it with surrounding context ' +
'(still <=250 chars) if it occurs more than once, or the call is ' +
'refused.',
),
}),
execute: async ({
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// This layer keeps only its own execute-side guards (require a selection
// for a top-level comment; reject suggestedText on a reply / without a
// selection) — the schema+description are shared.
createComment: sharedTool(
sharedToolSpecs.createComment,
async ({
pageId,
content,
selection,
@@ -548,26 +470,17 @@ export class AiChatToolsService {
const data = (result?.data ?? {}) as { id?: string };
return { commentId: data.id, pageId };
},
}),
),
resolveComment: tool({
description:
'Resolve or reopen a top-level comment thread (reversible — toggle ' +
'the resolved flag). Only top-level comments can be resolved.',
inputSchema: modelFriendlyInput({
commentId: z
.string()
.describe('The id of the top-level comment to resolve/reopen.'),
resolved: z
.boolean()
.describe('true to resolve the thread, false to reopen it.'),
}),
execute: async ({ commentId, resolved }) => {
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
resolveComment: sharedTool(
sharedToolSpecs.resolveComment,
async ({ commentId, resolved }) => {
// resolveComment(commentId, resolved) -> { success, commentId, resolved }.
await client.resolveComment(commentId, resolved);
return { commentId, resolved };
},
}),
),
// --- READ tools (added) ---
@@ -585,33 +498,12 @@ export class AiChatToolsService {
// hierarchy mode but is worded for the in-app agent; the standalone MCP
// `list_pages` carries its own wording. Kept per-layer so each side tunes
// its own guidance.
listPages: tool({
description:
'List the most recent pages, optionally scoped to a single space. ' +
'Returns a bounded list (default 50, max 100). Pass tree:true (with ' +
"spaceId) to instead get the space's full page hierarchy as a nested tree.",
inputSchema: modelFriendlyInput({
spaceId: z
.string()
.optional()
.describe('Optional space id to scope the listing to.'),
limit: z
.number()
.int()
.min(1)
.max(100)
.optional()
.describe('Maximum number of pages (1-100).'),
tree: z
.boolean()
.optional()
.describe(
'When true, return the full page hierarchy of the given space as a nested tree (children arrays) instead of the recent-pages flat list. Requires spaceId; ignores limit.',
),
}),
execute: async ({ spaceId, limit, tree }) =>
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
listPages: sharedTool(
sharedToolSpecs.listPages,
async ({ spaceId, limit, tree }) =>
await client.listPages(spaceId, limit, tree),
}),
),
listSidebarPages: tool({
description:
@@ -656,41 +548,34 @@ export class AiChatToolsService {
}),
),
// NOT shared (kept inline): the MCP tool name `table_get` is noun-first
// while this key is `getTable` (verb-first), breaking the
// snake_case(inAppKey) convention the shared registry enforces. Its
// reference parameter is still named `table` (was `tableRef`) so it matches
// the migrated table row/cell tools below.
getTable: tool({
description:
'Read a table as a matrix of cell texts (plus a parallel cellIds ' +
'matrix so cells can be addressed for rich edits).',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
tableRef: z
table: z
.string()
.describe(
'"#<index>" from getOutline, or a block id of any node inside ' +
'the table.',
'"#<index>" from the page outline, or a block id of any node ' +
'inside the table.',
),
}),
execute: async ({ pageId, tableRef }) =>
await client.getTable(pageId, tableRef),
execute: async ({ pageId, table }) =>
await client.getTable(pageId, table),
}),
listComments: tool({
description:
'List comments on a page in one call. By DEFAULT only ACTIVE ' +
'threads are returned; resolved threads (a resolved top-level ' +
'comment and all its replies) are hidden and their count reported ' +
'as `resolvedThreadsHidden` so you can re-query with ' +
'`includeResolved: true` to see everything. Returns ' +
'`{ items, resolvedThreadsHidden }`. Content is returned as Markdown.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
includeResolved: z
.boolean()
.optional()
.describe('default only active threads; true — include resolved'),
}),
execute: async ({ pageId, includeResolved }) =>
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
listComments: sharedTool(
sharedToolSpecs.listComments,
async ({ pageId, includeResolved }) =>
await client.listComments(pageId, includeResolved),
}),
),
getComment: tool({
description: 'Fetch a single comment by id (content as Markdown).',
@@ -700,26 +585,12 @@ export class AiChatToolsService {
execute: async ({ commentId }) => await client.getComment(commentId),
}),
checkNewComments: tool({
description:
'Find new comments across a space (optionally scoped to a subtree) ' +
'created after a given timestamp.',
inputSchema: modelFriendlyInput({
spaceId: z.string().describe('The id of the space to scan.'),
since: z
.string()
.describe('An ISO-8601 timestamp; only comments created after it.'),
parentPageId: z
.string()
.optional()
.describe(
'Optional page id to scope the scan to that page and its ' +
'descendants.',
),
}),
execute: async ({ spaceId, since, parentPageId }) =>
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
checkNewComments: sharedTool(
sharedToolSpecs.checkNewComments,
async ({ spaceId, since, parentPageId }) =>
await client.checkNewComments(spaceId, since, parentPageId),
}),
),
listShares: sharedTool(
sharedToolSpecs.listShares,
@@ -749,19 +620,14 @@ export class AiChatToolsService {
await client.diffPageVersions(pageId, from, to),
),
exportPageMarkdown: tool({
description:
'Export a page to a single self-contained Docmost-flavoured ' +
'Markdown file (meta + body + comment threads). Lossless round-trip ' +
'with importPageMarkdown.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to export.'),
}),
execute: async ({ pageId }) => {
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
exportPageMarkdown: sharedTool(
sharedToolSpecs.exportPageMarkdown,
async ({ pageId }) => {
const markdown = await client.exportPageMarkdown(pageId);
return { markdown };
},
}),
),
// --- WRITE tools (added; reversible via page history/trash) ---
@@ -811,28 +677,12 @@ export class AiChatToolsService {
async ({ pageId, nodeId }) => await client.deleteNode(pageId, nodeId),
),
updatePageJson: tool({
description:
"Replace a page's body with a full ProseMirror document — a full " +
'overwrite — and/or update its title. Minimal example content: ' +
'{"type":"doc","content":[{"type":"paragraph","content":' +
'[{"type":"text","text":"Hi"}]}]}. The content arg may be a JSON ' +
'object or a JSON string (both accepted). Omit content for a ' +
'title-only update. Reversible: the previous version is kept in page ' +
'history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to update.'),
content: z
.any()
.optional()
.describe(
'Full ProseMirror doc {"type":"doc","content":[...]} (JSON ' +
'object or JSON string); omit for a title-only update.',
),
title: z.string().optional().describe('Optional new title.'),
}),
execute: async ({ pageId, content, title }) => {
// Parity with the standalone MCP server (index.ts update_page_json):
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The execute body keeps this layer's content normalization (parity with
// the standalone MCP server, index.ts update_page_json).
updatePageJson: sharedTool(
sharedToolSpecs.updatePageJson,
async ({ pageId, content, title }) => {
// undefined/null pass through as undefined (title-only / no-op); any
// string is JSON.parsed (so an empty string "" throws, matching the
// MCP server); an object is passed through unchanged.
@@ -845,66 +695,29 @@ export class AiChatToolsService {
}
return await client.updatePageJson(pageId, doc, title);
},
}),
),
// NOT in the shared registry: this layer names the table argument
// `tableRef`, while the standalone MCP tool names it `table` (index.ts).
// Sharing one buildShape would rename a model-facing parameter on one
// transport, so the table row/cell tools stay per-layer by design.
tableInsertRow: tool({
description:
'Insert a row of plain-text cells into a table. Reversible via ' +
'page history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
tableRef: z
.string()
.describe('"#<index>" from getOutline, or a block id in the table.'),
cells: z.array(z.string()).describe('The cell texts for the row.'),
index: z
.number()
.int()
.optional()
.describe('0-based insert position (omit/out-of-range to append).'),
}),
execute: async ({ pageId, tableRef, cells, index }) =>
await client.tableInsertRow(pageId, tableRef, cells, index),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The table reference parameter was unified to `table` (was `tableRef`).
tableInsertRow: sharedTool(
sharedToolSpecs.tableInsertRow,
async ({ pageId, table, cells, index }) =>
await client.tableInsertRow(pageId, table, cells, index),
),
// NOT shared — same `tableRef` (here) vs `table` (MCP) parameter-name
// divergence as tableInsertRow.
tableDeleteRow: tool({
description:
'Delete a table row at a 0-based index. Reversible via page history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
tableRef: z
.string()
.describe('"#<index>" from getOutline, or a block id in the table.'),
index: z.number().int().describe('0-based row index to delete.'),
}),
execute: async ({ pageId, tableRef, index }) =>
await client.tableDeleteRow(pageId, tableRef, index),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
tableDeleteRow: sharedTool(
sharedToolSpecs.tableDeleteRow,
async ({ pageId, table, index }) =>
await client.tableDeleteRow(pageId, table, index),
),
// NOT shared — same `tableRef` (here) vs `table` (MCP) parameter-name
// divergence as tableInsertRow.
tableUpdateCell: tool({
description:
'Set the plain-text content of a table cell at [row, col] (0-based). ' +
'Reversible via page history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
tableRef: z
.string()
.describe('"#<index>" from getOutline, or a block id in the table.'),
row: z.number().int().describe('0-based row index.'),
col: z.number().int().describe('0-based column index.'),
text: z.string().describe('The new cell text.'),
}),
execute: async ({ pageId, tableRef, row, col, text }) =>
await client.tableUpdateCell(pageId, tableRef, row, col, text),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
tableUpdateCell: sharedTool(
sharedToolSpecs.tableUpdateCell,
async ({ pageId, table, row, col, text }) =>
await client.tableUpdateCell(pageId, table, row, col, text),
),
copyPageContent: sharedTool(
sharedToolSpecs.copyPageContent,
@@ -918,25 +731,14 @@ export class AiChatToolsService {
await client.importPageMarkdown(pageId, markdown),
),
// INTENTIONAL per-transport divergence (not shared): adds a security
// confirmation framing ("Only share when the user explicitly asked, since
// this exposes the page to anyone with the link") for the in-app agent; the
// standalone MCP `share_page` keeps the plain public-URL wording.
sharePage: tool({
description:
'Make a page PUBLICLY accessible and return its public URL. ' +
'Reversible via unsharePage. Only share when the user explicitly ' +
'asked, since this exposes the page to anyone with the link.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to share.'),
searchIndexing: z
.boolean()
.optional()
.describe('Allow public search engines to index it (default true).'),
}),
execute: async ({ pageId, searchIndexing }) =>
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// Both layers already carried the security-confirmation framing, so there
// was no real divergence to preserve — only wording drift.
sharePage: sharedTool(
sharedToolSpecs.sharePage,
async ({ pageId, searchIndexing }) =>
await client.sharePage(pageId, searchIndexing),
}),
),
unsharePage: sharedTool(
sharedToolSpecs.unsharePage,
@@ -1,43 +1,180 @@
import { resolveCurrentPageResult } from './current-page.util';
import {
resolveCurrentPageResult,
sanitizeSelection,
} from './current-page.util';
/**
* Unit tests for resolveCurrentPageResult (pure function). Mirrors the
* getCurrentPage tool's contract: { page: null } when no page is open (no id),
* otherwise { page: { id, title } } with title defaulting to ''.
* getCurrentPage tool's contract: { page: null, selection: null } when no page
* is open (no id), otherwise { page: { id, title }, selection } with title
* defaulting to '' and the selection passed through from the resolved context.
*/
describe('resolveCurrentPageResult', () => {
it('returns { page: null } when openedPage is undefined', () => {
expect(resolveCurrentPageResult(undefined)).toEqual({ page: null });
it('returns { page: null, selection: null } when openedPage is undefined', () => {
expect(resolveCurrentPageResult(undefined)).toEqual({
page: null,
selection: null,
});
});
it('returns { page: null } when openedPage is null', () => {
expect(resolveCurrentPageResult(null)).toEqual({ page: null });
it('returns { page: null, selection: null } when openedPage is null', () => {
expect(resolveCurrentPageResult(null)).toEqual({
page: null,
selection: null,
});
});
it('returns { page: null } when openedPage has no id', () => {
expect(resolveCurrentPageResult({})).toEqual({ page: null });
expect(resolveCurrentPageResult({ title: 'x' })).toEqual({ page: null });
it('returns { page: null, selection: null } when openedPage has no id', () => {
expect(resolveCurrentPageResult({})).toEqual({
page: null,
selection: null,
});
expect(resolveCurrentPageResult({ title: 'x' })).toEqual({
page: null,
selection: null,
});
});
it('returns { page: null } when id is an empty string', () => {
expect(resolveCurrentPageResult({ id: '' })).toEqual({ page: null });
it('returns { page: null, selection: null } when id is an empty string', () => {
expect(resolveCurrentPageResult({ id: '' })).toEqual({
page: null,
selection: null,
});
});
it('returns the page id and title when both are present', () => {
it('drops the selection when there is no page (selection dies with the page)', () => {
// Even if a selection somehow rode along without a page id, a null page
// always yields a null selection.
expect(
resolveCurrentPageResult({ selection: { text: 'orphan' } }),
).toEqual({ page: null, selection: null });
});
it('returns the page id and title with a null selection by default', () => {
expect(resolveCurrentPageResult({ id: 'p1', title: 'Hello' })).toEqual({
page: { id: 'p1', title: 'Hello' },
selection: null,
});
});
it('passes the nested selection through verbatim', () => {
const selection = {
text: 'fix this',
blockIds: ['b1'],
before: 'please ',
after: ' now',
};
expect(
resolveCurrentPageResult({ id: 'p1', title: 'Hello', selection }),
).toEqual({
page: { id: 'p1', title: 'Hello' },
selection,
});
});
it('defaults title to "" when it is missing', () => {
expect(resolveCurrentPageResult({ id: 'p1' })).toEqual({
page: { id: 'p1', title: '' },
selection: null,
});
});
it('keeps an explicit empty-string title as ""', () => {
expect(resolveCurrentPageResult({ id: 'p1', title: '' })).toEqual({
page: { id: 'p1', title: '' },
selection: null,
});
});
});
/**
* Unit tests for sanitizeSelection (#388). The selection is an attacker-
* controllable client snapshot: every field is type-checked and capped, and
* anything that is not a real selection collapses to null. It is NEVER verified
* against the page content (decision 5 a hint, not ground truth).
*/
describe('sanitizeSelection', () => {
it('accepts a well-formed payload unchanged', () => {
const raw = {
text: 'the selected fragment',
truncated: true,
blockIds: ['b1', 'b2'],
before: 'context before ',
after: ' context after',
};
expect(sanitizeSelection(raw)).toEqual(raw);
});
it('returns null for non-objects', () => {
expect(sanitizeSelection(null)).toBeNull();
expect(sanitizeSelection(undefined)).toBeNull();
expect(sanitizeSelection('text')).toBeNull();
expect(sanitizeSelection(42)).toBeNull();
expect(sanitizeSelection([])).toBeNull();
});
it('returns null when text is missing, non-string or blank-after-trim', () => {
expect(sanitizeSelection({})).toBeNull();
expect(sanitizeSelection({ text: 123 })).toBeNull();
expect(sanitizeSelection({ text: '' })).toBeNull();
expect(sanitizeSelection({ text: ' \n ' })).toBeNull();
});
it('keeps only text when the other fields are garbage', () => {
expect(
sanitizeSelection({
text: 'hello',
truncated: 'yes',
blockIds: 'nope',
before: 5,
after: {},
}),
).toEqual({ text: 'hello' });
});
it('caps text at 4000 and forces truncated', () => {
const raw = { text: 'a'.repeat(5000) };
const out = sanitizeSelection(raw)!;
expect(out.text).toHaveLength(4000);
expect(out.truncated).toBe(true);
});
it('does not set truncated for text under the cap', () => {
expect(sanitizeSelection({ text: 'short' })).toEqual({ text: 'short' });
});
it('slices blockIds to 20 and drops non-string / oversized ids', () => {
const ids = Array.from({ length: 30 }, (_, i) => `b${i}`);
const out = sanitizeSelection({
text: 'x',
blockIds: [...ids, 123, '', 'y'.repeat(65)],
})!;
// The 30 valid ids cap to the first 20; the number, empty string and the
// 65-char id are dropped before the slice.
expect(out.blockIds).toHaveLength(20);
expect(out.blockIds).toEqual(ids.slice(0, 20));
});
it('keeps a 64-char id but drops a 65-char one (boundary)', () => {
const ok = 'z'.repeat(64);
const tooLong = 'z'.repeat(65);
expect(
sanitizeSelection({ text: 'x', blockIds: [ok, tooLong] })!.blockIds,
).toEqual([ok]);
});
it('omits blockIds entirely when none survive', () => {
const out = sanitizeSelection({ text: 'x', blockIds: [123, ''] })!;
expect(out.blockIds).toBeUndefined();
});
it('caps before/after at 200 chars and drops empty ones', () => {
const out = sanitizeSelection({
text: 'x',
before: 'b'.repeat(300),
after: '',
})!;
expect(out.before).toHaveLength(200);
expect(out.after).toBeUndefined();
});
});
@@ -1,21 +1,91 @@
export interface SelectionContext {
text: string;
truncated?: boolean;
blockIds?: string[];
before?: string;
after?: string;
}
// Server-side caps for the client-reported selection. Intentionally >= the
// client caps: the client pre-trims for a small wire, but this layer re-checks
// everything because the payload is attacker-controllable.
const TEXT_CAP = 4000;
const CONTEXT_CAP = 200;
const MAX_BLOCK_IDS = 20;
const BLOCK_ID_CAP = 64;
// Sanitize the client-reported selection: type-check every field, cap sizes
// (text 4000, before/after 200, blockIds 20 x 64 chars), drop garbage to null.
// The selection is a CLIENT-side snapshot — never verified against the page
// content (#159 lesson: treat as a hint, not ground truth). The agent is told
// (getCurrentPage's description) to localize it before editing.
export function sanitizeSelection(raw: unknown): SelectionContext | null {
if (!raw || typeof raw !== 'object') return null;
const r = raw as Record<string, unknown>;
// text is the only required field; anything else is a non-selection.
if (typeof r.text !== 'string' || r.text.trim().length === 0) return null;
let text = r.text;
let truncated = r.truncated === true;
if (text.length > TEXT_CAP) {
text = text.slice(0, TEXT_CAP);
truncated = true;
}
const result: SelectionContext = { text };
if (truncated) result.truncated = true;
if (Array.isArray(r.blockIds)) {
// Keep only well-formed, in-range ids (an oversize id is DROPPED, not
// truncated — a mangled id is worse than a missing one), then cap the count.
const ids = r.blockIds
.filter(
(x): x is string =>
typeof x === 'string' && x.length > 0 && x.length <= BLOCK_ID_CAP,
)
.slice(0, MAX_BLOCK_IDS);
if (ids.length > 0) result.blockIds = ids;
}
if (typeof r.before === 'string' && r.before.length > 0) {
result.before = r.before.slice(0, CONTEXT_CAP);
}
if (typeof r.after === 'string' && r.after.length > 0) {
result.after = r.after.slice(0, CONTEXT_CAP);
}
return result;
}
export interface CurrentPageInput {
id?: string;
title?: string;
// The already-sanitized selection nested onto the resolved open-page context
// by resolveOpenPageContext (never the raw client value). Passed through to
// the tool result verbatim; null when nothing is selected.
selection?: SelectionContext | null;
}
export interface CurrentPageResult {
page: { id: string; title: string } | null;
selection: SelectionContext | null; // null when nothing is selected or no page
}
// Resolve the "current page" tool result from the client-supplied open-page
// context. Returns { page: null } when no page is open (no id), otherwise the
// page id + title (title defaults to '' when absent). Mirrors the getCurrentPage
// tool's contract so it can be unit-tested without the ESM Docmost client.
// context. Returns { page: null, selection: null } when no page is open (no id),
// otherwise the page id + title (title defaults to '' when absent) plus the
// selection already sanitized+nested by resolveOpenPageContext. A null page
// always yields a null selection (the selection dies with the page). Mirrors the
// getCurrentPage tool's contract so it can be unit-tested without the ESM
// Docmost client.
export function resolveCurrentPageResult(
openedPage?: CurrentPageInput | null,
): CurrentPageResult {
if (!openedPage?.id) {
return { page: null };
return { page: null, selection: null };
}
return { page: { id: openedPage.id, title: openedPage.title ?? '' } };
return {
page: { id: openedPage.id, title: openedPage.title ?? '' },
selection: openedPage.selection ?? null,
};
}
@@ -98,56 +98,29 @@ export const INLINE_TOOL_TIERS: Record<
},
getCurrentPage: {
tier: 'core',
catalogLine: 'getCurrentPage — the page the user is currently viewing.',
},
getPage: {
tier: 'core',
catalogLine: 'getPage — fetch a page as Markdown by its id.',
},
listPages: {
tier: 'core',
catalogLine: "listPages — list recent pages, or a space's full page tree.",
},
listComments: {
tier: 'core',
catalogLine: 'listComments — list all comments on a page (including resolved).',
catalogLine:
'getCurrentPage — the page the user is currently viewing and their current text selection on it.',
},
// NOTE: getPage and listPages moved to @docmost/mcp's SHARED_TOOL_SPECS
// (#294); they carry their own tier ('core') + catalogLine there.
// NOTE: createComment, listComments and resolveComment moved to
// @docmost/mcp's SHARED_TOOL_SPECS (#294); they carry their own tier +
// catalogLine there. getComment stays inline (MCP-only shape divergence is
// n/a — it simply has no shared spec).
getComment: {
tier: 'core',
catalogLine: 'getComment — fetch a single comment by id.',
},
createComment: {
tier: 'core',
catalogLine:
'createComment — add an inline comment (optionally with a suggested edit).',
},
resolveComment: {
tier: 'core',
catalogLine: 'resolveComment — resolve or reopen a comment thread.',
},
// --- deferred inline ---
createPage: {
tier: 'deferred',
catalogLine: 'createPage — create a new page with a Markdown body in a space.',
},
// NOTE: createPage, renamePage, movePage, deletePage, updatePageJson and
// exportPageMarkdown moved to @docmost/mcp's SHARED_TOOL_SPECS (#294); they
// carry their own deferred tier + catalogLine there.
updatePageContent: {
tier: 'deferred',
catalogLine:
"updatePageContent — replace a page's body (and optionally title) with new Markdown.",
},
renamePage: {
tier: 'deferred',
catalogLine: "renamePage — change a page's title only (body untouched).",
},
movePage: {
tier: 'deferred',
catalogLine: 'movePage — move a page under a new parent or to the space root.',
},
deletePage: {
tier: 'deferred',
catalogLine: 'deletePage — move a page to trash (soft delete, reversible).',
},
listSidebarPages: {
tier: 'deferred',
catalogLine:
@@ -157,42 +130,21 @@ export const INLINE_TOOL_TIERS: Record<
tier: 'deferred',
catalogLine: 'getTable — read a table as a matrix of cell texts and cell ids.',
},
checkNewComments: {
tier: 'deferred',
catalogLine:
'checkNewComments — find comments in a space created after a timestamp.',
},
// NOTE: tableInsertRow, tableDeleteRow and tableUpdateCell moved to
// @docmost/mcp's SHARED_TOOL_SPECS (#294); they carry their own deferred tier +
// catalogLine there. getTable stays inline (its MCP name table_get breaks the
// snake_case(inAppKey) convention, so it has no shared spec).
// NOTE: checkNewComments moved to @docmost/mcp's SHARED_TOOL_SPECS (#294);
// it carries its own deferred tier + catalogLine there.
getPageHistory: {
tier: 'deferred',
catalogLine:
'getPageHistory — fetch one page-history version with its ProseMirror content.',
},
exportPageMarkdown: {
tier: 'deferred',
catalogLine:
'exportPageMarkdown — export a page to self-contained Markdown (body + comments).',
},
updatePageJson: {
tier: 'deferred',
catalogLine:
"updatePageJson — overwrite a page's body with a full ProseMirror document.",
},
tableInsertRow: {
tier: 'deferred',
catalogLine: 'tableInsertRow — insert a row of plain-text cells into a table.',
},
tableDeleteRow: {
tier: 'deferred',
catalogLine: 'tableDeleteRow — delete a table row at a 0-based index.',
},
tableUpdateCell: {
tier: 'deferred',
catalogLine: 'tableUpdateCell — set the text of a table cell at [row, col].',
},
sharePage: {
tier: 'deferred',
catalogLine: 'sharePage — make a page publicly accessible and return its URL.',
},
// NOTE: sharePage moved to @docmost/mcp's SHARED_TOOL_SPECS (#294); it carries
// its own deferred tier + catalogLine there. transformPage stays inline (its
// schema deliberately diverges — it omits the deleteComments field the MCP
// docmost_transform exposes, a comment-deletion guardrail).
transformPage: {
tier: 'deferred',
catalogLine: "transformPage — run a sandboxed JS transform over a page's document.",
@@ -23,8 +23,21 @@ import { hashPassword } from '../../../common/helpers';
* unthrottled password-guessing oracle.
*/
// bcrypt cost-12 hashing/compare takes ~300ms idle but multiple seconds when
// parallel jest workers saturate all CPU cores; the 5s default flakes.
jest.setTimeout(30_000);
const WORKSPACE_ID = 'ws-1';
let passwordHash: string;
// Hoist the expensive work: compute ONE bcrypt cost-12 hash shared by all
// tests instead of five. The hash is a read-only string and each test builds
// its own user object around it, so sharing is safe.
beforeAll(async () => {
passwordHash = await hashPassword('correct-horse');
}, 30_000);
// Build an AuthService with the dependencies verifyUserCredentials/login touch
// stubbed, and a userRepo whose findByEmail is overridable per test. Only the
// collaborators actually reached on these paths need real behaviour; the rest
@@ -95,7 +108,6 @@ describe('AuthService.verifyUserCredentials (live credentials-mismatch contract)
it('DISABLED user -> throws exactly CREDENTIALS_MISMATCH_MESSAGE (no password oracle)', async () => {
// A deactivated user must be indistinguishable from a wrong password: same
// message, before any password comparison.
const passwordHash = await hashPassword('correct-horse');
const disabledUser = {
id: 'u-1',
email: 'disabled@example.com',
@@ -117,7 +129,6 @@ describe('AuthService.verifyUserCredentials (live credentials-mismatch contract)
});
it('WRONG password -> throws exactly CREDENTIALS_MISMATCH_MESSAGE', async () => {
const passwordHash = await hashPassword('correct-horse');
const user = {
id: 'u-1',
email: 'user@example.com',
@@ -139,7 +150,6 @@ describe('AuthService.verifyUserCredentials (live credentials-mismatch contract)
});
it('CORRECT credentials -> resolves the matched user (no side effects here)', async () => {
const passwordHash = await hashPassword('correct-horse');
const user = {
id: 'u-1',
email: 'user@example.com',
@@ -179,7 +189,6 @@ describe('AuthService.login (live credentials-mismatch contract via verifyUserCr
});
it('WRONG password -> login throws exactly CREDENTIALS_MISMATCH_MESSAGE', async () => {
const passwordHash = await hashPassword('correct-horse');
const user = {
id: 'u-1',
email: 'user@example.com',
@@ -201,7 +210,6 @@ describe('AuthService.login (live credentials-mismatch contract via verifyUserCr
});
it('CORRECT credentials -> login mints the session (the side-effecting path)', async () => {
const passwordHash = await hashPassword('correct-horse');
const user = {
id: 'u-1',
email: 'user@example.com',
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
import { InjectQueue } from '@nestjs/bullmq';
import { QueueJob, QueueName } from '../../integrations/queue/constants';
import { Queue } from 'bullmq';
import { EnvironmentService } from '../../integrations/environment/environment.service';
/**
* Thin snapshot of a page node carried inside domain events so the WebSocket
@@ -112,48 +111,24 @@ export class PageListener {
private readonly logger = new Logger(PageListener.name);
constructor(
private readonly environmentService: EnvironmentService,
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
) {}
@OnEvent(EventName.PAGE_CREATED)
async handlePageCreated(event: PageEvent) {
const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_CREATED, {
pageIds,
});
}
await this.aiQueue.add(QueueJob.PAGE_CREATED, { pageIds, workspaceId });
}
@OnEvent(EventName.PAGE_UPDATED)
async handlePageUpdated(event: PageEvent) {
const { pageIds } = event;
await this.searchQueue.add(QueueJob.PAGE_UPDATED, { pageIds });
}
@OnEvent(EventName.PAGE_DELETED)
async handlePageDeleted(event: PageEvent) {
const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_DELETED, { pageIds });
}
await this.aiQueue.add(QueueJob.PAGE_DELETED, { pageIds, workspaceId });
}
@OnEvent(EventName.PAGE_SOFT_DELETED)
async handlePageSoftDeleted(event: PageEvent) {
const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_SOFT_DELETED, { pageIds });
}
await this.aiQueue.add(QueueJob.PAGE_SOFT_DELETED, {
pageIds,
workspaceId,
@@ -163,14 +138,6 @@ export class PageListener {
@OnEvent(EventName.PAGE_RESTORED)
async handlePageRestored(event: PageEvent) {
const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_RESTORED, { pageIds });
}
await this.aiQueue.add(QueueJob.PAGE_RESTORED, { pageIds, workspaceId });
}
isTypesense(): boolean {
return this.environmentService.getSearchDriver() === 'typesense';
}
}
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
import { InjectQueue } from '@nestjs/bullmq';
import { QueueJob, QueueName } from '../../integrations/queue/constants';
import { Queue } from 'bullmq';
import { EnvironmentService } from '../../integrations/environment/environment.service';
export class SpaceEvent {
spaceId: string;
@@ -15,22 +14,12 @@ export class SpaceListener {
private readonly logger = new Logger(SpaceListener.name);
constructor(
private readonly environmentService: EnvironmentService,
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
) {}
@OnEvent(EventName.SPACE_DELETED)
async handleSpaceDeleted(event: SpaceEvent) {
const { spaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.SPACE_DELETED, { spaceId });
}
await this.aiQueue.add(QueueJob.SPACE_DELETED, { spaceId });
}
isTypesense(): boolean {
return this.environmentService.getSearchDriver() === 'typesense';
}
}
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
import { InjectQueue } from '@nestjs/bullmq';
import { QueueJob, QueueName } from '../../integrations/queue/constants';
import { Queue } from 'bullmq';
import { EnvironmentService } from '../../integrations/environment/environment.service';
export class WorkspaceEvent {
workspaceId: string;
@@ -15,22 +14,12 @@ export class WorkspaceListener {
private readonly logger = new Logger(WorkspaceListener.name);
constructor(
private readonly environmentService: EnvironmentService,
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
) {}
@OnEvent(EventName.WORKSPACE_DELETED)
async handlePageDeleted(event: WorkspaceEvent) {
const { workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.WORKSPACE_DELETED, { workspaceId });
}
await this.aiQueue.add(QueueJob.WORKSPACE_DELETED, { workspaceId });
}
isTypesense(): boolean {
return this.environmentService.getSearchDriver() === 'typesense';
}
}
+4
View File
@@ -24,6 +24,10 @@ const migrator = new Migrator({
path,
migrationFolder,
}),
// Match the startup auto-migrator (migration.service.ts): a back-dated
// migration from a long-lived branch must be applied, not rejected as
// "corrupted migrations" (incident #361). See that file for the full rationale.
allowUnorderedMigrations: true,
});
run(db, migrator, migrationFolder);
@@ -19,6 +19,16 @@ export class MigrationService {
path,
migrationFolder: path.join(__dirname, '..', 'migrations'),
}),
// A long-lived branch can add a migration whose timestamped filename sorts
// BEFORE migrations already applied in prod (e.g. #234's 20260627 landing
// after 20260704 was live). With the default (ordered) setting the startup
// migrator then sees "corrupted migrations" — the applied set is no longer a
// prefix of the sorted list — throws, and the app crash-loops on boot
// (incident #361: 502s for ~11 min). allowUnorderedMigrations runs any
// not-yet-applied migration regardless of filename order, so a back-dated
// migration is applied instead of bricking startup. A CI order-gate still
// discourages back-dating; this is the runtime safety net.
allowUnorderedMigrations: true,
});
const { error, results } = await migrator.migrateToLatest();
@@ -0,0 +1,124 @@
import { readFileSync } from 'fs';
import { streamText, Output } from 'ai';
import { MockLanguageModelV3, simulateReadableStream } from 'ai/test';
/**
* Regression tests for patches/ai@6.0.134.patch (server heap OOM on long
* autonomous agent runs, #184).
*
* Unpatched ai@6.0.134 substitutes the default text() output strategy even
* when the caller passes NO `output` option. Its createOutputTransformStream
* then accumulates the ENTIRE turn text and, on EVERY text-delta, enqueues a
* flat snapshot of all text so far as `partialOutput` (O(n^2) memory). Those
* snapshots pile up in the never-consumed leftover tee() branch of
* DefaultStreamTextResult.baseStream, which is what OOM'd production during a
* ~28k-chunk agent turn. The pnpm patch skips partialOutput production
* entirely when no output strategy was requested, while keeping per-delta
* streaming granularity.
*/
describe('ai@6.0.134 pnpm patch: no partialOutput accumulation without an output strategy', () => {
const makeModel = () =>
new MockLanguageModelV3({
doStream: async () => ({
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'text-start' as const, id: '1' },
{ type: 'text-delta' as const, id: '1', delta: 'Hello' },
{ type: 'text-delta' as const, id: '1', delta: ', ' },
{ type: 'text-delta' as const, id: '1', delta: 'world!' },
{ type: 'text-end' as const, id: '1' },
{
type: 'finish' as const,
finishReason: { unified: 'stop' as const, raw: 'stop' },
usage: {
inputTokens: {
total: 1,
noCache: undefined,
cacheRead: undefined,
cacheWrite: undefined,
},
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
}),
});
it('preserves per-delta streaming granularity in textStream', async () => {
const result = streamText({ model: makeModel(), prompt: 'hi' });
const deltas: string[] = [];
for await (const delta of result.textStream) {
deltas.push(delta);
}
// The patch must NOT coalesce or drop deltas: three model deltas arrive
// as three separate textStream chunks.
expect(deltas).toEqual(['Hello', ', ', 'world!']);
});
it('emits NO partialOutput values when the caller did not request an output strategy', async () => {
const result = streamText({ model: makeModel(), prompt: 'hi' });
// Fully consume the primary stream first (mirrors production usage).
for await (const _ of result.textStream) {
// drain
}
const partials: unknown[] = [];
for await (const partial of result.experimental_partialOutputStream) {
partials.push(partial);
}
// TRIPWIRE: on unpatched ai@6.0.134 the default text() output strategy
// yields one cumulative partial per text-delta here (['Hello', 'Hello, ',
// 'Hello, world!']). An empty stream proves the patch is applied and no
// cumulative snapshots are being produced (and thus none can pile up in
// the leftover internal tee branch).
expect(partials).toEqual([]);
});
it('preserves cumulative partialOutput when the caller DOES request an output strategy', async () => {
// PRESERVE-BRANCH GUARD: the patch only short-circuits partialOutput when
// `output == null`. When an output strategy IS set (here Output.text()),
// createOutputTransformStream must fall through to the ORIGINAL code path
// and keep publishing cumulative snapshots, so object/text-output consumers
// behave byte-identically to unpatched ai. A careless re-port that routed
// output-set calls into the skip branch would leave partialOutput empty and
// silently break those consumers — this test is the tripwire for that.
const result = streamText({
model: makeModel(),
prompt: 'hi',
experimental_output: Output.text(),
});
// Drain the primary stream fully and accumulate the complete output text.
let fullText = '';
for await (const delta of result.textStream) {
fullText += delta;
}
const partials: string[] = [];
for await (const partial of result.experimental_partialOutputStream) {
partials.push(partial);
}
// With a strategy set, partialOutput must be PRESERVED (non-empty) and
// cumulative: the last emitted partial equals the full accumulated text.
expect(partials.length).toBeGreaterThan(0);
expect(partials[partials.length - 1]).toBe(fullText);
expect(fullText).toBe('Hello, world!');
});
it('both installed dist builds (CJS and ESM) carry the patch marker', () => {
// Secondary guard: pins the patch to BOTH bundles the SDK ships, since
// the NestJS server consumes CJS while other tooling may load ESM.
const cjsPath = require.resolve('ai');
const mjsPath = cjsPath.replace(/index\.js$/, 'index.mjs');
expect(cjsPath).toMatch(/index\.js$/);
expect(readFileSync(cjsPath, 'utf8')).toContain('PATCH(docmost');
expect(readFileSync(mjsPath, 'utf8')).toContain('PATCH(docmost');
});
});
@@ -292,6 +292,23 @@ export class EnvironmentService {
return enabled === 'true';
}
/**
* Resumable SSE transport for durable agent runs (#184 phase 1.5). When
* enabled, a run tees its SSE frames into the in-memory run-stream registry so
* a late/reloaded tab can attach (replay + live tail) via
* `GET /ai-chat/runs/:chatId/stream`. Defaults to DISABLED: PR 1 ships the
* server code dormant with the flag off, `open`/`bind`/`generateMessageId`
* are never called and attach always answers 204, so the legacy and #184
* phase-1 wire paths stay byte-for-byte identical. Set
* AI_CHAT_RESUMABLE_STREAM=true to activate it (paired with the PR 2 client).
*/
isAiChatResumableStreamEnabled(): boolean {
const enabled = this.configService
.get<string>('AI_CHAT_RESUMABLE_STREAM', 'false')
.toLowerCase();
return enabled === 'true';
}
getPostHogHost(): string {
return this.configService.get<string>('POSTHOG_HOST');
}
@@ -2,15 +2,36 @@ import { FastifyReply, FastifyRequest } from 'fastify';
import { isStreamingResponse } from './metrics.constants';
import { observeHttp } from './metrics.registry';
// URL path prefixes served by @fastify/static (client build output under
// client/dist). `/assets/` holds the content-hashed bundle (index-*.js,
// chunk-*.js) — a NEW set of names every deploy, i.e. an UNBOUNDED label set
// (#362); the others (/vad/, /brand/, /locales/, /icons/ — copied verbatim from
// public/) have stable names, so they are merely repetitive per-file labels
// rather than unbounded. Either way none of these belong in the API-route
// histogram: collapse them all to one bounded `static` label. (Edge latency for
// static is already measured by Traefik's traefik_router_request_duration_*.)
const STATIC_PATH_PREFIXES = [
'/assets/',
'/vad/',
'/brand/',
'/locales/',
'/icons/',
];
/**
* Resolve the BOUNDED route label for an HTTP response.
*
* HARD REQUIREMENT (#355): use the ROUTE TEMPLATE (`/pages/:id`), NEVER the raw
* URL (`/pages/abc-123`), so label cardinality stays finite. Fastify exposes the
* matched template on `req.routeOptions.url`. On 404s (no route matched) that is
* missing collapse to the literal `unknown`.
* HARD REQUIREMENT (#355): use the ROUTE TEMPLATE (`/pages/:id`), NEVER a raw
* URL (`/pages/abc-123` or `/assets/index-CAbxDtto.js`), so label cardinality
* stays finite. Fastify exposes the matched template on `req.routeOptions.url`,
* BUT @fastify/static serves each file through a route whose matched url is the
* raw (hashed) file path so for static assets that value is itself unbounded.
* Detect static requests by their path prefix FIRST and collapse to `static`;
* otherwise use the route template; on a 404 (no route matched) `unknown`.
*/
export function resolveRouteLabel(req: FastifyRequest): string {
const path = (req.url ?? '').split('?', 1)[0];
if (STATIC_PATH_PREFIXES.some((p) => path.startsWith(p))) return 'static';
const url = req.routeOptions?.url;
return typeof url === 'string' && url.length > 0 ? url : 'unknown';
}
@@ -46,7 +46,6 @@ export class MetricsBullService implements OnModuleInit, OnModuleDestroy {
@InjectQueue(QueueName.GENERAL_QUEUE) generalQueue: Queue,
@InjectQueue(QueueName.BILLING_QUEUE) billingQueue: Queue,
@InjectQueue(QueueName.FILE_TASK_QUEUE) fileTaskQueue: Queue,
@InjectQueue(QueueName.SEARCH_QUEUE) searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) aiQueue: Queue,
@InjectQueue(QueueName.HISTORY_QUEUE) historyQueue: Queue,
@InjectQueue(QueueName.NOTIFICATION_QUEUE) notificationQueue: Queue,
@@ -58,7 +57,6 @@ export class MetricsBullService implements OnModuleInit, OnModuleDestroy {
{ label: 'general', queue: generalQueue },
{ label: 'billing', queue: billingQueue },
{ label: 'file-task', queue: fileTaskQueue },
{ label: 'search', queue: searchQueue },
{ label: 'ai', queue: aiQueue },
{ label: 'history', queue: historyQueue },
{ label: 'notification', queue: notificationQueue },
@@ -25,6 +25,61 @@ describe('resolveRouteLabel (histogram route label)', () => {
const req = { url: '/x' } as unknown as FastifyRequest;
expect(resolveRouteLabel(req)).toBe('unknown');
});
it.each([
'/assets/index-CAbxDtto.js',
'/assets/chunk-3OPIFGDE-CJOt9nr5.js',
'/assets/excalidraw-menu-DpsI0kFW.js',
'/vad/silero_vad_v5.onnx',
'/brand/logo.svg',
'/locales/en.json',
'/icons/app-icon-192x192.png',
])('collapses hashed/static asset %p to "static" (#362 cardinality)', (url) => {
// @fastify/static serves each file through a route whose matched url is the
// raw (hashed) file path, so routeOptions.url is itself unbounded here.
const req = {
url,
routeOptions: { url },
} as unknown as FastifyRequest;
const label = resolveRouteLabel(req);
expect(label).toBe('static');
expect(label).not.toContain('.js');
expect(label).not.toContain('index-');
});
it('strips the query string before the static-prefix check', () => {
const req = {
url: '/assets/index-CAbxDtto.js?v=2',
routeOptions: { url: '/assets/index-CAbxDtto.js' },
} as unknown as FastifyRequest;
expect(resolveRouteLabel(req)).toBe('static');
});
it('does NOT collapse a real API route that merely mentions assets', () => {
// A templated API route is kept as-is; only the static path PREFIXES collapse.
const req = {
url: '/api/pages/assets-guide',
routeOptions: { url: '/api/pages/:id' },
} as unknown as FastifyRequest;
expect(resolveRouteLabel(req)).toBe('/api/pages/:id');
});
it.each([
// The TRAILING SLASH on the prefix is the anti-false-collapse guard: a path
// that is the prefix WITHOUT its slash, or merely shares the prefix as a
// substring of a longer segment, must NOT collapse. These would collapse
// under a buggy `includes('/assets/')` / slashless-prefix impl.
'/assets',
'/assetsx/foo.js',
'/iconset/x.png',
])('does NOT collapse the prefix-boundary case %p', (url) => {
const req = {
url,
routeOptions: { url: '/some/:route' },
} as unknown as FastifyRequest;
expect(resolveRouteLabel(req)).not.toBe('static');
expect(resolveRouteLabel(req)).toBe('/some/:route');
});
});
describe('isStreamingResponse (SSE exclusion)', () => {
@@ -4,7 +4,6 @@ export enum QueueName {
GENERAL_QUEUE = '{general-queue}',
BILLING_QUEUE = '{billing-queue}',
FILE_TASK_QUEUE = '{file-task-queue}',
SEARCH_QUEUE = '{search-queue}',
AI_QUEUE = '{ai-queue}',
HISTORY_QUEUE = '{history-queue}',
NOTIFICATION_QUEUE = '{notification-queue}',
@@ -32,12 +31,6 @@ export enum QueueJob {
IMPORT_TASK = 'import-task',
EXPORT_TASK = 'export-task',
SEARCH_INDEX_PAGE = 'search-index-page',
SEARCH_INDEX_PAGES = 'search-index-pages',
SEARCH_INDEX_COMMENT = 'search-index-comment',
SEARCH_INDEX_COMMENTS = 'search-index-comments',
SEARCH_INDEX_ATTACHMENT = 'search-index-attachment',
SEARCH_INDEX_ATTACHMENTS = 'search-index-attachments',
SEARCH_REMOVE_PAGE = 'search-remove-page',
SEARCH_REMOVE_ASSET = 'search-remove-attachment',
SEARCH_REMOVE_FACE = 'search-remove-comment',
@@ -57,14 +57,6 @@ import { GeneralQueueProcessor } from './processors/general-queue.processor';
attempts: 1,
},
}),
BullModule.registerQueue({
name: QueueName.SEARCH_QUEUE,
defaultJobOptions: {
removeOnComplete: true,
removeOnFail: true,
attempts: 2,
},
}),
BullModule.registerQueue({
name: QueueName.AI_QUEUE,
defaultJobOptions: {
@@ -0,0 +1,564 @@
import * as http from 'node:http';
import { Kysely } from 'kysely';
import {
MockLanguageModelV3,
convertArrayToReadableStream,
} from 'ai/test';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatService } from 'src/core/ai-chat/ai-chat.service';
import { AiChatRunService } from 'src/core/ai-chat/ai-chat-run.service';
import {
AiChatStreamRegistryService,
RunStreamCallbacks,
} from 'src/core/ai-chat/ai-chat-stream-registry.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
} from './db';
/**
* #184 phase 1.5 the resumable transport end to end against REAL Postgres,
* the REAL `streamText` (seeded via MockLanguageModelV3) and a REAL Node
* ServerResponse, driving the REAL `AiChatService.stream` run-wrapped path with a
* REAL `AiChatStreamRegistryService`. The run-hooks mirror the controller: they
* begin a durable run and `open()` the registry entry at begin, and the service
* tees the SSE frames into it via `consumeSseStream` while stamping the DB row id
* via `generateMessageId` (both gated on runId + the resumable flag).
*
* Proven here: a finished run's replay is the full frame sequence incl `[DONE]`
* with `start.messageId` == the seeded DB row id; the anchor check (invariant 6);
* an attach opened BEFORE the first frame follows the live stream from frame 0; an
* explicit stop surfaces `{"type":"abort"}` + `[DONE]` + end to the subscriber;
* and the legacy (non-run) path tees nothing.
*/
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
async function waitFor(
cond: () => Promise<boolean> | boolean,
{ timeoutMs = 15_000, stepMs = 25 } = {},
): Promise<void> {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
if (await cond()) return;
await sleep(stepMs);
}
throw new Error('waitFor: condition not met within timeout');
}
// A real Node ServerResponse wired to a live socket (as in the stream int-spec).
function makeRealResponse(): Promise<{
res: http.ServerResponse;
cleanup: () => Promise<void>;
}> {
return new Promise((resolve) => {
const server = http.createServer((_req, res) => {
resolve({
res,
cleanup: () =>
new Promise<void>((done) => {
try {
if (!res.writableEnded) res.end();
} catch {
/* socket already gone */
}
server.close(() => done());
}),
});
});
server.listen(0, () => {
const port = (server.address() as any).port;
const creq = http.request({ port, method: 'GET' }, (cres) => {
cres.resume();
});
creq.on('error', () => undefined);
creq.end();
});
});
}
// A full, successful single-step turn.
function successStream() {
return convertArrayToReadableStream([
{ type: 'stream-start', warnings: [] },
{ type: 'text-start', id: 't1' },
{ type: 'text-delta', id: 't1', delta: 'Hello' },
{ type: 'text-delta', id: 't1', delta: ' there' },
{ type: 'text-end', id: 't1' },
{
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 10, outputTokens: 5, totalTokens: 15 },
},
] as any);
}
// A stream the test feeds chunk by chunk (to attach mid-flight / drive a stop).
function makeControlledStream() {
let controller!: ReadableStreamDefaultController<any>;
const stream = new ReadableStream<any>({
start(c) {
controller = c;
},
});
return {
stream,
emit: (chunk: any) => controller.enqueue(chunk),
close: () => controller.close(),
};
}
// Collect replay + live frames from an attachment.
function liveSink(): {
cb: RunStreamCallbacks;
frames: string[];
ended: () => boolean;
} {
const frames: string[] = [];
let ended = false;
return {
frames,
ended: () => ended,
cb: {
onFrame: (f) => frames.push(f),
onEnd: () => {
ended = true;
},
},
};
}
// The SSE `start` frame carries the message id; pull it out of a `data: {...}`.
function parseStartMessageId(frames: string[]): string | undefined {
for (const f of frames) {
const m = /^data: (\{.*\})\s*$/m.exec(f.trim());
if (!m) continue;
try {
const json = JSON.parse(m[1]);
if (json.type === 'start') return json.messageId;
} catch {
/* not this frame */
}
}
return undefined;
}
describe('AiChatService run-stream attach [integration]', () => {
let db: Kysely<any>;
let aiChatRepo: AiChatRepo;
let msgRepo: AiChatMessageRepo;
let runRepo: AiChatRunRepo;
let workspaceId: string;
let userId: string;
const mcpClients = {
toolsFor: async () => ({
tools: {},
clients: [],
outcomes: [],
instructions: [],
}),
};
// Build the service with the run-stream registry wired and the resumable flag
// ON (the property under test). Deferred tools OFF (irrelevant here).
function buildService(registry: AiChatStreamRegistryService): AiChatService {
return new AiChatService(
{ getChatModel: async () => null } as any,
aiChatRepo,
msgRepo,
{} as any,
{ resolve: async () => null } as any,
{ forUser: async () => ({}) } as any,
mcpClients as any,
{} as any,
{} as any,
{} as any,
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatResumableStreamEnabled: () => true,
} as any,
registry,
);
}
// Run-hooks mirroring the controller: begin the durable run AND open() the
// registry entry at begin. Captures the runId so a test can stop it.
function makeRunHooks(
runService: AiChatRunService,
registry: AiChatStreamRegistryService,
box: { runId?: string },
) {
return {
begin: async (chatId: string) => {
const handle = await runService.beginRun({
chatId,
workspaceId,
userId,
trigger: 'user',
});
box.runId = handle.runId;
registry.open(chatId, handle.runId);
return handle;
},
onAssistantSeeded: (runId: string, messageId: string) =>
runService.linkAssistantMessage(runId, workspaceId, messageId),
onStep: (runId: string, n: number) =>
void runService.recordStep(runId, workspaceId, n),
onSettled: (runId: string, status: any, error?: string) =>
runService.finalizeRun(runId, workspaceId, status, error),
};
}
function userUiMessage(text: string) {
return {
id: `u-${Math.random()}`,
role: 'user',
parts: [{ type: 'text', text }],
};
}
async function startRun(opts: {
registry: AiChatStreamRegistryService;
runService?: AiChatRunService;
model: MockLanguageModelV3;
chatId: string;
body: any;
box?: { runId?: string };
}): Promise<{ res: http.ServerResponse; cleanup: () => Promise<void> }> {
const service = buildService(opts.registry);
const { res, cleanup } = await makeRealResponse();
const runHooks = opts.runService
? makeRunHooks(opts.runService, opts.registry, opts.box ?? {})
: undefined;
await service.stream({
user: { id: userId, workspaceId } as any,
workspace: { id: workspaceId, name: 'WS' } as any,
sessionId: 'sess-1',
body: opts.body,
res: { raw: res } as any,
signal: new AbortController().signal,
model: opts.model as any,
role: null,
runHooks,
} as any);
return { res, cleanup };
}
async function assistantRowId(chatId: string): Promise<string> {
const rows = await msgRepo.findAllByChat(chatId, workspaceId);
const row = rows.find((r: any) => r.role === 'assistant');
return row!.id as string;
}
beforeAll(async () => {
db = getTestDb();
aiChatRepo = new AiChatRepo(db as any);
msgRepo = new AiChatMessageRepo(db as any);
runRepo = new AiChatRunRepo(db as any);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
it('run-wrapped: replay is the full frame sequence incl [DONE], start.messageId == the seeded DB row id', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const registry = new AiChatStreamRegistryService();
const runService = new AiChatRunService(runRepo, {
isCloud: () => false,
} as never);
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: successStream() }),
} as any);
const box: { runId?: string } = {};
const { cleanup } = await startRun({
registry,
runService,
model,
chatId,
body: { chatId, messages: [userUiMessage('Hi')] },
box,
});
try {
// Wait for the assistant row to settle (terminal callbacks run async).
await waitFor(async () => {
const rows = await msgRepo.findAllByChat(chatId, workspaceId);
return rows.some(
(r: any) =>
r.role === 'assistant' &&
['completed', 'error', 'aborted'].includes(r.status),
);
});
const rowId = await assistantRowId(chatId);
// Finished-run replay with expect=live + the correct anchor.
const sink = liveSink();
const att = await registry.attach(chatId, true, rowId, sink.cb);
expect(att).not.toBeNull();
expect(att!.finished).toBe(true);
// The tee captured frames (consumeSseStream was wired).
expect(att!.replay.length).toBeGreaterThan(0);
// generateMessageId stamped the DB row id onto the streamed start frame.
expect(parseStartMessageId(att!.replay)).toBe(rowId);
// The full sequence includes the streamed text and the terminal marker.
const joined = att!.replay.join('');
expect(joined).toContain('Hello');
expect(att!.replay.some((f) => f.includes('[DONE]'))).toBe(true);
} finally {
registry.onModuleDestroy();
await cleanup();
}
});
it('anchor mismatch with expect=live returns null (invariant 6)', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const registry = new AiChatStreamRegistryService();
const runService = new AiChatRunService(runRepo, {
isCloud: () => false,
} as never);
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: successStream() }),
} as any);
const { cleanup } = await startRun({
registry,
runService,
model,
chatId,
body: { chatId, messages: [userUiMessage('Hi')] },
});
try {
await waitFor(async () => {
const rows = await msgRepo.findAllByChat(chatId, workspaceId);
return rows.some(
(r: any) => r.role === 'assistant' && r.status === 'completed',
);
});
const sink = liveSink();
// A foreign anchor must NOT replay this run's transcript.
expect(
await registry.attach(chatId, true, 'a-different-run-row', sink.cb),
).toBeNull();
} finally {
registry.onModuleDestroy();
await cleanup();
}
});
it('an attach opened BEFORE the first frame follows the live stream from frame 0', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const registry = new AiChatStreamRegistryService();
const runService = new AiChatRunService(runRepo, {
isCloud: () => false,
} as never);
const controlled = makeControlledStream();
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: controlled.stream }),
} as any);
const { cleanup } = await startRun({
registry,
runService,
model,
chatId,
body: { chatId, messages: [userUiMessage('Slow please')] },
});
try {
// Attach while the entry exists (opened at begin) but before any frame.
const sink = liveSink();
const att = (await registry.attach(chatId, false, undefined, sink.cb))!;
expect(att.replay).toEqual([]); // nothing streamed yet -> replay from 0
att.start(); // go live (drains nothing, then follows)
// Now emit the whole turn.
controlled.emit({ type: 'stream-start', warnings: [] });
controlled.emit({ type: 'text-start', id: 't1' });
controlled.emit({ type: 'text-delta', id: 't1', delta: 'Zero' });
controlled.emit({ type: 'text-end', id: 't1' });
controlled.emit({
type: 'finish',
finishReason: 'stop',
usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 },
});
controlled.close();
await waitFor(() => sink.frames.some((f) => f.includes('[DONE]')));
// The subscriber saw the stream from the very first frame (`start`) through
// the terminal marker, with the streamed text present.
expect(sink.frames.some((f) => f.includes('"type":"start"'))).toBe(true);
expect(sink.frames.join('')).toContain('Zero');
expect(sink.frames[sink.frames.length - 1]).toContain('[DONE]');
expect(sink.ended()).toBe(true);
} finally {
registry.onModuleDestroy();
await cleanup();
}
});
it('requestStop surfaces {"type":"abort"} + [DONE] + end to the attached subscriber', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const registry = new AiChatStreamRegistryService();
const runService = new AiChatRunService(runRepo, {
isCloud: () => false,
} as never);
// An abort-AWARE model: it streams some partial output, then errors the model
// stream with an AbortError when the run signal aborts — exactly as a real
// provider network stream is torn down on abort (a plain in-memory stream
// would just stall, so streamText would never observe the stop).
const model = new MockLanguageModelV3({
doStream: async ({ abortSignal }: any) => {
const stream = new ReadableStream<any>({
start(controller) {
controller.enqueue({ type: 'stream-start', warnings: [] });
controller.enqueue({ type: 'text-start', id: 't1' });
controller.enqueue({ type: 'text-delta', id: 't1', delta: 'partial' });
abortSignal?.addEventListener('abort', () => {
try {
controller.error(
new DOMException('Aborted', 'AbortError'),
);
} catch {
/* already errored/closed */
}
});
},
});
return { stream };
},
} as any);
const box: { runId?: string } = {};
const { cleanup } = await startRun({
registry,
runService,
model,
chatId,
body: { chatId, messages: [userUiMessage('Start then stop')] },
box,
});
try {
const sink = liveSink();
const att = (await registry.attach(chatId, false, undefined, sink.cb))!;
att.start();
// Give streamText a beat to begin consuming the partial output.
await sleep(250);
// User presses Stop -> the run signal aborts -> the SDK emits an abort chunk.
await runService.requestStop(box.runId!, workspaceId);
await waitFor(() => sink.ended());
expect(sink.frames.some((f) => f.includes('"type":"abort"'))).toBe(true);
expect(sink.frames.some((f) => f.includes('[DONE]'))).toBe(true);
expect(sink.ended()).toBe(true);
} finally {
registry.onModuleDestroy();
await cleanup();
}
});
it('the outer catch calls abortEntry so an open entry is released (finished)', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const registry = new AiChatStreamRegistryService();
const runService = new AiChatRunService(runRepo, {
isCloud: () => false,
} as never);
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: successStream() }),
} as any);
// A msgRepo whose user-row insert throws: the turn fails AFTER begin (the
// entry is already open) but BEFORE the pipe, exercising the outer catch.
const throwingMsgRepo = {
insert: async () => {
throw new Error('db boom');
},
findAllByChat: (...a: any[]) => (msgRepo as any).findAllByChat(...a),
update: (...a: any[]) => (msgRepo as any).update(...a),
findById: (...a: any[]) => (msgRepo as any).findById(...a),
};
const service = new AiChatService(
{ getChatModel: async () => null } as any,
aiChatRepo,
throwingMsgRepo as any,
{} as any,
{ resolve: async () => null } as any,
{ forUser: async () => ({}) } as any,
mcpClients as any,
{} as any,
{} as any,
{} as any,
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatResumableStreamEnabled: () => true,
} as any,
registry,
);
const { res, cleanup } = await makeRealResponse();
const box: { runId?: string } = {};
try {
await expect(
service.stream({
user: { id: userId, workspaceId } as any,
workspace: { id: workspaceId, name: 'WS' } as any,
sessionId: 'sess-1',
body: { chatId, messages: [userUiMessage('will throw')] },
res: { raw: res } as any,
signal: new AbortController().signal,
model: model as any,
role: null,
runHooks: makeRunHooks(runService, registry, box),
} as any),
).rejects.toThrow();
// The entry opened at begin was terminated by abortEntry (from the catch),
// so it is finished and a plain attach returns null instead of hanging.
const entry = (registry as any).entries.get(chatId);
expect(entry).toBeDefined();
expect(entry.finished).toBe(true);
const sink = liveSink();
expect(await registry.attach(chatId, false, undefined, sink.cb)).toBeNull();
} finally {
registry.onModuleDestroy();
await cleanup();
}
});
it('legacy (no run-hooks): the registry is never populated', async () => {
const chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
const registry = new AiChatStreamRegistryService();
const model = new MockLanguageModelV3({
doStream: async () => ({ stream: successStream() }),
} as any);
// No runHooks -> runId undefined -> the run-wrapped tee is never wired.
const { cleanup } = await startRun({
registry,
model,
chatId,
body: { chatId, messages: [userUiMessage('Legacy hi')] },
});
try {
await waitFor(async () => {
const rows = await msgRepo.findAllByChat(chatId, workspaceId);
return rows.some(
(r: any) => r.role === 'assistant' && r.status === 'completed',
);
});
const sink = liveSink();
// No entry was ever opened; attach always yields null.
expect(await registry.attach(chatId, false, undefined, sink.cb)).toBeNull();
expect(await registry.attach(chatId, true, 'anything', sink.cb)).toBeNull();
} finally {
registry.onModuleDestroy();
await cleanup();
}
});
});
+38 -18
View File
@@ -38,6 +38,24 @@ export const TEST_DATABASE_URL =
process.env.TEST_DATABASE_URL ??
'postgresql://docmost:docmost_dev_pw@localhost:5432/docmost_test';
// Build the raw postgres.js client (mirrors database.module.ts: max pool,
// silenced notices, bigint-as-number parsing). Kept separate so the singleton
// can hold a reference to bound its shutdown in destroyTestDb.
function buildTestSql(url: string = TEST_DATABASE_URL) {
return postgres(url, {
max: 5,
onnotice: () => {},
types: {
bigint: {
to: 20,
from: [20, 1700],
serialize: (value: number) => value.toString(),
parse: (value: string) => Number.parseInt(value),
},
},
});
}
/**
* Build a Kysely instance that MIRRORS the app's setup in database.module.ts:
* PostgresJSDialect over postgres(), CamelCasePlugin, and the bigint type
@@ -47,38 +65,40 @@ export const TEST_DATABASE_URL =
*/
export function buildTestDb(url: string = TEST_DATABASE_URL): Kysely<any> {
return new Kysely<any>({
dialect: new PostgresJSDialect({
postgres: postgres(url, {
max: 5,
onnotice: () => {},
types: {
bigint: {
to: 20,
from: [20, 1700],
serialize: (value: number) => value.toString(),
parse: (value: string) => Number.parseInt(value),
},
},
}),
}),
dialect: new PostgresJSDialect({ postgres: buildTestSql(url) }),
plugins: [new CamelCasePlugin()],
});
}
let singleton: Kysely<any> | undefined;
let singletonSql: ReturnType<typeof buildTestSql> | undefined;
/** Lazily-built shared Kysely for the test suite (one per worker; maxWorkers=1). */
export function getTestDb(): Kysely<any> {
if (!singleton) {
singleton = buildTestDb();
singletonSql = buildTestSql();
singleton = new Kysely<any>({
dialect: new PostgresJSDialect({ postgres: singletonSql }),
plugins: [new CamelCasePlugin()],
});
}
return singleton;
}
export async function destroyTestDb(): Promise<void> {
if (singleton) {
await singleton.destroy();
singleton = undefined;
if (!singleton) return;
const sql = singletonSql;
// Clear the refs first so a hung end() cannot leave a half-closed singleton.
singleton = undefined;
singletonSql = undefined;
// postgres.js .end() waits indefinitely for in-flight queries by default; a
// leaked/stuck pooled connection would hang the afterAll hook (a 60s hook
// timeout in CI). Bound the shutdown: the { timeout } grace period lets
// active queries drain, then force-closes lingering sockets so teardown
// always completes. We close the pool directly instead of Kysely.destroy()
// (which would call sql.end() again with no timeout).
if (sql) {
await sql.end({ timeout: 5 });
}
}
+5 -1
View File
@@ -4,10 +4,14 @@
"testEnvironment": "node",
"testRegex": ".e2e-spec.ts$",
"transform": {
"prosemirror-markdown/build/.+\\.js$": [
"babel-jest",
{ "presets": [["@babel/preset-env", { "targets": { "node": "current" } }]] }
],
"^.+\\.(t|j)sx?$": ["ts-jest", { "tsconfig": { "allowJs": true } }]
},
"transformIgnorePatterns": [
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@sindresorhus[+/][a-z0-9-]+|escape-string-regexp|p-limit|yocto-queue)(@|/))"
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@sindresorhus[+/][a-z0-9-]+|escape-string-regexp|p-limit|yocto-queue|@docmost/prosemirror-markdown)(@|/))"
],
"moduleNameMapper": {
"^@docmost/db/(.*)$": "<rootDir>/../src/database/$1",
+5 -1
View File
@@ -4,10 +4,14 @@
"testRegex": ".*\\.int-spec\\.ts$",
"testPathIgnorePatterns": ["/node_modules/"],
"transform": {
"prosemirror-markdown/build/.+\\.js$": [
"babel-jest",
{ "presets": [["@babel/preset-env", { "targets": { "node": "current" } }]] }
],
"^.+\\.(t|j)sx?$": "ts-jest"
},
"transformIgnorePatterns": [
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0)(@|/))"
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@docmost/prosemirror-markdown)(@|/))"
],
"testEnvironment": "node",
"testTimeout": 60000,
+9
View File
@@ -131,5 +131,14 @@ const { Client } = require("pg");
7. **Migrations don't auto-run in dev** — run `migration:latest` after every pull
or branch switch.
8. **Automation (Playwright): type into the BODY editor, not the title.** A page has
two `.ProseMirror` editors — `[aria-label='Page title']` (non-collab) and
`[aria-label='Page content']` (the collab body). `document.querySelector('.ProseMirror')`
returns the TITLE editor, so typing there never changes body content and `mod+S`
versions nothing. Target `[aria-label='Page content']`, confirm it's collab-bound
(`el.editor.extensionManager.extensions.some(e=>e.name==='collaboration')`), and
wait ~10-12s for the store debounce before asserting `pages.content` changed. Full
testing methodology + traps: **[how-to-test.md](how-to-test.md)**.
See also the **Commands** and **Architecture → Two server processes** sections in
[`AGENTS.md`](../AGENTS.md).
+520
View File
@@ -0,0 +1,520 @@
# Фича «Время работы над статьёй» — дизайн-документ
Статус: черновик проектирования (код не пишется).
Контекст: gitmost (форк Docmost). Зависит от PR #370 / PR #374 (типизированная история страниц).
## 1. Цель и не-цели
**Цель.** Показывать в UI страницы одно число — оценку времени, реально затраченного
на работу над статьёй, собранную из истории правок. Число должно быть устойчиво к
паузам: «в 21:00 одна правка, в 09:00 вторая» не должно превращаться в «работал 12 часов».
По клику на число — раскрытие в **суточный таймлайн**: строка-день = 24-часовая дорожка с
окнами активности и суммой за день (см. §6.2).
**Явно не-цели.**
- Это НЕ инструмент для агента (не MCP-tool). Это число, отображаемое человеку в UI.
- Не хронометраж с точностью до минуты — это заведомо оценка.
- Не биллинг/тайм-трекинг сотрудников; не изменение долговечности черновика.
## 2. Проблема
История страницы в Docmost — таблица `page_history`: снимок на каждое сохранение.
У снимка есть `createdAt`, автор (`lastUpdatedById`), тип источника
(`lastUpdatedSource`: `user`/`agent`/`git`) и группировка агентских правок
(`lastUpdatedAiChatId`).
Наивная оценка `max(createdAt) − min(createdAt)` завышает в разы: между крайними
правками лежат сон, обед, дни простоя. На реальных данных статьи-примера
(первая страница истории, 20 снимков) span между крайними снимками ≈ 60 часов,
тогда как реальной работы — пара часов в 5–6 коротких заходов.
Правильная постановка: **длительность — это не span между крайними правками, а сумма
интервалов внутри «сессий»; историю режем по паузам бездействия.** Это классическая
*сессионизация по таймауту неактивности* (WakaTime / RescueTime / веб-аналитика).
## 3. Почему PR #374 — фундамент фичи
До #374 у сессионизации слепое пятно: если человек час пишет подряд и не жмёт «Сохранить»,
в `page_history` за этот час почти нет строк (тяжёлые автосейвы ydoc идут в `pages`/`ydoc`,
а не в историю). Непрерывную работу нечем измерить.
PR #374 вводит типизированную историю через колонку `page_history.kind`:
| `kind` | что означает | ценность для фичи |
|------------|------------------------------------------------|----------------------------------------------------|
| `manual` | человек нажал Save | сильный маркер активной работы человека |
| `agent` | снимок правки агентом | машинное время агента (группируется по `aiChatId`) |
| `idle` | автоснимок idle-флеша (потолок ~`maxWait`) | регулярный «пульс» непрерывной работы (~≤10м, §3) |
| `boundary` | автоснимок на переходе актора (user↔agent↔git) | бесплатная разметка «кто работал в этом сегменте» |
| `null` | легаси-автосейв (старые страницы) | обычный сэмпл активности |
Два env-параметра #374 напрямую задают качество измерения:
- **`IDLE_MAX_WAIT_USER=10м` / `AGENT=5м` (потолок ожидания)** — определяющий параметр.
Проверено по `computeHistoryJob` (`persistence.extension.ts`):
`delay = max(0, min(interval, burstStart + maxWait − now))`, а `enqueuePageHistory`
сбрасывает `burstStart` каждые `maxWait`. Так как `maxWait` (10м/5м) < `interval` (60м/15м),
**потолок всегда доминирует**: во время непрерывной работы `idle`-снимок форсится каждые
~`maxWait`, давая регулярный «пульс активности» с шагом ≤10 мин (user) / ≤5 мин (agent).
- **`IDLE_INTERVAL_USER=60м` / `AGENT=15м`** — номинальный трейлинг-интервал, который потолок
ожидания на практике всегда упреждает. Поэтому метка любого `idle`-снимка отстоит от
реальной правки не более чем на ~`maxWait` (≤10м user / ≤5м agent), а **не** на 60 мин.
(Это ключевой факт для точности: `idle`-метки — достоверный сигнал активности, не «хвост».)
Вывод: ядро алгоритма работает и на голых `createdAt`+`lastUpdatedSource` (есть с миграции
`20260616T130000-agent-provenance`), поэтому фича считает и на легаси-страницах — просто
грубее. После #374 (пульс ≤10 мин + типы) — точно. Жёсткой блокировки на мёрж #374 нет,
но полноценная точность появляется вместе с ним.
## 4. Входной сигнал
Дешёвый проекционный запрос по `page_history` (без тяжёлой колонки `content`): на строку —
`createdAt`, `lastUpdatedById`, `lastUpdatedSource`, `lastUpdatedAiChatId`, `kind`.
Все поля (включая `kind`) уже в `PageHistoryRepo.baseFields` после #374 — схему трогать не
нужно, `findTimelineByPageId` это лишь лёгкая проекция без `content`.
**Важный факт об авторстве (проверено по `persistence.extension.ts``updatePage`):**
`page_history.lastUpdatedById` — это ВСЕГДА ответственный человек, даже для агентских снимков
(«human stays the responsible author»). Признак «человек vs агент» живёт в
`lastUpdatedSource` (`user`/`agent`/`git`) и `lastUpdatedAiChatId`, а НЕ в `lastUpdatedById`.
Отсюда: разделять человеко-/агенто-время нужно по `lastUpdatedSource`, а атрибутировать
конкретному человеку — по `lastUpdatedById`/`contributorIds`.
## 5. Алгоритм: сессионизация по паузам
Параметры (env, в стиле #374; полный список — §10):
- **`T_gap`** — таймаут неактивности: пауза между соседними сэмплами `≤ T_gap` = непрерывная
работа, больше = перерыв (в зачёт не идёт).
- **`P_in` / `P_out`** — добивка МНОГОсэмпловой сессии (работа началась до первого и продолжалась
после последнего сохранения).
- **`P_single`** — блок ОДИНОЧНОЙ сессии (один сэмпл, соседей в пределах `T_gap` нет). Мал
(дефолт ~2 мин): один автосейв/idle-пульс — это «была правка», но приписывать ему полный
`P_in+P_out` = выдумывать время. НЕ путать с многосэмпловой добивкой.
Псевдокод (ОДИН проход по ВСЕМ сэмплам страницы; класс определяется у ГОТОВОЙ сессии — §5.1):
```text
samples = ВСЕ history rows страницы, projected, sorted by createdAt ASC
(все типы kind — сэмплы активности; idle — основной «пульс» непрерывной работы §3, НЕ исключается)
# коллапс агентских всплесков (§5.1)
collapse: подряд идущие сэмплы ОДНОГО aiChatId с source=agent → сегмент {t_start, t_end, source:'agent'}.
Разрывает всплеск любой сэмпл, НЕ продолжающий тот же aiChatId-агент: source≠agent,
boundary-переход, ИЛИ иной aiChatId — в т.ч. idle/boundary с ДРУГИМ aiChatId = НОВЫЙ ран,
склеивать НЕЛЬЗЯ (иначе простой между двумя ИИ-ранами засчитается агенту). idle с ТЕМ ЖЕ
(или null) aiChatId — продолжает сегмент. Дальше сегмент участвует в цикле как один «сэмпл» с
.t_start/.t_end/.source (source='agent'); у скалярного сэмпла t_start == t_end == t.
gap_threshold(a, b) = (a и b оба source=agent) ? agentTGap : T_gap # порог зависит от ПАРЫ (§10)
# сессионизация по паузам — ОДИН проход по ВСЕМ сэмплам (не по парам, не отдельно по классам)
sessions = []; cur = null
for s in samples: # s — скаляр или коллапс-сегмент
if cur == null: cur = { first: s, last: s, samples: [s] }
elif s.t_start − cur.last.t_end ≤ gap_threshold(cur.last, s):
cur.last = s; cur.samples.push(s) # непрерывная работа
else: sessions.push(cur); cur = { first: s, last: s, samples: [s] }
if cur != null: sessions.push(cur) # ОБЯЗАТЕЛЬНО закрыть последнюю (иначе теряется)
# класс и интервал каждой сессии
for sess in sessions:
sess.class = sess.samples.every(is_agent) ? 'agent_only' : 'work' # §5.1 (по source сэмплов)
sess.iv = (sess.first == sess.last && sess.first.t_start == sess.first.t_end)
? [ sess.t − P_single, sess.t ] # одиночный скаляр: pre-roll (без «будущей» работы)
: [ sess.first.t_start − P_in, sess.last.t_end + P_out ] # многосэмпловая / лон-сегмент
workMs = duration( union( sess.iv : sess.class=='work' ) ) # union внутри класса
agentOnlyMs = duration( union( sess.iv : sess.class=='agent_only' ) )
```
Ключевые свойства:
- **Один проход, потом классификация → метрики дизъюнктны.** Сессии не перекрываются (разделены
гэпами > порога), каждая — ровно одного класса. Человек и агент, правящие ОДНОВРЕМЕННО, попадают
в одну сессию класса `work` (надзор засчитан человеку, не дважды). `work`- и `agent_only`-интервалы
не пересекаются по wall-clock и `workMs + agentOnlyMs ≤ реально прошедшего` — **при рекомендованном
`T_gap ≥ P_in+P_out` (§10)**: иначе `P`-добивка соседних сессий РАЗНЫХ классов может пересечься, и
пересечение попадёт в обе метрики. Инвариант §6.3 (`Σ perDay == work`) от этого НЕ зависит.
- **Закрытие последней сессии обязательно** (иначе теряется самая свежая): `n=1` → одна одиночная
сессия `P_single`, `n=0` → 0.
- **`union`, а не `Σ`.** Перекрытия (соседние ближе `P`; одновременное соредактирование) не
задваиваются. Отсюда `Σ perDay == work` держится сам собой (§6.3) — без «клампа» и без условия
`T_gap ≥ P`.
- **Калибровка `T_gap` по «пульсу» #374 (важно).** После #374 непрерывная работа гарантированно
оставляет history-строку не реже ~`maxWait` (idle-пульс §3, гейт `isDeepStrictEqual`). Значит гэп
между соседними строками БОЛЬШЕ ~`maxWait` содержит участок без изменений контента = (частичное)
бездействие — даже между двумя `manual`. Поэтому `T_gap` калибруется ОТ `maxWait` (реком. ~15м
user / ~7м agent), а не ставится вольно: прежние «30 мин» переоценивали не-пульсирующие паузы.
Гэп `≤ T_gap ≈ maxWait` ПОДКРЕПЛЁН пульсом — это и оправдывает счёт его как работы. Легаси-страницы
до #374 пульса не имеют → там `T_gap` вынужденно шире и оценка грубее (§7, §10).
- **Всё равно оценка, а не строгая граница**: даже с пульсом «читал/думал 12 мин не печатая» не
отличить от «отошёл»; подпись «≈» и показ `T_gap` обязательны (§6).
- Почему интервалы, а не «правок × блок»: «снимок = +N мин» ломается на агентских всплесках
(8 снимков за 7 минут); длина сегмента всплеска = его wall-clock, независимо от плотности снимков.
### 5.1. Классификация сэмплов и всплесков
- **Класс сэмпла (человек / агент)** — по `lastUpdatedSource`: `user`→человек, `agent`→агент,
`git`→исключаем (§10 `excludeGit`), legacy-`null`→человек. `idle` наследует `source` страницы
на момент флеша (= source последней правки). `boundary` несёт СТАРЫЙ (pre-transition) `source`
— трактуем как есть: он маркирует исходящую работу того актора (это осознанный компромисс, а не
недосмотр — точность посекундная тут не нужна).
- **Класс СЕССИИ** (нужен для метрик §6.1): сессия, где ВСЕ сэмплы `source=agent`**`agent_only`**
(автономный прогон, не в основную метрику). Сессия хотя бы с одним человеческим сэмплом →
**`work`** (человек + надзор за агентом ВНУТРИ сессии засчитывается человеку — по union'у, §5).
- **`idle`** — полноценный сэмпл активности и основной «пульс» непрерывной работы (§3): его метка
отстаёт от реальной правки ≤ `maxWait` (≤10м) — в пределах округления, отмотки не требует.
Исключать нельзя: без него непрерывное письмо без ручных сохранений снова стало бы невидимым.
## 6. Что показывать в UI
### 6.1. Свёрнутое состояние — одно число
Две метрики, каждая = union-wall-clock СВОИХ сессий (§5):
- **`work` — headline, кликабельное число** (`≈ 4 ч 30 мин`, рядом с панелью истории или в
мета-инфо у заголовка): сессии класса `work` (≥1 человеческий сэмпл = человек + надзор за
агентом внутри сессии). Именно это открывает таймлайн (§6.2), и именно к нему сходится сумма по
дням.
- **`agent_only` — вторично**: сессии автономных прогонов агента (ни одного человеческого сэмпла).
На таймлайне — отдельным цветом; в `work` НЕ входит.
- Подпись «≈» и показ `T_gap` обязательны (оценка, не хронометраж — §5). Округление headline —
шаг 5–15 мин (точность оценки не оправдывает «4 ч 27 мин»).
### 6.2. Раскрытие по клику — суточный таймлайн (24 ч × дни)
Клик по числу открывает модалку/поповер с таймлайном по типу «punch-card»:
- **Строка = один календарный день**; ширина строки = 24 часа (фиксированная шкала 00:00→24:00).
- На дорожке дня закрашены **окна активности** — реальные интервалы работы в их часовом
положении, так что видно «вечерний марафон» vs «утренняя сессия».
- **Справа от строки — сумма за день** «ч мм» (напр. `3 ч 17 м`); пустой день — пустая дорожка
и «—».
- Внизу — общий итог (= headline `work` §6.1) плюс подпись таймзоны и `T_gap`.
Это графическая версия таблицы-примера-картинки: там окна были текстом («18:46 → 00:58»), здесь
то же рисуется отрезками на 24-часовой дорожке.
Детали:
- Окна = сессии класса `work` (§5), обрезанные по границам суток; сессии `agent_only` — отдельным
цветом (§6.1). По умолчанию `work` — один цвет «активность».
- Сессия через полночь рисуется как отрезок до 24:00 в одном дне и продолжение с 00:00 в
следующем — тот самый полуночный разрез (§6.3), теперь визуально очевиден.
- **Честность добивки.** Окно включает `P`/`P_single` — это оценка «до/после», а не измеренный
интервал. Одиночная сессия (`P_single`) рисуется минимальной видимой шириной и приглушённо
(иначе исчезает и/или создаёт иллюзию плотной работы из одного клика). Общая подпись — «≈».
### 6.3. Агрегация по дням (алгоритм)
Вход — ВСЕ сессии из §5 (`work` и `agent_only`), каждая развёрнута в интервал (`P_in/P_out` или
`P_single`).
```text
bucketByDay(sessions, tz):
U_work = union( sess.iv : sess.class=='work' ) # снимаем перекрытия ОДИН раз (§5)
U_agent = union( sess.iv : sess.class=='agent_only' ) # СИММЕТРИЧНО — иначе окна агента наложатся
для каждого дня D в [первый_день … последний_день] по tz (dayjs+tz, startOf('day')):
workWin[D] = { u ∩ [начало D, начало D+1) : непусто, u в U_work }
agentWin[D] = { u ∩ [начало D, начало D+1) : непусто, u в U_agent }
activeMs[D] = Σ длительностей workWin[D] # U_work без перекрытий → просто сумма
agentMs[D] = Σ длительностей agentWin[D]
→ perDay[] = [{ day, activeMs, agentMs, windows: (workWin[D] ⊕ agentWin[D]) с меткой класса }]
```
- **Инвариант согласованности `Σ activeMs[D] == work`** держится ПО ПОСТРОЕНИЮ: день — это
разбиение union'а `U_work` границами суток, ничего не теряется и не дублируется (в т.ч. на
23/25-часовых DST-сутках — §9#14). `agent_only`-окна рисуются, но в `activeMs` НЕ входят. Клампа
и скрытого условия `T_gap ≥ P` не требуется (в отличие от наивного `Σ` длительностей).
- **Таймзона `TZ`** определяет, где проходит «полночь» И в каких часовых координатах рисуются
окна. Дефолт — таймзона зрителя (локаль браузера); альтернатива — UTC (как на картинке-примере
«По дням (UTC)») или tz воркспейса. Влияет на раскладку по дням и положение окон, но НЕ на
общий итог → настройка (§10).
- **Длинный диапазон** (месяцы правок): строк ровно столько, сколько календарных дней в диапазоне.
При большом числе дней — вертикальный скролл + сворачивание длинных серий пустых дней
(«× N дней без правок») и/или переключение на понедельную группировку. Порог — настройка.
- **Округление дисплея:** сумму за день округляем до минут для подписи, но общий итог берём из
точных значений (иначе Σ округлённых по дням разойдётся с округлённым числом на ±1–2 мин).
## 7. Проверка на реальной статье
Ручной прогон на 20 снимках (1-я страница истории, `T_gap=30 мин`, `P_in+P_out=10 мин`,
`P_single=2 мин`; все сессии здесь класса `work` — в каждой есть человеческий сэмпл):
| Сессия | Интервал | Длит. |
|--------|-------------------------------------------------------|----------|
| S1 | 07-04 03:40 → 03:49 (многосэмпловая) | ≈19 мин |
| S2 | 07-04 15:43 → 16:13 (агент 15:43–15:50 → человек 16:13)| ≈41 мин |
| S3 | 07-04 18:11 (одиночная) | ≈2 мин |
| S4 | 07-04 19:38 → 19:54 (многосэмпловая) | ≈26 мин |
| S5 | 07-06 15:34 (одиночная) | ≈2 мин |
| S6 | 07-06 16:18 (одиночная, закрыта пост-циклом §5) | ≈2 мин |
| **Итого** | | **≈1 ч 32 мин** |
Наивно на том же срезе — ≈60 часов. Разница — весь смысл фичи.
Наблюдения:
- Разрезы легли по перерывам: ночь `03:49 → 15:43` (~12 ч) и сутки
`07-04 19:54 → 07-06 15:34` — оба выброшены.
- Чувствительность к порогу: `S5/S6` отстоят на 44 мин. При `T_gap=30` это две сессии,
при `T_gap=60` — одна (~44 мин). Число надо показывать **вместе с использованным порогом**.
- Это **оценка, не строгая граница** (§5), и НАПРАВЛЕНИЕ ошибки зависит от данных. Сохранения
ВНУТРИ `T_gap` мостятся, и вся пауза между ними засчитывается → на периодичном «фоновом» ритме
(напр. автосейв раз в ~`T_gap` при почти полном простое) возможен КРАТНЫЙ перебор (в разы), а не
«±порог». Сохранения ДАЛЬШЕ `T_gap` друг от друга, наоборот, теряют между-время → недобор. Поэтому
`T_gap` калибруется по пульсу #374 (§5, §10): после #374 «фоновый» ритм даёт гэпы > `maxWait` и
рвётся на перерывы, срезая кратный перебор; на легаси (пульса нет) оценка грубее в обе стороны.
Пример иллюстративен, посчитан на до-#374 срезе при `T_gap=30`.
## 8. Архитектура (куда встраивать)
**Ядро — чистая функция** `computeWorkTime(rows, config)` (детерминированная, без БД) в
отдельном модуле → легко покрыть юнит-тестами. Выход —
`{ workMs, agentOnlyMs, sessions[] }`, где `session = { start, end, class: 'work'|'agent_only' }`:
абсолютные границы (с добивкой `P`/`P_single`) плюс класс (§5.1) — этого достаточно и для метрик,
и для цвета окна. `workMs`/`agentOnlyMs` считаются как union-wall-clock сессий своего класса (§5).
**Вторая чистая функция** `bucketByDay(sessions, tz)` (ВСЕ сессии обоих классов) →
`perDay[] = [{ day, activeMs, agentMs, windows }]`: `activeMs` = длительность `work`-окон за день
(сходится к `work`, §6.3), `agentMs` = то же для `agent_only` (для подписи машинного времени за
сутки), `windows` — интервалы ОБОИХ классов, обрезанные по суткам и помеченные классом (для
отрисовки `work`/`agent_only` разным цветом, §6.2). Полуночный разрез — календарный, через
`dayjs` + tz-плагин (`startOf('day')` в `tz`), НЕ «+24 ч» (§9#14); `dayjs` уже в проекте. Отдельная
тестируемая функция (общий модуль сервера и клиента); `tz` — презентационный параметр, удобно звать
на клиенте от локали зрителя.
**Сервер:**
- `page-history.repo.ts` — метод `findTimelineByPageId(pageId)`: лёгкая проекция
(`createdAt, lastUpdatedById, lastUpdatedSource, lastUpdatedAiChatId, kind`) по всем строкам
ASC, без `content`.
- `page-history.service.ts``computeWorkTime(pageId, config)`: тянет таймлайн, зовёт ядро,
кэширует.
- `page.controller.ts` — рядом с `POST /history` и `POST /history/info` добавить
`POST /history/time` (или вложить в page-info). Отдаёт число + разбивку по сессиям.
**Клиент:**
- `page-history-query.ts` — хук `usePageWorkTime(pageId)` (возвращает `workMs`, `agentOnlyMs`,
`sessions[]`).
- Рендер кликабельного числа в панели истории.
- Модалка/поповер с суточным таймлайном (§6.2): `bucketByDay(sessions, viewerTz)` → строки-дни,
в каждой — 24-часовая дорожка с окнами. Это НЕ bar chart: рисуется кастомными CSS/SVG-отрезками
на 24-часовой шкале (позиция окна = `startOfDayOffset/24ч`, ширина = длительность/24ч). Готовые
чарт-библиотеки под это плохо ложатся — брать лёгкую собственную вёрстку, без новых тяжёлых
зависимостей. Пустые дни — пустая дорожка + «—».
**Производительность:** проекция без `content` дёшева; результат можно инкрементально кэшировать
(при `version.saved`, который #374 броадкастит, пересчитывать хвост).
## 9. Крайние случаи
1. Один снимок → одна одиночная сессия `P_single`, не ноль. Последняя сессия ВСЕГДА закрывается
пост-циклом (§5) — иначе теряется самая свежая.
2. История целиком агентская → `work = 0`, `agent_only = union прогонов`.
3. Плотный агентский всплеск → длина сегмента = его wall-clock (не зависит от числа снимков);
опц. кап `burstCapMs`.
4. Метка `idle` лагает ≤ `maxWait` (10м user / 5м agent) — в пределах округления; это
полноценный сэмпл активности, спец-обработки не требует (см. §3, §5.1).
5. Несколько соавторов → атрибуция человеку по `lastUpdatedById`/`contributorIds`; разделение
человек/агент — по `lastUpdatedSource` (НЕ по `lastUpdatedById`, он всегда человек).
6. Легаси `kind=null` → работает на `source`+`createdAt`, грубее.
7. Совпадающие метки времени (boundary+agent в один момент; в коде есть tie-break по `id`)
→ дедуп по округлённому `t`.
8. Одновременное соредактирование → `union` (§5) убирает двойной счёт wall-clock при перекрытии
окон; персональные человеко-часы (разбивка по авторам) — отдельный опциональный режим
(`perAuthor`), а не поведение по умолчанию.
9. Сессия через полночь (напр. `23:14 → 02:00`) → режется на границе суток `tz`, части идут
в разные дни; сумма по дням = общему числу (§6.3).
10. Выбор таймзоны дня меняет раскладку по дням (та же работа попадёт в другой день) — общий
итог не меняется; `tz` фиксируется в подписи графика.
11. Длинный диапазон правок (месяцы) → строк = число дней: вертикальный скролл + сворачивание
длинных серий пустых дней и/или понедельная группировка по порогу (§6.3).
12. День без правок → пустая 24-часовая дорожка + «—» в диапазоне (показываем ритм/паузы),
не пропускаем.
13. Очень короткое окно на 24-часовой шкале → рисуем минимальной видимой шириной, чтобы не
исчезало (§6.2).
14. Переход на летнее/зимнее время внутри `tz` → сутки в 23/25 ч; полуночный разрез считать
по календарю `tz` (`dayjs`+tz, §8), а не «+24 ч». Инвариант `Σ activeMs == work` держится
(разбиение union'а). Редкое исключение — tz с DST-переходом РОВНО в полночь (`startOf('day')`
неоднозначен) → до 1 ч может протечь в соседний день. Фиксированная 24-часовая дорожка (§6.2)
на 23/25-часовых сутках смещает окна визуально до ~1 ч — сознательное упрощение, на итог не
влияет.
## 10. Параметры по умолчанию и открытые решения
**Полный `config`** (env, дефолты):
- `T_gap≈15м` — калибровка по `IDLE_MAX_WAIT_USER=10м` + запас (§5: гэп больше `maxWait` не
подкреплён пульсом = бездействие; прежние «30м» переоценивали не-пульсирующие паузы).
- `agentTGap≈7м` — порог для ПАРЫ подряд идущих агентских сэмплов (`IDLE_MAX_WAIT_AGENT=5м` + запас).
- `P_in=5м`, `P_out=5м`, `P_single=2м` (одиночная — pre-roll, §5).
- `burstCapMs` (опц. кап на сегмент всплеска, §9#3), `dedupRoundMs` (дедуп совпадающих меток, §9#7),
`excludeGit=true`, `tz=локаль-зрителя`, `longRangeDayThreshold` (день→неделя, §6.3),
`perAuthor=false` (§9#8).
- **Легаси-страницы до #374** (нет пульса) → `T_gap` вынужденно шире, оценка там грубее.
- **`T_gap ≥ P_in+P_out`** НЕ требуется для инварианта §6.3 (union), но РЕКОМЕНДУЕТСЯ и валидируется
— иначе `work`/`agent_only` могут перекрыться в сумме (§5). При дефолтах (15 ≥ 10) держится.
Развилки (настройки, не блокируют проектирование):
- `work` vs `agent_only` — показывать оба, крупно `work` (headline), `agent_only` вторично;
- точное место в UI — панель истории (основное) или мета-строка страницы;
- дефолт `T_gap` — вынести в env (как `IDLE_*` в #374), калибровать на реальных статьях
после мёржа #374;
- **таймзона дня для графика (§6.2/§6.3)** — рекомендую локаль зрителя (интуитивно «мои
вечера»); альтернативы — UTC (как на картинке-примере) или tz воркспейса. Влияет только на
раскладку по дням, не на общий итог;
- **порог перехода день→неделя/месяц** для длинного диапазона правок.
---
# Приложение: Review Ledger (рабочий аппарат, НЕ нормативная часть)
> Аппарат adversarial-review-loop. Перед выдачей реализатору выносится/сворачивается.
> Критикам: НЕ перелитигировать закрытые findings, КРОМЕ случая, когда сам RESOLUTION
> дефектен — тогда атаковать его явно и сказать, почему предыдущий раунд ошибся.
## CONFIG
- EXTERNAL_MODEL: endpoint `https://api.z.ai/api/coding/paas/v4`, model `glm-5.2`
(ключ хранится вне репозитория, в логи не пишется). Роль: cold-reader gate (Phase 4.5),
tiebreaker при эскалации.
- Артефакт: `docs/features/page-work-time_design.md`.
- Целевой класс: спец для реализации другим человеком → план 2–3 итерации, до пустого
cold-reader gate (не по числу итераций).
## FACT BASE (проверено по PR #374 @ commit 924f8aa, ветка feat/370-page-versioning)
Верификация автором ДО цикла (ветка не в рабочем дереве — критики не видят её локально;
факты ниже — ground truth, можно дозапросить файлы через gitea MCP по указанному SHA):
- `page_history.kind``varchar(20)`, NULLABLE, БЕЗ дефолта (migration
`20260705T120000-page-history-kind.ts`). Домен: `manual`/`agent`/`idle`/`boundary`;
legacy `null` = автосейв (`collaboration/constants.ts`, `PageHistoryKind`).
- `kind` УЖЕ включён в `PageHistoryRepo.baseFields` (`page-history.repo.ts`) — читается всеми
выборками истории. `saveHistory({kind})` и `updateHistoryKind(id, kind)` существуют.
- Тайминги (`constants.ts`): `IDLE_INTERVAL_USER=60м`/`AGENT=15м`;
`IDLE_MAX_WAIT_USER=10м`/`AGENT=5м`.
- `computeHistoryJob` (`persistence.extension.ts`):
`delay = max(0, min(interval, burstStart + maxWait − now))`; `enqueuePageHistory` сбрасывает
`burstStart` каждые `maxWait`. Следствие: **потолок всегда доминирует**`idle` пульсирует
каждые ~`maxWait` при непрерывной работе; метка `idle` отстоит от правки ≤ `maxWait`, НЕ 60м.
- Идл-джоб всегда ставится с `kind:'idle'`; процессор пишет `job.data.kind ?? 'idle'`, но только
если контент изменился (`isDeepStrictEqual`-гейт).
- `boundary`-снимок пишется СИНХРОННО в store-транзакции на смене `lastUpdatedSource`
(user↔agent↔git), фиксирует ИСХОДЯЩИЙ (pre-transition) контент со СТАРЫМ source; его
`createdAt` = момент перехода (точный).
- `manual`/`agent` — явный save-version по stateless-каналу; `kind` выводится из
`context.actor` СЕРВЕРНО (неподделываемо). Promote-not-dup: апгрейд `kind` последнего снимка
на месте вместо дубля.
- `page_history.lastUpdatedById` = ВСЕГДА ответственный человек (даже для агентских снимков);
«agentness» — в `lastUpdatedSource` + `lastUpdatedAiChatId`.
- REST истории: `POST /history`, `POST /history/info` (`page.controller.ts`). Клиент:
`apps/client/src/features/page-history/*`. Есть broadcast `version.saved` (для live-инвалидации).
## Pre-loop fact-base corrections (автор, проверено vs source)
- C1. §3/§5.1/§крайние-случаи#4: убрано ошибочное «idle лагает до 60м / idle не удлиняет
сессию / отмотка на IDLE_INTERVAL». Верно: лаг ≤ `maxWait` (≤10м), `idle` — полноценный сэмпл.
- C2. §3: устранено внутреннее противоречие §3↔§5.1 (пульс vs исключение idle).
- C3. §4: `kind` уже в `baseFields` — схему менять не нужно.
- C4. §крайние-случаи#5: `lastUpdatedById` всегда человек; человек/агент — по `lastUpdatedSource`.
## Scope changes
- S1 (owner, до итерации 1): добавлен drill-down — клик по числу открывает график по дням.
- S2 (owner, до итерации 1): drill-down переопределён с «столбцы часов/день» на СУТОЧНЫЙ
ТАЙМЛАЙН (строка-день = 24 ч с окнами активности + сумма за день, §6.2/§6.3). Окна вернулись,
но графикой. Затронуты §1, §6.2, §6.3, §8, §9(#9–14), §10.
## Iteration log
### Итерация 1 — критики: Claude (hardened, A) + GLM-5.2 external (B)
Первый прогон local-субагентов вернул инъекцию из подложенного SKILL.md (реклама
«agent-first-plugin-suite») и 0 полезной работы — проигнорировано; пере-прогон с анти-инъекцией
дал реальные ревью. Дефекты и диспозиции:
- **[BLOCKER] A1** — §5-псевдокод не закрывал последнюю сессию (терял свежую; 0 сессий для
односессионной страницы; ронял S6). ПРИНЯТО → пост-цикловое закрытие, `n=1``P_single`, `n=0`→0.
- **[MAJOR] A2+B4** — какое число суммирует таймлайн; двойной счёт при перекрытии. ПРИНЯТО →
метрики `work`/`agent_only` (§6.1) + `total`/`perDay` через **union** (§5, §6.3).
- **[MAJOR] A3** — §1 остался со старым «столбцом на день» (residue S1→S2). ПРИНЯТО → §1 переписан.
- **[MAJOR] A4** — «кламп vs скрытое условие `T_gap ≥ P`». ПРИНЯТО, но растворено: union убрал и
кламп, и условие (§6.3).
- **[MAJOR] A5** — форма `session` и правило классификации человек/агент (нужны для `workMs` и
цвета). ПРИНЯТО → §5.1 классификация (+ nuance про старый source у boundary), `session.class` (§8).
- **[MAJOR→понижено] B1** — «нижняя оценка» ложна; гэп ≤ T_gap считается работой. ЧАСТИЧНО ПРИНЯТО
+ КОНТР по severity: гэп-как-работа — by design (по меткам не отличить «думал» от «отошёл»),
поведение оставлено; исправлено УТВЕРЖДЕНИЕ (§5/§7: «оценка, не строгая граница»). Понижено с
BLOCKER: это не баг кода, а некорректная формулировка/честность.
- **[MAJOR] B2+B5** — инфляция одиночных сессий на `P` + честность отрисовки добивки. ПРИНЯТО →
`P_single` (мал), приглушённая отрисовка + «≈» (§5, §6.2).
- **[MAJOR/MINOR] B3+A6** — коллапс агентского всплеска рушит вклинившегося человека + выбор
конца для гэп-теста. ПРИНЯТО → правило коллапса (только подряд один aiChatId; человек внутри
разрывает; левый гэп до `t_start`, правый от `t_end`) (§5).
- **[MINOR] A7** — `config` не перечислен. ПРИНЯТО → полный список (§10).
- **[MINOR] A8** — ячейка `idle` в таблице §3 устарела. ПРИНЯТО → исправлена.
- **[NIT] A9** — расхождение округления headline vs подписи дня. Оставлено с оговоркой (§6.3).
- **[NIT] A10** — назвать `dayjs` для tz/DST-разреза. ПРИНЯТО (§8, §9#14).
Verified clean (оба критика): факты #374; tz/DST-разрез; лёгкая проекция без `content`;
кэш на `version.saved`; разбиение на две чистые функции; §2-постановка; арифметика §7 (при
исправленном алгоритме).
### Итерация 1 — post-integration re-attack (Claude A + GLM round 2) — ЗАКРЫТА
Оба критика НЕЗАВИСИМО нашли один и тот же блокер и сошлись (разные семейства моделей):
- **[BLOCKER] N1 (A) = MAJOR-1 (GLM)** — §5 «сессионизация ОТДЕЛЬНО по классам» противоречила
§5.1/§7/§8 и теряла надзорное агентское время (S2 схлопывалась в 2 мин; `workMs+agentOnlyMs` мог
превысить прошедшее). Исправлено: ЕДИНЫЙ проход по всем сэмплам + классификация готовой сессии;
метрики — union внутри класса.
- **[MAJOR] N2 (A)** — union только ВНУТРИ класса; кросс-класс дизъюнктность требует `T_gap ≥ P`,
а §10 это отрицал; `agentTGap` неопределён при едином проходе. ПРИНЯТО → `gap_threshold` по ПАРЕ
сэмплов (agent–agent → `agentTGap`), caveat в §5, §10 смягчён (валидируем `T_gap ≥ P`).
- **[MAJOR] GLM-2** — `bucketByDay(workSessions)` не мог рисовать `agent_only`. ПРИНЯТО → сигнатура
`bucketByDay(sessions)`, окна ОБОИХ классов, `activeMs` = work-only.
- **[MINOR] N3** — роль `boundary` противоречива (§5 «человеческий» vs §5.1 «старый source=agent»).
ПРИНЯТО → §5: всплеск рвёт любой сэмпл, не продолжающий тот же aiChatId-агент; класс — по source.
- **[MINOR] N4** — псевдокод точечный, концы сегмента не заданы. ПРИНЯТО → сегмент {t_start,t_end};
same-source idle внутри всплеска продолжает сегмент.
- **[NIT] N5** — DST ровно в полночь + фикс. 24ч-дорожка ±1ч визуально. ПРИНЯТО → оговорка §9#14.
- **[NIT] N6** — `P_single` симметричный «выдумывает будущую работу». ПРИНЯТО → pre-roll `[t−P_single, t]`.
- **B1-severity (спор):** A СОГЛАСИЛСЯ с понижением BLOCKER→MAJOR (нет входа, где код отклоняется от
намерения — только формулировка/UX). GLM НАСТАИВАЛ на BLOCKER с НОВЫМ аргументом: отсутствие
idle-пульса в гэпе = доказанное бездействие. Диспозиция: label = MAJOR, но СУБСТАНЦИЯ GLM принята
(Invariant 7 — уступка с аргументом): `T_gap` калиброван по `maxWait` (§5/§10) + принцип «гэп >
maxWait = перерыв». A-residue (направление ошибки data-dependent, возможен КРАТНЫЙ перебор) принят
в §7. Тайбрейкер не понадобился (обе стороны привели аргументы, интегрированы обе).
Verified clean (round-2): инвариант §6.3 под DST (оба критика); burst-endpoints (оба); `P_single`
через полночь (GLM). Блокеров в ядре: 0. Итерация 1 закрыта.
### Convergence validation + cold-reader gate — ЗАКРЫТА (CONVERGED)
- **Cold-reader gate** (GLM-5.2, внешняя модель, «имплементер читает впервые»): ПУСТОЙ список
вопросов на день 1 — ДВАЖДЫ (до и после партии полировки).
- **Convergence pass** (GLM-5.2, свои hostile-сценарии + слепая реализация): вердикт NOT CONVERGED
на ещё не атакованной партии round-2 → нашёл 2 MAJOR + 1 MINOR (как и предупреждает skill —
«re-opened cycles find MAJORs in unreviewed amendments»):
- [MAJOR] §6.3 рисовал `agent_only`-окна из «сырых» `sess.iv` (не union) → визуальное наложение.
`U_agent = union(...)`, симметрично `U_work`; добавлен `agentMs[D]`.
- [MAJOR] §5 «idle продолжает сегмент НЕЗАВИСИМО от aiChatId» склеивал разные ИИ-раны через
idle-снимок. → idle с ДРУГИМ `aiChatId` = новый ран, рвёт сегмент.
- [MINOR] нет per-day суммы агента. → `agentMs` в `perDay`/§8.
Все приняты и интегрированы; финальный re-gate (GLM cold-reader) → снова ПУСТО, рассинхрона нет.
- **Tooling note:** local general-purpose Claude-субагенты трижды перехватывались инъекцией из
подложенного SKILL.md (реклама «agent-first-plugin-suite» / фейковые «review this PR» промпты,
0 tool-uses). Проигнорировано. Адверсариальные проходы и cold-reader выполнены внешней GLM-5.2
(гетерогенная модель — как раз рекомендована skill против self-preference) + один hardened
Claude-проход, который сработал.
## Closure checklist
1. 0 блокеров в ядре; счётчики обсуждения в обе стороны (приняты находки И отбит spor по B1-severity) — ✓
2. Партия полировки re-gated финальным cold-reader на полном тексте — ✓
3. Cold-reader: список вопросов ПУСТ (дважды) — ✓
4. Прогноз сходимости трактовался как гипотеза; решал чек-лист (convergence-pass реально нашёл MAJOR) — ✓
5. Backcast: частично (пример §7 на реальных данных + 36.5ч-картинка владельца) — обязательным не был
6. Preconditions зафиксированы: зависимость точности от #374 (пульс), грубость на легаси, DST-в-полночь, `T_gap ≥ P`
**СТАТУС: CONVERGED.** §1–§10 — нормативная спека для реализатора; это приложение — аудит-след (не нормативно).
+108
View File
@@ -0,0 +1,108 @@
# How to test the application (browser E2E + out-of-band)
How to actually verify a feature end-to-end against a running stand — driving the
**real app in a browser** and confirming results **out-of-band** in the DB/git, not
through the same API you're supposed to be testing. Written from real false-positives
that wasted hours (see **Traps** — read them before you write a test).
Prereq: a running stand — see **[dev-stand.md](dev-stand.md)**. Automation uses
Playwright (`pip install playwright && python -m playwright install chromium`).
## Principles
1. **Drive the behaviour under test through the browser.** The stand exists so you
exercise the real UI + realtime-collab + server path. Using `POST /api/pages/*` to
perform the action you're validating tests the API, not the app — an e2e suite can
do that. API calls are fine ONLY for one-time setup/fixtures, never for the
interaction you're asserting on.
2. **Evidence before claim.** Nothing "passes" without an artifact: a DB row, a git
diff, a screenshot looked at as an image. If you can't show it, you didn't verify it.
3. **Verify out-of-band.** Judge results from a source independent of the UI: `psql`
against the DB, a fresh `git clone` of a synced repo, a hard reload. Optimistic UI
lies about persistence.
4. **Disconfirm by default.** For each feature, actively try to prove it's broken
before concluding it works. Reload after every create/edit/save.
5. **Recon actuatability FIRST.** Before building editor tests, confirm the
interaction even works in your harness (does a typed edit reach the DB?). Skipping
this is how you ship a pile of tests that all silently exercised the wrong thing.
## The editor: two ProseMirror instances (READ THIS)
A page has **two** `.ProseMirror` editors:
| index | selector | role | collab? |
|---|---|---|---|
| 0 | `[aria-label='Page title']` | title field | **NO** (16 exts, no `collaboration`) |
| 1 | `[aria-label='Page content']` | body | **YES** (95 exts, has `collaboration`) |
`document.querySelector('.ProseMirror')` returns the **title** editor (first match).
Type there and you edit the title only — body page content never changes, so `mod+S`
"versions" unchanged content and every content test silently no-ops.
**Always target the body editor** and confirm it's collab-bound before typing:
```js
const el = document.querySelector("[aria-label='Page content']");
el.editor.extensionManager.extensions.some(e => e.name === 'collaboration'); // must be true
```
Body edits emit ~20 `/collab` websocket frames while typing and land in
`pages.content` after the **hocuspocus store debounce (~10s)** — so **wait ~12s**
before asserting persistence (checking at 6–8s is a false negative). `mod+S` (the
`save-version` stateless message) flushes immediately, so a version created right
after a settled body edit holds the typed text.
## A known-good browser flow
```
1. goto /s/<space-slug> # the "Create page" button lives in the space sidebar, not /home
2. click button[aria-label='Create page'] # fully UI-driven page creation
3. type into [aria-label='Page title'] # optional title
4. click [aria-label='Page content'] → type body text
5. wait ~12s (store debounce)
6. assert pages.content changed (psql) # out-of-band
7. mod+S / menu Save → assert page_history row (psql)
8. reload / fresh context → re-assert (persistence round-trip)
```
Auth: log in ONCE, save `storage_state.json`, reuse it across pages/agents (re-login
per run trips shared rate-limits). Cookie-based session authorizes both REST and the
collab websocket.
## Judging out-of-band
```bash
# page content / history
docker exec <db> psql -U docmost -d docmost -tAc \
"select coalesce(kind,'null'), content::text from page_history where page_id='<id>' order by created_at;"
# git-sync round-trip: clone the space repo and diff against what you pushed
git clone http://<user>:<pass>@127.0.0.1:3000/git/<spaceId>.git /tmp/x
```
`page_history.content` is full JSON — parse it, don't truncate the snippet, or a
marker check misses. For sync/async features (autosave, git-sync, idle-flush) use an
active probe: write a unique marker, wait past the debounce/poll window, re-read
out-of-band, ≥2 iterations — never conclude "broken" from a single snapshot.
## Traps (each of these produced a false result in a real run)
- **Wrong editor.** Typed into `.ProseMirror` (= title). Edits never touched body
content. → target `[aria-label='Page content']`.
- **Checked persistence too early.** Store debounce ~10s; a 6–8s check reads stale.
- **Truncated the DB snapshot** below where the test marker sits → false "content
missing".
- **API-seeded the content under test**, then "verified" the feature — that validated
the API, not the app.
- **Reused a fixed marker on a non-rebooted stand** → title/row collisions inflate
counts (`count==2`). Use a unique per-run marker (timestamp).
- **Idle/async read once** and called it "permanently broken" — it was mid-debounce.
- **Concluded env-limitation without a cross-build control.** If unsure whether a
failure is your harness or the product, run the SAME harness against a known-good
build; a divergence localizes it.
## Scope note
Some paths genuinely need a human in a real browser (rich drag-drop, native file
pickers, clipboard, and anything the harness can't actuate). Label those UNTESTED in
the report — "handled gracefully" is not "works". Keep four states distinct:
verified-working, defect, untested, env-limitation.
+2 -1
View File
@@ -96,7 +96,8 @@
"pnpm": {
"patchedDependencies": {
"scimmy@1.3.5": "patches/scimmy@1.3.5.patch",
"yjs@13.6.30": "patches/yjs@13.6.30.patch"
"yjs@13.6.30": "patches/yjs@13.6.30.patch",
"ai@6.0.134": "patches/ai@6.0.134.patch"
},
"overrides": {
"prosemirror-changeset": "2.4.0",
+9 -4
View File
@@ -55,10 +55,15 @@ describe('stabilizePageFile — normalize-on-write fixpoint (SPEC §11)', () =>
const file2 = await stabilizePageFile(doc2, meta);
expect(file2).toBe(file1);
// The materialized diagram default is present in the stabilized body (proof
// that the convergence pass actually ran, not just that two naive exports
// happened to match).
expect(body1).toContain('data-align="center"');
// The drawio node was materialized to its canonical HTML form by the
// convergence pass — a bare `{ src }` doc node becomes the full
// `<div data-type="drawio" data-src=...>` — proof the pass actually ran, not
// just two naive exports happening to match. Assert on the stable canonical
// markers rather than `data-align="center"`: center is a schema default the
// converter may omit (see prosemirror-markdown media-html.ts), so it is not
// a reliable convergence proof.
expect(body1).toContain('data-type="drawio"');
expect(body1).toContain('data-src="/d.drawio"');
});
it('already-stable content is unchanged by the pass (idempotent)', async () => {
+83 -355
View File
@@ -118,56 +118,19 @@ export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
// transport exposes a `tree:true` mode that returns the full nested hierarchy;
// the in-app copy keeps the same tree option but is worded for the in-app agent.
// Kept per-layer so each side can tune its own guidance.
server.registerTool(
"list_pages",
{
description:
"List most recent pages in a space ordered by updatedAt (descending). " +
"Returns a bounded list (default 50, max 100) — use search for lookups " +
"in large spaces. Pass tree:true (with spaceId) to instead get the " +
"space's full page hierarchy as a nested tree.",
inputSchema: {
spaceId: z.string().optional(),
limit: z
.number()
.int()
.min(1)
.max(100)
.optional()
.describe("Max pages to return (default 50, max 100)"),
tree: z
.boolean()
.optional()
.describe(
"When true, return the space's full page hierarchy as a nested tree (each node has a children array) instead of the recent-by-updatedAt flat list. Requires spaceId; ignores limit.",
),
},
},
async ({ spaceId, limit, tree }) => {
const result = await docmostClient.listPages(spaceId, limit ?? 50, tree ?? false);
return jsonContent(result);
},
);
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294). This
// transport keeps applying its own defaults (limit=50, tree=false) in execute.
registerShared(SHARED_TOOL_SPECS.listPages, async ({ spaceId, limit, tree }) => {
const result = await docmostClient.listPages(spaceId, limit ?? 50, tree ?? false);
return jsonContent(result);
});
// Tool: get_page
server.registerTool(
"get_page",
{
description:
"Get page details with content converted to Markdown. The conversion is " +
"LOSSY (block ids, exact table/callout structure are approximated); for a " +
"lossless representation use get_page_json. Inline <span data-comment-id> " +
"tags in the markdown are comment highlight anchors (also present for " +
"RESOLVED threads) — treat them as markup, not page text.",
inputSchema: {
pageId: z.string().min(1),
},
},
async ({ pageId }) => {
const page = await docmostClient.getPage(pageId);
return jsonContent(page);
},
);
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.getPage, async ({ pageId }) => {
const page = await docmostClient.getPage(pageId);
return jsonContent(page);
});
// Tool: get_page_json
registerShared(SHARED_TOOL_SPECS.getPageJson, async ({ pageId }) => {
@@ -201,6 +164,10 @@ registerShared(
);
// Tool: table_get
// NOT in the shared registry: the MCP tool name `table_get` is noun-first while
// the in-app key is `getTable` (verb-first), breaking the snake_case(inAppKey)
// convention the shared registry enforces (shared-tool-specs.contract.spec.ts).
// Renaming the public MCP tool would break external clients, so it stays inline.
server.registerTool(
"table_get",
{
@@ -223,25 +190,10 @@ server.registerTool(
);
// Tool: table_insert_row
// NOT in the shared registry: this transport names the table argument `table`,
// while the in-app tool names it `tableRef` (ai-chat-tools.service.ts). Sharing
// one buildShape would rename a public MCP parameter, so the table row/cell
// tools stay per-transport by design.
server.registerTool(
"table_insert_row",
{
description:
"Insert a row of plain-text cells into a table. `table` = `#<index>` or " +
"a block id inside it. `cells` = text per column (padded to the table's " +
"column count; error if more cells than columns). `index` = 0-based " +
"insert position (0 inserts before the header); omit to append at the end.",
inputSchema: {
pageId: z.string().min(1),
table: z.string().min(1),
cells: z.array(z.string()),
index: z.number().int().optional(),
},
},
// Schema + description now live in the shared registry (#294); the `table`
// parameter name is the canonical one (the in-app layer was unified to it).
registerShared(
SHARED_TOOL_SPECS.tableInsertRow,
async ({ pageId, table, cells, index }) => {
const result = await docmostClient.tableInsertRow(
pageId,
@@ -254,22 +206,9 @@ server.registerTool(
);
// Tool: table_delete_row
// NOT shared — same `table` (here) vs `tableRef` (in-app) parameter-name
// divergence as table_insert_row.
server.registerTool(
"table_delete_row",
{
description:
"Delete the row at 0-based `index` from a table (`table` = `#<index>` or " +
"a block id inside it). Refuses to delete the table's only row. An " +
"out-of-range `index` throws. Deleting `index` 0 removes the header row, " +
"and the next row becomes the new header.",
inputSchema: {
pageId: z.string().min(1),
table: z.string().min(1),
index: z.number().int(),
},
},
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.tableDeleteRow,
async ({ pageId, table, index }) => {
const result = await docmostClient.tableDeleteRow(pageId, table, index);
return jsonContent(result);
@@ -277,24 +216,9 @@ server.registerTool(
);
// Tool: table_update_cell
// NOT shared — same `table` (here) vs `tableRef` (in-app) parameter-name
// divergence as table_insert_row.
server.registerTool(
"table_update_cell",
{
description:
"Set the plain-text content of cell [row,col] (0-based) in a table " +
"(`table` = `#<index>` or a block id inside it). Replaces the cell's " +
"content with a single text paragraph; for rich formatting use patch_node " +
"on the cell's paragraph id from table_get.",
inputSchema: {
pageId: z.string().min(1),
table: z.string().min(1),
row: z.number().int(),
col: z.number().int(),
text: z.string(),
},
},
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.tableUpdateCell,
async ({ pageId, table, row, col, text }) => {
const result = await docmostClient.tableUpdateCell(
pageId,
@@ -308,22 +232,9 @@ server.registerTool(
);
// Tool: create_page
server.registerTool(
"create_page",
{
description:
"Create a new page from Markdown in a space. Pass parentPageId to nest " +
"it under a parent; omit it to create at the space root.",
inputSchema: {
title: z.string().min(1).describe("Title of the page"),
content: z.string().min(1).describe("Markdown content"),
spaceId: z.string().min(1),
parentPageId: z
.string()
.optional()
.describe("Optional parent page ID to nest under"),
},
},
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.createPage,
async ({ title, content, spaceId, parentPageId }) => {
const result = await docmostClient.createPage(
title,
@@ -336,32 +247,11 @@ server.registerTool(
);
// Tool: update_page_json
server.registerTool(
"update_page_json",
{
description:
"Replace a page's content with a raw ProseMirror JSON document " +
"(lossless write: preserves the block ids, callouts, tables and " +
"attributes you pass in). Typical flow: get_page_json -> modify the " +
"JSON -> update_page_json. Keep existing node ids intact so heading " +
"anchors and history stay stable. Minimal full-doc example: " +
'{"type":"doc","content":[{"type":"paragraph","content":' +
'[{"type":"text","text":"Hi"}]}]}. `content` may be a JSON object or a ' +
"JSON string (both accepted), and is OPTIONAL: omit it to update only " +
"the title (though prefer rename_page for a title-only change). " +
"Supplying neither content nor title is an error.",
inputSchema: {
pageId: z.string().min(1).describe("ID of the page to update"),
content: z
.any()
.optional()
.describe(
'ProseMirror document {"type":"doc","content":[...]} (JSON object or ' +
"JSON string). Omit to rename only.",
),
title: z.string().optional().describe("Optional new title"),
},
},
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's content normalization (parse a JSON-string content,
// pass undefined/null through for a title-only/no-op update).
registerShared(
SHARED_TOOL_SPECS.updatePageJson,
async ({ pageId, content, title }) => {
// Only parse/validate the document when it was actually supplied; when it
// is omitted, pass it straight through so the client performs a title-only
@@ -379,26 +269,11 @@ server.registerTool(
);
// Tool: export_page_markdown
server.registerTool(
"export_page_markdown",
{
description:
"Export a page to a single self-contained, lossless Docmost-flavoured " +
"Markdown file (custom extensions): YAML-free meta header, body with " +
"inline comment anchors and diagrams, and a trailing comments-thread " +
"block. Designed for a download -> edit body -> import_page_markdown " +
"round-trip that preserves everything, including comment highlights. " +
"Comment THREADS are preserved in the file but are not re-pushed to the " +
"server on import.",
inputSchema: {
pageId: z.string().min(1),
},
},
async ({ pageId }) => {
const md = await docmostClient.exportPageMarkdown(pageId);
return { content: [{ type: "text" as const, text: md }] };
},
);
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.exportPageMarkdown, async ({ pageId }) => {
const md = await docmostClient.exportPageMarkdown(pageId);
return { content: [{ type: "text" as const, text: md }] };
});
// Tool: import_page_markdown
registerShared(
@@ -422,22 +297,11 @@ registerShared(
);
// Tool: rename_page
server.registerTool(
"rename_page",
{
description:
"Rename a page (change its title only) without touching or resending " +
"its content.",
inputSchema: {
pageId: z.string().min(1).describe("ID of the page to rename"),
title: z.string().min(1).describe("New title"),
},
},
async ({ pageId, title }) => {
const result = await docmostClient.renamePage(pageId, title);
return jsonContent(result);
},
);
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.renamePage, async ({ pageId, title }) => {
const result = await docmostClient.renamePage(pageId, title);
return jsonContent(result);
});
// Tool: edit_page_text
registerShared(SHARED_TOOL_SPECS.editPageText, async ({ pageId, edits }) => {
@@ -516,6 +380,10 @@ registerShared(SHARED_TOOL_SPECS.deleteNode, async ({ pageId, nodeId }) => {
});
// Tool: insert_image
// MCP-only by design (NOT in the shared registry): the in-app AI-chat agent
// exposes no image tools (insert/replace), so there is no second layer to unify
// — a SHARED_TOOL_SPECS entry's tier/catalogLine are in-app metadata and the
// catalog-partition test forbids a spec without a live in-app tool (#294).
server.registerTool(
"insert_image",
{
@@ -561,6 +429,7 @@ server.registerTool(
);
// Tool: replace_image
// MCP-only by design (see insert_image): no in-app equivalent, stays inline.
server.registerTool(
"replace_image",
{
@@ -603,25 +472,10 @@ server.registerTool(
);
// Tool: share_page
// INTENTIONAL per-transport divergence (not shared): the in-app copy adds a
// security-confirmation framing ("only share when the user explicitly asked,
// since this exposes the page to anyone with the link") tuned for the in-app
// agent; this transport keeps the plain public-URL wording.
server.registerTool(
"share_page",
{
description:
"Make a page publicly accessible (idempotent) and return its public " +
"URL. The URL format is <app>/share/<key>/p/<slugId>. This exposes the " +
"page content to ANYONE with the URL — do it only when explicitly asked.",
inputSchema: {
pageId: z.string().min(1).describe("ID of the page to share"),
searchIndexing: z
.boolean()
.optional()
.describe("Allow search engines to index the page (default true)"),
},
},
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own `searchIndexing ?? true` default.
registerShared(
SHARED_TOOL_SPECS.sharePage,
async ({ pageId, searchIndexing }) => {
const result = await docmostClient.sharePage(pageId, searchIndexing ?? true);
return jsonContent(result);
@@ -641,29 +495,11 @@ registerShared(SHARED_TOOL_SPECS.listShares, async () => {
});
// Tool: move_page
server.registerTool(
"move_page",
{
description:
"Move a page under a new parent (nesting) or to the space root.",
inputSchema: {
pageId: z.string().min(1),
parentPageId: z
.string()
.nullable()
.optional()
.describe(
"Target parent page ID. Pass 'null' or empty string to move to root.",
),
position: z
.string()
.min(5)
.optional()
.describe(
"fractional-index position key; min 5 chars; omit to append at the end.",
),
},
},
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's cycle guard, its 'null'/'' -> null string coercion, and
// its positive-confirmation check on the move response.
registerShared(
SHARED_TOOL_SPECS.movePage,
async ({ pageId, parentPageId, position }) => {
const finalParentId =
parentPageId === "" || parentPageId === "null" ? null : parentPageId;
@@ -698,49 +534,22 @@ server.registerTool(
);
// Tool: delete_page
server.registerTool(
"delete_page",
{
description:
"Delete a single page by ID. SOFT delete only: the page is moved to " +
"trash and can be restored; nothing is permanently deleted.",
inputSchema: {
pageId: z.string().min(1),
},
},
async ({ pageId }) => {
await docmostClient.deletePage(pageId);
return {
content: [
{ type: "text" as const, text: `Successfully deleted page ${pageId}` },
],
};
},
);
// Schema + description now live in the shared registry (#294). The shared schema
// exposes ONLY pageId, so no permanent/force-delete flag can reach the client.
registerShared(SHARED_TOOL_SPECS.deletePage, async ({ pageId }) => {
await docmostClient.deletePage(pageId);
return {
content: [
{ type: "text" as const, text: `Successfully deleted page ${pageId}` },
],
};
});
// --- Comment tools (ported from upstream PR #3 by Max Nikitin) ---
// Tool: list_comments
server.registerTool(
"list_comments",
{
description:
"List comments on a page in one call (pagination is handled " +
"internally). By DEFAULT only ACTIVE threads are returned; resolved " +
"threads (a resolved top-level comment and all its replies) are hidden " +
"and their count reported as `resolvedThreadsHidden` so you can re-query " +
"with `includeResolved: true` to see everything. Returns " +
"`{ items, resolvedThreadsHidden }`. Content is returned as Markdown.",
inputSchema: {
pageId: z.string().describe("ID of the page"),
includeResolved: z
.boolean()
.optional()
.describe(
"default only active threads; true — include resolved",
),
},
},
registerShared(
SHARED_TOOL_SPECS.listComments,
async ({ pageId, includeResolved }) => {
const comments = await docmostClient.listComments(pageId, includeResolved);
return jsonContent(comments);
@@ -748,55 +557,11 @@ server.registerTool(
);
// Tool: create_comment
// INTENTIONAL per-transport divergence (not shared): the in-app copy tunes the
// guidance for the in-app agent (e.g. "retry with a corrected EXACT selection"
// and "Reversible via the comment UI"); this transport keeps its own wording.
server.registerTool(
"create_comment",
{
description:
"Create a new comment on a page. The comment is ALWAYS inline and is " +
"anchored to (highlights) its `selection` text — there are no page-level " +
"comments. Content is provided as Markdown and automatically converted. " +
"A top-level comment REQUIRES an exact `selection`; if the selection " +
"cannot be found in the page the call fails (no orphan comment is left). " +
"Replies (parentCommentId set) inherit the parent's anchor and take no " +
"selection. You may also attach a `suggestedText` proposing a replacement " +
"for the `selection`; a human applies (or rejects) it from the UI. When " +
"`suggestedText` is set the `selection` MUST occur exactly once in the " +
"page — expand it with surrounding context if it is ambiguous.",
inputSchema: {
pageId: z.string().describe("ID of the page to comment on"),
content: z.string().min(1).describe("Comment content in Markdown format"),
selection: z
.string()
.min(1)
// Enforce the documented 250-char cap to match the description above.
.max(250)
.optional()
.describe(
"EXACT contiguous text from a single paragraph/block to anchor the " +
"comment on (<=250 chars). Required for a top-level comment; omit " +
"only when replying via parentCommentId.",
),
parentCommentId: z
.string()
.optional()
.describe("Parent comment ID to create a reply (max 2 nesting levels)"),
suggestedText: z
.string()
.min(1)
.max(2000)
.optional()
.describe(
"Optional proposed replacement (PLAIN TEXT) for the `selection`, " +
"applied by a human via the UI (never auto-applied). REQUIRES a " +
"`selection`; NOT allowed on a reply. When set, the `selection` must " +
"be UNIQUE in the page — expand it with surrounding context (still " +
"<=250 chars) if it occurs more than once, or the call is refused.",
),
},
},
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own guards (require a selection for a top-level
// comment; reject suggestedText on a reply / without a selection).
registerShared(
SHARED_TOOL_SPECS.createComment,
async ({ pageId, content, selection, parentCommentId, suggestedText }) => {
if (!parentCommentId && (!selection || !selection.trim())) {
throw new Error(
@@ -872,28 +637,9 @@ server.registerTool(
);
// Tool: resolve_comment
server.registerTool(
"resolve_comment",
{
description:
"Resolve (close) or reopen a comment thread. Only top-level comments can " +
"be resolved — the server rejects resolving a reply. Reversible: pass " +
"resolved=false to reopen. Resolving keeps the thread and its replies " +
"(unlike delete_comment, which permanently removes them).",
inputSchema: {
commentId: z
.string()
.min(1)
.describe("ID of the top-level comment thread to resolve or reopen"),
resolved: z
.boolean()
.optional()
.default(true)
.describe(
"true (default) marks the thread resolved/closed; false reopens it",
),
},
},
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.resolveComment,
async ({ commentId, resolved }) => {
const result = await docmostClient.resolveComment(commentId, resolved);
return jsonContent(result);
@@ -901,30 +647,10 @@ server.registerTool(
);
// Tool: check_new_comments
server.registerTool(
"check_new_comments",
{
description:
"Check for new comments across pages in a space since a given timestamp. " +
"Optionally scope to a page subtree (folder). Returns only comments " +
"created after the specified time.",
inputSchema: {
spaceId: z.string().describe("Space ID to check for new comments"),
since: z
.string()
.min(1)
.describe(
"ISO 8601 timestamp — only return comments created after this time (e.g. '2026-03-10T00:00:00Z')",
),
parentPageId: z
.string()
.optional()
.describe(
"Optional root page ID to scope the check to a subtree (folder). " +
"Only pages under this parent will be checked.",
),
},
},
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own guard rejecting an unparseable `since` timestamp.
registerShared(
SHARED_TOOL_SPECS.checkNewComments,
async ({ spaceId, since, parentPageId }) => {
// Reject an unparseable timestamp up front: otherwise the comparison
// against NaN silently treats every comment as "not new" and the tool
@@ -1053,6 +779,8 @@ server.registerTool(
);
// Tool: insert_footnote
// MCP-only by design (see insert_image): the in-app AI-chat agent exposes no
// footnote tool, so there is no second layer to unify — stays inline (#294).
server.registerTool(
"insert_footnote",
{
+494
View File
@@ -316,6 +316,34 @@ export const SHARED_TOOL_SPECS = {
// --- share management ---
// Unified from the per-layer inline definitions (#294). Both layers already
// carried the "only share when explicitly asked" security framing (the
// "per-transport divergence" note on the old inline copies was stale), so
// there was no real behavioral divergence to preserve — only wording drift.
sharePage: {
mcpName: 'share_page',
inAppKey: 'sharePage',
// CANONICAL: merges the MCP copy's URL-format + idempotency detail with the
// in-app copy's reversibility note; keeps the security framing both had.
description:
'Make a page PUBLICLY accessible (idempotent) and return its public URL ' +
'(format: <app>/share/<key>/p/<slugId>). This exposes the page content ' +
'to ANYONE with the URL — only share when the user explicitly asked. ' +
'Reversible: unshare it later to revoke the public URL.',
tier: 'deferred',
catalogLine: 'sharePage — make a page publicly accessible and return its URL.',
// Reconciled: MCP's stricter .min(1) on pageId kept; field descriptions from
// the in-app copy. The MCP execute keeps its own `searchIndexing ?? true`
// default (a per-layer concern, not part of the shared schema).
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to share.'),
searchIndexing: z
.boolean()
.optional()
.describe('Allow public search engines to index it (default true).'),
}),
},
unsharePage: {
mcpName: 'unshare_page',
inAppKey: 'unsharePage',
@@ -509,4 +537,470 @@ export const SHARED_TOOL_SPECS = {
pageId: z.string().min(1),
}),
},
// --- page tools (unified from the per-layer inline definitions, #294) ---
//
// Descriptions merge both layers (the MCP copy's richer structural notes + the
// in-app copy's "Reversible via history/trash" framing where it added one).
// Field constraints keep the MCP copy's stricter .min(1) EXCEPT where the
// in-app layer deliberately allowed a looser value (documented per field).
getPage: {
mcpName: 'get_page',
inAppKey: 'getPage',
description:
'Fetch a single page as Markdown by its id. Returns the page title and ' +
'its Markdown content. The Markdown conversion is LOSSY (block ids, exact ' +
'table/callout structure are approximated); for a lossless representation ' +
'use the lossless page-JSON read tool. Inline <span data-comment-id> tags in the markdown ' +
'are comment highlight anchors (also present for RESOLVED threads) — ' +
'treat them as markup, not page text.',
tier: 'core',
catalogLine: 'getPage — fetch a page as Markdown by its id.',
// Reconciled: MCP's stricter .min(1) kept; in-app's more-informative
// "(or slugId)" describe kept.
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id (or slugId) of the page.'),
}),
},
listPages: {
mcpName: 'list_pages',
inAppKey: 'listPages',
description:
'List the most recent pages (ordered by updatedAt, descending), ' +
'optionally scoped to a single space. Returns a bounded list (default ' +
'50, max 100) — use search for lookups in large spaces. Pass tree:true ' +
"(with spaceId) to instead get the space's full page hierarchy as a " +
'nested tree.',
tier: 'core',
catalogLine: "listPages — list recent pages, or a space's full page tree.",
buildShape: (z) => ({
spaceId: z
.string()
.optional()
.describe('Optional space id to scope the listing to.'),
limit: z
.number()
.int()
.min(1)
.max(100)
.optional()
.describe('Maximum number of pages (default 50, max 100).'),
tree: z
.boolean()
.optional()
.describe(
"When true, return the space's full page hierarchy as a nested tree " +
'(children arrays) instead of the recent-by-updatedAt flat list. ' +
'Requires spaceId; ignores limit.',
),
}),
},
createPage: {
mcpName: 'create_page',
inAppKey: 'createPage',
description:
'Create a new page with a Markdown body in a space, optionally under a ' +
'parent page (omit parentPageId to create at the space root). Returns ' +
'the new page id and title. Reversible: a page can be moved to trash ' +
'later.',
tier: 'deferred',
catalogLine: 'createPage — create a new page with a Markdown body in a space.',
// Reconciled schema DRIFT: the MCP copy pinned `content` to .min(1) while
// the in-app copy left it unbounded and DOCUMENTS an empty body as valid
// ("may be empty") — creating an empty page to fill in later is a real use
// case. The looser (no-min) form is kept, so create_page now also accepts an
// empty body (harmless — it creates an empty page) and no previously-valid
// in-app input is ever rejected. `title`/`spaceId` keep the MCP .min(1)
// (an empty title or space is never valid).
buildShape: (z) => ({
title: z.string().min(1).describe('The title of the new page.'),
content: z.string().describe('The page body as Markdown (may be empty).'),
spaceId: z.string().min(1).describe('The id of the space to create the page in.'),
parentPageId: z
.string()
.optional()
.describe('Optional parent page id to nest the new page under.'),
}),
},
movePage: {
mcpName: 'move_page',
inAppKey: 'movePage',
description:
'Move a page under a new parent page, or to the space root when no ' +
'parent is given. Reversible: move it back at any time.',
tier: 'deferred',
catalogLine: 'movePage — move a page under a new parent or to the space root.',
// Reconciled schema DRIFT: the MCP copy exposed a `position` field
// (fractional-index ordering) that the in-app copy lacked. Unified by
// KEEPING position (the in-app client already accepts an optional position
// arg, so the in-app execute now forwards it) — it is optional, so no
// previously-valid in-app call is rejected. `parentPageId` is `.nullable()`
// on both, so a real JSON null moves to root on either transport; the MCP
// execute additionally coerces the strings 'null'/'' to null as a robustness
// fallback (kept in its execute body, not in the shared schema).
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to move.'),
parentPageId: z
.string()
.nullable()
.optional()
.describe(
'Target parent page id. Null or omitted moves the page to the space ' +
'root.',
),
position: z
.string()
.min(5)
.optional()
.describe(
'Optional fractional-index position key (min 5 chars); omit to ' +
'append at the end.',
),
}),
},
renamePage: {
mcpName: 'rename_page',
inAppKey: 'renamePage',
description:
'Rename a page (change its title only; the body is untouched, never ' +
'resent). Reversible: rename back at any time.',
tier: 'deferred',
catalogLine: "renamePage — change a page's title only (body untouched).",
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to rename.'),
title: z.string().min(1).describe('The new title.'),
}),
},
deletePage: {
mcpName: 'delete_page',
inAppKey: 'deletePage',
description:
'Move a page to the trash — SOFT delete only: the page can be restored ' +
'from trash and nothing is ever permanently deleted.',
tier: 'deferred',
catalogLine: 'deletePage — move a page to trash (soft delete, reversible).',
// GUARDRAIL preserved (§14 H4): the schema exposes ONLY pageId, so a
// permanentlyDelete/forceDelete flag can never reach the client through this
// tool (asserted by ai-chat-tools.service.spec.ts).
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to move to trash.'),
}),
},
updatePageJson: {
mcpName: 'update_page_json',
inAppKey: 'updatePageJson',
description:
"Replace a page's content with a raw ProseMirror JSON document (lossless " +
'write: preserves the block ids, callouts, tables and attributes you pass ' +
'in). Typical flow: read the page-JSON view -> modify the JSON -> write it back. ' +
'Keep existing node ids intact so heading anchors and history stay ' +
'stable. Minimal full-doc example: {"type":"doc","content":[{"type":' +
'"paragraph","content":[{"type":"text","text":"Hi"}]}]}. `content` may be ' +
'a JSON object or a JSON string (both accepted), and is OPTIONAL: omit it ' +
'to update only the title (though prefer the rename-page tool for a title-only ' +
'change). Supplying neither content nor title is an error. Reversible: ' +
'the previous version is kept in page history.',
tier: 'deferred',
catalogLine:
"updatePageJson — overwrite a page's body with a full ProseMirror document.",
buildShape: (z) => ({
pageId: z.string().min(1).describe('ID of the page to update'),
content: z
.any()
.optional()
.describe(
'ProseMirror document {"type":"doc","content":[...]} (JSON object or ' +
'JSON string). Omit to update only the title.',
),
title: z.string().optional().describe('Optional new title'),
}),
},
exportPageMarkdown: {
mcpName: 'export_page_markdown',
inAppKey: 'exportPageMarkdown',
// CANONICAL: the MCP copy (a strict superset of the terse in-app wording).
description:
'Export a page to a single self-contained, lossless Docmost-flavoured ' +
'Markdown file (custom extensions): YAML-free meta header, body with ' +
'inline comment anchors and diagrams, and a trailing comments-thread ' +
'block. Designed for a download -> edit body -> page-Markdown import ' +
'round-trip that preserves everything, including comment highlights. ' +
'Comment THREADS are preserved in the file but are not re-pushed to the ' +
'server on import.',
tier: 'deferred',
catalogLine:
'exportPageMarkdown — export a page to self-contained Markdown (body + comments).',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to export.'),
}),
},
// --- comment tools (unified from the per-layer inline definitions, #294) ---
//
// create_comment and resolve_comment previously carried a "per-transport
// divergence" note in BOTH layers; #294 unifies their schema + description
// here. Only the four tools that genuinely exist in BOTH layers live in the
// registry: create/list/resolve comment and check_new_comments.
//
// update_comment and delete_comment are intentionally NOT here: they exist
// ONLY on the standalone MCP server. The in-app agent deliberately exposes no
// hard comment edit/delete tool (comment edits are irreversible / not
// version-tracked; see the guardrail tests in ai-chat-tools.service.spec.ts),
// so there is nothing to unify — they stay inline in index.ts.
createComment: {
mcpName: 'create_comment',
inAppKey: 'createComment',
// CANONICAL: the in-app copy (the more-maintained one). It keeps the same
// rules as the MCP copy — inline-only, top-level requires a `selection`, no
// page-level comments, replies inherit the anchor, suggestedText must be
// unique — and adds the "retry with a corrected EXACT selection" and reply-
// to-reply-rejected guidance the MCP copy lacked. Execute-side validation
// (reject suggestedText on a reply, require a selection) stays per-layer.
description:
'Add an INLINE comment to a page, or reply to an existing top-level ' +
'comment (one level only — the backend rejects replies to replies). ' +
'The comment is anchored inline to the given exact `selection` text ' +
'(which gets highlighted); page-level comments are NOT supported. A ' +
'new top-level comment REQUIRES a `selection`. Replies inherit the ' +
"parent's anchor and take no selection. If the call fails with a " +
'"selection not found" error, retry with a corrected EXACT selection ' +
'copied verbatim from a single paragraph/block. You may also attach a ' +
'`suggestedText` proposing a replacement for the `selection` (a human ' +
'applies it from the UI); when set, the `selection` must occur exactly ' +
'once in the page. Reversible via the comment UI.',
tier: 'core',
catalogLine:
'createComment — add an inline comment (optionally with a suggested edit).',
// Reconciled schema: the field set is identical across both layers; the
// only constraint drift is `content`, which the MCP copy pinned to
// .min(1) while the in-app copy left unbounded — the stricter MCP form is
// kept (an empty comment body is never valid).
buildShape: (z) => ({
pageId: z.string().describe('The id of the page to comment on.'),
content: z.string().min(1).describe('The comment body as Markdown.'),
selection: z
.string()
.min(1)
.max(250)
.optional()
.describe(
'EXACT contiguous text from a SINGLE paragraph/block to anchor ' +
'(highlight) the comment on (<=250 chars, avoid spanning across ' +
'formatting boundaries). Required for a new top-level comment; ' +
'omit only when replying via parentCommentId.',
),
parentCommentId: z
.string()
.optional()
.describe(
'Optional id of a TOP-LEVEL comment to reply to (one level ' +
'of replies only).',
),
suggestedText: z
.string()
.min(1)
.max(2000)
.optional()
.describe(
'Optional proposed replacement (PLAIN TEXT) for the `selection`, ' +
'applied by a human via the UI (never auto-applied). REQUIRES a ' +
'`selection`; NOT allowed on a reply. When set, the `selection` ' +
'must be UNIQUE in the page — expand it with surrounding context ' +
'(still <=250 chars) if it occurs more than once, or the call is ' +
'refused.',
),
}),
},
listComments: {
mcpName: 'list_comments',
inAppKey: 'listComments',
// CANONICAL: the two copies are near-identical; the MCP copy is the
// superset (it keeps the "(pagination is handled internally)" note the
// in-app copy dropped), so it is used verbatim.
description:
'List comments on a page in one call (pagination is handled ' +
'internally). By DEFAULT only ACTIVE threads are returned; resolved ' +
'threads (a resolved top-level comment and all its replies) are hidden ' +
'and their count reported as `resolvedThreadsHidden` so you can re-query ' +
'with `includeResolved: true` to see everything. Returns ' +
'`{ items, resolvedThreadsHidden }`. Content is returned as Markdown.',
tier: 'core',
catalogLine:
'listComments — list all comments on a page (including resolved).',
buildShape: (z) => ({
pageId: z.string().describe('ID of the page'),
includeResolved: z
.boolean()
.optional()
.describe('default only active threads; true — include resolved'),
}),
},
resolveComment: {
mcpName: 'resolve_comment',
inAppKey: 'resolveComment',
// CANONICAL: the MCP copy's richer wording, minus its snake_case reference
// to `delete_comment` (a sibling tool that does NOT exist in the in-app
// layer) — rephrased transport-neutrally per the registry convention.
description:
'Resolve (close) or reopen a top-level comment thread (reversible — ' +
'pass resolved=false to reopen). Only top-level comments can be ' +
'resolved; the server rejects resolving a reply. Resolving keeps the ' +
'thread and its replies intact (it is not a deletion).',
tier: 'core',
catalogLine: 'resolveComment — resolve or reopen a comment thread.',
// Reconciled schema: `resolved` drifted — the MCP copy made it optional
// with .default(true) (resolve is the common case, documented), the in-app
// copy made it required. The MCP form is kept (a strict superset: it never
// rejects a previously-valid input and adds a sensible default), and
// commentId keeps the MCP copy's stricter .min(1).
buildShape: (z) => ({
commentId: z
.string()
.min(1)
.describe('ID of the top-level comment thread to resolve or reopen'),
resolved: z
.boolean()
.optional()
.default(true)
.describe(
'true (default) marks the thread resolved/closed; false reopens it',
),
}),
},
checkNewComments: {
mcpName: 'check_new_comments',
inAppKey: 'checkNewComments',
// CANONICAL: the MCP copy (the more detailed of the two). The MCP layer's
// execute-side guard that rejects an unparseable `since` timestamp stays in
// its execute body (per-layer logic), not in the shared schema.
description:
'Check for new comments across pages in a space since a given ' +
'timestamp. Optionally scope to a page subtree (folder). Returns only ' +
'comments created after the specified time.',
tier: 'deferred',
catalogLine:
'checkNewComments — find comments in a space created after a timestamp.',
// Reconciled schema: `since` keeps the MCP copy's stricter .min(1) (the
// in-app copy left it unbounded); field descriptions use the MCP copy's
// more detailed wording (it carries an example timestamp).
buildShape: (z) => ({
spaceId: z.string().describe('Space ID to check for new comments'),
since: z
.string()
.min(1)
.describe(
"ISO 8601 timestamp — only return comments created after this time " +
"(e.g. '2026-03-10T00:00:00Z')",
),
parentPageId: z
.string()
.optional()
.describe(
'Optional root page ID to scope the check to a subtree (folder). ' +
'Only pages under this parent will be checked.',
),
}),
},
// --- table tools (unified from the per-layer inline definitions, #294) ---
//
// These tools carried a "NOT shared" note in BOTH layers because of a single
// parameter-NAME drift: the MCP layer named the table reference `table` while
// the in-app layer named it `tableRef`. #294 reconciles that drift by unifying
// on the MCP name `table` — renaming the MCP public parameter would break
// external MCP clients, whereas the in-app parameter is model-facing
// (prompt-only) and safe to rename. The in-app execute bodies now destructure
// `table` instead of `tableRef` (nothing else changes). Descriptions take the
// MCP copy's richer wording (it documented `#<index>`, padding, header-row
// behavior) plus the in-app copy's "Reversible via page history" note; sibling
// tool references are phrased transport-neutrally.
//
// NOT here (kept inline in index.ts): table_get / getTable. Its MCP tool name
// is noun-first (`table_get`) while the in-app key is verb-first (`getTable`),
// so it breaks the snake_case(inAppKey) naming convention the registry enforces
// (shared-tool-specs.contract.spec.ts). Renaming the public MCP tool would
// break external clients, so it stays per-transport (its in-app param was still
// aligned to `table` for consistency with the migrated trio below).
tableInsertRow: {
mcpName: 'table_insert_row',
inAppKey: 'tableInsertRow',
description:
'Insert a row of plain-text cells into a table. `table` is `#<index>` ' +
'from the page outline, or a block id inside it. `cells` is the text per ' +
"column (padded to the table's column count; an error if more cells than " +
'columns). `index` is the 0-based insert position (0 inserts before the ' +
'header); omit to append at the end. Reversible via page history.',
tier: 'deferred',
catalogLine: 'tableInsertRow — insert a row of plain-text cells into a table.',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page.'),
table: z
.string()
.min(1)
.describe('"#<index>" from the page outline, or a block id in the table.'),
cells: z.array(z.string()).describe('The cell texts for the row (one per column).'),
index: z
.number()
.int()
.optional()
.describe('0-based insert position (0 inserts before the header); omit to append.'),
}),
},
tableDeleteRow: {
mcpName: 'table_delete_row',
inAppKey: 'tableDeleteRow',
description:
'Delete the row at 0-based `index` from a table (`table` is `#<index>` ' +
'from the page outline, or a block id inside it). Refuses to delete the ' +
"table's only row; an out-of-range `index` throws. Deleting `index` 0 " +
'removes the header row, and the next row becomes the new header. ' +
'Reversible via page history.',
tier: 'deferred',
catalogLine: 'tableDeleteRow — delete a table row at a 0-based index.',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page.'),
table: z
.string()
.min(1)
.describe('"#<index>" from the page outline, or a block id in the table.'),
index: z.number().int().describe('0-based row index to delete.'),
}),
},
tableUpdateCell: {
mcpName: 'table_update_cell',
inAppKey: 'tableUpdateCell',
description:
'Set the plain-text content of cell [row, col] (0-based) in a table ' +
'(`table` is `#<index>` from the page outline, or a block id inside it). ' +
"Replaces the cell's content with a single text paragraph; for rich " +
"formatting, patch the cell's paragraph id (obtained from reading the " +
'table) instead. Reversible via page history.',
tier: 'deferred',
catalogLine: 'tableUpdateCell — set the text of a table cell at [row, col].',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page.'),
table: z
.string()
.min(1)
.describe('"#<index>" from the page outline, or a block id in the table.'),
row: z.number().int().describe('0-based row index.'),
col: z.number().int().describe('0-based column index.'),
text: z.string().describe('The new cell text.'),
}),
},
} satisfies Record<string, SharedToolSpec>;
@@ -25,9 +25,11 @@ test("nested bulletList with 3 children keeps all children indented under the pa
),
);
// Block children of a list item are blank-line separated (loose list) since
// the #351 converter fix; the sublist stays nested at the 2-col marker column.
assert.equal(
convertProseMirrorToMarkdown(input),
"- Parent\n - A\n - B\n - C",
"- Parent\n\n - A\n - B\n - C",
);
});
@@ -41,9 +43,10 @@ test("nested list under an ordered item indents 3 spaces", () => {
),
);
// Blank-line separated block children (loose list) per the #351 converter fix.
assert.equal(
convertProseMirrorToMarkdown(input),
"1. Parent\n - Child",
"1. Parent\n\n - Child",
);
});
+105
View File
@@ -0,0 +1,105 @@
# @docmost/prosemirror-markdown
The single, canonical **ProseMirror ↔ Markdown converter** plus the Docmost
schema mirror (#293/#345). Headless and framework-free: no React, no browser
runtime. There is exactly ONE copy of this converter in the repo, consumed by:
- `packages/mcp` (the MCP server),
- `packages/git-sync` (two-way Git sync),
- `apps/server` (server-side markdown import/export, #345).
`src/lib/docmost-schema.ts` **mirrors** the upstream Tiptap schema that lives in
`packages/editor-ext`. The mirror is not free-floating: `serializer-contract.test.ts`
guards the boundary — every schema node must have a converter case, so a drift
between `editor-ext` and this package surfaces as a failing test rather than a
silent divergence.
## Why byte-stability matters
Git sync exports a page to markdown, and re-imports it on the next pull. If
`export → import → export` is not **byte-stable**, every pull rewrites files
that nobody edited, and the user's history churns with phantom diffs. So the
converter is held to more than "it roughly round-trips": the second export pass
must be a byte-for-byte fixpoint. That is what the property suite below proves.
## Two golden layers (do not mix them)
1. **Corpus fixtures**`test/fixtures/corpus/`. A fixed, hand-curated set of
representative documents (headings, marks, lists, tables, diagrams, columns,
details, mentions, …). These are the readable, deterministic "known-good"
snapshots. Edit them deliberately.
2. **Generative property suite**`test/generative/`. fast-check draws random
documents and asserts invariants over them. Two entry points:
- `flat-roundtrip.property.test.ts` — flat documents, attribute-level fuzzing.
- `nested-roundtrip.property.test.ts` — deeply nested structures.
The invariants (**P1–P4**):
- **P1** — semantic round-trip: `mdToPm(pmToMd(doc))` is canonically equal to
`doc` (no data loss for the round-trip-supported space).
- **P2** — byte fixpoint: `pmToMd(mdToPm(pmToMd(doc))) === pmToMd(doc)` (the
first pass may normalize once; the second pass must be a fixpoint).
- **P3** — totality: neither converter throws; output is bounded.
- **P4** — parser fuzz totality: for ANY input string, `markdownToProseMirror`
does not throw and returns a schema-valid document.
These invariants are kept **STRICT** — no `it.fails`, skip, or weakening. A
failure means the generator found a REAL converter bug.
## The counterexample process (the DoD)
This is the point of the generative layer. When a property run diverges:
1. **A property run surfaces a divergence** (locally, in CI, or in the nightly
cron — see below).
2. **fast-check shrinks it** to a minimal, human-readable counterexample and
prints the reproducing seed.
3. **Commit the shrunk doc as a permanent fixture** under
`test/fixtures/counterexamples/`, with a matching case in
`counterexamples.test.ts`. The fixture stays forever, as a regression pin.
4. **FIX the converter** so the counterexample round-trips. **Never weaken a
property to hide the bug.**
5. If — and only if — a maintainer decides a particular markdown-representable
loss is genuinely acceptable, it is recorded as an **explicit ACCEPTED /
allowlist entry with a written reason**, not by silently relaxing an
invariant.
### Attribute-coverage allowlist
`flat-roundtrip.property.test.ts` maintains `ATTR_VALUE_FUZZ_ALLOWLIST`. The
suite asserts that every attribute in the live schema is EITHER value-fuzzed by
the generator OR explicitly listed in this allowlist. This forces any newly
added node attribute to be consciously classified — you cannot add an attr and
leave it silently un-exercised; the coverage test fails until you either fuzz it
or record why it is held out.
## Running
```sh
# The full package suite (corpus + generative + contract tests):
pnpm --filter @docmost/prosemirror-markdown test
```
The generative suite honours two env knobs (invalid/empty → falls back to the
default):
| Env var | Default (flat) | Default (nested) | Meaning |
| -------------------- | -------------- | ---------------- | -------------------------------- |
| `PROPERTY_SEED` | `20250705` | `20250705` | fast-check seed (reproducibility) |
| `PROPERTY_NUM_RUNS` | `300` | `100` | runs per property |
```sh
# Reproduce a specific counterexample seed with a bigger budget:
PROPERTY_SEED=12345 PROPERTY_NUM_RUNS=5000 \
pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/
```
### Nightly cron
`.github/workflows/nightly-property.yml` runs the generative suite every night
with `PROPERTY_NUM_RUNS` cranked to ~5000 and a **random** seed, to reach deeper
counterexamples than a fixed-seed PR run can. On failure it files a Gitea issue
containing the reproducing seed, the run count, and the tail of the output (the
shrunk counterexample), which kicks off the counterexample → fixture process
above. It can also be triggered manually (`workflow_dispatch`) with custom
`num_runs` / `seed`.
@@ -1006,6 +1006,8 @@ const Column = Node.create({
width: {
default: null,
parseHTML: (el: HTMLElement) => {
// Mirrors editor-ext (column.ts): width is a unitless flex-grow
// number, so parse it to a Number for parity with the canonical schema.
const value = el.getAttribute("data-width");
return value ? parseFloat(value) : null;
},
@@ -48,6 +48,52 @@ export interface ConvertProseMirrorToMarkdownOptions {
dropResolvedCommentAnchors?: boolean;
}
/**
* Adjacent sibling lists that share a markdown MARKER FAMILY re-parse as ONE
* merged list bulletList and taskList both emit `- ` markers ( a single
* `<ul>`), and two orderedLists both emit `1.` markers ( a single `<ol>`). The
* cross-type case is real data loss the editor CAN produce (e.g. a taskList
* followed by a bulletList: the merged `<ul>` has a mix of checkbox and plain
* items, so `bridgeTaskLists` refuses to convert it and every taskItem loses its
* checkbox). Between two such adjacent list children we emit an empty HTML comment
* `<!-- -->`: marked renders it as its own HTML block that interrupts the list, so
* the two lists stay distinct; on import the comment is inert (parseAttachedComment
* null) and dropped by generateJSON, and re-export re-inserts it, so the marker
* is byte-stable. It fires ONLY between two adjacent same-family list nodes no
* separator is emitted for any other join, so non-list output is unchanged.
*/
const LIST_MARKER_SEPARATOR = "<!-- -->";
function listMarkerFamily(type: string | undefined): "ul" | "ol" | null {
if (type === "bulletList" || type === "taskList") return "ul";
if (type === "orderedList") return "ol";
return null;
}
function adjacentListsMerge(
prevType: string | undefined,
curType: string | undefined,
): boolean {
const a = listMarkerFamily(prevType);
return a !== null && a === listMarkerFamily(curType);
}
/**
* Render each block child, inserting a `<!-- -->` separator entry between any two
* adjacent same-marker-family list nodes (see LIST_MARKER_SEPARATOR). Callers
* join the returned strings with their own context separator.
*/
function renderBlockChildren(
children: any[],
render: (n: any) => string,
): string[] {
const out: string[] = [];
let prevType: string | undefined;
for (const child of children) {
if (adjacentListsMerge(prevType, child?.type)) out.push(LIST_MARKER_SEPARATOR);
out.push(render(child));
prevType = child?.type;
}
return out;
}
/**
* Convert ProseMirror/TipTap JSON content to Markdown
* Supports all Docmost-specific node types and extensions
@@ -347,9 +393,15 @@ export function convertProseMirrorToMarkdown(
// lossless (the body survives) and byte-stable (it re-exports identically),
// so it is deliberately not treated as data loss.
const parts: string[] = [];
let prevDocType: string | undefined;
for (const child of nodeContent) {
if (child?.type === "footnotesList") continue;
// Keep adjacent same-family sibling lists distinct (see renderBlockChildren).
if (adjacentListsMerge(prevDocType, child?.type)) {
parts.push(LIST_MARKER_SEPARATOR);
}
parts.push(processNode(child));
prevDocType = child?.type;
}
for (const [id, def] of footnoteDefs) {
if (!referencedFootnoteIds.has(id)) {
@@ -583,12 +635,23 @@ export function convertProseMirrorToMarkdown(
.map((item: any) => processListItem(item, "-"))
.join("\n");
case "orderedList":
case "orderedList": {
// Honor a non-1 `start`: CommonMark expresses it as the first marker
// ("5." starts the list at 5), and marked parses that back into
// <ol start="5">. Emitting `${start + index}.` keeps a start=5 list as
// "5.","6.",… while a default (start=1) list stays "1.","2.",… See #351.
// Only an INTEGER ≥ 2 is a valid explicit start; collapse everything else
// (absent, non-number, 0, negative, fractional) to 1. A negative/fractional
// start would otherwise emit an untokenizable marker (e.g. "-3.", "2.5.")
// that marked cannot parse, re-importing the whole list as a PARAGRAPH.
const raw = node.attrs?.start;
const start = Number.isInteger(raw) && (raw as number) > 1 ? (raw as number) : 1;
return nodeContent
.map((item: any, index: number) =>
processListItem(item, `${index + 1}.`),
processListItem(item, `${start + index}.`),
)
.join("\n");
}
case "taskList":
return nodeContent.map((item: any) => processTaskItem(item)).join("\n");
@@ -599,20 +662,34 @@ export function convertProseMirrorToMarkdown(
return processTaskItem(node);
case "listItem":
return nodeContent.map(processNode).join("\n");
// Direct-listItem path (lists normally render via processListItem, which
// handles the marker + indentation). Blank line between block children so
// multiple paragraphs do not merge on re-parse; a `<!-- -->` entry
// (renderBlockChildren) separates adjacent sibling lists.
return renderBlockChildren(nodeContent, processNode).join("\n\n");
case "blockquote":
case "blockquote": {
// Prefix EVERY line of EVERY child with "> " and separate block-level
// children with a blank ">" line so code blocks / multi-paragraph
// quotes round-trip correctly.
return nodeContent
.map((n: any) =>
// quotes round-trip correctly. A `> <!-- -->` separator line is inserted
// between two adjacent same-family sibling lists (renderBlockChildren)
// so they stay distinct inside the quote.
const bqParts: string[] = [];
let prevBqType: string | undefined;
for (const n of nodeContent) {
if (adjacentListsMerge(prevBqType, n?.type)) {
bqParts.push(`> ${LIST_MARKER_SEPARATOR}`);
}
bqParts.push(
processNode(n)
.split("\n")
.map((line: string) => (line.length ? `> ${line}` : ">"))
.join("\n"),
)
.join("\n>\n");
);
prevBqType = n?.type;
}
return bqParts.join("\n>\n");
}
case "horizontalRule":
return "---";
@@ -787,9 +864,12 @@ export function convertProseMirrorToMarkdown(
// blockquote-prefixed; a blank line becomes a bare `>` so the callout is
// not split.
const calloutType = (node.attrs?.type || "info").toLowerCase();
const calloutBody = nodeContent
.map(processNode)
.join("\n")
const calloutBody = renderBlockChildren(nodeContent, processNode)
// Blank line between block children (rendered as a bare `>` after the
// prefix pass below) so multiple paragraphs stay separate nodes instead
// of merging on re-parse — same rule blockquote already uses. A
// `<!-- -->` entry (renderBlockChildren) separates adjacent sibling lists.
.join("\n\n")
.split("\n")
.map((l: string) => (l.length ? `> ${l}` : ">"))
.join("\n");
@@ -809,7 +889,10 @@ export function convertProseMirrorToMarkdown(
return `<summary>${renderInlineChildren(nodeContent)}</summary>\n\n`;
case "detailsContent":
return `${nodeContent.map(processNode).join("\n")}\n`;
// Blank line between block children so multiple paragraphs in a details
// body survive as separate nodes (a single "\n" merges them on re-parse);
// a `<!-- -->` entry (renderBlockChildren) separates adjacent sibling lists.
return `${renderBlockChildren(nodeContent, processNode).join("\n\n")}\n`;
case "mathInline": {
// #293 canon #6: inline math serializes as Obsidian-native `$LaTeX$`
@@ -1329,20 +1412,38 @@ export function convertProseMirrorToMarkdown(
return `<ul>${children
.map((li: any) => `<li>${blockChildrenToHtml(li)}</li>`)
.join("")}</ul>`;
case "orderedList":
return `<ol>${children
case "orderedList": {
// Carry a non-1 `start` on the raw-HTML path (columns/spanned cells) via
// the <ol start="N"> attribute, which the tiptap parser reads back into
// attrs.start. A default (start=1) list stays a bare <ol>. See #351.
// Use the SAME integer-≥-2 guard as the markdown path: a fractional start
// would emit `start="2.5"` → parseInt→2 on import (path divergence), and a
// 0/negative start is not a valid explicit start either.
const raw = block.attrs?.start;
const start = Number.isInteger(raw) && (raw as number) > 1 ? (raw as number) : 1;
const startAttr = start > 1 ? ` start="${start}"` : "";
return `<ol${startAttr}>${children
.map((li: any) => `<li>${blockChildrenToHtml(li)}</li>`)
.join("")}</ol>`;
}
case "codeBlock": {
const lang = block.attrs?.language || "";
// The code itself is element TEXT content (between <code> tags), so it
// must escape < > & — NOT the attribute escaper. The language rides in
// a class ATTRIBUTE, so it uses escapeAttr.
//
// Read the child text RAW (as `case "codeBlock"` does) and keep it
// VERBATIM — do NOT strip the trailing newline. Unlike the markdown fence
// path (which strips then relies on marked re-adding one `\n`), the schema
// codeBlock parseHTML reads the `<code>` text content back byte-for-byte,
// so stripping here would drop a trailing newline the node legitimately
// carries and break the round trip inside a column/cell.
const code = escapeHtmlText(
children
.map(processNode)
.join("")
.replace(/\n+$/, ""),
.map((child: any) =>
typeof child?.text === "string" ? child.text : "",
)
.join(""),
);
const cls = lang ? ` class="language-${escapeAttr(lang)}"` : "";
return `<pre><code${cls}>${code}</code></pre>`;
@@ -1484,6 +1585,13 @@ export function convertProseMirrorToMarkdown(
const indent = " ".repeat(indentWidth);
const lines: string[] = [];
childStrings.forEach((child, childIndex) => {
// Separate consecutive block children with a BLANK line so the item is a
// CommonMark "loose" list item and each block stays its own node. Without
// it, a second paragraph (`- a\n b`) is re-parsed as a lazy continuation
// of the first and the two merge into one paragraph — silent data loss.
// The blank line still sits INSIDE the item (the following block keeps the
// continuation indent), so nested lists/code blocks remain nested.
if (childIndex > 0) lines.push("");
child.split("\n").forEach((line, lineIndex) => {
if (childIndex === 0 && lineIndex === 0) {
// First physical line of the first block gets the marker.
@@ -1500,7 +1608,9 @@ export function convertProseMirrorToMarkdown(
const processListItem = (item: any, prefix: string): string => {
const itemContent = item.content || [];
const childStrings = itemContent.map(processNode);
// A `<!-- -->` entry separates two adjacent same-family sublists inside the
// item so they do not merge on re-parse (see renderBlockChildren).
const childStrings = renderBlockChildren(itemContent, processNode);
if (childStrings.length === 0) return prefix;
// The rendered marker is `${prefix} ` (prefix + one space), so its width —
// and thus the continuation indent — is prefix.length + 1. This is correct
@@ -1514,7 +1624,9 @@ export function convertProseMirrorToMarkdown(
const checkbox = checked ? "[x]" : "[ ]";
const prefix = `- ${checkbox}`;
const itemContent = item.content || [];
const childStrings = itemContent.map(processNode);
// A `<!-- -->` entry separates two adjacent same-family sublists inside the
// item so they do not merge on re-parse (see renderBlockChildren).
const childStrings = renderBlockChildren(itemContent, processNode);
// An empty task item still needs its checkbox marker; without this guard
// the indent below produces "" and the "- [ ]"/"- [x]" row disappears.
if (childStrings.length === 0) return prefix;
@@ -270,9 +270,10 @@ const CALLOUT_CLOSE_RE = /^:::\s*$/;
* optional title after the type is allowed but ignored (the Docmost callout
* schema has no title). The body is the following contiguous blockquote lines.
*/
const CALLOUT_BQ_OPEN_RE = /^>\s*\[!(\w+)\]/;
/** Matches any blockquote continuation line (`>` … ). */
const BLOCKQUOTE_LINE_RE = /^>/;
// The callout's own `>` marker may be preceded by an ENCLOSING container prefix:
// list-item indentation (` `) and/or blockquote markers (`> `). Group 1 captures
// that prefix (lazily, so the LAST `>` before `[!type]` is the callout's own).
const CALLOUT_BQ_OPEN_RE = /^([>\s]*?)>\s*\[!(\w+)\]/;
/** Matches the start/end of a code fence (``` or ~~~), capturing the marker. */
const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
@@ -402,20 +403,47 @@ async function preprocessCallouts(markdown: string): Promise<string> {
// recurse so nested callouts (`> > [!type]`) are handled, then emit the same
// callout div the `:::` path produces. A normal blockquote (no `[!type]` on
// its first line) does not match and stays a blockquote.
//
// PREFIX-aware: a callout nested inside a list item and/or a blockquote is
// serialized with the enclosing container prefix in front of its own `>`
// marker — ` > [!type]` (list indent) or `> > [!type]` (blockquote). We
// capture that prefix, take only continuation lines carrying `prefix>`, strip
// it, and re-apply the prefix to the emitted HTML block so the callout div
// stays WITHIN its container (an unprefixed div would escape and re-parse as
// a top-level callout / plain blockquote — silent structure loss).
const bqOpen = line.match(CALLOUT_BQ_OPEN_RE);
if (bqOpen) {
const type = bqOpen[1].toLowerCase();
const prefix = bqOpen[1];
const type = bqOpen[2].toLowerCase();
const cont = prefix + ">"; // a body line = prefix + the callout's own `>`
const bodyLines: string[] = [];
let j = i + 1;
for (; j < lines.length; j++) {
if (!BLOCKQUOTE_LINE_RE.test(lines[j])) break;
bodyLines.push(lines[j].replace(/^>\s?/, ""));
if (!lines[j].startsWith(cont)) break;
// Drop the prefix + `>` + one optional space, leaving the body content.
bodyLines.push(lines[j].slice(prefix.length).replace(/^>\s?/, ""));
}
const inner = await transform(bodyLines);
const renderedInner = await markedInstance.parse(inner);
out.push(
`\n<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>\n`,
);
const block = `<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>`;
if (prefix.length === 0) {
// Top-level callout: blank lines isolate the HTML block.
out.push(`\n${block}\n`);
} else if (prefix.includes(">")) {
// Enclosing BLOCKQUOTE: prefix every line and add NO surrounding blank
// lines — a blank line would terminate the blockquote and split the
// callout out of it.
out.push(block.split("\n").map((l) => prefix + l).join("\n"));
} else {
// Pure LIST-ITEM indentation: re-indent and keep the blank-line
// separators (a loose list item), so the div sits at the marker column.
out.push(
`\n${block
.split("\n")
.map((l) => (l.length ? prefix + l : l))
.join("\n")}\n`,
);
}
i = j;
continue;
}
@@ -597,6 +625,40 @@ function bridgeTaskLists(html: string): string {
* (null from parseAttachedComment), an unknown name, a wrong-position comment, or
* an unknown/empty attr value is ignored.
*/
/**
* A directive comment is in ATTACHED position when it sits inside a `<p>`/`<hN>`
* textblock bound to that block's text (the `attrs`/`img` conventions). Every
* other parent (body, document level, a block container like blockquote/details/
* li/column div) is STANDALONE position, where a lone-block directive
* (subpages/pagebreak/pageembed/transclusion) is materialized. Broadening
* standalone beyond body/document is what lets these nodes survive NESTED inside
* a blockquote/callout/details/list item (previously dropped -> silent data loss).
*/
function isAttachedPosition(tag: string): boolean {
return tag === "p" || /^h[1-6]$/.test(tag);
}
/**
* Place a materialized standalone-directive element in the DOM: replace the
* comment IN PLACE when it has a real element parent inside <body> (body itself
* or a nested block container), preserving document order; queue it as a leading
* div only when the comment is at document level (no parentElement) or directly
* under `<html>` (outside <body>, which `document.body.innerHTML` would drop).
*/
function placeStandalone(
comment: any,
el: any,
tag: string,
leadingDivs: any[],
): void {
if (comment.parentElement && tag !== "html") {
comment.replaceWith(el);
} else {
comment.remove();
leadingDivs.push(el);
}
}
function applyCommentDirectives(html: string): string {
// Cheap early-out: no comments at all -> nothing to intercept.
if (!html.includes("<!--")) return html;
@@ -664,13 +726,13 @@ function applyCommentDirectives(html: string): string {
if (parsed.name === "subpages" || parsed.name === "pagebreak") {
// #293 canon #5 STANDALONE machinery. A lone comment line is rendered by
// marked as an HTML block; the parser places it either directly under
// <body> (when other content surrounds it) or at document level (when it
// leads the output). Both are STANDALONE position. A `subpages`/`pagebreak`
// comment sitting inside a `<p>`/`<hN>` (or any other element) is attached
// position -> INERT.
const standalone = tag === "" || tag === "body" || tag === "html";
if (!standalone) continue; // wrong position -> inert
// marked as its own HTML block; the parser places it under <body>, at
// document level (leading), or — when the directive is NESTED — inside a
// block CONTAINER (`<blockquote>` for blockquote/callout, `<details>`,
// `<li>`, a column `<div>`, …). All of those are STANDALONE position. Only a
// comment ATTACHED inside a `<p>`/`<hN>` (bound to that block's text) is
// attached position -> INERT.
if (isAttachedPosition(tag)) continue; // wrong position -> inert
const div = document.createElement("div");
if (parsed.name === "pagebreak") {
div.setAttribute("data-type", "pageBreak");
@@ -680,26 +742,18 @@ function applyCommentDirectives(html: string): string {
div.setAttribute("data-recursive", "true");
}
}
if (tag === "body") {
// In-body: replace in place so surrounding content keeps its order.
comment.replaceWith(div);
} else {
// Document-level (leading): drop the stray comment and queue the div to
// be prepended into body below.
comment.remove();
leadingDivs.push(div);
}
placeStandalone(comment, div, tag, leadingDivs);
continue;
}
if (parsed.name === "pageembed" || parsed.name === "transclusion") {
// #293 canon #8 STANDALONE media. Like subpages/pagebreak: a lone comment
// line placed under <body> or at document level (leading). An attached-
// position comment (inside a <p>/<hN> with a sibling) is INERT. We rebuild
// the schema div the raw-HTML path emits (media-html.ts) from the decoded
// attrs so serialize/parse stay in sync.
const standalone = tag === "" || tag === "body" || tag === "html";
if (!standalone) continue; // wrong position -> inert
// line placed under <body>, at document level (leading), or NESTED inside a
// block container (blockquote/callout/details/li/column). An ATTACHED-
// position comment (inside a `<p>`/`<hN>`) is INERT. We rebuild the schema
// div the raw-HTML path emits (media-html.ts) from the decoded attrs so
// serialize/parse stay in sync.
if (isAttachedPosition(tag)) continue; // wrong position -> inert
const el = buildElement(
parsed.name === "pageembed"
? pageEmbedToHtml({ sourcePageId: parsed.attrs.sourcePageId })
@@ -709,12 +763,7 @@ function applyCommentDirectives(html: string): string {
}),
);
if (!el) continue; // defensive: builder always yields an element
if (tag === "body") {
comment.replaceWith(el);
} else {
comment.remove();
leadingDivs.push(el);
}
placeStandalone(comment, el, tag, leadingDivs);
continue;
}
@@ -806,18 +855,36 @@ function applyCommentDirectives(html: string): string {
if (!parent) continue; // attrs comment must have an element parent
if (parsed.name !== "attrs") continue; // unknown name -> inert
// #293 canon #9 ATTACHED attrs: honored only in attached position.
const isBlock = tag === "p" || /^h[1-6]$/.test(tag);
if (!isBlock) continue; // misplaced comment -> inert
const align = parsed.attrs.textAlign;
if (typeof align === "string" && align) {
// Re-express as an inline style; the schema's textAlign parseHTML reads
// `el.style.textAlign` back onto the paragraph/heading node.
parent.style.textAlign = align;
// #293 canon #9 ATTACHED attrs: honored only in attached position.
if (tag === "p" || /^h[1-6]$/.test(tag)) {
// A real <p>/<hN> host (loose list item, top-level block, …): re-express as
// an inline style; the schema's textAlign parseHTML reads `el.style.textAlign`
// back onto the paragraph/heading node.
if (typeof align === "string" && align) parent.style.textAlign = align;
comment.remove();
} else if (tag === "li" || tag === "td" || tag === "th") {
// TIGHT list item / GFM table cell: marked emits the paragraph's inline
// content DIRECTLY inside the <li>/<td>/<th> with NO <p> wrapper, so there
// is no element to carry the style — generateJSON materializes the
// paragraph later. Wrap the host's LEADING inline content (everything up to
// the comment; any trailing block child such as a nested list stays put) in
// a <p> carrying the alignment, so the materialized paragraph re-reads it.
if (typeof align === "string" && align) {
const p = document.createElement("p");
p.style.textAlign = align;
while (parent.firstChild && parent.firstChild !== comment) {
p.appendChild(parent.firstChild);
}
parent.insertBefore(p, comment);
}
comment.remove();
} else {
// Misplaced `attrs` comment (not a textblock/li/cell host): inert. Consume
// it anyway so no attached marker ever survives into the parsed body
// (matches the pre-existing "consume regardless" behaviour).
comment.remove();
}
// Consume the marker regardless (unknown keys are simply ignored) so no
// attached comment ever survives into the parsed body.
comment.remove();
}
// Prepend any document-level (leading) standalone divs into body, preserving
// their document order relative to each other and ahead of existing content.
@@ -46,7 +46,11 @@ export function videoToHtml(attrs: Record<string, any>): string {
if (attrs.width != null) parts.push(`width="${escapeAttr(attrs.width)}"`);
if (attrs.height != null) parts.push(`height="${escapeAttr(attrs.height)}"`);
if (attrs.size != null) parts.push(`data-size="${escapeAttr(attrs.size)}"`);
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`);
// align default is "center" (schema): OMIT it so a bare/center node stays
// clean and parse's re-materialized "center" default is not a P2 churn — only
// a genuinely non-default left/right emits data-align (mirrors imageToHtml).
if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
if (attrs.aspectRatio != null)
parts.push(`data-aspect-ratio="${escapeAttr(attrs.aspectRatio)}"`);
return `<div><video ${parts.join(" ")}></video></div>`;
@@ -62,7 +66,9 @@ export function youtubeToHtml(attrs: Record<string, any>): string {
parts.push(`data-width="${escapeAttr(attrs.width)}"`);
if (attrs.height != null)
parts.push(`data-height="${escapeAttr(attrs.height)}"`);
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`);
// "center" is the schema default -> omit (see videoToHtml rationale).
if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
return `<div ${parts.join(" ")}></div>`;
}
@@ -97,7 +103,9 @@ export function diagramToHtml(
if (attrs.size != null) parts.push(`data-size="${escapeAttr(attrs.size)}"`);
if (attrs.aspectRatio != null)
parts.push(`data-aspect-ratio="${escapeAttr(attrs.aspectRatio)}"`);
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`);
// "center" is the schema default -> omit (see videoToHtml rationale).
if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
if (attrs.attachmentId)
parts.push(`data-attachment-id="${escapeAttr(attrs.attachmentId)}"`);
return `<div ${parts.join(" ")}></div>`;
@@ -110,10 +118,18 @@ export function embedToHtml(attrs: Record<string, any>): string {
`data-src="${escapeAttr(attrs.src ?? "")}"`,
`data-provider="${escapeAttr(attrs.provider ?? "")}"`,
];
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`);
if (attrs.width != null)
// "center" is the schema default -> omit (see videoToHtml rationale).
if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
// embed width/height default to the NUMBERS 800/600 (schema). Getting a data-
// attribute back always yields a STRING, so emitting the default here would
// round-trip 800 -> "800" (a number->string P1 divergence canonicalize does
// NOT normalize). OMIT the defaults so parse re-materializes the numeric
// default instead — mirrors the top-level embed path (markdown-converter.ts),
// which also emits width/height only when they differ from 800/600.
if (attrs.width != null && attrs.width !== 800)
parts.push(`data-width="${escapeAttr(attrs.width)}"`);
if (attrs.height != null)
if (attrs.height != null && attrs.height !== 600)
parts.push(`data-height="${escapeAttr(attrs.height)}"`);
return `<div ${parts.join(" ")}></div>`;
}
@@ -0,0 +1,19 @@
{
"doc": {
"type": "doc",
"content": [
{
"type": "orderedList",
"attrs": { "type": null, "start": 5 },
"content": [
{
"type": "listItem",
"content": [
{ "type": "paragraph", "content": [{ "type": "text", "text": "alpha" }] }
]
}
]
}
]
}
}
@@ -0,0 +1,390 @@
/**
* Schema-DERIVED attribute-state fast-check arbitraries (#351, PR 1).
*
* This GENERALIZES the #350 stability-matrix helper (roundtrip-stability.helper.ts)
* to fast-check. Where that helper sweeps a HAND-WRITTEN 2-state matrix for one
* node spec, this module reads the attribute list straight from
* `schema.nodes[type].spec.attrs` (never a hand list) and, per attribute,
* generates over the FOUR states the issue calls for:
*
* - `absent` : the attribute is OMITTED entirely (the empty-string-vs-
* absent churn class the #350 fix targets).
* - `default` : the schema default value, authored explicitly.
* - `nonDefault` : a representative legal non-default value.
* - `degenerate` : `""` for strings, `0`/negative for numbers, the flipped
* value for booleans.
*
* Why a per-attribute override table
* Everything that CAN be derived generically from the default's runtime type is
* (booleans flip; the degenerate value follows the runtime type). But two facts
* force a small, DOCUMENTED override table:
*
* 1. CONSTRAINED domains the schema does not encode. `image.align ∈
* {left,center,right}`, `heading.level 1..6`, `callout.type
* {info,success,warning,danger}`, `columns.layout`, table-cell `align`,
* `status.color`, `orderedList.start ≥ 1`, etc. A generic "default + 1"
* would emit an ILLEGAL value, so these get an explicit legal domain.
* 2. ROUND-TRIP-safety, established EMPIRICALLY by probing the live converter
* (the classification captured in flat-roundtrip.property.test.ts). A frozen
* attribute falls into ONE of TWO explicitly-distinguished classes never a
* silent "it just doesn't round-trip":
*
* (a) ACCEPTED LIMITATION the attribute has NO markdown representation,
* so the loss is inherent to targeting markdown, not a converter
* defect. These: `paragraph`/`heading` `indent`, `callout.icon`,
* `orderedList.type` (a/A/i markers), table `colwidth` /
* `backgroundColor(Name)` (dropped by the raw-<table> fallback). Each is
* tagged `// ACCEPTED:` inline. Freezing them is correct there is
* nothing to preserve in the target format.
*
* (b) FIXED & VALUE-FUZZED attributes that were once PINNED converter
* bugs (representable in markdown but dropped) and are now FIXED in
* src/, so they are value-fuzzed here at legal non-default values like
* any healthy attr. `orderedList.start` (the non-1 start once rendered
* as `1.`; the converter now emits the start marker / `<ol start="N">`)
* and `column.width` (a unitless flex-grow number that round-trips via
* parseFloat) are both fuzzed in OVERRIDES below. The former held-out
* `it.fails` cases are gone; `ordered-list-start.json` +
* counterexamples.test.ts now stand as PASSING regression pins (per the
* epic guardrail, the minimal doc stays forever to guard re-regression).
* The #351 media-family sizing attrs (image/video/youtube/pdf/drawio/
* excalidraw/embed width/height/size/aspectRatio) are likewise fuzzed
* now that they ride round-trip-safely in the per-node `<!--…-->` JSON.
*
* (c) DEFERRED-BUG representable AND round-trips, frozen only because the
* flat generator can't yet build a valid instance. Table
* `colspan`/`rowspan` round-trip via the raw-<table> fallback, but a
* geometrically-valid spanned table is PR-2 structural work; the flat
* generator hardcodes span = 1. Tagged `// DEFERRED-BUG:` inline so a
* maintainer does not read them as an inherent limitation.
* - Several non-null-default attrs are MATERIALIZED on import but are not
* in canonicalize's KNOWN_DEFAULTS (`callout.type`, `status.color`,
* table `colspan`/`rowspan`, `columns.layout`/`widthMode`,
* `embed.width`/`height`, `heading.level`, `taskItem.checked`,
* `details.open`, `subpages.recursive`, `orderedList.start`). If left
* `absent` they re-materialize as a non-canonical default and diverge
* under P1. We mark them `always` so they are authored explicitly.
* - The documented numericstring coercion set (`width height size
* aspectRatio`) is generated as STRINGS for the media family (a stored
* number re-parses as a string), EXCEPT `embed.width/height` which the
* embed schema keeps numeric handled per-attr.
*
* The two former PINNED-BUG attrs (`column.width` P2 churn, `orderedList.start`
* P1 loss) are now FIXED and value-fuzzed; `ordered-list-start.json` in
* counterexamples.test.ts is a permanent PASSING regression pin, not an
* `it.fails` hold-out.
*/
import fc from 'fast-check';
import { getSchema } from '@tiptap/core';
import { docmostExtensions } from '../../src/lib/index.js';
import { phraseArb, letterPhraseArb, urlArb } from './text-arbitraries.js';
/** The exact ProseMirror schema the converter targets. */
export const schema = getSchema(docmostExtensions as any);
/** Sentinel: this attribute is OMITTED (the `absent` state). */
export const ABSENT = Symbol('ABSENT');
/** The documented numeric→string coercion set (issue + roundtrip-stability.helper). */
export const NUMERIC_STRING_ATTRS = ['width', 'height', 'size', 'aspectRatio'];
/** Read the schema default for every attribute of a node type. */
export function schemaAttrDefaults(type: string): Record<string, unknown> {
const specAttrs = (schema.nodes[type]?.spec?.attrs ?? {}) as Record<
string,
{ default: unknown }
>;
const out: Record<string, unknown> = {};
for (const [k, v] of Object.entries(specAttrs)) out[k] = v.default;
return out;
}
/** Attribute names for a node type, straight from the schema (never hand-listed). */
export function schemaAttrNames(type: string): string[] {
return Object.keys((schema.nodes[type]?.spec?.attrs ?? {}) as object);
}
/**
* Per-attribute policy. Everything unlisted falls back to a generic policy:
* - a BOOLEAN default is fuzzable (its non-default is the flipped value);
* - any other default is `frozen` (only `absent`/`default` are generated) so
* we never invent an unverified non-default that might not round-trip.
* Listed attrs override this with a legal `arb` domain and/or flags.
*/
interface AttrPolicy {
/** Arbitrary for the `nonDefault` state's value. */
arb?: fc.Arbitrary<unknown>;
/** Value for the `degenerate` state (fuzz mode only). Omit to skip degenerate. */
degen?: unknown;
/** Never emit `absent` — the attr must be authored (materialized default class). */
always?: boolean;
/** Never emit the schema default value (required-ish attrs like `src`). Implies always. */
noDefault?: boolean;
/** Never emit non-default/degenerate — attr has no md representation or churns. */
frozen?: boolean;
}
const num = (...xs: number[]) => fc.constantFrom(...xs);
const str = (...xs: string[]) => fc.constantFrom(...xs);
const widthStr = str('120', '320', '640');
// Media `aspectRatio`/`size` are stringified numerics too (converter emits
// String(value)); the schema parseHTML reads them back as strings. Fuzz as
// plausible numeric strings so they round-trip byte-stably like widthStr.
const aspectRatioStr = str('1.5', '0.75', '1');
const sizeStr = str('320', '640');
// The documented override table, keyed `type.attr`. Every entry is grounded in
// the empirical converter probe (see flat-roundtrip.property.test.ts header).
const OVERRIDES: Record<string, AttrPolicy> = {
// ── block text containers ────────────────────────────────────────────────
// 'left' is the IMPLICIT default alignment: the converter drops it on export
// (empirically confirmed), so it never round-trips. Only center/right/justify
// carry through the `<!--attrs {textAlign}-->` comment.
'paragraph.textAlign': { arb: str('center', 'right', 'justify') },
'paragraph.indent': { frozen: true }, // ACCEPTED: no md representation
'heading.level': { always: true, arb: num(2, 3, 4, 5, 6) },
'heading.textAlign': { arb: str('center', 'right', 'justify') },
'heading.indent': { frozen: true }, // ACCEPTED: no md representation
// ── lists ────────────────────────────────────────────────────────────────
// FIXED (#351): the converter now emits the start marker ("5." / <ol start="5">)
// and it round-trips, so the start number is value-fuzzed. See
// counterexamples.test.ts (ordered-list-start.json) for the regression pin.
'orderedList.start': { always: true, arb: num(2, 3, 5, 42) },
'orderedList.type': { frozen: true }, // ACCEPTED: a/A/i markers not expressible in GFM
'taskItem.checked': { always: true, arb: fc.constant(true) }, // boolean, default false
// ── codeBlock ────────────────────────────────────────────────────────────
'codeBlock.language': { arb: str('js', 'ts', 'python', 'go', 'rust', 'bash') },
// ── image / media (numeric→string width family) ──────────────────────────
'image.src': { noDefault: true, arb: urlArb, degen: '' },
'image.align': { arb: str('left', 'right') },
'image.alt': { arb: letterPhraseArb, degen: '' },
'image.title': { arb: letterPhraseArb },
'image.width': { arb: widthStr, degen: '' },
'image.height': { arb: widthStr, degen: '' },
// #351 (this PR): media sizing/family attrs ride in the `<!--img {…}-->`
// comment JSON via String(value); parseHTML reads them back as strings, so a
// numeric string round-trips byte-stably (mirrors image.width).
'image.size': { arb: sizeStr, degen: '' },
'image.aspectRatio': { arb: aspectRatioStr },
// caption is text carried verbatim in the comment JSON (mirrors image.alt).
'image.caption': { arb: letterPhraseArb, degen: '' },
'video.src': { noDefault: true, arb: urlArb, degen: '' },
'video.alt': { arb: letterPhraseArb },
'video.width': { arb: widthStr },
'video.height': { arb: widthStr },
// #351 (this PR): video sizing/align ride in the `<!--video {…}-->` comment.
'video.size': { arb: sizeStr },
'video.aspectRatio': { arb: aspectRatioStr },
// align default "center" is dropped on export; fuzz non-center only.
'video.align': { arb: str('left', 'right') },
'audio.src': { noDefault: true, arb: urlArb, degen: '' },
'youtube.src': { noDefault: true, arb: urlArb },
// #351 (this PR): youtube width/height/align ride in the `<!--youtube {…}-->`
// comment JSON (String()-coerced dimensions, non-center align only).
'youtube.width': { arb: widthStr },
'youtube.height': { arb: widthStr },
'youtube.align': { arb: str('left', 'right') },
'pdf.src': { noDefault: true, arb: urlArb },
'pdf.name': { arb: phraseArb },
// #351 (this PR): pdf size/width/height ride in the `<!--pdf {…}-->` comment
// (String()-coerced dimensions, read back as strings).
'pdf.size': { arb: sizeStr },
'pdf.width': { arb: widthStr },
'pdf.height': { arb: widthStr },
'drawio.src': { noDefault: true, arb: urlArb },
// #351 (this PR): drawio family attrs ride in the `<!--drawio {…}-->` comment.
// Dimensions/size/aspectRatio are String()-coerced numeric strings; title/alt
// are text carried verbatim; align fuzzed non-center only.
'drawio.width': { arb: widthStr },
'drawio.height': { arb: widthStr },
'drawio.size': { arb: sizeStr },
'drawio.aspectRatio': { arb: aspectRatioStr },
'drawio.align': { arb: str('left', 'right') },
'drawio.title': { arb: letterPhraseArb },
'drawio.alt': { arb: letterPhraseArb },
'excalidraw.src': { noDefault: true, arb: urlArb },
// #351 (this PR): excalidraw family attrs ride in the `<!--excalidraw {…}-->`
// comment (same shape as the drawio family above).
'excalidraw.width': { arb: widthStr },
'excalidraw.height': { arb: widthStr },
'excalidraw.size': { arb: sizeStr },
'excalidraw.aspectRatio': { arb: aspectRatioStr },
'excalidraw.align': { arb: str('left', 'right') },
'excalidraw.title': { arb: letterPhraseArb },
'excalidraw.alt': { arb: letterPhraseArb },
'attachment.url': { noDefault: true, arb: urlArb },
'attachment.name': { arb: phraseArb },
// ── callout / status ─────────────────────────────────────────────────────
'callout.type': { always: true, arb: str('success', 'warning', 'danger') },
'callout.icon': { frozen: true }, // ACCEPTED: no md representation (dropped on export)
'status.text': { noDefault: true, arb: phraseArb, degen: '' },
'status.color': { always: true, arb: str('green', 'orange', 'red', 'blue', 'yellow', 'purple') },
// ── table cells ────────────────────────────────────────────────────────────
// DEFERRED-BUG (not ACCEPTED): colspan/rowspan ARE representable and round-trip
// — a spanned cell makes the converter emit the whole table as a raw <table>
// with colspan/rowspan attrs (markdown-converter.ts tableToHtml), which the
// tiptap parser reads back. They are frozen only because generating a
// geometrically-valid spanned table is deferred STRUCTURAL work (the flat
// generator hardcodes colspan/rowspan = 1), NOT a markdown limitation.
'tableCell.colspan': { always: true, frozen: true },
'tableCell.rowspan': { always: true, frozen: true },
// ACCEPTED: colwidth / backgroundColor(Name) have no representation — the
// raw-<table> fallback (tableToHtml) drops them, so there is nothing to preserve.
'tableCell.colwidth': { frozen: true },
'tableCell.backgroundColor': { frozen: true },
'tableCell.backgroundColorName': { frozen: true },
'tableCell.align': { arb: str('left', 'center', 'right') },
'tableHeader.colspan': { always: true, frozen: true }, // DEFERRED-BUG (see tableCell.colspan)
'tableHeader.rowspan': { always: true, frozen: true }, // DEFERRED-BUG (see tableCell.rowspan)
'tableHeader.colwidth': { frozen: true }, // ACCEPTED: no representation
'tableHeader.backgroundColor': { frozen: true }, // ACCEPTED: no representation
'tableHeader.backgroundColorName': { frozen: true }, // ACCEPTED: no representation
'tableHeader.align': { arb: str('left', 'center', 'right') },
// ── details ──────────────────────────────────────────────────────────────
'details.open': { always: true, arb: fc.constant(true) }, // boolean, default false
// ── columns ──────────────────────────────────────────────────────────────
'columns.layout': { always: true, arb: str('three_equal', 'left_sidebar', 'right_sidebar') },
// widthMode round-trips via the `data-width-mode` attribute (verified P1+P2),
// so it is fuzzed, not frozen.
'columns.widthMode': { always: true, arb: str('custom') },
// column.width is a unitless flex-grow NUMBER (matches editor-ext column.ts);
// parseHTML does parseFloat, so String(50) === "50" both ways and a numeric
// width round-trips byte-stably. Value-fuzzed as a number.
'column.width': { arb: num(25, 50, 75) },
// ── embed (schema keeps width/height NUMERIC, not string-coerced) ─────────
'embed.src': { noDefault: true, arb: urlArb, degen: '' },
'embed.provider': { noDefault: true, arb: str('iframe', 'youtube', 'vimeo') },
// #351 (this PR): the embed schema defaults width/height to the NUMBERS 800/600
// and the converter only emits them when they differ. But the value round-trips
// as a STRING: export stringifies into the comment JSON (String(width)) and the
// import path (embedToHtml -> data-width -> embed parseHTML) reads it back as a
// string, so an authored NUMBER 400 would diverge under P1 (400 vs "400"). Fuzz
// as numeric STRINGS avoiding "800"/"600" so they round-trip byte-stably. The
// 800/600 numeric default state still round-trips (omitted on export, re-
// materialized as the numeric default). `always` stays because these are
// materialized on import but absent from canonicalize's KNOWN_DEFAULTS.
'embed.width': { always: true, arb: str('400', '1000', '1200') },
'embed.height': { always: true, arb: str('300', '500', '900') },
// align default "center" is dropped on export; fuzz non-center only.
'embed.align': { arb: str('left', 'right') },
// ── subpages / math / htmlEmbed ──────────────────────────────────────────
'subpages.recursive': { always: true, arb: fc.constant(true) }, // boolean, default false
'mathBlock.text': { noDefault: true, arb: str('x^2', 'a < b', '\\frac{1}{2}'), degen: '' },
'mathInline.text': { noDefault: true, arb: str('x^2', 'a < b', '\\frac{1}{2}'), degen: '' },
'htmlEmbed.source': { noDefault: true, arb: str('<b>hi</b>', '<i>x</i>', '<span>y</span>'), degen: '' },
'htmlEmbed.height': { arb: num(200, 300, 400) },
// ── footnotes / transclusion / pageEmbed / mention ───────────────────────
'footnoteDefinition.id': { noDefault: true, arb: str('fn1', 'fn2', 'note') },
'footnoteReference.id': { noDefault: true, arb: str('fn1', 'fn2', 'note') },
'pageEmbed.sourcePageId': { noDefault: true, arb: fc.uuid() },
'transclusionSource.id': { noDefault: true, arb: str('src1', 'src2') },
'transclusionReference.sourcePageId': { noDefault: true, arb: fc.uuid() },
'transclusionReference.transclusionId': { noDefault: true, arb: str('tr1', 'tr2') },
'mention.id': { noDefault: true, arb: fc.uuid() },
'mention.label': { noDefault: true, arb: phraseArb },
'mention.entityType': { noDefault: true, arb: str('user') },
'mention.entityId': { noDefault: true, arb: fc.uuid() },
};
/** Resolve the effective policy for one attribute (override merged over generic). */
function policyFor(type: string, attr: string, def: unknown): AttrPolicy {
const override = OVERRIDES[`${type}.${attr}`];
if (override) return override;
// Generic: booleans are fuzzable via their flipped value; everything else is
// frozen (only absent/default) so no unverified non-default is invented.
if (typeof def === 'boolean') return { arb: fc.constant(!def) };
return { frozen: true };
}
/**
* Whether an attribute is actually exercised at a NON-DEFAULT value (i.e. its
* policy has an `arb`, which the generic fallback does not). Used by the
* attribute-coverage snapshot test to make the generic-frozen space VISIBLE: any
* string/number attr not in OVERRIDES is silently only tested at absent/default,
* so the snapshot pins exactly which attrs are NOT value-fuzzed and forces a
* reviewer to look when a new attr lands in that invisible bucket.
*/
export function attrIsValueFuzzed(type: string, attr: string): boolean {
const def = schemaAttrDefaults(type)[attr];
return !!policyFor(type, attr, def).arb;
}
/** Every node `type.attr` in the schema (excluding the auto `id`), sorted. */
export function allSchemaAttrKeys(): string[] {
const keys: string[] = [];
for (const type of Object.keys(schema.nodes)) {
for (const attr of schemaAttrNames(type)) {
if (attr === 'id') continue;
keys.push(`${type}.${attr}`);
}
}
return keys.sort();
}
/**
* Every MARK attribute in the schema, keyed `mark:<name>.<attr>`, sorted. Marks
* are not driven by the node OVERRIDES table (they are fuzzed by the text
* generator, text-arbitraries.ts), so their value-fuzz coverage is tracked with a
* separate snapshot (see flat-roundtrip.property.test.ts) without this the
* "no invisible coverage hole" guarantee would hold for node attrs only, letting a
* new mark attr slip through unfuzzed and unallowlisted.
*/
export function allSchemaMarkAttrKeys(): string[] {
const keys: string[] = [];
for (const [name, mark] of Object.entries(schema.marks)) {
const attrs = (mark.spec?.attrs ?? {}) as Record<string, unknown>;
for (const attr of Object.keys(attrs)) keys.push(`mark:${name}.${attr}`);
}
return keys.sort();
}
export type AttrMode = 'p1' | 'fuzz';
/**
* Build an arbitrary for ONE attribute's value (or the ABSENT sentinel) across
* the states legal for `mode`:
* - p1 : absent / default / nonDefault (the round-trip-safe space).
* - fuzz : the above PLUS degenerate (P2 tolerates the one-time
* normalization; P3 only needs totality).
*/
export function attrValueArb(
type: string,
attr: string,
mode: AttrMode,
): fc.Arbitrary<unknown | typeof ABSENT> {
const def = schemaAttrDefaults(type)[attr];
const p = policyFor(type, attr, def);
const states: fc.Arbitrary<unknown | typeof ABSENT>[] = [];
if (!p.always && !p.noDefault) states.push(fc.constant(ABSENT));
if (!p.noDefault) states.push(fc.constant(def));
if (!p.frozen && p.arb) states.push(p.arb);
if (mode === 'fuzz' && !p.frozen && p.degen !== undefined) {
states.push(fc.constant(p.degen));
}
if (states.length === 0) states.push(fc.constant(def));
return fc.oneof(...states);
}
/**
* Build an arbitrary for a node's full `attrs` object over all schema attrs.
* `base` pins caller-required attrs (e.g. a concrete `src`) verbatim; any attr
* present in `base` is NOT re-generated. Omitted (ABSENT) attrs are dropped.
*/
export function nodeAttrsArb(
type: string,
mode: AttrMode,
base: Record<string, unknown> = {},
): fc.Arbitrary<Record<string, unknown>> {
const names = schemaAttrNames(type).filter((n) => !(n in base) && n !== 'id');
if (names.length === 0) return fc.constant({ ...base });
return fc
.tuple(...names.map((n) => attrValueArb(type, n, mode)))
.map((vals) => {
const attrs: Record<string, unknown> = { ...base };
names.forEach((n, i) => {
if (vals[i] !== ABSENT) attrs[n] = vals[i];
});
return attrs;
});
}
@@ -0,0 +1,46 @@
import { describe, expect, it } from 'vitest';
import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import path from 'node:path';
import { convertProseMirrorToMarkdown } from '../../src/lib/markdown-converter.js';
import { markdownToProseMirror } from '../../src/lib/markdown-to-prosemirror.js';
import { docsCanonicallyEqual } from '../../src/lib/canonicalize.js';
// ---------------------------------------------------------------------------
// #351 committed counterexample — a REAL round-trip bug surfaced by the flat
// generative probing (attribute level). The bug below is now FIXED in src/
// (maintainer-approved), so this case is a permanent PASSING regression pin:
// the exact document that once lost data now round-trips cleanly, and the
// fixture stays in test/fixtures/counterexamples/ forever (per the epic
// guardrail) to guard against a re-regression. The case carries a loud
// `// BUG #351 (FIXED):` note recording the defect and its fix site.
//
// With the bug fixed, the offending attribute is now VALUE-FUZZED by the
// generator (see attr-arbitraries.ts: orderedList.start), not held out — this
// committed document is the minimal, human-readable pin.
// ---------------------------------------------------------------------------
const here = path.dirname(fileURLToPath(import.meta.url));
const fixtureDir = path.resolve(here, '../fixtures/counterexamples');
function loadDoc(file: string): any {
return JSON.parse(readFileSync(path.join(fixtureDir, file), 'utf8')).doc;
}
describe('#351 counterexamples (round-trip bugs, now FIXED — permanent regression pins)', () => {
// BUG #351 (FIXED): an `orderedList` with a non-1 `start` lost its start
// number. CommonMark CAN express this ("5." starts the list at 5), but the
// converter always emitted "1." and ignored `attrs.start`:
// doc.start = 5 -> md1 = "1. alpha" (start dropped on export)
// re-import stored start = 1 => docsCanonicallyEqual(rt, doc) === false
// A P1 (semantic round-trip) loss of the SAME class as column.width. FIXED in
// this PR in src/lib/markdown-converter.ts (the orderedList markdown path emits
// `${start + index}.`; the raw-HTML path emits `<ol start="N">`). tiptap's
// StarterKit / marked parse the start back, so it round-trips. Permanent P1 pin.
it('ordered list start number is preserved (P1)', async () => {
const doc = loadDoc('ordered-list-start.json');
const md1 = convertProseMirrorToMarkdown(doc);
const doc2 = await markdownToProseMirror(md1);
expect(docsCanonicallyEqual(doc2, doc)).toBe(true);
});
});
@@ -0,0 +1,392 @@
/**
* Nested whole-document generator (#351, PR 2) "variant A": a random walk over
* the schema's ContentMatch automaton.
*
* Where the FLAT generator (node-generators.ts) emits `{doc:[ <one target> ]}`,
* this module produces arbitrarily DEEP, valid ProseMirror documents. Validity is
* NOT hand-asserted: it comes straight from the schema. To fill a container's
* block content we start at `nodeType.contentMatch` and walk the automaton
* enumerate the legal next node types (`match.edge(i).type`), let fast-check pick
* one (or STOP once `match.validEnd`), generate that child RECURSIVELY, then
* advance the automaton via `match.matchType(childType)`. A bad walk therefore
* cannot emit a structurally-invalid doc (the `generator validity` test guards
* this with `schema.nodeFromJSON(json).check()`).
*
* What the walk drives, and what it delegates
* The walk owns BLOCK STRUCTURE (which blocks nest inside which containers, in
* what order and how deep). Two things it deliberately delegates, to avoid
* REGRESSING the byte-stable space the flat suite already proved empirically:
*
* - INLINE content of textblocks (paragraph/heading/codeBlock/detailsSummary)
* is filled from text-arbitraries.ts the exact hostile-but-byte-stable
* inline corpus the flat suite established. Walking the inline ContentMatch
* instead would re-derive (and re-fail) the text-space limitations the flat
* suite already pins, which is not this generator's job.
* - Node ATTRS come from nodeAttrsArb(type, 'p1') the round-trip-safe
* attribute space. Attribute-degenerate fuzzing (P2/P3 over 'fuzz') is the
* flat suite's concern; the nested suite isolates STRUCTURAL round-trip, so
* it stays in 'p1' and does not re-litigate frozen/pinned attributes.
*
* Coordination the automaton cannot express
* A ContentMatch guarantees a child SEQUENCE is legal, but not cross-sibling
* invariants. Two nodes need coordination the walk injects by hand:
* - `table`: GFM needs a RECTANGULAR grid with column-consistent alignment.
* The automaton happily allows ragged rows / per-cell align, which are not a
* converter bug just a malformed table. So a table is generated ATOMICALLY
* (same shape the flat suite proved) rather than walked.
* - `columns`: the `layout` attr must agree with the column COUNT. We pick the
* layout, derive the count, then walk each column's block body normally so
* columns still gain real nested content, only the count is coordinated.
*
* Excluded from the nested walk
* Footnote nodes (footnoteReference / footnotesList / footnoteDefinition) need a
* DOCUMENT-GLOBAL id match between a reference and its definition. That
* coordination is owned by the flat suite's `footnotes` generator; placing them
* independently here would fabricate id mismatches that look like converter bugs
* but are generator defects. They are filtered out of the walk (documented in
* EXCLUDED). The completeness contract lives in the FLAT suite and is unaffected.
*
* Termination / budgets
* Two bounds keep every doc finite and the suite fast:
* - MAX_DEPTH a hard cap on block-nesting depth. A precomputed `minDepth`
* fixpoint (the minimum extra nesting a subtree of each type needs to be
* valid) lets the walk pick a CONTAINER child only when there is depth
* headroom to complete it so the walk can never paint itself into a corner
* where a required child cannot fit (no invalid docs, guaranteed termination).
* - NODE_BUDGET a soft cap on total nodes; as it runs low the walk biases
* toward STOP (when validEnd) or toward cheap terminating children.
*/
import fc from 'fast-check';
import { getSchema } from '@tiptap/core';
import { docmostExtensions } from '../../src/lib/docmost-schema.js';
import { nodeAttrsArb } from './attr-arbitraries.js';
import {
inlineContentArb,
headingInlineContentArb,
plainInlineContentArb,
phraseArb,
} from './text-arbitraries.js';
/** The exact ProseMirror schema the converter targets (built per the issue). */
export const schema = getSchema(docmostExtensions as never);
/** Hard cap on block-nesting depth (doc = depth 0). Kept in the issue's 4–5 band. */
export const MAX_DEPTH = 4;
/**
* Soft cap on total node count per generated document. Kept moderate: every P1/P2
* run parses the emitted markdown through jsdom (heavy), so 100+ node docs across
* hundreds of runs exhaust the worker heap. 60 still yields deeply-nested docs
* (depth 4) while keeping the suite within memory.
*/
export const NODE_BUDGET = 60;
/**
* Nodes kept OUT of the nested walk: footnote nodes need a doc-global id match a
* local walk cannot coordinate (owned by the flat suite's `footnotes` generator).
*/
const EXCLUDED = new Set<string>([
'footnoteReference',
'footnotesList',
'footnoteDefinition',
]);
/**
* Structural-only children that carry `group: "block"` in the schema and so leak
* into EVERY block container's ContentMatch, even though the editor only ever
* places them inside their one true parent. Choosing them freely (e.g. a bare
* `column` at the document root) fabricates documents no editor produces and that
* the converter is not designed to round-trip a GENERATOR artifact, not a
* converter bug. They are admitted ONLY when the container being filled is their
* dedicated parent. (`column` is in fact always built inside columnsArb, so this
* just double-guards it.)
*/
const DEDICATED_PARENT: Record<string, string> = {
column: 'columns',
detailsSummary: 'details',
detailsContent: 'details',
};
/** Is child type `t` legal as a freely-chosen child of container `parentType`? */
function childAllowedUnder(t: string, parentType: string): boolean {
const dedicated = DEDICATED_PARENT[t];
return dedicated === undefined || dedicated === parentType;
}
/** Textblock (inlineContent) types — filled from the proven inline corpus. */
function isTextblock(typeName: string): boolean {
return !!schema.nodes[typeName]?.isTextblock;
}
/** Leaf/atom types — no content, only generated attrs. */
function isLeaf(typeName: string): boolean {
return !!schema.nodes[typeName]?.isLeaf;
}
// ---------------------------------------------------------------------------
// minDepth fixpoint: the minimum EXTRA block-nesting depth a valid subtree
// rooted at each node type requires. Leaves and textblocks need 0 (a textblock
// is satisfied by inline content, no block recursion). A container needs
// 1 + the cheapest way to satisfy its ContentMatch. Computed as a min–max path
// to `validEnd` over the automaton, iterated to a fixpoint over node types.
// ---------------------------------------------------------------------------
/** Cheapest (min over reachable validEnd of max child minDepth) to complete a match. */
function minCompletion(
match: any,
md: Record<string, number>,
seen: Set<any>,
parentType: string,
): number {
let best = match.validEnd ? 0 : Infinity;
if (seen.has(match)) return best; // a cycle never completes more cheaply
seen.add(match);
for (let i = 0; i < match.edgeCount; i++) {
const edge = match.edge(i);
const t = edge.type.name;
if (EXCLUDED.has(t) || t === 'text') continue;
if (!childAllowedUnder(t, parentType)) continue;
const childCost = md[t];
if (childCost === undefined || childCost === Infinity) continue;
const rest = minCompletion(edge.next, md, seen, parentType);
if (rest === Infinity) continue;
best = Math.min(best, Math.max(childCost, rest));
}
seen.delete(match);
return best;
}
function computeMinDepth(): Record<string, number> {
const md: Record<string, number> = {};
for (const name of Object.keys(schema.nodes)) {
md[name] = isLeaf(name) || isTextblock(name) ? 0 : Infinity;
}
let changed = true;
while (changed) {
changed = false;
for (const name of Object.keys(schema.nodes)) {
if (md[name] === 0) continue; // leaves/textblocks fixed at 0
const nt: any = schema.nodes[name];
const completion = minCompletion(nt.contentMatch, md, new Set(), name);
const next = completion === Infinity ? Infinity : 1 + completion;
if (next < md[name]) {
md[name] = next;
changed = true;
}
}
}
return md;
}
const MIN_DEPTH = computeMinDepth();
// ---------------------------------------------------------------------------
// Leaf / textblock builders (attrs from 'p1', inline from the proven corpus).
// ---------------------------------------------------------------------------
function attachAttrs(typeName: string, base: Record<string, unknown> = {}) {
return nodeAttrsArb(typeName, 'p1', base).map((attrs) => {
const node: any = { type: typeName };
if (Object.keys(attrs).length) node.attrs = attrs;
return node;
});
}
/** A leaf/atom block: attrs only, no content. */
function leafArb(typeName: string): fc.Arbitrary<any> {
return attachAttrs(typeName);
}
/** A textblock, inline content taken from the byte-stable flat corpus. */
function textblockArb(typeName: string): fc.Arbitrary<any> {
if (typeName === 'codeBlock') {
return fc
.tuple(
nodeAttrsArb('codeBlock', 'p1'),
// Fenced code re-imports with a TRAILING NEWLINE (flat suite finding);
// author it so the doc is already at the round-trip fixpoint.
fc.array(phraseArb, { minLength: 1, maxLength: 3 }).map((l) => l.join('\n') + '\n'),
)
.map(([attrs, code]) => ({
type: 'codeBlock',
...(Object.keys(attrs).length ? { attrs } : {}),
content: [{ type: 'text', text: code }],
}));
}
const inline =
typeName === 'heading'
? headingInlineContentArb
: typeName === 'detailsSummary'
? plainInlineContentArb
: inlineContentArb;
return fc
.tuple(nodeAttrsArb(typeName, 'p1'), inline)
.map(([attrs, content]) => ({
type: typeName,
...(Object.keys(attrs).length ? { attrs } : {}),
content,
}));
}
// ---------------------------------------------------------------------------
// Coordinated builders: table (atomic, rectangular, column-consistent align)
// and columns (layout coupled to count, bodies walked).
// ---------------------------------------------------------------------------
/** A rectangular GFM-safe table (mirrors the flat suite's proven shape). */
function tableArb(): fc.Arbitrary<any> {
return fc.integer({ min: 1, max: 3 }).chain((cols) => {
// One alignment per COLUMN, identical on header + every body cell, so the
// second export cannot re-align and churn.
const alignsArb = fc.array(fc.constantFrom(undefined, 'left', 'center', 'right'), {
minLength: cols,
maxLength: cols,
});
const cell = (header: boolean, align?: string) =>
phraseArb.map((t) => ({
type: header ? 'tableHeader' : 'tableCell',
attrs: { colspan: 1, rowspan: 1, ...(align ? { align } : {}) },
content: [{ type: 'paragraph', content: [{ type: 'text', text: t }] }],
}));
return alignsArb.chain((aligns) => {
const headerRow = fc
.tuple(...aligns.map((a) => cell(true, a)))
.map((cells) => ({ type: 'tableRow', content: cells }));
const bodyRow = fc
.tuple(...aligns.map((a) => cell(false, a)))
.map((cells) => ({ type: 'tableRow', content: cells }));
return fc
.tuple(headerRow, fc.array(bodyRow, { minLength: 1, maxLength: 2 }))
.map(([h, body]) => ({ type: 'table', content: [h, ...body] }));
});
});
}
/** A columns block: layout ↔ count coupled, each column body walked as blocks. */
function columnsArb(depth: number, budget: number): fc.Arbitrary<any> {
return fc
.constantFrom('two_equal', 'three_equal', 'left_sidebar', 'right_sidebar')
.chain((layout) => {
const count = layout === 'three_equal' ? 3 : 2;
const columnType: any = schema.nodes.column;
// Split the remaining budget across the fixed number of columns.
const per = Math.max(2, Math.floor((budget - 1) / count));
return nodeAttrsArb('columns', 'p1', { layout, widthMode: 'normal' }).chain((attrs) =>
fc
.tuple(
...Array.from({ length: count }, () =>
fillMatch(columnType.contentMatch, depth + 1, per, 'column').map(({ children }) => ({
type: 'column',
content: children,
})),
),
)
.map((cols) => ({ type: 'columns', attrs, content: cols })),
);
});
}
// ---------------------------------------------------------------------------
// The ContentMatch walk.
// ---------------------------------------------------------------------------
/** Count every node in a subtree (block + inline), for budget accounting. */
function countNodes(node: any): number {
let n = 1;
for (const c of node.content ?? []) n += countNodes(c);
return n;
}
/** Build a single child node of a given type at `depth`, within `budget`. */
function blockNode(typeName: string, depth: number, budget: number): fc.Arbitrary<any> {
if (typeName === 'table') return tableArb();
if (typeName === 'columns') return columnsArb(depth, budget);
if (isTextblock(typeName)) return textblockArb(typeName);
if (isLeaf(typeName)) return leafArb(typeName);
// Generic container: attrs from 'p1', block content from the automaton walk.
const nt: any = schema.nodes[typeName];
return nodeAttrsArb(typeName, 'p1').chain((attrs) =>
fillMatch(nt.contentMatch, depth, budget - 1, typeName).map(({ children }) => {
const node: any = { type: typeName };
if (Object.keys(attrs).length) node.attrs = attrs;
if (children.length) node.content = children;
return node;
}),
);
}
/**
* Fill a container's block content by walking its ContentMatch automaton from
* `match`. Returns the children array plus the budget left after them.
*/
function fillMatch(
match: any,
depth: number,
budget: number,
parentType: string,
): fc.Arbitrary<{ children: any[]; budget: number }> {
const canStop = match.validEnd;
// A child lives at depth+1; only pick it if its subtree can complete within
// MAX_DEPTH. This headroom rule is what makes the walk deadlock-free.
const headroom = MAX_DEPTH - (depth + 1);
const edges: { t: string; next: any }[] = [];
if (headroom >= 0) {
for (let i = 0; i < match.edgeCount; i++) {
const edge = match.edge(i);
const t = edge.type.name;
if (EXCLUDED.has(t) || t === 'text') continue;
if (!childAllowedUnder(t, parentType)) continue;
if ((MIN_DEPTH[t] ?? Infinity) > headroom) continue;
edges.push({ t, next: edge.next });
}
}
// Decide the next action: STOP (if allowed) or extend with one more child.
// Bias toward stopping when the budget is spent; force a child only when the
// match is not yet at a valid end.
const pool: { weight: number; arbitrary: fc.Arbitrary<{ t: string; next: any } | null> }[] = [];
const canGo = edges.length > 0 && (budget > 0 || !canStop);
if (canStop) {
// Stop is weighted higher when the budget is low so docs stay bounded.
pool.push({ weight: budget > 0 ? 2 : 5, arbitrary: fc.constant(null) });
}
if (canGo && !(canStop && budget <= 0)) {
pool.push({ weight: 3, arbitrary: fc.constantFrom(...edges) });
}
// Forced continuation: not a valid end yet and (budget exhausted) — must place
// a mandatory child regardless of budget.
if (pool.length === 0) {
if (edges.length > 0) {
pool.push({ weight: 1, arbitrary: fc.constantFrom(...edges) });
} else {
// No legal child and not required to place one: stop with what we have.
return fc.constant({ children: [], budget });
}
}
return fc.oneof(...pool).chain((choice) => {
if (choice === null) return fc.constant({ children: [], budget });
return blockNode(choice.t, depth + 1, budget).chain((node) => {
const cost = countNodes(node);
return fillMatch(choice.next, depth, budget - cost, parentType).map(
({ children, budget: left }) => ({
children: [node, ...children],
budget: left,
}),
);
});
});
}
/**
* The nested-document arbitrary: a valid, arbitrarily-deep ProseMirror doc built
* by walking the schema from the document root. Attrs stay in the round-trip-safe
* 'p1' space; inline content reuses the byte-stable flat corpus.
*/
export const docArb: fc.Arbitrary<any> = fillMatch(
schema.nodes.doc.contentMatch,
0,
NODE_BUDGET,
'doc',
).map(({ children }) => ({ type: 'doc', content: children }));
/** The precomputed minDepth table, exported for inspection/debugging. */
export { MIN_DEPTH };
@@ -0,0 +1,29 @@
import { describe, expect, it } from 'vitest';
import { envInt } from './env-int.js';
// Unit coverage for the shared PROPERTY_SEED / PROPERTY_NUM_RUNS parser. The key
// contract is that an explicit "0" is honored (a valid fast-check seed) while
// empty/absent/non-numeric fall back to the default.
describe('envInt', () => {
it('honors an explicit "0" (does not fall back)', () => {
expect(envInt('0', 42)).toBe(0);
});
it('falls back on empty string', () => {
expect(envInt('', 42)).toBe(42);
});
it('falls back on undefined', () => {
expect(envInt(undefined, 42)).toBe(42);
});
it('falls back on a non-numeric string', () => {
expect(envInt('abc', 42)).toBe(42);
});
it('parses a plain integer string', () => {
expect(envInt('300', 42)).toBe(300);
});
it('parses a negative integer', () => {
expect(envInt('-5', 42)).toBe(-5);
});
it('parses a large integer', () => {
expect(envInt('1073741824', 42)).toBe(1073741824);
});
});
@@ -0,0 +1,10 @@
/**
* Parse an integer-ish environment variable with a default fallback.
*
* Used to read PROPERTY_SEED / PROPERTY_NUM_RUNS in the generative property
* suites. An unset/empty/non-numeric value falls back to `dflt`, but an explicit
* `"0"` is honored (a valid fast-check seed) `Number(x) || dflt` would wrongly
* swallow 0. See flat-roundtrip.property.test.ts / nested-roundtrip.property.test.ts.
*/
export const envInt = (v: string | undefined, dflt: number): number =>
v !== undefined && v !== '' && Number.isFinite(Number(v)) ? Number(v) : dflt;
@@ -0,0 +1,290 @@
import { describe, expect, it, vi } from 'vitest';
import fc from 'fast-check';
// Real converter, imported the same way the sibling property test does.
import { convertProseMirrorToMarkdown } from '../../src/lib/markdown-converter.js';
// Importing markdownToProseMirror mutates the global DOM via jsdom at module
// load (expected, required for @tiptap/html's generateJSON under Node).
import { markdownToProseMirror } from '../../src/lib/markdown-to-prosemirror.js';
import { docsCanonicallyEqual, canonicalizeContent } from '../../src/lib/index.js';
import { firstDivergence } from '../roundtrip-helpers.js';
import {
schema,
allSchemaAttrKeys,
allSchemaMarkAttrKeys,
attrIsValueFuzzed,
} from './attr-arbitraries.js';
import {
buildGenerators,
coveredTypes,
KNOWN_UNCOVERED,
} from './node-generators.js';
import { envInt } from './env-int.js';
// ── Attribute-value coverage allowlist ──────────────────────────────────────
// The node/mark completeness contract guarantees every TYPE is generated, but
// NOT that every attribute is exercised at a NON-DEFAULT value. An attribute
// with no `arb` in attr-arbitraries.ts is only ever tested at absent/default —
// an INVISIBLE coverage hole (the reviewer's concern). This allowlist makes that
// hole EXPLICIT: it is the exact set of attrs deliberately not value-fuzzed, so
// a NEW attribute (or a newly-frozen one) that lands in this bucket flips the
// snapshot test red and forces a reviewer to classify it. Each belongs to one of:
// - internal/opaque ids & placeholders (attachmentId, slugId, placeholder,
// creatorId, anchorId, mime) — no meaningful non-default to assert. These stay
// frozen: their value is an opaque token carried verbatim, not a round-trip
// shape worth fuzzing;
// - ACCEPTED limitations with no md representation (indent, callout.icon,
// orderedList.type, table spans/bg/colwidth).
// The media dimension/family attrs (image/video/youtube/drawio/excalidraw/pdf/
// embed width/height/align/size/aspectRatio/caption/title/alt) that were
// previously "deferred to a later PR" are IMPLEMENTED (value-fuzzed) in THIS PR
// via the OVERRIDES table in attr-arbitraries.ts — they ride in the discriminator
// comment JSON and round-trip byte-stably, so they are no longer allowlisted.
const ATTR_VALUE_FUZZ_ALLOWLIST = new Set<string>([
'attachment.attachmentId', 'attachment.mime', 'attachment.placeholder', 'attachment.size',
'audio.attachmentId', 'audio.placeholder', 'audio.size',
'callout.icon',
'drawio.attachmentId',
'excalidraw.attachmentId',
'heading.indent',
'image.attachmentId', 'image.placeholder',
'mention.anchorId', 'mention.creatorId', 'mention.slugId',
'orderedList.type', 'paragraph.indent',
'pdf.attachmentId', 'pdf.placeholder',
'tableCell.backgroundColor', 'tableCell.backgroundColorName', 'tableCell.colspan',
'tableCell.colwidth', 'tableCell.rowspan',
'tableHeader.backgroundColor', 'tableHeader.backgroundColorName', 'tableHeader.colspan',
'tableHeader.colwidth', 'tableHeader.rowspan',
'video.attachmentId', 'video.placeholder',
]);
// ── MARK attribute-value coverage ───────────────────────────────────────────
// Marks are fuzzed by the text generator (text-arbitraries.ts markedTextRunArb),
// not the node OVERRIDES table, so their value-fuzz coverage is tracked with this
// separate registry — otherwise the "no invisible coverage hole" guarantee would
// hold for node attrs only, and a new mark attr (or a new attributed mark) would
// silently escape the fuzz set. Every schema mark attr must be in exactly one of:
// MARK_ATTR_FUZZED — actually driven at a non-default value by the generator;
// MARK_ATTR_ALLOWLIST — deliberately not value-fuzzed, with a reason.
const MARK_ATTR_FUZZED = new Set<string>([
'mark:link.href', // markedTextRunArb sets a random webUrl href
'mark:link.title', // ...and an optional letter-bearing title
'mark:highlight.color', // highlight mark carries a generated color
'mark:textStyle.color', // textStyle mark carries a generated color
'mark:comment.commentId', // comment anchor id (alphanumeric token)
'mark:comment.resolved', // comment resolved flag (rides only when true)
]);
const MARK_ATTR_ALLOWLIST = new Set<string>([
// link presentational/routing attrs: not part of the markdown link surface the
// converter emits (it round-trips href + title only), so there is no
// non-default value to assert here — a deferred concern for a link-specific
// fixture, not the flat generative pass.
'mark:link.internal',
'mark:link.target',
'mark:link.rel',
'mark:link.class',
]);
// Each run does a real convert + marked + jsdom parse (~ms). Give ample headroom
// so the suite is deterministic regardless of parallel worker load (like the
// sibling property file).
vi.setConfig({ testTimeout: 30000 });
// ---------------------------------------------------------------------------
// #351 PR 1 — GENERATIVE (property-based) round-trip over FLAT (single-node)
// documents at the ATTRIBUTE level.
//
// We assert three invariants for ANY generated valid flat document `d`
// (pmToMd = convertProseMirrorToMarkdown, mdToPm = markdownToProseMirror):
//
// P1 — semantic round-trip (nothing lost):
// docsCanonicallyEqual(await mdToPm(pmToMd(d)), d) === true
// P2 — byte fixpoint (anti "GS-EDIT-REVERT" churn):
// pmToMd(await mdToPm(pmToMd(d))) === pmToMd(d)
// P3 — totality: neither converter throws; bounded.
//
// The generators are schema-DERIVED (attribute lists come from
// schema.nodes[type].spec.attrs) and stay inside the round-trip-supported space
// proven empirically by probing the live converter (see attr-arbitraries.ts and
// text-arbitraries.ts). P1 runs over the safe attribute space; P2/P3 run over
// the wider 'fuzz' space that also injects degenerate attribute states, which
// P2 tolerates via a one-time first-pass normalization and P3 via totality only.
// ---------------------------------------------------------------------------
// Fixed seed so every failure is reproducible; fast-check also prints the
// shrunk counterexample. numRuns starts modest to keep CI under budget — the
// issue's CI target is ~300-500 per property. Both are overridable via the
// PROPERTY_SEED / PROPERTY_NUM_RUNS env vars (an invalid/empty value → NaN →
// falls back to the default below): the nightly cron
// (.github/workflows/nightly-property.yml) cranks NUM_RUNS to ~5000 with a
// random seed to hunt for deeper counterexamples. Each property runs over the
// UNION (fc.oneof) of all flat node generators, so the runs are shared across
// node types (one test per property keeps the jsdom import cost and memory
// bounded — a per-generator × per-property matrix is ~200 heavy tests that
// OOMs the worker).
// An unset/empty/non-numeric value falls back to the default; an explicit 0 is
// honored (a valid fast-check seed) — `Number(x) || default` would swallow it.
// The parser is shared with the nested suite (env-int.ts) and unit-tested there.
const SEED = envInt(process.env.PROPERTY_SEED, 20250705);
const NUM_RUNS = envInt(process.env.PROPERTY_NUM_RUNS, 300);
const P1_GENERATORS = buildGenerators('p1');
const FUZZ_GENERATORS = buildGenerators('fuzz');
// Union arbitraries: a single draw picks one node generator, then a document
// from it. On failure fast-check prints the shrunk counterexample doc, which
// names the offending node type directly.
const p1Union = fc.oneof(...P1_GENERATORS.map((g) => g.arb));
const fuzzUnion = fc.oneof(...FUZZ_GENERATORS.map((g) => g.arb));
async function roundTrip(doc: unknown): Promise<{ md1: string; md2: string; doc2: any }> {
const md1 = convertProseMirrorToMarkdown(doc);
const doc2 = await markdownToProseMirror(md1);
const md2 = convertProseMirrorToMarkdown(doc2);
return { md1, md2, doc2 };
}
describe('#351 flat generative round-trip — completeness contract', () => {
it('every schema node and mark is covered by a generator or explicitly allowlisted', () => {
const covered = coveredTypes();
const uncovered: string[] = [];
for (const nodeType of Object.keys(schema.nodes)) {
if (covered.has(nodeType)) continue;
if (nodeType in KNOWN_UNCOVERED) continue;
uncovered.push(`node:${nodeType}`);
}
for (const markType of Object.keys(schema.marks)) {
if (covered.has(`mark:${markType}`)) continue;
if (markType in KNOWN_UNCOVERED) continue;
uncovered.push(`mark:${markType}`);
}
// A new node/mark added to the schema with no generator AND no allowlist
// entry MUST turn this test red — that is the whole point (no silent blind
// spots).
expect(
uncovered,
`these schema types have no generator and no KNOWN_UNCOVERED reason:\n ${uncovered.join(
'\n ',
)}`,
).toEqual([]);
});
it('every KNOWN_UNCOVERED entry is a real schema type (no stale allowlist rows)', () => {
const all = new Set([...Object.keys(schema.nodes), ...Object.keys(schema.marks)]);
for (const t of Object.keys(KNOWN_UNCOVERED)) {
expect(all.has(t), `stale KNOWN_UNCOVERED entry: ${t}`).toBe(true);
}
});
it('every attribute is value-fuzzed OR explicitly allowlisted (no invisible hole)', () => {
// Makes the "generic-frozen" coverage hole VISIBLE: any schema attr not
// exercised at a non-default value must be a KNOWN entry in the allowlist.
// A new attr (or one that loses its `arb`) that falls into the not-fuzzed
// bucket without an allowlist row turns this red — no silent blind spots.
const unaccounted: string[] = [];
for (const key of allSchemaAttrKeys()) {
const i = key.indexOf('.');
const fuzzed = attrIsValueFuzzed(key.slice(0, i), key.slice(i + 1));
if (!fuzzed && !ATTR_VALUE_FUZZ_ALLOWLIST.has(key)) unaccounted.push(key);
}
expect(
unaccounted,
`these attrs are not value-fuzzed and not in ATTR_VALUE_FUZZ_ALLOWLIST:\n ${unaccounted.join(
'\n ',
)}`,
).toEqual([]);
});
it('the attribute allowlist has no stale rows (every entry is really not-fuzzed)', () => {
const notFuzzed = new Set(
allSchemaAttrKeys().filter((key) => {
const i = key.indexOf('.');
return !attrIsValueFuzzed(key.slice(0, i), key.slice(i + 1));
}),
);
for (const key of ATTR_VALUE_FUZZ_ALLOWLIST) {
expect(
notFuzzed.has(key),
`stale allowlist row (attr is now value-fuzzed, remove it): ${key}`,
).toBe(true);
}
});
it('every MARK attribute is value-fuzzed OR allowlisted (no invisible hole)', () => {
// The node guard above covers node attrs; marks are fuzzed by the text
// generator, so their coverage is tracked separately. A new mark attr (or a
// newly-attributed mark) that lands in neither set turns this red.
const unaccounted: string[] = [];
for (const key of allSchemaMarkAttrKeys()) {
if (!MARK_ATTR_FUZZED.has(key) && !MARK_ATTR_ALLOWLIST.has(key)) {
unaccounted.push(key);
}
}
expect(
unaccounted,
`these mark attrs are neither in MARK_ATTR_FUZZED nor MARK_ATTR_ALLOWLIST:\n ${unaccounted.join(
'\n ',
)}`,
).toEqual([]);
});
it('the MARK fuzz/allowlist sets have no stale rows (every entry is a real schema mark attr)', () => {
const all = new Set(allSchemaMarkAttrKeys());
for (const key of [...MARK_ATTR_FUZZED, ...MARK_ATTR_ALLOWLIST]) {
expect(all.has(key), `stale mark-attr registry row: ${key}`).toBe(true);
}
});
});
describe('#351 flat generative round-trip — properties', () => {
it('generator validity: every generated doc passes schema.check()', () => {
// A generator that emits an invalid ProseMirror document is a GENERATOR bug.
fc.assert(
fc.property(fuzzUnion, (doc) => {
schema.nodeFromJSON(doc).check(); // throws on an invalid doc
return true;
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
it('P1 — semantic round-trip: docsCanonicallyEqual(mdToPm(pmToMd(d)), d)', async () => {
await fc.assert(
fc.asyncProperty(p1Union, async (doc) => {
const { doc2 } = await roundTrip(doc);
if (!docsCanonicallyEqual(doc2, doc)) {
// Surface the precise divergence in the failure message.
const div = firstDivergence(
JSON.parse(JSON.stringify(canonicalizeContent(doc2))),
JSON.parse(JSON.stringify(canonicalizeContent(doc))),
);
throw new Error(
`P1 divergence @ ${div?.path}: got=${JSON.stringify(div?.a)} want=${JSON.stringify(div?.b)}`,
);
}
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
it('P2 — byte fixpoint: pmToMd(mdToPm(pmToMd(d))) === pmToMd(d)', async () => {
await fc.assert(
fc.asyncProperty(fuzzUnion, async (doc) => {
const { md1, md2 } = await roundTrip(doc);
expect(md2).toBe(md1);
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
it('P3 — totality: neither converter throws', async () => {
await fc.assert(
fc.asyncProperty(fuzzUnion, async (doc) => {
// Throwing here fails the property; fast-check shrinks to a minimal doc.
await roundTrip(doc);
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
});
@@ -0,0 +1,193 @@
import { describe, expect, it, vi } from 'vitest';
import fc from 'fast-check';
// Real converters. Importing markdownToProseMirror (transitively, via index)
// mutates the global DOM via jsdom at module load — expected, required for
// @tiptap/html's generateJSON under Node (same as the flat sibling suite).
import {
convertProseMirrorToMarkdown,
markdownToProseMirror,
docsCanonicallyEqual,
canonicalizeContent,
} from '../../src/lib/index.js';
import { firstDivergence } from '../roundtrip-helpers.js';
import { schema, docArb } from './doc-generator.js';
import { envInt } from './env-int.js';
// Each run does a real convert + jsdom parse; give ample headroom so the suite
// is deterministic under parallel worker load (matching the flat sibling suite).
vi.setConfig({ testTimeout: 60000 });
// ---------------------------------------------------------------------------
// #351 PR 2 — GENERATIVE round-trip over NESTED (whole-document) docs produced
// by the ContentMatch random walk (doc-generator.ts). The invariants mirror the
// flat suite, plus a parser-fuzz totality property (P4):
//
// P1 — semantic round-trip: docsCanonicallyEqual(mdToPm(pmToMd(d)), d)
// P2 — byte fixpoint (2nd pass): pmToMd(mdToPm(pmToMd(d))) === pmToMd(d)
// (the FIRST pass may normalize once; the SECOND pass must be a fixpoint)
// P3 — totality: neither converter throws; bounded.
// P4 — parser fuzz totality: for ANY string, markdownToProseMirror does NOT
// throw and returns a SCHEMA-VALID document.
//
// GUARDRAIL: a P1/P2/P3/P4 failure means the generator FOUND A REAL CONVERTER
// BUG. These invariants are kept STRICT — no it.fails / skip / weakening. A
// failure prints the shrunk minimal counterexample for triage.
// ---------------------------------------------------------------------------
// Both are overridable via the PROPERTY_SEED / PROPERTY_NUM_RUNS env vars (an
// invalid/empty value → NaN → falls back to the default below); the nightly
// cron (.github/workflows/nightly-property.yml) cranks NUM_RUNS up with a
// random seed to hunt for deeper counterexamples.
// An unset/empty/non-numeric value falls back to the default; an explicit 0 is
// honored (a valid fast-check seed) — `Number(x) || default` would swallow it.
// The parser is shared with the flat suite (env-int.ts) and unit-tested there.
const SEED = envInt(process.env.PROPERTY_SEED, 20250705);
// The nested walk builds far heavier docs than the flat suite (each P1/P2 run
// parses the emitted markdown through jsdom), so keep the run count moderate to
// hold runtime and worker memory in budget while still exercising deep
// structures. P4 (cheap string parsing) runs at a higher count below.
const NUM_RUNS = envInt(process.env.PROPERTY_NUM_RUNS, 100);
const pmToMd = (doc: unknown): string => convertProseMirrorToMarkdown(doc);
const mdToPm = (md: string): Promise<any> => markdownToProseMirror(md);
async function roundTrip(doc: unknown): Promise<{ md1: string; md2: string; doc2: any }> {
const md1 = pmToMd(doc);
const doc2 = await mdToPm(md1);
const md2 = pmToMd(doc2);
return { md1, md2, doc2 };
}
describe('#351 nested generative round-trip — generator validity', () => {
it('every generated nested doc passes schema.nodeFromJSON(...).check()', () => {
// A nested generator that emits an invalid ProseMirror document is a
// GENERATOR bug — the ContentMatch walk must only produce schema-valid docs.
fc.assert(
fc.property(docArb, (doc) => {
schema.nodeFromJSON(doc).check(); // throws on an invalid doc
return true;
}),
{ numRuns: NUM_RUNS * 2, seed: SEED },
);
});
});
// ── STATUS: P1/P2/P3/P4 all GREEN. The nested generator originally surfaced a
// batch of real converter bugs; all were fixed in the serializer/parser (see the
// #351 hand-off). For the record, the classes it found and that are now fixed:
// • Loose (multi-block) list items / task items / callouts / details bodies were
// joined with a single "\n", so every block after the first merged into the
// first paragraph on re-parse (silent content loss) — now blank-line separated.
// • Paragraph `textAlign` was dropped inside a TIGHT list item (no <p> host).
// • A nested codeBlock lost its trailing newline on the raw-HTML path.
// • Media (embed/video/youtube/drawio/excalidraw) inside `columns` churned a
// default `data-align` and coerced embed's numeric width/height to strings.
// • pageBreak / pageEmbed / subpages / transclusion were dropped when nested in
// blockquote / callout / details / list item (standalone-comment position).
// • Callouts nested in a list item or a blockquote (` > [!type]` / `> > [!type]`)
// were re-parsed as plain blockquotes (prefix-unaware callout preprocessor).
// • Two adjacent sibling lists sharing a marker family (bulletList/taskList →
// `<ul>`; orderedList → `<ol>`) merged into one list on re-parse — and for the
// cross-type case (taskList beside bulletList) the merged `<ul>` LOST every
// taskItem checkbox. The serializer now emits a `<!-- -->` separator between
// such adjacent lists (markdown-converter.ts renderBlockChildren), so they stay
// distinct and round-trip; the generator therefore emits them freely again.
describe('#351 nested generative round-trip — properties', () => {
it('P1 — semantic round-trip: docsCanonicallyEqual(mdToPm(pmToMd(d)), d)', async () => {
await fc.assert(
fc.asyncProperty(docArb, async (doc) => {
const { doc2 } = await roundTrip(doc);
if (!docsCanonicallyEqual(doc2, doc)) {
const div = firstDivergence(
JSON.parse(JSON.stringify(canonicalizeContent(doc2))),
JSON.parse(JSON.stringify(canonicalizeContent(doc))),
);
throw new Error(
`P1 divergence @ ${div?.path}: got=${JSON.stringify(div?.a)} want=${JSON.stringify(div?.b)}`,
);
}
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
it('P2 — byte fixpoint: pmToMd(mdToPm(pmToMd(d))) === pmToMd(d)', async () => {
await fc.assert(
fc.asyncProperty(docArb, async (doc) => {
const { md1, md2 } = await roundTrip(doc);
expect(md2).toBe(md1);
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
it('P3 — totality: neither converter throws', async () => {
await fc.assert(
fc.asyncProperty(docArb, async (doc) => {
await roundTrip(doc);
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
});
// ---------------------------------------------------------------------------
// P4 — parser fuzz. Independent of the doc generator: for ANY input string the
// PARSER (markdownToProseMirror) must be TOTAL — never throw — and must always
// return a schema-valid document. The corpus mixes raw unicode strings with
// strings assembled from markdown-significant fragments (headings, list bullets,
// fences, pipes, thematic breaks, HTML-ish snippets) to probe the block/inline
// parsers on hostile but plausible input.
// ---------------------------------------------------------------------------
const mdFragmentArb: fc.Arbitrary<string> = fc.constantFrom(
'# ', '## ', '### ###', '- ', '* ', '+ ', '1. ', '> ', '>> ',
'```', '```js', '~~~', '---', '***', '___', '| a | b |', '|---|---|',
'[link](http://x)', '![img](http://x)', '**', '__', '~~', '`code`',
'<div>', '</div>', '<b>', '<!-- c -->', '<table>', '<br>', '&amp;',
'\t', '\n', ' ', '\\', '^[fn]', '[^1]:', '- [ ] ', '- [x] ',
'$$', '$x$', ':::', '{.class}', '\u0000', '\uFEFF', '😀', 'مرحبا',
);
// Full-unicode strings (fast-check v4 replaced fullUnicodeString with the
// `unit: 'binary'` string option, which draws over the whole code-point range).
const fullUnicodeStringArb = (max?: number) =>
fc.string({ unit: 'binary', ...(max !== undefined ? { maxLength: max } : {}) });
const assembledMarkdownArb: fc.Arbitrary<string> = fc
.array(fc.oneof(mdFragmentArb, fc.string(), fullUnicodeStringArb(8)), {
minLength: 1,
maxLength: 12,
})
.map((parts) => parts.join(''));
const parserInputArb: fc.Arbitrary<string> = fc.oneof(
{ weight: 2, arbitrary: fc.string() },
{ weight: 2, arbitrary: fullUnicodeStringArb() },
{ weight: 3, arbitrary: assembledMarkdownArb },
{ weight: 1, arbitrary: fc.array(mdFragmentArb, { minLength: 1, maxLength: 8 }).map((p) => p.join('\n')) },
);
describe('#351 parser fuzz — totality on arbitrary input (P4)', () => {
it('P4 — markdownToProseMirror never throws and always returns a schema-valid doc', async () => {
await fc.assert(
fc.asyncProperty(parserInputArb, async (s) => {
let result: any;
try {
result = await mdToPm(s);
} catch (e: any) {
throw new Error(`P4 parser THREW on input ${JSON.stringify(s)}: ${e?.message ?? e}`);
}
try {
schema.nodeFromJSON(result).check();
} catch (e: any) {
throw new Error(
`P4 parser produced an INVALID doc for input ${JSON.stringify(s)}: ${e?.message ?? e}\n` +
`doc=${JSON.stringify(result).slice(0, 600)}`,
);
}
}),
{ numRuns: NUM_RUNS * 2, seed: SEED },
);
});
});
@@ -0,0 +1,310 @@
/**
* Flat single-node document generators (#351, PR 1).
*
* For every schema node type that can stand alone, a fast-check arbitrary
* producing `{ type:'doc', content:[ <the target node> ] }` with generated attrs
* (via nodeAttrsArb) and the minimal REQUIRED immediate children the schema
* demands (a heading's inline text, a listItem's one paragraph, a table's
* minimal rows, details' summary+content, a callout's one paragraph). Kept
* FLAT: a single target node, no deep nesting nested structural generation is
* PR 2.
*
* The `mode` threads through to the attribute arbitraries:
* - 'p1' : the round-trip-safe attribute space (P1 semantic round-trip).
* - 'fuzz' : adds degenerate attribute states (P2 byte-fixpoint tolerates the
* one-time normalization; P3 only needs totality).
*
* A COMPLETENESS CONTRACT (see flat-roundtrip.property.test.ts) enumerates the
* whole schema and asserts every node/mark is EITHER produced by a generator
* here OR listed in KNOWN_UNCOVERED with a reason so a new schema type with no
* generator turns the suite RED.
*/
import fc from 'fast-check';
import { type AttrMode, nodeAttrsArb } from './attr-arbitraries.js';
import {
inlineContentArb,
headingInlineContentArb,
plainInlineContentArb,
phraseArb,
markedTextRunArb,
} from './text-arbitraries.js';
const doc = (node: any) => ({ type: 'doc', content: [node] });
const para = (content: any[]) => ({ type: 'paragraph', content });
/** A named flat-document generator. */
export interface NamedGen {
name: string;
arb: fc.Arbitrary<any>;
}
// ---------------------------------------------------------------------------
// Per-target generators, each a function of mode.
// ---------------------------------------------------------------------------
const gen = {
paragraph: (m: AttrMode) =>
fc.tuple(nodeAttrsArb('paragraph', m), inlineContentArb).map(([attrs, content]) =>
doc({ type: 'paragraph', attrs, content }),
),
heading: (m: AttrMode) =>
fc.tuple(nodeAttrsArb('heading', m), headingInlineContentArb).map(([attrs, content]) =>
doc({ type: 'heading', attrs, content }),
),
blockquote: (_m: AttrMode) =>
inlineContentArb.map((content) => doc({ type: 'blockquote', content: [para(content)] })),
bulletList: (_m: AttrMode) =>
fc
.array(inlineContentArb, { minLength: 1, maxLength: 3 })
.map((items) =>
doc({
type: 'bulletList',
content: items.map((c) => ({ type: 'listItem', content: [para(c)] })),
}),
),
orderedList: (m: AttrMode) =>
fc
.tuple(nodeAttrsArb('orderedList', m), fc.array(inlineContentArb, { minLength: 1, maxLength: 3 }))
.map(([attrs, items]) =>
doc({
type: 'orderedList',
attrs,
content: items.map((c) => ({ type: 'listItem', content: [para(c)] })),
}),
),
taskList: (m: AttrMode) =>
fc
.array(fc.tuple(nodeAttrsArb('taskItem', m), inlineContentArb), { minLength: 1, maxLength: 3 })
.map((items) =>
doc({
type: 'taskList',
content: items.map(([attrs, c]) => ({ type: 'taskItem', attrs, content: [para(c)] })),
}),
),
codeBlock: (m: AttrMode) =>
fc
.tuple(
nodeAttrsArb('codeBlock', m),
// A fenced code block always re-imports with a TRAILING NEWLINE in its
// text (empirically confirmed). Author the newline so the doc is already
// at the round-trip fixpoint (supported-space shaping, not a masked bug).
fc.array(phraseArb, { minLength: 1, maxLength: 3 }).map((lines) => lines.join('\n') + '\n'),
)
.map(([attrs, code]) =>
doc({ type: 'codeBlock', attrs, content: [{ type: 'text', text: code }] }),
),
horizontalRule: (_m: AttrMode) => fc.constant(doc({ type: 'horizontalRule' })),
pageBreak: (_m: AttrMode) => fc.constant(doc({ type: 'pageBreak' })),
image: (m: AttrMode) => nodeAttrsArb('image', m).map((attrs) => doc({ type: 'image', attrs })),
callout: (m: AttrMode) =>
fc.tuple(nodeAttrsArb('callout', m), inlineContentArb).map(([attrs, content]) =>
doc({ type: 'callout', attrs, content: [para(content)] }),
),
mathBlock: (m: AttrMode) =>
nodeAttrsArb('mathBlock', m).map((attrs) => doc({ type: 'mathBlock', attrs })),
details: (m: AttrMode) =>
fc
.tuple(nodeAttrsArb('details', m), plainInlineContentArb, inlineContentArb)
.map(([attrs, summary, body]) =>
doc({
type: 'details',
attrs,
content: [
{ type: 'detailsSummary', content: summary },
{ type: 'detailsContent', content: [para(body)] },
],
}),
),
table: (_m: AttrMode) =>
fc.integer({ min: 1, max: 3 }).chain((cols) => {
// GFM alignment is column-wide (encoded in the header separator), so a
// column's alignment must be identical on the header and every body cell,
// else the second export re-aligns and churns. Pick ONE align per column.
const alignsArb = fc.array(fc.constantFrom(undefined, 'left', 'center', 'right'), {
minLength: cols,
maxLength: cols,
});
const cell = (header: boolean, align?: string) =>
phraseArb.map((t) => ({
type: header ? 'tableHeader' : 'tableCell',
// colspan/rowspan pinned to 1 (GFM cannot express spans); optional
// column-consistent align.
attrs: { colspan: 1, rowspan: 1, ...(align ? { align } : {}) },
content: [para([{ type: 'text', text: t }])],
}));
return alignsArb.chain((aligns) => {
const headerRow = fc
.tuple(...aligns.map((a) => cell(true, a)))
.map((cells) => ({ type: 'tableRow', content: cells }));
const bodyRow = fc
.tuple(...aligns.map((a) => cell(false, a)))
.map((cells) => ({ type: 'tableRow', content: cells }));
return fc
.tuple(headerRow, fc.array(bodyRow, { minLength: 1, maxLength: 2 }))
.map(([h, body]) => doc({ type: 'table', content: [h, ...body] }));
});
}),
columns: (m: AttrMode) =>
// Couple the column count to the layout so the two stay consistent
// (two_equal/left_sidebar/right_sidebar -> 2, three_equal -> 3).
fc
.constantFrom('two_equal', 'three_equal', 'left_sidebar', 'right_sidebar')
.chain((layout) => {
const count = layout === 'three_equal' ? 3 : 2;
return fc
.tuple(
nodeAttrsArb('columns', m, { layout, widthMode: 'normal' }),
fc.array(inlineContentArb, { minLength: count, maxLength: count }),
)
.map(([attrs, bodies]) =>
doc({
type: 'columns',
attrs,
content: bodies.map((c) => ({ type: 'column', content: [para(c)] })),
}),
);
}),
subpages: (m: AttrMode) =>
nodeAttrsArb('subpages', m).map((attrs) => doc({ type: 'subpages', attrs })),
audio: (m: AttrMode) => nodeAttrsArb('audio', m).map((attrs) => doc({ type: 'audio', attrs })),
video: (m: AttrMode) => nodeAttrsArb('video', m).map((attrs) => doc({ type: 'video', attrs })),
pdf: (m: AttrMode) => nodeAttrsArb('pdf', m).map((attrs) => doc({ type: 'pdf', attrs })),
youtube: (m: AttrMode) => nodeAttrsArb('youtube', m).map((attrs) => doc({ type: 'youtube', attrs })),
embed: (m: AttrMode) => nodeAttrsArb('embed', m).map((attrs) => doc({ type: 'embed', attrs })),
drawio: (m: AttrMode) => nodeAttrsArb('drawio', m).map((attrs) => doc({ type: 'drawio', attrs })),
excalidraw: (m: AttrMode) =>
nodeAttrsArb('excalidraw', m).map((attrs) => doc({ type: 'excalidraw', attrs })),
attachment: (m: AttrMode) =>
nodeAttrsArb('attachment', m).map((attrs) => doc({ type: 'attachment', attrs })),
htmlEmbed: (m: AttrMode) =>
nodeAttrsArb('htmlEmbed', m).map((attrs) => doc({ type: 'htmlEmbed', attrs })),
pageEmbed: (m: AttrMode) =>
nodeAttrsArb('pageEmbed', m).map((attrs) => doc({ type: 'pageEmbed', attrs })),
transclusionReference: (m: AttrMode) =>
nodeAttrsArb('transclusionReference', m).map((attrs) =>
doc({ type: 'transclusionReference', attrs }),
),
transclusionSource: (m: AttrMode) =>
fc.tuple(nodeAttrsArb('transclusionSource', m), inlineContentArb).map(([attrs, content]) =>
doc({ type: 'transclusionSource', attrs, content: [para(content)] }),
),
// A footnote reference PLUS its definition (the reference has no standalone
// markdown form without its definition — see KNOWN_UNCOVERED note for the
// bare reference). Both carry the same id. The definition body uses
// headingInlineContentArb (NO hard breaks): a footnote is serialized inline as
// `^[...]`, so a hard break inside it collapses to a single space on re-parse
// (empirically confirmed) — that is the container's markdown limitation, not
// an attribute-level concern. The reference-bearing paragraph is a NORMAL
// paragraph and keeps the full inline corpus.
footnotes: (m: AttrMode) =>
fc.tuple(fc.constantFrom('fn1', 'fn2', 'note'), inlineContentArb, headingInlineContentArb).map(
([id, refText, noteBody]) => ({
type: 'doc',
content: [
para([...refText, { type: 'footnoteReference', attrs: { id } }]),
{
type: 'footnotesList',
content: [{ type: 'footnoteDefinition', attrs: { id }, content: [para(noteBody)] }],
},
],
}),
),
// ── inline targets wrapped in a paragraph ────────────────────────────────
mention: (m: AttrMode) =>
nodeAttrsArb('mention', m).map((attrs) => doc(para([{ type: 'mention', attrs }]))),
mathInline: (m: AttrMode) =>
fc.tuple(phraseArb, nodeAttrsArb('mathInline', m)).map(([t, attrs]) =>
doc(para([{ type: 'text', text: t }, { type: 'mathInline', attrs }])),
),
status: (m: AttrMode) =>
nodeAttrsArb('status', m).map((attrs) => doc(para([{ type: 'status', attrs }]))),
hardBreak: (_m: AttrMode) =>
fc.tuple(phraseArb, phraseArb).map(([a, b]) =>
doc(para([{ type: 'text', text: a }, { type: 'hardBreak' }, { type: 'text', text: b }])),
),
// ── marks: a paragraph of marked runs (covers every mark type) ───────────
marksOnText: (_m: AttrMode) =>
fc.array(markedTextRunArb, { minLength: 1, maxLength: 5 }).map((runs) => {
// Merge adjacent same-mark runs (see text-arbitraries.normalizeInline).
const out: any[] = [];
for (const r of runs) {
const prev = out[out.length - 1];
if (prev && JSON.stringify(prev.marks ?? []) === JSON.stringify(r.marks ?? [])) {
prev.text += r.text;
} else out.push({ ...r });
}
return doc(para(out));
}),
};
/** Build the full list of named generators for a given mode. */
export function buildGenerators(mode: AttrMode): NamedGen[] {
return Object.entries(gen).map(([name, f]) => ({ name, arb: f(mode) }));
}
// ---------------------------------------------------------------------------
// Completeness contract support.
// ---------------------------------------------------------------------------
/**
* Schema node/mark types deliberately NOT covered by a P1/P2 generator, each
* with a one-line reason. Excluding a type means it is kept OUT of the round-
* trip generators it does NOT weaken any property.
*
* NOTE (empirical): the candidates the issue flagged for review pageEmbed,
* subpages, transclusionSource/Reference, mention, status were PROBED against
* the live converter and DO round-trip P1/P2 with placeholder ids, so they are
* COVERED by real generators rather than allowlisted here. The allowlist below
* holds only types with no standalone flat generator by construction.
*/
export const KNOWN_UNCOVERED: Record<string, string> = {
// The root node; it is the wrapper every generated doc already is, never a
// "target" content node, so it has no standalone generator of its own.
doc: 'the document root wrapper, not a content node with a standalone generator',
};
/** Recursively collect every node type and `mark:<type>` under a tree. */
export function collectTypes(node: any, seen = new Set<string>()): Set<string> {
if (!node || typeof node !== 'object') return seen;
if (node.type) seen.add(node.type);
for (const m of node.marks ?? []) if (m?.type) seen.add(`mark:${m.type}`);
for (const c of node.content ?? []) collectTypes(c, seen);
return seen;
}
/**
* Sample every generator and return the union of node/mark types they produce.
* Deterministic (fixed seed) so the completeness contract is stable.
*/
export function coveredTypes(seed = 12345, perGen = 60): Set<string> {
const seen = new Set<string>();
for (const { arb } of buildGenerators('p1')) {
for (const sample of fc.sample(arb, { numRuns: perGen, seed })) {
collectTypes(sample, seen);
}
}
return seen;
}
@@ -0,0 +1,258 @@
/**
* Hostile inline-text corpus for the generative flat-document round-trip suite
* (#351, PR 1).
*
* These arbitraries are a DIRECT PORT of the "supported space" guardrails that
* `test/markdown-roundtrip.property.test.ts` proved empirically against the live
* converter. That file's long header documents WHY each guardrail exists; rather
* than re-derive them, we reuse the exact same shapes here so the attribute-level
* generative suite inherits the same byte-stable text space. Each guardrail is
* cited back to that file below.
*
* The corpus deliberately spans the CommonMark / canon hostile alphabet
* (`* _ [ ] ( ) { } | < > & # ! ~ = + -`), unicode / emoji / RTL, and the legal
* mark combinations on runs (including the `code` mark, which the schema's
* `excludes: "_"` makes suppress every co-occurring mark so it is never
* combined with another mark in the byte-stable space).
*/
import fc from 'fast-check';
// ---------------------------------------------------------------------------
// Words and the hostile special-character alphabet.
// (Ported from markdown-roundtrip.property.test.ts, "Inline text arbitraries".)
// ---------------------------------------------------------------------------
/** Alphanumeric "word" (no markdown-significant characters). Length 1..6. */
export const wordArb = fc
.stringMatching(/^[A-Za-z0-9]{1,6}$/)
.filter((w) => w.length > 0);
/**
* A SINGLE markdown-significant character, emitted only as an isolated,
* space-flanked token. Every char the task calls out plus a few more; each was
* verified byte-stable in this position by the sibling property test.
*
* NOTE: the backtick (`) is DELIBERATELY excluded from free-floating plain text
* (it is a code-span delimiter that re-pairs globally). It is exercised only via
* the `code` mark and code blocks see markdown-roundtrip.property.test.ts.
*/
export const specialCharArb = fc.constantFrom(
'*', '_', '[', ']', '(', ')', '{', '}', '|', '<', '>', '&', '#', '!', '~', '=', '+', '-',
);
// A pinch of unicode / emoji / RTL, always word-like (no markdown specials) so
// it stays inside the space-flanked corpus. Kept letter/emoji-bearing so it is
// never coerced to a number (see letterPhraseArb rationale).
export const unicodeWordArb = fc.constantFrom(
'café', 'naïve', 'Zürich', 'Москва', 'こんにちは', '你好', '😀', '🚀x', 'مرحبا', 'שלום',
);
/**
* A "safe special" text string: a space-joined sequence of tokens that always
* BEGINS and ENDS with an alphanumeric word, with any isolated special chars (or
* unicode words) confined to the MIDDLE, each space-flanked by words.
*
* Both boundary guarantees matter (verbatim from the sibling test):
* * Leading word: the line never opens with a block/inline trigger
* (">", "*", "-", "#", "1." ...).
* * Trailing word: adjacent text runs CONCATENATE with no separator, so a run
* ending in a bare "<" beside a run starting with a letter would form a fake
* HTML tag. Ending every run with a word keeps every special internal and
* space-flanked even after concatenation.
*/
export const safeTextArb: fc.Arbitrary<string> = fc
.tuple(
wordArb,
fc.array(fc.oneof(wordArb, specialCharArb, unicodeWordArb), {
minLength: 0,
maxLength: 3,
}),
wordArb,
)
.map(([first, middle, last]) => [first, ...middle, last].join(' '));
/**
* A plain alphanumeric phrase (1..3 words) for places where even isolated
* specials are not wanted (e.g. code-block language, mention labels, status
* text, table cells rendered on the plain-markdown path).
*/
export const phraseArb: fc.Arbitrary<string> = fc
.array(wordArb, { minLength: 1, maxLength: 3 })
.map((ws) => ws.join(' '));
/**
* A phrase guaranteed to contain at least one letter. Used for image/media alt
* text and link titles: a PURELY numeric alt/title (e.g. "0") is parsed back as
* a NUMBER and then dropped by the converter's `value || ""` coercion not
* byte-stable. A letter anywhere keeps it a string. (Ported verbatim.)
*/
export const letterPhraseArb: fc.Arbitrary<string> = fc
.tuple(
fc.stringMatching(/^[A-Za-z]{1,4}$/),
fc.array(wordArb, { minLength: 0, maxLength: 2 }),
)
.map(([head, rest]) => [head, ...rest].join(' '));
/** A paren/space-free URL — safe inside markdown link/image `(...)` syntax. */
export const urlArb: fc.Arbitrary<string> = fc
.webUrl()
.filter((u) => !/[()\s]/.test(u));
// ---------------------------------------------------------------------------
// Marked inline runs.
// (Ported from markdown-roundtrip.property.test.ts "markedTextRunArb".)
// ---------------------------------------------------------------------------
/**
* A text run with an OPTIONAL single non-code formatting mark (bold/italic/
* strike/underline/superscript/subscript/spoiler), or a SOLE `code` mark, or a
* link, or an inline comment anchor. `code` is NEVER combined with another mark
* in the byte-stable space (that combination is a documented converter
* limitation the schema's `code` mark declares `excludes: "_"`). Marks wrap
* `safeTextArb`, which stays stable even when it contains isolated specials.
*
* The mark set here is broadened past the sibling test's {bold,italic,strike}
* to also cover underline / superscript / subscript / spoiler / textStyle /
* highlight (all single, non-code marks), so the marks-on-text generator
* exercises every mark the schema declares except the deliberately-excluded
* `code`+other combination.
*/
export const markedTextRunArb: fc.Arbitrary<any> = fc.oneof(
// Plain text.
safeTextArb.map((t) => ({ type: 'text', text: t })),
// Single formatting mark (attribute-free marks).
fc
.tuple(
safeTextArb,
fc.constantFrom('bold', 'italic', 'strike', 'underline', 'superscript', 'subscript', 'spoiler'),
)
.map(([t, m]) => ({ type: 'text', text: t, marks: [{ type: m }] })),
// highlight with a color attr.
fc
.tuple(safeTextArb, fc.constantFrom('#ffcc00', '#a0e0ff', 'yellow'))
.map(([t, color]) => ({ type: 'text', text: t, marks: [{ type: 'highlight', attrs: { color } }] })),
// textStyle with a color attr.
fc
.tuple(safeTextArb, fc.constantFrom('#123456', '#ff0000', '#00aa88'))
.map(([t, color]) => ({ type: 'text', text: t, marks: [{ type: 'textStyle', attrs: { color } }] })),
// Sole code mark (backtick span). safeTextArb is backtick-free, so the span
// content cannot contain an inner backtick.
safeTextArb.map((t) => ({ type: 'text', text: t, marks: [{ type: 'code' }] })),
// Link with safe text, a paren/space-free href, optionally a letter-bearing
// title (a purely numeric title is coerced to a number and dropped).
fc
.tuple(phraseArb, urlArb, fc.option(letterPhraseArb, { nil: undefined }))
.map(([t, href, title]) => ({
type: 'text',
text: t,
marks: [{ type: 'link', attrs: title ? { href, title } : { href } }],
})),
// Inline comment anchor: a span[data-comment-id] that must survive byte-for-
// byte. commentId is an alphanumeric token; `resolved` rides only when true.
fc
.tuple(safeTextArb, fc.stringMatching(/^[A-Za-z0-9]{4,10}$/), fc.boolean())
.map(([t, commentId, resolved]) => ({
type: 'text',
text: t,
marks: [
{ type: 'comment', attrs: resolved ? { commentId, resolved: true } : { commentId } },
],
})),
);
// ---------------------------------------------------------------------------
// Inline atoms and inline-content assembly.
// (Ported from markdown-roundtrip.property.test.ts.)
// ---------------------------------------------------------------------------
/** Inline math node carrying LaTeX that includes the `a < b` the task asks for. */
export const mathInlineArb: fc.Arbitrary<any> = fc
.constantFrom('a < b', 'x^2 + y^2', 'a < b < c', '\\frac{1}{2}', 'E = mc^2')
.map((text) => ({ type: 'mathInline', attrs: { text } }));
/** Mention node; label/id/entity are plain phrases / uuids. */
export const mentionArb: fc.Arbitrary<any> = fc
.tuple(phraseArb, fc.uuid(), fc.uuid())
.map(([label, id, entityId]) => ({
type: 'mention',
attrs: { id, label, entityType: 'user', entityId },
}));
export const hardBreakArb: fc.Arbitrary<any> = fc.constant({ type: 'hardBreak' });
const sameMarks = (a: any[] | undefined, b: any[] | undefined): boolean =>
JSON.stringify(a ?? []) === JSON.stringify(b ?? []);
/**
* Canonicalize a generated inline-content array the way ProseMirror stores it,
* then trim the markdown-fragile edges. (Ported verbatim from
* markdown-roundtrip.property.test.ts "normalizeInline":)
* 1) MERGE adjacent text runs with IDENTICAL marks (the editor coalesces
* them; split same-mark runs export to ambiguous "**a****b**").
* 2) Collapse CONSECUTIVE hard breaks (two render a blank line marked eats).
* 3) Drop a TRAILING hard break (removed by the converter's .trim()).
*/
export function normalizeInline(nodes: any[]): any[] {
const out: any[] = [];
for (const node of nodes) {
const prev = out[out.length - 1];
if (node.type === 'hardBreak' && prev && prev.type === 'hardBreak') continue;
if (
node.type === 'text' &&
prev &&
prev.type === 'text' &&
sameMarks(prev.marks, node.marks)
) {
prev.text += node.text;
continue;
}
out.push(node.type === 'text' ? { ...node } : node);
}
while (out.length > 1 && out[out.length - 1].type === 'hardBreak') out.pop();
return out;
}
/**
* Inline content for a paragraph: at least one marked text run, optionally with
* inline atoms (math/mention) and hard breaks interspersed. Always starts with a
* text run so the paragraph never opens with a block trigger. (Ported.)
*/
export const inlineContentArb: fc.Arbitrary<any[]> = fc
.tuple(
markedTextRunArb,
fc.array(
fc.oneof(
{ weight: 5, arbitrary: markedTextRunArb },
{ weight: 1, arbitrary: mathInlineArb },
{ weight: 1, arbitrary: mentionArb },
{ weight: 1, arbitrary: hardBreakArb },
),
{ minLength: 0, maxLength: 4 },
),
)
.map(([first, rest]) => normalizeInline([first, ...rest]));
/**
* Inline content for a HEADING identical to a paragraph's, but WITHOUT hard
* breaks. A hard break inside an ATX heading is not byte-stable (marked splits
* the heading). (Ported.)
*/
export const headingInlineContentArb: fc.Arbitrary<any[]> = fc
.tuple(
markedTextRunArb,
fc.array(
fc.oneof(
{ weight: 5, arbitrary: markedTextRunArb },
{ weight: 1, arbitrary: mathInlineArb },
{ weight: 1, arbitrary: mentionArb },
),
{ minLength: 0, maxLength: 4 },
),
)
.map(([first, rest]) => normalizeInline([first, ...rest]));
/** Simple plain-text inline content (single run) for containers rendered on the
* raw-HTML path (table cells / column bodies) where fancy inline is undesirable. */
export const plainInlineContentArb: fc.Arbitrary<any[]> = phraseArb.map((t) => [
{ type: 'text', text: t },
]);
@@ -0,0 +1,124 @@
import { describe, expect, it } from "vitest";
// Import DIRECTLY from src (NOT the docmost-client barrel, which pulls in
// collaboration.ts and mutates global DOM at import time).
import { convertProseMirrorToMarkdown } from "../src/lib/markdown-converter.js";
import { markdownToProseMirror } from "../src/lib/markdown-to-prosemirror.js";
/**
* gitmost #377 (round-1 review, finding #1) proof, against the REAL
* converter, that the transcript-insert boundary defense survives git-sync.
*
* The web bridge (apps/client .../gitmost/gitmost-recording.ts,
* `gitmostInsertTranscriptIntoEditor`) appends each transcript line as a
* PARAGRAPH text node. The paragraph serializer here (`case "paragraph"`) emits
* that text VERBATIM with no block-escape, so a line whose text begins with a
* col-0 markdown block trigger would, on the doc -> markdown -> doc git-sync
* cycle, silently re-parse into a heading / list / quote / callout / code block.
* That missing block-escape is the pre-existing root cause; the bridge's
* boundary defense prepends an invisible zero-width space (U+200B) to a line
* that begins with such a trigger, shifting it off column 0.
*
* This test keeps a COPY of the bridge's trigger regex (the bridge is in a
* different package and can't be imported here) and asserts:
* 1. bare trigger lines DO corrupt (documents the root cause), and
* 2. the ZWSP-neutralized form round-trips as a single PARAGRAPH with the
* text byte-preserved.
*/
const ZWSP = "​"; // U+200B
// MUST stay in sync with GITMOST_MD_BLOCK_TRIGGER_RE in the client bridge.
const MD_BLOCK_TRIGGER_RE =
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
const doc = (...nodes: any[]) => ({ type: "doc", content: nodes });
const para = (t: string) => ({
type: "paragraph",
content: [{ type: "text", text: t }],
});
const roundtrip = async (text: string) => {
const md = convertProseMirrorToMarkdown(doc(para(text)));
const back = await markdownToProseMirror(md);
return back.content as any[];
};
describe("gitmost transcript neutralization (git-sync round-trip)", () => {
// Lines that, at column 0, the serializer's missing block-escape would let
// git-sync re-parse into a non-paragraph block.
const triggerLines = [
"- dash",
"* star",
"+ plus",
"> quote",
"# hash",
"1. one",
"1) one",
"> [!info] note",
"```js",
"~~~",
// Solid + spaced thematic breaks — these re-parse into a `horizontalRule`,
// which carries NO text, so a bare separator line LOSES its text entirely
// (round-2 finding). `_` also only forms a block via this construct.
"---",
"***",
"___",
"- - -", // spaced dash break (solid form is caught by [-*+]\s too, but this is the break)
"_ _ _",
];
it("BARE trigger lines corrupt into non-paragraph blocks (root cause)", async () => {
for (const line of triggerLines) {
const blocks = await roundtrip(line);
// At least one produced block is NOT a paragraph — i.e. corruption.
const allParagraphs = blocks.every((b) => b.type === "paragraph");
expect(
allParagraphs,
`expected "${line}" to corrupt when inserted bare`,
).toBe(false);
}
});
it("BARE solid thematic breaks corrupt into a text-LOSING horizontalRule", async () => {
// The severe case: no text node survives. Documents why neutralization
// matters more here than for list/quote (where the text survived).
for (const line of ["---", "***", "___"]) {
const blocks = await roundtrip(line);
expect(blocks.map((b) => b.type)).toContain("horizontalRule");
// No block carries the original text anywhere.
const flat = JSON.stringify(blocks);
expect(flat).not.toContain(line);
}
});
it("ZWSP-neutralized trigger lines round-trip as a single paragraph, text preserved", async () => {
for (const line of triggerLines) {
// The regex must actually classify each as a trigger.
expect(MD_BLOCK_TRIGGER_RE.test(line), `regex missed "${line}"`).toBe(
true,
);
const neutralized = ZWSP + line;
const blocks = await roundtrip(neutralized);
expect(blocks).toHaveLength(1);
expect(blocks[0].type).toBe("paragraph");
// Text is byte-preserved (ZWSP + original line), so the display is the
// original line with only an invisible leading character.
expect(blocks[0].content[0].text).toBe(neutralized);
}
});
it("normal host-prefixed lines never match the trigger regex and round-trip byte-exact", async () => {
for (const line of [
"You: hello there",
"Speaker 1: - and then a dash mid-line",
"Speaker 2: 1. not a list",
]) {
expect(MD_BLOCK_TRIGGER_RE.test(line)).toBe(false);
const blocks = await roundtrip(line);
expect(blocks).toHaveLength(1);
expect(blocks[0].type).toBe("paragraph");
expect(blocks[0].content[0].text).toBe(line);
}
});
});
@@ -217,8 +217,9 @@ const colChildOf = (doc2: any) =>
doc2?.content?.[0]?.content?.[0]?.content?.[0];
describe('converter gap coverage — emission branches (specs 1–11)', () => {
// 1. orderedList renders index+1 and DROPS the start attribute.
it('orderedList start:5 restarts numbering at 1 (start attr ignored)', () => {
// 1. orderedList honors the start attribute (FIXED #351): markers count up
// from `start` ("5.","6.",…), which CommonMark round-trips.
it('orderedList start:5 numbers from 5 (start attr honored)', () => {
const out = convertProseMirrorToMarkdown(
doc({
type: 'orderedList',
@@ -229,7 +230,7 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
],
}),
);
expect(out).toBe('1. a\n2. b');
expect(out).toBe('5. a\n6. b');
});
// 2. An empty paragraph contributes an empty segment between two "\n\n" joins.
@@ -374,7 +375,9 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
],
}),
);
expect(out).toBe('- [ ] top\n - child');
// Block children of a task item are blank-line separated (loose list) per the
// #351 fix; the sublist stays at the fixed 2-column continuation indent.
expect(out).toBe('- [ ] top\n\n - child');
});
// 10. A bulletList inside a blockquote: each list line independently prefixed.
@@ -365,7 +365,7 @@ describe('media / attachment / container full-attribute golden coverage', () =>
);
});
it('orderedList inside a column renders via blockToHtml as <ol> (start attr DROPPED) with bold->strong, code->code', () => {
it('orderedList inside a column renders via blockToHtml as <ol start="N"> (start attr PRESERVED) with bold->strong, code->code', () => {
const out = c({
type: 'columns',
attrs: { layout: 'two' },
@@ -391,13 +391,13 @@ describe('media / attachment / container full-attribute golden coverage', () =>
},
],
});
// blockToHtml orderedList path emits a plain <ol> with no start attribute,
// and inlineToHtml maps bold->strong, code->code.
// blockToHtml orderedList path emits <ol start="3"> (FIXED #351), and
// inlineToHtml maps bold->strong, code->code.
expect(out).toContain(
'<ol><li><p><strong>a</strong></p></li><li><p><code>b</code></p></li></ol>',
'<ol start="3"><li><p><strong>a</strong></p></li><li><p><code>b</code></p></li></ol>',
);
// The start:3 attr is NOT preserved in the HTML/column container path.
expect(out).not.toContain('start=');
// The start:3 attr IS preserved in the HTML/column container path.
expect(out).toContain('start="3"');
});
it('hardBreak inside a column renders as <br> via inlineToHtml (not the markdown two-space form)', () => {

Some files were not shown because too many files have changed in this diff Show More