Compare commits

..

39 Commits

Author SHA1 Message Date
vvzvlad 20703d06c2 Merge pull request 'feat(gitmost-bridge): вставка transcript в страницу записи (#377)' (#378) from fix/377-bridge-transcript into develop
Reviewed-on: #378
2026-07-06 00:06:59 +03:00
agent_coder dab2660999 fix(gitmost-bridge): нейтрализовать сплошные thematic breaks в транскрипте
Round-1 нейтрализация ловила только spaced-разделители (`- - -`),
но пропускала сплошные `---`/`***`/`___`: на git-sync round-trip такая
строка становится horizontalRule, а он не несёт текста — строка терялась
целиком (хуже list/quote-порчи). Символ `_` вообще отсутствовал в regexе.

GITMOST_MD_BLOCK_TRIGGER_RE дополнен альтернативой на целую строку-
thematic-break `([-*_])(?:\s*\1){2,}\s*$` (3+ одинаковых `-`/`*`/`_`,
solid или через пробел); прежние группы переведены в non-capturing,
чтобы `\1` ссылался на единственную capture-группу. Обе копии regexа
(bridge и pm-markdown тест) синхронны.

Тесты: bare `---`/`***`/`___` документированы как text-losing
horizontalRule; ZWSP-нейтрализованная форма round-trip'ится параграфом
с сохранённым текстом. Кейсы падают на старом regexе.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 00:02:29 +03:00
agent_coder 751d55e9db fix(gitmost-bridge): neutralize col-0 block triggers in transcript lines + lock text-not-HTML (#378 review round 1)
Round-1 review found two issues:
- [regressions] Inserting a transcript line VERBATIM as a paragraph is unsafe on the
  git-sync doc->markdown->doc round-trip: the paragraph serializer emits text with no
  block-escape, so a line starting at col 0 with `- `/`> `/`# `/`1. `/```` ``` ````/`|`/
  `> [!info]` silently re-parses into a list/quote/heading/code/table/callout (the last
  hits the #359 callout machinery). Safe under the host contract (every line prefixed
  `You:`/`Speaker N:`, which starts with a letter) but the helper didn't enforce it, and
  leading-whitespace / stray lines exist. Fix: trim each kept line (drops the indent leak),
  and if it STILL begins with a col-0 markdown block trigger, prepend an invisible
  zero-width space (U+200B) so the trigger isn't at col 0 — the round-trip keeps it a
  paragraph. (Backslash-escape was rejected: `marked` consumes the `\` on re-import, so it
  would render visibly then vanish. ZWSP is invisible and never markdown-escaped.) The
  serializer's missing block-escape is the pre-existing root cause; this is the boundary
  defense. Prefixed transcript lines never match the trigger regex → left byte-exact.
- [test-coverage] The "inserted as TEXT, not HTML/markdown" contract wasn't locked (all
  test strings were plain alphabet). Added a test that inserts `<b>bold</b>
  <script>alert(1)</script> and *stars* and [link](x)` (with Bold/Italic/Link marks
  registered so an insertContent(html) regression would parse them) and asserts one
  verbatim text node, no marks, and getHTML() has no live tags.

Verified: apps/client tsc --noEmit 0 errors; gitmost bridge tests 4 passed; a NEW
prosemirror-markdown round-trip test (real convertProseMirrorToMarkdown ->
markdownToProseMirror) proves bare triggers corrupt while the ZWSP-neutralized form stays
a single byte-preserved paragraph and normal You:/Speaker N: lines round-trip byte-exact —
3 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 00:01:25 +03:00
agent_coder 02308012a6 feat(gitmost-bridge): insert transcript into the recording page (#377)
The native gitmost.app (stage 2) now sends a `transcript` field to
window.gitmost.createPageWithRecording, but the web bridge ignored it — the page
was created with audio only. Insert it.

- gitmost-recording.ts: add `transcript?: string` to GitmostCreatePagePayload
  (plain text, \n-separated `You:` / `Speaker N:` lines, ready to insert). Add a
  gitmostInsertTranscriptIntoEditor(editor, transcript) helper: a "Transcript"
  heading + one paragraph per non-empty line, each line inserted VERBATIM as a
  text node (never HTML → no injection), appended at doc end (below the audio).
  No-op when transcript is undefined/empty/whitespace-only/non-string.
- gitmost-global-bridge.tsx (createPageWithRecording): after the audio insert
  succeeds, call the helper inside a try/catch — best-effort, so a transcript
  failure can never turn the already-successful recording into an error (logs +
  still returns ok). Absent transcript → audio-only page, exactly as today.

DoD: recording with speech → page has audio AND a labeled transcript block;
without transcript → unchanged. Closes the web-side dependency of gitmost-app
stage 2. Verified: apps/client tsc --noEmit 0 errors; 2 unit tests
(transcript present → heading+paragraphs; undefined/""/whitespace/number/object
/null → no-op, doc unchanged).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 00:01:25 +03:00
vvzvlad 0665fcb630 Merge pull request 'fix(queue): убрать мёртвую очередь {search-queue} (#379)' (#380) from fix/379-remove-dead-search-queue into develop
Reviewed-on: #380
2026-07-05 23:44:54 +03:00
agent_coder 97bd554cb5 fix(queue): remove the dead {search-queue} — producers with no consumer (#379)
The {search-queue} BullMQ queue had producers on every page/space/workspace change
but NO consumer/@Processor — it lives in Docmost's EE edition (external search
drivers), never in this fork. Search works independently via the pages tsvector DB
trigger; the queue was pure dead weight, growing forever (1902 stuck jobs in Redis,
the first real hit of the #355 queue-growing alert — which fired correctly).

Remove the plumbing:
- the 3 producers: drop @InjectQueue(SEARCH_QUEUE) + every searchQueue.add(...) from
  page/space/workspace.listener.ts (and the isTypesense() gate that only wrapped the
  search enqueue). page.listener.handlePageUpdated did ONLY the search enqueue, so its
  @OnEvent(PAGE_UPDATED) handler is removed; the other handlers keep their aiQueue.add.
- the registration (queue.module.ts) and the metrics injection + 'search' depth-metric
  entry (metrics-bull.service.ts).
- the constants: QueueName.SEARCH_QUEUE + the 6 unused SEARCH_INDEX_* QueueJob members
  (grep-confirmed unreferenced). Shared QueueJob members (PAGE_*, SPACE_DELETED,
  WORKSPACE_DELETED — used by the AI queue / embedding/history/notification processors)
  are kept.

Search (tsvector trigger, search.service) and the PAGE_UPDATED websocket listener are
untouched. Verified: apps/server tsc --noEmit 0 errors; 6 spec suites / 52 tests green.

Ops (maintainer, out of code scope): one-time Redis cleanup
`redis-cli --scan --pattern "bull:{search-queue}:*" | xargs -r redis-cli del`, then
confirm bullmq_queue_depth{queue="search"} stays 0 (the metric no longer injects it).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 23:35:04 +03:00
vvzvlad f77a6b42de Merge pull request 'docs: how to test the application (browser E2E + out-of-band)' (#376) from docs/how-to-test into develop
Reviewed-on: #376
2026-07-05 22:40:12 +03:00
C9 Tester 134b627806 docs: add how-to-test.md (browser E2E + out-of-band) and link from AGENTS.md
Adds a testing guide covering how to verify features against a running stand:
drive the behaviour under test through the browser (not the API), verify
out-of-band in the DB/git, and the non-obvious traps. Notably the page has two
ProseMirror editors — [aria-label='Page title'] (non-collab) and
[aria-label='Page content'] (the collab body); querySelector('.ProseMirror')
returns the title, so tests must target the body editor and wait ~10s for the
hocuspocus store debounce. Links the new doc from AGENTS.md next to dev-stand.md
and adds a matching gotcha #8 to dev-stand.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 22:39:01 +03:00
vvzvlad 3267512ed9 Merge pull request 'refactor(#345): серверный экспорт/импорт markdown через @docmost/prosemirror-markdown' (#369) from refactor/345-server-converter into develop
Reviewed-on: #369
2026-07-05 20:41:30 +03:00
vvzvlad 48bd27b83c Merge pull request 'test(#351 PR 1): генеративное round-trip-тестирование конвертера — атрибутный уровень' (#373) from test/351-generative-converter into develop
Reviewed-on: #373
2026-07-05 20:40:40 +03:00
vvzvlad 265b81c93d Merge pull request 'fix(db): миграции «задним числом» из долгоживущих веток не роняют старт — CI-гейт + allowUnorderedMigrations (#363, инцидент #361)' (#365) from fix/363-migration-order into develop
Reviewed-on: #365
2026-07-05 20:40:06 +03:00
vvzvlad ed808876be Merge pull request 'fix(ai): patch ai@6.0.134 — drop O(n²) partialOutput accumulation causing heap OOM on long agent runs (#184)' (#368) from fix/ai-sdk-partial-output-oom into develop
Reviewed-on: #368
2026-07-05 20:39:51 +03:00
vvzvlad a72ddbbe86 Merge pull request 'refactor(ai-chat): единый реестр спеков инструментов — унификация tables/pages/misc/comments (#294)' (#367) from refactor/294-spec-registry-cont into develop
Reviewed-on: #367
2026-07-05 20:39:41 +03:00
agent_coder d8fc724d90 test(ai): cover the partialOutput PRESERVE branch of the ai@6.0.134 patch (#184, review F1)
The patch forks createOutputTransformStream: output==null skips partialOutput
(the OOM fix, already tested), output!=null preserves the original cumulative
accumulation. Only the skip branch was tested; the preserve branch — on which the
patch's "byte-identical when an output strategy is set" safety claim rests — had no
coverage, so a future re-port (patches are re-created via `pnpm patch` on every ai
bump) could silently route output-set calls into the skip branch and leave
partialOutput empty for object/text-output consumers, uncaught.

Add a 4th test: streamText({ ..., experimental_output: Output.text() }), drain
textStream, collect experimental_partialOutputStream, and assert it is non-empty and
cumulative (last partial == full text "Hello, world!"). Reuses the existing
makeModel() harness. Verified on the patched dist: partials are
["Hello","Hello, ","Hello, world!"]. `npx jest ai-sdk-partial-output.patch.spec.ts`
→ 4 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 19:35:14 +03:00
vvzvlad e4bfbcabaa Merge pull request 'feat(#371): редизайн модалки каталога ролей — карточки-наборы + per-role результаты импорта' (#375) from feat/371-roles-catalog into develop
Reviewed-on: #375
2026-07-05 16:43:19 +03:00
agent_coder 4c1ee50dc9 test(#351): close the mark-attr coverage hole + reclassify table spans (review round 1)
F1 [WARNING] The 'no invisible coverage hole' guard enumerated only
schema.nodes, so MARK attributes silently escaped the value-fuzz completeness
check — link.internal/target/rel/class are never fuzzed and nothing flagged it,
and a new attributed mark would slip through. Added allSchemaMarkAttrKeys() plus a
MARK_ATTR_FUZZED / MARK_ATTR_ALLOWLIST registry and two tests: every schema mark
attr must be in exactly one set (a new one turns it red), and neither set may hold
a stale row.

F2 [WARNING] The ACCEPTED annotation misclassified table colspan/rowspan as
having 'no md representation'. They DO round-trip — a spanned cell makes the
converter emit the whole table as a raw <table> with colspan/rowspan, which the
tiptap parser reads back. They are frozen only because generating a
geometrically-valid spanned table is deferred PR-2 structural work (the flat
generator hardcodes span = 1), not a markdown limit. Reclassified them as
DEFERRED-BUG (distinct from ACCEPTED) so a maintainer does not read them as an
inherent limitation; colwidth / backgroundColor(Name) stay ACCEPTED (the
raw-<table> fallback drops them).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 06:40:13 +03:00
agent_coder b8cce4f814 fix(#371): skipped role is not 'allInstalled', test the reason->action branch (review round 1)
F1 [WARNING] bundlePhase returned 'allInstalled' when a bundle's only
non-installed role was skipped (0 installed for it), so the collapsed green 'All
installed · up to date' header contradicted the open 'Installed 0 · 1 skipped'
plaque. It now returns 'mixed' whenever a skipped role is present. Fixed the test
that encoded the wrong behavior.

F2 [WARNING] The reason->action branch (name-conflict -> transient overlay +
'Rename & install'; already-installed -> informational, no button) lived only in
the component, untested. Extracted the two decisions into pure, unit-tested
helpers nameConflictSlugs() and partialOffersRename() and wired them into the
modal; both reason values are now covered.

F3 [low] Removed the unused useRef import (client eslint no-unused-vars is off, so
it shipped silently).

F4 [low] Extracted bundleCounts() as the single tally pass; bundlePhase and the
panel both derive from it instead of rescanning the roles array ~5x per render
(the same model<->component consolidation this PR is about).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 06:09:58 +03:00
agent_coder c5bff2d84a fix(#345): normalize CRLF before front-matter strip (review round 3)
F9 [WARNING] The line-anchored front-matter regex from round 2 requires a bare
LF after the opening `---`, so a Windows/CRLF foreign file (`---\r\n...`) slips
past the strip and leaks its front-matter into the body (where `title: Foo`
renders as a setext heading that title extraction hijacks). The canonical parser
whose regex shape this copied (page-file.ts) normalizes CRLF -> LF BEFORE its
FRONTMATTER_RE; the import path copied the regex but missed the normalization.
normalizeForeignMarkdown now replaces CRLF with LF first (which also makes
convertReferenceFootnotes' split('\n') consistent). Adds a CRLF fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 05:38:07 +03:00
agent_coder a325ddbabd feat(#371): roles catalog modal redesign — bundle cards + per-role import results
Integrates the designer-handoff Roles Catalog modal, wired to the real API; the
parent ai-agent-roles.tsx and the { opened, onClose, roles } contract are
unchanged.

- Server importFromCatalog now returns per-role lists (createdRoles /
  skippedRoles with a reason) alongside the existing counters (compat-preserving),
  so the UI can name the conflicting/installed roles.
- New pure view-model (catalog-bundle-model.ts): bundlePhase (empty | allNew |
  allInstalled | updates | mixed, ignoring the transient 'skipped'),
  installedLangForRole (same-slug-different-language hint), mapCatalogRoleToView —
  all unit-tested without mounting.
- Bundle cards with a summary status in the collapsed header (eager useQueries
  fan-out over all bundles, sharing the existing per-bundle cache keys), a single
  primary action per bundle, checkboxes + select/deselect-all, an inline result
  plaque that keeps the modal open, per-bundle and global 'Update all' request
  series with progress, and the other-language hint.
- The partial-result plaque distinguishes the skip reason: only a name-conflict
  offers 'Rename & install'; an already-installed race is informational (a rename
  re-import would just skip again and self-heal into a false success).
- All strings i18n'd (en/ru); mock handoff code (SEED/mockImport/delay) removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 05:35:58 +03:00
agent_coder 80fc30633b fix(#345): replace id-alternation regex with a fixed generic scanner + line-anchor frontmatter (review round 2)
F7 [CRITICAL] The round-1 F2(a) fix built ONE alternation regex over all
definition ids (`(id1|id2|...)`). On prefix-chain ids (a, aa, aaa, ...) V8's
regex compiler blows its stack with a fatal, UNCATCHABLE 'RegExpCompiler
Allocation failed' that kills the whole process — strictly worse than the
original per-def thread-hang, and its match cost was still O(text x defs).
Replaced with a single FIXED generic scanner `/\[\^([^\]]+)\]/g` plus a map
lookup in the replacer: genuinely O(total text), no per-document regex
compilation, cannot blow up. Output is identical (only real def ids are inlined).

F8 [WARNING] The frontmatter strip regex was not line-anchored: it closed on the
FIRST `---` anywhere, so a value containing a triple-dash (e.g.
'title: Q1 --- Q2') truncated the frontmatter and leaked the rest into the body.
Replaced with the line-anchored shape the canonical parser already uses
(page-file.ts): open on `---\n`, close on a `\n---` line.

Adds tests: 4000 prefix-chain ids do not crash and stay fast; a frontmatter
value containing '---' is stripped whole.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 05:18:44 +03:00
agent_coder e17d5bc060 fix(#345): restore prom-client, harden normalizer against ReDoS, strip frontmatter (review round 1)
Addresses the round-1 review of #369:

F1 [CRITICAL] Restore prom-client. The prior commit removed it as a 'stray dep',
but metrics.registry.ts imports it unconditionally at startup (main.ts boot), so
a clean frozen install had no prom-client -> server tsc TS2307 + boot crash. It
was surviving only via hoisting from a warm store. Restored to apps/server
dependencies + regenerated the lock (prom-client/tdigest/bintrees return),
keeping the @docmost/prosemirror-markdown dep. Verified: clean frozen install ->
require.resolve('prom-client') ok, server tsc EXIT 0.

F2 [HIGH] Two quadratic ReDoS vectors in foreign-markdown.ts on untrusted import
(runs synchronously on the request thread, 30MB cap):
  (a) pass-2 was O(lines x defs) — a per-def RegExp rebuilt and run over every
      line. Replaced with ONE precompiled alternation regex over all def ids,
      built once per document, with an id->body lookup in the replacer: O(text).
  (b) the inline-code split alternation backtracks quadratically on a long
      UNCLOSED backtick run. Lines over 8KB now skip the split (left untouched) —
      a real footnote line is never that long.

F3 [WARNING] Restore the leading YAML front-matter strip that the retired
markdownToHtml layer did. Without it, Obsidian/Hugo/Jekyll/git-sync files leak
their front-matter into the body (and 'title:' renders as a setext heading that
title extraction can hijack).

F4 [WARNING] Extend the zip-import spec with an image (width+align) + callout
fidelity assertion through the PM->HTML->PM hop (the one hop the package suite
does not cover).

F5/F6 Update AGENTS.md (apps/server is now a prosemirror-markdown consumer) and
make the server pretest build prosemirror-markdown too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 04:54:07 +03:00
agent_coder bfcee6dddc test(prosemirror-markdown): generative round-trip testing — attribute level, flat docs (#351 PR 1)
Schema-derived, property-based (fast-check) round-trip tests over flat
single-node ProseMirror documents. One test PR — src/ is untouched; the two
real bugs found are pinned as loud it.fails counterexamples, not fixed here.

- attr-arbitraries.ts: per-attribute four-state arbitraries (absent/default/
  nonDefault/degenerate), attribute list sourced from schema.nodes[t].spec.attrs;
  a documented override table supplies legal domains for constrained attrs and
  distinguishes two frozen classes explicitly — ACCEPTED limitations (no md
  representation) vs PINNED bugs (representable but dropped, tracked as
  counterexamples).
- text-arbitraries.ts: hostile text corpus (ported from the existing property
  test's supported-space guarantees).
- node-generators.ts: flat single-node generators + a completeness contract —
  every one of the schema's 45 nodes / 12 marks is either generated or listed in
  KNOWN_UNCOVERED with a reason.
- flat-roundtrip.property.test.ts: P1 (semantic round-trip via
  docsCanonicallyEqual), P2 (second-pass byte fixpoint — anti GS-EDIT-REVERT),
  P3 (totality), generator validity via schema.check(), and an explicit
  attribute-value-coverage snapshot so the not-fuzzed set can never grow silently.
- counterexamples: column.width (% dropped on parseFloat -> P2 churn) and
  orderedList.start (non-1 start renders as '1.' -> P1 loss) pinned as it.fails.

SEED=20250705, NUM_RUNS=300 per property; ~17s, no OOM (union arbitraries).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 04:38:40 +03:00
agent_coder 2c2d60a5dc fix(#345): protect inline-code refs and escape footnote-body brackets
The foreign-markdown import normalizer rewrote GFM reference footnotes
(`[^id]` + `[^id]: def`) into canonical inline `^[def]` footnotes, but two
edge cases corrupted content:

1. A `[^id]` inside an inline-code span (backticks) was rewritten like prose
   text — only fenced code blocks were protected. Now the rewrite pass splits
   each line on inline-code spans and only touches the text outside them.

2. An unbalanced `]` in a definition body truncated the resulting `^[...]`
   footnote at the canonical tokenizer, leaking the tail as literal text. The
   body's square brackets are now backslash-escaped before wrapping.

Adds golden cases for both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 03:39:01 +03:00
agent_coder 1417209915 fix(#345): drop stray prom-client dep + add prosemirror-markdown to the lock
The step-1 package.json declared the new @docmost/prosemirror-markdown workspace
dep but the lock was not regenerated (CI frozen install would fail), and it also
added a stray prom-client dep (a coder env-workaround for a pre-existing hoisted
import, unrelated to #345 — removed). Regenerated the lock with only the
prosemirror-markdown dep; faithful frozen install now passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 03:27:01 +03:00
agent_coder f555fc87da refactor(#345 step 2): server markdown IMPORT via canonical parser + normalizer
Move every SERVER Markdown->ProseMirror path off the editor-ext markdown layer
(`markdownToHtml`, a second marked-based parser) onto the canonical
`@docmost/prosemirror-markdown` package, and add a foreign-markdown normalizer at
the import boundary.

Code:
- `ImportService.processMarkdown` (single `.md` upload) now parses
  `markdownToProseMirror(normalizeForeignMarkdown(md))` directly — no HTML hop.
- `PageService.parseProsemirrorContent` markdown case (page create/update with
  `format: 'markdown'`) same.
- `FileImportTaskService` (zip import) parses markdown with the package, then
  serializes to HTML (`jsonToHtml`) so the SHARED HTML attachment / internal-link
  pipeline (processAttachments + formatImportHtml + processHTML) keeps handling
  `.md` and `.html` imports uniformly. The markdown PARSE — the drift source — no
  longer goes through editor-ext; the PM->HTML->PM hop that follows is lossless
  plumbing for attachment resolution, not a second parse.
- `canonicalizeFootnotes` stays as an idempotent #228 safety net for the HTML
  path (a no-op on the already-canonical markdown output).

Normalizer (`integrations/import/utils/foreign-markdown.ts`): a TEXT pre-pass,
NOT a parser fork. The strict canonical parser does not accept GFM `[^id]`
reference footnotes (and would misread `[^id]: def` as a CommonMark link-ref
definition, silently corrupting the ref into a bogus link), so the normalizer
rewrites reference footnotes into canonical inline `^[def]` before parsing.
Callout surfaces (`:::type` and `> [!type]`) are intentionally NOT touched — the
canonical parser already accepts BOTH natively, so normalizing them would be
redundant and risk degrading its nesting/code-fence-aware handling.

Fixtures-first: foreign-markdown.spec pins the normalizer and the end-to-end
acceptance (no literal `[^id]`/`:::` leaks; re-export is canonical). The two
footnote-canonicalize specs are updated to the canonical output — the parser
assigns fresh `fn-*` ids, so they now assert by definition BODY order (still
reference-ordered, deduped, orphan-free).

FINAL CHECK: `grep -rn "htmlToMarkdown\|markdownToHtml" apps/server/src` (non
-test) is now empty — both editor-ext markdown-layer functions are gone from the
server.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 03:21:07 +03:00
agent_coder d6d1195abd refactor(#345 step 1): server markdown EXPORT via canonical converter
Move every SERVER ProseMirror->Markdown path off the editor-ext markdown layer
(`htmlToMarkdown`, a second turndown-based converter) onto the canonical
`@docmost/prosemirror-markdown` package.

- `ExportService.exportPage` (page/space markdown export) and
  `collaboration.util.jsonToMarkdown` (used by page.controller's markdown
  responses and the AI public-share chat tool) now serialize DIRECTLY from
  ProseMirror JSON via `convertProseMirrorToMarkdown` — no HTML intermediate, no
  `<colgroup>` scrub (the converter emits GFM tables directly).

This is the SAME serializer the git-sync vault writer feeds, so an exported page
BODY is byte-identical to its vault representation: no more export-md vs vault-md
drift. The HTML export path is unchanged (still `jsonToHtml`).

Emitted markdown moves to the canonical forms: callouts `> [!type]` (not
`:::type`), inline footnotes `^[…]` (not `[^id]`), lossless images
`![alt](src) <!--img {…}-->` (editor-ext dropped width/height/align).

Fixtures-first: export-markdown.spec asserts those canonical forms and the
export==vault-by-construction equality (both call the package converter). The
one deliberate export/vault delta — export prepends the page title as an H1
while the vault carries it in frontmatter — is pinned by a test.

Test infra: declare the `@docmost/prosemirror-markdown` workspace dep; teach
jest to load its ESM build (babel-jest) and stub `@tiptap/react` (server code
imports editor-ext, whose node views reference React renderers only used in a
live browser editor — never on the server).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 03:20:25 +03:00
agent_coder 36b940fdb8 fix(#294 review F1-F2): test the changed execute wirings + transport-neutral descriptions
- F1: added in-app execute tests for the two wirings that ACTUALLY changed in the
  migration (the contract-parity test only checks advertised schema keys, not
  execute bodies): movePage forwards the newly-added optional `position` to
  client.movePage (and passes undefined position + null parent when omitted); the
  table trio (insert/delete/updateCell) forwards the unified `table` param
  positionally. A field destructured under the wrong name would have silently
  passed undefined to the client (execute is any-cast, tsc won't catch it).
- F2: rewrote the three migrated descriptions that hardcoded snake_case sibling
  tool names (which the in-app camelCase layer exposes under different ids,
  violating the registry's own transport-neutral-prose convention) into neutral
  prose: getPage "use get_page_json" -> "use the lossless page-JSON read tool";
  updatePageJson "get_page_json -> ... -> update_page_json" -> "read the page-JSON
  view -> modify -> write it back", "prefer rename_page" -> "prefer the rename-page
  tool"; exportPageMarkdown "import_page_markdown round-trip" -> "page-Markdown
  import round-trip" (the last was a direct regress — the in-app base said the
  camelCase importPageMarkdown). (stashPage's pre-existing get_page_json mention is
  out of scope, per the reviewer.)

Gate: mcp build 0; ai-chat-tools.service + tool-tiers (catalog-partition) pass,
incl. the 5 new execute-wiring tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 03:10:42 +03:00
agent_coder 0050ad7ebb docs(#363 review F2): update AGENTS.md migration-ordering to the new tolerant behavior
The "Migration ordering" section still described the OLD crash-loop-at-boot
behavior this PR removes ("Kysely refuses to start … rejected at boot"). Rewrote
it to the new two-layer model: the CI migration-order gate is the primary defense
(rename to a current timestamp), and the runtime now sets allowUnorderedMigrations
so the app applies a back-dated migration instead of crash-looping (with the note
that #ensureNoMissingMigrations still guards a removed applied migration, and that
migrations must stay independent since apply order can differ across instances).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:57:11 +03:00
agent_coder ce70fab1df refactor(ai-chat): unify share_page into SHARED_TOOL_SPECS (#294, misc family)
Migrates share_page / sharePage into the transport-agnostic spec registry
(schema + description declared once; each transport keeps only its execute/auth):
- sharePage (deferred) -> SHARED_TOOL_SPECS; index.ts uses registerShared(),
  ai-chat uses sharedTool(); removed from INLINE_TOOL_TIERS.

Drift reconciled (documented inline): both inline copies already carried the
"only share when the user explicitly asked" security framing, so the old
"per-transport divergence" note in BOTH layers was STALE — there was no real
behavioral divergence, only wording drift. The canonical description merges the
MCP copy's URL-format + idempotency detail with the in-app copy's reversibility
note and keeps the shared security framing. pageId keeps the MCP copy's stricter
.min(1). The MCP execute keeps its own `searchIndexing ?? true` default
(per-layer, not part of the shared schema).

Intentionally NOT migrated (kept inline — genuinely divergent, as their existing
notes state):
- search / searchPages: the in-app tool is a semantic+keyword hybrid (RRF) with
  in-process access control and a tuned schema (limit 1-20); the MCP `search` is
  a plain REST full-text search (limit up to 100). Different behavior AND schema.
- docmost_transform / transformPage: the in-app tool deliberately omits the
  `deleteComments` schema field (a comment-deletion guardrail) and carries a
  shorter description. Different schema.

Gate: mcp build 0 + node --test 458/458 (page-search excluded — hangs only under
the local re2->RegExp type-shim, its source untouched), server jest 775 incl.
tool-tiers catalog-partition + shared-spec contract parity, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:27:50 +03:00
agent_coder 7b4617db70 fix(#363 review F1): make the migration-order gate fail CLOSED (not open)
The CI gate — whose whole job is to BLOCK a back-dated migration — could pass
open in exactly the scenario it guards (a long branch vs a moving base, i.e. #361):

- Dropped the redundant `git fetch --depth=1`: the checkout already did
  fetch-depth:0 (full history), and the shallow graft truncated the BASE history,
  so `merge-base` (thus the three-dot `origin/base...HEAD` diff) failed when the
  base had moved ahead of the PR merge commit.
- Removed `|| true` on the diff: it swallowed that failure → `added` empty → loop
  skipped → bad=0 → gate PASS. Now `set -e` aborts the job (fail CLOSED) on any
  diff error — a gate must never pass on error.

Verified: yaml parses (jobs migration-order, test); a broken-ref diff with set -e
and no `|| true` aborts before bad=0 (fail-closed) instead of passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:27:16 +03:00
agent_coder b51dae16a6 docs(mcp): mark media tools MCP-only in index.ts (#294, media family)
The media tools — insert_image, replace_image, insert_footnote — are MCP-only
by design: the in-app AI-chat agent exposes no image or footnote tools, so there
is no second layer to unify into SHARED_TOOL_SPECS. A registry spec's
tier/catalogLine are in-app metadata and the catalog-partition test forbids a
spec without a live in-app tool, so forcing them into the registry would break
the invariant. They stay per-transport (inline in index.ts).

No behavior change — documentation only (adds the rationale above each tool so a
future migrator does not re-investigate why these are not shared).

Gate: mcp tsc 0 (comment-only change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:14:39 +03:00
agent_coder 39735afd73 refactor(ai-chat): unify page tools into SHARED_TOOL_SPECS (#294, pages family)
Migrates the three-layer page tools into the transport-agnostic spec registry
(schema + description declared once; each transport keeps only its execute/auth):
- getPage, listPages (core), createPage, movePage, renamePage, deletePage,
  updatePageJson, exportPageMarkdown (deferred) -> SHARED_TOOL_SPECS; index.ts
  uses registerShared(), ai-chat uses sharedTool(); removed from
  INLINE_TOOL_TIERS. Tiers preserved from CORE_TOOL_KEYS (getPage/listPages =
  core, the rest deferred).

delete_page is genuinely three-layer (in-app deletePage exists), so it IS
migrated — not MCP-only. Its H4 guardrail is preserved: the shared schema
exposes ONLY pageId, so no permanentlyDelete/forceDelete flag can reach the
client (still asserted by ai-chat-tools.service.spec.ts).

Descriptions merged (documented inline): each canonical text takes the MCP
copy's richer structural notes plus the in-app copy's reversibility framing.

Schema DRIFT reconciled (documented inline):
- createPage.content: MCP pinned .min(1) but the in-app copy left it unbounded
  and DOCUMENTS an empty body as valid ("may be empty" — creating an empty page
  to fill later is a real use). Kept the looser no-min form: create_page now also
  accepts an empty body (harmless) and no previously-valid in-app input is
  rejected. title/spaceId keep the MCP .min(1) (empty is never valid).
- movePage: MCP exposed an optional `position` (fractional-index) field the
  in-app copy lacked. Unified by KEEPING position — the in-app client already
  accepts an optional position arg, so the in-app execute now forwards it;
  optional, so no previously-valid call is rejected. `parentPageId` is nullable
  on both (real JSON null -> root); the MCP execute keeps its 'null'/'' string
  coercion as a per-layer robustness fallback.
- getPage/renamePage/updatePageJson/exportPageMarkdown/listPages: kept the MCP
  copy's stricter .min(1) on ids where the in-app copy was unbounded.

Per-transport execute logic preserved: getPage's {title,markdown} projection,
updatePageJson's JSON-string normalization, list_pages' default limit/tree, and
move_page's cycle guard + positive-confirmation check all stay in their execute
bodies.

Intentionally NOT touched: updatePageContent (Markdown-based body update; no MCP
equivalent) and getTable (name-convention divergence, see tables family) stay
inline.

Gate: mcp build 0 + node --test 458/458 (page-search excluded — hangs only under
the local re2->RegExp type-shim, its source untouched), server jest 770 incl.
tool-tiers catalog-partition + shared-spec contract parity + deletePage H4
guardrail, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:13:41 +03:00
agent_vscode 9b4b38a611 fix(ai): patch ai@6.0.134 — drop O(n²) partialOutput accumulation causing heap OOM on long agent runs (#184)
Production OOM'd (JS heap 1.85 GB / 2 GB limit) during a ~20-step,
~28k-chunk autonomous agent turn. Heap snapshot analysis (memlab) showed a
single DefaultStreamTextResult retaining ~1.7 GB via the never-consumed
leftover tee() branch of its internal baseStream.

Root cause in ai@6.0.134: streamText substitutes the default text() output
strategy even when the caller passes NO `output` option. Its
createOutputTransformStream then accumulates the ENTIRE turn text and, on
EVERY text-delta, enqueues `{ part, partialOutput }` where partialOutput is
a flat snapshot of all text so far (JSON.stringify flattens the
cons-string) — O(n²) memory across the turn. Every consumer accessor tees
baseStream and keeps the second branch as the new baseStream; the final
leftover branch is never read, so its controller queue holds every chunk
(28,225 x ~164 KB in the OOM'd run) for the life of the turn.

Fix (pnpm patch on both dist/index.js and dist/index.mjs):
- pass the raw, possibly-undefined `output` option into
  createOutputTransformStream instead of defaulting to text()
- when output == null, publish each text-delta immediately without
  accumulating turn text or producing partialOutput snapshots; streaming
  granularity is unchanged, and callers that DO request an output strategy
  keep the original behavior

Our server never uses partialOutputStream / experimental_output / the
output option, so no behavior changes for us beyond memory.

Regression spec ai-sdk-partial-output.patch.spec.ts drives the real
patched SDK with MockLanguageModelV3: asserts per-delta textStream
granularity, an EMPTY experimental_partialOutputStream (tripwire — yields
one cumulative partial per delta when unpatched), and the PATCH(docmost
marker in both installed dist bundles. Also documents the patch in
AGENTS.md (must be re-created when bumping `ai`) and CHANGELOG.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 02:13:17 +03:00
agent_coder eebbe6717c refactor(ai-chat): unify table row/cell tools into SHARED_TOOL_SPECS (#294, tables family)
Migrates the three-layer table WRITE tools into the transport-agnostic spec
registry (schema + description declared once; each transport keeps only its
execute/auth):
- tableInsertRow, tableDeleteRow, tableUpdateCell -> SHARED_TOOL_SPECS;
  index.ts uses registerShared(), ai-chat uses sharedTool(); removed from
  INLINE_TOOL_TIERS (all three are deferred; not in CORE_TOOL_KEYS).

Drift reconciled (documented inline): the four table tools previously carried a
"NOT shared" note in both layers over a single parameter-NAME drift — the MCP
layer named the table reference `table`, the in-app layer `tableRef`. Unified on
the MCP name `table` (renaming the public MCP parameter would break external MCP
clients; the in-app parameter is model-facing/prompt-only and safe to rename).
The in-app execute bodies now destructure `table`. Descriptions took the MCP
copy's richer wording (documents `#<index>`, padding, header-row behavior) plus
the in-app copy's "Reversible via page history" note; both fields keep the MCP
copy's stricter .min(1) (in-app left them unbounded); sibling tool references
phrased transport-neutrally.

Intentionally NOT migrated (kept inline): table_get / getTable. Its MCP tool
name is noun-first (`table_get`) while the in-app key is verb-first (`getTable`),
which breaks the snake_case(inAppKey) naming convention the registry enforces
(shared-tool-specs.contract.spec.ts). Renaming the public MCP tool would break
external clients, so it stays per-transport — but its in-app reference param was
still aligned to `table` (was `tableRef`) for consistency with the migrated trio.

Gate: mcp tsc 0 + node --test 458/458 (page-search excluded — hangs only under
the local re2->RegExp type-shim, its source is untouched), server jest 730 incl.
tool-tiers catalog-partition + shared-spec contract parity, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 02:06:52 +03:00
agent_coder e348433a39 refactor(ai-chat): unify comment tools into SHARED_TOOL_SPECS (#294, comments family)
Migrates the three-layer comment tools into the single transport-agnostic spec
registry (schema + model-facing description declared once; each transport keeps
only its execute/auth):
- createComment, listComments, resolveComment, checkNewComments — moved to
  SHARED_TOOL_SPECS; index.ts uses registerShared(), ai-chat uses sharedTool();
  removed from INLINE_TOOL_TIERS (tier/catalogLine now on the spec). Tiers
  preserved from CORE_TOOL_KEYS (create/list/resolve = core, check = deferred).

Intentionally NOT migrated (kept MCP-inline): update_comment / delete_comment —
they are MCP-only by design; the in-app AI-chat layer deliberately has no
updateComment/deleteComment (comment edits are irreversible / not
version-tracked), asserted by ai-chat-tools.service.spec.ts. A registry spec's
tier/catalogLine are in-app metadata and the catalog-partition test forbids a
deferred spec without a live in-app tool, so these stay per-transport.

Drift reconciled (documented inline): createComment/listComments/checkNewComments
took the more-maintained/superset description + stricter .min(1) guards.
resolveComment: `resolved` drifted (MCP optional+default(true) vs in-app
required) — kept the MCP superset, so in-app resolveComment now accepts an
omitted `resolved` (defaults to resolve) — a deliberate, backward-compatible
unification (never rejects a previously-valid input).

Gate: mcp build 0 + node --test 480/480, ai-chat 654, tool-tiers (incl. F3
catalog-partition) 16/16, server tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 01:45:43 +03:00
agent_coder 459d636ffb fix(db): prevent the migration-order crash-loop from long-lived branches (#363, incident #361)
A long-lived branch can add a migration whose timestamped filename sorts BEFORE
migrations already applied in prod (#234's 20260627T130000-ai-chat-runs merged
after 20260704T120000-client-metrics was live). Kysely's migrator with the
default ordered setting then rejects the applied set as "corrupted migrations"
(no longer a prefix of the sorted list), throws, and the app crash-loops on boot
— exactly incident #361 (502s for ~11 min after a develop deploy). #119 and #120
(June branches) are the next such threats.

Two levels, both:
1. CI migration-order gate (a new `migration-order` job in test.yml, PR-only):
   fails the PR when an added migration sorts at/before the newest migration on
   the base branch, with an actionable message to rename it to a current
   timestamp before merge. This is the primary defense — makes back-dating
   impossible to merge accidentally.
2. `allowUnorderedMigrations: true` on BOTH Migrators (migration.service.ts
   startup auto-migrate + migrate.ts CLI): the runtime safety net — Kysely applies
   a not-yet-applied older migration instead of bricking startup, so a back-dated
   migration that bypasses the gate (manual push / hotfix branch) still boots.
   Trade-off documented inline: apply order across instances may differ from
   lexicographic, so migrations must stay independent (ours each create their own
   objects); the CI gate remains the primary line.

Verified: allowUnorderedMigrations is a valid Kysely 0.28.17 Migrator option;
server tsc clean; the gate script rejects a back-dated filename and passes a
current one. No new deps, no migration, no runtime behavior change beyond the
migrator resilience.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 01:36:57 +03:00
agent_vscode e89ac627dd fix(migrations): rename ai-chat-runs migration to post-merge timestamp
20260627T130000-ai-chat-runs sorted before the already-executed
20260702T120000-ai-chat-page-snapshot, so Kysely's strict ordering
check ("corrupted migrations") crash-looped the server on startup.

- rename 20260627T130000-ai-chat-runs.ts -> 20260704T130000-ai-chat-runs.ts
- update the mirror comment in database/types/db.d.ts
2026-07-05 00:59:05 +03:00
vvzvlad f665f6fdd2 Merge pull request 'feat(ai-chat): autonomous agent runs — phase 1: durable detached runs (#184)' (#234) from feat/184-autonomous-agent-runs into develop
Reviewed-on: #234
2026-07-05 00:40:26 +03:00
agent_coder 68899a2c2e feat(ai-chat): durable detached agent runs — phase 1 (#184/#234)
Squashed for a clean rebase onto develop (was 19 commits; the reviewer approved
the net diff at fb246080). Detaches an agent run from the HTTP request/browser
window: a run is a first-class lifecycle object (ai_chat_runs), a browser
disconnect no longer kills it, a concurrent-run insert-gate prevents double runs,
and a reopened chat live-follows a still-running run via a polled observer merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 23:35:26 +03:00
107 changed files with 9978 additions and 2493 deletions
+14
View File
@@ -209,6 +209,20 @@ MCP_DOCMOST_PASSWORD=
# active" behavior. # active" behavior.
# AI_CHAT_DEFERRED_TOOLS=true # AI_CHAT_DEFERRED_TOOLS=true
# --- Autonomous / detached agent runs (settings.ai.autonomousRuns) ---
# Opt-in per workspace (AI settings; off by default). When on, a chat turn becomes
# a server-side RUN that survives a browser disconnect — only an explicit Stop ends
# it, and a client reconnects/live-follows the run.
#
# DEPLOY CONSTRAINT — SINGLE-INSTANCE ONLY in phase 1: Stop and the in-process
# AbortController that backs it are process-local, so a Stop only aborts a run
# executing on the SAME replica that owns it (cross-instance pub/sub stop is phase
# 2 and not yet reliable). Do NOT enable autonomousRuns on a horizontally-scaled
# deployment (multiple replicas behind a load balancer, or Docmost cloud
# CLOUD=true) — run a single instance instead. The server logs a startup WARNING
# when it detects a multi-instance deployment (CLOUD=true) so the constraint is
# visible, and a startup sweep settles any run left dangling by a restart.
# --- Anonymous public-share AI assistant --- # --- Anonymous public-share AI assistant ---
# Opt-in per workspace (AI settings -> "public share assistant"; off by default). # Opt-in per workspace (AI settings -> "public share assistant"; off by default).
# When enabled, anonymous visitors of a published share can ask an AI about that # When enabled, anonymous visitors of a published share can ask an AI about that
+43
View File
@@ -13,6 +13,49 @@ permissions:
contents: read contents: read
jobs: jobs:
# Guard against a long-lived branch adding a migration whose timestamped
# filename sorts BEFORE migrations already applied on the target branch (and
# thus in prod). The Kysely startup migrator rejects that as "corrupted
# migrations" and crash-loops the app on boot (incident #361). This gate fails
# the PR so the migration is renamed to a current timestamp before merge. Only
# runs for pull_request events (needs a base branch to diff against).
migration-order:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Checkout (full history for the base-branch diff)
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Added migrations must sort after the newest on the base branch
env:
TARGET_BRANCH: ${{ github.base_ref }}
run: |
set -euo pipefail
MIG_DIR="apps/server/src/database/migrations"
# checkout above already did fetch-depth:0 (full history). Fetch the base
# WITHOUT --depth (a shallow graft would truncate the base history and
# break the merge-base when the base has moved ahead of the PR merge —
# exactly the long-branch-vs-moving-base case this gate guards, #361).
git fetch --no-tags origin "$TARGET_BRANCH"
newest_on_target=$(git ls-tree -r --name-only "origin/${TARGET_BRANCH}" "$MIG_DIR" | sort | tail -1)
# NO `|| true`: a diff failure (e.g. an unresolved merge-base) must fail
# the job CLOSED — a gate whose job is to BLOCK must never pass on error.
# `set -e` above already aborts on a non-zero diff exit.
added=$(git diff --diff-filter=A --name-only "origin/${TARGET_BRANCH}...HEAD" -- "$MIG_DIR")
bad=0
for f in $added; do
if [[ "$f" < "$newest_on_target" || "$f" == "$newest_on_target" ]]; then
echo "::error::Migration $f sorts at or before the newest on ${TARGET_BRANCH} ($newest_on_target) — rename it with a CURRENT timestamp before merge (do not change its contents). See incident #361."
bad=1
fi
done
if [ "$bad" -eq 0 ]; then
echo "Migration order OK (added migrations all sort after $newest_on_target)."
fi
exit $bad
test: test:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 20 timeout-minutes: 20
+14 -4
View File
@@ -201,7 +201,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend | | `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server | | `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy | | `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp` and `git-sync`; there is exactly ONE copy of the converter now | | `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`. `build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
@@ -214,6 +214,12 @@ Run from the repo root unless noted. The dev workflow needs **Postgres (with the
> server, `APP_SECRET` mismatch between processes, a stale `editor-ext` white- > server, `APP_SECRET` mismatch between processes, a stale `editor-ext` white-
> screening the client, LAN exposure. See **[docs/dev-stand.md](docs/dev-stand.md)** > screening the client, LAN exposure. See **[docs/dev-stand.md](docs/dev-stand.md)**
> for the step-by-step and the traps. > for the step-by-step and the traps.
>
> **Testing the app against a stand** (browser E2E + out-of-band verification) has
> its own non-obvious traps — the page has two ProseMirror editors (only the body is
> collab-bound), a ~10s store debounce, and API-seeding the thing under test is a
> silent no-test. See **[docs/how-to-test.md](docs/how-to-test.md)** before writing
> UI tests.
```bash ```bash
pnpm install # install all workspaces (uses pnpm patches; see package.json `pnpm.patchedDependencies`) pnpm install # install all workspaces (uses pnpm patches; see package.json `pnpm.patchedDependencies`)
@@ -250,7 +256,10 @@ pnpm --filter server migration:codegen # regenerate src/databa
``` ```
Migration files live in `apps/server/src/database/migrations/` and are named `YYYYMMDDThhmmss-description.ts`. Fork-specific migrations only **add** tables (`page_embeddings`, `ai_chats`, `ai_chat_messages`, `ai_provider_credentials`, `ai_mcp_servers`, `page_template_references`) and columns (e.g. `pages.is_template`, a `NOT NULL DEFAULT false` boolean) — never drop/rewrite Docmost data. Migration files live in `apps/server/src/database/migrations/` and are named `YYYYMMDDThhmmss-description.ts`. Fork-specific migrations only **add** tables (`page_embeddings`, `ai_chats`, `ai_chat_messages`, `ai_provider_credentials`, `ai_mcp_servers`, `page_template_references`) and columns (e.g. `pages.is_template`, a `NOT NULL DEFAULT false` boolean) — never drop/rewrite Docmost data.
**Migration ordering — always check when merging branches/features.** Kysely runs migrations in **alphabetical (= timestamp) order** and refuses to start if a *new* migration sorts **before** one already applied to the DB (`corrupted migrations: ... must always have a name that comes alphabetically after the last executed migration`). When you merge a branch or land a feature, verify your migration's timestamp still sorts **after every migration that may already be applied on the target** (`/bin/ls -1 apps/server/src/database/migrations | sort | tail`). Branches developed in parallel routinely break this: a feature branch adds `…T130000-…`, `main` meanwhile ships and deploys `…T150000-…`, and after the merge the older-timestamped file is rejected at boot. **Fix = rename your migration to a timestamp after the latest one already in the target** (content unchanged — the filename is the ordering key), then rebuild so the compiled `dist/database/migrations/` picks up the new name. **Migration ordering — always check when merging branches/features.** Kysely runs migrations in **alphabetical (= timestamp) order**. A *new* migration that sorts **before** one already applied to the DB is a "back-dated" migration, which branches developed in parallel routinely produce: a feature branch adds `…T130000-…`, `develop` meanwhile ships and deploys `…T150000-…`, and after the merge the older-timestamped file has been skipped. Two layers guard this (both added for incident #361, where a back-dated migration crash-looped prod for ~11 min):
- **CI gate (primary):** the `migration-order` job in `.github/workflows/test.yml` fails a PR whose added migration sorts at/before the newest on the base branch. **So the fix is to rename your migration to a timestamp after the latest one already in the target** (`/bin/ls -1 apps/server/src/database/migrations | sort | tail`; content unchanged — the filename is the ordering key), then rebuild so the compiled `dist/database/migrations/` picks up the new name.
- **Runtime safety net:** both Migrators (`migration.service.ts` startup auto-migrate + `migrate.ts` CLI) set `allowUnorderedMigrations: true`, so the app does **not** refuse to start on an out-of-order migration — it applies the skipped older one instead of crash-looping. Kysely's `#ensureNoMissingMigrations` guard is still on (a *removed* applied migration is still an error). Because apply order can then differ from lexicographic across instances, migrations must stay **independent** (each creates its own objects) — the CI gate remains the primary line; this net only covers a gate bypass (manual push / hotfix branch).
## Architecture — the big picture ## Architecture — the big picture
@@ -279,11 +288,12 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
- `core/ai-chat/tools/` — the agent's ~40 read+write tools. Every tool runs under the **calling user's** CASL permissions via a per-user loopback access token (`docmost-client.loader.ts`), so the agent can never exceed what the user could do. Only **reversible** operations are exposed (page history + trash; no permanent delete). Agent edits get an "AI agent" provenance badge in page history (`20260616T130000-agent-provenance` migration). - `core/ai-chat/tools/` — the agent's ~40 read+write tools. Every tool runs under the **calling user's** CASL permissions via a per-user loopback access token (`docmost-client.loader.ts`), so the agent can never exceed what the user could do. Only **reversible** operations are exposed (page history + trash; no permanent delete). Agent edits get an "AI agent" provenance badge in page history (`20260616T130000-agent-provenance` migration).
- `core/ai-chat/embedding/` — RAG indexer + a BullMQ consumer on `AI_QUEUE` that embeds pages into `page_embeddings` (vector search), complementing Postgres full-text search. Pages are (re)indexed on edit; `AI_EMBEDDING_TIMEOUT_MS` bounds a hung embeddings endpoint. - `core/ai-chat/embedding/` — RAG indexer + a BullMQ consumer on `AI_QUEUE` that embeds pages into `page_embeddings` (vector search), complementing Postgres full-text search. Pages are (re)indexed on edit; `AI_EMBEDDING_TIMEOUT_MS` bounds a hung embeddings endpoint.
- `core/ai-chat/external-mcp/` — admins can attach external MCP servers (e.g. Tavily) to give the agent web access. **`ssrf-guard.ts` validates outbound MCP URLs against SSRF** — keep that guard in the path when touching external-MCP connection logic. - `core/ai-chat/external-mcp/` — admins can attach external MCP servers (e.g. Tavily) to give the agent web access. **`ssrf-guard.ts` validates outbound MCP URLs against SSRF** — keep that guard in the path when touching external-MCP connection logic.
- `core/ai-chat/ai-chat-run.service.ts` + `ai_chat_runs`**detached/autonomous agent runs** (`#184`), behind the per-workspace `settings.ai.autonomousRuns` flag (off by default). When on, a turn becomes a server-side RUN that survives a browser disconnect; only an explicit `POST /ai-chat/stop` ends it, and a client reconnects/live-follows via `POST /ai-chat/run`. **DEPLOY CONSTRAINT — single-instance only in phase 1:** Stop and the AbortController that backs it are process-local, so a Stop only aborts a run executing on the **same** replica that owns it (cross-instance pub/sub stop is phase 2). Do **not** enable `autonomousRuns` on a horizontally-scaled deployment (multiple replicas behind a load balancer, or Docmost cloud `CLOUD=true`) — run a single instance instead. The server logs a startup WARNING when it detects a multi-instance deployment (`CLOUD=true`) so the constraint is visible. The startup sweep settles any run left dangling by a restart.
### Client structure ### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions: Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI. - **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, import/export) — editor schema changes often need to be made in `editor-ext`, not just the client. The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by both `mcp` and `git-sync` — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. - The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`. - API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`. - Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
@@ -293,7 +303,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- **Errors must never be swallowed or shown as generic messages.** Every caught error MUST (1) be logged in full to the console/logger — error name, message, stack, `cause`, and (for HTTP/provider failures) the status code and response body — and (2) be surfaced to the user with a *specific, human-readable explanation of what actually went wrong*, never a bare generic string like "Something went wrong" / "Could not start recording" / "Transcription failed". Include the real reason (the underlying error/provider message) in the user-facing text. On the server, wrap third-party/provider failures with `describeProviderError` (or equivalent) and rethrow as a meaningful HTTP status + message — never let them collapse into an opaque 500. On the client, `console.error(<context>, err)` the raw error AND show the extracted reason (e.g. `err.response?.data?.message`, or the error `name: message`) in the notification. - **Errors must never be swallowed or shown as generic messages.** Every caught error MUST (1) be logged in full to the console/logger — error name, message, stack, `cause`, and (for HTTP/provider failures) the status code and response body — and (2) be surfaced to the user with a *specific, human-readable explanation of what actually went wrong*, never a bare generic string like "Something went wrong" / "Could not start recording" / "Transcription failed". Include the real reason (the underlying error/provider message) in the user-facing text. On the server, wrap third-party/provider failures with `describeProviderError` (or equivalent) and rethrow as a meaningful HTTP status + message — never let them collapse into an opaque 500. On the client, `console.error(<context>, err)` the raw error AND show the extracted reason (e.g. `err.response?.data?.message`, or the error `name: message`) in the notification.
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`. - The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly. - Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. - Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire test: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`) — it MUST be re-created via `pnpm patch` when bumping `ai`.
- **Adding/renaming/removing an MCP tool requires updating `SERVER_INSTRUCTIONS`** in `packages/mcp/src/index.ts` — the intent-routing guide MCP clients receive on initialize. This applies both to inline `server.registerTool(...)` calls in `index.ts` and to specs in `packages/mcp/src/tool-specs.ts`. Enforced by `packages/mcp/test/unit/server-instructions.test.mjs`, which fails when a registered tool is not mentioned in the guide (deliberate opt-outs go into its `EXCEPTIONS` list). `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests. - **Adding/renaming/removing an MCP tool requires updating `SERVER_INSTRUCTIONS`** in `packages/mcp/src/index.ts` — the intent-routing guide MCP clients receive on initialize. This applies both to inline `server.registerTool(...)` calls in `index.ts` and to specs in `packages/mcp/src/tool-specs.ts`. Enforced by `packages/mcp/test/unit/server-instructions.test.mjs`, which fails when a registered tool is not mentioned in the guide (deliberate opt-outs go into its `EXCEPTIONS` list). `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release ## CI / release
+21
View File
@@ -72,6 +72,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
append/prepend fragments, nor to COMMENT bodies — a comment may legitimately append/prepend fragments, nor to COMMENT bodies — a comment may legitimately
contain a standalone footnote definition, which canonicalization would drop. contain a standalone footnote definition, which canonicalization would drop.
(#228) (#228)
- **Detached, autonomous agent runs that survive a browser disconnect.** When the
new `settings.ai.autonomousRuns` workspace flag is on (off by default), an
AI-chat turn becomes a first-class, server-side RUN tracked in a new
`ai_chat_runs` table instead of a socket-bound stream: closing the tab or
losing the connection no longer aborts the turn — it keeps executing and
persisting server-side, and only an explicit Stop ends it. A client can
reconnect and live-follow (or stop) an in-flight run via `POST /ai-chat/run`
(resolve the latest run + its assistant message for a chat) and
`POST /ai-chat/stop` (stop by `runId` or `chatId`). A partial unique index
enforces one active run per chat, and a startup sweep settles any run left
dangling by a restart. Phase 1 is single-instance-only (cross-instance Stop is
not yet reliable); the server warns at startup on a horizontally-scaled
deployment. (#184)
- **Out-of-band page transfer via an in-RAM blob sandbox (`stash_page`).** A - **Out-of-band page transfer via an in-RAM blob sandbox (`stash_page`).** A
new MCP tool serializes a whole page (its full ProseMirror JSON, with every new MCP tool serializes a whole page (its full ProseMirror JSON, with every
internal image/file mirrored) into an ephemeral in-RAM blob and returns only internal image/file mirrored) into an ephemeral in-RAM blob and returns only
@@ -156,6 +169,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed ### Fixed
- **The server no longer runs out of heap during long autonomous agent runs.** A
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
snapshot of the ENTIRE turn text on every streamed text-delta when no output
strategy was requested (our server never requests one). Unpatched, those
O(n²) `partialOutput` snapshots piled up in a never-consumed internal
`tee()` branch of the stream result — a ~20-step, ~28k-chunk agent run
retained ~1.7 GB and OOM'd the 2 GB JS heap. Streaming granularity is
unchanged; the patch must be re-created if `ai` is ever bumped. (#184)
- **Internal links in exported Markdown no longer lose their visible text.** A - **Internal links in exported Markdown no longer lose their visible text.** A
link whose target page name had no file extension (e.g. a bare title) was link whose target page name had no file extension (e.g. a bare title) was
collapsed to empty text during export, producing an unclickable, label-less collapsed to empty text during export, producing an unclickable, label-less
@@ -1373,6 +1373,39 @@
"The role catalog is unavailable": "The role catalog is unavailable", "The role catalog is unavailable": "The role catalog is unavailable",
"Please try again later.": "Please try again later.", "Please try again later.": "Please try again later.",
"No bundles available": "No bundles available", "No bundles available": "No bundles available",
"Content": "Content",
"Content language of the roles": "Content language of the roles",
"{{count}} updates available in {{bundles}} bundles": "{{count}} updates available in {{bundles}} bundles",
"Update all ({{count}})": "Update all ({{count}})",
"Updating {{current}}/{{total}}…": "Updating {{current}}/{{total}}…",
"{{count}} roles are installed in another language. A different language installs separately and appears as new.": "{{count}} roles are installed in another language. A different language installs separately and appears as new.",
"{{count}} roles": "{{count}} roles",
"{{count}} new — none installed": "{{count}} new — none installed",
"All installed · up to date": "All installed · up to date",
"{{count}} updates · {{installed}} up to date": "{{count}} updates · {{installed}} up to date",
"{{count}} new": "{{count}} new",
"{{count}} installed": "{{count}} installed",
"{{count}} updates": "{{count}} updates",
"Install bundle": "Install bundle",
"Install {{count}} selected": "Install {{count}} selected",
"Install bundle ({{count}})": "Install bundle ({{count}})",
"{{selected}} of {{total}} selected": "{{selected}} of {{total}} selected",
"Select all": "Select all",
"Deselect all": "Deselect all",
"Skipped": "Skipped",
"v{{version}}": "v{{version}}",
"{{count}} roles installed": "{{count}} roles installed",
"{{count}} roles installed · {{renamed}} renamed": "{{count}} roles installed · {{renamed}} renamed",
"{{count}} roles updated": "{{count}} roles updated",
"Installed {{installed}} · {{skipped}} skipped": "Installed {{installed}} · {{skipped}} skipped",
"A role named \"{{name}}\" already exists in this workspace.": "A role named \"{{name}}\" already exists in this workspace.",
"\"{{name}}\" is already installed.": "\"{{name}}\" is already installed.",
"Rename & install": "Rename & install",
"Couldn’t load the catalog": "Couldn’t load the catalog",
"Check your connection and try again. Installed roles are not affected.": "Check your connection and try again. Installed roles are not affected.",
"Retry": "Retry",
"The catalog is empty": "The catalog is empty",
"No role bundles are published for this language yet. Try switching the content language.": "No role bundles are published for this language yet. Try switching the content language.",
"Already up to date": "Already up to date", "Already up to date": "Already up to date",
"Updated to the latest version": "Updated to the latest version", "Updated to the latest version": "Updated to the latest version",
"This role is no longer in the catalog": "This role is no longer in the catalog", "This role is no longer in the catalog": "This role is no longer in the catalog",
@@ -1235,6 +1235,39 @@
"The role catalog is unavailable": "Каталог ролей недоступен", "The role catalog is unavailable": "Каталог ролей недоступен",
"Please try again later.": "Попробуйте позже.", "Please try again later.": "Попробуйте позже.",
"No bundles available": "Наборы недоступны", "No bundles available": "Наборы недоступны",
"Content": "Язык контента",
"Content language of the roles": "Язык контента ролей",
"{{count}} updates available in {{bundles}} bundles": "Доступно обновлений: {{count}} в наборах: {{bundles}}",
"Update all ({{count}})": "Обновить все ({{count}})",
"Updating {{current}}/{{total}}…": "Обновление {{current}}/{{total}}…",
"{{count}} roles are installed in another language. A different language installs separately and appears as new.": "Ролей установлено на другом языке: {{count}}. Другой язык устанавливается отдельно и отображается как новый.",
"{{count}} roles": "ролей: {{count}}",
"{{count}} new — none installed": "новых: {{count}} — ничего не установлено",
"All installed · up to date": "Все установлены · актуальны",
"{{count}} updates · {{installed}} up to date": "обновлений: {{count}} · актуальны: {{installed}}",
"{{count}} new": "новых: {{count}}",
"{{count}} installed": "установлено: {{count}}",
"{{count}} updates": "обновлений: {{count}}",
"Install bundle": "Установить набор",
"Install {{count}} selected": "Установить выбранные ({{count}})",
"Install bundle ({{count}})": "Установить набор ({{count}})",
"{{selected}} of {{total}} selected": "выбрано {{selected}} из {{total}}",
"Select all": "Выбрать все",
"Deselect all": "Снять выбор",
"Skipped": "Пропущено",
"v{{version}}": "v{{version}}",
"{{count}} roles installed": "Установлено ролей: {{count}}",
"{{count}} roles installed · {{renamed}} renamed": "Установлено ролей: {{count}} · переименовано: {{renamed}}",
"{{count}} roles updated": "Обновлено ролей: {{count}}",
"Installed {{installed}} · {{skipped}} skipped": "Установлено: {{installed}} · пропущено: {{skipped}}",
"A role named \"{{name}}\" already exists in this workspace.": "Роль с именем «{{name}}» уже существует в этом рабочем пространстве.",
"\"{{name}}\" is already installed.": "«{{name}}» уже установлена.",
"Rename & install": "Переименовать и установить",
"Couldn’t load the catalog": "Не удалось загрузить каталог",
"Check your connection and try again. Installed roles are not affected.": "Проверьте подключение и попробуйте снова. Установленные роли не затронуты.",
"Retry": "Повторить",
"The catalog is empty": "Каталог пуст",
"No role bundles are published for this language yet. Try switching the content language.": "Для этого языка ещё не опубликовано ни одного набора ролей. Попробуйте сменить язык контента.",
"No roles configured": "Роли не настроены", "No roles configured": "Роли не настроены",
"Already up to date": "Уже актуальна", "Already up to date": "Уже актуальна",
"Updated to the latest version": "Обновлено до последней версии", "Updated to the latest version": "Обновлено до последней версии",
@@ -19,7 +19,7 @@ import {
IconPlus, IconPlus,
IconX, IconX,
} from "@tabler/icons-react"; } from "@tabler/icons-react";
import { useAtom, useSetAtom } from "jotai"; import { useAtom, useAtomValue, useSetAtom } from "jotai";
import { useLocation, useMatch } from "react-router-dom"; import { useLocation, useMatch } from "react-router-dom";
import { useTranslation } from "react-i18next"; import { useTranslation } from "react-i18next";
import { useQueryClient } from "@tanstack/react-query"; import { useQueryClient } from "@tanstack/react-query";
@@ -41,13 +41,24 @@ import { extractPageSlugId } from "@/lib";
import { import {
AI_CHATS_RQ_KEY, AI_CHATS_RQ_KEY,
AI_CHAT_MESSAGES_RQ_KEY, AI_CHAT_MESSAGES_RQ_KEY,
AI_CHAT_RUN_RQ_KEY,
useAiChatMessagesQuery, useAiChatMessagesQuery,
useAiChatRunQuery,
useAiChatsQuery, useAiChatsQuery,
useAiRolesQuery, useAiRolesQuery,
} from "@/features/ai-chat/queries/ai-chat-query.ts"; } from "@/features/ai-chat/queries/ai-chat-query.ts";
import {
shouldClearLatchOnQueryError,
shouldClearStoppingLatch,
shouldObserveRun,
} from "@/features/ai-chat/utils/run-polling.ts";
import { workspaceAtom } from "@/features/user/atoms/current-user-atom";
import ConversationList from "@/features/ai-chat/components/conversation-list.tsx"; import ConversationList from "@/features/ai-chat/components/conversation-list.tsx";
import ChatThread from "@/features/ai-chat/components/chat-thread.tsx"; import ChatThread from "@/features/ai-chat/components/chat-thread.tsx";
import { exportAiChat } from "@/features/ai-chat/services/ai-chat-service.ts"; import {
exportAiChat,
stopRun,
} from "@/features/ai-chat/services/ai-chat-service.ts";
import { useChatSession } from "@/features/ai-chat/hooks/use-chat-session.ts"; import { useChatSession } from "@/features/ai-chat/hooks/use-chat-session.ts";
import { import {
shouldCollapseOnOutsidePointer, shouldCollapseOnOutsidePointer,
@@ -234,6 +245,147 @@ export default function AiChatWindow() {
const { data: messageRows, isLoading: messagesLoading } = const { data: messageRows, isLoading: messagesLoading } =
useAiChatMessagesQuery(activeChatId ?? undefined); useAiChatMessagesQuery(activeChatId ?? undefined);
// #184 reconnect-and-live-follow. Whether detached agent runs are enabled for
// this workspace. The reconnect endpoint itself is NOT flag-gated server-side
// (it is only owner-gated and returns `{ run: null }` when the chat has no
// run); but when the feature is off no runs are ever created, so polling it
// would always come back empty — we gate it off here to avoid pointless polls.
const workspace = useAtomValue(workspaceAtom);
const autonomousRunsEnabled =
workspace?.settings?.ai?.autonomousRuns === true;
// Whether THIS tab is the one actively streaming the open chat's run locally
// (it started the run here and holds the SSE). Reported up from ChatThread. We
// are the STREAMER while true and a passive OBSERVER while false — the basis of
// the observer-vs-streamer detection. Reset to false by the fresh ChatThread's
// mount effect on every chat switch.
const [localStreaming, setLocalStreaming] = useState(false);
const onStreamingChange = useCallback((streaming: boolean) => {
setLocalStreaming(streaming);
}, []);
// #184 Stop wiring. While a detached run is being stopped we SUPPRESS the
// observer merge so the stopping run's still-persisting output does not
// re-stream back into view between the moment the user pressed Stop and the run
// actually settling as 'aborted' server-side. Polling itself keeps running (so
// the terminal transition is still detected) — only the visual merge is gated.
// Cleared when the run is observed terminal (below) or the chat is switched.
const [stoppingRun, setStoppingRun] = useState(false);
// Reset the stopping latch whenever the open chat changes: it is scoped to the
// run of the previously-open chat.
useEffect(() => {
setStoppingRun(false);
}, [activeChatId]);
// Authoritative stop of the open chat's detached run (the Stop button in
// autonomous mode). Latch "stopping" first (suppresses the re-stream flash),
// then request the server stop — the ONLY thing that ends a detached run; a mere
// local SSE abort is a client disconnect the server ignores. On failure we
// release the latch so the observer resumes (better to show the live run than to
// freeze the view) and surface the error.
const handleServerStop = useCallback(
(chatId: string): void => {
setStoppingRun(true);
// #234 F4: drop the PREVIOUS turn's run from the cache so `run` becomes null
// until the CURRENT turn's run is fetched fresh. Without this, once the local
// stream aborts (localStreaming -> false) the run query re-enables and
// react-query SYNCHRONOUSLY returns the still-cached prior terminal run; the
// terminal effect would then clear the stopping latch against that STALE run
// before the current turn's (still-running, detached, growing) run is ever
// observed — re-opening the observer merge and flashing the growing output
// over the frozen row. With the cache cleared the terminal effect's
// `if (!run) return` holds the latch until the current run itself is observed
// terminal (see shouldClearStoppingLatch).
queryClient.removeQueries({ queryKey: AI_CHAT_RUN_RQ_KEY(chatId) });
void stopRun(chatId).catch(() => {
setStoppingRun(false);
notifications.show({
message: t("Failed to stop the run"),
color: "red",
});
});
},
[t, queryClient],
);
// Poll the latest run of the open chat ONLY when we are a passive observer:
// feature on, a chat is open, and we are NOT the local streamer (the streamer
// already has the live SSE — polling/merging too would double-render). The
// query's own status-keyed refetchInterval stops once the run is terminal.
const { data: runData, isError: runQueryFailed } = useAiChatRunQuery(
activeChatId ?? undefined,
autonomousRunsEnabled && !localStreaming,
);
const run = runData?.run ?? null;
// Safety net (#234 F4 review): after handleServerStop clears the run cache,
// `run` is null until the current turn's run is fetched fresh, and the terminal
// effect below holds the latch via `if (!run) return`. If that refetch instead
// ERRORS PERMANENTLY (the GET-run keeps failing) while we are no longer the
// streamer, the run stays null, its status-keyed refetchInterval is off, and
// nothing would ever observe a terminal run — freezing the view with the
// observer merge suppressed. Release the latch on that error so the live view
// resumes rather than stays stuck (the local stopRun may already have succeeded
// independently).
//
// #234 F7: this must NOT fire on a TRANSIENT error while `run` is still an
// ACTIVE held run. In TanStack Query v5 (retry:false) the query's `data` is
// RETAINED on error, so `runQueryFailed` can be true while `run` is still
// pending/running — releasing then would re-open the observer merge and flash
// the growing detached run over the frozen row (the very flash F4 prevents). The
// decision is the pure, unit-tested `shouldClearLatchOnQueryError`, which gates
// on the run NOT being active: it cures only the genuine permanent-null-freeze
// (`run === null`) and never releases against an active run.
useEffect(() => {
if (
shouldClearLatchOnQueryError({
stoppingRun,
isLocalStreaming: localStreaming,
runQueryFailed,
run,
})
)
setStoppingRun(false);
}, [stoppingRun, localStreaming, runQueryFailed, run]);
// The run's incrementally-persisted assistant message to merge into the thread,
// but only while we are an observer (never when we are the streamer — guards
// against a stale poll fighting the live stream). Includes a terminal run so the
// final persisted output is shown on reopen.
const observedRow =
shouldObserveRun(run, localStreaming) && !stoppingRun
? (runData?.message ?? null)
: null;
// When the observed run reaches a terminal status, do a final messages refetch
// so the persisted final state (token/context badge, export source) is shown,
// then the query's refetchInterval has already stopped polling. Deduped per run
// id so it fires exactly once per run, not on every subsequent poll-less render.
const finalizedRunIdRef = useRef<string | null>(null);
useEffect(() => {
if (!run || !activeChatId) return;
if (run.status === "pending" || run.status === "running") {
// Active again (a new run) — re-arm so its terminal transition fires once.
finalizedRunIdRef.current = null;
return;
}
// Terminal: a stop we requested has landed (or the run finished on its own),
// so release the stopping latch — the observer merge can now show the final
// persisted (aborted/finished) output without any live re-stream. The decision
// is the pure, unit-tested `shouldClearStoppingLatch` (run-polling.ts): release
// ONLY when we requested a stop, this tab is no longer the streamer, AND the
// CURRENT run is terminal. The #234 F4 cache removal in handleServerStop makes
// `run` null (this branch's `if (!run) return` above holds) until the current
// turn's run is fetched fresh, so the latch can never clear against a stale
// cached run.
if (shouldClearStoppingLatch({ stoppingRun, run, isLocalStreaming: localStreaming }))
setStoppingRun(false);
if (finalizedRunIdRef.current === run.id) return;
finalizedRunIdRef.current = run.id;
queryClient.invalidateQueries({
queryKey: AI_CHAT_MESSAGES_RQ_KEY(activeChatId),
});
}, [run, activeChatId, queryClient, stoppingRun, localStreaming]);
// The page the user is currently viewing. AiChatWindow lives in a pathless // The page the user is currently viewing. AiChatWindow lives in a pathless
// parent layout route, so useParams() can't see :pageSlug. Match the full // parent layout route, so useParams() can't see :pageSlug. Match the full
// pathname against the authenticated page route instead so "the current page" // pathname against the authenticated page route instead so "the current page"
@@ -882,6 +1034,18 @@ export default function AiChatWindow() {
assistantName={currentRole?.name} assistantName={currentRole?.name}
onTurnFinished={onTurnFinished} onTurnFinished={onTurnFinished}
onServerChatId={onServerChatId} onServerChatId={onServerChatId}
// #184: live-follow a still-running run when we reopened the chat as
// a passive observer; null when there is nothing to observe or this
// tab is the streamer. onStreamingChange lets the window stop polling
// while we are the streamer.
observedRow={observedRow}
onStreamingChange={onStreamingChange}
// #184: in autonomous mode the Stop button must hit the authoritative
// server stop (a local SSE abort is a client disconnect the server
// ignores). onServerStop also arms the "stopping" latch above so the
// stopped run's output does not re-stream via the observer merge.
autonomousRunsEnabled={autonomousRunsEnabled}
onServerStop={handleServerStop}
/> />
)} )}
</div> </div>
@@ -11,6 +11,7 @@ const h = vi.hoisted(() => ({
onFinish: null as null | ((arg: Record<string, unknown>) => void), onFinish: null as null | ((arg: Record<string, unknown>) => void),
sendMessage: vi.fn(), sendMessage: vi.fn(),
stop: vi.fn(), stop: vi.fn(),
setMessages: vi.fn(),
transport: null as null | { transport: null as null | {
prepareSendMessagesRequest: (arg: { prepareSendMessagesRequest: (arg: {
messages: unknown[]; messages: unknown[];
@@ -30,6 +31,8 @@ vi.mock("@ai-sdk/react", () => ({
status: h.state.status, status: h.state.status,
stop: h.state.stop, stop: h.state.stop,
error: null, error: null,
// #184: ChatThread reads setMessages to merge a polled observer run.
setMessages: h.state.setMessages,
}; };
}, },
})); }));
@@ -228,3 +231,56 @@ describe("ChatThread — turn-end decision (onFinish)", () => {
} }
}); });
}); });
// #184 passive-observer merge: when reconnecting to a still-running run, the
// parent feeds the polled run message via `observedRow`; ChatThread merges it via
// setMessages — but ONLY when this tab is NOT itself streaming (the streamer's
// SSE owns the view, so a stale observedRow must never overwrite it).
describe("ChatThread — observer run merge (#184)", () => {
beforeEach(() => {
h.state.onFinish = null;
h.state.setMessages.mockReset();
});
const observedRow = {
id: "a-run",
role: "assistant",
content: "step 1\nstep 2",
metadata: {
parts: [{ type: "text", text: "step 1\nstep 2" }],
},
createdAt: "2026-01-01T00:00:00Z",
} as const;
function renderObserver(status: string) {
h.state.status = status;
render(
<MantineProvider>
<ChatThread
chatId="c1"
initialRows={[]}
onTurnFinished={vi.fn()}
observedRow={observedRow as never}
/>
</MantineProvider>,
);
}
it("merges the polled run message when this tab is a passive observer", () => {
renderObserver("ready");
expect(h.state.setMessages).toHaveBeenCalledTimes(1);
// The updater replaces/append the observed assistant row by id.
const updater = h.state.setMessages.mock.calls[0][0] as (
prev: { id: string; parts: { text: string }[] }[],
) => { id: string; parts: { text: string }[] }[];
const merged = updater([{ id: "u1", parts: [{ text: "hi" }] }]);
expect(merged).toHaveLength(2);
expect(merged[1].id).toBe("a-run");
expect(merged[1].parts[0].text).toBe("step 1\nstep 2");
});
it("does NOT merge while THIS tab is the streamer (no double-render)", () => {
renderObserver("streaming");
expect(h.state.setMessages).not.toHaveBeenCalled();
});
});
@@ -24,6 +24,7 @@ import {
} from "@/features/ai-chat/utils/role-launch.ts"; } from "@/features/ai-chat/utils/role-launch.ts";
import { describeChatError } from "@/features/ai-chat/utils/error-message.ts"; import { describeChatError } from "@/features/ai-chat/utils/error-message.ts";
import { extractServerChatId } from "@/features/ai-chat/utils/adopt-chat-id.ts"; import { extractServerChatId } from "@/features/ai-chat/utils/adopt-chat-id.ts";
import { mergeObservedMessage } from "@/features/ai-chat/utils/run-polling.ts";
import { import {
dequeue, dequeue,
enqueueMessage, enqueueMessage,
@@ -86,6 +87,29 @@ interface ChatThreadProps {
* Copy/export button available mid-stream). Distinct from onTurnFinished, * Copy/export button available mid-stream). Distinct from onTurnFinished,
* which fires only at the terminal outcome. */ * which fires only at the terminal outcome. */
onServerChatId?: (serverChatId?: string) => void; onServerChatId?: (serverChatId?: string) => void;
/** #184 reconnect-and-live-follow. When THIS tab reopened a chat whose agent
* run is still going (it is a PASSIVE OBSERVER — it did not start the run here),
* the parent polls the reconnect endpoint and feeds the run's incrementally-
* persisted assistant message here; we merge it into the live list so new
* steps/tool-calls appear as they are persisted. Null when there is nothing to
* observe (no run, feature off, or this tab IS the streamer). The merge is
* ADDITIONALLY guarded by our own `isStreaming`, so a stale value can never
* fight the local stream when we are the streamer. */
observedRow?: IAiChatMessageRow | null;
/** Report this tab's live streaming status up to the parent, so it can stop
* polling the run while WE are the active streamer (the SSE owns the view) and
* resume once we go idle. Called from an effect on every transition. */
onStreamingChange?: (streaming: boolean) => void;
/** #184: whether detached/autonomous agent runs are enabled for this workspace.
* When true the Stop button must additionally hit the AUTHORITATIVE server stop
* (via onServerStop) — aborting only the local SSE is just a client disconnect,
* which the server deliberately ignores, so the detached run would keep going. */
autonomousRunsEnabled?: boolean;
/** #184: request the server-side stop of this chat's active run (the parent owns
* the endpoint call + the "stopping" latch that keeps observer-polling from
* immediately re-streaming the stopping run's output). Called with the resolved
* chat id when the user presses Stop in autonomous mode. */
onServerStop?: (chatId: string) => void;
} }
/** /**
@@ -131,6 +155,10 @@ export default function ChatThread({
assistantName, assistantName,
onTurnFinished, onTurnFinished,
onServerChatId, onServerChatId,
observedRow,
onStreamingChange,
autonomousRunsEnabled,
onServerStop,
}: ChatThreadProps) { }: ChatThreadProps) {
const { t } = useTranslation(); const { t } = useTranslation();
@@ -216,6 +244,16 @@ export default function ChatThread({
const flushOnAbortRef = useRef(false); const flushOnAbortRef = useRef(false);
const interruptNextSendRef = useRef(false); const interruptNextSendRef = useRef(false);
// #234 F5: the user pressed Stop while streaming a BRAND-NEW chat whose server
// chat id has not been adopted yet (the `start` chunk carrying it hadn't landed
// when Stop was pressed). A local SSE abort alone does NOT stop the DETACHED
// autonomous run — it keeps burning tokens and WRITING TO PAGES — so we cannot
// just no-op. We latch the stop as PENDING and fire the authoritative server
// stop the moment onServerChatId adopts the id (below). Read-and-cleared there;
// also defused on every new turn start so it can never fire against a later,
// unrelated turn's run.
const stopPendingRef = useRef(false);
// FIFO dequeue + send the next queued message (no-op when the queue is empty). // FIFO dequeue + send the next queued message (no-op when the queue is empty).
// Returns whether a message was actually sent, so callers can tell an empty // Returns whether a message was actually sent, so callers can tell an empty
// dequeue (nothing to flush) from a real send. // dequeue (nothing to flush) from a real send.
@@ -274,7 +312,7 @@ export default function ChatThread({
[], [],
); );
const { messages, sendMessage, status, stop, error } = useChat({ const { messages, sendMessage, status, stop, error, setMessages } = useChat({
// Stable per-mount key. Existing chats use their real id; new chats use a // Stable per-mount key. Existing chats use their real id; new chats use a
// generated client id (never `undefined`) so the store is NOT re-created on // generated client id (never `undefined`) so the store is NOT re-created on
// every render mid-stream (see `chatStoreId` above). // every render mid-stream (see `chatStoreId` above).
@@ -365,7 +403,14 @@ export default function ChatThread({
return; return;
lastForwardedChatIdRef.current = serverChatId; lastForwardedChatIdRef.current = serverChatId;
onServerChatId(serverChatId); onServerChatId(serverChatId);
}, [messages, onServerChatId]); // #234 F5: if Stop was pressed before the id was known, the authoritative
// server stop was deferred to this adoption point — fire it now with the
// just-adopted id. One-shot (read-and-clear) so it can't fire twice.
if (stopPendingRef.current) {
stopPendingRef.current = false;
onServerStop?.(serverChatId);
}
}, [messages, onServerChatId, onServerStop]);
// Live "turn was interrupted" marker for the CURRENT session. The red error // Live "turn was interrupted" marker for the CURRENT session. The red error
// banner (driven by `error`) covers the error case; this covers an aborted // banner (driven by `error`) covers the error case; this covers an aborted
@@ -378,6 +423,27 @@ export default function ChatThread({
const isStreaming = status === "submitted" || status === "streaming"; const isStreaming = status === "submitted" || status === "streaming";
// #184: report our live streaming status up so the parent stops polling the run
// while WE are the streamer (the SSE owns the view) and resumes once we go idle.
// Effect (not render) so it never updates parent state during our own render;
// fires on mount with `false`, which also re-syncs the parent after a chat
// switch remounts this thread (a fresh mount is idle until the user sends).
useEffect(() => {
onStreamingChange?.(isStreaming);
}, [isStreaming, onStreamingChange]);
// #184 passive-observer merge: when the parent feeds a polled run message (we
// reopened a chat whose run is still going and did NOT start it here), merge it
// into the live list so new steps/tool-calls appear as they are persisted. Hard-
// gated by `!isStreaming`: if THIS tab is actually the streamer, the local SSE
// owns the view and a stale observedRow must never overwrite it. `observedRow`
// is a stable per-poll object, so this runs once per poll, not per render.
useEffect(() => {
if (isStreaming || !observedRow) return;
const observed = rowToUiMessage(observedRow);
setMessages((prev) => mergeObservedMessage(prev, observed));
}, [observedRow, isStreaming, setMessages]);
// "Send now" on a queued message: interrupt the current turn and immediately // "Send now" on a queued message: interrupt the current turn and immediately
// send THIS message, keeping the agent's partial output. Other queued messages // send THIS message, keeping the agent's partial output. Other queued messages
// stay queued and flush normally after the new turn. Reuses the existing // stay queued and flush normally after the new turn. Reuses the existing
@@ -409,6 +475,40 @@ export default function ChatThread({
[setQueue, stop], [setQueue, stop],
); );
// Stop the current turn. ALWAYS abort the local SSE (`stop()`) so the composer
// returns to idle immediately. In AUTONOMOUS mode the turn is a DETACHED run:
// aborting the local SSE is only a client disconnect, which the server ignores,
// so the run would keep executing — we ADDITIONALLY request the authoritative
// server-side stop (the parent owns that call + the "stopping" latch that keeps
// observer-polling from re-streaming the stopping run's output). The chat id is
// read live from chatIdRef (adopted early at the stream's `start` chunk); if it
// is not known yet — a brand-new chat in the first moment of its first turn —
// only the local abort happens (there is no server-side run handle to stop yet).
const handleStop = useCallback(() => {
stop();
if (!autonomousRunsEnabled) return;
if (chatIdRef.current) {
onServerStop?.(chatIdRef.current);
} else {
// #234 F5: no chat id yet (brand-new chat in the first moment of its first
// turn, before the `start` chunk adopted the id). Latch the stop as pending;
// the onServerChatId adoption effect fires the deferred server stop as soon
// as the id appears, so the detached run is still authoritatively stopped
// instead of left running by a silent local-only abort.
//
// KNOWN LIMITATION (#234 F5 review): `stop()` above has already aborted the
// local SSE reader. In the rare sub-window where Stop is pressed while still
// `submitted` (request sent, not one chunk read yet), that abort can cancel
// the reader BEFORE the `start` chunk is applied to `messages`, so the
// adoption effect never runs and this pending stop never fires. The detached
// run then keeps going for that turn. This is not a regression (the pre-fix
// behavior sent no server stop at all); closing it fully would require
// deferring the local abort until adoption, which is riskier and out of scope
// for this fix. Documented so a future change can address the abort-ordering.
stopPendingRef.current = true;
}
}, [stop, autonomousRunsEnabled, onServerStop]);
// Clear the stopped marker as soon as a new turn begins streaming, and drop any // Clear the stopped marker as soon as a new turn begins streaming, and drop any
// stale "Send now" interrupt flags. On the legit interrupt path both refs are // stale "Send now" interrupt flags. On the legit interrupt path both refs are
// already consumed synchronously (onFinish + prepareSendMessagesRequest) before // already consumed synchronously (onFinish + prepareSendMessagesRequest) before
@@ -420,6 +520,11 @@ export default function ChatThread({
setStopNotice(null); setStopNotice(null);
flushOnAbortRef.current = false; flushOnAbortRef.current = false;
interruptNextSendRef.current = false; interruptNextSendRef.current = false;
// #234 F5: a new turn is starting — drop any pending deferred-stop from a
// previous turn that never adopted an id, so it can never fire against this
// (or a later) unrelated turn's run. A deferred stop for the CURRENT turn is
// set AFTER this effect (on the Stop click), so this does not clobber it.
stopPendingRef.current = false;
} }
}, [isStreaming]); }, [isStreaming]);
@@ -539,7 +644,7 @@ export default function ChatThread({
<ChatInput <ChatInput
onSend={(text) => sendMessage({ text })} onSend={(text) => sendMessage({ text })}
onQueue={enqueue} onQueue={enqueue}
onStop={stop} onStop={handleStop}
isStreaming={isStreaming} isStreaming={isStreaming}
/> />
</Stack> </Stack>
@@ -1,6 +1,7 @@
import { import {
useInfiniteQuery, useInfiniteQuery,
useMutation, useMutation,
useQueries,
useQuery, useQuery,
useQueryClient, useQueryClient,
} from "@tanstack/react-query"; } from "@tanstack/react-query";
@@ -12,6 +13,7 @@ import {
deleteAiChat, deleteAiChat,
deleteAiRole, deleteAiRole,
getAiChatMessages, getAiChatMessages,
getAiChatRun,
getAiChats, getAiChats,
getAiRoleCatalog, getAiRoleCatalog,
getAiRoleCatalogBundle, getAiRoleCatalogBundle,
@@ -24,6 +26,7 @@ import {
import { import {
IAiChat, IAiChat,
IAiChatMessageRow, IAiChatMessageRow,
IAiChatRunResponse,
IAiRole, IAiRole,
IAiRoleCatalog, IAiRoleCatalog,
IAiRoleCatalogBundle, IAiRoleCatalogBundle,
@@ -34,6 +37,7 @@ import {
IAiRoleUpdateFromCatalogResult, IAiRoleUpdateFromCatalogResult,
} from "@/features/ai-chat/types/ai-chat.types.ts"; } from "@/features/ai-chat/types/ai-chat.types.ts";
import { IPagination } from "@/lib/types.ts"; import { IPagination } from "@/lib/types.ts";
import { runPollInterval } from "@/features/ai-chat/utils/run-polling.ts";
export const AI_CHATS_RQ_KEY = ["ai-chats"]; export const AI_CHATS_RQ_KEY = ["ai-chats"];
export const AI_ROLES_RQ_KEY = ["ai-roles"]; export const AI_ROLES_RQ_KEY = ["ai-roles"];
@@ -51,16 +55,18 @@ export const AI_CHAT_MESSAGES_RQ_KEY = (chatId: string) => [
"ai-chat-messages", "ai-chat-messages",
chatId, chatId,
]; ];
export const AI_CHAT_RUN_RQ_KEY = (chatId: string) => ["ai-chat-run", chatId];
/** Paginated list of the current user's chats (auto-loads further pages). */ /** Paginated list of the current user's chats (auto-loads further pages). */
export function useAiChatsQuery() { export function useAiChatsQuery() {
const query = useInfiniteQuery({ const query = useInfiniteQuery({
queryKey: AI_CHATS_RQ_KEY, queryKey: AI_CHATS_RQ_KEY,
queryFn: ({ pageParam }) => queryFn: ({ pageParam }) => getAiChats({ cursor: pageParam, limit: 50 }),
getAiChats({ cursor: pageParam, limit: 50 }),
initialPageParam: undefined as string | undefined, initialPageParam: undefined as string | undefined,
getNextPageParam: (lastPage) => getNextPageParam: (lastPage) =>
lastPage.meta.hasNextPage ? (lastPage.meta.nextCursor ?? undefined) : undefined, lastPage.meta.hasNextPage
? (lastPage.meta.nextCursor ?? undefined)
: undefined,
}); });
const data = useMemo<IPagination<IAiChat> | undefined>(() => { const data = useMemo<IPagination<IAiChat> | undefined>(() => {
@@ -90,7 +96,9 @@ export function useAiChatMessagesQuery(chatId: string | undefined) {
getAiChatMessages({ chatId: chatId as string, cursor: pageParam }), getAiChatMessages({ chatId: chatId as string, cursor: pageParam }),
initialPageParam: undefined as string | undefined, initialPageParam: undefined as string | undefined,
getNextPageParam: (lastPage) => getNextPageParam: (lastPage) =>
lastPage.meta.hasNextPage ? (lastPage.meta.nextCursor ?? undefined) : undefined, lastPage.meta.hasNextPage
? (lastPage.meta.nextCursor ?? undefined)
: undefined,
enabled: !!chatId, enabled: !!chatId,
}); });
@@ -131,6 +139,34 @@ export function useAiChatMessagesQuery(chatId: string | undefined) {
}; };
} }
/**
* Reconnect to a chat's latest agent run and LIVE-FOLLOW it (#184). While the run
* is active the query re-polls every {@link runPollInterval} ms (driven off the
* fetched `run.status`, the same status-keyed refetchInterval pattern as the
* embeddings reindex polling); once the run reaches a terminal status — or there
* is no run — the interval returns `false` and polling stops on its own. Polling
* is thus naturally bounded by the run terminating; no separate timeout cap.
*
* `enabled` gates the whole thing: callers pass `false` when the autonomous-runs
* feature is off (the endpoint is NOT flag-gated server-side, but with the feature
* off the chat has no runs, so polling would only ever return `{ run: null }`) OR
* when THIS tab is the one actively streaming the run (the live SSE owns the view,
* so we must not also poll/merge). The global `retry: false` means a failed fetch
* leaves `data` undefined, so refetchInterval(undefined run) returns false — a
* failed fetch can never spin a tight loop.
*/
export function useAiChatRunQuery(
chatId: string | undefined,
enabled: boolean,
) {
return useQuery<IAiChatRunResponse, Error>({
queryKey: AI_CHAT_RUN_RQ_KEY(chatId ?? ""),
queryFn: () => getAiChatRun(chatId as string),
enabled: !!chatId && enabled,
refetchInterval: (query) => runPollInterval(query.state.data?.run),
});
}
export function useRenameAiChatMutation() { export function useRenameAiChatMutation() {
const queryClient = useQueryClient(); const queryClient = useQueryClient();
const { t } = useTranslation(); const { t } = useTranslation();
@@ -272,6 +308,29 @@ export function useAiRoleCatalogBundleQuery(
}); });
} }
/**
* Eagerly open EVERY listed bundle's content in parallel for one language. The
* redesigned catalog shows each bundle's status summary in its COLLAPSED header,
* which needs every role's install state up front — so contents can no longer be
* lazy-loaded on expand. The catalog is small, so a fan-out of `useQueries` (one
* cached read per bundle, sharing the same cache keys as
* `useAiRoleCatalogBundleQuery`) is cheap. Gated by `enabled` (modal open + a
* resolved language) so nothing fetches while the modal is closed.
*/
export function useAiRoleCatalogBundlesQueries(
bundleIds: string[],
language: string,
enabled: boolean,
) {
return useQueries({
queries: bundleIds.map((bundleId) => ({
queryKey: AI_ROLE_CATALOG_BUNDLE_RQ_KEY(bundleId, language),
queryFn: () => getAiRoleCatalogBundle(bundleId, language),
enabled: enabled && !!language,
})),
});
}
export function useImportAiRolesFromCatalogMutation() { export function useImportAiRolesFromCatalogMutation() {
const queryClient = useQueryClient(); const queryClient = useQueryClient();
const { t } = useTranslation(); const { t } = useTranslation();
@@ -280,11 +339,14 @@ export function useImportAiRolesFromCatalogMutation() {
mutationFn: (payload) => importAiRolesFromCatalog(payload), mutationFn: (payload) => importAiRolesFromCatalog(payload),
onSuccess: (result) => { onSuccess: (result) => {
notifications.show({ notifications.show({
message: t("Imported {{created}}, renamed {{renamed}}, skipped {{skipped}}", { message: t(
created: result.created, "Imported {{created}}, renamed {{renamed}}, skipped {{skipped}}",
renamed: result.renamed, {
skipped: result.skipped, created: result.created,
}), renamed: result.renamed,
skipped: result.skipped,
},
),
}); });
// Surface partial failures (e.g. unique-name races) as a red warning. // Surface partial failures (e.g. unique-name races) as a red warning.
if (result.errors.length > 0) { if (result.errors.length > 0) {
@@ -0,0 +1,92 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import React from "react";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import type { IAiChatRunResponse } from "@/features/ai-chat/types/ai-chat.types.ts";
// react-i18next is pulled in transitively by ai-chat-query.ts (the mutation hooks
// use it); stub it so the module imports cleanly in this hook test.
vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (key: string) => key }),
}));
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
// Mock the whole service module; only getAiChatRun is exercised here, but the
// other named exports must exist so ai-chat-query.ts imports resolve.
vi.mock("@/features/ai-chat/services/ai-chat-service.ts", () => ({
getAiChatRun: vi.fn(),
getAiChatMessages: vi.fn(),
getAiChats: vi.fn(),
getAiRoleCatalog: vi.fn(),
getAiRoleCatalogBundle: vi.fn(),
getAiRoles: vi.fn(),
importAiRolesFromCatalog: vi.fn(),
createAiRole: vi.fn(),
deleteAiChat: vi.fn(),
deleteAiRole: vi.fn(),
renameAiChat: vi.fn(),
updateAiRole: vi.fn(),
updateAiRoleFromCatalog: vi.fn(),
}));
import { getAiChatRun } from "@/features/ai-chat/services/ai-chat-service.ts";
import { useAiChatRunQuery } from "@/features/ai-chat/queries/ai-chat-query.ts";
function createWrapper() {
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
return function Wrapper({ children }: { children: React.ReactNode }) {
return (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
);
};
}
const runningResponse: IAiChatRunResponse = {
run: { id: "run-1", chatId: "c1", status: "running" },
message: {
id: "a1",
role: "assistant",
content: "working...",
createdAt: "2026-01-01T00:00:00Z",
},
};
describe("useAiChatRunQuery — enable gating", () => {
beforeEach(() => {
vi.clearAllMocks();
});
it("fetches the run when enabled (passive observer, feature on)", async () => {
vi.mocked(getAiChatRun).mockResolvedValue(runningResponse);
const { result } = renderHook(() => useAiChatRunQuery("c1", true), {
wrapper: createWrapper(),
});
await waitFor(() => expect(result.current.isSuccess).toBe(true));
expect(getAiChatRun).toHaveBeenCalledWith("c1");
expect(result.current.data?.run?.status).toBe("running");
});
it("does NOT fetch when disabled (this tab is the streamer / feature off)", async () => {
vi.mocked(getAiChatRun).mockResolvedValue(runningResponse);
renderHook(() => useAiChatRunQuery("c1", false), {
wrapper: createWrapper(),
});
// Give any errant fetch a chance to fire, then assert none did.
await new Promise((r) => setTimeout(r, 20));
expect(getAiChatRun).not.toHaveBeenCalled();
});
it("does NOT fetch when there is no chat id", async () => {
vi.mocked(getAiChatRun).mockResolvedValue(runningResponse);
renderHook(() => useAiChatRunQuery(undefined, true), {
wrapper: createWrapper(),
});
await new Promise((r) => setTimeout(r, 20));
expect(getAiChatRun).not.toHaveBeenCalled();
});
});
@@ -77,7 +77,14 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
}); });
it("errors:[] -> only the summary notification (counts interpolated)", async () => { it("errors:[] -> only the summary notification (counts interpolated)", async () => {
await runMutation({ created: 3, renamed: 1, skipped: 2, errors: [] }); await runMutation({
created: 3,
renamed: 1,
skipped: 2,
errors: [],
createdRoles: [],
skippedRoles: [],
});
expect(notificationsShowMock).toHaveBeenCalledTimes(1); expect(notificationsShowMock).toHaveBeenCalledTimes(1);
expect(notificationsShowMock).toHaveBeenCalledWith({ expect(notificationsShowMock).toHaveBeenCalledWith({
message: "Imported 3, renamed 1, skipped 2", message: "Imported 3, renamed 1, skipped 2",
@@ -93,6 +100,8 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
{ slug: "a", message: "name taken" }, { slug: "a", message: "name taken" },
{ slug: "b", message: "name taken" }, { slug: "b", message: "name taken" },
], ],
createdRoles: [{ slug: "ok", name: "Ok" }],
skippedRoles: [],
}); });
expect(notificationsShowMock).toHaveBeenCalledTimes(2); expect(notificationsShowMock).toHaveBeenCalledTimes(2);
expect(notificationsShowMock).toHaveBeenNthCalledWith(1, { expect(notificationsShowMock).toHaveBeenNthCalledWith(1, {
@@ -5,6 +5,7 @@ import {
IAiChatListParams, IAiChatListParams,
IAiChatMessageRow, IAiChatMessageRow,
IAiChatMessagesParams, IAiChatMessagesParams,
IAiChatRunResponse,
IAiRole, IAiRole,
IAiRoleCatalog, IAiRoleCatalog,
IAiRoleCatalogBundle, IAiRoleCatalogBundle,
@@ -42,6 +43,38 @@ export async function getAiChatMessages(
return req.data; return req.data;
} }
/**
* Reconnect to the latest agent run of a chat (#184). Returns the run's
* persisted lifecycle state and the assistant message it materializes (the
* partial output while the run is in-flight, the final output once it finished).
* The DB is the source of truth, so this works for an in-flight run (the browser
* dropped, the run kept going) and a finished one alike; `{ run: null }` when the
* chat has never had a run. Owner-gated server-side (the requesting user must own
* the chat); it is NOT flag-gated — when the feature is off the chat simply has no
* runs, so the endpoint returns `{ run: null }`.
*/
export async function getAiChatRun(
chatId: string,
): Promise<IAiChatRunResponse> {
const req = await api.post<IAiChatRunResponse>("/ai-chat/run", { chatId });
return req.data;
}
/**
* Explicitly STOP the active agent run of a chat (#184). This is the ONLY thing
* that ends a DETACHED run — a mere browser disconnect (aborting the local SSE)
* is deliberately ignored server-side, so the client must call this to actually
* stop an autonomous run. Targeted by `chatId` (the server resolves whatever run
* is active on it); owner-gated server-side. Returns `{ stopped }` — false when
* there was nothing active to stop.
*/
export async function stopRun(
chatId: string,
): Promise<{ stopped: boolean }> {
const req = await api.post<{ stopped: boolean }>("/ai-chat/stop", { chatId });
return req.data;
}
/** /**
* Resolve the chat bound to a document (the current user's most-recent chat * Resolve the chat bound to a document (the current user's most-recent chat
* created on that page), or null when there is none. Drives auto-open-on-page. * created on that page), or null when there is none. Drives auto-open-on-page.
@@ -108,12 +108,25 @@ export interface IAiRoleImportPayload {
conflict: "skip" | "rename"; conflict: "skip" | "rename";
} }
/** Import result counts (mirrors `importFromCatalog()`). */ /**
* Import result (mirrors `importFromCatalog()`). The counters (`created`,
* `skipped`, `renamed`) drive the summary notification; the per-role lists
* (`createdRoles`, `skippedRoles`) drive the redesigned catalog modal's inline
* result plaque — which roles were installed (and any rename) and which were
* skipped and why (so the plaque can name the conflicting role and offer
* "Rename & install").
*/
export interface IAiRoleImportResult { export interface IAiRoleImportResult {
created: number; created: number;
skipped: number; skipped: number;
renamed: number; renamed: number;
errors: { slug: string; message: string }[]; errors: { slug: string; message: string }[];
createdRoles: { slug: string; name: string; renamedTo?: string }[];
skippedRoles: {
slug: string;
name: string;
reason: "name-conflict" | "already-installed";
}[];
} }
/** /**
@@ -200,6 +213,38 @@ export interface IAiChatMessageRow {
createdAt: string; createdAt: string;
} }
/**
* A persisted agent-run row (#184), mirroring the `ai_chat_runs` fields the
* client reads from `POST /ai-chat/run`. Only `status` is load-bearing for the
* reconnect-and-live-update UX (it drives the poll cadence); the rest are carried
* for display/diagnostics. The DB is the source of truth, so this resolves for an
* in-flight run (the browser dropped, the run kept going) and a finished one.
*/
export interface IAiChatRun {
id: string;
chatId: string;
// 'pending' | 'running' | 'succeeded' | 'failed' | 'aborted'. The first two are
// ACTIVE (keep polling); the rest are TERMINAL (stop polling).
status: "pending" | "running" | "succeeded" | "failed" | "aborted" | string;
error?: string | null;
stepCount?: number;
assistantMessageId?: string | null;
startedAt?: string | null;
finishedAt?: string | null;
createdAt?: string;
updatedAt?: string;
}
/**
* Response of `POST /ai-chat/run` (#184): the latest run of a chat and the
* assistant message it materializes (the partial/final output, projected from the
* persisted rows). Both are `null` when the chat has never had a run.
*/
export interface IAiChatRunResponse {
run: IAiChatRun | null;
message: IAiChatMessageRow | null;
}
export interface IAiChatListParams extends QueryParams {} export interface IAiChatListParams extends QueryParams {}
export interface IAiChatMessagesParams { export interface IAiChatMessagesParams {
@@ -0,0 +1,234 @@
import { describe, it, expect } from "vitest";
import {
bundleCounts,
bundlePhase,
installedLangForRole,
mapBundleRolesToView,
mapCatalogRoleToView,
nameConflictSlugs,
partialOffersRename,
type CatalogViewRole,
} from "./catalog-bundle-model.ts";
import type {
IAiRole,
IAiRoleCatalogRole,
} from "@/features/ai-chat/types/ai-chat.types.ts";
function installedRole(
source: { slug: string; language: string; version: number },
overrides: Partial<IAiRole> = {},
): IAiRole {
return {
id: `role-${source.slug}-${source.language}`,
name: source.slug,
emoji: null,
description: null,
enabled: true,
autoStart: true,
launchMessage: null,
source,
...overrides,
};
}
function catalogRole(
overrides: Partial<IAiRoleCatalogRole> = {},
): IAiRoleCatalogRole {
return {
slug: "writer",
emoji: "✍️",
name: "Writer",
description: "Drafts copy.",
instructions: "be a writer",
autoStart: true,
launchMessage: null,
version: 3,
...overrides,
};
}
// Build a minimal view role for bundlePhase tests.
function viewRole(status: CatalogViewRole["status"]): CatalogViewRole {
return { slug: `s-${status}`, name: status, description: "", version: 1, status };
}
describe("bundlePhase", () => {
it("empty bundle -> empty", () => {
expect(bundlePhase([])).toBe("empty");
});
it("all importable, none installed -> allNew", () => {
expect(bundlePhase([viewRole("import"), viewRole("import")])).toBe(
"allNew",
);
});
it("nothing to import or update -> allInstalled", () => {
expect(bundlePhase([viewRole("installed"), viewRole("installed")])).toBe(
"allInstalled",
);
});
it("updates present, nothing to import -> updates", () => {
expect(bundlePhase([viewRole("update"), viewRole("installed")])).toBe(
"updates",
);
});
it("import + installed (no updates) -> mixed", () => {
expect(bundlePhase([viewRole("import"), viewRole("installed")])).toBe(
"mixed",
);
});
it("import + update -> mixed", () => {
expect(bundlePhase([viewRole("import"), viewRole("update")])).toBe("mixed");
});
it("a skipped role with nothing installed -> mixed (NOT allInstalled)", () => {
// F1: a bundle whose only non-installed role was skipped has 0 installed for
// it, so the collapsed 'All installed · up to date' header would contradict
// the open 'Installed 0 · 1 skipped' plaque. It must be mixed until resolved.
expect(bundlePhase([viewRole("skipped")])).toBe("mixed");
});
it("installed + a skipped role -> mixed (partial success is not allInstalled)", () => {
expect(bundlePhase([viewRole("installed"), viewRole("skipped")])).toBe(
"mixed",
);
});
});
describe("bundleCounts", () => {
it("tallies each status once", () => {
expect(
bundleCounts([
viewRole("import"),
viewRole("import"),
viewRole("installed"),
viewRole("update"),
viewRole("skipped"),
]),
).toEqual({ importable: 2, installed: 1, update: 1, skipped: 1 });
});
});
describe("nameConflictSlugs / partialOffersRename (reason -> action)", () => {
it("only name-conflict skips become the transient overlay / offer rename", () => {
const skipped = [
{ slug: "writer", name: "Writer", reason: "name-conflict" as const },
{ slug: "editor", name: "Editor", reason: "already-installed" as const },
];
expect(nameConflictSlugs(skipped)).toEqual(["writer"]);
expect(partialOffersRename(skipped)).toBe(true);
});
it("an already-installed-only skip is informational: no overlay, no rename", () => {
const skipped = [
{ slug: "editor", name: "Editor", reason: "already-installed" as const },
];
expect(nameConflictSlugs(skipped)).toEqual([]);
expect(partialOffersRename(skipped)).toBe(false);
});
});
describe("installedLangForRole", () => {
it("returns the other language when the same slug is installed elsewhere", () => {
const roles = [installedRole({ slug: "writer", language: "ru", version: 2 })];
expect(installedLangForRole("writer", roles, "en")).toBe("ru");
});
it("returns undefined when the same slug is installed in the SAME language", () => {
const roles = [installedRole({ slug: "writer", language: "en", version: 2 })];
expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
});
it("returns undefined when no install of the slug exists", () => {
expect(installedLangForRole("writer", [], "en")).toBeUndefined();
});
it("ignores manually-created roles (no source)", () => {
const roles = [
installedRole({ slug: "writer", language: "ru", version: 2 }, {
source: null,
}),
];
expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
});
});
describe("mapCatalogRoleToView", () => {
it("no install -> import status, catalog version, emoji preserved", () => {
const view = mapCatalogRoleToView(catalogRole(), [], "en");
expect(view).toMatchObject({
slug: "writer",
emoji: "✍️",
name: "Writer",
description: "Drafts copy.",
status: "import",
version: 3,
});
expect(view.installedRoleId).toBeUndefined();
expect(view.installedLang).toBeUndefined();
});
it("import with the slug installed in another language -> installedLang set", () => {
const roles = [installedRole({ slug: "writer", language: "ru", version: 9 })];
const view = mapCatalogRoleToView(catalogRole(), roles, "en");
expect(view.status).toBe("import");
expect(view.installedLang).toBe("ru");
});
it("installed (up to date) -> installed status, catalog version, installedRoleId", () => {
const installed = installedRole({
slug: "writer",
language: "en",
version: 3,
});
const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
expect(view).toMatchObject({
status: "installed",
version: 3,
installedRoleId: installed.id,
});
});
it("update -> version=from, newVersion=to, installedRoleId", () => {
const installed = installedRole({
slug: "writer",
language: "en",
version: 1,
});
const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
expect(view).toMatchObject({
status: "update",
version: 1,
newVersion: 3,
installedRoleId: installed.id,
});
});
it("missing emoji -> emoji undefined; null description -> empty string", () => {
const view = mapCatalogRoleToView(
catalogRole({ emoji: null, description: null }),
[],
"en",
);
expect(view.emoji).toBeUndefined();
expect(view.description).toBe("");
});
});
describe("mapBundleRolesToView", () => {
it("maps a bundle's roles preserving order", () => {
const roles = [
catalogRole({ slug: "a", name: "A", version: 1 }),
catalogRole({ slug: "b", name: "B", version: 1 }),
];
const installed = [installedRole({ slug: "a", language: "en", version: 1 })];
const view = mapBundleRolesToView(roles, installed, "en");
expect(view.map((r) => r.slug)).toEqual(["a", "b"]);
expect(view[0].status).toBe("installed");
expect(view[1].status).toBe("import");
});
});
@@ -0,0 +1,206 @@
import type {
IAiRole,
IAiRoleCatalogRole,
} from "@/features/ai-chat/types/ai-chat.types.ts";
import { catalogRoleInstallState } from "@/features/ai-chat/utils/catalog-role-install-state.ts";
/**
* The redesigned catalog modal renders bundles as cards with a summary status
* (readable without expanding) and a single primary action. The per-role and
* per-bundle view model that drives that UI is derived here as PURE functions so
* the mapping, the "installed in another language" hint, and the bundle-phase
* computation are unit-testable without mounting the component (mirrors the
* `catalogRoleInstallState` precedent).
*/
/**
* A role's status in the catalog view model.
* - `import` — not installed in the current content language.
* - `installed` — installed and up to date.
* - `update` — installed, but the catalog ships a newer version.
* - `skipped` — TRANSIENT client-only status set after a conflicted import
* (a name collision under `conflict:'skip'`); never from the
* backend.
*/
export type RoleStatus = "import" | "installed" | "update" | "skipped";
/** A catalog role mapped into the modal's view model. */
export interface CatalogViewRole {
// Slug is the stable identity within a bundle; used as the row key and as the
// `slugs[]` payload for import.
slug: string;
// Optional in the catalog — the row reserves space and renders nothing when
// absent.
emoji?: string;
name: string;
description: string;
// For `installed`/`import`: the catalog version. For `update`: the installed
// (from) version, with `newVersion` holding the catalog (to) version.
version: number;
newVersion?: number;
status: RoleStatus;
// The language a same-slug role is installed under, when it differs from the
// current content language (drives the Р5 hint). Only set for `import` roles.
installedLang?: string;
// The workspace role id, present for `installed`/`update` — needed to call the
// update-from-catalog mutation.
installedRoleId?: string;
}
/**
* The summary phase of a bundle, derived from its roles' statuses. Determines
* the collapsed-header summary and the bundle's single primary action.
* - `empty` — the bundle has no roles.
* - `allNew` — everything is importable, nothing installed.
* - `allInstalled` — everything installed & up to date; nothing else pending.
* - `updates` — updates available and nothing left to import.
* - `mixed` — any other combination.
*/
export type BundlePhase =
| "empty"
| "allNew"
| "allInstalled"
| "updates"
| "mixed";
/** Per-status tallies for a bundle's roles (the single source of truth). */
export interface BundleCounts {
importable: number;
installed: number;
update: number;
skipped: number;
}
/**
* Count a bundle's roles by status ONCE. Both `bundlePhase` and the panel derive
* from this, so the tally logic lives in exactly one place (no rescans / drift).
*/
export function bundleCounts(roles: CatalogViewRole[]): BundleCounts {
const counts: BundleCounts = {
importable: 0,
installed: 0,
update: 0,
skipped: 0,
};
for (const r of roles) {
if (r.status === "import") counts.importable += 1;
else if (r.status === "installed") counts.installed += 1;
else if (r.status === "update") counts.update += 1;
else if (r.status === "skipped") counts.skipped += 1;
}
return counts;
}
export function bundlePhase(roles: CatalogViewRole[]): BundlePhase {
if (roles.length === 0) return "empty";
const { importable, installed, update, skipped } = bundleCounts(roles);
// A `skipped` role is a pending post-import conflict (0 installed for it), so a
// bundle that has ANY skipped role is NOT "all installed & up to date" — that
// would make the collapsed green "up to date" header contradict the open
// panel's "Installed 0 · 1 skipped" plaque. It is `mixed` until resolved.
if (importable === 0 && update === 0 && skipped === 0) return "allInstalled";
if (update > 0 && importable === 0 && skipped === 0) return "updates";
if (importable > 0 && installed === 0 && update === 0 && skipped === 0)
return "allNew";
return "mixed";
}
/**
* The subset of a skip result that should be shown as a TRANSIENT `skipped`
* overlay in the bundle (so the row offers a re-import path). Only NAME-CONFLICT
* skips qualify: an `already-installed` skip (a concurrent-import race) has
* nothing to act on — re-importing the same slug would just skip again — so it
* must NOT be overlaid (else the row shows a misleading "Rename & install" that
* self-heals into a false "installed"). Pure so both reason branches are tested.
*/
export function nameConflictSlugs(
skipped: { slug: string; reason: "name-conflict" | "already-installed" }[],
): string[] {
return skipped
.filter((s) => s.reason === "name-conflict")
.map((s) => s.slug);
}
/**
* Whether a partial-import result should offer the "Rename & install" action:
* only when at least one skip is a name conflict (renameable). An
* `already-installed`-only partial is informational.
*/
export function partialOffersRename(
skipped: { reason: "name-conflict" | "already-installed" }[],
): boolean {
return skipped.some((s) => s.reason === "name-conflict");
}
/**
* For a role NOT installed in the current `language`, find a workspace role with
* the same catalog `slug` installed under a DIFFERENT language, and return that
* language. Drives the "installed in another language" hint (Р5): a different
* language of the same slug is a separate install and appears as `import`.
*/
export function installedLangForRole(
slug: string,
workspaceRoles: IAiRole[],
language: string,
): string | undefined {
const other = workspaceRoles.find(
(r) =>
r.source?.slug === slug &&
!!r.source?.language &&
r.source.language !== language,
);
return other?.source?.language;
}
/**
* Map one catalog role to the view model, computing its install status against
* the workspace roles (via `catalogRoleInstallState`) and, for importable roles,
* the other-language hint.
*/
export function mapCatalogRoleToView(
role: IAiRoleCatalogRole,
workspaceRoles: IAiRole[],
language: string,
): CatalogViewRole {
const state = catalogRoleInstallState(role, workspaceRoles, language);
const base = {
slug: role.slug,
emoji: role.emoji ?? undefined,
name: role.name,
description: role.description ?? "",
};
if (state.state === "update") {
return {
...base,
status: "update",
version: state.fromVersion,
newVersion: state.toVersion,
installedRoleId: state.installed.id,
};
}
if (state.state === "installed") {
return {
...base,
status: "installed",
version: role.version,
installedRoleId: state.installed.id,
};
}
return {
...base,
status: "import",
version: role.version,
installedLang: installedLangForRole(role.slug, workspaceRoles, language),
};
}
/**
* Map a whole bundle's catalog roles to the view model, preserving order.
*/
export function mapBundleRolesToView(
roles: IAiRoleCatalogRole[],
workspaceRoles: IAiRole[],
language: string,
): CatalogViewRole[] {
return roles.map((r) => mapCatalogRoleToView(r, workspaceRoles, language));
}
@@ -0,0 +1,303 @@
import { describe, it, expect } from "vitest";
import type { UIMessage } from "@ai-sdk/react";
import type { IAiChatRun } from "@/features/ai-chat/types/ai-chat.types.ts";
import {
RUN_POLL_INTERVAL_MS,
isRunActive,
runPollInterval,
shouldObserveRun,
shouldClearStoppingLatch,
shouldClearLatchOnQueryError,
mergeObservedMessage,
} from "./run-polling.ts";
function makeRun(status: string): IAiChatRun {
return { id: "run-1", chatId: "c1", status };
}
function makeMsg(id: string, text: string): UIMessage {
return {
id,
role: "assistant",
parts: [{ type: "text", text }],
} as UIMessage;
}
describe("isRunActive", () => {
it("treats pending and running as active", () => {
expect(isRunActive(makeRun("pending"))).toBe(true);
expect(isRunActive(makeRun("running"))).toBe(true);
});
it("treats terminal / unknown / nullish as not active", () => {
expect(isRunActive(makeRun("succeeded"))).toBe(false);
expect(isRunActive(makeRun("failed"))).toBe(false);
expect(isRunActive(makeRun("aborted"))).toBe(false);
expect(isRunActive(makeRun("weird-future-status"))).toBe(false);
expect(isRunActive(null)).toBe(false);
expect(isRunActive(undefined)).toBe(false);
});
});
describe("runPollInterval (the refetchInterval helper)", () => {
it("returns 2000ms while the run is pending/running", () => {
expect(runPollInterval(makeRun("pending"))).toBe(RUN_POLL_INTERVAL_MS);
expect(runPollInterval(makeRun("running"))).toBe(RUN_POLL_INTERVAL_MS);
expect(RUN_POLL_INTERVAL_MS).toBe(2000);
});
it("returns false (stop polling) once the run is terminal", () => {
expect(runPollInterval(makeRun("succeeded"))).toBe(false);
expect(runPollInterval(makeRun("failed"))).toBe(false);
expect(runPollInterval(makeRun("aborted"))).toBe(false);
});
it("returns false (no polling) when there is no run", () => {
expect(runPollInterval(null)).toBe(false);
expect(runPollInterval(undefined)).toBe(false);
});
});
describe("shouldObserveRun (observer-vs-streamer decision)", () => {
it("observes an active run when this tab is NOT the local streamer", () => {
expect(shouldObserveRun(makeRun("running"), false)).toBe(true);
expect(shouldObserveRun(makeRun("pending"), false)).toBe(true);
});
it("observes a terminal run too (so the final output shows on reopen)", () => {
expect(shouldObserveRun(makeRun("succeeded"), false)).toBe(true);
});
it("does NOT observe when this tab IS the streamer (no double-render)", () => {
expect(shouldObserveRun(makeRun("running"), true)).toBe(false);
expect(shouldObserveRun(makeRun("succeeded"), true)).toBe(false);
});
it("does NOT observe when there is no run", () => {
expect(shouldObserveRun(null, false)).toBe(false);
expect(shouldObserveRun(undefined, false)).toBe(false);
});
});
describe("shouldClearStoppingLatch (#234 latch-release decision)", () => {
// The one case the latch SHOULD clear: we requested a stop, we are the passive
// observer (not streaming), and the CURRENT run is terminal.
it("clears only when stopping, observing, and the run is terminal", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("aborted"),
isLocalStreaming: false,
}),
).toBe(true);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("succeeded"),
isLocalStreaming: false,
}),
).toBe(true);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("failed"),
isLocalStreaming: false,
}),
).toBe(true);
});
// Round-3 regression: clearing while THIS tab is still the local streamer would
// re-open the flash for the current turn the moment we switch to observer role.
// A predicate lacking the streaming gate would (wrongly) return true here.
it("does NOT clear while this tab is the local streamer", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("aborted"),
isLocalStreaming: true,
}),
).toBe(false);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("succeeded"),
isLocalStreaming: true,
}),
).toBe(false);
});
// The detached run keeps growing after a local abort — while it is still
// active the latch MUST hold so the observer merge stays suppressed.
it("does NOT clear while the run is still active", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("running"),
isLocalStreaming: false,
}),
).toBe(false);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: makeRun("pending"),
isLocalStreaming: false,
}),
).toBe(false);
});
// #234 F4: on Stop the stale PREVIOUS-turn run is removed from the cache, so the
// observed `run` is null until the current turn's run is fetched fresh. A null
// run HOLDS the latch — it can never clear against the just-removed stale run,
// only against the current turn's own terminal run once observed.
it("does NOT clear against a removed/absent run (F4 stale-run guard)", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: null,
isLocalStreaming: false,
}),
).toBe(false);
expect(
shouldClearStoppingLatch({
stoppingRun: true,
run: undefined,
isLocalStreaming: false,
}),
).toBe(false);
});
it("does NOT clear when no stop was requested", () => {
expect(
shouldClearStoppingLatch({
stoppingRun: false,
run: makeRun("aborted"),
isLocalStreaming: false,
}),
).toBe(false);
});
});
describe("shouldClearLatchOnQueryError (#234 F7 error-safety-net decision)", () => {
// This guards the REAL anti-flash decision the component's run-query-error
// safety-net effect uses (ai-chat-window.tsx wires the effect to THIS helper,
// not a copy — so the test is non-vacuous vs the live code).
// (b) The F7 hole: a TRANSIENT run-query error while `run` is STILL ACTIVE must
// NOT clear the latch. TanStack Query v5 retains `data` on error, so
// runQueryFailed can be true while the held run is still pending/running.
// Against the PRE-F7 condition (without `!isRunActive(run)`) this would return
// true — so this assertion fails on the buggy code (non-vacuous).
it("does NOT clear on a transient error while the run is still ACTIVE (F7)", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: makeRun("running"),
}),
).toBe(false);
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: makeRun("pending"),
}),
).toBe(false);
});
// (a) The genuine permanent-null-freeze: run cache cleared by removeQueries +
// the refetch keeps ERRORING, so `run === null`. This is the ONLY case the
// safety-net exists to cure — it MUST clear so the frozen view resumes.
it("clears on a permanent error when the run is null (permanent-null-freeze)", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: null,
}),
).toBe(true);
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: undefined,
}),
).toBe(true);
});
// A TERMINAL run also satisfies `!isRunActive`; clearing then is harmless — the
// terminal effect (shouldClearStoppingLatch) already clears for a terminal run,
// so this only ever agrees with it. Asserted so the (c) reasoning is pinned.
it("clears on an error when the run is terminal (harmless, agrees with terminal effect)", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: true,
run: makeRun("aborted"),
}),
).toBe(true);
});
it("does NOT clear without an actual query error", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: false,
runQueryFailed: false,
run: null,
}),
).toBe(false);
});
it("does NOT clear while this tab is the local streamer", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: true,
isLocalStreaming: true,
runQueryFailed: true,
run: null,
}),
).toBe(false);
});
it("does NOT clear when no stop was requested", () => {
expect(
shouldClearLatchOnQueryError({
stoppingRun: false,
isLocalStreaming: false,
runQueryFailed: true,
run: null,
}),
).toBe(false);
});
});
describe("mergeObservedMessage", () => {
it("replaces the message with the same id in place (per-step growth)", () => {
const prev = [makeMsg("u1", "hi"), makeMsg("a1", "step 1")];
const observed = makeMsg("a1", "step 1\nstep 2");
const next = mergeObservedMessage(prev, observed);
expect(next).toHaveLength(2);
expect(next[1]).toBe(observed);
expect(next[0]).toBe(prev[0]); // untouched
expect(next).not.toBe(prev); // new array (never mutates input)
});
it("appends when the observed message is not yet present", () => {
const prev = [makeMsg("u1", "hi")];
const observed = makeMsg("a1", "first token");
const next = mergeObservedMessage(prev, observed);
expect(next).toHaveLength(2);
expect(next[1]).toBe(observed);
});
it("returns the original list unchanged when there is nothing to merge", () => {
const prev = [makeMsg("u1", "hi")];
expect(mergeObservedMessage(prev, null)).toBe(prev);
expect(mergeObservedMessage(prev, undefined)).toBe(prev);
});
});
@@ -0,0 +1,151 @@
import type { UIMessage } from "@ai-sdk/react";
import type { IAiChatRun } from "@/features/ai-chat/types/ai-chat.types.ts";
/**
* Reconnect-and-live-follow helpers (#184). When a chat is reopened while its
* agent run is STILL going, this tab is a PASSIVE OBSERVER: it did not start the
* run here (no local SSE stream), so it catches up by POLLING the reconnect
* endpoint (`POST /ai-chat/run`) and merging the run's incrementally-persisted
* assistant message into the rendered thread. These are the small pure decisions
* that machinery hangs off, extracted so they can be unit-tested in isolation
* (mirrors how reindex polling / editor-sync-state are tested).
*/
/** How often to re-poll the reconnect endpoint while a run is ACTIVE. */
export const RUN_POLL_INTERVAL_MS = 2000;
// 'pending' and 'running' are the two ACTIVE statuses; 'succeeded' | 'failed' |
// 'aborted' are TERMINAL (and any unknown future status is treated as terminal,
// so a stale/odd value never polls forever).
const ACTIVE_STATUSES = new Set(["pending", "running"]);
/** Whether a run is still going (worth polling / merging live updates from). */
export function isRunActive(run: IAiChatRun | null | undefined): boolean {
return !!run && ACTIVE_STATUSES.has(run.status);
}
/**
* The TanStack Query `refetchInterval` value for the run query: poll every
* {@link RUN_POLL_INTERVAL_MS} while the run is active, and `false` (stop) once
* it is terminal or there is no run. Polling is thus naturally bounded by the run
* reaching a terminal status — no separate timeout cap is needed.
*/
export function runPollInterval(
run: IAiChatRun | null | undefined,
): number | false {
return isRunActive(run) ? RUN_POLL_INTERVAL_MS : false;
}
/**
* Observer-vs-streamer decision. We render the polled run message (catch up +
* keep advancing) ONLY when this tab is a passive observer: there IS a run AND
* this tab is NOT the one locally streaming it (we reconnected, we didn't start
* it here). When this tab is the streamer, the live SSE stream owns the view, so
* we neither poll nor merge — avoiding a double-render fight. Terminal runs still
* merge (so the final persisted output is shown on reopen); the poll itself is
* stopped separately by {@link runPollInterval}.
*/
export function shouldObserveRun(
run: IAiChatRun | null | undefined,
localStreaming: boolean,
): boolean {
return !!run && !localStreaming;
}
/**
* Should the "stopping" latch — which suppresses the observer re-stream flash
* after the user pressed Stop — be RELEASED now? All three must hold:
* - `stoppingRun`: we actually requested a stop (otherwise nothing to release);
* - `!isLocalStreaming`: this tab is NOT the local streamer. While we are the
* streamer the run query is disabled, so the observed `run` is not the run we
* are following — releasing the latch then would re-open the flash for the
* current turn the instant we switch to observer role;
* - the observed `run` EXISTS and has reached a TERMINAL status.
*
* The null / still-active `run` case is the #234 F4 invariant. On Stop the stale
* PREVIOUS-turn run is removed from the query cache (`removeQueries`), so `run`
* is null until the CURRENT turn's run is re-fetched fresh; a null or active run
* therefore HOLDS the latch, so it can only ever clear against the current turn's
* OWN terminal run — never a stale cached one. (The cache removal itself is
* integration-level in AiChatWindow; this predicate encodes the decision given
* whatever run is currently observed, and a stale terminal run is
* indistinguishable from a current terminal run at the predicate level — hence
* the cache removal is what guarantees only the current run is ever passed here.)
*/
export function shouldClearStoppingLatch(args: {
stoppingRun: boolean;
run: IAiChatRun | null | undefined;
isLocalStreaming: boolean;
}): boolean {
const { stoppingRun, run, isLocalStreaming } = args;
if (!stoppingRun || isLocalStreaming) return false;
return !!run && !isRunActive(run);
}
/**
* Should the "stopping" latch be RELEASED by the run-query ERROR safety-net?
* (#234 F7 — a NEW path of the same re-stream flash the F4 latch exists to
* prevent.) After Stop, `handleServerStop` clears the run cache; the terminal
* effect then holds the latch via `if (!run) return` until the CURRENT turn's run
* is fetched fresh. If that refetch instead ERRORS permanently, `run` stays null,
* its status-keyed refetchInterval is off, and nothing would ever observe a
* terminal run — freezing the view with the observer merge suppressed. This
* safety-net cures ONLY that genuine permanent-null-freeze.
*
* All four must hold:
* - `stoppingRun`: we actually requested a stop (otherwise nothing to release);
* - `!isLocalStreaming`: this tab is NOT the local streamer (same reason as
* {@link shouldClearStoppingLatch});
* - `runQueryFailed`: the run query is in its error state (TanStack Query v5 with
* retry:false — isError);
* - `!isRunActive(run)`: the observed `run` is NOT an active (pending/running)
* held run. This is the F7 gate. In TanStack Query v5 the query's `data` is
* RETAINED on error, so `runQueryFailed` can be true while `run` is STILL an
* ACTIVE run (a single transient GET-run failure in the window between Stop and
* settle). Without this gate a transient error would release the latch early —
* re-opening the observer merge and flashing the growing detached run over the
* frozen row (exactly the F4 flash). Gating on the run NOT being active means we
* only ever cure the permanent-null-freeze (`run === null`, so
* `isRunActive(null)` is false), never release against an active run.
*
* (A terminal `run` also satisfies `!isRunActive(run)`; clearing then is harmless
* — the terminal effect's {@link shouldClearStoppingLatch} already clears the
* latch for a terminal run, so this only ever agrees with it, never conflicts.)
*
* INVARIANT (do not break): clearing the latch on the `run === null` branch is safe
* ONLY because the run query's `refetchInterval` (see {@link runPollInterval}) stops
* polling when the data is empty — so after we clear on null+error there is no
* subsequent auto-poll that could return a still-active detached run and re-open the
* merge. If `refetchInterval` is ever changed to keep polling on `run === null`/on
* error, this null-branch clear would re-open the F7 flash through the null path.
* Do not change the run query's refetchInterval without re-checking this path.
*/
export function shouldClearLatchOnQueryError(args: {
stoppingRun: boolean;
isLocalStreaming: boolean;
runQueryFailed: boolean;
run: IAiChatRun | null | undefined;
}): boolean {
const { stoppingRun, isLocalStreaming, runQueryFailed, run } = args;
return (
stoppingRun && !isLocalStreaming && runQueryFailed && !isRunActive(run)
);
}
/**
* Merge an observed assistant message into the rendered list: replace the message
* with the same id in place (the in-progress assistant row is already seeded from
* history, so per-step growth replaces it), or append it when absent. Returns a
* new array; the input is never mutated.
*/
export function mergeObservedMessage(
messages: UIMessage[],
observed: UIMessage | null | undefined,
): UIMessage[] {
if (!observed) return messages;
const idx = messages.findIndex((m) => m.id === observed.id);
if (idx === -1) return [...messages, observed];
const next = messages.slice();
next[idx] = observed;
return next;
}
@@ -24,6 +24,7 @@ import {
GitmostListPagesResult, GitmostListPagesResult,
GitmostListSpacesResult, GitmostListSpacesResult,
gitmostDecodePayloadToFile, gitmostDecodePayloadToFile,
gitmostInsertTranscriptIntoEditor,
gitmostUploadFileToEditor, gitmostUploadFileToEditor,
} from "@/features/editor/gitmost/gitmost-recording.ts"; } from "@/features/editor/gitmost/gitmost-recording.ts";
@@ -281,6 +282,18 @@ export default function GitmostGlobalBridge() {
pageId: page.id, pageId: page.id,
}; };
} }
// Best-effort: append the transcript (heading + one paragraph per line)
// below the just-inserted audio node. The audio insert already
// succeeded, so a transcript failure must NOT turn this into an error —
// wrap it and, on any throw, log and still return ok. A missing/empty/
// non-string transcript is a no-op inside the helper (audio only).
try {
gitmostInsertTranscriptIntoEditor(editor, payload?.transcript);
} catch (err) {
console.error("[gitmost] transcript insert failed", err);
}
return { ok: true, pageId: page.id }; return { ok: true, pageId: page.id };
} catch (err: any) { } catch (err: any) {
console.error("[gitmost] createPageWithRecording failed", err); console.error("[gitmost] createPageWithRecording failed", err);
@@ -0,0 +1,150 @@
import { describe, it, expect } from "vitest";
import { Editor } from "@tiptap/core";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Heading } from "@tiptap/extension-heading";
import { Bold } from "@tiptap/extension-bold";
import { Italic } from "@tiptap/extension-italic";
import { Link } from "@tiptap/extension-link";
import { gitmostInsertTranscriptIntoEditor } from "./gitmost-recording.ts";
const ZWSP = "​"; // U+200B, the helper's block-trigger neutralizer
/**
* #377 — the web-side bridge must append the native host's transcript below the
* recording. These exercise the pure insert helper through a REAL Tiptap editor
* (Document/Paragraph/Text/Heading + Bold/Italic/Link marks so an HTML-parsing
* regression would be caught), asserting the resulting document rather than
* mocking the editor: transcript present -> "Transcript" heading + one paragraph
* per non-empty line; content is inserted as LITERAL TEXT (no HTML/markdown
* parsing); col-0 markdown block triggers are neutralized so git-sync keeps them
* paragraphs; absent/empty/non-string -> no-op.
*/
describe("gitmostInsertTranscriptIntoEditor", () => {
const makeEditor = () =>
new Editor({
// Bold/Italic/Link are registered specifically so that IF the helper ever
// regressed to inserting an HTML/markdown string (instead of a text node),
// TipTap would parse `<b>`/`*..*`/`[..](..)` into marks and the literal-
// text assertions below would fail.
extensions: [Document, Paragraph, Text, Heading, Bold, Italic, Link],
// Start from a single empty paragraph (a fresh page's baseline). The
// helper appends at the end of the doc, i.e. below existing content.
content: { type: "doc", content: [{ type: "paragraph" }] },
});
it("inserts a Transcript heading + one paragraph per non-empty line, verbatim", () => {
const editor = makeEditor();
const inserted = gitmostInsertTranscriptIntoEditor(
editor,
"You: hello there\nSpeaker 1: hi\n\nYou: bye",
);
expect(inserted).toBe(true);
const nodes = (editor.getJSON().content ?? []) as any[];
// A level-2 "Transcript" heading is present.
const heading = nodes.find((n) => n.type === "heading");
expect(heading?.attrs?.level).toBe(2);
expect(heading?.content?.[0]?.text).toBe("Transcript");
// Every non-empty transcript line becomes a paragraph, in order, verbatim;
// the blank line between them is dropped.
const texts = nodes
.filter((n) => n.type === "paragraph")
.map((n) => n.content?.[0]?.text)
.filter((t) => typeof t === "string");
expect(texts).toEqual(["You: hello there", "Speaker 1: hi", "You: bye"]);
editor.destroy();
});
it("inserts HTML + markdown metacharacters as LITERAL text (no injection / no mark parsing)", () => {
const editor = makeEditor();
const line =
"You: <b>bold</b> <script>alert(1)</script> and *stars* and [link](x)";
const inserted = gitmostInsertTranscriptIntoEditor(editor, line);
expect(inserted).toBe(true);
const paras = (editor.getJSON().content ?? []).filter(
(n: any) => n.type === "paragraph",
) as any[];
// The transcript line is exactly ONE paragraph holding a SINGLE text node
// whose text is the verbatim string — not split into bold/link/other nodes,
// not carrying any marks, not raw HTML. This FAILS if the helper switched to
// insertContent(htmlString): TipTap would then parse <b>/[link](x)/*stars*.
const content = paras[paras.length - 1].content;
expect(content).toHaveLength(1);
expect(content[0].type).toBe("text");
expect(content[0].marks ?? []).toEqual([]);
expect(content[0].text).toBe(line);
// And no bold/italic/link mark exists anywhere in the document.
const html = editor.getHTML();
expect(html).not.toMatch(/<(strong|b|em|i|a)\b/);
// The angle brackets survived as escaped entities (literal text), not a live
// <script>/<b> element.
expect(html).not.toMatch(/<script/i);
editor.destroy();
});
it("neutralizes col-0 markdown block triggers with a leading ZWSP (git-sync safety)", () => {
const editor = makeEditor();
// Trigger lines (some with a leaked indent) + a normal prefixed line.
const inserted = gitmostInsertTranscriptIntoEditor(
editor,
[
"- dash",
" > quote", // leading indent must be trimmed then neutralized
"# hash",
"1. one",
"> [!info] note",
"```js",
"---", // solid thematic break -> horizontalRule (text-losing) if unneutralized
"***",
"___",
"You: normal line",
].join("\n"),
);
expect(inserted).toBe(true);
const texts = (editor.getJSON().content ?? [])
.filter((n: any) => n.type === "paragraph")
.map((n: any) => n.content?.[0]?.text)
.filter((t: any) => typeof t === "string") as string[];
// Every block-trigger line is prefixed with the invisible ZWSP (indent
// trimmed first); the normal `You:` line is left byte-exact.
expect(texts).toEqual([
ZWSP + "- dash",
ZWSP + "> quote",
ZWSP + "# hash",
ZWSP + "1. one",
ZWSP + "> [!info] note",
ZWSP + "```js",
ZWSP + "---",
ZWSP + "***",
ZWSP + "___",
"You: normal line",
]);
editor.destroy();
});
it("is a no-op for undefined / empty / whitespace-only / non-string transcripts", () => {
for (const value of [undefined, "", " \n \n", 42, {}, null]) {
const editor = makeEditor();
const before = JSON.stringify(editor.getJSON());
const inserted = gitmostInsertTranscriptIntoEditor(editor, value as any);
expect(inserted).toBe(false);
// Document is untouched (audio-only behavior preserved).
expect(JSON.stringify(editor.getJSON())).toBe(before);
editor.destroy();
}
});
});
@@ -65,6 +65,11 @@ export interface GitmostCreatePagePayload {
base64: string; base64: string;
filename: string; filename: string;
mimeType: string; mimeType: string;
// Optional transcript for the recording: plain text, `\n`-separated, each
// line already formatted as `You: ...` / `Speaker N: ...` by the native host
// (ready to insert, no parsing needed). Omitted (no speech / no models) ->
// audio only.
transcript?: string;
} }
export interface GitmostCreatePageResult { export interface GitmostCreatePageResult {
@@ -235,6 +240,83 @@ export async function gitmostUploadFileToEditor(
} }
} }
// Zero-width space (U+200B). Prepended to a transcript line that begins with a
// markdown BLOCK trigger: it is invisible in the rendered doc but shifts the
// trigger off column 0, so the git-sync doc->markdown->doc round-trip keeps the
// line a plain paragraph (see GITMOST_MD_BLOCK_TRIGGER_RE).
const GITMOST_ZWSP = "​";
// A markdown BLOCK-level construct that, sitting at column 0 of a paragraph
// line, the git-sync markdown serializer (packages/prosemirror-markdown
// markdown-converter.ts, `case "paragraph"`) would re-parse into a NON-paragraph
// block on the doc->markdown->doc cycle. That serializer emits paragraph text
// verbatim with NO block-escape (the pre-existing root cause), so a leading
// `#`/`-`/`*`/`+`/`>`, an ordered-list `N.`/`N)`, a code fence ```/~~~, a table
// `|`, or a `> [!info]` callout opener would silently become a heading / list /
// quote / code block / table / callout. The final alternative matches a WHOLE-
// LINE thematic break — solid `---`/`***`/`___` or spaced `- - -`/`_ _ _` (3+ of
// the same `-`/`*`/`_`) — which round-trips into a `horizontalRule`; because
// that node carries NO text, an un-neutralized separator line would LOSE its
// text entirely (worse than the list/quote case). This matches a TRIMMED line's
// start; the transcript's own `You:` / `Speaker N:` prefix begins with a letter
// and never matches, so prefixed lines are left byte-exact.
const GITMOST_MD_BLOCK_TRIGGER_RE =
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
// Append a transcript block BELOW the recording's audio node in a live editor:
// a "Transcript" heading followed by one paragraph per non-empty transcript
// line. The transcript is plain text, `\n`-separated, each line already
// formatted as `You: ...` / `Speaker N: ...` by the native host — line text is
// inserted as a TEXT node (never HTML/markdown), so there is no injection or
// mark-parsing surface. Each kept line is trimmed (drops an indent that would
// both leak into the display and, at col 0, form a markdown block trigger) and,
// if it still begins with a col-0 markdown block trigger, gets an invisible
// zero-width space prepended so the git-sync round-trip cannot turn it into a
// list/quote/heading/callout/code/table (defensive boundary against the
// serializer's missing block-escape). This is best-effort and meant to run
// AFTER the audio has already been inserted; the caller must guard against a
// throw so a transcript failure never fails the (already successful) recording.
// Returns true when a block was inserted, false when there was nothing to
// insert (transcript undefined/empty/not-a-string). A non-string value is a
// no-op, not an error.
export function gitmostInsertTranscriptIntoEditor(
editor: Editor,
transcript: unknown,
): boolean {
if (typeof transcript !== "string") return false;
const lines = transcript
.split("\n")
// Trim each line and drop blank (whitespace-only) ones.
.map((line) => line.trim())
.filter((line) => line.length > 0)
// Neutralize a col-0 markdown block trigger with an invisible ZWSP so the
// git-sync round-trip keeps the line a paragraph. Host lines (`You:` /
// `Speaker N:`) never match and stay byte-exact.
.map((line) =>
GITMOST_MD_BLOCK_TRIGGER_RE.test(line) ? GITMOST_ZWSP + line : line,
);
if (lines.length === 0) return false;
const content = [
{
type: "heading",
attrs: { level: 2 },
content: [{ type: "text", text: "Transcript" }],
},
...lines.map((line) => ({
type: "paragraph",
content: [{ type: "text", text: line }],
})),
];
// Append at the end of the document. On a freshly-created recording page the
// audio node is the last block, so the end position places the transcript
// directly below it.
const endPos = editor.state.doc.content.size;
editor.chain().focus().insertContentAt(endPos, content).run();
return true;
}
// Full insert path used by the open-page bridge (insertRecording): guard the // Full insert path used by the open-page bridge (insertRecording): guard the
// editor, validate/decode the payload, then upload. Never throws — resolves to // editor, validate/decode the payload, then upload. Never throws — resolves to
// a result code. // a result code.
@@ -394,6 +394,10 @@ export default function AiProviderSettings() {
useState<boolean>( useState<boolean>(
workspace?.settings?.ai?.publicShareAssistant ?? false, workspace?.settings?.ai?.publicShareAssistant ?? false,
); );
// #184: detached/autonomous agent runs (settings.ai.autonomousRuns).
const [autonomousRunsEnabled, setAutonomousRunsEnabled] = useState<boolean>(
workspace?.settings?.ai?.autonomousRuns ?? false,
);
const [chatToggleLoading, setChatToggleLoading] = useState(false); const [chatToggleLoading, setChatToggleLoading] = useState(false);
const [searchToggleLoading, setSearchToggleLoading] = useState(false); const [searchToggleLoading, setSearchToggleLoading] = useState(false);
const [dictationToggleLoading, setDictationToggleLoading] = useState(false); const [dictationToggleLoading, setDictationToggleLoading] = useState(false);
@@ -403,6 +407,8 @@ export default function AiProviderSettings() {
publicShareAssistantToggleLoading, publicShareAssistantToggleLoading,
setPublicShareAssistantToggleLoading, setPublicShareAssistantToggleLoading,
] = useState(false); ] = useState(false);
const [autonomousRunsToggleLoading, setAutonomousRunsToggleLoading] =
useState(false);
// Whether a key is currently stored server-side (drives the placeholder). // Whether a key is currently stored server-side (drives the placeholder).
const [hasApiKey, setHasApiKey] = useState(false); const [hasApiKey, setHasApiKey] = useState(false);
@@ -730,6 +736,37 @@ export default function AiProviderSettings() {
} }
} }
// Optimistic toggle for detached/autonomous agent runs
// (settings.ai.autonomousRuns). When on, a chat turn becomes a server-side run
// that survives a browser disconnect and can be reconnected to / live-followed;
// only an explicit Stop ends it. Off by default; single-instance-only in phase 1.
async function handleToggleAutonomousRuns(value: boolean) {
setAutonomousRunsToggleLoading(true);
const previous = autonomousRunsEnabled;
setAutonomousRunsEnabled(value);
try {
const updated = await updateWorkspace({ autonomousRuns: value });
setWorkspace({
...updated,
settings: {
...updated.settings,
ai: { ...updated.settings?.ai, autonomousRuns: value },
},
});
notifications.show({ message: t("Updated successfully") });
} catch (err) {
setAutonomousRunsEnabled(previous);
const message = (err as { response?: { data?: { message?: string } } })
?.response?.data?.message;
notifications.show({
message: message ?? t("Failed to update data"),
color: "red",
});
} finally {
setAutonomousRunsToggleLoading(false);
}
}
// Admins only — match the previous behavior. // Admins only — match the previous behavior.
if (!isAdmin) { if (!isAdmin) {
return ( return (
@@ -960,6 +997,31 @@ export default function AiProviderSettings() {
{...form.getInputProps("publicShareAssistantRoleId")} {...form.getInputProps("publicShareAssistantRoleId")}
/> />
{/* Detached/autonomous agent runs: a chat turn becomes a server-side run
that survives a browser disconnect; only an explicit Stop ends it.
Single-instance-only in phase 1. */}
<Group justify="space-between" align="center" wrap="nowrap" mt="md">
<Stack gap={0}>
<Text fw={600} size="sm">
{t("Autonomous agent runs")}
</Text>
<Text size="xs" c="dimmed">
{t(
"Keep an agent turn running server-side even if the browser disconnects; reconnect and follow it on reopen. Single-instance deployments only.",
)}
</Text>
</Stack>
<Switch
label={t("Enabled")}
labelPosition="left"
checked={autonomousRunsEnabled}
disabled={autonomousRunsToggleLoading}
onChange={(e) =>
handleToggleAutonomousRuns(e.currentTarget.checked)
}
/>
</Group>
<Group mt="md" align="center"> <Group mt="md" align="center">
<Button <Button
variant="default" variant="default"
@@ -26,6 +26,9 @@ export interface IWorkspace {
aiDictation?: boolean; aiDictation?: boolean;
aiDictationStreaming?: boolean; aiDictationStreaming?: boolean;
aiPublicShareAssistant?: boolean; aiPublicShareAssistant?: boolean;
// Write-only field for updateWorkspace({ autonomousRuns }). Read state lives at
// settings.ai.autonomousRuns.
autonomousRuns?: boolean;
trashRetentionDays?: number; trashRetentionDays?: number;
// Default lifetime (HOURS) for new temporary notes; frozen per-note at creation. // Default lifetime (HOURS) for new temporary notes; frozen per-note at creation.
temporaryNoteHours?: number; temporaryNoteHours?: number;
@@ -65,6 +68,9 @@ export interface IWorkspaceAiSettings {
dictation?: boolean; dictation?: boolean;
dictationStreaming?: boolean; dictationStreaming?: boolean;
publicShareAssistant?: boolean; publicShareAssistant?: boolean;
// #184: detached agent runs (a run survives a browser disconnect and can be
// reconnected to / live-followed on reopen). Gates the run-reconnect polling.
autonomousRuns?: boolean;
} }
export interface IWorkspaceSharingSettings { export interface IWorkspaceSharingSettings {
+6 -4
View File
@@ -23,7 +23,7 @@
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS", "migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts", "migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix", "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pretest": "pnpm --filter @docmost/editor-ext build", "pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build",
"test": "jest", "test": "jest",
"test:int": "jest --config test/jest-integration.json", "test:int": "jest --config test/jest-integration.json",
"test:watch": "jest --watch", "test:watch": "jest --watch",
@@ -43,6 +43,7 @@
"@clickhouse/client": "^1.18.2", "@clickhouse/client": "^1.18.2",
"@docmost/mcp": "workspace:*", "@docmost/mcp": "workspace:*",
"@docmost/pdf-inspector": "1.9.6", "@docmost/pdf-inspector": "1.9.6",
"@docmost/prosemirror-markdown": "workspace:*",
"@fastify/cookie": "^11.0.2", "@fastify/cookie": "^11.0.2",
"@fastify/multipart": "^10.0.0", "@fastify/multipart": "^10.0.0",
"@fastify/static": "^9.1.3", "@fastify/static": "^9.1.3",
@@ -175,7 +176,7 @@
"/node_modules/" "/node_modules/"
], ],
"transform": { "transform": {
"happy-dom.+\\.js$": [ "(happy-dom.+|prosemirror-markdown/build/.+)\\.js$": [
"babel-jest", "babel-jest",
{ {
"presets": [ "presets": [
@@ -193,7 +194,7 @@
"^.+\\.(t|j)sx?$": "ts-jest" "^.+\\.(t|j)sx?$": "ts-jest"
}, },
"transformIgnorePatterns": [ "transformIgnorePatterns": [
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0)(@|/))" "/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@docmost/prosemirror-markdown)(@|/))"
], ],
"collectCoverageFrom": [ "collectCoverageFrom": [
"**/*.(t|j)s" "**/*.(t|j)s"
@@ -204,7 +205,8 @@
"^@docmost/db/(.*)$": "<rootDir>/database/$1", "^@docmost/db/(.*)$": "<rootDir>/database/$1",
"^@docmost/transactional/(.*)$": "<rootDir>/integrations/transactional/$1", "^@docmost/transactional/(.*)$": "<rootDir>/integrations/transactional/$1",
"^@docmost/ee/(.*)$": "<rootDir>/ee/$1", "^@docmost/ee/(.*)$": "<rootDir>/ee/$1",
"^src/(.*)$": "<rootDir>/$1" "^src/(.*)$": "<rootDir>/$1",
"^@tiptap/react$": "<rootDir>/../test/stubs/tiptap-react.js"
} }
} }
} }
@@ -43,7 +43,6 @@ import {
Column, Column,
Status, Status,
addUniqueIdsToDoc, addUniqueIdsToDoc,
htmlToMarkdown,
TransclusionSource, TransclusionSource,
TransclusionReference, TransclusionReference,
FootnoteReference, FootnoteReference,
@@ -51,6 +50,7 @@ import {
FootnoteDefinition, FootnoteDefinition,
PageEmbed, PageEmbed,
} from '@docmost/editor-ext'; } from '@docmost/editor-ext';
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
import { generateText, getSchema, JSONContent } from '@tiptap/core'; import { generateText, getSchema, JSONContent } from '@tiptap/core';
import { generateHTML, generateJSON } from '../common/helpers/prosemirror/html'; import { generateHTML, generateJSON } from '../common/helpers/prosemirror/html';
// @tiptap/html library works best for generating prosemirror json state but not HTML // @tiptap/html library works best for generating prosemirror json state but not HTML
@@ -239,6 +239,10 @@ export function prosemirrorNodeToYElement(node: any): Y.XmlElement | Y.XmlText {
} }
export function jsonToMarkdown(tiptapJson: any): string { export function jsonToMarkdown(tiptapJson: any): string {
const html = jsonToHtml(tiptapJson); // Direct ProseMirror JSON -> Markdown via the canonical converter
return htmlToMarkdown(html); // (`@docmost/prosemirror-markdown`) — no HTML intermediate, no second
// editor-ext markdown layer. Same serializer as the page/space export and the
// git-sync vault writer, so every server PM->MD path emits identical canonical
// markdown (issue #345).
return convertProseMirrorToMarkdown(tiptapJson);
} }
@@ -1,10 +1,3 @@
export const HISTORY_INTERVAL = 5 * 60 * 1000; export const HISTORY_INTERVAL = 5 * 60 * 1000;
export const HISTORY_FAST_INTERVAL = 60 * 1000; export const HISTORY_FAST_INTERVAL = 60 * 1000;
export const HISTORY_FAST_THRESHOLD = 5 * 60 * 1000; export const HISTORY_FAST_THRESHOLD = 5 * 60 * 1000;
// #348 — debounce window for the per-page RAG re-embed job. Repeated saves
// within this window collapse to a single delayed job (coalesced by a stable
// jobId), so active editing does not pile up expensive re-embeds (external API
// + page_embeddings rewrite, concurrency 1). The worker reads the CURRENT page
// state at run time, so the last content within the window wins.
export const EMBED_DEBOUNCE_MS = 30 * 1000;
@@ -431,17 +431,7 @@ describe('PersistenceExtension.onStoreDocument — Approach-A boundary snapshot'
it('uses the canonical page.id (not the slugId doc name) for post-store side effects (#260)', async () => { it('uses the canonical page.id (not the slugId doc name) for post-store side effects (#260)', async () => {
const SLUG = 'slug-1'; // persistedHumanPage.slugId; findById resolves it const SLUG = 'slug-1'; // persistedHumanPage.slugId; findById resolves it
const document = ydocFor(doc('NEW AGENT CONTENT')); const document = ydocFor(doc('NEW AGENT CONTENT'));
// #348 — the transclusion sync now runs only when the new OR the previously pageRepo.findById.mockResolvedValue(persistedHumanPage('NEW AGENT CONTENT'));
// persisted content carries a transclusion-family node. Give the persisted
// (old) content a pageEmbed so the sync path is exercised and the #260
// UUID-vs-slugId contract asserted below is still verified.
pageRepo.findById.mockResolvedValue({
...persistedHumanPage('NEW AGENT CONTENT'),
content: {
type: 'doc',
content: [{ type: 'pageEmbed', attrs: { sourcePageId: 'src-1' } }],
},
});
pageHistoryRepo.findPageLastHistory.mockResolvedValue(null); pageHistoryRepo.findPageLastHistory.mockResolvedValue(null);
// A `page.<slugId>` document name (the bug's smoking gun), agent store over // A `page.<slugId>` document name (the bug's smoking gun), agent store over
@@ -36,13 +36,11 @@ import {
import { Page } from '@docmost/db/types/entity.types'; import { Page } from '@docmost/db/types/entity.types';
import { CollabHistoryService } from '../services/collab-history.service'; import { CollabHistoryService } from '../services/collab-history.service';
import { import {
EMBED_DEBOUNCE_MS,
HISTORY_FAST_INTERVAL, HISTORY_FAST_INTERVAL,
HISTORY_FAST_THRESHOLD, HISTORY_FAST_THRESHOLD,
HISTORY_INTERVAL, HISTORY_INTERVAL,
} from '../constants'; } from '../constants';
import { TransclusionService } from '../../core/page/transclusion/transclusion.service'; import { TransclusionService } from '../../core/page/transclusion/transclusion.service';
import { hasTransclusionFamilyNodes } from '../../core/page/transclusion/utils/transclusion-prosemirror.util';
import { observeCollabStore } from '../../integrations/metrics/metrics.registry'; import { observeCollabStore } from '../../integrations/metrics/metrics.registry';
/** /**
@@ -417,18 +415,7 @@ export class PersistenceExtension implements Extension {
// Use the canonical page UUID (page.id), not the doc-name id, which may be // Use the canonical page UUID (page.id), not the doc-name id, which may be
// a slugId for a `page.<slugId>` doc (#260). The transclusion/reference // a slugId for a `page.<slugId>` doc (#260). The transclusion/reference
// syncs write uuid-typed columns, so a slugId here threw Postgres 22P02. // syncs write uuid-typed columns, so a slugId here threw Postgres 22P02.
// await this.syncTransclusion(page.id, page.workspaceId, tiptapJson);
// #348 — skip the three sync SELECTs when neither the new content nor the
// previously-persisted content has any transclusion/reference/pageEmbed
// node: nothing to insert, and (the DB mirrors the old content) nothing to
// delete. Whenever either side has one, run the idempotent sync exactly as
// before so removals are still reconciled.
if (
hasTransclusionFamilyNodes(tiptapJson) ||
hasTransclusionFamilyNodes(page.content)
) {
await this.syncTransclusion(page.id, page.workspaceId, tiptapJson);
}
} }
if (page) { if (page) {
@@ -444,17 +431,7 @@ export class PersistenceExtension implements Extension {
(m) => m.entityId, (m) => m.entityId,
); );
// #348 — only enqueue when the mentioned-user set actually GAINED a member. if (userMentions.length > 0) {
// The processor (processPageMention) already no-ops when every current
// mention was present before (newMentions.length === 0), so skipping the
// enqueue in that case is behavior-identical and avoids piling up no-op jobs
// on every save of a page that merely CONTAINS (unchanged) mentions.
const oldMentionedUserIdSet = new Set(oldMentionedUserIds);
const hasNewMentionedUser = userMentions.some(
(m) => !oldMentionedUserIdSet.has(m.entityId),
);
if (hasNewMentionedUser) {
await this.notificationQueue.add(QueueJob.PAGE_MENTION_NOTIFICATION, { await this.notificationQueue.add(QueueJob.PAGE_MENTION_NOTIFICATION, {
userMentions: userMentions.map((m) => ({ userMentions: userMentions.map((m) => ({
userId: m.entityId, userId: m.entityId,
@@ -469,23 +446,12 @@ export class PersistenceExtension implements Extension {
} as IPageMentionNotificationJob); } as IPageMentionNotificationJob);
} }
await this.aiQueue.add( await this.aiQueue.add(QueueJob.PAGE_CONTENT_UPDATED, {
QueueJob.PAGE_CONTENT_UPDATED, // Canonical UUID: the embedding reindex resolves pages by uuid, so a
{ // slugId here threw Postgres 22P02 invalid-uuid (#260).
// Canonical UUID: the embedding reindex resolves pages by uuid, so a pageIds: [page.id],
// slugId here threw Postgres 22P02 invalid-uuid (#260). workspaceId: page.workspaceId,
pageIds: [page.id], });
workspaceId: page.workspaceId,
},
// #348 — coalesce re-embeds during active editing. A stable per-page
// jobId + delay means repeated saves within EMBED_DEBOUNCE_MS collapse
// to one delayed job instead of one expensive re-embed per save. The
// worker reads the current page state at run time, so last content wins.
// BullMQ forbids ':' in custom job ids (Redis key separator), so '-' is
// used; page.id is a UUID, so the id is unique per page. removeOnComplete
// (queue.module) frees the id after each run so the next window re-arms.
{ jobId: `embed-${page.id}`, delay: EMBED_DEBOUNCE_MS },
);
await this.enqueuePageHistory(page, lastUpdatedSource); await this.enqueuePageHistory(page, lastUpdatedSource);
} }
@@ -220,13 +220,6 @@ export class RedisSyncExtension<TCE extends CustomEvents> implements Extension {
}; };
async maintainLock(documentName: string) { async maintainLock(documentName: string) {
// #348 — clear any existing timer for this document before installing a new
// one. Without this, a second maintainLock for the same document (a
// reload-without-unload) overwrites this.locks[documentName] and leaks the
// previous interval, which keeps firing SET forever with no way to clear it.
if (this.locks[documentName]) {
clearInterval(this.locks[documentName]);
}
this.locks[documentName] = setInterval(() => { this.locks[documentName] = setInterval(() => {
this.pub.set( this.pub.set(
this.getKey(documentName), this.getKey(documentName),
@@ -4,21 +4,8 @@ export const CacheKey = {
`perm:space-roles:${userId}:${spaceId}`, `perm:space-roles:${userId}:${spaceId}`,
PAGE_CAN_EDIT: (userId: string, pageId: string) => PAGE_CAN_EDIT: (userId: string, pageId: string) =>
`perm:can-edit:${userId}:${pageId}`, `perm:can-edit:${userId}:${pageId}`,
// #348 — DomainMiddleware workspace resolution. Self-hosted resolves the single
// workspace (constant key); cloud resolves by the request subdomain (lowercased
// to match the case-insensitive `LOWER(hostname)` lookup). Every WorkspaceRepo
// mutator busts these, so staleness is bounded by both explicit invalidation and
// the short TTL below.
WORKSPACE_SELF_HOSTED: 'workspace:self-hosted',
WORKSPACE_BY_HOST: (subdomain: string) =>
`workspace:byhost:${subdomain.toLowerCase()}`,
}; };
// Permission caches dedupe repeated checks within and across short request bursts. // Permission caches dedupe repeated checks within and across short request bursts.
// 5s keeps staleness on revocations bounded. // 5s keeps staleness on revocations bounded.
export const PERMISSION_CACHE_TTL_MS = 5_000; export const PERMISSION_CACHE_TTL_MS = 5_000;
// #348 — workspace row changes rarely; a short TTL bounds staleness of
// security-relevant fields (enforceSso/enforceMfa/status) even if an explicit
// bust is ever missed, while still removing the per-request workspace query.
export const WORKSPACE_CACHE_TTL_MS = 15_000;
@@ -1,42 +1,13 @@
import { Inject, Injectable, NestMiddleware } from '@nestjs/common'; import { Injectable, NestMiddleware, NotFoundException } from '@nestjs/common';
import { FastifyRequest, FastifyReply } from 'fastify'; import { FastifyRequest, FastifyReply } from 'fastify';
import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Cache } from 'cache-manager';
import { EnvironmentService } from '../../integrations/environment/environment.service'; import { EnvironmentService } from '../../integrations/environment/environment.service';
import { WorkspaceRepo } from '@docmost/db/repos/workspace/workspace.repo'; import { WorkspaceRepo } from '@docmost/db/repos/workspace/workspace.repo';
import { Workspace } from '@docmost/db/types/entity.types';
import { withCache } from '../helpers/with-cache';
import { CacheKey, WORKSPACE_CACHE_TTL_MS } from '../helpers/cache-keys';
// #348 — timestamptz columns on the workspace row. The cache store (Keyv/Redis)
// JSON-serializes values, so a cached workspace comes back with these fields as
// ISO strings. Reviving them to Date keeps the cached path byte-identical to the
// direct DB path (postgres.js returns Date), so nothing downstream can observe a
// cache hit vs miss. Idempotent: `new Date(date)` on an already-Date value is a
// no-op-equivalent. Keep in sync with the workspace timestamptz columns.
const WORKSPACE_DATE_FIELDS: Array<keyof Workspace> = [
'createdAt',
'updatedAt',
'deletedAt',
'trialEndAt',
];
function reviveWorkspaceDates(workspace: Workspace): Workspace {
for (const field of WORKSPACE_DATE_FIELDS) {
const value = workspace[field];
if (value != null) {
(workspace as any)[field] = new Date(value as any);
}
}
return workspace;
}
@Injectable() @Injectable()
export class DomainMiddleware implements NestMiddleware { export class DomainMiddleware implements NestMiddleware {
constructor( constructor(
private workspaceRepo: WorkspaceRepo, private workspaceRepo: WorkspaceRepo,
private environmentService: EnvironmentService, private environmentService: EnvironmentService,
@Inject(CACHE_MANAGER) private readonly cacheManager: Cache,
) {} ) {}
async use( async use(
req: FastifyRequest['raw'], req: FastifyRequest['raw'],
@@ -44,21 +15,13 @@ export class DomainMiddleware implements NestMiddleware {
next: () => void, next: () => void,
) { ) {
if (this.environmentService.isSelfHosted()) { if (this.environmentService.isSelfHosted()) {
// #348 — cache the single-workspace lookup that runs on every request. const workspace = await this.workspaceRepo.findFirst();
// Invalidated by every WorkspaceRepo mutator (see bustWorkspaceCache).
const workspace = await withCache(
this.cacheManager,
CacheKey.WORKSPACE_SELF_HOSTED,
WORKSPACE_CACHE_TTL_MS,
() => this.workspaceRepo.findFirst(),
);
if (!workspace) { if (!workspace) {
//throw new NotFoundException('Workspace not found'); //throw new NotFoundException('Workspace not found');
(req as any).workspaceId = null; (req as any).workspaceId = null;
return next(); return next();
} }
reviveWorkspaceDates(workspace);
// TODO: unify // TODO: unify
(req as any).workspaceId = workspace.id; (req as any).workspaceId = workspace.id;
(req as any).workspace = workspace; (req as any).workspace = workspace;
@@ -66,21 +29,13 @@ export class DomainMiddleware implements NestMiddleware {
const header = req.headers.host; const header = req.headers.host;
const subdomain = header.split('.')[0]; const subdomain = header.split('.')[0];
// #348 — cache per-subdomain workspace resolution. Keyed by subdomain (the const workspace = await this.workspaceRepo.findByHostname(subdomain);
// hostname column); busted per hostname by every WorkspaceRepo mutator.
const workspace = await withCache(
this.cacheManager,
CacheKey.WORKSPACE_BY_HOST(subdomain),
WORKSPACE_CACHE_TTL_MS,
() => this.workspaceRepo.findByHostname(subdomain),
);
if (!workspace) { if (!workspace) {
(req as any).workspaceId = null; (req as any).workspaceId = null;
return next(); return next();
} }
reviveWorkspaceDates(workspace);
(req as any).workspaceId = workspace.id; (req as any).workspaceId = workspace.id;
(req as any).workspace = workspace; (req as any).workspace = workspace;
} }
@@ -0,0 +1,527 @@
import { Logger } from '@nestjs/common';
import {
AiChatRunService,
RunAlreadyActiveError,
ONE_ACTIVE_RUN_PER_CHAT_INDEX,
mapTurnStatusToRun,
} from './ai-chat-run.service';
/** Shape a Postgres unique-violation the way the postgres.js driver surfaces it:
* SQLSTATE 23505 + the offending index in `constraint_name`. */
function uniqueViolation(constraintName: string): Error & {
code: string;
constraint_name: string;
} {
return Object.assign(
new Error('duplicate key value violates unique constraint'),
{
code: '23505',
constraint_name: constraintName,
},
);
}
/**
* Unit coverage for the #184 phase-1 run lifecycle (AiChatRunService) with a
* hand-rolled mock repo no Nest graph, no DB. The invariant under test is the
* one that makes a run "autonomous": a run keeps going when its SUBSCRIBER (the
* browser) detaches, and ONLY an explicit stop aborts it. We assert that at the
* abort-signal level (the signal the agent loop actually consumes).
*/
/** Minimal EnvironmentService stub. Single-instance (CLOUD unset) by default. */
function makeEnv(isCloud = false) {
return { isCloud: () => isCloud };
}
function makeRepo(overrides: Record<string, jest.Mock> = {}) {
return {
insert: jest.fn(async (v: any) => ({
id: 'run-1',
status: v.status ?? 'running',
chatId: v.chatId,
workspaceId: v.workspaceId,
})),
update: jest.fn(async () => ({ id: 'run-1' })),
markStopRequested: jest.fn(async () => ({ id: 'run-1' })),
findActiveByChat: jest.fn(async () => undefined),
findLatestByChat: jest.fn(async () => undefined),
findById: jest.fn(async () => undefined),
sweepRunning: jest.fn(async () => 0),
...overrides,
};
}
describe('mapTurnStatusToRun', () => {
it('maps the turn terminal status to the run terminal status', () => {
expect(mapTurnStatusToRun('completed')).toBe('succeeded');
expect(mapTurnStatusToRun('error')).toBe('failed');
expect(mapTurnStatusToRun('aborted')).toBe('aborted');
});
});
describe('AiChatRunService.onModuleInit (startup sweep)', () => {
afterEach(() => jest.restoreAllMocks());
it('calls sweepRunning and resolves; logs when > 0', async () => {
const repo = makeRepo({ sweepRunning: jest.fn(async () => 2) });
const logSpy = jest
.spyOn(Logger.prototype, 'log')
.mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await expect(svc.onModuleInit()).resolves.toBeUndefined();
expect(repo.sweepRunning).toHaveBeenCalledTimes(1);
expect(logSpy).toHaveBeenCalledTimes(1);
expect(String(logSpy.mock.calls[0][0])).toContain('2');
});
it('a sweep failure is swallowed (never blocks startup)', async () => {
const repo = makeRepo({
sweepRunning: jest.fn(async () => {
throw new Error('db down');
}),
});
const warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await expect(svc.onModuleInit()).resolves.toBeUndefined();
// The first warn is the sweep failure (the multi-instance warn never fires
// single-instance), so the message is the db error.
expect(String(warnSpy.mock.calls[0][0])).toContain('db down');
});
it('F1 (DECISION C): the boot sweep is UNCONDITIONAL — sweepRunning is called with NO staleness window, so a fresh running run (updatedAt = now) is settled, not skipped', async () => {
// The bug: a fast restart (deploy/OOM within minutes of the last step) left a
// run stuck 'running' under the old 10-min window, 409ing every later turn in
// the chat. The fix settles ALL pending|running on boot. We assert the service
// invokes sweepRunning with no `staleMs` (the unconditional path); the repo's
// own spec proves no-window => no updatedAt filter.
const repo = makeRepo({ sweepRunning: jest.fn(async () => 1) });
jest.spyOn(Logger.prototype, 'log').mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.onModuleInit();
expect(repo.sweepRunning).toHaveBeenCalledTimes(1);
const callArgs = repo.sweepRunning.mock.calls[0] as unknown[];
const firstArg = callArgs[0] as { staleMs?: number } | undefined;
// Either no opts at all, or opts without a staleMs window => unconditional.
expect(firstArg?.staleMs).toBeUndefined();
});
it('F2 (DECISION A): warns at startup that autonomousRuns is single-instance-only when a horizontally-scaled deployment (CLOUD) is detected', async () => {
const repo = makeRepo();
const warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv(true) as never);
await svc.onModuleInit();
const warned = warnSpy.mock.calls.some((c) =>
/single-instance-only/i.test(String(c[0])),
);
expect(warned).toBe(true);
});
it('F2: does NOT warn about multi-instance on a single-instance (CLOUD unset) deployment', async () => {
const repo = makeRepo();
const warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv(false) as never);
await svc.onModuleInit();
const warned = warnSpy.mock.calls.some((c) =>
/single-instance-only/i.test(String(c[0])),
);
expect(warned).toBe(false);
});
});
describe('AiChatRunService run lifecycle', () => {
it('beginRun inserts a running row and registers a live abort controller', async () => {
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
expect(repo.insert).toHaveBeenCalledWith(
expect.objectContaining({
chatId: 'chat-1',
workspaceId: 'ws-1',
createdBy: 'user-1',
status: 'running',
trigger: 'user',
}),
);
expect(handle.runId).toBe('run-1');
expect(handle.signal.aborted).toBe(false);
expect(svc.isLocallyActive('run-1')).toBe(true);
});
it('beginRun REJECTS the racer: a 23505 on the one-active-per-chat index throws RunAlreadyActiveError (not swallowed) and registers no controller', async () => {
// The race: the controller's cheap pre-check passed for BOTH concurrent
// turns, so the loser's INSERT hits the partial unique index. That rejection
// is the authoritative gate — it must surface, not be swallowed into an
// untracked turn.
const repo = makeRepo({
insert: jest.fn(async () => {
throw uniqueViolation(ONE_ACTIVE_RUN_PER_CHAT_INDEX);
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await expect(
svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'user-1' }),
).rejects.toBeInstanceOf(RunAlreadyActiveError);
// No controller leaked for a rejected start.
expect(svc.isLocallyActive('run-1')).toBe(false);
});
it('beginRun does NOT mask an unrelated unique violation as already-active', async () => {
// A 23505 on some OTHER constraint is a real bug, not the race — it must
// propagate unchanged so it is never silently treated as "already active".
const other = uniqueViolation('ai_chat_runs_pkey');
const repo = makeRepo({
insert: jest.fn(async () => {
throw other;
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await expect(
svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'user-1' }),
).rejects.toBe(other);
});
it('beginRun propagates a non-unique insert failure unchanged', async () => {
const boom = new Error('connection reset');
const repo = makeRepo({
insert: jest.fn(async () => {
throw boom;
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await expect(
svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'user-1' }),
).rejects.toBe(boom);
});
it('two concurrent begins on one chat: exactly one wins, the other is rejected as already-active', async () => {
// Integration-style: model the DB partial unique index with a one-shot slot.
// The first insert claims it; the second hits a 23505 on the active index.
let slotTaken = false;
const repo = makeRepo({
insert: jest.fn(async (v: any) => {
if (slotTaken) throw uniqueViolation(ONE_ACTIVE_RUN_PER_CHAT_INDEX);
slotTaken = true;
return { id: 'run-win', status: v.status, chatId: v.chatId };
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const results = await Promise.allSettled([
svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'user-1' }),
svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'user-1' }),
]);
const fulfilled = results.filter((r) => r.status === 'fulfilled');
const rejected = results.filter((r) => r.status === 'rejected');
expect(fulfilled).toHaveLength(1);
expect(rejected).toHaveLength(1);
expect((rejected[0] as PromiseRejectedResult).reason).toBeInstanceOf(
RunAlreadyActiveError,
);
// Exactly the winner is locally active.
expect(svc.isLocallyActive('run-win')).toBe(true);
});
it('a SUBSCRIBER detaching does NOT abort the run (only an explicit stop does)', async () => {
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
// Model a browser disconnect: nothing in the run service is told to stop.
// The signal the agent loop consumes must stay un-aborted and the run stays
// locally active — i.e. it keeps running server-side.
expect(handle.signal.aborted).toBe(false);
expect(svc.isLocallyActive('run-1')).toBe(true);
// markStopRequested was never called by a mere detach.
expect(repo.markStopRequested).not.toHaveBeenCalled();
});
it('requestStop aborts the live controller, marks the row, and reports true', async () => {
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
const aborted = jest.fn();
handle.signal.addEventListener('abort', aborted);
const result = await svc.requestStop('run-1', 'ws-1');
expect(result).toBe(true);
expect(handle.signal.aborted).toBe(true);
expect(aborted).toHaveBeenCalledTimes(1);
expect(repo.markStopRequested).toHaveBeenCalledWith('run-1', 'ws-1');
});
it('requestStop on a run this replica does NOT hold still marks the row (true)', async () => {
// e.g. after a restart, or a sibling replica owns the controller. The row is
// marked so the owning replica/sweep settles it; we report a stop took effect.
const repo = makeRepo({
markStopRequested: jest.fn(async () => ({ id: 'run-9' })),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const result = await svc.requestStop('run-9', 'ws-1');
expect(result).toBe(true);
expect(svc.isLocallyActive('run-9')).toBe(false);
});
it('requestStop still aborts the live controller when markStopRequested rejects (transient DB error)', async () => {
// F15: the in-memory abort is the ONLY thing that stops a run and must not be
// hostage to the audit write of stop_requested_at. A transient failure on
// markStopRequested must NOT prevent abort() nor make requestStop throw.
const warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined);
const repo = makeRepo({
markStopRequested: jest.fn(async () => {
throw new Error('pool exhausted');
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
const aborted = jest.fn();
handle.signal.addEventListener('abort', aborted);
// Does NOT throw despite the DB write rejecting.
const result = await svc.requestStop('run-1', 'ws-1');
// The live turn was aborted even though the audit write failed...
expect(handle.signal.aborted).toBe(true);
expect(aborted).toHaveBeenCalledTimes(1);
expect(repo.markStopRequested).toHaveBeenCalledWith('run-1', 'ws-1');
// ...the catch branch logged the swallowed failure...
expect(warnSpy).toHaveBeenCalledTimes(1);
// ...and a stop is reported as having taken effect (the entry existed).
expect(result).toBe(true);
warnSpy.mockRestore();
});
it('requestStop on an already-settled run (nothing active) reports false', async () => {
const repo = makeRepo({
markStopRequested: jest.fn(async () => undefined),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const result = await svc.requestStop('run-done', 'ws-1');
expect(result).toBe(false);
});
it('finalizeRun settles the row to the mapped status with finishedAt and drops the in-memory entry', async () => {
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
expect(svc.isLocallyActive('run-1')).toBe(true);
await svc.finalizeRun('run-1', 'ws-1', 'error', 'provider blew up');
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({
status: 'failed',
error: 'provider blew up',
finishedAt: expect.any(Date),
}),
);
});
it('finalizeRun is IDEMPOTENT: a second settle no-ops (single terminal write)', async () => {
// The #184 review fix: AiChatService.stream wraps the turn in a safety-net
// catch that settles a failed turn AND streamText's terminal callback may
// also settle — both routes call finalizeRun. Only the FIRST may write the
// terminal row; the second must no-op so a late settle can never clobber the
// real terminal status or double-write the row.
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
await svc.finalizeRun('run-1', 'ws-1', 'error', 'first');
expect(svc.isLocallyActive('run-1')).toBe(false);
// A second settle (e.g. a streamText callback firing after the catch) no-ops.
await svc.finalizeRun('run-1', 'ws-1', 'completed', undefined);
expect(repo.update).toHaveBeenCalledTimes(1);
expect(repo.update).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'failed', error: 'first' }),
);
});
it('CONCURRENCY: two simultaneous finalizeRun on the same run write the terminal row EXACTLY ONCE (the 2nd caller exits synchronously at the atomic claim)', async () => {
// The CRITICAL race: AiChatService.stream's safety-net catch settles the turn
// to 'error' while a streamText terminal callback also settles it — both call
// finalizeRun for the SAME runId. The once-gate must close ATOMICALLY: a
// `settled.has` check alone is read BEFORE the awaited UPDATE, so both callers
// would pass it and BOTH write the row (last-write-wins clobber + double
// write). The fix claims the run with a SYNCHRONOUS `active.delete` before any
// await, so the second caller returns in the same tick, before the UPDATE.
//
// We force the two calls to overlap by making `update` return a promise we
// resolve only AFTER both finalizeRun calls have run their synchronous bodies.
let resolveUpdate!: (v: unknown) => void;
const updateGate = new Promise((res) => {
resolveUpdate = res;
});
const update = jest.fn(() => updateGate);
const repo = makeRepo({ update });
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
// Fire both before the (pending) update resolves. The first synchronously
// claims the entry (active.delete) and awaits update; the second, started in
// the same macrotask, finds the entry already gone and returns at the claim
// WITHOUT ever calling update.
const p1 = svc.finalizeRun('run-1', 'ws-1', 'completed');
const p2 = svc.finalizeRun('run-1', 'ws-1', 'error', 'safety-net');
// The decisive assertion: exactly one caller reached the terminal UPDATE.
expect(update).toHaveBeenCalledTimes(1);
// Let the single in-flight update land; both calls resolve cleanly.
resolveUpdate({ id: 'run-1' });
await Promise.all([p1, p2]);
expect(update).toHaveBeenCalledTimes(1);
// The winner is the FIRST caller ('completed' -> 'succeeded'); the late
// 'error' settle never wrote, so it could not clobber the real status.
expect(update).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
expect(svc.isLocallyActive('run-1')).toBe(false);
});
it('F6: a TRANSIENT terminal-write failure is ridden out by the bounded retry — the run is settled, not stranded', async () => {
// The bug: finalizeRun used to DROP the in-memory entry BEFORE the terminal
// UPDATE, then only warn-log a failure. A single transient blip (pool
// exhaustion / deadlock / connection hiccup) on that PK UPDATE left the row
// 'running' with nothing left to recover it -> every later turn in that chat
// 409s until a restart. The fix updates FIRST and retries.
let calls = 0;
const repo = makeRepo({
update: jest.fn(async () => {
calls += 1;
if (calls === 1) throw new Error('deadlock detected');
return { id: 'run-1' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
await svc.finalizeRun('run-1', 'ws-1', 'completed');
// The retry landed the terminal write: the entry is dropped (slot freed) and
// the row carries the real terminal status — NOT stranded at 'running'.
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenCalledTimes(2);
expect(repo.update).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
});
it('F6: if the terminal write keeps failing, the entry is RETAINED and a LATER settle completes it (chat not permanently 409d)', async () => {
// Worst case: the DB is down for the whole first finalize (all attempts fail).
// The run must NOT be silently lost — the entry stays so a subsequent settle
// (a streamText callback, requestStop -> onAbort, or a future sweep) can retry.
let healthy = false;
const repo = makeRepo({
update: jest.fn(async () => {
if (!healthy) throw new Error('pool exhausted');
return { id: 'run-1' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
workspaceId: 'ws-1',
userId: 'user-1',
});
// First settle: every bounded attempt fails -> entry retained, NOT settled.
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(true);
// F12: the give-up emits ONE explicit, greppable ERROR (run + chat context)
// so an operator can tell "gave up, run held in memory" from a per-attempt
// blip — distinct from the per-attempt warns.
const gaveUp = errorSpy.mock.calls.some(
(c) =>
/NON-TERMINAL/.test(String(c[0])) &&
/run-1/.test(String(c[0])) &&
/chat-1/.test(String(c[0])),
);
expect(gaveUp).toBe(true);
// The DB recovers; a later settle now succeeds and frees the slot.
healthy = true;
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
// And it is now idempotent: a further settle no-ops (terminal row already
// written), so a double-settle can never clobber the real status.
const callsBefore = repo.update.mock.calls.length;
await svc.finalizeRun('run-1', 'ws-1', 'error', 'late');
expect(repo.update).toHaveBeenCalledTimes(callsBefore);
});
it('recordStep / linkAssistantMessage are best-effort: a repo failure is swallowed', async () => {
const repo = makeRepo({
update: jest.fn(async () => {
throw new Error('transient');
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await expect(svc.recordStep('run-1', 'ws-1', 3)).resolves.toBeUndefined();
await expect(
svc.linkAssistantMessage('run-1', 'ws-1', 'msg-1'),
).resolves.toBeUndefined();
});
});
@@ -0,0 +1,452 @@
import { Injectable, Logger, OnModuleInit } from '@nestjs/common';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatRun } from '@docmost/db/types/entity.types';
import { isUniqueViolation, violatedConstraint } from '@docmost/db/utils';
import { EnvironmentService } from '../../integrations/environment/environment.service';
/** Name of the partial unique index enforcing "one active run per chat" (see the
* ai_chat_runs migration). A 23505 on THIS constraint is the race-safe signal
* that a concurrent turn already owns the chat distinct from any other unique
* collision, which must NOT be silently treated as "already active". */
export const ONE_ACTIVE_RUN_PER_CHAT_INDEX = 'ai_chat_runs_one_active_per_chat';
/**
* Thrown by {@link AiChatRunService.beginRun} when the run-row INSERT loses the
* race for a chat's single active slot (the partial unique index rejects it with
* a 23505). This is the AUTHORITATIVE concurrency gate: the controller's cheap
* pre-check is only a fast-path, and a request that slips past it must NOT run
* untracked. The caller (AiChatService.stream) translates this into a 409 and
* aborts the turn BEFORE any AI/provider call.
*/
export class RunAlreadyActiveError extends Error {
constructor(public readonly chatId: string) {
super(`An agent run is already in progress for chat ${chatId}`);
this.name = 'RunAlreadyActiveError';
}
}
/**
* The terminal status of a TURN (the #183 assistant-row lifecycle) maps onto the
* terminal status of a RUN (#184). A turn that completed -> the run succeeded; a
* turn that errored -> the run failed; a turn aborted (explicit user stop) -> the
* run aborted. Pure + unit-testable.
*/
export type TurnTerminalStatus = 'completed' | 'error' | 'aborted';
export type RunTerminalStatus = 'succeeded' | 'failed' | 'aborted';
export function mapTurnStatusToRun(
status: TurnTerminalStatus,
): RunTerminalStatus {
switch (status) {
case 'completed':
return 'succeeded';
case 'error':
return 'failed';
case 'aborted':
return 'aborted';
}
}
/** An in-flight run held in process memory: its AbortController is the ONLY thing
* that can stop the turn (an explicit user stop), independent of the browser
* socket. A mere disconnect never touches it, so the run keeps going. */
interface ActiveRun {
controller: AbortController;
chatId: string;
workspaceId: string;
}
/** The live handle the streaming path drives a run through (returned by
* {@link AiChatRunService.beginRun}). The `signal` governs the agent loop's
* abort wired to the run, NOT to the HTTP socket. */
export interface RunHandle {
runId: string;
signal: AbortSignal;
}
/**
* AiChatRunService (#184 phase 1) owns the agent RUN as a first-class,
* server-side lifecycle object detached from the HTTP request / browser window.
*
* Responsibilities:
* - create a run row when a turn starts (inserted directly as 'running'; the
* 'pending' status is only the column default + a reserved value, never
* written by code in phase 1) and register an in-memory AbortController for it
* (the explicit-stop lever);
* - finalize the run row (succeeded / failed / aborted) and unregister it;
* - service an EXPLICIT user stop (`requestStop`) the ONLY thing that aborts a
* run; a browser disconnect deliberately does NOT;
* - crash-recovery sweep of dangling runs on startup.
*
* The agent loop itself still runs in AiChatService.stream (reusing #183's
* step-granular durable write path, `consumeStream` already drains it independent
* of the socket); this service only wraps it in a durable lifecycle and an
* abort handle that outlives the subscriber.
*/
@Injectable()
export class AiChatRunService implements OnModuleInit {
private readonly logger = new Logger(AiChatRunService.name);
// runId -> ActiveRun. Process-local on purpose (phase 1 is single-process /
// in-memory transport; a cross-process BullMQ runner + Redis stop-signal is
// deferred to phase 2). A stop for a runId not in this map (e.g. after a
// restart) still records `stop_requested_at` on the row.
private readonly active = new Map<string, ActiveRun>();
// runIds whose TERMINAL row write has SUCCEEDED — the idempotency once-gate
// (F6). A finalize must short-circuit only AFTER the terminal write has landed,
// NOT merely after the in-memory entry was dropped: a transient UPDATE failure
// has to stay retryable, so "already settled" means "row already terminal", not
// "entry already gone". Grows by one short UUID per finished run over process
// uptime — negligible in phase 1's single process.
private readonly settled = new Set<string>();
// Bounded retry for the terminal write (F6): a single PK UPDATE can fail
// transiently under many fire-and-forget writes (pool exhaustion, deadlock, a
// brief connection blip). Riding out that blip in-place matters because the
// dominant success path (streamText onFinish) settles exactly ONCE — if that
// write is dropped and never retried, the row is stranded 'running' and the
// one-active-run gate 409s every future turn in the chat until a restart (no
// periodic sweep in phase 1).
private static readonly FINALIZE_MAX_ATTEMPTS = 3;
private static readonly FINALIZE_RETRY_BASE_MS = 50;
constructor(
private readonly runRepo: AiChatRunRepo,
private readonly environment: EnvironmentService,
) {}
/**
* Crash-recovery sweep on server start: settle EVERY run still left
* pending/running to 'aborted' (F1 / DECISION C). The boot sweep is
* UNCONDITIONAL no staleness window because phase 1 is single-process: on a
* fresh boot any pending|running run is definitionally hung (no live runner owns
* it), so even a fast restart (deploy/OOM within minutes of the last step) can
* no longer leave a run stuck 'running' forever (which would make the
* one-active-run gate 409 every future turn in that chat). The staleness window
* is reintroduced only for the phase-2 multi-instance timer sweep, where a
* booting replica must not abort a run another replica is actively executing.
* Best-effort a sweep failure is logged but MUST NOT block startup (mirrors
* AiChatService.onModuleInit for #183).
*/
async onModuleInit(): Promise<void> {
this.warnIfMultiInstance();
try {
// No `staleMs`: unconditional boot sweep (F1). See AiChatRunRepo.sweepRunning.
const swept = await this.runRepo.sweepRunning();
if (swept > 0) {
this.logger.log(
`Startup sweep: marked ${swept} dangling agent run(s) as 'aborted'.`,
);
}
} catch (err) {
this.logger.warn(
`Startup sweep of dangling runs failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
/**
* F2 (DECISION A): autonomous runs are SINGLE-INSTANCE-ONLY in phase 1. An
* explicit Stop, and the in-memory AbortController that backs it, are
* process-local: a Stop only aborts the live turn if it lands on the SAME
* replica that owns the run (it still stamps `stop_requested_at` cross-instance,
* but nothing reads that flag during an active run yet). Cross-instance pub/sub
* stop is phase 2. So if the deployment is horizontally scaled, warn loudly at
* startup that a Stop may not reach a run executing on another replica.
*
* DETECTION: this codebase always wires the socket.io Redis adapter (REDIS_URL
* is mandatory), so the adapter alone is NOT a horizontal-scaling signal. The
* authoritative signal the codebase has is `CLOUD=true` (EnvironmentService
* .isCloud()), the Docmost-cloud multi-replica deployment. We warn whenever that
* is set, because any workspace could enable settings.ai.autonomousRuns. A
* self-hosted operator running multiple replicas behind a load balancer is also
* multi-instance; the deploy docs (.env.example / AGENTS.md) spell out the
* single-instance constraint for that case.
*/
private warnIfMultiInstance(): void {
if (this.environment.isCloud()) {
this.logger.warn(
'Autonomous agent runs (settings.ai.autonomousRuns) are SINGLE-INSTANCE-ONLY ' +
'in phase 1: a horizontally-scaled deployment was detected (CLOUD=true). ' +
'An explicit Stop only aborts a run executing on the same replica that owns ' +
'it (cross-instance Stop is not yet reliable — phase 2). Run a single ' +
'instance if you enable autonomousRuns, or keep the flag off.',
);
}
}
/**
* Start a run for a turn: insert the run row (status 'running', startedAt now),
* register a fresh AbortController for it, and return a {@link RunHandle} whose
* `signal` the agent loop uses. The DB partial unique index guarantees at most
* one active run per chat a second concurrent start on the same chat REJECTS
* at the insert (a 23505 on {@link ONE_ACTIVE_RUN_PER_CHAT_INDEX}). That
* rejection is the AUTHORITATIVE race gate: it is surfaced as a distinct
* {@link RunAlreadyActiveError} (NOT swallowed), so the caller turns it into a
* 409 and never streams an untracked turn. The controller is registered AFTER a
* successful insert so a rejected start leaks nothing.
*/
async beginRun(args: {
chatId: string;
workspaceId: string;
userId: string;
trigger?: string;
}): Promise<RunHandle> {
let run: AiChatRun;
try {
run = await this.runRepo.insert({
chatId: args.chatId,
workspaceId: args.workspaceId,
createdBy: args.userId,
trigger: args.trigger ?? 'user',
status: 'running',
startedAt: new Date(),
});
} catch (err) {
// The race backstop: a concurrent turn already holds this chat's single
// active slot, so the partial unique index rejected our insert. Surface a
// distinct signal — the caller MUST reject this turn (409), not run it
// untracked. Any OTHER error propagates unchanged.
if (
isUniqueViolation(err) &&
violatedConstraint(err) === ONE_ACTIVE_RUN_PER_CHAT_INDEX
) {
throw new RunAlreadyActiveError(args.chatId);
}
throw err;
}
const controller = new AbortController();
this.active.set(run.id, {
controller,
chatId: args.chatId,
workspaceId: args.workspaceId,
});
return { runId: run.id, signal: controller.signal };
}
/** Link the assistant message (the #183 projection) to its run. Best-effort. */
async linkAssistantMessage(
runId: string,
workspaceId: string,
assistantMessageId: string,
): Promise<void> {
try {
await this.runRepo.update(runId, workspaceId, { assistantMessageId });
} catch (err) {
this.logger.warn(
`Failed to link assistant message to run ${runId}: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
/** Persist progress: bump the run's finished-step count. Best-effort (never
* blocks or breaks the stream). */
async recordStep(
runId: string,
workspaceId: string,
stepCount: number,
): Promise<void> {
try {
await this.runRepo.update(runId, workspaceId, { stepCount });
} catch (err) {
this.logger.warn(
`Failed to record step for run ${runId}: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
/**
* Finalize a run to its terminal status (succeeded / failed / aborted),
* stamping finishedAt + any error. Best-effort, but ROBUST against a transient
* terminal-write failure (F6) AND atomically safe against a concurrent settle.
*
* ATOMIC ONCE-CLAIM (the gate must close in ONE synchronous tick): two
* finalizeRun calls for the SAME run can race the documented real path is
* AiChatService.stream's safety-net catch settling the turn to 'error' while a
* streamText terminal callback (onFinish/onAbort/onError) ALSO settles it. The
* `settled.has` check alone is NOT a gate: it is read BEFORE the awaited UPDATE,
* so two callers can both see `false` and both write the row (last-write-wins
* clobbers the real terminal status, and the bounded retry only widens that
* window). The claim therefore happens via `active.delete`, a SYNCHRONOUS
* check-and-clear with NO await between the gate and the entry removal: the
* second concurrent caller finds the entry already gone and returns in the same
* tick, before any UPDATE. The transition "nobody is finalizing" -> "I am
* finalizing" is thus a single atomic step.
*
* ORDER MATTERS (F6): once we own the claim, the terminal UPDATE happens FIRST;
* only once it SUCCEEDS do we record the run as settled. If the UPDATE fails on
* every bounded attempt we RESTORE the in-memory entry, leave the run UNsettled,
* and emit an ERROR signal that the row is left non-terminal 'running' (which
* would 409 every future turn in the chat until recovery). An in-process retry
* by a LATER settle is only POSSIBLE, never guaranteed: it needs (a) the entry
* to have been restored at the give-up path AND (b) a fresh settler to arrive
* AFTER that restore. A concurrent settler that arrives DURING the retry window
* while the entry is deleted for backoff and not yet restored is consumed at
* the synchronous `active.delete` claim (it finds nothing to delete and returns
* a no-op), so it does NOT become an in-process retrier. The NO-streamText path
* (the turn threw before streamText was wired, so ONLY the safety-net ever
* settles) likewise has no second in-process settler at all. The UNCONDITIONAL
* backstop in every case is the boot sweep on the next restart (phase 1 has no
* periodic in-process sweep); the retained entry is bounded (cleared on restart)
* and harmless meanwhile.
*
* IDEMPOTENT on SUCCESS (#184 review): the terminal write happens AT MOST ONCE
* per run. After a successful write the once-gate keys off {@link settled} (the
* terminal row already written) so a settle arriving AFTER the entry was already
* dropped-and-settled returns early; a settle racing the in-flight write is
* stopped earlier still, by the `active.delete` claim. Either way a genuine
* double-settle collapses to a single write and a late settle can never clobber
* the real terminal status or double-write the row.
*/
async finalizeRun(
runId: string,
workspaceId: string,
turnStatus: TurnTerminalStatus,
error?: string,
): Promise<void> {
// ---- Atomic once-claim (synchronous; NO await before the gate closes) ----
// Already terminally written -> idempotent no-op.
if (this.settled.has(runId)) return;
// Capture the entry BEFORE the delete so a total-failure path can restore it.
const entry = this.active.get(runId);
// SYNCHRONOUS check-and-clear: the FIRST caller deletes (claims) the entry;
// any concurrent SECOND caller finds nothing to delete and returns HERE, in
// the same tick, before any await — so it can never reach the UPDATE.
if (!this.active.delete(runId)) return;
let lastError: unknown;
for (
let attempt = 1;
attempt <= AiChatRunService.FINALIZE_MAX_ATTEMPTS;
attempt++
) {
try {
await this.runRepo.update(runId, workspaceId, {
status: mapTurnStatusToRun(turnStatus),
finishedAt: new Date(),
error: error ?? null,
});
// Terminal write landed: arm the once-gate. The entry is already gone
// (claimed above); we do NOT restore it. The slot is now free.
this.settled.add(runId);
return;
} catch (err) {
lastError = err;
this.logger.warn(
`Failed to finalize run ${runId} (attempt ${attempt}/${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
}): ${err instanceof Error ? err.message : 'unknown error'}`,
);
if (attempt < AiChatRunService.FINALIZE_MAX_ATTEMPTS) {
await this.delay(AiChatRunService.FINALIZE_RETRY_BASE_MS * attempt);
}
}
}
// Every attempt failed: this is a give-up, materially worse than a per-attempt
// blip — the row is left NON-TERMINAL ('running'), so emit ONE explicit,
// greppable ERROR so an operator can tell "survived a blip" from "gave up, run
// held in memory until recovery" (the last warn alone says only "attempt 3/3").
this.logger.error(
`Run ${runId} (chat ${entry?.chatId ?? 'unknown'}) left NON-TERMINAL ` +
`('running'): terminal write failed after ${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
} attempts; entry retained in memory, recovery deferred to next settle / ` +
`boot sweep`,
lastError,
);
// RESTORE the claimed entry (and leave the run UNsettled) so a LATER settle
// that arrives AFTER this restore MAY retry the terminal write — but that
// in-process retry is NOT guaranteed (a concurrent settler caught in the retry
// window above is consumed at the `active.delete` claim, and the no-streamText
// path has no second settler at all). The UNCONDITIONAL backstop in every case
// is the boot sweep on the next restart; the restored entry is bounded and
// cleared on restart.
if (entry) this.active.set(runId, entry);
}
/** Small async backoff between terminal-write retries (F6). Isolated so it is
* trivial to stub/fake-time in tests. */
private delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/**
* Request an EXPLICIT stop of a run (the user pressed Stop). This is the ONLY
* thing that aborts a run distinct from a browser disconnect, which leaves
* the run going. Aborts the in-process controller FIRST (the only thing that
* actually stops the run, if this replica owns it), then makes a best-effort
* attempt to stamp `stop_requested_at` that audit write stamps only while the
* row is active and may be skipped on a DB error or lost to the finalize race,
* which is acceptable since the row still settles as 'aborted'. Returns true
* when a stop took effect (row marked and/or controller aborted), false when
* there was nothing active to stop.
*/
async requestStop(runId: string, workspaceId: string): Promise<boolean> {
const entry = this.active.get(runId);
if (entry) {
// Abort the live turn FIRST -> streamText onAbort fires -> the partial is
// persisted (#183) and finalizeRun settles the row as 'aborted'. This is
// the ONLY thing that aborts a run, so it MUST NOT be hostage to the audit
// write below: a transient failure on `markStopRequested` (pool exhaustion,
// deadlock, dropped connection) must never leave the run executing despite
// an explicit Stop. At worst only the `stop_requested_at` timestamp is lost.
entry.controller.abort();
}
// Record `stop_requested_at` (best-effort). A transient DB failure here is
// logged and treated as `marked = false`; the abort above already took
// effect, so we never rethrow and skip stopping the run. Note: because
// markStopRequested only stamps while the row is active, aborting first means
// even a healthy write can lose the race against the resulting finalize and
// skip the stamp — acceptable, as the row still settles as 'aborted' and only
// this audit timestamp may be lost.
let marked: unknown;
try {
marked = await this.runRepo.markStopRequested(runId, workspaceId);
} catch (err) {
marked = undefined;
this.logger.warn(
`requestStop: markStopRequested failed for run ${runId} ` +
`(stop_requested_at not recorded); abort already issued: ` +
`${err instanceof Error ? err.message : String(err)}`,
);
}
return Boolean(marked) || Boolean(entry);
}
/** Latest persisted run for a chat the reconnect target (an in-flight or
* finished run). Pure read-through to the repo. */
getLatestForChat(
chatId: string,
workspaceId: string,
): Promise<AiChatRun | undefined> {
return this.runRepo.findLatestByChat(chatId, workspaceId);
}
/** Fetch a run by id (workspace-scoped). Used to resolve + ownership-check an
* explicit stop targeting a runId. */
getRun(runId: string, workspaceId: string): Promise<AiChatRun | undefined> {
return this.runRepo.findById(runId, workspaceId);
}
/** The active run on a chat, if any (used to reject a concurrent start with a
* clean 409 before committing to the stream). */
getActiveForChat(
chatId: string,
workspaceId: string,
): Promise<AiChatRun | undefined> {
return this.runRepo.findActiveByChat(chatId, workspaceId);
}
/** Test/diagnostic seam: whether this replica is holding a live controller for
* the run. */
isLocallyActive(runId: string): boolean {
return this.active.has(runId);
}
}
@@ -25,6 +25,7 @@ describe('AiChatController.boundChat', () => {
}; };
const controller = new AiChatController( const controller = new AiChatController(
{} as never, {} as never,
{} as never, // aiChatRunService
aiChatRepo as never, aiChatRepo as never,
{} as never, {} as never,
{} as never, {} as never,
@@ -53,6 +53,7 @@ describe('AiChatController.export', () => {
}; };
const controller = new AiChatController( const controller = new AiChatController(
{} as never, {} as never,
{} as never, // aiChatRunService
aiChatRepo as never, aiChatRepo as never,
aiChatMessageRepo as never, aiChatMessageRepo as never,
{} as never, {} as never,
@@ -0,0 +1,164 @@
import { BadRequestException, ForbiddenException } from '@nestjs/common';
import { AiChatController } from './ai-chat.controller';
import type { User, Workspace } from '@docmost/db/types/entity.types';
/**
* Wiring spec for the #184 run-reconnect / run-stop endpoints
* (`POST /ai-chat/run` and `POST /ai-chat/stop`). Both are OWNER-gated via
* assertOwnedChat (the requesting user must own the chat) and NOT flag-gated.
* Exercised with hand-rolled mocks no Nest graph, no DB. The controller's
* constructor order is (aiChatService, aiChatRunService, aiChatRepo,
* aiChatMessageRepo, aiTranscription).
*/
describe('AiChatController run endpoints (#184)', () => {
const user = { id: 'u1' } as User;
const workspace = { id: 'ws1' } as Workspace;
function makeController(opts: {
chat?: unknown; // what aiChatRepo.findById returns (owner-gate)
run?: unknown; // getLatestForChat / getRun result
activeRun?: unknown; // getActiveForChat result
message?: unknown; // aiChatMessageRepo.findById result
stopped?: boolean; // requestStop result
}) {
const aiChatRunService = {
getLatestForChat: jest.fn().mockResolvedValue(opts.run),
getRun: jest.fn().mockResolvedValue(opts.run),
getActiveForChat: jest.fn().mockResolvedValue(opts.activeRun),
requestStop: jest.fn().mockResolvedValue(opts.stopped ?? false),
};
const aiChatRepo = {
findById: jest.fn().mockResolvedValue(opts.chat),
};
const aiChatMessageRepo = {
findById: jest.fn().mockResolvedValue(opts.message),
};
const controller = new AiChatController(
{} as never, // aiChatService
aiChatRunService as never,
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiTranscription
{} as never, // pageRepo
);
return { controller, aiChatRunService, aiChatRepo, aiChatMessageRepo };
}
describe('POST /ai-chat/run (getRun)', () => {
it('owner-gates: a chat the user does not own throws ForbiddenException', async () => {
const { controller, aiChatRunService } = makeController({
chat: { id: 'c1', creatorId: 'someone-else' },
});
await expect(
controller.getRun({ chatId: 'c1' }, user, workspace),
).rejects.toBeInstanceOf(ForbiddenException);
// It must NOT reach the run lookup once the owner-gate fails.
expect(aiChatRunService.getLatestForChat).not.toHaveBeenCalled();
});
it('returns { run: null, message: null } when the chat has never had a run', async () => {
const { controller, aiChatRunService } = makeController({
chat: { id: 'c1', creatorId: 'u1' },
run: undefined,
});
const res = await controller.getRun({ chatId: 'c1' }, user, workspace);
expect(res).toEqual({ run: null, message: null });
expect(aiChatRunService.getLatestForChat).toHaveBeenCalledWith(
'c1',
'ws1',
);
});
it('returns the run and its projected assistant message', async () => {
const run = { id: 'run-1', chatId: 'c1', assistantMessageId: 'm1' };
const message = { id: 'm1', role: 'assistant' };
const { controller, aiChatMessageRepo } = makeController({
chat: { id: 'c1', creatorId: 'u1' },
run,
message,
});
const res = await controller.getRun({ chatId: 'c1' }, user, workspace);
expect(res).toEqual({ run, message });
expect(aiChatMessageRepo.findById).toHaveBeenCalledWith('m1', 'ws1');
});
it('returns message: null when the run has no linked assistant message', async () => {
const run = { id: 'run-1', chatId: 'c1', assistantMessageId: null };
const { controller, aiChatMessageRepo } = makeController({
chat: { id: 'c1', creatorId: 'u1' },
run,
});
const res = await controller.getRun({ chatId: 'c1' }, user, workspace);
expect(res).toEqual({ run, message: null });
expect(aiChatMessageRepo.findById).not.toHaveBeenCalled();
});
});
describe('POST /ai-chat/stop (stopRun)', () => {
it('throws BadRequestException when neither runId nor chatId is given', async () => {
const { controller } = makeController({});
await expect(
controller.stopRun({}, user, workspace),
).rejects.toBeInstanceOf(BadRequestException);
});
it('stops by runId: owner-gates via the run’s chat, then requests the stop', async () => {
const { controller, aiChatRunService, aiChatRepo } = makeController({
run: { id: 'run-1', chatId: 'c1' },
chat: { id: 'c1', creatorId: 'u1' },
stopped: true,
});
const res = await controller.stopRun({ runId: 'run-1' }, user, workspace);
expect(res).toEqual({ stopped: true });
expect(aiChatRunService.getRun).toHaveBeenCalledWith('run-1', 'ws1');
expect(aiChatRepo.findById).toHaveBeenCalledWith('c1', 'ws1');
expect(aiChatRunService.requestStop).toHaveBeenCalledWith('run-1', 'ws1');
});
it('stops by runId: a foreign run’s chat throws ForbiddenException (no stop)', async () => {
const { controller, aiChatRunService } = makeController({
run: { id: 'run-1', chatId: 'c1' },
chat: { id: 'c1', creatorId: 'someone-else' },
});
await expect(
controller.stopRun({ runId: 'run-1' }, user, workspace),
).rejects.toBeInstanceOf(ForbiddenException);
expect(aiChatRunService.requestStop).not.toHaveBeenCalled();
});
it('stops by runId: an unknown run reports { stopped: false }', async () => {
const { controller, aiChatRunService } = makeController({
run: undefined,
});
const res = await controller.stopRun({ runId: 'gone' }, user, workspace);
expect(res).toEqual({ stopped: false });
expect(aiChatRunService.requestStop).not.toHaveBeenCalled();
});
it('stops by chatId: owner-gates, resolves the active run, requests the stop', async () => {
const { controller, aiChatRunService, aiChatRepo } = makeController({
chat: { id: 'c1', creatorId: 'u1' },
activeRun: { id: 'run-9' },
stopped: true,
});
const res = await controller.stopRun({ chatId: 'c1' }, user, workspace);
expect(res).toEqual({ stopped: true });
expect(aiChatRepo.findById).toHaveBeenCalledWith('c1', 'ws1');
expect(aiChatRunService.getActiveForChat).toHaveBeenCalledWith(
'c1',
'ws1',
);
expect(aiChatRunService.requestStop).toHaveBeenCalledWith('run-9', 'ws1');
});
it('stops by chatId: reports { stopped: false } when no run is active', async () => {
const { controller, aiChatRunService } = makeController({
chat: { id: 'c1', creatorId: 'u1' },
activeRun: undefined,
});
const res = await controller.stopRun({ chatId: 'c1' }, user, workspace);
expect(res).toEqual({ stopped: false });
expect(aiChatRunService.requestStop).not.toHaveBeenCalled();
});
});
});
@@ -1,6 +1,7 @@
import { import {
BadRequestException, BadRequestException,
Body, Body,
ConflictException,
Controller, Controller,
ForbiddenException, ForbiddenException,
HttpCode, HttpCode,
@@ -20,7 +21,13 @@ import { JwtAuthGuard } from '../../common/guards/jwt-auth.guard';
import { AuthUser } from '../../common/decorators/auth-user.decorator'; import { AuthUser } from '../../common/decorators/auth-user.decorator';
import { AuthWorkspace } from '../../common/decorators/auth-workspace.decorator'; import { AuthWorkspace } from '../../common/decorators/auth-workspace.decorator';
import { SkipTransform } from '../../common/decorators/skip-transform.decorator'; import { SkipTransform } from '../../common/decorators/skip-transform.decorator';
import { AiChat, User, Workspace } from '@docmost/db/types/entity.types'; import {
AiChat,
AiChatMessage,
AiChatRun,
User,
Workspace,
} from '@docmost/db/types/entity.types';
import { PaginationOptions } from '@docmost/db/pagination/pagination-options'; import { PaginationOptions } from '@docmost/db/pagination/pagination-options';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo'; import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo'; import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
@@ -28,7 +35,12 @@ import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { UserThrottlerGuard } from '../../integrations/throttle/user-throttler.guard'; import { UserThrottlerGuard } from '../../integrations/throttle/user-throttler.guard';
import { AI_CHAT_THROTTLER } from '../../integrations/throttle/throttler-names'; import { AI_CHAT_THROTTLER } from '../../integrations/throttle/throttler-names';
import { FileInterceptor } from '../../common/interceptors/file.interceptor'; import { FileInterceptor } from '../../common/interceptors/file.interceptor';
import { AiChatService, AiChatStreamBody } from './ai-chat.service'; import {
AiChatRunHooks,
AiChatService,
AiChatStreamBody,
} from './ai-chat.service';
import { AiChatRunService } from './ai-chat-run.service';
import { AiTranscriptionService } from './ai-transcription.service'; import { AiTranscriptionService } from './ai-transcription.service';
import { import {
BoundChatDto, BoundChatDto,
@@ -36,7 +48,9 @@ import {
ExportChatDto, ExportChatDto,
GeneratePageTitleDto, GeneratePageTitleDto,
GetChatMessagesDto, GetChatMessagesDto,
GetRunDto,
RenameChatDto, RenameChatDto,
StopRunDto,
} from './dto/ai-chat.dto'; } from './dto/ai-chat.dto';
import { describeProviderError } from '../../integrations/ai/ai-error.util'; import { describeProviderError } from '../../integrations/ai/ai-error.util';
import { buildChatMarkdown } from './chat-markdown.util'; import { buildChatMarkdown } from './chat-markdown.util';
@@ -53,6 +67,7 @@ export class AiChatController {
constructor( constructor(
private readonly aiChatService: AiChatService, private readonly aiChatService: AiChatService,
private readonly aiChatRunService: AiChatRunService,
private readonly aiChatRepo: AiChatRepo, private readonly aiChatRepo: AiChatRepo,
private readonly aiChatMessageRepo: AiChatMessageRepo, private readonly aiChatMessageRepo: AiChatMessageRepo,
private readonly aiTranscription: AiTranscriptionService, private readonly aiTranscription: AiTranscriptionService,
@@ -149,6 +164,75 @@ export class AiChatController {
return { markdown }; return { markdown };
} }
/**
* Reconnect to the latest run of a chat (#184 phase 1). Returns the run's
* persisted lifecycle state ({ status, error, stepCount, timings, ... }) plus
* the assistant message it projects (the partial/final output) the DB is the
* source of truth, so this works for an in-flight run (the browser dropped, the
* run kept going) and a finished one alike. Owner-gated via assertOwnedChat.
* `{ run: null }` when the chat has never had a run.
*/
@HttpCode(HttpStatus.OK)
@Post('run')
async getRun(
@Body() dto: GetRunDto,
@AuthUser() user: User,
@AuthWorkspace() workspace: Workspace,
): Promise<{ run: AiChatRun | null; message: AiChatMessage | null }> {
await this.assertOwnedChat(dto.chatId, user, workspace);
const run = await this.aiChatRunService.getLatestForChat(
dto.chatId,
workspace.id,
);
if (!run) return { run: null, message: null };
const message = run.assistantMessageId
? await this.aiChatMessageRepo.findById(
run.assistantMessageId,
workspace.id,
)
: undefined;
return { run, message: message ?? null };
}
/**
* Explicitly STOP an agent run (#184 phase 1) the user pressed Stop. This is
* the ONLY thing that ends a detached run; a browser disconnect deliberately
* does not. Target by `runId` (from the streamed start metadata) or by `chatId`
* (stop whatever run is active on it). Owner-gated. Returns
* `{ stopped }` false when there was nothing active to stop.
*/
@HttpCode(HttpStatus.OK)
@Post('stop')
async stopRun(
@Body() dto: StopRunDto,
@AuthUser() user: User,
@AuthWorkspace() workspace: Workspace,
): Promise<{ stopped: boolean }> {
let runId = dto.runId;
if (!runId && !dto.chatId) {
throw new BadRequestException('runId or chatId is required');
}
if (runId) {
// Resolve the run to its chat and owner-gate via that chat.
const run = await this.aiChatRunService.getRun(runId, workspace.id);
if (!run) return { stopped: false };
await this.assertOwnedChat(run.chatId, user, workspace);
} else {
await this.assertOwnedChat(dto.chatId!, user, workspace);
const active = await this.aiChatRunService.getActiveForChat(
dto.chatId!,
workspace.id,
);
if (!active) return { stopped: false };
runId = active.id;
}
const stopped = await this.aiChatRunService.requestStop(
runId,
workspace.id,
);
return { stopped };
}
/** Rename a chat. */ /** Rename a chat. */
@HttpCode(HttpStatus.OK) @HttpCode(HttpStatus.OK)
@Post('rename') @Post('rename')
@@ -200,11 +284,20 @@ export class AiChatController {
@AuthWorkspace() workspace: Workspace, @AuthWorkspace() workspace: Workspace,
): Promise<void> { ): Promise<void> {
// A7 gate: the workspace must have AI chat explicitly enabled. // A7 gate: the workspace must have AI chat explicitly enabled.
const settings = (workspace.settings ?? {}) as { ai?: { chat?: boolean } }; const settings = (workspace.settings ?? {}) as {
ai?: { chat?: boolean; autonomousRuns?: boolean };
};
if (settings.ai?.chat !== true) { if (settings.ai?.chat !== true) {
throw new ForbiddenException('AI chat is disabled'); throw new ForbiddenException('AI chat is disabled');
} }
// #184 phase 1 flag: when ON, the turn becomes a detached, durable RUN — its
// lifecycle is tracked in ai_chat_runs, a browser disconnect no longer aborts
// it, and only an explicit /ai-chat/stop ends it. When OFF (the default) the
// turn is socket-bound exactly as before, so existing deployments are
// unaffected.
const autonomousRuns = settings.ai?.autonomousRuns === true;
const sessionId = (req.raw as { sessionId?: string }).sessionId; const sessionId = (req.raw as { sessionId?: string }).sessionId;
if (!sessionId) { if (!sessionId) {
// The chat requires an interactive session to mint loopback tokens // The chat requires an interactive session to mint loopback tokens
@@ -228,6 +321,58 @@ export class AiChatController {
// HttpException) instead of breaking mid-stream. // HttpException) instead of breaking mid-stream.
const model = await this.aiChatService.getChatModel(workspace.id, role); const model = await this.aiChatService.getChatModel(workspace.id, role);
// #184: one active run per chat. For an EXISTING chat reject a concurrent
// start with a clean 409 BEFORE hijack (the common double-submit / second-tab
// case), so the user gets JSON, not a mid-stream error. A brand-new chat
// (no chatId) cannot have a prior run, and the DB partial unique index is the
// backstop against any race that slips past this check.
if (autonomousRuns && body.chatId) {
const active = await this.aiChatRunService.getActiveForChat(
body.chatId,
workspace.id,
);
if (active) {
throw new ConflictException({
message: 'An agent run is already in progress for this chat',
code: 'A_RUN_ALREADY_ACTIVE',
});
}
}
// Run-lifecycle hooks (#184), only when the flag is on. They wrap the turn in
// a durable run whose abort is governed by the run (explicit stop), persist
// its progress, and settle its terminal status — see AiChatRunService.
const runHooks: AiChatRunHooks | undefined = autonomousRuns
? {
begin: (chatId) =>
this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
}),
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
workspace.id,
messageId,
),
onStep: (runId, stepCount) =>
void this.aiChatRunService.recordStep(
runId,
workspace.id,
stepCount,
),
onSettled: (runId, status, error) =>
this.aiChatRunService.finalizeRun(
runId,
workspace.id,
status,
error,
),
}
: undefined;
// Abort the agent loop when the client disconnects. `close` also fires on // Abort the agent loop when the client disconnects. `close` also fires on
// normal completion, so only abort when the response has not finished // normal completion, so only abort when the response has not finished
// writing (a genuine disconnect). `once` fires at most once and self-removes; // writing (a genuine disconnect). `once` fires at most once and self-removes;
@@ -242,18 +387,44 @@ export class AiChatController {
// A genuine disconnect leaves the response unfinished (unlike a normal // A genuine disconnect leaves the response unfinished (unlike a normal
// completion, which also fires `close`). Such a drop — e.g. a reverse // completion, which also fires `close`). Such a drop — e.g. a reverse
// proxy cutting the SSE mid-answer — is otherwise invisible server-side, // proxy cutting the SSE mid-answer — is otherwise invisible server-side,
// so log it here before aborting the agent loop. // so log it here.
if (!res.raw.writableEnded) { if (!res.raw.writableEnded) {
this.logger.warn( if (autonomousRuns) {
`AI chat stream: client disconnected before completion; aborting turn ` + // #184: the turn is a DETACHED run. A disconnect must NOT abort it —
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`, // the run keeps executing and persisting server-side; the client
); // reconnects via /ai-chat/run (or re-stops via /ai-chat/stop). Log only.
controller.abort(); this.logger.log(
`AI chat stream: client disconnected; run continues server-side ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
} else {
this.logger.warn(
`AI chat stream: client disconnected before completion; aborting turn ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
controller.abort();
}
} }
}; };
req.raw.once('close', onClose); req.raw.once('close', onClose);
res.raw.once('finish', () => req.raw.off('close', onClose)); res.raw.once('finish', () => req.raw.off('close', onClose));
// #184: in detached mode the turn is NOT aborted on disconnect, so the SDK's
// pipe keeps writing to a socket the client may have dropped — for the rest of
// the (continuing) run. A write to the dead socket can emit an 'error' on the
// raw response; without a listener that surfaces as an unhandled error event.
// Swallow it (the run continues server-side regardless). Legacy mode aborts on
// disconnect, so it does not need this and keeps its exact prior behavior.
if (autonomousRuns) {
res.raw.on('error', (err) => {
this.logger.debug(
`AI chat detached stream: post-disconnect socket error swallowed: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
}
// Commit to streaming: hijack so Fastify stops managing the response and // Commit to streaming: hijack so Fastify stops managing the response and
// the AI SDK can write the UI-message stream directly to the Node socket. // the AI SDK can write the UI-message stream directly to the Node socket.
res.hijack(); res.hijack();
@@ -268,15 +439,32 @@ export class AiChatController {
signal: controller.signal, signal: controller.signal,
model, model,
role, role,
// #184: present only when the flag is on; wraps the turn in a durable run.
runHooks,
}); });
} catch (err) { } catch (err) {
// Any failure AFTER hijack can no longer send a clean JSON error, so emit // Any failure AFTER hijack can no longer go through Nest's exception
// a minimal error on the raw socket if nothing has been written yet. // filter, so emit the error on the raw socket if nothing has been written
this.logger.error('AI chat stream failed', err as Error); // yet. The lost-the-race 409 (RunAlreadyActiveError -> ConflictException)
// is raised by stream() BEFORE it writes a byte, so headers are still
// unsent here: honor the HttpException's real status + body (a clean 409),
// not a blanket 500. Everything else stays a 500.
const isHttp = err instanceof HttpException;
if (!isHttp) {
this.logger.error('AI chat stream failed', err as Error);
}
if (!res.raw.headersSent) { if (!res.raw.headersSent) {
res.raw.statusCode = 500; const status = isHttp ? err.getStatus() : 500;
const payload = isHttp
? err.getResponse()
: { error: 'Internal server error' };
res.raw.statusCode = status;
res.raw.setHeader('Content-Type', 'application/json'); res.raw.setHeader('Content-Type', 'application/json');
res.raw.end(JSON.stringify({ error: 'Internal server error' })); res.raw.end(
JSON.stringify(
typeof payload === 'string' ? { message: payload } : payload,
),
);
} else if (!res.raw.writableEnded) { } else if (!res.raw.writableEnded) {
res.raw.end(); res.raw.end();
} }
@@ -57,6 +57,7 @@ describe('AiChatController.generatePageTitle', () => {
const aiChatService = { generatePageTitle: generate }; const aiChatService = { generatePageTitle: generate };
const controller = new AiChatController( const controller = new AiChatController(
aiChatService as never, aiChatService as never,
{} as never, // aiChatRunService
{} as never, {} as never,
{} as never, {} as never,
{} as never, {} as never,
@@ -3,6 +3,7 @@ import { AiModule } from '../../integrations/ai/ai.module';
import { TokenModule } from '../auth/token.module'; import { TokenModule } from '../auth/token.module';
import { AiChatController } from './ai-chat.controller'; import { AiChatController } from './ai-chat.controller';
import { AiChatService } from './ai-chat.service'; import { AiChatService } from './ai-chat.service';
import { AiChatRunService } from './ai-chat-run.service';
import { AiTranscriptionService } from './ai-transcription.service'; import { AiTranscriptionService } from './ai-transcription.service';
import { AiChatToolsService } from './tools/ai-chat-tools.service'; import { AiChatToolsService } from './tools/ai-chat-tools.service';
import { EmbeddingModule } from './embedding/embedding.module'; import { EmbeddingModule } from './embedding/embedding.module';
@@ -42,6 +43,7 @@ import { PublicShareChatToolsService } from './tools/public-share-chat-tools.ser
controllers: [AiChatController, PublicShareChatController], controllers: [AiChatController, PublicShareChatController],
providers: [ providers: [
AiChatService, AiChatService,
AiChatRunService,
AiTranscriptionService, AiTranscriptionService,
AiChatToolsService, AiChatToolsService,
PublicShareChatService, PublicShareChatService,
@@ -1,5 +1,7 @@
import { Logger } from '@nestjs/common'; import { Logger } from '@nestjs/common';
import { AiChatService } from './ai-chat.service'; import { AiChatService, AiChatRunHooks } from './ai-chat.service';
import { AiChatRunService } from './ai-chat-run.service';
import type { User, Workspace } from '@docmost/db/types/entity.types';
/** /**
* Lifecycle unit tests for AiChatService.onModuleInit (#183 crash-recovery * Lifecycle unit tests for AiChatService.onModuleInit (#183 crash-recovery
@@ -61,3 +63,99 @@ describe('AiChatService.onModuleInit (startup sweep)', () => {
expect(String(warnSpy.mock.calls[0][0])).toContain('db unavailable'); expect(String(warnSpy.mock.calls[0][0])).toContain('db unavailable');
}); });
}); });
/**
* #184 CRITICAL run-lifecycle safety net (review fix). A transient failure
* AFTER a successful beginRun but BEFORE streamText's terminal callbacks own the
* lifecycle must STILL settle the run otherwise the run row is stuck 'running'
* forever (sweepRunning only runs at startup) and the partial unique index + the
* controller pre-check 409 every future turn in that chat until a restart. Here
* we model the very first bare await after beginRun (the user-message insert)
* throwing, wiring the run hooks to a REAL AiChatRunService (mock repo) exactly
* as the controller does, and assert the run is settled to 'error' and its
* in-memory entry dropped (so a follow-up turn would NOT be 409'd).
*/
describe('AiChatService.stream run-lifecycle safety net (#184)', () => {
const user = { id: 'u1' } as User;
const workspace = { id: 'ws1' } as Workspace;
afterEach(() => jest.restoreAllMocks());
it('an exception after beginRun settles the run to error and drops the in-memory entry', async () => {
jest.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
// Real run service over a mock repo, so finalizeRun's in-memory bookkeeping
// (active.delete) is exercised for real.
const runRepo = {
insert: jest.fn().mockResolvedValue({ id: 'run-1', status: 'running' }),
update: jest.fn().mockResolvedValue({ id: 'run-1' }),
};
const runService = new AiChatRunService(runRepo as never, { isCloud: () => false } as never);
// The user-message insert (the first bare await after beginRun) throws.
const aiChatMessageRepo = {
insert: jest.fn().mockRejectedValue(new Error('insert boom')),
};
const aiChatRepo = {
// Existing chat -> chatId stays, no new-chat insert path.
findById: jest.fn().mockResolvedValue({ id: 'chat-1', creatorId: 'u1' }),
};
const service = new AiChatService(
{} as never, // ai
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
{} as never, // aiSettings
{} as never, // tools
{} as never, // mcpClients
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{} as never, // environment
);
const runHooks: AiChatRunHooks = {
begin: (chatId) =>
runService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
}),
onSettled: (runId, status, error) =>
runService.finalizeRun(runId, workspace.id, status, error),
};
await expect(
service.stream({
user,
workspace,
sessionId: 'sess',
body: {
chatId: 'chat-1',
messages: [
{ id: 'm', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
},
res: {} as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks,
}),
).rejects.toThrow('insert boom');
// The run was begun...
expect(runRepo.insert).toHaveBeenCalledTimes(1);
// ...then settled to a terminal FAILED status by the safety net...
expect(runRepo.update).toHaveBeenCalledTimes(1);
expect(runRepo.update).toHaveBeenCalledWith(
'run-1',
'ws1',
expect.objectContaining({ status: 'failed' }),
);
// ...and the in-memory entry is gone, so a follow-up turn is NOT 409'd.
expect(runService.isLocallyActive('run-1')).toBe(false);
});
});
@@ -0,0 +1,489 @@
import { ConflictException, Logger } from '@nestjs/common';
// Mock the AI SDK so we can PROVE no provider call is made for the turn we are
// about to reject. The race rejection happens at runHooks.begin(), long before
// any streamText/generateText, so these never resolve a real model.
jest.mock('ai', () => ({
streamText: jest.fn(),
generateText: jest.fn(),
convertToModelMessages: jest.fn(() => []),
stepCountIs: jest.fn(() => () => false),
}));
import { streamText, generateText } from 'ai';
import { AiChatService } from './ai-chat.service';
import { RunAlreadyActiveError } from './ai-chat-run.service';
/**
* Race-closure coverage for the "one active run per chat" guard (#184).
*
* THE BUG: two simultaneous POST /ai-chat/stream on the same chat both pass the
* controller's cheap pre-check (TOCTOU), so the loser's run-row INSERT hits the
* partial unique index. Previously that 23505 was SWALLOWED and the second turn
* streamed UNTRACKED (no runId, not stoppable). THE FIX: beginRun surfaces a
* RunAlreadyActiveError and stream() turns it into a 409 BEFORE any AI call
* the second turn never runs.
*/
describe('AiChatService.stream — concurrent-run race rejection (#184)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
const generateTextMock = generateText as unknown as jest.Mock;
beforeEach(() => {
streamTextMock.mockReset();
generateTextMock.mockReset();
});
// Minimal service whose only reachable deps before begin() are aiChatRepo
// (resolve the existing chat) — everything past begin must remain untouched.
function makeService(beginImpl: () => Promise<unknown>) {
const aiChatMessageRepo = { insert: jest.fn() };
const aiChatRepo = {
// An existing chat: stream keeps the supplied chatId and skips creation.
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const svc = new AiChatService(
{} as never, // ai
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
{} as never, // aiSettings
{} as never, // tools
{} as never, // mcpClients
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
);
const begin = jest.fn(beginImpl);
return { svc, begin, aiChatRepo, aiChatMessageRepo };
}
const baseArgs = (begin: jest.Mock) => ({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: { chatId: 'chat-1', messages: [] } as never,
res: { raw: {} } as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: {
begin,
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled: jest.fn(),
} as never,
});
it('rejects the racer with a 409 ConflictException BEFORE any AI call, and never persists an untracked turn', async () => {
// begin loses the unique-index race -> RunAlreadyActiveError.
const { svc, begin, aiChatMessageRepo } = makeService(() => {
throw new RunAlreadyActiveError('chat-1');
});
const promise = svc.stream(baseArgs(begin));
await expect(promise).rejects.toBeInstanceOf(ConflictException);
await promise.catch((err: ConflictException) => {
expect(err.getStatus()).toBe(409);
expect((err.getResponse() as { code?: string }).code).toBe(
'A_RUN_ALREADY_ACTIVE',
);
});
// The decisive assertions: the rejected racer spent NO tokens and left NO
// untracked turn behind.
expect(begin).toHaveBeenCalledTimes(1);
expect(streamTextMock).not.toHaveBeenCalled();
expect(generateTextMock).not.toHaveBeenCalled();
expect(aiChatMessageRepo.insert).not.toHaveBeenCalled();
});
});
/**
* F3 the LOAD-BEARING run-detach wiring: `effectiveSignal = handle.signal`
* after runHooks.begin, then `abortSignal: effectiveSignal` passed to streamText.
* That single line is what makes a run survive a browser disconnect (the agent
* loop's abort is governed by the RUN's signal, not the socket): a regression to
* the socket-bound signal would still pass every other test green while silently
* breaking Stop + durability. These two tests pin the exact signal streamText
* consumes on both paths.
*/
describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
// A streamText result stub: the post-call drain + pipe are no-ops here; we only
// care WHICH abortSignal streamText was handed.
function makeStreamResult() {
return {
consumeStream: jest.fn(),
pipeUIMessageStreamToResponse: jest.fn(),
};
}
// A raw-response stub sufficient for the post-streamText wiring
// (stripStreamingHopByHopHeaders binds writeHead; startSseHeartbeat registers
// close/finish listeners; flushHeaders is belt-and-braces).
function makeRes() {
return {
raw: {
writeHead: jest.fn(),
write: jest.fn(),
once: jest.fn(),
on: jest.fn(),
flushHeaders: jest.fn(),
writableEnded: false,
destroyed: false,
},
};
}
// Wire only the deps reached on the way to streamText: resolve the existing
// chat, persist the user + seed the assistant row, load (empty) history, the
// admin settings, an empty external toolset + Docmost toolset.
function makeService() {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const aiChatMessageRepo = {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
const mcpClients = {
toolsFor: jest.fn(async () => ({
tools: {},
clients: [],
outcomes: [],
instructions: [],
})),
};
const svc = new AiChatService(
{} as never, // ai
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
aiSettings as never,
tools as never,
mcpClients as never,
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo (openPage undefined -> never touched)
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
);
return { svc };
}
const body = {
chatId: 'chat-1',
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
};
beforeEach(() => {
streamTextMock.mockReset();
streamTextMock.mockImplementation(() => makeStreamResult());
jest
.spyOn(Logger.prototype, 'log')
.mockImplementation(() => undefined as never);
});
afterEach(() => jest.restoreAllMocks());
it('happy path (run-wrapped): streamText is driven with abortSignal === handle.signal (the RUN signal, NOT the socket)', async () => {
const { svc } = makeService();
const runController = new AbortController();
const runSignal = runController.signal;
const socketSignal = new AbortController().signal;
const begin = jest.fn(async () => ({ runId: 'run-1', signal: runSignal }));
await svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: makeRes() as never,
signal: socketSignal,
model: {} as never,
role: null,
runHooks: {
begin,
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled: jest.fn(),
} as never,
});
expect(begin).toHaveBeenCalledTimes(1);
expect(streamTextMock).toHaveBeenCalledTimes(1);
// THE assertion: the agent loop's abort is wired to the RUN, so a browser
// disconnect (which aborts only `socketSignal`) cannot end the turn.
expect(streamTextMock.mock.calls[0][0].abortSignal).toBe(runSignal);
expect(streamTextMock.mock.calls[0][0].abortSignal).not.toBe(socketSignal);
});
it('legacy path (no runHooks): streamText is driven with the SOCKET signal', async () => {
const { svc } = makeService();
const socketSignal = new AbortController().signal;
await svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: makeRes() as never,
signal: socketSignal,
model: {} as never,
role: null,
// No runHooks -> the turn stays socket-bound (flag off / default).
});
expect(streamTextMock).toHaveBeenCalledTimes(1);
expect(streamTextMock.mock.calls[0][0].abortSignal).toBe(socketSignal);
});
/**
* F9 streamText's TERMINAL callbacks carry the #184 run lifecycle:
* onStepFinish -> runHooks.onStep(runId, stepCount)
* onFinish -> runHooks.onSettled(runId, 'completed') (dominant path)
* onAbort -> runHooks.onSettled(runId, 'aborted')
* onError -> runHooks.onSettled(runId, 'error', cause)
* makeStreamResult() ignores the streamText options, so these callbacks never
* fire on their own a regression in this wiring (esp. the success path) would
* strand the run with NO test catching it. Here we CAPTURE the options streamText
* was handed and invoke each callback with the real wiring, asserting the run
* hooks fire with the right args.
*/
// Drive stream() to the point streamText is called, capturing the options object
// (which carries onStepFinish/onFinish/onError/onAbort) and the run hooks.
async function captureStreamCallbacks() {
const { svc } = makeService();
let capturedOpts: any;
streamTextMock.mockImplementation((opts: any) => {
capturedOpts = opts;
return makeStreamResult();
});
const runHooks = {
begin: jest.fn(async () => ({
runId: 'run-1',
signal: new AbortController().signal,
})),
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled: jest.fn(),
};
await svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: makeRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: runHooks as never,
});
expect(capturedOpts).toBeDefined();
return { capturedOpts, runHooks };
}
it('F9: onStepFinish bumps the run step count, onFinish settles the run "completed" (the dominant autonomous-run path)', async () => {
const { capturedOpts, runHooks } = await captureStreamCallbacks();
// A finished step -> onStep(runId, finishedStepCount).
capturedOpts.onStepFinish({ text: 'step one', toolCalls: [], content: [] });
expect(runHooks.onStep).toHaveBeenCalledWith('run-1', 1);
capturedOpts.onStepFinish({ text: 'step two', toolCalls: [], content: [] });
expect(runHooks.onStep).toHaveBeenLastCalledWith('run-1', 2);
// The success terminal callback settles the run.
await capturedOpts.onFinish({
text: 'done',
finishReason: 'stop',
totalUsage: {},
usage: {},
steps: [],
});
expect(runHooks.onSettled).toHaveBeenCalledWith('run-1', 'completed');
});
it('F9: onAbort settles the run "aborted"', async () => {
jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as never);
const { capturedOpts, runHooks } = await captureStreamCallbacks();
await capturedOpts.onAbort({ steps: [] });
expect(runHooks.onSettled).toHaveBeenCalledWith('run-1', 'aborted');
});
it('F9: onError settles the run "error" carrying the provider cause', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as never);
const { capturedOpts, runHooks } = await captureStreamCallbacks();
await capturedOpts.onError({ error: new Error('provider exploded') });
expect(runHooks.onSettled).toHaveBeenCalledWith(
'run-1',
'error',
expect.stringContaining('provider exploded'),
);
});
});
/**
* F14 the begin-failure RESILIENCE branch (the `else` of the run-race guard).
*
* stream() wraps runHooks.begin in try/catch with TWO branches:
* - RunAlreadyActiveError -> 409 ConflictException (pinned above).
* - ANY OTHER begin failure -> SWALLOW + continue UNTRACKED on the socket signal
* (legacy fallback): it logs "...streaming without run tracking", leaves
* `effectiveSignal = signal` (runId undefined) and serves the turn anyway.
*
* The contract: a transient beginRun failure (e.g. a non-unique DB error inserting
* the run row) must STILL serve the user's turn it must NOT re-throw and must NOT
* be misclassified as a 409. A regression that re-threw here would break EVERY turn
* on a begin failure with nothing to catch it. This branch is otherwise undriven by
* any spec, so it is pinned here SEPARATELY from the 409 path: a plain begin error
* proceeds to streamText with the SOCKET signal and still persists the user turn.
*/
describe('AiChatService.stream — begin-failure resilience / legacy fallback (#184 F14)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
function makeStreamResult() {
return {
consumeStream: jest.fn(),
pipeUIMessageStreamToResponse: jest.fn(),
};
}
function makeRes() {
return {
raw: {
writeHead: jest.fn(),
write: jest.fn(),
once: jest.fn(),
on: jest.fn(),
flushHeaders: jest.fn(),
writableEnded: false,
destroyed: false,
},
};
}
// Same harness as the F3 abortSignal block, but it also exposes
// aiChatMessageRepo so we can assert the user turn IS persisted (the turn really
// streamed) despite begin() blowing up.
function makeService() {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const aiChatMessageRepo = {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
const mcpClients = {
toolsFor: jest.fn(async () => ({
tools: {},
clients: [],
outcomes: [],
instructions: [],
})),
};
const svc = new AiChatService(
{} as never, // ai
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
aiSettings as never,
tools as never,
mcpClients as never,
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
);
return { svc, aiChatMessageRepo };
}
const body = {
chatId: 'chat-1',
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
};
beforeEach(() => {
streamTextMock.mockReset();
streamTextMock.mockImplementation(() => makeStreamResult());
jest
.spyOn(Logger.prototype, 'log')
.mockImplementation(() => undefined as never);
});
afterEach(() => jest.restoreAllMocks());
it('a PLAIN begin() failure (NOT RunAlreadyActiveError) does NOT 409 — it swallows, logs, and streams the turn UNTRACKED on the socket signal', async () => {
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
const { svc, aiChatMessageRepo } = makeService();
const socketSignal = new AbortController().signal;
// A transient, NON-race begin failure (e.g. a non-unique DB error inserting
// the run row). This is the `else` branch of the begin try/catch.
const begin = jest.fn(async () => {
throw new Error('insert failed');
});
const promise = svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: makeRes() as never,
signal: socketSignal,
model: {} as never,
role: null,
runHooks: {
begin,
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled: jest.fn(),
} as never,
});
// The turn proceeds: NO throw at all (in particular NOT a 409).
await expect(promise).resolves.toBeUndefined();
expect(begin).toHaveBeenCalledTimes(1);
// The resilience branch logged the legacy-fallback warning.
expect(errorSpy).toHaveBeenCalledWith(
expect.stringContaining('streaming without run tracking'),
expect.anything(),
);
// The turn really streamed: the user message was persisted and streamText ran.
expect(aiChatMessageRepo.insert).toHaveBeenCalled();
expect(streamTextMock).toHaveBeenCalledTimes(1);
// The decisive wiring: with no run handle, the fallback uses the SOCKET signal
// (effectiveSignal = signal, runId undefined) — not a run-bound signal.
expect(streamTextMock.mock.calls[0][0].abortSignal).toBe(socketSignal);
});
});
@@ -453,6 +453,12 @@ describe('chatStreamMetadata', () => {
}); });
}); });
it('attaches the runId on the start part when a run wraps the turn (#184)', () => {
expect(
chatStreamMetadata({ type: 'start' }, 'chat-1', undefined, 'run-1'),
).toEqual({ chatId: 'chat-1', runId: 'run-1' });
});
it('returns the CUMULATIVE step usage passed in for the finish-step part', () => { it('returns the CUMULATIVE step usage passed in for the finish-step part', () => {
// finish-step usage is per-step in v6; the caller accumulates and passes the // finish-step usage is per-step in v6; the caller accumulates and passes the
// running sum, which this just wraps. // running sum, which this just wraps.
File diff suppressed because it is too large Load Diff
@@ -43,6 +43,30 @@ export class BoundChatDto {
pageId: string; pageId: string;
} }
/**
* Reconnect to the latest run of a chat (#184): fetch its persisted lifecycle
* state (and the assistant message it projects) for an in-flight or finished run.
*/
export class GetRunDto {
@IsString()
chatId: string;
}
/**
* Explicitly STOP an agent run (#184): the user pressed Stop distinct from a
* browser disconnect, which never stops a run. Either the run id (preferred, from
* the streamed start metadata) or the chat id (stop whatever run is active on it).
*/
export class StopRunDto {
@IsOptional()
@IsString()
runId?: string;
@IsOptional()
@IsString()
chatId?: string;
}
/** Export a chat to Markdown (#183). `lang` localizes the few fixed /** Export a chat to Markdown (#183). `lang` localizes the few fixed
* role/tool-action labels; defaults to English server-side. */ * role/tool-action labels; defaults to English server-side. */
export class ExportChatDto { export class ExportChatDto {
@@ -610,6 +610,63 @@ describe('AiAgentRolesService guards', () => {
expect(repo.insert.mock.calls[0][0].name).toBe('Researcher (2)'); expect(repo.insert.mock.calls[0][0].name).toBe('Researcher (2)');
}); });
it('createdRoles lists the installed role (no renamedTo when not renamed)', async () => {
const { service } = makeImportService({});
const res = await service.importFromCatalog('ws-1', 'u1', dto());
expect(res.createdRoles).toEqual([
{ slug: 'researcher', name: 'Researcher' },
]);
expect(res.skippedRoles).toEqual([]);
});
it('createdRoles carries renamedTo on a rename', async () => {
const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
const { service } = makeImportService({ existing });
const res = await service.importFromCatalog(
'ws-1',
'u1',
dto({ conflict: 'rename' }),
);
expect(res.createdRoles).toEqual([
{ slug: 'researcher', name: 'Researcher', renamedTo: 'Researcher (2)' },
]);
expect(res.skippedRoles).toEqual([]);
});
it('skippedRoles: already-installed slug carries reason "already-installed"', async () => {
const existing = [
makeRow({
id: 'r-existing',
name: 'Old researcher',
source: { slug: 'researcher', language: 'en', version: 1 } as never,
}),
];
const { service } = makeImportService({ existing });
const res = await service.importFromCatalog('ws-1', 'u1', dto());
expect(res.skippedRoles).toEqual([
{
slug: 'researcher',
name: 'Researcher',
reason: 'already-installed',
},
]);
expect(res.createdRoles).toEqual([]);
});
it('skippedRoles: a name collision under conflict:skip carries reason "name-conflict"', async () => {
const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
const { service } = makeImportService({ existing });
const res = await service.importFromCatalog(
'ws-1',
'u1',
dto({ conflict: 'skip' }),
);
expect(res.skippedRoles).toEqual([
{ slug: 'researcher', name: 'Researcher', reason: 'name-conflict' },
]);
expect(res.createdRoles).toEqual([]);
});
it('dto.slugs filters; an unknown slug becomes an error entry', async () => { it('dto.slugs filters; an unknown slug becomes an error entry', async () => {
const { service, repo } = makeImportService({ const { service, repo } = makeImportService({
bundleRoles: [catalogRole()], bundleRoles: [catalogRole()],
@@ -677,6 +734,15 @@ describe('AiAgentRolesService guards', () => {
// 'a' converged on the concurrent install (skip); 'b' imported; no errors. // 'a' converged on the concurrent install (skip); 'b' imported; no errors.
expect(res).toMatchObject({ created: 1, skipped: 1, renamed: 0 }); expect(res).toMatchObject({ created: 1, skipped: 1, renamed: 0 });
expect(res.errors).toEqual([]); expect(res.errors).toEqual([]);
// The per-role list records 'a' as an already-installed skip (the UI reads
// skippedRoles, not the counter, to render its plaque — assert the array,
// not just the count).
expect(res.skippedRoles).toContainEqual({
slug: 'a',
name: 'A',
reason: 'already-installed',
});
expect(res.createdRoles.map((r) => r.slug)).toEqual(['b']);
// Both inserts were attempted (the batch did not abort on the 23505). // Both inserts were attempted (the batch did not abort on the 23505).
expect(repo.insert).toHaveBeenCalledTimes(2); expect(repo.insert).toHaveBeenCalledTimes(2);
}); });
@@ -305,6 +305,16 @@ export class AiAgentRolesService {
skipped: number; skipped: number;
renamed: number; renamed: number;
errors: { slug: string; message: string }[]; errors: { slug: string; message: string }[];
// Per-role lists alongside the counters (kept for back-compat). The redesigned
// catalog UI needs the actual roles — which were created (and any rename) and
// which were skipped and why — to render an inline result plaque with the
// conflicting role's name and a "Rename & install" affordance.
createdRoles: { slug: string; name: string; renamedTo?: string }[];
skippedRoles: {
slug: string;
name: string;
reason: 'name-conflict' | 'already-installed';
}[];
}> { }> {
const { file, versions } = await this.loadBundleById( const { file, versions } = await this.loadBundleById(
dto.bundleId, dto.bundleId,
@@ -312,6 +322,13 @@ export class AiAgentRolesService {
); );
const errors: { slug: string; message: string }[] = []; const errors: { slug: string; message: string }[] = [];
const createdRoles: { slug: string; name: string; renamedTo?: string }[] =
[];
const skippedRoles: {
slug: string;
name: string;
reason: 'name-conflict' | 'already-installed';
}[] = [];
// Resolve the selected catalog roles (honor dto.slugs; flag unknown ones). // Resolve the selected catalog roles (honor dto.slugs; flag unknown ones).
let selected = file.roles; let selected = file.roles;
@@ -351,16 +368,27 @@ export class AiAgentRolesService {
// Already installed from the catalog in THIS language => skip (use // Already installed from the catalog in THIS language => skip (use
// update-from-catalog). A different language of the same slug still imports. // update-from-catalog). A different language of the same slug still imports.
const installKey = `${role.slug}:${dto.language}`; const installKey = `${role.slug}:${dto.language}`;
const originalName = role.name.trim();
if (installedKeys.has(installKey)) { if (installedKeys.has(installKey)) {
skipped++; skipped++;
skippedRoles.push({
slug: role.slug,
name: originalName,
reason: 'already-installed',
});
continue; continue;
} }
let name = role.name.trim(); let name = originalName;
let didRename = false; let didRename = false;
if (takenNames.has(name.toLowerCase())) { if (takenNames.has(name.toLowerCase())) {
if (dto.conflict === 'skip') { if (dto.conflict === 'skip') {
skipped++; skipped++;
skippedRoles.push({
slug: role.slug,
name: originalName,
reason: 'name-conflict',
});
continue; continue;
} }
// conflict === 'rename': find a free " (N)" suffix. // conflict === 'rename': find a free " (N)" suffix.
@@ -380,6 +408,11 @@ export class AiAgentRolesService {
}); });
created++; created++;
if (didRename) renamed++; if (didRename) renamed++;
createdRoles.push({
slug: role.slug,
name: originalName,
...(didRename ? { renamedTo: name } : {}),
});
takenNames.add(name.toLowerCase()); takenNames.add(name.toLowerCase());
installedKeys.add(installKey); installedKeys.add(installKey);
} catch (err) { } catch (err) {
@@ -391,6 +424,11 @@ export class AiAgentRolesService {
// skipped (already installed) and continue; do NOT abort or error. // skipped (already installed) and continue; do NOT abort or error.
if (isSourceUniqueViolation(err)) { if (isSourceUniqueViolation(err)) {
skipped++; skipped++;
skippedRoles.push({
slug: role.slug,
name: originalName,
reason: 'already-installed',
});
installedKeys.add(installKey); installedKeys.add(installKey);
continue; continue;
} }
@@ -407,7 +445,7 @@ export class AiAgentRolesService {
} }
} }
return { created, skipped, renamed, errors }; return { created, skipped, renamed, errors, createdRoles, skippedRoles };
} }
/** /**
@@ -539,3 +539,115 @@ describe('AiChatToolsService model-friendly input validation (#190)', () => {
expect(result.error?.message).toContain('parameter "pageId": missing (required)'); expect(result.error?.message).toContain('parameter "pageId": missing (required)');
}); });
}); });
/**
* #294 F1 the contract-parity test introspects only the ADVERTISED schema keys
* (buildShape), not the execute bodies. Most execs are unchanged pass-throughs,
* but two wirings actually CHANGED in the migration and are otherwise untested:
* - movePage now forwards the newly-added optional `position` field to the
* client (client.movePage(pageId, parentPageId, position));
* - the table trio unified its `tableRef` param to `table` and must forward it
* positionally. A field destructured under the wrong name would silently pass
* `undefined` to the client (execute is `any`-cast, so tsc won't catch it).
*/
describe('AiChatToolsService #294 changed execute wirings', () => {
const calls: Record<string, unknown[][]> = {
movePage: [],
tableInsertRow: [],
tableDeleteRow: [],
tableUpdateCell: [],
};
const fakeClient: Partial<DocmostClientLike> = {
movePage: (...args: unknown[]) => {
calls.movePage.push(args);
return Promise.resolve({ success: true });
},
tableInsertRow: (...args: unknown[]) => {
calls.tableInsertRow.push(args);
return Promise.resolve({ ok: true });
},
tableDeleteRow: (...args: unknown[]) => {
calls.tableDeleteRow.push(args);
return Promise.resolve({ ok: true });
},
tableUpdateCell: (...args: unknown[]) => {
calls.tableUpdateCell.push(args);
return Promise.resolve({ ok: true });
},
};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
generateCollabToken: jest.fn().mockResolvedValue('collab-token'),
};
let service: AiChatToolsService;
beforeEach(() => {
for (const k of Object.keys(calls)) calls[k].length = 0;
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue(
mockLoaded(function () {
return fakeClient as DocmostClientLike;
} as unknown as loader.DocmostClientCtor),
);
service = new AiChatToolsService(
tokenServiceStub as never,
{} as never,
{} as never,
{} as never,
{} as never,
{
asSink: () => ({ put: jest.fn(), has: jest.fn(), evict: jest.fn() }),
} as never,
);
});
afterEach(() => jest.restoreAllMocks());
const buildTools = () =>
service.forUser(
{ id: 'user-1', email: 'u@example.com', workspaceId: 'ws-1' } as never,
'session-1',
'ws-1',
'chat-1',
);
it('movePage forwards the optional position to the client', async () => {
const tools = await buildTools();
await tools.movePage.execute(
{ pageId: 'p1', parentPageId: 'parent1', position: 'a5' } as never,
{} as never,
);
expect(calls.movePage).toEqual([['p1', 'parent1', 'a5']]);
});
it('movePage passes undefined position and null parent when omitted (unchanged behavior)', async () => {
const tools = await buildTools();
await tools.movePage.execute({ pageId: 'p2' } as never, {} as never);
expect(calls.movePage).toEqual([['p2', null, undefined]]);
});
it('tableInsertRow forwards the unified `table` param positionally', async () => {
const tools = await buildTools();
await tools.tableInsertRow.execute(
{ pageId: 'p1', table: '#0', cells: ['a', 'b'], index: 2 } as never,
{} as never,
);
expect(calls.tableInsertRow).toEqual([['p1', '#0', ['a', 'b'], 2]]);
});
it('tableDeleteRow forwards `table` positionally', async () => {
const tools = await buildTools();
await tools.tableDeleteRow.execute(
{ pageId: 'p1', table: '#0', index: 1 } as never,
{} as never,
);
expect(calls.tableDeleteRow).toEqual([['p1', '#0', 1]]);
});
it('tableUpdateCell forwards `table` positionally', async () => {
const tools = await buildTools();
await tools.tableUpdateCell.execute(
{ pageId: 'p1', table: '#0', row: 1, col: 2, text: 'x' } as never,
{} as never,
);
expect(calls.tableUpdateCell).toEqual([['p1', '#0', 1, 2, 'x']]);
});
});
@@ -316,50 +316,27 @@ export class AiChatToolsService {
execute: async () => resolveCurrentPageResult(openedPage), execute: async () => resolveCurrentPageResult(openedPage),
}), }),
getPage: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: // The execute body keeps this layer's { title, markdown } projection.
'Fetch a single page as Markdown by its page id. Returns the page ' + getPage: sharedTool(sharedToolSpecs.getPage, async ({ pageId }) => {
'title and its Markdown content. Inline <span data-comment-id> tags ' + // getPage(pageId) -> { data: filterPage(page, markdown), success }.
'in the markdown are comment highlight anchors (also present for ' + const result = await client.getPage(pageId);
'RESOLVED threads) — treat them as markup, not page text.', const data = (result?.data ?? {}) as {
inputSchema: modelFriendlyInput({ title?: string;
pageId: z.string().describe('The id (or slugId) of the page.'), content?: string;
}), };
execute: async ({ pageId }) => { return {
// getPage(pageId) -> { data: filterPage(page, markdown), success }. title: data.title ?? '',
const result = await client.getPage(pageId); markdown: typeof data.content === 'string' ? data.content : '',
const data = (result?.data ?? {}) as { };
title?: string;
content?: string;
};
return {
title: data.title ?? '',
markdown: typeof data.content === 'string' ? data.content : '',
};
},
}), }),
// --- WRITE tools (all reversible — history/trash; §6.5 / D3) --- // --- WRITE tools (all reversible — history/trash; §6.5 / D3) ---
createPage: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: createPage: sharedTool(
'Create a new page with a Markdown body in a space, optionally under ' + sharedToolSpecs.createPage,
'a parent page. Returns the new page id and title. Reversible: a page ' + async ({ title, content, spaceId, parentPageId }) => {
'can be moved to trash later.',
inputSchema: modelFriendlyInput({
title: z.string().describe('The title of the new page.'),
content: z
.string()
.describe('The page body as Markdown (may be empty).'),
spaceId: z
.string()
.describe('The id of the space to create the page in.'),
parentPageId: z
.string()
.optional()
.describe('Optional parent page id to nest the new page under.'),
}),
execute: async ({ title, content, spaceId, parentPageId }) => {
// createPage(title, content, spaceId, parentPageId?) -> // createPage(title, content, spaceId, parentPageId?) ->
// { data: filterPage(page, markdown), success }. // { data: filterPage(page, markdown), success }.
const result = await client.createPage( const result = await client.createPage(
@@ -375,7 +352,7 @@ export class AiChatToolsService {
}; };
return { id: data.id ?? data.slugId, title: data.title ?? title }; return { id: data.id ?? data.slugId, title: data.title ?? title };
}, },
}), ),
updatePageContent: tool({ updatePageContent: tool({
description: description:
@@ -399,115 +376,46 @@ export class AiChatToolsService {
}, },
}), }),
renamePage: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: renamePage: sharedTool(
"Rename a page (change its title only; the body is untouched). " + sharedToolSpecs.renamePage,
'Reversible: rename back at any time.', async ({ pageId, title }) => {
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to rename.'),
title: z.string().describe('The new title.'),
}),
execute: async ({ pageId, title }) => {
// renamePage(pageId, title) -> { success, pageId, title }. // renamePage(pageId, title) -> { success, pageId, title }.
await client.renamePage(pageId, title); await client.renamePage(pageId, title);
return { pageId, title }; return { pageId, title };
}, },
}), ),
movePage: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: // The shared schema adds the optional `position` field this layer lacked
'Move a page under a new parent page, or to the space root when no ' + // before; the execute now forwards it (the client already accepted it).
'parent is given. Reversible: move it back at any time.', movePage: sharedTool(
inputSchema: modelFriendlyInput({ sharedToolSpecs.movePage,
pageId: z.string().describe('The id of the page to move.'), async ({ pageId, parentPageId, position }) => {
parentPageId: z
.string()
.nullable()
.optional()
.describe(
'Target parent page id. Null/omitted moves the page to the ' +
'space root.',
),
}),
execute: async ({ pageId, parentPageId }) => {
// movePage(pageId, parentPageId, position?) -> raw move response. // movePage(pageId, parentPageId, position?) -> raw move response.
await client.movePage(pageId, parentPageId ?? null); await client.movePage(pageId, parentPageId ?? null, position);
return { pageId, parentPageId: parentPageId ?? null, moved: true }; return { pageId, parentPageId: parentPageId ?? null, moved: true };
}, },
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// GUARDRAIL (§14 H4) preserved: the shared schema exposes ONLY pageId, so
// permanentlyDelete/forceDelete are never part of the input and can never
// be forwarded — the agent physically cannot permanently delete a page.
deletePage: sharedTool(sharedToolSpecs.deletePage, async ({ pageId }) => {
// deletePage(pageId) hits POST /pages/delete with { pageId } only,
// which is the soft-delete (trash) path on the server.
await client.deletePage(pageId);
return { pageId, trashed: true };
}), }),
deletePage: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: // This layer keeps only its own execute-side guards (require a selection
'Move a page to the trash (SOFT delete only — fully reversible; the ' + // for a top-level comment; reject suggestedText on a reply / without a
'page can be restored from trash). This NEVER permanently deletes.', // selection) — the schema+description are shared.
inputSchema: modelFriendlyInput({ createComment: sharedTool(
pageId: z.string().describe('The id of the page to move to trash.'), sharedToolSpecs.createComment,
}), async ({
// GUARDRAIL (§14 H4): the only field ever passed to the client is
// pageId. permanentlyDelete/forceDelete are not part of the schema and
// are never forwarded, so the agent physically cannot permanently
// delete a page through this tool.
execute: async ({ pageId }) => {
// deletePage(pageId) hits POST /pages/delete with { pageId } only,
// which is the soft-delete (trash) path on the server.
await client.deletePage(pageId);
return { pageId, trashed: true };
},
}),
// INTENTIONAL per-transport divergence (not shared): the description is
// tuned for the in-app agent (e.g. "retry with a corrected EXACT selection"
// and "Reversible via the comment UI"); the standalone MCP `create_comment`
// keeps its own wording. Kept per-layer.
createComment: tool({
description:
'Add an INLINE comment to a page, or reply to an existing top-level ' +
'comment (one level only — the backend rejects replies to replies). ' +
'The comment is anchored inline to the given exact `selection` text ' +
'(which gets highlighted); page-level comments are NOT supported. A ' +
"new top-level comment REQUIRES a `selection`. Replies inherit the " +
"parent's anchor and take no selection. If the call fails with a " +
'"selection not found" error, retry with a corrected EXACT selection ' +
'copied verbatim from a single paragraph/block. You may also attach a ' +
'`suggestedText` proposing a replacement for the `selection` (a human ' +
'applies it from the UI); when set, the `selection` must occur exactly ' +
'once in the page. Reversible via the comment UI.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to comment on.'),
content: z.string().describe('The comment body as Markdown.'),
selection: z
.string()
.min(1)
.max(250)
.optional()
.describe(
'EXACT contiguous text from a SINGLE paragraph/block to anchor ' +
'(highlight) the comment on (<=250 chars, avoid spanning across ' +
'formatting boundaries). Required for a new top-level comment; ' +
'omit only when replying via parentCommentId.',
),
parentCommentId: z
.string()
.optional()
.describe(
'Optional id of a TOP-LEVEL comment to reply to (one level ' +
'of replies only).',
),
suggestedText: z
.string()
.min(1)
.max(2000)
.optional()
.describe(
'Optional proposed replacement (PLAIN TEXT) for the `selection`, ' +
'applied by a human via the UI (never auto-applied). REQUIRES a ' +
'`selection`; NOT allowed on a reply. When set, the `selection` ' +
'must be UNIQUE in the page — expand it with surrounding context ' +
'(still <=250 chars) if it occurs more than once, or the call is ' +
'refused.',
),
}),
execute: async ({
pageId, pageId,
content, content,
selection, selection,
@@ -548,26 +456,17 @@ export class AiChatToolsService {
const data = (result?.data ?? {}) as { id?: string }; const data = (result?.data ?? {}) as { id?: string };
return { commentId: data.id, pageId }; return { commentId: data.id, pageId };
}, },
}), ),
resolveComment: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: resolveComment: sharedTool(
'Resolve or reopen a top-level comment thread (reversible — toggle ' + sharedToolSpecs.resolveComment,
'the resolved flag). Only top-level comments can be resolved.', async ({ commentId, resolved }) => {
inputSchema: modelFriendlyInput({
commentId: z
.string()
.describe('The id of the top-level comment to resolve/reopen.'),
resolved: z
.boolean()
.describe('true to resolve the thread, false to reopen it.'),
}),
execute: async ({ commentId, resolved }) => {
// resolveComment(commentId, resolved) -> { success, commentId, resolved }. // resolveComment(commentId, resolved) -> { success, commentId, resolved }.
await client.resolveComment(commentId, resolved); await client.resolveComment(commentId, resolved);
return { commentId, resolved }; return { commentId, resolved };
}, },
}), ),
// --- READ tools (added) --- // --- READ tools (added) ---
@@ -585,33 +484,12 @@ export class AiChatToolsService {
// hierarchy mode but is worded for the in-app agent; the standalone MCP // hierarchy mode but is worded for the in-app agent; the standalone MCP
// `list_pages` carries its own wording. Kept per-layer so each side tunes // `list_pages` carries its own wording. Kept per-layer so each side tunes
// its own guidance. // its own guidance.
listPages: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: listPages: sharedTool(
'List the most recent pages, optionally scoped to a single space. ' + sharedToolSpecs.listPages,
'Returns a bounded list (default 50, max 100). Pass tree:true (with ' + async ({ spaceId, limit, tree }) =>
"spaceId) to instead get the space's full page hierarchy as a nested tree.",
inputSchema: modelFriendlyInput({
spaceId: z
.string()
.optional()
.describe('Optional space id to scope the listing to.'),
limit: z
.number()
.int()
.min(1)
.max(100)
.optional()
.describe('Maximum number of pages (1-100).'),
tree: z
.boolean()
.optional()
.describe(
'When true, return the full page hierarchy of the given space as a nested tree (children arrays) instead of the recent-pages flat list. Requires spaceId; ignores limit.',
),
}),
execute: async ({ spaceId, limit, tree }) =>
await client.listPages(spaceId, limit, tree), await client.listPages(spaceId, limit, tree),
}), ),
listSidebarPages: tool({ listSidebarPages: tool({
description: description:
@@ -656,41 +534,34 @@ export class AiChatToolsService {
}), }),
), ),
// NOT shared (kept inline): the MCP tool name `table_get` is noun-first
// while this key is `getTable` (verb-first), breaking the
// snake_case(inAppKey) convention the shared registry enforces. Its
// reference parameter is still named `table` (was `tableRef`) so it matches
// the migrated table row/cell tools below.
getTable: tool({ getTable: tool({
description: description:
'Read a table as a matrix of cell texts (plus a parallel cellIds ' + 'Read a table as a matrix of cell texts (plus a parallel cellIds ' +
'matrix so cells can be addressed for rich edits).', 'matrix so cells can be addressed for rich edits).',
inputSchema: modelFriendlyInput({ inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'), pageId: z.string().describe('The id of the page.'),
tableRef: z table: z
.string() .string()
.describe( .describe(
'"#<index>" from getOutline, or a block id of any node inside ' + '"#<index>" from the page outline, or a block id of any node ' +
'the table.', 'inside the table.',
), ),
}), }),
execute: async ({ pageId, tableRef }) => execute: async ({ pageId, table }) =>
await client.getTable(pageId, tableRef), await client.getTable(pageId, table),
}), }),
listComments: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: listComments: sharedTool(
'List comments on a page in one call. By DEFAULT only ACTIVE ' + sharedToolSpecs.listComments,
'threads are returned; resolved threads (a resolved top-level ' + async ({ pageId, includeResolved }) =>
'comment and all its replies) are hidden and their count reported ' +
'as `resolvedThreadsHidden` so you can re-query with ' +
'`includeResolved: true` to see everything. Returns ' +
'`{ items, resolvedThreadsHidden }`. Content is returned as Markdown.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
includeResolved: z
.boolean()
.optional()
.describe('default only active threads; true — include resolved'),
}),
execute: async ({ pageId, includeResolved }) =>
await client.listComments(pageId, includeResolved), await client.listComments(pageId, includeResolved),
}), ),
getComment: tool({ getComment: tool({
description: 'Fetch a single comment by id (content as Markdown).', description: 'Fetch a single comment by id (content as Markdown).',
@@ -700,26 +571,12 @@ export class AiChatToolsService {
execute: async ({ commentId }) => await client.getComment(commentId), execute: async ({ commentId }) => await client.getComment(commentId),
}), }),
checkNewComments: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: checkNewComments: sharedTool(
'Find new comments across a space (optionally scoped to a subtree) ' + sharedToolSpecs.checkNewComments,
'created after a given timestamp.', async ({ spaceId, since, parentPageId }) =>
inputSchema: modelFriendlyInput({
spaceId: z.string().describe('The id of the space to scan.'),
since: z
.string()
.describe('An ISO-8601 timestamp; only comments created after it.'),
parentPageId: z
.string()
.optional()
.describe(
'Optional page id to scope the scan to that page and its ' +
'descendants.',
),
}),
execute: async ({ spaceId, since, parentPageId }) =>
await client.checkNewComments(spaceId, since, parentPageId), await client.checkNewComments(spaceId, since, parentPageId),
}), ),
listShares: sharedTool( listShares: sharedTool(
sharedToolSpecs.listShares, sharedToolSpecs.listShares,
@@ -749,19 +606,14 @@ export class AiChatToolsService {
await client.diffPageVersions(pageId, from, to), await client.diffPageVersions(pageId, from, to),
), ),
exportPageMarkdown: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: exportPageMarkdown: sharedTool(
'Export a page to a single self-contained Docmost-flavoured ' + sharedToolSpecs.exportPageMarkdown,
'Markdown file (meta + body + comment threads). Lossless round-trip ' + async ({ pageId }) => {
'with importPageMarkdown.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to export.'),
}),
execute: async ({ pageId }) => {
const markdown = await client.exportPageMarkdown(pageId); const markdown = await client.exportPageMarkdown(pageId);
return { markdown }; return { markdown };
}, },
}), ),
// --- WRITE tools (added; reversible via page history/trash) --- // --- WRITE tools (added; reversible via page history/trash) ---
@@ -811,28 +663,12 @@ export class AiChatToolsService {
async ({ pageId, nodeId }) => await client.deleteNode(pageId, nodeId), async ({ pageId, nodeId }) => await client.deleteNode(pageId, nodeId),
), ),
updatePageJson: tool({ // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
description: // The execute body keeps this layer's content normalization (parity with
"Replace a page's body with a full ProseMirror document — a full " + // the standalone MCP server, index.ts update_page_json).
'overwrite — and/or update its title. Minimal example content: ' + updatePageJson: sharedTool(
'{"type":"doc","content":[{"type":"paragraph","content":' + sharedToolSpecs.updatePageJson,
'[{"type":"text","text":"Hi"}]}]}. The content arg may be a JSON ' + async ({ pageId, content, title }) => {
'object or a JSON string (both accepted). Omit content for a ' +
'title-only update. Reversible: the previous version is kept in page ' +
'history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to update.'),
content: z
.any()
.optional()
.describe(
'Full ProseMirror doc {"type":"doc","content":[...]} (JSON ' +
'object or JSON string); omit for a title-only update.',
),
title: z.string().optional().describe('Optional new title.'),
}),
execute: async ({ pageId, content, title }) => {
// Parity with the standalone MCP server (index.ts update_page_json):
// undefined/null pass through as undefined (title-only / no-op); any // undefined/null pass through as undefined (title-only / no-op); any
// string is JSON.parsed (so an empty string "" throws, matching the // string is JSON.parsed (so an empty string "" throws, matching the
// MCP server); an object is passed through unchanged. // MCP server); an object is passed through unchanged.
@@ -845,66 +681,29 @@ export class AiChatToolsService {
} }
return await client.updatePageJson(pageId, doc, title); return await client.updatePageJson(pageId, doc, title);
}, },
}), ),
// NOT in the shared registry: this layer names the table argument // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// `tableRef`, while the standalone MCP tool names it `table` (index.ts). // The table reference parameter was unified to `table` (was `tableRef`).
// Sharing one buildShape would rename a model-facing parameter on one tableInsertRow: sharedTool(
// transport, so the table row/cell tools stay per-layer by design. sharedToolSpecs.tableInsertRow,
tableInsertRow: tool({ async ({ pageId, table, cells, index }) =>
description: await client.tableInsertRow(pageId, table, cells, index),
'Insert a row of plain-text cells into a table. Reversible via ' + ),
'page history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
tableRef: z
.string()
.describe('"#<index>" from getOutline, or a block id in the table.'),
cells: z.array(z.string()).describe('The cell texts for the row.'),
index: z
.number()
.int()
.optional()
.describe('0-based insert position (omit/out-of-range to append).'),
}),
execute: async ({ pageId, tableRef, cells, index }) =>
await client.tableInsertRow(pageId, tableRef, cells, index),
}),
// NOT shared — same `tableRef` (here) vs `table` (MCP) parameter-name // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// divergence as tableInsertRow. tableDeleteRow: sharedTool(
tableDeleteRow: tool({ sharedToolSpecs.tableDeleteRow,
description: async ({ pageId, table, index }) =>
'Delete a table row at a 0-based index. Reversible via page history.', await client.tableDeleteRow(pageId, table, index),
inputSchema: modelFriendlyInput({ ),
pageId: z.string().describe('The id of the page.'),
tableRef: z
.string()
.describe('"#<index>" from getOutline, or a block id in the table.'),
index: z.number().int().describe('0-based row index to delete.'),
}),
execute: async ({ pageId, tableRef, index }) =>
await client.tableDeleteRow(pageId, tableRef, index),
}),
// NOT shared — same `tableRef` (here) vs `table` (MCP) parameter-name // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// divergence as tableInsertRow. tableUpdateCell: sharedTool(
tableUpdateCell: tool({ sharedToolSpecs.tableUpdateCell,
description: async ({ pageId, table, row, col, text }) =>
'Set the plain-text content of a table cell at [row, col] (0-based). ' + await client.tableUpdateCell(pageId, table, row, col, text),
'Reversible via page history.', ),
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page.'),
tableRef: z
.string()
.describe('"#<index>" from getOutline, or a block id in the table.'),
row: z.number().int().describe('0-based row index.'),
col: z.number().int().describe('0-based column index.'),
text: z.string().describe('The new cell text.'),
}),
execute: async ({ pageId, tableRef, row, col, text }) =>
await client.tableUpdateCell(pageId, tableRef, row, col, text),
}),
copyPageContent: sharedTool( copyPageContent: sharedTool(
sharedToolSpecs.copyPageContent, sharedToolSpecs.copyPageContent,
@@ -918,25 +717,14 @@ export class AiChatToolsService {
await client.importPageMarkdown(pageId, markdown), await client.importPageMarkdown(pageId, markdown),
), ),
// INTENTIONAL per-transport divergence (not shared): adds a security // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// confirmation framing ("Only share when the user explicitly asked, since // Both layers already carried the security-confirmation framing, so there
// this exposes the page to anyone with the link") for the in-app agent; the // was no real divergence to preserve — only wording drift.
// standalone MCP `share_page` keeps the plain public-URL wording. sharePage: sharedTool(
sharePage: tool({ sharedToolSpecs.sharePage,
description: async ({ pageId, searchIndexing }) =>
'Make a page PUBLICLY accessible and return its public URL. ' +
'Reversible via unsharePage. Only share when the user explicitly ' +
'asked, since this exposes the page to anyone with the link.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to share.'),
searchIndexing: z
.boolean()
.optional()
.describe('Allow public search engines to index it (default true).'),
}),
execute: async ({ pageId, searchIndexing }) =>
await client.sharePage(pageId, searchIndexing), await client.sharePage(pageId, searchIndexing),
}), ),
unsharePage: sharedTool( unsharePage: sharedTool(
sharedToolSpecs.unsharePage, sharedToolSpecs.unsharePage,
@@ -100,54 +100,26 @@ export const INLINE_TOOL_TIERS: Record<
tier: 'core', tier: 'core',
catalogLine: 'getCurrentPage — the page the user is currently viewing.', catalogLine: 'getCurrentPage — the page the user is currently viewing.',
}, },
getPage: { // NOTE: getPage and listPages moved to @docmost/mcp's SHARED_TOOL_SPECS
tier: 'core', // (#294); they carry their own tier ('core') + catalogLine there.
catalogLine: 'getPage — fetch a page as Markdown by its id.', // NOTE: createComment, listComments and resolveComment moved to
}, // @docmost/mcp's SHARED_TOOL_SPECS (#294); they carry their own tier +
listPages: { // catalogLine there. getComment stays inline (MCP-only shape divergence is
tier: 'core', // n/a — it simply has no shared spec).
catalogLine: "listPages — list recent pages, or a space's full page tree.",
},
listComments: {
tier: 'core',
catalogLine: 'listComments — list all comments on a page (including resolved).',
},
getComment: { getComment: {
tier: 'core', tier: 'core',
catalogLine: 'getComment — fetch a single comment by id.', catalogLine: 'getComment — fetch a single comment by id.',
}, },
createComment: {
tier: 'core',
catalogLine:
'createComment — add an inline comment (optionally with a suggested edit).',
},
resolveComment: {
tier: 'core',
catalogLine: 'resolveComment — resolve or reopen a comment thread.',
},
// --- deferred inline --- // --- deferred inline ---
createPage: { // NOTE: createPage, renamePage, movePage, deletePage, updatePageJson and
tier: 'deferred', // exportPageMarkdown moved to @docmost/mcp's SHARED_TOOL_SPECS (#294); they
catalogLine: 'createPage — create a new page with a Markdown body in a space.', // carry their own deferred tier + catalogLine there.
},
updatePageContent: { updatePageContent: {
tier: 'deferred', tier: 'deferred',
catalogLine: catalogLine:
"updatePageContent — replace a page's body (and optionally title) with new Markdown.", "updatePageContent — replace a page's body (and optionally title) with new Markdown.",
}, },
renamePage: {
tier: 'deferred',
catalogLine: "renamePage — change a page's title only (body untouched).",
},
movePage: {
tier: 'deferred',
catalogLine: 'movePage — move a page under a new parent or to the space root.',
},
deletePage: {
tier: 'deferred',
catalogLine: 'deletePage — move a page to trash (soft delete, reversible).',
},
listSidebarPages: { listSidebarPages: {
tier: 'deferred', tier: 'deferred',
catalogLine: catalogLine:
@@ -157,42 +129,21 @@ export const INLINE_TOOL_TIERS: Record<
tier: 'deferred', tier: 'deferred',
catalogLine: 'getTable — read a table as a matrix of cell texts and cell ids.', catalogLine: 'getTable — read a table as a matrix of cell texts and cell ids.',
}, },
checkNewComments: { // NOTE: tableInsertRow, tableDeleteRow and tableUpdateCell moved to
tier: 'deferred', // @docmost/mcp's SHARED_TOOL_SPECS (#294); they carry their own deferred tier +
catalogLine: // catalogLine there. getTable stays inline (its MCP name table_get breaks the
'checkNewComments — find comments in a space created after a timestamp.', // snake_case(inAppKey) convention, so it has no shared spec).
}, // NOTE: checkNewComments moved to @docmost/mcp's SHARED_TOOL_SPECS (#294);
// it carries its own deferred tier + catalogLine there.
getPageHistory: { getPageHistory: {
tier: 'deferred', tier: 'deferred',
catalogLine: catalogLine:
'getPageHistory — fetch one page-history version with its ProseMirror content.', 'getPageHistory — fetch one page-history version with its ProseMirror content.',
}, },
exportPageMarkdown: { // NOTE: sharePage moved to @docmost/mcp's SHARED_TOOL_SPECS (#294); it carries
tier: 'deferred', // its own deferred tier + catalogLine there. transformPage stays inline (its
catalogLine: // schema deliberately diverges — it omits the deleteComments field the MCP
'exportPageMarkdown — export a page to self-contained Markdown (body + comments).', // docmost_transform exposes, a comment-deletion guardrail).
},
updatePageJson: {
tier: 'deferred',
catalogLine:
"updatePageJson — overwrite a page's body with a full ProseMirror document.",
},
tableInsertRow: {
tier: 'deferred',
catalogLine: 'tableInsertRow — insert a row of plain-text cells into a table.',
},
tableDeleteRow: {
tier: 'deferred',
catalogLine: 'tableDeleteRow — delete a table row at a 0-based index.',
},
tableUpdateCell: {
tier: 'deferred',
catalogLine: 'tableUpdateCell — set the text of a table cell at [row, col].',
},
sharePage: {
tier: 'deferred',
catalogLine: 'sharePage — make a page publicly accessible and return its URL.',
},
transformPage: { transformPage: {
tier: 'deferred', tier: 'deferred',
catalogLine: "transformPage — run a sandboxed JS transform over a page's document.", catalogLine: "transformPage — run a sandboxed JS transform over a page's document.",
@@ -51,21 +51,7 @@ export class JwtStrategy extends PassportStrategy(Strategy, 'jwt') {
throw new UnauthorizedException(); throw new UnauthorizedException();
} }
// #348 — reuse the workspace DomainMiddleware already loaded for this request const workspace = await this.workspaceRepo.findById(payload.workspaceId);
// instead of re-querying it. `validate()` above has confirmed
// `req.raw.workspaceId === payload.workspaceId` (or that it is unset), and the
// middleware sets `req.raw.workspace` alongside `req.raw.workspaceId` from the
// SAME workspace row, so when the ids match this is that row. NOTE it is the
// middleware's `selectAll` object (a superset of the fallback `findById` base
// fields — it also carries licenseKey/auditRetentionDays); that is harmless
// here because every consumer reads this workspace via the AuthWorkspace
// decorator, which already preferred `req.raw.workspace` (the selectAll object)
// over `req.user.workspace` before this change. Fall back to the query if the
// middleware did not populate it (a path that bypasses DomainMiddleware).
const workspace =
req.raw.workspace && req.raw.workspaceId === payload.workspaceId
? req.raw.workspace
: await this.workspaceRepo.findById(payload.workspaceId);
if (!workspace) { if (!workspace) {
throw new UnauthorizedException(); throw new UnauthorizedException();
@@ -38,8 +38,6 @@ export class FavoriteService {
await this.pagePermissionRepo.filterAccessiblePageIds({ await this.pagePermissionRepo.filterAccessiblePageIds({
pageIds: result.items, pageIds: result.items,
userId, userId,
// #348 — favorites load at app-start; enable the workspace short-circuit.
workspaceId,
}); });
const accessibleSet = new Set(accessibleIds); const accessibleSet = new Set(accessibleIds);
result.items = result.items.filter((id) => accessibleSet.has(id)); result.items = result.items.filter((id) => accessibleSet.has(id));
@@ -127,8 +125,6 @@ export class FavoriteService {
await this.pagePermissionRepo.filterAccessiblePageIds({ await this.pagePermissionRepo.filterAccessiblePageIds({
pageIds, pageIds,
userId, userId,
// #348 — workspace-level short-circuit for the favorites list.
workspaceId,
}); });
accessiblePageSet = new Set(accessibleIds); accessiblePageSet = new Set(accessibleIds);
} }
@@ -23,12 +23,7 @@ export class NotificationController {
@Body() dto: ListNotificationsDto, @Body() dto: ListNotificationsDto,
@AuthUser() user: User, @AuthUser() user: User,
) { ) {
return this.notificationService.findByUserId( return this.notificationService.findByUserId(user.id, dto, dto.type);
user.id,
dto,
dto.type,
user.workspaceId,
);
} }
@HttpCode(HttpStatus.OK) @HttpCode(HttpStatus.OK)
@@ -45,7 +45,6 @@ export class NotificationService {
userId: string, userId: string,
pagination: PaginationOptions, pagination: PaginationOptions,
type: NotificationTab = 'all', type: NotificationTab = 'all',
workspaceId?: string | null,
) { ) {
const result = await this.notificationRepo.findByUserId( const result = await this.notificationRepo.findByUserId(
userId, userId,
@@ -62,8 +61,6 @@ export class NotificationService {
await this.pagePermissionRepo.filterAccessiblePageIds({ await this.pagePermissionRepo.filterAccessiblePageIds({
pageIds, pageIds,
userId, userId,
// #348 — notifications list; enable the workspace short-circuit.
workspaceId,
}); });
const accessibleSet = new Set(accessiblePageIds); const accessibleSet = new Set(accessiblePageIds);
+2 -12
View File
@@ -446,11 +446,7 @@ export class PageController {
); );
} }
return this.pageService.getRecentPages( return this.pageService.getRecentPages(user.id, pagination);
user.id,
pagination,
user.workspaceId,
);
} }
@HttpCode(HttpStatus.OK) @HttpCode(HttpStatus.OK)
@@ -473,13 +469,7 @@ export class PageController {
} }
} }
return this.pageService.getCreatedByPages( return this.pageService.getCreatedByPages(targetUserId, user.id, pagination, dto.spaceId);
targetUserId,
user.id,
pagination,
dto.spaceId,
user.workspaceId,
);
} }
@HttpCode(HttpStatus.OK) @HttpCode(HttpStatus.OK)
@@ -52,7 +52,9 @@ import {
INTERNAL_LINK_REGEX, INTERNAL_LINK_REGEX,
extractPageSlugId, extractPageSlugId,
} from '../../../integrations/export/utils'; } from '../../../integrations/export/utils';
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext'; import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../../../integrations/import/utils/foreign-markdown';
import { WatcherService } from '../../watcher/watcher.service'; import { WatcherService } from '../../watcher/watcher.service';
import { sql } from 'kysely'; import { sql } from 'kysely';
import { TransclusionService } from '../transclusion/transclusion.service'; import { TransclusionService } from '../transclusion/transclusion.service';
@@ -1163,7 +1165,6 @@ export class PageService {
async getRecentPages( async getRecentPages(
userId: string, userId: string,
pagination: PaginationOptions, pagination: PaginationOptions,
workspaceId?: string | null,
): Promise<CursorPaginationResult<Page>> { ): Promise<CursorPaginationResult<Page>> {
const result = await this.pageRepo.getRecentPages(userId, pagination); const result = await this.pageRepo.getRecentPages(userId, pagination);
@@ -1173,8 +1174,6 @@ export class PageService {
await this.pagePermissionRepo.filterAccessiblePageIds({ await this.pagePermissionRepo.filterAccessiblePageIds({
pageIds, pageIds,
userId, userId,
// #348 — cross-space "recent"; enable the workspace short-circuit.
workspaceId,
}); });
const accessibleSet = new Set(accessibleIds); const accessibleSet = new Set(accessibleIds);
result.items = result.items.filter((p) => accessibleSet.has(p.id)); result.items = result.items.filter((p) => accessibleSet.has(p.id));
@@ -1188,7 +1187,6 @@ export class PageService {
requestingUserId: string, requestingUserId: string,
pagination: PaginationOptions, pagination: PaginationOptions,
spaceId?: string, spaceId?: string,
workspaceId?: string | null,
): Promise<CursorPaginationResult<Page>> { ): Promise<CursorPaginationResult<Page>> {
const result = await this.pageRepo.getCreatedByPages( const result = await this.pageRepo.getCreatedByPages(
creatorId, creatorId,
@@ -1203,9 +1201,6 @@ export class PageService {
await this.pagePermissionRepo.filterAccessiblePageIds({ await this.pagePermissionRepo.filterAccessiblePageIds({
pageIds, pageIds,
userId: requestingUserId, userId: requestingUserId,
spaceId,
// #348 — enable the workspace short-circuit when not space-scoped.
workspaceId,
}); });
const accessibleSet = new Set(accessibleIds); const accessibleSet = new Set(accessibleIds);
result.items = result.items.filter((p) => accessibleSet.has(p.id)); result.items = result.items.filter((p) => accessibleSet.has(p.id));
@@ -1308,8 +1303,14 @@ export class PageService {
switch (format) { switch (format) {
case 'markdown': { case 'markdown': {
const html = await markdownToHtml(content as string); // Canonical markdown -> ProseMirror JSON directly via
prosemirrorJson = htmlToJson(html as string); // `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate,
// no editor-ext markdown layer. Foreign markdown surfaces the strict
// parser rejects (GFM `[^id]` reference footnotes) are normalized to the
// canonical inline form first.
prosemirrorJson = await markdownToProseMirror(
normalizeForeignMarkdown(content as string),
);
break; break;
} }
case 'html': { case 'html': {
@@ -93,41 +93,6 @@ function collectNodes<T>(
return Array.from(byKey.values()); return Array.from(byKey.values());
} }
/**
* #348 cheap early-exit probe: does this doc contain ANY node the transclusion
* syncs care about (`transclusionSource` / `transclusionReference` / `pageEmbed`)?
* Lets the collab store skip the three sync SELECTs when neither the previous nor
* the new content has any such node there is nothing to insert, and (since the
* DB mirrors the previously-persisted content) nothing to delete. Walks once and
* short-circuits on the first match; uses the same depth ceiling as the
* collectors. Deliberately does NOT skip `transclusionSource` subtrees: it only
* answers "any node present?", so descending everywhere is strictly conservative
* (it can never wrongly report "none").
*/
export function hasTransclusionFamilyNodes(doc: unknown): boolean {
const visit = (node: any, depth: number): boolean => {
if (!node || typeof node !== 'object') return false;
if (depth > MAX_PM_WALK_DEPTH) return false;
if (
node.type === TRANSCLUSION_TYPE ||
node.type === REFERENCE_TYPE ||
node.type === PAGE_EMBED_TYPE
) {
return true;
}
if (Array.isArray(node.content)) {
for (const child of node.content) {
if (visit(child, depth + 1)) return true;
}
}
return false;
};
return visit(doc, 0);
}
/** /**
* Walks a ProseMirror JSON document and returns one snapshot per top-level * Walks a ProseMirror JSON document and returns one snapshot per top-level
* `transclusion` node. Does not recurse into transclusions (schema disallows * `transclusion` node. Does not recurse into transclusions (schema disallows
@@ -155,8 +155,6 @@ export class SearchService {
pageIds, pageIds,
userId: opts.userId, userId: opts.userId,
spaceId: searchParams.spaceId, spaceId: searchParams.spaceId,
// #348 — enables the workspace-level short-circuit when not space-scoped.
workspaceId: opts.workspaceId,
}); });
const accessibleSet = new Set(accessibleIds); const accessibleSet = new Set(accessibleIds);
results = results.filter((r: any) => accessibleSet.has(r.id)); results = results.filter((r: any) => accessibleSet.has(r.id));
@@ -268,8 +266,6 @@ export class SearchService {
await this.pagePermissionRepo.filterAccessiblePageIds({ await this.pagePermissionRepo.filterAccessiblePageIds({
pageIds, pageIds,
userId, userId,
// #348 — workspace-level short-circuit for the suggest path.
workspaceId,
}); });
const accessibleSet = new Set(accessibleIds); const accessibleSet = new Set(accessibleIds);
pages = pages.filter((p) => accessibleSet.has(p.id)); pages = pages.filter((p) => accessibleSet.has(p.id));
@@ -55,6 +55,14 @@ export class UpdateWorkspaceDto extends PartialType(CreateWorkspaceDto) {
@IsBoolean() @IsBoolean()
aiDictationStreaming: boolean; aiDictationStreaming: boolean;
// #184: detached/autonomous agent runs (settings.ai.autonomousRuns). When on, a
// chat turn becomes a server-side RUN that survives a browser disconnect; only
// an explicit /ai-chat/stop ends it. Off by default; single-instance-only in
// phase 1 (see AiChatRunService.warnIfMultiInstance / AGENTS.md).
@IsOptional()
@IsBoolean()
autonomousRuns: boolean;
// Workspace master toggle that enables/disables the HTML embed block type. // Workspace master toggle that enables/disables the HTML embed block type.
// Persisted at settings.htmlEmbed. ABSENT/false => OFF (default). The block // Persisted at settings.htmlEmbed. ABSENT/false => OFF (default). The block
// itself renders in a sandboxed iframe, so this is a feature switch, not a // itself renders in a sandboxed iframe, so this is a feature switch, not a
@@ -526,6 +526,20 @@ export class WorkspaceService {
); );
} }
if (typeof updateWorkspaceDto.autonomousRuns !== 'undefined') {
const prev = settingsBefore?.ai?.autonomousRuns ?? false;
if (prev !== updateWorkspaceDto.autonomousRuns) {
before.autonomousRuns = prev;
after.autonomousRuns = updateWorkspaceDto.autonomousRuns;
}
await this.workspaceRepo.updateAiSettings(
workspaceId,
'autonomousRuns',
updateWorkspaceDto.autonomousRuns,
trx,
);
}
if (typeof updateWorkspaceDto.htmlEmbed !== 'undefined') { if (typeof updateWorkspaceDto.htmlEmbed !== 'undefined') {
const prev = settingsBefore?.htmlEmbed ?? false; const prev = settingsBefore?.htmlEmbed ?? false;
if (prev !== updateWorkspaceDto.htmlEmbed) { if (prev !== updateWorkspaceDto.htmlEmbed) {
@@ -579,6 +593,7 @@ export class WorkspaceService {
delete updateWorkspaceDto.aiChat; delete updateWorkspaceDto.aiChat;
delete updateWorkspaceDto.aiDictation; delete updateWorkspaceDto.aiDictation;
delete updateWorkspaceDto.aiDictationStreaming; delete updateWorkspaceDto.aiDictationStreaming;
delete updateWorkspaceDto.autonomousRuns;
delete updateWorkspaceDto.htmlEmbed; delete updateWorkspaceDto.htmlEmbed;
delete updateWorkspaceDto.trackerHead; delete updateWorkspaceDto.trackerHead;
delete updateWorkspaceDto.aiPublicShareAssistant; delete updateWorkspaceDto.aiPublicShareAssistant;
@@ -31,6 +31,7 @@ import { FavoriteRepo } from '@docmost/db/repos/favorite/favorite.repo';
import { TemplateRepo } from '@docmost/db/repos/template/template.repo'; import { TemplateRepo } from '@docmost/db/repos/template/template.repo';
import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo'; import { AiChatRepo } from '@docmost/db/repos/ai-chat/ai-chat.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo'; import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo'; import { AiChatPageSnapshotRepo } from '@docmost/db/repos/ai-chat/ai-chat-page-snapshot.repo';
import { AiProviderCredentialsRepo } from '@docmost/db/repos/ai-chat/ai-provider-credentials.repo'; import { AiProviderCredentialsRepo } from '@docmost/db/repos/ai-chat/ai-provider-credentials.repo';
import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo'; import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo';
@@ -122,6 +123,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
TemplateRepo, TemplateRepo,
AiChatRepo, AiChatRepo,
AiChatMessageRepo, AiChatMessageRepo,
AiChatRunRepo,
AiChatPageSnapshotRepo, AiChatPageSnapshotRepo,
AiProviderCredentialsRepo, AiProviderCredentialsRepo,
AiMcpServerRepo, AiMcpServerRepo,
@@ -156,6 +158,7 @@ import { firstSqlToken } from '../integrations/metrics/metrics.constants';
TemplateRepo, TemplateRepo,
AiChatRepo, AiChatRepo,
AiChatMessageRepo, AiChatMessageRepo,
AiChatRunRepo,
AiChatPageSnapshotRepo, AiChatPageSnapshotRepo,
AiProviderCredentialsRepo, AiProviderCredentialsRepo,
AiMcpServerRepo, AiMcpServerRepo,
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
import { InjectQueue } from '@nestjs/bullmq'; import { InjectQueue } from '@nestjs/bullmq';
import { QueueJob, QueueName } from '../../integrations/queue/constants'; import { QueueJob, QueueName } from '../../integrations/queue/constants';
import { Queue } from 'bullmq'; import { Queue } from 'bullmq';
import { EnvironmentService } from '../../integrations/environment/environment.service';
/** /**
* Thin snapshot of a page node carried inside domain events so the WebSocket * Thin snapshot of a page node carried inside domain events so the WebSocket
@@ -112,48 +111,24 @@ export class PageListener {
private readonly logger = new Logger(PageListener.name); private readonly logger = new Logger(PageListener.name);
constructor( constructor(
private readonly environmentService: EnvironmentService,
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue, @InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
) {} ) {}
@OnEvent(EventName.PAGE_CREATED) @OnEvent(EventName.PAGE_CREATED)
async handlePageCreated(event: PageEvent) { async handlePageCreated(event: PageEvent) {
const { pageIds, workspaceId } = event; const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_CREATED, {
pageIds,
});
}
await this.aiQueue.add(QueueJob.PAGE_CREATED, { pageIds, workspaceId }); await this.aiQueue.add(QueueJob.PAGE_CREATED, { pageIds, workspaceId });
} }
@OnEvent(EventName.PAGE_UPDATED)
async handlePageUpdated(event: PageEvent) {
const { pageIds } = event;
await this.searchQueue.add(QueueJob.PAGE_UPDATED, { pageIds });
}
@OnEvent(EventName.PAGE_DELETED) @OnEvent(EventName.PAGE_DELETED)
async handlePageDeleted(event: PageEvent) { async handlePageDeleted(event: PageEvent) {
const { pageIds, workspaceId } = event; const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_DELETED, { pageIds });
}
await this.aiQueue.add(QueueJob.PAGE_DELETED, { pageIds, workspaceId }); await this.aiQueue.add(QueueJob.PAGE_DELETED, { pageIds, workspaceId });
} }
@OnEvent(EventName.PAGE_SOFT_DELETED) @OnEvent(EventName.PAGE_SOFT_DELETED)
async handlePageSoftDeleted(event: PageEvent) { async handlePageSoftDeleted(event: PageEvent) {
const { pageIds, workspaceId } = event; const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_SOFT_DELETED, { pageIds });
}
await this.aiQueue.add(QueueJob.PAGE_SOFT_DELETED, { await this.aiQueue.add(QueueJob.PAGE_SOFT_DELETED, {
pageIds, pageIds,
workspaceId, workspaceId,
@@ -163,14 +138,6 @@ export class PageListener {
@OnEvent(EventName.PAGE_RESTORED) @OnEvent(EventName.PAGE_RESTORED)
async handlePageRestored(event: PageEvent) { async handlePageRestored(event: PageEvent) {
const { pageIds, workspaceId } = event; const { pageIds, workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.PAGE_RESTORED, { pageIds });
}
await this.aiQueue.add(QueueJob.PAGE_RESTORED, { pageIds, workspaceId }); await this.aiQueue.add(QueueJob.PAGE_RESTORED, { pageIds, workspaceId });
} }
isTypesense(): boolean {
return this.environmentService.getSearchDriver() === 'typesense';
}
} }
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
import { InjectQueue } from '@nestjs/bullmq'; import { InjectQueue } from '@nestjs/bullmq';
import { QueueJob, QueueName } from '../../integrations/queue/constants'; import { QueueJob, QueueName } from '../../integrations/queue/constants';
import { Queue } from 'bullmq'; import { Queue } from 'bullmq';
import { EnvironmentService } from '../../integrations/environment/environment.service';
export class SpaceEvent { export class SpaceEvent {
spaceId: string; spaceId: string;
@@ -15,22 +14,12 @@ export class SpaceListener {
private readonly logger = new Logger(SpaceListener.name); private readonly logger = new Logger(SpaceListener.name);
constructor( constructor(
private readonly environmentService: EnvironmentService,
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue, @InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
) {} ) {}
@OnEvent(EventName.SPACE_DELETED) @OnEvent(EventName.SPACE_DELETED)
async handleSpaceDeleted(event: SpaceEvent) { async handleSpaceDeleted(event: SpaceEvent) {
const { spaceId } = event; const { spaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.SPACE_DELETED, { spaceId });
}
await this.aiQueue.add(QueueJob.SPACE_DELETED, { spaceId }); await this.aiQueue.add(QueueJob.SPACE_DELETED, { spaceId });
} }
isTypesense(): boolean {
return this.environmentService.getSearchDriver() === 'typesense';
}
} }
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
import { InjectQueue } from '@nestjs/bullmq'; import { InjectQueue } from '@nestjs/bullmq';
import { QueueJob, QueueName } from '../../integrations/queue/constants'; import { QueueJob, QueueName } from '../../integrations/queue/constants';
import { Queue } from 'bullmq'; import { Queue } from 'bullmq';
import { EnvironmentService } from '../../integrations/environment/environment.service';
export class WorkspaceEvent { export class WorkspaceEvent {
workspaceId: string; workspaceId: string;
@@ -15,22 +14,12 @@ export class WorkspaceListener {
private readonly logger = new Logger(WorkspaceListener.name); private readonly logger = new Logger(WorkspaceListener.name);
constructor( constructor(
private readonly environmentService: EnvironmentService,
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue, @InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
) {} ) {}
@OnEvent(EventName.WORKSPACE_DELETED) @OnEvent(EventName.WORKSPACE_DELETED)
async handlePageDeleted(event: WorkspaceEvent) { async handlePageDeleted(event: WorkspaceEvent) {
const { workspaceId } = event; const { workspaceId } = event;
if (this.isTypesense()) {
await this.searchQueue.add(QueueJob.WORKSPACE_DELETED, { workspaceId });
}
await this.aiQueue.add(QueueJob.WORKSPACE_DELETED, { workspaceId }); await this.aiQueue.add(QueueJob.WORKSPACE_DELETED, { workspaceId });
} }
isTypesense(): boolean {
return this.environmentService.getSearchDriver() === 'typesense';
}
} }
+4
View File
@@ -24,6 +24,10 @@ const migrator = new Migrator({
path, path,
migrationFolder, migrationFolder,
}), }),
// Match the startup auto-migrator (migration.service.ts): a back-dated
// migration from a long-lived branch must be applied, not rejected as
// "corrupted migrations" (incident #361). See that file for the full rationale.
allowUnorderedMigrations: true,
}); });
run(db, migrator, migrationFolder); run(db, migrator, migrationFolder);
@@ -0,0 +1,106 @@
import { type Kysely, sql } from 'kysely';
/**
* `ai_chat_runs` the agent RUN as a first-class, server-side lifecycle object
* (#184 phase 1: autonomous agent runs detached from the browser window).
*
* Until now an agent turn lived ONLY as long as the HTTP request was open
* (`res.hijack()` in ai-chat.controller.ts); a browser disconnect aborted it.
* This table makes a turn a persistent object the server owns: it is created
* when a run starts (inserted directly as 'running' in phase 1 'pending' is
* only this column's default + a reserved value, never written by code yet) and
* advances to succeeded|failed|aborted, surviving the subscriber (browser) going
* away when it settles. The DB is the source of
* truth a later client reconnects/sees the result by reading this row plus the
* assistant message it projects (`assistant_message_id`).
*
* The assistant message row (#183 step-granular durability) is the PROJECTION of
* a run's output; this row is the run's LIFECYCLE. They are linked by
* `assistant_message_id` (SET NULL if the message is later pruned).
*
* `status` : 'pending' | 'running' | 'succeeded' | 'failed' | 'aborted'.
* `trigger` : 'user' | 'autostart' | 'schedule' | 'api' | 'continue' only
* 'user' is produced in phase 1; the others are reserved for the
* autonomy triggers deferred to phase 2 so they need no later
* migration.
*
* ONE ACTIVE RUN PER CHAT is enforced by a partial unique index on `chat_id`
* WHERE status IN ('pending','running'): an autonomous run and a user run can
* never trample each other on the same chat. Settled runs (succeeded/failed/
* aborted) are excluded from the index so a chat can accumulate any number of
* historical runs.
*/
export async function up(db: Kysely<any>): Promise<void> {
await db.schema
.createTable('ai_chat_runs')
.ifNotExists()
.addColumn('id', 'uuid', (col) =>
col.primaryKey().defaultTo(sql`gen_uuid_v7()`),
)
.addColumn('chat_id', 'uuid', (col) =>
col.references('ai_chats.id').onDelete('cascade').notNull(),
)
.addColumn('workspace_id', 'uuid', (col) =>
col.references('workspaces.id').onDelete('cascade').notNull(),
)
// The human who triggered the run (audit). SET NULL on user deletion so the
// run history outlives its author; NULL is also the natural value for a
// future system/cron/api trigger with no human actor.
.addColumn('created_by', 'uuid', (col) =>
col.references('users.id').onDelete('set null'),
)
// The assistant message this run materializes (the #183 projection). SET NULL
// if that message row is later deleted; nullable because the run row is
// created a moment BEFORE the assistant row is seeded.
.addColumn('assistant_message_id', 'uuid', (col) =>
col.references('ai_chat_messages.id').onDelete('set null'),
)
.addColumn('trigger', 'varchar(20)', (col) =>
col.notNull().defaultTo('user'),
)
.addColumn('status', 'varchar(20)', (col) =>
col.notNull().defaultTo('pending'),
)
// Terminal error message for a failed run (provider/transport cause),
// mirroring the assistant message's metadata.error.
.addColumn('error', 'text', (col) => col)
// Number of agent steps finished so far (kept monotonic with the projection).
.addColumn('step_count', 'integer', (col) => col.notNull().defaultTo(0))
// Set when an EXPLICIT user stop is requested (distinct from a mere browser
// disconnect, which never stops a run). The runner aborts the turn and the
// run settles as 'aborted'.
.addColumn('stop_requested_at', 'timestamptz', (col) => col)
.addColumn('started_at', 'timestamptz', (col) => col)
.addColumn('finished_at', 'timestamptz', (col) => col)
.addColumn('created_at', 'timestamptz', (col) =>
col.notNull().defaultTo(sql`now()`),
)
.addColumn('updated_at', 'timestamptz', (col) =>
col.notNull().defaultTo(sql`now()`),
)
.execute();
// Reconnect / "latest run for this chat" reads hit chat_id first.
await db.schema
.createIndex('ai_chat_runs_chat_id_idx')
.ifNotExists()
.on('ai_chat_runs')
.column('chat_id')
.execute();
// One ACTIVE run per chat (advisory at the DB level): a second pending/running
// run on the same chat is rejected, so a user turn and an autonomous turn can
// never race on the same chat. Partial so settled runs do not collide.
await db.schema
.createIndex('ai_chat_runs_one_active_per_chat')
.ifNotExists()
.on('ai_chat_runs')
.column('chat_id')
.unique()
.where(sql.ref('status'), 'in', sql`('pending','running')`)
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema.dropTable('ai_chat_runs').execute();
}
@@ -1,118 +0,0 @@
import { type Kysely, sql } from 'kysely';
/**
* #348 targeted hot-path indexes.
*
* 1. GIN trigram indexes for `/search/suggest`. That endpoint runs a
* leading-wildcard `LOWER(f_unaccent(col)) LIKE '%q%'` per keystroke, which
* is a sequential scan without a trigram index. The index EXPRESSIONS below
* are `LOWER(f_unaccent(title|name))`, matching the predicates in
* search.service.ts exactly so the planner uses them (verified with EXPLAIN:
* the suggest predicate resolves to a Bitmap Index Scan on these indexes).
*
* IMMUTABLE-wrapper fix (required for the index to build): `f_unaccent` was
* defined as `SELECT unaccent('unaccent', $1)` (the two-arg, dictionary-named
* unaccent). That body CANNOT be used in an index expression: when Postgres
* inlines the IMMUTABLE SQL wrapper while building the index it fails to
* resolve the two-arg call (`function unaccent(unknown, text) does not exist`,
* the `'unaccent'` literal loses its regdictionary coercion). The single-arg
* `unaccent($1)` is the same operation (the default text-search dictionary IS
* `unaccent`; verified byte-equal on accented samples), and crucially
* SCHEMA-QUALIFIED as `public.unaccent($1)` it inlines cleanly, so the index
* builds. We therefore `CREATE OR REPLACE` `f_unaccent` to the qualified
* single-arg body. This is output-identical for every existing caller (the
* tsvector trigger, the main `tsv @@` search, and the suggest LIKE), so no
* reindex/backfill is needed; `down()` restores the original two-arg body.
* (The `unaccent` extension is installed in `public` in this codebase, which
* is why `public.unaccent` is the correct qualification.)
*
* 2. Composite indexes for two ORDER-BY-only-on-id queries that currently sort
* on top of a created_at index:
* - page_history: `findPageHistoryByPageId` does WHERE page_id ORDER BY id
* DESC, but only `(page_id, created_at DESC)` exists extra sort.
* - comments: `findPageComments` does WHERE page_id ORDER BY id ASC, but only
* `(page_id)` exists extra sort.
*
* DEPLOY-TIME LOCK WARNING: these are plain (non-CONCURRENT) CREATE INDEX
* statements CONCURRENTLY is impossible because Kysely runs each migration in a
* transaction. They take a SHARE lock that BLOCKS writes (INSERT/UPDATE/DELETE) on
* pages/users/groups/comments/page_history for the duration of the build. The two
* GIN trigram builds on pages.title / users.name are the slow ones and can take
* minutes on a large tenant a write-outage window during the deploy migration.
* For large installations, run this migration in a maintenance window, or build
* the trigram indexes out-of-band with CREATE INDEX CONCURRENTLY before deploying
* (then this migration's `IF NOT EXISTS` is a no-op). Small/typical tenants are
* unaffected.
*/
export async function up(db: Kysely<any>): Promise<void> {
// Index-compatible, output-identical redefinition of f_unaccent (see header).
await sql`
CREATE OR REPLACE FUNCTION f_unaccent(text)
RETURNS text
LANGUAGE sql
IMMUTABLE PARALLEL SAFE STRICT
AS $func$
SELECT public.unaccent($1);
$func$
`.execute(db);
// Search-suggest trigram indexes. Expressions match search.service.ts.
await sql`
CREATE INDEX IF NOT EXISTS idx_pages_title_trgm
ON pages USING gin ((LOWER(f_unaccent(title))) gin_trgm_ops)
`.execute(db);
await sql`
CREATE INDEX IF NOT EXISTS idx_users_name_trgm
ON users USING gin ((LOWER(f_unaccent(name))) gin_trgm_ops)
`.execute(db);
await sql`
CREATE INDEX IF NOT EXISTS idx_groups_name_trgm
ON groups USING gin ((LOWER(f_unaccent(name))) gin_trgm_ops)
`.execute(db);
// page_history: WHERE page_id ORDER BY id DESC (findPageHistoryByPageId).
await sql`
CREATE INDEX IF NOT EXISTS idx_page_history_page_id
ON page_history (page_id, id DESC)
`.execute(db);
// comments: WHERE page_id ORDER BY id ASC (findPageComments).
await sql`
CREATE INDEX IF NOT EXISTS idx_comments_page_id_id
ON comments (page_id, id)
`.execute(db);
// page_access(workspace_id): #348 made hasRestrictedPagesInWorkspace uncached
// (F1 fix), so `EXISTS(SELECT 1 FROM page_access WHERE workspace_id=?)` now runs
// per-request on every whole-workspace list endpoint (global search + suggest,
// favorites, notifications, recent, created-by). page_access only had a
// space_id index → that EXISTS was a seq scan in the common zero-restriction
// case. This index makes it an index-only existence probe.
await sql`
CREATE INDEX IF NOT EXISTS idx_page_access_workspace_id
ON page_access (workspace_id)
`.execute(db);
}
export async function down(db: Kysely<any>): Promise<void> {
// Drop the expression indexes before restoring the function body.
await sql`DROP INDEX IF EXISTS idx_pages_title_trgm`.execute(db);
await sql`DROP INDEX IF EXISTS idx_users_name_trgm`.execute(db);
await sql`DROP INDEX IF EXISTS idx_groups_name_trgm`.execute(db);
await sql`DROP INDEX IF EXISTS idx_page_history_page_id`.execute(db);
await sql`DROP INDEX IF EXISTS idx_comments_page_id_id`.execute(db);
await sql`DROP INDEX IF EXISTS idx_page_access_workspace_id`.execute(db);
// Restore the original two-arg (dictionary-named) f_unaccent body.
await sql`
CREATE OR REPLACE FUNCTION f_unaccent(text)
RETURNS text
LANGUAGE sql
IMMUTABLE PARALLEL SAFE STRICT
AS $func$
SELECT unaccent('unaccent', $1);
$func$
`.execute(db);
}
@@ -121,6 +121,23 @@ export class AiChatMessageRepo {
return rows.reverse(); return rows.reverse();
} }
/** Fetch a single message by id + workspace (e.g. a run's projection row for
* the #184 reconnect read). Returns undefined when nothing matches. */
async findById(
id: string,
workspaceId: string,
trx?: KyselyTransaction,
): Promise<AiChatMessage | undefined> {
const db = dbOrTx(this.db, trx);
return db
.selectFrom('aiChatMessages')
.select(this.baseFields)
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('deletedAt', 'is', null)
.executeTakeFirst();
}
async insert( async insert(
insertable: InsertableAiChatMessage, insertable: InsertableAiChatMessage,
trx?: KyselyTransaction, trx?: KyselyTransaction,
@@ -0,0 +1,82 @@
import { AiChatRunRepo, SWEEP_RUN_STALE_MS } from './ai-chat-run.repo';
import type { KyselyDB } from '../../types/kysely.types';
/**
* Unit coverage for AiChatRunRepo.sweepRunning over a chainable builder mock (no
* live DB). The F1 invariant under test (DECISION C): the BOOT sweep is
* UNCONDITIONAL it adds NO `updatedAt <` predicate, so a fresh 'running' run
* (updatedAt = now) IS settled rather than skipped by a staleness window. The
* window is added ONLY when an explicit `staleMs` is supplied (the future phase-2
* multi-instance timer sweep). We assert the EXACT predicates the spec mandates.
*/
describe('AiChatRunRepo.sweepRunning', () => {
type Recorded = {
table?: string;
set?: Record<string, unknown>;
wheres: Array<[string, string, unknown]>;
returning?: string;
};
function makeDb(swept: Array<{ id: string }>): {
db: KyselyDB;
rec: Recorded;
} {
const rec: Recorded = { wheres: [] };
const builder: Record<string, unknown> = {};
builder.set = (v: Record<string, unknown>) => {
rec.set = v;
return builder;
};
builder.where = (col: string, op: string, val: unknown) => {
rec.wheres.push([col, op, val]);
return builder;
};
builder.returning = (col: string) => {
rec.returning = col;
return builder;
};
builder.execute = () => Promise.resolve(swept);
const db = {
updateTable: (table: string) => {
rec.table = table;
return builder;
},
} as unknown as KyselyDB;
return { db, rec };
}
it('F1: the boot sweep (no staleMs) is UNCONDITIONAL — only a status filter, NO updatedAt window', async () => {
const { db, rec } = makeDb([{ id: 'r1' }, { id: 'r2' }]);
const repo = new AiChatRunRepo(db);
const swept = await repo.sweepRunning();
expect(swept).toBe(2);
expect(rec.table).toBe('aiChatRuns');
// The status filter is always present...
expect(rec.wheres).toContainEqual([
'status',
'in',
expect.arrayContaining(['pending', 'running']),
]);
// ...but a fresh 'running' run (updatedAt = now) must NOT be skipped: no
// updatedAt predicate at all on the boot path.
expect(rec.wheres.some(([col]) => col === 'updatedAt')).toBe(false);
// It flips to 'aborted' and stamps finishedAt.
expect(rec.set).toEqual(
expect.objectContaining({ status: 'aborted', finishedAt: expect.any(Date) }),
);
});
it('phase-2 path: an explicit staleMs reintroduces the updatedAt window', async () => {
const { db, rec } = makeDb([]);
const repo = new AiChatRunRepo(db);
await repo.sweepRunning({ staleMs: SWEEP_RUN_STALE_MS });
const updatedAtWhere = rec.wheres.find(([col]) => col === 'updatedAt');
expect(updatedAtWhere).toBeDefined();
expect(updatedAtWhere![1]).toBe('<');
expect(updatedAtWhere![2]).toBeInstanceOf(Date);
});
});
@@ -0,0 +1,212 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { sql } from 'kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import {
AiChatRun,
InsertableAiChatRun,
} from '@docmost/db/types/entity.types';
// Statuses that count as "the run is still live" (an autonomous and a user run
// must never both be live on one chat — enforced by the partial unique index and
// checked here for friendly 409s before the insert races the constraint).
export const ACTIVE_RUN_STATUSES = ['pending', 'running'] as const;
// Crash-recovery sweep recency threshold (mirrors AiChatMessageRepo.sweepStreaming,
// #183): when a staleness window is supplied, a 'running'/'pending' run is only
// swept to 'aborted' once it has been UNTOUCHED for this long, so a sibling
// replica's boot-sweep can never abort a run another replica is actively
// executing. The runner bumps `updatedAt` on every step, so a live run never
// matches. PHASE 1 is single-process and the boot sweep passes NO window (every
// dangling run is settled unconditionally — see sweepRunning / F1). This constant
// is the window to reintroduce for the phase-2 multi-instance timer sweep.
export const SWEEP_RUN_STALE_MS = 10 * 60 * 1000; // 10 minutes
/**
* Repository for `ai_chat_runs` (#184 phase 1): the agent run as a first-class,
* server-side lifecycle object detached from the HTTP request. The run row is the
* point a client subscribes/reconnects to (by `id` or by chat); the assistant
* message it links to (`assistantMessageId`) is the #183 projection of its output.
*/
@Injectable()
export class AiChatRunRepo {
private readonly logger = new Logger(AiChatRunRepo.name);
private baseFields: Array<keyof AiChatRun> = [
'id',
'chatId',
'workspaceId',
'createdBy',
'assistantMessageId',
'trigger',
'status',
'error',
'stepCount',
'stopRequestedAt',
'startedAt',
'finishedAt',
'createdAt',
'updatedAt',
];
constructor(@InjectKysely() private readonly db: KyselyDB) {}
async insert(
insertable: InsertableAiChatRun,
trx?: KyselyTransaction,
): Promise<AiChatRun> {
const db = dbOrTx(this.db, trx);
return db
.insertInto('aiChatRuns')
.values(insertable)
.returning(this.baseFields)
.executeTakeFirst();
}
async findById(
id: string,
workspaceId: string,
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
return db
.selectFrom('aiChatRuns')
.select(this.baseFields)
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.executeTakeFirst();
}
/** The currently-active (pending|running) run for a chat, if any. At most one
* exists thanks to the partial unique index. */
async findActiveByChat(
chatId: string,
workspaceId: string,
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
return db
.selectFrom('aiChatRuns')
.select(this.baseFields)
.where('chatId', '=', chatId)
.where('workspaceId', '=', workspaceId)
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.executeTakeFirst();
}
/** The most-recent run for a chat (active or settled) — the reconnect target. */
async findLatestByChat(
chatId: string,
workspaceId: string,
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
return db
.selectFrom('aiChatRuns')
.select(this.baseFields)
.where('chatId', '=', chatId)
.where('workspaceId', '=', workspaceId)
.orderBy('createdAt', 'desc')
.orderBy('id', 'desc')
.limit(1)
.executeTakeFirst();
}
/**
* Patch a run by id + workspace; always bumps `updatedAt`. Used for every
* lifecycle transition (mark running, link the assistant message, bump
* step_count, finalize succeeded/failed/aborted). Returns the updated row or
* undefined when nothing matched (e.g. a foreign workspace).
*/
async update(
id: string,
workspaceId: string,
patch: Partial<{
status: string;
error: string | null;
stepCount: number;
assistantMessageId: string | null;
stopRequestedAt: Date | null;
startedAt: Date | null;
finishedAt: Date | null;
}>,
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatRuns')
.set({ ...(patch as Record<string, unknown>), updatedAt: new Date() })
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* Mark an EXPLICIT stop request on an active run (distinct from a browser
* disconnect, which never stops a run). Stamps `stop_requested_at` ONLY while
* the run is still active, so a late stop on an already-settled run is a no-op.
* Returns the row when a stop was recorded, else undefined (nothing active).
*/
async markStopRequested(
id: string,
workspaceId: string,
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatRuns')
.set({ stopRequestedAt: new Date(), updatedAt: new Date() })
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* Crash-recovery sweep (mirrors AiChatMessageRepo.sweepStreaming): flip every
* run still left pending/running a run whose process died before reaching a
* terminal status to 'aborted', stamping `finished_at`. Returns the number
* swept. Workspace-wide on purpose (a crash can dangle runs in any workspace).
*
* F1 (DECISION C): the BOOT sweep is UNCONDITIONAL it passes no `staleMs`, so
* EVERY dangling run is settled regardless of how recently it was touched. On a
* fresh single-process boot any pending|running run is definitionally hung (no
* runner is alive to own it), so a fast restart (deploy/OOM within minutes of
* the last step) no longer leaves a run stuck 'running' forever which would
* make the one-active-run gate 409 every future turn in that chat.
*
* The optional `staleMs` window is reintroduced ONLY for the future phase-2
* multi-instance timer sweep (see {@link SWEEP_RUN_STALE_MS}): there a booting
* replica must NOT abort a run another replica is actively executing, so it
* sweeps only runs UNTOUCHED past the window. Phase 1 is single-process, so the
* boot path supplies no window.
*/
async sweepRunning(
opts: { staleMs?: number } = {},
trx?: KyselyTransaction,
): Promise<number> {
const db = dbOrTx(this.db, trx);
const now = new Date();
let query = db
.updateTable('aiChatRuns')
.set({
status: 'aborted',
finishedAt: now,
updatedAt: now,
error: sql`coalesce(error, ${'Run interrupted by a server restart.'})`,
})
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[]);
// Multi-instance (phase 2) only: skip runs touched within the window so a
// sibling replica's live run is never aborted. Omitted on the phase-1 boot
// sweep -> unconditional.
if (typeof opts.staleMs === 'number') {
const staleBefore = new Date(now.getTime() - opts.staleMs);
query = query.where('updatedAt', '<', staleBefore);
}
const rows = await query.returning('id').execute();
return rows.length;
}
}
@@ -657,9 +657,8 @@ export class PagePermissionRepo {
pageIds: string[]; pageIds: string[];
userId: string; userId: string;
spaceId?: string; spaceId?: string;
workspaceId?: string | null;
}): Promise<string[]> { }): Promise<string[]> {
const { pageIds, userId, spaceId, workspaceId } = opts; const { pageIds, userId, spaceId } = opts;
if (pageIds.length === 0) return []; if (pageIds.length === 0) return [];
if (spaceId) { if (spaceId) {
@@ -667,17 +666,6 @@ export class PagePermissionRepo {
if (!hasRestrictions) { if (!hasRestrictions) {
return pageIds; return pageIds;
} }
} else if (workspaceId) {
// #348 — whole-workspace callers (no spaceId: favorites, notifications,
// recent, created-by, global search) skip the recursive-ancestor CTE + anti
// -join entirely when the workspace has ZERO restricted pages. When any
// restriction DOES exist, fall through to the identical CTE below, so
// behavior is unchanged whenever restrictions are present.
const hasRestrictions =
await this.hasRestrictedPagesInWorkspace(workspaceId);
if (!hasRestrictions) {
return pageIds;
}
} }
const results = await this.db const results = await this.db
@@ -915,39 +903,6 @@ export class PagePermissionRepo {
return Boolean(result?.exists); return Boolean(result?.exists);
} }
/**
* Workspace-level analogue of hasRestrictedPagesInSpace: does ANY page in the
* whole workspace carry a restriction? Lets whole-workspace access filters
* short-circuit the recursive-ancestor CTE when nothing is restricted at all.
*
* UNCACHED (like the sibling hasRestrictedPagesInSpace) a single cheap
* `EXISTS(pageAccess WHERE workspaceId=?)` per call. This is an ACCESS-CONTROL
* gate on whole-workspace list endpoints, so it must never go stale: caching it
* (even 5s) reintroduced a leak the space-path never had a concurrent
* whole-workspace read in the insert->commit window of the FIRST restricted page
* could re-populate `false` under withCache (read-then-set, no del-during-read
* guard) and override the insert bust, leaking that page to unauthorized users
* for up to the TTL (#348 review F1). An uncached EXISTS removes both the
* cache/DB asymmetry with hasRestrictedPagesInSpace and that race; the space
* path already accepts this exact per-call cost.
*/
async hasRestrictedPagesInWorkspace(workspaceId: string): Promise<boolean> {
const result = await this.db
.selectNoFrom((eb) =>
eb
.exists(
eb
.selectFrom('pageAccess')
.select(sql`1`.as('one'))
.where('pageAccess.workspaceId', '=', workspaceId),
)
.as('exists'),
)
.executeTakeFirst();
return Boolean(result?.exists);
}
/** /**
* Given a list of parent page IDs, return which ones have at least one accessible child. * Given a list of parent page IDs, return which ones have at least one accessible child.
* Efficient batch query for sidebar hasChildren calculation. * Efficient batch query for sidebar hasChildren calculation.
@@ -581,9 +581,6 @@ export class PageRepo {
const query = this.db const query = this.db
.selectFrom('pages') .selectFrom('pages')
.select(this.baseFields) .select(this.baseFields)
// NOTE: `content` IS needed here — the trash UI reads page.content to render
// the deleted-page preview modal (trash.tsx handlePageClick ->
// TrashPageContentModal pageContent). Do NOT drop it (see #348 review F3).
.select('content') .select('content')
.select((eb) => this.withSpace(eb)) .select((eb) => this.withSpace(eb))
.select((eb) => this.withDeletedBy(eb)) .select((eb) => this.withDeletedBy(eb))
@@ -1,6 +1,4 @@
import { Inject, Injectable } from '@nestjs/common'; import { Injectable } from '@nestjs/common';
import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Cache } from 'cache-manager';
import { InjectKysely } from 'nestjs-kysely'; import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types'; import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils'; import { dbOrTx } from '../../utils';
@@ -11,7 +9,6 @@ import {
} from '@docmost/db/types/entity.types'; } from '@docmost/db/types/entity.types';
import { ExpressionBuilder, sql } from 'kysely'; import { ExpressionBuilder, sql } from 'kysely';
import { DB, Workspaces } from '@docmost/db/types/db'; import { DB, Workspaces } from '@docmost/db/types/db';
import { CacheKey } from '../../../common/helpers/cache-keys';
/** /**
* Writable `settings.ai.provider` keys, enforced at this generic SQL layer. This * Writable `settings.ai.provider` keys, enforced at this generic SQL layer. This
@@ -64,34 +61,7 @@ export class WorkspaceRepo {
'temporaryNoteHours', 'temporaryNoteHours',
'isScimEnabled', 'isScimEnabled',
]; ];
constructor( constructor(@InjectKysely() private readonly db: KyselyDB) {}
@InjectKysely() private readonly db: KyselyDB,
@Inject(CACHE_MANAGER) private readonly cacheManager: Cache,
) {}
/**
* #348 bust the DomainMiddleware workspace caches after any workspace write.
* Deletes BOTH the self-hosted (constant) key and the cloud per-hostname key so
* a single implementation covers either deployment mode (the irrelevant key is a
* harmless no-op). Best-effort: a cache error must never fail the write, and a
* missed bust is bounded by WORKSPACE_CACHE_TTL_MS. Note: a hostname RENAME only
* busts the NEW hostname's key (the row returned here carries the new hostname);
* the old key expires via TTL.
*/
private async bustWorkspaceCache(
workspace?: Pick<Workspace, 'hostname'> | undefined,
): Promise<void> {
try {
await this.cacheManager.del(CacheKey.WORKSPACE_SELF_HOSTED);
if (workspace?.hostname) {
await this.cacheManager.del(
CacheKey.WORKSPACE_BY_HOST(workspace.hostname),
);
}
} catch {
// cache is best-effort; TTL is the backstop
}
}
async findById( async findById(
workspaceId: string, workspaceId: string,
@@ -174,14 +144,12 @@ export class WorkspaceRepo {
trx?: KyselyTransaction, trx?: KyselyTransaction,
): Promise<Workspace> { ): Promise<Workspace> {
const db = dbOrTx(this.db, trx); const db = dbOrTx(this.db, trx);
const workspace = await db return db
.updateTable('workspaces') .updateTable('workspaces')
.set({ ...updatableWorkspace, updatedAt: new Date() }) .set({ ...updatableWorkspace, updatedAt: new Date() })
.where('id', '=', workspaceId) .where('id', '=', workspaceId)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
await this.bustWorkspaceCache(workspace);
return workspace;
} }
async insertWorkspace( async insertWorkspace(
@@ -189,14 +157,11 @@ export class WorkspaceRepo {
trx?: KyselyTransaction, trx?: KyselyTransaction,
): Promise<Workspace> { ): Promise<Workspace> {
const db = dbOrTx(this.db, trx); const db = dbOrTx(this.db, trx);
const workspace = await db return db
.insertInto('workspaces') .insertInto('workspaces')
.values(insertableWorkspace) .values(insertableWorkspace)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
// Bust the cached "not found" so a fresh install / new tenant is seen at once.
await this.bustWorkspaceCache(workspace);
return workspace;
} }
async count(): Promise<number> { async count(): Promise<number> {
@@ -238,7 +203,7 @@ export class WorkspaceRepo {
trx?: KyselyTransaction, trx?: KyselyTransaction,
) { ) {
const db = dbOrTx(this.db, trx); const db = dbOrTx(this.db, trx);
const workspace = await db return db
.updateTable('workspaces') .updateTable('workspaces')
.set({ .set({
settings: sql`COALESCE(settings, '{}'::jsonb) settings: sql`COALESCE(settings, '{}'::jsonb)
@@ -249,8 +214,6 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId) .where('id', '=', workspaceId)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
await this.bustWorkspaceCache(workspace);
return workspace;
} }
async updateAiSettings( async updateAiSettings(
@@ -260,7 +223,7 @@ export class WorkspaceRepo {
trx?: KyselyTransaction, trx?: KyselyTransaction,
) { ) {
const db = dbOrTx(this.db, trx); const db = dbOrTx(this.db, trx);
const workspace = await db return db
.updateTable('workspaces') .updateTable('workspaces')
.set({ .set({
settings: sql`COALESCE(settings, '{}'::jsonb) settings: sql`COALESCE(settings, '{}'::jsonb)
@@ -271,8 +234,6 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId) .where('id', '=', workspaceId)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
await this.bustWorkspaceCache(workspace);
return workspace;
} }
/** /**
@@ -311,7 +272,7 @@ export class WorkspaceRepo {
entries.flatMap(([k, v]) => [sql.lit(k), sql`${v}::text`]), entries.flatMap(([k, v]) => [sql.lit(k), sql`${v}::text`]),
)})` )})`
: sql`'{}'::jsonb`; : sql`'{}'::jsonb`;
const workspace = await db return db
.updateTable('workspaces') .updateTable('workspaces')
.set({ .set({
settings: sql`COALESCE(settings, '{}'::jsonb) || jsonb_build_object( settings: sql`COALESCE(settings, '{}'::jsonb) || jsonb_build_object(
@@ -326,8 +287,6 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId) .where('id', '=', workspaceId)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
await this.bustWorkspaceCache(workspace);
return workspace;
} }
/** /**
@@ -344,7 +303,7 @@ export class WorkspaceRepo {
trx?: KyselyTransaction, trx?: KyselyTransaction,
) { ) {
const db = dbOrTx(this.db, trx); const db = dbOrTx(this.db, trx);
const workspace = await db return db
.updateTable('workspaces') .updateTable('workspaces')
.set({ .set({
settings: sql`COALESCE(settings, '{}'::jsonb) settings: sql`COALESCE(settings, '{}'::jsonb)
@@ -354,8 +313,6 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId) .where('id', '=', workspaceId)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
await this.bustWorkspaceCache(workspace);
return workspace;
} }
async updateSharingSettings( async updateSharingSettings(
@@ -365,7 +322,7 @@ export class WorkspaceRepo {
trx?: KyselyTransaction, trx?: KyselyTransaction,
) { ) {
const db = dbOrTx(this.db, trx); const db = dbOrTx(this.db, trx);
const workspace = await db return db
.updateTable('workspaces') .updateTable('workspaces')
.set({ .set({
settings: sql`COALESCE(settings, '{}'::jsonb) settings: sql`COALESCE(settings, '{}'::jsonb)
@@ -376,8 +333,6 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId) .where('id', '=', workspaceId)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
await this.bustWorkspaceCache(workspace);
return workspace;
} }
async updateTemplateSettings( async updateTemplateSettings(
@@ -387,7 +342,7 @@ export class WorkspaceRepo {
trx?: KyselyTransaction, trx?: KyselyTransaction,
) { ) {
const db = dbOrTx(this.db, trx); const db = dbOrTx(this.db, trx);
const workspace = await db return db
.updateTable('workspaces') .updateTable('workspaces')
.set({ .set({
settings: sql`COALESCE(settings, '{}'::jsonb) settings: sql`COALESCE(settings, '{}'::jsonb)
@@ -398,8 +353,6 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId) .where('id', '=', workspaceId)
.returning(this.baseFields) .returning(this.baseFields)
.executeTakeFirst(); .executeTakeFirst();
await this.bustWorkspaceCache(workspace);
return workspace;
} }
} }
@@ -19,6 +19,16 @@ export class MigrationService {
path, path,
migrationFolder: path.join(__dirname, '..', 'migrations'), migrationFolder: path.join(__dirname, '..', 'migrations'),
}), }),
// A long-lived branch can add a migration whose timestamped filename sorts
// BEFORE migrations already applied in prod (e.g. #234's 20260627 landing
// after 20260704 was live). With the default (ordered) setting the startup
// migrator then sees "corrupted migrations" — the applied set is no longer a
// prefix of the sorted list — throws, and the app crash-loops on boot
// (incident #361: 502s for ~11 min). allowUnorderedMigrations runs any
// not-yet-applied migration regardless of filename order, so a back-dated
// migration is applied instead of bricking startup. A CI order-gate still
// discourages back-dating; this is the runtime safety net.
allowUnorderedMigrations: true,
}); });
const { error, results } = await migrator.migrateToLatest(); const { error, results } = await migrator.migrateToLatest();
+30
View File
@@ -659,6 +659,35 @@ export interface AiChatMessages {
deletedAt: Timestamp | null; deletedAt: Timestamp | null;
} }
// The agent RUN as a first-class server-side lifecycle object (#184 phase 1).
// Mirrors migration 20260704T130000-ai-chat-runs.ts. A run is created when an
// agent turn starts and survives the browser disconnecting; the DB is the source
// of truth a later client reconnects to. `assistantMessageId` links to the #183
// projection row (the assistant message this run materializes).
export interface AiChatRuns {
id: Generated<string>;
chatId: string;
workspaceId: string;
// SET NULL on user deletion (the run history outlives its author); also NULL
// for a future non-human trigger (cron/api).
createdBy: string | null;
// The assistant message this run materializes; SET NULL if it is pruned.
assistantMessageId: string | null;
// 'user' | 'autostart' | 'schedule' | 'api' | 'continue' (only 'user' is
// produced in phase 1; the rest are reserved for the deferred autonomy triggers).
trigger: Generated<string>;
// 'pending' | 'running' | 'succeeded' | 'failed' | 'aborted'.
status: Generated<string>;
error: string | null;
stepCount: Generated<number>;
// Set when an EXPLICIT user stop is requested (distinct from a disconnect).
stopRequestedAt: Timestamp | null;
startedAt: Timestamp | null;
finishedAt: Timestamp | null;
createdAt: Generated<Timestamp>;
updatedAt: Generated<Timestamp>;
}
// Per-(chat,page) snapshot of the open page's Markdown at the END of the agent's // Per-(chat,page) snapshot of the open page's Markdown at the END of the agent's
// previous turn (#274). Mirrors migration 20260702T120000-ai-chat-page-snapshot.ts. // previous turn (#274). Mirrors migration 20260702T120000-ai-chat-page-snapshot.ts.
// The next turn diffs the CURRENT Markdown against `contentMd` to surface edits a // The next turn diffs the CURRENT Markdown against `contentMd` to surface edits a
@@ -695,6 +724,7 @@ export interface DB {
aiAgentRoles: AiAgentRoles; aiAgentRoles: AiAgentRoles;
aiChats: AiChats; aiChats: AiChats;
aiChatMessages: AiChatMessages; aiChatMessages: AiChatMessages;
aiChatRuns: AiChatRuns;
aiChatPageSnapshots: AiChatPageSnapshots; aiChatPageSnapshots: AiChatPageSnapshots;
apiKeys: ApiKeys; apiKeys: ApiKeys;
attachments: Attachments; attachments: Attachments;
+15 -7
View File
@@ -3,6 +3,7 @@ import {
AiAgentRoles, AiAgentRoles,
AiChats, AiChats,
AiChatMessages, AiChatMessages,
AiChatRuns,
AiChatPageSnapshots, AiChatPageSnapshots,
Attachments, Attachments,
Comments, Comments,
@@ -56,10 +57,12 @@ export type UpdatableAiChat = Updateable<Omit<AiChats, 'id'>>;
// full-text search. It is omitted from the public type so it never leaks // full-text search. It is omitted from the public type so it never leaks
// into HTTP responses or the chat history fed to the language model. // into HTTP responses or the chat history fed to the language model.
export type AiChatMessage = Omit<Selectable<AiChatMessages>, 'tsv'>; export type AiChatMessage = Omit<Selectable<AiChatMessages>, 'tsv'>;
export type InsertableAiChatMessage = Omit< export type InsertableAiChatMessage = Omit<Insertable<AiChatMessages>, 'tsv'>;
Insertable<AiChatMessages>,
'tsv' // AI Chat Run (#184 phase 1): the agent run as a first-class lifecycle object,
>; // detached from the HTTP request / browser window.
export type AiChatRun = Selectable<AiChatRuns>;
export type InsertableAiChatRun = Insertable<AiChatRuns>;
// AI Chat Page Snapshot (#274): per-(chat,page) Markdown snapshot taken at the // AI Chat Page Snapshot (#274): per-(chat,page) Markdown snapshot taken at the
// end of the agent's previous turn, diffed against the current page next turn to // end of the agent's previous turn, diffed against the current page next turn to
@@ -214,11 +217,14 @@ export type UpdatableFavorite = Updateable<Omit<Favorites, 'id'>>;
// Page Transclusion // Page Transclusion
export type PageTransclusion = Selectable<PageTransclusions>; export type PageTransclusion = Selectable<PageTransclusions>;
export type InsertablePageTransclusion = Insertable<PageTransclusions>; export type InsertablePageTransclusion = Insertable<PageTransclusions>;
export type UpdatablePageTransclusion = Updateable<Omit<PageTransclusions, 'id'>>; export type UpdatablePageTransclusion = Updateable<
Omit<PageTransclusions, 'id'>
>;
// Page Transclusion Reference // Page Transclusion Reference
export type PageTransclusionReference = Selectable<PageTransclusionReferences>; export type PageTransclusionReference = Selectable<PageTransclusionReferences>;
export type InsertablePageTransclusionReference = Insertable<PageTransclusionReferences>; export type InsertablePageTransclusionReference =
Insertable<PageTransclusionReferences>;
export type UpdatablePageTransclusionReference = Updateable< export type UpdatablePageTransclusionReference = Updateable<
Omit<PageTransclusionReferences, 'id'> Omit<PageTransclusionReferences, 'id'>
>; >;
@@ -288,7 +294,9 @@ export type UpdatablePagePermission = Updateable<Omit<_PagePermissions, 'id'>>;
// Page Verification // Page Verification
export type PageVerification = Selectable<_PageVerifications>; export type PageVerification = Selectable<_PageVerifications>;
export type InsertablePageVerification = Insertable<_PageVerifications>; export type InsertablePageVerification = Insertable<_PageVerifications>;
export type UpdatablePageVerification = Updateable<Omit<_PageVerifications, 'id'>>; export type UpdatablePageVerification = Updateable<
Omit<_PageVerifications, 'id'>
>;
// Page Verifier // Page Verifier
export type PageVerifier = Selectable<_PageVerifiers>; export type PageVerifier = Selectable<_PageVerifiers>;
@@ -0,0 +1,124 @@
import { readFileSync } from 'fs';
import { streamText, Output } from 'ai';
import { MockLanguageModelV3, simulateReadableStream } from 'ai/test';
/**
* Regression tests for patches/ai@6.0.134.patch (server heap OOM on long
* autonomous agent runs, #184).
*
* Unpatched ai@6.0.134 substitutes the default text() output strategy even
* when the caller passes NO `output` option. Its createOutputTransformStream
* then accumulates the ENTIRE turn text and, on EVERY text-delta, enqueues a
* flat snapshot of all text so far as `partialOutput` (O(n^2) memory). Those
* snapshots pile up in the never-consumed leftover tee() branch of
* DefaultStreamTextResult.baseStream, which is what OOM'd production during a
* ~28k-chunk agent turn. The pnpm patch skips partialOutput production
* entirely when no output strategy was requested, while keeping per-delta
* streaming granularity.
*/
describe('ai@6.0.134 pnpm patch: no partialOutput accumulation without an output strategy', () => {
const makeModel = () =>
new MockLanguageModelV3({
doStream: async () => ({
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'text-start' as const, id: '1' },
{ type: 'text-delta' as const, id: '1', delta: 'Hello' },
{ type: 'text-delta' as const, id: '1', delta: ', ' },
{ type: 'text-delta' as const, id: '1', delta: 'world!' },
{ type: 'text-end' as const, id: '1' },
{
type: 'finish' as const,
finishReason: { unified: 'stop' as const, raw: 'stop' },
usage: {
inputTokens: {
total: 1,
noCache: undefined,
cacheRead: undefined,
cacheWrite: undefined,
},
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
}),
});
it('preserves per-delta streaming granularity in textStream', async () => {
const result = streamText({ model: makeModel(), prompt: 'hi' });
const deltas: string[] = [];
for await (const delta of result.textStream) {
deltas.push(delta);
}
// The patch must NOT coalesce or drop deltas: three model deltas arrive
// as three separate textStream chunks.
expect(deltas).toEqual(['Hello', ', ', 'world!']);
});
it('emits NO partialOutput values when the caller did not request an output strategy', async () => {
const result = streamText({ model: makeModel(), prompt: 'hi' });
// Fully consume the primary stream first (mirrors production usage).
for await (const _ of result.textStream) {
// drain
}
const partials: unknown[] = [];
for await (const partial of result.experimental_partialOutputStream) {
partials.push(partial);
}
// TRIPWIRE: on unpatched ai@6.0.134 the default text() output strategy
// yields one cumulative partial per text-delta here (['Hello', 'Hello, ',
// 'Hello, world!']). An empty stream proves the patch is applied and no
// cumulative snapshots are being produced (and thus none can pile up in
// the leftover internal tee branch).
expect(partials).toEqual([]);
});
it('preserves cumulative partialOutput when the caller DOES request an output strategy', async () => {
// PRESERVE-BRANCH GUARD: the patch only short-circuits partialOutput when
// `output == null`. When an output strategy IS set (here Output.text()),
// createOutputTransformStream must fall through to the ORIGINAL code path
// and keep publishing cumulative snapshots, so object/text-output consumers
// behave byte-identically to unpatched ai. A careless re-port that routed
// output-set calls into the skip branch would leave partialOutput empty and
// silently break those consumers — this test is the tripwire for that.
const result = streamText({
model: makeModel(),
prompt: 'hi',
experimental_output: Output.text(),
});
// Drain the primary stream fully and accumulate the complete output text.
let fullText = '';
for await (const delta of result.textStream) {
fullText += delta;
}
const partials: string[] = [];
for await (const partial of result.experimental_partialOutputStream) {
partials.push(partial);
}
// With a strategy set, partialOutput must be PRESERVED (non-empty) and
// cumulative: the last emitted partial equals the full accumulated text.
expect(partials.length).toBeGreaterThan(0);
expect(partials[partials.length - 1]).toBe(fullText);
expect(fullText).toBe('Hello, world!');
});
it('both installed dist builds (CJS and ESM) carry the patch marker', () => {
// Secondary guard: pins the patch to BOTH bundles the SDK ships, since
// the NestJS server consumes CJS while other tooling may load ESM.
const cjsPath = require.resolve('ai');
const mjsPath = cjsPath.replace(/index\.js$/, 'index.mjs');
expect(cjsPath).toMatch(/index\.js$/);
expect(readFileSync(cjsPath, 'utf8')).toContain('PATCH(docmost');
expect(readFileSync(mjsPath, 'utf8')).toContain('PATCH(docmost');
});
});
@@ -0,0 +1,145 @@
// export.service.ts imports the ESM-only @sindresorhus/slugify (not in jest's
// transform allowlist). It is irrelevant to the markdown-serialization path under
// test (only used for page-mention link slugs on the DB path), so it is mocked
// out to keep the module graph loadable under ts-jest (mirrors the import specs).
jest.mock('@sindresorhus/slugify', () => ({
__esModule: true,
default: (input: string) => String(input),
}));
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
import { ExportService } from './export.service';
import { ExportFormat } from './dto/export-dto';
/**
* STEP 1 golden test for issue #345: server MARKDOWN export runs DIRECTLY through
* the canonical converter (`convertProseMirrorToMarkdown`) no HTML intermediate
* and no `@docmost/editor-ext` markdown layer so the emitted markdown is in the
* canonical package forms and is byte-identical to the git-sync vault body.
*
* These are the goldens the swap has to satisfy: they assert the CANONICAL
* surface (callout `> [!type]`, inline footnote `^[…]`, lossless image
* `<!--img …-->`) rather than the old editor-ext forms (`:::type`, `[^id]`,
* lossy `![alt](src)`).
*
* `exportPage(..., singlePage=false)` takes no DB path (no mention rewriting), so
* the service is constructed with null collaborators and only the pure
* PM -> Markdown path is exercised.
*/
function makeService(): ExportService {
return new ExportService(
null as any, // pageRepo
null as any, // pagePermissionRepo
null as any, // db
null as any, // storageService
null as any, // environmentService
null as any, // domainService
);
}
// A representative page exercising the node types whose canonical markdown form
// changed with the move off the editor-ext layer: callout, inline footnote, and a
// lossless image carrying width/align attrs that the old layer dropped.
const REPRESENTATIVE_DOC = {
type: 'doc',
content: [
{
type: 'paragraph',
content: [
{ type: 'text', text: 'Body ' },
{ type: 'footnoteReference', attrs: { id: 'fn-1' } },
{ type: 'text', text: ' end.' },
],
},
{
type: 'callout',
attrs: { type: 'info', icon: null },
content: [
{
type: 'paragraph',
content: [{ type: 'text', text: 'Heads up' }],
},
],
},
{
type: 'image',
attrs: {
src: '/files/pic.png',
alt: 'Pic',
width: 320,
align: 'left',
},
},
{
type: 'footnotesList',
content: [
{
type: 'footnoteDefinition',
attrs: { id: 'fn-1' },
content: [
{
type: 'paragraph',
content: [{ type: 'text', text: 'the note' }],
},
],
},
],
},
],
};
describe('ExportService — markdown export via the canonical converter (#345)', () => {
it('emits canonical callout, inline footnote and lossless image forms', async () => {
const service = makeService();
const md = (await service.exportPage(ExportFormat.Markdown, {
title: '',
content: REPRESENTATIVE_DOC,
} as any)) as string;
// Callout: Obsidian `> [!type]`, NOT the legacy `:::type`.
expect(md).toContain('> [!info]');
expect(md).not.toContain(':::');
// Inline footnote: `^[…]`, NOT the reference `[^id]` form.
expect(md).toContain('^[the note]');
expect(md).not.toMatch(/\[\^/);
// Lossless image: trailing `<!--img …-->` carrying the dropped attrs.
expect(md).toContain('![Pic](/files/pic.png)');
expect(md).toContain('<!--img');
expect(md).toContain('"width":"320"');
expect(md).toContain('"align":"left"');
});
it('export body is byte-identical to the git-sync vault serializer (export == vault)', async () => {
const service = makeService();
// A title-less page: exportPage prepends NO heading, so the whole output is
// the page BODY — exactly what git-sync serializes (git-sync stores the title
// in frontmatter / the filename, never as an in-body H1).
const exported = (await service.exportPage(ExportFormat.Markdown, {
title: '',
content: REPRESENTATIVE_DOC,
} as any)) as string;
// The git-sync vault writer feeds this SAME converter (git-sync
// `stabilizePageBody` = convertProseMirrorToMarkdown(content) at the
// fixpoint). For an already-stable doc the single pass IS the fixpoint, so
// the two are byte-identical by construction — assert it.
const vaultBody = convertProseMirrorToMarkdown(REPRESENTATIVE_DOC);
expect(exported).toBe(vaultBody);
});
it('prepends the page title as an H1 heading (the one documented export/vault delta)', async () => {
const service = makeService();
const md = (await service.exportPage(ExportFormat.Markdown, {
title: 'My Page',
content: { type: 'doc', content: [] },
} as any)) as string;
// Export makes standalone files, so it prepends the title as an H1. This is
// the ONE deliberate difference from the vault body (which carries the title
// in frontmatter). The body below the heading still serializes canonically.
expect(md.startsWith('# My Page')).toBe(true);
});
});
@@ -37,7 +37,7 @@ import {
getAttachmentIds, getAttachmentIds,
getProsemirrorContent, getProsemirrorContent,
} from '../../common/helpers/prosemirror/utils'; } from '../../common/helpers/prosemirror/utils';
import { htmlToMarkdown } from '@docmost/editor-ext'; import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
type AllowedAttachment = { id: string; fileName: string; filePath: string }; type AllowedAttachment = { id: string; fileName: string; filePath: string };
@@ -79,9 +79,8 @@ export class ExportService {
prosemirrorJson.content.unshift(titleNode); prosemirrorJson.content.unshift(titleNode);
} }
const pageHtml = jsonToHtml(prosemirrorJson);
if (format === ExportFormat.HTML) { if (format === ExportFormat.HTML) {
const pageHtml = jsonToHtml(prosemirrorJson);
return `<!DOCTYPE html> return `<!DOCTYPE html>
<html> <html>
<head> <head>
@@ -92,11 +91,14 @@ export class ExportService {
} }
if (format === ExportFormat.Markdown) { if (format === ExportFormat.Markdown) {
const newPageHtml = pageHtml.replace( // Direct ProseMirror JSON -> Markdown via the canonical converter
/<colgroup[^>]*>[\s\S]*?<\/colgroup>/gim, // (`@docmost/prosemirror-markdown`). This is the SAME serializer the
'', // git-sync vault writer feeds (see git-sync `stabilizePageBody`), so an
); // exported page body is byte-identical to its vault representation — no
return htmlToMarkdown(newPageHtml); // HTML intermediate, no second markdown layer, no format drift (issue
// #345). The old `<colgroup>` scrub is gone with the HTML step: the
// converter emits GFM tables directly and never produces `<colgroup>`.
return convertProseMirrorToMarkdown(prosemirrorJson);
} }
return; return;
@@ -17,6 +17,22 @@ jest.mock('image-dimensions', () => ({
__esModule: true, __esModule: true,
imageDimensionsFromData: () => undefined, imageDimensionsFromData: () => undefined,
})); }));
// FileImportTaskService -> PageService -> collaboration.gateway ->
// metrics.registry imports `prom-client`, which is not resolvable in this
// workspace's node_modules (types-only stub, no runtime entry). Metrics are
// disabled on this path, so a virtual no-op mock keeps the module graph loadable.
jest.mock(
'prom-client',
() => ({
collectDefaultMetrics: () => undefined,
Registry: class {},
Histogram: class {},
Gauge: class {},
Counter: class {},
Summary: class {},
}),
{ virtual: true },
);
import { promises as fs } from 'fs'; import { promises as fs } from 'fs';
import * as os from 'os'; import * as os from 'os';
@@ -26,14 +42,17 @@ import { ImportService } from './import.service';
/** /**
* Binding test for issue #228 / review #5: FileImportTaskService.processGenericImport * Binding test for issue #228 / review #5: FileImportTaskService.processGenericImport
* is a NON-editor write path (markdownToHtml -> processHTML -> JSON, never runs * is a NON-editor write path, so a zip-imported `.md` page ends up with canonical
* footnoteSyncPlugin), so it canonicalizes footnotes before persisting. This pins * footnotes before persisting: ordered by first reference, reused refs deduped,
* that binding the same one import.service has a spec for which previously had * orphan definitions dropped.
* NO spec at all.
* *
* The markdown -> HTML -> ProseMirror conversion is REAL (a real ImportService, * Since #345 the `.md` parse runs `normalizeForeignMarkdown` ->
* its createYdoc stubbed); the filesystem is a real temp dir with one .md file; * `markdownToProseMirror` -> `jsonToHtml` (feeding the shared HTML attachment /
* the DB transaction is stubbed to capture the persisted page content. * link pipeline) -> `processHTML` -> `canonicalizeFootnotes`. The parser assigns
* fresh `fn-*` ids, so we assert by definition BODY order rather than the source
* labels. The conversion is REAL (a real ImportService, its createYdoc stubbed);
* the filesystem is a real temp dir with one .md file; the DB transaction is
* stubbed to capture the persisted page content.
*/ */
// Out-of-order references (c, a, b), a REUSED reference ([^a] twice), and an // Out-of-order references (c, a, b), a REUSED reference ([^a] twice), and an
@@ -49,13 +68,14 @@ const MARKDOWN = [
'[^z]: orphan note', '[^z]: orphan note',
].join('\n'); ].join('\n');
function footnoteListIds(content: any): string[] { /** Definition body texts of the (single) footnotesList, in list order. */
function footnoteListBodies(content: any): string[] {
const list = (content?.content ?? []).find( const list = (content?.content ?? []).find(
(n: any) => n.type === 'footnotesList', (n: any) => n.type === 'footnotesList',
); );
return (list?.content ?? []) return (list?.content ?? [])
.filter((n: any) => n.type === 'footnoteDefinition') .filter((n: any) => n.type === 'footnoteDefinition')
.map((n: any) => n.attrs?.id); .map((n: any) => n.content?.[0]?.content?.[0]?.text);
} }
// A permissive chainable stub for the spaces lookup (selectFrom(...).select(...) // A permissive chainable stub for the spaces lookup (selectFrom(...).select(...)
@@ -71,80 +91,127 @@ function chainable(result: any): any {
return proxy; return proxy;
} }
/**
* Run one markdown file through the REAL zip-import pipeline
* (`processGenericImport` -> `markdownToProseMirror` -> `jsonToHtml` ->
* `processHTML`/`htmlToJson`) and return the persisted page `content`. This is
* the server-specific PM->HTML->PM hop that the package's own PM<->MD tests do
* NOT cover.
*/
async function runZipImport(markdown: string): Promise<any> {
const extractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fit-canon-'));
await fs.writeFile(path.join(extractDir, 'note.md'), markdown, 'utf-8');
const importService = new ImportService(
{} as any,
{} as any,
{} as any,
{} as any,
);
jest
.spyOn(importService as any, 'createYdoc')
.mockResolvedValue(Buffer.from([]) as any);
let captured: any = null;
const trx = {
insertInto: (table: string) => ({
values: (v: any) => {
if (table === 'pages') captured = v;
return { execute: async () => {} };
},
}),
};
const db: any = {
selectFrom: () => chainable({ slug: 'space-slug' }),
transaction: () => ({ execute: (fn: any) => fn(trx) }),
};
const importAttachmentService = {
processAttachments: async ({ html }: any) => html,
};
const service = new FileImportTaskService(
{} as any, // storageService
importService as any,
{ nextPagePosition: async () => 'a0' } as any,
{ insertBacklink: jest.fn() } as any,
db,
importAttachmentService as any,
{ emit: jest.fn() } as any,
{ logBatchWithContext: jest.fn() } as any,
);
const fileTask: any = {
id: 'task-1',
source: 'generic',
spaceId: 'space-1',
workspaceId: 'ws-1',
creatorId: 'user-1',
};
try {
await service.processGenericImport({ extractDir, fileTask });
expect(captured).toBeTruthy();
return captured.content;
} finally {
await fs.rm(extractDir, { recursive: true, force: true });
}
}
/** Find the first node of a given type anywhere in a PM content tree. */
function findFirst(node: any, type: string): any {
if (!node || typeof node !== 'object') return null;
if (node.type === type) return node;
for (const child of node.content ?? []) {
const hit = findFirst(child, type);
if (hit) return hit;
}
return null;
}
describe('FileImportTaskService.processGenericImport — footnote canonicalization (#228)', () => { describe('FileImportTaskService.processGenericImport — footnote canonicalization (#228)', () => {
it('orders footnotes by first reference, dedupes reuse, and drops orphans on zip import', async () => { it('orders footnotes by first reference, dedupes reuse, and drops orphans on zip import', async () => {
const extractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fit-canon-')); const content = await runZipImport(MARKDOWN);
await fs.writeFile(path.join(extractDir, 'note.md'), MARKDOWN, 'utf-8'); // Definitions ordered by FIRST REFERENCE (C, A, B), NOT the markdown
// definition order (A, B, C). Ids are the parser's fresh `fn-*`, so pin
// Real ImportService for the html -> JSON conversion; stub the yjs encode. // the BODIES.
const importService = new ImportService( expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
{} as any, // Orphan [^z] dropped; reused [^a] collapses to one definition; one list.
{} as any, expect(footnoteListBodies(content)).not.toContain('orphan note');
{} as any, const lists = (content.content ?? []).filter(
{} as any, (n: any) => n.type === 'footnotesList',
); );
jest expect(lists).toHaveLength(1);
.spyOn(importService as any, 'createYdoc') expect(
.mockResolvedValue(Buffer.from([]) as any); footnoteListBodies(content).filter((b) => b === 'note A'),
).toHaveLength(1);
});
let captured: any = null; // #345 F4: the zip path routes markdown through jsonToHtml -> processHTML ->
const trx = { // htmlToJson (the shared HTML attachment pipeline). #345's headline is LOSSLESS
insertInto: (table: string) => ({ // image width/align via the `<!--img {...}-->` comment; a callout carries its
values: (v: any) => { // `type`. This asserts those survive the PM->HTML->PM hop — the one hop the
if (table === 'pages') captured = v; // package's PM<->MD suite does not exercise.
return { execute: async () => {} }; it('preserves image width/align and callout type through the PM->HTML->PM hop', async () => {
}, const md = [
}), '# Doc',
}; '',
const db: any = { '![a picture](https://example.com/i.png) <!--img {"width":"320","align":"left"}-->',
selectFrom: () => chainable({ slug: 'space-slug' }), '',
transaction: () => ({ execute: (fn: any) => fn(trx) }), ':::warning',
}; 'Careful now.',
':::',
].join('\n');
const importAttachmentService = { const content = await runZipImport(md);
processAttachments: async ({ html }: any) => html,
};
const backlinkRepo = { insertBacklink: jest.fn() };
const eventEmitter = { emit: jest.fn() };
const auditService = { logBatchWithContext: jest.fn() };
const pageService = { nextPagePosition: async () => 'a0' }; const image = findFirst(content, 'image');
expect(image).toBeTruthy();
// The lossless sizing/alignment must survive the HTML hop.
expect(String(image.attrs?.width)).toBe('320');
expect(image.attrs?.align).toBe('left');
const service = new FileImportTaskService( const callout = findFirst(content, 'callout');
{} as any, // storageService expect(callout).toBeTruthy();
importService as any, expect(callout.attrs?.type).toBe('warning');
pageService as any,
backlinkRepo as any,
db,
importAttachmentService as any,
eventEmitter as any,
auditService as any,
);
const fileTask: any = {
id: 'task-1',
source: 'generic',
spaceId: 'space-1',
workspaceId: 'ws-1',
creatorId: 'user-1',
};
try {
await service.processGenericImport({ extractDir, fileTask });
expect(captured).toBeTruthy();
const content = captured.content;
// Reference order is c, a, b (NOT the markdown definition order a, b, c).
expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']);
// Orphan [^z] dropped; reused [^a] collapses to one definition; one list.
expect(footnoteListIds(content)).not.toContain('z');
const lists = (content.content ?? []).filter(
(n: any) => n.type === 'footnotesList',
);
expect(lists).toHaveLength(1);
expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1);
} finally {
await fs.rm(extractDir, { recursive: true, force: true });
}
}); });
}); });
@@ -1,6 +1,9 @@
import { Inject, Injectable, Logger } from '@nestjs/common'; import { Inject, Injectable, Logger } from '@nestjs/common';
import * as path from 'path'; import * as path from 'path';
import { jsonToText } from '../../../collaboration/collaboration.util'; import {
jsonToHtml,
jsonToText,
} from '../../../collaboration/collaboration.util';
import { InjectKysely } from 'nestjs-kysely'; import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types'; import { KyselyDB } from '@docmost/db/types/kysely.types';
import { import {
@@ -18,9 +21,11 @@ import { generateSlugId } from '../../../common/helpers';
import { v7 } from 'uuid'; import { v7 } from 'uuid';
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered'; import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types'; import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext'; import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils'; import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
import { formatImportHtml } from '../utils/import-formatter'; import { formatImportHtml } from '../utils/import-formatter';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import { import {
buildAttachmentCandidates, buildAttachmentCandidates,
collectMarkdownAndHtmlFiles, collectMarkdownAndHtmlFiles,
@@ -461,7 +466,18 @@ export class FileImportTaskService {
content = await fs.readFile(absPath, 'utf-8'); content = await fs.readFile(absPath, 'utf-8');
if (page.fileExtension.toLowerCase() === '.md') { if (page.fileExtension.toLowerCase() === '.md') {
content = await markdownToHtml(content); // Parse markdown with the single canonical converter
// (`@docmost/prosemirror-markdown`), after normalizing foreign
// reference footnotes, then serialize to HTML so the shared HTML
// pipeline below (processAttachments + formatImportHtml +
// processHTML) keeps handling `.md` and `.html` imports
// uniformly. The markdown PARSE no longer goes through the
// editor-ext markdown layer (issue #345) — the drift source is
// gone. The PM -> HTML -> PM hop that follows is lossless
// plumbing for attachment/link resolution, NOT a second parse.
content = jsonToHtml(
await markdownToProseMirror(normalizeForeignMarkdown(content)),
);
} }
} catch (err: any) { } catch (err: any) {
if (err?.code === 'ENOENT') { if (err?.code === 'ENOENT') {
@@ -500,10 +516,12 @@ export class FileImportTaskService {
this.importService.extractTitleAndRemoveHeading(pmState); this.importService.extractTitleAndRemoveHeading(pmState);
// Canonicalize footnote topology on this non-editor write path // Canonicalize footnote topology on this non-editor write path
// (markdownToHtml/processHTML never runs footnoteSyncPlugin), so a // (the HTML pipeline's processHTML never runs footnoteSyncPlugin), so
// zip-imported page's footnotes are reference-ordered, deduped, and // a zip-imported page's footnotes are reference-ordered, deduped, and
// orphan-free like the editor's invariant (issue #228). Pure + // orphan-free like the editor's invariant (issue #228). Pure +
// idempotent + shape-safe; a footnote-free doc is unchanged. // idempotent + shape-safe; a footnote-free doc is unchanged. (For a
// `.md` file the package parser already yields canonical footnotes,
// so this is a no-op there.)
// (Future consolidation, architecture B: like import.service, this // (Future consolidation, architecture B: like import.service, this
// path persists directly rather than via PageService — a shared // path persists directly rather than via PageService — a shared
// "prepare JSON for persist" helper would centralize this call.) // "prepare JSON for persist" helper would centralize this call.)
@@ -12,13 +12,19 @@ import { canonicalizeFootnotes } from '@docmost/editor-ext';
/** /**
* Integration-ish test for the USER-FACING markdown import path * Integration-ish test for the USER-FACING markdown import path
* (`ImportService.importPage`). It exercises the REAL markdown -> HTML -> JSON * (`ImportService.importPage`). It exercises the REAL markdown -> ProseMirror
* conversion and asserts that the stored page content has its footnotes * conversion and asserts the stored page's footnotes are canonical: ordered by
* canonicalized the gap that issue #228 fixes: the import path builds * FIRST REFERENCE (not markdown definition order), reused references deduped to a
* ProseMirror JSON directly (never running the editor's footnoteSyncPlugin), so * single definition, and orphan definitions dropped.
* before this wiring the stored footnotes kept the markdown's physical *
* definition order (out of order vs. references), retained orphan definitions, * Since #345 the markdown parse runs through the canonical package
* and did not collapse reused references. * (`normalizeForeignMarkdown` -> `markdownToProseMirror`), which owns this
* canonicalization: the input's GFM `[^id]` reference footnotes are normalized to
* inline `^[…]`, and the parser assigns fresh sequential ids (`fn-*`) in
* reference order while merging identical bodies so we assert by definition
* BODY order, not by the source labels. `canonicalizeFootnotes` remains wired as
* an idempotent safety net (issue #228) and is a no-op on this already-canonical
* output.
* *
* The DB/ydoc side-effects are stubbed: `getNewPagePosition` (DB query) and * The DB/ydoc side-effects are stubbed: `getNewPagePosition` (DB query) and
* `createYdoc` (Yjs encode) are spied, and `pageRepo.insertPage` captures the * `createYdoc` (Yjs encode) are spied, and `pageRepo.insertPage` captures the
@@ -67,24 +73,14 @@ function makeService() {
} }
/** List the footnote-definition ids of the (single) footnotesList, in order. */ /** List the footnote-definition ids of the (single) footnotesList, in order. */
function footnoteListIds(content: any): string[] { /** Definition body texts of the (single) footnotesList, in list order. */
function footnoteListBodies(content: any): string[] {
const list = (content.content ?? []).find( const list = (content.content ?? []).find(
(n: any) => n.type === 'footnotesList', (n: any) => n.type === 'footnotesList',
); );
if (!list) return []; return (list?.content ?? [])
return (list.content ?? [])
.filter((n: any) => n.type === 'footnoteDefinition') .filter((n: any) => n.type === 'footnoteDefinition')
.map((n: any) => n.attrs?.id); .map((n: any) => n.content?.[0]?.content?.[0]?.text);
}
function definitionText(content: any, id: string): string | undefined {
const list = (content.content ?? []).find(
(n: any) => n.type === 'footnotesList',
);
const def = (list?.content ?? []).find(
(n: any) => n.type === 'footnoteDefinition' && n.attrs?.id === id,
);
return def?.content?.[0]?.content?.[0]?.text;
} }
describe('ImportService.importPage — footnote canonicalization (#228)', () => { describe('ImportService.importPage — footnote canonicalization (#228)', () => {
@@ -101,23 +97,23 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
const content = getCaptured().content; const content = getCaptured().content;
expect(content).toBeTruthy(); expect(content).toBeTruthy();
// Reference order is c, a, b (NOT the markdown definition order a, b, c). // Definitions ordered by FIRST REFERENCE (C, A, B) — NOT the markdown
expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']); // definition order (A, B, C) — with the orphan [^z] dropped and the reused
// [^a] collapsed to a single definition. (Ids are the parser's fresh `fn-*`,
// Definitions preserved and attached to the right ids. // so we pin the BODIES.)
expect(definitionText(content, 'c')).toBe('note C'); expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
expect(definitionText(content, 'a')).toBe('note A');
expect(definitionText(content, 'b')).toBe('note B');
// Orphan definition [^z] is dropped. // Orphan definition [^z] is dropped.
expect(footnoteListIds(content)).not.toContain('z'); expect(footnoteListBodies(content)).not.toContain('orphan note');
// Reused [^a] yields exactly ONE definition, and exactly one list. // Reused [^a] yields exactly ONE definition, and exactly one list.
const lists = (content.content ?? []).filter( const lists = (content.content ?? []).filter(
(n: any) => n.type === 'footnotesList', (n: any) => n.type === 'footnotesList',
); );
expect(lists).toHaveLength(1); expect(lists).toHaveLength(1);
expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1); expect(
footnoteListBodies(content).filter((b) => b === 'note A'),
).toHaveLength(1);
}); });
it('is idempotent: canonicalizing the stored output again is a no-op', async () => { it('is idempotent: canonicalizing the stored output again is a no-op', async () => {
@@ -134,6 +130,6 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
// time must not change it (safe to wire into every write path). // time must not change it (safe to wire into every write path).
const second = canonicalizeFootnotes(stored); const second = canonicalizeFootnotes(stored);
expect(second).toEqual(stored); expect(second).toEqual(stored);
expect(footnoteListIds(second)).toEqual(['c', 'a', 'b']); expect(footnoteListBodies(second)).toEqual(['note C', 'note A', 'note B']);
}); });
}); });
@@ -17,7 +17,9 @@ import {
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered'; import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { TiptapTransformer } from '@hocuspocus/transformer'; import { TiptapTransformer } from '@hocuspocus/transformer';
import * as Y from 'yjs'; import * as Y from 'yjs';
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext'; import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import { import {
FileTaskStatus, FileTaskStatus,
FileTaskType, FileTaskType,
@@ -85,11 +87,13 @@ export class ImportService {
const extracted = this.extractTitleAndRemoveHeading(prosemirrorState); const extracted = this.extractTitleAndRemoveHeading(prosemirrorState);
const title = extracted.title; const title = extracted.title;
// Imported markdown/HTML is built via markdownToHtml -> htmlToJson, which // The markdown path now canonicalizes footnotes itself (the package parser),
// never runs the editor's footnoteSyncPlugin, so the footnote topology keeps // but the HTML path (processHTML -> htmlToJson) does NOT run the editor's
// the source's PHYSICAL definition order (out of order vs. references), // footnoteSyncPlugin, so an imported HTML doc can keep its source's PHYSICAL
// retains orphan definitions, and is not deduped. Canonicalize before // definition order (out of order vs. references), retain orphan definitions,
// persisting so the stored page matches the editor's invariant (issue #228). // and not be deduped. Canonicalize before persisting so the stored page
// matches the editor's invariant (issue #228); it is an idempotent no-op on
// the already-canonical markdown output.
// Pure + idempotent + shape-safe: a doc with no footnotes is unchanged. // Pure + idempotent + shape-safe: a doc with no footnotes is unchanged.
// (Future consolidation, architecture B: this import path persists directly // (Future consolidation, architecture B: this import path persists directly
// via pageRepo.insertPage rather than through PageService.createPage, so the // via pageRepo.insertPage rather than through PageService.createPage, so the
@@ -133,12 +137,15 @@ export class ImportService {
} }
async processMarkdown(markdownInput: string): Promise<any> { async processMarkdown(markdownInput: string): Promise<any> {
try { // Canonical markdown -> ProseMirror JSON directly via
const html = await markdownToHtml(markdownInput); // `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate and no
return this.processHTML(html); // second editor-ext markdown layer. Foreign markdown surfaces the strict
} catch (err) { // canonical parser does not accept (GFM `[^id]` reference footnotes) are
throw err; // rewritten to the canonical inline form by `normalizeForeignMarkdown` first.
} // The HTML-cleanup pass (`normalizeImportHtml`) is intentionally skipped here:
// it targets foreign *HTML* (Notion/XWiki), which only ever arrives on the
// `.html` path (`processHTML`), never as canonical markdown.
return markdownToProseMirror(normalizeForeignMarkdown(markdownInput));
} }
async processHTML(htmlInput: string): Promise<any> { async processHTML(htmlInput: string): Promise<any> {
@@ -0,0 +1,218 @@
import {
convertProseMirrorToMarkdown,
markdownToProseMirror,
} from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from './foreign-markdown';
/**
* STEP 2 goldens for issue #345: the foreign-markdown normalizer that runs at the
* import boundary BEFORE the strict canonical parser (`markdownToProseMirror`).
*
* Two layers:
* 1. PURE stringstring cases pinning the normalizer's own behavior (GFM
* reference footnotes inline `^[…]`).
* 2. END-TO-END acceptance: for a foreign corpus, `normalizeForeignMarkdown`
* then `markdownToProseMirror` then `convertProseMirrorToMarkdown` must leave
* NO literal `[^id]` / `:::` garbage in the document and must re-export in the
* canonical forms.
*/
describe('normalizeForeignMarkdown — GFM reference footnotes', () => {
it('inlines a single-line reference footnote and drops its definition', () => {
const out = normalizeForeignMarkdown(
'A note[^1] here.\n\n[^1]: The definition.',
);
expect(out).toBe('A note^[The definition.] here.\n');
});
it('inlines every reference to a reused id (downstream dedups)', () => {
const out = normalizeForeignMarkdown(
'X[^a] and Y[^a].\n\n[^a]: shared.',
);
expect(out).toBe('X^[shared.] and Y^[shared.].\n');
});
it('joins indented continuation lines of a definition with a space', () => {
const out = normalizeForeignMarkdown(
'See[^n].\n\n[^n]: line one\n line two',
);
expect(out).toBe('See^[line one line two].\n');
});
it('never rewrites a reference inside a fenced code block', () => {
const out = normalizeForeignMarkdown(
'```\ncode[^1] here\n```\n\n[^1]: def.',
);
expect(out).toContain('code[^1] here');
// The (now orphaned) definition line is still removed.
expect(out).not.toContain('[^1]: def.');
});
it('never rewrites a reference inside an INLINE-code span (backticks)', () => {
// The `[^1]` inside backticks is literal code and must survive verbatim;
// the one outside is rewritten. (Bug #1: only fenced blocks were protected.)
const out = normalizeForeignMarkdown(
'Use `arr[^1]` in code but note[^1] in prose.\n\n[^1]: def.',
);
expect(out).toBe('Use `arr[^1]` in code but note^[def.] in prose.\n');
});
it('escapes brackets in a body so an unbalanced ] cannot truncate the footnote', () => {
// A foreign definition body with a stray `]` would, unescaped, close the
// canonical `^[...]` early and leak the tail as text (bug #2). The body's
// brackets are backslash-escaped so the footnote stays whole.
const out = normalizeForeignMarkdown(
'Ref[^1] here.\n\n[^1]: see item ] and [more] later',
);
expect(out).toBe('Ref^[see item \\] and \\[more\\] later] here.\n');
// The tokenizer must see exactly one unescaped closing bracket (our own).
expect(out.match(/(?<!\\)\]/g)).toHaveLength(1);
});
it('leaves a reference with no matching definition literal (no body to inline)', () => {
const out = normalizeForeignMarkdown('Dangling[^x] ref.');
expect(out).toBe('Dangling[^x] ref.');
});
it('returns the input unchanged when there are no reference footnotes', () => {
const md = '# Title\n\nJust text with `inline code` and a [link](/x).';
expect(normalizeForeignMarkdown(md)).toBe(md);
});
it('does NOT touch callout surfaces — the canonical parser handles them', () => {
const callouts = ':::info\nHi\n:::\n\n> [!warning]\n> Careful';
expect(normalizeForeignMarkdown(callouts)).toBe(callouts);
});
it('strips a leading YAML front-matter block (Obsidian/Hugo/git-sync files)', () => {
const out = normalizeForeignMarkdown(
'---\ntitle: My Page\ntags: [a, b]\n---\n\n# Heading\n\nBody.',
);
expect(out).toBe('# Heading\n\nBody.');
// The front-matter must not leak into the body as a setext heading.
expect(out).not.toContain('title: My Page');
expect(out).not.toContain('---');
});
it('does not strip a horizontal rule that is not leading front-matter', () => {
const md = 'Intro paragraph.\n\n---\n\nAfter the rule.';
expect(normalizeForeignMarkdown(md)).toBe(md);
});
it('is linear on a document with thousands of definitions (no quadratic blowup)', () => {
// F2(a): the pass-2 rewrite must be O(text), not O(text × defs). Build a
// pathological doc (many defs + many plain text lines) and assert it
// completes well under a second — a quadratic implementation took ~14s.
const N = 4000;
const refs = Array.from({ length: N }, (_, i) => `line ${i} plain text`).join('\n');
const defs = Array.from({ length: N }, (_, i) => `[^n${i}]: def ${i}`).join('\n');
const doc = `start[^n0] and[^n${N - 1}] end\n\n${refs}\n\n${defs}`;
const t0 = Date.now();
const out = normalizeForeignMarkdown(doc);
const elapsed = Date.now() - t0;
expect(elapsed).toBeLessThan(2000);
// Sanity: the two real references were still inlined.
expect(out).toContain('^[def 0]');
expect(out).toContain(`^[def ${N - 1}]`);
});
it('is bounded on a long unclosed backtick run (no inline-split ReDoS)', () => {
// F2(b): a huge unterminated backtick run must not cause quadratic
// backtracking in the inline-code split. Oversized lines skip the split
// entirely (left untouched), so this returns promptly.
const line = 'x' + '`'.repeat(200000);
const doc = `${line}\n\n[^1]: def`;
const t0 = Date.now();
normalizeForeignMarkdown(doc);
expect(Date.now() - t0).toBeLessThan(2000);
});
it('does not crash or slow down on thousands of prefix-chain definition ids', () => {
// F7: the rewrite must use a FIXED generic scanner, not an alternation built
// from the ids. A `(a|aa|aaa|…)` alternation over prefix-chain ids blows the
// V8 regex compiler (FATAL RegExpCompiler Allocation failed — uncatchable,
// kills the process). A fixed scanner has no id-dependent compilation cost.
const N = 4000;
const ids = Array.from({ length: N }, (_, i) => 'a'.repeat(i + 1));
const defs = ids.map((id) => `[^${id}]: body ${id.length}`).join('\n');
const doc = `ref[^${ids[0]}] and[^${ids[N - 1]}] end\n\n${defs}`;
const t0 = Date.now();
const out = normalizeForeignMarkdown(doc);
expect(Date.now() - t0).toBeLessThan(2000);
// Prefix disambiguation is correct: [^a] and [^aaaa...] inline their OWN body.
expect(out).toContain('^[body 1]');
expect(out).toContain(`^[body ${N}]`);
});
it('strips a CRLF (Windows) front-matter block, not just LF', () => {
// F9: the line-anchored regex needs LF after the opening `---`, so a Windows
// file (`---\r\n…`) would slip past the strip and leak the front-matter into
// the body. normalizeForeignMarkdown normalizes CRLF -> LF first.
const out = normalizeForeignMarkdown(
'---\r\ntitle: Foo\r\ntags: [a]\r\n---\r\n\r\n# Heading\r\n\r\nBody.',
);
expect(out).toBe('# Heading\n\nBody.');
expect(out).not.toContain('title: Foo');
expect(out).not.toContain('---');
});
it('strips front-matter whose value contains a triple-dash (line-anchored)', () => {
// F8: the block must close only on a `\n---` LINE, not the first inline
// `---`. A value like `title: Q1 --- Q2` must not truncate the front-matter
// and leak the rest (author/closing ---) into the body.
const out = normalizeForeignMarkdown(
'---\ntitle: Q1 --- Q2 results\nauthor: bob\n---\n\nReal body.',
);
expect(out).toBe('Real body.');
expect(out).not.toContain('author: bob');
expect(out).not.toContain('Q2 results');
});
});
describe('foreign markdown import acceptance (normalizer + canonical parser)', () => {
const FOREIGN = [
'# Doc',
'',
'Body refs [^c] and [^a] and [^b] and again [^a].',
'',
':::info',
'A legacy callout.',
':::',
'',
'| h1 | h2 |',
'| --- | --- |',
'| 1 | 2 |',
'',
'[^a]: note A',
'[^b]: note B',
'[^c]: note C',
'[^z]: orphan note',
].join('\n');
it('leaves no literal [^id] or ::: in the imported doc and re-exports canonically', async () => {
const normalized = normalizeForeignMarkdown(FOREIGN);
const doc = await markdownToProseMirror(normalized);
const reexport = convertProseMirrorToMarkdown(doc);
// No foreign garbage leaks into the document.
expect(reexport).not.toMatch(/\[\^/); // no reference footnote refs/defs
expect(reexport).not.toContain(':::'); // no legacy callout fences
// Canonical forms are present.
expect(reexport).toContain('^[note C]');
expect(reexport).toContain('> [!info]');
expect(reexport).toContain('| h1 | h2 |');
// Footnotes: ordered by first reference (C, A, B), reused [^a] deduped to one,
// orphan [^z] dropped (it had no reference after normalization).
const list = doc.content.find((n: any) => n.type === 'footnotesList');
const bodies = list.content.map(
(d: any) => d.content[0].content[0].text,
);
expect(bodies).toEqual(['note C', 'note A', 'note B']);
expect(bodies).not.toContain('orphan note');
expect(
doc.content.filter((n: any) => n.type === 'footnotesList'),
).toHaveLength(1);
});
});
@@ -0,0 +1,265 @@
/**
* Foreign-markdown normalizer an input-liberal / output-canonical adapter that
* runs at the IMPORT boundary, BEFORE the canonical parser
* (`markdownToProseMirror` from `@docmost/prosemirror-markdown`).
*
* The canonical parser is deliberately STRICT: it only understands Docmost's
* canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
* inline footnotes `^[body]`, lossless `![alt](src) <!--img {...}-->` images, ).
* Import, however, ingests FOREIGN files (GitHub/GFM, Notion, old Docmost
* exports). Those use surfaces the canonical parser does not accept, most notably
* GitHub-flavoured *reference* footnotes:
*
* Text with a note[^1] and another[^long].
*
* [^1]: The first definition.
* [^long]: A second one.
*
* Left untouched, the parser does NOT recognise `[^id]` (it only parses `^[body]`),
* so the reference leaks as literal text and worse, the trailing `[^id]: def`
* line is a valid CommonMark *link-reference definition*, so `[^id]` is silently
* rendered as a bogus link. This normalizer rewrites reference footnotes into the
* canonical inline form so the parser materialises real footnote nodes.
*
* This is a TEXT pre-pass, NOT a second parser fork: it does not re-implement any
* converter logic. Callout surfaces (`:::type` and `> [!type]`) are intentionally
* NOT touched here the canonical parser already accepts BOTH natively (its
* `preprocessCallouts` pass), so normalizing them would be redundant and would
* only risk degrading the parser's nesting/code-fence-aware handling.
*/
/** Matches a fenced code block delimiter (``` or ~~~), capturing the marker run. */
const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
/**
* Matches a GFM footnote DEFINITION line: `[^id]: body`. The id is any run of
* non-`]` characters; the body is the remainder of the line (possibly empty).
*/
const FOOTNOTE_DEF_RE = /^\[\^([^\]]+)\]:[ \t]?(.*)$/;
/** True when a line is a code-fence delimiter that toggles fenced-code state. */
function fenceMarker(line: string): string | null {
const m = line.match(CODE_FENCE_RE);
return m ? m[2] : null;
}
/** True when a line is indented (leading space/tab) and not blank — a continuation. */
function isIndentedContinuation(line: string): boolean {
return /^[ \t]+\S/.test(line);
}
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
/**
* Backslash-escape any square bracket in a footnote body before it is wrapped in
* `^[...]`. The canonical inline-footnote tokenizer scans the body with bracket
* balancing and closes on the first UNMATCHED `]`, so an unbalanced bracket in a
* foreign definition (e.g. `[^1]: see item ] later`) would otherwise truncate the
* footnote and leak the tail as literal text. Escaping every `[`/`]` makes the
* body an inert run of characters the tokenizer then closes only on our own
* closing `]`. (A balanced `[link](url)` inside a body still round-trips because
* the escaped form renders the literal brackets, which is the safe reading for a
* footnote body; the alternative brittle balance tracking risks worse.)
*/
function escapeFootnoteBody(body: string): string {
return body.replace(/[[\]]/g, '\\$&');
}
/**
* Rewrite every `[^id]` reference on a line to its `^[body]` form, but ONLY in the
* text OUTSIDE inline-code spans. A `[^id]` inside backticks is literal code
* content and must be preserved verbatim (a footnote ref never lives inside code).
* We split the line on inline-code spans (paired backtick runs) and rewrite only
* the non-code segments.
*/
// Above this length a single line is not split into inline-code spans (see
// below). A genuine markdown line carrying a footnote reference is never tens of
// KB; the cap only bypasses the inline-code protection for pathological lines.
const INLINE_SPLIT_MAX_LINE = 8192;
function rewriteRefsOutsideInlineCode(
line: string,
replace: (text: string) => string,
): string {
// The inline-code split alternation `(`+)(?:(?!\1)[\s\S])*\1` backtracks
// quadratically on a long UNCLOSED backtick run (its middle can consume the
// rest of the line, then fail to find a closing run and retry from each
// position). On an untrusted import this is a request-thread ReDoS. A real
// footnote line is short, so for an oversized line we skip the inline-code
// protection entirely and leave the line UNTOUCHED (rewriting it wholesale
// could corrupt a `[^id]` that legitimately lives inside inline code). This is
// a conservative bypass: an over-8KB line simply does not get its reference
// footnotes inlined — acceptable for a pathological input.
if (line.length > INLINE_SPLIT_MAX_LINE) return line;
// Alternation: an inline-code span (one or more backticks, then anything up to
// the SAME run of backticks) OR a run of non-backtick text. Unterminated
// backticks fall through as ordinary text (matched by the second branch on the
// leftover), so a stray backtick never swallows the rest of the line.
const parts = line.match(/(`+)(?:(?!\1)[\s\S])*\1|[^`]+|`+/g);
if (!parts) return line;
return parts
.map((seg) => (seg.startsWith('`') ? seg : replace(seg)))
.join('');
}
/**
* Convert GFM reference footnotes (`[^id]` + `[^id]: def`) into canonical inline
* footnotes (`^[def]`).
*
* - Definitions are collected first (a leading `[^id]: text` line plus any
* immediately-following indented continuation lines, joined with a space) and
* removed from the output.
* - Each in-text reference `[^id]` for which a definition was found is replaced by
* `^[def]`. References with no matching definition are left literal (there is no
* body to inline; the parser fails them open the same way).
* - Code is respected on both passes: `[^id]` inside a fenced ``` / ~~~ block is
* never rewritten and a `[^id]:` line inside a fence is never a definition; and
* on the rewrite pass a `[^id]` inside an INLINE-code span (backticks) is left
* literal too.
* - The inlined body is bracket-escaped so an unbalanced `[`/`]` in a foreign
* definition cannot truncate the resulting `^[...]` footnote.
*
* Deduplication / reference-ordering / orphan-dropping of the resulting footnotes
* is handled downstream by the canonical parser (`assembleFootnotes`); this pass
* only changes the surface syntax.
*/
function convertReferenceFootnotes(markdown: string): string {
const lines = markdown.split('\n');
// Pass 1: collect definitions and mark their lines for removal.
const defs = new Map<string, string>();
const dropped = new Array<boolean>(lines.length).fill(false);
let inFence = false;
let fence = '';
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
const marker = fenceMarker(line);
if (inFence) {
if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
inFence = false;
fence = '';
}
continue;
}
if (marker) {
inFence = true;
fence = marker;
continue;
}
const def = line.match(FOOTNOTE_DEF_RE);
if (!def) continue;
const id = def[1];
const body: string[] = [def[2].trim()];
dropped[i] = true;
// Consume immediately-following indented continuation lines (GFM lazy
// continuation is not supported by design — keep it simple and predictable).
let j = i + 1;
while (j < lines.length && isIndentedContinuation(lines[j])) {
body.push(lines[j].trim());
dropped[j] = true;
j++;
}
i = j - 1;
// Last definition wins for a duplicated id (matches CommonMark link-ref
// semantics closely enough for a foreign-input adapter).
defs.set(id, body.filter((s) => s.length > 0).join(' '));
}
if (defs.size === 0) {
return markdown;
}
// ONE fixed, generic scanner regex — NOT one built from the definition ids.
// It matches ANY `[^id]` shape, and the replacer decides per match via a map
// lookup whether that id is a real definition (replace) or not (leave as-is).
// This is genuinely O(total text) with no per-document regex compilation.
//
// Do NOT rebuild this as an alternation over `[...defs.keys()]`: a giant
// `(id1|id2|...)` alternation over thousands of ids can blow the V8 regex
// compiler's stack — a fatal, UNCATCHABLE "RegExpCompiler Allocation failed"
// on prefix-chain ids (`a`, `aa`, `aaa`, ...) that kills the whole process
// (worse than the earlier per-def thread-hang). A fixed scanner has no
// id-dependent compilation cost and cannot blow up.
const refRe = /\[\^([^\]]+)\]/g;
const rewriteSegment = (segment: string): string =>
segment.replace(refRe, (whole, id: string) => {
const body = defs.get(id);
// Only real definitions are inlined; an unknown id is left literal (same as
// the old per-def loop, which simply never matched it).
return body === undefined ? whole : `^[${escapeFootnoteBody(body)}]`;
});
// Pass 2: rewrite in-text references, skipping fenced code and dropped lines.
const out: string[] = [];
inFence = false;
fence = '';
for (let i = 0; i < lines.length; i++) {
if (dropped[i]) continue;
let line = lines[i];
const marker = fenceMarker(line);
if (inFence) {
out.push(line);
if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
inFence = false;
fence = '';
}
continue;
}
if (marker) {
inFence = true;
fence = marker;
out.push(line);
continue;
}
line = rewriteRefsOutsideInlineCode(line, rewriteSegment);
out.push(line);
}
return out.join('\n');
}
/**
* Strip a single leading YAML front-matter block (`---\n…\n---`). Foreign files
* from Obsidian / Hugo / Jekyll / Notion and Docmost's OWN git-sync page files
* open with front-matter that the canonical parser does not consume, so
* without this it leaks into the body (and `title: Foo` above the closing `---`
* renders as a setext `<h2>` that `extractTitleAndRemoveHeading` can hijack as
* the page title). It is a no-op for front-matter-free input.
*
* LINE-ANCHORED (the same shape the canonical parser uses in
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
* very start and closes only on a `\n---` line. The retired `markdownToHtml`
* strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
* and leaked the rest into the body. An optional leading BOM is tolerated.
*/
const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
/**
* Normalize a foreign markdown string into Docmost's canonical markdown surface
* so the strict canonical parser accepts it losslessly: normalize line endings,
* strip a leading YAML front-matter block, then rewrite GFM reference footnotes
* into inline footnotes. Add further fixture-driven foreign-surface cases here as
* they are found.
*/
export function normalizeForeignMarkdown(markdown: string): string {
if (!markdown) return markdown;
// Normalize CRLF -> LF FIRST. The line-anchored front-matter regex requires a
// bare `\n` after the opening `---`, and convertReferenceFootnotes splits on
// `\n`; a Windows/CRLF foreign file (`---\r\n…`) would otherwise slip past the
// front-matter strip and leak into the body. The canonical parser
// (page-file.ts parsePageFile) normalizes the same way before its FRONTMATTER_RE.
const src = markdown.replace(/\r\n/g, '\n');
const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
return convertReferenceFootnotes(withoutFrontMatter);
}
@@ -46,7 +46,6 @@ export class MetricsBullService implements OnModuleInit, OnModuleDestroy {
@InjectQueue(QueueName.GENERAL_QUEUE) generalQueue: Queue, @InjectQueue(QueueName.GENERAL_QUEUE) generalQueue: Queue,
@InjectQueue(QueueName.BILLING_QUEUE) billingQueue: Queue, @InjectQueue(QueueName.BILLING_QUEUE) billingQueue: Queue,
@InjectQueue(QueueName.FILE_TASK_QUEUE) fileTaskQueue: Queue, @InjectQueue(QueueName.FILE_TASK_QUEUE) fileTaskQueue: Queue,
@InjectQueue(QueueName.SEARCH_QUEUE) searchQueue: Queue,
@InjectQueue(QueueName.AI_QUEUE) aiQueue: Queue, @InjectQueue(QueueName.AI_QUEUE) aiQueue: Queue,
@InjectQueue(QueueName.HISTORY_QUEUE) historyQueue: Queue, @InjectQueue(QueueName.HISTORY_QUEUE) historyQueue: Queue,
@InjectQueue(QueueName.NOTIFICATION_QUEUE) notificationQueue: Queue, @InjectQueue(QueueName.NOTIFICATION_QUEUE) notificationQueue: Queue,
@@ -58,7 +57,6 @@ export class MetricsBullService implements OnModuleInit, OnModuleDestroy {
{ label: 'general', queue: generalQueue }, { label: 'general', queue: generalQueue },
{ label: 'billing', queue: billingQueue }, { label: 'billing', queue: billingQueue },
{ label: 'file-task', queue: fileTaskQueue }, { label: 'file-task', queue: fileTaskQueue },
{ label: 'search', queue: searchQueue },
{ label: 'ai', queue: aiQueue }, { label: 'ai', queue: aiQueue },
{ label: 'history', queue: historyQueue }, { label: 'history', queue: historyQueue },
{ label: 'notification', queue: notificationQueue }, { label: 'notification', queue: notificationQueue },
@@ -4,7 +4,6 @@ export enum QueueName {
GENERAL_QUEUE = '{general-queue}', GENERAL_QUEUE = '{general-queue}',
BILLING_QUEUE = '{billing-queue}', BILLING_QUEUE = '{billing-queue}',
FILE_TASK_QUEUE = '{file-task-queue}', FILE_TASK_QUEUE = '{file-task-queue}',
SEARCH_QUEUE = '{search-queue}',
AI_QUEUE = '{ai-queue}', AI_QUEUE = '{ai-queue}',
HISTORY_QUEUE = '{history-queue}', HISTORY_QUEUE = '{history-queue}',
NOTIFICATION_QUEUE = '{notification-queue}', NOTIFICATION_QUEUE = '{notification-queue}',
@@ -32,12 +31,6 @@ export enum QueueJob {
IMPORT_TASK = 'import-task', IMPORT_TASK = 'import-task',
EXPORT_TASK = 'export-task', EXPORT_TASK = 'export-task',
SEARCH_INDEX_PAGE = 'search-index-page',
SEARCH_INDEX_PAGES = 'search-index-pages',
SEARCH_INDEX_COMMENT = 'search-index-comment',
SEARCH_INDEX_COMMENTS = 'search-index-comments',
SEARCH_INDEX_ATTACHMENT = 'search-index-attachment',
SEARCH_INDEX_ATTACHMENTS = 'search-index-attachments',
SEARCH_REMOVE_PAGE = 'search-remove-page', SEARCH_REMOVE_PAGE = 'search-remove-page',
SEARCH_REMOVE_ASSET = 'search-remove-attachment', SEARCH_REMOVE_ASSET = 'search-remove-attachment',
SEARCH_REMOVE_FACE = 'search-remove-comment', SEARCH_REMOVE_FACE = 'search-remove-comment',
@@ -57,14 +57,6 @@ import { GeneralQueueProcessor } from './processors/general-queue.processor';
attempts: 1, attempts: 1,
}, },
}), }),
BullModule.registerQueue({
name: QueueName.SEARCH_QUEUE,
defaultJobOptions: {
removeOnComplete: true,
removeOnFail: true,
attempts: 2,
},
}),
BullModule.registerQueue({ BullModule.registerQueue({
name: QueueName.AI_QUEUE, name: QueueName.AI_QUEUE,
defaultJobOptions: { defaultJobOptions: {
@@ -0,0 +1,304 @@
import { Kysely } from 'kysely';
import {
AiChatRunRepo,
SWEEP_RUN_STALE_MS,
} from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunService } from '../../src/core/ai-chat/ai-chat-run.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
} from './db';
/**
* Integration coverage for the #184 phase-1 durable agent run: real SQL against
* docmost_test. Proves the core invariant primitives a run is a first-class
* lifecycle row, at most one is active per chat, a detached run's progress
* survives with NO subscriber, an explicit stop settles it as aborted, a
* reconnect read returns the persisted state, and a crash sweep recovers
* dangling runs.
*/
describe('AiChatRun durable lifecycle [integration]', () => {
let db: Kysely<any>;
let runRepo: AiChatRunRepo;
let messageRepo: AiChatMessageRepo;
let service: AiChatRunService;
let workspaceId: string;
let otherWorkspaceId: string;
let userId: string;
let chatId: string;
beforeAll(async () => {
db = getTestDb();
runRepo = new AiChatRunRepo(db as any);
messageRepo = new AiChatMessageRepo(db as any);
// Boot-sweep isn't triggered here; the isCloud stub is all the service needs
// for these direct-call integration cases (F7).
service = new AiChatRunService(runRepo, { isCloud: () => false } as never);
workspaceId = (await createWorkspace(db)).id;
otherWorkspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
chatId = (await createChat(db, { workspaceId, creatorId: userId })).id;
});
afterAll(async () => {
await destroyTestDb();
});
// Each test that creates an active run settles it (or uses its own chat) so the
// partial unique index does not bleed across tests.
it('insert + findById round-trips a run row, defaulting status/trigger', async () => {
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
});
expect(run.status).toBe('pending');
expect(run.trigger).toBe('user');
expect(run.stepCount).toBe(0);
const found = await runRepo.findById(run.id, workspaceId);
expect(found!.id).toBe(run.id);
// Workspace-scoped: a foreign workspace sees nothing.
expect(await runRepo.findById(run.id, otherWorkspaceId)).toBeUndefined();
// settle so it does not occupy the active slot
await runRepo.update(run.id, workspaceId, {
status: 'succeeded',
finishedAt: new Date(),
});
});
it('enforces ONE ACTIVE run per chat (partial unique index rejects a second)', async () => {
const activeChat = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const first = await runRepo.insert({
chatId: activeChat,
workspaceId,
createdBy: userId,
status: 'running',
});
// A second pending/running run on the SAME chat must be rejected by the DB.
await expect(
runRepo.insert({
chatId: activeChat,
workspaceId,
createdBy: userId,
status: 'running',
}),
).rejects.toThrow();
// findActiveByChat returns exactly the one active run.
const active = await runRepo.findActiveByChat(activeChat, workspaceId);
expect(active!.id).toBe(first.id);
// Once it settles, the slot frees and a new run may start.
await runRepo.update(first.id, workspaceId, {
status: 'succeeded',
finishedAt: new Date(),
});
expect(
await runRepo.findActiveByChat(activeChat, workspaceId),
).toBeUndefined();
const second = await runRepo.insert({
chatId: activeChat,
workspaceId,
createdBy: userId,
status: 'running',
});
expect(second.id).not.toBe(first.id);
await runRepo.update(second.id, workspaceId, {
status: 'aborted',
finishedAt: new Date(),
});
});
it('DETACHED run: persists + finalizes succeeded with NO subscriber, reconnect returns state', async () => {
// A dedicated chat so the active-run slot is clean.
const runChat = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
// beginRun = the runner starts the turn (registers an in-memory controller).
const handle = await service.beginRun({
chatId: runChat,
workspaceId,
userId,
});
expect(handle.signal.aborted).toBe(false);
expect(service.isLocallyActive(handle.runId)).toBe(true);
// The assistant projection row (#183) is seeded + linked.
const seeded = await messageRepo.insert({
chatId: runChat,
workspaceId,
userId,
role: 'assistant',
content: '',
status: 'streaming',
metadata: { parts: [] } as never,
});
await service.linkAssistantMessage(handle.runId, workspaceId, seeded.id);
// Progress is persisted as steps finish — NO HTTP socket involved here at all.
await service.recordStep(handle.runId, workspaceId, 1);
await messageRepo.update(seeded.id, workspaceId, {
content: 'partial work',
metadata: { parts: [{ type: 'text', text: 'partial work' }] },
});
// The turn completes; finalize the projection then the run.
await messageRepo.update(seeded.id, workspaceId, {
content: 'final answer',
status: 'completed',
});
await service.finalizeRun(handle.runId, workspaceId, 'completed');
expect(service.isLocallyActive(handle.runId)).toBe(false);
// Reconnect: the latest run for the chat + its projected message, from the DB.
const run = await service.getLatestForChat(runChat, workspaceId);
expect(run!.status).toBe('succeeded');
expect(run!.stepCount).toBe(1);
expect(run!.assistantMessageId).toBe(seeded.id);
expect(run!.finishedAt).toBeTruthy();
const message = await messageRepo.findById(seeded.id, workspaceId);
expect(message!.status).toBe('completed');
expect(message!.content).toBe('final answer');
});
it('EXPLICIT stop aborts the run signal, marks the row, and settles as aborted', async () => {
const runChat = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const handle = await service.beginRun({
chatId: runChat,
workspaceId,
userId,
});
// User presses Stop.
const stopped = await service.requestStop(handle.runId, workspaceId);
expect(stopped).toBe(true);
expect(handle.signal.aborted).toBe(true);
// The row carries the stop request (distinct from a disconnect, which would
// leave stop_requested_at NULL).
const afterStop = await runRepo.findById(handle.runId, workspaceId);
expect(afterStop!.stopRequestedAt).toBeTruthy();
// The terminal callback (onAbort) settles the run.
await service.finalizeRun(handle.runId, workspaceId, 'aborted');
const run = await service.getLatestForChat(runChat, workspaceId);
expect(run!.status).toBe('aborted');
});
it('markStopRequested is a no-op on an already-settled run (returns undefined)', async () => {
const runChat = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const run = await runRepo.insert({
chatId: runChat,
workspaceId,
createdBy: userId,
status: 'running',
});
await runRepo.update(run.id, workspaceId, {
status: 'succeeded',
finishedAt: new Date(),
});
const marked = await runRepo.markStopRequested(run.id, workspaceId);
expect(marked).toBeUndefined();
});
it('sweepRunning aborts STALE dangling runs but not fresh or settled ones', async () => {
const sweepChat1 = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const sweepChat2 = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const sweepChat3 = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const stale = await runRepo.insert({
chatId: sweepChat1,
workspaceId,
createdBy: userId,
status: 'running',
});
const fresh = await runRepo.insert({
chatId: sweepChat2,
workspaceId,
createdBy: userId,
status: 'running',
});
const settled = await runRepo.insert({
chatId: sweepChat3,
workspaceId,
createdBy: userId,
status: 'running',
});
await runRepo.update(settled.id, workspaceId, {
status: 'succeeded',
finishedAt: new Date(),
});
// Backdate the stale run's updatedAt past the 10-minute staleness window.
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 20 * 60 * 1000) })
.where('id', '=', stale.id)
.execute();
// WINDOWED sweep (phase-2 multi-instance timer path): only runs older than the
// staleness window are aborted, so a sibling replica's fresh run survives. The
// no-arg boot sweep (variant C) is unconditional — covered separately below.
const swept = await runRepo.sweepRunning({ staleMs: SWEEP_RUN_STALE_MS });
expect(swept).toBeGreaterThanOrEqual(1);
expect((await runRepo.findById(stale.id, workspaceId))!.status).toBe(
'aborted',
);
// Fresh (recently-updated) running run survives the WINDOWED sweep — a sibling
// replica may still be executing it.
expect((await runRepo.findById(fresh.id, workspaceId))!.status).toBe(
'running',
);
expect((await runRepo.findById(settled.id, workspaceId))!.status).toBe(
'succeeded',
);
// cleanup active fresh run
await runRepo.update(fresh.id, workspaceId, {
status: 'aborted',
finishedAt: new Date(),
});
});
it('sweepRunning() with NO args (boot sweep / variant C) aborts even a FRESH running run', async () => {
// F1/DECISION C at the SQL level: the unconditional boot sweep has NO
// staleness window, so a run updated just now (a fast restart) is settled too
// — otherwise it would stay 'running' forever and 409 every future turn.
const bootChat = (
await createChat(db, { workspaceId, creatorId: userId })
).id;
const fresh = await runRepo.insert({
chatId: bootChat,
workspaceId,
createdBy: userId,
status: 'running',
});
// updatedAt = now (fresh, untouched). The no-arg sweep settles it anyway.
const swept = await runRepo.sweepRunning();
expect(swept).toBeGreaterThanOrEqual(1);
expect((await runRepo.findById(fresh.id, workspaceId))!.status).toBe(
'aborted',
);
});
});
@@ -1,113 +0,0 @@
import { Kysely } from 'kysely';
import { PagePermissionRepo } from '@docmost/db/repos/page/page-permission.repo';
import { GroupRepo } from '@docmost/db/repos/group/group.repo';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
createUser,
createPage,
} from './db';
/**
* #348 the whole-workspace access-filter short-circuit is an ACCESS-CONTROL
* path, so it must produce the SAME result as the full recursive-ancestor CTE.
*
* filterAccessiblePageIds({ workspaceId }) (no spaceId the favorites /
* notifications / recent / created-by / global-search callers) skips the CTE only
* when the workspace has ZERO restricted pages. A page is "restricted &
* inaccessible" when it (or an ancestor) has a `pageAccess` row and the user has
* no matching `pagePermissions`. Driven against real Postgres, asserts:
* 1. zero restrictions -> short-circuit returns the full input set;
* 2. a restriction present -> the CTE runs and drops the page the user can't
* reach while keeping the reachable ones (behavior unchanged);
* 3. inserting the FIRST pageAccess flips hasRestrictedPagesInWorkspace
* false -> true immediately (the 0->1 transition now uncached, no stale
* window, review F1); it is scoped per workspace.
*/
describe('#348 filterAccessiblePageIds workspace short-circuit (real PG)', () => {
let db: Kysely<any>;
let repo: PagePermissionRepo;
let workspaceId: string;
let otherWorkspaceId: string;
let userId: string;
let spaceId: string;
beforeAll(async () => {
db = getTestDb();
// hasRestrictedPagesInWorkspace is now uncached, and no other cached
// permission path is exercised here, so a no-op cache stub suffices.
const cacheStub = {
get: async () => undefined,
set: async () => undefined,
del: async () => undefined,
} as never;
repo = new PagePermissionRepo(db, new GroupRepo(db), cacheStub);
const ws = await createWorkspace(db);
workspaceId = ws.id;
const other = await createWorkspace(db);
otherWorkspaceId = other.id;
const user = await createUser(db, workspaceId);
userId = user.id;
const space = await createSpace(db, workspaceId);
spaceId = space.id;
});
afterAll(async () => {
await destroyTestDb();
});
it('zero restrictions: short-circuit returns the full input set', async () => {
const p1 = await createPage(db, { workspaceId, spaceId });
const p2 = await createPage(db, { workspaceId, spaceId });
expect(await repo.hasRestrictedPagesInWorkspace(workspaceId)).toBe(false);
const ids = [p1.id, p2.id];
const filtered = await repo.filterAccessiblePageIds({
pageIds: ids,
userId,
workspaceId,
});
expect(new Set(filtered)).toEqual(new Set(ids));
});
it('a restriction present: filters out the page the user cannot reach', async () => {
const openPage = await createPage(db, { workspaceId, spaceId });
const restrictedPage = await createPage(db, { workspaceId, spaceId });
// Add a pageAccess row on restrictedPage with NO matching pagePermissions for
// `userId` → the CTE anti-join marks it inaccessible for this user.
await db
.insertInto('pageAccess')
.values({
pageId: restrictedPage.id,
workspaceId,
spaceId,
accessLevel: 'read',
creatorId: userId,
})
.execute();
// 0->1 transition is reflected immediately (uncached).
expect(await repo.hasRestrictedPagesInWorkspace(workspaceId)).toBe(true);
const filtered = await repo.filterAccessiblePageIds({
pageIds: [openPage.id, restrictedPage.id],
userId,
workspaceId,
});
expect(filtered).toContain(openPage.id);
expect(filtered).not.toContain(restrictedPage.id);
});
it('hasRestrictedPagesInWorkspace is scoped per workspace', async () => {
// The other workspace has no pageAccess rows → still false, unaffected by the
// restriction added above in `workspaceId`.
expect(await repo.hasRestrictedPagesInWorkspace(otherWorkspaceId)).toBe(
false,
);
});
});
@@ -1,25 +1,7 @@
import { Kysely } from 'kysely'; import { Kysely } from 'kysely';
import { WorkspaceRepo } from '@docmost/db/repos/workspace/workspace.repo'; import { WorkspaceRepo } from '@docmost/db/repos/workspace/workspace.repo';
import { CacheKey } from 'src/common/helpers/cache-keys';
import { getTestDb, destroyTestDb, createWorkspace } from './db'; import { getTestDb, destroyTestDb, createWorkspace } from './db';
// A minimal Map-backed cache double with a working `del` (the previous `{}` stub
// made bustWorkspaceCache's `del` throw into its own try/catch, so the #348
// invalidation was never actually exercised — review F6).
function makeCacheDouble() {
const store = new Map<string, unknown>();
return {
store,
get: async (k: string) => store.get(k),
set: async (k: string, v: unknown) => {
store.set(k, v);
},
del: async (k: string) => {
store.delete(k);
},
};
}
/** /**
* A WorkspaceRepo.updateSetting jsonb-MERGE (the html-embed kill-switch * A WorkspaceRepo.updateSetting jsonb-MERGE (the html-embed kill-switch
* write-half). Setting a single top-level key must NOT clobber sibling * write-half). Setting a single top-level key must NOT clobber sibling
@@ -33,9 +15,7 @@ describe('WorkspaceRepo.updateSetting (jsonb merge) [integration]', () => {
beforeAll(() => { beforeAll(() => {
db = getTestDb(); db = getTestDb();
// Repos are plain classes taking @InjectKysely() db — instantiate directly. // Repos are plain classes taking @InjectKysely() db — instantiate directly.
// 2nd arg is CACHE_MANAGER (used only to bust the #348 workspace cache); a repo = new WorkspaceRepo(db as any);
// stub is fine here since bustWorkspaceCache is best-effort (try/catch).
repo = new WorkspaceRepo(db as any, {} as any);
}); });
afterAll(async () => { afterAll(async () => {
@@ -78,62 +58,3 @@ describe('WorkspaceRepo.updateSetting (jsonb merge) [integration]', () => {
expect(updated.settings).toEqual({ htmlEmbed: false }); expect(updated.settings).toEqual({ htmlEmbed: false });
}); });
}); });
/**
* #348 F6 the DomainMiddleware workspace cache (WORKSPACE_SELF_HOSTED /
* WORKSPACE_BY_HOST, 15s TTL) caches security-relevant fields (enforceSso/
* enforceMfa/status). Its correctness rests entirely on bustWorkspaceCache being
* called from every mutator. This exercises the real invalidation with a working
* cache double (not the {} stub, whose del throws-and-swallows): warm the cache
* like DomainMiddleware, mutate, and assert the busted key is gone so a stale
* workspace row can't outlive the mutation.
*/
describe('WorkspaceRepo bustWorkspaceCache invalidation [integration]', () => {
let db: Kysely<any>;
beforeAll(() => {
db = getTestDb();
});
afterAll(async () => {
await destroyTestDb();
});
it('updateSetting busts the self-hosted workspace cache key', async () => {
const cache = makeCacheDouble();
const repo = new WorkspaceRepo(db as any, cache as any);
const ws = await createWorkspace(db, { settings: {} });
// Warm the cache as DomainMiddleware would (self-hosted key).
cache.store.set(CacheKey.WORKSPACE_SELF_HOSTED, ws);
expect(cache.store.has(CacheKey.WORKSPACE_SELF_HOSTED)).toBe(true);
await repo.updateSetting(ws.id, 'htmlEmbed', true);
// The mutation must have invalidated the cached row.
expect(cache.store.has(CacheKey.WORKSPACE_SELF_HOSTED)).toBe(false);
});
it('updateSharingSettings busts the by-host workspace cache key too', async () => {
const cache = makeCacheDouble();
const repo = new WorkspaceRepo(db as any, cache as any);
const ws = await createWorkspace(db, { settings: {} });
// createWorkspace assigns a unique hostname; read it back for the by-host key.
const { hostname } = await db
.selectFrom('workspaces')
.select(['hostname'])
.where('id', '=', ws.id)
.executeTakeFirstOrThrow();
// Warm BOTH keys (self-hosted + by-host); the by-host bust needs the row's
// hostname, which the mutator returns from the DB.
cache.store.set(CacheKey.WORKSPACE_SELF_HOSTED, ws);
cache.store.set(CacheKey.WORKSPACE_BY_HOST(hostname as string), ws);
await repo.updateSharingSettings(ws.id, 'allowInvite', true);
expect(cache.store.has(CacheKey.WORKSPACE_SELF_HOSTED)).toBe(false);
expect(cache.store.has(CacheKey.WORKSPACE_BY_HOST(hostname as string))).toBe(
false,
);
});
});
+23
View File
@@ -0,0 +1,23 @@
// Jest stub for @tiptap/react.
//
// The server export/import code paths transitively import editor-ext, whose node
// extensions import from `@tiptap/react`. The real module re-exports all of
// `@tiptap/core` (headless, safe under node) AND adds React view helpers
// (`ReactNodeViewRenderer`, …) that eagerly pull in react-dom — which throws
// `navigator is not defined` under jest's node environment.
//
// So this stub DELEGATES to the real `@tiptap/core` (keeping `mergeAttributes`,
// `Node`, `Mark`, `nodeInputRule`, … working — they are used by
// `jsonToHtml`/`htmlToJson` on the server) and overrides ONLY the React view
// helpers with no-ops. Those helpers are referenced solely inside `addNodeView()`
// — code that runs only in a live browser editor, never on the server; if any
// were actually invoked here it would (correctly) surface as a test failure.
const core = require('@tiptap/core');
module.exports = {
...core,
ReactNodeViewRenderer: () => () => ({}),
NodeViewWrapper: () => null,
NodeViewContent: () => null,
ReactRenderer: class {},
};
+9
View File
@@ -131,5 +131,14 @@ const { Client } = require("pg");
7. **Migrations don't auto-run in dev** — run `migration:latest` after every pull 7. **Migrations don't auto-run in dev** — run `migration:latest` after every pull
or branch switch. or branch switch.
8. **Automation (Playwright): type into the BODY editor, not the title.** A page has
two `.ProseMirror` editors — `[aria-label='Page title']` (non-collab) and
`[aria-label='Page content']` (the collab body). `document.querySelector('.ProseMirror')`
returns the TITLE editor, so typing there never changes body content and `mod+S`
versions nothing. Target `[aria-label='Page content']`, confirm it's collab-bound
(`el.editor.extensionManager.extensions.some(e=>e.name==='collaboration')`), and
wait ~10-12s for the store debounce before asserting `pages.content` changed. Full
testing methodology + traps: **[how-to-test.md](how-to-test.md)**.
See also the **Commands** and **Architecture → Two server processes** sections in See also the **Commands** and **Architecture → Two server processes** sections in
[`AGENTS.md`](../AGENTS.md). [`AGENTS.md`](../AGENTS.md).
+108
View File
@@ -0,0 +1,108 @@
# How to test the application (browser E2E + out-of-band)
How to actually verify a feature end-to-end against a running stand — driving the
**real app in a browser** and confirming results **out-of-band** in the DB/git, not
through the same API you're supposed to be testing. Written from real false-positives
that wasted hours (see **Traps** — read them before you write a test).
Prereq: a running stand — see **[dev-stand.md](dev-stand.md)**. Automation uses
Playwright (`pip install playwright && python -m playwright install chromium`).
## Principles
1. **Drive the behaviour under test through the browser.** The stand exists so you
exercise the real UI + realtime-collab + server path. Using `POST /api/pages/*` to
perform the action you're validating tests the API, not the app — an e2e suite can
do that. API calls are fine ONLY for one-time setup/fixtures, never for the
interaction you're asserting on.
2. **Evidence before claim.** Nothing "passes" without an artifact: a DB row, a git
diff, a screenshot looked at as an image. If you can't show it, you didn't verify it.
3. **Verify out-of-band.** Judge results from a source independent of the UI: `psql`
against the DB, a fresh `git clone` of a synced repo, a hard reload. Optimistic UI
lies about persistence.
4. **Disconfirm by default.** For each feature, actively try to prove it's broken
before concluding it works. Reload after every create/edit/save.
5. **Recon actuatability FIRST.** Before building editor tests, confirm the
interaction even works in your harness (does a typed edit reach the DB?). Skipping
this is how you ship a pile of tests that all silently exercised the wrong thing.
## The editor: two ProseMirror instances (READ THIS)
A page has **two** `.ProseMirror` editors:
| index | selector | role | collab? |
|---|---|---|---|
| 0 | `[aria-label='Page title']` | title field | **NO** (16 exts, no `collaboration`) |
| 1 | `[aria-label='Page content']` | body | **YES** (95 exts, has `collaboration`) |
`document.querySelector('.ProseMirror')` returns the **title** editor (first match).
Type there and you edit the title only — body page content never changes, so `mod+S`
"versions" unchanged content and every content test silently no-ops.
**Always target the body editor** and confirm it's collab-bound before typing:
```js
const el = document.querySelector("[aria-label='Page content']");
el.editor.extensionManager.extensions.some(e => e.name === 'collaboration'); // must be true
```
Body edits emit ~20 `/collab` websocket frames while typing and land in
`pages.content` after the **hocuspocus store debounce (~10s)** — so **wait ~12s**
before asserting persistence (checking at 6–8s is a false negative). `mod+S` (the
`save-version` stateless message) flushes immediately, so a version created right
after a settled body edit holds the typed text.
## A known-good browser flow
```
1. goto /s/<space-slug> # the "Create page" button lives in the space sidebar, not /home
2. click button[aria-label='Create page'] # fully UI-driven page creation
3. type into [aria-label='Page title'] # optional title
4. click [aria-label='Page content'] → type body text
5. wait ~12s (store debounce)
6. assert pages.content changed (psql) # out-of-band
7. mod+S / menu Save → assert page_history row (psql)
8. reload / fresh context → re-assert (persistence round-trip)
```
Auth: log in ONCE, save `storage_state.json`, reuse it across pages/agents (re-login
per run trips shared rate-limits). Cookie-based session authorizes both REST and the
collab websocket.
## Judging out-of-band
```bash
# page content / history
docker exec <db> psql -U docmost -d docmost -tAc \
"select coalesce(kind,'null'), content::text from page_history where page_id='<id>' order by created_at;"
# git-sync round-trip: clone the space repo and diff against what you pushed
git clone http://<user>:<pass>@127.0.0.1:3000/git/<spaceId>.git /tmp/x
```
`page_history.content` is full JSON — parse it, don't truncate the snippet, or a
marker check misses. For sync/async features (autosave, git-sync, idle-flush) use an
active probe: write a unique marker, wait past the debounce/poll window, re-read
out-of-band, ≥2 iterations — never conclude "broken" from a single snapshot.
## Traps (each of these produced a false result in a real run)
- **Wrong editor.** Typed into `.ProseMirror` (= title). Edits never touched body
content. → target `[aria-label='Page content']`.
- **Checked persistence too early.** Store debounce ~10s; a 6–8s check reads stale.
- **Truncated the DB snapshot** below where the test marker sits → false "content
missing".
- **API-seeded the content under test**, then "verified" the feature — that validated
the API, not the app.
- **Reused a fixed marker on a non-rebooted stand** → title/row collisions inflate
counts (`count==2`). Use a unique per-run marker (timestamp).
- **Idle/async read once** and called it "permanently broken" — it was mid-debounce.
- **Concluded env-limitation without a cross-build control.** If unsure whether a
failure is your harness or the product, run the SAME harness against a known-good
build; a divergence localizes it.
## Scope note
Some paths genuinely need a human in a real browser (rich drag-drop, native file
pickers, clipboard, and anything the harness can't actuate). Label those UNTESTED in
the report — "handled gracefully" is not "works". Keep four states distinct:
verified-working, defect, untested, env-limitation.
+2 -1
View File
@@ -96,7 +96,8 @@
"pnpm": { "pnpm": {
"patchedDependencies": { "patchedDependencies": {
"scimmy@1.3.5": "patches/scimmy@1.3.5.patch", "scimmy@1.3.5": "patches/scimmy@1.3.5.patch",
"yjs@13.6.30": "patches/yjs@13.6.30.patch" "yjs@13.6.30": "patches/yjs@13.6.30.patch",
"ai@6.0.134": "patches/ai@6.0.134.patch"
}, },
"overrides": { "overrides": {
"prosemirror-changeset": "2.4.0", "prosemirror-changeset": "2.4.0",
+83 -355
View File
@@ -118,56 +118,19 @@ export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
// transport exposes a `tree:true` mode that returns the full nested hierarchy; // transport exposes a `tree:true` mode that returns the full nested hierarchy;
// the in-app copy keeps the same tree option but is worded for the in-app agent. // the in-app copy keeps the same tree option but is worded for the in-app agent.
// Kept per-layer so each side can tune its own guidance. // Kept per-layer so each side can tune its own guidance.
server.registerTool( // Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294). This
"list_pages", // transport keeps applying its own defaults (limit=50, tree=false) in execute.
{ registerShared(SHARED_TOOL_SPECS.listPages, async ({ spaceId, limit, tree }) => {
description: const result = await docmostClient.listPages(spaceId, limit ?? 50, tree ?? false);
"List most recent pages in a space ordered by updatedAt (descending). " + return jsonContent(result);
"Returns a bounded list (default 50, max 100) — use search for lookups " + });
"in large spaces. Pass tree:true (with spaceId) to instead get the " +
"space's full page hierarchy as a nested tree.",
inputSchema: {
spaceId: z.string().optional(),
limit: z
.number()
.int()
.min(1)
.max(100)
.optional()
.describe("Max pages to return (default 50, max 100)"),
tree: z
.boolean()
.optional()
.describe(
"When true, return the space's full page hierarchy as a nested tree (each node has a children array) instead of the recent-by-updatedAt flat list. Requires spaceId; ignores limit.",
),
},
},
async ({ spaceId, limit, tree }) => {
const result = await docmostClient.listPages(spaceId, limit ?? 50, tree ?? false);
return jsonContent(result);
},
);
// Tool: get_page // Tool: get_page
server.registerTool( // Schema + description now live in the shared registry (#294).
"get_page", registerShared(SHARED_TOOL_SPECS.getPage, async ({ pageId }) => {
{ const page = await docmostClient.getPage(pageId);
description: return jsonContent(page);
"Get page details with content converted to Markdown. The conversion is " + });
"LOSSY (block ids, exact table/callout structure are approximated); for a " +
"lossless representation use get_page_json. Inline <span data-comment-id> " +
"tags in the markdown are comment highlight anchors (also present for " +
"RESOLVED threads) — treat them as markup, not page text.",
inputSchema: {
pageId: z.string().min(1),
},
},
async ({ pageId }) => {
const page = await docmostClient.getPage(pageId);
return jsonContent(page);
},
);
// Tool: get_page_json // Tool: get_page_json
registerShared(SHARED_TOOL_SPECS.getPageJson, async ({ pageId }) => { registerShared(SHARED_TOOL_SPECS.getPageJson, async ({ pageId }) => {
@@ -201,6 +164,10 @@ registerShared(
); );
// Tool: table_get // Tool: table_get
// NOT in the shared registry: the MCP tool name `table_get` is noun-first while
// the in-app key is `getTable` (verb-first), breaking the snake_case(inAppKey)
// convention the shared registry enforces (shared-tool-specs.contract.spec.ts).
// Renaming the public MCP tool would break external clients, so it stays inline.
server.registerTool( server.registerTool(
"table_get", "table_get",
{ {
@@ -223,25 +190,10 @@ server.registerTool(
); );
// Tool: table_insert_row // Tool: table_insert_row
// NOT in the shared registry: this transport names the table argument `table`, // Schema + description now live in the shared registry (#294); the `table`
// while the in-app tool names it `tableRef` (ai-chat-tools.service.ts). Sharing // parameter name is the canonical one (the in-app layer was unified to it).
// one buildShape would rename a public MCP parameter, so the table row/cell registerShared(
// tools stay per-transport by design. SHARED_TOOL_SPECS.tableInsertRow,
server.registerTool(
"table_insert_row",
{
description:
"Insert a row of plain-text cells into a table. `table` = `#<index>` or " +
"a block id inside it. `cells` = text per column (padded to the table's " +
"column count; error if more cells than columns). `index` = 0-based " +
"insert position (0 inserts before the header); omit to append at the end.",
inputSchema: {
pageId: z.string().min(1),
table: z.string().min(1),
cells: z.array(z.string()),
index: z.number().int().optional(),
},
},
async ({ pageId, table, cells, index }) => { async ({ pageId, table, cells, index }) => {
const result = await docmostClient.tableInsertRow( const result = await docmostClient.tableInsertRow(
pageId, pageId,
@@ -254,22 +206,9 @@ server.registerTool(
); );
// Tool: table_delete_row // Tool: table_delete_row
// NOT shared — same `table` (here) vs `tableRef` (in-app) parameter-name // Schema + description now live in the shared registry (#294).
// divergence as table_insert_row. registerShared(
server.registerTool( SHARED_TOOL_SPECS.tableDeleteRow,
"table_delete_row",
{
description:
"Delete the row at 0-based `index` from a table (`table` = `#<index>` or " +
"a block id inside it). Refuses to delete the table's only row. An " +
"out-of-range `index` throws. Deleting `index` 0 removes the header row, " +
"and the next row becomes the new header.",
inputSchema: {
pageId: z.string().min(1),
table: z.string().min(1),
index: z.number().int(),
},
},
async ({ pageId, table, index }) => { async ({ pageId, table, index }) => {
const result = await docmostClient.tableDeleteRow(pageId, table, index); const result = await docmostClient.tableDeleteRow(pageId, table, index);
return jsonContent(result); return jsonContent(result);
@@ -277,24 +216,9 @@ server.registerTool(
); );
// Tool: table_update_cell // Tool: table_update_cell
// NOT shared — same `table` (here) vs `tableRef` (in-app) parameter-name // Schema + description now live in the shared registry (#294).
// divergence as table_insert_row. registerShared(
server.registerTool( SHARED_TOOL_SPECS.tableUpdateCell,
"table_update_cell",
{
description:
"Set the plain-text content of cell [row,col] (0-based) in a table " +
"(`table` = `#<index>` or a block id inside it). Replaces the cell's " +
"content with a single text paragraph; for rich formatting use patch_node " +
"on the cell's paragraph id from table_get.",
inputSchema: {
pageId: z.string().min(1),
table: z.string().min(1),
row: z.number().int(),
col: z.number().int(),
text: z.string(),
},
},
async ({ pageId, table, row, col, text }) => { async ({ pageId, table, row, col, text }) => {
const result = await docmostClient.tableUpdateCell( const result = await docmostClient.tableUpdateCell(
pageId, pageId,
@@ -308,22 +232,9 @@ server.registerTool(
); );
// Tool: create_page // Tool: create_page
server.registerTool( // Schema + description now live in the shared registry (#294).
"create_page", registerShared(
{ SHARED_TOOL_SPECS.createPage,
description:
"Create a new page from Markdown in a space. Pass parentPageId to nest " +
"it under a parent; omit it to create at the space root.",
inputSchema: {
title: z.string().min(1).describe("Title of the page"),
content: z.string().min(1).describe("Markdown content"),
spaceId: z.string().min(1),
parentPageId: z
.string()
.optional()
.describe("Optional parent page ID to nest under"),
},
},
async ({ title, content, spaceId, parentPageId }) => { async ({ title, content, spaceId, parentPageId }) => {
const result = await docmostClient.createPage( const result = await docmostClient.createPage(
title, title,
@@ -336,32 +247,11 @@ server.registerTool(
); );
// Tool: update_page_json // Tool: update_page_json
server.registerTool( // Schema + description now live in the shared registry (#294). The execute body
"update_page_json", // keeps this transport's content normalization (parse a JSON-string content,
{ // pass undefined/null through for a title-only/no-op update).
description: registerShared(
"Replace a page's content with a raw ProseMirror JSON document " + SHARED_TOOL_SPECS.updatePageJson,
"(lossless write: preserves the block ids, callouts, tables and " +
"attributes you pass in). Typical flow: get_page_json -> modify the " +
"JSON -> update_page_json. Keep existing node ids intact so heading " +
"anchors and history stay stable. Minimal full-doc example: " +
'{"type":"doc","content":[{"type":"paragraph","content":' +
'[{"type":"text","text":"Hi"}]}]}. `content` may be a JSON object or a ' +
"JSON string (both accepted), and is OPTIONAL: omit it to update only " +
"the title (though prefer rename_page for a title-only change). " +
"Supplying neither content nor title is an error.",
inputSchema: {
pageId: z.string().min(1).describe("ID of the page to update"),
content: z
.any()
.optional()
.describe(
'ProseMirror document {"type":"doc","content":[...]} (JSON object or ' +
"JSON string). Omit to rename only.",
),
title: z.string().optional().describe("Optional new title"),
},
},
async ({ pageId, content, title }) => { async ({ pageId, content, title }) => {
// Only parse/validate the document when it was actually supplied; when it // Only parse/validate the document when it was actually supplied; when it
// is omitted, pass it straight through so the client performs a title-only // is omitted, pass it straight through so the client performs a title-only
@@ -379,26 +269,11 @@ server.registerTool(
); );
// Tool: export_page_markdown // Tool: export_page_markdown
server.registerTool( // Schema + description now live in the shared registry (#294).
"export_page_markdown", registerShared(SHARED_TOOL_SPECS.exportPageMarkdown, async ({ pageId }) => {
{ const md = await docmostClient.exportPageMarkdown(pageId);
description: return { content: [{ type: "text" as const, text: md }] };
"Export a page to a single self-contained, lossless Docmost-flavoured " + });
"Markdown file (custom extensions): YAML-free meta header, body with " +
"inline comment anchors and diagrams, and a trailing comments-thread " +
"block. Designed for a download -> edit body -> import_page_markdown " +
"round-trip that preserves everything, including comment highlights. " +
"Comment THREADS are preserved in the file but are not re-pushed to the " +
"server on import.",
inputSchema: {
pageId: z.string().min(1),
},
},
async ({ pageId }) => {
const md = await docmostClient.exportPageMarkdown(pageId);
return { content: [{ type: "text" as const, text: md }] };
},
);
// Tool: import_page_markdown // Tool: import_page_markdown
registerShared( registerShared(
@@ -422,22 +297,11 @@ registerShared(
); );
// Tool: rename_page // Tool: rename_page
server.registerTool( // Schema + description now live in the shared registry (#294).
"rename_page", registerShared(SHARED_TOOL_SPECS.renamePage, async ({ pageId, title }) => {
{ const result = await docmostClient.renamePage(pageId, title);
description: return jsonContent(result);
"Rename a page (change its title only) without touching or resending " + });
"its content.",
inputSchema: {
pageId: z.string().min(1).describe("ID of the page to rename"),
title: z.string().min(1).describe("New title"),
},
},
async ({ pageId, title }) => {
const result = await docmostClient.renamePage(pageId, title);
return jsonContent(result);
},
);
// Tool: edit_page_text // Tool: edit_page_text
registerShared(SHARED_TOOL_SPECS.editPageText, async ({ pageId, edits }) => { registerShared(SHARED_TOOL_SPECS.editPageText, async ({ pageId, edits }) => {
@@ -516,6 +380,10 @@ registerShared(SHARED_TOOL_SPECS.deleteNode, async ({ pageId, nodeId }) => {
}); });
// Tool: insert_image // Tool: insert_image
// MCP-only by design (NOT in the shared registry): the in-app AI-chat agent
// exposes no image tools (insert/replace), so there is no second layer to unify
// — a SHARED_TOOL_SPECS entry's tier/catalogLine are in-app metadata and the
// catalog-partition test forbids a spec without a live in-app tool (#294).
server.registerTool( server.registerTool(
"insert_image", "insert_image",
{ {
@@ -561,6 +429,7 @@ server.registerTool(
); );
// Tool: replace_image // Tool: replace_image
// MCP-only by design (see insert_image): no in-app equivalent, stays inline.
server.registerTool( server.registerTool(
"replace_image", "replace_image",
{ {
@@ -603,25 +472,10 @@ server.registerTool(
); );
// Tool: share_page // Tool: share_page
// INTENTIONAL per-transport divergence (not shared): the in-app copy adds a // Schema + description now live in the shared registry (#294). The execute body
// security-confirmation framing ("only share when the user explicitly asked, // keeps this transport's own `searchIndexing ?? true` default.
// since this exposes the page to anyone with the link") tuned for the in-app registerShared(
// agent; this transport keeps the plain public-URL wording. SHARED_TOOL_SPECS.sharePage,
server.registerTool(
"share_page",
{
description:
"Make a page publicly accessible (idempotent) and return its public " +
"URL. The URL format is <app>/share/<key>/p/<slugId>. This exposes the " +
"page content to ANYONE with the URL — do it only when explicitly asked.",
inputSchema: {
pageId: z.string().min(1).describe("ID of the page to share"),
searchIndexing: z
.boolean()
.optional()
.describe("Allow search engines to index the page (default true)"),
},
},
async ({ pageId, searchIndexing }) => { async ({ pageId, searchIndexing }) => {
const result = await docmostClient.sharePage(pageId, searchIndexing ?? true); const result = await docmostClient.sharePage(pageId, searchIndexing ?? true);
return jsonContent(result); return jsonContent(result);
@@ -641,29 +495,11 @@ registerShared(SHARED_TOOL_SPECS.listShares, async () => {
}); });
// Tool: move_page // Tool: move_page
server.registerTool( // Schema + description now live in the shared registry (#294). The execute body
"move_page", // keeps this transport's cycle guard, its 'null'/'' -> null string coercion, and
{ // its positive-confirmation check on the move response.
description: registerShared(
"Move a page under a new parent (nesting) or to the space root.", SHARED_TOOL_SPECS.movePage,
inputSchema: {
pageId: z.string().min(1),
parentPageId: z
.string()
.nullable()
.optional()
.describe(
"Target parent page ID. Pass 'null' or empty string to move to root.",
),
position: z
.string()
.min(5)
.optional()
.describe(
"fractional-index position key; min 5 chars; omit to append at the end.",
),
},
},
async ({ pageId, parentPageId, position }) => { async ({ pageId, parentPageId, position }) => {
const finalParentId = const finalParentId =
parentPageId === "" || parentPageId === "null" ? null : parentPageId; parentPageId === "" || parentPageId === "null" ? null : parentPageId;
@@ -698,49 +534,22 @@ server.registerTool(
); );
// Tool: delete_page // Tool: delete_page
server.registerTool( // Schema + description now live in the shared registry (#294). The shared schema
"delete_page", // exposes ONLY pageId, so no permanent/force-delete flag can reach the client.
{ registerShared(SHARED_TOOL_SPECS.deletePage, async ({ pageId }) => {
description: await docmostClient.deletePage(pageId);
"Delete a single page by ID. SOFT delete only: the page is moved to " + return {
"trash and can be restored; nothing is permanently deleted.", content: [
inputSchema: { { type: "text" as const, text: `Successfully deleted page ${pageId}` },
pageId: z.string().min(1), ],
}, };
}, });
async ({ pageId }) => {
await docmostClient.deletePage(pageId);
return {
content: [
{ type: "text" as const, text: `Successfully deleted page ${pageId}` },
],
};
},
);
// --- Comment tools (ported from upstream PR #3 by Max Nikitin) --- // --- Comment tools (ported from upstream PR #3 by Max Nikitin) ---
// Tool: list_comments // Tool: list_comments
server.registerTool( registerShared(
"list_comments", SHARED_TOOL_SPECS.listComments,
{
description:
"List comments on a page in one call (pagination is handled " +
"internally). By DEFAULT only ACTIVE threads are returned; resolved " +
"threads (a resolved top-level comment and all its replies) are hidden " +
"and their count reported as `resolvedThreadsHidden` so you can re-query " +
"with `includeResolved: true` to see everything. Returns " +
"`{ items, resolvedThreadsHidden }`. Content is returned as Markdown.",
inputSchema: {
pageId: z.string().describe("ID of the page"),
includeResolved: z
.boolean()
.optional()
.describe(
"default only active threads; true — include resolved",
),
},
},
async ({ pageId, includeResolved }) => { async ({ pageId, includeResolved }) => {
const comments = await docmostClient.listComments(pageId, includeResolved); const comments = await docmostClient.listComments(pageId, includeResolved);
return jsonContent(comments); return jsonContent(comments);
@@ -748,55 +557,11 @@ server.registerTool(
); );
// Tool: create_comment // Tool: create_comment
// INTENTIONAL per-transport divergence (not shared): the in-app copy tunes the // Schema + description now live in the shared registry (#294). The execute body
// guidance for the in-app agent (e.g. "retry with a corrected EXACT selection" // keeps this transport's own guards (require a selection for a top-level
// and "Reversible via the comment UI"); this transport keeps its own wording. // comment; reject suggestedText on a reply / without a selection).
server.registerTool( registerShared(
"create_comment", SHARED_TOOL_SPECS.createComment,
{
description:
"Create a new comment on a page. The comment is ALWAYS inline and is " +
"anchored to (highlights) its `selection` text — there are no page-level " +
"comments. Content is provided as Markdown and automatically converted. " +
"A top-level comment REQUIRES an exact `selection`; if the selection " +
"cannot be found in the page the call fails (no orphan comment is left). " +
"Replies (parentCommentId set) inherit the parent's anchor and take no " +
"selection. You may also attach a `suggestedText` proposing a replacement " +
"for the `selection`; a human applies (or rejects) it from the UI. When " +
"`suggestedText` is set the `selection` MUST occur exactly once in the " +
"page — expand it with surrounding context if it is ambiguous.",
inputSchema: {
pageId: z.string().describe("ID of the page to comment on"),
content: z.string().min(1).describe("Comment content in Markdown format"),
selection: z
.string()
.min(1)
// Enforce the documented 250-char cap to match the description above.
.max(250)
.optional()
.describe(
"EXACT contiguous text from a single paragraph/block to anchor the " +
"comment on (<=250 chars). Required for a top-level comment; omit " +
"only when replying via parentCommentId.",
),
parentCommentId: z
.string()
.optional()
.describe("Parent comment ID to create a reply (max 2 nesting levels)"),
suggestedText: z
.string()
.min(1)
.max(2000)
.optional()
.describe(
"Optional proposed replacement (PLAIN TEXT) for the `selection`, " +
"applied by a human via the UI (never auto-applied). REQUIRES a " +
"`selection`; NOT allowed on a reply. When set, the `selection` must " +
"be UNIQUE in the page — expand it with surrounding context (still " +
"<=250 chars) if it occurs more than once, or the call is refused.",
),
},
},
async ({ pageId, content, selection, parentCommentId, suggestedText }) => { async ({ pageId, content, selection, parentCommentId, suggestedText }) => {
if (!parentCommentId && (!selection || !selection.trim())) { if (!parentCommentId && (!selection || !selection.trim())) {
throw new Error( throw new Error(
@@ -872,28 +637,9 @@ server.registerTool(
); );
// Tool: resolve_comment // Tool: resolve_comment
server.registerTool( // Schema + description now live in the shared registry (#294).
"resolve_comment", registerShared(
{ SHARED_TOOL_SPECS.resolveComment,
description:
"Resolve (close) or reopen a comment thread. Only top-level comments can " +
"be resolved — the server rejects resolving a reply. Reversible: pass " +
"resolved=false to reopen. Resolving keeps the thread and its replies " +
"(unlike delete_comment, which permanently removes them).",
inputSchema: {
commentId: z
.string()
.min(1)
.describe("ID of the top-level comment thread to resolve or reopen"),
resolved: z
.boolean()
.optional()
.default(true)
.describe(
"true (default) marks the thread resolved/closed; false reopens it",
),
},
},
async ({ commentId, resolved }) => { async ({ commentId, resolved }) => {
const result = await docmostClient.resolveComment(commentId, resolved); const result = await docmostClient.resolveComment(commentId, resolved);
return jsonContent(result); return jsonContent(result);
@@ -901,30 +647,10 @@ server.registerTool(
); );
// Tool: check_new_comments // Tool: check_new_comments
server.registerTool( // Schema + description now live in the shared registry (#294). The execute body
"check_new_comments", // keeps this transport's own guard rejecting an unparseable `since` timestamp.
{ registerShared(
description: SHARED_TOOL_SPECS.checkNewComments,
"Check for new comments across pages in a space since a given timestamp. " +
"Optionally scope to a page subtree (folder). Returns only comments " +
"created after the specified time.",
inputSchema: {
spaceId: z.string().describe("Space ID to check for new comments"),
since: z
.string()
.min(1)
.describe(
"ISO 8601 timestamp — only return comments created after this time (e.g. '2026-03-10T00:00:00Z')",
),
parentPageId: z
.string()
.optional()
.describe(
"Optional root page ID to scope the check to a subtree (folder). " +
"Only pages under this parent will be checked.",
),
},
},
async ({ spaceId, since, parentPageId }) => { async ({ spaceId, since, parentPageId }) => {
// Reject an unparseable timestamp up front: otherwise the comparison // Reject an unparseable timestamp up front: otherwise the comparison
// against NaN silently treats every comment as "not new" and the tool // against NaN silently treats every comment as "not new" and the tool
@@ -1053,6 +779,8 @@ server.registerTool(
); );
// Tool: insert_footnote // Tool: insert_footnote
// MCP-only by design (see insert_image): the in-app AI-chat agent exposes no
// footnote tool, so there is no second layer to unify — stays inline (#294).
server.registerTool( server.registerTool(
"insert_footnote", "insert_footnote",
{ {
+494
View File
@@ -316,6 +316,34 @@ export const SHARED_TOOL_SPECS = {
// --- share management --- // --- share management ---
// Unified from the per-layer inline definitions (#294). Both layers already
// carried the "only share when explicitly asked" security framing (the
// "per-transport divergence" note on the old inline copies was stale), so
// there was no real behavioral divergence to preserve — only wording drift.
sharePage: {
mcpName: 'share_page',
inAppKey: 'sharePage',
// CANONICAL: merges the MCP copy's URL-format + idempotency detail with the
// in-app copy's reversibility note; keeps the security framing both had.
description:
'Make a page PUBLICLY accessible (idempotent) and return its public URL ' +
'(format: <app>/share/<key>/p/<slugId>). This exposes the page content ' +
'to ANYONE with the URL — only share when the user explicitly asked. ' +
'Reversible: unshare it later to revoke the public URL.',
tier: 'deferred',
catalogLine: 'sharePage — make a page publicly accessible and return its URL.',
// Reconciled: MCP's stricter .min(1) on pageId kept; field descriptions from
// the in-app copy. The MCP execute keeps its own `searchIndexing ?? true`
// default (a per-layer concern, not part of the shared schema).
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to share.'),
searchIndexing: z
.boolean()
.optional()
.describe('Allow public search engines to index it (default true).'),
}),
},
unsharePage: { unsharePage: {
mcpName: 'unshare_page', mcpName: 'unshare_page',
inAppKey: 'unsharePage', inAppKey: 'unsharePage',
@@ -509,4 +537,470 @@ export const SHARED_TOOL_SPECS = {
pageId: z.string().min(1), pageId: z.string().min(1),
}), }),
}, },
// --- page tools (unified from the per-layer inline definitions, #294) ---
//
// Descriptions merge both layers (the MCP copy's richer structural notes + the
// in-app copy's "Reversible via history/trash" framing where it added one).
// Field constraints keep the MCP copy's stricter .min(1) EXCEPT where the
// in-app layer deliberately allowed a looser value (documented per field).
getPage: {
mcpName: 'get_page',
inAppKey: 'getPage',
description:
'Fetch a single page as Markdown by its id. Returns the page title and ' +
'its Markdown content. The Markdown conversion is LOSSY (block ids, exact ' +
'table/callout structure are approximated); for a lossless representation ' +
'use the lossless page-JSON read tool. Inline <span data-comment-id> tags in the markdown ' +
'are comment highlight anchors (also present for RESOLVED threads) — ' +
'treat them as markup, not page text.',
tier: 'core',
catalogLine: 'getPage — fetch a page as Markdown by its id.',
// Reconciled: MCP's stricter .min(1) kept; in-app's more-informative
// "(or slugId)" describe kept.
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id (or slugId) of the page.'),
}),
},
listPages: {
mcpName: 'list_pages',
inAppKey: 'listPages',
description:
'List the most recent pages (ordered by updatedAt, descending), ' +
'optionally scoped to a single space. Returns a bounded list (default ' +
'50, max 100) — use search for lookups in large spaces. Pass tree:true ' +
"(with spaceId) to instead get the space's full page hierarchy as a " +
'nested tree.',
tier: 'core',
catalogLine: "listPages — list recent pages, or a space's full page tree.",
buildShape: (z) => ({
spaceId: z
.string()
.optional()
.describe('Optional space id to scope the listing to.'),
limit: z
.number()
.int()
.min(1)
.max(100)
.optional()
.describe('Maximum number of pages (default 50, max 100).'),
tree: z
.boolean()
.optional()
.describe(
"When true, return the space's full page hierarchy as a nested tree " +
'(children arrays) instead of the recent-by-updatedAt flat list. ' +
'Requires spaceId; ignores limit.',
),
}),
},
createPage: {
mcpName: 'create_page',
inAppKey: 'createPage',
description:
'Create a new page with a Markdown body in a space, optionally under a ' +
'parent page (omit parentPageId to create at the space root). Returns ' +
'the new page id and title. Reversible: a page can be moved to trash ' +
'later.',
tier: 'deferred',
catalogLine: 'createPage — create a new page with a Markdown body in a space.',
// Reconciled schema DRIFT: the MCP copy pinned `content` to .min(1) while
// the in-app copy left it unbounded and DOCUMENTS an empty body as valid
// ("may be empty") — creating an empty page to fill in later is a real use
// case. The looser (no-min) form is kept, so create_page now also accepts an
// empty body (harmless — it creates an empty page) and no previously-valid
// in-app input is ever rejected. `title`/`spaceId` keep the MCP .min(1)
// (an empty title or space is never valid).
buildShape: (z) => ({
title: z.string().min(1).describe('The title of the new page.'),
content: z.string().describe('The page body as Markdown (may be empty).'),
spaceId: z.string().min(1).describe('The id of the space to create the page in.'),
parentPageId: z
.string()
.optional()
.describe('Optional parent page id to nest the new page under.'),
}),
},
movePage: {
mcpName: 'move_page',
inAppKey: 'movePage',
description:
'Move a page under a new parent page, or to the space root when no ' +
'parent is given. Reversible: move it back at any time.',
tier: 'deferred',
catalogLine: 'movePage — move a page under a new parent or to the space root.',
// Reconciled schema DRIFT: the MCP copy exposed a `position` field
// (fractional-index ordering) that the in-app copy lacked. Unified by
// KEEPING position (the in-app client already accepts an optional position
// arg, so the in-app execute now forwards it) — it is optional, so no
// previously-valid in-app call is rejected. `parentPageId` is `.nullable()`
// on both, so a real JSON null moves to root on either transport; the MCP
// execute additionally coerces the strings 'null'/'' to null as a robustness
// fallback (kept in its execute body, not in the shared schema).
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to move.'),
parentPageId: z
.string()
.nullable()
.optional()
.describe(
'Target parent page id. Null or omitted moves the page to the space ' +
'root.',
),
position: z
.string()
.min(5)
.optional()
.describe(
'Optional fractional-index position key (min 5 chars); omit to ' +
'append at the end.',
),
}),
},
renamePage: {
mcpName: 'rename_page',
inAppKey: 'renamePage',
description:
'Rename a page (change its title only; the body is untouched, never ' +
'resent). Reversible: rename back at any time.',
tier: 'deferred',
catalogLine: "renamePage — change a page's title only (body untouched).",
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to rename.'),
title: z.string().min(1).describe('The new title.'),
}),
},
deletePage: {
mcpName: 'delete_page',
inAppKey: 'deletePage',
description:
'Move a page to the trash — SOFT delete only: the page can be restored ' +
'from trash and nothing is ever permanently deleted.',
tier: 'deferred',
catalogLine: 'deletePage — move a page to trash (soft delete, reversible).',
// GUARDRAIL preserved (§14 H4): the schema exposes ONLY pageId, so a
// permanentlyDelete/forceDelete flag can never reach the client through this
// tool (asserted by ai-chat-tools.service.spec.ts).
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to move to trash.'),
}),
},
updatePageJson: {
mcpName: 'update_page_json',
inAppKey: 'updatePageJson',
description:
"Replace a page's content with a raw ProseMirror JSON document (lossless " +
'write: preserves the block ids, callouts, tables and attributes you pass ' +
'in). Typical flow: read the page-JSON view -> modify the JSON -> write it back. ' +
'Keep existing node ids intact so heading anchors and history stay ' +
'stable. Minimal full-doc example: {"type":"doc","content":[{"type":' +
'"paragraph","content":[{"type":"text","text":"Hi"}]}]}. `content` may be ' +
'a JSON object or a JSON string (both accepted), and is OPTIONAL: omit it ' +
'to update only the title (though prefer the rename-page tool for a title-only ' +
'change). Supplying neither content nor title is an error. Reversible: ' +
'the previous version is kept in page history.',
tier: 'deferred',
catalogLine:
"updatePageJson — overwrite a page's body with a full ProseMirror document.",
buildShape: (z) => ({
pageId: z.string().min(1).describe('ID of the page to update'),
content: z
.any()
.optional()
.describe(
'ProseMirror document {"type":"doc","content":[...]} (JSON object or ' +
'JSON string). Omit to update only the title.',
),
title: z.string().optional().describe('Optional new title'),
}),
},
exportPageMarkdown: {
mcpName: 'export_page_markdown',
inAppKey: 'exportPageMarkdown',
// CANONICAL: the MCP copy (a strict superset of the terse in-app wording).
description:
'Export a page to a single self-contained, lossless Docmost-flavoured ' +
'Markdown file (custom extensions): YAML-free meta header, body with ' +
'inline comment anchors and diagrams, and a trailing comments-thread ' +
'block. Designed for a download -> edit body -> page-Markdown import ' +
'round-trip that preserves everything, including comment highlights. ' +
'Comment THREADS are preserved in the file but are not re-pushed to the ' +
'server on import.',
tier: 'deferred',
catalogLine:
'exportPageMarkdown — export a page to self-contained Markdown (body + comments).',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to export.'),
}),
},
// --- comment tools (unified from the per-layer inline definitions, #294) ---
//
// create_comment and resolve_comment previously carried a "per-transport
// divergence" note in BOTH layers; #294 unifies their schema + description
// here. Only the four tools that genuinely exist in BOTH layers live in the
// registry: create/list/resolve comment and check_new_comments.
//
// update_comment and delete_comment are intentionally NOT here: they exist
// ONLY on the standalone MCP server. The in-app agent deliberately exposes no
// hard comment edit/delete tool (comment edits are irreversible / not
// version-tracked; see the guardrail tests in ai-chat-tools.service.spec.ts),
// so there is nothing to unify — they stay inline in index.ts.
createComment: {
mcpName: 'create_comment',
inAppKey: 'createComment',
// CANONICAL: the in-app copy (the more-maintained one). It keeps the same
// rules as the MCP copy — inline-only, top-level requires a `selection`, no
// page-level comments, replies inherit the anchor, suggestedText must be
// unique — and adds the "retry with a corrected EXACT selection" and reply-
// to-reply-rejected guidance the MCP copy lacked. Execute-side validation
// (reject suggestedText on a reply, require a selection) stays per-layer.
description:
'Add an INLINE comment to a page, or reply to an existing top-level ' +
'comment (one level only — the backend rejects replies to replies). ' +
'The comment is anchored inline to the given exact `selection` text ' +
'(which gets highlighted); page-level comments are NOT supported. A ' +
'new top-level comment REQUIRES a `selection`. Replies inherit the ' +
"parent's anchor and take no selection. If the call fails with a " +
'"selection not found" error, retry with a corrected EXACT selection ' +
'copied verbatim from a single paragraph/block. You may also attach a ' +
'`suggestedText` proposing a replacement for the `selection` (a human ' +
'applies it from the UI); when set, the `selection` must occur exactly ' +
'once in the page. Reversible via the comment UI.',
tier: 'core',
catalogLine:
'createComment — add an inline comment (optionally with a suggested edit).',
// Reconciled schema: the field set is identical across both layers; the
// only constraint drift is `content`, which the MCP copy pinned to
// .min(1) while the in-app copy left unbounded — the stricter MCP form is
// kept (an empty comment body is never valid).
buildShape: (z) => ({
pageId: z.string().describe('The id of the page to comment on.'),
content: z.string().min(1).describe('The comment body as Markdown.'),
selection: z
.string()
.min(1)
.max(250)
.optional()
.describe(
'EXACT contiguous text from a SINGLE paragraph/block to anchor ' +
'(highlight) the comment on (<=250 chars, avoid spanning across ' +
'formatting boundaries). Required for a new top-level comment; ' +
'omit only when replying via parentCommentId.',
),
parentCommentId: z
.string()
.optional()
.describe(
'Optional id of a TOP-LEVEL comment to reply to (one level ' +
'of replies only).',
),
suggestedText: z
.string()
.min(1)
.max(2000)
.optional()
.describe(
'Optional proposed replacement (PLAIN TEXT) for the `selection`, ' +
'applied by a human via the UI (never auto-applied). REQUIRES a ' +
'`selection`; NOT allowed on a reply. When set, the `selection` ' +
'must be UNIQUE in the page — expand it with surrounding context ' +
'(still <=250 chars) if it occurs more than once, or the call is ' +
'refused.',
),
}),
},
listComments: {
mcpName: 'list_comments',
inAppKey: 'listComments',
// CANONICAL: the two copies are near-identical; the MCP copy is the
// superset (it keeps the "(pagination is handled internally)" note the
// in-app copy dropped), so it is used verbatim.
description:
'List comments on a page in one call (pagination is handled ' +
'internally). By DEFAULT only ACTIVE threads are returned; resolved ' +
'threads (a resolved top-level comment and all its replies) are hidden ' +
'and their count reported as `resolvedThreadsHidden` so you can re-query ' +
'with `includeResolved: true` to see everything. Returns ' +
'`{ items, resolvedThreadsHidden }`. Content is returned as Markdown.',
tier: 'core',
catalogLine:
'listComments — list all comments on a page (including resolved).',
buildShape: (z) => ({
pageId: z.string().describe('ID of the page'),
includeResolved: z
.boolean()
.optional()
.describe('default only active threads; true — include resolved'),
}),
},
resolveComment: {
mcpName: 'resolve_comment',
inAppKey: 'resolveComment',
// CANONICAL: the MCP copy's richer wording, minus its snake_case reference
// to `delete_comment` (a sibling tool that does NOT exist in the in-app
// layer) — rephrased transport-neutrally per the registry convention.
description:
'Resolve (close) or reopen a top-level comment thread (reversible — ' +
'pass resolved=false to reopen). Only top-level comments can be ' +
'resolved; the server rejects resolving a reply. Resolving keeps the ' +
'thread and its replies intact (it is not a deletion).',
tier: 'core',
catalogLine: 'resolveComment — resolve or reopen a comment thread.',
// Reconciled schema: `resolved` drifted — the MCP copy made it optional
// with .default(true) (resolve is the common case, documented), the in-app
// copy made it required. The MCP form is kept (a strict superset: it never
// rejects a previously-valid input and adds a sensible default), and
// commentId keeps the MCP copy's stricter .min(1).
buildShape: (z) => ({
commentId: z
.string()
.min(1)
.describe('ID of the top-level comment thread to resolve or reopen'),
resolved: z
.boolean()
.optional()
.default(true)
.describe(
'true (default) marks the thread resolved/closed; false reopens it',
),
}),
},
checkNewComments: {
mcpName: 'check_new_comments',
inAppKey: 'checkNewComments',
// CANONICAL: the MCP copy (the more detailed of the two). The MCP layer's
// execute-side guard that rejects an unparseable `since` timestamp stays in
// its execute body (per-layer logic), not in the shared schema.
description:
'Check for new comments across pages in a space since a given ' +
'timestamp. Optionally scope to a page subtree (folder). Returns only ' +
'comments created after the specified time.',
tier: 'deferred',
catalogLine:
'checkNewComments — find comments in a space created after a timestamp.',
// Reconciled schema: `since` keeps the MCP copy's stricter .min(1) (the
// in-app copy left it unbounded); field descriptions use the MCP copy's
// more detailed wording (it carries an example timestamp).
buildShape: (z) => ({
spaceId: z.string().describe('Space ID to check for new comments'),
since: z
.string()
.min(1)
.describe(
"ISO 8601 timestamp — only return comments created after this time " +
"(e.g. '2026-03-10T00:00:00Z')",
),
parentPageId: z
.string()
.optional()
.describe(
'Optional root page ID to scope the check to a subtree (folder). ' +
'Only pages under this parent will be checked.',
),
}),
},
// --- table tools (unified from the per-layer inline definitions, #294) ---
//
// These tools carried a "NOT shared" note in BOTH layers because of a single
// parameter-NAME drift: the MCP layer named the table reference `table` while
// the in-app layer named it `tableRef`. #294 reconciles that drift by unifying
// on the MCP name `table` — renaming the MCP public parameter would break
// external MCP clients, whereas the in-app parameter is model-facing
// (prompt-only) and safe to rename. The in-app execute bodies now destructure
// `table` instead of `tableRef` (nothing else changes). Descriptions take the
// MCP copy's richer wording (it documented `#<index>`, padding, header-row
// behavior) plus the in-app copy's "Reversible via page history" note; sibling
// tool references are phrased transport-neutrally.
//
// NOT here (kept inline in index.ts): table_get / getTable. Its MCP tool name
// is noun-first (`table_get`) while the in-app key is verb-first (`getTable`),
// so it breaks the snake_case(inAppKey) naming convention the registry enforces
// (shared-tool-specs.contract.spec.ts). Renaming the public MCP tool would
// break external clients, so it stays per-transport (its in-app param was still
// aligned to `table` for consistency with the migrated trio below).
tableInsertRow: {
mcpName: 'table_insert_row',
inAppKey: 'tableInsertRow',
description:
'Insert a row of plain-text cells into a table. `table` is `#<index>` ' +
'from the page outline, or a block id inside it. `cells` is the text per ' +
"column (padded to the table's column count; an error if more cells than " +
'columns). `index` is the 0-based insert position (0 inserts before the ' +
'header); omit to append at the end. Reversible via page history.',
tier: 'deferred',
catalogLine: 'tableInsertRow — insert a row of plain-text cells into a table.',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page.'),
table: z
.string()
.min(1)
.describe('"#<index>" from the page outline, or a block id in the table.'),
cells: z.array(z.string()).describe('The cell texts for the row (one per column).'),
index: z
.number()
.int()
.optional()
.describe('0-based insert position (0 inserts before the header); omit to append.'),
}),
},
tableDeleteRow: {
mcpName: 'table_delete_row',
inAppKey: 'tableDeleteRow',
description:
'Delete the row at 0-based `index` from a table (`table` is `#<index>` ' +
'from the page outline, or a block id inside it). Refuses to delete the ' +
"table's only row; an out-of-range `index` throws. Deleting `index` 0 " +
'removes the header row, and the next row becomes the new header. ' +
'Reversible via page history.',
tier: 'deferred',
catalogLine: 'tableDeleteRow — delete a table row at a 0-based index.',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page.'),
table: z
.string()
.min(1)
.describe('"#<index>" from the page outline, or a block id in the table.'),
index: z.number().int().describe('0-based row index to delete.'),
}),
},
tableUpdateCell: {
mcpName: 'table_update_cell',
inAppKey: 'tableUpdateCell',
description:
'Set the plain-text content of cell [row, col] (0-based) in a table ' +
'(`table` is `#<index>` from the page outline, or a block id inside it). ' +
"Replaces the cell's content with a single text paragraph; for rich " +
"formatting, patch the cell's paragraph id (obtained from reading the " +
'table) instead. Reversible via page history.',
tier: 'deferred',
catalogLine: 'tableUpdateCell — set the text of a table cell at [row, col].',
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page.'),
table: z
.string()
.min(1)
.describe('"#<index>" from the page outline, or a block id in the table.'),
row: z.number().int().describe('0-based row index.'),
col: z.number().int().describe('0-based column index.'),
text: z.string().describe('The new cell text.'),
}),
},
} satisfies Record<string, SharedToolSpec>; } satisfies Record<string, SharedToolSpec>;
@@ -0,0 +1,16 @@
{
"_bug": "BUG #351: a `column` whose `width` is a percentage string (e.g. \"50%\") is NOT byte-stable across export->import->export (violates P2). The `column` schema's parseHTML does `parseFloat(getAttribute('data-width'))`, which silently drops the '%' unit and returns the NUMBER 50. So the first export emits data-width=\"50%\" but the re-import stores width=50, and the second export emits data-width=\"50\": md2 !== md1, a permanent GS-EDIT-REVERT churn (every git-sync pull rewrites the column width). The editor authors column widths as percentages, so this is a real data/round-trip defect. Fix belongs in src/lib/docmost-schema.ts column.width parseHTML (preserve the unit / keep the string), which is OUT OF SCOPE for this test-only PR and must be a separate, maintainer-approved change. This flat generator therefore keeps `column.width` frozen (never generates a non-default width).",
"doc": {
"type": "doc",
"content": [
{
"type": "columns",
"attrs": { "layout": "two_equal", "widthMode": "normal" },
"content": [
{ "type": "column", "attrs": { "width": "50%" }, "content": [{ "type": "paragraph", "content": [{ "type": "text", "text": "L" }] }] },
{ "type": "column", "attrs": { "width": "50%" }, "content": [{ "type": "paragraph", "content": [{ "type": "text", "text": "R" }] }] }
]
}
]
}
}
@@ -0,0 +1,19 @@
{
"doc": {
"type": "doc",
"content": [
{
"type": "orderedList",
"attrs": { "type": null, "start": 5 },
"content": [
{
"type": "listItem",
"content": [
{ "type": "paragraph", "content": [{ "type": "text", "text": "alpha" }] }
]
}
]
}
]
}
}
@@ -0,0 +1,325 @@
/**
* Schema-DERIVED attribute-state fast-check arbitraries (#351, PR 1).
*
* This GENERALIZES the #350 stability-matrix helper (roundtrip-stability.helper.ts)
* to fast-check. Where that helper sweeps a HAND-WRITTEN 2-state matrix for one
* node spec, this module reads the attribute list straight from
* `schema.nodes[type].spec.attrs` (never a hand list) and, per attribute,
* generates over the FOUR states the issue calls for:
*
* - `absent` : the attribute is OMITTED entirely (the empty-string-vs-
* absent churn class the #350 fix targets).
* - `default` : the schema default value, authored explicitly.
* - `nonDefault` : a representative legal non-default value.
* - `degenerate` : `""` for strings, `0`/negative for numbers, the flipped
* value for booleans.
*
* Why a per-attribute override table
* Everything that CAN be derived generically from the default's runtime type is
* (booleans flip; the degenerate value follows the runtime type). But two facts
* force a small, DOCUMENTED override table:
*
* 1. CONSTRAINED domains the schema does not encode. `image.align ∈
* {left,center,right}`, `heading.level 1..6`, `callout.type
* {info,success,warning,danger}`, `columns.layout`, table-cell `align`,
* `status.color`, `orderedList.start ≥ 1`, etc. A generic "default + 1"
* would emit an ILLEGAL value, so these get an explicit legal domain.
* 2. ROUND-TRIP-safety, established EMPIRICALLY by probing the live converter
* (the classification captured in flat-roundtrip.property.test.ts). A frozen
* attribute falls into ONE of TWO explicitly-distinguished classes never a
* silent "it just doesn't round-trip":
*
* (a) ACCEPTED LIMITATION the attribute has NO markdown representation,
* so the loss is inherent to targeting markdown, not a converter
* defect. These: `paragraph`/`heading` `indent`, `callout.icon`,
* `orderedList.type` (a/A/i markers), table `colwidth` /
* `backgroundColor(Name)` (dropped by the raw-<table> fallback). Each is
* tagged `// ACCEPTED:` inline. Freezing them is correct there is
* nothing to preserve in the target format.
*
* (b) PINNED BUG the attribute IS representable in markdown but the
* converter drops it anyway (a real defect). These are NOT silently
* frozen: each is captured as a LOUD `it.fails` counterexample in
* test/fixtures/counterexamples/ + counterexamples.test.ts, and the
* freeze here only keeps the P1/P2 union green until a MAINTAINER rules
* on accept-vs-fix (the epic guardrail reserves that call). These:
* `column.width` (parseFloat drops `%`), `orderedList.start` (non-1
* start renders as `1.`). Tagged `// PINNED-BUG:` inline.
*
* (c) DEFERRED-BUG representable AND round-trips, frozen only because the
* flat generator can't yet build a valid instance. Table
* `colspan`/`rowspan` round-trip via the raw-<table> fallback, but a
* geometrically-valid spanned table is PR-2 structural work; the flat
* generator hardcodes span = 1. Tagged `// DEFERRED-BUG:` inline so a
* maintainer does not read them as an inherent limitation.
* - Several non-null-default attrs are MATERIALIZED on import but are not
* in canonicalize's KNOWN_DEFAULTS (`callout.type`, `status.color`,
* table `colspan`/`rowspan`, `columns.layout`/`widthMode`,
* `embed.width`/`height`, `heading.level`, `taskItem.checked`,
* `details.open`, `subpages.recursive`, `orderedList.start`). If left
* `absent` they re-materialize as a non-canonical default and diverge
* under P1. We mark them `always` so they are authored explicitly.
* - The documented numericstring coercion set (`width height size
* aspectRatio`) is generated as STRINGS for the media family (a stored
* number re-parses as a string), EXCEPT `embed.width/height` which the
* embed schema keeps numeric handled per-attr.
*
* Both PINNED-BUG attrs (`column.width` P2 churn, `orderedList.start` P1 loss)
* are captured as committed `it.fails` counterexamples NOT hidden here.
*/
import fc from 'fast-check';
import { getSchema } from '@tiptap/core';
import { docmostExtensions } from '../../src/lib/index.js';
import { phraseArb, letterPhraseArb, urlArb } from './text-arbitraries.js';
/** The exact ProseMirror schema the converter targets. */
export const schema = getSchema(docmostExtensions as any);
/** Sentinel: this attribute is OMITTED (the `absent` state). */
export const ABSENT = Symbol('ABSENT');
/** The documented numeric→string coercion set (issue + roundtrip-stability.helper). */
export const NUMERIC_STRING_ATTRS = ['width', 'height', 'size', 'aspectRatio'];
/** Read the schema default for every attribute of a node type. */
export function schemaAttrDefaults(type: string): Record<string, unknown> {
const specAttrs = (schema.nodes[type]?.spec?.attrs ?? {}) as Record<
string,
{ default: unknown }
>;
const out: Record<string, unknown> = {};
for (const [k, v] of Object.entries(specAttrs)) out[k] = v.default;
return out;
}
/** Attribute names for a node type, straight from the schema (never hand-listed). */
export function schemaAttrNames(type: string): string[] {
return Object.keys((schema.nodes[type]?.spec?.attrs ?? {}) as object);
}
/**
* Per-attribute policy. Everything unlisted falls back to a generic policy:
* - a BOOLEAN default is fuzzable (its non-default is the flipped value);
* - any other default is `frozen` (only `absent`/`default` are generated) so
* we never invent an unverified non-default that might not round-trip.
* Listed attrs override this with a legal `arb` domain and/or flags.
*/
interface AttrPolicy {
/** Arbitrary for the `nonDefault` state's value. */
arb?: fc.Arbitrary<unknown>;
/** Value for the `degenerate` state (fuzz mode only). Omit to skip degenerate. */
degen?: unknown;
/** Never emit `absent` — the attr must be authored (materialized default class). */
always?: boolean;
/** Never emit the schema default value (required-ish attrs like `src`). Implies always. */
noDefault?: boolean;
/** Never emit non-default/degenerate — attr has no md representation or churns. */
frozen?: boolean;
}
const num = (...xs: number[]) => fc.constantFrom(...xs);
const str = (...xs: string[]) => fc.constantFrom(...xs);
const widthStr = str('120', '320', '640');
// The documented override table, keyed `type.attr`. Every entry is grounded in
// the empirical converter probe (see flat-roundtrip.property.test.ts header).
const OVERRIDES: Record<string, AttrPolicy> = {
// ── block text containers ────────────────────────────────────────────────
// 'left' is the IMPLICIT default alignment: the converter drops it on export
// (empirically confirmed), so it never round-trips. Only center/right/justify
// carry through the `<!--attrs {textAlign}-->` comment.
'paragraph.textAlign': { arb: str('center', 'right', 'justify') },
'paragraph.indent': { frozen: true }, // ACCEPTED: no md representation
'heading.level': { always: true, arb: num(2, 3, 4, 5, 6) },
'heading.textAlign': { arb: str('center', 'right', 'justify') },
'heading.indent': { frozen: true }, // ACCEPTED: no md representation
// ── lists ────────────────────────────────────────────────────────────────
// PINNED-BUG: markdown CAN express a non-1 start ("5."), but the converter
// renders "1." and drops it -> P1 loss. See counterexamples.test.ts
// (ordered-list-start.json). Frozen only until the maintainer rules accept-vs-fix.
'orderedList.start': { always: true, frozen: true },
'orderedList.type': { frozen: true }, // ACCEPTED: a/A/i markers not expressible in GFM
'taskItem.checked': { always: true, arb: fc.constant(true) }, // boolean, default false
// ── codeBlock ────────────────────────────────────────────────────────────
'codeBlock.language': { arb: str('js', 'ts', 'python', 'go', 'rust', 'bash') },
// ── image / media (numeric→string width family) ──────────────────────────
'image.src': { noDefault: true, arb: urlArb, degen: '' },
'image.align': { arb: str('left', 'right') },
'image.alt': { arb: letterPhraseArb, degen: '' },
'image.title': { arb: letterPhraseArb },
'image.width': { arb: widthStr, degen: '' },
'image.height': { arb: widthStr, degen: '' },
'video.src': { noDefault: true, arb: urlArb, degen: '' },
'video.alt': { arb: letterPhraseArb },
'video.width': { arb: widthStr },
'video.height': { arb: widthStr },
'audio.src': { noDefault: true, arb: urlArb, degen: '' },
'youtube.src': { noDefault: true, arb: urlArb },
'pdf.src': { noDefault: true, arb: urlArb },
'pdf.name': { arb: phraseArb },
'drawio.src': { noDefault: true, arb: urlArb },
'excalidraw.src': { noDefault: true, arb: urlArb },
'attachment.url': { noDefault: true, arb: urlArb },
'attachment.name': { arb: phraseArb },
// ── callout / status ─────────────────────────────────────────────────────
'callout.type': { always: true, arb: str('success', 'warning', 'danger') },
'callout.icon': { frozen: true }, // ACCEPTED: no md representation (dropped on export)
'status.text': { noDefault: true, arb: phraseArb, degen: '' },
'status.color': { always: true, arb: str('green', 'orange', 'red', 'blue', 'yellow', 'purple') },
// ── table cells ────────────────────────────────────────────────────────────
// DEFERRED-BUG (not ACCEPTED): colspan/rowspan ARE representable and round-trip
// — a spanned cell makes the converter emit the whole table as a raw <table>
// with colspan/rowspan attrs (markdown-converter.ts tableToHtml), which the
// tiptap parser reads back. They are frozen only because generating a
// geometrically-valid spanned table is deferred STRUCTURAL work (the flat
// generator hardcodes colspan/rowspan = 1), NOT a markdown limitation.
'tableCell.colspan': { always: true, frozen: true },
'tableCell.rowspan': { always: true, frozen: true },
// ACCEPTED: colwidth / backgroundColor(Name) have no representation — the
// raw-<table> fallback (tableToHtml) drops them, so there is nothing to preserve.
'tableCell.colwidth': { frozen: true },
'tableCell.backgroundColor': { frozen: true },
'tableCell.backgroundColorName': { frozen: true },
'tableCell.align': { arb: str('left', 'center', 'right') },
'tableHeader.colspan': { always: true, frozen: true }, // DEFERRED-BUG (see tableCell.colspan)
'tableHeader.rowspan': { always: true, frozen: true }, // DEFERRED-BUG (see tableCell.rowspan)
'tableHeader.colwidth': { frozen: true }, // ACCEPTED: no representation
'tableHeader.backgroundColor': { frozen: true }, // ACCEPTED: no representation
'tableHeader.backgroundColorName': { frozen: true }, // ACCEPTED: no representation
'tableHeader.align': { arb: str('left', 'center', 'right') },
// ── details ──────────────────────────────────────────────────────────────
'details.open': { always: true, arb: fc.constant(true) }, // boolean, default false
// ── columns ──────────────────────────────────────────────────────────────
'columns.layout': { always: true, arb: str('three_equal', 'left_sidebar', 'right_sidebar') },
// widthMode round-trips via the `data-width-mode` attribute (verified P1+P2),
// so it is fuzzed, not frozen.
'columns.widthMode': { always: true, arb: str('custom') },
// PINNED-BUG: parseFloat import drops the `%` unit -> P2 churn. See
// counterexamples.test.ts (columns-column-width-percent.json).
'column.width': { frozen: true },
// ── embed (schema keeps width/height NUMERIC, not string-coerced) ─────────
'embed.src': { noDefault: true, arb: urlArb, degen: '' },
'embed.provider': { noDefault: true, arb: str('iframe', 'youtube', 'vimeo') },
'embed.width': { always: true, frozen: true },
'embed.height': { always: true, frozen: true },
// ── subpages / math / htmlEmbed ──────────────────────────────────────────
'subpages.recursive': { always: true, arb: fc.constant(true) }, // boolean, default false
'mathBlock.text': { noDefault: true, arb: str('x^2', 'a < b', '\\frac{1}{2}'), degen: '' },
'mathInline.text': { noDefault: true, arb: str('x^2', 'a < b', '\\frac{1}{2}'), degen: '' },
'htmlEmbed.source': { noDefault: true, arb: str('<b>hi</b>', '<i>x</i>', '<span>y</span>'), degen: '' },
'htmlEmbed.height': { arb: num(200, 300, 400) },
// ── footnotes / transclusion / pageEmbed / mention ───────────────────────
'footnoteDefinition.id': { noDefault: true, arb: str('fn1', 'fn2', 'note') },
'footnoteReference.id': { noDefault: true, arb: str('fn1', 'fn2', 'note') },
'pageEmbed.sourcePageId': { noDefault: true, arb: fc.uuid() },
'transclusionSource.id': { noDefault: true, arb: str('src1', 'src2') },
'transclusionReference.sourcePageId': { noDefault: true, arb: fc.uuid() },
'transclusionReference.transclusionId': { noDefault: true, arb: str('tr1', 'tr2') },
'mention.id': { noDefault: true, arb: fc.uuid() },
'mention.label': { noDefault: true, arb: phraseArb },
'mention.entityType': { noDefault: true, arb: str('user') },
'mention.entityId': { noDefault: true, arb: fc.uuid() },
};
/** Resolve the effective policy for one attribute (override merged over generic). */
function policyFor(type: string, attr: string, def: unknown): AttrPolicy {
const override = OVERRIDES[`${type}.${attr}`];
if (override) return override;
// Generic: booleans are fuzzable via their flipped value; everything else is
// frozen (only absent/default) so no unverified non-default is invented.
if (typeof def === 'boolean') return { arb: fc.constant(!def) };
return { frozen: true };
}
/**
* Whether an attribute is actually exercised at a NON-DEFAULT value (i.e. its
* policy has an `arb`, which the generic fallback does not). Used by the
* attribute-coverage snapshot test to make the generic-frozen space VISIBLE: any
* string/number attr not in OVERRIDES is silently only tested at absent/default,
* so the snapshot pins exactly which attrs are NOT value-fuzzed and forces a
* reviewer to look when a new attr lands in that invisible bucket.
*/
export function attrIsValueFuzzed(type: string, attr: string): boolean {
const def = schemaAttrDefaults(type)[attr];
return !!policyFor(type, attr, def).arb;
}
/** Every node `type.attr` in the schema (excluding the auto `id`), sorted. */
export function allSchemaAttrKeys(): string[] {
const keys: string[] = [];
for (const type of Object.keys(schema.nodes)) {
for (const attr of schemaAttrNames(type)) {
if (attr === 'id') continue;
keys.push(`${type}.${attr}`);
}
}
return keys.sort();
}
/**
* Every MARK attribute in the schema, keyed `mark:<name>.<attr>`, sorted. Marks
* are not driven by the node OVERRIDES table (they are fuzzed by the text
* generator, text-arbitraries.ts), so their value-fuzz coverage is tracked with a
* separate snapshot (see flat-roundtrip.property.test.ts) without this the
* "no invisible coverage hole" guarantee would hold for node attrs only, letting a
* new mark attr slip through unfuzzed and unallowlisted.
*/
export function allSchemaMarkAttrKeys(): string[] {
const keys: string[] = [];
for (const [name, mark] of Object.entries(schema.marks)) {
const attrs = (mark.spec?.attrs ?? {}) as Record<string, unknown>;
for (const attr of Object.keys(attrs)) keys.push(`mark:${name}.${attr}`);
}
return keys.sort();
}
export type AttrMode = 'p1' | 'fuzz';
/**
* Build an arbitrary for ONE attribute's value (or the ABSENT sentinel) across
* the states legal for `mode`:
* - p1 : absent / default / nonDefault (the round-trip-safe space).
* - fuzz : the above PLUS degenerate (P2 tolerates the one-time
* normalization; P3 only needs totality).
*/
export function attrValueArb(
type: string,
attr: string,
mode: AttrMode,
): fc.Arbitrary<unknown | typeof ABSENT> {
const def = schemaAttrDefaults(type)[attr];
const p = policyFor(type, attr, def);
const states: fc.Arbitrary<unknown | typeof ABSENT>[] = [];
if (!p.always && !p.noDefault) states.push(fc.constant(ABSENT));
if (!p.noDefault) states.push(fc.constant(def));
if (!p.frozen && p.arb) states.push(p.arb);
if (mode === 'fuzz' && !p.frozen && p.degen !== undefined) {
states.push(fc.constant(p.degen));
}
if (states.length === 0) states.push(fc.constant(def));
return fc.oneof(...states);
}
/**
* Build an arbitrary for a node's full `attrs` object over all schema attrs.
* `base` pins caller-required attrs (e.g. a concrete `src`) verbatim; any attr
* present in `base` is NOT re-generated. Omitted (ABSENT) attrs are dropped.
*/
export function nodeAttrsArb(
type: string,
mode: AttrMode,
base: Record<string, unknown> = {},
): fc.Arbitrary<Record<string, unknown>> {
const names = schemaAttrNames(type).filter((n) => !(n in base) && n !== 'id');
if (names.length === 0) return fc.constant({ ...base });
return fc
.tuple(...names.map((n) => attrValueArb(type, n, mode)))
.map((vals) => {
const attrs: Record<string, unknown> = { ...base };
names.forEach((n, i) => {
if (vals[i] !== ABSENT) attrs[n] = vals[i];
});
return attrs;
});
}

Some files were not shown because too many files have changed in this diff Show More