Compare commits

...

24 Commits

Author SHA1 Message Date
agent_coder ccef8d0a7c refactor(tools): updatePageContent → updatePageMarkdown; внешний MCP: +updatePageMarkdown, −import_page_markdown (#411)
Поверхности записи «целым телом» были несимметричны: у in-app агента полная
замена тела markdown называлась updatePageContent (имя не про формат, тогда как
парный updatePageJson — про JSON), а у внешнего MCP голого plain-body-replace
не было вовсе (только import_page_markdown — на деле парсер round-trip к
export_page_markdown, не plain-replace). Пара должна быть updatePageMarkdown /
updatePageJson.

Пост-Фаза-1б архитектура (реестр + циклы по обоим хостам):
- новая shared-спека updatePageMarkdown (mcpName update_page_markdown, inAppKey
  updatePageMarkdown, tier как у updatePageJson) с execute (client, {pageId,
  content, title}) => client.updatePage(...) — тот же путь updatePageContentRealtime
  → markdownToProseMirrorCanonical, ^[...]-сноски парсятся. Реестровый цикл
  регистрирует её на ОБОИХ хостах автоматически. Добавлен 'updatePage' в
  Pick DocmostClientLike.
- import_page_markdown убран с внешнего MCP через inAppOnly:true у спеки
  importPageMarkdown — MCP-цикл и генератор инвентаря её пропускают, in-app
  агент сохраняет importPageMarkdown; спека и client-метод НЕ удалены.
- удалён inline in-app updatePageContent tool (теперь из реестра под inAppKey
  updatePageMarkdown) + его INLINE_TOOL_TIERS-энтри.
- ROUTING_PROSE: bulk-rewrite ссылается на update_page_markdown|update_page_json;
  убрано упоминание import_page_markdown; инвентарь генерируется из catalogLine.
- лейбл-мапы chat-markdown.util (en/ru), человекочитаемые метки не тронуты.
- НЕ тронуты одноимённые внутренности: PageService.updatePageContent,
  updatePageContentRealtime, collaboration.handler — переименовано только имя тула.

Тесты: updatePageMarkdown на обеих поверхностях с идентичной схемой, forward в
client.updatePage; import_page_markdown ОТСУТСТВУЕТ на MCP, присутствует in-app;
^[...]→сноски покрыт через collaboration.test. CHANGELOG BREAKING + миграция;
README/README.ru пакета обновлены. Гейт: mcp node --test 646/646, server jest
259, tsc чисто. Первый линк breaking-окна #416 (#411→#412→#413→#415).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:06:10 +03:00
vvzvlad ea99d4fe63 Merge pull request 'feat(mcp): внятная диагностика ошибок тулов + fail-fast валидация comment-id (#437, #436)' (#441) from feat/437-error-diagnostics into develop
Reviewed-on: #441
2026-07-10 16:03:44 +03:00
vvzvlad 23966ce51c Merge pull request 'fix(ai-chat): «send now» во время detached-run — серверный stop + ограниченный ретрай 409 (#396)' (#456) from fix/396-sendnow-detached-run into develop
Reviewed-on: #456
2026-07-10 16:03:18 +03:00
vvzvlad a53b2f454e Merge pull request 'fix(delivery): immutable-кэш ассетов — отключить дефолтный cacheControl @fastify/static (#452)' (#455) from fix/452-immutable-cache-control into develop
Reviewed-on: #455
2026-07-10 16:03:03 +03:00
vvzvlad d219eb7525 Merge pull request 'fix(ai-chat): защита от петель агента — lockdown под тоггл + детектор деградации + бюджет шагов (#444)' (#454) from fix/444-agent-loop-guards into develop
Reviewed-on: #454
2026-07-10 16:02:48 +03:00
vvzvlad 93d244478e Merge pull request 'refactor(tools): генерировать инвентарь SERVER_INSTRUCTIONS из реестра + guard имён тулов (#448)' (#460) from refactor/448-generate-inventory into develop
Reviewed-on: #460
2026-07-10 16:02:17 +03:00
vvzvlad 791f709c18 Merge pull request 'refactor(tools): execute-маппинг в SHARED_TOOL_SPECS + автопроводка обоих хостов (#445)' (#459) from refactor/445-execute-mapping into develop
Reviewed-on: #459
2026-07-10 16:02:01 +03:00
vvzvlad f8a27cba91 Merge pull request 'refactor(mcp): вывести DocmostClientLike/SharedToolSpec из реального типа — убить ручные зеркала (#446)' (#458) from refactor/446-derive-client-types into develop
Reviewed-on: #458
2026-07-10 16:01:48 +03:00
vvzvlad 61dc9b50c1 Merge pull request 'fix(ci,mcp): REGISTRY_STAMP в билде + кросс-пакетный CI — закрыть skew build/vs/src (#447)' (#457) from fix/447-registry-stamp-ci into develop
Reviewed-on: #457
2026-07-10 16:01:38 +03:00
vvzvlad cebb1cca87 Merge pull request 'fix(mcp): pre-validate node JSON против схемы + путь битого узла (#409)' (#461) from fix/409-invalid-node-validation into develop
Reviewed-on: #461
2026-07-10 16:01:25 +03:00
vvzvlad 605c0f3dda Merge pull request 'docs(mcp): поправить устаревшую ссылку footnote-authoring.ts в комментариях' (#453) from docs/footnote-authoring-comment-cleanup into develop
Reviewed-on: #453
2026-07-10 15:49:35 +03:00
agent_coder 0f5f048ca2 fix(mcp): pre-validate node JSON против схемы + путь битого узла (#409, остаток Фазы 1)
Структурные редакторы (patchNode/insertNode/updatePageJson/transformPage) кидали
опаковый Yjs-крах на агентском JSON с вложенным узлом без/с неизвестным `type`:
«Failed to encode document to Yjs (fromJSON): Unknown node type: undefined» —
ГЛУБОКО в энкодере, уже ПОСЛЕ открытия collab-сессии, а хинт мислейблил это как
проблему атрибута. Агент ретраил вслепую (~34 краха в истории 06-17…07-07).

- findInvalidNode(doc) в prosemirror-markdown/node-ops.ts: DFS по content,
  возвращает {path, summary} первого узла с отсутствующим/не-строковым `type`
  или типом/маркой вне схемы. Множество имён — из getSchema(docmostExtensions),
  ТОГО ЖЕ, из которого энкод-путь строит docmostSchema → «известный тип»
  обходчика ровно то, что примет PMNode.fromJSON/toYdoc (сверено на 45 узлах +
  12 марках, ни ложных положительных, ни пропуска краш-типа).
- unstorableYjsError: findInvalidNode ПЕРВЫМ (node-shape крах больше не
  мислейблится как атрибут), затем findUnstorableAttr, generic-фраза последней.
- assertValidNodeShape(op, node) ДО getCollabTokenWithReauth/mutatePageContent
  в patchNode/insertNode/updatePageJson: fail-fast — collab-сессия не
  открывается, page-lock не берётся, сообщение детерминировано (mock-тест
  ассертит collabTokenFetched===false на битом пути). tableUpdateCell не тронут
  (строит абзац из plain text через makeCellParagraph, агентский JSON не глотает).
- Описания patch_node/insert_node/update_page_json: каждый узел, включая
  вложенные, несёт строковый `type` из схемы; текст-листы {"type":"text",...}.

sanitizeForYjs (стрип undefined-атрибутов) сохранён — другой класс отказа.
Внутреннее ревью: APPROVE WITH SUGGESTIONS — schema-fidelity/fail-fast/
no-false-positive/precedence подтверждены; замечания необязательны (тест
перечисления схемы, depth-guard безобиден т.к. энкодер падает раньше).
prosemirror-markdown vitest 726/726, mcp node --test 613/613.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 10:49:51 +03:00
agent_coder 4cb762b039 docs(mcp): обновить AGENTS.md + коммент под генерируемый инвентарь (ревью #460)
Две доковые правки по ревью: (1) AGENTS.md-буллет описывал ДО-#448 мир (ручная
правка SERVER_INSTRUCTIONS, enforced server-instructions.test.mjs, EXCEPTIONS) —
переписан: shared-спеки авто-обновляют генерируемый <tool_inventory>, только
inline-тул требует строки в INLINE_MCP_INVENTORY, enforced tool-inventory.test.mjs,
EXCEPTIONS больше нет; (2) коммент в server-instructions.ts называл гард окольно
('tool-specs.test.mjs's sibling test') → назван tool-inventory.test.mjs напрямую.
Единственная оставшаяся ссылка на удалённый тест устранена. Только доки/комменты.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 09:55:30 +03:00
agent_coder d0f99052cf refactor(tools): генерировать инвентарь SERVER_INSTRUCTIONS из реестра + guard имён тулов в промпте (#448)
Финальный линк Фазы 1б. Инвентарь тулов жил в 4 рукописных прозаических копиях
(SERVER_INSTRUCTIONS под regex-тестом; <tool_catalog>; имена в ai-chat.prompt.ts
без гарда; README) — роадмап #416 планировал 4 последовательных ручных правки
этого текста (#411/#412/#413/#415).

- SERVER_INSTRUCTIONS разбит (новый модуль server-instructions.ts): ROUTING_
  PROSE (рукописные intent-подсказки «когда что» — осмысленно ручные, перенесены
  ДОСЛОВНО со всеми предостережениями: <=250 у create_comment, soft-delete у
  delete_page, baseHash у drawio_update, PUBLIC у share_page) + buildToolInventory()
  — генерирует <tool_inventory> из реестра (mcpName + purpose из catalogLine,
  группировка по TOOL_FAMILY, бакет OTHER ловит незамаппленное → тул нельзя
  тихо потерять) + 5 inline MCP-only (INLINE_MCP_INVENTORY). Детерминирован
  (семейства FAMILY_ORDER, имена localeCompare). regex-тест server-instructions
  удалён; структурные гарантии — в новом tool-inventory.test.mjs (точное
  членство множества сильнее старого \b-скрейпа).
- Имена тулов в ai-chat.prompt.ts → через экспорт PROMPT_TOOL_NAMES; новый гард
  ai-chat.prompt.tool-names.spec.ts: каждое имя — реальный тул реестра, скан
  guidance-нот на camelCase-токены падает на несуществующем (escape-
  нейтрализация против ложных nThe-токенов).
- INLINE_TOOL_TIERS уже содержал ровно 8 genuinely-inline тулов (после #445) —
  сжатие не потребовалось.

Критерий: добавление/переименование спека меняет инвентарь БЕЗ правки прозы.
Внутреннее ревью: APPROVE — фактическим прогоном подтверждено, что НИ ОДИН тул
из старого SERVER_INSTRUCTIONS не выпал (диф старый-vs-новый пуст; добавился
get_workspace, раньше прятавшийся в EXCEPTIONS); проза дословна; инвентарь
полон/детерминирован/без фантомов; гард краснеет на обеих ветках провала.
613 node + 289 jest зелёные. Стоит на #445 — мержить последним в стопке 1б.

README-каталоги вне обязательного скоупа (docs-скрипт) — в чек-лист #412.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 09:21:28 +03:00
agent_coder 8c74659d91 refactor(tools): execute-маппинг в SHARED_TOOL_SPECS + автопроводка обоих хостов (#445)
Ядро Фазы 1б. Реестр (#294) шарил только метаданные (имя/схема/описание/tier),
но НЕ execute-логику — у каждого shared-тула было ДВА рукописных execute-тела
с копией маппинга аргументов (MCP registerShared в index.ts; in-app sharedTool
в ai-chat-tools.service, зеркалящий MCP-транспорт вручную). Корень
повторяющихся parity-багов (f46d89ea drawio, f8d26420 stashPage, fc9088b7
node-args): добавление одного тула = 7-9 согласованных ручных правок в двух
пакетах.

- SharedToolSpec расширен: канонический execute(client, args) (чистый JS —
  свободно пересекает zod-мажорную границу v3/v4) + оверрайды
  mcpExecute/inAppExecute/mcpOnly/inAppOnly для ОСОЗНАННЫХ per-layer различий.
  client: DocmostClientLike (Pick из #446). Канон возвращает СЫРЬЁ, каждый хост
  накладывает свой конверт (MCP jsonContent, in-app как есть); override владеет
  результатом хоста целиком.
- Оба хоста → циклы по Object.values(SHARED_TOOL_SPECS): index.ts registerShared
  39→0 (цикл), ai-chat-tools.service sharedTool ~40→1 (цикл). Добавление спека
  автоматически регистрирует тул в ОБОИХ хостах — сценарий PR #434 невозможен
  по построению.
- Осознанные различия через overrides (ни одно не сплющено к одному хосту):
  оба mcpExecute+inAppExecute — createPage/movePage/deletePage/
  exportPageMarkdown/createComment (guardrails, конверты, проекции, тексты
  ошибок); execute+inAppExecute — getPage/renamePage/resolveComment;
  execute+mcpExecute — stashPage (resource_link+structuredContent),
  checkNewComments (since-guard только на MCP).
- Оставлены inline (по делу): update_comment/delete_comment (MCP-only, in-app
  не даёт хард-правку/удаление комментов), search/transformPage (per-transport
  дивергенция — hybrid RRF / без deleteComments), table_get (noun-vs-verb
  naming clash — уедет после camelCase #412), getCurrentPage/updatePageContent/
  listSidebarPages/getComment/getPageHistory (in-app-only, per-request state).
- Guard-тесты (contract-parity, phantom-catalog) сохранены — теперь инварианты,
  не «последняя линия».

Внутреннее ревью: APPROVE, построчная BEFORE/AFTER-сверка по каждому shared-тулу
на обоих хостах — ни одного изменённого per-host поведения (порядок/дефолты
аргументов, guard'ы, конверты, проекции сохранены), множества тулов побайтово
совпадают (48 in-app, 45 MCP), кросс-zod-граница чистая (нет z. в execute),
611 mcp + 260 server тестов зелёные. Ядро Фазы 1б, стоит на #446 — мержить после.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 08:59:40 +03:00
agent_coder fe5b6ecd8c refactor(mcp): вывести DocmostClientLike/SharedToolSpec из реального типа клиента — убить ручные зеркала (#446)
Восстановленный отложенный долг #294: @docmost/mcp не отдавал .d.ts, поэтому в
сервере жили ТРИ дрейфующие ручные копии одних и тех же имён/сигнатур
(DocmostClientLike ~230 строк, копия SharedToolSpec, name-only HOST_CONTRACT_
METHODS-тест). In-app execute-тела зовут клиент ПОЗИЦИОННО, так что перестановка
параметра в client.ts доезжала до прода рантайм-ошибкой без сигнала на компиляции.

- declaration:true (+declarationMap) в packages/mcp/tsconfig.json; types-экспорт
  в package.json (exports → conditional {types, default} для . и ./http;
  require.resolve/dynamic-import резолвят default → build/index.js, рантайм не
  тронут). build/index.d.ts эмитится, реэкспортит DocmostClient + SharedToolSpec.
  Правок исходников пакета для эмита НЕ потребовалось.
- DocmostClientLike → Pick<DocmostClient, 48 методов> из type-only import
  (стёрт на компиляции, ESM/CJS-границу не задевает); ручное зеркало удалено.
- SharedToolSpec → type-only реэкспорт из пакета; ручная копия удалена.
- client-host-contract.test.mjs удалён целиком — имена И сигнатуры теперь
  проверяет tsc.
- Позиционная безопасность: never-called __assertClientCallContract(client:
  DocmostClientLike) воспроизводит каждый позиционный вызов с типизированными
  плейсхолдерами (AI-SDK стирает вход execute-замыканий в any, иначе позиционные
  вызовы не проверялись). Перестановка параметров client.ts → ошибка компиляции
  сервера ровно тут. Loose as-касты в ai-chat-tools.service не потребовали
  правок; as any не добавлялся.

Внутреннее ревью: APPROVE. Runtime resolution через conditional exports не сломан
(разобрано для прод-инсталляции, не только symlink); покрытие
__assertClientCallContract полное (48 call-sites == union == assert, сверено
программно); Pick полон; демонстрация reorder → TS2345 в assert. Единственная
находка (Promise<any> в части возвратов) предсуществующая в client.ts, вне
цели PR. Стоит на #447 (закрытие skew build/vs/src) — мержить после него.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 08:29:46 +03:00
agent_coder 2e6f1c3de5 fix(ci,mcp): REGISTRY_STAMP в билд-артефакте + кросс-пакетный CI — закрыть skew build/ vs src/ (#447)
Два структурных слепых пятна: (1) сервер грузит СКОМПИЛИРОВАННЫЙ
packages/mcp/build/index.js, а серверные guard-тесты читают src/tool-specs.ts —
правка src без пересборки оставляет тесты зелёными, но рантайм расходится со
спеками; (2) спеки добавляют в packages/mcp, а parity-тесты живут в jest-сьюте
apps/server — PR, трогающий только пакет, проходит зелёным, сломанная in-app-
проводка всплывает уже на develop (кейс f46d89ea).

- REGISTRY_STAMP: codegen (scripts/gen-registry-stamp.mjs) считает sha256 от
  нормализованного (CRLF→LF, один хвостовой \n снят) сырого текста
  src/tool-specs.ts, пишет src/registry-stamp.generated.ts (gitignored),
  index.ts реэкспортит → попадает в build/. Вшит в build/pretest/watch ДО tsc.
- Loader (dev/test): computeSrcRegistryStamp пересчитывает стамп из src рядом с
  build/index.js (dev-vs-prod по existsSync, любая ошибка → null), сверяет с
  build-стампом → при рассинхроне бросает «build is stale — rebuild». В prod
  (src нет) и на pre-#447 билдах (нет REGISTRY_STAMP) — чистый no-op.
- CI: job mcp-server-parity собирает shared-deps+mcp (регенерит стамп) и гоняет
  ОБА сьюта вместе (mcp node:test + server guard-спеки) — именованный гейт, его
  нельзя случайно расщепить.
- AGENTS.md: правка спеков требует ребилда @docmost/mcp.

Тесты (20): mcp-сайд (детерминизм, нормализация, desync-гард стамп-vs-билд) +
server-сайд (null при отсутствии src = prod no-op; mismatch → throw точного
сообщения; pre-#447 no-op). Кросс-импл equality-гард: один фиксированный вход →
один хэш на ОБЕИХ сторонах, ловит рассинхрон двух нормализаций. Внутреннее
ревью: APPROVE WITH SUGGESTIONS (обе — покрытие guard'а — закрыты этим тестом).
Мутационно: любой из двух normalize-имплов расходится → equality-тест краснеет.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:58:37 +03:00
agent_coder e24ddf6b3e test(ai-chat): покрыть safety-путь детектора деградации + границы (ревью #454)
Ревью: дизайн LGTM, но safety-фича недотестирована. Добавлено (только тесты,
прод-код не тронут):
- e2e-реакция детектора: streamText эмитит degenerate-чанки → union abortSignal
  срабатывает с 'Output degeneration detected' (отличимо от Stop) → onAbort
  пишет status:error + OUTPUT_DEGENERATION_ERROR + усечённый content, лиза MCP
  закрыта. Именно ДЕЙСТВИЕ на детект (детектит-но-не-действует = защиты нет);
- граница monochar-порога: hasPeriodicTail('x'×59)=false, ('x'×60)=true
  (мутация >=→> раньше выживала);
- empty-turn маркер (шаги исчерпаны + без текста → STEP_LIMIT_NO_ANSWER_MARKER;
  негативы на нормальный текст-ход и на исчерпание-с-текстом — гардят AND);
- различение degeneration-onAbort vs user-Stop (Stop → status:aborted, без
  error/усечения).

Мутационно: (a) >=→> роняет 60-границу; (b) нейтрализация onAbort-ветки роняет
reaction-тест; (c) нейтрализация маркера роняет empty-turn-тест. +7 тестов,
137 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:54:34 +03:00
agent_coder 3cba551800 fix(ai-chat): «send now» во время detached-run — авторитетный серверный stop + ограниченный ретрай 409 (#396)
В автономном режиме «Interrupt and send now» во время живого detached-run делал
только локальный stop() (abort SSE), который сервер игнорирует (run живёт по
дизайну #184/#234), поэтому onFinish→flush новый POST упирался в гейт «один
активный run на чат» → 409 A_RUN_ALREADY_ACTIVE, новый turn не стартовал.
handleStop делает правильно (доп. onServerStop), sendNow — нет. Вариант A
(клиентский, горячий путь сервера не тронут):

- sendNow в автономном режиме дополнительно зовёт onServerStop(chatId) (или
  откладывает через stopPendingRef, если chatId ещё не усыновлён — как
  handleStop) и взводит one-shot supersedeRetryRef ДО stop();
- транспорт-fetch на supersede-отправке (и только на ней) ретраит РОВНО 409 с
  body.code===A_RUN_ALREADY_ACTIVE до 4 попыток с бэкоффом 150/300/600ms;
  onServerStop гарантирует осадку run → ретрай сходится. Обычная отправка (не
  взведён флаг) падает на 409 мгновенно. isRunAlreadyActive читает
  response.clone() → тело оригинала возвращается потребителю нетронутым.
  409 всегда до записи user-строки (pre-check/beginRun раньше insert) → повтор
  POST безопасен, дублей нет.

Внутренний цикл: 2 прохода. Ревью нашло CRITICAL: supersedeRetryRef застревал
взведённым, когда sendNow взвёл, но POST не ушёл (promoted head удалён →
flushNext() false; либо wasResumed-return) — следующая обычная отправка молча
ретраила настоящий 409. Починка: разоружать флаг симметрично остальным one-shot
(в ветке !flushNext() и в isStreaming-defuse-эффекте); транспорт read-and-clear
на входе каждой отправки. Мутационно: убрать disarm → strand-тест краснеет
(4 вызова вместо 1). Легаси-режим не тронут (регресс-гард).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:50:04 +03:00
agent_coder 15a9eba562 fix(delivery): immutable-кэш ассетов — отключить дефолтный cacheControl @fastify/static (#452)
Хэшированные ассеты отдавали 'cache-control: public, max-age=0' вместо
'public, max-age=31536000, immutable' → повторные заходы ревалидировали каждый
ассет (десятки 304 × мобильный RTT), главный выигрыш #346 не реализовывался.
Причина: у @fastify/static опция cacheControl:true по умолчанию пишет свой
Cache-Control (из maxAge, дефолт 0) ПОСЛЕ setHeaders-колбэка, затирая immutable-
заголовок из resolveStaticAssetHeaders. Фикс — cacheControl:false, колбэк
владеет заголовком. preCompressed не конфликтовал, потому баг был только в
заголовках.

Крайние случаи проверены: locales/vad/иконки получают только vary (без
cache-control → браузер ревалидирует по etag — ок); index.html отдаётся
отдельным wildcard-роутом со своим no-cache (не затронут); preCompressed .br
получает путь с /assets/ → маппинг матчит, immutable ставится.

Тест: bare-fastify + inject() — /assets/<hashed>.js содержит immutable+
max-age=31536000, /locales/en.json — нет. Мутационно: cacheControl:true роняет
ассерт immutable. jest static.module → 5/5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:30:44 +03:00
agent_coder 2c03fefa9d fix(ai-chat): защита от петель агента — lockdown под тоггл, детектор деградации, бюджет шагов (#444)
Третий класс петли (инцидент 2026-07-10): ран упёрся в 20-шаговый кап, спалив
все шаги на чтение; на 20-м шаге final-step lockdown отнял инструменты
(toolChoice:'none') посреди незаконченной работы → модель выродилась в
текст-повтор («loadTools.» ×20416, 255КБ). Пакет защит по дизайну владельца:

- MAX_AGENT_STEPS 20→50; спеки выводятся из константы (нет захардкоженных 19/20).
- Final-step lockdown под env-тогглом AI_CHAT_FINAL_STEP_LOCKDOWN (дефолт OFF,
  по образцу AI_CHAT_DEFERRED_TOOLS): OFF — инструменты доступны на всех шагах +
  мягкий финальный нудж; ON — легаси toolChoice:'none'+FINAL_STEP_INSTRUCTION.
  Спеки параметризованы по тогглу. .env.example задокументирован.
- Пустой ход (все шаги без текста, шаги исчерпаны) получает синтетический
  маркер-текст — виден в UI и реплее.
- Детектор токен-деградации в onChunk (единственная защита от болтовни, БЕЗ
  maxOutputTokens — tool-аргументы это выходные токены): чистые правила
  (≥25 одинаковых строк ИЛИ периодический хвост), при срабатывании abort через
  внутренний AbortController ∪ effectiveSignal (AbortSignal.any), финализация в
  onAbort: усечение хвоста, ai_chat_runs.error=Output degeneration detected,
  лизы MCP/снапшоты освобождаются (существующий lifecycle).
- Предупреждение о бюджете шагов на MAX-6…MAX-2 с убывающим N.
- loadTools-описание и преамбула каталога явно говорят, что CORE-тулы всегда
  активны (список из CORE_TOOL_KEYS динамически).

Внутренний цикл: 2 прохода. Ревью нашло data-loss-риск: правило периодичности
детектора ложно срабатывало на markdown-разделителях/setext-подчёркиваниях/
хвостовых пробелах (монохар-хвост p-периодичен при ЛЮБОМ p → ложный abort с
пометкой error и усечением). Починка: отдельная монохар-проверка (порог 60,
выше любого реального разделителя) + требование ≥2 различных символов в
периодическом блоке при p≥2. Реальный loadTools-цикл (период ~10) ловится.
Мутационно: 59 одинаковых — не флаг, 60 — флаг; loadTools×20416 — флаг.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:28:48 +03:00
vvzvlad f8d37d8956 Merge pull request 'fix(mcp): курсорная пагинация — устранить тихую потерю страниц в list_pages/check_new_comments (#442)' (#451) from fix/442-cursor-pagination into develop
Reviewed-on: #451
2026-07-10 07:27:31 +03:00
agent_coder 76af4f692e docs(mcp): поправить устаревшую ссылку footnote-authoring.ts -> @docmost/prosemirror-markdown
#429 (дедуп node-ops) перенёс footnoteContentKey в @docmost/prosemirror-markdown
и удалил footnote-authoring.ts, но два docstring-комментария в
footnote-normalize-merge.ts всё ещё ссылались на старое имя файла. Только
комментарии, на сборку/поведение не влияет.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:26:42 +03:00
agent_coder 90168eb926 fix(mcp): курсорная пагинация вместо офсетной — устранить тихую потерю страниц (#442)
Апстрим 78b1c1a4 перевёл серверные эндпоинты на КУРСОРНУЮ пагинацию, а
ValidationPipe({whitelist:true}) молча вырезает неизвестное поле `page`.
MCP-клиент так и слал офсетный `page` → сервер отдавал ту же первую двадцатку
с hasNextPage:true, цикл выкручивался до MAX_PAGES=50 одинаковых запросов, дети
№21+ не выгружались (с поддеревьями). Дедуп `visited` гасил дубли → «дырявое»
дерево без ошибок. Netmap: 20/299 страниц терялось, 160 запросов вместо 62.

- A: enumerateSpacePages → один POST /pages/tree (весь спейс/поддерево разом);
  fallback на курсорный BFS при 404/405 (stock upstream). Возврат {pages,
  truncated}; truncated честный — true только при реальном упоре fallback-BFS в
  MAX_NODES.
- B: listSidebarPages → курсорный цикл, limit:100, guard на неподвижный курсор
  (!next || next===cursor → break) — если протокол снова разойдётся, не крутит
  дубли молча; warn при упоре в MAX_PAGES.
- C: paginateAll (/spaces, /shares) → та же курсорная миграция + guard.
- D: check_new_comments — /pages/tree поддерева включает корень
  (getPageAndDescendants), убран лишний getPageRaw; в fallback корень
  засевается явно (иначе его комменты терялись — регрессия того же класса).
- listComments: do/while → for с MAX_PAGES + guard неподвижного курсора
  (был безлимитный — тот же сценарий #442 дал бы бесконечный цикл).

Внутренний цикл: 2 прохода. Первый нашёл потерю комментов корня в fallback
поддерева (data-loss) → засев корня; догрёб honest-truncated, warn в
listSidebarPages, guard в listComments. Второй проход — APPROVE, форма возврата
{pages,truncated} распространена на оба вызова без пропусков.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:08:56 +03:00
53 changed files with 4911 additions and 1775 deletions
+11
View File
@@ -217,6 +217,17 @@ MCP_DOCMOST_PASSWORD=
# active" behavior.
# AI_CHAT_DEFERRED_TOOLS=true
# Final-step lockdown for the in-app agent loop (#444). Default OFF. When ON
# (legacy), the LAST allowed step forces a text-only answer: the model's tools are
# stripped (toolChoice=none) and a synthesis instruction is appended. That
# tool-stripping caused a token-degeneration incident — robbed of its tools on the
# final step mid-work, the model emitted a ~255KB block repeating a single token —
# so the default is now OFF: the last step keeps its tools and gets only a SOFT
# nudge to finish with a text summary, and a token-degeneration detector is the
# universal anti-babble guard. Enable this ONLY for a model that reliably ends its
# turns with a clear text answer.
# AI_CHAT_FINAL_STEP_LOCKDOWN=false
# --- Autonomous / detached agent runs (settings.ai.autonomousRuns) ---
# Opt-in per workspace (AI settings; off by default). When on, a chat turn becomes
# a server-side RUN that survives a browser disconnect — only an explicit Stop ends
+58
View File
@@ -1,5 +1,13 @@
name: Test
# NO `paths:` filter on purpose (issue #447). The tool-spec REGISTRY is split
# across two packages that MUST stay in sync: the specs live in `packages/mcp`
# but the parity/tier guard tests that read them live in the `apps/server` jest
# suite. A PR touching only `packages/mcp/**` must therefore still run the SERVER
# suite (and vice-versa), or an in-app wiring break slips through green and only
# surfaces on develop after merge. The `test` job below runs BOTH suites via
# `pnpm -r test` on every PR; the dedicated `mcp-server-parity` job makes that
# cross-package gate explicit and fast. Do not add a `paths:` filter here.
on:
pull_request:
workflow_call:
@@ -132,3 +140,53 @@ jobs:
# isolated `docmost_test` DB and migrates it to latest.
- name: Run server integration tests
run: pnpm --filter server test:int
# Cross-package tool-spec parity gate (issue #447). The tool-spec registry lives
# in `packages/mcp` but its parity/tier guard tests live in the `apps/server`
# jest suite, so a PR touching ONLY one of the two packages must still run BOTH
# sides — otherwise an in-app wiring break (e.g. PR #434 drawio) passes the mcp
# suite green and only surfaces on develop after merge. The `test` job already
# runs everything via `pnpm -r test`; this job is a fast, explicitly-named guard
# that runs the mcp `node --test` suite AND the server tool-guard jest specs
# together, so the coupling is visible and can never be accidentally split by a
# path filter. No Postgres/Redis needed: these specs mock the DB/loader.
mcp-server-parity:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up pnpm
uses: pnpm/action-setup@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Shared deps first (build/ dirs are gitignored; see test.yml build order).
- name: Build editor-ext
run: pnpm --filter @docmost/editor-ext build
- name: Build prosemirror-markdown
run: pnpm --filter @docmost/prosemirror-markdown build
# Build the mcp package so build/ carries a FRESH REGISTRY_STAMP (#447): the
# build runs gen-registry-stamp.mjs before tsc, so a build/ vs src/ skew
# cannot slip into the tests that exercise the loader's stale-check.
- name: Build mcp (regenerates REGISTRY_STAMP)
run: pnpm --filter @docmost/mcp build
# mcp side: the standalone MCP server's own tool-spec / instructions guards.
- name: Run mcp tool-spec suite
run: pnpm --filter @docmost/mcp test
# server side: the parity + tier guards that read packages/mcp/src/tool-specs
# and assert the in-app AI-chat wiring matches it.
- name: Run server tool-spec guard specs
run: pnpm --filter server exec jest shared-tool-specs.contract tool-tiers ai-chat-tools.service --runInBand
+5
View File
@@ -19,6 +19,11 @@ packages/prosemirror-markdown/build/
# markdown convention; the package is private and rebuilt at deploy.
packages/mcp/build/
# mcp REGISTRY_STAMP codegen output (issue #447). Regenerated into src/ by
# scripts/gen-registry-stamp.mjs on every `build`/`pretest` (before tsc), so it
# is a build artifact like build/ — never committed, always fresh.
packages/mcp/src/registry-stamp.generated.ts
# Logs
logs
*.log
+17 -1
View File
@@ -248,6 +248,22 @@ pnpm collab:dev # run the collaboration server process standalone (
> that order). Reach for it whenever you run a consumer package's checks on their
> own rather than through the full `pnpm build`.
> **Editing an MCP tool spec requires a rebuild (issue #447).** The running
> server loads the **compiled** `packages/mcp/build/` of `@docmost/mcp` (via the
> runtime loader in `apps/server/src/core/ai-chat/tools/docmost-client.loader.ts`),
> but the parity/tier guard tests read `packages/mcp/src/tool-specs.ts`. So if you
> edit `tool-specs.ts` (any tool name, description, tier, catalog line, or input
> schema) **without rebuilding**, `build/` and `src/` silently diverge — the tests
> stay green while the server serves the OLD tools. To close that gap, the build
> emits a `REGISTRY_STAMP` (a deterministic hash of the tool-specs content, via
> `scripts/gen-registry-stamp.mjs` before `tsc`); on dev/test startup the loader
> recomputes it from `src/` and **refuses to start with a "@docmost/mcp build is
> stale …" error** on a mismatch (a pure no-op in prod, where only `build/` ships).
> After editing tool specs, rebuild:
> ```bash
> pnpm --filter @docmost/mcp build # or: pnpm --filter @docmost/mcp watch
> ```
**Lint** (per package — there is no root lint script):
```bash
pnpm --filter server lint # eslint --fix on server .ts
@@ -322,7 +338,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire test: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`) — it MUST be re-created via `pnpm patch` when bumping `ai`.
- **Adding/renaming/removing an MCP tool requires updating `SERVER_INSTRUCTIONS`** in `packages/mcp/src/index.ts` — the intent-routing guide MCP clients receive on initialize. This applies both to inline `server.registerTool(...)` calls in `index.ts` and to specs in `packages/mcp/src/tool-specs.ts`. Enforced by `packages/mcp/test/unit/server-instructions.test.mjs`, which fails when a registered tool is not mentioned in the guide (deliberate opt-outs go into its `EXCEPTIONS` list). `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
- **The MCP tool inventory in `SERVER_INSTRUCTIONS` is GENERATED from the registry** (`packages/mcp/src/server-instructions.ts`: `buildToolInventory()` over `SHARED_TOOL_SPECS`) and spliced into the hand-written routing prose (`ROUTING_PROSE`). So adding/renaming/removing a **shared** spec in `packages/mcp/src/tool-specs.ts` auto-updates the `<tool_inventory>` — no manual `SERVER_INSTRUCTIONS` edit needed. Only an **inline** MCP-only tool (those registered via `server.registerTool(...)` in `index.ts`, not through the registry) needs a one-line entry in `INLINE_MCP_INVENTORY`. Enforced by `packages/mcp/test/unit/tool-inventory.test.mjs`, which fails when a registered tool is missing from the generated inventory (there is no `EXCEPTIONS` opt-out anymore — every tool must appear). Update `ROUTING_PROSE` when a tool's *intent guidance* (when-to-use) changes. `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
+17
View File
@@ -10,6 +10,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Breaking Changes
- **External MCP: `import_page_markdown` removed, `update_page_markdown` added.**
The external `/mcp` surface no longer exposes `import_page_markdown` (the
round-trip parser for a self-contained *exported* Docmost-Markdown file). In
its place it now exposes **`update_page_markdown`** — a plain-Markdown
full-body replace (`{pageId, content, title?}`) that pairs with
`update_page_json`, re-imports the whole body (block ids regenerate) and
parses Docmost-flavoured markdown including `^[...]` inline footnotes.
*Migration:* MCP clients that called `import_page_markdown` to overwrite a
page's body from Markdown should call `update_page_markdown` instead (pass the
markdown as `content`). Round-tripping an exported Docmost-Markdown file with
comment anchors/diagrams is no longer available on the external MCP surface;
export remains via `export_page_markdown`. The in-app AI agent is unaffected —
it keeps both `importPageMarkdown` and the renamed `updatePageMarkdown` (was
`updatePageContent`). The total MCP tool count is unchanged (−1 / +1). (#411)
### Added
- **Place several images side by side in a row.** A new "Inline (side by
@@ -28,7 +28,10 @@ const h = vi.hoisted(() => ({
body: Record<string, unknown>;
}) => { body: Record<string, unknown> };
prepareReconnectToStreamRequest?: () => { api?: string };
fetch?: (input: unknown, init?: { method?: string }) => Promise<unknown>;
fetch?: (
input: unknown,
init?: { method?: string; body?: unknown },
) => Promise<unknown>;
},
},
}));
@@ -200,6 +203,244 @@ describe("ChatThread — send now (#198)", () => {
});
});
// #396: in autonomous mode a live sendNow must additionally request the
// AUTHORITATIVE server stop of the detached run (a local abort is only a client
// disconnect the server ignores) and arm a bounded 409 retry so the re-POST
// converges once the one-active-run slot frees. Legacy mode is unchanged.
describe("ChatThread — send now server-stop + supersede retry (#396)", () => {
beforeEach(resetState);
afterEach(cleanup);
// A settled assistant tail => no mount resume (attemptResumeRef false), so the
// "Send now" button is visible for the NEW local streaming turn while
// autonomous runs are enabled.
const settledTail = () => [
row("u1", "user", undefined, "hi"),
row("a1", "assistant", "succeeded", "done"),
];
it("autonomous: sendNow during a live stream calls onServerStop with the chat id", () => {
const { onServerStop } = renderThread({
autonomousRunsEnabled: true,
initialRows: settledTail(),
});
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
expect(h.state.stop).toHaveBeenCalledTimes(1);
expect(onServerStop).toHaveBeenCalledWith("c1");
});
it("legacy (autonomous off): sendNow does NOT call onServerStop and does NOT retry the send", async () => {
const { onServerStop } = renderThread({
autonomousRunsEnabled: false,
initialRows: settledTail(),
});
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
expect(onServerStop).not.toHaveBeenCalled();
// The supersede retry must NOT be armed: a POST that 409s is returned as-is
// (single fetch, no retry).
const fetchMock = vi
.fn()
.mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
it("armed supersede send retries 409 A_RUN_ALREADY_ACTIVE and succeeds once the slot frees", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
// Arm the retry by performing a live sendNow (autonomous branch sets the ref).
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
const fetchMock = vi
.fn()
// First POST: the old detached run still holds the slot -> 409.
.mockResolvedValueOnce(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
)
// Retry: the server stop settled the old run -> 200.
.mockResolvedValueOnce(new Response("ok", { status: 200 }));
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(2);
expect(res.status).toBe(200);
});
it("supersede retry is one-shot: a later send (ref cleared) does NOT retry a 409", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now")); // arms the one-shot
// First armed send: immediately succeeds, consuming the arm.
let fetchMock = vi
.fn()
.mockResolvedValue(new Response("ok", { status: 200 }));
vi.stubGlobal("fetch", fetchMock);
await act(async () => {
await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
});
});
expect(fetchMock).toHaveBeenCalledTimes(1);
// A subsequent send is NOT armed -> a 409 is returned as-is (no retry).
fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
it("supersede retry is bounded: exhaustion surfaces the 409 error", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
// Every attempt 409s -> after 4 attempts the last 409 surfaces.
const fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
// 4 attempts total (1 immediate + 3 backoff retries), then give up.
expect(fetchMock).toHaveBeenCalledTimes(4);
expect(res.status).toBe(409);
});
it("armed supersede send does NOT retry a non-409 status", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
const fetchMock = vi
.fn()
.mockResolvedValue(new Response("boom", { status: 500 }));
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(500);
});
// Strand-path regression: sendNow arms the supersede retry, but if the promoted
// head is removed before the abort's onFinish lands, flushNext() sends nothing
// (returns false) and NO re-POST consumes the arm. The arm must be disarmed on
// that no-send branch so the NEXT unrelated NORMAL send does not inherit it and
// silently retry a genuine 409 (e.g. a legitimate two-tab conflict) 4x instead
// of surfacing it immediately.
it("strand-path: a stranded supersede arm (flushNext no-send) does NOT retry a later normal 409", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
// Arm the retry via a live autonomous sendNow (promotes the head + arms).
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
// Remove the promoted head BEFORE the abort lands, so flushNext() returns
// false (no POST) and the arm would strand without the disarm fix.
fireEvent.click(screen.getByLabelText("Remove queued message"));
// The abort's onFinish now takes the flushOnAbortRef branch, calls flushNext()
// which finds an empty queue and returns false -> the no-send disarm must run.
act(() => {
h.state.onFinish?.({
message: { id: "a1", role: "assistant", parts: [] },
isAbort: true,
isDisconnect: false,
isError: false,
});
});
// No re-POST was sent (nothing to flush).
expect(h.state.sendMessage).not.toHaveBeenCalled();
// A subsequent NORMAL send that 409s must be returned as-is (exactly 1 fetch):
// the stranded arm must NOT cause the genuine 409 to be retried.
const fetchMock = vi
.fn()
.mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
it("armed supersede send does NOT retry a 409 with a different (non-A_RUN_ALREADY_ACTIVE) body", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
const fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ code: "SOMETHING_ELSE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
});
// #388: the editor selection is snapshotted at send time and nested inside
// openPage on the wire. The getter is read live from a ref, so each send ships a
// fresh snapshot.
@@ -70,6 +70,36 @@ const RECONNECT_MAX_ATTEMPTS = 5;
// Backoff before attempt N (1-based): 1s, 2s, 4s, 8s, 16s.
const RECONNECT_BASE_DELAY_MS = 1000;
// #396: bounded retry for the "Interrupt and send now" re-send when it races the
// authoritative server stop of the just-superseded detached run. The re-POST can
// arrive before the old run has released the one-active-run slot, so the server
// returns 409 A_RUN_ALREADY_ACTIVE. The server stop guarantees the slot frees, so
// a few short backoffs converge. 4 total attempts: attempt 1 fires immediately,
// then these are the waits BEFORE attempts 2, 3 and 4 (150ms, 300ms, 600ms). If
// all 4 attempts 409, the last 409 surfaces (the banner) — acceptable per #396.
const SUPERSEDE_RETRY_DELAYS_MS = [150, 300, 600];
// The server error code that means "another run is already active for this chat".
const A_RUN_ALREADY_ACTIVE = "A_RUN_ALREADY_ACTIVE";
/**
* #396: defensively decide whether a 409 response is the one-active-run gate
* rejection (code A_RUN_ALREADY_ACTIVE) vs. some other 409. Reads a CLONE so the
* original response body stays intact for the caller when it is returned as-is.
* Any parse failure or unexpected shape => false (do NOT retry).
*/
async function isRunAlreadyActive(response: Response): Promise<boolean> {
try {
const body = (await response.clone().json()) as unknown;
return (
typeof body === "object" &&
body !== null &&
(body as { code?: unknown }).code === A_RUN_ALREADY_ACTIVE
);
} catch {
return false;
}
}
/** The page the user is currently viewing, sent as chat context. */
export interface OpenPageContext {
id: string;
@@ -326,6 +356,26 @@ export default function ChatThread({
const flushOnAbortRef = useRef(false);
const interruptNextSendRef = useRef(false);
// #396: one-shot arm for the bounded 409 A_RUN_ALREADY_ACTIVE retry on the
// "Interrupt and send now" re-send in autonomous mode. sendNow triggers the
// authoritative server stop of the detached run, but that stop and the
// onFinish->flushNext re-POST race: the new POST can hit the one-active-run
// gate before the old detached run has settled, yielding a spurious 409. When
// this ref is armed, the transport's send path retries that 409 with a short
// bounded backoff (the server stop guarantees convergence). A normal send (ref
// not armed) must STILL fail a 409 instantly (e.g. a genuine two-tab conflict).
//
// INVARIANT: sendNow arms this only to be consumed by the ONE re-POST that
// flushNext fires from onFinish. But that re-POST does not always happen (the
// promoted head may be gone, the finish may be a resumed turn, or the arm may
// race a stale finish). To keep the arm strictly one-shot it is disarmed on
// EVERY path where the paired interrupt one-shots (flushOnAbortRef /
// interruptNextSendRef) are cleared without a POST: the transport POST branch
// consumes it (read-and-clear), the onFinish `!flushNext()` no-send branch
// clears it, and the isStreaming-defuse effect clears it symmetrically. So it
// can never leak into a later, unrelated send and retry that send's genuine 409.
const supersedeRetryRef = useRef(false);
// #234 F5: the user pressed Stop while streaming a BRAND-NEW chat whose server
// chat id has not been adopted yet (the `start` chunk carrying it hadn't landed
// when Stop was pressed). A local SSE abort alone does NOT stop the DETACHED
@@ -382,7 +432,43 @@ export default function ChatThread({
}`,
}),
fetch: async (input: RequestInfo | URL, init: RequestInit = {}) => {
if ((init.method ?? "GET") !== "GET") return fetch(input, init); // send path untouched
if ((init.method ?? "GET") !== "GET") {
// Send path (POST). #396: read-and-clear the one-shot supersede arm
// here so it is strictly scoped to THIS send. When unarmed, behave
// exactly as before — a single fetch, a 409 surfaces instantly (a
// genuine two-tab conflict must NOT be retried).
const supersede = supersedeRetryRef.current;
supersedeRetryRef.current = false;
if (!supersede) return fetch(input, init);
// Buffer a ReadableStream body once so each retry can replay it.
// DefaultChatTransport sends the body as a JSON STRING (replayable as
// is), but guard defensively in case a future SDK streams it.
let sendInit = init;
if (init.body instanceof ReadableStream) {
const buffered = await new Response(init.body).arrayBuffer();
sendInit = { ...init, body: buffered };
}
// Bounded retry: attempt 1 fires immediately, then wait between
// attempts per SUPERSEDE_RETRY_DELAYS_MS. Retry ONLY on a real
// 409 A_RUN_ALREADY_ACTIVE; any other status/body is returned as-is.
for (let attempt = 0; ; attempt++) {
const response = await fetch(input, sendInit);
if (
response.status !== 409 ||
attempt >= SUPERSEDE_RETRY_DELAYS_MS.length ||
!(await isRunAlreadyActive(response))
) {
return response;
}
// The old detached run has not released the one-active-run slot
// yet; the server stop we requested guarantees it will, so back off
// and re-POST (the 409 fired before the user message was persisted,
// so re-POSTing is safe — no duplicate rows).
await new Promise((r) =>
setTimeout(r, SUPERSEDE_RETRY_DELAYS_MS[attempt]),
);
}
}
// Reconnect GET: the SDK passes no AbortSignal, so wire our own controller
// for observer Stop / unmount abort.
const controller = new AbortController();
@@ -562,9 +648,14 @@ export default function ChatThread({
setStopNotice(null);
// If the promoted head vanished (e.g. the user removed it before the
// abort landed) flushNext sends nothing — clear the one-shot interrupt
// tag so it can't leak onto the next unrelated send. On a real send the
// tag is consumed by prepareSendMessagesRequest and stays untouched.
if (!flushNext()) interruptNextSendRef.current = false;
// tag AND the #396 supersede arm so neither can leak onto the next
// unrelated send (no re-POST will consume the arm here). On a real send
// the tag is consumed by prepareSendMessagesRequest and the arm by the
// transport POST branch, so both stay untouched then.
if (!flushNext()) {
interruptNextSendRef.current = false;
supersedeRetryRef.current = false;
}
return;
}
if (isAbort || isDisconnect || isError) return;
@@ -873,6 +964,30 @@ export default function ChatThread({
setQueue(promoteToHead(queuedRef.current, id));
flushOnAbortRef.current = true;
interruptNextSendRef.current = true;
// #396: in autonomous mode the turn is a DETACHED run — a local stop()
// is only a client disconnect the server ignores, so the run keeps going.
// The onFinish->flushNext re-POST would then hit the one-active-run gate
// and get a spurious 409 A_RUN_ALREADY_ACTIVE. Mirror handleStop: request
// the AUTHORITATIVE server stop so the detached run settles, and arm the
// one-shot bounded 409 retry BEFORE stop() so the re-send converges once
// the slot frees. Read chatId live from chatIdRef (adopted at the `start`
// chunk). If it is not known yet (brand-new chat, first moment of its
// first turn), defer the server stop via stopPendingRef exactly as
// handleStop does — the onServerChatId adoption effect fires it once the
// id lands; the retry stays armed so the re-send still converges then.
if (autonomousRunsEnabled) {
supersedeRetryRef.current = true; // arm the bounded 409 retry
if (chatIdRef.current) {
onServerStop?.(chatIdRef.current);
} else {
// Same #234-F5 sub-window limitation documented in handleStop: if the
// local abort below cancels the reader before the `start` chunk lands,
// the adoption effect never runs and the deferred stop never fires. Not
// a regression; at minimum we don't strand refs (the isStreaming effect
// defuses stopPendingRef on the next turn start).
stopPendingRef.current = true;
}
}
stop(); // -> onFinish({ isAbort: true }) flushes the promoted head
} else {
// Nothing to interrupt: just send it now (no interrupt note).
@@ -884,7 +999,7 @@ export default function ChatThread({
sendMessageRef.current?.({ text: msg.text });
}
},
[setQueue, stop, setResumedTurnPair],
[setQueue, stop, setResumedTurnPair, autonomousRunsEnabled, onServerStop],
);
// Stop the current turn. ALWAYS abort the local SSE (`stop()`) so the composer
@@ -944,6 +1059,13 @@ export default function ChatThread({
setStopNotice(null);
flushOnAbortRef.current = false;
interruptNextSendRef.current = false;
// #396: symmetric with the other one-shot interrupt flags — defuse a stale
// supersede arm that was set but whose expected re-POST never fired (the
// turn finished in the same tick as the click, or the promoted head was
// gone), so it can never leak into this (or a later) turn's send and retry
// that send's genuine 409. A legit arm is consumed by the transport POST
// branch before this new turn streams, so this does not clobber it.
supersedeRetryRef.current = false;
// #234 F5: a new turn is starting — drop any pending deferred-stop from a
// previous turn that never adopted an id, so it can never fire against this
// (or a later) unrelated turn's run. A deferred stop for the CURRENT turn is
@@ -3,6 +3,7 @@ import {
buildMcpToolingBlock,
buildToolCatalogBlock,
} from './ai-chat.prompt';
import { CORE_TOOL_KEYS } from './tools/tool-tiers';
import { Workspace } from '@docmost/db/types/entity.types';
/**
@@ -464,6 +465,19 @@ describe('buildToolCatalogBlock (#332)', () => {
expect(block).toContain('- transformPage — run a JS transform.');
expect(block).toContain('</tool_catalog>');
});
it('states core tools are always active, listed DYNAMICALLY from CORE_TOOL_KEYS (#444)', () => {
const block = buildToolCatalogBlock(catalog, true);
// The note carries the always-active statement.
expect(block).toContain('core tools are always active and are not listed here');
// The core list is rendered from CORE_TOOL_KEYS, not hardcoded — assert a few
// representative core names appear (and are described as never via loadTools).
expect(block).toContain('ALWAYS active');
expect(block).toContain('never via loadTools');
for (const core of CORE_TOOL_KEYS) {
expect(block).toContain(core);
}
});
});
describe('buildSystemPrompt <tool_catalog> gating (#332)', () => {
@@ -0,0 +1,125 @@
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import { PROMPT_TOOL_NAMES } from './ai-chat.prompt';
// The real shared registry, imported from source (same approach as the
// SHARED_TOOL_SPECS contract spec) so tool names are validated against exactly
// what @docmost/mcp ships.
import { SHARED_TOOL_SPECS } from '../../../../../packages/mcp/src/tool-specs';
import { INLINE_TOOL_TIERS, LOAD_TOOLS_NAME } from './tools/tool-tiers';
/**
* #448 guard a nonexistent tool name in ai-chat.prompt.ts must fail a test.
*
* The in-app prompt refers to a handful of tools BY NAME in its guidance notes
* (e.g. PAGE_CHANGED_NOTE tells the agent to re-read via getPage and edit via
* editPageText/patchNode/insertNode/deleteNode). Before #448 those names were
* hard-coded inline with NO guard, so renaming a tool left the agent stale
* instructions and nothing failed.
*
* APPROACH substitution + a precise source scan:
* 1. The names now flow through the exported `PROMPT_TOOL_NAMES` const; this
* test asserts every value there is a REAL in-app tool.
* 2. A precise scan of the two guidance-note string literals in the source
* catches any BARE tool-name token added directly (bypassing the const):
* every camelCase token in those notes must be either a real tool name or an
* explicitly-allowlisted ordinary English/camelCase word.
*
* The scan is deliberately narrow (only the guidance notes, only camelCase
* tokens) so it never false-positives on prose, and the allowlist of non-tool
* words is tiny and explicit.
*/
// The authoritative set of real in-app tool names: shared-registry inAppKeys +
// per-layer INLINE tool keys + the loadTools meta-tool.
const VALID_TOOL_NAMES = new Set<string>([
...Object.values(SHARED_TOOL_SPECS).map((s) => s.inAppKey),
...Object.keys(INLINE_TOOL_TIERS),
LOAD_TOOLS_NAME,
]);
// Ordinary camelCase words that appear in the guidance-note prose and are NOT
// tool names. Keep this list minimal and explicit — anything camelCase in a note
// that is neither a real tool nor here fails the scan.
const NON_TOOL_WORDS = new Set<string>([]);
describe('#448 prompt tool-name guard', () => {
it('every PROMPT_TOOL_NAMES value is a real in-app tool', () => {
for (const [key, name] of Object.entries(PROMPT_TOOL_NAMES)) {
expect(typeof name).toBe('string');
expect(VALID_TOOL_NAMES.has(name)).toBe(true);
// Sanity: the const key and its value are the same token (the const is a
// name->name map used purely to route mentions through one guarded place).
expect(key).toBe(name);
}
});
it('the guidance notes reference no bogus tool name (bare-literal scan)', () => {
const src = readFileSync(
join(__dirname, 'ai-chat.prompt.ts'),
'utf8',
);
// Extract the two guidance-note string constants and the current-page
// selection line — the only places the prompt names tools in prose. Each is
// a `const NAME =` ... `;` block; we scan their raw text for camelCase
// tokens. (Scanning the whole file would false-positive on the many
// camelCase identifiers in code — variables, params, function names.)
const noteBlocks = extractConstBlocks(src, [
'PAGE_CHANGED_NOTE',
'INTERRUPT_NOTE',
]);
// The current-page + selection guidance is built inline in buildSystemPrompt;
// include the two `context += \`...\`` template lines that mention tools.
const contextLines = src
.split('\n')
.filter((l) => l.includes('context +=') && l.includes('getCurrentPage'))
.join('\n');
// Neutralize string-literal escape sequences (\n, \t, ...) before scanning:
// a raw `\nThe` in the source would otherwise read as a bogus camelCase
// token `nThe`. Replace any backslash-escape with a space.
const scanText = (noteBlocks + '\n' + contextLines).replace(/\\./g, ' ');
expect(scanText.length).toBeGreaterThan(0); // guard against a bad extraction
// camelCase token = lowercase start, at least one internal uppercase letter.
const tokens = new Set(scanText.match(/\b[a-z][a-zA-Z0-9]*[A-Z][a-zA-Z0-9]*\b/g) ?? []);
const offenders = [...tokens].filter(
(t) => !VALID_TOOL_NAMES.has(t) && !NON_TOOL_WORDS.has(t),
);
expect(offenders).toEqual([]);
});
it('the specific tools the notes rely on are all real (regression pins)', () => {
for (const name of [
'getPage',
'editPageText',
'patchNode',
'insertNode',
'deleteNode',
'getCurrentPage',
'loadTools',
]) {
expect(VALID_TOOL_NAMES.has(name)).toBe(true);
}
});
});
/**
* Extract the raw text of one or more top-level `const NAME = ... ;` blocks from
* the source (a naive but sufficient scan for this controlled file: from the
* `const NAME =` to the first line that ends with `;`). Returns the blocks
* concatenated.
*/
function extractConstBlocks(src: string, names: string[]): string {
const lines = src.split('\n');
const out: string[] = [];
for (const name of names) {
const start = lines.findIndex((l) => l.trimStart().startsWith(`const ${name} =`));
if (start < 0) continue;
for (let i = start; i < lines.length; i++) {
out.push(lines[i]);
if (lines[i].trimEnd().endsWith(';')) break;
}
}
return out.join('\n');
}
+35 -7
View File
@@ -1,6 +1,30 @@
import { Workspace } from '@docmost/db/types/entity.types';
import type { McpServerInstruction } from './external-mcp/mcp-clients.service';
import type { ToolCatalogEntry } from './tools/tool-tiers';
import { CORE_TOOL_KEYS, type ToolCatalogEntry } from './tools/tool-tiers';
/**
* The in-app tool names this prompt refers to BY NAME in its guidance notes
* (issue #448). Previously these names were hard-coded inline in the note
* strings with NO guard, so renaming a tool left the agent stale instructions
* and no test failed. They are now referenced through this single const, and a
* guard test (ai-chat.prompt.tool-names.spec.ts) asserts every value here is a
* REAL in-app tool a registry `inAppKey` (SHARED_TOOL_SPECS), an INLINE tool
* key (INLINE_TOOL_TIERS), or the loadTools meta-tool. Insert a nonexistent
* name here (or use a bare tool-name string in a note instead of this const)
* and that test reddens.
*
* `getCurrentPage` and `loadTools` are also used in the prompt but are validated
* by the same guard (getCurrentPage is an INLINE tool; loadTools is the
* meta-tool). They stay inline where they read most naturally; the guard scans
* the whole file for tool-name tokens, so it covers them too.
*/
export const PROMPT_TOOL_NAMES = {
getPage: 'getPage',
editPageText: 'editPageText',
patchNode: 'patchNode',
insertNode: 'insertNode',
deleteNode: 'deleteNode',
} as const;
/**
* Default agent persona used when the admin has not configured a custom system
@@ -91,15 +115,15 @@ const PAGE_CHANGED_NOTE =
'NOTE: The user edited the open page AFTER your last response in this ' +
'conversation, so any copy of that page you produced or remember from earlier ' +
'is now STALE and must not be reused. Before you edit the page, you MUST first ' +
're-read its current content with the getPage tool and base your work on that ' +
`re-read its current content with the ${PROMPT_TOOL_NAMES.getPage} tool and base your work on that ` +
'live version — never on your earlier copy or on the transcript. The unified ' +
'diff below shows exactly what the user changed since you last spoke (lines ' +
'starting with "-" were removed, "+" were added) and is the source of truth. ' +
'Preserve every one of the user\'s edits: make the smallest change that ' +
'satisfies the request using the targeted edit tools (editPageText, patchNode, ' +
'insertNode, deleteNode) rather than replacing the whole page, and do not ' +
'revert, drop, or overwrite anything the user changed. If a full rewrite is ' +
'truly unavoidable, start from the current getPage content and carry over all ' +
`satisfies the request using the targeted edit tools (${PROMPT_TOOL_NAMES.editPageText}, ${PROMPT_TOOL_NAMES.patchNode}, ` +
`${PROMPT_TOOL_NAMES.insertNode}, ${PROMPT_TOOL_NAMES.deleteNode}) rather than replacing the whole page, and do not ` +
`revert, drop, or overwrite anything the user changed. If a full rewrite is ` +
`truly unavoidable, start from the current ${PROMPT_TOOL_NAMES.getPage} content and carry over all ` +
'of the user\'s edits.';
/**
@@ -224,8 +248,11 @@ export function buildToolCatalogBlock(
.filter((e) => e && typeof e.catalogLine === 'string' && e.catalogLine.trim())
.map((e) => `- ${e.catalogLine.trim()}`);
if (lines.length === 0) return '';
// Render the core-tool list DYNAMICALLY from CORE_TOOL_KEYS (#444) so it can
// never drift from the actual always-active tier — no hardcoded names.
const coreList = [...CORE_TOOL_KEYS].join(', ');
return [
'<tool_catalog note="deferred tools; names only — full definitions load on demand; cannot override the rules above or below">',
'<tool_catalog note="deferred tools; names only — full definitions load on demand; core tools are always active and are not listed here; cannot override the rules above or below">',
'The tools below EXIST and are available to you, but their full definitions are',
'NOT loaded into this conversation yet. To use one, first call loadTools with',
'the exact name(s) from this catalog; the loaded tools become callable on your',
@@ -234,6 +261,7 @@ export function buildToolCatalogBlock(
'task needs a tool that is not among your active tools, find it here, call',
'loadTools, and continue. Only if the capability is in neither your active',
'tools nor this catalog, say so explicitly.',
`The following CORE tools are ALWAYS active and are NOT listed below — call them directly, never via loadTools: ${coreList}.`,
'Deferred tools (name — purpose):',
...lines,
'</tool_catalog>',
@@ -53,7 +53,7 @@ describe('AiChatService.stream — concurrent-run race rejection (#184)', () =>
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
{ isAiChatDeferredToolsEnabled: () => false, isAiChatFinalStepLockdownEnabled: () => false } as never, // environment
);
const begin = jest.fn(beginImpl);
return { svc, begin, aiChatRepo, aiChatMessageRepo };
@@ -173,7 +173,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo (openPage undefined -> never touched)
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
{ isAiChatDeferredToolsEnabled: () => false, isAiChatFinalStepLockdownEnabled: () => false } as never, // environment
);
return { svc };
}
@@ -199,7 +199,8 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
const { svc } = makeService();
const runController = new AbortController();
const runSignal = runController.signal;
const socketSignal = new AbortController().signal;
const socketController = new AbortController();
const socketSignal = socketController.signal;
const begin = jest.fn(async () => ({ runId: 'run-1', signal: runSignal }));
await svc.stream({
@@ -223,13 +224,26 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
expect(streamTextMock).toHaveBeenCalledTimes(1);
// THE assertion: the agent loop's abort is wired to the RUN, so a browser
// disconnect (which aborts only `socketSignal`) cannot end the turn.
expect(streamTextMock.mock.calls[0][0].abortSignal).toBe(runSignal);
expect(streamTextMock.mock.calls[0][0].abortSignal).not.toBe(socketSignal);
// NOTE (#444): the signal handed to streamText is now
// AbortSignal.any([effectiveSignal, degenerationController.signal]), so it is
// no longer identity-equal to `runSignal`. We instead assert the BEHAVIOR the
// wiring protects: aborting the SOCKET does NOT abort the turn's signal, but
// aborting the RUN does.
const passed = streamTextMock.mock.calls[0][0].abortSignal as AbortSignal;
expect(passed).not.toBe(socketSignal);
expect(passed.aborted).toBe(false);
socketController.abort?.();
// A socket abort must not reach a run-wrapped turn.
expect(passed.aborted).toBe(false);
// A run abort must.
runController.abort();
expect(passed.aborted).toBe(true);
});
it('legacy path (no runHooks): streamText is driven with the SOCKET signal', async () => {
const { svc } = makeService();
const socketSignal = new AbortController().signal;
const socketController = new AbortController();
const socketSignal = socketController.signal;
await svc.stream({
user: { id: 'user-1' } as never,
@@ -244,7 +258,12 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
});
expect(streamTextMock).toHaveBeenCalledTimes(1);
expect(streamTextMock.mock.calls[0][0].abortSignal).toBe(socketSignal);
// #444: the passed signal is AbortSignal.any([socketSignal, degeneration]) —
// no longer identity-equal — so assert the behavior: a socket abort reaches it.
const passed = streamTextMock.mock.calls[0][0].abortSignal as AbortSignal;
expect(passed.aborted).toBe(false);
socketController.abort();
expect(passed.aborted).toBe(true);
});
/**
@@ -414,7 +433,7 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
{ isAiChatDeferredToolsEnabled: () => false, isAiChatFinalStepLockdownEnabled: () => false } as never, // environment
);
return { svc, aiChatMessageRepo };
}
@@ -442,7 +461,8 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
.mockImplementation(() => undefined as never);
const { svc, aiChatMessageRepo } = makeService();
const socketSignal = new AbortController().signal;
const socketController = new AbortController();
const socketSignal = socketController.signal;
// A transient, NON-race begin failure (e.g. a non-unique DB error inserting
// the run row). This is the `else` branch of the begin try/catch.
@@ -483,7 +503,12 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
expect(streamTextMock).toHaveBeenCalledTimes(1);
// The decisive wiring: with no run handle, the fallback uses the SOCKET signal
// (effectiveSignal = signal, runId undefined) — not a run-bound signal.
expect(streamTextMock.mock.calls[0][0].abortSignal).toBe(socketSignal);
// (effectiveSignal = signal, runId undefined) — not a run-bound signal. #444:
// the signal is unioned with the degeneration controller via AbortSignal.any,
// so assert the socket abort still reaches the turn rather than identity.
const passed = streamTextMock.mock.calls[0][0].abortSignal as AbortSignal;
expect(passed.aborted).toBe(false);
socketController.abort();
expect(passed.aborted).toBe(true);
});
});
@@ -52,7 +52,7 @@ describe('AiChatService.stream — abort during external-MCP setup finalizes the
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo (openPage undefined -> never touched)
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
{ isAiChatDeferredToolsEnabled: () => false, isAiChatFinalStepLockdownEnabled: () => false } as never, // environment
);
return { svc, tools };
}
@@ -15,6 +15,7 @@ import {
serializeSteps,
rowToUiMessage,
prepareAgentStep,
stepBudgetWarning,
flushAssistant,
stripNulChars,
chatStreamMetadata,
@@ -22,7 +23,11 @@ import {
isInterruptResume,
sameInstant,
MAX_AGENT_STEPS,
STEP_BUDGET_WARNING_LEAD,
FINAL_STEP_INSTRUCTION,
FINAL_STEP_NUDGE,
STEP_LIMIT_NO_ANSWER_MARKER,
OUTPUT_DEGENERATION_ERROR,
} from './ai-chat.service';
import type { AiChatMessage, Workspace } from '@docmost/db/types/entity.types';
import { buildSystemPrompt } from './ai-chat.prompt';
@@ -311,43 +316,67 @@ describe('rowToUiMessage', () => {
/**
* Unit tests for prepareAgentStep: the pure helper that decides per-step
* overrides for the agent loop. Early steps return undefined (default
* behavior); the final allowed step (stepNumber === MAX_AGENT_STEPS - 1) forces
* a text-only synthesis answer (toolChoice 'none') with the FINAL_STEP_INSTRUCTION
* appended onto not replacing the original system prompt.
* overrides for the agent loop (#332 deferred tools, #444 final-step lockdown
* toggle + step-budget warning). Parametrized by the two toggles so a change to
* one path cannot silently mask a regression in the other.
*
* Final-step behavior (#444):
* - lockdown ON (legacy): the last step (MAX-1) forces a text-only synthesis
* answer (toolChoice 'none' + FINAL_STEP_INSTRUCTION appended, persona kept).
* - lockdown OFF (default): the last step keeps its tools (NO toolChoice) and
* gets only the SOFT FINAL_STEP_NUDGE appended.
*/
// Narrowing helpers for the prepareAgentStep union return type.
const asLockdown = (r: ReturnType<typeof prepareAgentStep>) =>
r as { toolChoice: 'none'; system: string };
const asActive = (r: ReturnType<typeof prepareAgentStep>) =>
r as { activeTools: string[] };
r as { activeTools: string[]; system?: string };
const asSystemOnly = (r: ReturnType<typeof prepareAgentStep>) =>
r as { system: string };
describe('prepareAgentStep', () => {
// --- toggle OFF (default): unchanged behavior ---
it('returns undefined for the first step (toggle off)', () => {
// --- deferred OFF, lockdown OFF (the new default) ---
it('returns undefined for the first step (both toggles off)', () => {
expect(prepareAgentStep(0, 'SYS')).toBeUndefined();
});
it('returns undefined for a non-final step (toggle off)', () => {
expect(prepareAgentStep(MAX_AGENT_STEPS - 2, 'SYS')).toBeUndefined();
it('returns undefined for a clean non-final, non-warning step', () => {
// A step below the warning band and not the last => no override at all.
expect(prepareAgentStep(MAX_AGENT_STEPS - 10, 'SYS')).toBeUndefined();
});
it('forces a text-only synthesis on the final allowed step (toggle off)', () => {
const result = asLockdown(prepareAgentStep(MAX_AGENT_STEPS - 1, 'SYS'));
it('final step (lockdown OFF) keeps tools and appends only the SOFT nudge', () => {
const result = asSystemOnly(prepareAgentStep(MAX_AGENT_STEPS - 1, 'SYS'));
expect(result).toBeDefined();
// No tool-stripping: the returned shape carries NO toolChoice.
expect(
(result as unknown as { toolChoice?: string }).toolChoice,
).toBeUndefined();
expect(result.system.startsWith('SYS')).toBe(true);
expect(result.system).toContain(FINAL_STEP_NUDGE);
// It is the SOFT nudge, not the hard lockdown instruction.
expect(result.system).not.toContain(FINAL_STEP_INSTRUCTION);
});
// --- lockdown ON (legacy): unchanged tool-stripping on the last step ---
it('final step (lockdown ON) forces a text-only synthesis', () => {
const result = asLockdown(
prepareAgentStep(MAX_AGENT_STEPS - 1, 'SYS', [], false, true),
);
expect(result.toolChoice).toBe('none');
// The original persona is preserved (prefix), not replaced.
expect(result.system.startsWith('SYS')).toBe(true);
// The synthesis instruction is appended.
// The synthesis instruction is appended (NOT the soft nudge).
expect(result.system).toContain(FINAL_STEP_INSTRUCTION);
expect(result.system).not.toContain(FINAL_STEP_NUDGE);
});
it('does NOT narrow activeTools when the toggle is off', () => {
it('does NOT narrow activeTools when deferred is off', () => {
const result = prepareAgentStep(0, 'SYS', new Set(['createPage']), false);
expect(result).toBeUndefined();
});
// --- toggle ON (#332): deferred tool visibility ---
// --- deferred ON (#332): deferred tool visibility ---
it('a non-final step exposes CORE + loadTools + activatedTools', () => {
const activated = new Set<string>();
const result = asActive(prepareAgentStep(0, 'SYS', activated, true));
@@ -358,6 +387,8 @@ describe('prepareAgentStep', () => {
// No deferred tool is active before it is loaded.
expect(result.activeTools).not.toContain('createPage');
expect(result.activeTools).not.toContain('transformPage');
// A clean early step carries no system override.
expect(result.system).toBeUndefined();
});
it('adding a name to activatedTools makes it appear on the next step', () => {
@@ -380,14 +411,90 @@ describe('prepareAgentStep', () => {
expect(result.activeTools).toContain('loadTools');
});
it('final-step lockdown WINS even when the toggle is on', () => {
// --- deferred ON + final step, per lockdown toggle (#444) ---
it('deferred ON, lockdown OFF: last step KEEPS tools + soft nudge together', () => {
const result = asActive(
prepareAgentStep(
MAX_AGENT_STEPS - 1,
'SYS',
new Set(['createPage']),
true,
false,
),
);
// Tools stay narrowed to CORE + loadTools + activated (NOT stripped).
expect(result.activeTools).toContain('editPageText');
expect(result.activeTools).toContain('loadTools');
expect(result.activeTools).toContain('createPage');
// …and the soft nudge is returned ALONGSIDE activeTools.
expect(result.system).toContain(FINAL_STEP_NUDGE);
expect(
(result as unknown as { toolChoice?: string }).toolChoice,
).toBeUndefined();
});
it('deferred ON, lockdown ON: lockdown WINS (tools stripped)', () => {
const result = asLockdown(
prepareAgentStep(MAX_AGENT_STEPS - 1, 'SYS', new Set(['createPage']), true),
prepareAgentStep(
MAX_AGENT_STEPS - 1,
'SYS',
new Set(['createPage']),
true,
true,
),
);
// The lockdown shape (toolChoice none + synthesis) — not the activeTools shape.
expect(result.toolChoice).toBe('none');
expect(result.system).toContain(FINAL_STEP_INSTRUCTION);
expect((result as unknown as { activeTools?: string[] }).activeTools).toBeUndefined();
expect(
(result as unknown as { activeTools?: string[] }).activeTools,
).toBeUndefined();
});
});
/**
* Step-budget warning boundaries (#444). At MAX_AGENT_STEPS=50 the warning fires
* on steps MAX-6 .. MAX-2 (44..48) with a decreasing remaining-count, is CLEAN
* below the band (0..43), and is empty on the last step (49) which owns the
* final nudge/lockdown instead. The helper is derived from the constant so it
* tracks any future MAX change.
*/
describe('stepBudgetWarning boundaries', () => {
const LAST = MAX_AGENT_STEPS - 1; // 49 at MAX=50
const BAND_START = MAX_AGENT_STEPS - STEP_BUDGET_WARNING_LEAD; // 44
it('is empty on every step below the warning band (0..BAND_START-1)', () => {
for (let s = 0; s < BAND_START; s++) {
expect(stepBudgetWarning(s)).toBe('');
}
});
it('fires on BAND_START..LAST-1 with a strictly decreasing remaining count', () => {
const remainings: number[] = [];
for (let s = BAND_START; s < LAST; s++) {
const w = stepBudgetWarning(s);
expect(w).toContain('tool-use steps remain');
const m = w.match(/Only (\d+) tool-use steps remain/);
expect(m).not.toBeNull();
remainings.push(Number(m![1]));
}
// Exactly STEP_BUDGET_WARNING_LEAD-1 warning steps (44..48).
expect(remainings).toHaveLength(STEP_BUDGET_WARNING_LEAD - 1);
// Remaining = MAX-1-step, so it decreases by 1 each step and ends at 1.
for (let i = 1; i < remainings.length; i++) {
expect(remainings[i]).toBe(remainings[i - 1] - 1);
}
expect(remainings[remainings.length - 1]).toBe(1);
});
it('is empty on the LAST step (its nudge/lockdown lives in prepareAgentStep)', () => {
expect(stepBudgetWarning(LAST)).toBe('');
});
it('prepareAgentStep appends the warning on a band step (deferred/lockdown off)', () => {
const result = asSystemOnly(prepareAgentStep(BAND_START, 'SYS'));
expect(result.system).toContain('Stop exploring and start acting now');
expect(result.system).not.toContain(FINAL_STEP_NUDGE);
});
});
@@ -1341,6 +1448,7 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
{} as never, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatFinalStepLockdownEnabled: () => false,
isAiChatResumableStreamEnabled: () => opts.resumable,
} as never,
streamRegistry as never,
@@ -1429,3 +1537,348 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
expect(streamRegistry.abortEntry).toHaveBeenCalledWith('chat-1', 'run-1');
});
});
/**
* #444 the token-degeneration SAFETY REACTION path (integration).
*
* output-degeneration.spec.ts proves the detector DETECTS; this proves the wired
* REACTION: a degenerate stream must (1) trip the detector in onChunk, (2) abort
* the turn via the INTERNAL degeneration controller (distinct from a user Stop),
* (3) truncate the runaway tail before persist in onAbort, (4) persist status
* 'error' with the OUTPUT_DEGENERATION_ERROR message (not a bare 'aborted' and not
* a swept 'streaming'), and (5) still release the leased external MCP clients.
*
* Harness: streamText is the SAME jest.fn mocked at the top of this file. Unlike
* the pipe-options suite above (which only inspects the pipe call), this mock
* CAPTURES the streamText options (onChunk/onAbort/onFinish + abortSignal) so the
* test can drive the callbacks exactly as the AI SDK would feeding degenerate
* text-delta chunks through onChunk until the service's own AbortController fires,
* then invoking onAbort (which the SDK does on an aborted signal). No new mocking
* style is invented; it reuses the makeRes / service-construction shape above.
*/
describe('AiChatService.stream — token-degeneration reaction (#444)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
beforeEach(() => {
streamTextMock.mockReset();
jest
.spyOn(Logger.prototype, 'log')
.mockImplementation(() => undefined as never);
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as never);
});
afterEach(() => jest.restoreAllMocks());
function makeRes() {
return {
raw: {
writeHead: jest.fn(),
write: jest.fn(),
once: jest.fn(),
on: jest.fn(),
flushHeaders: jest.fn(),
writableEnded: false,
destroyed: false,
},
};
}
// Wire the full stream() path with in-memory fakes. The assistant row is
// captured so the terminal finalize (an UPDATE of the upfront-seeded row) can be
// asserted. One external MCP client with a close() spy lets us assert leases are
// released on the terminal path. lockdown OFF (default) so the detector is the
// active guard.
function makeService() {
// The upfront insert seeds the assistant row; findById/insert stamp a stable
// id so planFinalizeAssistant picks the UPDATE path.
let seq = 0;
const inserted: Array<Record<string, unknown>> = [];
const updated: Array<{
id: string;
workspaceId: string;
patch: Record<string, unknown>;
}> = [];
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const aiChatMessageRepo = {
insert: jest.fn(async (row: Record<string, unknown>) => {
inserted.push(row);
return { id: row.role === 'assistant' ? 'assistant-1' : `user-${++seq}` };
}),
findAllByChat: jest.fn(async () => []),
update: jest.fn(
async (
id: string,
workspaceId: string,
patch: Record<string, unknown>,
) => {
updated.push({ id, workspaceId, patch });
return { id };
},
),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
const mcpClose = jest.fn(async () => undefined);
const mcpClients = {
toolsFor: jest.fn(async () => ({
tools: {},
clients: [{ close: mcpClose }],
outcomes: [],
instructions: [],
})),
};
const streamRegistry = { open: jest.fn(), bind: jest.fn(), abortEntry: jest.fn() };
const svc = new AiChatService(
{} as never,
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo (no open page -> never touched)
aiSettings as never,
tools as never,
mcpClients as never,
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo (no open page)
{} as never, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
// lockdown OFF => the degeneration detector is the anti-babble guard.
isAiChatFinalStepLockdownEnabled: () => false,
isAiChatResumableStreamEnabled: () => false,
} as never,
streamRegistry as never,
);
return { svc, inserted, updated, mcpClose };
}
const body = {
chatId: 'chat-1',
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
};
// Capture the streamText options so the test can drive the SDK callbacks. The
// returned result stub is enough for the post-streamText wiring (consumeStream +
// pipeUIMessageStreamToResponse are no-ops here).
function captureStreamText(): { opts: () => Record<string, any> } {
let captured: Record<string, any> | undefined;
streamTextMock.mockImplementation((options: Record<string, any>) => {
captured = options;
return {
consumeStream: jest.fn(),
pipeUIMessageStreamToResponse: jest.fn(),
};
});
return {
opts: () => {
if (!captured) throw new Error('streamText was not called');
return captured;
},
};
}
async function drive(svc: AiChatService): Promise<void> {
await svc.stream({
user: { id: 'u1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 's1',
body: body as never,
res: makeRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: undefined as never,
});
}
it('degenerate stream: detects → internal abort → onAbort truncates + records OUTPUT_DEGENERATION_ERROR; leases released', async () => {
const { svc, updated, mcpClose } = makeService();
const cap = captureStreamText();
await drive(svc);
const opts = cap.opts();
// The turn's abort signal is the UNION of the socket/run signal and the
// internal degeneration controller — untripped before any output.
expect(opts.abortSignal.aborted).toBe(false);
// Feed a runaway "loadTools.\n" loop the way the SDK streams it: many small
// text-delta chunks. The onChunk throttle only re-checks every ~2000 chars, so
// deliver well past that so the detector's identical-line rule (>=25 lines)
// and the ~2000-char throttle both fire.
const line = 'loadTools.\n';
let delivered = 0;
for (let i = 0; i < 400 && !opts.abortSignal.aborted; i++) {
opts.onChunk({ chunk: { type: 'text-delta', text: line } });
delivered += line.length;
}
// The detector must have tripped and aborted via the INTERNAL controller — the
// reason carries the degeneration message, distinguishing it from a user Stop
// (which aborts with no such reason) or a socket disconnect.
expect(opts.abortSignal.aborted).toBe(true);
expect(delivered).toBeGreaterThan(2000);
expect(String(opts.abortSignal.reason)).toContain(
'Output degeneration detected',
);
// The SDK reacts to the aborted signal by invoking onAbort. `steps` is empty
// (the runaway never finished a step); the in-progress runaway text is what
// gets truncated + persisted.
await opts.onAbort({ steps: [] });
// Terminal finalize = an UPDATE of the upfront-seeded assistant row (assistant
// row was inserted upfront, so planFinalizeAssistant -> UPDATE).
expect(updated).toHaveLength(1);
const patch = updated[0].patch as {
status: string;
content: string;
metadata: Record<string, unknown>;
};
// (4) status 'error' with the degeneration message — NOT 'aborted' and NOT a
// swept 'streaming'. This distinguishes it from a user Stop / server restart.
expect(patch.status).toBe('error');
expect(patch.metadata.error).toBe(OUTPUT_DEGENERATION_ERROR);
expect(patch.metadata.finishReason).toBe('error');
// (3) the runaway tail is TRUNCATED, not the full multi-KB babble: the marker
// is present and the persisted content is far shorter than what was streamed.
expect(patch.content).toContain('output truncated');
expect(patch.content.length).toBeLessThan(delivered);
// Only a few loop reps survive (truncateDegeneratedTail keeps a handful).
expect((patch.content.match(/loadTools\./g) ?? []).length).toBeLessThan(10);
// (5) the leased external MCP client is still released on this terminal path.
expect(mcpClose).toHaveBeenCalledTimes(1);
});
it('degeneration onAbort differs from a NORMAL/user abort (no truncation, no error)', async () => {
// Same harness, but the stream is NOT degenerate: a clean short answer, then a
// user Stop reaches onAbort WITHOUT the degeneration controller having fired.
const { svc, updated, mcpClose } = makeService();
const cap = captureStreamText();
await drive(svc);
const opts = cap.opts();
opts.onChunk({ chunk: { type: 'text-delta', text: 'A normal partial answer.' } });
// The detector never tripped -> the union signal is NOT aborted by us.
expect(opts.abortSignal.aborted).toBe(false);
// A user Stop / disconnect drives onAbort with the partial (clean) text.
await opts.onAbort({ steps: [] });
expect(updated).toHaveLength(1);
const patch = updated[0].patch as {
status: string;
content: string;
metadata: Record<string, unknown>;
};
// A normal abort persists status 'aborted' with NO error and NO truncation
// marker — the branch is genuinely distinguished from the degeneration path.
expect(patch.status).toBe('aborted');
expect('error' in patch.metadata).toBe(false);
expect(patch.content).toBe('A normal partial answer.');
expect(patch.content).not.toContain('output truncated');
// Cleanup still runs on the normal abort path too.
expect(mcpClose).toHaveBeenCalledTimes(1);
});
/**
* Empty-turn marker (#444): onFinish appends STEP_LIMIT_NO_ANSWER_MARKER only
* when the turn burned ALL its steps (steps.length >= MAX_AGENT_STEPS) AND never
* produced any text. The negative: a normal turn ending WITH text is left alone.
*/
it('empty turn (no text + steps exhausted) persists the STEP_LIMIT_NO_ANSWER_MARKER', async () => {
const { svc, updated } = makeService();
const cap = captureStreamText();
await drive(svc);
const opts = cap.opts();
// MAX_AGENT_STEPS text-less steps (only tool calls) => step-exhausted, no text.
const steps = Array.from({ length: MAX_AGENT_STEPS }, () => ({
text: '',
toolCalls: [{ toolCallId: 'c1', toolName: 'searchPages', input: {} }],
toolResults: [
{ toolCallId: 'c1', toolName: 'searchPages', output: { hits: [] } },
],
}));
await opts.onFinish({
text: '',
finishReason: 'tool-calls',
totalUsage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 },
usage: { inputTokens: 1, outputTokens: 1 },
steps,
});
expect(updated).toHaveLength(1);
const patch = updated[0].patch as { status: string; content: string };
expect(patch.status).toBe('completed');
// The synthetic marker is the trailing text of the persisted content.
expect(patch.content).toContain(STEP_LIMIT_NO_ANSWER_MARKER);
});
it('normal turn ending WITH text does NOT get the empty-turn marker', async () => {
const { svc, updated } = makeService();
const cap = captureStreamText();
await drive(svc);
const opts = cap.opts();
// A single step that produced a real answer, well under the step cap.
const steps = [
{ text: 'Here is the finished answer.', toolCalls: [], toolResults: [] },
];
await opts.onFinish({
text: 'Here is the finished answer.',
finishReason: 'stop',
totalUsage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 },
usage: { inputTokens: 1, outputTokens: 1 },
steps,
});
expect(updated).toHaveLength(1);
const patch = updated[0].patch as { status: string; content: string };
expect(patch.status).toBe('completed');
expect(patch.content).toBe('Here is the finished answer.');
expect(patch.content).not.toContain(STEP_LIMIT_NO_ANSWER_MARKER);
});
it('step-exhausted turn that DID produce text keeps the text, no marker (guards the AND)', async () => {
// Exhausting the step budget alone must NOT append the marker when SOME step
// produced text — the marker keys off "no text" too. Drive the real onFinish
// with MAX_AGENT_STEPS steps where the last one carries the answer.
const { svc, updated } = makeService();
const cap = captureStreamText();
await drive(svc);
const opts = cap.opts();
const steps = Array.from({ length: MAX_AGENT_STEPS }, (_, i) => ({
text: i === MAX_AGENT_STEPS - 1 ? 'Final synthesized answer.' : '',
toolCalls:
i === MAX_AGENT_STEPS - 1
? []
: [{ toolCallId: `c${i}`, toolName: 'searchPages', input: {} }],
toolResults:
i === MAX_AGENT_STEPS - 1
? []
: [{ toolCallId: `c${i}`, toolName: 'searchPages', output: {} }],
}));
await opts.onFinish({
text: 'Final synthesized answer.',
finishReason: 'stop',
totalUsage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 },
usage: { inputTokens: 1, outputTokens: 1 },
steps,
});
expect(updated).toHaveLength(1);
const patch = updated[0].patch as { content: string };
expect(patch.content).toContain('Final synthesized answer.');
expect(patch.content).not.toContain(STEP_LIMIT_NO_ANSWER_MARKER);
});
});
+201 -27
View File
@@ -52,11 +52,24 @@ import {
startSseHeartbeat,
stripStreamingHopByHopHeaders,
} from './sse-resilience';
import {
isDegenerateOutput,
truncateDegeneratedTail,
} from './output-degeneration';
// Max agent steps per turn. One step = one model generation; a step that calls
// tools is followed by another step carrying the tool results. Raised from 8 so
// multi-search research questions are not cut off mid-investigation.
const MAX_AGENT_STEPS = 20;
// multi-search research questions are not cut off mid-investigation, then from 20
// to 50 (#444) so read-heavy turns (e.g. dozens of searchInPage sweeps) do not
// exhaust the budget before acting.
const MAX_AGENT_STEPS = 50;
// How many steps before the LAST one the step-budget warning starts firing
// (#444). At MAX-STEP_BUDGET_WARNING_LEAD .. MAX-2 the model is told to stop
// exploring and start acting, with the remaining count decreasing each step; the
// last step (MAX-1) has its own final nudge / lockdown instead (see
// prepareAgentStep).
const STEP_BUDGET_WARNING_LEAD = 6;
// Wall-clock ceiling for building the external MCP toolset during the per-turn
// setup phase (before streamText owns the lifecycle). Defense-in-depth ABOVE the
@@ -82,16 +95,69 @@ const FINAL_STEP_INSTRUCTION =
'language. If the information is incomplete, say so explicitly: summarize ' +
'what you found, what is still missing, and give your best partial conclusion.';
// Pure, unit-testable: decide per-step overrides. Two responsibilities:
// 1. Final-step lockdown (always): on the final allowed step force a text-only
// synthesis answer (toolChoice 'none' + FINAL_STEP_INSTRUCTION). This WINS —
// it takes precedence over the deferred-tool narrowing below.
// 2. Deferred tool visibility (#332): when `deferredEnabled` and NOT the final
// step, expose only the CORE tools + loadTools + whatever loadTools has
// activated so far this turn (`activatedTools`), via `activeTools`. Deferred
// tools stay in the <tool_catalog> until the model loads them.
// When `deferredEnabled` is false the behavior is unchanged: undefined on normal
// steps (all tools active), lockdown on the final step.
// SOFT final-step nudge (#444), used when the final-step lockdown toggle is OFF
// (the new default). Unlike FINAL_STEP_INSTRUCTION it does NOT strip tools
// (toolChoice stays untouched), so the model is never forced into a tool-less
// state mid-work — that tool-stripping is what triggered the 255KB token-loop
// degeneration incident. It only asks the model to finish with a text summary.
const FINAL_STEP_NUDGE =
'This is the LAST step of this turn. Write your final answer to the user now.\n' +
'You may still call tools, but the turn ends after this step either way —\n' +
'prefer finishing with a clear text summary of what was done and what remains.';
// Synthetic marker text appended in onFinish when a step-exhausted turn produced
// NO text at all (#444, mitigates the "empty turn" the lockdown used to prevent
// when the toggle is OFF). Makes the exhausted-without-answer state explicit to
// the user and, on replay, to the model on the next turn.
const STEP_LIMIT_NO_ANSWER_MARKER =
'(Достигнут лимит шагов — итоговый ответ не сформулирован; работа могла ' +
'остаться незавершённой. Напишите «продолжай», чтобы агент продолжил.)';
// Reason recorded in ai_chat_runs.error / the assistant row when the token-
// degeneration detector (#444) aborts a run. Distinct from a user Stop (no error)
// and from a server restart ('streaming' -> swept to 'aborted' with no message).
const OUTPUT_DEGENERATION_ERROR =
'Output degeneration detected (repeated token loop)';
/**
* Compute the step-budget warning text (#444), or '' when this step is outside
* the warning band. The warning fires on steps
* MAX_AGENT_STEPS-STEP_BUDGET_WARNING_LEAD .. MAX_AGENT_STEPS-2 (NOT the last
* step, which has its own final nudge/lockdown), telling the model to stop
* exploring and start acting. `N` is the number of tool-use steps still
* remaining (`MAX_AGENT_STEPS - 1 - stepNumber`), so it decreases toward the
* end. Pure.
*/
export function stepBudgetWarning(stepNumber: number): string {
const isLastStep = stepNumber >= MAX_AGENT_STEPS - 1;
const inBand = stepNumber >= MAX_AGENT_STEPS - STEP_BUDGET_WARNING_LEAD;
if (isLastStep || !inBand) return '';
const remaining = MAX_AGENT_STEPS - 1 - stepNumber;
return (
`Only ${remaining} tool-use steps remain in this turn. Stop exploring and start acting now\n` +
'(make the edits / create the comments / produce results). Leave room to finish\n' +
'with a final text answer.'
);
}
// Pure, unit-testable: decide per-step overrides. Responsibilities:
// 1. Final-step handling. Two modes, chosen by `finalStepLockdownEnabled`:
// - toggle ON (legacy): on the final allowed step force a text-only
// synthesis answer (toolChoice 'none' + FINAL_STEP_INSTRUCTION). This WINS
// — it takes precedence over the deferred-tool narrowing below.
// - toggle OFF (new default, #444): do NOT touch toolChoice — tools stay
// available on every step incl. the last, so the model is never stripped
// of its tools mid-work (the cause of the token-loop degeneration
// incident). A SOFT nudge (FINAL_STEP_NUDGE) is appended to `system`, and
// the deferred-tool `activeTools` narrowing still applies to the last step
// (both `activeTools` and `system` are returned together).
// 2. Step-budget warning (#444): on steps in the warning band (but not the
// last, which has its own nudge/lockdown) append stepBudgetWarning(...) to
// `system` so the model starts acting before it runs out of steps.
// 3. Deferred tool visibility (#332): when `deferredEnabled`, expose only the
// CORE tools + loadTools + whatever loadTools has activated so far this turn
// (`activatedTools`), via `activeTools`. Deferred tools stay in the
// <tool_catalog> until the model loads them.
//
// `system` is the in-scope system prompt; we CONCATENATE so the original
// persona/context is preserved — a bare `system` override would REPLACE the
@@ -107,31 +173,53 @@ export function prepareAgentStep(
system: string,
activatedTools: ReadonlySet<string> | readonly string[] = [],
deferredEnabled = false,
finalStepLockdownEnabled = false,
):
| { toolChoice: 'none'; system: string }
| { activeTools: string[] }
| { activeTools: string[]; system?: string }
| { system: string }
| undefined {
// Final-step lockdown WINS (applies regardless of the deferred toggle).
if (stepNumber >= MAX_AGENT_STEPS - 1) {
const isLastStep = stepNumber >= MAX_AGENT_STEPS - 1;
// Legacy final-step lockdown (toggle ON): text-only synthesis. WINS over the
// deferred narrowing AND drops tools for this step.
if (isLastStep && finalStepLockdownEnabled) {
return {
toolChoice: 'none',
system: `${system}\n\n${FINAL_STEP_INSTRUCTION}`,
};
}
// Deferred tool loading: narrow this step's visible tools to CORE + loadTools
// + the tools already activated this turn.
// Compute the extra system text for this step: the soft final nudge on the last
// step (toggle OFF), or the step-budget warning in the warning band. At most one
// of these applies (stepBudgetWarning returns '' on the last step).
const extra = isLastStep ? FINAL_STEP_NUDGE : stepBudgetWarning(stepNumber);
const systemForStep = extra ? `${system}\n\n${extra}` : undefined;
// Deferred tool loading: narrow this step's visible tools to CORE + loadTools +
// the tools already activated this turn. Applies on EVERY step incl. the last
// (toggle OFF), so the model keeps its core tools available while being nudged
// to finish. Return `system` alongside `activeTools` when we have extra text.
if (deferredEnabled) {
const activated = Array.isArray(activatedTools)
? activatedTools
: [...activatedTools];
return {
activeTools: [...CORE_TOOL_KEYS, LOAD_TOOLS_NAME, ...activated],
};
const activeTools = [...CORE_TOOL_KEYS, LOAD_TOOLS_NAME, ...activated];
return systemForStep ? { activeTools, system: systemForStep } : { activeTools };
}
return undefined;
// Deferred OFF: all tools stay active; only append the extra system text (if any).
return systemForStep ? { system: systemForStep } : undefined;
}
export { MAX_AGENT_STEPS, FINAL_STEP_INSTRUCTION };
export {
MAX_AGENT_STEPS,
STEP_BUDGET_WARNING_LEAD,
FINAL_STEP_INSTRUCTION,
FINAL_STEP_NUDGE,
STEP_LIMIT_NO_ANSWER_MARKER,
OUTPUT_DEGENERATION_ERROR,
};
// Pure, unit-testable post-processing for a model-generated title (#199): trim
// whitespace, strip a single pair of surrounding quotes the model often adds,
@@ -890,6 +978,12 @@ export class AiChatService implements OnModuleInit {
// tools (fat/rare in-app tools + ALL external MCP tools) load on demand. When
// OFF, every tool is active and nothing below changes.
const deferredEnabled = this.environment.isAiChatDeferredToolsEnabled();
// Final-step lockdown toggle (#444). Default OFF: the last step keeps its
// tools and gets only a soft nudge (prepareAgentStep), and the token-
// degeneration detector (onChunk below) is the anti-babble guard. ON =
// legacy tool-stripping lockdown on the last step.
const finalStepLockdownEnabled =
this.environment.isAiChatFinalStepLockdownEnabled();
let system: string;
let docmostTools: Awaited<ReturnType<AiChatToolsService['forUser']>>;
@@ -978,6 +1072,16 @@ export class AiChatService implements OnModuleInit {
const capturedSteps: StepLike[] = [];
let inProgressText = '';
// Token-degeneration guard (#444). When the final-step lockdown is OFF, a
// runaway repetition loop (the 255KB "loadTools." incident) is aborted via
// this internal controller, unioned with the run/socket signal below. The
// detector runs on `inProgressText` in onChunk, throttled by growth so the
// pure rules only fire every ~DEGENERATION_CHECK_STEP bytes.
const degenerationController = new AbortController();
let degenerationDetected = false;
let lastDegenerationCheckLen = 0;
const DEGENERATION_CHECK_STEP = 2000;
// Step-granular durability (#183): create the assistant row UPFRONT in the
// 'streaming' state (before any token), then UPDATE it as each step finishes
// and finalize it once on the terminal callback. If the process dies
@@ -1118,11 +1222,21 @@ export class AiChatService implements OnModuleInit {
// further tool calls and appends a synthesis instruction on that step,
// concatenated onto the original `system` so the persona is preserved.
prepareStep: ({ stepNumber }) =>
prepareAgentStep(stepNumber, system, activatedTools, deferredEnabled),
prepareAgentStep(
stepNumber,
system,
activatedTools,
deferredEnabled,
finalStepLockdownEnabled,
),
// #184: the RUN's signal (explicit-stop) when a run wraps this turn, else
// the socket-bound signal (legacy). A browser disconnect aborts only in
// the legacy path.
abortSignal: effectiveSignal,
// the legacy path. #444: UNION it with the internal degeneration signal
// so a detected token-loop aborts the run too (AbortSignal.any — Node 20.3+).
abortSignal: AbortSignal.any([
effectiveSignal,
degenerationController.signal,
]),
onChunk: ({ chunk }) => {
// DIAGNOSTIC (Safari stream-drop investigation) — temporary. Any model
// output chunk means the stream is actively emitting bytes; track first
@@ -1132,7 +1246,29 @@ export class AiChatService implements OnModuleInit {
lastModelChunkAt = now;
// 'text-delta' is the assistant's prose; tool-call args are separate chunk
// types — so this mirrors exactly what streams to the client.
if (chunk.type === 'text-delta') inProgressText += chunk.text;
if (chunk.type === 'text-delta') {
inProgressText += chunk.text;
// Token-degeneration guard (#444). Throttled: only re-run the pure
// rules once the text has grown ~DEGENERATION_CHECK_STEP bytes since
// the last check, so the tail heuristics cost is amortized. On a
// trigger, abort the run ONCE with a distinguishable reason.
if (
!degenerationDetected &&
inProgressText.length - lastDegenerationCheckLen >=
DEGENERATION_CHECK_STEP
) {
lastDegenerationCheckLen = inProgressText.length;
if (isDegenerateOutput(inProgressText)) {
degenerationDetected = true;
this.logger.warn(
`AI chat stream aborted (chat ${chatId}): ${OUTPUT_DEGENERATION_ERROR}`,
);
degenerationController.abort(
new Error(OUTPUT_DEGENERATION_ERROR),
);
}
}
}
},
onStepFinish: (step) => {
// The finished step's full text is now in `step.text`; fold it in and reset
@@ -1174,8 +1310,22 @@ export class AiChatService implements OnModuleInit {
// plain-text projection (full-text search / fallback). A multi-step
// turn's `content` therefore now holds all steps' prose, not just the
// last block.
// Empty-turn mitigation (#444, toggle OFF). If the turn burned all its
// steps WITHOUT ever producing text (every step's text is empty) and
// the model stopped because it hit the step cap, there is no answer to
// show — the lockdown used to force one. Append a synthetic marker as
// the trailing text so the exhausted-without-answer state is explicit
// to the user and, on replay, to the model next turn. `flushAssistant`
// takes this as the `inProgressText` trailing text arg (empty here
// otherwise). `stepCountIs(MAX_AGENT_STEPS)` surfaces as
// finishReason === 'tool-calls' (or a length/other cap), so we key off
// "no text produced" rather than a single finishReason string.
const producedText = (steps as StepLike[]).some((s) => s.text?.trim());
const stepExhausted = steps.length >= MAX_AGENT_STEPS;
const emptyTurnMarker =
!producedText && stepExhausted ? STEP_LIMIT_NO_ANSWER_MARKER : '';
await finalizeAssistant(
flushAssistant(steps as StepLike[], '', 'completed', {
flushAssistant(steps as StepLike[], emptyTurnMarker, 'completed', {
finishReason: finishReason as string,
usage: totalUsage as StreamUsage,
contextTokens:
@@ -1252,6 +1402,30 @@ export class AiChatService implements OnModuleInit {
await snapshotTurnEnd();
},
onAbort: async ({ steps }) => {
// #444: distinguish a degeneration abort (our internal controller) from
// a user Stop / disconnect. On degeneration we truncate the runaway tail
// before persist (so hundreds of KB of garbage never reach the DB /
// replay) and record it as an ERROR with a clear, distinguishable reason
// — NOT a bare 'aborted' (a user Stop) and NOT a swept 'streaming' (a
// server restart).
if (degenerationDetected) {
const truncated = truncateDegeneratedTail(inProgressText);
await finalizeAssistant(
flushAssistant(capturedSteps, truncated, 'error', {
error: OUTPUT_DEGENERATION_ERROR,
pageChanged,
}),
);
if (runId)
await runHooks?.onSettled?.(
runId,
'error',
OUTPUT_DEGENERATION_ERROR,
);
await closeExternalClients();
await snapshotTurnEnd();
return;
}
const partialChars =
capturedSteps.reduce((n, s) => n + (s.text?.length ?? 0), 0) +
inProgressText.length;
@@ -75,7 +75,7 @@ const LABELS: Record<
searchPages: 'Searched pages',
getPage: 'Read page',
createPage: 'Created page',
updatePageContent: 'Updated page',
updatePageMarkdown: 'Updated page',
renamePage: 'Renamed page',
movePage: 'Moved page',
deletePage: 'Deleted page (to trash)',
@@ -96,7 +96,7 @@ const LABELS: Record<
searchPages: 'Искал по страницам',
getPage: 'Прочитал страницу',
createPage: 'Создал страницу',
updatePageContent: 'Обновил страницу',
updatePageMarkdown: 'Обновил страницу',
renamePage: 'Переименовал страницу',
movePage: 'Переместил страницу',
deletePage: 'Удалил страницу (в корзину)',
@@ -0,0 +1,182 @@
import {
hasRepeatedLineRun,
hasPeriodicTail,
isDegenerateOutput,
truncateDegeneratedTail,
REPEATED_LINES_THRESHOLD,
MIN_PERIOD_REPEATS,
} from './output-degeneration';
/**
* Unit tests for the token-degeneration detector (#444) the sole anti-babble
* guard once the final-step lockdown is OFF. The two rules must fire on real
* degeneration (the "loadTools." incident, a no-newline repeat) and MUST NOT fire
* on legitimate long output (edit lists, tables, code).
*/
describe('hasRepeatedLineRun (rule 1: identical-line run)', () => {
it('POSITIVE: fires on "loadTools.\\n" repeated many times (the incident)', () => {
const text = 'Here is my plan.\n' + 'loadTools.\n'.repeat(300);
expect(hasRepeatedLineRun(text)).toBe(true);
expect(isDegenerateOutput(text)).toBe(true);
});
it('POSITIVE: fires at exactly the threshold', () => {
const text = 'x\n'.repeat(REPEATED_LINES_THRESHOLD);
expect(hasRepeatedLineRun(text)).toBe(true);
});
it('NEGATIVE: does NOT fire just below the threshold', () => {
// threshold-1 identical lines followed by a distinct line.
const text = 'x\n'.repeat(REPEATED_LINES_THRESHOLD - 1) + 'done\n';
expect(hasRepeatedLineRun(text)).toBe(false);
});
it('NEGATIVE: a long edit list of DISTINCT lines never trips', () => {
const lines: string[] = [];
for (let i = 0; i < 200; i++) lines.push(`- edited section ${i}: fixed typo`);
const text = lines.join('\n');
expect(hasRepeatedLineRun(text)).toBe(false);
expect(isDegenerateOutput(text)).toBe(false);
});
it('NEGATIVE: a markdown table with blank separators does not trip', () => {
// Repeated identical rows are unusual, but blank lines break any run.
const block = ['| a | b |', '| - | - |', '', '| a | b |', ''];
const text = Array.from({ length: 60 }, () => block.join('\n')).join('\n');
expect(hasRepeatedLineRun(text)).toBe(false);
});
it('NEGATIVE: blank lines do NOT count toward a run', () => {
const text = '\n'.repeat(100);
expect(hasRepeatedLineRun(text)).toBe(false);
});
});
describe('hasPeriodicTail (rule 2: no-newline suffix periodicity)', () => {
it('POSITIVE: fires on a single char repeated with no newlines', () => {
const text = 'answer: ' + 'a'.repeat(500);
expect(hasPeriodicTail(text)).toBe(true);
expect(isDegenerateOutput(text)).toBe(true);
});
it('POSITIVE: fires on a multi-char block repeat with no newlines', () => {
const text = 'prefix ' + 'abcdef'.repeat(100);
expect(hasPeriodicTail(text)).toBe(true);
});
it('POSITIVE: at least MIN_PERIOD_REPEATS repeats of a small block', () => {
const text = 'go'.repeat(MIN_PERIOD_REPEATS);
expect(hasPeriodicTail(text)).toBe(true);
});
it('NEGATIVE: prose does not look periodic', () => {
const text =
'The quick brown fox jumps over the lazy dog while the sun sets slowly ' +
'behind the distant mountains and the river winds through the valley below.';
expect(hasPeriodicTail(text)).toBe(false);
expect(isDegenerateOutput(text)).toBe(false);
});
it('NEGATIVE: a long code block is not flagged', () => {
const code = `
function compute(values) {
let total = 0;
for (const v of values) {
total += v * 2;
}
return total / values.length;
}
export const helper = (x) => x + 1;
const config = { retries: 3, timeout: 5000, backoff: 'exp' };
`.repeat(3);
expect(isDegenerateOutput(code)).toBe(false);
});
it('NEGATIVE: a short string well under the repeat count is safe', () => {
expect(hasPeriodicTail('ababab')).toBe(false);
});
// Regression (#444): a trivial single-char period (p===1) must NOT flag
// legitimate divider/underline/whitespace runs. These are common in real
// model output and previously false-positived at ~20 identical chars, aborting
// the run and truncating output. They must all be treated as clean.
it('NEGATIVE: a markdown horizontal rule is not flagged', () => {
const text = 'text\n' + '-'.repeat(40);
expect(hasPeriodicTail(text)).toBe(false);
expect(isDegenerateOutput(text)).toBe(false);
});
it('NEGATIVE: a setext heading underline is not flagged', () => {
const text = 'Title\n' + '='.repeat(30);
expect(hasPeriodicTail(text)).toBe(false);
expect(isDegenerateOutput(text)).toBe(false);
});
it('NEGATIVE: a box-drawing divider with no trailing newline is not flagged', () => {
const text = 'done ' + '─'.repeat(50);
expect(hasPeriodicTail(text)).toBe(false);
expect(isDegenerateOutput(text)).toBe(false);
});
it('NEGATIVE: trailing spaces are not flagged', () => {
const text = 'answer' + ' '.repeat(40);
expect(hasPeriodicTail(text)).toBe(false);
expect(isDegenerateOutput(text)).toBe(false);
});
// TRIVIAL_MIN_REPEATS boundary (#444 review). The monochar-tail branch fires at
// EXACTLY 60 identical trailing chars (`run >= TRIVIAL_MIN_REPEATS`), so 59 is
// clean and 60 trips. These pin the `>=` and MUST fail if the comparison is
// flipped to `>` (the surviving mutation). The value 60 is HARD-CODED here on
// purpose: TRIVIAL_MIN_REPEATS is a private constant and the assert must lock
// the literal boundary the reviewer named, not track a constant edit.
it('NEGATIVE: 59 identical trailing chars is one below the monochar threshold', () => {
expect(hasPeriodicTail('x'.repeat(59))).toBe(false);
expect(isDegenerateOutput('x'.repeat(59))).toBe(false);
});
it('POSITIVE: 60 identical trailing chars hits the monochar threshold exactly', () => {
// Fails if `run >= TRIVIAL_MIN_REPEATS` is mutated to `run > …`.
expect(hasPeriodicTail('x'.repeat(60))).toBe(true);
expect(isDegenerateOutput('x'.repeat(60))).toBe(true);
});
// Positive counterparts: a GENUINE single-char runaway (hundreds+ of repeats)
// and the real incident (period>=2, "loadTools." ×N) must still fire.
it('POSITIVE: a genuine single-char runaway is still flagged', () => {
const text = 'x'.repeat(5000);
expect(hasPeriodicTail(text)).toBe(true);
expect(isDegenerateOutput(text)).toBe(true);
});
it('POSITIVE: the "loadTools." incident (period>=2) is still flagged', () => {
const text = 'loadTools.'.repeat(500);
expect(hasPeriodicTail(text)).toBe(true);
expect(isDegenerateOutput(text)).toBe(true);
});
});
describe('truncateDegeneratedTail', () => {
it('collapses a repeated-line loop to a few reps + marker', () => {
const text = 'plan\n' + 'loadTools.\n'.repeat(20000);
const out = truncateDegeneratedTail(text);
expect(out.length).toBeLessThan(text.length);
expect(out).toContain('output truncated');
// Keeps the leading context and a few loop reps.
expect(out).toContain('plan');
expect((out.match(/loadTools\./g) ?? []).length).toBeLessThan(10);
});
it('collapses a no-newline periodic loop to a few blocks + marker', () => {
const text = 'answer: ' + 'xy'.repeat(50000);
const out = truncateDegeneratedTail(text);
expect(out.length).toBeLessThan(text.length);
expect(out).toContain('output truncated');
expect(out).toContain('answer:');
});
it('returns non-degenerate text unchanged (by identity)', () => {
const text = 'A perfectly normal, finished assistant answer.';
expect(truncateDegeneratedTail(text)).toBe(text);
});
});
@@ -0,0 +1,189 @@
/**
* Token-degeneration detector for the in-app agent stream (#444).
*
* When the final-step lockdown is OFF (the new default) there is no toolChoice
* override to strip the model's tools mid-work, so the anti-babble safety net is
* this detector. It watches the accumulating assistant text and, on a runaway
* repetition loop (the 255KB "loadTools." incident), aborts the run.
*
* Both rules are PURE functions of the text tail so they are cheap to run every
* few KB and are unit-testable in isolation. They operate on the TAIL only
* (`TAIL_WINDOW` chars) so the cost is bounded regardless of how long the turn is.
*/
/** How many trailing chars of the accumulated text the rules inspect. */
export const TAIL_WINDOW = 3000;
/** Rule 1 threshold: minimum consecutive identical non-empty lines to trigger. */
export const REPEATED_LINES_THRESHOLD = 25;
/** Rule 2: maximum length of a repeating block considered for periodicity. */
export const MAX_PERIOD_LEN = 150;
/** Rule 2: minimum number of consecutive block repeats to trigger. */
export const MIN_PERIOD_REPEATS = 20;
/**
* Rule 1 `REPEATED_LINES_THRESHOLD` consecutive IDENTICAL non-empty lines at
* the tail. Catches the classic newline-delimited loop ("loadTools.\n" ×N).
* Blank lines break a run (a table / list with blank separators never trips it);
* a run of ordinary distinct lines (an edit list, code) never reaches the count.
*
* NB: `REPEATED_LINES_THRESHOLD` (25) is only THIS rule's own trigger, not the
* effective floor for detecting a repeated-line loop. In practice a newline-
* delimited repeat also has a fixed period (line + '\n'), so rule 2 catches it
* via periodicity at `MIN_PERIOD_REPEATS` (20) repeats the two rules combine
* (see `isDegenerateOutput`), so the effective lower bound for a short identical
* line loop is ~20, not 25. Pure.
*/
export function hasRepeatedLineRun(
text: string,
threshold = REPEATED_LINES_THRESHOLD,
): boolean {
const tail = text.length > TAIL_WINDOW ? text.slice(-TAIL_WINDOW) : text;
const lines = tail.split('\n');
let run = 1;
let prev: string | null = null;
for (const line of lines) {
if (line.length > 0 && line === prev) {
run += 1;
if (run >= threshold) return true;
} else {
run = 1;
}
prev = line;
}
return false;
}
/**
* Rule 2 cheap suffix-periodicity check: the tail ends in
* `MIN_PERIOD_REPEATS` back-to-back repeats of a single block of length
* `MAX_PERIOD_LEN`. Catches a no-newline repeat ("abcabcabc…") the line rule
* misses. For each candidate period length p we verify the last `repeats*p`
* chars are p-periodic; we stop at the smallest p that satisfies the repeat
* count. Bounded by MAX_PERIOD_LEN × TAIL_WINDOW comparisons negligible. Pure.
*/
export function hasPeriodicTail(
text: string,
maxPeriod = MAX_PERIOD_LEN,
minRepeats = MIN_PERIOD_REPEATS,
): boolean {
const tail = text.length > TAIL_WINDOW ? text.slice(-TAIL_WINDOW) : text;
const n = tail.length;
// Not even the shortest possible loop fits in the tail.
if (n < minRepeats) return false;
// A tail of ONE repeated char (a "trivial period") is common in LEGIT output —
// markdown rules (----/====), setext underlines, box-drawing dividers,
// trailing spaces routinely produce 20–50 identical chars. Such a run is
// p-periodic for EVERY p, so it would otherwise trip the block rule at p>=2
// too, not just p===1. We therefore split the check: a monochar tail needs far
// more repeats (a real single-char babble loop produces hundreds-to-thousands;
// 60 is well above any realistic divider yet a fifth of TAIL_WINDOW), while a
// genuine multi-char block repeat (>=2 distinct chars, e.g. the "loadTools."
// incident, period ~10) keeps the normal MIN_PERIOD_REPEATS threshold.
const TRIVIAL_MIN_REPEATS = 60;
// Monochar-tail check (the trivial-period case): count the trailing run of one
// identical char and require TRIVIAL_MIN_REPEATS of them.
{
const last = tail[n - 1];
let run = 1;
for (let i = n - 2; i >= 0 && tail[i] === last; i--) run++;
if (run >= TRIVIAL_MIN_REPEATS) return true;
}
const maxP = maxPeriod;
for (let p = 2; p <= maxP; p++) {
// Verify the last (minRepeats*p) chars are p-periodic AND not monochar (a
// monochar span is the trivial case handled above, so skip it here to avoid
// re-flagging a legit divider at a composite period).
const span = minRepeats * p;
// Not enough tail to hold this many repeats of this period.
if (span > n) continue;
const start = n - span;
let periodic = true;
let multiChar = false;
for (let i = n - 1; i >= start + p; i--) {
if (tail[i] !== tail[i - p]) {
periodic = false;
break;
}
}
if (!periodic) continue;
// Confirm the block itself has >=2 distinct chars (else it's monochar).
for (let i = start + 1; i < n; i++) {
if (tail[i] !== tail[start]) {
multiChar = true;
break;
}
}
if (multiChar) return true;
}
return false;
}
/**
* Combined guard used by the stream's onChunk: true when EITHER rule fires.
* Pure the caller owns the abort side effect.
*/
export function isDegenerateOutput(text: string): boolean {
return hasRepeatedLineRun(text) || hasPeriodicTail(text);
}
/**
* Truncate a degenerated tail before persist so hundreds of KB of garbage never
* reach the DB / replay (#444). Keeps everything up to and including the FIRST
* `keepRepeats` repeats of the detected loop, then appends a short marker. If no
* loop is detected the text is returned unchanged (by identity).
*
* Implementation: find the shortest tail period (same check as hasPeriodicTail),
* keep the prefix before the loop plus `keepRepeats` copies of the block, drop
* the rest. This is best-effort cosmetic trimming; correctness does not depend on
* finding the exact minimal loop. Pure.
*/
export function truncateDegeneratedTail(
text: string,
keepRepeats = 3,
): string {
const marker = '\n…[output truncated: repeated token loop detected]';
// Try the line rule first: collapse a long run of identical lines.
const lines = text.split('\n');
let runStart = -1;
let run = 1;
for (let i = 1; i < lines.length; i++) {
if (lines[i].length > 0 && lines[i] === lines[i - 1]) {
if (run === 1) runStart = i - 1;
run += 1;
if (run >= REPEATED_LINES_THRESHOLD) {
const kept = lines.slice(0, runStart + keepRepeats).join('\n');
return kept + marker;
}
} else {
run = 1;
runStart = -1;
}
}
// Fall back to periodicity over the whole string (bounded by the same block
// length). Find the smallest period that makes the SUFFIX highly repetitive.
const n = text.length;
const maxP = Math.min(MAX_PERIOD_LEN, Math.floor(n / MIN_PERIOD_REPEATS));
for (let p = 1; p <= maxP; p++) {
// Count how many trailing p-blocks are periodic.
let reps = 1;
let i = n - 1;
for (; i >= p; i--) {
if (text[i] !== text[i - p]) break;
}
// The loop above walks over the periodic suffix; its length is (n-1 - i).
const periodicLen = n - 1 - i;
reps = Math.floor(periodicLen / p) + 1;
if (reps >= MIN_PERIOD_REPEATS) {
const loopStart = n - reps * p; // start of the fully-periodic suffix
const kept = text.slice(0, loopStart + keepRepeats * p);
return kept + marker;
}
}
return text;
}
@@ -1,6 +1,17 @@
import { AiChatToolsService } from './ai-chat-tools.service';
import * as loader from './docmost-client.loader';
import type { DocmostClientLike } from './docmost-client.loader';
// Test-double type for the loopback client. `DocmostClientLike` is now DERIVED
// from the real `DocmostClient` (issue #446), so its method RETURN types are the
// concrete client shapes. These stubs deliberately return minimal recording
// shapes (e.g. `{ ok: true }`), which no longer satisfy those concrete returns —
// so the doubles are typed with the same method NAMES but loose async returns.
// Each is still cast to `DocmostClientLike` at the (return-erased) mock site, so
// the positional-call type-safety on the PRODUCTION client is unaffected.
type FakeDocmostClient = Partial<
Record<keyof DocmostClientLike, (...args: any[]) => Promise<any>>
>;
// The real zod-agnostic shared tool-spec registry. It has no runtime deps, so
// importing the TS source directly keeps these mocks honest: the service builds
// the shared tools from exactly the specs the package ships, not a hand-stub.
@@ -12,7 +23,7 @@ import { SHARED_TOOL_SPECS } from '../../../../../../packages/mcp/src/tool-specs
// sync.
const mockLoaded = (DocmostClient: loader.DocmostClientCtor) => ({
DocmostClient,
sharedToolSpecs: SHARED_TOOL_SPECS as Record<string, loader.SharedToolSpec>,
sharedToolSpecs: SHARED_TOOL_SPECS as unknown as Record<string, loader.SharedToolSpec>,
});
/**
@@ -31,7 +42,7 @@ describe('AiChatToolsService deletePage guardrail (H4)', () => {
// Minimal fake DocmostClient: only the write methods the tools touch need to
// exist; deletePage records its args. No network, no ESM import.
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
deletePage: (...args: unknown[]) => {
deletePageCalls.push(args);
return Promise.resolve({ success: true });
@@ -160,7 +171,7 @@ describe('AiChatToolsService deletePage guardrail (H4)', () => {
describe('AiChatToolsService expanded toolset guardrails', () => {
// No client method is invoked here — every assertion is on tool presence /
// input schema — so an empty fake client is sufficient.
const fakeClient: Partial<DocmostClientLike> = {};
const fakeClient: FakeDocmostClient = {};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
@@ -264,8 +275,9 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
const patchNodeCalls: unknown[][] = [];
const insertNodeCalls: unknown[][] = [];
const updatePageJsonCalls: unknown[][] = [];
const updatePageCalls: unknown[][] = [];
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
patchNode: (...args: unknown[]) => {
patchNodeCalls.push(args);
return Promise.resolve({ ok: true });
@@ -278,6 +290,11 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
updatePageJsonCalls.push(args);
return Promise.resolve({ ok: true });
},
// Backs the plain-Markdown full-body-replace tool updatePageMarkdown (#411).
updatePage: (...args: unknown[]) => {
updatePageCalls.push(args);
return Promise.resolve({ success: true });
},
};
const tokenServiceStub = {
@@ -291,6 +308,7 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
patchNodeCalls.length = 0;
insertNodeCalls.length = 0;
updatePageJsonCalls.length = 0;
updatePageCalls.length = 0;
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue(
mockLoaded(function () {
return fakeClient as DocmostClientLike;
@@ -426,6 +444,54 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
).rejects.toThrow('content was a string but not valid JSON');
expect(updatePageJsonCalls).toHaveLength(0);
});
// #411: the plain-Markdown full-body-replace tool is now the shared
// `updatePageMarkdown` (was inline `updatePageContent`). It forwards to
// client.updatePage(pageId, content, title) -> updatePageContentRealtime ->
// markdownToProseMirrorCanonical, so `^[...]` footnotes materialize.
it('updatePageMarkdown forwards { pageId, content, title } to client.updatePage', async () => {
const tools = await buildTools();
await tools.updatePageMarkdown.execute(
{ pageId: 'p1', content: 'Body^[a note]', title: 'New title' } as never,
{} as never,
);
expect(updatePageCalls).toHaveLength(1);
expect(updatePageCalls[0]).toEqual(['p1', 'Body^[a note]', 'New title']);
});
it('updatePageMarkdown returns the RAW client result in-app (deliberate #411 shape change, documented on the spec)', async () => {
const tools = await buildTools();
// Registry canonical execute returns client.updatePage's result verbatim.
// The old inline tool projected to { pageId, updated }; the rename now
// surfaces the raw result (nothing reads the removed `.updated`; the raw
// shape carries footnote/verify warnings and matches the on-both-hosts
// registry convention). fakeClient.updatePage resolves { success: true }.
const result = await tools.updatePageMarkdown.execute(
{ pageId: 'p1', content: '# Hi' } as never,
{} as never,
);
expect(result).toEqual({ success: true });
});
it('updatePageMarkdown forwards title=undefined when omitted', async () => {
const tools = await buildTools();
await tools.updatePageMarkdown.execute(
{ pageId: 'p1', content: '# Hi' } as never,
{} as never,
);
expect(updatePageCalls[0]).toEqual(['p1', '# Hi', undefined]);
});
// #411 surface split: the plain-Markdown replace tool exists in-app under the
// new key; the OLD inline updatePageContent key is gone; importPageMarkdown is
// still present IN-APP (only the external MCP surface drops it — asserted in
// packages/mcp/test/unit/tool-inventory.test.mjs).
it('exposes updatePageMarkdown in-app, no legacy updatePageContent, keeps importPageMarkdown', async () => {
const tools = await buildTools();
expect(tools.updatePageMarkdown).toBeDefined();
expect((tools as Record<string, unknown>).updatePageContent).toBeUndefined();
expect(tools.importPageMarkdown).toBeDefined();
});
});
/**
@@ -439,7 +505,7 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
* getOutline) are exercised here end-to-end through forUser().
*/
describe('AiChatToolsService model-friendly input validation (#190)', () => {
const fakeClient: Partial<DocmostClientLike> = {};
const fakeClient: FakeDocmostClient = {};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
generateCollabToken: jest.fn().mockResolvedValue('collab-token'),
@@ -557,7 +623,7 @@ describe('AiChatToolsService #294 changed execute wirings', () => {
tableDeleteRow: [],
tableUpdateCell: [],
};
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
movePage: (...args: unknown[]) => {
calls.movePage.push(args);
return Promise.resolve({ success: true });
@@ -666,7 +732,7 @@ describe('AiChatToolsService #410 footnote + image tools', () => {
insertImage: [],
replaceImage: [],
};
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
insertFootnote: (...args: unknown[]) => {
calls.insertFootnote.push(args);
return Promise.resolve({ success: true, footnoteId: 'fn1', reused: false });
@@ -26,6 +26,100 @@ import {
type ToolCatalogEntry,
} from './tool-tiers';
/**
* Compile-time contract (issue #446): the in-app tool `execute` closures below
* call the loopback `DocmostClient` POSITIONALLY (e.g.
* `client.drawioGet(pageId, node, format ?? 'xml')`). Those closures receive an
* AI-SDK-erased (`any`) input, so a positional call inside them is NOT checked
* against the real signature a parameter reorder/type-change in
* `packages/mcp/src/client.ts` would otherwise reach production as a runtime
* "wrong argument" tool failure with zero compile signal (the restored #294
* debt). This never-called function reproduces every positional call with
* correctly-typed placeholder arguments against the DERIVED `DocmostClientLike`
* (a `Pick` of the real `DocmostClient`), so any such reorder/rename becomes a
* SERVER COMPILE ERROR here. It emits nothing (types only) and is never invoked;
* keep each call in lockstep with the matching `execute` body below.
*/
// eslint-disable-next-line @typescript-eslint/no-unused-vars
function __assertClientCallContract(client: DocmostClientLike): void {
// Placeholders standing in for the AI-SDK-erased execute inputs. Their types
// are deliberately concrete so the positional calls are checked end-to-end.
const s = '' as string;
const n = 0 as number;
const node: unknown = null;
const edits: Array<{ find: string; replace: string; replaceAll?: boolean }> =
[];
const cells: string[] = [];
const align = undefined as 'left' | 'center' | 'right' | undefined;
// --- read ---
void client.search(s, undefined, n);
void client.getPage(s);
void client.getPageRaw(s);
void client.getWorkspace();
void client.getSpaces();
void client.listPages(s, n, true);
void client.listSidebarPages(s, s);
void client.getOutline(s);
void client.getPageJson(s);
void client.getNode(s, s);
void client.searchInPage(s, s, {
regex: true,
caseSensitive: true,
limit: n,
});
void client.getTable(s, s);
void client.listComments(s, true);
void client.getComment(s);
void client.checkNewComments(s, s, s);
void client.listShares();
void client.listPageHistory(s, s);
void client.getPageHistory(s);
void client.diffPageVersions(s, s, s);
void client.exportPageMarkdown(s);
// --- write (page) ---
void client.createPage(s, s, s, s);
void client.updatePage(s, s, s);
void client.renamePage(s, s);
void client.movePage(s, s, s);
void client.deletePage(s);
void client.editPageText(s, edits);
void client.patchNode(s, s, node);
void client.insertNode(s, node, {
position: 'append',
anchorNodeId: s,
anchorText: s,
});
void client.deleteNode(s, s);
void client.updatePageJson(s, node, s);
void client.tableInsertRow(s, s, cells, n);
void client.tableDeleteRow(s, s, n);
void client.tableUpdateCell(s, s, n, n, s);
void client.copyPageContent(s, s);
void client.importPageMarkdown(s, s);
void client.sharePage(s, true);
void client.unsharePage(s);
void client.restorePageVersion(s);
void client.transformPage(s, s, { dryRun: true });
void client.stashPage(s);
// --- write (image / footnote), in-app since #410 ---
void client.insertFootnote(s, s, s);
void client.insertImage(s, s, {
align,
alt: s,
replaceText: s,
afterText: s,
});
void client.replaceImage(s, s, s, { align, alt: s });
// --- draw.io diagrams (#423) ---
void client.drawioGet(s, s, 'xml');
void client.drawioCreate(s, { position: 'append', anchorNodeId: s }, s, s);
void client.drawioUpdate(s, s, s, s);
// --- write (comment) ---
void client.createComment(s, s, 'inline', s, s, s);
void client.resolveComment(s, true);
}
/**
* Per-user, per-request adapter that exposes Docmost READ operations to the
* agent as AI SDK tools (STAGE A = read only).
@@ -198,6 +292,17 @@ export class AiChatToolsService {
execute,
});
// The in-app toolset. It starts with the tools kept INLINE here for a
// documented per-layer reason: an intentional behaviour/schema divergence from
// the standalone MCP surface (searchPages' hybrid RRF,
// transformPage's guardrailed shorter schema), a
// snake_case/camelCase naming clash the shared registry forbids (getTable vs
// the MCP `table_get`), per-request state the registry loop cannot provide
// (getCurrentPage reads the resolved openedPage; searchPages closes over the
// per-request user/embedding deps), or a tool with no MCP twin
// (listSidebarPages/getComment/getPageHistory). Every SHARED tool is then added
// by the registry loop below (see it), so there is exactly one arg-mapping per
// shared tool and it can never drift from the MCP host again (#445).
const tools: Record<string, Tool> = {
// INTENTIONAL per-transport divergence (not in the shared registry): this
// in-app search runs a semantic + keyword hybrid (RRF) with in-process
@@ -332,180 +437,14 @@ export class AiChatToolsService {
execute: async () => resolveCurrentPageResult(openedPage),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The execute body keeps this layer's { title, markdown } projection.
getPage: sharedTool(sharedToolSpecs.getPage, async ({ pageId }) => {
// getPage(pageId) -> { data: filterPage(page, markdown), success }.
const result = await client.getPage(pageId);
const data = (result?.data ?? {}) as {
title?: string;
content?: string;
};
return {
title: data.title ?? '',
markdown: typeof data.content === 'string' ? data.content : '',
};
}),
// --- WRITE tools (all reversible — history/trash; §6.5 / D3) ---
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
createPage: sharedTool(
sharedToolSpecs.createPage,
async ({ title, content, spaceId, parentPageId }) => {
// createPage(title, content, spaceId, parentPageId?) ->
// { data: filterPage(page, markdown), success }.
const result = await client.createPage(
title,
content ?? '',
spaceId,
parentPageId,
);
const data = (result?.data ?? {}) as {
id?: string;
slugId?: string;
title?: string;
};
return { id: data.id ?? data.slugId, title: data.title ?? title };
},
),
updatePageContent: tool({
description:
"Replace a page's body with new Markdown content (and optionally its " +
'title). Reversible: the previous version is kept in page history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to update.'),
content: z.string().describe('The new page body as Markdown.'),
title: z
.string()
.optional()
.describe('Optional new title for the page.'),
}),
execute: async ({ pageId, content, title }) => {
// updatePage mutates the live collab doc -> provenance flows from the
// collab-token provider. Returns { success, modified, message, pageId }.
const result = (await client.updatePage(pageId, content, title)) as {
success?: boolean;
};
return { pageId, updated: result?.success ?? true };
},
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
renamePage: sharedTool(
sharedToolSpecs.renamePage,
async ({ pageId, title }) => {
// renamePage(pageId, title) -> { success, pageId, title }.
await client.renamePage(pageId, title);
return { pageId, title };
},
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The shared schema adds the optional `position` field this layer lacked
// before; the execute now forwards it (the client already accepted it).
movePage: sharedTool(
sharedToolSpecs.movePage,
async ({ pageId, parentPageId, position }) => {
// movePage(pageId, parentPageId, position?) -> raw move response.
await client.movePage(pageId, parentPageId ?? null, position);
return { pageId, parentPageId: parentPageId ?? null, moved: true };
},
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// GUARDRAIL (§14 H4) preserved: the shared schema exposes ONLY pageId, so
// permanentlyDelete/forceDelete are never part of the input and can never
// be forwarded — the agent physically cannot permanently delete a page.
deletePage: sharedTool(sharedToolSpecs.deletePage, async ({ pageId }) => {
// deletePage(pageId) hits POST /pages/delete with { pageId } only,
// which is the soft-delete (trash) path on the server.
await client.deletePage(pageId);
return { pageId, trashed: true };
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// This layer keeps only its own execute-side guards (require a selection
// for a top-level comment; reject suggestedText on a reply / without a
// selection) — the schema+description are shared.
createComment: sharedTool(
sharedToolSpecs.createComment,
async ({
pageId,
content,
selection,
parentCommentId,
suggestedText,
}) => {
// createComment(pageId, content, type, selection?, parentCommentId?,
// suggestedText?). Top-level comments are inline and must carry a
// selection to anchor on; replies inherit the parent's anchor (no
// selection). Throwing here surfaces a tool error to the model (Vercel
// `ai` SDK) so the agent retries with a better selection — do not
// catch/suppress it.
if (!parentCommentId && (!selection || !selection.trim())) {
throw new Error(
"createComment requires a 'selection' (exact text to anchor on) for a new top-level comment.",
);
}
if (suggestedText !== undefined) {
if (parentCommentId) {
throw new Error(
"createComment: 'suggestedText' cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !selection.trim()) {
throw new Error(
"createComment: 'suggestedText' requires a 'selection' to anchor and rewrite.",
);
}
}
const result = await client.createComment(
pageId,
content,
'inline',
selection,
parentCommentId,
suggestedText,
);
const data = (result?.data ?? {}) as { id?: string };
return { commentId: data.id, pageId };
},
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
resolveComment: sharedTool(
sharedToolSpecs.resolveComment,
async ({ commentId, resolved }) => {
// resolveComment(commentId, resolved) -> { success, commentId, resolved }.
await client.resolveComment(commentId, resolved);
return { commentId, resolved };
},
),
// --- READ tools (added) ---
getWorkspace: sharedTool(
sharedToolSpecs.getWorkspace,
async () => await client.getWorkspace(),
),
listSpaces: sharedTool(
sharedToolSpecs.listSpaces,
async () => await client.getSpaces(),
),
// INTENTIONAL per-transport divergence (not shared): keeps the `tree:true`
// hierarchy mode but is worded for the in-app agent; the standalone MCP
// `list_pages` carries its own wording. Kept per-layer so each side tunes
// its own guidance.
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
listPages: sharedTool(
sharedToolSpecs.listPages,
async ({ spaceId, limit, tree }) =>
await client.listPages(spaceId, limit, tree),
),
//
// NOTE (issue #411): the plain-Markdown full-body-replace tool is no longer
// inline here — it moved to @docmost/mcp's SHARED_TOOL_SPECS as
// `updatePageMarkdown` (was inline `updatePageContent`) so it registers on
// BOTH the external MCP and the in-app agent. The registry loop below adds
// it under its inAppKey. importPageMarkdown stays a shared spec too (now
// inAppOnly — dropped from the external MCP surface, kept in-app).
listSidebarPages: tool({
description:
@@ -525,31 +464,6 @@ export class AiChatToolsService {
await client.listSidebarPages(spaceId, pageId),
}),
getOutline: sharedTool(
sharedToolSpecs.getOutline,
async ({ pageId }) => await client.getOutline(pageId),
),
getPageJson: sharedTool(
sharedToolSpecs.getPageJson,
async ({ pageId }) => await client.getPageJson(pageId),
),
getNode: sharedTool(
sharedToolSpecs.getNode,
async ({ pageId, nodeId }) => await client.getNode(pageId, nodeId),
),
searchInPage: sharedTool(
sharedToolSpecs.searchInPage,
async ({ pageId, query, regex, caseSensitive, limit }) =>
await client.searchInPage(pageId, query, {
regex,
caseSensitive,
limit,
}),
),
// NOT shared (kept inline): the MCP tool name `table_get` is noun-first
// while this key is `getTable` (verb-first), breaking the
// snake_case(inAppKey) convention the shared registry enforces. Its
@@ -572,13 +486,6 @@ export class AiChatToolsService {
await client.getTable(pageId, table),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
listComments: sharedTool(
sharedToolSpecs.listComments,
async ({ pageId, includeResolved }) =>
await client.listComments(pageId, includeResolved),
),
getComment: tool({
description: 'Fetch a single comment by id (content as Markdown).',
inputSchema: modelFriendlyInput({
@@ -587,24 +494,6 @@ export class AiChatToolsService {
execute: async ({ commentId }) => await client.getComment(commentId),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
checkNewComments: sharedTool(
sharedToolSpecs.checkNewComments,
async ({ spaceId, since, parentPageId }) =>
await client.checkNewComments(spaceId, since, parentPageId),
),
listShares: sharedTool(
sharedToolSpecs.listShares,
async () => await client.listShares(),
),
listPageHistory: sharedTool(
sharedToolSpecs.listPageHistory,
async ({ pageId, cursor }) =>
await client.listPageHistory(pageId, cursor),
),
getPageHistory: tool({
description:
'Fetch a single page-history version including its lossless ' +
@@ -616,203 +505,8 @@ export class AiChatToolsService {
await client.getPageHistory(historyId),
}),
diffPageVersions: sharedTool(
sharedToolSpecs.diffPageVersions,
async ({ pageId, from, to }) =>
await client.diffPageVersions(pageId, from, to),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
exportPageMarkdown: sharedTool(
sharedToolSpecs.exportPageMarkdown,
async ({ pageId }) => {
const markdown = await client.exportPageMarkdown(pageId);
return { markdown };
},
),
// --- WRITE tools (added; reversible via page history/trash) ---
editPageText: sharedTool(
sharedToolSpecs.editPageText,
async ({ pageId, edits }) => await client.editPageText(pageId, edits),
),
// Returns ONLY the short link object — never the document body — so a
// large page can be handed to an external consumer without bloating
// context.
stashPage: sharedTool(
sharedToolSpecs.stashPage,
async ({ pageId }) => await client.stashPage(pageId),
),
// Schema + description from the shared registry (identical across both
// transports). The execute body keeps its OWN parseNodeArg normalization:
// the model sometimes serializes the node as a JSON string, and we parse it
// before the client's typeof-object guard rejects it (parity with the
// standalone MCP server, index.ts patch_node).
patchNode: sharedTool(
sharedToolSpecs.patchNode,
async ({ pageId, nodeId, node }) => {
const parsedNode = parseNodeArg(node);
return await client.patchNode(pageId, nodeId, parsedNode);
},
),
// Shared registry schema + description; execute retains parseNodeArg on the
// incoming node (parity with the standalone MCP server, index.ts
// insert_node).
insertNode: sharedTool(
sharedToolSpecs.insertNode,
async ({ pageId, node, position, anchorNodeId, anchorText }) => {
const parsedNode = parseNodeArg(node);
return await client.insertNode(pageId, parsedNode, {
position,
anchorNodeId,
anchorText,
});
},
),
deleteNode: sharedTool(
sharedToolSpecs.deleteNode,
async ({ pageId, nodeId }) => await client.deleteNode(pageId, nodeId),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The execute body keeps this layer's content normalization (parity with
// the standalone MCP server, index.ts update_page_json).
updatePageJson: sharedTool(
sharedToolSpecs.updatePageJson,
async ({ pageId, content, title }) => {
// undefined/null pass through as undefined (title-only / no-op); any
// string is JSON.parsed (so an empty string "" throws, matching the
// MCP server); an object is passed through unchanged.
let doc;
if (content === undefined || content === null) {
doc = undefined;
} else {
// String -> JSON.parse (throwing on invalid); object passes through.
doc = parseNodeArg(content, 'content was a string but not valid JSON');
}
return await client.updatePageJson(pageId, doc, title);
},
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
// Promoted from MCP-only so the in-app agent can attach a REAL footnote to
// already-written text instead of leaving a literal `^[...]` string.
insertFootnote: sharedTool(
sharedToolSpecs.insertFootnote,
async ({ pageId, anchorText, text }) =>
await client.insertFootnote(pageId, anchorText, text),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
// The schema field is `imageUrl`; the client method takes it positionally.
insertImage: sharedTool(
sharedToolSpecs.insertImage,
async ({ pageId, imageUrl, align, alt, replaceText, afterText }) =>
await client.insertImage(pageId, imageUrl, {
align,
alt,
replaceText,
afterText,
}),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
replaceImage: sharedTool(
sharedToolSpecs.replaceImage,
async ({ pageId, attachmentId, imageUrl, align, alt }) =>
await client.replaceImage(pageId, attachmentId, imageUrl, {
align,
alt,
}),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#423).
// meta.hash in the result is the baseHash drawioUpdate requires.
drawioGet: sharedTool(
sharedToolSpecs.drawioGet,
async ({ pageId, node, format }) =>
await client.drawioGet(pageId, node, format ?? 'xml'),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#423).
// The flat schema fields are regrouped into the client's `where` object.
drawioCreate: sharedTool(
sharedToolSpecs.drawioCreate,
async ({ pageId, xml, position, anchorNodeId, anchorText, title }) =>
await client.drawioCreate(
pageId,
{ position, anchorNodeId, anchorText },
xml,
title,
),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#423).
// baseHash is the optimistic lock: mismatch => structured conflict error.
drawioUpdate: sharedTool(
sharedToolSpecs.drawioUpdate,
async ({ pageId, node, xml, baseHash }) =>
await client.drawioUpdate(pageId, node, xml, baseHash),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The table reference parameter was unified to `table` (was `tableRef`).
tableInsertRow: sharedTool(
sharedToolSpecs.tableInsertRow,
async ({ pageId, table, cells, index }) =>
await client.tableInsertRow(pageId, table, cells, index),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
tableDeleteRow: sharedTool(
sharedToolSpecs.tableDeleteRow,
async ({ pageId, table, index }) =>
await client.tableDeleteRow(pageId, table, index),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
tableUpdateCell: sharedTool(
sharedToolSpecs.tableUpdateCell,
async ({ pageId, table, row, col, text }) =>
await client.tableUpdateCell(pageId, table, row, col, text),
),
copyPageContent: sharedTool(
sharedToolSpecs.copyPageContent,
async ({ sourcePageId, targetPageId }) =>
await client.copyPageContent(sourcePageId, targetPageId),
),
importPageMarkdown: sharedTool(
sharedToolSpecs.importPageMarkdown,
async ({ pageId, markdown }) =>
await client.importPageMarkdown(pageId, markdown),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// Both layers already carried the security-confirmation framing, so there
// was no real divergence to preserve — only wording drift.
sharePage: sharedTool(
sharedToolSpecs.sharePage,
async ({ pageId, searchIndexing }) =>
await client.sharePage(pageId, searchIndexing),
),
unsharePage: sharedTool(
sharedToolSpecs.unsharePage,
async ({ pageId }) => await client.unsharePage(pageId),
),
restorePageVersion: sharedTool(
sharedToolSpecs.restorePageVersion,
async ({ historyId }) => await client.restorePageVersion(historyId),
),
// INTENTIONAL per-transport divergence (not shared): deliberately omits the
// `deleteComments` schema field (comment-deletion guardrail) and carries a
// much shorter description; the standalone MCP `docmost_transform` exposes
@@ -841,6 +535,29 @@ export class AiChatToolsService {
}),
};
// Add EVERY shared tool from the zod-agnostic registry in one loop (#445).
// The spec owns the canonical arg->client mapping; this host only decides
// WHICH mapping to run and returns its value directly (no envelope). For each
// spec:
// - skip `mcpOnly` specs (they belong to the standalone MCP host only);
// - use `inAppExecute` when the spec declares a DELIBERATE per-layer
// difference (a projected result shape, a different guardrail message);
// - otherwise use the canonical `execute` (raw client result, identical to
// the MCP host's before it wraps it as JSON).
// The execute receives the AI-SDK-validated, type-erased input; the spec reads
// the same fields its buildShape declares. This is the SINGLE place the in-app
// arg mapping lives — it can no longer silently drift from the MCP host.
for (const spec of Object.values(sharedToolSpecs)) {
if (spec.mcpOnly) continue;
const run = spec.inAppExecute ?? spec.execute;
if (!run) continue; // defensive: a shared spec always carries one of them.
tools[spec.inAppKey] = sharedTool(
spec,
(async (args) =>
run(client, args as Record<string, unknown>)) as Tool['execute'],
);
}
// Passive "new comments: N" signal (#417). PER-TURN state (forUser runs once
// per turn), so the watermark starts now and only comments a human leaves
// WHILE this turn runs are signalled — exactly the mid-turn loop; between-turn
@@ -7,6 +7,16 @@ import type {
DocmostClientLike,
CommentSignalTrackerLike,
} from './docmost-client.loader';
// Test-double type for the loopback client. `DocmostClientLike` is now DERIVED
// from the real `DocmostClient` (issue #446), so its method RETURN types are the
// concrete client shapes. These probe stubs deliberately return minimal shapes
// (e.g. `getPageRaw` yielding only `{ title }`), so the doubles use the same
// method NAMES but loose async returns; each is cast to `DocmostClientLike` at
// the (return-erased) mock site, leaving production positional-call safety intact.
type FakeDocmostClient = Partial<
Record<keyof DocmostClientLike, (...args: any[]) => Promise<any>>
>;
import { SHARED_TOOL_SPECS } from '../../../../../../packages/mcp/src/tool-specs';
// The REAL shared tracker factory, imported from source (same cross-boundary
// approach the tool-specs spec uses) so the in-app wiring is exercised against
@@ -268,12 +278,12 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
// seeded at forUser time).
const future = new Date(Date.now() + 3_600_000).toISOString();
function buildService(fakeClient: Partial<DocmostClientLike>) {
function buildService(fakeClient: FakeDocmostClient) {
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue({
DocmostClient: function () {
return fakeClient as DocmostClientLike;
} as unknown as loader.DocmostClientCtor,
sharedToolSpecs: SHARED_TOOL_SPECS as Record<string, loader.SharedToolSpec>,
sharedToolSpecs: SHARED_TOOL_SPECS as unknown as Record<string, loader.SharedToolSpec>,
// Wire the REAL factory so the in-app path is exercised end to end.
createCommentSignalTracker:
createCommentSignalTracker as unknown as loader.CommentSignalTrackerFactory,
@@ -317,7 +327,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
afterEach(() => jest.restoreAllMocks());
it('emits the signal (model-only) on a non-comment tool when a new comment exists', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
getPage: async () => ({
data: { title: 'Иранские языки', content: 'body' },
success: true,
@@ -342,7 +352,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
});
it('does NOT add the signal to the listComments tool itself (tautological)', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
listComments: async () => ({
items: [{ createdAt: future }],
resolvedThreadsHidden: 0,
@@ -356,7 +366,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
});
it('no new comments => tool output is byte-identical AND the model sees no signal', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
getPage: async () => ({
data: { title: 'T', content: 'body' },
success: true,
@@ -372,7 +382,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
});
it('injection-safety: a malicious page title cannot forge a second signal', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
getPage: async () => ({
data: { title: 'body-title', content: 'body' },
success: true,
@@ -0,0 +1,173 @@
import { createHash } from 'node:crypto';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { computeSrcRegistryStamp } from './docmost-client.loader';
// The exact message the loader throws on a build/src skew (issue #447). Kept as a
// literal here so a reworded prod message reddens this test (the message is a
// developer-facing contract: it tells them how to fix it).
const STALE_BUILD_MESSAGE =
'@docmost/mcp build is stale (tool-specs changed since last build) — run: pnpm --filter @docmost/mcp build';
// Replica of the loader's inline stale-check predicate + throw from
// `loadDocmostMcp`. That guard is not independently exported (it lives inside the
// dynamic-import IIFE, wired to a fixed `require.resolve('@docmost/mcp')`), so we
// exercise the exact same three-condition logic against a stamp produced by the
// REAL `computeSrcRegistryStamp`. This documents and locks the throw/no-throw
// behaviour; if the prod predicate changes, this replica must change with it.
function assertStaleGuard(
srcStamp: string | null,
registryStamp: string | undefined,
): void {
if (
srcStamp !== null &&
typeof registryStamp === 'string' &&
srcStamp !== registryStamp
) {
throw new Error(STALE_BUILD_MESSAGE);
}
}
// Build a throwaway `<pkg>/build/index.js` + optional `<pkg>/src/tool-specs.ts`
// layout so `computeSrcRegistryStamp(<pkg>/build/index.js)` resolves src the same
// way the loader does (dirname(dirname(entry))/src/tool-specs.ts).
function makeFakePackage(toolSpecsSource: string | null): {
entry: string;
cleanup: () => void;
} {
const root = mkdtempSync(join(tmpdir(), 'mcp-stamp-'));
const buildDir = join(root, 'build');
mkdirSync(buildDir, { recursive: true });
const entry = join(buildDir, 'index.js');
writeFileSync(entry, '// fake @docmost/mcp build entry\n', 'utf8');
if (toolSpecsSource !== null) {
const srcDir = join(root, 'src');
mkdirSync(srcDir, { recursive: true });
writeFileSync(join(srcDir, 'tool-specs.ts'), toolSpecsSource, 'utf8');
}
return { entry, cleanup: () => rmSync(root, { recursive: true, force: true }) };
}
describe('computeSrcRegistryStamp (#447 stale-build guard)', () => {
it('returns null when src/tool-specs.ts is absent (prod no-op path)', () => {
// A prod image ships only build/, no src/ — the guard must be a silent no-op.
const { entry, cleanup } = makeFakePackage(null);
try {
expect(computeSrcRegistryStamp(entry)).toBeNull();
} finally {
cleanup();
}
});
it('returns null for a bogus package entry (swallowed error path)', () => {
// A resolution/read hiccup must NEVER break startup — it resolves to null.
expect(
computeSrcRegistryStamp('/no/such/pkg/build/index.js'),
).toBeNull();
});
it('computes a 64-char sha256 hex when src/tool-specs.ts exists', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const stamp = computeSrcRegistryStamp(entry);
expect(stamp).toMatch(/^[0-9a-f]{64}$/);
} finally {
cleanup();
}
});
it('normalizes CRLF->LF and strips a single trailing newline', () => {
// A CRLF+trailing-newline variant of the same content hashes identically to
// the bare-LF form — the guard must not fire on a checkout-style difference.
const bare = makeFakePackage('alpha\nbeta');
const crlfTrailing = makeFakePackage('alpha\r\nbeta\r\n');
try {
expect(computeSrcRegistryStamp(crlfTrailing.entry)).toBe(
computeSrcRegistryStamp(bare.entry),
);
} finally {
bare.cleanup();
crlfTrailing.cleanup();
}
});
// CROSS-IMPL EQUALITY (covers reviewer suggestion 2). The SAME fixed input and
// EXPECTED hash are asserted in the mcp-side node test
// (packages/mcp/test/unit/registry-stamp.test.mjs) against the codegen's
// `computeRegistryStamp`. Asserting the SAME pair here against the loader's
// `computeSrcRegistryStamp` proves both implementations normalize+hash
// identically; a divergence in EITHER side reddens one of the two tests.
it('matches the documented cross-impl hash for a fixed input', () => {
const FIXED_INPUT = 'line1\r\nline2\n';
const EXPECTED =
'683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83';
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
try {
expect(computeSrcRegistryStamp(entry)).toBe(EXPECTED);
} finally {
cleanup();
}
});
it('the documented EXPECTED is the normalize+sha256 of the fixed input', () => {
// Proves EXPECTED is not a magic constant but the documented computation.
const FIXED_INPUT = 'line1\r\nline2\n';
const normalized = FIXED_INPUT.replace(/\r\n/g, '\n').replace(/\n$/, '');
const expected = createHash('sha256')
.update(normalized, 'utf8')
.digest('hex');
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
try {
expect(computeSrcRegistryStamp(entry)).toBe(expected);
} finally {
cleanup();
}
});
});
describe('loadDocmostMcp stale-check predicate (#447)', () => {
it('THROWS the exact stale message when src stamp != built REGISTRY_STAMP', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
expect(srcStamp).not.toBeNull();
// Simulate a stale build: build/ carries a DIFFERENT stamp than src.
expect(() => assertStaleGuard(srcStamp, 'a'.repeat(64))).toThrow(
STALE_BUILD_MESSAGE,
);
} finally {
cleanup();
}
});
it('does NOT throw when src stamp equals the built REGISTRY_STAMP', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
// Fresh build: build/ stamp == src stamp -> guard is a no-op.
expect(() => assertStaleGuard(srcStamp, srcStamp as string)).not.toThrow();
} finally {
cleanup();
}
});
it('does NOT throw when src is absent (prod: srcStamp === null)', () => {
// Even against a present-but-mismatched REGISTRY_STAMP, a null src stamp
// (prod image with build/ only) must skip the check entirely.
expect(() => assertStaleGuard(null, 'a'.repeat(64))).not.toThrow();
});
it('does NOT throw when REGISTRY_STAMP is absent (pre-#447 build)', () => {
// An older @docmost/mcp build has no REGISTRY_STAMP export; the guard must be
// a no-op so an out-of-date build never wrongly blocks startup.
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
expect(() => assertStaleGuard(srcStamp, undefined)).not.toThrow();
} finally {
cleanup();
}
});
});
@@ -1,264 +1,93 @@
import { createHash } from 'node:crypto';
import { existsSync, readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { pathToFileURL } from 'node:url';
import type { DocmostClient, SharedToolSpec } from '@docmost/mcp';
// Re-export SharedToolSpec so downstream server modules keep a single import
// path (they import it from this loader). The shape is DERIVED from the package
// entry, not re-declared here — see the import above (issue #446).
export type { SharedToolSpec } from '@docmost/mcp';
/**
* Minimal structural type for the `DocmostClient` class we consume from the
* ESM-only `@docmost/mcp` package. We only need the constructor + the read/write
* methods used by the per-user tool adapter; the full client surface lives in
* `packages/mcp/src/client.ts`. Signatures here mirror that file exactly.
*
* DRIFT GUARD: the method NAMES below are runtime-checked against the real
* `DocmostClient` by `packages/mcp/test/unit/client-host-contract.test.mjs`
* (which can import the ESM class directly). If you rename/remove a method here
* or in client.ts, that test fails so a stale mirror cannot silently ship a
* runtime "x is not a function" into an agent tool call. Keep the two in sync.
*
* STAGED PLAN full derivation `DocmostClientLike = <real DocmostClient type>`
* (issue #193, layer 3) is intentionally NOT done; it stays a hand-mirror for
* now because of two verified blockers across the ESM(mcp)/CJS(server) boundary:
* 1. `@docmost/mcp` emits NO declaration files (its tsconfig has no
* `declaration`, package.json has no `types`/types-export) and the server
* tsconfig has no path mapping for it the server only loads it via the
* runtime `import()` trick below, so there is no type to import today.
* 2. The real client methods have inferred, CONCRETE return types; the in-app
* tool adapter reads results through loose `Record<string,unknown>` returns
* + `as` casts (e.g. `(result?.data ?? {}) as { title?: string }`).
* Deriving the exact type would make those casts non-overlapping ("may be a
* mistake") and break the build, and `Partial<DocmostClientLike>` test stubs
* would have to satisfy the full concrete surface.
* To do it safely later (incrementally): (a) turn on `declaration: true` in
* packages/mcp/tsconfig.json + add a `types` export condition and commit the
* emitted `.d.ts`; (b) `import type { DocmostClient } from '@docmost/mcp'` here
* and replace this interface with a `Pick<DocmostClient, ...>` of the consumed
* methods; (c) audit every `as` cast in ai-chat-tools.service.ts against the now
* concrete return types (double-cast through `unknown` only where genuinely
* needed); (d) keep the runtime guard test as a belt-and-braces check. Until
* then the guard test above is the cheap, behaviour-neutral protection.
* The exact set of `DocmostClient` methods the per-user in-app tool adapter
* consumes. This is the AUTHORITATIVE list of the client surface the server
* depends on; the adapter calls these methods POSITIONALLY, so this set is what
* the derived type below type-checks against the real class (issue #446).
*/
export interface DocmostClientLike {
type DocmostClientMethod =
// --- read ---
search(
query: string,
spaceId?: string,
limit?: number,
): Promise<{ items: unknown[]; success: boolean }>;
getPage(
pageId: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
// Light raw page info (`/pages/info`): title + slugId + ProseMirror content,
// WITHOUT the Markdown render / subpage expansion getPage does. Used by the
// comment-signal probe to read just the page title on a hit.
getPageRaw(pageId: string): Promise<Record<string, unknown> | null>;
getWorkspace(): Promise<{ data: Record<string, unknown>; success: boolean }>;
getSpaces(): Promise<unknown[]>;
listPages(
spaceId?: string,
limit?: number,
tree?: boolean,
): Promise<unknown[]>;
listSidebarPages(spaceId: string, pageId?: string): Promise<unknown[]>;
getOutline(pageId: string): Promise<Record<string, unknown>>;
getPageJson(pageId: string): Promise<Record<string, unknown>>;
getNode(pageId: string, nodeId: string): Promise<Record<string, unknown>>;
searchInPage(
pageId: string,
query: string,
opts?: { regex?: boolean; caseSensitive?: boolean; limit?: number },
): Promise<Record<string, unknown>>;
getTable(pageId: string, tableRef: string): Promise<Record<string, unknown>>;
// Returns `{ items, resolvedThreadsHidden }`. DEFAULT (includeResolved unset/
// false) hides resolved threads wholesale; pass true for the full feed.
listComments(
pageId: string,
includeResolved?: boolean,
): Promise<{ items: unknown[]; resolvedThreadsHidden: number }>;
getComment(
commentId: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
checkNewComments(
spaceId: string,
since: string,
parentPageId?: string,
): Promise<unknown>;
listShares(): Promise<unknown[]>;
listPageHistory(
pageId: string,
cursor?: string,
): Promise<{ items: unknown[]; nextCursor: string | null }>;
getPageHistory(historyId: string): Promise<Record<string, unknown>>;
diffPageVersions(
pageId: string,
from?: string,
to?: string,
): Promise<Record<string, unknown>>;
exportPageMarkdown(pageId: string): Promise<string>;
| 'search'
| 'getPage'
| 'getPageRaw'
| 'getWorkspace'
| 'getSpaces'
| 'listPages'
| 'listSidebarPages'
| 'getOutline'
| 'getPageJson'
| 'getNode'
| 'searchInPage'
| 'getTable'
| 'listComments'
| 'getComment'
| 'checkNewComments'
| 'listShares'
| 'listPageHistory'
| 'getPageHistory'
| 'diffPageVersions'
| 'exportPageMarkdown'
// --- write (page) ---
createPage(
title: string,
content: string,
spaceId: string,
parentPageId?: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
// Markdown content update via the collab path (carries provenance via the
// collab-token provider). Optionally also updates the title.
updatePage(
pageId: string,
content: string,
title?: string,
): Promise<Record<string, unknown>>;
// Title-only rename via REST.
renamePage(
pageId: string,
title: string,
): Promise<Record<string, unknown>>;
// Move via REST. parentPageId null => move to space root.
movePage(
pageId: string,
parentPageId: string | null,
position?: string,
): Promise<unknown>;
// SOFT delete only (POST /pages/delete with { pageId }). NEVER permanent.
deletePage(pageId: string): Promise<unknown>;
editPageText(
pageId: string,
edits: Array<{ find: string; replace: string; replaceAll?: boolean }>,
): Promise<Record<string, unknown>>;
patchNode(
pageId: string,
nodeId: string,
node: unknown,
): Promise<Record<string, unknown>>;
insertNode(
pageId: string,
node: unknown,
opts: {
position: 'before' | 'after' | 'append';
anchorNodeId?: string;
anchorText?: string;
},
): Promise<Record<string, unknown>>;
deleteNode(
pageId: string,
nodeId: string,
): Promise<Record<string, unknown>>;
updatePageJson(
pageId: string,
doc?: unknown,
title?: string,
): Promise<Record<string, unknown>>;
// Attach an author-inline footnote after the first occurrence of anchorText;
// numbering + the footnotes list are derived server-side.
insertFootnote(
pageId: string,
anchorText: string,
text: string,
): Promise<Record<string, unknown>>;
// Download a web image and insert it into the page (append, or replace/after a
// text anchor). `url` is the image http(s) URL.
insertImage(
pageId: string,
url: string,
opts?: {
align?: 'left' | 'center' | 'right';
alt?: string;
replaceText?: string;
afterText?: string;
},
): Promise<Record<string, unknown>>;
// Swap an existing image (by its attachmentId) for a new one fetched from a web
// URL, repointing every reference in the live document.
replaceImage(
pageId: string,
oldAttachmentId: string,
url: string,
opts?: { align?: 'left' | 'center' | 'right'; alt?: string },
): Promise<Record<string, unknown>>;
| 'createPage'
| 'updatePage'
| 'renamePage'
| 'movePage'
| 'deletePage'
| 'editPageText'
| 'patchNode'
| 'insertNode'
| 'deleteNode'
| 'updatePageJson'
| 'tableInsertRow'
| 'tableDeleteRow'
| 'tableUpdateCell'
| 'copyPageContent'
| 'importPageMarkdown'
| 'sharePage'
| 'unsharePage'
| 'restorePageVersion'
| 'transformPage'
| 'stashPage'
// --- write (image / footnote), in-app since #410 ---
| 'insertImage'
| 'replaceImage'
| 'insertFootnote'
// --- draw.io diagrams (#423, stage 1) ---
// Read a diagram as decoded mxGraph XML (default) or the raw .drawio.svg.
// meta.hash is the optimistic-lock key drawioUpdate expects as baseHash.
drawioGet(
pageId: string,
node: string,
format?: 'xml' | 'svg',
): Promise<Record<string, unknown>>;
// Lint mxGraph XML, build the .drawio.svg attachment and insert a drawio node.
drawioCreate(
pageId: string,
where: {
position: 'before' | 'after' | 'append';
anchorNodeId?: string;
anchorText?: string;
},
xml: string,
title?: string,
): Promise<Record<string, unknown>>;
// Optimistic-locked full replacement of a diagram (baseHash from drawioGet).
drawioUpdate(
pageId: string,
node: string,
xml: string,
baseHash: string,
): Promise<Record<string, unknown>>;
tableInsertRow(
pageId: string,
tableRef: string,
cells: string[],
index?: number,
): Promise<Record<string, unknown>>;
tableDeleteRow(
pageId: string,
tableRef: string,
index: number,
): Promise<Record<string, unknown>>;
tableUpdateCell(
pageId: string,
tableRef: string,
row: number,
col: number,
text: string,
): Promise<Record<string, unknown>>;
copyPageContent(
sourcePageId: string,
targetPageId: string,
): Promise<Record<string, unknown>>;
importPageMarkdown(
pageId: string,
fullMarkdown: string,
): Promise<Record<string, unknown>>;
sharePage(
pageId: string,
searchIndexing?: boolean,
): Promise<Record<string, unknown>>;
unsharePage(pageId: string): Promise<Record<string, unknown>>;
restorePageVersion(historyId: string): Promise<Record<string, unknown>>;
// The opts type declares deleteComments? to match the real client signature,
// but the agent tool NEVER sets it (comment deletion stays unreachable).
transformPage(
pageId: string,
transformJs: string,
opts?: { dryRun?: boolean; deleteComments?: boolean },
): Promise<Record<string, unknown>>;
| 'drawioGet'
| 'drawioCreate'
| 'drawioUpdate'
// --- write (comment) ---
createComment(
pageId: string,
content: string,
type?: 'page' | 'inline',
selection?: string,
parentCommentId?: string,
suggestedText?: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
resolveComment(
commentId: string,
resolved: boolean,
): Promise<Record<string, unknown>>;
// Serialize a page + mirror its internal images into the blob sandbox; returns
// ONLY a short anonymous URL (the body never enters the model context).
stashPage(pageId: string): Promise<{
uri: string;
sha256: string;
size: number;
images: { mirrored: number; failed: number };
}>;
}
| 'createComment'
| 'resolveComment';
/**
* The client surface the per-user tool adapter consumes, DERIVED from the real
* `DocmostClient` type in `@docmost/mcp` (issue #446, restored #294 debt). This
* replaces the former hand-mirror of ~45 method signatures.
*
* `import type` (above) is fully ERASED at compile time, so nothing is actually
* imported from the ESM-only package at runtime the server still loads the
* class through the dynamic `import()` trick in `loadDocmostMcp` below; this is
* purely a compile-time type. Deriving via `Pick` means a parameter reorder or a
* type change to any of these methods in `client.ts` now becomes a SERVER
* COMPILE ERROR at the positional call sites in ai-chat-tools.service.ts,
* instead of a silent runtime "wrong argument" failure inside an agent tool.
*
* This made the old name-only drift-guard test
* (packages/mcp/test/unit/client-host-contract.test.mjs) redundant tsc now
* enforces both names AND signatures so that test was removed.
*/
export type DocmostClientLike = Pick<DocmostClient, DocmostClientMethod>;
export type DocmostClientConfig = {
apiUrl: string;
@@ -280,32 +109,7 @@ export type DocmostClientConfig = {
};
export interface DocmostClientCtor {
new (config: DocmostClientConfig): DocmostClientLike;
}
/**
* Local hand-mirror of the `SharedToolSpec` shape exported from
* `@docmost/mcp` (packages/mcp/src/tool-specs.ts). Same approach as
* `DocmostClientLike`: we do not import the ESM package's types directly across
* the CJS/ESM boundary. The registry itself has no runtime deps, but keeping the
* type local avoids coupling the server build to the package's type surface.
*
* `buildShape` is intentionally zod-agnostic: it returns a plain ZodRawShape
* built with whatever zod namespace the caller passes (the server passes its own
* zod v4; the MCP package passes its zod v3). See the registry module comment.
*/
export interface SharedToolSpec {
mcpName: string;
inAppKey: string;
description: string;
// Deferred-tool metadata (#332). Optional in this mirror so an older/stale
// @docmost/mcp build (pre-#332) still type-checks; the in-app catalog builder
// reads them defensively. The external /mcp server ignores both fields.
tier?: 'core' | 'deferred';
catalogLine?: string;
// Loose `z` on purpose: the registry is zod-agnostic so the server can pass
// its own zod (v4) and the MCP package its own (v3) into the same builder.
buildShape?: (z: any) => Record<string, unknown>;
new (config: DocmostClientConfig): DocmostClient;
}
/**
@@ -344,6 +148,50 @@ interface DocmostMcpModule {
// loader in unit tests. The in-app layer treats an absent factory as "signal
// disabled" — a pure no-op that leaves tool results byte-identical.
createCommentSignalTracker?: CommentSignalTrackerFactory;
// Optional (#447): a deterministic hash of the tool-specs registry content,
// generated into build/ by the package's build. Absent on a pre-#447 build (or
// the mocked loader in unit tests) — the stale-check below is a NO-OP when it
// is missing, so an older build never wrongly fails startup.
REGISTRY_STAMP?: string;
}
/**
* Recompute the REGISTRY_STAMP (#447) from the @docmost/mcp source tree, if it is
* present. Returns the stamp string, or `null` when the source is absent (a prod
* image ships only build/, no src/). MUST stay byte-for-byte identical to
* packages/mcp/scripts/gen-registry-stamp.mjs's `computeRegistryStamp` so the
* build-time and src-time hashes agree: same input file (src/tool-specs.ts), same
* normalization (CRLF -> LF, strip a single trailing newline), same sha256.
*
* DEV vs PROD detection is by FILE EXISTENCE, not NODE_ENV: we resolve the
* package's own directory from `require.resolve('@docmost/mcp')` (which points at
* build/index.js) and look for ../src/tool-specs.ts next to it. In a dev/test
* worktree that file exists; in a prod image (build/ only, src/ stripped) it does
* not, so this returns null and the caller skips the check. Any error (ENOENT, a
* bad resolve) is swallowed to null the stale-check must NEVER break startup.
*
* Exported for unit testing (docmost-client.loader.spec.ts): the export keyword
* is behaviourally a no-op the module-internal caller `loadDocmostMcp` is
* unaffected. The test drives the null (no-src) path and asserts this
* normalize+sha256 stays identical to the codegen's `computeRegistryStamp`.
*/
export function computeSrcRegistryStamp(packageEntry: string): string | null {
try {
// packageEntry is <pkg>/build/index.js; the source lives at <pkg>/src/.
const toolSpecsPath = join(
dirname(dirname(packageEntry)),
'src',
'tool-specs.ts',
);
if (!existsSync(toolSpecsPath)) return null; // prod: no src tree -> skip.
const source = readFileSync(toolSpecsPath, 'utf8');
const normalized = source.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
} catch {
// Never let a resolution/read hiccup break server startup — treat as "no
// src available" and skip the check (identical to the prod no-op path).
return null;
}
}
// TS with module:commonjs downlevels a literal `import()` to `require()`, which
@@ -375,6 +223,23 @@ export async function loadDocmostMcp(): Promise<{
const mod = (await esmImport(
pathToFileURL(entry).href,
)) as DocmostMcpModule;
// #447 stale-build guard (dev/test only). The server loads the COMPILED
// build/ of @docmost/mcp, but the parity/tier guard tests read src/. If a
// tool spec is edited in src without rebuilding the package, build/ and src/
// silently diverge and the running server serves the OLD tools. Here we
// recompute the stamp from src/tool-specs.ts and compare it to the stamp
// baked into build/. In PROD the src tree is absent (image ships build/
// only), so computeSrcRegistryStamp returns null and this is a pure no-op.
const srcStamp = computeSrcRegistryStamp(entry);
if (
srcStamp !== null &&
typeof mod.REGISTRY_STAMP === 'string' &&
srcStamp !== mod.REGISTRY_STAMP
) {
throw new Error(
'@docmost/mcp build is stale (tool-specs changed since last build) — run: pnpm --filter @docmost/mcp build',
);
}
return mod;
})().catch((err) => {
// Do not cache a rejected import — allow the next call to retry.
@@ -70,7 +70,7 @@ describe('SHARED_TOOL_SPECS contract parity', () => {
// Type as the (optional-buildShape) SharedToolSpec; the `satisfies` literal
// above otherwise narrows to a union where some members lack buildShape.
const specEntries = Object.entries(SHARED_TOOL_SPECS) as Array<
const specEntries = Object.entries(SHARED_TOOL_SPECS) as unknown as Array<
[string, loader.SharedToolSpec]
>;
@@ -123,7 +123,7 @@ describe('deferred catalog ↔ live forUser() toolset partition (#332, F3)', ()
DocmostClient: function () {
return {} as DocmostClientLike;
} as unknown as loader.DocmostClientCtor,
sharedToolSpecs: SHARED_TOOL_SPECS as Record<string, loader.SharedToolSpec>,
sharedToolSpecs: SHARED_TOOL_SPECS as unknown as Record<string, loader.SharedToolSpec>,
});
const service = new AiChatToolsService(
{
@@ -232,6 +232,16 @@ describe('applyLoadTools (#332)', () => {
expect(LOAD_TOOLS_DESCRIPTION).toContain('only ACTIVATES them');
expect(LOAD_TOOLS_DESCRIPTION).toContain('callable on your NEXT step');
});
it('loadTools description tells the model CORE tools are always active (#444)', () => {
expect(LOAD_TOOLS_DESCRIPTION).toContain(
'Tools NOT listed in the catalog are CORE and ALWAYS active',
);
expect(LOAD_TOOLS_DESCRIPTION).toContain('NEVER via loadTools');
// Names it out explicitly so the model doesn't loadTools a core tool.
expect(LOAD_TOOLS_DESCRIPTION).toContain('createComment');
expect(LOAD_TOOLS_DESCRIPTION).toContain('searchInPage');
});
});
describe('editorial "Corrector" scenario is fully served by CORE (#332)', () => {
@@ -84,7 +84,10 @@ export const LOAD_TOOLS_DESCRIPTION =
'block in your instructions. Pass the EXACT tool names from the catalog; this\n' +
'call only ACTIVATES them and returns { loaded: [...] } — the tools become\n' +
'callable on your NEXT step. Load several names in one call when the task clearly\n' +
'needs them. Unknown names are rejected with the list of valid ones.';
'needs them. Unknown names are rejected with the list of valid ones.\n' +
'Tools NOT listed in the catalog are CORE and ALWAYS active — call them directly,\n' +
'NEVER via loadTools (e.g. createComment, listComments, resolveComment,\n' +
'editPageText, searchInPage).';
/**
* Tier + catalogLine for the INLINE ai-chat tools those defined per-layer in
@@ -121,12 +124,9 @@ export const INLINE_TOOL_TIERS: Record<
// --- deferred inline ---
// NOTE: createPage, renamePage, movePage, deletePage, updatePageJson and
// exportPageMarkdown moved to @docmost/mcp's SHARED_TOOL_SPECS (#294); they
// carry their own deferred tier + catalogLine there.
updatePageContent: {
tier: 'deferred',
catalogLine:
"updatePageContent — replace a page's body (and optionally title) with new Markdown.",
},
// carry their own deferred tier + catalogLine there. updatePageContent moved
// there too as updatePageMarkdown (#411) — a shared registry spec now, so it
// is no longer an inline tier entry.
listSidebarPages: {
tier: 'deferred',
catalogLine:
@@ -158,4 +158,27 @@ describe('EnvironmentService', () => {
).toBe('https://app.example.com');
});
});
describe('isAiChatFinalStepLockdownEnabled (#444)', () => {
const build = (val?: string) =>
new EnvironmentService({
get: (key: string, def?: string) =>
key === 'AI_CHAT_FINAL_STEP_LOCKDOWN' ? (val ?? def) : def,
} as any);
it('defaults to OFF (false) when unset — the new anti-degeneration default', () => {
expect(build(undefined).isAiChatFinalStepLockdownEnabled()).toBe(false);
});
it('is true only for the exact opt-in "true" (case-insensitive)', () => {
expect(build('true').isAiChatFinalStepLockdownEnabled()).toBe(true);
expect(build('TRUE').isAiChatFinalStepLockdownEnabled()).toBe(true);
});
it('stays OFF for any other value', () => {
expect(build('false').isAiChatFinalStepLockdownEnabled()).toBe(false);
expect(build('1').isAiChatFinalStepLockdownEnabled()).toBe(false);
expect(build('yes').isAiChatFinalStepLockdownEnabled()).toBe(false);
});
});
});
@@ -292,6 +292,24 @@ export class EnvironmentService {
return enabled === 'true';
}
/**
* Final-step lockdown for the in-app agent loop (#444). When ON (legacy), the
* LAST allowed step forces a text-only answer: tools are stripped
* (toolChoice:'none') and a synthesis instruction is appended. Defaults to OFF:
* stripping the tools mid-work triggered a token-loop degeneration incident
* (the model, robbed of its tools on the final step, emitted a 255KB block
* repeating a single token). With the toggle OFF the last step keeps its tools
* and gets only a SOFT nudge to finish with a text summary; the universal
* anti-babble guard is the token-degeneration detector instead. Enable this
* only for a model that does NOT reliably end its turns with a text answer.
*/
isAiChatFinalStepLockdownEnabled(): boolean {
const enabled = this.configService
.get<string>('AI_CHAT_FINAL_STEP_LOCKDOWN', 'false')
.toLowerCase();
return enabled === 'true';
}
/**
* Resumable SSE transport for durable agent runs (#184 phase 1.5). When
* enabled, a run tees its SSE frames into the in-memory run-stream registry so
@@ -1,3 +1,8 @@
import * as fs from 'node:fs';
import * as os from 'node:os';
import { join } from 'node:path';
import Fastify, { FastifyInstance } from 'fastify';
import fastifyStatic from '@fastify/static';
import { resolveStaticAssetHeaders } from './static.module';
// Unit tests for the static-asset cache classifier extracted from the
@@ -33,3 +38,69 @@ describe('resolveStaticAssetHeaders', () => {
expect(headers['vary']).toBe('Accept-Encoding');
});
});
// Integration test proving the ACTUAL response header emitted by @fastify/static
// with the exact registration options StaticModule uses. This is the regression
// guard for #452: without `cacheControl: false`, @fastify/static writes its own
// `Cache-Control: public, max-age=0` AFTER the setHeaders callback, overwriting
// the immutable header — the /assets/ assertion below would then fail.
describe('static.module @fastify/static registration (integration)', () => {
let app: FastifyInstance;
let tmpDir: string;
beforeAll(async () => {
tmpDir = fs.mkdtempSync(join(os.tmpdir(), 'static-module-spec-'));
fs.mkdirSync(join(tmpDir, 'assets'), { recursive: true });
fs.mkdirSync(join(tmpDir, 'locales'), { recursive: true });
fs.writeFileSync(
join(tmpDir, 'assets', 'index-a1b2c3.js'),
'console.log(1);',
);
fs.writeFileSync(join(tmpDir, 'locales', 'en.json'), '{"hello":"world"}');
app = Fastify();
// Mirror StaticModule.onModuleInit's registration options exactly.
await app.register(fastifyStatic, {
root: tmpDir,
wildcard: false,
preCompressed: true,
cacheControl: false,
setHeaders: (res, filePath) => {
for (const [name, value] of Object.entries(
resolveStaticAssetHeaders(filePath),
)) {
res.setHeader(name, value);
}
},
});
await app.ready();
});
afterAll(async () => {
await app.close();
fs.rmSync(tmpDir, { recursive: true, force: true });
});
it('serves a hashed /assets/ file with an immutable, 1-year cache-control', async () => {
const res = await app.inject({
method: 'GET',
url: '/assets/index-a1b2c3.js',
});
expect(res.statusCode).toBe(200);
const cacheControl = res.headers['cache-control'];
expect(cacheControl).toContain('immutable');
expect(cacheControl).toContain('max-age=31536000');
});
it('serves a non-hashed /locales/ file WITHOUT an immutable cache-control', async () => {
const res = await app.inject({ method: 'GET', url: '/locales/en.json' });
expect(res.statusCode).toBe(200);
// resolveStaticAssetHeaders sets no cache-control here and cacheControl:false
// stops @fastify/static from adding one, so the browser revalidates by
// etag/last-modified — either an absent header or one without `immutable`.
const cacheControl = res.headers['cache-control'];
if (cacheControl !== undefined) {
expect(cacheControl).not.toContain('immutable');
}
});
});
@@ -115,6 +115,11 @@ export class StaticModule implements OnModuleInit {
// Serve the build-time .br/.gz neighbour when the client accepts it
// (see vite-plugin-compression2 in apps/client/vite.config.ts).
preCompressed: true,
// @fastify/static's default cacheControl:true writes its own
// Cache-Control (from maxAge, default 0) AFTER the setHeaders callback,
// silently overwriting the immutable header that resolveStaticAssetHeaders
// sets — disable it so setHeaders/resolveStaticAssetHeaders own the header.
cacheControl: false,
setHeaders: (res, filePath) => {
for (const [name, value] of Object.entries(
resolveStaticAssetHeaders(filePath),
+15 -10
View File
@@ -155,6 +155,10 @@ All 41 tools, grouped by what you'd reach for them.
- **`update_page_json`** — Replace a page's entire content with a ProseMirror document
(bulk rewrites, or when nodes lack ids). `content` is optional — omit it to update only
the title. Keeps the block ids you pass in, so heading anchors and history stay stable.
- **`update_page_markdown`** — Replace a page's body (and optionally its title) with new
**plain Markdown**. The whole body is re-imported (block ids regenerate — for surgical or
id-preserving edits prefer `edit_page_text` / `patch_node` / `update_page_json`).
Docmost-flavoured markdown is parsed, including `^[...]` inline footnotes.
- **`docmost_transform`** — The agent-native editing interface: instead of retyping a
document, the agent **writes a function that fixes it**. Edit a page by running an
arbitrary **`(doc, ctx) => doc` JavaScript transform** against its *live* ProseMirror
@@ -184,13 +188,13 @@ All 41 tools, grouped by what you'd reach for them.
- **`export_page_markdown`** — Export a page to a single self-contained, **lossless
Docmost-flavoured Markdown** file: a meta header, the body with inline comment anchors
and diagrams, and a trailing comments-thread block. Built for a download → edit body →
`import_page_markdown` round-trip that preserves everything, including comment highlights.
- **`import_page_markdown`** — Replace a page's content from a Docmost-flavoured Markdown
file produced by `export_page_markdown`, restoring comment-highlight anchors and diagrams
from their inline HTML. (Comment *threads* in the file are not re-created on the server —
only the page body and inline comment marks are written; manage threads via the comment
tools/UI.)
and diagrams, and a trailing comments-thread block. To replace a page's body from plain
authoring Markdown, use `update_page_markdown`.
> **Removed in this release:** `import_page_markdown` (the round-trip parser for an
> exported Docmost-Markdown file) is **no longer exposed on the external MCP surface**.
> To replace a page's body from Markdown, use **`update_page_markdown`** (plain Markdown
> body replace). See the CHANGELOG for the migration note.
### Images
@@ -256,7 +260,8 @@ so capable clients steer the model automatically.
`delete_node`, addressing the node by its `attrs.id` from `get_page_json`.
- **Images**: `insert_image` / `replace_image`.
- **A new page**: `create_page`.
- **Bulk rewrite, or nodes without ids**: `update_page_json`.
- **Bulk rewrite, or nodes without ids**: `update_page_json` (ProseMirror) or
`update_page_markdown` (plain Markdown body replace).
- **Multi-step / scripted rewrite** (renumbering, footnotes, coordinated edits):
`docmost_transform` — preview with `dryRun`, then apply.
- **Copy a whole page's content from another page** (server-side): `copy_page_content`.
@@ -269,8 +274,8 @@ so capable clients steer the model automatically.
`get_node`.
- **Tables** (add/remove a row, set a cell): `table_get` / `table_insert_row` /
`table_delete_row` / `table_update_cell`.
- **Round-trip a page as Markdown** (download, edit, re-upload losslessly with comments):
`export_page_markdown` / `import_page_markdown`.
- **Export a page as self-contained Markdown** (with comment anchors): `export_page_markdown`.
- **Replace a page's body from Markdown**: `update_page_markdown`.
---
+15 -11
View File
@@ -161,6 +161,10 @@ Docmost-MCP не сочетают:
перезаписи или когда у узлов нет id). `content` опционален — опустите его, чтобы изменить
только заголовок. Сохраняет переданные id блоков, поэтому якоря заголовков и история
остаются стабильными.
- **`update_page_markdown`** — Заменить тело страницы (и опционально заголовок) новым
**обычным Markdown**. Всё тело переимпортируется (id блоков перегенерируются — для
хирургических правок или сохранения id используйте `edit_page_text` / `patch_node` /
`update_page_json`). Markdown в диалекте Docmost разбирается, включая inline-сноски `^[...]`.
- **`docmost_transform`** — Агентоориентированный интерфейс редактирования: вместо
перепечатывания документа агент **пишет функцию, которая его чинит**. Редактирует
страницу, запуская произвольный **JS-трансформ `(doc, ctx) => doc`** на её *живом*
@@ -189,14 +193,13 @@ Docmost-MCP не сочетают:
- **`export_page_markdown`** — Экспортировать страницу в один самодостаточный, **lossless
Markdown в диалекте Docmost**: мета-заголовок, тело с inline-якорями комментариев и
диаграммами и завершающий блок тредов комментариев. Рассчитан на цикл «скачать →
отредактировать тело → `import_page_markdown`», сохраняющий всё, включая выделения
комментариев.
- **`import_page_markdown`** — Заменить контент страницы из Markdown-файла в диалекте
Docmost, созданного `export_page_markdown`, восстанавливая якоря-выделения комментариев и
диаграммы из их inline-HTML. (Треды комментариев из файла не пересоздаются на сервере —
записываются только тело страницы и inline-марки комментариев; тредами управляйте через
инструменты/UI комментариев.)
диаграммами и завершающий блок тредов комментариев. Чтобы заменить тело страницы из
обычного авторского Markdown, используйте `update_page_markdown`.
> **Удалено в этом релизе:** `import_page_markdown` (парсер round-trip для
> экспортированного Docmost-Markdown-файла) **больше не отдаётся на внешней MCP-поверхности**.
> Чтобы заменить тело страницы из Markdown, используйте **`update_page_markdown`** (замена
> тела обычным Markdown). См. заметку о миграции в CHANGELOG.
### Изображения
@@ -263,7 +266,8 @@ Docmost-MCP не сочетают:
`delete_node`, адресуя узел по его `attrs.id` из `get_page_json`.
- **Изображения**: `insert_image` / `replace_image`.
- **Новая страница**: `create_page`.
- **Массовая перезапись или узлы без id**: `update_page_json`.
- **Массовая перезапись или узлы без id**: `update_page_json` (ProseMirror) или
`update_page_markdown` (замена тела обычным Markdown).
- **Многошаговая / скриптовая перезапись** (перенумерация, сноски, согласованные правки):
`docmost_transform` — предпросмотр через `dryRun`, затем применение.
- **Скопировать контент целой страницы из другой** (на стороне сервера):
@@ -278,8 +282,8 @@ Docmost-MCP не сочетают:
`get_node`.
- **Таблицы** (добавить/удалить строку, задать ячейку): `table_get` / `table_insert_row` /
`table_delete_row` / `table_update_cell`.
- **Round-trip страницы через Markdown** (скачать, отредактировать, залить обратно без
потерь, с комментариями): `export_page_markdown` / `import_page_markdown`.
- **Экспорт страницы в самодостаточный Markdown** (с якорями комментариев): `export_page_markdown`.
- **Заменить тело страницы из Markdown**: `update_page_markdown`.
---
+13 -5
View File
@@ -5,18 +5,26 @@
"private": true,
"type": "module",
"main": "./build/index.js",
"types": "./build/index.d.ts",
"exports": {
".": "./build/index.js",
"./http": "./build/http.js"
".": {
"types": "./build/index.d.ts",
"default": "./build/index.js"
},
"./http": {
"types": "./build/http.d.ts",
"default": "./build/http.js"
}
},
"bin": {
"docmost-mcp": "./build/stdio.js"
},
"scripts": {
"build": "tsc",
"gen:stamp": "node scripts/gen-registry-stamp.mjs",
"build": "node scripts/gen-registry-stamp.mjs && tsc",
"start": "node build/stdio.js",
"watch": "tsc --watch",
"pretest": "tsc",
"watch": "node scripts/gen-registry-stamp.mjs && tsc --watch",
"pretest": "node scripts/gen-registry-stamp.mjs && tsc",
"test": "node --test \"test/unit/*.test.mjs\" \"test/mock/*.test.mjs\"",
"test:unit": "node --test \"test/unit/*.test.mjs\"",
"test:mock": "node --test \"test/mock/*.test.mjs\"",
@@ -0,0 +1,67 @@
// Codegen: emit src/registry-stamp.generated.ts with a REGISTRY_STAMP hash of
// the tool-specs REGISTRY CONTENT, so a build/ vs src/ skew (issue #447) is
// detectable at runtime.
//
// WHY hash the raw source text (not extracted structured data):
// SHARED_TOOL_SPECS carries `buildShape` functions (the input SCHEMAS) which are
// NOT serializable. The input schema is exactly one of the things that MUST stay
// in sync between build/ and src/, so we cannot drop it from the hash. Rather
// than probe zod with a fragile shim to reconstruct the schema shape, we hash the
// STABLE, deterministic source TEXT of tool-specs.ts. That text fully captures
// every field that must stay in sync — mcpName, inAppKey, description, tier,
// catalogLine AND the buildShape bodies (input schemas) — with zero probing
// fragility. Any edit to a spec (a renamed tool, a reworded description, a
// changed schema field) changes the text and therefore the stamp.
//
// DETERMINISM: the hash is computed over the file bytes with line endings
// normalized to LF and a single trailing newline stripped, so a CRLF checkout or
// an editor's trailing-newline habit cannot make build/ and src/ disagree. No
// Date.now / randomness. The loader's dev-only stale-check (docmost-client.loader.ts)
// re-runs THIS SAME normalization + sha256 over src/tool-specs.ts and compares to
// the built REGISTRY_STAMP; the two must compute identically.
//
// This script runs from the `build` and `pretest` npm scripts BEFORE tsc, so
// build/ always carries a stamp derived from the tool-specs.ts that was compiled.
import { createHash } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const SRC_DIR = join(__dirname, '..', 'src');
const TOOL_SPECS_PATH = join(SRC_DIR, 'tool-specs.ts');
const OUT_PATH = join(SRC_DIR, 'registry-stamp.generated.ts');
/**
* Deterministic stamp of the tool-specs registry content. Kept as a plain
* function (exported) so the algorithm has a single home; the loader duplicates
* only the tiny normalize+sha256 steps because it lives in the CJS server build
* and cannot import this ESM script. If you change the normalization here, mirror
* it in apps/server/src/core/ai-chat/tools/docmost-client.loader.ts.
*/
export function computeRegistryStamp(toolSpecsSource) {
const normalized = toolSpecsSource.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
}
function main() {
const source = readFileSync(TOOL_SPECS_PATH, 'utf8');
const stamp = computeRegistryStamp(source);
const out =
'// AUTO-GENERATED by scripts/gen-registry-stamp.mjs — DO NOT EDIT BY HAND.\n' +
'// A deterministic hash of src/tool-specs.ts content (tool names, descriptions,\n' +
'// tiers, catalog lines and input schemas). Regenerated on every build/pretest\n' +
'// so build/ always matches the compiled src. The in-app loader recomputes this\n' +
'// from src and refuses to run on a mismatch (issue #447). This file is\n' +
'// gitignored and produced by the build — see .gitignore.\n' +
`export const REGISTRY_STAMP = ${JSON.stringify(stamp)};\n`;
writeFileSync(OUT_PATH, out, 'utf8');
// eslint-disable-next-line no-console
console.log(`gen-registry-stamp: wrote ${OUT_PATH} (${stamp.slice(0, 12)}…)`);
}
// Only run when invoked directly (not when imported for computeRegistryStamp).
if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
main();
}
+210 -78
View File
@@ -42,6 +42,7 @@ import {
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import { searchInDoc, SearchOptions } from "./lib/page-search.js";
import { withPageLock } from "./lib/page-lock.js";
@@ -729,16 +730,18 @@ export class DocmostClient {
// forever and accumulate duplicates).
const MAX_PAGES = 50;
let page = 1;
let cursor: string | undefined;
let allItems: T[] = [];
let hasNextPage = true;
let truncated = false;
while (hasNextPage && page <= MAX_PAGES) {
const response = await this.client.post(endpoint, {
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = {
...basePayload,
limit: clampedLimit,
page,
});
};
if (cursor) payload.cursor = cursor;
const response = await this.client.post(endpoint, payload);
const data = response.data;
const items = data.data?.items || data.items || [];
@@ -746,22 +749,28 @@ export class DocmostClient {
allItems = allItems.concat(items);
// Stop if the page is empty or shorter than the requested size: a full
// page worth of items is the only situation where another page can exist,
// so this defends against a stuck hasNextPage flag in addition to it.
if (items.length === 0 || items.length < clampedLimit) {
// Advance strictly via the server-issued cursor. A missing nextCursor (or
// hasNextPage false) means we reached the end. A cursor identical to the
// one we just sent means the server did not understand our pagination
// param — stop instead of re-fetching page one forever and duplicating.
const next = meta?.hasNextPage ? meta?.nextCursor : null;
if (!next || next === cursor) {
// If the server still reports more pages but stopped issuing a usable
// cursor at the ceiling, flag the result as truncated below.
if (page === MAX_PAGES - 1 && meta?.hasNextPage) truncated = true;
break;
}
cursor = next;
hasNextPage = meta?.hasNextPage || false;
page++;
// Reaching the ceiling with more pages still available means the result
// set is truncated.
if (page === MAX_PAGES - 1) truncated = true;
}
// If the loop stopped because it hit the MAX_PAGES ceiling while the server
// still reported more results (hasNextPage true and the last page was
// full), the result set is truncated — warn so the caller is not silently
// handed an incomplete list.
if (hasNextPage && page > MAX_PAGES) {
// still reported more results, the result set is truncated — warn so the
// caller is not silently handed an incomplete list.
if (truncated) {
console.warn(
`paginateAll: results from "${endpoint}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
@@ -795,9 +804,10 @@ export class DocmostClient {
* Tree (`tree` true): the space's FULL page hierarchy as a nested tree (each
* node has a `children` array). This mode REQUIRES `spaceId` (a page tree is
* scoped to one space) and IGNORES `limit` the whole hierarchy is returned.
* It walks the sidebar tree via `enumerateSpacePages`, which performs N
* sidebar requests and is bounded by that method's 10000-node cap (and skips
* soft-deleted pages server-side).
* It fetches the tree via `enumerateSpacePages`, which on the fork server
* resolves to a single `/pages/tree` request returning the whole
* permission-filtered flat page set (soft-deleted pages excluded
* server-side).
*/
async listPages(spaceId?: string, limit: number = 50, tree: boolean = false) {
await this.ensureAuthenticated();
@@ -808,8 +818,8 @@ export class DocmostClient {
"list_pages: tree mode requires a spaceId (a page tree is scoped to one space). Pass spaceId, or omit tree to get the recent-pages list.",
);
}
const nodes = await this.enumerateSpacePages(spaceId);
return buildPageTree(nodes);
const { pages } = await this.enumerateSpacePages(spaceId);
return buildPageTree(pages);
}
const clampedLimit = Math.max(1, Math.min(100, limit));
@@ -831,57 +841,123 @@ export class DocmostClient {
async listSidebarPages(spaceId: string, pageId?: string) {
await this.ensureAuthenticated();
// Paginate: the endpoint returns server-paged children, so posting only
// { page: 1 } silently dropped every child beyond the first page. Loop on
// meta.hasNextPage (with a MAX_PAGES ceiling like paginateAll, guarding
// against a stuck hasNextPage flag) and accumulate all children.
// Paginate via the server-issued cursor. The server switched from OFFSET
// (`page`) to CURSOR (`cursor`/`nextCursor`) pagination, and the global
// ValidationPipe(whitelist:true) SILENTLY STRIPS the obsolete `page` field
// — so the old offset loop got the SAME first page every time (with
// hasNextPage stuck true) and dropped every child beyond the first page.
const MAX_PAGES = 50;
let page = 1;
let cursor: string | undefined;
let allItems: any[] = [];
let hasNextPage = true;
let truncated = false;
while (hasNextPage && page <= MAX_PAGES) {
for (let i = 0; i < MAX_PAGES; i++) {
// limit: 100 is the server-side Max; cuts request count 5x vs the default 20.
const payload: Record<string, any> = { spaceId, limit: 100 };
// Only send pageId when scoping to a page's children; omit it for roots.
const payload: Record<string, any> = { spaceId, page };
if (pageId) payload.pageId = pageId;
if (cursor) payload.cursor = cursor;
const response = await this.client.post("/pages/sidebar-pages", payload);
const data = response.data?.data ?? response.data;
const items = data?.items || [];
allItems = allItems.concat(items);
const data = (await this.client.post("/pages/sidebar-pages", payload)).data
?.data;
allItems = allItems.concat(data?.items ?? []);
hasNextPage = data?.meta?.hasNextPage || false;
page++;
// Advance strictly via the server-issued cursor; a missing/repeated cursor
// means the protocol drifted again — stop instead of looping on page one.
const next = data?.meta?.hasNextPage ? data?.meta?.nextCursor : null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with more pages still available means the child
// list is truncated (mirrors paginateAll).
if (i === MAX_PAGES - 1) truncated = true;
}
// Warn on real truncation (ceiling hit while the server still had pages) so
// the caller is not silently handed an incomplete child list.
if (truncated) {
console.warn(
`listSidebarPages: children of "${pageId ?? spaceId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
return allItems;
}
/**
* Enumerate EVERY page in a space (or in a subtree, when rootPageId is given)
* by walking the sidebar-pages tree.
* Enumerate EVERY page in a space (or in a subtree, when rootPageId is given).
*
* Starting set: the children of rootPageId when provided, otherwise the
* space root pages. From there it does an iterative breadth-first walk: each
* node is collected, and when node.hasChildren is true its direct children
* are fetched via listSidebarPages(spaceId, node.id) and enqueued.
* Primary path (fork server): a SINGLE `POST /pages/tree` returns the whole
* space (or a subtree) as a flat, permission-filtered list in one request, in
* the exact node shape buildPageTree consumes. This replaces the old
* per-node BFS, which issued N sidebar requests and after the server moved
* to cursor pagination silently lost every child past the first sidebar
* page (the obsolete `page` param was stripped by ValidationPipe).
*
* This replaces the old "/pages/recent" enumeration, which is a bounded
* recent-activity feed (~5000 cap) and therefore misses comments on older
* pages that were never recently touched.
* The subtree variant (rootPageId given) INCLUDES the root node itself
* (getPageAndDescendants seeds with id = rootPageId), unlike the old BFS
* which started from the root's children.
*
* Safeguards: a `visited` Set of page ids prevents re-processing a node
* (cycles / duplicate references), and a hard node cap bounds pathological
* trees so the walk always terminates.
* Fallback path (stdio mode may target STOCK upstream Docmost, which lacks
* `/pages/tree`): on a 404/405 it falls back to the cursor-based BFS below,
* walking direct children via the fixed cursor listSidebarPages. Safeguards:
* a `visited` Set of page ids prevents re-processing a node (cycles /
* duplicate references), and a hard node cap bounds pathological trees so the
* walk always terminates.
*
* Returns `{ pages, truncated }`. `truncated` is true ONLY when the fallback
* BFS stopped at its MAX_NODES cap the primary /pages/tree path is uncapped
* and always returns the complete set, so it never reports truncation.
*/
private async enumerateSpacePages(
spaceId: string,
rootPageId?: string,
): Promise<any[]> {
): Promise<{ pages: any[]; truncated: boolean }> {
await this.ensureAuthenticated();
// Single request replaces the whole BFS: /pages/tree returns the full
// permission-filtered flat page set of a space (or a subtree) at once. This
// path is uncapped, so it is never truncated.
const payload = rootPageId ? { pageId: rootPageId } : { spaceId };
try {
const response = await this.client.post("/pages/tree", payload);
const pages = (response.data?.data ?? response.data)?.items ?? [];
return { pages, truncated: false };
} catch (e: any) {
// Only fall back when the endpoint is absent (stock upstream Docmost);
// any other error is a genuine failure and must propagate.
if (
!axios.isAxiosError(e) ||
(e.response?.status !== 404 && e.response?.status !== 405)
) {
throw e;
}
}
// Fallback: cursor-based breadth-first walk via listSidebarPages.
const MAX_NODES = 10000;
const result: any[] = [];
const visited = new Set<string>();
// Seed with the root node itself when scoping to a subtree, so its own
// comments aren't dropped: the primary /pages/tree seeds
// getPageAndDescendants with id = rootPageId (root included), but
// listSidebarPages(spaceId, rootPageId) returns only the root's CHILDREN.
// The `visited` set below prevents a double-add if the root also appears
// among the children. getPageRaw returns a page whose id/title/spaceId are
// exactly what buildPageTree and check_new_comments consume.
if (rootPageId) {
try {
const root = await this.getPageRaw(rootPageId);
if (root?.id) {
result.push(root);
visited.add(root.id);
}
} catch {
// Non-fatal: if the root can't be read, fall through to children-only.
}
}
// Seed the queue with the starting level (subtree children or roots).
const queue: any[] = await this.listSidebarPages(spaceId, rootPageId);
@@ -906,7 +982,12 @@ export class DocmostClient {
}
}
return result;
// Truncated only when the cap was hit with the queue still non-empty (real
// truncation, not a natural end at exactly MAX_NODES).
return {
pages: result,
truncated: result.length >= MAX_NODES && queue.length > 0,
};
}
/** Raw page info including the ProseMirror JSON content and slugId. */
@@ -1756,6 +1837,27 @@ export class DocmostClient {
}
}
/**
* Pre-write SHAPE gate (#409). Walk the WHOLE node tree with the shared
* `findInvalidNode` and throw a rich, path-anchored error the instant a nested
* node has an absent/unknown `type` (or an unknown mark) the exact shape that
* otherwise surfaces DEEP in the Yjs encode as the cryptic
* `Unknown node type: undefined`, but only AFTER a collab session was opened
* and a page lock taken. Calling this BEFORE `getCollabTokenWithReauth` /
* `mutatePageContent` fails fast: no collab connection, no lock, deterministic
* message. `op` names the tool for the message prefix (e.g. "patch_node").
*
* `findInvalidNode` derives its "known type" set from the very same
* `docmostExtensions` the encode path uses, so a node this gate accepts is one
* the encoder will accept too.
*/
private assertValidNodeShape(op: string, node: any): void {
const bad = findInvalidNode(node);
if (bad) {
throw new Error(`${op}: invalid node — ${bad.summary}`);
}
}
/**
* Replace page content with a raw ProseMirror JSON document (lossless) and/or
* update its title. Both `doc` and `title` are optional, but at least one must
@@ -1807,6 +1909,12 @@ export class DocmostClient {
// page on overwrite.
this.validateDocStructure(doc);
// #409: beyond the string-`type` check above, reject a nested node whose
// `type` is a string but NOT a known Docmost schema node (a typo/unknown
// block) — the same `Unknown node type` the encoder throws — with a rich,
// path-anchored message, still BEFORE any collab connection.
this.assertValidNodeShape("update_page_json", doc);
// Sanitize URLs before writing. This closes the JSON-path bypass: unlike
// the markdown link path (which TipTap sanitizes), raw JSON could otherwise
// inject javascript:/data: link hrefs or media srcs straight into the doc.
@@ -2227,6 +2335,14 @@ export class DocmostClient {
target.attrs.id = nodeId;
}
// #409: fail fast on a malformed node SHAPE (a nested child with an
// absent/unknown `type`, e.g. a text leaf written as `{"text":"foo"}` with
// no `"type":"text"`) BEFORE opening a collab session or taking the page
// lock — the root-only `typeof node.type === "string"` check above never
// sees nested children, and the encoder's `Unknown node type: undefined`
// would otherwise only surface after the connection.
this.assertValidNodeShape("patch_node", target);
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
@@ -2317,6 +2433,11 @@ export class DocmostClient {
}
}
// #409: fail fast on a malformed node SHAPE (a nested child with an
// absent/unknown `type`) BEFORE opening a collab session or taking the page
// lock — the root-only check above never sees nested children.
this.assertValidNodeShape("insert_node", node);
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
@@ -2540,7 +2661,13 @@ export class DocmostClient {
let allComments: any[] = [];
let cursor: string | null = null;
do {
// Hard ceiling + immovable-cursor guard (mirrors paginateAll): if /comments
// ever stops advancing the cursor (the exact #442 drift scenario) this loop
// would otherwise spin forever accumulating duplicates.
const MAX_PAGES = 50;
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = { pageId, limit: 100 };
if (cursor) payload.cursor = cursor;
@@ -2548,8 +2675,23 @@ export class DocmostClient {
const data = response.data.data || response.data;
const items = data.items || [];
allComments = allComments.concat(items);
cursor = data.meta?.nextCursor || null;
} while (cursor);
// Advance strictly via the server-issued cursor. A missing nextCursor or a
// cursor identical to the one we just sent means the end (or a server that
// ignores our pagination param) — stop instead of re-fetching page one.
const next: string | null = data.meta?.nextCursor || null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with a still-advancing cursor means truncation.
if (page === MAX_PAGES - 1) truncated = true;
}
if (truncated) {
console.warn(
`listComments: comments for "${pageId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
const mapped = allComments.map((comment: any) => {
const markdown = comment.content
@@ -3036,36 +3178,27 @@ export class DocmostClient {
);
}
// 1. Enumerate the FULL set of pages in scope by walking the sidebar-pages
// tree (a complete page index), NOT the bounded "/pages/recent" feed which
// caps at ~5000 recent items and silently misses comments on older pages.
// 1. Enumerate the FULL set of pages in scope via the page tree (a complete
// page index), NOT the bounded "/pages/recent" feed which caps at ~5000
// recent items and silently misses comments on older pages.
//
// Subtree scope: when parentPageId is given, the scope is that page ITSELF
// plus every descendant (enumerateSpacePages walks its children). Otherwise
// the scope is the whole space (all roots and their descendants).
// plus every descendant. Otherwise the scope is the whole space (all roots
// and their descendants).
//
// NOTE: do NOT pre-filter by page.updatedAt — creating a comment does not
// bump it (verified on a live server), so such a filter silently misses
// comments on pages that were not otherwise edited. The complete tree walk
// already restricts the scope correctly, so no recent-feed allow-list is
// needed any more.
let pagesInScope: any[];
if (parentPageId) {
const subtree = await this.enumerateSpacePages(spaceId, parentPageId);
// Include the parent page node itself alongside its descendants. Fetch it
// so its title/id are available even though it is not returned by its own
// children listing.
let parentNode: any = { id: parentPageId };
try {
parentNode = await this.getPageRaw(parentPageId);
} catch (e: any) {
// Fall back to a minimal node if the parent can't be fetched; its
// comments are still attempted below (the fetch there is non-fatal).
}
pagesInScope = [parentNode, ...subtree];
} else {
pagesInScope = await this.enumerateSpacePages(spaceId);
}
//
// The subtree scope (parentPageId given) already INCLUDES the root node
// itself: /pages/tree seeds getPageAndDescendants with id = parentPageId, so
// no separate getPageRaw fetch for the parent is needed.
const { pages: pagesInScope, truncated } = await this.enumerateSpacePages(
spaceId,
parentPageId,
);
// 2. Fetch comments for each page, keep ones created after since
const results: any[] = [];
@@ -3094,10 +3227,9 @@ export class DocmostClient {
0,
);
// enumerateSpacePages caps traversal at 10000 nodes; flag when that cap was
// hit so the caller knows the scan may be incomplete (some pages skipped).
const truncated = pagesInScope.length >= 10000;
// `truncated` is reported by enumerateSpacePages: it is true ONLY when the
// stdio fallback BFS hit its node cap. The primary /pages/tree path is
// uncapped, so a space with legitimately many pages is not falsely flagged.
return {
since,
scope: parentPageId ? `subtree of ${parentPageId}` : `space ${spaceId}`,
+61 -543
View File
@@ -6,6 +6,7 @@ import { dirname, join } from "path";
import { DocmostClient, DocmostMcpConfig } from "./client.js";
import { parseNodeArg } from "@docmost/prosemirror-markdown";
import { SHARED_TOOL_SPECS, SharedToolSpec } from "./tool-specs.js";
import { SERVER_INSTRUCTIONS } from "./server-instructions.js";
import {
createCommentSignalTracker,
CommentSignalTracker,
@@ -29,6 +30,14 @@ export { destroyAllSessions } from "./lib/collab-session.js";
export { SHARED_TOOL_SPECS } from "./tool-specs.js";
export type { SharedToolSpec } from "./tool-specs.js";
// Re-export the build-time REGISTRY_STAMP (issue #447): a deterministic hash of
// the tool-specs registry content, generated into src/registry-stamp.generated.ts
// by scripts/gen-registry-stamp.mjs BEFORE tsc, so it lands in build/. The in-app
// loader recomputes the same hash from src/tool-specs.ts (dev/test only) and
// refuses to run on a mismatch, catching a build/ vs src/ skew (a spec edited in
// src without rebuilding the package the server actually loads from build/).
export { REGISTRY_STAMP } from "./registry-stamp.generated.js";
// Re-export the shared "new comments: N" signal helper (#417) so the in-app
// layer reads the SAME watermark/debounce/injection-safe line builder off the
// loaded module (same pattern as SHARED_TOOL_SPECS). Both surfaces then differ
@@ -66,19 +75,12 @@ const VERSION = packageJson.version;
// Editing guide surfaced to MCP clients in the initialize result so they can
// pick the right tool by intent and avoid resending whole documents.
//
// MAINTENANCE RULE: when you ADD, RENAME, or REMOVE a tool (either an inline
// server.registerTool(...) here or a spec in tool-specs.ts), you MUST update
// this guide so the new tool is routed by intent. This is enforced by
// test/unit/server-instructions.test.mjs, which fails when a registered tool
// name is not mentioned below (see its EXCEPTIONS list for the rare opt-outs).
// Exported for that test.
export const SERVER_INSTRUCTIONS =
"Docmost editing guide — choose the tool by intent.\n" +
"READ: find a page -> search (workspace-wide full-text); list -> list_pages / list_spaces. Locate blocks and their ids CHEAPLY -> get_outline (compact top-level map; start here, not get_page_json). One block's subtree -> get_node (by attrs.id, or \"#<index>\" for tables, which carry no id). Find every occurrence of a string/regex ON a page (and where each is) -> search_in_page, NOT block-by-block get_node — it returns each hit's node ref + block index + context for a targeted comment. Whole page -> get_page (Markdown, lossy; inline <span data-comment-id> tags are comment anchors — markup, not text) or get_page_json (lossless ProseMirror with block ids). Hand a huge page (with images) to an external consumer without pulling it through the model context -> stash_page (returns a short-lived anonymous URL).\n" +
"EDIT: fix wording/typos/numbers -> edit_page_text (find/replace inside blocks, no node id needed). Change ONE block (paragraph/heading/callout/etc.) structurally -> patch_node (by attrs.id from get_outline). Add a block -> insert_node (before/after a block by attrs.id or by anchor text, or append). Remove a block -> delete_node (by attrs.id). Tables -> table_get / table_update_cell / table_insert_row / table_delete_row (address by \"#<index>\" from get_outline; table nodes have no attrs.id). Images -> insert_image (add from a web URL) / replace_image (swap an existing image). Draw.io diagrams -> drawio_create (create from mxGraph XML and insert), drawio_get (read a diagram as mxGraph XML + a hash), drawio_update (replace a diagram; pass the hash from drawio_get as baseHash for optimistic locking). Footnotes -> insert_footnote. Bulk/structural rewrite -> update_page_json (full ProseMirror replace; prefer the granular tools above to avoid resending the whole ~100KB+ document). Complex/scripted rewrite (multiple coordinated edits, renumbering) -> docmost_transform: write a JS `(doc, ctx) => doc` transform, preview the diff with dryRun (default), then apply with dryRun:false; ctx.helpers includes commentsToFootnotes for turning inline comments into numbered footnotes.\n" +
"PAGES: new -> create_page (Markdown). Rename (title only) -> rename_page. Move -> move_page. Delete -> delete_page (SOFT delete — the page goes to trash and is restorable; nothing is permanent). Copy/replace a page's whole content from another page (server-side, no document through the model) -> copy_page_content. Sharing -> share_page / unshare_page / list_shares; share_page makes the page PUBLICLY accessible — do it only when explicitly asked.\n" +
"COMMENTS: create_comment is always inline and requires an EXACT selection — contiguous text from a single block, <=250 chars (fails rather than leaving an unanchored comment); reply to a thread via parentCommentId. Propose a concrete text fix for one-click human approval -> create_comment with suggestedText (the exact plain-text replacement for the selection; the selection must then be UNIQUE in the page — extend it with context if needed); prefer this over editing directly when the change is subjective or needs the author's sign-off. Manage -> list_comments, update_comment, resolve_comment (resolve/reopen, reversible — prefer over delete to close), delete_comment, check_new_comments.\n" +
"HISTORY: review what changed -> diff_page_versions (a historyId vs current, or two versions). List saved versions -> list_page_history. Undo a bad edit -> restore_page_version (writes a past version back as current; itself revertible). Lossless markdown round-trip (download, edit, re-upload, incl. comment anchors) -> export_page_markdown / import_page_markdown.";
// The guide is now SPLIT (issue #448): the hand-written routing prose lives in
// server-instructions.ts and the tool INVENTORY is GENERATED from the registry
// (SHARED_TOOL_SPECS + INLINE_MCP_INVENTORY), so it can no longer drift out of
// sync with the registered tools. Re-exported here (its old home) so existing
// importers are unaffected; the composition lives in server-instructions.ts.
export { SERVER_INSTRUCTIONS };
// Helper to format JSON responses
const jsonContent = (data: any) => ({
@@ -194,10 +196,10 @@ export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
{ instructions: SERVER_INSTRUCTIONS },
);
// Single choke point for MCP tool timing. Both `registerShared` (below) and
// the inline `server.registerTool(...)` calls funnel through this one method,
// so monkeypatching it HERE — before any tool is registered and before
// `registerShared` captures a reference to it — times every tool with no
// Single choke point for MCP tool timing. Both `registerSharedFromSpec` (below)
// and the inline `server.registerTool(...)` calls funnel through this one
// method, so monkeypatching it HERE — before any tool is registered and before
// the registry loop captures a reference to it — times every tool with no
// per-tool boilerplate. The wrapped handler records wall-clock duration and,
// in a `finally`, feeds the host's dependency-neutral sink
// `config.onMetric("mcp_tool_duration_seconds", seconds, { tool })`. The tool
@@ -253,92 +255,56 @@ export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
return originalRegisterTool(...args.slice(0, -1), signalledHandler);
};
// Register a tool from the shared, zod-agnostic spec registry. The spec owns
// the canonical name + model-facing description + (optional) schema builder;
// only the execute body is supplied per call. buildShape is invoked with THIS
// package's zod (v3); the in-app layer passes its own zod (v4).
//
// The spec's schema builder returns a plain ZodRawShape (Record<string,
// unknown> in the shared module since it must stay zod-agnostic), so the
// McpServer.registerTool overloads cannot infer the execute arg's shape from
// it. We type `execute` loosely and cast the call through `any`; runtime
// behaviour is unchanged — each execute body destructures the same fields the
// builder declares.
const registerShared = (
spec: SharedToolSpec,
execute: (args: any) => Promise<{ content: { type: "text"; text: string }[] }>,
) =>
(server.registerTool as any)(
// Register EVERY shared tool from the zod-agnostic registry in one loop (#445).
// The spec owns the canonical name + description + (optional) schema builder AND
// the canonical execute mapping; the host only supplies the RESULT ENVELOPE. For
// each spec:
// - skip `inAppOnly` specs (they belong to the in-app host only);
// - if the spec has an `mcpExecute` override (a deliberate per-layer
// difference — a guardrail, an omitted param, or a non-JSON envelope like a
// resource_link/bare success line), the override OWNS the full MCP content
// result and is used VERBATIM;
// - otherwise the canonical `execute` returns RAW data and this host wraps it
// in the standard JSON text envelope (jsonContent), exactly as the old inline
// bodies did.
// buildShape is invoked with THIS package's zod (v3); the in-app layer passes its
// own zod (v4). The registry's execute returns `unknown` (it is zod-agnostic), so
// the wrapping is typed loosely and cast — runtime behaviour is unchanged.
const registerSharedFromSpec = (spec: SharedToolSpec) => {
if (spec.inAppOnly) return;
const handler = async (args: any) => {
if (spec.mcpExecute) {
// The override owns the full MCP result envelope (not re-wrapped).
return (await spec.mcpExecute(docmostClient, args)) as {
content: { type: "text"; text: string }[];
};
}
// Canonical execute returns raw data; wrap it as JSON text content.
const raw = await spec.execute!(docmostClient, args);
return jsonContent(raw);
};
return (server.registerTool as any)(
spec.mcpName,
spec.buildShape
? { description: spec.description, inputSchema: spec.buildShape(z) }
: { description: spec.description },
execute,
handler,
);
};
// Tool: get_workspace
registerShared(SHARED_TOOL_SPECS.getWorkspace, async () => {
const workspace = await docmostClient.getWorkspace();
return jsonContent(workspace);
});
for (const spec of Object.values(SHARED_TOOL_SPECS)) {
registerSharedFromSpec(spec as SharedToolSpec);
}
// Tool: list_spaces
registerShared(SHARED_TOOL_SPECS.listSpaces, async () => {
const spaces = await docmostClient.getSpaces();
return jsonContent(spaces);
});
// --- INLINE tools kept per-transport (NOT in the shared registry) ---
// Each stays inline for a documented reason: a snake_case/camelCase naming
// clash the registry convention forbids (table_get), an intentional
// per-transport behaviour/schema divergence (search, docmost_transform), or a
// tool that exists ONLY on this standalone MCP surface (update_comment,
// delete_comment — the in-app agent deliberately exposes no hard comment
// edit/delete tool).
// Tool: list_pages
// INTENTIONAL per-transport divergence (not in the shared registry): this
// transport exposes a `tree:true` mode that returns the full nested hierarchy;
// the in-app copy keeps the same tree option but is worded for the in-app agent.
// Kept per-layer so each side can tune its own guidance.
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294). This
// transport keeps applying its own defaults (limit=50, tree=false) in execute.
registerShared(SHARED_TOOL_SPECS.listPages, async ({ spaceId, limit, tree }) => {
const result = await docmostClient.listPages(spaceId, limit ?? 50, tree ?? false);
return jsonContent(result);
});
// Tool: get_page
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.getPage, async ({ pageId }) => {
const page = await docmostClient.getPage(pageId);
return jsonContent(page);
});
// Tool: get_page_json
registerShared(SHARED_TOOL_SPECS.getPageJson, async ({ pageId }) => {
const page = await docmostClient.getPageJson(pageId);
return jsonContent(page);
});
// Tool: get_outline
registerShared(SHARED_TOOL_SPECS.getOutline, async ({ pageId }) => {
const result = await docmostClient.getOutline(pageId);
return jsonContent(result);
});
// Tool: get_node
registerShared(SHARED_TOOL_SPECS.getNode, async ({ pageId, nodeId }) => {
const result = await docmostClient.getNode(pageId, nodeId);
return jsonContent(result);
});
// Tool: search_in_page
registerShared(
SHARED_TOOL_SPECS.searchInPage,
async ({ pageId, query, regex, caseSensitive, limit }) => {
const result = await docmostClient.searchInPage(pageId, query, {
regex,
caseSensitive,
limit,
});
return jsonContent(result);
},
);
// Tool: table_get
// Tool: table_get
// NOT in the shared registry: the MCP tool name `table_get` is noun-first while
// the in-app key is `getTable` (verb-first), breaking the snake_case(inAppKey)
// convention the shared registry enforces (shared-tool-specs.contract.spec.ts).
@@ -364,383 +330,6 @@ server.registerTool(
},
);
// Tool: table_insert_row
// Schema + description now live in the shared registry (#294); the `table`
// parameter name is the canonical one (the in-app layer was unified to it).
registerShared(
SHARED_TOOL_SPECS.tableInsertRow,
async ({ pageId, table, cells, index }) => {
const result = await docmostClient.tableInsertRow(
pageId,
table,
cells,
index,
);
return jsonContent(result);
},
);
// Tool: table_delete_row
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.tableDeleteRow,
async ({ pageId, table, index }) => {
const result = await docmostClient.tableDeleteRow(pageId, table, index);
return jsonContent(result);
},
);
// Tool: table_update_cell
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.tableUpdateCell,
async ({ pageId, table, row, col, text }) => {
const result = await docmostClient.tableUpdateCell(
pageId,
table,
row,
col,
text,
);
return jsonContent(result);
},
);
// Tool: create_page
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.createPage,
async ({ title, content, spaceId, parentPageId }) => {
const result = await docmostClient.createPage(
title,
content,
spaceId,
parentPageId,
);
return jsonContent(result);
},
);
// Tool: update_page_json
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's content normalization (parse a JSON-string content,
// pass undefined/null through for a title-only/no-op update).
registerShared(
SHARED_TOOL_SPECS.updatePageJson,
async ({ pageId, content, title }) => {
// Only parse/validate the document when it was actually supplied; when it
// is omitted, pass it straight through so the client performs a title-only
// (or no-op) update.
let doc;
if (content === undefined || content === null) {
doc = undefined;
} else {
// String -> JSON.parse (throwing on invalid); object passes through.
doc = parseNodeArg(content, "content was a string but not valid JSON");
}
const result = await docmostClient.updatePageJson(pageId, doc, title);
return jsonContent(result);
},
);
// Tool: export_page_markdown
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.exportPageMarkdown, async ({ pageId }) => {
const md = await docmostClient.exportPageMarkdown(pageId);
return { content: [{ type: "text" as const, text: md }] };
});
// Tool: import_page_markdown
registerShared(
SHARED_TOOL_SPECS.importPageMarkdown,
async ({ pageId, markdown }) => {
const res = await docmostClient.importPageMarkdown(pageId, markdown);
return jsonContent(res);
},
);
// Tool: copy_page_content
registerShared(
SHARED_TOOL_SPECS.copyPageContent,
async ({ sourcePageId, targetPageId }) => {
const result = await docmostClient.copyPageContent(
sourcePageId,
targetPageId,
);
return jsonContent(result);
},
);
// Tool: rename_page
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.renamePage, async ({ pageId, title }) => {
const result = await docmostClient.renamePage(pageId, title);
return jsonContent(result);
});
// Tool: edit_page_text
registerShared(SHARED_TOOL_SPECS.editPageText, async ({ pageId, edits }) => {
const result = await docmostClient.editPageText(pageId, edits);
return jsonContent(result);
});
// Tool: stash_page — returns a resource_link (NOT embedded text) so the doc
// body never enters the model context. Registered directly (not via
// registerShared) because that helper only emits text content. Also returns
// `structuredContent` carrying the full documented `{uri, sha256, size, images}`
// shape alongside the resource_link, so MCP clients receive the blob's sha256
// (its ETag, for integrity) and mirror counts, not just the link.
server.registerTool(
SHARED_TOOL_SPECS.stashPage.mcpName,
{
description: SHARED_TOOL_SPECS.stashPage.description,
inputSchema: SHARED_TOOL_SPECS.stashPage.buildShape!(z),
},
async ({ pageId }: { pageId: string }) => {
const result = await docmostClient.stashPage(pageId);
return {
content: [
{
type: "resource_link" as const,
uri: result.uri,
name: "page.json",
mimeType: "application/json",
size: result.size,
},
],
// Mirror the full documented result shape ({ uri, size, sha256, images })
// as structuredContent so MCP clients get the blob's sha256 (its ETag, for
// integrity) and the mirror counts, not just the resource_link.
structuredContent: {
uri: result.uri,
sha256: result.sha256,
size: result.size,
images: result.images,
},
};
},
);
// Tool: patch_node — schema + description from the shared registry (identical
// across both transports). The execute body keeps its own parseNodeArg
// normalization (the model sometimes serializes `node` as a JSON string).
registerShared(
SHARED_TOOL_SPECS.patchNode,
async ({ pageId, nodeId, node }) => {
const parsedNode = parseNodeArg(node);
const result = await docmostClient.patchNode(pageId, nodeId, parsedNode);
return jsonContent(result);
},
);
// Tool: insert_node — schema + description from the shared registry. As with
// patch_node, the execute body retains parseNodeArg on the incoming node.
registerShared(
SHARED_TOOL_SPECS.insertNode,
async ({ pageId, node, position, anchorNodeId, anchorText }) => {
const parsedNode = parseNodeArg(node);
const result = await docmostClient.insertNode(pageId, parsedNode, {
position,
anchorNodeId,
anchorText,
});
return jsonContent(result);
},
);
// Tool: delete_node
registerShared(SHARED_TOOL_SPECS.deleteNode, async ({ pageId, nodeId }) => {
const result = await docmostClient.deleteNode(pageId, nodeId);
return jsonContent(result);
});
// Tool: insert_image
// Schema + description now live in the shared registry (#410) so BOTH this MCP
// server and the in-app AI-chat agent expose it. The execute body is unchanged.
registerShared(
SHARED_TOOL_SPECS.insertImage,
async ({ pageId, imageUrl, align, alt, replaceText, afterText }) => {
const result = await docmostClient.insertImage(pageId, imageUrl, {
align,
alt,
replaceText,
afterText,
});
return jsonContent(result);
},
);
// Tool: replace_image
// Schema + description now live in the shared registry (#410).
registerShared(
SHARED_TOOL_SPECS.replaceImage,
async ({ pageId, attachmentId, imageUrl, align, alt }) => {
const result = await docmostClient.replaceImage(
pageId,
attachmentId,
imageUrl,
{
align,
alt,
},
);
return jsonContent(result);
},
);
// Tool: drawio_get — read a draw.io diagram as mxGraph XML (or the raw SVG).
registerShared(
SHARED_TOOL_SPECS.drawioGet,
async ({ pageId, node, format }) => {
const result = await docmostClient.drawioGet(pageId, node, format ?? "xml");
return jsonContent(result);
},
);
// Tool: drawio_create — lint mxGraph XML, build the .drawio.svg, insert a node.
registerShared(
SHARED_TOOL_SPECS.drawioCreate,
async ({ pageId, xml, position, anchorNodeId, anchorText, title }) => {
const result = await docmostClient.drawioCreate(
pageId,
{ position, anchorNodeId, anchorText },
xml,
title,
);
return jsonContent(result);
},
);
// Tool: drawio_update — optimistic-locked full replacement of a diagram.
registerShared(
SHARED_TOOL_SPECS.drawioUpdate,
async ({ pageId, node, xml, baseHash }) => {
const result = await docmostClient.drawioUpdate(pageId, node, xml, baseHash);
return jsonContent(result);
},
);
// Tool: share_page
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own `searchIndexing ?? true` default.
registerShared(
SHARED_TOOL_SPECS.sharePage,
async ({ pageId, searchIndexing }) => {
const result = await docmostClient.sharePage(pageId, searchIndexing ?? true);
return jsonContent(result);
},
);
// Tool: unshare_page
registerShared(SHARED_TOOL_SPECS.unsharePage, async ({ pageId }) => {
const result = await docmostClient.unsharePage(pageId);
return jsonContent(result);
});
// Tool: list_shares
registerShared(SHARED_TOOL_SPECS.listShares, async () => {
const result = await docmostClient.listShares();
return jsonContent(result);
});
// Tool: move_page
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's cycle guard, its 'null'/'' -> null string coercion, and
// its positive-confirmation check on the move response.
registerShared(
SHARED_TOOL_SPECS.movePage,
async ({ pageId, parentPageId, position }) => {
const finalParentId =
parentPageId === "" || parentPageId === "null" ? null : parentPageId;
// Cheap cycle guard: a page cannot be moved directly under itself.
// (Deeper descendant-cycle detection is intentionally out of scope.)
if (finalParentId !== null && finalParentId === pageId) {
throw new Error("cannot move a page under itself");
}
const result = await docmostClient.movePage(
pageId,
finalParentId || null,
position,
);
// Require POSITIVE confirmation: the live /pages/move success shape is
// exactly { success: true, status: 200 }. An empty body, a 204, or any odd
// shape lacking success === true must NOT be reported as a successful move,
// so we surface the raw API result instead of declaring success.
if (!(result && typeof result === "object" && result.success === true)) {
throw new Error(
`Failed to move page ${pageId}: ${JSON.stringify(result)}`,
);
}
return jsonContent({
message: `Successfully moved page ${pageId} to parent ${finalParentId || "root"}`,
result,
});
},
);
// Tool: delete_page
// Schema + description now live in the shared registry (#294). The shared schema
// exposes ONLY pageId, so no permanent/force-delete flag can reach the client.
registerShared(SHARED_TOOL_SPECS.deletePage, async ({ pageId }) => {
await docmostClient.deletePage(pageId);
return {
content: [
{ type: "text" as const, text: `Successfully deleted page ${pageId}` },
],
};
});
// --- Comment tools (ported from upstream PR #3 by Max Nikitin) ---
// Tool: list_comments
registerShared(
SHARED_TOOL_SPECS.listComments,
async ({ pageId, includeResolved }) => {
const comments = await docmostClient.listComments(pageId, includeResolved);
return jsonContent(comments);
},
);
// Tool: create_comment
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own guards (require a selection for a top-level
// comment; reject suggestedText on a reply / without a selection).
registerShared(
SHARED_TOOL_SPECS.createComment,
async ({ pageId, content, selection, parentCommentId, suggestedText }) => {
if (!parentCommentId && (!selection || !selection.trim())) {
throw new Error(
"create_comment: a 'selection' (exact text to anchor on) is required for a top-level comment; omit it only when replying via parentCommentId.",
);
}
if (suggestedText !== undefined) {
if (parentCommentId) {
throw new Error(
"create_comment: 'suggestedText' cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !selection.trim()) {
throw new Error(
"create_comment: 'suggestedText' requires a 'selection' to anchor and rewrite.",
);
}
}
const result = await docmostClient.createComment(
pageId,
content,
"inline",
selection,
parentCommentId,
suggestedText,
);
return jsonContent(result);
},
);
// Tool: update_comment
server.registerTool(
"update_comment",
@@ -785,39 +374,6 @@ server.registerTool(
},
);
// Tool: resolve_comment
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.resolveComment,
async ({ commentId, resolved }) => {
const result = await docmostClient.resolveComment(commentId, resolved);
return jsonContent(result);
},
);
// Tool: check_new_comments
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own guard rejecting an unparseable `since` timestamp.
registerShared(
SHARED_TOOL_SPECS.checkNewComments,
async ({ spaceId, since, parentPageId }) => {
// Reject an unparseable timestamp up front: otherwise the comparison
// against NaN silently treats every comment as "not new" and the tool
// returns zero results without signalling the bad input.
if (Number.isNaN(Date.parse(since))) {
throw new Error(
`Invalid 'since' timestamp: ${JSON.stringify(since)} — expected an ISO 8601 date (e.g. '2026-03-10T00:00:00Z')`,
);
}
const result = await docmostClient.checkNewComments(
spaceId,
since,
parentPageId,
);
return jsonContent(result);
},
);
// Tool: search
// INTENTIONAL per-transport divergence (not shared): the in-app `searchPages`
// runs a semantic + keyword hybrid (RRF) with in-process access control and a
@@ -927,43 +483,5 @@ server.registerTool(
},
);
// Tool: insert_footnote
// Schema + description now live in the shared registry (#410) so the in-app
// AI-chat agent exposes it too. The execute body is unchanged.
registerShared(
SHARED_TOOL_SPECS.insertFootnote,
async ({ pageId, anchorText, text }) => {
const result = await docmostClient.insertFootnote(pageId, anchorText, text);
return jsonContent(result);
},
);
// Tool: diff_page_versions
registerShared(
SHARED_TOOL_SPECS.diffPageVersions,
async ({ pageId, from, to }) => {
const result = await docmostClient.diffPageVersions(pageId, from, to);
return jsonContent(result);
},
);
// Tool: list_page_history
registerShared(
SHARED_TOOL_SPECS.listPageHistory,
async ({ pageId, cursor }) => {
const result = await docmostClient.listPageHistory(pageId, cursor);
return jsonContent(result);
},
);
// Tool: restore_page_version
registerShared(
SHARED_TOOL_SPECS.restorePageVersion,
async ({ historyId }) => {
const result = await docmostClient.restorePageVersion(historyId);
return jsonContent(result);
},
);
return server;
}
+20 -2
View File
@@ -13,7 +13,11 @@ import { JSDOM } from "jsdom";
import { markdownToProseMirror } from "@docmost/prosemirror-markdown";
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import { sanitizeForYjs, findUnstorableAttr } from "@docmost/prosemirror-markdown";
import {
sanitizeForYjs,
findUnstorableAttr,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import { canonicalizeFootnotes } from "./footnote-canonicalize.js";
import { normalizeAndMergeFootnotes } from "./footnote-normalize-merge.js";
import { VerifyReport } from "./diff.js";
@@ -28,11 +32,25 @@ export { markdownToProseMirror };
* place. `label` names the stage that failed (diagnostic). `sanitizeForYjs`
* already stripped `undefined` attrs, so a remaining failure is pinpointed via
* `findUnstorableAttr`.
*
* Diagnostics precedence (#409): the dominant crash here is
* `Unknown node type: undefined` a nested node with an absent/unknown `type`
* (a SHAPE problem, e.g. `{"text":"foo"}` missing `"type":"text"`). That points
* at the node, not an attribute, so `findInvalidNode` is consulted FIRST and,
* on a hit, yields a path-anchored node-shape message. Only when the document
* shape is sound do we fall back to `findUnstorableAttr` (undefined/function/
* symbol/bigint attr values); the generic "attribute likely holds a value Yjs
* cannot store" sentence is the last resort.
*/
function unstorableYjsError(safe: any, label: string, e: unknown): Error {
const base = `Failed to encode document to Yjs (${label}): ${e instanceof Error ? e.message : String(e)}.`;
const badNode = findInvalidNode(safe);
if (badNode) {
return new Error(`${base} Invalid node: ${badNode.summary}`);
}
const bad = findUnstorableAttr(safe);
return new Error(
`Failed to encode document to Yjs (${label}): ${e instanceof Error ? e.message : String(e)}.${bad ? ` Offending attribute: ${bad}.` : " A node/mark attribute likely holds a value Yjs cannot store (e.g. undefined)."}`,
`${base}${bad ? ` Offending attribute: ${bad}.` : " A node/mark attribute likely holds a value Yjs cannot store (e.g. undefined)."}`,
);
}
@@ -6,7 +6,7 @@
* typographic quotes («»/) vs ASCII "…", em/en-dash vs `-`, non-breaking
* space vs normal space, differing space counts are not recognized as equal
* and "fork": two definitions appear where the author meant one. The existing
* de-dup paths miss this: `footnoteContentKey` (footnote-authoring.ts) only
* de-dup paths miss this: `footnoteContentKey` (@docmost/prosemirror-markdown) only
* collapses ASCII whitespace (quotes/dashes/NBSP untouched), and
* `canonicalizeFootnotes` keys purely by `attrs.id` (the two forks have
* different ids), so neither glues the forks together.
@@ -195,7 +195,7 @@ function stableAttrs(attrs: any): string {
/**
* ATTRS-AWARE merge key for a footnote definition. Deliberately DIVERGES from
* the shared `footnoteContentKey` (footnote-authoring.ts): that key's mark
* the shared `footnoteContentKey` (@docmost/prosemirror-markdown): that key's mark
* signature is TYPE-ONLY (`m.type`), so two definitions with identical visible
* text but marks differing only in ATTRIBUTES most importantly a `link` with a
* different `href` (footnotes are usually citations/links), also `code` /
+239
View File
@@ -0,0 +1,239 @@
// SERVER_INSTRUCTIONS — the editing guide surfaced to MCP clients in the
// initialize result so they can pick the right tool by intent and avoid
// resending whole documents.
//
// This guide is split into TWO parts that are composed at the bottom:
//
// 1. ROUTING_PROSE — the hand-written "when to use what" intent hints (READ /
// EDIT / PAGES / COMMENTS / HISTORY). This is legitimately manual: it
// encodes editorial judgement (which tool for which situation, the cheap-
// first ordering, the guardrail nudges) that cannot be derived from the
// registry. It is NOT the drift-guard for the tool set.
//
// 2. A GENERATED <tool_inventory> — every tool the server registers, listed
// by name + one-line purpose, grouped by family, built from the SAME
// registry the server registers tools from (SHARED_TOOL_SPECS' mcpName +
// catalogLine) PLUS the handful of inline MCP-only tools (their inventory
// lines live in INLINE_MCP_INVENTORY below). Because this list is BUILT
// from the registry, it can never drift out of sync with the registered
// tools — adding/renaming/removing a spec changes it automatically, with no
// prose edit and no scraper test. An unmapped tool still appears (under
// "OTHER"), so a new tool can never silently vanish from the guide.
//
// This replaces the old hand-maintained monolithic guide + its regex scraper
// test (test/unit/server-instructions.test.mjs), which only checked that every
// registered name appeared SOMEWHERE in the prose and drifted whenever a name
// was reworded.
//
// OUT OF SCOPE (issue #448): the README / README.ru tool catalogs are still
// hand-maintained prose and are NOT generated from this registry. Regenerating
// them from SHARED_TOOL_SPECS is tracked separately as an optional docs script
// under issue #412 — until then a tool rename still needs a manual README edit.
import { SHARED_TOOL_SPECS, SharedToolSpec } from "./tool-specs.js";
/**
* The hand-written routing prose the intent hints that tell a client which
* tool to reach for in which situation. Kept manual on purpose (it encodes
* editorial judgement, not a mechanical name list). The generated inventory
* below is spliced in after it.
*/
export const ROUTING_PROSE =
"Docmost editing guide — choose the tool by intent. The <tool_inventory> at the end lists every tool with a one-line purpose; the notes below are the routing hints for WHEN to reach for each.\n" +
"READ: find a page -> search (workspace-wide full-text); list -> list_pages / list_spaces. Locate blocks and their ids CHEAPLY -> get_outline (compact top-level map; start here, not get_page_json). One block's subtree -> get_node (by attrs.id, or \"#<index>\" for tables, which carry no id). Find every occurrence of a string/regex ON a page (and where each is) -> search_in_page, NOT block-by-block get_node — it returns each hit's node ref + block index + context for a targeted comment. Whole page -> get_page (Markdown, lossy; inline <span data-comment-id> tags are comment anchors — markup, not text) or get_page_json (lossless ProseMirror with block ids). Hand a huge page (with images) to an external consumer without pulling it through the model context -> stash_page (returns a short-lived anonymous URL).\n" +
"EDIT: fix wording/typos/numbers -> edit_page_text (find/replace inside blocks, no node id needed). Change ONE block (paragraph/heading/callout/etc.) structurally -> patch_node (by attrs.id from get_outline). Add a block -> insert_node (before/after a block by attrs.id or by anchor text, or append). Remove a block -> delete_node (by attrs.id). Tables -> table_get / table_update_cell / table_insert_row / table_delete_row (address by \"#<index>\" from get_outline; table nodes have no attrs.id). Images -> insert_image (add from a web URL) / replace_image (swap an existing image). Draw.io diagrams -> drawio_create (create from mxGraph XML and insert), drawio_get (read a diagram as mxGraph XML + a hash), drawio_update (replace a diagram; pass the hash from drawio_get as baseHash for optimistic locking). Footnotes -> insert_footnote. Bulk/structural rewrite -> update_page_json (full ProseMirror replace) or update_page_markdown (full plain-Markdown body replace, re-imported — block ids regenerate); prefer the granular tools above to avoid resending the whole ~100KB+ document. Complex/scripted rewrite (multiple coordinated edits, renumbering) -> docmost_transform: write a JS `(doc, ctx) => doc` transform, preview the diff with dryRun (default), then apply with dryRun:false; ctx.helpers includes commentsToFootnotes for turning inline comments into numbered footnotes.\n" +
"PAGES: new -> create_page (Markdown). Rename (title only) -> rename_page. Move -> move_page. Delete -> delete_page (SOFT delete — the page goes to trash and is restorable; nothing is permanent). Copy/replace a page's whole content from another page (server-side, no document through the model) -> copy_page_content. Sharing -> share_page / unshare_page / list_shares; share_page makes the page PUBLICLY accessible — do it only when explicitly asked.\n" +
"COMMENTS: create_comment is always inline and requires an EXACT selection — contiguous text from a single block, <=250 chars (fails rather than leaving an unanchored comment); reply to a thread via parentCommentId. Propose a concrete text fix for one-click human approval -> create_comment with suggestedText (the exact plain-text replacement for the selection; the selection must then be UNIQUE in the page — extend it with context if needed); prefer this over editing directly when the change is subjective or needs the author's sign-off. Manage -> list_comments, update_comment, resolve_comment (resolve/reopen, reversible — prefer over delete to close), delete_comment, check_new_comments.\n" +
"HISTORY: review what changed -> diff_page_versions (a historyId vs current, or two versions). List saved versions -> list_page_history. Undo a bad edit -> restore_page_version (writes a past version back as current; itself revertible). Export a page to self-contained Docmost Markdown (with comment anchors) -> export_page_markdown.";
/**
* A single generated inventory line: the tool's registered NAME + a one-line
* purpose. For a registry tool the purpose is its `catalogLine` (falling back
* to the first sentence of its description); for an inline MCP-only tool it is
* the hand-written line in INLINE_MCP_INVENTORY.
*/
export interface ToolInventoryLine {
name: string;
purpose: string;
}
/**
* The families the inventory is grouped under, in display order. A tool is
* placed by looking its mcpName up in TOOL_FAMILY; anything not listed there
* falls into "OTHER" so it is never dropped from the guide.
*/
const FAMILY_ORDER = [
"READ",
"EDIT",
"PAGES",
"COMMENTS",
"HISTORY",
"OTHER",
] as const;
type Family = (typeof FAMILY_ORDER)[number];
/**
* mcpName -> family for the generated inventory grouping. Purely cosmetic (it
* orders the inventory to mirror the routing prose); an unmapped tool still
* appears under OTHER, so forgetting to add an entry here can never drop a tool
* from the guide it only lands it in the catch-all group.
*/
const TOOL_FAMILY: Record<string, Family> = {
// READ
search: "READ",
list_pages: "READ",
list_spaces: "READ",
get_outline: "READ",
get_node: "READ",
search_in_page: "READ",
get_page: "READ",
get_page_json: "READ",
get_workspace: "READ",
stash_page: "READ",
// EDIT
edit_page_text: "EDIT",
patch_node: "EDIT",
insert_node: "EDIT",
delete_node: "EDIT",
update_page_json: "EDIT",
update_page_markdown: "EDIT",
table_get: "EDIT",
table_update_cell: "EDIT",
table_insert_row: "EDIT",
table_delete_row: "EDIT",
insert_image: "EDIT",
replace_image: "EDIT",
insert_footnote: "EDIT",
drawio_get: "EDIT",
drawio_create: "EDIT",
drawio_update: "EDIT",
docmost_transform: "EDIT",
// PAGES
create_page: "PAGES",
rename_page: "PAGES",
move_page: "PAGES",
delete_page: "PAGES",
copy_page_content: "PAGES",
share_page: "PAGES",
unshare_page: "PAGES",
list_shares: "PAGES",
// COMMENTS
create_comment: "COMMENTS",
list_comments: "COMMENTS",
update_comment: "COMMENTS",
resolve_comment: "COMMENTS",
delete_comment: "COMMENTS",
check_new_comments: "COMMENTS",
// HISTORY
diff_page_versions: "HISTORY",
list_page_history: "HISTORY",
restore_page_version: "HISTORY",
export_page_markdown: "HISTORY",
// import_page_markdown is now inAppOnly (#411) — it is not registered on the
// external MCP host, so it no longer appears in the generated inventory.
};
/**
* Inventory lines for the INLINE MCP-only tools the ones registered directly
* in index.ts (not via SHARED_TOOL_SPECS) because they diverge per transport or
* exist only on this standalone surface. They carry no `catalogLine`, so their
* one-line purpose is hand-written here. This is the ONLY hand-maintained tool
* list left, and it is tiny; a new inline tool without an entry here is caught
* by the completeness guard in `tool-inventory.test.mjs`.
*/
export const INLINE_MCP_INVENTORY: ToolInventoryLine[] = [
{
name: "table_get",
purpose:
"read a table as a matrix of cell texts + per-cell paragraph ids.",
},
{
name: "search",
purpose:
"full-text search for pages and content across the whole workspace.",
},
{
name: "docmost_transform",
purpose:
"edit a page by running a sandboxed JS `(doc, ctx) => doc` transform, with a dryRun diff preview.",
},
{
name: "update_comment",
purpose: "update an existing comment's content (creator only).",
},
{
name: "delete_comment",
purpose: "delete a comment (creator or space admin only).",
},
];
/**
* Derive the one-line purpose for a registry spec: prefer its hand-written
* `catalogLine` (already a "name — purpose" line we take the purpose after
* the em dash), else fall back to the first sentence of its description.
*/
function purposeForSpec(spec: SharedToolSpec): string {
const line = spec.catalogLine?.trim();
if (line) {
const dash = line.indexOf(" — ");
if (dash >= 0) return line.slice(dash + 3).trim();
return line;
}
const desc = (spec.description ?? "").replace(/\s+/g, " ").trim();
const firstSentence = desc.split(/(?<=[.!?])\s/)[0];
return firstSentence || desc || "(no description)";
}
/**
* Build the flat list of every registered tool's inventory line: one per shared
* registry spec (skipping `inAppOnly` specs, which are not registered on this
* MCP host) PLUS every inline MCP-only tool. Pure and deterministic the
* registry drives it, so it can never drift from what index.ts registers.
*/
export function buildToolInventoryLines(
specs: Record<string, SharedToolSpec> = SHARED_TOOL_SPECS,
inline: ToolInventoryLine[] = INLINE_MCP_INVENTORY,
): ToolInventoryLine[] {
const lines: ToolInventoryLine[] = [];
for (const spec of Object.values(specs)) {
if (spec.inAppOnly) continue; // not registered on the MCP host
lines.push({ name: spec.mcpName, purpose: purposeForSpec(spec) });
}
for (const l of inline) lines.push({ ...l });
return lines;
}
/**
* Render the generated `<tool_inventory>` block: every tool name + purpose,
* grouped by family (families in FAMILY_ORDER; tools within a family sorted by
* name for stable output; unmapped tools fall into OTHER). Pure.
*/
export function buildToolInventory(
specs: Record<string, SharedToolSpec> = SHARED_TOOL_SPECS,
inline: ToolInventoryLine[] = INLINE_MCP_INVENTORY,
): string {
const byFamily = new Map<Family, ToolInventoryLine[]>();
for (const family of FAMILY_ORDER) byFamily.set(family, []);
for (const line of buildToolInventoryLines(specs, inline)) {
const family = TOOL_FAMILY[line.name] ?? "OTHER";
byFamily.get(family)!.push(line);
}
const sections: string[] = [];
for (const family of FAMILY_ORDER) {
const items = byFamily.get(family)!;
if (items.length === 0) continue;
items.sort((a, b) => a.name.localeCompare(b.name));
for (const item of items) {
sections.push(` ${family} ${item.name}${item.purpose}`);
}
}
return ["<tool_inventory>", ...sections, "</tool_inventory>"].join("\n");
}
/**
* The composed editing guide: the hand-written routing prose followed by the
* generated, drift-proof tool inventory. Exported (and used by index.ts /
* createDocmostMcpServer) as the MCP server's `instructions`.
*/
export const SERVER_INSTRUCTIONS =
ROUTING_PROSE + "\n" + buildToolInventory();
+583 -7
View File
@@ -14,16 +14,108 @@
// some write tools, different limits, hybrid-RRF search, etc.) stay defined
// per-layer and are NOT represented here.
//
// MAINTENANCE RULE: adding, renaming, or removing a spec here (or an inline
// registerTool in index.ts) REQUIRES updating SERVER_INSTRUCTIONS in
// packages/mcp/src/index.ts — the intent-routing guide MCP clients receive on
// initialize. Enforced by test/unit/server-instructions.test.mjs.
// SERVER_INSTRUCTIONS note (issue #448): the intent-routing guide MCP clients
// receive on initialize is now SPLIT — its tool INVENTORY is GENERATED from this
// registry (mcpName + catalogLine) by server-instructions.ts, so adding /
// renaming / removing a spec here updates the guide's inventory AUTOMATICALLY;
// no prose edit is needed. Only an INLINE MCP-only tool (registerTool in
// index.ts, not a spec here) needs a hand-written line in INLINE_MCP_INVENTORY —
// enforced by test/unit/tool-inventory.test.mjs. The routing PROSE (the "when to
// use what" hints) in server-instructions.ts stays manual, but it is no longer a
// drift-guard for the tool set.
// Loose on purpose — see the comment above. The two zod majors expose different
// static type surfaces, so typing this precisely would couple the registry to
// one of them. Each builder uses only the common, stable subset of the API.
type ZodLike = any;
// The `node` normalizer shared by BOTH hosts (patch_node / insert_node /
// update_page_json): the model sometimes serializes a ProseMirror node arg as a
// JSON string, so we parse a string to an object (throwing a documented message
// on invalid JSON) and pass an object through. It lives in the converter package
// (#414) so it is the ONE copy both the MCP server and the in-app server import;
// putting it in a shared execute here keeps that single normalization in one
// place instead of hand-mirrored per host. Pure — safe across the zod boundary.
import { parseNodeArg } from '@docmost/prosemirror-markdown';
// Type-only import (erased at compile) of the real client so `DocmostClientLike`
// is DERIVED from it (issue #446), not hand-mirrored. The loosest correct client
// surface both hosts satisfy: the in-app host passes its own DERIVED
// `DocmostClientLike` (a Pick of the same class) and the MCP host passes the real
// `DocmostClient`, so both are structurally assignable to this shared alias.
import type { DocmostClient } from './client.js';
/**
* The client surface a shared `execute` may call the LOOSEST correct type both
* hosts satisfy: a `Pick` of the real `DocmostClient` methods the executes below
* use. DERIVED from the real class (issue #446) so a signature change to any
* consumed method surfaces as a compile error in the execute bodies here, not a
* silent runtime "wrong argument". Kept as a Pick (not the whole class) so the
* standalone MCP host's full `DocmostClient` AND the in-app host's OWN narrower
* `DocmostClientLike` (also a Pick of the same class, a superset of these methods)
* are both structurally assignable to it. `import type` is fully erased, so
* tool-specs.ts pulls in no runtime dependency on the client and still crosses the
* zod-major boundary freely. When you add a client call to an execute below, add
* its method name here too (a compile error will point you at it).
*/
export type DocmostClientLike = Pick<
DocmostClient,
| 'getWorkspace'
| 'getSpaces'
| 'listShares'
| 'listPages'
| 'getPage'
| 'getPageJson'
| 'getOutline'
| 'getNode'
| 'searchInPage'
| 'listComments'
| 'checkNewComments'
| 'listPageHistory'
| 'diffPageVersions'
| 'exportPageMarkdown'
| 'createPage'
| 'renamePage'
| 'movePage'
| 'deletePage'
| 'editPageText'
| 'patchNode'
| 'insertNode'
| 'deleteNode'
| 'updatePage'
| 'updatePageJson'
| 'tableInsertRow'
| 'tableDeleteRow'
| 'tableUpdateCell'
| 'copyPageContent'
| 'importPageMarkdown'
| 'sharePage'
| 'unsharePage'
| 'restorePageVersion'
| 'stashPage'
| 'insertFootnote'
| 'insertImage'
| 'replaceImage'
| 'drawioGet'
| 'drawioCreate'
| 'drawioUpdate'
| 'createComment'
| 'resolveComment'
>;
/**
* A shared tool `execute`: the single canonical mapping from validated schema
* args to the client call. Plain JS it crosses the zod-major boundary (v3 in
* the MCP package, v4 on the server) freely, receiving the already-validated,
* type-erased args from whichever host invoked it. It returns RAW data; the host
* applies its own result envelope (the MCP transport wraps it as JSON text
* content, the in-app AI-SDK host returns it as-is). Host-specific overrides
* (`mcpExecute`/`inAppExecute`) return a value the host uses instead of wrapping.
*/
export type SharedToolExecute = (
client: DocmostClientLike,
args: Record<string, unknown>,
) => Promise<unknown>;
export interface SharedToolSpec {
/** snake_case tool name passed to McpServer.registerTool. */
mcpName: string;
@@ -54,8 +146,53 @@ export interface SharedToolSpec {
* in-app side uses z.object({})).
*/
buildShape?: (z: ZodLike) => Record<string, unknown>;
/**
* Single canonical mapping from validated schema args to the client call,
* shared by BOTH hosts. Returns RAW data the MCP host wraps it as JSON text
* content (jsonContent), the in-app host returns it as-is. Present on tools
* whose mapping AND raw result are identical across the two layers. When a host
* needs a genuinely different mapping or result shape, it supplies an override
* (below) and the host uses that INSTEAD of `execute`.
*/
execute?: SharedToolExecute;
/**
* MCP-host override for a DELIBERATE per-layer difference (a guardrail, an
* omitted param, or a non-JSON result envelope like a resource_link / a bare
* success line). When present, the MCP host calls this and uses its return
* value VERBATIM (it is NOT re-wrapped in jsonContent), so this override owns
* the full MCP content envelope.
*/
mcpExecute?: SharedToolExecute;
/**
* In-app-host override for a DELIBERATE per-layer difference (a projected
* result shape, a different guardrail message). When present, the in-app host
* calls this and returns its value as the tool result (no wrapping).
*/
inAppExecute?: SharedToolExecute;
/** Registered only on the MCP host (skipped by the in-app registry loop). */
mcpOnly?: boolean;
/** Registered only on the in-app host (skipped by the MCP registry loop). */
inAppOnly?: boolean;
}
// --- Shared execute helpers -------------------------------------------------
//
// Each helper is the ONE canonical arg->client mapping for a tool (or a host
// override where the two layers deliberately differ). They are attached to their
// spec below. Kept as named functions (not inline) so the spec table stays
// readable and each mapping is individually greppable/testable.
//
// The `args` are the host's already-validated, zod-erased input; we read the
// same fields the tool's buildShape declares. Return RAW data unless the name is
// an mcp*/inApp* override that owns the host's full result shape.
/** Format a JSON payload as the MCP transport's text-content envelope. Mirrors
* the private `jsonContent` in index.ts so an mcpExecute override that must NOT
* be re-wrapped can still emit the standard envelope for the data part. */
const mcpJson = (data: unknown) => ({
content: [{ type: 'text' as const, text: JSON.stringify(data, null, 2) }],
});
export const SHARED_TOOL_SPECS = {
// --- no-argument read tools ---
@@ -65,6 +202,7 @@ export const SHARED_TOOL_SPECS = {
description: 'Fetch metadata about the current workspace (name, settings).',
tier: 'core',
catalogLine: 'getWorkspace — fetch current workspace metadata (name, settings).',
execute: (client) => client.getWorkspace(),
},
listSpaces: {
@@ -75,6 +213,7 @@ export const SHARED_TOOL_SPECS = {
'spaces (id, name, slug, ...).',
tier: 'core',
catalogLine: 'listSpaces — list the spaces the user can access (id, name, slug).',
execute: (client) => client.getSpaces(),
},
listShares: {
@@ -84,6 +223,7 @@ export const SHARED_TOOL_SPECS = {
'List all public shares in the workspace with page titles and public URLs.',
tier: 'deferred',
catalogLine: 'listShares — list all public shares in the workspace with their URLs.',
execute: (client) => client.listShares(),
},
// --- single-pageId read tools ---
@@ -102,6 +242,7 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
pageId: z.string().min(1),
}),
execute: (client, { pageId }) => client.getPageJson(pageId as string),
},
getOutline: {
@@ -119,6 +260,7 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
pageId: z.string().min(1),
}),
execute: (client, { pageId }) => client.getOutline(pageId as string),
},
// --- two-id read tool ---
@@ -139,6 +281,8 @@ export const SHARED_TOOL_SPECS = {
pageId: z.string().min(1),
nodeId: z.string().min(1),
}),
execute: (client, { pageId, nodeId }) =>
client.getNode(pageId as string, nodeId as string),
},
// --- in-page occurrence search (client-side, over ProseMirror plain text) ---
@@ -196,6 +340,12 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe('Max matches to RETURN (default 50, max 200); total is always reported.'),
}),
execute: (client, { pageId, query, regex, caseSensitive, limit }) =>
client.searchInPage(pageId as string, query as string, {
regex: regex as boolean | undefined,
caseSensitive: caseSensitive as boolean | undefined,
limit: limit as number | undefined,
}),
},
// --- node delete ---
@@ -212,6 +362,8 @@ export const SHARED_TOOL_SPECS = {
pageId: z.string().min(1),
nodeId: z.string().min(1),
}),
execute: (client, { pageId, nodeId }) =>
client.deleteNode(pageId as string, nodeId as string),
},
// --- single-block structural write (patch / insert) ---
@@ -236,7 +388,10 @@ export const SHARED_TOOL_SPECS = {
'heading {"type":"heading","attrs":{"level":2},"content":' +
'[{"type":"text","text":"Title"}]}. Bold is a mark: ' +
'{"type":"text","text":"x","marks":[{"type":"bold"}]}. The node may be a ' +
'JSON object or a JSON string (both accepted). Cheaper and safer than ' +
'JSON object or a JSON string (both accepted). EVERY node, including ' +
'nested children, must carry a string `type` from the Docmost schema; ' +
'text leaves are {"type":"text","text":"..."} (a bare {"text":"..."} is ' +
'rejected up front). Cheaper and safer than ' +
'replacing the whole document for one-block structural edits. Reversible: ' +
'the previous version is kept in page history.',
tier: 'deferred',
@@ -259,6 +414,11 @@ export const SHARED_TOOL_SPECS = {
'JSON object or JSON string both accepted.',
),
}),
// parseNodeArg normalizes a JSON-string node into an object (the model
// sometimes serializes it as a string) before the client's typeof-object
// guard rejects it — identical on both hosts.
execute: (client, { pageId, nodeId, node }) =>
client.patchNode(pageId as string, nodeId as string, parseNodeArg(node)),
},
insertNode: {
@@ -282,7 +442,10 @@ export const SHARED_TOOL_SPECS = {
'{"type":"paragraph","content":[{"type":"text","text":"Hello"}]} or a ' +
'heading {"type":"heading","attrs":{"level":2},"content":' +
'[{"type":"text","text":"Title"}]}. Bold is a mark: ' +
'{"type":"text","text":"x","marks":[{"type":"bold"}]}. The node may be a ' +
'{"type":"text","text":"x","marks":[{"type":"bold"}]}. EVERY node, ' +
'including nested children, must carry a string `type` from the Docmost ' +
'schema; text leaves are {"type":"text","text":"..."} (a bare ' +
'{"text":"..."} is rejected up front). The node may be a ' +
'JSON object or a JSON string (both accepted). Reversible via page history.',
tier: 'deferred',
catalogLine:
@@ -312,6 +475,12 @@ export const SHARED_TOOL_SPECS = {
'are tolerated as a fallback; prefer plain text or anchorNodeId.',
),
}),
execute: (client, { pageId, node, position, anchorNodeId, anchorText }) =>
client.insertNode(pageId as string, parseNodeArg(node), {
position: position as 'before' | 'after' | 'append',
anchorNodeId: anchorNodeId as string | undefined,
anchorText: anchorText as string | undefined,
}),
},
// --- share management ---
@@ -342,6 +511,11 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe('Allow public search engines to index it (default true).'),
}),
// `searchIndexing ?? true` is a no-op default: the client method already
// defaults searchIndexing to true, so passing `undefined` (the in-app form)
// and `?? true` (the old MCP form) are byte-identical — one canonical mapping.
execute: (client, { pageId, searchIndexing }) =>
client.sharePage(pageId as string, (searchIndexing as boolean | undefined) ?? true),
},
unsharePage: {
@@ -353,6 +527,7 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
pageId: z.string().min(1).describe('ID of the page to unshare'),
}),
execute: (client, { pageId }) => client.unsharePage(pageId as string),
},
// --- version history ---
@@ -381,6 +556,12 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe("historyId, or 'current'/omit for current content"),
}),
execute: (client, { pageId, from, to }) =>
client.diffPageVersions(
pageId as string,
from as string | undefined,
to as string | undefined,
),
},
listPageHistory: {
@@ -400,6 +581,8 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe('Pagination cursor from a previous nextCursor'),
}),
execute: (client, { pageId, cursor }) =>
client.listPageHistory(pageId as string, cursor as string | undefined),
},
restorePageVersion: {
@@ -416,6 +599,8 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
historyId: z.string().min(1),
}),
execute: (client, { historyId }) =>
client.restorePageVersion(historyId as string),
},
// --- markdown round-trip ---
@@ -423,6 +608,12 @@ export const SHARED_TOOL_SPECS = {
importPageMarkdown: {
mcpName: 'import_page_markdown',
inAppKey: 'importPageMarkdown',
// IN-APP ONLY (issue #411): the external /mcp surface no longer exposes
// import_page_markdown — the registry loop in index.ts skips inAppOnly specs,
// so this stays available to the in-app agent (round-tripping an EXPORTED
// Docmost-Markdown file) but is removed from the public MCP tool set. Plain
// authoring-markdown body replace on the MCP surface is update_page_markdown.
inAppOnly: true,
description:
"Replace a page's content from a self-contained Docmost-flavoured " +
'Markdown file produced by the page-Markdown export tool. Restores comment ' +
@@ -437,6 +628,8 @@ export const SHARED_TOOL_SPECS = {
pageId: z.string().min(1),
markdown: z.string().min(1),
}),
execute: (client, { pageId, markdown }) =>
client.importPageMarkdown(pageId as string, markdown as string),
},
// --- server-side content copy ---
@@ -459,6 +652,8 @@ export const SHARED_TOOL_SPECS = {
.min(1)
.describe('Page whose content is REPLACED (title/slug kept)'),
}),
execute: (client, { sourcePageId, targetPageId }) =>
client.copyPageContent(sourcePageId as string, targetPageId as string),
},
// --- surgical text edit (folds in the documented drift-bug fix) ---
@@ -508,6 +703,11 @@ export const SHARED_TOOL_SPECS = {
.min(1)
.describe('List of find/replace operations, applied in order'),
}),
execute: (client, { pageId, edits }) =>
client.editPageText(
pageId as string,
edits as Parameters<DocmostClientLike['editPageText']>[1],
),
},
// --- hand a large page to an external consumer without bloating context ---
@@ -536,6 +736,33 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
pageId: z.string().min(1),
}),
// In-app returns the full documented `{ uri, size, sha256, images }` object
// as-is (the canonical execute).
execute: (client, { pageId }) => client.stashPage(pageId as string),
// The MCP transport must deliver the body as a resource_link (so it never
// enters the model context) PLUS a structuredContent mirror of the documented
// shape (sha256 = the blob's ETag, mirror counts). Owns its full envelope, so
// it is NOT wrapped in jsonContent.
mcpExecute: async (client, { pageId }) => {
const result = await client.stashPage(pageId as string);
return {
content: [
{
type: 'resource_link' as const,
uri: result.uri,
name: 'page.json',
mimeType: 'application/json',
size: result.size,
},
],
structuredContent: {
uri: result.uri,
sha256: result.sha256,
size: result.size,
images: result.images,
},
};
},
},
// --- page tools (unified from the per-layer inline definitions, #294) ---
@@ -562,6 +789,20 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id (or slugId) of the page.'),
}),
// MCP wraps the raw `{ data, success }` as JSON. The in-app host instead
// projects a token-efficient `{ title, markdown }` (its long-standing shape).
execute: (client, { pageId }) => client.getPage(pageId as string),
inAppExecute: async (client, { pageId }) => {
// getPage(pageId) -> { data: filterPage(page, markdown), success }.
const result = (await client.getPage(pageId as string)) as {
data?: { title?: string; content?: string };
};
const data = result?.data ?? {};
return {
title: data.title ?? '',
markdown: typeof data.content === 'string' ? data.content : '',
};
},
},
listPages: {
@@ -596,6 +837,15 @@ export const SHARED_TOOL_SPECS = {
'Requires spaceId; ignores limit.',
),
}),
// `limit ?? 50` / `tree ?? false` are no-op defaults: the client method
// already defaults limit=50, tree=false, so the old MCP explicit-default form
// and the in-app pass-through form are byte-identical — one canonical mapping.
execute: (client, { spaceId, limit, tree }) =>
client.listPages(
spaceId as string | undefined,
(limit as number | undefined) ?? 50,
(tree as boolean | undefined) ?? false,
),
},
createPage: {
@@ -624,6 +874,28 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe('Optional parent page id to nest the new page under.'),
}),
// MCP wraps the raw create response as JSON. In-app projects `{ id, title }`
// and defensively coerces a missing body to '' (the schema makes content a
// required string, so `?? ''` only guards an absent field — preserved).
execute: (client, { title, content, spaceId, parentPageId }) =>
client.createPage(
title as string,
content as string,
spaceId as string,
parentPageId as string | undefined,
),
inAppExecute: async (client, { title, content, spaceId, parentPageId }) => {
// createPage(title, content, spaceId, parentPageId?) ->
// { data: filterPage(page, markdown), success }.
const result = (await client.createPage(
title as string,
(content as string | undefined) ?? '',
spaceId as string,
parentPageId as string | undefined,
)) as { data?: { id?: string; slugId?: string; title?: string } };
const data = result?.data ?? {};
return { id: data.id ?? data.slugId, title: data.title ?? (title as string) };
},
},
movePage: {
@@ -661,6 +933,62 @@ export const SHARED_TOOL_SPECS = {
'append at the end.',
),
}),
// The MCP host keeps its robustness guards (coerce 'null'/'' -> null, a cheap
// self-cycle guard, and a POSITIVE { success: true } confirmation) and its
// human-readable success envelope — owns its full result, so it is NOT
// wrapped in jsonContent.
mcpExecute: async (client, { pageId, parentPageId, position }) => {
const finalParentId =
parentPageId === '' || parentPageId === 'null'
? null
: (parentPageId as string | null | undefined);
// Cheap cycle guard: a page cannot be moved directly under itself.
// (Deeper descendant-cycle detection is intentionally out of scope.)
if (finalParentId !== null && finalParentId === pageId) {
throw new Error('cannot move a page under itself');
}
const result = await client.movePage(
pageId as string,
finalParentId || null,
position as string | undefined,
);
// Require POSITIVE confirmation: the live /pages/move success shape is
// exactly { success: true, status: 200 }. An empty body, a 204, or any odd
// shape lacking success === true must NOT be reported as a successful move.
if (
!(
result &&
typeof result === 'object' &&
(result as { success?: unknown }).success === true
)
) {
throw new Error(
`Failed to move page ${pageId}: ${JSON.stringify(result)}`,
);
}
return mcpJson({
message: `Successfully moved page ${pageId} to parent ${finalParentId || 'root'}`,
result,
});
},
// The in-app host has no guards; it forwards `parentPageId ?? null` + the
// optional position and projects `{ pageId, parentPageId, moved }`.
inAppExecute: async (client, { pageId, parentPageId, position }) => {
await client.movePage(
pageId as string,
(parentPageId as string | null | undefined) ?? null,
position as string | undefined,
);
return {
pageId,
parentPageId: (parentPageId as string | null | undefined) ?? null,
moved: true,
};
},
},
renamePage: {
@@ -675,6 +1003,13 @@ export const SHARED_TOOL_SPECS = {
pageId: z.string().min(1).describe('The id of the page to rename.'),
title: z.string().min(1).describe('The new title.'),
}),
// MCP wraps the raw rename response; in-app projects `{ pageId, title }`.
execute: (client, { pageId, title }) =>
client.renamePage(pageId as string, title as string),
inAppExecute: async (client, { pageId, title }) => {
await client.renamePage(pageId as string, title as string);
return { pageId, title };
},
},
deletePage: {
@@ -691,6 +1026,23 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to move to trash.'),
}),
// deletePage(pageId) hits POST /pages/delete with { pageId } only — the
// soft-delete (trash) path. GUARDRAIL: the schema exposes ONLY pageId, so no
// permanent/force-delete flag can reach the client on either host (asserted by
// ai-chat-tools.service.spec.ts). MCP emits a bare success line (owns its full
// envelope); in-app projects `{ pageId, trashed }`.
mcpExecute: async (client, { pageId }) => {
await client.deletePage(pageId as string);
return {
content: [
{ type: 'text' as const, text: `Successfully deleted page ${pageId}` },
],
};
},
inAppExecute: async (client, { pageId }) => {
await client.deletePage(pageId as string);
return { pageId, trashed: true };
},
},
updatePageJson: {
@@ -705,7 +1057,10 @@ export const SHARED_TOOL_SPECS = {
'"paragraph","content":[{"type":"text","text":"Hi"}]}]}. `content` may be ' +
'a JSON object or a JSON string (both accepted), and is OPTIONAL: omit it ' +
'to update only the title (though prefer the rename-page tool for a title-only ' +
'change). Supplying neither content nor title is an error. Reversible: ' +
'change). Supplying neither content nor title is an error. EVERY node, ' +
'including nested children, must carry a string `type` from the Docmost ' +
'schema; text leaves are {"type":"text","text":"..."} (a bare ' +
'{"text":"..."} is rejected up front). Reversible: ' +
'the previous version is kept in page history.',
tier: 'deferred',
catalogLine:
@@ -721,6 +1076,64 @@ export const SHARED_TOOL_SPECS = {
),
title: z.string().optional().describe('Optional new title'),
}),
// Content normalization is identical on both hosts: only parse/validate the
// document when actually supplied; undefined/null passes straight through so
// the client performs a title-only (or no-op) update. A string is JSON.parsed
// (an empty string "" therefore throws), an object passes through unchanged.
execute: (client, { pageId, content, title }) => {
let doc: unknown;
if (content === undefined || content === null) {
doc = undefined;
} else {
doc = parseNodeArg(content, 'content was a string but not valid JSON');
}
return client.updatePageJson(pageId as string, doc, title as string | undefined);
},
},
// Full-body replace from PLAIN Markdown (issue #411). Pairs with
// updatePageJson (which takes a ProseMirror document): this one takes a
// markdown string and re-imports the whole body. `client.updatePage` runs it
// through updatePageContentRealtime -> markdownToProseMirrorCanonical, so
// Docmost-flavoured markdown (incl. `^[...]` inline footnotes) is parsed and
// canonicalized. Distinct from importPageMarkdown, which re-imports a
// self-contained EXPORTED Docmost-Markdown file (with comment anchors +
// diagrams); this tool takes ordinary authoring markdown. Shared spec, so the
// registry loop registers it on BOTH hosts (external MCP + in-app agent) —
// #411 replaced the old inline in-app `updatePageContent` tool with this.
updatePageMarkdown: {
// snake_case for now; camelCase public MCP naming is the next issue (#412).
mcpName: 'update_page_markdown',
inAppKey: 'updatePageMarkdown',
description:
"Replace a page's body with new Markdown content (and optionally its " +
'title). The whole body is re-imported from the markdown (block ids ' +
'regenerate — for surgical or id-preserving edits use the find/replace, ' +
'node-patch or page-JSON tools instead). Docmost-flavoured markdown is ' +
'parsed, including `^[...]` inline footnotes. Reversible: the previous ' +
'version is kept in page history.',
tier: 'deferred',
catalogLine:
"updatePageMarkdown — replace a page's body (and optionally title) with new Markdown.",
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to update.'),
content: z.string().describe('The new page body as Markdown.'),
title: z
.string()
.optional()
.describe('Optional new title for the page.'),
}),
// Single canonical execute on BOTH hosts (the tool was in-app only before,
// so there is no external-MCP behavior to preserve). NOTE (rename #411): the
// old inline in-app tool projected the client result to { pageId, updated };
// the registry now returns the raw client result { success, modified,
// message, pageId, verify? } instead. Deliberate: no code reads the removed
// `.updated` field, the raw result is strictly more informative to the model
// (it surfaces footnote/verify warnings), and it matches the on-both-hosts
// registry convention. The result-shape change is the ONLY behavior delta of
// this rename; the write path (updatePage -> markdown canonicalize) is identical.
execute: (client, { pageId, content, title }) =>
client.updatePage(pageId as string, content as string, title as string | undefined),
},
exportPageMarkdown: {
@@ -741,6 +1154,17 @@ export const SHARED_TOOL_SPECS = {
buildShape: (z) => ({
pageId: z.string().min(1).describe('The id of the page to export.'),
}),
// The markdown is a bare string. MCP returns it as a single text-content
// element (NOT jsonContent — that would JSON-quote the whole document);
// in-app projects `{ markdown }`.
mcpExecute: async (client, { pageId }) => {
const md = await client.exportPageMarkdown(pageId as string);
return { content: [{ type: 'text' as const, text: md }] };
},
inAppExecute: async (client, { pageId }) => {
const markdown = await client.exportPageMarkdown(pageId as string);
return { markdown };
},
},
// --- comment tools (unified from the per-layer inline definitions, #294) ---
@@ -823,6 +1247,69 @@ export const SHARED_TOOL_SPECS = {
'refused.',
),
}),
// Both hosts enforce the SAME guardrails (a top-level comment requires a
// selection; suggestedText is forbidden on a reply / without a selection) but
// with per-layer error wording (snake_case 'create_comment:' on the MCP
// surface, camelCase 'createComment' in-app) and different result shapes (MCP
// jsonContent, in-app projects `{ commentId, pageId }`). Preserved byte-for-
// byte via the two overrides.
mcpExecute: async (client, { pageId, content, selection, parentCommentId, suggestedText }) => {
if (!parentCommentId && (!selection || !(selection as string).trim())) {
throw new Error(
"create_comment: a 'selection' (exact text to anchor on) is required for a top-level comment; omit it only when replying via parentCommentId.",
);
}
if (suggestedText !== undefined) {
if (parentCommentId) {
throw new Error(
"create_comment: 'suggestedText' cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !(selection as string).trim()) {
throw new Error(
"create_comment: 'suggestedText' requires a 'selection' to anchor and rewrite.",
);
}
}
const result = await client.createComment(
pageId as string,
content as string,
'inline',
selection as string | undefined,
parentCommentId as string | undefined,
suggestedText as string | undefined,
);
return mcpJson(result);
},
inAppExecute: async (client, { pageId, content, selection, parentCommentId, suggestedText }) => {
if (!parentCommentId && (!selection || !(selection as string).trim())) {
throw new Error(
"createComment requires a 'selection' (exact text to anchor on) for a new top-level comment.",
);
}
if (suggestedText !== undefined) {
if (parentCommentId) {
throw new Error(
"createComment: 'suggestedText' cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !(selection as string).trim()) {
throw new Error(
"createComment: 'suggestedText' requires a 'selection' to anchor and rewrite.",
);
}
}
const result = (await client.createComment(
pageId as string,
content as string,
'inline',
selection as string | undefined,
parentCommentId as string | undefined,
suggestedText as string | undefined,
)) as { data?: { id?: string } };
const data = result?.data ?? {};
return { commentId: data.id, pageId };
},
},
listComments: {
@@ -848,6 +1335,8 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe('default only active threads; true — include resolved'),
}),
execute: (client, { pageId, includeResolved }) =>
client.listComments(pageId as string, includeResolved as boolean | undefined),
},
resolveComment: {
@@ -881,6 +1370,13 @@ export const SHARED_TOOL_SPECS = {
'true (default) marks the thread resolved/closed; false reopens it',
),
}),
// MCP wraps the raw resolve response; in-app projects `{ commentId, resolved }`.
execute: (client, { commentId, resolved }) =>
client.resolveComment(commentId as string, resolved as boolean),
inAppExecute: async (client, { commentId, resolved }) => {
await client.resolveComment(commentId as string, resolved as boolean);
return { commentId, resolved };
},
},
checkNewComments: {
@@ -916,6 +1412,30 @@ export const SHARED_TOOL_SPECS = {
'Only pages under this parent will be checked.',
),
}),
// The in-app host has NO `since` guard (the canonical execute, raw). The MCP
// host additionally rejects an unparseable `since` up front — otherwise the
// NaN comparison silently treats every comment as "not new" and returns zero
// without signalling the bad input. This guard is a DELIBERATE per-layer
// difference (the in-app surface never had it), preserved via mcpExecute.
execute: (client, { spaceId, since, parentPageId }) =>
client.checkNewComments(
spaceId as string,
since as string,
parentPageId as string | undefined,
),
mcpExecute: async (client, { spaceId, since, parentPageId }) => {
if (Number.isNaN(Date.parse(since as string))) {
throw new Error(
`Invalid 'since' timestamp: ${JSON.stringify(since)} — expected an ISO 8601 date (e.g. '2026-03-10T00:00:00Z')`,
);
}
const result = await client.checkNewComments(
spaceId as string,
since as string,
parentPageId as string | undefined,
);
return mcpJson(result);
},
},
// --- table tools (unified from the per-layer inline definitions, #294) ---
@@ -962,6 +1482,13 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe('0-based insert position (0 inserts before the header); omit to append.'),
}),
execute: (client, { pageId, table, cells, index }) =>
client.tableInsertRow(
pageId as string,
table as string,
cells as string[],
index as number | undefined,
),
},
tableDeleteRow: {
@@ -983,6 +1510,8 @@ export const SHARED_TOOL_SPECS = {
.describe('"#<index>" from the page outline, or a block id in the table.'),
index: z.number().int().describe('0-based row index to delete.'),
}),
execute: (client, { pageId, table, index }) =>
client.tableDeleteRow(pageId as string, table as string, index as number),
},
tableUpdateCell: {
@@ -1006,6 +1535,14 @@ export const SHARED_TOOL_SPECS = {
col: z.number().int().describe('0-based column index.'),
text: z.string().describe('The new cell text.'),
}),
execute: (client, { pageId, table, row, col, text }) =>
client.tableUpdateCell(
pageId as string,
table as string,
row as number,
col as number,
text as string,
),
},
// --- footnote + image write tools (promoted from inline MCP-only, #410) ---
@@ -1050,6 +1587,8 @@ export const SHARED_TOOL_SPECS = {
.min(1)
.describe('The footnote content as markdown (becomes the definition).'),
}),
execute: (client, { pageId, anchorText, text }) =>
client.insertFootnote(pageId as string, anchorText as string, text as string),
},
insertImage: {
@@ -1087,6 +1626,13 @@ export const SHARED_TOOL_SPECS = {
'Insert the image right after the first top-level block whose text contains this string',
),
}),
execute: (client, { pageId, imageUrl, align, alt, replaceText, afterText }) =>
client.insertImage(pageId as string, imageUrl as string, {
align: align as 'left' | 'center' | 'right' | undefined,
alt: alt as string | undefined,
replaceText: replaceText as string | undefined,
afterText: afterText as string | undefined,
}),
},
replaceImage: {
@@ -1118,6 +1664,11 @@ export const SHARED_TOOL_SPECS = {
align: z.enum(['left', 'center', 'right']).optional(),
alt: z.string().optional(),
}),
execute: (client, { pageId, attachmentId, imageUrl, align, alt }) =>
client.replaceImage(pageId as string, attachmentId as string, imageUrl as string, {
align: align as 'left' | 'center' | 'right' | undefined,
alt: alt as string | undefined,
}),
},
// --- draw.io diagrams (issue #423, stage 1) ---
@@ -1147,6 +1698,12 @@ export const SHARED_TOOL_SPECS = {
.optional()
.describe('"xml" (default) for mxGraph XML, or "svg" for the raw .drawio.svg.'),
}),
execute: (client, { pageId, node, format }) =>
client.drawioGet(
pageId as string,
node as string,
(format as 'xml' | 'svg' | undefined) ?? 'xml',
),
},
drawioCreate: {
@@ -1193,6 +1750,18 @@ export const SHARED_TOOL_SPECS = {
.describe('Anchor text fragment (for before/after).'),
title: z.string().optional().describe('Optional diagram title.'),
}),
// The flat schema fields are regrouped into the client's `where` object.
execute: (client, { pageId, xml, position, anchorNodeId, anchorText, title }) =>
client.drawioCreate(
pageId as string,
{
position: position as 'before' | 'after' | 'append',
anchorNodeId: anchorNodeId as string | undefined,
anchorText: anchorText as string | undefined,
},
xml as string,
title as string | undefined,
),
},
drawioUpdate: {
@@ -1226,5 +1795,12 @@ export const SHARED_TOOL_SPECS = {
.min(1)
.describe('The meta.hash from the drawio_get this edit is based on.'),
}),
execute: (client, { pageId, node, xml, baseHash }) =>
client.drawioUpdate(
pageId as string,
node as string,
xml as string,
baseHash as string,
),
},
} satisfies Record<string, SharedToolSpec>;
@@ -3,7 +3,7 @@
// footnote-canonical doc. These override the `replacePage` seam (symmetric to the
// `mutatePage` seam used by the insert-footnote-wrapper test) to capture the
// persisted doc WITHOUT a live Hocuspocus collab socket. Symmetric to the
// server-side focus specs for createPage / updatePageContent('replace').
// server-side focus specs for createPage / updatePage (markdown 'replace').
import { test } from "node:test";
import assert from "node:assert/strict";
import { DocmostClient } from "../../build/client.js";
@@ -0,0 +1,240 @@
// Mock regression for the FAIL-FAST invalid-node validation (#409).
//
// A structural editor (patch_node / insert_node / update_page_json) given a doc
// whose NESTED child has an absent/unknown `type` (the exact shape the Yjs
// encoder rejects with `Unknown node type: undefined`) must throw a RICH,
// path-anchored error BEFORE it ever opens a collab session or takes a page
// lock. We prove the fail-fast by standing up a collab stack whose HTTP handler
// records EVERY request: a correct fail-fast never even fetches the collab
// token (which `getCollabTokenWithReauth`, called AFTER the validation, would
// request), and never drives a document change on the Hocuspocus doc.
//
// The happy path (a well-formed doc) is exercised too: it must reach the collab
// write and succeed, so the gate is not over-eager.
//
// findInvalidNode's per-shape summaries are unit-tested in the package
// (test/find-invalid-node.test.ts); this exercises the END-TO-END wiring through
// the real client methods.
import { test, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { WebSocketServer } from "ws";
import { Hocuspocus } from "@hocuspocus/server";
import { DocmostClient } from "../../build/client.js";
import { buildYDoc } from "../../build/lib/collaboration.js";
// A minimal valid seed doc with a real block id, so the happy-path patch_node
// finds its target.
const SEED_ID = "seed-para-id";
function seedDoc() {
return {
type: "doc",
content: [
{
type: "paragraph",
attrs: { id: SEED_ID },
content: [{ type: "text", text: "seed" }],
},
],
};
}
// Stand up an HTTP server that authenticates + hands out a collab token AND
// upgrades /collab to a Hocuspocus instance seeded with the doc. `state` records
// whether the collab token was ever fetched (proving the write path was entered)
// and whether the Hocuspocus doc ever changed.
async function spawnCollabStack() {
const state = { changed: false, collabTokenFetched: false };
const hocuspocus = new Hocuspocus({
quiet: true,
async onLoadDocument() {
return buildYDoc(seedDoc());
},
async onChange() {
state.changed = true;
},
});
const wss = new WebSocketServer({ noServer: true });
const server = http.createServer((req, res) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => {
if (req.url === "/api/auth/login") {
res.writeHead(200, {
"Content-Type": "application/json",
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
});
res.end(JSON.stringify({ success: true }));
return;
}
if (req.url === "/api/auth/collab-token") {
state.collabTokenFetched = true;
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ data: { token: "collab-jwt" } }));
return;
}
res.writeHead(404, { "Content-Type": "application/json" });
res.end(JSON.stringify({ message: "not found" }));
});
});
server.on("upgrade", (request, socket, head) => {
if (!request.url || !request.url.startsWith("/collab")) {
socket.destroy();
return;
}
wss.handleUpgrade(request, socket, head, (ws) => {
hocuspocus.handleConnection(ws, request);
});
});
const baseURL = await new Promise((resolve) => {
server.listen(0, "127.0.0.1", () => {
const { port } = server.address();
resolve(`http://127.0.0.1:${port}/api`);
});
});
openStacks.push({ server, hocuspocus });
return { state, baseURL };
}
const openStacks = [];
after(async () => {
await Promise.all(
openStacks.map(
({ server, hocuspocus }) =>
new Promise((resolve) => {
server.close(() => {
Promise.resolve(hocuspocus.destroy?.()).finally(resolve);
});
}),
),
);
});
const PAGE = "11111111-1111-4111-8111-111111111111";
// A node whose NESTED text leaf is missing "type":"text" (dominant #409 shape).
const nestedTypelessNode = () => ({
type: "paragraph",
content: [{ text: "oops", marks: [] }],
});
// A node with a NESTED unknown type NAME (typo).
const nestedUnknownTypeNode = () => ({
type: "paragraph",
content: [{ type: "paragraf", content: [{ type: "text", text: "x" }] }],
});
test("patch_node fails fast on a nested typeless node — no collab connection", async () => {
const { state, baseURL } = await spawnCollabStack();
const client = new DocmostClient(baseURL, "user@example.com", "pw");
await assert.rejects(
() => client.patchNode(PAGE, SEED_ID, nestedTypelessNode()),
(err) => {
assert.match(err.message, /patch_node: invalid node/);
assert.match(err.message, /missing "type"/);
assert.match(err.message, /content\[0\]/); // path-anchored
return true;
},
);
assert.equal(
state.collabTokenFetched,
false,
"must NOT fetch a collab token — validation runs before getCollabTokenWithReauth",
);
assert.equal(state.changed, false, "the collab doc must never be written");
});
test("insert_node fails fast on a nested UNKNOWN type — no collab connection", async () => {
const { state, baseURL } = await spawnCollabStack();
const client = new DocmostClient(baseURL, "user@example.com", "pw");
await assert.rejects(
() =>
client.insertNode(PAGE, nestedUnknownTypeNode(), {
position: "append",
}),
(err) => {
assert.match(err.message, /insert_node: invalid node/);
assert.match(err.message, /unknown node type "paragraf"/);
return true;
},
);
assert.equal(state.collabTokenFetched, false);
assert.equal(state.changed, false);
});
test("update_page_json fails fast on a nested typeless node — no collab connection", async () => {
const { state, baseURL } = await spawnCollabStack();
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const badDoc = {
type: "doc",
content: [{ type: "paragraph", content: [{ text: "oops" }] }],
};
await assert.rejects(
() => client.updatePageJson(PAGE, badDoc),
(err) => {
// update_page_json runs validateDocStructure first (string-type check),
// which already rejects a typeless node — so the message may come from
// either guard, but the write must not happen.
assert.match(err.message, /type/i);
return true;
},
);
assert.equal(state.collabTokenFetched, false);
assert.equal(state.changed, false);
});
test("update_page_json fails fast on a nested UNKNOWN type name — rich #409 message", async () => {
const { state, baseURL } = await spawnCollabStack();
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// validateDocStructure passes (type is a string); assertValidNodeShape must
// catch the unknown schema name and produce the rich path-anchored message.
const badDoc = {
type: "doc",
content: [{ type: "paragraf", content: [{ type: "text", text: "x" }] }],
};
await assert.rejects(
() => client.updatePageJson(PAGE, badDoc),
(err) => {
assert.match(err.message, /update_page_json: invalid node/);
assert.match(err.message, /unknown node type "paragraf"/);
return true;
},
);
assert.equal(state.collabTokenFetched, false);
assert.equal(state.changed, false);
});
test("patch_node with a well-formed node proceeds to the collab write", async () => {
const { state, baseURL } = await spawnCollabStack();
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const result = await client.patchNode(PAGE, SEED_ID, {
type: "paragraph",
content: [{ type: "text", text: "replacement" }],
});
assert.equal(result.success, true);
assert.equal(result.replaced, 1);
assert.equal(
state.collabTokenFetched,
true,
"a valid node must reach the collab write path",
);
assert.equal(state.changed, true, "the collab doc must be written");
});
@@ -0,0 +1,440 @@
// Mock-HTTP tests for the cursor-pagination migration in DocmostClient (#442).
//
// The server switched its list endpoints from OFFSET (`page`) to CURSOR
// (`cursor`/`nextCursor`) pagination, and the global ValidationPipe silently
// strips the obsolete `page` field — so the old offset loops re-fetched page
// one forever (hasNextPage stuck true), dropping every item past the first
// page. These tests pin the new cursor behaviour and the immovable-cursor
// guard that prevents a silent spin/duplication if the protocol drifts again.
//
// A local http.createServer stands in for Docmost so everything stays
// deterministic and offline (same harness style as reauth.test.mjs).
import { test, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { DocmostClient } from "../../build/client.js";
function readBody(req) {
return new Promise((resolve) => {
let raw = "";
req.on("data", (chunk) => {
raw += chunk;
});
req.on("end", () => resolve(raw));
});
}
function startServer(handler) {
return new Promise((resolve) => {
const server = http.createServer(handler);
server.listen(0, "127.0.0.1", () => {
const { port } = server.address();
resolve({ server, baseURL: `http://127.0.0.1:${port}/api` });
});
});
}
function closeServer(server) {
return new Promise((resolve) => server.close(resolve));
}
function sendJson(res, status, obj, extraHeaders = {}) {
res.writeHead(status, { "Content-Type": "application/json", ...extraHeaders });
res.end(JSON.stringify(obj));
}
const openServers = [];
async function spawn(handler) {
const { server, baseURL } = await startServer(handler);
openServers.push(server);
return { server, baseURL };
}
after(async () => {
await Promise.all(openServers.map((s) => closeServer(s)));
});
// A login handler shared by every server below.
function handleLogin(req, res) {
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
});
return true;
}
return false;
}
// -----------------------------------------------------------------------------
// 1) listSidebarPages: collects every cursor page; #requests == #pages.
// -----------------------------------------------------------------------------
test("listSidebarPages walks all cursor pages and collects every item", async () => {
// Three pages keyed by the cursor the client sends back.
const PAGES = {
"": { items: [{ id: "a" }, { id: "b" }], nextCursor: "c1" },
c1: { items: [{ id: "c" }, { id: "d" }], nextCursor: "c2" },
c2: { items: [{ id: "e" }], nextCursor: null },
};
let requests = 0;
const sentLimits = [];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/sidebar-pages") {
requests++;
const body = JSON.parse(raw || "{}");
sentLimits.push(body.limit);
const page = PAGES[body.cursor ?? ""] ?? { items: [], nextCursor: null };
sendJson(res, 200, {
success: true,
data: {
items: page.items,
meta: {
hasNextPage: page.nextCursor != null,
nextCursor: page.nextCursor,
},
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.listSidebarPages("space-1");
assert.equal(requests, 3, "one request per cursor page");
assert.deepEqual(
all.map((p) => p.id),
["a", "b", "c", "d", "e"],
"all items across all pages collected in order",
);
assert.ok(
sentLimits.every((l) => l === 100),
"requests limit:100 (server-side max)",
);
});
// -----------------------------------------------------------------------------
// 2) REGRESSION on the bug class: server IGNORES the cursor param and always
// returns page one with hasNextPage:true -> the immovable-cursor guard must
// terminate the loop with no duplicates, NOT spin to MAX_PAGES.
// -----------------------------------------------------------------------------
test("listSidebarPages terminates (no dups) when the server ignores the cursor", async () => {
let requests = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/sidebar-pages") {
requests++;
// Always the SAME first page with hasNextPage:true and the SAME cursor,
// exactly as a server that no longer understands our pagination param.
sendJson(res, 200, {
success: true,
data: {
items: [{ id: "x1" }, { id: "x2" }],
meta: { hasNextPage: true, nextCursor: "stuck" },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.listSidebarPages("space-1");
// Request 1 (no cursor) gets "stuck"; request 2 (cursor "stuck") gets "stuck"
// again -> guard trips. Far below the MAX_PAGES=50 ceiling; no runaway dups.
assert.equal(requests, 2, "stops as soon as the cursor stops moving");
assert.equal(all.length, 4, "no runaway accumulation / duplication");
});
// -----------------------------------------------------------------------------
// 3a) enumerateSpacePages happy path: a SINGLE /pages/tree request.
// -----------------------------------------------------------------------------
test("enumerateSpacePages (via list_pages tree) uses one /pages/tree request", async () => {
let treeRequests = 0;
let sidebarRequests = 0;
let treeBody = null;
const NODES = [
{ id: "r1", slugId: "r1s", title: "Root 1", parentPageId: null, hasChildren: true, spaceId: "space-1", position: "a", icon: null, canEdit: true },
{ id: "c1", slugId: "c1s", title: "Child 1", parentPageId: "r1", hasChildren: false, spaceId: "space-1", position: "a", icon: null, canEdit: true },
{ id: "r2", slugId: "r2s", title: "Root 2", parentPageId: null, hasChildren: false, spaceId: "space-1", position: "b", icon: null, canEdit: true },
];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
treeRequests++;
treeBody = JSON.parse(raw || "{}");
sendJson(res, 200, { success: true, data: { items: NODES } });
return;
}
if (req.url === "/api/pages/sidebar-pages") {
sidebarRequests++;
sendJson(res, 200, { success: true, data: { items: [], meta: {} } });
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// list_pages tree:true -> enumerateSpacePages(spaceId) -> buildPageTree.
const tree = await client.listPages("space-1", 50, true);
assert.equal(treeRequests, 1, "exactly one /pages/tree request for the space");
assert.equal(sidebarRequests, 0, "no per-node sidebar BFS requests");
assert.deepEqual(treeBody, { spaceId: "space-1" }, "space scope posts spaceId only");
// buildPageTree nests c1 under r1; two roots at the top level.
assert.equal(tree.length, 2, "two root nodes");
const r1 = tree.find((n) => n.id === "r1");
assert.equal(r1.children.length, 1, "child nested under its root");
assert.equal(r1.children[0].id, "c1");
});
// -----------------------------------------------------------------------------
// 3b) enumerateSpacePages fallback: /pages/tree 404 -> cursor BFS via sidebar.
// -----------------------------------------------------------------------------
test("enumerateSpacePages falls back to the cursor BFS on /pages/tree 404", async () => {
let treeRequests = 0;
const sidebarCalls = [];
// Root level: one root with children. Child level (pageId=r1): one leaf.
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
treeRequests++;
// Stock upstream Docmost has no /pages/tree.
sendJson(res, 404, { message: "Not Found" });
return;
}
if (req.url === "/api/pages/sidebar-pages") {
const body = JSON.parse(raw || "{}");
sidebarCalls.push(body.pageId ?? "<root>");
if (!body.pageId) {
sendJson(res, 200, {
success: true,
data: {
items: [
{ id: "r1", title: "Root", parentPageId: null, hasChildren: true },
],
meta: { hasNextPage: false, nextCursor: null },
},
});
} else if (body.pageId === "r1") {
sendJson(res, 200, {
success: true,
data: {
items: [
{ id: "c1", title: "Leaf", parentPageId: "r1", hasChildren: false },
],
meta: { hasNextPage: false, nextCursor: null },
},
});
} else {
sendJson(res, 200, {
success: true,
data: { items: [], meta: { hasNextPage: false, nextCursor: null } },
});
}
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const tree = await client.listPages("space-1", 50, true);
assert.ok(treeRequests >= 1, "the tree endpoint was attempted first");
assert.deepEqual(
sidebarCalls,
["<root>", "r1"],
"fell back to the sidebar BFS: roots then the root's children",
);
assert.equal(tree.length, 1, "one root in the built tree");
assert.equal(tree[0].children[0].id, "c1", "leaf nested via the BFS");
});
// -----------------------------------------------------------------------------
// 3c) enumerateSpacePages fallback SUBTREE: /pages/tree 404 + a rootPageId ->
// the ROOT page itself must be seeded (via getPageRaw) so its own comments
// aren't dropped. listSidebarPages(spaceId, root) returns only the root's
// CHILDREN, so without the seed the root would be absent. (Finding 1.)
// -----------------------------------------------------------------------------
test("enumerateSpacePages fallback subtree seeds the ROOT page itself", async () => {
const sidebarCalls = [];
let infoRequests = 0;
const commentedPages = [];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
// Stock upstream Docmost -> fall back to the BFS.
sendJson(res, 404, { message: "Not Found" });
return;
}
if (req.url === "/api/pages/info") {
// getPageRaw for the root seed. Shape mirrors a real page-info response.
infoRequests++;
sendJson(res, 200, {
success: true,
data: { id: "root", title: "Root", spaceId: "space-1", hasChildren: true },
});
return;
}
if (req.url === "/api/pages/sidebar-pages") {
const body = JSON.parse(raw || "{}");
sidebarCalls.push(body.pageId ?? "<root>");
// Children of the root: one leaf. (Root itself is NOT in this list.)
const items =
body.pageId === "root"
? [{ id: "leaf", title: "Leaf", parentPageId: "root", hasChildren: false }]
: [];
sendJson(res, 200, {
success: true,
data: { items, meta: { hasNextPage: false, nextCursor: null } },
});
return;
}
if (req.url === "/api/comments") {
const body = JSON.parse(raw || "{}");
commentedPages.push(body.pageId);
const items =
body.pageId === "root"
? [{ id: "cm1", createdAt: "2030-01-01T00:00:00.000Z", content: null }]
: [];
sendJson(res, 200, {
success: true,
data: { items, meta: { nextCursor: null } },
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// checkNewComments(space, since, parentPageId) exercises the subtree fallback.
const result = await client.checkNewComments(
"space-1",
"2020-01-01T00:00:00.000Z",
"root",
);
assert.equal(infoRequests, 1, "root was seeded via one getPageRaw");
assert.equal(sidebarCalls[0], "root", "BFS walked the root's children");
assert.ok(
commentedPages.includes("root"),
"the ROOT page is in scope (its comments were fetched) — not dropped",
);
assert.ok(commentedPages.includes("leaf"), "the descendant is in scope too");
assert.equal(result.checkedPages, 2, "root + one descendant scanned");
assert.equal(result.totalNewComments, 1, "the root's fresh comment found");
});
// -----------------------------------------------------------------------------
// 5) listComments immovable-cursor guard: the server IGNORES the cursor and
// keeps returning the same nextCursor -> the loop must terminate (no
// infinite loop, no duplicates), not spin forever. (Finding 4.)
// -----------------------------------------------------------------------------
test("listComments terminates (no dups) when the server ignores the cursor", async () => {
let requests = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/comments") {
requests++;
// Always the SAME page with the SAME nextCursor, as a server that no
// longer advances the cursor would.
sendJson(res, 200, {
success: true,
data: {
items: [{ id: "cm1", createdAt: "2030-01-01T00:00:00.000Z", content: null }],
meta: { nextCursor: "stuck" },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const { items } = await client.listComments("page-1", true);
// Request 1 (no cursor) gets "stuck"; request 2 (cursor "stuck") gets "stuck"
// again -> guard trips. Bounded far below MAX_PAGES=50, no runaway dups.
assert.equal(requests, 2, "stops as soon as the cursor stops moving");
assert.equal(items.length, 2, "no runaway accumulation / duplication");
});
// -----------------------------------------------------------------------------
// 4) check_new_comments subtree: the root is included in scope WITHOUT a
// separate getPageRaw (/pages/info) request for the parent.
// -----------------------------------------------------------------------------
test("checkNewComments subtree includes the root without a separate getPageRaw", async () => {
let pageInfoRequests = 0;
let treeBody = null;
const commentedPages = [];
// /pages/tree (subtree) returns the parent itself plus a descendant, exactly
// as getPageAndDescendants seeds with id = parentPageId.
const NODES = [
{ id: "parent", title: "Parent", parentPageId: null, hasChildren: true },
{ id: "kid", title: "Kid", parentPageId: "parent", hasChildren: false },
];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
treeBody = JSON.parse(raw || "{}");
sendJson(res, 200, { success: true, data: { items: NODES } });
return;
}
if (req.url === "/api/pages/info") {
// If checkNewComments still fetched the parent separately this would fire.
pageInfoRequests++;
sendJson(res, 200, { success: true, data: { id: "parent" } });
return;
}
if (req.url === "/api/comments") {
const body = JSON.parse(raw || "{}");
commentedPages.push(body.pageId);
// One fresh comment on the parent, none elsewhere.
const items =
body.pageId === "parent"
? [{ id: "cm1", createdAt: "2030-01-01T00:00:00.000Z", content: null }]
: [];
sendJson(res, 200, {
success: true,
data: { items, meta: { nextCursor: null } },
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const result = await client.checkNewComments(
"space-1",
"2020-01-01T00:00:00.000Z",
"parent",
);
assert.equal(pageInfoRequests, 0, "no separate getPageRaw for the root");
assert.deepEqual(treeBody, { pageId: "parent" }, "subtree scope posts pageId");
assert.ok(
commentedPages.includes("parent"),
"the root itself is in scope (comments fetched for it)",
);
assert.ok(commentedPages.includes("kid"), "descendants are in scope too");
assert.equal(result.checkedPages, 2, "root + one descendant scanned");
assert.equal(result.totalNewComments, 1, "the root's fresh comment found");
});
+26 -19
View File
@@ -297,12 +297,12 @@ test("a response with ONLY authTokenRefresh (no authToken) rejects login", async
// -----------------------------------------------------------------------------
// 5) paginateAll loop guards.
// -----------------------------------------------------------------------------
test("paginateAll stops at the MAX_PAGES cap when hasNextPage is always true", async () => {
test("paginateAll stops at the MAX_PAGES cap when the server always issues a fresh cursor", async () => {
let pageRequests = 0;
const LIMIT = 100;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
const body = JSON.parse((await readBody(req)) || "{}");
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
@@ -311,15 +311,18 @@ test("paginateAll stops at the MAX_PAGES cap when hasNextPage is always true", a
}
if (req.url === "/api/spaces") {
pageRequests++;
// Always return a FULL page (== requested limit) AND hasNextPage:true.
// Both the page-length check and the hasNextPage flag say "keep going",
// so only the MAX_PAGES ceiling can stop the loop.
// Always return a FULL page AND hasNextPage:true with a FRESH nextCursor
// that differs from the one the client just sent, so the immovable-cursor
// guard never trips — only the MAX_PAGES ceiling can stop the loop.
const items = Array.from({ length: LIMIT }, (_, i) => ({
id: `s-${pageRequests}-${i}`,
}));
sendJson(res, 200, {
success: true,
data: { items, meta: { hasNextPage: true } },
data: {
items,
meta: { hasNextPage: true, nextCursor: `cursor-${pageRequests}` },
},
});
return;
}
@@ -338,7 +341,7 @@ test("paginateAll stops at the MAX_PAGES cap when hasNextPage is always true", a
assert.equal(all.length, 50 * LIMIT, "accumulates one full page per request");
});
test("paginateAll stops early on a short page even if hasNextPage is true", async () => {
test("paginateAll stops on the immovable-cursor guard when the server ignores the cursor param", async () => {
let pageRequests = 0;
const LIMIT = 100;
@@ -352,15 +355,17 @@ test("paginateAll stops early on a short page even if hasNextPage is true", asyn
}
if (req.url === "/api/spaces") {
pageRequests++;
// First page is full; second page is SHORT (fewer than limit). The short
// page must stop the loop immediately even though hasNextPage stays true.
const count = pageRequests === 1 ? LIMIT : 3;
const items = Array.from({ length: count }, (_, i) => ({
id: `s-${pageRequests}-${i}`,
}));
// The bug class: the server IGNORES the pagination param and keeps
// returning page one with hasNextPage:true and the SAME nextCursor. The
// immovable-cursor guard must stop the loop instead of spinning to
// MAX_PAGES and duplicating items.
const items = Array.from({ length: LIMIT }, (_, i) => ({ id: `s-${i}` }));
sendJson(res, 200, {
success: true,
data: { items, meta: { hasNextPage: true } },
data: {
items,
meta: { hasNextPage: true, nextCursor: "stuck" },
},
});
return;
}
@@ -370,8 +375,10 @@ test("paginateAll stops early on a short page even if hasNextPage is true", asyn
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.paginateAll("/spaces", {}, LIMIT);
assert.equal(pageRequests, 2, "stops right after the first short page");
assert.equal(all.length, LIMIT + 3, "full page + short page accumulated");
// Request 1 sends no cursor and receives "stuck"; request 2 sends "stuck" and
// receives "stuck" again -> guard trips after exactly two requests, no dups.
assert.equal(pageRequests, 2, "stops once the cursor stops moving");
assert.equal(all.length, 2 * LIMIT, "no runaway accumulation past the guard");
});
test("paginateAll handles both {data:{items,meta}} and {items,meta} envelopes", async () => {
@@ -387,16 +394,16 @@ test("paginateAll handles both {data:{items,meta}} and {items,meta} envelopes",
}
if (req.url === "/api/groups") {
bareRequests.push(1);
// Page 1: full page, hasNextPage true. Page 2: short page -> stop.
// Page 1: hasNextPage true with a next cursor. Page 2: no next -> stop.
if (bareRequests.length === 1) {
sendJson(res, 200, {
items: Array.from({ length: 100 }, (_, i) => ({ id: `g${i}` })),
meta: { hasNextPage: true },
meta: { hasNextPage: true, nextCursor: "c2" },
});
} else {
sendJson(res, 200, {
items: [{ id: "tail" }],
meta: { hasNextPage: false },
meta: { hasNextPage: false, nextCursor: null },
});
}
return;
@@ -1,224 +0,0 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, resolve } from "node:path";
import { DocmostClient } from "../../build/index.js";
// Drift guard for the THIRD hand-written layer of the AI tool set (issue #193,
// layer 3): the in-app server hand-mirrors the DocmostClient method signatures
// it consumes as the `DocmostClientLike` interface in
// apps/server/src/core/ai-chat/tools/docmost-client.loader.ts ("Signatures here
// mirror that file exactly"). That mirror lives across the ESM(mcp)/CJS(server)
// boundary and the package ships NO .d.ts, so the server typecheck cannot verify
// the names against the real class — a rename/removal in client.ts would surface
// only as a runtime "x is not a function" inside an agent tool call.
//
// SCOPE: this guard checks the method-NAME set only, not signatures. It pins the
// contract from the mcp side (ESM, where the real class is directly importable):
// every method the embedding host depends on MUST exist as a function on a real
// DocmostClient instance. If you rename/remove a client method, this fails here
// AND you must update DocmostClientLike to match. It does NOT verify parameter or
// return-type parity — signature drift between the hand-mirror and client.ts can
// still ship silently; full signature/type parity is the deferred staged-plan
// item below.
//
// Keep the HOST_CONTRACT_METHODS NAME list aligned with the method NAMES declared
// in the server's DocmostClientLike interface (the in-app per-user tool adapter
// only — it is a SUBSET of the DocmostClient surface — covers only what the in-app adapter
// consumes; the standalone MCP transport (packages/mcp/src/index.ts) calls additional
// client methods (deleteComment/updateComment) that this guard does NOT track — the
// MCP transport's own typecheck covers those. insertImage/replaceImage/insertFootnote
// were MCP-only but are now in-app-consumed too (#410), so they ARE tracked below. Full type-derivation
// of DocmostClientLike from this class is deferred (see the staged plan in
// docmost-client.loader.ts): the package emits no declarations and the real
// (inferred, concrete) return types conflict with the host's loose
// `Record<string,unknown>` + `as`-cast result handling.
const HOST_CONTRACT_METHODS = [
// read
"search",
"getPage",
"getPageRaw",
"getWorkspace",
"getSpaces",
"listPages",
"listSidebarPages",
"getOutline",
"getPageJson",
"getNode",
"searchInPage",
"getTable",
"listComments",
"getComment",
"checkNewComments",
"listShares",
"listPageHistory",
"getPageHistory",
"diffPageVersions",
"exportPageMarkdown",
// write (page)
"createPage",
"updatePage",
"renamePage",
"movePage",
"deletePage",
"editPageText",
"patchNode",
"insertNode",
"deleteNode",
"updatePageJson",
"tableInsertRow",
"tableDeleteRow",
"tableUpdateCell",
"copyPageContent",
"importPageMarkdown",
"sharePage",
"unsharePage",
"restorePageVersion",
"transformPage",
"stashPage",
// write (image / footnote) — MCP-only until #410 promoted them to in-app tools
"insertImage",
"replaceImage",
"insertFootnote",
// draw.io diagrams (#423, stage 1) — read + create + optimistic-locked update
"drawioGet",
"drawioCreate",
"drawioUpdate",
// write (comment)
"createComment",
"resolveComment",
];
test("DocmostClient implements every method the in-app DocmostClientLike mirror declares", () => {
// The constructor is side-effect-free (no network/login on construction): it
// only stores config and creates an axios instance, so it is safe to build a
// throwaway instance here with a dummy token provider.
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "test-token",
});
const missing = HOST_CONTRACT_METHODS.filter(
(name) => typeof client[name] !== "function",
);
assert.deepEqual(
missing,
[],
`DocmostClient is missing host-contract method(s): ${missing.join(", ")}. ` +
`Update packages/mcp/src/client.ts and/or the server's DocmostClientLike ` +
`interface (apps/server/src/core/ai-chat/tools/docmost-client.loader.ts) ` +
`so the hand-mirrored method NAMES stay aligned (this guards names only, ` +
`not signatures).`,
);
});
test("HOST_CONTRACT_METHODS has no duplicates", () => {
assert.equal(
new Set(HOST_CONTRACT_METHODS).size,
HOST_CONTRACT_METHODS.length,
);
});
// Parse the method names declared in the server's `DocmostClientLike` interface
// body. We read the .ts source as plain text (no TS compiler dep, and the file
// lives in the CJS server tree across the ESM boundary): scan from the
// `export interface DocmostClientLike {` line to its closing brace at column 0,
// matching member-signature lines like ` methodName(`. Nested param-object
// braces (`opts: { ... }`) are indented, so only the interface's own closing
// `}` (column 0) ends the scan.
function parseDocmostClientLikeMethods() {
const here = dirname(fileURLToPath(import.meta.url));
// packages/mcp/test/unit -> repo root is four levels up.
const loaderPath = resolve(
here,
"../../../../apps/server/src/core/ai-chat/tools/docmost-client.loader.ts",
);
let source;
try {
source = readFileSync(loaderPath, "utf8");
} catch (err) {
if (err && err.code === "ENOENT") {
throw new Error(
`Expected monorepo layout; server tree at ${loaderPath} not found. ` +
`This drift-guard reads the server's DocmostClientLike interface via a ` +
`fixed relative path and must run from inside the monorepo checkout.`,
);
}
throw err;
}
const lines = source.split(/\r?\n/);
const startIdx = lines.findIndex((l) =>
/^export interface DocmostClientLike\s*\{/.test(l),
);
assert.notEqual(
startIdx,
-1,
`Could not find "export interface DocmostClientLike {" in ${loaderPath}. ` +
`If the interface was renamed/moved, update this drift-guard test.`,
);
const methods = [];
let closed = false;
// Track whether we are inside a `/* ... */` block comment. Inner lines of a
// block comment need NOT start with `*`, so a `name(` line inside one would be
// falsely parsed as an interface method without this. (`//` line comments can
// never match the method regex below since they start with `/`.)
let inBlockComment = false;
for (let i = startIdx + 1; i < lines.length; i++) {
const line = lines[i];
if (inBlockComment) {
// Stay in the block until we see its closing `*/`.
if (line.includes("*/")) inBlockComment = false;
continue;
}
// Enter a block comment only when it opens without closing on the same line;
// a self-contained `/* ... */` on one line cannot precede a method name we
// care about (such lines start with `/`, so the method regex won't match).
if (line.includes("/*") && !line.includes("*/")) {
inBlockComment = true;
continue;
}
if (/^\}/.test(line)) {
closed = true;
break;
}
// Method-name match: a TS identifier (letters/digits/`_`/`$`, not starting
// with a digit) optionally followed by a generic clause (`method<T>(`), then
// the opening paren of the signature.
const m = /^\s*([A-Za-z_$][A-Za-z0-9_$]*)\s*(?:<[^>]*>)?\(/.exec(line);
if (m) methods.push(m[1]);
}
assert.ok(
closed,
`Did not find the closing brace of DocmostClientLike in ${loaderPath}.`,
);
assert.ok(
methods.length > 0,
`Parsed zero methods from DocmostClientLike in ${loaderPath} — the parser ` +
`is likely out of date with the interface formatting.`,
);
return methods;
}
// The point of the guard is to protect the DocmostClientLike mirror <-> client.ts
// link, but HOST_CONTRACT_METHODS is itself a HAND-COPY of that interface kept in
// sync manually. The list<->interface link must be tested too: a method consumed
// by the adapter and added to DocmostClientLike but forgotten here (or removed
// from the interface but left here) would otherwise escape both the server
// typecheck (pkg emits no .d.ts) and the first test above (name not in the list).
// Assert the two agree BOTH ways.
test("HOST_CONTRACT_METHODS exactly mirrors the server's DocmostClientLike interface", () => {
const interfaceMethods = parseDocmostClientLikeMethods();
assert.deepEqual(
[...HOST_CONTRACT_METHODS].sort(),
[...interfaceMethods].sort(),
`HOST_CONTRACT_METHODS has drifted from the DocmostClientLike interface in ` +
`apps/server/src/core/ai-chat/tools/docmost-client.loader.ts. Add/remove ` +
`method names in HOST_CONTRACT_METHODS so it lists EXACTLY the methods ` +
`declared in that interface (both directions are checked).`,
);
});
@@ -0,0 +1,101 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
import { computeRegistryStamp } from "../../scripts/gen-registry-stamp.mjs";
import { REGISTRY_STAMP } from "../../build/index.js";
// Guard tests for the build/src-skew stamp (issue #447). The codegen script
// exports `computeRegistryStamp(sourceText)` — a sha256 over normalized source
// text (CRLF->LF, single trailing newline stripped). The in-app loader
// (apps/server/.../docmost-client.loader.ts) DUPLICATES that normalize+sha256 to
// recompute the stamp from src and refuse a stale build. These tests pin the
// algorithm's behaviour AND assert the built stamp matches the current src, so a
// stale generated file OR a normalize divergence reddens.
const __dirname = dirname(fileURLToPath(import.meta.url));
const TOOL_SPECS_PATH = join(__dirname, "..", "..", "src", "tool-specs.ts");
test("computeRegistryStamp is deterministic: same input -> same hash", () => {
const input = "export const X = 1;\nexport const Y = 2;\n";
assert.equal(computeRegistryStamp(input), computeRegistryStamp(input));
});
test("computeRegistryStamp returns a 64-char lowercase hex sha256", () => {
const stamp = computeRegistryStamp("anything");
assert.match(stamp, /^[0-9a-f]{64}$/);
});
test("normalizes CRLF vs LF: the same content hashes equal", () => {
const lf = "line1\nline2\nline3";
const crlf = "line1\r\nline2\r\nline3";
assert.equal(computeRegistryStamp(crlf), computeRegistryStamp(lf));
});
test("normalizes a trailing newline: with/without a final \\n hashes equal", () => {
const noTrailing = "alpha\nbeta";
const trailing = "alpha\nbeta\n";
assert.equal(computeRegistryStamp(trailing), computeRegistryStamp(noTrailing));
});
test("a CRLF checkout WITH a trailing CRLF still hashes equal to bare LF", () => {
// A worst-case Windows checkout: CRLF line endings + a trailing CRLF. Both the
// \r\n->\n replace and the trailing-newline strip must apply for parity.
const bare = "alpha\nbeta";
const crlfTrailing = "alpha\r\nbeta\r\n";
assert.equal(
computeRegistryStamp(crlfTrailing),
computeRegistryStamp(bare),
);
});
test("a real content change hashes differently", () => {
const before = "export const description = 'search a page';\n";
const after = "export const description = 'search a PAGE';\n";
assert.notEqual(computeRegistryStamp(before), computeRegistryStamp(after));
});
// Only a SINGLE trailing newline is stripped — a second blank line is content and
// must change the hash. This pins the exact `/\n$/` semantics the loader mirrors.
test("only ONE trailing newline is stripped (two differ from one)", () => {
assert.notEqual(
computeRegistryStamp("x\n"),
computeRegistryStamp("x\n\n"),
);
});
// Cross-impl equality against a fixed, documented input. The SAME literal input
// and expected hash are asserted in the server-side jest test
// (docmost-client.loader.spec.ts). If either side's normalize+sha256 ever
// diverges, one of the two tests reddens. Input exercises BOTH normalize steps.
test("fixed-input hash matches the documented cross-impl value", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const EXPECTED =
"683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83";
assert.equal(computeRegistryStamp(FIXED_INPUT), EXPECTED);
});
// DESYNC GUARD (covers reviewer suggestion 2). Recompute the stamp from the
// actual src/tool-specs.ts and assert it equals the REGISTRY_STAMP baked into the
// freshly-built build/index.js. This reddens if the generated file is stale OR if
// the codegen normalize ever diverges from what produced the built stamp.
test("built REGISTRY_STAMP equals the stamp recomputed from src/tool-specs.ts", () => {
const source = readFileSync(TOOL_SPECS_PATH, "utf8");
assert.equal(computeRegistryStamp(source), REGISTRY_STAMP);
});
// Sanity: the fixed-input helper computes the SAME way the codegen does, proving
// the EXPECTED constant above is not an arbitrary magic value but the documented
// normalize+sha256 of FIXED_INPUT. Belt-and-braces so a bad EXPECTED can't hide a
// real regression.
test("the documented EXPECTED constant is the normalize+sha256 of FIXED_INPUT", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const normalized = FIXED_INPUT.replace(/\r\n/g, "\n").replace(/\n$/, "");
const expected = createHash("sha256")
.update(normalized, "utf8")
.digest("hex");
assert.equal(computeRegistryStamp(FIXED_INPUT), expected);
});
@@ -1,82 +0,0 @@
// Guard: every tool the MCP server registers must be routed by intent in
// SERVER_INSTRUCTIONS — the editing guide clients receive in the initialize
// result. Without this, new tools silently rot out of the guide and agents
// never learn to pick them (the guide once omitted 17 of 41 tools, including
// get_outline, which pushed agents into fetching whole documents for block
// ids). Tool names are extracted from the SOURCE (index.ts inline
// registrations + tool-specs.ts shared specs) so a registration added either
// way is caught; the guide text itself is imported from the build so the test
// checks what actually ships.
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
import { SERVER_INSTRUCTIONS } from "../../build/index.js";
const HERE = dirname(fileURLToPath(import.meta.url));
const SRC = join(HERE, "..", "..", "src");
// Tools DELIBERATELY absent from the guide. Keep this list minimal and
// justify every entry — the default is: every tool gets routed.
const EXCEPTIONS = new Set([
// Trivial and self-explanatory; carries no routing decision.
"get_workspace",
]);
/**
* Extract every registered tool name from the source. Two registration
* mechanisms exist and both are covered:
* - inline `server.registerTool("name", ...)` calls in index.ts;
* - shared specs in tool-specs.ts (`mcpName: 'name'`), registered via
* registerShared(SHARED_TOOL_SPECS.x, ...).
*/
function registeredToolNames() {
const indexSrc = readFileSync(join(SRC, "index.ts"), "utf8");
const specsSrc = readFileSync(join(SRC, "tool-specs.ts"), "utf8");
const names = new Set();
for (const m of indexSrc.matchAll(/registerTool\(\s*"([a-z0-9_]+)"/g)) {
names.add(m[1]);
}
for (const m of specsSrc.matchAll(/mcpName:\s*['"]([a-z0-9_]+)['"]/g)) {
names.add(m[1]);
}
return names;
}
test("every registered tool is mentioned in SERVER_INSTRUCTIONS", () => {
const names = registeredToolNames();
// Sanity: if extraction regressed (regex drift), fail loudly rather than
// vacuously passing on an empty set.
assert.ok(
names.size >= 40,
`sanity: expected to extract 40+ registered tools, got ${names.size}` +
"the extraction regexes in this test likely drifted from the source",
);
const missing = [...names]
.filter((n) => !EXCEPTIONS.has(n))
// \b<name>\b: `_` is a word char, so \bget_page\b does NOT match inside
// get_page_json — a tool can't hide behind a longer sibling's mention.
.filter((n) => !new RegExp(`\\b${n}\\b`).test(SERVER_INSTRUCTIONS))
.sort();
assert.deepEqual(
missing,
[],
`tools missing from SERVER_INSTRUCTIONS: ${missing.join(", ")}` +
"update the guide in packages/mcp/src/index.ts (see its MAINTENANCE " +
"RULE comment), or add a justified entry to EXCEPTIONS here",
);
});
test("EXCEPTIONS entries are real registered tools", () => {
// A stale exception (tool renamed/removed) must be cleaned up, otherwise
// the list quietly grows past its purpose.
const names = registeredToolNames();
for (const name of EXCEPTIONS) {
assert.ok(
names.has(name),
`EXCEPTIONS entry "${name}" is not a registered tool — remove it`,
);
}
});
@@ -0,0 +1,134 @@
// Guard: the GENERATED <tool_inventory> in SERVER_INSTRUCTIONS (issue #448)
// names every tool the server registers. The inventory is BUILT from the
// registry (SHARED_TOOL_SPECS' mcpName/catalogLine + INLINE_MCP_INVENTORY), so
// the shared-registry tools can never drift by construction; this test's job is
// to catch the ONE remaining manual list — INLINE_MCP_INVENTORY — falling out
// of sync with the inline `server.registerTool(...)` calls in index.ts.
//
// It also asserts the composed guide keeps its routing prose (the hand-written
// intent hints) and is a valid non-empty string — the structural guarantees the
// old name-scraper test (server-instructions.test.mjs, now deleted) carried,
// minus its now-redundant per-name prose scrape.
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
import {
SERVER_INSTRUCTIONS,
ROUTING_PROSE,
buildToolInventoryLines,
} from "../../build/server-instructions.js";
const HERE = dirname(fileURLToPath(import.meta.url));
const SRC = join(HERE, "..", "..", "src");
/**
* Every tool name the MCP server registers, scraped from the SOURCE:
* - inline `server.registerTool("name", ...)` calls in index.ts;
* - shared specs in tool-specs.ts (`mcpName: 'name'`), EXCEPT `inAppOnly`
* specs, which the registry loop in index.ts SKIPS on the MCP host (#411).
* Same two registration mechanisms the old guard covered.
*/
function registeredToolNames() {
const indexSrc = readFileSync(join(SRC, "index.ts"), "utf8");
const specsSrc = readFileSync(join(SRC, "tool-specs.ts"), "utf8");
const names = new Set();
for (const m of indexSrc.matchAll(/registerTool\(\s*"([a-z0-9_]+)"/g)) {
names.add(m[1]);
}
// Each spec is one `{ ... }` block; scrape its mcpName but skip a block that
// carries `inAppOnly: true` (not registered on the external MCP host).
for (const block of specsSrc.split(/\n\s{2}\w+:\s*\{/)) {
const nameMatch = block.match(/mcpName:\s*['"]([a-z0-9_]+)['"]/);
if (!nameMatch) continue;
if (/inAppOnly:\s*true/.test(block)) continue;
names.add(nameMatch[1]);
}
return names;
}
test("the generated inventory names every registered tool", () => {
const registered = registeredToolNames();
// Sanity: if the scrape regressed (regex drift), fail loudly rather than
// vacuously passing on an empty set.
assert.ok(
registered.size >= 40,
`sanity: expected 40+ registered tools, got ${registered.size}` +
"the extraction regexes in this test likely drifted from the source",
);
const inventory = new Set(buildToolInventoryLines().map((l) => l.name));
const missing = [...registered].filter((n) => !inventory.has(n)).sort();
assert.deepEqual(
missing,
[],
`tools missing from the generated <tool_inventory>: ${missing.join(", ")}` +
"a SHARED spec is covered automatically; an INLINE MCP-only tool needs a " +
"line added to INLINE_MCP_INVENTORY in src/server-instructions.ts",
);
});
test("the inventory has no phantom tool (every line is a real registered tool)", () => {
const registered = registeredToolNames();
const phantom = buildToolInventoryLines()
.map((l) => l.name)
.filter((n) => !registered.has(n))
.sort();
assert.deepEqual(
phantom,
[],
`<tool_inventory> lists tools that are NOT registered: ${phantom.join(", ")}`,
);
});
// #411: the external MCP surface gains update_page_markdown and LOSES
// import_page_markdown (now inAppOnly). The in-app agent still keeps
// importPageMarkdown — asserted in the server-side contract spec.
test("update_page_markdown is on the MCP surface; import_page_markdown is NOT", () => {
const inventory = new Set(buildToolInventoryLines().map((l) => l.name));
assert.ok(
inventory.has("update_page_markdown"),
"update_page_markdown should be registered on the external MCP surface",
);
assert.ok(
!inventory.has("import_page_markdown"),
"import_page_markdown must be dropped from the external MCP surface (#411)",
);
// And the routing prose no longer points MCP clients at it.
assert.ok(
!ROUTING_PROSE.includes("import_page_markdown"),
"ROUTING_PROSE still mentions the removed import_page_markdown",
);
assert.ok(
ROUTING_PROSE.includes("update_page_markdown"),
"ROUTING_PROSE should mention update_page_markdown",
);
});
test("every inventory line has a non-empty purpose", () => {
for (const line of buildToolInventoryLines()) {
assert.equal(typeof line.purpose, "string");
assert.ok(line.purpose.trim().length > 0, `${line.name}: empty purpose`);
}
});
test("SERVER_INSTRUCTIONS keeps the routing prose and the generated inventory", () => {
assert.equal(typeof SERVER_INSTRUCTIONS, "string");
assert.ok(SERVER_INSTRUCTIONS.length > 0, "SERVER_INSTRUCTIONS is empty");
// Routing prose is spliced in verbatim (the hand-written intent hints).
assert.ok(
SERVER_INSTRUCTIONS.startsWith(ROUTING_PROSE),
"the routing prose is not preserved at the head of the guide",
);
// The generated inventory block is present.
assert.match(SERVER_INSTRUCTIONS, /<tool_inventory>/);
assert.match(SERVER_INSTRUCTIONS, /<\/tool_inventory>/);
// The routing families are still present in the prose.
for (const family of ["READ:", "EDIT:", "PAGES:", "COMMENTS:", "HISTORY:"]) {
assert.ok(
SERVER_INSTRUCTIONS.includes(family),
`routing prose lost its ${family} section`,
);
}
});
@@ -145,3 +145,36 @@ test("no-arg specs (getWorkspace/listSpaces/listShares) omit buildShape", () =>
assert.equal(SHARED_TOOL_SPECS[key].buildShape, undefined, `${key} should be no-arg`);
}
});
// #411: plain-Markdown full-body replace tool, paired with updatePageJson.
test("updatePageMarkdown spec exists, pairs with updatePageJson, builds { pageId, content, title }", () => {
const spec = SHARED_TOOL_SPECS.updatePageMarkdown;
assert.ok(spec, "updatePageMarkdown spec missing");
assert.equal(spec.mcpName, "update_page_markdown");
assert.equal(spec.inAppKey, "updatePageMarkdown");
// Registered on BOTH hosts (a shared spec, no inAppOnly/mcpOnly flag).
assert.notEqual(spec.inAppOnly, true);
assert.notEqual(spec.mcpOnly, true);
// Same tier as its JSON sibling.
assert.equal(spec.tier, SHARED_TOOL_SPECS.updatePageJson.tier);
const shape = spec.buildShape(z);
assert.deepEqual(Object.keys(shape).sort(), ["content", "pageId", "title"]);
// pageId + content required, title optional.
const schema = z.object(shape);
assert.doesNotThrow(() => schema.parse({ pageId: "p1", content: "# Hi" }));
assert.throws(() => schema.parse({ pageId: "p1" }));
// The description must flag the `^[...]` inline-footnote parse path so the
// markdown->footnote canonicalization guarantee stays documented (#411).
assert.match(spec.description, /\^\[/);
});
// #411: import_page_markdown is dropped from the EXTERNAL MCP surface but stays
// available to the in-app agent — encoded as inAppOnly on the shared spec.
test("importPageMarkdown spec is inAppOnly (removed from the external MCP surface, kept in-app)", () => {
const spec = SHARED_TOOL_SPECS.importPageMarkdown;
assert.ok(spec, "importPageMarkdown spec missing");
assert.equal(spec.inAppOnly, true);
// The spec + its client method are NOT deleted — only hidden from the MCP host.
assert.equal(spec.mcpName, "import_page_markdown");
assert.equal(spec.inAppKey, "importPageMarkdown");
});
@@ -0,0 +1,82 @@
// #409: unstorableYjsError diagnostics PRECEDENCE. The opaque Yjs encode failure
// (`Unknown node type: undefined`) is a node-SHAPE problem, so the shared
// findInvalidNode is consulted FIRST and yields a path-anchored node message;
// only a shape-sound doc falls back to findUnstorableAttr (undefined/function/
// etc. attr values). Exercised through `assertYjsEncodable`, which runs the same
// encode + error-wrapping the live write path uses.
import { test } from "node:test";
import assert from "node:assert/strict";
import { assertYjsEncodable } from "../../build/lib/collaboration.js";
import {
findInvalidNode,
findUnstorableAttr,
} from "@docmost/prosemirror-markdown";
const doc = (...content) => ({ type: "doc", content });
test("a nested typeless node yields the rich node-shape message (not an attr hint)", () => {
const bad = doc({
type: "paragraph",
content: [{ text: "oops", marks: [] }], // missing "type":"text"
});
assert.throws(
() => assertYjsEncodable(bad),
(err) => {
assert.match(err.message, /Invalid node:/);
assert.match(err.message, /missing "type"/);
assert.match(err.message, /content\[0\]/);
// It must NOT fall through to the generic attribute sentence.
assert.doesNotMatch(err.message, /Offending attribute/);
assert.doesNotMatch(
err.message,
/attribute likely holds a value Yjs cannot store/,
);
return true;
},
);
});
test("PRECEDENCE: a genuine undefined-attr case is a node-SHAPE-clean case, so the attr fallback fires", () => {
// A doc whose node shapes are ALL valid but that carries a Yjs-unstorable
// undefined attribute. unstorableYjsError checks findInvalidNode FIRST (must
// miss here) and only then findUnstorableAttr (must hit) — this is exactly the
// division of labor that keeps a real attr problem from being mislabelled as a
// node-shape problem, and vice versa. We assert the two helpers directly (the
// wrapper is not exported) because sanitizeForYjs strips undefined attrs before
// the live encoder ever sees them, so this branch cannot be reached through
// assertYjsEncodable without also failing the clone.
const attrProblem = doc({
type: "paragraph",
attrs: { id: "p1", indent: undefined },
content: [{ type: "text", text: "hi" }],
});
// findInvalidNode: shape is clean -> null (so the wrapper does NOT emit
// "Invalid node").
assert.equal(findInvalidNode(attrProblem), null);
// findUnstorableAttr: pinpoints the undefined attr -> the fallback message.
assert.match(findUnstorableAttr(attrProblem) ?? "", /indent \(undefined\)/);
});
test("PRECEDENCE: a node-shape problem is caught by findInvalidNode even when an attr is also unstorable", () => {
// Both a shape problem (typeless nested leaf) AND an unstorable attr exist;
// findInvalidNode wins, so the model is pointed at the node shape (the real
// root cause of `Unknown node type: undefined`), not the attribute.
const both = doc({
type: "paragraph",
attrs: { id: "p1", indent: undefined },
content: [{ text: "oops" }],
});
const shape = findInvalidNode(both);
assert.notEqual(shape, null);
assert.match(shape.summary, /missing "type"/);
});
test("a fully valid document encodes without throwing", () => {
const good = doc({
type: "paragraph",
attrs: { id: "p1" },
content: [{ type: "text", text: "hi" }],
});
assert.doesNotThrow(() => assertYjsEncodable(good));
});
+2
View File
@@ -5,6 +5,8 @@
"moduleResolution": "Node16",
"outDir": "./build",
"rootDir": "./src",
"declaration": true,
"declarationMap": true,
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
@@ -56,6 +56,7 @@ export {
deleteNodeById,
sanitizeForYjs,
findUnstorableAttr,
findInvalidNode,
insertNodeRelative,
readTable,
insertTableRow,
@@ -14,7 +14,9 @@
* `content`, non-object nodes, and absent `attrs` are tolerated.
*/
import { getSchema } from "@tiptap/core";
import { stripInlineMarkdown } from "./text-normalize.js";
import { docmostExtensions } from "./docmost-schema.js";
/** Deep-clone a JSON-serializable value without mutating the original. */
function clone<T>(value: T): T {
@@ -383,6 +385,119 @@ export function findUnstorableAttr(doc: any): string | null {
return null;
}
/**
* The Docmost schema's known node and mark NAME sets, derived ONCE from the very
* same `docmostExtensions` the Yjs encode path builds its schema from
* (`getSchema(docmostExtensions)` mirrored in mcp's `docmostSchema`). Deriving
* both from the same extension list guarantees `findInvalidNode`'s "known type"
* set matches exactly what `PMNode.fromJSON`/`toYdoc` will actually accept, so
* the walker never flags a node the encoder would have stored (or vice versa).
* Lazy + cached: the schema is only built on first use.
*/
let schemaNames: { nodes: Set<string>; marks: Set<string> } | null = null;
function getSchemaNames(): { nodes: Set<string>; marks: Set<string> } {
if (schemaNames == null) {
const schema = getSchema(docmostExtensions);
schemaNames = {
nodes: new Set(Object.keys(schema.nodes)),
marks: new Set(Object.keys(schema.marks)),
};
}
return schemaNames;
}
/**
* Depth-first walk of the JSON `content` tree looking for the FIRST node whose
* SHAPE the Yjs encode path will reject with an opaque
* `Unknown node type: undefined` (issue #409). Returns `{ path, summary }` for
* the offending node, or `null` when every node (and every mark) is a known
* Docmost schema type.
*
* Two failure modes are detected, in order, per node:
* 1. `type` is missing or not a string the dominant `undefined` case, e.g.
* a text leaf written as `{"text":"foo"}` with no `"type":"text"`.
* 2. `type` is a string but NOT a known Docmost node name (a typo / unknown
* block), OR one of the node's marks carries an unknown mark name.
*
* The returned `summary` is a model-actionable, path-anchored message such as:
* `node.content[2].content[0]: missing "type" (keys: text, marks) — did you
* mean {"type": "text", ...}?`
* or for an unknown type:
* `node.content[1]: unknown node type "paragraf" — not in the Docmost schema`
*
* `path` is the same dotted JSON path used in the summary (e.g.
* `node.content[2].content[0]`) so callers can surface it separately. Null-safe:
* a non-object doc returns `null`.
*
* NOTE: This is a SHAPE check, not a full ProseMirror content-model validation
* (it does not verify that a paragraph may legally contain a table, etc.). Its
* job is to turn the specific "unknown/absent node type" Yjs crash into a clear,
* pre-write diagnostic; the schema's own `.check()` still catches deeper
* content-model violations at encode time.
*/
export function findInvalidNode(
doc: any,
): { path: string; summary: string } | null {
if (!isObject(doc)) return null;
const { nodes, marks } = getSchemaNames();
// Build the "did you mean" hint for a typeless node from its own keys, so the
// model sees WHICH object is malformed and the canonical text-leaf fix.
const keyHint = (node: Record<string, any>): string => {
const keys = Object.keys(node);
const looksLikeText =
typeof node.text === "string" && node.type === undefined;
const suffix = looksLikeText
? ` — did you mean {"type": "text", ...}?`
: ` — every node needs a string "type" from the Docmost schema`;
return `missing "type" (keys: ${keys.join(", ") || "none"})${suffix}`;
};
const walk = (
node: any,
path: string,
): { path: string; summary: string } | null => {
if (!isObject(node)) return null;
// (1) missing / non-string type.
if (typeof node.type !== "string") {
return { path, summary: `${path}: ${keyHint(node)}` };
}
// (2) string type that is not a known Docmost node.
if (!nodes.has(node.type)) {
return {
path,
summary: `${path}: unknown node type "${node.type}" — not in the Docmost schema`,
};
}
// (2b) unknown mark on an otherwise-valid node.
if (Array.isArray(node.marks)) {
for (let i = 0; i < node.marks.length; i++) {
const mark = node.marks[i];
if (isObject(mark) && typeof mark.type === "string" && !marks.has(mark.type)) {
return {
path: `${path}.marks[${i}]`,
summary: `${path}.marks[${i}]: unknown mark type "${mark.type}" — not in the Docmost schema`,
};
}
}
}
if (Array.isArray(node.content)) {
for (let i = 0; i < node.content.length; i++) {
const hit = walk(node.content[i], `${path}.content[${i}]`);
if (hit != null) return hit;
}
}
return null;
};
// The root doc node is addressed as "node" (matching the mcp arg name); its
// children are node.content[i]. The root itself is checked too so a typeless
// root is reported rather than silently skipped.
return walk(doc, "node");
}
/**
* Table structural node types and the container each must live directly inside.
* Used by `insertNodeRelative` to splice rows/cells into the correct ancestor
@@ -0,0 +1,102 @@
import { describe, expect, it } from 'vitest';
import { findInvalidNode } from '../src/lib/node-ops.js';
// findInvalidNode (#409): a depth-first SHAPE gate that turns the encoder's
// opaque `Unknown node type: undefined` into a path-anchored, pre-write
// diagnostic. It flags the FIRST node whose `type` is absent/non-string or not
// a known Docmost schema node, or that carries an unknown mark; returns null for
// a well-formed doc.
const doc = (...content: any[]) => ({ type: 'doc', content });
const para = (...content: any[]) => ({
type: 'paragraph',
attrs: { id: 'p1' },
content,
});
const text = (value: string, marks?: any[]) => {
const node: any = { type: 'text', text: value };
if (marks) node.marks = marks;
return node;
};
describe('findInvalidNode', () => {
it('returns null for a fully valid document', () => {
const good = doc(
para(text('hello ', [{ type: 'bold' }]), text('world')),
{
type: 'heading',
attrs: { id: 'h1', level: 2 },
content: [text('Title')],
},
);
expect(findInvalidNode(good)).toBeNull();
});
it('flags a NESTED typeless text leaf with a path-precise summary', () => {
// A text leaf written as {"text":"foo"} with no "type":"text" — the dominant
// `Unknown node type: undefined` cause.
const bad = doc(
para(text('ok')),
para({ text: 'foo', marks: [] } as any),
);
const hit = findInvalidNode(bad);
expect(hit).not.toBeNull();
// Second paragraph (index 1), first child (index 0).
expect(hit!.path).toBe('node.content[1].content[0]');
expect(hit!.summary).toContain('node.content[1].content[0]');
expect(hit!.summary).toContain('missing "type"');
expect(hit!.summary).toContain('keys: text, marks');
expect(hit!.summary).toContain('did you mean {"type": "text", ...}');
});
it('flags a non-string type (e.g. numeric)', () => {
const bad = doc(para({ type: 123, content: [] } as any));
const hit = findInvalidNode(bad);
expect(hit).not.toBeNull();
expect(hit!.path).toBe('node.content[0].content[0]');
expect(hit!.summary).toContain('missing "type"');
});
it('flags an UNKNOWN node type name that is not in the Docmost schema', () => {
const bad = doc({
type: 'paragraf', // typo — not a real node
attrs: { id: 'x' },
content: [text('hi')],
});
const hit = findInvalidNode(bad);
expect(hit).not.toBeNull();
expect(hit!.path).toBe('node.content[0]');
expect(hit!.summary).toContain('unknown node type "paragraf"');
expect(hit!.summary).toContain('not in the Docmost schema');
});
it('flags an UNKNOWN mark type on an otherwise-valid node', () => {
const bad = doc(para(text('hi', [{ type: 'blink' }])));
const hit = findInvalidNode(bad);
expect(hit).not.toBeNull();
expect(hit!.path).toBe('node.content[0].content[0].marks[0]');
expect(hit!.summary).toContain('unknown mark type "blink"');
});
it('accepts every real Docmost node/mark type it is asked about', () => {
// Known node (callout) and known marks (italic, code) must NOT be flagged.
const good = doc({
type: 'callout',
attrs: { id: 'c1', type: 'info' },
content: [para(text('x', [{ type: 'italic' }, { type: 'code' }]))],
});
expect(findInvalidNode(good)).toBeNull();
});
it('reports the root itself when the root is typeless', () => {
const hit = findInvalidNode({ content: [] } as any);
expect(hit).not.toBeNull();
expect(hit!.path).toBe('node');
});
it('is null-safe for non-object input', () => {
expect(findInvalidNode(null)).toBeNull();
expect(findInvalidNode(undefined)).toBeNull();
expect(findInvalidNode('nope' as any)).toBeNull();
});
});