Compare commits

...

40 Commits

Author SHA1 Message Date
agent_coder 7f9f6c7585 fix(mcp): не нормализовать текст под маркой code в сносках (порча code-литералов)
Ревью #422: безусловная нормализация переписывала в ASCII и текст inline-code
внутри сносок (кавычки/тире/спецпробелы) — для code-литерала это не типографика,
а изменение смысла (эмпирически на raw-JSON путях: code-нода "a—b «x»" -> "a-b").
normalizeDefinitionText теперь пропускает текст-ноды с маркой code (verbatim), и
краевой trim определения тоже не трогает крайние code-ноды. footnoteMergeKey
читает сырой текст code-нод -> сноски, различающиеся только глифами в коде, НЕ
сливаются, а форки в прозе по-прежнему сливаются. Тесты: code verbatim +
непослияние по code-глифам.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 04:30:46 +03:00
agent_coder 91c4cc925a feat(mcp): детерминированная нормализация текста сносок + склейка форков по нормализованному тексту
Новый чистый модуль footnote-normalize-merge.ts (normalizeAndMergeFootnotes):
нормализует текст определений сносок (типографские кавычки->ASCII, тире->'-',
NBSP/спецпробелы->пробел, схлопывание пробелов, trim) и сливает определения с
совпавшим нормализованным текстом, перевешивая ссылки на канонический id;
дубли-сироты добивает canonicalizeFootnotes. Ключ слияния attrs-aware
(footnoteMergeKey/stableAttrs) — сноски с одинаковым текстом, но разными attrs
марок (напр. link.href) НЕ сливаются (защита от потери target). Пасс вызывается
строго ПЕРЕД canonicalizeFootnotes на 5 write-путях MCP (markdown-импорт,
updatePageJson, copyPageContent, docmost_transform, insertInlineFootnote).
Глиф-карты продублированы из comment-anchor.ts (там private+завязаны на golden).
Идемпотентен, чистый (deep-clone), scope строго внутри footnoteDefinition.

closes #419

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 04:30:46 +03:00
vvzvlad ee33a293b9 Merge pull request 'feat(mcp): drawio стадия 1 — CRUD-инструменты drawio_get/create/update (сырой XML)' (#434) from feat/423-drawio-crud into develop
Reviewed-on: #434
2026-07-10 04:27:32 +03:00
vvzvlad 86830b860d Merge pull request 'feat(ai-chat): авто-реконнект к detached-рану после живого обрыва SSE' (#432) from feat/430-live-reconnect into develop
Reviewed-on: #432
2026-07-10 04:27:16 +03:00
vvzvlad d0d2a7880f Merge pull request 'feat(client): сноски — рендер без сдвига (номер инлайн через ::before) и шрифтом sm' (#421) from feat/420-footnote-render into develop
Reviewed-on: #421
2026-07-10 04:26:49 +03:00
vvzvlad 9acbc07f7d Merge pull request 'feat(ai-chat): персист tool-error частей — упавшие тулы видны в истории и сохраняют текст ошибки' (#426) from feat/407-persist-tool-errors into develop
Reviewed-on: #426
2026-07-10 04:26:37 +03:00
vvzvlad a0eb3131a6 Merge pull request 'feat(mcp): createComment — подсказки самокоррекции якоря (closest-block, markdown-strip, multi-block)' (#427) from feat/408-createcomment-hints into develop
Reviewed-on: #427
2026-07-10 04:26:25 +03:00
vvzvlad 50bb086edf Merge pull request 'perf(mcp): кэш живого collab-соединения (CollabSession) — серия правок за один connect/sync' (#431) from perf/400-collab-session into develop
Reviewed-on: #431
2026-07-10 04:26:04 +03:00
vvzvlad f2ad0121a5 Merge pull request 'perf(collab): три фикса горячего пути — connect-vs-unload гонка, двойная перекодировка, isDeepStrictEqual' (#433) from perf/401-collab-hotpath into develop
Reviewed-on: #433
2026-07-10 04:25:47 +03:00
agent_coder 2194f423a1 test(mcp): покрыть error-ветки drawio-тулов + escaping round-trip; удалить мёртвый freshBlockId (ревью #434)
- drawio_get: битый ref -> 'no node found'; drawio-нода без src -> 'has no src';
  drawio_update: узел не drawio -> чистая ошибка + ноль upload/mutate;
  drawio_create: anchor не найден -> ошибка 'unreferenced orphan' + attachment
  залит и назван в сообщении, drawio-узел не записан.
- escaping round-trip: title с < > " & переживает encode/build/decode байт-в-байт,
  внешний content="..." остаётся well-formed.
- удалён неиспользуемый freshBlockId() (0 call-site после фикса #index-хендла).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 03:41:52 +03:00
agent_coder 5a6009c750 feat(mcp): drawio стадия 1 — CRUD-инструменты drawio_get/create/update (сырой mxGraph XML)
Узел drawio для агента был непрозрачен: round-trip хранит узел, но содержимое
диаграммы недоступно. Стадия 1 даёт минимальный CRUD без рендеринга на беке.

Новые модули:
- drawio-xml.ts: decode-chain (content= base64 -> plain/entity-encoded XML или
  compressed <diagram> через pako.inflateRaw raw-deflate + decodeURIComponent;
  инфляция потоковая с капом 16 MiB — защита от decompression-bomb); encode
  (plain uncompressed по контракту createDrawioSvg); линтер (все правила ->
  структурированный tool-error с cellId; edge без дочернего mxGeometry — ошибка
  №1); stable mxHash (sha256 по нормализованному XML) — ключ optimistic-lock.
- drawio-preview.ts: чистый TS schematic SVG (rect/ellipse/rhombus/edge/label,
  контейнеры -> абсолютные координаты, unknown-стенсил -> подписанный rect),
  без зависимостей и без бэкенд-рендера.

Инструменты (SHARED_TOOL_SPECS, deferred-тир):
- drawio_get(pageId, node, format?) -> XML/SVG + мета {attachmentId,title,w,h,
  cellCount,hash}; читает человеческий compressed-экспорт losslessly.
- drawio_create(pageId, where, xml, title?) -> lint -> preview -> .drawio.svg ->
  upload (тот же attachment-конвейер + validateDocUrls) -> insert drawio-узел;
  возвращает адресуемый '#<index>'-хендл (у схемы drawio нет атрибута id).
- drawio_update(pageId, node, xml, baseHash) -> baseHash-конфликт до записи;
  перепривязывает РОВНО адресованный узел (не все с общим attachmentId).

pako@2.0.3 — единственная новая зависимость (lockfile синхронизирован).

closes #423

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 03:25:29 +03:00
agent_coder 9685074237 perf(collab): три фикса горячего пути сервера — connect-vs-unload гонка, двойная перекодировка, isDeepStrictEqual (замер)
Побочные находки инцидента #400 (правка большой таблицы через MCP подвешивает всё).

1. Гонка connect-vs-unload в @hocuspocus/server 3.4.4 (вероятный источник 25s
   connect-таймаутов): createDocument проверяет loadingDocuments/documents, но НЕ
   ждёт unloadingDocuments -> новое соединение может захендшейкаться на умирающий
   Document -> redis-sync идёт по пути 'doc не загружен', провайдер висит до
   таймаута. Апстрим (main) не починен. pnpm-патч (инфра как у yjs-патча): в начале
   createDocument await in-flight unload (обёрнут в try/catch — отклонённый unload
   не отравляет открытие, поведение как до патча), в ОБОИХ рантаймах (cjs+esm).
   Тест hocuspocus-unload-race: реальный createDocument с засеянным in-flight
   unload -> не грузит пока unload не осел; при откате патча тест краснеет.

2. Двойная перекодировка в onLoadDocument (persistence.extension.ts): хук строил
   НОВЫЙ Y.Doc и возвращал его -> hocuspocus делал applyUpdate(encodeStateAsUpdate)
   ВТОРОЙ раз (315КБ на каждую холодную загрузку); в JSON-ветке результат encode
   выбрасывался (мёртвый вызов). Теперь стейт применяется прямо в data.document,
   возврат undefined (hocuspocus мержит только при возврате Doc); мёртвый encode
   убран. Содержимое документа не меняется — только меньше encode/alloc.

3. isDeepStrictEqual по 84КБ JSON на каждом store: замерил — 1.32мс на 90КБ
   (immaterial, <50мс порога; доминируют fromYdoc+encodeStateAsUpdate). Изменений
   кода НЕТ по правилу задачи (dirty-флаг только при material).

closes #401

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 02:53:23 +03:00
agent_coder 22f687c39e feat(ai-chat): авто-реконнект к detached-рану после живого обрыва SSE
Автономный ран продолжается на сервере при обрыве SSE (Safari роняет длинный
стрим), но клиент показывал баннер 'Lost connection' и мёртвую вкладку до ручной
перезагрузки: resumeStream() звался только на mount. Добавлен недостающий триггер.

- chat-thread.tsx: в onFinish на живом isDisconnect (гард !wasResumed &&
  autonomousRunsEnabled && mounted && assistant) -> beginReconnect с экспон.
  backoff (1/2/4/8/16с, лимит 5). Стоп: status->streaming / 2xx re-attach /
  терминальный хвост reconcile / stop / unmount. Исчерпание -> Retry.
- Дедуп (главный риск): зеркалит mount strip/anchor — пиннит текущий streaming-ряд
  как anchor (id ассистент-строки), стрипает его из стора ДО replay, сервер
  ?expect=live&anchor=<id> пересобирает без дублей; на отказе/204 строка
  восстанавливается через onNoActiveStream (контент не теряется).
- 204/overflow -> degraded poll через существующий onNoActiveStream.
- RUN_STREAM_MAX_BUFFER_BYTES 4->32МБ (марафонские раны 11-25мин переполняли 4МБ);
  204->poll остаётся backstop. Degraded-poll: фиксированный 10-мин-от-старта кап
  заменён на inactivity-кап (продлевается пока приходят новые ряды).
- UI: баннер 'reconnecting… (N/5)' + ручной Retry на исчерпании.

closes #430

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 02:51:58 +03:00
agent_coder 0d4f719f47 fix(mcp): fail-fast гард на одновременный mutate одной CollabSession (ревью #431)
mutate трекает in-flight одним полем inflightReject; наложенный второй вызов
перезаписал бы рехджектор первого -> при disconnect отклонился бы только второй,
первый бы висел до PERSIST_TIMEOUT_MS (20с). В проде безопасно (оба call-site
сериализуют через per-page withPageLock), но это футган на разделяемом примитиве.
Гард в начале mutate (после ready-проверки, до касания inflightReject): наличие
in-flight -> reject нового вызова без порчи состояния первого. Docstring CONCURRENCY.
Последовательные mutate не задеты (localFinish синхронно чистит inflightReject до
резолва). +2 теста: конкурентный второй реджектится, первый цел; последовательные
оба успешны.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 02:44:40 +03:00
agent_coder 572f0a2ab9 perf(mcp): кэш живого collab-соединения (CollabSession) — серия правок за один connect/sync
Инцидент: агент заполнял большую таблицу десятками table_update_cell, каждый —
полный цикл connect/auth/load/initial-sync/store/unload; часть падала с 25s connect-
timeout, event loop lag до 1.7с. Вариант B: кэшировать живой HocuspocusProvider
per page на серию правок — пока провайдер жив (connections>0), сервер не входит в
store->unload->reload, дебаунс реально коалесцирует записи (N ячеек -> 1-2 store).

Новый модуль collab-session.ts: класс CollabSession (connecting->ready->dead) +
реестр (ключ wsUrl+pageId+token) с idle-TTL/max-age/LRU-evict. mutatePageContent
(collaboration.ts) и mutateLiveContentUnlocked (client.ts, replaceImage) переведены
на acquireCollabSession; one-shot Promise-машина (~360 строк дублирования) удалена.

5 инвариантов (подтверждены внутренним ревью): (1) read->write атомарна — между
fromYdoc и applyDocToFragment нет await; (2) per-edit ack сохранён дословно (гард
connectionLost от false-success при реконнекте); (3) disconnect=смерть сессии (без
авто-реконнекта, in-flight реджектится теми же текстами ошибок); (4) изоляция
identity (токен в ключе); (5) валидация при reuse. replaceImage работает под
внешним page-локом без дедлока (acquire лок не берёт). Тексты ошибок == develop.

Env: MCP_COLLAB_SESSION_IDLE_MS (0=выкл кэш, точное легаси), _MAX_AGE_MS, _MAX_ENTRIES.
Teardown: destroyAllSessions обвязан в stdio (exit/SIGINT/SIGTERM) + реэкспорт из
index для встраивающего хоста.

closes #400

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 02:28:30 +03:00
agent_coder 72c2d1687e docs(mcp): починить осиротевший docstring + уточнить коммент про exact-wins (ревью #427)
- text-normalize.ts: closestBlockHint был вставлен между docstring'ом
  stripInlineMarkdown и его определением -> docstring осиротел. closestBlockHint
  перенесён ПОСЛЕ stripInlineMarkdown, каждый docstring снова примыкает к своей
  функции. Поведение не менялось (только порядок объявлений).
- comment-anchor.ts: header-коммент завышал маршрутизацию — countAnchorMatches НЕ
  зовёт resolveAnchorSelection, у него своя параллельная реализация exact-wins.
  Коммент уточнён: can/get/apply идут через resolveAnchorSelection, count держит
  свой счётчик-примитив, синхронный с ним; обе реализации exact-wins должны
  держаться в синхроне при правках.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 00:06:28 +03:00
agent_coder 96faa28220 docs: отразить ключ error в заглавной секции reading-ai-logs (устранить противоречие)
Ревью #426: секция «How tool calls are stored — READ THIS» всё ещё утверждала,
что единственные ключи элемента — toolName/input/output и «нет error», хотя этот
же PR добавляет error и подробно описывает его ниже. Заглавный абзац приведён в
соответствие: error — возможный ключ для брошенных ошибок на строках после #407;
подсчёт инвокаций и пайринг учитывают error как парный результат.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:50:05 +03:00
agent_coder c9293e316b feat(mcp): createComment — подсказки самокоррекции якоря (closest-block, markdown-strip, multi-block)
createComment — топ-хотспот ошибок агента (промахи по якорю, слепые ретраи).
Портированы аффордансы самокоррекции из editPageText:
- Closest-block hint: общий хелпер closestBlockHint вынесен в text-normalize.ts
  (json-edit.ts теперь тоже его зовёт), подключён во все 3 throw-а createComment.
- Markdown-strip fallback в comment-anchor.ts согласованно по всем 4 функциям
  (can/count/apply/get) через единый resolveAnchorSelection: exact-verbatim wins
  глобально, stripped — только если raw не якорится нигде; soft warning как в
  editPageText. Инвариант уникальности suggestion (0/1/>=2) сохранён: raw-unique
  никогда не запускает fallback -> не может стать ambiguous. Хранимый selection
  остаётся СЫРОЙ подстрокой документа (strip только для поиска).
- Multi-block detection: явное сообщение 'selection spans multiple blocks' когда
  per-block поиск провалился, но выделение есть в объединённом тексте блоков.
- tool-spec createComment: копировать selection дословно из getPage/searchInPage.
Известное мелкое ограничение (нит внутреннего ревью): детектор multi-block
использует raw selection, поэтому markdown-стилизованное выделение через границу
блоков получит generic-подсказку вместо spans-multiple-blocks (редко, guidance
всё равно корректный).

closes #408

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:45:11 +03:00
agent_coder 654ba9f249 feat(ai-chat): персист tool-error частей — упавшие тулы видны в истории и сохраняют текст ошибки при реплее
В ai@6 упавший тул — это tool-error часть в step.content ({type,toolCallId,
toolName,input,error}), а не элемент toolResults. Раньше serializeSteps писал
только toolCalls+toolResults (ошибка терялась, orphan tool-call без результата),
а assistantParts эмитил заглушку 'Tool call did not complete.' (реальный текст
ошибки терялся для мультиходового реплея — модель не знала, почему упало, и
повторяла ошибку).

- StepLike расширен полем content; новый хелпер normalizeToolError (Error/string/
  object -> строка, обрезка через существующий compactValue/лимиты).
- serializeSteps: на каждый tool-error пушит парный {toolName, error} тем же
  паттерном, что успешный {toolName, output} -> колонка tool_calls фиксирует сбой.
- assistantParts: при наличии tool-error эмитит output-error с РЕАЛЬНЫМ текстом;
  заглушка остаётся только для по-настоящему непарных вызовов (прерванных).
- docs/reading-ai-logs.md обновлён под новую форму + cutover-оговорка.
Обратно совместимо: старые строки читаются как раньше, error-элемент аддитивен.

closes #407

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:35:32 +03:00
agent_coder 9120ad3b2d feat(client): сноски — рендер без сдвига (номер инлайн через ::before) и шрифтом sm
Убран отдельный столбец-маркер .definitionMarker (order:-1, min-width:1.5em),
дававший висячий отступ. Номер сноски теперь рисуется инлайн в начале первого
параграфа через .definitionContent > :first-child::before из CSS-переменной
--footnote-number (в модель документа не попадает, экспорт не затрагивает).
Кегль сносок уменьшен до var(--mantine-font-size-sm). Инвариант #146 (contentDOM
первый в DOM) и логика мульти-бэклинков #168 сохранены.

closes #420

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:12:35 +03:00
agent_vscode d90c3b8b9e fix(ai-chat): stop trimming tool outputs in replayed history
Read-tool outputs were compacted at a 4000-byte gate before being stored
in metadata.parts, which is re-sent to the model every later turn. Whole-page
reads (tens of KB) got shrunk to a 500-char preview plus a "[truncated N chars]"
marker; on the next turn the model read that marker as a source truncation and
re-read the page, wasting tokens and producing wrong behavior.

- Raise MAX_TOOL_OUTPUT_BYTES 4000 -> 200_000 so normal reads are stored and
  replayed verbatim; only a single >200 KB output is compacted as a backstop.
- Reword the inline string marker so it reads as a replay-history elision, not
  a source truncation, and tells the model it can re-call the tool.
- Update doc comments and enlarge the compactToolOutput unit-test inputs above
  the new 200 KB gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 17:49:47 +03:00
agent_vscode b24347fd96 chore(vscode): update git sync task to fetch and merge from gitea 2026-07-07 21:42:46 +03:00
vvzvlad 6ee581a0a9 Merge pull request 'feat(ai-chat): insertFootnote/insertImage/replaceImage для in-app агента (#410)' (#418) from feat/410-agent-footnote-image into develop
Reviewed-on: #418
2026-07-07 21:38:59 +03:00
agent_coder 984b95df9f test(mcp): #410 review — align HOST_CONTRACT_METHODS drift-guard (#410)
Promoting insertFootnote/insertImage/replaceImage into the in-app
DocmostClientLike interface requires their mirror in the drift-guard's
HOST_CONTRACT_METHODS list (the bidirectional deepEqual — 40 vs 43 — was
red, exactly what the guard exists to catch). Added the three names and
corrected the header comment: they were MCP-only but are now in-app-consumed
and tracked; only deleteComment/updateComment remain untracked MCP-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:35:08 +03:00
agent_coder 327737b701 feat(ai-chat): give the in-app agent insertFootnote/insertImage/replaceImage (#410)
The Researcher role wrote 40 literal `^[...]` and zero real footnotes: its
incremental write path (insertNode/editPageText) doesn't parse markdown, and
the footnote-capable tool was MCP-only. Promote three tools from inline
MCP-only to the shared registry so the in-app agent gets them too.

- tool-specs.ts: insertFootnote/insertImage/replaceImage added to
  SHARED_TOOL_SPECS (mcpName/schema/description moved VERBATIM from the inline
  registrations — MCP names + behaviour unchanged for external clients).
- index.ts: the 3 inline registerTool calls become registerShared; drop the
  "MCP-only by design" comments.
- ai-chat-tools.service.ts: register the 3 in-app via sharedTool ->
  client.insertFootnote/insertImage/replaceImage (imageUrl->url,
  attachmentId->oldAttachmentId mapping).
- tool-tiers.ts: insertFootnote -> core (else the original asymmetry recurs —
  footnote tool hidden while editPageText is core); images -> deferred.
- research/{en,ru}.yaml FOOTNOTES: `^[...]` parses ONLY on a whole-markdown
  write (create/update/import); for a pinpoint citation to existing text use
  insertFootnote; via editPageText/insertNode it stays literal.
- json-edit.ts guardrail: an edit_page_text `replace` containing a `^[...]`
  token is refused into failed[] with an insert_footnote hint, mirroring the
  existing formatting-marker refusal. (Slightly broader net than that mirror —
  a literal `^[a-z]` regex class in a replace is also refused; accepted
  defense-in-depth, has a no-false-positive test.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:18:01 +03:00
agent_vscode abd61041fe Merge branch 'develop' of https://gitea.vvzvlad.xyz/vvzvlad/gitmost into develop 2026-07-07 21:00:18 +03:00
vvzvlad f55191e2a0 Merge pull request 'test(ai-chat): упрочнение фикса MCP-зависания — покрытие defense-in-depth + параллельная сборка (#397 follow-up)' (#405) from fix/397-mcp-hang-hardening into develop
Reviewed-on: #405
2026-07-07 20:57:40 +03:00
agent_vscode 7100d28629 docs: add reading-ai-logs documentation 2026-07-07 20:43:41 +03:00
agent_vscode 7538f98a3d feat(agent-roles): increase instruction max length to 100000
Raise validation limit from 20000 to 100000 characters to allow more detailed instructions.
2026-07-07 19:30:11 +03:00
agent_vscode a984366309 feat(agent-roles): add PROSE, NOT NOTES guidelines to researcher role
Add detailed "PROSE, NOT NOTES" instructions to the English and Russian researcher role bundles, clarifying report writing standards. Update the researcher role version to 8 in the index and content-hashes files.
2026-07-07 19:16:30 +03:00
agent_coder 41480bc44f test(ai-chat): harden the MCP-hang fix — cover defense-in-depth paths + parallelize build (#397)
Follow-up to the merged MCP-hang fix (16b476a2); post-hoc review DO. The
fix itself is unchanged and prod-working — this adds the coverage + one
coherence fix the review asked for.

Tests (ai-chat.service.setup-abort.spec.ts):
- onLateResolve late-close: a toolset that resolves AFTER the setup race
  was lost has its leased clients released (close spy asserted).
- pure 60s deadline (signal NOT aborted): the turn proceeds Docmost-only
  (reaches streamText, run NOT finalized 'aborted') — the defense-in-depth
  backstop, previously untested.
- legacy no-runId: a setup abort does NOT re-throw (the `runId &&` guard);
  together with the deadline test this locks both halves of the catch guard.

Coherence (external-mcp/mcp-clients.service.ts buildEntry):
- The per-server connects now run via Promise.all instead of a sequential
  for-await, so total build time is bounded by the slowest single server
  (~2×CONNECT_TIMEOUT_MS) instead of the sum. At >6 all-timing-out servers
  the sequential build exceeded the outer 60s deadline, inverting the
  "per-server bound primary, outer deadline is backstop" invariant. Merge
  is done in a sequential post-Promise.all loop in server order, so tool-key
  precedence/disambiguation, outcomes and client ordering are byte-identical
  to the sequential build; each server keeps its own timeout + failure-close.

Comments: the onLateResolve note now says it releases the lease (refcount),
not force-closes transports (the cache owns them); the invariant comment
reflects the parallel build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 18:05:10 +03:00
agent_vscode f68c7ba7ef feat(agent-roles): rewrite researcher prompt (pages-read budget, working-memory doc, review pass)
Replace the researcher role instructions in both bundles with the new prompt:
- Budget measured in PAGES READ, not searches; snippets never enter the report.
- The document is working memory: live plan, "Log"/"Open Questions" sections,
  hard flush cadence (~8-10 pages), context discipline with re-reads.
- Mandatory CRITICAL REVIEW PASS + BUDGET REMAINDER PROTOCOL (adversarial
  verification, primary sources, lateral expansion).
- Source hierarchy, dates/staleness, dead-end handling, inline ^[...] footnotes,
  report language/terminology rules, finalization checklist.

ru.yaml carries the text verbatim (Russian report); en.yaml is the English-
adapted mirror (report language + working-section names/examples translated).
Bump researcher role version and refresh the content-hash lock.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 05:39:28 +03:00
agent_vscode 23cbc0cc91 feat(agent-roles): researcher cites sources inline, reads pages, defaults to 50 searches
Rework the researcher role prompt (ru + en bundles):
- Add a "CITING SOURCES INLINE (FOOTNOTES)" section: every non-trivial claim
  must carry an inline `^[...]` footnote (the only form this system parses);
  explicitly forbid the unsupported `[^1]` reference style.
- Add a HARD CADENCE rule: flush findings to the document at least every 10
  searches, reinforced in the WORK LOOP.
- Raise the default search volume: STEP 0 estimate and the VOLUME floor now
  default to ~50 searches (15 -> 50), with a carve-out for a single trivial fact.
- Replace the weak "FULL PAGES, NOT SNIPPETS" bullet with a strong rule to open
  and read (extract) pages, not just run searches (~2-3 pages per search).

Bump researcher role version and refresh the content-hash lock.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 05:21:12 +03:00
agent_vscode 4e9f47b4a5 fix(ai-chat): strip NUL chars before persisting assistant rows
A single NUL (U+0000) in model/tool output (e.g. a truncated multibyte
read of a web page) is rejected by Postgres in BOTH the content (text) and
toolCalls/metadata (jsonb) columns, so it failed EVERY write of the
streaming assistant row ("invalid input syntax for type json") and silently
dropped the turn's content from the DB while the live stream still showed it.

- add stripNulChars: deep-strips NUL from all strings, returns the same
  reference when there is nothing to strip (no needless clone)
- apply it at the flushAssistant choke point (covers content + toolCalls +
  metadata for the seed, per-step and terminal writes)
- tests: deep-strip, same-reference, end-to-end via flushAssistant

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 04:40:03 +03:00
agent_vscode 888c87f984 feat(config): lower MCP timeouts and raise JSON body limit
Reduce the default external-MCP silence timeout from 5 min to 1 min and the
overall call timeout from 15 min to 2 min. Update related tests and comments.
Increase Fastify's JSON body limit to 25 MiB (configurable via
HTTP_JSON_BODY_LIMIT) to accommodate large AI‑chat payloads.

BREAKING CHANGE: shorter MCP timeouts may abort long‑running tool calls that
previously succeeded with the older defaults.
2026-07-07 04:19:26 +03:00
vvzvlad f0afb2d729 Merge pull request 'feat(metrics): наблюдаемость collab-цикла и MCP (#402, follow-up #355)' (#403) from feat/402-collab-mcp-metrics into develop
Reviewed-on: #403
2026-07-07 02:41:10 +03:00
agent_coder 8f5f5877b3 test(metrics): #403 review — lock the registerTool monkeypatch + reword overhead note (#402)
DO-1: add an integration test (test/mock/tool-timing-server.test.mjs) that
constructs a real createDocmostMcpServer with a spy onMetric, links a Client
over InMemoryTransport, invokes get_workspace (no input schema, so the wrapped
handler always runs) and asserts onMetric fired with
("mcp_tool_duration_seconds", <number>, { tool: "get_workspace" }). This locks
that the monkeypatch wraps every tool AND labels with the registration name —
which the isolated timeToolHandler unit test does not. Mutation-verified
(mislabel -> test fails). The tool's expected backend failure (ECONNREFUSED)
is tolerated: the wrapper times in a finally on throw too, so the metric fires.

DO-2: reword the mcp.service.ts "zero overhead when disabled" comment to
"negligible overhead" — the registerTool wrapper still runs performance.now()
+ an async try/finally per tool call when onMetric is undefined (the
`onMetric?.()` short-circuits the label build; cost is immaterial at
tool-call rate), so "zero" was literally inaccurate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 02:38:11 +03:00
agent_coder 7a9d719877 feat(metrics): #402 pass 2 — MCP tool + connect-timeout via dependency-neutral callback
packages/mcp stays free of prom-client/server: it only calls an optional
DocmostMcpConfig.onMetric(name, value, labels?) sink the host provides.

- client.ts: onMetric on the config; fires collab_connect_timeouts_total
  once, only inside the 25s connect-timeout callback (the connect-vs-unload
  signal). cleanup() clears the timer on every other finish path, so no
  double-count.
- index.ts: createDocmostMcpServer monkeypatches server.registerTool (before
  registerShared + inline tools are registered) to wrap every handler with
  timeToolHandler — times in a finally on success AND throw, re-throws
  unchanged, emits mcp_tool_duration_seconds{tool=<registered name>} (bounded
  cardinality). Single choke point catches all tools.
- mcp.service.ts: the per-request config resolver injects onMetric ONLY when
  isMetricsEnabled(), routing mcp_tool_duration_seconds -> observeMcpTool and
  collab_connect_timeouts_total -> incConnectTimeout. Disabled / standalone
  (stdio, no onMetric) -> undefined -> zero-overhead no-op.

New node:test unit (tool-timing.test.mjs) covers the wrapper's value/throw
preservation and the standalone no-op. packages/mcp/build/ is gitignored,
not committed (CI rebuilds it).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 02:20:46 +03:00
agent_coder 96db9b6c7f feat(metrics): #402 pass 1 — collab-cycle observability (load/lifecycle/connect/auth)
Extends the #355 perf-metrics registry (all behind the METRICS_PORT hard
gate — nullable instruments, no-op helpers when unset). New families:

- collab_doc_load_duration_seconds{size_bucket} — onLoadDocument timed on
  the real DB-load paths only (the already-loaded early-return is skipped).
- size_bucket added to collab_store_duration_seconds; storeDocument returns
  the ydoc byte length (reusing its single Y.encodeStateAsUpdate, no second
  encode) so onStoreDocument observes size without extra cost.
- collab_docs_open (gauge, read-on-scrape via collect() from
  hocuspocus.getDocumentsCount() — never inc/dec'd, so it can't drift) +
  collab_doc_loads_total / collab_doc_unloads_total (afterLoad/afterUnload).
- collab_connect_duration_seconds — onConnect->connected, correlated by a
  WeakMap keyed on the shared request (leak-free; the current client uses
  one socket per document).
- collab_auth_duration_seconds — wraps onAuthenticate (success and failure).
- sizeBucket() shared helper (lt64k|lt256k|lt1m|ge1m, 4 bounded values).

The mcp_tool_duration_seconds histogram + collab_connect_timeouts_total
counter helpers are registered here but wired in pass 2 (MCP callback).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 02:10:54 +03:00
agent_vscode 16b476a205 fix(ai-chat): bound MCP connect + guard turn setup so a hung handshake can't wedge every run
A transient network blip during an external-MCP handshake left createMCPClient
pending forever (@ai-sdk/mcp does not settle on abort). getOrBuildEntry caches the
per-workspace build PROMISE, so the never-settling connect poisoned the cache and
every later turn hung at step_count=0 before streamText — the run never finalized,
the row stayed 'running', and the chat was permanently blocked with
A_RUN_ALREADY_ACTIVE (an explicit Stop could not interrupt the un-signalled setup).

- mcp-clients: wrap connect() in a settling timeout (connectWithTimeout) that
  closes a late-arriving client, so a hung handshake rejects instead of poisoning
  the build cache; the bad server is skipped and the build completes.
- mcp-clients: close a connected-but-unregistered client when tools() fails,
  fixing a pre-existing transport leak in buildEntry.
- ai-chat.service: bound the toolsFor build with the run's abort signal AND a
  deadline (raceAgainstAbortAndTimeout); re-throw an explicit Stop only when a run
  exists (runId) so the run finalizes as 'aborted', and keep the legacy
  socket-bound path unchanged; settle the run 'aborted' vs 'error' accordingly.
- tests: cache-not-poisoned + orphan-client-close-once + Stop-during-setup
  finalizes the run once.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 23:27:51 +03:00
68 changed files with 8555 additions and 935 deletions
+13 -5
View File
@@ -191,16 +191,24 @@ MCP_DOCMOST_PASSWORD=
# Silence timeout (ms) for EXTERNAL-MCP transport ONLY (not the chat provider).
# Tighter than AI_STREAM_TIMEOUT_MS so a byte-silent/hung MCP server is broken in
# ~5 min instead of 15. Note it also cuts a legitimately long but byte-silent
# ~1 min instead of 15. Note it also cuts a legitimately long but byte-silent
# single tool call (a slow crawl that emits nothing until done) and an SSE
# transport idling >5 min BETWEEN tool calls. Default 300000 (5 min).
# AI_MCP_STREAM_TIMEOUT_MS=300000
# transport idling >1 min BETWEEN tool calls. Default 60000 (1 min).
# AI_MCP_STREAM_TIMEOUT_MS=60000
# Total wall-clock cap (ms) for ONE external MCP tool call (app-level, not
# transport). Aborts a tool that keeps the socket warm (SSE heartbeats / trickle)
# but never returns a result — which the silence timeout above never breaks.
# Default 900000 (15 min).
# AI_MCP_CALL_TIMEOUT_MS=900000
# Default 120000 (2 min).
# AI_MCP_CALL_TIMEOUT_MS=120000
# Max JSON/urlencoded request body size (bytes). Fastify's 1 MiB default is too
# small for a long AI-chat research turn: the client resends the FULL message
# history (every tool call + search result) on each turn, so a deep conversation's
# POST to /api/ai-chat/stream can be several MB and would otherwise be rejected
# with FST_ERR_CTP_BODY_TOO_LARGE (413). Does NOT affect multipart file uploads
# (see FILE_UPLOAD_SIZE_LIMIT). Default 26214400 (25 MiB).
# HTTP_JSON_BODY_LIMIT=26214400
# Deferred tool loading for the in-app AI chat (#332). Default ON: the agent sees
# a compact <tool_catalog> and only CORE tools + a loadTools meta-tool are active
+2 -2
View File
@@ -3,9 +3,9 @@
"version": "2.0.0",
"tasks": [
{
"label": "git push (github + gitea)",
"label": "git sync (pull gitea -> push github + gitea)",
"type": "shell",
"command": "git push github develop && git push gitea develop",
"command": "git fetch gitea && git merge --no-edit gitea/develop && git push github develop && git push gitea develop",
"options": { "cwd": "${workspaceFolder}" },
"presentation": { "reveal": "never", "focus": false, "panel": "shared", "showReuseMessage": false, "close": true },
"problemMatcher": []
+303 -105
View File
@@ -16,139 +16,337 @@ roles:
whatever language is most effective, but deliver the report in English.
═══════════════════════════════════════════════
STEP 0. PLAN (always do this first)
THE BUDGET: PAGES READ, NOT SEARCHES
═══════════════════════════════════════════════
Before searching for anything, draft and show a research plan:
- Break down the query: what exactly is needed, what sub-questions are
inside it, which terms are ambiguous or have synonyms/jargon.
- Formulate 5–10 search directions, including adjacent perspectives that
may prove useful even if the user did not ask about them directly.
- Fix the "research budget" — how many searches to run. If the USER named a
budget (e.g. "budget 100"), that number is BINDING and MUST be spent in
full: it defines the volume of the research, so keep searching until it is
used up. If the user gave no number, estimate one yourself from the task's
complexity (a simple fact: under 5; a medium task: 5–15; a hard task:
more).
- Decide which languages it makes sense to search in (see below).
The unit of research work is a PAGE READ IN FULL — opening a source with the
page-reading/extraction tool and actually reading it. Search queries are free
and unlimited: they are navigation, not research. A search result snippet is a
POINTER, never a source. Nothing learned only from a snippet may enter the
report.
- If the user named a budget (e.g. "budget 100"), that is 100 pages read, and
it is BINDING — a floor you MUST reach. Spend it in full even past the point
where the topic feels covered (see BUDGET REMAINDER PROTOCOL below).
- If no budget is given, default to about 50 pages read; fewer only for a
single trivial fact, well over 50 for a hard, broad task. Absent an explicit
budget, stop only at genuine saturation — when further reading stops
yielding new relevant information — not when it "seems like enough".
- A page counts toward the budget only if you read it and extracted something
(a finding, a dead-end note, a contradiction). Skimming a snippet does not
count. Re-opening the same page does not count twice.
- Rule of thumb: for every search that surfaces relevant hits, open and read
at least 2–3 of the most promising results BEFORE running the next search.
Chaining searches with no page reads in between is a critical failure —
snippets carry ~5 % of the available content and reading pages is the whole
job. If you catch yourself doing it, stop and go read what you already
found.
BUDGET REMAINDER PROTOCOL. When the topic already feels covered but budget
remains, do NOT pad with junk or near-duplicate reads. Spend the remainder in
this priority order:
1. ADVERSARIAL VERIFICATION — for each key claim in the document, run
searches deliberately trying to REFUTE it or find a competing version;
read what you find. Results go into the "Contradictions" section (or
strengthen the claim's footnote).
2. PRIMARY SOURCES — for every important claim currently backed by a
retelling, aggregator, or news piece, hunt down and read the original:
the study, spec, dataset, filing, repository, interview.
3. LATERAL EXPANSION — adjacent disciplines, industries with the same
problem, historical analogues, criticism and opposing schools.
Every remainder read must still be a genuine attempt to learn or verify
something.
═══════════════════════════════════════════════
WHERE TO WRITE THE RESULT
THE DOCUMENT IS YOUR WORKING MEMORY
═══════════════════════════════════════════════
- Reuse the current/already-open document ONLY if either (a) the user
explicitly asked to work in it, or (b) it is empty or has very little on
it AND its title matches the topic of the research. In every other case —
a non-empty page, or one whose title is about something else — create a
NEW document for the report.
- Set up this document at the VERY START — right after the plan (STEP 0) and
BEFORE running any searches. Seed it immediately with the query, the plan,
and a skeleton of the sections you expect to fill.
- Fill the document DYNAMICALLY as you work: after every meaningful finding,
write it in straight away (fact → source → reliability assessment) and
grow or reshape the structure as your understanding evolves.
- Do NOT hoard everything in your head or in notes and dump the whole report
in one pass at the end. The document is a LIVING artifact: it must exist
from the first minute and be updated continuously throughout the run, so
that by the finalization stage it is already almost complete and only
needs cleanup, ordering, and self-verification.
Your context window is small and lossy; the document is not. Treat the
document — not your head — as the single source of truth and your external
memory. You are not "taking notes to compile later"; you are building the
report itself, live, from the first minute.
SETUP. Create/claim the document at the VERY START, before any searches.
Reuse the currently open document ONLY if (a) the user explicitly asked to
work in it, or (b) it is empty or near-empty AND its title matches the topic.
Otherwise create a new one.
Seed it immediately with:
- the user's query, restated;
- the RESEARCH PLAN (see below) — the plan lives in the document, not in
chat; do not wait for approval, write it and proceed;
- a skeleton of the report sections you expect to fill;
- a "Log" section (working log) and an "Open Questions" section.
RESEARCH PLAN (written into the document before searching):
- Break down the query: what exactly is needed, what sub-questions are
inside it, which terms are ambiguous or have synonyms/jargon.
- 5–10 search directions, including adjacent angles the user did not ask
about directly.
- The budget (user-given or default) and how you expect to allocate it
across directions — a rough split, revisable.
- Which languages to search in.
THE LOG. In the "Log" section keep a numbered list of pages read:
`N. [query →] source — what I took / empty / contradiction`. One line each.
This is your budget counter and your flush-cadence counter — count by the log,
not from memory. Dead ends and paywalls go in the log too (they count toward
the budget only if you actually read a cached/alternative copy; a hard dead
end is logged but not counted).
FLUSH CADENCE — HARD RULE. Never read more than ~8–10 pages without writing
everything gathered since the last flush into the report sections. Check the
log: if the last flush was 10 reads ago, the next action is writing, not
reading. Frequent small updates are the norm; a long streak of reads with
nothing written is a mistake to correct immediately.
A flush means writing REPORT PROSE, not dumping notes. Every flush produces
finished paragraphs in the report sections, written to the standard of
"PROSE, NOT NOTES" below. Telegraphic fragments are allowed ONLY in the
"Log" and "Open Questions" working sections — never in the report body.
Do not plan to "expand the notes into text later": later never comes, and a
report assembled from unexpanded notes is a failed report.
CONTEXT DISCIPLINE. After flushing a finding into the document, compress it in
your head to 2–3 sentences of conclusions and let the raw page text go. Do not
carry full page contents forward in context. When you need to re-orient — and
ALWAYS before deciding what to research next after a flush — RE-READ the
document (at minimum: the skeleton, "Open Questions", and the sections you
touched). The document you re-read, not your memory of it, defines the current
state of the research.
═══════════════════════════════════════════════
WORK LOOP (repeat until saturation)
WORK LOOP
═══════════════════════════════════════════════
Work iteratively through an observe → orient → decide → act loop:
1. Observe: what has been gathered, what is still missing, what tools exist.
2. Orient: which query or source would best close the gap; update your
understanding of the topic based on what you've found.
3. Decide: choose a specific next action.
4. Act: run the search or open the source.
After EVERY result, reason about it: what you learned, what new questions
arose, what to search next. Maintain an internal list of open questions and
gaps, and close them.
Iterate observe → orient → decide → act:
1. Observe: re-read the relevant parts of the DOCUMENT — what is filled,
what is thin, what "Open Questions" lists.
2. Orient: which query or source best closes the biggest gap; update the
plan section if your understanding of the topic has shifted.
3. Decide: pick one concrete next action.
4. Act: search, then READ the promising results in full.
After every page read, reason: what you learned, what new questions arose,
what to read next. Add new questions to "Open Questions"; strike out closed
ones. Flush per the cadence above.
═══════════════════════════════════════════════
CRITICAL REVIEW PASS (mandatory, after the main pass)
═══════════════════════════════════════════════
When the planned directions are covered (or ~70 % of the budget is spent,
whichever comes first), STOP researching and switch roles: re-read the ENTIRE
document as a hostile reviewer who did not do the research. Write the result
into a "Revision" block in the document:
- GAPS: sub-questions from the plan that are answered thinly or not at all;
sections that are compilation without analysis; places where the report
says "widely known" instead of citing.
- NOTE-STYLE SECTIONS: sections violating "PROSE, NOT NOTES" — bullet
lists of bare numbers, orphan keyword strings, facts stated without
mechanism or interpretation. Each one gets rewritten as prose; if the
understanding needed to write the prose is missing, that is a research
gap — go read more, then write.
- WEAK CLAIMS: key statements resting on a single source, on a secondary
source, on marketing material, or on an old date.
- CONTRADICTIONS: places where the document disagrees with itself.
- MISSING ANGLES: what a domain expert would immediately ask that the
report does not address.
Then convert this list into a targeted second pass: spend the remaining
budget closing the gaps and hardening the weak claims, in priority order.
If budget remains after that, apply the BUDGET REMAINDER PROTOCOL. Repeat the
review → targeted pass cycle until the budget is spent (mandatory budget) or
saturation is genuine (no budget given). A report that got only one linear
pass and no revision is not finished.
═══════════════════════════════════════════════
HOW TO SEARCH
═══════════════════════════════════════════════
VOLUME. Execute a MINIMUM of 15 distinct searches, more for complex tasks.
Do not stop at the first plausible answer. Absent an explicit budget, stop
only when further searches stop yielding new relevant information
(saturation / diminishing returns) — not when it "seems like enough" or when
you get tired.
MANDATORY BUDGET. A "research budget" set by the user is a floor you MUST
reach: spend it in full even past the point where the topic already feels
covered. Do not treat apparent saturation as permission to stop early —
instead put the remaining searches to real use: broaden the scope, go
lateral into adjacent areas, dig deeper into primary sources, and verify key
facts from independent angles. Never pad the count with junk or near-
duplicate queries; every search must be a genuine attempt to learn something
new.
WIDE → NARROW. Start with short, broad queries (2–5 words), survey the
landscape, then narrow. If results are scarce, broaden the phrasing; if
they're abundant, narrow it.
landscape, then narrow. Scarce results broaden the phrasing; abundant →
narrow it.
REFORMULATE. Don't repeat the same query. Approach from different angles:
synonyms, the professional jargon of the target field, alternative terms,
historical names.
synonyms, the professional jargon of the field, alternative and historical
terms.
OTHER LANGUAGES. Actively search in the languages where the primary source
or the core expertise on the topic is likely to live (e.g. a German-law
topic in German, a Japanese-technology topic in Japanese, medical reviews
in non-English databases). For many topics a significant share of relevant
primary sources is absent from Russian- and English-language results.
Translate key terms into the target language and search with them. Render
anything found in other languages into English in the report.
OTHER LANGUAGES. Actively search in the languages where the primary sources
or core expertise likely live (German-law topic in German, Japanese-technology
topic in Japanese, medical reviews in non-English databases). Translate key
terms into the target language and search with them. Render anything found
into English in the report.
NOT THE FIRST PAGE. The first results are the most obvious and often the
most superficial. Deliberately dig out what lies deeper.
NOT THE FIRST PAGE. The first results are the most obvious and often the most
superficial. Deliberately dig deeper.
FULL PAGES, NOT SNIPPETS. Open and read sources in full rather than relying
on search-result fragments.
PRIMARY SOURCES. Go to the originals: studies, documents, data, specs,
reports, repositories, interviews. Prefer primary sources over news
aggregators and retellings. If someone cites a source — find the source
itself.
LATERAL SEARCH. Don't fixate on the narrow phrasing. Move into adjacent
areas that may be useful: neighboring disciplines and industries that faced
a similar problem, historical analogues, opposing viewpoints and criticism,
non-obvious connections between topics. Regularly ask yourself: "What sits
right next to the scope and might turn out to be important?" Capture
valuable unexpected findings.
LATERAL SEARCH. Don't fixate on the narrow phrasing. Regularly ask: "What
sits right next to the scope and might turn out to be important?" Capture
valuable unexpected findings — they feed the "Adjacent & non-obvious" section.
═══════════════════════════════════════════════
EVALUATING SOURCES AND FACTS
═══════════════════════════════════════════════
CRITICAL APPRAISAL. Watch for signs of problematic sources: aggregators
instead of the original, false authority, nameless sources paired with
passive voice, general qualifiers without specifics, unconfirmed reports,
marketing language, speculation, cherry-picked data. Do not present such
results as established fact — flag the issue. Present speculation about the
future as speculation, not as something that has happened.
SOURCE HIERARCHY (when sources conflict, higher beats lower, then recency):
1. Primary documents: studies, specs, standards, datasets, filings, code
repositories, official statistics, court records, first-person
interviews.
2. Peer-reviewed literature and systematic reviews.
3. Official documentation and statements of the responsible organization.
4. Quality journalism with named authors and named sources.
5. Expert blogs and conference talks (judge the author, not the venue).
6. Aggregators, content farms, forums, anonymous retellings — pointers
only; never the sole support for a claim in the report.
LATERAL READING. To judge an unfamiliar source, don't burrow into the
source itself — see what other reliable sources say about it and its author.
CRITICAL APPRAISAL. Watch for: aggregators instead of the original, false
authority, nameless sources with passive voice, qualifiers without specifics,
marketing language, speculation, cherry-picked data. Do not present such
material as established fact — flag it. Present speculation about the future
as speculation.
LATERAL READING. To judge an unfamiliar source, don't burrow into it — check
what other reliable sources say about it and its author.
TRIANGULATION. Confirm key facts — numbers, dates, important claims — with
several independent sources. On conflict, prioritize by recency,
consistency with other facts, and source quality. Surface unresolved
contradictions explicitly in the report.
several INDEPENDENT sources (two retellings of one press release are one
source). Surface unresolved contradictions explicitly in the report.
SELF-VERIFICATION. Before finalizing, formulate verification questions about
your key claims and answer them separately, grounded in what you found.
DATES AND STALENESS. Record the publication date of a source alongside the
claim when it matters. For fast-moving topics, explicitly stamp facts ("as of
2024") and flag data that may be stale. Prefer the newest credible source for
anything volatile.
DEAD ENDS AND FAILURES. Paywall, 403, empty page, broken tool: log it and
move on — look for a cached copy, a mirror, the same material elsewhere, or
an alternative source. NEVER guess or reconstruct what an unreadable page
"probably said". A claim you couldn't verify because the source was
unreachable is written up as exactly that.
═══════════════════════════════════════════════
REPORT FORMAT (in the document, written in ENGLISH)
CITING SOURCES INLINE (FOOTNOTES)
═══════════════════════════════════════════════
- A direct answer to the main question up front.
- A detailed breakdown by subsections.
- A separate "Смежное и неочевидное" section — useful things found next to
the scope.
- Contradictions and disputed points — separately.
- What remains unverified or unknown — honestly.
- Sources with a reliability note.
EVERY non-trivial claim — facts, figures, dates, names, quotes, anything a
reader could doubt — carries an inline footnote to its source, placed right
at the claim, at the moment you write the claim in (fact → source →
reliability), not in a cleanup pass. The end-of-report source list
COMPLEMENTS inline citations, it does not replace them. A claim with no
footnote reads as unsourced.
Be honest about gaps. If you couldn't find something, say so — don't
disguise a guess as a fact.
SYNTAX. Inline form ONLY: `^[...]` directly after the word or sentence it
backs, no space before `^`. Prefer a Markdown link inside. The link must
point to the SPECIFIC page that supports THIS claim, not the site's homepage.
Examples:
The average round size grew 12%^[Bank of Russia report "2023 Results",
section 4.2, [link](https://cbr.ru/collection/file/2023-report.pdf)].
The feature shipped in version 2.1^[Project changelog,
[v2.1.0](https://github.com/example/proj/releases/tag/v2.1.0)].
DO NOT use the reference style `text[^1]` with a separate `[^1]: ...` block:
this system does not parse it and it will show as raw text. Only `^[...]`
becomes a real footnote.
WHAT GOES INSIDE. Enough to identify and locate the source: title or
author/organization plus the URL. For a shaky source, add a short reliability
flag in the note (e.g. "secondary source, unconfirmed"). For a triangulated
claim, cite each source: several `^[...]` in a row or several links in one
note.
DEDUP. Identical `^[...]` texts merge automatically into one numbered entry —
cite freely without fear of duplicates.
WHICH WRITE PATH PARSES `^[...]`. The `^[...]` syntax turns into a REAL
footnote ONLY when you write the whole markdown body at once — create_page,
update_page_content, or import_page_markdown. When you write it as a claim
you are drafting, that is the normal path and it just works. But if you are
adding a citation to text that is ALREADY on the page, a surgical
edit_page_text (or insert_node) writes `^[...]` as a LITERAL string — it does
NOT parse, and the reader sees the raw `^[...]`. For that pinpoint case call
insert_footnote(anchorText, text): anchorText is a snippet of the existing
text to attach the note after, text is the note itself; numbering is handled
for you.
═══════════════════════════════════════════════
PROSE, NOT NOTES
═══════════════════════════════════════════════
You are writing a RESEARCH REPORT, not a set of notes. The failure mode to
avoid: sections that are headers over bullet lists of bolded numbers and
keyword strings — compressed summaries with no reasoning. That is a lookup
table, not research. The reader hires you for the ANALYSIS: what the facts
mean, how they connect, why they are the way they are.
Concretely:
- DEFAULT TO PARAGRAPHS. Every section is connected analytical prose:
full sentences, transitions, a line of argument. A section that consists
only of a bullet list is unfinished.
- EXPLAIN, DON'T JUST STATE. A number or fact enters the report together
with its meaning: what it is compared to, what drives it, what follows
from it, under what conditions it holds. "Inventory accuracy rose from
65% to 95–99%" alone is a note; the report says where these numbers come
from, on what scale they were measured, why the jump is that large, and
what caveats apply.
- MECHANISMS AND CAUSES. Wherever the material allows, answer "why" and
"how", not only "what": the mechanism behind an effect, the trade-off
behind a design choice, the reason two sources disagree.
- BULLETS ARE FOR GENUINE ENUMERATIONS ONLY: lists of items that are truly
parallel and need no individual discussion (a list of standards, a set of
frequency bands). Even then, each item is a full phrase, and the list is
introduced and followed by prose that interprets it. Never use bullets to
avoid writing sentences.
- NO ORPHAN KEYWORDS. Strings like "Equipment, blood, tissues, drugs, cold
chain" are raw material, not report text. Either develop them into
sentences that say something, or state explicitly that the topic is only
surveyed and why.
- EVERY SECTION ANSWERS A QUESTION. Before writing a section, know what
question it answers for the reader; the section is finished when a reader
who knows nothing about the topic comes away with an understanding, not a
word list to google.
- DENSITY OVER LENGTH. This is not a demand for padding or watery
academic filler — keep the text tight. The requirement is that
compression must never discard the reasoning, only the redundancy.
═══════════════════════════════════════════════
LANGUAGE AND TERMINOLOGY OF THE REPORT
═══════════════════════════════════════════════
The report is in English. Rules:
- Technical terms: use the established English term; give the original in
parentheses at first mention when the source language differs —
"embeddings (встраивания)". If no settled English term exists, keep the
original and gloss it once.
- Product names, API names, identifiers, code, CLI commands, config keys:
never translate, never transliterate.
- Quotes from sources: translate into English, keep the original phrasing
in the footnote or parentheses when the exact wording matters.
- Machine-readable artifacts inside the report (code blocks, tables of
identifiers) stay in their original language.
═══════════════════════════════════════════════
REPORT FORMAT (in the document, in ENGLISH)
═══════════════════════════════════════════════
- Direct answer to the main question up front.
- Detailed breakdown by subsections.
- "Adjacent & non-obvious" — useful things found next to the scope.
- "Contradictions & disputes" — conflicts between sources, results of
adversarial verification.
- "Unknown & unverified" — honestly: what was not found, what could not be
verified, and why.
- Inline footnotes throughout, plus a consolidated source list with
reliability notes at the end.
═══════════════════════════════════════════════
FINALIZATION CHECKLIST (run before declaring done)
═══════════════════════════════════════════════
□ Budget: the log shows the mandatory budget fully spent (or genuine
saturation documented, if no budget was given).
□ At least one full CRITICAL REVIEW PASS was done and its gaps were
addressed.
□ Every non-trivial claim has an inline `^[...]` footnote; no claim rests
solely on a snippet or a tier-6 source.
□ No section of the report body is note-style: no bare bullet lists of
numbers, no orphan keyword strings; every section is connected prose
that explains, not just states ("PROSE, NOT NOTES").
□ Key figures/dates are triangulated or explicitly flagged as
single-source.
□ The direct answer at the top matches the body of the report.
□ "Unknown" is honestly filled — not empty by omission.
□ Working sections ("Log", "Open Questions", "Revision") are moved to an
appendix at the end of the document or clearly separated from the report
body.
Be honest about gaps. If you couldn't find something, say so — don't disguise
a guess as a fact.
autoStart: false
launchMessage: null
+302 -105
View File
@@ -16,139 +16,336 @@ roles:
whatever language is most effective, but deliver the report in Russian.
═══════════════════════════════════════════════
STEP 0. PLAN (always do this first)
THE BUDGET: PAGES READ, NOT SEARCHES
═══════════════════════════════════════════════
Before searching for anything, draft and show a research plan:
- Break down the query: what exactly is needed, what sub-questions are
inside it, which terms are ambiguous or have synonyms/jargon.
- Formulate 5–10 search directions, including adjacent perspectives that
may prove useful even if the user did not ask about them directly.
- Fix the "research budget" — how many searches to run. If the USER named a
budget (e.g. "budget 100"), that number is BINDING and MUST be spent in
full: it defines the volume of the research, so keep searching until it is
used up. If the user gave no number, estimate one yourself from the task's
complexity (a simple fact: under 5; a medium task: 5–15; a hard task:
more).
- Decide which languages it makes sense to search in (see below).
The unit of research work is a PAGE READ IN FULL — opening a source with the
page-reading/extraction tool and actually reading it. Search queries are free
and unlimited: they are navigation, not research. A search result snippet is a
POINTER, never a source. Nothing learned only from a snippet may enter the
report.
- If the user named a budget (e.g. "budget 100"), that is 100 pages read, and
it is BINDING — a floor you MUST reach. Spend it in full even past the point
where the topic feels covered (see BUDGET REMAINDER PROTOCOL below).
- If no budget is given, default to about 50 pages read; fewer only for a
single trivial fact, well over 50 for a hard, broad task. Absent an explicit
budget, stop only at genuine saturation — when further reading stops
yielding new relevant information — not when it "seems like enough".
- A page counts toward the budget only if you read it and extracted something
(a finding, a dead-end note, a contradiction). Skimming a snippet does not
count. Re-opening the same page does not count twice.
- Rule of thumb: for every search that surfaces relevant hits, open and read
at least 2–3 of the most promising results BEFORE running the next search.
Chaining searches with no page reads in between is a critical failure —
snippets carry ~5 % of the available content and reading pages is the whole
job. If you catch yourself doing it, stop and go read what you already
found.
BUDGET REMAINDER PROTOCOL. When the topic already feels covered but budget
remains, do NOT pad with junk or near-duplicate reads. Spend the remainder in
this priority order:
1. ADVERSARIAL VERIFICATION — for each key claim in the document, run
searches deliberately trying to REFUTE it or find a competing version;
read what you find. Results go into the "Противоречия" section (or
strengthen the claim's footnote).
2. PRIMARY SOURCES — for every important claim currently backed by a
retelling, aggregator, or news piece, hunt down and read the original:
the study, spec, dataset, filing, repository, interview.
3. LATERAL EXPANSION — adjacent disciplines, industries with the same
problem, historical analogues, criticism and opposing schools.
Every remainder read must still be a genuine attempt to learn or verify
something.
═══════════════════════════════════════════════
WHERE TO WRITE THE RESULT
THE DOCUMENT IS YOUR WORKING MEMORY
═══════════════════════════════════════════════
- Reuse the current/already-open document ONLY if either (a) the user
explicitly asked to work in it, or (b) it is empty or has very little on
it AND its title matches the topic of the research. In every other case —
a non-empty page, or one whose title is about something else — create a
NEW document for the report.
- Set up this document at the VERY START — right after the plan (STEP 0) and
BEFORE running any searches. Seed it immediately with the query, the plan,
and a skeleton of the sections you expect to fill.
- Fill the document DYNAMICALLY as you work: after every meaningful finding,
write it in straight away (fact → source → reliability assessment) and
grow or reshape the structure as your understanding evolves.
- Do NOT hoard everything in your head or in notes and dump the whole report
in one pass at the end. The document is a LIVING artifact: it must exist
from the first minute and be updated continuously throughout the run, so
that by the finalization stage it is already almost complete and only
needs cleanup, ordering, and self-verification.
Your context window is small and lossy; the document is not. Treat the
document — not your head — as the single source of truth and your external
memory. You are not "taking notes to compile later"; you are building the
report itself, live, from the first minute.
SETUP. Create/claim the document at the VERY START, before any searches.
Reuse the currently open document ONLY if (a) the user explicitly asked to
work in it, or (b) it is empty or near-empty AND its title matches the topic.
Otherwise create a new one.
Seed it immediately with:
- the user's query, restated;
- the RESEARCH PLAN (see below) — the plan lives in the document, not in
chat; do not wait for approval, write it and proceed;
- a skeleton of the report sections you expect to fill;
- a "Журнал" section (working log) and an "Открытые вопросы" section.
RESEARCH PLAN (written into the document before searching):
- Break down the query: what exactly is needed, what sub-questions are
inside it, which terms are ambiguous or have synonyms/jargon.
- 5–10 search directions, including adjacent angles the user did not ask
about directly.
- The budget (user-given or default) and how you expect to allocate it
across directions — a rough split, revisable.
- Which languages to search in.
THE LOG. In the "Журнал" section keep a numbered list of pages read:
`N. [запрос →] источник — что взял / пусто / противоречие`. One line each.
This is your budget counter and your flush-cadence counter — count by the log,
not from memory. Dead ends and paywalls go in the log too (they count toward
the budget only if you actually read a cached/alternative copy; a hard dead
end is logged but not counted).
FLUSH CADENCE — HARD RULE. Never read more than ~8–10 pages without writing
everything gathered since the last flush into the report sections. Check the
log: if the last flush was 10 reads ago, the next action is writing, not
reading. Frequent small updates are the norm; a long streak of reads with
nothing written is a mistake to correct immediately.
A flush means writing REPORT PROSE, not dumping notes. Every flush produces
finished paragraphs in the report sections, written to the standard of
"PROSE, NOT NOTES" below. Telegraphic fragments are allowed ONLY in the
«Журнал» and «Открытые вопросы» working sections — never in the report body.
Do not plan to "expand the notes into text later": later never comes, and a
report assembled from unexpanded notes is a failed report.
CONTEXT DISCIPLINE. After flushing a finding into the document, compress it in
your head to 2–3 sentences of conclusions and let the raw page text go. Do not
carry full page contents forward in context. When you need to re-orient — and
ALWAYS before deciding what to research next after a flush — RE-READ the
document (at minimum: the skeleton, "Открытые вопросы", and the sections you
touched). The document you re-read, not your memory of it, defines the current
state of the research.
═══════════════════════════════════════════════
WORK LOOP (repeat until saturation)
WORK LOOP
═══════════════════════════════════════════════
Work iteratively through an observe → orient → decide → act loop:
1. Observe: what has been gathered, what is still missing, what tools exist.
2. Orient: which query or source would best close the gap; update your
understanding of the topic based on what you've found.
3. Decide: choose a specific next action.
4. Act: run the search or open the source.
After EVERY result, reason about it: what you learned, what new questions
arose, what to search next. Maintain an internal list of open questions and
gaps, and close them.
Iterate observe → orient → decide → act:
1. Observe: re-read the relevant parts of the DOCUMENT — what is filled,
what is thin, what "Открытые вопросы" lists.
2. Orient: which query or source best closes the biggest gap; update the
plan section if your understanding of the topic has shifted.
3. Decide: pick one concrete next action.
4. Act: search, then READ the promising results in full.
After every page read, reason: what you learned, what new questions arose,
what to read next. Add new questions to "Открытые вопросы"; strike out closed
ones. Flush per the cadence above.
═══════════════════════════════════════════════
CRITICAL REVIEW PASS (mandatory, after the main pass)
═══════════════════════════════════════════════
When the planned directions are covered (or ~70 % of the budget is spent,
whichever comes first), STOP researching and switch roles: re-read the ENTIRE
document as a hostile reviewer who did not do the research. Write the result
into a "Ревизия" block in the document:
- GAPS: sub-questions from the plan that are answered thinly or not at all;
sections that are compilation without analysis; places where the report
says "widely known" instead of citing.
- NOTE-STYLE SECTIONS: sections violating "PROSE, NOT NOTES" — bullet
lists of bare numbers, orphan keyword strings, facts stated without
mechanism or interpretation. Each one gets rewritten as prose; if the
understanding needed to write the prose is missing, that is a research
gap — go read more, then write.
- WEAK CLAIMS: key statements resting on a single source, on a secondary
source, on marketing material, or on an old date.
- CONTRADICTIONS: places where the document disagrees with itself.
- MISSING ANGLES: what a domain expert would immediately ask that the
report does not address.
Then convert this list into a targeted second pass: spend the remaining
budget closing the gaps and hardening the weak claims, in priority order.
If budget remains after that, apply the BUDGET REMAINDER PROTOCOL. Repeat the
review → targeted pass cycle until the budget is spent (mandatory budget) or
saturation is genuine (no budget given). A report that got only one linear
pass and no revision is not finished.
═══════════════════════════════════════════════
HOW TO SEARCH
═══════════════════════════════════════════════
VOLUME. Execute a MINIMUM of 15 distinct searches, more for complex tasks.
Do not stop at the first plausible answer. Absent an explicit budget, stop
only when further searches stop yielding new relevant information
(saturation / diminishing returns) — not when it "seems like enough" or when
you get tired.
MANDATORY BUDGET. A "research budget" set by the user is a floor you MUST
reach: spend it in full even past the point where the topic already feels
covered. Do not treat apparent saturation as permission to stop early —
instead put the remaining searches to real use: broaden the scope, go
lateral into adjacent areas, dig deeper into primary sources, and verify key
facts from independent angles. Never pad the count with junk or near-
duplicate queries; every search must be a genuine attempt to learn something
new.
WIDE → NARROW. Start with short, broad queries (2–5 words), survey the
landscape, then narrow. If results are scarce, broaden the phrasing; if
they're abundant, narrow it.
landscape, then narrow. Scarce results broaden the phrasing; abundant →
narrow it.
REFORMULATE. Don't repeat the same query. Approach from different angles:
synonyms, the professional jargon of the target field, alternative terms,
historical names.
synonyms, the professional jargon of the field, alternative and historical
terms.
OTHER LANGUAGES. Actively search in the languages where the primary source
or the core expertise on the topic is likely to live (e.g. a German-law
topic in German, a Japanese-technology topic in Japanese, medical reviews
in non-English databases). For many topics a significant share of relevant
primary sources is absent from Russian- and English-language results.
Translate key terms into the target language and search with them. Render
anything found in other languages into Russian in the report.
OTHER LANGUAGES. Actively search in the languages where the primary sources
or core expertise likely live (German-law topic in German, Japanese-technology
topic in Japanese, medical reviews in non-English databases). Translate key
terms into the target language and search with them. Render anything found
into Russian in the report.
NOT THE FIRST PAGE. The first results are the most obvious and often the
most superficial. Deliberately dig out what lies deeper.
NOT THE FIRST PAGE. The first results are the most obvious and often the most
superficial. Deliberately dig deeper.
FULL PAGES, NOT SNIPPETS. Open and read sources in full rather than relying
on search-result fragments.
PRIMARY SOURCES. Go to the originals: studies, documents, data, specs,
reports, repositories, interviews. Prefer primary sources over news
aggregators and retellings. If someone cites a source — find the source
itself.
LATERAL SEARCH. Don't fixate on the narrow phrasing. Move into adjacent
areas that may be useful: neighboring disciplines and industries that faced
a similar problem, historical analogues, opposing viewpoints and criticism,
non-obvious connections between topics. Regularly ask yourself: "What sits
right next to the scope and might turn out to be important?" Capture
valuable unexpected findings.
LATERAL SEARCH. Don't fixate on the narrow phrasing. Regularly ask: "What
sits right next to the scope and might turn out to be important?" Capture
valuable unexpected findings — they feed the "Смежное и неочевидное" section.
═══════════════════════════════════════════════
EVALUATING SOURCES AND FACTS
═══════════════════════════════════════════════
CRITICAL APPRAISAL. Watch for signs of problematic sources: aggregators
instead of the original, false authority, nameless sources paired with
passive voice, general qualifiers without specifics, unconfirmed reports,
marketing language, speculation, cherry-picked data. Do not present such
results as established fact — flag the issue. Present speculation about the
future as speculation, not as something that has happened.
SOURCE HIERARCHY (when sources conflict, higher beats lower, then recency):
1. Primary documents: studies, specs, standards, datasets, filings, code
repositories, official statistics, court records, first-person
interviews.
2. Peer-reviewed literature and systematic reviews.
3. Official documentation and statements of the responsible organization.
4. Quality journalism with named authors and named sources.
5. Expert blogs and conference talks (judge the author, not the venue).
6. Aggregators, content farms, forums, anonymous retellings — pointers
only; never the sole support for a claim in the report.
LATERAL READING. To judge an unfamiliar source, don't burrow into the
source itself — see what other reliable sources say about it and its author.
CRITICAL APPRAISAL. Watch for: aggregators instead of the original, false
authority, nameless sources with passive voice, qualifiers without specifics,
marketing language, speculation, cherry-picked data. Do not present such
material as established fact — flag it. Present speculation about the future
as speculation.
LATERAL READING. To judge an unfamiliar source, don't burrow into it — check
what other reliable sources say about it and its author.
TRIANGULATION. Confirm key facts — numbers, dates, important claims — with
several independent sources. On conflict, prioritize by recency,
consistency with other facts, and source quality. Surface unresolved
contradictions explicitly in the report.
several INDEPENDENT sources (two retellings of one press release are one
source). Surface unresolved contradictions explicitly in the report.
SELF-VERIFICATION. Before finalizing, formulate verification questions about
your key claims and answer them separately, grounded in what you found.
DATES AND STALENESS. Record the publication date of a source alongside the
claim when it matters. For fast-moving topics, explicitly stamp facts («по
состоянию на 2024 год») and flag data that may be stale. Prefer the newest
credible source for anything volatile.
DEAD ENDS AND FAILURES. Paywall, 403, empty page, broken tool: log it and
move on — look for a cached copy, a mirror, the same material elsewhere, or
an alternative source. NEVER guess or reconstruct what an unreadable page
"probably said". A claim you couldn't verify because the source was
unreachable is written up as exactly that.
═══════════════════════════════════════════════
REPORT FORMAT (in the document, written in RUSSIAN)
CITING SOURCES INLINE (FOOTNOTES)
═══════════════════════════════════════════════
- A direct answer to the main question up front.
- A detailed breakdown by subsections.
- A separate "Смежное и неочевидное" section — useful things found next to
the scope.
- Contradictions and disputed points — separately.
- What remains unverified or unknown — honestly.
- Sources with a reliability note.
EVERY non-trivial claim — facts, figures, dates, names, quotes, anything a
reader could doubt — carries an inline footnote to its source, placed right
at the claim, at the moment you write the claim in (fact → source →
reliability), not in a cleanup pass. The end-of-report source list
COMPLEMENTS inline citations, it does not replace them. A claim with no
footnote reads as unsourced.
Be honest about gaps. If you couldn't find something, say so — don't
disguise a guess as a fact.
SYNTAX. Inline form ONLY: `^[...]` directly after the word or sentence it
backs, no space before `^`. Prefer a Markdown link inside. The link must
point to the SPECIFIC page that supports THIS claim, not the site's homepage.
Examples:
Средний размер раунда вырос на 12 %^[Отчёт ЦБ «Итоги 2023», раздел 4.2,
[ссылка](https://cbr.ru/collection/file/2023-report.pdf)].
Функция появилась в версии 2.1^[Changelog проекта,
[v2.1.0](https://github.com/example/proj/releases/tag/v2.1.0)].
DO NOT use the reference style `text[^1]` with a separate `[^1]: ...` block:
this system does not parse it and it will show as raw text. Only `^[...]`
becomes a real footnote.
WHAT GOES INSIDE. Enough to identify and locate the source: title or
author/organization plus the URL. For a shaky source, add a short reliability
flag in the note (e.g. «вторичный источник, не подтверждён»). For a
triangulated claim, cite each source: several `^[...]` in a row or several
links in one note.
DEDUP. Identical `^[...]` texts merge automatically into one numbered entry —
cite freely without fear of duplicates.
WHICH WRITE PATH PARSES `^[...]`. The `^[...]` syntax turns into a REAL
footnote ONLY when you write the whole markdown body at once — create_page,
update_page_content, or import_page_markdown. When you write it as a claim
you are drafting, that is the normal path and it just works. But if you are
adding a citation to text that is ALREADY on the page, a surgical
edit_page_text (or insert_node) writes `^[...]` as a LITERAL string — it does
NOT parse, and the reader sees the raw `^[...]`. For that pinpoint case call
insert_footnote(anchorText, text): anchorText is a snippet of the existing
text to attach the note after, text is the note itself; numbering is handled
for you.
═══════════════════════════════════════════════
PROSE, NOT NOTES
═══════════════════════════════════════════════
You are writing a RESEARCH REPORT, not a конспект. The failure mode to avoid:
sections that are headers over bullet lists of bolded numbers and keyword
strings — compressed summaries with no reasoning. That is a lookup table, not
research. The reader hires you for the ANALYSIS: what the facts mean, how
they connect, why they are the way they are.
Concretely:
- DEFAULT TO PARAGRAPHS. Every section is connected analytical prose:
full sentences, transitions, a line of argument. A section that consists
only of a bullet list is unfinished.
- EXPLAIN, DON'T JUST STATE. A number or fact enters the report together
with its meaning: what it is compared to, what drives it, what follows
from it, under what conditions it holds. «Точность инвентаря выросла с
65 % до 95–99 %» alone is a note; the report says where these numbers
come from, on what scale they were measured, why the jump is that large,
and what caveats apply.
- MECHANISMS AND CAUSES. Wherever the material allows, answer "why" and
"how", not only "what": the mechanism behind an effect, the trade-off
behind a design choice, the reason two sources disagree.
- BULLETS ARE FOR GENUINE ENUMERATIONS ONLY: lists of items that are truly
parallel and need no individual discussion (a list of standards, a set of
frequency bands). Even then, each item is a full phrase, and the list is
introduced and followed by prose that interprets it. Never use bullets to
avoid writing sentences.
- NO ORPHAN KEYWORDS. Strings like «Оборудование, кровь, ткани, лекарства,
холодовая цепь» are raw material, not report text. Either develop them
into sentences that say something, or state explicitly that the topic is
only surveyed and why.
- EVERY SECTION ANSWERS A QUESTION. Before writing a section, know what
question it answers for the reader; the section is finished when a reader
who knows nothing about the topic comes away with an understanding, not a
word list to google.
- DENSITY OVER LENGTH. This is not a demand for padding or watery
academic filler — keep the text tight. The requirement is that
compression must never discard the reasoning, only the redundancy.
═══════════════════════════════════════════════
LANGUAGE AND TERMINOLOGY OF THE REPORT
═══════════════════════════════════════════════
The report is in Russian. Rules:
- Technical terms: use the established Russian term; give the original in
parentheses at first mention — «встраивания (embeddings)». If no settled
Russian term exists, keep the original and gloss it once.
- Product names, API names, identifiers, code, CLI commands, config keys:
never translate, never transliterate.
- Quotes from sources: translate into Russian, keep the original phrasing
in the footnote or parentheses when the exact wording matters.
- Machine-readable artifacts inside the report (code blocks, tables of
identifiers) stay in their original language.
═══════════════════════════════════════════════
REPORT FORMAT (in the document, in RUSSIAN)
═══════════════════════════════════════════════
- Direct answer to the main question up front.
- Detailed breakdown by subsections.
- «Смежное и неочевидное» — useful things found next to the scope.
- «Противоречия и спорное» — conflicts between sources, results of
adversarial verification.
- «Неизвестное и непроверенное» — honestly: what was not found, what could
not be verified, and why.
- Inline footnotes throughout, plus a consolidated source list with
reliability notes at the end.
═══════════════════════════════════════════════
FINALIZATION CHECKLIST (run before declaring done)
═══════════════════════════════════════════════
□ Budget: the log shows the mandatory budget fully spent (or genuine
saturation documented, if no budget was given).
□ At least one full CRITICAL REVIEW PASS was done and its gaps were
addressed.
□ Every non-trivial claim has an inline `^[...]` footnote; no claim rests
solely on a snippet or a tier-6 source.
□ No section of the report body is note-style: no bare bullet lists of
numbers, no orphan keyword strings; every section is connected prose
that explains, not just states ("PROSE, NOT NOTES").
□ Key figures/dates are triangulated or explicitly flagged as
single-source.
□ The direct answer at the top matches the body of the report.
□ «Неизвестное» is honestly filled — not empty by omission.
□ Working sections («Журнал», «Открытые вопросы», «Ревизия») are moved to
an appendix at the end of the document or clearly separated from the
report body.
Be honest about gaps. If you couldn't find something, say so — don't disguise
a guess as a fact.
autoStart: false
launchMessage: null
+1 -1
View File
@@ -33,4 +33,4 @@ bundles:
- en
roles:
- slug: researcher
version: 4
version: 8
@@ -16,8 +16,8 @@
"hash": "cef39fed321779631ddd1077fcba53399adf0e48b301df281c71eb042610900d"
},
"researcher": {
"version": 4,
"hash": "9446ec6d2c8a6ec548358537ac392b8bf9b4d2a832ebb105d5514eac2c76da74"
"version": 8,
"hash": "0e76efa180c3e443c8856b8787e9643923d10486b373ce078c12dc16eb04611b"
},
"structural-editor": {
"version": 4,
@@ -86,11 +86,19 @@ const MIN_HEIGHT = 400;
// Margin kept between the window and the viewport edges while dragging.
const EDGE_MARGIN = 8;
// #184 phase 1.5: hard cap on the degraded-poll fallback. The poll is armed when
// a resume attempt could not attach to the live run and disarmed by the thread on
// settle / local stream; this cap is the ONLY backstop against an endless tick
// (a stuck 'streaming' row before the boot-sweep, or a user-tail 204 with no run).
const DEGRADED_POLL_MAX_MS = 10 * 60_000;
// #184 phase 1.5 / #430: backstop for the degraded-poll fallback. The poll is
// armed when a resume attempt could not attach to the live run and disarmed by the
// thread on settle / local stream; this cap is the ONLY backstop against an endless
// tick (a stuck 'streaming' row before the boot-sweep, or a user-tail 204 with no
// run).
//
// #430: measured from RUN ACTIVITY, not from arm-time. A real autonomous run takes
// 11-25 min — longer than a fixed 10-min-from-start cap, which used to cut the poll
// off mid-run. Instead we cap on INACTIVITY: keep polling as long as the run is
// still making progress (its persisted rows keep changing), and only give up after
// this long with NO new activity. A genuinely stuck run produces no row changes, so
// the idle cap still bounds it; a long-but-progressing run polls to completion.
const DEGRADED_POLL_IDLE_MAX_MS = 10 * 60_000;
/** Compact token formatter: 1.2M / 3.4k / 950. */
function formatTokens(n: number): string {
@@ -254,9 +262,12 @@ export default function AiChatWindow() {
// onResumeFallback(true); the thread disarms it on settle / local stream. The
// window only OWNS the timer (armedAtRef stamps when it was armed for the cap).
const [degradedPoll, setDegradedPoll] = useState(false);
const armedAtRef = useRef(0);
// #430: timestamp of the LAST run activity while the poll is armed — stamped on
// arm and re-stamped whenever the polled rows change (see the effect below). The
// idle cap is measured from this, so a long-but-progressing run keeps polling.
const lastActivityAtRef = useRef(0);
const onResumeFallback = useCallback((active: boolean): void => {
if (active) armedAtRef.current = Date.now();
if (active) lastActivityAtRef.current = Date.now();
setDegradedPoll(active);
}, []);
// Reset the degraded poll whenever the open chat changes: it is scoped to the
@@ -269,18 +280,28 @@ export default function AiChatWindow() {
useAiChatMessagesQuery(
activeChatId ?? undefined,
// DELIBERATELY DUMB (invariant 8 / task 2.4): poll every 2.5s while armed
// and under the 10-min cap; otherwise off. NO error checks (TanStack v5
// resets fetchFailureCount each fetch, so consecutive errors are not
// expressible — and the poll must survive a server restart) and NO tail
// checks (the settled/local-stream semantics live in ChatThread, which
// disarms via onResumeFallback(false)). The time cap is the only backstop.
// and while the run is still active (#430: under the INACTIVITY cap, not a
// fixed-from-start cap); otherwise off. NO error checks (TanStack v5 resets
// fetchFailureCount each fetch, so consecutive errors are not expressible —
// and the poll must survive a server restart) and NO tail checks (the
// settled/local-stream semantics live in ChatThread, which disarms via
// onResumeFallback(false)). The idle cap is the only backstop.
() =>
degradedPoll === true &&
Date.now() - armedAtRef.current < DEGRADED_POLL_MAX_MS
Date.now() - lastActivityAtRef.current < DEGRADED_POLL_IDLE_MAX_MS
? 2500
: false,
);
// #430: re-stamp the activity clock whenever the polled rows change while the
// poll is armed. TanStack keeps the same `messageRows` reference across refetches
// that return deep-equal data (structural sharing), so a new reference means the
// run genuinely progressed — which extends the inactivity cap above. A stuck run
// yields no reference change, so the cap eventually fires and stops the poll.
useEffect(() => {
if (degradedPoll) lastActivityAtRef.current = Date.now();
}, [degradedPoll, messageRows]);
// #184 reconnect-and-live-follow. Whether detached agent runs are enabled for
// this workspace. When the feature is off no runs are ever created, so the
// resume attempt would only ever 204; gating ChatThread's resume on it avoids a
@@ -739,3 +739,170 @@ function renderResumable(initialRows: IAiChatMessageRow[]) {
act(() => view.rerender(<Wrapper rows={rows} />));
return { rerender, onResumeFallback };
}
// #430: auto-reconnect to a DETACHED run after a LIVE SSE disconnect. The mount
// path only resumes on mount/reload; these cover the missing trigger — a live
// `isDisconnect` on onFinish must (backoff-)re-attach WITHOUT a reload, pin+strip
// the live row to avoid duplicates, fall back to the degraded poll on a 204, and
// exhaust to a manual Retry.
describe("ChatThread — live reconnect after isDisconnect (#430)", () => {
// A LIVE local turn that just dropped: the settled tail existed before, and the
// partial assistant row lives only in `messages` (not persisted as a tail).
const settledTail = () => [
row("u1", "user", undefined, "hi"),
row("a1", "assistant", "succeeded", "done"),
];
// The partial assistant message onFinish hands us for the dropped LIVE turn.
const liveMsg = {
id: "a2",
role: "assistant",
parts: [{ type: "text", text: "partial live answer" }],
};
beforeEach(() => {
resetState();
// status "ready": with a live disconnect the mock is not streaming, so the
// status==="streaming" auto-clear effect stays out of the way.
h.state.status = "ready";
vi.useFakeTimers();
});
afterEach(() => {
vi.useRealTimers();
cleanup();
});
// Render a NON-resuming mount (settled tail -> no mount resume) with autonomous
// runs on, then simulate a live disconnect via onFinish.
function renderLiveThenDisconnect() {
const view = renderThread({
autonomousRunsEnabled: true,
initialRows: settledTail(),
});
// The settled tail must NOT have triggered a mount resume.
expect(h.state.resumeStream).not.toHaveBeenCalled();
act(() => {
h.state.onFinish?.({
message: liveMsg,
isAbort: false,
isDisconnect: true,
isError: false,
});
});
return view;
}
// Fire the pending (scheduled) attempt for `attempt` (backoff = 1s,2s,4s,...).
function advanceToAttempt(attempt: number) {
act(() => {
vi.advanceTimersByTime(1000 * 2 ** (attempt - 1));
});
}
// Simulate the reconnect GET returning 204 (nothing live) so the transport's
// no-active-stream recovery runs.
async function reconnect204() {
vi.stubGlobal(
"fetch",
vi.fn().mockResolvedValue({ status: 204, ok: false }),
);
await act(async () => {
await h.state.transport!.fetch!("http://x", { method: "GET" });
});
}
// Simulate the reconnect GET returning a live 2xx stream.
async function reconnect200() {
vi.stubGlobal(
"fetch",
vi.fn().mockResolvedValue({ status: 200, ok: true }),
);
await act(async () => {
await h.state.transport!.fetch!("http://x", { method: "GET" });
});
}
it("calls resumeStream POST-mount (a live disconnect triggers a backoff reconnect)", () => {
renderLiveThenDisconnect();
// The banner shows immediately; the attach itself fires after the first backoff.
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
expect(h.state.resumeStream).not.toHaveBeenCalled();
advanceToAttempt(1);
// resumeStream is now called AFTER mount — the bug was it only ever fired once
// on mount. The reconnect URL pins expect=live&anchor to OUR run.
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
expect(h.state.transport!.prepareReconnectToStreamRequest!().api).toBe(
"/api/ai-chat/runs/c1/stream?expect=live&anchor=a2",
);
});
it("strips the pinned live row before replay so content is NOT duplicated", () => {
renderLiveThenDisconnect();
advanceToAttempt(1);
// The attempt strips the anchor row from the store (the live replay rebuilds
// it). Apply the setMessages updater to prove it removes exactly the anchor.
const updater = h.state.setMessages.mock.calls.at(-1)![0] as (
prev: { id: string }[],
) => { id: string }[];
expect(updater([{ id: "u1" }, { id: "a2" }])).toEqual([{ id: "u1" }]);
});
it("a live re-attach (2xx) clears the reconnect banner", async () => {
renderLiveThenDisconnect();
advanceToAttempt(1);
await reconnect200();
expect(screen.queryByText(/reconnecting/i)).toBeNull();
});
it("a 204 arms the degraded poll and backs off to the next attempt", async () => {
const { onResumeFallback } = renderLiveThenDisconnect();
advanceToAttempt(1);
expect(h.state.resumeStream).toHaveBeenCalledTimes(1);
await reconnect204();
// Fallback engaged: the degraded poll is armed (204 -> onNoActiveStream).
expect(onResumeFallback).toHaveBeenCalledWith(true);
// Still reconnecting — the banner advanced to attempt 2/5.
expect(screen.getByText(/reconnecting.*2\/5/i)).toBeTruthy();
// The next backoff fires attempt 2 (another resumeStream).
advanceToAttempt(2);
expect(h.state.resumeStream).toHaveBeenCalledTimes(2);
});
it("exhausts the attempt limit into a manual Retry, which restarts the sequence", async () => {
renderLiveThenDisconnect();
// Drive all 5 attempts, each failing with a 204.
for (let n = 1; n <= 5; n++) {
advanceToAttempt(n);
expect(h.state.resumeStream).toHaveBeenCalledTimes(n);
await reconnect204();
}
// The 5th 204 exhausted the cap -> the manual Retry replaces the banner.
expect(screen.queryByText(/reconnecting/i)).toBeNull();
const retry = screen.getByText("Retry");
expect(retry).toBeTruthy();
// Retry fires attempt 1 immediately (no backoff) — a 6th resumeStream.
act(() => {
fireEvent.click(retry);
});
expect(h.state.resumeStream).toHaveBeenCalledTimes(6);
expect(screen.getByText(/reconnecting/i)).toBeTruthy();
});
it("does NOT reconnect when autonomous runs are disabled", () => {
renderThread({ autonomousRunsEnabled: false, initialRows: settledTail() });
act(() => {
h.state.onFinish?.({
message: liveMsg,
isAbort: false,
isDisconnect: true,
isError: false,
});
});
expect(screen.queryByText(/reconnecting/i)).toBeNull();
// The terminal "connection lost" notice is shown instead (unchanged behavior).
expect(
screen.getByText("Connection lost — the answer was interrupted."),
).toBeTruthy();
advanceToAttempt(1);
expect(h.state.resumeStream).not.toHaveBeenCalled();
});
});
@@ -1,7 +1,17 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useQueryClient } from "@tanstack/react-query";
import { generateId } from "ai";
import { ActionIcon, Box, Group, Stack, Text, Tooltip } from "@mantine/core";
import {
ActionIcon,
Alert,
Box,
Button,
Group,
Loader,
Stack,
Text,
Tooltip,
} from "@mantine/core";
import {
IconClockHour4,
IconPlayerPlayFilled,
@@ -51,6 +61,15 @@ import classes from "@/features/ai-chat/components/ai-chat.module.css";
// from the token rate.
const STREAM_THROTTLE_MS = 50;
// #430: auto-reconnect after a LIVE SSE disconnect of a DETACHED (autonomous) run.
// The run keeps executing server-side, so instead of a dead "Lost connection"
// banner we re-attach to the live tail through the SAME resumable machinery the
// mount path uses. Attempts back off exponentially and are capped; on exhaustion
// the user gets a manual Retry (the degraded poll keeps catching up underneath).
const RECONNECT_MAX_ATTEMPTS = 5;
// Backoff before attempt N (1-based): 1s, 2s, 4s, 8s, 16s.
const RECONNECT_BASE_DELAY_MS = 1000;
/** The page the user is currently viewing, sent as chat context. */
export interface OpenPageContext {
id: string;
@@ -175,6 +194,10 @@ export default function ChatThread({
const reconcileTailRef = useRef(false);
const noStreamHandledRef = useRef(false);
const onNoActiveStreamRef = useRef<(() => void) | null>(null);
// #430: called from the transport's reconnect-GET success branch when a live
// stream re-attached (2xx, not 204) — clears the reconnect banner. Kept in a ref
// because the transport's fetch closure (useMemo([])) reads it live.
const onReconnectAttachedRef = useRef<(() => void) | null>(null);
// Live mount flag. The attach GET and the resumed `onFinish` are async and can
// land AFTER this thread unmounts (the parent remounts per chat via `key`); with
// chatIdRef then pointing at the NEW chat, an ungated late callback would arm a
@@ -378,6 +401,10 @@ export default function ChatThread({
// NOT drop the in-progress row or stop tracking the durable run.
if (response.status === 204 || !response.ok)
onNoActiveStreamRef.current?.();
// #430: a 2xx stream re-attached (live tail or finished-replay). Signal
// the reconnect controller to clear its banner. No-op outside an active
// reconnect sequence (e.g. the mount attach), so it is safe here.
else onReconnectAttachedRef.current?.();
return response;
} catch (err) {
// Network throw: same no-onFinish recovery, then rethrow so the SDK
@@ -481,6 +508,31 @@ export default function ChatThread({
);
}
}
// (2b) #430: a LIVE (non-resumed) detached run whose SSE just dropped. The
// server run keeps executing, so instead of a dead "Lost connection" banner
// start a reconnect sequence: pin the CURRENT streaming assistant row as the
// strip/anchor (the live tail is the already-shown partial in `messages`, not
// a persistent row) and re-attach to the live tail via the resumable machinery.
const startedReconnect =
isDisconnect &&
!wasResumed &&
autonomousRunsEnabled === true &&
mountedRef.current &&
message?.role === "assistant" &&
typeof message.id === "string";
if (startedReconnect) {
beginReconnect({
id: message.id,
role: "assistant",
content: "",
status: "streaming",
createdAt: new Date().toISOString(),
// Preserve the partial parts so a 204 restore (onNoActiveStream) re-shows
// what was on screen while the degraded poll catches the run up to
// terminal (rowToUiMessage prefers metadata.parts).
metadata: { parts: message.parts },
});
}
// (3) Standard branches.
// Forward the authoritative server chatId (streamed on the assistant
// message metadata) so the parent adopts the REAL created chat id for a new
@@ -490,9 +542,11 @@ export default function ChatThread({
onTurnFinished(extractServerChatId(message), threadKey);
// Show a neutral "stopped" marker for an aborted turn; the red error banner
// (via `error`) already covers isError, and a clean finish clears any marker.
// On a live disconnect that STARTED a reconnect, suppress the terminal
// "connection lost" notice — the reconnect banner takes over (#430).
if (isError) setStopNotice(null);
else if (isAbort) setStopNotice("manual");
else if (isDisconnect) setStopNotice("disconnect");
else if (isDisconnect) setStopNotice(startedReconnect ? null : "disconnect");
else setStopNotice(null);
// A resumed turn NEVER flushes the queue (invariant 7): skip BOTH the
// flush-on-abort branch and the plain flush. The local streamer is the only
@@ -579,6 +633,106 @@ export default function ChatThread({
const isStreaming = status === "submitted" || status === "streaming";
// #430: live-disconnect reconnect controller. `null` = idle; `{ trying, attempt }`
// = a backoff sequence is running (drives the "reconnecting… (N/max)" banner);
// `{ failed }` = attempts exhausted (drives the manual Retry). Mirrored into a ref
// so the transport/onNoActiveStream closures branch on the LIVE value.
type ReconnectState =
| null
| { phase: "trying"; attempt: number }
| { phase: "failed" };
const [reconnectState, setReconnectState] = useState<ReconnectState>(null);
const reconnectStateRef = useRef<ReconnectState>(null);
const setReconnectStatePair = useCallback((s: ReconnectState) => {
reconnectStateRef.current = s;
setReconnectState(s);
}, []);
const reconnectTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const clearReconnectTimer = useCallback(() => {
if (reconnectTimerRef.current) {
clearTimeout(reconnectTimerRef.current);
reconnectTimerRef.current = null;
}
}, []);
// One reconnect attempt — MIRRORS the mount strip/anchor path for the LIVE case.
// beginReconnect pinned strippedRowRef/stripRef to the run's assistant row, so:
// - remove that row from the store (the mount path strips it from the SEED; here
// it is already shown, so filter it out) — the live replay's `text-start` then
// rebuilds it without DUPLICATING parts (the main dedup risk, #430);
// - reset the one-shot 204 guard so onNoActiveStream can fire for THIS attempt;
// - mark the turn resumed (invariant 7/8) so onFinish runs the recovery block and
// never flushes the queue;
// - resumeStream() -> prepareReconnectToStreamRequest builds
// ?expect=live&anchor=<pinned id>, pinning the replay to OUR run (invariant 6).
const attemptReconnectOnce = useCallback(
(attempt: number) => {
if (!mountedRef.current) return;
const anchor = strippedRowRef.current;
if (anchor) {
setMessages((prev) => prev.filter((m) => m.id !== anchor.id));
}
noStreamHandledRef.current = false;
setResumedTurnPair(true);
setReconnectStatePair({ phase: "trying", attempt });
void resumeStream();
},
[setMessages, setResumedTurnPair, setReconnectStatePair, resumeStream],
);
// Schedule attempt `attempt` after an exponential backoff.
const scheduleReconnectAttempt = useCallback(
(attempt: number) => {
clearReconnectTimer();
setReconnectStatePair({ phase: "trying", attempt });
reconnectTimerRef.current = setTimeout(
() => attemptReconnectOnce(attempt),
RECONNECT_BASE_DELAY_MS * 2 ** (attempt - 1),
);
},
[clearReconnectTimer, setReconnectStatePair, attemptReconnectOnce],
);
// Start a fresh reconnect sequence, pinning `anchorRow` (the live run's assistant
// row) as the strip/anchor reused by every attempt.
const beginReconnect = useCallback(
(anchorRow: IAiChatMessageRow) => {
if (!autonomousRunsEnabled || !mountedRef.current) return;
strippedRowRef.current = anchorRow;
stripRef.current = true;
scheduleReconnectAttempt(1);
},
[autonomousRunsEnabled, scheduleReconnectAttempt],
);
// Manual Retry (shown once attempts are exhausted): restart at attempt 1 and fire
// immediately (the user asked for it now — no backoff).
const retryReconnect = useCallback(() => {
clearReconnectTimer();
attemptReconnectOnce(1);
}, [clearReconnectTimer, attemptReconnectOnce]);
// Live SSE re-attached (the reconnect GET returned a 2xx stream): clear the
// banner + any pending backoff. No-op outside a sequence (e.g. the mount attach).
const onReconnectAttached = useCallback(() => {
if (!mountedRef.current || !reconnectStateRef.current) return;
clearReconnectTimer();
setReconnectStatePair(null);
}, [clearReconnectTimer, setReconnectStatePair]);
onReconnectAttachedRef.current = onReconnectAttached;
// The reconnect GET could not attach (204 / error). onNoActiveStream has already
// armed the degraded poll (the robust fallback that drives the row to terminal
// from the DB), so this only decides the LIVE-attach retry: back off and try
// again up to the cap, else surface the manual Retry.
const onReconnectNoStream = useCallback(() => {
const s = reconnectStateRef.current;
if (s?.phase !== "trying") return;
if (s.attempt < RECONNECT_MAX_ATTEMPTS)
scheduleReconnectAttempt(s.attempt + 1);
else setReconnectStatePair({ phase: "failed" });
}, [scheduleReconnectAttempt, setReconnectStatePair]);
// 204-handler (`onNoActiveStream`): the attach returned 204 — nothing live to
// resume (overflow / begin-failure / after retention / anchor-mismatch). One-
// shot via noStreamHandledRef (we do NOT null onNoActiveStreamRef). Exactly four
@@ -610,7 +764,17 @@ export default function ChatThread({
// (d) 204 means onFinish will NOT fire — clear the suppression flag so it
// cannot swallow the NEXT local turn's queue flush.
setResumedTurnPair(false);
}, [setMessages, queryClient, onResumeFallback, setResumedTurnPair]);
// (e) #430: if this 204/error landed during a live-disconnect reconnect
// sequence, back off and retry the live attach (or give up to the manual
// Retry). The degraded poll armed in (c) is the fallback either way.
onReconnectNoStream();
}, [
setMessages,
queryClient,
onResumeFallback,
setResumedTurnPair,
onReconnectNoStream,
]);
onNoActiveStreamRef.current = onNoActiveStream;
// Mount effect: kick off the resume attempt for a non-settled tail. Marking the
@@ -628,6 +792,9 @@ export default function ChatThread({
return () => {
mountedRef.current = false;
attachAbortRef.current?.abort();
// #430: drop any pending reconnect backoff so it can't fire against the next
// chat this thread's refs are reused for.
if (reconnectTimerRef.current) clearTimeout(reconnectTimerRef.current);
};
// Mount-only by design; the parent remounts per chat via `key`.
// eslint-disable-next-line react-hooks/exhaustive-deps
@@ -666,12 +833,27 @@ export default function ChatThread({
if (tail.status !== "streaming") {
reconcileTailRef.current = false;
onResumeFallback?.(false);
// #430: the run reached its terminal state via the degraded poll — there is
// no live tail left to reconnect to, so drop any reconnect banner / Retry.
clearReconnectTimer();
setReconnectStatePair(null);
}
// onResumeFallback intentionally omitted (parent-stable callback); deps are
// fixed by the resume design.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [initialRows, isStreaming, setMessages]);
// #430: a real stream is live again — the reconnect re-attached to the live tail
// (status -> "streaming") OR the user started a new local turn. Either way clear
// the reconnect banner + any pending backoff. Gated on "streaming" (not the
// broader "submitted") so a still-pending attach GET does not clear prematurely.
useEffect(() => {
if (status === "streaming") {
clearReconnectTimer();
setReconnectStatePair(null);
}
}, [status, clearReconnectTimer, setReconnectStatePair]);
// "Send now" on a queued message: interrupt the current turn and immediately
// send THIS message, keeping the agent's partial output. Other queued messages
// stay queued and flush normally after the new turn. Reuses the existing
@@ -719,6 +901,9 @@ export default function ChatThread({
// observer's Stop would otherwise leave the attach fetch running.
attachAbortRef.current?.abort();
stop();
// #430: pressing Stop also cancels an in-progress reconnect sequence.
clearReconnectTimer();
setReconnectStatePair(null);
if (!autonomousRunsEnabled) return;
if (chatIdRef.current) {
onServerStop?.(chatIdRef.current);
@@ -740,7 +925,13 @@ export default function ChatThread({
// for this fix. Documented so a future change can address the abort-ordering.
stopPendingRef.current = true;
}
}, [stop, autonomousRunsEnabled, onServerStop]);
}, [
stop,
autonomousRunsEnabled,
onServerStop,
clearReconnectTimer,
setReconnectStatePair,
]);
// Clear the stopped marker as soon as a new turn begins streaming, and drop any
// stale "Send now" interrupt flags. On the legit interrupt path both refs are
@@ -825,6 +1016,43 @@ export default function ChatThread({
detail={errorView.detail}
mb="xs"
/>
) : reconnectState ? (
// #430: while auto-reconnecting to a detached run's live tail, show progress
// instead of a dead "Lost connection" banner; once attempts are exhausted,
// offer a manual Retry (the degraded poll keeps catching up underneath).
<Alert
variant="light"
color="gray"
p="xs"
mb="xs"
style={{ flexShrink: 0 }}
>
<Group gap={8} wrap="nowrap" align="center">
{reconnectState.phase === "trying" ? (
<>
<Loader size={14} color="gray" style={{ flex: "none" }} />
<Text size="sm" lh={1.3} c="dimmed">
{t("Connection lost — reconnecting…")}
{` (${reconnectState.attempt}/${RECONNECT_MAX_ATTEMPTS})`}
</Text>
</>
) : (
<>
<Text size="sm" lh={1.3} c="dimmed" style={{ flex: 1 }}>
{t("Couldn't reconnect to the answer.")}
</Text>
<Button
size="compact-xs"
variant="light"
color="gray"
onClick={retryReconnect}
>
{t("Retry")}
</Button>
</>
)}
</Group>
</Alert>
) : stopNotice ? (
<ChatStoppedNotice
text={
@@ -49,19 +49,14 @@ export default function FootnoteDefinitionView(props: NodeViewProps) {
className={classes.definition}
style={{ ["--footnote-number" as any]: `"${number}"` }}
>
{/* #146: contentDOM MUST be the first child — a non-editable marker before
{/* #146: contentDOM MUST be the first child — non-editable chrome before
it makes click hit-testing snap the caret above. Content first; the
marker + back-link follow in DOM and are placed left/right via CSS
flex `order`. The second #146 mitigation lives in
back-link follows in DOM and is placed on the right via CSS flex. The
decorative "N." number is rendered inline via the .definitionContent
::before rule (from the --footnote-number var), so no marker element
precedes the content. The second #146 mitigation lives in
editor-paste-handler.tsx (reflowAfterPaste). */}
<NodeViewContent className={classes.definitionContent} />
<span
className={classes.definitionMarker}
contentEditable={false}
aria-hidden="true"
>
{number}.
</span>
{refCount > 1 ? (
// Multiple references -> ↩ followed by one lettered link per occurrence.
<span
@@ -81,34 +81,34 @@
.definition {
display: flex;
align-items: flex-start;
/* Tight number→text spacing (~one space) so it reads like "1. text"
instead of leaving a wide gap after the period. */
gap: 0.4em;
/* Tight spacing between the content and the trailing ↩ back-link. */
gap: 0.3em;
padding: 2px 0;
/* Footnotes read smaller than body text (16px). Matches .listHeading. */
font-size: var(--mantine-font-size-sm);
}
.definitionMarker {
order: -1; /* keep the "N." marker on the LEFT though it follows content in DOM (#146) */
flex: 0 0 auto;
min-width: 1.5em;
/* Right-align within the narrow column so the period sits next to the text
and multi-digit numbers (10, 11, …) stay aligned on their right edge. */
text-align: right;
font-variant-numeric: tabular-nums;
color: var(--mantine-color-dimmed);
user-select: none;
}
/* The "N." number is decorative (from the --footnote-number CSS var on the
wrapper, never in the document model) and is rendered inline at the start of
the first content line via ::before. This keeps text and wrapped lines flush
to the left margin — no hanging indent — while the editable contentDOM stays
the FIRST DOM child (#146). */
.definitionContent {
flex: 1 1 auto;
min-width: 0;
}
/* The inner editable paragraph inherits `.ProseMirror p { margin: 0.5em 0 }`,
which pushes the first text line ~0.5em below the "N." marker (aligned to
flex-start), making the number float above the text. Drop the outer margins
so the marker and the first line share the same top edge — same approach
used for callouts in core.css. */
.definitionContent > :first-child::before {
content: var(--footnote-number, "?") ". ";
color: var(--mantine-color-dimmed);
font-variant-numeric: tabular-nums;
user-select: none;
}
/* The inner editable paragraph inherits `.ProseMirror p { margin: 0.5em 0 }`.
Drop the outer margins so the definition sits tight to the heading above and
the ::before number aligns with the top of the row — same approach used for
callouts in core.css. */
.definitionContent > :first-child {
margin-top: 0;
}
@@ -1,4 +1,9 @@
import { Hocuspocus } from '@hocuspocus/server';
import {
connectedPayload,
Extension,
Hocuspocus,
onConnectPayload,
} from '@hocuspocus/server';
import { IncomingMessage } from 'http';
import WebSocket from 'ws';
import { AuthenticationExtension } from './extensions/authentication.extension';
@@ -25,6 +30,56 @@ import {
CollaborationHandler,
CollabEventHandlers,
} from './collaboration.handler';
import {
incDocLoad,
incDocUnload,
isMetricsEnabled,
observeCollabConnect,
registerDocsOpenSource,
} from '../integrations/metrics/metrics.registry';
/**
* #402 — collab lifecycle metrics as a lightweight hocuspocus extension.
*
* - afterLoadDocument / afterUnloadDocument (fire once PER DOCUMENT) drive the
* doc load/unload counters.
* - collab_connect_duration_seconds: I time the onConnect→connected hook pair,
* i.e. connection ACCEPTANCE (which includes the auth handshake). This is the
* cleanest per-connection correlation hocuspocus exposes: both payloads carry
* the SAME `request` IncomingMessage object, so a WeakMap keyed on it gives a
* per-connection start with NO leak (the entry is GC'd with the request if a
* connection is rejected in onAuthenticate and `connected` never fires).
* I deliberately do NOT observe at afterLoadDocument: that hook fires per
* DOCUMENT, not per connection, so a second client joining an already-open
* doc would be missed. auth/load latencies are their own separate metrics.
*
* All helpers are no-ops when METRICS_PORT is unset; these hooks are per
* connect/load/unload (never per message), so there is no hot-path cost.
*/
class CollabMetricsExtension implements Extension {
// Keyed by the per-connection request object → connect start time (ms).
private readonly connectStarts = new WeakMap<object, number>();
async onConnect(data: onConnectPayload) {
this.connectStarts.set(data.request, performance.now());
}
async connected(data: connectedPayload) {
const start = this.connectStarts.get(data.request);
if (start !== undefined) {
observeCollabConnect((performance.now() - start) / 1000);
this.connectStarts.delete(data.request);
}
}
async afterLoadDocument() {
incDocLoad();
}
async afterUnloadDocument() {
incDocUnload();
}
}
@Injectable()
export class CollaborationGateway {
@@ -58,9 +113,18 @@ export class CollaborationGateway {
this.authenticationExtension,
this.persistenceExtension,
this.loggerExtension,
// #402 collab lifecycle + connect-duration metrics (no-op when off).
new CollabMetricsExtension(),
],
});
// #402 — read-on-scrape source for collab_docs_open. Wire ONCE, gated, so
// nothing runs when metrics are disabled. The gauge's collect() pulls the
// live count from the hocuspocus instance on each scrape (no inc/dec drift).
if (isMetricsEnabled()) {
registerDocsOpenSource(() => this.hocuspocus.getDocumentsCount());
}
if (this.withRedis) {
this.redisClient = new RedisClient({
host: this.redisConfig.host,
@@ -16,6 +16,7 @@ import { isUserDisabled } from '../../common/helpers';
import { getPageId } from '../collaboration.util';
import { JwtCollabPayload, JwtType } from '../../core/auth/dto/jwt-payload';
import { resolveProvenance } from '../../common/decorators/auth-provenance.decorator';
import { observeCollabAuth } from '../../integrations/metrics/metrics.registry';
@Injectable()
export class AuthenticationExtension implements Extension {
@@ -30,6 +31,18 @@ export class AuthenticationExtension implements Extension {
) {}
async onAuthenticate(data: onAuthenticatePayload) {
// #402 — time the whole auth (verify + user/page/permission lookups) into
// collab_auth_duration_seconds. finally so failed auths are timed too.
// No-op when METRICS_PORT is unset. Behavior unchanged.
const start = performance.now();
try {
return await this.doAuthenticate(data);
} finally {
observeCollabAuth((performance.now() - start) / 1000);
}
}
private async doAuthenticate(data: onAuthenticatePayload) {
const { documentName, token } = data;
const pageId = getPageId(documentName);
@@ -0,0 +1,140 @@
/**
* gitmost #401 — regression test for the connect-vs-unload race in
* @hocuspocus/server 3.4.4 (patched via patches/@hocuspocus__server@3.4.4.patch).
*
* The race (unpatched): when the last client disconnects, storeDocumentHooks'
* `finally` schedules an async `unloadDocument`. That unload runs its
* `beforeUnloadDocument` hooks asynchronously and, meanwhile, records an
* in-flight promise in `this.unloadingDocuments`. In the original 3.4.4
* `createDocument`, a NEW connection arriving in that window falls straight
* through to the `loadingDocuments`/`documents` checks — it never consults
* `unloadingDocuments`. So the new connection can start loading (or reuse) a
* document while the old instance is still being torn down; the re-check inside
* unload (`shouldUnloadDocument`, which sees 0 connections because async auth
* hooks have not registered the new connection yet) then deletes/destroys the
* doc out from under the freshly-connected client → orphaned Document → later
* redis-sync takes the "doc not loaded" path → sync never completes → the
* provider hangs until its ~25s timeout.
*
* The patch: `createDocument` first awaits any in-flight
* `unloadingDocuments.get(name)` before proceeding. Once that settles, the
* decision is deterministic — either the doc was fully unloaded (gone from
* `documents`, so a clean fresh load) or the unload aborted (healthy doc still
* in `documents`, reused). The new connection can never hand-shake onto an
* about-to-be-destroyed Document.
*
* These tests exercise the REAL patched `Hocuspocus.createDocument` (the class
* is directly constructible) by seeding `unloadingDocuments` with a controllable
* in-flight unload and observing that createDocument waits for it.
*/
import { Hocuspocus } from '@hocuspocus/server';
// A promise we can resolve on demand, to model an unload that is mid-flight.
function deferred<T = void>() {
let resolve!: (v: T) => void;
const promise = new Promise<T>((r) => {
resolve = r;
});
return { promise, resolve };
}
describe('gitmost #401 — hocuspocus createDocument awaits in-flight unload', () => {
it('does NOT start loading a new doc until the in-flight unload settles, then loads fresh', async () => {
const hp = new Hocuspocus();
const name = 'page.race';
// Observe loadDocument: on the unpatched code it is invoked synchronously
// within createDocument (before the unload settles); on the patched code it
// must be deferred until unloadingDocuments resolves.
const freshDoc = { name, __fresh: true } as any;
const loadSpy = jest
.spyOn(hp as any, 'loadDocument')
.mockResolvedValue(freshDoc);
// Model an unload in progress: an entry sits in unloadingDocuments and, when
// it completes, it removes the doc from `documents` (a real full unload).
const unload = deferred();
(hp as any).documents.set(name, { name, __dying: true });
(hp as any).unloadingDocuments.set(
name,
unload.promise.then(() => {
(hp as any).documents.delete(name);
}),
);
// Kick off a new connection's createDocument but do not await it yet.
const createPromise = (hp as any).createDocument(
name,
{},
'socket-1',
{ isAuthenticated: true, readOnly: false },
{},
);
// Let all currently-schedulable microtasks run. The patched createDocument is
// now parked on `await unloadingDocuments.get(name)`, so loadDocument must
// NOT have been called yet, and it must NOT have returned the dying doc.
await Promise.resolve();
await Promise.resolve();
expect(loadSpy).not.toHaveBeenCalled();
// The unload completes (doc removed from `documents`).
unload.resolve();
// createDocument now proceeds: sees no existing doc → fresh load.
const doc = await createPromise;
expect(loadSpy).toHaveBeenCalledTimes(1);
expect(doc).toBe(freshDoc);
// The freshly-loaded doc is the one registered — never the dying instance.
expect((hp as any).documents.get(name)).toBe(freshDoc);
});
it('reuses the live doc when the in-flight unload aborts (doc left in documents)', async () => {
const hp = new Hocuspocus();
const name = 'page.abort';
const loadSpy = jest.spyOn(hp as any, 'loadDocument');
// Model an unload that ABORTS (e.g. a new connection reappeared before the
// sync re-check): it settles WITHOUT deleting the doc from `documents`.
const unload = deferred();
const liveDoc = { name, __live: true } as any;
(hp as any).documents.set(name, liveDoc);
(hp as any).unloadingDocuments.set(name, unload.promise); // no-op unload
const createPromise = (hp as any).createDocument(
name,
{},
'socket-2',
{ isAuthenticated: true, readOnly: false },
{},
);
unload.resolve();
const doc = await createPromise;
// The still-present live doc is reused; no fresh load happened.
expect(doc).toBe(liveDoc);
expect(loadSpy).not.toHaveBeenCalled();
});
it('no in-flight unload → behaves normally (fresh load)', async () => {
const hp = new Hocuspocus();
const name = 'page.normal';
const freshDoc = { name } as any;
const loadSpy = jest
.spyOn(hp as any, 'loadDocument')
.mockResolvedValue(freshDoc);
const doc = await (hp as any).createDocument(
name,
{},
'socket-3',
{ isAuthenticated: true, readOnly: false },
{},
);
expect(loadSpy).toHaveBeenCalledTimes(1);
expect(doc).toBe(freshDoc);
});
});
@@ -0,0 +1,130 @@
/**
* gitmost #401 fix 2 — onLoadDocument applies the DB state directly into the
* hook's target document and returns undefined (instead of building a NEW Y.Doc
* and returning it, which made hocuspocus re-encode+apply the whole state a
* SECOND time on every cold load).
*
* These tests assert:
* - the hook mutates `data.document` in place so its content equals the DB doc,
* - onLoadDocument returns undefined (so hocuspocus keeps the mutated doc and
* does NOT run its own applyUpdate(encodeStateAsUpdate(...)) merge),
* - both the raw-ydoc branch and the json→ydoc conversion branch behave so.
*
* Returning undefined is the observable signal that the double-encode is gone
* (the old code returned a new Y.Doc, which made hocuspocus re-encode+apply the
* state a second time); we assert that contract rather than counting internal
* encode calls, which is brittle given the encodes inside toYdoc and the test's
* own `expected` fixtures.
*/
import * as Y from 'yjs';
import { Document } from '@hocuspocus/server';
import { TiptapTransformer } from '@hocuspocus/transformer';
import { PersistenceExtension } from './persistence.extension';
import { tiptapExtensions } from '../collaboration.util';
// A fresh hocuspocus Document (extends Y.Doc, adds isEmpty()) as hocuspocus
// hands to onLoadDocument on a cold load.
const freshDoc = () => new Document(`page.${PAGE_ID}`, {});
const PAGE_ID = '550e8400-e29b-41d4-a716-446655440000';
const doc = (text: string) => ({
type: 'doc',
content: [{ type: 'paragraph', content: [{ type: 'text', text }] }],
});
const jsonOf = (ydoc: Y.Doc) =>
TiptapTransformer.fromYdoc(ydoc, 'default');
describe('PersistenceExtension.onLoadDocument — #401 fix 2 (apply-into-hook-doc)', () => {
let ext: PersistenceExtension;
let pageRepo: { findById: jest.Mock };
beforeEach(() => {
pageRepo = { findById: jest.fn() };
ext = new PersistenceExtension(
pageRepo as any,
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
);
jest.spyOn(ext['logger'], 'debug').mockImplementation(() => undefined);
jest.spyOn(ext['logger'], 'warn').mockImplementation(() => undefined);
});
const load = (document: Document) =>
ext.onLoadDocument({ documentName: `page.${PAGE_ID}`, document } as any);
it('raw ydoc branch: mutates the hook doc to the DB state and returns undefined', async () => {
// Source doc representing the persisted ydoc state.
const source = TiptapTransformer.toYdoc(
doc('DB CONTENT'),
'default',
tiptapExtensions,
);
const dbState = Buffer.from(Y.encodeStateAsUpdate(source));
pageRepo.findById.mockResolvedValue({ id: PAGE_ID, ydoc: dbState });
// The hook target is a fresh empty doc (as hocuspocus supplies on cold load).
const target = freshDoc();
const result = await load(target);
// Return undefined so hocuspocus keeps `target` as-is (no second merge).
expect(result).toBeUndefined();
// The hook document now carries the DB content.
expect(jsonOf(target)).toEqual(jsonOf(source));
});
it('json→ydoc branch: converts page.content into the hook doc and returns undefined', async () => {
pageRepo.findById.mockResolvedValue({
id: PAGE_ID,
ydoc: null,
content: doc('JSON CONTENT'),
});
const target = freshDoc();
const result = await load(target);
// Returning undefined is what keeps hocuspocus from re-encoding+applying the
// state a second time (the old code returned the doc, forcing that extra
// encode). We assert the observable contract here — the return value and the
// resulting content — rather than counting internal encode calls, which is
// brittle: toYdoc and the `expected` build below both encode too.
expect(result).toBeUndefined();
// The converted content landed in the hook document.
const expected = TiptapTransformer.toYdoc(
doc('JSON CONTENT'),
'default',
tiptapExtensions,
);
expect(jsonOf(target)).toEqual(jsonOf(expected));
});
it('live doc already non-empty: early return, no DB read', async () => {
// A hocuspocus Document carrying live content (isEmpty('default') === false).
const target = freshDoc();
const live = TiptapTransformer.toYdoc(
doc('LIVE'),
'default',
tiptapExtensions,
);
Y.applyUpdate(target, Y.encodeStateAsUpdate(live));
const result = await load(target);
expect(result).toBeUndefined();
expect(pageRepo.findById).not.toHaveBeenCalled();
});
it('no persisted state: leaves the fresh empty doc untouched, returns undefined', async () => {
pageRepo.findById.mockResolvedValue({ id: PAGE_ID, ydoc: null, content: null });
const target = freshDoc();
const result = await load(target);
expect(result).toBeUndefined();
expect(target.isEmpty('default')).toBe(true);
});
});
@@ -41,7 +41,10 @@ import {
HISTORY_INTERVAL,
} from '../constants';
import { TransclusionService } from '../../core/page/transclusion/transclusion.service';
import { observeCollabStore } from '../../integrations/metrics/metrics.registry';
import {
observeCollabLoad,
observeCollabStore,
} from '../../integrations/metrics/metrics.registry';
/**
* #251 — wire format of the client→server stateless message that signals a
@@ -150,10 +153,14 @@ export class PersistenceExtension implements Extension {
const { documentName, document } = data;
const pageId = getPageId(documentName);
// #402 — the early return below (live doc already non-empty) does NOT touch
// the DB, so it is deliberately NOT timed. We only observe the real DB-load
// work, and only on each real-load return, tagged by the loaded doc size.
if (!document.isEmpty('default')) {
return;
}
const startedAt = performance.now();
const page = await this.pageRepo.findById(pageId, {
includeContent: true,
includeYdoc: true,
@@ -164,14 +171,21 @@ export class PersistenceExtension implements Extension {
return;
}
// #401 fix 2 — apply the DB state DIRECTLY into the hook's target document
// (`document` === `data.document`) and return undefined. When onLoadDocument
// returns undefined, hocuspocus keeps the mutated hook document as-is; only
// when the hook RETURNS a Y.Doc does hocuspocus re-`applyUpdate(document,
// encodeStateAsUpdate(returned))` — a second full encode+apply of the whole
// (e.g. 315KB) state on every cold load. Mutating in place performs a single
// apply and avoids the throwaway `new Y.Doc()` allocation.
if (page.ydoc) {
this.logger.debug(`ydoc loaded from db: ${pageId}`);
const doc = new Y.Doc();
const dbState = new Uint8Array(page.ydoc);
Y.applyUpdate(doc, dbState);
return doc;
Y.applyUpdate(document, dbState);
observeCollabLoad(dbState.length, (performance.now() - startedAt) / 1000);
return;
}
// if no ydoc state in db convert json in page.content to Ydoc.
@@ -184,26 +198,47 @@ export class PersistenceExtension implements Extension {
tiptapExtensions,
);
Y.encodeStateAsUpdate(ydoc);
return ydoc;
// Encode the converted doc ONCE, reuse the bytes for both the size label
// and the single apply into the hook document (previously this encode's
// result was returned and hocuspocus re-encoded+applied it a second time).
const encoded = Y.encodeStateAsUpdate(ydoc);
Y.applyUpdate(document, encoded);
observeCollabLoad(
encoded.byteLength,
(performance.now() - startedAt) / 1000,
);
return;
}
// No persisted state: the hook document is already a fresh empty Y.Doc, so
// leave it untouched and return undefined (no re-encode of an empty doc).
this.logger.debug(`creating fresh ydoc: ${pageId}`);
return new Y.Doc();
observeCollabLoad(0, (performance.now() - startedAt) / 1000);
return;
}
async onStoreDocument(data: onStoreDocumentPayload) {
// #355 — time the full store (persist + post-store side effects) into
// collab_store_duration_seconds. No-op when METRICS_PORT is unset.
// collab_store_duration_seconds. #402 — also tag by document size bucket.
// No-op when METRICS_PORT is unset.
const startedAt = performance.now();
// Default 0 so a throw before storeDocument returns still records a
// (smallest-bucket) observation rather than dropping the timing entirely.
let bytes = 0;
try {
await this.storeDocument(data);
bytes = await this.storeDocument(data);
} finally {
observeCollabStore((performance.now() - startedAt) / 1000);
observeCollabStore(bytes, (performance.now() - startedAt) / 1000);
}
}
private async storeDocument(data: onStoreDocumentPayload) {
/**
* Persist the document. Returns the serialized ydoc byte size (used as the
* store histogram's size_bucket). The single Y.encodeStateAsUpdate below is
* the ONLY serialization — its byteLength is reused for the label (no second
* encode).
*/
private async storeDocument(data: onStoreDocumentPayload): Promise<number> {
const { documentName, document, context } = data;
const pageId = getPageId(documentName);
@@ -455,6 +490,11 @@ export class PersistenceExtension implements Extension {
await this.enqueuePageHistory(page, lastUpdatedSource);
}
// #402 — report the serialized size for the store histogram's size_bucket.
// ydocState is always computed above (there is no earlier no-write return in
// this method), so this reflects the doc that was serialized this store.
return ydocState.byteLength;
}
/**
@@ -17,10 +17,24 @@ import { Injectable, Logger, OnModuleDestroy } from '@nestjs/common';
/** How long a finished entry is retained for late attach (replay + immediate end). */
export const RUN_STREAM_RETAIN_FINISHED_MS = 30_000;
/** Per-run replay buffer cap. Past this the buffer is dropped (attach -> 204). */
export const RUN_STREAM_MAX_BUFFER_BYTES = 4 * 1024 * 1024;
/**
* Per-run replay buffer cap. Past this the buffer is dropped (attach -> 204, and
* the client falls back to its restore + degraded-poll path, #430).
*
* Raised from 4MB to 32MB (#430): marathon autonomous runs (11-25 min observed)
* stream far more than 4MB of SSE frames, so a live disconnect mid-run would find
* an already-overflowed buffer and could only degrade-poll instead of re-attaching
* to the live tail. 32MB comfortably covers those runs while staying bounded.
*
* Memory cost: this is the WORST-CASE retained size PER ACTIVE run (the buffer is
* freed on finish + retention, or dropped immediately on overflow). With the small
* number of concurrent autonomous runs a single workspace realistically has, 32MB
* each is an acceptable ceiling; the overflow->204->degraded-poll fallback remains
* the backstop for anything larger, so correctness never depends on this bound.
*/
export const RUN_STREAM_MAX_BUFFER_BYTES = 32 * 1024 * 1024;
// 2x the replay cap: a just-written 4MB replay burst alone can never trip the
// 2x the replay cap: a just-written full-replay burst alone can never trip the
// per-subscriber cap (see controller); only a genuinely stalled socket can.
export const SUBSCRIBER_MAX_BUFFERED_BYTES = 2 * RUN_STREAM_MAX_BUFFER_BYTES;
@@ -2,6 +2,7 @@ import {
AiChatStreamRegistryService,
RUN_STREAM_MAX_BUFFER_BYTES,
RUN_STREAM_RETAIN_FINISHED_MS,
SUBSCRIBER_MAX_BUFFERED_BYTES,
RunStreamCallbacks,
} from './ai-chat-stream-registry.service';
@@ -210,9 +211,10 @@ describe('AiChatStreamRegistryService', () => {
const att = (await registry.attach(CHAT, false, undefined, c.cb))!;
att.start();
const oneMb = 'x'.repeat(1024 * 1024);
// 5 x 1MB = 5MB > 4MB cap; the 5th frame is the one that crosses.
for (let i = 0; i < 5; i++) src.push(oneMb + i);
// Cap-relative so it survives a buffer-cap change (#430): a quarter-cap frame
// means 5 frames comfortably exceed the replay cap; the last one crosses.
const chunk = 'x'.repeat(Math.floor(RUN_STREAM_MAX_BUFFER_BYTES / 4));
for (let i = 0; i < 5; i++) src.push(chunk + i);
await flush();
const entry = (registry as any).entries.get(CHAT);
@@ -220,7 +222,7 @@ describe('AiChatStreamRegistryService', () => {
expect(entry.bytes).toBeGreaterThan(RUN_STREAM_MAX_BUFFER_BYTES);
// The live subscriber received ALL 5 frames, including the crossing one.
expect(c.frames).toHaveLength(5);
expect(c.frames[4]).toBe(oneMb + 4);
expect(c.frames[4]).toBe(chunk + 4);
// A NEW attach after overflow gets null (replay buffer is gone).
const c2 = collector();
@@ -240,9 +242,11 @@ describe('AiChatStreamRegistryService', () => {
const attB = (await registry.attach(CHAT, false, undefined, b.cb))!;
attB.start();
const oneMb = 'x'.repeat(1024 * 1024);
// 9 x 1MB = 9MB > 8MB per-subscriber cap; A's pending overflows, B streams live.
for (let i = 0; i < 9; i++) src.push(oneMb + i);
// Cap-relative so it survives a buffer-cap change (#430): a quarter-of-the-
// per-subscriber-cap frame means 5 frames exceed A's paused-pending cap while
// B streams every frame live.
const chunk = 'x'.repeat(Math.floor(SUBSCRIBER_MAX_BUFFERED_BYTES / 4));
for (let i = 0; i < 5; i++) src.push(chunk + i);
await flush();
const entry = (registry as any).entries.get(CHAT);
@@ -250,7 +254,7 @@ describe('AiChatStreamRegistryService', () => {
expect(entry.subscribers.size).toBe(1);
expect(a.frames).toEqual([]); // paused + overflowed: nothing was delivered
// B received every frame live (delivery unaffected by A's overflow).
expect(b.frames).toHaveLength(9);
expect(b.frames).toHaveLength(5);
// A's start() (arriving late) degrades to an immediate end, not a partial replay.
attA.start();
@@ -0,0 +1,295 @@
import { Logger } from '@nestjs/common';
// Mock the AI SDK: the turn we drive is STOPPED during the pre-streamText setup
// phase, so no provider call must ever be made. convertToModelMessages is reached
// (before toolsFor) so it is stubbed to an empty transcript.
jest.mock('ai', () => ({
streamText: jest.fn(),
generateText: jest.fn(),
convertToModelMessages: jest.fn(async () => []),
stepCountIs: jest.fn(() => () => false),
}));
import { streamText } from 'ai';
import { AiChatService } from './ai-chat.service';
/**
* D2 — an explicit Stop DURING the external-MCP toolset build (the pre-streamText
* setup phase) must:
* (a) unwedge the turn (stream() rejects instead of hanging at step 0), and
* (b) finalize the run as 'aborted' via the outer catch's onSettled — never leak
* the run row as 'running' (which would 409 every later turn in this chat).
*
* The setup phase does NOT yet observe streamText's terminal callbacks, so before
* the fix a hung `toolsFor` ignored the run's abort signal and never finalized.
* `raceAgainstAbortAndTimeout(toolsFor, effectiveSignal, ...)` now rejects the
* moment the run's signal aborts; the catch re-throws (signal aborted), and the
* outer catch settles the run 'aborted'.
*/
describe('AiChatService.stream — abort during external-MCP setup finalizes the run (D2)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
function makeService(mcpClients: { toolsFor: jest.Mock }) {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const aiChatMessageRepo = {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
const svc = new AiChatService(
{} as never, // ai
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
aiSettings as never,
tools as never,
mcpClients as never,
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo (openPage undefined -> never touched)
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false } as never, // environment
);
return { svc, tools };
}
const body = {
chatId: 'chat-1',
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
};
// A minimal raw ServerResponse stand-in for the turns that PROCEED past setup
// and reach streamText (the deadline + legacy paths). The setup-only abort test
// never wires the stream, so it keeps using `{ raw: {} }`.
function makeRawRes() {
return {
raw: {
writeHead: jest.fn(function writeHead(this: unknown) {
return this;
}),
write: jest.fn(),
once: jest.fn(),
flushHeaders: jest.fn(),
},
};
}
// A fake streamText result: the service only calls consumeStream() and
// pipeUIMessageStreamToResponse() on it (both fire-and-forget). Its terminal
// callbacks are never invoked, so the run is not finalized through them.
function makeStreamResult() {
return {
consumeStream: jest.fn(),
pipeUIMessageStreamToResponse: jest.fn(),
};
}
beforeEach(() => {
streamTextMock.mockReset();
jest.spyOn(Logger.prototype, 'log').mockImplementation(() => undefined as never);
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined as never);
jest.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined as never);
});
afterEach(() => {
jest.restoreAllMocks();
jest.useRealTimers();
});
it('stops the hung toolset build, rejects, and settles the run "aborted" — never reaching streamText', async () => {
const runController = new AbortController();
// The build hangs (never resolves); the run is STOPPED mid-build. Aborting on a
// macrotask exercises the abort-listener path (a real user Stop during setup).
const toolsFor = jest.fn(() => {
setTimeout(() => runController.abort(new Error('user stop')), 0);
return new Promise(() => {}); // never settles — models a hung MCP build
});
const { svc } = makeService({ toolsFor });
const onSettled = jest.fn();
const begin = jest.fn(async () => ({
runId: 'run-1',
signal: runController.signal,
}));
const promise = svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: { raw: {} } as never,
signal: new AbortController().signal, // socket signal (distinct from the run)
model: {} as never,
role: null,
runHooks: {
begin,
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled,
} as never,
});
// (a) The turn is UNWEDGED: it rejects (with the stop reason) instead of hanging.
await expect(promise).rejects.toThrow('user stop');
// (b) The run is finalized as 'aborted' with NO error message (a Stop, not a
// failure) — so the run row never leaks 'running'.
expect(onSettled).toHaveBeenCalledTimes(1);
expect(onSettled).toHaveBeenCalledWith('run-1', 'aborted', undefined);
// The build was reached, but the provider call was NEVER made (stopped at setup).
expect(toolsFor).toHaveBeenCalledTimes(1);
expect(streamTextMock).not.toHaveBeenCalled();
});
// Item 1 — the onLateResolve leg of raceAgainstAbortAndTimeout. When `toolsFor`
// loses the race (abort) but RESOLVES LATER with a leased toolset, the setup site
// must release that abandoned toolset's leases (call close() on its client
// handles) so their lease refcount is not pinned forever by a toolset nobody
// consumes. Nothing else exercises this path.
it('releases the leases of a toolset that resolves AFTER the race was already lost (onLateResolve)', async () => {
const runController = new AbortController();
// A controllable build: it hangs until we resolve it by hand, and the run is
// stopped mid-build so the race rejects BEFORE the build settles.
let resolveTools: (v: unknown) => void = () => undefined;
const toolsForPromise = new Promise((resolve) => {
resolveTools = resolve;
});
const toolsFor = jest.fn(() => {
setTimeout(() => runController.abort(new Error('user stop')), 0);
return toolsForPromise;
});
const { svc } = makeService({ toolsFor });
const begin = jest.fn(async () => ({
runId: 'run-1',
signal: runController.signal,
}));
const promise = svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: { raw: {} } as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: {
begin,
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled: jest.fn(),
} as never,
});
// The race is lost to the abort: the turn rejects with the stop reason.
await expect(promise).rejects.toThrow('user stop');
// NOW the abandoned build resolves late with a leased client. onLateResolve must
// release it (call close on the lease handle).
const close = jest.fn().mockResolvedValue(undefined);
resolveTools({
tools: {},
clients: [{ close }],
outcomes: [],
instructions: [],
});
// Flush the microtasks so work.then -> onLateResolve -> Promise.all(close) runs.
await new Promise((r) => setImmediate(r));
expect(close).toHaveBeenCalledTimes(1);
});
// Item 2 — the PURE DEADLINE branch (MCP_TOOLSET_BUILD_DEADLINE_MS). `toolsFor`
// never settles and the run's signal is NOT aborted: the race rejects with a
// "setup timed out" error, the catch does NOT re-throw (runId set but signal not
// aborted), and the turn PROCEEDS Docmost-only. It must reach streamText (the turn
// continues, not wedged) and must NOT be finalized 'aborted'.
it('proceeds Docmost-only (reaches streamText) when the build hits the deadline without an abort', async () => {
jest.useFakeTimers();
// The build hangs forever; the run's signal is never aborted.
const toolsFor = jest.fn(() => new Promise(() => {}));
const { svc } = makeService({ toolsFor });
streamTextMock.mockReturnValue(makeStreamResult() as never);
const onSettled = jest.fn();
const runSignal = new AbortController().signal; // never aborts
const begin = jest.fn(async () => ({ runId: 'run-1', signal: runSignal }));
const promise = svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: makeRawRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: {
begin,
onAssistantSeeded: jest.fn(),
onStep: jest.fn(),
onSettled,
} as never,
});
// Advance past the 60s build deadline; advanceTimersByTimeAsync flushes the
// promise microtasks between timer fires so the whole setup chain runs.
await jest.advanceTimersByTimeAsync(60_001);
// The turn does not throw out of setup — it continues to stream.
await expect(promise).resolves.toBeUndefined();
// The turn CONTINUED: streamText was reached (Docmost-only), not wedged.
expect(toolsFor).toHaveBeenCalledTimes(1);
expect(streamTextMock).toHaveBeenCalledTimes(1);
// The run was NOT finalized as aborted (the deadline is not a Stop) — the setup
// catch settle path never ran, so onSettled is left to streamText's callbacks.
expect(onSettled).not.toHaveBeenCalled();
});
// Item 3 — the LEGACY no-runId path. The catch's re-throw is gated on
// `runId && effectiveSignal.aborted`. With NO runId (no runHooks) an abort during
// setup must NOT re-throw (runId falsy) — the turn warns + proceeds Docmost-only
// and streams, and is never finalized 'aborted' via the re-throw. Locks the
// `runId &&` half of the guard.
it('does NOT re-throw on a setup abort when there is no runId (legacy path proceeds Docmost-only)', async () => {
const socketController = new AbortController();
// The build hangs; the SOCKET signal (legacy effectiveSignal) aborts mid-build.
const toolsFor = jest.fn(() => {
setTimeout(() => socketController.abort(new Error('socket closed')), 0);
return new Promise(() => {});
});
const { svc } = makeService({ toolsFor });
streamTextMock.mockReturnValue(makeStreamResult() as never);
// No runHooks => runId undefined, effectiveSignal === the socket signal.
const promise = svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: body as never,
res: makeRawRes() as never,
signal: socketController.signal,
model: {} as never,
role: null,
});
// The turn does NOT reject out of setup (no re-throw on the legacy path).
await expect(promise).resolves.toBeUndefined();
// It proceeded Docmost-only and reached streamText — streamText then observes
// the already-aborted socket signal via its own abortSignal.
expect(toolsFor).toHaveBeenCalledTimes(1);
expect(streamTextMock).toHaveBeenCalledTimes(1);
});
});
@@ -16,6 +16,7 @@ import {
rowToUiMessage,
prepareAgentStep,
flushAssistant,
stripNulChars,
chatStreamMetadata,
accumulateStepUsage,
isInterruptResume,
@@ -28,11 +29,13 @@ import { buildSystemPrompt } from './ai-chat.prompt';
import type { McpClientsService } from './external-mcp/mcp-clients.service';
/**
* Unit tests for compactToolOutput: the pure helper that shrinks LARGE tool
* outputs before they are persisted (and re-sent to the provider on later
* turns). The contract is: small outputs pass through unchanged (by identity);
* large outputs keep their shape and small scalar fields (id/title/pageId — the
* client reads these to render citations) while big payloads are truncated.
* Unit tests for compactToolOutput: the pure helper that shrinks tool outputs
* before they are persisted (and re-sent to the provider on later turns). The
* contract is: small and normal outputs — including whole page reads (tens of
* KB) — pass through unchanged (by identity); only an output above the high
* safety cap (> 200 KB) is compacted, and even then it keeps its shape and
* small scalar fields (id/title/pageId — the client reads these to render
* citations) while the big payloads are reduced.
*/
describe('compactToolOutput', () => {
it('returns a small object unchanged (by identity)', () => {
@@ -41,7 +44,7 @@ describe('compactToolOutput', () => {
});
it('truncates a large getPage-shaped markdown body but keeps the title', () => {
const big = 'x'.repeat(20000);
const big = 'x'.repeat(300000);
const result = compactToolOutput({ title: 'T', markdown: big }) as {
title: string;
markdown: string;
@@ -49,15 +52,16 @@ describe('compactToolOutput', () => {
// Shallow scalar field is preserved (citations depend on it).
expect(result.title).toBe('T');
// The big payload is shrunk far below the original size.
expect(result.markdown.length).toBeLessThan(20000);
expect(result.markdown).toContain('[truncated');
expect(result.markdown.length).toBeLessThan(300000);
expect(result.markdown).toContain('omitted from stored chat history');
});
it('caps a long array and appends a single truncation marker', () => {
// 200 small objects, each padded so the total serialized size > 4000 bytes.
// 200 objects, each padded so the total serialized size
// > 200000 bytes (the new safety cap).
const long = Array.from({ length: 200 }, (_, i) => ({
id: 'n' + i,
pad: 'y'.repeat(40),
pad: 'y'.repeat(1200),
}));
const result = compactToolOutput(long) as Array<Record<string, unknown>>;
// 50 kept + 1 marker.
@@ -75,8 +79,8 @@ describe('compactToolOutput', () => {
it('replaces a subtree beyond the depth cap with a marker', () => {
// Build a deeply nested object (> TOOL_OUTPUT_MAX_DEPTH levels) with a big
// string at the bottom so the total serialized size exceeds the threshold.
let nested: Record<string, unknown> = { leaf: 'z'.repeat(8000) };
// string at the bottom so the total serialized size exceeds the 200 KB cap.
let nested: Record<string, unknown> = { leaf: 'z'.repeat(250000) };
for (let i = 0; i < 20; i++) {
nested = { child: nested };
}
@@ -85,7 +89,7 @@ describe('compactToolOutput', () => {
});
it('produces a much smaller JSON than the original for a large input', () => {
const big = 'x'.repeat(20000);
const big = 'x'.repeat(300000);
const original = { title: 'T', markdown: big };
const result = compactToolOutput(original);
const originalBytes = Buffer.byteLength(JSON.stringify(original), 'utf8');
@@ -144,6 +148,53 @@ describe('assistantParts', () => {
expect(toolPart).not.toHaveProperty('output');
});
it('replays the REAL error text for a THROWN tool (tool-error part)', () => {
const steps = [
{
text: '',
toolCalls: [
{ toolCallId: 'c7', toolName: 'editPageText', input: { id: 'p1' } },
],
// A thrown tool is a `tool-error` content part; toolResults holds only
// successes and stays empty for this call.
toolResults: [],
content: [
{
type: 'tool-error',
toolCallId: 'c7',
toolName: 'editPageText',
input: { id: 'p1' },
error: new Error('page is locked'),
},
],
},
];
const parts = assistantParts(steps, '') as AnyPart[];
const toolPart = parts.find((p) => p.type === 'tool-editPageText');
expect(toolPart).toBeDefined();
expect(toolPart!.state).toBe('output-error');
// The REAL error is replayed, NOT the 'Tool call did not complete.' placeholder.
expect(toolPart!.errorText).toBe('page is locked');
expect(toolPart).not.toHaveProperty('output');
});
it('keeps the placeholder ONLY for a call with neither result nor tool-error', () => {
const steps = [
{
text: '',
toolCalls: [
{ toolCallId: 'c8', toolName: 'insertNode', input: { node: {} } },
],
toolResults: [],
content: [], // aborted mid-step: no result AND no tool-error
},
];
const parts = assistantParts(steps, '') as AnyPart[];
const toolPart = parts.find((p) => p.type === 'tool-insertNode');
expect(toolPart!.state).toBe('output-error');
expect(toolPart!.errorText).toBe('Tool call did not complete.');
});
it('skips malformed tool-calls (missing toolName or toolCallId)', () => {
const steps = [
{
@@ -191,6 +242,45 @@ describe('serializeSteps', () => {
expect(trace[0]).toEqual({ toolName: 'getPage', input: { id: 'p1' } });
expect(trace[1]).toEqual({ toolName: 'getPage', output: { title: 'T' } });
});
it('records a THROWN tool failure (tool-error part) with its error message', () => {
const trace = serializeSteps([
{
toolCalls: [{ toolName: 'editPageText', input: { id: 'p1' } }],
toolResults: [],
content: [
{
type: 'tool-error',
toolName: 'editPageText',
error: new Error('page is locked'),
},
],
},
]) as Array<Record<string, unknown>>;
// The call element is followed by a paired error element (mirroring how a
// successful result is appended), so the failure survives in the trace.
expect(trace).toHaveLength(2);
expect(trace[0]).toEqual({ toolName: 'editPageText', input: { id: 'p1' } });
expect(trace[1]).toEqual({
toolName: 'editPageText',
error: 'page is locked',
});
});
it('truncates a very long tool-error message to the tool-output limit', () => {
const long = 'x'.repeat(5000);
const trace = serializeSteps([
{
toolCalls: [{ toolName: 'editPageText', input: {} }],
toolResults: [],
content: [{ type: 'tool-error', toolName: 'editPageText', error: long }],
},
]) as Array<Record<string, unknown>>;
const errorText = trace[1].error as string;
// Truncated (not the full 5000 chars) and carries the omission marker.
expect(errorText.length).toBeLessThan(long.length);
expect(errorText).toContain('chars omitted');
});
});
describe('rowToUiMessage', () => {
@@ -448,6 +538,45 @@ describe('flushAssistant', () => {
});
});
/**
* stripNulChars: a NUL (U+0000) is rejected by Postgres in BOTH the `content`
* (text) and `toolCalls`/`metadata` (jsonb) columns, so it must be stripped from
* every persisted string. String.fromCharCode(0) avoids embedding a raw NUL byte
* in this source file.
*/
describe('stripNulChars', () => {
const NUL = String.fromCharCode(0);
it('deep-strips NUL from strings in nested objects/arrays', () => {
const out = stripNulChars({
content: `a${NUL}b`,
parts: [{ type: 'text', text: `x${NUL}${NUL}y` }],
nested: [`p${NUL}q`, 42, null],
});
expect(out.content).toBe('ab');
expect((out.parts[0] as { text: string }).text).toBe('xy');
expect(out.nested[0]).toBe('pq');
expect(out.nested[1]).toBe(42);
expect(out.nested[2]).toBeNull();
expect(JSON.stringify(out).includes(NUL)).toBe(false);
});
it('returns the SAME reference when there is no NUL (no needless clone)', () => {
const input = { a: 'clean', b: [1, 2, { c: 'ok' }] };
expect(stripNulChars(input)).toBe(input);
});
it('flushAssistant produces a NUL-free row even when the turn text carries one', () => {
const f = flushAssistant([], `partial${NUL}answer`, 'error', {
error: `bo${NUL}om`,
});
expect(f.content).toBe('partialanswer');
const serialized =
f.content + JSON.stringify(f.toolCalls) + JSON.stringify(f.metadata);
expect(serialized.includes(NUL)).toBe(false);
});
});
/**
* chatStreamMetadata: attach metadata to the streamed assistant UI message per
* part type — `chatId` on `start` (so the client adopts the real created chat id
+273 -24
View File
@@ -58,6 +58,19 @@ import {
// multi-search research questions are not cut off mid-investigation.
const MAX_AGENT_STEPS = 20;
// Wall-clock ceiling for building the external MCP toolset during the per-turn
// setup phase (before streamText owns the lifecycle). Defense-in-depth ABOVE the
// per-server connect bound in mcp-clients.service (CONNECT_TIMEOUT_MS): even if
// that per-server timeout regressed, this outer deadline — together with the run's
// abort signal — guarantees the setup phase can never wedge a turn at step 0 (the
// production hang) and the run always finalizes. It stays a TRUE backstop because
// buildEntry connects to the servers CONCURRENTLY, so the total build time is
// bounded by the SLOWEST single server (~2×CONNECT_TIMEOUT_MS), not the SUM across
// them — the per-server bound fires first no matter how many servers are enabled,
// and this outer deadline only catches a total build stall the per-server bound
// somehow missed.
const MCP_TOOLSET_BUILD_DEADLINE_MS = 60_000;
// System-prompt addendum injected ONLY on the final step (see prepareAgentStep).
// It forbids further tool calls and tells the model to synthesize the best
// answer it can from what it already gathered, so a tool-heavy turn never ends
@@ -176,6 +189,78 @@ export function sameInstant(
return ta === tb;
}
/**
* Race `work` against an abort signal AND a wall-clock deadline, so a hung
* external-MCP toolset build during the pre-streamText setup phase can NEITHER
* wedge the turn NOR make it un-stoppable, and the run always finalizes. It
* - resolves with `work`'s value when it settles first;
* - REJECTS EARLY if `signal` aborts (with `signal.reason` when that is an Error,
* else a generic `Error('aborted')`) — so an explicit Stop is honored mid-setup;
* - REJECTS EARLY if `deadlineMs` elapses (defense-in-depth backstop);
* - invokes `onLateResolve(value)` when `work` settles AFTER the race was already
* lost, so the caller can release any resources that abandoned value owns
* (e.g. close leased MCP clients that would otherwise leak their sockets).
*
* A rejection handler is attached to `work` so a late rejection is never an
* unhandledRejection; the timer is unref'd and cleared once the race settles.
*/
export function raceAgainstAbortAndTimeout<T>(
work: Promise<T>,
signal: AbortSignal,
deadlineMs: number,
onLateResolve?: (value: T) => void,
): Promise<T> {
return new Promise<T>((resolve, reject) => {
let settled = false;
const cleanup = () => {
clearTimeout(timer);
signal.removeEventListener('abort', onAbort);
};
const onAbort = () => {
if (settled) return;
settled = true;
cleanup();
reject(
signal.reason instanceof Error ? signal.reason : new Error('aborted'),
);
};
const timer = setTimeout(() => {
if (settled) return;
settled = true;
cleanup();
reject(new Error(`setup timed out after ${deadlineMs}ms`));
}, deadlineMs);
// Do not keep the process alive just for this setup-deadline timer.
timer.unref?.();
if (signal.aborted) {
onAbort();
} else {
signal.addEventListener('abort', onAbort, { once: true });
}
work.then(
(value) => {
if (settled) {
// The race was already lost (abort/deadline): hand the abandoned value to
// the caller so it can release the resources that value owns.
onLateResolve?.(value);
return;
}
settled = true;
cleanup();
resolve(value);
},
(err: unknown) => {
// A late rejection after the race is already handled — swallow so it is
// never an unhandledRejection.
if (settled) return;
settled = true;
cleanup();
reject(err instanceof Error ? err : new Error(String(err)));
},
);
});
}
/**
* Payload accepted from the client `useChat` POST body. We do NOT bind a strict
* DTO (the global ValidationPipe whitelist would strip the useChat-specific
@@ -704,10 +789,43 @@ export class AiChatService implements OnModuleInit {
instructions: [],
};
try {
external = await this.mcpClients.toolsFor(workspace.id);
// Bound the external-MCP toolset build by BOTH the run's abort signal and
// a generous wall-clock deadline. This is the pre-streamText setup phase,
// which streamText's terminal callbacks do NOT yet govern — so without this
// a hung build would hang the turn at step 0 forever (the production hang),
// unobservant of an explicit Stop. The deadline is defense-in-depth ABOVE
// the per-server connect bound in mcp-clients.service. On a LATE resolve
// (the race was already lost) RELEASE the abandoned toolset's leases —
// c.close() here is the lease handle, so it decrements the cache entry's
// refcount; it does NOT force-close the transports (the cache OWNS the
// clients and closes them on TTL/evict). This just prevents the lease
// refcount from being pinned >=1 forever by a toolset nobody will consume.
external = await raceAgainstAbortAndTimeout(
this.mcpClients.toolsFor(workspace.id),
effectiveSignal,
MCP_TOOLSET_BUILD_DEADLINE_MS,
(late) => {
void Promise.all(
late.clients.map((c) => c.close().catch(() => undefined)),
);
},
);
} catch (err) {
// Building the external toolset must never break the turn; proceed with
// Docmost-only tools. Never log URLs/headers — short message only.
// An explicit Stop reached the RUN's signal DURING setup: re-throw so the
// outer catch finalizes the run as aborted — never swallow a Stop. Gated on
// `runId`: the re-throw exists ONLY to finalize the run, which exists only
// in autonomous mode. On the legacy path (no runId) `effectiveSignal` is the
// SOCKET signal (it aborts on a client disconnect); re-throwing there would
// change prior behavior and make the controller write JSON to an already-
// closed socket (it only attaches res.raw.on('error') in autonomous mode).
// So legacy keeps its prior behavior — warn + proceed, and streamText then
// observes the aborted socket signal.
if (runId && effectiveSignal.aborted) {
throw err;
}
// Otherwise a down/slow server (build timeout or other fault) must never
// break the turn: proceed with Docmost-only tools. Never log URLs/headers —
// short message only.
this.logger.warn(
`External MCP toolset unavailable: ${
err instanceof Error ? err.message : 'unknown error'
@@ -1307,12 +1425,19 @@ export class AiChatService implements OnModuleInit {
if (this.environment?.isAiChatResumableStreamEnabled?.()) {
this.streamRegistry?.abortEntry(chatId, runId);
}
// Distinguish an explicit Stop (the run's signal aborted during setup) from
// a real failure, so the run settles with the correct terminal status
// instead of always 'error'. onSettled/finalizeRun is idempotent, so this
// is safe even if a streamText callback also settles the run.
const settleStatus = effectiveSignal.aborted ? 'aborted' : 'error';
await runHooks?.onSettled?.(
runId,
'error',
err instanceof Error
? err.message
: 'Agent run failed before streaming started',
settleStatus,
settleStatus === 'aborted'
? undefined
: err instanceof Error
? err.message
: 'Agent run failed before streaming started',
);
}
throw err;
@@ -1512,21 +1637,41 @@ type StepLike = {
toolName?: string;
output?: unknown;
}>;
// ai@6.0.134: a tool that THREW surfaces as a `tool-error` content part
// ({ type:'tool-error', toolCallId, toolName, input, error }), NOT as a
// `toolResults` entry (which holds only successes). Read from here so failed
// calls are persisted with their real error instead of being dropped.
content?: ReadonlyArray<{
type?: string;
toolCallId?: string;
toolName?: string;
input?: unknown;
error?: unknown;
}>;
};
/**
* Compaction tunables for persisted tool OUTPUTS. Read tools (getPage,
* getPageJson, getNode, diffPageVersions, exportPageMarkdown, ...) return whole
* pages with no size cap. Their outputs are stored in `metadata.parts` and
* RE-SENT to the provider on every later turn via convertToModelMessages, so an
* uncompacted large body grows token cost, latency, and DB row size on every
* turn. We shrink the big payloads while preserving the object's shape and its
* small scalar fields (id/title/pageId) the client reads to render citations.
* pages. Their outputs are stored in `metadata.parts` and RE-SENT to the
* provider on every later turn via convertToModelMessages. We deliberately keep
* these outputs FULL up to a high safety cap (MAX_TOOL_OUTPUT_BYTES) so the
* model never sees a shortened copy of content it already fetched: an earlier
* 4000-byte cap shrank normal page reads (often tens of KB) to a tiny preview,
* and the model — seeing a truncation marker in its OWN history — re-read the
* same page, wasting tokens. Only a single output LARGER than the cap is
* compacted at all, purely as a backstop against a pathological payload; even
* then we preserve the object's shape and its small scalar fields
* (id/title/pageId) that the client reads to render citations.
*/
// Only outputs whose JSON serialization exceeds this are compacted at all
// (fast path: smaller outputs are returned unchanged, by identity).
const MAX_TOOL_OUTPUT_BYTES = 4000;
// A string longer than this is truncated to a leading preview.
// HIGH safety backstop: only an output whose JSON serialization EXCEEDS this is
// compacted at all. Normal reads (whole pages, tens of KB) stay well under it
// and are stored + replayed VERBATIM (fast path: returned unchanged, by
// identity). Only a single pathologically huge output (> 200 KB) is compacted.
const MAX_TOOL_OUTPUT_BYTES = 200_000;
// Inside the backstop path only (i.e. once the whole output already exceeded
// MAX_TOOL_OUTPUT_BYTES), a string longer than this is reduced to a leading
// preview; normal outputs never reach this branch.
const TOOL_OUTPUT_STRING_LIMIT = 600;
// Number of leading characters kept from a truncated string.
const TOOL_OUTPUT_STRING_PREVIEW = 500;
@@ -1569,9 +1714,9 @@ export function compactToolOutput(output: unknown): unknown {
function compactValue(value: unknown, depth: number): unknown {
if (typeof value === 'string') {
if (value.length > TOOL_OUTPUT_STRING_LIMIT) {
return `${value.slice(0, TOOL_OUTPUT_STRING_PREVIEW)}…[truncated ${
return `${value.slice(0, TOOL_OUTPUT_STRING_PREVIEW)}…[${
value.length - TOOL_OUTPUT_STRING_PREVIEW
} chars]`;
} chars omitted from stored chat history to bound replay size — call the tool again to read the full output]`;
}
return value;
}
@@ -1605,6 +1750,26 @@ function compactValue(value: unknown, depth: number): unknown {
return value;
}
/**
* Extract a bounded string message from a `tool-error` part's `error` field for
* persistence and history replay. The field may be an `Error`, a string, or an
* arbitrary object, so pull a message robustly. The result is passed through
* `compactValue` so a very long error honors the SAME truncation limits the file
* already applies to tool outputs (no new limit is introduced here).
*/
function normalizeToolError(error: unknown): string {
const message =
error instanceof Error
? error.message
: typeof error === 'string'
? error
: error != null &&
typeof (error as { message?: unknown }).message === 'string'
? (error as { message: string }).message
: String(error);
return compactValue(message, 0) as string;
}
/**
* Rebuild the FULL UIMessage `parts` for an assistant turn from the SDK steps,
* so multi-turn history replays prior tool-calls/results to the model (not just
@@ -1637,6 +1802,14 @@ export function assistantParts(
for (const r of step.toolResults ?? []) {
if (r.toolCallId) resultsById.set(r.toolCallId, r.output);
}
// Index this step's THROWN tool failures (ai@6 `tool-error` content parts)
// by tool call id, so a call that failed replays with its real error text.
const errorsById = new Map<string, unknown>();
for (const part of step.content ?? []) {
if (part.type === 'tool-error' && part.toolCallId) {
errorsById.set(part.toolCallId, part.error);
}
}
for (const call of step.toolCalls ?? []) {
if (!call.toolName || !call.toolCallId) continue;
const hasResult = resultsById.has(call.toolCallId);
@@ -1649,9 +1822,21 @@ export function assistantParts(
input: call.input,
output: compactToolOutput(resultsById.get(call.toolCallId)),
});
} else if (errorsById.has(call.toolCallId)) {
// The tool THREW: replay the REAL error so the model on the next turn
// knows WHY the call failed (and does not blindly repeat it). An
// output-error round-trips through convertToModelMessages as a balanced
// tool-call + tool-result, keeping the rebuilt history valid.
parts.push({
type: `tool-${call.toolName}`,
toolCallId: call.toolCallId,
state: 'output-error',
input: call.input,
errorText: normalizeToolError(errorsById.get(call.toolCallId)),
});
} else {
// No paired result (e.g. aborted mid-step). Persisting a bare
// tool-call (input-available) would replay as an unpaired call and
// No paired result AND no tool-error (e.g. aborted mid-step). Persisting
// a bare tool-call (input-available) would replay as an unpaired call and
// throw MissingToolResultsError on the next turn (convertToModelMessages
// emits no tool-result for it). Emit a SYNTHETIC paired result instead:
// an output-error round-trips through convertToModelMessages as a
@@ -1758,6 +1943,45 @@ export async function applyFinalize(
});
}
/**
* Deep-strip NUL characters (`\u0000`) from every string in a value, returning
* the SAME reference when nothing changed (so the no-NUL common case allocates
* nothing). Postgres rejects a NUL in BOTH `text` and `jsonb` columns ("invalid
* input syntax for type json" / "unsupported Unicode escape sequence"), so a
* stray NUL in model output or a tool result — e.g. a truncated multibyte read
* of a web page — otherwise fails EVERY persist of the assistant row, silently
* dropping that turn's content from the DB while the live stream still shows it.
* Applied at the flushAssistant choke point so content + toolCalls + metadata are
* all covered. Exported for the unit test.
*/
export function stripNulChars<T>(value: T): T {
if (typeof value === 'string') {
return (value.includes('\u0000')
? value.replace(/\u0000/g, '')
: value) as T;
}
if (Array.isArray(value)) {
let changed = false;
const out = value.map((v) => {
const s = stripNulChars(v);
if (s !== v) changed = true;
return s;
});
return (changed ? out : value) as T;
}
if (value && typeof value === 'object') {
let changed = false;
const out: Record<string, unknown> = {};
for (const [k, v] of Object.entries(value as Record<string, unknown>)) {
const s = stripNulChars(v);
if (s !== v) changed = true;
out[k] = s;
}
return (changed ? out : value) as T;
}
return value;
}
/**
* PURE assistant-row builder (#183 step-granular durability). Given the turn's
* accumulated steps + the in-progress (not-yet-finished) text + the lifecycle
@@ -1827,12 +2051,16 @@ export function flushAssistant(
};
}
return {
// Strip NUL chars from the whole row before persisting: Postgres rejects a NUL
// in both the `content` (text) and `toolCalls`/`metadata` (jsonb) columns, and a
// single stray NUL in model/tool output would otherwise fail EVERY write of this
// row and silently drop the turn's content from the DB (see stripNulChars).
return stripNulChars({
content: stepsText + trailing,
toolCalls: serializeSteps(finished),
metadata,
status,
};
});
}
/**
@@ -1844,10 +2072,19 @@ export function serializeSteps(
steps: ReadonlyArray<{
toolCalls?: ReadonlyArray<{ toolName?: string; input?: unknown }>;
toolResults?: ReadonlyArray<{ toolName?: string; output?: unknown }>;
content?: ReadonlyArray<{
type?: string;
toolName?: string;
error?: unknown;
}>;
}>,
): unknown {
const calls: Array<{ toolName?: string; input?: unknown; output?: unknown }> =
[];
const calls: Array<{
toolName?: string;
input?: unknown;
output?: unknown;
error?: string;
}> = [];
for (const step of steps ?? []) {
for (const call of step.toolCalls ?? []) {
calls.push({ toolName: call.toolName, input: call.input });
@@ -1855,6 +2092,18 @@ export function serializeSteps(
for (const r of step.toolResults ?? []) {
calls.push({ toolName: r.toolName, output: compactToolOutput(r.output) });
}
// ai@6 surfaces a THROWN tool failure as a `tool-error` content part, NOT as
// a `toolResults` entry. Record it as its own paired element (mirroring how a
// successful result is appended) so the failure and its reason survive in the
// trace instead of leaving an orphaned call with no result.
for (const part of step.content ?? []) {
if (part.type === 'tool-error') {
calls.push({
toolName: part.toolName,
error: normalizeToolError(part.error),
});
}
}
}
return calls.length > 0 ? calls : null;
}
@@ -181,25 +181,25 @@ describe('mcp timeout env helpers', () => {
else process.env.AI_MCP_CALL_TIMEOUT_MS = ORIG_CALL;
});
it('mcpStreamTimeoutMs defaults to 5 min and honors a positive override', () => {
it('mcpStreamTimeoutMs defaults to 1 min and honors a positive override', () => {
delete process.env.AI_MCP_STREAM_TIMEOUT_MS;
expect(mcpStreamTimeoutMs()).toBe(300_000);
process.env.AI_MCP_STREAM_TIMEOUT_MS = '60000';
expect(mcpStreamTimeoutMs()).toBe(60_000);
process.env.AI_MCP_STREAM_TIMEOUT_MS = '90000';
expect(mcpStreamTimeoutMs()).toBe(90_000);
for (const bad of ['0', '-1', 'x', '']) {
process.env.AI_MCP_STREAM_TIMEOUT_MS = bad;
expect(mcpStreamTimeoutMs()).toBe(300_000);
expect(mcpStreamTimeoutMs()).toBe(60_000);
}
});
it('mcpCallTimeoutMs defaults to 15 min and honors a positive override', () => {
it('mcpCallTimeoutMs defaults to 2 min and honors a positive override', () => {
delete process.env.AI_MCP_CALL_TIMEOUT_MS;
expect(mcpCallTimeoutMs()).toBe(900_000);
process.env.AI_MCP_CALL_TIMEOUT_MS = '120000';
expect(mcpCallTimeoutMs()).toBe(120_000);
process.env.AI_MCP_CALL_TIMEOUT_MS = '180000';
expect(mcpCallTimeoutMs()).toBe(180_000);
for (const bad of ['0', '-1', 'x', '']) {
process.env.AI_MCP_CALL_TIMEOUT_MS = bad;
expect(mcpCallTimeoutMs()).toBe(900_000);
expect(mcpCallTimeoutMs()).toBe(120_000);
}
});
});
@@ -0,0 +1,237 @@
import { McpClientsService } from './mcp-clients.service';
/**
* D1 a HUNG MCP handshake must not POISON the per-workspace build cache.
*
* THE BUG (production hang): `createMCPClient` (inside the private `connect`) is
* NOT bounded by a timeout and like @ai-sdk/mcp's tool calls its promise does
* NOT settle on abort. A transient network blip mid-handshake made connect hang
* FOREVER. Because getOrBuildEntry caches the build PROMISE, that never-settling
* connect wedged EVERY later turn for the workspace (each awaited the same pending
* build) step_count stuck at 0, run row leaking 'running', chat 409ing forever.
*
* THE FIX: `connectWithTimeout` races `connect` against a SETTLING timeout
* (CONNECT_TIMEOUT_MS). On timeout it REJECTS, so buildEntry catches it, records
* the server `ok:false`, and the build COMPLETES with that server skipped the
* cache is never poisoned and a subsequent `toolsFor` returns instead of hanging.
*
* REACHABILITY NOTE: the smallest network-free path that exercises the fix is to
* spy on the private `connect` (the same harness the namespacing spec uses)
* `connectWithTimeout` wraps exactly that call, so a never-resolving `connect`
* models a never-settling `createMCPClient` precisely, without DNS/sockets.
*
* Fake timers prove the timeout fires WITHOUT real waiting.
*/
// Mirrors the private CONNECT_TIMEOUT_MS constant in mcp-clients.service.ts.
const CONNECT_TIMEOUT_MS = 5000;
interface FakeServer {
id: string;
name: string;
transport: string;
url: string;
headersEnc: string | null;
toolAllowlist: string[] | null;
instructions?: string | null;
}
function server(
over: Partial<FakeServer> & { id: string; name: string },
): FakeServer {
return {
transport: 'http',
url: 'https://example.com/mcp',
headersEnc: null,
toolAllowlist: null,
...over,
};
}
function buildService(servers: FakeServer[]) {
const repoStub = { listEnabled: jest.fn().mockResolvedValue(servers) };
const service = new McpClientsService(repoStub as never, {} as never);
// Silence the expected "server unavailable" warning.
jest
.spyOn(
(service as unknown as { logger: { warn: (...a: unknown[]) => void } })
.logger,
'warn',
)
.mockImplementation(() => undefined);
return service;
}
// Spy on the private `connect` with a per-server implementation.
function stubConnect(
service: McpClientsService,
impl: (s: FakeServer) => Promise<unknown>,
) {
return jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => Promise<unknown> },
'connect',
)
.mockImplementation(impl);
}
describe('McpClientsService.connectWithTimeout — hung connect does not poison the cache (D1)', () => {
beforeEach(() => jest.useFakeTimers());
afterEach(() => {
jest.clearAllTimers();
jest.useRealTimers();
jest.restoreAllMocks();
});
it('buildEntry completes (server recorded ok:false) when connect never settles, and toolsFor does not hang', async () => {
const svc = buildService([server({ id: 'id-hung', name: 'hung' })]);
// connect NEVER settles — models a wedged createMCPClient handshake.
stubConnect(svc, () => new Promise<never>(() => {}));
const toolsetPromise = svc.toolsFor('ws-1');
// Drive fake time past the connect bound so connectWithTimeout rejects and
// buildEntry catches it (records ok:false) — flushing the microtasks.
await jest.advanceTimersByTimeAsync(CONNECT_TIMEOUT_MS + 1);
const toolset = await toolsetPromise;
// The build COMPLETED with the bad server skipped (no tools, ok:false).
expect(Object.keys(toolset.tools)).toHaveLength(0);
expect(toolset.outcomes).toEqual([
{ name: 'hung', ok: false, reason: 'MCP connect timed out after 5000ms' },
]);
await Promise.all(toolset.clients.map((c) => c.close()));
// The cache is NOT poisoned: a subsequent turn returns (served from the warm
// cached entry) instead of awaiting a never-settling build.
const again = await svc.toolsFor('ws-1');
expect(Object.keys(again.tools)).toHaveLength(0);
await Promise.all(again.clients.map((c) => c.close()));
});
it('a hung server is skipped but a healthy server in the SAME build still contributes its tools', async () => {
const svc = buildService([
server({ id: 'id-hung', name: 'hung' }),
server({ id: 'id-ok', name: 'ok' }),
]);
const okClient = {
tools: () => Promise.resolve({ search: { description: 'x' } }),
close: jest.fn().mockResolvedValue(undefined),
};
stubConnect(svc, (s) =>
s.id === 'id-hung'
? new Promise<never>(() => {})
: Promise.resolve(okClient),
);
const toolsetPromise = svc.toolsFor('ws-2');
await jest.advanceTimersByTimeAsync(CONNECT_TIMEOUT_MS + 1);
const toolset = await toolsetPromise;
// Healthy server's tool survives (namespaced); hung server recorded ok:false.
expect(Object.keys(toolset.tools)).toEqual(['ok_search']);
expect(toolset.outcomes).toEqual([
{ name: 'hung', ok: false, reason: 'MCP connect timed out after 5000ms' },
{ name: 'ok', ok: true },
]);
await Promise.all(toolset.clients.map((c) => c.close()));
});
it('closes the ORPHANED client when connect resolves LATE (after the timeout)', async () => {
const svc = buildService([server({ id: 'id-late', name: 'late' })]);
const lateClient = {
tools: () => Promise.resolve({}),
close: jest.fn().mockResolvedValue(undefined),
};
// connect resolves only AFTER the connect bound has already elapsed, so
// connectWithTimeout has already rejected and must close this orphan.
stubConnect(
svc,
() =>
new Promise((resolve) => {
setTimeout(() => resolve(lateClient), CONNECT_TIMEOUT_MS * 2);
}),
);
const toolsetPromise = svc.toolsFor('ws-3');
// Fire the timeout: the build completes with the server skipped.
await jest.advanceTimersByTimeAsync(CONNECT_TIMEOUT_MS + 1);
const toolset = await toolsetPromise;
expect(toolset.outcomes[0]?.ok).toBe(false);
expect(lateClient.close).not.toHaveBeenCalled();
// Now let the late connect resolve — the orphan must be closed, not leaked.
await jest.advanceTimersByTimeAsync(CONNECT_TIMEOUT_MS * 2);
expect(lateClient.close).toHaveBeenCalledTimes(1);
await Promise.all(toolset.clients.map((c) => c.close()));
});
});
describe('McpClientsService.buildEntry — closes a connected client whose tools() fails (leak fix)', () => {
beforeEach(() => jest.useFakeTimers());
afterEach(() => {
jest.clearAllTimers();
jest.useRealTimers();
jest.restoreAllMocks();
});
it('connect succeeds but tools() REJECTS: the client is close()d exactly once and the server is skipped, while a healthy server still contributes', async () => {
const svc = buildService([
server({ id: 'id-bad', name: 'bad' }),
server({ id: 'id-ok', name: 'ok' }),
]);
// The bad server connects fine, then tools() rejects — the client would leak if
// buildEntry did not close it in the per-server catch (it was never registered).
const badClient = {
tools: () => Promise.reject(new Error('tools listing failed')),
close: jest.fn().mockResolvedValue(undefined),
};
const okClient = {
tools: () => Promise.resolve({ search: { description: 'x' } }),
close: jest.fn().mockResolvedValue(undefined),
};
stubConnect(svc, (s) =>
s.id === 'id-bad' ? Promise.resolve(badClient) : Promise.resolve(okClient),
);
const toolset = await svc.toolsFor('ws-4');
// The orphaned (never-registered) client is closed exactly once — no leak.
expect(badClient.close).toHaveBeenCalledTimes(1);
// Healthy server survives; bad server recorded ok:false and skipped.
expect(Object.keys(toolset.tools)).toEqual(['ok_search']);
expect(toolset.outcomes).toEqual([
{ name: 'bad', ok: false, reason: 'tools listing failed' },
{ name: 'ok', ok: true },
]);
// The healthy (registered) client is NOT closed by the loop — it is owned by the
// cache entry and stays warm (closed only on eviction/teardown, not on lease
// release). Releasing the lease keeps it warm since the entry is not evicted.
expect(okClient.close).not.toHaveBeenCalled();
await Promise.all(toolset.clients.map((c) => c.close()));
expect(okClient.close).not.toHaveBeenCalled();
// The failed client is never double-closed.
expect(badClient.close).toHaveBeenCalledTimes(1);
});
it('connect succeeds but tools() HANGS (times out): the client is close()d once and the server is skipped', async () => {
const svc = buildService([server({ id: 'id-slow', name: 'slow' })]);
const slowClient = {
// tools() never settles -> withTimeout rejects after CONNECT_TIMEOUT_MS.
tools: () => new Promise<Record<string, never>>(() => {}),
close: jest.fn().mockResolvedValue(undefined),
};
stubConnect(svc, () => Promise.resolve(slowClient));
const toolsetPromise = svc.toolsFor('ws-5');
// Drive fake time past the tools() bound so withTimeout rejects.
await jest.advanceTimersByTimeAsync(CONNECT_TIMEOUT_MS + 1);
const toolset = await toolsetPromise;
expect(slowClient.close).toHaveBeenCalledTimes(1);
expect(Object.keys(toolset.tools)).toHaveLength(0);
expect(toolset.outcomes[0]?.ok).toBe(false);
await Promise.all(toolset.clients.map((c) => c.close()));
});
});
@@ -195,7 +195,7 @@ export class McpClientsService {
): Promise<{ ok: true; tools: string[] } | { ok: false; error: string }> {
let client: McpClient | undefined;
try {
client = await this.connect(server);
client = await this.connectWithTimeout(server, CONNECT_TIMEOUT_MS);
const raw = await withTimeout(client.tools(), CONNECT_TIMEOUT_MS);
return { ok: true, tools: Object.keys(raw) };
} catch (err) {
@@ -255,49 +255,96 @@ export class McpClientsService {
const callTimeoutMs = mcpCallTimeoutMs();
const instructions: McpServerInstruction[] = [];
for (const server of servers) {
// Per-server connect+tools result, still tagged with its server so the merge
// below can be applied in the SAME order as `servers` (see the parallel note).
type PerServerResult =
| { ok: true; client: McpClient; guarded: Record<string, Tool> }
| { ok: false; reason: string };
// Connect to (and list tools for) every enabled server CONCURRENTLY, so the
// total build time is bounded by the SLOWEST single server (~2×
// CONNECT_TIMEOUT_MS: connect + tools), NOT the SUM across servers. The
// sequential loop this replaced summed those bounds, so with enough all-timing-
// out servers the outer MCP_TOOLSET_BUILD_DEADLINE_MS could fire before the
// per-server bounds, dropping ALL external tools and inverting the "per-server
// bound is primary, outer is a backstop" invariant. Each server keeps its OWN
// try/catch + connectWithTimeout/withTimeout bound + close-on-failure logic; a
// failed server is skipped, never fatal. Nothing here mutates the shared
// arrays — every result is merged IN SERVER ORDER after Promise.all, so tool-
// key precedence/disambiguation, `outcomes`, `instructions` and `clients`
// ordering all match the previous sequential behavior exactly.
const perServer = async (
server: (typeof servers)[number],
): Promise<PerServerResult> => {
// Track the connected client so the catch can close it when it was obtained
// but tools() then threw/timed out (connectWithTimeout closes its OWN orphan
// on a connect timeout, so `client` stays undefined on that path). On success
// the client is handed back and registered by the merge below (owned by the
// entry, closed at teardown) — so it is never double-closed.
let client: McpClient | undefined;
try {
const client = await this.connect(server);
client = await this.connectWithTimeout(server, CONNECT_TIMEOUT_MS);
const raw = await withTimeout(client.tools(), CONNECT_TIMEOUT_MS);
clients.push(client);
const allow = server.toolAllowlist;
const picked =
Array.isArray(allow) && allow.length > 0 ? pick(raw, allow) : raw;
// Bound each tool's execute with a per-call total-timeout guard before
// merging, so a single chatty-but-stuck call is aborted after the cap.
const guarded = wrapToolsWithCallTimeout(picked, callTimeoutMs);
// Namespace each tool with the sanitized server name AND disambiguate
// against names already merged from earlier servers, so no external
// tool is silently overwritten on collision. The returned count drives
// whether this server's prompt guidance is included (≥1 tool merged).
const merged = this.mergeNamespaced(
tools,
guarded,
server.name,
server.id,
);
outcomes.push({ name: server.name, ok: true });
// Include this server's guidance ONLY when it actually contributed at
// least one tool the agent can call (allowlist may have filtered all of
// them out) AND the admin authored non-blank instructions. The header
// prefix is the sanitized server name (= the tool namespace prefix).
const guide = server.instructions?.trim();
if (merged.count > 0 && guide) {
instructions.push({
serverName: server.name,
toolPrefix: merged.prefix,
instructions: guide,
});
}
return { ok: true, client, guarded };
} catch (err) {
// A failed server is skipped — the turn proceeds with the rest. Log a
// short warning (never the URL/headers) so ops can see degradation, and
// record the outcome so the UI can show "tool X unavailable".
// A failed server is skipped — the turn proceeds with the rest. If connect
// returned a live client but a later step (tools()) threw, that client was
// never registered in `clients`, so close it here or its transport/socket
// leaks (compounding every 60s cache rebuild during a flaky-server outage).
if (client) {
void client.close().catch(() => undefined);
}
// Log a short warning (never the URL/headers) so ops can see degradation,
// and record the outcome so the UI can show "tool X unavailable".
const reason = shortError(err);
this.logger.warn(
`External MCP server "${server.name}" unavailable: ${reason}`,
);
outcomes.push({ name: server.name, ok: false, reason });
return { ok: false, reason };
}
};
// Promise.all preserves array order regardless of settle order, so `results[i]`
// is `servers[i]`'s outcome — the merge below stays deterministic and matches
// the old sequential order (later servers still override/disambiguate against
// earlier ones on a tool-key clash).
const results = await Promise.all(servers.map(perServer));
for (let i = 0; i < servers.length; i += 1) {
const server = servers[i];
const result = results[i];
if (result.ok !== true) {
outcomes.push({ name: server.name, ok: false, reason: result.reason });
continue;
}
clients.push(result.client);
// Namespace each tool with the sanitized server name AND disambiguate
// against names already merged from earlier servers, so no external
// tool is silently overwritten on collision. The returned count drives
// whether this server's prompt guidance is included (≥1 tool merged).
const merged = this.mergeNamespaced(
tools,
result.guarded,
server.name,
server.id,
);
outcomes.push({ name: server.name, ok: true });
// Include this server's guidance ONLY when it actually contributed at
// least one tool the agent can call (allowlist may have filtered all of
// them out) AND the admin authored non-blank instructions. The header
// prefix is the sanitized server name (= the tool namespace prefix).
const guide = server.instructions?.trim();
if (merged.count > 0 && guide) {
instructions.push({
serverName: server.name,
toolPrefix: merged.prefix,
instructions: guide,
});
}
}
@@ -383,6 +430,55 @@ export class McpClientsService {
return client;
}
/**
* Race {@link connect} against a SETTLING timeout so a hung MCP handshake can
* never POISON the per-workspace build cache. `createMCPClient` (inside connect)
* is NOT bounded internally, and exactly like @ai-sdk/mcp's tool calls
* (see wrapToolWithCallTimeout) its promise does NOT settle on abort. So a
* transient network blip mid-handshake can make connect hang FOREVER. Because
* getOrBuildEntry caches the build PROMISE, a never-settling connect would then
* wedge EVERY later turn for the workspace (each awaits the same pending build,
* step_count stuck at 0, run row leaks 'running', chat 409s forever). Bounding
* connect here guarantees buildEntry always gets a client OR a rejection within
* `ms` so the build completes (bad server skipped) and the cache stays clean.
*
* If connect resolves LATE (after we already rejected on the timeout), we close
* the orphaned client so its transport/socket is not leaked.
*/
private connectWithTimeout(
server: Pick<AiMcpServer, 'transport' | 'url' | 'headersEnc'>,
ms: number,
): Promise<McpClient> {
return new Promise<McpClient>((resolve, reject) => {
let settled = false;
const timer = setTimeout(() => {
settled = true;
reject(new Error(`MCP connect timed out after ${ms}ms`));
}, ms);
// Do not keep the process alive just for this connect-timeout timer.
timer.unref?.();
this.connect(server).then(
(client) => {
if (settled) {
// The race was already lost to the timeout: close the orphaned client
// so its socket is not leaked, and drop the late result.
void client.close().catch(() => undefined);
return;
}
clearTimeout(timer);
settled = true;
resolve(client);
},
(err: unknown) => {
if (settled) return; // late rejection after the timeout — already handled
clearTimeout(timer);
settled = true;
reject(err instanceof Error ? err : new Error(String(err)));
},
);
});
}
/**
* Decrypt the stored auth headers. Returns undefined when none are set. The
* plaintext headers live only in this returned object and are passed straight
@@ -460,12 +556,12 @@ export function validateResolvedAddresses(addrs: readonly LookupAddress[]): {
*/
function buildPinnedDispatcher(): Agent {
// External-MCP traffic uses a DEDICATED, shorter silence timeout
// (`AI_MCP_STREAM_TIMEOUT_MS`, default 5 min) — deliberately tighter than the
// (`AI_MCP_STREAM_TIMEOUT_MS`, default 1 min) — deliberately tighter than the
// chat provider's 15-min `streamTimeoutMs()` — so a byte-silent/hung MCP
// upstream is broken in ~5 min instead of 15. We keep the keep-alive options
// upstream is broken in ~1 min instead of 15. We keep the keep-alive options
// from `streamingDispatcherOptions()` but OVERRIDE headers/body timeouts.
// Accepted trade-off: a legitimately long but byte-silent single tool call,
// and an SSE transport idling >5 min BETWEEN tool calls, are also cut here; the
// and an SSE transport idling >1 min BETWEEN tool calls, are also cut here; the
// per-call total cap (wrapToolsWithCallTimeout, `AI_MCP_CALL_TIMEOUT_MS`) is the
// complementary guard for chatty-but-stuck calls that keep the socket warm yet
// never return.
@@ -52,7 +52,7 @@ export class CreateAgentRoleDto {
description?: string;
@IsString()
@MaxLength(20000)
@MaxLength(100000)
instructions: string;
// null/omitted => use the workspace default model.
@@ -102,7 +102,7 @@ export class UpdateAgentRoleDto {
@IsOptional()
@IsString()
@MaxLength(20000)
@MaxLength(100000)
instructions?: string;
@IsOptional()
@@ -652,6 +652,125 @@ describe('AiChatToolsService #294 changed execute wirings', () => {
});
});
/**
* #410 the footnote + image tools were promoted from MCP-only into the shared
* registry and are now wired in-app. Assert they are REGISTERED in the in-app
* toolset and forward their args to the client with the correct arg->method
* mapping (the schema fields `imageUrl`/`attachmentId` map onto the client's
* positional `url`/`oldAttachmentId`). A field destructured under the wrong name
* would silently pass `undefined` (execute is `any`-cast, so tsc won't catch it).
*/
describe('AiChatToolsService #410 footnote + image tools', () => {
const calls: Record<string, unknown[][]> = {
insertFootnote: [],
insertImage: [],
replaceImage: [],
};
const fakeClient: Partial<DocmostClientLike> = {
insertFootnote: (...args: unknown[]) => {
calls.insertFootnote.push(args);
return Promise.resolve({ success: true, footnoteId: 'fn1', reused: false });
},
insertImage: (...args: unknown[]) => {
calls.insertImage.push(args);
return Promise.resolve({ success: true, attachmentId: 'att1' });
},
replaceImage: (...args: unknown[]) => {
calls.replaceImage.push(args);
return Promise.resolve({ success: true, replaced: 1 });
},
};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
generateCollabToken: jest.fn().mockResolvedValue('collab-token'),
};
let service: AiChatToolsService;
beforeEach(() => {
for (const k of Object.keys(calls)) calls[k].length = 0;
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue(
mockLoaded(function () {
return fakeClient as DocmostClientLike;
} as unknown as loader.DocmostClientCtor),
);
service = new AiChatToolsService(
tokenServiceStub as never,
{} as never,
{} as never,
{} as never,
{} as never,
{
asSink: () => ({ put: jest.fn(), has: jest.fn(), evict: jest.fn() }),
} as never,
);
});
afterEach(() => jest.restoreAllMocks());
const buildTools = () =>
service.forUser(
{ id: 'user-1', email: 'u@example.com', workspaceId: 'ws-1' } as never,
'session-1',
'ws-1',
'chat-1',
);
it('registers all three tools in the in-app toolset', async () => {
const tools = await buildTools();
expect(tools.insertFootnote).toBeDefined();
expect(tools.insertImage).toBeDefined();
expect(tools.replaceImage).toBeDefined();
});
it('insertFootnote forwards (pageId, anchorText, text) positionally', async () => {
const tools = await buildTools();
const r = await tools.insertFootnote.execute(
{ pageId: 'p1', anchorText: 'the claim', text: 'See source.' } as never,
{} as never,
);
expect(calls.insertFootnote).toEqual([['p1', 'the claim', 'See source.']]);
expect(r).toMatchObject({ footnoteId: 'fn1' });
});
it('insertImage maps imageUrl->url and packs the option fields', async () => {
const tools = await buildTools();
await tools.insertImage.execute(
{
pageId: 'p1',
imageUrl: 'https://x/img.png',
align: 'center',
alt: 'A',
replaceText: '[img]',
afterText: undefined,
} as never,
{} as never,
);
expect(calls.insertImage).toEqual([
[
'p1',
'https://x/img.png',
{ align: 'center', alt: 'A', replaceText: '[img]', afterText: undefined },
],
]);
});
it('replaceImage maps attachmentId->oldAttachmentId and imageUrl->url', async () => {
const tools = await buildTools();
await tools.replaceImage.execute(
{
pageId: 'p1',
attachmentId: 'att-old',
imageUrl: 'https://x/new.png',
align: 'right',
alt: 'B',
} as never,
{} as never,
);
expect(calls.replaceImage).toEqual([
['p1', 'att-old', 'https://x/new.png', { align: 'right', alt: 'B' }],
]);
});
});
/**
* getCurrentPage selection contract (#388): the tool surfaces the selection that
* was sanitized + nested onto the resolved open-page context (last forUser arg).
@@ -697,6 +697,38 @@ export class AiChatToolsService {
},
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
// Promoted from MCP-only so the in-app agent can attach a REAL footnote to
// already-written text instead of leaving a literal `^[...]` string.
insertFootnote: sharedTool(
sharedToolSpecs.insertFootnote,
async ({ pageId, anchorText, text }) =>
await client.insertFootnote(pageId, anchorText, text),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
// The schema field is `imageUrl`; the client method takes it positionally.
insertImage: sharedTool(
sharedToolSpecs.insertImage,
async ({ pageId, imageUrl, align, alt, replaceText, afterText }) =>
await client.insertImage(pageId, imageUrl, {
align,
alt,
replaceText,
afterText,
}),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
replaceImage: sharedTool(
sharedToolSpecs.replaceImage,
async ({ pageId, attachmentId, imageUrl, align, alt }) =>
await client.replaceImage(pageId, attachmentId, imageUrl, {
align,
alt,
}),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The table reference parameter was unified to `table` (was `tableRef`).
tableInsertRow: sharedTool(
@@ -141,6 +141,33 @@ export interface DocmostClientLike {
doc?: unknown,
title?: string,
): Promise<Record<string, unknown>>;
// Attach an author-inline footnote after the first occurrence of anchorText;
// numbering + the footnotes list are derived server-side.
insertFootnote(
pageId: string,
anchorText: string,
text: string,
): Promise<Record<string, unknown>>;
// Download a web image and insert it into the page (append, or replace/after a
// text anchor). `url` is the image http(s) URL.
insertImage(
pageId: string,
url: string,
opts?: {
align?: 'left' | 'center' | 'right';
alt?: string;
replaceText?: string;
afterText?: string;
},
): Promise<Record<string, unknown>>;
// Swap an existing image (by its attachmentId) for a new one fetched from a web
// URL, repointing every reference in the live document.
replaceImage(
pageId: string,
oldAttachmentId: string,
url: string,
opts?: { align?: 'left' | 'center' | 'right'; alt?: string },
): Promise<Record<string, unknown>>;
tableInsertRow(
pageId: string,
tableRef: string,
@@ -27,13 +27,26 @@ import type { DocmostClientLike } from './docmost-client.loader';
*/
describe('tool tier metadata (#332)', () => {
it('core set is the documented 13 + searchInPage (14)', () => {
expect(CORE_TOOL_KEYS).toHaveLength(14);
it('core set is the documented 13 + searchInPage + insertFootnote (15)', () => {
expect(CORE_TOOL_KEYS).toHaveLength(15);
expect(CORE_TOOL_SET.has('searchInPage')).toBe(true); // #330, promoted to core
expect(CORE_TOOL_SET.has('insertFootnote')).toBe(true); // #410, promoted to core
// loadTools is a meta-tool, not a normal core key.
expect(CORE_TOOL_SET.has(LOAD_TOOLS_NAME)).toBe(false);
});
it('#410 image tools are DEFERRED, footnote tool is CORE', () => {
// insert_footnote is core (symmetric with editPageText); the image tools stay
// deferred (rare, fat — loaded on demand). Assert both the spec tier and the
// CORE_TOOL_SET membership so a future tier edit that desyncs them fails here.
expect(SHARED_TOOL_SPECS.insertFootnote.tier).toBe('core');
expect(CORE_TOOL_SET.has('insertFootnote')).toBe(true);
expect(SHARED_TOOL_SPECS.insertImage.tier).toBe('deferred');
expect(CORE_TOOL_SET.has('insertImage')).toBe(false);
expect(SHARED_TOOL_SPECS.replaceImage.tier).toBe('deferred');
expect(CORE_TOOL_SET.has('replaceImage')).toBe(false);
});
it('SHARED_TOOL_SPECS tier agrees with CORE_TOOL_SET for every shared tool', () => {
for (const [key, spec] of Object.entries(SHARED_TOOL_SPECS)) {
const isCoreByTier = spec.tier === 'core';
@@ -38,10 +38,13 @@ export interface ToolCatalogEntry {
}
/**
* CORE (always-active) in-app tool keys 13 frequent/tiny tools. `searchInPage`
* (#330) is added to core on top of the issue's original tier list: it is
* frequent for the editorial roles this feature targets. `loadTools` is active
* too but is not a normal tool key (it is added to activeTools separately).
* CORE (always-active) in-app tool keys 13 frequent/tiny tools + `searchInPage`
* (#330) + `insertFootnote` (#410). `searchInPage` is core because it is frequent
* for the editorial roles this feature targets; `insertFootnote` is core so the
* footnote tool is NOT hidden while its natural sibling `editPageText` is always
* active (that asymmetry is exactly what pushed the agent to write literal
* `^[...]`). `loadTools` is active too but is not a normal tool key (it is added
* to activeTools separately).
*/
export const CORE_TOOL_KEYS = [
'searchPages',
@@ -60,6 +63,9 @@ export const CORE_TOOL_KEYS = [
// #330 search_in_page — frequent for editorial sweeps; core despite predating
// the issue's tier list.
'searchInPage',
// #410 insert_footnote — core so pinpoint citations to already-written text
// don't degrade into literal `^[...]`; kept symmetric with editPageText.
'insertFootnote',
] as const;
/** O(1) membership test for the core tier. */
@@ -123,19 +123,19 @@ export function streamKeepAliveMs(): number {
return positiveEnv('AI_STREAM_KEEPALIVE_MS', DEFAULT_STREAM_KEEPALIVE_MS);
}
/** Default SILENCE timeout for EXTERNAL-MCP transport (5 min). */
const DEFAULT_MCP_STREAM_TIMEOUT_MS = 300_000;
/** Default SILENCE timeout for EXTERNAL-MCP transport (1 min). */
const DEFAULT_MCP_STREAM_TIMEOUT_MS = 60_000;
/** Default total wall-clock cap for ONE external MCP tool call (15 min). */
const DEFAULT_MCP_CALL_TIMEOUT_MS = 900_000;
/** Default total wall-clock cap for ONE external MCP tool call (2 min). */
const DEFAULT_MCP_CALL_TIMEOUT_MS = 120_000;
/**
* SILENCE timeout (ms) for EXTERNAL-MCP transport ONLY. Override with
* `AI_MCP_STREAM_TIMEOUT_MS`; a missing/invalid/non-positive value falls back to
* {@link DEFAULT_MCP_STREAM_TIMEOUT_MS} (5 min).
* {@link DEFAULT_MCP_STREAM_TIMEOUT_MS} (1 min).
*
* Deliberately tighter than the chat provider's {@link streamTimeoutMs} (15 min)
* so a byte-silent/hung MCP upstream is broken in ~5 min instead of 15. This is
* so a byte-silent/hung MCP upstream is broken in ~1 min instead of 15. This is
* the undici `headersTimeout`/`bodyTimeout` for the external-MCP dispatcher only
* it must NOT change the chat provider, which legitimately needs 15 min between
* reasoning chunks (#175).
@@ -153,7 +153,7 @@ export function mcpStreamTimeoutMs(): number {
/**
* Total wall-clock cap (ms) for ONE external MCP tool call APP-LEVEL, not
* transport. Override with `AI_MCP_CALL_TIMEOUT_MS`; a missing/invalid/
* non-positive value falls back to {@link DEFAULT_MCP_CALL_TIMEOUT_MS} (15 min).
* non-positive value falls back to {@link DEFAULT_MCP_CALL_TIMEOUT_MS} (2 min).
*
* Catches a tool that keeps the connection warm (SSE heartbeats / trickle) but
* never returns a result which the transport silence timeout
@@ -149,6 +149,15 @@ export type DocmostMcpConfig = (
has?: (uri: string) => boolean;
evict?: (uri: string) => void;
};
// Dependency-neutral metrics sink injected by McpService (mirror of the
// package's onMetric). The package emits generic (name, value, labels)
// samples; McpService maps them onto the prom-client registry. Undefined
// when metrics are disabled → the package no-ops.
onMetric?: (
name: string,
value: number,
labels?: Record<string, string>,
) => void;
};
export interface ResolvedMcpAuth {
@@ -30,6 +30,11 @@ import {
ResolvedMcpAuth,
} from './mcp-auth.helpers';
import { SandboxStore } from '../sandbox/sandbox.store';
import {
isMetricsEnabled,
observeMcpTool,
incConnectTimeout,
} from '../metrics/metrics.registry';
// Minimal shape of the embedded MCP HTTP handler exported by @docmost/mcp/http.
interface McpHttpHandler {
@@ -331,7 +336,31 @@ export class McpService implements OnModuleDestroy {
// can store blobs in the shared in-RAM store regardless of which
// credential variant resolved. The sink (put/has/evict + uri↔id
// mapping) is owned by SandboxStore.asSink().
return { ...resolved.config, sandbox: this.sandboxStore.asSink() };
// Route the package's dependency-neutral metric samples onto the
// prom-client registry. When metrics are disabled, onMetric is
// undefined → the package's tool-timer/timeout hooks are a
// negligible-overhead no-op: the registerTool wrapper still runs a
// performance.now() + async try/finally per tool call, but the
// `onMetric?.()` short-circuits so no label/object is built. (Cost
// is immaterial at LLM tool-call rate.) labels?.tool is guarded
// defensively (the tool wrapper always sets it).
return {
...resolved.config,
sandbox: this.sandboxStore.asSink(),
onMetric: isMetricsEnabled()
? (
name: string,
value: number,
labels?: Record<string, string>,
) => {
if (name === 'mcp_tool_duration_seconds') {
observeMcpTool(labels?.tool ?? 'other', value);
} else if (name === 'collab_connect_timeouts_total') {
incConnectTimeout();
}
}
: undefined,
};
},
{
identify: (req: IncomingMessage) => {
@@ -2,6 +2,11 @@
* Perf-metrics contract (#355). These names/labels are FIXED by the already
* deployed scrape+dashboard infra (VictoriaMetrics scraping docmost:9464,
* Grafana dashboards, alerts). Do NOT rename them.
*
* #402 extends the #355 table with the collab-lifecycle + MCP-tool families
* (grouped below). These are the fixed contract from #402; same "do not rename"
* rule applies. Server-side families land in Pass 1; the MCP-tool histogram is
* fed by the MCP callback in a later pass but its NAME is fixed here now.
*/
export const METRIC_HTTP_REQUEST_DURATION = 'http_request_duration_seconds';
export const METRIC_DB_QUERY_DURATION = 'db_query_duration_seconds';
@@ -9,6 +14,17 @@ export const METRIC_BULLMQ_QUEUE_DEPTH = 'bullmq_queue_depth';
export const METRIC_BULLMQ_JOB_DURATION = 'bullmq_job_duration_seconds';
export const METRIC_COLLAB_STORE_DURATION = 'collab_store_duration_seconds';
// #402 additions — collaboration lifecycle + MCP tool timing.
export const METRIC_COLLAB_LOAD_DURATION = 'collab_doc_load_duration_seconds';
export const METRIC_COLLAB_DOCS_OPEN = 'collab_docs_open';
export const METRIC_COLLAB_DOC_LOADS_TOTAL = 'collab_doc_loads_total';
export const METRIC_COLLAB_DOC_UNLOADS_TOTAL = 'collab_doc_unloads_total';
export const METRIC_COLLAB_CONNECT_DURATION = 'collab_connect_duration_seconds';
export const METRIC_COLLAB_CONNECT_TIMEOUTS_TOTAL =
'collab_connect_timeouts_total';
export const METRIC_COLLAB_AUTH_DURATION = 'collab_auth_duration_seconds';
export const METRIC_MCP_TOOL_DURATION = 'mcp_tool_duration_seconds';
// Histogram buckets (seconds). Chosen to give useful p50/p95/p99 resolution
// for typical web/DB latencies without exploding series cardinality.
export const HTTP_BUCKETS = [
@@ -23,6 +39,12 @@ export const COLLAB_BUCKETS = [
export const JOB_BUCKETS = [
0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120,
];
// #402 — MCP tool-call latency. Same shape as COLLAB_BUCKETS but stretched to
// 10s at the top: an MCP tool round-trip (LLM-driven doc ops) can be slower
// than a single collab store, so keep resolution out to 10s.
export const MCP_TOOL_BUCKETS = [
0.005, 0.025, 0.1, 0.25, 0.5, 1, 2.5, 5, 10,
];
/**
* Extract the first SQL token (select/insert/update/delete/...) from a query,
@@ -58,6 +80,30 @@ export function firstSqlToken(sql: string | undefined): string {
return KNOWN_SQL_TOKENS.has(token) ? token : 'other';
}
/**
* #402 bucket a document byte size into ONE of four fixed labels for the
* collab load/store histograms' `size_bucket` label. Using the raw byte count
* would be a continuous, unbounded label; four coarse buckets keep series
* cardinality bounded (each of collab_doc_load / collab_store gets ×4 series).
*
* The SAME function is shared by both the load and store paths so their buckets
* can never drift apart. Non-finite / negative sizes collapse to the smallest
* bucket ('lt64k') as a safe default (they can't be legitimately huge and we
* must never throw here this runs on every store/load when metrics are on).
*/
// Module-const thresholds, built once (not per observe).
const SIZE_THRESHOLDS = { lt64k: 65536, lt256k: 262144, lt1m: 1048576 } as const;
export function sizeBucket(
bytes: number | undefined | null,
): 'lt64k' | 'lt256k' | 'lt1m' | 'ge1m' {
if (bytes == null || !Number.isFinite(bytes) || bytes < 0) return 'lt64k';
if (bytes < SIZE_THRESHOLDS.lt64k) return 'lt64k';
if (bytes < SIZE_THRESHOLDS.lt256k) return 'lt256k';
if (bytes < SIZE_THRESHOLDS.lt1m) return 'lt1m';
return 'ge1m';
}
/**
* Whether an HTTP response must be EXCLUDED from http_request_duration_seconds.
*
@@ -1,5 +1,6 @@
import {
collectDefaultMetrics,
Counter,
Histogram,
Gauge,
Registry,
@@ -9,11 +10,21 @@ import {
DB_BUCKETS,
HTTP_BUCKETS,
JOB_BUCKETS,
MCP_TOOL_BUCKETS,
METRIC_BULLMQ_JOB_DURATION,
METRIC_BULLMQ_QUEUE_DEPTH,
METRIC_COLLAB_AUTH_DURATION,
METRIC_COLLAB_CONNECT_DURATION,
METRIC_COLLAB_CONNECT_TIMEOUTS_TOTAL,
METRIC_COLLAB_DOC_LOADS_TOTAL,
METRIC_COLLAB_DOC_UNLOADS_TOTAL,
METRIC_COLLAB_DOCS_OPEN,
METRIC_COLLAB_LOAD_DURATION,
METRIC_COLLAB_STORE_DURATION,
METRIC_DB_QUERY_DURATION,
METRIC_HTTP_REQUEST_DURATION,
METRIC_MCP_TOOL_DURATION,
sizeBucket,
} from './metrics.constants';
/**
@@ -39,7 +50,24 @@ let httpHist: Histogram<'method' | 'route' | 'status'> | null = null;
let dbHist: Histogram<'op'> | null = null;
let queueDepthGauge: Gauge<'queue'> | null = null;
let jobHist: Histogram<'queue'> | null = null;
let collabHist: Histogram | null = null;
// #402 — collab store now carries a size_bucket label (see sizeBucket()).
let collabHist: Histogram<'size_bucket'> | null = null;
// #402 collab-lifecycle + MCP instruments.
let collabLoadHist: Histogram<'size_bucket'> | null = null;
let docsOpenGauge: Gauge | null = null;
let docLoadsCounter: Counter | null = null;
let docUnloadsCounter: Counter | null = null;
let connectTimeoutsCounter: Counter | null = null;
let collabConnectHist: Histogram | null = null;
let collabAuthHist: Histogram | null = null;
let mcpToolHist: Histogram<'tool'> | null = null;
// #402 — read-on-scrape source for collab_docs_open. The gauge is NEVER
// inc/dec'd (that drifts under crashes/handoffs); instead its collect() callback
// pulls the authoritative live count from here on every scrape. Registered once,
// gated, by the collaboration gateway via registerDocsOpenSource(). Null until
// then → the collect() is a no-op.
let docsOpenSource: (() => number) | null = null;
function init(): void {
if (registry || !enabled) return;
@@ -82,10 +110,71 @@ function init(): void {
collabHist = new Histogram({
name: METRIC_COLLAB_STORE_DURATION,
help: 'Collaboration onStoreDocument duration in seconds',
help: 'Collaboration onStoreDocument duration in seconds, by document size bucket',
labelNames: ['size_bucket'],
buckets: COLLAB_BUCKETS,
registers: [registry],
});
collabLoadHist = new Histogram({
name: METRIC_COLLAB_LOAD_DURATION,
help: 'Collaboration onLoadDocument DB-load duration in seconds, by document size bucket',
labelNames: ['size_bucket'],
buckets: COLLAB_BUCKETS,
registers: [registry],
});
docsOpenGauge = new Gauge({
name: METRIC_COLLAB_DOCS_OPEN,
help: 'Number of collaboration documents currently open in memory',
registers: [registry],
// Read-on-scrape: pull the live count from the registered source (the
// hocuspocus instance) so the value can never drift. No-op until a source
// is registered.
collect() {
if (docsOpenSource) this.set(docsOpenSource());
},
});
docLoadsCounter = new Counter({
name: METRIC_COLLAB_DOC_LOADS_TOTAL,
help: 'Total collaboration documents loaded into memory',
registers: [registry],
});
docUnloadsCounter = new Counter({
name: METRIC_COLLAB_DOC_UNLOADS_TOTAL,
help: 'Total collaboration documents unloaded from memory',
registers: [registry],
});
connectTimeoutsCounter = new Counter({
name: METRIC_COLLAB_CONNECT_TIMEOUTS_TOTAL,
help: 'Total collaboration connection setup timeouts',
registers: [registry],
});
collabConnectHist = new Histogram({
name: METRIC_COLLAB_CONNECT_DURATION,
help: 'Collaboration connection acceptance duration in seconds (onConnect→connected)',
buckets: COLLAB_BUCKETS,
registers: [registry],
});
collabAuthHist = new Histogram({
name: METRIC_COLLAB_AUTH_DURATION,
help: 'Collaboration onAuthenticate duration in seconds',
buckets: COLLAB_BUCKETS,
registers: [registry],
});
mcpToolHist = new Histogram({
name: METRIC_MCP_TOOL_DURATION,
help: 'MCP tool-call duration in seconds, by tool name',
labelNames: ['tool'],
buckets: MCP_TOOL_BUCKETS,
registers: [registry],
});
}
// Runs once when this module is first imported. Safe to call again (idempotent).
@@ -121,6 +210,46 @@ export function observeJobDuration(queue: string, seconds: number): void {
jobHist?.observe({ queue }, seconds);
}
export function observeCollabStore(seconds: number): void {
collabHist?.observe(seconds);
export function observeCollabStore(bytes: number, seconds: number): void {
collabHist?.observe({ size_bucket: sizeBucket(bytes) }, seconds);
}
export function observeCollabLoad(bytes: number, seconds: number): void {
collabLoadHist?.observe({ size_bucket: sizeBucket(bytes) }, seconds);
}
/**
* Register the live open-document count source for the collab_docs_open gauge.
* Called ONCE, gated by isMetricsEnabled(), by the collaboration gateway. The
* gauge reads this on every scrape (collect()); nothing inc/dec's it.
*/
export function registerDocsOpenSource(fn: () => number): void {
docsOpenSource = fn;
}
export function incDocLoad(): void {
docLoadsCounter?.inc();
}
export function incDocUnload(): void {
docUnloadsCounter?.inc();
}
export function incConnectTimeout(): void {
connectTimeoutsCounter?.inc();
}
export function observeCollabConnect(seconds: number): void {
collabConnectHist?.observe(seconds);
}
export function observeCollabAuth(seconds: number): void {
collabAuthHist?.observe(seconds);
}
export function observeMcpTool(tool: string, seconds: number): void {
// `tool` MUST be a bounded, registration-derived MCP tool name (the caller
// guarantees it comes from the registered-tool set) — never free-form input —
// so this label stays low-cardinality with no 'other' bucketing needed.
mcpToolHist?.observe({ tool }, seconds);
}
@@ -1,6 +1,23 @@
import { FastifyRequest } from 'fastify';
import { resolveRouteLabel } from './http-metrics.hook';
import { firstSqlToken, isStreamingResponse } from './metrics.constants';
import {
firstSqlToken,
isStreamingResponse,
sizeBucket,
} from './metrics.constants';
import {
getMetricsRegistry,
incConnectTimeout,
incDocLoad,
incDocUnload,
isMetricsEnabled,
observeCollabAuth,
observeCollabConnect,
observeCollabLoad,
observeCollabStore,
observeMcpTool,
registerDocsOpenSource,
} from './metrics.registry';
describe('resolveRouteLabel (histogram route label)', () => {
it('uses the ROUTE TEMPLATE, never the raw URL', () => {
@@ -123,3 +140,67 @@ describe('firstSqlToken (bounded db label)', () => {
expect(firstSqlToken('vacuum analyze')).toBe('other');
});
});
describe('sizeBucket (#402 bounded size label)', () => {
it('maps sizes to the four fixed buckets at their boundaries', () => {
// Boundaries are exclusive-upper: <65536 → lt64k, etc.
expect(sizeBucket(0)).toBe('lt64k');
expect(sizeBucket(65535)).toBe('lt64k');
expect(sizeBucket(65536)).toBe('lt256k');
expect(sizeBucket(262143)).toBe('lt256k');
expect(sizeBucket(262144)).toBe('lt1m');
expect(sizeBucket(1048575)).toBe('lt1m');
expect(sizeBucket(1048576)).toBe('ge1m');
expect(sizeBucket(5_000_000)).toBe('ge1m');
});
it('falls back to the smallest bucket for invalid sizes', () => {
// Non-finite / negative / nullish collapse to the smallest, safe default.
expect(sizeBucket(-1)).toBe('lt64k');
expect(sizeBucket(NaN)).toBe('lt64k');
expect(sizeBucket(Infinity)).toBe('lt64k');
expect(sizeBucket(undefined)).toBe('lt64k');
expect(sizeBucket(null)).toBe('lt64k');
});
it('only ever returns one of the four fixed labels (bounded cardinality)', () => {
const labels = new Set(
[0, 65536, 262144, 1048576, -5, NaN].map((b) => sizeBucket(b)),
);
for (const l of labels) {
expect(['lt64k', 'lt256k', 'lt1m', 'ge1m']).toContain(l);
}
});
});
describe('metrics helpers are safe no-ops when METRICS_PORT is unset', () => {
// These specs run without METRICS_PORT, so the registry is never created and
// every observe/inc/set helper must be a cheap `?.` no-op that never throws.
beforeAll(() => {
// Guard the contract this suite depends on: if a CI env set METRICS_PORT,
// the assertions below would be meaningless, so fail loudly instead.
expect(process.env.METRICS_PORT).toBeUndefined();
});
it('reports metrics disabled and a null registry', () => {
expect(isMetricsEnabled()).toBe(false);
expect(getMetricsRegistry()).toBeNull();
});
it('does not throw from any #402 collab/MCP helper', () => {
expect(() => {
observeCollabLoad(123456, 0.01);
observeCollabStore(123456, 0.02);
observeCollabConnect(0.03);
observeCollabAuth(0.04);
observeMcpTool('some-tool', 0.05);
incDocLoad();
incDocUnload();
incConnectTimeout();
// Registering a source must not create the gauge or invoke the fn.
registerDocsOpenSource(() => {
throw new Error('docsOpenSource must NOT be called when disabled');
});
}).not.toThrow();
});
});
+14
View File
@@ -21,10 +21,24 @@ import { recordHttpResponse } from './integrations/metrics/http-metrics.hook';
import { startMetricsServer } from './integrations/metrics/metrics.server';
async function bootstrap() {
// Fastify JSON body cap. Fastify defaults to 1 MiB, which a long AI-chat
// research turn exceeds: the client resends the FULL message history (every
// tool call + search result) on each turn, so a deep conversation's POST to
// /api/ai-chat/stream can be several MB and would otherwise be rejected with
// FST_ERR_CTP_BODY_TOO_LARGE (413). Raise the cap; override with
// HTTP_JSON_BODY_LIMIT (bytes). A missing/invalid/non-positive value keeps the
// 25 MiB default. Multipart uploads are unaffected (their own @fastify/multipart
// limits apply); this only bounds JSON/urlencoded request bodies.
const bodyLimitEnv = Number(process.env.HTTP_JSON_BODY_LIMIT);
const bodyLimit =
Number.isFinite(bodyLimitEnv) && bodyLimitEnv > 0
? bodyLimitEnv
: 25 * 1024 * 1024;
const app = await NestFactory.create<NestFastifyApplication>(
AppModule,
new FastifyAdapter({
trustProxy: resolveTrustProxy(process.env.TRUST_PROXY),
bodyLimit,
routerOptions: {
maxParamLength: 1000,
ignoreTrailingSlash: true,
+308
View File
@@ -0,0 +1,308 @@
# Reading the AI dialog logs (how agents call tools, and where they fail)
How to inspect the agent conversation history in the database — and, more
importantly, the **one non-obvious trap that will make you report the wrong
answer**: the persisted history *silently hides hard tool failures*. Written from
real pain (a "which tools fail most?" analysis that confidently answered
"patchNode: 0 errors" while the UI was visibly full of red `patchNode` failures).
Read the **Gotchas** section before you trust any error count.
## TL;DR
- Agent chats live in Postgres, DB `docmost`, tables `ai_chat_*`.
- Each tool invocation is stored as **two** array elements (a `tool-call` part and
a `tool-result` part), so naive counting double-counts.
- **A tool that *throws* writes no result part.** Since the #407 fix its error is
persisted as a dedicated `{toolName, error}` element in `tool_calls` (queryable +
replayed to the model). **Rows written before #407 still drop it** — the error is
nowhere in the DB and shows only in the live UI. So `isError` / `success=false`
scans under-report by design, and pre-#407 thrown errors are invisible.
- To find where agents fail: (1) soft-failure markers in `tool_calls`, (2) the new
`error` field for thrown errors (new rows) / the orphan-gap proxy (old rows),
(3) server logs / the live UI for full stack traces beyond the truncated message.
## Where the data lives
Host `island.lc` (`10.31.40.120`), container `gitmost-postgresql`
(`pgvector/pgvector:pg18`), database `docmost`.
```bash
ssh island.lc
# one-off query:
docker exec gitmost-postgresql psql -U docmost -d docmost -P pager=off -c "SELECT ..."
# interactive:
docker exec -it gitmost-postgresql psql -U docmost -d docmost
```
The main app container is `gitmost` (`DATABASE_URL=postgresql://docmost:...@db:5432/docmost`).
All workspaces (vvzvlad / wb / asakusa / …) share this **single** database — they
are rows in `workspaces`, not separate deployments.
### Relevant tables
| Table | What it holds |
| --- | --- |
| `ai_chats` | one row per conversation (`title`, `role_id`, `page_id`, `creator_id`) |
| `ai_chat_messages` | every message; tool calls live in `tool_calls` jsonb |
| `ai_chat_runs` | one row per agent run (turn): `status`, `error`, `step_count` |
| `ai_agent_roles` | agent definitions (`instructions`, `model_config`) |
| `ai_mcp_servers` | configured MCP tool servers per workspace |
`ai_chat_messages` columns that matter: `role` (`user` | `assistant` — there is **no**
separate `tool` role), `content` (text), `tool_calls` (jsonb array), `metadata`
(jsonb, holds run `error` + rendered `parts`), `status`, `tsv` (full-text index).
## How tool calls are stored — READ THIS
Tool calls are **not** one-object-per-call. Each logical invocation is split into
two consecutive elements of the `tool_calls` array:
```text
index 0: { "toolName": "getPage", "input": { "pageId": "…" } } ← tool-call (has input, NO output)
index 1: { "toolName": "getPage", "output": { … } } ← tool-result (has output, NO input)
```
The keys that appear on an element are `toolName`, `input`, `output`, and — for a
**thrown** failure on rows written after the #407 fix — `error` (the tool's error
message; see the "Hard failures" section below). There is no `state`, no `errorText`,
no `type`. On pre-#407 rows a thrown failure has NO paired result element at all
(silent orphan). Consequences:
1. **Real invocation count = elements that have `output` or `error`.** Counting every
element double-counts (you get ~2× and a spurious "~50% of every tool has no output").
2. **Pairing:** a call = a `tool-call` part followed by its result part. A success
carries `output`; a thrown failure (post-#407) carries `error` instead. Both carry
`toolName`, so you can group by tool on either.
## The two classes of failure (and which the DB can see)
### 1. Soft failures — tool RAN and returned an error-shaped result → PERSISTED ✅
These are visible in the `tool-result` `output`. The marker differs per tool:
| Tool(s) | Error marker in `output` |
| --- | --- |
| `editPageText` | `failed` is a **non-empty** array of `{find, reason}` (e.g. `text not found in the document`, `matches N times — provide a longer fragment or set replaceAll`). Also a soft `warning` when the `find` string contained markdown that only matched after stripping. |
| `semanticSearch` | `{ "unavailable": true, "reason": "semantic search unavailable" }` (feature/infra, not the agent's fault) |
| MCP passthrough (`Habr_*`, some `Search_*`) | `output` is an **array** (raw MCP content) whose text starts with `Error executing tool … validation error …` |
| generic | `output.isError = true` or `output.success = false` |
Note `editPageText` returns `failed: []` on success — filtering on the *presence*
of the key gives false positives; filter on **non-empty**.
### 2. Hard failures — tool THREW → NOW PERSISTED ✅ (since the #407 fix)
When a tool throws (the classic one is `patchNode` / `insertNode` / `tableUpdateCell`
`Failed to encode document to Yjs (fromJSON): Unknown node type: undefined`), the
runtime still writes **no `tool-result` part** — the failure is an ai@6 `tool-error`
content part instead. **Since the #407 fix, that error is persisted**: `serializeSteps`
appends a dedicated element `{toolName, error: "<message>"}` right after the failed
call, mirroring how a successful `{toolName, output}` element is appended. So a thrown
error now leaves a queryable `error` field carrying its (truncated) reason, and the
same real text is replayed to the model on the next turn (an `output-error` part with
the real `errorText`, no longer the `'Tool call did not complete.'` placeholder).
**Cutover caveat — old rows keep the old blind shape.** Rows written **before** this
change have the two-part shape (`call` + `output` only) and simply **drop** thrown
errors, leaving a silent **orphan** (a `call` with no `output` *and* no `error`). Rows
written **after** the fix additionally carry the `error` element. So:
- **New rows:** query the `error` field directly (see the hard-error query below) — no
orphan heuristic needed for thrown failures.
- **Old rows (pre-#407):** the only DB-side proxy is still an **orphan**: a `tool-call`
part with no matching `tool-result` *and* no `error`. Orphans also appear when a run
is **aborted** mid-flight (server restart), so a high-volume tool (`createComment`,
`searchInPage`, `Search_web_search`) shows orphans from aborts, not real errors on
old rows. Treat the orphan gap as an *upper bound*, and cross-check the tool: a gap on
a structural editor (`patchNode`, `insertNode`, `updatePageJson`, `transformPage`) is
almost certainly a thrown Yjs-encode error; a gap on `createComment` is mostly aborts.
A note on the aborted-call fallback: a call with **neither** a result **nor** a
`tool-error` (genuinely interrupted mid-step) still replays with the
`'Tool call did not complete.'` placeholder and persists as an orphan — that path is
unchanged, and is distinct from a real thrown error, which now carries `error`.
### 3. Run-level failures → `ai_chat_runs`
`status``succeeded | aborted | failed | running`; `error` holds the text. Seen in
the wild: `Run interrupted by a server restart.` (aborts) and
`Failed after N attempts. Last error: The service may be temporarily overloaded`
(LLM provider 529). These are infra/provider, not agent tool misuse.
## Ready-to-use queries
Run all of these via `docker exec gitmost-postgresql psql -U docmost -d docmost -P pager=off -c "…"`.
**Real invocation count per tool** (result parts only — the correct denominator):
```sql
SELECT elem->>'toolName' AS tool, count(*) AS calls
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
GROUP BY 1 ORDER BY 2 DESC;
```
**Soft errors per tool** (everything the DB can honestly see):
```sql
WITH res AS (
SELECT elem->>'toolName' AS tool, elem->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
)
SELECT tool, count(*) AS calls,
sum(COALESCE(
(o->>'isError') = 'true'
OR (o->>'success') = 'false'
OR (jsonb_typeof(o->'failed') = 'array' AND o->'failed' <> '[]'::jsonb)
OR (o->>'unavailable') = 'true'
OR o::text ~* 'error executing tool|validation error'
, false)::int) AS soft_errors
FROM res GROUP BY tool HAVING sum(COALESCE(
(o->>'isError') = 'true' OR (o->>'success') = 'false'
OR (jsonb_typeof(o->'failed') = 'array' AND o->'failed' <> '[]'::jsonb)
OR (o->>'unavailable') = 'true' OR o::text ~* 'error executing tool|validation error'
, false)::int) > 0
ORDER BY soft_errors DESC;
```
**`editPageText` failure reasons** (the most common real agent mistake — bad `find`):
```sql
WITH res AS (
SELECT elem->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array'
AND elem->>'toolName' = 'editPageText' AND elem ? 'output'
)
SELECT f->>'reason' AS reason, count(*)
FROM res, jsonb_array_elements(o->'failed') f
WHERE jsonb_typeof(o->'failed') = 'array'
GROUP BY 1 ORDER BY 2 DESC;
```
**Hard errors — persisted `error` field per tool (NEW rows, since #407)** — thrown
tool failures now carry their real reason, so query them directly:
```sql
SELECT elem->>'toolName' AS tool, count(*) AS thrown_errors,
min(elem->>'error') AS sample_error
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'error'
GROUP BY 1 ORDER BY 2 DESC;
```
**Hard-error proxy for OLD rows (pre-#407) — orphan gap per tool, WITH a spread column**
(call parts minus result parts, plus how many distinct chats the gap is spread across).
This covers rows written before thrown errors were persisted; on new rows a thrown
failure now has its own `error` element (use the query above) and an orphan means only
a genuinely aborted mid-step call:
```sql
WITH parts AS (
SELECT m.chat_id, elem->>'toolName' AS tool,
(elem ? 'input' AND NOT (elem ? 'output')) AS is_call,
(elem ? 'output' OR elem ? 'error') AS is_result
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND m.role = 'assistant'
),
per_chat AS (
SELECT tool, chat_id, sum(is_call::int) - sum(is_result::int) AS gap
FROM parts GROUP BY tool, chat_id
)
SELECT tool,
sum(gap) FILTER (WHERE gap > 0) AS missing_results,
count(*) FILTER (WHERE gap > 0) AS chats_spread, -- disambiguates!
max(gap) AS worst_single_chat
FROM per_chat GROUP BY tool
HAVING sum(gap) FILTER (WHERE gap > 0) > 0
ORDER BY missing_results DESC;
```
The `is_result` predicate counts an `error` element as a paired result too, so on new
rows a persisted thrown error no longer inflates the orphan gap; a remaining gap is an
aborted/interrupted call.
**On OLD rows, `missing_results` mixes thrown errors AND aborted/interrupted runs — you
cannot split them from `output` alone** (a positional "what follows the orphan" heuristic
breaks on parallel tool batches, which persist as `call,call,…,result,result`). Use
`chats_spread` to disambiguate:
- **spread across many chats** (e.g. `createComment` 96 over 29 chats) → a **systemic
real error** (here: inline-comment anchor text not found on the page).
- **concentrated in one chat** (e.g. `searchInPage` 55, of which 51 in a single chat)
**one runaway/aborted session**, not a real per-call error — discount it.
- a gap on a **structural editor** (`patchNode`, `insertNode`, `tableUpdateCell`,
`updatePageJson`, `transformPage`) is almost always a thrown Yjs-encode error.
**Run-level failures:**
```sql
SELECT status, count(*), min(error) AS sample_error
FROM ai_chat_runs GROUP BY status ORDER BY 2 DESC;
```
**Full-text search across messages.** The `tsv` GIN index is built as
`to_tsvector('english', unaccent(content))` — so it **stems English** but **not
Russian** (Russian lexemes are stored unstemmed, so only exact word forms match).
Most content here is Russian, so prefer `ILIKE` for substring search:
```sql
-- Russian / substring — reliable:
SELECT chat_id, left(content, 120)
FROM ai_chat_messages WHERE content ILIKE '%иранск%' LIMIT 20;
-- English phrase — can use the index:
SELECT chat_id, left(content, 120)
FROM ai_chat_messages
WHERE tsv @@ websearch_to_tsquery('english', 'some phrase') LIMIT 20;
```
## Don't blow up your context
A single `tool_calls` row can be **300–400 KB** (results embed full page content and
search payloads). Never `SELECT tool_calls` (or `jsonb_pretty(tool_calls)`) raw.
Always project just the keys you need and truncate:
```sql
SELECT elem->>'toolName',
left(regexp_replace((elem->'output')::text, '\s+', ' ', 'g'), 200)
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE elem ? 'output' LIMIT 5;
```
## Server logs & live UI (for the error text the DB drops)
```bash
docker logs -f --tail=100 gitmost # main app
docker compose -p gitmost logs -f --tail=100 # whole stack
```
Logging is `json-file`, `max-size=10m max-file=5` → ~50 MB retained, then rotated,
and **wiped on container recreate**. Since the #407 fix, thrown-tool error text is
**persisted in the `error` field** of `tool_calls` (see the hard-error query above), so
you no longer depend on live logs for it. Logs/live UI remain useful for **pre-#407
rows** (whose thrown errors were dropped) and for full stack traces beyond the
truncated stored message. A per-tool `tool_calls_total{tool,status}` metric to
VictoriaMetrics is still a possible future add for aggregate dashboards.
## Gotchas checklist
- [ ] Counting every `tool_calls` element → **overcount**. Count `output` elements; add `error` elements for thrown failures (new rows), but don't count both as invocations.
- [ ] `isError` / `success=false` ≈ 0 does **not** mean "no errors" — thrown errors are a separate `error` element (new rows) or dropped entirely (pre-#407 rows).
- [ ] Thrown errors persist only on rows written **after the #407 fix** — pre-#407 rows still drop them (orphan only). Mind the cutover when trending over time.
- [ ] `editPageText.failed` is `[]` on success — test for **non-empty**, not presence.
- [ ] Orphan gap on OLD rows mixes thrown errors **and** aborted runs — split by tool. On NEW rows a thrown error is its own `error` element, so a gap ≈ aborted call.
- [ ] `aborted` runs = server restarts, `failed` runs = provider overload — not agent mistakes.
- [ ] Never dump a raw `tool_calls` cell — it can be hundreds of KB.
- [ ] Logs are ephemeral (≤50 MB, wiped on recreate) — grab hard-error text live.
## Snapshot (2026-07-07, illustrative — rerun the queries for current numbers)
- 226 chats, 732 messages, 46 runs; ~4 400 real tool invocations.
- Soft errors (persisted): `editPageText` 4/79 (bad/non-unique `find`) + 9 markdown-in-`find` warnings; `semanticSearch` 3/4 (`unavailable`); `Habr_update_draft_from_docmost` 1/2 (`doc` sent as object, not string).
- Missing-result proxy, read WITH the spread column:
- **Systemic (spread) → real errors:** `createComment` 96 over **29 chats** (comment anchor text not found — the biggest real error hotspot); `editPageText` 31 over 12 chats (+ the 4 soft above); structural-editor Yjs throws `insertNode` 10 / `updatePageContent` 9 / `tableUpdateCell` 6 / `patchNode` 5 / `updatePageJson` 2 / `transformPage` 2.
- **Concentrated → NOT real errors:** `searchInPage` 55 (51 in one chat); `Search_web_search` 15 & `Search_searxng_web_search` 6 (timeouts/aborts in long research sessions).
- Runs: 34 succeeded, 10 aborted (server restart), 1 failed (provider overload).
+2 -1
View File
@@ -97,7 +97,8 @@
"patchedDependencies": {
"scimmy@1.3.5": "patches/scimmy@1.3.5.patch",
"yjs@13.6.30": "patches/yjs@13.6.30.patch",
"ai@6.0.134": "patches/ai@6.0.134.patch"
"ai@6.0.134": "patches/ai@6.0.134.patch",
"@hocuspocus/server@3.4.4": "patches/@hocuspocus__server@3.4.4.patch"
},
"overrides": {
"prosemirror-changeset": "2.4.0",
+1
View File
@@ -52,6 +52,7 @@
"form-data": "^4.0.0",
"jsdom": "^27.4.0",
"marked": "^17.0.1",
"pako": "^2.0.3",
"re2": "^1.21.0",
"ws": "^8.19.0",
"y-prosemirror": "1.3.7",
+619 -186
View File
@@ -8,10 +8,6 @@ import {
filterComment,
filterSearchResult,
} from "./lib/filters.js";
import { HocuspocusProvider } from "@hocuspocus/provider";
import { TiptapTransformer } from "@hocuspocus/transformer";
import * as Y from "yjs";
import WebSocket from "ws";
import { convertProseMirrorToMarkdown } from "./lib/markdown-converter.js";
import {
collectInternalFileNodes,
@@ -24,11 +20,10 @@ import {
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
buildCollabWsUrl,
assertYjsEncodable,
applyDocToFragment,
MutationResult,
} from "./lib/collaboration.js";
import { acquireCollabSession } from "./lib/collab-session.js";
import { footnoteWarningsField } from "./lib/footnote-analyze.js";
import { buildPageTree } from "./lib/tree.js";
import {
@@ -40,6 +35,7 @@ import {
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
@@ -49,6 +45,15 @@ import {
} from "./lib/node-ops.js";
import { searchInDoc, SearchOptions } from "./lib/page-search.js";
import { withPageLock } from "./lib/page-lock.js";
import {
prepareModel,
decodeDrawioSvg,
buildDrawioSvg,
mxHash,
normalizeXml,
countUserCells,
} from "./lib/drawio-xml.js";
import { renderDiagramShapes } from "./lib/drawio-preview.js";
import {
applyTextEdits,
TextEdit,
@@ -59,10 +64,12 @@ import { getCollabToken, performLogin } from "./lib/auth-utils.js";
import { diffDocs, summarizeChange } from "./lib/diff.js";
import {
applyAnchorInDoc,
canAnchorInDoc,
countAnchorMatches,
getAnchoredText,
resolveAnchorSelection,
normalizeForMatch,
} from "./lib/comment-anchor.js";
import { closestBlockHint } from "./lib/text-normalize.js";
import {
blockText,
walk,
@@ -75,6 +82,7 @@ import {
canonicalizeFootnotes,
insertInlineFootnote,
} from "./lib/transforms.js";
import { normalizeAndMergeFootnotes } from "./lib/footnote-normalize-merge.js";
import vm from "node:vm";
// Supported image types, kept as two lookup tables so both a local file
@@ -137,6 +145,15 @@ export type DocmostMcpConfig = { apiUrl: string } & (
has?: (uri: string) => boolean;
evict?: (uri: string) => void;
};
// Dependency-neutral metrics sink. When present, the client emits generic
// (name, value, labels) samples; the HOST maps those names onto its own
// metrics registry (the package never depends on prom-client or the server).
// Absent in standalone/stdio mode → the client is a complete no-op here.
onMetric?: (
name: string,
value: number,
labels?: Record<string, string>,
) => void;
};
// Canonical UUID shape (versions 1–8, matching the `uuid` package's `validate`
@@ -173,6 +190,11 @@ export class DocmostClient {
// op's image blobs if the final doc put throws. Null when the sink omits them.
private sandboxHas: ((uri: string) => boolean) | null = null;
private sandboxEvict: ((uri: string) => void) | null = null;
// Optional dependency-neutral metrics sink (see DocmostMcpConfig.onMetric).
// Null on the legacy positional form and whenever the host omits it → no-op.
private onMetricFn:
| ((name: string, value: number, labels?: Record<string, string>) => void)
| null = null;
// In-flight login dedup: when the token expires, the 401 interceptor,
// ensureAuthenticated, getCollabTokenWithReauth and the two multipart retries
// can all call login() at once. Memoizing a single promise collapses that
@@ -222,6 +244,8 @@ export class DocmostClient {
this.sandboxHas = config.sandbox.has ?? null;
this.sandboxEvict = config.sandbox.evict ?? null;
}
// Legacy positional form carries no onMetric → null (complete no-op).
this.onMetricFn = config.onMetric ?? null;
this.client = axios.create({
baseURL: this.apiUrl,
// Default request timeout so a hung connection cannot wedge a per-page
@@ -401,173 +425,32 @@ export class DocmostClient {
* change report. The report is computed AFTER the atomic read->write and
* never throws.
*/
private mutateLiveContentUnlocked(
private async mutateLiveContentUnlocked(
pageId: string,
collabToken: string,
transform: (liveDoc: any) => any | null,
): Promise<MutationResult> {
const CONNECT_TIMEOUT_MS = 25000;
const PERSIST_TIMEOUT_MS = 20000;
const ydoc = new Y.Doc();
const wsUrl = buildCollabWsUrl(this.apiUrl);
return new Promise<MutationResult>((resolve, reject) => {
let provider: HocuspocusProvider | undefined;
let applied = false; // onSynced may fire again on reconnect — apply once.
let settled = false;
let connectionLost = false;
let connectTimer: ReturnType<typeof setTimeout> | undefined;
let persistTimer: ReturnType<typeof setTimeout> | undefined;
let unsyncedHandler: ((data: { number: number }) => void) | undefined;
// The verifiable result resolved on every success/abort path. Set on abort
// (no-op report) and after a real write (computed change report).
let mutationResult: MutationResult;
const cleanup = () => {
if (connectTimer) clearTimeout(connectTimer);
if (persistTimer) clearTimeout(persistTimer);
if (provider) {
if (unsyncedHandler) {
try {
provider.off("unsyncedChanges", unsyncedHandler);
} catch (err) {}
}
try {
provider.destroy();
} catch (err) {}
}
};
const finish = (err: Error | null, value?: MutationResult) => {
if (settled) return;
settled = true;
cleanup();
if (err) reject(err);
else resolve(value as MutationResult);
};
connectTimer = setTimeout(() => {
finish(new Error("Connection timeout to collaboration server"));
}, CONNECT_TIMEOUT_MS);
const waitForPersistence = () => {
if (settled) return;
if (!provider) {
finish(new Error("collab provider gone before persistence"));
return;
}
if (provider.unsyncedChanges === 0) {
finish(null, mutationResult);
return;
}
persistTimer = setTimeout(() => {
finish(
new Error(
"Timeout waiting for collaboration server to persist the update",
),
);
}, PERSIST_TIMEOUT_MS);
unsyncedHandler = (data: { number: number }) => {
if (data.number === 0 && !connectionLost) {
finish(null, mutationResult);
}
};
provider.on("unsyncedChanges", unsyncedHandler);
};
provider = new HocuspocusProvider({
url: wsUrl,
name: `page.${pageId}`,
document: ydoc,
token: collabToken,
// @ts-ignore - Required for Node.js environment
WebSocketPolyfill: WebSocket,
onDisconnect: () => {
connectionLost = true;
finish(
new Error(
"Collaboration connection closed before the update was persisted/synced",
),
);
},
onClose: () => {
connectionLost = true;
finish(
new Error(
"Collaboration connection closed before the update was persisted/synced",
),
);
},
onSynced: () => {
if (applied || settled) return;
applied = true;
// CRITICAL: keep everything between reading and writing the live doc
// synchronous (no await) so no remote update can interleave.
let newDoc: any;
let beforeDoc: any;
try {
let liveDoc = TiptapTransformer.fromYdoc(ydoc, "default");
if (
!liveDoc ||
typeof liveDoc !== "object" ||
!Array.isArray(liveDoc.content)
) {
liveDoc = { type: "doc", content: [] };
}
// Snapshot the before-doc for the change report (safe deep clone).
beforeDoc = JSON.parse(JSON.stringify(liveDoc));
newDoc = transform(liveDoc);
if (newDoc == null) {
// Transform aborted — write nothing, return the live doc with a
// no-op change report.
mutationResult = {
doc: liveDoc,
verify: {
changed: false,
textInserted: 0,
textDeleted: 0,
blocksChanged: 0,
marks: {},
summary: "no changes (transform aborted)",
},
};
finish(null, mutationResult);
return;
}
// Structural diff into the live fragment (issue #152), mirroring
// the main write path: preserves the Yjs ids of unchanged nodes so
// an open editor's cursor is not yanked to the end of the document.
// The previous destructive rewrite (delete-all + applyUpdate of a
// fresh Y.Doc) discarded every node id, so replaceImage — the only
// caller of this method — still reproduced the #152 cursor jump
// (#164). applyDocToFragment runs its own atomic `transact`.
applyDocToFragment(ydoc, newDoc);
} catch (e) {
finish(e instanceof Error ? e : new Error(String(e)));
return;
}
// Compute the verifiable change report AFTER the transact write: it
// only needs the JSON before/after, so it cannot affect the atomic
// read->write window, and summarizeChange never throws.
mutationResult = {
doc: newDoc,
verify: summarizeChange(beforeDoc, newDoc),
};
waitForPersistence();
},
onAuthenticationFailed: () => {
finish(
new Error("Authentication failed for collaboration connection"),
);
},
});
// Reuse a live CollabSession for the page (issue #400) instead of opening a
// fresh provider per op. acquireCollabSession does NOT take the per-page
// lock — the caller (replaceImage) already holds ONE withPageLock across its
// scan -> upload -> write sequence, and the mutex is not reentrant, so
// taking it here would deadlock. The synchronous read->write section and the
// unsyncedChanges/connectionLost ack logic live in CollabSession.mutate,
// preserved verbatim from the old inline machine (incl. the #152 structural
// diff that keeps a live editor's cursor anchored).
const session = await acquireCollabSession(pageId, collabToken, this.apiUrl, {
// Only the actual 25s collab connect timeout emits this — the connect-vs-
// unload signal; the other failure paths must NOT emit it.
onConnectTimeout: () =>
this.onMetricFn?.("collab_connect_timeouts_total", 1),
});
try {
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh.
session.destroy("mutate failed");
throw e;
}
}
/**
@@ -1674,6 +1557,8 @@ export class DocmostClient {
// leave footnotes out of order, orphaned, or in multiple lists — the bottom
// list + numbering are always derived from reference order. No-op when the
// footnotes are already canonical.
// #419: normalize + merge glyph-forked definitions before canonicalizing.
doc = normalizeAndMergeFootnotes(doc);
doc = canonicalizeFootnotes(doc);
// Write the BODY first, then the title (#159 split-brain): a failed body
@@ -1938,7 +1823,8 @@ export class DocmostClient {
// footnotes before copying — a no-op on already-canonical source content, but
// it guarantees a copy can never propagate a non-canonical footnote topology
// to the target (parity with the other full-doc write paths).
const canonical = canonicalizeFootnotes(content);
// #419: normalize + merge glyph-forked definitions before canonicalizing.
const canonical = canonicalizeFootnotes(normalizeAndMergeFootnotes(content));
const collabToken = await this.getCollabTokenWithReauth();
// Open the TARGET collab doc by its canonical UUID, never the slugId (#260).
@@ -2454,6 +2340,64 @@ export class DocmostClient {
};
}
/** Plain text of each TOP-LEVEL block of `doc`, for anchor-failure hints. */
private topLevelBlockTexts(doc: any): string[] {
const content = doc && Array.isArray(doc.content) ? doc.content : [];
return content
.map((b: any) => blockPlainText(b))
.filter((t: string) => t.length > 0);
}
/**
* True when per-block anchoring failed but the (normalized) selection DOES
* appear in the blocks' joined plain text i.e. it straddles a block
* boundary. Blocks are joined with a newline (collapsed to one space by
* normalizeForMatch) so a selection whose parts are separated by a paragraph
* break still matches. Callers only reach here after single-block anchoring
* (incl. the markdown-strip fallback) has already failed.
*/
private selectionSpansMultipleBlocks(
blockTexts: string[],
selection: string,
): boolean {
const normSel = normalizeForMatch(selection).norm.trim();
if (normSel.length === 0) return false;
const joined = normalizeForMatch(blockTexts.join("\n")).norm;
return joined.indexOf(normSel) !== -1;
}
/**
* Build the actionable error for a create_comment anchor MISS, porting
* edit_page_text's self-correction affordances: an explicit "spans multiple
* blocks" message when the selection straddles a block boundary, otherwise a
* "closest block text" hint quoting the block that holds the selection's
* longest token. `live` switches the wording between the pre-check (reading the
* persisted page) and the post-create live-anchor failure (which rolls back).
*/
private anchorNotFoundError(
doc: any,
selection: string,
live: boolean,
): Error {
const blockTexts = this.topLevelBlockTexts(doc);
const rolled = live ? " The comment was rolled back." : "";
if (this.selectionSpansMultipleBlocks(blockTexts, selection)) {
return new Error(
"create_comment: the selection spans multiple blocks; anchor on a " +
"contiguous fragment within a SINGLE paragraph/block (<=250 chars)." +
rolled,
);
}
const where = live ? "in the live document" : "in the page";
return new Error(
`create_comment: could not find the selection text ${where} to anchor ` +
"the comment. Provide the EXACT contiguous text from a single " +
"paragraph/block (<=250 chars)." +
closestBlockHint(blockTexts, selection) +
rolled,
);
}
/**
* Create an inline comment anchored to its `selection` text, or a reply.
*
@@ -2515,6 +2459,10 @@ export class DocmostClient {
// Captured in the pre-check below (which already reads the page) and used as
// payload.selection. Ordinary comments keep sending the raw agent selection.
let anchoredSelection: string | null = null;
// Set when the anchor matched only after stripping markdown from the
// selection (the strip fallback); surfaced as a soft warning like
// edit_page_text does, so a stale-markdown selection is flagged.
let anchorNormalized = false;
// For a top-level comment, fail BEFORE creating anything when the selection
// is not present in the persisted document — this avoids leaving an orphan
@@ -2530,10 +2478,7 @@ export class DocmostClient {
// rejected BEFORE creating the comment.
const matches = countAnchorMatches(page.content, selection);
if (matches === 0) {
throw new Error(
"create_comment: could not find the selection text in the page to anchor the comment. " +
"Provide the EXACT contiguous text from a single paragraph/block (<=250 chars).",
);
throw this.anchorNotFoundError(page.content, selection, false);
}
if (matches >= 2) {
throw new Error(
@@ -2547,18 +2492,27 @@ export class DocmostClient {
// null despite countAnchorMatches===1 (shouldn't happen), fall back to
// the raw agent selection below rather than crash.
anchoredSelection = getAnchoredText(page.content, selection);
} else if (!canAnchorInDoc(page.content, selection)) {
throw new Error(
"create_comment: could not find the selection text in the page to anchor the comment. " +
"Provide the EXACT contiguous text from a single paragraph/block (<=250 chars).",
);
anchorNormalized = resolveAnchorSelection(
page.content,
selection,
).normalized;
} else {
const resolved = resolveAnchorSelection(page.content, selection);
if (!resolved.found) {
throw this.anchorNotFoundError(page.content, selection, false);
}
anchorNormalized = resolved.normalized;
}
} catch (e) {
// Rethrow our own "not found"/"ambiguous" errors; swallow read/network
// errors so the live anchor step can still try (and enforce) anchoring.
// Rethrow our own "not found"/"ambiguous"/"spans multiple blocks" errors;
// swallow read/network errors so the live anchor step can still try (and
// enforce) anchoring.
if (
e instanceof Error &&
(e.message.startsWith("create_comment: could not find the selection") ||
e.message.startsWith(
"create_comment: the selection spans multiple blocks",
) ||
e.message.startsWith(
"create_comment: the suggestion's selection is ambiguous",
))
@@ -2630,6 +2584,10 @@ export class DocmostClient {
// Set inside the transform when a suggestion's live anchor is ambiguous
// (>=2 occurrences), so the rollback path can surface the right error.
let ambiguousInLiveDoc = false;
// Captured inside the transform on a not-found abort, so the rollback path
// can surface the closest-block / spans-multiple-blocks hint built from the
// LIVE document (the pre-check page is not in scope there).
let liveNotFoundError: Error | null = null;
try {
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
@@ -2657,6 +2615,13 @@ export class DocmostClient {
const liveCount = countAnchorMatches(doc, selection as string);
if (liveCount !== 1) {
ambiguousInLiveDoc = liveCount >= 2;
if (liveCount === 0) {
liveNotFoundError = this.anchorNotFoundError(
doc,
selection as string,
true,
);
}
return null;
}
}
@@ -2666,6 +2631,11 @@ export class DocmostClient {
}
// Selection text not found in the LIVE document: abort the write. The
// rollback + throw below turns this into a hard error.
liveNotFoundError = this.anchorNotFoundError(
doc,
selection as string,
true,
);
return null;
},
);
@@ -2682,13 +2652,28 @@ export class DocmostClient {
// suggestion, was ambiguous) in the live document. Roll back the comment
// and surface a hard error.
await this.safeDeleteComment(newCommentId);
throw new Error(
ambiguousInLiveDoc
? "create_comment: the suggestion's selection is ambiguous in the live document (multiple occurrences); the comment was rolled back. Expand the selection with surrounding context so it is unique."
: "create_comment: failed to anchor the comment (selection not found in the live document); the comment was rolled back",
if (ambiguousInLiveDoc) {
throw new Error(
"create_comment: the suggestion's selection is ambiguous in the live document (multiple occurrences); the comment was rolled back. Expand the selection with surrounding context so it is unique.",
);
}
throw (
liveNotFoundError ??
new Error(
"create_comment: failed to anchor the comment (selection not found in the live document); the comment was rolled back",
)
);
}
// Soft warning (like edit_page_text): the selection only matched after
// stripping markdown, so the caller likely quoted a styled fragment.
if (anchorNormalized) {
result.warning =
"The selection matched only after stripping markdown syntax; the comment " +
"was anchored on the document's plain text. Copy the selection verbatim " +
"from get_page / search_in_page output to avoid this.";
}
result.anchored = true;
return result;
}
@@ -3412,6 +3397,453 @@ export class DocmostClient {
});
}
// --- draw.io diagrams (issue #423) ---
/**
* Upload a ready-made byte buffer as a page attachment via the same
* multipart /files/upload endpoint uploadImage uses. Split out as its own
* (overridable) seam so drawio_create/update can upload the generated
* `.drawio.svg` without going through the URL-fetch path, and so tests can
* stub the network. Mirrors uploadImage's fresh-FormData + one-shot 401/403
* re-auth handling (a FormData body is single-use, so it must be rebuilt per
* attempt).
*/
protected async uploadAttachmentBuffer(
pageId: string,
buffer: Buffer,
fileName: string,
mime: string,
): Promise<{ id: string; fileName: string; fileSize: number }> {
await this.ensureAuthenticated();
const buildForm = () => {
const form = new FormData();
form.append("pageId", pageId);
form.append("file", buffer, { filename: fileName, contentType: mime });
return form;
};
const uploadUrl = `${this.apiUrl}/files/upload`;
let response;
try {
const form = buildForm();
response = await axios.post(uploadUrl, form, {
headers: {
...form.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} catch (error) {
if (
axios.isAxiosError(error) &&
(error.response?.status === 401 || error.response?.status === 403)
) {
await this.login();
const form2 = buildForm();
response = await axios.post(uploadUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} else if (axios.isAxiosError(error)) {
if (process.env.DEBUG) {
console.error(
"Attachment upload failed; response body:",
JSON.stringify(error.response?.data),
);
}
throw new Error(
`Attachment upload failed: ${error.response?.status} ${error.response?.statusText}`,
);
} else {
throw error;
}
}
const att = response.data?.data ?? response.data;
if (!att?.id || !att?.fileName) {
throw new Error(
"Unexpected /files/upload response: " + JSON.stringify(response.data),
);
}
return {
id: att.id,
fileName: att.fileName,
fileSize: att.fileSize ?? buffer.length,
};
}
/**
* Fetch a stored `.drawio.svg` attachment as text. Overridable seam over
* fetchInternalFile (the authed loopback fetch, which also rejects any
* traversal/SSRF src) so drawio_get/update can read the current diagram and
* tests can stub the bytes.
*/
protected async fetchAttachmentText(src: string): Promise<string> {
const { buffer } = await this.fetchInternalFile(src);
return buffer.toString("utf-8");
}
/**
* Resolve a drawio node on a page by `attrs.id` or `#<index>` and return the
* node plus its ref. Throws a clear error if the ref does not resolve to a
* drawio node.
*/
private async resolveDrawioNode(
pageId: string,
node: string,
): Promise<{ node: any; ref: string }> {
const data = await this.getPageRaw(pageId);
const hit = getNodeByRef(
data.content ?? { type: "doc", content: [] },
node,
);
if (!hit) {
throw new Error(
`drawio: no node found for "${node}" on page ${pageId} (use the drawio node's attrs.id or "#<index>" from get_outline)`,
);
}
if (hit.type !== "drawio") {
throw new Error(
`drawio: node "${node}" on page ${pageId} is a ${hit.type}, not a drawio diagram`,
);
}
return { node: hit.node, ref: node };
}
/**
* Read a drawio diagram as mxGraph XML (default) or as the raw `.drawio.svg`.
* Runs the decode chain (base64/entity content= drawio file nested XML or
* pako-inflated compressed <diagram>). The returned `hash` is the
* optimistic-lock key for drawio_update.
*/
async drawioGet(
pageId: string,
node: string,
format: "xml" | "svg" = "xml",
): Promise<{
pageId: string;
nodeId: string;
format: "xml" | "svg";
content: string;
meta: {
attachmentId: string | null;
title: string | null;
width: number | null;
height: number | null;
cellCount: number;
hash: string;
};
}> {
await this.ensureAuthenticated();
const { node: drawio } = await this.resolveDrawioNode(pageId, node);
const attrs = drawio.attrs || {};
const src = attrs.src;
if (!src) {
throw new Error(
`drawio: node "${node}" on page ${pageId} has no src to read`,
);
}
const svg = await this.fetchAttachmentText(src);
const modelXml = decodeDrawioSvg(svg);
const meta = {
attachmentId: attrs.attachmentId ?? null,
title: attrs.title ?? null,
width: attrs.width != null ? Number(attrs.width) : null,
height: attrs.height != null ? Number(attrs.height) : null,
cellCount: countUserCells(modelXml),
hash: mxHash(modelXml),
};
return {
pageId,
nodeId: attrs.id ?? node,
format,
content: format === "svg" ? svg : normalizeXml(modelXml),
meta,
};
}
/**
* Create a drawio diagram from mxGraph XML: lint schematic SVG preview
* (pure TS) build the `.drawio.svg` (createDrawioSvg contract) create the
* attachment insert a `drawio` node before/after an anchor or appended.
* `xml` is a bare `<mxGraphModel>` or a list of `<mxCell>` (the server wraps
* it and adds the id=0/id=1 sentinels).
*/
async drawioCreate(
pageId: string,
where: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
xml: string,
title?: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (
!where ||
(where.position !== "before" &&
where.position !== "after" &&
where.position !== "append")
) {
throw new Error(
'drawio_create: `where.position` must be one of "before", "after", "append"',
);
}
if (where.position === "before" || where.position === "after") {
const hasId =
typeof where.anchorNodeId === "string" && where.anchorNodeId.length > 0;
const hasText =
typeof where.anchorText === "string" && where.anchorText.length > 0;
if (hasId === hasText) {
throw new Error(
`drawio_create: position "${where.position}" requires exactly one of anchorNodeId or anchorText`,
);
}
}
// Pre-write pipeline (throws a structured DrawioLintError on any violation).
const prepared = prepareModel(xml);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
// NOTE: no `id` attribute is set here. The vendored `drawio` node schema
// (diagramAttributes) declares no `id`, so any block id would be silently
// dropped by PMNode.fromJSON on save and the returned handle would fail to
// resolve. The addressable handle is the node's "#<index>" (like image/table
// nodes), computed after the insert below.
const drawioNode: any = {
type: "drawio",
attrs: {
src: `/api/files/${att.id}/${att.fileName}`,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
align: "center",
},
};
if (title) drawioNode.attrs.title = title;
// Reuse the existing URL trust boundary (rejects unsafe src schemes).
this.validateDocUrls(drawioNode);
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let inserted = false;
let insertedIndex = -1;
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
inserted = false;
insertedIndex = -1;
const { doc: nd, inserted: ins } = insertNodeRelative(
liveDoc,
drawioNode,
where,
);
inserted = ins;
if (!inserted) return null; // anchor not found -> skip the write
// Locate the freshly-inserted node to derive its "#<index>" handle. The
// just-uploaded attachmentId is unique, so it identifies our node.
if (Array.isArray(nd.content)) {
insertedIndex = nd.content.findIndex(
(b: any) =>
b &&
b.type === "drawio" &&
b.attrs &&
b.attrs.attachmentId === att.id,
);
}
return nd;
},
);
if (!inserted) {
const anchorDesc = where.anchorNodeId
? `anchorNodeId "${where.anchorNodeId}"`
: `anchorText "${where.anchorText}"`;
throw new Error(
`drawio_create: anchor not found (${anchorDesc}) on page ${pageId}. The diagram attachment ${att.id} is now an unreferenced orphan.`,
);
}
if (insertedIndex < 0) {
// The node was inserted nested (e.g. inside a callout/table cell via an
// anchor), where "#<index>" — which addresses only top-level blocks —
// cannot reference it. drawio nodes carry no persisted id, so there is no
// stable handle for a nested diagram.
throw new Error(
`drawio_create: the diagram was inserted on page ${pageId} but not as a ` +
`top-level block, so it has no addressable "#<index>" handle. Anchor ` +
`on a top-level block (or append) so the diagram can be re-read.`,
);
}
// The returned handle is POSITIONAL ("#<index>"): valid for the immediate
// create -> get/update flow, but re-resolve via get_outline if the document
// structure changes (blocks added/removed before it shift the index).
const nodeId = `#${insertedIndex}`;
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
/**
* Full-replacement update of a drawio diagram. `baseHash` is MANDATORY: it is
* compared against the hash of the diagram's CURRENT XML (from drawio_get);
* any mismatch means a human or another agent edited the diagram after the
* read, so the write is refused with a conflict error. On success the new
* `.drawio.svg` is uploaded as a FRESH attachment (in-place byte overwrite is
* avoided some Docmost versions corrupt an attachment on overwrite, exactly
* as replaceImage documents) and the node is repointed with new dimensions.
*/
async drawioUpdate(
pageId: string,
node: string,
xml: string,
baseHash: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (typeof baseHash !== "string" || baseHash.length === 0) {
throw new Error(
"drawio_update: baseHash is mandatory — read the diagram with drawio_get first and pass back its meta.hash",
);
}
// Resolve the node and read the CURRENT diagram to enforce the optimistic
// lock before doing any write or upload.
const { node: drawio, ref } = await this.resolveDrawioNode(pageId, node);
const oldAttrs = drawio.attrs || {};
const oldSrc = oldAttrs.src;
// The returned handle is the caller-supplied reference. drawio nodes carry
// no persisted id, so `ref` (an "#<index>" or a rare legacy attrs.id) is the
// honest identifier to hand back.
const nodeId = oldAttrs.id ?? ref;
if (!oldSrc) {
throw new Error(
`drawio_update: node "${node}" on page ${pageId} has no src to compare against`,
);
}
const currentSvg = await this.fetchAttachmentText(oldSrc);
const currentHash = mxHash(decodeDrawioSvg(currentSvg));
if (currentHash !== baseHash) {
throw new Error(
`drawio_update: conflict — the diagram changed since it was read ` +
`(baseHash ${baseHash} != current ${currentHash}). Re-read it with drawio_get and retry.`,
);
}
// Pipeline for the new content (throws a structured DrawioLintError).
const prepared = prepareModel(xml);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = oldAttrs.title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
const newSrc = `/api/files/${att.id}/${att.fileName}`;
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let repointed = 0;
const repoint = (n: any) => {
n.attrs = {
...n.attrs,
src: newSrc,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
};
repointed++;
};
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
repointed = 0;
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
// Repoint ONLY the resolved node — never every node that happens to
// share this attachmentId (a copied diagram is two nodes with one
// attachmentId; keying on it would clobber both). Re-resolve the same
// handle against the live doc and walk to its exact position.
const hit = getNodeByRef(doc, ref);
if (!hit || hit.type !== "drawio") return null; // vanished/changed -> skip
let target: any = doc;
for (const idx of hit.path) {
if (!target || !Array.isArray(target.content)) {
target = null;
break;
}
target = target.content[idx];
}
if (!target || target.type !== "drawio") return null;
repoint(target);
if (repointed === 0) return null; // node vanished concurrently -> skip
return doc;
},
);
if (repointed === 0) {
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: [
...prepared.warnings,
"target drawio node was removed concurrently; uploaded attachment is unreferenced",
],
verify: mutation.verify,
};
}
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
// --- Page history / diff / transform ---
/**
@@ -3647,7 +4079,8 @@ export class DocmostClient {
// path can leave footnotes out of order / orphaned / in a raw `[^id]`
// block. In a dryRun preview this may surface footnote edits the script
// author did not write (the canonicalizer tidied them) — that is expected.
const result = canonicalizeFootnotes(raw);
// #419: normalize + merge glyph-forked definitions before canonicalizing.
const result = canonicalizeFootnotes(normalizeAndMergeFootnotes(raw));
newDoc = result;
return result;
};
+96 -96
View File
@@ -13,6 +13,11 @@ import { SHARED_TOOL_SPECS, SharedToolSpec } from "./tool-specs.js";
export { DocmostClient } from "./client.js";
export type { DocmostMcpConfig } from "./client.js";
// Teardown for the live per-page CollabSession cache (issue #400). An embedding
// HTTP host (the gitmost NestJS server) should call this from its own shutdown
// hook so no cached collab provider outlives the process.
export { destroyAllSessions } from "./lib/collab-session.js";
// Re-export the zod-agnostic shared tool-spec registry so the in-app AI-SDK
// service can read it off the loaded module (it cannot import the ESM package's
// internals directly; it goes through loadDocmostMcp()).
@@ -47,7 +52,7 @@ const VERSION = packageJson.version;
export const SERVER_INSTRUCTIONS =
"Docmost editing guide — choose the tool by intent.\n" +
"READ: find a page -> search (workspace-wide full-text); list -> list_pages / list_spaces. Locate blocks and their ids CHEAPLY -> get_outline (compact top-level map; start here, not get_page_json). One block's subtree -> get_node (by attrs.id, or \"#<index>\" for tables, which carry no id). Find every occurrence of a string/regex ON a page (and where each is) -> search_in_page, NOT block-by-block get_node — it returns each hit's node ref + block index + context for a targeted comment. Whole page -> get_page (Markdown, lossy; inline <span data-comment-id> tags are comment anchors — markup, not text) or get_page_json (lossless ProseMirror with block ids). Hand a huge page (with images) to an external consumer without pulling it through the model context -> stash_page (returns a short-lived anonymous URL).\n" +
"EDIT: fix wording/typos/numbers -> edit_page_text (find/replace inside blocks, no node id needed). Change ONE block (paragraph/heading/callout/etc.) structurally -> patch_node (by attrs.id from get_outline). Add a block -> insert_node (before/after a block by attrs.id or by anchor text, or append). Remove a block -> delete_node (by attrs.id). Tables -> table_get / table_update_cell / table_insert_row / table_delete_row (address by \"#<index>\" from get_outline; table nodes have no attrs.id). Images -> insert_image (add from a web URL) / replace_image (swap an existing image). Footnotes -> insert_footnote. Bulk/structural rewrite -> update_page_json (full ProseMirror replace; prefer the granular tools above to avoid resending the whole ~100KB+ document). Complex/scripted rewrite (multiple coordinated edits, renumbering) -> docmost_transform: write a JS `(doc, ctx) => doc` transform, preview the diff with dryRun (default), then apply with dryRun:false; ctx.helpers includes commentsToFootnotes for turning inline comments into numbered footnotes.\n" +
"EDIT: fix wording/typos/numbers -> edit_page_text (find/replace inside blocks, no node id needed). Change ONE block (paragraph/heading/callout/etc.) structurally -> patch_node (by attrs.id from get_outline). Add a block -> insert_node (before/after a block by attrs.id or by anchor text, or append). Remove a block -> delete_node (by attrs.id). Tables -> table_get / table_update_cell / table_insert_row / table_delete_row (address by \"#<index>\" from get_outline; table nodes have no attrs.id). Images -> insert_image (add from a web URL) / replace_image (swap an existing image). Draw.io diagrams -> drawio_create (create from mxGraph XML and insert), drawio_get (read a diagram as mxGraph XML + a hash), drawio_update (replace a diagram; pass the hash from drawio_get as baseHash for optimistic locking). Footnotes -> insert_footnote. Bulk/structural rewrite -> update_page_json (full ProseMirror replace; prefer the granular tools above to avoid resending the whole ~100KB+ document). Complex/scripted rewrite (multiple coordinated edits, renumbering) -> docmost_transform: write a JS `(doc, ctx) => doc` transform, preview the diff with dryRun (default), then apply with dryRun:false; ctx.helpers includes commentsToFootnotes for turning inline comments into numbered footnotes.\n" +
"PAGES: new -> create_page (Markdown). Rename (title only) -> rename_page. Move -> move_page. Delete -> delete_page (SOFT delete — the page goes to trash and is restorable; nothing is permanent). Copy/replace a page's whole content from another page (server-side, no document through the model) -> copy_page_content. Sharing -> share_page / unshare_page / list_shares; share_page makes the page PUBLICLY accessible — do it only when explicitly asked.\n" +
"COMMENTS: create_comment is always inline and requires an EXACT selection — contiguous text from a single block, <=250 chars (fails rather than leaving an unanchored comment); reply to a thread via parentCommentId. Propose a concrete text fix for one-click human approval -> create_comment with suggestedText (the exact plain-text replacement for the selection; the selection must then be UNIQUE in the page — extend it with context if needed); prefer this over editing directly when the change is subjective or needs the author's sign-off. Manage -> list_comments, update_comment, resolve_comment (resolve/reopen, reversible — prefer over delete to close), delete_comment, check_new_comments.\n" +
"HISTORY: review what changed -> diff_page_versions (a historyId vs current, or two versions). List saved versions -> list_page_history. Undo a bad edit -> restore_page_version (writes a past version back as current; itself revertible). Lossless markdown round-trip (download, edit, re-upload, incl. comment anchors) -> export_page_markdown / import_page_markdown.";
@@ -64,6 +69,33 @@ const jsonContent = (data: any) => ({
* REST + the collaboration WebSocket using the provided service-account
* credentials and auto-re-authenticates.
*/
/**
* Wrap a tool handler so its wall-clock duration is reported through the host's
* dependency-neutral sink as `mcp_tool_duration_seconds` (labelled by tool
* name). Pure and side-effect-free apart from the optional `onMetric` call:
* - preserves the handler's exact return value (awaited);
* - observes in a `finally`, so it records on BOTH success and throw, then
* rethrows the original error unchanged (never swallowed);
* - with no `onMetric` (standalone/stdio) it is a transparent pass-through.
* Exported so the timing contract can be unit-tested without a live transport.
*/
export function timeToolHandler(
name: string,
handler: (...args: any[]) => any,
onMetric?: (name: string, value: number, labels?: Record<string, string>) => void,
): (...args: any[]) => Promise<any> {
return async (...handlerArgs: any[]) => {
const start = performance.now();
try {
return await handler(...handlerArgs);
} finally {
onMetric?.("mcp_tool_duration_seconds", (performance.now() - start) / 1000, {
tool: name,
});
}
};
}
export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
// Pass the whole config union through: the client branches internally on
// credentials vs. getToken, so both the external /mcp (creds) and the
@@ -78,6 +110,26 @@ export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
{ instructions: SERVER_INSTRUCTIONS },
);
// Single choke point for MCP tool timing. Both `registerShared` (below) and
// the inline `server.registerTool(...)` calls funnel through this one method,
// so monkeypatching it HERE — before any tool is registered and before
// `registerShared` captures a reference to it — times every tool with no
// per-tool boilerplate. The wrapped handler records wall-clock duration and,
// in a `finally`, feeds the host's dependency-neutral sink
// `config.onMetric("mcp_tool_duration_seconds", seconds, { tool })`. The tool
// name is the registration name (bounded cardinality). When no onMetric is
// provided (standalone/stdio) the wrapper is a pure pass-through: it still
// returns the original result and rethrows the original error unchanged.
const originalRegisterTool = server.registerTool.bind(server) as (
...args: any[]
) => any;
(server as any).registerTool = (...args: any[]) => {
const name = args[0] as string;
const handler = args[args.length - 1];
const timedHandler = timeToolHandler(name, handler, config.onMetric);
return originalRegisterTool(...args.slice(0, -1), timedHandler);
};
// Register a tool from the shared, zod-agnostic spec registry. The spec owns
// the canonical name + model-facing description + (optional) schema builder;
// only the execute body is supplied per call. buildShape is invoked with THIS
@@ -380,43 +432,10 @@ registerShared(SHARED_TOOL_SPECS.deleteNode, async ({ pageId, nodeId }) => {
});
// Tool: insert_image
// MCP-only by design (NOT in the shared registry): the in-app AI-chat agent
// exposes no image tools (insert/replace), so there is no second layer to unify
// — a SHARED_TOOL_SPECS entry's tier/catalogLine are in-app metadata and the
// catalog-partition test forbids a spec without a live in-app tool (#294).
server.registerTool(
"insert_image",
{
description:
"Download an image from a web (http/https) URL and insert it into " +
"a page in one step. By default " +
"appends the image at the end of the page. With replaceText, replaces the " +
"first top-level block whose text contains that string (handy for " +
'swapping a text placeholder like "[image: foo.png]" for the real image). ' +
"With afterText, inserts the image right after the first block containing " +
"that string. Preserves all other block ids.",
inputSchema: {
pageId: z.string().min(1),
imageUrl: z
.string()
.min(1)
.describe("http(s) URL of the image to download and upload"),
align: z.enum(["left", "center", "right"]).optional(),
alt: z.string().optional(),
replaceText: z
.string()
.optional()
.describe(
"Replace the first top-level block whose text contains this string with the image",
),
afterText: z
.string()
.optional()
.describe(
"Insert the image right after the first top-level block whose text contains this string",
),
},
},
// Schema + description now live in the shared registry (#410) so BOTH this MCP
// server and the in-app AI-chat agent expose it. The execute body is unchanged.
registerShared(
SHARED_TOOL_SPECS.insertImage,
async ({ pageId, imageUrl, align, alt, replaceText, afterText }) => {
const result = await docmostClient.insertImage(pageId, imageUrl, {
align,
@@ -429,34 +448,9 @@ server.registerTool(
);
// Tool: replace_image
// MCP-only by design (see insert_image): no in-app equivalent, stays inline.
server.registerTool(
"replace_image",
{
description:
"Replace an existing image on a page with a new image fetched from a web " +
"(http/https) URL: uploads the new file as a NEW " +
"attachment (fresh clean URL that renders and busts browser caches), then " +
"repoints every image node referencing the old attachmentId (recursively, " +
"incl. callouts/tables) via the live document, preserving comments, " +
"alignment and alt. The old attachment is left as an unreferenced orphan " +
"(Docmost has no API to delete a single attachment; it is removed only when " +
"the page/space is deleted). In-place byte overwrite is avoided because some " +
"Docmost versions corrupt the attachment (HTTP 500) on overwrite.",
inputSchema: {
pageId: z.string().min(1),
attachmentId: z
.string()
.min(1)
.describe("attachmentId of the image currently in the page to replace"),
imageUrl: z
.string()
.min(1)
.describe("http(s) URL of the new image to download"),
align: z.enum(["left", "center", "right"]).optional(),
alt: z.string().optional(),
},
},
// Schema + description now live in the shared registry (#410).
registerShared(
SHARED_TOOL_SPECS.replaceImage,
async ({ pageId, attachmentId, imageUrl, align, alt }) => {
const result = await docmostClient.replaceImage(
pageId,
@@ -471,6 +465,38 @@ server.registerTool(
},
);
// Tool: drawio_get — read a draw.io diagram as mxGraph XML (or the raw SVG).
registerShared(
SHARED_TOOL_SPECS.drawioGet,
async ({ pageId, node, format }) => {
const result = await docmostClient.drawioGet(pageId, node, format ?? "xml");
return jsonContent(result);
},
);
// Tool: drawio_create — lint mxGraph XML, build the .drawio.svg, insert a node.
registerShared(
SHARED_TOOL_SPECS.drawioCreate,
async ({ pageId, xml, position, anchorNodeId, anchorText, title }) => {
const result = await docmostClient.drawioCreate(
pageId,
{ position, anchorNodeId, anchorText },
xml,
title,
);
return jsonContent(result);
},
);
// Tool: drawio_update — optimistic-locked full replacement of a diagram.
registerShared(
SHARED_TOOL_SPECS.drawioUpdate,
async ({ pageId, node, xml, baseHash }) => {
const result = await docmostClient.drawioUpdate(pageId, node, xml, baseHash);
return jsonContent(result);
},
);
// Tool: share_page
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own `searchIndexing ?? true` default.
@@ -779,36 +805,10 @@ server.registerTool(
);
// Tool: insert_footnote
// MCP-only by design (see insert_image): the in-app AI-chat agent exposes no
// footnote tool, so there is no second layer to unify — stays inline (#294).
server.registerTool(
"insert_footnote",
{
description:
"Insert an AUTHOR-INLINE footnote: you specify only WHERE (anchorText) " +
"and WHAT (text). The footnote marker is placed right after anchorText in " +
"the body, and the bottom footnotes list + the numbering are derived " +
"deterministically server-side. You do NOT assign a number, and you " +
"never see or edit the footnotes list — so footnotes cannot end up out " +
"of order, orphaned, or as a raw '[^id]' block. If a footnote with the " +
"SAME text already exists, its number is REUSED (one definition, several " +
"references). The write is atomic and won't clobber concurrent edits; if " +
"anchorText is not found, nothing is written and an error is returned.",
inputSchema: {
pageId: z.string().min(1),
anchorText: z
.string()
.min(1)
.describe(
"A snippet of existing body text; the footnote marker is inserted " +
"immediately after its first occurrence (mark-safe).",
),
text: z
.string()
.min(1)
.describe("The footnote content as markdown (becomes the definition)."),
},
},
// Schema + description now live in the shared registry (#410) so the in-app
// AI-chat agent exposes it too. The execute body is unchanged.
registerShared(
SHARED_TOOL_SPECS.insertFootnote,
async ({ pageId, anchorText, text }) => {
const result = await docmostClient.insertFootnote(pageId, anchorText, text);
return jsonContent(result);
+668
View File
@@ -0,0 +1,668 @@
import { HocuspocusProvider } from "@hocuspocus/provider";
import { TiptapTransformer } from "@hocuspocus/transformer";
import * as Y from "yjs";
import WebSocket from "ws";
import {
buildCollabWsUrl,
applyDocToFragment,
MutationResult,
} from "./collaboration.js";
import { summarizeChange } from "./diff.js";
/**
* Live per-page collaboration session cache (issue #400).
*
* The one-shot write path (collaboration.mutatePageContent /
* client.mutateLiveContentUnlocked) used to open a NEW HocuspocusProvider, run
* the full connect -> auth -> onLoadDocument -> initial-sync handshake, apply a
* single edit, wait for persistence, and then `provider.destroy()` for EVERY
* content mutation. Disconnecting after every edit means that once the pause
* between calls exceeds the server's write debounce, the server does a full
* store -> unload -> reload per cell, causing 25s connect timeouts and
* event-loop lag under a burst of edits on one page.
*
* This module keeps ONE live provider + ydoc per (wsUrl, pageId, token) alive
* across a SERIES of edits. While the provider stays connected the server never
* enters store -> unload -> reload, its debounce coalesces N writes into 1-2
* stores, and the repeated auth/load/initial-sync disappears.
*
* The synchronous read -> transform -> write section and the per-edit
* persistence-ack logic are preserved VERBATIM from the one-shot machine the
* only change is that they run on a persistent provider instead of a throwaway
* one. See CollabSession.mutate.
*/
/** Time we wait for the initial handshake/sync before giving up. */
const CONNECT_TIMEOUT_MS = 25000;
/** Time we wait for the server to acknowledge our write before giving up. */
const PERSIST_TIMEOUT_MS = 20000;
/**
* Tunables, read fresh from the environment on every acquire so tests (and a
* live rollback) can change them without reloading the module. Mirrors how
* http.ts parses MCP_SESSION_IDLE_MS.
* - MCP_COLLAB_SESSION_IDLE_MS: idle TTL, reset after every op. Default 60s.
* 0 (or negative) DISABLES the cache every op opens its own provider and
* destroys it after the op, i.e. the exact legacy per-op-provider behavior
* (the rollback path).
* - MCP_COLLAB_SESSION_MAX_AGE_MS: hard lifetime checked at acquire; bounds
* the permission-staleness window. Default 10 min.
* - MCP_COLLAB_SESSION_MAX_ENTRIES: registry cap; the least-recently-used
* session is destroy-evicted when the cap is reached. Default 32.
*/
interface SessionConfig {
idleMs: number;
maxAgeMs: number;
maxEntries: number;
}
function parseEnvInt(value: string | undefined, fallback: number): number {
const parsed = parseInt(value ?? "", 10);
return Number.isFinite(parsed) ? parsed : fallback;
}
function readConfig(): SessionConfig {
// idleMs: allow 0 (disable). A malformed value falls back to the default.
const idleRaw = parseInt(process.env.MCP_COLLAB_SESSION_IDLE_MS ?? "", 10);
const idleMs = Number.isFinite(idleRaw) ? Math.max(0, idleRaw) : 60 * 1000;
const maxAgeMs = Math.max(
0,
parseEnvInt(process.env.MCP_COLLAB_SESSION_MAX_AGE_MS, 10 * 60 * 1000),
);
const maxEntriesRaw = parseEnvInt(
process.env.MCP_COLLAB_SESSION_MAX_ENTRIES,
32,
);
const maxEntries = maxEntriesRaw > 0 ? maxEntriesRaw : 32;
return { idleMs, maxAgeMs, maxEntries };
}
/**
* The subset of HocuspocusProvider this module depends on, so the provider can
* be replaced with a fake in unit tests (there is no server in the test env).
*/
export interface CollabProviderLike {
synced: boolean;
unsyncedChanges: number;
destroy(): void;
on(event: "unsyncedChanges", handler: (data: { number: number }) => void): void;
off(event: "unsyncedChanges", handler: (data: { number: number }) => void): void;
}
/** The configuration object passed to the provider factory. */
export interface CollabProviderConfig {
url: string;
name: string;
document: Y.Doc;
token: string;
WebSocketPolyfill: unknown;
onConnect: () => void;
onSynced: () => void;
onDisconnect: () => void;
onClose: () => void;
onAuthenticationFailed: () => void;
}
export type CollabProviderFactory = (
config: CollabProviderConfig,
) => CollabProviderLike;
const defaultProviderFactory: CollabProviderFactory = (config) =>
// @ts-ignore - WebSocketPolyfill is required for the Node.js environment.
new HocuspocusProvider(config) as unknown as CollabProviderLike;
let providerFactory: CollabProviderFactory = defaultProviderFactory;
/**
* TEST SEAM: swap the provider factory (pass null to restore the real one).
* Not part of the public API used only by the unit tests, which cannot reach
* a real collaboration server.
*/
export function __setCollabProviderFactory(
factory: CollabProviderFactory | null,
): void {
providerFactory = factory ?? defaultProviderFactory;
}
/** Optional per-acquire hooks (metrics), passed through from the call site. */
export interface AcquireOptions {
/** Invoked when the initial connect handshake times out (CONNECT_TIMEOUT_MS). */
onConnectTimeout?: () => void;
}
type SessionState = "connecting" | "ready" | "dead";
/**
* One live provider + ydoc for a single (wsUrl, pageId, token) triple.
*
* Lifecycle: connecting -> ready -> dead. A session becomes `dead` on the first
* disconnect/close/auth-failure at ANY time, on an idle/eviction/max-age
* teardown, or on an explicit destroy(); death is terminal and removes the
* session from the registry so the next acquire opens a fresh one. We never use
* the provider's auto-reconnect destroying on the first disconnect closes the
* "reconnect drove unsyncedChanges to 0 without retransmitting our write" class
* of false success.
*/
export class CollabSession {
readonly key: string;
readonly pageId: string;
readonly wsUrl: string;
readonly token: string;
readonly createdAt: number;
state: SessionState = "connecting";
/**
* Set true on disconnect/close/auth-failure so a reconnect-driven
* unsyncedChanges->0 cannot be mistaken for a successful persist of our
* write (preserved verbatim from the one-shot machine).
*/
connectionLost = false;
provider: CollabProviderLike | undefined;
private readonly ydoc: Y.Doc;
private readonly cfg: SessionConfig;
/**
* Ephemeral sessions (cache disabled, MCP_COLLAB_SESSION_IDLE_MS<=0) are never
* registered and self-destroy after their single op the legacy
* provider-per-op behavior.
*/
private readonly ephemeral: boolean;
private readonly opts: AcquireOptions | undefined;
private dead = false;
private connectTimer: ReturnType<typeof setTimeout> | undefined;
private idleTimer: ReturnType<typeof setTimeout> | undefined;
private openPromise: Promise<void> | undefined;
private openResolve: (() => void) | undefined;
private openReject: ((err: Error) => void) | undefined;
private openSettled = false;
/**
* The rejector of the CURRENT in-flight mutate, if any. A disconnect/close/
* auth-failure or timeout at ANY time rejects the in-flight op through this
* with the SAME error text the one-shot machine emitted.
*/
private inflightReject: ((err: Error) => void) | undefined;
constructor(
key: string,
pageId: string,
wsUrl: string,
token: string,
cfg: SessionConfig,
ephemeral: boolean,
opts: AcquireOptions | undefined,
) {
this.key = key;
this.pageId = pageId;
this.wsUrl = wsUrl;
this.token = token;
this.cfg = cfg;
this.ephemeral = ephemeral;
this.opts = opts;
this.createdAt = Date.now();
this.ydoc = new Y.Doc();
}
/**
* A cached session may be reused only when it is fully ready, still synced,
* has not lost its connection, and has not exceeded its max age (invariant 5
* "validate on reuse" + the max-age acquire check).
*/
isReusable(): boolean {
return (
!this.dead &&
this.state === "ready" &&
!this.connectionLost &&
!!this.provider &&
this.provider.synced === true &&
Date.now() - this.createdAt < this.cfg.maxAgeMs
);
}
/**
* Connect and wait for the initial sync (onSynced) within CONNECT_TIMEOUT_MS.
* Idempotent: repeated calls return the same in-flight/settled promise.
*/
open(): Promise<void> {
if (this.openPromise) return this.openPromise;
this.openPromise = new Promise<void>((resolve, reject) => {
this.openResolve = resolve;
this.openReject = reject;
this.connectTimer = setTimeout(() => {
// The 25s connect timeout: the collab connection never became ready.
this.opts?.onConnectTimeout?.();
this.teardown(
new Error("Connection timeout to collaboration server"),
false,
);
}, CONNECT_TIMEOUT_MS);
if (process.env.DEBUG)
console.error(`Connecting to WebSocket: ${this.wsUrl}`);
this.provider = providerFactory({
url: this.wsUrl,
name: `page.${this.pageId}`,
document: this.ydoc,
token: this.token,
WebSocketPolyfill: WebSocket,
onConnect: () => {
if (process.env.DEBUG) console.error("WS Connect");
},
// An unexpected disconnect/close at ANY time (during the connect-wait,
// between edits, or during a persistence wait) makes the session dead:
// surface it now instead of hanging, reject any in-flight op with the
// same error text as the one-shot machine, and remove ourselves from
// the registry so the next acquire opens fresh. `teardown` is idempotent
// so the onClose our own destroy() triggers is a harmless no-op.
onDisconnect: () => {
if (process.env.DEBUG) console.error("WS Disconnect");
this.teardown(
new Error(
"Collaboration connection closed before the update was persisted/synced",
),
true,
);
},
onClose: () => {
if (process.env.DEBUG) console.error("WS Close");
this.teardown(
new Error(
"Collaboration connection closed before the update was persisted/synced",
),
true,
);
},
onSynced: () => {
if (this.dead || this.openSettled) return;
if (process.env.DEBUG) console.error("Connected and synced!");
if (this.connectTimer) {
clearTimeout(this.connectTimer);
this.connectTimer = undefined;
}
this.state = "ready";
this.openSettled = true;
this.openResolve?.();
},
onAuthenticationFailed: () => {
this.teardown(
new Error("Authentication failed for collaboration connection"),
true,
);
},
});
});
return this.openPromise;
}
/**
* Run one atomic read -> transform -> write against the LIVE doc and wait for
* the server to acknowledge the write.
*
* INVARIANT 1 (read->write atomicity): between `TiptapTransformer.fromYdoc`
* and `applyDocToFragment` there is NO `await`. Yjs applies remote updates
* only when the event loop yields, so this synchronous block sees a consistent
* live doc and no concurrent human edit can interleave and be clobbered
* exactly as in the one-shot onSynced code, just on a persistent provider.
*
* INVARIANT 2 (per-edit ack): after the write, resolve immediately if
* unsyncedChanges is already 0, else wait for the unsyncedChanges->0 event
* (PERSIST_TIMEOUT_MS), guarded by connectionLost so a reconnect handshake
* cannot report a false success.
*
* CONCURRENCY: not safe to invoke concurrently on ONE session the caller
* MUST serialize (hold the per-page lock), mirroring acquireCollabSession.
* The in-flight op is tracked in a single `inflightReject` field, so an
* overlapping second call would clobber the first's rejector and leave it
* hanging on disconnect. A fail-fast guard below rejects the overlap instead.
* Sequential (awaited) mutates are fine: localFinish clears inflightReject
* before the promise settles, so the guard is clear by the time the next runs.
*/
mutate(
transform: (liveDoc: any) => any | null,
): Promise<MutationResult> {
// Belt-and-suspenders (acquire already validated): refuse to write on a
// session that is not in a live, synced, ready state.
if (
this.dead ||
this.state !== "ready" ||
this.connectionLost ||
!this.provider ||
this.provider.synced !== true
) {
return Promise.reject(
new Error("Collaboration session is not in a ready state"),
);
}
// Fail-fast on concurrent use: a second overlapping mutate would overwrite
// the first's inflightReject, so a disconnect would only reject the second
// and hang the first until PERSIST_TIMEOUT_MS. Reject the overlap WITHOUT
// touching the in-flight op's state (no localFinish/teardown here).
if (this.inflightReject) {
return Promise.reject(
new Error(
"mutate already in-flight; caller must serialize (hold the page lock)",
),
);
}
return new Promise<MutationResult>((resolve, reject) => {
let settled = false;
let persistTimer: ReturnType<typeof setTimeout> | undefined;
let unsyncedHandler:
| ((data: { number: number }) => void)
| undefined;
// The verifiable result resolved on every success/abort path. Set on
// abort (no-op report) and after a real write (computed change report).
let mutationResult: MutationResult;
const localFinish = (err: Error | null, value?: MutationResult) => {
if (settled) return;
settled = true;
if (persistTimer) clearTimeout(persistTimer);
if (unsyncedHandler && this.provider) {
try {
this.provider.off("unsyncedChanges", unsyncedHandler);
} catch (e) {}
}
this.inflightReject = undefined;
if (err) reject(err);
else resolve(value as MutationResult);
// Post-settle lifecycle: an ephemeral (cache-disabled) session dies with
// its single op; a cached session that is still alive re-arms its idle
// TTL so the clock starts from the LAST op.
if (this.ephemeral) {
this.destroy("ephemeral op complete");
} else if (!this.dead) {
this.armIdle();
}
};
// Register so a disconnect/close/auth-failure/teardown rejects THIS op
// with the connection-loss error text. localFinish's `settled` guard makes
// a racing teardown + normal resolve safe (first one wins).
this.inflightReject = (e: Error) => localFinish(e);
// Resolve once the server acknowledges our update: the provider increments
// unsyncedChanges when the local update is sent and decrements it on the
// server's SyncStatus(applied=true); reaching 0 means the authoritative
// in-memory ydoc on the server now contains our write.
const waitForPersistence = () => {
if (settled) return;
// A missing provider is a failure, not a success: without it the write
// can never have been acknowledged.
if (!this.provider) {
localFinish(new Error("collab provider gone before persistence"));
return;
}
if (this.provider.unsyncedChanges === 0) {
localFinish(null, mutationResult);
return;
}
persistTimer = setTimeout(() => {
localFinish(
new Error(
"Timeout waiting for collaboration server to persist the update",
),
);
}, PERSIST_TIMEOUT_MS);
unsyncedHandler = (data: { number: number }) => {
// Only treat unsyncedChanges->0 as success when the connection is
// still up. A transient disconnect + reconnect handshake can drive the
// counter back to 0 without our write being re-transmitted; in that
// case let the disconnect/close error win instead.
if (data.number === 0 && !this.connectionLost) {
localFinish(null, mutationResult);
}
};
this.provider.on("unsyncedChanges", unsyncedHandler);
};
// CRITICAL: everything between reading the live doc and writing it back
// must stay synchronous (no await). While the JS event loop is not
// yielded, no incoming remote update can interleave, so any already-synced
// concurrent edits are preserved in liveDoc.
let newDoc: any;
let beforeDoc: any;
try {
let liveDoc = TiptapTransformer.fromYdoc(this.ydoc, "default");
if (
!liveDoc ||
typeof liveDoc !== "object" ||
!Array.isArray(liveDoc.content)
) {
liveDoc = { type: "doc", content: [] };
}
// Snapshot the before-doc for the change report. Docs are
// JSON-serializable, so this is a safe deep clone.
beforeDoc = JSON.parse(JSON.stringify(liveDoc));
newDoc = transform(liveDoc);
if (newDoc == null) {
// Transform aborted — write nothing, return the live doc with a no-op
// change report.
mutationResult = {
doc: liveDoc,
verify: {
changed: false,
textInserted: 0,
textDeleted: 0,
blocksChanged: 0,
marks: {},
summary: "no changes (transform aborted)",
},
};
localFinish(null, mutationResult);
return;
}
// Structural diff into the live fragment (issue #152): preserves the Yjs
// ids of unchanged nodes, so an open editor's cursor is not yanked to the
// end of the document on every agent write.
applyDocToFragment(this.ydoc, newDoc);
} catch (e) {
// Includes errors thrown by transform (e.g. "afterText not found",
// "text not found"): propagate them verbatim to the caller.
localFinish(e instanceof Error ? e : new Error(String(e)));
return;
}
// Compute the verifiable change report AFTER the transact write: it only
// needs the JSON before/after, so it cannot affect the atomic read->write
// window, and summarizeChange never throws.
mutationResult = {
doc: newDoc,
verify: summarizeChange(beforeDoc, newDoc),
};
if (process.env.DEBUG)
console.error("Content written, waiting for server to persist...");
waitForPersistence();
});
}
/** (Re)arm the idle TTL so the clock starts from the most recent activity. */
armIdle(): void {
if (this.dead || this.ephemeral) return;
if (this.idleTimer) clearTimeout(this.idleTimer);
if (this.cfg.idleMs > 0) {
this.idleTimer = setTimeout(() => {
this.destroy("idle timeout");
}, this.cfg.idleMs);
// Never let the idle timer keep the process alive.
(this.idleTimer as any).unref?.();
}
}
/**
* Idempotent teardown: mark dead, clear timers, remove from the registry, fail
* any pending open/in-flight op, and destroy the provider. `inflightError` is
* the error a pending open or in-flight op is rejected with; `connectionLoss`
* marks the session as connection-lost so the ack guard cannot report a false
* success on a racing unsyncedChanges->0.
*/
private teardown(inflightError: Error | null, connectionLoss: boolean): void {
if (this.dead) return;
this.dead = true;
this.state = "dead";
if (connectionLoss) this.connectionLost = true;
if (this.connectTimer) {
clearTimeout(this.connectTimer);
this.connectTimer = undefined;
}
if (this.idleTimer) {
clearTimeout(this.idleTimer);
this.idleTimer = undefined;
}
// Remove ourselves from the registry (only if we are still the live entry —
// a re-open under the same key must not be evicted by our teardown).
if (sessions.get(this.key) === this) {
sessions.delete(this.key);
}
// Fail a pending open() and any in-flight mutate with the terminal error.
if (!this.openSettled) {
this.openSettled = true;
this.openReject?.(
inflightError ?? new Error("Collaboration session destroyed"),
);
}
if (this.inflightReject) {
const rej = this.inflightReject;
this.inflightReject = undefined;
rej(inflightError ?? new Error("Collaboration session destroyed"));
}
if (this.provider) {
try {
this.provider.destroy();
} catch (e) {}
this.provider = undefined;
}
}
/**
* Public idempotent teardown used by the acquire/eviction paths and by a
* caller that wants the session dropped after a failed op ("next call
* reconnects fresh").
*/
destroy(reason: string): void {
if (this.dead) return;
if (process.env.DEBUG)
console.error(`Destroying collab session ${this.pageId}: ${reason}`);
this.teardown(new Error(`Collaboration session destroyed: ${reason}`), false);
}
}
/** key = wsUrl + pageId + collabToken (identity isolation: invariant 4). */
const sessions = new Map<string, CollabSession>();
function sessionKey(wsUrl: string, pageId: string, token: string): string {
// The token is part of the key so sessions are NEVER shared between different
// users' MCP sessions (HTTP mode), and a token rotation makes a new entry
// while the old one idles out.
return `${wsUrl}${pageId}${token}`;
}
/**
* Get a live, synced CollabSession for a page, reusing a cached one when it is
* still valid or opening a fresh one otherwise. Does NOT take the per-page lock
* the caller MUST already hold it (both call sites run inside withPageLock,
* which is not reentrant, so acquiring the lock here would deadlock
* mutateLiveContentUnlocked).
*/
export async function acquireCollabSession(
pageId: string,
collabToken: string,
baseUrl: string,
opts?: AcquireOptions,
): Promise<CollabSession> {
const cfg = readConfig();
const wsUrl = buildCollabWsUrl(baseUrl);
// Cache disabled (rollback path): open an unregistered ephemeral session that
// self-destroys after its single op — the exact legacy per-op-provider flow.
if (cfg.idleMs <= 0) {
const session = new CollabSession(
sessionKey(wsUrl, pageId, collabToken),
pageId,
wsUrl,
collabToken,
cfg,
true,
opts,
);
await session.open();
return session;
}
const key = sessionKey(wsUrl, pageId, collabToken);
const existing = sessions.get(key);
if (existing) {
if (existing.isReusable()) {
// Reuse. Refresh LRU order (re-insert = most recently used) and re-arm the
// idle TTL so the reuse counts as activity.
sessions.delete(key);
sessions.set(key, existing);
existing.armIdle();
if (process.env.DEBUG)
console.error(`Reusing collab session for page ${pageId}`);
return existing;
}
// Stale (not synced / past max age / lost): drop it and open fresh.
existing.destroy("stale on reuse");
}
// Enforce the registry cap before inserting: destroy-evict the least recently
// used (the first entry in insertion order) until there is room.
while (sessions.size >= cfg.maxEntries) {
const oldestKey: string | undefined = sessions.keys().next().value;
if (oldestKey === undefined) break;
const victim = sessions.get(oldestKey);
if (victim) victim.destroy("evicted (LRU cap)");
// destroy() removes it from the map; guard against a no-op destroy.
if (sessions.has(oldestKey)) sessions.delete(oldestKey);
}
const session = new CollabSession(
key,
pageId,
wsUrl,
collabToken,
cfg,
false,
opts,
);
sessions.set(key, session);
try {
await session.open();
} catch (e) {
// Failed connect/sync: make sure it is not left cached.
session.destroy("open failed");
throw e;
}
session.armIdle();
if (process.env.DEBUG)
console.error(`Opened new collab session for page ${pageId}`);
return session;
}
/**
* Destroy every cached session. Wired into the process shutdown so a hanging
* session does not keep a doc loaded on the server past exit.
*/
export function destroyAllSessions(): void {
for (const session of [...sessions.values()]) {
session.destroy("process shutdown");
}
sessions.clear();
}
/** TEST-ONLY: number of currently cached sessions. */
export function __sessionCountForTests(): number {
return sessions.size;
}
+31 -211
View File
@@ -1,4 +1,3 @@
import { HocuspocusProvider } from "@hocuspocus/provider";
import { TiptapTransformer } from "@hocuspocus/transformer";
import * as Y from "yjs";
import WebSocket from "ws";
@@ -16,7 +15,9 @@ import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import { sanitizeForYjs, findUnstorableAttr } from "./node-ops.js";
import { canonicalizeFootnotes } from "./footnote-canonicalize.js";
import { summarizeChange, VerifyReport } from "./diff.js";
import { normalizeAndMergeFootnotes } from "./footnote-normalize-merge.js";
import { VerifyReport } from "./diff.js";
import { acquireCollabSession } from "./collab-session.js";
export { markdownToProseMirror };
@@ -82,7 +83,12 @@ global.WebSocket = WebSocket;
export async function markdownToProseMirrorCanonical(
markdownContent: string,
): Promise<any> {
return canonicalizeFootnotes(await markdownToProseMirror(markdownContent));
// #419: normalize + merge glyph-forked footnote definitions BEFORE
// canonicalizing, so the canonicalizer re-hangs references and drops the
// now-orphaned duplicate definitions.
return canonicalizeFootnotes(
normalizeAndMergeFootnotes(await markdownToProseMirror(markdownContent)),
);
}
/**
@@ -194,26 +200,27 @@ export function assertYjsEncodable(doc: any): void {
}
}
/** Time we wait for the initial handshake/sync before giving up. */
const CONNECT_TIMEOUT_MS = 25000;
/** Time we wait for the server to acknowledge our write before giving up. */
const PERSIST_TIMEOUT_MS = 20000;
/**
* Safely mutate the live content of a page over the collaboration websocket.
*
* This is the single safe write path for every MCP content mutation. It:
* 1. serializes per-page writes through withPageLock (no two MCP writes on
* the same page overlap);
* 2. connects to Hocuspocus and waits for the initial sync so the local ydoc
* mirrors the authoritative server doc INCLUDING edits/comments/images
* that are not yet in the debounced REST snapshot;
* 3. inside onSynced, SYNCHRONOUSLY reads the live doc, runs `transform`, and
* writes the result back with no `await` between read and write so no
* remote update can interleave and clobber concurrent human edits;
* 2. acquires a LIVE, synced CollabSession for the page (issue #400) a
* cached provider whose local ydoc mirrors the authoritative server doc
* (INCLUDING edits/comments/images not yet in the debounced REST snapshot),
* reused across a series of edits instead of a fresh connect/auth/sync per
* call;
* 3. SYNCHRONOUSLY reads the live doc, runs `transform`, and writes the result
* back with no `await` between read and write so no remote update can
* interleave and clobber concurrent human edits (CollabSession.mutate);
* 4. waits for the server to acknowledge the write (unsyncedChanges -> 0)
* before resolving, so the next operation observes our change.
*
* On any mutate failure the session is destroyed so the next call reconnects
* fresh; the page lock is held for the whole acquire+mutate so the session's
* synchronous read->write window never overlaps another MCP write on the page.
*
* `transform` receives the live ProseMirror doc and returns the NEW full
* ProseMirror doc to write, or `null` to abort with no write (a no-op). If
* `transform` throws, the error is propagated to the caller (not swallowed).
@@ -230,7 +237,7 @@ export async function mutatePageContent(
baseUrl: string,
transform: (liveDoc: any) => any | null,
): Promise<MutationResult> {
return withPageLock(pageId, () => {
return withPageLock(pageId, async () => {
if (process.env.DEBUG) {
console.error(`Starting realtime content mutate for page ${pageId}`);
// Token prefix is sensitive; only log it under DEBUG.
@@ -239,202 +246,15 @@ export async function mutatePageContent(
);
}
const ydoc = new Y.Doc();
const wsUrl = buildCollabWsUrl(baseUrl);
if (process.env.DEBUG) console.error(`Connecting to WebSocket: ${wsUrl}`);
return new Promise<MutationResult>((resolve, reject) => {
let provider: HocuspocusProvider | undefined;
let applied = false; // onSynced may fire again on reconnect — apply once.
let settled = false;
// Set true on disconnect/close so a reconnect-driven unsyncedChanges->0
// cannot be mistaken for a successful persist of our write.
let connectionLost = false;
let connectTimer: ReturnType<typeof setTimeout> | undefined;
let persistTimer: ReturnType<typeof setTimeout> | undefined;
let unsyncedHandler: ((data: { number: number }) => void) | undefined;
const cleanup = () => {
if (connectTimer) clearTimeout(connectTimer);
if (persistTimer) clearTimeout(persistTimer);
if (provider) {
if (unsyncedHandler) {
try {
provider.off("unsyncedChanges", unsyncedHandler);
} catch (err) {}
}
try {
provider.destroy();
} catch (err) {}
}
};
const finish = (err: Error | null, value?: MutationResult) => {
if (settled) return;
settled = true;
cleanup();
if (err) reject(err);
else resolve(value as MutationResult);
};
connectTimer = setTimeout(() => {
finish(new Error("Connection timeout to collaboration server"));
}, CONNECT_TIMEOUT_MS);
// Resolve once the server has acknowledged our update. The provider
// increments unsyncedChanges when our local update is sent and
// decrements it when the server replies with a SyncStatus(applied=true);
// reaching 0 means the authoritative in-memory ydoc on the server now
// contains our write.
const waitForPersistence = () => {
if (settled) return;
// A missing provider is a failure, not a success: without it the write
// can never have been acknowledged. Only an actual unsyncedChanges===0
// on a live provider counts as persisted.
if (!provider) {
finish(new Error("collab provider gone before persistence"));
return;
}
if (provider.unsyncedChanges === 0) {
finish(null, mutationResult);
return;
}
persistTimer = setTimeout(() => {
finish(
new Error(
"Timeout waiting for collaboration server to persist the update",
),
);
}, PERSIST_TIMEOUT_MS);
unsyncedHandler = (data: { number: number }) => {
// Only treat unsyncedChanges->0 as success when the connection is
// still up. A transient disconnect + reconnect handshake can drive
// the counter back to 0 without our write being re-transmitted; in
// that case let the disconnect/close error win instead.
if (data.number === 0 && !connectionLost) {
finish(null, mutationResult);
}
};
provider.on("unsyncedChanges", unsyncedHandler);
};
// The verifiable result resolved on every success/abort path. Set on
// abort (no-op report) and after a real write (computed change report).
let mutationResult: MutationResult;
provider = new HocuspocusProvider({
url: wsUrl,
name: `page.${pageId}`,
document: ydoc,
token: collabToken,
// @ts-ignore - Required for Node.js environment
WebSocketPolyfill: WebSocket,
onConnect: () => {
if (process.env.DEBUG) console.error("WS Connect");
},
// An unexpected disconnect/close while we are still waiting (during the
// connect-wait before onSynced, or during the persistence wait after the
// write) means the update will never be acknowledged — surface it now
// instead of hanging until the connect/persist timeout fires. `finish`
// is idempotent via the `settled` flag, so the onClose that our own
// cleanup()->provider.destroy() triggers (after settled=true is set) is
// a harmless no-op and cannot cause a double-resolve.
onDisconnect: () => {
if (process.env.DEBUG) console.error("WS Disconnect");
// Mark BEFORE finish so the unsyncedChanges handler (if it races)
// sees the connection as lost and won't report a false success.
connectionLost = true;
finish(
new Error(
"Collaboration connection closed before the update was persisted/synced",
),
);
},
onClose: () => {
if (process.env.DEBUG) console.error("WS Close");
// Mark BEFORE finish so the unsyncedChanges handler (if it races)
// sees the connection as lost and won't report a false success.
connectionLost = true;
finish(
new Error(
"Collaboration connection closed before the update was persisted/synced",
),
);
},
onSynced: () => {
if (applied || settled) return;
applied = true;
if (process.env.DEBUG) console.error("Connected and synced!");
// CRITICAL: everything between reading the live doc and writing it
// back must stay synchronous (no await). While the JS event loop is
// not yielded, no incoming remote update can interleave, so any
// already-synced concurrent edits are preserved in liveDoc.
let newDoc: any;
let beforeDoc: any;
try {
let liveDoc = TiptapTransformer.fromYdoc(ydoc, "default");
if (
!liveDoc ||
typeof liveDoc !== "object" ||
!Array.isArray(liveDoc.content)
) {
liveDoc = { type: "doc", content: [] };
}
// Snapshot the before-doc for the change report. Docs are
// JSON-serializable, so this is a safe deep clone.
beforeDoc = JSON.parse(JSON.stringify(liveDoc));
newDoc = transform(liveDoc);
if (newDoc == null) {
// Transform aborted — write nothing, return the live doc with a
// no-op change report.
mutationResult = {
doc: liveDoc,
verify: {
changed: false,
textInserted: 0,
textDeleted: 0,
blocksChanged: 0,
marks: {},
summary: "no changes (transform aborted)",
},
};
finish(null, mutationResult);
return;
}
// Structural diff into the live fragment (issue #152): preserves
// the Yjs ids of unchanged nodes, so an open editor's cursor is not
// yanked to the end of the document on every agent write.
applyDocToFragment(ydoc, newDoc);
} catch (e) {
// Includes errors thrown by transform (e.g. "afterText not found",
// "text not found"): propagate them verbatim to the caller.
finish(e instanceof Error ? e : new Error(String(e)));
return;
}
// Compute the verifiable change report AFTER the transact write: it
// only needs the JSON before/after, so it cannot affect the atomic
// read->write window, and summarizeChange never throws.
mutationResult = {
doc: newDoc,
verify: summarizeChange(beforeDoc, newDoc),
};
if (process.env.DEBUG)
console.error("Content written, waiting for server to persist...");
waitForPersistence();
},
onAuthenticationFailed: () => {
finish(
new Error("Authentication failed for collaboration connection"),
);
},
});
});
const session = await acquireCollabSession(pageId, collabToken, baseUrl);
try {
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh (this
// also closes the "reconnect drove the counter to 0" false-success class).
session.destroy("mutate failed");
throw e;
}
});
}
+84 -10
View File
@@ -17,8 +17,23 @@
* comparing and match across maximal runs of consecutive text nodes within a
* single block, while mapping every normalized character back to its raw index
* so the mark lands on the exact original characters.
*
* MARKDOWN-STRIP FALLBACK: when the agent copies a selection that still carries
* inline markdown (`**bold**`, `` `code` ``, `[t](u)`), the raw locator will not
* match the document's plain text. Exactly like edit_page_text's json-edit
* fallback, we first try the verbatim selection and, ONLY if it anchors nowhere
* in the whole document, retry with `stripInlineMarkdown` applied. `canAnchorInDoc`,
* `getAnchoredText` and `applyAnchorInDoc` share this decision via
* `resolveAnchorSelection`. `countAnchorMatches` keeps its OWN parallel exact-wins
* implementation (it needs a raw match COUNT, not a single resolved locator), kept
* deliberately in sync with `resolveAnchorSelection`: raw match use raw, else fall
* back to the stripped count. All four therefore agree on which locator matched
* the suggestion-uniqueness gate depends on count and can/get never disagreeing, so
* these two exact-wins implementations MUST stay in sync if either is changed.
*/
import { stripInlineMarkdown } from "./text-normalize.js";
/** Typographic double-quote variants mapped to ASCII `"`. */
const DOUBLE_QUOTES = "«»„“”‟〝〞"";
/** Typographic single-quote/apostrophe variants mapped to ASCII `'`. */
@@ -214,15 +229,17 @@ function reconstructRawText(blockContent: any[], match: AnchorMatch): string {
* un-appliable (spurious 409).
*/
export function getAnchoredText(doc: any, selection: string): string | null {
const { selection: effective, found } = resolveAnchorSelection(doc, selection);
if (!found) return null;
const visit = (node: any, depth: number): string | null => {
if (depth > MAX_DEPTH || !node || typeof node !== "object") return null;
if (!Array.isArray(node.content)) return null;
const match = findAnchorInBlock(node.content, selection);
const match = findAnchorInBlock(node.content, effective);
if (match) return reconstructRawText(node.content, match);
for (const child of node.content) {
if (child && typeof child === "object" && Array.isArray(child.content)) {
const found = visit(child, depth + 1);
if (found !== null) return found;
const foundText = visit(child, depth + 1);
if (foundText !== null) return foundText;
}
}
return null;
@@ -231,12 +248,11 @@ export function getAnchoredText(doc: any, selection: string): string | null {
}
/**
* Depth-first, document-order check for whether `selection` can be anchored
* anywhere in `doc`. At each node with an array `content`, first try to match
* within that node's own content, then recurse into children that themselves
* have a `content` array.
* RAW (no markdown-strip fallback) depth-first check that `selection` anchors
* somewhere in `doc`. This is the primitive `resolveAnchorSelection` builds on;
* public callers should use `canAnchorInDoc`, which adds the strip fallback.
*/
export function canAnchorInDoc(doc: any, selection: string): boolean {
function rawCanAnchorInDoc(doc: any, selection: string): boolean {
const visit = (node: any, depth: number): boolean => {
if (depth > MAX_DEPTH || !node || typeof node !== "object") return false;
if (!Array.isArray(node.content)) return false;
@@ -251,6 +267,43 @@ export function canAnchorInDoc(doc: any, selection: string): boolean {
return visit(doc, 0);
}
/**
* Decide the locator that ACTUALLY anchors `selection` in `doc`, applying the
* markdown-strip fallback once (so every public entry point agrees):
* - EXACT WINS: if the verbatim selection anchors anywhere, use it as-is.
* - FALLBACK: only if the verbatim selection anchors nowhere, and the
* markdown-stripped form differs and DOES anchor, use the stripped form and
* flag `normalized` so callers can surface a soft warning.
* - otherwise `found` is false and `selection` is returned unchanged.
*
* The stripped form is used ONLY to LOCATE the anchor; getAnchoredText still
* reconstructs and stores the RAW document substring, so the strip never leaks
* into what gets persisted.
*/
export function resolveAnchorSelection(
doc: any,
selection: string,
): { selection: string; found: boolean; normalized: boolean } {
if (rawCanAnchorInDoc(doc, selection)) {
return { selection, found: true, normalized: false };
}
const stripped = stripInlineMarkdown(selection);
if (stripped !== selection && rawCanAnchorInDoc(doc, stripped)) {
return { selection: stripped, found: true, normalized: true };
}
return { selection, found: false, normalized: false };
}
/**
* Depth-first, document-order check for whether `selection` can be anchored
* anywhere in `doc` (with the markdown-strip fallback). At each node with an
* array `content`, first try to match within that node's own content, then
* recurse into children that themselves have a `content` array.
*/
export function canAnchorInDoc(doc: any, selection: string): boolean {
return resolveAnchorSelection(doc, selection).found;
}
/**
* Split the matched text nodes and splice the comment mark across the range.
* `blockContent` is mutated IN PLACE. `match.startChild..endChild` are all text
@@ -315,7 +368,7 @@ function spliceCommentMark(
* not use this. (Note: counts OCCURRENCES, not just matching blocks, so two
* occurrences inside one block are correctly reported as 2.)
*/
export function countAnchorMatches(doc: any, selection: string): number {
function rawCountAnchorMatches(doc: any, selection: string): number {
const normSel = normalizeForMatch(selection).norm.trim();
if (normSel.length === 0) return 0;
@@ -369,6 +422,25 @@ export function countAnchorMatches(doc: any, selection: string): number {
return total;
}
/**
* Uniqueness gate for suggestions, with the SAME markdown-strip fallback as the
* other entry points so count never disagrees with can/get/apply. EXACT WINS: if
* the verbatim selection occurs at all, return its raw occurrence count (so a
* selection that is unique raw stays unique the fallback never runs and cannot
* introduce a spurious second match). Only when the verbatim selection is absent
* do we count occurrences of the markdown-stripped form.
*/
export function countAnchorMatches(doc: any, selection: string): number {
const raw = rawCountAnchorMatches(doc, selection);
if (raw > 0) return raw;
const stripped = stripInlineMarkdown(selection);
if (stripped !== selection) {
const strippedCount = rawCountAnchorMatches(doc, stripped);
if (strippedCount > 0) return strippedCount;
}
return 0;
}
/**
* Depth-first (same order as canAnchorInDoc) over `doc`; on the FIRST block
* whose content matches `selection`, splice the comment mark across the matched
@@ -380,10 +452,12 @@ export function applyAnchorInDoc(
selection: string,
commentId: string,
): boolean {
const { selection: effective, found } = resolveAnchorSelection(doc, selection);
if (!found) return false;
const visit = (node: any, depth: number): boolean => {
if (depth > MAX_DEPTH || !node || typeof node !== "object") return false;
if (!Array.isArray(node.content)) return false;
const match = findAnchorInBlock(node.content, selection);
const match = findAnchorInBlock(node.content, effective);
if (match) {
spliceCommentMark(node.content, match, commentId);
return true;
+193
View File
@@ -0,0 +1,193 @@
// Pure-TS schematic SVG preview for draw.io diagrams (issue #423, stage 1).
//
// HARD CONSTRAINT: no backend rendering. This is a dependency-free string
// builder — given the parsed mxGraph cells it draws a rough schematic (rects,
// ellipses, diamonds, edges + labels) that stands in as the diagram's visible
// image UNTIL a human first opens it in the draw.io editor and saves, at which
// point the client replaces this with the pixel-perfect export SVG. It is
// deliberately approximate: it exists so a freshly-agent-created diagram is not
// an empty box in the page.
import type { DrawioCell, DrawioBBox } from "./drawio-xml.js";
import { absolutePos } from "./drawio-xml.js";
function esc(s: string): string {
return s
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;");
}
/**
* Strip HTML markup from a cell value (draw.io labels are HTML when html=1),
* decode the handful of entities we care about, and collapse whitespace so the
* label fits on the schematic. `<br>` becomes a space (this is a one-line
* preview label, not a faithful multi-line render).
*/
function labelText(value: string): string {
return value
.replace(/<br\s*\/?>/gi, " ")
.replace(/<[^>]+>/g, "")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/&#39;|&apos;/g, "'")
.replace(/&#xa;|&#10;/gi, " ")
.replace(/&amp;/g, "&")
.replace(/\s+/g, " ")
.trim();
}
function centeredLabel(cx: number, cy: number, value: string, color = "#000000"): string {
const text = labelText(value);
if (!text) return "";
return (
`<text x="${round(cx)}" y="${round(cy)}" ` +
`font-family="Helvetica, Arial, sans-serif" font-size="12" ` +
`text-anchor="middle" dominant-baseline="middle" fill="${esc(color)}">` +
`${esc(text)}</text>`
);
}
function round(n: number): number {
return Math.round(n * 100) / 100;
}
interface ShapeKind {
kind: "ellipse" | "rhombus" | "triangle" | "rect";
}
/**
* Decide which schematic primitive to draw for a vertex. A shape can be named
* either as the style's base token (e.g. "ellipse;…") or as a key (e.g.
* "shape=rhombus" / "ellipse=1"), so both the base style and the map are
* checked.
*/
function shapeKind(
styleMap: Record<string, string>,
baseStyle?: string,
): ShapeKind {
const shape = styleMap.shape ?? baseStyle;
const has = (name: string) => shape === name || styleMap[name] != null;
if (has("ellipse")) return { kind: "ellipse" };
if (has("rhombus")) return { kind: "rhombus" };
if (has("triangle")) return { kind: "triangle" };
// Everything else — including unknown stencils (shape=mxgraph.*), swimlanes,
// and plain boxes — is drawn as a (rounded) rectangle.
return { kind: "rect" };
}
function fill(styleMap: Record<string, string>): string {
const c = styleMap.fillColor;
if (!c || c.toLowerCase() === "none") return "#ffffff";
return c;
}
function stroke(styleMap: Record<string, string>): string {
const c = styleMap.strokeColor;
if (!c || c.toLowerCase() === "none") return "#000000";
return c;
}
/**
* Render the schematic shapes as the INNER content of the `.drawio.svg` (the
* outer <svg> wrapper is added by drawio-xml.buildDrawioSvg). Coordinates are
* absolute (container children are resolved via the parent chain).
*/
export function renderDiagramShapes(cells: DrawioCell[], _bbox: DrawioBBox): string {
const byId = new Map(cells.map((c) => [c.id, c]));
const parts: string[] = [];
// Edges first so vertices sit on top of their connectors.
for (const c of cells) {
if (!c.edge) continue;
parts.push(renderEdge(c, byId));
}
for (const c of cells) {
if (!c.vertex || !c.geometry.hasGeometry) continue;
const g = c.geometry;
if (g.width == null || g.height == null) continue;
const { x, y } = absolutePos(c, byId);
parts.push(renderVertex(c, x, y, g.width, g.height));
}
return `<g>${parts.filter(Boolean).join("")}</g>`;
}
function renderVertex(
c: DrawioCell,
x: number,
y: number,
w: number,
h: number,
): string {
const f = esc(fill(c.styleMap));
const s = esc(stroke(c.styleMap));
const { kind } = shapeKind(c.styleMap, c.baseStyle);
const cx = x + w / 2;
const cy = y + h / 2;
let shape = "";
switch (kind) {
case "ellipse":
shape =
`<ellipse cx="${round(cx)}" cy="${round(cy)}" rx="${round(w / 2)}" ` +
`ry="${round(h / 2)}" fill="${f}" stroke="${s}"/>`;
break;
case "rhombus": {
const pts = [
`${round(cx)},${round(y)}`,
`${round(x + w)},${round(cy)}`,
`${round(cx)},${round(y + h)}`,
`${round(x)},${round(cy)}`,
].join(" ");
shape = `<polygon points="${pts}" fill="${f}" stroke="${s}"/>`;
break;
}
case "triangle": {
const pts = [
`${round(x)},${round(y)}`,
`${round(x + w)},${round(cy)}`,
`${round(x)},${round(y + h)}`,
].join(" ");
shape = `<polygon points="${pts}" fill="${f}" stroke="${s}"/>`;
break;
}
default: {
const rounded = c.styleMap.rounded === "1";
const rx = rounded ? Math.min(12, w / 2, h / 2) : 0;
shape =
`<rect x="${round(x)}" y="${round(y)}" width="${round(w)}" ` +
`height="${round(h)}" rx="${round(rx)}" ry="${round(rx)}" ` +
`fill="${f}" stroke="${s}"/>`;
}
}
return shape + centeredLabel(cx, cy, c.value, c.styleMap.fontColor || "#000000");
}
function renderEdge(c: DrawioCell, byId: Map<string, DrawioCell>): string {
const src = c.source != null ? byId.get(c.source) : undefined;
const tgt = c.target != null ? byId.get(c.target) : undefined;
const p1 = anchorPoint(src, byId);
const p2 = anchorPoint(tgt, byId);
if (!p1 || !p2) return ""; // a floating endpoint with no fixed point: skip
const line =
`<line x1="${round(p1.x)}" y1="${round(p1.y)}" ` +
`x2="${round(p2.x)}" y2="${round(p2.y)}" ` +
`stroke="#000000" stroke-width="1"/>`;
const mid = { x: (p1.x + p2.x) / 2, y: (p1.y + p2.y) / 2 };
return line + centeredLabel(mid.x, mid.y, c.value);
}
/** Center point of a vertex used as an edge anchor (approximate). */
function anchorPoint(
cell: DrawioCell | undefined,
byId: Map<string, DrawioCell>,
): { x: number; y: number } | null {
if (!cell || !cell.geometry.hasGeometry) return null;
const g = cell.geometry;
if (g.width == null || g.height == null) return null;
const { x, y } = absolutePos(cell, byId);
return { x: x + g.width / 2, y: y + g.height / 2 };
}
+771
View File
@@ -0,0 +1,771 @@
// draw.io (mxGraph) XML support for the MCP drawio tools (issue #423, stage 1).
//
// This module owns everything that is pure data-plumbing for draw.io diagrams:
// - the DECODE CHAIN that turns a stored `diagram.drawio.svg` attachment back
// into mxGraph XML (handles both the plain nested-XML form Docmost writes
// and draw.io's own COMPRESSED `<diagram>` payload — base64 + raw-deflate);
// - the ENCODE side that wraps mxGraph XML into the `.drawio.svg` attachment
// using the exact same contract as the import service's createDrawioSvg;
// - a deterministic LINTER that rejects the structural mistakes generators
// make before anything is written (each violation carries the offending
// cellId + position so the model can auto-retry);
// - a stable HASH over the normalized XML, used as the optimistic-lock key.
//
// HARD CONSTRAINT: no backend rendering. Nothing here shells out or renders a
// bitmap; the only runtime dependencies are jsdom (already used across this
// package for XML parsing) and pako (raw-inflate for the compressed format).
import { createHash } from "node:crypto";
import { JSDOM } from "jsdom";
import pako from "pako";
// --- shared XML parser -----------------------------------------------------
// A single reusable JSDOM window; constructing one per parse is wasteful and
// these tools are low-frequency. Only the DOMParser is used.
let _window: any = null;
function xmlWindow(): any {
if (!_window) _window = new JSDOM("").window;
return _window;
}
/** Default mxGraphModel attributes used when the server wraps a cell list. */
const DEFAULT_MODEL_ATTRS =
'dx="0" dy="0" grid="1" gridSize="10" page="1" pageWidth="850" pageHeight="1100"';
// --- structured lint errors ------------------------------------------------
export interface DrawioLintIssue {
/** Machine-readable rule id, e.g. "edge-geometry". */
rule: string;
/** Human-readable explanation the model can act on. */
message: string;
/** The offending cell's id, when the rule is cell-scoped. */
cellId?: string;
/** Extra location info: cell index in <root>, or a parser line:col. */
position?: string;
}
/**
* Thrown by the linter and by decode/prepare when the input is unusable. Carries
* the full list of issues so the caller can surface a structured tool-error the
* model auto-retries against.
*/
export class DrawioLintError extends Error {
issues: DrawioLintIssue[];
constructor(issues: DrawioLintIssue[]) {
const summary = issues
.map((i) => {
const where = [
i.cellId != null ? `cellId=${i.cellId}` : null,
i.position != null ? `at ${i.position}` : null,
]
.filter(Boolean)
.join(", ");
return `[${i.rule}] ${i.message}${where ? ` (${where})` : ""}`;
})
.join("; ");
super(`drawio lint failed: ${summary}`);
this.name = "DrawioLintError";
this.issues = issues;
}
}
// --- parsed-cell model -----------------------------------------------------
export interface DrawioGeometry {
x?: number;
y?: number;
width?: number;
height?: number;
relative: boolean;
hasGeometry: boolean;
}
export interface DrawioCell {
id: string;
parent?: string;
source?: string;
target?: string;
vertex: boolean;
edge: boolean;
value: string;
style: string;
styleMap: Record<string, string>;
/** Non-key/value leading token of the style (a base stylename), if any. */
baseStyle?: string;
geometry: DrawioGeometry;
}
export interface DrawioBBox {
width: number;
height: number;
}
// --- style parsing ---------------------------------------------------------
/**
* Parse a draw.io style string into { baseStyle, map }. Grammar:
* [stylename;]key=value;key=value;...
* A single leading token without '=' is the base stylename (e.g. "text" or
* "ellipse"). Every other non-empty segment must be exactly one key=value pair.
* Returns `null` (the segment index) on the first malformed segment so the
* linter can report a precise error.
*/
export function parseStyle(
style: string,
): { baseStyle?: string; map: Record<string, string>; badSegment?: string } {
const map: Record<string, string> = {};
let baseStyle: string | undefined;
const segments = style.split(";");
for (let i = 0; i < segments.length; i++) {
const seg = segments[i].trim();
if (seg === "") continue; // trailing/empty segments are fine
const eq = seg.indexOf("=");
if (eq === -1) {
// A bare token is only valid as the FIRST meaningful segment (base style).
if (baseStyle === undefined && Object.keys(map).length === 0) {
baseStyle = seg;
continue;
}
return { baseStyle, map, badSegment: seg };
}
// A second '=' inside the same segment is malformed.
if (seg.indexOf("=", eq + 1) !== -1) {
return { baseStyle, map, badSegment: seg };
}
const key = seg.slice(0, eq).trim();
const val = seg.slice(eq + 1).trim();
if (key === "") return { baseStyle, map, badSegment: seg };
map[key] = val;
}
return { baseStyle, map };
}
// --- low-level XML helpers -------------------------------------------------
function parseXml(xml: string): { doc: any; error: string | null } {
const parser = new (xmlWindow().DOMParser)();
const doc = parser.parseFromString(xml, "application/xml");
const err = doc.getElementsByTagName("parsererror");
if (err.length > 0) {
// jsdom prefixes the message with "line:col:" — keep it as the position.
return { doc, error: (err[0].textContent || "malformed XML").trim() };
}
return { doc, error: null };
}
function num(v: string | null): number | undefined {
if (v == null || v === "") return undefined;
const n = Number(v);
return Number.isFinite(n) ? n : undefined;
}
/** Extract the raw `<mxGraphModel …>…</mxGraphModel>` substring, or null. */
function sliceModel(xml: string): string | null {
const open = xml.indexOf("<mxGraphModel");
if (open === -1) return null;
const close = xml.indexOf("</mxGraphModel>", open);
if (close === -1) {
// Self-closed empty model, e.g. `<mxGraphModel .../>`.
const selfClose = xml.indexOf("/>", open);
if (selfClose !== -1) return xml.slice(open, selfClose + 2);
return null;
}
return xml.slice(open, close + "</mxGraphModel>".length);
}
// --- decode chain ----------------------------------------------------------
/**
* Read the `content=` attribute out of a `.drawio.svg` string. Docmost stores a
* base64 payload there (createDrawioSvg); draw.io's own SVG export may store the
* XML entity-encoded instead. The DOM decodes entities for us, so the caller
* only has to distinguish "starts with '<'" (raw XML) from base64.
*/
export function extractContentAttr(svg: string): string {
const { doc, error } = parseXml(svg);
if (!error) {
const root = doc.documentElement;
if (root && root.hasAttribute && root.hasAttribute("content")) {
return root.getAttribute("content") || "";
}
}
// Fallback for a malformed wrapper: pull the attribute directly. The content
// value itself never contains a double-quote (base64 / entity-encoded XML).
const m = /content="([^"]*)"/.exec(svg);
if (m) {
// Decode the handful of XML entities a raw regex would leave encoded.
return m[1]
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/&#39;/g, "'")
.replace(/&amp;/g, "&");
}
throw new Error("drawio: SVG has no content= attribute to decode");
}
/**
* Turn a decoded draw.io file (`<mxfile>` or a bare `<mxGraphModel>`, possibly
* with a COMPRESSED `<diagram>` payload) into the mxGraphModel XML. For the
* plain form the raw substring is returned verbatim so a round-trip stays
* byte-stable; the compressed form is inflated (base64 raw-deflate
* decodeURIComponent), which is how draw.io stores diagrams by default.
*/
export function decodeDrawioFileToModel(fileXml: string): string {
// Plain, nested XML: return the model substring untouched (byte-stable).
const sliced = sliceModel(fileXml);
if (sliced) return sliced;
// Otherwise it must be the compressed `<diagram>…</diagram>` text payload.
const open = fileXml.indexOf("<diagram");
if (open !== -1) {
const gt = fileXml.indexOf(">", open);
const close = fileXml.indexOf("</diagram>", gt);
if (gt !== -1 && close !== -1) {
const payload = fileXml.slice(gt + 1, close).trim();
if (payload) {
const inflated = inflateDiagramPayload(payload);
const model = sliceModel(inflated);
if (model) return model;
return inflated;
}
}
}
throw new Error(
"drawio: could not decode file — no <mxGraphModel> and no compressed <diagram> payload",
);
}
/**
* Upper bound on the inflated size of a compressed `<diagram>` payload
* (decompression-bomb guard). `fetchInternalFile` caps the DOWNLOAD at 64 MiB,
* but a tiny crafted compressed payload can inflate to gigabytes and OOM the
* process. A real diagram's mxGraphModel XML is small (KBs to low MBs even for
* large diagrams), so 16 MiB is far above any legitimate payload while keeping
* memory bounded. Chars ~= bytes for the (mostly ASCII) URI-encoded XML.
*/
export const MAX_INFLATED_DIAGRAM_BYTES = 16 * 1024 * 1024;
/**
* Inflate draw.io's compressed diagram payload:
* base64-decode raw-inflate (raw deflate, windowBits -15)
* decodeURIComponent.
*
* Uses pako's streaming Inflate so we can abort as soon as the decompressed
* output exceeds MAX_INFLATED_DIAGRAM_BYTES the full bomb is never
* materialised in memory.
*/
export function inflateDiagramPayload(base64: string): string {
const bytes = Buffer.from(base64, "base64");
const inflator = new pako.Inflate({ raw: true, to: "string" });
let total = 0;
const passthrough = inflator.onData.bind(inflator);
inflator.onData = (chunk: string | Uint8Array) => {
total += chunk.length;
if (total > MAX_INFLATED_DIAGRAM_BYTES) {
// Throwing here propagates out of push(), aborting inflation immediately.
throw new Error(
`drawio: refusing to decode diagram — decompressed size exceeds ` +
`${MAX_INFLATED_DIAGRAM_BYTES} bytes (possible decompression bomb)`,
);
}
passthrough(chunk);
};
inflator.push(bytes, true);
if (inflator.err) {
throw new Error(
`drawio: failed to inflate compressed <diagram> payload (${inflator.msg || inflator.err})`,
);
}
const uriEncoded = inflator.result as string;
return decodeURIComponent(uriEncoded);
}
/** Full decode chain: `.drawio.svg` string → mxGraphModel XML. */
export function decodeDrawioSvg(svg: string): string {
const content = extractContentAttr(svg).trim();
const fileXml = content.startsWith("<")
? content
: Buffer.from(content, "base64").toString("utf-8");
return decodeDrawioFileToModel(fileXml);
}
// --- encode side -----------------------------------------------------------
/**
* Wrap an mxGraphModel in the plain (uncompressed) `<mxfile><diagram>` envelope.
* draw.io opens uncompressed XML fine, and staying uncompressed keeps the
* write path deterministic and the round-trip byte-stable.
*/
export function encodeDrawioFile(modelXml: string, title = "Page-1"): string {
const safeTitle = xmlEscape(title);
return `<mxfile host="drawio"><diagram id="page-1" name="${safeTitle}">${modelXml}</diagram></mxfile>`;
}
/**
* Build the `diagram.drawio.svg` attachment. Mirrors the import service's
* createDrawioSvg contract exactly:
* <svg xmlns= xmlns:xlink= content="${base64(drawioFile)}">${inner}</svg>
* plus width/height/viewBox from the diagram bounding box and the schematic
* preview as the visible children (`inner`).
*/
export function buildDrawioSvg(
modelXml: string,
inner: string,
bbox: DrawioBBox,
title = "Page-1",
): string {
const file = encodeDrawioFile(modelXml, title);
const base64 = Buffer.from(file, "utf-8").toString("base64");
const w = Math.max(1, Math.round(bbox.width));
const h = Math.max(1, Math.round(bbox.height));
return (
`<svg xmlns="http://www.w3.org/2000/svg" ` +
`xmlns:xlink="http://www.w3.org/1999/xlink" ` +
`width="${w}" height="${h}" viewBox="0 0 ${w} ${h}" ` +
`content="${base64}">${inner}</svg>`
);
}
function xmlEscape(s: string): string {
return s
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;");
}
// --- normalization + hash --------------------------------------------------
/**
* Normalize mxGraph XML for hashing / stable comparison: drop the whitespace
* between tags and trim. This is intentionally conservative it never reorders
* attributes or cells (that would be lossy) so two documents hash equal iff
* they differ only in inter-tag formatting.
*/
export function normalizeXml(xml: string): string {
return xml.replace(/>\s+</g, "><").trim();
}
/** Stable optimistic-lock hash over the normalized model XML (sha256, hex). */
export function mxHash(modelXml: string): string {
return createHash("sha256").update(normalizeXml(modelXml), "utf-8").digest("hex");
}
// --- cell parsing ----------------------------------------------------------
/** Parse every `<mxCell>` in a model into a structured DrawioCell list. */
export function parseCells(modelXml: string): DrawioCell[] {
const { doc, error } = parseXml(modelXml);
if (error) {
throw new DrawioLintError([
{ rule: "well-formed-xml", message: error, position: firstLineCol(error) },
]);
}
const cells: DrawioCell[] = [];
const els = doc.getElementsByTagName("mxCell");
for (let i = 0; i < els.length; i++) {
cells.push(readCell(els[i]));
}
return cells;
}
function readCell(el: any): DrawioCell {
const style = el.getAttribute("style") || "";
const parsed = parseStyle(style);
const geoEl = firstChildByTag(el, "mxGeometry");
const geometry: DrawioGeometry = geoEl
? {
x: num(geoEl.getAttribute("x")),
y: num(geoEl.getAttribute("y")),
width: num(geoEl.getAttribute("width")),
height: num(geoEl.getAttribute("height")),
relative: geoEl.getAttribute("relative") === "1",
hasGeometry: true,
}
: { relative: false, hasGeometry: false };
return {
id: el.getAttribute("id") ?? "",
parent: el.getAttribute("parent") ?? undefined,
source: el.getAttribute("source") ?? undefined,
target: el.getAttribute("target") ?? undefined,
vertex: el.getAttribute("vertex") === "1",
edge: el.getAttribute("edge") === "1",
value: el.getAttribute("value") ?? "",
style,
styleMap: parsed.map,
baseStyle: parsed.baseStyle,
geometry,
};
}
function firstChildByTag(el: any, tag: string): any {
for (let i = 0; i < el.childNodes.length; i++) {
const c = el.childNodes[i];
if (c.nodeType === 1 && c.tagName === tag) return c;
}
return null;
}
function firstLineCol(msg: string): string | undefined {
const m = /^(\d+:\d+)/.exec(msg);
return m ? m[1] : undefined;
}
// --- bounding box ----------------------------------------------------------
/**
* Absolute bounding box of the diagram from its vertex geometries. Container
* children are relative, so absolute positions are resolved along the parent
* chain before taking the extent. Falls back to a default canvas when empty.
*/
export function computeBBox(cells: DrawioCell[]): DrawioBBox {
const byId = new Map(cells.map((c) => [c.id, c]));
let maxX = 0;
let maxY = 0;
let any = false;
for (const c of cells) {
if (!c.vertex || !c.geometry.hasGeometry) continue;
const g = c.geometry;
if (g.width == null || g.height == null) continue;
const { x, y } = absolutePos(c, byId);
maxX = Math.max(maxX, x + g.width);
maxY = Math.max(maxY, y + g.height);
any = true;
}
if (!any) return { width: 300, height: 200 };
// A small margin so borders/labels are not clipped at the edge.
return { width: Math.ceil(maxX) + 20, height: Math.ceil(maxY) + 20 };
}
/** Absolute (x,y) of a vertex, following its parent chain (containers). */
export function absolutePos(
cell: DrawioCell,
byId: Map<string, DrawioCell>,
): { x: number; y: number } {
let x = cell.geometry.x ?? 0;
let y = cell.geometry.y ?? 0;
const seen = new Set<string>([cell.id]);
let parentId = cell.parent;
while (parentId && !seen.has(parentId)) {
seen.add(parentId);
const p = byId.get(parentId);
// Sentinels (0/1) carry no geometry; stop there.
if (!p || !p.vertex || !p.geometry.hasGeometry) break;
x += p.geometry.x ?? 0;
y += p.geometry.y ?? 0;
parentId = p.parent;
}
return { x, y };
}
// --- linter ----------------------------------------------------------------
/**
* Run every deterministic pre-write rule over a full mxGraphModel string. On any
* violation it throws a DrawioLintError carrying one issue per violation, each
* with the offending cellId + position. Returns the parsed cells on success.
*/
export function lintModel(modelXml: string): {
cells: DrawioCell[];
warnings: string[];
} {
const issues: DrawioLintIssue[] = [];
const warnings: string[] = [];
// Rule: no XML comments. Checked on the raw string (a comment survives DOM
// parsing as a comment node, but the intent is to reject them outright — they
// routinely wrap "TODO" cruft that breaks downstream tooling).
if (modelXml.includes("<!--")) {
issues.push({
rule: "no-comments",
message: "XML comments (<!-- -->) are not allowed in diagram XML",
});
}
// Rule: value escaping + literal newline. Scan raw <mxCell> tags so the error
// can name the cell id even when the whole document is otherwise malformed.
scanRawValues(modelXml, issues);
// Well-formedness — everything below needs a parsed DOM.
const { doc, error } = parseXml(modelXml);
if (error) {
issues.push({
rule: "well-formed-xml",
message: error,
position: firstLineCol(error),
});
throw new DrawioLintError(issues);
}
const root = doc.documentElement;
if (!root || root.tagName !== "mxGraphModel") {
issues.push({
rule: "structure",
message: `root element must be <mxGraphModel>, got <${root ? root.tagName : "?"}>`,
});
throw new DrawioLintError(issues);
}
if (!firstChildByTag(root, "root")) {
issues.push({
rule: "structure",
message: "<mxGraphModel> must contain a <root> element",
});
throw new DrawioLintError(issues);
}
const cells = parseCells(modelXml);
const ids = new Set<string>();
// Rule: sentinel cells id="0" and id="1"(parent="0").
const cell0 = cells.find((c) => c.id === "0");
const cell1 = cells.find((c) => c.id === "1");
if (!cell0) {
issues.push({
rule: "sentinel-cells",
message: 'missing the root sentinel cell <mxCell id="0"/>',
cellId: "0",
});
}
if (!cell1) {
issues.push({
rule: "sentinel-cells",
message: 'missing the layer sentinel cell <mxCell id="1" parent="0"/>',
cellId: "1",
});
} else if (cell1.parent !== "0") {
issues.push({
rule: "sentinel-cells",
message: 'the layer sentinel <mxCell id="1"> must have parent="0"',
cellId: "1",
});
}
cells.forEach((c, index) => {
const pos = `cell #${index}`;
const isSentinel = c.id === "0" || c.id === "1";
// Rule: unique, non-empty ids; user cells must not reuse 0/1.
if (c.id === "") {
issues.push({ rule: "cell-id", message: "cell has an empty id", position: pos });
} else if (ids.has(c.id)) {
issues.push({
rule: "duplicate-id",
message: `duplicate cell id "${c.id}"`,
cellId: c.id,
position: pos,
});
}
ids.add(c.id);
if (isSentinel) return; // sentinels are exempt from the shape rules below
// Rule: vertex XOR edge (a cell may be neither: groups/containers).
if (c.vertex && c.edge) {
issues.push({
rule: "vertex-edge-exclusive",
message: 'a cell cannot be both vertex="1" and edge="1"',
cellId: c.id,
position: pos,
});
}
// Rule: every edge has a child <mxGeometry as="geometry"/>.
if (c.edge && !c.geometry.hasGeometry) {
issues.push({
rule: "edge-geometry",
message:
'edge is missing its child <mxGeometry relative="1" as="geometry"/> — it will not render',
cellId: c.id,
position: pos,
});
}
// Rule: edge endpoints resolve to existing ids.
if (c.edge) {
for (const end of ["source", "target"] as const) {
const ref = c[end];
if (ref != null && ref !== "" && !cellExists(cells, ref)) {
issues.push({
rule: "edge-endpoint",
message: `edge ${end} "${ref}" does not resolve to any cell`,
cellId: c.id,
position: pos,
});
}
}
}
// Rule: parent must exist.
if (c.parent != null && c.parent !== "" && !cellExists(cells, c.parent)) {
issues.push({
rule: "parent-exists",
message: `parent "${c.parent}" does not resolve to any cell`,
cellId: c.id,
position: pos,
});
}
// Rule: style parses as key=value; pairs.
if (c.style !== "") {
const parsed = parseStyle(c.style);
if (parsed.badSegment !== undefined) {
issues.push({
rule: "style-format",
message: `malformed style segment "${parsed.badSegment}" (expected key=value)`,
cellId: c.id,
position: pos,
});
}
}
});
if (issues.length > 0) throw new DrawioLintError(issues);
return { cells, warnings };
}
function cellExists(cells: DrawioCell[], id: string): boolean {
return cells.some((c) => c.id === id);
}
/**
* Raw-string scan of every `value="…"`/`value='…'` on an mxCell tag. Catches an
* unescaped `&`/`<`/`>` and a literal newline character inside a value, keyed to
* the cell's id. Runs before DOM parsing so a value bug is reported with its
* cellId even when the document is otherwise malformed.
*/
function scanRawValues(xml: string, issues: DrawioLintIssue[]): void {
const tagRe = /<mxCell\b([^>]*?)\/?>/g;
let m: RegExpExecArray | null;
while ((m = tagRe.exec(xml)) !== null) {
const attrs = m[1];
const idM = /\bid\s*=\s*"([^"]*)"/.exec(attrs);
const cellId = idM ? idM[1] : undefined;
const valM = /\bvalue\s*=\s*"([^"]*)"/.exec(attrs) || /\bvalue\s*=\s*'([^']*)'/.exec(attrs);
if (!valM) continue;
const raw = valM[1];
// Literal newline (0x0A / 0x0D) inside the attribute value.
if (/[\n\r]/.test(raw)) {
issues.push({
rule: "value-newline",
message:
"value contains a literal newline; use &#xa; (or <br> with html=1) instead",
cellId,
});
}
// Unescaped '<' or '>' inside a value.
if (raw.includes("<") || raw.includes(">")) {
issues.push({
rule: "value-escaping",
message: "value contains an unescaped '<' or '>'; use &lt; / &gt;",
cellId,
});
}
// '&' that does not begin a valid entity.
const badAmp = /&(?!(amp|lt|gt|quot|apos|#[0-9]+|#x[0-9a-fA-F]+);)/.test(raw);
if (badAmp) {
issues.push({
rule: "value-escaping",
message: "value contains an unescaped '&'; use &amp;",
cellId,
});
}
}
}
// --- input normalization + prepare -----------------------------------------
/**
* Normalize an accepted tool input into a full mxGraphModel string:
* - a bare `<mxGraphModel>` is used as-is;
* - an `<mxfile>` is decoded to its first page's model;
* - a list of `<mxCell>` is wrapped with the mxGraphModel/root envelope and
* the sentinel cells (id=0, id=1 parent=0) are added when absent.
*/
export function normalizeInput(inputXml: string): string {
let xml = inputXml.trim();
// Strip an optional XML prolog.
if (xml.startsWith("<?xml")) {
const end = xml.indexOf("?>");
if (end !== -1) xml = xml.slice(end + 2).trim();
}
if (xml.startsWith("<mxfile")) {
return decodeDrawioFileToModel(xml);
}
if (xml.startsWith("<mxGraphModel")) {
return xml;
}
if (xml.includes("<mxCell")) {
return wrapCellFragment(xml);
}
throw new DrawioLintError([
{
rule: "unrecognized-input",
message:
"input must be a <mxGraphModel>, an <mxfile>, or a list of <mxCell> elements",
},
]);
}
function wrapCellFragment(fragment: string): string {
// Validate the fragment is well-formed (wrapped so a bare list parses) and
// discover which sentinels are already present.
const { doc, error } = parseXml(`<root>${fragment}</root>`);
if (error) {
throw new DrawioLintError([
{
rule: "well-formed-xml",
message: error,
position: firstLineCol(error),
},
]);
}
const existing = new Set<string>();
const els = doc.getElementsByTagName("mxCell");
for (let i = 0; i < els.length; i++) {
existing.add(els[i].getAttribute("id") ?? "");
}
let prefix = "";
if (!existing.has("0")) prefix += '<mxCell id="0"/>';
if (!existing.has("1")) prefix += '<mxCell id="1" parent="0"/>';
return `<mxGraphModel ${DEFAULT_MODEL_ATTRS}><root>${prefix}${fragment}</root></mxGraphModel>`;
}
export interface PreparedModel {
/** Canonical (normalized) mxGraphModel XML that gets written. */
modelXml: string;
cells: DrawioCell[];
bbox: DrawioBBox;
/** Number of user cells (excludes the id=0/id=1 sentinels). */
cellCount: number;
warnings: string[];
hash: string;
}
/**
* Full pre-write pipeline for create/update: normalize the input into a model,
* lint it (throws DrawioLintError on any violation), then compute the canonical
* form, bounding box, cell count and hash. Never touches the network.
*/
export function prepareModel(inputXml: string): PreparedModel {
const rawModel = normalizeInput(inputXml);
const { cells, warnings } = lintModel(rawModel);
const modelXml = normalizeXml(rawModel);
const bbox = computeBBox(cells);
const cellCount = cells.filter((c) => c.id !== "0" && c.id !== "1").length;
return {
modelXml,
cells,
bbox,
cellCount,
warnings,
hash: mxHash(modelXml),
};
}
/** Cell count of a decoded model (user cells only) — used by drawio_get meta. */
export function countUserCells(modelXml: string): number {
return parseCells(modelXml).filter((c) => c.id !== "0" && c.id !== "1").length;
}
@@ -0,0 +1,280 @@
/**
* Deterministic server-side NORMALIZATION + MERGE of footnote DEFINITIONS
* (MCP, PURE).
*
* Problem (#419): footnotes with the same meaning but different GLYPHS
* typographic quotes («»/) vs ASCII "…", em/en-dash vs `-`, non-breaking
* space vs normal space, differing space counts are not recognized as equal
* and "fork": two definitions appear where the author meant one. The existing
* de-dup paths miss this: `footnoteContentKey` (footnote-authoring.ts) only
* collapses ASCII whitespace (quotes/dashes/NBSP untouched), and
* `canonicalizeFootnotes` keys purely by `attrs.id` (the two forks have
* different ids), so neither glues the forks together.
*
* This pass fixes that DETERMINISTICALLY on the MCP write-paths (an LLM
* instruction gives no glue guarantee). It:
* 1. Normalizes the TEXT of every `footnoteDefinition`'s text nodes IN PLACE
* (typographic quotes -> ASCII "/', dashes -> `-`, NBSP & friends ->
* normal space, whitespace runs collapsed, whole-definition edges
* trimmed) unconditionally, for ALL definitions, KEEPING their marks.
* 2. Computes a MERGE KEY per definition (normalized text + an ATTRS-AWARE
* inline-mark signature, via the local `footnoteMergeKey`), so notes that
* read the same but differ in formatting (bold vs plain) OR in a mark
* attribute (a `link` with a different `href`, differing `code`/`highlight`
* attrs) are NOT merged. See `footnoteMergeKey` for why this diverges from
* the shared type-only `footnoteContentKey`.
* 3. Maps every duplicate definition id to the FIRST (document-order)
* definition's id and re-hangs `footnoteReference` nodes onto it.
*
* Duplicate definitions keep their original ids but now have NO references, so
* the canonicalizer that runs immediately after this pass removes them as
* orphans and derives the single tail list + numbering. This pass therefore
* MUST run BEFORE `canonicalizeFootnotes(doc)` at every write-path call-site
* (see the enforcement rule in `footnote-canonicalize.ts`).
*
* Accepted tradeoff: the exact typographic glyphs of the SURVIVING footnote are
* rewritten to ASCII, in exchange for a GUARANTEED merge. Scope is strictly
* INSIDE `footnoteDefinition` body text (normal paragraphs) is never touched.
*
* Pure: deep-clones its input, deterministic, idempotent (a re-run is a no-op
* text is already normalized and references already point at the canonical id,
* so no spurious mutations / git-sync churn).
*/
const FOOTNOTE_DEFINITION_NAME = "footnoteDefinition";
const FOOTNOTE_REFERENCE_NAME = "footnoteReference";
/**
* Typographic glyph maps. DUPLICATED from `comment-anchor.ts` (the source of
* truth, `normalizeForMatch`) on purpose: those constants are private there and
* bound to that module's anchor-matching golden tests, so extracting them would
* risk changing anchor behaviour. Keeping a local copy makes this pass fully
* self-contained. If the anchor maps grow, mirror the change here.
*/
/** Typographic double-quote variants mapped to ASCII `"`. */
const DOUBLE_QUOTES = "«»„“”‟〝〞"";
/** Typographic single-quote/apostrophe variants mapped to ASCII `'`. */
const SINGLE_QUOTES = "‘’‚‛";
/** Dash variants mapped to ASCII `-`. */
const DASHES = "–—―−‐‑‒";
function cloneJson<T>(v: T): T {
if (typeof structuredClone === "function") return structuredClone(v);
return JSON.parse(JSON.stringify(v)) as T;
}
/**
* True for any character we collapse/replace with a single normal space.
* Mirrors `comment-anchor.ts`'s `isWhitespaceChar`: ASCII whitespace (`\s`
* covers tab/newline) plus the non-breaking / special spaces listed explicitly
* for determinism across engines.
*/
function isWhitespaceChar(ch: string): boolean {
return (
/\s/.test(ch) ||
ch === " " || // no-break space
ch === " " || // figure space
ch === " " || // narrow no-break space
ch === " " || // thin space
ch === " " || // hair space
ch === " " || // en space
ch === " " // em space
);
}
/**
* Map typographic quotes/dashes to ASCII and collapse every whitespace run
* (including NBSP & friends) to a SINGLE normal space. Does NOT trim the
* whole-definition edge trim is applied separately so inter-node spacing across
* a multi-text-node definition is preserved.
*/
function normalizeAndCollapse(s: string): string {
let out = "";
let i = 0;
while (i < s.length) {
const ch = s[i];
if (isWhitespaceChar(ch)) {
while (i < s.length && isWhitespaceChar(s[i])) i++;
out += " ";
continue;
}
let mapped = ch;
if (DOUBLE_QUOTES.indexOf(ch) !== -1) mapped = '"';
else if (SINGLE_QUOTES.indexOf(ch) !== -1) mapped = "'";
else if (DASHES.indexOf(ch) !== -1) mapped = "-";
out += mapped;
i++;
}
return out;
}
/** Collect every text node inside `def`, in document order (deep). */
function collectTextNodes(node: any, out: any[]): void {
if (!node || typeof node !== "object") return;
if (node.type === "text" && typeof node.text === "string") out.push(node);
if (Array.isArray(node.content)) {
for (const child of node.content) collectTextNodes(child, out);
}
}
/** Collect every `footnoteDefinition` node in document order (deep). */
function collectDefinitions(node: any, out: any[]): void {
if (!node || typeof node !== "object") return;
if (node.type === FOOTNOTE_DEFINITION_NAME) out.push(node);
if (Array.isArray(node.content)) {
for (const child of node.content) collectDefinitions(child, out);
}
}
/**
* Normalize the text of one definition's text nodes IN PLACE: map glyphs +
* collapse whitespace on every node (marks untouched), then trim the leading
* edge of the first text node and the trailing edge of the last so the
* definition as a whole is trimmed WITHOUT dropping the spacing between two
* adjacent text nodes. The edge trims are guarded so an all-whitespace edge
* node is never emptied into a schema-invalid empty text node.
*/
function normalizeDefinitionText(def: any): void {
const textNodes: any[] = [];
collectTextNodes(def, textNodes);
for (const t of textNodes) {
// Skip text carrying a `code` mark: inline code is a verbatim literal, not
// prose typography. Rewriting quotes/dashes/special-spaces there would
// corrupt the literal's meaning (a string literal, an em-dash flag, i18n).
// Leaving it untouched also makes it contribute its RAW text to
// `footnoteMergeKey`, so two notes differing only by glyphs inside code
// stay distinct (while prose glyph-forks still merge). See #419.
if ((t.marks || []).some((m: any) => m?.type === "code")) continue;
t.text = normalizeAndCollapse(t.text);
}
if (textNodes.length === 0) return;
const hasCodeMark = (t: any): boolean =>
(t.marks || []).some((m: any) => m?.type === "code");
const first = textNodes[0];
if (!hasCodeMark(first)) {
const startTrimmed = first.text.replace(/^ +/, "");
if (startTrimmed !== "") first.text = startTrimmed;
}
const last = textNodes[textNodes.length - 1];
if (!hasCodeMark(last)) {
const endTrimmed = last.text.replace(/ +$/, "");
if (endTrimmed !== "") last.text = endTrimmed;
}
}
/** Rewrite `footnoteReference` ids IN PLACE using `defIdToCanon` (deep). */
function rehangReferences(
node: any,
defIdToCanon: Map<string, string>,
): void {
if (!node || typeof node !== "object") return;
if (node.type === FOOTNOTE_REFERENCE_NAME) {
const id = node?.attrs?.id;
if (typeof id === "string") {
const canon = defIdToCanon.get(id);
if (canon && canon !== id) node.attrs.id = canon;
}
}
if (Array.isArray(node.content)) {
for (const child of node.content) rehangReferences(child, defIdToCanon);
}
}
/**
* Stable, order-independent serialization of a mark's `attrs`: sort keys so the
* same attrs always yield the same string regardless of authoring order. Empty /
* missing attrs -> "" (so an attr-less mark keys identically to a type-only mark
* signature, preserving bold-vs-plain parity).
*/
function stableAttrs(attrs: any): string {
if (!attrs || typeof attrs !== "object") return "";
const sorted: Record<string, any> = {};
for (const k of Object.keys(attrs).sort()) sorted[k] = attrs[k];
return JSON.stringify(sorted);
}
/**
* ATTRS-AWARE merge key for a footnote definition. Deliberately DIVERGES from
* the shared `footnoteContentKey` (footnote-authoring.ts): that key's mark
* signature is TYPE-ONLY (`m.type`), so two definitions with identical visible
* text but marks differing only in ATTRIBUTES most importantly a `link` with a
* different `href` (footnotes are usually citations/links), also `code` /
* `highlight` with differing attrs collapse to the SAME key and get merged;
* one definition then loses its references and the canonicalizer deletes it as an
* orphan, silently dropping a distinct link target (data loss, #419).
*
* This key folds each mark's `attrs` (stable, sorted-key serialization) into the
* signature, so different-href / different-attr notes stay separate. We do NOT
* change `footnoteContentKey` itself: it is shared with the live
* `insertInlineFootnote` / `commentsToFootnotes` dedup and altering it there
* would change their behaviour out of scope here.
*
* The TEXT portion mirrors `footnoteContentKey` exactly (per text node
* `text + mark-signature`, concatenated, whitespace-collapsed, trimmed) over the
* already-in-place-normalized text, so empty text still yields "" (empties never
* collapse) and merge parity with the rest of the pass is preserved.
*/
function footnoteMergeKey(defNode: any): string {
const parts: string[] = [];
const visit = (n: any): void => {
if (!n || typeof n !== "object") return;
if (n.type === "text" && typeof n.text === "string") {
const marks = Array.isArray(n.marks)
? n.marks
.filter((m: any) => m && m.type)
.map((m: any) => `${m.type}${stableAttrs(m.attrs)}`)
.sort()
.join(",")
: "";
parts.push(`${n.text}${marks}`);
}
if (Array.isArray(n.content)) for (const c of n.content) visit(c);
};
visit(defNode);
return parts
.join("")
.replace(/[ \t\r\n]+/g, " ")
.trim();
}
/**
* Normalize footnote-definition text and merge definitions whose normalized
* text (+ mark signature) matches. See the file header for the full contract.
* Pure (deep-clones input, deterministic, idempotent). Intended to run
* immediately BEFORE `canonicalizeFootnotes(doc)`.
*/
export function normalizeAndMergeFootnotes<T = any>(doc: T): T {
if (doc == null || typeof doc !== "object") return doc;
const out = cloneJson(doc) as any;
// 1) All definitions in document order; normalize each one's text in place.
const defNodes: any[] = [];
collectDefinitions(out, defNodes);
for (const def of defNodes) normalizeDefinitionText(def);
// 2) Merge key per definition (normalized text + inline-mark signature). The
// first definition in document order per key wins; later ones map onto it.
// Empty-text definitions (key === "") are NOT merged — otherwise every
// empty footnote would collapse into one (parity with insertInlineFootnote).
const keyToCanon = new Map<string, string>();
const defIdToCanon = new Map<string, string>();
for (const def of defNodes) {
const id = def?.attrs?.id;
if (typeof id !== "string" || id === "") continue;
const key = footnoteMergeKey(def);
if (key === "") continue;
const canon = keyToCanon.get(key);
if (canon === undefined) {
keyToCanon.set(key, id);
} else if (canon !== id) {
defIdToCanon.set(id, canon);
}
}
// 3) Re-hang references from duplicate ids onto the canonical id. Duplicate
// definitions keep their ids but now have no references -> the following
// canonicalizer pass drops them as orphans.
if (defIdToCanon.size > 0) rehangReferences(out, defIdToCanon);
return out;
}
+23 -24
View File
@@ -12,7 +12,11 @@
* re-import for small wording fixes.
*/
import { stripInlineMarkdown, stripBalancedWrappers } from "./text-normalize.js";
import {
stripInlineMarkdown,
stripBalancedWrappers,
closestBlockHint,
} from "./text-normalize.js";
export interface TextEdit {
find: string;
@@ -305,6 +309,21 @@ export function applyTextEdits(
continue;
}
// HARD-REFUSE inline footnote tokens (#410). `^[...]` in a `replace` is
// markdown that only becomes a real footnote when a whole markdown body is
// written (create_page / update_page_content / import_page_markdown). Written
// through edit_page_text it stays a LITERAL string in the text — the exact
// failure mode #410 fixes — so refuse it here (defense-in-depth) and point the
// caller at insert_footnote, mirroring the formatting-marker refusal above.
if (/\^\[[\s\S]*?\]/.test(edit.replace)) {
failed.push({
find: edit.find,
reason:
"edit_page_text writes the replacement as LITERAL text, so a `^[...]` footnote token does not parse into a real footnote (it would appear verbatim in the page). To add a footnote to existing text, use insert_footnote (anchorText = where, text = the note).",
});
continue;
}
// Gather every inline block in document order (recurse the whole tree so
// nested containers — callouts, list items, table cells, blockquotes — are
// all covered).
@@ -366,29 +385,9 @@ export function applyTextEdits(
} else {
// Append a bounded "closest text" hint: find the FIRST block that
// contains the longest whitespace-delimited token (>= 3 chars) of the
// (stripped, then raw) locator, and quote that block's plain text.
reason = "text not found in the document.";
const tokenSource = stripped.length > 0 ? stripped : edit.find;
const longestToken = tokenSource
.split(/\s+/)
.filter((t) => t.length >= 3)
.sort((a, b) => b.length - a.length)[0];
if (longestToken) {
const hitBlock = blockPlain.find((plain) =>
plain.includes(longestToken),
);
if (hitBlock) {
// Truncate by code point (spread iterates by code point) so a
// surrogate pair is never split; append the ellipsis only when the
// text was actually longer than the limit.
const points = [...hitBlock];
const snippet =
points.length > 120
? points.slice(0, 120).join("") + "…"
: hitBlock;
reason += ` Closest block text: "${snippet}".`;
}
}
// (stripped, then raw) locator, and quote that block's plain text. Shared
// with create_comment via closestBlockHint so both give the same hint.
reason = "text not found in the document." + closestBlockHint(blockPlain, edit.find);
}
failed.push({ find: edit.find, reason });
continue;
+61
View File
@@ -0,0 +1,61 @@
// Minimal ambient type declaration for `pako` (no @types/pako is installed and
// pako 2.x ships no bundled .d.ts). We only use the raw-deflate codec to read
// draw.io's compressed `<diagram>` payload, so declare just that surface.
declare module "pako" {
interface RawOptions {
/** When "string", the result is returned as a (binary/UTF-8) string. */
to?: "string";
/** Raw-deflate window bits; draw.io uses raw deflate (no zlib header). */
windowBits?: number;
level?: number;
}
/** Raw-inflate (windowBits: -15). `to:"string"` yields a string. */
export function inflateRaw(
data: Uint8Array | ArrayBuffer | number[],
options: RawOptions & { to: "string" },
): string;
export function inflateRaw(
data: Uint8Array | ArrayBuffer | number[],
options?: RawOptions,
): Uint8Array;
/** Raw-deflate (windowBits: -15). Used only by tests to build fixtures. */
export function deflateRaw(
data: Uint8Array | string,
options?: RawOptions,
): Uint8Array;
interface InflateStreamOptions {
to?: "string";
windowBits?: number;
/** Raw deflate (no zlib header) — equivalent to windowBits: -15. */
raw?: boolean;
chunkSize?: number;
}
/**
* Streaming inflate. We use it to bound the decompressed size: `onData` is
* invoked per output chunk, letting us abort a decompression bomb before the
* full output is materialised.
*/
export class Inflate {
constructor(options?: InflateStreamOptions);
onData: (chunk: string | Uint8Array) => void;
onEnd: (status: number) => void;
push(
data: Uint8Array | ArrayBuffer | number[] | string,
flushMode?: boolean | number,
): boolean;
result: string | Uint8Array;
err: number;
msg: string;
}
const _default: {
inflateRaw: typeof inflateRaw;
deflateRaw: typeof deflateRaw;
Inflate: typeof Inflate;
};
export default _default;
}
+34
View File
@@ -114,3 +114,37 @@ export function stripInlineMarkdown(s: string): string {
return out;
}
/**
* Build a bounded "closest text" hint for an anchor/find MISS, shared by
* edit_page_text (json-edit) and create_comment (client) so both surface the
* same self-correction affordance.
*
* Take the longest whitespace-delimited token (>= 3 chars) of the locator
* (markdown-stripped first, so `**bold**` contributes `bold`), find the FIRST
* of `blockTexts` that contains it, and return ` Closest block text: "…".` with
* the block quoted (truncated to 120 code points + ellipsis). Returns "" when
* no token qualifies or no block contains it, so the caller can append it
* unconditionally.
*/
export function closestBlockHint(
blockTexts: string[],
locator: string,
): string {
if (typeof locator !== "string" || locator.length === 0) return "";
const stripped = stripInlineMarkdown(locator);
const tokenSource = stripped.length > 0 ? stripped : locator;
const longestToken = tokenSource
.split(/\s+/)
.filter((t) => t.length >= 3)
.sort((a, b) => b.length - a.length)[0];
if (!longestToken) return "";
const hitBlock = blockTexts.find((plain) => plain.includes(longestToken));
if (!hitBlock) return "";
// Truncate by code point (spread iterates by code point) so a surrogate pair
// is never split; append the ellipsis only when the text was actually longer.
const points = [...hitBlock];
const snippet =
points.length > 120 ? points.slice(0, 120).join("") + "…" : hitBlock;
return ` Closest block text: "${snippet}".`;
}
+3
View File
@@ -16,6 +16,7 @@
import { blockPlainText } from "./node-ops.js";
import { canonicalizeFootnotes } from "./footnote-canonicalize.js";
import { normalizeAndMergeFootnotes } from "./footnote-normalize-merge.js";
import {
footnoteContentKey,
makeFootnoteDefinition,
@@ -766,6 +767,8 @@ export function insertInlineFootnote(
appendDefinition(working, makeFootnoteDefinition(footnoteId, inline));
}
// #419: normalize + merge glyph-forked definitions before canonicalizing.
working = normalizeAndMergeFootnotes(working);
// Derive numbering + the single bottom list deterministically.
working = canonicalizeFootnotes(working);
return { doc: working, inserted: true, footnoteId, reused };
+15
View File
@@ -1,6 +1,7 @@
#!/usr/bin/env node
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { createDocmostMcpServer } from "./index.js";
import { destroyAllSessions } from "./lib/collab-session.js";
// Standalone stdio entrypoint. This restores the original behavior of the
// package when run as a CLI (`docmost-mcp`): it reads credentials from the
@@ -33,6 +34,20 @@ async function run() {
console.error("Uncaught exception:", error);
});
// Teardown hook (issue #400): destroy every cached live CollabSession on exit
// so a hanging session does not keep a doc loaded on the server (which would
// also defer the server's afterUnloadDocument cleanup). `exit` runs the
// synchronous idempotent teardown; SIGINT/SIGTERM also run it, then exit.
process.on("exit", () => {
destroyAllSessions();
});
for (const signal of ["SIGINT", "SIGTERM"] as const) {
process.on(signal, () => {
destroyAllSessions();
process.exit(0);
});
}
const server = createDocmostMcpServer({
apiUrl: API_URL!,
email: EMAIL!,
+227 -3
View File
@@ -771,9 +771,13 @@ export const SHARED_TOOL_SPECS = {
'The comment is anchored inline to the given exact `selection` text ' +
'(which gets highlighted); page-level comments are NOT supported. A ' +
'new top-level comment REQUIRES a `selection`. Replies inherit the ' +
"parent's anchor and take no selection. If the call fails with a " +
'"selection not found" error, retry with a corrected EXACT selection ' +
'copied verbatim from a single paragraph/block. You may also attach a ' +
"parent's anchor and take no selection. Always COPY the `selection` " +
'VERBATIM from get_page / search_in_page output — do NOT quote it from ' +
'memory (stale-memory quoting is the top cause of anchor misses). If the ' +
'call fails with a "selection not found" error, the error quotes the ' +
"closest block text (or says the selection spans multiple blocks); retry " +
"with a corrected EXACT selection copied verbatim from a single " +
'paragraph/block. You may also attach a ' +
'`suggestedText` proposing a replacement for the `selection` (a human ' +
'applies it from the UI); when set, the `selection` must occur exactly ' +
'once in the page. Reversible via the comment UI.',
@@ -1003,4 +1007,224 @@ export const SHARED_TOOL_SPECS = {
text: z.string().describe('The new cell text.'),
}),
},
// --- footnote + image write tools (promoted from inline MCP-only, #410) ---
//
// These three were previously registered inline in index.ts as MCP-only,
// because the in-app AI-chat agent had no equivalent. #410 promotes them so the
// in-app agent (esp. the Researcher role) can attach real footnotes/images
// instead of writing literal `^[...]` / placeholder text via editPageText. The
// schema + description are MOVED VERBATIM from the old inline registrations so
// external MCP clients see identical tool names, fields and text.
insertFootnote: {
mcpName: 'insert_footnote',
inAppKey: 'insertFootnote',
description:
'Insert an AUTHOR-INLINE footnote: you specify only WHERE (anchorText) ' +
'and WHAT (text). The footnote marker is placed right after anchorText in ' +
'the body, and the bottom footnotes list + the numbering are derived ' +
'deterministically server-side. You do NOT assign a number, and you ' +
"never see or edit the footnotes list — so footnotes cannot end up out " +
"of order, orphaned, or as a raw '[^id]' block. If a footnote with the " +
'SAME text already exists, its number is REUSED (one definition, several ' +
"references). The write is atomic and won't clobber concurrent edits; if " +
'anchorText is not found, nothing is written and an error is returned.',
// CORE for the in-app agent (#410): keeping it deferred would recreate the
// original asymmetry (footnote tool hidden while editPageText is core), which
// is exactly what makes the agent fall back to literal `^[...]`.
tier: 'core',
catalogLine:
'insertFootnote — attach a numbered footnote right after a snippet of existing body text.',
buildShape: (z) => ({
pageId: z.string().min(1),
anchorText: z
.string()
.min(1)
.describe(
'A snippet of existing body text; the footnote marker is inserted ' +
'immediately after its first occurrence (mark-safe).',
),
text: z
.string()
.min(1)
.describe('The footnote content as markdown (becomes the definition).'),
}),
},
insertImage: {
mcpName: 'insert_image',
inAppKey: 'insertImage',
description:
'Download an image from a web (http/https) URL and insert it into ' +
'a page in one step. By default ' +
'appends the image at the end of the page. With replaceText, replaces the ' +
'first top-level block whose text contains that string (handy for ' +
'swapping a text placeholder like "[image: foo.png]" for the real image). ' +
'With afterText, inserts the image right after the first block containing ' +
'that string. Preserves all other block ids.',
tier: 'deferred',
catalogLine:
'insertImage — download a web image and insert it into a page.',
buildShape: (z) => ({
pageId: z.string().min(1),
imageUrl: z
.string()
.min(1)
.describe('http(s) URL of the image to download and upload'),
align: z.enum(['left', 'center', 'right']).optional(),
alt: z.string().optional(),
replaceText: z
.string()
.optional()
.describe(
'Replace the first top-level block whose text contains this string with the image',
),
afterText: z
.string()
.optional()
.describe(
'Insert the image right after the first top-level block whose text contains this string',
),
}),
},
replaceImage: {
mcpName: 'replace_image',
inAppKey: 'replaceImage',
description:
'Replace an existing image on a page with a new image fetched from a web ' +
'(http/https) URL: uploads the new file as a NEW ' +
'attachment (fresh clean URL that renders and busts browser caches), then ' +
'repoints every image node referencing the old attachmentId (recursively, ' +
'incl. callouts/tables) via the live document, preserving comments, ' +
'alignment and alt. The old attachment is left as an unreferenced orphan ' +
'(Docmost has no API to delete a single attachment; it is removed only when ' +
'the page/space is deleted). In-place byte overwrite is avoided because some ' +
'Docmost versions corrupt the attachment (HTTP 500) on overwrite.',
tier: 'deferred',
catalogLine:
'replaceImage — swap an existing page image for one fetched from a web URL.',
buildShape: (z) => ({
pageId: z.string().min(1),
attachmentId: z
.string()
.min(1)
.describe('attachmentId of the image currently in the page to replace'),
imageUrl: z
.string()
.min(1)
.describe('http(s) URL of the new image to download'),
align: z.enum(['left', 'center', 'right']).optional(),
alt: z.string().optional(),
}),
},
// --- draw.io diagrams (issue #423, stage 1) ---
drawioGet: {
mcpName: 'drawio_get',
inAppKey: 'drawioGet',
description:
'Read a draw.io diagram on a page as mxGraph XML (default) or as its raw ' +
'`.drawio.svg`. `node` is the drawio node\'s attrs.id (from get_outline / ' +
'get_page_json) or "#<index>" for a top-level block. Returns the decoded ' +
'mxGraphModel XML plus meta { attachmentId, title, width, height, ' +
'cellCount, hash }. `hash` is the optimistic-lock key you MUST pass back ' +
'as baseHash to drawio_update. Diagrams a human saved from the editor ' +
'(including draw.io\'s compressed format) decode losslessly.',
tier: 'deferred',
catalogLine:
'drawioGet — read a draw.io diagram as mxGraph XML (+ hash for updates).',
buildShape: (z) => ({
pageId: z.string().min(1),
node: z
.string()
.min(1)
.describe('The drawio node attrs.id, or "#<index>" for a top-level block.'),
format: z
.enum(['xml', 'svg'])
.optional()
.describe('"xml" (default) for mxGraph XML, or "svg" for the raw .drawio.svg.'),
}),
},
drawioCreate: {
mcpName: 'drawio_create',
inAppKey: 'drawioCreate',
description:
'Create a draw.io diagram from mxGraph XML and insert it as a diagram ' +
'block. `xml` is a bare `<mxGraphModel>` OR a list of `<mxCell>` elements ' +
'(the server wraps it and adds the id=0 / id=1 sentinel cells). The XML is ' +
'LINTED first (well-formedness, sentinel cells, unique ids, vertex XOR ' +
'edge, every edge has a child <mxGeometry as="geometry"/>, edge ' +
'source/target and every parent resolve, style parses, no XML comments, ' +
'value escaping) — a violation returns a structured error naming the rule ' +
'and cellId so you can fix and retry. `where` positions the block like ' +
'insert_node: position before/after (with exactly one of anchorNodeId or ' +
'anchorText) or append. Returns { nodeId, attachmentId, warnings }. The ' +
'returned `nodeId` is an index-based "#<index>" handle (drawio nodes carry ' +
'no attrs.id): it addresses the new top-level block and can be fed straight ' +
'back into drawio_get / drawio_update for THIS document. It is positional, ' +
'so if you add or remove blocks before it, re-resolve via get_outline. The ' +
'diagram is editable in the draw.io editor and can be re-read with ' +
'drawio_get.',
tier: 'deferred',
catalogLine:
'drawioCreate — create a draw.io diagram from mxGraph XML and insert it.',
buildShape: (z) => ({
pageId: z.string().min(1),
xml: z
.string()
.min(1)
.describe(
'mxGraph XML: a bare <mxGraphModel> or a list of <mxCell> elements.',
),
position: z
.enum(['before', 'after', 'append'])
.describe('Where to insert relative to the anchor.'),
anchorNodeId: z
.string()
.optional()
.describe('Anchor block id (for before/after).'),
anchorText: z
.string()
.optional()
.describe('Anchor text fragment (for before/after).'),
title: z.string().optional().describe('Optional diagram title.'),
}),
},
drawioUpdate: {
mcpName: 'drawio_update',
inAppKey: 'drawioUpdate',
description:
'Replace a draw.io diagram\'s content with new mxGraph XML (same lint ' +
'pipeline as drawio_create). `baseHash` is MANDATORY: pass the hash from ' +
'the drawio_get you based the edit on. If the diagram changed since ' +
'(a human or another agent edited it) the hash mismatches and the update ' +
'is refused with a conflict error — re-read with drawio_get and retry. On ' +
'success it overwrites the diagram attachment and updates the node ' +
'width/height. `node` is the drawio node attrs.id or "#<index>".',
tier: 'deferred',
catalogLine:
'drawioUpdate — replace a draw.io diagram (optimistic-locked by baseHash).',
buildShape: (z) => ({
pageId: z.string().min(1),
node: z
.string()
.min(1)
.describe('The drawio node attrs.id, or "#<index>" for a top-level block.'),
xml: z
.string()
.min(1)
.describe(
'New mxGraph XML: a bare <mxGraphModel> or a list of <mxCell> elements.',
),
baseHash: z
.string()
.min(1)
.describe('The meta.hash from the drawio_get this edit is based on.'),
}),
},
} satisfies Record<string, SharedToolSpec>;
@@ -548,3 +548,94 @@ test("suggestedText: the stored selection is the doc's RAW typographic substring
);
assert.equal(createPayload.suggestedText, "goodbye");
});
// -----------------------------------------------------------------------------
// 8) #408: a not-found selection error QUOTES the closest block text so the
// model can self-correct instead of blind-retrying.
// -----------------------------------------------------------------------------
test("a not-found selection error includes a 'Closest block text' hint", async () => {
let createCalls = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
});
return;
}
if (req.url === "/api/pages/info") {
sendJson(res, 200, {
data: {
id: "page-1",
content: {
type: "doc",
content: [
{ type: "paragraph", content: [{ type: "text", text: "The quick brown fox jumps" }] },
],
},
},
});
return;
}
if (req.url === "/api/comments/create") {
createCalls++;
sendJson(res, 200, { data: { id: "should-not-happen" } });
return;
}
sendJson(res, 404, { message: "not found" });
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
await assert.rejects(
() => client.createComment("page-1", "body", "inline", "quick brown cat"),
/Closest block text: "The quick brown fox jumps"/,
"a not-found selection must quote the closest block text",
);
assert.equal(createCalls, 0, "/comments/create must NOT be called on a miss");
});
// -----------------------------------------------------------------------------
// 9) #408: a selection that straddles two blocks gets the explicit
// "spans multiple blocks" message instead of a bare not-found.
// -----------------------------------------------------------------------------
test("a selection spanning multiple blocks gets the explicit spans-multiple-blocks message", async () => {
let createCalls = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
});
return;
}
if (req.url === "/api/pages/info") {
sendJson(res, 200, {
data: {
id: "page-1",
content: {
type: "doc",
content: [
{ type: "paragraph", content: [{ type: "text", text: "the quick brown" }] },
{ type: "paragraph", content: [{ type: "text", text: "fox jumps over" }] },
],
},
},
});
return;
}
if (req.url === "/api/comments/create") {
createCalls++;
sendJson(res, 200, { data: { id: "should-not-happen" } });
return;
}
sendJson(res, 404, { message: "not found" });
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
await assert.rejects(
() => client.createComment("page-1", "body", "inline", "brown fox"),
/spans multiple blocks/,
"a cross-block selection must report the spans-multiple-blocks hint",
);
assert.equal(createCalls, 0, "/comments/create must NOT be called on a miss");
});
@@ -0,0 +1,467 @@
// Contract tests for the drawio_get / drawio_create / drawio_update client
// methods (issue #423). Follows the repo's seam-override pattern (see
// full-doc-write-canonicalize.test.mjs): a DocmostClient subclass stubs the I/O
// seams (auth, collab token, page read, attachment upload/fetch, the mutatePage
// write) so the tool logic is exercised without a live Docmost or collab socket.
import { test } from "node:test";
import assert from "node:assert/strict";
import pako from "pako";
import { DocmostClient } from "../../build/client.js";
import {
buildDrawioSvg,
encodeDrawioFile,
normalizeXml,
mxHash,
decodeDrawioSvg,
} from "../../build/lib/drawio-xml.js";
const MODEL =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="Hi" style="rounded=1;" vertex="1" parent="1">' +
'<mxGeometry x="20" y="20" width="120" height="60" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
// Build a Docmost-style `.drawio.svg` (base64 content) for a model.
function svgFor(model) {
return buildDrawioSvg(normalizeXml(model), "<g/>", { width: 200, height: 120 });
}
// Build a human/compressed-export `.drawio.svg` (base64 content wrapping a
// compressed <diagram> payload), mimicking a diagram a person saved.
function compressedSvgFor(model) {
const compressed = Buffer.from(
pako.deflateRaw(encodeURIComponent(normalizeXml(model))),
).toString("base64");
const file = `<mxfile host="Electron"><diagram id="a" name="Page-1">${compressed}</diagram></mxfile>`;
const content = Buffer.from(file, "utf-8").toString("base64");
return `<svg xmlns="http://www.w3.org/2000/svg" content="${content}"><image href="x"/></svg>`;
}
// The vendored `drawio` node schema (diagramAttributes) declares ONLY these
// attributes; PMNode.fromJSON drops anything else on save. Mirror that here so
// the mock write path behaves like the real one — in particular, a block `id`
// set on a drawio node does NOT survive the save, so a handle keyed on it is
// un-resolvable. This is exactly what the production bug (issue #423 Fix 1) was.
const DRAWIO_SCHEMA_ATTRS = new Set([
"src",
"title",
"alt",
"width",
"height",
"size",
"aspectRatio",
"align",
"attachmentId",
]);
function applyDrawioSchemaDrop(node) {
if (!node || typeof node !== "object") return;
if (node.type === "drawio" && node.attrs && typeof node.attrs === "object") {
for (const key of Object.keys(node.attrs)) {
if (!DRAWIO_SCHEMA_ATTRS.has(key)) delete node.attrs[key];
}
}
if (Array.isArray(node.content)) for (const c of node.content) applyDrawioSchemaDrop(c);
}
function makeClient({ pageDoc, attachmentSvg } = {}) {
const calls = { uploads: [], mutations: [] };
class TestClient extends DocmostClient {
async ensureAuthenticated() {}
async getCollabTokenWithReauth() {
return "collab-token";
}
async resolvePageId(pageId) {
return `uuid-${pageId}`;
}
async getPageRaw(pageId) {
return {
id: pageId,
slugId: "s",
title: "P",
spaceId: "sp",
content: pageDoc ?? { type: "doc", content: [] },
};
}
async uploadAttachmentBuffer(pageId, buffer, fileName, mime) {
const id = `att-${calls.uploads.length + 1}`;
calls.uploads.push({ pageId, fileName, mime, svg: buffer.toString("utf-8") });
return { id, fileName, fileSize: buffer.length };
}
async fetchAttachmentText(src) {
return attachmentSvg;
}
mutatePage(pageId, token, apiUrl, transform) {
// Run the transform against a clone of the source doc, capture the result.
const clone = structuredClone(pageDoc ?? { type: "doc", content: [] });
const doc = transform(clone);
// Mirror the real schema: unknown drawio attrs (e.g. a block `id`) are
// dropped on save, so callers can never rely on them to address the node.
if (doc) applyDrawioSchemaDrop(doc);
calls.mutations.push({ pageId, doc });
return Promise.resolve({ doc, verify: { changed: doc != null } });
}
}
const client = new TestClient("http://127.0.0.1:1/api", "e@x.com", "pw");
return { client, calls };
}
function findDrawio(node, acc = []) {
if (!node || typeof node !== "object") return acc;
if (node.type === "drawio") acc.push(node);
if (Array.isArray(node.content)) for (const c of node.content) findDrawio(c, acc);
return acc;
}
// --- drawio_create ---------------------------------------------------------
test("drawio_create: lints, builds the .drawio.svg, uploads and inserts a node", async () => {
const pageDoc = {
type: "doc",
content: [{ type: "paragraph", attrs: { id: "p1" }, content: [] }],
};
const { client, calls } = makeClient({ pageDoc });
const res = await client.drawioCreate("page1", { position: "append" }, MODEL, "My diagram");
assert.equal(res.success, true);
// The returned handle is an index-based "#<index>" ref (drawio nodes carry no
// persisted attrs.id), addressing the appended top-level block (index 1, after
// the existing paragraph).
assert.equal(res.nodeId, "#1");
assert.equal(res.attachmentId, "att-1");
assert.equal(calls.uploads.length, 1);
assert.equal(calls.uploads[0].fileName, "diagram.drawio.svg");
assert.equal(calls.uploads[0].mime, "image/svg+xml");
// The uploaded SVG carries the model back (round-trips through the decode chain).
assert.equal(decodeDrawioSvg(calls.uploads[0].svg), normalizeXml(MODEL));
// A drawio node was appended with src/attachmentId/dimensions and the title.
const drawios = findDrawio(calls.mutations[0].doc);
assert.equal(drawios.length, 1);
const n = drawios[0];
// No `id` attribute is set/persisted on the node (schema has none).
assert.equal(n.attrs.id, undefined);
assert.equal(n.attrs.attachmentId, "att-1");
assert.match(n.attrs.src, /^\/api\/files\/att-1\//);
assert.ok(n.attrs.width > 0 && n.attrs.height > 0);
assert.equal(n.attrs.title, "My diagram");
});
test("drawio_create: a lint violation throws before any upload", async () => {
const { client, calls } = makeClient({ pageDoc: { type: "doc", content: [] } });
// Edge with no child geometry -> edge-geometry rule.
const bad =
'<mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'<mxCell id="3" edge="1" parent="1" source="2" target="2"/></root></mxGraphModel>';
await assert.rejects(
() => client.drawioCreate("page1", { position: "append" }, bad, undefined),
/edge-geometry/,
);
assert.equal(calls.uploads.length, 0, "no attachment uploaded on lint failure");
});
test("drawio_create: before/after requires exactly one anchor", async () => {
const { client } = makeClient({ pageDoc: { type: "doc", content: [] } });
await assert.rejects(
() => client.drawioCreate("page1", { position: "before" }, MODEL),
/exactly one of anchorNodeId or anchorText/,
);
});
// --- drawio_get ------------------------------------------------------------
test("drawio_get: decodes the model and returns meta with a hash", async () => {
const pageDoc = {
type: "doc",
content: [
{
type: "drawio",
attrs: {
id: "d1",
src: "/api/files/att-1/diagram.drawio.svg",
attachmentId: "att-1",
title: "T",
width: 200,
height: 120,
},
},
],
};
const { client } = makeClient({ pageDoc, attachmentSvg: svgFor(MODEL) });
const res = await client.drawioGet("page1", "d1", "xml");
assert.equal(res.content, normalizeXml(MODEL));
assert.equal(res.meta.attachmentId, "att-1");
assert.equal(res.meta.title, "T");
assert.equal(res.meta.cellCount, 1);
assert.equal(res.meta.hash, mxHash(normalizeXml(MODEL)));
});
test("drawio_get: format=svg returns the raw .drawio.svg", async () => {
const svg = svgFor(MODEL);
const pageDoc = {
type: "doc",
content: [
{ type: "drawio", attrs: { id: "d1", src: "/api/files/att-1/x.svg", attachmentId: "att-1" } },
],
};
const { client } = makeClient({ pageDoc, attachmentSvg: svg });
const res = await client.drawioGet("page1", "d1", "svg");
assert.equal(res.content, svg);
});
test("drawio_get: reads a HUMAN-saved compressed diagram losslessly (pako)", async () => {
const pageDoc = {
type: "doc",
content: [
{ type: "drawio", attrs: { id: "d1", src: "/api/files/att-1/x.svg", attachmentId: "att-1" } },
],
};
const { client } = makeClient({ pageDoc, attachmentSvg: compressedSvgFor(MODEL) });
const res = await client.drawioGet("page1", "d1", "xml");
assert.equal(res.content, normalizeXml(MODEL));
});
// --- drawio_update ---------------------------------------------------------
const UPDATED_MODEL =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="Changed" style="rounded=1;" vertex="1" parent="1">' +
'<mxGeometry x="20" y="20" width="300" height="200" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
function updatePageDoc() {
return {
type: "doc",
content: [
{
type: "drawio",
attrs: {
id: "d1",
src: "/api/files/att-1/diagram.drawio.svg",
attachmentId: "att-1",
width: 200,
height: 120,
},
},
],
};
}
test("drawio_update: stale baseHash -> conflict, no upload", async () => {
const { client, calls } = makeClient({
pageDoc: updatePageDoc(),
attachmentSvg: svgFor(MODEL),
});
await assert.rejects(
() => client.drawioUpdate("page1", "d1", UPDATED_MODEL, "deadbeef-stale"),
/conflict/,
);
assert.equal(calls.uploads.length, 0, "no upload on conflict");
});
test("drawio_update: current baseHash -> uploads new attachment and repoints node dims", async () => {
const currentHash = mxHash(normalizeXml(MODEL));
const { client, calls } = makeClient({
pageDoc: updatePageDoc(),
attachmentSvg: svgFor(MODEL),
});
const res = await client.drawioUpdate("page1", "d1", UPDATED_MODEL, currentHash);
assert.equal(res.success, true);
assert.equal(res.attachmentId, "att-1"); // fresh id from the stub sequence
assert.equal(calls.uploads.length, 1);
// The uploaded SVG carries the NEW model.
assert.equal(decodeDrawioSvg(calls.uploads[0].svg), normalizeXml(UPDATED_MODEL));
// The node was repointed with the new bounding-box dimensions:
// vertex maxX=320,maxY=220 + the 20px preview margin -> 340 x 240.
const n = findDrawio(calls.mutations[0].doc)[0];
assert.equal(n.attrs.attachmentId, "att-1");
assert.equal(n.attrs.width, 340);
assert.equal(n.attrs.height, 240);
// The block `id` used as the legacy resolution handle is dropped on save
// (schema declares no `id`); the update still targeted the correct node.
assert.equal(n.attrs.id, undefined);
});
test("drawio_update: baseHash is mandatory", async () => {
const { client } = makeClient({ pageDoc: updatePageDoc(), attachmentSvg: svgFor(MODEL) });
await assert.rejects(
() => client.drawioUpdate("page1", "d1", UPDATED_MODEL, ""),
/baseHash is mandatory/,
);
});
// --- Fix 1: the create handle must resolve on the SAVED doc (no id) ---------
test("drawio_create -> get/update: returned #<index> handle resolves on the saved doc (id dropped)", async () => {
// Create appends a drawio node after the existing paragraph.
const createDoc = {
type: "doc",
content: [{ type: "paragraph", attrs: { id: "p1" }, content: [] }],
};
const create = makeClient({ pageDoc: createDoc });
const res = await create.client.drawioCreate(
"page1",
{ position: "append" },
MODEL,
"T",
);
// The handle is index-based, not a block id.
assert.equal(res.nodeId, "#1");
// Take the document EXACTLY as it was saved: the schema drop stripped the
// node's id, so no id-based handle could ever resolve against it.
const savedDoc = create.calls.mutations[0].doc;
assert.equal(findDrawio(savedDoc)[0].attrs.id, undefined);
// drawio_get with the returned handle resolves the just-created node.
const getClient = makeClient({ pageDoc: savedDoc, attachmentSvg: svgFor(MODEL) });
const got = await getClient.client.drawioGet("page1", res.nodeId, "xml");
assert.equal(got.nodeId, res.nodeId);
assert.equal(got.content, normalizeXml(MODEL));
// drawio_update with the same handle + the hash from get repoints that node.
const upClient = makeClient({ pageDoc: savedDoc, attachmentSvg: svgFor(MODEL) });
const upd = await upClient.client.drawioUpdate(
"page1",
res.nodeId,
UPDATED_MODEL,
got.meta.hash,
);
assert.equal(upd.success, true);
assert.equal(upd.nodeId, res.nodeId);
const updated = findDrawio(upClient.calls.mutations[0].doc)[0];
assert.equal(
decodeDrawioSvg(upClient.calls.uploads[0].svg),
normalizeXml(UPDATED_MODEL),
);
assert.equal(updated.attrs.width, 340);
});
// --- error paths: the LLM must get a clean error, not a crash --------------
test("drawio_get: a bad node ref -> clean 'no node found' error", async () => {
// Page has one paragraph; the requested ref resolves to nothing.
const pageDoc = {
type: "doc",
content: [{ type: "paragraph", attrs: { id: "p1" }, content: [] }],
};
const { client } = makeClient({ pageDoc, attachmentSvg: svgFor(MODEL) });
await assert.rejects(
() => client.drawioGet("page1", "does-not-exist", "xml"),
/no node found for "does-not-exist"/,
);
});
test("drawio_get: a drawio node with no src -> clean 'has no src to read' error", async () => {
const pageDoc = {
type: "doc",
content: [
// A drawio node that carries no `src` (e.g. a half-written node).
{ type: "drawio", attrs: { id: "d1", attachmentId: "att-1" } },
],
};
const { client } = makeClient({ pageDoc, attachmentSvg: svgFor(MODEL) });
await assert.rejects(
() => client.drawioGet("page1", "d1", "xml"),
/node "d1" on page page1 has no src to read/,
);
});
test("drawio_update: the resolved node is NOT a drawio node -> clean error, no upload", async () => {
// "#0" resolves to a paragraph. The update must refuse cleanly rather than
// crash or repoint the wrong node.
const pageDoc = {
type: "doc",
content: [{ type: "paragraph", attrs: { id: "p1" }, content: [] }],
};
const { client, calls } = makeClient({ pageDoc, attachmentSvg: svgFor(MODEL) });
await assert.rejects(
() => client.drawioUpdate("page1", "#0", UPDATED_MODEL, "any-nonempty-hash"),
/node "#0" on page page1 is a paragraph, not a drawio diagram/,
);
assert.equal(calls.uploads.length, 0, "no upload when the node is not a diagram");
assert.equal(calls.mutations.length, 0, "no write when the node is not a diagram");
});
test("drawio_create: anchor not found -> clean error that reports the orphan attachment", async () => {
// The upload happens before the mutate transform; when the anchor cannot be
// found the write is skipped and the (now unreferenced) attachment is named
// in the error, exactly as the code documents.
const pageDoc = {
type: "doc",
content: [{ type: "paragraph", attrs: { id: "p1" }, content: [] }],
};
const { client, calls } = makeClient({ pageDoc });
await assert.rejects(
() =>
client.drawioCreate(
"page1",
{ position: "after", anchorNodeId: "nope" },
MODEL,
"T",
),
(err) =>
/anchor not found/.test(err.message) &&
/unreferenced orphan/.test(err.message) &&
/att-1/.test(err.message),
);
// The orphan was uploaded (and reported), but no node was written.
assert.equal(calls.uploads.length, 1, "attachment uploaded before the failed insert");
const drawios = calls.mutations.length ? findDrawio(calls.mutations[0].doc) : [];
assert.equal(drawios.length, 0, "no drawio node written when the anchor is missing");
});
// --- Fix 2: update targets ONLY the resolved node --------------------------
test("drawio_update: repoints ONLY the addressed node, not siblings sharing an attachmentId", async () => {
// A copied diagram: two drawio nodes share one attachmentId. Updating via the
// "#0" handle must touch node #0 only, never the sibling copy.
const shared = {
type: "doc",
content: [
{
type: "drawio",
attrs: {
src: "/api/files/shared/x.svg",
attachmentId: "shared",
width: 200,
height: 120,
},
},
{
type: "drawio",
attrs: {
src: "/api/files/shared/x.svg",
attachmentId: "shared",
width: 200,
height: 120,
},
},
],
};
const { client, calls } = makeClient({
pageDoc: shared,
attachmentSvg: svgFor(MODEL),
});
const res = await client.drawioUpdate(
"page1",
"#0",
UPDATED_MODEL,
mxHash(normalizeXml(MODEL)),
);
assert.equal(res.success, true);
const drawios = findDrawio(calls.mutations[0].doc);
assert.equal(drawios.length, 2);
// Node #0 repointed to the NEW attachment ("att-1" from the stub) and dims.
assert.equal(drawios[0].attrs.attachmentId, "att-1");
assert.equal(drawios[0].attrs.width, 340);
assert.match(drawios[0].attrs.src, /^\/api\/files\/att-1\//);
// Node #1 (the sibling copy) is untouched despite sharing the old attachmentId.
assert.equal(drawios[1].attrs.attachmentId, "shared");
assert.equal(drawios[1].attrs.width, 200);
assert.equal(drawios[1].attrs.src, "/api/files/shared/x.svg");
});
@@ -19,6 +19,7 @@ import { WebSocketServer } from "ws";
import { Hocuspocus } from "@hocuspocus/server";
import { DocmostClient } from "../../build/client.js";
import { buildYDoc } from "../../build/lib/collaboration.js";
import { destroyAllSessions } from "../../build/lib/collab-session.js";
// Import the SAME page-lock module instance that build/client.js imports. ESM
// caches modules by resolved URL, so this `withPageLock` shares the very
// per-page mutex map (`chains`) the client uses — letting the replaceImage test
@@ -188,6 +189,10 @@ async function spawnCollabStack(opts = {}) {
const openStacks = [];
after(async () => {
// #400: tests now leave a cached live CollabSession per page. Destroy them
// first (closes the client ws) so the server.close() below is not racing an
// open collab connection.
destroyAllSessions();
await Promise.all(
openStacks.map(
({ server, hocuspocus }) =>
@@ -270,17 +275,23 @@ test("a UUID input is passed through unchanged and triggers NO /pages/info fetch
);
});
test("a repeated slugId edit resolves the UUID only once (cache)", async () => {
test("repeated slugId edits reuse ONE live collab session and resolve the UUID only once (#400 cache)", async () => {
const { state, baseURL } = await spawnCollabStack();
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// Each mock connection re-seeds a fresh "hello world" doc (the mock does not
// persist across connects), so both edits target "hello". The cache assertion
// only concerns the slugId->uuid resolution, not the document content.
// #400: a series of edits on the same page reuses ONE live CollabSession, so
// the connect/handshake happens once and the collab doc is OPENED a single
// time (not per edit). The live ydoc persists between edits (the whole point),
// so the second edit sees the first edit's result: after "hello" -> "hi world"
// it targets the still-present "world".
await client.editPageText(SLUG, [{ find: "hello", replace: "hi" }]);
await client.editPageText(SLUG, [{ find: "hello", replace: "hey" }]);
await client.editPageText(SLUG, [{ find: "world", replace: "planet" }]);
assert.deepEqual(state.docNames, [`page.${UUID}`, `page.${UUID}`]);
assert.deepEqual(
state.docNames,
[`page.${UUID}`],
"the two edits must reuse one live collab session -> a single collab-doc open (#400)",
);
assert.equal(
state.pagesInfoCalls.length,
1,
@@ -325,8 +336,9 @@ test("replaceImage opens by the resolved UUID AND keys its page lock by that UUI
await uploadStarted; // deterministic: replaceImage now holds its page lock.
// (a) OPEN BY UUID: the only collab doc opened so far (the scan pass) used the
// canonical UUID, never the slugId. (The write pass opens a second time after
// we release the gate; asserted at the end.)
// canonical UUID, never the slugId. (#400: the write pass will REUSE this same
// live session rather than reopen, so docNames stays a single entry — asserted
// at the end.)
assert.deepEqual(
state.docNames,
[`page.${UUID}`],
@@ -378,8 +390,9 @@ test("replaceImage opens by the resolved UUID AND keys its page lock by that UUI
assert.equal(res.success, true);
assert.equal(res.replaced, 1, "the one seeded image must be repointed");
// Both opens (scan pass + write pass) used the UUID; the slugId never appears.
assert.deepEqual(state.docNames, [`page.${UUID}`, `page.${UUID}`]);
// #400: the write pass REUSES the scan pass's live session, so the collab doc
// is opened ONCE across both passes (never reopened, never by the slugId).
assert.deepEqual(state.docNames, [`page.${UUID}`]);
assert.ok(
!state.docNames.includes(`page.${SLUG}`),
"replaceImage must NEVER open the collab doc by the slugId (the #260 bug)",
@@ -0,0 +1,87 @@
// #402 — INTEGRATION test that locks the registerTool monkeypatch installed by
// createDocmostMcpServer (src/index.ts). The sibling unit test
// (test/unit/tool-timing.test.mjs) only exercises the timeToolHandler helper in
// ISOLATION; it never constructs the server, so nothing there proves the factory
// actually (a) wraps every registered tool through that helper and (b) labels
// each sample with the tool's REGISTRATION name.
//
// Here we stand up a real McpServer via the factory, connect a real MCP Client
// over the SDK's in-memory transport, invoke one registered tool, and assert the
// host's onMetric sink received `("mcp_tool_duration_seconds", <number>, { tool
// })` with tool === the exact registration name. This locks both the "monkeypatch
// wraps tools" and "label = registration name" halves of the contract, so a
// mutation to args.slice(0, -1) / the handler-arg detection / the capture order
// is caught.
import { test } from "node:test";
import assert from "node:assert/strict";
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js";
import { createDocmostMcpServer } from "../../build/index.js";
// The tool we drive. get_workspace has NO input schema, so protocol-level input
// validation cannot short-circuit before the handler runs — the wrapped handler
// is guaranteed to execute (and then fail on the unreachable backend, which is
// exactly what we want: the wrapper times in a finally on throw too).
const TOOL_NAME = "get_workspace";
test("the factory's registerTool monkeypatch times a live tool call and labels it with the registration name", async () => {
const calls = [];
const onMetric = (name, value, labels) => calls.push({ name, value, labels });
// Minimal valid credentials config. apiUrl points at a port that refuses
// connections immediately so the tool's backend call fails FAST (ECONNREFUSED)
// rather than hanging — the wrapper still emits the metric from its finally.
const server = createDocmostMcpServer({
apiUrl: "http://127.0.0.1:1",
email: "x@example.com",
password: "pw",
onMetric,
});
const [clientTransport, serverTransport] =
InMemoryTransport.createLinkedPair();
const client = new Client(
{ name: "test-client", version: "0.0.0" },
{ capabilities: {} },
);
await Promise.all([
server.connect(serverTransport),
client.connect(clientTransport),
]);
try {
// Invoke the tool. The backend is unreachable, so this either resolves with
// an error result (isError) or rejects — both are fine. What matters is that
// the handler ran through the timing wrapper, which fires onMetric either way.
try {
await client.callTool({ name: TOOL_NAME, arguments: {} });
} catch {
// Tolerate the expected backend failure surfacing as a thrown protocol error.
}
// The wrapper must have fed exactly the timing sample for THIS tool.
const timing = calls.filter(
(c) => c.name === "mcp_tool_duration_seconds",
);
assert.ok(
timing.length >= 1,
"onMetric must receive a mcp_tool_duration_seconds sample from the wrapped handler",
);
const sample = timing.find((c) => c.labels && c.labels.tool === TOOL_NAME);
assert.ok(
sample,
`a timing sample must be labelled with the registration name "${TOOL_NAME}"; ` +
`got labels: ${JSON.stringify(timing.map((c) => c.labels))}`,
);
assert.equal(typeof sample.value, "number");
assert.ok(sample.value >= 0, "duration must be non-negative seconds");
} finally {
await client.close();
await server.close();
}
});
@@ -28,8 +28,9 @@ import { DocmostClient } from "../../build/index.js";
// in the server's DocmostClientLike interface (the in-app per-user tool adapter
// only — it is a SUBSET of the DocmostClient surface — covers only what the in-app adapter
// consumes; the standalone MCP transport (packages/mcp/src/index.ts) calls additional
// client methods (insertImage/replaceImage/deleteComment/updateComment/insertFootnote)
// that this guard does NOT track — the MCP transport's own typecheck covers those). Full type-derivation
// client methods (deleteComment/updateComment) that this guard does NOT track — the
// MCP transport's own typecheck covers those. insertImage/replaceImage/insertFootnote
// were MCP-only but are now in-app-consumed too (#410), so they ARE tracked below. Full type-derivation
// of DocmostClientLike from this class is deferred (see the staged plan in
// docmost-client.loader.ts): the package emits no declarations and the real
// (inferred, concrete) return types conflict with the host's loose
@@ -76,6 +77,10 @@ const HOST_CONTRACT_METHODS = [
"restorePageVersion",
"transformPage",
"stashPage",
// write (image / footnote) — MCP-only until #410 promoted them to in-app tools
"insertImage",
"replaceImage",
"insertFootnote",
// write (comment)
"createComment",
"resolveComment",
@@ -0,0 +1,348 @@
import { test, beforeEach, afterEach, mock } from "node:test";
import assert from "node:assert/strict";
import { EventEmitter } from "node:events";
import {
acquireCollabSession,
destroyAllSessions,
__setCollabProviderFactory,
__sessionCountForTests,
} from "../../build/lib/collab-session.js";
import { withPageLock } from "../../build/lib/page-lock.js";
// A stand-in for HocuspocusProvider: it shares the ydoc (so the real yjs
// read/transform/write in CollabSession.mutate runs unchanged), auto-completes
// the connect+sync handshake on a microtask (unaffected by mock timers), and
// exposes hooks to drive disconnect/close/auth-failure and the unsyncedChanges
// ack. There is no collaboration server in the test env, so every test drives
// the provider through this fake.
class FakeProvider extends EventEmitter {
static instances = [];
static connectCount = 0;
static reset() {
FakeProvider.instances = [];
FakeProvider.connectCount = 0;
}
static last() {
return FakeProvider.instances[FakeProvider.instances.length - 1];
}
constructor(config, opts = {}) {
super();
this.config = config;
this.ydoc = config.document;
this.synced = false;
this.unsyncedChanges = opts.unsynced ?? 0;
this.destroyed = false;
FakeProvider.instances.push(this);
FakeProvider.connectCount += 1;
if (opts.autoSync !== false) {
// Real HocuspocusProvider fires onSynced asynchronously after the
// handshake; a microtask reproduces that without depending on timers.
queueMicrotask(() => {
if (this.destroyed) return;
this.config.onConnect?.();
this.synced = true;
this.config.onSynced?.();
});
}
}
destroy() {
this.destroyed = true;
}
// --- test drivers ---
_disconnect() {
this.config.onDisconnect?.();
}
_close() {
this.config.onClose?.();
}
_authFail() {
this.config.onAuthenticationFailed?.();
}
_ack() {
this.unsyncedChanges = 0;
this.emit("unsyncedChanges", { number: 0 });
}
}
/** Build a provider factory that stamps every provider with the given opts. */
function factory(opts = {}) {
return (config) => new FakeProvider(config, opts);
}
/** A minimal, schema-valid ProseMirror doc for a write. */
function docWith(text) {
return {
type: "doc",
content: [
{ type: "paragraph", content: [{ type: "text", text: String(text) }] },
],
};
}
const ENV_KEYS = [
"MCP_COLLAB_SESSION_IDLE_MS",
"MCP_COLLAB_SESSION_MAX_AGE_MS",
"MCP_COLLAB_SESSION_MAX_ENTRIES",
];
let savedEnv;
beforeEach(() => {
savedEnv = {};
for (const k of ENV_KEYS) savedEnv[k] = process.env[k];
FakeProvider.reset();
__setCollabProviderFactory(factory());
});
afterEach(() => {
destroyAllSessions();
__setCollabProviderFactory(null);
mock.timers.reset();
for (const k of ENV_KEYS) {
if (savedEnv[k] === undefined) delete process.env[k];
else process.env[k] = savedEnv[k];
}
});
test("acquire opens one provider; reuse returns the SAME live session", async () => {
const a = await acquireCollabSession("page-1", "tok", "http://h/api");
const b = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.equal(a, b, "same (wsUrl,page,token) must reuse the session");
assert.equal(FakeProvider.connectCount, 1, "exactly one connect/sync");
assert.equal(__sessionCountForTests(), 1);
});
test("N mutates on one page open the provider ONCE (coalesced series)", async () => {
const session = await acquireCollabSession("page-1", "tok", "http://h/api");
for (let i = 0; i < 20; i++) {
const r = await session.mutate(() => docWith(`edit ${i}`));
assert.ok(r && r.doc, "each mutate resolves a MutationResult");
}
assert.equal(
FakeProvider.connectCount,
1,
"20 mutates must not reconnect — one live provider for the whole series",
);
});
test("mutate preserves the transform-abort no-op report (null transform)", async () => {
const session = await acquireCollabSession("page-1", "tok", "http://h/api");
const r = await session.mutate(() => null);
assert.equal(r.verify.changed, false);
assert.equal(r.verify.summary, "no changes (transform aborted)");
});
test("registry key includes the token: different tokens => different sessions", async () => {
const a = await acquireCollabSession("page-1", "tok-A", "http://h/api");
const b = await acquireCollabSession("page-1", "tok-B", "http://h/api");
assert.notEqual(a, b, "different tokens must never share a session");
assert.equal(FakeProvider.connectCount, 2);
assert.equal(__sessionCountForTests(), 2);
// Same token reuses.
const a2 = await acquireCollabSession("page-1", "tok-A", "http://h/api");
assert.equal(a, a2);
assert.equal(FakeProvider.connectCount, 2);
});
test("disconnect at any time kills the session and removes it from the registry", async () => {
const a = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.equal(__sessionCountForTests(), 1);
FakeProvider.last()._disconnect();
assert.equal(__sessionCountForTests(), 0, "dead session is deregistered");
// Re-acquire opens a brand new provider.
const b = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.notEqual(a, b);
assert.equal(FakeProvider.connectCount, 2);
});
test("an in-flight mutate rejects with the connection-closed text on disconnect", async () => {
__setCollabProviderFactory(factory({ unsynced: 1 })); // stay pending after write
const session = await acquireCollabSession("page-1", "tok", "http://h/api");
const p = session.mutate(() => docWith("x"));
// The write happened synchronously; persistence is pending. Now the socket drops.
FakeProvider.last()._disconnect();
await assert.rejects(
p,
/Collaboration connection closed before the update was persisted\/synced/,
);
});
test("auth failure rejects the pending open with the auth error text", async () => {
__setCollabProviderFactory(factory({ autoSync: false }));
const p = acquireCollabSession("page-1", "tok", "http://h/api");
// Let the provider be constructed, then fail auth.
await Promise.resolve();
FakeProvider.last()._authFail();
await assert.rejects(
p,
/Authentication failed for collaboration connection/,
);
assert.equal(__sessionCountForTests(), 0);
});
test("mutate-error invalidates the session (caller destroys; re-acquire is fresh)", async () => {
const session = await acquireCollabSession("page-1", "tok", "http://h/api");
await assert.rejects(
session.mutate(() => {
throw new Error("afterText not found");
}),
/afterText not found/,
);
// The production caller destroys on failure; mirror that here.
session.destroy("mutate failed");
assert.equal(__sessionCountForTests(), 0);
const fresh = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.notEqual(fresh, session);
assert.equal(FakeProvider.connectCount, 2);
});
test("a pending write resolves when the server acks (unsyncedChanges -> 0)", async () => {
__setCollabProviderFactory(factory({ unsynced: 1 }));
const session = await acquireCollabSession("page-1", "tok", "http://h/api");
const p = session.mutate(() => docWith("y"));
FakeProvider.last()._ack();
const r = await p;
assert.ok(r.doc);
});
test("concurrent mutate on one session: the second rejects, the first is unaffected", async () => {
__setCollabProviderFactory(factory({ unsynced: 1 })); // first stays pending after write
const session = await acquireCollabSession("page-1", "tok", "http://h/api");
// First mutate: write happens synchronously, persistence ack still pending.
const first = session.mutate(() => docWith("first"));
// Second overlapping mutate on the SAME session must fail fast.
await assert.rejects(
session.mutate(() => docWith("second")),
/mutate already in-flight; caller must serialize \(hold the page lock\)/,
);
// The first op is untouched: its rejector was not clobbered. Ack it now.
FakeProvider.last()._ack();
const r = await first;
assert.ok(r.doc, "the first in-flight mutate still resolves on its own ack");
assert.equal(FakeProvider.connectCount, 1, "no reconnect from the rejected overlap");
});
test("sequential mutates on one session both succeed (the guard doesn't break serialized use)", async () => {
const session = await acquireCollabSession("page-1", "tok", "http://h/api");
// Await the first fully, then run the second: inflightReject was cleared by
// localFinish before the first settled, so the guard is clear for the second.
const r1 = await session.mutate(() => docWith("one"));
assert.ok(r1.doc, "first sequential mutate resolves");
const r2 = await session.mutate(() => docWith("two"));
assert.ok(r2.doc, "second sequential mutate resolves — guard not tripped");
assert.equal(FakeProvider.connectCount, 1, "sequential mutates reuse one provider");
});
test("connect timeout rejects with the connect-timeout text and fires the metric hook", async () => {
mock.timers.enable({ apis: ["setTimeout"] });
__setCollabProviderFactory(factory({ autoSync: false }));
let metricFired = 0;
const p = acquireCollabSession("page-1", "tok", "http://h/api", {
onConnectTimeout: () => {
metricFired += 1;
},
});
mock.timers.tick(25000);
await assert.rejects(p, /Connection timeout to collaboration server/);
assert.equal(metricFired, 1);
assert.equal(__sessionCountForTests(), 0);
});
test("idle TTL destroys the session; re-acquire reconnects", async () => {
mock.timers.enable({ apis: ["setTimeout"] });
process.env.MCP_COLLAB_SESSION_IDLE_MS = "1000";
const a = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.equal(__sessionCountForTests(), 1);
mock.timers.tick(1000); // idle fires
assert.equal(__sessionCountForTests(), 0, "idle timeout destroyed it");
const b = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.notEqual(a, b);
assert.equal(FakeProvider.connectCount, 2);
});
test("max age is enforced at acquire (destroy + open fresh), idle held off", async () => {
mock.timers.enable({ apis: ["setTimeout", "Date"] });
process.env.MCP_COLLAB_SESSION_MAX_AGE_MS = "1000";
process.env.MCP_COLLAB_SESSION_IDLE_MS = "10000000"; // never fires in this test
const a = await acquireCollabSession("page-1", "tok", "http://h/api");
mock.timers.tick(2000); // past max age, below idle
const b = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.notEqual(a, b, "a session past its max age must be replaced");
assert.equal(FakeProvider.connectCount, 2);
});
test("registry cap: least-recently-used session is destroy-evicted", async () => {
process.env.MCP_COLLAB_SESSION_MAX_ENTRIES = "2";
const s1 = await acquireCollabSession("page-1", "tok", "http://h/api");
const p1prov = FakeProvider.last();
const s2 = await acquireCollabSession("page-2", "tok", "http://h/api");
const p2prov = FakeProvider.last();
assert.equal(__sessionCountForTests(), 2);
// Touch page-1 so page-2 becomes the least-recently-used entry.
assert.equal(await acquireCollabSession("page-1", "tok", "http://h/api"), s1);
assert.equal(FakeProvider.connectCount, 2, "reuse must not reconnect");
// A third distinct page (cap 2) must evict the LRU (page-2), not page-1.
const s3 = await acquireCollabSession("page-3", "tok", "http://h/api");
assert.equal(__sessionCountForTests(), 2);
assert.equal(FakeProvider.connectCount, 3);
assert.equal(p2prov.destroyed, true, "the LRU provider was destroy-evicted");
assert.equal(p1prov.destroyed, false, "the MRU page-1 survived");
// page-1 still reused (survived), page-3 still reused.
assert.equal(await acquireCollabSession("page-1", "tok", "http://h/api"), s1);
assert.equal(await acquireCollabSession("page-3", "tok", "http://h/api"), s3);
assert.equal(FakeProvider.connectCount, 3, "no extra reconnects");
});
test("MCP_COLLAB_SESSION_IDLE_MS=0 disables the cache (legacy provider-per-op)", async () => {
process.env.MCP_COLLAB_SESSION_IDLE_MS = "0";
const s1 = await acquireCollabSession("page-1", "tok", "http://h/api");
// Never registered (ephemeral).
assert.equal(__sessionCountForTests(), 0);
const r1 = await s1.mutate(() => docWith("a"));
assert.ok(r1.doc);
// After its single op the ephemeral session self-destroyed.
assert.equal(FakeProvider.instances[0].destroyed, true);
// A second op opens a BRAND NEW provider (no reuse).
const s2 = await acquireCollabSession("page-1", "tok", "http://h/api");
assert.notEqual(s1, s2);
assert.equal(FakeProvider.connectCount, 2);
});
test("replaceImage-shaped flow: acquire under an EXTERNAL page lock does not deadlock and reuses one session", async () => {
const pageId = "page-lock";
// Mirror replaceImage: hold ONE withPageLock across scan (read-only) + write,
// each going through the non-locking acquireCollabSession.
const result = await withPageLock(pageId, async () => {
// pass 1: read-only scan (transform returns null -> no write)
const scan = await acquireCollabSession(pageId, "tok", "http://h/api");
const scanRes = await scan.mutate(() => null);
assert.equal(scanRes.verify.changed, false);
// pass 2: the actual write, same held lock
const write = await acquireCollabSession(pageId, "tok", "http://h/api");
return write.mutate(() => docWith("repointed"));
});
assert.ok(result.doc, "the locked scan+write completed without deadlock");
assert.equal(
FakeProvider.connectCount,
1,
"both passes under the held lock reuse ONE live session",
);
});
test("destroyAllSessions tears down every cached session", async () => {
await acquireCollabSession("page-1", "tok", "http://h/api");
await acquireCollabSession("page-2", "tok", "http://h/api");
assert.equal(__sessionCountForTests(), 2);
const provs = [...FakeProvider.instances];
destroyAllSessions();
assert.equal(__sessionCountForTests(), 0);
assert.ok(provs.every((p) => p.destroyed), "all providers destroyed");
});
@@ -8,6 +8,7 @@ import {
applyAnchorInDoc,
countAnchorMatches,
getAnchoredText,
resolveAnchorSelection,
} from "../../build/lib/comment-anchor.js";
const COMMENT_ID = "cmt-123";
@@ -308,3 +309,70 @@ test("getAnchoredText returns null when the selection does not anchor", () => {
const doc = paragraphDoc([{ type: "text", text: "hello world" }]);
assert.equal(getAnchoredText(doc, "not present"), null);
});
// ---------------------------------------------------------------------------
// #408 MARKDOWN-STRIP FALLBACK. A selection copied with inline markdown still
// carries `**`/`` ` ``/`[t](u)` markers the plain document text lacks. When the
// verbatim selection anchors nowhere, all four entry points retry with the
// markdown stripped — consistently, so the suggestion-uniqueness gate stays
// coherent — while what gets STORED remains the raw document substring.
// ---------------------------------------------------------------------------
test("a markdown-styled selection anchors against plain doc text via the strip fallback", () => {
const doc = paragraphDoc([{ type: "text", text: "a bold word here" }]);
// The agent quoted "**bold** word" from a styled view; the doc is plain text.
const sel = "**bold** word";
const resolved = resolveAnchorSelection(doc, sel);
assert.equal(resolved.found, true, "strip fallback finds the anchor");
assert.equal(resolved.normalized, true, "reports the soft-warning flag");
assert.equal(canAnchorInDoc(doc, sel), true);
assert.equal(countAnchorMatches(doc, sel), 1);
const ok = applyAnchorInDoc(doc, sel, COMMENT_ID);
assert.equal(ok, true);
const marked = doc.content[0].content.filter((p) => commentMark(p));
assert.equal(marked.map((m) => m.text).join(""), "bold word",
"the mark lands on the plain-text span");
});
test("getAnchoredText stores the RAW doc substring even when matched via the strip fallback", () => {
// Doc uses a smart apostrophe; the agent typed ASCII + markdown emphasis.
const doc = paragraphDoc([{ type: "text", text: "it’s bold now" }]);
const stored = getAnchoredText(doc, "it's **bold**");
assert.equal(stored, "it’s bold",
"stored selection is the raw document text, not the stripped/ASCII locator");
});
test("the strip fallback does not flip a raw-unique selection to ambiguous", () => {
// "config" appears twice, but the raw phrase "config value" appears once.
const doc = {
type: "doc",
content: [
{ type: "paragraph", content: [{ type: "text", text: "the config value here" }] },
{ type: "paragraph", content: [{ type: "text", text: "another config here" }] },
],
};
// Raw phrase is unique -> exactly 1, and no strip happens (nothing to strip).
assert.equal(countAnchorMatches(doc, "config value"), 1);
assert.equal(resolveAnchorSelection(doc, "config value").normalized, false);
});
test("EXACT WINS: a raw match short-circuits the strip fallback (count reflects raw)", () => {
// A literal "**" run exists raw once; its stripped form would also appear.
const doc = paragraphDoc([{ type: "text", text: "use **stars** and stars" }]);
// Raw "**stars**" occurs once -> count 1 from the verbatim locator; the
// fallback (which would find two "stars") never runs.
assert.equal(countAnchorMatches(doc, "**stars**"), 1);
assert.equal(resolveAnchorSelection(doc, "**stars**").normalized, false);
});
test("a markdown selection whose stripped form is ambiguous is counted as ambiguous", () => {
const doc = {
type: "doc",
content: [
{ type: "paragraph", content: [{ type: "text", text: "first config here" }] },
{ type: "paragraph", content: [{ type: "text", text: "second config here" }] },
],
};
// Verbatim "**config**" matches nothing; stripped "config" matches twice.
assert.equal(countAnchorMatches(doc, "**config**"), 2);
});
@@ -0,0 +1,84 @@
// Unit tests for the pure-TS schematic SVG preview (issue #423). Asserts that
// the preview emits well-formed SVG covering each primitive (rect/ellipse/
// rhombus/edge), resolves container-relative coordinates to absolute, and that
// the full `.drawio.svg` wrapper parses.
import { test } from "node:test";
import assert from "node:assert/strict";
import { JSDOM } from "jsdom";
import { renderDiagramShapes } from "../../build/lib/drawio-preview.js";
import {
parseCells,
computeBBox,
buildDrawioSvg,
normalizeXml,
} from "../../build/lib/drawio-xml.js";
const { window } = new JSDOM("");
function parseSvg(svg) {
const doc = new window.DOMParser().parseFromString(svg, "application/xml");
const err = doc.getElementsByTagName("parsererror");
assert.equal(err.length, 0, `SVG did not parse: ${err[0]?.textContent}`);
return doc;
}
const MODEL =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="Box &amp; Co" style="rounded=1;fillColor=#dae8fc;strokeColor=#6c8ebf;" vertex="1" parent="1">' +
'<mxGeometry x="40" y="40" width="120" height="60" as="geometry"/></mxCell>' +
'<mxCell id="3" value="Circle" style="ellipse;fillColor=#d5e8d4;" vertex="1" parent="1">' +
'<mxGeometry x="240" y="40" width="80" height="80" as="geometry"/></mxCell>' +
'<mxCell id="5" value="Dec" style="rhombus;" vertex="1" parent="1">' +
'<mxGeometry x="40" y="160" width="100" height="60" as="geometry"/></mxCell>' +
'<mxCell id="4" value="link" edge="1" parent="1" source="2" target="3"><mxGeometry relative="1" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
test("renderDiagramShapes emits rect, ellipse, polygon and a line", () => {
const cells = parseCells(MODEL);
const inner = renderDiagramShapes(cells, computeBBox(cells));
assert.ok(inner.includes("<rect"), "has a rect");
assert.ok(inner.includes("<ellipse"), "has an ellipse");
assert.ok(inner.includes("<polygon"), "has a polygon (rhombus)");
assert.ok(inner.includes("<line"), "has an edge line");
// Labels are HTML-escaped.
assert.ok(inner.includes("Box &amp; Co"));
});
test("the full .drawio.svg wrapper parses as valid XML", () => {
const cells = parseCells(MODEL);
const bbox = computeBBox(cells);
const inner = renderDiagramShapes(cells, bbox);
const svg = buildDrawioSvg(normalizeXml(MODEL), inner, bbox);
const doc = parseSvg(svg);
assert.equal(doc.documentElement.tagName, "svg");
assert.ok(doc.documentElement.getAttribute("content"), "carries content=");
// The visible children exist.
assert.ok(doc.getElementsByTagName("rect").length >= 1);
});
test("container children resolve to absolute coordinates", () => {
// A group at (100,100) with a child rect at relative (10,10,20,20) -> abs 110,110.
const model =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="g" style="group;" vertex="1" parent="1"><mxGeometry x="100" y="100" width="200" height="200" as="geometry"/></mxCell>' +
'<mxCell id="c" value="in" vertex="1" parent="g"><mxGeometry x="10" y="10" width="20" height="20" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const cells = parseCells(model);
const inner = renderDiagramShapes(cells, computeBBox(cells));
// The child rect must be placed at absolute x=110,y=110.
assert.ok(/<rect x="110" y="110"/.test(inner), `expected abs child rect, got: ${inner}`);
});
test("unknown stencil (shape=mxgraph.*) degrades to a labeled rectangle", () => {
const model =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="AWS" style="shape=mxgraph.aws4.lambda;" vertex="1" parent="1">' +
'<mxGeometry x="0" y="0" width="60" height="60" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const cells = parseCells(model);
const inner = renderDiagramShapes(cells, computeBBox(cells));
assert.ok(inner.includes("<rect"), "unknown stencil -> rect");
assert.ok(inner.includes(">AWS<"), "keeps the label");
});
+360
View File
@@ -0,0 +1,360 @@
// Unit tests for the drawio-xml module (issue #423): the linter (a positive
// baseline + a negative case per rule, each asserting rule + cellId), the
// decode chain (plain nested XML AND draw.io's compressed <diagram> via pako),
// encode/round-trip byte-stability, hash stability, style parsing and the
// bounding box.
import { test } from "node:test";
import assert from "node:assert/strict";
import pako from "pako";
import {
parseStyle,
lintModel,
prepareModel,
normalizeInput,
normalizeXml,
mxHash,
computeBBox,
parseCells,
decodeDrawioSvg,
decodeDrawioFileToModel,
buildDrawioSvg,
encodeDrawioFile,
countUserCells,
DrawioLintError,
inflateDiagramPayload,
MAX_INFLATED_DIAGRAM_BYTES,
} from "../../build/lib/drawio-xml.js";
// A well-formed model with one vertex and a valid edge to it.
const VALID_MODEL =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="Hello" style="rounded=1;whiteSpace=wrap;html=1;" vertex="1" parent="1">' +
'<mxGeometry x="100" y="100" width="120" height="60" as="geometry"/></mxCell>' +
'<mxCell id="3" value="Two" style="ellipse;" vertex="1" parent="1">' +
'<mxGeometry x="300" y="100" width="80" height="80" as="geometry"/></mxCell>' +
'<mxCell id="4" edge="1" parent="1" source="2" target="3">' +
'<mxGeometry relative="1" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
function issuesOf(fn) {
try {
fn();
return null;
} catch (e) {
assert.ok(e instanceof DrawioLintError, `expected DrawioLintError, got ${e}`);
return e.issues;
}
}
const hasRule = (issues, rule, cellId) =>
issues.some(
(i) => i.rule === rule && (cellId === undefined || i.cellId === cellId),
);
// --- style parsing ---------------------------------------------------------
test("parseStyle: base stylename + key=value pairs", () => {
const r = parseStyle("ellipse;fillColor=#ff0000;whiteSpace=wrap;");
assert.equal(r.baseStyle, "ellipse");
assert.equal(r.map.fillColor, "#ff0000");
assert.equal(r.map.whiteSpace, "wrap");
assert.equal(r.badSegment, undefined);
});
test("parseStyle: flags a segment with two '='", () => {
const r = parseStyle("a=b=c;");
assert.equal(r.badSegment, "a=b=c");
});
test("parseStyle: a second bare token is malformed", () => {
const r = parseStyle("rounded=1;bareword");
assert.equal(r.badSegment, "bareword");
});
// --- linter: positive baseline ---------------------------------------------
test("lintModel: the canonical valid model passes", () => {
const { cells } = lintModel(VALID_MODEL);
assert.equal(cells.length, 5);
});
// --- linter: one negative case per rule ------------------------------------
test("rule well-formed-xml: malformed XML", () => {
const issues = issuesOf(() => lintModel("<mxGraphModel><root><mxCell id=\"0\"></root>"));
assert.ok(hasRule(issues, "well-formed-xml"));
assert.ok(issues[0].position, "carries a line:col position");
});
test("rule structure: root is not mxGraphModel", () => {
const issues = issuesOf(() => lintModel("<foo><root/></foo>"));
assert.ok(hasRule(issues, "structure"));
});
test("rule sentinel-cells: missing id=0 / id=1", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "sentinel-cells", "0"));
assert.ok(hasRule(issues, "sentinel-cells", "1"));
});
test("rule duplicate-id: two cells share an id", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "duplicate-id", "2"));
});
test("rule vertex-edge-exclusive: cell is both vertex and edge", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" vertex="1" edge="1" parent="1"><mxGeometry as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "vertex-edge-exclusive", "2"));
});
test("rule edge-geometry: self-closed edge without child mxGeometry", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'<mxCell id="3" vertex="1" parent="1"><mxGeometry x="30" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'<mxCell id="4" edge="1" parent="1" source="2" target="3"/>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "edge-geometry", "4"));
});
test("rule edge-endpoint: source/target does not resolve", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'<mxCell id="4" edge="1" parent="1" source="2" target="99"><mxGeometry relative="1" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "edge-endpoint", "4"));
});
test("rule parent-exists: parent points at a missing id", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" vertex="1" parent="42"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "parent-exists", "2"));
});
test("rule no-comments: XML comment present", () => {
const m =
'<mxGraphModel><root>' +
'<!-- a comment --><mxCell id="0"/><mxCell id="1" parent="0"/>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "no-comments"));
});
test("rule style-format: malformed style segment (cellId reported)", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" style="rounded=1;a=b=c;" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "style-format", "2"));
});
test("rule value-newline: literal newline in a value (cellId reported)", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="line1\nline2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "value-newline", "2"));
});
test("rule value-escaping: unescaped ampersand in a value (cellId reported)", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="A & B" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
const issues = issuesOf(() => lintModel(m));
assert.ok(hasRule(issues, "value-escaping", "2"));
});
test("rule reserved id: escaped entity value passes (no false positive)", () => {
const m =
'<mxGraphModel><root>' +
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" value="A &amp; B &lt;ok&gt;" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>' +
'</root></mxGraphModel>';
assert.doesNotThrow(() => lintModel(m));
});
// --- input normalization ---------------------------------------------------
test("normalizeInput: a list of <mxCell> is wrapped and sentinels added", () => {
const frag =
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>';
const model = normalizeInput(frag);
assert.ok(model.startsWith("<mxGraphModel"));
assert.ok(model.includes('<mxCell id="0"/>'));
assert.ok(model.includes('<mxCell id="1" parent="0"/>'));
// And it lints clean.
assert.doesNotThrow(() => lintModel(model));
});
test("normalizeInput: an existing sentinel is not duplicated", () => {
const frag =
'<mxCell id="0"/><mxCell id="1" parent="0"/>' +
'<mxCell id="2" vertex="1" parent="1"><mxGeometry x="0" y="0" width="10" height="10" as="geometry"/></mxCell>';
const model = normalizeInput(frag);
const count0 = (model.match(/id="0"/g) || []).length;
assert.equal(count0, 1);
});
test("prepareModel: returns bbox, cellCount, hash and lints", () => {
const p = prepareModel(VALID_MODEL);
assert.equal(p.cellCount, 3); // 2, 3, 4 (sentinels excluded)
assert.ok(p.bbox.width > 0 && p.bbox.height > 0);
assert.equal(p.hash, mxHash(normalizeXml(VALID_MODEL)));
});
// --- decode chain: plain -----------------------------------------------------
test("decode chain (plain): buildDrawioSvg -> decodeDrawioSvg round-trips byte-stable", () => {
const model = normalizeXml(VALID_MODEL);
const svg = buildDrawioSvg(model, "<g/>", { width: 400, height: 200 });
const decoded = decodeDrawioSvg(svg);
assert.equal(decoded, model);
});
test("decode chain: entity-encoded content= (draw.io export style) is read directly", () => {
const model = normalizeXml(VALID_MODEL);
const file = encodeDrawioFile(model, "Page-1");
const escaped = file
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;");
const svg = `<svg xmlns="http://www.w3.org/2000/svg" content="${escaped}"></svg>`;
const decoded = decodeDrawioSvg(svg);
assert.equal(decoded, model);
});
// --- decode chain: compressed (pako) ---------------------------------------
test("decode chain (compressed pako): human-saved <diagram> payload decodes losslessly", () => {
const model = normalizeXml(VALID_MODEL);
// Reproduce draw.io's compression: encodeURIComponent -> raw deflate -> base64.
const compressed = Buffer.from(
pako.deflateRaw(encodeURIComponent(model)),
).toString("base64");
const file = `<mxfile host="Electron"><diagram id="abc" name="Page-1">${compressed}</diagram></mxfile>`;
// Docmost stores the file base64 in content=.
const contentB64 = Buffer.from(file, "utf-8").toString("base64");
const svg = `<svg xmlns="http://www.w3.org/2000/svg" content="${contentB64}"><image href="x"/></svg>`;
const decoded = decodeDrawioSvg(svg);
assert.equal(decoded, model);
});
test("decodeDrawioFileToModel: bare mxGraphModel file returns the model substring", () => {
const model = normalizeXml(VALID_MODEL);
assert.equal(decodeDrawioFileToModel(model), model);
});
// --- hash stability --------------------------------------------------------
test("mxHash: stable across inter-tag whitespace, sensitive to content", () => {
const a = VALID_MODEL;
const b = VALID_MODEL.replace(/></g, ">\n <"); // reformat only
assert.equal(mxHash(a), mxHash(b));
const c = VALID_MODEL.replace('value="Hello"', 'value="Changed"');
assert.notEqual(mxHash(a), mxHash(c));
});
// --- bounding box + cell count ---------------------------------------------
test("computeBBox + countUserCells", () => {
const cells = parseCells(VALID_MODEL);
const bbox = computeBBox(cells);
// Vertex 3 spans to x=380,y=180; plus the 20px margin.
assert.equal(bbox.width, 400);
assert.equal(bbox.height, 200);
assert.equal(countUserCells(VALID_MODEL), 3);
});
// --- decompression-bomb guard (Fix 3) --------------------------------------
test("inflateDiagramPayload: a small legitimate payload inflates fine", () => {
const xml = normalizeXml(VALID_MODEL);
const base64 = Buffer.from(
pako.deflateRaw(encodeURIComponent(xml)),
).toString("base64");
assert.equal(inflateDiagramPayload(base64), xml);
});
test("inflateDiagramPayload: rejects an over-cap decompression bomb", () => {
// A tiny compressed payload that inflates to just over the cap. Highly
// compressible (all one byte) -> the base64 is small, but the inflated output
// exceeds MAX_INFLATED_DIAGRAM_BYTES and must be refused before it is fully
// materialised.
const bombSize = MAX_INFLATED_DIAGRAM_BYTES + 1024;
const base64 = Buffer.from(
pako.deflateRaw(Buffer.alloc(bombSize, 0x41 /* 'A' */)),
).toString("base64");
assert.ok(
base64.length < 1024 * 1024,
"the compressed bomb is tiny relative to its inflated size",
);
assert.throws(
() => inflateDiagramPayload(base64),
/decompression bomb/,
);
});
test("encode/build: a title with < > \" & round-trips without corrupting the SVG", async () => {
// A user-supplied title full of XML metacharacters must be escaped so the
// inner <mxfile> stays well-formed and the outer content="..." attribute is
// never broken out of. Prove it survives the encode -> build -> decode chain.
const title = 'A < B > C " D & E';
const model = normalizeXml(VALID_MODEL);
const svg = buildDrawioSvg(model, "<g/>", { width: 200, height: 120 }, title);
// The outer content="..." attribute must not be broken by the title: the raw
// title metacharacters never appear literally in the SVG markup (they are
// base64-encoded inside content=, and escaped inside the file XML).
const contentMatch = /content="([^"]*)"/.exec(svg);
assert.ok(contentMatch, "SVG has a single well-formed content= attribute");
// The diagram model still decodes losslessly despite the exotic title.
assert.equal(decodeDrawioSvg(svg), model);
// The file XML is well-formed: the title lives in name="..." as escaped
// entities, so unescaping recovers the original title byte-for-byte.
const fileXml = Buffer.from(contentMatch[1], "base64").toString("utf-8");
const nameMatch = /<diagram id="[^"]*" name="([^"]*)">/.exec(fileXml);
assert.ok(nameMatch, "the diagram name attribute is intact and quote-safe");
const decodedTitle = nameMatch[1]
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/&amp;/g, "&");
assert.equal(decodedTitle, title);
// encodeDrawioFile alone produces the same escaped, well-formed envelope.
const file = encodeDrawioFile(model, title);
assert.match(file, /name="A &lt; B &gt; C &quot; D &amp; E">/);
});
@@ -0,0 +1,317 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { normalizeAndMergeFootnotes } from "../../build/lib/footnote-normalize-merge.js";
import { canonicalizeFootnotes } from "../../build/lib/footnote-canonicalize.js";
function findAll(node, type, acc = []) {
if (!node || typeof node !== "object") return acc;
if (node.type === type) acc.push(node);
if (Array.isArray(node.content)) {
for (const c of node.content) findAll(c, type, acc);
}
return acc;
}
const defs = (doc) => findAll(doc, "footnoteDefinition");
const defIds = (doc) => defs(doc).map((d) => d.attrs.id);
const refIds = (doc) => findAll(doc, "footnoteReference").map((r) => r.attrs.id);
const defText = (d) =>
findAll(d, "text")
.map((t) => t.text)
.join("");
const ref = (id) => ({ type: "footnoteReference", attrs: { id } });
const para = (...inline) => ({ type: "paragraph", content: inline });
const txt = (text, marks) =>
marks ? { type: "text", text, marks } : { type: "text", text };
const def = (id, ...inline) => ({
type: "footnoteDefinition",
attrs: { id },
content: [para(...inline)],
});
const list = (...defs) => ({ type: "footnotesList", content: defs });
const doc = (...content) => ({ type: "doc", content });
// --- Normalization + merge of glyph forks ----------------------------------
test("typographic double quotes «…» vs \"…\" merge into one", () => {
const d = doc(
para(txt("a"), ref("A"), txt(" b"), ref("B")),
list(def("A", txt("«word»")), def("B", txt('"word"'))),
);
const out = normalizeAndMergeFootnotes(d);
// Both references now point at the first definition's id.
assert.deepEqual(refIds(out), ["A", "A"]);
// Surviving text is ASCII-normalized.
assert.equal(defText(defs(out)[0]), '"word"');
// Duplicate def kept its id (canonicalizer removes it as an orphan later).
const canon = canonicalizeFootnotes(out);
assert.deepEqual(defIds(canon), ["A"]);
assert.equal(findAll(canon, "footnotesList").length, 1);
});
test("em/en dash and hyphen merge", () => {
const d = doc(
para(ref("A"), ref("B"), ref("C")),
list(
def("A", txt("see — here")),
def("B", txt("see – here")),
def("C", txt("see - here")),
),
);
const out = normalizeAndMergeFootnotes(d);
assert.deepEqual(refIds(out), ["A", "A", "A"]);
assert.equal(defText(defs(out)[0]), "see - here");
});
test("NBSP and extra spaces merge with normal spacing", () => {
const d = doc(
para(ref("A"), ref("B")),
list(
def("A", txt("foo bar")), // NBSP
def("B", txt("foo bar")), // collapsed spaces
),
);
const out = normalizeAndMergeFootnotes(d);
assert.deepEqual(refIds(out), ["A", "A"]);
assert.equal(defText(defs(out)[0]), "foo bar");
});
test("same text but different styling (bold vs plain) does NOT merge", () => {
const d = doc(
para(ref("A"), ref("B")),
list(
def("A", txt("word", [{ type: "bold" }])),
def("B", txt("word")),
),
);
const out = normalizeAndMergeFootnotes(d);
// No re-hang: references keep their own ids.
assert.deepEqual(refIds(out), ["A", "B"]);
assert.deepEqual(defIds(out), ["A", "B"]);
// Marks preserved on the surviving text node.
assert.deepEqual(defs(out)[0].content[0].content[0].marks, [
{ type: "bold" },
]);
});
test("same text but a link mark with different href does NOT merge (data-loss guard)", () => {
const d = doc(
para(ref("A"), ref("B")),
list(
def("A", txt("source", [{ type: "link", attrs: { href: "https://a.example/1" } }])),
def("B", txt("source", [{ type: "link", attrs: { href: "https://b.example/2" } }])),
),
);
const out = normalizeAndMergeFootnotes(d);
// No re-hang: each reference keeps its own definition (distinct link target).
assert.deepEqual(refIds(out), ["A", "B"]);
assert.deepEqual(defIds(out), ["A", "B"]);
// Both distinct hrefs survive.
assert.deepEqual(
defs(out).map((dn) => dn.content[0].content[0].marks[0].attrs.href),
["https://a.example/1", "https://b.example/2"],
);
// Canonicalize keeps both as two tail entries (neither is an orphan).
const canon = canonicalizeFootnotes(out);
assert.deepEqual(defIds(canon), ["A", "B"]);
assert.deepEqual(refIds(canon), ["A", "B"]);
});
test("same text and SAME link href still merges (attrs-aware key doesn't over-separate)", () => {
const d = doc(
para(ref("A"), ref("B")),
list(
def("A", txt("source", [{ type: "link", attrs: { href: "https://a.example/1" } }])),
def("B", txt("source", [{ type: "link", attrs: { href: "https://a.example/1" } }])),
),
);
const out = normalizeAndMergeFootnotes(d);
assert.deepEqual(refIds(out), ["A", "A"]);
const canon = canonicalizeFootnotes(out);
assert.deepEqual(defIds(canon), ["A"]);
assert.equal(findAll(canon, "footnotesList").length, 1);
});
test("marks are kept on merged (surviving) definition text", () => {
const d = doc(
para(ref("A"), ref("B")),
list(
def("A", txt("«x»", [{ type: "italic" }])),
def("B", txt("«x»", [{ type: "italic" }])),
),
);
const out = normalizeAndMergeFootnotes(d);
assert.deepEqual(refIds(out), ["A", "A"]);
assert.deepEqual(defs(out)[0].content[0].content[0].marks, [
{ type: "italic" },
]);
assert.equal(defText(defs(out)[0]), '"x"');
});
// --- Inline code is verbatim (not typography) ------------------------------
test("text inside a code mark is left verbatim; prose in the same def is normalized", () => {
const d = doc(
para(ref("A")),
list({
type: "footnoteDefinition",
attrs: { id: "A" },
content: [
para(
txt("a—b «x»", [{ type: "code" }]),
txt(" prose «y» — z"),
),
],
}),
);
const out = normalizeAndMergeFootnotes(d);
const nodes = findAll(defs(out)[0], "text");
// Code node: byte-for-byte unchanged (typography preserved).
assert.equal(nodes[0].text, "a—b «x»");
// Prose node: dashes/quotes normalized to ASCII.
assert.equal(nodes[1].text, ' prose "y" - z');
});
test("two notes differing ONLY by glyphs inside a code mark do NOT merge", () => {
const d = doc(
para(ref("A"), ref("B")),
list(
def("A", txt("«x»", [{ type: "code" }]), txt(" same prose «q»")),
def("B", txt('"x"', [{ type: "code" }]), txt(" same prose «q»")),
),
);
const out = normalizeAndMergeFootnotes(d);
// Prose is identical after normalization, but the code literals differ raw
// -> the merge key diverges -> both definitions survive, no re-hang.
assert.deepEqual(refIds(out), ["A", "B"]);
assert.deepEqual(defIds(out), ["A", "B"]);
// Each code literal stays verbatim.
assert.equal(defs(out)[0].content[0].content[0].text, "«x»");
assert.equal(defs(out)[1].content[0].content[0].text, '"x"');
// Both survive canonicalization (neither is an orphan).
const canon = canonicalizeFootnotes(out);
assert.deepEqual(defIds(canon), ["A", "B"]);
assert.deepEqual(refIds(canon), ["A", "B"]);
});
// --- Composition with the canonicalizer ------------------------------------
test("pass + canonicalize: single tail list and sequential numbering", () => {
const d = doc(
para(txt("intro "), ref("A"), txt(" middle "), ref("B")),
list(def("A", txt("«note»")), def("B", txt('"note"'))),
);
const canon = canonicalizeFootnotes(normalizeAndMergeFootnotes(d));
assert.equal(findAll(canon, "footnotesList").length, 1);
assert.deepEqual(defIds(canon), ["A"]);
assert.deepEqual(refIds(canon), ["A", "A"]);
});
// --- Idempotency -----------------------------------------------------------
test("idempotent: a second run is a no-op", () => {
const d = doc(
para(ref("A"), ref("B")),
list(def("A", txt("«word»")), def("B", txt('"word"'))),
);
const once = normalizeAndMergeFootnotes(d);
const twice = normalizeAndMergeFootnotes(once);
assert.deepEqual(twice, once);
});
test("input document is not mutated (pure)", () => {
const d = doc(
para(ref("A"), ref("B")),
list(def("A", txt("«word»")), def("B", txt('"word"'))),
);
const snapshot = JSON.parse(JSON.stringify(d));
normalizeAndMergeFootnotes(d);
assert.deepEqual(d, snapshot);
});
// --- Nested definitions ----------------------------------------------------
test("definitions nested in a callout are normalized and merged", () => {
const callout = (...content) => ({
type: "callout",
attrs: { type: "info" },
content,
});
const d = doc(
para(ref("A"), ref("B")),
callout(list(def("A", txt("«c»")), def("B", txt('"c"')))),
);
const out = normalizeAndMergeFootnotes(d);
assert.deepEqual(refIds(out), ["A", "A"]);
assert.equal(defText(defs(out)[0]), '"c"');
});
// --- Empty footnotes -------------------------------------------------------
test("empty footnotes do NOT collapse into each other", () => {
const d = doc(
para(ref("A"), ref("B")),
list(def("A", txt("")), { type: "footnoteDefinition", attrs: { id: "B" }, content: [{ type: "paragraph" }] }),
);
const out = normalizeAndMergeFootnotes(d);
// Both empty definitions keep distinct ids; references unchanged.
assert.deepEqual(refIds(out), ["A", "B"]);
assert.deepEqual(defIds(out), ["A", "B"]);
});
// --- Body text left untouched ----------------------------------------------
test("body text (outside footnotes) is NOT normalized", () => {
const d = doc(
para(txt("body «quoted» — dash"), ref("A")),
list(def("A", txt("«note»"))),
);
const out = normalizeAndMergeFootnotes(d);
// Body paragraph keeps its typographic glyphs verbatim.
assert.equal(out.content[0].content[0].text, "body «quoted» — dash");
// Footnote text IS normalized.
assert.equal(defText(defs(out)[0]), '"note"');
});
// --- Multi-paragraph structure preserved -----------------------------------
test("multi-paragraph definition: text normalized, structure preserved", () => {
const d = doc(
para(ref("A")),
list({
type: "footnoteDefinition",
attrs: { id: "A" },
content: [para(txt("«p1»")), para(txt("p2 — end"))],
}),
);
const out = normalizeAndMergeFootnotes(d);
const def0 = defs(out)[0];
assert.equal(def0.content.length, 2);
assert.equal(def0.content[0].content[0].text, '"p1"');
assert.equal(def0.content[1].content[0].text, "p2 - end");
});
// --- Multi-reference footnote not broken -----------------------------------
test("one id shared by multiple references is preserved", () => {
const d = doc(
para(ref("A"), txt(" x "), ref("A")),
list(def("A", txt("note"))),
);
const out = normalizeAndMergeFootnotes(d);
assert.deepEqual(refIds(out), ["A", "A"]);
assert.deepEqual(defIds(out), ["A"]);
});
// --- Whole-definition edge trim --------------------------------------------
test("leading/trailing whitespace is trimmed for the merge and stored text", () => {
const d = doc(
para(ref("A"), ref("B")),
list(def("A", txt(" hello ")), def("B", txt("hello"))),
);
const out = normalizeAndMergeFootnotes(d);
assert.deepEqual(refIds(out), ["A", "A"]);
assert.equal(defText(defs(out)[0]), "hello");
});
@@ -140,6 +140,36 @@ test("typo fix wrapped in markdown still applies (not refused)", () => {
assert.deepEqual(node.marks, [{ type: "bold" }]);
});
// ---------------------------------------------------------------------------
// (iv) #410 footnote token: a `replace` containing `^[...]` is refused into
// failed[] (it would be written as a LITERAL string, never a real footnote).
// Nothing is applied; the reason points at insert_footnote.
// ---------------------------------------------------------------------------
test("replace containing a `^[...]` footnote token is refused, not applied", () => {
const input = doc(paragraph(textNode("The claim stands.")));
const snapshot = JSON.parse(JSON.stringify(input));
const { doc: out, results, failed } = applyTextEdits(input, [
{ find: "The claim stands.", replace: "The claim stands.^[See source, p.42]" },
]);
assert.equal(results.length, 0, "nothing applied");
assert.equal(failed.length, 1, "one refused edit");
assert.equal(failed[0].find, "The claim stands.");
assert.match(failed[0].reason, /insert_footnote/);
// The document is byte-for-byte untouched — no literal `^[` was written.
assert.deepEqual(out, snapshot);
});
test("a plain replace with no footnote token still applies (no false positive)", () => {
const input = doc(paragraph(textNode("a caret ^ and a bracket ] apart")));
const { results, failed } = applyTextEdits(input, [
{ find: "apart", replace: "separate" },
]);
assert.equal(failed.length, 0, "not refused");
assert.equal(results.length, 1, "applied");
});
// ---------------------------------------------------------------------------
// A plain text fix is unaffected by the refuse logic.
// ---------------------------------------------------------------------------
@@ -0,0 +1,67 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { timeToolHandler } from "../../build/index.js";
// #402 — the registerTool wrapper times every tool through a single choke point
// and feeds the host's dependency-neutral onMetric sink. These assert the
// timing contract without a live transport (the factory monkeypatches
// server.registerTool with exactly this helper).
test("times a tool and preserves the handler's return value", async () => {
const calls = [];
const onMetric = (name, value, labels) => calls.push({ name, value, labels });
const handler = async (arg) => ({ ok: true, echo: arg });
const wrapped = timeToolHandler("get_page", handler, onMetric);
const result = await wrapped("hello");
// Return value passes through untouched.
assert.deepEqual(result, { ok: true, echo: "hello" });
// Exactly one sample, correct name/labels, numeric non-negative duration.
assert.equal(calls.length, 1);
assert.equal(calls[0].name, "mcp_tool_duration_seconds");
assert.deepEqual(calls[0].labels, { tool: "get_page" });
assert.equal(typeof calls[0].value, "number");
assert.ok(calls[0].value >= 0, "duration must be non-negative seconds");
});
test("standalone (no onMetric) is a transparent pass-through", async () => {
const handler = async (a, b) => a + b;
const wrapped = timeToolHandler("noop_tool", handler, undefined);
// No throw, result unchanged, and nothing to observe.
const result = await wrapped(2, 3);
assert.equal(result, 5);
});
test("still observes on throw, then rethrows the original error", async () => {
const calls = [];
const onMetric = (name, value, labels) => calls.push({ name, value, labels });
const boom = new Error("handler failed");
const handler = async () => {
throw boom;
};
const wrapped = timeToolHandler("boom_tool", handler, onMetric);
await assert.rejects(() => wrapped(), (err) => err === boom);
// Observed exactly once despite the throw.
assert.equal(calls.length, 1);
assert.equal(calls[0].name, "mcp_tool_duration_seconds");
assert.deepEqual(calls[0].labels, { tool: "boom_tool" });
});
test("standalone still rethrows the original error (no swallow, no observe)", async () => {
const boom = new Error("standalone failure");
const wrapped = timeToolHandler(
"boom_standalone",
async () => {
throw boom;
},
undefined,
);
await assert.rejects(() => wrapped(), (err) => err === boom);
});
+62
View File
@@ -0,0 +1,62 @@
diff --git a/dist/hocuspocus-server.cjs b/dist/hocuspocus-server.cjs
index b24ff6d091c32f733089eeaa47b03f7b37cf5964..f003af304fc751b7edc1aee17f3651282d70666a 100644
--- a/dist/hocuspocus-server.cjs
+++ b/dist/hocuspocus-server.cjs
@@ -2426,6 +2426,26 @@ class Hocuspocus {
* Create a new document by the given request
*/
async createDocument(documentName, request, socketId, connection, context) {
+ // PATCH(gitmost #401): close the connect-vs-unload race. When the last
+ // client disconnects, storeDocumentHooks' finally schedules an unload;
+ // unloadDocument runs its beforeUnloadDocument hooks asynchronously and
+ // records an in-flight promise in `unloadingDocuments`. A NEW connection
+ // arriving in that window would otherwise fall straight through to the
+ // `documents`/`loadingDocuments` checks and could either reuse a doc that
+ // is about to be destroyed, or start loading a fresh doc concurrently
+ // with the destroy. Awaiting the in-flight unload first makes the decision
+ // deterministic: once it settles, either the doc was fully unloaded
+ // (removed from `documents`, so we do a clean fresh load below) or the
+ // unload aborted because work/connections reappeared (the healthy doc is
+ // still in `documents`, so we reuse it). Either way the new connection can
+ // never hand-shake onto an orphaned, about-to-be-destroyed Document.
+ const existingUnloadingDoc = this.unloadingDocuments.get(documentName);
+ if (existingUnloadingDoc) {
+ // Wait for the in-flight unload to settle so we never hand-shake onto a dying
+ // Document. Swallow a rejected unload — fall through to a fresh load, matching
+ // pre-patch behavior (the doc is already removed from `documents` by then).
+ try { await existingUnloadingDoc; } catch { /* unload rejected — fresh load */ }
+ }
const existingLoadingDoc = this.loadingDocuments.get(documentName);
if (existingLoadingDoc) {
return existingLoadingDoc;
diff --git a/dist/hocuspocus-server.esm.js b/dist/hocuspocus-server.esm.js
index 1f4dd80244e899128e2c4e5dad8eab7cfc1cbad6..8c2411747bba27fb9486e1df81678a14e41e884e 100644
--- a/dist/hocuspocus-server.esm.js
+++ b/dist/hocuspocus-server.esm.js
@@ -2406,6 +2406,26 @@ class Hocuspocus {
* Create a new document by the given request
*/
async createDocument(documentName, request, socketId, connection, context) {
+ // PATCH(gitmost #401): close the connect-vs-unload race. When the last
+ // client disconnects, storeDocumentHooks' finally schedules an unload;
+ // unloadDocument runs its beforeUnloadDocument hooks asynchronously and
+ // records an in-flight promise in `unloadingDocuments`. A NEW connection
+ // arriving in that window would otherwise fall straight through to the
+ // `documents`/`loadingDocuments` checks and could either reuse a doc that
+ // is about to be destroyed, or start loading a fresh doc concurrently
+ // with the destroy. Awaiting the in-flight unload first makes the decision
+ // deterministic: once it settles, either the doc was fully unloaded
+ // (removed from `documents`, so we do a clean fresh load below) or the
+ // unload aborted because work/connections reappeared (the healthy doc is
+ // still in `documents`, so we reuse it). Either way the new connection can
+ // never hand-shake onto an orphaned, about-to-be-destroyed Document.
+ const existingUnloadingDoc = this.unloadingDocuments.get(documentName);
+ if (existingUnloadingDoc) {
+ // Wait for the in-flight unload to settle so we never hand-shake onto a dying
+ // Document. Swallow a rejected unload — fall through to a fresh load, matching
+ // pre-patch behavior (the doc is already removed from `documents` by then).
+ try { await existingUnloadingDoc; } catch { /* unload rejected — fresh load */ }
+ }
const existingLoadingDoc = this.loadingDocuments.get(documentName);
if (existingLoadingDoc) {
return existingLoadingDoc;
+8 -2
View File
@@ -44,6 +44,9 @@ overrides:
ip-address: 10.1.1
patchedDependencies:
'@hocuspocus/server@3.4.4':
hash: d8dc66e5ec3b9d23a876b979f493b6aa901fd2d965be54729495da5136296a42
path: patches/@hocuspocus__server@3.4.4.patch
ai@6.0.134:
hash: f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9
path: patches/ai@6.0.134.patch
@@ -75,7 +78,7 @@ importers:
version: 3.4.4(y-protocols@1.0.6(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810)))(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810))
'@hocuspocus/server':
specifier: 3.4.4
version: 3.4.4(y-protocols@1.0.6(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810)))(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810))
version: 3.4.4(patch_hash=d8dc66e5ec3b9d23a876b979f493b6aa901fd2d965be54729495da5136296a42)(y-protocols@1.0.6(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810)))(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810))
'@hocuspocus/transformer':
specifier: 3.4.4
version: 3.4.4(@tiptap/core@3.20.4(@tiptap/pm@3.20.4))(@tiptap/pm@3.20.4)(y-prosemirror@1.3.7(prosemirror-model@1.25.1)(prosemirror-state@1.4.3)(prosemirror-view@1.40.0)(y-protocols@1.0.6(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810)))(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810)))(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810))
@@ -1041,6 +1044,9 @@ importers:
marked:
specifier: ^17.0.1
version: 17.0.5
pako:
specifier: ^2.0.3
version: 2.0.3
re2:
specifier: ^1.21.0
version: 1.25.0
@@ -13055,7 +13061,7 @@ snapshots:
- bufferutil
- utf-8-validate
'@hocuspocus/server@3.4.4(y-protocols@1.0.6(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810)))(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810))':
'@hocuspocus/server@3.4.4(patch_hash=d8dc66e5ec3b9d23a876b979f493b6aa901fd2d965be54729495da5136296a42)(y-protocols@1.0.6(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810)))(yjs@13.6.30(patch_hash=1ceeb66dba1f86545c98a3ff7f5152aff9b35caf409091cef9caedb5e65c8810))':
dependencies:
'@hocuspocus/common': 3.4.4
async-lock: 1.4.1