Compare commits

..

20 Commits

Author SHA1 Message Date
agent_coder 3d8bc655eb docs(mcp): точный коммент toolAbortSignal — set-and-leave, не restore (#487, ревью nit)
Внутреннее ревью: docstring поля/метода toolAbortSignal утверждал «restores the
prior value on unwind», но wrapInAppToolWithCap намеренно НЕ восстанавливает
сигнал (set-and-leave) — именно это заставляет брошенного проигравшего race
бросить на следующем safe-point; корректность держится на свежем клиенте на ход +
перезаписи следующим вызовом. Комментарии приведены в соответствие с механизмом.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 05:34:46 +03:00
agent_coder b8aac9635b feat(ai-chat): ретраи finalizeAssistant + owner-write приоритет + двусторонний reconcile (#487)
Раньше finalizeAssistant ставил finalized=true ДО записи и не ретраил → один
неудачный UPDATE = строка вечно 'streaming'; свип был boot-only; run-свип
безусловный — асимметрия «run succeeded / message streaming навсегда».

- finalizeAssistant: bounded-ретраи; once-гейт закрывается ТОЛЬКО после успешной
  записи; возвращает ok. Правило owner-write: терминальная запись owner'а
  условна на status='streaming' OR metadata.finalizeFailed (repo.finalizeOwner) —
  перетирает reconcile-штамп, но не проставленный терминал. ВСЕ status-only
  штампы reconcile (stampTerminalIfStreaming, sweepStreaming) пишут строго
  onlyIfStreaming И мёржат metadata.finalizeFailed:true (иначе поздний owner-write
  не перетрёт).
- Порядок: попытка message-finalize → ран финализируется ВСЕГДА; при провале
  message onFinish помечает ран 'error' (не 'completed'). Ран не гейтится на
  message.
- Периодический reconcile-джоб (setInterval, env-tunable) клаузами по порядку:
  (a) пере-драйв зомби; (b) message streaming + ран терминален → штамп по статусу
  рана (succeeded-ран + зависшая строка → 'aborted'+finalizeFailed, НЕ
  'completed'-empty); (c) run running + НЕТ entry И НЕТ zombie + staleness →
  aborted (гейт «нет entry» первичный, staleness от last-progress updatedAt,
  X=max(2×per-call cap,15мин)+boot-warn); (d) message streaming + возраст>X + нет
  активной run-строки → aborted (двойной гейт). isInterruptResume исключает
  finalizeFailed-строки. Оппортунистический одно-чатовый reconcile при старте хода
  (best-effort, не фейлит ход). sweepStreaming boot-only → периодический.

Тесты (реальная БД): owner finalizeOwner чистит finalizeFailed; штамп не перетирает
терминал; поздний owner-write перетирает aborted-штамп; клаузы b/c/d (+живой entry
не трогать, двойной гейт d); «убить БД на finish → после восстановления ни строка,
ни ран не застряли». Юниты: finalizeFailed исключает interrupt-resume;
reconcileStaleRuns «нет entry».

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 05:15:46 +03:00
agent_coder d72e101216 feat(ai-chat): единый гейт конкурентности + серверный supersede (CAS) (#487)
Раньше в legacy-режиме (дефолт!) гейта конкурентности НЕ было вообще — проверка
409 сидела внутри if (autonomousRuns && chatId): два таба = два параллельных
стрима на один чат (интерливинг истории, падения convertToModelMessages).

Серверная часть (клиентская лестница ретраев — отдельной клиентской итерацией
FSM, см. ниже):
- Run-строка ОБЯЗАТЕЛЬНА в ОБОИХ режимах: legacy тоже проходит beginRun/
  finalizeRun (гейт партиального уникального индекса + runId в start-метаданных).
  Различие режимов сведено к семантике abort: legacy onClose при дисконнекте
  зовёт requestStop(runId) вместо аборта сокет-сигнала (которого streamText уже
  не потребляет). Второй таб в ЛЮБОМ режиме → 409 A_RUN_ALREADY_ACTIVE.
- Supersede CAS POST /stream { supersede: { runId: X } } (AiChatRunService
  .supersede): валидация X.chatId===body.chatId (иначе 400 SUPERSEDE_INVALID);
  нет активного рана → degrade в обычный send; активный ≠ X → 409
  SUPERSEDE_TARGET_MISMATCH + текущий runId; активный = X → requestStop →
  awaitSettled (таймаут W=10c) → «записано/сдался» (сдался → settleZombie
  применяет intended условным UPDATE) → ready; таймаут → 409 SUPERSEDE_TIMEOUT,
  ничего не персистится. W обоснован race-on-abort'ом коммита 1; DB-brownout →
  TIMEOUT штатен, W не увеличивать (env-tunable).
- Задокументированы ограничения: нет квиесценции сайд-эффектов (в промпт нового
  рана добавлена строка SUPERSEDE_NOTE «предыдущий ран прерван, его последние
  операции могли примениться с задержкой»); кража слота между освобождением и
  beginRun (→ MISMATCH с новым runId, бэкстоп — уникальный индекс).

Тесты: два параллельных старта (оба режима) → строго 409 второму; supersede при
живом долгом ТУЛЕ (не UPDATE-задержке) settle'ится быстро; все CAS-ветки; дубль
supersede-POST → degrade; зомби-путь через settleZombie; HTTP-маппинг ветвей.
Кейс «beginRun ok → insert user-msg упал → ран settled, слот свободен» покрыт
lifecycle-спекой против реального safety-net catch.

КЛИЕНТ (deferred, точно flagged): удаление лестницы SUPERSEDE_RETRY_DELAYS_MS/
isRunAlreadyActive/supersedeRetryRef и переход на supersede-в-body требуют
адопции runId из start-метаданных (сейчас клиент runId не трекает вовсе) — это
и есть та «итерация клиентского FSM», которую данный серверный контракт
разблокирует. Не трогаю фрагильный клиентский FSM без возможности E2E-валидации
в браузере, чтобы не внести непроверяемую регрессию.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 05:15:46 +03:00
agent_coder 89fcc60782 feat(ai-chat): зомби-механика finalizeRun + settledPromise + условные терминальные UPDATE (#487)
Раньше give-up-путь finalizeRun ВОССТАНАВЛИВАЛ entry (зомби неотличим от живого
рана), а терминальный runRepo.update был БЕЗУСЛОВНЫМ (последний писатель перетирал
терминальный статус).

- Все терминальные UPDATE ранов теперь УСЛОВНЫЕ: новый repo.finalizeIfActive
  пишет только пока строка pending|running (зеркально onlyIfStreaming у сообщений)
  → двойной settle схлопывается в benign no-op, терминальный статус не перетереть.
- При исчерпании ретраев finalizeRun НЕ восстанавливает entry, а оставляет
  ЗОМБИ-запись { terminalWriteFailed, intended:{status,error} } в отдельной map;
  settleZombie пере-драйвит intended условным UPDATE (зовётся reconcile/supersede/
  boot sweep). Зомби удаляется после успешного UPDATE или обнаружения строки уже
  терминальной.
- Per-run settledPromise в ОТДЕЛЬНОЙ map runId→deferred (переживает active.delete),
  создаётся в beginRun, резолвится ровно один раз исходом (записано/сдался); поздний
  подписчик через ранее взятую ссылку получает резолвленный; peekSettled: live
  deferred → зомби-синтез → undefined (читать строку). Обе map bounded.
- Задокументирована потеря (single-process): рестарт до пере-драйва → boot sweep
  пишет 'aborted' поверх фактического intended — неустранимо в phase 1.

Тесты: юниты give-up-пути (зомби вместо restore + notifier terminalWriteFailed +
settleZombie), двойного settle, позднего подписчика; интеграционные против реальной
Б, что условный finalizeIfActive не перетирает терминал и двойной settle схлопывается.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 05:15:46 +03:00
agent_coder 917c406489 fix(ai-chat): in-app тулы — race-on-abort + safe-points + per-call cap (#487)
In-app тул-обёртки отбрасывали второй аргумент options с abortSignal — это был
единственный класс тулов без отмены и без wall-clock cap. Контент-мутации идут
через collab-WS (mutatePageContent), не через axios, поэтому «прокинуть signal в
axios» главный write-путь не покрывал.

- Переиспользован race-паттерн wrapToolWithCallTimeout: каждый in-app тул гонится
  против композитного сигнала (Stop + per-call cap); на abort — реджект
  немедленно, проигравший промис отбрасывается (латентность мс, не сетевой
  teardown — от этого зависит таймаут supersede в коммите 3).
- Safe-point проверки сигнала между последовательными вызовами paginateAll и
  пре-коммитная проверка в mutatePageContent (+ реентрантный близнец) через
  DocmostClient.setToolAbortSignal, который обёртка публикует перед каждым вызовом.
- Per-call cap покрывает весь вызов, env-tunable (AI_CHAT_INAPP_TOOL_CALL_CAP_MS,
  дефолт 2 мин).
- Задокументировано ограничение (#487): abort между вызовами многовызовного
  write-тула оставляет частично применённую операцию — отмена гарантирует «новый
  вызов не стартует», не «запись не доехала».

Тесты (честное свойство «после Stop не стартует новый HTTP/WS-вызов»): юнит
wrapInAppToolWithCap (реджект-на-abort, отсутствие новых вызовов, per-call cap,
публикация сигнала) + mock-HTTP тест реального paginateAll safe-point.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 05:15:46 +03:00
agent_coder 6bf9358387 fix(#486): ревью — CHANGELOG + AGENTS.md + два теста
Ре-ревью PR #500 (changes-requested, 4 мелких, все doc/test):

F1 [CHANGELOG] CHANGELOG.md [Unreleased]:
- Breaking Changes: metrics-листенер 0.0.0.0→127.0.0.1 — кросс-контейнерный
  скрейп (docmost:9464) молча умрёт без METRICS_BIND=0.0.0.0 + METRICS_TOKEN
  (миграция в .env.example).
- Security: утечка errorText тулов/провайдера анониму (closes #394);
  /metrics под Bearer (METRICS_TOKEN).
- Fixed: ioredis-утечка в /health; ELK вешал event loop; beginRun-призрак →
  честный 503 A_RUN_BEGIN_FAILED; ai drain-hang.

F2 [AGENTS.md] строка про ai-патч: теперь он несёт ДВА фикса (#184 O(n²)
partialOutput И #486 drain-hang), оба тривайра названы.

F3 [test] metrics.server.spec: добавлен кейс токена ТОЙ ЖЕ длины
(Bearer topsecreX) → 401 — пиннит constant-time сравнение (прежние кейсы
коротили на length-guard, до timingSafeEqual не доходили).

F4 [test] output-degeneration.spec: behavior-тест, гоняющий РЕАЛЬНЫЕ
onChunk/onStepFinish из stream() — длинный чистый шаг → граница → свежий
дегенеративный бёрст → ассерт abortSignal.aborted (было хардкодом
resetWatermark=0, ревёрт правки не краснил).

Мутационные пруфы (non-vacuous): F3 — форс compare→true роняет same-length
кейс (401→200); F4 — ревёрт lastDegenerationCheckLen=0 роняет behavior-тест
(aborted true→false). Оба восстановлены, специи зелёные (34/34).
2026-07-11 04:38:16 +03:00
agent_coder ccfbf3343b fix(#486): ревью — run-race контракт под новую политику + timing-safe metrics-токен
Ревью полной ветки #486 нашло два пункта в моих коммитах (3 и 4):

- BLOCKER (коммит 4): ai-chat.service.run-race.spec.ts (#184 F14) пинил СТАРУЮ
  политику «plain begin() failure → swallow + стрим UNTRACKED» (resolves
  toBeUndefined). Коммит 4 её развернул → тест падал (1 failed/6). Кейс
  ИНВЕРТИРОВАН под новую политику: plain begin-failure теперь REJECTS с 503
  A_RUN_BEGIN_FAILED, до первого байта, user-строка не вставлена, streamText не
  вызван — путь остаётся явно запинён, а не удалён.

- NIT (коммит 3): metrics-токен сравнивался наивным !== (единственный слой auth
  эндпоинта) → тайминг-утечка токена. Заменено на crypto.timingSafeEqual с
  length-guard (разная длина → reject), семантика 401/200 без изменений.

Внесено отдельным fixup-коммитом (rebase -i недоступен в окружении; ветка не
запушена). Тесты: run-race 7/7 + metrics.server 7/7 зелёные.
2026-07-11 03:54:50 +03:00
agent_coder b86c747245 chore(mcp): McpService.onModuleDestroy → destroyAllSessions + удаление мёртвого кода (#486)
onModuleDestroy чистил только sweep-таймер; loopback-collab-сессии держали доки
открытыми на collab-сервере до идла — рестарт мог гонять с доком, запиненным
умирающим воркером. Теперь дергает destroyAllSessions() через переопределяемый
шов (для юнит-теста без ESM-пакета), best-effort. Плюс удалён мёртвый код:
неиспользуемый импорт parseNodeArg в mcp/index.ts и мёртвые enum-члены
SEARCH_REMOVE_* в queue.constants (подтверждено grep'ом — ни диспетчера, ни
процессора).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:31:24 +03:00
agent_coder dc96736d44 fix(mcp): пробросить truncated в tree-mode listPages (#486)
listPages(tree:true) деструктурировал только pages из enumerateSpacePages и
возвращал голое дерево, теряя truncated — на неполном дереве (stdio-fallback BFS
упёрся в node-cap) вызывающий не знал, что страницы потеряны. Возвращаем
{ tree, truncated } (по образцу check_new_comments); основной /pages/tree путь
беспредельный, там false.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:28:08 +03:00
agent_coder ef1a2b3960 fix(auth): провенанс для API-key пути (#486)
validateApiKey возвращал результат до резолва провенанса — REST-записи по
is_agent API-ключу не получали маркер 'agent'. Перенести «выше» нельзя: payload
API-ключа не несёт подписанный actor-клейм, а user (с isAgent) неизвестен до
валидации ключа. Резолвим провенанс от возвращённого user: isAgent -> 'agent',
иначе 'user'; aiChatId у API-ключа всегда null (нет ai_chats-строки). Загрузка
EE ApiKeyService вынесена в переопределяемый шов resolveApiKeyService для юнит-
теста без EE-бандла.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:25:56 +03:00
agent_coder 0be44e307e fix(client): flushNext в onFinish гейтится mountedRef (#486)
Финальный onFinish->flushNext() не проверял live-mount флаг. Чистый onFinish
может прийти ПОСЛЕ анмаунта треда (New-chat / переключение чата мид-стрим —
асинхронные attach/resume оседают поздно): flush дергал очередь и re-POST'ил
сообщение из брошенного треда — «призрачные» отправки/чаты-призраки. Остальные
обращения к очереди уже гейтятся mountedRef; закрываем последнюю дыру.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:24:19 +03:00
agent_coder 9e99309096 fix(ai-chat): сброс lastDegenerationCheckLen в onStepFinish (#486)
onStepFinish обнулял inProgressText, но НЕ lastDegenerationCheckLen — а это
смещение В аккумулятор. После первого длинного шага протухшая (большая) отметка
делала throttle-условие отрицательным, и детектор токен-лупов молчал весь
следующий шаг, пока текст не перерастал старую отметку. Обнуляем отметку в
onStepFinish. Throttle-предикат вынесен в output-degeneration
(shouldCheckDegeneration + DEGENERATION_CHECK_STEP), чтобы юнит гонял ту же
логику, что и стрим.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:22:14 +03:00
agent_coder 433bd822b6 fix(mcp): сброс collab-токена на WS-auth-failure с ретраем (#486)
Кэш collab-токена (#435) инвалидировался только на HTTP-401/403 (REST-
интерцептор и login()); отклонённый Hocuspocus-handshake оставлял протухший
токен в кэше — каждая последующая мутация переотправляла тот же битый токен до
истечения TTL (минуты) без self-heal. collab-session помечает ошибку
onAuthenticationFailed маркером; клиентские write-швы (mutatePage/replacePage/
mutateLiveContentUnlocked) обёрнуты в writeWithCollabAuthRetry: на помеченной
ошибке кэш сбрасывается и запись ретраится ровно один раз со свежим токеном —
симметрично HTTP-пути.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:19:15 +03:00
agent_coder ece84924f3 fix(mcp): REGISTRY_STAMP хэширует всё src/**, а не только tool-specs.ts (#486)
Стамп детектил build/src-skew только по tool-specs.ts — правка client.ts,
client/*-модуля, comment-signal или drawio-* без пересборки проходила молча,
и build/ отдавал старый код. Теперь codegen и рантайм-лоадер хэшируют весь
src/**/*.ts (кроме *.generated.ts — иначе цикл через собственный выход),
симметрично: одинаковый обход, POSIX-сортировка, нормализация и sha256.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:16:02 +03:00
agent_coder a14174a726 fix(ai): drain-hang в writeToServerResponse — расширить patches/ai@6.0.134.patch (#486)
Серверная ai@6.0.134 в writeToServerResponse при backpressure (write()===false)
ждала ТОЛЬКО once('drain'). Если клиент отвалился мид-запись, сокет не дренится,
await не резолвится: read-цикл паркуется НАВСЕГДА, finally{response.end()}
недостижим, reader и буферы висят до рестарта. В autonomous ран продолжает лить
вывод после дисконнекта → КАЖДЫЙ дисконнект мид-ран оставляет висящий пайп.
Плюс read() — fire-and-forget с throw → unhandledRejection.

Патч расширен (index.js и index.mjs): Promise.race drain/close/error с гигиеной
once-слушателей (все три снимаются на первом settle — не копятся по одному на
stall); при close/error — reader.cancel() и выход (безопасно для detached-ранов:
независимый дренаж делает consumeStream); rejection read() поглощается с логом.
pnpm-lock patch_hash перегенерён (patch-commit). Выравнивание версии ai —
отдельный коммит (#495), не здесь.

Тест: трипвайр-спека в apps/server (только там резолвится патченная копия) по
образцу ai-sdk-partial-output.patch.spec — дисконнект мид-запись без drain
завершает ответ (end() вызван, reader не висит), read()-throw не даёт
unhandledRejection, оба dist-билда несут маркер PATCH(docmost #486).
2026-07-11 02:54:03 +03:00
agent_coder d822ceeaf7 fix(share): не утекать errorText тулов и провайдера анониму в публичном шэре (closes #394)
SECURITY. В публичном share-чате сырой текст ошибки тула или провайдера утекал
анонимному читателю. Три слоя, все обязательны:

(1) Рендер-гейт: prop showErrors в ToolCallCard (протянут через MessageList/
MessageItem), share-виджет передаёт false — сырой errorText не рисуется. Но
рендер-гейт маскирует только DOM, не байты.

(2) Санитизация на уровне share-тулсета (авторитетно): forShare оборачивает
execute каждого тула catch'ем. Своя ShareToolError (безопасные строки: «page not
available in this share») пробрасывается, ЛЮБАЯ другая ошибка → generic «tool
could not complete», полный текст только в серверный лог. Одно место закрывает
байты (атомарный tool-output-error фрейм v6), рендер и контекст модели; self-
correction сохранена.

(3) Анонимный onError пайпа: ShareToolError → её безопасное сообщение; иначе
describeProviderError (statusCode + тело: внутренний baseUrl/модель) только в лог,
читателю — фиксированная классифицированная строка (rate-limited/unavailable/
provider error).

Тест: интеграционный с РЕАЛЬНЫМ падением тула и провайдера — assert по СЫРЫМ SSE-
БАЙТАМ (не по DOM): секрет/baseUrl/стек отсутствуют, видна безопасная строка,
полный текст провайдера ушёл в серверный лог.
2026-07-11 02:50:19 +03:00
agent_coder 4ac014def2 fix(ai-chat): провал beginRun фейлит ход честным 503 (A_RUN_BEGIN_FAILED) (#486)
Раньше провал beginRun (кроме unique-violation), напр. blip пула БД, логировался
и ход ПРОДОЛЖАЛСЯ без run-строки. В autonomous такой ран никто не абортит: /stop
его не видит, дисконнект не абортит, one-run-гейт пропускает ВТОРОЙ ран — невидимый
неостанавливаемый ран до рестарта.

Теперь провал beginRun (кроме RunAlreadyActiveError → прежний 409) бросает
ServiceUnavailableException с кодом A_RUN_BEGIN_FAILED ДО первого байта и до
вставки user-строки (post-hijack catch контроллера отдаёт честный 503 на raw-
сокет). Без ветвления по режимам — #487 наследует ту же политику. В тело кладём
statusCode: 503 (object-arg исключение его не добавляет), чтобы клиент видел
статус.

Клиентский классификатор: ветка A_RUN_BEGIN_FAILED добавлена СТРОГО ДО generic-
503-матча — иначе показал бы «provider is not configured» вместо «временно,
повторите».

Тесты: unit fail-fast (stream() бросает 503 A_RUN_BEGIN_FAILED, ни байта в сокет,
user-строка не вставлена; RunAlreadyActiveError по-прежнему 409); unit клиентского
классификатора из ПОЛНОГО реального тела ответа с гвардом порядка.
2026-07-11 02:36:43 +03:00
agent_coder 799dcfe7c5 fix(metrics): bind на 127.0.0.1 по умолчанию + METRICS_BIND/METRICS_TOKEN (#486)
/metrics слушал на 0.0.0.0 без какой-либо аутентификации — auth-less
эндпоинт на всех интерфейсах. Теперь дефолтный bind — loopback 127.0.0.1;
env METRICS_BIND переопределяет интерфейс (0.0.0.0 для скрейпера в
отдельном контейнере, docmost:9464); опциональный METRICS_TOKEN включает
Bearer-аутентификацию (запросы без точного токена получают 401). Доки
скрейпа в .env.example обновлены.

Тест: unit на дефолтный bind + env-переопределение + резолв токена;
интеграционный по РЕАЛЬНОМУ сокету — listener забинден на 127.0.0.1,
без токена /metrics отдаётся, с токеном без/с неверным Bearer → 401, с
верным → 200.
2026-07-11 02:32:21 +03:00
agent_coder 14c864f5c6 fix(health): устранить утечку ioredis-клиента в /health-пробе (#486)
pingCheck строил new Redis(...) на КАЖДЫЙ вызов и делал disconnect() только
на success-пути. Пока Redis лежит, каждый тик health-пробы добавлял свежий,
вечно реконнектящийся клиент — неограниченный рост хэндлов/клиентов на всё
время недоступности Redis.

Теперь один долгоживущий probe-клиент, переиспользуемый между тиками:
lazyConnect (конструктор не бросает и не коннектится жадно),
maxRetriesPerRequest: 1 и enableOfflineQueue: false (проба фейлится быстро,
команды не буферизуются), плюс listener на 'error' (иначе unhandled error
роняет процесс). onModuleDestroy закрывает клиент при shutdown.

Тест: интеграционный — N проб при лежащем Redis (реальный refused-порт, не
мок поведения) создают РОВНО ОДИН клиент (на баге было бы N); onModuleDestroy
освобождает клиент, следующая проба лениво строит новый.
2026-07-11 02:29:41 +03:00
agent_coder e292ed5117 fix(mcp): ELK-лейаут в worker_thread — таймаут через terminate() (#486)
elkjs.layout() возвращает Promise, но саму раскладку крутит СИНХРОННО и
блокирует поток целиком. На in-app хосте это был главный event loop:
патологический граф у капа 500 узлов вешал ВСЕ HTTP/SSE/loopback. Прежняя
защита (Promise.race с setTimeout(5s)) была иллюзией — таймер физически не
мог сработать, пока тот же поток заблокирован внутри elkjs (комментарий в
коде это сам признавал).

Теперь elk.layout() исполняется в worker_thread, а таймаут форсится
worker.terminate() — единственный способ прервать синхронный JS. Главный
поток остаётся свободным; на таймауте/ошибке — best-effort откат к
исходной модели, как и раньше. Лживый комментарий «can never wedge the
server» убран.

Тесты: unit на terminate-по-таймауту (крошечный ceiling → hard-kill →
исходная модель нетронута) и бенчмарк-гард на worst-case графе у капа
(500 узлов/~1000 рёбер раскладывается, а главный event loop продолжает
тикать во время раскладки).
2026-07-11 02:25:28 +03:00
114 changed files with 7344 additions and 1614 deletions
+14
View File
@@ -335,6 +335,20 @@ MCP_DOCMOST_PASSWORD=
# VictoriaMetrics/Prometheus reaching it as <host>:<port>/metrics.
# METRICS_PORT=9464
#
# METRICS_BIND — interface the /metrics listener binds to. DEFAULT 127.0.0.1
# (loopback only), so the unauthenticated endpoint is NOT exposed on all
# interfaces. If the scraper runs in a SEPARATE container and reaches this as
# docmost:9464, set METRICS_BIND=0.0.0.0 — but then also set METRICS_TOKEN
# and/or keep the port on a private network, since /metrics is otherwise open.
# METRICS_BIND=127.0.0.1
#
# METRICS_TOKEN — optional Bearer token guarding /metrics. When set, every
# scrape MUST send `Authorization: Bearer <token>` (others get 401). Configure
# the scraper with the same bearer token (e.g. VictoriaMetrics/vmagent
# `bearer_token`, Prometheus `authorization.credentials`). Leave unset only
# when the endpoint is bound to loopback or an otherwise-trusted network.
# METRICS_TOKEN=
#
# 2) CLIENT_TELEMETRY_ENABLED — the public client perf-telemetry sink.
# OFF by default. When true, the unauthenticated POST /api/telemetry/vitals
# endpoint is registered and browsers collect + send web-vitals / editor
+3 -3
View File
@@ -334,7 +334,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked; jsdom (Node only) | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, `apps/server` (server-side markdown import/export, #345), AND `apps/client` (markdown paste/copy + AI-chat render, via the `browser` entry — native `DOMParser`, no jsdom in the client bundle, #347); there is exactly ONE copy of the converter now |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
@@ -460,7 +460,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, `apps/server` (#345), and `apps/client` (#347) — do NOT reintroduce a per-package copy. The client uses the package's `browser` entry (`@docmost/prosemirror-markdown/browser`): markdown paste (`markdown-clipboard.ts`), copy-as-markdown, and AI-chat rendering now all go through the canonical converter, so the hand-written `marked`/`turndown` markdown layer that used to live in `editor-ext` was deleted (#347). The browser entry runs the HTML→DOM stage on the native `DOMParser`, so jsdom stays out of the client bundle. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
@@ -470,7 +470,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- **Errors must never be swallowed or shown as generic messages.** Every caught error MUST (1) be logged in full to the console/logger — error name, message, stack, `cause`, and (for HTTP/provider failures) the status code and response body — and (2) be surfaced to the user with a *specific, human-readable explanation of what actually went wrong*, never a bare generic string like "Something went wrong" / "Could not start recording" / "Transcription failed". Include the real reason (the underlying error/provider message) in the user-facing text. On the server, wrap third-party/provider failures with `describeProviderError` (or equivalent) and rethrow as a meaningful HTTP status + message — never let them collapse into an opaque 500. On the client, `console.error(<context>, err)` the raw error AND show the extracted reason (e.g. `err.response?.data?.message`, or the error `name: message`) in the notification.
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire test: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`) — it MUST be re-created via `pnpm patch` when bumping `ai`.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch carries TWO independent server fixes, each with its own tripwire test: (1) it disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`); (2) it fixes `writeToServerResponse`'s drain-hang — the loop awaited only `"drain"` under backpressure, so a mid-write client disconnect parked the pipe forever and leaked the reader/buffers until restart; it now races `"drain"` against `"close"`/`"error"`, cancels the reader on disconnect, and swallows the fire-and-forget read rejection (#486; tripwire: `apps/server/src/integrations/ai/ai-sdk-drain-hang.patch.spec.ts`). Both tripwires assert BOTH installed dist builds carry their patch marker. The patch MUST be re-created via `pnpm patch` when bumping `ai`.
- **The MCP tool inventory in `SERVER_INSTRUCTIONS` is GENERATED from the registry** (`packages/mcp/src/server-instructions.ts`: `buildToolInventory()` over `SHARED_TOOL_SPECS`) and spliced into the hand-written routing prose (`ROUTING_PROSE`). So adding/renaming/removing a **shared** spec in `packages/mcp/src/tool-specs.ts` auto-updates the `<tool_inventory>` — no manual `SERVER_INSTRUCTIONS` edit needed. Only an **inline** MCP-only tool (those registered via `server.registerTool(...)` in `index.ts`, not through the registry) needs a one-line entry in `INLINE_MCP_INVENTORY`. Enforced by `packages/mcp/test/unit/tool-inventory.test.mjs`, which fails when a registered tool is missing from the generated inventory (there is no `EXCEPTIONS` opt-out anymore — every tool must appear). Update `ROUTING_PROSE` when a tool's *intent guidance* (when-to-use) changes. `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
+63 -12
View File
@@ -115,6 +115,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the old ProseMirror-JSON output. Released together with the `#411`/`#412`
breaking window so external configs break exactly once. (#413)
- **The Prometheus `/metrics` listener now binds to `127.0.0.1` (loopback) by
default instead of `0.0.0.0` (all interfaces).** This closes an unauthenticated
endpoint that was previously reachable on every interface. **DEPLOY MIGRATION —
cross-container scraping breaks silently otherwise:** if your scraper runs in a
SEPARATE container and reaches the app as `docmost:9464` (the exact topology the
old `0.0.0.0` hardcode served), you MUST now set `METRICS_BIND=0.0.0.0` — and,
because that re-exposes the endpoint, also set `METRICS_TOKEN=<secret>` and
configure the scraper with a matching Bearer token. Without `METRICS_BIND`, the
scraper can no longer connect and metrics go dark with no error. See the
`METRICS_BIND` / `METRICS_TOKEN` block in `.env.example` for the migration.
Same-host (loopback) scrapers need no change. (#486)
### Added
- **Place several images side by side in a row.** A new "Inline (side by
@@ -270,18 +282,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **Client markdown paste/copy and AI-chat rendering now go through the canonical
converter.** Pasting markdown into the editor, "Copy as markdown", the AI title
generator, and the AI-chat markdown renderer all now use
`@docmost/prosemirror-markdown` (via its new `browser` entry — native
`DOMParser`, no jsdom in the client bundle) instead of the hand-written
`marked`/`turndown` markdown layer in `editor-ext`, which was **deleted**. As a
result, pasting canonical markdown (`^[…]` footnotes, `<!--img …-->`,
`> [!type]` callouts, `$…$` math, `==…==` highlight, standalone `<!--subpages-->`
comments) now produces the SAME nodes the server import produces for the same
text. Chat/reasoning markdown now renders through the editor schema (list items
are wrapped in `<p>`; CSS keeps them tight). (#347)
- **Enabling a public share no longer auto-shares the whole sub-tree.** Turning
a page "Shared to web" now defaults to the page alone; descendant pages become
public only when you explicitly turn on the dedicated "Include sub-pages"
@@ -310,6 +310,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`tee()` branch of the stream result — a ~20-step, ~28k-chunk agent run
retained ~1.7 GB and OOM'd the 2 GB JS heap. Streaming granularity is
unchanged; the patch must be re-created if `ai` is ever bumped. (#184)
- **The server no longer leaks a hung stream pipe on every mid-run client
disconnect.** The same `ai@6.0.134` pnpm patch now also fixes the SDK's
`writeToServerResponse`, which awaited only a `"drain"` event under
backpressure: when a client disconnected mid-write the socket never drained, so
the write loop parked forever, `response.end()` was unreachable, and the stream
reader plus buffered chunks were pinned until process restart (every mid-run
disconnect in autonomous mode leaked one). The patch races `"drain"` against
`"close"`/`"error"`, cancels the reader and ends the response on disconnect, and
swallows the fire-and-forget read rejection instead of crashing on an
unhandledRejection. (#486)
- **A failed autonomous agent-run start no longer becomes an unstoppable ghost
run.** When `beginRun` failed for a transient reason (e.g. a DB-pool blip),
the turn previously continued with NO run row — invisible to `/stop`, not
aborted on disconnect, and able to slip a second run past the one-run-per-chat
gate, leaving an unstoppable run until restart. The turn now fails fast with an
honest `503 A_RUN_BEGIN_FAILED` before the first byte (no orphan state), and the
client shows a "temporary — please try again" message instead of a misleading
"provider not configured". (#486)
- **A pathological draw.io graph can no longer wedge the whole server.** The ELK
auto-layout (`layout:"elk"`) ran elkjs synchronously on the main event loop, so
a graph at the node/edge cap blocked ALL HTTP/SSE/loopback traffic while it
churned — and the old `setTimeout` "timeout" could never fire because the same
thread was blocked. Layout now runs in a worker thread with the timeout enforced
by `worker.terminate()`; the main loop stays responsive. (#486)
- **The `/health` Redis probe no longer leaks a client on every tick while Redis
is down.** It built a new `ioredis` client per probe and disconnected it only on
success, so during an outage each health tick added another forever-reconnecting
client (an unbounded handle leak). A single long-lived probe client is now
reused and closed on shutdown. (#486)
- **Internal links in exported Markdown no longer lose their visible text.** A
link whose target page name had no file extension (e.g. a bare title) was
collapsed to empty text during export, producing an unclickable, label-less
@@ -386,6 +419,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
share); any other value now returns the generic "not found" instead of
serving the page. (#218)
- **Tool and provider error text no longer leaks to anonymous readers in the
public-share AI chat.** A failing tool's raw error (which could carry an
internal page title or a stack fragment) and a provider error (which bundles the
provider `statusCode` and response body — potentially the internal baseUrl or
model name) were streamed verbatim to the anonymous reader over SSE. Errors are
now sanitized at the source: the share toolset collapses any unclassified tool
error to a safe generic string (safe, classified tool messages still pass
through for the model's self-correction), and the anonymous stream `onError`
maps provider failures to a fixed set of neutral strings — the full detail goes
only to the server log. A UI render gate is layered on top. (closes #394)
- **The Prometheus `/metrics` endpoint can now require Bearer authentication and
is loopback-bound by default.** Previously it listened on all interfaces with no
auth. Setting `METRICS_TOKEN` requires every scrape to present
`Authorization: Bearer <token>` (compared in constant time), and the listener
defaults to `127.0.0.1` (see the Breaking Changes entry for the cross-container
migration). (#486)
## [0.94.0] - 2026-06-26
This release makes AI chat durable and fast: assistant turns are persisted to
-1
View File
@@ -21,7 +21,6 @@
"@atlaskit/pragmatic-drag-and-drop-live-region": "1.3.4",
"@casl/react": "5.0.1",
"@docmost/editor-ext": "workspace:*",
"@docmost/prosemirror-markdown": "workspace:*",
"@excalidraw/excalidraw": "0.18.0-3a5ef40",
"@mantine/core": "8.3.18",
"@mantine/dates": "8.3.18",
@@ -55,15 +55,6 @@
padding-inline-start: 1.4em;
}
/* The canonical converter renders list items through the editor schema, which
wraps each item's content in a <p> (listItem content is `paragraph+`). Drop
that paragraph's block margin so list items render TIGHT (no extra vertical
gap), matching the previous marked output same rule already applied to
table cells above (issue #347). */
.markdown li p {
margin: 0;
}
/* GFM tables in assistant markdown. The chat lives in a NARROW side panel, so a
wide LLM table must scroll horizontally instead of collapsing its columns:
`.markdown` sets `word-break: break-word`, which (with the default table
@@ -181,14 +172,6 @@
margin: 0 0 4px;
}
/* Same as `.markdown li p` above: the canonical converter wraps every list
item's content in a <p>, so without this each reasoning-panel list item would
pick up `.reasoningText p`'s 4px bottom margin and render too loose. Drop it
so Reasoning-panel lists stay tight, mirroring the pre-#347 marked output. */
.reasoningText li p {
margin: 0;
}
.inputWrapper {
flex: 0 0 auto;
padding-top: var(--mantine-spacing-xs);
@@ -203,6 +203,52 @@ describe("ChatThread — send now (#198)", () => {
});
});
// #486: the final onFinish -> flushNext() must be gated on the live-mount flag.
// A clean onFinish can land AFTER the thread unmounts (New-chat / chat-switch
// mid-stream — the async attach/resume settles late); flushing then dequeues and
// re-POSTs a queued message from an abandoned thread (a "ghost" send).
describe("ChatThread — onFinish flush gated on mount (#486)", () => {
beforeEach(resetState);
afterEach(cleanup);
it("a clean onFinish WHILE MOUNTED flushes the queued message (control)", () => {
renderThread();
fireEvent.click(screen.getByTestId("queue-btn")); // enqueue "queued text"
expect(h.state.sendMessage).not.toHaveBeenCalled();
act(() => {
h.state.onFinish?.({
message: { id: "a", role: "assistant", parts: [] },
isAbort: false,
isDisconnect: false,
isError: false,
});
});
// Mounted: the queue flushes normally.
expect(h.state.sendMessage).toHaveBeenCalledWith({ text: "queued text" });
});
it("a clean onFinish AFTER unmount does NOT flush (no ghost send)", () => {
const { unmount } = renderThread();
fireEvent.click(screen.getByTestId("queue-btn")); // enqueue "queued text"
h.state.sendMessage.mockClear();
// Chat switched away mid-stream: the streamer unmounts...
unmount();
// ...and a late, clean onFinish lands on the abandoned thread.
act(() => {
h.state.onFinish?.({
message: { id: "a", role: "assistant", parts: [] },
isAbort: false,
isDisconnect: false,
isError: false,
});
});
// Gated on mountedRef: NOTHING is sent from the dead thread.
expect(h.state.sendMessage).not.toHaveBeenCalled();
});
});
// #396: in autonomous mode a live sendNow must additionally request the
// AUTHORITATIVE server stop of the detached run (a local abort is only a client
// disconnect the server ignores) and arm a bounded 409 retry so the re-POST
@@ -659,7 +659,13 @@ export default function ChatThread({
return;
}
if (isAbort || isDisconnect || isError) return;
flushNext();
// Gate the final flush on the live-mount flag (#486): a clean onFinish can
// land AFTER this thread unmounted (a New-chat / chat-switch mid-stream —
// the async attach/resume settles late). Flushing then dequeues and POSTs a
// queued message from an abandoned thread — a "ghost" send / ghost chat.
// Every other queue side effect already guards on mountedRef; this last one
// was the gap.
if (mountedRef.current) flushNext();
},
// `onError` runs in addition to `onFinish` (which ai@6 also calls on error).
// Log the raw failure here for devtools; the UI shows a friendly classified
@@ -47,6 +47,13 @@ interface MessageItemProps {
* agent's raw query/argument text.
*/
showInput?: boolean;
/**
* Forwarded to ToolCallCard: whether a failed tool card renders its raw
* errorText. Defaults to true (internal chat). The public share passes false so
* internal detail in a tool error is never painted (belt to the server-side
* byte sanitization).
*/
showErrors?: boolean;
/**
* Neutralize internal/relative markdown links in the rendered answer (drop
* their href so they become inert text). Defaults to false (internal chat,
@@ -125,6 +132,7 @@ function MessageItem({
message,
showCitations = true,
showInput = true,
showErrors = true,
neutralizeInternalLinks = false,
assistantName,
turnStreaming = false,
@@ -219,6 +227,7 @@ function MessageItem({
part={part as unknown as ToolUiPart}
showCitations={showCitations}
showInput={showInput}
showErrors={showErrors}
/>
);
}
@@ -284,6 +293,7 @@ export function arePropsEqual(
prev.signature === next.signature &&
prev.showCitations === next.showCitations &&
prev.showInput === next.showInput &&
prev.showErrors === next.showErrors &&
prev.neutralizeInternalLinks === next.neutralizeInternalLinks &&
prev.assistantName === next.assistantName &&
// The turn-end flip re-renders every row once (cheap, terminal event) —
@@ -32,6 +32,12 @@ interface MessageListProps {
* doesn't see the agent's raw query/argument text.
*/
showInput?: boolean;
/**
* Forwarded to MessageItem -> ToolCallCard: whether a failed tool card renders
* its raw errorText. Defaults to true (internal chat). The public share passes
* false so internal detail in a tool error is never painted.
*/
showErrors?: boolean;
/**
* Forwarded to MessageItem: neutralize internal/relative markdown links in
* the rendered answers (drop their href so they render as inert text).
@@ -127,6 +133,7 @@ export default function MessageList({
emptyState,
showCitations = true,
showInput = true,
showErrors = true,
neutralizeInternalLinks = false,
assistantName,
}: MessageListProps) {
@@ -217,6 +224,7 @@ export default function MessageList({
signature={messageSignature(message)}
showCitations={showCitations}
showInput={showInput}
showErrors={showErrors}
neutralizeInternalLinks={neutralizeInternalLinks}
assistantName={assistantName}
// Turn-level liveness, gated to the TAIL row: only the tail message
@@ -30,6 +30,16 @@ interface ToolCallCardProps {
* the extra summary line, leaving the card (the action log) intact.
*/
showInput?: boolean;
/**
* Whether to render the tool's raw errorText on a failed call. Defaults to true
* (the internal chat, where the operator may debug). The public share passes
* false: a tool error string can carry internal detail (an internal page title,
* a stack fragment, a provider message). This is the RENDER gate only the
* authoritative fix also sanitizes the bytes server-side (see
* PublicShareChatToolsService.forShare), so a share reader never receives raw
* error text over the wire, not just never sees it painted (#394).
*/
showErrors?: boolean;
}
/**
@@ -41,6 +51,7 @@ export default function ToolCallCard({
part,
showCitations = true,
showInput = true,
showErrors = true,
}: ToolCallCardProps) {
const { t } = useTranslation();
const toolName = getToolName(part);
@@ -74,7 +85,7 @@ export default function ToolCallCard({
</Text>
)}
{state === "error" && part.errorText && (
{state === "error" && showErrors && part.errorText && (
<Text size="xs" c="red" mt={2}>
{part.errorText}
</Text>
@@ -33,44 +33,29 @@ describe("collapseBlankLines", () => {
});
});
describe("collapseBlankLines + renderChatMarkdown (canonical converter)", () => {
// Chat markdown now renders through @docmost/prosemirror-markdown (issue #347):
// the SAME converter the editor/import use. Its list items are schema-shaped —
// each <li>'s content is wrapped in a <p> (listItem content is `paragraph+`) —
// so the HTML always carries `<li><p>…</p></li>` regardless of blank-line
// looseness in the source (the converter has no tight/loose distinction). The
// visual tightness that `collapseBlankLines` used to buy is now provided by
// CSS (`.markdown li p { margin: 0 }`), not the HTML shape.
it("renders a blank-line-separated bullet list as a real <ul> list", () => {
describe("collapseBlankLines + renderChatMarkdown (tight reasoning rendering)", () => {
it("renders a blank-line-separated list as a TIGHT list (no <li><p>)", () => {
const loose =
"Intro paragraph.\n\n- item one\n\n- item two\n\n- item three";
const html = renderChatMarkdown(collapseBlankLines(loose), {});
// Clean, un-namespaced HTML (DOMSerializer, not XMLSerializer) — no xmlns.
// Tight list: each <li> holds the text directly, not wrapped in a <p>.
expect(html).toContain("<li>item one</li>");
expect(html).not.toContain("<li><p>");
// The list still parses as a list after the paragraph (not a paragraph+<br>).
expect(html).toContain("<ul>");
expect(html).not.toMatch(/<ul[^>]*xmlns/);
// The item text is present (inside the schema's <li><p> wrapper).
expect(html).toContain("item one");
// The intro paragraph renders as its own paragraph before the list.
expect(html).toContain("<p>Intro paragraph.</p>");
});
it("renders an ordered list (1. 2.) as a real <ol> list", () => {
it("renders an ordered list (1. 2.) as tight after collapsing", () => {
const loose = "Intro.\n\n1. first\n\n2. second";
const html = renderChatMarkdown(collapseBlankLines(loose), {});
expect(html).toContain("<ol>");
expect(html).not.toMatch(/<ol[^>]*xmlns/);
expect(html).toContain("first");
expect(html).toContain("second");
expect(html).toContain("<li>first</li>");
expect(html).not.toContain("<li><p>");
});
it("wraps list-item content in <p> (schema shape; tightness is CSS)", () => {
// The canonical converter always wraps a list item's content in a paragraph,
// whether or not the source had blank lines between items.
it("the loose source WOULD render <li><p> without collapsing (control)", () => {
const loose = "- a\n\n- b";
expect(renderChatMarkdown(loose, {})).toContain("<li><p>");
// And a "tight" source produces the identical wrapping (no distinction).
expect(renderChatMarkdown(collapseBlankLines(loose), {})).toContain(
"<li><p>",
);
});
});
@@ -23,6 +23,25 @@ describe("describeChatError", () => {
});
});
it("classifies an A_RUN_BEGIN_FAILED 503 as a temporary run-start failure, NOT provider-not-configured (#486)", () => {
// The FULL real body the server writes for a beginRun failure: a
// ServiceUnavailableException(object) whose response is serialized verbatim
// onto the raw socket, self-describing statusCode 503 + the run-start code.
const body =
'{"message":"Could not start the agent run. This is usually temporary — please try again.","code":"A_RUN_BEGIN_FAILED","statusCode":503}';
expect(describeChatError(body, t)).toEqual({
title: "Could not start the run",
detail:
"The agent run could not be started. This is usually temporary — please try again.",
});
// ORDER GUARD: even though the body ALSO carries statusCode 503 (which the
// generic branch matches), the A_RUN_BEGIN_FAILED branch runs first, so it is
// never mislabeled "AI provider not configured".
expect(describeChatError(body, t).title).not.toBe(
"AI provider not configured",
);
});
it("classifies a dropped connection (ECONNRESET) as a lost-connection error", () => {
expect(
describeChatError("Cannot connect to API: read ECONNRESET", t).title,
@@ -24,6 +24,21 @@ export function describeChatError(
): ChatErrorView {
const msg = message ?? "";
// Our own "could not start the run" gate (A_RUN_BEGIN_FAILED, #486): a 503
// whose body carries this code is a TEMPORARY server-side failure while
// starting the run (e.g. a DB-pool blip), NOT an unconfigured provider. It MUST
// be matched STRICTLY BEFORE the generic 503 branch below, which would
// otherwise mislabel it "The AI provider is not configured" and tell the user
// to call an admin instead of just retrying.
if (/"code"\s*:\s*"A_RUN_BEGIN_FAILED"/.test(msg)) {
return {
title: t("Could not start the run"),
detail: t(
"The agent run could not be started. This is usually temporary — please try again.",
),
};
}
if (/"statusCode"\s*:\s*403\b/.test(msg)) {
return {
title: t("AI chat is disabled"),
@@ -1,37 +1,6 @@
import {
markdownToProseMirrorSync,
docmostExtensions,
} from "@docmost/prosemirror-markdown/browser";
import { getSchema } from "@tiptap/core";
import { Node as PMNode, DOMSerializer } from "@tiptap/pm/model";
import { markdownToHtml } from "@docmost/editor-ext";
import DOMPurify from "dompurify";
// The Docmost editor schema, built once. Chat markdown is rendered through the
// SAME schema the editor/import use (issue #347), so chat output matches how the
// page would render the same markdown.
const chatSchema = getSchema(docmostExtensions);
/**
* Markdown -> HTML for chat display, via the canonical converter. We serialize
* the ProseMirror doc with `DOMSerializer` into a real element and read its
* `innerHTML` (rather than `@tiptap/html`'s `generateHTML`, whose browser path
* uses `XMLSerializer` and stamps a `xmlns` on every block) so the markup is
* clean HTML. `li > p` wrapping is inherent to the schema (listItem content is
* `paragraph+`); the chat CSS zeroes those paragraph margins so lists still
* render tight.
*/
function markdownToChatHtml(markdown: string): string {
const doc = markdownToProseMirrorSync(markdown);
const node = PMNode.fromJSON(chatSchema, doc);
const div = document.createElement("div");
DOMSerializer.fromSchema(chatSchema).serializeFragment(
node.content,
{ document },
div,
);
return div.innerHTML;
}
export interface RenderChatMarkdownOptions {
/**
* Neutralize INTERNAL links so they render as inert text (no `href`/`target`).
@@ -94,32 +63,22 @@ function neutralizeInternalLinksHook(node: Element): void {
/**
* Render AI markdown to sanitized HTML for read-only display. We reuse the
* canonical converter (issue #347): markdown -> ProseMirror JSON (the SAME
* `markdownToProseMirrorSync` the editor paste/import path uses, so chat output
* matches the editor's markdown flavor) -> HTML via `markdownToChatHtml`
* (DOMSerializer), then sanitize with DOMPurify LLM output is untrusted, so it
* must never reach the DOM unsanitized.
* app's `markdownToHtml` (the same `marked` pipeline used for paste/import) so
* chat output matches the editor's markdown flavor, then sanitize with
* DOMPurify LLM output is untrusted, so it must never reach the DOM unsanitized.
*
* Stays SYNCHRONOUS: both callers render inside React (a memo and a useMemo),
* so the whole pipeline must resolve without awaiting. The converter's sync
* entry makes that possible; on any conversion error we return "" so the caller
* falls back to raw text (the same fallback the old Promise-guard produced).
* `markdownToHtml` can return `string | Promise<string>` (it has async marked
* extensions registered). In practice plain chat markdown resolves
* synchronously, but we guard the Promise case by returning a safe empty string
* for that branch (the caller renders the raw text fallback instead).
*/
export function renderChatMarkdown(
markdown: string,
options: RenderChatMarkdownOptions = {},
): string {
if (!markdown) return "";
let html: string;
try {
// markdown -> canonical PM JSON -> HTML (native DOMParser in the browser;
// jsdom is never bundled — see @docmost/prosemirror-markdown/browser).
html = markdownToChatHtml(markdown);
} catch {
// Malformed/unsupported markdown must not crash the chat render; fall back
// to raw text (empty return -> caller shows the plain-text branch).
return "";
}
const html = markdownToHtml(markdown);
if (typeof html !== "string") return "";
if (!options.neutralizeInternalLinks) {
// Internal chat: unchanged behavior, no hook registered.
@@ -1,206 +0,0 @@
import { describe, it, expect } from "vitest";
import { Editor } from "@tiptap/core";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Bold } from "@tiptap/extension-bold";
import { Italic } from "@tiptap/extension-italic";
import { MarkdownClipboard } from "./markdown-clipboard";
/**
* Integration coverage for the async `handlePaste` seam (issue #347). The paste
* conversion moved to `@docmost/prosemirror-markdown`'s browser entry, whose
* `markdownToProseMirror` is async so `handlePaste` captures the range, claims
* the event (returns true), and dispatches the insert on the next microtask.
* These tests drive that path end to end on a minimal schema (a plain-markdown
* paste whose converted nodes fit paragraph/text/bold/italic), asserting the
* text lands with the right marks and that the raw markdown syntax is consumed
* (recognized as markdown, not inserted literally).
*/
function makeEditor() {
const element = document.createElement("div");
document.body.appendChild(element);
return new Editor({
element,
extensions: [
Document,
Paragraph,
Text,
Bold,
Italic,
MarkdownClipboard.configure({ transformPastedText: true }),
],
content: { type: "doc", content: [{ type: "paragraph" }] },
});
}
// Locate the markdownClipboard plugin and invoke its handlePaste directly with a
// synthetic clipboard event (jsdom has no real paste pipeline). The plugin's
// handlePaste closes over the extension `this`, so calling it off the plugin
// props preserves `this.editor`/`this.options`.
function paste(editor: Editor, text: string): boolean {
const view = editor.view;
const plugin = view.state.plugins.find(
(p: any) => p.props && p.spec?.key,
) as any;
const event = {
clipboardData: {
getData: (type: string) => (type === "text/plain" ? text : ""),
},
} as unknown as ClipboardEvent;
// Find the specific handlePaste that belongs to the markdown clipboard plugin.
const md = view.state.plugins.find(
(p: any) => typeof p.props?.handlePaste === "function",
) as any;
return md.props.handlePaste(view, event, view.state.selection.content());
}
// Flush the microtask queue so the async .then() dispatch runs.
const flush = () => new Promise((r) => setTimeout(r, 0));
describe("MarkdownClipboard handlePaste (async md -> PM)", () => {
it("converts a plain-markdown paste with bold/italic into marked text", async () => {
const editor = makeEditor();
const claimed = paste(editor, "hello **bold** and *italic*");
// The paste is claimed synchronously (async insert follows).
expect(claimed).toBe(true);
await flush();
const json = editor.getJSON();
const text = JSON.stringify(json);
// The raw markdown asterisks are consumed (recognized), not inserted literally.
expect(editor.getText()).not.toContain("**");
expect(editor.getText()).toContain("bold");
expect(editor.getText()).toContain("italic");
// The bold/italic marks materialized.
expect(text).toContain('"bold"');
expect(text).toContain('"italic"');
editor.destroy();
});
it("recognizes a bullet list paste as list structure (not literal '-')", async () => {
// A bullet list is not representable in this minimal schema, so the converter
// output would fail PMNode.fromJSON and the catch inserts raw text. Use a
// paste whose nodes DO fit the schema to assert the happy path instead: two
// paragraphs separated by a blank line.
const editor = makeEditor();
paste(editor, "first para\n\nsecond para");
await flush();
const json = editor.getJSON() as any;
const paras = (json.content || []).filter(
(n: any) => n.type === "paragraph",
);
// Two paragraphs materialized from the blank-line-separated markdown.
expect(paras.length).toBeGreaterThanOrEqual(2);
expect(editor.getText()).toContain("first para");
expect(editor.getText()).toContain("second para");
editor.destroy();
});
it("falls back to raw text when conversion yields nodes the schema lacks", async () => {
// `# heading` converts to a `heading` node absent from this minimal schema,
// so PMNode.fromJSON throws and the catch re-inserts the raw text — the user
// never loses their clipboard content.
const editor = makeEditor();
paste(editor, "# a heading line");
await flush();
// Content is preserved (either as heading text or literal), never dropped.
expect(editor.getText()).toContain("a heading line");
editor.destroy();
});
});
// The async seam captures the target range synchronously, then replaces on the
// next microtask. If the document changed under it between capture and resolve
// (impossible in prod — same microtask — but pinned here), BOTH the success
// (replaceRange) and the fail-open (insertText) branches must fall back to the
// LIVE selection rather than a stale absolute range, so neither clobbers content
// nor throws a RangeError. We force the mid-flight change by dispatching a
// doc-mutating transaction AFTER the synchronous claim but BEFORE flushing the
// microtask that runs the `.then`/`.catch`.
describe("MarkdownClipboard handlePaste — doc-changed-mid-flight guard", () => {
// Replace the whole doc with one paragraph of `text` (synchronous dispatch).
// An empty string yields an empty paragraph (a text node may not be empty).
function seedContent(editor: Editor, text: string) {
editor.commands.setContent({
type: "doc",
content: [
text
? { type: "paragraph", content: [{ type: "text", text }] }
: { type: "paragraph" },
],
});
}
it("success branch: mid-flight doc change routes the paste to the LIVE selection, never the stale range (clobber-proving)", async () => {
// The paste captures a NON-EMPTY range {1,5} (over "AAAA"). Then, before the
// async resolve, the doc GROWS ("MARKER" inserted at the start) and the cursor
// is parked at the doc END. The captured {1,5} is now stale and points INTO
// "MARKER". A WORKING guard replaces at the live (end) selection → MARKER is
// untouched. A BROKEN guard replaces the stale {1,5} → it erases the first
// characters of MARKER (this is what a zero-width `from==to` range could never
// reveal, which is why the earlier version was vacuous).
const editor = makeEditor();
seedContent(editor, "AAAABBBB");
editor.commands.setTextSelection({ from: 1, to: 5 }); // captured range = {1,5}
const claimed = paste(editor, "hello **bold**");
expect(claimed).toBe(true);
// Mid-flight: grow the doc and move the cursor to a KNOWN-safe end position.
editor.view.dispatch(editor.view.state.tr.insertText("MARKER", 1));
const end = editor.state.doc.content.size;
editor.commands.setTextSelection({ from: end, to: end });
await flush();
const text = editor.getText();
// MARKER intact only if the guard used the live selection, not the stale range.
expect(text).toContain("MARKER");
expect(text).toContain("bold");
expect(text).not.toContain("**");
editor.destroy();
});
it("fail-open branch: a mid-flight doc SHRINK makes the stale `to` out of bounds — the guard must avoid a RangeError (throw-proving)", async () => {
// The paste captures a range {1,9} over an 8-char paragraph, then the
// conversion FAILS (`# heading` -> a heading node the minimal schema lacks,
// so PMNode.fromJSON throws -> the fail-open catch runs). Before the reject,
// the doc is SHRUNK to an empty paragraph, so the captured `to` (9) is now far
// past the doc's end. A WORKING guard inserts the raw text at the live (valid)
// selection → "raw heading" lands. A BROKEN guard does insertText(md, 1, 9) on
// a size-2 doc → RangeError, so the dispatch never runs and "raw heading" is
// absent (the assertion reddens). A zero-width/growing-doc setup could never
// push `to` out of bounds, which is why the earlier version was vacuous.
const editor = makeEditor();
seedContent(editor, "AAAABBBB");
editor.commands.setTextSelection({ from: 1, to: 9 }); // captured range = {1,9}
paste(editor, "# raw heading");
// Mid-flight: shrink the doc so the captured `to` = 9 is now out of bounds.
seedContent(editor, "");
await flush();
const text = editor.getText();
// Raw text lands (via the live selection) only if the guard avoided the
// stale, now-out-of-bounds range.
expect(text).toContain("raw heading");
editor.destroy();
});
it("two pastes in flight: neither payload is lost (no data loss)", async () => {
// Prod-unreachable (two paste events are separate macrotasks, and each
// conversion resolves on a microtask before the next), but pinned here: when
// both resolve back-to-back, the second sees the changed doc and inserts at
// the live selection the first left — so the two payloads may INTERLEAVE, but
// neither is dropped. We assert no data loss, not contiguity.
const editor = makeEditor();
paste(editor, "alphaword");
paste(editor, "betaword");
await flush();
const text = editor.getText();
// Neither payload fully dropped (interleaving may split one of them).
expect(text).toContain("alpha");
expect(text).toContain("beta");
editor.destroy();
});
});
@@ -1,12 +1,5 @@
import { describe, it, expect } from "vitest";
// Markdown conversion now goes through the canonical package's BROWSER entry
// (issue #347): the same converter the server import/export uses, resolved via
// the `browser` exports condition so it runs on the native `DOMParser` (the
// client jsdom vitest env provides one) with jsdom never bundled.
import {
convertProseMirrorToMarkdown,
markdownToProseMirrorSync,
} from "@docmost/prosemirror-markdown/browser";
import { htmlToMarkdown } from "@docmost/editor-ext";
import {
normalizeTableColumnWidths,
classifyClipboardSelection,
@@ -182,13 +175,10 @@ describe("classifyClipboardSelection", () => {
// Output-level tests for the table clipboard regression: copying a table must
// yield a real GFM pipe table, NOT one-value-per-line concatenated cells.
// These exercise the actual markdown produced by convertProseMirrorToMarkdown
// the same serializer step the clipboardTextSerializer now runs (issue #347) —
// so they pin the OUTPUT shape that the classifier-flag tests above do not cover.
// Input is ProseMirror JSON (what the copied slice serializes to), matching the
// clipboardTextSerializer's new call: it wraps the slice content in a synthetic
// `doc` (and the bare-rows case in a `table`) and calls the converter.
describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () => {
// These exercise the actual markdown produced by htmlToMarkdown (the same
// serializer step the clipboardTextSerializer runs), so they pin the OUTPUT
// shape that the classifier-flag tests above do not cover.
describe("table clipboard markdown output (htmlToMarkdown)", () => {
// Trim each line and drop blanks so structural assertions are whitespace-robust.
function lines(md: string): string[] {
return md
@@ -198,10 +188,10 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
}
// A GFM separator row like "| --- | --- |" (any number of columns), tolerant
// of the padding the serializer emits.
// of the padding turndown emits.
function isSeparatorRow(line: string): boolean {
const compact = line.replace(/\s+/g, "");
return /^\|(?::?-{2,}:?\|)+$/.test(compact);
return /^\|(?:-{3,}\|)+$/.test(compact);
}
// Split a pipe-delimited row into trimmed cell values.
@@ -213,33 +203,42 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
.map((c) => c.trim());
}
const cell = (t: string) => ({
type: "tableCell",
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
});
const headerCell = (t: string) => ({
type: "tableHeader",
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
});
const row = (nodes: any[]) => ({ type: "tableRow", content: nodes });
it("serializes a header-less partial cell selection (bare rows) as a valid GFM pipe table", () => {
// Mirror the serializer's `wrapBareRows` branch: bare tableRow nodes are
// wrapped in a synthetic `table` and convertProseMirrorToMarkdown is called
// (see markdown-clipboard.ts clipboardTextSerializer).
const rows = [
row([cell("a"), cell("b")]),
row([cell("c"), cell("d")]),
];
const md = convertProseMirrorToMarkdown({
type: "doc",
content: [{ type: "table", content: rows }],
});
// Mirror the serializer's `wrapBareRows` branch exactly: bare <tr> nodes are
// wrapped in <table><tbody> and htmlToMarkdown(div.innerHTML) is called.
// See markdown-clipboard.ts clipboardTextSerializer:
// const table = document.createElement("table");
// const tbody = document.createElement("tbody");
// tbody.appendChild(fragment); table.appendChild(tbody);
// div.appendChild(table);
// return htmlToMarkdown(div.innerHTML);
const div = document.createElement("div");
const table = document.createElement("table");
const tbody = document.createElement("tbody");
for (const [c1, c2] of [
["a", "b"],
["c", "d"],
]) {
const tr = document.createElement("tr");
const td1 = document.createElement("td");
td1.textContent = c1;
const td2 = document.createElement("td");
td2.textContent = c2;
tr.appendChild(td1);
tr.appendChild(td2);
tbody.appendChild(tr);
}
table.appendChild(tbody);
div.appendChild(table);
const md = htmlToMarkdown(div.innerHTML);
const ls = lines(md);
// Valid GFM: a header/data separator row is present.
// Valid GFM: a header/data separator row is present (an empty header is
// synthesized by the GFM turndown plugin for a header-less table — fine).
expect(ls.some(isSeparatorRow)).toBe(true);
// NOT the old broken "one value per line" shape: every line is pipe-delimited.
// NOT the old broken "one value per line" shape: every line is pipe-delimited
// and no line is a bare cell value on its own.
expect(ls.every((l) => l.includes("|"))).toBe(true);
expect(md).not.toMatch(/^\s*(a|b|c|d)\s*$/m);
// The cell values land in real pipe-delimited data rows.
@@ -249,21 +248,39 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
});
it("serializes a whole table with a header row as a proper GFM table (headline regression)", () => {
// Mirror the serializer's non-wrap branch: the full `table` node is the
// slice content and convertProseMirrorToMarkdown runs on it.
const md = convertProseMirrorToMarkdown({
type: "doc",
content: [
{
type: "table",
content: [
row([headerCell("Name"), headerCell("Age")]),
row([cell("Alice"), cell("30")]),
row([cell("Bob"), cell("25")]),
],
},
],
});
// Mirror the serializer's non-wrap branch: the full <table> node is appended
// directly (div.appendChild(fragment)) and htmlToMarkdown(div.innerHTML) runs.
const div = document.createElement("div");
const table = document.createElement("table");
const thead = document.createElement("thead");
const headerRow = document.createElement("tr");
for (const h of ["Name", "Age"]) {
const th = document.createElement("th");
th.textContent = h;
headerRow.appendChild(th);
}
thead.appendChild(headerRow);
table.appendChild(thead);
const tbody = document.createElement("tbody");
for (const [name, age] of [
["Alice", "30"],
["Bob", "25"],
]) {
const tr = document.createElement("tr");
const td1 = document.createElement("td");
td1.textContent = name;
const td2 = document.createElement("td");
td2.textContent = age;
tr.appendChild(td1);
tr.appendChild(td2);
tbody.appendChild(tr);
}
table.appendChild(tbody);
div.appendChild(table);
const md = htmlToMarkdown(div.innerHTML);
const ls = lines(md);
// Proper GFM structure: separator row + all rows pipe-delimited.
@@ -279,146 +296,3 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
expect(md).not.toMatch(/^\s*(Name|Age|Alice|Bob|30|25)\s*$/m);
});
});
// #347 acceptance: pasting CANONICAL markdown yields the SAME nodes the server
// import produces for the same text. The paste path calls markdownToProseMirror
// (the package browser entry) — the identical converter the server import uses —
// so asserting the converter (via the browser entry, on the native DOMParser)
// recognizes each canon form pins the paste-parity guarantee. These forms were
// NOT recognized by the old editor-ext marked layer the paste used before.
describe("canonical markdown paste recognition (browser entry parity)", () => {
// Collect every node type present in a doc (recursively).
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
if (!n || typeof n !== "object") return set;
if (n.type) set.add(n.type);
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
return set;
};
const findNode = (n: any, type: string): any => {
if (!n || typeof n !== "object") return undefined;
if (n.type === type) return n;
if (Array.isArray(n.content)) {
for (const c of n.content) {
const hit = findNode(c, type);
if (hit) return hit;
}
}
return undefined;
};
const allText = (n: any): string => {
if (!n || typeof n !== "object") return "";
if (typeof n.text === "string") return n.text;
if (Array.isArray(n.content)) return n.content.map(allText).join("");
return "";
};
it("^[…] inline footnote -> footnoteReference + footnotesList", () => {
const doc = markdownToProseMirrorSync("Body^[a note here].");
const types = collectTypes(doc);
expect(types.has("footnoteReference")).toBe(true);
expect(types.has("footnotesList")).toBe(true);
expect(types.has("footnoteDefinition")).toBe(true);
});
it('<!--img {…}--> attached image comment -> image with align', () => {
const doc = markdownToProseMirrorSync(
'![alt](/files/x.png) <!--img {"align":"left"}-->',
);
const img = findNode(doc, "image");
expect(img).toBeTruthy();
expect(img.attrs?.align).toBe("left");
expect(img.attrs?.src).toBe("/files/x.png");
});
it("> [!type] Obsidian callout -> callout node with type", () => {
const doc = markdownToProseMirrorSync("> [!warning]\n> be careful");
const callout = findNode(doc, "callout");
expect(callout).toBeTruthy();
expect(callout.attrs?.type).toBe("warning");
expect(allText(callout)).toContain("be careful");
});
it("$…$ inline math -> mathInline node", () => {
const doc = markdownToProseMirrorSync("Euler: $e^{i\\pi}+1=0$ done");
const math = findNode(doc, "mathInline");
expect(math).toBeTruthy();
expect(math.attrs?.text).toContain("e^{i\\pi}");
});
it("==…== highlight -> highlight mark", () => {
const doc = markdownToProseMirrorSync("A ==marked== word");
const marked = findNode(doc, "text");
// The highlighted run carries a `highlight` mark somewhere in the doc.
const hasHighlight = (n: any): boolean => {
if (!n || typeof n !== "object") return false;
if (
n.type === "text" &&
(n.marks || []).some((m: any) => m.type === "highlight")
)
return true;
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
};
expect(marked).toBeTruthy();
expect(hasHighlight(doc)).toBe(true);
});
it("<!--subpages--> standalone comment -> subpages node", () => {
const doc = markdownToProseMirrorSync("intro\n\n<!--subpages-->\n\nafter");
expect(collectTypes(doc).has("subpages")).toBe(true);
});
});
// #347 negatives: plain text carrying markdown-LIKE punctuation must NOT be
// silently converted/mangled (currency, bare `==`, a `[^1]` reference form).
describe("plain-text paste negatives (no phantom conversion)", () => {
const findNode = (n: any, type: string): any => {
if (!n || typeof n !== "object") return undefined;
if (n.type === type) return n;
if (Array.isArray(n.content)) {
for (const c of n.content) {
const hit = findNode(c, type);
if (hit) return hit;
}
}
return undefined;
};
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
if (!n || typeof n !== "object") return set;
if (n.type) set.add(n.type);
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
return set;
};
const allText = (n: any): string => {
if (!n || typeof n !== "object") return "";
if (typeof n.text === "string") return n.text;
if (Array.isArray(n.content)) return n.content.map(allText).join("");
return "";
};
it("currency `$5 and $10` is NOT turned into math", () => {
const doc = markdownToProseMirrorSync("It costs $5 and $10 total");
expect(findNode(doc, "mathInline")).toBeFalsy();
expect(allText(doc)).toContain("$5 and $10");
});
it("a lone `==` is NOT turned into a highlight", () => {
const doc = markdownToProseMirrorSync("compare a == b in code");
const hasHighlight = (n: any): boolean => {
if (!n || typeof n !== "object") return false;
if (
n.type === "text" &&
(n.marks || []).some((m: any) => m.type === "highlight")
)
return true;
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
};
expect(hasHighlight(doc)).toBe(false);
expect(allText(doc)).toContain("== b");
});
it("a `[^1]` reference form (no `^[`) is NOT turned into a footnote", () => {
const doc = markdownToProseMirrorSync("see note [^1] for details");
expect(collectTypes(doc).has("footnoteReference")).toBe(false);
expect(allText(doc)).toContain("[^1]");
});
});
@@ -1,23 +1,15 @@
// adapted from: https://github.com/aguingand/tiptap-markdown/blob/main/src/extensions/tiptap/clipboard.js - MIT
import { Extension } from "@tiptap/core";
import { Plugin, PluginKey, TextSelection } from "@tiptap/pm/state";
import { DOMParser, DOMSerializer, Fragment, Slice, Node as PMNode } from "@tiptap/pm/model";
import { DOMParser, DOMSerializer, Fragment, Slice } from "@tiptap/pm/model";
import { find } from "linkifyjs";
import {
markdownToHtml,
htmlToMarkdown,
canonicalizeFootnotes,
FOOTNOTES_LIST_NAME,
FOOTNOTE_REFERENCE_NAME,
} from "@docmost/editor-ext";
// Markdown <-> ProseMirror conversion now lives ONLY in the canonical
// `@docmost/prosemirror-markdown` package (issue #347). The BROWSER entry uses
// the native `DOMParser` for its HTML->DOM stage (jsdom stays out of the client
// bundle) while producing the SAME nodes the server import does — so a paste of
// canonical markdown (`^[…]`, `<!--img …-->`, `> [!type]`, `$…$`, `==…==`,
// standalone comments) is recognized identically to import.
import {
markdownToProseMirror,
convertProseMirrorToMarkdown,
} from "@docmost/prosemirror-markdown/browser";
import type { Schema } from "@tiptap/pm/model";
export const MarkdownClipboard = Extension.create({
@@ -47,24 +39,25 @@ export const MarkdownClipboard = Extension.create({
classifyClipboardSelection(topLevelNodes);
if (!asMarkdown) return null;
// Convert the copied selection to Markdown through the canonical
// package (issue #347), the SAME serializer the server export uses,
// so a copied table/list matches the on-disk markdown form. The
// converter takes a ProseMirror `doc` JSON, so wrap the slice's
// top-level content in a synthetic doc.
const content = slice.content.toJSON() as any[];
const div = document.createElement("div");
const serializer = DOMSerializer.fromSchema(this.editor.schema);
const fragment = serializer.serializeFragment(slice.content);
if (wrapBareRows) {
// A partial table cell-selection serializes to bare `tableRow`
// nodes (prosemirror-tables yields the whole `table` node only for
// a full-table selection). The converter's table case expects a
// `table` wrapper, so wrap the bare rows in one — mirroring the old
// <table><tbody> wrap that the HTML->markdown step needed.
return convertProseMirrorToMarkdown({
type: "doc",
content: [{ type: "table", content }],
});
// A partial table cell-selection serializes to bare <tr> nodes
// (prosemirror-tables returns the whole `table` node only when the
// entire table is selected). Bare <tr> would be foster-parented
// away by the HTML parser inside htmlToMarkdown, so wrap them in
// <table><tbody> first for the GFM turndown rule to detect them.
const table = document.createElement("table");
const tbody = document.createElement("tbody");
tbody.appendChild(fragment);
table.appendChild(tbody);
div.appendChild(table);
} else {
div.appendChild(fragment);
}
return convertProseMirrorToMarkdown({ type: "doc", content });
return htmlToMarkdown(div.innerHTML);
},
handlePaste: (view, event, slice) => {
if (!event.clipboardData) {
@@ -102,115 +95,37 @@ export const MarkdownClipboard = Extension.create({
}
}
const schema = this.editor.schema;
// Capture the target range NOW. markdownToProseMirror RETURNS A
// PROMISE (kept async only for the Node consumers' contract; the
// conversion pipeline itself is synchronous), so the actual replace
// happens on the next microtask. No user input can interleave a
// microtask, so the state is unchanged when we dispatch — but we
// still re-read the live state before replacing and, if the doc did
// change under us, fall back to the live selection rather than the
// captured (now-stale) range.
const from = view.state.selection.from;
const to = view.state.selection.to;
const startDoc = view.state.doc;
const md = text.replace(/\n+$/, "");
const { tr } = view.state;
const { from, to } = view.state.selection;
void markdownToProseMirror(md)
.then((doc) => {
if (view.isDestroyed) return;
// Canonical PM-JSON -> HTML via the LIVE editor schema, then
// reuse the UNCHANGED downstream seam (normalizeTableColumnWidths
// + parseSlice + canonicalizePastedFootnotes). The JSON->HTML->
// JSON hop is lossless (same schema both directions); it lets the
// existing paste-insertion logic stay byte-identical — only the
// SOURCE of the markdown conversion changed (issue #347 guardrail:
// no converter logic in the client, only a call into the package).
const node = PMNode.fromJSON(schema, doc);
const div = document.createElement("div");
DOMSerializer.fromSchema(schema).serializeFragment(
node.content,
{ document },
div,
);
const parsed = markdownToHtml(text.replace(/\n+$/, ""));
const body = elementFromString(parsed);
normalizeTableColumnWidths(body);
const body = elementFromString(div.innerHTML);
normalizeTableColumnWidths(body);
const parsedSlice = DOMParser.fromSchema(
this.editor.schema,
).parseSlice(body, {
preserveWhitespace: true,
});
const parsedSlice = DOMParser.fromSchema(schema).parseSlice(
body,
{ preserveWhitespace: true },
);
// A markdown paste builds its ProseMirror fragment directly (DOM ->
// parseSlice), bypassing the editor's footnoteSyncPlugin, which never
// reorders an existing list. So a pasted markdown block whose footnote
// definitions are out of order (or contains orphan defs) would be
// stored out of order. Canonicalize the self-contained pasted block so
// its footnotes come out reference-ordered, deduped and orphan-free
// (issue #228). See canonicalizePastedFootnotes for why this is scoped
// to whole-block pastes that carry their own footnotesList.
const contentNodes = canonicalizePastedFootnotes(
parsedSlice,
this.editor.schema,
);
// A markdown paste builds its ProseMirror fragment directly (DOM
// -> parseSlice), bypassing the editor's footnoteSyncPlugin, which
// never reorders an existing list. So a pasted markdown block whose
// footnote definitions are out of order (or contains orphan defs)
// would be stored out of order. Canonicalize the self-contained
// pasted block so its footnotes come out reference-ordered, deduped
// and orphan-free (issue #228). See canonicalizePastedFootnotes for
// why this is scoped to whole-block pastes that carry their own
// footnotesList.
const contentNodes = canonicalizePastedFootnotes(
parsedSlice,
schema,
);
// Target the captured range (normally still valid — same
// microtask). If the doc changed under us since capture, the
// captured absolute from/to are stale, so fall back to the live
// selection rather than StepMap-mapping the old range.
const tr = view.state.tr;
let mappedFrom = from;
let mappedTo = to;
if (view.state.doc !== startDoc) {
// Defensive: if the doc changed under us, fall back to the
// current selection rather than a stale absolute range.
mappedFrom = view.state.selection.from;
mappedTo = view.state.selection.to;
}
tr.replaceRange(mappedFrom, mappedTo, contentNodes);
const insertEnd = tr.mapping.map(mappedFrom, 1);
tr.setSelection(
TextSelection.near(
tr.doc.resolve(Math.max(mappedFrom, insertEnd - 2)),
-1,
),
);
tr.setMeta("paste", true);
view.dispatch(tr);
})
.catch((err) => {
// Fail-open: a conversion error must not swallow the paste
// silently in a way that loses the text. We already claimed the
// event (returned true), so re-insert the raw text as a plain
// paragraph so the user never loses their clipboard content.
// Log it: this catch covers BOTH the converter and the success
// `.then` body (e.g. PMNode.fromJSON throwing on a schema drift
// between the canonical package and the live editor schema), so a
// silent degrade to raw text would otherwise be an invisible,
// non-reproducible regression ("my table pasted as text").
console.error(
"markdown paste conversion failed, inserting raw text",
err,
);
if (view.isDestroyed) return;
const tr = view.state.tr;
// Same guard the success path uses: if the doc changed under us
// since the range was captured (normally never — same microtask),
// the captured absolute from/to are stale and would throw a
// RangeError here (an unhandled rejection on a hot paste path).
// Fall back to the live selection instead of a stale range.
if (view.state.doc !== startDoc) {
const sel = view.state.selection;
tr.insertText(md, sel.from, sel.to);
} else {
tr.insertText(md, from, to);
}
tr.setMeta("paste", true);
view.dispatch(tr);
});
// Claim the paste: we insert asynchronously above.
tr.replaceRange(from, to, contentNodes);
const insertEnd = tr.mapping.map(from, 1);
tr.setSelection(TextSelection.near(tr.doc.resolve(Math.max(from, insertEnd - 2)), -1));
tr.setMeta('paste', true)
view.dispatch(tr);
return true;
},
// Strip trailing whitespace-only paragraphs from pasted content.
@@ -33,11 +33,10 @@ vi.mock("@/lib/local-emitter.ts", () => ({
default: { emit: (...args: unknown[]) => localEmitMock(...args) },
}));
// convertProseMirrorToMarkdown echoes a marker carried on the fake editor's
// getJSON() doc, so each test controls the markdown purely via the fake page
// editor (issue #347: the hook now serializes editor JSON through the package).
vi.mock("@docmost/prosemirror-markdown/browser", () => ({
convertProseMirrorToMarkdown: (doc: { __md?: string }) => doc?.__md ?? "",
// htmlToMarkdown just echoes the editor HTML so each test controls the markdown
// purely via the fake page editor's getHTML().
vi.mock("@docmost/editor-ext", () => ({
htmlToMarkdown: (html: string) => html,
}));
const notificationsShowMock = vi.fn();
@@ -54,12 +53,10 @@ import { useGeneratePageTitle } from "./use-generate-page-title.ts";
// --- Test helpers -------------------------------------------------------------
function makePageEditor(pageId: string, md = "content"): Editor {
function makePageEditor(pageId: string, html = "<p>content</p>"): Editor {
return {
isDestroyed: false,
// The mocked convertProseMirrorToMarkdown reads `__md` back off this doc,
// so `md` is exactly the markdown the hook will send to the title service.
getJSON: () => ({ type: "doc", __md: md }),
getHTML: () => html,
storage: { pageId },
} as unknown as Editor;
}
@@ -3,7 +3,7 @@ import { useMutation } from "@tanstack/react-query";
import { useAtomValue } from "jotai";
import { notifications } from "@mantine/notifications";
import { useTranslation } from "react-i18next";
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser";
import { htmlToMarkdown } from "@docmost/editor-ext";
import {
pageEditorAtom,
titleEditorAtom,
@@ -49,9 +49,7 @@ export function useGeneratePageTitle(pageId: string) {
mutationFn: async () => {
if (!pageEditor || pageEditor.isDestroyed) return;
// Serialize the live editor content to markdown through the canonical
// converter (issue #347), matching the on-disk/export markdown form.
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON()).trim();
const markdown = htmlToMarkdown(pageEditor.getHTML()).trim();
if (!markdown) {
notifications.show({ message: t("The note is empty"), color: "yellow" });
return;
@@ -37,7 +37,7 @@ import { useTreeMutation } from "@/features/page/tree/hooks/use-tree-mutation.ts
import { PageWidthToggle } from "@/features/user/components/page-width-pref.tsx";
import { Trans, useTranslation } from "react-i18next";
import ExportModal from "@/components/common/export-modal";
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser";
import { htmlToMarkdown } from "@docmost/editor-ext";
import {
pageEditorAtom,
yjsConnectionStatusAtom,
@@ -199,9 +199,8 @@ function PageActionMenu({ readOnly }: PageActionMenuProps) {
const handleCopyAsMarkdown = () => {
if (!pageEditor) return;
// Copy the page as canonical markdown through the shared converter (issue
// #347), so "Copy as markdown" matches the server export byte-for-byte.
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON());
const html = pageEditor.getHTML();
const markdown = htmlToMarkdown(html);
const title = page?.title ? `# ${page.title}\n\n` : "";
clipboard.copy(`${title}${markdown}`);
notifications.show({ message: t("Copied") });
@@ -168,6 +168,10 @@ export default function ShareAiWidget({
// Anonymous reader: suppress the tool-argument summary line so the
// agent's raw query/argument text isn't shown on the public share.
showInput={false}
// Anonymous reader: never paint a tool's raw errorText (it can carry
// internal detail). This is the render gate; the bytes are also
// sanitized server-side in PublicShareChatToolsService.forShare (#394).
showErrors={false}
// Anonymous reader: neutralize internal/relative links in the
// assistant's markdown so internal UUIDs/auth-gated routes don't
// leak as clickable links (external http(s) links are kept).
@@ -1,5 +1,4 @@
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { encodeHtmlEmbedSource } from '@docmost/editor-ext';
import { markdownToHtml, encodeHtmlEmbedSource } from '@docmost/editor-ext';
import { htmlToJson } from '../../../collaboration/collaboration.util';
import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
@@ -11,12 +10,13 @@ import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
*
* The block renders inside a sandboxed iframe, so this is not an XSS surface;
* this exercises the REAL server import conversion path that ImportService uses
* (`markdownToProseMirror`, the canonical converter issue #345/#347) and
* asserts that such a node is DETECTED and STRIPPABLE so the share read path's
* (`markdownToHtml` then `htmlToJson`; `processHTML` adds only a cheerio
* link/iframe normalize pass which does not touch htmlEmbed divs) and asserts
* that such a node is DETECTED and STRIPPABLE so the share read path's
* master-toggle strip can remove it when the workspace toggle is OFF.
*/
describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTML', () => {
it('round-trips through markdownToProseMirror and is DETECTED (base64 data-source)', async () => {
it('round-trips through markdownToHtml -> htmlToJson and is DETECTED (base64 data-source)', async () => {
const source = '<script>steal()</script>';
const encoded = encodeHtmlEmbedSource(source);
const md = [
@@ -27,9 +27,12 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
'World',
].join('\n');
// The canonical importer parses the raw block-level div into a real
// htmlEmbed node carrying the decoded source.
const json = await markdownToProseMirror(md);
const html = await markdownToHtml(md);
// marked preserves the raw block-level div verbatim.
expect(html).toContain('data-type="htmlEmbed"');
const json = htmlToJson(html);
// The div parses into a real htmlEmbed node carrying the decoded source.
expect(hasHtmlEmbedNode(json)).toBe(true);
// Because it is detected, the share master-toggle strip can remove it.
@@ -56,7 +59,8 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
// therefore stripping) does not depend on the source being well-formed, so
// the bypass cannot be hidden by sending a malformed data-source.
const md = `<div data-type="htmlEmbed" data-source="&lt;script&gt;x&lt;/script&gt;"></div>`;
const json = await markdownToProseMirror(md);
const html = await markdownToHtml(md);
const json = htmlToJson(html);
expect(hasHtmlEmbedNode(json)).toBe(true);
expect(hasHtmlEmbedNode(stripHtmlEmbedNodes(json))).toBe(false);
});
@@ -43,6 +43,9 @@ function makeRepo(overrides: Record<string, jest.Mock> = {}) {
workspaceId: v.workspaceId,
})),
update: jest.fn(async () => ({ id: 'run-1' })),
// #487: terminal finalize now goes through the CONDITIONAL write. Default
// returns a truthy row (the run WAS active -> this call wrote it).
finalizeIfActive: jest.fn(async () => ({ id: 'run-1', status: 'succeeded' })),
markStopRequested: jest.fn(async () => ({ id: 'run-1' })),
findActiveByChat: jest.fn(async () => undefined),
findLatestByChat: jest.fn(async () => undefined),
@@ -336,14 +339,12 @@ describe('AiChatRunService run lifecycle', () => {
await svc.finalizeRun('run-1', 'ws-1', 'error', 'provider blew up');
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenCalledWith(
// #487: the terminal write is CONDITIONAL (finalizeIfActive); finishedAt is
// stamped inside the repo method, so the service passes just status + error.
expect(repo.finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({
status: 'failed',
error: 'provider blew up',
finishedAt: expect.any(Date),
}),
expect.objectContaining({ status: 'failed', error: 'provider blew up' }),
);
});
@@ -366,8 +367,8 @@ describe('AiChatRunService run lifecycle', () => {
// A second settle (e.g. a streamText callback firing after the catch) no-ops.
await svc.finalizeRun('run-1', 'ws-1', 'completed', undefined);
expect(repo.update).toHaveBeenCalledTimes(1);
expect(repo.update).toHaveBeenCalledWith(
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(1);
expect(repo.finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'failed', error: 'first' }),
@@ -389,8 +390,8 @@ describe('AiChatRunService run lifecycle', () => {
const updateGate = new Promise((res) => {
resolveUpdate = res;
});
const update = jest.fn(() => updateGate);
const repo = makeRepo({ update });
const finalizeIfActive = jest.fn(() => updateGate);
const repo = makeRepo({ finalizeIfActive });
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
@@ -399,23 +400,23 @@ describe('AiChatRunService run lifecycle', () => {
});
// Fire both before the (pending) update resolves. The first synchronously
// claims the entry (active.delete) and awaits update; the second, started in
// the same macrotask, finds the entry already gone and returns at the claim
// WITHOUT ever calling update.
// claims the entry (active.delete) and awaits the write; the second, started
// in the same macrotask, finds the entry already gone and returns at the claim
// WITHOUT ever writing.
const p1 = svc.finalizeRun('run-1', 'ws-1', 'completed');
const p2 = svc.finalizeRun('run-1', 'ws-1', 'error', 'safety-net');
// The decisive assertion: exactly one caller reached the terminal UPDATE.
expect(update).toHaveBeenCalledTimes(1);
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
// Let the single in-flight update land; both calls resolve cleanly.
resolveUpdate({ id: 'run-1' });
resolveUpdate({ id: 'run-1', status: 'succeeded' });
await Promise.all([p1, p2]);
expect(update).toHaveBeenCalledTimes(1);
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
// The winner is the FIRST caller ('completed' -> 'succeeded'); the late
// 'error' settle never wrote, so it could not clobber the real status.
expect(update).toHaveBeenCalledWith(
expect(finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
@@ -431,10 +432,10 @@ describe('AiChatRunService run lifecycle', () => {
// 409s until a restart. The fix updates FIRST and retries.
let calls = 0;
const repo = makeRepo({
update: jest.fn(async () => {
finalizeIfActive: jest.fn(async () => {
calls += 1;
if (calls === 1) throw new Error('deadlock detected');
return { id: 'run-1' };
return { id: 'run-1', status: 'succeeded' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
@@ -447,26 +448,29 @@ describe('AiChatRunService run lifecycle', () => {
await svc.finalizeRun('run-1', 'ws-1', 'completed');
// The retry landed the terminal write: the entry is dropped (slot freed) and
// the row carries the real terminal status — NOT stranded at 'running'.
// The retry landed the terminal write: the entry is dropped (slot freed), no
// zombie left, and the row carries the real terminal status.
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenCalledTimes(2);
expect(repo.update).toHaveBeenLastCalledWith(
expect(svc.hasZombie('run-1')).toBe(false);
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(2);
expect(repo.finalizeIfActive).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
});
it('F6: if the terminal write keeps failing, the entry is RETAINED and a LATER settle completes it (chat not permanently 409d)', async () => {
it('#487 give-up: if the terminal write keeps failing, finalizeRun leaves a ZOMBIE (does NOT restore the entry) and settleZombie re-drives it', async () => {
// Worst case: the DB is down for the whole first finalize (all attempts fail).
// The run must NOT be silently lost — the entry stays so a subsequent settle
// (a streamText callback, requestStop -> onAbort, or a future sweep) can retry.
// #487 changes the give-up behaviour: the entry is NOT restored (a restored
// entry is indistinguishable from a live run). Instead a ZOMBIE record holds
// the intended terminal status, and a re-drive (settleZombie — called by the
// reconcile / supersede / opportunistic paths) applies it later.
let healthy = false;
const repo = makeRepo({
update: jest.fn(async () => {
finalizeIfActive: jest.fn(async () => {
if (!healthy) throw new Error('pool exhausted');
return { id: 'run-1' };
return { id: 'run-1', status: 'succeeded' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
@@ -480,35 +484,83 @@ describe('AiChatRunService run lifecycle', () => {
userId: 'user-1',
});
// First settle: every bounded attempt fails -> entry retained, NOT settled.
// First settle: every bounded attempt fails -> ZOMBIE, entry NOT restored.
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(true);
// F12: the give-up emits ONE explicit, greppable ERROR (run + chat context)
// so an operator can tell "gave up, run held in memory" from a per-attempt
// blip — distinct from the per-attempt warns.
expect(svc.isLocallyActive('run-1')).toBe(false); // NOT a live entry
expect(svc.hasZombie('run-1')).toBe(true);
expect(svc.zombieRunIds()).toContain('run-1');
// The give-up emits ONE explicit, greppable ERROR mentioning the zombie.
const gaveUp = errorSpy.mock.calls.some(
(c) =>
/NON-TERMINAL/.test(String(c[0])) &&
/ZOMBIE/.test(String(c[0])) &&
/run-1/.test(String(c[0])) &&
/chat-1/.test(String(c[0])),
);
expect(gaveUp).toBe(true);
// The settle notifier resolved as terminalWriteFailed (a subscriber learns the
// slot still needs the intended status applied).
const outcome = await svc.peekSettled('run-1');
expect(outcome).toEqual({
status: 'succeeded',
error: null,
terminalWriteFailed: true,
});
// The DB recovers; a later settle now succeeds and frees the slot.
// The DB recovers; a re-drive settles the zombie via the conditional UPDATE.
healthy = true;
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenLastCalledWith(
const redriven = await svc.settleZombie('run-1');
expect(redriven).toBe(true);
expect(svc.hasZombie('run-1')).toBe(false);
expect(repo.finalizeIfActive).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
// And it is now idempotent: a further settle no-ops (terminal row already
// written), so a double-settle can never clobber the real status.
const callsBefore = repo.update.mock.calls.length;
// A later finalizeRun is idempotent (row already terminal): it no-ops at the
// once-gate, never re-writing.
const callsBefore = repo.finalizeIfActive.mock.calls.length;
await svc.finalizeRun('run-1', 'ws-1', 'error', 'late');
expect(repo.update).toHaveBeenCalledTimes(callsBefore);
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(callsBefore);
});
it('#487 double-settle collapses to a benign no-op (conditional write; notifier resolves once)', async () => {
// A second concurrent settle is stopped at the synchronous active.delete
// claim, so the terminal write runs exactly once and the notifier resolves
// exactly once with the FIRST settler's outcome.
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'u1' });
await svc.finalizeRun('run-1', 'ws-1', 'aborted');
await svc.finalizeRun('run-1', 'ws-1', 'error', 'late'); // no-op
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(1);
const outcome = await svc.peekSettled('run-1');
// peekSettled after resolve+delete falls through (notifier dropped, no zombie)
// -> undefined; the FIRST settler already resolved any earlier subscriber.
expect(outcome).toBeUndefined();
});
it('#487 late settledPromise subscriber gets the resolved outcome', async () => {
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'u1' });
// Subscribe BEFORE settle: hold the promise reference (as supersede does).
const early = svc.peekSettled('run-1');
expect(early).toBeDefined();
await svc.finalizeRun('run-1', 'ws-1', 'completed');
// The reference grabbed before settle resolves with the written outcome, even
// though the notifier was dropped from the map on resolve (bounded).
await expect(early).resolves.toEqual({
status: 'succeeded',
error: null,
terminalWriteFailed: false,
});
});
it('recordStep / linkAssistantMessage are best-effort: a repo failure is swallowed', async () => {
@@ -525,3 +577,197 @@ describe('AiChatRunService run lifecycle', () => {
).resolves.toBeUndefined();
});
});
describe('#487 AiChatRunService.supersede (CAS)', () => {
const chat = 'chat-1';
const ws = 'ws-1';
it('degrade: no active run on the chat -> caller sends a normal turn', async () => {
const repo = makeRepo({
findById: jest.fn(async () => undefined),
findActiveByChat: jest.fn(async () => undefined),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({ kind: 'degrade' });
});
it('invalid: the target run belongs to a DIFFERENT chat -> 400', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-x',
chatId: 'other-chat',
workspaceId: ws,
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({ kind: 'invalid' });
});
it('mismatch: a DIFFERENT run is active than the one targeted -> current runId', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-x', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => ({
id: 'run-live',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({
kind: 'mismatch',
activeRunId: 'run-live',
});
});
it('ready: the target IS active -> stop it, await its (fast) settle, free the slot', async () => {
// Simulate a live long TOOL (NOT a slow UPDATE): the run stays active until an
// explicit Stop unwinds it; commit-1's race makes that settle land quickly.
// The abort listener stands in for streamText's onAbort -> finalizeRun.
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'aborted',
error: null,
})),
findActiveByChat: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
handle.signal.addEventListener('abort', () => {
void svc.finalizeRun('run-1', ws, 'aborted');
});
// supersede: getRun -> getActiveByChat(==target) -> requestStop -> the abort
// listener settles the run -> awaitSettled resolves -> ready.
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
expect(handle.signal.aborted).toBe(true); // Stop reached the run
});
it('timeout: the target never settles within W -> 409 SUPERSEDE_TIMEOUT (nothing persisted)', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-1', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
// Do NOT settle the run: a tiny W elapses -> timeout.
const result = await svc.supersede(chat, 'run-1', ws, 30);
expect(result).toEqual({ kind: 'timeout' });
});
it('ready then a DUPLICATE supersede POST degrades (the run is already gone)', async () => {
let active: unknown = {
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
};
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'aborted',
error: null,
})),
findActiveByChat: jest.fn(async () => active),
finalizeIfActive: jest.fn(async () => {
active = undefined; // settling frees the active slot
return { id: 'run-1', status: 'aborted' };
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
handle.signal.addEventListener('abort', () => {
void svc.finalizeRun('run-1', ws, 'aborted');
});
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
// The duplicate POST for the same target now finds no active run -> degrade.
expect(await svc.supersede(chat, 'run-1', ws)).toEqual({ kind: 'degrade' });
});
it('reconcileStaleRuns: aborts a stale run with NO entry/zombie; NEVER touches a live entry', async () => {
const finalizeIfActive = jest.fn(async () => ({ id: 'x', status: 'aborted' }));
const repo = makeRepo({
insert: jest.fn(async (v: any) => ({
id: 'live-1',
status: 'running',
chatId: v.chatId,
workspaceId: v.workspaceId,
})),
finalizeIfActive,
findStaleActive: jest.fn(async () => [
{ id: 'orphan-1', workspaceId: ws, chatId: 'c-orphan' },
{ id: 'live-1', workspaceId: ws, chatId: 'c-live' },
]),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
// A LIVE run this replica owns (in the `active` map).
await svc.beginRun({ chatId: 'c-live', workspaceId: ws, userId: 'u1' });
expect(svc.isLocallyActive('live-1')).toBe(true);
const aborted = await svc.reconcileStaleRuns(15 * 60 * 1000);
expect(aborted).toBe(1);
// The orphan (no entry) was aborted; the live entry was NEVER passed to the DB.
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
expect(finalizeIfActive).toHaveBeenCalledWith(
'orphan-1',
ws,
expect.objectContaining({ status: 'aborted' }),
);
expect(svc.isLocallyActive('live-1')).toBe(true);
});
it('gave-up zombie: supersede applies the intended status (settleZombie) then is ready', async () => {
let healthy = false;
let active: unknown = {
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
};
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-1', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => active),
finalizeIfActive: jest.fn(async () => {
if (!healthy) throw new Error('db down');
active = undefined;
return { id: 'run-1', status: 'aborted' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
jest.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
// The run's terminal write gives up -> zombie (row still 'running').
await svc.finalizeRun('run-1', ws, 'aborted');
expect(svc.hasZombie('run-1')).toBe(true);
// The DB recovers; supersede awaits the (already-resolved, terminalWriteFailed)
// settle, then settleZombie applies the intended status -> ready.
healthy = true;
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
expect(svc.hasZombie('run-1')).toBe(false);
});
});
@@ -34,6 +34,88 @@ export class RunAlreadyActiveError extends Error {
export type TurnTerminalStatus = 'completed' | 'error' | 'aborted';
export type RunTerminalStatus = 'succeeded' | 'failed' | 'aborted';
/** The terminal run statuses — the row is done once it reads one of these. */
export const RUN_TERMINAL_STATUSES: readonly RunTerminalStatus[] = [
'succeeded',
'failed',
'aborted',
];
/** Whether a persisted run status is terminal (settled). */
export function isRunTerminal(status: string | null | undefined): boolean {
return (
status === 'succeeded' || status === 'failed' || status === 'aborted'
);
}
/**
* #487: the outcome a run's {@link AiChatRunService.finalizeRun} settled with.
* `terminalWriteFailed` = the terminal write GAVE UP after the bounded retry, so
* the row is still non-terminal ('running') and a ZOMBIE record holds the
* `intended` status for a later re-drive (reconcile / supersede / boot sweep). A
* subscriber (supersede, #487 commit 3) uses this to decide whether the slot is
* genuinely free or must first have the intended status applied.
*/
export interface RunSettleOutcome {
status: RunTerminalStatus;
error: string | null;
terminalWriteFailed: boolean;
}
/**
* #487: how long a supersede waits for the target run to settle after Stop before
* it degrades to `SUPERSEDE_TIMEOUT`. W=10s is generous under a HEALTHY DB: commit
* 1's race-on-abort makes an in-app tool abort->settle in ms/hundreds of ms, so a
* live run releases its slot well within the window. Under a DB brownout the
* timeout is normal (the write cannot land); W must NOT be raised to paper
* over a slow DB a SUPERSEDE_TIMEOUT is the honest signal (nothing persisted,
* the composer keeps the user's text). Env-tunable for ops, default 10s.
*/
export const SUPERSEDE_SETTLE_TIMEOUT_MS = (() => {
const raw = Number(process.env.AI_CHAT_SUPERSEDE_TIMEOUT_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 10_000;
})();
/**
* #487: the result of the supersede CAS ({@link AiChatRunService.supersede}).
* - `degrade` : no active run on the chat (it ended between click and POST)
* the caller sends a NORMAL turn (NOT a mismatch);
* - `invalid` : the target runId belongs to a DIFFERENT chat (malformed CAS 400);
* - `mismatch` : a DIFFERENT run is active than the one the client targeted
* 409 SUPERSEDE_TARGET_MISMATCH carrying the current `activeRunId`
* (the client does NOT auto-retry);
* - `timeout` : the target did not settle within W 409 SUPERSEDE_TIMEOUT,
* nothing persisted;
* - `ready` : the target was stopped AND settled (or its zombie's intended was
* applied) the slot is free; the caller may beginRun the new run.
*/
export type SupersedeResult =
| { kind: 'degrade' }
| { kind: 'invalid' }
| { kind: 'mismatch'; activeRunId: string }
| { kind: 'timeout' }
| { kind: 'ready' };
/** A one-shot settle notifier (#487): `resolve` is called EXACTLY ONCE. */
interface Deferred<T> {
promise: Promise<T>;
resolve: (value: T) => void;
}
/**
* #487: a run whose terminal write GAVE UP (every bounded attempt failed). The
* row is stranded non-terminal ('running'); this record is the ONLY thing that
* distinguishes it from a live run, and carries the `intended` terminal status so
* a re-drive can apply it via the conditional UPDATE. Process-local (phase-1
* single-process assumption): a restart drops it, and the boot sweep then writes
* 'aborted' over the intended a documented loss (see finalizeRun).
*/
interface ZombieRun {
workspaceId: string;
chatId: string;
intended: { status: RunTerminalStatus; error: string | null };
}
export function mapTurnStatusToRun(
status: TurnTerminalStatus,
): RunTerminalStatus {
@@ -101,6 +183,22 @@ export class AiChatRunService implements OnModuleInit {
// uptime — negligible in phase 1's single process.
private readonly settled = new Set<string>();
// #487 runId -> one-shot settle notifier. Kept in a SEPARATE map from `active`
// ON PURPOSE: it must OUTLIVE the `active.delete` claim inside finalizeRun (the
// claim frees the slot the instant finalize starts), so a subscriber can still
// await the outcome after the entry is gone. Created in beginRun, resolved
// EXACTLY ONCE in finalizeRun, then removed (bounded). Absence => this replica
// has no live notifier: a subscriber falls back to the zombie map, then to the
// row (see peekSettled). Process-local (phase-1 single-process assumption).
private readonly settledPromises = new Map<string, Deferred<RunSettleOutcome>>();
// #487 runId -> ZOMBIE record: a run whose terminal write gave up (row stranded
// non-terminal). BOUNDED — an entry is added only on give-up and removed on a
// successful re-drive (settleZombie) or when the row is found already terminal;
// a process restart clears it (and the boot sweep settles the stranded row).
// Process-local (phase-1 single-process assumption).
private readonly zombies = new Map<string, ZombieRun>();
// Bounded retry for the terminal write (F6): a single PK UPDATE can fail
// transiently under many fire-and-forget writes (pool exhaustion, deadlock, a
// brief connection blip). Riding out that blip in-place matters because the
@@ -224,6 +322,10 @@ export class AiChatRunService implements OnModuleInit {
chatId: args.chatId,
workspaceId: args.workspaceId,
});
// #487: arm the one-shot settle notifier BEFORE returning, so a subscriber
// that races in immediately after begin always finds a promise to await. It
// is resolved exactly once when the run settles (or gives up).
this.settledPromises.set(run.id, this.makeDeferred<RunSettleOutcome>());
return { runId: run.id, signal: controller.signal };
}
@@ -263,47 +365,43 @@ export class AiChatRunService implements OnModuleInit {
}
/**
* Finalize a run to its terminal status (succeeded / failed / aborted),
* stamping finishedAt + any error. Best-effort, but ROBUST against a transient
* terminal-write failure (F6) AND atomically safe against a concurrent settle.
* Finalize a run to its terminal status (succeeded / failed / aborted) via a
* CONDITIONAL UPDATE, stamping finishedAt + any error. Atomically safe against a
* concurrent settle AND robust against a transient terminal-write failure.
*
* ATOMIC ONCE-CLAIM (the gate must close in ONE synchronous tick): two
* finalizeRun calls for the SAME run can race the documented real path is
* AiChatService.stream's safety-net catch settling the turn to 'error' while a
* streamText terminal callback (onFinish/onAbort/onError) ALSO settles it. The
* `settled.has` check alone is NOT a gate: it is read BEFORE the awaited UPDATE,
* so two callers can both see `false` and both write the row (last-write-wins
* clobbers the real terminal status, and the bounded retry only widens that
* window). The claim therefore happens via `active.delete`, a SYNCHRONOUS
* check-and-clear with NO await between the gate and the entry removal: the
* second concurrent caller finds the entry already gone and returns in the same
* tick, before any UPDATE. The transition "nobody is finalizing" -> "I am
* finalizing" is thus a single atomic step.
* claim happens via `active.delete`, a SYNCHRONOUS check-and-clear with NO await
* between the gate and the entry removal: the second concurrent caller finds the
* entry already gone and returns in the same tick, before any UPDATE.
*
* ORDER MATTERS (F6): once we own the claim, the terminal UPDATE happens FIRST;
* only once it SUCCEEDS do we record the run as settled. If the UPDATE fails on
* every bounded attempt we RESTORE the in-memory entry, leave the run UNsettled,
* and emit an ERROR signal that the row is left non-terminal 'running' (which
* would 409 every future turn in the chat until recovery). An in-process retry
* by a LATER settle is only POSSIBLE, never guaranteed: it needs (a) the entry
* to have been restored at the give-up path AND (b) a fresh settler to arrive
* AFTER that restore. A concurrent settler that arrives DURING the retry window
* while the entry is deleted for backoff and not yet restored is consumed at
* the synchronous `active.delete` claim (it finds nothing to delete and returns
* a no-op), so it does NOT become an in-process retrier. The NO-streamText path
* (the turn threw before streamText was wired, so ONLY the safety-net ever
* settles) likewise has no second in-process settler at all. The UNCONDITIONAL
* backstop in every case is the boot sweep on the next restart (phase 1 has no
* periodic in-process sweep); the retained entry is bounded (cleared on restart)
* and harmless meanwhile.
* ALL TERMINAL WRITES ARE CONDITIONAL (#487): `finalizeIfActive` only flips a
* row still in pending|running (mirror of the assistant message's
* `onlyIfStreaming`). So even a settle that DID reach the UPDATE (e.g. a
* reconcile stamp racing an owner finalize) can never clobber a terminal status
* the loser matches nothing and is a benign no-op. `active.delete` is the
* fast, in-process gate; the conditional WHERE is the authoritative one.
*
* IDEMPOTENT on SUCCESS (#184 review): the terminal write happens AT MOST ONCE
* per run. After a successful write the once-gate keys off {@link settled} (the
* terminal row already written) so a settle arriving AFTER the entry was already
* dropped-and-settled returns early; a settle racing the in-flight write is
* stopped earlier still, by the `active.delete` claim. Either way a genuine
* double-settle collapses to a single write and a late settle can never clobber
* the real terminal status or double-write the row.
* ZOMBIE ON GIVE-UP (#487): if every bounded attempt THROWS (the DB is down for
* the whole finalize), we do NOT restore the entry. The row is stranded
* non-terminal ('running'); we record a ZOMBIE `{ terminalWriteFailed, intended
* }` (the ONLY thing distinguishing this dead run from a live one) and resolve
* the settle notifier with `terminalWriteFailed: true`. A restore would make the
* zombie indistinguishable from a live run to every reader; instead a re-drive
* (settleZombie, called by the periodic reconcile / supersede / opportunistic
* paths) applies the intended status later via the same conditional UPDATE.
*
* DOCUMENTED LOSS (#487, single-process phase 1): if the process RESTARTS before
* a zombie is re-driven, the in-memory zombie map is gone and the boot sweep
* (unconditional) writes 'aborted' over the ACTUAL intended status. This is
* unavoidable while the run lifecycle is single-process there is no durable
* record of `intended`; a cross-process durable intent is deferred to phase 2.
*
* IDEMPOTENT: the settle notifier resolves EXACTLY ONCE; a second settle is
* stopped at `settled.has` or the `active.delete` claim, so a double-settle
* collapses to a single write and can never double-resolve or clobber the row.
*/
async finalizeRun(
runId: string,
@@ -314,13 +412,17 @@ export class AiChatRunService implements OnModuleInit {
// ---- Atomic once-claim (synchronous; NO await before the gate closes) ----
// Already terminally written -> idempotent no-op.
if (this.settled.has(runId)) return;
// Capture the entry BEFORE the delete so a total-failure path can restore it.
// Capture the entry BEFORE the delete for the give-up log context.
const entry = this.active.get(runId);
// SYNCHRONOUS check-and-clear: the FIRST caller deletes (claims) the entry;
// any concurrent SECOND caller finds nothing to delete and returns HERE, in
// the same tick, before any await — so it can never reach the UPDATE.
if (!this.active.delete(runId)) return;
const status = mapTurnStatusToRun(turnStatus);
const err = error ?? null;
const chatId = entry?.chatId ?? 'unknown';
let lastError: unknown;
for (
let attempt = 1;
@@ -328,47 +430,294 @@ export class AiChatRunService implements OnModuleInit {
attempt++
) {
try {
await this.runRepo.update(runId, workspaceId, {
status: mapTurnStatusToRun(turnStatus),
finishedAt: new Date(),
error: error ?? null,
const row = await this.runRepo.finalizeIfActive(runId, workspaceId, {
status,
error: err,
});
// Terminal write landed: arm the once-gate. The entry is already gone
// (claimed above); we do NOT restore it. The slot is now free.
// No throw => the row is now terminal (we wrote it, or it was ALREADY
// terminal — another writer won the conditional UPDATE, a benign no-op).
this.settled.add(runId);
this.zombies.delete(runId);
// Resolve with the persisted outcome: our status when WE wrote it, else
// the row's real terminal status (re-read on the already-terminal path so
// a subscriber never sees a status we did not actually persist).
const outcome: RunSettleOutcome = row
? { status, error: err, terminalWriteFailed: false }
: await this.readTerminalOutcome(runId, workspaceId, status, err);
this.resolveSettled(runId, outcome);
return;
} catch (err) {
lastError = err;
} catch (err2) {
lastError = err2;
this.logger.warn(
`Failed to finalize run ${runId} (attempt ${attempt}/${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
}): ${err instanceof Error ? err.message : 'unknown error'}`,
}): ${err2 instanceof Error ? err2.message : 'unknown error'}`,
);
if (attempt < AiChatRunService.FINALIZE_MAX_ATTEMPTS) {
await this.delay(AiChatRunService.FINALIZE_RETRY_BASE_MS * attempt);
}
}
}
// Every attempt failed: this is a give-up, materially worse than a per-attempt
// blip — the row is left NON-TERMINAL ('running'), so emit ONE explicit,
// greppable ERROR so an operator can tell "survived a blip" from "gave up, run
// held in memory until recovery" (the last warn alone says only "attempt 3/3").
// Every attempt threw: GIVE UP. The row is stranded non-terminal ('running').
// Do NOT restore the entry (a restored entry is indistinguishable from a live
// run); leave a ZOMBIE record instead, and resolve the notifier as
// terminalWriteFailed so a subscriber knows the slot still needs the intended
// status applied. One explicit, greppable ERROR so an operator can tell a
// give-up from a per-attempt blip.
this.logger.error(
`Run ${runId} (chat ${entry?.chatId ?? 'unknown'}) left NON-TERMINAL ` +
`('running'): terminal write failed after ${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
} attempts; entry retained in memory, recovery deferred to next settle / ` +
`boot sweep`,
`Run ${runId} (chat ${chatId}) left NON-TERMINAL ('running'): terminal ` +
`write failed after ${AiChatRunService.FINALIZE_MAX_ATTEMPTS} attempts; ` +
`ZOMBIE recorded (intended '${status}'), recovery deferred to reconcile / ` +
`supersede / boot sweep`,
lastError,
);
// RESTORE the claimed entry (and leave the run UNsettled) so a LATER settle
// that arrives AFTER this restore MAY retry the terminal write — but that
// in-process retry is NOT guaranteed (a concurrent settler caught in the retry
// window above is consumed at the `active.delete` claim, and the no-streamText
// path has no second settler at all). The UNCONDITIONAL backstop in every case
// is the boot sweep on the next restart; the restored entry is bounded and
// cleared on restart.
if (entry) this.active.set(runId, entry);
this.zombies.set(runId, {
workspaceId,
chatId,
intended: { status, error: err },
});
this.resolveSettled(runId, { status, error: err, terminalWriteFailed: true });
}
/**
* #487: re-drive a zombie run's intended terminal write (the conditional
* UPDATE). Called by the periodic reconcile (commit 4), an opportunistic
* single-chat reconcile, and supersede (commit 3). On success the row is now
* terminal (written OR found already terminal) the zombie is cleared and the
* once-gate armed; on another failure the zombie is kept for a later retry.
* Returns true when the row is now terminal. Best-effort; never throws.
*/
async settleZombie(runId: string): Promise<boolean> {
const z = this.zombies.get(runId);
if (!z) return false;
try {
await this.runRepo.finalizeIfActive(runId, z.workspaceId, {
status: z.intended.status,
error: z.intended.error,
});
this.zombies.delete(runId);
this.settled.add(runId);
return true;
} catch (err) {
this.logger.warn(
`Re-drive of zombie run ${runId} (chat ${z.chatId}) failed; will retry ` +
`later: ${err instanceof Error ? err.message : 'unknown error'}`,
);
return false;
}
}
/**
* #487 reconcile clause (c): abort runs the DB still shows active (pending|
* running) but that this replica does NOT own NO live entry AND NO zombie
* and that have been UNTOUCHED past `staleMs` (from last-progress `updated_at`,
* NOT startedAt, so a legit long marathon is never a candidate). "No entry" is
* the PRIMARY gate: a live entry (an actively-executing run on this replica) is
* NEVER aborted, whatever its age. Returns the number aborted. Best-effort
* never throws (a periodic-job failure must not crash the process).
*/
async reconcileStaleRuns(staleMs: number): Promise<number> {
let candidates: Array<{ id: string; workspaceId: string; chatId: string }>;
try {
candidates = await this.runRepo.findStaleActive(staleMs);
} catch (err) {
this.logger.warn(
`Reconcile (stale runs) query failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
return 0;
}
let aborted = 0;
for (const c of candidates) {
// PRIMARY gate: never touch a live entry, and never race a zombie we are
// already re-driving (settleZombie owns those).
if (this.active.has(c.id) || this.zombies.has(c.id)) continue;
try {
const row = await this.runRepo.finalizeIfActive(c.id, c.workspaceId, {
status: 'aborted',
error: 'Run aborted by reconcile: no live runner (stale).',
});
if (row) {
aborted += 1;
this.settled.add(c.id);
}
} catch (err) {
this.logger.warn(
`Reconcile abort of stale run ${c.id} failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
return aborted;
}
/**
* #487: the run's settle outcome as seen by THIS replica, or undefined when it
* has no record (the caller then reads the row the DB is the source of truth).
* A LIVE deferred (still settling, or resolved-but-not-yet-consumed) wins; a
* ZOMBIE synthesizes the give-up outcome. A subscriber (supersede) races this
* against a timeout.
*/
peekSettled(runId: string): Promise<RunSettleOutcome> | undefined {
const d = this.settledPromises.get(runId);
if (d) return d.promise;
const z = this.zombies.get(runId);
if (z) {
return Promise.resolve({
status: z.intended.status,
error: z.intended.error,
terminalWriteFailed: true,
});
}
return undefined;
}
/**
* #487: await a run's settle outcome, bounded by `timeoutMs`. Returns the
* outcome on settle, or undefined on TIMEOUT (or when this replica has no record
* of the run and its row is not terminal). Uses the LIVE settle notifier / the
* zombie synth when present; else reads the row (the DB is the source of truth
* once the in-memory record is gone). The subscriber (supersede) grabs this
* right after Stop; commit 1's race makes the settle land in ms on a healthy DB.
*/
async awaitSettled(
runId: string,
workspaceId: string,
timeoutMs: number,
): Promise<RunSettleOutcome | undefined> {
const pending = this.peekSettled(runId);
if (pending) {
let timer: ReturnType<typeof setTimeout> | undefined;
const timeout = new Promise<undefined>((resolve) => {
timer = setTimeout(() => resolve(undefined), timeoutMs);
timer.unref?.();
});
try {
return await Promise.race([pending, timeout]);
} finally {
if (timer) clearTimeout(timer);
}
}
// No live notifier and no zombie: read the row (already settled-and-written,
// or unknown here). A terminal row is an outcome; anything else -> undefined.
const row = await this.runRepo.findById(runId, workspaceId);
if (row && isRunTerminal(row.status)) {
return {
status: row.status as RunTerminalStatus,
error: row.error ?? null,
terminalWriteFailed: false,
};
}
return undefined;
}
/**
* #487: the SERVER supersede CAS for `POST /stream { supersede: { runId: X } }`.
* Atomically transitions "X is the chat's active run" -> "X is stopped, settled,
* slot free" so the caller can start a replacement run. See {@link
* SupersedeResult} for the branch semantics.
*
* On a `ready` result the caller MUST still go through the normal beginRun gate
* (the partial unique index) between the slot freeing here and beginRun a
* neighbouring tab's ordinary POST can win the slot (documented SLOT-THEFT: the
* loser then gets a MISMATCH carrying the NEW runId). There is also NO side-
* effect quiescence: an in-flight write of the stopped run may still land AFTER
* the new run starts (commit 1 stops the NEXT call, not one already committing),
* so the caller adds a prompt note to the new run.
*/
async supersede(
chatId: string,
targetRunId: string,
workspaceId: string,
timeoutMs: number = SUPERSEDE_SETTLE_TIMEOUT_MS,
): Promise<SupersedeResult> {
// Validate the target belongs to THIS chat (a CAS targeting another chat's run
// is malformed -> 400). A missing row is NOT invalid: the run may have ended
// and been pruned; the active-run check below decides degrade vs mismatch.
const target = await this.getRun(targetRunId, workspaceId);
if (target && target.chatId !== chatId) return { kind: 'invalid' };
const active = await this.getActiveForChat(chatId, workspaceId);
// No active run: it ended between the client's click and this POST — this is a
// DEGRADE to a normal send, NOT a mismatch (the user's intent still holds).
if (!active) return { kind: 'degrade' };
// A DIFFERENT run is active than the one the client saw -> mismatch. The
// client does not auto-retry; it surfaces the new runId.
if (active.id !== targetRunId) {
return { kind: 'mismatch', activeRunId: active.id };
}
// The target IS active: stop it, then await its settle within W.
await this.requestStop(targetRunId, workspaceId);
const outcome = await this.awaitSettled(targetRunId, workspaceId, timeoutMs);
if (!outcome) return { kind: 'timeout' };
// Gave up (terminal write failed): apply the intended status via the
// conditional UPDATE so the slot actually frees. If that ALSO fails, the row
// is still stranded -> treat as a timeout (nothing persisted for the new run).
if (outcome.terminalWriteFailed) {
const settled = await this.settleZombie(targetRunId);
if (!settled) return { kind: 'timeout' };
}
return { kind: 'ready' };
}
/** #487 test/diagnostic seam: whether a give-up zombie is held for this run. */
hasZombie(runId: string): boolean {
return this.zombies.has(runId);
}
/** #487: every zombie runId held on this replica (reconcile clause a, commit 4). */
zombieRunIds(): string[] {
return [...this.zombies.keys()];
}
/** #487: create a one-shot deferred (resolve captured for a later single call). */
private makeDeferred<T>(): Deferred<T> {
let resolve!: (value: T) => void;
const promise = new Promise<T>((r) => {
resolve = r;
});
return { promise, resolve };
}
/** #487: resolve a run's settle notifier EXACTLY ONCE, then drop it (bounded).
* A subscriber that already grabbed the promise still resolves; a later one
* falls back to the zombie map / the row (see peekSettled). */
private resolveSettled(runId: string, outcome: RunSettleOutcome): void {
const d = this.settledPromises.get(runId);
if (!d) return;
this.settledPromises.delete(runId);
d.resolve(outcome);
}
/** #487: read the persisted terminal outcome when the conditional finalize was a
* no-op (the row was already terminal). Falls back to the intended status when
* the read fails or the row is unexpectedly missing/non-terminal. */
private async readTerminalOutcome(
runId: string,
workspaceId: string,
fallbackStatus: RunTerminalStatus,
fallbackError: string | null,
): Promise<RunSettleOutcome> {
try {
const row = await this.runRepo.findById(runId, workspaceId);
if (row && isRunTerminal(row.status)) {
return {
status: row.status as RunTerminalStatus,
error: row.error ?? null,
terminalWriteFailed: false,
};
}
} catch {
// Fall through to the intended status — best-effort only.
}
return {
status: fallbackStatus,
error: fallbackError,
terminalWriteFailed: false,
};
}
/** Small async backoff between terminal-write retries (F6). Isolated so it is
@@ -0,0 +1,109 @@
import {
ConflictException,
Logger,
ServiceUnavailableException,
} from '@nestjs/common';
import { AiChatService } from './ai-chat.service';
import { RunAlreadyActiveError } from './ai-chat-run.service';
/**
* Fail-fast guard for beginRun failures (#486, commit 4).
*
* When runHooks.begin() rejects for a reason OTHER than RunAlreadyActiveError
* (e.g. a DB-pool blip), the turn must NOT continue untracked. The old code
* logged and streamed anyway, leaving a run with NO run-row: in autonomous mode
* nobody could abort it (/stop can't see it, disconnect doesn't abort it, and the
* one-run gate would admit a SECOND run) an unstoppable invisible run until
* restart. The fix throws A_RUN_BEGIN_FAILED (503) BEFORE the first byte and
* before the user row is persisted.
*
* We drive `stream()` directly on a prototype instance wired with only the
* collaborators it touches before the throw, so the assertion is on the REAL
* control flow, not a mock of it.
*/
describe('AiChatService beginRun failure (#486)', () => {
function makeService(insertSpy: jest.Mock): AiChatService {
// Bypass the (heavy) DI constructor: exercise the real stream() method on a
// bare prototype instance with just the fields reached before the throw.
// `any` because the private `logger` field makes a typed intersection collapse.
const svc = Object.create(AiChatService.prototype);
svc.aiChatRepo = {
// Existing chat -> no insert path; chatId is kept as-is.
findById: jest.fn().mockResolvedValue({ id: 'chat1' }),
};
svc.aiChatMessageRepo = { insert: insertSpy };
svc.logger = new Logger('test');
return svc as AiChatService;
}
const baseArgs = () => {
const write = jest.fn();
const res = {
raw: { write, writableEnded: false, headersSent: false },
};
return {
user: { id: 'u1' } as never,
workspace: { id: 'w1' } as never,
sessionId: 's1',
// openPage undefined -> resolveOpenPageContext returns null without any DB
// call; chatId present -> the existing-chat path.
body: { chatId: 'chat1', messages: [] } as never,
res: res as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
write,
};
};
it('throws A_RUN_BEGIN_FAILED (503) before the first byte and before persisting the user turn', async () => {
const insertSpy = jest.fn();
const svc = makeService(insertSpy);
const { write, ...args } = baseArgs();
const runHooks = {
begin: jest.fn().mockRejectedValue(new Error('DB pool exhausted')),
} as never;
let caught: unknown;
try {
await svc.stream({ ...args, runHooks });
} catch (e) {
caught = e;
}
expect(caught).toBeInstanceOf(ServiceUnavailableException);
const http = caught as ServiceUnavailableException;
expect(http.getStatus()).toBe(503);
expect(http.getResponse()).toMatchObject({ code: 'A_RUN_BEGIN_FAILED' });
// Fail-fast: nothing was written to the socket and NO user message row was
// persisted, so the turn left no orphan state to clean up.
expect(write).not.toHaveBeenCalled();
expect(insertSpy).not.toHaveBeenCalled();
});
it('still maps a lost-the-race RunAlreadyActiveError to a 409, not A_RUN_BEGIN_FAILED', async () => {
const insertSpy = jest.fn();
const svc = makeService(insertSpy);
const { write, ...args } = baseArgs();
const runHooks = {
begin: jest.fn().mockRejectedValue(new RunAlreadyActiveError('chat1')),
} as never;
let caught: unknown;
try {
await svc.stream({ ...args, runHooks });
} catch (e) {
caught = e;
}
expect(caught).toBeInstanceOf(ConflictException);
expect((caught as ConflictException).getResponse()).toMatchObject({
code: 'A_RUN_ALREADY_ACTIVE',
});
expect(write).not.toHaveBeenCalled();
expect(insertSpy).not.toHaveBeenCalled();
});
});
@@ -115,7 +115,7 @@ describe('finalizeAssistant dispatch (planFinalizeAssistant + applyFinalize)', (
// Drive the SAME applyFinalize the service calls (no duplicated logic).
async function dispatchFinalize(
repo: { insert: jest.Mock; update: jest.Mock },
repo: { insert: jest.Mock; finalizeOwner: jest.Mock },
assistantId: string | undefined,
flushed: AssistantFlush,
): Promise<void> {
@@ -135,21 +135,22 @@ describe('finalizeAssistant dispatch (planFinalizeAssistant + applyFinalize)', (
expect(planFinalizeAssistant(undefined)).toEqual({ kind: 'insert' });
});
it('(a) upfront insert succeeded -> finalize UPDATEs the row by id', async () => {
const repo = { insert: jest.fn(), update: jest.fn() };
it('(a) upfront insert succeeded -> finalize CONDITIONALLY updates the row by id (#487 owner-write)', async () => {
const repo = { insert: jest.fn(), finalizeOwner: jest.fn() };
const flushed = flushAssistant([], 'final answer', 'completed', {
finishReason: 'stop',
});
await dispatchFinalize(repo, 'a1', flushed);
expect(repo.update).toHaveBeenCalledWith('a1', workspaceId, flushed);
// #487: the owner write is the CONDITIONAL finalizeOwner, not a raw update.
expect(repo.finalizeOwner).toHaveBeenCalledWith('a1', workspaceId, flushed);
expect(repo.insert).not.toHaveBeenCalled();
});
it('(b) upfront insert failed -> finalize INSERTs the terminal payload', async () => {
const repo = { insert: jest.fn(), update: jest.fn() };
const repo = { insert: jest.fn(), finalizeOwner: jest.fn() };
const flushed = flushAssistant([], 'partial', 'error', { error: 'boom' });
await dispatchFinalize(repo, undefined, flushed);
expect(repo.update).not.toHaveBeenCalled();
expect(repo.finalizeOwner).not.toHaveBeenCalled();
expect(repo.insert).toHaveBeenCalledTimes(1);
const arg = repo.insert.mock.calls[0][0];
// The fallback insert carries the terminal content/status/metadata.
@@ -0,0 +1,221 @@
import {
BadRequestException,
ConflictException,
HttpException,
} from '@nestjs/common';
import { AiChatController } from './ai-chat.controller';
import type { User, Workspace } from '@docmost/db/types/entity.types';
/**
* #487 commit 3 the single concurrency GATE (both modes) + the server supersede
* CAS, at the controller boundary. The gate + CAS run BEFORE res.hijack(), so a
* rejected concurrent start / a CAS branch returns clean JSON (an HttpException
* the controller's post-hijack catch re-serializes). These assert the OBSERVABLE
* HTTP contract against the real controller + a stubbed run service.
*/
describe('#487 AiChatController.stream — gate + supersede', () => {
const user = { id: 'u1' } as User;
function wsWith(autonomousRuns: boolean): Workspace {
return {
id: 'ws1',
settings: { ai: { chat: true, autonomousRuns } },
} as unknown as Workspace;
}
function makeReqRes(body: Record<string, unknown>) {
const req = {
raw: { sessionId: 'sess', once: jest.fn(), destroyed: false },
body,
};
const res = {
raw: {
writableEnded: false,
headersSent: false,
on: jest.fn(),
once: jest.fn(),
setHeader: jest.fn(),
end: jest.fn(),
statusCode: 200,
flushHeaders: jest.fn(),
},
hijack: jest.fn(),
status: jest.fn().mockReturnThis(),
send: jest.fn(),
};
return { req, res };
}
function makeController(runServiceOverrides: Record<string, jest.Mock>) {
const aiChatService = {
resolveRoleForRequest: jest.fn().mockResolvedValue(null),
getChatModel: jest.fn().mockResolvedValue({}),
stream: jest.fn().mockResolvedValue(undefined),
};
const aiChatRunService = {
getActiveForChat: jest.fn().mockResolvedValue(undefined),
supersede: jest.fn(),
beginRun: jest.fn().mockResolvedValue({
runId: 'run-new',
signal: new AbortController().signal,
}),
linkAssistantMessage: jest.fn(),
recordStep: jest.fn(),
finalizeRun: jest.fn(),
requestStop: jest.fn(),
...runServiceOverrides,
};
const controller = new AiChatController(
aiChatService as never,
aiChatRunService as never,
{} as never, // aiChatRepo
{} as never, // aiChatMessageRepo
{} as never, // aiTranscription
{} as never, // pageRepo
);
return { controller, aiChatService, aiChatRunService };
}
const codeOf = (err: unknown) =>
(((err as HttpException).getResponse() as Record<string, unknown>) ?? {})
.code;
describe('single concurrency gate — BOTH modes reject the second tab with 409', () => {
for (const autonomousRuns of [true, false]) {
it(`rejects a concurrent start with 409 A_RUN_ALREADY_ACTIVE (autonomousRuns=${autonomousRuns})`, async () => {
const { controller, aiChatRunService } = makeController({
getActiveForChat: jest
.fn()
.mockResolvedValue({ id: 'run-live', chatId: 'c1' }),
});
const { req, res } = makeReqRes({ chatId: 'c1' });
let thrown: unknown;
try {
await controller.stream(
req as never,
res as never,
user,
wsWith(autonomousRuns),
);
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect((thrown as HttpException).getStatus()).toBe(409);
expect(codeOf(thrown)).toBe('A_RUN_ALREADY_ACTIVE');
// Rejected BEFORE committing to the stream (no hijack, no service.stream).
expect(res.hijack).not.toHaveBeenCalled();
expect(aiChatRunService.getActiveForChat).toHaveBeenCalledWith(
'c1',
'ws1',
);
});
}
});
it('supersede MISMATCH -> 409 SUPERSEDE_TARGET_MISMATCH carrying the current runId', async () => {
const { controller } = makeController({
supersede: jest
.fn()
.mockResolvedValue({ kind: 'mismatch', activeRunId: 'run-other' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect(codeOf(thrown)).toBe('SUPERSEDE_TARGET_MISMATCH');
expect(
((thrown as HttpException).getResponse() as Record<string, unknown>)
.activeRunId,
).toBe('run-other');
expect(res.hijack).not.toHaveBeenCalled();
});
it('supersede TIMEOUT -> 409 SUPERSEDE_TIMEOUT, nothing streamed', async () => {
const { controller } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'timeout' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(false));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect(codeOf(thrown)).toBe('SUPERSEDE_TIMEOUT');
expect(res.hijack).not.toHaveBeenCalled();
});
it('supersede INVALID (target on another chat) -> 400 SUPERSEDE_INVALID', async () => {
const { controller } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'invalid' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(BadRequestException);
expect(codeOf(thrown)).toBe('SUPERSEDE_INVALID');
});
it('supersede without chatId -> 400 SUPERSEDE_INVALID', async () => {
const { controller, aiChatRunService } = makeController({});
const { req, res } = makeReqRes({ supersede: { runId: 'run-x' } });
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(BadRequestException);
expect(codeOf(thrown)).toBe('SUPERSEDE_INVALID');
expect(aiChatRunService.supersede).not.toHaveBeenCalled();
});
it('supersede READY -> proceeds to stream with superseded=true', async () => {
const { controller, aiChatService } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'ready' }),
getActiveForChat: jest.fn().mockResolvedValue(undefined), // slot free after CAS
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
await controller.stream(req as never, res as never, user, wsWith(true));
expect(res.hijack).toHaveBeenCalled();
expect(aiChatService.stream).toHaveBeenCalledTimes(1);
expect(aiChatService.stream.mock.calls[0][0].superseded).toBe(true);
// The run hooks are always present now (both modes).
expect(aiChatService.stream.mock.calls[0][0].runHooks).toBeDefined();
});
it('supersede DEGRADE -> proceeds to a normal send (superseded=false)', async () => {
const { controller, aiChatService } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'degrade' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
await controller.stream(req as never, res as never, user, wsWith(false));
expect(aiChatService.stream).toHaveBeenCalledTimes(1);
expect(aiChatService.stream.mock.calls[0][0].superseded).toBe(false);
});
});
@@ -432,12 +432,66 @@ export class AiChatController {
// HttpException) instead of breaking mid-stream.
const model = await this.aiChatService.getChatModel(workspace.id, role);
// #184: one active run per chat. For an EXISTING chat reject a concurrent
// start with a clean 409 BEFORE hijack (the common double-submit / second-tab
// case), so the user gets JSON, not a mid-stream error. A brand-new chat
// (no chatId) cannot have a prior run, and the DB partial unique index is the
// backstop against any race that slips past this check.
if (autonomousRuns && body.chatId) {
// #487: server-side supersede CAS ("interrupt and send now"). When the client
// asks to replace a live run, atomically STOP it and wait for it to settle
// before this turn claims the slot. Runs BEFORE hijack so every branch returns
// clean JSON (the client keeps the composer text on a 409). See
// AiChatRunService.supersede for the branch semantics.
let superseded = false;
const supersedeRunId = body.supersede?.runId;
if (supersedeRunId) {
if (!body.chatId) {
throw new BadRequestException({
message: 'supersede requires chatId',
code: 'SUPERSEDE_INVALID',
});
}
const result = await this.aiChatRunService.supersede(
body.chatId,
supersedeRunId,
workspace.id,
);
switch (result.kind) {
case 'invalid':
throw new BadRequestException({
message: 'The run to supersede does not belong to this chat',
code: 'SUPERSEDE_INVALID',
});
case 'mismatch':
// A DIFFERENT run is active than the one the client targeted. Surface
// the CURRENT runId; the client does NOT auto-retry (a stale CAS).
throw new ConflictException({
message: 'A different agent run is now active on this chat',
code: 'SUPERSEDE_TARGET_MISMATCH',
activeRunId: result.activeRunId,
});
case 'timeout':
// The target did not settle within W — nothing was persisted, the
// composer keeps the text. NOT a rollback: the stop is already issued.
throw new ConflictException({
message:
'The previous run did not stop in time; nothing was sent — please try again',
code: 'SUPERSEDE_TIMEOUT',
});
case 'ready':
// The target stopped and settled: the slot is free. Prompt the new run
// that the old run's last operations may still be applying.
superseded = true;
break;
case 'degrade':
// The run already ended between click and POST — send normally.
break;
}
}
// #487: one active run per chat — ENFORCED IN BOTH MODES now (legacy mode used
// to have NO gate, so two tabs streamed two parallel turns on one chat, which
// interleaved history and crashed convertToModelMessages). Reject a concurrent
// start with a clean pre-hijack 409 (double-submit / second-tab). A brand-new
// chat (no chatId) cannot have a prior run, and the DB partial unique index in
// beginRun is the authoritative backstop for any race that slips past here
// (including a slot stolen between a supersede release and beginRun).
if (body.chatId) {
const active = await this.aiChatRunService.getActiveForChat(
body.chatId,
workspace.id,
@@ -446,107 +500,94 @@ export class AiChatController {
throw new ConflictException({
message: 'An agent run is already in progress for this chat',
code: 'A_RUN_ALREADY_ACTIVE',
activeRunId: active.id,
});
}
}
// Run-lifecycle hooks (#184), only when the flag is on. They wrap the turn in
// a durable run whose abort is governed by the run (explicit stop), persist
// its progress, and settle its terminal status — see AiChatRunService.
const runHooks: AiChatRunHooks | undefined = autonomousRuns
? {
begin: async (chatId) => {
const handle = await this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
});
// #184 phase 1.5: register the run-stream entry at BEGIN (before any
// frame) so a tab that attaches in the begin->seed window finds an
// entry to wait on. Gated on AI_CHAT_RESUMABLE_STREAM: with the flag
// off nothing is registered and attach always 204s.
if (
handle?.runId &&
this.environment?.isAiChatResumableStreamEnabled?.()
) {
this.streamRegistry?.open(chatId, handle.runId);
}
return handle;
},
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
workspace.id,
messageId,
),
onStep: (runId, stepCount) =>
void this.aiChatRunService.recordStep(
runId,
workspace.id,
stepCount,
),
onSettled: (runId, status, error) =>
this.aiChatRunService.finalizeRun(
runId,
workspace.id,
status,
error,
),
// #487: the turn is ALWAYS a first-class RUN now (both modes). The mode
// difference is only the abort semantics on a browser disconnect (onClose
// below). currentRunId is captured at begin so a legacy disconnect can stop
// the run through its stop lever.
let currentRunId: string | undefined;
const runHooks: AiChatRunHooks = {
begin: async (chatId) => {
const handle = await this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
});
currentRunId = handle?.runId;
// #184 phase 1.5: register the run-stream entry at BEGIN (before any
// frame) so a tab that attaches in the begin->seed window finds an entry
// to wait on. Gated on AI_CHAT_RESUMABLE_STREAM.
if (
handle?.runId &&
this.environment?.isAiChatResumableStreamEnabled?.()
) {
this.streamRegistry?.open(chatId, handle.runId);
}
: undefined;
return handle;
},
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
workspace.id,
messageId,
),
onStep: (runId, stepCount) =>
void this.aiChatRunService.recordStep(runId, workspace.id, stepCount),
onSettled: (runId, status, error) =>
this.aiChatRunService.finalizeRun(runId, workspace.id, status, error),
};
// Abort the agent loop when the client disconnects. `close` also fires on
// normal completion, so only abort when the response has not finished
// writing (a genuine disconnect). `once` fires at most once and self-removes;
// we also drop it on response `finish` so it never lingers after the stream
// completes normally (the AI SDK pipes the response fire-and-forget, so we
// cannot simply remove it once `stream()` returns).
// Handle a client disconnect. `close` also fires on normal completion, so only
// act when the response has not finished writing (a genuine disconnect). `once`
// fires at most once and self-removes; we also drop it on response `finish`.
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: wall-clock at
// which a Safari disconnect is observed, measured from request receipt.
const reqStartedAt = Date.now();
const controller = new AbortController();
const onClose = (): void => {
// A genuine disconnect leaves the response unfinished (unlike a normal
// completion, which also fires `close`). Such a drop — e.g. a reverse
// proxy cutting the SSE mid-answer — is otherwise invisible server-side,
// so log it here.
if (!res.raw.writableEnded) {
if (autonomousRuns) {
// #184: the turn is a DETACHED run. A disconnect must NOT abort it —
// the run keeps executing and persisting server-side; the client
// reconnects via /ai-chat/run (or re-stops via /ai-chat/stop). Log only.
// #184: a DETACHED run — a disconnect must NOT stop it. The run keeps
// executing and persisting server-side; the client reconnects via
// /ai-chat/run (or re-stops via /ai-chat/stop). Log only.
this.logger.log(
`AI chat stream: client disconnected; run continues server-side ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
} else {
// #487: legacy — a disconnect ENDS the turn, but the turn is now a RUN,
// so stop it through the run's stop lever (requestStop). streamText no
// longer consumes the socket signal (effectiveSignal is the run signal),
// so aborting `controller` would do nothing; requestStop aborts the run.
this.logger.warn(
`AI chat stream: client disconnected before completion; aborting turn ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
`AI chat stream: client disconnected before completion; stopping the ` +
`run (elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
controller.abort();
if (currentRunId) {
void this.aiChatRunService.requestStop(currentRunId, workspace.id);
}
}
}
};
req.raw.once('close', onClose);
res.raw.once('finish', () => req.raw.off('close', onClose));
// #184: in detached mode the turn is NOT aborted on disconnect, so the SDK's
// pipe keeps writing to a socket the client may have dropped — for the rest of
// the (continuing) run. A write to the dead socket can emit an 'error' on the
// raw response; without a listener that surfaces as an unhandled error event.
// Swallow it (the run continues server-side regardless). Legacy mode aborts on
// disconnect, so it does not need this and keeps its exact prior behavior.
if (autonomousRuns) {
res.raw.on('error', (err) => {
this.logger.debug(
`AI chat detached stream: post-disconnect socket error swallowed: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
}
// #184/#487: the run/pipe can outlive the socket in BOTH modes now (autonomous
// keeps going; legacy keeps going until requestStop's abort unwinds the turn).
// The SDK's pipe may then write to a dropped socket and emit an 'error' on the
// raw response — swallow it so it never surfaces as an unhandled error event.
res.raw.on('error', (err) => {
this.logger.debug(
`AI chat stream: post-disconnect socket error swallowed: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
// Commit to streaming: hijack so Fastify stops managing the response and
// the AI SDK can write the UI-message stream directly to the Node socket.
@@ -562,8 +603,10 @@ export class AiChatController {
signal: controller.signal,
model,
role,
// #184: present only when the flag is on; wraps the turn in a durable run.
// #487: the turn is always run-wrapped now (both modes).
runHooks,
// #487: warn the new run that a superseded run's last ops may still apply.
superseded,
});
} catch (err) {
// Any failure AFTER hijack can no longer go through Nest's exception
@@ -101,6 +101,22 @@ const INTERRUPT_NOTE =
'assume your previous response was complete, and do not silently restart the ' +
'partial work — build on it or follow the new instruction.';
/**
* #487: injected on a turn started by SUPERSEDING a previous run (the user hit
* "interrupt and send now" while a run was live). The previous run was Stopped,
* but there is NO side-effect quiescence a write it had already committed, or
* one committing at the moment of Stop, may land with a small delay AFTER this new
* run starts. So the model is told its picture of the page/state may be a beat
* stale and to re-read before assuming an edit did or did not apply.
*/
const SUPERSEDE_NOTE =
'NOTE: A previous agent run in this conversation was just interrupted so this ' +
'new turn could start. That run was stopped, but any operation it had already ' +
'begun (e.g. a page edit) may still be applied with a short delay. Do not ' +
'assume the document/state is exactly as the interrupted run left it — if you ' +
'need to rely on the current content, RE-READ it with the page tools before ' +
'acting rather than trusting a cached view.';
/**
* Injected on a turn where the open page was hand-edited by the user (or anyone
* else) AFTER the agent's previous response ended (#274). The server takes a
@@ -203,6 +219,14 @@ export interface BuildSystemPromptInput {
* (partial) answer was cut off by the user's new message.
*/
interrupted?: boolean;
/**
* #487: true when THIS turn was started by superseding a still-live previous run
* ("interrupt and send now"). Adds SUPERSEDE_NOTE so the model knows the previous
* run's last operations may still be applying and to re-read state it depends on.
* Distinct from `interrupted` (which is about a PARTIAL prior answer in history);
* both can be set together. Self-clears set only for the superseding turn.
*/
superseded?: boolean;
/**
* Set only when the open page was edited by the user AFTER the agent's previous
* turn ended (#274), confirmed server-side by diffing the current page against
@@ -311,6 +335,7 @@ export function buildSystemPrompt({
openedPage,
mcpInstructions,
interrupted,
superseded,
pageChanged,
deferredToolsEnabled,
toolCatalog,
@@ -360,6 +385,13 @@ export function buildSystemPrompt({
context += `\n${INTERRUPT_NOTE}`;
}
// Supersede note (#487): present only for a turn that stopped and replaced a
// still-live previous run — warns the model the previous run's last operations
// may still be applying (no side-effect quiescence).
if (superseded) {
context += `\n${SUPERSEDE_NOTE}`;
}
// Per-turn page-change note (#274). Added to the context section (inside the
// safety sandwich), present only when the server detected that the open page
// was edited by the user since the agent's last turn ended. The diff content is
@@ -89,6 +89,11 @@ describe('AiChatService.stream run-lifecycle safety net (#184)', () => {
const runRepo = {
insert: jest.fn().mockResolvedValue({ id: 'run-1', status: 'running' }),
update: jest.fn().mockResolvedValue({ id: 'run-1' }),
// #487: the terminal settle now goes through the CONDITIONAL write.
finalizeIfActive: jest
.fn()
.mockResolvedValue({ id: 'run-1', status: 'failed' }),
findById: jest.fn().mockResolvedValue(undefined),
};
const runService = new AiChatRunService(runRepo as never, { isCloud: () => false } as never);
@@ -148,9 +153,10 @@ describe('AiChatService.stream run-lifecycle safety net (#184)', () => {
// The run was begun...
expect(runRepo.insert).toHaveBeenCalledTimes(1);
// ...then settled to a terminal FAILED status by the safety net...
expect(runRepo.update).toHaveBeenCalledTimes(1);
expect(runRepo.update).toHaveBeenCalledWith(
// ...then settled to a terminal FAILED status by the safety net (via the
// #487 conditional write)...
expect(runRepo.finalizeIfActive).toHaveBeenCalledTimes(1);
expect(runRepo.finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws1',
expect.objectContaining({ status: 'failed' }),
@@ -1,4 +1,8 @@
import { ConflictException, Logger } from '@nestjs/common';
import {
ConflictException,
Logger,
ServiceUnavailableException,
} from '@nestjs/common';
// Mock the AI SDK so we can PROVE no provider call is made for the turn we are
// about to reject. The race rejection happens at runHooks.begin(), long before
@@ -151,6 +155,8 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -328,7 +334,13 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
usage: {},
steps: [],
});
expect(runHooks.onSettled).toHaveBeenCalledWith('run-1', 'completed');
// #487: onFinish passes the (undefined) error slot so a message-finalize
// failure could error-mark the run; on the success path it is undefined.
expect(runHooks.onSettled).toHaveBeenCalledWith(
'run-1',
'completed',
undefined,
);
});
it('F9: onAbort settles the run "aborted"', async () => {
@@ -360,22 +372,22 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
});
/**
* F14 the begin-failure RESILIENCE branch (the `else` of the run-race guard).
* F14 the begin-failure branch (the `else` of the run-race guard).
*
* stream() wraps runHooks.begin in try/catch with TWO branches:
* - RunAlreadyActiveError -> 409 ConflictException (pinned above).
* - ANY OTHER begin failure -> SWALLOW + continue UNTRACKED on the socket signal
* (legacy fallback): it logs "...streaming without run tracking", leaves
* `effectiveSignal = signal` (runId undefined) and serves the turn anyway.
* - ANY OTHER begin failure -> throw ServiceUnavailableException(A_RUN_BEGIN_FAILED)
* BEFORE the first byte (#486, commit 4).
*
* The contract: a transient beginRun failure (e.g. a non-unique DB error inserting
* the run row) must STILL serve the user's turn it must NOT re-throw and must NOT
* be misclassified as a 409. A regression that re-threw here would break EVERY turn
* on a begin failure with nothing to catch it. This branch is otherwise undriven by
* any spec, so it is pinned here SEPARATELY from the 409 path: a plain begin error
* proceeds to streamText with the SOCKET signal and still persists the user turn.
* POLICY CHANGE (#486): the OLD contract here was "SWALLOW + stream the turn
* UNTRACKED on the socket signal". That was reversed: an untracked run is
* invisible to /stop, is not aborted on disconnect, and slips past the one-run
* gate an unstoppable ghost run in autonomous mode. Now a plain begin failure
* FAILS the turn fast with a 503 A_RUN_BEGIN_FAILED, before any user row is
* persisted and before streamText runs. This case is INVERTED (not deleted) so
* the "plain begin failure" path stays explicitly pinned under the new policy.
*/
describe('AiChatService.stream — begin-failure resilience / legacy fallback (#184 F14)', () => {
describe('AiChatService.stream — begin-failure fails the turn (#184 F14 / #486)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
function makeStreamResult() {
@@ -411,6 +423,8 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -455,7 +469,7 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
afterEach(() => jest.restoreAllMocks());
it('a PLAIN begin() failure (NOT RunAlreadyActiveError) does NOT 409 — it swallows, logs, and streams the turn UNTRACKED on the socket signal', async () => {
it('a PLAIN begin() failure (NOT RunAlreadyActiveError) FAILS the turn with a 503 A_RUN_BEGIN_FAILED before the first byte — NO untracked stream (#486)', async () => {
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
@@ -487,28 +501,26 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
} as never,
});
// The turn proceeds: NO throw at all (in particular NOT a 409).
await expect(promise).resolves.toBeUndefined();
// NEW POLICY: the turn is REJECTED with a 503 A_RUN_BEGIN_FAILED (not a 409,
// and NOT swallowed into an untracked stream).
await expect(promise).rejects.toBeInstanceOf(ServiceUnavailableException);
const err = (await promise.catch(
(e) => e,
)) as ServiceUnavailableException;
expect(err.getStatus()).toBe(503);
expect(err.getResponse()).toMatchObject({ code: 'A_RUN_BEGIN_FAILED' });
expect(begin).toHaveBeenCalledTimes(1);
// The resilience branch logged the legacy-fallback warning.
// It logged the fail-the-turn line.
expect(errorSpy).toHaveBeenCalledWith(
expect.stringContaining('streaming without run tracking'),
expect.stringContaining('failing the turn'),
expect.anything(),
);
// The turn really streamed: the user message was persisted and streamText ran.
expect(aiChatMessageRepo.insert).toHaveBeenCalled();
expect(streamTextMock).toHaveBeenCalledTimes(1);
// The decisive wiring: with no run handle, the fallback uses the SOCKET signal
// (effectiveSignal = signal, runId undefined) — not a run-bound signal. #444:
// the signal is unioned with the degeneration controller via AbortSignal.any,
// so assert the socket abort still reaches the turn rather than identity.
const passed = streamTextMock.mock.calls[0][0].abortSignal as AbortSignal;
expect(passed.aborted).toBe(false);
socketController.abort();
expect(passed.aborted).toBe(true);
// Fail-fast: the turn NEVER streamed — no user row persisted, no streamText
// call, so no orphan/untracked run was left behind.
expect(aiChatMessageRepo.insert).not.toHaveBeenCalled();
expect(streamTextMock).not.toHaveBeenCalled();
});
});
@@ -1315,8 +1315,12 @@ describe('AiChatService page-change lifecycle (#274)', () => {
describe('isInterruptResume', () => {
// history tail is the just-inserted user row; [len-2] is the previous turn.
const withPrev = (
prev: { role: string; status?: string | null } | null,
): Array<{ role: string; status?: string | null }> =>
prev: {
role: string;
status?: string | null;
metadata?: unknown;
} | null,
): Array<{ role: string; status?: string | null; metadata?: unknown }> =>
prev
? [prev, { role: 'user', status: null }]
: [{ role: 'user', status: null }];
@@ -1357,6 +1361,33 @@ describe('isInterruptResume', () => {
it('false when there is no preceding turn (only the new user row)', () => {
expect(isInterruptResume(withPrev(null), true)).toBe(false);
});
it('#487 EXCLUDES a reconcile stamp (finalizeFailed) — not a genuine interruption', () => {
// A row a reconcile settled to 'aborted' carries metadata.finalizeFailed. It
// must NOT be treated as an interrupt-resume (that would inject a false
// "you were interrupted" note), even though its status is 'aborted'.
expect(
isInterruptResume(
withPrev({
role: 'assistant',
status: 'aborted',
metadata: { finalizeFailed: true },
}),
true,
),
).toBe(false);
// A genuine abort (no finalizeFailed) still counts.
expect(
isInterruptResume(
withPrev({
role: 'assistant',
status: 'aborted',
metadata: { parts: [] },
}),
true,
),
).toBe(true);
});
});
/**
@@ -1419,6 +1450,9 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
// #487: the terminal owner-write + the opportunistic reconcile query.
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -1623,6 +1657,19 @@ describe('AiChatService.stream — token-degeneration reaction (#444)', () => {
return { id };
},
),
// #487: the terminal owner-write records into the SAME `updated` recorder so
// assertions on the terminal 'completed'/'error'/'aborted' write still hold.
finalizeOwner: jest.fn(
async (
id: string,
workspaceId: string,
patch: Record<string, unknown>,
) => {
updated.push({ id, workspaceId, patch });
return { id };
},
),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
+320 -41
View File
@@ -3,7 +3,9 @@ import {
ForbiddenException,
Injectable,
Logger,
OnModuleDestroy,
OnModuleInit,
ServiceUnavailableException,
} from '@nestjs/common';
import { FastifyReply } from 'fastify';
import {
@@ -41,7 +43,11 @@ import {
makeLoadToolsTool,
buildExternalToolCatalog,
} from './tools/tool-tiers';
import { RunAlreadyActiveError } from './ai-chat-run.service';
import {
RunAlreadyActiveError,
AiChatRunService,
} from './ai-chat-run.service';
import { inAppToolCallCapMs } from './tools/ai-chat-tools.service';
import { computePageChange } from './page-change/page-change.util';
import {
sanitizeSelection,
@@ -55,6 +61,7 @@ import {
import {
isDegenerateOutput,
truncateDegeneratedTail,
shouldCheckDegeneration,
} from './output-degeneration';
// Max agent steps per turn. One step = one model generation; a step that calls
@@ -248,15 +255,23 @@ export function cleanGeneratedTitle(text: string): string {
* partial output is already in history thanks to the step-granular write path).
*/
export function isInterruptResume(
history: Array<{ role: string; status?: string | null }>,
history: Array<{
role: string;
status?: string | null;
metadata?: unknown;
}>,
clientInterrupted: boolean | undefined,
): boolean {
if (clientInterrupted !== true) return false;
const prev = history[history.length - 2];
return (
prev?.role === 'assistant' &&
(prev.status === 'aborted' || prev.status === 'streaming')
);
if (prev?.role !== 'assistant') return false;
// #487: a reconcile STAMP (metadata.finalizeFailed) is NOT a genuine user
// interruption — the previous turn's process died and a reconcile settled the
// row as 'aborted'. Treating it as an interrupt-resume would inject a false
// "you were interrupted" note. Exclude any finalizeFailed row.
const meta = prev.metadata as { finalizeFailed?: unknown } | null | undefined;
if (meta && meta.finalizeFailed === true) return false;
return prev.status === 'aborted' || prev.status === 'streaming';
}
/**
@@ -377,6 +392,14 @@ export interface AiChatStreamBody {
// it against persisted history (`isInterruptResume`) before injecting the
// interrupt note, so a spoofed/stale flag on an ordinary turn is ignored.
interrupted?: boolean;
// #487: server-side supersede CAS. When present, this POST asks the server to
// STOP the run `supersede.runId` (which the client saw as the chat's active run)
// and, once it has settled, start THIS turn in its place. The server validates
// the target against the chat and answers 400 (wrong chat) / 409
// SUPERSEDE_TARGET_MISMATCH / 409 SUPERSEDE_TIMEOUT, or proceeds normally
// (degrade / ready). Absent => an ordinary send (rejected with 409
// A_RUN_ALREADY_ACTIVE if a run is already active on the chat).
supersede?: { runId?: string } | null;
// useChat sends the full UIMessage list; the last one is the new user turn.
messages?: UIMessage[];
}
@@ -426,6 +449,11 @@ export interface AiChatStreamArgs {
// chat row (existing chat) or the request body (new chat). null => universal
// assistant. Carried here so the turn never re-loads it.
role: AiAgentRole | null;
// #487: true when this turn was started by SUPERSEDING a still-live previous run
// (the controller ran the supersede CAS to a `ready` result). Adds the
// SUPERSEDE_NOTE to the system prompt (the previous run's last ops may still be
// applying — no side-effect quiescence). Absent on an ordinary send.
superseded?: boolean;
}
/**
@@ -442,7 +470,7 @@ export interface AiChatStreamArgs {
* can be rebuilt for `convertToModelMessages`.
*/
@Injectable()
export class AiChatService implements OnModuleInit {
export class AiChatService implements OnModuleInit, OnModuleDestroy {
private readonly logger = new Logger(AiChatService.name);
constructor(
@@ -463,8 +491,17 @@ export class AiChatService implements OnModuleInit {
// constructions (int-specs) compile unchanged; Nest always injects the real
// provider in production. Only ever touched on the run-wrapped + flag-on path.
private readonly streamRegistry?: AiChatStreamRegistryService,
// #487: the run lifecycle service, for the periodic + opportunistic reconcile
// (zombie re-drive + stale-run abort). OPTIONAL so positional test
// constructions compile unchanged; Nest always injects the real singleton, so
// reconcile sees the SAME in-memory active/zombie maps the runner mutates.
private readonly aiChatRunService?: AiChatRunService,
) {}
// #487: periodic reconcile timer (single-process phase 1). Started in
// onModuleInit, cleared in onModuleDestroy.
private reconcileTimer?: ReturnType<typeof setInterval>;
/**
* Crash-recovery sweep on server start (#183): any assistant row left in the
* 'streaming' state is the relic of a turn whose process died before it
@@ -489,6 +526,158 @@ export class AiChatService implements OnModuleInit {
}`,
);
}
// #487: start the PERIODIC reconcile (was boot-only). It heals both directions
// of the run<->message lifecycle asymmetry that a boot sweep alone left to the
// NEXT restart. Single-process phase 1: the in-memory active/zombie maps are
// authoritative, so "no live entry" is a safe primary gate.
const staleMs = this.reconcileStalenessMs();
// boot-warn if the per-call cap is configured so high the derived staleness is
// unusually long (a stale run then lingers longer before reconcile aborts it).
if (staleMs > 30 * 60 * 1000) {
this.logger.warn(
`#487 reconcile staleness is ${Math.round(staleMs / 60000)}min ` +
`(derived from max(2 x per-call cap, 15min)); a per-call cap this high ` +
`delays stale-run recovery. Review AI_CHAT_INAPP_TOOL_CALL_CAP_MS.`,
);
}
const intervalMs = this.reconcileIntervalMs();
this.reconcileTimer = setInterval(() => {
void this.reconcile().catch((err) => {
this.logger.warn(
`Periodic reconcile failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
});
}, intervalMs);
this.reconcileTimer.unref?.();
}
/** #487: stop the periodic reconcile timer on shutdown. */
onModuleDestroy(): void {
if (this.reconcileTimer) {
clearInterval(this.reconcileTimer);
this.reconcileTimer = undefined;
}
}
/**
* #487: reconcile staleness threshold X a run/message is only a "no live
* runner" abort candidate once UNTOUCHED past this. Derived as
* max(2 x per-call cap, 15min): 2x the longest legitimate single tool call plus
* a floor, so a marathon turn making steady progress (updatedAt bumped each
* step) is never swept.
*/
private reconcileStalenessMs(): number {
return Math.max(2 * inAppToolCallCapMs(), 15 * 60 * 1000);
}
/** #487: how often the periodic reconcile runs (env-tunable, default 2min). */
private reconcileIntervalMs(): number {
const raw = Number(process.env.AI_CHAT_RECONCILE_INTERVAL_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 2 * 60 * 1000;
}
/**
* #487: the periodic BIDIRECTIONAL reconcile. Runs the clauses IN ORDER; each is
* best-effort (a failure of one never blocks the others). Single-process phase 1
* the run service's in-memory maps are authoritative for "live entry".
*
* (a) re-drive ZOMBIE runs (a terminal write that gave up) apply the intended
* status via the conditional UPDATE;
* (b) message 'streaming' + its RUN terminal -> stamp the message by the run's
* status (succeeded-run + stuck row -> 'aborted'+finalizeFailed, NOT
* 'completed' with empty parts the final text lived only in the dead
* process's memory, a documented loss);
* (c) run active + NO live entry + NO zombie + stale -> aborted (the run
* service applies the "no entry" primary gate + last-progress staleness);
* (d) message 'streaming' + age>X + NO active run on the chat -> aborted
* (historical-row safety, double-gated).
*/
async reconcile(): Promise<void> {
const staleMs = this.reconcileStalenessMs();
// (a) zombie re-drive.
if (this.aiChatRunService) {
for (const runId of this.aiChatRunService.zombieRunIds()) {
try {
await this.aiChatRunService.settleZombie(runId);
} catch (err) {
this.logger.warn(
`Reconcile (a) zombie ${runId} re-drive failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
}
// (b) message streaming + run terminal -> stamp message by run status.
try {
const stuck = await this.aiChatMessageRepo.findStreamingWithTerminalRun();
for (const s of stuck) {
// succeeded-run -> 'aborted' (NOT 'completed'-empty); failed -> 'error';
// aborted -> 'aborted'. All via the finalizeFailed stamp.
const status = s.runStatus === 'failed' ? 'error' : 'aborted';
await this.aiChatMessageRepo.stampTerminalIfStreaming(
s.messageId,
s.workspaceId,
status,
);
}
} catch (err) {
this.logger.warn(
`Reconcile (b) message<-run failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
// (c) stale active run with no live runner -> aborted.
if (this.aiChatRunService) {
try {
await this.aiChatRunService.reconcileStaleRuns(staleMs);
} catch (err) {
this.logger.warn(
`Reconcile (c) stale-run abort failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
// (d) historical streaming row, no active run on the chat, stale -> aborted.
try {
await this.aiChatMessageRepo.sweepStreamingWithoutActiveRun(staleMs);
} catch (err) {
this.logger.warn(
`Reconcile (d) historical-row sweep failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
/**
* #487: OPPORTUNISTIC single-chat reconcile at the start of a turn (beginRun /
* supersede path), so a user who returns to a chat with a stuck streaming row
* (its run already terminal) sees it settled WITHOUT waiting for the periodic
* job. Best-effort a failure NEVER fails the turn (swallowed by the caller).
*/
async reconcileChat(chatId: string, workspaceId: string): Promise<void> {
const stuck = await this.aiChatMessageRepo.findStreamingWithTerminalRun(50, {
chatId,
workspaceId,
});
for (const s of stuck) {
const status = s.runStatus === 'failed' ? 'error' : 'aborted';
await this.aiChatMessageRepo.stampTerminalIfStreaming(
s.messageId,
s.workspaceId,
status,
);
}
}
/**
@@ -725,6 +914,7 @@ export class AiChatService implements OnModuleInit {
model,
role,
runHooks,
superseded,
}: AiChatStreamArgs): Promise<void> {
// Resolve / create the chat. A new chat is created when no valid chatId is
// supplied or the supplied one does not belong to this workspace.
@@ -797,15 +987,49 @@ export class AiChatService implements OnModuleInit {
code: 'A_RUN_ALREADY_ACTIVE',
});
}
// Any OTHER run-start failure must not break the turn — fall back to the
// socket signal (legacy behavior) and stream anyway.
// Any OTHER run-start failure (e.g. a DB-pool blip) must FAIL THE TURN,
// not silently stream without a run-row. The old fallback let the turn
// continue untracked: in autonomous mode nobody could then abort it —
// /stop can't see a run that doesn't exist, a client disconnect doesn't
// abort it, and the one-run-per-chat gate would let a SECOND run in. That
// is an unstoppable, invisible run until process restart. Reject NOW,
// BEFORE the first byte (nothing is written yet, no user row inserted, no
// MCP lease taken), so the controller's post-hijack catch turns this
// HttpException into an honest 503 on the raw socket. Same policy for BOTH
// modes — #487 inherits it (no mode-branching here).
this.logger.error(
`Failed to begin agent run (chat ${chatId}); streaming without run tracking`,
`Failed to begin agent run (chat ${chatId}); failing the turn`,
err as Error,
);
throw new ServiceUnavailableException({
message:
'Could not start the agent run. This is usually temporary — please try again.',
code: 'A_RUN_BEGIN_FAILED',
// Self-describe the status in the body: the controller's post-hijack
// catch writes getResponse() verbatim onto the raw socket, and an
// object-arg HttpException does NOT inject statusCode. Without it the
// client's 503 classifier (which reads the body JSON) could not see the
// status. With it present, the client's A_RUN_BEGIN_FAILED branch (which
// runs strictly before the generic-503 branch) shows "temporary, retry".
statusCode: 503,
});
}
}
// #487: opportunistic single-chat reconcile — settle any streaming row on this
// chat whose run is already terminal BEFORE this turn's history load, so the
// user never waits on the periodic job and the new turn's model history is not
// polluted by a stuck 'streaming' row. Best-effort: it must NEVER fail the turn.
try {
await this.reconcileChat(chatId, workspace.id);
} catch (err) {
this.logger.debug(
`Opportunistic reconcile for chat ${chatId} failed (ignored): ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
try {
// Extract the incoming user turn (the last user message from useChat).
const incoming = lastUserMessage(body.messages);
@@ -1011,6 +1235,9 @@ export class AiChatService implements OnModuleInit {
// History-confirmed interrupt-resume flag (#198): adds the interrupt note
// so the model treats the partial answer above as cut off, not finished.
interrupted,
// #487: this turn superseded a still-live run — warn the model the
// previous run's last ops may still be applying (no quiescence).
superseded,
// Detected between-turns human edit to the open page (#274): adds the
// page_changed note + unified diff so the agent doesn't overwrite it.
pageChanged,
@@ -1080,7 +1307,6 @@ export class AiChatService implements OnModuleInit {
const degenerationController = new AbortController();
let degenerationDetected = false;
let lastDegenerationCheckLen = 0;
const DEGENERATION_CHECK_STEP = 2000;
// Step-granular durability (#183): create the assistant row UPFRONT in the
// 'streaming' state (before any token), then UPDATE it as each step finishes
@@ -1166,29 +1392,59 @@ export class AiChatService implements OnModuleInit {
// callbacks — mirroring the pre-#183 persist-at-most-once guard for the
// TERMINAL status (the row may be updated many times with 'streaming' before
// this fires once).
// #487: the once-gate closes ONLY AFTER a successful write, and the write is
// BOUNDED-RETRIED. Previously `finalized` was set BEFORE the write and never
// retried, so a single failed UPDATE stranded the row 'streaming' forever
// (the boot-only sweep was the only recovery). Now a transient blip is ridden
// out in place; a total give-up leaves the gate OPEN and logs, and the
// periodic reconcile (clauses b/d) later settles the row. Returns whether the
// terminal write LANDED, so the caller can error-mark the RUN on a message
// failure (the run is finalized regardless — never gated on the message).
let finalized = false;
const FINALIZE_MSG_MAX_ATTEMPTS = 3;
const finalizeAssistant = async (
flushed: AssistantFlush,
): Promise<void> => {
if (finalized) return;
finalized = true;
): Promise<boolean> => {
if (finalized) return true;
const plan = planFinalizeAssistant(assistantId);
try {
// Shared dispatch (see applyFinalize): UPDATE the upfront row, or — when
// the upfront insert failed (kind 'insert') — INSERT the terminal row as
// the only safety against losing the turn entirely.
await applyFinalize(
this.aiChatMessageRepo,
plan,
{ chatId, workspaceId: workspace.id, userId: user.id },
flushed,
);
} catch (err) {
this.logger.error(
`Failed to finalize assistant message (kind=${plan.kind})`,
err as Error,
);
let lastError: unknown;
for (let attempt = 1; attempt <= FINALIZE_MSG_MAX_ATTEMPTS; attempt++) {
try {
// Shared dispatch (see applyFinalize): conditionally UPDATE the upfront
// row (owner-write priority), or — when the upfront insert failed (kind
// 'insert') — INSERT the terminal row as the only safety against losing
// the turn entirely.
await applyFinalize(
this.aiChatMessageRepo,
plan,
{ chatId, workspaceId: workspace.id, userId: user.id },
flushed,
);
finalized = true; // gate closes ONLY after a successful write
return true;
} catch (err) {
lastError = err;
this.logger.warn(
`Assistant message finalize attempt ${attempt}/${FINALIZE_MSG_MAX_ATTEMPTS} ` +
`failed (kind=${plan.kind}): ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
if (attempt < FINALIZE_MSG_MAX_ATTEMPTS) {
await new Promise((r) => setTimeout(r, 50 * attempt));
}
}
}
// Gave up: leave the gate OPEN (no in-process second settler exists — the
// terminal callbacks are mutually exclusive) and log. The periodic reconcile
// settles the stranded row; a late owner-write is impossible for this turn,
// so the reconcile stamp (aborted+finalizeFailed) is the final state.
this.logger.error(
`Assistant message finalize GAVE UP after ${FINALIZE_MSG_MAX_ATTEMPTS} ` +
`attempts (row left 'streaming', chat ${chatId}); reconcile will settle it`,
lastError as Error,
);
return false;
};
// DIAGNOSTIC (Safari stream-drop investigation) — temporary. Measure
@@ -1254,8 +1510,10 @@ export class AiChatService implements OnModuleInit {
// trigger, abort the run ONCE with a distinguishable reason.
if (
!degenerationDetected &&
inProgressText.length - lastDegenerationCheckLen >=
DEGENERATION_CHECK_STEP
shouldCheckDegeneration(
inProgressText.length,
lastDegenerationCheckLen,
)
) {
lastDegenerationCheckLen = inProgressText.length;
if (isDegenerateOutput(inProgressText)) {
@@ -1275,6 +1533,13 @@ export class AiChatService implements OnModuleInit {
// the in-progress accumulator for the next step.
capturedSteps.push(step as StepLike);
inProgressText = '';
// Reset the degeneration-check watermark too (#486): it tracks a byte
// offset INTO inProgressText, so once that resets to '' a stale (large)
// mark makes `inProgressText.length - lastDegenerationCheckLen` go
// negative and the throttled detector stays silent until a later step's
// text re-grows past the old offset — a whole degenerate step could slip
// through undetected. Zeroing it re-arms the check from the next byte.
lastDegenerationCheckLen = 0;
// Step-granular durability (#183): persist this finished step (its text +
// tool calls + tool RESULTS) the moment it ends, so a process death after
// this point still recovers the step. Not awaited here (never block the
@@ -1324,7 +1589,7 @@ export class AiChatService implements OnModuleInit {
const stepExhausted = steps.length >= MAX_AGENT_STEPS;
const emptyTurnMarker =
!producedText && stepExhausted ? STEP_LIMIT_NO_ANSWER_MARKER : '';
await finalizeAssistant(
const msgOk = await finalizeAssistant(
flushAssistant(steps as StepLike[], emptyTurnMarker, 'completed', {
finishReason: finishReason as string,
usage: totalUsage as StreamUsage,
@@ -1338,9 +1603,19 @@ export class AiChatService implements OnModuleInit {
pageChanged,
}),
);
// #184: settle the RUN as succeeded (best-effort, after the projection
// is finalized above).
if (runId) await runHooks?.onSettled?.(runId, 'completed');
// #184/#487: the RUN is finalized ALWAYS (never gated on the message).
// If the message finalize GAVE UP, error-mark the run so the asymmetry
// "run succeeded / message streaming forever" cannot arise; the
// periodic reconcile then settles the stuck message from this run.
if (runId) {
await runHooks?.onSettled?.(
runId,
msgOk ? 'completed' : 'error',
msgOk
? undefined
: 'Assistant message could not be persisted (finalize failed).',
);
}
// Lifecycle: release the external MCP clients leased for this turn.
await closeExternalClients();
@@ -2081,7 +2356,10 @@ export function planFinalizeAssistant(
* a test mock both satisfy it). */
export interface FinalizeRepo {
insert(insertable: Record<string, unknown>): Promise<unknown>;
update(
// #487: the OWNER terminal write is CONDITIONAL (status='streaming' OR
// metadata.finalizeFailed) so the owner overwrites a reconcile stamp but never
// an already-proper terminal row (owner-write priority).
finalizeOwner(
id: string,
workspaceId: string,
patch: AssistantFlush,
@@ -2090,10 +2368,11 @@ export interface FinalizeRepo {
/**
* Apply a finalize `plan` to the repo with the terminal `flushed` payload (#183):
* UPDATE the upfront row, or INSERT a fresh terminal row as the fallback when the
* upfront insert failed. The SINGLE dispatch shared by the service's
* finalizeAssistant and its test, so the test exercises the real path instead of
* a copy (#186 review). Pure of error handling the caller wraps it.
* conditionally UPDATE the upfront row (owner-write priority, #487), or INSERT a
* fresh terminal row as the fallback when the upfront insert failed. The SINGLE
* dispatch shared by the service's finalizeAssistant and its test, so the test
* exercises the real path instead of a copy (#186 review). Pure of error
* handling the caller wraps it (and RETRIES it, #487).
*/
export async function applyFinalize(
repo: FinalizeRepo,
@@ -2102,7 +2381,7 @@ export async function applyFinalize(
flushed: AssistantFlush,
): Promise<void> {
if (plan.kind === 'update') {
await repo.update(plan.id, base.workspaceId, flushed);
await repo.finalizeOwner(plan.id, base.workspaceId, flushed);
return;
}
await repo.insert({
@@ -1,11 +1,24 @@
import { Logger } from '@nestjs/common';
import { streamText } from 'ai';
import {
hasRepeatedLineRun,
hasPeriodicTail,
isDegenerateOutput,
truncateDegeneratedTail,
shouldCheckDegeneration,
DEGENERATION_CHECK_STEP,
REPEATED_LINES_THRESHOLD,
MIN_PERIOD_REPEATS,
} from './output-degeneration';
import { AiChatService } from './ai-chat.service';
// Mock ONLY streamText so we can capture the onChunk/onStepFinish callbacks the
// service registers and drive them by hand; every other `ai` export the service
// uses (convertToModelMessages, stepCountIs, …) stays real.
jest.mock('ai', () => {
const actual = jest.requireActual('ai');
return { ...actual, streamText: jest.fn() };
});
/**
* Unit tests for the token-degeneration detector (#444) the sole anti-babble
@@ -180,3 +193,188 @@ describe('truncateDegeneratedTail', () => {
expect(truncateDegeneratedTail(text)).toBe(text);
});
});
/**
* Throttle + step-boundary reset (#486). The stream keeps a watermark
* (`lastDegenerationCheckLen`) that is an OFFSET into the accumulated step text.
* On a step boundary the accumulator resets to '', so the watermark MUST reset to
* 0 too otherwise the throttle goes silent for the whole next step. These tests
* pin the pure decision AND the reset property that ai-chat.service.onStepFinish
* now enforces.
*/
describe('shouldCheckDegeneration (throttle) + step-boundary reset (#486)', () => {
it('fires once the text grows a full DEGENERATION_CHECK_STEP past the mark', () => {
expect(shouldCheckDegeneration(DEGENERATION_CHECK_STEP, 0)).toBe(true);
expect(shouldCheckDegeneration(DEGENERATION_CHECK_STEP - 1, 0)).toBe(false);
expect(shouldCheckDegeneration(5000, 3000)).toBe(true); // grew 2000 since mark
expect(shouldCheckDegeneration(4000, 3000)).toBe(false); // grew only 1000
});
it('BUG (no reset): a stale large watermark silences the next step', () => {
// End of a long step: the watermark sits at 5000. The step ends and the
// accumulator resets to '' — but if the watermark is NOT reset, a fresh short
// degenerate burst (length 2000) never triggers a check: 2000 - 5000 < STEP.
const staleWatermark = 5000;
const nextStepLen = DEGENERATION_CHECK_STEP; // a fresh 2KB burst
expect(shouldCheckDegeneration(nextStepLen, staleWatermark)).toBe(false);
});
it('FIX (reset to 0): the same short degenerate burst IS checked and detected', () => {
// onStepFinish now zeroes the watermark, so the fresh burst re-arms the check.
const resetWatermark = 0;
const degenerateBurst = 'loadTools.\n'.repeat(300); // real degeneration
expect(degenerateBurst.length).toBeGreaterThanOrEqual(DEGENERATION_CHECK_STEP);
// The throttle now fires...
expect(
shouldCheckDegeneration(degenerateBurst.length, resetWatermark),
).toBe(true);
// ...and the detector catches the loop that would otherwise stream unchecked.
expect(isDegenerateOutput(degenerateBurst)).toBe(true);
});
});
/**
* BEHAVIOR guard for the ACTUAL fix (#486, ai-chat.service.onStepFinish resets
* lastDegenerationCheckLen to 0). The pure tests above use a hard-coded
* resetWatermark, so a REVERT of the real `lastDegenerationCheckLen = 0` line
* would not redden any of them. This drives the REAL onChunk/onStepFinish
* closures from stream() end to end and asserts the run is aborted when a fresh
* degenerate burst arrives in the step AFTER a long clean step which only
* happens if the watermark was actually zeroed on the step boundary.
*/
describe('AiChatService: onStepFinish re-arms the degeneration watermark (#486)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
function makeRes() {
return {
raw: {
writeHead: jest.fn(),
write: jest.fn(),
once: jest.fn(),
on: jest.fn(),
flushHeaders: jest.fn(),
writableEnded: false,
destroyed: false,
},
};
}
function makeService() {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const aiChatMessageRepo = {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
const mcpClients = {
toolsFor: jest.fn(async () => ({
tools: {},
clients: [],
outcomes: [],
instructions: [],
})),
};
return new AiChatService(
{} as never, // ai
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
aiSettings as never,
tools as never,
mcpClients as never,
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
// Lockdown OFF -> the degeneration guard is the active anti-babble path.
isAiChatFinalStepLockdownEnabled: () => false,
} as never, // environment
);
}
beforeEach(() => {
streamTextMock.mockReset();
jest.spyOn(Logger.prototype, 'log').mockImplementation(() => undefined as never);
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined as never);
});
afterEach(() => jest.restoreAllMocks());
it('aborts on a fresh degenerate burst in the NEXT step (reverting the reset line reddens this)', async () => {
let captured:
| {
onChunk?: (e: { chunk: { type: string; text: string } }) => void;
onStepFinish?: (step: unknown) => void;
abortSignal?: AbortSignal;
}
| undefined;
streamTextMock.mockImplementation((opts: never) => {
captured = opts;
return {
consumeStream: jest.fn(),
pipeUIMessageStreamToResponse: jest.fn(),
};
});
const svc = makeService();
await svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: {
chatId: 'chat-1',
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
} as never,
res: makeRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
// No runHooks -> legacy path (socket signal), degeneration guard active.
});
expect(streamTextMock).toHaveBeenCalledTimes(1);
const onChunk = captured!.onChunk!;
const onStepFinish = captured!.onStepFinish!;
const abortSignal = captured!.abortSignal!;
expect(abortSignal.aborted).toBe(false);
// STEP 1: a LONG, non-degenerate first step. Distinct lines never trip the
// detector, but they advance the throttle watermark far past the burst size
// that follows (to ~5x the step). This is the stale watermark that, WITHOUT
// the reset, would silence step 2.
let counter = 0;
let accumulated = 0;
while (accumulated < DEGENERATION_CHECK_STEP * 5) {
const line = `unique clean line number ${counter++} with distinct words\n`;
accumulated += line.length;
onChunk({ chunk: { type: 'text-delta', text: line } });
}
expect(abortSignal.aborted).toBe(false); // clean step must not abort
// STEP BOUNDARY: the real onStepFinish resets inProgressText AND (the fix)
// zeroes lastDegenerationCheckLen.
onStepFinish({ text: 'a clean first step', toolCalls: [], toolResults: [] });
// STEP 2: a FRESH, short degenerate burst (~3.3KB). Its length is far below
// the step-1 stale watermark (~10KB), so WITHOUT the reset the throttle stays
// silent and this streams unchecked. WITH the reset (watermark 0) it re-arms,
// the detector fires, and the run aborts.
const burst = 'loadTools.\n'.repeat(300);
expect(burst.length).toBeGreaterThanOrEqual(DEGENERATION_CHECK_STEP);
expect(burst.length).toBeLessThan(DEGENERATION_CHECK_STEP * 5);
onChunk({ chunk: { type: 'text-delta', text: burst } });
// The decisive assertion: the composed abortSignal (unioned with the
// degeneration controller) is now aborted. Reverting `lastDegenerationCheckLen
// = 0` in onStepFinish makes this stay false.
expect(abortSignal.aborted).toBe(true);
});
});
@@ -131,6 +131,32 @@ export function isDegenerateOutput(text: string): boolean {
return hasRepeatedLineRun(text) || hasPeriodicTail(text);
}
/**
* How many bytes the in-progress text must grow before the (amortized) tail
* heuristics are re-run. Shared with ai-chat.service so the throttle the stream
* applies is the SAME one the unit test drives.
*/
export const DEGENERATION_CHECK_STEP = 2000;
/**
* Throttle decision for the degeneration guard (#444/#486). Returns true when
* the accumulated text has grown at least DEGENERATION_CHECK_STEP bytes past the
* last-checked offset, so the pure rules only fire every ~2KB. Pure; the caller
* updates its watermark to `textLen` when this returns true.
*
* The watermark is an offset INTO the accumulator, so when the accumulator is
* reset to '' on a step boundary the caller MUST reset the watermark to 0 too
* (#486). Otherwise `textLen - lastCheckLen` goes negative after the reset and
* this returns false until a later step re-grows past the stale offset a whole
* degenerate step could stream unchecked.
*/
export function shouldCheckDegeneration(
textLen: number,
lastCheckLen: number,
): boolean {
return textLen - lastCheckLen >= DEGENERATION_CHECK_STEP;
}
/**
* Truncate a degenerated tail before persist so hundreds of KB of garbage never
* reach the DB / replay (#444). Keeps everything up to and including the FIRST
@@ -0,0 +1,241 @@
// Break the editor-ext import chain (share.service -> collaboration.util ->
// @docmost/editor-ext -> @tiptap/core) that is unresolvable in this jest env and
// pre-existingly breaks these specs. jsonToMarkdown is never reached in these
// tests (the tools fail before rendering markdown).
jest.mock('../../collaboration/collaboration.util', () => ({
jsonToMarkdown: () => '',
}));
import { Logger } from '@nestjs/common';
import { MockLanguageModelV3, simulateReadableStream } from 'ai/test';
import { PublicShareChatService } from './public-share-chat.service';
import { PublicShareChatToolsService } from './tools/public-share-chat-tools.service';
/**
* SECURITY integration guard for #394 (commit 5): a tool's or the provider's raw
* error text must NOT leak to an anonymous public-share reader.
*
* The render gate (ToolCallCard showErrors=false) hides the text in the DOM but
* NOT on the wire, so this test asserts on the RAW SSE BYTES the server writes
* exactly the channel the render gate masks. We drive the real
* PublicShareChatService.stream() with a real share toolset (its underlying
* services mocked to fail) and a mock model, then inspect every byte piped to the
* fake socket.
*/
// A minimal ServerResponse stand-in that records every written chunk.
class FakeSocket {
chunks: string[] = [];
statusCode = 200;
writableEnded = false;
destroyed = false;
headersSent = false;
writeHead(): this {
this.headersSent = true;
return this;
}
setHeader(): void {}
removeHeader(): void {}
getHeader(): undefined {
return undefined;
}
flushHeaders(): void {}
write(chunk: unknown): boolean {
this.chunks.push(
typeof chunk === 'string' ? chunk : Buffer.from(chunk as never).toString('utf8'),
);
return true;
}
end(chunk?: unknown): void {
if (chunk) this.write(chunk);
this.writableEnded = true;
}
on(): this {
return this;
}
once(): this {
return this;
}
get body(): string {
return this.chunks.join('');
}
}
/** Mock model that issues one getSharePage tool call, then finishes with text. */
function toolCallingModel(): MockLanguageModelV3 {
let call = 0;
return new MockLanguageModelV3({
doStream: async () => {
call++;
if (call === 1) {
return {
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'tool-input-start' as const, id: 't1', toolName: 'getSharePage' },
{ type: 'tool-input-end' as const, id: 't1' },
{
type: 'tool-call' as const,
toolCallId: 't1',
toolName: 'getSharePage',
input: '{"pageId":"secret-page"}',
},
{
type: 'finish' as const,
finishReason: { unified: 'tool-calls' as const, raw: 'tool_calls' },
usage: {
inputTokens: { total: 1, noCache: undefined, cacheRead: undefined, cacheWrite: undefined },
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
};
}
return {
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'text-start' as const, id: '1' },
{ type: 'text-delta' as const, id: '1', delta: 'Sorry.' },
{ type: 'text-end' as const, id: '1' },
{
type: 'finish' as const,
finishReason: { unified: 'stop' as const, raw: 'stop' },
usage: {
inputTokens: { total: 1, noCache: undefined, cacheRead: undefined, cacheWrite: undefined },
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
};
},
});
}
/** Mock model whose stream emits a provider error carrying an internal secret. */
function providerErrorModel(secret: string): MockLanguageModelV3 {
return new MockLanguageModelV3({
doStream: async () => ({
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{
type: 'error' as const,
error: {
statusCode: 503,
message: 'Service Unavailable',
responseBody: `upstream ${secret} model=internal-gpt`,
},
},
],
}),
}),
});
}
function makeService(toolsService: PublicShareChatToolsService): {
svc: PublicShareChatService;
logSpy: jest.SpyInstance;
} {
const svc = Object.create(PublicShareChatService.prototype);
const logger = new Logger('test');
const logSpy = jest.spyOn(logger, 'error').mockImplementation(() => undefined);
jest.spyOn(logger, 'warn').mockImplementation(() => undefined);
svc.tools = toolsService;
svc.logger = logger;
svc.tokenBudget = { record: jest.fn().mockResolvedValue(undefined) };
return { svc, logSpy };
}
async function runStream(
svc: PublicShareChatService,
model: MockLanguageModelV3,
): Promise<FakeSocket> {
const socket = new FakeSocket();
await svc.stream({
workspaceId: 'ws1',
shareId: 'share1',
share: { id: 'share1', pageId: 'p1', sharedPage: { id: 'p1', title: 'Docs' } },
openedPage: null,
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'read the page' }] } as never,
],
res: { raw: socket } as never,
signal: new AbortController().signal,
model: model as never,
role: null,
});
// Let the piped stream drain fully.
await new Promise((r) => setTimeout(r, 300));
return socket;
}
describe('public share chat error leak (#394)', () => {
afterEach(() => jest.restoreAllMocks());
it('does NOT leak a tool\'s raw internal error to the SSE bytes (generic classified string instead)', async () => {
const SECRET = 'INTERNAL_baseUrl_http://provider.internal:8080/v1';
const shareService = {
// The canonical boundary throws a RAW internal error (with a secret).
resolveReadableSharePage: jest
.fn()
.mockRejectedValue(new Error(`db failed at ${SECRET} stack@line42`)),
};
const tools = new PublicShareChatToolsService(
shareService as never,
{} as never,
{} as never,
);
const { svc } = makeService(tools);
const socket = await runStream(svc, toolCallingModel());
// The tool-output-error frame is present on the wire...
expect(socket.body).toContain('tool-output-error');
// ...but it carries ONLY the generic classified string — never the secret,
// the raw driver message, or a stack fragment.
expect(socket.body).toContain('The tool could not complete the request.');
expect(socket.body).not.toContain(SECRET);
expect(socket.body).not.toContain('stack@line42');
expect(socket.body).not.toContain('db failed');
});
it('passes a SAFE ShareToolError message (page not available) through to the bytes', async () => {
const shareService = {
// Not found in this share -> the tool throws the classified SAFE message.
resolveReadableSharePage: jest.fn().mockResolvedValue(null),
};
const tools = new PublicShareChatToolsService(
shareService as never,
{} as never,
{} as never,
);
const { svc } = makeService(tools);
const socket = await runStream(svc, toolCallingModel());
expect(socket.body).toContain('tool-output-error');
expect(socket.body).toContain('not available in this share');
});
it('does NOT leak a provider error (statusCode + response body) to the SSE bytes', async () => {
const SECRET = 'http://provider.internal:8080';
const tools = new PublicShareChatToolsService(
{} as never,
{} as never,
{} as never,
);
const { svc, logSpy } = makeService(tools);
const socket = await runStream(svc, providerErrorModel(SECRET));
// The anon sees a fixed classified string, not the provider body/baseUrl/model.
expect(socket.body).toContain('temporarily unavailable');
expect(socket.body).not.toContain(SECRET);
expect(socket.body).not.toContain('internal-gpt');
// The FULL provider detail is logged server-side only.
const logged = logSpy.mock.calls.map((c) => String(c[0])).join('\n');
expect(logged).toContain(SECRET);
});
});
@@ -12,7 +12,10 @@ import { AiAgentRoleRepo } from '@docmost/db/repos/ai-agent-roles/ai-agent-roles
import { AiAgentRole } from '@docmost/db/types/entity.types';
import { AiService } from '../../integrations/ai/ai.service';
import { AiSettingsService } from '../../integrations/ai/ai-settings.service';
import { PublicShareChatToolsService } from './tools/public-share-chat-tools.service';
import {
PublicShareChatToolsService,
ShareToolError,
} from './tools/public-share-chat-tools.service';
import { buildShareSystemPrompt } from './public-share-chat.prompt';
import { roleModelOverride } from './roles/role-model-config';
import {
@@ -102,6 +105,30 @@ export function filterShareTranscript(messages: UIMessage[]): UIMessage[] {
);
}
/**
* Fixed, classified strings an ANONYMOUS share reader may see when the assistant
* stream fails (#394). These reveal NOTHING about the internal provider, its
* baseUrl, the model name, or the raw response body unlike describeProviderError
* (which is for the server log / the authenticated operator only). We classify by
* HTTP status where available so the reader still gets a useful hint (retry vs.
* give up) without any internal detail.
*/
export function classifyAnonStreamError(error: unknown): string {
const status =
typeof error === 'object' && error !== null
? (error as { statusCode?: number }).statusCode
: undefined;
if (status === 429) {
return 'The assistant is receiving too many requests right now. Please try again shortly.';
}
if (typeof status === 'number' && status >= 500) {
return 'The assistant is temporarily unavailable. Please try again.';
}
// Any other failure (including a bare connection error with no status): a
// single neutral line. No provider identity, no config, no response body.
return 'The assistant could not complete your request. Please try again.';
}
/**
* Anonymous, read-only AI assistant for a single PUBLIC share tree.
*
@@ -318,11 +345,28 @@ export class PublicShareChatService {
result.pipeUIMessageStreamToResponse(res.raw, {
headers: { 'X-Accel-Buffering': 'no' },
onError: (error: unknown) => {
// Reuse the shared formatter so provider error formatting stays
// unified between the log line and the streamed error message — a
// share reader sees 402/429/503 causes consistently with the
// authenticated path.
return describeProviderError(error, 'AI stream error');
// SECURITY (#394): the string this returns is written verbatim into the
// SSE error frame delivered to an ANONYMOUS reader (for a tool failure
// it becomes the atomic `tool-output-error` frame's errorText; for a
// stream/provider failure, the terminal error frame).
//
// A ShareToolError is already a classified, safe tool message (see
// PublicShareChatToolsService.wrapToolErrors) — pass it through so the
// reader still gets the useful "page not available in this share" hint.
if (error instanceof ShareToolError) {
return error.message;
}
// Anything else is a provider/stream error. describeProviderError
// bundles the provider statusCode AND response body, which can carry the
// internal baseUrl or model name — NEVER expose that to the public. Log
// the full detail server-side only and return a fixed classified string.
this.logger.error(
`Public share chat pipe error: ${describeProviderError(
error,
'AI stream error',
)}`,
);
return classifyAnonStreamError(error);
},
});
@@ -0,0 +1,160 @@
import {
wrapInAppToolWithCap,
inAppToolCallCapMs,
type ToolAbortSignalSink,
} from './ai-chat-tools.service';
import type { Tool, ToolCallOptions } from 'ai';
/**
* #487 commit 1 in-app tool race-on-abort + safe-points + per-call cap.
*
* Tests assert the HONEST observable property the spec names "after Stop, NO
* new HTTP/WS call STARTS; an already-started single call may take either
* outcome" against the REAL wrapper mechanism (the composite abort signal it
* publishes on the client + the RACE it runs), NOT a timing-dependent proxy like
* "the write didn't land".
*/
// A minimal stand-in for the client's `toolAbortSignal` field. In production the
// wrapper publishes the composite here and the client's paginateAll /
// mutatePageContent safe-points read it; the fake "tool" below reads it the same
// way, so this exercises the real contract without a live DB / collab socket.
class FakeClient implements ToolAbortSignalSink {
private signal: AbortSignal | null = null;
setToolAbortSignal(signal: AbortSignal | null): void {
this.signal = signal;
}
getToolAbortSignal(): AbortSignal | null {
return this.signal;
}
}
// A ToolCallOptions with just the field the wrapper reads (abortSignal). The AI
// SDK passes a fuller object; the wrapper only spreads it and reads abortSignal.
const opts = (abortSignal?: AbortSignal): ToolCallOptions =>
({ toolCallId: 't1', messages: [], abortSignal }) as unknown as ToolCallOptions;
const tick = (ms = 5) => new Promise((r) => setTimeout(r, ms));
describe('#487 wrapInAppToolWithCap — race-on-abort + safe-points', () => {
it('after Stop, no NEW simulated call starts (multi-call tool)', async () => {
const client = new FakeClient();
const started: number[] = [];
// A multi-call tool that mirrors paginateAll: it consults the client signal
// at a safe-point BEFORE starting each simulated network call.
const multiCall: Tool = {
execute: (async (_args: unknown) => {
for (let i = 0; i < 6; i++) {
// Safe-point: exactly what paginateAll / mutatePageContent do.
client.getToolAbortSignal()?.throwIfAborted();
started.push(i);
await tick(10);
}
return 'done';
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(multiCall, client, 10_000);
const ac = new AbortController();
const call = (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
// Let one or two calls start, then Stop.
await tick(12);
ac.abort(new Error('user stop'));
await expect(call).rejects.toThrow(); // wrapper rejects promptly
const startedAtStop = started.length;
// Give the abandoned loser ample time; its next safe-point must throw because
// the (aborted) composite is still published on the client.
await tick(60);
expect(started.length).toBe(startedAtStop);
// It must NOT have run the whole sequence (that would mean Stop did nothing).
expect(started.length).toBeLessThan(6);
});
it('rejects immediately on Stop even if the call never settles (discard loser)', async () => {
const client = new FakeClient();
let settled = false;
const hang: Tool = {
execute: (async () => {
await new Promise(() => undefined); // never resolves
settled = true;
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(hang, client, 10_000);
const ac = new AbortController();
const call = (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
await tick(5);
ac.abort();
await expect(call).rejects.toThrow();
expect(settled).toBe(false);
});
it('per-call cap rejects a hung call with no Stop signal', async () => {
const client = new FakeClient();
const hang: Tool = {
execute: (async () => {
await new Promise(() => undefined);
}) as unknown as Tool['execute'],
} as Tool;
// Tiny cap; no options.abortSignal at all (composite = cap only).
const wrapped = wrapInAppToolWithCap(hang, client, 20);
const start = Date.now();
await expect(
(wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>)(
{},
opts(undefined),
),
).rejects.toThrow(/per-call cap/);
expect(Date.now() - start).toBeLessThan(2000);
});
it('publishes a composite signal on the client for the duration of the call', async () => {
const client = new FakeClient();
let seenDuringCall: AbortSignal | null = null;
const probe: Tool = {
execute: (async () => {
seenDuringCall = client.getToolAbortSignal();
return 'ok';
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(probe, client, 10_000);
const ac = new AbortController();
await (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
expect(seenDuringCall).not.toBeNull();
// The published composite must reflect the turn's Stop signal.
ac.abort();
expect((seenDuringCall as unknown as AbortSignal).aborted).toBe(true);
});
it('a completed call returns its raw result unchanged', async () => {
const client = new FakeClient();
const ok: Tool = {
execute: (async () => ({ items: [1, 2, 3] })) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(ok, client, 10_000);
const res = await (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(new AbortController().signal));
expect(res).toEqual({ items: [1, 2, 3] });
});
it('cap is env-tunable with a 2-minute default', () => {
const prev = process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
delete process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
expect(inAppToolCallCapMs()).toBe(120_000);
process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = '5000';
expect(inAppToolCallCapMs()).toBe(5000);
process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = 'not-a-number';
expect(inAppToolCallCapMs()).toBe(120_000);
if (prev === undefined) delete process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
else process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = prev;
});
});
@@ -1,5 +1,5 @@
import { Injectable, Logger } from '@nestjs/common';
import { tool, type Tool } from 'ai';
import { tool, type Tool, type ToolCallOptions } from 'ai';
import { z } from 'zod';
import { User } from '@docmost/db/types/entity.types';
import { TokenService } from '../../auth/services/token.service';
@@ -159,6 +159,129 @@ function __assertClientCallContract(client: DocmostClientLike): void {
* existing service-account `/mcp` path already calls loopback successfully, so
* this works for single-workspace self-host.
*/
/**
* #487: wall-clock cap for a SINGLE in-app tool call, env-tunable via
* `AI_CHAT_INAPP_TOOL_CALL_CAP_MS`. Bounds a read tool that would otherwise
* paginate for minutes and a content write whose collab commit hangs, and is the
* per-call CAP half of the composite abort signal every in-app tool is wrapped
* with (the other half is the turn's Stop signal). Default 2 minutes: generous
* for a legitimate long read/write, tight enough that a stuck call cannot pin the
* turn. The reconcile staleness floor (#487 commit 4) is derived as
* max(2 x this cap, 15 min), so keep this well under that.
*/
export function inAppToolCallCapMs(): number {
const raw = Number(process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 120_000;
}
/** #487: the composite signal's reason as an Error (informative thrown value). */
function inAppAbortReason(signal: AbortSignal): Error {
const r = signal.reason;
return r instanceof Error
? r
: new Error(typeof r === 'string' ? r : 'In-app tool call aborted');
}
/**
* The client surface {@link wrapInAppToolWithCap} drives (#487). Both methods are
* OPTIONAL: the real loopback DocmostClient implements them (so a Stop/cap reaches
* its pagination / pre-commit safe-points), but a client that omits them still
* gets the OUTER guarantee the race rejects on abort regardless. This keeps the
* wrapper decoupled from the exact client shape (unit-test doubles need not stub
* the plumbing).
*/
export interface ToolAbortSignalSink {
setToolAbortSignal?(signal: AbortSignal | null): void;
getToolAbortSignal?(): AbortSignal | null;
}
/**
* #487: wrap an in-app tool so a Stop (the turn's `options.abortSignal`) OR the
* per-call wall-clock cap REJECTS the call immediately, and so that SAME
* composite signal reaches the client's pagination / pre-commit safe-points (via
* `client.setToolAbortSignal`) making a Stop stop the NEXT HTTP/WS call from
* starting.
*
* Reuses the RACE pattern of `wrapToolWithCallTimeout` (mcp-clients.service.ts):
* the call is raced against the composite signal, so on abort we reject in the
* SAME tick and DISCARD the loser promise. Its network / collab teardown latency
* therefore never blocks the turn the supersede timeout W=10s (#487 commit 3)
* relies on this abort->settle latency being milliseconds, not a socket teardown.
* Awaiting the client's own signal-into-write path alone would NOT satisfy this
* (the loser could still be tearing down a collab socket).
*
* The composite is SET on the client at entry and deliberately NOT restored on
* unwind: after this wrapper rejects on abort, the ABANDONED loser promise keeps
* running, and its safe-points read the client field leaving the (aborted)
* composite there is exactly what makes the loser's NEXT call throw and stop. The
* next in-app tool call overwrites the field with its own fresh composite before
* any of its safe-points run, so a stale settled signal never leaks forward.
* SINGLE-WRITER by phase-1 assumption see DocmostClientContext.toolAbortSignal
* for the parallel-call caveat (#487).
*
* KNOWN LIMITATION (#487): a write tool that issues SEVERAL sequential collab
* commits can be aborted BETWEEN commits, leaving a partially-applied operation.
* Cancel guarantees "no NEW call starts", NOT "the write didn't land".
*/
export function wrapInAppToolWithCap(
toolDef: Tool,
client: ToolAbortSignalSink,
capMs: number,
): Tool {
const original = toolDef.execute;
if (typeof original !== 'function') return toolDef;
const execute = async (args: unknown, options: ToolCallOptions) => {
const capController = new AbortController();
const timer = setTimeout(() => {
capController.abort(
new Error(`In-app tool call exceeded the ${capMs}ms per-call cap`),
);
}, capMs);
timer.unref?.();
const composite = options?.abortSignal
? AbortSignal.any([options.abortSignal, capController.signal])
: capController.signal;
// Reject the MOMENT the composite fires, independent of whether `original`
// ever settles (a hung collab write / read would otherwise pin the turn). The
// losing `original` is left pending; Promise.race attaches a rejection
// handler to both inputs so a late rejection is never unhandled.
const aborted = new Promise<never>((_, reject) => {
const fail = () => reject(inAppAbortReason(composite));
if (composite.aborted) fail();
else composite.addEventListener('abort', fail, { once: true });
});
// Publish the composite so the client's pagination / pre-commit safe-points
// observe it (see the "not restored on unwind" rationale above). Guarded: a
// client without the plumbing still gets the OUTER race guarantee below.
client.setToolAbortSignal?.(composite);
try {
return await Promise.race([
(original as (a: unknown, o: ToolCallOptions) => Promise<unknown>)(
args,
{ ...options, abortSignal: composite },
),
aborted,
]);
} finally {
clearTimeout(timer);
}
};
return { ...toolDef, execute } as unknown as Tool;
}
/** #487: apply {@link wrapInAppToolWithCap} to every tool in a set. */
export function wrapInAppToolsWithCap(
tools: Record<string, Tool>,
client: ToolAbortSignalSink,
capMs: number,
): Record<string, Tool> {
const out: Record<string, Tool> = {};
for (const [name, t] of Object.entries(tools)) {
out[name] = wrapInAppToolWithCap(t, client, capMs);
}
return out;
}
@Injectable()
export class AiChatToolsService {
private readonly logger = new Logger(AiChatToolsService.name);
@@ -186,7 +309,12 @@ export class AiChatToolsService {
sessionId: string,
workspaceId: string,
aiChatId: string,
): Promise<DocmostClientLike> {
// #487: the returned client also carries the tool-cancellation plumbing
// (setToolAbortSignal/getToolAbortSignal). These are host plumbing, NOT part
// of the tool-execute surface (DocmostClientMethod), so they are surfaced here
// as an intersection rather than by widening that Pick — keeping the
// positional-call drift-guard (#446) scoped to the actual tool methods.
): Promise<DocmostClientLike & ToolAbortSignalSink> {
const apiUrl =
process.env.MCP_DOCMOST_API_URL ||
`http://127.0.0.1:${process.env.PORT || 3000}/api`;
@@ -630,7 +758,15 @@ export class AiChatToolsService {
// dependency and reuses the CASL enforcement already on `client`. When the
// loaded package predates #417 (factory undefined) or the loader is mocked in
// a unit test, signalling is a pure no-op and results are byte-identical.
if (!createCommentSignalTracker) return tools;
// #487: wrap every in-app tool with the race-on-abort + per-call cap guard so
// a Stop / cap rejects immediately AND reaches the client's write/pagination
// safe-points. Applied as the OUTERMOST wrapper (over the comment-signal
// wrapper below) so the race governs the whole call. The client carries the
// per-call composite signal via setToolAbortSignal.
const capMs = inAppToolCallCapMs();
if (!createCommentSignalTracker) {
return wrapInAppToolsWithCap(tools, client, capMs);
}
const tracker = createCommentSignalTracker({
probe: async (pageId: string, sinceMs: number) => {
@@ -659,7 +795,11 @@ export class AiChatToolsService {
},
});
return wrapToolsWithCommentSignal(tools, tracker);
return wrapInAppToolsWithCap(
wrapToolsWithCommentSignal(tools, tracker),
client,
capMs,
);
}
}
@@ -1,7 +1,15 @@
import { createHash } from 'node:crypto';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import {
mkdtempSync,
mkdirSync,
writeFileSync,
rmSync,
readdirSync,
statSync,
readFileSync,
} from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { dirname, join, relative, sep } from 'node:path';
import { computeSrcRegistryStamp } from './docmost-client.loader';
@@ -30,10 +38,14 @@ function assertStaleGuard(
}
}
// Build a throwaway `<pkg>/build/index.js` + optional `<pkg>/src/tool-specs.ts`
// layout so `computeSrcRegistryStamp(<pkg>/build/index.js)` resolves src the same
// way the loader does (dirname(dirname(entry))/src/tool-specs.ts).
function makeFakePackage(toolSpecsSource: string | null): {
// Build a throwaway `<pkg>/build/index.js` + optional `<pkg>/src/` tree so
// `computeSrcRegistryStamp(<pkg>/build/index.js)` resolves src the same way the
// loader does (dirname(dirname(entry))/src). Since #486 the stamp hashes the WHOLE
// src tree, so a fixture is a { relPath: content } map. A bare string is sugar for
// a single `tool-specs.ts`; `null` means "no src tree" (the prod no-op path).
function makeFakePackage(
src: string | Record<string, string> | null,
): {
entry: string;
cleanup: () => void;
} {
@@ -42,10 +54,15 @@ function makeFakePackage(toolSpecsSource: string | null): {
mkdirSync(buildDir, { recursive: true });
const entry = join(buildDir, 'index.js');
writeFileSync(entry, '// fake @docmost/mcp build entry\n', 'utf8');
if (toolSpecsSource !== null) {
if (src !== null) {
const files =
typeof src === 'string' ? { 'tool-specs.ts': src } : src;
const srcDir = join(root, 'src');
mkdirSync(srcDir, { recursive: true });
writeFileSync(join(srcDir, 'tool-specs.ts'), toolSpecsSource, 'utf8');
for (const [rel, content] of Object.entries(files)) {
const full = join(srcDir, rel);
mkdirSync(dirname(full), { recursive: true });
writeFileSync(full, content, 'utf8');
}
}
return { entry, cleanup: () => rmSync(root, { recursive: true, force: true }) };
}
@@ -93,34 +110,109 @@ describe('computeSrcRegistryStamp (#447 stale-build guard)', () => {
}
});
// CROSS-IMPL EQUALITY (covers reviewer suggestion 2). The SAME fixed input and
// #486 CORE (negative): an edit to a NON-tool-specs src file (client.ts) with a
// rebuild NOT run must move the src stamp away from the built REGISTRY_STAMP, so
// the loader's stale-check refuses. Under the old tool-specs.ts-only hash this
// edit was invisible and a stale build/ served the old client.ts silently.
it('a client.ts edit (no rebuild) moves the src stamp -> loader refuses (#486)', () => {
// "Built" state: the package as it was compiled.
const built = makeFakePackage({
'tool-specs.ts': 'export const SPECS = 1;\n',
'client.ts': "export const impl = 'v1';\n",
});
// "Dev edited src, forgot to rebuild": client.ts changed, tool-specs.ts not.
const edited = makeFakePackage({
'tool-specs.ts': 'export const SPECS = 1;\n',
'client.ts': "export const impl = 'v2';\n",
});
try {
const builtStamp = computeSrcRegistryStamp(built.entry);
const editedStamp = computeSrcRegistryStamp(edited.entry);
expect(builtStamp).not.toBeNull();
expect(editedStamp).not.toBe(builtStamp);
// build/ still carries builtStamp; src now hashes to editedStamp -> refuse.
expect(() => assertStaleGuard(editedStamp, builtStamp as string)).toThrow(
STALE_BUILD_MESSAGE,
);
} finally {
built.cleanup();
edited.cleanup();
}
});
// *.generated.ts is excluded (the codegen's own output — a fixed-point cycle
// otherwise): its presence/content must not move the stamp.
it('excludes *.generated.ts from the stamp', () => {
const without = makeFakePackage({ 'tool-specs.ts': 'x\n' });
const withGen = makeFakePackage({
'tool-specs.ts': 'x\n',
'registry-stamp.generated.ts': 'export const REGISTRY_STAMP = "abc";\n',
});
try {
expect(computeSrcRegistryStamp(withGen.entry)).toBe(
computeSrcRegistryStamp(without.entry),
);
} finally {
without.cleanup();
withGen.cleanup();
}
});
// CROSS-IMPL EQUALITY (covers reviewer suggestion 2). The SAME fixed tree and
// EXPECTED hash are asserted in the mcp-side node test
// (packages/mcp/test/unit/registry-stamp.test.mjs) against the codegen's
// `computeRegistryStamp`. Asserting the SAME pair here against the loader's
// `computeSrcRegistryStamp` proves both implementations normalize+hash
// `computeSrcRegistryStamp` proves both implementations enumerate+normalize+hash
// identically; a divergence in EITHER side reddens one of the two tests.
it('matches the documented cross-impl hash for a fixed input', () => {
const FIXED_INPUT = 'line1\r\nline2\n';
const EXPECTED =
'683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83';
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
const CROSS_IMPL_TREE = {
'tool-specs.ts': 'line1\r\nline2\n',
'client/read.ts': 'export const R = 1;\n',
'registry-stamp.generated.ts': 'export const REGISTRY_STAMP="ignored";\n',
};
const CROSS_IMPL_EXPECTED =
'131c1b9e4e2f5a7d6cef91ca8df619822b442f52bc45ebd09474a4c1d6728616';
it('matches the documented cross-impl hash for a fixed tree', () => {
const { entry, cleanup } = makeFakePackage(CROSS_IMPL_TREE);
try {
expect(computeSrcRegistryStamp(entry)).toBe(EXPECTED);
expect(computeSrcRegistryStamp(entry)).toBe(CROSS_IMPL_EXPECTED);
} finally {
cleanup();
}
});
it('the documented EXPECTED is the normalize+sha256 of the fixed input', () => {
// Proves EXPECTED is not a magic constant but the documented computation.
const FIXED_INPUT = 'line1\r\nline2\n';
const normalized = FIXED_INPUT.replace(/\r\n/g, '\n').replace(/\n$/, '');
const expected = createHash('sha256')
.update(normalized, 'utf8')
.digest('hex');
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
it('the documented EXPECTED is the enumerate+normalize+sha256 of the tree', () => {
// Proves EXPECTED is not a magic constant but the documented computation — a
// local re-implementation of the loader's tree walk.
const { entry, cleanup } = makeFakePackage(CROSS_IMPL_TREE);
try {
expect(computeSrcRegistryStamp(entry)).toBe(expected);
const srcDir = join(dirname(dirname(entry)), 'src');
const collect = (dir: string): string[] => {
const out: string[] = [];
for (const e of readdirSync(dir)) {
const f = join(dir, e);
if (statSync(f).isDirectory()) out.push(...collect(f));
else if (e.endsWith('.ts') && !e.endsWith('.generated.ts'))
out.push(f);
}
return out;
};
const files = collect(srcDir)
.map((abs) => ({ rel: relative(srcDir, abs).split(sep).join('/'), abs }))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const h = createHash('sha256');
for (const { rel, abs } of files) {
const n = readFileSync(abs, 'utf8')
.replace(/\r\n/g, '\n')
.replace(/\n$/, '');
h.update(rel, 'utf8');
h.update('\0', 'utf8');
h.update(n, 'utf8');
h.update('\0', 'utf8');
}
const localHash = h.digest('hex');
expect(computeSrcRegistryStamp(entry)).toBe(localHash);
expect(localHash).toBe(CROSS_IMPL_EXPECTED);
} finally {
cleanup();
}
@@ -1,6 +1,6 @@
import { createHash } from 'node:crypto';
import { existsSync, readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs';
import { dirname, join, relative, sep } from 'node:path';
import { pathToFileURL } from 'node:url';
import type { DocmostClient, SharedToolSpec } from '@docmost/mcp';
@@ -191,33 +191,52 @@ interface DocmostMcpModule {
* present. Returns the stamp string, or `null` when the source is absent (a prod
* image ships only build/, no src/). MUST stay byte-for-byte identical to
* packages/mcp/scripts/gen-registry-stamp.mjs's `computeRegistryStamp` so the
* build-time and src-time hashes agree: same input file (src/tool-specs.ts), same
* normalization (CRLF -> LF, strip a single trailing newline), same sha256.
* build-time and src-time hashes agree: same file set (every src/**\/*.ts except
* *.generated.ts), same POSIX-relative sort, same per-file normalization (CRLF ->
* LF, strip a single trailing newline) with the same path+content framing, same
* sha256. Hashing the WHOLE src tree (not just tool-specs.ts) is #486: an edit to
* client.ts / a client/* module / comment-signal / drawio-* without a rebuild
* must also be caught, otherwise build/ silently serves the old code.
*
* DEV vs PROD detection is by FILE EXISTENCE, not NODE_ENV: we resolve the
* package's own directory from `require.resolve('@docmost/mcp')` (which points at
* build/index.js) and look for ../src/tool-specs.ts next to it. In a dev/test
* worktree that file exists; in a prod image (build/ only, src/ stripped) it does
* not, so this returns null and the caller skips the check. Any error (ENOENT, a
* bad resolve) is swallowed to null the stale-check must NEVER break startup.
* build/index.js) and look for ../src next to it. In a dev/test worktree that
* directory exists; in a prod image (build/ only, src/ stripped) it does not, so
* this returns null and the caller skips the check. Any error (ENOENT, a bad
* resolve) is swallowed to null the stale-check must NEVER break startup.
*
* Exported for unit testing (docmost-client.loader.spec.ts): the export keyword
* is behaviourally a no-op the module-internal caller `loadDocmostMcp` is
* unaffected. The test drives the null (no-src) path and asserts this
* normalize+sha256 stays identical to the codegen's `computeRegistryStamp`.
* enumerate+normalize+sha256 stays identical to the codegen's
* `computeRegistryStamp`.
*/
export function computeSrcRegistryStamp(packageEntry: string): string | null {
try {
// packageEntry is <pkg>/build/index.js; the source lives at <pkg>/src/.
const toolSpecsPath = join(
dirname(dirname(packageEntry)),
'src',
'tool-specs.ts',
);
if (!existsSync(toolSpecsPath)) return null; // prod: no src tree -> skip.
const source = readFileSync(toolSpecsPath, 'utf8');
const normalized = source.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
const srcDir = join(dirname(dirname(packageEntry)), 'src');
if (!existsSync(srcDir)) return null; // prod: no src tree -> skip.
// Enumerate every src/**\/*.ts except the codegen's own *.generated.ts
// output (including it would be a fixed-point cycle). Sort by POSIX-relative
// path so ordering is platform-independent, then fold each file's relative
// path + normalized content into one hash — identical to the codegen.
const files = collectStampFiles(srcDir)
.map((abs) => ({
rel: relative(srcDir, abs).split(sep).join('/'),
abs,
}))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const hash = createHash('sha256');
for (const { rel, abs } of files) {
const normalized = readFileSync(abs, 'utf8')
.replace(/\r\n/g, '\n')
.replace(/\n$/, '');
hash.update(rel, 'utf8');
hash.update('\0', 'utf8');
hash.update(normalized, 'utf8');
hash.update('\0', 'utf8');
}
return hash.digest('hex');
} catch {
// Never let a resolution/read hiccup break server startup — treat as "no
// src available" and skip the check (identical to the prod no-op path).
@@ -225,6 +244,24 @@ export function computeSrcRegistryStamp(packageEntry: string): string | null {
}
}
/**
* Recursively enumerate every `*.ts` under `dir`, EXCLUDING `*.generated.ts`.
* Mirror of the codegen's `collectStampFiles` (packages/mcp/scripts/
* gen-registry-stamp.mjs) keep the two walk/filter rules identical.
*/
function collectStampFiles(dir: string): string[] {
const out: string[] = [];
for (const entry of readdirSync(dir)) {
const full = join(dir, entry);
if (statSync(full).isDirectory()) {
out.push(...collectStampFiles(full));
} else if (entry.endsWith('.ts') && !entry.endsWith('.generated.ts')) {
out.push(full);
}
}
return out;
}
// TS with module:commonjs downlevels a literal `import()` to `require()`, which
// cannot load the ESM-only `@docmost/mcp` package. Indirect through Function so
// the real dynamic `import()` survives compilation and can load ESM from
@@ -7,6 +7,22 @@ import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { jsonToMarkdown } from '../../../collaboration/collaboration.util';
import { modelFriendlyInput } from './model-friendly-input';
/**
* A tool error whose message is DELIBERATELY safe to expose to an anonymous
* share reader (and to the model, for self-correction). Every OTHER thrown error
* is treated as internal and replaced with a generic string by `wrapToolErrors`,
* so a raw exception message an internal page title, a DB/stack fragment, a
* driver detail never rides the public UI stream (#394).
*/
export class ShareToolError extends Error {}
// The only two classified strings an anonymous reader may ever see from a tool
// failure. The specific one keeps the model's self-correction useful ("try a
// different page"); the generic one reveals nothing about the internal fault.
const SHARE_TOOL_ERROR_NOT_AVAILABLE =
'The requested page is not available in this share.';
const SHARE_TOOL_ERROR_GENERIC = 'The tool could not complete the request.';
/**
* Isolated, READ-ONLY toolset for the ANONYMOUS public-share assistant.
*
@@ -44,7 +60,7 @@ export class PublicShareChatToolsService {
* are NO write tools, NO comments/history, NO cross-space or external tools.
*/
forShare(shareId: string, workspaceId: string): Record<string, Tool> {
return {
return this.wrapToolErrors({
searchSharePages: tool({
description:
'Search the pages of THIS published documentation share for a ' +
@@ -96,7 +112,7 @@ export class PublicShareChatToolsService {
execute: async ({ pageId }) => {
const id = (pageId ?? '').trim();
if (!id) {
throw new Error('A pageId is required.');
throw new ShareToolError('A pageId is required.');
}
// Resolve via the SINGLE canonical share-access boundary: confirms the
// page resolves to THIS share (recursive CTE up the tree, honouring
@@ -112,7 +128,7 @@ export class PublicShareChatToolsService {
workspaceId,
);
if (!resolved) {
throw new Error('That page is not part of this published share.');
throw new ShareToolError(SHARE_TOOL_ERROR_NOT_AVAILABLE);
}
const { page } = resolved;
@@ -193,6 +209,57 @@ export class PublicShareChatToolsService {
}
},
}),
};
});
}
/**
* Wrap every tool's `execute` so a THROWN error is sanitized in ONE place
* closing the byte leak, the render, and the model context at once (#394).
*
* The AI SDK surfaces a tool-execution throw as an atomic `tool-output-error`
* frame on the v6 UI stream whose `errorText` is the thrown message; on the
* public share that frame goes straight to an anonymous reader. Unwrapped, a
* raw exception (an internal page title, a DB/stack fragment, a driver detail)
* would ride that frame verbatim. Here we catch it, LOG the full detail
* server-side only, and re-throw a CLASSIFIED, safe error: the tool's own
* intentional ShareToolError messages pass through (they keep the model's
* self-correction useful), everything else collapses to a generic string.
*/
private wrapToolErrors(
tools: Record<string, Tool>,
): Record<string, Tool> {
const wrapped: Record<string, Tool> = {};
for (const [name, t] of Object.entries(tools)) {
const original = t.execute;
if (typeof original !== 'function') {
wrapped[name] = t;
continue;
}
wrapped[name] = {
...t,
execute: async (args: unknown, options: unknown) => {
try {
return await (
original as (a: unknown, o: unknown) => Promise<unknown>
)(args, options);
} catch (err) {
const safe =
err instanceof ShareToolError
? err.message
: SHARE_TOOL_ERROR_GENERIC;
// Full detail to the server log ONLY — never to the anon.
this.logger.warn(
`Public share tool "${name}" failed: ${
err instanceof Error ? err.message : String(err)
}`,
);
// This safe string is ALL that rides the tool-output-error frame,
// becomes model context, and could be rendered — one choke point.
throw new ShareToolError(safe);
}
},
} as Tool;
}
return wrapped;
}
}
@@ -120,3 +120,102 @@ describe('JwtStrategy — provenance derivation', () => {
expect(req.raw.actor).toBeUndefined();
});
});
/**
* Provenance derivation on the API-KEY path (jwt.strategy.validateApiKey, #486).
*
* The access-token path stamped provenance; the API-key path returned early
* WITHOUT it, so an is_agent API key's REST writes recorded no 'agent' marker.
* The API-key payload carries no signed claim, so provenance is resolved from the
* SERVER-SIDE user returned by ApiKeyService.validateApiKey: isAgent -> 'agent',
* otherwise 'user'; aiChatId is always null (an API key has no ai_chats row).
*
* The enterprise ApiKeyService is not bundled in the OSS build, so the strategy
* loads it through an overridable `resolveApiKeyService` seam that we stub here.
*/
describe('JwtStrategy — API-key provenance derivation (#486)', () => {
function makeApiKeyStrategy(validateApiKeyImpl: (p: any) => Promise<any>) {
const userRepo: any = { findById: jest.fn() };
const workspaceRepo: any = { findById: jest.fn() };
const userSessionRepo: any = { findActiveById: jest.fn() };
const sessionActivityService: any = { trackActivity: jest.fn() };
const environmentService: any = { getAppSecret: () => 'test-secret' };
const moduleRef: any = {};
const strategy = new JwtStrategy(
userRepo,
workspaceRepo,
userSessionRepo,
sessionActivityService,
environmentService,
moduleRef,
);
// Stub the EE ApiKeyService seam (the real module is not in the OSS build).
const validateApiKey = jest.fn(validateApiKeyImpl);
jest
.spyOn(strategy as any, 'resolveApiKeyService')
.mockReturnValue({ validateApiKey });
return { strategy, validateApiKey };
}
const makeReq = () => ({ raw: {} as Record<string, any> });
const apiKeyPayload = () => ({
sub: 'svc-1',
workspaceId: 'ws-1',
apiKeyId: 'key-1',
type: JwtType.API_KEY,
});
it("stamps actor='agent' for an is_agent API key (from the validated user)", async () => {
const validated = {
user: { id: 'svc-1', isAgent: true },
workspace: { id: 'ws-1' },
};
const { strategy, validateApiKey } = makeApiKeyStrategy(
async () => validated,
);
const req = makeReq();
const result = await strategy.validate(req, apiKeyPayload() as any);
expect(validateApiKey).toHaveBeenCalledTimes(1);
expect(req.raw.actor).toBe('agent');
// API keys carry no internal ai_chats row -> null.
expect(req.raw.aiChatId).toBeNull();
// The validated auth object is returned unchanged (req.user shape preserved).
expect(result).toBe(validated);
});
it("stamps actor='user' for an ordinary (non-agent) API key", async () => {
const { strategy } = makeApiKeyStrategy(async () => ({
user: { id: 'u-1', isAgent: false },
workspace: { id: 'ws-1' },
}));
const req = makeReq();
await strategy.validate(req, apiKeyPayload() as any);
expect(req.raw.actor).toBe('user');
expect(req.raw.aiChatId).toBeNull();
});
it('throws Unauthorized (and stamps nothing) when the EE module is missing', async () => {
const userRepo: any = { findById: jest.fn() };
const strategy = new JwtStrategy(
userRepo,
{ findById: jest.fn() } as any,
{ findActiveById: jest.fn() } as any,
{ trackActivity: jest.fn() } as any,
{ getAppSecret: () => 'test-secret' } as any,
{} as any,
);
// EE not bundled: the seam returns null.
jest.spyOn(strategy as any, 'resolveApiKeyService').mockReturnValue(null);
const req = makeReq();
await expect(
strategy.validate(req, apiKeyPayload() as any),
).rejects.toThrow(UnauthorizedException);
expect(req.raw.actor).toBeUndefined();
});
});
@@ -102,28 +102,49 @@ export class JwtStrategy extends PassportStrategy(Strategy, 'jwt') {
}
private async validateApiKey(req: any, payload: JwtApiKeyPayload) {
let ApiKeyModule: any;
let isApiKeyModuleReady = false;
const apiKeyService = this.resolveApiKeyService();
if (!apiKeyService) {
throw new UnauthorizedException('Enterprise API Key module missing');
}
const result = await apiKeyService.validateApiKey(payload);
// Stamp the agent-edit provenance for the API-KEY path too (#486). Unlike the
// access-token path above, it CANNOT be resolved before this point: the
// API-key payload carries no signed actor/aiChatId claim, and the user (with
// its isAgent flag) is unknown until the key is validated. Claim semantics for
// API keys: an is_agent API key (an agent service account) stamps 'agent' on
// every REST write; an ordinary API key resolves to 'user'. An API key has no
// internal ai_chats row, so aiChatId is always null. Derived from the
// SERVER-SIDE user (never a client field), so an 'agent' badge is unspoofable
// — mirroring the access-token path. Passing `null` for the claim means the
// actor is decided solely by user.isAgent.
const provenance = resolveProvenance((result as any)?.user, null);
req.raw.actor = provenance.actor;
req.raw.aiChatId = provenance.aiChatId;
return result;
}
/**
* Resolve the enterprise ApiKeyService, or `null` when the EE module is not
* bundled in this build (community build). Extracted as an overridable seam so
* the API-key provenance stamping can be unit-tested without the EE package
* present (docmost is OSS + a separate EE bundle; `require` of the EE path
* throws here). Any load/resolve error is treated as "module missing".
*/
protected resolveApiKeyService(): {
validateApiKey: (payload: JwtApiKeyPayload) => Promise<unknown>;
} | null {
try {
// eslint-disable-next-line @typescript-eslint/no-require-imports
ApiKeyModule = require('./../../../ee/api-key/api-key.service');
isApiKeyModuleReady = true;
const ApiKeyModule = require('./../../../ee/api-key/api-key.service');
return this.moduleRef.get(ApiKeyModule.ApiKeyService, { strict: false });
} catch (err) {
this.logger.debug(
'API Key module requested but enterprise module not bundled in this build',
);
isApiKeyModuleReady = false;
return null;
}
if (isApiKeyModuleReady) {
const ApiKeyService = this.moduleRef.get(ApiKeyModule.ApiKeyService, {
strict: false,
});
return ApiKeyService.validateApiKey(payload);
}
throw new UnauthorizedException('Enterprise API Key module missing');
}
}
@@ -1,5 +1,6 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { sql } from 'kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import {
@@ -188,6 +189,144 @@ export class AiChatMessageRepo {
return query.returning(this.baseFields).executeTakeFirst();
}
/**
* #487 OWNER terminal write the streamText terminal callback's finalize. Like
* `update` but CONDITIONAL on `status='streaming' OR metadata.finalizeFailed`:
* the owner writes its real content EITHER when the row is still streaming (the
* normal case) OR when a reconcile stamp already flipped it to a terminal status
* but marked `finalizeFailed:true` the owner's real content OVERWRITES that
* placeholder stamp (owner-write priority, #487). A row that is properly terminal
* (no finalizeFailed) is left untouched (undefined) idempotent. The `patch`
* carries the real metadata WITHOUT finalizeFailed, so a successful write CLEARS
* the flag. Returns the updated row, or undefined when nothing matched.
*/
async finalizeOwner(
id: string,
workspaceId: string,
patch: Partial<{
content: string | null;
toolCalls: unknown;
metadata: unknown;
status: string | null;
}>,
trx?: KyselyTransaction,
): Promise<AiChatMessage | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatMessages')
.set({ ...(patch as Record<string, unknown>), updatedAt: new Date() })
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where((eb) =>
eb.or([
eb('status', '=', 'streaming'),
eb(sql<string>`(metadata->>'finalizeFailed')`, '=', 'true'),
]),
)
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* #487 RECONCILE status-only stamp settle a stuck 'streaming' row to a
* terminal status WITHOUT the owner's real content (which lived only in the
* dead process's memory — a documented loss). CONDITIONAL on `status='streaming'`
* (never touches an already-terminal row) AND it MERGES `finalizeFailed:true`
* into metadata (preserving the partial `parts` already persisted) so a LATER
* owner-write (finalizeOwner) can still OVERWRITE this placeholder with real
* content, and so `isInterruptResume` can EXCLUDE this row (a reconcile stamp is
* not a genuine user interruption). Returns the updated row, or undefined.
*/
async stampTerminalIfStreaming(
id: string,
workspaceId: string,
status: 'aborted' | 'error' | 'completed',
trx?: KyselyTransaction,
): Promise<AiChatMessage | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatMessages')
.set({
status,
metadata: sql`coalesce(metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('status', '=', 'streaming')
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* #487 reconcile clause (b): streaming assistant rows whose linked RUN has
* already reached a terminal status an asymmetry ("run settled / message
* streaming forever") the periodic reconcile heals by stamping the message.
* Returns the message id + its run's terminal status, bounded.
*/
async findStreamingWithTerminalRun(
limit = 200,
// #487: scope to ONE chat for the opportunistic per-turn reconcile (removes
// reconcile latency from the user-visible path); omit for the periodic sweep.
chat?: { chatId: string; workspaceId: string },
): Promise<
Array<{ messageId: string; workspaceId: string; runStatus: string }>
> {
let query = this.db
.selectFrom('aiChatMessages as m')
.innerJoin('aiChatRuns as r', 'r.assistantMessageId', 'm.id')
.select([
'm.id as messageId',
'm.workspaceId as workspaceId',
'r.status as runStatus',
])
.where('m.status', '=', 'streaming')
.where('r.status', 'in', ['succeeded', 'failed', 'aborted']);
if (chat) {
query = query
.where('m.chatId', '=', chat.chatId)
.where('m.workspaceId', '=', chat.workspaceId);
}
return query.limit(limit).execute();
}
/**
* #487 reconcile clause (d) historical-row safety: streaming rows older than
* `staleMs` whose chat has NO active run row (double-gated). Settle them to
* 'aborted' + finalizeFailed (so a late owner-write could still overwrite).
* Returns the count. Used ONLY by the periodic reconcile, never at boot.
*/
async sweepStreamingWithoutActiveRun(
staleMs: number,
trx?: KyselyTransaction,
): Promise<number> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - staleMs);
const rows = await db
.updateTable('aiChatMessages as m')
.set({
status: 'aborted',
metadata: sql`coalesce(m.metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('m.status', '=', 'streaming')
.where('m.updatedAt', '<', staleBefore)
.where((eb) =>
eb.not(
eb.exists(
eb
.selectFrom('aiChatRuns as r')
.select('r.id')
.whereRef('r.chatId', '=', 'm.chatId')
.where('r.status', 'in', ['pending', 'running']),
),
),
)
.returning('m.id')
.execute();
return rows.length;
}
/**
* Crash-recovery sweep (#183): flip every assistant row still left in the
* 'streaming' state (a turn that died mid-write before reaching a terminal
@@ -200,13 +339,20 @@ export class AiChatMessageRepo {
* step, so an actively-streaming row never matches; this prevents a fresh
* replica's boot-sweep from aborting a turn another replica is still streaming
* in a multi-instance deploy.
*
* #487: the sweep now ALSO marks `finalizeFailed:true` so a late owner-write can
* overwrite this placeholder with real content (owner-write priority).
*/
async sweepStreaming(trx?: KyselyTransaction): Promise<number> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - SWEEP_STREAMING_STALE_MS);
const rows = await db
.updateTable('aiChatMessages')
.set({ status: 'aborted', updatedAt: new Date() })
.set({
status: 'aborted',
metadata: sql`coalesce(metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('status', '=', 'streaming')
.where('updatedAt', '<', staleBefore)
.returning('id')
@@ -143,6 +143,41 @@ export class AiChatRunRepo {
.executeTakeFirst();
}
/**
* #487: CONDITIONAL terminal finalize flip a run to a terminal status and
* stamp `finished_at` ONLY while it is still active (pending|running), mirroring
* the assistant message's `onlyIfStreaming` guard. A double-settle (a late or
* second writer, a supersede applying a zombie's intended, a reconcile stamp)
* matches NOTHING once the row is terminal and is a benign no-op so a terminal
* status can never be clobbered by a later writer (last-writer-wins is gone).
*
* Returns the updated row when it WAS active (this call wrote it), else
* undefined (the row was already terminal another writer won). The caller
* distinguishes the two to resolve the correct settle outcome.
*/
async finalizeIfActive(
id: string,
workspaceId: string,
patch: { status: string; error: string | null },
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
const now = new Date();
return db
.updateTable('aiChatRuns')
.set({
status: patch.status,
error: patch.error,
finishedAt: now,
updatedAt: now,
})
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* Mark an EXPLICIT stop request on an active run (distinct from a browser
* disconnect, which never stops a run). Stamps `stop_requested_at` ONLY while
@@ -184,6 +219,31 @@ export class AiChatRunRepo {
* sweeps only runs UNTOUCHED past the window. Phase 1 is single-process, so the
* boot path supplies no window.
*/
/**
* #487 reconcile clause (c): active (pending|running) runs UNTOUCHED past
* `staleMs` candidates for "no live runner" abort. Staleness is measured from
* `updated_at` (the LAST-PROGRESS timestamp recordStep bumps it), NOT
* `started_at`, so a legitimate long-running marathon (1125 min of steady
* progress) is never a candidate. The caller filters these against its in-memory
* `active` / zombie maps ("no entry" is the PRIMARY gate a live entry is never
* aborted) before settling any of them. Bounded.
*/
async findStaleActive(
staleMs: number,
limit = 200,
trx?: KyselyTransaction,
): Promise<Array<{ id: string; workspaceId: string; chatId: string }>> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - staleMs);
return db
.selectFrom('aiChatRuns')
.select(['id', 'workspaceId', 'chatId'])
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.where('updatedAt', '<', staleBefore)
.limit(limit)
.execute();
}
async sweepRunning(
opts: { staleMs?: number } = {},
trx?: KyselyTransaction,
@@ -0,0 +1,133 @@
import { readFileSync } from 'fs';
import { EventEmitter } from 'node:events';
import { streamText } from 'ai';
import { MockLanguageModelV3, simulateReadableStream } from 'ai/test';
/**
* Regression tests for the writeToServerResponse drain-hang fix in
* patches/ai@6.0.134.patch (#486, commit 6).
*
* Unpatched ai@6.0.134's writeToServerResponse awaits ONLY `once("drain")` when
* response.write() returns false (backpressure). If the client disconnects
* mid-write the socket never drains, so that await never resolves: the read loop
* parks FOREVER, its `finally { response.end() }` is unreachable, and the stream
* reader + buffered chunks are pinned until process restart. In autonomous mode
* the run keeps producing output after the disconnect, so EVERY mid-run
* disconnect leaks a hung pipe. The patch races drain against close/error, and on
* a terminal socket event cancels the reader and breaks so `finally` always runs.
*
* This drives the REAL patched writeToServerResponse through the public
* pipeUIMessageStreamToResponse API with a response that never drains and closes
* mid-write exactly the leak scenario.
*/
/** A ServerResponse-like emitter whose first write() stalls (returns false) and
* then "closes" like a disconnecting client never firing 'drain'. */
class DisconnectingResponse extends EventEmitter {
ended = false;
writeCount = 0;
statusCode = 200;
writableEnded = false;
destroyed = false;
writeHead(): this {
return this;
}
setHeader(): void {}
flushHeaders(): void {}
write(): boolean {
this.writeCount++;
if (this.writeCount === 1) {
// Simulate the client vanishing mid-write: backpressure (false) and then a
// 'close' on the next tick, and CRUCIALLY never a 'drain'. Unpatched, the
// loop would await drain forever here.
setImmediate(() => this.emit('close'));
return false;
}
return true;
}
end(): void {
this.ended = true;
this.writableEnded = true;
this.emit('finish');
}
}
function makeModel() {
return new MockLanguageModelV3({
doStream: async () => ({
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'text-start' as const, id: '1' },
{ type: 'text-delta' as const, id: '1', delta: 'hello ' },
{ type: 'text-delta' as const, id: '1', delta: 'world' },
{ type: 'text-end' as const, id: '1' },
{
type: 'finish' as const,
finishReason: { unified: 'stop' as const, raw: 'stop' },
usage: {
inputTokens: { total: 1, noCache: undefined, cacheRead: undefined, cacheWrite: undefined },
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
}),
});
}
describe('ai@6.0.134 pnpm patch: writeToServerResponse drain-hang (#486)', () => {
it('ends the response (does NOT hang) when the socket closes mid-write without draining', async () => {
const result = streamText({ model: makeModel(), prompt: 'hi' });
const res = new DisconnectingResponse();
// Drain the SDK stream independently, like the production detached path.
void result.consumeStream({ onError: () => undefined });
result.pipeUIMessageStreamToResponse(res as never);
// TRIPWIRE: the patched loop exits on 'close' and runs finally -> end().
// Unpatched, it awaits 'drain' forever and this never becomes true.
await new Promise<void>((resolve, reject) => {
const started = Date.now();
const poll = setInterval(() => {
if (res.ended) {
clearInterval(poll);
resolve();
} else if (Date.now() - started > 3000) {
clearInterval(poll);
reject(new Error('writeToServerResponse hung: response never ended'));
}
}, 20);
});
expect(res.ended).toBe(true);
});
it('does not emit an unhandledRejection when the fire-and-forget read() throws', async () => {
// The patch swallows read()'s rejection (fire-and-forget) with a log instead
// of letting it surface as a process-killing unhandledRejection.
const rejections: unknown[] = [];
const onUnhandled = (e: unknown) => rejections.push(e);
process.on('unhandledRejection', onUnhandled);
// Silence the patch's diagnostic console.error for the throwing read().
const errSpy = jest.spyOn(console, 'error').mockImplementation(() => undefined);
try {
const result = streamText({ model: makeModel(), prompt: 'hi' });
const res = new DisconnectingResponse();
void result.consumeStream({ onError: () => undefined });
result.pipeUIMessageStreamToResponse(res as never);
await new Promise((r) => setTimeout(r, 300));
} finally {
process.off('unhandledRejection', onUnhandled);
errSpy.mockRestore();
}
expect(rejections).toEqual([]);
});
it('both installed dist builds (CJS and ESM) carry the #486 patch marker', () => {
const cjsPath = require.resolve('ai');
const mjsPath = cjsPath.replace(/index\.js$/, 'index.mjs');
expect(cjsPath).toMatch(/index\.js$/);
expect(readFileSync(cjsPath, 'utf8')).toContain('PATCH(docmost #486)');
expect(readFileSync(mjsPath, 'utf8')).toContain('PATCH(docmost #486)');
});
});
@@ -0,0 +1,95 @@
import type { HealthIndicatorService } from '@nestjs/terminus';
import type { EnvironmentService } from '../environment/environment.service';
/**
* Integration guard for the /health Redis-probe handle leak (#486, commit 2).
*
* The bug: `pingCheck` built `new Redis(...)` per call and only disconnected on
* the SUCCESS path, so when Redis is DOWN every probe tick added ANOTHER
* forever-reconnecting client an unbounded handle/client leak for the duration
* of the outage. The fix reuses ONE long-lived probe client.
*
* This is an OBSERVABLE-property test, not an assertion on a mocked return value:
* we point the indicator at a REAL, refused TCP endpoint (a dead port) so ioredis
* genuinely fails to connect, run many probes, and assert the number of live
* Redis CLIENTS created stays at exactly ONE. `ioredis` is delegated to its real
* implementation (requireActual) only the constructor is wrapped to COUNT the
* real clients it creates, which is precisely the leaking resource.
*/
const mockLiveClients: Array<{ status: string; disconnect: () => void }> = [];
jest.mock('ioredis', () => {
const actual = jest.requireActual('ioredis');
const RealRedis = actual.Redis ?? actual.default ?? actual;
class CountingRedis extends RealRedis {
constructor(...args: unknown[]) {
super(...(args as []));
mockLiveClients.push(this as never);
}
}
return { ...actual, Redis: CountingRedis, default: CountingRedis };
});
// Import AFTER the mock is registered so the class picks up the counting client.
import { RedisHealthIndicator } from './redis.health';
describe('RedisHealthIndicator handle leak (#486)', () => {
const indicatorService = {
check: (key: string) => ({
up: () => ({ [key]: { status: 'up' } }),
down: (message: string) => ({ [key]: { status: 'down', message } }),
}),
} as unknown as HealthIndicatorService;
// A port with (almost certainly) nothing listening -> connection refused fast.
const environmentService = {
getRedisUrl: () => 'redis://127.0.0.1:6399/0',
} as unknown as EnvironmentService;
let indicator: RedisHealthIndicator;
beforeEach(() => {
mockLiveClients.length = 0;
indicator = new RedisHealthIndicator(indicatorService, environmentService);
});
afterEach(() => {
indicator.onModuleDestroy();
// Belt-and-braces: tear down anything the test created so ioredis reconnect
// timers do not keep the jest worker alive.
for (const c of mockLiveClients) {
try {
c.disconnect();
} catch {
/* already gone */
}
}
});
it('creates exactly ONE Redis client across many probes while Redis is DOWN', async () => {
const N = 8;
for (let i = 0; i < N; i++) {
const result = await indicator.pingCheck('redis');
// Down endpoint -> every probe reports "down" (not an unhandled crash).
expect(result.redis.status).toBe('down');
}
// THE OBSERVABLE LEAK: on the buggy code this is N (a fresh, never-cleaned
// reconnecting client per probe). The fix reuses one shared client.
expect(mockLiveClients).toHaveLength(1);
});
it('onModuleDestroy releases the probe client (a later probe builds a fresh one)', async () => {
await indicator.pingCheck('redis');
expect(mockLiveClients).toHaveLength(1);
indicator.onModuleDestroy();
// A second destroy is a safe no-op (probeClient was nulled).
indicator.onModuleDestroy();
// After shutdown the indicator lazily builds a NEW client on the next probe,
// proving the old one was truly released rather than reused.
await indicator.pingCheck('redis');
expect(mockLiveClients).toHaveLength(2);
});
});
@@ -2,33 +2,78 @@ import {
HealthIndicatorResult,
HealthIndicatorService,
} from '@nestjs/terminus';
import { Injectable, Logger } from '@nestjs/common';
import { Injectable, Logger, OnModuleDestroy } from '@nestjs/common';
import { EnvironmentService } from '../environment/environment.service';
import { Redis } from 'ioredis';
@Injectable()
export class RedisHealthIndicator {
export class RedisHealthIndicator implements OnModuleDestroy {
private readonly logger = new Logger(RedisHealthIndicator.name);
/**
* ONE long-lived probe connection, reused across every /health tick. The old
* code built `new Redis(...)` per call and only `disconnect()`d on the SUCCESS
* path, so while Redis was DOWN every probe added a fresh, forever-reconnecting
* client a handle leak that grew without bound for as long as the outage (and
* the health checker keeps polling) lasted. A single shared client keeps at most
* ONE background reconnect loop regardless of how many probes run.
*/
private probeClient: Redis | null = null;
constructor(
private readonly healthIndicatorService: HealthIndicatorService,
private environmentService: EnvironmentService,
) {}
private getProbeClient(): Redis {
if (!this.probeClient) {
this.probeClient = new Redis(this.environmentService.getRedisUrl(), {
// Constructing must never throw or eagerly connect; the first ping opens
// the socket. This lets us build the client once and reuse it.
lazyConnect: true,
// A health probe must fail FAST, not queue behind a stuck reconnect: one
// retry per request, and no offline queue so a ping while disconnected
// rejects immediately instead of buffering commands that pile up in RAM.
maxRetriesPerRequest: 1,
enableOfflineQueue: false,
});
// ioredis emits 'error' on every failed (re)connect; with no listener that
// surfaces as an unhandled 'error' event and can crash the process. Swallow
// it here — pingCheck already reports health — and log at debug so a Redis
// outage does not flood the logs.
this.probeClient.on('error', (err) => {
this.logger.debug(
`Redis probe connection error: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
}
return this.probeClient;
}
async pingCheck(key: string): Promise<HealthIndicatorResult> {
const indicator = this.healthIndicatorService.check(key);
try {
const redis = new Redis(this.environmentService.getRedisUrl(), {
maxRetriesPerRequest: 15,
});
const redis = this.getProbeClient();
await redis.ping();
redis.disconnect();
return indicator.up();
} catch (e) {
this.logger.error(e);
return indicator.down(`${key} is not available`);
}
}
onModuleDestroy(): void {
if (this.probeClient) {
// disconnect() (not quit()) tears the socket + reconnect loop down
// immediately without waiting on a round-trip to a possibly-down server.
// Do NOT removeAllListeners() with no event name — that would also strip
// ioredis' OWN internal listeners and break its teardown; our 'error'
// listener is harmless and dies with the dropped client reference.
this.probeClient.disconnect();
this.probeClient = null;
}
}
}
@@ -238,9 +238,8 @@ function convertReferenceFootnotes(markdown: string): string {
*
* LINE-ANCHORED (the same shape the canonical parser uses in
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
* very start and closes only on a `\n---` line. The retired editor-ext
* `markdownToHtml` front-matter strip (removed in #347) closed on the FIRST
* `---` ANYWHERE (an unanchored close), so a value
* very start and closes only on a `\n---` line. The retired `markdownToHtml`
* strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
* and leaked the rest into the body. An optional leading BOM is tolerated.
*/
@@ -16,6 +16,7 @@ import {
} from './mcp-auth.helpers';
import { JwtType } from '../../core/auth/dto/jwt-payload';
import { CREDENTIALS_MISMATCH_MESSAGE } from '../../core/auth/auth.constants';
import { McpService } from './mcp.service';
// The /mcp per-user auth decision logic is tested through the framework-free
// `resolveMcpSessionConfig` helper that McpService delegates to. McpService
@@ -1179,3 +1180,46 @@ describe('mapAuthResultToResponse (handle status/body mapping, refactor R2)', ()
});
});
});
// #486: onModuleDestroy must ALSO tear down the live loopback CollabSessions, not
// just clear the sweep timer — otherwise the embedded MCP's collab sockets keep
// docs pinned open on the collab server past process exit. The teardown goes
// through an overridable seam (destroyAllMcpSessions) so it can be spied without
// loading the ESM-only @docmost/mcp package.
describe('McpService.onModuleDestroy — CollabSession teardown (#486)', () => {
function makeService(): McpService {
// The constructor only stores its deps and starts the (unref'd) sweep timer,
// so bare stubs suffice. onModuleDestroy clears that timer, so no leak.
return new McpService(
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
);
}
it('destroys all sessions AND clears the sweep timer on shutdown', async () => {
const svc = makeService();
const destroy = jest.fn().mockResolvedValue(undefined);
(svc as any).destroyAllMcpSessions = destroy;
const clearSpy = jest.spyOn(global, 'clearInterval');
await svc.onModuleDestroy();
expect(destroy).toHaveBeenCalledTimes(1);
expect(clearSpy).toHaveBeenCalledWith((svc as any).sweepTimer);
clearSpy.mockRestore();
});
it('swallows a teardown failure so shutdown never throws', async () => {
const svc = makeService();
(svc as any).destroyAllMcpSessions = jest
.fn()
.mockRejectedValue(new Error('collab teardown boom'));
await expect(svc.onModuleDestroy()).resolves.toBeUndefined();
});
});
@@ -117,10 +117,42 @@ export class McpService implements OnModuleDestroy {
this.sweepTimer.unref?.();
}
onModuleDestroy(): void {
async onModuleDestroy(): Promise<void> {
clearInterval(this.sweepTimer);
// Tear down any live loopback CollabSession providers at shutdown (#486). The
// embedded MCP (and the in-app AI agent) open Hocuspocus collab sockets against
// THIS process; without an explicit teardown those sessions keep their docs
// "open" on the collab server and hold providers/buffers until they idle out,
// so a restart can race a doc still pinned by the dying worker. Best-effort:
// any failure is logged, never allowed to break shutdown.
try {
await this.destroyAllMcpSessions();
} catch (err) {
this.logger.error(
'MCP CollabSession teardown on shutdown failed',
err as Error,
);
}
}
/**
* Resolve @docmost/mcp's `destroyAllSessions` and invoke it (#486). The live
* CollabSession registry is a module-level singleton in the ESM package, shared
* by every entry (`.`/`./http`), so this tears down ALL sessions regardless of
* which surface opened them. The module is already loaded whenever MCP was used;
* if it was never loaded (or is absent) the import + no-op is harmless.
*
* Held as an overridable field so a unit test can spy the teardown without
* loading the ESM-only package or standing up the DI graph.
*/
private destroyAllMcpSessions: () => Promise<void> = async () => {
const entry = require.resolve('@docmost/mcp');
const mod = (await esmImport(pathToFileURL(entry).href)) as {
destroyAllSessions?: () => void;
};
mod.destroyAllSessions?.();
};
// Service account the embedded MCP uses to talk back to this Docmost
// instance over loopback REST + the collaboration WebSocket. Now OPTIONAL:
// it is only a fallback when no per-user Basic/Bearer credentials are sent.
@@ -0,0 +1,148 @@
import { get as httpGet } from 'node:http';
import { AddressInfo } from 'node:net';
import { createServer } from 'node:http';
// Drive the metrics HTTP server without the load-time METRICS_PORT gate: mock the
// registry so isMetricsEnabled()/getMetricsRegistry() are always satisfied. What
// we assert is observed over a REAL socket (bind address, status codes), not on
// the mock.
jest.mock('./metrics.registry', () => ({
isMetricsEnabled: () => true,
getMetricsRegistry: () => ({
metrics: async () => '# HELP up test\nup 1\n',
contentType: 'text/plain; version=0.0.4',
}),
}));
import {
startMetricsServer,
closeMetricsServer,
resolveMetricsBind,
resolveMetricsToken,
} from './metrics.server';
/** Find a free TCP port (the metrics server requires METRICS_PORT > 0). */
function freePort(): Promise<number> {
return new Promise((resolve, reject) => {
const s = createServer();
s.once('error', reject);
s.listen(0, '127.0.0.1', () => {
const p = (s.address() as AddressInfo).port;
s.close(() => resolve(p));
});
});
}
/** Minimal GET against 127.0.0.1:port with optional Authorization header. */
function req(
port: number,
headers: Record<string, string> = {},
): Promise<{ status: number; body: string }> {
return new Promise((resolve, reject) => {
const r = httpGet(
{ host: '127.0.0.1', port, path: '/metrics', headers },
(res) => {
let body = '';
res.on('data', (c) => (body += c));
res.on('end', () =>
resolve({ status: res.statusCode ?? 0, body }),
);
},
);
r.on('error', reject);
});
}
describe('metrics server bind + auth (#486)', () => {
const saved = {
bind: process.env.METRICS_BIND,
token: process.env.METRICS_TOKEN,
port: process.env.METRICS_PORT,
};
afterEach(async () => {
await closeMetricsServer();
process.env.METRICS_BIND = saved.bind;
process.env.METRICS_TOKEN = saved.token;
process.env.METRICS_PORT = saved.port;
delete process.env.METRICS_BIND;
delete process.env.METRICS_TOKEN;
});
describe('resolveMetricsBind', () => {
it('defaults to loopback 127.0.0.1', () => {
delete process.env.METRICS_BIND;
expect(resolveMetricsBind()).toBe('127.0.0.1');
});
it('honours the METRICS_BIND override', () => {
process.env.METRICS_BIND = '0.0.0.0';
expect(resolveMetricsBind()).toBe('0.0.0.0');
});
it('treats a blank override as unset (loopback)', () => {
process.env.METRICS_BIND = ' ';
expect(resolveMetricsBind()).toBe('127.0.0.1');
});
});
describe('resolveMetricsToken', () => {
it('is null when unset', () => {
delete process.env.METRICS_TOKEN;
expect(resolveMetricsToken()).toBeNull();
});
it('returns the trimmed token when set', () => {
process.env.METRICS_TOKEN = ' s3cret ';
expect(resolveMetricsToken()).toBe('s3cret');
});
});
it('binds to loopback by default and serves /metrics without auth when no token', async () => {
delete process.env.METRICS_BIND;
delete process.env.METRICS_TOKEN;
const port = await freePort();
process.env.METRICS_PORT = String(port);
const server = startMetricsServer();
expect(server).not.toBeNull();
await new Promise<void>((resolve) => {
if (server!.listening) resolve();
else server!.once('listening', () => resolve());
});
// OBSERVABLE: the listener bound to loopback, not 0.0.0.0.
expect((server!.address() as AddressInfo).address).toBe('127.0.0.1');
const res = await req(port);
expect(res.status).toBe(200);
expect(res.body).toContain('up 1');
});
it('rejects unauthenticated scrapes with 401 and accepts the exact Bearer token', async () => {
delete process.env.METRICS_BIND;
process.env.METRICS_TOKEN = 'topsecret';
const port = await freePort();
process.env.METRICS_PORT = String(port);
const server = startMetricsServer();
expect(server).not.toBeNull();
// No auth -> 401.
const noAuth = await req(port);
expect(noAuth.status).toBe(401);
// Wrong token, DIFFERENT length -> 401 (short-circuits on the length guard).
const wrong = await req(port, { authorization: 'Bearer nope' });
expect(wrong.status).toBe(401);
// Wrong token, SAME length -> 401. This drives the timingSafeEqual compare
// itself (the length guard passes: 'Bearer topsecreX' has the same length as
// 'Bearer topsecret'). Pins the constant-time compare: a regression that made
// it return true would let this equal-length wrong token through — the
// different-length case above would NOT catch that.
const sameLen = await req(port, { authorization: 'Bearer topsecreX' });
expect(sameLen.status).toBe(401);
// Correct token -> 200 with the metrics body.
const ok = await req(port, { authorization: 'Bearer topsecret' });
expect(ok.status).toBe(200);
expect(ok.body).toContain('up 1');
});
});
@@ -1,7 +1,27 @@
import { createServer, Server } from 'node:http';
import { timingSafeEqual } from 'node:crypto';
import { Logger } from '@nestjs/common';
import { getMetricsRegistry, isMetricsEnabled } from './metrics.registry';
/**
* Constant-time compare of the presented Authorization header against the
* expected `Bearer <token>`. This is the ONLY auth layer for the metrics
* endpoint, so a naive `!==` would leak the token byte-by-byte via timing.
* timingSafeEqual requires equal-length buffers, so a length mismatch short-
* circuits to "not equal" (its own length is not itself a useful oracle: the
* expected string length is fixed by config, not secret-derived).
*/
function bearerMatches(
presented: string | undefined,
expected: string,
): boolean {
if (typeof presented !== 'string') return false;
const a = Buffer.from(presented);
const b = Buffer.from(expected);
if (a.length !== b.length) return false;
return timingSafeEqual(a, b);
}
/**
* Start the Prometheus scrape endpoint on a SEPARATE port, taken from
* `METRICS_PORT`. There is NO default port: when `METRICS_PORT` is unset the
@@ -16,6 +36,30 @@ import { getMetricsRegistry, isMetricsEnabled } from './metrics.registry';
*/
let metricsServer: Server | null = null;
/**
* Interface the metrics endpoint binds to. Defaults to LOOPBACK (127.0.0.1) so
* the unauthenticated `/metrics` surface is NOT exposed on all interfaces by
* default the old `0.0.0.0` bind put an auth-less endpoint on every interface.
* Deployments where the scraper runs in a SEPARATE container (and reaches this as
* `docmost:9464`) set `METRICS_BIND=0.0.0.0`, ideally together with METRICS_TOKEN
* and/or a private network so the port is not world-readable.
*/
export function resolveMetricsBind(): string {
const raw = (process.env.METRICS_BIND ?? '').trim();
return raw.length > 0 ? raw : '127.0.0.1';
}
/**
* Optional Bearer token guarding `/metrics`. When `METRICS_TOKEN` is set, every
* scrape must present `Authorization: Bearer <token>`; unset (default) leaves the
* endpoint open (safe when bound to loopback / a trusted network). Returns the
* trimmed token or null when unset/blank.
*/
export function resolveMetricsToken(): string | null {
const raw = (process.env.METRICS_TOKEN ?? '').trim();
return raw.length > 0 ? raw : null;
}
export function startMetricsServer(): Server | null {
if (!isMetricsEnabled()) return null;
@@ -31,8 +75,22 @@ export function startMetricsServer(): Server | null {
return null;
}
const bind = resolveMetricsBind();
const token = resolveMetricsToken();
const server = createServer(async (req, res) => {
if (req.method === 'GET' && req.url === '/metrics') {
// Optional Bearer auth: reject scrapes without the exact token when one is
// configured. This is the auth layer the old all-interfaces bind lacked.
if (token) {
const auth = req.headers['authorization'];
if (!bearerMatches(auth, `Bearer ${token}`)) {
res.statusCode = 401;
res.setHeader('WWW-Authenticate', 'Bearer');
res.end();
return;
}
}
try {
const body = await register.metrics();
res.setHeader('Content-Type', register.contentType);
@@ -48,10 +106,14 @@ export function startMetricsServer(): Server | null {
res.end();
});
// Bind on all interfaces: the scraper (VictoriaMetrics) reaches this from
// another container as docmost:9464. The port is not published to the host.
server.listen(port, '0.0.0.0', () => {
logger.log(`Metrics endpoint listening on :${port}/metrics`);
// Bind to loopback by default so the auth-less endpoint is not exposed on all
// interfaces. Set METRICS_BIND=0.0.0.0 (ideally with METRICS_TOKEN) when the
// scraper runs in a separate container and reaches this as docmost:9464.
server.listen(port, bind, () => {
logger.log(
`Metrics endpoint listening on ${bind}:${port}/metrics` +
(token ? ' (Bearer auth required)' : ''),
);
});
server.on('error', (err) => {
@@ -31,9 +31,6 @@ export enum QueueJob {
IMPORT_TASK = 'import-task',
EXPORT_TASK = 'export-task',
SEARCH_REMOVE_PAGE = 'search-remove-page',
SEARCH_REMOVE_ASSET = 'search-remove-attachment',
SEARCH_REMOVE_FACE = 'search-remove-comment',
TYPESENSE_FLUSH = 'typesense-flush',
PAGE_CREATED = 'page-created',
@@ -0,0 +1,305 @@
import { Kysely } from 'kysely';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatRunService } from '../../src/core/ai-chat/ai-chat-run.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #487 commit 4 bidirectional reconcile + owner-write priority, real SQL.
*
* Proves the OBSERVABLE recovery properties against docmost_test:
* - the CONDITIONAL owner-write beats a reconcile stamp, and a stamp never
* clobbers a proper terminal row;
* - a LATE owner-finalize with real content OVERWRITES a reconcile 'aborted'
* stamp (finalizeFailed);
* - each reconcile clause (b message<-run, c stale-run, d historical row) settles
* the stuck row/run, and a LIVE run entry is never touched;
* - the "kill DB on finish" recovery: after the DB comes back, neither the
* message row nor the run row stays stuck.
*/
describe('#487 reconcile + owner-write priority [integration]', () => {
let db: Kysely<any>;
let messageRepo: AiChatMessageRepo;
let runRepo: AiChatRunRepo;
let runService: AiChatRunService;
let workspaceId: string;
let userId: string;
beforeAll(async () => {
db = getTestDb();
messageRepo = new AiChatMessageRepo(db as any);
runRepo = new AiChatRunRepo(db as any);
runService = new AiChatRunService(runRepo, { isCloud: () => false } as never);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
const newChat = async () =>
(await createChat(db, { workspaceId, creatorId: userId })).id;
const metaOf = async (id: string): Promise<Record<string, unknown> | null> => {
const row = await messageRepo.findById(id, workspaceId);
return (row?.metadata as Record<string, unknown> | null) ?? null;
};
it('owner finalizeOwner writes a streaming row and CLEARS finalizeFailed', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
const wrote = await messageRepo.finalizeOwner(m.id, workspaceId, {
content: 'final answer',
status: 'completed',
metadata: { parts: [{ type: 'text', text: 'final answer' }] },
} as never);
expect(wrote!.status).toBe('completed');
expect((await metaOf(m.id))?.finalizeFailed).toBeUndefined();
});
it('a reconcile stamp NEVER clobbers a proper terminal row (finalizeOwner is a no-op there)', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'completed',
content: 'real',
metadata: { parts: [] },
});
// The reconcile stamp is onlyIfStreaming -> no-op on a completed row.
const stamped = await messageRepo.stampTerminalIfStreaming(
m.id,
workspaceId,
'aborted',
);
expect(stamped).toBeUndefined();
expect((await messageRepo.findById(m.id, workspaceId))!.status).toBe(
'completed',
);
});
it('LATE owner-finalize with real content OVERWRITES a reconcile aborted stamp', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [{ type: 'text', text: 'partial' }] },
});
// Reconcile stamps it aborted + finalizeFailed (final text lived only in mem).
const stamped = await messageRepo.stampTerminalIfStreaming(
m.id,
workspaceId,
'aborted',
);
expect(stamped!.status).toBe('aborted');
expect((await metaOf(m.id))?.finalizeFailed).toBe(true);
// A LATE owner-write (finalizeFailed=true satisfies the OR) overwrites it with
// real content, clearing the flag — owner-write priority.
const wrote = await messageRepo.finalizeOwner(m.id, workspaceId, {
content: 'the real final answer',
status: 'completed',
metadata: { parts: [{ type: 'text', text: 'the real final answer' }] },
} as never);
expect(wrote!.status).toBe('completed');
expect(wrote!.content).toBe('the real final answer');
expect((await metaOf(m.id))?.finalizeFailed).toBeUndefined();
});
it('clause (c): a stale active run with NO live entry -> aborted; a LIVE entry is untouched', async () => {
// Stale run, NOT owned by this replica (no entry) -> reconcile aborts it.
const staleChat = await newChat();
const stale = await runRepo.insert({
chatId: staleChat,
workspaceId,
createdBy: userId,
status: 'running',
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', stale.id)
.execute();
// A live run OWNED by this replica (beginRun registers an in-memory entry),
// ALSO backdated stale — the "no entry" primary gate must protect it.
const liveChat = await newChat();
const live = await runService.beginRun({
chatId: liveChat,
workspaceId,
userId,
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', live.runId)
.execute();
const aborted = await runService.reconcileStaleRuns(15 * 60 * 1000);
expect(aborted).toBeGreaterThanOrEqual(1);
expect((await runRepo.findById(stale.id, workspaceId))!.status).toBe(
'aborted',
);
// The live entry is NEVER aborted, however stale its row looks.
expect((await runRepo.findById(live.runId, workspaceId))!.status).toBe(
'running',
);
expect(runService.isLocallyActive(live.runId)).toBe(true);
// cleanup the live run
await runService.finalizeRun(live.runId, workspaceId, 'aborted');
});
it('clause (b): a streaming message whose RUN is terminal is stamped by run status (succeeded -> aborted, NOT completed-empty)', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
// A SUCCEEDED run linked to the still-streaming message (the asymmetry).
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
assistantMessageId: msg.id,
});
await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'succeeded',
error: null,
});
const stuck = await messageRepo.findStreamingWithTerminalRun();
const mine = stuck.find((s) => s.messageId === msg.id);
expect(mine?.runStatus).toBe('succeeded');
// Reconcile clause (b): succeeded run -> message 'aborted' (NOT 'completed'),
// the final text lived only in memory (documented loss), +finalizeFailed.
const status = mine!.runStatus === 'failed' ? 'error' : 'aborted';
await messageRepo.stampTerminalIfStreaming(msg.id, workspaceId, status);
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
it('clause (d): a stale streaming row with NO active run on the chat -> aborted+finalizeFailed', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
const swept = await messageRepo.sweepStreamingWithoutActiveRun(
15 * 60 * 1000,
);
expect(swept).toBeGreaterThanOrEqual(1);
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
it('clause (d) is DOUBLE-GATED: a stale streaming row WITH an active run on the chat is left alone', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
// An ACTIVE run on the same chat -> clause (d) must NOT touch the message.
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
});
await messageRepo.sweepStreamingWithoutActiveRun(15 * 60 * 1000);
expect((await messageRepo.findById(msg.id, workspaceId))!.status).toBe(
'streaming',
);
await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'aborted',
error: null,
});
});
it('"kill DB on finish" recovery: after the DB is back, reconcile leaves NEITHER the row nor the run stuck', async () => {
// Simulate a process that seeded the assistant row + run, then died before
// finalizing EITHER (a mid-turn crash): a streaming message + a running run,
// both stale, with no in-memory entry (fresh service = fresh maps).
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [{ type: 'text', text: 'partial' }] },
});
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
assistantMessageId: msg.id,
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', run.id)
.execute();
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
// Reconcile (as the periodic job would): (c) aborts the orphan run, then
// (b) settles the message from the now-terminal run.
await runService.reconcileStaleRuns(15 * 60 * 1000);
const stuck = await messageRepo.findStreamingWithTerminalRun();
for (const s of stuck) {
const status = s.runStatus === 'failed' ? 'error' : 'aborted';
await messageRepo.stampTerminalIfStreaming(s.messageId, s.workspaceId, status);
}
// Neither is stuck: the run is terminal AND the message is terminal.
expect((await runRepo.findById(run.id, workspaceId))!.status).toBe('aborted');
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
});
@@ -281,6 +281,52 @@ describe('AiChatRun durable lifecycle [integration]', () => {
});
});
it('#487 finalizeIfActive is CONDITIONAL: a late terminal write cannot clobber the settled status (real SQL)', async () => {
const c = (await createChat(db, { workspaceId, creatorId: userId })).id;
const run = await runRepo.insert({
chatId: c,
workspaceId,
createdBy: userId,
status: 'running',
});
// First terminal write: the run IS active, so it flips + returns the row.
const first = await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'succeeded',
error: null,
});
expect(first!.status).toBe('succeeded');
expect(first!.finishedAt).toBeTruthy();
// A late/second writer tries to flip it to 'aborted' — the WHERE status IN
// ('pending','running') guard matches NOTHING now, so it is a benign no-op.
const second = await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'aborted',
error: 'late clobber attempt',
});
expect(second).toBeUndefined();
// The persisted terminal status is UNCHANGED — last-writer-wins is gone.
const row = await runRepo.findById(run.id, workspaceId);
expect(row!.status).toBe('succeeded');
expect(row!.error).toBeNull();
});
it('#487 double-settle through the service collapses to one write at the SQL gate', async () => {
const c = (await createChat(db, { workspaceId, creatorId: userId })).id;
const handle = await service.beginRun({ chatId: c, workspaceId, userId });
// First settle writes 'aborted' via the conditional write.
await service.finalizeRun(handle.runId, workspaceId, 'aborted');
// A late safety-net settle to 'error' is a no-op (row already terminal).
await service.finalizeRun(handle.runId, workspaceId, 'error', 'late');
const row = await runRepo.findById(handle.runId, workspaceId);
expect(row!.status).toBe('aborted');
expect(service.isLocallyActive(handle.runId)).toBe(false);
expect(service.hasZombie(handle.runId)).toBe(false);
});
it('sweepRunning() with NO args (boot sweep / variant C) aborts even a FRESH running run', async () => {
// F1/DECISION C at the SQL level: the unconditional boot sweep has NO
// staleness window, so a run updated just now (a fast restart) is settled too
+3
View File
@@ -11,6 +11,9 @@
"main": "dist/index.js",
"module": "./src/index.ts",
"types": "dist/index.d.ts",
"dependencies": {
"marked": "17.0.5"
},
"devDependencies": {
"@vitest/coverage-v8": "4.1.6",
"vitest": "4.1.6"
+1
View File
@@ -18,6 +18,7 @@ export * from "./lib/excalidraw";
export * from "./lib/embed";
export * from "./lib/html-embed/html-embed";
export * from "./lib/mention";
export * from "./lib/markdown";
export * from "./lib/search-and-replace";
export * from "./lib/embed-provider";
export * from "./lib/subpages";
@@ -14,8 +14,7 @@ import {
* ProseMirror JSON directly (never running the editor's plugins), so the
* canonical footnote topology was never enforced on those writes. The consumers
* of this editor-ext copy are: the server markdown/HTML import
* (`markdownToProseMirror` from @docmost/prosemirror-markdown in import.service /
* file-import-task.service),
* (`markdownToHtml -> htmlToJson` in import.service / file-import-task.service),
* `PageService` create/update (`parseProsemirrorContent` for the JSON/markdown/
* HTML REST write paths), and the client markdown PASTE path
* (`markdown-clipboard.ts`). (The MCP package mirrors this canonicalizer in
@@ -0,0 +1,131 @@
import { describe, it, expect } from "vitest";
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
import { markdownToHtml } from "../markdown/utils/marked.utils";
import { extractFootnoteDefinitions } from "../markdown/utils/footnote.marked";
// HTML the editor-ext nodes render (sup[data-footnote-ref], section/div).
const HTML =
`<p>Water<sup data-footnote-ref data-id="fn1"></sup> and clay<sup data-footnote-ref data-id="fn2"></sup>.</p>` +
`<section data-footnotes>` +
`<div data-footnote-def data-id="fn1"><p>First note.</p></div>` +
`<div data-footnote-def data-id="fn2"><p>Second note.</p></div>` +
`</section>`;
describe("footnote markdown round-trip", () => {
it("HTML -> Markdown produces pandoc footnote syntax", () => {
const md = htmlToMarkdown(HTML);
expect(md).toContain("[^fn1]");
expect(md).toContain("[^fn2]");
expect(md).toContain("[^fn1]: First note.");
expect(md).toContain("[^fn2]: Second note.");
});
it("Markdown -> HTML rebuilds the footnote nodes' HTML", async () => {
const md = htmlToMarkdown(HTML);
const html = await markdownToHtml(md);
expect(html).toContain('data-footnote-ref data-id="fn1"');
expect(html).toContain('data-footnote-ref data-id="fn2"');
expect(html).toContain("data-footnotes");
expect(html).toContain('data-footnote-def data-id="fn1"');
expect(html).toContain("First note.");
expect(html).toContain("Second note.");
});
it("preserves a [^id]: line shown inside a fenced code block (not a definition)", async () => {
// A document that DOCUMENTS footnote syntax inside a code fence. The
// `[^demo]: ...` line is example text, not a real definition, and must
// survive the Markdown -> HTML conversion verbatim.
const md = [
"Here is how footnotes look:",
"",
"```markdown",
"Some text[^demo]",
"",
"[^demo]: this is the definition",
"```",
"",
"End of doc.",
].join("\n");
const html = await markdownToHtml(md);
// The example definition line is kept inside the rendered code block.
expect(html).toContain("[^demo]: this is the definition");
// It did NOT get pulled out into a real footnotes section.
expect(html).not.toContain("data-footnotes");
expect(html).not.toContain("data-footnote-def");
});
it("extractFootnoteDefinitions keeps the FIRST duplicate definition and reuses markers", () => {
// Two definitions share id `d`, and the body has two `[^d]` markers. Under
// the import model (#166) duplicate definition ids are FIRST-WINS: only the
// first definition is kept; markers are NEVER rewritten, so the two `[^d]`
// references reuse the single footnote.
const md = [
"See here[^d] and there[^d].",
"",
"[^d]: first",
"[^d]: second",
].join("\n");
const { body, section } = extractFootnoteDefinitions(md);
const defIds = Array.from(
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
expect(defIds).toEqual(["d"]); // first-wins: one definition
expect(section).toContain("first");
expect(section).not.toContain("second"); // duplicate dropped
// Both markers stay `[^d]` (reuse) — no `d__2` minting.
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
(m) => m[1],
);
expect(refIds).toEqual(["d", "d"]);
});
it("extractFootnoteDefinitions is DETERMINISTIC and stable (same input -> same output)", () => {
// The output must be a pure function of the input markdown so importing the
// same source twice (or via the editor and the MCP mirror) is identical.
const md = [
"See[^d] one[^d] two[^d].",
"",
"[^d]: first",
"[^d]: second",
"[^d]: third",
].join("\n");
const run = () => {
const { body, section } = extractFootnoteDefinitions(md);
const defIds = Array.from(
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
(m) => m[1],
);
return { defIds, refIds };
};
const a = run();
const b = run();
expect(a).toEqual(b);
// First-wins: one kept definition `d`; all three reuse markers stay `d`.
expect(a.defIds).toEqual(["d"]);
expect(a.refIds).toEqual(["d", "d", "d"]);
});
it("markdownToHtml with a reused id renders ONE shared footnote def", async () => {
const md = [
"See here[^d] and there[^d].",
"",
"[^d]: first",
"[^d]: second",
].join("\n");
const html = await markdownToHtml(md);
const defIds = Array.from(
html.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
expect(defIds).toEqual(["d"]); // one shared definition
expect(html).toContain("first");
expect(html).not.toContain("second");
});
});
@@ -103,9 +103,8 @@ interface CollisionPlan {
* `X__2`, `X__3`, collision-bumped) so it survives as a distinct footnote which,
* having no matching reference, then falls under the normal orphan policy. It is
* only ever dropped for lacking a reference, never for colliding. The IMPORT
* paths (@docmost/prosemirror-markdown / MCP extractFootnotes) instead apply
* first-wins + drop + warn for duplicate definitions; that divergence is
* intentional import
* paths (footnote.marked.ts / MCP extractFootnotes) instead apply first-wins +
* drop + warn for duplicate definitions; that divergence is intentional import
* is an agent-authored artifact we sanitize, the editor is live user data we must
* not lose.
*
@@ -6,9 +6,8 @@ import { deriveFootnoteId } from "./footnote-util";
*
* `deriveFootnoteId` lives ONLY in editor-ext now it is used by
* `resolveCollisions` (re-id of a duplicate definition) and `footnotePastePlugin`
* (re-id of a pasted colliding definition). The MCP / @docmost/prosemirror-markdown
* import paths no longer derive ids (duplicate definitions there are
* first-wins-dropped, #166), so there
* (re-id of a pasted colliding definition). The MCP/marked import paths no longer
* derive ids (duplicate definitions there are first-wins-dropped, #166), so there
* is no cross-package copy and no parity test to keep in sync. This table pins the
* deterministic scheme so a future change to it is a conscious one.
*/
@@ -63,9 +63,8 @@ export function generateFootnoteId(): string {
* its own seen-set before requesting the next derived id.
*
* Used only inside editor-ext now (resolveCollisions for a re-id'd duplicate
* DEFINITION, and footnotePastePlugin). The MCP / @docmost/prosemirror-markdown
* import paths no longer derive ids duplicate definitions there are
* first-wins-dropped (#166) so
* DEFINITION, and footnotePastePlugin). The MCP/marked import paths no longer
* derive ids duplicate definitions there are first-wins-dropped (#166) so
* there is no cross-package copy to keep in sync. The golden table in
* footnote-util.derive-id.test.ts pins the scheme.
*/
@@ -0,0 +1,68 @@
import { describe, it, expect } from "vitest";
import { generateJSON } from "@tiptap/html";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
import { markdownToHtml } from "../markdown/utils/marked.utils";
import { TiptapImage } from "./image";
// Minimal schema for parsing markdownToHtml output back to JSON (mirrors
// image.spec.ts), so we can assert the recovered caption EXACTLY.
const parseExtensions = [Document, Paragraph, Text, TiptapImage];
// Lossless markdown round-trip for image captions (issue #221). An image WITH a
// caption can't be expressed as `![alt](src)`, so it is emitted as a raw <img>
// (carrying data-caption) wrapped in a block <div>, the same trick the <video>
// rule uses. marked passes the raw HTML through, so markdownToHtml keeps the
// data-caption, and the image extension's parseHTML restores the attribute.
describe("image caption markdown round-trip", () => {
it("HTML -> Markdown emits a raw <img data-caption> for captioned images", () => {
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
const md = htmlToMarkdown(html);
expect(md).toContain("data-caption=\"A grey cat\"");
expect(md).toContain('src="/files/a.png"');
expect(md).toContain('alt="cat"');
// It must NOT degrade to the lossy ![]() form.
expect(md).not.toContain("![cat]");
});
it("Markdown -> HTML restores data-caption on the <img>", async () => {
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
const md = htmlToMarkdown(html);
const back = await markdownToHtml(md);
expect(back).toContain('data-caption="A grey cat"');
expect(back).toContain('src="/files/a.png"');
});
it("special characters in the caption survive the round-trip (escaped)", async () => {
// The source caption is the decoded string `Tom & "Jerry"` (both an `&` and
// a `"`). escapeHtmlAttr must encode `&` -> `&amp;` and `"` -> `&quot;`.
const html = `<p><img src="/files/a.png" data-caption='Tom &amp; &quot;Jerry&quot;'></p>`;
const md = htmlToMarkdown(html);
// (a) The intermediate Markdown must carry the EXACT escaped attribute. This
// fails if escapeHtmlAttr stopped escaping `"` (attribute break-out:
// data-caption="Tom & "Jerry"") or double-encoded `&` (`&amp;amp;`).
expect(md).toContain('data-caption="Tom &amp; &quot;Jerry&quot;"');
const back = await markdownToHtml(md);
expect(back).toContain("data-caption=");
expect(back).toContain("Jerry");
expect(back).toContain("Tom");
// (b) Re-parse the rendered HTML through the image extension's parseHTML and
// assert the recovered caption is EXACTLY the original (no corruption, loss,
// or double-encoding).
const json = generateJSON(back, parseExtensions);
expect(json.content?.[0]?.attrs?.caption).toBe('Tom & "Jerry"');
});
it("caption-less images stay a clean ![alt](src) with no raw HTML", () => {
const html = `<p><img src="/files/a.png" alt="cat"></p>`;
const md = htmlToMarkdown(html);
expect(md).toContain("![cat](/files/a.png)");
expect(md).not.toContain("data-caption");
expect(md).not.toContain("<img");
});
});
@@ -0,0 +1,105 @@
import { describe, expect, it } from "vitest";
import { htmlEmbedExtension } from "./utils/html-embed.marked";
import { markdownToHtml } from "./index";
import { encodeHtmlEmbedSource } from "../html-embed/html-embed";
// CONTRACT tests for the marked block tokenizer that rebuilds an htmlEmbed node
// from the `<!--html-embed:BASE64-->` marker (html-embed.marked.ts), plus the
// observable round-trip through markdownToHtml.
//
// These pin the REAL tokenizer behaviour the import path depends on:
// - the tokenizer rule is anchored (^) and only accepts the base64 alphabet
// [A-Za-z0-9+/=], so a marker with non-base64 chars is NOT tokenized and
// survives as a literal HTML comment (not silently turned into something the
// server's strip no longer recognizes);
// - start() reports the correct index of the next marker so marked invokes the
// tokenizer at the right offset when a marker sits mid-document / after text;
// - a marker with surrounding text on the SAME line is split out into its own
// embed div while the surrounding text becomes ordinary paragraphs.
//
// The contract is asserted against the actual exported extension and pipeline —
// no behaviour is invented; the expectations were read off the real tokenizer.
const SAMPLE = "<b>x</b>";
const ENC = encodeHtmlEmbedSource(SAMPLE);
describe("htmlEmbed marked tokenizer — start()", () => {
it("returns the index of a marker that sits mid-document", () => {
const src = `hello world <!--html-embed:${ENC}-->`;
expect(htmlEmbedExtension.start(src)).toBe(src.indexOf("<!--html-embed:"));
});
it("returns 0 when the marker is at the very start", () => {
expect(htmlEmbedExtension.start(`<!--html-embed:${ENC}-->`)).toBe(0);
});
it("returns -1 when there is no marker", () => {
expect(htmlEmbedExtension.start("no marker here")).toBe(-1);
});
});
describe("htmlEmbed marked tokenizer — tokenizer()", () => {
it("tokenizes a marker at the start of the input, capturing the base64 payload", () => {
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`);
expect(token).toBeTruthy();
expect(token!.type).toBe("htmlEmbed");
expect(token!.raw).toBe(`<!--html-embed:${ENC}-->`);
expect(token!.encoded).toBe(ENC);
});
it("tokenizes an EMPTY marker (the [A-Za-z0-9+/=]* class allows zero chars)", () => {
const token = htmlEmbedExtension.tokenizer("<!--html-embed:-->");
expect(token).toBeTruthy();
expect(token!.encoded).toBe("");
expect(token!.raw).toBe("<!--html-embed:-->");
});
it("does NOT tokenize when text precedes the marker (rule is anchored ^)", () => {
// marked relies on start() to advance to the marker; the tokenizer itself
// only matches at offset 0, so a non-anchored call returns undefined.
expect(
htmlEmbedExtension.tokenizer(`hello <!--html-embed:${ENC}-->`),
).toBeUndefined();
});
it("does NOT tokenize a marker containing a non-base64 char ('$')", () => {
expect(
htmlEmbedExtension.tokenizer("<!--html-embed:ab$cd-->"),
).toBeUndefined();
});
it("does NOT tokenize a marker containing a space", () => {
expect(
htmlEmbedExtension.tokenizer("<!--html-embed:ab cd-->"),
).toBeUndefined();
});
it("renderer emits the embed div the node's parseHTML recognizes", () => {
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`)!;
const html = htmlEmbedExtension.renderer(token as any);
expect(html).toBe(
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
);
});
});
describe("htmlEmbed marked tokenizer — markdownToHtml round-trip", () => {
it("splits a marker out of surrounding same-line text into its own embed div", async () => {
const html = await markdownToHtml(`before <!--html-embed:${ENC}--> after`);
// The marker became the embed div...
expect(html).toContain(
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
);
// ...and the surrounding text survived as ordinary paragraph content.
expect(html).toContain("before");
expect(html).toContain("after");
});
it("leaves a marker with non-base64 chars as a literal comment (NOT an embed div)", async () => {
const html = await markdownToHtml("<!--html-embed:ab$cd-->");
// It is NOT tokenized into an embed div the server would strip...
expect(html).not.toContain('data-type="htmlEmbed"');
// ...it passes through unchanged as a literal HTML comment.
expect(html).toContain("<!--html-embed:ab$cd-->");
});
});
@@ -0,0 +1,2 @@
export * from "./utils/marked.utils";
export * from "./utils/turndown.utils";
@@ -0,0 +1,112 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml, htmlToMarkdown } from "./index";
import {
encodeHtmlEmbedSource,
decodeHtmlEmbedSource,
} from "../html-embed/html-embed";
// SECURITY (Variant C admin gate, import attack surface).
//
// The markdown import path is the only write path where an htmlEmbed reaches
// the server purely from file bytes (no editor / collab socket). The marked
// tokenizer in `html-embed.marked.ts` and the turndown rule in
// `turndown.utils.ts` are what materialize the `<!--html-embed:BASE64-->`
// marker into the `<div data-type="htmlEmbed" data-source="BASE64">` element
// that the server then parses into an htmlEmbed node and the admin gate strips.
//
// If either the tokenizer regex or the turndown rule shape drifts, the marker
// would either (a) stop becoming an htmlEmbed node (silently dropping admin
// content) or (b) become some OTHER tag the server's `hasHtmlEmbedNode` no
// longer recognizes (a strip bypass). These tests pin the marker <-> embed-div
// contract that the server-side strip relies on. editor-ext had ZERO tests
// before this file; this adds the runner + the round-trip coverage.
// The server parses the embed div by matching `data-type="htmlEmbed"` and
// decoding `data-source`; mirror that here so the assertion is exactly what the
// real `htmlToJson` -> htmlEmbed node parse depends on (the node's parseHTML in
// html-embed.ts uses the same selector + decodeHtmlEmbedSource).
const EMBED_DIV_RE = /<div[^>]*\bdata-type="htmlEmbed"[^>]*>/;
function extractEmbedSource(html: string): string | undefined {
const div = EMBED_DIV_RE.exec(html);
if (!div) return undefined;
const enc = /data-source="([^"]*)"/.exec(div[0]);
if (!enc) return undefined;
return decodeHtmlEmbedSource(enc[1]);
}
// Replicates the server's `hasHtmlEmbedNode` decision against the embed *div*
// (the HTML form the server immediately converts to JSON). If this matches, the
// server's JSON-level `hasHtmlEmbedNode` will too, because htmlToJson maps this
// exact div to an htmlEmbed node.
function htmlHasHtmlEmbed(html: string): boolean {
return EMBED_DIV_RE.test(html);
}
describe("markdown <!--html-embed--> import round-trip", () => {
const source = "<script>x</script>";
it("markdownToHtml turns the marker into an htmlEmbed div carrying the source", async () => {
const md = "<!--html-embed:" + encodeHtmlEmbedSource(source) + "-->";
const html = await markdownToHtml(md);
// The marker became the embed div the server recognizes as an htmlEmbed
// node (so the server's hasHtmlEmbedNode would match it after htmlToJson).
expect(htmlHasHtmlEmbed(html)).toBe(true);
// The decoded source is the original script, intact.
expect(extractEmbedSource(html)).toBe(source);
// The raw script is NOT inlined into the HTML — it stays base64 in the
// attribute (the marker itself must not be a direct injection vector).
expect(html).not.toContain("<script>x</script>");
});
it("preserves UTF-8 / special chars in the embedded source", async () => {
const utf8 = '<script>console.log("héllo → 世界")</script>';
const md = "<!--html-embed:" + encodeHtmlEmbedSource(utf8) + "-->";
const html = await markdownToHtml(md);
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe(utf8);
});
it("an empty marker still produces an htmlEmbed div (empty source)", async () => {
const html = await markdownToHtml("<!--html-embed:-->");
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe("");
});
it("round-trips htmlToMarkdown -> markdownToHtml preserving the embed marker", async () => {
const encoded = encodeHtmlEmbedSource(source);
// NOTE: turndown drops a *blank* (childless) element before any custom rule
// runs, and the htmlEmbed div is normally childless. The export pipeline
// therefore must give the rule a non-blank div to fire on; we add an inert
// text child here to exercise the real turndown htmlEmbed rule. (A blank
// embed div serializing to "" is asserted separately below as a documented
// edge so this contract drift is visible.)
const startHtml = `<div data-type="htmlEmbed" data-source="${encoded}">x</div>`;
// Export to markdown: the turndown rule emits the <!--html-embed:..-->
// marker (lossless, inert in plain markdown viewers).
const md = htmlToMarkdown(startHtml);
expect(md).toContain("<!--html-embed:" + encoded + "-->");
// Re-import: the marker round-trips back into an embed div with the same
// decoded source — this is the marker <-> embed-div contract the server's
// import strip depends on.
const html = await markdownToHtml(md);
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe(source);
});
it("documents that a BLANK embed div serializes to empty markdown (turndown drops childless blocks)", () => {
const encoded = encodeHtmlEmbedSource(source);
const blank = `<div data-type="htmlEmbed" data-source="${encoded}"></div>`;
// This pins current behavior so a future change to the turndown rule (e.g.
// making it fire on blank nodes) is caught rather than silently shipping.
expect(htmlToMarkdown(blank)).toBe("");
});
it("the base64 codec itself round-trips (no '<' leaks into the attribute)", () => {
const encoded = encodeHtmlEmbedSource(source);
expect(encoded).not.toContain("<");
expect(decodeHtmlEmbedSource(encoded)).toBe(source);
});
});
@@ -0,0 +1,29 @@
/**
* Flexible `basename` implementation for node and the browser
* @see https://stackoverflow.com/a/59907288/2228771
*/
export function getBasename(path: string) {
// make sure the basename is not empty, if string ends with separator
let end = path.length - 1;
while (path[end] === '/' || path[end] === '\\') {
--end;
}
// support mixing of Win + Unix path separators
const i1 = path.lastIndexOf('/', end);
const i2 = path.lastIndexOf('\\', end);
let start: number;
if (i1 === -1) {
if (i2 === -1) {
// no separator in the whole thing
return path;
}
start = i2;
} else if (i2 === -1) {
start = i1;
} else {
start = Math.max(i1, i2);
}
return path.substring(start + 1, end + 1);
}
@@ -0,0 +1,33 @@
/**
* Shared pieces for the two callout tokenizers `callout.marked.ts` (the
* `:::type` fenced form) and `github-callout.marked.ts` (the `> [!type]` GitHub
* alert form). Both emit the SAME callout node, so the banner type dictionary
* and the HTML renderer live here once instead of drifting apart in two files.
* The tokenizers themselves stay separate (different syntaxes / source matching).
*/
/** The four callout banner types the editor schema supports. */
export const CALLOUT_TYPES = ['info', 'success', 'warning', 'danger'] as const;
export type CalloutType = (typeof CALLOUT_TYPES)[number];
/**
* Coerce an arbitrary type name onto a supported banner type, defaulting to
* `info` for anything unrecognized (the shared fallback both tokenizers use).
*/
export function normalizeCalloutType(type: string): CalloutType {
return (CALLOUT_TYPES as readonly string[]).includes(type)
? (type as CalloutType)
: 'info';
}
/**
* Render a callout node to the editor's HTML shape. `body` is the already
* markdown-parsed inner content (marked may hand back a string synchronously).
*/
export function renderCalloutHtml(
type: string,
body: string | Promise<string>,
): string {
return `<div data-type="callout" data-callout-type="${type}">${body}</div>`;
}
@@ -0,0 +1,37 @@
import { Token, marked } from 'marked';
import { normalizeCalloutType, renderCalloutHtml } from './callout-common.marked';
interface CalloutToken {
type: 'callout';
calloutType: string;
text: string;
raw: string;
}
export const calloutExtension = {
name: 'callout',
level: 'block',
start(src: string) {
return src.match(/:::/)?.index ?? -1;
},
tokenizer(src: string): CalloutToken | undefined {
const rule = /^:::([a-zA-Z0-9]+)\s+([\s\S]+?):::/;
const match = rule.exec(src);
if (match) {
return {
type: 'callout',
calloutType: normalizeCalloutType(match[1]),
raw: match[0],
text: match[2].trim(),
};
}
},
renderer(token: Token) {
const calloutToken = token as CalloutToken;
return renderCalloutHtml(
calloutToken.calloutType,
marked.parse(calloutToken.text),
);
},
};
@@ -0,0 +1,72 @@
import { describe, it, expect } from "vitest";
import { extractFootnoteDefinitions } from "./footnote.marked";
/** Pull the ordered list of `data-footnote-def` ids out of the rendered section. */
function defIds(section: string): string[] {
return [...section.matchAll(/data-footnote-def data-id="([^"]+)"/g)].map(
(m) => m[1],
);
}
/** Pull the ordered list of `[^id]` markers that remain in the body. */
function bodyMarkers(body: string): string[] {
return [...body.matchAll(/\[\^([^\]\s]+)\]/g)].map((m) => m[1]);
}
describe("extractFootnoteDefinitions: duplicate definition ids (first-wins)", () => {
// Body has ONE `[^d]` reference but THREE `[^d]:` definitions. Under the
// import model (#166) a duplicate definition id is FIRST-WINS: only the first
// definition is kept; the rest are DROPPED (and surfaced by analyzeFootnotes,
// not silently re-id'd into orphan footnotes as before). Reference markers are
// never rewritten, so repeated references would reuse the single footnote.
const md = ["See[^d].", "", "[^d]: a", "[^d]: b", "[^d]: c"].join("\n");
it("keeps only the FIRST definition for the id (first-wins)", () => {
const { section } = extractFootnoteDefinitions(md);
const ids = defIds(section);
expect(ids).toEqual(["d"]);
});
it("keeps the first definition's text and drops the duplicates", () => {
const { section } = extractFootnoteDefinitions(md);
expect(section).toContain('data-footnote-def data-id="d"><p>a</p>');
// No derived `d__2` / `d__3` ids are emitted anymore.
expect(section).not.toContain("d__2");
expect(section).not.toContain("d__3");
// The dropped duplicate texts are not in the section.
expect(section).not.toContain("<p>b</p>");
expect(section).not.toContain("<p>c</p>");
});
it("leaves the SINGLE body marker as [^d] (markers are never rewritten)", () => {
const { body } = extractFootnoteDefinitions(md);
expect(bodyMarkers(body)).toEqual(["d"]);
expect(body).toContain("See[^d].");
// The definition lines themselves were pulled OUT of the body.
expect(body).not.toContain("[^d]: a");
expect(body).not.toContain("[^d]: b");
expect(body).not.toContain("[^d]: c");
});
it("does not crash and produces a well-formed footnotes section", () => {
const { section } = extractFootnoteDefinitions(md);
expect(section.startsWith("<section data-footnotes>")).toBe(true);
expect(section.endsWith("</section>")).toBe(true);
// Exactly one definition div (first-wins).
expect([...section.matchAll(/<div data-footnote-def/g)]).toHaveLength(1);
});
});
describe("extractFootnoteDefinitions: reuse (repeated references, one definition)", () => {
// Pandoc semantics: many `[^a]` references + one `[^a]:` definition = one
// footnote, shared. Markers are left intact so the editor numbers them as one.
const md = ["A[^a] B[^a] C[^a].", "", "[^a]: shared note"].join("\n");
it("emits exactly one definition and leaves every reference marker as [^a]", () => {
const { section, body } = extractFootnoteDefinitions(md);
expect(defIds(section)).toEqual(["a"]);
expect(section).toContain('data-footnote-def data-id="a"><p>shared note</p>');
// All three reference markers stay `a` (no `a__2`/`a__3` minting).
expect(bodyMarkers(body)).toEqual(["a", "a", "a"]);
});
});
@@ -0,0 +1,131 @@
import { marked } from "marked";
/**
* Pandoc/GFM footnote support for the marked (Markdown -> HTML) pipeline.
*
* Two pieces:
* - an INLINE tokenizer for `[^id]` references -> <sup data-footnote-ref
* data-id="id"> (matches the editor-ext FootnoteReference renderHTML);
* - a document hook (`preprocess`/`walkTokens` is awkward for collecting +
* removing definitions, so we use a regex preprocessing step instead) that
* pulls every `[^id]: text` definition line out of the body and appends a
* single <section data-footnotes> with one <div data-footnote-def> per
* definition, so the round-trip rebuilds footnotesList + footnoteDefinition.
*
* Every FIRST definition line is emitted duplicate ids are first-wins (the
* rest are dropped, and surfaced via analyzeFootnotes), and reference markers are
* left untouched so repeated `[^a]` references reuse the one footnote (#166).
* Orphan definitions (no matching reference) are still emitted here; the editor's
* sync plugin reconciles the final reference/definition set (drops orphans,
* synthesizes a single empty definition for a reference that lacks one).
*/
const DEFINITION_RE = /^\[\^([^\]\s]+)\]:[ \t]*(.*)$/;
const REFERENCE_RE = /\[\^([^\]\s]+)\]/;
interface FootnoteRefToken {
type: "footnoteRef";
raw: string;
id: string;
}
export const footnoteReferenceExtension = {
name: "footnoteRef",
level: "inline" as const,
start(src: string) {
return src.match(/\[\^/)?.index ?? -1;
},
tokenizer(src: string): FootnoteRefToken | undefined {
const match = REFERENCE_RE.exec(src);
// Only match at the very start of the remaining inline source.
if (match && match.index === 0) {
return {
type: "footnoteRef",
raw: match[0],
id: match[1],
};
}
return undefined;
},
renderer(token: FootnoteRefToken) {
return `<sup data-footnote-ref data-id="${escapeAttr(token.id)}"></sup>`;
},
};
function escapeAttr(value: string): string {
return String(value).replace(/&/g, "&amp;").replace(/"/g, "&quot;");
}
/**
* Extract `[^id]: text` definition lines from the markdown body, returning the
* cleaned body plus a rendered <section data-footnotes> (empty string when no
* definitions). Call this BEFORE marked.parse and append the section to the
* resulting HTML.
*/
export function extractFootnoteDefinitions(markdown: string): {
body: string;
section: string;
} {
const lines = markdown.split("\n");
const bodyLines: string[] = [];
const definitions: Array<{ id: string; text: string }> = [];
// Track fenced-code state so a `[^id]: ...` line that merely SHOWS footnote
// syntax inside a ``` / ~~~ code block is left in the body verbatim and not
// mistaken for a real definition.
let fence: string | null = null;
for (const line of lines) {
const fenceMatch = /^(\s*)(`{3,}|~{3,})/.exec(line);
if (fenceMatch) {
const marker = fenceMatch[2][0];
if (fence === null) {
fence = marker; // opening fence
} else if (marker === fence) {
fence = null; // closing fence (matching delimiter type)
}
bodyLines.push(line);
continue;
}
const m = fence === null ? DEFINITION_RE.exec(line) : null;
if (m) {
definitions.push({ id: m[1], text: m[2] });
} else {
bodyLines.push(line);
}
}
if (definitions.length === 0) {
return { body: markdown, section: "" };
}
// Duplicate definition ids (e.g. `[^d]: first` / `[^d]: second`): FIRST WINS,
// the rest are DROPPED. Reference markers are left UNTOUCHED so repeated `[^a]`
// references reuse the single footnote (Pandoc semantics, #166). This differs
// from the live editor's never-lose policy (resolveCollisions re-ids a
// duplicate definition into an orphan) on purpose: an import is an
// agent-authored artifact we sanitize, and the dropped duplicate is surfaced
// to the caller via analyzeFootnotes' `duplicateDefinitions` warning instead.
const firstById = new Map<string, string>(); // id -> first definition text
for (const def of definitions) {
if (!firstById.has(def.id)) firstById.set(def.id, def.text);
}
const defsHtml = [...firstById.entries()]
.map(([id, text]) => {
// Render the definition text as inline markdown so emphasis/links inside
// a footnote survive the round-trip; wrap in a paragraph (the node's
// content is paragraph+).
const inner = marked.parseInline(text || "");
return `<div data-footnote-def data-id="${escapeAttr(
id,
)}"><p>${inner}</p></div>`;
})
.join("");
return {
body: bodyLines.join("\n"),
section: `<section data-footnotes>${defsHtml}</section>`,
};
}
@@ -0,0 +1,54 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml } from "./marked.utils";
/**
* Regression for issue #192: pasting a GitHub-style `> [!type]` alert produced a
* literal `<blockquote>` containing `[!info]` instead of a callout node, because
* only the `:::type` form was tokenized. The editor paste path runs the same
* `markdownToHtml`, so these assertions pin the conversion at the source.
*/
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
describe("markdownToHtml: GitHub `> [!type]` callouts", () => {
it("converts `> [!info]` to a callout node, not a literal blockquote", () => {
const out = html("> [!info]\n> Callout body text here");
expect(out).toContain('data-type="callout"');
expect(out).toContain('data-callout-type="info"');
expect(out).toContain("Callout body text here");
expect(out).not.toContain("[!info]");
expect(out).not.toContain("<blockquote");
});
it("maps GitHub alert aliases onto the supported banner types", () => {
expect(html("> [!NOTE]\n> x")).toContain('data-callout-type="info"');
expect(html("> [!TIP]\n> x")).toContain('data-callout-type="success"');
expect(html("> [!WARNING]\n> x")).toContain('data-callout-type="warning"');
expect(html("> [!CAUTION]\n> x")).toContain('data-callout-type="danger"');
});
it("accepts the editor's own type names directly", () => {
expect(html("> [!success]\n> x")).toContain('data-callout-type="success"');
expect(html("> [!danger]\n> x")).toContain('data-callout-type="danger"');
});
it("falls back to info for an unknown type", () => {
expect(html("> [!bogus]\n> x")).toContain('data-callout-type="info"');
});
it("preserves multi-line callout bodies", () => {
const out = html("> [!warning]\n> line one\n> line two");
expect(out).toContain('data-callout-type="warning"');
expect(out).toContain("line one");
expect(out).toContain("line two");
});
it("still converts the `:::type` form", () => {
const out = html(":::info\nbody\n:::");
expect(out).toContain('data-type="callout"');
expect(out).toContain('data-callout-type="info"');
});
});
@@ -0,0 +1,81 @@
import { Token, marked } from 'marked';
import { renderCalloutHtml } from './callout-common.marked';
interface GithubCalloutToken {
type: 'githubCallout';
calloutType: string;
text: string;
raw: string;
}
/**
* Map GitHub "alert" blockquote markers (`> [!NOTE]`, `> [!WARNING]`, ) onto
* the four callout banner types the editor schema supports. The editor's own
* type names (`info`/`success`/`warning`/`danger`) are also accepted directly,
* because users paste both forms. Anything unrecognized falls back to `info`,
* matching the `:::type` callout tokenizer.
*/
const GITHUB_ALERT_TYPE_MAP: Record<string, string> = {
note: 'info',
tip: 'success',
important: 'info',
warning: 'warning',
caution: 'danger',
info: 'info',
success: 'success',
danger: 'danger',
};
/**
* Tokenizer for GitHub-flavored alert callouts written as a blockquote whose
* first line is `[!type]`:
*
* > [!info]
* > body line one
* > body line two
*
* Without this, the default blockquote tokenizer wins and the marker renders as
* a literal `[!info]` inside a `<blockquote>`. The editor's paste path runs the
* same `markdownToHtml`, so registering this here also fixes pasting the syntax
* into the editor (issue #192), not just markdown import.
*/
export const githubCalloutExtension = {
name: 'githubCallout',
level: 'block' as const,
start(src: string) {
return src.match(/^ {0,3}>[ \t]*\[!/m)?.index ?? -1;
},
tokenizer(src: string): GithubCalloutToken | undefined {
const rule =
/^ {0,3}>[ \t]*\[!([a-zA-Z]+)\][^\n]*(?:\n {0,3}>[^\n]*)*(?:\n|$)/;
const match = rule.exec(src);
if (!match) return undefined;
const rawType = match[1].toLowerCase();
const calloutType = GITHUB_ALERT_TYPE_MAP[rawType] ?? 'info';
const text = match[0]
.replace(/\n+$/, '')
.split('\n')
// Strip the blockquote marker (`>` + optional space) from every line.
.map((line) => line.replace(/^ {0,3}>[ \t]?/, ''))
// Drop the `[!type]` marker that opens the first line.
.map((line, i) => (i === 0 ? line.replace(/^\[![a-zA-Z]+\][ \t]*/, '') : line))
.join('\n')
.trim();
return {
type: 'githubCallout',
calloutType,
raw: match[0],
text,
};
},
renderer(token: Token) {
const calloutToken = token as GithubCalloutToken;
return renderCalloutHtml(
calloutToken.calloutType,
marked.parse(calloutToken.text),
);
},
};
@@ -0,0 +1,41 @@
import { Token } from "marked";
interface HtmlEmbedToken {
type: "htmlEmbed";
raw: string;
encoded: string;
}
/**
* Marked extension that rebuilds an `htmlEmbed` node from the HTML comment
* marker produced by the turndown rule (`<!--html-embed:<base64>-->`).
*
* It emits the same marker div the node's `parseHTML` recognizes, so the
* pipeline MD -> HTML -> ProseMirror JSON restores the node (and its
* base64 `data-source`) exactly. We do NOT expand the raw markup here; the
* source stays base64-encoded in the attribute and is only executed by the
* client NodeView.
*/
export const htmlEmbedExtension = {
name: "htmlEmbed",
level: "block" as const,
start(src: string) {
return src.indexOf("<!--html-embed:");
},
tokenizer(src: string): HtmlEmbedToken | undefined {
const rule = /^<!--html-embed:([A-Za-z0-9+/=]*)-->/;
const match = rule.exec(src);
if (match) {
return {
type: "htmlEmbed",
raw: match[0],
encoded: match[1] ?? "",
};
}
},
renderer(token: Token) {
const htmlEmbedToken = token as HtmlEmbedToken;
return `<div data-type="htmlEmbed" data-source="${htmlEmbedToken.encoded}"></div>`;
},
};
@@ -0,0 +1,76 @@
import { marked } from "marked";
import { calloutExtension } from "./callout.marked";
import { githubCalloutExtension } from "./github-callout.marked";
import { mathBlockExtension } from "./math-block.marked";
import { mathInlineExtension } from "./math-inline.marked";
import {
footnoteReferenceExtension,
extractFootnoteDefinitions,
} from "./footnote.marked";
import { htmlEmbedExtension } from "./html-embed.marked";
marked.use({
renderer: {
list({ ordered, start, items }) {
let body = "";
for (const item of items) {
body += this.listitem(item);
}
if (ordered) {
const startAttr = start !== 1 ? ` start="${start}"` : "";
return `<ol${startAttr}>\n${body}</ol>\n`;
}
const isTaskList = items.some((item) => item.task);
const dataType = isTaskList ? ' data-type="taskList"' : "";
return `<ul${dataType}>\n${body}</ul>\n`;
},
listitem({ tokens, task: isTask, checked: isChecked }) {
const text = this.parser.parse(tokens);
if (!isTask) {
return `<li>${text}</li>\n`;
}
const checkedAttr = isChecked
? 'data-checked="true"'
: 'data-checked="false"';
return `<li data-type="taskItem" ${checkedAttr}>${text}</li>\n`;
},
},
});
marked.use({
extensions: [
calloutExtension,
githubCalloutExtension,
mathBlockExtension,
mathInlineExtension,
footnoteReferenceExtension,
htmlEmbedExtension,
],
});
marked.setOptions({ breaks: true });
export function markdownToHtml(
markdownInput: string,
): string | Promise<string> {
const YAML_FONT_MATTER_REGEX = /^\s*---[\s\S]*?---\s*/;
const markdown = markdownInput
.replace(YAML_FONT_MATTER_REGEX, "")
.trimStart();
// Pull `[^id]: ...` definition lines out of the body, render the body, then
// append a single <section data-footnotes> so the round-trip rebuilds the
// footnotesList + footnoteDefinition nodes.
const { body, section } = extractFootnoteDefinitions(markdown);
const parsed = marked.parse(body);
if (!section) return parsed;
if (typeof parsed === "string") {
return parsed + section;
}
return parsed.then((html) => html + section);
}
@@ -0,0 +1,37 @@
import { Token, marked } from 'marked';
interface MathBlockToken {
type: 'mathBlock';
text: string;
raw: string;
}
export const mathBlockExtension = {
name: 'mathBlock',
level: 'block',
start(src: string) {
return src.match(/\$\$/)?.index ?? -1;
},
tokenizer(src: string): MathBlockToken | undefined {
const rule = /^\$\$(?!(\$))([\s\S]+?)\$\$/;
const match = rule.exec(src);
if (match) {
return {
type: 'mathBlock',
raw: match[0],
text: match[2]?.trim(),
};
}
},
renderer(token: Token) {
const mathBlockToken = token as MathBlockToken;
// parse to prevent escaping slashes
const latex = marked
.parse(mathBlockToken.text)
.toString()
.replace(/<(\/)?p>/g, '');
return `<div data-type="${mathBlockToken.type}" data-katex="true">${latex}</div>`;
},
};
@@ -0,0 +1,50 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml } from "./marked.utils";
/**
* Data-integrity regression (issue #204, Phase 2): plain prose that mentions
* prices like `$5 and $6` must NOT be misread as inline math. The inline-math
* tokenizer mutates a global `marked` singleton at import time
* (`marked.utils.ts`), so math behaviour can only be exercised safely through
* the public `markdownToHtml`; importing the tokenizer in isolation would give
* a different, non-representative result. These assertions therefore drive the
* real conversion path.
*/
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
const MATH_MARKERS = ['data-type="mathInline"', 'data-katex="true"'];
function hasInlineMath(out: string): boolean {
return MATH_MARKERS.some((m) => out.includes(m));
}
describe("markdownToHtml: inline-math false positives", () => {
it("does not treat prices `$5 and $6` as inline math", () => {
const out = html("It costs $5 and $6 today.");
expect(hasInlineMath(out)).toBe(false);
// The text survives verbatim (no katex span swallowing it).
expect(out).toContain("$5 and $6");
});
it("does not treat a single trailing price `$5` as inline math", () => {
const out = html("Lunch was $5.");
expect(hasInlineMath(out)).toBe(false);
expect(out).toContain("$5");
});
it("does not treat `$5, $6, $7` (multiple prices) as inline math", () => {
const out = html("Choose $5, $6, $7 plans.");
expect(hasInlineMath(out)).toBe(false);
});
it("STILL converts a genuine inline-math expression `$x + y$`", () => {
// Guard the positive path so the false-positive guard above can't be
// satisfied by simply disabling math entirely.
const out = html("The sum $x + y$ is shown.");
expect(hasInlineMath(out)).toBe(true);
});
});
@@ -0,0 +1,55 @@
import { Token, marked } from 'marked';
interface MathInlineToken {
type: 'mathInline';
text: string;
raw: string;
}
const inlineMathRegex = /^\$(?!\s)(.+?)(?<!\s)\$(?!\d)/;
export const mathInlineExtension = {
name: 'mathInline',
level: 'inline',
start(src: string) {
let index: number;
let indexSrc = src;
while (indexSrc) {
index = indexSrc.indexOf('$');
if (index === -1) {
return;
}
const f = index === 0 || indexSrc.charAt(index - 1) === ' ';
if (f) {
const possibleKatex = indexSrc.substring(index);
if (possibleKatex.match(inlineMathRegex)) {
return index;
}
}
indexSrc = indexSrc.substring(index + 1).replace(/^\$+/, '');
}
},
tokenizer(src: string): MathInlineToken | undefined {
const match = inlineMathRegex.exec(src);
if (match) {
return {
type: 'mathInline',
raw: match[0],
text: match[1]?.trim(),
};
}
},
renderer(token: Token) {
const mathInlineToken = token as MathInlineToken;
// parse to prevent escaping slashes
const latex = marked
.parse(mathInlineToken.text)
.toString()
.replace(/<(\/)?p>/g, '');
return `<span data-type="${mathInlineToken.type}" data-katex="true">${latex}</span>`;
},
};
@@ -0,0 +1,128 @@
import { describe, it, expect } from "vitest";
import { getSchema } from "@tiptap/core";
import { generateHTML, generateJSON } from "@tiptap/html";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Bold } from "@tiptap/extension-bold";
import { htmlToMarkdown } from "./turndown.utils";
import { markdownToHtml } from "./marked.utils";
import { Spoiler } from "../../spoiler/spoiler";
// The spoiler mark has no native Markdown syntax, so it is preserved losslessly
// as raw inline HTML (`<span data-spoiler="true">…</span>`), the same approach
// htmlEmbed uses. This test drives the full editor round-trip:
// JSON -> HTML -> Markdown -> HTML -> JSON
// and asserts the `spoiler` mark survives end to end. We use the same
// getSchema + @tiptap/html generateHTML/generateJSON utilities the other
// editor-ext schema tests use.
const extensions = [Document, Paragraph, Text, Bold, Spoiler];
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
// Count text nodes carrying a `spoiler` mark anywhere in a ProseMirror JSON doc.
function countSpoilerMarks(doc: any): number {
let count = 0;
const walk = (node: any) => {
if (!node || typeof node !== "object") return;
if (Array.isArray(node.marks)) {
for (const mark of node.marks) {
if (mark?.type === "spoiler") count++;
}
}
if (Array.isArray(node.content)) node.content.forEach(walk);
};
walk(doc);
return count;
}
describe("Spoiler mark schema", () => {
it("registers the spoiler mark in the schema", () => {
const schema = getSchema(extensions);
expect(schema.marks.spoiler).toBeTruthy();
});
it("recovers the spoiler mark from span[data-spoiler] (HTML -> JSON)", () => {
const json = generateJSON(
'<p>before <span data-spoiler="true">hidden</span> after</p>',
extensions,
);
expect(countSpoilerMarks(json)).toBe(1);
});
it("emits data-spoiler + class on render (JSON -> HTML)", () => {
const doc = {
type: "doc",
content: [
{
type: "paragraph",
content: [
{
type: "text",
text: "hidden",
marks: [{ type: "spoiler" }],
},
],
},
],
};
const out = generateHTML(doc, extensions);
expect(out).toContain('data-spoiler="true"');
expect(out).toContain('class="spoiler"');
});
});
describe("Spoiler Markdown round-trip is lossless", () => {
const docWith = (textNode: any) => ({
type: "doc",
content: [
{
type: "paragraph",
content: [{ type: "text", text: "before " }, textNode, { type: "text", text: " after" }],
},
],
});
it("preserves the spoiler mark through JSON -> MD -> HTML -> JSON", () => {
const startDoc = docWith({
type: "text",
text: "hidden",
marks: [{ type: "spoiler" }],
});
// JSON -> HTML
const html1 = generateHTML(startDoc, extensions);
expect(html1).toContain('data-spoiler="true"');
// HTML -> Markdown (raw inline HTML, lossless)
const md = htmlToMarkdown(html1);
expect(md).toContain('<span data-spoiler="true">hidden</span>');
// MD -> HTML -> JSON (mark restored via parseHTML)
const endJson = generateJSON(html(md), extensions);
expect(countSpoilerMarks(endJson)).toBe(1);
// The visible text survives.
expect(JSON.stringify(endJson)).toContain("hidden");
});
it("keeps the spoiler intact when it intersects a bold mark", () => {
const startDoc = docWith({
type: "text",
text: "secret",
marks: [{ type: "bold" }, { type: "spoiler" }],
});
const md = htmlToMarkdown(generateHTML(startDoc, extensions));
expect(md).toContain("data-spoiler=\"true\"");
const endJson = generateJSON(html(md), extensions);
expect(countSpoilerMarks(endJson)).toBe(1);
// Bold survives alongside the spoiler.
expect(JSON.stringify(endJson)).toContain('"bold"');
});
});
@@ -0,0 +1,12 @@
// Map @joplin/turndown types to @types/turndown
declare module "@joplin/turndown" {
import TurndownService from "turndown";
export = TurndownService;
}
declare module "@joplin/turndown-plugin-gfm" {
import TurndownService from "turndown";
export const tables: TurndownService.Plugin;
export const strikethrough: TurndownService.Plugin;
export const highlightedCodeBlock: TurndownService.Plugin;
}
@@ -0,0 +1,147 @@
import { describe, it, expect } from "vitest";
import { htmlToMarkdown } from "./turndown.utils";
import { markdownToHtml } from "./marked.utils";
/**
* #206 mdrt-2 Markdown export must never SILENTLY drop a block. (FIXED)
*
* `htmlToMarkdown` (turndown) historically only registered rules for a fixed
* set of custom nodes (callout, taskItem, details, math, iframe, htmlEmbed,
* image, video, footnote). Any other custom node `transclusionReference`,
* `pageBreak`, `mention`, `status` fell through to turndown's default
* handling: an empty wrapper is "blank" and removed, so the block disappeared
* from the exported Markdown with no trace, and `mention`/`status` collapsed to
* bare text, losing their identity (data-id / data-color). The invariant
* "never silently lose a block" was broken.
*
* The fix adds lossless turndown rules that re-emit each of these nodes as raw
* HTML carrying every `data-*` attribute. Plain-Markdown viewers ignore the
* inert tag; the import path round-trips it (`markdownToHtml` passes the raw
* HTML through and each node's `parseHTML` rebuilds the ProseMirror node). These
* tests assert the surviving contract (the block is preserved AND its identity
* round-trips back through import).
*/
describe("htmlToMarkdown — custom nodes are preserved losslessly (#206 mdrt-2)", () => {
const wrap = (inner: string) => `<p>before</p>${inner}<p>after</p>`;
it("preserves a pageBreak block on Markdown export", () => {
const md = htmlToMarkdown(
wrap('<div data-type="pageBreak" class="page-break"></div>'),
);
expect(md).toContain("before");
expect(md).toContain("after");
// The break survives as an inert raw-HTML tag, not silently dropped.
expect(md).toMatch(/data-type="pageBreak"/);
expect(md).toMatch(/page-?break/i);
});
it("preserves a transclusionReference's identity on Markdown export", () => {
const md = htmlToMarkdown(
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
);
expect(md).toContain("before");
expect(md).toContain("after");
// The data-id (the only thing that gives the reference identity) survives.
expect(md).toContain("abc");
expect(md).toMatch(/data-type="transclusionReference"/);
});
it("preserves a mention's data-id (stable identity) on Markdown export", () => {
const md = htmlToMarkdown(
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
);
// The mention keeps its stable identity (data-id), not just the text.
expect(md).toContain("u1");
expect(md).toContain("Bob");
expect(md).toMatch(/data-type="mention"/);
});
it("preserves a status chip's color on Markdown export", () => {
const md = htmlToMarkdown(
'<p>s <span data-type="status" data-color="green">Done</span></p>',
);
// The chip's color (its identity) survives, not just the visible text.
expect(md).toContain("green");
expect(md).toContain("Done");
expect(md).toMatch(/data-type="status"/);
});
// The export form is only lossless if the import path can rebuild it. These
// assert the full MD -> HTML round-trip restores the node + its attributes,
// which is the marker <-> node contract each `parseHTML` relies on.
describe("import round-trip (markdownToHtml restores the node)", () => {
it("round-trips a pageBreak through export + import", async () => {
const md = htmlToMarkdown(
wrap('<div data-type="pageBreak" class="page-break"></div>'),
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<div[^>]*data-type="pageBreak"[^>]*>/);
expect(html).toContain("before");
expect(html).toContain("after");
});
it("round-trips a transclusionReference (keeps data-id)", async () => {
const md = htmlToMarkdown(
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<div[^>]*data-type="transclusionReference"[^>]*>/);
expect(html).toContain("abc");
});
it("round-trips a mention (keeps data-id + data-label)", async () => {
const md = htmlToMarkdown(
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<span[^>]*data-type="mention"[^>]*>/);
expect(html).toContain("u1");
expect(html).toContain("Bob");
});
it("round-trips a status chip (keeps data-color)", async () => {
const md = htmlToMarkdown(
'<p>s <span data-type="status" data-color="green">Done</span></p>',
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<span[^>]*data-type="status"[^>]*>/);
expect(html).toContain("green");
});
// HTML special chars in an attribute value or in a node's text must be
// ESCAPED when re-emitted as raw HTML, otherwise the exported tag is
// malformed and `markdownToHtml`'s parser cannot restore the original value
// (the same silent data loss this PR fixes). Dropping `<`/`>` escaping is the
// dangerous regression: a stray `<` or `>` corrupts the tag (or injects new
// markup), so the test data carries ALL of `&`, `"`, `<`, `>` in BOTH the
// data-label attribute and the visible text. That fully exercises
// escapeHtmlAttr's `&,",<,>` branches and escapeHtmlText's `&,<,>` branches
// (escapeHtmlText leaves `"` literal); the alphanumeric-only cases above hit
// none of them.
it("escapes HTML special chars (& \" < >) in attrs + text and round-trips them", async () => {
const md = htmlToMarkdown(
`<p>hi <span data-type="mention" data-id="u1" data-label="A &amp; &lt;B&gt; &quot;C&quot;">@A &amp; &lt;B&gt; "C"</span> there</p>`,
);
// (a) The exported Markdown carries a WELL-FORMED, correctly-escaped tag:
// the attribute escapes `&`, `<`, `>` AND `"`; the text escapes `&`, `<`,
// `>` (a `"` inside text content is legal, so it stays literal).
expect(md).toContain('data-label="A &amp; &lt;B&gt; &quot;C&quot;"');
expect(md).toContain('>@A &amp; &lt;B&gt; "C"</span>');
// And explicitly NOT the raw, tag-corrupting forms: a literal `<B>` (would
// mean `<`/`>` escaping was dropped in either the attr or the text)...
expect(md).not.toContain("<B>");
// ...nor the malformed attribute that an unescaped `"` would produce.
expect(md).not.toContain('data-label="A &amp; &lt;B&gt; "C""');
// (b) Import restores the ORIGINAL (unescaped) values, attribute and text.
const html = await markdownToHtml(md);
const dom = new DOMParser().parseFromString(html as string, "text/html");
const span = dom.querySelector('span[data-type="mention"]');
expect(span).not.toBeNull();
expect(span!.getAttribute("data-id")).toBe("u1");
expect(span!.getAttribute("data-label")).toBe('A & <B> "C"');
expect(span!.textContent).toBe('@A & <B> "C"');
});
});
});
@@ -0,0 +1,488 @@
import * as _TurndownService from '@joplin/turndown';
import * as TurndownPluginGfm from '@joplin/turndown-plugin-gfm';
import { getBasename } from './basename';
// CJS/ESM interop: .default exists in Vite, not in NestJS
const TurndownService = (_TurndownService as any).default || _TurndownService;
function sanitizeMdLinkText(value: string): string {
return value
.replace(/\\/g, '\\\\')
.replace(/([\[\]!])/g, '\\$1')
.replace(/[\r\n]+/g, ' ');
}
// Tags turndown treats as void (self-closing). Footnote references render as an
// empty <sup data-footnote-ref> whose meaning lives entirely in its data-id;
// without marking it void, turndown's blank-node removal drops it before our
// rule runs, losing the `[^id]` marker. Mirrors turndown's built-in list.
const TURNDOWN_VOID_ELEMENTS = [
'AREA', 'BASE', 'BR', 'COL', 'COMMAND', 'EMBED', 'HR', 'IMG', 'INPUT',
'KEYGEN', 'LINK', 'META', 'PARAM', 'SOURCE', 'TRACK', 'WBR',
];
function isVoidNode(node: any): boolean {
const name = node?.nodeName?.toUpperCase?.();
if (!name) return false;
if (name === 'SUP' && node.hasAttribute?.('data-footnote-ref')) {
return true;
}
return TURNDOWN_VOID_ELEMENTS.indexOf(name) !== -1;
}
/**
* An empty <sup data-footnote-ref> is "blank" to turndown, which removes blank
* inline nodes (RootNode/Node use a module-level isVoid the options cannot
* override). To survive, inject the id as text content so the node is non-blank;
* the footnoteReference rule then reads data-id and emits `[^id]`.
*/
function fillEmptyFootnoteRefs(html: string): string {
return html.replace(
/<sup\b([^>]*\bdata-footnote-ref\b[^>]*)>\s*<\/sup>/gi,
(_m, attrs) => `<sup${attrs}>​</sup>`,
);
}
/**
* `pageBreak` and `transclusionReference` are childless atom <div>s. Like an
* empty footnote ref (see above), turndown treats a childless block as "blank"
* and replaces it with the blankRule BEFORE any custom rule can fire so the
* node disappears from the export with no trace (#206 mdrt-2). Inject a
* zero-width space so the node is non-blank and our lossless rule runs; the
* rule rebuilds the tag from the element's attributes, so the injected char
* never reaches the output.
*/
function fillEmptyAtomBlocks(html: string): string {
return html.replace(
/<div\b([^>]*\bdata-type="(?:pageBreak|transclusionReference)"[^>]*)>\s*<\/div>/gi,
(_m, attrs) => `<div${attrs}>​</div>`,
);
}
/** HTML-escape an attribute value so a re-emitted raw-HTML tag is well-formed. */
function escapeHtmlAttr(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
/** HTML-escape text placed inside a re-emitted raw-HTML element. */
function escapeHtmlText(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
/**
* Serialize ALL of an element's attributes back to a raw-HTML attribute string
* (leading space included). Generic on purpose: a custom node's identity lives
* entirely in its `data-*` attributes (data-id, data-color, data-source-page-id,
* data-transclusion-id, ), and serializing every attribute keeps the export
* lossless regardless of which attributes a given node carries.
*/
function serializeAttrs(node: any): string {
const attrs = node?.attributes;
if (!attrs) return '';
return Array.from(attrs as ArrayLike<{ name: string; value: string }>)
.map((attr) => ` ${attr.name}="${escapeHtmlAttr(attr.value ?? '')}"`)
.join('');
}
export function htmlToMarkdown(html: string): string {
const turndownService = new TurndownService({
headingStyle: 'atx',
codeBlockStyle: 'fenced',
hr: '---',
bulletListMarker: '-',
isVoid: isVoidNode,
});
turndownService.use([
TurndownPluginGfm.tables,
TurndownPluginGfm.strikethrough,
TurndownPluginGfm.highlightedCodeBlock,
taskList,
callout,
preserveDetail,
listParagraph,
orderedListItem,
mathInline,
mathBlock,
iframeEmbed,
htmlEmbed,
spoiler,
image,
video,
footnoteReference,
footnotesList,
pageBreak,
transclusionReference,
mention,
status,
]);
return turndownService
.turndown(fillEmptyAtomBlocks(fillEmptyFootnoteRefs(html)))
.replaceAll('<br>', ' ');
}
/**
* Lossless export rules for custom nodes that have NO native Markdown syntax
* (#206 mdrt-2). Markdown cannot represent a page break, a transclusion
* reference, a mention's stable id, or a status chip's color so rather than
* letting turndown silently drop them, each rule re-emits the node as raw HTML
* carrying every `data-*` attribute. Plain-Markdown viewers ignore the inert
* tag, and the import path round-trips it: `markdownToHtml` passes raw HTML
* through and each node's `parseHTML` (`div[data-type="…"]`, `span[…]`) rebuilds
* the ProseMirror node with its attributes intact.
*/
function pageBreak(turndownService: _TurndownService) {
turndownService.addRule('pageBreak', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'pageBreak'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
},
});
}
function transclusionReference(turndownService: _TurndownService) {
turndownService.addRule('transclusionReference', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'transclusionReference'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
},
});
}
function mention(turndownService: _TurndownService) {
turndownService.addRule('mention', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-type') === 'mention'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const text = escapeHtmlText(node.textContent || '');
return `<span${serializeAttrs(node)}>${text}</span>`;
},
});
}
function status(turndownService: _TurndownService) {
turndownService.addRule('status', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' && node.getAttribute('data-type') === 'status'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const text = escapeHtmlText(node.textContent || '');
return `<span${serializeAttrs(node)}>${text}</span>`;
},
});
}
/**
* Serialize the `htmlEmbed` node to Markdown.
*
* Markdown has no native representation for an arbitrary-HTML block, so we
* preserve the node losslessly as an HTML comment carrying the base64-encoded
* source (the same `data-source` payload the node stores). `markdownToHtml`
* recognizes the same marker and rebuilds the node, so the round-trip
* MD -> HTML -> JSON keeps the source intact. The comment also keeps the raw
* markup inert in the exported `.md` file (it does not render in plain Markdown
* viewers).
*/
function htmlEmbed(turndownService: _TurndownService) {
turndownService.addRule('htmlEmbed', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'htmlEmbed'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const encoded = node.getAttribute('data-source') || '';
return `\n\n<!--html-embed:${encoded}-->\n\n`;
},
});
}
/**
* Serialize the `spoiler` inline mark to lossless raw inline HTML.
*
* Markdown has no native spoiler syntax, so we emit the same `<span
* data-spoiler="true"></span>` the mark renders. `marked` passes inline raw HTML
* through untouched, and `generateJSON` restores the mark via its parseHTML, so
* the round-trip MD -> HTML -> JSON keeps the spoiler intact. The UI-only
* `is-revealed` state is never serialized.
*/
function spoiler(turndownService: _TurndownService) {
turndownService.addRule('spoiler', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-spoiler') === 'true'
);
},
replacement: function (content: string) {
return `<span data-spoiler="true">${content}</span>`;
},
});
}
function listParagraph(turndownService: _TurndownService) {
turndownService.addRule('paragraph', {
filter: ['p'],
replacement: (content: string, node: HTMLInputElement) => {
if (node.parentElement?.nodeName === 'LI') {
return content;
}
return `\n\n${content}\n\n`;
},
});
}
function orderedListItem(turndownService: _TurndownService) {
turndownService.addRule('orderedListItem', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'LI' && node.getAttribute('data-type') !== 'taskItem';
},
replacement: (content: string, node: HTMLInputElement, options: any) => {
const parent = node.parentNode as HTMLElement;
if (parent.nodeName !== 'OL' && parent.nodeName !== 'UL') {
return content;
}
content = content
.replace(/^\n+/, '')
.replace(/\n+$/, '\n')
.replace(/\n/gm, '\n ');
let prefix: string;
if (parent.nodeName === 'OL') {
const start = parseInt(parent.getAttribute('start') || '1', 10);
const index = Array.prototype.indexOf.call(parent.children, node);
prefix = `${start + index}. `;
} else {
prefix = `${options.bulletListMarker} `;
}
return (
prefix +
content +
(node.nextSibling && !/\n$/.test(content) ? '\n' : '')
);
},
});
}
function callout(turndownService: _TurndownService) {
turndownService.addRule('callout', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' && node.getAttribute('data-type') === 'callout'
);
},
replacement: function (content: string, node: HTMLInputElement) {
const calloutType = node.getAttribute('data-callout-type');
return `\n\n:::${calloutType}\n${content.trim()}\n:::\n\n`;
},
});
}
function taskList(turndownService: _TurndownService) {
turndownService.addRule('taskListItem', {
filter: function (node: HTMLInputElement) {
return (
node.getAttribute('data-type') === 'taskItem' &&
node.parentNode.nodeName === 'UL'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const isChecked = node.getAttribute('data-checked') === 'true';
const div = node.querySelector('div');
const text = div ? div.textContent.trim() : node.textContent.trim();
const prefix = `- ${isChecked ? '[x]' : '[ ]'} `;
return (
prefix +
text +
(node.nextSibling && !/\n$/.test(text) ? '\n' : '')
);
},
});
}
function preserveDetail(turndownService: _TurndownService) {
turndownService.addRule('preserveDetail', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'DETAILS';
},
replacement: function (_content: string, node: HTMLInputElement) {
const summary = node.querySelector(':scope > summary');
let detailSummary = '';
if (summary) {
detailSummary = `<summary>${turndownService.turndown(summary.innerHTML)}</summary>`;
}
const detailsContent = Array.from(node.childNodes)
.filter((child) => child.nodeName !== 'SUMMARY')
.map((child) =>
child.nodeType === 1
? turndownService.turndown((child as HTMLElement).outerHTML)
: child.textContent,
)
.join('');
return `\n<details>\n${detailSummary}\n\n${detailsContent}\n\n</details>\n`;
},
});
}
function mathInline(turndownService: _TurndownService) {
turndownService.addRule('mathInline', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-type') === 'mathInline'
);
},
replacement: function (content: string) {
return `$${content}$`;
},
});
}
function mathBlock(turndownService: _TurndownService) {
turndownService.addRule('mathBlock', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'mathBlock'
);
},
replacement: function (content: string) {
return `\n$$\n${content}\n$$\n`;
},
});
}
function iframeEmbed(turndownService: _TurndownService) {
turndownService.addRule('iframeEmbed', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'IFRAME';
},
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src');
return '[' + src + '](' + src + ')';
},
});
}
function image(turndownService: _TurndownService) {
turndownService.addRule('image', {
filter: 'img',
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src') || '';
if (!src) return '';
const caption = node.getAttribute('data-caption') || '';
if (caption) {
// ![]() can't carry a caption, so emit a raw <img> wrapped in a block
// <div>. marked passes it through and the image extension's parseHTML
// restores the caption from data-caption.
const parts = [`src="${escapeHtmlAttr(src)}"`];
const alt = node.getAttribute('alt') || '';
if (alt) parts.push(`alt="${escapeHtmlAttr(alt)}"`);
parts.push(`data-caption="${escapeHtmlAttr(caption)}"`);
return `<div><img ${parts.join(' ')}></div>`;
}
const alt = sanitizeMdLinkText(node.getAttribute('alt') || '');
const title = node.getAttribute('title') || '';
const titlePart = title ? ' "' + title.replace(/"/g, '\\"') + '"' : '';
return '![' + alt + '](' + src + titlePart + ')';
},
});
}
/**
* Footnote reference (inline atom) -> pandoc/GFM marker `[^id]`.
* The visible number is derived (not stored), so the id is the stable anchor.
*/
function footnoteReference(turndownService: _TurndownService) {
turndownService.addRule('footnoteReference', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SUP' && node.hasAttribute('data-footnote-ref')
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const id = node.getAttribute('data-id') || '';
return id ? `[^${id}]` : '';
},
});
}
/**
* Footnotes container -> the list of `[^id]: text` definitions at the end of
* the document (one per line). Each footnoteDefinition inside emits its own
* `[^id]: ...` line; turndown joins them with the surrounding block spacing.
*/
function footnotesList(turndownService: _TurndownService) {
turndownService.addRule('footnoteDefinition', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' && node.hasAttribute('data-footnote-def')
);
},
replacement: function (content: string, node: HTMLInputElement) {
const id = node.getAttribute('data-id') || '';
// Collapse internal newlines so the definition stays a single MD line;
// continuation lines are a v2 refinement.
const text = content.replace(/\s*\n+\s*/g, ' ').trim();
return id ? `\n[^${id}]: ${text}\n` : '';
},
});
turndownService.addRule('footnotesList', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SECTION' && node.hasAttribute('data-footnotes')
);
},
replacement: function (content: string) {
return `\n\n${content.trim()}\n`;
},
});
}
function video(turndownService: _TurndownService) {
turndownService.addRule('video', {
filter: function (node: HTMLInputElement) {
return node.tagName === 'VIDEO';
},
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src') || '';
const ariaLabel = node.getAttribute('aria-label');
const name = sanitizeMdLinkText(
ariaLabel || getBasename(src) || src,
);
return '[' + name + '](' + src + ')';
},
});
}
+1 -6
View File
@@ -14,15 +14,10 @@ export default defineConfig({
provider: "v8",
reporter: ["text-summary", "text"],
all: false,
// functions lowered 60 -> 57 after issue #347 removed the editor-ext
// markdown layer (src/lib/markdown) and its image/footnote round-trip
// specs: that markdown behavior now lives in — and is tested by —
// @docmost/prosemirror-markdown, so the editor-ext baseline shifts down.
// Still a real gate (a few points below the post-removal measured level).
thresholds: {
statements: 54,
branches: 44,
functions: 57,
functions: 60,
lines: 54,
},
},
+77 -37
View File
@@ -1,60 +1,100 @@
// Codegen: emit src/registry-stamp.generated.ts with a REGISTRY_STAMP hash of
// the tool-specs REGISTRY CONTENT, so a build/ vs src/ skew (issue #447) is
// detectable at runtime.
// the ENTIRE src/ tree, so a build/ vs src/ skew (issue #447) is detectable at
// runtime for ANY source file — not just tool-specs.ts.
//
// WHY hash the raw source text (not extracted structured data):
// SHARED_TOOL_SPECS carries `buildShape` functions (the input SCHEMAS) which are
// NOT serializable. The input schema is exactly one of the things that MUST stay
// in sync between build/ and src/, so we cannot drop it from the hash. Rather
// than probe zod with a fragile shim to reconstruct the schema shape, we hash the
// STABLE, deterministic source TEXT of tool-specs.ts. That text fully captures
// every field that must stay in sync — mcpName, inAppKey, description, tier,
// catalogLine AND the buildShape bodies (input schemas) — with zero probing
// fragility. Any edit to a spec (a renamed tool, a reworded description, a
// changed schema field) changes the text and therefore the stamp.
// WHY hash the whole src tree (not just tool-specs.ts): the runtime tools are
// assembled from far more than the spec registry — client.ts, the client/*
// domain modules, comment-signal.ts and the drawio-* helpers all ship in build/
// and are loaded by the in-app server. Hashing ONLY tool-specs.ts meant an edit
// to any of those (e.g. a behavioural fix in client.ts) left the stamp unchanged,
// so a stale build/ served the OLD code silently (issue #486). Hashing every
// src/**/*.ts closes that gap: any source edit changes the stamp.
//
// DETERMINISM: the hash is computed over the file bytes with line endings
// normalized to LF and a single trailing newline stripped, so a CRLF checkout or
// an editor's trailing-newline habit cannot make build/ and src/ disagree. No
// Date.now / randomness. The loader's dev-only stale-check (docmost-client.loader.ts)
// re-runs THIS SAME normalization + sha256 over src/tool-specs.ts and compares to
// the built REGISTRY_STAMP; the two must compute identically.
// WHY hash the raw source text (not extracted structured data): the tool input
// SCHEMAS live as `buildShape` functions which are NOT serializable, so we cannot
// reduce them to structured data without a fragile zod shim. Hashing the STABLE,
// deterministic source TEXT captures every field that must stay in sync with zero
// probing fragility. Any edit to any source file changes the text → the stamp.
//
// DETERMINISM: files are enumerated recursively, filtered to *.ts EXCLUDING
// *.generated.ts (the codegen's OWN output — including it would create a
// fixed-point cycle), and sorted by their POSIX-normalized path relative to src/
// so the order is platform-independent. Each file contributes its relative path
// AND its content with line endings normalized to LF and a single trailing
// newline stripped, so a CRLF checkout or an editor's trailing-newline habit
// cannot make build/ and src/ disagree. No Date.now / randomness. The loader's
// dev-only stale-check (docmost-client.loader.ts) re-runs THIS SAME enumeration +
// normalization + sha256 and compares to the built REGISTRY_STAMP; the two must
// compute identically.
//
// This script runs from the `build` and `pretest` npm scripts BEFORE tsc, so
// build/ always carries a stamp derived from the tool-specs.ts that was compiled.
// build/ always carries a stamp derived from the src/ tree that was compiled.
import { createHash } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
import { readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { dirname, join, relative, sep } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const SRC_DIR = join(__dirname, '..', 'src');
const TOOL_SPECS_PATH = join(SRC_DIR, 'tool-specs.ts');
const OUT_PATH = join(SRC_DIR, 'registry-stamp.generated.ts');
/**
* Deterministic stamp of the tool-specs registry content. Kept as a plain
* function (exported) so the algorithm has a single home; the loader duplicates
* only the tiny normalize+sha256 steps because it lives in the CJS server build
* and cannot import this ESM script. If you change the normalization here, mirror
* it in apps/server/src/core/ai-chat/tools/docmost-client.loader.ts.
* Recursively enumerate every `*.ts` file under `dir`, EXCLUDING the codegen's
* own `*.generated.ts` output (a self-referential cycle otherwise). Returns
* absolute paths, unsorted (the caller sorts by relative path for determinism).
* Kept as a plain exported function so the algorithm has a single home; the
* loader duplicates it because it lives in the CJS server build and cannot import
* this ESM script. If you change the walk/filter here, mirror it in
* apps/server/src/core/ai-chat/tools/docmost-client.loader.ts.
*/
export function computeRegistryStamp(toolSpecsSource) {
const normalized = toolSpecsSource.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
export function collectStampFiles(dir) {
const out = [];
for (const entry of readdirSync(dir)) {
const full = join(dir, entry);
if (statSync(full).isDirectory()) {
out.push(...collectStampFiles(full));
} else if (entry.endsWith('.ts') && !entry.endsWith('.generated.ts')) {
out.push(full);
}
}
return out;
}
/**
* Deterministic stamp of the whole src/ tree. Enumerate + sort by POSIX-relative
* path, then fold each file's relative path AND normalized content into one
* sha256. MUST stay byte-for-byte identical to the loader's recompute.
*/
export function computeRegistryStamp(srcDir) {
const files = collectStampFiles(srcDir)
.map((abs) => ({
rel: relative(srcDir, abs).split(sep).join('/'),
abs,
}))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const hash = createHash('sha256');
for (const { rel, abs } of files) {
const normalized = readFileSync(abs, 'utf8')
.replace(/\r\n/g, '\n')
.replace(/\n$/, '');
hash.update(rel, 'utf8');
hash.update('\0', 'utf8');
hash.update(normalized, 'utf8');
hash.update('\0', 'utf8');
}
return hash.digest('hex');
}
function main() {
const source = readFileSync(TOOL_SPECS_PATH, 'utf8');
const stamp = computeRegistryStamp(source);
const stamp = computeRegistryStamp(SRC_DIR);
const out =
'// AUTO-GENERATED by scripts/gen-registry-stamp.mjs — DO NOT EDIT BY HAND.\n' +
'// A deterministic hash of src/tool-specs.ts content (tool names, descriptions,\n' +
'// tiers, catalog lines and input schemas). Regenerated on every build/pretest\n' +
'// so build/ always matches the compiled src. The in-app loader recomputes this\n' +
'// from src and refuses to run on a mismatch (issue #447). This file is\n' +
'// gitignored and produced by the build — see .gitignore.\n' +
'// A deterministic hash of the whole src/ tree (every src/**/*.ts except\n' +
'// *.generated.ts). Regenerated on every build/pretest so build/ always\n' +
'// matches the compiled src. The in-app loader recomputes this from src and\n' +
'// refuses to run on a mismatch (issue #447/#486). This file is gitignored\n' +
'// and produced by the build — see .gitignore.\n' +
`export const REGISTRY_STAMP = ${JSON.stringify(stamp)};\n`;
writeFileSync(OUT_PATH, out, 'utf8');
// eslint-disable-next-line no-console
+123 -15
View File
@@ -19,7 +19,10 @@ import {
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { acquireCollabSession } from "../lib/collab-session.js";
import {
acquireCollabSession,
isCollabAuthFailedError,
} from "../lib/collab-session.js";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import { getCollabToken, performLogin } from "../lib/auth-utils.js";
import { formatDocmostAxiosError } from "./errors.js";
@@ -159,6 +162,43 @@ export abstract class DocmostClientContext {
// bypassed on a forced refresh (the 401/403 reauth path). null = no token yet.
protected collabTokenCache: { token: string; mintedAt: number } | null = null;
// #487: an OPTIONAL abort signal the in-app tool host sets before each tool
// call (a composite of the turn's Stop signal + a per-call wall-clock cap). It
// is checked at safe-points BETWEEN the sequential HTTP calls of a paginated
// read (paginateAll) and just before the atomic collab commit of a write (the
// mutatePage/replacePage/mutateLiveContentUnlocked seams), so a Stop / cap
// stops the NEXT network call from STARTING. An already-started single call may
// still land — a documented limitation (#487).
//
// SINGLE-WRITER by phase-1 assumption: exactly one DocmostClient is built per
// turn and shared by every tool call; the host sets this per call and does NOT
// restore the prior value on unwind (set-and-leave) — a fresh client per turn
// plus overwrite-by-the-next-call keeps it correct, and leaving a settled
// call's signal in place is what makes a discarded race-loser throw on its
// next safe-point. If the model emits PARALLEL in-app
// tool calls they share this one field, so the per-call CAP of one call is not
// guaranteed to bound another's in-flight pagination — but every composite the
// host sets carries the SAME turn Stop signal, so a Stop still aborts whichever
// signal is current. #487.
protected toolAbortSignal: AbortSignal | null = null;
/**
* #487: set (or clear with null) the in-app tool abort signal governing the
* NEXT client call's safe-points. The host wraps each in-app tool call: it sets
* the composite (Stop + per-call cap) here before invoking the tool and leaves
* it in place afterwards (set-and-leave, NOT restored) the next call
* overwrites it, and a fresh client is built per turn. Public so the
* server-side tool wrapper can reach it; harmless (a no-op) when never set.
*/
public setToolAbortSignal(signal: AbortSignal | null): void {
this.toolAbortSignal = signal;
}
/** #487: the abort signal currently governing this client's safe-points. */
public getToolAbortSignal(): AbortSignal | null {
return this.toolAbortSignal;
}
// Two construction forms:
// - new DocmostClient(config) // discriminated union (current)
// - new DocmostClient(baseURL, email, password) // legacy positional creds
@@ -484,6 +524,37 @@ export abstract class DocmostClientContext {
);
}
/**
* Run a collab write and, on a Hocuspocus HANDSHAKE auth failure, self-heal
* once (#486). Symmetric to the HTTP-401 path in getCollabTokenWithReauth: the
* REST interceptor and login() already drop the cached collab token on a 401/
* 403, but a rejected WEBSOCKET handshake left the stale token in the cache, so
* every subsequent mutation kept re-presenting the same bad token for up to the
* collab-token TTL (minutes) with no self-heal. Here, when the write rejects
* with the tagged collab-auth error, we invalidate the cached token and retry
* the write EXACTLY once with a force-refreshed token. Not a loop: a second
* failure (or any non-auth error) propagates unchanged.
*
* `write` receives the token to use, so the retry can hand it a genuinely fresh
* one rather than re-running with the same stale string.
*/
protected async writeWithCollabAuthRetry<T>(
collabToken: string,
write: (token: string) => Promise<T>,
): Promise<T> {
try {
return await write(collabToken);
} catch (e) {
if (!isCollabAuthFailedError(e)) throw e;
// The WS handshake rejected our token: drop it from the cache so it can't
// be reused for the rest of the TTL, mint a fresh one (forceRefresh bypasses
// the cache and re-invokes the provider/login), and retry the write once.
this.collabTokenCache = null;
const fresh = await this.getCollabTokenWithReauth(true);
return await write(fresh);
}
}
/**
* Connect to the collaboration websocket, read the live doc, apply
* `transform`, write the result, and wait for the server to persist it
@@ -518,19 +589,28 @@ export abstract class DocmostClientContext {
// unsyncedChanges/connectionLost ack logic live in CollabSession.mutate,
// preserved verbatim from the old inline machine (incl. the #152 structural
// diff that keeps a live editor's cursor anchored).
const session = await acquireCollabSession(pageId, collabToken, this.apiUrl, {
// Only the actual 25s collab connect timeout emits this — the connect-vs-
// unload signal; the other failure paths must NOT emit it.
onConnectTimeout: () =>
this.onMetricFn?.("collab_connect_timeouts_total", 1),
// Wrap in the collab-auth self-heal (#486): a rejected WS handshake drops the
// cached collab token and retries once with a fresh one (the retry passes the
// refreshed token down to acquireCollabSession via `token`).
return this.writeWithCollabAuthRetry(collabToken, async (token) => {
const session = await acquireCollabSession(pageId, token, this.apiUrl, {
// Only the actual 25s collab connect timeout emits this — the connect-vs-
// unload signal; the other failure paths must NOT emit it.
onConnectTimeout: () =>
this.onMetricFn?.("collab_connect_timeouts_total", 1),
});
try {
// #487 PRE-COMMIT safe-point (reentrant twin of mutatePageContent): a
// Stop/cap after acquiring the session but before the atomic write skips
// this commit. Same limitation applies (stops the NEXT commit only).
this.toolAbortSignal?.throwIfAborted();
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh.
session.destroy("mutate failed");
throw e;
}
});
try {
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh.
session.destroy("mutate failed");
throw e;
}
}
/**
@@ -555,6 +635,11 @@ export abstract class DocmostClientContext {
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
// #487 safe-point: a Stop (or the in-app tool per-call cap) that fires
// BETWEEN sequential page fetches must stop the NEXT request from starting
// — a read tool that would otherwise paginate for minutes is interrupted
// here. throwIfAborted() rejects with the signal's reason.
this.toolAbortSignal?.throwIfAborted();
const payload: Record<string, any> = {
...basePayload,
limit: clampedLimit,
@@ -659,7 +744,19 @@ export abstract class DocmostClientContext {
transform: (doc: any) => any,
): Promise<{ doc?: any; verify?: any }> {
const pageUuid = await this.resolvePageId(pageId);
return mutatePageContent(pageUuid, collabToken, apiUrl, transform);
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
// #487: thread the in-app tool signal to mutatePageContent's pre-commit
// safe-point so a Stop/cap during the connect/lock window skips the write.
mutatePageContent(
pageUuid,
token,
apiUrl,
transform,
this.toolAbortSignal ?? undefined,
),
);
}
/**
@@ -679,7 +776,18 @@ export abstract class DocmostClientContext {
apiUrl: string,
): Promise<{ doc?: any; verify?: any }> {
const pageUuid = await this.resolvePageId(pageId);
return replacePageContent(pageUuid, doc, collabToken, apiUrl);
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
// #487: same pre-commit safe-point as mutatePage, for full-document writes.
replacePageContent(
pageUuid,
doc,
token,
apiUrl,
this.toolAbortSignal ?? undefined,
),
);
}
/**
+7 -2
View File
@@ -120,8 +120,13 @@ export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base
"listPages: tree mode requires a spaceId (a page tree is scoped to one space). Pass spaceId, or omit tree to get the recent-pages list.",
);
}
const { pages } = await this.enumerateSpacePages(spaceId);
return buildPageTree(pages);
// #486: propagate `truncated` (same pattern as check_new_comments). The old
// code dropped it, so a caller handed an INCOMPLETE tree (the stdio-fallback
// BFS hit its node cap) had no way to know pages were missing. Return the
// tree alongside the flag; the primary /pages/tree path is uncapped so this
// is false there.
const { pages, truncated } = await this.enumerateSpacePages(spaceId);
return { tree: buildPageTree(pages), truncated };
}
const clampedLimit = Math.max(1, Math.min(100, limit));
-1
View File
@@ -4,7 +4,6 @@ import { readFileSync } from "fs";
import { fileURLToPath } from "url";
import { dirname, join } from "path";
import { DocmostClient, DocmostMcpConfig } from "./client.js";
import { parseNodeArg } from "@docmost/prosemirror-markdown";
import { searchShapes } from "./lib/drawio-shapes.js";
import { getGuideSection } from "./lib/drawio-guide.js";
import { SHARED_TOOL_SPECS, SharedToolSpec } from "./tool-specs.js";
+26 -4
View File
@@ -37,6 +37,25 @@ const CONNECT_TIMEOUT_MS = 25000;
/** Time we wait for the server to acknowledge our write before giving up. */
const PERSIST_TIMEOUT_MS = 20000;
/**
* Marker property set on the Error thrown when the Hocuspocus handshake REJECTS
* our collab token (onAuthenticationFailed). The client wraps content writes so
* that on this specific failure it invalidates its cached collab token and
* retries once with a fresh one symmetric to the HTTP-401 reauth path (#486).
* A plain message-match would be brittle; a tagged property is unambiguous and
* survives teardown (which rejects pending ops with this SAME error object).
*/
const COLLAB_AUTH_FAILED_MARKER = "collabAuthFailed";
/** True when `e` is the tagged collab-WS auth-failure error (see marker above). */
export function isCollabAuthFailedError(e: unknown): boolean {
return !!(
e &&
typeof e === "object" &&
(e as Record<string, unknown>)[COLLAB_AUTH_FAILED_MARKER] === true
);
}
/**
* Tunables, read fresh from the environment on every acquire so tests (and a
* live rollback) can change them without reloading the module. Mirrors how
@@ -302,10 +321,13 @@ export class CollabSession {
this.openResolve?.();
},
onAuthenticationFailed: () => {
this.teardown(
new Error("Authentication failed for collaboration connection"),
true,
);
// Tag the error so the client can tell a REJECTED collab token apart
// from a generic disconnect and invalidate + refresh it (#486).
const err = new Error(
"Authentication failed for collaboration connection",
) as Error & { [COLLAB_AUTH_FAILED_MARKER]?: boolean };
err[COLLAB_AUTH_FAILED_MARKER] = true;
this.teardown(err, true);
},
});
});
+16
View File
@@ -254,6 +254,12 @@ export async function mutatePageContent(
collabToken: string,
baseUrl: string,
transform: (liveDoc: any) => any | null,
// #487: optional abort signal carrying the turn's Stop + the in-app tool
// per-call cap. Checked as the PRE-COMMIT safe-point below (after the session
// is acquired, immediately before the atomic read->write), so a Stop that
// arrives during the connect/lock window stops THIS write from landing. See the
// limitation note at the check.
signal?: AbortSignal,
): Promise<MutationResult> {
return withPageLock(pageId, async () => {
if (process.env.DEBUG) {
@@ -266,6 +272,13 @@ export async function mutatePageContent(
const session = await acquireCollabSession(pageId, collabToken, baseUrl);
try {
// #487 PRE-COMMIT safe-point: if the turn was Stopped (or the in-app tool
// per-call cap fired) after we acquired the collab session but before the
// atomic write, throw NOW so this commit never runs. KNOWN LIMITATION
// (#487): this only stops THIS commit — a write tool that already committed
// an EARLIER call this turn leaves that op applied. Cancel guarantees "no
// NEW commit starts", NOT "the write didn't land".
signal?.throwIfAborted();
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh (this
@@ -291,6 +304,8 @@ export async function replacePageContent(
prosemirrorDoc: any,
collabToken: string,
baseUrl: string,
// #487: threaded straight to mutatePageContent's pre-commit safe-point.
signal?: AbortSignal,
): Promise<MutationResult> {
// Fail fast on a bad document instead of deferring the failure into the
// collaboration write (where TiptapTransformer.toYdoc(undefined) used to
@@ -307,6 +322,7 @@ export async function replacePageContent(
collabToken,
baseUrl,
() => prosemirrorDoc,
signal,
);
}
+74 -31
View File
@@ -10,7 +10,7 @@
// exactly mxGraph's convention for a child of a container, so they map across
// directly. Container sizes are computed by ELK; leaf sizes are preserved.
import ELK from "elkjs/lib/elk.bundled.js";
import { Worker } from "node:worker_threads";
import { JSDOM } from "jsdom";
import { normalizeInput, parseCells, type DrawioCell } from "./drawio-xml.js";
@@ -18,22 +18,33 @@ import { normalizeInput, parseCells, type DrawioCell } from "./drawio-xml.js";
const DEFAULT_W = 140;
const DEFAULT_H = 60;
// DoS bounds for the in-process ELK layout. The mxGraph XML is LLM-supplied
// (layout:"elk" in drawioCreate/drawioUpdate) and elkjs runs synchronously on
// the MCP server's event loop, so an unbounded graph would block it for
// seconds-to-minutes. A ~1MB XML (well under the stage-1 16MB cap) can carry
// thousands of nodes. We cap the graph size and race the layout against a
// wall-clock timeout; on either bound we fall back to the ORIGINAL model, the
// same best-effort contract the catch already honours.
// DoS bounds for the ELK layout. The mxGraph XML is LLM-supplied (layout:"elk"
// in drawioCreate/drawioUpdate). elkjs' layout() returns a Promise but runs the
// crossing-minimisation SYNCHRONOUSLY — it blocks whatever thread it runs on for
// the whole pass. A ~1MB XML (well under the stage-1 16MB cap) can carry
// thousands of nodes. We (a) cap the graph size before ever calling ELK and
// (b) run the layout in a WORKER THREAD so the main event loop stays free, with
// the wall-clock timeout enforced by terminating that worker. On either bound we
// fall back to the ORIGINAL model, the same best-effort contract the catch honours.
// - 500 nodes lays out in well under a second; beyond that ELK cost climbs
// steeply, so refuse and leave the (already-valid) model untouched.
// - Edges dominate the layered-crossing cost, so allow a bit more headroom
// (1000) than nodes but still bound them.
// - 5s is generous for any graph within the caps yet short enough that a
// pathological input can never wedge the server.
// - The timeout is a HARD kill of the worker thread — the only way to interrupt
// synchronous JS. The in-process setTimeout race we used before was an
// illusion: the timer could never fire while the SAME thread was blocked
// inside elkjs, so it "protected" nothing. Now the timer runs on the main
// thread while ELK runs on the worker, so it can actually fire and terminate.
const ELK_MAX_NODES = 500;
const ELK_MAX_EDGES = 1000;
const ELK_TIMEOUT_MS = 5000;
// Wall-clock ceiling for a single layout pass. Overridable for tests (a tiny
// value forces the terminate-on-timeout path deterministically); a non-positive
// or unparseable override falls back to the default.
const ELK_TIMEOUT_DEFAULT_MS = 5000;
function resolveElkTimeoutMs(): number {
const raw = Number(process.env.DRAWIO_ELK_TIMEOUT_MS);
return Number.isFinite(raw) && raw > 0 ? Math.floor(raw) : ELK_TIMEOUT_DEFAULT_MS;
}
// Spacing is set >=150px on purpose so an ELK layout never trips the linter's
// "gap between adjacent shapes < 150px" quality warning (acceptance #3).
@@ -78,13 +89,57 @@ interface ElkGraph extends ElkNode {
edges?: ElkEdge[];
}
/**
* Run one ELK layered layout on a worker thread and resolve with the laid-out
* graph. The timeout is enforced by `worker.terminate()` a HARD kill, which is
* the only way to interrupt elkjs' synchronous crossing-minimisation once it has
* started. Rejects on timeout, worker error, or an early exit; the caller treats
* any rejection as "keep the original model" (best-effort layout).
*/
function layoutInWorker(graph: ElkGraph, timeoutMs: number): Promise<ElkGraph> {
return new Promise((resolve, reject) => {
const worker = new Worker(
new URL("./drawio-layout.worker.js", import.meta.url),
{ workerData: { graph } },
);
let settled = false;
const finish = (fn: () => void) => {
if (settled) return;
settled = true;
clearTimeout(timer);
// Always tear the worker down: on the happy path so it does not linger,
// on timeout so the blocked synchronous ELK run is actually interrupted.
void worker.terminate();
fn();
};
const timer = setTimeout(
() => finish(() => reject(new Error("ELK layout timed out"))),
timeoutMs,
);
worker.once("message", (msg: { ok?: boolean; laid?: ElkGraph; error?: string }) => {
finish(() =>
msg?.ok
? resolve(msg.laid as ElkGraph)
: reject(new Error(msg?.error ?? "ELK layout failed")),
);
});
worker.once("error", (err) => finish(() => reject(err)));
worker.once("exit", (code) => {
// A clean exit after we already settled is normal (terminate()); only an
// unexpected early exit while still pending is a failure.
if (settled) return;
finish(() => reject(new Error(`ELK worker exited early (code ${code})`)));
});
});
}
/**
* Apply an ELK layered layout to a drawio input and return a full mxGraphModel
* string with rewritten geometry. Accepts the same three input forms as
* drawioCreate (a bare model, an <mxfile>, or a <mxCell> list). Async because
* elkjs' layout() is promise-based. On any layout failure the ORIGINAL
* (normalized) model is returned unchanged layout is best-effort polish, never
* a reason to fail the write.
* the layout runs on a worker thread. On any layout failure (including a
* terminate-on-timeout) the ORIGINAL (normalized) model is returned unchanged
* layout is best-effort polish, never a reason to fail the write.
*/
export async function applyElkLayout(inputXml: string): Promise<string> {
const modelXml = normalizeInput(inputXml);
@@ -150,26 +205,14 @@ export async function applyElkLayout(inputXml: string): Promise<string> {
};
let laid: ElkGraph;
let timer: ReturnType<typeof setTimeout> | undefined;
try {
// elkjs ships a CJS default export whose interop shape varies across
// module systems; resolve the real constructor at runtime, then cast (the
// runtime call is verified — see the layout unit test).
const Ctor: any = (ELK as any).default ?? ELK;
const elk = new Ctor();
// Race the layout against a wall-clock timeout so a graph that is under the
// node/edge caps but still pathologically slow can never wedge the server.
const timeout = new Promise<never>((_, reject) => {
timer = setTimeout(
() => reject(new Error("ELK layout timed out")),
ELK_TIMEOUT_MS,
);
});
laid = (await Promise.race([elk.layout(graph as any), timeout])) as ElkGraph;
// Run the (synchronous-under-the-hood) ELK pass on a worker thread so the
// main event loop is never blocked, and enforce the wall-clock ceiling by
// terminating that worker on timeout. A graph under the node/edge caps but
// still pathologically slow is hard-killed instead of wedging anything.
laid = await layoutInWorker(graph, resolveElkTimeoutMs());
} catch {
return modelXml; // best-effort: keep the model as-is on timeout or ELK failure
} finally {
if (timer) clearTimeout(timer);
}
// Collect computed geometry per node id (coords are parent-relative already).
@@ -0,0 +1,36 @@
// Worker-thread entry for the ELK layered layout (issue #486, commit 1).
//
// elkjs' layout() returns a Promise but runs the actual crossing-minimisation
// SYNCHRONOUSLY — it blocks whatever thread it runs on for the whole pass. On
// the in-app MCP host that thread used to be the main NestJS event loop, so a
// pathological graph at the node/edge cap could wedge ALL HTTP/SSE/loopback
// traffic while it churned. Running it HERE, on a dedicated worker thread, keeps
// the main loop free; the parent enforces the wall-clock timeout by calling
// `worker.terminate()` — the only way to interrupt synchronous JS — since the
// in-process `setTimeout` race the parent used before could never fire while the
// same thread was blocked inside elkjs.
import { parentPort, workerData } from "node:worker_threads";
import ELK from "elkjs/lib/elk.bundled.js";
interface WorkerInput {
graph: unknown;
}
const { graph } = (workerData ?? {}) as WorkerInput;
// elkjs ships a CJS default export whose interop shape varies across module
// systems; resolve the real constructor at runtime (same as the parent did).
const Ctor: any = (ELK as any).default ?? ELK;
const elk = new Ctor();
elk
.layout(graph as any)
.then((laid: unknown) => {
parentPort?.postMessage({ ok: true, laid });
})
.catch((err: unknown) => {
parentPort?.postMessage({
ok: false,
error: err instanceof Error ? err.message : String(err),
});
});
@@ -0,0 +1,137 @@
// Unit tests for the collab-token reset on a Hocuspocus WS auth failure (#486).
//
// Before this fix the cached collab token (#435) was dropped ONLY on an HTTP
// 401/403 (the REST interceptor + login()); a rejected collab-WEBSOCKET handshake
// left the stale token in the cache, so every subsequent mutation re-presented
// the SAME bad token for up to the collab-token TTL (minutes) with no self-heal.
//
// The fix wraps collab writes in `writeWithCollabAuthRetry`: when the write
// rejects with the tagged collab-auth error (collab-session.ts's
// onAuthenticationFailed), it invalidates the cached token and retries the write
// ONCE with a force-refreshed token — symmetric to the HTTP-401 path.
//
// writeWithCollabAuthRetry / getCollabTokenWithReauth are protected in TS but
// plain methods on the compiled build, so the tests call them directly (same
// convention as collab-token-cache.test.mjs).
import { test, afterEach } from "node:test";
import assert from "node:assert/strict";
import { DocmostClient } from "../../build/client.js";
const ENV_KEY = "MCP_COLLAB_TOKEN_TTL_MS";
afterEach(() => {
delete process.env[ENV_KEY];
});
// A counting provider that returns a distinct token each call so a cached
// (reused) token is visibly the SAME string while a fresh mint is different.
function countingProvider() {
let n = 0;
const fn = async () => {
n++;
return `provider-token-${n}`;
};
return {
fn,
get calls() {
return n;
},
};
}
// The tagged error collab-session.ts throws on a rejected WS handshake.
function collabAuthError() {
const err = new Error("Authentication failed for collaboration connection");
err.collabAuthFailed = true;
return err;
}
test("a WS auth failure clears the cached token and retries the write with a FRESH one (#486)", async () => {
process.env[ENV_KEY] = "300000"; // 5 min: the cache is warm across the burst.
const p = countingProvider();
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
getCollabToken: p.fn,
});
// Warm the cache the way a real write would (mints provider-token-1).
const initial = await client.getCollabTokenWithReauth();
assert.equal(initial, "provider-token-1");
assert.equal(p.calls, 1);
const tokensSeen = [];
const write = async (token) => {
tokensSeen.push(token);
// The FIRST attempt (with the stale cached token) fails the WS handshake;
// the retry (with a fresh token) succeeds.
if (tokensSeen.length === 1) throw collabAuthError();
return `written-with:${token}`;
};
const result = await client.writeWithCollabAuthRetry(initial, write);
assert.equal(tokensSeen.length, 2, "write attempted exactly twice (one retry)");
assert.equal(tokensSeen[0], "provider-token-1", "first attempt used the stale token");
assert.equal(
tokensSeen[1],
"provider-token-2",
"retry used a FRESH force-refreshed token, not the stale cached one",
);
assert.equal(result, "written-with:provider-token-2", "the retry's result wins");
assert.equal(p.calls, 2, "exactly one extra mint for the retry — no loop");
// The cache now holds the fresh token, so a subsequent op reuses it (proving
// the stale token was evicted and the fresh one cached, not re-minted).
const next = await client.getCollabTokenWithReauth();
assert.equal(next, "provider-token-2", "the fresh token replaced the stale cache");
assert.equal(p.calls, 2, "served from cache — provider not re-invoked");
});
test("a successful write is NOT retried and mints nothing extra", async () => {
process.env[ENV_KEY] = "300000";
const p = countingProvider();
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
getCollabToken: p.fn,
});
const initial = await client.getCollabTokenWithReauth(); // provider-token-1
let attempts = 0;
const result = await client.writeWithCollabAuthRetry(initial, async (token) => {
attempts++;
return `ok:${token}`;
});
assert.equal(attempts, 1, "no retry on success");
assert.equal(result, "ok:provider-token-1");
assert.equal(p.calls, 1, "no extra mint");
});
test("a NON-auth write error propagates unchanged (no reset, no retry)", async () => {
process.env[ENV_KEY] = "300000";
const p = countingProvider();
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
getCollabToken: p.fn,
});
const initial = await client.getCollabTokenWithReauth(); // provider-token-1
let attempts = 0;
await assert.rejects(
client.writeWithCollabAuthRetry(initial, async () => {
attempts++;
throw new Error("collab connection closed before persist"); // NOT tagged.
}),
/closed before persist/,
);
assert.equal(attempts, 1, "a non-auth error is not retried");
assert.equal(p.calls, 1, "the cache is untouched -> no fresh mint");
// Cache still holds the original token (was never invalidated).
const still = await client.getCollabTokenWithReauth();
assert.equal(still, "provider-token-1");
assert.equal(p.calls, 1);
});
@@ -0,0 +1,55 @@
// Unit test: listPages tree mode must propagate the `truncated` flag (#486).
//
// enumerateSpacePages returns { pages, truncated } — truncated is true ONLY when
// the stdio-fallback BFS hit its node cap (the primary /pages/tree path is
// uncapped). The old tree-mode listPages destructured only `pages` and returned a
// bare tree, dropping `truncated`, so a caller handed an INCOMPLETE tree had no
// way to know pages were missing. The fix returns { tree, truncated } (same
// pattern check_new_comments uses).
//
// Reaching the real cap (MAX_NODES = 10000) in a mock is impractical, so we stub
// enumerateSpacePages directly to assert the flag is threaded through verbatim.
import { test } from "node:test";
import assert from "node:assert/strict";
import { DocmostClient } from "../../build/client.js";
function stubClient() {
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
});
// No network: the tree path only calls ensureAuthenticated + enumerateSpacePages.
client.ensureAuthenticated = async () => {};
return client;
}
const onePage = [{ id: "r1", title: "Root", parentPageId: null }];
test("tree mode carries truncated:true when the enumeration truncated (#486)", async () => {
const client = stubClient();
client.enumerateSpacePages = async () => ({ pages: onePage, truncated: true });
const res = await client.listPages("space-1", 50, true);
assert.equal(res.truncated, true, "the truncated flag is threaded through");
assert.ok(Array.isArray(res.tree), "the built tree rides alongside the flag");
assert.equal(res.tree[0].id, "r1");
});
test("tree mode carries truncated:false for a complete enumeration", async () => {
const client = stubClient();
client.enumerateSpacePages = async () => ({ pages: onePage, truncated: false });
const res = await client.listPages("space-1", 50, true);
assert.equal(res.truncated, false);
assert.equal(res.tree[0].id, "r1");
});
test("tree mode still requires a spaceId", async () => {
const client = stubClient();
await assert.rejects(
client.listPages(undefined, 50, true),
/tree mode requires a spaceId/,
);
});
@@ -0,0 +1,143 @@
// #487 commit 1 — the in-app tool cancellation safe-point inside paginateAll.
//
// The in-app tool host sets a composite abort signal on the client
// (setToolAbortSignal) before each tool call; paginateAll checks it at a
// safe-point BEFORE every sequential page fetch, so a Stop that lands mid-read
// stops the NEXT HTTP request from STARTING (a read tool can no longer paginate
// for minutes past a Stop). This pins the HONEST observable property against the
// REAL client + a real HTTP server: "after Stop, no NEW request starts".
import { test, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { DocmostClient } from "../../build/client.js";
function readBody(req) {
return new Promise((resolve) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => resolve(raw));
});
}
function sendJson(res, status, obj, extra = {}) {
res.writeHead(status, { "Content-Type": "application/json", ...extra });
res.end(JSON.stringify(obj));
}
const openServers = [];
async function spawn(handler) {
const server = await new Promise((resolve) => {
const s = http.createServer(handler);
s.listen(0, "127.0.0.1", () => resolve(s));
});
openServers.push(server);
const { port } = server.address();
return { baseURL: `http://127.0.0.1:${port}/api` };
}
after(async () => {
await Promise.all(openServers.map((s) => new Promise((r) => s.close(r))));
});
function handleLogin(req, res) {
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
});
return true;
}
return false;
}
// A Stop that lands DURING pagination: the server aborts the client signal as it
// serves page 1 (more pages remain). The loop's next safe-point must throw before
// the page-2 request is sent.
test("paginateAll stops the NEXT request when the signal aborts mid-pagination", async () => {
let requests = 0;
const ac = new AbortController();
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/spaces") {
requests++;
// Simulate a user Stop that lands while page 1 is in flight.
if (requests === 1) ac.abort(new Error("user stop"));
sendJson(res, 200, {
success: true,
data: {
items: [{ id: `p${requests}` }],
meta: { hasNextPage: true, nextCursor: `c${requests}` },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
client.setToolAbortSignal(ac.signal);
await assert.rejects(
() => client.paginateAll("/spaces", {}),
/user stop/,
"the aborted safe-point rejects with the signal's reason",
);
assert.equal(requests, 1, "page 2 never started after the Stop");
});
// A Stop that is already in effect before the read starts: zero requests fire.
test("paginateAll starts no request when the signal is already aborted", async () => {
let requests = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/spaces") {
requests++;
sendJson(res, 200, {
success: true,
data: { items: [], meta: { hasNextPage: false, nextCursor: null } },
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// Warm the auth so ensureAuthenticated does not itself POST after the abort.
await client.ensureAuthenticated();
const ac = new AbortController();
ac.abort(new Error("already stopped"));
client.setToolAbortSignal(ac.signal);
await assert.rejects(() => client.paginateAll("/spaces", {}), /already stopped/);
assert.equal(requests, 0, "no /spaces request started once already aborted");
});
// Without a tool signal (default), pagination is unaffected — the safe-point is a
// pure no-op, so pre-#487 behaviour is byte-identical.
test("paginateAll is unaffected when no tool signal is set", async () => {
let requests = 0;
const PAGES = {
"": { items: [{ id: "a" }], nextCursor: "c1" },
c1: { items: [{ id: "b" }], nextCursor: null },
};
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/spaces") {
requests++;
const body = JSON.parse(raw || "{}");
const page = PAGES[body.cursor ?? ""] ?? { items: [], nextCursor: null };
sendJson(res, 200, {
success: true,
data: {
items: page.items,
meta: { hasNextPage: page.nextCursor != null, nextCursor: page.nextCursor },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.paginateAll("/spaces", {});
assert.equal(requests, 2, "both pages fetched with no signal set");
assert.deepEqual(all.map((p) => p.id), ["a", "b"]);
});
@@ -184,12 +184,14 @@ test("enumerateSpacePages (via listPages tree) uses one /pages/tree request", as
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// listPages tree:true -> enumerateSpacePages(spaceId) -> buildPageTree.
const tree = await client.listPages("space-1", 50, true);
// listPages tree:true -> enumerateSpacePages(spaceId) -> { tree, truncated }.
const { tree, truncated } = await client.listPages("space-1", 50, true);
assert.equal(treeRequests, 1, "exactly one /pages/tree request for the space");
assert.equal(sidebarRequests, 0, "no per-node sidebar BFS requests");
assert.deepEqual(treeBody, { spaceId: "space-1" }, "space scope posts spaceId only");
// The uncapped /pages/tree path is never truncated (#486).
assert.equal(truncated, false, "primary /pages/tree path is not truncated");
// buildPageTree nests c1 under r1; two roots at the top level.
assert.equal(tree.length, 2, "two root nodes");
const r1 = tree.find((n) => n.id === "r1");
@@ -249,7 +251,7 @@ test("enumerateSpacePages falls back to the cursor BFS on /pages/tree 404", asyn
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const tree = await client.listPages("space-1", 50, true);
const { tree, truncated } = await client.listPages("space-1", 50, true);
assert.ok(treeRequests >= 1, "the tree endpoint was attempted first");
assert.deepEqual(
@@ -257,6 +259,8 @@ test("enumerateSpacePages falls back to the cursor BFS on /pages/tree 404", asyn
["<root>", "r1"],
"fell back to the sidebar BFS: roots then the root's children",
);
// Small fallback walk well under the node cap -> not truncated (#486).
assert.equal(truncated, false, "fallback BFS below the cap is not truncated");
assert.equal(tree.length, 1, "one root in the built tree");
assert.equal(tree[0].children[0].id, "c1", "leaf nested via the BFS");
});
@@ -101,6 +101,88 @@ test("DoS guard: a graph over the node cap is returned unchanged, quickly", asyn
assert.ok(dt < 2000, `cap path should be fast, took ${dt}ms`);
});
/** Build a layered DAG near the caps: `n` vertices, up to ~2 edges each into the
* next layer of `layerSize`. Used as a real worst-case graph for the benchmark. */
function layeredGraph(n, layerSize) {
let cells = "";
for (let i = 2; i < 2 + n; i++) {
cells +=
`<mxCell id="${i}" value="N${i}" style="rounded=1;html=1;" vertex="1" parent="1">` +
`<mxGeometry x="10" y="10" width="120" height="60" as="geometry"/></mxCell>`;
}
let ei = 0;
for (let i = 2; i < 2 + n; i++) {
for (const off of [layerSize, layerSize + 1]) {
const t = i + off;
if (t < 2 + n) cells += `<mxCell id="e${ei++}" edge="1" parent="1" source="${i}" target="${t}"><mxGeometry relative="1" as="geometry"/></mxCell>`;
}
}
return (
'<mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/>' +
cells +
"</root></mxGraphModel>"
);
}
test("terminate-on-timeout: a layout that exceeds the wall-clock ceiling is hard-killed and the original model is returned (#486)", async () => {
// A 1ms ceiling fires before the worker can even finish loading elkjs, so the
// parent must terminate() the worker and fall back to the ORIGINAL model. On
// the OLD in-process race this timer could never fire while the SAME thread was
// blocked inside elkjs — the fallback path was unreachable; here it works.
const prev = process.env.DRAWIO_ELK_TIMEOUT_MS;
process.env.DRAWIO_ELK_TIMEOUT_MS = "1";
try {
const model = layeredGraph(400, 20);
const t0 = Date.now();
const laid = await applyElkLayout(model);
const dt = Date.now() - t0;
// Original geometry is preserved verbatim: every vertex is still stacked at
// (10,10), proving NO ELK coordinates were applied (the pass was killed).
const verts = parseCells(laid).filter((c) => c.vertex);
assert.equal(verts.length, 400, "all vertices survived the fallback");
for (const v of verts) {
assert.equal(v.geometry.x, 10, "x untouched -> layout was terminated");
assert.equal(v.geometry.y, 10, "y untouched -> layout was terminated");
}
// The kill is prompt: terminate() returns the call well under the natural
// layout time for a 400-node graph.
assert.ok(dt < 2000, `terminate path should be prompt, took ${dt}ms`);
} finally {
if (prev === undefined) delete process.env.DRAWIO_ELK_TIMEOUT_MS;
else process.env.DRAWIO_ELK_TIMEOUT_MS = prev;
}
});
test("benchmark guard: a worst-case graph AT the cap lays out without wedging the main event loop (#486)", async () => {
// ~500 nodes / ~1000 edges — a real worst case at the node/edge caps. The
// layout runs on a WORKER thread, so the MAIN event loop must stay responsive
// throughout: a timer scheduled on the main thread keeps firing while ELK
// churns. On the OLD synchronous-on-main-thread code this counter would be
// pinned at 0 for the whole layout (event loop wedged) — exactly the prod fire.
const model = layeredGraph(500, 20);
let mainLoopTicks = 0;
const iv = setInterval(() => {
mainLoopTicks++;
}, 2);
const t0 = Date.now();
const laid = await applyElkLayout(model);
const dt = Date.now() - t0;
clearInterval(iv);
assert.ok(
mainLoopTicks > 0,
"main event loop must stay responsive while ELK runs on the worker",
);
// Benchmark guard: the worst-case graph actually LAYS OUT within the default
// ceiling (it did not fall back). At least one vertex moved off the stack.
const verts = parseCells(laid).filter((c) => c.vertex);
assert.equal(verts.length, 500, "all vertices survived");
const moved = verts.some((v) => v.geometry.x !== 10 || v.geometry.y !== 10);
assert.ok(moved, "layout was applied (did not time out / fall back)");
// Sanity ceiling well under the 5s wall-clock timeout.
assert.ok(dt < 5000, `worst-case layout should be under the ceiling, took ${dt}ms`);
});
test("layout is best-effort: an empty/degenerate model is returned intact", async () => {
const model =
'<mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/></root></mxGraphModel>';
+189 -70
View File
@@ -1,101 +1,220 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import {
mkdtempSync,
mkdirSync,
writeFileSync,
rmSync,
readdirSync,
statSync,
readFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
import { dirname, join, relative, sep } from "node:path";
import { createHash } from "node:crypto";
import { computeRegistryStamp } from "../../scripts/gen-registry-stamp.mjs";
import { REGISTRY_STAMP } from "../../build/index.js";
// Guard tests for the build/src-skew stamp (issue #447). The codegen script
// exports `computeRegistryStamp(sourceText)` — a sha256 over normalized source
// text (CRLF->LF, single trailing newline stripped). The in-app loader
// (apps/server/.../docmost-client.loader.ts) DUPLICATES that normalize+sha256 to
// recompute the stamp from src and refuse a stale build. These tests pin the
// algorithm's behaviour AND assert the built stamp matches the current src, so a
// stale generated file OR a normalize divergence reddens.
// Guard tests for the build/src-skew stamp (issues #447/#486). The codegen script
// exports `computeRegistryStamp(srcDir)` — a sha256 over the WHOLE src/ tree
// (every src/**/*.ts EXCEPT *.generated.ts), each file folded in as its
// POSIX-relative path + its normalized content (CRLF->LF, single trailing newline
// stripped). Hashing the whole tree (not just tool-specs.ts) is #486: an edit to
// client.ts / a client/* module without a rebuild must ALSO redden. The in-app
// loader (apps/server/.../docmost-client.loader.ts) DUPLICATES this enumerate+
// normalize+sha256 to refuse a stale build. These tests pin the algorithm and
// assert the built stamp matches the current src.
const __dirname = dirname(fileURLToPath(import.meta.url));
const TOOL_SPECS_PATH = join(__dirname, "..", "..", "src", "tool-specs.ts");
const SRC_DIR = join(__dirname, "..", "..", "src");
test("computeRegistryStamp is deterministic: same input -> same hash", () => {
const input = "export const X = 1;\nexport const Y = 2;\n";
assert.equal(computeRegistryStamp(input), computeRegistryStamp(input));
// Build a throwaway src/ tree from a { relPath: content } map and return its dir.
function makeSrcTree(files) {
const root = mkdtempSync(join(tmpdir(), "mcp-stamp-tree-"));
const src = join(root, "src");
for (const [rel, content] of Object.entries(files)) {
const full = join(src, rel);
mkdirSync(dirname(full), { recursive: true });
writeFileSync(full, content, "utf8");
}
return { src, cleanup: () => rmSync(root, { recursive: true, force: true }) };
}
test("computeRegistryStamp is deterministic: same tree -> same hash", () => {
const a = makeSrcTree({ "tool-specs.ts": "export const X = 1;\n" });
const b = makeSrcTree({ "tool-specs.ts": "export const X = 1;\n" });
try {
assert.equal(computeRegistryStamp(a.src), computeRegistryStamp(b.src));
} finally {
a.cleanup();
b.cleanup();
}
});
test("computeRegistryStamp returns a 64-char lowercase hex sha256", () => {
const stamp = computeRegistryStamp("anything");
assert.match(stamp, /^[0-9a-f]{64}$/);
const t = makeSrcTree({ "tool-specs.ts": "anything\n" });
try {
assert.match(computeRegistryStamp(t.src), /^[0-9a-f]{64}$/);
} finally {
t.cleanup();
}
});
test("normalizes CRLF vs LF: the same content hashes equal", () => {
const lf = "line1\nline2\nline3";
const crlf = "line1\r\nline2\r\nline3";
assert.equal(computeRegistryStamp(crlf), computeRegistryStamp(lf));
// #486 CORE: an edit to a NON-tool-specs source file (client.ts) must change the
// stamp. Under the old single-file (tool-specs.ts only) hash this edit was
// invisible and a stale build/ served the old client.ts silently.
test("editing client.ts (not tool-specs.ts) changes the stamp (#486)", () => {
const before = makeSrcTree({
"tool-specs.ts": "export const SPECS = 1;\n",
"client.ts": "export const impl = 'v1';\n",
});
const after = makeSrcTree({
"tool-specs.ts": "export const SPECS = 1;\n",
"client.ts": "export const impl = 'v2';\n",
});
try {
assert.notEqual(
computeRegistryStamp(before.src),
computeRegistryStamp(after.src),
"a client.ts edit with an unchanged tool-specs.ts must move the stamp",
);
} finally {
before.cleanup();
after.cleanup();
}
});
test("normalizes a trailing newline: with/without a final \\n hashes equal", () => {
const noTrailing = "alpha\nbeta";
const trailing = "alpha\nbeta\n";
assert.equal(computeRegistryStamp(trailing), computeRegistryStamp(noTrailing));
test("editing a nested client/* module changes the stamp", () => {
const before = makeSrcTree({
"tool-specs.ts": "x\n",
"client/read.ts": "export const READ = 1;\n",
});
const after = makeSrcTree({
"tool-specs.ts": "x\n",
"client/read.ts": "export const READ = 2;\n",
});
try {
assert.notEqual(
computeRegistryStamp(before.src),
computeRegistryStamp(after.src),
);
} finally {
before.cleanup();
after.cleanup();
}
});
test("a CRLF checkout WITH a trailing CRLF still hashes equal to bare LF", () => {
// A worst-case Windows checkout: CRLF line endings + a trailing CRLF. Both the
// \r\n->\n replace and the trailing-newline strip must apply for parity.
const bare = "alpha\nbeta";
const crlfTrailing = "alpha\r\nbeta\r\n";
assert.equal(
computeRegistryStamp(crlfTrailing),
computeRegistryStamp(bare),
);
// *.generated.ts is EXCLUDED (else the codegen's own output is a fixed-point
// cycle): adding/removing/changing it must not move the stamp.
test("*.generated.ts is excluded from the stamp", () => {
const without = makeSrcTree({ "tool-specs.ts": "x\n" });
const withGen = makeSrcTree({
"tool-specs.ts": "x\n",
"registry-stamp.generated.ts": 'export const REGISTRY_STAMP = "abc";\n',
});
try {
assert.equal(
computeRegistryStamp(without.src),
computeRegistryStamp(withGen.src),
"a *.generated.ts file must not affect the stamp",
);
} finally {
without.cleanup();
withGen.cleanup();
}
});
test("a real content change hashes differently", () => {
const before = "export const description = 'search a page';\n";
const after = "export const description = 'search a PAGE';\n";
assert.notEqual(computeRegistryStamp(before), computeRegistryStamp(after));
test("a CRLF checkout WITH trailing CRLF hashes equal to bare LF", () => {
const bare = makeSrcTree({ "tool-specs.ts": "alpha\nbeta" });
const crlfTrailing = makeSrcTree({ "tool-specs.ts": "alpha\r\nbeta\r\n" });
try {
assert.equal(
computeRegistryStamp(crlfTrailing.src),
computeRegistryStamp(bare.src),
);
} finally {
bare.cleanup();
crlfTrailing.cleanup();
}
});
// Only a SINGLE trailing newline is stripped — a second blank line is content and
// must change the hash. This pins the exact `/\n$/` semantics the loader mirrors.
// Only a SINGLE trailing newline is stripped — a second blank line is content.
test("only ONE trailing newline is stripped (two differ from one)", () => {
assert.notEqual(
computeRegistryStamp("x\n"),
computeRegistryStamp("x\n\n"),
);
const one = makeSrcTree({ "tool-specs.ts": "x\n" });
const two = makeSrcTree({ "tool-specs.ts": "x\n\n" });
try {
assert.notEqual(
computeRegistryStamp(one.src),
computeRegistryStamp(two.src),
);
} finally {
one.cleanup();
two.cleanup();
}
});
// Cross-impl equality against a fixed, documented input. The SAME literal input
// and expected hash are asserted in the server-side jest test
// (docmost-client.loader.spec.ts). If either side's normalize+sha256 ever
// diverges, one of the two tests reddens. Input exercises BOTH normalize steps.
test("fixed-input hash matches the documented cross-impl value", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const EXPECTED =
"683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83";
assert.equal(computeRegistryStamp(FIXED_INPUT), EXPECTED);
// Cross-impl equality against a fixed, documented tree. The SAME literal tree and
// expected hash are asserted in the server-side jest test
// (docmost-client.loader.spec.ts). If either side's enumerate+normalize+sha256
// ever diverges, one of the two tests reddens. The tree exercises: a nested file,
// BOTH normalize steps (tool-specs.ts uses CRLF + trailing \n) and the
// *.generated.ts exclusion.
const CROSS_IMPL_TREE = {
"tool-specs.ts": "line1\r\nline2\n",
"client/read.ts": "export const R = 1;\n",
"registry-stamp.generated.ts": 'export const REGISTRY_STAMP="ignored";\n',
};
const CROSS_IMPL_EXPECTED =
"131c1b9e4e2f5a7d6cef91ca8df619822b442f52bc45ebd09474a4c1d6728616";
test("fixed-tree hash matches the documented cross-impl value", () => {
const t = makeSrcTree(CROSS_IMPL_TREE);
try {
assert.equal(computeRegistryStamp(t.src), CROSS_IMPL_EXPECTED);
} finally {
t.cleanup();
}
});
// DESYNC GUARD (covers reviewer suggestion 2). Recompute the stamp from the
// actual src/tool-specs.ts and assert it equals the REGISTRY_STAMP baked into the
// freshly-built build/index.js. This reddens if the generated file is stale OR if
// the codegen normalize ever diverges from what produced the built stamp.
test("built REGISTRY_STAMP equals the stamp recomputed from src/tool-specs.ts", () => {
const source = readFileSync(TOOL_SPECS_PATH, "utf8");
assert.equal(computeRegistryStamp(source), REGISTRY_STAMP);
// Sanity: the EXPECTED constant is not a magic value but the documented
// enumerate+normalize+sha256 of CROSS_IMPL_TREE (a local re-implementation).
test("the documented EXPECTED is the enumerate+normalize+sha256 of the tree", () => {
const t = makeSrcTree(CROSS_IMPL_TREE);
try {
const collect = (dir) => {
const out = [];
for (const e of readdirSync(dir)) {
const f = join(dir, e);
if (statSync(f).isDirectory()) out.push(...collect(f));
else if (e.endsWith(".ts") && !e.endsWith(".generated.ts")) out.push(f);
}
return out;
};
const files = collect(t.src)
.map((abs) => ({ rel: relative(t.src, abs).split(sep).join("/"), abs }))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const h = createHash("sha256");
for (const { rel, abs } of files) {
const n = readFileSync(abs, "utf8")
.replace(/\r\n/g, "\n")
.replace(/\n$/, "");
h.update(rel, "utf8");
h.update("\0", "utf8");
h.update(n, "utf8");
h.update("\0", "utf8");
}
assert.equal(h.digest("hex"), CROSS_IMPL_EXPECTED);
} finally {
t.cleanup();
}
});
// Sanity: the fixed-input helper computes the SAME way the codegen does, proving
// the EXPECTED constant above is not an arbitrary magic value but the documented
// normalize+sha256 of FIXED_INPUT. Belt-and-braces so a bad EXPECTED can't hide a
// real regression.
test("the documented EXPECTED constant is the normalize+sha256 of FIXED_INPUT", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const normalized = FIXED_INPUT.replace(/\r\n/g, "\n").replace(/\n$/, "");
const expected = createHash("sha256")
.update(normalized, "utf8")
.digest("hex");
assert.equal(computeRegistryStamp(FIXED_INPUT), expected);
// DESYNC GUARD. Recompute the stamp from the REAL src/ tree and assert it equals
// the REGISTRY_STAMP baked into the freshly-built build/index.js. This reddens if
// the generated file is stale OR if the codegen ever diverges from what produced
// the built stamp.
test("built REGISTRY_STAMP equals the stamp recomputed from src/", () => {
assert.equal(computeRegistryStamp(SRC_DIR), REGISTRY_STAMP);
});
+3 -22
View File
@@ -1,31 +1,12 @@
# @docmost/prosemirror-markdown
The single, canonical **ProseMirror ↔ Markdown converter** plus the Docmost
schema mirror (#293/#345/#347). Headless and framework-free: no React. There is
exactly ONE copy of this converter in the repo, consumed by:
schema mirror (#293/#345). Headless and framework-free: no React, no browser
runtime. There is exactly ONE copy of this converter in the repo, consumed by:
- `packages/mcp` (the MCP server),
- `packages/git-sync` (two-way Git sync),
- `apps/server` (server-side markdown import/export, #345),
- `apps/client` (markdown paste/copy + AI-chat render, #347).
### Node vs browser entry
The HTML→DOM stage of markdown import runs on `jsdom` in Node and the native
`DOMParser` in the browser, injected per environment so **jsdom never enters a
client bundle**:
- default entry (`@docmost/prosemirror-markdown`) — Node: registers jsdom +
`@tiptap/html`'s happy-dom `server` `generateJSON`. Used by mcp / git-sync /
apps/server.
- `browser` entry (`@docmost/prosemirror-markdown/browser`, via the `"browser"`
exports condition) — registers the native `DOMParser` + `@tiptap/html`'s
browser `generateJSON`. Used by `apps/client`; carries no jsdom/happy-dom.
Both entries expose the identical converter surface; only the injected
DOM/`generateJSON` implementations differ (`src/lib/dom-parser.ts`). A
`markdownToProseMirrorSync` variant exists for callers that cannot await (the
client's synchronous chat renderer).
- `apps/server` (server-side markdown import/export, #345).
`src/lib/docmost-schema.ts` **mirrors** the upstream Tiptap schema that lives in
`packages/editor-ext`. The mirror is not free-floating: `serializer-contract.test.ts`

Some files were not shown because too many files have changed in this diff Show More