Compare commits

..

15 Commits

Author SHA1 Message Date
agent_coder 7b664f9078 fix(security): share-alias reassign 409 — гейт раскрытия по view-праву (доведение #495 к.4)
Коммит 4 закрыл утечку currentPageId на анонимном /availability, но
эквивалентная (и более широкая) дыра оставалась на POST /aliases/set (reassign):
при занятом алиасе и confirmReassign=false 409 отдавал currentPageId И
currentPageTitle ЦЕЛЕВОЙ страницы, а контроллер гейтил только validateCanEdit на
ИСХОДНОЙ странице. Любой участник с одной редактируемой+расшаренной страницей мог
перебирать имена алиасов и мапить их на (id, title) чужих страниц без права
просмотра — тот же класс перечисления, плюс ещё и заголовок.

Фикс: setAlias теперь гейтит раскрытие. currentPageId НЕ отдаётся никогда
(клиент им не пользуется, это перечислимая идентичность). currentPageTitle —
только если validateCanView(целевая, user) проходит; иначе голый «занят» (клиент
и так показывает generic confirm-модалку без заголовка — UX не ломается).
Гейт живёт в сервисе, где строится раскрытие (PageAccessService — @Global, без
цикла); контроллер прокидывает user. Поправлен неточный коммент checkAvailability.

Тесты: viewer → 409 с title, БЕЗ id; не-viewer → 409 без title и без id.
Mutation-verify: вернул утечку (id + безусловный title) → оба теста краснеют.
Контроллер-спек и int-spec обновлены под новую сигнатуру; tsc чист.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 22:45:28 +03:00
agent_coder d50295dc2a feat(ai,mcp): идентичность прогона реиндекса + брендированные PageId/SlugId (#495 item 14)
Аспект A — идентичность прогона реиндекса эмбеддингов.
У статуса/поллинга реиндекса не было идентичности КОНКРЕТНОГО прогона, из-за
чего класс багов «это тот же прогон или новый?» чинили дважды. Теперь каждый
прогон получает свой runId (crypto.randomUUID в start()), он хранится в Redis-
хэше рядом с total/done/startedAt и возвращается в ReindexProgress. Эндпоинт
статуса (getMasked -> MaskedAiSettings) отдаёт runId и reindexStartedAt. Клиент
кеит поллинг на (runId, startedAt): смена runId = НОВЫЙ прогон (сбрасываем
залатанное per-run состояние поллинга), тот же runId = тот же прогон. Всё
best-effort/косметика как и остальной код прогресса: пустой/отсутствующий runId
деградирует мягко и никогда не ломает реиндекс.

Аспект B — брендированные PageId/SlugId в MCP-клиенте + валидация формата в
серверных DTO (семейство инцидентов #435: двойная идентичность страницы —
внутренний id и slugId — гонялась как голая строка и молча путалась).
- packages/mcp/src/lib/page-id.ts: номинальные типы PageId/SlugId + валидирующие
  конструкторы asPageId/asSlugId и гварды isPageId/isSlugId (формат UUID и
  10-символьного slugId). PageId протянут через единственный узел канонизации и
  записи: resolvePageId() теперь возвращает PageId, а ключ per-page лока
  (withPageLock) и точки записи в collab (mutatePageContent/replacePageContent/
  updatePageContentRealtime) требуют бренд — сырой slugId/непроверенный id больше
  не проходит проверку типов (инвариант #260 «resolve-then-lock» теперь на уровне
  компилятора). Публичный вход методов остаётся строкой (это legitimно UUID ИЛИ
  slugId), брендируется каноническое значение.
- apps/server core/page/dto: PageIdDto.pageId получает валидацию формата
  (@Matches, UUID или 10-символьный slugId), так что кривая/подменённая
  идентичность отклоняется на границе, а не падает в repo голой строкой.

Тесты (все зелёные, с mutation-verify каждого):
- server: getMasked отдаёт runId/reindexStartedAt; стор пишет/читает runId и
  мягко деградирует его до ''; DTO отклоняет кривой pageId/slugId и принимает
  валидный.
- client: чистый хелпер reindexRunKey/isNewReindexRun (кеинг поллинга на runId).
- mcp: конструкторы/гварды PageId/SlugId (валидация формата).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:34:18 +03:00
agent_coder bf087c0a10 fix(ci): дедуп ночного фаззера по хэшу контрпримера, а не по префиксу заголовка
Ночной property-фаззер при находке контрпримера заводил/обновлял issue,
дедуплицируя по ПРЕФИКСУ ЗАГОЛОВКА. Из-за этого при уже открытом issue по
багу A другой баг B с тем же префиксом заголовка считался дубликатом и
молча терялся до закрытия первого issue — реальные вторые баги глотались.

Теперь дедуп идёт по стабильному короткому хэшу самого контрпримера:
- из вывода fast-check извлекается блок «Counterexample:» (минимальный
  падающий вход) до строки «Shrunk N time(s)»/«Got error»; сид, path и
  счётчик усадки в хэш НЕ входят, поэтому один и тот же баг с разными
  сидами даёт один хэш;
- sha256, первые 12 hex-символов, кладутся в заголовок и в
  машиночитаемый маркер тела `<!-- counterexample-hash: ... -->`;
- поиск открытого issue матчит этот хэш в заголовке или маркер в теле.

Итог: два РАЗНЫХ контрпримера дают два РАЗНЫХ issue, а повторная находка
ТОГО ЖЕ контрпримера по-прежнему схлопывается в существующий.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:13:23 +03:00
agent_coder 185f184989 i18n(ru): перевести недостающие строки в ru-RU
Добавлены переводы для 151 ключа, присутствовавшего в en-US, но
отсутствовавшего в ru-RU (настройки ИИ, диктовка, MCP, HTML-вставки,
роли агента и др.), включая ранее непереведённые «Streaming dictation»
и «Save and test». Технические токены и бренды (MCP, URL, Docmost AI,
плейсхолдеры версий) намеренно оставлены как есть. Добавлены русские
формы множественного числа (_few/_many) для «result found». Файл
переупорядочен в соответствии с порядком ключей en-US.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:13:23 +03:00
agent_coder 2a660687a0 fix(editor): защитить чтение приватных стеков y-undo в тулбаре
canUndo/canRedo в use-toolbar-state читали приватные внутренности y-undo
(undoManager.undoStack.length / redoStack.length). Апгрейд yjs / y-undo,
переименовавший или перестроивший эти поля, тихо сломал бы состояние кнопок
undo/redo (или упал бы на .length у undefined) без единой ошибки.

Оставляем дешёвое чтение длины стеков (сознательно не используем дорогой
editor.can().undo()/.redo(), который делает dry-run на каждый keystroke,
см. комментарий в файле), но теперь feature-detect: доверяем стекам только
если это реально массивы, иначе откатываемся на безопасный дефолт
(prosemirror-history undoDepth/redoDepth -> 0). Логика вынесена в чистую
функцию yHistoryAvailability.

Добавлен pin-test, фиксирующий текущую форму библиотеки: реальный
Y.UndoManager по-прежнему отдаёт undoStack/redoStack массивами. Апгрейд,
меняющий контракт, упадёт громко в тесте, а не тихо в UI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:13:23 +03:00
agent_coder 93abc01969 fix(client): авто-перезагрузка чанков в окне 5 минут вместо one-shot флага
При ChunkLoadError граница перезагружала страницу один раз, гейтируя
булевым флагом в sessionStorage, который никогда не сбрасывался. Из-за
этого ВТОРОЙ деплой за время жизни вкладки не давал авто-восстановления:
пользователь застревал на битом чанке без перезагрузки.

Заменяю one-shot флаг на счётчик по временному окну: не более одной
авто-перезагрузки за 5 минут. В sessionStorage храню метку времени
последней перезагрузки; на ChunkLoadError перезагружаемся только если
прошлая была раньше окна (или её не было), иначе проваливаемся в ручной
UI без перезагрузки. Это восстанавливает работу через несколько деплоев,
но не даёт бесконечного цикла при навсегда битом lazy-чанке (сброс флага
после успешного маунта отвергнут: оболочка монтируется, чанк 404 —
и цикл).

Решение об окне вынесено в чистый хелпер shouldAutoReload(now,
lastReloadAt, windowMs) и покрыто юнит-тестами: никогда-не-грузили →
можно; 6 минут назад → можно; 1 минуту назад → нельзя.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:13:23 +03:00
agent_coder cd5dc9e25b perf(db): GIN trigram-индексы строятся CONCURRENTLY вне транзакции
2 GIN trigram-индекса (pages.title, pages.text_content) + users.name строились
plain CREATE INDEX внутри Kysely-транзакции миграции: SHARE-lock блокирует записи
на pages/users на минуты при автодеплое.

Kysely гоняет миграцию в транзакции, а CONCURRENTLY внутри транзакции нельзя.
Поэтому ensureConcurrentIndexes (concurrent-indexes.ts) пре-строит эти индексы
через CREATE INDEX CONCURRENTLY (raw, вне транзакции) ДО миграатора — на
существующей БД миграционный CREATE INDEX IF NOT EXISTS становится no-op и лок не
берётся. Best-effort: на свежей БД (нет pages/f_unaccent) пре-build молча
пропускается, а миграция строит индекс на пустой таблице. Худший случай = прежнее
поведение, лучший — без лока.

CONCURRENT_INDEXES — канонические определения; drift-guard тест сверяет их с
выражениями в миграциях. Тест раннера: CONCURRENTLY+IF NOT EXISTS, вне
транзакции, best-effort (падение одного не рвёт остальные).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:11:49 +03:00
agent_coder 89be673854 fix(temp-notes): свип на старте + FOR UPDATE SKIP LOCKED в транзакции
Две дырки свипера временных заметок:
- Не было свипа на старте: заметки, протухшие во время простоя, ждали до часа
  до первого тика @Interval. Добавил onApplicationBootstrap (best-effort, не
  блокирует boot).
- Гонка без блокировки: re-check и removePage не были атомарны, «Сделать
  постоянной» могла проскочить между ними. Теперь re-check идёт в транзакции с
  FOR UPDATE SKIP LOCKED: сериализуется с toggleTemporary (тот же row-lock) и
  пропускает строки, захваченные другим воркером/инстансом (без двойной
  обработки). Удаление идёт ВНУТРИ этой транзакции.

removePage получил опциональный trx: чтобы удалять под блокировкой без deadlock
на вложенной независимой транзакции. Броадкаст PAGE_SOFT_DELETED при переданном
trx отложен на commit через registerAfterCommit (откат больше не рассылает
фантомное удаление); без trx поведение прежнее.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:05:16 +03:00
agent_coder 3d4166f5be fix(security): client-vitals — whitelist route/attr на анонимном эндпоинте
POST /api/telemetry/vitals анонимный, а route/attr не сверялись со словарём:
любой мог писать свободный текст в client_metrics (высокая кардинальность,
инъекция текста/PII/разметки).
- route: экспортировал полный словарь шаблонов из клиентского route-template.ts
  (KNOWN_ROUTE_TEMPLATES — канонический источник), сервер валидирует по зеркалу
  ALLOWED_ROUTE_TEMPLATES: нет в словаре → drop (null), событие остаётся.
- attr: это web-vitals attribution target (CSS-селектор), а не enum — ограничил
  консервативным charset CSS-селектора; всё вне набора → drop.
Клиентский self-consistency тест: templateRoute выдаёт ТОЛЬКО значения из
словаря (иначе легитимные метрики отбрасывались бы). Серверные тесты: raw-путь
и инъекция в route отброшены, PII/разметка в attr отброшены.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:58:40 +03:00
agent_coder f59b84ace2 fix(cache): bustWorkspaceCache — после коммита, не внутри транзакции
bustWorkspaceCache звался сразу после write, но ВНУТРИ переданной транзакции
(до коммита). Окно: параллельный читатель промахивается мимо инвалидированного
ключа, читает ещё НЕ закоммиченную (старую) строку и репопулирует кэш старым
значением; после коммита кэш держит устаревшее до TTL (15 c).

Добавил post-commit-хук в executeTx: registerAfterCommit(trx, fn) регистрирует
side-effect, который дренится ТОЛЬКО после коммита транзакции — причём внешним
executeTx, владеющим trx (проброшенный existingTrx срабатывает на настоящей
границе коммита, а не во вложенном вызове). WeakMap по trx — без утечки.
bustWorkspaceCache теперь: без trx — del сразу (уже автокоммит); с trx —
регистрирует del на post-commit. Ошибка хука не валит уже закоммиченный write.

Тест: порядок body→commit→hook, дренаж на внешней границе, глушение падения хука.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:55:21 +03:00
agent_coder beb61e77bb fix(page): move-position — валидация charset вместо длины
MovePageDto.position — это fractional-indexing ключ (generateJitteredKeyBetween,
тот же генератор, что в page.service). @MinLength(5)/@MaxLength(12) не совпадали
с реальным диапазоном генератора: плотные between-вставки в глубоком дереве
растят ключ далеко за 12 символов (замерено >40), и валидный ключ, который
сервер сам сгенерил, отклонялся 400 (Gitea #139, п.6).

Теперь валидируем по charset — base-62 алфавит [0-9A-Za-z] — плюс щедрый
@MaxLength(256) как чистый DoS-guard, сильно выше любого реального ключа.
test.failing (bug-lock) распинен в обычный it; добавлен кейс на отклонение
символов вне алфавита (control/separator/инъекция/пустая строка).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:49:13 +03:00
agent_coder 73dcc4553d fix(security): share-alias availability не отдаёт currentPageId
Проба доступности алиаса возвращала currentPageId — id страницы, на которую
алиас уже указывает — ЛЮБОМУ аутентифицированному участнику воркспейса без
проверки прав на просмотр этой страницы. Перебором имён алиасов можно было
смапить их на id страниц, к которым доступа нет.

Теперь checkAvailability отдаёт только {alias, valid, available}. Бита
taken/free достаточно для пробы; заголовок целевой страницы всплывает лишь
ПОСЛЕ реальной попытки setAlias (путь 409 ALIAS_REASSIGN_REQUIRED), который
проверяет права. Клиент currentPageId нигде не использовал — убран из типа,
стейта и теста. Серверный спек утверждает отсутствие currentPageId в ответе.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:47:27 +03:00
agent_coder a7fdeb47c8 chore(ai-chat): guard-пороги в env + единый маркер дегенерации
Три хвоста детектора петель (#444):
- Пороги детектора дегенерации теперь конфигурируются из env (по образцу
  AI_CHAT_FINAL_STEP_LOCKDOWN): AI_CHAT_DEGENERATION_REPEATED_LINES /
  _PERIOD_MAX_LEN / _PERIOD_MIN_REPEATS / _CHECK_STEP. Резолвер читает сырую
  строку (пусто = unset → компилируемый дефолт), требует ≥1, иначе безопасный
  откат к дефолту (0/отриц. сломал бы детектор). Оператор перенастраивает
  анти-babble-guard без редеплоя.
- Единый маркер: при дегенерации live-стрим показывает нейтральное «Response
  stopped.» (клиент не отличает от ручного Stop), а персист-баннер после refetch
  падал в дженерик. Классифицировал OUTPUT_DEGENERATION_ERROR на клиенте под тот
  же заголовок «Response stopped.» + деталь про петлю — live и refetch больше не
  расходятся.
- STEP_LIMIT_NO_ANSWER_MARKER: вместо хардкодной русской строки в content —
  локаль-нейтральная английская (базовая локаль = i18n-ключ этого репо, читаема
  моделью на реплее); не-русские юзеры больше не видят русский текст.

Тесты: env-пороги гоняются против реальных форм повторов с mutation-проверкой
(поднятый checkStep глушит burst; сниженный repeatedLines триггерит короткий
ран); клиентская классификация — новый кейс в error-message.test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:43:07 +03:00
agent_coder 490ff89467 chore(deps): вендорные ai-патчи — upstream-трекинг и план выравнивания версии
Оформил как план, а не сюрприз деплоя (строго после drain-патча из #486):
- upstream-репортинг двух ai-фиксов (O(n²) partialOutput heap-OOM; drain-hang
  в writeToServerResponse) и hocuspocus connect-vs-unload (#401) — анализ уже
  в PATCH()-заголовках самих патчей; ссылки держим в AGENTS.md, а НЕ в .patch
  (байты патча идут в patch_hash lockfile e8c599b3 — правка десинхронит пин и
  ломает pnpm install).
- рассинхрон версии ai в монорепе (клиент 6.0.207 vs сервер 6.0.134-patched):
  пока безвреден (серверные фиксы — мёртвый код в браузере), но это дрейф.
  Выравнивание — install-gated шаг с явным «портировать все три ai-патча на
  целевую версию»: оффсеты строк сдвинутся, текущий патч НЕ приложится как есть,
  pnpm install падёт на неприложенном патче — это и есть страховка.

Патч-файлы и lockfile не тронуты; оба tripwire-спека находят маркеры.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:32:47 +03:00
agent_coder 97d45ffce5 fix(ai): gemini/ollama через aiStreamingFetch + явный maxRetries
Провайдер-фабрики gemini и ollama (chat-путь) шли на глобальном undici-fetch:
без keep-alive recycle, без ретраев на pre-response reset, с дефолтным
(безграничным по паузе) таймаутом. Классы инцидентов #140/#175/#310 для них
воспроизводимы так же, как для openai. Прокинул this.aiProviderFetch (одна
строка на провайдера) — тот же слоёный instrumented streaming fetch, что уже
стоит на openai.

Плюс явно закрепил maxRetries=2 в обоих streamText-вызовах (authenticated и
public-share): совпадает с дефолтом SDK, но фиксирует потолок против дрейфа
дефолта. Арифметика коннектов на ход: (1 + maxRetries=2) × (1 +
AI_STREAM_PRE_RESPONSE_RETRIES) — два слоя ретраев композируются.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:29:36 +03:00
112 changed files with 4240 additions and 9585 deletions
+3 -41
View File
@@ -225,26 +225,11 @@ MCP_DOCMOST_PASSWORD=
# Silence timeout (ms) for EXTERNAL-MCP transport ONLY (not the chat provider).
# Tighter than AI_STREAM_TIMEOUT_MS so a byte-silent/hung MCP server is broken in
# ~1 min instead of 15. It cuts a legitimately long but byte-silent single tool
# call (a slow crawl that emits nothing until done) on the HTTP (streamable)
# transport, which opens a fresh request per call. The SSE transport — one
# long-lived body across many calls — is NO LONGER governed by this timeout
# (as of #489): its idle-BETWEEN-calls window has its own, raised bodyTimeout,
# AI_MCP_SSE_BODY_TIMEOUT_MS below. Default 60000 (1 min).
# ~1 min instead of 15. Note it also cuts a legitimately long but byte-silent
# single tool call (a slow crawl that emits nothing until done) and an SSE
# transport idling >1 min BETWEEN tool calls. Default 60000 (1 min).
# AI_MCP_STREAM_TIMEOUT_MS=60000
# bodyTimeout (ms) for the EXTERNAL-MCP SSE transport ONLY (#489). The SSE
# transport holds ONE response body open across many tool calls, so undici's
# bodyTimeout (time between body bytes) counts the LEGITIMATE silence BETWEEN the
# model's tool calls, not just a hung single call. At the tight 1-min silence
# timeout above, a normal >1-min gap between calls would break the SSE socket and
# the cache would serve a dead client until TTL — so the SSE transport gets its
# OWN, RAISED bodyTimeout. A single stuck call is still bounded by the per-call
# cap (AI_MCP_CALL_TIMEOUT_MS), and a socket that does break is healed by the
# in-run transport-error retry. The HTTP (streamable) transport keeps the tight
# timeout. Default 600000 (10 min).
# AI_MCP_SSE_BODY_TIMEOUT_MS=600000
# Total wall-clock cap (ms) for ONE external MCP tool call (app-level, not
# transport). Aborts a tool that keeps the socket warm (SSE heartbeats / trickle)
# but never returns a result — which the silence timeout above never breaks.
@@ -303,29 +288,6 @@ MCP_DOCMOST_PASSWORD=
# registry is process-local).
# AI_CHAT_RESUMABLE_STREAM=false
# --- Run lifecycle tunables (#487) ---
# These govern the universal run machinery (every turn is now a first-class run,
# both modes) and rarely need changing.
#
# How long a server-side SUPERSEDE ("interrupt and send now") waits for the target
# run to settle after issuing Stop before it degrades to a 409 SUPERSEDE_TIMEOUT
# (nothing sent, the composer keeps the user's text). 10s is generous under a
# healthy DB; do NOT raise it to paper over a slow DB — a SUPERSEDE_TIMEOUT is the
# honest signal. Default 10000 (10s).
# AI_CHAT_SUPERSEDE_TIMEOUT_MS=10000
#
# How often the periodic bidirectional reconcile job runs (heals runs/messages
# left dangling by a crash or a lost terminal write). Default 120000 (2 min).
# AI_CHAT_RECONCILE_INTERVAL_MS=120000
#
# Wall-clock cap for a SINGLE in-app tool call (a long paginated read, or a content
# write whose collab commit hangs) — the per-call half of the composite abort
# signal every in-app tool is wrapped with (the other half is the turn's Stop).
# The reconcile staleness floor is derived as max(2 x this cap, 15min), so a very
# high value delays stale-run recovery (the server boot-warns above 30min). Default
# 120000 (2 min).
# AI_CHAT_INAPP_TOOL_CALL_CAP_MS=120000
# --- Anonymous public-share AI assistant ---
# Opt-in per workspace (AI settings -> "public share assistant"; off by default).
# When enabled, anonymous visitors of a published share can ask an AI about that
+41 -10
View File
@@ -124,9 +124,17 @@ jobs:
exit "$FAILED"
# A GENUINE counterexample: fast-check printed a shrunk minimal case and its
# reproducing seed into property-output.txt. File a dedup-guarded issue whose
# title prefix is UNIQUE to counterexamples, so an infra failure (handled by
# the next step under a different title) can never poison this dedup.
# reproducing seed into property-output.txt. File a dedup-guarded issue.
#
# Dedup is keyed on a HASH of the SHRUNK COUNTEREXAMPLE (the minimal failing
# input), NOT on the issue title prefix. Keying on the prefix would let a
# single open issue swallow every OTHER counterexample (a different bug B whose
# title shares the prefix would be treated as a duplicate and stay silent until
# the first issue is closed). Hashing the shrunk example instead means two
# DIFFERENT counterexamples get two DIFFERENT issues, while a re-find of the
# SAME counterexample still dedupes onto the existing one. The infra-failure
# step (below) still keys on its own distinct title, so it can never poison
# this dedup either.
- name: File counterexample issue
# always() is REQUIRED: the fuzz step exits nonzero on a failing shard,
# so a bare `if:` (implicitly success() && ...) would skip this step
@@ -146,25 +154,48 @@ jobs:
echo "No fast-check counterexample signature — infra failure, handled by the next step."
exit 0
fi
TITLE="${TITLE_PREFIX} (seed=${FAIL_SEED})"
# Extract the SHRUNK counterexample block: the "Counterexample:" line(s)
# up to (but excluding) the "Shrunk N time(s)" / "Got error" line. This is
# the minimal failing INPUT and is STABLE across the different seeds/paths
# that reach the same bug — unlike the seed, path, or shrink count (which
# precede/follow this block and vary run-to-run) and unlike the whole
# output (which embeds those varying parts). Hashing THIS is what makes the
# dedup identity the bug itself rather than an incidental run detail.
CE_TEXT=$(awk '/Counterexample:/{c=1} /Shrunk [0-9]+ time|Got error/{c=0} c{print}' property-output.txt)
if [ -z "$CE_TEXT" ]; then
# No parseable shrunk block (unexpected — the signature check above
# already confirmed fast-check output). Fall back to the reproducing
# seed so we still emit a stable identity instead of silently deduping.
CE_TEXT="seed:${FAIL_SEED}"
fi
# Stable short id: first 12 hex chars of sha256 over the counterexample.
CE_HASH=$(printf '%s' "$CE_TEXT" | sha256sum | cut -c1-12)
# Machine-readable marker embedded in the issue body; the open-issue search
# below matches on it (and on the hash in the title) so identity travels
# with the issue regardless of any human title edits.
CE_MARKER="<!-- counterexample-hash: ${CE_HASH} -->"
export CE_HASH CE_MARKER
TITLE="${TITLE_PREFIX} [${CE_HASH}] (seed=${FAIL_SEED})"
# Best-effort dedup: skip if an open issue with the counterexample title
# prefix already exists. A failure of this check must NOT block creation.
# Dedup on the counterexample hash: skip only if an OPEN issue already
# carries this exact hash (in its title or its body marker). A different
# counterexample has a different hash and is NOT deduped. A failure of this
# check must NOT block creation.
EXISTING=""
if EXISTING=$(curl -sS \
-H "Authorization: token ${GITHUB_TOKEN}" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues?state=open&limit=100"); then
if printf '%s' "$EXISTING" \
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const p=process.env.TITLE_PREFIX;process.exit(a.some(i=>typeof i.title==="string"&&i.title.startsWith(p))?0:1)})'; then
echo "An open '${TITLE_PREFIX}' issue already exists — skipping creation."
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const h=process.env.CE_HASH,m=process.env.CE_MARKER;process.exit(a.some(i=>(typeof i.title==="string"&&i.title.includes(h))||(typeof i.body==="string"&&i.body.includes(m)))?0:1)})'; then
echo "An open issue for counterexample ${CE_HASH} already exists — skipping creation."
exit 0
fi
fi
# Build the JSON body with the test output SAFELY escaped (never hand-
# interpolate the counterexample into JSON).
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed with a fast-check counterexample.\n\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nReproduce locally:\n\n```\nPROPERTY_SEED=%s PROPERTY_NUM_RUNS=%s pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/\n```\n\nfast-check shrinks the failure to a minimal counterexample. Commit it as a permanent fixture under `packages/prosemirror-markdown/test/fixtures/counterexamples/` + a case in `counterexamples.test.ts`, then FIX the converter (do not weaken a property). See `packages/prosemirror-markdown/README.md`.\n\nTail of the test output (contains the shrunk counterexample):\n\n```\n%s\n```\n' \
"$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$FAIL_SEED" "$NUM_RUNS" "$(tail -n 120 property-output.txt)")
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed with a fast-check counterexample.\n\n- counterexample hash: `%s`\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nReproduce locally:\n\n```\nPROPERTY_SEED=%s PROPERTY_NUM_RUNS=%s pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/\n```\n\nfast-check shrinks the failure to a minimal counterexample. Commit it as a permanent fixture under `packages/prosemirror-markdown/test/fixtures/counterexamples/` + a case in `counterexamples.test.ts`, then FIX the converter (do not weaken a property). See `packages/prosemirror-markdown/README.md`.\n\nTail of the test output (contains the shrunk counterexample):\n\n```\n%s\n```\n\n%s\n' \
"$CE_HASH" "$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$FAIL_SEED" "$NUM_RUNS" "$(tail -n 120 property-output.txt)" "$CE_MARKER")
jq -n --arg title "$TITLE" --arg body "$BODY_TEXT" \
'{title: $title, body: $body}' > payload.json
-4
View File
@@ -29,10 +29,6 @@ packages/mcp/build/
# is a build artifact like build/ — never committed, always fresh.
packages/mcp/src/registry-stamp.generated.ts
# token-estimate compiled output (#490; built in CI/Docker via `pnpm build` /
# the server `pretest`, never committed, so src/ and prod can never diverge).
packages/token-estimate/dist/
# Logs
logs
*.log
+3 -1
View File
@@ -455,7 +455,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
- `core/ai-chat/tools/` — the agent's ~40 read+write tools. Every tool runs under the **calling user's** CASL permissions via a per-user loopback access token (`docmost-client.loader.ts`), so the agent can never exceed what the user could do. Only **reversible** operations are exposed (page history + trash; no permanent delete). Agent edits get an "AI agent" provenance badge in page history (`20260616T130000-agent-provenance` migration).
- `core/ai-chat/embedding/` — RAG indexer + a BullMQ consumer on `AI_QUEUE` that embeds pages into `page_embeddings` (vector search), complementing Postgres full-text search. Pages are (re)indexed on edit; `AI_EMBEDDING_TIMEOUT_MS` bounds a hung embeddings endpoint.
- `core/ai-chat/external-mcp/` — admins can attach external MCP servers (e.g. Tavily) to give the agent web access. **`ssrf-guard.ts` validates outbound MCP URLs against SSRF** — keep that guard in the path when touching external-MCP connection logic.
- `core/ai-chat/ai-chat-run.service.ts` + `ai_chat_runs`**every agent turn is now a first-class server-side RUN** (`#184`, universalized in `#487`): its lifecycle is tracked in `ai_chat_runs` in **both** modes, and the single-active-run-per-chat concurrency gate is enforced universally (a legacy second tab now gets a clean `409 A_RUN_ALREADY_ACTIVE` instead of a second parallel stream that interleaved history). The per-workspace `settings.ai.autonomousRuns` flag (off by default) **no longer gates whether a turn is a run** — it now controls **only the browser-disconnect semantics**: when ON the run is *detached* (a disconnect leaves it executing server-side; only an explicit `POST /ai-chat/stop` ends it, and a client reconnects/live-follows via `POST /ai-chat/run`); when OFF (legacy) a disconnect ends the turn by stopping its run via the run's stop lever. `#487` also adds a server-side **supersede** CAS ("interrupt and send now") to `POST /ai-chat/stream` (`supersede: { runId }`): it atomically stops the chat's currently-active run and waits for it to settle before the new turn claims the slot, returning `SUPERSEDE_INVALID` / `SUPERSEDE_TARGET_MISMATCH` / `SUPERSEDE_TIMEOUT` on the non-proceed branches. **DEPLOY CONSTRAINT — single-instance only in phase 1:** Stop and the AbortController that backs it are process-local, so a Stop only aborts a run executing on the **same** replica that owns it (cross-instance pub/sub stop is phase 2). Do **not** enable `autonomousRuns` on a horizontally-scaled deployment (multiple replicas behind a load balancer, or Docmost cloud `CLOUD=true`) — run a single instance instead. The server logs a startup WARNING when it detects a multi-instance deployment (`CLOUD=true`) so the constraint is visible. The startup sweep settles any run left dangling by a restart.
- `core/ai-chat/ai-chat-run.service.ts` + `ai_chat_runs`**detached/autonomous agent runs** (`#184`), behind the per-workspace `settings.ai.autonomousRuns` flag (off by default). When on, a turn becomes a server-side RUN that survives a browser disconnect; only an explicit `POST /ai-chat/stop` ends it, and a client reconnects/live-follows via `POST /ai-chat/run`. **DEPLOY CONSTRAINT — single-instance only in phase 1:** Stop and the AbortController that backs it are process-local, so a Stop only aborts a run executing on the **same** replica that owns it (cross-instance pub/sub stop is phase 2). Do **not** enable `autonomousRuns` on a horizontally-scaled deployment (multiple replicas behind a load balancer, or Docmost cloud `CLOUD=true`) — run a single instance instead. The server logs a startup WARNING when it detects a multi-instance deployment (`CLOUD=true`) so the constraint is visible. The startup sweep settles any run left dangling by a restart.
### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
@@ -471,6 +471,8 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch carries TWO independent server fixes, each with its own tripwire test: (1) it disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`); (2) it fixes `writeToServerResponse`'s drain-hang — the loop awaited only `"drain"` under backpressure, so a mid-write client disconnect parked the pipe forever and leaked the reader/buffers until restart; it now races `"drain"` against `"close"`/`"error"`, cancels the reader on disconnect, and swallows the fire-and-forget read rejection (#486; tripwire: `apps/server/src/integrations/ai/ai-sdk-drain-hang.patch.spec.ts`). Both tripwires assert BOTH installed dist builds carry their patch marker. The patch MUST be re-created via `pnpm patch` when bumping `ai`.
- **Upstream tracking (report the analysis upstream, don't just carry it):** both `ai` fixes and the hocuspocus one are candidates for upstreaming so we can eventually drop the local patch — the analysis is already written up in each patch's `PATCH(...)` header comments. File (a) an upstream **issue** on `vercel/ai` for the O(n²) cumulative `partialOutput` accumulation (heap OOM), (b) an upstream **issue** on `vercel/ai` for the `writeToServerResponse` drain-hang, and (c) an upstream **PR** on `@hocuspocus/server` for the connect-vs-unload race (local marker `PATCH(gitmost #401)` in `patches/@hocuspocus__server@3.4.4.patch`). Do NOT edit the patch files to add links — the patch bytes feed `patch_hash` in `pnpm-lock.yaml` (`ai@6.0.134``e8c599b3…`), so any content change there desyncs the lockfile pin and breaks `pnpm install`; keep upstream references here instead.
- **`ai` version is split across the monorepo and MUST be aligned deliberately, NOT casually:** the server pins `ai@6.0.134` (patched, exact — the `patchedDependencies` key forces that version), while the client declares `ai@6.0.207` (unpatched — the server-side `writeToServerResponse`/`partialOutput` fixes are dead code in the browser, so the mismatch is currently benign but is real drift). Alignment is a **planned, install-gated step**, never a bare `package.json` edit: (1) choose the target version; (2) re-create ALL THREE patch hunks (partialOutput publish-each, the `DefaultStreamTextResult` lazy-`output` wiring, and the drain-hang race) against the target dist via `pnpm patch` — the line offsets shift between versions, so the current patch WILL fail to apply as-is; (3) run a full `pnpm install` so the lockfile + new `patch_hash` regenerate together; (4) confirm both tripwire specs still find their markers. `pnpm install` FAILS HARD on an unapplied patch — that failure is the guardrail, so treat the port as a deliberate plan rather than discovering it as a deploy-time surprise.
- **The MCP tool inventory in `SERVER_INSTRUCTIONS` is GENERATED from the registry** (`packages/mcp/src/server-instructions.ts`: `buildToolInventory()` over `SHARED_TOOL_SPECS`) and spliced into the hand-written routing prose (`ROUTING_PROSE`). So adding/renaming/removing a **shared** spec in `packages/mcp/src/tool-specs.ts` auto-updates the `<tool_inventory>` — no manual `SERVER_INSTRUCTIONS` edit needed. Only an **inline** MCP-only tool (those registered via `server.registerTool(...)` in `index.ts`, not through the registry) needs a one-line entry in `INLINE_MCP_INVENTORY`. Enforced by `packages/mcp/test/unit/tool-inventory.test.mjs`, which fails when a registered tool is missing from the generated inventory (there is no `EXCEPTIONS` opt-out anymore — every tool must appear). Update `ROUTING_PROSE` when a tool's *intent guidance* (when-to-use) changes. `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
+9 -60
View File
@@ -202,17 +202,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
dangling by a restart. Phase 1 is single-instance-only (cross-instance Stop is
not yet reliable); the server warns at startup on a horizontally-scaled
deployment. (#184)
- **Server-side "interrupt and send now" (supersede) for AI chat.** `POST
/ai-chat/stream` now accepts a `supersede: { runId }` field: when the user sends
a new message while a run is active, the server atomically stops that run and
waits for it to settle before the new turn claims the chat's single run slot,
instead of the send being rejected as concurrent. The compare-and-set surfaces
three codes on its non-proceed branches — `SUPERSEDE_INVALID` (the targeted run
is malformed / belongs to another chat), `SUPERSEDE_TARGET_MISMATCH` (a
different run is now active; carries the current `activeRunId`), and
`SUPERSEDE_TIMEOUT` (the previous run did not stop within the settle window, so
nothing was sent and the composer keeps the text). Tunable via
`AI_CHAT_SUPERSEDE_TIMEOUT_MS` (default 10s). (#487)
- **Out-of-band page transfer via an in-RAM blob sandbox (`stash_page`).** A
new MCP tool serializes a whole page (its full ProseMirror JSON, with every
internal image/file mirrored) into an ephemeral in-RAM blob and returns only
@@ -293,17 +282,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **Every AI-chat turn is now a first-class server-side run, and one run per chat
is enforced in both modes.** The run machinery from `#184` was universalized: a
turn is tracked in `ai_chat_runs` and gated by the single-active-run-per-chat
index regardless of the `settings.ai.autonomousRuns` flag. **Behavior change:**
a second tab (or a double-submit) that starts a turn while one is already active
on the chat is now rejected up front with `409 A_RUN_ALREADY_ACTIVE` (carrying
the `activeRunId`); previously, on the legacy path, it opened a second parallel
stream on the same chat that interleaved history. The `autonomousRuns` flag no
longer controls whether a turn is a run — it now governs **only** the
browser-disconnect semantics (ON = detached/survives a disconnect; OFF = a
disconnect stops the run). (#487)
- **Vendor `ai` patch: upstream-tracking + version-alignment plan documented.**
The two local `ai@6.0.134` fixes (O(n²) `partialOutput` heap-OOM; the
`writeToServerResponse` drain-hang) and the hocuspocus connect-vs-unload race
now have explicit upstream-reporting and `ai`-version-alignment steps recorded
in `AGENTS.md` (client `ai@6.0.207` vs server `ai@6.0.134`-patched drift). The
patch bytes are unchanged — they feed the lockfile `patch_hash`, so the
alignment is called out as an install-gated plan rather than a bare version
bump. No runtime change.
- **Client markdown paste/copy and AI-chat rendering now go through the canonical
converter.** Pasting markdown into the editor, "Copy as markdown", the AI title
generator, and the AI-chat markdown renderer all now use
@@ -336,44 +323,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **A long AI chat no longer bricks on the model's context window, and each turn
stops re-persisting the whole tool-output history.** Tool outputs are now
stored ONCE, in `metadata.parts`; the `tool_calls` trace keeps only per-step
outcome flags (a v2 trace shape), ending the O(N²) write amplification that
re-wrote every prior output on every step (measured on a live Postgres via the
`pg_current_wal_lsn()` delta: the trace column shrank ~3200×, the full
assistant row ~51%). The persisted record is unchanged in content — the full
history still lives in `metadata.parts`. At REPLAY time only, the history sent
to the provider is now bounded by a deterministic, prompt-cache-friendly token
budget: `floor(0.7 × chatContextWindow)` when a window is configured (no cap —
anti-brick protection, not a cost limiter), a flat 100k fallback for installs
with no window set (exactly the ones that hit terminal overflow), or off when
the window is explicitly `0`. Trimming truncates old tool outputs first, then
mechanically collapses the oldest turns, always keeping the recent turns full
and the tool-call/result pairing balanced. A provider context-overflow 400 is
now classified and used as a reactive signal: the row is stamped so the NEXT
turn re-trims aggressively (0.5×), which un-bricks a chat that just 400'd. The
client token badge and the server budgeter now share one estimator (new
`@docmost/token-estimate` package) so they can never diverge. Deferred-tool
activation is also cached in the chat metadata to avoid re-resolving it each
turn. (#490)
- **A chat with one malformed message part no longer 500s on every turn, and a
failed send no longer duplicates the user's message.** Incoming client parts
are now whitelisted to `text` (a forged tool-result part can no longer reach
the persisted history or the model context), and the turn is converted BEFORE
the user row is inserted, so a mid-flight failure cannot leave a duplicate
user row that a retry then compounds. A single part that still fails to convert
degrades to a `[tool context omitted]` marker on that one row instead of
bricking the whole chat. (#489)
- **A transport drop to an external MCP server now heals within the same turn.**
On an undici transport error, a read-only MCP tool reconnects its server and
retries once within the run; a write is never auto-retried (it may already have
applied). One flapping server no longer nulls the shared client cache, so other
servers' cached clients are untouched. The SSE transport also gets a raised
body-timeout so a legitimate >1-min idle between the model's tool calls no
longer breaks a long-lived SSE socket (new `AI_MCP_SSE_BODY_TIMEOUT_MS`, default
10 min; see `.env.example`). (#489)
- **The server no longer runs out of heap during long autonomous agent runs.** A
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
snapshot of the ENTIRE turn text on every streamed text-delta when no output
-1
View File
@@ -22,7 +22,6 @@
"@casl/react": "5.0.1",
"@docmost/editor-ext": "workspace:*",
"@docmost/prosemirror-markdown": "workspace:*",
"@docmost/token-estimate": "workspace:*",
"@excalidraw/excalidraw": "0.18.0-3a5ef40",
"@mantine/core": "8.3.18",
"@mantine/dates": "8.3.18",
+230 -78
View File
@@ -256,6 +256,9 @@
"Invite link": "Ссылка для приглашения",
"Copy": "Копировать",
"Copy to space": "Копировать в пространство",
"Copy chat": "Копировать чат",
"Dock to sidebar": "Закрепить в боковой панели",
"Undock": "Открепить",
"Copied": "Скопировано",
"Failed to export chat": "Не удалось экспортировать чат",
"Duplicate": "Дублировать",
@@ -285,6 +288,9 @@
"Alt text": "Альтернативный текст",
"Describe this for accessibility.": "Опишите это для специальных возможностей.",
"Add a description": "Добавить описание",
"Caption": "Подпись",
"Add a caption": "Добавить подпись",
"Shown below the image.": "Отображается под изображением.",
"Justify": "По ширине",
"Merge cells": "Объединить ячейки",
"Split cell": "Разделить ячейку",
@@ -388,22 +394,6 @@
"Quote": "Цитата",
"Image": "Изображение",
"Audio": "Аудио",
"Transcribe": "Транскрибировать",
"Transcribing…": "Транскрибация…",
"No speech detected": "Речь не распознана",
"Transcription failed": "Не удалось распознать речь",
"Voice dictation is not configured": "Голосовой ввод не настроен",
"Start dictation": "Начать диктовку",
"Stop recording": "Остановить запись",
"Microphone access denied": "Доступ к микрофону запрещён",
"No microphone found": "Микрофон не найден",
"Microphone is unavailable or already in use": "Микрофон недоступен или уже используется",
"Could not start recording": "Не удалось начать запись",
"Audio recording is not available in this browser/context": "Запись аудио недоступна в этом браузере/контексте",
"Dictation": "Диктовка",
"Dictation becomes available once the page finishes connecting": "Диктовка станет доступна после подключения к документу",
"No connection to the collaboration server — dictation unavailable": "Нет связи с сервером совместного редактирования — диктовка недоступна",
"This page is read-only": "Страница открыта только для чтения",
"Embed PDF": "Встроить PDF",
"Upload and embed a PDF file.": "Загрузите и встроите PDF-файл.",
"Embed as PDF": "Встроить как PDF",
@@ -419,9 +409,6 @@
"Footnote {{number}}": "Сноска {{number}}",
"Go to footnote": "Перейти к сноске",
"Back to reference": "Вернуться к ссылке",
"Back to references": "Вернуться к ссылкам",
"Back to reference {{label}}": "Вернуться к ссылке {{label}}",
"Empty footnote": "Пустая сноска",
"Math inline": "Строчная формула",
"Insert inline math equation.": "Вставить математическое выражение в строку.",
"Math block": "Блок формулы",
@@ -447,6 +434,9 @@
"{{count}} command available_other": "Доступно {{count}} команд",
"{{count}} result available_one": "Доступен 1 результат",
"{{count}} result available_other": "Доступно {{count}} результатов",
"{{count}} result found_one": "Найден {{count}} результат",
"{{count}} result found_few": "Найдено {{count}} результата",
"{{count}} result found_other": "Найдено {{count}} результатов",
"Equal columns": "Равные столбцы",
"Left sidebar": "Левая боковая панель",
"Right sidebar": "Правая боковая панель",
@@ -456,6 +446,7 @@
"Names do not match": "Названия не совпадают",
"Today, {{time}}": "Сегодня, {{time}}",
"Yesterday, {{time}}": "Вчера, {{time}}",
"now": "сейчас",
"Space created successfully": "Пространство успешно создано",
"Space updated successfully": "Пространство успешно обновлено",
"Space deleted successfully": "Пространство успешно удалено",
@@ -559,6 +550,7 @@
"Add 2FA method": "Добавить метод 2FA",
"Backup codes": "Резервные коды",
"Disable": "Отключить",
"disabled": "отключено",
"Invalid verification code": "Недействительный код подтверждения",
"New backup codes have been generated": "Новые резервные коды сгенерированы",
"Failed to regenerate backup codes": "Не удалось заново сгенерировать резервные коды",
@@ -702,62 +694,6 @@
"AI search": "Поиск ИИ",
"AI Answer": "Ответ ИИ",
"Ask AI": "Спросить ИИ",
"AI agent": "AI-агент",
"Take a look at the current document": "Посмотри текущий документ",
"Start automatically": "Запускать автоматически",
"When on, picking this role sends a launch message and starts the chat. When off, the role is selected and you type the first message yourself.": "Когда включено, выбор этой роли отправляет стартовое сообщение и начинает чат. Когда выключено, роль выбирается, а первое сообщение вы вводите сами.",
"Launch message": "Стартовое сообщение",
"Sent automatically when this role is picked. Leave empty to use the default text. Ignored when “Start automatically” is off.": "Отправляется автоматически при выборе этой роли. Оставьте пустым, чтобы использовать текст по умолчанию. Игнорируется, когда «Запускать автоматически» выключено.",
"AI agent is typing…": "AI-агент печатает…",
"{{name}} is typing…": "{{name}} печатает…",
"Thinking…": "Думаю…",
"Thinking… · {{count}} tokens": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_one": "Думаю… · {{count}} токен",
"Thinking… · {{count}} tokens_few": "Думаю… · {{count}} токена",
"Thinking… · {{count}} tokens_many": "Думаю… · {{count}} токенов",
"Thinking · {{count}} tokens": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_one": "Размышления · {{count}} токен",
"Thinking · {{count}} tokens_few": "Размышления · {{count}} токена",
"Thinking · {{count}} tokens_many": "Размышления · {{count}} токенов",
"Agent role": "Роль агента",
"AI chat": "AI-чат",
"AI chat is disabled for this workspace.": "AI-чат отключён для этого рабочего пространства.",
"Ask a question about this documentation.": "Задайте вопрос об этой документации.",
"Ask a question…": "Задайте вопрос…",
"Ask the AI agent anything about your workspace.": "Спросите AI-агента о чём угодно по вашему рабочему пространству.",
"Ask the AI agent…": "Спросите AI-агента…",
"Copy chat": "Копировать чат",
"Dock to sidebar": "Закрепить в боковой панели",
"Undock": "Открепить",
"Created successfully": "Успешно создано",
"Context size / model limit": "Размер контекста / лимит модели",
"Context window (tokens)": "Окно контекста (токены)",
"Shown as used / total in the chat header. Leave empty to hide the limit.": "Показывается в шапке чата как использовано / всего. Пусто — лимит скрыт.",
"Delete this chat?": "Удалить этот чат?",
"Deleted successfully": "Успешно удалено",
"AI agent «{{role}}» on behalf of {{person}}": "AI-агент «{{role}}» от имени {{person}}",
"AI agent {{name}}": "AI-агент {{name}}",
"Failed to delete chat": "Не удалось удалить чат",
"Failed to rename chat": "Не удалось переименовать чат",
"Failed": "Ошибка",
"OK · {{n}}": "OK · {{n}}",
"Test": "Тест",
"No tools available": "Инструменты недоступны",
"Available tools": "Доступные инструменты",
"Minimize": "Свернуть",
"No chats yet.": "Чатов пока нет.",
"Send": "Отправить",
"Send when the agent finishes": "Отправить, когда агент закончит",
"Queue message": "Поставить в очередь",
"Remove queued message": "Убрать из очереди",
"Send now": "Отправить сейчас",
"Interrupt and send now": "Прервать и отправить сейчас",
"Something went wrong": "Что-то пошло не так",
"Stop": "Стоп",
"The AI agent could not respond. Please try again.": "AI-агент не смог ответить. Попробуйте ещё раз.",
"The AI provider is not configured. Ask an administrator to set it up.": "AI-провайдер не настроен. Попросите администратора настроить его.",
"Universal assistant": "Универсальный ассистент",
"You": "Вы",
"AI is thinking...": "ИИ обрабатывает запрос...",
"Thinking": "Думаю",
"Ask a question...": "Задайте вопрос...",
@@ -784,8 +720,40 @@
"Manage API keys for all users in the workspace. View the <anchor>API documentation</anchor> for usage details.": "Управляйте API-ключами для всех пользователей в рабочем пространстве. Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>API documentation</anchor> for usage details.": "Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>MCP documentation</anchor>.": "Смотрите <anchor>документацию по MCP</anchor>.",
"Instructions": нструкции",
"AI / Models": И / Модели",
"AI / External tools (MCP)": "ИИ / Внешние инструменты (MCP)",
"Add server": "Добавить сервер",
"Edit server": "Изменить сервер",
"Delete server": "Удалить сервер",
"Are you sure you want to delete this MCP server?": "Вы уверены, что хотите удалить этот MCP-сервер?",
"No external servers configured": "Внешние серверы не настроены",
"Server name": "Имя сервера",
"Transport": "Транспорт",
"URL": "URL",
"Authorization header": "Заголовок авторизации",
"Tool allowlist": "Список разрешённых инструментов",
"Optional. Leave empty to allow all tools the server exposes.": "Необязательно. Оставьте пустым, чтобы разрешить все инструменты, которые предоставляет сервер.",
"Optional guidance for the agent on how and when to use this server's tools. Injected into the system prompt. The server's tools are namespaced as \"<server name>_*\".": "Необязательное указание агенту, как и когда использовать инструменты этого сервера. Добавляется в системный промпт. Инструменты сервера именуются с префиксом «<имя сервера>_*».",
"Test": "Тест",
"Available tools": "Доступные инструменты",
"No tools available": "Инструменты недоступны",
"Failed": "Ошибка",
"OK · {{n}}": "OK · {{n}}",
"Created successfully": "Успешно создано",
"Deleted successfully": "Успешно удалено",
"Clear": "Очистить",
"Provider": "Провайдер",
"•••• set": "•••• задан",
"Clear key": "Очистить ключ",
"Base URL": "Базовый URL",
"Chat model": "Модель чата",
"Embedding model": "Модель эмбеддингов",
"System message": "Системное сообщение",
"A built-in safety framework is always appended.": "Встроенный набор правил безопасности всегда добавляется автоматически.",
"Test connection": "Проверить соединение",
"Connection successful": "Соединение установлено",
"Connection failed": "Не удалось установить соединение",
"Only workspace admins can manage AI provider settings.": "Управлять настройками провайдера ИИ могут только администраторы рабочего пространства.",
"Sources": "Источники",
"AI Answers not available for attachments": "Ответы ИИ недоступны для вложений",
"No answer available": "Ответ недоступен",
@@ -1013,6 +981,7 @@
"Try again": "Попробовать снова",
"Untitled chat": "Чат без названия",
"No document": "Без документа",
"You": "Вы",
"What can I help you with?": "Чем я могу вам помочь?",
"Are you sure you want to revoke this {{credential}}": "Вы уверены, что хотите отозвать этот {{credential}}",
"Automatically provision users and groups from your identity provider via SCIM.": "Автоматически предоставляйте доступ пользователям и группам из вашего провайдера удостоверений через SCIM.",
@@ -1041,6 +1010,9 @@
"Page menu": "Меню страницы",
"Expand": "Развернуть",
"Collapse": "Свернуть",
"Expand all": "Развернуть все",
"Collapse all": "Свернуть все",
"Couldn't expand the tree: {{reason}}": "Не удалось развернуть дерево: {{reason}}",
"Comment menu": "Меню комментария",
"Group menu": "Меню группы",
"Show hidden breadcrumbs": "Показать скрытые хлебные крошки",
@@ -1077,7 +1049,7 @@
"Search pages and spaces...": "Поиск страниц и пространств...",
"No results found": "Результаты не найдены",
"You don't have permission to create pages here": "У вас нет прав на создание страниц здесь",
"Chat menu": "Меню чата",
"Chat menu for {{title}}": "Меню чата для {{title}}",
"API key menu": "Меню API-ключа",
"Jump to comment selection": "Перейти к выбору комментария",
"Slash commands": "Команды со слешем",
@@ -1131,6 +1103,9 @@
"Undo": "Отменить",
"Redo": "Повторить",
"Backlinks": "Обратные ссылки",
"Back to references": "Вернуться к ссылкам",
"Back to reference {{label}}": "Вернуться к ссылке {{label}}",
"Empty footnote": "Пустая сноска",
"Last updated by": "Последний изменивший",
"Last updated": "Последнее обновление",
"Stats": "Статистика",
@@ -1164,6 +1139,7 @@
"Page title": "Заголовок страницы",
"Page content": "Содержимое страницы",
"Member actions": "Действия с участником",
"Member actions for {{name}}": "Действия с участником {{name}}",
"Toggle password visibility": "Переключить видимость пароля",
"Send comment": "Отправить комментарий",
"Token actions": "Действия с токеном",
@@ -1183,11 +1159,187 @@
"Removed from favorites": "Удалено из избранного",
"Added {{name}} to favorites": "{{name}} добавлено в избранное",
"Removed {{name}} from favorites": "{{name}} удалено из избранного",
"Label added": "Метка добавлена",
"Label removed": "Метка удалена",
"Image updated": "Изображение обновлено",
"Unsupported image type": "Неподдерживаемый тип изображения",
"Member deactivated": "Участник деактивирован",
"Member activated": "Участник активирован",
"Name is required": "Укажите имя",
"Name must be 40 characters or fewer": "Имя должно содержать не более 40 символов",
"Group name must be at least 2 characters": "Название группы должно содержать не менее 2 символов",
"Group name must be 100 characters or fewer": "Название группы должно содержать не более 100 символов",
"Description must be 500 characters or fewer": "Описание должно содержать не более 500 символов",
"Invalid invitation link": "Недействительная ссылка-приглашение",
"Page menu for {{name}}": "Меню страницы для {{name}}",
"Create subpage of {{name}}": "Создать подстраницу для {{name}}",
"AI chat": "AI-чат",
"Ask a question about this documentation.": "Задайте вопрос об этой документации.",
"Ask a question…": "Задайте вопрос…",
"Thinking…": "Думаю…",
"Thinking… · {{count}} tokens": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_one": "Думаю… · {{count}} токен",
"Thinking… · {{count}} tokens_few": "Думаю… · {{count}} токена",
"Thinking… · {{count}} tokens_many": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_other": "Думаю… · {{count}} токенов",
"Thinking · {{count}} tokens": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_one": "Размышления · {{count}} токен",
"Thinking · {{count}} tokens_few": "Размышления · {{count}} токена",
"Thinking · {{count}} tokens_many": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_other": "Размышления · {{count}} токенов",
"The assistant is unavailable right now. Please try again.": "Ассистент сейчас недоступен. Попробуйте ещё раз.",
"Public share assistant": "Ассистент публичного доступа",
"Let anonymous visitors of public shares ask an AI assistant scoped to that share's pages. You pay for the tokens.": "Позвольте анонимным посетителям публичных ссылок обращаться к ИИ-ассистенту в рамках страниц этой публикации. Токены оплачиваете вы.",
"Public assistant model": "Модель публичного ассистента",
"Defaults to the chat model": "По умолчанию используется модель чата",
"Optional cheaper model id for the public assistant. Empty uses the chat model above.": "Необязательный более дешёвый идентификатор модели для публичного ассистента. Если пусто, используется модель чата выше.",
"Assistant identity": "Личность ассистента",
"Pick an agent role whose persona the public assistant adopts. The safety rules always still apply.": "Выберите роль агента, чью личность примет публичный ассистент. Правила безопасности всегда остаются в силе.",
"Built-in assistant persona": "Встроенная личность ассистента",
"Minimize": "Свернуть",
"Context size / model limit": "Размер контекста / лимит модели",
"Context window (tokens)": "Окно контекста (токены)",
"Shown as used / total in the chat header. Leave empty to hide the limit.": "Показывается в шапке чата как использовано / всего. Пусто — лимит скрыт.",
"AI agent": "AI-агент",
"Take a look at the current document": "Посмотри текущий документ",
"AI agent is typing…": "AI-агент печатает…",
"{{name}} is typing…": "{{name}} печатает…",
"Send": "Отправить",
"Send when the agent finishes": "Отправить, когда агент закончит",
"Queue message": "Поставить в очередь",
"Remove queued message": "Убрать из очереди",
"Send now": "Отправить сейчас",
"Interrupt and send now": "Прервать и отправить сейчас",
"Stop": "Стоп",
"Response stopped.": "Ответ остановлен.",
"Connection lost — the answer was interrupted.": "Соединение потеряно — ответ был прерван.",
"Response stopped (manually or the connection dropped).": "Ответ остановлен (вручную или из-за разрыва соединения).",
"Chat menu": "Меню чата",
"No chats yet.": "Чатов пока нет.",
"Delete this chat?": "Удалить этот чат?",
"Ask the AI agent…": "Спросите AI-агента…",
"Ask the AI agent anything about your workspace.": "Спросите AI-агента о чём угодно по вашему рабочему пространству.",
"Failed to rename chat": "Не удалось переименовать чат",
"Failed to delete chat": "Не удалось удалить чат",
"Something went wrong": "Что-то пошло не так",
"AI chat is disabled for this workspace.": "AI-чат отключён для этого рабочего пространства.",
"The AI provider is not configured. Ask an administrator to set it up.": "AI-провайдер не настроен. Попросите администратора настроить его.",
"The AI agent could not respond. Please try again.": "AI-агент не смог ответить. Попробуйте ещё раз.",
"Searched pages": "Поиск по страницам",
"Read page": "Прочитана страница",
"Created page": "Создана страница",
"Updated page": "Обновлена страница",
"Renamed page": "Переименована страница",
"Moved page": "Перемещена страница",
"Deleted page (to trash)": "Удалена страница (в корзину)",
"Commented": "Добавлен комментарий",
"Resolved comment": "Комментарий решён",
"Ran tool {{name}}": "Выполнен инструмент {{name}}",
"AI agent «{{role}}» on behalf of {{person}}": "AI-агент «{{role}}» от имени {{person}}",
"AI agent {{name}}": "AI-агент {{name}}",
"Endpoints": "Эндпоинты",
"where we fetch models": "откуда мы получаем модели",
"All endpoints are OpenAI-compatible. Point the Base URL at OpenAI, OpenRouter, a local Ollama, or any self-hosted server.": "Все эндпоинты совместимы с OpenAI. Укажите в базовом URL адрес OpenAI, OpenRouter, локального Ollama или любого self-hosted сервера.",
"Chat / LLM": "Чат / LLM",
"root": "корневой",
"Semantic search": "Семантический поиск",
"Voice / STT": "Голос / STT",
"Voice dictation": "Голосовой ввод",
"Streaming dictation": "Потоковый голосовой ввод",
"Transcribe as you speak, cutting on pauses": "Транскрибирование по мере речи, с разбивкой на паузах",
"Voice dictation is not available yet.": "Голосовой ввод пока недоступен.",
"Test endpoint": "Проверить эндпоинт",
"Save and test": "Сохранить и проверить",
"Save endpoints": "Сохранить эндпоинты",
"Configured and enabled": "Настроено и включено",
"Configured but disabled": "Настроено, но отключено",
"Enabled but not configured": "Включено, но не настроено",
"Not configured": "Не настроено",
"External tools": "Внешние инструменты",
"Gitmost as MCP client": "Gitmost как MCP-клиент",
"Servers the agent calls out to.": "Серверы, к которым обращается агент.",
"MCP server": "MCP-сервер",
"expose the workspace": "открыть доступ к рабочему пространству",
"Enable MCP server": "Включить MCP-сервер",
"Exposes the workspace as an MCP server at /mcp — this provides a capability, it doesn't consume a model.": "Открывает рабочее пространство как MCP-сервер по адресу /mcp — это предоставляет возможность, а не потребляет модель.",
"Resolves to {{url}}": "Разрешается в {{url}}",
"Model": "Модель",
"Done": "Готово",
"shared prompt · safety framework appended automatically": "общий промпт · правила безопасности добавляются автоматически",
"/v1/chat/completions · root endpoint — Embeddings and Voice inherit its URL and key": "/v1/chat/completions · корневой эндпоинт — Эмбеддинги и Голос наследуют его URL и ключ",
"/v1/embeddings · embeds pages so semantic search can find them": "/v1/embeddings · создаёт эмбеддинги страниц, чтобы их находил семантический поиск",
"/v1/audio/transcriptions · works with local whisper (speaches / faster-whisper-server)": "/v1/audio/transcriptions · работает с локальным whisper (speaches / faster-whisper-server)",
"Vector search · requires pgvector": "Векторный поиск · требуется pgvector",
"Embedding API key": "API-ключ для эмбеддингов",
"Embeddings": "Эмбеддинги",
"Leave empty to use the chat API key": "Оставьте пустым, чтобы использовать API-ключ чата",
"Leave empty to use the chat base URL": "Оставьте пустым, чтобы использовать базовый URL чата",
"Reindex now": "Переиндексировать сейчас",
"Start dictation": "Начать диктовку",
"Stop recording": "Остановить запись",
"Transcribing…": "Транскрибация…",
"Microphone access denied": "Доступ к микрофону запрещён",
"No microphone found": "Микрофон не найден",
"Could not start recording": "Не удалось начать запись",
"Transcription failed": "Не удалось распознать речь",
"Transcribe": "Транскрибировать",
"No speech detected": "Речь не распознана",
"Voice dictation is not configured": "Голосовой ввод не настроен",
"Microphone is unavailable or already in use": "Микрофон недоступен или уже используется",
"Audio recording is not available in this browser/context": "Запись аудио недоступна в этом браузере/контексте",
"Dictation": "Диктовка",
"Dictation becomes available once the page finishes connecting": "Диктовка станет доступна после подключения к документу",
"No connection to the collaboration server — dictation unavailable": "Нет связи с сервером совместного редактирования — диктовка недоступна",
"This page is read-only": "Страница открыта только для чтения",
"Request format": "Формат запроса",
"How transcription requests are sent to the endpoint": "Как запросы на транскрибирование отправляются на эндпоинт",
"OpenAI-compatible (multipart/form-data)": "Совместимо с OpenAI (multipart/form-data)",
"OpenRouter (JSON, base64 audio)": "OpenRouter (JSON, аудио в base64)",
"Dictation language": "Язык диктовки",
"Auto-detect": "Автоопределение",
"Spoken language hint sent to the transcription model. Auto-detect lets the model decide.": "Подсказка языка речи для модели транскрипции. «Автоопределение» оставляет выбор за моделью.",
"Agent role": "Роль агента",
"Universal assistant": "Универсальный ассистент",
"Add role": "Добавить роль",
"Edit role": "Изменить роль",
"Role name": "Название роли",
"e.g. Proofreader": "напр. Корректор",
"Optional. Shown as the chat badge.": "Необязательно. Отображается как значок чата.",
"Optional. A short note about what this role does.": "Необязательно. Краткое описание того, что делает эта роль.",
"Instructions": "Инструкции",
"The built-in safety framework is always added automatically.": "Встроенный набор правил безопасности всегда добавляется автоматически.",
"Model provider override": "Переопределение провайдера модели",
"Optional. Defaults to the workspace provider.": "Необязательно. По умолчанию используется провайдер рабочего пространства.",
"Model override": "Переопределение модели",
"Optional. Defaults to the workspace model.": "Необязательно. По умолчанию используется модель рабочего пространства.",
"e.g. gpt-4o-mini": "напр. gpt-4o-mini",
"If you choose a different provider, it must already be configured in AI settings.": "Если вы выбираете другого провайдера, он уже должен быть настроен в настройках ИИ.",
"Start automatically": "Запускать автоматически",
"When on, picking this role sends a launch message and starts the chat. When off, the role is selected and you type the first message yourself.": "Когда включено, выбор этой роли отправляет стартовое сообщение и начинает чат. Когда выключено, роль выбирается, а первое сообщение вы вводите сами.",
"Launch message": "Стартовое сообщение",
"Sent automatically when this role is picked. Leave empty to use the default text. Ignored when “Start automatically” is off.": "Отправляется автоматически при выборе этой роли. Оставьте пустым, чтобы использовать текст по умолчанию. Игнорируется, когда «Запускать автоматически» выключено.",
"Agent roles": "Роли агента",
"Reusable presets that shape the agent's behavior (and optionally its model). Picked when starting a new chat.": "Многоразовые пресеты, определяющие поведение агента (и, при желании, его модель). Выбираются при запуске нового чата.",
"No roles configured": "Роли не настроены",
"Delete role": "Удалить роль",
"Are you sure you want to delete this role?": "Вы уверены, что хотите удалить эту роль?",
"HTML embed": "HTML-вставка",
"Edit HTML embed": "Изменить HTML-вставку",
"HTML embed is disabled in this workspace": "HTML-вставки отключены в этом рабочем пространстве",
"Click to add HTML / CSS / JS": "Нажмите, чтобы добавить HTML / CSS / JS",
"This HTML/CSS/JS runs in a sandboxed frame and cannot access the viewer's session, cookies, or API.": "Этот HTML/CSS/JS выполняется в изолированном фрейме и не имеет доступа к сессии, cookie или API просматривающего.",
"<script>...</script>": "<script>...</script>",
"Height (px, blank = auto)": "Высота (px, пусто = авто)",
"advanced": "дополнительно",
"Enable HTML embed": "Включить HTML-вставки",
"Allow members to insert raw HTML/CSS/JavaScript blocks. The block renders in a sandboxed frame and cannot access the viewer's session, cookies, or API. Off by default.": "Разрешить участникам вставлять блоки с необработанным HTML/CSS/JavaScript. Блок отображается в изолированном фрейме и не имеет доступа к сессии, cookie или API просматривающего. По умолчанию выключено.",
"When enabled, any member can insert an HTML embed block. The toggle just enables or disables the block type workspace-wide.": "Когда включено, любой участник может вставить блок HTML-вставки. Переключатель просто включает или отключает этот тип блока во всём рабочем пространстве.",
"Embeds run inside a sandboxed iframe with a separate origin, so they cannot read or modify the page they are embedded in.": "Вставки выполняются в изолированном iframe с отдельным источником, поэтому они не могут читать или изменять страницу, в которую встроены.",
"Turning this off hides existing embeds (they render as a disabled placeholder) and stops serving them on public share pages.": "Отключение этой опции скрывает существующие вставки (они отображаются как отключённая заглушка) и прекращает их показ на публичных страницах.",
"Analytics / tracker": "Аналитика / трекер",
"Injected verbatim into the <head> of PUBLIC SHARE pages only (same-origin). For analytics snippets (Google Analytics, Yandex.Metrika, etc.). Admin only.": "Вставляется дословно в <head> только ПУБЛИЧНЫХ страниц (тот же источник). Для сниппетов аналитики (Google Analytics, Яндекс.Метрика и т. п.). Только для администраторов.",
"Go to login page": "Перейти на страницу входа",
"Move to space": "Переместить в пространство",
"Float left (wrap text)": "Обтекание слева",
"Float right (wrap text)": "Обтекание справа",
"Inline (side by side)": "В ряд",
@@ -1199,6 +1351,7 @@
"Showing {{count}} subpages_one": "Показано {{count}} подстраница",
"Showing {{count}} subpages_few": "Показано {{count}} подстраницы",
"Showing {{count}} subpages_many": "Показано {{count}} подстраниц",
"Showing {{count}} subpages_other": "Показано {{count}} подстраниц",
"Protocol": "Протокол",
"How chat requests are sent and how reasoning is surfaced": "Как отправляются запросы чата и как показывается reasoning",
"OpenAI-compatible (surfaces reasoning)": "OpenAI-совместимый (показывает reasoning)",
@@ -1268,7 +1421,6 @@
"Retry": "Повторить",
"The catalog is empty": "Каталог пуст",
"No role bundles are published for this language yet. Try switching the content language.": "Для этого языка ещё не опубликовано ни одного набора ролей. Попробуйте сменить язык контента.",
"No roles configured": "Роли не настроены",
"Already up to date": "Уже актуальна",
"Updated to the latest version": "Обновлено до последней версии",
"This role is no longer in the catalog": "Эта роль больше не представлена в каталоге",
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { isChunkLoadError } from "./chunk-load-error-boundary";
import { isChunkLoadError, shouldAutoReload } from "./chunk-load-error-boundary";
// The detector decides whether a caught render error is a stale-deploy chunk-404
// (→ auto-reload to fetch the new manifest) vs a genuine app error (→ generic
@@ -35,3 +35,31 @@ describe("isChunkLoadError", () => {
expect(isChunkLoadError(err)).toBe(false);
});
});
// The window gate replaces the old one-shot flag: it must permit recovery across
// several deploys in one tab (each > window apart) while still stopping an infinite
// reload loop when a lazy chunk is permanently broken (a second failure < window).
describe("shouldAutoReload", () => {
const WINDOW = 5 * 60 * 1000;
const NOW = 1_000_000_000_000;
it("allows a reload when we have never auto-reloaded", () => {
expect(shouldAutoReload(NOW, null, WINDOW)).toBe(true);
});
it("allows a reload when the last one was 6 minutes ago (outside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 6 * 60 * 1000, WINDOW)).toBe(true);
});
it("blocks a reload when the last one was 1 minute ago (inside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 1 * 60 * 1000, WINDOW)).toBe(false);
});
it("blocks a reload exactly at the window boundary (not strictly older)", () => {
expect(shouldAutoReload(NOW, NOW - WINDOW, WINDOW)).toBe(false);
});
it("allows a reload when the stored timestamp is unparseable (NaN)", () => {
expect(shouldAutoReload(NOW, NaN, WINDOW)).toBe(true);
});
});
@@ -2,7 +2,25 @@ import { ReactNode } from "react";
import { ErrorBoundary } from "react-error-boundary";
import { Button, Center, Stack, Text } from "@mantine/core";
const RELOAD_FLAG = "chunk-reload-attempted";
// sessionStorage key holding the epoch-ms timestamp of the last automatic reload.
const RELOAD_AT_KEY = "chunk-reload-at";
// Allow at most one automatic reload per this window. A stale-deploy 404 is cured
// by a single reload, so anything inside the window is treated as a reload loop
// (permanently-broken chunk) and falls through to the manual UI. A window (rather
// than a one-shot flag) lets a SECOND deploy in the same tab's lifetime recover too.
const RELOAD_WINDOW_MS = 5 * 60 * 1000;
// Pure window decision, unit-tested in isolation: auto-reload only if we have never
// auto-reloaded (lastReloadAt null/NaN) or the last one was strictly older than the
// window. Anything inside the window is suppressed to break an infinite reload loop.
export function shouldAutoReload(
now: number,
lastReloadAt: number | null,
windowMs: number,
): boolean {
if (lastReloadAt === null || Number.isNaN(lastReloadAt)) return true;
return now - lastReloadAt > windowMs;
}
// Heuristic detection of a failed dynamic import. Since the code-splitting work,
// every route (plus Aside / AiChatWindow) is React.lazy: when a new deploy
@@ -24,12 +42,16 @@ export function isChunkLoadError(error: unknown): boolean {
function handleError(error: unknown) {
if (!isChunkLoadError(error)) return;
// A stale-chunk 404 is cured by a full reload that re-fetches index.html and
// the new chunk manifest. Auto-reload once, guarding against a reload loop
// (e.g. a genuinely missing chunk) with a one-shot sessionStorage flag. If the
// flag is already set we fall through to the manual recovery UI below.
// the new chunk manifest. Auto-reload at most once per RELOAD_WINDOW_MS: this
// recovers across multiple deploys in a single tab's lifetime, yet a
// permanently-broken lazy chunk (which would loop) is stopped after the first
// reload and falls through to the manual recovery UI below.
try {
if (sessionStorage.getItem(RELOAD_FLAG)) return;
sessionStorage.setItem(RELOAD_FLAG, "1");
const raw = sessionStorage.getItem(RELOAD_AT_KEY);
const lastReloadAt = raw === null ? null : Number.parseInt(raw, 10);
const now = Date.now();
if (!shouldAutoReload(now, lastReloadAt, RELOAD_WINDOW_MS)) return;
sessionStorage.setItem(RELOAD_AT_KEY, String(now));
} catch {
// sessionStorage unavailable (private mode / disabled): skip the automatic
// reload rather than risk an unguarded loop; the fallback UI still recovers.
@@ -86,11 +86,19 @@ const MIN_HEIGHT = 400;
// Margin kept between the window and the viewport edges while dragging.
const EDGE_MARGIN = 8;
// #184 phase 1.5 / #430 / #488: the degraded-poll fallback. The window owns only
// a DUMB 2.5s timer, gated by an armed flag; the THREAD's run-lifecycle FSM owns
// arm/disarm AND the inactivity cap that turns a stuck run into a `stalled` banner
// (#488 commit 4a — the cap moved into the thread so polling->stalled is a single
// FSM transition; the window no longer silently stops polling at the cap).
// #184 phase 1.5 / #430: backstop for the degraded-poll fallback. The poll is
// armed when a resume attempt could not attach to the live run and disarmed by the
// thread on settle / local stream; this cap is the ONLY backstop against an endless
// tick (a stuck 'streaming' row before the boot-sweep, or a user-tail 204 with no
// run).
//
// #430: measured from RUN ACTIVITY, not from arm-time. A real autonomous run takes
// 11-25 min — longer than a fixed 10-min-from-start cap, which used to cut the poll
// off mid-run. Instead we cap on INACTIVITY: keep polling as long as the run is
// still making progress (its persisted rows keep changing), and only give up after
// this long with NO new activity. A genuinely stuck run produces no row changes, so
// the idle cap still bounds it; a long-but-progressing run polls to completion.
const DEGRADED_POLL_IDLE_MAX_MS = 10 * 60_000;
/** Compact token formatter: 1.2M / 3.4k / 950. */
function formatTokens(n: number): string {
@@ -251,13 +259,17 @@ export default function AiChatWindow() {
[roles],
);
// #184 phase 1.5 / #488: degraded-poll fallback. ChatThread's FSM arms this via
// onResumeFallback(true) when it enters a poll-bearing recovery (attach 204 /
// starved finish / stop) and disarms it on settle / local stream / stalled. The
// window owns ONLY the dumb 2.5s timer; the THREAD owns arm/disarm AND the
// inactivity cap (a stuck run -> the thread's `stalled` banner disarms this).
// #184 phase 1.5: degraded-poll fallback (replaces the F4/F5/F7 latches). When
// ChatThread could not attach to a still-running run it arms this via
// onResumeFallback(true); the thread disarms it on settle / local stream. The
// window only OWNS the timer (armedAtRef stamps when it was armed for the cap).
const [degradedPoll, setDegradedPoll] = useState(false);
// #430: timestamp of the LAST run activity while the poll is armed — stamped on
// arm and re-stamped whenever the polled rows change (see the effect below). The
// idle cap is measured from this, so a long-but-progressing run keeps polling.
const lastActivityAtRef = useRef(0);
const onResumeFallback = useCallback((active: boolean): void => {
if (active) lastActivityAtRef.current = Date.now();
setDegradedPoll(active);
}, []);
// Reset the degraded poll whenever the open chat changes: it is scoped to the
@@ -269,17 +281,33 @@ export default function AiChatWindow() {
const { data: messageRows, isLoading: messagesLoading } =
useAiChatMessagesQuery(
activeChatId ?? undefined,
// DELIBERATELY DUMB: poll every 2.5s WHILE ARMED, otherwise off. NO error
// checks (TanStack resets fetchFailureCount each fetch; the poll must survive
// a server restart), NO tail checks, NO cap here — the settled/stalled/idle-cap
// semantics all live in ChatThread's FSM, which disarms via onResumeFallback.
() => (degradedPoll === true ? 2500 : false),
// DELIBERATELY DUMB (invariant 8 / task 2.4): poll every 2.5s while armed
// and while the run is still active (#430: under the INACTIVITY cap, not a
// fixed-from-start cap); otherwise off. NO error checks (TanStack v5 resets
// fetchFailureCount each fetch, so consecutive errors are not expressible —
// and the poll must survive a server restart) and NO tail checks (the
// settled/local-stream semantics live in ChatThread, which disarms via
// onResumeFallback(false)). The idle cap is the only backstop.
() =>
degradedPoll === true &&
Date.now() - lastActivityAtRef.current < DEGRADED_POLL_IDLE_MAX_MS
? 2500
: false,
// #344: gate on windowOpen too — no message history is fetched (and no
// degraded poll runs) while the window is closed; it loads when the window
// opens with an active chat.
windowOpen,
);
// #430: re-stamp the activity clock whenever the polled rows change while the
// poll is armed. TanStack keeps the same `messageRows` reference across refetches
// that return deep-equal data (structural sharing), so a new reference means the
// run genuinely progressed — which extends the inactivity cap above. A stuck run
// yields no reference change, so the cap eventually fires and stops the poll.
useEffect(() => {
if (degradedPoll) lastActivityAtRef.current = Date.now();
}, [degradedPoll, messageRows]);
// #184 reconnect-and-live-follow. Whether detached agent runs are enabled for
// this workspace. When the feature is off no runs are ever created, so the
// resume attempt would only ever 204; gating ChatThread's resume on it avoids a
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -57,25 +57,6 @@ export async function stopRun(
return req.data;
}
/**
* #488: the run-fact "is a run active on this chat?" first-class from the
* server (POST /ai-chat/run). Called on mount to seed the client FSM's run-fact
* and to VERIFY after a supersede mismatch (an observer following a superseded
* run asks for the latest run and follows it). Returns the latest run row (with
* its `id` and `status`) and its projected assistant message, or `run: null` when
* the chat has never had a run. Owner-gated server-side.
*/
export async function getRun(chatId: string): Promise<{
run: { id: string; status: string } | null;
message: IAiChatMessageRow | null;
}> {
const req = await api.post<{
run: { id: string; status: string } | null;
message: IAiChatMessageRow | null;
}>("/ai-chat/run", { chatId });
return req.data;
}
/**
* Resolve the chat bound to a document (the current user's most-recent chat
* created on that page), or null when there is none. Drives auto-open-on-page.
@@ -1,183 +0,0 @@
# AI-chat run-lifecycle FSM — design spec (#488)
This is the written design that `run-fsm.ts` implements. It ships in the PR (issue
#488 commit 1: "the spec is written FIRST and enters the PR"). It has four parts:
(1) the event × state transition table, (2) the map of every `chat-thread.tsx` ref
to {FSM state | FSM context | stays data}, (3) the run-fact protocol, (4) the
invariants.
The reducer is a **pure function** `reduce(machine, event) → machine`. The returned
machine carries the **command effects** for that transition; a thin runtime in
`chat-thread.tsx` dispatches events and executes effects. Because it is pure, the
whole machine is enumerable and unit-tested directly (event × state → next state is
the observable property) — see `run-fsm.test.ts`.
---
## 1. Event × state transition table
Phases: `idle | sending | streaming | attaching | reconnecting(attempt,failed) |
polling(reason) | stalled | stopping | superseding | error(kind)`.
Context (orthogonal): `epoch`, `ownership: local|observer`, `runFact: {runId}|null`,
`liveFollow` (are we following a live run we locally streamed — the reconnect
ladder — vs a one-shot mount-attach resume? both are `observer`, but a live-follow
drop RE-ENTERS the ladder (#488 commit 3) while a mount-resume drop polls).
Legend: **†** = command-transition (bumps `epoch`, I1). Effects in `[…]`.
| Event (source) | From phase(s) | → To phase | Effects / ctx |
|---|---|---|---|
| `SEND_LOCAL` (user send) | idle, error, polling, stalled, reconnecting | sending **†** | `[cancelReconnect, disarmPoll]`, ownership=local |
| `STREAM_START{runId}` (SDK `start` metadata) | sending, attaching, reconnecting, superseding | streaming | `[cancelReconnect, disarmPoll]`, runFact←runId |
| `FINISH_CLEAN` (onFinish clean) | streaming, … | idle | `[disarmPoll, cancelReconnect]`, runFact←null |
| `FINISH_ABORT` (onFinish isAbort) | streaming, stopping | idle | `[disarmPoll, cancelReconnect]`, runFact←null (I4 exits stopping by this DATA) |
| `FINISH_DISCONNECT` (observer, NOT liveFollow) | streaming(observer) | polling(disconnect-visible) | `[armPoll]` (a mount-resume drop polls) |
| `FINISH_DISCONNECT{hasVisibleContent}` (local drop OR liveFollow) | streaming | reconnecting(1) **†** *iff runFact\|liveFollow* | `[scheduleReconnect(1)]` (+`armPoll` if visible), ownership=observer, liveFollow=true (commit 3: repeatable) |
| `FINISH_DISCONNECT` (no runFact, not liveFollow) | streaming | idle | runFact←null (plain terminal "connection lost") |
| `STREAM_INCOMPLETE{reason}` (observer starved/torn clean finish) | streaming(observer) | polling(reason) | `[armPoll(reason)]` |
| `FINISH_ERROR{kind}` (onFinish isError) | any | error(kind) | `[disarmPoll, cancelReconnect]`, runFact←null |
| `STREAM_START{runId}` (first assistant frame of a local turn) | sending | streaming | runFact←runId, `[cancelReconnect, disarmPoll]` |
| `ATTACH_START{runId}` (mount resume) | **idle only** (F2) | attaching **†** | `[resumeStream]`, ownership=observer, runFact←runId; ignored from any non-idle phase |
| `ATTACH_LIVE` (attach GET 2xx) | attaching | streaming | — |
| `ATTACH_NONE` (attach GET 204/err/throw) | attaching | polling(attach-none) | `[armPoll(attach-none)]` |
| `RECONNECT_ATTEMPT{n}` (backoff timer) | reconnecting | reconnecting(n) **†** | `[resumeStream]` |
| `RECONNECT_ATTACHED` (reconnect GET 2xx) | reconnecting | streaming | `[cancelReconnect, disarmPoll]`**counter reset** (commit 3) |
| `RECONNECT_NONE` (reconnect GET 204/err), attempt<MAX | reconnecting | reconnecting(n+1) **†** | `[armPoll(attach-none), scheduleReconnect(n+1)]` |
| `RECONNECT_NONE`, attempt=MAX | reconnecting | reconnecting(MAX, failed) | `[armPoll(reconnect-exhausted)]` |
| `RETRY` (manual, failed banner) | reconnecting(failed) | reconnecting(1) **†** | `[resumeStream]` |
| `RETRY` (manual, stalled banner) | stalled | polling(attach-none) **†** | `[armPoll]` |
| `POLL_TERMINAL` (settled tail merged) | polling, reconnecting, stopping | idle | `[disarmPoll, cancelReconnect]`, runFact←null (I4) |
| `POLL_IDLE_CAP` (inactivity cap) | polling, reconnecting | stalled | `[disarmPoll, cancelReconnect]` (commit 4a — no more silent) |
| `RUN_FACT{null}` (POST /run → null/terminal, 204) | reconnecting/attaching/polling/stopping | idle | `[cancelReconnect, disarmPoll]`, runFact←null (I3 fresh-negative gate) |
| `RUN_FACT{runId}` | any | (same) | runFact←runId (pessimism toward an attempt) |
| `STOP_REQUESTED` (user Stop) | streaming, reconnecting, polling | stopping **†** | `[stopRun, abortAttach, cancelReconnect, armPoll]` (poll drives the terminal — I4 exit by data) |
| `SUPERSEDE_REQUESTED{targetRunId}` (interrupt+send) | streaming, reconnecting, polling, error | superseding **†** | `[supersede(target), cancelReconnect, disarmPoll]` |
| `SUPERSEDE_READY{runId}` (CAS ok) | superseding | streaming | ownership=local, runFact←runId |
| `SUPERSEDE_MISMATCH{currentRunId}` (409 SUPERSEDE_TARGET_MISMATCH) | superseding | error(supersede-mismatch) | `[postRun(verify)]`, runFact←currentRunId |
| `SUPERSEDE_TIMEOUT` (409 SUPERSEDE_TIMEOUT) | superseding | error(supersede-timeout) | — (composer keeps text; no auto-retry) |
| `SUPERSEDE_INVALID` (409 SUPERSEDE_INVALID) | superseding | error(supersede-invalid) | — |
| `RUN_ALREADY_ACTIVE{activeRunId}` (409 A_RUN_ALREADY_ACTIVE, plain POST) | sending | error(run-already-active) | runFact←activeRunId (composer offers supersede; NO auto-retry) |
| `DISPOSE` (unmount) | any | idle **†** | `[abortAttach, cancelReconnect, disarmPoll]` (I1/I5 — epoch++ kills late callbacks) |
**`stopping` honors any finish (re-review MEDIUM):** BEFORE the epoch filter, a
stream finish (`FINISH_*`/`STREAM_INCOMPLETE`) arriving in phase `stopping` exits
`stopping -> idle` regardless of generation. A plain Stop has no successor stream,
so the aborted stream's finish IS the expected end (I4 exit by data) — and it
carries the PRE-stop generation (STOP_REQUESTED bumped the epoch), so the filter
would otherwise strand the machine in `stopping` (no idle-cap covers it). The filter
stays in force for `superseding` (that is the F1 supersede drop).
**Epoch filter (I1):** the reducer then drops any event carrying an `epoch` that
does not equal the current `ctx.epoch`. Outcome events (`STREAM_START`, `ATTACH_*`,
`RECONNECT_*`, `SUPERSEDE_*`, **`FINISH_*`/`STREAM_INCOMPLETE`**, `RUN_FACT`) are
stamped with the generation the corresponding STREAM started under (the runtime
holds a per-owned-stream `turnEpoch`); trigger events (user actions, fresh
disconnects) carry no epoch. **F1:** this is what makes a SUPERSEDED stream's late
`onFinish` (a dead stream A closing after the CAS started stream B) get dropped, so
A cannot drive the live new run into a false reconnect or reset its run-fact. The
supersede path additionally ABORTS A and starts B only from A's onFinish (a
microtask), because ai@6 `AbstractChat.makeRequest` corrupts overlapping streams
(A's `finally` reads then nulls the shared `activeResponse`).
**Removed events (scope-cut, internal review):** `RUN_SUPERSEDED` (a ghost feature —
never dispatched; the observer-superseded case is handled by the degraded poll,
which follows the latest rows regardless of runId), `RECONNECT_BEGIN` (reconnect is
entered by `FINISH_DISCONNECT`), and `POLL_ACTIVITY` (the window's activity clock was
removed when the idle-cap moved into the thread). The reducer and this table now
share exactly the dispatched event set.
### 409-code → event map (the real #487 contract consumed here)
| Server response | Event dispatched | error kind → banner |
|---|---|---|
| 409 `A_RUN_ALREADY_ACTIVE` (+ body.activeRunId) | `RUN_ALREADY_ACTIVE{activeRunId}` | run-already-active → "already answering / interrupt & send" |
| 409 `SUPERSEDE_TARGET_MISMATCH` (+ body.activeRunId) | `SUPERSEDE_MISMATCH{currentRunId}` | supersede-mismatch → verify via /run |
| 409 `SUPERSEDE_TIMEOUT` | `SUPERSEDE_TIMEOUT` | supersede-timeout → "couldn't interrupt in time, resend" |
| 409 `SUPERSEDE_INVALID` | `SUPERSEDE_INVALID` | supersede-invalid → "couldn't interrupt this run" |
| 503 `A_RUN_BEGIN_FAILED` | `FINISH_ERROR{begin-failed}` | begin-failed → "could not start, temporary" |
---
## 2. Ref-map — every `chat-thread.tsx` ref → its new home (MIGRATION RESOLVED)
The migration is COMPLETE: the 13 run-lifecycle FLAGS below are GONE from
`chat-thread.tsx` (collapsed into FSM phase/ctx/effects, or deleted). What remains
are identity/data mirrors, effect-owned controllers/timers, and ONE React-liveness
bit — none of which is a run-lifecycle flag, so the post-merge "no new flags" rule
holds. **Pending column: empty.**
| # | Old ref | Resolved to | Where now |
|---|---|---|---|
| 1 | `reconcileTailRef` | **FSM phase** | reconcile-merge gated on `phase ∈ {polling, reconnecting, stopping}` |
| 2 | `noStreamHandledRef` | **FSM epoch (I1)** | the attach outcome's epoch guard drops the stale/second outcome |
| 3 | `onNoActiveStreamRef` | **FSM event** | transport → `handleAttachOutcome` dispatches `ATTACH_NONE`/`RECONNECT_NONE` |
| 4 | `onReconnectAttachedRef` | **FSM event** | transport dispatches `ATTACH_LIVE` / `RECONNECT_ATTACHED` |
| 5 | `resumedTurnRef` + `resumedTurn` state | **FSM ctx `ownership`** | `ownership==='observer'` ⇒ never flush; hides "Send now" |
| 6 | `reconnectStateRef` + `reconnectState` state | **FSM phase** | `reconnecting(attempt,failed)` renders the banner |
| 7 | `reconnectTimerRef` | **effect-owned timer** | owned by `scheduleReconnect`/`cancelReconnect` effects (not a flag) |
| 8 | `flushOnAbortRef` | **DELETED** | the stop→flush dance is replaced by the CAS supersede (commit 5) |
| 9 | `interruptNextSendRef` | **DELETED** | the server injects the interrupt note from the supersede itself |
| 10 | `supersedeRetryRef` | **DELETED** (commit 5) | the client 409 retry ladder is gone; CAS supersede replaces it |
| 11 | `stopPendingRef` | **FSM phase `stopping`** | the deferred stop fires from the chat-id adoption effect while `stopping` |
| 12 | `mountedRef` | **retained (React liveness)** | orthogonal to run-lifecycle; gates imperative onFinish side-effects post-unmount. Epoch (I1) handles stale COMMAND-outcomes; DISPOSE bumps it |
| 13 | `attemptResumeRef` | **FSM `ATTACH_START` + run-fact** | mount arms attach ONLY on a confirmed active run (commit 4b: streaming-tail status, or POST /run for a user tail) |
| 14 | `stripRef` | **data** (attachStrategy) | strip+replay detail; the `resumeStream` effect reads it |
| 15 | `strippedRowRef` | **data** (attachStrategy) | the anchor row |
| 16 | `attachAbortRef` | **effect-owned controller** | aborted by the `abortAttach` effect in cleanup (I5) |
| 17–25 | `chatIdRef`, `openPageRef`, `getEditorSelectionRef`, `roleIdRef`, `stableIdRef`, `queuedRef`, `sendMessageRef`, `statusRef`, `lastForwardedChatIdRef` | **data** (identity/send mirrors) | unchanged — not lifecycle flags |
| NEW | `pendingSupersedeRef` | **data** (send-plumbing) | the runId injected into the next `POST /stream {supersede}`; the single replacement for the 3 DELETED one-shots (#8/#9/#10) — net −2 refs |
| NEW | `idleCapTimerRef` | **effect-owned timer** | the stalled inactivity cap → `POLL_IDLE_CAP` (commit 4a); not a flag |
Net: the 13 lifecycle flags (#1#13) are eliminated: **8** → FSM phase/ctx/epoch/event
(#1#6, #11, #13), **3** deleted (#8/#9/#10), **`reconnectTimerRef` (#7)** becomes an
effect-owned controller, and **`mountedRef` (#12)** is retained as React liveness
(8 + 3 + 1 + 1 = 13). (`attachAbortRef` (#16) is outside the #1#13 set — it was
already an effect-owned controller.) Two effect-owned timers + one send-plumbing data
ref are added — none is a boolean lifecycle latch.
---
## 3. Run-fact protocol (`runFact: {runId} | null`) — I3
"A run is active" is first-class from the SERVER, not inferred from an assistant
message. Sources, in the order they update `ctx.runFact`:
1. **Init (mount):** `POST /ai-chat/run { chatId }``{ run, message }`. A `run`
with a non-terminal `status` seeds `runFact = { runId: run.id }`; a null/terminal
run seeds `null`. This is what arms the resume attempt (`ATTACH_START`) — the
attempt is armed ONLY on a positive fact (commit 4b: a user-tail with no active
run no longer arms a pointless poll on every open).
2. **Live update:** the `start` stream metadata carries `runId``STREAM_START{runId}`.
3. **Attach outcomes:** `ATTACH_LIVE` (2xx) confirms active; a 204 on a non-stripped
path is an authoritative NEGATIVE fact → the runtime dispatches `RUN_FACT{null}`,
which cancels recovery (I3 fresh-negative gate).
4. **Poll (future resume-stack iteration #491):** the delta will carry the run field;
until then the poll drives to a terminal ROW, dispatched as `POLL_TERMINAL`.
Pessimism rule: a stale-but-positive fact PERMITS entering recovery (attach); the
204 then cuts it. A fresh negative fact gates recovery OUT immediately.
---
## 4. Invariants
- **I1 — Epoch (generation counter).** Every command-emitting transition bumps
`ctx.epoch`; every async outcome event carries its issuing epoch; the reducer
drops stale-epoch outcomes. Replaces the one-shot-ref zoo (`noStreamHandledRef`,
the flush/interrupt/supersede one-shots, the `mountedRef` late-callback gate).
- **I2 — Ownership is context, not state.** `local | observer` is orthogonal to the
transport phase. The queue flushes ONLY under local ownership; an observer
following a detached run never flushes (was `resumedTurnRef`).
- **I3 — Run-fact is first-class from the server.** Reconnect is entered by the
run-fact, not by an assistant message (commit 2). A fresh negative fact cancels
recovery.
- **I4 — Exit `stopping` by DATA.** A terminal row / negative run-fact / terminal
finish exits `stopping`, never the stopRun HTTP response (which returns after the
abort but before finalization — keying off it would unlock the composer on a 409).
- **I5 — Dispose protocol.** Command controllers (attach GET, POST /stream, POST
/run) are effect-owned and aborted in cleanup (`abortAttach` on `DISPOSE`), not
render-phase refs. A client abort of an already-sent POST does not cancel the
server action, so disarming on unmount is safe.
- **attachStrategy** (strip+replay today) is behind the `resumeStream` effect; the
resume-stack iteration (#491) swaps it to tail-only WITHOUT touching the FSM.
- **Queue** stays a data structure; flush/interrupt decisions are transitions.
@@ -1,482 +0,0 @@
import { describe, it, expect } from "vitest";
import {
reduce,
initialMachine,
reconnectDelayMs,
RECONNECT_MAX_ATTEMPTS,
type Machine,
type Effect,
type Event,
} from "./run-fsm";
// Drive a sequence of events through the reducer, returning the final machine.
function run(m: Machine, ...events: Event[]): Machine {
return events.reduce(reduce, m);
}
function withRunFact(runId = "run-1"): Machine {
return {
...initialMachine(),
ctx: { epoch: 0, ownership: "local", runFact: { runId }, liveFollow: false },
};
}
function effectTypes(m: Machine): string[] {
return m.effects.map((e) => e.type);
}
function hasEffect(m: Machine, type: Effect["type"]): boolean {
return m.effects.some((e) => e.type === type);
}
describe("run-fsm — epoch invariant (I1)", () => {
it("drops an outcome carrying a stale epoch", () => {
// A command bumps the epoch; an outcome stamped with the OLD epoch is dropped.
const m0 = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" }); // epoch 0->1, attaching
expect(m0.ctx.epoch).toBe(1);
expect(m0.phase.name).toBe("attaching");
// A late ATTACH_LIVE from a SUPERSEDED attempt (epoch 0) must NOT drive us.
const stale = reduce(m0, { type: "ATTACH_LIVE", epoch: 0 });
expect(stale.phase.name).toBe("attaching");
expect(stale.effects).toEqual([]);
});
it("applies an outcome carrying the current epoch", () => {
const m0 = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
const live = reduce(m0, { type: "ATTACH_LIVE", epoch: m0.ctx.epoch });
expect(live.phase.name).toBe("streaming");
});
it("an outcome with no epoch is never dropped (trigger events)", () => {
const m0 = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
const disposed = reduce(m0, { type: "DISPOSE" });
expect(disposed.phase.name).toBe("idle");
expect(hasEffect(disposed, "abortAttach")).toBe(true);
});
it("every command-transition increments the epoch exactly once", () => {
let m = initialMachine();
const before = m.ctx.epoch;
m = reduce(m, { type: "SEND_LOCAL" });
expect(m.ctx.epoch).toBe(before + 1);
m = reduce(m, { type: "STOP_REQUESTED" });
expect(m.ctx.epoch).toBe(before + 2);
});
});
describe("run-fsm — local turn", () => {
it("SEND_LOCAL → sending, local ownership, cancels recovery", () => {
const m = reduce(withRunFact(), { type: "SEND_LOCAL" });
expect(m.phase.name).toBe("sending");
expect(m.ctx.ownership).toBe("local");
expect(effectTypes(m)).toEqual(
expect.arrayContaining(["cancelReconnect", "disarmPoll"]),
);
});
it("STREAM_START adopts the runId into the run-fact and goes streaming", () => {
const m = run(initialMachine(), { type: "SEND_LOCAL" });
const s = reduce(m, { type: "STREAM_START", runId: "run-9", epoch: m.ctx.epoch });
expect(s.phase.name).toBe("streaming");
expect(s.ctx.runFact).toEqual({ runId: "run-9" });
});
it("FINISH_CLEAN → idle, run-fact cleared, poll/reconnect disarmed", () => {
const streaming = run(initialMachine(), { type: "SEND_LOCAL" }, { type: "STREAM_START", runId: "r" });
const done = reduce(streaming, { type: "FINISH_CLEAN" });
expect(done.phase.name).toBe("idle");
expect(done.ctx.runFact).toBeNull();
});
});
// #488 commit 2 — SSE break BEFORE the first assistant frame must still recover.
describe("run-fsm — commit 2: reconnect by run-fact, not by assistant message", () => {
it("FINISH_DISCONNECT with an active run-fact → reconnecting (even with no visible content)", () => {
// Setup-phase break: no assistant frame yet, but a run-fact exists.
const streaming = withRunFact("run-2");
const m = reduce(streaming, {
type: "FINISH_DISCONNECT",
hasVisibleContent: false,
epoch: streaming.ctx.epoch,
});
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") expect(m.phase.attempt).toBe(1);
expect(m.ctx.ownership).toBe("observer");
expect(hasEffect(m, "scheduleReconnect")).toBe(true);
// No visible content -> no poll arm yet (the reconnect ladder rebuilds it).
expect(hasEffect(m, "armPoll")).toBe(false);
});
it("FINISH_DISCONNECT WITH visible content also arms the poll", () => {
const m = reduce(withRunFact("run-2"), {
type: "FINISH_DISCONNECT",
hasVisibleContent: true,
epoch: 0,
});
expect(m.phase.name).toBe("reconnecting");
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("FINISH_DISCONNECT with NO run-fact → idle (plain connection-lost)", () => {
const m = reduce(initialMachine(), {
type: "FINISH_DISCONNECT",
hasVisibleContent: true,
epoch: 0,
});
expect(m.phase.name).toBe("idle");
});
});
// #488 commit 3 — a SECOND break after a successful re-attach starts a NEW ladder.
describe("run-fsm — commit 3: repeated reconnect cycles", () => {
it("two breaks in a row produce two reconnect cycles (counter resets on attach)", () => {
let m = withRunFact("run-3");
// First break -> reconnecting(1).
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
// Attempt fires, re-attaches live.
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: 1, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_ATTACHED", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("streaming");
// SECOND break: the counter was reset, so a fresh ladder starts at attempt 1
// (the old one-shot !wasResumed gate would have sent this to silent poll).
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") expect(m.phase.attempt).toBe(1);
expect(hasEffect(m, "scheduleReconnect")).toBe(true);
});
it("a MOUNT-attach observer drop falls to POLL, not the reconnect ladder", () => {
// Distinguishes commit 3 from a one-shot resume: an observer that never
// live-followed (liveFollow false) polls on a drop.
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
expect(m.ctx.ownership).toBe("observer");
expect(m.ctx.liveFollow).toBe(false);
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: true, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("polling");
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("STREAM_INCOMPLETE (observer starved/torn finish) → polling", () => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
m = reduce(m, { type: "STREAM_INCOMPLETE", reason: "starved", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "polling", reason: "starved" });
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("liveFollow is set on the first local drop and kept across a re-attach", () => {
let m = withRunFact("run-3");
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
expect(m.ctx.liveFollow).toBe(true);
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: 1, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_ATTACHED", epoch: m.ctx.epoch });
expect(m.ctx.liveFollow).toBe(true); // kept — so a second drop reconnects
// A clean finish clears it.
m = reduce(m, { type: "FINISH_CLEAN", epoch: m.ctx.epoch });
expect(m.ctx.liveFollow).toBe(false);
});
it("RECONNECT_NONE backs off through the ladder, then fails at the cap", () => {
let m = withRunFact("run-3");
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
for (let n = 1; n < RECONNECT_MAX_ATTEMPTS; n++) {
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: n, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_NONE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") {
expect(m.phase.attempt).toBe(n + 1);
expect(m.phase.failed).toBe(false);
}
// The belt-and-suspenders poll is armed each failed attempt.
expect(hasEffect(m, "armPoll")).toBe(true);
}
// Final attempt fails -> failed banner (Retry), poll armed.
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: RECONNECT_MAX_ATTEMPTS, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_NONE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") expect(m.phase.failed).toBe(true);
// RETRY restarts at attempt 1.
m = reduce(m, { type: "RETRY" });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") {
expect(m.phase.attempt).toBe(1);
expect(m.phase.failed).toBe(false);
}
expect(hasEffect(m, "resumeStream")).toBe(true);
});
it("reconnectDelayMs is the exponential backoff 1s,2s,4s,8s,16s", () => {
expect([1, 2, 3, 4, 5].map(reconnectDelayMs)).toEqual([1000, 2000, 4000, 8000, 16000]);
});
});
// #488 commit 4 — polling stalled-state + user-tail gating.
describe("run-fsm — commit 4: stalled + run-fact gating", () => {
it("POLL_IDLE_CAP: polling → stalled with a banner (poll disarmed), not silent", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("polling");
m = reduce(m, { type: "POLL_IDLE_CAP" });
expect(m.phase.name).toBe("stalled");
expect(hasEffect(m, "disarmPoll")).toBe(true);
});
it("RETRY from stalled re-arms the poll", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
m = reduce(m, { type: "POLL_IDLE_CAP" });
m = reduce(m, { type: "RETRY" });
expect(m.phase.name).toBe("polling");
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("a fresh NEGATIVE run-fact while attaching cancels recovery (user-tail, no active run)", () => {
// The mount POST /run returns no active run: attaching → idle, no poll armed.
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "RUN_FACT", runFact: null, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("idle");
expect(m.ctx.runFact).toBeNull();
expect(hasEffect(m, "disarmPoll")).toBe(true);
});
it("a negative run-fact while polling stops the poll", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
m = reduce(m, { type: "RUN_FACT", runFact: null, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("idle");
});
it("POLL_TERMINAL settles polling → idle (I4 data-driven exit)", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
m = reduce(m, { type: "POLL_TERMINAL" });
expect(m.phase.name).toBe("idle");
expect(m.ctx.runFact).toBeNull();
});
});
// #488 commit 5 — error classification + supersede CAS transitions.
describe("run-fsm — commit 5: supersede CAS + error classification", () => {
it("SUPERSEDE_REQUESTED → superseding, fires the CAS effect, bumps epoch", () => {
const streaming = withRunFact("run-old");
const m = reduce(streaming, { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
expect(m.phase.name).toBe("superseding");
expect(m.ctx.epoch).toBe(streaming.ctx.epoch + 1);
const sup = m.effects.find((e) => e.type === "supersede");
expect(sup).toEqual({ type: "supersede", targetRunId: "run-old" });
});
it("SUPERSEDE_READY → streaming as the new local owner", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_READY", runId: "run-new", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("streaming");
expect(m.ctx.ownership).toBe("local");
expect(m.ctx.runFact).toEqual({ runId: "run-new" });
});
it("SUPERSEDE_MISMATCH → error(supersede-mismatch) + verify via /run (no blind banner)", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_MISMATCH", currentRunId: "run-x", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "error", kind: "supersede-mismatch" });
expect(hasEffect(m, "postRun")).toBe(true);
expect(m.ctx.runFact).toEqual({ runId: "run-x" });
});
it("SUPERSEDE_TIMEOUT → error(supersede-timeout), no auto-retry effect", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_TIMEOUT", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "error", kind: "supersede-timeout" });
expect(m.effects).toEqual([]);
});
it("SUPERSEDE_INVALID → error(supersede-invalid)", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_INVALID", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "error", kind: "supersede-invalid" });
});
it("a stale SUPERSEDE outcome from a superseded epoch is dropped", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
const supersedingEpoch = m.ctx.epoch;
// The user retriggers, bumping the epoch again.
m = reduce(m, { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
// The first CAS's late TIMEOUT (old epoch) must NOT knock us out of superseding.
const late = reduce(m, { type: "SUPERSEDE_TIMEOUT", epoch: supersedingEpoch });
expect(late.phase.name).toBe("superseding");
});
it("RUN_ALREADY_ACTIVE (plain POST gate) → error(run-already-active), no retry effect", () => {
const m = reduce(run(initialMachine(), { type: "SEND_LOCAL" }), { type: "RUN_ALREADY_ACTIVE" });
expect(m.phase).toEqual({ name: "error", kind: "run-already-active" });
expect(m.effects).toEqual([]);
});
it("#497/S4: RUN_ALREADY_ACTIVE{activeRunId} ADOPTS the server's active run as the run-fact", () => {
// The server sends `activeRunId` so a later supersede can TARGET that run
// instead of a blind promote+abort. Absorb it into runFact.
const m = reduce(run(initialMachine(), { type: "SEND_LOCAL" }), {
type: "RUN_ALREADY_ACTIVE",
activeRunId: "run-foreign",
});
expect(m.phase).toEqual({ name: "error", kind: "run-already-active" });
expect(m.ctx.runFact).toEqual({ runId: "run-foreign" });
expect(m.effects).toEqual([]);
});
it("#497/S4: RUN_ALREADY_ACTIVE without an activeRunId keeps the prior run-fact", () => {
const seeded = reduce(run(initialMachine(), { type: "SEND_LOCAL" }), {
type: "RUN_FACT",
runFact: { runId: "run-prior" },
});
const m = reduce(seeded, { type: "RUN_ALREADY_ACTIVE" });
expect(m.ctx.runFact).toEqual({ runId: "run-prior" });
});
});
// #488 F2 — a late mount `getRun → ATTACH_START` must not hijack a local turn.
describe("run-fsm — F2: ATTACH_START only from idle", () => {
it("ATTACH_START from a local `sending` turn is ignored (no observer hijack)", () => {
const sending = reduce(initialMachine(), { type: "SEND_LOCAL" }); // idle -> sending, local
const m = reduce(sending, { type: "ATTACH_START", runId: "r" });
expect(m.phase.name).toBe("sending");
expect(m.ctx.ownership).toBe("local"); // NOT flipped to observer
expect(m.effects).toEqual([]); // no resumeStream
});
it("ATTACH_START from idle attaches as normal", () => {
const m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
expect(m.phase.name).toBe("attaching");
expect(m.ctx.ownership).toBe("observer");
expect(hasEffect(m, "resumeStream")).toBe(true);
});
});
describe("run-fsm — stop (I4: exit by data)", () => {
it("STOP_REQUESTED → stopping, fires stopRun + abortAttach, no data-independent exit", () => {
const m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
expect(m.phase.name).toBe("stopping");
expect(effectTypes(m)).toEqual(expect.arrayContaining(["stopRun", "abortAttach"]));
});
it("stopping exits on the aborted stream's finish carrying the PRE-STOP epoch", () => {
// MEDIUM (#488 re-review): STOP_REQUESTED is a command that BUMPS the epoch, but
// the runtime stamps the aborted stream's onFinish with the stream's START (pre-
// stop) generation — exactly what the component sends. `stopping` must HONOR
// that finish regardless of generation (no idle-cap covers `stopping`).
// MUTATION-VERIFY: remove the honor-in-`stopping` branch and this hangs in
// `stopping` (the epoch filter drops the pre-stop finish) -> red.
const preStopEpoch = withRunFact().ctx.epoch; // E1 (the stream's start epoch)
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" }); // E1 -> E2, stopping
expect(m.ctx.epoch).toBe(preStopEpoch + 1);
m = reduce(m, { type: "FINISH_ABORT", epoch: preStopEpoch }); // NOT the current epoch
expect(m.phase.name).toBe("idle");
expect(m.ctx.runFact).toBeNull();
});
it("stopping exits on a clean finish carrying the pre-stop epoch too", () => {
const preStopEpoch = withRunFact().ctx.epoch;
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
m = reduce(m, { type: "FINISH_CLEAN", epoch: preStopEpoch });
expect(m.phase.name).toBe("idle");
});
it("stopping exits on a negative run-fact (data)", () => {
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
m = reduce(m, { type: "RUN_FACT", runFact: null, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("idle");
});
// Review #4: `stopping` arms the poll but had no inactivity backstop.
it("review-4: POLL_IDLE_CAP in `stopping` exits to idle (bounded), NOT stalled", () => {
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
expect(m.phase.name).toBe("stopping");
expect(hasEffect(m, "armPoll")).toBe(true);
// MUTATION-VERIFY: drop the `stopping` branch in POLL_IDLE_CAP and this hangs
// in `stopping` (poll forever) -> red.
m = reduce(m, { type: "POLL_IDLE_CAP" });
expect(m.phase.name).toBe("idle");
expect(hasEffect(m, "disarmPoll")).toBe(true);
expect(m.ctx.ownership).toBe("local");
});
});
// Review #1: positive attach outcomes must be guarded by the SOURCE phase — the
// epoch filter alone is insufficient because POLL_TERMINAL uses to() (no epoch
// bump) and does not abort the in-flight GET.
describe("run-fsm — review-1: attach outcomes guarded by source phase", () => {
it("a late RECONNECT_ATTACHED after POLL_TERMINAL stays idle (no phantom streaming)", () => {
let m = withRunFact("run-1");
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: true, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: 1, epoch: m.ctx.epoch }); // attach GET
const epoch = m.ctx.epoch;
// The armed degraded poll reaches the terminal row FIRST (epoch unchanged).
m = reduce(m, { type: "POLL_TERMINAL" });
expect(m.phase.name).toBe("idle");
expect(m.ctx.epoch).toBe(epoch); // POLL_TERMINAL did NOT bump the epoch
// The slow GET returns live 2xx under the SAME epoch — must NOT resurrect.
m = reduce(m, { type: "RECONNECT_ATTACHED", epoch });
expect(m.phase.name).toBe("idle");
});
it("a late ATTACH_LIVE / ATTACH_NONE after leaving `attaching` is ignored", () => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
const epoch = m.ctx.epoch;
m = reduce(m, { type: "ATTACH_NONE", epoch }); // attaching -> polling
m = reduce(m, { type: "POLL_TERMINAL" }); // -> idle (epoch unchanged)
expect(m.phase.name).toBe("idle");
m = reduce(m, { type: "ATTACH_LIVE", epoch }); // late 2xx, same epoch
expect(m.phase.name).toBe("idle");
// And a late ATTACH_NONE (not `attaching`) is a no-op too.
m = reduce(m, { type: "ATTACH_NONE", epoch });
expect(m.phase.name).toBe("idle");
});
});
// Review #2: every terminal transition resets ownership to local.
describe("run-fsm — review-2: terminal transitions reset ownership to local", () => {
const observer = (): Machine => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
expect(m.ctx.ownership).toBe("observer");
return m;
};
it("FINISH_CLEAN resets ownership", () => {
const m = reduce(observer(), { type: "FINISH_CLEAN", epoch: observer().ctx.epoch });
expect(m.ctx.ownership).toBe("local");
});
it("FINISH_ERROR / POLL_TERMINAL / RUN_FACT(null) reset ownership", () => {
let o = observer();
expect(reduce(o, { type: "FINISH_ERROR", kind: "stream", epoch: o.ctx.epoch }).ctx.ownership).toBe("local");
// POLL_TERMINAL from an observer polling phase
let p = reduce(observer(), { type: "STREAM_INCOMPLETE", reason: "starved", epoch: observer().ctx.epoch });
expect(reduce(p, { type: "POLL_TERMINAL" }).ctx.ownership).toBe("local");
// RUN_FACT(null) from an observer attaching phase
let a = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
expect(reduce(a, { type: "RUN_FACT", runFact: null, epoch: a.ctx.epoch }).ctx.ownership).toBe("local");
});
});
describe("run-fsm — ownership (I2) is context, orthogonal to phase", () => {
it("attach/reconnect set observer; send/supersede-ready set local", () => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
expect(m.ctx.ownership).toBe("observer");
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("streaming");
expect(m.ctx.ownership).toBe("observer"); // still observing a detached run
// A local send flips ownership back to local.
m = reduce(m, { type: "SEND_LOCAL" });
expect(m.ctx.ownership).toBe("local");
});
});
describe("run-fsm — dispose (I5)", () => {
it("DISPOSE from any phase aborts controllers and bumps epoch", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
const before = m.ctx.epoch;
m = reduce(m, { type: "DISPOSE" });
expect(m.phase.name).toBe("idle");
expect(m.ctx.epoch).toBe(before + 1);
expect(effectTypes(m)).toEqual(
expect.arrayContaining(["abortAttach", "cancelReconnect", "disarmPoll"]),
);
});
});
@@ -1,600 +0,0 @@
/**
* Run-lifecycle finite state machine for a single AI-chat thread (#488).
*
* ============================================================================
* WHY THIS EXISTS
* ----------------------------------------------------------------------------
* The resume/reconnect/poll/stop/supersede lifecycle used to be spread across
* ~26 `useRef` one-shot flags in `chat-thread.tsx`, each disarmed "on every
* path". Ownerless flag combinations produced silent UI freezes, and every fix
* added another ref (the #381 -> #432 -> #456 spiral). This module replaces that
* ref-zoo with ONE pure reducer whose transitions are enumerable and unit-
* testable in isolation (event x state -> next state is the observable property).
*
* The reducer is PURE: it owns no timers, no fetches, no React state. It maps
* `(machine, event) -> machine`, where the returned machine carries the list of
* COMMAND EFFECTS to run for that transition. A thin runtime in `chat-thread.tsx`
* dispatches events (from SDK callbacks / HTTP outcomes) and executes the
* effects (attach GET, POST /stream, POST /run, POST /stop, backoff timers,
* poll arm/disarm). The runtime lives in a THREAD, not the window, so a late SDK
* callback dies with the owner (kills the "event from a dead view" class, #161).
*
* ============================================================================
* INVARIANTS (see run-fsm.spec.md for the full spec + tables)
* ----------------------------------------------------------------------------
* I1 EPOCH (generation counter). Commands (`resumeStream`, `postRun`, `stop`,
* `supersede`, `scheduleReconnect`) are async; their outcomes arrive on the
* SAME SDK/HTTP callbacks. Every command-emitting transition increments
* `ctx.epoch`; every OUTCOME event carries the epoch it was issued under;
* the reducer DROPS an outcome whose epoch != the current epoch. This is
* what the one-shot-ref zoo used to approximate by hand.
* I2 OWNERSHIP is a CONTEXT FIELD (`'local' | 'observer'`), not a state
* orthogonal to the transport phase. The queue is flushed ONLY by a local
* owner (an observer following a detached run never flushes).
* I3 RUN-FACT ("a run is active") is first-class from the server: `runFact`
* holds the server-confirmed active run id (POST /run on mount, the `start`
* metadata runId, attach outcomes). Reconnect is entered by the RUN-FACT,
* not by the presence of an assistant message (#488 commit 2). A fresh
* negative fact (null) cancels reconnect immediately.
* I4 Exit `stopping` by DATA (a terminal row / negative run-fact), NEVER by the
* stopRun HTTP response (which returns after abort, before finalization).
* I5 Command controllers are effect-owned (abort in cleanup), NOT render-phase
* refs expressed here as the `abortAttach` effect on disposing transitions.
* ============================================================================
*/
// ---------------------------------------------------------------------------
// Phases (the transport lifecycle). Ownership / runFact are CONTEXT, not here.
// ---------------------------------------------------------------------------
/** Why the degraded poll is the active recovery. */
export type PollReason =
| "attach-none" // mount attach returned 204 / error — nothing live to attach
| "starved" // a resumed finish carried no visible content
| "disconnect-visible" // a live disconnect WITH on-screen content — poll to terminal
| "reconnect-exhausted"; // the live re-attach ladder gave up
/** The classified error kind (drives the banner text + composer behavior). */
export type ErrorKind =
| "stream" // a generic provider/network stream error (useChat error)
| "run-already-active" // 409 A_RUN_ALREADY_ACTIVE (a plain POST hit the gate)
| "supersede-mismatch" // 409 SUPERSEDE_TARGET_MISMATCH (CAS target moved)
| "supersede-timeout" // 409 SUPERSEDE_TIMEOUT (old run did not settle in W)
| "supersede-invalid" // 409 SUPERSEDE_INVALID (bad supersede target)
| "begin-failed"; // 503 A_RUN_BEGIN_FAILED (could not start the run)
export type Phase =
| { name: "idle" }
| { name: "sending" } // local POST in flight, before the first frame
| { name: "streaming" } // receiving frames
| { name: "attaching" } // mount-time attach GET in flight
| { name: "reconnecting"; attempt: number; failed: boolean }
| { name: "polling"; reason: PollReason }
| { name: "stalled" } // poll hit the inactivity cap — banner + Retry
| { name: "stopping" }
| { name: "superseding" }
| { name: "error"; kind: ErrorKind };
export type Ownership = "local" | "observer";
/** The server-confirmed active run, or null when no run is active. */
export type RunFact = { runId: string } | null;
export interface Ctx {
/** I1: generation counter — every command-transition increments it. */
epoch: number;
/** I2: does THIS client own the turn's writes (local streamer) or observe? */
ownership: Ownership;
/** I3: the server-confirmed active run. */
runFact: RunFact;
/**
* Are we FOLLOWING a live run we were locally streaming (the reconnect ladder),
* as opposed to a one-shot mount-attach resume? Both are `ownership: 'observer'`,
* but they recover DIFFERENTLY on a drop: a live-follow drop RE-ENTERS the
* reconnect ladder (#488 commit 3 the second break after a successful re-attach
* must reconnect again, not fall to silent poll), while a mount-resume drop falls
* to the degraded poll. This is the ctx bit that separates the two WITHOUT a new
* component ref (it is why commit 3 needs the FSM, not a surgical patch).
*/
liveFollow: boolean;
}
export interface Machine {
phase: Phase;
ctx: Ctx;
/** Command effects to run for the transition that produced THIS machine.
* The runtime executes them and does not read them again. */
effects: Effect[];
}
// ---------------------------------------------------------------------------
// Command effects (the reducer's only side-channel — executed by the runtime).
// ---------------------------------------------------------------------------
export type Effect =
/** POST /run to (re)establish or verify the run-fact. `reason` is diagnostic. */
| { type: "postRun"; reason: "mount" | "verify" }
/** Trigger the SDK `resumeStream()` (attach GET via prepareReconnectToStream). */
| { type: "resumeStream" }
/** Schedule a reconnect attempt after a backoff, then dispatch RECONNECT_ATTEMPT. */
| { type: "scheduleReconnect"; attempt: number; delayMs: number }
/** Cancel any pending reconnect backoff timer. */
| { type: "cancelReconnect" }
/** Arm the degraded poll (the window's dumb timer follows the run in the DB). */
| { type: "armPoll"; reason: PollReason }
/** Disarm the degraded poll. */
| { type: "disarmPoll" }
/** POST /stop the chat's active run (authoritative detached-run stop). */
| { type: "stopRun" }
/** POST /stream { supersede: { runId } } — the CAS "interrupt and send now". */
| { type: "supersede"; targetRunId: string }
/** Abort the in-flight attach/reconnect GET controller (dispose / observer stop). */
| { type: "abortAttach" };
// ---------------------------------------------------------------------------
// Events. An OUTCOME event MAY carry `epoch`; if it does and it does not equal
// the current epoch, the reducer drops it (I1). Trigger events (user actions,
// fresh disconnects) carry no epoch and are never dropped.
// ---------------------------------------------------------------------------
export type Event =
// -- local turn --
| { type: "SEND_LOCAL" }
| { type: "STREAM_START"; runId?: string; epoch?: number }
/** An OBSERVER's attached stream ended WITHOUT reaching terminal (a starved
* clean replay, or a torn resume) fall to the degraded poll to drive the row
* to its real terminal state. (A live-follow drop uses FINISH_DISCONNECT.) */
| { type: "STREAM_INCOMPLETE"; reason: PollReason; epoch?: number }
| { type: "FINISH_CLEAN"; epoch?: number }
| { type: "FINISH_ABORT"; epoch?: number }
| { type: "FINISH_DISCONNECT"; hasVisibleContent: boolean; epoch?: number }
| { type: "FINISH_ERROR"; kind: ErrorKind; epoch?: number }
// -- mount attach (resume) --
| { type: "ATTACH_START"; runId?: string }
| { type: "ATTACH_LIVE"; epoch?: number }
| { type: "ATTACH_NONE"; epoch?: number }
// -- reconnect after a live disconnect (entered by FINISH_DISCONNECT, #488 c2) --
| { type: "RECONNECT_ATTEMPT"; attempt: number; epoch?: number }
| { type: "RECONNECT_ATTACHED"; epoch?: number }
| { type: "RECONNECT_NONE"; epoch?: number }
| { type: "RETRY" }
// -- degraded poll --
| { type: "POLL_TERMINAL" }
| { type: "POLL_IDLE_CAP" }
// -- run-fact (server-confirmed active run) --
| { type: "RUN_FACT"; runFact: RunFact; epoch?: number }
// -- stop --
| { type: "STOP_REQUESTED" }
// -- supersede (CAS) --
| { type: "SUPERSEDE_REQUESTED"; targetRunId: string }
| { type: "SUPERSEDE_READY"; runId?: string; epoch?: number }
| { type: "SUPERSEDE_MISMATCH"; currentRunId?: string; epoch?: number }
| { type: "SUPERSEDE_TIMEOUT"; epoch?: number }
| { type: "SUPERSEDE_INVALID"; epoch?: number }
| { type: "RUN_ALREADY_ACTIVE"; activeRunId?: string }
// -- lifecycle --
| { type: "DISPOSE" };
export const RECONNECT_MAX_ATTEMPTS = 5;
export const RECONNECT_BASE_DELAY_MS = 1000;
/** Backoff before attempt N (1-based): 1s, 2s, 4s, 8s, 16s. */
export function reconnectDelayMs(attempt: number): number {
return RECONNECT_BASE_DELAY_MS * 2 ** (attempt - 1);
}
// ---------------------------------------------------------------------------
// Constructors / helpers.
// ---------------------------------------------------------------------------
export function initialMachine(overrides?: Partial<Ctx>): Machine {
return {
phase: { name: "idle" },
ctx: { epoch: 0, ownership: "local", runFact: null, liveFollow: false, ...overrides },
effects: [],
};
}
/** Build a machine result: a phase, optional ctx patch, and effects. Empty
* effects by default. Never mutates the input. */
function to(
m: Machine,
phase: Phase,
opts?: { ctx?: Partial<Ctx>; effects?: Effect[] },
): Machine {
return {
phase,
ctx: { ...m.ctx, ...(opts?.ctx ?? {}) },
effects: opts?.effects ?? [],
};
}
/** No transition: keep the phase, clear effects (so a re-run does not re-fire). */
function stay(m: Machine): Machine {
return { phase: m.phase, ctx: m.ctx, effects: [] };
}
/** A command-transition: same as `to` but bumps the epoch (I1). Any outcome
* event issued under the old epoch is dropped once this lands. */
function command(
m: Machine,
phase: Phase,
effects: Effect[],
ctx?: Partial<Ctx>,
): Machine {
return {
phase,
ctx: { ...m.ctx, ...(ctx ?? {}), epoch: m.ctx.epoch + 1 },
effects,
};
}
// ---------------------------------------------------------------------------
// The pure reducer.
// ---------------------------------------------------------------------------
/** The terminal stream-finish events (one turn's stream ended). */
function isFinishEvent(event: Event): boolean {
return (
event.type === "FINISH_ABORT" ||
event.type === "FINISH_CLEAN" ||
event.type === "FINISH_DISCONNECT" ||
event.type === "FINISH_ERROR" ||
event.type === "STREAM_INCOMPLETE"
);
}
export function reduce(m: Machine, event: Event): Machine {
// MEDIUM (#488 re-review): honor ANY stream finish in `stopping` regardless of
// generation. A plain user Stop has NO successor stream — the aborted stream's
// finish IS the expected end of the stop, so exit `stopping -> idle` by that DATA
// (I4). The epoch filter below must NOT drop it: STOP_REQUESTED bumped the epoch,
// but the finish carries the PRE-stop generation (the runtime stamps it with the
// stream's start epoch), so I1 would otherwise strand the machine in `stopping`
// forever (no idle-cap covers `stopping`). The epoch filter stays in force for
// `superseding` (a successor B owns) — that is the F1 supersede drop.
if (m.phase.name === "stopping" && isFinishEvent(event)) {
return to(m, { name: "idle" }, {
// Reset ownership to local on this terminal transition (review #2): otherwise
// an observer-stop leaves ownership 'observer' and hides "Send now" forever.
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
}
// I1: drop a stale outcome (an event issued under a superseded epoch).
if ("epoch" in event && event.epoch !== undefined && event.epoch !== m.ctx.epoch) {
return stay(m);
}
switch (event.type) {
// ---- local turn ----------------------------------------------------
case "SEND_LOCAL":
// A local send owns the view: leave any recovery, become the local
// streamer, disarm poll/reconnect. epoch++ so a late recovery outcome
// from the previous phase is dropped.
return command(
m,
{ name: "sending" },
[{ type: "cancelReconnect" }, { type: "disarmPoll" }],
{ ownership: "local", liveFollow: false },
);
case "STREAM_INCOMPLETE":
// An OBSERVER's attached stream ended incomplete (starved / torn) — follow
// the run to terminal via the degraded poll.
return to(m, { name: "polling", reason: event.reason }, {
effects: [{ type: "armPoll", reason: event.reason }],
});
case "STREAM_START": {
// First frame arrived. Adopt the run-fact runId if present. sending ->
// streaming; a reconnect/attach that just went live also lands here.
const runFact = event.runId ? { runId: event.runId } : m.ctx.runFact;
return to(m, { name: "streaming" }, {
ctx: { runFact },
effects: [{ type: "cancelReconnect" }, { type: "disarmPoll" }],
});
}
case "FINISH_CLEAN":
// A clean terminal outcome. The run is done — clear the run-fact and go
// idle. (The queue flush is a component concern gated by ownership; the
// FSM only models the phase.) Review #2: reset ownership to local so a
// just-finished observer-attach turn re-exposes "Send now" for the queue.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
case "FINISH_ABORT":
// A user Stop / intentional abort finished. If we were stopping, the
// terminal data has now arrived (I4) — go idle. The run-fact is cleared.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
case "FINISH_DISCONNECT":
// A LIVE SSE drop. Recovery depends on WHO we are (I2 + liveFollow):
// - a mount-attach OBSERVER (a one-shot resume, NOT live-follow) that drops
// -> the degraded poll drives the row to terminal from the DB.
if (m.ctx.ownership === "observer" && !m.ctx.liveFollow) {
return to(m, { name: "polling", reason: "disconnect-visible" }, {
effects: [{ type: "armPoll", reason: "disconnect-visible" }],
});
}
// - a LOCAL live turn (first drop) OR a live-follow re-attach (a SUBSEQUENT
// drop) -> (re-)enter the reconnect ladder. #488 commit 3: allowed
// REPEATEDLY — `liveFollow` is kept across a successful re-attach, so the
// second break reconnects again instead of falling to silent poll.
// #488 commit 2: gated on the RUN-FACT (or an existing live-follow), NOT on
// the presence of an assistant message — a setup-phase break still recovers.
// - visible content already on screen -> keep it, ALSO poll to terminal
// (a full replay could clobber the fuller live tail);
// - no visible content -> the reconnect ladder rebuilds it.
if (m.ctx.runFact || m.ctx.liveFollow) {
const effects: Effect[] = [
{ type: "scheduleReconnect", attempt: 1, delayMs: reconnectDelayMs(1) },
];
if (event.hasVisibleContent) effects.push({ type: "armPoll", reason: "disconnect-visible" });
return command(m, { name: "reconnecting", attempt: 1, failed: false }, effects, {
ownership: "observer",
liveFollow: true,
});
}
// No run to recover: a plain disconnect. Surface the terminal notice.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
});
case "FINISH_ERROR":
return to(m, { name: "error", kind: event.kind }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
// ---- mount attach (resume) ----------------------------------------
case "ATTACH_START":
// A reopened tab attaches to a still-running run: observer ownership.
// #488 F2: ONLY from idle. The mount `getRun` round-trip resolves async, and
// a local send may have started meanwhile (phase `sending`, ownership local);
// a late ATTACH_START must NOT hijack that local turn into an observer-attach
// (queue would stop flushing, "Send now" would hide). Guarding in the reducer
// covers every dispatch source.
if (m.phase.name !== "idle") return stay(m);
return command(m, { name: "attaching" }, [{ type: "resumeStream" }], {
ownership: "observer",
runFact: event.runId ? { runId: event.runId } : m.ctx.runFact,
});
case "ATTACH_LIVE":
// The attach GET returned a live 2xx stream — follow it as an observer.
// Review #1: guard by SOURCE phase. The epoch filter alone is not enough — a
// POLL_TERMINAL uses to() (no epoch bump) and does not abort the in-flight
// GET, so a slow 2xx landing after the machine already left `attaching` (e.g.
// the armed poll saw the terminal row -> idle) would resurrect a settled run
// into a phantom `streaming`. Only enter streaming FROM `attaching`.
if (m.phase.name !== "attaching") return stay(m);
return to(m, { name: "streaming" });
case "ATTACH_NONE":
// 204 / non-2xx / throw: nothing live to attach. Arm the degraded poll to
// follow the run to terminal from the DB. This is a soft-negative run-fact
// (204 on a non-stripped path is authoritative-negative; the runtime may
// pass a RUN_FACT null separately). Keep the run-fact as-is here.
// Review #1: guard by source phase for consistency (a late outcome after the
// machine already left `attaching` must not re-arm a poll).
if (m.phase.name !== "attaching") return stay(m);
return to(m, { name: "polling", reason: "attach-none" }, {
effects: [{ type: "armPoll", reason: "attach-none" }],
});
// ---- reconnect after a live disconnect ----------------------------
case "RECONNECT_ATTEMPT":
// A scheduled backoff fired — fire the attach GET. epoch++ so the previous
// attempt's late outcome cannot drive this one.
if (m.phase.name !== "reconnecting") return stay(m);
return command(
m,
{ name: "reconnecting", attempt: event.attempt, failed: false },
[{ type: "resumeStream" }],
);
case "RECONNECT_ATTACHED":
// #488 commit 3: a live re-attach succeeded. Reset to streaming — the
// attempt counter is dropped, so a LATER disconnect can start a fresh
// ladder from attempt 1 (the old one-shot `!wasResumed` gate forbade a
// second cycle, sending the second break to silent poll).
// Review #1: guard by SOURCE phase. The armed degraded poll can reach the
// terminal row (POLL_TERMINAL -> idle, via to(), NO epoch bump, GET not
// aborted) BEFORE a slow reconnect GET returns 2xx; without this guard that
// late RECONNECT_ATTACHED (same epoch) would resurrect a settled run into a
// phantom `streaming`. Only re-enter streaming FROM `reconnecting`.
if (m.phase.name !== "reconnecting") return stay(m);
return to(m, { name: "streaming" }, {
effects: [{ type: "cancelReconnect" }, { type: "disarmPoll" }],
});
case "RECONNECT_NONE": {
// 204 / error during a reconnect attempt. Arm the degraded poll as the
// belt-and-suspenders fallback, then either back off to the next attempt
// or, at the cap, surface the manual Retry ("failed").
if (m.phase.name !== "reconnecting") return stay(m);
const attempt = m.phase.attempt;
if (attempt < RECONNECT_MAX_ATTEMPTS) {
return command(
m,
{ name: "reconnecting", attempt: attempt + 1, failed: false },
[
{ type: "armPoll", reason: "attach-none" },
{ type: "scheduleReconnect", attempt: attempt + 1, delayMs: reconnectDelayMs(attempt + 1) },
],
);
}
return to(m, { name: "reconnecting", attempt, failed: true }, {
effects: [{ type: "armPoll", reason: "reconnect-exhausted" }],
});
}
case "RETRY":
// Manual Retry from the "failed" reconnect banner OR the stalled banner.
if (m.phase.name === "reconnecting" && m.phase.failed) {
return command(
m,
{ name: "reconnecting", attempt: 1, failed: false },
[{ type: "resumeStream" }],
);
}
if (m.phase.name === "stalled") {
// Re-arm the poll to try to catch the run up again.
return command(m, { name: "polling", reason: "attach-none" }, [
{ type: "armPoll", reason: "attach-none" },
]);
}
return stay(m);
// ---- degraded poll -------------------------------------------------
case "POLL_TERMINAL":
// The run reached a terminal row via the poll (or the reconcile merge). Go
// idle and disarm everything (I4: this is a DATA-driven exit, incl. exit
// from `stopping`). Review #2: reset ownership to local.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
case "POLL_IDLE_CAP":
// Review #4: `stopping` also arms the poll (STOP_REQUESTED) but has NO other
// backstop — an observer-stop with no SDK stream to fire onFinish, whose
// server stop never drives the run terminal, would poll the DB forever. Give
// it a bounded exit: cap -> idle + disarm (NOT `stalled`; Stop was already
// pressed, so there is nothing for the user to retry).
if (m.phase.name === "stopping") {
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
}
// #488 commit 4a: the poll hit the inactivity cap. Instead of going SILENT
// (the old "forever half-done answer"), surface a stalled banner + Retry.
if (m.phase.name !== "polling" && m.phase.name !== "reconnecting") return stay(m);
return to(m, { name: "stalled" }, {
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
// ---- run-fact ------------------------------------------------------
case "RUN_FACT": {
const runFact = event.runFact;
// A fresh NEGATIVE fact (no active run) cancels recovery immediately (I3):
// there is nothing to reconnect to / poll for.
if (!runFact) {
if (
m.phase.name === "reconnecting" ||
m.phase.name === "attaching" ||
m.phase.name === "polling" ||
m.phase.name === "stopping"
) {
return to(m, { name: "idle" }, {
// Review #2: reset ownership to local on this terminal transition.
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "cancelReconnect" }, { type: "disarmPoll" }],
});
}
return to(m, m.phase, { ctx: { runFact: null } });
}
// A positive fact just updates the context (pessimism toward an attempt: a
// stale-but-positive fact permits entering recovery; a 204 will cut it).
return to(m, m.phase, { ctx: { runFact } });
}
// ---- stop ----------------------------------------------------------
case "STOP_REQUESTED":
// Authoritative stop of a detached run. Enter `stopping` and fire stopRun +
// abort the local/attach reader. ALSO arm the poll so the terminal row is
// observed — the exit is by DATA (I4: a terminal row / negative run-fact),
// never by the stopRun HTTP response (which returns after abort, before
// finalization). For a local turn the aborted stream's onFinish (ANY finish)
// is HONORED in `stopping` at the top of reduce() — regardless of generation
// — and exits to idle; the armed poll is the fallback for an observer stop
// with no local onFinish.
return command(
m,
{ name: "stopping" },
[
{ type: "stopRun" },
{ type: "abortAttach" },
{ type: "cancelReconnect" },
{ type: "armPoll", reason: "attach-none" },
],
);
// ---- supersede (CAS) ----------------------------------------------
case "SUPERSEDE_REQUESTED":
// "Interrupt and send now": CAS POST /stream { supersede }. epoch++ so a
// late outcome of the interrupted run is dropped.
return command(
m,
{ name: "superseding" },
[{ type: "supersede", targetRunId: event.targetRunId }, { type: "cancelReconnect" }, { type: "disarmPoll" }],
);
case "SUPERSEDE_READY": {
// CAS succeeded (old run stopped/settled, slot taken, new run begun). We
// are now the local streamer of the NEW run. Adopt its runId if provided.
const runFact = event.runId ? { runId: event.runId } : m.ctx.runFact;
return to(m, { name: "streaming" }, {
ctx: { ownership: "local", runFact, liveFollow: false },
});
}
case "SUPERSEDE_MISMATCH":
// The active run moved between the click and the CAS. Per the spec: verify
// via /run rather than blindly banner — the mismatch may be our own already-
// superseded run. Surface a classified error AND fire a run-fact verify.
return to(m, { name: "error", kind: "supersede-mismatch" }, {
ctx: { runFact: event.currentRunId ? { runId: event.currentRunId } : m.ctx.runFact },
effects: [{ type: "postRun", reason: "verify" }],
});
case "SUPERSEDE_TIMEOUT":
// The old run did not settle within W. Nothing persisted; the composer keeps
// its text. Classified error, NO auto-retry (the old client retry ladder is
// removed in #488 commit 5).
return to(m, { name: "error", kind: "supersede-timeout" });
case "SUPERSEDE_INVALID":
return to(m, { name: "error", kind: "supersede-invalid" });
case "RUN_ALREADY_ACTIVE":
// A plain POST hit the one-active-run gate. NO auto-retry — the composer
// offers "interrupt and send" (supersede) instead. #497/S4: adopt the
// server's activeRunId as the run-fact so that supersede can TARGET the
// (possibly foreign-tab) active run via the CAS, rather than a blind
// promote+abort that just 409s again. A stale/absent id keeps the prior fact.
return to(m, { name: "error", kind: "run-already-active" }, {
ctx: { runFact: event.activeRunId ? { runId: event.activeRunId } : m.ctx.runFact },
});
// ---- lifecycle -----------------------------------------------------
case "DISPOSE":
// Unmount: abort in-flight controllers, drop timers, and bump the epoch so
// NO late callback can drive this (now dead) machine (I5).
return command(
m,
{ name: "idle" },
[
{ type: "abortAttach" },
{ type: "cancelReconnect" },
{ type: "disarmPoll" },
],
{ liveFollow: false },
);
default: {
// Exhaustiveness guard.
const _never: never = event;
void _never;
return stay(m);
}
}
}
@@ -3,7 +3,6 @@ import {
resolveAdoptedChatId,
newlyAddedChatIds,
extractServerChatId,
extractRunId,
} from "./adopt-chat-id";
describe("resolveAdoptedChatId", () => {
@@ -71,17 +70,3 @@ describe("extractServerChatId", () => {
expect(extractServerChatId(undefined)).toBeUndefined();
});
});
describe("extractRunId", () => {
it("reads a string runId from the start metadata", () => {
expect(extractRunId({ metadata: { runId: "run-1" } })).toBe("run-1");
});
it("returns undefined when runId is absent", () => {
expect(extractRunId({ metadata: { chatId: "c" } })).toBeUndefined();
expect(extractRunId({})).toBeUndefined();
expect(extractRunId(undefined)).toBeUndefined();
});
it("returns undefined for a non-string runId", () => {
expect(extractRunId({ metadata: { runId: 7 } })).toBeUndefined();
});
});
@@ -56,20 +56,6 @@ export function extractServerChatId(
return typeof m?.chatId === "string" ? m.chatId : undefined;
}
/**
* #488: read the authoritative RUN id off a streaming assistant message. The
* server attaches it as `message.metadata.runId` on the `start` part when a run
* wraps the turn (see server `chatStreamMetadata`, #184/#487). This is the live
* run-fact update the client FSM adopts (mirrors `extractServerChatId`). Returns
* it only when it is a string; undefined otherwise.
*/
export function extractRunId(
message: { metadata?: unknown } | undefined,
): string | undefined {
const m = message?.metadata as { runId?: string } | undefined;
return typeof m?.runId === "string" ? m.runId : undefined;
}
/**
* The deduped set of ids present in `afterIds` but not in `beforeIds`. A
* paginated/flatMapped list can repeat the same id, so dedupe: one genuinely-new
@@ -6,13 +6,10 @@ describe("estimateTokens", () => {
expect(estimateTokens("")).toBe(0);
});
// #490: migrated onto the shared @docmost/token-estimate module (chars/2.5, up
// from the old client-only chars/4) so the client counter and the server replay
// budgeter can never diverge.
it("ceils chars/2.5 so any non-empty text is at least 1 token", () => {
it("ceils chars/4 so any non-empty text is at least 1 token", () => {
expect(estimateTokens("a")).toBe(1);
expect(estimateTokens("ab")).toBe(1);
expect(estimateTokens("abcde")).toBe(2); // 5 / 2.5 = 2
expect(estimateTokens("x".repeat(10))).toBe(4); // 10 / 2.5 = 4
expect(estimateTokens("abcd")).toBe(1);
expect(estimateTokens("abcde")).toBe(2);
expect(estimateTokens("12345678")).toBe(2);
});
});
@@ -2,10 +2,18 @@
* Rough client-side token estimation for AI-chat UI affordances.
*
* No provider streams exact per-token usage mid-stream, so any in-flight figure
* is a CLIENT ESTIMATE. This re-exports the SHARED estimator from
* `@docmost/token-estimate` (chars/2.5) so the in-body counter and the server's
* replay budgeter use the SAME heuristic two divergent estimators would mean
* "the badge shows 60%" while "the budgeter already trimmed" (#490). Used by the
* in-body reasoning counter ("Thinking · N tokens").
* is a CLIENT ESTIMATE (chars/4 heuristic). Pure + unit-testable: it never runs
* a real BPE tokenizer (that would be O(n²) on the hot path, bloat the bundle,
* and be wrong for Gemini/Ollama anyway). Used by the in-body reasoning counter
* ("Thinking · N tokens").
*/
export { estimateTokens } from "@docmost/token-estimate";
/**
* Rough token estimate for a piece of text using the standard chars/4 heuristic.
* Returns 0 for empty/whitespace-free-of-content input, and ceils so any
* non-empty text counts as at least one token.
*/
export function estimateTokens(text: string): number {
if (!text) return 0;
return Math.ceil(text.length / 4);
}
@@ -42,51 +42,21 @@ describe("describeChatError", () => {
);
});
// #488 commit 5: the #487 concurrency-gate / supersede 409s. FULL real bodies:
// a ConflictException(object) whose response is serialized verbatim, carrying a
// `code` and statusCode 409. Each must classify to a human text, not raw JSON.
it("classifies A_RUN_ALREADY_ACTIVE (409) as already-answering, not raw JSON", () => {
const body =
'{"message":"A run is already active for this chat","code":"A_RUN_ALREADY_ACTIVE","statusCode":409}';
expect(describeChatError(body, t).title).toBe(
"The agent is already answering",
it("classifies a token-degeneration abort under the SAME 'Response stopped.' marker the live view shows (#495)", () => {
// The exact reason the server persists in metadata.error on a degeneration
// abort (ai-chat.service OUTPUT_DEGENERATION_ERROR). Live, this event shows
// the neutral "Response stopped." notice; the persisted banner MUST match it
// so live and refetch never disagree.
const view = describeChatError(
"Output degeneration detected (repeated token loop)",
t,
);
// Never leaks the raw code as the detail.
expect(describeChatError(body, t).detail).not.toContain("A_RUN_ALREADY_ACTIVE");
});
it("classifies SUPERSEDE_TARGET_MISMATCH (409) as run-changed", () => {
// Real server body shape: the current run id is `activeRunId` (NOT `runId`) —
// see ai-chat.controller.ts. describeChatError classifies off `code` only.
const body =
'{"message":"active run does not match the supersede target","code":"SUPERSEDE_TARGET_MISMATCH","activeRunId":"run-x","statusCode":409}';
expect(describeChatError(body, t).title).toBe(
"Couldn't interrupt — the run changed",
expect(view.title).toBe("Response stopped.");
expect(view.detail).toBe(
"The answer was stopped automatically because the model fell into a repeated output loop.",
);
});
it("classifies SUPERSEDE_TIMEOUT (409) as couldn't-interrupt-in-time", () => {
const body =
'{"message":"the run did not settle within the supersede window","code":"SUPERSEDE_TIMEOUT","statusCode":409}';
expect(describeChatError(body, t).title).toBe("Couldn't interrupt in time");
});
it("classifies SUPERSEDE_INVALID (409) as couldn't-interrupt-that-run", () => {
const body =
'{"message":"supervise requires chatId","code":"SUPERSEDE_INVALID","statusCode":409}';
expect(describeChatError(body, t).title).toBe(
"Couldn't interrupt that run",
);
});
it("ORDER GUARD: A_RUN_ALREADY_ACTIVE wins over any generic status branch", () => {
// Even though the body could superficially look 4xx-ish, the code branch runs
// first, so it is never mislabeled by a generic status heading.
const body =
'{"message":"conflict","code":"A_RUN_ALREADY_ACTIVE","statusCode":409}';
const view = describeChatError(body, t);
// Regression guard: it must NOT fall through to the generic heading.
expect(view.title).not.toBe("Something went wrong");
expect(view.title).not.toBe("AI provider not configured");
});
it("classifies a dropped connection (ECONNRESET) as a lost-connection error", () => {
@@ -39,40 +39,18 @@ export function describeChatError(
};
}
// #488 commit 5: the #487 concurrency-gate / supersede 409s. These arrive as a
// ConflictException(object) body carrying a `code` (and statusCode 409). They
// MUST be classified by `code` STRICTLY BEFORE any generic status branch, or the
// user sees the raw JSON `{"code":"A_RUN_ALREADY_ACTIVE",…}`. The code strings
// are the real #487 server contract (ai-chat.controller.ts) — do not invent.
if (/"code"\s*:\s*"A_RUN_ALREADY_ACTIVE"/.test(msg)) {
// Our own token-degeneration abort (#444): the server aborts a runaway
// repetition loop and persists this exact reason in metadata.error. LIVE, the
// same abort surfaces as the neutral "Response stopped." notice (the client
// cannot tell it from a manual Stop mid-stream), so the persisted banner must
// read the SAME "Response stopped." marker — otherwise the live view and a
// later refetch show two different texts for one event. The detail explains the
// loop-guard cause without contradicting the shared heading.
if (/output degeneration detected|repeated token loop/i.test(msg)) {
return {
title: t("The agent is already answering"),
title: t("Response stopped."),
detail: t(
"This chat already has a run in progress. Wait for it to finish, or interrupt it and send now.",
),
};
}
if (/"code"\s*:\s*"SUPERSEDE_TARGET_MISMATCH"/.test(msg)) {
return {
title: t("Couldn't interrupt — the run changed"),
detail: t(
"The run you tried to interrupt is no longer the active one. Check the latest answer and try again.",
),
};
}
if (/"code"\s*:\s*"SUPERSEDE_TIMEOUT"/.test(msg)) {
return {
title: t("Couldn't interrupt in time"),
detail: t(
"The previous run didn't stop in time. Nothing was sent — try sending again.",
),
};
}
if (/"code"\s*:\s*"SUPERSEDE_INVALID"/.test(msg)) {
return {
title: t("Couldn't interrupt that run"),
detail: t(
"The run to interrupt doesn't belong to this chat. Reload and try again.",
"The answer was stopped automatically because the model fell into a repeated output loop.",
),
};
}
@@ -0,0 +1,65 @@
import { describe, it, expect } from "vitest";
import * as Y from "yjs";
import { yHistoryAvailability } from "./use-toolbar-state.ts";
// Undo/redo availability is derived from the Yjs UndoManager's PRIVATE
// `undoStack` / `redoStack` fields (see use-toolbar-state.ts for why we read the
// stack lengths directly instead of the expensive `editor.can().undo()` dry-run).
// These tests lock in the behavior AND pin the library shape so a yjs / y-undo
// upgrade that renames/restructures those internals fails loudly here rather than
// silently enabling/disabling the toolbar buttons in production.
describe("yHistoryAvailability", () => {
it("reports availability from the stack lengths", () => {
expect(yHistoryAvailability({ undoStack: [], redoStack: [] })).toEqual({
canUndo: false,
canRedo: false,
});
expect(
yHistoryAvailability({ undoStack: [{}], redoStack: [] }),
).toEqual({ canUndo: true, canRedo: false });
expect(
yHistoryAvailability({ undoStack: [{}], redoStack: [{}, {}] }),
).toEqual({ canUndo: true, canRedo: true });
});
it("returns null when the private stack shape is unrecognized (upgrade guard)", () => {
// Simulates a yjs / y-undo upgrade that renames or restructures the private
// fields: the caller then falls back to the safe prosemirror-history default
// instead of throwing on `.length` of undefined or reading garbage.
expect(yHistoryAvailability(undefined)).toBeNull();
expect(yHistoryAvailability(null)).toBeNull();
expect(yHistoryAvailability({})).toBeNull();
expect(yHistoryAvailability({ undoStack: 5, redoStack: 5 })).toBeNull();
// Only one stack present (partial rename) is still not trusted.
expect(yHistoryAvailability({ undoStack: [] })).toBeNull();
});
it("pin-test: a real yjs UndoManager still exposes undoStack/redoStack arrays", () => {
const doc = new Y.Doc();
const text = doc.getText("prosemirror");
const undoManager = new Y.UndoManager(text);
// Fresh manager: both stacks empty -> nothing to undo/redo.
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: false,
canRedo: false,
});
// A tracked edit must push onto the private undoStack. If a future yjs
// renames these fields, yHistoryAvailability(undoManager) returns null and
// the expectation below fails loudly.
text.insert(0, "hello");
undoManager.stopCapturing();
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: true,
canRedo: false,
});
// Undoing moves the item to the redoStack -> redo becomes available.
undoManager.undo();
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: false,
canRedo: true,
});
});
});
@@ -35,6 +35,30 @@ export interface ToolbarState {
// When neither history backend is installed (the pre-sync static editor —
// mainExtensions only, undoRedo disabled), both fall through to 0 -> false,
// matching the previous `safeCan` behavior.
// Reads the Yjs UndoManager's undo/redo availability from its stack lengths.
//
// `undoStack` / `redoStack` are PRIVATE y-undo / yjs internals, so we touch them
// defensively: a yjs or y-undo upgrade that renames or restructures these fields
// must not silently mis-drive the toolbar buttons (nor throw on `.length` of
// `undefined`). We only trust them when they are actually arrays; otherwise this
// returns null and the caller falls back to a safe default. The pin-test in
// use-toolbar-state.test.ts asserts the current library shape, so an upgrade that
// breaks this contract fails loudly there instead of failing silently in the UI.
export function yHistoryAvailability(
undoManager: unknown,
): { canUndo: boolean; canRedo: boolean } | null {
if (!undoManager || typeof undoManager !== "object") return null;
const { undoStack, redoStack } = undoManager as {
undoStack?: unknown;
redoStack?: unknown;
};
if (!Array.isArray(undoStack) || !Array.isArray(redoStack)) return null;
return {
canUndo: undoStack.length > 0,
canRedo: redoStack.length > 0,
};
}
function historyAvailability(editor: Editor): {
canUndo: boolean;
canRedo: boolean;
@@ -43,16 +67,14 @@ function historyAvailability(editor: Editor): {
// Collaboration history (Yjs) takes precedence when present.
const yState = yUndoPluginKey.getState(state) as
| { undoManager?: { undoStack: unknown[]; redoStack: unknown[] } }
| { undoManager?: unknown }
| undefined;
if (yState?.undoManager) {
return {
canUndo: yState.undoManager.undoStack.length > 0,
canRedo: yState.undoManager.redoStack.length > 0,
};
}
const yAvail = yHistoryAvailability(yState?.undoManager);
if (yAvail) return yAvail;
// Plain prosemirror-history (returns 0 when the history plugin is absent).
// This is also the safe default when a Yjs UndoManager is present but its
// private stack shape is no longer recognized (yHistoryAvailability -> null).
return {
canUndo: undoDepth(state) > 0,
canRedo: redoDepth(state) > 0,
@@ -13,8 +13,7 @@ let currentAlias: IShareAlias | null = null;
let availabilityResult: {
valid: boolean;
available: boolean;
currentPageId: string | null;
} = { valid: true, available: true, currentPageId: null };
} = { valid: true, available: true };
vi.mock("@/features/share/queries/share-query.ts", () => ({
useShareAliasForPageQuery: () => ({ data: currentAlias }),
@@ -56,7 +55,7 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
beforeEach(() => {
setMutateAsync.mockReset();
currentAlias = null;
availabilityResult = { valid: true, available: true, currentPageId: null };
availabilityResult = { valid: true, available: true };
});
it("shows a 'will move it here' HINT (not a terminal error) when the name belongs to another page, and keeps Save enabled", async () => {
@@ -65,7 +64,6 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
availabilityResult = {
valid: true,
available: false,
currentPageId: "page-X",
};
renderSection("page-Y");
@@ -97,7 +95,6 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
availabilityResult = {
valid: true,
available: false,
currentPageId: "page-X",
};
// The server rejects the un-confirmed save asking the client to confirm.
setMutateAsync.mockRejectedValueOnce({
@@ -106,7 +103,6 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
status: 409,
data: {
code: "ALIAS_REASSIGN_REQUIRED",
currentPageId: "page-X",
currentPageTitle: "Alias Test Page X",
},
},
@@ -48,7 +48,6 @@ export default function ShareAliasSection({
const [availability, setAvailability] = useState<{
valid: boolean;
available: boolean;
currentPageId: string | null;
} | null>(null);
const [reassign, setReassign] = useState<{
alias: string;
@@ -76,7 +75,6 @@ export default function ShareAliasSection({
setAvailability({
valid: res.valid,
available: res.available,
currentPageId: res.currentPageId,
});
} catch {
setAvailability(null);
@@ -108,7 +108,6 @@ export interface IShareAliasAvailability {
alias: string;
valid: boolean;
available: boolean;
currentPageId: string | null;
}
export interface ISharedPageTree {
@@ -6,6 +6,8 @@ import {
nextReindexPollInterval,
isReindexComplete,
isReindexButtonLoading,
reindexRunKey,
isNewReindexRun,
} from './ai-provider-settings';
describe('resolveCardStatus', () => {
@@ -221,6 +223,128 @@ describe('isReindexComplete', () => {
});
});
describe('reindexRunKey', () => {
it('is null when the status carries no run identity', () => {
expect(reindexRunKey(undefined)).toBeNull();
expect(
reindexRunKey({ reindexing: false, indexedPages: 5, totalPages: 5 }),
).toBeNull();
});
it('is null for a legacy/degraded record with an empty runId', () => {
// The server sends runId='' for a record written before the field existed;
// the client must treat that as "no identity" (fall back to prior behaviour).
expect(
reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: '',
reindexStartedAt: 1000,
}),
).toBeNull();
});
it('folds runId and startedAt into one stable key', () => {
expect(
reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
}),
).toBe('run-a:1000');
});
it('changes when the runId changes for the same startedAt', () => {
const a = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
});
const b = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-b',
reindexStartedAt: 1000,
});
expect(a).not.toBe(b);
});
it('changes when the same runId restarts at a new startedAt', () => {
const a = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
});
const b = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 2000,
});
expect(a).not.toBe(b);
});
});
describe('isNewReindexRun (poll keying on runId)', () => {
// Derive the status shape from the helper itself so the test needs no export
// of the component-internal ReindexStatus type.
type ReindexStatusLike = NonNullable<Parameters<typeof reindexRunKey>[0]>;
const run = (runId: string, startedAt: number): ReindexStatusLike => ({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId,
reindexStartedAt: startedAt,
});
it('first identity after none latched is a NEW run', () => {
expect(isNewReindexRun(null, run('run-a', 1000))).toBe(true);
});
it('the SAME identity is not a new run (same run being watched)', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(isNewReindexRun(key, run('run-a', 1000))).toBe(false);
});
it('a DIFFERENT runId is a new run (reset per-run poll state)', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(isNewReindexRun(key, run('run-b', 1000))).toBe(true);
});
it('an identity-less poll (no runId / cleared record) is never a new run', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(
isNewReindexRun(key, {
reindexing: false,
indexedPages: 10,
totalPages: 10,
}),
).toBe(false);
});
it('a legacy empty-runId poll does not spuriously reset a latched run', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(
isNewReindexRun(key, {
reindexing: true,
indexedPages: 3,
totalPages: 10,
runId: '',
reindexStartedAt: 1000,
}),
).toBe(false);
});
});
describe('isReindexButtonLoading', () => {
it('loads while the POST mutation is pending', () => {
expect(
@@ -173,9 +173,43 @@ export function resolveKeyField(
// Subset of the status payload that drives the reindex poll decisions.
type ReindexStatus = Pick<
IAiSettings,
"reindexing" | "indexedPages" | "totalPages"
"reindexing" | "indexedPages" | "totalPages" | "runId" | "reindexStartedAt"
>;
/**
* A stable per-RUN key for the reindex poll: `runId:startedAt`, or `null` when
* the status carries no run identity (no active run, or a legacy/degraded
* server record with an empty runId). Two polls of the SAME run share a key; a
* new run mints a fresh runId and so a different key.
*
* This is the single place the client turns the server's run identity into the
* value it keys on it removes the "is this the same run I've been watching or
* a brand-new one?" ambiguity that made a class of reindex-status bugs (a stale
* pre-reindex snapshot vs a fresh run) get fixed twice (#262). `startedAt` is
* folded in so a run that somehow reuses a runId but restarted is still new.
*/
export function reindexRunKey(status: ReindexStatus | undefined): string | null {
const runId = status?.runId;
if (!runId) return null;
return `${runId}:${status?.reindexStartedAt ?? ""}`;
}
/**
* Decide whether the latest poll represents a NEW reindex run relative to the
* run key the client last latched (`prevKey`, `null` if none yet). True only
* when the status carries an identity AND it differs from the latched one the
* signal to reset any per-run poll state (the "seen active" latch / progress the
* UI held). The same identity (or no identity) is NOT a new run, so an unchanged
* or identity-less poll never resets mid-run.
*/
export function isNewReindexRun(
prevKey: string | null,
status: ReindexStatus | undefined,
): boolean {
const key = reindexRunKey(status);
return key !== null && key !== prevKey;
}
/**
* Decide the TanStack Query `refetchInterval` while a reindex may be running.
* Returns the poll interval (ms) to keep polling, or `false` to stop.
@@ -320,6 +354,13 @@ export default function AiProviderSettings() {
// counter at 0 until a manual reload. A ref (not state) because it must not
// trigger a render and is only ever read where `reindexing` is already false.
const reindexSeenActiveRef = useRef(false);
// The run identity (runId:startedAt) the current poll window is keyed on. When
// a poll reports a DIFFERENT runId the server has started a NEW run, so we
// re-latch to it and reset `reindexSeenActiveRef` — a fresh run must never
// inherit the previous run's "seen active"/completion state (which would stop
// polling immediately or read the old run's counters as this run's). null =
// no run keyed yet (steady state, or a legacy record without a runId).
const reindexRunKeyRef = useRef<string | null>(null);
// Only admins may read the (masked) AI settings; the server enforces this too.
const { data: settings, isLoading } = useAiSettingsQuery(isAdmin, (query) =>
@@ -336,6 +377,14 @@ export default function AiProviderSettings() {
// unmount because the deadline state goes away with the component.
useEffect(() => {
if (reindexDeadline === null) return;
// Key the poll on the run identity: if this poll carries a runId different
// from the one we latched, the server started a NEW run, so adopt it and
// drop the per-run "seen active" latch (a fresh run must not inherit the
// previous run's completion state). Same runId => same run, leave it alone.
if (isNewReindexRun(reindexRunKeyRef.current, settings)) {
reindexRunKeyRef.current = reindexRunKey(settings);
reindexSeenActiveRef.current = false;
}
// Latch "we have seen the active run" the moment a poll reports it, so the
// completion check below (and the refetchInterval's) only fires once the run
// has genuinely started — never on the stale pre-reindex snapshot.
@@ -1220,6 +1269,10 @@ export default function AiProviderSettings() {
// immediately.
onSuccess: () => {
reindexSeenActiveRef.current = false;
// Forget the previous run's identity so the first poll of
// this window (carrying the new run's runId) is recognized
// as a new run and keyed afresh.
reindexRunKeyRef.current = null;
setReindexDeadline(Date.now() + REINDEX_POLL_CAP_MS);
},
})
@@ -51,6 +51,14 @@ export interface IAiSettings {
// True while a full workspace reindex is actively running; the counts above
// then reflect the live run progress (done climbs 0 -> total).
reindexing?: boolean;
// Identity of the ACTIVE reindex run (present only while `reindexing`). The
// poll keys on `runId`: a changed value means a NEW run (reset the per-run
// poll state the UI latched), the same value is the run already being watched.
// Absent/empty ('') => no identity available; the client keeps prior behaviour.
runId?: string;
// Epoch-ms the active run started; paired with `runId` so a restart with a
// recycled id is still detected as a new run.
reindexStartedAt?: number;
}
// Update payload. Key semantics (same for `apiKey` and `embeddingApiKey`):
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { templateRoute } from "./route-template";
import { templateRoute, KNOWN_ROUTE_TEMPLATES } from "./route-template";
describe("templateRoute", () => {
it("templates a space page path (never leaks slugs)", () => {
@@ -32,4 +32,30 @@ describe("templateRoute", () => {
expect(templateRoute("/weird/unknown/thing")).toBe("other");
expect(templateRoute("/s/team/p/slug/extra/segments")).toBe("other");
});
// The server's /api/telemetry/vitals mirror (ALLOWED_ROUTE_TEMPLATES) drops any
// route outside KNOWN_ROUTE_TEMPLATES, so templateRoute must NEVER emit a label
// that is not in that dictionary — otherwise legit client metrics get dropped.
it("only ever emits labels contained in KNOWN_ROUTE_TEMPLATES (#495)", () => {
const samples = [
"/",
"/home",
"/settings/members",
"/settings/groups/g-1",
"/s/team",
"/s/team/trash",
"/s/team/p/slug",
"/p/slug",
"/share/abc",
"/share/abc/p/slug",
"/share/p/slug",
"/labels/urgent",
"/invites/inv-1",
"/weird/unknown/thing", // -> "other"
"/deep/unmatched/x/y/z", // -> "other"
];
for (const path of samples) {
expect(KNOWN_ROUTE_TEMPLATES.has(templateRoute(path))).toBe(true);
}
});
});
@@ -44,6 +44,22 @@ const STATIC_ROUTES = new Set<string>([
'/settings/sharing',
]);
/**
* The COMPLETE, finite vocabulary `templateRoute` can ever emit: the two
* synthetic labels (`/` and `other`), the static routes, and the dynamic
* templates. Exported so the public `/api/telemetry/vitals` endpoint can reject
* any `route` outside this dictionary server-side (the endpoint is anonymous, so
* an un-checked `route` is a free-text write surface). The server keeps a mirror
* (`ALLOWED_ROUTE_TEMPLATES` in client-metrics.constants.ts) this is the
* canonical source; keep them in lockstep.
*/
export const KNOWN_ROUTE_TEMPLATES: ReadonlySet<string> = new Set<string>([
'/',
'other',
...STATIC_ROUTES,
...ROUTE_PATTERNS.map((p) => p.template),
]);
export function templateRoute(pathname: string): string {
// Normalise a trailing slash (except root).
const path =
+1 -3
View File
@@ -23,7 +23,7 @@
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build",
"test": "jest",
"test:int": "jest --config test/jest-integration.json",
"test:watch": "jest --watch",
@@ -44,7 +44,6 @@
"@docmost/mcp": "workspace:*",
"@docmost/pdf-inspector": "1.9.6",
"@docmost/prosemirror-markdown": "workspace:*",
"@docmost/token-estimate": "workspace:*",
"@fastify/compress": "^9.0.0",
"@fastify/cookie": "^11.0.2",
"@fastify/multipart": "^10.0.0",
@@ -207,7 +206,6 @@
"^@docmost/db/(.*)$": "<rootDir>/database/$1",
"^@docmost/transactional/(.*)$": "<rootDir>/integrations/transactional/$1",
"^@docmost/ee/(.*)$": "<rootDir>/ee/$1",
"^@docmost/token-estimate$": "<rootDir>/../../../packages/token-estimate/src/index.ts",
"^src/(.*)$": "<rootDir>/$1",
"^@tiptap/react$": "<rootDir>/../test/stubs/tiptap-react.js"
}
@@ -43,9 +43,6 @@ function makeRepo(overrides: Record<string, jest.Mock> = {}) {
workspaceId: v.workspaceId,
})),
update: jest.fn(async () => ({ id: 'run-1' })),
// #487: terminal finalize now goes through the CONDITIONAL write. Default
// returns a truthy row (the run WAS active -> this call wrote it).
finalizeIfActive: jest.fn(async () => ({ id: 'run-1', status: 'succeeded' })),
markStopRequested: jest.fn(async () => ({ id: 'run-1' })),
findActiveByChat: jest.fn(async () => undefined),
findLatestByChat: jest.fn(async () => undefined),
@@ -339,12 +336,14 @@ describe('AiChatRunService run lifecycle', () => {
await svc.finalizeRun('run-1', 'ws-1', 'error', 'provider blew up');
expect(svc.isLocallyActive('run-1')).toBe(false);
// #487: the terminal write is CONDITIONAL (finalizeIfActive); finishedAt is
// stamped inside the repo method, so the service passes just status + error.
expect(repo.finalizeIfActive).toHaveBeenCalledWith(
expect(repo.update).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'failed', error: 'provider blew up' }),
expect.objectContaining({
status: 'failed',
error: 'provider blew up',
finishedAt: expect.any(Date),
}),
);
});
@@ -367,8 +366,8 @@ describe('AiChatRunService run lifecycle', () => {
// A second settle (e.g. a streamText callback firing after the catch) no-ops.
await svc.finalizeRun('run-1', 'ws-1', 'completed', undefined);
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(1);
expect(repo.finalizeIfActive).toHaveBeenCalledWith(
expect(repo.update).toHaveBeenCalledTimes(1);
expect(repo.update).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'failed', error: 'first' }),
@@ -390,8 +389,8 @@ describe('AiChatRunService run lifecycle', () => {
const updateGate = new Promise((res) => {
resolveUpdate = res;
});
const finalizeIfActive = jest.fn(() => updateGate);
const repo = makeRepo({ finalizeIfActive });
const update = jest.fn(() => updateGate);
const repo = makeRepo({ update });
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
@@ -400,23 +399,23 @@ describe('AiChatRunService run lifecycle', () => {
});
// Fire both before the (pending) update resolves. The first synchronously
// claims the entry (active.delete) and awaits the write; the second, started
// in the same macrotask, finds the entry already gone and returns at the claim
// WITHOUT ever writing.
// claims the entry (active.delete) and awaits update; the second, started in
// the same macrotask, finds the entry already gone and returns at the claim
// WITHOUT ever calling update.
const p1 = svc.finalizeRun('run-1', 'ws-1', 'completed');
const p2 = svc.finalizeRun('run-1', 'ws-1', 'error', 'safety-net');
// The decisive assertion: exactly one caller reached the terminal UPDATE.
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
expect(update).toHaveBeenCalledTimes(1);
// Let the single in-flight update land; both calls resolve cleanly.
resolveUpdate({ id: 'run-1', status: 'succeeded' });
resolveUpdate({ id: 'run-1' });
await Promise.all([p1, p2]);
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
expect(update).toHaveBeenCalledTimes(1);
// The winner is the FIRST caller ('completed' -> 'succeeded'); the late
// 'error' settle never wrote, so it could not clobber the real status.
expect(finalizeIfActive).toHaveBeenCalledWith(
expect(update).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
@@ -432,10 +431,10 @@ describe('AiChatRunService run lifecycle', () => {
// 409s until a restart. The fix updates FIRST and retries.
let calls = 0;
const repo = makeRepo({
finalizeIfActive: jest.fn(async () => {
update: jest.fn(async () => {
calls += 1;
if (calls === 1) throw new Error('deadlock detected');
return { id: 'run-1', status: 'succeeded' };
return { id: 'run-1' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
@@ -448,29 +447,26 @@ describe('AiChatRunService run lifecycle', () => {
await svc.finalizeRun('run-1', 'ws-1', 'completed');
// The retry landed the terminal write: the entry is dropped (slot freed), no
// zombie left, and the row carries the real terminal status.
// The retry landed the terminal write: the entry is dropped (slot freed) and
// the row carries the real terminal status — NOT stranded at 'running'.
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(svc.hasZombie('run-1')).toBe(false);
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(2);
expect(repo.finalizeIfActive).toHaveBeenLastCalledWith(
expect(repo.update).toHaveBeenCalledTimes(2);
expect(repo.update).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
});
it('#487 give-up: if the terminal write keeps failing, finalizeRun leaves a ZOMBIE (does NOT restore the entry) and settleZombie re-drives it', async () => {
it('F6: if the terminal write keeps failing, the entry is RETAINED and a LATER settle completes it (chat not permanently 409d)', async () => {
// Worst case: the DB is down for the whole first finalize (all attempts fail).
// #487 changes the give-up behaviour: the entry is NOT restored (a restored
// entry is indistinguishable from a live run). Instead a ZOMBIE record holds
// the intended terminal status, and a re-drive (settleZombie — called by the
// reconcile / supersede / opportunistic paths) applies it later.
// The run must NOT be silently lost — the entry stays so a subsequent settle
// (a streamText callback, requestStop -> onAbort, or a future sweep) can retry.
let healthy = false;
const repo = makeRepo({
finalizeIfActive: jest.fn(async () => {
update: jest.fn(async () => {
if (!healthy) throw new Error('pool exhausted');
return { id: 'run-1', status: 'succeeded' };
return { id: 'run-1' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
@@ -484,83 +480,35 @@ describe('AiChatRunService run lifecycle', () => {
userId: 'user-1',
});
// First settle: every bounded attempt fails -> ZOMBIE, entry NOT restored.
// First settle: every bounded attempt fails -> entry retained, NOT settled.
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(false); // NOT a live entry
expect(svc.hasZombie('run-1')).toBe(true);
expect(svc.zombieRunIds()).toContain('run-1');
// The give-up emits ONE explicit, greppable ERROR mentioning the zombie.
expect(svc.isLocallyActive('run-1')).toBe(true);
// F12: the give-up emits ONE explicit, greppable ERROR (run + chat context)
// so an operator can tell "gave up, run held in memory" from a per-attempt
// blip — distinct from the per-attempt warns.
const gaveUp = errorSpy.mock.calls.some(
(c) =>
/NON-TERMINAL/.test(String(c[0])) &&
/ZOMBIE/.test(String(c[0])) &&
/run-1/.test(String(c[0])) &&
/chat-1/.test(String(c[0])),
);
expect(gaveUp).toBe(true);
// The settle notifier resolved as terminalWriteFailed (a subscriber learns the
// slot still needs the intended status applied).
const outcome = await svc.peekSettled('run-1');
expect(outcome).toEqual({
status: 'succeeded',
error: null,
terminalWriteFailed: true,
});
// The DB recovers; a re-drive settles the zombie via the conditional UPDATE.
// The DB recovers; a later settle now succeeds and frees the slot.
healthy = true;
const redriven = await svc.settleZombie('run-1');
expect(redriven).toBe(true);
expect(svc.hasZombie('run-1')).toBe(false);
expect(repo.finalizeIfActive).toHaveBeenLastCalledWith(
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
// A later finalizeRun is idempotent (row already terminal): it no-ops at the
// once-gate, never re-writing.
const callsBefore = repo.finalizeIfActive.mock.calls.length;
// And it is now idempotent: a further settle no-ops (terminal row already
// written), so a double-settle can never clobber the real status.
const callsBefore = repo.update.mock.calls.length;
await svc.finalizeRun('run-1', 'ws-1', 'error', 'late');
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(callsBefore);
});
it('#487 double-settle collapses to a benign no-op (conditional write; notifier resolves once)', async () => {
// A second concurrent settle is stopped at the synchronous active.delete
// claim, so the terminal write runs exactly once and the notifier resolves
// exactly once with the FIRST settler's outcome.
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'u1' });
await svc.finalizeRun('run-1', 'ws-1', 'aborted');
await svc.finalizeRun('run-1', 'ws-1', 'error', 'late'); // no-op
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(1);
const outcome = await svc.peekSettled('run-1');
// peekSettled after resolve+delete falls through (notifier dropped, no zombie)
// -> undefined; the FIRST settler already resolved any earlier subscriber.
expect(outcome).toBeUndefined();
});
it('#487 late settledPromise subscriber gets the resolved outcome', async () => {
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'u1' });
// Subscribe BEFORE settle: hold the promise reference (as supersede does).
const early = svc.peekSettled('run-1');
expect(early).toBeDefined();
await svc.finalizeRun('run-1', 'ws-1', 'completed');
// The reference grabbed before settle resolves with the written outcome, even
// though the notifier was dropped from the map on resolve (bounded).
await expect(early).resolves.toEqual({
status: 'succeeded',
error: null,
terminalWriteFailed: false,
});
expect(repo.update).toHaveBeenCalledTimes(callsBefore);
});
it('recordStep / linkAssistantMessage are best-effort: a repo failure is swallowed', async () => {
@@ -577,197 +525,3 @@ describe('AiChatRunService run lifecycle', () => {
).resolves.toBeUndefined();
});
});
describe('#487 AiChatRunService.supersede (CAS)', () => {
const chat = 'chat-1';
const ws = 'ws-1';
it('degrade: no active run on the chat -> caller sends a normal turn', async () => {
const repo = makeRepo({
findById: jest.fn(async () => undefined),
findActiveByChat: jest.fn(async () => undefined),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({ kind: 'degrade' });
});
it('invalid: the target run belongs to a DIFFERENT chat -> 400', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-x',
chatId: 'other-chat',
workspaceId: ws,
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({ kind: 'invalid' });
});
it('mismatch: a DIFFERENT run is active than the one targeted -> current runId', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-x', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => ({
id: 'run-live',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({
kind: 'mismatch',
activeRunId: 'run-live',
});
});
it('ready: the target IS active -> stop it, await its (fast) settle, free the slot', async () => {
// Simulate a live long TOOL (NOT a slow UPDATE): the run stays active until an
// explicit Stop unwinds it; commit-1's race makes that settle land quickly.
// The abort listener stands in for streamText's onAbort -> finalizeRun.
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'aborted',
error: null,
})),
findActiveByChat: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
handle.signal.addEventListener('abort', () => {
void svc.finalizeRun('run-1', ws, 'aborted');
});
// supersede: getRun -> getActiveByChat(==target) -> requestStop -> the abort
// listener settles the run -> awaitSettled resolves -> ready.
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
expect(handle.signal.aborted).toBe(true); // Stop reached the run
});
it('timeout: the target never settles within W -> 409 SUPERSEDE_TIMEOUT (nothing persisted)', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-1', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
// Do NOT settle the run: a tiny W elapses -> timeout.
const result = await svc.supersede(chat, 'run-1', ws, 30);
expect(result).toEqual({ kind: 'timeout' });
});
it('ready then a DUPLICATE supersede POST degrades (the run is already gone)', async () => {
let active: unknown = {
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
};
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'aborted',
error: null,
})),
findActiveByChat: jest.fn(async () => active),
finalizeIfActive: jest.fn(async () => {
active = undefined; // settling frees the active slot
return { id: 'run-1', status: 'aborted' };
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
handle.signal.addEventListener('abort', () => {
void svc.finalizeRun('run-1', ws, 'aborted');
});
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
// The duplicate POST for the same target now finds no active run -> degrade.
expect(await svc.supersede(chat, 'run-1', ws)).toEqual({ kind: 'degrade' });
});
it('reconcileStaleRuns: aborts a stale run with NO entry/zombie; NEVER touches a live entry', async () => {
const finalizeIfActive = jest.fn(async () => ({ id: 'x', status: 'aborted' }));
const repo = makeRepo({
insert: jest.fn(async (v: any) => ({
id: 'live-1',
status: 'running',
chatId: v.chatId,
workspaceId: v.workspaceId,
})),
finalizeIfActive,
findStaleActive: jest.fn(async () => [
{ id: 'orphan-1', workspaceId: ws, chatId: 'c-orphan' },
{ id: 'live-1', workspaceId: ws, chatId: 'c-live' },
]),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
// A LIVE run this replica owns (in the `active` map).
await svc.beginRun({ chatId: 'c-live', workspaceId: ws, userId: 'u1' });
expect(svc.isLocallyActive('live-1')).toBe(true);
const aborted = await svc.reconcileStaleRuns(15 * 60 * 1000);
expect(aborted).toBe(1);
// The orphan (no entry) was aborted; the live entry was NEVER passed to the DB.
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
expect(finalizeIfActive).toHaveBeenCalledWith(
'orphan-1',
ws,
expect.objectContaining({ status: 'aborted' }),
);
expect(svc.isLocallyActive('live-1')).toBe(true);
});
it('gave-up zombie: supersede applies the intended status (settleZombie) then is ready', async () => {
let healthy = false;
let active: unknown = {
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
};
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-1', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => active),
finalizeIfActive: jest.fn(async () => {
if (!healthy) throw new Error('db down');
active = undefined;
return { id: 'run-1', status: 'aborted' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
jest.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
// The run's terminal write gives up -> zombie (row still 'running').
await svc.finalizeRun('run-1', ws, 'aborted');
expect(svc.hasZombie('run-1')).toBe(true);
// The DB recovers; supersede awaits the (already-resolved, terminalWriteFailed)
// settle, then settleZombie applies the intended status -> ready.
healthy = true;
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
expect(svc.hasZombie('run-1')).toBe(false);
});
});
@@ -34,88 +34,6 @@ export class RunAlreadyActiveError extends Error {
export type TurnTerminalStatus = 'completed' | 'error' | 'aborted';
export type RunTerminalStatus = 'succeeded' | 'failed' | 'aborted';
/** The terminal run statuses — the row is done once it reads one of these. */
export const RUN_TERMINAL_STATUSES: readonly RunTerminalStatus[] = [
'succeeded',
'failed',
'aborted',
];
/** Whether a persisted run status is terminal (settled). */
export function isRunTerminal(status: string | null | undefined): boolean {
return (
status === 'succeeded' || status === 'failed' || status === 'aborted'
);
}
/**
* #487: the outcome a run's {@link AiChatRunService.finalizeRun} settled with.
* `terminalWriteFailed` = the terminal write GAVE UP after the bounded retry, so
* the row is still non-terminal ('running') and a ZOMBIE record holds the
* `intended` status for a later re-drive (reconcile / supersede / boot sweep). A
* subscriber (supersede, #487 commit 3) uses this to decide whether the slot is
* genuinely free or must first have the intended status applied.
*/
export interface RunSettleOutcome {
status: RunTerminalStatus;
error: string | null;
terminalWriteFailed: boolean;
}
/**
* #487: how long a supersede waits for the target run to settle after Stop before
* it degrades to `SUPERSEDE_TIMEOUT`. W=10s is generous under a HEALTHY DB: commit
* 1's race-on-abort makes an in-app tool abort->settle in ms/hundreds of ms, so a
* live run releases its slot well within the window. Under a DB brownout the
* timeout is normal (the write cannot land); W must NOT be raised to paper
* over a slow DB a SUPERSEDE_TIMEOUT is the honest signal (nothing persisted,
* the composer keeps the user's text). Env-tunable for ops, default 10s.
*/
export const SUPERSEDE_SETTLE_TIMEOUT_MS = (() => {
const raw = Number(process.env.AI_CHAT_SUPERSEDE_TIMEOUT_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 10_000;
})();
/**
* #487: the result of the supersede CAS ({@link AiChatRunService.supersede}).
* - `degrade` : no active run on the chat (it ended between click and POST)
* the caller sends a NORMAL turn (NOT a mismatch);
* - `invalid` : the target runId belongs to a DIFFERENT chat (malformed CAS 400);
* - `mismatch` : a DIFFERENT run is active than the one the client targeted
* 409 SUPERSEDE_TARGET_MISMATCH carrying the current `activeRunId`
* (the client does NOT auto-retry);
* - `timeout` : the target did not settle within W 409 SUPERSEDE_TIMEOUT,
* nothing persisted;
* - `ready` : the target was stopped AND settled (or its zombie's intended was
* applied) the slot is free; the caller may beginRun the new run.
*/
export type SupersedeResult =
| { kind: 'degrade' }
| { kind: 'invalid' }
| { kind: 'mismatch'; activeRunId: string }
| { kind: 'timeout' }
| { kind: 'ready' };
/** A one-shot settle notifier (#487): `resolve` is called EXACTLY ONCE. */
interface Deferred<T> {
promise: Promise<T>;
resolve: (value: T) => void;
}
/**
* #487: a run whose terminal write GAVE UP (every bounded attempt failed). The
* row is stranded non-terminal ('running'); this record is the ONLY thing that
* distinguishes it from a live run, and carries the `intended` terminal status so
* a re-drive can apply it via the conditional UPDATE. Process-local (phase-1
* single-process assumption): a restart drops it, and the boot sweep then writes
* 'aborted' over the intended a documented loss (see finalizeRun).
*/
interface ZombieRun {
workspaceId: string;
chatId: string;
intended: { status: RunTerminalStatus; error: string | null };
}
export function mapTurnStatusToRun(
status: TurnTerminalStatus,
): RunTerminalStatus {
@@ -183,22 +101,6 @@ export class AiChatRunService implements OnModuleInit {
// uptime — negligible in phase 1's single process.
private readonly settled = new Set<string>();
// #487 runId -> one-shot settle notifier. Kept in a SEPARATE map from `active`
// ON PURPOSE: it must OUTLIVE the `active.delete` claim inside finalizeRun (the
// claim frees the slot the instant finalize starts), so a subscriber can still
// await the outcome after the entry is gone. Created in beginRun, resolved
// EXACTLY ONCE in finalizeRun, then removed (bounded). Absence => this replica
// has no live notifier: a subscriber falls back to the zombie map, then to the
// row (see peekSettled). Process-local (phase-1 single-process assumption).
private readonly settledPromises = new Map<string, Deferred<RunSettleOutcome>>();
// #487 runId -> ZOMBIE record: a run whose terminal write gave up (row stranded
// non-terminal). BOUNDED — an entry is added only on give-up and removed on a
// successful re-drive (settleZombie) or when the row is found already terminal;
// a process restart clears it (and the boot sweep settles the stranded row).
// Process-local (phase-1 single-process assumption).
private readonly zombies = new Map<string, ZombieRun>();
// Bounded retry for the terminal write (F6): a single PK UPDATE can fail
// transiently under many fire-and-forget writes (pool exhaustion, deadlock, a
// brief connection blip). Riding out that blip in-place matters because the
@@ -322,10 +224,6 @@ export class AiChatRunService implements OnModuleInit {
chatId: args.chatId,
workspaceId: args.workspaceId,
});
// #487: arm the one-shot settle notifier BEFORE returning, so a subscriber
// that races in immediately after begin always finds a promise to await. It
// is resolved exactly once when the run settles (or gives up).
this.settledPromises.set(run.id, this.makeDeferred<RunSettleOutcome>());
return { runId: run.id, signal: controller.signal };
}
@@ -365,43 +263,47 @@ export class AiChatRunService implements OnModuleInit {
}
/**
* Finalize a run to its terminal status (succeeded / failed / aborted) via a
* CONDITIONAL UPDATE, stamping finishedAt + any error. Atomically safe against a
* concurrent settle AND robust against a transient terminal-write failure.
* Finalize a run to its terminal status (succeeded / failed / aborted),
* stamping finishedAt + any error. Best-effort, but ROBUST against a transient
* terminal-write failure (F6) AND atomically safe against a concurrent settle.
*
* ATOMIC ONCE-CLAIM (the gate must close in ONE synchronous tick): two
* finalizeRun calls for the SAME run can race the documented real path is
* AiChatService.stream's safety-net catch settling the turn to 'error' while a
* streamText terminal callback (onFinish/onAbort/onError) ALSO settles it. The
* claim happens via `active.delete`, a SYNCHRONOUS check-and-clear with NO await
* between the gate and the entry removal: the second concurrent caller finds the
* entry already gone and returns in the same tick, before any UPDATE.
* `settled.has` check alone is NOT a gate: it is read BEFORE the awaited UPDATE,
* so two callers can both see `false` and both write the row (last-write-wins
* clobbers the real terminal status, and the bounded retry only widens that
* window). The claim therefore happens via `active.delete`, a SYNCHRONOUS
* check-and-clear with NO await between the gate and the entry removal: the
* second concurrent caller finds the entry already gone and returns in the same
* tick, before any UPDATE. The transition "nobody is finalizing" -> "I am
* finalizing" is thus a single atomic step.
*
* ALL TERMINAL WRITES ARE CONDITIONAL (#487): `finalizeIfActive` only flips a
* row still in pending|running (mirror of the assistant message's
* `onlyIfStreaming`). So even a settle that DID reach the UPDATE (e.g. a
* reconcile stamp racing an owner finalize) can never clobber a terminal status
* the loser matches nothing and is a benign no-op. `active.delete` is the
* fast, in-process gate; the conditional WHERE is the authoritative one.
* ORDER MATTERS (F6): once we own the claim, the terminal UPDATE happens FIRST;
* only once it SUCCEEDS do we record the run as settled. If the UPDATE fails on
* every bounded attempt we RESTORE the in-memory entry, leave the run UNsettled,
* and emit an ERROR signal that the row is left non-terminal 'running' (which
* would 409 every future turn in the chat until recovery). An in-process retry
* by a LATER settle is only POSSIBLE, never guaranteed: it needs (a) the entry
* to have been restored at the give-up path AND (b) a fresh settler to arrive
* AFTER that restore. A concurrent settler that arrives DURING the retry window
* while the entry is deleted for backoff and not yet restored is consumed at
* the synchronous `active.delete` claim (it finds nothing to delete and returns
* a no-op), so it does NOT become an in-process retrier. The NO-streamText path
* (the turn threw before streamText was wired, so ONLY the safety-net ever
* settles) likewise has no second in-process settler at all. The UNCONDITIONAL
* backstop in every case is the boot sweep on the next restart (phase 1 has no
* periodic in-process sweep); the retained entry is bounded (cleared on restart)
* and harmless meanwhile.
*
* ZOMBIE ON GIVE-UP (#487): if every bounded attempt THROWS (the DB is down for
* the whole finalize), we do NOT restore the entry. The row is stranded
* non-terminal ('running'); we record a ZOMBIE `{ terminalWriteFailed, intended
* }` (the ONLY thing distinguishing this dead run from a live one) and resolve
* the settle notifier with `terminalWriteFailed: true`. A restore would make the
* zombie indistinguishable from a live run to every reader; instead a re-drive
* (settleZombie, called by the periodic reconcile / supersede / opportunistic
* paths) applies the intended status later via the same conditional UPDATE.
*
* DOCUMENTED LOSS (#487, single-process phase 1): if the process RESTARTS before
* a zombie is re-driven, the in-memory zombie map is gone and the boot sweep
* (unconditional) writes 'aborted' over the ACTUAL intended status. This is
* unavoidable while the run lifecycle is single-process there is no durable
* record of `intended`; a cross-process durable intent is deferred to phase 2.
*
* IDEMPOTENT: the settle notifier resolves EXACTLY ONCE; a second settle is
* stopped at `settled.has` or the `active.delete` claim, so a double-settle
* collapses to a single write and can never double-resolve or clobber the row.
* IDEMPOTENT on SUCCESS (#184 review): the terminal write happens AT MOST ONCE
* per run. After a successful write the once-gate keys off {@link settled} (the
* terminal row already written) so a settle arriving AFTER the entry was already
* dropped-and-settled returns early; a settle racing the in-flight write is
* stopped earlier still, by the `active.delete` claim. Either way a genuine
* double-settle collapses to a single write and a late settle can never clobber
* the real terminal status or double-write the row.
*/
async finalizeRun(
runId: string,
@@ -412,17 +314,13 @@ export class AiChatRunService implements OnModuleInit {
// ---- Atomic once-claim (synchronous; NO await before the gate closes) ----
// Already terminally written -> idempotent no-op.
if (this.settled.has(runId)) return;
// Capture the entry BEFORE the delete for the give-up log context.
// Capture the entry BEFORE the delete so a total-failure path can restore it.
const entry = this.active.get(runId);
// SYNCHRONOUS check-and-clear: the FIRST caller deletes (claims) the entry;
// any concurrent SECOND caller finds nothing to delete and returns HERE, in
// the same tick, before any await — so it can never reach the UPDATE.
if (!this.active.delete(runId)) return;
const status = mapTurnStatusToRun(turnStatus);
const err = error ?? null;
const chatId = entry?.chatId ?? 'unknown';
let lastError: unknown;
for (
let attempt = 1;
@@ -430,294 +328,47 @@ export class AiChatRunService implements OnModuleInit {
attempt++
) {
try {
const row = await this.runRepo.finalizeIfActive(runId, workspaceId, {
status,
error: err,
await this.runRepo.update(runId, workspaceId, {
status: mapTurnStatusToRun(turnStatus),
finishedAt: new Date(),
error: error ?? null,
});
// No throw => the row is now terminal (we wrote it, or it was ALREADY
// terminal — another writer won the conditional UPDATE, a benign no-op).
// Terminal write landed: arm the once-gate. The entry is already gone
// (claimed above); we do NOT restore it. The slot is now free.
this.settled.add(runId);
this.zombies.delete(runId);
// Resolve with the persisted outcome: our status when WE wrote it, else
// the row's real terminal status (re-read on the already-terminal path so
// a subscriber never sees a status we did not actually persist).
const outcome: RunSettleOutcome = row
? { status, error: err, terminalWriteFailed: false }
: await this.readTerminalOutcome(runId, workspaceId, status, err);
this.resolveSettled(runId, outcome);
return;
} catch (err2) {
lastError = err2;
} catch (err) {
lastError = err;
this.logger.warn(
`Failed to finalize run ${runId} (attempt ${attempt}/${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
}): ${err2 instanceof Error ? err2.message : 'unknown error'}`,
}): ${err instanceof Error ? err.message : 'unknown error'}`,
);
if (attempt < AiChatRunService.FINALIZE_MAX_ATTEMPTS) {
await this.delay(AiChatRunService.FINALIZE_RETRY_BASE_MS * attempt);
}
}
}
// Every attempt threw: GIVE UP. The row is stranded non-terminal ('running').
// Do NOT restore the entry (a restored entry is indistinguishable from a live
// run); leave a ZOMBIE record instead, and resolve the notifier as
// terminalWriteFailed so a subscriber knows the slot still needs the intended
// status applied. One explicit, greppable ERROR so an operator can tell a
// give-up from a per-attempt blip.
// Every attempt failed: this is a give-up, materially worse than a per-attempt
// blip — the row is left NON-TERMINAL ('running'), so emit ONE explicit,
// greppable ERROR so an operator can tell "survived a blip" from "gave up, run
// held in memory until recovery" (the last warn alone says only "attempt 3/3").
this.logger.error(
`Run ${runId} (chat ${chatId}) left NON-TERMINAL ('running'): terminal ` +
`write failed after ${AiChatRunService.FINALIZE_MAX_ATTEMPTS} attempts; ` +
`ZOMBIE recorded (intended '${status}'), recovery deferred to reconcile / ` +
`supersede / boot sweep`,
`Run ${runId} (chat ${entry?.chatId ?? 'unknown'}) left NON-TERMINAL ` +
`('running'): terminal write failed after ${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
} attempts; entry retained in memory, recovery deferred to next settle / ` +
`boot sweep`,
lastError,
);
this.zombies.set(runId, {
workspaceId,
chatId,
intended: { status, error: err },
});
this.resolveSettled(runId, { status, error: err, terminalWriteFailed: true });
}
/**
* #487: re-drive a zombie run's intended terminal write (the conditional
* UPDATE). Called by the periodic reconcile (commit 4), an opportunistic
* single-chat reconcile, and supersede (commit 3). On success the row is now
* terminal (written OR found already terminal) the zombie is cleared and the
* once-gate armed; on another failure the zombie is kept for a later retry.
* Returns true when the row is now terminal. Best-effort; never throws.
*/
async settleZombie(runId: string): Promise<boolean> {
const z = this.zombies.get(runId);
if (!z) return false;
try {
await this.runRepo.finalizeIfActive(runId, z.workspaceId, {
status: z.intended.status,
error: z.intended.error,
});
this.zombies.delete(runId);
this.settled.add(runId);
return true;
} catch (err) {
this.logger.warn(
`Re-drive of zombie run ${runId} (chat ${z.chatId}) failed; will retry ` +
`later: ${err instanceof Error ? err.message : 'unknown error'}`,
);
return false;
}
}
/**
* #487 reconcile clause (c): abort runs the DB still shows active (pending|
* running) but that this replica does NOT own NO live entry AND NO zombie
* and that have been UNTOUCHED past `staleMs` (from last-progress `updated_at`,
* NOT startedAt, so a legit long marathon is never a candidate). "No entry" is
* the PRIMARY gate: a live entry (an actively-executing run on this replica) is
* NEVER aborted, whatever its age. Returns the number aborted. Best-effort
* never throws (a periodic-job failure must not crash the process).
*/
async reconcileStaleRuns(staleMs: number): Promise<number> {
let candidates: Array<{ id: string; workspaceId: string; chatId: string }>;
try {
candidates = await this.runRepo.findStaleActive(staleMs);
} catch (err) {
this.logger.warn(
`Reconcile (stale runs) query failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
return 0;
}
let aborted = 0;
for (const c of candidates) {
// PRIMARY gate: never touch a live entry, and never race a zombie we are
// already re-driving (settleZombie owns those).
if (this.active.has(c.id) || this.zombies.has(c.id)) continue;
try {
const row = await this.runRepo.finalizeIfActive(c.id, c.workspaceId, {
status: 'aborted',
error: 'Run aborted by reconcile: no live runner (stale).',
});
if (row) {
aborted += 1;
this.settled.add(c.id);
}
} catch (err) {
this.logger.warn(
`Reconcile abort of stale run ${c.id} failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
return aborted;
}
/**
* #487: the run's settle outcome as seen by THIS replica, or undefined when it
* has no record (the caller then reads the row the DB is the source of truth).
* A LIVE deferred (still settling, or resolved-but-not-yet-consumed) wins; a
* ZOMBIE synthesizes the give-up outcome. A subscriber (supersede) races this
* against a timeout.
*/
peekSettled(runId: string): Promise<RunSettleOutcome> | undefined {
const d = this.settledPromises.get(runId);
if (d) return d.promise;
const z = this.zombies.get(runId);
if (z) {
return Promise.resolve({
status: z.intended.status,
error: z.intended.error,
terminalWriteFailed: true,
});
}
return undefined;
}
/**
* #487: await a run's settle outcome, bounded by `timeoutMs`. Returns the
* outcome on settle, or undefined on TIMEOUT (or when this replica has no record
* of the run and its row is not terminal). Uses the LIVE settle notifier / the
* zombie synth when present; else reads the row (the DB is the source of truth
* once the in-memory record is gone). The subscriber (supersede) grabs this
* right after Stop; commit 1's race makes the settle land in ms on a healthy DB.
*/
async awaitSettled(
runId: string,
workspaceId: string,
timeoutMs: number,
): Promise<RunSettleOutcome | undefined> {
const pending = this.peekSettled(runId);
if (pending) {
let timer: ReturnType<typeof setTimeout> | undefined;
const timeout = new Promise<undefined>((resolve) => {
timer = setTimeout(() => resolve(undefined), timeoutMs);
timer.unref?.();
});
try {
return await Promise.race([pending, timeout]);
} finally {
if (timer) clearTimeout(timer);
}
}
// No live notifier and no zombie: read the row (already settled-and-written,
// or unknown here). A terminal row is an outcome; anything else -> undefined.
const row = await this.runRepo.findById(runId, workspaceId);
if (row && isRunTerminal(row.status)) {
return {
status: row.status as RunTerminalStatus,
error: row.error ?? null,
terminalWriteFailed: false,
};
}
return undefined;
}
/**
* #487: the SERVER supersede CAS for `POST /stream { supersede: { runId: X } }`.
* Atomically transitions "X is the chat's active run" -> "X is stopped, settled,
* slot free" so the caller can start a replacement run. See {@link
* SupersedeResult} for the branch semantics.
*
* On a `ready` result the caller MUST still go through the normal beginRun gate
* (the partial unique index) between the slot freeing here and beginRun a
* neighbouring tab's ordinary POST can win the slot (documented SLOT-THEFT: the
* loser then gets a MISMATCH carrying the NEW runId). There is also NO side-
* effect quiescence: an in-flight write of the stopped run may still land AFTER
* the new run starts (commit 1 stops the NEXT call, not one already committing),
* so the caller adds a prompt note to the new run.
*/
async supersede(
chatId: string,
targetRunId: string,
workspaceId: string,
timeoutMs: number = SUPERSEDE_SETTLE_TIMEOUT_MS,
): Promise<SupersedeResult> {
// Validate the target belongs to THIS chat (a CAS targeting another chat's run
// is malformed -> 400). A missing row is NOT invalid: the run may have ended
// and been pruned; the active-run check below decides degrade vs mismatch.
const target = await this.getRun(targetRunId, workspaceId);
if (target && target.chatId !== chatId) return { kind: 'invalid' };
const active = await this.getActiveForChat(chatId, workspaceId);
// No active run: it ended between the client's click and this POST — this is a
// DEGRADE to a normal send, NOT a mismatch (the user's intent still holds).
if (!active) return { kind: 'degrade' };
// A DIFFERENT run is active than the one the client saw -> mismatch. The
// client does not auto-retry; it surfaces the new runId.
if (active.id !== targetRunId) {
return { kind: 'mismatch', activeRunId: active.id };
}
// The target IS active: stop it, then await its settle within W.
await this.requestStop(targetRunId, workspaceId);
const outcome = await this.awaitSettled(targetRunId, workspaceId, timeoutMs);
if (!outcome) return { kind: 'timeout' };
// Gave up (terminal write failed): apply the intended status via the
// conditional UPDATE so the slot actually frees. If that ALSO fails, the row
// is still stranded -> treat as a timeout (nothing persisted for the new run).
if (outcome.terminalWriteFailed) {
const settled = await this.settleZombie(targetRunId);
if (!settled) return { kind: 'timeout' };
}
return { kind: 'ready' };
}
/** #487 test/diagnostic seam: whether a give-up zombie is held for this run. */
hasZombie(runId: string): boolean {
return this.zombies.has(runId);
}
/** #487: every zombie runId held on this replica (reconcile clause a, commit 4). */
zombieRunIds(): string[] {
return [...this.zombies.keys()];
}
/** #487: create a one-shot deferred (resolve captured for a later single call). */
private makeDeferred<T>(): Deferred<T> {
let resolve!: (value: T) => void;
const promise = new Promise<T>((r) => {
resolve = r;
});
return { promise, resolve };
}
/** #487: resolve a run's settle notifier EXACTLY ONCE, then drop it (bounded).
* A subscriber that already grabbed the promise still resolves; a later one
* falls back to the zombie map / the row (see peekSettled). */
private resolveSettled(runId: string, outcome: RunSettleOutcome): void {
const d = this.settledPromises.get(runId);
if (!d) return;
this.settledPromises.delete(runId);
d.resolve(outcome);
}
/** #487: read the persisted terminal outcome when the conditional finalize was a
* no-op (the row was already terminal). Falls back to the intended status when
* the read fails or the row is unexpectedly missing/non-terminal. */
private async readTerminalOutcome(
runId: string,
workspaceId: string,
fallbackStatus: RunTerminalStatus,
fallbackError: string | null,
): Promise<RunSettleOutcome> {
try {
const row = await this.runRepo.findById(runId, workspaceId);
if (row && isRunTerminal(row.status)) {
return {
status: row.status as RunTerminalStatus,
error: row.error ?? null,
terminalWriteFailed: false,
};
}
} catch {
// Fall through to the intended status — best-effort only.
}
return {
status: fallbackStatus,
error: fallbackError,
terminalWriteFailed: false,
};
// RESTORE the claimed entry (and leave the run UNsettled) so a LATER settle
// that arrives AFTER this restore MAY retry the terminal write — but that
// in-process retry is NOT guaranteed (a concurrent settler caught in the retry
// window above is consumed at the `active.delete` claim, and the no-streamText
// path has no second settler at all). The UNCONDITIONAL backstop in every case
// is the boot sweep on the next restart; the restored entry is bounded and
// cleared on restart.
if (entry) this.active.set(runId, entry);
}
/** Small async backoff between terminal-write retries (F6). Isolated so it is
@@ -115,7 +115,7 @@ describe('finalizeAssistant dispatch (planFinalizeAssistant + applyFinalize)', (
// Drive the SAME applyFinalize the service calls (no duplicated logic).
async function dispatchFinalize(
repo: { insert: jest.Mock; finalizeOwner: jest.Mock },
repo: { insert: jest.Mock; update: jest.Mock },
assistantId: string | undefined,
flushed: AssistantFlush,
): Promise<void> {
@@ -135,22 +135,21 @@ describe('finalizeAssistant dispatch (planFinalizeAssistant + applyFinalize)', (
expect(planFinalizeAssistant(undefined)).toEqual({ kind: 'insert' });
});
it('(a) upfront insert succeeded -> finalize CONDITIONALLY updates the row by id (#487 owner-write)', async () => {
const repo = { insert: jest.fn(), finalizeOwner: jest.fn() };
it('(a) upfront insert succeeded -> finalize UPDATEs the row by id', async () => {
const repo = { insert: jest.fn(), update: jest.fn() };
const flushed = flushAssistant([], 'final answer', 'completed', {
finishReason: 'stop',
});
await dispatchFinalize(repo, 'a1', flushed);
// #487: the owner write is the CONDITIONAL finalizeOwner, not a raw update.
expect(repo.finalizeOwner).toHaveBeenCalledWith('a1', workspaceId, flushed);
expect(repo.update).toHaveBeenCalledWith('a1', workspaceId, flushed);
expect(repo.insert).not.toHaveBeenCalled();
});
it('(b) upfront insert failed -> finalize INSERTs the terminal payload', async () => {
const repo = { insert: jest.fn(), finalizeOwner: jest.fn() };
const repo = { insert: jest.fn(), update: jest.fn() };
const flushed = flushAssistant([], 'partial', 'error', { error: 'boom' });
await dispatchFinalize(repo, undefined, flushed);
expect(repo.finalizeOwner).not.toHaveBeenCalled();
expect(repo.update).not.toHaveBeenCalled();
expect(repo.insert).toHaveBeenCalledTimes(1);
const arg = repo.insert.mock.calls[0][0];
// The fallback insert carries the terminal content/status/metadata.
@@ -1,279 +0,0 @@
import {
BadRequestException,
ConflictException,
ForbiddenException,
HttpException,
} from '@nestjs/common';
import { AiChatController } from './ai-chat.controller';
import type { User, Workspace } from '@docmost/db/types/entity.types';
/**
* #487 commit 3 the single concurrency GATE (both modes) + the server supersede
* CAS, at the controller boundary. The gate + CAS run BEFORE res.hijack(), so a
* rejected concurrent start / a CAS branch returns clean JSON (an HttpException
* the controller's post-hijack catch re-serializes). These assert the OBSERVABLE
* HTTP contract against the real controller + a stubbed run service.
*/
describe('#487 AiChatController.stream — gate + supersede', () => {
const user = { id: 'u1' } as User;
function wsWith(autonomousRuns: boolean): Workspace {
return {
id: 'ws1',
settings: { ai: { chat: true, autonomousRuns } },
} as unknown as Workspace;
}
function makeReqRes(body: Record<string, unknown>) {
const req = {
raw: { sessionId: 'sess', once: jest.fn(), destroyed: false },
body,
};
const res = {
raw: {
writableEnded: false,
headersSent: false,
on: jest.fn(),
once: jest.fn(),
setHeader: jest.fn(),
end: jest.fn(),
statusCode: 200,
flushHeaders: jest.fn(),
},
hijack: jest.fn(),
status: jest.fn().mockReturnThis(),
send: jest.fn(),
};
return { req, res };
}
function makeController(
runServiceOverrides: Record<string, jest.Mock>,
// The chat assertOwnedChat resolves. Default: a chat OWNED by `user` (u1), so
// the ownership gate is transparent to the gate/CAS assertions below. Pass a
// foreign-owner (or undefined) chat to exercise the #487 owner rejection.
chat: { creatorId: string } | undefined = { creatorId: 'u1' },
) {
const aiChatService = {
resolveRoleForRequest: jest.fn().mockResolvedValue(null),
getChatModel: jest.fn().mockResolvedValue({}),
stream: jest.fn().mockResolvedValue(undefined),
};
const aiChatRunService = {
getActiveForChat: jest.fn().mockResolvedValue(undefined),
supersede: jest.fn(),
beginRun: jest.fn().mockResolvedValue({
runId: 'run-new',
signal: new AbortController().signal,
}),
linkAssistantMessage: jest.fn(),
recordStep: jest.fn(),
finalizeRun: jest.fn(),
requestStop: jest.fn(),
...runServiceOverrides,
};
const aiChatRepo = { findById: jest.fn().mockResolvedValue(chat) };
const controller = new AiChatController(
aiChatService as never,
aiChatRunService as never,
aiChatRepo as never, // aiChatRepo
{} as never, // aiChatMessageRepo
{} as never, // aiTranscription
{} as never, // pageRepo
);
return { controller, aiChatService, aiChatRunService, aiChatRepo };
}
const codeOf = (err: unknown) =>
(((err as HttpException).getResponse() as Record<string, unknown>) ?? {})
.code;
describe('single concurrency gate — BOTH modes reject the second tab with 409', () => {
for (const autonomousRuns of [true, false]) {
it(`rejects a concurrent start with 409 A_RUN_ALREADY_ACTIVE (autonomousRuns=${autonomousRuns})`, async () => {
const { controller, aiChatRunService } = makeController({
getActiveForChat: jest
.fn()
.mockResolvedValue({ id: 'run-live', chatId: 'c1' }),
});
const { req, res } = makeReqRes({ chatId: 'c1' });
let thrown: unknown;
try {
await controller.stream(
req as never,
res as never,
user,
wsWith(autonomousRuns),
);
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect((thrown as HttpException).getStatus()).toBe(409);
expect(codeOf(thrown)).toBe('A_RUN_ALREADY_ACTIVE');
// Rejected BEFORE committing to the stream (no hijack, no service.stream).
expect(res.hijack).not.toHaveBeenCalled();
expect(aiChatRunService.getActiveForChat).toHaveBeenCalledWith(
'c1',
'ws1',
);
});
}
});
// #487 [security, F1]: stream() MUST owner-gate an existing chat exactly like its
// six sibling endpoints, BEFORE the supersede CAS. Otherwise a same-workspace
// non-owner could POST a supersede against another user's chat and (a) harvest
// that user's active runId from the 409 SUPERSEDE_TARGET_MISMATCH body, then (b)
// requestStop the foreign run. The gate must reject FIRST — no run lookup, no
// supersede, no stop, no runId leak.
describe('cross-user ownership gate (F1)', () => {
it('a non-owner streaming against someone else\'s chat is rejected (403) with NO runId leak and NO foreign requestStop', async () => {
// A live run exists on the victim's chat. Without the gate the supersede CAS
// would run and (faithful to the run service) return a MISMATCH carrying the
// victim's runId — the exact leak. With the gate it must never be reached.
const getActiveForChat = jest
.fn()
.mockResolvedValue({ id: 'run-victim', chatId: 'c-other' });
const supersede = jest
.fn()
.mockResolvedValue({ kind: 'mismatch', activeRunId: 'run-victim' });
const requestStop = jest.fn();
const { controller, aiChatService } = makeController(
{ getActiveForChat, supersede, requestStop },
{ creatorId: 'someone-else' }, // the chat is NOT owned by u1
);
const { req, res } = makeReqRes({
chatId: 'c-other',
supersede: { runId: 'guessed-uuid' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
// Rejected by the ownership gate (403), the SAME shape the neighbors use.
expect(thrown).toBeInstanceOf(ForbiddenException);
expect((thrown as HttpException).getStatus()).toBe(403);
// Crucially NOT a 409 that would carry activeRunId — no runId is leaked.
const payload = JSON.stringify(
(thrown as HttpException).getResponse() ?? {},
);
expect(payload).not.toContain('run-victim');
expect(codeOf(thrown)).not.toBe('SUPERSEDE_TARGET_MISMATCH');
// The gate short-circuits BEFORE any run machinery runs.
expect(getActiveForChat).not.toHaveBeenCalled();
expect(supersede).not.toHaveBeenCalled();
expect(requestStop).not.toHaveBeenCalled();
expect(aiChatService.stream).not.toHaveBeenCalled();
expect(res.hijack).not.toHaveBeenCalled();
});
});
it('supersede MISMATCH -> 409 SUPERSEDE_TARGET_MISMATCH carrying the current runId', async () => {
const { controller } = makeController({
supersede: jest
.fn()
.mockResolvedValue({ kind: 'mismatch', activeRunId: 'run-other' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect(codeOf(thrown)).toBe('SUPERSEDE_TARGET_MISMATCH');
expect(
((thrown as HttpException).getResponse() as Record<string, unknown>)
.activeRunId,
).toBe('run-other');
expect(res.hijack).not.toHaveBeenCalled();
});
it('supersede TIMEOUT -> 409 SUPERSEDE_TIMEOUT, nothing streamed', async () => {
const { controller } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'timeout' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(false));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect(codeOf(thrown)).toBe('SUPERSEDE_TIMEOUT');
expect(res.hijack).not.toHaveBeenCalled();
});
it('supersede INVALID (target on another chat) -> 400 SUPERSEDE_INVALID', async () => {
const { controller } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'invalid' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(BadRequestException);
expect(codeOf(thrown)).toBe('SUPERSEDE_INVALID');
});
it('supersede without chatId -> 400 SUPERSEDE_INVALID', async () => {
const { controller, aiChatRunService } = makeController({});
const { req, res } = makeReqRes({ supersede: { runId: 'run-x' } });
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(BadRequestException);
expect(codeOf(thrown)).toBe('SUPERSEDE_INVALID');
expect(aiChatRunService.supersede).not.toHaveBeenCalled();
});
it('supersede READY -> proceeds to stream with superseded=true', async () => {
const { controller, aiChatService } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'ready' }),
getActiveForChat: jest.fn().mockResolvedValue(undefined), // slot free after CAS
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
await controller.stream(req as never, res as never, user, wsWith(true));
expect(res.hijack).toHaveBeenCalled();
expect(aiChatService.stream).toHaveBeenCalledTimes(1);
expect(aiChatService.stream.mock.calls[0][0].superseded).toBe(true);
// The run hooks are always present now (both modes).
expect(aiChatService.stream.mock.calls[0][0].runHooks).toBeDefined();
});
it('supersede DEGRADE -> proceeds to a normal send (superseded=false)', async () => {
const { controller, aiChatService } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'degrade' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
await controller.stream(req as never, res as never, user, wsWith(false));
expect(aiChatService.stream).toHaveBeenCalledTimes(1);
expect(aiChatService.stream.mock.calls[0][0].superseded).toBe(false);
});
});
@@ -418,19 +418,6 @@ export class AiChatController {
const body = (req.body ?? {}) as AiChatStreamBody;
// #487 [security]: gate cross-user access to an EXISTING chat BEFORE anything
// reads its runs. Every sibling endpoint (getRun/stop/history/rename/delete/
// attachRunStream) owner-checks the chat via assertOwnedChat; stream() must too.
// Without this a same-workspace member who is NOT the chat owner could POST a
// supersede against another user's chat and (a) harvest that user's active runId
// out of the 409 SUPERSEDE_TARGET_MISMATCH body, then (b) requestStop the foreign
// run. Gate on the chatId the client sent, when present — a brand-new chat (no
// chatId) has no prior owner to check. Mirrors /stop's owner check (403 as the
// neighbors do), and runs pre-hijack so it returns clean JSON.
if (body.chatId) {
await this.assertOwnedChat(body.chatId, user, workspace);
}
// Resolve the agent role for this turn BEFORE hijack: existing chats read it
// from ai_chats.role_id (authoritative), a new chat from body.roleId. The
// role drives both the persona and the optional model override below.
@@ -445,66 +432,12 @@ export class AiChatController {
// HttpException) instead of breaking mid-stream.
const model = await this.aiChatService.getChatModel(workspace.id, role);
// #487: server-side supersede CAS ("interrupt and send now"). When the client
// asks to replace a live run, atomically STOP it and wait for it to settle
// before this turn claims the slot. Runs BEFORE hijack so every branch returns
// clean JSON (the client keeps the composer text on a 409). See
// AiChatRunService.supersede for the branch semantics.
let superseded = false;
const supersedeRunId = body.supersede?.runId;
if (supersedeRunId) {
if (!body.chatId) {
throw new BadRequestException({
message: 'supersede requires chatId',
code: 'SUPERSEDE_INVALID',
});
}
const result = await this.aiChatRunService.supersede(
body.chatId,
supersedeRunId,
workspace.id,
);
switch (result.kind) {
case 'invalid':
throw new BadRequestException({
message: 'The run to supersede does not belong to this chat',
code: 'SUPERSEDE_INVALID',
});
case 'mismatch':
// A DIFFERENT run is active than the one the client targeted. Surface
// the CURRENT runId; the client does NOT auto-retry (a stale CAS).
throw new ConflictException({
message: 'A different agent run is now active on this chat',
code: 'SUPERSEDE_TARGET_MISMATCH',
activeRunId: result.activeRunId,
});
case 'timeout':
// The target did not settle within W — nothing was persisted, the
// composer keeps the text. NOT a rollback: the stop is already issued.
throw new ConflictException({
message:
'The previous run did not stop in time; nothing was sent — please try again',
code: 'SUPERSEDE_TIMEOUT',
});
case 'ready':
// The target stopped and settled: the slot is free. Prompt the new run
// that the old run's last operations may still be applying.
superseded = true;
break;
case 'degrade':
// The run already ended between click and POST — send normally.
break;
}
}
// #487: one active run per chat — ENFORCED IN BOTH MODES now (legacy mode used
// to have NO gate, so two tabs streamed two parallel turns on one chat, which
// interleaved history and crashed convertToModelMessages). Reject a concurrent
// start with a clean pre-hijack 409 (double-submit / second-tab). A brand-new
// chat (no chatId) cannot have a prior run, and the DB partial unique index in
// beginRun is the authoritative backstop for any race that slips past here
// (including a slot stolen between a supersede release and beginRun).
if (body.chatId) {
// #184: one active run per chat. For an EXISTING chat reject a concurrent
// start with a clean 409 BEFORE hijack (the common double-submit / second-tab
// case), so the user gets JSON, not a mid-stream error. A brand-new chat
// (no chatId) cannot have a prior run, and the DB partial unique index is the
// backstop against any race that slips past this check.
if (autonomousRuns && body.chatId) {
const active = await this.aiChatRunService.getActiveForChat(
body.chatId,
workspace.id,
@@ -513,94 +446,107 @@ export class AiChatController {
throw new ConflictException({
message: 'An agent run is already in progress for this chat',
code: 'A_RUN_ALREADY_ACTIVE',
activeRunId: active.id,
});
}
}
// #487: the turn is ALWAYS a first-class RUN now (both modes). The mode
// difference is only the abort semantics on a browser disconnect (onClose
// below). currentRunId is captured at begin so a legacy disconnect can stop
// the run through its stop lever.
let currentRunId: string | undefined;
const runHooks: AiChatRunHooks = {
begin: async (chatId) => {
const handle = await this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
});
currentRunId = handle?.runId;
// #184 phase 1.5: register the run-stream entry at BEGIN (before any
// frame) so a tab that attaches in the begin->seed window finds an entry
// to wait on. Gated on AI_CHAT_RESUMABLE_STREAM.
if (
handle?.runId &&
this.environment?.isAiChatResumableStreamEnabled?.()
) {
this.streamRegistry?.open(chatId, handle.runId);
// Run-lifecycle hooks (#184), only when the flag is on. They wrap the turn in
// a durable run whose abort is governed by the run (explicit stop), persist
// its progress, and settle its terminal status — see AiChatRunService.
const runHooks: AiChatRunHooks | undefined = autonomousRuns
? {
begin: async (chatId) => {
const handle = await this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
});
// #184 phase 1.5: register the run-stream entry at BEGIN (before any
// frame) so a tab that attaches in the begin->seed window finds an
// entry to wait on. Gated on AI_CHAT_RESUMABLE_STREAM: with the flag
// off nothing is registered and attach always 204s.
if (
handle?.runId &&
this.environment?.isAiChatResumableStreamEnabled?.()
) {
this.streamRegistry?.open(chatId, handle.runId);
}
return handle;
},
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
workspace.id,
messageId,
),
onStep: (runId, stepCount) =>
void this.aiChatRunService.recordStep(
runId,
workspace.id,
stepCount,
),
onSettled: (runId, status, error) =>
this.aiChatRunService.finalizeRun(
runId,
workspace.id,
status,
error,
),
}
return handle;
},
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
workspace.id,
messageId,
),
onStep: (runId, stepCount) =>
void this.aiChatRunService.recordStep(runId, workspace.id, stepCount),
onSettled: (runId, status, error) =>
this.aiChatRunService.finalizeRun(runId, workspace.id, status, error),
};
: undefined;
// Handle a client disconnect. `close` also fires on normal completion, so only
// act when the response has not finished writing (a genuine disconnect). `once`
// fires at most once and self-removes; we also drop it on response `finish`.
// Abort the agent loop when the client disconnects. `close` also fires on
// normal completion, so only abort when the response has not finished
// writing (a genuine disconnect). `once` fires at most once and self-removes;
// we also drop it on response `finish` so it never lingers after the stream
// completes normally (the AI SDK pipes the response fire-and-forget, so we
// cannot simply remove it once `stream()` returns).
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: wall-clock at
// which a Safari disconnect is observed, measured from request receipt.
const reqStartedAt = Date.now();
const controller = new AbortController();
const onClose = (): void => {
// A genuine disconnect leaves the response unfinished (unlike a normal
// completion, which also fires `close`). Such a drop — e.g. a reverse
// proxy cutting the SSE mid-answer — is otherwise invisible server-side,
// so log it here.
if (!res.raw.writableEnded) {
if (autonomousRuns) {
// #184: a DETACHED run — a disconnect must NOT stop it. The run keeps
// executing and persisting server-side; the client reconnects via
// /ai-chat/run (or re-stops via /ai-chat/stop). Log only.
// #184: the turn is a DETACHED run. A disconnect must NOT abort it —
// the run keeps executing and persisting server-side; the client
// reconnects via /ai-chat/run (or re-stops via /ai-chat/stop). Log only.
this.logger.log(
`AI chat stream: client disconnected; run continues server-side ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
} else {
// #487: legacy — a disconnect ENDS the turn, but the turn is now a RUN,
// so stop it through the run's stop lever (requestStop). streamText no
// longer consumes the socket signal (effectiveSignal is the run signal),
// so aborting `controller` would do nothing; requestStop aborts the run.
this.logger.warn(
`AI chat stream: client disconnected before completion; stopping the ` +
`run (elapsed=${Date.now() - reqStartedAt}ms since request received)`,
`AI chat stream: client disconnected before completion; aborting turn ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
if (currentRunId) {
void this.aiChatRunService.requestStop(currentRunId, workspace.id);
}
controller.abort();
}
}
};
req.raw.once('close', onClose);
res.raw.once('finish', () => req.raw.off('close', onClose));
// #184/#487: the run/pipe can outlive the socket in BOTH modes now (autonomous
// keeps going; legacy keeps going until requestStop's abort unwinds the turn).
// The SDK's pipe may then write to a dropped socket and emit an 'error' on the
// raw response — swallow it so it never surfaces as an unhandled error event.
res.raw.on('error', (err) => {
this.logger.debug(
`AI chat stream: post-disconnect socket error swallowed: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
// #184: in detached mode the turn is NOT aborted on disconnect, so the SDK's
// pipe keeps writing to a socket the client may have dropped — for the rest of
// the (continuing) run. A write to the dead socket can emit an 'error' on the
// raw response; without a listener that surfaces as an unhandled error event.
// Swallow it (the run continues server-side regardless). Legacy mode aborts on
// disconnect, so it does not need this and keeps its exact prior behavior.
if (autonomousRuns) {
res.raw.on('error', (err) => {
this.logger.debug(
`AI chat detached stream: post-disconnect socket error swallowed: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
}
// Commit to streaming: hijack so Fastify stops managing the response and
// the AI SDK can write the UI-message stream directly to the Node socket.
@@ -616,10 +562,8 @@ export class AiChatController {
signal: controller.signal,
model,
role,
// #487: the turn is always run-wrapped now (both modes).
// #184: present only when the flag is on; wraps the turn in a durable run.
runHooks,
// #487: warn the new run that a superseded run's last ops may still apply.
superseded,
});
} catch (err) {
// Any failure AFTER hijack can no longer go through Nest's exception
@@ -1,142 +0,0 @@
// #489 — client-parts validation + resilient history conversion.
//
// These unit tests exercise the two exported helpers against the REAL
// `convertToModelMessages` from `ai` (NOT a mock): a genuinely malformed part
// (a `null` element inside a parts array) makes the real converter throw
// ("Cannot read properties of null"), which is the actual production
// "bricked chat" mechanism this fix defends against. Asserting against the real
// converter (rather than a mock-shaped error) is the whole point — a mock would
// hide a version change in the converter's throw behaviour.
import { convertToModelMessages, type UIMessage } from 'ai';
import {
sanitizeUserParts,
convertHistoryResilient,
TOOL_CONTEXT_OMITTED_MARKER,
} from './ai-chat.service';
type Row = Omit<UIMessage, 'id'> & { id: string };
describe('sanitizeUserParts (#489, branch: validation on receipt)', () => {
it('keeps whitelisted text parts unchanged', () => {
const drops: string[] = [];
const out = sanitizeUserParts(
[
{ type: 'text', text: 'a' },
{ type: 'text', text: 'b' },
] as UIMessage['parts'],
(t) => drops.push(t),
);
expect(out).toEqual([
{ type: 'text', text: 'a' },
{ type: 'text', text: 'b' },
]);
expect(drops).toEqual([]);
});
it('drops a non-text part (a tool-part in input-available) and reports its type', () => {
const drops: string[] = [];
const out = sanitizeUserParts(
[
{ type: 'text', text: 'hi' },
{
type: 'tool-getPage',
toolCallId: 't1',
state: 'input-available',
input: { pageId: 'p' },
},
] as unknown as UIMessage['parts'],
(t) => drops.push(t),
);
expect(out).toEqual([{ type: 'text', text: 'hi' }]);
expect(drops).toEqual(['tool-getPage']);
});
it('drops a null part (the shape that would poison convertToModelMessages)', () => {
const drops: string[] = [];
const out = sanitizeUserParts(
[{ type: 'text', text: 'hi' }, null] as unknown as UIMessage['parts'],
(t) => drops.push(t),
);
expect(out).toEqual([{ type: 'text', text: 'hi' }]);
expect(drops).toEqual(['(unknown)']);
});
it('returns undefined when nothing survives (so a null metadata is persisted)', () => {
const out = sanitizeUserParts(
[
{ type: 'tool-x', toolCallId: 't', state: 'input-available' },
] as unknown as UIMessage['parts'],
() => undefined,
);
expect(out).toBeUndefined();
});
it('returns undefined for a non-array input', () => {
expect(
sanitizeUserParts(undefined as unknown as UIMessage['parts'], () => undefined),
).toBeUndefined();
});
});
describe('convertHistoryResilient (#489, branches: happy + per-row degradation)', () => {
it('happy path: healthy history converts identically to convertToModelMessages, no degrade', async () => {
const history: Row[] = [
{ id: 'u1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
{ id: 'a1', role: 'assistant', parts: [{ type: 'text', text: 'hello' }] },
];
const degrades: number[] = [];
const out = await convertHistoryResilient(history, (i) => degrades.push(i));
const expected = await convertToModelMessages(history as UIMessage[]);
expect(out).toEqual(expected);
expect(degrades).toEqual([]);
});
it('REAL poison: a null part throws in the batch converter but is isolated and degraded to a marker', async () => {
// Sanity: the real converter genuinely throws on this shape.
const poisoned: Row = {
id: 'a1',
role: 'assistant',
parts: [
{ type: 'text', text: 'earlier answer' },
null,
] as unknown as UIMessage['parts'],
};
await expect(
convertToModelMessages([poisoned as UIMessage]),
).rejects.toThrow();
const history: Row[] = [
{ id: 'u1', role: 'user', parts: [{ type: 'text', text: 'first' }] },
poisoned,
{ id: 'u2', role: 'user', parts: [{ type: 'text', text: 'second' }] },
];
const degrades: number[] = [];
const out = await convertHistoryResilient(history, (i) => degrades.push(i));
// Only the poisoned row (index 1) is degraded.
expect(degrades).toEqual([1]);
// Healthy rows survive verbatim.
const flat = JSON.stringify(out);
expect(flat).toContain('first');
expect(flat).toContain('second');
// The degraded row carries its readable text AND the truncation marker so the
// model sees that tool context was omitted (never a silent loss).
expect(flat).toContain('earlier answer');
expect(flat).toContain(TOOL_CONTEXT_OMITTED_MARKER);
// The whole batch converted (3 model messages, none dropped).
expect(out).toHaveLength(3);
});
it('a fully-poisoned row (no readable text) still degrades to just the marker', async () => {
const history: Row[] = [
{
id: 'a1',
role: 'assistant',
parts: [null] as unknown as UIMessage['parts'],
},
];
const out = await convertHistoryResilient(history, () => undefined);
expect(out).toHaveLength(1);
expect(JSON.stringify(out)).toContain(TOOL_CONTEXT_OMITTED_MARKER);
});
});
@@ -101,22 +101,6 @@ const INTERRUPT_NOTE =
'assume your previous response was complete, and do not silently restart the ' +
'partial work — build on it or follow the new instruction.';
/**
* #487: injected on a turn started by SUPERSEDING a previous run (the user hit
* "interrupt and send now" while a run was live). The previous run was Stopped,
* but there is NO side-effect quiescence a write it had already committed, or
* one committing at the moment of Stop, may land with a small delay AFTER this new
* run starts. So the model is told its picture of the page/state may be a beat
* stale and to re-read before assuming an edit did or did not apply.
*/
const SUPERSEDE_NOTE =
'NOTE: A previous agent run in this conversation was just interrupted so this ' +
'new turn could start. That run was stopped, but any operation it had already ' +
'begun (e.g. a page edit) may still be applied with a short delay. Do not ' +
'assume the document/state is exactly as the interrupted run left it — if you ' +
'need to rely on the current content, RE-READ it with the page tools before ' +
'acting rather than trusting a cached view.';
/**
* Injected on a turn where the open page was hand-edited by the user (or anyone
* else) AFTER the agent's previous response ended (#274). The server takes a
@@ -219,14 +203,6 @@ export interface BuildSystemPromptInput {
* (partial) answer was cut off by the user's new message.
*/
interrupted?: boolean;
/**
* #487: true when THIS turn was started by superseding a still-live previous run
* ("interrupt and send now"). Adds SUPERSEDE_NOTE so the model knows the previous
* run's last operations may still be applying and to re-read state it depends on.
* Distinct from `interrupted` (which is about a PARTIAL prior answer in history);
* both can be set together. Self-clears set only for the superseding turn.
*/
superseded?: boolean;
/**
* Set only when the open page was edited by the user AFTER the agent's previous
* turn ended (#274), confirmed server-side by diffing the current page against
@@ -335,7 +311,6 @@ export function buildSystemPrompt({
openedPage,
mcpInstructions,
interrupted,
superseded,
pageChanged,
deferredToolsEnabled,
toolCatalog,
@@ -385,13 +360,6 @@ export function buildSystemPrompt({
context += `\n${INTERRUPT_NOTE}`;
}
// Supersede note (#487): present only for a turn that stopped and replaced a
// still-live previous run — warns the model the previous run's last operations
// may still be applying (no side-effect quiescence).
if (superseded) {
context += `\n${SUPERSEDE_NOTE}`;
}
// Per-turn page-change note (#274). Added to the context section (inside the
// safety sandwich), present only when the server detected that the open page
// was edited by the user since the agent's last turn ended. The diff content is
@@ -89,22 +89,11 @@ describe('AiChatService.stream run-lifecycle safety net (#184)', () => {
const runRepo = {
insert: jest.fn().mockResolvedValue({ id: 'run-1', status: 'running' }),
update: jest.fn().mockResolvedValue({ id: 'run-1' }),
// #487: the terminal settle now goes through the CONDITIONAL write.
finalizeIfActive: jest
.fn()
.mockResolvedValue({ id: 'run-1', status: 'failed' }),
findById: jest.fn().mockResolvedValue(undefined),
};
const runService = new AiChatRunService(runRepo as never, { isCloud: () => false } as never);
// The user-message insert throws. #489 runs the history load + convert BEFORE
// the insert (convert-before-insert, so a retry cannot duplicate the user row),
// so `findAllByChat` (a real repo method) is now called first — stub it to an
// empty history so the flow reaches the insert. Both awaits are AFTER beginRun,
// so the "exception after beginRun -> settled to error" invariant is unchanged;
// the throw point simply moved from insert to a later insert after a no-op load.
// The user-message insert (the first bare await after beginRun) throws.
const aiChatMessageRepo = {
findAllByChat: jest.fn().mockResolvedValue([]),
insert: jest.fn().mockRejectedValue(new Error('insert boom')),
};
const aiChatRepo = {
@@ -159,10 +148,9 @@ describe('AiChatService.stream run-lifecycle safety net (#184)', () => {
// The run was begun...
expect(runRepo.insert).toHaveBeenCalledTimes(1);
// ...then settled to a terminal FAILED status by the safety net (via the
// #487 conditional write)...
expect(runRepo.finalizeIfActive).toHaveBeenCalledTimes(1);
expect(runRepo.finalizeIfActive).toHaveBeenCalledWith(
// ...then settled to a terminal FAILED status by the safety net...
expect(runRepo.update).toHaveBeenCalledTimes(1);
expect(runRepo.update).toHaveBeenCalledWith(
'run-1',
'ws1',
expect.objectContaining({ status: 'failed' }),
@@ -155,8 +155,6 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -181,7 +179,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false, isAiChatFinalStepLockdownEnabled: () => false } as never, // environment
);
return { svc, aiChatMessageRepo };
return { svc };
}
const body = {
@@ -287,7 +285,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
// Drive stream() to the point streamText is called, capturing the options object
// (which carries onStepFinish/onFinish/onError/onAbort) and the run hooks.
async function captureStreamCallbacks() {
const { svc, aiChatMessageRepo } = makeService();
const { svc } = makeService();
let capturedOpts: any;
streamTextMock.mockImplementation((opts: any) => {
capturedOpts = opts;
@@ -314,7 +312,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
runHooks: runHooks as never,
});
expect(capturedOpts).toBeDefined();
return { capturedOpts, runHooks, aiChatMessageRepo };
return { capturedOpts, runHooks };
}
it('F9: onStepFinish bumps the run step count, onFinish settles the run "completed" (the dominant autonomous-run path)', async () => {
@@ -334,13 +332,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
usage: {},
steps: [],
});
// #487: onFinish passes the (undefined) error slot so a message-finalize
// failure could error-mark the run; on the success path it is undefined.
expect(runHooks.onSettled).toHaveBeenCalledWith(
'run-1',
'completed',
undefined,
);
expect(runHooks.onSettled).toHaveBeenCalledWith('run-1', 'completed');
});
it('F9: onAbort settles the run "aborted"', async () => {
@@ -369,51 +361,6 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
expect.stringContaining('provider exploded'),
);
});
// #490 reactive branch: a provider CONTEXT-OVERFLOW 400 in onError is classified,
// records a distinguishable cause, and stamps metadata.replayOverflow so the NEXT
// turn's budgeter trims aggressively (the recovery that un-bricks the chat).
it('#490: a context-overflow 400 stamps replayOverflow on the finalized row', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as never);
const { capturedOpts, aiChatMessageRepo } = await captureStreamCallbacks();
const overflow = Object.assign(new Error('too large'), {
statusCode: 400,
message:
"This model's maximum context length is 128000 tokens. However, your messages resulted in 214000 tokens. Please reduce the length.",
});
await capturedOpts.onError({ error: overflow });
// The seed row exists (finalizeOwner is the owner-write path).
expect(aiChatMessageRepo.finalizeOwner).toHaveBeenCalled();
const calls = aiChatMessageRepo.finalizeOwner.mock.calls as any[][];
const patch = calls[calls.length - 1][2] as {
status: string;
metadata: Record<string, unknown>;
};
expect(patch.status).toBe('error');
expect(patch.metadata.replayOverflow).toBe(true);
expect(patch.metadata.error).toContain('контекстное окно');
});
it('#490: a non-overflow error does NOT stamp replayOverflow', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
const { capturedOpts, aiChatMessageRepo } = await captureStreamCallbacks();
await capturedOpts.onError({ error: new Error('network reset') });
const calls = aiChatMessageRepo.finalizeOwner.mock.calls as any[][];
const patch = calls[calls.length - 1][2] as {
status: string;
metadata: Record<string, unknown>;
};
expect('replayOverflow' in patch.metadata).toBe(false);
});
});
/**
@@ -468,8 +415,6 @@ describe('AiChatService.stream — begin-failure fails the turn (#184 F14 / #486
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -13,7 +13,6 @@ import {
compactToolOutput,
assistantParts,
serializeSteps,
type StepPartsCache,
rowToUiMessage,
prepareAgentStep,
stepBudgetWarning,
@@ -29,14 +28,10 @@ import {
FINAL_STEP_NUDGE,
STEP_LIMIT_NO_ANSWER_MARKER,
OUTPUT_DEGENERATION_ERROR,
lastAssistantContextTokens,
lastAssistantReplayOverflow,
seedActivatedTools,
} from './ai-chat.service';
import type { AiChatMessage, Workspace } from '@docmost/db/types/entity.types';
import { buildSystemPrompt } from './ai-chat.prompt';
import type { McpClientsService } from './external-mcp/mcp-clients.service';
import { resolveEffectiveReplayThreshold } from './history-budget';
/**
* Unit tests for compactToolOutput: the pure helper that shrinks tool outputs
@@ -119,54 +114,6 @@ describe('compactToolOutput', () => {
describe('assistantParts', () => {
type AnyPart = Record<string, unknown>;
// #490 memoization: assistantParts builds each step's parts once and caches
// them by the step OBJECT's identity, so a mid-stream flush does not
// re-stringify every prior step's (large) output. Observable property: with a
// shared cache, the second call over the SAME step object returns the cached
// (identical) part array even if the step's underlying output was swapped —
// proving the work was memoized, not redone.
it('memoizes a step by identity (shared cache => one build per step)', () => {
const cache: StepPartsCache = new WeakMap();
const step = {
text: 'x',
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: {} }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage', output: { v: 1 } }],
};
const first = assistantParts([step], '', cache) as AnyPart[];
expect((first.find((p) => p.type === 'tool-getPage')!.output as any).v).toBe(
1,
);
// Swap the output for a NEW value; a re-build would pick it up, a cache hit
// keeps the first result.
step.toolResults[0] = {
toolCallId: 'c1',
toolName: 'getPage',
output: { v: 2 },
};
const second = assistantParts([step], '', cache) as AnyPart[];
expect((second.find((p) => p.type === 'tool-getPage')!.output as any).v).toBe(
1,
);
// Same cached part objects are reused.
expect(second.find((p) => p.type === 'tool-getPage')).toBe(
first.find((p) => p.type === 'tool-getPage'),
);
});
it('without a cache, each call rebuilds (no stale memo)', () => {
const step = {
text: 'x',
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: {} }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage', output: { v: 1 } }],
};
const first = assistantParts([step], '') as AnyPart[];
step.toolResults[0].output = { v: 2 };
const second = assistantParts([step], '') as AnyPart[];
expect((second.find((p) => p.type === 'tool-getPage')!.output as any).v).toBe(
2,
);
});
it('emits output-available for a tool-call WITH a paired result', () => {
const steps = [
{
@@ -284,320 +231,61 @@ describe('assistantParts', () => {
});
});
// #490 trace format v2: per call the trace stores { input } for the call and an
// OUTCOME element — { ok: true } on success, { error, kind: 'thrown' } on a
// thrown tool-error, { error, kind: 'interrupted' } on a mid-step abort. The tool
// OUTPUT is no longer duplicated here (it lives once in metadata.parts).
describe('serializeSteps (trace v2)', () => {
describe('serializeSteps', () => {
it('returns null when there are no calls or results', () => {
expect(serializeSteps([])).toBeNull();
});
it('pairs a successful call with an { ok: true } outcome and NO output', () => {
it('flattens calls and results into a compact trace', () => {
const trace = serializeSteps([
{
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: { id: 'p1' } }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage' }],
toolCalls: [{ toolName: 'getPage', input: { id: 'p1' } }],
toolResults: [{ toolName: 'getPage', output: { title: 'T' } }],
},
]) as Array<Record<string, unknown>>;
expect(trace).toHaveLength(2);
expect(trace[0]).toEqual({ toolName: 'getPage', input: { id: 'p1' } });
expect(trace[1]).toEqual({ toolName: 'getPage', ok: true });
// The output is NOT stored in the trace any more (dedup: it lives in parts).
expect(trace.some((e) => 'output' in e)).toBe(false);
expect(trace[1]).toEqual({ toolName: 'getPage', output: { title: 'T' } });
});
it('records a THROWN failure with { error, kind: "thrown" }', () => {
it('records a THROWN tool failure (tool-error part) with its error message', () => {
const trace = serializeSteps([
{
toolCalls: [
{ toolCallId: 'c1', toolName: 'editPageText', input: { id: 'p1' } },
],
toolCalls: [{ toolName: 'editPageText', input: { id: 'p1' } }],
toolResults: [],
content: [
{
type: 'tool-error',
toolCallId: 'c1',
toolName: 'editPageText',
error: new Error('page is locked'),
},
],
},
]) as Array<Record<string, unknown>>;
// The call element is followed by a paired error element (mirroring how a
// successful result is appended), so the failure survives in the trace.
expect(trace).toHaveLength(2);
expect(trace[0]).toEqual({ toolName: 'editPageText', input: { id: 'p1' } });
expect(trace[1]).toEqual({
toolName: 'editPageText',
error: 'page is locked',
kind: 'thrown',
});
});
it('marks an interrupted call (no result, no throw) with kind "interrupted"', () => {
const trace = serializeSteps([
{
toolCalls: [
{ toolCallId: 'c1', toolName: 'createComment', input: { x: 1 } },
],
toolResults: [],
content: [],
},
]) as Array<Record<string, unknown>>;
expect(trace).toHaveLength(2);
expect(trace[1]).toEqual({
toolName: 'createComment',
error: 'Tool call did not complete.',
kind: 'interrupted',
});
// Structurally distinct from a thrown hard-fail so it never inflates an
// error-rate scan.
expect((trace[1] as { kind: string }).kind).not.toBe('thrown');
});
it('truncates a very long thrown-error message to the tool-output limit', () => {
it('truncates a very long tool-error message to the tool-output limit', () => {
const long = 'x'.repeat(5000);
const trace = serializeSteps([
{
toolCalls: [{ toolCallId: 'c1', toolName: 'editPageText', input: {} }],
toolCalls: [{ toolName: 'editPageText', input: {} }],
toolResults: [],
content: [
{
type: 'tool-error',
toolCallId: 'c1',
toolName: 'editPageText',
error: long,
},
],
content: [{ type: 'tool-error', toolName: 'editPageText', error: long }],
},
]) as Array<Record<string, unknown>>;
const errorText = trace[1].error as string;
// Truncated (not the full 5000 chars) and carries the omission marker.
expect(errorText.length).toBeLessThan(long.length);
expect(errorText).toContain('chars omitted');
});
it('pairs parallel calls in one step with their outcomes by id', () => {
const trace = serializeSteps([
{
toolCalls: [
{ toolCallId: 'a', toolName: 'getPage', input: {} },
{ toolCallId: 'b', toolName: 'searchPages', input: {} },
],
toolResults: [{ toolCallId: 'b', toolName: 'searchPages' }],
content: [
{ type: 'tool-error', toolCallId: 'a', toolName: 'getPage', error: 'nope' },
],
},
]) as Array<Record<string, unknown>>;
// call a, outcome a (thrown), call b, outcome b (ok)
expect(trace).toHaveLength(4);
expect(trace[1]).toEqual({ toolName: 'getPage', error: 'nope', kind: 'thrown' });
expect(trace[3]).toEqual({ toolName: 'searchPages', ok: true });
});
});
// #490: every assistant row flushAssistant writes carries the v2 era marker so a
// dual-shape diagnostic query can branch on the trace shape without inspecting it.
describe('toolTraceVersion era marker (#490)', () => {
it('stamps metadata.toolTraceVersion = 2 on every flushed row', () => {
const seed = flushAssistant([], '', 'streaming');
expect(seed.metadata.toolTraceVersion).toBe(2);
const done = flushAssistant(
[
{
text: 'ok',
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: {} }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage' }],
},
],
'',
'completed',
{ finishReason: 'stop' },
);
expect(done.metadata.toolTraceVersion).toBe(2);
});
});
// #490 replay-budget signal helpers over persisted history.
describe('lastAssistantContextTokens', () => {
const row = (
role: string,
metadata: Record<string, unknown> | null,
): AiChatMessage => ({ role, metadata }) as unknown as AiChatMessage;
it('reads the most recent assistant turn contextTokens (provider fact)', () => {
const hist = [
row('user', null),
row('assistant', { contextTokens: 12000 }),
row('user', null),
row('assistant', { contextTokens: 41000 }),
];
expect(lastAssistantContextTokens(hist)).toBe(41000);
});
it('returns undefined when the last assistant turn recorded no usage', () => {
const hist = [row('assistant', { error: 'boom' }), row('user', null)];
expect(lastAssistantContextTokens(hist)).toBeUndefined();
expect(lastAssistantContextTokens([])).toBeUndefined();
});
});
// #490 snapshotOpenPage fast-path: skip the full Markdown export + upsert when a
// snapshot already exists at the page's CURRENT version (same updated_at instant).
describe('snapshotOpenPage fast-path (#490)', () => {
function makeSvc(existingSnapshot: unknown, pageUpdatedAt: Date) {
const exportPageMarkdown = jest.fn(async () => '# md');
const upsert = jest.fn(async () => undefined);
const findByChatPage = jest.fn(async () => existingSnapshot);
const pageRepo = {
findById: jest.fn(async () => ({
id: 'p1',
workspaceId: 'ws1',
updatedAt: pageUpdatedAt,
})),
};
const svc = new AiChatService(
{} as never, // ai
{} as never, // aiChatRepo
{} as never, // aiChatMessageRepo
{ findByChatPage, upsert } as never, // aiChatPageSnapshotRepo
{} as never, // aiSettings
{ exportPageMarkdown } as never, // tools
{} as never, // mcpClients
{} as never, // aiAgentRoleRepo
pageRepo as never, // pageRepo
{} as never, // pageAccess
{} as never, // environment
);
return { svc, exportPageMarkdown, upsert, findByChatPage };
}
const args = () =>
[
'chat1',
'p1',
{ id: 'ws1' } as never,
{ id: 'u1' } as never,
'sess',
] as const;
it('skips export + upsert when the snapshot is already at this page version', async () => {
const t = new Date('2026-07-07T10:00:00Z');
const { svc, exportPageMarkdown, upsert } = makeSvc(
{ pageUpdatedAt: t, contentMd: '# md' },
t,
);
await (svc as unknown as { snapshotOpenPage: (...a: unknown[]) => Promise<void> })
.snapshotOpenPage(...args());
expect(exportPageMarkdown).not.toHaveBeenCalled();
expect(upsert).not.toHaveBeenCalled();
});
it('exports + upserts when the page advanced since the snapshot', async () => {
const { svc, exportPageMarkdown, upsert } = makeSvc(
{ pageUpdatedAt: new Date('2026-07-07T10:00:00Z'), contentMd: 'old' },
new Date('2026-07-07T11:00:00Z'),
);
await (svc as unknown as { snapshotOpenPage: (...a: unknown[]) => Promise<void> })
.snapshotOpenPage(...args());
expect(exportPageMarkdown).toHaveBeenCalledTimes(1);
expect(upsert).toHaveBeenCalledTimes(1);
});
it('seeds (exports + upserts) on the first turn (no snapshot yet)', async () => {
const { svc, exportPageMarkdown, upsert } = makeSvc(
undefined,
new Date('2026-07-07T10:00:00Z'),
);
await (svc as unknown as { snapshotOpenPage: (...a: unknown[]) => Promise<void> })
.snapshotOpenPage(...args());
expect(exportPageMarkdown).toHaveBeenCalledTimes(1);
expect(upsert).toHaveBeenCalledTimes(1);
});
});
// #490 deferred-tool activation persisted across turns.
describe('seedActivatedTools', () => {
const valid = new Set(['Search_web', 'getPageJson', 'diffPageVersions']);
it('seeds from persisted metadata, intersected with current valid names', () => {
expect(
seedActivatedTools(
{ activatedTools: ['Search_web', 'getPageJson'] },
valid,
),
).toEqual(['Search_web', 'getPageJson']);
});
it('drops a stored tool that is no longer valid (allowlist/role changed)', () => {
// 'Habr_publish' was activated before but is not in the current allowlist.
expect(
seedActivatedTools({ activatedTools: ['Search_web', 'Habr_publish'] }, valid),
).toEqual(['Search_web']);
});
it('is empty/robust for missing, non-array, or unknown-shaped metadata', () => {
expect(seedActivatedTools(undefined, valid)).toEqual([]);
expect(seedActivatedTools({}, valid)).toEqual([]);
expect(seedActivatedTools({ activatedTools: 'nope' }, valid)).toEqual([]);
expect(
seedActivatedTools({ activatedTools: [1, 'getPageJson', null] }, valid),
).toEqual(['getPageJson']);
});
it('de-duplicates stored names', () => {
expect(
seedActivatedTools(
{ activatedTools: ['getPageJson', 'getPageJson'] },
valid,
),
).toEqual(['getPageJson']);
});
});
describe('lastAssistantReplayOverflow', () => {
const row = (
role: string,
metadata: Record<string, unknown> | null,
): AiChatMessage => ({ role, metadata }) as unknown as AiChatMessage;
it('is true only when the LAST assistant turn overflowed', () => {
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
row('user', null),
]),
).toBe(true);
// A recovered (later, non-overflow) assistant turn clears it.
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
row('user', null),
row('assistant', { contextTokens: 5 }),
]),
).toBe(false);
expect(lastAssistantReplayOverflow([])).toBe(false);
});
// #490 reactive recovery: a prior turn stamped `replayOverflow` must make the
// NEXT turn's effective budget the AGGRESSIVE 0.5x cut — that harder trim is
// what un-bricks a chat that just 400'd on the context window. This exercises
// the exact wiring the service uses: read the stamp, then scale the threshold.
it('#490: a prior replayOverflow drives the next turn to the 0.5x aggressive budget', () => {
const history = [
row('assistant', { replayOverflow: true }),
row('user', null),
];
const priorOverflowed = lastAssistantReplayOverflow(history);
expect(priorOverflowed).toBe(true);
// Base budget 100k -> aggressive recovery halves it to 50k this turn.
expect(resolveEffectiveReplayThreshold(100_000, priorOverflowed)).toBe(50_000);
// Odd base floors, not rounds.
expect(resolveEffectiveReplayThreshold(99_999, true)).toBe(49_999);
// No prior overflow -> the base budget is used verbatim (no aggressive cut).
expect(resolveEffectiveReplayThreshold(100_000, false)).toBe(100_000);
// An explicit off-switch (null) is never overridden, even on recovery.
expect(resolveEffectiveReplayThreshold(null, true)).toBeNull();
});
});
describe('rowToUiMessage', () => {
@@ -930,23 +618,6 @@ describe('flushAssistant', () => {
expect(flushed.metadata.error).toBe('boom');
});
// #490 observability: the replay budgeter's decision is stamped on the turn.
it('records replayTrimmedToTokens + replayOverflow when provided', () => {
const f = flushAssistant([], '', 'error', {
error: 'ctx',
replayTrimmedToTokens: 42_000,
replayOverflow: true,
});
expect(f.metadata.replayTrimmedToTokens).toBe(42_000);
expect(f.metadata.replayOverflow).toBe(true);
});
it('omits the replay metadata when not provided', () => {
const f = flushAssistant([], '', 'completed', { finishReason: 'stop' });
expect('replayTrimmedToTokens' in f.metadata).toBe(false);
expect('replayOverflow' in f.metadata).toBe(false);
});
// #274 observability: the page-change diff the agent saw this turn is persisted
// to metadata.pageChanged when a non-empty diff was injected, and omitted when
// the diff is empty/whitespace or the arg is not supplied.
@@ -1644,12 +1315,8 @@ describe('AiChatService page-change lifecycle (#274)', () => {
describe('isInterruptResume', () => {
// history tail is the just-inserted user row; [len-2] is the previous turn.
const withPrev = (
prev: {
role: string;
status?: string | null;
metadata?: unknown;
} | null,
): Array<{ role: string; status?: string | null; metadata?: unknown }> =>
prev: { role: string; status?: string | null } | null,
): Array<{ role: string; status?: string | null }> =>
prev
? [prev, { role: 'user', status: null }]
: [{ role: 'user', status: null }];
@@ -1690,33 +1357,6 @@ describe('isInterruptResume', () => {
it('false when there is no preceding turn (only the new user row)', () => {
expect(isInterruptResume(withPrev(null), true)).toBe(false);
});
it('#487 EXCLUDES a reconcile stamp (finalizeFailed) — not a genuine interruption', () => {
// A row a reconcile settled to 'aborted' carries metadata.finalizeFailed. It
// must NOT be treated as an interrupt-resume (that would inject a false
// "you were interrupted" note), even though its status is 'aborted'.
expect(
isInterruptResume(
withPrev({
role: 'assistant',
status: 'aborted',
metadata: { finalizeFailed: true },
}),
true,
),
).toBe(false);
// A genuine abort (no finalizeFailed) still counts.
expect(
isInterruptResume(
withPrev({
role: 'assistant',
status: 'aborted',
metadata: { parts: [] },
}),
true,
),
).toBe(true);
});
});
/**
@@ -1769,7 +1409,7 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
}
// Wire only the deps reached on the way to the pipe call, plus a spy registry.
function makeService(opts: { resumable: boolean; history?: unknown[] }) {
function makeService(opts: { resumable: boolean }) {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
@@ -1777,11 +1417,8 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
const aiChatMessageRepo = {
// Both the user insert and the assistant seed return the same row id.
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => opts.history ?? []),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
// #487: the terminal owner-write + the opportunistic reconcile query.
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -1816,7 +1453,7 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
} as never,
streamRegistry as never,
);
return { svc, streamRegistry, aiChatMessageRepo };
return { svc, streamRegistry };
}
const body = {
@@ -1899,86 +1536,6 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
await expect(drive(svc, makeRunHooks())).rejects.toThrow('boom');
expect(streamRegistry.abortEntry).toHaveBeenCalledWith('chat-1', 'run-1');
});
// #489 REGRESSION (against the REAL convertToModelMessages — not mocked here):
// a persisted history row whose parts contain a `null` element makes the real
// convertToModelMessages THROW ("Cannot read properties of null"). Pre-fix that
// 500-ed every turn forever and each retry appended a duplicate user row. The
// fix converts BEFORE the insert and isolates the poisoned row per-row, degrading
// it to text with a "[tool context omitted]" marker. Assert the turn still runs,
// the marker reaches the model, and exactly ONE user row is inserted.
it('#489: a poisoned OLD-history row keeps the chat working; the marker reaches the model; one user insert', async () => {
const { svc, aiChatMessageRepo } = makeService({
resumable: false,
history: [
{
id: 'old-1',
role: 'assistant',
content: 'earlier answer',
// A null part is the poison: rowToUiMessage keeps it (the array is
// non-empty) and the real convertToModelMessages throws on it.
metadata: { parts: [{ type: 'text', text: 'earlier answer' }, null] },
status: 'completed',
},
],
});
// Must NOT throw — the poisoned row is degraded, not fatal.
await drive(svc, makeRunHooks());
expect(streamTextMock).toHaveBeenCalledTimes(1);
const passedMessages = streamTextMock.mock.calls[0][0].messages;
const serialized = JSON.stringify(passedMessages);
// The model sees the truncation marker (silent tool-context loss is not ok)
// AND the row's readable text is preserved alongside it.
expect(serialized).toContain('[tool context omitted]');
expect(serialized).toContain('earlier answer');
// Exactly ONE user row inserted (no duplicate), inserted AFTER conversion.
const userInserts = aiChatMessageRepo.insert.mock.calls
.map((c: unknown[]) => c[0] as { role?: string })
.filter((r) => r.role === 'user');
expect(userInserts).toHaveLength(1);
});
// #489: client-supplied non-text parts (a tool-part in `input-available`, the
// exact "bricking" payload) are dropped ON RECEIPT — never persisted — so they
// can never poison future turns. Only the text survives into metadata.parts.
it('#489: a non-text client part is stripped before persist (only text survives)', async () => {
const { svc, aiChatMessageRepo } = makeService({ resumable: false });
await svc.stream({
user: { id: 'u1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 's1',
body: {
chatId: 'chat-1',
messages: [
{
id: 'm1',
role: 'user',
parts: [
{ type: 'text', text: 'hello' },
// untrusted tool-part — must be dropped, never persisted
{
type: 'tool-getPage',
toolCallId: 't1',
state: 'input-available',
input: { pageId: 'p' },
},
],
},
],
} as never,
res: makeRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: makeRunHooks() as never,
});
const userInsert = aiChatMessageRepo.insert.mock.calls
.map((c: unknown[]) => c[0] as { role?: string; metadata?: unknown })
.find((r) => r.role === 'user');
const parts = (userInsert?.metadata as { parts?: Array<{ type: string }> })
?.parts;
expect(parts).toEqual([{ type: 'text', text: 'hello' }]);
});
});
/**
@@ -2066,19 +1623,6 @@ describe('AiChatService.stream — token-degeneration reaction (#444)', () => {
return { id };
},
),
// #487: the terminal owner-write records into the SAME `updated` recorder so
// assertions on the terminal 'completed'/'error'/'aborted' write still hold.
finalizeOwner: jest.fn(
async (
id: string,
workspaceId: string,
patch: Record<string, unknown>,
) => {
updated.push({ id, workspaceId, patch });
return { id };
},
),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -2338,148 +1882,3 @@ describe('AiChatService.stream — token-degeneration reaction (#444)', () => {
expect(patch.content).not.toContain(STEP_LIMIT_NO_ANSWER_MARKER);
});
});
// #487 F3 — the reconcile() / reconcileChat() ORCHESTRATORS. The individual
// clauses are exercised elsewhere; these pin the production orchestration the
// per-clause specs do not: the clause ORDER, the per-clause try/catch ISOLATION
// (one clause throwing must NOT abort the others), and reconcileChat() (which runs
// at the start of every turn and was entirely uncovered).
describe('AiChatService.reconcile / reconcileChat orchestrators (#487 F3)', () => {
let warnSpy: jest.SpyInstance;
beforeEach(() => {
// Silence the intentional clause-failure warnings (kept out of test output).
warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined);
});
afterEach(() => {
warnSpy.mockRestore();
});
function makeService(opts: {
messageRepo?: Record<string, jest.Mock>;
runService?: Record<string, jest.Mock>;
}) {
const aiChatMessageRepo = {
findStreamingWithTerminalRun: jest.fn(async () => []),
stampTerminalIfStreaming: jest.fn(async () => undefined),
sweepStreamingWithoutActiveRun: jest.fn(async () => 0),
...(opts.messageRepo ?? {}),
};
const aiChatRunService = opts.runService
? {
zombieRunIds: jest.fn(() => []),
settleZombie: jest.fn(async () => true),
reconcileStaleRuns: jest.fn(async () => 0),
...opts.runService,
}
: undefined;
const svc = new AiChatService(
{} as never, // ai
{} as never, // aiChatRepo
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
{} as never, // aiSettings
{} as never, // tools
{} as never, // mcpClients
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{} as never, // environment
{} as never, // streamRegistry
aiChatRunService as never, // aiChatRunService (#487)
);
return { svc, aiChatMessageRepo, aiChatRunService };
}
it('reconcile() fires all four clauses IN ORDER (a -> b -> c -> d)', async () => {
const order: string[] = [];
const { svc } = makeService({
messageRepo: {
findStreamingWithTerminalRun: jest.fn(async () => {
order.push('b:find');
return [
{ messageId: 'm1', workspaceId: 'ws1', runStatus: 'succeeded' },
];
}),
stampTerminalIfStreaming: jest.fn(async () => {
order.push('b:stamp');
}),
sweepStreamingWithoutActiveRun: jest.fn(async () => {
order.push('d');
return 0;
}),
},
runService: {
zombieRunIds: jest.fn(() => ['z1']),
settleZombie: jest.fn(async () => {
order.push('a');
return true;
}),
reconcileStaleRuns: jest.fn(async () => {
order.push('c');
return 0;
}),
},
});
await svc.reconcile();
expect(order).toEqual(['a', 'b:find', 'b:stamp', 'c', 'd']);
});
it('a clause that THROWS does not abort the remaining clauses (per-clause try/catch isolation)', async () => {
const { svc, aiChatMessageRepo, aiChatRunService } = makeService({
messageRepo: {
// Clause (b) blows up mid-reconcile.
findStreamingWithTerminalRun: jest.fn(async () => {
throw new Error('clause b DB blip');
}),
},
runService: {
zombieRunIds: jest.fn(() => ['z1']),
},
});
// reconcile() must SETTLE (the clause-b failure is swallowed), not reject.
await expect(svc.reconcile()).resolves.toBeUndefined();
// (a) ran before (b); crucially (c) and (d) STILL ran despite (b) throwing —
// the property a missing try/catch would break. MUTATION-VERIFY: drop clause
// (b)'s try/catch and this reddens (the throw propagates, skipping c + d).
expect(aiChatRunService!.settleZombie).toHaveBeenCalled(); // (a)
expect(aiChatRunService!.reconcileStaleRuns).toHaveBeenCalled(); // (c)
expect(
aiChatMessageRepo.sweepStreamingWithoutActiveRun,
).toHaveBeenCalled(); // (d)
});
it('reconcileChat() settles THIS chat\'s stuck streaming rows by their run status', async () => {
const { svc, aiChatMessageRepo } = makeService({
messageRepo: {
findStreamingWithTerminalRun: jest.fn(async () => [
{ messageId: 'm1', workspaceId: 'ws1', runStatus: 'failed' },
{ messageId: 'm2', workspaceId: 'ws1', runStatus: 'succeeded' },
]),
},
});
await svc.reconcileChat('chat-1', 'ws1');
// Scoped to THIS chat and bounded at 50 (the user-facing opportunistic path).
expect(
aiChatMessageRepo.findStreamingWithTerminalRun,
).toHaveBeenCalledWith(50, { chatId: 'chat-1', workspaceId: 'ws1' });
// failed-run -> 'error'; every other terminal status -> 'aborted'.
expect(aiChatMessageRepo.stampTerminalIfStreaming).toHaveBeenCalledWith(
'm1',
'ws1',
'error',
);
expect(aiChatMessageRepo.stampTerminalIfStreaming).toHaveBeenCalledWith(
'm2',
'ws1',
'aborted',
);
});
});
File diff suppressed because it is too large Load Diff
@@ -1,209 +0,0 @@
import { randomBytes } from 'crypto';
import { Client } from 'pg';
import { flushAssistant, serializeSteps } from './ai-chat.service';
/**
* #490 write-volume regression an OBSERVABLE-PROPERTY test on a LIVE Postgres,
* not "bytes through a mock repo" (a mock measures exactly the thing that does not
* hurt). It drives a realistic 50-step run where each step returns a ~100 KB tool
* output and, at every `onStepFinish`, UPDATEs the assistant row the way the
* service does then reads the REAL write volume via the `pg_current_wal_lsn()`
* delta around the run.
*
* The property proven: v2 stores each tool OUTPUT only in `metadata.parts`, no
* longer ALSO in the `tool_calls` trace. So:
* 1. the trace (`tool_calls`) column's write volume is now O(Σ steps) tiny,
* linear outcome flags vs the pre-#490 O(N²) that re-persisted every prior
* output on every step; and
* 2. the FULL-row write volume drops sharply (the duplicated output copy is gone).
*
* Connects to the local gitmost test Postgres (docker `gitmost-test-pg` on :5432);
* SKIPS cleanly when that DB is not reachable so it never breaks a DB-less CI.
*/
const CONN =
process.env.WAL_TEST_DATABASE_URL ??
'postgresql://docmost:docmost_dev_pw@localhost:5432/docmost';
// A step whose tool output is ~100 KB (a page read), in the SDK StepLike shape.
// The body is INCOMPRESSIBLE random text — a `'x'.repeat()` filler would TOAST-
// compress to nothing and hide the real write volume (a page body does not).
function makeStep(i: number, outputBytes = 100_000) {
const body = randomBytes(Math.ceil(outputBytes * 0.75)).toString('base64');
return {
text: `step ${i} reasoning`,
toolCalls: [{ toolCallId: `c${i}`, toolName: 'getPage', input: { id: `p${i}` } }],
toolResults: [
{
toolCallId: `c${i}`,
toolName: 'getPage',
output: { id: `p${i}`, title: `Page ${i}`, body },
},
],
};
}
// The pre-#490 (v1) trace: outputs stored a SECOND time in `tool_calls`
// (the duplication #490 removed). Mirrors the OLD serializeSteps shape.
function v1Trace(steps: ReturnType<typeof makeStep>[]): unknown {
const calls: unknown[] = [];
for (const s of steps) {
for (const c of s.toolCalls) calls.push({ toolName: c.toolName, input: c.input });
for (const r of s.toolResults)
calls.push({ toolName: r.toolName, output: r.output });
}
return calls;
}
async function walDelta(
client: Client,
fn: () => Promise<void>,
): Promise<number> {
const before = (await client.query('SELECT pg_current_wal_lsn() AS l')).rows[0]
.l as string;
await fn();
// NOTE: do NOT pg_switch_wal() here — a segment switch pads the LSN to the next
// 16 MB boundary and would swamp the actual write delta. The raw LSN advances by
// the bytes of WAL emitted, which is exactly what we want to measure.
const after = (await client.query('SELECT pg_current_wal_lsn() AS l')).rows[0]
.l as string;
return Number(
(await client.query('SELECT pg_wal_lsn_diff($1,$2) AS d', [after, before]))
.rows[0].d,
);
}
describe('#490 write-volume on a live Postgres (pg_current_wal_lsn delta)', () => {
let client: Client | undefined;
let available = false;
beforeAll(async () => {
try {
client = new Client(CONN);
await client.connect();
await client.query('SELECT pg_current_wal_lsn()');
available = true;
} catch {
available = false;
client = undefined;
}
});
afterAll(async () => {
await client?.end().catch(() => undefined);
});
const STEPS = 50;
it('v2 trace write volume is O(Σ steps) — a tiny fraction of the v1 duplicate', async () => {
if (!available || !client) {
console.warn('SKIP: gitmost-test-pg not reachable; skipping WAL test.');
return;
}
const c = client;
// Isolated table so we measure only the tool_calls (trace) column's writes.
await c.query('DROP TABLE IF EXISTS _wal_trace');
await c.query('CREATE TABLE _wal_trace(id int primary key, tool_calls jsonb)');
await c.query("INSERT INTO _wal_trace VALUES (1, '[]'::jsonb)");
const steps: ReturnType<typeof makeStep>[] = [];
// v1: each step re-persists ALL prior outputs into the trace (the O(N²) churn).
const v1 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
await c.query('UPDATE _wal_trace SET tool_calls=$1 WHERE id=1', [
JSON.stringify(v1Trace(acc)),
]);
}
steps.push(...acc);
});
await c.query("UPDATE _wal_trace SET tool_calls='[]'::jsonb WHERE id=1");
// v2: the REAL serializeSteps — outcome flags only, NO outputs.
const v2 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
await c.query('UPDATE _wal_trace SET tool_calls=$1 WHERE id=1', [
JSON.stringify(serializeSteps(acc)),
]);
}
});
await c.query('DROP TABLE IF EXISTS _wal_trace');
// eslint-disable-next-line no-console
console.log(
`[#490 WAL] trace column over ${STEPS} steps: v1=${(v1 / 1e6).toFixed(1)}MB ` +
`v2=${(v2 / 1e6).toFixed(2)}MB (${(v1 / v2).toFixed(0)}x smaller)`,
);
// The trace no longer carries outputs: v2 is a tiny fraction of v1's WAL.
expect(v2).toBeLessThan(v1 * 0.1);
// And v2's trace WAL is small in absolute terms — O(Σ steps) of flags, not
// O(N² × output). 50 steps of ~40-byte flags is well under a few MB of WAL.
expect(v2).toBeLessThan(5_000_000);
// v1's duplicate alone is huge (≈ the 100 KB output re-written N² times).
expect(v1).toBeGreaterThan(50_000_000);
}, 120_000);
it('the full assistant row write drops sharply once the duplicate is gone', async () => {
if (!available || !client) return;
const c = client;
await c.query('DROP TABLE IF EXISTS _wal_full');
await c.query(
'CREATE TABLE _wal_full(id int primary key, content text, tool_calls jsonb, metadata jsonb, status text)',
);
await c.query("INSERT INTO _wal_full VALUES (1, '', '[]'::jsonb, '{}'::jsonb, 'streaming')");
const writeRow = async (patch: {
content: string;
toolCalls: unknown;
metadata: unknown;
status: string;
}) =>
c.query(
'UPDATE _wal_full SET content=$1, tool_calls=$2, metadata=$3, status=$4 WHERE id=1',
[
patch.content,
JSON.stringify(patch.toolCalls ?? null),
JSON.stringify(patch.metadata),
patch.status,
],
);
// v2 (real flushAssistant): outputs live once, in metadata.parts.
const v2 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
await writeRow(flushAssistant(acc as never, '', 'streaming'));
}
});
await c.query("UPDATE _wal_full SET content='', tool_calls='[]'::jsonb, metadata='{}'::jsonb WHERE id=1");
// v1: same row PLUS the duplicated outputs in the trace column.
const v1 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
const f = flushAssistant(acc as never, '', 'streaming');
await writeRow({ ...f, toolCalls: v1Trace(acc) });
}
});
await c.query('DROP TABLE IF EXISTS _wal_full');
// eslint-disable-next-line no-console
console.log(
`[#490 WAL] full row over ${STEPS} steps: v1=${(v1 / 1e6).toFixed(1)}MB ` +
`v2=${(v2 / 1e6).toFixed(1)}MB (saved ${((1 - v2 / v1) * 100).toFixed(0)}%)`,
);
// Removing the duplicated trace copy is a large, real write-volume reduction.
expect(v2).toBeLessThan(v1 * 0.75);
}, 120_000);
});
@@ -1,261 +0,0 @@
import { errors } from 'undici';
import {
McpClientsService,
isRetryableConnectError,
} from './mcp-clients.service';
/**
* #489 external-MCP in-run transport recovery.
*
* The transport-error classification + retry gate are exercised against the REAL
* undici error CLASSES prod throws (`errors.SocketError` / `errors.BodyTimeoutError`,
* carrying the true `UND_ERR_*` codes and class names), wrapped EXACTLY as undici's
* `fetch` wraps them a `TypeError('fetch failed'|'terminated')` whose `.cause` is
* the undici error. These are the real classes, not hand-rolled `{code:'...'}`
* mocks: constructing the genuine class is what makes this a faithful test of the
* prod predicate (epic root-cause #4 a mock-shaped predicate would leave the
* evict/retry path silently dead in production while CI stays green). We construct
* rather than drive a live fetch because Jest's environment degrades the live-fetch
* error to a generic `Error` cause (no undici code), which would NOT be the prod
* shape.
*/
/** A REAL undici socket reset, wrapped as fetch wraps it. */
function realSocketResetError(): unknown {
const err = new TypeError('fetch failed');
(err as { cause?: unknown }).cause = new errors.SocketError('other side closed');
return err;
}
/** A REAL undici body timeout, wrapped as fetch wraps it. */
function realBodyTimeoutError(): unknown {
const err = new TypeError('terminated');
(err as { cause?: unknown }).cause = new errors.BodyTimeoutError();
return err;
}
type FakeServer = {
id: string;
name: string;
transport: 'http' | 'sse';
url: string;
headersEnc: string | null;
toolAllowlist: string[] | null;
instructions: string | null;
};
const server = (over: Partial<FakeServer> = {}): FakeServer => ({
id: 's1',
name: 'srv',
transport: 'http',
url: 'http://example.test/mcp',
headersEnc: null,
toolAllowlist: null,
instructions: null,
...over,
});
function buildService(servers: FakeServer[], trusted = false) {
const repo = { listEnabled: jest.fn().mockResolvedValue(servers) };
const service = new McpClientsService(repo as never, {} as never);
// Seed a DETERMINISTIC write-class map so the retry gate is controlled here
// (the production map loads from @docmost/mcp via a dynamic ESM import). getPage
// is a read, patchNode is a write — the real classifications.
(
service as unknown as { writeClassMapPromise: Promise<unknown> }
).writeClassMapPromise = Promise.resolve({
getPage: 'readOnly',
patchNode: 'write',
});
// The service only APPLIES that map to a TRUSTED internal Docmost server
// (isInternalDocmostServer, really false for every third-party row). A retry
// test needs a trusted server to exercise the readOnly-retry path at all, so it
// passes trusted=true to model a Docmost-origin server; the third-party
// double-apply test leaves it at the real value (false).
if (trusted) {
jest
.spyOn(
service as unknown as {
isInternalDocmostServer: (s: FakeServer) => boolean;
},
'isInternalDocmostServer',
)
.mockReturnValue(true);
}
return { service, repo };
}
/** Spy the private `connect` so each call yields a controlled fake client whose
* single tool's execute is the supplied function. Returns the connect spy. */
function stubConnect(
service: McpClientsService,
toolName: string,
execs: Array<(...a: unknown[]) => Promise<unknown>>,
) {
let n = 0;
return jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => Promise<unknown> },
'connect',
)
.mockImplementation(async () => {
const exec = execs[Math.min(n, execs.length - 1)];
n += 1;
return {
tools: async () => ({ [toolName]: { description: 'x', execute: exec } }),
close: jest.fn().mockResolvedValue(undefined),
};
});
}
const opts = (abortSignal?: AbortSignal) =>
({ toolCallId: 't', messages: [], abortSignal }) as never;
describe('isRetryableConnectError (#489, REAL error shapes)', () => {
it('classifies a real undici socket reset and body timeout as retryable', async () => {
const socketErr = await realSocketResetError();
const bodyErr = await realBodyTimeoutError();
expect(isRetryableConnectError(socketErr)).toBe(true);
expect(isRetryableConnectError(bodyErr)).toBe(true);
// Unwraps a wrapped cause chain (e.g. an MCPClientError around the socket err).
const wrapped = new Error('mcp call failed');
(wrapped as { cause?: unknown }).cause = socketErr;
expect(isRetryableConnectError(wrapped)).toBe(true);
});
it('does NOT classify an application-level error as a transport break', () => {
expect(isRetryableConnectError(new Error('validation failed'))).toBe(false);
expect(isRetryableConnectError({ name: 'HttpError', status: 400 })).toBe(false);
expect(isRetryableConnectError(undefined)).toBe(false);
expect(isRetryableConnectError('boom')).toBe(false);
});
});
describe('McpClientsService in-run transport recovery (#489)', () => {
afterEach(() => jest.restoreAllMocks());
it('a readOnly tool whose transport breaks reconnects and retries WITHIN the same run', async () => {
const realErr = await realSocketResetError();
const { service } = buildService([server()], true);
const first = jest.fn().mockRejectedValue(realErr);
const second = jest.fn().mockResolvedValue({ ok: true });
const connectSpy = stubConnect(service, 'getPage', [first, second]);
const toolset = await service.toolsFor('ws-1');
const tool = toolset.tools['srv_getPage'];
const result = await (tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
);
// The repeat call within the run got a LIVE client and succeeded.
expect(result).toEqual({ ok: true });
expect(first).toHaveBeenCalledTimes(1);
expect(second).toHaveBeenCalledTimes(1);
// Exactly one reconnect was minted (initial build connect + one recovery).
expect(connectSpy).toHaveBeenCalledTimes(2);
// The run accumulated BOTH leases (old + reconnected) — released together at end.
expect(toolset.clients).toHaveLength(2);
await Promise.all(toolset.clients.map((c) => c.close()));
});
it('a WRITE tool does NOT auto-retry on a transport error (indeterminate)', async () => {
const realErr = await realSocketResetError();
const { service } = buildService([server()], true);
const exec = jest.fn().mockRejectedValue(realErr);
const connectSpy = stubConnect(service, 'patchNode', [exec]);
const toolset = await service.toolsFor('ws-2');
const tool = toolset.tools['srv_patchNode'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
),
).rejects.toThrow(/MAY have already applied/);
// Called exactly once — NO blind retry (avoids double-apply, the #435 class).
expect(exec).toHaveBeenCalledTimes(1);
// No fresh connection was minted for a write.
expect(connectSpy).toHaveBeenCalledTimes(1);
await Promise.all(toolset.clients.map((c) => c.close()));
});
it('does NOT retry (or reconnect) after the run is aborted (Stop)', async () => {
const realErr = await realSocketResetError();
const { service } = buildService([server()], true);
const controller = new AbortController();
// The transport error arrives, but the run was Stopped in the same tick.
const first = jest.fn().mockImplementation(async () => {
controller.abort();
throw realErr;
});
const second = jest.fn().mockResolvedValue({ ok: true });
const connectSpy = stubConnect(service, 'getPage', [first, second]);
const toolset = await service.toolsFor('ws-3');
const tool = toolset.tools['srv_getPage'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(controller.signal),
),
).rejects.toBeDefined();
// getPage IS readOnly, but the Stop blocks the retry — no second call, no mint.
expect(second).not.toHaveBeenCalled();
expect(connectSpy).toHaveBeenCalledTimes(1);
await Promise.all(toolset.clients.map((c) => c.close()));
});
it('an app-level (non-transport) tool error is surfaced verbatim, never retried', async () => {
const { service } = buildService([server()], true);
const appErr = new Error('tool says: bad input');
const exec = jest.fn().mockRejectedValue(appErr);
const connectSpy = stubConnect(service, 'getPage', [exec]);
const toolset = await service.toolsFor('ws-4');
const tool = toolset.tools['srv_getPage'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
),
).rejects.toThrow('tool says: bad input');
expect(exec).toHaveBeenCalledTimes(1);
expect(connectSpy).toHaveBeenCalledTimes(1); // no reconnect for an app error
await Promise.all(toolset.clients.map((c) => c.close()));
});
// #489 (review, MEDIUM) — the Docmost write-class map keys by DOCMOST tool
// names; a THIRD-PARTY server may name a WRITE tool `getPage` (a Docmost read
// name). It must NOT inherit readOnly and must NOT auto-retry on a transport
// error — a blind retry of that write is a double-apply (the #435 class). Here
// the server is UNTRUSTED (buildService default, isInternalDocmostServer=false),
// so the map is not applied and `getPage` classifies as a write.
//
// MUTATION-VERIFY: forcing the server "trusted" (buildService(..., true)) makes
// `getPage` inherit readOnly -> it WOULD reconnect+retry (connect twice) and the
// assertions below fail — i.e. removing the trust scope re-opens the bug.
it('a THIRD-PARTY WRITE tool named like a Docmost read does NOT auto-retry (no double-apply)', async () => {
const realErr = await realSocketResetError();
// Untrusted: default trusted=false — a real third-party server.
const { service } = buildService([server()]);
const exec = jest.fn().mockRejectedValue(realErr);
const connectSpy = stubConnect(service, 'getPage', [exec, exec]);
const toolset = await service.toolsFor('ws-5');
const tool = toolset.tools['srv_getPage'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
),
).rejects.toThrow(/MAY have already applied/);
// Exactly one call, NO reconnect — the name collision granted no readOnly-retry.
expect(exec).toHaveBeenCalledTimes(1);
expect(connectSpy).toHaveBeenCalledTimes(1);
await Promise.all(toolset.clients.map((c) => c.close()));
});
});
@@ -106,11 +106,8 @@ describe('McpClientsService.decryptHeaders', () => {
describe('McpClientsService.guardedFetch (SSRF per-request guard)', () => {
// The bound guardedFetch closure lives on the instance as a private field.
// #489 split it into per-transport HTTP/SSE bindings (they differ only in the
// dispatcher's bodyTimeout); the SSRF guard is identical, so testing the HTTP
// one is sufficient.
const guardedFetchOf = (service: McpClientsService) =>
(service as unknown as { guardedFetchHttp: typeof fetch }).guardedFetchHttp;
(service as unknown as { guardedFetch: typeof fetch }).guardedFetch;
let fetchSpy: jest.SpiedFunction<typeof fetch>;
@@ -1,6 +1,5 @@
import { isIP } from 'node:net';
import { lookup as dnsLookup, type LookupAddress } from 'node:dns';
import { pathToFileURL } from 'node:url';
import { Injectable, Logger } from '@nestjs/common';
import { type Tool, type ToolCallOptions } from 'ai';
import { createMCPClient } from '@ai-sdk/mcp';
@@ -11,29 +10,9 @@ import {
streamingDispatcherOptions,
mcpStreamTimeoutMs,
mcpCallTimeoutMs,
mcpSseBodyTimeoutMs,
} from '../../../integrations/ai/ai-streaming-fetch';
import { SecretBoxService } from '../../../integrations/crypto/secret-box';
import { isUrlAllowed, isIpAllowed } from './ssrf-guard';
// TYPE-ONLY (erased at compile): @docmost/mcp is ESM-only and cannot be a runtime
// `require()` from this commonjs module (same constraint as docmost-client.loader).
// The write-class MAP is loaded lazily via the dynamic-import trick below.
import type { ToolWriteClass } from '@docmost/mcp';
// TS(commonjs) downlevels a literal `import()` to `require()`, which cannot load
// the ESM-only @docmost/mcp. Indirect through Function so the real dynamic
// `import()` survives compilation (same trick as docmost-client.loader.ts).
const esmImport = new Function(
'specifier',
'return import(specifier)',
) as (specifier: string) => Promise<unknown>;
/** Local read-only predicate avoids a value import of the ESM-only package.
* Only a pure read is retry-safe after a transport break (a write is
* indeterminate). Kept in lockstep with @docmost/mcp's isRetryableWriteClass. */
function isReadOnlyWriteClass(writeClass: ToolWriteClass | undefined): boolean {
return writeClass === 'readOnly';
}
/** A closable external MCP client handle. */
export interface Closable {
@@ -102,52 +81,12 @@ const MAX_TOOL_NAME_LENGTH = 64;
* close until the turn releases it, so a TTL expiry mid-turn never closes a
* client a stream is still executing against.
*/
/**
* Where a merged (namespaced) tool came from, so the per-run recovery wrapper
* (#489) can, on a transport error, reconnect THAT server and re-resolve the SAME
* underlying tool by its raw name. `writeClass` gates the single auto-retry (a
* read is retry-safe; a write is indeterminate). `serverIndex` indexes the
* entry's `servers` array (which server config to reconnect).
*/
interface ToolProvenance {
serverIndex: number;
rawName: string;
writeClass: ToolWriteClass | undefined;
}
/** A live reconnected server (its fresh client + raw call-timeout-wrapped tools). */
interface RecoveredServerState {
client: McpClient;
tools: Record<string, Tool>;
}
/**
* Per-run, per-server recovery binding (#489). `current` is the server's LIVE
* target for this run: `null` means "use the ORIGINAL cached client/template";
* a non-null value is a reconnected throwaway client all this server's tools now
* call. `reconnecting` dedupes concurrent reconnects so only ONE fresh client is
* minted per death (a losing concurrent call awaits it and retries on the SAME
* new client the CAS-by-identity rule).
*/
interface ServerBinding {
current: RecoveredServerState | null;
reconnecting?: Promise<RecoveredServerState>;
}
interface CacheEntry {
tools: Record<string, Tool>;
clients: McpClient[];
outcomes: ServerOutcome[];
/** Prompt guidance for qualifying servers (see McpServerInstruction). */
instructions: McpServerInstruction[];
/**
* The enabled server configs used to build this entry (#489), so the per-run
* recovery wrapper can reconnect a specific server by index. Parallel to the
* indices referenced by {@link toolMeta}.
*/
servers: AiMcpServer[];
/** merged-tool-key -> provenance (#489), for the per-run recovery wrapper. */
toolMeta: Record<string, ToolProvenance>;
expiresAt: number;
/** Active leases (turns currently using these clients). */
refCount: number;
@@ -181,82 +120,20 @@ export class McpClientsService {
*/
private readonly cache = new Map<string, Promise<CacheEntry>>();
/**
* SSRF-pinned dispatchers for outbound external-MCP fetches. Both use the SAME
* custom connect.lookup (so every connection is IP-validated), but carry a
* DIFFERENT `bodyTimeout` (#489): the HTTP (streamable) transport opens a fresh
* request per call, so it keeps the tight silence timeout; the SSE transport
* holds ONE long-lived body open across many calls, so a >1-min idle BETWEEN
* calls is LEGITIMATE and must not break the socket it gets a much larger
* bodyTimeout. (headersTimeout stays tight on both.)
* A single shared SSRF-pinned dispatcher for ALL outbound external-MCP fetches.
* Its custom connect.lookup runs per connection, so one instance safely guards
* every server's connections (we never connect to an unvalidated IP).
*/
private readonly dispatcherHttp: Dispatcher = buildPinnedDispatcher(
mcpStreamTimeoutMs(),
);
private readonly dispatcherSse: Dispatcher = buildPinnedDispatcher(
mcpSseBodyTimeoutMs(),
);
/** guardedFetch bound to each dispatcher; picked by transport type in connect(). */
private readonly guardedFetchHttp: typeof fetch = (input, init) =>
guardedFetch(this.dispatcherHttp, input, init);
private readonly guardedFetchSse: typeof fetch = (input, init) =>
guardedFetch(this.dispatcherSse, input, init);
/**
* Memoized write-class map (#489), loaded lazily from @docmost/mcp via the
* dynamic-import trick. Keyed by tool name (=== mcpName). A tool NOT in the map
* (any third-party external MCP tool) classifies as `undefined` -> treated as a
* write by the retry gate (the safe default: never blind-retry an unknown tool).
* On any load failure the map is `{}` (every tool -> no auto-retry), so a
* missing/older @docmost/mcp build only DISABLES retries, never mis-retries.
*/
private writeClassMapPromise: Promise<Record<string, ToolWriteClass>> | null =
null;
private readonly dispatcher: Dispatcher = buildPinnedDispatcher();
/** guardedFetch bound to the pinned dispatcher; reused by every transport. */
private readonly guardedFetch: typeof fetch = (input, init) =>
guardedFetch(this.dispatcher, input, init);
constructor(
private readonly repo: AiMcpServerRepo,
private readonly secretBox: SecretBoxService,
) {}
/**
* Whether an external MCP server is the TRUSTED internal Docmost MCP server
* the only server whose tools may be classified by the Docmost write-class map
* (#489 review). Today this is ALWAYS false: every `ai_mcp_servers` row is an
* admin-configured THIRD-PARTY endpoint (there is no builtin/self flag, sentinel
* URL, or synthetic server in this path Docmost's OWN tools are exposed via the
* separate in-app tools path, never through this external-MCP client). So no
* third-party tool can inherit `readOnly` by a name collision with a Docmost read
* tool, and none is ever auto-retried on a transport error (which would risk a
* double-apply the #435 class). Flip this (an explicit `kind`/`isBuiltin`
* column, or a configured self-MCP URL) if a trusted internal server is ever
* introduced. A method (not a free function) so it is a single, mockable seam.
*/
private isInternalDocmostServer(_server: AiMcpServer): boolean {
return false;
}
/** Lazily load + memoize the shared write-class map (see the field doc). */
private getWriteClassMap(): Promise<Record<string, ToolWriteClass>> {
if (!this.writeClassMapPromise) {
this.writeClassMapPromise = (async () => {
try {
const entry = require.resolve('@docmost/mcp');
const mod = (await esmImport(pathToFileURL(entry).href)) as {
SHARED_TOOL_WRITE_CLASS?: Record<string, ToolWriteClass>;
};
return mod.SHARED_TOOL_WRITE_CLASS ?? {};
} catch (err) {
this.logger.warn(
`Could not load MCP write-class map (auto-retry disabled): ${shortError(
err,
)}`,
);
return {};
}
})();
}
return this.writeClassMapPromise;
}
/**
* Build (or reuse a cached) external toolset for a workspace. Returns the
* merged tools, the open client handles to release, and per-server outcomes.
@@ -285,37 +162,11 @@ export class McpClientsService {
}
},
};
// #489: the run accumulates a SET of leases — the primary cache lease PLUS any
// throwaway client minted by an in-run transport-recovery reconnect. They are
// NEVER released mid-run (releasing a swapped-out client while a concurrent
// in-flight call still holds it would INDUCE a second failure); the caller
// releases the WHOLE set together at turn-end. A recovery reconnect pushes its
// lease onto this live array, which the consumer closes over.
const leaseSet: Closable[] = [release];
// #489: per-RUN transport-recovery binding, one per server, SHARED by all of
// that server's tools so a swap by one call is seen by the next (CAS by
// identity). Kept per-run (here, not in the cached entry) because the binding
// + lease-set state is per-run.
const bindings = new Map<number, ServerBinding>();
const capMs = mcpCallTimeoutMs();
// Wrap each cached tool with the recovery layer. On a transport error a
// declared readOnly tool reconnects its server and retries ONCE; a write is
// never blind-retried (indeterminate — may have applied before the reset). A
// tool without provenance (a minimal stub entry in a test) passes through raw.
const tools: Record<string, Tool> = {};
for (const [key, tool] of Object.entries(entry.tools)) {
const meta = entry.toolMeta?.[key];
tools[key] = meta
? this.wrapWithTransportRecovery(entry, meta, tool, leaseSet, bindings, capMs)
: tool;
}
// One release handle drives the whole leased entry; closing it releases all
// underlying clients together (they share the same lease lifecycle).
return {
tools,
clients: leaseSet,
tools: entry.tools,
clients: [release],
outcomes: entry.outcomes,
instructions: entry.instructions,
};
@@ -403,16 +254,6 @@ export class McpClientsService {
// Per-call total wall-clock cap, read once for this build (env-overridable).
const callTimeoutMs = mcpCallTimeoutMs();
const instructions: McpServerInstruction[] = [];
// merged-key -> provenance for the per-run recovery wrapper (#489).
const toolMeta: Record<string, ToolProvenance> = {};
// Shared Docmost write-class map (#489) — classifies a tool by its raw name.
// Loaded ONLY when at least one server is a TRUSTED internal Docmost server
// (see isInternalDocmostServer): for third-party servers the map is never
// applied (a name collision must not grant readOnly-retry), so we skip the
// dynamic ESM load entirely in that (currently universal) case.
const writeClassMap = servers.some((s) => this.isInternalDocmostServer(s))
? await this.getWriteClassMap()
: null;
// Per-server connect+tools result, still tagged with its server so the merge
// below can be applied in the SAME order as `servers` (see the parallel note).
@@ -486,23 +327,11 @@ export class McpClientsService {
// against names already merged from earlier servers, so no external
// tool is silently overwritten on collision. The returned count drives
// whether this server's prompt guidance is included (≥1 tool merged).
// #489 (review): the Docmost write-class map keys by DOCMOST tool names and
// may ONLY be trusted for a server KNOWN to be the internal Docmost MCP
// server. Every row here is an admin-configured THIRD-PARTY endpoint, so a
// third-party WRITE tool that happens to be named like a Docmost read
// (getPage, listPages, ...) must NOT inherit readOnly — that would auto-retry
// a mutation on a transport error (double-apply, the #435 class). Gate the
// map on the trust check; untrusted servers get writeClass=undefined -> the
// recovery wrapper treats them as writes and never auto-retries.
const trustWriteClass = this.isInternalDocmostServer(server);
const merged = this.mergeNamespaced(
tools,
result.guarded,
server.name,
server.id,
toolMeta,
i,
trustWriteClass ? writeClassMap : null,
);
outcomes.push({ name: server.name, ok: true });
// Include this server's guidance ONLY when it actually contributed at
@@ -524,8 +353,6 @@ export class McpClientsService {
clients,
outcomes,
instructions,
servers,
toolMeta,
expiresAt: Date.now() + CACHE_TTL_MS,
refCount: 0,
evicted: false,
@@ -552,33 +379,18 @@ export class McpClientsService {
picked: Record<string, Tool>,
serverName: string,
serverId: string,
toolMeta: Record<string, ToolProvenance>,
serverIndex: number,
// The Docmost write-class map, or `null` for an UNTRUSTED (third-party)
// server whose tools must all default to write (never auto-retried).
writeClassMap: Record<string, ToolWriteClass> | null,
): { count: number; prefix: string } {
let count = 0;
for (const { full, raw, tool } of namespace(picked, serverName)) {
let key = full;
for (const [name, tool] of Object.entries(namespace(picked, serverName))) {
let key = name;
if (key in target) {
const original = key;
key = disambiguate(full, serverId, (candidate) => candidate in target);
key = disambiguate(name, serverId, (candidate) => candidate in target);
this.logger.debug(
`External MCP tool name "${original}" collided; renamed to "${key}"`,
);
}
target[key] = tool;
// Record provenance so the per-run recovery wrapper (#489) can reconnect
// this tool's server and re-resolve it by its raw name. writeClass is set
// ONLY from a TRUSTED (internal-Docmost) map; for a third-party server the
// map is null -> writeClass stays undefined -> the wrapper treats the tool
// as a write and never auto-retries it (no double-apply on name collision).
toolMeta[key] = {
serverIndex,
rawName: raw,
writeClass: writeClassMap ? writeClassMap[raw] : undefined,
};
count += 1;
}
return { count, prefix: namespacePrefix(serverName) };
@@ -612,10 +424,7 @@ export class McpClientsService {
// Defense in depth: re-validate the actual request host on EVERY fetch
// AND pin the socket to a validated IP via the dispatcher's connect
// lookup, closing the DNS-rebinding TOCTOU between check and connect.
// #489: the SSE transport uses the raised-bodyTimeout dispatcher (idle
// between calls is legit); HTTP uses the tight one.
fetch:
transportType === 'sse' ? this.guardedFetchSse : this.guardedFetchHttp,
fetch: this.guardedFetch,
},
})) as unknown as McpClient;
return client;
@@ -696,176 +505,6 @@ export class McpClientsService {
}
}
/**
* Wrap one merged external tool with the per-run transport-recovery layer (#489).
*
* attempt 1 runs on the server's CURRENT binding (the cached client, or a client
* a sibling tool already reconnected this run). On a REAL transport error
* (undici/@ai-sdk socket/body-timeout shapes {@link isRetryableConnectError},
* NOT a mock) and ONLY for a declared readOnly tool, it reconnects the server
* and retries EXACTLY ONCE on the fresh client; a write is surfaced as an
* indeterminate error (it may have applied before the reset never
* blind-retried). A single per-call cap bounds BOTH attempts + the reconnect,
* and the run's abort signal is checked before the retry AND before minting a
* fresh connection (no connection is opened for a stopped run).
*/
private wrapWithTransportRecovery(
entry: CacheEntry,
meta: ToolProvenance,
template: Tool,
leaseSet: Closable[],
bindings: Map<number, ServerBinding>,
capMs: number,
): Tool {
const original = template.execute;
if (typeof original !== 'function') return template;
const service = this;
const { serverIndex, rawName, writeClass } = meta;
let binding = bindings.get(serverIndex);
if (!binding) {
binding = { current: null };
bindings.set(serverIndex, binding);
}
const boundBinding = binding;
const execute = async (args: unknown, options: ToolCallOptions) => {
// The per-call cap governs the WHOLE sequence (attempt1 + reconnect +
// attempt2). Compose it with the run's abort signal so a Stop or the cap
// ends any awaited call — @ai-sdk/mcp does not settle on abort, so we RACE.
const capController = new AbortController();
const capTimer = setTimeout(() => {
capController.abort(new Error(`MCP tool call timed out after ${capMs}ms`));
}, capMs);
capTimer.unref?.();
const runSignal = options?.abortSignal;
const composed = runSignal
? AbortSignal.any([runSignal, capController.signal])
: capController.signal;
const stopped = () => runSignal?.aborted === true || capController.signal.aborted;
const callOn = async (
exec: NonNullable<Tool['execute']>,
): Promise<unknown> => {
const aborted = new Promise<never>((_, reject) => {
const fail = () => reject(abortReason(composed));
if (composed.aborted) fail();
else composed.addEventListener('abort', fail, { once: true });
});
return Promise.race([exec(args, { ...options, abortSignal: composed }), aborted]);
};
const execFor = (
state: RecoveredServerState | null,
): NonNullable<Tool['execute']> | undefined =>
state ? (state.tools[rawName]?.execute as NonNullable<Tool['execute']>) : original;
try {
// Snapshot the target BEFORE the call so a swap by a concurrent call is
// detected by identity in the catch.
const attemptState = boundBinding.current;
const attemptExec = execFor(attemptState);
if (typeof attemptExec !== 'function') {
throw new Error(`external MCP tool "${rawName}" is not callable`);
}
try {
return await callOn(attemptExec);
} catch (err) {
// Never retry on a Stop or an exhausted cap.
if (stopped()) throw err;
// Only a genuine transport break is a recovery candidate.
if (!isRetryableConnectError(err)) throw err;
// A write tool is INDETERMINATE on a transport error (may have applied
// before the reset) — surface that; do NOT auto-retry (double-apply is
// the #435 incident class).
if (!isReadOnlyWriteClass(writeClass)) {
throw new Error(
`external MCP tool "${rawName}" hit a transport error and MAY have already ` +
`applied on the server — not retried automatically; verify state before ` +
`retrying. (${shortError(err)})`,
);
}
// Abort check BEFORE minting a fresh connection (no socket for a
// stopped run). LIMITATION (#489, LOW): the reconnect's own connect is
// bounded by CONNECT_TIMEOUT_MS but does NOT itself observe `composed`,
// so a Stop that lands DURING the handshake is only honored at the next
// `stopped()` gate (before the retry) — a bounded ≤5s late-abort window;
// the throwaway client is closed at turn-end regardless. Threading
// `composed` into the SHARED (CAS-deduped) reconnect is deliberately
// avoided: it would let the first caller's abort tear down a reconnect a
// concurrent still-live caller depends on.
if (stopped()) throw err;
// CAS-swap by IDENTITY: mint+swap only if nobody swapped since this
// call's snapshot; a losing concurrent call awaits the same reconnect
// and retries on the SAME fresh client.
let target: RecoveredServerState;
if (boundBinding.current === attemptState) {
if (!boundBinding.reconnecting) {
boundBinding.reconnecting = (async () => {
const server = entry.servers[serverIndex];
const fresh = await service.reconnectServer(server, capMs);
leaseSet.push(fresh.lease); // accumulate; released at turn-end
boundBinding.current = fresh.state;
return fresh.state;
})();
// Clear the in-flight marker once it settles (success or failure) so
// a LATER death of the new client can reconnect again.
void boundBinding.reconnecting.then(
() => (boundBinding.reconnecting = undefined),
() => (boundBinding.reconnecting = undefined),
);
}
target = await boundBinding.reconnecting;
} else {
target = boundBinding.current as RecoveredServerState;
}
// Abort check BEFORE the retry.
if (stopped()) throw err;
const retryExec = execFor(target);
if (typeof retryExec !== 'function') throw err;
return await callOn(retryExec);
}
} finally {
clearTimeout(capTimer);
}
};
return { ...template, execute } as unknown as Tool;
}
/**
* Reconnect ONE server for an in-run recovery (#489): open a fresh client and
* list+wrap its tools. The throwaway client is NOT cached it is owned by the
* RUN via the returned lease (closed at turn-end), independent of the shared
* cache entry (whose TTL rebuild heals future turns). On a failure the fresh
* client is closed so its socket never leaks.
*/
private async reconnectServer(
server: AiMcpServer,
capMs: number,
): Promise<{ state: RecoveredServerState; lease: Closable }> {
const client = await this.connectWithTimeout(server, CONNECT_TIMEOUT_MS);
let tools: Record<string, Tool>;
try {
const raw = await withTimeout(client.tools(), CONNECT_TIMEOUT_MS);
const allow = server.toolAllowlist;
const picked =
Array.isArray(allow) && allow.length > 0 ? pick(raw, allow) : raw;
tools = wrapToolsWithCallTimeout(picked, capMs);
} catch (err) {
void client.close().catch(() => undefined);
throw err;
}
let released = false;
const lease: Closable = {
close: async () => {
if (released) return;
released = true;
await client.close().catch(() => undefined);
},
};
return { state: { client, tools }, lease };
}
/** Mark an entry evicted; close its clients now if nothing is leasing them. */
private evict(entry: CacheEntry): void {
clearTimeout(entry.timer);
@@ -915,21 +554,22 @@ export function validateResolvedAddresses(addrs: readonly LookupAddress[]): {
* certificate validation still uses the real hostname (we never rewrite the URL
* to an IP literal).
*/
function buildPinnedDispatcher(bodyTimeoutMs: number): Agent {
// External-MCP traffic uses a DEDICATED, shorter HEADERS silence timeout
function buildPinnedDispatcher(): Agent {
// External-MCP traffic uses a DEDICATED, shorter silence timeout
// (`AI_MCP_STREAM_TIMEOUT_MS`, default 1 min) — deliberately tighter than the
// chat provider's 15-min `streamTimeoutMs()` — so a byte-silent/hung MCP
// upstream is broken in ~1 min instead of 15. We keep the keep-alive options
// from `streamingDispatcherOptions()` but OVERRIDE the timeouts. `bodyTimeout`
// is passed in per-transport (#489): tight for HTTP (fresh request per call),
// raised for SSE (one long-lived body across calls — idle BETWEEN calls is
// legit). The per-call total cap (`AI_MCP_CALL_TIMEOUT_MS`) is the complementary
// guard for chatty-but-stuck calls that keep the socket warm yet never return.
const headersMs = mcpStreamTimeoutMs();
// from `streamingDispatcherOptions()` but OVERRIDE headers/body timeouts.
// Accepted trade-off: a legitimately long but byte-silent single tool call,
// and an SSE transport idling >1 min BETWEEN tool calls, are also cut here; the
// per-call total cap (wrapToolsWithCallTimeout, `AI_MCP_CALL_TIMEOUT_MS`) is the
// complementary guard for chatty-but-stuck calls that keep the socket warm yet
// never return.
const mcpSilenceMs = mcpStreamTimeoutMs();
return new Agent({
...streamingDispatcherOptions(),
headersTimeout: headersMs,
bodyTimeout: bodyTimeoutMs,
headersTimeout: mcpSilenceMs,
bodyTimeout: mcpSilenceMs,
connect: {
lookup: (hostname, _options, callback) => {
// Always resolve ALL addresses ourselves; do not trust the caller's
@@ -1029,22 +669,18 @@ function pick(
function namespace(
tools: Record<string, Tool>,
serverName: string,
): Array<{ full: string; raw: string; tool: Tool }> {
): Record<string, Tool> {
const prefix = namespacePrefix(serverName);
const out: Array<{ full: string; raw: string; tool: Tool }> = [];
const taken: Record<string, true> = {};
const out: Record<string, Tool> = {};
for (const [name, t] of Object.entries(tools)) {
const safe = sanitizeName(name);
let full = capName(`${prefix}_${safe}`);
// Duplicate names within ONE server can still collide after sanitize/
// truncate — suffix-disambiguate so the second tool is not overwritten.
if (full in taken) {
full = disambiguate(full, '', (candidate) => candidate in taken);
if (full in out) {
full = disambiguate(full, '', (candidate) => candidate in out);
}
taken[full] = true;
// Keep the RAW (un-namespaced) name alongside the merged key so the per-run
// recovery wrapper (#489) can re-resolve the same tool on a fresh client.
out.push({ full, raw: name, tool: t });
out[full] = t;
}
return out;
}
@@ -1168,69 +804,6 @@ export function wrapToolWithCallTimeout(tool: Tool, ms: number): Tool {
return { ...tool, execute } as unknown as Tool;
}
/**
* undici / Node network error CODES that mean the connection broke (not an
* application-level error) a transient transport failure a readOnly call may
* safely retry after reconnecting. Matched against the REAL error shapes (#489):
* a socket reset surfaces as `TypeError: fetch failed` whose `.cause` is an
* undici `SocketError { code:'UND_ERR_SOCKET' }`; a body-timeout as
* `TypeError: terminated` whose `.cause` is `BodyTimeoutError`. Classifying by
* these real codes/names (not by mock errors) is essential a mock-shaped
* predicate would leave eviction silently dead in production while CI is green.
*/
const RETRYABLE_TRANSPORT_ERROR_CODES: ReadonlySet<string> = new Set([
'ECONNRESET',
'ECONNREFUSED',
'ECONNABORTED',
'EPIPE',
'ETIMEDOUT',
'EAI_AGAIN',
'ENETUNREACH',
'EHOSTUNREACH',
'UND_ERR_SOCKET',
'UND_ERR_CONNECT_TIMEOUT',
'UND_ERR_HEADERS_TIMEOUT',
'UND_ERR_BODY_TIMEOUT',
'UND_ERR_CLOSED',
'UND_ERR_DESTROYED',
]);
/** undici error CLASS names for the same transport-break conditions. */
const RETRYABLE_TRANSPORT_ERROR_NAMES: ReadonlySet<string> = new Set([
'SocketError',
'BodyTimeoutError',
'HeadersTimeoutError',
'ConnectTimeoutError',
'ClientClosedError',
'ClientDestroyedError',
]);
/**
* Whether `err` is a retryable TRANSPORT break (a broken socket / body timeout),
* classified by the REAL undici/@ai-sdk error shapes (#489). undici surfaces a
* reset as `TypeError('fetch failed'|'terminated')` with the real error in
* `.cause`, and @ai-sdk/mcp may wrap it again in an `MCPClientError` (cause
* chain), so we walk `.cause` (bounded depth) checking `.code` and `.name`. An
* app-level tool error (a 4xx, a validation failure) is NOT retryable and returns
* false only a connection-level failure heals with a reconnect.
*/
export function isRetryableConnectError(err: unknown, depth = 0): boolean {
if (!err || typeof err !== 'object' || depth > 6) return false;
const e = err as {
code?: unknown;
name?: unknown;
cause?: unknown;
};
if (typeof e.code === 'string' && RETRYABLE_TRANSPORT_ERROR_CODES.has(e.code)) {
return true;
}
if (typeof e.name === 'string' && RETRYABLE_TRANSPORT_ERROR_NAMES.has(e.name)) {
return true;
}
if (e.cause != null) return isRetryableConnectError(e.cause, depth + 1);
return false;
}
/** The signal's reason as an Error (informative thrown value on abort/timeout). */
function abortReason(signal: AbortSignal): Error {
const r = signal.reason;
@@ -1,266 +0,0 @@
import type { ModelMessage } from 'ai';
import {
resolveReplayBudget,
isContextOverflowError,
estimateMessagesTokens,
trimHistoryForReplay,
REPLAY_BUDGET_DEFAULT_TOKENS,
REPLAY_TRUNCATION_MARKER,
REPLAY_TURN_COLLAPSED_MARKER,
} from './history-budget';
describe('resolveReplayBudget', () => {
it('uses floor(0.7 x window) for a configured window (no cap)', () => {
// 0.7 x 60k = 42k
expect(resolveReplayBudget(60_000)).toEqual({
thresholdTokens: 42_000,
usedDefault: false,
});
// 0.7 x 1M = 700k — NOT capped (anti-brick vs the window, not a cost limiter).
expect(resolveReplayBudget(1_000_000)).toEqual({
thresholdTokens: 700_000,
usedDefault: false,
});
});
it('accepts the raw ::text stored form', () => {
expect(resolveReplayBudget('60000').thresholdTokens).toBe(42_000);
});
// The crux (#490): a chat with NO context window configured must STILL be
// budgeted — those are exactly the installs that hit terminal overflow.
it('applies the flat default when the window is unset/empty', () => {
expect(resolveReplayBudget(undefined)).toEqual({
thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS,
usedDefault: true,
});
expect(resolveReplayBudget('')).toEqual({
thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS,
usedDefault: true,
});
expect(resolveReplayBudget(' ')).toEqual({
thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS,
usedDefault: true,
});
});
it('treats an explicit 0 as the off-switch (distinct from unset)', () => {
expect(resolveReplayBudget(0)).toEqual({
thresholdTokens: null,
usedDefault: false,
});
expect(resolveReplayBudget('0')).toEqual({
thresholdTokens: null,
usedDefault: false,
});
});
it('falls back to the default on a negative/garbage value', () => {
expect(resolveReplayBudget(-5).usedDefault).toBe(true);
expect(resolveReplayBudget('abc').usedDefault).toBe(true);
});
});
describe('isContextOverflowError', () => {
it('classifies a real provider 400 context-overflow shape', () => {
// OpenAI-compatible shape.
expect(
isContextOverflowError({
statusCode: 400,
message:
"This model's maximum context length is 128000 tokens. However, your messages resulted in 214000 tokens. Please reduce the length of the messages.",
}),
).toBe(true);
// Anthropic-style wording.
expect(
isContextOverflowError({
status: 400,
message: 'prompt is too long: 250000 tokens > 200000 maximum',
}),
).toBe(true);
// Nested body + string status.
expect(
isContextOverflowError({
response: { status: '400' },
message: 'input is too long for the requested model',
}),
).toBe(true);
// Error instance with the cause carrying the body.
const e = new Error('Bad request');
(e as any).statusCode = 400;
(e as any).cause = new Error('maximum context window exceeded');
expect(isContextOverflowError(e)).toBe(true);
});
it('does NOT classify unrelated 400s or auth/rate-limit errors', () => {
expect(
isContextOverflowError({ statusCode: 400, message: 'invalid tool schema' }),
).toBe(false);
expect(
isContextOverflowError({
statusCode: 429,
message: 'context length exceeded but rate limited',
}),
).toBe(false);
expect(isContextOverflowError({ statusCode: 500, message: 'server error' })).toBe(
false,
);
expect(isContextOverflowError(undefined)).toBe(false);
expect(isContextOverflowError('some random string')).toBe(false);
});
});
// Helpers to build ModelMessage fixtures in the ai@6 shape.
const userMsg = (text: string): ModelMessage =>
({ role: 'user', content: [{ type: 'text', text }] }) as ModelMessage;
const assistantMsg = (
text: string,
toolCallId?: string,
toolName?: string,
): ModelMessage =>
({
role: 'assistant',
content: [
{ type: 'text', text },
...(toolCallId
? [{ type: 'tool-call', toolCallId, toolName, input: {} }]
: []),
],
}) as ModelMessage;
const toolMsg = (
toolCallId: string,
toolName: string,
value: unknown,
): ModelMessage =>
({
role: 'tool',
content: [
{ type: 'tool-result', toolCallId, toolName, output: { type: 'json', value } },
],
}) as ModelMessage;
describe('trimHistoryForReplay', () => {
it('null budget disables trimming (returns the same reference)', () => {
const msgs = [userMsg('hi'), assistantMsg('yo')];
const r = trimHistoryForReplay(msgs, null);
expect(r.trimmed).toBe(false);
expect(r.messages).toBe(msgs);
});
it('leaves history under budget untouched (same reference)', () => {
const msgs = [userMsg('hi'), assistantMsg('a short answer')];
const r = trimHistoryForReplay(msgs, 100_000);
expect(r.trimmed).toBe(false);
expect(r.messages).toBe(msgs);
});
it('truncates OLD tool outputs but keeps recent turns full', () => {
const big = 'X'.repeat(40_000); // ~16k tokens on its own
const msgs: ModelMessage[] = [];
// 6 OLD turns (indices 0..5), each with a huge tool output.
for (let i = 0; i < 6; i++) {
msgs.push(userMsg(`old q${i}`));
msgs.push(assistantMsg('looking', `c${i}`, 'getPage'));
msgs.push(toolMsg(`c${i}`, 'getPage', { body: big }));
msgs.push(assistantMsg(`old a${i}`));
}
// 3 small recent turns, then the CURRENT turn with its own huge tool output.
// With REPLAY_KEEP_RECENT_TURNS=4 the last 4 user-turns stay full, so only
// these small recent turns + the current big one are kept full; the 6 old
// turns above fall in the trim region.
for (let i = 0; i < 3; i++) {
msgs.push(userMsg(`recent q${i}`));
msgs.push(assistantMsg(`recent a${i}`));
}
msgs.push(userMsg('current q'));
msgs.push(assistantMsg('looking', 'cR', 'getPage'));
msgs.push(toolMsg('cR', 'getPage', { body: big }));
msgs.push(assistantMsg('current a'));
// Budget large enough that phase-1 tool truncation alone brings it under.
const r = trimHistoryForReplay(msgs, 30_000);
expect(r.trimmed).toBe(true);
const flat = JSON.stringify(r.messages);
// The CURRENT turn's tool output survives in full.
expect(flat).toContain(big);
// Old outputs were truncated with the marker.
expect(flat).toContain(REPLAY_TRUNCATION_MARKER);
// Phase 1 sufficed: the oldest turns were NOT collapsed.
expect(flat).not.toContain(REPLAY_TURN_COLLAPSED_MARKER);
expect(estimateMessagesTokens(r.messages)).toBeLessThan(
estimateMessagesTokens(msgs),
);
});
it('collapses the oldest turns when tool truncation is not enough', () => {
// Many turns with LARGE assistant TEXT (not tool output) so phase 1 can't help.
const bigText = 'слово '.repeat(8_000); // large Cyrillic text per turn
const msgs: ModelMessage[] = [];
for (let i = 0; i < 12; i++) {
msgs.push(userMsg(`q${i}`));
msgs.push(assistantMsg(bigText));
}
const r = trimHistoryForReplay(msgs, 30_000);
expect(r.trimmed).toBe(true);
// Oldest turns collapsed; result fits (best-effort) and is much smaller.
expect(estimateMessagesTokens(r.messages)).toBeLessThan(
estimateMessagesTokens(msgs),
);
// The LAST turn's text is preserved in full (recent turns stay full).
expect(JSON.stringify(r.messages[r.messages.length - 1])).toContain(bigText);
});
it('is deterministic / byte-stable for identical inputs', () => {
const big = 'Y'.repeat(30_000);
const build = (): ModelMessage[] => {
const m: ModelMessage[] = [];
for (let i = 0; i < 10; i++) {
m.push(userMsg(`q${i}`));
m.push(assistantMsg('t', `c${i}`, 'getPage'));
m.push(toolMsg(`c${i}`, 'getPage', { body: big }));
}
return m;
};
const a = trimHistoryForReplay(build(), 15_000);
const b = trimHistoryForReplay(build(), 15_000);
expect(JSON.stringify(a.messages)).toBe(JSON.stringify(b.messages));
});
it('never leaves an unpaired tool-call after collapsing (balanced history)', () => {
const big = 'Z'.repeat(40_000);
const msgs: ModelMessage[] = [];
for (let i = 0; i < 10; i++) {
msgs.push(userMsg(`q${i}`));
msgs.push(assistantMsg('t', `c${i}`, 'getPage'));
msgs.push(toolMsg(`c${i}`, 'getPage', { body: big }));
}
const r = trimHistoryForReplay(msgs, 8_000);
// Count tool-call vs tool-result parts in the trimmed output.
let calls = 0;
let results = 0;
for (const m of r.messages) {
if (!Array.isArray(m.content)) continue;
for (const p of m.content as Array<{ type?: string }>) {
if (p.type === 'tool-call') calls++;
if (p.type === 'tool-result' || p.type === 'tool-error') results++;
}
}
// Every surviving tool-call has a surviving result (collapsing drops BOTH).
expect(calls).toBe(results);
// Collapsed turns carry the marker.
expect(JSON.stringify(r.messages)).toContain(REPLAY_TURN_COLLAPSED_MARKER);
});
it('respects the provider fact: under-budget contextTokens skips trimming', () => {
const big = 'W'.repeat(60_000);
const msgs = [
userMsg('q'),
assistantMsg('t', 'c1', 'getPage'),
toolMsg('c1', 'getPage', { body: big }),
];
// char-estimate is high, but the provider says we are well under budget.
const r = trimHistoryForReplay(msgs, 100_000, 5_000);
expect(r.trimmed).toBe(false);
expect(r.messages).toBe(msgs);
});
});
@@ -1,375 +0,0 @@
/**
* History-replay token budget (#490).
*
* The whole persisted conversation is replayed to the provider on EVERY turn, so
* a long chat eventually exceeds the model's context window and the provider 400s
* on every turn terminally (the chat "bricks"). This module bounds the replayed
* history at REPLAY TIME only: it never mutates what is persisted (the DB stays
* the full record), and its output is a deterministic, byte-stable function of its
* input so the trimmed prefix is identical turn to turn (provider prompt-cache
* friendliness real money on long chats).
*
* The PRIMARY signal is the provider's own fact: `metadata.contextTokens` from the
* last turn. The chars-based {@link estimateTokens} (shared with the client) is
* used only for the DELTA of not-yet-sent messages, to decide WHAT to trim, and as
* the fallback for chats with no usage yet.
*/
import type { ModelMessage } from 'ai';
import { estimateTokens } from '@docmost/token-estimate';
/** Flat default budget when no context window is configured (tokens). */
export const REPLAY_BUDGET_DEFAULT_TOKENS = 100_000;
/** Fraction of a configured context window used as the budget. */
export const REPLAY_BUDGET_WINDOW_FRACTION = 0.7;
/**
* Fraction of the normal budget used for the REACTIVE re-trim after a provider
* context-overflow 400 the preventive estimate under-counted, so cut harder.
*/
export const REPLAY_AGGRESSIVE_FRACTION = 0.5;
/**
* Turns (a user message + its assistant/tool replies) kept FULL at the tail,
* including the current one never trimmed. Older turns are compacted first.
*/
export const REPLAY_KEEP_RECENT_TURNS = 4;
/** Leading chars kept from a truncated old tool output. */
export const REPLAY_TOOL_OUTPUT_HEAD = 800;
/** Trailing chars kept from a truncated old tool output. */
export const REPLAY_TOOL_OUTPUT_TAIL = 300;
/** Marker inserted where an old tool output was truncated for replay. */
export const REPLAY_TRUNCATION_MARKER =
'[…truncated for replay; call the tool again to read the full output]';
/** Marker for a whole old turn collapsed to its text. */
export const REPLAY_TURN_COLLAPSED_MARKER =
'[earlier tool activity omitted for replay]';
export interface ReplayBudget {
/** Token threshold above which replay history is trimmed; `null` = OFF. */
thresholdTokens: number | null;
/** True when the flat default was used (no context window configured). */
usedDefault: boolean;
}
/**
* Resolve the replay budget from the RAW stored `chatContextWindow` (text/number).
* - a positive value -> `floor(fraction × window)` (NO cap the budgeter is
* anti-brick protection against the window itself, not a cost/economy limiter,
* exactly as the codebase already treats maxOutputTokens; the reactive branch
* still guarantees anti-brick regardless of how high this budget is)
* - explicit `0` -> OFF (admin opt-out; `null` threshold)
* - unset/empty/invalid-> the flat default (still protects the installations
* that hit terminal overflow are exactly the ones that never set a window)
*
* Note the raw value is needed because the parsed `chatContextWindow` collapses
* both `0` and unset to `undefined`, which would erase the explicit off-switch.
*/
export function resolveReplayBudget(rawContextWindow: unknown): ReplayBudget {
let n: number | undefined;
if (typeof rawContextWindow === 'number') {
n = rawContextWindow;
} else if (typeof rawContextWindow === 'string') {
const t = rawContextWindow.trim();
n = t === '' ? undefined : Number(t);
}
// Unset / empty / non-numeric / negative -> flat default (the protective case).
if (n === undefined || !Number.isFinite(n) || n < 0) {
return { thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS, usedDefault: true };
}
// Explicit 0 -> off-switch.
if (n === 0) {
return { thresholdTokens: null, usedDefault: false };
}
return {
thresholdTokens: Math.floor(REPLAY_BUDGET_WINDOW_FRACTION * n),
usedDefault: false,
};
}
/**
* The effective replay threshold for THIS turn, given the base budget and whether
* the PREVIOUS turn hit a context-overflow 400 (the reactive-recovery signal,
* `metadata.replayOverflow`). On recovery the base budget is scaled down by
* {@link REPLAY_AGGRESSIVE_FRACTION}: the overflowing turn produced no usage
* signal, so the preventive estimate under-counted and a normal-threshold trim may
* not shrink enough to fit this harder cut is what un-bricks the chat.
*
* A `null` base budget (trimming OFF) is passed through unchanged: an explicit
* off-switch is never overridden by the recovery path.
*/
export function resolveEffectiveReplayThreshold(
thresholdTokens: number | null,
priorOverflowed: boolean,
): number | null {
if (!priorOverflowed || thresholdTokens == null) return thresholdTokens;
return Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION);
}
/**
* True when a provider error is a CONTEXT-OVERFLOW rejection (the prompt exceeds
* the model's window). Providers surface this as an HTTP 400 with a recognizable
* message; match both the status and the message patterns robustly across
* OpenAI-compatible / Anthropic / Gemini wordings, since the exact shape varies.
*/
export function isContextOverflowError(error: unknown): boolean {
const status = extractStatus(error);
const msg = extractMessage(error).toLowerCase();
// Message patterns seen across providers for "prompt too long".
const overflowPattern =
/context (?:length|window)|maximum context|too many tokens|too large for|reduce the length|prompt is too long|input (?:is )?too long|exceeds? the (?:maximum )?(?:context|token)|maximum.*tokens|string too long/;
if (!overflowPattern.test(msg)) return false;
// A 400/413 with an overflow-shaped message is an overflow. Some providers
// omit/rewrite the status, so accept the message match when the status is
// unknown, but reject it for auth/rate-limit statuses that never mean overflow.
if (status === 400 || status === 413) return true;
if (status === 401 || status === 403 || status === 429) return false;
return true;
}
function extractStatus(error: unknown): number | undefined {
if (!error || typeof error !== 'object') return undefined;
const e = error as Record<string, unknown>;
for (const k of ['statusCode', 'status']) {
const v = e[k];
if (typeof v === 'number') return v;
if (typeof v === 'string' && /^\d+$/.test(v)) return Number(v);
}
// Nested (e.g. { response: { status } } / { cause: { statusCode } }).
for (const k of ['response', 'cause', 'data']) {
const nested = e[k];
if (nested && typeof nested === 'object') {
const s = extractStatus(nested);
if (s !== undefined) return s;
}
}
return undefined;
}
function extractMessage(error: unknown): string {
if (error == null) return '';
if (typeof error === 'string') return error;
if (error instanceof Error) {
// Include nested causes (provider libs wrap the real body in `cause`).
const cause = (error as { cause?: unknown }).cause;
return `${error.message} ${cause ? extractMessage(cause) : ''}`;
}
if (typeof error === 'object') {
const e = error as Record<string, unknown>;
const parts: string[] = [];
for (const k of ['message', 'error', 'body', 'responseBody', 'data']) {
const v = e[k];
if (typeof v === 'string') parts.push(v);
else if (v && typeof v === 'object') parts.push(extractMessage(v));
}
return parts.join(' ');
}
return String(error);
}
/** Rough token size of a ModelMessage array via the shared chars estimator. */
export function estimateMessagesTokens(
messages: ReadonlyArray<ModelMessage>,
): number {
let total = 0;
for (const m of messages) {
total += estimateTokens(serializeContent(m.content));
}
return total;
}
function serializeContent(content: unknown): string {
if (typeof content === 'string') return content;
try {
return JSON.stringify(content) ?? '';
} catch {
return '';
}
}
/** Deep JSON string of an arbitrary value, bounded so estimation never throws. */
function stringifyValue(value: unknown): string {
if (typeof value === 'string') return value;
try {
return JSON.stringify(value) ?? String(value);
} catch {
return String(value);
}
}
export interface TrimResult {
messages: ModelMessage[];
/** Whether any trimming was applied. */
trimmed: boolean;
/** Estimated tokens of the returned messages (chars-based). */
estimatedTokens: number;
}
/**
* Bound the replayed history to `budgetTokens`, deterministically. Returns the
* SAME array reference (no copy) when nothing needs trimming, so the common case
* is free and byte-identical. Trimming order (spec #490):
* 1. truncate OLD turns' tool outputs (head+tail + marker) the bulk of the size
* 2. mechanically collapse the OLDEST turns to their text (concatenation, no LLM)
* 3. the current + last {@link REPLAY_KEEP_RECENT_TURNS} turns stay FULL
*
* `budgetTokens === null` disables trimming. `priorContextTokens` (the provider's
* fact from last turn) short-circuits the decision: when it is known and already
* under budget we skip trimming even if the char-estimate is higher (the provider
* count is authoritative). The char-estimate drives WHAT to cut.
*/
export function trimHistoryForReplay(
messages: ModelMessage[],
budgetTokens: number | null,
priorContextTokens?: number,
): TrimResult {
if (budgetTokens == null) {
return { messages, trimmed: false, estimatedTokens: 0 };
}
const estimated = estimateMessagesTokens(messages);
// Decision signal: prefer the provider's fact (last turn's contextTokens) plus
// the estimated delta of the messages appended since; fall back to the pure
// char-estimate for a chat with no usage yet.
const projected =
priorContextTokens != null
? Math.max(priorContextTokens, estimated)
: estimated;
if (projected <= budgetTokens) {
return { messages, trimmed: false, estimatedTokens: estimated };
}
// The tail we always keep full: from the Nth-from-last user message onward.
const boundary = recentBoundaryIndex(messages, REPLAY_KEEP_RECENT_TURNS);
const tail = messages.slice(boundary);
let head = messages.slice(0, boundary).map(cloneMessage);
// Phase 1: truncate old tool outputs.
for (const m of head) {
if (m.role === 'tool') truncateToolMessage(m);
}
let out = [...head, ...tail];
let est = estimateMessagesTokens(out);
if (est <= budgetTokens) {
return { messages: out, trimmed: true, estimatedTokens: est };
}
// Phase 2: collapse the oldest turns (in `head`) to their text, one at a time,
// from the oldest, until we fit or the whole head is collapsed.
const turns = splitTurns(head);
const collapsed: ModelMessage[] = [];
let i = 0;
for (; i < turns.length; i++) {
if (est <= budgetTokens) break;
collapsed.push(...collapseTurn(turns[i]));
// Re-estimate the whole prospective output.
const remaining = turns.slice(i + 1).flat();
out = [...collapsed, ...remaining, ...tail];
est = estimateMessagesTokens(out);
}
// Include any turns we didn't need to collapse.
const remaining = turns.slice(i).flat();
out = [...collapsed, ...remaining, ...tail];
est = estimateMessagesTokens(out);
return { messages: out, trimmed: true, estimatedTokens: est };
}
/** Index of the first message of the Nth-from-last user turn (0 if fewer). */
function recentBoundaryIndex(
messages: ReadonlyArray<ModelMessage>,
keepTurns: number,
): number {
const userIdx: number[] = [];
for (let i = 0; i < messages.length; i++) {
if (messages[i].role === 'user') userIdx.push(i);
}
if (userIdx.length <= keepTurns) return 0;
return userIdx[userIdx.length - keepTurns];
}
/** Split a message list into turns; each turn starts at a `user` message. */
function splitTurns(messages: ModelMessage[]): ModelMessage[][] {
const turns: ModelMessage[][] = [];
for (const m of messages) {
if (m.role === 'user' || turns.length === 0) turns.push([m]);
else turns[turns.length - 1].push(m);
}
return turns;
}
/**
* Collapse a whole turn to its plain text (mechanical concatenation, not an LLM
* summary). Keeps the user message; replaces the assistant/tool messages with a
* single assistant text message = the assistant's concatenated text + a marker
* when tool activity was dropped. Dropping BOTH the tool-call and tool-result
* parts together keeps the rebuilt history balanced (no unpaired calls).
*/
function collapseTurn(turn: ModelMessage[]): ModelMessage[] {
const out: ModelMessage[] = [];
let assistantText = '';
let hadTools = false;
for (const m of turn) {
if (m.role === 'user') {
out.push(m);
} else if (m.role === 'assistant') {
const { text, tools } = extractAssistantText(m.content);
assistantText += text;
hadTools = hadTools || tools;
} else if (m.role === 'tool') {
hadTools = true;
} else {
out.push(m);
}
}
const note =
(assistantText ? assistantText.trimEnd() : '') +
(hadTools
? `${assistantText ? '\n\n' : ''}${REPLAY_TURN_COLLAPSED_MARKER}`
: '');
if (note) out.push({ role: 'assistant', content: note } as ModelMessage);
return out;
}
function extractAssistantText(content: unknown): {
text: string;
tools: boolean;
} {
if (typeof content === 'string') return { text: content, tools: false };
if (!Array.isArray(content)) return { text: '', tools: false };
let text = '';
let tools = false;
for (const part of content) {
const type = (part as { type?: string })?.type;
if (type === 'text') text += (part as { text?: string }).text ?? '';
else if (type === 'tool-call') tools = true;
}
return { text, tools };
}
/** Truncate every tool-result output in a `tool` message to head+tail+marker. */
function truncateToolMessage(message: ModelMessage): void {
const content = message.content;
if (!Array.isArray(content)) return;
for (const part of content) {
const p = part as { type?: string; output?: { type?: string; value?: unknown } };
if (p.type !== 'tool-result' && p.type !== 'tool-error') continue;
if (!p.output) continue;
const raw = stringifyValue(p.output.value);
const budget = REPLAY_TOOL_OUTPUT_HEAD + REPLAY_TOOL_OUTPUT_TAIL;
if (raw.length <= budget + REPLAY_TRUNCATION_MARKER.length) continue;
const truncated =
raw.slice(0, REPLAY_TOOL_OUTPUT_HEAD) +
`\n${REPLAY_TRUNCATION_MARKER}\n` +
raw.slice(raw.length - REPLAY_TOOL_OUTPUT_TAIL);
// Represent the shrunk output as a text output (a valid tool-result output).
p.output = { type: 'text', value: truncated };
}
}
/** Shallow-ish clone so trimming never mutates the caller's (persisted-derived)
* message objects only the OLD region is cloned before it is edited. */
function cloneMessage(m: ModelMessage): ModelMessage {
if (typeof m.content === 'string') return { ...m };
return {
...m,
content: (m.content as unknown[]).map((p) =>
p && typeof p === 'object' ? { ...(p as object) } : p,
),
} as ModelMessage;
}
@@ -9,9 +9,73 @@ import {
DEGENERATION_CHECK_STEP,
REPEATED_LINES_THRESHOLD,
MIN_PERIOD_REPEATS,
degenerationThresholds,
} from './output-degeneration';
import { AiChatService } from './ai-chat.service';
// Part A (#495 iter10): the detector thresholds are env-tunable. These drive the
// resolver against real repeat-count shapes and mutation-verify that the env
// override actually changes the trigger point (not a vacuous read).
describe('degeneration thresholds are env-configurable', () => {
const VARS = [
'AI_CHAT_DEGENERATION_REPEATED_LINES',
'AI_CHAT_DEGENERATION_PERIOD_MAX_LEN',
'AI_CHAT_DEGENERATION_PERIOD_MIN_REPEATS',
'AI_CHAT_DEGENERATION_CHECK_STEP',
];
const saved: Record<string, string | undefined> = {};
beforeEach(() => {
for (const v of VARS) saved[v] = process.env[v];
});
afterEach(() => {
for (const v of VARS) {
if (saved[v] === undefined) delete process.env[v];
else process.env[v] = saved[v];
}
});
it('defaults to the compiled constants when unset', () => {
for (const v of VARS) delete process.env[v];
expect(degenerationThresholds()).toEqual({
repeatedLines: REPEATED_LINES_THRESHOLD,
maxPeriodLen: 150,
minPeriodRepeats: MIN_PERIOD_REPEATS,
checkStep: DEGENERATION_CHECK_STEP,
});
});
it('falls back to the default on blank / invalid / non-positive values', () => {
for (const bad of ['', ' ', 'abc', '0', '-3', '1.5']) {
process.env.AI_CHAT_DEGENERATION_REPEATED_LINES = bad;
// '1.5' floors to 1 (still ≥1, valid); every other bad value → default.
const expected = bad === '1.5' ? 1 : REPEATED_LINES_THRESHOLD;
expect(degenerationThresholds().repeatedLines).toBe(expected);
}
});
it('a RAISED check-step suppresses a burst the default would have flagged', () => {
// A ~3.3KB periodic burst is periodic-degenerate, but shouldCheckDegeneration
// is the throttle gate. Default checkStep=2000 arms on it; raising the step
// above the burst size means the throttle never re-fires for it.
const burstLen = 'loadTools.\n'.repeat(300).length; // ~3300
delete process.env.AI_CHAT_DEGENERATION_CHECK_STEP;
expect(shouldCheckDegeneration(burstLen, 0)).toBe(true); // default 2000
process.env.AI_CHAT_DEGENERATION_CHECK_STEP = String(burstLen + 1);
expect(shouldCheckDegeneration(burstLen, 0)).toBe(false); // raised gate
});
it('a LOWERED repeated-lines threshold trips on a shorter identical-line run', () => {
// 8 identical lines: below the default 25 (rule 1) and below the periodic
// rule's 20 repeats — so isDegenerateOutput is false by default.
const shortRun = 'x\n'.repeat(8);
delete process.env.AI_CHAT_DEGENERATION_REPEATED_LINES;
expect(isDegenerateOutput(shortRun)).toBe(false);
// Lower rule 1 to 5 → the 8-line run now trips.
process.env.AI_CHAT_DEGENERATION_REPEATED_LINES = '5';
expect(isDegenerateOutput(shortRun)).toBe(true);
});
});
// Mock ONLY streamText so we can capture the onChunk/onStepFinish callbacks the
// service registers and drive them by hand; every other `ai` export the service
// uses (convertToModelMessages, stepCountIs, …) stays real.
@@ -23,6 +23,54 @@ export const MAX_PERIOD_LEN = 150;
/** Rule 2: minimum number of consecutive block repeats to trigger. */
export const MIN_PERIOD_REPEATS = 20;
/**
* Read a positive-integer threshold from an env var, falling back to `fallback`
* on unset/blank/invalid/non-positive. Mirrors the `AI_STREAM_PRE_RESPONSE_RETRIES`
* resolver in `ai-streaming-fetch.ts`: read the RAW string first so a blank value
* is treated as "unset" ( fallback) rather than coercing to 0. Thresholds must
* stay 1 a 0/negative would make the detector fire on any text (or never), so
* a bad value degrades to the safe compiled default instead. Env-tunable so an
* operator can retune the anti-babble guard (#444) without a redeploy, following
* the `AI_CHAT_FINAL_STEP_LOCKDOWN` toggle convention.
*/
function envThreshold(name: string, fallback: number): number {
const rawStr = process.env[name];
if (rawStr === undefined || rawStr.trim() === '') return fallback;
const raw = Number(rawStr);
return Number.isFinite(raw) && raw >= 1 ? Math.floor(raw) : fallback;
}
/**
* Resolve the degeneration-detector thresholds from the environment, each
* defaulting to the compiled constant above. Read fresh per call (not cached at
* import) so a test or a runtime env change takes effect deterministically.
*/
export function degenerationThresholds(): {
repeatedLines: number;
maxPeriodLen: number;
minPeriodRepeats: number;
checkStep: number;
} {
return {
repeatedLines: envThreshold(
'AI_CHAT_DEGENERATION_REPEATED_LINES',
REPEATED_LINES_THRESHOLD,
),
maxPeriodLen: envThreshold(
'AI_CHAT_DEGENERATION_PERIOD_MAX_LEN',
MAX_PERIOD_LEN,
),
minPeriodRepeats: envThreshold(
'AI_CHAT_DEGENERATION_PERIOD_MIN_REPEATS',
MIN_PERIOD_REPEATS,
),
checkStep: envThreshold(
'AI_CHAT_DEGENERATION_CHECK_STEP',
DEGENERATION_CHECK_STEP,
),
};
}
/**
* Rule 1 `REPEATED_LINES_THRESHOLD` consecutive IDENTICAL non-empty lines at
* the tail. Catches the classic newline-delimited loop ("loadTools.\n" ×N).
@@ -128,7 +176,11 @@ export function hasPeriodicTail(
* Pure the caller owns the abort side effect.
*/
export function isDegenerateOutput(text: string): boolean {
return hasRepeatedLineRun(text) || hasPeriodicTail(text);
const cfg = degenerationThresholds();
return (
hasRepeatedLineRun(text, cfg.repeatedLines) ||
hasPeriodicTail(text, cfg.maxPeriodLen, cfg.minPeriodRepeats)
);
}
/**
@@ -154,7 +206,7 @@ export function shouldCheckDegeneration(
textLen: number,
lastCheckLen: number,
): boolean {
return textLen - lastCheckLen >= DEGENERATION_CHECK_STEP;
return textLen - lastCheckLen >= degenerationThresholds().checkStep;
}
/**
@@ -307,6 +307,10 @@ export class PublicShareChatService {
system,
messages: modelMessages,
tools,
// Pin the AI SDK per-request retry budget explicitly (matches the SDK
// default of 2). Connection arithmetic: (1 + maxRetries) × (1 +
// AI_STREAM_PRE_RESPONSE_RETRIES) worst-case connects per turn.
maxRetries: 2,
// Bound the agent loop for anonymous callers.
stopWhen: stepCountIs(5),
// Cap per-request output so one anonymous call cannot run up the provider
@@ -1,160 +0,0 @@
import {
wrapInAppToolWithCap,
inAppToolCallCapMs,
type ToolAbortSignalSink,
} from './ai-chat-tools.service';
import type { Tool, ToolCallOptions } from 'ai';
/**
* #487 commit 1 in-app tool race-on-abort + safe-points + per-call cap.
*
* Tests assert the HONEST observable property the spec names "after Stop, NO
* new HTTP/WS call STARTS; an already-started single call may take either
* outcome" against the REAL wrapper mechanism (the composite abort signal it
* publishes on the client + the RACE it runs), NOT a timing-dependent proxy like
* "the write didn't land".
*/
// A minimal stand-in for the client's `toolAbortSignal` field. In production the
// wrapper publishes the composite here and the client's paginateAll /
// mutatePageContent safe-points read it; the fake "tool" below reads it the same
// way, so this exercises the real contract without a live DB / collab socket.
class FakeClient implements ToolAbortSignalSink {
private signal: AbortSignal | null = null;
setToolAbortSignal(signal: AbortSignal | null): void {
this.signal = signal;
}
getToolAbortSignal(): AbortSignal | null {
return this.signal;
}
}
// A ToolCallOptions with just the field the wrapper reads (abortSignal). The AI
// SDK passes a fuller object; the wrapper only spreads it and reads abortSignal.
const opts = (abortSignal?: AbortSignal): ToolCallOptions =>
({ toolCallId: 't1', messages: [], abortSignal }) as unknown as ToolCallOptions;
const tick = (ms = 5) => new Promise((r) => setTimeout(r, ms));
describe('#487 wrapInAppToolWithCap — race-on-abort + safe-points', () => {
it('after Stop, no NEW simulated call starts (multi-call tool)', async () => {
const client = new FakeClient();
const started: number[] = [];
// A multi-call tool that mirrors paginateAll: it consults the client signal
// at a safe-point BEFORE starting each simulated network call.
const multiCall: Tool = {
execute: (async (_args: unknown) => {
for (let i = 0; i < 6; i++) {
// Safe-point: exactly what paginateAll / mutatePageContent do.
client.getToolAbortSignal()?.throwIfAborted();
started.push(i);
await tick(10);
}
return 'done';
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(multiCall, client, 10_000);
const ac = new AbortController();
const call = (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
// Let one or two calls start, then Stop.
await tick(12);
ac.abort(new Error('user stop'));
await expect(call).rejects.toThrow(); // wrapper rejects promptly
const startedAtStop = started.length;
// Give the abandoned loser ample time; its next safe-point must throw because
// the (aborted) composite is still published on the client.
await tick(60);
expect(started.length).toBe(startedAtStop);
// It must NOT have run the whole sequence (that would mean Stop did nothing).
expect(started.length).toBeLessThan(6);
});
it('rejects immediately on Stop even if the call never settles (discard loser)', async () => {
const client = new FakeClient();
let settled = false;
const hang: Tool = {
execute: (async () => {
await new Promise(() => undefined); // never resolves
settled = true;
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(hang, client, 10_000);
const ac = new AbortController();
const call = (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
await tick(5);
ac.abort();
await expect(call).rejects.toThrow();
expect(settled).toBe(false);
});
it('per-call cap rejects a hung call with no Stop signal', async () => {
const client = new FakeClient();
const hang: Tool = {
execute: (async () => {
await new Promise(() => undefined);
}) as unknown as Tool['execute'],
} as Tool;
// Tiny cap; no options.abortSignal at all (composite = cap only).
const wrapped = wrapInAppToolWithCap(hang, client, 20);
const start = Date.now();
await expect(
(wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>)(
{},
opts(undefined),
),
).rejects.toThrow(/per-call cap/);
expect(Date.now() - start).toBeLessThan(2000);
});
it('publishes a composite signal on the client for the duration of the call', async () => {
const client = new FakeClient();
let seenDuringCall: AbortSignal | null = null;
const probe: Tool = {
execute: (async () => {
seenDuringCall = client.getToolAbortSignal();
return 'ok';
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(probe, client, 10_000);
const ac = new AbortController();
await (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
expect(seenDuringCall).not.toBeNull();
// The published composite must reflect the turn's Stop signal.
ac.abort();
expect((seenDuringCall as unknown as AbortSignal).aborted).toBe(true);
});
it('a completed call returns its raw result unchanged', async () => {
const client = new FakeClient();
const ok: Tool = {
execute: (async () => ({ items: [1, 2, 3] })) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(ok, client, 10_000);
const res = await (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(new AbortController().signal));
expect(res).toEqual({ items: [1, 2, 3] });
});
it('cap is env-tunable with a 2-minute default', () => {
const prev = process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
delete process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
expect(inAppToolCallCapMs()).toBe(120_000);
process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = '5000';
expect(inAppToolCallCapMs()).toBe(5000);
process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = 'not-a-number';
expect(inAppToolCallCapMs()).toBe(120_000);
if (prev === undefined) delete process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
else process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = prev;
});
});
@@ -1,5 +1,5 @@
import { Injectable, Logger } from '@nestjs/common';
import { tool, type Tool, type ToolCallOptions } from 'ai';
import { tool, type Tool } from 'ai';
import { z } from 'zod';
import { User } from '@docmost/db/types/entity.types';
import { TokenService } from '../../auth/services/token.service';
@@ -159,129 +159,6 @@ function __assertClientCallContract(client: DocmostClientLike): void {
* existing service-account `/mcp` path already calls loopback successfully, so
* this works for single-workspace self-host.
*/
/**
* #487: wall-clock cap for a SINGLE in-app tool call, env-tunable via
* `AI_CHAT_INAPP_TOOL_CALL_CAP_MS`. Bounds a read tool that would otherwise
* paginate for minutes and a content write whose collab commit hangs, and is the
* per-call CAP half of the composite abort signal every in-app tool is wrapped
* with (the other half is the turn's Stop signal). Default 2 minutes: generous
* for a legitimate long read/write, tight enough that a stuck call cannot pin the
* turn. The reconcile staleness floor (#487 commit 4) is derived as
* max(2 x this cap, 15 min), so keep this well under that.
*/
export function inAppToolCallCapMs(): number {
const raw = Number(process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 120_000;
}
/** #487: the composite signal's reason as an Error (informative thrown value). */
function inAppAbortReason(signal: AbortSignal): Error {
const r = signal.reason;
return r instanceof Error
? r
: new Error(typeof r === 'string' ? r : 'In-app tool call aborted');
}
/**
* The client surface {@link wrapInAppToolWithCap} drives (#487). Both methods are
* OPTIONAL: the real loopback DocmostClient implements them (so a Stop/cap reaches
* its pagination / pre-commit safe-points), but a client that omits them still
* gets the OUTER guarantee the race rejects on abort regardless. This keeps the
* wrapper decoupled from the exact client shape (unit-test doubles need not stub
* the plumbing).
*/
export interface ToolAbortSignalSink {
setToolAbortSignal?(signal: AbortSignal | null): void;
getToolAbortSignal?(): AbortSignal | null;
}
/**
* #487: wrap an in-app tool so a Stop (the turn's `options.abortSignal`) OR the
* per-call wall-clock cap REJECTS the call immediately, and so that SAME
* composite signal reaches the client's pagination / pre-commit safe-points (via
* `client.setToolAbortSignal`) making a Stop stop the NEXT HTTP/WS call from
* starting.
*
* Reuses the RACE pattern of `wrapToolWithCallTimeout` (mcp-clients.service.ts):
* the call is raced against the composite signal, so on abort we reject in the
* SAME tick and DISCARD the loser promise. Its network / collab teardown latency
* therefore never blocks the turn the supersede timeout W=10s (#487 commit 3)
* relies on this abort->settle latency being milliseconds, not a socket teardown.
* Awaiting the client's own signal-into-write path alone would NOT satisfy this
* (the loser could still be tearing down a collab socket).
*
* The composite is SET on the client at entry and deliberately NOT restored on
* unwind: after this wrapper rejects on abort, the ABANDONED loser promise keeps
* running, and its safe-points read the client field leaving the (aborted)
* composite there is exactly what makes the loser's NEXT call throw and stop. The
* next in-app tool call overwrites the field with its own fresh composite before
* any of its safe-points run, so a stale settled signal never leaks forward.
* SINGLE-WRITER by phase-1 assumption see DocmostClientContext.toolAbortSignal
* for the parallel-call caveat (#487).
*
* KNOWN LIMITATION (#487): a write tool that issues SEVERAL sequential collab
* commits can be aborted BETWEEN commits, leaving a partially-applied operation.
* Cancel guarantees "no NEW call starts", NOT "the write didn't land".
*/
export function wrapInAppToolWithCap(
toolDef: Tool,
client: ToolAbortSignalSink,
capMs: number,
): Tool {
const original = toolDef.execute;
if (typeof original !== 'function') return toolDef;
const execute = async (args: unknown, options: ToolCallOptions) => {
const capController = new AbortController();
const timer = setTimeout(() => {
capController.abort(
new Error(`In-app tool call exceeded the ${capMs}ms per-call cap`),
);
}, capMs);
timer.unref?.();
const composite = options?.abortSignal
? AbortSignal.any([options.abortSignal, capController.signal])
: capController.signal;
// Reject the MOMENT the composite fires, independent of whether `original`
// ever settles (a hung collab write / read would otherwise pin the turn). The
// losing `original` is left pending; Promise.race attaches a rejection
// handler to both inputs so a late rejection is never unhandled.
const aborted = new Promise<never>((_, reject) => {
const fail = () => reject(inAppAbortReason(composite));
if (composite.aborted) fail();
else composite.addEventListener('abort', fail, { once: true });
});
// Publish the composite so the client's pagination / pre-commit safe-points
// observe it (see the "not restored on unwind" rationale above). Guarded: a
// client without the plumbing still gets the OUTER race guarantee below.
client.setToolAbortSignal?.(composite);
try {
return await Promise.race([
(original as (a: unknown, o: ToolCallOptions) => Promise<unknown>)(
args,
{ ...options, abortSignal: composite },
),
aborted,
]);
} finally {
clearTimeout(timer);
}
};
return { ...toolDef, execute } as unknown as Tool;
}
/** #487: apply {@link wrapInAppToolWithCap} to every tool in a set. */
export function wrapInAppToolsWithCap(
tools: Record<string, Tool>,
client: ToolAbortSignalSink,
capMs: number,
): Record<string, Tool> {
const out: Record<string, Tool> = {};
for (const [name, t] of Object.entries(tools)) {
out[name] = wrapInAppToolWithCap(t, client, capMs);
}
return out;
}
@Injectable()
export class AiChatToolsService {
private readonly logger = new Logger(AiChatToolsService.name);
@@ -309,12 +186,7 @@ export class AiChatToolsService {
sessionId: string,
workspaceId: string,
aiChatId: string,
// #487: the returned client also carries the tool-cancellation plumbing
// (setToolAbortSignal/getToolAbortSignal). These are host plumbing, NOT part
// of the tool-execute surface (DocmostClientMethod), so they are surfaced here
// as an intersection rather than by widening that Pick — keeping the
// positional-call drift-guard (#446) scoped to the actual tool methods.
): Promise<DocmostClientLike & ToolAbortSignalSink> {
): Promise<DocmostClientLike> {
const apiUrl =
process.env.MCP_DOCMOST_API_URL ||
`http://127.0.0.1:${process.env.PORT || 3000}/api`;
@@ -758,15 +630,7 @@ export class AiChatToolsService {
// dependency and reuses the CASL enforcement already on `client`. When the
// loaded package predates #417 (factory undefined) or the loader is mocked in
// a unit test, signalling is a pure no-op and results are byte-identical.
// #487: wrap every in-app tool with the race-on-abort + per-call cap guard so
// a Stop / cap rejects immediately AND reaches the client's write/pagination
// safe-points. Applied as the OUTERMOST wrapper (over the comment-signal
// wrapper below) so the race governs the whole call. The client carries the
// per-call composite signal via setToolAbortSignal.
const capMs = inAppToolCallCapMs();
if (!createCommentSignalTracker) {
return wrapInAppToolsWithCap(tools, client, capMs);
}
if (!createCommentSignalTracker) return tools;
const tracker = createCommentSignalTracker({
probe: async (pageId: string, sinceMs: number) => {
@@ -795,11 +659,7 @@ export class AiChatToolsService {
},
});
return wrapInAppToolsWithCap(
wrapToolsWithCommentSignal(tools, tracker),
client,
capMs,
);
return wrapToolsWithCommentSignal(tools, tracker);
}
}
@@ -17,9 +17,10 @@ import { MovePageDto } from './move-page.dto';
// a valid ordering key the server itself generated would be refused on move.
//
// The tests below assert the CORRECT contract: any key the generator can produce
// must satisfy the DTO. The genuinely-failing case is marked `test.failing` so the
// suite stays green while locking the bug; it flips red (alerting us) once the DTO
// bounds are widened to cover the generator's real range.
// must satisfy the DTO. FIXED (#495 item 9): the DTO now validates `position` by
// CHARSET ([0-9A-Za-z], the generator's base-62 alphabet) instead of the wrong
// @MaxLength(12) length bound, so dense between-inserts are accepted; the former
// `test.failing` bug-lock is now a passing assertion.
function constraintErrors(position: unknown) {
const dto = plainToInstance(MovePageDto, {
@@ -47,24 +48,30 @@ describe('MovePageDto.position vs generateJitteredKeyBetween parity', () => {
expect(hasError(errors, 'position')).toBe(false);
});
// BUG LOCK: dense between-inserts produce keys longer than 12 chars, which
// MaxLength(12) rejects even though they are valid ordering keys. This SHOULD
// pass; it currently fails. Flips green when the DTO bound is fixed.
test.failing(
'accepts dense between-inserted keys (currently rejected by MaxLength(12))',
async () => {
let lo = generateJitteredKeyBetween(null, null);
let hi = generateJitteredKeyBetween(lo, null);
// Repeatedly insert just above `lo`, shrinking the gap so the key grows.
let longest = lo;
for (let i = 0; i < 40; i++) {
const mid = generateJitteredKeyBetween(lo, hi);
if (mid.length > longest.length) longest = mid;
hi = mid;
}
expect(longest.length).toBeGreaterThan(12); // sanity: we produced a long key
const errors = await constraintErrors(longest);
expect(hasError(errors, 'position')).toBe(false);
},
);
// FIXED: dense between-inserts produce keys longer than 12 chars, which the old
// MaxLength(12) rejected even though they are valid ordering keys. Now accepted.
it('accepts dense between-inserted keys longer than 12 chars', async () => {
let lo = generateJitteredKeyBetween(null, null);
let hi = generateJitteredKeyBetween(lo, null);
// Repeatedly insert just above `lo`, shrinking the gap so the key grows.
let longest = lo;
for (let i = 0; i < 40; i++) {
const mid = generateJitteredKeyBetween(lo, hi);
if (mid.length > longest.length) longest = mid;
hi = mid;
}
expect(longest.length).toBeGreaterThan(12); // sanity: we produced a long key
const errors = await constraintErrors(longest);
expect(hasError(errors, 'position')).toBe(false);
});
// The charset guard replaces the length bound: reject anything outside the
// generator's [0-9A-Za-z] alphabet (control chars, separators, injection) and
// the empty string, while still accepting every real key.
it('rejects a position with characters outside the fractional-index alphabet', async () => {
for (const bad of ['a0/b', 'a b', 'a\n0', 'a.b', '', "a';--"]) {
const errors = await constraintErrors(bad);
expect(hasError(errors, 'position')).toBe(true);
}
});
});
+13 -3
View File
@@ -1,8 +1,8 @@
import {
IsString,
IsOptional,
MinLength,
MaxLength,
Matches,
IsNotEmpty,
} from 'class-validator';
@@ -10,9 +10,19 @@ export class MovePageDto {
@IsString()
pageId: string;
// `position` is a fractional-indexing key from `generateJitteredKeyBetween`
// (the SAME generator page.service uses). Validate by CHARSET, not length: the
// generator's default base-62 alphabet is [0-9A-Za-z], and DENSE between-inserts
// legitimately grow a key well past a dozen chars (measured >40), so the old
// @MinLength(5)/@MaxLength(12) bounds wrongly 400'd valid ordering keys the
// server itself produced (Gitea #139 item 6). The charset regex rejects control
// chars / separators / injection, and a generous MaxLength stays only as a
// DoS guard — far above any realistic key, so it never rejects a real move.
@IsString()
@MinLength(5)
@MaxLength(12)
@Matches(/^[0-9A-Za-z]+$/, {
message: 'position must be a fractional-index key ([0-9A-Za-z])',
})
@MaxLength(256)
position: string;
@IsOptional()
@@ -0,0 +1,49 @@
import 'reflect-metadata';
import { plainToInstance } from 'class-transformer';
import { validate } from 'class-validator';
import { PageIdDto } from './page.dto';
// #435: PageIdDto.pageId carries a page's DOUBLE identity (internal UUID OR
// public 10-char slugId), both as bare strings. The DTO must accept exactly
// those two FORMATS and reject a malformed / swapped identity at the boundary.
async function pageIdErrors(pageId: unknown) {
const dto = plainToInstance(PageIdDto, { pageId });
const errors = await validate(dto as object);
return errors.some((e) => e.property === 'pageId');
}
const UUID = '019f499a-9f8c-7d68-b7be-ce100d7c6c56';
const SLUG = 'aB3xQ7kR2p';
describe('PageIdDto pageId format validation', () => {
it('accepts a canonical page UUID', async () => {
expect(await pageIdErrors(UUID)).toBe(false);
});
it('accepts a 10-char slugId', async () => {
expect(await pageIdErrors(SLUG)).toBe(false);
});
it('rejects a truncated / wrong-length slug', async () => {
expect(await pageIdErrors('aB3xQ7kR2')).toBe(true); // 9 chars
expect(await pageIdErrors('aB3xQ7kR2pX')).toBe(true); // 11 chars
});
it('rejects a slug with an illegal character', async () => {
expect(await pageIdErrors('aB3xQ7kR2!')).toBe(true);
});
it('rejects a full URL / path-shaped identity (not the bare id)', async () => {
expect(await pageIdErrors(`my-page-title-${SLUG}`)).toBe(true);
expect(await pageIdErrors(`https://x/p/${SLUG}`)).toBe(true);
});
it('rejects a malformed UUID', async () => {
expect(await pageIdErrors('019f499a-9f8c-7d68-b7be')).toBe(true);
expect(await pageIdErrors('not-a-uuid-at-all-really')).toBe(true);
});
it('rejects an empty string', async () => {
expect(await pageIdErrors('')).toBe(true);
});
});
@@ -0,0 +1,39 @@
import { applyDecorators } from '@nestjs/common';
import { Matches, ValidationOptions } from 'class-validator';
/**
* A page identity at the API boundary is EITHER the internal page UUID or the
* public 10-char slugId (page.repo.findById matches a non-UUID input as a
* slugId). Both arrive as bare strings, which is exactly how the two got swapped
* silently (incident family #435). This regex pins the two accepted FORMATS so a
* malformed / cross-wired identity (a truncated slug, a full URL, an email, an
* id from another entity kind that isn't even shaped like either) is rejected at
* the boundary instead of falling through to a confusing 404.
*
* - UUID: canonical 8-4-4-4-12 hex (version-agnostic page ids are UUIDv7,
* so only the shape/length is enforced, matching the MCP's UUID_RE
* and the server's isValidUUID acceptance).
* - slugId: exactly 10 chars over [0-9A-Za-z] (nanoid `generateSlugId`).
*
* The two are disjoint (a UUID is 36 chars WITH dashes, a slugId 10 chars
* WITHOUT), so a value can only satisfy one branch.
*/
export const PAGE_ID_OR_SLUG_ID_REGEX =
/^(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9A-Za-z]{10})$/i;
/**
* Validate that a DTO string field is a well-formed page identity (page UUID OR
* 10-char slugId). Composed decorator so the same format rule is applied
* consistently wherever a DTO accepts a `pageId` that may be either form.
*/
export function IsPageIdOrSlugId(
validationOptions?: ValidationOptions,
): PropertyDecorator {
return applyDecorators(
Matches(PAGE_ID_OR_SLUG_ID_REGEX, {
message:
'pageId must be a page UUID or a 10-character slugId',
...validationOptions,
}),
);
}
@@ -9,10 +9,16 @@ import {
import { Transform } from 'class-transformer';
import { ContentFormat } from './create-page.dto';
import { IsPageIdOrSlugId } from './page-identity.validator';
export class PageIdDto {
@IsString()
@IsNotEmpty()
// Format-validate the double identity (#435): accept only a page UUID or a
// 10-char slugId so a malformed / swapped identity is rejected at the boundary
// rather than passed to the repo as a bare string. Base for PageInfoDto,
// DeletePageDto, BacklinksListDto, AddLabelsDto/RemoveLabelDto, etc.
@IsPageIdOrSlugId()
pageId: string;
}
@@ -1,19 +1,38 @@
import { TemporaryNoteCleanupService } from '../temporary-note-cleanup.service';
/**
* Chainable Kysely stub that records every `.where(...)` call so the test can
* assert the sweep only selects armed, expired, not-yet-trashed notes. The
* terminal `.execute()` resolves the configured expired rows (the batch SELECT);
* `.executeTakeFirst()` resolves the per-row deadline re-read done just before
* each `removePage`. By default the re-read reports the note as still armed and
* still expired (epoch deadline < now), so the sweep proceeds to delete it;
* tests override `reReadFirst` to simulate a concurrent "Make permanent".
* Chainable Kysely stub for the temporary-note sweep.
*
* `this.db` serves the non-locking candidate SELECT (selectFrom/select/where/
* limit/execute -> the configured expired rows) AND `.transaction().execute(cb)`,
* which runs `cb` with a separate `trx` builder. The `trx` builder serves the
* per-row LOCKED re-check (selectFrom/select/where/forUpdate/skipLocked/
* executeTakeFirst). `lockedRows` drives what that locked re-check returns per
* candidate an id/creator/workspace row means "still expired, delete it";
* `undefined` means the predicate no longer matched (made permanent / re-armed /
* already trashed) or the row was SKIP-LOCKED by another worker, so it is skipped.
*/
function makeDbStub(expiredRows: any[]) {
function makeDbStub(expiredRows: any[], lockedRows?: any[]) {
const whereCalls: any[][] = [];
const reReadFirst = jest
.fn()
.mockResolvedValue({ temporaryExpiresAt: new Date(0), deletedAt: null });
const locked = [
...(lockedRows ??
expiredRows.map((r) => ({
id: r.id,
creatorId: r.creatorId,
workspaceId: r.workspaceId,
}))),
];
const lockedTakeFirst = jest.fn(() => Promise.resolve(locked.shift()));
const forUpdate = jest.fn(() => trxBuilder);
const skipLocked = jest.fn(() => trxBuilder);
const trxBuilder: any = {
selectFrom: jest.fn(() => trxBuilder),
select: jest.fn(() => trxBuilder),
where: jest.fn(() => trxBuilder),
forUpdate,
skipLocked,
executeTakeFirst: lockedTakeFirst,
};
const builder: any = {
selectFrom: jest.fn(() => builder),
select: jest.fn(() => builder),
@@ -23,9 +42,11 @@ function makeDbStub(expiredRows: any[]) {
}),
limit: jest.fn(() => builder),
execute: jest.fn().mockResolvedValue(expiredRows),
executeTakeFirst: reReadFirst,
transaction: jest.fn(() => ({
execute: (cb: (trx: any) => Promise<any>) => Promise.resolve(cb(trxBuilder)),
})),
};
return { builder, whereCalls, reReadFirst };
return { builder, whereCalls, lockedTakeFirst, forUpdate, skipLocked };
}
describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
@@ -52,20 +73,36 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
expect(builder.limit.mock.calls[0][0]).toBeGreaterThan(0);
});
it('soft-deletes each expired note via removePage, attributed to its creator', async () => {
it('soft-deletes each expired note via removePage under a row lock, attributed to its creator', async () => {
const expired = [
{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' },
{ id: 'p2', creatorId: 'u2', workspaceId: 'w1' },
];
const { builder } = makeDbStub(expired);
const { builder, forUpdate, skipLocked } = makeDbStub(expired);
const pageRepo = { removePage: jest.fn().mockResolvedValue(undefined) } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).toHaveBeenCalledTimes(2);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(1, 'p1', 'u1', 'w1');
expect(pageRepo.removePage).toHaveBeenNthCalledWith(2, 'p2', 'u2', 'w1');
// The 4th arg is the locking transaction — the delete runs inside it.
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
1,
'p1',
'u1',
'w1',
expect.anything(),
);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
2,
'p2',
'u2',
'w1',
expect.anything(),
);
// The re-check acquired a FOR UPDATE SKIP LOCKED lock (once per candidate).
expect(forUpdate).toHaveBeenCalledTimes(2);
expect(skipLocked).toHaveBeenCalledTimes(2);
});
it('continues past a failing note (one bad removePage does not abort the sweep)', async () => {
@@ -86,60 +123,29 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
service.sweepExpiredTemporaryNotes(),
).resolves.toBeUndefined();
expect(pageRepo.removePage).toHaveBeenCalledTimes(2);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(2, 'good', 'u2', 'w1');
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
2,
'good',
'u2',
'w1',
expect.anything(),
);
});
it('does NOT trash a note made permanent in the race window', async () => {
// The batch SELECT saw the note as expired, but before its turn in the loop
// the user clicked "Make permanent" (temporary_expires_at -> null). The
// deadline re-read must catch this and skip the delete so the keep wins.
it('does NOT trash a note made permanent / re-armed / already trashed (locked re-check returns nothing)', async () => {
// The batch SELECT saw the note as expired, but by the time the LOCKED
// re-check runs the row no longer matches the still-armed+expired+not-trashed
// predicate (make-permanent, re-arm to a future deadline, or already trashed),
// OR another worker holds the row (SKIP LOCKED). In every case the locked
// SELECT returns nothing and the delete is skipped so the keep/other worker wins.
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: null,
deletedAt: null,
});
const { builder, lockedTakeFirst } = makeDbStub(expired, [undefined]);
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(reReadFirst).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('skips a note already trashed since the batch SELECT', async () => {
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: new Date(0),
deletedAt: new Date(),
});
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('does NOT trash a note re-armed to a future deadline in the race window', async () => {
// The batch SELECT saw the note as expired, but before its turn in the loop
// the user disarmed it and re-armed it to a fresh, still-future deadline
// (temporary_expires_at -> now + 1h). The deadline re-read must catch that
// the note is no longer expired and skip the delete so the keep wins.
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: new Date(Date.now() + 60 * 60 * 1000),
deletedAt: null,
});
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(reReadFirst).toHaveBeenCalledTimes(1);
expect(lockedTakeFirst).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
@@ -151,4 +157,31 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('sweeps once on application bootstrap (catches notes expired during downtime)', async () => {
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder } = makeDbStub(expired);
const pageRepo = { removePage: jest.fn().mockResolvedValue(undefined) } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.onApplicationBootstrap();
expect(pageRepo.removePage).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).toHaveBeenCalledWith(
'p1',
'u1',
'w1',
expect.anything(),
);
});
it('a startup-sweep failure never blocks application boot', async () => {
const { builder } = makeDbStub([]);
// Make the candidate SELECT throw to simulate a boot-time DB hiccup.
builder.execute.mockRejectedValueOnce(new Error('db not ready'));
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await expect(service.onApplicationBootstrap()).resolves.toBeUndefined();
});
});
@@ -1,8 +1,13 @@
import { Injectable, Logger } from '@nestjs/common';
import {
Injectable,
Logger,
OnApplicationBootstrap,
} from '@nestjs/common';
import { Interval } from '@nestjs/schedule';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { executeTx } from '@docmost/db/utils';
/**
* Background sweeper for temporary notes ("structure or die"). A note whose
@@ -11,7 +16,7 @@ import { PageRepo } from '@docmost/db/repos/page/page.repo';
* TrashCleanupService; `@nestjs/schedule` is already enabled globally.
*/
@Injectable()
export class TemporaryNoteCleanupService {
export class TemporaryNoteCleanupService implements OnApplicationBootstrap {
private readonly logger = new Logger(TemporaryNoteCleanupService.name);
// Cap a single sweep so a large backlog (e.g. many notes created during
@@ -24,6 +29,20 @@ export class TemporaryNoteCleanupService {
private readonly pageRepo: PageRepo,
) {}
// Sweep once at startup so notes that expired during downtime are trashed
// right away instead of waiting up to an hour for the first @Interval tick.
// Best-effort: never let a startup-sweep failure block application boot.
async onApplicationBootstrap() {
try {
await this.sweepExpiredTemporaryNotes();
} catch (error) {
this.logger.error(
'Temporary-note startup sweep failed',
error instanceof Error ? error.stack : undefined,
);
}
}
// Hourly granularity: lifetimes are configured in hours, so a sub-hour
// overshoot past the deadline is acceptable.
@Interval('temporary-note-cleanup', 60 * 60 * 1000)
@@ -31,9 +50,11 @@ export class TemporaryNoteCleanupService {
try {
const now = new Date();
// Candidate ids (non-locking). The authoritative re-check happens per row
// under a row lock below, so this cheap pass just bounds the batch.
const expired = await this.db
.selectFrom('pages')
.select(['id', 'creatorId', 'workspaceId'])
.select(['id'])
.where('temporaryExpiresAt', 'is not', null)
.where('temporaryExpiresAt', '<', now)
.where('deletedAt', 'is', null) // not already in trash
@@ -41,50 +62,53 @@ export class TemporaryNoteCleanupService {
.execute();
let trashed = 0;
for (const page of expired) {
for (const candidate of expired) {
try {
// Re-check the deadline at deletion time. The SELECT above is not
// transactional, so a user may click "Make permanent"
// (toggleTemporary sets temporary_expires_at = null) in the window
// between the SELECT and this per-row removePage. removePage deletes
// by id with only a `deletedAt IS NULL` filter and never re-reads the
// deadline, so without this guard a concurrently-kept note would
// still be trashed. Re-read the row and skip it unless it is still
// armed AND still expired, so a concurrent make-permanent wins.
const current = await this.db
.selectFrom('pages')
.select(['temporaryExpiresAt', 'deletedAt'])
.where('id', '=', page.id)
.executeTakeFirst();
const didTrash = await executeTx(this.db, async (trx) => {
// Re-check the row UNDER A LOCK inside the transaction. `FOR UPDATE
// SKIP LOCKED`:
// - serialises against a concurrent "Make permanent"
// (toggleTemporary UPDATE takes the same row lock): if it commits
// first, the deadline predicate below no longer matches and we
// skip; if we lock first, it waits until this delete commits.
// - SKIP LOCKED lets a second worker/instance skip a row another
// sweeper already claimed instead of blocking on it (no double
// processing, no thundering herd).
// The predicate re-asserts still-armed AND still-expired AND
// not-already-trashed, so a make-permanent / prior sweep drops the row.
const locked = await trx
.selectFrom('pages')
.select(['id', 'creatorId', 'workspaceId'])
.where('id', '=', candidate.id)
.where('temporaryExpiresAt', 'is not', null)
.where('temporaryExpiresAt', '<', now)
.where('deletedAt', 'is', null)
.forUpdate()
.skipLocked()
.executeTakeFirst();
if (
!current ||
current.deletedAt !== null ||
current.temporaryExpiresAt === null ||
new Date(current.temporaryExpiresAt) >= now
) {
// Made permanent, already trashed, or no longer expired since the
// SELECT — leave it alone.
continue;
}
if (!locked) return false;
// Reuse the exact soft-delete path: recursive over children, removes
// shares in a transaction, and emits PAGE_SOFT_DELETED (tree
// invalidation + watcher notifications). Attribute the automatic
// deletion to the note's creator (no schema change). Both the SELECT
// above and removePage filter `deletedAt IS NULL`, so a double sweep
// is idempotent.
await this.pageRepo.removePage(
page.id,
// creatorId is set on every created page; a temporary note always
// has one. Cast to satisfy the non-null deletedById parameter.
page.creatorId as string,
page.workspaceId,
);
trashed++;
// Reuse the exact soft-delete path (recursive children + share
// removal + PAGE_SOFT_DELETED broadcast), running IN this locked
// transaction so the delete is atomic with the re-check and cannot
// deadlock on a nested independent transaction. The broadcast is
// deferred by removePage to this transaction's commit. Attribute the
// automatic deletion to the note's creator (no schema change).
await this.pageRepo.removePage(
locked.id,
// creatorId is set on every created page; a temporary note always
// has one. Cast to satisfy the non-null deletedById parameter.
locked.creatorId as string,
locked.workspaceId,
trx,
);
return true;
});
if (didTrash) trashed++;
} catch (error) {
this.logger.error(
`Failed to trash expired temporary note ${page.id}`,
`Failed to trash expired temporary note ${candidate.id}`,
error instanceof Error ? error.stack : undefined,
);
}
@@ -133,6 +133,9 @@ describe('ShareAliasController authz gates', () => {
creatorId: 'u-1',
alias: 'promo',
confirmReassign: true,
// The requesting user is forwarded so setAlias can gate the reassign
// 409 title disclosure on target-page view permission (#495).
user,
});
expect(result).toEqual({ id: 'alias-1' });
});
@@ -79,6 +79,9 @@ export class ShareAliasController {
creatorId: user.id,
alias: dto.alias,
confirmReassign: dto.confirmReassign,
// Gates whether the reassign 409 may reveal the current target's title
// (view-permission check on that page) — see setAlias (#495).
user,
});
}
@@ -1,4 +1,8 @@
import { BadRequestException, ConflictException } from '@nestjs/common';
import {
BadRequestException,
ConflictException,
ForbiddenException,
} from '@nestjs/common';
import { NoResultError } from 'kysely';
import { ShareAliasService } from './share-alias.service';
@@ -7,6 +11,8 @@ import { ShareAliasService } from './share-alias.service';
* 409 reassign guard, uniqueness-race handling, availability probe, and the
* request-time readable-target resolution (which re-runs the share boundary).
*/
const USER = { id: 'u-1' } as any;
describe('ShareAliasService', () => {
// Sentinel handed to repo calls so tests can assert they ran inside the tx.
const trx = { __trx: true };
@@ -27,6 +33,10 @@ describe('ShareAliasService', () => {
resolveReadableSharePage: jest.fn(),
isSharingAllowed: jest.fn(),
};
// Default: the requester CAN view the target page (validateCanView resolves),
// so the reassign 409 may disclose its title. Tests override to reject to
// assert the no-leak path.
const pageAccessService = { validateCanView: jest.fn().mockResolvedValue(undefined) };
// Fake kysely db: only .transaction().execute(cb) is used by setAlias.
const db = {
transaction: jest.fn(() => ({
@@ -37,9 +47,10 @@ describe('ShareAliasService', () => {
shareAliasRepo as any,
pageRepo as any,
shareService as any,
pageAccessService as any,
db as any,
);
return { service, shareAliasRepo, pageRepo, shareService, db };
return { service, shareAliasRepo, pageRepo, shareService, pageAccessService, db };
}
describe('setAlias', () => {
@@ -50,6 +61,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'A', // too short + uppercase
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -66,6 +78,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: ' My Page ',
});
@@ -114,6 +127,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'ted',
});
@@ -144,6 +158,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
@@ -179,6 +194,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'new',
});
@@ -190,30 +206,77 @@ describe('ShareAliasService', () => {
);
});
it('throws 409 with current target when name is taken and not confirmed', async () => {
const { service, shareAliasRepo, pageRepo } = makeService();
it('throws 409 with the target TITLE (never its id) when the requester CAN view it', async () => {
const { service, shareAliasRepo, pageRepo, pageAccessService } =
makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
alias: 'foo',
pageId: 'p-other',
});
pageRepo.findById.mockResolvedValue({ id: 'p-other', title: 'Other' });
pageAccessService.validateCanView.mockResolvedValue(undefined); // can view
try {
await service.setAlias({
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
} catch (err) {
expect(err).toBeInstanceOf(ConflictException);
expect((err as ConflictException).getResponse()).toMatchObject({
const body = (err as ConflictException).getResponse();
expect(body).toMatchObject({
code: 'ALIAS_REASSIGN_REQUIRED',
currentPageId: 'p-other',
currentPageTitle: 'Other',
});
// SECURITY (#495): the page id is NEVER disclosed, even to a viewer.
expect(body).not.toHaveProperty('currentPageId');
expect(pageAccessService.validateCanView).toHaveBeenCalledWith(
expect.objectContaining({ id: 'p-other' }),
USER,
);
}
expect(shareAliasRepo.updatePageId).not.toHaveBeenCalled();
});
it('throws 409 WITHOUT the title or id when the requester CANNOT view the target (#495)', async () => {
const { service, shareAliasRepo, pageRepo, pageAccessService } =
makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
alias: 'foo',
pageId: 'p-secret',
});
pageRepo.findById.mockResolvedValue({ id: 'p-secret', title: 'Secret' });
// No view permission on the target page -> validateCanView throws.
pageAccessService.validateCanView.mockRejectedValue(
new ForbiddenException(),
);
try {
await service.setAlias({
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
} catch (err) {
expect(err).toBeInstanceOf(ConflictException);
const body = (err as ConflictException).getResponse() as Record<
string,
unknown
>;
expect(body).toMatchObject({ code: 'ALIAS_REASSIGN_REQUIRED' });
// The enumeration hole: neither the id nor the title of a page the
// requester cannot see may leak.
expect(body).not.toHaveProperty('currentPageId');
expect(body.currentPageTitle ?? null).toBeNull();
}
expect(shareAliasRepo.updatePageId).not.toHaveBeenCalled();
});
@@ -231,6 +294,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
confirmReassign: true,
});
@@ -269,6 +333,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -294,6 +359,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -317,6 +383,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -346,6 +413,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -375,6 +443,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
confirmReassign: true,
});
@@ -406,6 +475,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'ted',
});
fail('expected ConflictException');
@@ -428,6 +498,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -450,18 +521,22 @@ describe('ShareAliasService', () => {
alias: 'free-name',
valid: true,
available: true,
currentPageId: null,
});
// SECURITY (#495): the availability probe must NOT leak any page id.
expect(res).not.toHaveProperty('currentPageId');
});
it('reports taken with the current target page', async () => {
it('reports taken WITHOUT leaking the current target page id (#495)', async () => {
const { service, shareAliasRepo } = makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
pageId: 'p-9',
});
const res = await service.checkAvailability('taken', 'ws-1');
expect(res).toMatchObject({ available: false, currentPageId: 'p-9' });
expect(res).toMatchObject({ available: false });
// The row exists (available:false) but its pageId is never returned — an
// authenticated member cannot map an alias name to a page id it can't view.
expect(res).not.toHaveProperty('currentPageId');
});
});
@@ -7,7 +7,8 @@ import {
import { ShareAliasRepo } from '@docmost/db/repos/share-alias/share-alias.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { ShareService } from './share.service';
import { Page, ShareAlias } from '@docmost/db/types/entity.types';
import { PageAccessService } from '../page/page-access/page-access.service';
import { Page, ShareAlias, User } from '@docmost/db/types/entity.types';
import { isValidShareAlias, normalizeShareAlias } from './share-alias.util';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
@@ -43,6 +44,7 @@ export class ShareAliasService {
private readonly shareAliasRepo: ShareAliasRepo,
private readonly pageRepo: PageRepo,
private readonly shareService: ShareService,
private readonly pageAccessService: PageAccessService,
@InjectKysely() private readonly db: KyselyDB,
) {}
@@ -55,9 +57,13 @@ export class ShareAliasService {
* `/l/<old>` link survives
* - name already points at pageId -> no-op (idempotent)
* - name points at ANOTHER page -> the "swap". Without confirmReassign
* we throw 409 carrying the current target so the client can confirm;
* with it we UPDATE the single row's page_id (every /l/<alias> link
* follows the 302 to the new page instantly no stale cache).
* we throw 409 so the client can confirm. SECURITY (#495): the 409 reveals
* the current target's title ONLY when `user` may VIEW that page, and never
* its id otherwise any member with one editable+shared page could iterate
* alias names with confirmReassign=false and map them to (id, title) of
* pages they cannot see. With confirmReassign we UPDATE the single row's
* page_id (every /l/<alias> link follows the 302 to the new page instantly
* no stale cache).
*
* To keep the invariant self-healing we DELETE every other alias row still
* pointing at this page (a legacy duplicate, or the target page's own former
@@ -77,8 +83,12 @@ export class ShareAliasService {
creatorId: string;
alias: string;
confirmReassign?: boolean;
// The requesting user — used ONLY to gate whether the reassign 409 may reveal
// the current target page's title (view-permission check). Not an authz gate
// for the write itself (the controller already validated edit on `pageId`).
user: User;
}): Promise<ShareAlias> {
const { workspaceId, pageId, creatorId, confirmReassign } = opts;
const { workspaceId, pageId, creatorId, confirmReassign, user } = opts;
const alias = normalizeShareAlias(opts.alias);
if (!isValidShareAlias(alias)) {
throw new BadRequestException(
@@ -97,14 +107,30 @@ export class ShareAliasService {
// The name is occupied by a DIFFERENT (or dangling) target page.
if (byName && byName.pageId !== pageId) {
if (!confirmReassign) {
// SECURITY (#495): only disclose the current target's TITLE, and only
// when the requester may VIEW that page. Never disclose its id (the
// client's confirm-reassign UX doesn't use it, and it is an enumerable
// identity). A member with one editable+shared page must NOT be able to
// iterate alias names and map them to (id, title) of pages they cannot
// see. When view is denied (or the alias is dangling) the 409 is the
// bare "occupied" fact — the client still shows a generic confirm modal.
const currentPage = byName.pageId
? await this.pageRepo.findById(byName.pageId)
: null;
let currentPageTitle: string | null = null;
if (currentPage) {
try {
await this.pageAccessService.validateCanView(currentPage, user);
currentPageTitle = currentPage.title ?? null;
} catch {
// No view permission on the target -> do not reveal its title.
currentPageTitle = null;
}
}
throw new ConflictException({
message: 'Alias already in use',
code: 'ALIAS_REASSIGN_REQUIRED',
currentPageId: byName.pageId,
currentPageTitle: currentPage?.title ?? null,
currentPageTitle,
});
}
// Confirmed swap. ORDER MATTERS: the partial unique index on
@@ -223,21 +249,27 @@ export class ShareAliasService {
alias: string;
valid: boolean;
available: boolean;
currentPageId: string | null;
}> {
const alias = normalizeShareAlias(rawAlias);
if (!isValidShareAlias(alias)) {
return { alias, valid: false, available: false, currentPageId: null };
return { alias, valid: false, available: false };
}
const existing = await this.shareAliasRepo.findByAliasAndWorkspace(
alias,
workspaceId,
);
// SECURITY (#495): return ONLY the boolean availability. The previous shape
// leaked `currentPageId` — the id of whatever page the alias already targets —
// to ANY authenticated workspace member, with no view-permission check on that
// page. An attacker could enumerate alias names and map them to page ids they
// have no access to. The taken/free bit is all the "is this address free"
// probe needs. The reassign flow (setAlias 409) may surface the target's
// TITLE, but only behind a `validateCanView` on that page (see setAlias); it
// never returns the page id.
return {
alias,
valid: true,
available: !existing,
currentPageId: existing?.pageId ?? null,
};
}
@@ -24,6 +24,54 @@ export const ALLOWED_RATINGS = new Set<string>([
'poor',
]);
// The ONLY route labels accepted. The endpoint is anonymous, so an un-checked
// `route` is a free-text write surface (arbitrary high-cardinality strings /
// injected text into the metrics table). The client only ever sends a label from
// a finite template dictionary (`templateRoute`), so we drop anything not in it.
//
// PARITY: this MUST mirror `KNOWN_ROUTE_TEMPLATES` in the client's
// `apps/client/src/lib/telemetry/route-template.ts` (the canonical source). A
// drift means legit client routes get dropped — keep the two in lockstep; the
// client self-consistency test asserts `templateRoute` only emits these values.
export const ALLOWED_ROUTE_TEMPLATES = new Set<string>([
'/',
'other',
// Static routes.
'/home',
'/spaces',
'/favorites',
'/login',
'/forgot-password',
'/password-reset',
'/setup/register',
'/settings/account/profile',
'/settings/account/preferences',
'/settings/workspace',
'/settings/ai',
'/settings/members',
'/settings/groups',
'/settings/spaces',
'/settings/sharing',
// Dynamic templates (slugs/ids are already collapsed to `:param`).
'/share/:shareId/p/:slug',
'/share/p/:slug',
'/share/:shareId',
'/p/:slug',
'/s/:space/p/:slug',
'/s/:space/trash',
'/s/:space',
'/labels/:label',
'/invites/:invitationId',
'/settings/groups/:groupId',
]);
// `attr` is a web-vitals attribution TARGET: a CSS-selector-ish string (an
// element path like `html>body>div#app>button.cta`), never free prose. Constrain
// it to a conservative CSS-selector charset so the anonymous endpoint cannot be
// used to write arbitrary text / PII / markup into the metrics table. A value
// containing anything outside this set is DROPPED (-> null); the event is kept.
export const ATTR_ALLOWED_CHARSET = /^[A-Za-z0-9#.\-_> :()[\]="'*+~,]+$/;
// Max events accepted per batch; the rest are ignored.
export const MAX_EVENTS_PER_BATCH = 50;
@@ -77,14 +125,20 @@ export function sanitizeVitalEvent(
? e.rating
: null;
// route: accept ONLY a known template label (dictionary check), else drop to
// null. The length cap stays as a cheap pre-guard before the Set lookup.
let route: string | null = null;
if (typeof e.route === 'string' && e.route.length > 0) {
route = e.route.slice(0, MAX_ROUTE_LENGTH);
const candidate = e.route.slice(0, MAX_ROUTE_LENGTH);
route = ALLOWED_ROUTE_TEMPLATES.has(candidate) ? candidate : null;
}
// attr: truncate, then accept ONLY if it is a CSS-selector-shaped string
// (charset whitelist); anything with characters outside the set is dropped.
let attr: string | null = null;
if (typeof e.attr === 'string' && e.attr.length > 0) {
attr = e.attr.slice(0, MAX_ATTR_LENGTH);
const candidate = e.attr.slice(0, MAX_ATTR_LENGTH);
attr = ATTR_ALLOWED_CHARSET.test(candidate) ? candidate : null;
}
let docSize: number | null = null;
@@ -90,6 +90,44 @@ describe('VitalsService.buildRows', () => {
expect(rows[0].attr).toHaveLength(MAX_ATTR_LENGTH);
});
it('keeps a known route template but DROPS an unknown/free-text route (#495)', () => {
const rows = svc.buildRows(
{
events: [
{ name: 'INP', value: 1, route: '/s/:space/p/:slug' }, // known
{ name: 'INP', value: 2, route: '/s/acme-corp/p/secret-slug' }, // raw path (slugs) — not a template
{ name: 'INP', value: 3, route: 'DROP TABLE client_metrics;--' }, // injected free text
{ name: 'INP', value: 4, route: '/home' }, // known static
],
},
WS,
);
expect(rows.map((r) => r.route)).toEqual([
'/s/:space/p/:slug',
null, // raw path dropped
null, // free text dropped
'/home',
]);
});
it('DROPS an attr that is not a CSS-selector-shaped string (#495)', () => {
const rows = svc.buildRows(
{
events: [
{ name: 'INP', value: 1, attr: 'div#app>button.cta' }, // valid selector
{ name: 'INP', value: 2, attr: 'user@example.com wrote a note' }, // free text / PII
{ name: 'INP', value: 3, attr: '<script>alert(1)</script>' }, // markup
],
},
WS,
);
expect(rows.map((r) => r.attr)).toEqual([
'div#app>button.cta',
null,
null,
]);
});
it('caps the batch at 50 events', () => {
const events = Array.from({ length: 200 }, () => ({ name: 'CLS', value: 1 }));
const rows = svc.buildRows({ events }, WS);
@@ -0,0 +1,89 @@
import * as path from 'path';
import { readFileSync } from 'fs';
// Mock ONLY kysely's `sql.raw(...).execute()` so we can observe what
// ensureConcurrentIndexes runs and how it handles failures, without a DB.
const execMock = jest.fn((_db: unknown) => Promise.resolve(undefined));
const rawMock = jest.fn((_stmt: string) => ({ execute: execMock }));
jest.mock('kysely', () => {
const actual = jest.requireActual('kysely');
return { ...actual, sql: { ...actual.sql, raw: rawMock } };
});
import { CONCURRENT_INDEXES, ensureConcurrentIndexes } from './concurrent-indexes';
describe('ensureConcurrentIndexes', () => {
const fakeDb = { __topLevelKysely: true } as never;
beforeEach(() => {
execMock.mockReset().mockResolvedValue(undefined);
rawMock.mockClear();
});
it('runs every registered index CONCURRENTLY, IF NOT EXISTS, outside a transaction', async () => {
const onLog = jest.fn();
await ensureConcurrentIndexes(fakeDb, onLog);
expect(rawMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length);
for (const call of rawMock.mock.calls) {
const stmt = call[0] as string;
expect(stmt).toContain('CREATE INDEX CONCURRENTLY');
expect(stmt).toContain('IF NOT EXISTS');
}
// Executed against the top-level db (a transaction would forbid CONCURRENTLY).
for (const call of execMock.mock.calls) {
expect(call[0]).toBe(fakeDb);
}
expect(onLog).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length);
});
it('is best-effort: a failing index does not abort the rest and is reported', async () => {
// Fail the FIRST index; the remaining ones must still be attempted.
execMock
.mockRejectedValueOnce(new Error('relation "pages" does not exist'))
.mockResolvedValue(undefined);
const onLog = jest.fn();
await expect(
ensureConcurrentIndexes(fakeDb, onLog),
).resolves.toBeUndefined();
expect(execMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length);
// The failure surfaced with an error argument for the caller to log.
const errored = onLog.mock.calls.filter((c) => c[1] !== undefined);
expect(errored).toHaveLength(1);
expect(String(errored[0][1])).toContain('does not exist');
});
});
// DRIFT GUARD: each CONCURRENT_INDEXES entry pre-builds an index that a plain
// migration ALSO creates with `IF NOT EXISTS`. If the two expressions diverge,
// Postgres would treat them as different indexes and the pre-build would NOT
// make the migration a no-op. Assert the migration files still contain each
// index's functional expression.
describe('CONCURRENT_INDEXES parity with the migrations', () => {
const migrationsDir = path.join(__dirname, 'migrations');
const files = [
'20260705T120000-perf-indexes.ts',
'20260706T120000-search-lookup-trgm.ts',
].map((f) => readFileSync(path.join(migrationsDir, f), 'utf8'));
const allMigrationSrc = files.join('\n');
it.each(CONCURRENT_INDEXES.map((i) => [i.name, i]))(
'migration source still creates %s with the same expression',
(_name, idx) => {
// Extract the `USING gin ((...expr...) gin_trgm_ops)` tail from the
// canonical create and assert the migration source contains it verbatim.
const m = (idx as { create: string }).create.match(
/ON \w+ (USING gin .+)$/,
);
expect(m).not.toBeNull();
const expr = (m as RegExpMatchArray)[1];
expect(allMigrationSrc).toContain(expr);
// And the migration must build it by the SAME index name.
expect(allMigrationSrc).toContain(
`CREATE INDEX IF NOT EXISTS ${(idx as { name: string }).name}`,
);
},
);
});
@@ -0,0 +1,84 @@
import { Kysely, sql } from 'kysely';
/**
* Indexes that MUST be built with `CREATE INDEX CONCURRENTLY` so an auto-deploy
* migration never takes a `SHARE` lock that blocks writes on a hot table
* (`pages`) for the potentially minutes-long GIN trigram build (#495 item 12).
*
* Kysely runs each migration INSIDE a transaction (Postgres has transactional
* DDL), and `CREATE INDEX CONCURRENTLY` cannot run inside a transaction block, so
* these cannot live in an ordinary migration. Instead {@link ensureConcurrentIndexes}
* builds them out-of-band (no transaction) BEFORE the migrator runs; the matching
* migrations keep a plain `CREATE INDEX IF NOT EXISTS` as a backstop, which then
* no-ops because the index already exists. So:
* - existing prod DB, incremental deploy: pre-build runs CONCURRENTLY (no write
* lock), migration's IF NOT EXISTS no-ops the write-blocking build is gone;
* - fresh DB (or a DB that has not yet created `pages` / `f_unaccent`): the
* pre-build fails and is swallowed (best-effort), and the migration builds the
* index normally on an empty/small table where the lock is irrelevant.
* Worst case therefore equals the previous behaviour; best case removes the lock.
*
* The `create` text is the CANONICAL definition it MUST match the migration's
* `IF NOT EXISTS` create expression exactly (same functional expression + opclass)
* or Postgres would treat them as two different indexes.
*/
export const CONCURRENT_INDEXES: ReadonlyArray<{
name: string;
create: string;
}> = [
{
// #348 perf-indexes — pages.title trigram (coalesce-free functional expr).
name: 'idx_pages_title_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_title_trgm ' +
'ON pages USING gin ((LOWER(f_unaccent(title))) gin_trgm_ops)',
},
{
// #348 perf-indexes — users.name trigram (member search-suggest).
name: 'idx_users_name_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_name_trgm ' +
'ON users USING gin ((LOWER(f_unaccent(name))) gin_trgm_ops)',
},
{
// #443 search-lookup-trgm — pages.text_content trigram (the slow, large one).
name: 'idx_pages_text_content_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_text_content_trgm ' +
'ON pages USING gin ((LOWER(f_unaccent(text_content))) gin_trgm_ops)',
},
];
/**
* Best-effort, non-transactional pre-build of {@link CONCURRENT_INDEXES}. Run
* BEFORE the migrator so the blocking `CREATE INDEX` in the corresponding
* migration becomes an `IF NOT EXISTS` no-op.
*
* `db` MUST be the top-level Kysely instance (NOT a transaction): each statement
* then executes on its own connection with no surrounding `BEGIN`, which is
* required for `CONCURRENTLY`. Every statement is independent and swallowed on
* error (a missing `pages`/`f_unaccent` on a fresh DB, an unsupported driver
* path, a permissions gap): the migration backstop still builds the index, so a
* failure here is never fatal. `onLog` reports progress/failures for the caller
* to route to its logger.
*/
export async function ensureConcurrentIndexes(
db: Kysely<any>,
onLog?: (message: string, error?: unknown) => void,
): Promise<void> {
for (const idx of CONCURRENT_INDEXES) {
try {
await sql.raw(idx.create).execute(db);
onLog?.(`Concurrent index ensured: ${idx.name}`);
} catch (error) {
// Non-fatal by design — the migration's IF NOT EXISTS create is the
// backstop. Common benign cause: the table/function does not exist yet on
// a fresh DB (the migrations will build the index instead).
onLog?.(
`Concurrent index pre-build skipped for ${idx.name} ` +
`(will fall back to the in-migration build)`,
error,
);
}
}
}
@@ -0,0 +1,91 @@
import { executeTx, registerAfterCommit } from './utils';
import { KyselyDB, KyselyTransaction } from './types/kysely.types';
// Post-commit hook contract (#495 item 13): a side effect registered via
// registerAfterCommit must run ONLY AFTER the owning transaction commits, and a
// hook registered against a passed-through existingTrx must fire at the OUTER
// commit boundary — never inside the inner call. We fake the Kysely transaction
// runner so the ordering is observable without a real DB.
/**
* A minimal db whose `.transaction().execute(cb)` records the commit ORDER: it
* runs `cb(trx)`, pushes 'commit' onto `log` (simulating the real commit that
* happens after the callback resolves), then returns the callback's result.
*/
function fakeDb(log: string[]): { db: KyselyDB; trx: KyselyTransaction } {
const trx = { __fakeTrx: true } as unknown as KyselyTransaction;
const db = {
transaction: () => ({
execute: async (cb: (t: KyselyTransaction) => Promise<unknown>) => {
const result = await cb(trx);
log.push('commit');
return result;
},
}),
} as unknown as KyselyDB;
return { db, trx };
}
describe('executeTx post-commit hooks', () => {
it('runs an afterCommit hook only AFTER the transaction commits', async () => {
const log: string[] = [];
const { db } = fakeDb(log);
await executeTx(db, async (trx) => {
log.push('body');
registerAfterCommit(trx, () => {
log.push('hook');
});
// The hook must NOT have run yet — the tx is still open.
expect(log).toEqual(['body']);
});
// Order proves post-commit: body → commit → hook (never body → hook → commit).
expect(log).toEqual(['body', 'commit', 'hook']);
});
it('drains hooks registered against a passed-through existingTrx at the OUTER commit', async () => {
const log: string[] = [];
const { db, trx: outerTrx } = fakeDb(log);
await executeTx(db, async (outer) => {
// Nested executeTx reuses the outer trx: it must NOT commit or drain now.
await executeTx(
db,
async (inner) => {
registerAfterCommit(inner, () => {
log.push('inner-hook');
});
},
outer,
);
// Still inside the outer tx — the inner hook has not fired.
expect(log).toEqual([]);
});
// The single (outer) commit drains the hook registered on the shared trx.
expect(log).toEqual(['commit', 'inner-hook']);
// Sanity: the trx the hooks were registered against is the outer one.
expect(outerTrx).toBeDefined();
});
it('a hook failure does not reject the already-committed executeTx', async () => {
const log: string[] = [];
const { db } = fakeDb(log);
await expect(
executeTx(db, async (trx) => {
registerAfterCommit(trx, () => {
throw new Error('cache del blew up');
});
registerAfterCommit(trx, () => {
log.push('second-hook-still-runs');
});
return 'ok';
}),
).resolves.toBe('ok');
// The throwing hook is swallowed; a later hook still runs.
expect(log).toEqual(['commit', 'second-hook-still-runs']);
});
});
@@ -33,16 +33,18 @@ import { type Kysely, sql } from 'kysely';
* - comments: `findPageComments` does WHERE page_id ORDER BY id ASC, but only
* `(page_id)` exists extra sort.
*
* DEPLOY-TIME LOCK WARNING: these are plain (non-CONCURRENT) CREATE INDEX
* statements CONCURRENTLY is impossible because Kysely runs each migration in a
* transaction. They take a SHARE lock that BLOCKS writes (INSERT/UPDATE/DELETE) on
* pages/users/groups/comments/page_history for the duration of the build. The two
* GIN trigram builds on pages.title / users.name are the slow ones and can take
* minutes on a large tenant a write-outage window during the deploy migration.
* For large installations, run this migration in a maintenance window, or build
* the trigram indexes out-of-band with CREATE INDEX CONCURRENTLY before deploying
* (then this migration's `IF NOT EXISTS` is a no-op). Small/typical tenants are
* unaffected.
* DEPLOY-TIME LOCK: these are plain (non-CONCURRENT) CREATE INDEX statements
* CONCURRENTLY is impossible HERE because Kysely runs each migration in a
* transaction. The two GIN trigram builds on pages.title / users.name are the
* slow ones and would take a SHARE lock that BLOCKS writes on pages/users for
* minutes on a large tenant. To avoid that, `ensureConcurrentIndexes`
* (database/concurrent-indexes.ts) now pre-builds BOTH trigram indexes with
* CREATE INDEX CONCURRENTLY (no transaction) BEFORE the migrator runs, so on an
* existing DB the two `IF NOT EXISTS` trigram creates below no-op and no write
* lock is taken. On a fresh DB the pre-build is skipped and they build on an
* empty table. Keep the two trigram creates in lockstep with their CANONICAL
* definitions in CONCURRENT_INDEXES. (The plain b-tree indexes further down are
* fast metadata-only builds; they are not pre-built.)
*/
export async function up(db: Kysely<any>): Promise<void> {
// Index-compatible, output-identical redefinition of f_unaccent (see header).
@@ -26,13 +26,16 @@ import { type Kysely, sql } from 'kysely';
* to update than b-trees); on the small instances this fork targets that cost
* is acceptable and the read win on agent lookups is the priority.
*
* DEPLOY-TIME LOCK WARNING: plain (non-CONCURRENT) CREATE INDEX Kysely runs
* each migration in a transaction, so CONCURRENTLY is impossible. The build takes
* a SHARE lock that BLOCKS writes on `pages` for its duration. The text_content
* GIN build is the slow one and can take minutes on a large tenant. For large
* installations, run this in a maintenance window or build the index out-of-band
* with CREATE INDEX CONCURRENTLY before deploying (then `IF NOT EXISTS` no-ops
* here). Small/typical tenants are unaffected.
* DEPLOY-TIME LOCK: this is a plain (non-CONCURRENT) CREATE INDEX Kysely runs
* each migration in a transaction, so CONCURRENTLY is impossible HERE, and the
* build would take a SHARE lock that BLOCKS writes on `pages` for its duration
* (the text_content GIN build can take minutes on a large tenant). To avoid that,
* `ensureConcurrentIndexes` (database/concurrent-indexes.ts) now pre-builds this
* index with CREATE INDEX CONCURRENTLY (no transaction) BEFORE the migrator runs,
* so on an existing DB the `IF NOT EXISTS` below no-ops and no write lock is taken.
* On a fresh DB the pre-build is skipped and this builds it on an empty table
* where the lock is irrelevant. Keep this create in lockstep with the CANONICAL
* definition in CONCURRENT_INDEXES (same expression + opclass).
*/
export async function up(db: Kysely<any>): Promise<void> {
// The title predicate is served by #348's idx_pages_title_trgm — see header.
@@ -1,24 +0,0 @@
import { type Kysely, sql } from 'kysely';
export async function up(db: Kysely<any>): Promise<void> {
// Chat-level metadata bag (#490). First use: the deferred-tool ACTIVATION set
// (`activatedTools`) is persisted here so it survives across turns — previously
// the set was reset every turn, forcing the model to re-run loadTools and pay a
// fresh round-trip to re-activate the same tools each turn. On load the stored
// set is intersected with the current valid deferred names, so an allowlist /
// role change can never inject a now-nonexistent tool.
//
// jsonb, defaulted to '{}' so every row (incl. pre-migration ones, backfilled
// by the default) is a readable object — the app never has to null-guard the
// bag itself, only individual keys.
await db.schema
.alterTable('ai_chats')
.addColumn('metadata', 'jsonb', (col) =>
col.notNull().defaultTo(sql`'{}'::jsonb`),
)
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema.alterTable('ai_chats').dropColumn('metadata').execute();
}
@@ -1,6 +1,5 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { sql } from 'kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import {
@@ -189,144 +188,6 @@ export class AiChatMessageRepo {
return query.returning(this.baseFields).executeTakeFirst();
}
/**
* #487 OWNER terminal write the streamText terminal callback's finalize. Like
* `update` but CONDITIONAL on `status='streaming' OR metadata.finalizeFailed`:
* the owner writes its real content EITHER when the row is still streaming (the
* normal case) OR when a reconcile stamp already flipped it to a terminal status
* but marked `finalizeFailed:true` the owner's real content OVERWRITES that
* placeholder stamp (owner-write priority, #487). A row that is properly terminal
* (no finalizeFailed) is left untouched (undefined) idempotent. The `patch`
* carries the real metadata WITHOUT finalizeFailed, so a successful write CLEARS
* the flag. Returns the updated row, or undefined when nothing matched.
*/
async finalizeOwner(
id: string,
workspaceId: string,
patch: Partial<{
content: string | null;
toolCalls: unknown;
metadata: unknown;
status: string | null;
}>,
trx?: KyselyTransaction,
): Promise<AiChatMessage | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatMessages')
.set({ ...(patch as Record<string, unknown>), updatedAt: new Date() })
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where((eb) =>
eb.or([
eb('status', '=', 'streaming'),
eb(sql<string>`(metadata->>'finalizeFailed')`, '=', 'true'),
]),
)
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* #487 RECONCILE status-only stamp settle a stuck 'streaming' row to a
* terminal status WITHOUT the owner's real content (which lived only in the
* dead process's memory — a documented loss). CONDITIONAL on `status='streaming'`
* (never touches an already-terminal row) AND it MERGES `finalizeFailed:true`
* into metadata (preserving the partial `parts` already persisted) so a LATER
* owner-write (finalizeOwner) can still OVERWRITE this placeholder with real
* content, and so `isInterruptResume` can EXCLUDE this row (a reconcile stamp is
* not a genuine user interruption). Returns the updated row, or undefined.
*/
async stampTerminalIfStreaming(
id: string,
workspaceId: string,
status: 'aborted' | 'error' | 'completed',
trx?: KyselyTransaction,
): Promise<AiChatMessage | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatMessages')
.set({
status,
metadata: sql`coalesce(metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('status', '=', 'streaming')
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* #487 reconcile clause (b): streaming assistant rows whose linked RUN has
* already reached a terminal status an asymmetry ("run settled / message
* streaming forever") the periodic reconcile heals by stamping the message.
* Returns the message id + its run's terminal status, bounded.
*/
async findStreamingWithTerminalRun(
limit = 200,
// #487: scope to ONE chat for the opportunistic per-turn reconcile (removes
// reconcile latency from the user-visible path); omit for the periodic sweep.
chat?: { chatId: string; workspaceId: string },
): Promise<
Array<{ messageId: string; workspaceId: string; runStatus: string }>
> {
let query = this.db
.selectFrom('aiChatMessages as m')
.innerJoin('aiChatRuns as r', 'r.assistantMessageId', 'm.id')
.select([
'm.id as messageId',
'm.workspaceId as workspaceId',
'r.status as runStatus',
])
.where('m.status', '=', 'streaming')
.where('r.status', 'in', ['succeeded', 'failed', 'aborted']);
if (chat) {
query = query
.where('m.chatId', '=', chat.chatId)
.where('m.workspaceId', '=', chat.workspaceId);
}
return query.limit(limit).execute();
}
/**
* #487 reconcile clause (d) historical-row safety: streaming rows older than
* `staleMs` whose chat has NO active run row (double-gated). Settle them to
* 'aborted' + finalizeFailed (so a late owner-write could still overwrite).
* Returns the count. Used ONLY by the periodic reconcile, never at boot.
*/
async sweepStreamingWithoutActiveRun(
staleMs: number,
trx?: KyselyTransaction,
): Promise<number> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - staleMs);
const rows = await db
.updateTable('aiChatMessages as m')
.set({
status: 'aborted',
metadata: sql`coalesce(m.metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('m.status', '=', 'streaming')
.where('m.updatedAt', '<', staleBefore)
.where((eb) =>
eb.not(
eb.exists(
eb
.selectFrom('aiChatRuns as r')
.select('r.id')
.whereRef('r.chatId', '=', 'm.chatId')
.where('r.status', 'in', ['pending', 'running']),
),
),
)
.returning('m.id')
.execute();
return rows.length;
}
/**
* Crash-recovery sweep (#183): flip every assistant row still left in the
* 'streaming' state (a turn that died mid-write before reaching a terminal
@@ -339,20 +200,13 @@ export class AiChatMessageRepo {
* step, so an actively-streaming row never matches; this prevents a fresh
* replica's boot-sweep from aborting a turn another replica is still streaming
* in a multi-instance deploy.
*
* #487: the sweep now ALSO marks `finalizeFailed:true` so a late owner-write can
* overwrite this placeholder with real content (owner-write priority).
*/
async sweepStreaming(trx?: KyselyTransaction): Promise<number> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - SWEEP_STREAMING_STALE_MS);
const rows = await db
.updateTable('aiChatMessages')
.set({
status: 'aborted',
metadata: sql`coalesce(metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.set({ status: 'aborted', updatedAt: new Date() })
.where('status', '=', 'streaming')
.where('updatedAt', '<', staleBefore)
.returning('id')
@@ -143,41 +143,6 @@ export class AiChatRunRepo {
.executeTakeFirst();
}
/**
* #487: CONDITIONAL terminal finalize flip a run to a terminal status and
* stamp `finished_at` ONLY while it is still active (pending|running), mirroring
* the assistant message's `onlyIfStreaming` guard. A double-settle (a late or
* second writer, a supersede applying a zombie's intended, a reconcile stamp)
* matches NOTHING once the row is terminal and is a benign no-op so a terminal
* status can never be clobbered by a later writer (last-writer-wins is gone).
*
* Returns the updated row when it WAS active (this call wrote it), else
* undefined (the row was already terminal another writer won). The caller
* distinguishes the two to resolve the correct settle outcome.
*/
async finalizeIfActive(
id: string,
workspaceId: string,
patch: { status: string; error: string | null },
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
const now = new Date();
return db
.updateTable('aiChatRuns')
.set({
status: patch.status,
error: patch.error,
finishedAt: now,
updatedAt: now,
})
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* Mark an EXPLICIT stop request on an active run (distinct from a browser
* disconnect, which never stops a run). Stamps `stop_requested_at` ONLY while
@@ -219,31 +184,6 @@ export class AiChatRunRepo {
* sweeps only runs UNTOUCHED past the window. Phase 1 is single-process, so the
* boot path supplies no window.
*/
/**
* #487 reconcile clause (c): active (pending|running) runs UNTOUCHED past
* `staleMs` candidates for "no live runner" abort. Staleness is measured from
* `updated_at` (the LAST-PROGRESS timestamp recordStep bumps it), NOT
* `started_at`, so a legitimate long-running marathon (1125 min of steady
* progress) is never a candidate. The caller filters these against its in-memory
* `active` / zombie maps ("no entry" is the PRIMARY gate a live entry is never
* aborted) before settling any of them. Bounded.
*/
async findStaleActive(
staleMs: number,
limit = 200,
trx?: KyselyTransaction,
): Promise<Array<{ id: string; workspaceId: string; chatId: string }>> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - staleMs);
return db
.selectFrom('aiChatRuns')
.select(['id', 'workspaceId', 'chatId'])
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.where('updatedAt', '<', staleBefore)
.limit(limit)
.execute();
}
async sweepRunning(
opts: { staleMs?: number } = {},
trx?: KyselyTransaction,
@@ -1,7 +1,7 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx, executeTx } from '../../utils';
import { dbOrTx, executeTx, registerAfterCommit } from '../../utils';
import {
InsertablePage,
Page,
@@ -349,14 +349,23 @@ export class PageRepo {
pageId: string,
deletedById: string,
workspaceId: string,
// Optional caller transaction. When passed, the reads + soft-delete run in
// THAT transaction (so a caller holding a `FOR UPDATE` lock on the row — e.g.
// the temporary-note sweeper — can delete under the lock without deadlocking
// on a nested independent transaction) and the PAGE_SOFT_DELETED broadcast is
// deferred to the caller's COMMIT via registerAfterCommit (so a rolled-back
// delete never broadcasts). With no trx the behaviour is unchanged: own
// transaction, broadcast right after it commits.
existingTrx?: KyselyTransaction,
): Promise<void> {
const currentDate = new Date();
const readDb = dbOrTx(this.db, existingTrx);
// Read the root snapshot up front so PAGE_SOFT_DELETED can carry it without
// a post-commit DB read (variant A). Only the root of the deleted subtree is
// needed for the tree broadcast — the client `treeModel.remove` drops all
// descendants, so we don't snapshot/broadcast every descendant.
const rootSnapshot = await this.db
const rootSnapshot = await readDb
.selectFrom('pages')
.select([
'id',
@@ -371,7 +380,7 @@ export class PageRepo {
.where('deletedAt', 'is', null)
.executeTakeFirst();
const descendants = await this.db
const descendants = await readDb
.withRecursive('page_descendants', (db) =>
db
.selectFrom('pages')
@@ -393,39 +402,60 @@ export class PageRepo {
const pageIds = descendants.map((d) => d.id);
if (pageIds.length > 0) {
await executeTx(this.db, async (trx) => {
await trx
.updateTable('pages')
.set({
deletedById: deletedById,
deletedAt: currentDate,
})
.where('id', 'in', pageIds)
.where('deletedAt', 'is', null)
.execute();
// Reuse the caller's transaction when given (executeTx passes it straight
// through), else own a fresh one.
await executeTx(
this.db,
async (trx) => {
await trx
.updateTable('pages')
.set({
deletedById: deletedById,
deletedAt: currentDate,
})
.where('id', 'in', pageIds)
.where('deletedAt', 'is', null)
.execute();
await trx.deleteFrom('shares').where('pageId', 'in', pageIds).execute();
});
await trx
.deleteFrom('shares')
.where('pageId', 'in', pageIds)
.execute();
},
existingTrx,
);
this.eventEmitter.emit(EventName.PAGE_SOFT_DELETED, {
pageIds: pageIds,
workspaceId,
// Root-only snapshot: one `deleteTreeNode` is enough, the client removes
// the whole subtree. Skip if the root vanished between the two reads.
pages: rootSnapshot
? [
{
id: rootSnapshot.id,
slugId: rootSnapshot.slugId,
title: rootSnapshot.title,
icon: rootSnapshot.icon,
position: rootSnapshot.position,
spaceId: rootSnapshot.spaceId,
parentPageId: rootSnapshot.parentPageId,
},
]
: [],
});
const emitSoftDeleted = () => {
this.eventEmitter.emit(EventName.PAGE_SOFT_DELETED, {
pageIds: pageIds,
workspaceId,
// Root-only snapshot: one `deleteTreeNode` is enough, the client
// removes the whole subtree. Skip if the root vanished between reads.
pages: rootSnapshot
? [
{
id: rootSnapshot.id,
slugId: rootSnapshot.slugId,
title: rootSnapshot.title,
icon: rootSnapshot.icon,
position: rootSnapshot.position,
spaceId: rootSnapshot.spaceId,
parentPageId: rootSnapshot.parentPageId,
},
]
: [],
});
};
if (existingTrx) {
// Inside a caller transaction: the delete above is NOT committed yet.
// Defer the tree broadcast to the caller's commit so a rolled-back delete
// never broadcasts a phantom removal.
registerAfterCommit(existingTrx, emitSoftDeleted);
} else {
// Own transaction already committed above — broadcast now.
emitSoftDeleted();
}
}
}
@@ -3,7 +3,7 @@ import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Cache } from 'cache-manager';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import { dbOrTx, registerAfterCommit } from '../../utils';
import {
InsertableWorkspace,
UpdatableWorkspace,
@@ -80,16 +80,30 @@ export class WorkspaceRepo {
*/
private async bustWorkspaceCache(
workspace?: Pick<Workspace, 'hostname'> | undefined,
trx?: KyselyTransaction,
): Promise<void> {
try {
await this.cacheManager.del(CacheKey.WORKSPACE_SELF_HOSTED);
if (workspace?.hostname) {
await this.cacheManager.del(
CacheKey.WORKSPACE_BY_HOST(workspace.hostname),
);
const del = async () => {
try {
await this.cacheManager.del(CacheKey.WORKSPACE_SELF_HOSTED);
if (workspace?.hostname) {
await this.cacheManager.del(
CacheKey.WORKSPACE_BY_HOST(workspace.hostname),
);
}
} catch {
// cache is best-effort; TTL is the backstop
}
} catch {
// cache is best-effort; TTL is the backstop
};
if (trx) {
// Inside a caller transaction the write is NOT yet committed: busting now
// opens a repopulation window (a concurrent reader reloads the cache with
// the pre-commit / stale row, which then survives until TTL). Defer the del
// to the transaction's commit (drained by the owning executeTx) (#495).
registerAfterCommit(trx, del);
} else {
// No transaction: the mutation above already auto-committed, so this del is
// already post-commit.
await del();
}
}
@@ -180,7 +194,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -195,7 +209,7 @@ export class WorkspaceRepo {
.returning(this.baseFields)
.executeTakeFirst();
// Bust the cached "not found" so a fresh install / new tenant is seen at once.
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -249,7 +263,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -271,7 +285,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -326,7 +340,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -354,7 +368,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -376,7 +390,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -398,7 +412,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -4,6 +4,7 @@ import { promises as fs } from 'fs';
import { Migrator, FileMigrationProvider } from 'kysely';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { ensureConcurrentIndexes } from '@docmost/db/concurrent-indexes';
@Injectable()
export class MigrationService {
@@ -12,6 +13,16 @@ export class MigrationService {
constructor(@InjectKysely() private readonly db: KyselyDB) {}
async migrateToLatest(): Promise<void> {
// Build write-blocking trigram indexes CONCURRENTLY (no transaction) BEFORE
// the migrator runs, so the corresponding in-migration `CREATE INDEX IF NOT
// EXISTS` no-ops instead of taking a SHARE lock on `pages` during deploy
// (#495). Best-effort: on a fresh DB (no `pages`/`f_unaccent` yet) this is a
// no-op and the migrations build the index normally.
await ensureConcurrentIndexes(this.db, (message, error) => {
if (error) this.logger.warn(`${message}: ${String(error)}`);
else this.logger.log(message);
});
const migrator = new Migrator({
db: this.db,
provider: new FileMigrationProvider({
-3
View File
@@ -606,9 +606,6 @@ export interface AiChats {
// The document the chat was created in (open page at first message). NULL =>
// started outside any document. ON DELETE SET NULL on the page FK.
pageId: string | null;
// Chat-level metadata bag (#490). jsonb, defaulted to '{}'. First key:
// `activatedTools` — the deferred-tool activation set persisted across turns.
metadata: Generated<Json>;
createdAt: Generated<Timestamp>;
updatedAt: Generated<Timestamp>;
deletedAt: Timestamp | null;
+63 -3
View File
@@ -6,16 +6,76 @@ import { KyselyDB, KyselyTransaction } from './types/kysely.types';
* If an existing transaction is provided, it directly executes the callback with it.
* Otherwise, it starts a new transaction using the provided database instance and executes the callback within that transaction.
*/
/**
* Post-commit side-effect hooks, keyed by the transaction they were registered
* against. A WeakMap so an abandoned/never-drained transaction's entry is GC'd
* with the trx object (no leak). Used by {@link registerAfterCommit} /
* {@link executeTx}.
*/
const afterCommitHooks = new WeakMap<
KyselyTransaction,
Array<() => Promise<void> | void>
>();
/**
* Register a side effect to run ONLY AFTER the transaction that owns `trx`
* commits. THE fix for "bust the cache inside the open transaction" bugs: a
* cache-invalidation (or any read-your-write-visible side effect) done while the
* writing transaction is still open opens a window where a concurrent reader
* repopulates the cache with the PRE-COMMIT (stale) row, so after commit the
* cache holds the old value until its TTL. Deferring the effect to post-commit
* closes that window.
*
* The hook is drained by the OUTERMOST {@link executeTx} that actually owns
* (created) this transaction so registering against a passed-through
* `existingTrx` still fires at the real commit boundary, not at the inner call.
* NOTE: a hook registered against a transaction that was NOT created via
* `executeTx` (untracked) will never be drained always create transactions
* through `executeTx` when you rely on post-commit hooks.
*/
export function registerAfterCommit(
trx: KyselyTransaction,
hook: () => Promise<void> | void,
): void {
const existing = afterCommitHooks.get(trx);
if (existing) existing.push(hook);
else afterCommitHooks.set(trx, [hook]);
}
export async function executeTx<T>(
db: KyselyDB,
callback: (trx: KyselyTransaction) => Promise<T>,
existingTrx?: KyselyTransaction,
): Promise<T> {
if (existingTrx) {
return await callback(existingTrx); // Execute callback with existing transaction
} else {
return await db.transaction().execute((trx) => callback(trx)); // Start new transaction and execute callback
// Reuse the caller's transaction. Any post-commit hooks registered here are
// drained by the OUTER executeTx that created `existingTrx`, at the true
// commit boundary — so we must NOT drain them now.
return await callback(existingTrx);
}
// We OWN this transaction: run the body, then (only once it has COMMITTED)
// drain the post-commit hooks registered against it during the body.
let ownTrx: KyselyTransaction | undefined;
const result = await db.transaction().execute((trx) => {
ownTrx = trx;
return callback(trx);
});
if (ownTrx) {
const hooks = afterCommitHooks.get(ownTrx);
if (hooks) {
afterCommitHooks.delete(ownTrx);
for (const hook of hooks) {
// Best-effort: a failed side effect (e.g. a cache del) must not fail the
// already-committed transaction.
try {
await hook();
} catch {
// swallow — the durable write already committed
}
}
}
}
return result;
}
/*
@@ -99,10 +99,12 @@ describe('AiSettingsService.getMasked reindex progress', () => {
// actually pins the progress.total branch rather than coincidentally
// matching the DB fallback. With fix #1 the two sources agree in practice,
// but getMasked must still return progress.total when a record is active.
const startedAt = Date.now();
reindexProgress.get.mockResolvedValue({
total: 500,
done: 120,
startedAt: Date.now(),
startedAt,
runId: 'run-abc',
});
const masked = await service.getMasked(WORKSPACE_ID);
@@ -110,6 +112,10 @@ describe('AiSettingsService.getMasked reindex progress', () => {
expect(masked.indexedPages).toBe(120); // progress.done, not DB 478
expect(masked.totalPages).toBe(500); // progress.total, not DB 478
expect(masked.reindexing).toBe(true);
// The status payload must carry the run identity so the client can key its
// poll on it (a changed runId => a NEW run).
expect(masked.runId).toBe('run-abc');
expect(masked.reindexStartedAt).toBe(startedAt);
});
it('falls back to countIndexedPages when no reindex is active', async () => {
@@ -121,6 +127,10 @@ describe('AiSettingsService.getMasked reindex progress', () => {
expect(masked.indexedPages).toBe(478);
expect(masked.totalPages).toBe(478);
expect(masked.reindexing).toBe(false);
// No active run -> no run identity surfaced (the client keeps its prior
// steady-state behaviour).
expect(masked.runId).toBeUndefined();
expect(masked.reindexStartedAt).toBeUndefined();
});
});
@@ -245,9 +245,6 @@ export class AiSettingsService {
// Max context window for the chat header badge denominator. Stored as
// ::text; 0/unset/invalid = no limit (undefined).
chatContextWindow: parsePositiveInt(provider.chatContextWindow),
// RAW stored value (#490): the replay budgeter reads this to distinguish an
// explicit `0` (off-switch) from unset, which parsePositiveInt cannot.
chatContextWindowRaw: provider.chatContextWindow,
// Plain passthrough; getChatModel defaults unset to 'openai-compatible'.
chatApiStyle: provider.chatApiStyle,
// Cheap model id for the anonymous public-share assistant; reuses the chat
@@ -374,6 +371,12 @@ export class AiSettingsService {
totalPages,
// Optional hint for the client: a reindex run is currently in progress.
reindexing: progress != null,
// Per-run identity so the client can key its poll on a stable run id and
// reset its per-run state when a NEW run starts. Present only while a run
// is active; `runId` may be '' for a legacy/degraded record (the client
// treats that as "no identity").
runId: progress?.runId,
reindexStartedAt: progress?.startedAt,
};
}
@@ -129,12 +129,6 @@ const DEFAULT_MCP_STREAM_TIMEOUT_MS = 60_000;
/** Default total wall-clock cap for ONE external MCP tool call (2 min). */
const DEFAULT_MCP_CALL_TIMEOUT_MS = 120_000;
/**
* Default `bodyTimeout` for the EXTERNAL-MCP SSE transport (10 min) #489.
* Deliberately much LARGER than {@link DEFAULT_MCP_STREAM_TIMEOUT_MS}.
*/
const DEFAULT_MCP_SSE_BODY_TIMEOUT_MS = 600_000;
/**
* SILENCE timeout (ms) for EXTERNAL-MCP transport ONLY. Override with
* `AI_MCP_STREAM_TIMEOUT_MS`; a missing/invalid/non-positive value falls back to
@@ -170,26 +164,6 @@ export function mcpCallTimeoutMs(): number {
return positiveEnv('AI_MCP_CALL_TIMEOUT_MS', DEFAULT_MCP_CALL_TIMEOUT_MS);
}
/**
* `bodyTimeout` (ms) for the EXTERNAL-MCP **SSE** transport ONLY #489. Override
* with `AI_MCP_SSE_BODY_TIMEOUT_MS`; a missing/invalid/non-positive value falls
* back to {@link DEFAULT_MCP_SSE_BODY_TIMEOUT_MS} (10 min).
*
* The SSE transport holds ONE long-lived response body open across many tool
* calls, so undici's `bodyTimeout` (time between body bytes) counts the LEGITIMATE
* silence BETWEEN calls, not just a hung single call. At the tight HTTP silence
* timeout ({@link mcpStreamTimeoutMs}, 1 min) a normal >1-min gap between the
* model's tool calls would break the SSE socket, and the cache would then serve a
* dead client until TTL. So the SSE transport gets its OWN, RAISED bodyTimeout;
* the per-call total cap ({@link mcpCallTimeoutMs}) still bounds a single stuck
* call, and the app-level transport-error retry heals a socket that does break.
* The HTTP (streamable) transport keeps the tight timeout it opens a fresh
* request per call, so idle-between-calls does not apply there.
*/
export function mcpSseBodyTimeoutMs(): number {
return positiveEnv('AI_MCP_SSE_BODY_TIMEOUT_MS', DEFAULT_MCP_SSE_BODY_TIMEOUT_MS);
}
/**
* undici `Agent` options for streaming AI traffic the (generous, finite)
* silence timeouts plus the keep-alive recycle window. Shared by the chat
@@ -0,0 +1,86 @@
// `.provider` alone cannot prove the gemini/ollama chat factories were built
// with the instrumented streaming fetch — a regression dropping it (which drops
// them back to the global undici fetch: no keep-alive recycle, no reset retries,
// unbounded silence timeout; incident classes #140/#175/#310) would still pass.
// So mock the factories and assert the exact fetch argument. jest.mock is
// module-scoped, hence a dedicated file.
const mockGeminiModel = { provider: 'google.generative-ai', modelId: 'm' };
const mockOllamaModel = { provider: 'ollama.chat', modelId: 'm' };
// jest allows `mock`-prefixed vars inside a jest.mock factory.
const mockCreateGoogle = jest.fn((_settings: unknown) => () => mockGeminiModel);
const mockCreateOllama = jest.fn((_settings: unknown) => () => mockOllamaModel);
jest.mock('@ai-sdk/google', () => ({
createGoogleGenerativeAI: (settings: unknown) => mockCreateGoogle(settings),
}));
jest.mock('ai-sdk-ollama', () => ({
createOllama: (settings: unknown) => mockCreateOllama(settings),
}));
import { AiService } from './ai.service';
describe('AiService.getChatModel provider transport fetch (gemini/ollama)', () => {
function serviceWith(cfg: Record<string, unknown>) {
const aiSettings = {
resolve: jest.fn().mockResolvedValue(cfg),
};
return new AiService(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
aiSettings as any,
{ find: jest.fn() } as never,
{ decryptSecret: jest.fn() } as never,
);
}
beforeEach(() => {
mockCreateGoogle.mockClear();
mockCreateOllama.mockClear();
});
it('builds the gemini chat model with the instrumented streaming fetch', async () => {
await serviceWith({
driver: 'gemini',
chatModel: 'gemini-2.5-pro',
apiKey: 'the-key',
}).getChatModel('ws-1');
expect(mockCreateGoogle).toHaveBeenCalledTimes(1);
expect(mockCreateGoogle).toHaveBeenCalledWith(
expect.objectContaining({
apiKey: 'the-key',
fetch: expect.any(Function),
}),
);
});
it('builds the ollama chat model with the instrumented streaming fetch', async () => {
await serviceWith({
driver: 'ollama',
chatModel: 'llama3',
baseUrl: 'http://localhost:11434/api',
}).getChatModel('ws-1');
expect(mockCreateOllama).toHaveBeenCalledTimes(1);
expect(mockCreateOllama).toHaveBeenCalledWith(
expect.objectContaining({
baseURL: 'http://localhost:11434/api',
fetch: expect.any(Function),
}),
);
});
it('reuses ONE service-lifetime fetch instance across both providers', async () => {
const svc = serviceWith({
driver: 'gemini',
chatModel: 'gemini-2.5-pro',
apiKey: 'k',
});
await svc.getChatModel('ws-1');
const geminiFetch = mockCreateGoogle.mock.calls[0][0] as { fetch: unknown };
// Same instance on a second call — the fetch is held for the service
// lifetime to reuse the streaming dispatcher's connection pool.
await svc.getChatModel('ws-1');
const geminiFetch2 = mockCreateGoogle.mock.calls[1][0] as { fetch: unknown };
expect(geminiFetch.fetch).toBe(geminiFetch2.fetch);
});
});
+15 -3
View File
@@ -190,10 +190,22 @@ export class AiService {
}).chat(chatModel);
}
case 'gemini':
return createGoogleGenerativeAI({ apiKey })(chatModel);
// Route gemini through the same instrumented streaming fetch as openai
// (finite silence timeouts + keep-alive recycling + pre-response
// connection-reset retry). Without it the provider ran on the global
// undici fetch — no keep-alive recycle, no reset retries, default
// (unbounded silence) timeout — so incident classes #140/#175/#310 were
// reproducible for gemini too.
return createGoogleGenerativeAI({
apiKey,
fetch: this.aiProviderFetch,
})(chatModel);
case 'ollama':
// Ollama needs no API key.
return createOllama({ baseURL: baseUrl })(chatModel);
// Ollama needs no API key. Same transport hardening as above (#140/#175/#310).
return createOllama({
baseURL: baseUrl,
fetch: this.aiProviderFetch,
})(chatModel);
default:
throw new AiNotConfiguredException();
}
+10 -4
View File
@@ -105,10 +105,6 @@ export interface ResolvedAiConfig extends Partial<AiProviderSettings> {
// Max context window in tokens; surfaced to the chat header badge as the
// "current / max" denominator. 0/unset = no limit.
chatContextWindow?: number;
// RAW stored context window (::text), BEFORE parsePositiveInt collapses `0` and
// unset to `undefined`. The #490 replay budgeter needs the raw value to honor an
// explicit `0` off-switch distinctly from "unset -> flat default".
chatContextWindowRaw?: string | number;
// Cheap model id for the public-share assistant; reuses the chat creds.
publicShareChatModel?: string;
// Agent-role id whose persona the public-share assistant adopts (empty/unset
@@ -153,4 +149,14 @@ export interface MaskedAiSettings {
// True while a full workspace reindex is actively running (the counts above
// then reflect the live run progress rather than the steady-state DB count).
reindexing?: boolean;
// Identity of the ACTIVE reindex run (present only while `reindexing`). The
// client keys its poll on `runId`: a changed value means a NEW run (reset the
// per-run poll state it latched), the same value means the run it is already
// watching — removing the "same run or a fresh one?" ambiguity a stale
// pre-reindex snapshot otherwise causes. Absent/empty degrades gracefully.
runId?: string;
// Epoch-ms the active run started (present only while `reindexing`). Paired
// with `runId` so a run that restarts with the same (recycled) id is still
// seen as new.
reindexStartedAt?: number;
}
@@ -48,19 +48,38 @@ describe('EmbeddingReindexProgressService', () => {
}
describe('get', () => {
it('maps a valid hash to a ReindexProgress object', async () => {
it('maps a valid hash to a ReindexProgress object (incl. the run identity)', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({ total: '478', done: '120', startedAt: '1000' });
hgetall.mockResolvedValue({
total: '478',
done: '120',
startedAt: '1000',
runId: 'run-xyz',
});
const service = makeService(redis);
await expect(service.get(WORKSPACE_ID)).resolves.toEqual({
total: 478,
done: 120,
startedAt: 1000,
runId: 'run-xyz',
});
expect(hgetall).toHaveBeenCalledWith(KEY);
});
it('degrades a missing runId to an empty string (legacy/partial record)', async () => {
const { redis, hgetall } = makeRedis();
// A record written before runId existed: get() must still succeed and
// report runId='' so the client treats it as "no identity", never breaks.
hgetall.mockResolvedValue({ total: '10', done: '3', startedAt: '5' });
await expect(makeService(redis).get(WORKSPACE_ID)).resolves.toEqual({
total: 10,
done: 3,
startedAt: 5,
runId: '',
});
});
it('returns null for an empty hash (no record)', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({});
@@ -87,11 +106,17 @@ describe('EmbeddingReindexProgressService', () => {
it('coerces a non-finite startedAt to 0', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({ total: '10', done: '2', startedAt: 'nope' });
hgetall.mockResolvedValue({
total: '10',
done: '2',
startedAt: 'nope',
runId: 'run-1',
});
await expect(makeService(redis).get(WORKSPACE_ID)).resolves.toEqual({
total: 10,
done: 2,
startedAt: 0,
runId: 'run-1',
});
});
@@ -115,6 +140,21 @@ describe('EmbeddingReindexProgressService', () => {
expect(multiObj.exec).toHaveBeenCalledTimes(1);
});
it('mints a fresh non-empty runId into the record on each start', async () => {
const { redis, multiObj } = makeRedis();
const service = makeService(redis);
await service.start(WORKSPACE_ID, 1);
await service.start(WORKSPACE_ID, 1);
const firstRunId = multiObj.hset.mock.calls[0][1].runId;
const secondRunId = multiObj.hset.mock.calls[1][1].runId;
expect(typeof firstRunId).toBe('string');
expect(firstRunId).not.toBe('');
// Each run gets its OWN identity so the client can tell a re-trigger apart
// from the run it is already watching.
expect(secondRunId).not.toBe(firstRunId);
});
it('defaults the expire TTL to the full 1h record TTL', async () => {
const { redis, multiObj } = makeRedis();
await makeService(redis).start(WORKSPACE_ID, 478);
@@ -1,17 +1,28 @@
import { Injectable, Logger } from '@nestjs/common';
import { RedisService } from '@nestjs-labs/nestjs-ioredis';
import { randomUUID } from 'node:crypto';
import type { Redis } from 'ioredis';
/**
* Live progress of an in-flight workspace embeddings reindex run.
* `total` is the number of pages the run will process, `done` how many it has
* already processed (success OR handled failure), `startedAt` the epoch-ms the
* record was created.
* record was created, and `runId` a per-run identity minted at `start()`.
*
* `runId` gives each reindex run a stable identity so a poller can tell "same
* run I've been watching" from "a NEW run started" WITHOUT guessing from the
* progress counters (the ambiguity that a stale pre-reindex snapshot vs a fresh
* run otherwise causes the bug class fixed twice under #262). It is best-
* effort like the rest of this record: a record written before this field
* existed (or a Redis hiccup) yields an empty `runId`, which the client must
* treat as "no identity available" and degrade to its prior behaviour, never
* break.
*/
export interface ReindexProgress {
total: number;
done: number;
startedAt: number;
runId: string;
}
/** Redis key namespace for the per-workspace reindex-progress record. */
@@ -86,12 +97,18 @@ export class EmbeddingReindexProgressService {
): Promise<void> {
const key = this.key(workspaceId);
try {
// A fresh identity per run so the client poll can key on it: a changed
// runId means a genuinely NEW run (reset any latched per-run poll state),
// the same runId means the run the client is already watching. Best-effort
// like the counters — never surfaced to the user, only used to disambiguate.
const runId = randomUUID();
await this.redis
.multi()
.hset(key, {
total: String(total),
done: '0',
startedAt: String(Date.now()),
runId,
})
.expire(key, ttlSeconds)
.exec();
@@ -150,7 +167,15 @@ export class EmbeddingReindexProgressService {
const done = Number(data.done);
const startedAt = Number(data.startedAt);
if (!Number.isFinite(total) || !Number.isFinite(done)) return null;
return { total, done, startedAt: Number.isFinite(startedAt) ? startedAt : 0 };
// `runId` degrades gracefully: a pre-existing record (written before this
// field) or a stripped value reads as '' — the client treats that as "no
// identity" and keeps its prior behaviour rather than breaking the poll.
return {
total,
done,
startedAt: Number.isFinite(startedAt) ? startedAt : 0,
runId: typeof data.runId === 'string' ? data.runId : '',
};
} catch (err) {
this.logger.warn(
`reindex-progress read failed for workspace ${workspaceId}; ` +
@@ -1,305 +0,0 @@
import { Kysely } from 'kysely';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatRunService } from '../../src/core/ai-chat/ai-chat-run.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #487 commit 4 bidirectional reconcile + owner-write priority, real SQL.
*
* Proves the OBSERVABLE recovery properties against docmost_test:
* - the CONDITIONAL owner-write beats a reconcile stamp, and a stamp never
* clobbers a proper terminal row;
* - a LATE owner-finalize with real content OVERWRITES a reconcile 'aborted'
* stamp (finalizeFailed);
* - each reconcile clause (b message<-run, c stale-run, d historical row) settles
* the stuck row/run, and a LIVE run entry is never touched;
* - the "kill DB on finish" recovery: after the DB comes back, neither the
* message row nor the run row stays stuck.
*/
describe('#487 reconcile + owner-write priority [integration]', () => {
let db: Kysely<any>;
let messageRepo: AiChatMessageRepo;
let runRepo: AiChatRunRepo;
let runService: AiChatRunService;
let workspaceId: string;
let userId: string;
beforeAll(async () => {
db = getTestDb();
messageRepo = new AiChatMessageRepo(db as any);
runRepo = new AiChatRunRepo(db as any);
runService = new AiChatRunService(runRepo, { isCloud: () => false } as never);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
const newChat = async () =>
(await createChat(db, { workspaceId, creatorId: userId })).id;
const metaOf = async (id: string): Promise<Record<string, unknown> | null> => {
const row = await messageRepo.findById(id, workspaceId);
return (row?.metadata as Record<string, unknown> | null) ?? null;
};
it('owner finalizeOwner writes a streaming row and CLEARS finalizeFailed', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
const wrote = await messageRepo.finalizeOwner(m.id, workspaceId, {
content: 'final answer',
status: 'completed',
metadata: { parts: [{ type: 'text', text: 'final answer' }] },
} as never);
expect(wrote!.status).toBe('completed');
expect((await metaOf(m.id))?.finalizeFailed).toBeUndefined();
});
it('a reconcile stamp NEVER clobbers a proper terminal row (finalizeOwner is a no-op there)', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'completed',
content: 'real',
metadata: { parts: [] },
});
// The reconcile stamp is onlyIfStreaming -> no-op on a completed row.
const stamped = await messageRepo.stampTerminalIfStreaming(
m.id,
workspaceId,
'aborted',
);
expect(stamped).toBeUndefined();
expect((await messageRepo.findById(m.id, workspaceId))!.status).toBe(
'completed',
);
});
it('LATE owner-finalize with real content OVERWRITES a reconcile aborted stamp', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [{ type: 'text', text: 'partial' }] },
});
// Reconcile stamps it aborted + finalizeFailed (final text lived only in mem).
const stamped = await messageRepo.stampTerminalIfStreaming(
m.id,
workspaceId,
'aborted',
);
expect(stamped!.status).toBe('aborted');
expect((await metaOf(m.id))?.finalizeFailed).toBe(true);
// A LATE owner-write (finalizeFailed=true satisfies the OR) overwrites it with
// real content, clearing the flag — owner-write priority.
const wrote = await messageRepo.finalizeOwner(m.id, workspaceId, {
content: 'the real final answer',
status: 'completed',
metadata: { parts: [{ type: 'text', text: 'the real final answer' }] },
} as never);
expect(wrote!.status).toBe('completed');
expect(wrote!.content).toBe('the real final answer');
expect((await metaOf(m.id))?.finalizeFailed).toBeUndefined();
});
it('clause (c): a stale active run with NO live entry -> aborted; a LIVE entry is untouched', async () => {
// Stale run, NOT owned by this replica (no entry) -> reconcile aborts it.
const staleChat = await newChat();
const stale = await runRepo.insert({
chatId: staleChat,
workspaceId,
createdBy: userId,
status: 'running',
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', stale.id)
.execute();
// A live run OWNED by this replica (beginRun registers an in-memory entry),
// ALSO backdated stale — the "no entry" primary gate must protect it.
const liveChat = await newChat();
const live = await runService.beginRun({
chatId: liveChat,
workspaceId,
userId,
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', live.runId)
.execute();
const aborted = await runService.reconcileStaleRuns(15 * 60 * 1000);
expect(aborted).toBeGreaterThanOrEqual(1);
expect((await runRepo.findById(stale.id, workspaceId))!.status).toBe(
'aborted',
);
// The live entry is NEVER aborted, however stale its row looks.
expect((await runRepo.findById(live.runId, workspaceId))!.status).toBe(
'running',
);
expect(runService.isLocallyActive(live.runId)).toBe(true);
// cleanup the live run
await runService.finalizeRun(live.runId, workspaceId, 'aborted');
});
it('clause (b): a streaming message whose RUN is terminal is stamped by run status (succeeded -> aborted, NOT completed-empty)', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
// A SUCCEEDED run linked to the still-streaming message (the asymmetry).
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
assistantMessageId: msg.id,
});
await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'succeeded',
error: null,
});
const stuck = await messageRepo.findStreamingWithTerminalRun();
const mine = stuck.find((s) => s.messageId === msg.id);
expect(mine?.runStatus).toBe('succeeded');
// Reconcile clause (b): succeeded run -> message 'aborted' (NOT 'completed'),
// the final text lived only in memory (documented loss), +finalizeFailed.
const status = mine!.runStatus === 'failed' ? 'error' : 'aborted';
await messageRepo.stampTerminalIfStreaming(msg.id, workspaceId, status);
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
it('clause (d): a stale streaming row with NO active run on the chat -> aborted+finalizeFailed', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
const swept = await messageRepo.sweepStreamingWithoutActiveRun(
15 * 60 * 1000,
);
expect(swept).toBeGreaterThanOrEqual(1);
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
it('clause (d) is DOUBLE-GATED: a stale streaming row WITH an active run on the chat is left alone', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
// An ACTIVE run on the same chat -> clause (d) must NOT touch the message.
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
});
await messageRepo.sweepStreamingWithoutActiveRun(15 * 60 * 1000);
expect((await messageRepo.findById(msg.id, workspaceId))!.status).toBe(
'streaming',
);
await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'aborted',
error: null,
});
});
it('"kill DB on finish" recovery: after the DB is back, reconcile leaves NEITHER the row nor the run stuck', async () => {
// Simulate a process that seeded the assistant row + run, then died before
// finalizing EITHER (a mid-turn crash): a streaming message + a running run,
// both stale, with no in-memory entry (fresh service = fresh maps).
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [{ type: 'text', text: 'partial' }] },
});
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
assistantMessageId: msg.id,
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', run.id)
.execute();
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
// Reconcile (as the periodic job would): (c) aborts the orphan run, then
// (b) settles the message from the now-terminal run.
await runService.reconcileStaleRuns(15 * 60 * 1000);
const stuck = await messageRepo.findStreamingWithTerminalRun();
for (const s of stuck) {
const status = s.runStatus === 'failed' ? 'error' : 'aborted';
await messageRepo.stampTerminalIfStreaming(s.messageId, s.workspaceId, status);
}
// Neither is stuck: the run is terminal AND the message is terminal.
expect((await runRepo.findById(run.id, workspaceId))!.status).toBe('aborted');
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
});
@@ -281,52 +281,6 @@ describe('AiChatRun durable lifecycle [integration]', () => {
});
});
it('#487 finalizeIfActive is CONDITIONAL: a late terminal write cannot clobber the settled status (real SQL)', async () => {
const c = (await createChat(db, { workspaceId, creatorId: userId })).id;
const run = await runRepo.insert({
chatId: c,
workspaceId,
createdBy: userId,
status: 'running',
});
// First terminal write: the run IS active, so it flips + returns the row.
const first = await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'succeeded',
error: null,
});
expect(first!.status).toBe('succeeded');
expect(first!.finishedAt).toBeTruthy();
// A late/second writer tries to flip it to 'aborted' — the WHERE status IN
// ('pending','running') guard matches NOTHING now, so it is a benign no-op.
const second = await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'aborted',
error: 'late clobber attempt',
});
expect(second).toBeUndefined();
// The persisted terminal status is UNCHANGED — last-writer-wins is gone.
const row = await runRepo.findById(run.id, workspaceId);
expect(row!.status).toBe('succeeded');
expect(row!.error).toBeNull();
});
it('#487 double-settle through the service collapses to one write at the SQL gate', async () => {
const c = (await createChat(db, { workspaceId, creatorId: userId })).id;
const handle = await service.beginRun({ chatId: c, workspaceId, userId });
// First settle writes 'aborted' via the conditional write.
await service.finalizeRun(handle.runId, workspaceId, 'aborted');
// A late safety-net settle to 'error' is a no-op (row already terminal).
await service.finalizeRun(handle.runId, workspaceId, 'error', 'late');
const row = await runRepo.findById(handle.runId, workspaceId);
expect(row!.status).toBe('aborted');
expect(service.isLocallyActive(handle.runId)).toBe(false);
expect(service.hasZombie(handle.runId)).toBe(false);
});
it('sweepRunning() with NO args (boot sweep / variant C) aborts even a FRESH running run', async () => {
// F1/DECISION C at the SQL level: the unconditional boot sweep has NO
// staleness window, so a run updated just now (a fast restart) is settled too
@@ -33,6 +33,11 @@ describe('share_aliases one-per-page invariant [integration]', () => {
const pageRepo = {
findById: async (id: string) => ({ id, title: `title-${id}` }),
};
// The requester can view the target page (permissive), so the reassign 409 may
// include its title — these tests exercise the one-per-page invariant, not the
// #495 disclosure gate (that is unit-tested in share-alias.service.spec.ts).
const pageAccessService = { validateCanView: async () => {} };
const USER = { id: 'u-int' } as any;
beforeAll(async () => {
db = getTestDb();
@@ -41,6 +46,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
repo as any,
pageRepo as any,
{} as any, // shareService — unused by setAlias
pageAccessService as any,
db as any,
);
wsId = (await createWorkspace(db)).id;
@@ -188,6 +194,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId,
user: USER,
alias: 'te',
});
expect(first.alias).toBe('te');
@@ -196,6 +203,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId,
user: USER,
alias: 'ted',
});
// Same row id — a RENAME, not a new insert.
@@ -217,12 +225,14 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'hello',
});
const again = await service.setAlias({
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'hello',
});
expect(again.id).toBe(inserted.id);
@@ -244,6 +254,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
flakyRepo as any,
pageRepo as any,
{} as any,
pageAccessService as any,
db as any,
);
@@ -252,6 +263,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'rollback-me',
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -275,6 +287,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageA,
creatorId: null as any,
user: USER,
alias: 'shared',
});
@@ -283,6 +296,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared',
}),
).rejects.toBeInstanceOf(ConflictException);
@@ -291,6 +305,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared',
confirmReassign: true,
});
@@ -317,12 +332,14 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageA,
creatorId: null as any,
user: USER,
alias: 'shared-target',
});
await service.setAlias({
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'bee',
});
@@ -330,6 +347,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared-target',
confirmReassign: true,
});
-1
View File
@@ -22,7 +22,6 @@
"^@docmost/db/(.*)$": "<rootDir>/src/database/$1",
"^@docmost/transactional/(.*)$": "<rootDir>/src/integrations/transactional/$1",
"^@docmost/ee/(.*)$": "<rootDir>/src/ee/$1",
"^@docmost/token-estimate$": "<rootDir>/../../packages/token-estimate/src/index.ts",
"^src/(.*)$": "<rootDir>/src/$1"
}
}
+78 -174
View File
@@ -8,35 +8,19 @@ real pain (a "which tools fail most?" analysis that confidently answered
Read the **Gotchas** section before you trust any error count.
> **TWO ERAS — check the marker first.** The `tool_calls` shape changed in **#490
> (trace v2)**. A row written by v2 carries `metadata.toolTraceVersion = 2`; older
> rows have no such key. The two shapes store DIFFERENT things (v2 dropped the tool
> OUTPUT from the trace), so **every query below is dual-shape** — branch on the
> marker. **Never compare an aggregate or trend across the era boundary**: a metric
> jump on the cut-over week is an artifact of the shape change, not a behavior
> change.
## TL;DR
- Agent chats live in Postgres, DB `docmost`, tables `ai_chat_*`.
- **Era marker:** `metadata.toolTraceVersion = 2` ⇒ v2 (#490) row; absent ⇒ legacy row.
- Each tool invocation is stored as **two** consecutive array elements — a
`tool-call` part then an OUTCOME part — so naive counting double-counts.
- **v2 (#490):** outcome is `{toolName, ok: true}` on success, or
`{toolName, error, kind: 'thrown'|'interrupted'}` on failure. The tool **OUTPUT
is NOT in `tool_calls`** any more — it lives once in `metadata.parts` (this
removed a hundreds-of-MB-per-run write duplication). Soft-failure analysis
therefore reads `metadata.parts`, not `tool_calls`.
- **legacy:** outcome is `{toolName, output}` on success; a **thrown** failure is
a `{toolName, error}` element **only on rows after #407**, and is dropped
entirely (silent orphan) on pre-#407 rows.
- **A tool that *throws* writes no result part.** In v2 it is a
`{error, kind:'thrown'}` element; an interrupted/aborted call is a distinct
`{error, kind:'interrupted'}`. `isError`/`success=false` scans read the *output*
and so under-report thrown failures in every era.
- To find where agents fail: (1) soft-failure markers in `metadata.parts` outputs
(v2) / `tool_calls` outputs (legacy), (2) the `error`/`kind` fields for thrown
failures (v2 + post-#407), (3) server logs / the live UI for full stack traces.
- Each tool invocation is stored as **two** array elements (a `tool-call` part and
a `tool-result` part), so naive counting double-counts.
- **A tool that *throws* writes no result part.** Since the #407 fix its error is
persisted as a dedicated `{toolName, error}` element in `tool_calls` (queryable +
replayed to the model). **Rows written before #407 still drop it** — the error is
nowhere in the DB and shows only in the live UI. So `isError` / `success=false`
scans under-report by design, and pre-#407 thrown errors are invisible.
- To find where agents fail: (1) soft-failure markers in `tool_calls`, (2) the new
`error` field for thrown errors (new rows) / the orphan-gap proxy (old rows),
(3) server logs / the live UI for full stack traces beyond the truncated message.
## Where the data lives
@@ -69,67 +53,33 @@ are rows in `workspaces`, not separate deployments.
separate `tool` role), `content` (text), `tool_calls` (jsonb array), `metadata`
(jsonb, holds run `error` + rendered `parts`), `status`, `tsv` (full-text index).
## Era marker — check this before every query
```sql
-- how many rows are in each era?
SELECT COALESCE((metadata->>'toolTraceVersion'), 'legacy') AS era, count(*)
FROM ai_chat_messages
WHERE role = 'assistant' AND jsonb_typeof(tool_calls) = 'array'
GROUP BY 1 ORDER BY 2 DESC;
```
- `toolTraceVersion = '2'`**v2** (#490): outcome flags, **no output in the trace**.
- `NULL` (`'legacy'`) → pre-#490: outcome carries the tool `output` inline.
**Do not trend a metric across the cut-over.** The shape change alone shifts counts
(e.g. "elements with `output`" collapses to zero for v2), so a week that straddles
the boundary shows an artifact, not a behavior change. Segment by era, or restrict to
one era, before comparing.
## How tool calls are stored — READ THIS
Tool calls are **not** one-object-per-call. Each logical invocation is split into
two consecutive elements of the `tool_calls` array — a **call** then an **outcome**.
The outcome shape is era-dependent:
two consecutive elements of the `tool_calls` array:
```text
# v2 (#490) — metadata.toolTraceVersion = 2
index 0: { "toolName":"getPage", "input":{...} } ← call (has input)
index 1: { "toolName":"getPage", "ok":true } ← success (NO output here)
or : { "toolName":"getPage", "error":"…", "kind":"thrown" } ← threw
or : { "toolName":"getPage", "error":"…", "kind":"interrupted" } ← aborted mid-step
# legacy — no toolTraceVersion
index 0: { "toolName":"getPage", "input":{...} } ← call (has input, NO output)
index 1: { "toolName":"getPage", "output":{...} } ← success (has output)
or : { "toolName":"getPage", "error":"…" } ← threw (post-#407 only)
index 0: { "toolName": "getPage", "input": { "pageId": "…" } } ← tool-call (has input, NO output)
index 1: { "toolName": "getPage", "output": { … } } ← tool-result (has output, NO input)
```
The keys that can appear: `toolName`, `input` (call), and on the outcome — **v2:**
`ok` **or** `error`+`kind`; **legacy:** `output` **or** (post-#407) `error`. There is
no `state`, no `errorText`, no `type` in `tool_calls` (those live on `metadata.parts`).
Consequences:
The keys that appear on an element are `toolName`, `input`, `output`, and — for a
**thrown** failure on rows written after the #407 fix — `error` (the tool's error
message; see the "Hard failures" section below). There is no `state`, no `errorText`,
no `type`. On pre-#407 rows a thrown failure has NO paired result element at all
(silent orphan). Consequences:
1. **Real invocation count** — count the OUTCOME elements, not every element (else you
double-count): **v2** = elements with `ok` or `error`; **legacy** = elements with
`output` or `error`.
2. **Pairing:** a call (`input`) is followed by its outcome. `toolName` is on both, so
you can group by tool on either. In v2 the `kind` field separates a real hard-fail
(`thrown`) from an aborted call (`interrupted`) — a distinction legacy rows cannot
make (both are orphans; see below).
3. **The tool OUTPUT is only in `metadata.parts` on v2 rows.** To inspect what a tool
returned (soft-error markers, page bodies) on a v2 row, read the parts
(`part->>'type' LIKE 'tool-%'`, `part->>'state' = 'output-available'`, `part->'output'`),
not `tool_calls`.
1. **Real invocation count = elements that have `output` or `error`.** Counting every
element double-counts (you get ~2× and a spurious "~50% of every tool has no output").
2. **Pairing:** a call = a `tool-call` part followed by its result part. A success
carries `output`; a thrown failure (post-#407) carries `error` instead. Both carry
`toolName`, so you can group by tool on either.
## The two classes of failure (and which the DB can see)
### 1. Soft failures — tool RAN and returned an error-shaped result → PERSISTED ✅
These are visible in the tool `output`**on v2 rows in `metadata.parts`** (the
`output-available` part's `output`), on **legacy rows in the `tool_calls` outcome
element's `output`**. The marker differs per tool:
These are visible in the `tool-result` `output`. The marker differs per tool:
| Tool(s) | Error marker in `output` |
| --- | --- |
@@ -141,32 +91,37 @@ element's `output`**. The marker differs per tool:
Note `editPageText` returns `failed: []` on success — filtering on the *presence*
of the key gives false positives; filter on **non-empty**.
### 2. Hard failures — tool THREW → PERSISTED ✅
### 2. Hard failures — tool THREW → NOW PERSISTED ✅ (since the #407 fix)
When a tool throws (the classic one is `patchNode` / `insertNode` / `tableUpdateCell`
`Failed to encode document to Yjs (fromJSON): Unknown node type: undefined`), the
runtime writes **no `tool-result` part** — the failure is an ai@6 `tool-error` content
part. How that lands in `tool_calls` depends on the era:
runtime still writes **no `tool-result` part** — the failure is an ai@6 `tool-error`
content part instead. **Since the #407 fix, that error is persisted**: `serializeSteps`
appends a dedicated element `{toolName, error: "<message>"}` right after the failed
call, mirroring how a successful `{toolName, output}` element is appended. So a thrown
error now leaves a queryable `error` field carrying its (truncated) reason, and the
same real text is replayed to the model on the next turn (an `output-error` part with
the real `errorText`, no longer the `'Tool call did not complete.'` placeholder).
- **v2 (#490):** a `{toolName, error, kind:'thrown'}` outcome element. An interrupted /
aborted mid-step call is a **distinct** `{toolName, error:'Tool call did not
complete.', kind:'interrupted'}` element — so you can tell a real hard-fail from an
abort **directly, without the orphan heuristic**. Query `kind = 'thrown'`.
- **post-#407 legacy:** a `{toolName, error}` element (no `kind`) right after the call.
- **pre-#407 legacy:** the error is **dropped** — a silent **orphan** (a `call` with no
`output` *and* no `error`).
**Cutover caveat — old rows keep the old blind shape.** Rows written **before** this
change have the two-part shape (`call` + `output` only) and simply **drop** thrown
errors, leaving a silent **orphan** (a `call` with no `output` *and* no `error`). Rows
written **after** the fix additionally carry the `error` element. So:
The same real error text is replayed to the model on the next turn (an `output-error`
part with the real `errorText`, from `metadata.parts`), in every era.
- **New rows:** query the `error` field directly (see the hard-error query below) — no
orphan heuristic needed for thrown failures.
- **Old rows (pre-#407):** the only DB-side proxy is still an **orphan**: a `tool-call`
part with no matching `tool-result` *and* no `error`. Orphans also appear when a run
is **aborted** mid-flight (server restart), so a high-volume tool (`createComment`,
`searchInPage`, `Search_web_search`) shows orphans from aborts, not real errors on
old rows. Treat the orphan gap as an *upper bound*, and cross-check the tool: a gap on
a structural editor (`patchNode`, `insertNode`, `updatePageJson`, `transformPage`) is
almost certainly a thrown Yjs-encode error; a gap on `createComment` is mostly aborts.
**Cutover caveat.** Only pre-#407 legacy rows need the orphan proxy: an orphan is a
`tool-call` with no matching outcome. Orphans there also appear when a run is **aborted**
mid-flight (server restart), so a high-volume tool (`createComment`, `searchInPage`,
`Search_web_search`) shows orphans from aborts, not real errors. Treat the orphan gap as
an *upper bound* and cross-check the tool: a gap on a structural editor (`patchNode`,
`insertNode`, `updatePageJson`, `transformPage`) is almost certainly a thrown Yjs-encode
error; a gap on `createComment` is mostly aborts. **On v2 rows this ambiguity is gone**
`kind` labels each outcome.
A note on the aborted-call fallback: a call with **neither** a result **nor** a
`tool-error` (genuinely interrupted mid-step) still replays with the
`'Tool call did not complete.'` placeholder and persists as an orphan — that path is
unchanged, and is distinct from a real thrown error, which now carries `error`.
### 3. Run-level failures → `ai_chat_runs`
@@ -179,34 +134,22 @@ the wild: `Run interrupted by a server restart.` (aborts) and
Run all of these via `docker exec gitmost-postgresql psql -U docmost -d docmost -P pager=off -c "…"`.
**Real invocation count per tool** (outcome parts only — the correct denominator).
Dual-shape: a v2 outcome has `ok` or `error`; a legacy outcome has `output` or `error`:
**Real invocation count per tool** (result parts only — the correct denominator):
```sql
SELECT elem->>'toolName' AS tool, count(*) AS calls
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array'
AND (elem ? 'ok' OR elem ? 'output' OR elem ? 'error')
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
GROUP BY 1 ORDER BY 2 DESC;
```
**Soft errors per tool.** The soft-error marker lives in the tool OUTPUT — which on
**v2 rows is in `metadata.parts`**, on **legacy rows is in the `tool_calls` outcome
element**. This query UNIONs both eras, projecting each output as `o`:
**Soft errors per tool** (everything the DB can honestly see):
```sql
WITH res AS (
-- v2 (#490): output is in metadata.parts (output-available tool parts)
SELECT part->>'type' AS tool, part->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.metadata->'parts') part
WHERE (m.metadata->>'toolTraceVersion') = '2'
AND part->>'type' LIKE 'tool-%' AND part->>'state' = 'output-available'
UNION ALL
-- legacy: output is inline in the tool_calls outcome element
SELECT elem->>'toolName' AS tool, elem->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE (m.metadata->>'toolTraceVersion') IS NULL
AND jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
)
SELECT tool, count(*) AS calls,
sum(COALESCE(
@@ -224,23 +167,13 @@ FROM res GROUP BY tool HAVING sum(COALESCE(
ORDER BY soft_errors DESC;
```
Note the v2 `tool` label is the part type (`tool-editPageText`); strip the `tool-`
prefix if you join it against the legacy `toolName`.
**`editPageText` failure reasons** (the most common real agent mistake — bad `find`).
Same dual-shape output source:
**`editPageText` failure reasons** (the most common real agent mistake — bad `find`):
```sql
WITH res AS (
SELECT part->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.metadata->'parts') part
WHERE (m.metadata->>'toolTraceVersion') = '2'
AND part->>'type' = 'tool-editPageText' AND part->>'state' = 'output-available'
UNION ALL
SELECT elem->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE (m.metadata->>'toolTraceVersion') IS NULL
AND jsonb_typeof(m.tool_calls) = 'array'
WHERE jsonb_typeof(m.tool_calls) = 'array'
AND elem->>'toolName' = 'editPageText' AND elem ? 'output'
)
SELECT f->>'reason' AS reason, count(*)
@@ -249,43 +182,30 @@ WHERE jsonb_typeof(o->'failed') = 'array'
GROUP BY 1 ORDER BY 2 DESC;
```
**Hard errors — persisted `error` field per tool (v2 + post-#407 rows)** — thrown tool
failures carry their real reason, so query them directly. On **v2** rows exclude the
`interrupted` kind so an aborted call is not counted as a hard-fail:
**Hard errors — persisted `error` field per tool (NEW rows, since #407)** — thrown
tool failures now carry their real reason, so query them directly:
```sql
SELECT elem->>'toolName' AS tool, count(*) AS thrown_errors,
min(elem->>'error') AS sample_error
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'error'
-- v2 rows label the kind; a legacy error element has no kind (count it).
AND COALESCE(elem->>'kind', 'thrown') = 'thrown'
GROUP BY 1 ORDER BY 2 DESC;
```
Aborted mid-step calls on v2 rows are a distinct, directly countable population:
```sql
SELECT elem->>'toolName' AS tool, count(*) AS interrupted
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem->>'kind' = 'interrupted'
GROUP BY 1 ORDER BY 2 DESC;
```
**Hard-error proxy for OLD rows (pre-#407) — orphan gap per tool, WITH a spread column**
(call parts minus outcome parts, plus how many distinct chats the gap is spread across).
This is needed ONLY for pre-#407 legacy rows (v2 and post-#407 rows carry the error /
`kind` directly — use the queries above). The `WHERE` restricts to the legacy era so v2
rows (where an `ok` outcome is not an `output`) never produce phantom orphans:
(call parts minus result parts, plus how many distinct chats the gap is spread across).
This covers rows written before thrown errors were persisted; on new rows a thrown
failure now has its own `error` element (use the query above) and an orphan means only
a genuinely aborted mid-step call:
```sql
WITH parts AS (
SELECT m.chat_id, elem->>'toolName' AS tool,
(elem ? 'input' AND NOT (elem ? 'output') AND NOT (elem ? 'ok')) AS is_call,
(elem ? 'output' OR elem ? 'error' OR elem ? 'ok') AS is_result
(elem ? 'input' AND NOT (elem ? 'output')) AS is_call,
(elem ? 'output' OR elem ? 'error') AS is_result
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND m.role = 'assistant'
AND (m.metadata->>'toolTraceVersion') IS NULL
),
per_chat AS (
SELECT tool, chat_id, sum(is_call::int) - sum(is_result::int) AS gap
@@ -341,21 +261,11 @@ WHERE tsv @@ websearch_to_tsquery('english', 'some phrase') LIMIT 20;
## Don't blow up your context
Tool outputs embed full page content and search payloads (hundreds of KB per row).
On **legacy** rows they are in `tool_calls`; on **v2** rows they moved to
`metadata->'parts'` (the `tool_calls` trace itself is now small). Never `SELECT
tool_calls` / `metadata` (or `jsonb_pretty(...)`) raw — project just the keys you need
and truncate:
A single `tool_calls` row can be **300–400 KB** (results embed full page content and
search payloads). Never `SELECT tool_calls` (or `jsonb_pretty(tool_calls)`) raw.
Always project just the keys you need and truncate:
```sql
-- v2: outputs live in metadata.parts
SELECT part->>'type',
left(regexp_replace((part->'output')::text, '\s+', ' ', 'g'), 200)
FROM ai_chat_messages m, jsonb_array_elements(m.metadata->'parts') part
WHERE (m.metadata->>'toolTraceVersion') = '2'
AND part->>'state' = 'output-available' LIMIT 5;
-- legacy: outputs live in tool_calls
SELECT elem->>'toolName',
left(regexp_replace((elem->'output')::text, '\s+', ' ', 'g'), 200)
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
@@ -370,32 +280,26 @@ docker compose -p gitmost logs -f --tail=100 # whole stack
```
Logging is `json-file`, `max-size=10m max-file=5` → ~50 MB retained, then rotated,
and **wiped on container recreate**. Thrown-tool error text is **persisted** — in the
`error` field of `tool_calls` (v2 `kind:'thrown'` / post-#407 legacy) — so you no longer
depend on live logs for it. Logs/live UI remain useful for **pre-#407 rows** (whose
thrown errors were dropped) and for full stack traces beyond the truncated stored
message. A per-tool `tool_calls_total{tool,status}` metric to VictoriaMetrics is still a
possible future add for aggregate dashboards.
and **wiped on container recreate**. Since the #407 fix, thrown-tool error text is
**persisted in the `error` field** of `tool_calls` (see the hard-error query above), so
you no longer depend on live logs for it. Logs/live UI remain useful for **pre-#407
rows** (whose thrown errors were dropped) and for full stack traces beyond the
truncated stored message. A per-tool `tool_calls_total{tool,status}` metric to
VictoriaMetrics is still a possible future add for aggregate dashboards.
## Gotchas checklist
- [ ] **Check `metadata.toolTraceVersion` first.** v2 (`= 2`) has no output in `tool_calls`; legacy has it inline. Never trend a metric across the era boundary.
- [ ] Counting every `tool_calls` element → **overcount**. Count OUTCOME elements — v2: `ok` or `error`; legacy: `output` or `error` — never both call+outcome as invocations.
- [ ] `isError` / `success=false` ≈ 0 does **not** mean "no errors" — thrown errors are an `error` element (v2 `kind:'thrown'` / post-#407), not in the output.
- [ ] **v2:** soft-error markers (the tool output) are in `metadata.parts`, NOT `tool_calls`. Legacy: they are in the `tool_calls` outcome `output`.
- [ ] **v2:** `kind` splits a real hard-fail (`thrown`) from an aborted call (`interrupted`) directly — no orphan heuristic needed. The orphan gap is a pre-#407-legacy-only proxy.
- [ ] Counting every `tool_calls` element → **overcount**. Count `output` elements; add `error` elements for thrown failures (new rows), but don't count both as invocations.
- [ ] `isError` / `success=false` ≈ 0 does **not** mean "no errors" — thrown errors are a separate `error` element (new rows) or dropped entirely (pre-#407 rows).
- [ ] Thrown errors persist only on rows written **after the #407 fix** — pre-#407 rows still drop them (orphan only). Mind the cutover when trending over time.
- [ ] `editPageText.failed` is `[]` on success — test for **non-empty**, not presence.
- [ ] Orphan gap on OLD rows mixes thrown errors **and** aborted runs — split by tool. On NEW rows a thrown error is its own `error` element, so a gap ≈ aborted call.
- [ ] `aborted` runs = server restarts, `failed` runs = provider overload — not agent mistakes.
- [ ] Never dump a raw `tool_calls` **or** `metadata.parts` cell — outputs are hundreds of KB.
- [ ] Logs are ephemeral (≤50 MB, wiped on recreate) — grab pre-#407 hard-error text live.
- [ ] Never dump a raw `tool_calls` cell — it can be hundreds of KB.
- [ ] Logs are ephemeral (≤50 MB, wiped on recreate) — grab hard-error text live.
## Snapshot (2026-07-07, illustrative — rerun the queries for current numbers)
> All rows in this snapshot predate #490, so they are **legacy-era** (outputs inline in
> `tool_calls`, orphan proxy for thrown errors). Do not trend these numbers against v2
> rows — segment by `toolTraceVersion` first.
- 226 chats, 732 messages, 46 runs; ~4 400 real tool invocations.
- Soft errors (persisted): `editPageText` 4/79 (bad/non-unique `find`) + 9 markdown-in-`find` warnings; `semanticSearch` 3/4 (`unavailable`); `Habr_update_draft_from_docmost` 1/2 (`doc` sent as object, not string).
- Missing-result proxy, read WITH the spread column:
+13
View File
@@ -33,6 +33,19 @@ import { TransformsMixin, type ITransformsMixin } from "./client/transforms.js";
export type { DocmostMcpConfig, SandboxPut } from "./client/context.js";
export { formatDocmostAxiosError, assertFullUuid } from "./client/errors.js";
// Branded page-identity types + validating constructors/guards (#435): a page's
// internal UUID (`PageId`) and public slug (`SlugId`) are distinct nominal types
// so the two can't be swapped as bare strings. Re-exported on the package
// surface so hosts/tests can validate + brand identities at their boundaries.
export type { PageId, SlugId, PageRef } from "./lib/page-id.js";
export {
asPageId,
asSlugId,
isPageId,
isSlugId,
SLUG_ID_RE,
} from "./lib/page-id.js";
// The full public + shared instance surface of the assembled client. Built by
// INTERSECTING each domain mixin's public interface (each DERIVED from its class
// and enforced by that class's `implements` clause — issue #446, no hand-mirror)
+20 -58
View File
@@ -59,39 +59,6 @@ import {
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Max concurrent per-page comment fetches in checkNewComments (#490). The scan is
// O(N) independent REST reads over the working set; running them one-at-a-time made
// a large space linear in round-trips. A small cap parallelizes without hammering
// the server (or exhausting sockets). 6 is a conservative middle of the 5–8 band.
const COMMENT_SCAN_CONCURRENCY = 6;
/**
* Map `items` through `fn` with at most `limit` in flight, preserving INPUT ORDER
* in the returned array. A tiny bounded pool (no p-limit dependency): `limit`
* workers pull the next index off a shared cursor until the list is drained.
*/
async function mapWithConcurrency<T, R>(
items: readonly T[],
limit: number,
fn: (item: T, index: number) => Promise<R>,
): Promise<R[]> {
const results = new Array<R>(items.length);
let cursor = 0;
const worker = async (): Promise<void> => {
for (;;) {
const i = cursor++;
if (i >= items.length) return;
results[i] = await fn(items[i], i);
}
};
const workers = Array.from(
{ length: Math.max(1, Math.min(limit, items.length)) },
() => worker(),
);
await Promise.all(workers);
return results;
}
// Public method surface of CommentsMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
@@ -688,32 +655,27 @@ export function CommentsMixin<TBase extends GConstructor<DocmostClientContext>>(
parentPageId,
);
// 2. Fetch comments for each page, keep ones created after since. Runs with
// bounded concurrency (#490) instead of one-at-a-time — the per-page reads are
// independent, so a large working set no longer costs O(N) serial round-trips.
// Order is preserved (mapWithConcurrency keeps input order), so the output is
// deterministic regardless of which fetch finishes first.
const perPage = await mapWithConcurrency(
pagesInScope,
COMMENT_SCAN_CONCURRENCY,
async (page: any) => {
try {
// Full feed (incl. resolved): a "new comments since" scan reports all
// recent activity; the active-only filter is scoped to listComments.
const comments = (await this.listComments(page.id, true)).items;
const newComments = comments.filter(
(c: any) => new Date(c.createdAt) > sinceDate,
);
return newComments.length > 0
? { pageId: page.id, pageTitle: page.title, comments: newComments }
: null;
} catch (e: any) {
// Skip pages with errors (e.g. deleted between calls)
return null;
// 2. Fetch comments for each page, keep ones created after since
const results: any[] = [];
for (const page of pagesInScope) {
try {
// Full feed (incl. resolved): a "new comments since" scan reports all
// recent activity; the active-only filter is scoped to listComments.
const comments = (await this.listComments(page.id, true)).items;
const newComments = comments.filter(
(c: any) => new Date(c.createdAt) > sinceDate,
);
if (newComments.length > 0) {
results.push({
pageId: page.id,
pageTitle: page.title,
comments: newComments,
});
}
},
);
const results: any[] = perPage.filter((r): r is any => r !== null);
} catch (e: any) {
// Skip pages with errors (e.g. deleted between calls)
}
}
const totalNewComments = results.reduce(
(sum, r) => sum + r.comments.length,
+14 -67
View File
@@ -24,6 +24,7 @@ import {
isCollabAuthFailedError,
} from "../lib/collab-session.js";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import type { PageId } from "../lib/page-id.js";
import { getCollabToken, performLogin } from "../lib/auth-utils.js";
import { formatDocmostAxiosError } from "./errors.js";
import { GetPageConversionCache } from "./getpage-cache.js";
@@ -170,43 +171,6 @@ export abstract class DocmostClientContext {
// cached conversion can never leak across identities. See getpage-cache.ts.
protected getPageCache = new GetPageConversionCache();
// #487: an OPTIONAL abort signal the in-app tool host sets before each tool
// call (a composite of the turn's Stop signal + a per-call wall-clock cap). It
// is checked at safe-points BETWEEN the sequential HTTP calls of a paginated
// read (paginateAll) and just before the atomic collab commit of a write (the
// mutatePage/replacePage/mutateLiveContentUnlocked seams), so a Stop / cap
// stops the NEXT network call from STARTING. An already-started single call may
// still land — a documented limitation (#487).
//
// SINGLE-WRITER by phase-1 assumption: exactly one DocmostClient is built per
// turn and shared by every tool call; the host sets this per call and does NOT
// restore the prior value on unwind (set-and-leave) — a fresh client per turn
// plus overwrite-by-the-next-call keeps it correct, and leaving a settled
// call's signal in place is what makes a discarded race-loser throw on its
// next safe-point. If the model emits PARALLEL in-app
// tool calls they share this one field, so the per-call CAP of one call is not
// guaranteed to bound another's in-flight pagination — but every composite the
// host sets carries the SAME turn Stop signal, so a Stop still aborts whichever
// signal is current. #487.
protected toolAbortSignal: AbortSignal | null = null;
/**
* #487: set (or clear with null) the in-app tool abort signal governing the
* NEXT client call's safe-points. The host wraps each in-app tool call: it sets
* the composite (Stop + per-call cap) here before invoking the tool and leaves
* it in place afterwards (set-and-leave, NOT restored) the next call
* overwrites it, and a fresh client is built per turn. Public so the
* server-side tool wrapper can reach it; harmless (a no-op) when never set.
*/
public setToolAbortSignal(signal: AbortSignal | null): void {
this.toolAbortSignal = signal;
}
/** #487: the abort signal currently governing this client's safe-points. */
public getToolAbortSignal(): AbortSignal | null {
return this.toolAbortSignal;
}
// Two construction forms:
// - new DocmostClient(config) // discriminated union (current)
// - new DocmostClient(baseURL, email, password) // legacy positional creds
@@ -608,10 +572,6 @@ export abstract class DocmostClientContext {
this.onMetricFn?.("collab_connect_timeouts_total", 1),
});
try {
// #487 PRE-COMMIT safe-point (reentrant twin of mutatePageContent): a
// Stop/cap after acquiring the session but before the atomic write skips
// this commit. Same limitation applies (stops the NEXT commit only).
this.toolAbortSignal?.throwIfAborted();
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh.
@@ -643,11 +603,6 @@ export abstract class DocmostClientContext {
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
// #487 safe-point: a Stop (or the in-app tool per-call cap) that fires
// BETWEEN sequential page fetches must stop the NEXT request from starting
// — a read tool that would otherwise paginate for minutes is interrupted
// here. throwIfAborted() rejects with the signal's reason.
this.toolAbortSignal?.throwIfAborted();
const payload: Record<string, any> = {
...basePayload,
limit: clampedLimit,
@@ -715,10 +670,17 @@ export abstract class DocmostClientContext {
* once via getPageRaw and cached (both slugId->uuid and uuid->uuid), so
* repeated edits on the same page add no extra request.
*/
protected async resolvePageId(pageId: string): Promise<string> {
if (isUuid(pageId)) return pageId;
protected async resolvePageId(pageId: string): Promise<PageId> {
// This is the ONE canonicalization seam, so it is where the `PageId` brand
// is minted (#435). The value is validated here — a UUID input by isUuid, a
// resolved id as the server's own page.id — so the downstream write path
// (withPageLock / mutatePageContent) can require the brand and reject any
// unresolved raw id at compile time. The brand is applied by cast, not the
// asPageId() validator, to avoid rejecting the fake ids the mock tests feed
// a server stub; asPageId() guards the untrusted PUBLIC boundary instead.
if (isUuid(pageId)) return pageId as PageId;
const cached = this.pageIdCache.get(pageId);
if (cached) return cached;
if (cached) return cached as PageId;
const data = await this.getPageRaw(pageId);
const uuid = data?.id;
if (typeof uuid !== "string" || !uuid) {
@@ -727,7 +689,7 @@ export abstract class DocmostClientContext {
);
}
this.pageIdCache.set(pageId, uuid);
return uuid;
return uuid as PageId;
}
@@ -755,15 +717,7 @@ export abstract class DocmostClientContext {
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
// #487: thread the in-app tool signal to mutatePageContent's pre-commit
// safe-point so a Stop/cap during the connect/lock window skips the write.
mutatePageContent(
pageUuid,
token,
apiUrl,
transform,
this.toolAbortSignal ?? undefined,
),
mutatePageContent(pageUuid, token, apiUrl, transform),
);
}
@@ -787,14 +741,7 @@ export abstract class DocmostClientContext {
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
// #487: same pre-commit safe-point as mutatePage, for full-document writes.
replacePageContent(
pageUuid,
doc,
token,
apiUrl,
this.toolAbortSignal ?? undefined,
),
replacePageContent(pageUuid, doc, token, apiUrl),
);
}
-7
View File
@@ -30,13 +30,6 @@ export { destroyAllSessions } from "./lib/collab-session.js";
// internals directly; it goes through loadDocmostMcp()).
export { SHARED_TOOL_SPECS } from "./tool-specs.js";
export type { SharedToolSpec } from "./tool-specs.js";
// #489 — write-class registry consumed by the in-app external-MCP retry gate.
export {
SHARED_TOOL_WRITE_CLASS,
isRetryableWriteClass,
assertEverySpecDeclaresWriteClass,
} from "./tool-specs.js";
export type { ToolWriteClass } from "./tool-specs.js";
// Re-export the build-time REGISTRY_STAMP (issue #447): a deterministic hash of
// the tool-specs registry content, generated into src/registry-stamp.generated.ts
+7 -19
View File
@@ -13,6 +13,7 @@ import { JSDOM } from "jsdom";
import { markdownToProseMirror } from "@docmost/prosemirror-markdown";
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import type { PageId } from "./page-id.js";
import {
sanitizeForYjs,
findUnstorableAttr,
@@ -250,16 +251,13 @@ export function assertYjsEncodable(doc: any): void {
* read->write window, and it never throws (it can NEVER break a write).
*/
export async function mutatePageContent(
pageId: string,
// Canonical UUID only (#260/#435): the brand forces every caller to
// resolvePageId() BEFORE this seam so the lock + CollabSession key can never
// be a raw slugId.
pageId: PageId,
collabToken: string,
baseUrl: string,
transform: (liveDoc: any) => any | null,
// #487: optional abort signal carrying the turn's Stop + the in-app tool
// per-call cap. Checked as the PRE-COMMIT safe-point below (after the session
// is acquired, immediately before the atomic read->write), so a Stop that
// arrives during the connect/lock window stops THIS write from landing. See the
// limitation note at the check.
signal?: AbortSignal,
): Promise<MutationResult> {
return withPageLock(pageId, async () => {
if (process.env.DEBUG) {
@@ -272,13 +270,6 @@ export async function mutatePageContent(
const session = await acquireCollabSession(pageId, collabToken, baseUrl);
try {
// #487 PRE-COMMIT safe-point: if the turn was Stopped (or the in-app tool
// per-call cap fired) after we acquired the collab session but before the
// atomic write, throw NOW so this commit never runs. KNOWN LIMITATION
// (#487): this only stops THIS commit — a write tool that already committed
// an EARLIER call this turn leaves that op applied. Cancel guarantees "no
// NEW commit starts", NOT "the write didn't land".
signal?.throwIfAborted();
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh (this
@@ -300,12 +291,10 @@ export async function mutatePageContent(
* mutatePageContent.
*/
export async function replacePageContent(
pageId: string,
pageId: PageId,
prosemirrorDoc: any,
collabToken: string,
baseUrl: string,
// #487: threaded straight to mutatePageContent's pre-commit safe-point.
signal?: AbortSignal,
): Promise<MutationResult> {
// Fail fast on a bad document instead of deferring the failure into the
// collaboration write (where TiptapTransformer.toYdoc(undefined) used to
@@ -322,7 +311,6 @@ export async function replacePageContent(
collabToken,
baseUrl,
() => prosemirrorDoc,
signal,
);
}
@@ -332,7 +320,7 @@ export async function replacePageContent(
* Tables and :::callout::: blocks survive thanks to the full schema.
*/
export async function updatePageContentRealtime(
pageId: string,
pageId: PageId,
markdownContent: string,
collabToken: string,
baseUrl: string,
+81
View File
@@ -0,0 +1,81 @@
/**
* Branded page-identity types for the MCP client (incident family #435).
*
* A Docmost page has TWO identities that are BOTH plain strings:
* - the internal `page.id` a canonical UUID (the server generates UUIDv7),
* - the public `slugId` a 10-char nanoid over [0-9A-Za-z] used in URLs.
* Because both are bare `string`s, they were passed around interchangeably and
* silently swapped (e.g. locking/keying a collab doc by the slugId instead of
* the UUID the #260 data-loss). Branding them as distinct nominal types makes
* a swap a COMPILE error at the seams that matter, and the validating
* constructors reject a malformed / cross-wired identity at runtime too.
*
* These are type-level + format-validation helpers ONLY: a `PageId`/`SlugId` is
* still a `string` at runtime (assignable INTO any `string` parameter with no
* change), so branding a value flows outward for free; only the few seams that
* REQUIRE a canonical id (resolvePageId's result, the per-page lock key, the
* collab write entrypoints) demand the brand and so catch an unresolved raw id.
*/
import { UUID_RE } from "./page-lock.js";
/** The internal canonical page id (`page.id`), a canonical UUID. */
export type PageId = string & { readonly __brand: "PageId" };
/** The public page slug id (`slugId`), a 10-char nanoid used in URLs. */
export type SlugId = string & { readonly __brand: "SlugId" };
/**
* A page REFERENCE an agent may supply: EITHER the canonical UUID `PageId` or
* the public `SlugId`. The server's page lookup accepts both (a non-UUID is
* matched as a slugId), so the public read/tool boundary legitimately takes
* either form; only the resolved-to-canonical value is a strict `PageId`.
*/
export type PageRef = PageId | SlugId;
// A slugId is exactly 10 chars from the nanoid alphabet the server uses for
// generateSlugId: [0-9A-Za-z] (see apps/server .../common/helpers/nanoid.utils).
// Disjoint from a UUID (which is 36 chars WITH dashes), so the two formats can
// never be confused for one another.
export const SLUG_ID_RE = /^[0-9A-Za-z]{10}$/;
/** Type guard: is `value` a canonical page UUID (`PageId`)? */
export function isPageId(value: unknown): value is PageId {
return typeof value === "string" && UUID_RE.test(value);
}
/** Type guard: is `value` a public slug id (`SlugId`)? */
export function isSlugId(value: unknown): value is SlugId {
return typeof value === "string" && SLUG_ID_RE.test(value);
}
/**
* Validate + brand a canonical page id. Throws an actionable error (BEFORE any
* network use) when `value` is not a canonical UUID, so a slugId or garbage
* cross-wired where the canonical id is required fails fast and loud instead of
* silently splitting a lock/doc key (#260). `label` names the offending param.
*/
export function asPageId(value: string, label = "pageId"): PageId {
if (!isPageId(value)) {
throw new Error(
`${label}: expected a canonical page UUID (36 chars, e.g. ` +
`019f499a-9f8c-7d68-b7be-ce100d7c6c56), got '${value}'. A slugId or ` +
`other identity must be resolved to the page UUID first.`,
);
}
return value;
}
/**
* Validate + brand a public slug id. Throws when `value` is not the 10-char
* slugId format, so a UUID or garbage cross-wired where a slugId is required is
* rejected at the boundary.
*/
export function asSlugId(value: string, label = "slugId"): SlugId {
if (!isSlugId(value)) {
throw new Error(
`${label}: expected a 10-char page slugId (e.g. 'aB3xQ7kR2p'), got ` +
`'${value}'.`,
);
}
return value;
}

Some files were not shown because too many files have changed in this diff Show More