Compare commits

..

45 Commits

Author SHA1 Message Date
agent_coder 45ff922dd4 refactor(mcp): убрать мёртвый runtime-слой page-id, оставить бренд PageId
Ревью #516: isPageId/asPageId/isSlugId/asSlugId/SLUG_ID_RE + типы SlugId/PageRef
имели НОЛЬ вызовов в монорепе — только барные ре-экспорты в client.ts и кейсы
теста. Несущий смысл — только compile-time бренд PageId (минтится as PageId в
resolvePageId, единственном узле канонизации; asPageId туда намеренно не звался).
Докстринг заявлял «asPageId() guards the untrusted PUBLIC boundary» — но никто не
звал: спекулятивный вес.

Удалил мёртвые runtime-символы + типы SlugId/PageRef + ре-экспорты + их тест-файл.
Оставил тип PageId + касты. Поправил врущий коммент в resolvePageId (бренд —
чистый compile-time маркер, гарантия — что этот узел единственный производитель).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder e3ca2dc1d5 test(db): executeTx — покрыть rollback-ветку after-commit-хука
Ревью #516: fakeDb.execute всегда пушил commit — моделировался только commit-путь,
несущий негатив (тело кидает → tx реджектит → дренаж после awaited-tx не бежит →
хук НЕ фаerr) не был заперт. Исходник корректен, это была дыра в тесте.

Добавлен тест: тело бросает → executeTx реджектит, commit не случился, хук не
побежал. Mutation-verify: перенос дренажа в finally → тест краснеет.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder f919ced8c9 fix(db): concurrent-pre-build индексов — переопределять f_unaccent ПЕРЕД циклом
Ревью #516: фича была мёртвой-при-рождении ровно на целевом кейсе. Все три
CONCURRENT_INDEXES используют LOWER(f_unaccent(col)), но index-инлайнящаяся
1-арг форма f_unaccent (SELECT public.unaccent($1)) создаётся ВНУТРИ миграции
20260705, а ensureConcurrentIndexes зовётся ПЕРЕД мигратором. На существующем
тенанте живой f_unaccent — ещё старая 2-арг форма из 20250729, которая НЕ
инлайнится: CREATE INDEX CONCURRENTLY падает («function unaccent(unknown, text)
does not exist … during inlining»), best-effort глотает, и мигратор строит
индексы НЕ-concurrently под тем самым SHARE-локом (эмпирически подтверждено
ревьюером на живом pg).

Фикс: перед циклом идемпотентно переопределяем f_unaccent в 1-арг форму
(output-identical, в lockstep с 20260705), в том же best-effort try/swallow.

Плюс честный докстринг: убрал ложное «worst case = previous behaviour».
Прерванный CONCURRENTLY оставляет INVALID-индекс, который name-based IF NOT
EXISTS не чинит — новый режим отказа (старый in-tx build такого не оставлял).

Тест: переопределялка вызывается ПЕРВОЙ (order), затем индексы CONCURRENTLY вне
транзакции; провал переопределялки не рвёт цикл. Mutation-verify: убрать
pre-loop redef → order-тест краснеет.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder 4109b2ef7f fix(security): share-alias reassign 409 — гейт раскрытия по view-праву (доведение #495 к.4)
Коммит 4 закрыл утечку currentPageId на анонимном /availability, но
эквивалентная (и более широкая) дыра оставалась на POST /aliases/set (reassign):
при занятом алиасе и confirmReassign=false 409 отдавал currentPageId И
currentPageTitle ЦЕЛЕВОЙ страницы, а контроллер гейтил только validateCanEdit на
ИСХОДНОЙ странице. Любой участник с одной редактируемой+расшаренной страницей мог
перебирать имена алиасов и мапить их на (id, title) чужих страниц без права
просмотра — тот же класс перечисления, плюс ещё и заголовок.

Фикс: setAlias теперь гейтит раскрытие. currentPageId НЕ отдаётся никогда
(клиент им не пользуется, это перечислимая идентичность). currentPageTitle —
только если validateCanView(целевая, user) проходит; иначе голый «занят» (клиент
и так показывает generic confirm-модалку без заголовка — UX не ломается).
Гейт живёт в сервисе, где строится раскрытие (PageAccessService — @Global, без
цикла); контроллер прокидывает user. Поправлен неточный коммент checkAvailability.

Тесты: viewer → 409 с title, БЕЗ id; не-viewer → 409 без title и без id.
Mutation-verify: вернул утечку (id + безусловный title) → оба теста краснеют.
Контроллер-спек и int-spec обновлены под новую сигнатуру; tsc чист.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder 6b27ff0652 feat(ai,mcp): идентичность прогона реиндекса + брендированные PageId/SlugId (#495 item 14)
Аспект A — идентичность прогона реиндекса эмбеддингов.
У статуса/поллинга реиндекса не было идентичности КОНКРЕТНОГО прогона, из-за
чего класс багов «это тот же прогон или новый?» чинили дважды. Теперь каждый
прогон получает свой runId (crypto.randomUUID в start()), он хранится в Redis-
хэше рядом с total/done/startedAt и возвращается в ReindexProgress. Эндпоинт
статуса (getMasked -> MaskedAiSettings) отдаёт runId и reindexStartedAt. Клиент
кеит поллинг на (runId, startedAt): смена runId = НОВЫЙ прогон (сбрасываем
залатанное per-run состояние поллинга), тот же runId = тот же прогон. Всё
best-effort/косметика как и остальной код прогресса: пустой/отсутствующий runId
деградирует мягко и никогда не ломает реиндекс.

Аспект B — брендированные PageId/SlugId в MCP-клиенте + валидация формата в
серверных DTO (семейство инцидентов #435: двойная идентичность страницы —
внутренний id и slugId — гонялась как голая строка и молча путалась).
- packages/mcp/src/lib/page-id.ts: номинальные типы PageId/SlugId + валидирующие
  конструкторы asPageId/asSlugId и гварды isPageId/isSlugId (формат UUID и
  10-символьного slugId). PageId протянут через единственный узел канонизации и
  записи: resolvePageId() теперь возвращает PageId, а ключ per-page лока
  (withPageLock) и точки записи в collab (mutatePageContent/replacePageContent/
  updatePageContentRealtime) требуют бренд — сырой slugId/непроверенный id больше
  не проходит проверку типов (инвариант #260 «resolve-then-lock» теперь на уровне
  компилятора). Публичный вход методов остаётся строкой (это legitimно UUID ИЛИ
  slugId), брендируется каноническое значение.
- apps/server core/page/dto: PageIdDto.pageId получает валидацию формата
  (@Matches, UUID или 10-символьный slugId), так что кривая/подменённая
  идентичность отклоняется на границе, а не падает в repo голой строкой.

Тесты (все зелёные, с mutation-verify каждого):
- server: getMasked отдаёт runId/reindexStartedAt; стор пишет/читает runId и
  мягко деградирует его до ''; DTO отклоняет кривой pageId/slugId и принимает
  валидный.
- client: чистый хелпер reindexRunKey/isNewReindexRun (кеинг поллинга на runId).
- mcp: конструкторы/гварды PageId/SlugId (валидация формата).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder e133b86982 fix(ci): дедуп ночного фаззера по хэшу контрпримера, а не по префиксу заголовка
Ночной property-фаззер при находке контрпримера заводил/обновлял issue,
дедуплицируя по ПРЕФИКСУ ЗАГОЛОВКА. Из-за этого при уже открытом issue по
багу A другой баг B с тем же префиксом заголовка считался дубликатом и
молча терялся до закрытия первого issue — реальные вторые баги глотались.

Теперь дедуп идёт по стабильному короткому хэшу самого контрпримера:
- из вывода fast-check извлекается блок «Counterexample:» (минимальный
  падающий вход) до строки «Shrunk N time(s)»/«Got error»; сид, path и
  счётчик усадки в хэш НЕ входят, поэтому один и тот же баг с разными
  сидами даёт один хэш;
- sha256, первые 12 hex-символов, кладутся в заголовок и в
  машиночитаемый маркер тела `<!-- counterexample-hash: ... -->`;
- поиск открытого issue матчит этот хэш в заголовке или маркер в теле.

Итог: два РАЗНЫХ контрпримера дают два РАЗНЫХ issue, а повторная находка
ТОГО ЖЕ контрпримера по-прежнему схлопывается в существующий.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder 579c82617b i18n(ru): перевести недостающие строки в ru-RU
Добавлены переводы для 151 ключа, присутствовавшего в en-US, но
отсутствовавшего в ru-RU (настройки ИИ, диктовка, MCP, HTML-вставки,
роли агента и др.), включая ранее непереведённые «Streaming dictation»
и «Save and test». Технические токены и бренды (MCP, URL, Docmost AI,
плейсхолдеры версий) намеренно оставлены как есть. Добавлены русские
формы множественного числа (_few/_many) для «result found». Файл
переупорядочен в соответствии с порядком ключей en-US.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder 173f35e473 fix(editor): защитить чтение приватных стеков y-undo в тулбаре
canUndo/canRedo в use-toolbar-state читали приватные внутренности y-undo
(undoManager.undoStack.length / redoStack.length). Апгрейд yjs / y-undo,
переименовавший или перестроивший эти поля, тихо сломал бы состояние кнопок
undo/redo (или упал бы на .length у undefined) без единой ошибки.

Оставляем дешёвое чтение длины стеков (сознательно не используем дорогой
editor.can().undo()/.redo(), который делает dry-run на каждый keystroke,
см. комментарий в файле), но теперь feature-detect: доверяем стекам только
если это реально массивы, иначе откатываемся на безопасный дефолт
(prosemirror-history undoDepth/redoDepth -> 0). Логика вынесена в чистую
функцию yHistoryAvailability.

Добавлен pin-test, фиксирующий текущую форму библиотеки: реальный
Y.UndoManager по-прежнему отдаёт undoStack/redoStack массивами. Апгрейд,
меняющий контракт, упадёт громко в тесте, а не тихо в UI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder 696b96ac18 fix(client): авто-перезагрузка чанков в окне 5 минут вместо one-shot флага
При ChunkLoadError граница перезагружала страницу один раз, гейтируя
булевым флагом в sessionStorage, который никогда не сбрасывался. Из-за
этого ВТОРОЙ деплой за время жизни вкладки не давал авто-восстановления:
пользователь застревал на битом чанке без перезагрузки.

Заменяю one-shot флаг на счётчик по временному окну: не более одной
авто-перезагрузки за 5 минут. В sessionStorage храню метку времени
последней перезагрузки; на ChunkLoadError перезагружаемся только если
прошлая была раньше окна (или её не было), иначе проваливаемся в ручной
UI без перезагрузки. Это восстанавливает работу через несколько деплоев,
но не даёт бесконечного цикла при навсегда битом lazy-чанке (сброс флага
после успешного маунта отвергнут: оболочка монтируется, чанк 404 —
и цикл).

Решение об окне вынесено в чистый хелпер shouldAutoReload(now,
lastReloadAt, windowMs) и покрыто юнит-тестами: никогда-не-грузили →
можно; 6 минут назад → можно; 1 минуту назад → нельзя.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder a96ca8e26b perf(db): GIN trigram-индексы строятся CONCURRENTLY вне транзакции
2 GIN trigram-индекса (pages.title, pages.text_content) + users.name строились
plain CREATE INDEX внутри Kysely-транзакции миграции: SHARE-lock блокирует записи
на pages/users на минуты при автодеплое.

Kysely гоняет миграцию в транзакции, а CONCURRENTLY внутри транзакции нельзя.
Поэтому ensureConcurrentIndexes (concurrent-indexes.ts) пре-строит эти индексы
через CREATE INDEX CONCURRENTLY (raw, вне транзакции) ДО миграатора — на
существующей БД миграционный CREATE INDEX IF NOT EXISTS становится no-op и лок не
берётся. Best-effort: на свежей БД (нет pages/f_unaccent) пре-build молча
пропускается, а миграция строит индекс на пустой таблице. Худший случай = прежнее
поведение, лучший — без лока.

CONCURRENT_INDEXES — канонические определения; drift-guard тест сверяет их с
выражениями в миграциях. Тест раннера: CONCURRENTLY+IF NOT EXISTS, вне
транзакции, best-effort (падение одного не рвёт остальные).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder f84386f24a fix(temp-notes): свип на старте + FOR UPDATE SKIP LOCKED в транзакции
Две дырки свипера временных заметок:
- Не было свипа на старте: заметки, протухшие во время простоя, ждали до часа
  до первого тика @Interval. Добавил onApplicationBootstrap (best-effort, не
  блокирует boot).
- Гонка без блокировки: re-check и removePage не были атомарны, «Сделать
  постоянной» могла проскочить между ними. Теперь re-check идёт в транзакции с
  FOR UPDATE SKIP LOCKED: сериализуется с toggleTemporary (тот же row-lock) и
  пропускает строки, захваченные другим воркером/инстансом (без двойной
  обработки). Удаление идёт ВНУТРИ этой транзакции.

removePage получил опциональный trx: чтобы удалять под блокировкой без deadlock
на вложенной независимой транзакции. Броадкаст PAGE_SOFT_DELETED при переданном
trx отложен на commit через registerAfterCommit (откат больше не рассылает
фантомное удаление); без trx поведение прежнее.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder c0c44fddb9 fix(security): client-vitals — whitelist route/attr на анонимном эндпоинте
POST /api/telemetry/vitals анонимный, а route/attr не сверялись со словарём:
любой мог писать свободный текст в client_metrics (высокая кардинальность,
инъекция текста/PII/разметки).
- route: экспортировал полный словарь шаблонов из клиентского route-template.ts
  (KNOWN_ROUTE_TEMPLATES — канонический источник), сервер валидирует по зеркалу
  ALLOWED_ROUTE_TEMPLATES: нет в словаре → drop (null), событие остаётся.
- attr: это web-vitals attribution target (CSS-селектор), а не enum — ограничил
  консервативным charset CSS-селектора; всё вне набора → drop.
Клиентский self-consistency тест: templateRoute выдаёт ТОЛЬКО значения из
словаря (иначе легитимные метрики отбрасывались бы). Серверные тесты: raw-путь
и инъекция в route отброшены, PII/разметка в attr отброшены.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder 91e58e3c9f fix(cache): bustWorkspaceCache — после коммита, не внутри транзакции
bustWorkspaceCache звался сразу после write, но ВНУТРИ переданной транзакции
(до коммита). Окно: параллельный читатель промахивается мимо инвалидированного
ключа, читает ещё НЕ закоммиченную (старую) строку и репопулирует кэш старым
значением; после коммита кэш держит устаревшее до TTL (15 c).

Добавил post-commit-хук в executeTx: registerAfterCommit(trx, fn) регистрирует
side-effect, который дренится ТОЛЬКО после коммита транзакции — причём внешним
executeTx, владеющим trx (проброшенный existingTrx срабатывает на настоящей
границе коммита, а не во вложенном вызове). WeakMap по trx — без утечки.
bustWorkspaceCache теперь: без trx — del сразу (уже автокоммит); с trx —
регистрирует del на post-commit. Ошибка хука не валит уже закоммиченный write.

Тест: порядок body→commit→hook, дренаж на внешней границе, глушение падения хука.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder 8d062728e5 fix(page): move-position — валидация charset вместо длины
MovePageDto.position — это fractional-indexing ключ (generateJitteredKeyBetween,
тот же генератор, что в page.service). @MinLength(5)/@MaxLength(12) не совпадали
с реальным диапазоном генератора: плотные between-вставки в глубоком дереве
растят ключ далеко за 12 символов (замерено >40), и валидный ключ, который
сервер сам сгенерил, отклонялся 400 (Gitea #139, п.6).

Теперь валидируем по charset — base-62 алфавит [0-9A-Za-z] — плюс щедрый
@MaxLength(256) как чистый DoS-guard, сильно выше любого реального ключа.
test.failing (bug-lock) распинен в обычный it; добавлен кейс на отклонение
символов вне алфавита (control/separator/инъекция/пустая строка).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:02:13 +03:00
agent_coder f1cffc2d0f fix(security): share-alias availability не отдаёт currentPageId
Проба доступности алиаса возвращала currentPageId — id страницы, на которую
алиас уже указывает — ЛЮБОМУ аутентифицированному участнику воркспейса без
проверки прав на просмотр этой страницы. Перебором имён алиасов можно было
смапить их на id страниц, к которым доступа нет.

Теперь checkAvailability отдаёт только {alias, valid, available}. Бита
taken/free достаточно для пробы; заголовок целевой страницы всплывает лишь
ПОСЛЕ реальной попытки setAlias (путь 409 ALIAS_REASSIGN_REQUIRED), который
проверяет права. Клиент currentPageId нигде не использовал — убран из типа,
стейта и теста. Серверный спек утверждает отсутствие currentPageId в ответе.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 00:53:44 +03:00
agent_coder bb9a6fd765 chore(ai-chat): guard-пороги в env + единый маркер дегенерации
Три хвоста детектора петель (#444):
- Пороги детектора дегенерации теперь конфигурируются из env (по образцу
  AI_CHAT_FINAL_STEP_LOCKDOWN): AI_CHAT_DEGENERATION_REPEATED_LINES /
  _PERIOD_MAX_LEN / _PERIOD_MIN_REPEATS / _CHECK_STEP. Резолвер читает сырую
  строку (пусто = unset → компилируемый дефолт), требует ≥1, иначе безопасный
  откат к дефолту (0/отриц. сломал бы детектор). Оператор перенастраивает
  анти-babble-guard без редеплоя.
- Единый маркер: при дегенерации live-стрим показывает нейтральное «Response
  stopped.» (клиент не отличает от ручного Stop), а персист-баннер после refetch
  падал в дженерик. Классифицировал OUTPUT_DEGENERATION_ERROR на клиенте под тот
  же заголовок «Response stopped.» + деталь про петлю — live и refetch больше не
  расходятся.
- STEP_LIMIT_NO_ANSWER_MARKER: вместо хардкодной русской строки в content —
  локаль-нейтральная английская (базовая локаль = i18n-ключ этого репо, читаема
  моделью на реплее); не-русские юзеры больше не видят русский текст.

Тесты: env-пороги гоняются против реальных форм повторов с mutation-проверкой
(поднятый checkStep глушит burst; сниженный repeatedLines триггерит короткий
ран); клиентская классификация — новый кейс в error-message.test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 00:53:44 +03:00
agent_coder 1c083fbd3d chore(deps): вендорные ai-патчи — upstream-трекинг и план выравнивания версии
Оформил как план, а не сюрприз деплоя (строго после drain-патча из #486):
- upstream-репортинг двух ai-фиксов (O(n²) partialOutput heap-OOM; drain-hang
  в writeToServerResponse) и hocuspocus connect-vs-unload (#401) — анализ уже
  в PATCH()-заголовках самих патчей; ссылки держим в AGENTS.md, а НЕ в .patch
  (байты патча идут в patch_hash lockfile e8c599b3 — правка десинхронит пин и
  ломает pnpm install).
- рассинхрон версии ai в монорепе (клиент 6.0.207 vs сервер 6.0.134-patched):
  пока безвреден (серверные фиксы — мёртвый код в браузере), но это дрейф.
  Выравнивание — install-gated шаг с явным «портировать все три ai-патча на
  целевую версию»: оффсеты строк сдвинутся, текущий патч НЕ приложится как есть,
  pnpm install падёт на неприложенном патче — это и есть страховка.

Патч-файлы и lockfile не тронуты; оба tripwire-спека находят маркеры.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 00:51:13 +03:00
agent_coder 52763998d3 fix(ai): gemini/ollama через aiStreamingFetch + явный maxRetries
Провайдер-фабрики gemini и ollama (chat-путь) шли на глобальном undici-fetch:
без keep-alive recycle, без ретраев на pre-response reset, с дефолтным
(безграничным по паузе) таймаутом. Классы инцидентов #140/#175/#310 для них
воспроизводимы так же, как для openai. Прокинул this.aiProviderFetch (одна
строка на провайдера) — тот же слоёный instrumented streaming fetch, что уже
стоит на openai.

Плюс явно закрепил maxRetries=2 в обоих streamText-вызовах (authenticated и
public-share): совпадает с дефолтом SDK, но фиксирует потолок против дрейфа
дефолта. Арифметика коннектов на ход: (1 + maxRetries=2) × (1 +
AI_STREAM_PRE_RESPONSE_RETRIES) — два слоя ретраев композируются.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 00:50:35 +03:00
vvzvlad bfb6a52eea Merge pull request 'epic #497 волна 1 — интеграция четырёх approved-итераций (#486/#487/#488/#489) → develop' (#511) from integ/497-wave1 into develop
Reviewed-on: #511
2026-07-11 19:29:27 +03:00
agent_coder 0503e8b4b1 fix(ai-chat): W1 — клиент читает 409-поле activeRunId (было runId=undefined)
Ревью волны 1 (agent_vscode): сервер эмитит id текущего рана в activeRunId
на обоих 409-бранчах (SUPERSEDE_TARGET_MISMATCH, A_RUN_ALREADY_ACTIVE), а
клиентский read409 читал runId → SUPERSEDE_MISMATCH{currentRunId} всегда
undefined: быстрый supersede-хинт мёртв в проде, а клиентские тесты
ложно-зелёные (мокали поле runId, которого сервер не шлёт).

- read409 читает activeRunId (undefined-safe); оба мока — на реальную форму
  + ассерт на усвоенный currentRunId (mutation: revert→runId краснит тесты).
- S4 (groundwork): activeRunId из A_RUN_ALREADY_ACTIVE поглощается в runFact
  (to(), без бампа эпохи/ownership — инварианты #488 целы). Полная обвязка
  supersede чужой вкладки из фазы error требует расширения gate в sendNow —
  отдельным follow-up; сейчас это безопасная заготовка, не рабочая фича.
- spec.md: 2 строки контракт-таблицы приведены к activeRunId.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 19:23:10 +03:00
agent_coder b97cad0ebe Merge remote-tracking branch 'gitea/feat/488-client-fsm' into integ/497-wave1 2026-07-11 17:29:35 +03:00
agent_coder 080dd82023 fix(client): ре-ревью #509 — 5 пунктов Do-листа (стабильность/регрессии) (#488)
1 [stability] Позитивные attach-исходы гвардятся по ИСХОДНОЙ фазе. Одного epoch-
фильтра мало: POLL_TERMINAL использует to() (epoch не инкрементит) и не шлёт
abortAttach, поэтому медленный GET, вернувший live 2xx уже ПОСЛЕ того как армленный
poll увёл машину в idle, воскрешал осевший ран в фантомный streaming. RECONNECT_
ATTACHED теперь bail'ит если фаза != reconnecting; ATTACH_LIVE/ATTACH_NONE — если
!= attaching. Тест на гонку (POLL_TERMINAL до RECONNECT_ATTACHED → idle) + mutation.

2 [regressions] ownership сбрасывается в "local" на ВСЕХ терминальных переходах
(FINISH_CLEAN/ABORT/ERROR, POLL_TERMINAL, RUN_FACT{null}→idle, honor-in-stopping,
disconnect→idle). Иначе observer-attach + очередь + clean-финиш → idle, но
ownership навсегда observer → «Send now» скрыт при свободном композере. Безопасно
для I2: рантайм захватывает wasObserver из machineRef ДО dispatch. Тест + mutation.

3 [coverage] Тест happy-path CAS-supersede: SUPERSEDE_READY-dispatch (200-исход
transport.fetch) раньше НЕ исполнялся ни в одном тесте — риск залипания в
superseding на весь стрим B. Тест вводит в superseding, гонит A.onFinish→B, затем
POST 200 → SUPERSEDE_READY → streaming, проверяет повторный supersede (не залип).
Сиблинги: 409 SUPERSEDE_TARGET_MISMATCH → getRun/verify; plain-409
A_RUN_ALREADY_ACTIVE → классифицированный баннер. Mutation (no-op READY → красный).

4 [regressions] Inactivity-бэкстоп для poll, армленного в stopping. STOP_REQUESTED
армит poll и входит в stopping, но idle-cap покрывал только polling/reconnecting →
observer-стоп без SDK-стрима и без серверного терминала поллил БД вечно. Добавлен
stopping в фаза-чек эффекта + переход POLL_IDLE_CAP: stopping→idle+disarm (НЕ
stalled — Stop уже нажат). FSM + компонент-тест (idle-cap→disarm) + mutation.

5 [docs] Счёт §2 «Net»: 8→FSM (#1-6,#11,#13) + 3 deleted + reconnectTimerRef
(effect-owned) + mountedRef (retained) = 13; attachAbortRef вне набора #1-13.

Не трогал DROP-блок (мёртвый FINISH_* в superseding, ErrorKind.kind, неиспользуемые
enum-варианты, epochRef-зеркало). Всё прошлое цело (disconnect-first, epoch, honor-
in-stopping, render-gate, supersede). Полный ai-chat 35 файлов / 388 / 0; tsc 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 11:07:02 +03:00
agent_coder 803bbd40b4 fix(client): фаза FSM решает баннер — reconnect не маскируется остаточным error (#488)
Браузерный QA: маршрутизация disconnect-first работает (лесенка реально стартует),
но пользователь всё равно видел терминальный «Lost connection… reload», а не
«reconnecting… (N/5)».

Трасса (подтверждена по коду): рендер `{errorView ? <ChatErrorAlert/> : phase ===
"reconnecting" ? ...}` даёт errorView ПРИОРИТЕТ над recovery-фазой. errorView =
`error && describeChatError(...)`, где `error` — из useChat. Реальный ai@6 при
isError выставляет useChat `error`, а дроп ВСЕГДА isError+isDisconnect → после
починки маршрутизации FSM уходит в `reconnecting`, но `error` остаётся выставленным
→ errorView перекрывает reconnect-баннер. Тот же класс вакуум-мока, что и с
isDisconnect, но на поле `error` (мок хардкодил error:null → в тестах маски не было).

Фикс рендера: фаза FSM — источник истины. Терминальный errorView показывается
ТОЛЬКО когда FSM реально терминален: `showError = errorView && phase === "error"`.
В recovery-фазах (reconnecting/polling/stalled/superseding/stopping) выигрывает
recovery-баннер (или контент стрима). Классифицированные 409 (#487) целы: supersede/
gate-409 ставят FSM в error(kind) → errorView показывается там, где должен.

Мок useChat сделан реалистичным СИСТЕМНО: добавлено поле h.state.error, мок его
возвращает; при любом isError-финише (дроп или провайдерская ошибка) тест ставит
error в реальную форму, зеркаля связку SDK. MUTATION-VERIFY: откат рендер-гейта
(errorView-first) → 8 reconnect/stalled-тестов краснеют (баннер маскируется);
с гейтом — зелёные. Плюс отдельный тест «reconnect-баннер виден, не замаскирован».

Всё прошлое цело: disconnect-first, epoch-штамп, honor-in-stopping, supersede-
исходы, stalled/no-poll. Полный ai-chat 35 файлов / 378 / 0; tsc ai-chat 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 09:55:39 +03:00
agent_coder c7a38e274e fix(client): реальный обрыв SSE → reconnect-лесенка, не терминальный error (#488)
Браузерный QA поймал баг, который юнит-тесты пропускали из-за вакуумной SDK-формы.

Премиса (подтверждена по ai@6.0.207 AbstractChat.makeRequest, catch ~13763): при
сетевом дропе SDK ставит isError=true БЕЗУСЛОВНО, а isError=true → isDisconnect=true
ставится РЯДОМ (только для fetch/network TypeError). Т.е. реальный обрыв ВСЕГДА
даёт { isError:true, isDisconnect:true }; форма { isDisconnect:true, isError:false }
SDK'ом НЕ эмитится.

Баг: в onFinish проверка `if (isError) return` стояла РАНЬШЕ ветки isDisconnect →
реальный дроп уходил в терминальный error-баннер, а FINISH_DISCONNECT (единственный
вход в reconnect-лесенку) не диспатчился НИКОГДА. Сценарии 1/2 (commit 2 «обрыв до
первого кадра» и commit 3 «два обрыва») в браузере не работали.

Фикс: маршрутизация по disconnect ПЕРВЫМ: isDisconnect → FINISH_DISCONNECT
(reconnect); НЕ-disconnect error (isError && !isDisconnect, напр. провайдерский 500)
→ FINISH_ERROR (терминал); затем isAbort; затем clean. Порядок веток supersede-
блока тоже disconnect-first (для консистентности; там всё равно всё дропается I1).

Инварианты сохранены: epoch-штамп turnEpochRef на всех FINISH_*; honor-in-stopping
в редьюсере честит ЛЮБОЙ финиш в фазе stopping → на пути Stop дроп-финиш уходит в
idle, а не в ложный reconnect (новый тест). F1 supersede-drop чужого поколения цел.

Тесты сделаны НЕ вакуумными: дроп подаётся реальной формой { isError:true,
isDisconnect:true }, терминальная ошибка — { isError:true, isDisconnect:false }.
MUTATION-VERIFY: багованный isError-first порядок → 6 reconnect-тестов краснеют
(нет баннера «reconnecting»); с фиксом зелены. Полный ai-chat 35 файлов / 377 / 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 09:37:07 +03:00
agent_coder 791a707eb6 docs(ai-chat): задокументировать поверхность #489 — env-var, устаревшая заметка, CHANGELOG
F1 (ревью #508):
- .env.example: новая AI_MCP_SSE_BODY_TIMEOUT_MS (дефолт 600000, 10 мин) —
  bodyTimeout SSE-транспорта external-MCP; idle между тул-вызовами легитимен.
- .env.example: поправлена ставшая ложной заметка про AI_MCP_STREAM_TIMEOUT_MS
  (SSE-idle-между-вызовами больше не режется им — с #489 это отдельная var;
  1-мин silence остаётся только для HTTP/headers и одиночного залипшего вызова).
- CHANGELOG [Unreleased]/Fixed: битый part больше не 500-ит каждый ход + не
  плодит дубль user-строки; MCP-транспорт-дропы восстанавливаются in-run
  (readOnly ретраится раз, write никогда) + новая env-var.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 09:11:32 +03:00
agent_coder 60205398bb test(ai-chat): застабить findAllByChat в lifecycle-моке под convert-before-insert (#489)
Коммит 1 (#489) прогоняет загрузку истории + convertToModelMessages ДО insert
user-строки (convert-before-insert, чтобы ретрай не плодил дубли), поэтому
stream() теперь зовёт реальный метод репо findAllByChat перед insert. Хэнд-роллед
мок в тесте «exception after beginRun → settled to error» стабил только insert,
и тест ловил «findAllByChat is not a function» вместо «insert boom».

Порядок инварианта НЕ изменился: и findAllByChat, и insert идут ПОСЛЕ beginRun
(reconcileChat между ними — best-effort, глотает ошибку), так что бросок по-
прежнему происходит ПОСЛЕ начала рана и обязан сеттлить ран в error. Застабил
findAllByChat → [] (реальный репо-метод, см. ai-chat.controller.ts), тест снова
доходит до insert boom и проверяет settle-to-error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:25:29 +03:00
agent_coder 1685b32333 fix(client): ре-ревью F1 — правка задетых Stop и supersede-таймингов (#488)
MEDIUM — эпоха-штамп из F1 сломал выход из `stopping` для локального Stop.
STOP_REQUESTED бампит epoch (E1→E2), а onFinish аборчиваемого стрима стампится
ПРЕД-стоп-эпохой E1 (start-эпоха стрима), поэтому фильтр I1 дропал FINISH_ABORT и
машина зависала в `stopping` навсегда (idle-cap покрывает только polling/
reconnecting). Фикс в редьюсере: ЛЮБОЙ finish (`FINISH_*`/`STREAM_INCOMPLETE`) в
фазе `stopping` честится и выводит `stopping→idle` МИНУЯ epoch-фильтр — у обычного
Stop нет преемника, финиш аборта и есть ожидаемое завершение (I4). Для `superseding`
фильтр сохранён (это и есть F1-drop). Тест переписан на ПРЕД-стоп-эпоху E1 (реальная
проводка); MUTATION-VERIFY: снятие honor-in-stopping → зависание → красный.

LOW-1 — supersede терял B, если A уже settled, а statusRef ещё «streaming».
Ливнесс в sendNow теперь берётся из ФАЗЫ FSM (machineRef, обновляется onFinish'ем
СИНХРОННО), а не из отрендеренного statusRef: settled-A (фаза idle) → B шлётся
немедленно, без аборта мёртвого стрима и залипания в `superseding`. statusRef
удалён (больше не нужен). Тест на под-кадровое окно.

LOW-2 — двойной «Send now» в окне аборта A перезаписывал pendingSupersedeTextRef.
Второй клик при уже летящем supersede (pendingText взведён / фаза `superseding`) —
NO-OP: сообщение остаётся в очереди, ничего не теряется/не перетирается. Тест.

Тесты: FSM 36 + chat-thread 39 (+error 26 +adopt 16) = 117 зелёных; tsc ai-chat 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:02:04 +03:00
agent_coder 4b78e336a2 fix(client): внутреннее ревью миграции FSM — F1..F4 (#488)
F1 [correctness] — supersede: перекрытие стримов рушило свежий run.
ai@6 AbstractChat.makeRequest в finally читает и обнуляет общий activeResponse,
поэтому параллельные стрим-A и стрим-B корёжат друг друга, а незастемпленный
onFinish мёртвого A уводил ЖИВОЙ новый run в ложный reconnect / сбрасывал runFact.
Фикс: (1) FINISH_*/STREAM_INCOMPLETE штампуются per-stream generation (turnEpochRef,
взводится в момент старта стрима) → фильтр I1 отбрасывает финиш чужого поколения;
(2) sendNow-supersede АБОРТИТ A и стартует B только из onFinish A (микротаск,
после того как finally A обнулил activeResponse) — гарантия отсутствия перекрытия.
Тест на поздний isDisconnect A после SUPERSEDE_REQUESTED: машина НЕ уходит в
reconnect, B отправлен. MUTATION-VERIFY: снятие epoch-штампа у FINISH_DISCONNECT →
тест краснеет («Connection lost — reconnecting»).

F2 [correctness] — гонка mount getRun→ATTACH_START с локальным send. Редьюсер
ATTACH_START теперь игнорирует любую не-idle фазу, поэтому поздний резолв getRun не
перехватывает начавшийся локальный турн в observer-attach. Тест на гонку.

F3 [ghost feature] — RUN_SUPERSEDED объявлен+покрыт тестом, но НИКОГДА не
диспатчился. Удалён (событие+обработчик+тест+postRun-reason observer-follow) как
сознательный scope-cut: наблюдатель убитого supersede-рана и так следует за новым
через деградированный поллинг (свежие строки истории, независимо от runId).

F4 [hygiene] — мёртвые события. STREAM_START подключён (первый ассистент-фрейм
локального турна: sending→streaming, спека↔код совпали). RECONNECT_BEGIN и
POLL_ACTIVITY удалены (не диспатчились). Множество событий редьюсера = множеству
диспатчащихся; редьюсер тотален.

Тесты: FSM 35 + chat-thread 37 (+error 26 +adopt 16) = 114 зелёных; tsc ai-chat 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:02:04 +03:00
agent_coder 77e20ecb41 fix(client): миграция chat-thread на FSM — reconnect×N, stalled, supersede (#488)
Полная миграция chat-thread.tsx на автомат run-fsm: 13 ref-флагов жизненного
цикла resume/reconnect/poll/ownership УБРАНЫ (карта ref'ов в run-fsm.spec.md,
колонка pending пуста). Коммиты 3/4/5 приезжают одной атомарной миграцией —
три фикса делят одну модель состояний (раздельные коммиты давали бы несобираемые
промежуточные состояния, что противоречит смыслу единого автомата).

Коммит 3 — повторные циклы reconnect: attached→reconnecting разрешён многократно.
Различие «live-follow (лестница reconnect) vs mount-resume» вынесено в ctx-поле
liveFollow (НЕ новый ref — это и есть смысл FSM): live-follow-обрыв перезаходит в
лестницу (сброс счётчика после успешного re-attach), mount-resume-обрыв уходит в
poll. Тест «два обрыва подряд → два цикла».

Коммит 4 — (a) polling→stalled по idle-капу (баннер+Retry вместо тихого
«вечно-полуготового»); кап переехал в тред (idleCapTimerRef, effect-owned, не
флаг), окно теперь тупо поллит по armed-флагу. (b) resume армится ТОЛЬКО при
серверном подтверждении активного рана: streaming-tail (статус) или POST /run для
user-tail — чат без активного рана больше не порождает ~240 req/10мин. Тесты:
stalled-баннер; user-tail с/без активного рана.

Коммит 5 (supersede) — удалены SUPERSEDE_RETRY_DELAYS_MS/isRunAlreadyActive/
supersedeRetryRef (клиентская лестница ретраев). «Прервать и отправить» идёт через
FSM superseding → POST /stream {supersede:{runId}} (runId из start-метаданных,
extractRunId). Транспорт разбирает CAS-исход: ok→SUPERSEDE_READY (новый стрим),
409 MISMATCH→verify через /run, TIMEOUT/INVALID→классифицированная ошибка без
авто-ретрая; голый 409 A_RUN_ALREADY_ACTIVE→RUN_ALREADY_ACTIVE. pendingSupersedeRef
(send-плумбинг data) — единственная замена трёх удалённых one-shot'ов.

Инвариант epoch (I1) гейтит каждый command-outcome (attach/reconnect/supersede/
postRun): устаревшее поколение колбэка отбрасывается редьюсером; DISPOSE на unmount
инкрементит epoch. mountedRef оставлен как React-liveness (ортогонален lifecycle).

Тесты: FSM 37 переходов; chat-thread 35 (переписан на FSM-переходы); все зелёные.
E2E (реальный SSE/reconnect/supersede через редиплой) — на staging QA.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:02:04 +03:00
agent_coder d533daa45f fix(client): обрыв SSE до первого кадра ассистента → ран подхватывается (#488)
Коммит 2. Раньше вход в reconnect требовал `message?.role === "assistant"`, и
обрыв в setup-фазе (до первого кадра ассистента, включая сборку MCP-тулсета с
дедлайном до 60 c) не давал НИ реконнекта, НИ поллинга — а detached-ран
продолжал писать в страницы (тихая дыра целостности).

Правка: вход в reconnect по РАН-ФАКТУ (активный detached-ран), а не по наличию
assistant-сообщения. В autonomous-режиме ран активен весь ход, поэтому здесь
сигнал run-факта — сам autonomousRunsEnabled; более богатый серверный run-факт
(POST /run / runId из start-метаданных) смоделирован и покрыт тестами в FSM
(run-fsm.ts FINISH_DISCONNECT по ctx.runFact) и приезжает с полной миграцией
компонента. При отсутствии assistant-строки reconnect идёт БЕЗ strip/anchor
(простой live-attach — на экране нечего перестраивать).

Тест: «обрыв до первого кадра → баннер reconnect + resumeStream + attach без
anchor», плюс FSM-переход (уже зелёный в коммите 1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:02:04 +03:00
agent_coder ba58493909 fix(client): классификация 409-кодов + run-факт плумбинг (#488)
Потребляет реальный контракт #487 на клиенте (без выдумывания кодов):
- error-message.ts: ветки для 409 A_RUN_ALREADY_ACTIVE / SUPERSEDE_TARGET_MISMATCH
  / SUPERSEDE_TIMEOUT / SUPERSEDE_INVALID — человеческие тексты СТРОГО ДО generic-
  веток по статусу (иначе юзер видит сырой JSON {"code":"A_RUN_ALREADY_ACTIVE"});
- extractRunId(message): чтение runId из start-метаданных (зеркало
  extractServerChatId) — live-обновление run-факта для FSM;
- getRun(chatId): POST /ai-chat/run — first-class run-факт с сервера (init на
  маунте + verify после supersede-mismatch).

Плумбинг под FSM-обвязку коммитов 2–5. Тесты: классификатор (все 4 кода + order-
guard), extractRunId.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:02:04 +03:00
agent_coder 947796adef refactor(client): FSM skeleton + spec для run-lifecycle (#488)
Заменяет зоопарк из ~26 useRef-флагов в chat-thread.tsx на один чистый
редьюсер с перечислимыми переходами (event × state → next state — впервые
юнит-тестируемо напрямую).

Коммит 1 из 5. Содержит СПЕКУ (пишется первой, входит в PR) и каркас:
- run-fsm.spec.md: таблица «событие × состояние», карта всех ref'ов
  → {состояние | контекст | данные}, протокол run-факта, список инвариантов;
- run-fsm.ts: чистый reduce(machine, event) → machine с epoch-инвариантом (I1),
  состояниями idle|sending|streaming|attaching|reconnecting|polling|stalled|
  stopping|superseding|error, ownership как ПОЛЕ контекста (I2), run-фактом
  как first-class (I3), выходом из stopping по данным (I4), dispose-протоколом
  (I5) и слоем command-эффектов (attach/postStream/postRun/stop/supersede);
- run-fsm.test.ts: 31 тест переходов, включая поведение коммитов 2–5 как
  переходы автомата (reconnect по run-факту; повторные циклы reconnect;
  polling→stalled; supersede CAS-исходы; фильтрация позднего колбэка по epoch).

8 зафиксированных решений реализованы; epoch-инвариант неотключаем.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:02:04 +03:00
agent_coder 38a09c5ca1 fix(ai-chat): write-class-ретрай только для доверенного внутреннего Docmost-сервера (#489 ревью)
MEDIUM (реальная брешь): SHARED_TOOL_WRITE_CLASS — имена ТУЛОВ Docmost, но
mergeNamespaced применял мапу к тулам ЛЮБОГО стороннего MCP-сервера по совпадению
rawName. Сторонний WRITE-тул, названный как Docmost-read (getPage/listPages/…),
наследовал readOnly → авто-ретрай при транспортной ошибке → double-apply (класс
#435). Гарантия «unknown third-party → write → не ретраится» была ЛОЖНОЙ при
коллизии имён.

Фикс: write-class мапа применяется ТОЛЬКО к серверу, про который известно, что
это внутренний Docmost-MCP (isInternalDocmostServer). Сейчас это ВСЕГДА false —
в этом пути нет встроенного/доверенного Docmost-сервера: все ai_mcp_servers суть
сторонние admin-конфиги, а собственные тулы Docmost идут отдельным in-app путём
(docmostTools), не через mcp-clients. Значит НИ ОДИН сторонний тул не получает
readOnly по коллизии и не авто-ретраится (undefined → трактуется как write).
Мапа грузится лениво только если есть доверенный сервер (иначе ESM-импорт
пропускается). isInternalDocmostServer — метод-сим, флипается при появлении
доверенного сервера (kind/isBuiltin-колонка или сконфигурённый self-MCP URL).

LOW: reconnect не наблюдает composed abort во время 5-сек handshake — задокумен-
тировано (окно поздней отмены ≤5s, сокет закрывается на turn-end); проброс
composed в общий CAS-дедуплицированный reconnect намеренно НЕ сделан (отменил бы
реконнект, нужный конкурентному живому вызову).

Тест: сторонний WRITE-тул с именем getPage при транспортной ошибке НЕ ретраится;
mutation-verify — форсирование trusted делает тест красным (чужой getPage ретраится).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:01:33 +03:00
agent_coder 1b05224b27 fix(ai-chat): MCP-кэш — in-run восстановление, ретраи только для readOnly (#489)
Кэш внешних MCP-клиентов не делал health-check/reconnect-on-error: после
разрыва SSE-транспорта (bodyTimeout режет при тишине МЕЖДУ вызовами) отдавал
труп до конца TTL, а модель жгла шаги на ретраях ВНУТРИ текущего рана.

- Новое поле writeClass ('readOnly'|'write') на КАЖДОМ SHARED_TOOL_SPEC +
  registration-time assert (и compile-time через satisfies). Все 48 спеков
  расклассифицированы: чтения → readOnly, любая мутация страницы/коммента/шэра/
  диаграммы → write. Экспортированы SHARED_TOOL_WRITE_CLASS + isRetryableWriteClass.
- Per-run обёртка восстановления транспорта: при транспортной ошибке readOnly-тул
  реконнектит свой сервер и ретраит РОВНО 1 раз ВНУТРИ рана; write-тул НЕ
  авторетраится (indeterminate — «могло примениться, проверь», класс инцидента
  #435). CAS-своп байндинга по identity (проигравший конкурентный вызов ретраит
  на текущем клиенте, не минтит второй). Лизы не освобождаются mid-run — ран
  копит set (старая+новая) и релизит на turn-end.
- Проверка abortSignal ПЕРЕД ретраем И ПЕРЕД чеканкой свежего клиента; per-call
  cap покрывает оба attempt'а + connect.
- Классификация транспортной ошибки по РЕАЛЬНЫМ шейпам undici (SocketError/
  BodyTimeoutError, cause-цепочка), не по мок-ошибкам.
- Отдельный, поднятый bodyTimeout для SSE-транспорта MCP (тишина между вызовами
  легальна) — DEFAULT 10 мин, AI_MCP_SSE_BODY_TIMEOUT_MS.

write-class map грузится в mcp-clients лениво через dynamic import (пакет ESM),
type-only импорт — без static require ESM из commonjs.

Тесты на РЕАЛЬНЫХ error-шейпах: «повтор после обрыва ВНУТРИ рана получает живой
клиент», «write-тул не авторетраится», «ретрай после Stop не происходит»,
+ writeClass-контракт в mcp node --test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:01:33 +03:00
agent_coder b50b32bf64 fix(ai-chat): валидация клиентских parts + insert после конвертации (#489)
Юзер-сообщение персистилось в metadata БЕЗ валидации и реплеилось через
convertToModelMessages на каждом ходе; insert user-строки шёл ДО конвертации.
Битая структура parts (например, null-элемент в массиве) → конвертация кидает
→ 500 на КАЖДОМ ходе навсегда, а каждый ретрай добавлял дубль user-строки.

- Санитизация parts при приёме: whitelist { text }, прочее (в т.ч. tool-part
  в input-available) отбрасывается с warn — не попадает в metadata.
- convertToModelMessages прогоняется ДО insert'а user-строки (ретрай не плодит
  дубли); при падении на СТАРОЙ истории — per-row конвертация изолирует битую
  строку и деградирует её до plain-text с маркером «[tool context omitted]»
  (молчаливая потеря tool-контекста недопустима).

Тесты против РЕАЛЬНОГО convertToModelMessages (null-part реально кидает):
unit трёх веток + сервис-регресс «чат с битым сообщением в истории работает,
маркер доходит до модели, одна user-строка».

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 08:01:33 +03:00
agent_coder 1d704d4ec5 test(mcp): покрыть write-safe-point «после Stop новая запись не стартует» (#487, F4)
paginate-abort-safepoint покрывал только READ-safe-point; WRITE-шов
(collaboration.mutatePageContent / context.mutateLiveContentUnlocked —
signal?.throwIfAborted() перед session.mutate) был без теста. Добавляю
мок-collab юнит через тест-сим __setCollabProviderFactory (фейковый провайдер
с мгновенным onSynced): предварительно abort-нутый сигнал → оба шва реджектят
ДО session.mutate, трансформ не вызывается; контрольные кейсы (живой сигнал)
подтверждают, что session.mutate достижим.

MUTATION-VERIFY: снятие throwIfAborted в обоих швах делает красными ровно два
abort-кейса, контрольные остаются зелёными.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:49:30 +03:00
agent_coder 2cb07ef7fb test(ai-chat): покрыть оркестраторы reconcile()/reconcileChat() (#487, F3)
Клаузы (a-d) тестировались по отдельности, но сам оркестратор reconcile()
(ПОРЯДОК клауз + пер-клаузная try/catch-изоляция «одна упавшая не блокирует
остальные») не проверялся, а reconcileChat() (старт каждого хода) не был
покрыт вовсе. Добавляю: тест порядка a→b→c→d; тест изоляции (клауза b кидает
— c и d всё равно отрабатывают, reconcile не реджектит); тест reconcileChat
(скоуп по чату, bound 50, failed→error / прочее→aborted).

MUTATION-VERIFY: снятие try/catch у клаузы (b) делает тест изоляции красным
(исключение пробрасывается, c+d пропускаются).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:49:30 +03:00
agent_coder 9ba97b6d2b docs(ai-chat): задокументировать наблюдаемую поверхность #487 (F2)
- CHANGELOG [Unreleased]: серверный supersede + три кода (SUPERSEDE_INVALID /
  SUPERSEDE_TARGET_MISMATCH / SUPERSEDE_TIMEOUT) в Added; смена поведения
  (легаси вторая вкладка теперь → 409 A_RUN_ALREADY_ACTIVE вместо второго
  параллельного стрима; каждый ход — ран в обоих режимах) в Changed.
- .env.example: три новых AI_CHAT_* (SUPERSEDE_TIMEOUT_MS / RECONCILE_INTERVAL_MS
  / INAPP_TOOL_CALL_CAP_MS, дефолты 10000/120000/120000, связь cap↔staleness).
- AGENTS.md:458: ран теперь универсален (оба режима), флаг autonomousRuns =
  только disconnect-семантика; плюс упоминание supersede.
- ai-chat.service.ts:1136: устаревший коммент про «legacy → socket signal» →
  оба режима на run-signal, guard runId защищает лишь no-handle fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:49:14 +03:00
agent_coder 0b9de2c25e fix(ai-chat): owner-gate stream() до supersede — закрыть утечку чужого runId (#487, F1)
stream() был единственным эндпоинтом ai-chat без assertOwnedChat: участник
того же воркспейса (НЕ владелец чата) мог послать POST /stream
{chatId:<чужой>, supersede:{runId:<любой>}} и (а) выудить активный runId
жертвы из ответа 409 SUPERSEDE_TARGET_MISMATCH, затем (б) requestStop чужого
рана. Добавляю owner-check в начале stream() (когда есть body.chatId), ровно
как в /stop и соседях — pre-hijack, чистый 403. Это заодно закрывает и
негейченную кросс-юзерную запись через тот же stream() (база #500).

Тест: не-владелец POST /stream с чужим chatId → ForbiddenException,
runId не утёк, supersede/requestStop не вызваны. MUTATION-VERIFY: снятие
assertOwnedChat делает тест красным (возвращается путь утечки).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:49:02 +03:00
agent_coder 123cba7de1 fix(mcp): убрать залипший маркер конфликта в context.ts (артефакт ребейза)
После ребейза в src/client/context.ts:209 остался одиночный маркер
`>>>>>>> 917c4064` без парного открытия — он попал в коммит и ломал сборку
всего пакета mcp (TS1185 Merge conflict marker), из-за чего не запускались
никакие node --test. Код выше маркера цел; удаляю только строку маркера.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:48:52 +03:00
agent_coder 001eb1c923 docs(mcp): точный коммент toolAbortSignal — set-and-leave, не restore (#487, ревью nit)
Внутреннее ревью: docstring поля/метода toolAbortSignal утверждал «restores the
prior value on unwind», но wrapInAppToolWithCap намеренно НЕ восстанавливает
сигнал (set-and-leave) — именно это заставляет брошенного проигравшего race
бросить на следующем safe-point; корректность держится на свежем клиенте на ход +
перезаписи следующим вызовом. Комментарии приведены в соответствие с механизмом.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:26:01 +03:00
agent_coder 002c931c6b feat(ai-chat): ретраи finalizeAssistant + owner-write приоритет + двусторонний reconcile (#487)
Раньше finalizeAssistant ставил finalized=true ДО записи и не ретраил → один
неудачный UPDATE = строка вечно 'streaming'; свип был boot-only; run-свип
безусловный — асимметрия «run succeeded / message streaming навсегда».

- finalizeAssistant: bounded-ретраи; once-гейт закрывается ТОЛЬКО после успешной
  записи; возвращает ok. Правило owner-write: терминальная запись owner'а
  условна на status='streaming' OR metadata.finalizeFailed (repo.finalizeOwner) —
  перетирает reconcile-штамп, но не проставленный терминал. ВСЕ status-only
  штампы reconcile (stampTerminalIfStreaming, sweepStreaming) пишут строго
  onlyIfStreaming И мёржат metadata.finalizeFailed:true (иначе поздний owner-write
  не перетрёт).
- Порядок: попытка message-finalize → ран финализируется ВСЕГДА; при провале
  message onFinish помечает ран 'error' (не 'completed'). Ран не гейтится на
  message.
- Периодический reconcile-джоб (setInterval, env-tunable) клаузами по порядку:
  (a) пере-драйв зомби; (b) message streaming + ран терминален → штамп по статусу
  рана (succeeded-ран + зависшая строка → 'aborted'+finalizeFailed, НЕ
  'completed'-empty); (c) run running + НЕТ entry И НЕТ zombie + staleness →
  aborted (гейт «нет entry» первичный, staleness от last-progress updatedAt,
  X=max(2×per-call cap,15мин)+boot-warn); (d) message streaming + возраст>X + нет
  активной run-строки → aborted (двойной гейт). isInterruptResume исключает
  finalizeFailed-строки. Оппортунистический одно-чатовый reconcile при старте хода
  (best-effort, не фейлит ход). sweepStreaming boot-only → периодический.

Тесты (реальная БД): owner finalizeOwner чистит finalizeFailed; штамп не перетирает
терминал; поздний owner-write перетирает aborted-штамп; клаузы b/c/d (+живой entry
не трогать, двойной гейт d); «убить БД на finish → после восстановления ни строка,
ни ран не застряли». Юниты: finalizeFailed исключает interrupt-resume;
reconcileStaleRuns «нет entry».

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:26:01 +03:00
agent_coder 307ad49324 feat(ai-chat): единый гейт конкурентности + серверный supersede (CAS) (#487)
Раньше в legacy-режиме (дефолт!) гейта конкурентности НЕ было вообще — проверка
409 сидела внутри if (autonomousRuns && chatId): два таба = два параллельных
стрима на один чат (интерливинг истории, падения convertToModelMessages).

Серверная часть (клиентская лестница ретраев — отдельной клиентской итерацией
FSM, см. ниже):
- Run-строка ОБЯЗАТЕЛЬНА в ОБОИХ режимах: legacy тоже проходит beginRun/
  finalizeRun (гейт партиального уникального индекса + runId в start-метаданных).
  Различие режимов сведено к семантике abort: legacy onClose при дисконнекте
  зовёт requestStop(runId) вместо аборта сокет-сигнала (которого streamText уже
  не потребляет). Второй таб в ЛЮБОМ режиме → 409 A_RUN_ALREADY_ACTIVE.
- Supersede CAS POST /stream { supersede: { runId: X } } (AiChatRunService
  .supersede): валидация X.chatId===body.chatId (иначе 400 SUPERSEDE_INVALID);
  нет активного рана → degrade в обычный send; активный ≠ X → 409
  SUPERSEDE_TARGET_MISMATCH + текущий runId; активный = X → requestStop →
  awaitSettled (таймаут W=10c) → «записано/сдался» (сдался → settleZombie
  применяет intended условным UPDATE) → ready; таймаут → 409 SUPERSEDE_TIMEOUT,
  ничего не персистится. W обоснован race-on-abort'ом коммита 1; DB-brownout →
  TIMEOUT штатен, W не увеличивать (env-tunable).
- Задокументированы ограничения: нет квиесценции сайд-эффектов (в промпт нового
  рана добавлена строка SUPERSEDE_NOTE «предыдущий ран прерван, его последние
  операции могли примениться с задержкой»); кража слота между освобождением и
  beginRun (→ MISMATCH с новым runId, бэкстоп — уникальный индекс).

Тесты: два параллельных старта (оба режима) → строго 409 второму; supersede при
живом долгом ТУЛЕ (не UPDATE-задержке) settle'ится быстро; все CAS-ветки; дубль
supersede-POST → degrade; зомби-путь через settleZombie; HTTP-маппинг ветвей.
Кейс «beginRun ok → insert user-msg упал → ран settled, слот свободен» покрыт
lifecycle-спекой против реального safety-net catch.

КЛИЕНТ (deferred, точно flagged): удаление лестницы SUPERSEDE_RETRY_DELAYS_MS/
isRunAlreadyActive/supersedeRetryRef и переход на supersede-в-body требуют
адопции runId из start-метаданных (сейчас клиент runId не трекает вовсе) — это
и есть та «итерация клиентского FSM», которую данный серверный контракт
разблокирует. Не трогаю фрагильный клиентский FSM без возможности E2E-валидации
в браузере, чтобы не внести непроверяемую регрессию.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:26:01 +03:00
agent_coder 1031ed1ae8 feat(ai-chat): зомби-механика finalizeRun + settledPromise + условные терминальные UPDATE (#487)
Раньше give-up-путь finalizeRun ВОССТАНАВЛИВАЛ entry (зомби неотличим от живого
рана), а терминальный runRepo.update был БЕЗУСЛОВНЫМ (последний писатель перетирал
терминальный статус).

- Все терминальные UPDATE ранов теперь УСЛОВНЫЕ: новый repo.finalizeIfActive
  пишет только пока строка pending|running (зеркально onlyIfStreaming у сообщений)
  → двойной settle схлопывается в benign no-op, терминальный статус не перетереть.
- При исчерпании ретраев finalizeRun НЕ восстанавливает entry, а оставляет
  ЗОМБИ-запись { terminalWriteFailed, intended:{status,error} } в отдельной map;
  settleZombie пере-драйвит intended условным UPDATE (зовётся reconcile/supersede/
  boot sweep). Зомби удаляется после успешного UPDATE или обнаружения строки уже
  терминальной.
- Per-run settledPromise в ОТДЕЛЬНОЙ map runId→deferred (переживает active.delete),
  создаётся в beginRun, резолвится ровно один раз исходом (записано/сдался); поздний
  подписчик через ранее взятую ссылку получает резолвленный; peekSettled: live
  deferred → зомби-синтез → undefined (читать строку). Обе map bounded.
- Задокументирована потеря (single-process): рестарт до пере-драйва → boot sweep
  пишет 'aborted' поверх фактического intended — неустранимо в phase 1.

Тесты: юниты give-up-пути (зомби вместо restore + notifier terminalWriteFailed +
settleZombie), двойного settle, позднего подписчика; интеграционные против реальной
Б, что условный finalizeIfActive не перетирает терминал и двойной settle схлопывается.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:26:01 +03:00
agent_coder 3b6634c6be fix(ai-chat): in-app тулы — race-on-abort + safe-points + per-call cap (#487)
In-app тул-обёртки отбрасывали второй аргумент options с abortSignal — это был
единственный класс тулов без отмены и без wall-clock cap. Контент-мутации идут
через collab-WS (mutatePageContent), не через axios, поэтому «прокинуть signal в
axios» главный write-путь не покрывал.

- Переиспользован race-паттерн wrapToolWithCallTimeout: каждый in-app тул гонится
  против композитного сигнала (Stop + per-call cap); на abort — реджект
  немедленно, проигравший промис отбрасывается (латентность мс, не сетевой
  teardown — от этого зависит таймаут supersede в коммите 3).
- Safe-point проверки сигнала между последовательными вызовами paginateAll и
  пре-коммитная проверка в mutatePageContent (+ реентрантный близнец) через
  DocmostClient.setToolAbortSignal, который обёртка публикует перед каждым вызовом.
- Per-call cap покрывает весь вызов, env-tunable (AI_CHAT_INAPP_TOOL_CALL_CAP_MS,
  дефолт 2 мин).
- Задокументировано ограничение (#487): abort между вызовами многовызовного
  write-тула оставляет частично применённую операцию — отмена гарантирует «новый
  вызов не стартует», не «запись не доехала».

Тесты (честное свойство «после Stop не стартует новый HTTP/WS-вызов»): юнит
wrapInAppToolWithCap (реджект-на-abort, отсутствие новых вызовов, per-call cap,
публикация сигнала) + mock-HTTP тест реального paginateAll safe-point.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 07:26:01 +03:00
92 changed files with 9232 additions and 2201 deletions
+41 -3
View File
@@ -225,11 +225,26 @@ MCP_DOCMOST_PASSWORD=
# Silence timeout (ms) for EXTERNAL-MCP transport ONLY (not the chat provider).
# Tighter than AI_STREAM_TIMEOUT_MS so a byte-silent/hung MCP server is broken in
# ~1 min instead of 15. Note it also cuts a legitimately long but byte-silent
# single tool call (a slow crawl that emits nothing until done) and an SSE
# transport idling >1 min BETWEEN tool calls. Default 60000 (1 min).
# ~1 min instead of 15. It cuts a legitimately long but byte-silent single tool
# call (a slow crawl that emits nothing until done) on the HTTP (streamable)
# transport, which opens a fresh request per call. The SSE transport — one
# long-lived body across many calls — is NO LONGER governed by this timeout
# (as of #489): its idle-BETWEEN-calls window has its own, raised bodyTimeout,
# AI_MCP_SSE_BODY_TIMEOUT_MS below. Default 60000 (1 min).
# AI_MCP_STREAM_TIMEOUT_MS=60000
# bodyTimeout (ms) for the EXTERNAL-MCP SSE transport ONLY (#489). The SSE
# transport holds ONE response body open across many tool calls, so undici's
# bodyTimeout (time between body bytes) counts the LEGITIMATE silence BETWEEN the
# model's tool calls, not just a hung single call. At the tight 1-min silence
# timeout above, a normal >1-min gap between calls would break the SSE socket and
# the cache would serve a dead client until TTL — so the SSE transport gets its
# OWN, RAISED bodyTimeout. A single stuck call is still bounded by the per-call
# cap (AI_MCP_CALL_TIMEOUT_MS), and a socket that does break is healed by the
# in-run transport-error retry. The HTTP (streamable) transport keeps the tight
# timeout. Default 600000 (10 min).
# AI_MCP_SSE_BODY_TIMEOUT_MS=600000
# Total wall-clock cap (ms) for ONE external MCP tool call (app-level, not
# transport). Aborts a tool that keeps the socket warm (SSE heartbeats / trickle)
# but never returns a result — which the silence timeout above never breaks.
@@ -288,6 +303,29 @@ MCP_DOCMOST_PASSWORD=
# registry is process-local).
# AI_CHAT_RESUMABLE_STREAM=false
# --- Run lifecycle tunables (#487) ---
# These govern the universal run machinery (every turn is now a first-class run,
# both modes) and rarely need changing.
#
# How long a server-side SUPERSEDE ("interrupt and send now") waits for the target
# run to settle after issuing Stop before it degrades to a 409 SUPERSEDE_TIMEOUT
# (nothing sent, the composer keeps the user's text). 10s is generous under a
# healthy DB; do NOT raise it to paper over a slow DB — a SUPERSEDE_TIMEOUT is the
# honest signal. Default 10000 (10s).
# AI_CHAT_SUPERSEDE_TIMEOUT_MS=10000
#
# How often the periodic bidirectional reconcile job runs (heals runs/messages
# left dangling by a crash or a lost terminal write). Default 120000 (2 min).
# AI_CHAT_RECONCILE_INTERVAL_MS=120000
#
# Wall-clock cap for a SINGLE in-app tool call (a long paginated read, or a content
# write whose collab commit hangs) — the per-call half of the composite abort
# signal every in-app tool is wrapped with (the other half is the turn's Stop).
# The reconcile staleness floor is derived as max(2 x this cap, 15min), so a very
# high value delays stale-run recovery (the server boot-warns above 30min). Default
# 120000 (2 min).
# AI_CHAT_INAPP_TOOL_CALL_CAP_MS=120000
# --- Anonymous public-share AI assistant ---
# Opt-in per workspace (AI settings -> "public share assistant"; off by default).
# When enabled, anonymous visitors of a published share can ask an AI about that
+41 -10
View File
@@ -124,9 +124,17 @@ jobs:
exit "$FAILED"
# A GENUINE counterexample: fast-check printed a shrunk minimal case and its
# reproducing seed into property-output.txt. File a dedup-guarded issue whose
# title prefix is UNIQUE to counterexamples, so an infra failure (handled by
# the next step under a different title) can never poison this dedup.
# reproducing seed into property-output.txt. File a dedup-guarded issue.
#
# Dedup is keyed on a HASH of the SHRUNK COUNTEREXAMPLE (the minimal failing
# input), NOT on the issue title prefix. Keying on the prefix would let a
# single open issue swallow every OTHER counterexample (a different bug B whose
# title shares the prefix would be treated as a duplicate and stay silent until
# the first issue is closed). Hashing the shrunk example instead means two
# DIFFERENT counterexamples get two DIFFERENT issues, while a re-find of the
# SAME counterexample still dedupes onto the existing one. The infra-failure
# step (below) still keys on its own distinct title, so it can never poison
# this dedup either.
- name: File counterexample issue
# always() is REQUIRED: the fuzz step exits nonzero on a failing shard,
# so a bare `if:` (implicitly success() && ...) would skip this step
@@ -146,25 +154,48 @@ jobs:
echo "No fast-check counterexample signature — infra failure, handled by the next step."
exit 0
fi
TITLE="${TITLE_PREFIX} (seed=${FAIL_SEED})"
# Extract the SHRUNK counterexample block: the "Counterexample:" line(s)
# up to (but excluding) the "Shrunk N time(s)" / "Got error" line. This is
# the minimal failing INPUT and is STABLE across the different seeds/paths
# that reach the same bug — unlike the seed, path, or shrink count (which
# precede/follow this block and vary run-to-run) and unlike the whole
# output (which embeds those varying parts). Hashing THIS is what makes the
# dedup identity the bug itself rather than an incidental run detail.
CE_TEXT=$(awk '/Counterexample:/{c=1} /Shrunk [0-9]+ time|Got error/{c=0} c{print}' property-output.txt)
if [ -z "$CE_TEXT" ]; then
# No parseable shrunk block (unexpected — the signature check above
# already confirmed fast-check output). Fall back to the reproducing
# seed so we still emit a stable identity instead of silently deduping.
CE_TEXT="seed:${FAIL_SEED}"
fi
# Stable short id: first 12 hex chars of sha256 over the counterexample.
CE_HASH=$(printf '%s' "$CE_TEXT" | sha256sum | cut -c1-12)
# Machine-readable marker embedded in the issue body; the open-issue search
# below matches on it (and on the hash in the title) so identity travels
# with the issue regardless of any human title edits.
CE_MARKER="<!-- counterexample-hash: ${CE_HASH} -->"
export CE_HASH CE_MARKER
TITLE="${TITLE_PREFIX} [${CE_HASH}] (seed=${FAIL_SEED})"
# Best-effort dedup: skip if an open issue with the counterexample title
# prefix already exists. A failure of this check must NOT block creation.
# Dedup on the counterexample hash: skip only if an OPEN issue already
# carries this exact hash (in its title or its body marker). A different
# counterexample has a different hash and is NOT deduped. A failure of this
# check must NOT block creation.
EXISTING=""
if EXISTING=$(curl -sS \
-H "Authorization: token ${GITHUB_TOKEN}" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues?state=open&limit=100"); then
if printf '%s' "$EXISTING" \
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const p=process.env.TITLE_PREFIX;process.exit(a.some(i=>typeof i.title==="string"&&i.title.startsWith(p))?0:1)})'; then
echo "An open '${TITLE_PREFIX}' issue already exists — skipping creation."
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const h=process.env.CE_HASH,m=process.env.CE_MARKER;process.exit(a.some(i=>(typeof i.title==="string"&&i.title.includes(h))||(typeof i.body==="string"&&i.body.includes(m)))?0:1)})'; then
echo "An open issue for counterexample ${CE_HASH} already exists — skipping creation."
exit 0
fi
fi
# Build the JSON body with the test output SAFELY escaped (never hand-
# interpolate the counterexample into JSON).
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed with a fast-check counterexample.\n\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nReproduce locally:\n\n```\nPROPERTY_SEED=%s PROPERTY_NUM_RUNS=%s pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/\n```\n\nfast-check shrinks the failure to a minimal counterexample. Commit it as a permanent fixture under `packages/prosemirror-markdown/test/fixtures/counterexamples/` + a case in `counterexamples.test.ts`, then FIX the converter (do not weaken a property). See `packages/prosemirror-markdown/README.md`.\n\nTail of the test output (contains the shrunk counterexample):\n\n```\n%s\n```\n' \
"$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$FAIL_SEED" "$NUM_RUNS" "$(tail -n 120 property-output.txt)")
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed with a fast-check counterexample.\n\n- counterexample hash: `%s`\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nReproduce locally:\n\n```\nPROPERTY_SEED=%s PROPERTY_NUM_RUNS=%s pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/\n```\n\nfast-check shrinks the failure to a minimal counterexample. Commit it as a permanent fixture under `packages/prosemirror-markdown/test/fixtures/counterexamples/` + a case in `counterexamples.test.ts`, then FIX the converter (do not weaken a property). See `packages/prosemirror-markdown/README.md`.\n\nTail of the test output (contains the shrunk counterexample):\n\n```\n%s\n```\n\n%s\n' \
"$CE_HASH" "$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$FAIL_SEED" "$NUM_RUNS" "$(tail -n 120 property-output.txt)" "$CE_MARKER")
jq -n --arg title "$TITLE" --arg body "$BODY_TEXT" \
'{title: $title, body: $body}' > payload.json
+3 -1
View File
@@ -455,7 +455,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
- `core/ai-chat/tools/` — the agent's ~40 read+write tools. Every tool runs under the **calling user's** CASL permissions via a per-user loopback access token (`docmost-client.loader.ts`), so the agent can never exceed what the user could do. Only **reversible** operations are exposed (page history + trash; no permanent delete). Agent edits get an "AI agent" provenance badge in page history (`20260616T130000-agent-provenance` migration).
- `core/ai-chat/embedding/` — RAG indexer + a BullMQ consumer on `AI_QUEUE` that embeds pages into `page_embeddings` (vector search), complementing Postgres full-text search. Pages are (re)indexed on edit; `AI_EMBEDDING_TIMEOUT_MS` bounds a hung embeddings endpoint.
- `core/ai-chat/external-mcp/` — admins can attach external MCP servers (e.g. Tavily) to give the agent web access. **`ssrf-guard.ts` validates outbound MCP URLs against SSRF** — keep that guard in the path when touching external-MCP connection logic.
- `core/ai-chat/ai-chat-run.service.ts` + `ai_chat_runs`**detached/autonomous agent runs** (`#184`), behind the per-workspace `settings.ai.autonomousRuns` flag (off by default). When on, a turn becomes a server-side RUN that survives a browser disconnect; only an explicit `POST /ai-chat/stop` ends it, and a client reconnects/live-follows via `POST /ai-chat/run`. **DEPLOY CONSTRAINT — single-instance only in phase 1:** Stop and the AbortController that backs it are process-local, so a Stop only aborts a run executing on the **same** replica that owns it (cross-instance pub/sub stop is phase 2). Do **not** enable `autonomousRuns` on a horizontally-scaled deployment (multiple replicas behind a load balancer, or Docmost cloud `CLOUD=true`) — run a single instance instead. The server logs a startup WARNING when it detects a multi-instance deployment (`CLOUD=true`) so the constraint is visible. The startup sweep settles any run left dangling by a restart.
- `core/ai-chat/ai-chat-run.service.ts` + `ai_chat_runs`**every agent turn is now a first-class server-side RUN** (`#184`, universalized in `#487`): its lifecycle is tracked in `ai_chat_runs` in **both** modes, and the single-active-run-per-chat concurrency gate is enforced universally (a legacy second tab now gets a clean `409 A_RUN_ALREADY_ACTIVE` instead of a second parallel stream that interleaved history). The per-workspace `settings.ai.autonomousRuns` flag (off by default) **no longer gates whether a turn is a run** — it now controls **only the browser-disconnect semantics**: when ON the run is *detached* (a disconnect leaves it executing server-side; only an explicit `POST /ai-chat/stop` ends it, and a client reconnects/live-follows via `POST /ai-chat/run`); when OFF (legacy) a disconnect ends the turn by stopping its run via the run's stop lever. `#487` also adds a server-side **supersede** CAS ("interrupt and send now") to `POST /ai-chat/stream` (`supersede: { runId }`): it atomically stops the chat's currently-active run and waits for it to settle before the new turn claims the slot, returning `SUPERSEDE_INVALID` / `SUPERSEDE_TARGET_MISMATCH` / `SUPERSEDE_TIMEOUT` on the non-proceed branches. **DEPLOY CONSTRAINT — single-instance only in phase 1:** Stop and the AbortController that backs it are process-local, so a Stop only aborts a run executing on the **same** replica that owns it (cross-instance pub/sub stop is phase 2). Do **not** enable `autonomousRuns` on a horizontally-scaled deployment (multiple replicas behind a load balancer, or Docmost cloud `CLOUD=true`) — run a single instance instead. The server logs a startup WARNING when it detects a multi-instance deployment (`CLOUD=true`) so the constraint is visible. The startup sweep settles any run left dangling by a restart.
### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
@@ -471,6 +471,8 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch carries TWO independent server fixes, each with its own tripwire test: (1) it disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`); (2) it fixes `writeToServerResponse`'s drain-hang — the loop awaited only `"drain"` under backpressure, so a mid-write client disconnect parked the pipe forever and leaked the reader/buffers until restart; it now races `"drain"` against `"close"`/`"error"`, cancels the reader on disconnect, and swallows the fire-and-forget read rejection (#486; tripwire: `apps/server/src/integrations/ai/ai-sdk-drain-hang.patch.spec.ts`). Both tripwires assert BOTH installed dist builds carry their patch marker. The patch MUST be re-created via `pnpm patch` when bumping `ai`.
- **Upstream tracking (report the analysis upstream, don't just carry it):** both `ai` fixes and the hocuspocus one are candidates for upstreaming so we can eventually drop the local patch — the analysis is already written up in each patch's `PATCH(...)` header comments. File (a) an upstream **issue** on `vercel/ai` for the O(n²) cumulative `partialOutput` accumulation (heap OOM), (b) an upstream **issue** on `vercel/ai` for the `writeToServerResponse` drain-hang, and (c) an upstream **PR** on `@hocuspocus/server` for the connect-vs-unload race (local marker `PATCH(gitmost #401)` in `patches/@hocuspocus__server@3.4.4.patch`). Do NOT edit the patch files to add links — the patch bytes feed `patch_hash` in `pnpm-lock.yaml` (`ai@6.0.134``e8c599b3…`), so any content change there desyncs the lockfile pin and breaks `pnpm install`; keep upstream references here instead.
- **`ai` version is split across the monorepo and MUST be aligned deliberately, NOT casually:** the server pins `ai@6.0.134` (patched, exact — the `patchedDependencies` key forces that version), while the client declares `ai@6.0.207` (unpatched — the server-side `writeToServerResponse`/`partialOutput` fixes are dead code in the browser, so the mismatch is currently benign but is real drift). Alignment is a **planned, install-gated step**, never a bare `package.json` edit: (1) choose the target version; (2) re-create ALL THREE patch hunks (partialOutput publish-each, the `DefaultStreamTextResult` lazy-`output` wiring, and the drain-hang race) against the target dist via `pnpm patch` — the line offsets shift between versions, so the current patch WILL fail to apply as-is; (3) run a full `pnpm install` so the lockfile + new `patch_hash` regenerate together; (4) confirm both tripwire specs still find their markers. `pnpm install` FAILS HARD on an unapplied patch — that failure is the guardrail, so treat the port as a deliberate plan rather than discovering it as a deploy-time surprise.
- **The MCP tool inventory in `SERVER_INSTRUCTIONS` is GENERATED from the registry** (`packages/mcp/src/server-instructions.ts`: `buildToolInventory()` over `SHARED_TOOL_SPECS`) and spliced into the hand-written routing prose (`ROUTING_PROSE`). So adding/renaming/removing a **shared** spec in `packages/mcp/src/tool-specs.ts` auto-updates the `<tool_inventory>` — no manual `SERVER_INSTRUCTIONS` edit needed. Only an **inline** MCP-only tool (those registered via `server.registerTool(...)` in `index.ts`, not through the registry) needs a one-line entry in `INLINE_MCP_INVENTORY`. Enforced by `packages/mcp/test/unit/tool-inventory.test.mjs`, which fails when a registered tool is missing from the generated inventory (there is no `EXCEPTIONS` opt-out anymore — every tool must appear). Update `ROUTING_PROSE` when a tool's *intent guidance* (when-to-use) changes. `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
+47
View File
@@ -202,6 +202,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
dangling by a restart. Phase 1 is single-instance-only (cross-instance Stop is
not yet reliable); the server warns at startup on a horizontally-scaled
deployment. (#184)
- **Server-side "interrupt and send now" (supersede) for AI chat.** `POST
/ai-chat/stream` now accepts a `supersede: { runId }` field: when the user sends
a new message while a run is active, the server atomically stops that run and
waits for it to settle before the new turn claims the chat's single run slot,
instead of the send being rejected as concurrent. The compare-and-set surfaces
three codes on its non-proceed branches — `SUPERSEDE_INVALID` (the targeted run
is malformed / belongs to another chat), `SUPERSEDE_TARGET_MISMATCH` (a
different run is now active; carries the current `activeRunId`), and
`SUPERSEDE_TIMEOUT` (the previous run did not stop within the settle window, so
nothing was sent and the composer keeps the text). Tunable via
`AI_CHAT_SUPERSEDE_TIMEOUT_MS` (default 10s). (#487)
- **Out-of-band page transfer via an in-RAM blob sandbox (`stash_page`).** A
new MCP tool serializes a whole page (its full ProseMirror JSON, with every
internal image/file mirrored) into an ephemeral in-RAM blob and returns only
@@ -282,6 +293,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **Every AI-chat turn is now a first-class server-side run, and one run per chat
is enforced in both modes.** The run machinery from `#184` was universalized: a
turn is tracked in `ai_chat_runs` and gated by the single-active-run-per-chat
index regardless of the `settings.ai.autonomousRuns` flag. **Behavior change:**
a second tab (or a double-submit) that starts a turn while one is already active
on the chat is now rejected up front with `409 A_RUN_ALREADY_ACTIVE` (carrying
the `activeRunId`); previously, on the legacy path, it opened a second parallel
stream on the same chat that interleaved history. The `autonomousRuns` flag no
longer controls whether a turn is a run — it now governs **only** the
browser-disconnect semantics (ON = detached/survives a disconnect; OFF = a
disconnect stops the run). (#487)
- **Vendor `ai` patch: upstream-tracking + version-alignment plan documented.**
The two local `ai@6.0.134` fixes (O(n²) `partialOutput` heap-OOM; the
`writeToServerResponse` drain-hang) and the hocuspocus connect-vs-unload race
now have explicit upstream-reporting and `ai`-version-alignment steps recorded
in `AGENTS.md` (client `ai@6.0.207` vs server `ai@6.0.134`-patched drift). The
patch bytes are unchanged — they feed the lockfile `patch_hash`, so the
alignment is called out as an install-gated plan rather than a bare version
bump. No runtime change.
- **Client markdown paste/copy and AI-chat rendering now go through the canonical
converter.** Pasting markdown into the editor, "Copy as markdown", the AI title
generator, and the AI-chat markdown renderer all now use
@@ -314,6 +344,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **A chat with one malformed message part no longer 500s on every turn, and a
failed send no longer duplicates the user's message.** Incoming client parts
are now whitelisted to `text` (a forged tool-result part can no longer reach
the persisted history or the model context), and the turn is converted BEFORE
the user row is inserted, so a mid-flight failure cannot leave a duplicate
user row that a retry then compounds. A single part that still fails to convert
degrades to a `[tool context omitted]` marker on that one row instead of
bricking the whole chat. (#489)
- **A transport drop to an external MCP server now heals within the same turn.**
On an undici transport error, a read-only MCP tool reconnects its server and
retries once within the run; a write is never auto-retried (it may already have
applied). One flapping server no longer nulls the shared client cache, so other
servers' cached clients are untouched. The SSE transport also gets a raised
body-timeout so a legitimate >1-min idle between the model's tool calls no
longer breaks a long-lived SSE socket (new `AI_MCP_SSE_BODY_TIMEOUT_MS`, default
10 min; see `.env.example`). (#489)
- **The server no longer runs out of heap during long autonomous agent runs.** A
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
snapshot of the ENTIRE turn text on every streamed text-delta when no output
+230 -78
View File
@@ -256,6 +256,9 @@
"Invite link": "Ссылка для приглашения",
"Copy": "Копировать",
"Copy to space": "Копировать в пространство",
"Copy chat": "Копировать чат",
"Dock to sidebar": "Закрепить в боковой панели",
"Undock": "Открепить",
"Copied": "Скопировано",
"Failed to export chat": "Не удалось экспортировать чат",
"Duplicate": "Дублировать",
@@ -285,6 +288,9 @@
"Alt text": "Альтернативный текст",
"Describe this for accessibility.": "Опишите это для специальных возможностей.",
"Add a description": "Добавить описание",
"Caption": "Подпись",
"Add a caption": "Добавить подпись",
"Shown below the image.": "Отображается под изображением.",
"Justify": "По ширине",
"Merge cells": "Объединить ячейки",
"Split cell": "Разделить ячейку",
@@ -388,22 +394,6 @@
"Quote": "Цитата",
"Image": "Изображение",
"Audio": "Аудио",
"Transcribe": "Транскрибировать",
"Transcribing…": "Транскрибация…",
"No speech detected": "Речь не распознана",
"Transcription failed": "Не удалось распознать речь",
"Voice dictation is not configured": "Голосовой ввод не настроен",
"Start dictation": "Начать диктовку",
"Stop recording": "Остановить запись",
"Microphone access denied": "Доступ к микрофону запрещён",
"No microphone found": "Микрофон не найден",
"Microphone is unavailable or already in use": "Микрофон недоступен или уже используется",
"Could not start recording": "Не удалось начать запись",
"Audio recording is not available in this browser/context": "Запись аудио недоступна в этом браузере/контексте",
"Dictation": "Диктовка",
"Dictation becomes available once the page finishes connecting": "Диктовка станет доступна после подключения к документу",
"No connection to the collaboration server — dictation unavailable": "Нет связи с сервером совместного редактирования — диктовка недоступна",
"This page is read-only": "Страница открыта только для чтения",
"Embed PDF": "Встроить PDF",
"Upload and embed a PDF file.": "Загрузите и встроите PDF-файл.",
"Embed as PDF": "Встроить как PDF",
@@ -419,9 +409,6 @@
"Footnote {{number}}": "Сноска {{number}}",
"Go to footnote": "Перейти к сноске",
"Back to reference": "Вернуться к ссылке",
"Back to references": "Вернуться к ссылкам",
"Back to reference {{label}}": "Вернуться к ссылке {{label}}",
"Empty footnote": "Пустая сноска",
"Math inline": "Строчная формула",
"Insert inline math equation.": "Вставить математическое выражение в строку.",
"Math block": "Блок формулы",
@@ -447,6 +434,9 @@
"{{count}} command available_other": "Доступно {{count}} команд",
"{{count}} result available_one": "Доступен 1 результат",
"{{count}} result available_other": "Доступно {{count}} результатов",
"{{count}} result found_one": "Найден {{count}} результат",
"{{count}} result found_few": "Найдено {{count}} результата",
"{{count}} result found_other": "Найдено {{count}} результатов",
"Equal columns": "Равные столбцы",
"Left sidebar": "Левая боковая панель",
"Right sidebar": "Правая боковая панель",
@@ -456,6 +446,7 @@
"Names do not match": "Названия не совпадают",
"Today, {{time}}": "Сегодня, {{time}}",
"Yesterday, {{time}}": "Вчера, {{time}}",
"now": "сейчас",
"Space created successfully": "Пространство успешно создано",
"Space updated successfully": "Пространство успешно обновлено",
"Space deleted successfully": "Пространство успешно удалено",
@@ -559,6 +550,7 @@
"Add 2FA method": "Добавить метод 2FA",
"Backup codes": "Резервные коды",
"Disable": "Отключить",
"disabled": "отключено",
"Invalid verification code": "Недействительный код подтверждения",
"New backup codes have been generated": "Новые резервные коды сгенерированы",
"Failed to regenerate backup codes": "Не удалось заново сгенерировать резервные коды",
@@ -702,62 +694,6 @@
"AI search": "Поиск ИИ",
"AI Answer": "Ответ ИИ",
"Ask AI": "Спросить ИИ",
"AI agent": "AI-агент",
"Take a look at the current document": "Посмотри текущий документ",
"Start automatically": "Запускать автоматически",
"When on, picking this role sends a launch message and starts the chat. When off, the role is selected and you type the first message yourself.": "Когда включено, выбор этой роли отправляет стартовое сообщение и начинает чат. Когда выключено, роль выбирается, а первое сообщение вы вводите сами.",
"Launch message": "Стартовое сообщение",
"Sent automatically when this role is picked. Leave empty to use the default text. Ignored when “Start automatically” is off.": "Отправляется автоматически при выборе этой роли. Оставьте пустым, чтобы использовать текст по умолчанию. Игнорируется, когда «Запускать автоматически» выключено.",
"AI agent is typing…": "AI-агент печатает…",
"{{name}} is typing…": "{{name}} печатает…",
"Thinking…": "Думаю…",
"Thinking… · {{count}} tokens": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_one": "Думаю… · {{count}} токен",
"Thinking… · {{count}} tokens_few": "Думаю… · {{count}} токена",
"Thinking… · {{count}} tokens_many": "Думаю… · {{count}} токенов",
"Thinking · {{count}} tokens": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_one": "Размышления · {{count}} токен",
"Thinking · {{count}} tokens_few": "Размышления · {{count}} токена",
"Thinking · {{count}} tokens_many": "Размышления · {{count}} токенов",
"Agent role": "Роль агента",
"AI chat": "AI-чат",
"AI chat is disabled for this workspace.": "AI-чат отключён для этого рабочего пространства.",
"Ask a question about this documentation.": "Задайте вопрос об этой документации.",
"Ask a question…": "Задайте вопрос…",
"Ask the AI agent anything about your workspace.": "Спросите AI-агента о чём угодно по вашему рабочему пространству.",
"Ask the AI agent…": "Спросите AI-агента…",
"Copy chat": "Копировать чат",
"Dock to sidebar": "Закрепить в боковой панели",
"Undock": "Открепить",
"Created successfully": "Успешно создано",
"Context size / model limit": "Размер контекста / лимит модели",
"Context window (tokens)": "Окно контекста (токены)",
"Shown as used / total in the chat header. Leave empty to hide the limit.": "Показывается в шапке чата как использовано / всего. Пусто — лимит скрыт.",
"Delete this chat?": "Удалить этот чат?",
"Deleted successfully": "Успешно удалено",
"AI agent «{{role}}» on behalf of {{person}}": "AI-агент «{{role}}» от имени {{person}}",
"AI agent {{name}}": "AI-агент {{name}}",
"Failed to delete chat": "Не удалось удалить чат",
"Failed to rename chat": "Не удалось переименовать чат",
"Failed": "Ошибка",
"OK · {{n}}": "OK · {{n}}",
"Test": "Тест",
"No tools available": "Инструменты недоступны",
"Available tools": "Доступные инструменты",
"Minimize": "Свернуть",
"No chats yet.": "Чатов пока нет.",
"Send": "Отправить",
"Send when the agent finishes": "Отправить, когда агент закончит",
"Queue message": "Поставить в очередь",
"Remove queued message": "Убрать из очереди",
"Send now": "Отправить сейчас",
"Interrupt and send now": "Прервать и отправить сейчас",
"Something went wrong": "Что-то пошло не так",
"Stop": "Стоп",
"The AI agent could not respond. Please try again.": "AI-агент не смог ответить. Попробуйте ещё раз.",
"The AI provider is not configured. Ask an administrator to set it up.": "AI-провайдер не настроен. Попросите администратора настроить его.",
"Universal assistant": "Универсальный ассистент",
"You": "Вы",
"AI is thinking...": "ИИ обрабатывает запрос...",
"Thinking": "Думаю",
"Ask a question...": "Задайте вопрос...",
@@ -784,8 +720,40 @@
"Manage API keys for all users in the workspace. View the <anchor>API documentation</anchor> for usage details.": "Управляйте API-ключами для всех пользователей в рабочем пространстве. Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>API documentation</anchor> for usage details.": "Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>MCP documentation</anchor>.": "Смотрите <anchor>документацию по MCP</anchor>.",
"Instructions": нструкции",
"AI / Models": И / Модели",
"AI / External tools (MCP)": "ИИ / Внешние инструменты (MCP)",
"Add server": "Добавить сервер",
"Edit server": "Изменить сервер",
"Delete server": "Удалить сервер",
"Are you sure you want to delete this MCP server?": "Вы уверены, что хотите удалить этот MCP-сервер?",
"No external servers configured": "Внешние серверы не настроены",
"Server name": "Имя сервера",
"Transport": "Транспорт",
"URL": "URL",
"Authorization header": "Заголовок авторизации",
"Tool allowlist": "Список разрешённых инструментов",
"Optional. Leave empty to allow all tools the server exposes.": "Необязательно. Оставьте пустым, чтобы разрешить все инструменты, которые предоставляет сервер.",
"Optional guidance for the agent on how and when to use this server's tools. Injected into the system prompt. The server's tools are namespaced as \"<server name>_*\".": "Необязательное указание агенту, как и когда использовать инструменты этого сервера. Добавляется в системный промпт. Инструменты сервера именуются с префиксом «<имя сервера>_*».",
"Test": "Тест",
"Available tools": "Доступные инструменты",
"No tools available": "Инструменты недоступны",
"Failed": "Ошибка",
"OK · {{n}}": "OK · {{n}}",
"Created successfully": "Успешно создано",
"Deleted successfully": "Успешно удалено",
"Clear": "Очистить",
"Provider": "Провайдер",
"•••• set": "•••• задан",
"Clear key": "Очистить ключ",
"Base URL": "Базовый URL",
"Chat model": "Модель чата",
"Embedding model": "Модель эмбеддингов",
"System message": "Системное сообщение",
"A built-in safety framework is always appended.": "Встроенный набор правил безопасности всегда добавляется автоматически.",
"Test connection": "Проверить соединение",
"Connection successful": "Соединение установлено",
"Connection failed": "Не удалось установить соединение",
"Only workspace admins can manage AI provider settings.": "Управлять настройками провайдера ИИ могут только администраторы рабочего пространства.",
"Sources": "Источники",
"AI Answers not available for attachments": "Ответы ИИ недоступны для вложений",
"No answer available": "Ответ недоступен",
@@ -1013,6 +981,7 @@
"Try again": "Попробовать снова",
"Untitled chat": "Чат без названия",
"No document": "Без документа",
"You": "Вы",
"What can I help you with?": "Чем я могу вам помочь?",
"Are you sure you want to revoke this {{credential}}": "Вы уверены, что хотите отозвать этот {{credential}}",
"Automatically provision users and groups from your identity provider via SCIM.": "Автоматически предоставляйте доступ пользователям и группам из вашего провайдера удостоверений через SCIM.",
@@ -1041,6 +1010,9 @@
"Page menu": "Меню страницы",
"Expand": "Развернуть",
"Collapse": "Свернуть",
"Expand all": "Развернуть все",
"Collapse all": "Свернуть все",
"Couldn't expand the tree: {{reason}}": "Не удалось развернуть дерево: {{reason}}",
"Comment menu": "Меню комментария",
"Group menu": "Меню группы",
"Show hidden breadcrumbs": "Показать скрытые хлебные крошки",
@@ -1077,7 +1049,7 @@
"Search pages and spaces...": "Поиск страниц и пространств...",
"No results found": "Результаты не найдены",
"You don't have permission to create pages here": "У вас нет прав на создание страниц здесь",
"Chat menu": "Меню чата",
"Chat menu for {{title}}": "Меню чата для {{title}}",
"API key menu": "Меню API-ключа",
"Jump to comment selection": "Перейти к выбору комментария",
"Slash commands": "Команды со слешем",
@@ -1131,6 +1103,9 @@
"Undo": "Отменить",
"Redo": "Повторить",
"Backlinks": "Обратные ссылки",
"Back to references": "Вернуться к ссылкам",
"Back to reference {{label}}": "Вернуться к ссылке {{label}}",
"Empty footnote": "Пустая сноска",
"Last updated by": "Последний изменивший",
"Last updated": "Последнее обновление",
"Stats": "Статистика",
@@ -1164,6 +1139,7 @@
"Page title": "Заголовок страницы",
"Page content": "Содержимое страницы",
"Member actions": "Действия с участником",
"Member actions for {{name}}": "Действия с участником {{name}}",
"Toggle password visibility": "Переключить видимость пароля",
"Send comment": "Отправить комментарий",
"Token actions": "Действия с токеном",
@@ -1183,11 +1159,187 @@
"Removed from favorites": "Удалено из избранного",
"Added {{name}} to favorites": "{{name}} добавлено в избранное",
"Removed {{name}} from favorites": "{{name}} удалено из избранного",
"Label added": "Метка добавлена",
"Label removed": "Метка удалена",
"Image updated": "Изображение обновлено",
"Unsupported image type": "Неподдерживаемый тип изображения",
"Member deactivated": "Участник деактивирован",
"Member activated": "Участник активирован",
"Name is required": "Укажите имя",
"Name must be 40 characters or fewer": "Имя должно содержать не более 40 символов",
"Group name must be at least 2 characters": "Название группы должно содержать не менее 2 символов",
"Group name must be 100 characters or fewer": "Название группы должно содержать не более 100 символов",
"Description must be 500 characters or fewer": "Описание должно содержать не более 500 символов",
"Invalid invitation link": "Недействительная ссылка-приглашение",
"Page menu for {{name}}": "Меню страницы для {{name}}",
"Create subpage of {{name}}": "Создать подстраницу для {{name}}",
"AI chat": "AI-чат",
"Ask a question about this documentation.": "Задайте вопрос об этой документации.",
"Ask a question…": "Задайте вопрос…",
"Thinking…": "Думаю…",
"Thinking… · {{count}} tokens": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_one": "Думаю… · {{count}} токен",
"Thinking… · {{count}} tokens_few": "Думаю… · {{count}} токена",
"Thinking… · {{count}} tokens_many": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_other": "Думаю… · {{count}} токенов",
"Thinking · {{count}} tokens": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_one": "Размышления · {{count}} токен",
"Thinking · {{count}} tokens_few": "Размышления · {{count}} токена",
"Thinking · {{count}} tokens_many": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_other": "Размышления · {{count}} токенов",
"The assistant is unavailable right now. Please try again.": "Ассистент сейчас недоступен. Попробуйте ещё раз.",
"Public share assistant": "Ассистент публичного доступа",
"Let anonymous visitors of public shares ask an AI assistant scoped to that share's pages. You pay for the tokens.": "Позвольте анонимным посетителям публичных ссылок обращаться к ИИ-ассистенту в рамках страниц этой публикации. Токены оплачиваете вы.",
"Public assistant model": "Модель публичного ассистента",
"Defaults to the chat model": "По умолчанию используется модель чата",
"Optional cheaper model id for the public assistant. Empty uses the chat model above.": "Необязательный более дешёвый идентификатор модели для публичного ассистента. Если пусто, используется модель чата выше.",
"Assistant identity": "Личность ассистента",
"Pick an agent role whose persona the public assistant adopts. The safety rules always still apply.": "Выберите роль агента, чью личность примет публичный ассистент. Правила безопасности всегда остаются в силе.",
"Built-in assistant persona": "Встроенная личность ассистента",
"Minimize": "Свернуть",
"Context size / model limit": "Размер контекста / лимит модели",
"Context window (tokens)": "Окно контекста (токены)",
"Shown as used / total in the chat header. Leave empty to hide the limit.": "Показывается в шапке чата как использовано / всего. Пусто — лимит скрыт.",
"AI agent": "AI-агент",
"Take a look at the current document": "Посмотри текущий документ",
"AI agent is typing…": "AI-агент печатает…",
"{{name}} is typing…": "{{name}} печатает…",
"Send": "Отправить",
"Send when the agent finishes": "Отправить, когда агент закончит",
"Queue message": "Поставить в очередь",
"Remove queued message": "Убрать из очереди",
"Send now": "Отправить сейчас",
"Interrupt and send now": "Прервать и отправить сейчас",
"Stop": "Стоп",
"Response stopped.": "Ответ остановлен.",
"Connection lost — the answer was interrupted.": "Соединение потеряно — ответ был прерван.",
"Response stopped (manually or the connection dropped).": "Ответ остановлен (вручную или из-за разрыва соединения).",
"Chat menu": "Меню чата",
"No chats yet.": "Чатов пока нет.",
"Delete this chat?": "Удалить этот чат?",
"Ask the AI agent…": "Спросите AI-агента…",
"Ask the AI agent anything about your workspace.": "Спросите AI-агента о чём угодно по вашему рабочему пространству.",
"Failed to rename chat": "Не удалось переименовать чат",
"Failed to delete chat": "Не удалось удалить чат",
"Something went wrong": "Что-то пошло не так",
"AI chat is disabled for this workspace.": "AI-чат отключён для этого рабочего пространства.",
"The AI provider is not configured. Ask an administrator to set it up.": "AI-провайдер не настроен. Попросите администратора настроить его.",
"The AI agent could not respond. Please try again.": "AI-агент не смог ответить. Попробуйте ещё раз.",
"Searched pages": "Поиск по страницам",
"Read page": "Прочитана страница",
"Created page": "Создана страница",
"Updated page": "Обновлена страница",
"Renamed page": "Переименована страница",
"Moved page": "Перемещена страница",
"Deleted page (to trash)": "Удалена страница (в корзину)",
"Commented": "Добавлен комментарий",
"Resolved comment": "Комментарий решён",
"Ran tool {{name}}": "Выполнен инструмент {{name}}",
"AI agent «{{role}}» on behalf of {{person}}": "AI-агент «{{role}}» от имени {{person}}",
"AI agent {{name}}": "AI-агент {{name}}",
"Endpoints": "Эндпоинты",
"where we fetch models": "откуда мы получаем модели",
"All endpoints are OpenAI-compatible. Point the Base URL at OpenAI, OpenRouter, a local Ollama, or any self-hosted server.": "Все эндпоинты совместимы с OpenAI. Укажите в базовом URL адрес OpenAI, OpenRouter, локального Ollama или любого self-hosted сервера.",
"Chat / LLM": "Чат / LLM",
"root": "корневой",
"Semantic search": "Семантический поиск",
"Voice / STT": "Голос / STT",
"Voice dictation": "Голосовой ввод",
"Streaming dictation": "Потоковый голосовой ввод",
"Transcribe as you speak, cutting on pauses": "Транскрибирование по мере речи, с разбивкой на паузах",
"Voice dictation is not available yet.": "Голосовой ввод пока недоступен.",
"Test endpoint": "Проверить эндпоинт",
"Save and test": "Сохранить и проверить",
"Save endpoints": "Сохранить эндпоинты",
"Configured and enabled": "Настроено и включено",
"Configured but disabled": "Настроено, но отключено",
"Enabled but not configured": "Включено, но не настроено",
"Not configured": "Не настроено",
"External tools": "Внешние инструменты",
"Gitmost as MCP client": "Gitmost как MCP-клиент",
"Servers the agent calls out to.": "Серверы, к которым обращается агент.",
"MCP server": "MCP-сервер",
"expose the workspace": "открыть доступ к рабочему пространству",
"Enable MCP server": "Включить MCP-сервер",
"Exposes the workspace as an MCP server at /mcp — this provides a capability, it doesn't consume a model.": "Открывает рабочее пространство как MCP-сервер по адресу /mcp — это предоставляет возможность, а не потребляет модель.",
"Resolves to {{url}}": "Разрешается в {{url}}",
"Model": "Модель",
"Done": "Готово",
"shared prompt · safety framework appended automatically": "общий промпт · правила безопасности добавляются автоматически",
"/v1/chat/completions · root endpoint — Embeddings and Voice inherit its URL and key": "/v1/chat/completions · корневой эндпоинт — Эмбеддинги и Голос наследуют его URL и ключ",
"/v1/embeddings · embeds pages so semantic search can find them": "/v1/embeddings · создаёт эмбеддинги страниц, чтобы их находил семантический поиск",
"/v1/audio/transcriptions · works with local whisper (speaches / faster-whisper-server)": "/v1/audio/transcriptions · работает с локальным whisper (speaches / faster-whisper-server)",
"Vector search · requires pgvector": "Векторный поиск · требуется pgvector",
"Embedding API key": "API-ключ для эмбеддингов",
"Embeddings": "Эмбеддинги",
"Leave empty to use the chat API key": "Оставьте пустым, чтобы использовать API-ключ чата",
"Leave empty to use the chat base URL": "Оставьте пустым, чтобы использовать базовый URL чата",
"Reindex now": "Переиндексировать сейчас",
"Start dictation": "Начать диктовку",
"Stop recording": "Остановить запись",
"Transcribing…": "Транскрибация…",
"Microphone access denied": "Доступ к микрофону запрещён",
"No microphone found": "Микрофон не найден",
"Could not start recording": "Не удалось начать запись",
"Transcription failed": "Не удалось распознать речь",
"Transcribe": "Транскрибировать",
"No speech detected": "Речь не распознана",
"Voice dictation is not configured": "Голосовой ввод не настроен",
"Microphone is unavailable or already in use": "Микрофон недоступен или уже используется",
"Audio recording is not available in this browser/context": "Запись аудио недоступна в этом браузере/контексте",
"Dictation": "Диктовка",
"Dictation becomes available once the page finishes connecting": "Диктовка станет доступна после подключения к документу",
"No connection to the collaboration server — dictation unavailable": "Нет связи с сервером совместного редактирования — диктовка недоступна",
"This page is read-only": "Страница открыта только для чтения",
"Request format": "Формат запроса",
"How transcription requests are sent to the endpoint": "Как запросы на транскрибирование отправляются на эндпоинт",
"OpenAI-compatible (multipart/form-data)": "Совместимо с OpenAI (multipart/form-data)",
"OpenRouter (JSON, base64 audio)": "OpenRouter (JSON, аудио в base64)",
"Dictation language": "Язык диктовки",
"Auto-detect": "Автоопределение",
"Spoken language hint sent to the transcription model. Auto-detect lets the model decide.": "Подсказка языка речи для модели транскрипции. «Автоопределение» оставляет выбор за моделью.",
"Agent role": "Роль агента",
"Universal assistant": "Универсальный ассистент",
"Add role": "Добавить роль",
"Edit role": "Изменить роль",
"Role name": "Название роли",
"e.g. Proofreader": "напр. Корректор",
"Optional. Shown as the chat badge.": "Необязательно. Отображается как значок чата.",
"Optional. A short note about what this role does.": "Необязательно. Краткое описание того, что делает эта роль.",
"Instructions": "Инструкции",
"The built-in safety framework is always added automatically.": "Встроенный набор правил безопасности всегда добавляется автоматически.",
"Model provider override": "Переопределение провайдера модели",
"Optional. Defaults to the workspace provider.": "Необязательно. По умолчанию используется провайдер рабочего пространства.",
"Model override": "Переопределение модели",
"Optional. Defaults to the workspace model.": "Необязательно. По умолчанию используется модель рабочего пространства.",
"e.g. gpt-4o-mini": "напр. gpt-4o-mini",
"If you choose a different provider, it must already be configured in AI settings.": "Если вы выбираете другого провайдера, он уже должен быть настроен в настройках ИИ.",
"Start automatically": "Запускать автоматически",
"When on, picking this role sends a launch message and starts the chat. When off, the role is selected and you type the first message yourself.": "Когда включено, выбор этой роли отправляет стартовое сообщение и начинает чат. Когда выключено, роль выбирается, а первое сообщение вы вводите сами.",
"Launch message": "Стартовое сообщение",
"Sent automatically when this role is picked. Leave empty to use the default text. Ignored when “Start automatically” is off.": "Отправляется автоматически при выборе этой роли. Оставьте пустым, чтобы использовать текст по умолчанию. Игнорируется, когда «Запускать автоматически» выключено.",
"Agent roles": "Роли агента",
"Reusable presets that shape the agent's behavior (and optionally its model). Picked when starting a new chat.": "Многоразовые пресеты, определяющие поведение агента (и, при желании, его модель). Выбираются при запуске нового чата.",
"No roles configured": "Роли не настроены",
"Delete role": "Удалить роль",
"Are you sure you want to delete this role?": "Вы уверены, что хотите удалить эту роль?",
"HTML embed": "HTML-вставка",
"Edit HTML embed": "Изменить HTML-вставку",
"HTML embed is disabled in this workspace": "HTML-вставки отключены в этом рабочем пространстве",
"Click to add HTML / CSS / JS": "Нажмите, чтобы добавить HTML / CSS / JS",
"This HTML/CSS/JS runs in a sandboxed frame and cannot access the viewer's session, cookies, or API.": "Этот HTML/CSS/JS выполняется в изолированном фрейме и не имеет доступа к сессии, cookie или API просматривающего.",
"<script>...</script>": "<script>...</script>",
"Height (px, blank = auto)": "Высота (px, пусто = авто)",
"advanced": "дополнительно",
"Enable HTML embed": "Включить HTML-вставки",
"Allow members to insert raw HTML/CSS/JavaScript blocks. The block renders in a sandboxed frame and cannot access the viewer's session, cookies, or API. Off by default.": "Разрешить участникам вставлять блоки с необработанным HTML/CSS/JavaScript. Блок отображается в изолированном фрейме и не имеет доступа к сессии, cookie или API просматривающего. По умолчанию выключено.",
"When enabled, any member can insert an HTML embed block. The toggle just enables or disables the block type workspace-wide.": "Когда включено, любой участник может вставить блок HTML-вставки. Переключатель просто включает или отключает этот тип блока во всём рабочем пространстве.",
"Embeds run inside a sandboxed iframe with a separate origin, so they cannot read or modify the page they are embedded in.": "Вставки выполняются в изолированном iframe с отдельным источником, поэтому они не могут читать или изменять страницу, в которую встроены.",
"Turning this off hides existing embeds (they render as a disabled placeholder) and stops serving them on public share pages.": "Отключение этой опции скрывает существующие вставки (они отображаются как отключённая заглушка) и прекращает их показ на публичных страницах.",
"Analytics / tracker": "Аналитика / трекер",
"Injected verbatim into the <head> of PUBLIC SHARE pages only (same-origin). For analytics snippets (Google Analytics, Yandex.Metrika, etc.). Admin only.": "Вставляется дословно в <head> только ПУБЛИЧНЫХ страниц (тот же источник). Для сниппетов аналитики (Google Analytics, Яндекс.Метрика и т. п.). Только для администраторов.",
"Go to login page": "Перейти на страницу входа",
"Move to space": "Переместить в пространство",
"Float left (wrap text)": "Обтекание слева",
"Float right (wrap text)": "Обтекание справа",
"Inline (side by side)": "В ряд",
@@ -1199,6 +1351,7 @@
"Showing {{count}} subpages_one": "Показано {{count}} подстраница",
"Showing {{count}} subpages_few": "Показано {{count}} подстраницы",
"Showing {{count}} subpages_many": "Показано {{count}} подстраниц",
"Showing {{count}} subpages_other": "Показано {{count}} подстраниц",
"Protocol": "Протокол",
"How chat requests are sent and how reasoning is surfaced": "Как отправляются запросы чата и как показывается reasoning",
"OpenAI-compatible (surfaces reasoning)": "OpenAI-совместимый (показывает reasoning)",
@@ -1268,7 +1421,6 @@
"Retry": "Повторить",
"The catalog is empty": "Каталог пуст",
"No role bundles are published for this language yet. Try switching the content language.": "Для этого языка ещё не опубликовано ни одного набора ролей. Попробуйте сменить язык контента.",
"No roles configured": "Роли не настроены",
"Already up to date": "Уже актуальна",
"Updated to the latest version": "Обновлено до последней версии",
"This role is no longer in the catalog": "Эта роль больше не представлена в каталоге",
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { isChunkLoadError } from "./chunk-load-error-boundary";
import { isChunkLoadError, shouldAutoReload } from "./chunk-load-error-boundary";
// The detector decides whether a caught render error is a stale-deploy chunk-404
// (→ auto-reload to fetch the new manifest) vs a genuine app error (→ generic
@@ -35,3 +35,31 @@ describe("isChunkLoadError", () => {
expect(isChunkLoadError(err)).toBe(false);
});
});
// The window gate replaces the old one-shot flag: it must permit recovery across
// several deploys in one tab (each > window apart) while still stopping an infinite
// reload loop when a lazy chunk is permanently broken (a second failure < window).
describe("shouldAutoReload", () => {
const WINDOW = 5 * 60 * 1000;
const NOW = 1_000_000_000_000;
it("allows a reload when we have never auto-reloaded", () => {
expect(shouldAutoReload(NOW, null, WINDOW)).toBe(true);
});
it("allows a reload when the last one was 6 minutes ago (outside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 6 * 60 * 1000, WINDOW)).toBe(true);
});
it("blocks a reload when the last one was 1 minute ago (inside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 1 * 60 * 1000, WINDOW)).toBe(false);
});
it("blocks a reload exactly at the window boundary (not strictly older)", () => {
expect(shouldAutoReload(NOW, NOW - WINDOW, WINDOW)).toBe(false);
});
it("allows a reload when the stored timestamp is unparseable (NaN)", () => {
expect(shouldAutoReload(NOW, NaN, WINDOW)).toBe(true);
});
});
@@ -2,7 +2,25 @@ import { ReactNode } from "react";
import { ErrorBoundary } from "react-error-boundary";
import { Button, Center, Stack, Text } from "@mantine/core";
const RELOAD_FLAG = "chunk-reload-attempted";
// sessionStorage key holding the epoch-ms timestamp of the last automatic reload.
const RELOAD_AT_KEY = "chunk-reload-at";
// Allow at most one automatic reload per this window. A stale-deploy 404 is cured
// by a single reload, so anything inside the window is treated as a reload loop
// (permanently-broken chunk) and falls through to the manual UI. A window (rather
// than a one-shot flag) lets a SECOND deploy in the same tab's lifetime recover too.
const RELOAD_WINDOW_MS = 5 * 60 * 1000;
// Pure window decision, unit-tested in isolation: auto-reload only if we have never
// auto-reloaded (lastReloadAt null/NaN) or the last one was strictly older than the
// window. Anything inside the window is suppressed to break an infinite reload loop.
export function shouldAutoReload(
now: number,
lastReloadAt: number | null,
windowMs: number,
): boolean {
if (lastReloadAt === null || Number.isNaN(lastReloadAt)) return true;
return now - lastReloadAt > windowMs;
}
// Heuristic detection of a failed dynamic import. Since the code-splitting work,
// every route (plus Aside / AiChatWindow) is React.lazy: when a new deploy
@@ -24,12 +42,16 @@ export function isChunkLoadError(error: unknown): boolean {
function handleError(error: unknown) {
if (!isChunkLoadError(error)) return;
// A stale-chunk 404 is cured by a full reload that re-fetches index.html and
// the new chunk manifest. Auto-reload once, guarding against a reload loop
// (e.g. a genuinely missing chunk) with a one-shot sessionStorage flag. If the
// flag is already set we fall through to the manual recovery UI below.
// the new chunk manifest. Auto-reload at most once per RELOAD_WINDOW_MS: this
// recovers across multiple deploys in a single tab's lifetime, yet a
// permanently-broken lazy chunk (which would loop) is stopped after the first
// reload and falls through to the manual recovery UI below.
try {
if (sessionStorage.getItem(RELOAD_FLAG)) return;
sessionStorage.setItem(RELOAD_FLAG, "1");
const raw = sessionStorage.getItem(RELOAD_AT_KEY);
const lastReloadAt = raw === null ? null : Number.parseInt(raw, 10);
const now = Date.now();
if (!shouldAutoReload(now, lastReloadAt, RELOAD_WINDOW_MS)) return;
sessionStorage.setItem(RELOAD_AT_KEY, String(now));
} catch {
// sessionStorage unavailable (private mode / disabled): skip the automatic
// reload rather than risk an unguarded loop; the fallback UI still recovers.
@@ -86,19 +86,11 @@ const MIN_HEIGHT = 400;
// Margin kept between the window and the viewport edges while dragging.
const EDGE_MARGIN = 8;
// #184 phase 1.5 / #430: backstop for the degraded-poll fallback. The poll is
// armed when a resume attempt could not attach to the live run and disarmed by the
// thread on settle / local stream; this cap is the ONLY backstop against an endless
// tick (a stuck 'streaming' row before the boot-sweep, or a user-tail 204 with no
// run).
//
// #430: measured from RUN ACTIVITY, not from arm-time. A real autonomous run takes
// 11-25 min — longer than a fixed 10-min-from-start cap, which used to cut the poll
// off mid-run. Instead we cap on INACTIVITY: keep polling as long as the run is
// still making progress (its persisted rows keep changing), and only give up after
// this long with NO new activity. A genuinely stuck run produces no row changes, so
// the idle cap still bounds it; a long-but-progressing run polls to completion.
const DEGRADED_POLL_IDLE_MAX_MS = 10 * 60_000;
// #184 phase 1.5 / #430 / #488: the degraded-poll fallback. The window owns only
// a DUMB 2.5s timer, gated by an armed flag; the THREAD's run-lifecycle FSM owns
// arm/disarm AND the inactivity cap that turns a stuck run into a `stalled` banner
// (#488 commit 4a — the cap moved into the thread so polling->stalled is a single
// FSM transition; the window no longer silently stops polling at the cap).
/** Compact token formatter: 1.2M / 3.4k / 950. */
function formatTokens(n: number): string {
@@ -259,17 +251,13 @@ export default function AiChatWindow() {
[roles],
);
// #184 phase 1.5: degraded-poll fallback (replaces the F4/F5/F7 latches). When
// ChatThread could not attach to a still-running run it arms this via
// onResumeFallback(true); the thread disarms it on settle / local stream. The
// window only OWNS the timer (armedAtRef stamps when it was armed for the cap).
// #184 phase 1.5 / #488: degraded-poll fallback. ChatThread's FSM arms this via
// onResumeFallback(true) when it enters a poll-bearing recovery (attach 204 /
// starved finish / stop) and disarms it on settle / local stream / stalled. The
// window owns ONLY the dumb 2.5s timer; the THREAD owns arm/disarm AND the
// inactivity cap (a stuck run -> the thread's `stalled` banner disarms this).
const [degradedPoll, setDegradedPoll] = useState(false);
// #430: timestamp of the LAST run activity while the poll is armed — stamped on
// arm and re-stamped whenever the polled rows change (see the effect below). The
// idle cap is measured from this, so a long-but-progressing run keeps polling.
const lastActivityAtRef = useRef(0);
const onResumeFallback = useCallback((active: boolean): void => {
if (active) lastActivityAtRef.current = Date.now();
setDegradedPoll(active);
}, []);
// Reset the degraded poll whenever the open chat changes: it is scoped to the
@@ -281,33 +269,17 @@ export default function AiChatWindow() {
const { data: messageRows, isLoading: messagesLoading } =
useAiChatMessagesQuery(
activeChatId ?? undefined,
// DELIBERATELY DUMB (invariant 8 / task 2.4): poll every 2.5s while armed
// and while the run is still active (#430: under the INACTIVITY cap, not a
// fixed-from-start cap); otherwise off. NO error checks (TanStack v5 resets
// fetchFailureCount each fetch, so consecutive errors are not expressible —
// and the poll must survive a server restart) and NO tail checks (the
// settled/local-stream semantics live in ChatThread, which disarms via
// onResumeFallback(false)). The idle cap is the only backstop.
() =>
degradedPoll === true &&
Date.now() - lastActivityAtRef.current < DEGRADED_POLL_IDLE_MAX_MS
? 2500
: false,
// DELIBERATELY DUMB: poll every 2.5s WHILE ARMED, otherwise off. NO error
// checks (TanStack resets fetchFailureCount each fetch; the poll must survive
// a server restart), NO tail checks, NO cap here — the settled/stalled/idle-cap
// semantics all live in ChatThread's FSM, which disarms via onResumeFallback.
() => (degradedPoll === true ? 2500 : false),
// #344: gate on windowOpen too — no message history is fetched (and no
// degraded poll runs) while the window is closed; it loads when the window
// opens with an active chat.
windowOpen,
);
// #430: re-stamp the activity clock whenever the polled rows change while the
// poll is armed. TanStack keeps the same `messageRows` reference across refetches
// that return deep-equal data (structural sharing), so a new reference means the
// run genuinely progressed — which extends the inactivity cap above. A stuck run
// yields no reference change, so the cap eventually fires and stops the poll.
useEffect(() => {
if (degradedPoll) lastActivityAtRef.current = Date.now();
}, [degradedPoll, messageRows]);
// #184 reconnect-and-live-follow. Whether detached agent runs are enabled for
// this workspace. When the feature is off no runs are ever created, so the
// resume attempt would only ever 204; gating ChatThread's resume on it avoids a
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -57,6 +57,25 @@ export async function stopRun(
return req.data;
}
/**
* #488: the run-fact "is a run active on this chat?" first-class from the
* server (POST /ai-chat/run). Called on mount to seed the client FSM's run-fact
* and to VERIFY after a supersede mismatch (an observer following a superseded
* run asks for the latest run and follows it). Returns the latest run row (with
* its `id` and `status`) and its projected assistant message, or `run: null` when
* the chat has never had a run. Owner-gated server-side.
*/
export async function getRun(chatId: string): Promise<{
run: { id: string; status: string } | null;
message: IAiChatMessageRow | null;
}> {
const req = await api.post<{
run: { id: string; status: string } | null;
message: IAiChatMessageRow | null;
}>("/ai-chat/run", { chatId });
return req.data;
}
/**
* Resolve the chat bound to a document (the current user's most-recent chat
* created on that page), or null when there is none. Drives auto-open-on-page.
@@ -0,0 +1,183 @@
# AI-chat run-lifecycle FSM — design spec (#488)
This is the written design that `run-fsm.ts` implements. It ships in the PR (issue
#488 commit 1: "the spec is written FIRST and enters the PR"). It has four parts:
(1) the event × state transition table, (2) the map of every `chat-thread.tsx` ref
to {FSM state | FSM context | stays data}, (3) the run-fact protocol, (4) the
invariants.
The reducer is a **pure function** `reduce(machine, event) → machine`. The returned
machine carries the **command effects** for that transition; a thin runtime in
`chat-thread.tsx` dispatches events and executes effects. Because it is pure, the
whole machine is enumerable and unit-tested directly (event × state → next state is
the observable property) — see `run-fsm.test.ts`.
---
## 1. Event × state transition table
Phases: `idle | sending | streaming | attaching | reconnecting(attempt,failed) |
polling(reason) | stalled | stopping | superseding | error(kind)`.
Context (orthogonal): `epoch`, `ownership: local|observer`, `runFact: {runId}|null`,
`liveFollow` (are we following a live run we locally streamed — the reconnect
ladder — vs a one-shot mount-attach resume? both are `observer`, but a live-follow
drop RE-ENTERS the ladder (#488 commit 3) while a mount-resume drop polls).
Legend: **†** = command-transition (bumps `epoch`, I1). Effects in `[…]`.
| Event (source) | From phase(s) | → To phase | Effects / ctx |
|---|---|---|---|
| `SEND_LOCAL` (user send) | idle, error, polling, stalled, reconnecting | sending **†** | `[cancelReconnect, disarmPoll]`, ownership=local |
| `STREAM_START{runId}` (SDK `start` metadata) | sending, attaching, reconnecting, superseding | streaming | `[cancelReconnect, disarmPoll]`, runFact←runId |
| `FINISH_CLEAN` (onFinish clean) | streaming, … | idle | `[disarmPoll, cancelReconnect]`, runFact←null |
| `FINISH_ABORT` (onFinish isAbort) | streaming, stopping | idle | `[disarmPoll, cancelReconnect]`, runFact←null (I4 exits stopping by this DATA) |
| `FINISH_DISCONNECT` (observer, NOT liveFollow) | streaming(observer) | polling(disconnect-visible) | `[armPoll]` (a mount-resume drop polls) |
| `FINISH_DISCONNECT{hasVisibleContent}` (local drop OR liveFollow) | streaming | reconnecting(1) **†** *iff runFact\|liveFollow* | `[scheduleReconnect(1)]` (+`armPoll` if visible), ownership=observer, liveFollow=true (commit 3: repeatable) |
| `FINISH_DISCONNECT` (no runFact, not liveFollow) | streaming | idle | runFact←null (plain terminal "connection lost") |
| `STREAM_INCOMPLETE{reason}` (observer starved/torn clean finish) | streaming(observer) | polling(reason) | `[armPoll(reason)]` |
| `FINISH_ERROR{kind}` (onFinish isError) | any | error(kind) | `[disarmPoll, cancelReconnect]`, runFact←null |
| `STREAM_START{runId}` (first assistant frame of a local turn) | sending | streaming | runFact←runId, `[cancelReconnect, disarmPoll]` |
| `ATTACH_START{runId}` (mount resume) | **idle only** (F2) | attaching **†** | `[resumeStream]`, ownership=observer, runFact←runId; ignored from any non-idle phase |
| `ATTACH_LIVE` (attach GET 2xx) | attaching | streaming | — |
| `ATTACH_NONE` (attach GET 204/err/throw) | attaching | polling(attach-none) | `[armPoll(attach-none)]` |
| `RECONNECT_ATTEMPT{n}` (backoff timer) | reconnecting | reconnecting(n) **†** | `[resumeStream]` |
| `RECONNECT_ATTACHED` (reconnect GET 2xx) | reconnecting | streaming | `[cancelReconnect, disarmPoll]`**counter reset** (commit 3) |
| `RECONNECT_NONE` (reconnect GET 204/err), attempt<MAX | reconnecting | reconnecting(n+1) **†** | `[armPoll(attach-none), scheduleReconnect(n+1)]` |
| `RECONNECT_NONE`, attempt=MAX | reconnecting | reconnecting(MAX, failed) | `[armPoll(reconnect-exhausted)]` |
| `RETRY` (manual, failed banner) | reconnecting(failed) | reconnecting(1) **†** | `[resumeStream]` |
| `RETRY` (manual, stalled banner) | stalled | polling(attach-none) **†** | `[armPoll]` |
| `POLL_TERMINAL` (settled tail merged) | polling, reconnecting, stopping | idle | `[disarmPoll, cancelReconnect]`, runFact←null (I4) |
| `POLL_IDLE_CAP` (inactivity cap) | polling, reconnecting | stalled | `[disarmPoll, cancelReconnect]` (commit 4a — no more silent) |
| `RUN_FACT{null}` (POST /run → null/terminal, 204) | reconnecting/attaching/polling/stopping | idle | `[cancelReconnect, disarmPoll]`, runFact←null (I3 fresh-negative gate) |
| `RUN_FACT{runId}` | any | (same) | runFact←runId (pessimism toward an attempt) |
| `STOP_REQUESTED` (user Stop) | streaming, reconnecting, polling | stopping **†** | `[stopRun, abortAttach, cancelReconnect, armPoll]` (poll drives the terminal — I4 exit by data) |
| `SUPERSEDE_REQUESTED{targetRunId}` (interrupt+send) | streaming, reconnecting, polling, error | superseding **†** | `[supersede(target), cancelReconnect, disarmPoll]` |
| `SUPERSEDE_READY{runId}` (CAS ok) | superseding | streaming | ownership=local, runFact←runId |
| `SUPERSEDE_MISMATCH{currentRunId}` (409 SUPERSEDE_TARGET_MISMATCH) | superseding | error(supersede-mismatch) | `[postRun(verify)]`, runFact←currentRunId |
| `SUPERSEDE_TIMEOUT` (409 SUPERSEDE_TIMEOUT) | superseding | error(supersede-timeout) | — (composer keeps text; no auto-retry) |
| `SUPERSEDE_INVALID` (409 SUPERSEDE_INVALID) | superseding | error(supersede-invalid) | — |
| `RUN_ALREADY_ACTIVE{activeRunId}` (409 A_RUN_ALREADY_ACTIVE, plain POST) | sending | error(run-already-active) | runFact←activeRunId (composer offers supersede; NO auto-retry) |
| `DISPOSE` (unmount) | any | idle **†** | `[abortAttach, cancelReconnect, disarmPoll]` (I1/I5 — epoch++ kills late callbacks) |
**`stopping` honors any finish (re-review MEDIUM):** BEFORE the epoch filter, a
stream finish (`FINISH_*`/`STREAM_INCOMPLETE`) arriving in phase `stopping` exits
`stopping -> idle` regardless of generation. A plain Stop has no successor stream,
so the aborted stream's finish IS the expected end (I4 exit by data) — and it
carries the PRE-stop generation (STOP_REQUESTED bumped the epoch), so the filter
would otherwise strand the machine in `stopping` (no idle-cap covers it). The filter
stays in force for `superseding` (that is the F1 supersede drop).
**Epoch filter (I1):** the reducer then drops any event carrying an `epoch` that
does not equal the current `ctx.epoch`. Outcome events (`STREAM_START`, `ATTACH_*`,
`RECONNECT_*`, `SUPERSEDE_*`, **`FINISH_*`/`STREAM_INCOMPLETE`**, `RUN_FACT`) are
stamped with the generation the corresponding STREAM started under (the runtime
holds a per-owned-stream `turnEpoch`); trigger events (user actions, fresh
disconnects) carry no epoch. **F1:** this is what makes a SUPERSEDED stream's late
`onFinish` (a dead stream A closing after the CAS started stream B) get dropped, so
A cannot drive the live new run into a false reconnect or reset its run-fact. The
supersede path additionally ABORTS A and starts B only from A's onFinish (a
microtask), because ai@6 `AbstractChat.makeRequest` corrupts overlapping streams
(A's `finally` reads then nulls the shared `activeResponse`).
**Removed events (scope-cut, internal review):** `RUN_SUPERSEDED` (a ghost feature —
never dispatched; the observer-superseded case is handled by the degraded poll,
which follows the latest rows regardless of runId), `RECONNECT_BEGIN` (reconnect is
entered by `FINISH_DISCONNECT`), and `POLL_ACTIVITY` (the window's activity clock was
removed when the idle-cap moved into the thread). The reducer and this table now
share exactly the dispatched event set.
### 409-code → event map (the real #487 contract consumed here)
| Server response | Event dispatched | error kind → banner |
|---|---|---|
| 409 `A_RUN_ALREADY_ACTIVE` (+ body.activeRunId) | `RUN_ALREADY_ACTIVE{activeRunId}` | run-already-active → "already answering / interrupt & send" |
| 409 `SUPERSEDE_TARGET_MISMATCH` (+ body.activeRunId) | `SUPERSEDE_MISMATCH{currentRunId}` | supersede-mismatch → verify via /run |
| 409 `SUPERSEDE_TIMEOUT` | `SUPERSEDE_TIMEOUT` | supersede-timeout → "couldn't interrupt in time, resend" |
| 409 `SUPERSEDE_INVALID` | `SUPERSEDE_INVALID` | supersede-invalid → "couldn't interrupt this run" |
| 503 `A_RUN_BEGIN_FAILED` | `FINISH_ERROR{begin-failed}` | begin-failed → "could not start, temporary" |
---
## 2. Ref-map — every `chat-thread.tsx` ref → its new home (MIGRATION RESOLVED)
The migration is COMPLETE: the 13 run-lifecycle FLAGS below are GONE from
`chat-thread.tsx` (collapsed into FSM phase/ctx/effects, or deleted). What remains
are identity/data mirrors, effect-owned controllers/timers, and ONE React-liveness
bit — none of which is a run-lifecycle flag, so the post-merge "no new flags" rule
holds. **Pending column: empty.**
| # | Old ref | Resolved to | Where now |
|---|---|---|---|
| 1 | `reconcileTailRef` | **FSM phase** | reconcile-merge gated on `phase ∈ {polling, reconnecting, stopping}` |
| 2 | `noStreamHandledRef` | **FSM epoch (I1)** | the attach outcome's epoch guard drops the stale/second outcome |
| 3 | `onNoActiveStreamRef` | **FSM event** | transport → `handleAttachOutcome` dispatches `ATTACH_NONE`/`RECONNECT_NONE` |
| 4 | `onReconnectAttachedRef` | **FSM event** | transport dispatches `ATTACH_LIVE` / `RECONNECT_ATTACHED` |
| 5 | `resumedTurnRef` + `resumedTurn` state | **FSM ctx `ownership`** | `ownership==='observer'` ⇒ never flush; hides "Send now" |
| 6 | `reconnectStateRef` + `reconnectState` state | **FSM phase** | `reconnecting(attempt,failed)` renders the banner |
| 7 | `reconnectTimerRef` | **effect-owned timer** | owned by `scheduleReconnect`/`cancelReconnect` effects (not a flag) |
| 8 | `flushOnAbortRef` | **DELETED** | the stop→flush dance is replaced by the CAS supersede (commit 5) |
| 9 | `interruptNextSendRef` | **DELETED** | the server injects the interrupt note from the supersede itself |
| 10 | `supersedeRetryRef` | **DELETED** (commit 5) | the client 409 retry ladder is gone; CAS supersede replaces it |
| 11 | `stopPendingRef` | **FSM phase `stopping`** | the deferred stop fires from the chat-id adoption effect while `stopping` |
| 12 | `mountedRef` | **retained (React liveness)** | orthogonal to run-lifecycle; gates imperative onFinish side-effects post-unmount. Epoch (I1) handles stale COMMAND-outcomes; DISPOSE bumps it |
| 13 | `attemptResumeRef` | **FSM `ATTACH_START` + run-fact** | mount arms attach ONLY on a confirmed active run (commit 4b: streaming-tail status, or POST /run for a user tail) |
| 14 | `stripRef` | **data** (attachStrategy) | strip+replay detail; the `resumeStream` effect reads it |
| 15 | `strippedRowRef` | **data** (attachStrategy) | the anchor row |
| 16 | `attachAbortRef` | **effect-owned controller** | aborted by the `abortAttach` effect in cleanup (I5) |
| 17–25 | `chatIdRef`, `openPageRef`, `getEditorSelectionRef`, `roleIdRef`, `stableIdRef`, `queuedRef`, `sendMessageRef`, `statusRef`, `lastForwardedChatIdRef` | **data** (identity/send mirrors) | unchanged — not lifecycle flags |
| NEW | `pendingSupersedeRef` | **data** (send-plumbing) | the runId injected into the next `POST /stream {supersede}`; the single replacement for the 3 DELETED one-shots (#8/#9/#10) — net −2 refs |
| NEW | `idleCapTimerRef` | **effect-owned timer** | the stalled inactivity cap → `POLL_IDLE_CAP` (commit 4a); not a flag |
Net: the 13 lifecycle flags (#1#13) are eliminated: **8** → FSM phase/ctx/epoch/event
(#1#6, #11, #13), **3** deleted (#8/#9/#10), **`reconnectTimerRef` (#7)** becomes an
effect-owned controller, and **`mountedRef` (#12)** is retained as React liveness
(8 + 3 + 1 + 1 = 13). (`attachAbortRef` (#16) is outside the #1#13 set — it was
already an effect-owned controller.) Two effect-owned timers + one send-plumbing data
ref are added — none is a boolean lifecycle latch.
---
## 3. Run-fact protocol (`runFact: {runId} | null`) — I3
"A run is active" is first-class from the SERVER, not inferred from an assistant
message. Sources, in the order they update `ctx.runFact`:
1. **Init (mount):** `POST /ai-chat/run { chatId }``{ run, message }`. A `run`
with a non-terminal `status` seeds `runFact = { runId: run.id }`; a null/terminal
run seeds `null`. This is what arms the resume attempt (`ATTACH_START`) — the
attempt is armed ONLY on a positive fact (commit 4b: a user-tail with no active
run no longer arms a pointless poll on every open).
2. **Live update:** the `start` stream metadata carries `runId``STREAM_START{runId}`.
3. **Attach outcomes:** `ATTACH_LIVE` (2xx) confirms active; a 204 on a non-stripped
path is an authoritative NEGATIVE fact → the runtime dispatches `RUN_FACT{null}`,
which cancels recovery (I3 fresh-negative gate).
4. **Poll (future resume-stack iteration #491):** the delta will carry the run field;
until then the poll drives to a terminal ROW, dispatched as `POLL_TERMINAL`.
Pessimism rule: a stale-but-positive fact PERMITS entering recovery (attach); the
204 then cuts it. A fresh negative fact gates recovery OUT immediately.
---
## 4. Invariants
- **I1 — Epoch (generation counter).** Every command-emitting transition bumps
`ctx.epoch`; every async outcome event carries its issuing epoch; the reducer
drops stale-epoch outcomes. Replaces the one-shot-ref zoo (`noStreamHandledRef`,
the flush/interrupt/supersede one-shots, the `mountedRef` late-callback gate).
- **I2 — Ownership is context, not state.** `local | observer` is orthogonal to the
transport phase. The queue flushes ONLY under local ownership; an observer
following a detached run never flushes (was `resumedTurnRef`).
- **I3 — Run-fact is first-class from the server.** Reconnect is entered by the
run-fact, not by an assistant message (commit 2). A fresh negative fact cancels
recovery.
- **I4 — Exit `stopping` by DATA.** A terminal row / negative run-fact / terminal
finish exits `stopping`, never the stopRun HTTP response (which returns after the
abort but before finalization — keying off it would unlock the composer on a 409).
- **I5 — Dispose protocol.** Command controllers (attach GET, POST /stream, POST
/run) are effect-owned and aborted in cleanup (`abortAttach` on `DISPOSE`), not
render-phase refs. A client abort of an already-sent POST does not cancel the
server action, so disarming on unmount is safe.
- **attachStrategy** (strip+replay today) is behind the `resumeStream` effect; the
resume-stack iteration (#491) swaps it to tail-only WITHOUT touching the FSM.
- **Queue** stays a data structure; flush/interrupt decisions are transitions.
@@ -0,0 +1,482 @@
import { describe, it, expect } from "vitest";
import {
reduce,
initialMachine,
reconnectDelayMs,
RECONNECT_MAX_ATTEMPTS,
type Machine,
type Effect,
type Event,
} from "./run-fsm";
// Drive a sequence of events through the reducer, returning the final machine.
function run(m: Machine, ...events: Event[]): Machine {
return events.reduce(reduce, m);
}
function withRunFact(runId = "run-1"): Machine {
return {
...initialMachine(),
ctx: { epoch: 0, ownership: "local", runFact: { runId }, liveFollow: false },
};
}
function effectTypes(m: Machine): string[] {
return m.effects.map((e) => e.type);
}
function hasEffect(m: Machine, type: Effect["type"]): boolean {
return m.effects.some((e) => e.type === type);
}
describe("run-fsm — epoch invariant (I1)", () => {
it("drops an outcome carrying a stale epoch", () => {
// A command bumps the epoch; an outcome stamped with the OLD epoch is dropped.
const m0 = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" }); // epoch 0->1, attaching
expect(m0.ctx.epoch).toBe(1);
expect(m0.phase.name).toBe("attaching");
// A late ATTACH_LIVE from a SUPERSEDED attempt (epoch 0) must NOT drive us.
const stale = reduce(m0, { type: "ATTACH_LIVE", epoch: 0 });
expect(stale.phase.name).toBe("attaching");
expect(stale.effects).toEqual([]);
});
it("applies an outcome carrying the current epoch", () => {
const m0 = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
const live = reduce(m0, { type: "ATTACH_LIVE", epoch: m0.ctx.epoch });
expect(live.phase.name).toBe("streaming");
});
it("an outcome with no epoch is never dropped (trigger events)", () => {
const m0 = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
const disposed = reduce(m0, { type: "DISPOSE" });
expect(disposed.phase.name).toBe("idle");
expect(hasEffect(disposed, "abortAttach")).toBe(true);
});
it("every command-transition increments the epoch exactly once", () => {
let m = initialMachine();
const before = m.ctx.epoch;
m = reduce(m, { type: "SEND_LOCAL" });
expect(m.ctx.epoch).toBe(before + 1);
m = reduce(m, { type: "STOP_REQUESTED" });
expect(m.ctx.epoch).toBe(before + 2);
});
});
describe("run-fsm — local turn", () => {
it("SEND_LOCAL → sending, local ownership, cancels recovery", () => {
const m = reduce(withRunFact(), { type: "SEND_LOCAL" });
expect(m.phase.name).toBe("sending");
expect(m.ctx.ownership).toBe("local");
expect(effectTypes(m)).toEqual(
expect.arrayContaining(["cancelReconnect", "disarmPoll"]),
);
});
it("STREAM_START adopts the runId into the run-fact and goes streaming", () => {
const m = run(initialMachine(), { type: "SEND_LOCAL" });
const s = reduce(m, { type: "STREAM_START", runId: "run-9", epoch: m.ctx.epoch });
expect(s.phase.name).toBe("streaming");
expect(s.ctx.runFact).toEqual({ runId: "run-9" });
});
it("FINISH_CLEAN → idle, run-fact cleared, poll/reconnect disarmed", () => {
const streaming = run(initialMachine(), { type: "SEND_LOCAL" }, { type: "STREAM_START", runId: "r" });
const done = reduce(streaming, { type: "FINISH_CLEAN" });
expect(done.phase.name).toBe("idle");
expect(done.ctx.runFact).toBeNull();
});
});
// #488 commit 2 — SSE break BEFORE the first assistant frame must still recover.
describe("run-fsm — commit 2: reconnect by run-fact, not by assistant message", () => {
it("FINISH_DISCONNECT with an active run-fact → reconnecting (even with no visible content)", () => {
// Setup-phase break: no assistant frame yet, but a run-fact exists.
const streaming = withRunFact("run-2");
const m = reduce(streaming, {
type: "FINISH_DISCONNECT",
hasVisibleContent: false,
epoch: streaming.ctx.epoch,
});
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") expect(m.phase.attempt).toBe(1);
expect(m.ctx.ownership).toBe("observer");
expect(hasEffect(m, "scheduleReconnect")).toBe(true);
// No visible content -> no poll arm yet (the reconnect ladder rebuilds it).
expect(hasEffect(m, "armPoll")).toBe(false);
});
it("FINISH_DISCONNECT WITH visible content also arms the poll", () => {
const m = reduce(withRunFact("run-2"), {
type: "FINISH_DISCONNECT",
hasVisibleContent: true,
epoch: 0,
});
expect(m.phase.name).toBe("reconnecting");
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("FINISH_DISCONNECT with NO run-fact → idle (plain connection-lost)", () => {
const m = reduce(initialMachine(), {
type: "FINISH_DISCONNECT",
hasVisibleContent: true,
epoch: 0,
});
expect(m.phase.name).toBe("idle");
});
});
// #488 commit 3 — a SECOND break after a successful re-attach starts a NEW ladder.
describe("run-fsm — commit 3: repeated reconnect cycles", () => {
it("two breaks in a row produce two reconnect cycles (counter resets on attach)", () => {
let m = withRunFact("run-3");
// First break -> reconnecting(1).
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
// Attempt fires, re-attaches live.
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: 1, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_ATTACHED", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("streaming");
// SECOND break: the counter was reset, so a fresh ladder starts at attempt 1
// (the old one-shot !wasResumed gate would have sent this to silent poll).
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") expect(m.phase.attempt).toBe(1);
expect(hasEffect(m, "scheduleReconnect")).toBe(true);
});
it("a MOUNT-attach observer drop falls to POLL, not the reconnect ladder", () => {
// Distinguishes commit 3 from a one-shot resume: an observer that never
// live-followed (liveFollow false) polls on a drop.
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
expect(m.ctx.ownership).toBe("observer");
expect(m.ctx.liveFollow).toBe(false);
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: true, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("polling");
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("STREAM_INCOMPLETE (observer starved/torn finish) → polling", () => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
m = reduce(m, { type: "STREAM_INCOMPLETE", reason: "starved", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "polling", reason: "starved" });
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("liveFollow is set on the first local drop and kept across a re-attach", () => {
let m = withRunFact("run-3");
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
expect(m.ctx.liveFollow).toBe(true);
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: 1, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_ATTACHED", epoch: m.ctx.epoch });
expect(m.ctx.liveFollow).toBe(true); // kept — so a second drop reconnects
// A clean finish clears it.
m = reduce(m, { type: "FINISH_CLEAN", epoch: m.ctx.epoch });
expect(m.ctx.liveFollow).toBe(false);
});
it("RECONNECT_NONE backs off through the ladder, then fails at the cap", () => {
let m = withRunFact("run-3");
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: false, epoch: m.ctx.epoch });
for (let n = 1; n < RECONNECT_MAX_ATTEMPTS; n++) {
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: n, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_NONE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") {
expect(m.phase.attempt).toBe(n + 1);
expect(m.phase.failed).toBe(false);
}
// The belt-and-suspenders poll is armed each failed attempt.
expect(hasEffect(m, "armPoll")).toBe(true);
}
// Final attempt fails -> failed banner (Retry), poll armed.
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: RECONNECT_MAX_ATTEMPTS, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_NONE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") expect(m.phase.failed).toBe(true);
// RETRY restarts at attempt 1.
m = reduce(m, { type: "RETRY" });
expect(m.phase.name).toBe("reconnecting");
if (m.phase.name === "reconnecting") {
expect(m.phase.attempt).toBe(1);
expect(m.phase.failed).toBe(false);
}
expect(hasEffect(m, "resumeStream")).toBe(true);
});
it("reconnectDelayMs is the exponential backoff 1s,2s,4s,8s,16s", () => {
expect([1, 2, 3, 4, 5].map(reconnectDelayMs)).toEqual([1000, 2000, 4000, 8000, 16000]);
});
});
// #488 commit 4 — polling stalled-state + user-tail gating.
describe("run-fsm — commit 4: stalled + run-fact gating", () => {
it("POLL_IDLE_CAP: polling → stalled with a banner (poll disarmed), not silent", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("polling");
m = reduce(m, { type: "POLL_IDLE_CAP" });
expect(m.phase.name).toBe("stalled");
expect(hasEffect(m, "disarmPoll")).toBe(true);
});
it("RETRY from stalled re-arms the poll", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
m = reduce(m, { type: "POLL_IDLE_CAP" });
m = reduce(m, { type: "RETRY" });
expect(m.phase.name).toBe("polling");
expect(hasEffect(m, "armPoll")).toBe(true);
});
it("a fresh NEGATIVE run-fact while attaching cancels recovery (user-tail, no active run)", () => {
// The mount POST /run returns no active run: attaching → idle, no poll armed.
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "RUN_FACT", runFact: null, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("idle");
expect(m.ctx.runFact).toBeNull();
expect(hasEffect(m, "disarmPoll")).toBe(true);
});
it("a negative run-fact while polling stops the poll", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
m = reduce(m, { type: "RUN_FACT", runFact: null, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("idle");
});
it("POLL_TERMINAL settles polling → idle (I4 data-driven exit)", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_NONE", epoch: m.ctx.epoch });
m = reduce(m, { type: "POLL_TERMINAL" });
expect(m.phase.name).toBe("idle");
expect(m.ctx.runFact).toBeNull();
});
});
// #488 commit 5 — error classification + supersede CAS transitions.
describe("run-fsm — commit 5: supersede CAS + error classification", () => {
it("SUPERSEDE_REQUESTED → superseding, fires the CAS effect, bumps epoch", () => {
const streaming = withRunFact("run-old");
const m = reduce(streaming, { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
expect(m.phase.name).toBe("superseding");
expect(m.ctx.epoch).toBe(streaming.ctx.epoch + 1);
const sup = m.effects.find((e) => e.type === "supersede");
expect(sup).toEqual({ type: "supersede", targetRunId: "run-old" });
});
it("SUPERSEDE_READY → streaming as the new local owner", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_READY", runId: "run-new", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("streaming");
expect(m.ctx.ownership).toBe("local");
expect(m.ctx.runFact).toEqual({ runId: "run-new" });
});
it("SUPERSEDE_MISMATCH → error(supersede-mismatch) + verify via /run (no blind banner)", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_MISMATCH", currentRunId: "run-x", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "error", kind: "supersede-mismatch" });
expect(hasEffect(m, "postRun")).toBe(true);
expect(m.ctx.runFact).toEqual({ runId: "run-x" });
});
it("SUPERSEDE_TIMEOUT → error(supersede-timeout), no auto-retry effect", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_TIMEOUT", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "error", kind: "supersede-timeout" });
expect(m.effects).toEqual([]);
});
it("SUPERSEDE_INVALID → error(supersede-invalid)", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
m = reduce(m, { type: "SUPERSEDE_INVALID", epoch: m.ctx.epoch });
expect(m.phase).toEqual({ name: "error", kind: "supersede-invalid" });
});
it("a stale SUPERSEDE outcome from a superseded epoch is dropped", () => {
let m = reduce(withRunFact("run-old"), { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
const supersedingEpoch = m.ctx.epoch;
// The user retriggers, bumping the epoch again.
m = reduce(m, { type: "SUPERSEDE_REQUESTED", targetRunId: "run-old" });
// The first CAS's late TIMEOUT (old epoch) must NOT knock us out of superseding.
const late = reduce(m, { type: "SUPERSEDE_TIMEOUT", epoch: supersedingEpoch });
expect(late.phase.name).toBe("superseding");
});
it("RUN_ALREADY_ACTIVE (plain POST gate) → error(run-already-active), no retry effect", () => {
const m = reduce(run(initialMachine(), { type: "SEND_LOCAL" }), { type: "RUN_ALREADY_ACTIVE" });
expect(m.phase).toEqual({ name: "error", kind: "run-already-active" });
expect(m.effects).toEqual([]);
});
it("#497/S4: RUN_ALREADY_ACTIVE{activeRunId} ADOPTS the server's active run as the run-fact", () => {
// The server sends `activeRunId` so a later supersede can TARGET that run
// instead of a blind promote+abort. Absorb it into runFact.
const m = reduce(run(initialMachine(), { type: "SEND_LOCAL" }), {
type: "RUN_ALREADY_ACTIVE",
activeRunId: "run-foreign",
});
expect(m.phase).toEqual({ name: "error", kind: "run-already-active" });
expect(m.ctx.runFact).toEqual({ runId: "run-foreign" });
expect(m.effects).toEqual([]);
});
it("#497/S4: RUN_ALREADY_ACTIVE without an activeRunId keeps the prior run-fact", () => {
const seeded = reduce(run(initialMachine(), { type: "SEND_LOCAL" }), {
type: "RUN_FACT",
runFact: { runId: "run-prior" },
});
const m = reduce(seeded, { type: "RUN_ALREADY_ACTIVE" });
expect(m.ctx.runFact).toEqual({ runId: "run-prior" });
});
});
// #488 F2 — a late mount `getRun → ATTACH_START` must not hijack a local turn.
describe("run-fsm — F2: ATTACH_START only from idle", () => {
it("ATTACH_START from a local `sending` turn is ignored (no observer hijack)", () => {
const sending = reduce(initialMachine(), { type: "SEND_LOCAL" }); // idle -> sending, local
const m = reduce(sending, { type: "ATTACH_START", runId: "r" });
expect(m.phase.name).toBe("sending");
expect(m.ctx.ownership).toBe("local"); // NOT flipped to observer
expect(m.effects).toEqual([]); // no resumeStream
});
it("ATTACH_START from idle attaches as normal", () => {
const m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
expect(m.phase.name).toBe("attaching");
expect(m.ctx.ownership).toBe("observer");
expect(hasEffect(m, "resumeStream")).toBe(true);
});
});
describe("run-fsm — stop (I4: exit by data)", () => {
it("STOP_REQUESTED → stopping, fires stopRun + abortAttach, no data-independent exit", () => {
const m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
expect(m.phase.name).toBe("stopping");
expect(effectTypes(m)).toEqual(expect.arrayContaining(["stopRun", "abortAttach"]));
});
it("stopping exits on the aborted stream's finish carrying the PRE-STOP epoch", () => {
// MEDIUM (#488 re-review): STOP_REQUESTED is a command that BUMPS the epoch, but
// the runtime stamps the aborted stream's onFinish with the stream's START (pre-
// stop) generation — exactly what the component sends. `stopping` must HONOR
// that finish regardless of generation (no idle-cap covers `stopping`).
// MUTATION-VERIFY: remove the honor-in-`stopping` branch and this hangs in
// `stopping` (the epoch filter drops the pre-stop finish) -> red.
const preStopEpoch = withRunFact().ctx.epoch; // E1 (the stream's start epoch)
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" }); // E1 -> E2, stopping
expect(m.ctx.epoch).toBe(preStopEpoch + 1);
m = reduce(m, { type: "FINISH_ABORT", epoch: preStopEpoch }); // NOT the current epoch
expect(m.phase.name).toBe("idle");
expect(m.ctx.runFact).toBeNull();
});
it("stopping exits on a clean finish carrying the pre-stop epoch too", () => {
const preStopEpoch = withRunFact().ctx.epoch;
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
m = reduce(m, { type: "FINISH_CLEAN", epoch: preStopEpoch });
expect(m.phase.name).toBe("idle");
});
it("stopping exits on a negative run-fact (data)", () => {
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
m = reduce(m, { type: "RUN_FACT", runFact: null, epoch: m.ctx.epoch });
expect(m.phase.name).toBe("idle");
});
// Review #4: `stopping` arms the poll but had no inactivity backstop.
it("review-4: POLL_IDLE_CAP in `stopping` exits to idle (bounded), NOT stalled", () => {
let m = reduce(withRunFact(), { type: "STOP_REQUESTED" });
expect(m.phase.name).toBe("stopping");
expect(hasEffect(m, "armPoll")).toBe(true);
// MUTATION-VERIFY: drop the `stopping` branch in POLL_IDLE_CAP and this hangs
// in `stopping` (poll forever) -> red.
m = reduce(m, { type: "POLL_IDLE_CAP" });
expect(m.phase.name).toBe("idle");
expect(hasEffect(m, "disarmPoll")).toBe(true);
expect(m.ctx.ownership).toBe("local");
});
});
// Review #1: positive attach outcomes must be guarded by the SOURCE phase — the
// epoch filter alone is insufficient because POLL_TERMINAL uses to() (no epoch
// bump) and does not abort the in-flight GET.
describe("run-fsm — review-1: attach outcomes guarded by source phase", () => {
it("a late RECONNECT_ATTACHED after POLL_TERMINAL stays idle (no phantom streaming)", () => {
let m = withRunFact("run-1");
m = reduce(m, { type: "FINISH_DISCONNECT", hasVisibleContent: true, epoch: m.ctx.epoch });
m = reduce(m, { type: "RECONNECT_ATTEMPT", attempt: 1, epoch: m.ctx.epoch }); // attach GET
const epoch = m.ctx.epoch;
// The armed degraded poll reaches the terminal row FIRST (epoch unchanged).
m = reduce(m, { type: "POLL_TERMINAL" });
expect(m.phase.name).toBe("idle");
expect(m.ctx.epoch).toBe(epoch); // POLL_TERMINAL did NOT bump the epoch
// The slow GET returns live 2xx under the SAME epoch — must NOT resurrect.
m = reduce(m, { type: "RECONNECT_ATTACHED", epoch });
expect(m.phase.name).toBe("idle");
});
it("a late ATTACH_LIVE / ATTACH_NONE after leaving `attaching` is ignored", () => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
const epoch = m.ctx.epoch;
m = reduce(m, { type: "ATTACH_NONE", epoch }); // attaching -> polling
m = reduce(m, { type: "POLL_TERMINAL" }); // -> idle (epoch unchanged)
expect(m.phase.name).toBe("idle");
m = reduce(m, { type: "ATTACH_LIVE", epoch }); // late 2xx, same epoch
expect(m.phase.name).toBe("idle");
// And a late ATTACH_NONE (not `attaching`) is a no-op too.
m = reduce(m, { type: "ATTACH_NONE", epoch });
expect(m.phase.name).toBe("idle");
});
});
// Review #2: every terminal transition resets ownership to local.
describe("run-fsm — review-2: terminal transitions reset ownership to local", () => {
const observer = (): Machine => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
expect(m.ctx.ownership).toBe("observer");
return m;
};
it("FINISH_CLEAN resets ownership", () => {
const m = reduce(observer(), { type: "FINISH_CLEAN", epoch: observer().ctx.epoch });
expect(m.ctx.ownership).toBe("local");
});
it("FINISH_ERROR / POLL_TERMINAL / RUN_FACT(null) reset ownership", () => {
let o = observer();
expect(reduce(o, { type: "FINISH_ERROR", kind: "stream", epoch: o.ctx.epoch }).ctx.ownership).toBe("local");
// POLL_TERMINAL from an observer polling phase
let p = reduce(observer(), { type: "STREAM_INCOMPLETE", reason: "starved", epoch: observer().ctx.epoch });
expect(reduce(p, { type: "POLL_TERMINAL" }).ctx.ownership).toBe("local");
// RUN_FACT(null) from an observer attaching phase
let a = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
expect(reduce(a, { type: "RUN_FACT", runFact: null, epoch: a.ctx.epoch }).ctx.ownership).toBe("local");
});
});
describe("run-fsm — ownership (I2) is context, orthogonal to phase", () => {
it("attach/reconnect set observer; send/supersede-ready set local", () => {
let m = reduce(initialMachine(), { type: "ATTACH_START", runId: "r" });
expect(m.ctx.ownership).toBe("observer");
m = reduce(m, { type: "ATTACH_LIVE", epoch: m.ctx.epoch });
expect(m.phase.name).toBe("streaming");
expect(m.ctx.ownership).toBe("observer"); // still observing a detached run
// A local send flips ownership back to local.
m = reduce(m, { type: "SEND_LOCAL" });
expect(m.ctx.ownership).toBe("local");
});
});
describe("run-fsm — dispose (I5)", () => {
it("DISPOSE from any phase aborts controllers and bumps epoch", () => {
let m = reduce(withRunFact(), { type: "ATTACH_START", runId: "r" });
const before = m.ctx.epoch;
m = reduce(m, { type: "DISPOSE" });
expect(m.phase.name).toBe("idle");
expect(m.ctx.epoch).toBe(before + 1);
expect(effectTypes(m)).toEqual(
expect.arrayContaining(["abortAttach", "cancelReconnect", "disarmPoll"]),
);
});
});
@@ -0,0 +1,600 @@
/**
* Run-lifecycle finite state machine for a single AI-chat thread (#488).
*
* ============================================================================
* WHY THIS EXISTS
* ----------------------------------------------------------------------------
* The resume/reconnect/poll/stop/supersede lifecycle used to be spread across
* ~26 `useRef` one-shot flags in `chat-thread.tsx`, each disarmed "on every
* path". Ownerless flag combinations produced silent UI freezes, and every fix
* added another ref (the #381 -> #432 -> #456 spiral). This module replaces that
* ref-zoo with ONE pure reducer whose transitions are enumerable and unit-
* testable in isolation (event x state -> next state is the observable property).
*
* The reducer is PURE: it owns no timers, no fetches, no React state. It maps
* `(machine, event) -> machine`, where the returned machine carries the list of
* COMMAND EFFECTS to run for that transition. A thin runtime in `chat-thread.tsx`
* dispatches events (from SDK callbacks / HTTP outcomes) and executes the
* effects (attach GET, POST /stream, POST /run, POST /stop, backoff timers,
* poll arm/disarm). The runtime lives in a THREAD, not the window, so a late SDK
* callback dies with the owner (kills the "event from a dead view" class, #161).
*
* ============================================================================
* INVARIANTS (see run-fsm.spec.md for the full spec + tables)
* ----------------------------------------------------------------------------
* I1 EPOCH (generation counter). Commands (`resumeStream`, `postRun`, `stop`,
* `supersede`, `scheduleReconnect`) are async; their outcomes arrive on the
* SAME SDK/HTTP callbacks. Every command-emitting transition increments
* `ctx.epoch`; every OUTCOME event carries the epoch it was issued under;
* the reducer DROPS an outcome whose epoch != the current epoch. This is
* what the one-shot-ref zoo used to approximate by hand.
* I2 OWNERSHIP is a CONTEXT FIELD (`'local' | 'observer'`), not a state
* orthogonal to the transport phase. The queue is flushed ONLY by a local
* owner (an observer following a detached run never flushes).
* I3 RUN-FACT ("a run is active") is first-class from the server: `runFact`
* holds the server-confirmed active run id (POST /run on mount, the `start`
* metadata runId, attach outcomes). Reconnect is entered by the RUN-FACT,
* not by the presence of an assistant message (#488 commit 2). A fresh
* negative fact (null) cancels reconnect immediately.
* I4 Exit `stopping` by DATA (a terminal row / negative run-fact), NEVER by the
* stopRun HTTP response (which returns after abort, before finalization).
* I5 Command controllers are effect-owned (abort in cleanup), NOT render-phase
* refs expressed here as the `abortAttach` effect on disposing transitions.
* ============================================================================
*/
// ---------------------------------------------------------------------------
// Phases (the transport lifecycle). Ownership / runFact are CONTEXT, not here.
// ---------------------------------------------------------------------------
/** Why the degraded poll is the active recovery. */
export type PollReason =
| "attach-none" // mount attach returned 204 / error — nothing live to attach
| "starved" // a resumed finish carried no visible content
| "disconnect-visible" // a live disconnect WITH on-screen content — poll to terminal
| "reconnect-exhausted"; // the live re-attach ladder gave up
/** The classified error kind (drives the banner text + composer behavior). */
export type ErrorKind =
| "stream" // a generic provider/network stream error (useChat error)
| "run-already-active" // 409 A_RUN_ALREADY_ACTIVE (a plain POST hit the gate)
| "supersede-mismatch" // 409 SUPERSEDE_TARGET_MISMATCH (CAS target moved)
| "supersede-timeout" // 409 SUPERSEDE_TIMEOUT (old run did not settle in W)
| "supersede-invalid" // 409 SUPERSEDE_INVALID (bad supersede target)
| "begin-failed"; // 503 A_RUN_BEGIN_FAILED (could not start the run)
export type Phase =
| { name: "idle" }
| { name: "sending" } // local POST in flight, before the first frame
| { name: "streaming" } // receiving frames
| { name: "attaching" } // mount-time attach GET in flight
| { name: "reconnecting"; attempt: number; failed: boolean }
| { name: "polling"; reason: PollReason }
| { name: "stalled" } // poll hit the inactivity cap — banner + Retry
| { name: "stopping" }
| { name: "superseding" }
| { name: "error"; kind: ErrorKind };
export type Ownership = "local" | "observer";
/** The server-confirmed active run, or null when no run is active. */
export type RunFact = { runId: string } | null;
export interface Ctx {
/** I1: generation counter — every command-transition increments it. */
epoch: number;
/** I2: does THIS client own the turn's writes (local streamer) or observe? */
ownership: Ownership;
/** I3: the server-confirmed active run. */
runFact: RunFact;
/**
* Are we FOLLOWING a live run we were locally streaming (the reconnect ladder),
* as opposed to a one-shot mount-attach resume? Both are `ownership: 'observer'`,
* but they recover DIFFERENTLY on a drop: a live-follow drop RE-ENTERS the
* reconnect ladder (#488 commit 3 the second break after a successful re-attach
* must reconnect again, not fall to silent poll), while a mount-resume drop falls
* to the degraded poll. This is the ctx bit that separates the two WITHOUT a new
* component ref (it is why commit 3 needs the FSM, not a surgical patch).
*/
liveFollow: boolean;
}
export interface Machine {
phase: Phase;
ctx: Ctx;
/** Command effects to run for the transition that produced THIS machine.
* The runtime executes them and does not read them again. */
effects: Effect[];
}
// ---------------------------------------------------------------------------
// Command effects (the reducer's only side-channel — executed by the runtime).
// ---------------------------------------------------------------------------
export type Effect =
/** POST /run to (re)establish or verify the run-fact. `reason` is diagnostic. */
| { type: "postRun"; reason: "mount" | "verify" }
/** Trigger the SDK `resumeStream()` (attach GET via prepareReconnectToStream). */
| { type: "resumeStream" }
/** Schedule a reconnect attempt after a backoff, then dispatch RECONNECT_ATTEMPT. */
| { type: "scheduleReconnect"; attempt: number; delayMs: number }
/** Cancel any pending reconnect backoff timer. */
| { type: "cancelReconnect" }
/** Arm the degraded poll (the window's dumb timer follows the run in the DB). */
| { type: "armPoll"; reason: PollReason }
/** Disarm the degraded poll. */
| { type: "disarmPoll" }
/** POST /stop the chat's active run (authoritative detached-run stop). */
| { type: "stopRun" }
/** POST /stream { supersede: { runId } } — the CAS "interrupt and send now". */
| { type: "supersede"; targetRunId: string }
/** Abort the in-flight attach/reconnect GET controller (dispose / observer stop). */
| { type: "abortAttach" };
// ---------------------------------------------------------------------------
// Events. An OUTCOME event MAY carry `epoch`; if it does and it does not equal
// the current epoch, the reducer drops it (I1). Trigger events (user actions,
// fresh disconnects) carry no epoch and are never dropped.
// ---------------------------------------------------------------------------
export type Event =
// -- local turn --
| { type: "SEND_LOCAL" }
| { type: "STREAM_START"; runId?: string; epoch?: number }
/** An OBSERVER's attached stream ended WITHOUT reaching terminal (a starved
* clean replay, or a torn resume) fall to the degraded poll to drive the row
* to its real terminal state. (A live-follow drop uses FINISH_DISCONNECT.) */
| { type: "STREAM_INCOMPLETE"; reason: PollReason; epoch?: number }
| { type: "FINISH_CLEAN"; epoch?: number }
| { type: "FINISH_ABORT"; epoch?: number }
| { type: "FINISH_DISCONNECT"; hasVisibleContent: boolean; epoch?: number }
| { type: "FINISH_ERROR"; kind: ErrorKind; epoch?: number }
// -- mount attach (resume) --
| { type: "ATTACH_START"; runId?: string }
| { type: "ATTACH_LIVE"; epoch?: number }
| { type: "ATTACH_NONE"; epoch?: number }
// -- reconnect after a live disconnect (entered by FINISH_DISCONNECT, #488 c2) --
| { type: "RECONNECT_ATTEMPT"; attempt: number; epoch?: number }
| { type: "RECONNECT_ATTACHED"; epoch?: number }
| { type: "RECONNECT_NONE"; epoch?: number }
| { type: "RETRY" }
// -- degraded poll --
| { type: "POLL_TERMINAL" }
| { type: "POLL_IDLE_CAP" }
// -- run-fact (server-confirmed active run) --
| { type: "RUN_FACT"; runFact: RunFact; epoch?: number }
// -- stop --
| { type: "STOP_REQUESTED" }
// -- supersede (CAS) --
| { type: "SUPERSEDE_REQUESTED"; targetRunId: string }
| { type: "SUPERSEDE_READY"; runId?: string; epoch?: number }
| { type: "SUPERSEDE_MISMATCH"; currentRunId?: string; epoch?: number }
| { type: "SUPERSEDE_TIMEOUT"; epoch?: number }
| { type: "SUPERSEDE_INVALID"; epoch?: number }
| { type: "RUN_ALREADY_ACTIVE"; activeRunId?: string }
// -- lifecycle --
| { type: "DISPOSE" };
export const RECONNECT_MAX_ATTEMPTS = 5;
export const RECONNECT_BASE_DELAY_MS = 1000;
/** Backoff before attempt N (1-based): 1s, 2s, 4s, 8s, 16s. */
export function reconnectDelayMs(attempt: number): number {
return RECONNECT_BASE_DELAY_MS * 2 ** (attempt - 1);
}
// ---------------------------------------------------------------------------
// Constructors / helpers.
// ---------------------------------------------------------------------------
export function initialMachine(overrides?: Partial<Ctx>): Machine {
return {
phase: { name: "idle" },
ctx: { epoch: 0, ownership: "local", runFact: null, liveFollow: false, ...overrides },
effects: [],
};
}
/** Build a machine result: a phase, optional ctx patch, and effects. Empty
* effects by default. Never mutates the input. */
function to(
m: Machine,
phase: Phase,
opts?: { ctx?: Partial<Ctx>; effects?: Effect[] },
): Machine {
return {
phase,
ctx: { ...m.ctx, ...(opts?.ctx ?? {}) },
effects: opts?.effects ?? [],
};
}
/** No transition: keep the phase, clear effects (so a re-run does not re-fire). */
function stay(m: Machine): Machine {
return { phase: m.phase, ctx: m.ctx, effects: [] };
}
/** A command-transition: same as `to` but bumps the epoch (I1). Any outcome
* event issued under the old epoch is dropped once this lands. */
function command(
m: Machine,
phase: Phase,
effects: Effect[],
ctx?: Partial<Ctx>,
): Machine {
return {
phase,
ctx: { ...m.ctx, ...(ctx ?? {}), epoch: m.ctx.epoch + 1 },
effects,
};
}
// ---------------------------------------------------------------------------
// The pure reducer.
// ---------------------------------------------------------------------------
/** The terminal stream-finish events (one turn's stream ended). */
function isFinishEvent(event: Event): boolean {
return (
event.type === "FINISH_ABORT" ||
event.type === "FINISH_CLEAN" ||
event.type === "FINISH_DISCONNECT" ||
event.type === "FINISH_ERROR" ||
event.type === "STREAM_INCOMPLETE"
);
}
export function reduce(m: Machine, event: Event): Machine {
// MEDIUM (#488 re-review): honor ANY stream finish in `stopping` regardless of
// generation. A plain user Stop has NO successor stream — the aborted stream's
// finish IS the expected end of the stop, so exit `stopping -> idle` by that DATA
// (I4). The epoch filter below must NOT drop it: STOP_REQUESTED bumped the epoch,
// but the finish carries the PRE-stop generation (the runtime stamps it with the
// stream's start epoch), so I1 would otherwise strand the machine in `stopping`
// forever (no idle-cap covers `stopping`). The epoch filter stays in force for
// `superseding` (a successor B owns) — that is the F1 supersede drop.
if (m.phase.name === "stopping" && isFinishEvent(event)) {
return to(m, { name: "idle" }, {
// Reset ownership to local on this terminal transition (review #2): otherwise
// an observer-stop leaves ownership 'observer' and hides "Send now" forever.
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
}
// I1: drop a stale outcome (an event issued under a superseded epoch).
if ("epoch" in event && event.epoch !== undefined && event.epoch !== m.ctx.epoch) {
return stay(m);
}
switch (event.type) {
// ---- local turn ----------------------------------------------------
case "SEND_LOCAL":
// A local send owns the view: leave any recovery, become the local
// streamer, disarm poll/reconnect. epoch++ so a late recovery outcome
// from the previous phase is dropped.
return command(
m,
{ name: "sending" },
[{ type: "cancelReconnect" }, { type: "disarmPoll" }],
{ ownership: "local", liveFollow: false },
);
case "STREAM_INCOMPLETE":
// An OBSERVER's attached stream ended incomplete (starved / torn) — follow
// the run to terminal via the degraded poll.
return to(m, { name: "polling", reason: event.reason }, {
effects: [{ type: "armPoll", reason: event.reason }],
});
case "STREAM_START": {
// First frame arrived. Adopt the run-fact runId if present. sending ->
// streaming; a reconnect/attach that just went live also lands here.
const runFact = event.runId ? { runId: event.runId } : m.ctx.runFact;
return to(m, { name: "streaming" }, {
ctx: { runFact },
effects: [{ type: "cancelReconnect" }, { type: "disarmPoll" }],
});
}
case "FINISH_CLEAN":
// A clean terminal outcome. The run is done — clear the run-fact and go
// idle. (The queue flush is a component concern gated by ownership; the
// FSM only models the phase.) Review #2: reset ownership to local so a
// just-finished observer-attach turn re-exposes "Send now" for the queue.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
case "FINISH_ABORT":
// A user Stop / intentional abort finished. If we were stopping, the
// terminal data has now arrived (I4) — go idle. The run-fact is cleared.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
case "FINISH_DISCONNECT":
// A LIVE SSE drop. Recovery depends on WHO we are (I2 + liveFollow):
// - a mount-attach OBSERVER (a one-shot resume, NOT live-follow) that drops
// -> the degraded poll drives the row to terminal from the DB.
if (m.ctx.ownership === "observer" && !m.ctx.liveFollow) {
return to(m, { name: "polling", reason: "disconnect-visible" }, {
effects: [{ type: "armPoll", reason: "disconnect-visible" }],
});
}
// - a LOCAL live turn (first drop) OR a live-follow re-attach (a SUBSEQUENT
// drop) -> (re-)enter the reconnect ladder. #488 commit 3: allowed
// REPEATEDLY — `liveFollow` is kept across a successful re-attach, so the
// second break reconnects again instead of falling to silent poll.
// #488 commit 2: gated on the RUN-FACT (or an existing live-follow), NOT on
// the presence of an assistant message — a setup-phase break still recovers.
// - visible content already on screen -> keep it, ALSO poll to terminal
// (a full replay could clobber the fuller live tail);
// - no visible content -> the reconnect ladder rebuilds it.
if (m.ctx.runFact || m.ctx.liveFollow) {
const effects: Effect[] = [
{ type: "scheduleReconnect", attempt: 1, delayMs: reconnectDelayMs(1) },
];
if (event.hasVisibleContent) effects.push({ type: "armPoll", reason: "disconnect-visible" });
return command(m, { name: "reconnecting", attempt: 1, failed: false }, effects, {
ownership: "observer",
liveFollow: true,
});
}
// No run to recover: a plain disconnect. Surface the terminal notice.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
});
case "FINISH_ERROR":
return to(m, { name: "error", kind: event.kind }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
// ---- mount attach (resume) ----------------------------------------
case "ATTACH_START":
// A reopened tab attaches to a still-running run: observer ownership.
// #488 F2: ONLY from idle. The mount `getRun` round-trip resolves async, and
// a local send may have started meanwhile (phase `sending`, ownership local);
// a late ATTACH_START must NOT hijack that local turn into an observer-attach
// (queue would stop flushing, "Send now" would hide). Guarding in the reducer
// covers every dispatch source.
if (m.phase.name !== "idle") return stay(m);
return command(m, { name: "attaching" }, [{ type: "resumeStream" }], {
ownership: "observer",
runFact: event.runId ? { runId: event.runId } : m.ctx.runFact,
});
case "ATTACH_LIVE":
// The attach GET returned a live 2xx stream — follow it as an observer.
// Review #1: guard by SOURCE phase. The epoch filter alone is not enough — a
// POLL_TERMINAL uses to() (no epoch bump) and does not abort the in-flight
// GET, so a slow 2xx landing after the machine already left `attaching` (e.g.
// the armed poll saw the terminal row -> idle) would resurrect a settled run
// into a phantom `streaming`. Only enter streaming FROM `attaching`.
if (m.phase.name !== "attaching") return stay(m);
return to(m, { name: "streaming" });
case "ATTACH_NONE":
// 204 / non-2xx / throw: nothing live to attach. Arm the degraded poll to
// follow the run to terminal from the DB. This is a soft-negative run-fact
// (204 on a non-stripped path is authoritative-negative; the runtime may
// pass a RUN_FACT null separately). Keep the run-fact as-is here.
// Review #1: guard by source phase for consistency (a late outcome after the
// machine already left `attaching` must not re-arm a poll).
if (m.phase.name !== "attaching") return stay(m);
return to(m, { name: "polling", reason: "attach-none" }, {
effects: [{ type: "armPoll", reason: "attach-none" }],
});
// ---- reconnect after a live disconnect ----------------------------
case "RECONNECT_ATTEMPT":
// A scheduled backoff fired — fire the attach GET. epoch++ so the previous
// attempt's late outcome cannot drive this one.
if (m.phase.name !== "reconnecting") return stay(m);
return command(
m,
{ name: "reconnecting", attempt: event.attempt, failed: false },
[{ type: "resumeStream" }],
);
case "RECONNECT_ATTACHED":
// #488 commit 3: a live re-attach succeeded. Reset to streaming — the
// attempt counter is dropped, so a LATER disconnect can start a fresh
// ladder from attempt 1 (the old one-shot `!wasResumed` gate forbade a
// second cycle, sending the second break to silent poll).
// Review #1: guard by SOURCE phase. The armed degraded poll can reach the
// terminal row (POLL_TERMINAL -> idle, via to(), NO epoch bump, GET not
// aborted) BEFORE a slow reconnect GET returns 2xx; without this guard that
// late RECONNECT_ATTACHED (same epoch) would resurrect a settled run into a
// phantom `streaming`. Only re-enter streaming FROM `reconnecting`.
if (m.phase.name !== "reconnecting") return stay(m);
return to(m, { name: "streaming" }, {
effects: [{ type: "cancelReconnect" }, { type: "disarmPoll" }],
});
case "RECONNECT_NONE": {
// 204 / error during a reconnect attempt. Arm the degraded poll as the
// belt-and-suspenders fallback, then either back off to the next attempt
// or, at the cap, surface the manual Retry ("failed").
if (m.phase.name !== "reconnecting") return stay(m);
const attempt = m.phase.attempt;
if (attempt < RECONNECT_MAX_ATTEMPTS) {
return command(
m,
{ name: "reconnecting", attempt: attempt + 1, failed: false },
[
{ type: "armPoll", reason: "attach-none" },
{ type: "scheduleReconnect", attempt: attempt + 1, delayMs: reconnectDelayMs(attempt + 1) },
],
);
}
return to(m, { name: "reconnecting", attempt, failed: true }, {
effects: [{ type: "armPoll", reason: "reconnect-exhausted" }],
});
}
case "RETRY":
// Manual Retry from the "failed" reconnect banner OR the stalled banner.
if (m.phase.name === "reconnecting" && m.phase.failed) {
return command(
m,
{ name: "reconnecting", attempt: 1, failed: false },
[{ type: "resumeStream" }],
);
}
if (m.phase.name === "stalled") {
// Re-arm the poll to try to catch the run up again.
return command(m, { name: "polling", reason: "attach-none" }, [
{ type: "armPoll", reason: "attach-none" },
]);
}
return stay(m);
// ---- degraded poll -------------------------------------------------
case "POLL_TERMINAL":
// The run reached a terminal row via the poll (or the reconcile merge). Go
// idle and disarm everything (I4: this is a DATA-driven exit, incl. exit
// from `stopping`). Review #2: reset ownership to local.
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
case "POLL_IDLE_CAP":
// Review #4: `stopping` also arms the poll (STOP_REQUESTED) but has NO other
// backstop — an observer-stop with no SDK stream to fire onFinish, whose
// server stop never drives the run terminal, would poll the DB forever. Give
// it a bounded exit: cap -> idle + disarm (NOT `stalled`; Stop was already
// pressed, so there is nothing for the user to retry).
if (m.phase.name === "stopping") {
return to(m, { name: "idle" }, {
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
}
// #488 commit 4a: the poll hit the inactivity cap. Instead of going SILENT
// (the old "forever half-done answer"), surface a stalled banner + Retry.
if (m.phase.name !== "polling" && m.phase.name !== "reconnecting") return stay(m);
return to(m, { name: "stalled" }, {
effects: [{ type: "disarmPoll" }, { type: "cancelReconnect" }],
});
// ---- run-fact ------------------------------------------------------
case "RUN_FACT": {
const runFact = event.runFact;
// A fresh NEGATIVE fact (no active run) cancels recovery immediately (I3):
// there is nothing to reconnect to / poll for.
if (!runFact) {
if (
m.phase.name === "reconnecting" ||
m.phase.name === "attaching" ||
m.phase.name === "polling" ||
m.phase.name === "stopping"
) {
return to(m, { name: "idle" }, {
// Review #2: reset ownership to local on this terminal transition.
ctx: { runFact: null, liveFollow: false, ownership: "local" },
effects: [{ type: "cancelReconnect" }, { type: "disarmPoll" }],
});
}
return to(m, m.phase, { ctx: { runFact: null } });
}
// A positive fact just updates the context (pessimism toward an attempt: a
// stale-but-positive fact permits entering recovery; a 204 will cut it).
return to(m, m.phase, { ctx: { runFact } });
}
// ---- stop ----------------------------------------------------------
case "STOP_REQUESTED":
// Authoritative stop of a detached run. Enter `stopping` and fire stopRun +
// abort the local/attach reader. ALSO arm the poll so the terminal row is
// observed — the exit is by DATA (I4: a terminal row / negative run-fact),
// never by the stopRun HTTP response (which returns after abort, before
// finalization). For a local turn the aborted stream's onFinish (ANY finish)
// is HONORED in `stopping` at the top of reduce() — regardless of generation
// — and exits to idle; the armed poll is the fallback for an observer stop
// with no local onFinish.
return command(
m,
{ name: "stopping" },
[
{ type: "stopRun" },
{ type: "abortAttach" },
{ type: "cancelReconnect" },
{ type: "armPoll", reason: "attach-none" },
],
);
// ---- supersede (CAS) ----------------------------------------------
case "SUPERSEDE_REQUESTED":
// "Interrupt and send now": CAS POST /stream { supersede }. epoch++ so a
// late outcome of the interrupted run is dropped.
return command(
m,
{ name: "superseding" },
[{ type: "supersede", targetRunId: event.targetRunId }, { type: "cancelReconnect" }, { type: "disarmPoll" }],
);
case "SUPERSEDE_READY": {
// CAS succeeded (old run stopped/settled, slot taken, new run begun). We
// are now the local streamer of the NEW run. Adopt its runId if provided.
const runFact = event.runId ? { runId: event.runId } : m.ctx.runFact;
return to(m, { name: "streaming" }, {
ctx: { ownership: "local", runFact, liveFollow: false },
});
}
case "SUPERSEDE_MISMATCH":
// The active run moved between the click and the CAS. Per the spec: verify
// via /run rather than blindly banner — the mismatch may be our own already-
// superseded run. Surface a classified error AND fire a run-fact verify.
return to(m, { name: "error", kind: "supersede-mismatch" }, {
ctx: { runFact: event.currentRunId ? { runId: event.currentRunId } : m.ctx.runFact },
effects: [{ type: "postRun", reason: "verify" }],
});
case "SUPERSEDE_TIMEOUT":
// The old run did not settle within W. Nothing persisted; the composer keeps
// its text. Classified error, NO auto-retry (the old client retry ladder is
// removed in #488 commit 5).
return to(m, { name: "error", kind: "supersede-timeout" });
case "SUPERSEDE_INVALID":
return to(m, { name: "error", kind: "supersede-invalid" });
case "RUN_ALREADY_ACTIVE":
// A plain POST hit the one-active-run gate. NO auto-retry — the composer
// offers "interrupt and send" (supersede) instead. #497/S4: adopt the
// server's activeRunId as the run-fact so that supersede can TARGET the
// (possibly foreign-tab) active run via the CAS, rather than a blind
// promote+abort that just 409s again. A stale/absent id keeps the prior fact.
return to(m, { name: "error", kind: "run-already-active" }, {
ctx: { runFact: event.activeRunId ? { runId: event.activeRunId } : m.ctx.runFact },
});
// ---- lifecycle -----------------------------------------------------
case "DISPOSE":
// Unmount: abort in-flight controllers, drop timers, and bump the epoch so
// NO late callback can drive this (now dead) machine (I5).
return command(
m,
{ name: "idle" },
[
{ type: "abortAttach" },
{ type: "cancelReconnect" },
{ type: "disarmPoll" },
],
{ liveFollow: false },
);
default: {
// Exhaustiveness guard.
const _never: never = event;
void _never;
return stay(m);
}
}
}
@@ -3,6 +3,7 @@ import {
resolveAdoptedChatId,
newlyAddedChatIds,
extractServerChatId,
extractRunId,
} from "./adopt-chat-id";
describe("resolveAdoptedChatId", () => {
@@ -70,3 +71,17 @@ describe("extractServerChatId", () => {
expect(extractServerChatId(undefined)).toBeUndefined();
});
});
describe("extractRunId", () => {
it("reads a string runId from the start metadata", () => {
expect(extractRunId({ metadata: { runId: "run-1" } })).toBe("run-1");
});
it("returns undefined when runId is absent", () => {
expect(extractRunId({ metadata: { chatId: "c" } })).toBeUndefined();
expect(extractRunId({})).toBeUndefined();
expect(extractRunId(undefined)).toBeUndefined();
});
it("returns undefined for a non-string runId", () => {
expect(extractRunId({ metadata: { runId: 7 } })).toBeUndefined();
});
});
@@ -56,6 +56,20 @@ export function extractServerChatId(
return typeof m?.chatId === "string" ? m.chatId : undefined;
}
/**
* #488: read the authoritative RUN id off a streaming assistant message. The
* server attaches it as `message.metadata.runId` on the `start` part when a run
* wraps the turn (see server `chatStreamMetadata`, #184/#487). This is the live
* run-fact update the client FSM adopts (mirrors `extractServerChatId`). Returns
* it only when it is a string; undefined otherwise.
*/
export function extractRunId(
message: { metadata?: unknown } | undefined,
): string | undefined {
const m = message?.metadata as { runId?: string } | undefined;
return typeof m?.runId === "string" ? m.runId : undefined;
}
/**
* The deduped set of ids present in `afterIds` but not in `beforeIds`. A
* paginated/flatMapped list can repeat the same id, so dedupe: one genuinely-new
@@ -42,6 +42,70 @@ describe("describeChatError", () => {
);
});
// #488 commit 5: the #487 concurrency-gate / supersede 409s. FULL real bodies:
// a ConflictException(object) whose response is serialized verbatim, carrying a
// `code` and statusCode 409. Each must classify to a human text, not raw JSON.
it("classifies A_RUN_ALREADY_ACTIVE (409) as already-answering, not raw JSON", () => {
const body =
'{"message":"A run is already active for this chat","code":"A_RUN_ALREADY_ACTIVE","statusCode":409}';
expect(describeChatError(body, t).title).toBe(
"The agent is already answering",
);
// Never leaks the raw code as the detail.
expect(describeChatError(body, t).detail).not.toContain("A_RUN_ALREADY_ACTIVE");
});
it("classifies SUPERSEDE_TARGET_MISMATCH (409) as run-changed", () => {
// Real server body shape: the current run id is `activeRunId` (NOT `runId`) —
// see ai-chat.controller.ts. describeChatError classifies off `code` only.
const body =
'{"message":"active run does not match the supersede target","code":"SUPERSEDE_TARGET_MISMATCH","activeRunId":"run-x","statusCode":409}';
expect(describeChatError(body, t).title).toBe(
"Couldn't interrupt — the run changed",
);
});
it("classifies SUPERSEDE_TIMEOUT (409) as couldn't-interrupt-in-time", () => {
const body =
'{"message":"the run did not settle within the supersede window","code":"SUPERSEDE_TIMEOUT","statusCode":409}';
expect(describeChatError(body, t).title).toBe("Couldn't interrupt in time");
});
it("classifies SUPERSEDE_INVALID (409) as couldn't-interrupt-that-run", () => {
const body =
'{"message":"supervise requires chatId","code":"SUPERSEDE_INVALID","statusCode":409}';
expect(describeChatError(body, t).title).toBe(
"Couldn't interrupt that run",
);
});
it("ORDER GUARD: A_RUN_ALREADY_ACTIVE wins over any generic status branch", () => {
// Even though the body could superficially look 4xx-ish, the code branch runs
// first, so it is never mislabeled by a generic status heading.
const body =
'{"message":"conflict","code":"A_RUN_ALREADY_ACTIVE","statusCode":409}';
const view = describeChatError(body, t);
expect(view.title).not.toBe("Something went wrong");
expect(view.title).not.toBe("AI provider not configured");
});
it("classifies a token-degeneration abort under the SAME 'Response stopped.' marker the live view shows (#495)", () => {
// The exact reason the server persists in metadata.error on a degeneration
// abort (ai-chat.service OUTPUT_DEGENERATION_ERROR). Live, this event shows
// the neutral "Response stopped." notice; the persisted banner MUST match it
// so live and refetch never disagree.
const view = describeChatError(
"Output degeneration detected (repeated token loop)",
t,
);
expect(view.title).toBe("Response stopped.");
expect(view.detail).toBe(
"The answer was stopped automatically because the model fell into a repeated output loop.",
);
// Regression guard: it must NOT fall through to the generic heading.
expect(view.title).not.toBe("Something went wrong");
});
it("classifies a dropped connection (ECONNRESET) as a lost-connection error", () => {
expect(
describeChatError("Cannot connect to API: read ECONNRESET", t).title,
@@ -39,6 +39,60 @@ export function describeChatError(
};
}
// #488 commit 5: the #487 concurrency-gate / supersede 409s. These arrive as a
// ConflictException(object) body carrying a `code` (and statusCode 409). They
// MUST be classified by `code` STRICTLY BEFORE any generic status branch, or the
// user sees the raw JSON `{"code":"A_RUN_ALREADY_ACTIVE",…}`. The code strings
// are the real #487 server contract (ai-chat.controller.ts) — do not invent.
if (/"code"\s*:\s*"A_RUN_ALREADY_ACTIVE"/.test(msg)) {
return {
title: t("The agent is already answering"),
detail: t(
"This chat already has a run in progress. Wait for it to finish, or interrupt it and send now.",
),
};
}
if (/"code"\s*:\s*"SUPERSEDE_TARGET_MISMATCH"/.test(msg)) {
return {
title: t("Couldn't interrupt — the run changed"),
detail: t(
"The run you tried to interrupt is no longer the active one. Check the latest answer and try again.",
),
};
}
if (/"code"\s*:\s*"SUPERSEDE_TIMEOUT"/.test(msg)) {
return {
title: t("Couldn't interrupt in time"),
detail: t(
"The previous run didn't stop in time. Nothing was sent — try sending again.",
),
};
}
if (/"code"\s*:\s*"SUPERSEDE_INVALID"/.test(msg)) {
return {
title: t("Couldn't interrupt that run"),
detail: t(
"The run to interrupt doesn't belong to this chat. Reload and try again.",
),
};
}
// Our own token-degeneration abort (#444): the server aborts a runaway
// repetition loop and persists this exact reason in metadata.error. LIVE, the
// same abort surfaces as the neutral "Response stopped." notice (the client
// cannot tell it from a manual Stop mid-stream), so the persisted banner must
// read the SAME "Response stopped." marker — otherwise the live view and a
// later refetch show two different texts for one event. The detail explains the
// loop-guard cause without contradicting the shared heading.
if (/output degeneration detected|repeated token loop/i.test(msg)) {
return {
title: t("Response stopped."),
detail: t(
"The answer was stopped automatically because the model fell into a repeated output loop.",
),
};
}
if (/"statusCode"\s*:\s*403\b/.test(msg)) {
return {
title: t("AI chat is disabled"),
@@ -0,0 +1,65 @@
import { describe, it, expect } from "vitest";
import * as Y from "yjs";
import { yHistoryAvailability } from "./use-toolbar-state.ts";
// Undo/redo availability is derived from the Yjs UndoManager's PRIVATE
// `undoStack` / `redoStack` fields (see use-toolbar-state.ts for why we read the
// stack lengths directly instead of the expensive `editor.can().undo()` dry-run).
// These tests lock in the behavior AND pin the library shape so a yjs / y-undo
// upgrade that renames/restructures those internals fails loudly here rather than
// silently enabling/disabling the toolbar buttons in production.
describe("yHistoryAvailability", () => {
it("reports availability from the stack lengths", () => {
expect(yHistoryAvailability({ undoStack: [], redoStack: [] })).toEqual({
canUndo: false,
canRedo: false,
});
expect(
yHistoryAvailability({ undoStack: [{}], redoStack: [] }),
).toEqual({ canUndo: true, canRedo: false });
expect(
yHistoryAvailability({ undoStack: [{}], redoStack: [{}, {}] }),
).toEqual({ canUndo: true, canRedo: true });
});
it("returns null when the private stack shape is unrecognized (upgrade guard)", () => {
// Simulates a yjs / y-undo upgrade that renames or restructures the private
// fields: the caller then falls back to the safe prosemirror-history default
// instead of throwing on `.length` of undefined or reading garbage.
expect(yHistoryAvailability(undefined)).toBeNull();
expect(yHistoryAvailability(null)).toBeNull();
expect(yHistoryAvailability({})).toBeNull();
expect(yHistoryAvailability({ undoStack: 5, redoStack: 5 })).toBeNull();
// Only one stack present (partial rename) is still not trusted.
expect(yHistoryAvailability({ undoStack: [] })).toBeNull();
});
it("pin-test: a real yjs UndoManager still exposes undoStack/redoStack arrays", () => {
const doc = new Y.Doc();
const text = doc.getText("prosemirror");
const undoManager = new Y.UndoManager(text);
// Fresh manager: both stacks empty -> nothing to undo/redo.
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: false,
canRedo: false,
});
// A tracked edit must push onto the private undoStack. If a future yjs
// renames these fields, yHistoryAvailability(undoManager) returns null and
// the expectation below fails loudly.
text.insert(0, "hello");
undoManager.stopCapturing();
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: true,
canRedo: false,
});
// Undoing moves the item to the redoStack -> redo becomes available.
undoManager.undo();
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: false,
canRedo: true,
});
});
});
@@ -35,6 +35,30 @@ export interface ToolbarState {
// When neither history backend is installed (the pre-sync static editor —
// mainExtensions only, undoRedo disabled), both fall through to 0 -> false,
// matching the previous `safeCan` behavior.
// Reads the Yjs UndoManager's undo/redo availability from its stack lengths.
//
// `undoStack` / `redoStack` are PRIVATE y-undo / yjs internals, so we touch them
// defensively: a yjs or y-undo upgrade that renames or restructures these fields
// must not silently mis-drive the toolbar buttons (nor throw on `.length` of
// `undefined`). We only trust them when they are actually arrays; otherwise this
// returns null and the caller falls back to a safe default. The pin-test in
// use-toolbar-state.test.ts asserts the current library shape, so an upgrade that
// breaks this contract fails loudly there instead of failing silently in the UI.
export function yHistoryAvailability(
undoManager: unknown,
): { canUndo: boolean; canRedo: boolean } | null {
if (!undoManager || typeof undoManager !== "object") return null;
const { undoStack, redoStack } = undoManager as {
undoStack?: unknown;
redoStack?: unknown;
};
if (!Array.isArray(undoStack) || !Array.isArray(redoStack)) return null;
return {
canUndo: undoStack.length > 0,
canRedo: redoStack.length > 0,
};
}
function historyAvailability(editor: Editor): {
canUndo: boolean;
canRedo: boolean;
@@ -43,16 +67,14 @@ function historyAvailability(editor: Editor): {
// Collaboration history (Yjs) takes precedence when present.
const yState = yUndoPluginKey.getState(state) as
| { undoManager?: { undoStack: unknown[]; redoStack: unknown[] } }
| { undoManager?: unknown }
| undefined;
if (yState?.undoManager) {
return {
canUndo: yState.undoManager.undoStack.length > 0,
canRedo: yState.undoManager.redoStack.length > 0,
};
}
const yAvail = yHistoryAvailability(yState?.undoManager);
if (yAvail) return yAvail;
// Plain prosemirror-history (returns 0 when the history plugin is absent).
// This is also the safe default when a Yjs UndoManager is present but its
// private stack shape is no longer recognized (yHistoryAvailability -> null).
return {
canUndo: undoDepth(state) > 0,
canRedo: redoDepth(state) > 0,
@@ -13,8 +13,7 @@ let currentAlias: IShareAlias | null = null;
let availabilityResult: {
valid: boolean;
available: boolean;
currentPageId: string | null;
} = { valid: true, available: true, currentPageId: null };
} = { valid: true, available: true };
vi.mock("@/features/share/queries/share-query.ts", () => ({
useShareAliasForPageQuery: () => ({ data: currentAlias }),
@@ -56,7 +55,7 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
beforeEach(() => {
setMutateAsync.mockReset();
currentAlias = null;
availabilityResult = { valid: true, available: true, currentPageId: null };
availabilityResult = { valid: true, available: true };
});
it("shows a 'will move it here' HINT (not a terminal error) when the name belongs to another page, and keeps Save enabled", async () => {
@@ -65,7 +64,6 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
availabilityResult = {
valid: true,
available: false,
currentPageId: "page-X",
};
renderSection("page-Y");
@@ -97,7 +95,6 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
availabilityResult = {
valid: true,
available: false,
currentPageId: "page-X",
};
// The server rejects the un-confirmed save asking the client to confirm.
setMutateAsync.mockRejectedValueOnce({
@@ -106,7 +103,6 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
status: 409,
data: {
code: "ALIAS_REASSIGN_REQUIRED",
currentPageId: "page-X",
currentPageTitle: "Alias Test Page X",
},
},
@@ -48,7 +48,6 @@ export default function ShareAliasSection({
const [availability, setAvailability] = useState<{
valid: boolean;
available: boolean;
currentPageId: string | null;
} | null>(null);
const [reassign, setReassign] = useState<{
alias: string;
@@ -76,7 +75,6 @@ export default function ShareAliasSection({
setAvailability({
valid: res.valid,
available: res.available,
currentPageId: res.currentPageId,
});
} catch {
setAvailability(null);
@@ -108,7 +108,6 @@ export interface IShareAliasAvailability {
alias: string;
valid: boolean;
available: boolean;
currentPageId: string | null;
}
export interface ISharedPageTree {
@@ -6,6 +6,8 @@ import {
nextReindexPollInterval,
isReindexComplete,
isReindexButtonLoading,
reindexRunKey,
isNewReindexRun,
} from './ai-provider-settings';
describe('resolveCardStatus', () => {
@@ -221,6 +223,128 @@ describe('isReindexComplete', () => {
});
});
describe('reindexRunKey', () => {
it('is null when the status carries no run identity', () => {
expect(reindexRunKey(undefined)).toBeNull();
expect(
reindexRunKey({ reindexing: false, indexedPages: 5, totalPages: 5 }),
).toBeNull();
});
it('is null for a legacy/degraded record with an empty runId', () => {
// The server sends runId='' for a record written before the field existed;
// the client must treat that as "no identity" (fall back to prior behaviour).
expect(
reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: '',
reindexStartedAt: 1000,
}),
).toBeNull();
});
it('folds runId and startedAt into one stable key', () => {
expect(
reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
}),
).toBe('run-a:1000');
});
it('changes when the runId changes for the same startedAt', () => {
const a = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
});
const b = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-b',
reindexStartedAt: 1000,
});
expect(a).not.toBe(b);
});
it('changes when the same runId restarts at a new startedAt', () => {
const a = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
});
const b = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 2000,
});
expect(a).not.toBe(b);
});
});
describe('isNewReindexRun (poll keying on runId)', () => {
// Derive the status shape from the helper itself so the test needs no export
// of the component-internal ReindexStatus type.
type ReindexStatusLike = NonNullable<Parameters<typeof reindexRunKey>[0]>;
const run = (runId: string, startedAt: number): ReindexStatusLike => ({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId,
reindexStartedAt: startedAt,
});
it('first identity after none latched is a NEW run', () => {
expect(isNewReindexRun(null, run('run-a', 1000))).toBe(true);
});
it('the SAME identity is not a new run (same run being watched)', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(isNewReindexRun(key, run('run-a', 1000))).toBe(false);
});
it('a DIFFERENT runId is a new run (reset per-run poll state)', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(isNewReindexRun(key, run('run-b', 1000))).toBe(true);
});
it('an identity-less poll (no runId / cleared record) is never a new run', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(
isNewReindexRun(key, {
reindexing: false,
indexedPages: 10,
totalPages: 10,
}),
).toBe(false);
});
it('a legacy empty-runId poll does not spuriously reset a latched run', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(
isNewReindexRun(key, {
reindexing: true,
indexedPages: 3,
totalPages: 10,
runId: '',
reindexStartedAt: 1000,
}),
).toBe(false);
});
});
describe('isReindexButtonLoading', () => {
it('loads while the POST mutation is pending', () => {
expect(
@@ -173,9 +173,43 @@ export function resolveKeyField(
// Subset of the status payload that drives the reindex poll decisions.
type ReindexStatus = Pick<
IAiSettings,
"reindexing" | "indexedPages" | "totalPages"
"reindexing" | "indexedPages" | "totalPages" | "runId" | "reindexStartedAt"
>;
/**
* A stable per-RUN key for the reindex poll: `runId:startedAt`, or `null` when
* the status carries no run identity (no active run, or a legacy/degraded
* server record with an empty runId). Two polls of the SAME run share a key; a
* new run mints a fresh runId and so a different key.
*
* This is the single place the client turns the server's run identity into the
* value it keys on it removes the "is this the same run I've been watching or
* a brand-new one?" ambiguity that made a class of reindex-status bugs (a stale
* pre-reindex snapshot vs a fresh run) get fixed twice (#262). `startedAt` is
* folded in so a run that somehow reuses a runId but restarted is still new.
*/
export function reindexRunKey(status: ReindexStatus | undefined): string | null {
const runId = status?.runId;
if (!runId) return null;
return `${runId}:${status?.reindexStartedAt ?? ""}`;
}
/**
* Decide whether the latest poll represents a NEW reindex run relative to the
* run key the client last latched (`prevKey`, `null` if none yet). True only
* when the status carries an identity AND it differs from the latched one the
* signal to reset any per-run poll state (the "seen active" latch / progress the
* UI held). The same identity (or no identity) is NOT a new run, so an unchanged
* or identity-less poll never resets mid-run.
*/
export function isNewReindexRun(
prevKey: string | null,
status: ReindexStatus | undefined,
): boolean {
const key = reindexRunKey(status);
return key !== null && key !== prevKey;
}
/**
* Decide the TanStack Query `refetchInterval` while a reindex may be running.
* Returns the poll interval (ms) to keep polling, or `false` to stop.
@@ -320,6 +354,13 @@ export default function AiProviderSettings() {
// counter at 0 until a manual reload. A ref (not state) because it must not
// trigger a render and is only ever read where `reindexing` is already false.
const reindexSeenActiveRef = useRef(false);
// The run identity (runId:startedAt) the current poll window is keyed on. When
// a poll reports a DIFFERENT runId the server has started a NEW run, so we
// re-latch to it and reset `reindexSeenActiveRef` — a fresh run must never
// inherit the previous run's "seen active"/completion state (which would stop
// polling immediately or read the old run's counters as this run's). null =
// no run keyed yet (steady state, or a legacy record without a runId).
const reindexRunKeyRef = useRef<string | null>(null);
// Only admins may read the (masked) AI settings; the server enforces this too.
const { data: settings, isLoading } = useAiSettingsQuery(isAdmin, (query) =>
@@ -336,6 +377,14 @@ export default function AiProviderSettings() {
// unmount because the deadline state goes away with the component.
useEffect(() => {
if (reindexDeadline === null) return;
// Key the poll on the run identity: if this poll carries a runId different
// from the one we latched, the server started a NEW run, so adopt it and
// drop the per-run "seen active" latch (a fresh run must not inherit the
// previous run's completion state). Same runId => same run, leave it alone.
if (isNewReindexRun(reindexRunKeyRef.current, settings)) {
reindexRunKeyRef.current = reindexRunKey(settings);
reindexSeenActiveRef.current = false;
}
// Latch "we have seen the active run" the moment a poll reports it, so the
// completion check below (and the refetchInterval's) only fires once the run
// has genuinely started — never on the stale pre-reindex snapshot.
@@ -1220,6 +1269,10 @@ export default function AiProviderSettings() {
// immediately.
onSuccess: () => {
reindexSeenActiveRef.current = false;
// Forget the previous run's identity so the first poll of
// this window (carrying the new run's runId) is recognized
// as a new run and keyed afresh.
reindexRunKeyRef.current = null;
setReindexDeadline(Date.now() + REINDEX_POLL_CAP_MS);
},
})
@@ -51,6 +51,14 @@ export interface IAiSettings {
// True while a full workspace reindex is actively running; the counts above
// then reflect the live run progress (done climbs 0 -> total).
reindexing?: boolean;
// Identity of the ACTIVE reindex run (present only while `reindexing`). The
// poll keys on `runId`: a changed value means a NEW run (reset the per-run
// poll state the UI latched), the same value is the run already being watched.
// Absent/empty ('') => no identity available; the client keeps prior behaviour.
runId?: string;
// Epoch-ms the active run started; paired with `runId` so a restart with a
// recycled id is still detected as a new run.
reindexStartedAt?: number;
}
// Update payload. Key semantics (same for `apiKey` and `embeddingApiKey`):
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { templateRoute } from "./route-template";
import { templateRoute, KNOWN_ROUTE_TEMPLATES } from "./route-template";
describe("templateRoute", () => {
it("templates a space page path (never leaks slugs)", () => {
@@ -32,4 +32,30 @@ describe("templateRoute", () => {
expect(templateRoute("/weird/unknown/thing")).toBe("other");
expect(templateRoute("/s/team/p/slug/extra/segments")).toBe("other");
});
// The server's /api/telemetry/vitals mirror (ALLOWED_ROUTE_TEMPLATES) drops any
// route outside KNOWN_ROUTE_TEMPLATES, so templateRoute must NEVER emit a label
// that is not in that dictionary — otherwise legit client metrics get dropped.
it("only ever emits labels contained in KNOWN_ROUTE_TEMPLATES (#495)", () => {
const samples = [
"/",
"/home",
"/settings/members",
"/settings/groups/g-1",
"/s/team",
"/s/team/trash",
"/s/team/p/slug",
"/p/slug",
"/share/abc",
"/share/abc/p/slug",
"/share/p/slug",
"/labels/urgent",
"/invites/inv-1",
"/weird/unknown/thing", // -> "other"
"/deep/unmatched/x/y/z", // -> "other"
];
for (const path of samples) {
expect(KNOWN_ROUTE_TEMPLATES.has(templateRoute(path))).toBe(true);
}
});
});
@@ -44,6 +44,22 @@ const STATIC_ROUTES = new Set<string>([
'/settings/sharing',
]);
/**
* The COMPLETE, finite vocabulary `templateRoute` can ever emit: the two
* synthetic labels (`/` and `other`), the static routes, and the dynamic
* templates. Exported so the public `/api/telemetry/vitals` endpoint can reject
* any `route` outside this dictionary server-side (the endpoint is anonymous, so
* an un-checked `route` is a free-text write surface). The server keeps a mirror
* (`ALLOWED_ROUTE_TEMPLATES` in client-metrics.constants.ts) this is the
* canonical source; keep them in lockstep.
*/
export const KNOWN_ROUTE_TEMPLATES: ReadonlySet<string> = new Set<string>([
'/',
'other',
...STATIC_ROUTES,
...ROUTE_PATTERNS.map((p) => p.template),
]);
export function templateRoute(pathname: string): string {
// Normalise a trailing slash (except root).
const path =
@@ -43,6 +43,9 @@ function makeRepo(overrides: Record<string, jest.Mock> = {}) {
workspaceId: v.workspaceId,
})),
update: jest.fn(async () => ({ id: 'run-1' })),
// #487: terminal finalize now goes through the CONDITIONAL write. Default
// returns a truthy row (the run WAS active -> this call wrote it).
finalizeIfActive: jest.fn(async () => ({ id: 'run-1', status: 'succeeded' })),
markStopRequested: jest.fn(async () => ({ id: 'run-1' })),
findActiveByChat: jest.fn(async () => undefined),
findLatestByChat: jest.fn(async () => undefined),
@@ -336,14 +339,12 @@ describe('AiChatRunService run lifecycle', () => {
await svc.finalizeRun('run-1', 'ws-1', 'error', 'provider blew up');
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenCalledWith(
// #487: the terminal write is CONDITIONAL (finalizeIfActive); finishedAt is
// stamped inside the repo method, so the service passes just status + error.
expect(repo.finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({
status: 'failed',
error: 'provider blew up',
finishedAt: expect.any(Date),
}),
expect.objectContaining({ status: 'failed', error: 'provider blew up' }),
);
});
@@ -366,8 +367,8 @@ describe('AiChatRunService run lifecycle', () => {
// A second settle (e.g. a streamText callback firing after the catch) no-ops.
await svc.finalizeRun('run-1', 'ws-1', 'completed', undefined);
expect(repo.update).toHaveBeenCalledTimes(1);
expect(repo.update).toHaveBeenCalledWith(
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(1);
expect(repo.finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'failed', error: 'first' }),
@@ -389,8 +390,8 @@ describe('AiChatRunService run lifecycle', () => {
const updateGate = new Promise((res) => {
resolveUpdate = res;
});
const update = jest.fn(() => updateGate);
const repo = makeRepo({ update });
const finalizeIfActive = jest.fn(() => updateGate);
const repo = makeRepo({ finalizeIfActive });
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({
chatId: 'chat-1',
@@ -399,23 +400,23 @@ describe('AiChatRunService run lifecycle', () => {
});
// Fire both before the (pending) update resolves. The first synchronously
// claims the entry (active.delete) and awaits update; the second, started in
// the same macrotask, finds the entry already gone and returns at the claim
// WITHOUT ever calling update.
// claims the entry (active.delete) and awaits the write; the second, started
// in the same macrotask, finds the entry already gone and returns at the claim
// WITHOUT ever writing.
const p1 = svc.finalizeRun('run-1', 'ws-1', 'completed');
const p2 = svc.finalizeRun('run-1', 'ws-1', 'error', 'safety-net');
// The decisive assertion: exactly one caller reached the terminal UPDATE.
expect(update).toHaveBeenCalledTimes(1);
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
// Let the single in-flight update land; both calls resolve cleanly.
resolveUpdate({ id: 'run-1' });
resolveUpdate({ id: 'run-1', status: 'succeeded' });
await Promise.all([p1, p2]);
expect(update).toHaveBeenCalledTimes(1);
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
// The winner is the FIRST caller ('completed' -> 'succeeded'); the late
// 'error' settle never wrote, so it could not clobber the real status.
expect(update).toHaveBeenCalledWith(
expect(finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
@@ -431,10 +432,10 @@ describe('AiChatRunService run lifecycle', () => {
// 409s until a restart. The fix updates FIRST and retries.
let calls = 0;
const repo = makeRepo({
update: jest.fn(async () => {
finalizeIfActive: jest.fn(async () => {
calls += 1;
if (calls === 1) throw new Error('deadlock detected');
return { id: 'run-1' };
return { id: 'run-1', status: 'succeeded' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
@@ -447,26 +448,29 @@ describe('AiChatRunService run lifecycle', () => {
await svc.finalizeRun('run-1', 'ws-1', 'completed');
// The retry landed the terminal write: the entry is dropped (slot freed) and
// the row carries the real terminal status — NOT stranded at 'running'.
// The retry landed the terminal write: the entry is dropped (slot freed), no
// zombie left, and the row carries the real terminal status.
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenCalledTimes(2);
expect(repo.update).toHaveBeenLastCalledWith(
expect(svc.hasZombie('run-1')).toBe(false);
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(2);
expect(repo.finalizeIfActive).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
});
it('F6: if the terminal write keeps failing, the entry is RETAINED and a LATER settle completes it (chat not permanently 409d)', async () => {
it('#487 give-up: if the terminal write keeps failing, finalizeRun leaves a ZOMBIE (does NOT restore the entry) and settleZombie re-drives it', async () => {
// Worst case: the DB is down for the whole first finalize (all attempts fail).
// The run must NOT be silently lost — the entry stays so a subsequent settle
// (a streamText callback, requestStop -> onAbort, or a future sweep) can retry.
// #487 changes the give-up behaviour: the entry is NOT restored (a restored
// entry is indistinguishable from a live run). Instead a ZOMBIE record holds
// the intended terminal status, and a re-drive (settleZombie — called by the
// reconcile / supersede / opportunistic paths) applies it later.
let healthy = false;
const repo = makeRepo({
update: jest.fn(async () => {
finalizeIfActive: jest.fn(async () => {
if (!healthy) throw new Error('pool exhausted');
return { id: 'run-1' };
return { id: 'run-1', status: 'succeeded' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
@@ -480,35 +484,83 @@ describe('AiChatRunService run lifecycle', () => {
userId: 'user-1',
});
// First settle: every bounded attempt fails -> entry retained, NOT settled.
// First settle: every bounded attempt fails -> ZOMBIE, entry NOT restored.
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(true);
// F12: the give-up emits ONE explicit, greppable ERROR (run + chat context)
// so an operator can tell "gave up, run held in memory" from a per-attempt
// blip — distinct from the per-attempt warns.
expect(svc.isLocallyActive('run-1')).toBe(false); // NOT a live entry
expect(svc.hasZombie('run-1')).toBe(true);
expect(svc.zombieRunIds()).toContain('run-1');
// The give-up emits ONE explicit, greppable ERROR mentioning the zombie.
const gaveUp = errorSpy.mock.calls.some(
(c) =>
/NON-TERMINAL/.test(String(c[0])) &&
/ZOMBIE/.test(String(c[0])) &&
/run-1/.test(String(c[0])) &&
/chat-1/.test(String(c[0])),
);
expect(gaveUp).toBe(true);
// The settle notifier resolved as terminalWriteFailed (a subscriber learns the
// slot still needs the intended status applied).
const outcome = await svc.peekSettled('run-1');
expect(outcome).toEqual({
status: 'succeeded',
error: null,
terminalWriteFailed: true,
});
// The DB recovers; a later settle now succeeds and frees the slot.
// The DB recovers; a re-drive settles the zombie via the conditional UPDATE.
healthy = true;
await svc.finalizeRun('run-1', 'ws-1', 'completed');
expect(svc.isLocallyActive('run-1')).toBe(false);
expect(repo.update).toHaveBeenLastCalledWith(
const redriven = await svc.settleZombie('run-1');
expect(redriven).toBe(true);
expect(svc.hasZombie('run-1')).toBe(false);
expect(repo.finalizeIfActive).toHaveBeenLastCalledWith(
'run-1',
'ws-1',
expect.objectContaining({ status: 'succeeded' }),
);
// And it is now idempotent: a further settle no-ops (terminal row already
// written), so a double-settle can never clobber the real status.
const callsBefore = repo.update.mock.calls.length;
// A later finalizeRun is idempotent (row already terminal): it no-ops at the
// once-gate, never re-writing.
const callsBefore = repo.finalizeIfActive.mock.calls.length;
await svc.finalizeRun('run-1', 'ws-1', 'error', 'late');
expect(repo.update).toHaveBeenCalledTimes(callsBefore);
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(callsBefore);
});
it('#487 double-settle collapses to a benign no-op (conditional write; notifier resolves once)', async () => {
// A second concurrent settle is stopped at the synchronous active.delete
// claim, so the terminal write runs exactly once and the notifier resolves
// exactly once with the FIRST settler's outcome.
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'u1' });
await svc.finalizeRun('run-1', 'ws-1', 'aborted');
await svc.finalizeRun('run-1', 'ws-1', 'error', 'late'); // no-op
expect(repo.finalizeIfActive).toHaveBeenCalledTimes(1);
const outcome = await svc.peekSettled('run-1');
// peekSettled after resolve+delete falls through (notifier dropped, no zombie)
// -> undefined; the FIRST settler already resolved any earlier subscriber.
expect(outcome).toBeUndefined();
});
it('#487 late settledPromise subscriber gets the resolved outcome', async () => {
const repo = makeRepo();
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: 'chat-1', workspaceId: 'ws-1', userId: 'u1' });
// Subscribe BEFORE settle: hold the promise reference (as supersede does).
const early = svc.peekSettled('run-1');
expect(early).toBeDefined();
await svc.finalizeRun('run-1', 'ws-1', 'completed');
// The reference grabbed before settle resolves with the written outcome, even
// though the notifier was dropped from the map on resolve (bounded).
await expect(early).resolves.toEqual({
status: 'succeeded',
error: null,
terminalWriteFailed: false,
});
});
it('recordStep / linkAssistantMessage are best-effort: a repo failure is swallowed', async () => {
@@ -525,3 +577,197 @@ describe('AiChatRunService run lifecycle', () => {
).resolves.toBeUndefined();
});
});
describe('#487 AiChatRunService.supersede (CAS)', () => {
const chat = 'chat-1';
const ws = 'ws-1';
it('degrade: no active run on the chat -> caller sends a normal turn', async () => {
const repo = makeRepo({
findById: jest.fn(async () => undefined),
findActiveByChat: jest.fn(async () => undefined),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({ kind: 'degrade' });
});
it('invalid: the target run belongs to a DIFFERENT chat -> 400', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-x',
chatId: 'other-chat',
workspaceId: ws,
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({ kind: 'invalid' });
});
it('mismatch: a DIFFERENT run is active than the one targeted -> current runId', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-x', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => ({
id: 'run-live',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
expect(await svc.supersede(chat, 'run-x', ws)).toEqual({
kind: 'mismatch',
activeRunId: 'run-live',
});
});
it('ready: the target IS active -> stop it, await its (fast) settle, free the slot', async () => {
// Simulate a live long TOOL (NOT a slow UPDATE): the run stays active until an
// explicit Stop unwinds it; commit-1's race makes that settle land quickly.
// The abort listener stands in for streamText's onAbort -> finalizeRun.
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'aborted',
error: null,
})),
findActiveByChat: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
handle.signal.addEventListener('abort', () => {
void svc.finalizeRun('run-1', ws, 'aborted');
});
// supersede: getRun -> getActiveByChat(==target) -> requestStop -> the abort
// listener settles the run -> awaitSettled resolves -> ready.
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
expect(handle.signal.aborted).toBe(true); // Stop reached the run
});
it('timeout: the target never settles within W -> 409 SUPERSEDE_TIMEOUT (nothing persisted)', async () => {
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-1', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
})),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
// Do NOT settle the run: a tiny W elapses -> timeout.
const result = await svc.supersede(chat, 'run-1', ws, 30);
expect(result).toEqual({ kind: 'timeout' });
});
it('ready then a DUPLICATE supersede POST degrades (the run is already gone)', async () => {
let active: unknown = {
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
};
const repo = makeRepo({
findById: jest.fn(async () => ({
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'aborted',
error: null,
})),
findActiveByChat: jest.fn(async () => active),
finalizeIfActive: jest.fn(async () => {
active = undefined; // settling frees the active slot
return { id: 'run-1', status: 'aborted' };
}),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
const handle = await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
handle.signal.addEventListener('abort', () => {
void svc.finalizeRun('run-1', ws, 'aborted');
});
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
// The duplicate POST for the same target now finds no active run -> degrade.
expect(await svc.supersede(chat, 'run-1', ws)).toEqual({ kind: 'degrade' });
});
it('reconcileStaleRuns: aborts a stale run with NO entry/zombie; NEVER touches a live entry', async () => {
const finalizeIfActive = jest.fn(async () => ({ id: 'x', status: 'aborted' }));
const repo = makeRepo({
insert: jest.fn(async (v: any) => ({
id: 'live-1',
status: 'running',
chatId: v.chatId,
workspaceId: v.workspaceId,
})),
finalizeIfActive,
findStaleActive: jest.fn(async () => [
{ id: 'orphan-1', workspaceId: ws, chatId: 'c-orphan' },
{ id: 'live-1', workspaceId: ws, chatId: 'c-live' },
]),
});
const svc = new AiChatRunService(repo as never, makeEnv() as never);
// A LIVE run this replica owns (in the `active` map).
await svc.beginRun({ chatId: 'c-live', workspaceId: ws, userId: 'u1' });
expect(svc.isLocallyActive('live-1')).toBe(true);
const aborted = await svc.reconcileStaleRuns(15 * 60 * 1000);
expect(aborted).toBe(1);
// The orphan (no entry) was aborted; the live entry was NEVER passed to the DB.
expect(finalizeIfActive).toHaveBeenCalledTimes(1);
expect(finalizeIfActive).toHaveBeenCalledWith(
'orphan-1',
ws,
expect.objectContaining({ status: 'aborted' }),
);
expect(svc.isLocallyActive('live-1')).toBe(true);
});
it('gave-up zombie: supersede applies the intended status (settleZombie) then is ready', async () => {
let healthy = false;
let active: unknown = {
id: 'run-1',
chatId: chat,
workspaceId: ws,
status: 'running',
};
const repo = makeRepo({
findById: jest.fn(async () => ({ id: 'run-1', chatId: chat, workspaceId: ws })),
findActiveByChat: jest.fn(async () => active),
finalizeIfActive: jest.fn(async () => {
if (!healthy) throw new Error('db down');
active = undefined;
return { id: 'run-1', status: 'aborted' };
}),
});
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
jest.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
const svc = new AiChatRunService(repo as never, makeEnv() as never);
await svc.beginRun({ chatId: chat, workspaceId: ws, userId: 'u1' });
// The run's terminal write gives up -> zombie (row still 'running').
await svc.finalizeRun('run-1', ws, 'aborted');
expect(svc.hasZombie('run-1')).toBe(true);
// The DB recovers; supersede awaits the (already-resolved, terminalWriteFailed)
// settle, then settleZombie applies the intended status -> ready.
healthy = true;
expect(await svc.supersede(chat, 'run-1', ws, 10_000)).toEqual({
kind: 'ready',
});
expect(svc.hasZombie('run-1')).toBe(false);
});
});
@@ -34,6 +34,88 @@ export class RunAlreadyActiveError extends Error {
export type TurnTerminalStatus = 'completed' | 'error' | 'aborted';
export type RunTerminalStatus = 'succeeded' | 'failed' | 'aborted';
/** The terminal run statuses — the row is done once it reads one of these. */
export const RUN_TERMINAL_STATUSES: readonly RunTerminalStatus[] = [
'succeeded',
'failed',
'aborted',
];
/** Whether a persisted run status is terminal (settled). */
export function isRunTerminal(status: string | null | undefined): boolean {
return (
status === 'succeeded' || status === 'failed' || status === 'aborted'
);
}
/**
* #487: the outcome a run's {@link AiChatRunService.finalizeRun} settled with.
* `terminalWriteFailed` = the terminal write GAVE UP after the bounded retry, so
* the row is still non-terminal ('running') and a ZOMBIE record holds the
* `intended` status for a later re-drive (reconcile / supersede / boot sweep). A
* subscriber (supersede, #487 commit 3) uses this to decide whether the slot is
* genuinely free or must first have the intended status applied.
*/
export interface RunSettleOutcome {
status: RunTerminalStatus;
error: string | null;
terminalWriteFailed: boolean;
}
/**
* #487: how long a supersede waits for the target run to settle after Stop before
* it degrades to `SUPERSEDE_TIMEOUT`. W=10s is generous under a HEALTHY DB: commit
* 1's race-on-abort makes an in-app tool abort->settle in ms/hundreds of ms, so a
* live run releases its slot well within the window. Under a DB brownout the
* timeout is normal (the write cannot land); W must NOT be raised to paper
* over a slow DB a SUPERSEDE_TIMEOUT is the honest signal (nothing persisted,
* the composer keeps the user's text). Env-tunable for ops, default 10s.
*/
export const SUPERSEDE_SETTLE_TIMEOUT_MS = (() => {
const raw = Number(process.env.AI_CHAT_SUPERSEDE_TIMEOUT_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 10_000;
})();
/**
* #487: the result of the supersede CAS ({@link AiChatRunService.supersede}).
* - `degrade` : no active run on the chat (it ended between click and POST)
* the caller sends a NORMAL turn (NOT a mismatch);
* - `invalid` : the target runId belongs to a DIFFERENT chat (malformed CAS 400);
* - `mismatch` : a DIFFERENT run is active than the one the client targeted
* 409 SUPERSEDE_TARGET_MISMATCH carrying the current `activeRunId`
* (the client does NOT auto-retry);
* - `timeout` : the target did not settle within W 409 SUPERSEDE_TIMEOUT,
* nothing persisted;
* - `ready` : the target was stopped AND settled (or its zombie's intended was
* applied) the slot is free; the caller may beginRun the new run.
*/
export type SupersedeResult =
| { kind: 'degrade' }
| { kind: 'invalid' }
| { kind: 'mismatch'; activeRunId: string }
| { kind: 'timeout' }
| { kind: 'ready' };
/** A one-shot settle notifier (#487): `resolve` is called EXACTLY ONCE. */
interface Deferred<T> {
promise: Promise<T>;
resolve: (value: T) => void;
}
/**
* #487: a run whose terminal write GAVE UP (every bounded attempt failed). The
* row is stranded non-terminal ('running'); this record is the ONLY thing that
* distinguishes it from a live run, and carries the `intended` terminal status so
* a re-drive can apply it via the conditional UPDATE. Process-local (phase-1
* single-process assumption): a restart drops it, and the boot sweep then writes
* 'aborted' over the intended a documented loss (see finalizeRun).
*/
interface ZombieRun {
workspaceId: string;
chatId: string;
intended: { status: RunTerminalStatus; error: string | null };
}
export function mapTurnStatusToRun(
status: TurnTerminalStatus,
): RunTerminalStatus {
@@ -101,6 +183,22 @@ export class AiChatRunService implements OnModuleInit {
// uptime — negligible in phase 1's single process.
private readonly settled = new Set<string>();
// #487 runId -> one-shot settle notifier. Kept in a SEPARATE map from `active`
// ON PURPOSE: it must OUTLIVE the `active.delete` claim inside finalizeRun (the
// claim frees the slot the instant finalize starts), so a subscriber can still
// await the outcome after the entry is gone. Created in beginRun, resolved
// EXACTLY ONCE in finalizeRun, then removed (bounded). Absence => this replica
// has no live notifier: a subscriber falls back to the zombie map, then to the
// row (see peekSettled). Process-local (phase-1 single-process assumption).
private readonly settledPromises = new Map<string, Deferred<RunSettleOutcome>>();
// #487 runId -> ZOMBIE record: a run whose terminal write gave up (row stranded
// non-terminal). BOUNDED — an entry is added only on give-up and removed on a
// successful re-drive (settleZombie) or when the row is found already terminal;
// a process restart clears it (and the boot sweep settles the stranded row).
// Process-local (phase-1 single-process assumption).
private readonly zombies = new Map<string, ZombieRun>();
// Bounded retry for the terminal write (F6): a single PK UPDATE can fail
// transiently under many fire-and-forget writes (pool exhaustion, deadlock, a
// brief connection blip). Riding out that blip in-place matters because the
@@ -224,6 +322,10 @@ export class AiChatRunService implements OnModuleInit {
chatId: args.chatId,
workspaceId: args.workspaceId,
});
// #487: arm the one-shot settle notifier BEFORE returning, so a subscriber
// that races in immediately after begin always finds a promise to await. It
// is resolved exactly once when the run settles (or gives up).
this.settledPromises.set(run.id, this.makeDeferred<RunSettleOutcome>());
return { runId: run.id, signal: controller.signal };
}
@@ -263,47 +365,43 @@ export class AiChatRunService implements OnModuleInit {
}
/**
* Finalize a run to its terminal status (succeeded / failed / aborted),
* stamping finishedAt + any error. Best-effort, but ROBUST against a transient
* terminal-write failure (F6) AND atomically safe against a concurrent settle.
* Finalize a run to its terminal status (succeeded / failed / aborted) via a
* CONDITIONAL UPDATE, stamping finishedAt + any error. Atomically safe against a
* concurrent settle AND robust against a transient terminal-write failure.
*
* ATOMIC ONCE-CLAIM (the gate must close in ONE synchronous tick): two
* finalizeRun calls for the SAME run can race the documented real path is
* AiChatService.stream's safety-net catch settling the turn to 'error' while a
* streamText terminal callback (onFinish/onAbort/onError) ALSO settles it. The
* `settled.has` check alone is NOT a gate: it is read BEFORE the awaited UPDATE,
* so two callers can both see `false` and both write the row (last-write-wins
* clobbers the real terminal status, and the bounded retry only widens that
* window). The claim therefore happens via `active.delete`, a SYNCHRONOUS
* check-and-clear with NO await between the gate and the entry removal: the
* second concurrent caller finds the entry already gone and returns in the same
* tick, before any UPDATE. The transition "nobody is finalizing" -> "I am
* finalizing" is thus a single atomic step.
* claim happens via `active.delete`, a SYNCHRONOUS check-and-clear with NO await
* between the gate and the entry removal: the second concurrent caller finds the
* entry already gone and returns in the same tick, before any UPDATE.
*
* ORDER MATTERS (F6): once we own the claim, the terminal UPDATE happens FIRST;
* only once it SUCCEEDS do we record the run as settled. If the UPDATE fails on
* every bounded attempt we RESTORE the in-memory entry, leave the run UNsettled,
* and emit an ERROR signal that the row is left non-terminal 'running' (which
* would 409 every future turn in the chat until recovery). An in-process retry
* by a LATER settle is only POSSIBLE, never guaranteed: it needs (a) the entry
* to have been restored at the give-up path AND (b) a fresh settler to arrive
* AFTER that restore. A concurrent settler that arrives DURING the retry window
* while the entry is deleted for backoff and not yet restored is consumed at
* the synchronous `active.delete` claim (it finds nothing to delete and returns
* a no-op), so it does NOT become an in-process retrier. The NO-streamText path
* (the turn threw before streamText was wired, so ONLY the safety-net ever
* settles) likewise has no second in-process settler at all. The UNCONDITIONAL
* backstop in every case is the boot sweep on the next restart (phase 1 has no
* periodic in-process sweep); the retained entry is bounded (cleared on restart)
* and harmless meanwhile.
* ALL TERMINAL WRITES ARE CONDITIONAL (#487): `finalizeIfActive` only flips a
* row still in pending|running (mirror of the assistant message's
* `onlyIfStreaming`). So even a settle that DID reach the UPDATE (e.g. a
* reconcile stamp racing an owner finalize) can never clobber a terminal status
* the loser matches nothing and is a benign no-op. `active.delete` is the
* fast, in-process gate; the conditional WHERE is the authoritative one.
*
* IDEMPOTENT on SUCCESS (#184 review): the terminal write happens AT MOST ONCE
* per run. After a successful write the once-gate keys off {@link settled} (the
* terminal row already written) so a settle arriving AFTER the entry was already
* dropped-and-settled returns early; a settle racing the in-flight write is
* stopped earlier still, by the `active.delete` claim. Either way a genuine
* double-settle collapses to a single write and a late settle can never clobber
* the real terminal status or double-write the row.
* ZOMBIE ON GIVE-UP (#487): if every bounded attempt THROWS (the DB is down for
* the whole finalize), we do NOT restore the entry. The row is stranded
* non-terminal ('running'); we record a ZOMBIE `{ terminalWriteFailed, intended
* }` (the ONLY thing distinguishing this dead run from a live one) and resolve
* the settle notifier with `terminalWriteFailed: true`. A restore would make the
* zombie indistinguishable from a live run to every reader; instead a re-drive
* (settleZombie, called by the periodic reconcile / supersede / opportunistic
* paths) applies the intended status later via the same conditional UPDATE.
*
* DOCUMENTED LOSS (#487, single-process phase 1): if the process RESTARTS before
* a zombie is re-driven, the in-memory zombie map is gone and the boot sweep
* (unconditional) writes 'aborted' over the ACTUAL intended status. This is
* unavoidable while the run lifecycle is single-process there is no durable
* record of `intended`; a cross-process durable intent is deferred to phase 2.
*
* IDEMPOTENT: the settle notifier resolves EXACTLY ONCE; a second settle is
* stopped at `settled.has` or the `active.delete` claim, so a double-settle
* collapses to a single write and can never double-resolve or clobber the row.
*/
async finalizeRun(
runId: string,
@@ -314,13 +412,17 @@ export class AiChatRunService implements OnModuleInit {
// ---- Atomic once-claim (synchronous; NO await before the gate closes) ----
// Already terminally written -> idempotent no-op.
if (this.settled.has(runId)) return;
// Capture the entry BEFORE the delete so a total-failure path can restore it.
// Capture the entry BEFORE the delete for the give-up log context.
const entry = this.active.get(runId);
// SYNCHRONOUS check-and-clear: the FIRST caller deletes (claims) the entry;
// any concurrent SECOND caller finds nothing to delete and returns HERE, in
// the same tick, before any await — so it can never reach the UPDATE.
if (!this.active.delete(runId)) return;
const status = mapTurnStatusToRun(turnStatus);
const err = error ?? null;
const chatId = entry?.chatId ?? 'unknown';
let lastError: unknown;
for (
let attempt = 1;
@@ -328,47 +430,294 @@ export class AiChatRunService implements OnModuleInit {
attempt++
) {
try {
await this.runRepo.update(runId, workspaceId, {
status: mapTurnStatusToRun(turnStatus),
finishedAt: new Date(),
error: error ?? null,
const row = await this.runRepo.finalizeIfActive(runId, workspaceId, {
status,
error: err,
});
// Terminal write landed: arm the once-gate. The entry is already gone
// (claimed above); we do NOT restore it. The slot is now free.
// No throw => the row is now terminal (we wrote it, or it was ALREADY
// terminal — another writer won the conditional UPDATE, a benign no-op).
this.settled.add(runId);
this.zombies.delete(runId);
// Resolve with the persisted outcome: our status when WE wrote it, else
// the row's real terminal status (re-read on the already-terminal path so
// a subscriber never sees a status we did not actually persist).
const outcome: RunSettleOutcome = row
? { status, error: err, terminalWriteFailed: false }
: await this.readTerminalOutcome(runId, workspaceId, status, err);
this.resolveSettled(runId, outcome);
return;
} catch (err) {
lastError = err;
} catch (err2) {
lastError = err2;
this.logger.warn(
`Failed to finalize run ${runId} (attempt ${attempt}/${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
}): ${err instanceof Error ? err.message : 'unknown error'}`,
}): ${err2 instanceof Error ? err2.message : 'unknown error'}`,
);
if (attempt < AiChatRunService.FINALIZE_MAX_ATTEMPTS) {
await this.delay(AiChatRunService.FINALIZE_RETRY_BASE_MS * attempt);
}
}
}
// Every attempt failed: this is a give-up, materially worse than a per-attempt
// blip — the row is left NON-TERMINAL ('running'), so emit ONE explicit,
// greppable ERROR so an operator can tell "survived a blip" from "gave up, run
// held in memory until recovery" (the last warn alone says only "attempt 3/3").
// Every attempt threw: GIVE UP. The row is stranded non-terminal ('running').
// Do NOT restore the entry (a restored entry is indistinguishable from a live
// run); leave a ZOMBIE record instead, and resolve the notifier as
// terminalWriteFailed so a subscriber knows the slot still needs the intended
// status applied. One explicit, greppable ERROR so an operator can tell a
// give-up from a per-attempt blip.
this.logger.error(
`Run ${runId} (chat ${entry?.chatId ?? 'unknown'}) left NON-TERMINAL ` +
`('running'): terminal write failed after ${
AiChatRunService.FINALIZE_MAX_ATTEMPTS
} attempts; entry retained in memory, recovery deferred to next settle / ` +
`boot sweep`,
`Run ${runId} (chat ${chatId}) left NON-TERMINAL ('running'): terminal ` +
`write failed after ${AiChatRunService.FINALIZE_MAX_ATTEMPTS} attempts; ` +
`ZOMBIE recorded (intended '${status}'), recovery deferred to reconcile / ` +
`supersede / boot sweep`,
lastError,
);
// RESTORE the claimed entry (and leave the run UNsettled) so a LATER settle
// that arrives AFTER this restore MAY retry the terminal write — but that
// in-process retry is NOT guaranteed (a concurrent settler caught in the retry
// window above is consumed at the `active.delete` claim, and the no-streamText
// path has no second settler at all). The UNCONDITIONAL backstop in every case
// is the boot sweep on the next restart; the restored entry is bounded and
// cleared on restart.
if (entry) this.active.set(runId, entry);
this.zombies.set(runId, {
workspaceId,
chatId,
intended: { status, error: err },
});
this.resolveSettled(runId, { status, error: err, terminalWriteFailed: true });
}
/**
* #487: re-drive a zombie run's intended terminal write (the conditional
* UPDATE). Called by the periodic reconcile (commit 4), an opportunistic
* single-chat reconcile, and supersede (commit 3). On success the row is now
* terminal (written OR found already terminal) the zombie is cleared and the
* once-gate armed; on another failure the zombie is kept for a later retry.
* Returns true when the row is now terminal. Best-effort; never throws.
*/
async settleZombie(runId: string): Promise<boolean> {
const z = this.zombies.get(runId);
if (!z) return false;
try {
await this.runRepo.finalizeIfActive(runId, z.workspaceId, {
status: z.intended.status,
error: z.intended.error,
});
this.zombies.delete(runId);
this.settled.add(runId);
return true;
} catch (err) {
this.logger.warn(
`Re-drive of zombie run ${runId} (chat ${z.chatId}) failed; will retry ` +
`later: ${err instanceof Error ? err.message : 'unknown error'}`,
);
return false;
}
}
/**
* #487 reconcile clause (c): abort runs the DB still shows active (pending|
* running) but that this replica does NOT own NO live entry AND NO zombie
* and that have been UNTOUCHED past `staleMs` (from last-progress `updated_at`,
* NOT startedAt, so a legit long marathon is never a candidate). "No entry" is
* the PRIMARY gate: a live entry (an actively-executing run on this replica) is
* NEVER aborted, whatever its age. Returns the number aborted. Best-effort
* never throws (a periodic-job failure must not crash the process).
*/
async reconcileStaleRuns(staleMs: number): Promise<number> {
let candidates: Array<{ id: string; workspaceId: string; chatId: string }>;
try {
candidates = await this.runRepo.findStaleActive(staleMs);
} catch (err) {
this.logger.warn(
`Reconcile (stale runs) query failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
return 0;
}
let aborted = 0;
for (const c of candidates) {
// PRIMARY gate: never touch a live entry, and never race a zombie we are
// already re-driving (settleZombie owns those).
if (this.active.has(c.id) || this.zombies.has(c.id)) continue;
try {
const row = await this.runRepo.finalizeIfActive(c.id, c.workspaceId, {
status: 'aborted',
error: 'Run aborted by reconcile: no live runner (stale).',
});
if (row) {
aborted += 1;
this.settled.add(c.id);
}
} catch (err) {
this.logger.warn(
`Reconcile abort of stale run ${c.id} failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
return aborted;
}
/**
* #487: the run's settle outcome as seen by THIS replica, or undefined when it
* has no record (the caller then reads the row the DB is the source of truth).
* A LIVE deferred (still settling, or resolved-but-not-yet-consumed) wins; a
* ZOMBIE synthesizes the give-up outcome. A subscriber (supersede) races this
* against a timeout.
*/
peekSettled(runId: string): Promise<RunSettleOutcome> | undefined {
const d = this.settledPromises.get(runId);
if (d) return d.promise;
const z = this.zombies.get(runId);
if (z) {
return Promise.resolve({
status: z.intended.status,
error: z.intended.error,
terminalWriteFailed: true,
});
}
return undefined;
}
/**
* #487: await a run's settle outcome, bounded by `timeoutMs`. Returns the
* outcome on settle, or undefined on TIMEOUT (or when this replica has no record
* of the run and its row is not terminal). Uses the LIVE settle notifier / the
* zombie synth when present; else reads the row (the DB is the source of truth
* once the in-memory record is gone). The subscriber (supersede) grabs this
* right after Stop; commit 1's race makes the settle land in ms on a healthy DB.
*/
async awaitSettled(
runId: string,
workspaceId: string,
timeoutMs: number,
): Promise<RunSettleOutcome | undefined> {
const pending = this.peekSettled(runId);
if (pending) {
let timer: ReturnType<typeof setTimeout> | undefined;
const timeout = new Promise<undefined>((resolve) => {
timer = setTimeout(() => resolve(undefined), timeoutMs);
timer.unref?.();
});
try {
return await Promise.race([pending, timeout]);
} finally {
if (timer) clearTimeout(timer);
}
}
// No live notifier and no zombie: read the row (already settled-and-written,
// or unknown here). A terminal row is an outcome; anything else -> undefined.
const row = await this.runRepo.findById(runId, workspaceId);
if (row && isRunTerminal(row.status)) {
return {
status: row.status as RunTerminalStatus,
error: row.error ?? null,
terminalWriteFailed: false,
};
}
return undefined;
}
/**
* #487: the SERVER supersede CAS for `POST /stream { supersede: { runId: X } }`.
* Atomically transitions "X is the chat's active run" -> "X is stopped, settled,
* slot free" so the caller can start a replacement run. See {@link
* SupersedeResult} for the branch semantics.
*
* On a `ready` result the caller MUST still go through the normal beginRun gate
* (the partial unique index) between the slot freeing here and beginRun a
* neighbouring tab's ordinary POST can win the slot (documented SLOT-THEFT: the
* loser then gets a MISMATCH carrying the NEW runId). There is also NO side-
* effect quiescence: an in-flight write of the stopped run may still land AFTER
* the new run starts (commit 1 stops the NEXT call, not one already committing),
* so the caller adds a prompt note to the new run.
*/
async supersede(
chatId: string,
targetRunId: string,
workspaceId: string,
timeoutMs: number = SUPERSEDE_SETTLE_TIMEOUT_MS,
): Promise<SupersedeResult> {
// Validate the target belongs to THIS chat (a CAS targeting another chat's run
// is malformed -> 400). A missing row is NOT invalid: the run may have ended
// and been pruned; the active-run check below decides degrade vs mismatch.
const target = await this.getRun(targetRunId, workspaceId);
if (target && target.chatId !== chatId) return { kind: 'invalid' };
const active = await this.getActiveForChat(chatId, workspaceId);
// No active run: it ended between the client's click and this POST — this is a
// DEGRADE to a normal send, NOT a mismatch (the user's intent still holds).
if (!active) return { kind: 'degrade' };
// A DIFFERENT run is active than the one the client saw -> mismatch. The
// client does not auto-retry; it surfaces the new runId.
if (active.id !== targetRunId) {
return { kind: 'mismatch', activeRunId: active.id };
}
// The target IS active: stop it, then await its settle within W.
await this.requestStop(targetRunId, workspaceId);
const outcome = await this.awaitSettled(targetRunId, workspaceId, timeoutMs);
if (!outcome) return { kind: 'timeout' };
// Gave up (terminal write failed): apply the intended status via the
// conditional UPDATE so the slot actually frees. If that ALSO fails, the row
// is still stranded -> treat as a timeout (nothing persisted for the new run).
if (outcome.terminalWriteFailed) {
const settled = await this.settleZombie(targetRunId);
if (!settled) return { kind: 'timeout' };
}
return { kind: 'ready' };
}
/** #487 test/diagnostic seam: whether a give-up zombie is held for this run. */
hasZombie(runId: string): boolean {
return this.zombies.has(runId);
}
/** #487: every zombie runId held on this replica (reconcile clause a, commit 4). */
zombieRunIds(): string[] {
return [...this.zombies.keys()];
}
/** #487: create a one-shot deferred (resolve captured for a later single call). */
private makeDeferred<T>(): Deferred<T> {
let resolve!: (value: T) => void;
const promise = new Promise<T>((r) => {
resolve = r;
});
return { promise, resolve };
}
/** #487: resolve a run's settle notifier EXACTLY ONCE, then drop it (bounded).
* A subscriber that already grabbed the promise still resolves; a later one
* falls back to the zombie map / the row (see peekSettled). */
private resolveSettled(runId: string, outcome: RunSettleOutcome): void {
const d = this.settledPromises.get(runId);
if (!d) return;
this.settledPromises.delete(runId);
d.resolve(outcome);
}
/** #487: read the persisted terminal outcome when the conditional finalize was a
* no-op (the row was already terminal). Falls back to the intended status when
* the read fails or the row is unexpectedly missing/non-terminal. */
private async readTerminalOutcome(
runId: string,
workspaceId: string,
fallbackStatus: RunTerminalStatus,
fallbackError: string | null,
): Promise<RunSettleOutcome> {
try {
const row = await this.runRepo.findById(runId, workspaceId);
if (row && isRunTerminal(row.status)) {
return {
status: row.status as RunTerminalStatus,
error: row.error ?? null,
terminalWriteFailed: false,
};
}
} catch {
// Fall through to the intended status — best-effort only.
}
return {
status: fallbackStatus,
error: fallbackError,
terminalWriteFailed: false,
};
}
/** Small async backoff between terminal-write retries (F6). Isolated so it is
@@ -115,7 +115,7 @@ describe('finalizeAssistant dispatch (planFinalizeAssistant + applyFinalize)', (
// Drive the SAME applyFinalize the service calls (no duplicated logic).
async function dispatchFinalize(
repo: { insert: jest.Mock; update: jest.Mock },
repo: { insert: jest.Mock; finalizeOwner: jest.Mock },
assistantId: string | undefined,
flushed: AssistantFlush,
): Promise<void> {
@@ -135,21 +135,22 @@ describe('finalizeAssistant dispatch (planFinalizeAssistant + applyFinalize)', (
expect(planFinalizeAssistant(undefined)).toEqual({ kind: 'insert' });
});
it('(a) upfront insert succeeded -> finalize UPDATEs the row by id', async () => {
const repo = { insert: jest.fn(), update: jest.fn() };
it('(a) upfront insert succeeded -> finalize CONDITIONALLY updates the row by id (#487 owner-write)', async () => {
const repo = { insert: jest.fn(), finalizeOwner: jest.fn() };
const flushed = flushAssistant([], 'final answer', 'completed', {
finishReason: 'stop',
});
await dispatchFinalize(repo, 'a1', flushed);
expect(repo.update).toHaveBeenCalledWith('a1', workspaceId, flushed);
// #487: the owner write is the CONDITIONAL finalizeOwner, not a raw update.
expect(repo.finalizeOwner).toHaveBeenCalledWith('a1', workspaceId, flushed);
expect(repo.insert).not.toHaveBeenCalled();
});
it('(b) upfront insert failed -> finalize INSERTs the terminal payload', async () => {
const repo = { insert: jest.fn(), update: jest.fn() };
const repo = { insert: jest.fn(), finalizeOwner: jest.fn() };
const flushed = flushAssistant([], 'partial', 'error', { error: 'boom' });
await dispatchFinalize(repo, undefined, flushed);
expect(repo.update).not.toHaveBeenCalled();
expect(repo.finalizeOwner).not.toHaveBeenCalled();
expect(repo.insert).toHaveBeenCalledTimes(1);
const arg = repo.insert.mock.calls[0][0];
// The fallback insert carries the terminal content/status/metadata.
@@ -0,0 +1,279 @@
import {
BadRequestException,
ConflictException,
ForbiddenException,
HttpException,
} from '@nestjs/common';
import { AiChatController } from './ai-chat.controller';
import type { User, Workspace } from '@docmost/db/types/entity.types';
/**
* #487 commit 3 the single concurrency GATE (both modes) + the server supersede
* CAS, at the controller boundary. The gate + CAS run BEFORE res.hijack(), so a
* rejected concurrent start / a CAS branch returns clean JSON (an HttpException
* the controller's post-hijack catch re-serializes). These assert the OBSERVABLE
* HTTP contract against the real controller + a stubbed run service.
*/
describe('#487 AiChatController.stream — gate + supersede', () => {
const user = { id: 'u1' } as User;
function wsWith(autonomousRuns: boolean): Workspace {
return {
id: 'ws1',
settings: { ai: { chat: true, autonomousRuns } },
} as unknown as Workspace;
}
function makeReqRes(body: Record<string, unknown>) {
const req = {
raw: { sessionId: 'sess', once: jest.fn(), destroyed: false },
body,
};
const res = {
raw: {
writableEnded: false,
headersSent: false,
on: jest.fn(),
once: jest.fn(),
setHeader: jest.fn(),
end: jest.fn(),
statusCode: 200,
flushHeaders: jest.fn(),
},
hijack: jest.fn(),
status: jest.fn().mockReturnThis(),
send: jest.fn(),
};
return { req, res };
}
function makeController(
runServiceOverrides: Record<string, jest.Mock>,
// The chat assertOwnedChat resolves. Default: a chat OWNED by `user` (u1), so
// the ownership gate is transparent to the gate/CAS assertions below. Pass a
// foreign-owner (or undefined) chat to exercise the #487 owner rejection.
chat: { creatorId: string } | undefined = { creatorId: 'u1' },
) {
const aiChatService = {
resolveRoleForRequest: jest.fn().mockResolvedValue(null),
getChatModel: jest.fn().mockResolvedValue({}),
stream: jest.fn().mockResolvedValue(undefined),
};
const aiChatRunService = {
getActiveForChat: jest.fn().mockResolvedValue(undefined),
supersede: jest.fn(),
beginRun: jest.fn().mockResolvedValue({
runId: 'run-new',
signal: new AbortController().signal,
}),
linkAssistantMessage: jest.fn(),
recordStep: jest.fn(),
finalizeRun: jest.fn(),
requestStop: jest.fn(),
...runServiceOverrides,
};
const aiChatRepo = { findById: jest.fn().mockResolvedValue(chat) };
const controller = new AiChatController(
aiChatService as never,
aiChatRunService as never,
aiChatRepo as never, // aiChatRepo
{} as never, // aiChatMessageRepo
{} as never, // aiTranscription
{} as never, // pageRepo
);
return { controller, aiChatService, aiChatRunService, aiChatRepo };
}
const codeOf = (err: unknown) =>
(((err as HttpException).getResponse() as Record<string, unknown>) ?? {})
.code;
describe('single concurrency gate — BOTH modes reject the second tab with 409', () => {
for (const autonomousRuns of [true, false]) {
it(`rejects a concurrent start with 409 A_RUN_ALREADY_ACTIVE (autonomousRuns=${autonomousRuns})`, async () => {
const { controller, aiChatRunService } = makeController({
getActiveForChat: jest
.fn()
.mockResolvedValue({ id: 'run-live', chatId: 'c1' }),
});
const { req, res } = makeReqRes({ chatId: 'c1' });
let thrown: unknown;
try {
await controller.stream(
req as never,
res as never,
user,
wsWith(autonomousRuns),
);
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect((thrown as HttpException).getStatus()).toBe(409);
expect(codeOf(thrown)).toBe('A_RUN_ALREADY_ACTIVE');
// Rejected BEFORE committing to the stream (no hijack, no service.stream).
expect(res.hijack).not.toHaveBeenCalled();
expect(aiChatRunService.getActiveForChat).toHaveBeenCalledWith(
'c1',
'ws1',
);
});
}
});
// #487 [security, F1]: stream() MUST owner-gate an existing chat exactly like its
// six sibling endpoints, BEFORE the supersede CAS. Otherwise a same-workspace
// non-owner could POST a supersede against another user's chat and (a) harvest
// that user's active runId from the 409 SUPERSEDE_TARGET_MISMATCH body, then (b)
// requestStop the foreign run. The gate must reject FIRST — no run lookup, no
// supersede, no stop, no runId leak.
describe('cross-user ownership gate (F1)', () => {
it('a non-owner streaming against someone else\'s chat is rejected (403) with NO runId leak and NO foreign requestStop', async () => {
// A live run exists on the victim's chat. Without the gate the supersede CAS
// would run and (faithful to the run service) return a MISMATCH carrying the
// victim's runId — the exact leak. With the gate it must never be reached.
const getActiveForChat = jest
.fn()
.mockResolvedValue({ id: 'run-victim', chatId: 'c-other' });
const supersede = jest
.fn()
.mockResolvedValue({ kind: 'mismatch', activeRunId: 'run-victim' });
const requestStop = jest.fn();
const { controller, aiChatService } = makeController(
{ getActiveForChat, supersede, requestStop },
{ creatorId: 'someone-else' }, // the chat is NOT owned by u1
);
const { req, res } = makeReqRes({
chatId: 'c-other',
supersede: { runId: 'guessed-uuid' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
// Rejected by the ownership gate (403), the SAME shape the neighbors use.
expect(thrown).toBeInstanceOf(ForbiddenException);
expect((thrown as HttpException).getStatus()).toBe(403);
// Crucially NOT a 409 that would carry activeRunId — no runId is leaked.
const payload = JSON.stringify(
(thrown as HttpException).getResponse() ?? {},
);
expect(payload).not.toContain('run-victim');
expect(codeOf(thrown)).not.toBe('SUPERSEDE_TARGET_MISMATCH');
// The gate short-circuits BEFORE any run machinery runs.
expect(getActiveForChat).not.toHaveBeenCalled();
expect(supersede).not.toHaveBeenCalled();
expect(requestStop).not.toHaveBeenCalled();
expect(aiChatService.stream).not.toHaveBeenCalled();
expect(res.hijack).not.toHaveBeenCalled();
});
});
it('supersede MISMATCH -> 409 SUPERSEDE_TARGET_MISMATCH carrying the current runId', async () => {
const { controller } = makeController({
supersede: jest
.fn()
.mockResolvedValue({ kind: 'mismatch', activeRunId: 'run-other' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect(codeOf(thrown)).toBe('SUPERSEDE_TARGET_MISMATCH');
expect(
((thrown as HttpException).getResponse() as Record<string, unknown>)
.activeRunId,
).toBe('run-other');
expect(res.hijack).not.toHaveBeenCalled();
});
it('supersede TIMEOUT -> 409 SUPERSEDE_TIMEOUT, nothing streamed', async () => {
const { controller } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'timeout' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(false));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(ConflictException);
expect(codeOf(thrown)).toBe('SUPERSEDE_TIMEOUT');
expect(res.hijack).not.toHaveBeenCalled();
});
it('supersede INVALID (target on another chat) -> 400 SUPERSEDE_INVALID', async () => {
const { controller } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'invalid' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(BadRequestException);
expect(codeOf(thrown)).toBe('SUPERSEDE_INVALID');
});
it('supersede without chatId -> 400 SUPERSEDE_INVALID', async () => {
const { controller, aiChatRunService } = makeController({});
const { req, res } = makeReqRes({ supersede: { runId: 'run-x' } });
let thrown: unknown;
try {
await controller.stream(req as never, res as never, user, wsWith(true));
} catch (e) {
thrown = e;
}
expect(thrown).toBeInstanceOf(BadRequestException);
expect(codeOf(thrown)).toBe('SUPERSEDE_INVALID');
expect(aiChatRunService.supersede).not.toHaveBeenCalled();
});
it('supersede READY -> proceeds to stream with superseded=true', async () => {
const { controller, aiChatService } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'ready' }),
getActiveForChat: jest.fn().mockResolvedValue(undefined), // slot free after CAS
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
await controller.stream(req as never, res as never, user, wsWith(true));
expect(res.hijack).toHaveBeenCalled();
expect(aiChatService.stream).toHaveBeenCalledTimes(1);
expect(aiChatService.stream.mock.calls[0][0].superseded).toBe(true);
// The run hooks are always present now (both modes).
expect(aiChatService.stream.mock.calls[0][0].runHooks).toBeDefined();
});
it('supersede DEGRADE -> proceeds to a normal send (superseded=false)', async () => {
const { controller, aiChatService } = makeController({
supersede: jest.fn().mockResolvedValue({ kind: 'degrade' }),
});
const { req, res } = makeReqRes({
chatId: 'c1',
supersede: { runId: 'run-x' },
});
await controller.stream(req as never, res as never, user, wsWith(false));
expect(aiChatService.stream).toHaveBeenCalledTimes(1);
expect(aiChatService.stream.mock.calls[0][0].superseded).toBe(false);
});
});
@@ -418,6 +418,19 @@ export class AiChatController {
const body = (req.body ?? {}) as AiChatStreamBody;
// #487 [security]: gate cross-user access to an EXISTING chat BEFORE anything
// reads its runs. Every sibling endpoint (getRun/stop/history/rename/delete/
// attachRunStream) owner-checks the chat via assertOwnedChat; stream() must too.
// Without this a same-workspace member who is NOT the chat owner could POST a
// supersede against another user's chat and (a) harvest that user's active runId
// out of the 409 SUPERSEDE_TARGET_MISMATCH body, then (b) requestStop the foreign
// run. Gate on the chatId the client sent, when present — a brand-new chat (no
// chatId) has no prior owner to check. Mirrors /stop's owner check (403 as the
// neighbors do), and runs pre-hijack so it returns clean JSON.
if (body.chatId) {
await this.assertOwnedChat(body.chatId, user, workspace);
}
// Resolve the agent role for this turn BEFORE hijack: existing chats read it
// from ai_chats.role_id (authoritative), a new chat from body.roleId. The
// role drives both the persona and the optional model override below.
@@ -432,12 +445,66 @@ export class AiChatController {
// HttpException) instead of breaking mid-stream.
const model = await this.aiChatService.getChatModel(workspace.id, role);
// #184: one active run per chat. For an EXISTING chat reject a concurrent
// start with a clean 409 BEFORE hijack (the common double-submit / second-tab
// case), so the user gets JSON, not a mid-stream error. A brand-new chat
// (no chatId) cannot have a prior run, and the DB partial unique index is the
// backstop against any race that slips past this check.
if (autonomousRuns && body.chatId) {
// #487: server-side supersede CAS ("interrupt and send now"). When the client
// asks to replace a live run, atomically STOP it and wait for it to settle
// before this turn claims the slot. Runs BEFORE hijack so every branch returns
// clean JSON (the client keeps the composer text on a 409). See
// AiChatRunService.supersede for the branch semantics.
let superseded = false;
const supersedeRunId = body.supersede?.runId;
if (supersedeRunId) {
if (!body.chatId) {
throw new BadRequestException({
message: 'supersede requires chatId',
code: 'SUPERSEDE_INVALID',
});
}
const result = await this.aiChatRunService.supersede(
body.chatId,
supersedeRunId,
workspace.id,
);
switch (result.kind) {
case 'invalid':
throw new BadRequestException({
message: 'The run to supersede does not belong to this chat',
code: 'SUPERSEDE_INVALID',
});
case 'mismatch':
// A DIFFERENT run is active than the one the client targeted. Surface
// the CURRENT runId; the client does NOT auto-retry (a stale CAS).
throw new ConflictException({
message: 'A different agent run is now active on this chat',
code: 'SUPERSEDE_TARGET_MISMATCH',
activeRunId: result.activeRunId,
});
case 'timeout':
// The target did not settle within W — nothing was persisted, the
// composer keeps the text. NOT a rollback: the stop is already issued.
throw new ConflictException({
message:
'The previous run did not stop in time; nothing was sent — please try again',
code: 'SUPERSEDE_TIMEOUT',
});
case 'ready':
// The target stopped and settled: the slot is free. Prompt the new run
// that the old run's last operations may still be applying.
superseded = true;
break;
case 'degrade':
// The run already ended between click and POST — send normally.
break;
}
}
// #487: one active run per chat — ENFORCED IN BOTH MODES now (legacy mode used
// to have NO gate, so two tabs streamed two parallel turns on one chat, which
// interleaved history and crashed convertToModelMessages). Reject a concurrent
// start with a clean pre-hijack 409 (double-submit / second-tab). A brand-new
// chat (no chatId) cannot have a prior run, and the DB partial unique index in
// beginRun is the authoritative backstop for any race that slips past here
// (including a slot stolen between a supersede release and beginRun).
if (body.chatId) {
const active = await this.aiChatRunService.getActiveForChat(
body.chatId,
workspace.id,
@@ -446,107 +513,94 @@ export class AiChatController {
throw new ConflictException({
message: 'An agent run is already in progress for this chat',
code: 'A_RUN_ALREADY_ACTIVE',
activeRunId: active.id,
});
}
}
// Run-lifecycle hooks (#184), only when the flag is on. They wrap the turn in
// a durable run whose abort is governed by the run (explicit stop), persist
// its progress, and settle its terminal status — see AiChatRunService.
const runHooks: AiChatRunHooks | undefined = autonomousRuns
? {
begin: async (chatId) => {
const handle = await this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
});
// #184 phase 1.5: register the run-stream entry at BEGIN (before any
// frame) so a tab that attaches in the begin->seed window finds an
// entry to wait on. Gated on AI_CHAT_RESUMABLE_STREAM: with the flag
// off nothing is registered and attach always 204s.
if (
handle?.runId &&
this.environment?.isAiChatResumableStreamEnabled?.()
) {
this.streamRegistry?.open(chatId, handle.runId);
}
return handle;
},
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
workspace.id,
messageId,
),
onStep: (runId, stepCount) =>
void this.aiChatRunService.recordStep(
runId,
workspace.id,
stepCount,
),
onSettled: (runId, status, error) =>
this.aiChatRunService.finalizeRun(
runId,
workspace.id,
status,
error,
),
// #487: the turn is ALWAYS a first-class RUN now (both modes). The mode
// difference is only the abort semantics on a browser disconnect (onClose
// below). currentRunId is captured at begin so a legacy disconnect can stop
// the run through its stop lever.
let currentRunId: string | undefined;
const runHooks: AiChatRunHooks = {
begin: async (chatId) => {
const handle = await this.aiChatRunService.beginRun({
chatId,
workspaceId: workspace.id,
userId: user.id,
trigger: 'user',
});
currentRunId = handle?.runId;
// #184 phase 1.5: register the run-stream entry at BEGIN (before any
// frame) so a tab that attaches in the begin->seed window finds an entry
// to wait on. Gated on AI_CHAT_RESUMABLE_STREAM.
if (
handle?.runId &&
this.environment?.isAiChatResumableStreamEnabled?.()
) {
this.streamRegistry?.open(chatId, handle.runId);
}
: undefined;
return handle;
},
onAssistantSeeded: (runId, messageId) =>
this.aiChatRunService.linkAssistantMessage(
runId,
workspace.id,
messageId,
),
onStep: (runId, stepCount) =>
void this.aiChatRunService.recordStep(runId, workspace.id, stepCount),
onSettled: (runId, status, error) =>
this.aiChatRunService.finalizeRun(runId, workspace.id, status, error),
};
// Abort the agent loop when the client disconnects. `close` also fires on
// normal completion, so only abort when the response has not finished
// writing (a genuine disconnect). `once` fires at most once and self-removes;
// we also drop it on response `finish` so it never lingers after the stream
// completes normally (the AI SDK pipes the response fire-and-forget, so we
// cannot simply remove it once `stream()` returns).
// Handle a client disconnect. `close` also fires on normal completion, so only
// act when the response has not finished writing (a genuine disconnect). `once`
// fires at most once and self-removes; we also drop it on response `finish`.
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: wall-clock at
// which a Safari disconnect is observed, measured from request receipt.
const reqStartedAt = Date.now();
const controller = new AbortController();
const onClose = (): void => {
// A genuine disconnect leaves the response unfinished (unlike a normal
// completion, which also fires `close`). Such a drop — e.g. a reverse
// proxy cutting the SSE mid-answer — is otherwise invisible server-side,
// so log it here.
if (!res.raw.writableEnded) {
if (autonomousRuns) {
// #184: the turn is a DETACHED run. A disconnect must NOT abort it —
// the run keeps executing and persisting server-side; the client
// reconnects via /ai-chat/run (or re-stops via /ai-chat/stop). Log only.
// #184: a DETACHED run — a disconnect must NOT stop it. The run keeps
// executing and persisting server-side; the client reconnects via
// /ai-chat/run (or re-stops via /ai-chat/stop). Log only.
this.logger.log(
`AI chat stream: client disconnected; run continues server-side ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
} else {
// #487: legacy — a disconnect ENDS the turn, but the turn is now a RUN,
// so stop it through the run's stop lever (requestStop). streamText no
// longer consumes the socket signal (effectiveSignal is the run signal),
// so aborting `controller` would do nothing; requestStop aborts the run.
this.logger.warn(
`AI chat stream: client disconnected before completion; aborting turn ` +
`(elapsed=${Date.now() - reqStartedAt}ms since request received)`,
`AI chat stream: client disconnected before completion; stopping the ` +
`run (elapsed=${Date.now() - reqStartedAt}ms since request received)`,
);
controller.abort();
if (currentRunId) {
void this.aiChatRunService.requestStop(currentRunId, workspace.id);
}
}
}
};
req.raw.once('close', onClose);
res.raw.once('finish', () => req.raw.off('close', onClose));
// #184: in detached mode the turn is NOT aborted on disconnect, so the SDK's
// pipe keeps writing to a socket the client may have dropped — for the rest of
// the (continuing) run. A write to the dead socket can emit an 'error' on the
// raw response; without a listener that surfaces as an unhandled error event.
// Swallow it (the run continues server-side regardless). Legacy mode aborts on
// disconnect, so it does not need this and keeps its exact prior behavior.
if (autonomousRuns) {
res.raw.on('error', (err) => {
this.logger.debug(
`AI chat detached stream: post-disconnect socket error swallowed: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
}
// #184/#487: the run/pipe can outlive the socket in BOTH modes now (autonomous
// keeps going; legacy keeps going until requestStop's abort unwinds the turn).
// The SDK's pipe may then write to a dropped socket and emit an 'error' on the
// raw response — swallow it so it never surfaces as an unhandled error event.
res.raw.on('error', (err) => {
this.logger.debug(
`AI chat stream: post-disconnect socket error swallowed: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
// Commit to streaming: hijack so Fastify stops managing the response and
// the AI SDK can write the UI-message stream directly to the Node socket.
@@ -562,8 +616,10 @@ export class AiChatController {
signal: controller.signal,
model,
role,
// #184: present only when the flag is on; wraps the turn in a durable run.
// #487: the turn is always run-wrapped now (both modes).
runHooks,
// #487: warn the new run that a superseded run's last ops may still apply.
superseded,
});
} catch (err) {
// Any failure AFTER hijack can no longer go through Nest's exception
@@ -0,0 +1,142 @@
// #489 — client-parts validation + resilient history conversion.
//
// These unit tests exercise the two exported helpers against the REAL
// `convertToModelMessages` from `ai` (NOT a mock): a genuinely malformed part
// (a `null` element inside a parts array) makes the real converter throw
// ("Cannot read properties of null"), which is the actual production
// "bricked chat" mechanism this fix defends against. Asserting against the real
// converter (rather than a mock-shaped error) is the whole point — a mock would
// hide a version change in the converter's throw behaviour.
import { convertToModelMessages, type UIMessage } from 'ai';
import {
sanitizeUserParts,
convertHistoryResilient,
TOOL_CONTEXT_OMITTED_MARKER,
} from './ai-chat.service';
type Row = Omit<UIMessage, 'id'> & { id: string };
describe('sanitizeUserParts (#489, branch: validation on receipt)', () => {
it('keeps whitelisted text parts unchanged', () => {
const drops: string[] = [];
const out = sanitizeUserParts(
[
{ type: 'text', text: 'a' },
{ type: 'text', text: 'b' },
] as UIMessage['parts'],
(t) => drops.push(t),
);
expect(out).toEqual([
{ type: 'text', text: 'a' },
{ type: 'text', text: 'b' },
]);
expect(drops).toEqual([]);
});
it('drops a non-text part (a tool-part in input-available) and reports its type', () => {
const drops: string[] = [];
const out = sanitizeUserParts(
[
{ type: 'text', text: 'hi' },
{
type: 'tool-getPage',
toolCallId: 't1',
state: 'input-available',
input: { pageId: 'p' },
},
] as unknown as UIMessage['parts'],
(t) => drops.push(t),
);
expect(out).toEqual([{ type: 'text', text: 'hi' }]);
expect(drops).toEqual(['tool-getPage']);
});
it('drops a null part (the shape that would poison convertToModelMessages)', () => {
const drops: string[] = [];
const out = sanitizeUserParts(
[{ type: 'text', text: 'hi' }, null] as unknown as UIMessage['parts'],
(t) => drops.push(t),
);
expect(out).toEqual([{ type: 'text', text: 'hi' }]);
expect(drops).toEqual(['(unknown)']);
});
it('returns undefined when nothing survives (so a null metadata is persisted)', () => {
const out = sanitizeUserParts(
[
{ type: 'tool-x', toolCallId: 't', state: 'input-available' },
] as unknown as UIMessage['parts'],
() => undefined,
);
expect(out).toBeUndefined();
});
it('returns undefined for a non-array input', () => {
expect(
sanitizeUserParts(undefined as unknown as UIMessage['parts'], () => undefined),
).toBeUndefined();
});
});
describe('convertHistoryResilient (#489, branches: happy + per-row degradation)', () => {
it('happy path: healthy history converts identically to convertToModelMessages, no degrade', async () => {
const history: Row[] = [
{ id: 'u1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
{ id: 'a1', role: 'assistant', parts: [{ type: 'text', text: 'hello' }] },
];
const degrades: number[] = [];
const out = await convertHistoryResilient(history, (i) => degrades.push(i));
const expected = await convertToModelMessages(history as UIMessage[]);
expect(out).toEqual(expected);
expect(degrades).toEqual([]);
});
it('REAL poison: a null part throws in the batch converter but is isolated and degraded to a marker', async () => {
// Sanity: the real converter genuinely throws on this shape.
const poisoned: Row = {
id: 'a1',
role: 'assistant',
parts: [
{ type: 'text', text: 'earlier answer' },
null,
] as unknown as UIMessage['parts'],
};
await expect(
convertToModelMessages([poisoned as UIMessage]),
).rejects.toThrow();
const history: Row[] = [
{ id: 'u1', role: 'user', parts: [{ type: 'text', text: 'first' }] },
poisoned,
{ id: 'u2', role: 'user', parts: [{ type: 'text', text: 'second' }] },
];
const degrades: number[] = [];
const out = await convertHistoryResilient(history, (i) => degrades.push(i));
// Only the poisoned row (index 1) is degraded.
expect(degrades).toEqual([1]);
// Healthy rows survive verbatim.
const flat = JSON.stringify(out);
expect(flat).toContain('first');
expect(flat).toContain('second');
// The degraded row carries its readable text AND the truncation marker so the
// model sees that tool context was omitted (never a silent loss).
expect(flat).toContain('earlier answer');
expect(flat).toContain(TOOL_CONTEXT_OMITTED_MARKER);
// The whole batch converted (3 model messages, none dropped).
expect(out).toHaveLength(3);
});
it('a fully-poisoned row (no readable text) still degrades to just the marker', async () => {
const history: Row[] = [
{
id: 'a1',
role: 'assistant',
parts: [null] as unknown as UIMessage['parts'],
},
];
const out = await convertHistoryResilient(history, () => undefined);
expect(out).toHaveLength(1);
expect(JSON.stringify(out)).toContain(TOOL_CONTEXT_OMITTED_MARKER);
});
});
@@ -101,6 +101,22 @@ const INTERRUPT_NOTE =
'assume your previous response was complete, and do not silently restart the ' +
'partial work — build on it or follow the new instruction.';
/**
* #487: injected on a turn started by SUPERSEDING a previous run (the user hit
* "interrupt and send now" while a run was live). The previous run was Stopped,
* but there is NO side-effect quiescence a write it had already committed, or
* one committing at the moment of Stop, may land with a small delay AFTER this new
* run starts. So the model is told its picture of the page/state may be a beat
* stale and to re-read before assuming an edit did or did not apply.
*/
const SUPERSEDE_NOTE =
'NOTE: A previous agent run in this conversation was just interrupted so this ' +
'new turn could start. That run was stopped, but any operation it had already ' +
'begun (e.g. a page edit) may still be applied with a short delay. Do not ' +
'assume the document/state is exactly as the interrupted run left it — if you ' +
'need to rely on the current content, RE-READ it with the page tools before ' +
'acting rather than trusting a cached view.';
/**
* Injected on a turn where the open page was hand-edited by the user (or anyone
* else) AFTER the agent's previous response ended (#274). The server takes a
@@ -203,6 +219,14 @@ export interface BuildSystemPromptInput {
* (partial) answer was cut off by the user's new message.
*/
interrupted?: boolean;
/**
* #487: true when THIS turn was started by superseding a still-live previous run
* ("interrupt and send now"). Adds SUPERSEDE_NOTE so the model knows the previous
* run's last operations may still be applying and to re-read state it depends on.
* Distinct from `interrupted` (which is about a PARTIAL prior answer in history);
* both can be set together. Self-clears set only for the superseding turn.
*/
superseded?: boolean;
/**
* Set only when the open page was edited by the user AFTER the agent's previous
* turn ended (#274), confirmed server-side by diffing the current page against
@@ -311,6 +335,7 @@ export function buildSystemPrompt({
openedPage,
mcpInstructions,
interrupted,
superseded,
pageChanged,
deferredToolsEnabled,
toolCatalog,
@@ -360,6 +385,13 @@ export function buildSystemPrompt({
context += `\n${INTERRUPT_NOTE}`;
}
// Supersede note (#487): present only for a turn that stopped and replaced a
// still-live previous run — warns the model the previous run's last operations
// may still be applying (no side-effect quiescence).
if (superseded) {
context += `\n${SUPERSEDE_NOTE}`;
}
// Per-turn page-change note (#274). Added to the context section (inside the
// safety sandwich), present only when the server detected that the open page
// was edited by the user since the agent's last turn ended. The diff content is
@@ -89,11 +89,22 @@ describe('AiChatService.stream run-lifecycle safety net (#184)', () => {
const runRepo = {
insert: jest.fn().mockResolvedValue({ id: 'run-1', status: 'running' }),
update: jest.fn().mockResolvedValue({ id: 'run-1' }),
// #487: the terminal settle now goes through the CONDITIONAL write.
finalizeIfActive: jest
.fn()
.mockResolvedValue({ id: 'run-1', status: 'failed' }),
findById: jest.fn().mockResolvedValue(undefined),
};
const runService = new AiChatRunService(runRepo as never, { isCloud: () => false } as never);
// The user-message insert (the first bare await after beginRun) throws.
// The user-message insert throws. #489 runs the history load + convert BEFORE
// the insert (convert-before-insert, so a retry cannot duplicate the user row),
// so `findAllByChat` (a real repo method) is now called first — stub it to an
// empty history so the flow reaches the insert. Both awaits are AFTER beginRun,
// so the "exception after beginRun -> settled to error" invariant is unchanged;
// the throw point simply moved from insert to a later insert after a no-op load.
const aiChatMessageRepo = {
findAllByChat: jest.fn().mockResolvedValue([]),
insert: jest.fn().mockRejectedValue(new Error('insert boom')),
};
const aiChatRepo = {
@@ -148,9 +159,10 @@ describe('AiChatService.stream run-lifecycle safety net (#184)', () => {
// The run was begun...
expect(runRepo.insert).toHaveBeenCalledTimes(1);
// ...then settled to a terminal FAILED status by the safety net...
expect(runRepo.update).toHaveBeenCalledTimes(1);
expect(runRepo.update).toHaveBeenCalledWith(
// ...then settled to a terminal FAILED status by the safety net (via the
// #487 conditional write)...
expect(runRepo.finalizeIfActive).toHaveBeenCalledTimes(1);
expect(runRepo.finalizeIfActive).toHaveBeenCalledWith(
'run-1',
'ws1',
expect.objectContaining({ status: 'failed' }),
@@ -155,6 +155,8 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -332,7 +334,13 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
usage: {},
steps: [],
});
expect(runHooks.onSettled).toHaveBeenCalledWith('run-1', 'completed');
// #487: onFinish passes the (undefined) error slot so a message-finalize
// failure could error-mark the run; on the success path it is undefined.
expect(runHooks.onSettled).toHaveBeenCalledWith(
'run-1',
'completed',
undefined,
);
});
it('F9: onAbort settles the run "aborted"', async () => {
@@ -415,6 +423,8 @@ describe('AiChatService.stream — begin-failure fails the turn (#184 F14 / #486
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -1315,8 +1315,12 @@ describe('AiChatService page-change lifecycle (#274)', () => {
describe('isInterruptResume', () => {
// history tail is the just-inserted user row; [len-2] is the previous turn.
const withPrev = (
prev: { role: string; status?: string | null } | null,
): Array<{ role: string; status?: string | null }> =>
prev: {
role: string;
status?: string | null;
metadata?: unknown;
} | null,
): Array<{ role: string; status?: string | null; metadata?: unknown }> =>
prev
? [prev, { role: 'user', status: null }]
: [{ role: 'user', status: null }];
@@ -1357,6 +1361,33 @@ describe('isInterruptResume', () => {
it('false when there is no preceding turn (only the new user row)', () => {
expect(isInterruptResume(withPrev(null), true)).toBe(false);
});
it('#487 EXCLUDES a reconcile stamp (finalizeFailed) — not a genuine interruption', () => {
// A row a reconcile settled to 'aborted' carries metadata.finalizeFailed. It
// must NOT be treated as an interrupt-resume (that would inject a false
// "you were interrupted" note), even though its status is 'aborted'.
expect(
isInterruptResume(
withPrev({
role: 'assistant',
status: 'aborted',
metadata: { finalizeFailed: true },
}),
true,
),
).toBe(false);
// A genuine abort (no finalizeFailed) still counts.
expect(
isInterruptResume(
withPrev({
role: 'assistant',
status: 'aborted',
metadata: { parts: [] },
}),
true,
),
).toBe(true);
});
});
/**
@@ -1409,7 +1440,7 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
}
// Wire only the deps reached on the way to the pipe call, plus a spy registry.
function makeService(opts: { resumable: boolean }) {
function makeService(opts: { resumable: boolean; history?: unknown[] }) {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
@@ -1417,8 +1448,11 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
const aiChatMessageRepo = {
// Both the user insert and the assistant seed return the same row id.
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
findAllByChat: jest.fn(async () => opts.history ?? []),
update: jest.fn(async () => ({ id: 'msg-1' })),
// #487: the terminal owner-write + the opportunistic reconcile query.
finalizeOwner: jest.fn(async () => ({ id: 'msg-1' })),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -1453,7 +1487,7 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
} as never,
streamRegistry as never,
);
return { svc, streamRegistry };
return { svc, streamRegistry, aiChatMessageRepo };
}
const body = {
@@ -1536,6 +1570,86 @@ describe('AiChatService.stream — resumable pipe options (#184 phase 1.5)', ()
await expect(drive(svc, makeRunHooks())).rejects.toThrow('boom');
expect(streamRegistry.abortEntry).toHaveBeenCalledWith('chat-1', 'run-1');
});
// #489 REGRESSION (against the REAL convertToModelMessages — not mocked here):
// a persisted history row whose parts contain a `null` element makes the real
// convertToModelMessages THROW ("Cannot read properties of null"). Pre-fix that
// 500-ed every turn forever and each retry appended a duplicate user row. The
// fix converts BEFORE the insert and isolates the poisoned row per-row, degrading
// it to text with a "[tool context omitted]" marker. Assert the turn still runs,
// the marker reaches the model, and exactly ONE user row is inserted.
it('#489: a poisoned OLD-history row keeps the chat working; the marker reaches the model; one user insert', async () => {
const { svc, aiChatMessageRepo } = makeService({
resumable: false,
history: [
{
id: 'old-1',
role: 'assistant',
content: 'earlier answer',
// A null part is the poison: rowToUiMessage keeps it (the array is
// non-empty) and the real convertToModelMessages throws on it.
metadata: { parts: [{ type: 'text', text: 'earlier answer' }, null] },
status: 'completed',
},
],
});
// Must NOT throw — the poisoned row is degraded, not fatal.
await drive(svc, makeRunHooks());
expect(streamTextMock).toHaveBeenCalledTimes(1);
const passedMessages = streamTextMock.mock.calls[0][0].messages;
const serialized = JSON.stringify(passedMessages);
// The model sees the truncation marker (silent tool-context loss is not ok)
// AND the row's readable text is preserved alongside it.
expect(serialized).toContain('[tool context omitted]');
expect(serialized).toContain('earlier answer');
// Exactly ONE user row inserted (no duplicate), inserted AFTER conversion.
const userInserts = aiChatMessageRepo.insert.mock.calls
.map((c: unknown[]) => c[0] as { role?: string })
.filter((r) => r.role === 'user');
expect(userInserts).toHaveLength(1);
});
// #489: client-supplied non-text parts (a tool-part in `input-available`, the
// exact "bricking" payload) are dropped ON RECEIPT — never persisted — so they
// can never poison future turns. Only the text survives into metadata.parts.
it('#489: a non-text client part is stripped before persist (only text survives)', async () => {
const { svc, aiChatMessageRepo } = makeService({ resumable: false });
await svc.stream({
user: { id: 'u1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 's1',
body: {
chatId: 'chat-1',
messages: [
{
id: 'm1',
role: 'user',
parts: [
{ type: 'text', text: 'hello' },
// untrusted tool-part — must be dropped, never persisted
{
type: 'tool-getPage',
toolCallId: 't1',
state: 'input-available',
input: { pageId: 'p' },
},
],
},
],
} as never,
res: makeRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
runHooks: makeRunHooks() as never,
});
const userInsert = aiChatMessageRepo.insert.mock.calls
.map((c: unknown[]) => c[0] as { role?: string; metadata?: unknown })
.find((r) => r.role === 'user');
const parts = (userInsert?.metadata as { parts?: Array<{ type: string }> })
?.parts;
expect(parts).toEqual([{ type: 'text', text: 'hello' }]);
});
});
/**
@@ -1623,6 +1737,19 @@ describe('AiChatService.stream — token-degeneration reaction (#444)', () => {
return { id };
},
),
// #487: the terminal owner-write records into the SAME `updated` recorder so
// assertions on the terminal 'completed'/'error'/'aborted' write still hold.
finalizeOwner: jest.fn(
async (
id: string,
workspaceId: string,
patch: Record<string, unknown>,
) => {
updated.push({ id, workspaceId, patch });
return { id };
},
),
findStreamingWithTerminalRun: jest.fn(async () => []),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
@@ -1882,3 +2009,148 @@ describe('AiChatService.stream — token-degeneration reaction (#444)', () => {
expect(patch.content).not.toContain(STEP_LIMIT_NO_ANSWER_MARKER);
});
});
// #487 F3 — the reconcile() / reconcileChat() ORCHESTRATORS. The individual
// clauses are exercised elsewhere; these pin the production orchestration the
// per-clause specs do not: the clause ORDER, the per-clause try/catch ISOLATION
// (one clause throwing must NOT abort the others), and reconcileChat() (which runs
// at the start of every turn and was entirely uncovered).
describe('AiChatService.reconcile / reconcileChat orchestrators (#487 F3)', () => {
let warnSpy: jest.SpyInstance;
beforeEach(() => {
// Silence the intentional clause-failure warnings (kept out of test output).
warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined);
});
afterEach(() => {
warnSpy.mockRestore();
});
function makeService(opts: {
messageRepo?: Record<string, jest.Mock>;
runService?: Record<string, jest.Mock>;
}) {
const aiChatMessageRepo = {
findStreamingWithTerminalRun: jest.fn(async () => []),
stampTerminalIfStreaming: jest.fn(async () => undefined),
sweepStreamingWithoutActiveRun: jest.fn(async () => 0),
...(opts.messageRepo ?? {}),
};
const aiChatRunService = opts.runService
? {
zombieRunIds: jest.fn(() => []),
settleZombie: jest.fn(async () => true),
reconcileStaleRuns: jest.fn(async () => 0),
...opts.runService,
}
: undefined;
const svc = new AiChatService(
{} as never, // ai
{} as never, // aiChatRepo
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
{} as never, // aiSettings
{} as never, // tools
{} as never, // mcpClients
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{} as never, // environment
{} as never, // streamRegistry
aiChatRunService as never, // aiChatRunService (#487)
);
return { svc, aiChatMessageRepo, aiChatRunService };
}
it('reconcile() fires all four clauses IN ORDER (a -> b -> c -> d)', async () => {
const order: string[] = [];
const { svc } = makeService({
messageRepo: {
findStreamingWithTerminalRun: jest.fn(async () => {
order.push('b:find');
return [
{ messageId: 'm1', workspaceId: 'ws1', runStatus: 'succeeded' },
];
}),
stampTerminalIfStreaming: jest.fn(async () => {
order.push('b:stamp');
}),
sweepStreamingWithoutActiveRun: jest.fn(async () => {
order.push('d');
return 0;
}),
},
runService: {
zombieRunIds: jest.fn(() => ['z1']),
settleZombie: jest.fn(async () => {
order.push('a');
return true;
}),
reconcileStaleRuns: jest.fn(async () => {
order.push('c');
return 0;
}),
},
});
await svc.reconcile();
expect(order).toEqual(['a', 'b:find', 'b:stamp', 'c', 'd']);
});
it('a clause that THROWS does not abort the remaining clauses (per-clause try/catch isolation)', async () => {
const { svc, aiChatMessageRepo, aiChatRunService } = makeService({
messageRepo: {
// Clause (b) blows up mid-reconcile.
findStreamingWithTerminalRun: jest.fn(async () => {
throw new Error('clause b DB blip');
}),
},
runService: {
zombieRunIds: jest.fn(() => ['z1']),
},
});
// reconcile() must SETTLE (the clause-b failure is swallowed), not reject.
await expect(svc.reconcile()).resolves.toBeUndefined();
// (a) ran before (b); crucially (c) and (d) STILL ran despite (b) throwing —
// the property a missing try/catch would break. MUTATION-VERIFY: drop clause
// (b)'s try/catch and this reddens (the throw propagates, skipping c + d).
expect(aiChatRunService!.settleZombie).toHaveBeenCalled(); // (a)
expect(aiChatRunService!.reconcileStaleRuns).toHaveBeenCalled(); // (c)
expect(
aiChatMessageRepo.sweepStreamingWithoutActiveRun,
).toHaveBeenCalled(); // (d)
});
it('reconcileChat() settles THIS chat\'s stuck streaming rows by their run status', async () => {
const { svc, aiChatMessageRepo } = makeService({
messageRepo: {
findStreamingWithTerminalRun: jest.fn(async () => [
{ messageId: 'm1', workspaceId: 'ws1', runStatus: 'failed' },
{ messageId: 'm2', workspaceId: 'ws1', runStatus: 'succeeded' },
]),
},
});
await svc.reconcileChat('chat-1', 'ws1');
// Scoped to THIS chat and bounded at 50 (the user-facing opportunistic path).
expect(
aiChatMessageRepo.findStreamingWithTerminalRun,
).toHaveBeenCalledWith(50, { chatId: 'chat-1', workspaceId: 'ws1' });
// failed-run -> 'error'; every other terminal status -> 'aborted'.
expect(aiChatMessageRepo.stampTerminalIfStreaming).toHaveBeenCalledWith(
'm1',
'ws1',
'error',
);
expect(aiChatMessageRepo.stampTerminalIfStreaming).toHaveBeenCalledWith(
'm2',
'ws1',
'aborted',
);
});
});
+442 -64
View File
@@ -3,6 +3,7 @@ import {
ForbiddenException,
Injectable,
Logger,
OnModuleDestroy,
OnModuleInit,
ServiceUnavailableException,
} from '@nestjs/common';
@@ -13,6 +14,7 @@ import {
convertToModelMessages,
stepCountIs,
type UIMessage,
type ModelMessage,
type LanguageModel,
} from 'ai';
import { AiService } from '../../integrations/ai/ai.service';
@@ -42,7 +44,11 @@ import {
makeLoadToolsTool,
buildExternalToolCatalog,
} from './tools/tool-tiers';
import { RunAlreadyActiveError } from './ai-chat-run.service';
import {
RunAlreadyActiveError,
AiChatRunService,
} from './ai-chat-run.service';
import { inAppToolCallCapMs } from './tools/ai-chat-tools.service';
import { computePageChange } from './page-change/page-change.util';
import {
sanitizeSelection,
@@ -111,9 +117,14 @@ const FINAL_STEP_NUDGE =
// NO text at all (#444, mitigates the "empty turn" the lockdown used to prevent
// when the toggle is OFF). Makes the exhausted-without-answer state explicit to
// the user and, on replay, to the model on the next turn.
// The persisted content is the app's base locale (en-US) — which is ALSO the
// i18n key the client localizes through `t()` — instead of a hardcoded Russian
// string (it used to render Russian for every locale, and fed Russian back to
// the model on replay). Keep it a plain, model-readable English sentence so the
// next turn's replay reads cleanly; the client resolves the locale.
const STEP_LIMIT_NO_ANSWER_MARKER =
'(Достигнут лимит шагов — итоговый ответ не сформулирован; работа могла ' +
'остаться незавершённой. Напишите «продолжай», чтобы агент продолжил.)';
'(Step limit reached — no final answer was produced; the work may be ' +
'unfinished. Reply "continue" to let the agent carry on.)';
// Reason recorded in ai_chat_runs.error / the assistant row when the token-
// degeneration detector (#444) aborts a run. Distinct from a user Stop (no error)
@@ -250,15 +261,23 @@ export function cleanGeneratedTitle(text: string): string {
* partial output is already in history thanks to the step-granular write path).
*/
export function isInterruptResume(
history: Array<{ role: string; status?: string | null }>,
history: Array<{
role: string;
status?: string | null;
metadata?: unknown;
}>,
clientInterrupted: boolean | undefined,
): boolean {
if (clientInterrupted !== true) return false;
const prev = history[history.length - 2];
return (
prev?.role === 'assistant' &&
(prev.status === 'aborted' || prev.status === 'streaming')
);
if (prev?.role !== 'assistant') return false;
// #487: a reconcile STAMP (metadata.finalizeFailed) is NOT a genuine user
// interruption — the previous turn's process died and a reconcile settled the
// row as 'aborted'. Treating it as an interrupt-resume would inject a false
// "you were interrupted" note. Exclude any finalizeFailed row.
const meta = prev.metadata as { finalizeFailed?: unknown } | null | undefined;
if (meta && meta.finalizeFailed === true) return false;
return prev.status === 'aborted' || prev.status === 'streaming';
}
/**
@@ -379,6 +398,14 @@ export interface AiChatStreamBody {
// it against persisted history (`isInterruptResume`) before injecting the
// interrupt note, so a spoofed/stale flag on an ordinary turn is ignored.
interrupted?: boolean;
// #487: server-side supersede CAS. When present, this POST asks the server to
// STOP the run `supersede.runId` (which the client saw as the chat's active run)
// and, once it has settled, start THIS turn in its place. The server validates
// the target against the chat and answers 400 (wrong chat) / 409
// SUPERSEDE_TARGET_MISMATCH / 409 SUPERSEDE_TIMEOUT, or proceeds normally
// (degrade / ready). Absent => an ordinary send (rejected with 409
// A_RUN_ALREADY_ACTIVE if a run is already active on the chat).
supersede?: { runId?: string } | null;
// useChat sends the full UIMessage list; the last one is the new user turn.
messages?: UIMessage[];
}
@@ -428,6 +455,11 @@ export interface AiChatStreamArgs {
// chat row (existing chat) or the request body (new chat). null => universal
// assistant. Carried here so the turn never re-loads it.
role: AiAgentRole | null;
// #487: true when this turn was started by SUPERSEDING a still-live previous run
// (the controller ran the supersede CAS to a `ready` result). Adds the
// SUPERSEDE_NOTE to the system prompt (the previous run's last ops may still be
// applying — no side-effect quiescence). Absent on an ordinary send.
superseded?: boolean;
}
/**
@@ -444,7 +476,7 @@ export interface AiChatStreamArgs {
* can be rebuilt for `convertToModelMessages`.
*/
@Injectable()
export class AiChatService implements OnModuleInit {
export class AiChatService implements OnModuleInit, OnModuleDestroy {
private readonly logger = new Logger(AiChatService.name);
constructor(
@@ -465,8 +497,17 @@ export class AiChatService implements OnModuleInit {
// constructions (int-specs) compile unchanged; Nest always injects the real
// provider in production. Only ever touched on the run-wrapped + flag-on path.
private readonly streamRegistry?: AiChatStreamRegistryService,
// #487: the run lifecycle service, for the periodic + opportunistic reconcile
// (zombie re-drive + stale-run abort). OPTIONAL so positional test
// constructions compile unchanged; Nest always injects the real singleton, so
// reconcile sees the SAME in-memory active/zombie maps the runner mutates.
private readonly aiChatRunService?: AiChatRunService,
) {}
// #487: periodic reconcile timer (single-process phase 1). Started in
// onModuleInit, cleared in onModuleDestroy.
private reconcileTimer?: ReturnType<typeof setInterval>;
/**
* Crash-recovery sweep on server start (#183): any assistant row left in the
* 'streaming' state is the relic of a turn whose process died before it
@@ -491,6 +532,158 @@ export class AiChatService implements OnModuleInit {
}`,
);
}
// #487: start the PERIODIC reconcile (was boot-only). It heals both directions
// of the run<->message lifecycle asymmetry that a boot sweep alone left to the
// NEXT restart. Single-process phase 1: the in-memory active/zombie maps are
// authoritative, so "no live entry" is a safe primary gate.
const staleMs = this.reconcileStalenessMs();
// boot-warn if the per-call cap is configured so high the derived staleness is
// unusually long (a stale run then lingers longer before reconcile aborts it).
if (staleMs > 30 * 60 * 1000) {
this.logger.warn(
`#487 reconcile staleness is ${Math.round(staleMs / 60000)}min ` +
`(derived from max(2 x per-call cap, 15min)); a per-call cap this high ` +
`delays stale-run recovery. Review AI_CHAT_INAPP_TOOL_CALL_CAP_MS.`,
);
}
const intervalMs = this.reconcileIntervalMs();
this.reconcileTimer = setInterval(() => {
void this.reconcile().catch((err) => {
this.logger.warn(
`Periodic reconcile failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
});
}, intervalMs);
this.reconcileTimer.unref?.();
}
/** #487: stop the periodic reconcile timer on shutdown. */
onModuleDestroy(): void {
if (this.reconcileTimer) {
clearInterval(this.reconcileTimer);
this.reconcileTimer = undefined;
}
}
/**
* #487: reconcile staleness threshold X a run/message is only a "no live
* runner" abort candidate once UNTOUCHED past this. Derived as
* max(2 x per-call cap, 15min): 2x the longest legitimate single tool call plus
* a floor, so a marathon turn making steady progress (updatedAt bumped each
* step) is never swept.
*/
private reconcileStalenessMs(): number {
return Math.max(2 * inAppToolCallCapMs(), 15 * 60 * 1000);
}
/** #487: how often the periodic reconcile runs (env-tunable, default 2min). */
private reconcileIntervalMs(): number {
const raw = Number(process.env.AI_CHAT_RECONCILE_INTERVAL_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 2 * 60 * 1000;
}
/**
* #487: the periodic BIDIRECTIONAL reconcile. Runs the clauses IN ORDER; each is
* best-effort (a failure of one never blocks the others). Single-process phase 1
* the run service's in-memory maps are authoritative for "live entry".
*
* (a) re-drive ZOMBIE runs (a terminal write that gave up) apply the intended
* status via the conditional UPDATE;
* (b) message 'streaming' + its RUN terminal -> stamp the message by the run's
* status (succeeded-run + stuck row -> 'aborted'+finalizeFailed, NOT
* 'completed' with empty parts the final text lived only in the dead
* process's memory, a documented loss);
* (c) run active + NO live entry + NO zombie + stale -> aborted (the run
* service applies the "no entry" primary gate + last-progress staleness);
* (d) message 'streaming' + age>X + NO active run on the chat -> aborted
* (historical-row safety, double-gated).
*/
async reconcile(): Promise<void> {
const staleMs = this.reconcileStalenessMs();
// (a) zombie re-drive.
if (this.aiChatRunService) {
for (const runId of this.aiChatRunService.zombieRunIds()) {
try {
await this.aiChatRunService.settleZombie(runId);
} catch (err) {
this.logger.warn(
`Reconcile (a) zombie ${runId} re-drive failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
}
// (b) message streaming + run terminal -> stamp message by run status.
try {
const stuck = await this.aiChatMessageRepo.findStreamingWithTerminalRun();
for (const s of stuck) {
// succeeded-run -> 'aborted' (NOT 'completed'-empty); failed -> 'error';
// aborted -> 'aborted'. All via the finalizeFailed stamp.
const status = s.runStatus === 'failed' ? 'error' : 'aborted';
await this.aiChatMessageRepo.stampTerminalIfStreaming(
s.messageId,
s.workspaceId,
status,
);
}
} catch (err) {
this.logger.warn(
`Reconcile (b) message<-run failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
// (c) stale active run with no live runner -> aborted.
if (this.aiChatRunService) {
try {
await this.aiChatRunService.reconcileStaleRuns(staleMs);
} catch (err) {
this.logger.warn(
`Reconcile (c) stale-run abort failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
// (d) historical streaming row, no active run on the chat, stale -> aborted.
try {
await this.aiChatMessageRepo.sweepStreamingWithoutActiveRun(staleMs);
} catch (err) {
this.logger.warn(
`Reconcile (d) historical-row sweep failed: ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
}
/**
* #487: OPPORTUNISTIC single-chat reconcile at the start of a turn (beginRun /
* supersede path), so a user who returns to a chat with a stuck streaming row
* (its run already terminal) sees it settled WITHOUT waiting for the periodic
* job. Best-effort a failure NEVER fails the turn (swallowed by the caller).
*/
async reconcileChat(chatId: string, workspaceId: string): Promise<void> {
const stuck = await this.aiChatMessageRepo.findStreamingWithTerminalRun(50, {
chatId,
workspaceId,
});
for (const s of stuck) {
const status = s.runStatus === 'failed' ? 'error' : 'aborted';
await this.aiChatMessageRepo.stampTerminalIfStreaming(
s.messageId,
s.workspaceId,
status,
);
}
}
/**
@@ -727,6 +920,7 @@ export class AiChatService implements OnModuleInit {
model,
role,
runHooks,
superseded,
}: AiChatStreamArgs): Promise<void> {
// Resolve / create the chat. A new chat is created when no valid chatId is
// supplied or the supplied one does not belong to this workspace.
@@ -835,12 +1029,77 @@ export class AiChatService implements OnModuleInit {
}
}
// #487: opportunistic single-chat reconcile — settle any streaming row on this
// chat whose run is already terminal BEFORE this turn's history load, so the
// user never waits on the periodic job and the new turn's model history is not
// polluted by a stuck 'streaming' row. Best-effort: it must NEVER fail the turn.
try {
await this.reconcileChat(chatId, workspace.id);
} catch (err) {
this.logger.debug(
`Opportunistic reconcile for chat ${chatId} failed (ignored): ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
try {
// Extract the incoming user turn (the last user message from useChat).
const incoming = lastUserMessage(body.messages);
const incomingText = uiMessageText(incoming);
// Persist the user message before contacting the model.
// #489: sanitize client-supplied parts ON RECEIPT. The client only ever
// sends `sendMessage({ text })` (a single text part); there is no
// file/attachment path. Any other part — most dangerously a tool-part in
// `input-available` state — is untrusted data that, once persisted to
// `metadata.parts` verbatim, is REPLAYED through convertToModelMessages on
// every later turn. A malformed tool-part makes that conversion throw,
// 500-ing every future turn of the chat forever ("bricked"). Drop any
// non-whitelisted part with a warn.
const sanitizedParts = sanitizeUserParts(incoming?.parts, (type) =>
this.logger.warn(
`Dropping unsupported user message part '${type}' on chat ${chatId}`,
),
);
// #489: rebuild the conversation from persisted history (not the client
// payload) and CONVERT it to model messages BEFORE persisting the user row.
// Load the OLD history (WITHOUT the new row) and append the incoming turn in
// memory for the conversion. This makes the insert happen only after a
// successful conversion, so a conversion failure cannot leave a DUPLICATE
// user row behind on the client's retry (the "bricked chat" that accreted a
// dup on every 500). `findAllByChat` returns chronological order (oldest ->
// newest) and keeps a 5000-row memory-safety backstop (on overflow it keeps
// the NEWEST rows and logs a warning); that is a safety net far above any
// realistic chat, not a conversational limit.
const oldHistory = await this.aiChatMessageRepo.findAllByChat(
chatId,
workspace.id,
);
const uiMessages: Array<Omit<UIMessage, 'id'> & { id: string }> = [
...oldHistory.map(rowToUiMessage),
{
id: 'pending-user',
role: 'user',
parts: (sanitizedParts && sanitizedParts.length > 0
? sanitizedParts
: textPart(incomingText)) as UIMessage['parts'],
},
];
// convertToModelMessages is async in ai@6.0.134 (returns Promise<ModelMessage[]>).
// Resilient (#489): a single poisoned row in the OLD history is isolated via
// per-row conversion and degraded to plain text with a "[tool context
// omitted]" marker rather than 500-ing the whole turn (silent loss of tool
// context is not acceptable — the model must see the truncation).
const messages = await convertHistoryResilient(uiMessages, (index, err) =>
this.logger.warn(
`Degraded unconvertible history row ${index} on chat ${chatId} to text: ${
err instanceof Error ? err.message : 'unknown error'
}`,
),
);
// Persist the user message only AFTER a successful conversion (#489).
await this.aiChatMessageRepo.insert({
chatId,
workspaceId: workspace.id,
@@ -848,31 +1107,21 @@ export class AiChatService implements OnModuleInit {
role: 'user',
content: incomingText,
// jsonb column: UIMessage parts are JSON-serializable at runtime but not
// structurally `JsonValue`, so cast through unknown.
metadata: (incoming?.parts ? { parts: incoming.parts } : null) as never,
// structurally `JsonValue`, so cast through unknown. Persist the SANITIZED
// parts (never the raw client parts) so the row is always convertible.
metadata: (sanitizedParts ? { parts: sanitizedParts } : null) as never,
});
// Rebuild the conversation from persisted history (not the client payload),
// so the model always sees the authoritative server-side transcript. Load
// the FULL history in chronological order (oldest -> newest, incl. the user
// message just inserted above) so NO turns are dropped — there is no
// recent-tail window anymore. `findAllByChat` keeps a 5000-row memory-safety
// backstop (on overflow it keeps the NEWEST rows and logs a warning); that
// is a safety net far above any realistic chat, not a conversational limit.
const history = await this.aiChatMessageRepo.findAllByChat(
chatId,
workspace.id,
);
const uiMessages = history.map(rowToUiMessage);
// convertToModelMessages is async in ai@6.0.134 (returns Promise<ModelMessage[]>).
const messages = await convertToModelMessages(uiMessages);
// Interrupt-resume detection (#198): the client "send now" flag is only a
// hint — confirm it against the persisted history (the preceding assistant
// turn must really be aborted/streaming) so a spoofed flag cannot inject the
// interrupt note onto an ordinary turn. The partial output the model needs is
// already in `messages` (the aborted assistant row replays via findRecent).
const interrupted = isInterruptResume(history, body.interrupted);
// Append the new user turn (shape-only) so index -2 is the prior assistant.
const interrupted = isInterruptResume(
[...oldHistory, { role: 'user', status: null, metadata: null }],
body.interrupted,
);
// Per-turn page-change detection (#274): if the open page was hand-edited by
// the user since the agent's last turn ended, compute the unified diff so the
@@ -929,14 +1178,13 @@ export class AiChatService implements OnModuleInit {
);
} catch (err) {
// An explicit Stop reached the RUN's signal DURING setup: re-throw so the
// outer catch finalizes the run as aborted — never swallow a Stop. Gated on
// `runId`: the re-throw exists ONLY to finalize the run, which exists only
// in autonomous mode. On the legacy path (no runId) `effectiveSignal` is the
// SOCKET signal (it aborts on a client disconnect); re-throwing there would
// change prior behavior and make the controller write JSON to an already-
// closed socket (it only attaches res.raw.on('error') in autonomous mode).
// So legacy keeps its prior behavior — warn + proceed, and streamText then
// observes the aborted socket signal.
// outer catch finalizes the run as aborted — never swallow a Stop. #487: the
// turn is ALWAYS run-wrapped now (both modes), so `effectiveSignal` is the
// RUN signal and `runId` is set in BOTH — a Stop (from /ai-chat/stop or a
// legacy disconnect's requestStop) aborts it identically. The `runId` guard
// now only defends the theoretical no-handle fallback (`begin` returned
// nothing, leaving `effectiveSignal` as the bare socket signal): there we
// keep the old warn-and-proceed rather than re-throw.
if (runId && effectiveSignal.aborted) {
throw err;
}
@@ -1040,6 +1288,9 @@ export class AiChatService implements OnModuleInit {
// History-confirmed interrupt-resume flag (#198): adds the interrupt note
// so the model treats the partial answer above as cut off, not finished.
interrupted,
// #487: this turn superseded a still-live run — warn the model the
// previous run's last ops may still be applying (no quiescence).
superseded,
// Detected between-turns human edit to the open page (#274): adds the
// page_changed note + unified diff so the agent doesn't overwrite it.
pageChanged,
@@ -1194,29 +1445,59 @@ export class AiChatService implements OnModuleInit {
// callbacks — mirroring the pre-#183 persist-at-most-once guard for the
// TERMINAL status (the row may be updated many times with 'streaming' before
// this fires once).
// #487: the once-gate closes ONLY AFTER a successful write, and the write is
// BOUNDED-RETRIED. Previously `finalized` was set BEFORE the write and never
// retried, so a single failed UPDATE stranded the row 'streaming' forever
// (the boot-only sweep was the only recovery). Now a transient blip is ridden
// out in place; a total give-up leaves the gate OPEN and logs, and the
// periodic reconcile (clauses b/d) later settles the row. Returns whether the
// terminal write LANDED, so the caller can error-mark the RUN on a message
// failure (the run is finalized regardless — never gated on the message).
let finalized = false;
const FINALIZE_MSG_MAX_ATTEMPTS = 3;
const finalizeAssistant = async (
flushed: AssistantFlush,
): Promise<void> => {
if (finalized) return;
finalized = true;
): Promise<boolean> => {
if (finalized) return true;
const plan = planFinalizeAssistant(assistantId);
try {
// Shared dispatch (see applyFinalize): UPDATE the upfront row, or — when
// the upfront insert failed (kind 'insert') — INSERT the terminal row as
// the only safety against losing the turn entirely.
await applyFinalize(
this.aiChatMessageRepo,
plan,
{ chatId, workspaceId: workspace.id, userId: user.id },
flushed,
);
} catch (err) {
this.logger.error(
`Failed to finalize assistant message (kind=${plan.kind})`,
err as Error,
);
let lastError: unknown;
for (let attempt = 1; attempt <= FINALIZE_MSG_MAX_ATTEMPTS; attempt++) {
try {
// Shared dispatch (see applyFinalize): conditionally UPDATE the upfront
// row (owner-write priority), or — when the upfront insert failed (kind
// 'insert') — INSERT the terminal row as the only safety against losing
// the turn entirely.
await applyFinalize(
this.aiChatMessageRepo,
plan,
{ chatId, workspaceId: workspace.id, userId: user.id },
flushed,
);
finalized = true; // gate closes ONLY after a successful write
return true;
} catch (err) {
lastError = err;
this.logger.warn(
`Assistant message finalize attempt ${attempt}/${FINALIZE_MSG_MAX_ATTEMPTS} ` +
`failed (kind=${plan.kind}): ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
if (attempt < FINALIZE_MSG_MAX_ATTEMPTS) {
await new Promise((r) => setTimeout(r, 50 * attempt));
}
}
}
// Gave up: leave the gate OPEN (no in-process second settler exists — the
// terminal callbacks are mutually exclusive) and log. The periodic reconcile
// settles the stranded row; a late owner-write is impossible for this turn,
// so the reconcile stamp (aborted+finalizeFailed) is the final state.
this.logger.error(
`Assistant message finalize GAVE UP after ${FINALIZE_MSG_MAX_ATTEMPTS} ` +
`attempts (row left 'streaming', chat ${chatId}); reconcile will settle it`,
lastError as Error,
);
return false;
};
// DIAGNOSTIC (Safari stream-drop investigation) — temporary. Measure
@@ -1238,6 +1519,13 @@ export class AiChatService implements OnModuleInit {
system,
messages,
tools,
// Pin the AI SDK per-request retry budget explicitly instead of relying
// on its default (which is also 2). Connection arithmetic per turn:
// (1 + maxRetries=2) × (1 + AI_STREAM_PRE_RESPONSE_RETRIES) network
// connects worst-case — the two retry layers compose, so making the SDK
// side explicit keeps that ceiling visible and pinned against SDK-default
// drift.
maxRetries: 2,
// No maxOutputTokens cap on the agent: tool-call arguments (e.g. a full
// page body for the write tools) are emitted as OUTPUT tokens, so a fixed
// cap would truncate complex tool calls mid-argument. Let the model use its
@@ -1361,7 +1649,7 @@ export class AiChatService implements OnModuleInit {
const stepExhausted = steps.length >= MAX_AGENT_STEPS;
const emptyTurnMarker =
!producedText && stepExhausted ? STEP_LIMIT_NO_ANSWER_MARKER : '';
await finalizeAssistant(
const msgOk = await finalizeAssistant(
flushAssistant(steps as StepLike[], emptyTurnMarker, 'completed', {
finishReason: finishReason as string,
usage: totalUsage as StreamUsage,
@@ -1375,9 +1663,19 @@ export class AiChatService implements OnModuleInit {
pageChanged,
}),
);
// #184: settle the RUN as succeeded (best-effort, after the projection
// is finalized above).
if (runId) await runHooks?.onSettled?.(runId, 'completed');
// #184/#487: the RUN is finalized ALWAYS (never gated on the message).
// If the message finalize GAVE UP, error-mark the run so the asymmetry
// "run succeeded / message streaming forever" cannot arise; the
// periodic reconcile then settles the stuck message from this run.
if (runId) {
await runHooks?.onSettled?.(
runId,
msgOk ? 'completed' : 'error',
msgOk
? undefined
: 'Assistant message could not be persisted (finalize failed).',
);
}
// Lifecycle: release the external MCP clients leased for this turn.
await closeExternalClients();
@@ -1830,6 +2128,82 @@ function textPart(text: string): Array<{ type: 'text'; text: string }> {
return text ? [{ type: 'text', text }] : [];
}
/**
* Part types accepted on an INCOMING user turn (#489). The client only ever
* sends `sendMessage({ text })` (a single text part); there is no file/attachment
* path. Everything else on a client-supplied user message most dangerously a
* tool-part in `input-available` state is untrusted data that would be
* persisted to `metadata.parts` verbatim and replayed through
* `convertToModelMessages` on every later turn, potentially bricking the chat.
*/
const ALLOWED_USER_PART_TYPES: ReadonlySet<string> = new Set(['text']);
/**
* Keep only whitelisted parts on a client-supplied user message; report each
* dropped part's type via `onDrop` (the caller warns). Returns `undefined` when
* nothing survives (no parts / none whitelisted), so the caller persists a null
* metadata rather than an empty-parts object. Never throws.
*/
export function sanitizeUserParts(
parts: UIMessage['parts'] | undefined,
onDrop: (type: string) => void,
): UIMessage['parts'] | undefined {
if (!Array.isArray(parts)) return undefined;
const kept = parts.filter((p) => {
const type =
typeof (p as { type?: unknown })?.type === 'string'
? (p as { type: string }).type
: '';
if (ALLOWED_USER_PART_TYPES.has(type)) return true;
onDrop(type || '(unknown)');
return false;
});
return kept.length > 0 ? (kept as UIMessage['parts']) : undefined;
}
/** Marker for a history row whose tool parts could not be replayed (#489). */
export const TOOL_CONTEXT_OMITTED_MARKER = '[tool context omitted]';
/**
* Convert persisted UI history to model messages, tolerating a single poisoned
* row (#489). `convertToModelMessages` over the WHOLE array throws if ANY row is
* malformed (e.g. a tool-part left unbalanced / in `input-available` state),
* which would otherwise 500 every turn of the chat forever. On a batch failure we
* fall back to per-row conversion so the bad row is isolated: it is degraded to
* plain text carrying its readable text plus a `[tool context omitted]` marker
* (the model MUST see that its tool context was truncated silent loss is not
* acceptable), while every healthy row converts normally. Because AI SDK v6
* carries a tool call and its result inside the SAME assistant UIMessage's parts,
* per-row conversion preserves call/result pairing.
*/
export async function convertHistoryResilient(
uiMessages: Array<Omit<UIMessage, 'id'> & { id: string }>,
onDegrade: (index: number, err: unknown) => void,
): Promise<ModelMessage[]> {
try {
return await convertToModelMessages(uiMessages as UIMessage[]);
} catch {
const out: ModelMessage[] = [];
for (let i = 0; i < uiMessages.length; i++) {
const m = uiMessages[i];
try {
out.push(...(await convertToModelMessages([m as UIMessage])));
} catch (err) {
onDegrade(i, err);
const text = uiMessageText(m as UIMessage);
const degraded = text
? `${text}\n\n${TOOL_CONTEXT_OMITTED_MARKER}`
: TOOL_CONTEXT_OMITTED_MARKER;
out.push({
role: m.role === 'assistant' ? 'assistant' : 'user',
content: degraded,
} as ModelMessage);
}
}
return out;
}
}
/**
* Minimal shapes of the AI SDK v6 step objects we read to rebuild UIMessage
* parts (see ai@6.0.134 `StepResult`: `text`, `toolCalls` -> TypedToolCall,
@@ -2118,7 +2492,10 @@ export function planFinalizeAssistant(
* a test mock both satisfy it). */
export interface FinalizeRepo {
insert(insertable: Record<string, unknown>): Promise<unknown>;
update(
// #487: the OWNER terminal write is CONDITIONAL (status='streaming' OR
// metadata.finalizeFailed) so the owner overwrites a reconcile stamp but never
// an already-proper terminal row (owner-write priority).
finalizeOwner(
id: string,
workspaceId: string,
patch: AssistantFlush,
@@ -2127,10 +2504,11 @@ export interface FinalizeRepo {
/**
* Apply a finalize `plan` to the repo with the terminal `flushed` payload (#183):
* UPDATE the upfront row, or INSERT a fresh terminal row as the fallback when the
* upfront insert failed. The SINGLE dispatch shared by the service's
* finalizeAssistant and its test, so the test exercises the real path instead of
* a copy (#186 review). Pure of error handling the caller wraps it.
* conditionally UPDATE the upfront row (owner-write priority, #487), or INSERT a
* fresh terminal row as the fallback when the upfront insert failed. The SINGLE
* dispatch shared by the service's finalizeAssistant and its test, so the test
* exercises the real path instead of a copy (#186 review). Pure of error
* handling the caller wraps it (and RETRIES it, #487).
*/
export async function applyFinalize(
repo: FinalizeRepo,
@@ -2139,7 +2517,7 @@ export async function applyFinalize(
flushed: AssistantFlush,
): Promise<void> {
if (plan.kind === 'update') {
await repo.update(plan.id, base.workspaceId, flushed);
await repo.finalizeOwner(plan.id, base.workspaceId, flushed);
return;
}
await repo.insert({
@@ -0,0 +1,261 @@
import { errors } from 'undici';
import {
McpClientsService,
isRetryableConnectError,
} from './mcp-clients.service';
/**
* #489 external-MCP in-run transport recovery.
*
* The transport-error classification + retry gate are exercised against the REAL
* undici error CLASSES prod throws (`errors.SocketError` / `errors.BodyTimeoutError`,
* carrying the true `UND_ERR_*` codes and class names), wrapped EXACTLY as undici's
* `fetch` wraps them a `TypeError('fetch failed'|'terminated')` whose `.cause` is
* the undici error. These are the real classes, not hand-rolled `{code:'...'}`
* mocks: constructing the genuine class is what makes this a faithful test of the
* prod predicate (epic root-cause #4 a mock-shaped predicate would leave the
* evict/retry path silently dead in production while CI stays green). We construct
* rather than drive a live fetch because Jest's environment degrades the live-fetch
* error to a generic `Error` cause (no undici code), which would NOT be the prod
* shape.
*/
/** A REAL undici socket reset, wrapped as fetch wraps it. */
function realSocketResetError(): unknown {
const err = new TypeError('fetch failed');
(err as { cause?: unknown }).cause = new errors.SocketError('other side closed');
return err;
}
/** A REAL undici body timeout, wrapped as fetch wraps it. */
function realBodyTimeoutError(): unknown {
const err = new TypeError('terminated');
(err as { cause?: unknown }).cause = new errors.BodyTimeoutError();
return err;
}
type FakeServer = {
id: string;
name: string;
transport: 'http' | 'sse';
url: string;
headersEnc: string | null;
toolAllowlist: string[] | null;
instructions: string | null;
};
const server = (over: Partial<FakeServer> = {}): FakeServer => ({
id: 's1',
name: 'srv',
transport: 'http',
url: 'http://example.test/mcp',
headersEnc: null,
toolAllowlist: null,
instructions: null,
...over,
});
function buildService(servers: FakeServer[], trusted = false) {
const repo = { listEnabled: jest.fn().mockResolvedValue(servers) };
const service = new McpClientsService(repo as never, {} as never);
// Seed a DETERMINISTIC write-class map so the retry gate is controlled here
// (the production map loads from @docmost/mcp via a dynamic ESM import). getPage
// is a read, patchNode is a write — the real classifications.
(
service as unknown as { writeClassMapPromise: Promise<unknown> }
).writeClassMapPromise = Promise.resolve({
getPage: 'readOnly',
patchNode: 'write',
});
// The service only APPLIES that map to a TRUSTED internal Docmost server
// (isInternalDocmostServer, really false for every third-party row). A retry
// test needs a trusted server to exercise the readOnly-retry path at all, so it
// passes trusted=true to model a Docmost-origin server; the third-party
// double-apply test leaves it at the real value (false).
if (trusted) {
jest
.spyOn(
service as unknown as {
isInternalDocmostServer: (s: FakeServer) => boolean;
},
'isInternalDocmostServer',
)
.mockReturnValue(true);
}
return { service, repo };
}
/** Spy the private `connect` so each call yields a controlled fake client whose
* single tool's execute is the supplied function. Returns the connect spy. */
function stubConnect(
service: McpClientsService,
toolName: string,
execs: Array<(...a: unknown[]) => Promise<unknown>>,
) {
let n = 0;
return jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => Promise<unknown> },
'connect',
)
.mockImplementation(async () => {
const exec = execs[Math.min(n, execs.length - 1)];
n += 1;
return {
tools: async () => ({ [toolName]: { description: 'x', execute: exec } }),
close: jest.fn().mockResolvedValue(undefined),
};
});
}
const opts = (abortSignal?: AbortSignal) =>
({ toolCallId: 't', messages: [], abortSignal }) as never;
describe('isRetryableConnectError (#489, REAL error shapes)', () => {
it('classifies a real undici socket reset and body timeout as retryable', async () => {
const socketErr = await realSocketResetError();
const bodyErr = await realBodyTimeoutError();
expect(isRetryableConnectError(socketErr)).toBe(true);
expect(isRetryableConnectError(bodyErr)).toBe(true);
// Unwraps a wrapped cause chain (e.g. an MCPClientError around the socket err).
const wrapped = new Error('mcp call failed');
(wrapped as { cause?: unknown }).cause = socketErr;
expect(isRetryableConnectError(wrapped)).toBe(true);
});
it('does NOT classify an application-level error as a transport break', () => {
expect(isRetryableConnectError(new Error('validation failed'))).toBe(false);
expect(isRetryableConnectError({ name: 'HttpError', status: 400 })).toBe(false);
expect(isRetryableConnectError(undefined)).toBe(false);
expect(isRetryableConnectError('boom')).toBe(false);
});
});
describe('McpClientsService in-run transport recovery (#489)', () => {
afterEach(() => jest.restoreAllMocks());
it('a readOnly tool whose transport breaks reconnects and retries WITHIN the same run', async () => {
const realErr = await realSocketResetError();
const { service } = buildService([server()], true);
const first = jest.fn().mockRejectedValue(realErr);
const second = jest.fn().mockResolvedValue({ ok: true });
const connectSpy = stubConnect(service, 'getPage', [first, second]);
const toolset = await service.toolsFor('ws-1');
const tool = toolset.tools['srv_getPage'];
const result = await (tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
);
// The repeat call within the run got a LIVE client and succeeded.
expect(result).toEqual({ ok: true });
expect(first).toHaveBeenCalledTimes(1);
expect(second).toHaveBeenCalledTimes(1);
// Exactly one reconnect was minted (initial build connect + one recovery).
expect(connectSpy).toHaveBeenCalledTimes(2);
// The run accumulated BOTH leases (old + reconnected) — released together at end.
expect(toolset.clients).toHaveLength(2);
await Promise.all(toolset.clients.map((c) => c.close()));
});
it('a WRITE tool does NOT auto-retry on a transport error (indeterminate)', async () => {
const realErr = await realSocketResetError();
const { service } = buildService([server()], true);
const exec = jest.fn().mockRejectedValue(realErr);
const connectSpy = stubConnect(service, 'patchNode', [exec]);
const toolset = await service.toolsFor('ws-2');
const tool = toolset.tools['srv_patchNode'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
),
).rejects.toThrow(/MAY have already applied/);
// Called exactly once — NO blind retry (avoids double-apply, the #435 class).
expect(exec).toHaveBeenCalledTimes(1);
// No fresh connection was minted for a write.
expect(connectSpy).toHaveBeenCalledTimes(1);
await Promise.all(toolset.clients.map((c) => c.close()));
});
it('does NOT retry (or reconnect) after the run is aborted (Stop)', async () => {
const realErr = await realSocketResetError();
const { service } = buildService([server()], true);
const controller = new AbortController();
// The transport error arrives, but the run was Stopped in the same tick.
const first = jest.fn().mockImplementation(async () => {
controller.abort();
throw realErr;
});
const second = jest.fn().mockResolvedValue({ ok: true });
const connectSpy = stubConnect(service, 'getPage', [first, second]);
const toolset = await service.toolsFor('ws-3');
const tool = toolset.tools['srv_getPage'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(controller.signal),
),
).rejects.toBeDefined();
// getPage IS readOnly, but the Stop blocks the retry — no second call, no mint.
expect(second).not.toHaveBeenCalled();
expect(connectSpy).toHaveBeenCalledTimes(1);
await Promise.all(toolset.clients.map((c) => c.close()));
});
it('an app-level (non-transport) tool error is surfaced verbatim, never retried', async () => {
const { service } = buildService([server()], true);
const appErr = new Error('tool says: bad input');
const exec = jest.fn().mockRejectedValue(appErr);
const connectSpy = stubConnect(service, 'getPage', [exec]);
const toolset = await service.toolsFor('ws-4');
const tool = toolset.tools['srv_getPage'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
),
).rejects.toThrow('tool says: bad input');
expect(exec).toHaveBeenCalledTimes(1);
expect(connectSpy).toHaveBeenCalledTimes(1); // no reconnect for an app error
await Promise.all(toolset.clients.map((c) => c.close()));
});
// #489 (review, MEDIUM) — the Docmost write-class map keys by DOCMOST tool
// names; a THIRD-PARTY server may name a WRITE tool `getPage` (a Docmost read
// name). It must NOT inherit readOnly and must NOT auto-retry on a transport
// error — a blind retry of that write is a double-apply (the #435 class). Here
// the server is UNTRUSTED (buildService default, isInternalDocmostServer=false),
// so the map is not applied and `getPage` classifies as a write.
//
// MUTATION-VERIFY: forcing the server "trusted" (buildService(..., true)) makes
// `getPage` inherit readOnly -> it WOULD reconnect+retry (connect twice) and the
// assertions below fail — i.e. removing the trust scope re-opens the bug.
it('a THIRD-PARTY WRITE tool named like a Docmost read does NOT auto-retry (no double-apply)', async () => {
const realErr = await realSocketResetError();
// Untrusted: default trusted=false — a real third-party server.
const { service } = buildService([server()]);
const exec = jest.fn().mockRejectedValue(realErr);
const connectSpy = stubConnect(service, 'getPage', [exec, exec]);
const toolset = await service.toolsFor('ws-5');
const tool = toolset.tools['srv_getPage'];
await expect(
(tool.execute as (a: unknown, o: unknown) => Promise<unknown>)(
{ pageId: 'p' },
opts(),
),
).rejects.toThrow(/MAY have already applied/);
// Exactly one call, NO reconnect — the name collision granted no readOnly-retry.
expect(exec).toHaveBeenCalledTimes(1);
expect(connectSpy).toHaveBeenCalledTimes(1);
await Promise.all(toolset.clients.map((c) => c.close()));
});
});
@@ -106,8 +106,11 @@ describe('McpClientsService.decryptHeaders', () => {
describe('McpClientsService.guardedFetch (SSRF per-request guard)', () => {
// The bound guardedFetch closure lives on the instance as a private field.
// #489 split it into per-transport HTTP/SSE bindings (they differ only in the
// dispatcher's bodyTimeout); the SSRF guard is identical, so testing the HTTP
// one is sufficient.
const guardedFetchOf = (service: McpClientsService) =>
(service as unknown as { guardedFetch: typeof fetch }).guardedFetch;
(service as unknown as { guardedFetchHttp: typeof fetch }).guardedFetchHttp;
let fetchSpy: jest.SpiedFunction<typeof fetch>;
@@ -1,5 +1,6 @@
import { isIP } from 'node:net';
import { lookup as dnsLookup, type LookupAddress } from 'node:dns';
import { pathToFileURL } from 'node:url';
import { Injectable, Logger } from '@nestjs/common';
import { type Tool, type ToolCallOptions } from 'ai';
import { createMCPClient } from '@ai-sdk/mcp';
@@ -10,9 +11,29 @@ import {
streamingDispatcherOptions,
mcpStreamTimeoutMs,
mcpCallTimeoutMs,
mcpSseBodyTimeoutMs,
} from '../../../integrations/ai/ai-streaming-fetch';
import { SecretBoxService } from '../../../integrations/crypto/secret-box';
import { isUrlAllowed, isIpAllowed } from './ssrf-guard';
// TYPE-ONLY (erased at compile): @docmost/mcp is ESM-only and cannot be a runtime
// `require()` from this commonjs module (same constraint as docmost-client.loader).
// The write-class MAP is loaded lazily via the dynamic-import trick below.
import type { ToolWriteClass } from '@docmost/mcp';
// TS(commonjs) downlevels a literal `import()` to `require()`, which cannot load
// the ESM-only @docmost/mcp. Indirect through Function so the real dynamic
// `import()` survives compilation (same trick as docmost-client.loader.ts).
const esmImport = new Function(
'specifier',
'return import(specifier)',
) as (specifier: string) => Promise<unknown>;
/** Local read-only predicate avoids a value import of the ESM-only package.
* Only a pure read is retry-safe after a transport break (a write is
* indeterminate). Kept in lockstep with @docmost/mcp's isRetryableWriteClass. */
function isReadOnlyWriteClass(writeClass: ToolWriteClass | undefined): boolean {
return writeClass === 'readOnly';
}
/** A closable external MCP client handle. */
export interface Closable {
@@ -81,12 +102,52 @@ const MAX_TOOL_NAME_LENGTH = 64;
* close until the turn releases it, so a TTL expiry mid-turn never closes a
* client a stream is still executing against.
*/
/**
* Where a merged (namespaced) tool came from, so the per-run recovery wrapper
* (#489) can, on a transport error, reconnect THAT server and re-resolve the SAME
* underlying tool by its raw name. `writeClass` gates the single auto-retry (a
* read is retry-safe; a write is indeterminate). `serverIndex` indexes the
* entry's `servers` array (which server config to reconnect).
*/
interface ToolProvenance {
serverIndex: number;
rawName: string;
writeClass: ToolWriteClass | undefined;
}
/** A live reconnected server (its fresh client + raw call-timeout-wrapped tools). */
interface RecoveredServerState {
client: McpClient;
tools: Record<string, Tool>;
}
/**
* Per-run, per-server recovery binding (#489). `current` is the server's LIVE
* target for this run: `null` means "use the ORIGINAL cached client/template";
* a non-null value is a reconnected throwaway client all this server's tools now
* call. `reconnecting` dedupes concurrent reconnects so only ONE fresh client is
* minted per death (a losing concurrent call awaits it and retries on the SAME
* new client the CAS-by-identity rule).
*/
interface ServerBinding {
current: RecoveredServerState | null;
reconnecting?: Promise<RecoveredServerState>;
}
interface CacheEntry {
tools: Record<string, Tool>;
clients: McpClient[];
outcomes: ServerOutcome[];
/** Prompt guidance for qualifying servers (see McpServerInstruction). */
instructions: McpServerInstruction[];
/**
* The enabled server configs used to build this entry (#489), so the per-run
* recovery wrapper can reconnect a specific server by index. Parallel to the
* indices referenced by {@link toolMeta}.
*/
servers: AiMcpServer[];
/** merged-tool-key -> provenance (#489), for the per-run recovery wrapper. */
toolMeta: Record<string, ToolProvenance>;
expiresAt: number;
/** Active leases (turns currently using these clients). */
refCount: number;
@@ -120,20 +181,82 @@ export class McpClientsService {
*/
private readonly cache = new Map<string, Promise<CacheEntry>>();
/**
* A single shared SSRF-pinned dispatcher for ALL outbound external-MCP fetches.
* Its custom connect.lookup runs per connection, so one instance safely guards
* every server's connections (we never connect to an unvalidated IP).
* SSRF-pinned dispatchers for outbound external-MCP fetches. Both use the SAME
* custom connect.lookup (so every connection is IP-validated), but carry a
* DIFFERENT `bodyTimeout` (#489): the HTTP (streamable) transport opens a fresh
* request per call, so it keeps the tight silence timeout; the SSE transport
* holds ONE long-lived body open across many calls, so a >1-min idle BETWEEN
* calls is LEGITIMATE and must not break the socket it gets a much larger
* bodyTimeout. (headersTimeout stays tight on both.)
*/
private readonly dispatcher: Dispatcher = buildPinnedDispatcher();
/** guardedFetch bound to the pinned dispatcher; reused by every transport. */
private readonly guardedFetch: typeof fetch = (input, init) =>
guardedFetch(this.dispatcher, input, init);
private readonly dispatcherHttp: Dispatcher = buildPinnedDispatcher(
mcpStreamTimeoutMs(),
);
private readonly dispatcherSse: Dispatcher = buildPinnedDispatcher(
mcpSseBodyTimeoutMs(),
);
/** guardedFetch bound to each dispatcher; picked by transport type in connect(). */
private readonly guardedFetchHttp: typeof fetch = (input, init) =>
guardedFetch(this.dispatcherHttp, input, init);
private readonly guardedFetchSse: typeof fetch = (input, init) =>
guardedFetch(this.dispatcherSse, input, init);
/**
* Memoized write-class map (#489), loaded lazily from @docmost/mcp via the
* dynamic-import trick. Keyed by tool name (=== mcpName). A tool NOT in the map
* (any third-party external MCP tool) classifies as `undefined` -> treated as a
* write by the retry gate (the safe default: never blind-retry an unknown tool).
* On any load failure the map is `{}` (every tool -> no auto-retry), so a
* missing/older @docmost/mcp build only DISABLES retries, never mis-retries.
*/
private writeClassMapPromise: Promise<Record<string, ToolWriteClass>> | null =
null;
constructor(
private readonly repo: AiMcpServerRepo,
private readonly secretBox: SecretBoxService,
) {}
/**
* Whether an external MCP server is the TRUSTED internal Docmost MCP server
* the only server whose tools may be classified by the Docmost write-class map
* (#489 review). Today this is ALWAYS false: every `ai_mcp_servers` row is an
* admin-configured THIRD-PARTY endpoint (there is no builtin/self flag, sentinel
* URL, or synthetic server in this path Docmost's OWN tools are exposed via the
* separate in-app tools path, never through this external-MCP client). So no
* third-party tool can inherit `readOnly` by a name collision with a Docmost read
* tool, and none is ever auto-retried on a transport error (which would risk a
* double-apply the #435 class). Flip this (an explicit `kind`/`isBuiltin`
* column, or a configured self-MCP URL) if a trusted internal server is ever
* introduced. A method (not a free function) so it is a single, mockable seam.
*/
private isInternalDocmostServer(_server: AiMcpServer): boolean {
return false;
}
/** Lazily load + memoize the shared write-class map (see the field doc). */
private getWriteClassMap(): Promise<Record<string, ToolWriteClass>> {
if (!this.writeClassMapPromise) {
this.writeClassMapPromise = (async () => {
try {
const entry = require.resolve('@docmost/mcp');
const mod = (await esmImport(pathToFileURL(entry).href)) as {
SHARED_TOOL_WRITE_CLASS?: Record<string, ToolWriteClass>;
};
return mod.SHARED_TOOL_WRITE_CLASS ?? {};
} catch (err) {
this.logger.warn(
`Could not load MCP write-class map (auto-retry disabled): ${shortError(
err,
)}`,
);
return {};
}
})();
}
return this.writeClassMapPromise;
}
/**
* Build (or reuse a cached) external toolset for a workspace. Returns the
* merged tools, the open client handles to release, and per-server outcomes.
@@ -162,11 +285,37 @@ export class McpClientsService {
}
},
};
// One release handle drives the whole leased entry; closing it releases all
// underlying clients together (they share the same lease lifecycle).
// #489: the run accumulates a SET of leases — the primary cache lease PLUS any
// throwaway client minted by an in-run transport-recovery reconnect. They are
// NEVER released mid-run (releasing a swapped-out client while a concurrent
// in-flight call still holds it would INDUCE a second failure); the caller
// releases the WHOLE set together at turn-end. A recovery reconnect pushes its
// lease onto this live array, which the consumer closes over.
const leaseSet: Closable[] = [release];
// #489: per-RUN transport-recovery binding, one per server, SHARED by all of
// that server's tools so a swap by one call is seen by the next (CAS by
// identity). Kept per-run (here, not in the cached entry) because the binding
// + lease-set state is per-run.
const bindings = new Map<number, ServerBinding>();
const capMs = mcpCallTimeoutMs();
// Wrap each cached tool with the recovery layer. On a transport error a
// declared readOnly tool reconnects its server and retries ONCE; a write is
// never blind-retried (indeterminate — may have applied before the reset). A
// tool without provenance (a minimal stub entry in a test) passes through raw.
const tools: Record<string, Tool> = {};
for (const [key, tool] of Object.entries(entry.tools)) {
const meta = entry.toolMeta?.[key];
tools[key] = meta
? this.wrapWithTransportRecovery(entry, meta, tool, leaseSet, bindings, capMs)
: tool;
}
return {
tools: entry.tools,
clients: [release],
tools,
clients: leaseSet,
outcomes: entry.outcomes,
instructions: entry.instructions,
};
@@ -254,6 +403,16 @@ export class McpClientsService {
// Per-call total wall-clock cap, read once for this build (env-overridable).
const callTimeoutMs = mcpCallTimeoutMs();
const instructions: McpServerInstruction[] = [];
// merged-key -> provenance for the per-run recovery wrapper (#489).
const toolMeta: Record<string, ToolProvenance> = {};
// Shared Docmost write-class map (#489) — classifies a tool by its raw name.
// Loaded ONLY when at least one server is a TRUSTED internal Docmost server
// (see isInternalDocmostServer): for third-party servers the map is never
// applied (a name collision must not grant readOnly-retry), so we skip the
// dynamic ESM load entirely in that (currently universal) case.
const writeClassMap = servers.some((s) => this.isInternalDocmostServer(s))
? await this.getWriteClassMap()
: null;
// Per-server connect+tools result, still tagged with its server so the merge
// below can be applied in the SAME order as `servers` (see the parallel note).
@@ -327,11 +486,23 @@ export class McpClientsService {
// against names already merged from earlier servers, so no external
// tool is silently overwritten on collision. The returned count drives
// whether this server's prompt guidance is included (≥1 tool merged).
// #489 (review): the Docmost write-class map keys by DOCMOST tool names and
// may ONLY be trusted for a server KNOWN to be the internal Docmost MCP
// server. Every row here is an admin-configured THIRD-PARTY endpoint, so a
// third-party WRITE tool that happens to be named like a Docmost read
// (getPage, listPages, ...) must NOT inherit readOnly — that would auto-retry
// a mutation on a transport error (double-apply, the #435 class). Gate the
// map on the trust check; untrusted servers get writeClass=undefined -> the
// recovery wrapper treats them as writes and never auto-retries.
const trustWriteClass = this.isInternalDocmostServer(server);
const merged = this.mergeNamespaced(
tools,
result.guarded,
server.name,
server.id,
toolMeta,
i,
trustWriteClass ? writeClassMap : null,
);
outcomes.push({ name: server.name, ok: true });
// Include this server's guidance ONLY when it actually contributed at
@@ -353,6 +524,8 @@ export class McpClientsService {
clients,
outcomes,
instructions,
servers,
toolMeta,
expiresAt: Date.now() + CACHE_TTL_MS,
refCount: 0,
evicted: false,
@@ -379,18 +552,33 @@ export class McpClientsService {
picked: Record<string, Tool>,
serverName: string,
serverId: string,
toolMeta: Record<string, ToolProvenance>,
serverIndex: number,
// The Docmost write-class map, or `null` for an UNTRUSTED (third-party)
// server whose tools must all default to write (never auto-retried).
writeClassMap: Record<string, ToolWriteClass> | null,
): { count: number; prefix: string } {
let count = 0;
for (const [name, tool] of Object.entries(namespace(picked, serverName))) {
let key = name;
for (const { full, raw, tool } of namespace(picked, serverName)) {
let key = full;
if (key in target) {
const original = key;
key = disambiguate(name, serverId, (candidate) => candidate in target);
key = disambiguate(full, serverId, (candidate) => candidate in target);
this.logger.debug(
`External MCP tool name "${original}" collided; renamed to "${key}"`,
);
}
target[key] = tool;
// Record provenance so the per-run recovery wrapper (#489) can reconnect
// this tool's server and re-resolve it by its raw name. writeClass is set
// ONLY from a TRUSTED (internal-Docmost) map; for a third-party server the
// map is null -> writeClass stays undefined -> the wrapper treats the tool
// as a write and never auto-retries it (no double-apply on name collision).
toolMeta[key] = {
serverIndex,
rawName: raw,
writeClass: writeClassMap ? writeClassMap[raw] : undefined,
};
count += 1;
}
return { count, prefix: namespacePrefix(serverName) };
@@ -424,7 +612,10 @@ export class McpClientsService {
// Defense in depth: re-validate the actual request host on EVERY fetch
// AND pin the socket to a validated IP via the dispatcher's connect
// lookup, closing the DNS-rebinding TOCTOU between check and connect.
fetch: this.guardedFetch,
// #489: the SSE transport uses the raised-bodyTimeout dispatcher (idle
// between calls is legit); HTTP uses the tight one.
fetch:
transportType === 'sse' ? this.guardedFetchSse : this.guardedFetchHttp,
},
})) as unknown as McpClient;
return client;
@@ -505,6 +696,176 @@ export class McpClientsService {
}
}
/**
* Wrap one merged external tool with the per-run transport-recovery layer (#489).
*
* attempt 1 runs on the server's CURRENT binding (the cached client, or a client
* a sibling tool already reconnected this run). On a REAL transport error
* (undici/@ai-sdk socket/body-timeout shapes {@link isRetryableConnectError},
* NOT a mock) and ONLY for a declared readOnly tool, it reconnects the server
* and retries EXACTLY ONCE on the fresh client; a write is surfaced as an
* indeterminate error (it may have applied before the reset never
* blind-retried). A single per-call cap bounds BOTH attempts + the reconnect,
* and the run's abort signal is checked before the retry AND before minting a
* fresh connection (no connection is opened for a stopped run).
*/
private wrapWithTransportRecovery(
entry: CacheEntry,
meta: ToolProvenance,
template: Tool,
leaseSet: Closable[],
bindings: Map<number, ServerBinding>,
capMs: number,
): Tool {
const original = template.execute;
if (typeof original !== 'function') return template;
const service = this;
const { serverIndex, rawName, writeClass } = meta;
let binding = bindings.get(serverIndex);
if (!binding) {
binding = { current: null };
bindings.set(serverIndex, binding);
}
const boundBinding = binding;
const execute = async (args: unknown, options: ToolCallOptions) => {
// The per-call cap governs the WHOLE sequence (attempt1 + reconnect +
// attempt2). Compose it with the run's abort signal so a Stop or the cap
// ends any awaited call — @ai-sdk/mcp does not settle on abort, so we RACE.
const capController = new AbortController();
const capTimer = setTimeout(() => {
capController.abort(new Error(`MCP tool call timed out after ${capMs}ms`));
}, capMs);
capTimer.unref?.();
const runSignal = options?.abortSignal;
const composed = runSignal
? AbortSignal.any([runSignal, capController.signal])
: capController.signal;
const stopped = () => runSignal?.aborted === true || capController.signal.aborted;
const callOn = async (
exec: NonNullable<Tool['execute']>,
): Promise<unknown> => {
const aborted = new Promise<never>((_, reject) => {
const fail = () => reject(abortReason(composed));
if (composed.aborted) fail();
else composed.addEventListener('abort', fail, { once: true });
});
return Promise.race([exec(args, { ...options, abortSignal: composed }), aborted]);
};
const execFor = (
state: RecoveredServerState | null,
): NonNullable<Tool['execute']> | undefined =>
state ? (state.tools[rawName]?.execute as NonNullable<Tool['execute']>) : original;
try {
// Snapshot the target BEFORE the call so a swap by a concurrent call is
// detected by identity in the catch.
const attemptState = boundBinding.current;
const attemptExec = execFor(attemptState);
if (typeof attemptExec !== 'function') {
throw new Error(`external MCP tool "${rawName}" is not callable`);
}
try {
return await callOn(attemptExec);
} catch (err) {
// Never retry on a Stop or an exhausted cap.
if (stopped()) throw err;
// Only a genuine transport break is a recovery candidate.
if (!isRetryableConnectError(err)) throw err;
// A write tool is INDETERMINATE on a transport error (may have applied
// before the reset) — surface that; do NOT auto-retry (double-apply is
// the #435 incident class).
if (!isReadOnlyWriteClass(writeClass)) {
throw new Error(
`external MCP tool "${rawName}" hit a transport error and MAY have already ` +
`applied on the server — not retried automatically; verify state before ` +
`retrying. (${shortError(err)})`,
);
}
// Abort check BEFORE minting a fresh connection (no socket for a
// stopped run). LIMITATION (#489, LOW): the reconnect's own connect is
// bounded by CONNECT_TIMEOUT_MS but does NOT itself observe `composed`,
// so a Stop that lands DURING the handshake is only honored at the next
// `stopped()` gate (before the retry) — a bounded ≤5s late-abort window;
// the throwaway client is closed at turn-end regardless. Threading
// `composed` into the SHARED (CAS-deduped) reconnect is deliberately
// avoided: it would let the first caller's abort tear down a reconnect a
// concurrent still-live caller depends on.
if (stopped()) throw err;
// CAS-swap by IDENTITY: mint+swap only if nobody swapped since this
// call's snapshot; a losing concurrent call awaits the same reconnect
// and retries on the SAME fresh client.
let target: RecoveredServerState;
if (boundBinding.current === attemptState) {
if (!boundBinding.reconnecting) {
boundBinding.reconnecting = (async () => {
const server = entry.servers[serverIndex];
const fresh = await service.reconnectServer(server, capMs);
leaseSet.push(fresh.lease); // accumulate; released at turn-end
boundBinding.current = fresh.state;
return fresh.state;
})();
// Clear the in-flight marker once it settles (success or failure) so
// a LATER death of the new client can reconnect again.
void boundBinding.reconnecting.then(
() => (boundBinding.reconnecting = undefined),
() => (boundBinding.reconnecting = undefined),
);
}
target = await boundBinding.reconnecting;
} else {
target = boundBinding.current as RecoveredServerState;
}
// Abort check BEFORE the retry.
if (stopped()) throw err;
const retryExec = execFor(target);
if (typeof retryExec !== 'function') throw err;
return await callOn(retryExec);
}
} finally {
clearTimeout(capTimer);
}
};
return { ...template, execute } as unknown as Tool;
}
/**
* Reconnect ONE server for an in-run recovery (#489): open a fresh client and
* list+wrap its tools. The throwaway client is NOT cached it is owned by the
* RUN via the returned lease (closed at turn-end), independent of the shared
* cache entry (whose TTL rebuild heals future turns). On a failure the fresh
* client is closed so its socket never leaks.
*/
private async reconnectServer(
server: AiMcpServer,
capMs: number,
): Promise<{ state: RecoveredServerState; lease: Closable }> {
const client = await this.connectWithTimeout(server, CONNECT_TIMEOUT_MS);
let tools: Record<string, Tool>;
try {
const raw = await withTimeout(client.tools(), CONNECT_TIMEOUT_MS);
const allow = server.toolAllowlist;
const picked =
Array.isArray(allow) && allow.length > 0 ? pick(raw, allow) : raw;
tools = wrapToolsWithCallTimeout(picked, capMs);
} catch (err) {
void client.close().catch(() => undefined);
throw err;
}
let released = false;
const lease: Closable = {
close: async () => {
if (released) return;
released = true;
await client.close().catch(() => undefined);
},
};
return { state: { client, tools }, lease };
}
/** Mark an entry evicted; close its clients now if nothing is leasing them. */
private evict(entry: CacheEntry): void {
clearTimeout(entry.timer);
@@ -554,22 +915,21 @@ export function validateResolvedAddresses(addrs: readonly LookupAddress[]): {
* certificate validation still uses the real hostname (we never rewrite the URL
* to an IP literal).
*/
function buildPinnedDispatcher(): Agent {
// External-MCP traffic uses a DEDICATED, shorter silence timeout
function buildPinnedDispatcher(bodyTimeoutMs: number): Agent {
// External-MCP traffic uses a DEDICATED, shorter HEADERS silence timeout
// (`AI_MCP_STREAM_TIMEOUT_MS`, default 1 min) — deliberately tighter than the
// chat provider's 15-min `streamTimeoutMs()` — so a byte-silent/hung MCP
// upstream is broken in ~1 min instead of 15. We keep the keep-alive options
// from `streamingDispatcherOptions()` but OVERRIDE headers/body timeouts.
// Accepted trade-off: a legitimately long but byte-silent single tool call,
// and an SSE transport idling >1 min BETWEEN tool calls, are also cut here; the
// per-call total cap (wrapToolsWithCallTimeout, `AI_MCP_CALL_TIMEOUT_MS`) is the
// complementary guard for chatty-but-stuck calls that keep the socket warm yet
// never return.
const mcpSilenceMs = mcpStreamTimeoutMs();
// from `streamingDispatcherOptions()` but OVERRIDE the timeouts. `bodyTimeout`
// is passed in per-transport (#489): tight for HTTP (fresh request per call),
// raised for SSE (one long-lived body across calls — idle BETWEEN calls is
// legit). The per-call total cap (`AI_MCP_CALL_TIMEOUT_MS`) is the complementary
// guard for chatty-but-stuck calls that keep the socket warm yet never return.
const headersMs = mcpStreamTimeoutMs();
return new Agent({
...streamingDispatcherOptions(),
headersTimeout: mcpSilenceMs,
bodyTimeout: mcpSilenceMs,
headersTimeout: headersMs,
bodyTimeout: bodyTimeoutMs,
connect: {
lookup: (hostname, _options, callback) => {
// Always resolve ALL addresses ourselves; do not trust the caller's
@@ -669,18 +1029,22 @@ function pick(
function namespace(
tools: Record<string, Tool>,
serverName: string,
): Record<string, Tool> {
): Array<{ full: string; raw: string; tool: Tool }> {
const prefix = namespacePrefix(serverName);
const out: Record<string, Tool> = {};
const out: Array<{ full: string; raw: string; tool: Tool }> = [];
const taken: Record<string, true> = {};
for (const [name, t] of Object.entries(tools)) {
const safe = sanitizeName(name);
let full = capName(`${prefix}_${safe}`);
// Duplicate names within ONE server can still collide after sanitize/
// truncate — suffix-disambiguate so the second tool is not overwritten.
if (full in out) {
full = disambiguate(full, '', (candidate) => candidate in out);
if (full in taken) {
full = disambiguate(full, '', (candidate) => candidate in taken);
}
out[full] = t;
taken[full] = true;
// Keep the RAW (un-namespaced) name alongside the merged key so the per-run
// recovery wrapper (#489) can re-resolve the same tool on a fresh client.
out.push({ full, raw: name, tool: t });
}
return out;
}
@@ -804,6 +1168,69 @@ export function wrapToolWithCallTimeout(tool: Tool, ms: number): Tool {
return { ...tool, execute } as unknown as Tool;
}
/**
* undici / Node network error CODES that mean the connection broke (not an
* application-level error) a transient transport failure a readOnly call may
* safely retry after reconnecting. Matched against the REAL error shapes (#489):
* a socket reset surfaces as `TypeError: fetch failed` whose `.cause` is an
* undici `SocketError { code:'UND_ERR_SOCKET' }`; a body-timeout as
* `TypeError: terminated` whose `.cause` is `BodyTimeoutError`. Classifying by
* these real codes/names (not by mock errors) is essential a mock-shaped
* predicate would leave eviction silently dead in production while CI is green.
*/
const RETRYABLE_TRANSPORT_ERROR_CODES: ReadonlySet<string> = new Set([
'ECONNRESET',
'ECONNREFUSED',
'ECONNABORTED',
'EPIPE',
'ETIMEDOUT',
'EAI_AGAIN',
'ENETUNREACH',
'EHOSTUNREACH',
'UND_ERR_SOCKET',
'UND_ERR_CONNECT_TIMEOUT',
'UND_ERR_HEADERS_TIMEOUT',
'UND_ERR_BODY_TIMEOUT',
'UND_ERR_CLOSED',
'UND_ERR_DESTROYED',
]);
/** undici error CLASS names for the same transport-break conditions. */
const RETRYABLE_TRANSPORT_ERROR_NAMES: ReadonlySet<string> = new Set([
'SocketError',
'BodyTimeoutError',
'HeadersTimeoutError',
'ConnectTimeoutError',
'ClientClosedError',
'ClientDestroyedError',
]);
/**
* Whether `err` is a retryable TRANSPORT break (a broken socket / body timeout),
* classified by the REAL undici/@ai-sdk error shapes (#489). undici surfaces a
* reset as `TypeError('fetch failed'|'terminated')` with the real error in
* `.cause`, and @ai-sdk/mcp may wrap it again in an `MCPClientError` (cause
* chain), so we walk `.cause` (bounded depth) checking `.code` and `.name`. An
* app-level tool error (a 4xx, a validation failure) is NOT retryable and returns
* false only a connection-level failure heals with a reconnect.
*/
export function isRetryableConnectError(err: unknown, depth = 0): boolean {
if (!err || typeof err !== 'object' || depth > 6) return false;
const e = err as {
code?: unknown;
name?: unknown;
cause?: unknown;
};
if (typeof e.code === 'string' && RETRYABLE_TRANSPORT_ERROR_CODES.has(e.code)) {
return true;
}
if (typeof e.name === 'string' && RETRYABLE_TRANSPORT_ERROR_NAMES.has(e.name)) {
return true;
}
if (e.cause != null) return isRetryableConnectError(e.cause, depth + 1);
return false;
}
/** The signal's reason as an Error (informative thrown value on abort/timeout). */
function abortReason(signal: AbortSignal): Error {
const r = signal.reason;
@@ -9,9 +9,73 @@ import {
DEGENERATION_CHECK_STEP,
REPEATED_LINES_THRESHOLD,
MIN_PERIOD_REPEATS,
degenerationThresholds,
} from './output-degeneration';
import { AiChatService } from './ai-chat.service';
// Part A (#495 iter10): the detector thresholds are env-tunable. These drive the
// resolver against real repeat-count shapes and mutation-verify that the env
// override actually changes the trigger point (not a vacuous read).
describe('degeneration thresholds are env-configurable', () => {
const VARS = [
'AI_CHAT_DEGENERATION_REPEATED_LINES',
'AI_CHAT_DEGENERATION_PERIOD_MAX_LEN',
'AI_CHAT_DEGENERATION_PERIOD_MIN_REPEATS',
'AI_CHAT_DEGENERATION_CHECK_STEP',
];
const saved: Record<string, string | undefined> = {};
beforeEach(() => {
for (const v of VARS) saved[v] = process.env[v];
});
afterEach(() => {
for (const v of VARS) {
if (saved[v] === undefined) delete process.env[v];
else process.env[v] = saved[v];
}
});
it('defaults to the compiled constants when unset', () => {
for (const v of VARS) delete process.env[v];
expect(degenerationThresholds()).toEqual({
repeatedLines: REPEATED_LINES_THRESHOLD,
maxPeriodLen: 150,
minPeriodRepeats: MIN_PERIOD_REPEATS,
checkStep: DEGENERATION_CHECK_STEP,
});
});
it('falls back to the default on blank / invalid / non-positive values', () => {
for (const bad of ['', ' ', 'abc', '0', '-3', '1.5']) {
process.env.AI_CHAT_DEGENERATION_REPEATED_LINES = bad;
// '1.5' floors to 1 (still ≥1, valid); every other bad value → default.
const expected = bad === '1.5' ? 1 : REPEATED_LINES_THRESHOLD;
expect(degenerationThresholds().repeatedLines).toBe(expected);
}
});
it('a RAISED check-step suppresses a burst the default would have flagged', () => {
// A ~3.3KB periodic burst is periodic-degenerate, but shouldCheckDegeneration
// is the throttle gate. Default checkStep=2000 arms on it; raising the step
// above the burst size means the throttle never re-fires for it.
const burstLen = 'loadTools.\n'.repeat(300).length; // ~3300
delete process.env.AI_CHAT_DEGENERATION_CHECK_STEP;
expect(shouldCheckDegeneration(burstLen, 0)).toBe(true); // default 2000
process.env.AI_CHAT_DEGENERATION_CHECK_STEP = String(burstLen + 1);
expect(shouldCheckDegeneration(burstLen, 0)).toBe(false); // raised gate
});
it('a LOWERED repeated-lines threshold trips on a shorter identical-line run', () => {
// 8 identical lines: below the default 25 (rule 1) and below the periodic
// rule's 20 repeats — so isDegenerateOutput is false by default.
const shortRun = 'x\n'.repeat(8);
delete process.env.AI_CHAT_DEGENERATION_REPEATED_LINES;
expect(isDegenerateOutput(shortRun)).toBe(false);
// Lower rule 1 to 5 → the 8-line run now trips.
process.env.AI_CHAT_DEGENERATION_REPEATED_LINES = '5';
expect(isDegenerateOutput(shortRun)).toBe(true);
});
});
// Mock ONLY streamText so we can capture the onChunk/onStepFinish callbacks the
// service registers and drive them by hand; every other `ai` export the service
// uses (convertToModelMessages, stepCountIs, …) stays real.
@@ -23,6 +23,54 @@ export const MAX_PERIOD_LEN = 150;
/** Rule 2: minimum number of consecutive block repeats to trigger. */
export const MIN_PERIOD_REPEATS = 20;
/**
* Read a positive-integer threshold from an env var, falling back to `fallback`
* on unset/blank/invalid/non-positive. Mirrors the `AI_STREAM_PRE_RESPONSE_RETRIES`
* resolver in `ai-streaming-fetch.ts`: read the RAW string first so a blank value
* is treated as "unset" ( fallback) rather than coercing to 0. Thresholds must
* stay 1 a 0/negative would make the detector fire on any text (or never), so
* a bad value degrades to the safe compiled default instead. Env-tunable so an
* operator can retune the anti-babble guard (#444) without a redeploy, following
* the `AI_CHAT_FINAL_STEP_LOCKDOWN` toggle convention.
*/
function envThreshold(name: string, fallback: number): number {
const rawStr = process.env[name];
if (rawStr === undefined || rawStr.trim() === '') return fallback;
const raw = Number(rawStr);
return Number.isFinite(raw) && raw >= 1 ? Math.floor(raw) : fallback;
}
/**
* Resolve the degeneration-detector thresholds from the environment, each
* defaulting to the compiled constant above. Read fresh per call (not cached at
* import) so a test or a runtime env change takes effect deterministically.
*/
export function degenerationThresholds(): {
repeatedLines: number;
maxPeriodLen: number;
minPeriodRepeats: number;
checkStep: number;
} {
return {
repeatedLines: envThreshold(
'AI_CHAT_DEGENERATION_REPEATED_LINES',
REPEATED_LINES_THRESHOLD,
),
maxPeriodLen: envThreshold(
'AI_CHAT_DEGENERATION_PERIOD_MAX_LEN',
MAX_PERIOD_LEN,
),
minPeriodRepeats: envThreshold(
'AI_CHAT_DEGENERATION_PERIOD_MIN_REPEATS',
MIN_PERIOD_REPEATS,
),
checkStep: envThreshold(
'AI_CHAT_DEGENERATION_CHECK_STEP',
DEGENERATION_CHECK_STEP,
),
};
}
/**
* Rule 1 `REPEATED_LINES_THRESHOLD` consecutive IDENTICAL non-empty lines at
* the tail. Catches the classic newline-delimited loop ("loadTools.\n" ×N).
@@ -128,7 +176,11 @@ export function hasPeriodicTail(
* Pure the caller owns the abort side effect.
*/
export function isDegenerateOutput(text: string): boolean {
return hasRepeatedLineRun(text) || hasPeriodicTail(text);
const cfg = degenerationThresholds();
return (
hasRepeatedLineRun(text, cfg.repeatedLines) ||
hasPeriodicTail(text, cfg.maxPeriodLen, cfg.minPeriodRepeats)
);
}
/**
@@ -154,7 +206,7 @@ export function shouldCheckDegeneration(
textLen: number,
lastCheckLen: number,
): boolean {
return textLen - lastCheckLen >= DEGENERATION_CHECK_STEP;
return textLen - lastCheckLen >= degenerationThresholds().checkStep;
}
/**
@@ -307,6 +307,10 @@ export class PublicShareChatService {
system,
messages: modelMessages,
tools,
// Pin the AI SDK per-request retry budget explicitly (matches the SDK
// default of 2). Connection arithmetic: (1 + maxRetries) × (1 +
// AI_STREAM_PRE_RESPONSE_RETRIES) worst-case connects per turn.
maxRetries: 2,
// Bound the agent loop for anonymous callers.
stopWhen: stepCountIs(5),
// Cap per-request output so one anonymous call cannot run up the provider
@@ -0,0 +1,160 @@
import {
wrapInAppToolWithCap,
inAppToolCallCapMs,
type ToolAbortSignalSink,
} from './ai-chat-tools.service';
import type { Tool, ToolCallOptions } from 'ai';
/**
* #487 commit 1 in-app tool race-on-abort + safe-points + per-call cap.
*
* Tests assert the HONEST observable property the spec names "after Stop, NO
* new HTTP/WS call STARTS; an already-started single call may take either
* outcome" against the REAL wrapper mechanism (the composite abort signal it
* publishes on the client + the RACE it runs), NOT a timing-dependent proxy like
* "the write didn't land".
*/
// A minimal stand-in for the client's `toolAbortSignal` field. In production the
// wrapper publishes the composite here and the client's paginateAll /
// mutatePageContent safe-points read it; the fake "tool" below reads it the same
// way, so this exercises the real contract without a live DB / collab socket.
class FakeClient implements ToolAbortSignalSink {
private signal: AbortSignal | null = null;
setToolAbortSignal(signal: AbortSignal | null): void {
this.signal = signal;
}
getToolAbortSignal(): AbortSignal | null {
return this.signal;
}
}
// A ToolCallOptions with just the field the wrapper reads (abortSignal). The AI
// SDK passes a fuller object; the wrapper only spreads it and reads abortSignal.
const opts = (abortSignal?: AbortSignal): ToolCallOptions =>
({ toolCallId: 't1', messages: [], abortSignal }) as unknown as ToolCallOptions;
const tick = (ms = 5) => new Promise((r) => setTimeout(r, ms));
describe('#487 wrapInAppToolWithCap — race-on-abort + safe-points', () => {
it('after Stop, no NEW simulated call starts (multi-call tool)', async () => {
const client = new FakeClient();
const started: number[] = [];
// A multi-call tool that mirrors paginateAll: it consults the client signal
// at a safe-point BEFORE starting each simulated network call.
const multiCall: Tool = {
execute: (async (_args: unknown) => {
for (let i = 0; i < 6; i++) {
// Safe-point: exactly what paginateAll / mutatePageContent do.
client.getToolAbortSignal()?.throwIfAborted();
started.push(i);
await tick(10);
}
return 'done';
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(multiCall, client, 10_000);
const ac = new AbortController();
const call = (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
// Let one or two calls start, then Stop.
await tick(12);
ac.abort(new Error('user stop'));
await expect(call).rejects.toThrow(); // wrapper rejects promptly
const startedAtStop = started.length;
// Give the abandoned loser ample time; its next safe-point must throw because
// the (aborted) composite is still published on the client.
await tick(60);
expect(started.length).toBe(startedAtStop);
// It must NOT have run the whole sequence (that would mean Stop did nothing).
expect(started.length).toBeLessThan(6);
});
it('rejects immediately on Stop even if the call never settles (discard loser)', async () => {
const client = new FakeClient();
let settled = false;
const hang: Tool = {
execute: (async () => {
await new Promise(() => undefined); // never resolves
settled = true;
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(hang, client, 10_000);
const ac = new AbortController();
const call = (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
await tick(5);
ac.abort();
await expect(call).rejects.toThrow();
expect(settled).toBe(false);
});
it('per-call cap rejects a hung call with no Stop signal', async () => {
const client = new FakeClient();
const hang: Tool = {
execute: (async () => {
await new Promise(() => undefined);
}) as unknown as Tool['execute'],
} as Tool;
// Tiny cap; no options.abortSignal at all (composite = cap only).
const wrapped = wrapInAppToolWithCap(hang, client, 20);
const start = Date.now();
await expect(
(wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>)(
{},
opts(undefined),
),
).rejects.toThrow(/per-call cap/);
expect(Date.now() - start).toBeLessThan(2000);
});
it('publishes a composite signal on the client for the duration of the call', async () => {
const client = new FakeClient();
let seenDuringCall: AbortSignal | null = null;
const probe: Tool = {
execute: (async () => {
seenDuringCall = client.getToolAbortSignal();
return 'ok';
}) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(probe, client, 10_000);
const ac = new AbortController();
await (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(ac.signal));
expect(seenDuringCall).not.toBeNull();
// The published composite must reflect the turn's Stop signal.
ac.abort();
expect((seenDuringCall as unknown as AbortSignal).aborted).toBe(true);
});
it('a completed call returns its raw result unchanged', async () => {
const client = new FakeClient();
const ok: Tool = {
execute: (async () => ({ items: [1, 2, 3] })) as unknown as Tool['execute'],
} as Tool;
const wrapped = wrapInAppToolWithCap(ok, client, 10_000);
const res = await (
wrapped.execute as (a: unknown, o: ToolCallOptions) => Promise<unknown>
)({}, opts(new AbortController().signal));
expect(res).toEqual({ items: [1, 2, 3] });
});
it('cap is env-tunable with a 2-minute default', () => {
const prev = process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
delete process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
expect(inAppToolCallCapMs()).toBe(120_000);
process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = '5000';
expect(inAppToolCallCapMs()).toBe(5000);
process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = 'not-a-number';
expect(inAppToolCallCapMs()).toBe(120_000);
if (prev === undefined) delete process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS;
else process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS = prev;
});
});
@@ -1,5 +1,5 @@
import { Injectable, Logger } from '@nestjs/common';
import { tool, type Tool } from 'ai';
import { tool, type Tool, type ToolCallOptions } from 'ai';
import { z } from 'zod';
import { User } from '@docmost/db/types/entity.types';
import { TokenService } from '../../auth/services/token.service';
@@ -159,6 +159,129 @@ function __assertClientCallContract(client: DocmostClientLike): void {
* existing service-account `/mcp` path already calls loopback successfully, so
* this works for single-workspace self-host.
*/
/**
* #487: wall-clock cap for a SINGLE in-app tool call, env-tunable via
* `AI_CHAT_INAPP_TOOL_CALL_CAP_MS`. Bounds a read tool that would otherwise
* paginate for minutes and a content write whose collab commit hangs, and is the
* per-call CAP half of the composite abort signal every in-app tool is wrapped
* with (the other half is the turn's Stop signal). Default 2 minutes: generous
* for a legitimate long read/write, tight enough that a stuck call cannot pin the
* turn. The reconcile staleness floor (#487 commit 4) is derived as
* max(2 x this cap, 15 min), so keep this well under that.
*/
export function inAppToolCallCapMs(): number {
const raw = Number(process.env.AI_CHAT_INAPP_TOOL_CALL_CAP_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 120_000;
}
/** #487: the composite signal's reason as an Error (informative thrown value). */
function inAppAbortReason(signal: AbortSignal): Error {
const r = signal.reason;
return r instanceof Error
? r
: new Error(typeof r === 'string' ? r : 'In-app tool call aborted');
}
/**
* The client surface {@link wrapInAppToolWithCap} drives (#487). Both methods are
* OPTIONAL: the real loopback DocmostClient implements them (so a Stop/cap reaches
* its pagination / pre-commit safe-points), but a client that omits them still
* gets the OUTER guarantee the race rejects on abort regardless. This keeps the
* wrapper decoupled from the exact client shape (unit-test doubles need not stub
* the plumbing).
*/
export interface ToolAbortSignalSink {
setToolAbortSignal?(signal: AbortSignal | null): void;
getToolAbortSignal?(): AbortSignal | null;
}
/**
* #487: wrap an in-app tool so a Stop (the turn's `options.abortSignal`) OR the
* per-call wall-clock cap REJECTS the call immediately, and so that SAME
* composite signal reaches the client's pagination / pre-commit safe-points (via
* `client.setToolAbortSignal`) making a Stop stop the NEXT HTTP/WS call from
* starting.
*
* Reuses the RACE pattern of `wrapToolWithCallTimeout` (mcp-clients.service.ts):
* the call is raced against the composite signal, so on abort we reject in the
* SAME tick and DISCARD the loser promise. Its network / collab teardown latency
* therefore never blocks the turn the supersede timeout W=10s (#487 commit 3)
* relies on this abort->settle latency being milliseconds, not a socket teardown.
* Awaiting the client's own signal-into-write path alone would NOT satisfy this
* (the loser could still be tearing down a collab socket).
*
* The composite is SET on the client at entry and deliberately NOT restored on
* unwind: after this wrapper rejects on abort, the ABANDONED loser promise keeps
* running, and its safe-points read the client field leaving the (aborted)
* composite there is exactly what makes the loser's NEXT call throw and stop. The
* next in-app tool call overwrites the field with its own fresh composite before
* any of its safe-points run, so a stale settled signal never leaks forward.
* SINGLE-WRITER by phase-1 assumption see DocmostClientContext.toolAbortSignal
* for the parallel-call caveat (#487).
*
* KNOWN LIMITATION (#487): a write tool that issues SEVERAL sequential collab
* commits can be aborted BETWEEN commits, leaving a partially-applied operation.
* Cancel guarantees "no NEW call starts", NOT "the write didn't land".
*/
export function wrapInAppToolWithCap(
toolDef: Tool,
client: ToolAbortSignalSink,
capMs: number,
): Tool {
const original = toolDef.execute;
if (typeof original !== 'function') return toolDef;
const execute = async (args: unknown, options: ToolCallOptions) => {
const capController = new AbortController();
const timer = setTimeout(() => {
capController.abort(
new Error(`In-app tool call exceeded the ${capMs}ms per-call cap`),
);
}, capMs);
timer.unref?.();
const composite = options?.abortSignal
? AbortSignal.any([options.abortSignal, capController.signal])
: capController.signal;
// Reject the MOMENT the composite fires, independent of whether `original`
// ever settles (a hung collab write / read would otherwise pin the turn). The
// losing `original` is left pending; Promise.race attaches a rejection
// handler to both inputs so a late rejection is never unhandled.
const aborted = new Promise<never>((_, reject) => {
const fail = () => reject(inAppAbortReason(composite));
if (composite.aborted) fail();
else composite.addEventListener('abort', fail, { once: true });
});
// Publish the composite so the client's pagination / pre-commit safe-points
// observe it (see the "not restored on unwind" rationale above). Guarded: a
// client without the plumbing still gets the OUTER race guarantee below.
client.setToolAbortSignal?.(composite);
try {
return await Promise.race([
(original as (a: unknown, o: ToolCallOptions) => Promise<unknown>)(
args,
{ ...options, abortSignal: composite },
),
aborted,
]);
} finally {
clearTimeout(timer);
}
};
return { ...toolDef, execute } as unknown as Tool;
}
/** #487: apply {@link wrapInAppToolWithCap} to every tool in a set. */
export function wrapInAppToolsWithCap(
tools: Record<string, Tool>,
client: ToolAbortSignalSink,
capMs: number,
): Record<string, Tool> {
const out: Record<string, Tool> = {};
for (const [name, t] of Object.entries(tools)) {
out[name] = wrapInAppToolWithCap(t, client, capMs);
}
return out;
}
@Injectable()
export class AiChatToolsService {
private readonly logger = new Logger(AiChatToolsService.name);
@@ -186,7 +309,12 @@ export class AiChatToolsService {
sessionId: string,
workspaceId: string,
aiChatId: string,
): Promise<DocmostClientLike> {
// #487: the returned client also carries the tool-cancellation plumbing
// (setToolAbortSignal/getToolAbortSignal). These are host plumbing, NOT part
// of the tool-execute surface (DocmostClientMethod), so they are surfaced here
// as an intersection rather than by widening that Pick — keeping the
// positional-call drift-guard (#446) scoped to the actual tool methods.
): Promise<DocmostClientLike & ToolAbortSignalSink> {
const apiUrl =
process.env.MCP_DOCMOST_API_URL ||
`http://127.0.0.1:${process.env.PORT || 3000}/api`;
@@ -630,7 +758,15 @@ export class AiChatToolsService {
// dependency and reuses the CASL enforcement already on `client`. When the
// loaded package predates #417 (factory undefined) or the loader is mocked in
// a unit test, signalling is a pure no-op and results are byte-identical.
if (!createCommentSignalTracker) return tools;
// #487: wrap every in-app tool with the race-on-abort + per-call cap guard so
// a Stop / cap rejects immediately AND reaches the client's write/pagination
// safe-points. Applied as the OUTERMOST wrapper (over the comment-signal
// wrapper below) so the race governs the whole call. The client carries the
// per-call composite signal via setToolAbortSignal.
const capMs = inAppToolCallCapMs();
if (!createCommentSignalTracker) {
return wrapInAppToolsWithCap(tools, client, capMs);
}
const tracker = createCommentSignalTracker({
probe: async (pageId: string, sinceMs: number) => {
@@ -659,7 +795,11 @@ export class AiChatToolsService {
},
});
return wrapToolsWithCommentSignal(tools, tracker);
return wrapInAppToolsWithCap(
wrapToolsWithCommentSignal(tools, tracker),
client,
capMs,
);
}
}
@@ -17,9 +17,10 @@ import { MovePageDto } from './move-page.dto';
// a valid ordering key the server itself generated would be refused on move.
//
// The tests below assert the CORRECT contract: any key the generator can produce
// must satisfy the DTO. The genuinely-failing case is marked `test.failing` so the
// suite stays green while locking the bug; it flips red (alerting us) once the DTO
// bounds are widened to cover the generator's real range.
// must satisfy the DTO. FIXED (#495 item 9): the DTO now validates `position` by
// CHARSET ([0-9A-Za-z], the generator's base-62 alphabet) instead of the wrong
// @MaxLength(12) length bound, so dense between-inserts are accepted; the former
// `test.failing` bug-lock is now a passing assertion.
function constraintErrors(position: unknown) {
const dto = plainToInstance(MovePageDto, {
@@ -47,24 +48,33 @@ describe('MovePageDto.position vs generateJitteredKeyBetween parity', () => {
expect(hasError(errors, 'position')).toBe(false);
});
// BUG LOCK: dense between-inserts produce keys longer than 12 chars, which
// MaxLength(12) rejects even though they are valid ordering keys. This SHOULD
// pass; it currently fails. Flips green when the DTO bound is fixed.
test.failing(
'accepts dense between-inserted keys (currently rejected by MaxLength(12))',
async () => {
let lo = generateJitteredKeyBetween(null, null);
let hi = generateJitteredKeyBetween(lo, null);
// Repeatedly insert just above `lo`, shrinking the gap so the key grows.
let longest = lo;
for (let i = 0; i < 40; i++) {
const mid = generateJitteredKeyBetween(lo, hi);
if (mid.length > longest.length) longest = mid;
hi = mid;
}
expect(longest.length).toBeGreaterThan(12); // sanity: we produced a long key
const errors = await constraintErrors(longest);
expect(hasError(errors, 'position')).toBe(false);
},
);
// FIXED: dense between-inserts produce keys longer than 12 chars, which the old
// MaxLength(12) rejected even though they are valid ordering keys. Now accepted.
it('accepts dense between-inserted keys longer than 12 chars', async () => {
let lo = generateJitteredKeyBetween(null, null);
let hi = generateJitteredKeyBetween(lo, null);
// Repeatedly insert just above `lo`, shrinking the gap so the key grows. The
// generator is JITTERED (random), so use enough iterations that the longest
// key reliably clears the old 12-char bound: measured min-over-50-trials is
// ~11 at 40 iterations (flaky) but ~36 at 200 (robust margin).
let longest = lo;
for (let i = 0; i < 200; i++) {
const mid = generateJitteredKeyBetween(lo, hi);
if (mid.length > longest.length) longest = mid;
hi = mid;
}
expect(longest.length).toBeGreaterThan(12); // sanity: we produced a long key
const errors = await constraintErrors(longest);
expect(hasError(errors, 'position')).toBe(false);
});
// The charset guard replaces the length bound: reject anything outside the
// generator's [0-9A-Za-z] alphabet (control chars, separators, injection) and
// the empty string, while still accepting every real key.
it('rejects a position with characters outside the fractional-index alphabet', async () => {
for (const bad of ['a0/b', 'a b', 'a\n0', 'a.b', '', "a';--"]) {
const errors = await constraintErrors(bad);
expect(hasError(errors, 'position')).toBe(true);
}
});
});
+13 -3
View File
@@ -1,8 +1,8 @@
import {
IsString,
IsOptional,
MinLength,
MaxLength,
Matches,
IsNotEmpty,
} from 'class-validator';
@@ -10,9 +10,19 @@ export class MovePageDto {
@IsString()
pageId: string;
// `position` is a fractional-indexing key from `generateJitteredKeyBetween`
// (the SAME generator page.service uses). Validate by CHARSET, not length: the
// generator's default base-62 alphabet is [0-9A-Za-z], and DENSE between-inserts
// legitimately grow a key well past a dozen chars (measured >40), so the old
// @MinLength(5)/@MaxLength(12) bounds wrongly 400'd valid ordering keys the
// server itself produced (Gitea #139 item 6). The charset regex rejects control
// chars / separators / injection, and a generous MaxLength stays only as a
// DoS guard — far above any realistic key, so it never rejects a real move.
@IsString()
@MinLength(5)
@MaxLength(12)
@Matches(/^[0-9A-Za-z]+$/, {
message: 'position must be a fractional-index key ([0-9A-Za-z])',
})
@MaxLength(256)
position: string;
@IsOptional()
@@ -0,0 +1,49 @@
import 'reflect-metadata';
import { plainToInstance } from 'class-transformer';
import { validate } from 'class-validator';
import { PageIdDto } from './page.dto';
// #435: PageIdDto.pageId carries a page's DOUBLE identity (internal UUID OR
// public 10-char slugId), both as bare strings. The DTO must accept exactly
// those two FORMATS and reject a malformed / swapped identity at the boundary.
async function pageIdErrors(pageId: unknown) {
const dto = plainToInstance(PageIdDto, { pageId });
const errors = await validate(dto as object);
return errors.some((e) => e.property === 'pageId');
}
const UUID = '019f499a-9f8c-7d68-b7be-ce100d7c6c56';
const SLUG = 'aB3xQ7kR2p';
describe('PageIdDto pageId format validation', () => {
it('accepts a canonical page UUID', async () => {
expect(await pageIdErrors(UUID)).toBe(false);
});
it('accepts a 10-char slugId', async () => {
expect(await pageIdErrors(SLUG)).toBe(false);
});
it('rejects a truncated / wrong-length slug', async () => {
expect(await pageIdErrors('aB3xQ7kR2')).toBe(true); // 9 chars
expect(await pageIdErrors('aB3xQ7kR2pX')).toBe(true); // 11 chars
});
it('rejects a slug with an illegal character', async () => {
expect(await pageIdErrors('aB3xQ7kR2!')).toBe(true);
});
it('rejects a full URL / path-shaped identity (not the bare id)', async () => {
expect(await pageIdErrors(`my-page-title-${SLUG}`)).toBe(true);
expect(await pageIdErrors(`https://x/p/${SLUG}`)).toBe(true);
});
it('rejects a malformed UUID', async () => {
expect(await pageIdErrors('019f499a-9f8c-7d68-b7be')).toBe(true);
expect(await pageIdErrors('not-a-uuid-at-all-really')).toBe(true);
});
it('rejects an empty string', async () => {
expect(await pageIdErrors('')).toBe(true);
});
});
@@ -0,0 +1,39 @@
import { applyDecorators } from '@nestjs/common';
import { Matches, ValidationOptions } from 'class-validator';
/**
* A page identity at the API boundary is EITHER the internal page UUID or the
* public 10-char slugId (page.repo.findById matches a non-UUID input as a
* slugId). Both arrive as bare strings, which is exactly how the two got swapped
* silently (incident family #435). This regex pins the two accepted FORMATS so a
* malformed / cross-wired identity (a truncated slug, a full URL, an email, an
* id from another entity kind that isn't even shaped like either) is rejected at
* the boundary instead of falling through to a confusing 404.
*
* - UUID: canonical 8-4-4-4-12 hex (version-agnostic page ids are UUIDv7,
* so only the shape/length is enforced, matching the MCP's UUID_RE
* and the server's isValidUUID acceptance).
* - slugId: exactly 10 chars over [0-9A-Za-z] (nanoid `generateSlugId`).
*
* The two are disjoint (a UUID is 36 chars WITH dashes, a slugId 10 chars
* WITHOUT), so a value can only satisfy one branch.
*/
export const PAGE_ID_OR_SLUG_ID_REGEX =
/^(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9A-Za-z]{10})$/i;
/**
* Validate that a DTO string field is a well-formed page identity (page UUID OR
* 10-char slugId). Composed decorator so the same format rule is applied
* consistently wherever a DTO accepts a `pageId` that may be either form.
*/
export function IsPageIdOrSlugId(
validationOptions?: ValidationOptions,
): PropertyDecorator {
return applyDecorators(
Matches(PAGE_ID_OR_SLUG_ID_REGEX, {
message:
'pageId must be a page UUID or a 10-character slugId',
...validationOptions,
}),
);
}
@@ -9,10 +9,16 @@ import {
import { Transform } from 'class-transformer';
import { ContentFormat } from './create-page.dto';
import { IsPageIdOrSlugId } from './page-identity.validator';
export class PageIdDto {
@IsString()
@IsNotEmpty()
// Format-validate the double identity (#435): accept only a page UUID or a
// 10-char slugId so a malformed / swapped identity is rejected at the boundary
// rather than passed to the repo as a bare string. Base for PageInfoDto,
// DeletePageDto, BacklinksListDto, AddLabelsDto/RemoveLabelDto, etc.
@IsPageIdOrSlugId()
pageId: string;
}
@@ -1,19 +1,38 @@
import { TemporaryNoteCleanupService } from '../temporary-note-cleanup.service';
/**
* Chainable Kysely stub that records every `.where(...)` call so the test can
* assert the sweep only selects armed, expired, not-yet-trashed notes. The
* terminal `.execute()` resolves the configured expired rows (the batch SELECT);
* `.executeTakeFirst()` resolves the per-row deadline re-read done just before
* each `removePage`. By default the re-read reports the note as still armed and
* still expired (epoch deadline < now), so the sweep proceeds to delete it;
* tests override `reReadFirst` to simulate a concurrent "Make permanent".
* Chainable Kysely stub for the temporary-note sweep.
*
* `this.db` serves the non-locking candidate SELECT (selectFrom/select/where/
* limit/execute -> the configured expired rows) AND `.transaction().execute(cb)`,
* which runs `cb` with a separate `trx` builder. The `trx` builder serves the
* per-row LOCKED re-check (selectFrom/select/where/forUpdate/skipLocked/
* executeTakeFirst). `lockedRows` drives what that locked re-check returns per
* candidate an id/creator/workspace row means "still expired, delete it";
* `undefined` means the predicate no longer matched (made permanent / re-armed /
* already trashed) or the row was SKIP-LOCKED by another worker, so it is skipped.
*/
function makeDbStub(expiredRows: any[]) {
function makeDbStub(expiredRows: any[], lockedRows?: any[]) {
const whereCalls: any[][] = [];
const reReadFirst = jest
.fn()
.mockResolvedValue({ temporaryExpiresAt: new Date(0), deletedAt: null });
const locked = [
...(lockedRows ??
expiredRows.map((r) => ({
id: r.id,
creatorId: r.creatorId,
workspaceId: r.workspaceId,
}))),
];
const lockedTakeFirst = jest.fn(() => Promise.resolve(locked.shift()));
const forUpdate = jest.fn(() => trxBuilder);
const skipLocked = jest.fn(() => trxBuilder);
const trxBuilder: any = {
selectFrom: jest.fn(() => trxBuilder),
select: jest.fn(() => trxBuilder),
where: jest.fn(() => trxBuilder),
forUpdate,
skipLocked,
executeTakeFirst: lockedTakeFirst,
};
const builder: any = {
selectFrom: jest.fn(() => builder),
select: jest.fn(() => builder),
@@ -23,9 +42,11 @@ function makeDbStub(expiredRows: any[]) {
}),
limit: jest.fn(() => builder),
execute: jest.fn().mockResolvedValue(expiredRows),
executeTakeFirst: reReadFirst,
transaction: jest.fn(() => ({
execute: (cb: (trx: any) => Promise<any>) => Promise.resolve(cb(trxBuilder)),
})),
};
return { builder, whereCalls, reReadFirst };
return { builder, whereCalls, lockedTakeFirst, forUpdate, skipLocked };
}
describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
@@ -52,20 +73,36 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
expect(builder.limit.mock.calls[0][0]).toBeGreaterThan(0);
});
it('soft-deletes each expired note via removePage, attributed to its creator', async () => {
it('soft-deletes each expired note via removePage under a row lock, attributed to its creator', async () => {
const expired = [
{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' },
{ id: 'p2', creatorId: 'u2', workspaceId: 'w1' },
];
const { builder } = makeDbStub(expired);
const { builder, forUpdate, skipLocked } = makeDbStub(expired);
const pageRepo = { removePage: jest.fn().mockResolvedValue(undefined) } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).toHaveBeenCalledTimes(2);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(1, 'p1', 'u1', 'w1');
expect(pageRepo.removePage).toHaveBeenNthCalledWith(2, 'p2', 'u2', 'w1');
// The 4th arg is the locking transaction — the delete runs inside it.
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
1,
'p1',
'u1',
'w1',
expect.anything(),
);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
2,
'p2',
'u2',
'w1',
expect.anything(),
);
// The re-check acquired a FOR UPDATE SKIP LOCKED lock (once per candidate).
expect(forUpdate).toHaveBeenCalledTimes(2);
expect(skipLocked).toHaveBeenCalledTimes(2);
});
it('continues past a failing note (one bad removePage does not abort the sweep)', async () => {
@@ -86,60 +123,29 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
service.sweepExpiredTemporaryNotes(),
).resolves.toBeUndefined();
expect(pageRepo.removePage).toHaveBeenCalledTimes(2);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(2, 'good', 'u2', 'w1');
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
2,
'good',
'u2',
'w1',
expect.anything(),
);
});
it('does NOT trash a note made permanent in the race window', async () => {
// The batch SELECT saw the note as expired, but before its turn in the loop
// the user clicked "Make permanent" (temporary_expires_at -> null). The
// deadline re-read must catch this and skip the delete so the keep wins.
it('does NOT trash a note made permanent / re-armed / already trashed (locked re-check returns nothing)', async () => {
// The batch SELECT saw the note as expired, but by the time the LOCKED
// re-check runs the row no longer matches the still-armed+expired+not-trashed
// predicate (make-permanent, re-arm to a future deadline, or already trashed),
// OR another worker holds the row (SKIP LOCKED). In every case the locked
// SELECT returns nothing and the delete is skipped so the keep/other worker wins.
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: null,
deletedAt: null,
});
const { builder, lockedTakeFirst } = makeDbStub(expired, [undefined]);
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(reReadFirst).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('skips a note already trashed since the batch SELECT', async () => {
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: new Date(0),
deletedAt: new Date(),
});
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('does NOT trash a note re-armed to a future deadline in the race window', async () => {
// The batch SELECT saw the note as expired, but before its turn in the loop
// the user disarmed it and re-armed it to a fresh, still-future deadline
// (temporary_expires_at -> now + 1h). The deadline re-read must catch that
// the note is no longer expired and skip the delete so the keep wins.
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: new Date(Date.now() + 60 * 60 * 1000),
deletedAt: null,
});
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(reReadFirst).toHaveBeenCalledTimes(1);
expect(lockedTakeFirst).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
@@ -151,4 +157,31 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('sweeps once on application bootstrap (catches notes expired during downtime)', async () => {
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder } = makeDbStub(expired);
const pageRepo = { removePage: jest.fn().mockResolvedValue(undefined) } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.onApplicationBootstrap();
expect(pageRepo.removePage).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).toHaveBeenCalledWith(
'p1',
'u1',
'w1',
expect.anything(),
);
});
it('a startup-sweep failure never blocks application boot', async () => {
const { builder } = makeDbStub([]);
// Make the candidate SELECT throw to simulate a boot-time DB hiccup.
builder.execute.mockRejectedValueOnce(new Error('db not ready'));
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await expect(service.onApplicationBootstrap()).resolves.toBeUndefined();
});
});
@@ -1,8 +1,13 @@
import { Injectable, Logger } from '@nestjs/common';
import {
Injectable,
Logger,
OnApplicationBootstrap,
} from '@nestjs/common';
import { Interval } from '@nestjs/schedule';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { executeTx } from '@docmost/db/utils';
/**
* Background sweeper for temporary notes ("structure or die"). A note whose
@@ -11,7 +16,7 @@ import { PageRepo } from '@docmost/db/repos/page/page.repo';
* TrashCleanupService; `@nestjs/schedule` is already enabled globally.
*/
@Injectable()
export class TemporaryNoteCleanupService {
export class TemporaryNoteCleanupService implements OnApplicationBootstrap {
private readonly logger = new Logger(TemporaryNoteCleanupService.name);
// Cap a single sweep so a large backlog (e.g. many notes created during
@@ -24,6 +29,20 @@ export class TemporaryNoteCleanupService {
private readonly pageRepo: PageRepo,
) {}
// Sweep once at startup so notes that expired during downtime are trashed
// right away instead of waiting up to an hour for the first @Interval tick.
// Best-effort: never let a startup-sweep failure block application boot.
async onApplicationBootstrap() {
try {
await this.sweepExpiredTemporaryNotes();
} catch (error) {
this.logger.error(
'Temporary-note startup sweep failed',
error instanceof Error ? error.stack : undefined,
);
}
}
// Hourly granularity: lifetimes are configured in hours, so a sub-hour
// overshoot past the deadline is acceptable.
@Interval('temporary-note-cleanup', 60 * 60 * 1000)
@@ -31,9 +50,11 @@ export class TemporaryNoteCleanupService {
try {
const now = new Date();
// Candidate ids (non-locking). The authoritative re-check happens per row
// under a row lock below, so this cheap pass just bounds the batch.
const expired = await this.db
.selectFrom('pages')
.select(['id', 'creatorId', 'workspaceId'])
.select(['id'])
.where('temporaryExpiresAt', 'is not', null)
.where('temporaryExpiresAt', '<', now)
.where('deletedAt', 'is', null) // not already in trash
@@ -41,50 +62,53 @@ export class TemporaryNoteCleanupService {
.execute();
let trashed = 0;
for (const page of expired) {
for (const candidate of expired) {
try {
// Re-check the deadline at deletion time. The SELECT above is not
// transactional, so a user may click "Make permanent"
// (toggleTemporary sets temporary_expires_at = null) in the window
// between the SELECT and this per-row removePage. removePage deletes
// by id with only a `deletedAt IS NULL` filter and never re-reads the
// deadline, so without this guard a concurrently-kept note would
// still be trashed. Re-read the row and skip it unless it is still
// armed AND still expired, so a concurrent make-permanent wins.
const current = await this.db
.selectFrom('pages')
.select(['temporaryExpiresAt', 'deletedAt'])
.where('id', '=', page.id)
.executeTakeFirst();
const didTrash = await executeTx(this.db, async (trx) => {
// Re-check the row UNDER A LOCK inside the transaction. `FOR UPDATE
// SKIP LOCKED`:
// - serialises against a concurrent "Make permanent"
// (toggleTemporary UPDATE takes the same row lock): if it commits
// first, the deadline predicate below no longer matches and we
// skip; if we lock first, it waits until this delete commits.
// - SKIP LOCKED lets a second worker/instance skip a row another
// sweeper already claimed instead of blocking on it (no double
// processing, no thundering herd).
// The predicate re-asserts still-armed AND still-expired AND
// not-already-trashed, so a make-permanent / prior sweep drops the row.
const locked = await trx
.selectFrom('pages')
.select(['id', 'creatorId', 'workspaceId'])
.where('id', '=', candidate.id)
.where('temporaryExpiresAt', 'is not', null)
.where('temporaryExpiresAt', '<', now)
.where('deletedAt', 'is', null)
.forUpdate()
.skipLocked()
.executeTakeFirst();
if (
!current ||
current.deletedAt !== null ||
current.temporaryExpiresAt === null ||
new Date(current.temporaryExpiresAt) >= now
) {
// Made permanent, already trashed, or no longer expired since the
// SELECT — leave it alone.
continue;
}
if (!locked) return false;
// Reuse the exact soft-delete path: recursive over children, removes
// shares in a transaction, and emits PAGE_SOFT_DELETED (tree
// invalidation + watcher notifications). Attribute the automatic
// deletion to the note's creator (no schema change). Both the SELECT
// above and removePage filter `deletedAt IS NULL`, so a double sweep
// is idempotent.
await this.pageRepo.removePage(
page.id,
// creatorId is set on every created page; a temporary note always
// has one. Cast to satisfy the non-null deletedById parameter.
page.creatorId as string,
page.workspaceId,
);
trashed++;
// Reuse the exact soft-delete path (recursive children + share
// removal + PAGE_SOFT_DELETED broadcast), running IN this locked
// transaction so the delete is atomic with the re-check and cannot
// deadlock on a nested independent transaction. The broadcast is
// deferred by removePage to this transaction's commit. Attribute the
// automatic deletion to the note's creator (no schema change).
await this.pageRepo.removePage(
locked.id,
// creatorId is set on every created page; a temporary note always
// has one. Cast to satisfy the non-null deletedById parameter.
locked.creatorId as string,
locked.workspaceId,
trx,
);
return true;
});
if (didTrash) trashed++;
} catch (error) {
this.logger.error(
`Failed to trash expired temporary note ${page.id}`,
`Failed to trash expired temporary note ${candidate.id}`,
error instanceof Error ? error.stack : undefined,
);
}
@@ -133,6 +133,9 @@ describe('ShareAliasController authz gates', () => {
creatorId: 'u-1',
alias: 'promo',
confirmReassign: true,
// The requesting user is forwarded so setAlias can gate the reassign
// 409 title disclosure on target-page view permission (#495).
user,
});
expect(result).toEqual({ id: 'alias-1' });
});
@@ -79,6 +79,9 @@ export class ShareAliasController {
creatorId: user.id,
alias: dto.alias,
confirmReassign: dto.confirmReassign,
// Gates whether the reassign 409 may reveal the current target's title
// (view-permission check on that page) — see setAlias (#495).
user,
});
}
@@ -1,4 +1,8 @@
import { BadRequestException, ConflictException } from '@nestjs/common';
import {
BadRequestException,
ConflictException,
ForbiddenException,
} from '@nestjs/common';
import { NoResultError } from 'kysely';
import { ShareAliasService } from './share-alias.service';
@@ -7,6 +11,8 @@ import { ShareAliasService } from './share-alias.service';
* 409 reassign guard, uniqueness-race handling, availability probe, and the
* request-time readable-target resolution (which re-runs the share boundary).
*/
const USER = { id: 'u-1' } as any;
describe('ShareAliasService', () => {
// Sentinel handed to repo calls so tests can assert they ran inside the tx.
const trx = { __trx: true };
@@ -27,6 +33,10 @@ describe('ShareAliasService', () => {
resolveReadableSharePage: jest.fn(),
isSharingAllowed: jest.fn(),
};
// Default: the requester CAN view the target page (validateCanView resolves),
// so the reassign 409 may disclose its title. Tests override to reject to
// assert the no-leak path.
const pageAccessService = { validateCanView: jest.fn().mockResolvedValue(undefined) };
// Fake kysely db: only .transaction().execute(cb) is used by setAlias.
const db = {
transaction: jest.fn(() => ({
@@ -37,9 +47,10 @@ describe('ShareAliasService', () => {
shareAliasRepo as any,
pageRepo as any,
shareService as any,
pageAccessService as any,
db as any,
);
return { service, shareAliasRepo, pageRepo, shareService, db };
return { service, shareAliasRepo, pageRepo, shareService, pageAccessService, db };
}
describe('setAlias', () => {
@@ -50,6 +61,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'A', // too short + uppercase
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -66,6 +78,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: ' My Page ',
});
@@ -114,6 +127,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'ted',
});
@@ -144,6 +158,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
@@ -179,6 +194,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'new',
});
@@ -190,30 +206,77 @@ describe('ShareAliasService', () => {
);
});
it('throws 409 with current target when name is taken and not confirmed', async () => {
const { service, shareAliasRepo, pageRepo } = makeService();
it('throws 409 with the target TITLE (never its id) when the requester CAN view it', async () => {
const { service, shareAliasRepo, pageRepo, pageAccessService } =
makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
alias: 'foo',
pageId: 'p-other',
});
pageRepo.findById.mockResolvedValue({ id: 'p-other', title: 'Other' });
pageAccessService.validateCanView.mockResolvedValue(undefined); // can view
try {
await service.setAlias({
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
} catch (err) {
expect(err).toBeInstanceOf(ConflictException);
expect((err as ConflictException).getResponse()).toMatchObject({
const body = (err as ConflictException).getResponse();
expect(body).toMatchObject({
code: 'ALIAS_REASSIGN_REQUIRED',
currentPageId: 'p-other',
currentPageTitle: 'Other',
});
// SECURITY (#495): the page id is NEVER disclosed, even to a viewer.
expect(body).not.toHaveProperty('currentPageId');
expect(pageAccessService.validateCanView).toHaveBeenCalledWith(
expect.objectContaining({ id: 'p-other' }),
USER,
);
}
expect(shareAliasRepo.updatePageId).not.toHaveBeenCalled();
});
it('throws 409 WITHOUT the title or id when the requester CANNOT view the target (#495)', async () => {
const { service, shareAliasRepo, pageRepo, pageAccessService } =
makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
alias: 'foo',
pageId: 'p-secret',
});
pageRepo.findById.mockResolvedValue({ id: 'p-secret', title: 'Secret' });
// No view permission on the target page -> validateCanView throws.
pageAccessService.validateCanView.mockRejectedValue(
new ForbiddenException(),
);
try {
await service.setAlias({
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
} catch (err) {
expect(err).toBeInstanceOf(ConflictException);
const body = (err as ConflictException).getResponse() as Record<
string,
unknown
>;
expect(body).toMatchObject({ code: 'ALIAS_REASSIGN_REQUIRED' });
// The enumeration hole: neither the id nor the title of a page the
// requester cannot see may leak.
expect(body).not.toHaveProperty('currentPageId');
expect(body.currentPageTitle ?? null).toBeNull();
}
expect(shareAliasRepo.updatePageId).not.toHaveBeenCalled();
});
@@ -231,6 +294,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
confirmReassign: true,
});
@@ -269,6 +333,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -294,6 +359,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -317,6 +383,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -346,6 +413,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -375,6 +443,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
confirmReassign: true,
});
@@ -406,6 +475,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'ted',
});
fail('expected ConflictException');
@@ -428,6 +498,7 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -450,18 +521,22 @@ describe('ShareAliasService', () => {
alias: 'free-name',
valid: true,
available: true,
currentPageId: null,
});
// SECURITY (#495): the availability probe must NOT leak any page id.
expect(res).not.toHaveProperty('currentPageId');
});
it('reports taken with the current target page', async () => {
it('reports taken WITHOUT leaking the current target page id (#495)', async () => {
const { service, shareAliasRepo } = makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
pageId: 'p-9',
});
const res = await service.checkAvailability('taken', 'ws-1');
expect(res).toMatchObject({ available: false, currentPageId: 'p-9' });
expect(res).toMatchObject({ available: false });
// The row exists (available:false) but its pageId is never returned — an
// authenticated member cannot map an alias name to a page id it can't view.
expect(res).not.toHaveProperty('currentPageId');
});
});
@@ -7,7 +7,8 @@ import {
import { ShareAliasRepo } from '@docmost/db/repos/share-alias/share-alias.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { ShareService } from './share.service';
import { Page, ShareAlias } from '@docmost/db/types/entity.types';
import { PageAccessService } from '../page/page-access/page-access.service';
import { Page, ShareAlias, User } from '@docmost/db/types/entity.types';
import { isValidShareAlias, normalizeShareAlias } from './share-alias.util';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
@@ -43,6 +44,7 @@ export class ShareAliasService {
private readonly shareAliasRepo: ShareAliasRepo,
private readonly pageRepo: PageRepo,
private readonly shareService: ShareService,
private readonly pageAccessService: PageAccessService,
@InjectKysely() private readonly db: KyselyDB,
) {}
@@ -55,9 +57,13 @@ export class ShareAliasService {
* `/l/<old>` link survives
* - name already points at pageId -> no-op (idempotent)
* - name points at ANOTHER page -> the "swap". Without confirmReassign
* we throw 409 carrying the current target so the client can confirm;
* with it we UPDATE the single row's page_id (every /l/<alias> link
* follows the 302 to the new page instantly no stale cache).
* we throw 409 so the client can confirm. SECURITY (#495): the 409 reveals
* the current target's title ONLY when `user` may VIEW that page, and never
* its id otherwise any member with one editable+shared page could iterate
* alias names with confirmReassign=false and map them to (id, title) of
* pages they cannot see. With confirmReassign we UPDATE the single row's
* page_id (every /l/<alias> link follows the 302 to the new page instantly
* no stale cache).
*
* To keep the invariant self-healing we DELETE every other alias row still
* pointing at this page (a legacy duplicate, or the target page's own former
@@ -77,8 +83,12 @@ export class ShareAliasService {
creatorId: string;
alias: string;
confirmReassign?: boolean;
// The requesting user — used ONLY to gate whether the reassign 409 may reveal
// the current target page's title (view-permission check). Not an authz gate
// for the write itself (the controller already validated edit on `pageId`).
user: User;
}): Promise<ShareAlias> {
const { workspaceId, pageId, creatorId, confirmReassign } = opts;
const { workspaceId, pageId, creatorId, confirmReassign, user } = opts;
const alias = normalizeShareAlias(opts.alias);
if (!isValidShareAlias(alias)) {
throw new BadRequestException(
@@ -97,14 +107,30 @@ export class ShareAliasService {
// The name is occupied by a DIFFERENT (or dangling) target page.
if (byName && byName.pageId !== pageId) {
if (!confirmReassign) {
// SECURITY (#495): only disclose the current target's TITLE, and only
// when the requester may VIEW that page. Never disclose its id (the
// client's confirm-reassign UX doesn't use it, and it is an enumerable
// identity). A member with one editable+shared page must NOT be able to
// iterate alias names and map them to (id, title) of pages they cannot
// see. When view is denied (or the alias is dangling) the 409 is the
// bare "occupied" fact — the client still shows a generic confirm modal.
const currentPage = byName.pageId
? await this.pageRepo.findById(byName.pageId)
: null;
let currentPageTitle: string | null = null;
if (currentPage) {
try {
await this.pageAccessService.validateCanView(currentPage, user);
currentPageTitle = currentPage.title ?? null;
} catch {
// No view permission on the target -> do not reveal its title.
currentPageTitle = null;
}
}
throw new ConflictException({
message: 'Alias already in use',
code: 'ALIAS_REASSIGN_REQUIRED',
currentPageId: byName.pageId,
currentPageTitle: currentPage?.title ?? null,
currentPageTitle,
});
}
// Confirmed swap. ORDER MATTERS: the partial unique index on
@@ -223,21 +249,27 @@ export class ShareAliasService {
alias: string;
valid: boolean;
available: boolean;
currentPageId: string | null;
}> {
const alias = normalizeShareAlias(rawAlias);
if (!isValidShareAlias(alias)) {
return { alias, valid: false, available: false, currentPageId: null };
return { alias, valid: false, available: false };
}
const existing = await this.shareAliasRepo.findByAliasAndWorkspace(
alias,
workspaceId,
);
// SECURITY (#495): return ONLY the boolean availability. The previous shape
// leaked `currentPageId` — the id of whatever page the alias already targets —
// to ANY authenticated workspace member, with no view-permission check on that
// page. An attacker could enumerate alias names and map them to page ids they
// have no access to. The taken/free bit is all the "is this address free"
// probe needs. The reassign flow (setAlias 409) may surface the target's
// TITLE, but only behind a `validateCanView` on that page (see setAlias); it
// never returns the page id.
return {
alias,
valid: true,
available: !existing,
currentPageId: existing?.pageId ?? null,
};
}
@@ -24,6 +24,54 @@ export const ALLOWED_RATINGS = new Set<string>([
'poor',
]);
// The ONLY route labels accepted. The endpoint is anonymous, so an un-checked
// `route` is a free-text write surface (arbitrary high-cardinality strings /
// injected text into the metrics table). The client only ever sends a label from
// a finite template dictionary (`templateRoute`), so we drop anything not in it.
//
// PARITY: this MUST mirror `KNOWN_ROUTE_TEMPLATES` in the client's
// `apps/client/src/lib/telemetry/route-template.ts` (the canonical source). A
// drift means legit client routes get dropped — keep the two in lockstep; the
// client self-consistency test asserts `templateRoute` only emits these values.
export const ALLOWED_ROUTE_TEMPLATES = new Set<string>([
'/',
'other',
// Static routes.
'/home',
'/spaces',
'/favorites',
'/login',
'/forgot-password',
'/password-reset',
'/setup/register',
'/settings/account/profile',
'/settings/account/preferences',
'/settings/workspace',
'/settings/ai',
'/settings/members',
'/settings/groups',
'/settings/spaces',
'/settings/sharing',
// Dynamic templates (slugs/ids are already collapsed to `:param`).
'/share/:shareId/p/:slug',
'/share/p/:slug',
'/share/:shareId',
'/p/:slug',
'/s/:space/p/:slug',
'/s/:space/trash',
'/s/:space',
'/labels/:label',
'/invites/:invitationId',
'/settings/groups/:groupId',
]);
// `attr` is a web-vitals attribution TARGET: a CSS-selector-ish string (an
// element path like `html>body>div#app>button.cta`), never free prose. Constrain
// it to a conservative CSS-selector charset so the anonymous endpoint cannot be
// used to write arbitrary text / PII / markup into the metrics table. A value
// containing anything outside this set is DROPPED (-> null); the event is kept.
export const ATTR_ALLOWED_CHARSET = /^[A-Za-z0-9#.\-_> :()[\]="'*+~,]+$/;
// Max events accepted per batch; the rest are ignored.
export const MAX_EVENTS_PER_BATCH = 50;
@@ -77,14 +125,20 @@ export function sanitizeVitalEvent(
? e.rating
: null;
// route: accept ONLY a known template label (dictionary check), else drop to
// null. The length cap stays as a cheap pre-guard before the Set lookup.
let route: string | null = null;
if (typeof e.route === 'string' && e.route.length > 0) {
route = e.route.slice(0, MAX_ROUTE_LENGTH);
const candidate = e.route.slice(0, MAX_ROUTE_LENGTH);
route = ALLOWED_ROUTE_TEMPLATES.has(candidate) ? candidate : null;
}
// attr: truncate, then accept ONLY if it is a CSS-selector-shaped string
// (charset whitelist); anything with characters outside the set is dropped.
let attr: string | null = null;
if (typeof e.attr === 'string' && e.attr.length > 0) {
attr = e.attr.slice(0, MAX_ATTR_LENGTH);
const candidate = e.attr.slice(0, MAX_ATTR_LENGTH);
attr = ATTR_ALLOWED_CHARSET.test(candidate) ? candidate : null;
}
let docSize: number | null = null;
@@ -90,6 +90,44 @@ describe('VitalsService.buildRows', () => {
expect(rows[0].attr).toHaveLength(MAX_ATTR_LENGTH);
});
it('keeps a known route template but DROPS an unknown/free-text route (#495)', () => {
const rows = svc.buildRows(
{
events: [
{ name: 'INP', value: 1, route: '/s/:space/p/:slug' }, // known
{ name: 'INP', value: 2, route: '/s/acme-corp/p/secret-slug' }, // raw path (slugs) — not a template
{ name: 'INP', value: 3, route: 'DROP TABLE client_metrics;--' }, // injected free text
{ name: 'INP', value: 4, route: '/home' }, // known static
],
},
WS,
);
expect(rows.map((r) => r.route)).toEqual([
'/s/:space/p/:slug',
null, // raw path dropped
null, // free text dropped
'/home',
]);
});
it('DROPS an attr that is not a CSS-selector-shaped string (#495)', () => {
const rows = svc.buildRows(
{
events: [
{ name: 'INP', value: 1, attr: 'div#app>button.cta' }, // valid selector
{ name: 'INP', value: 2, attr: 'user@example.com wrote a note' }, // free text / PII
{ name: 'INP', value: 3, attr: '<script>alert(1)</script>' }, // markup
],
},
WS,
);
expect(rows.map((r) => r.attr)).toEqual([
'div#app>button.cta',
null,
null,
]);
});
it('caps the batch at 50 events', () => {
const events = Array.from({ length: 200 }, () => ({ name: 'CLS', value: 1 }));
const rows = svc.buildRows({ events }, WS);
@@ -0,0 +1,114 @@
import * as path from 'path';
import { readFileSync } from 'fs';
// Mock ONLY kysely's `sql.raw(...).execute()` so we can observe what
// ensureConcurrentIndexes runs and how it handles failures, without a DB.
const execMock = jest.fn((_db: unknown) => Promise.resolve(undefined));
const rawMock = jest.fn((_stmt: string) => ({ execute: execMock }));
jest.mock('kysely', () => {
const actual = jest.requireActual('kysely');
return { ...actual, sql: { ...actual.sql, raw: rawMock } };
});
import { CONCURRENT_INDEXES, ensureConcurrentIndexes } from './concurrent-indexes';
describe('ensureConcurrentIndexes', () => {
const fakeDb = { __topLevelKysely: true } as never;
beforeEach(() => {
execMock.mockReset().mockResolvedValue(undefined);
rawMock.mockClear();
});
it('redefines the inlinable f_unaccent BEFORE building any index, then builds each CONCURRENTLY outside a transaction', async () => {
const onLog = jest.fn();
await ensureConcurrentIndexes(fakeDb, onLog);
// One f_unaccent redefine + one statement per index.
expect(rawMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
const statements = rawMock.mock.calls.map((c) => c[0] as string);
// ORDER: the f_unaccent redefine MUST be first — otherwise an existing
// tenant's old 2-arg f_unaccent makes every CONCURRENTLY build fail.
expect(statements[0]).toContain('CREATE OR REPLACE FUNCTION f_unaccent');
expect(statements[0]).toContain('SELECT public.unaccent($1)');
expect(statements[0]).not.toContain('CREATE INDEX');
// The rest are the CONCURRENTLY index builds.
for (const stmt of statements.slice(1)) {
expect(stmt).toContain('CREATE INDEX CONCURRENTLY');
expect(stmt).toContain('IF NOT EXISTS');
}
// Executed against the top-level db (a transaction would forbid CONCURRENTLY).
for (const call of execMock.mock.calls) {
expect(call[0]).toBe(fakeDb);
}
expect(onLog).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
});
it('still builds the indexes when the f_unaccent redefine fails (fresh DB, best-effort)', async () => {
// Redefine (first execute) throws; the index loop must still run.
execMock
.mockRejectedValueOnce(new Error('extension "unaccent" does not exist'))
.mockResolvedValue(undefined);
const onLog = jest.fn();
await expect(
ensureConcurrentIndexes(fakeDb, onLog),
).resolves.toBeUndefined();
// 1 redefine attempt + one per index.
expect(execMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
const errored = onLog.mock.calls.filter((c) => c[1] !== undefined);
expect(errored).toHaveLength(1);
expect(String(errored[0][1])).toContain('unaccent');
});
it('is best-effort: a failing index does not abort the rest and is reported', async () => {
// Redefine ok; fail the FIRST index; the remaining ones must still be tried.
execMock
.mockResolvedValueOnce(undefined) // f_unaccent redefine
.mockRejectedValueOnce(new Error('relation "pages" does not exist'))
.mockResolvedValue(undefined);
const onLog = jest.fn();
await expect(
ensureConcurrentIndexes(fakeDb, onLog),
).resolves.toBeUndefined();
expect(execMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
const errored = onLog.mock.calls.filter((c) => c[1] !== undefined);
expect(errored).toHaveLength(1);
expect(String(errored[0][1])).toContain('does not exist');
});
});
// DRIFT GUARD: each CONCURRENT_INDEXES entry pre-builds an index that a plain
// migration ALSO creates with `IF NOT EXISTS`. If the two expressions diverge,
// Postgres would treat them as different indexes and the pre-build would NOT
// make the migration a no-op. Assert the migration files still contain each
// index's functional expression.
describe('CONCURRENT_INDEXES parity with the migrations', () => {
const migrationsDir = path.join(__dirname, 'migrations');
const files = [
'20260705T120000-perf-indexes.ts',
'20260706T120000-search-lookup-trgm.ts',
].map((f) => readFileSync(path.join(migrationsDir, f), 'utf8'));
const allMigrationSrc = files.join('\n');
it.each(CONCURRENT_INDEXES.map((i) => [i.name, i]))(
'migration source still creates %s with the same expression',
(_name, idx) => {
// Extract the `USING gin ((...expr...) gin_trgm_ops)` tail from the
// canonical create and assert the migration source contains it verbatim.
const m = (idx as { create: string }).create.match(
/ON \w+ (USING gin .+)$/,
);
expect(m).not.toBeNull();
const expr = (m as RegExpMatchArray)[1];
expect(allMigrationSrc).toContain(expr);
// And the migration must build it by the SAME index name.
expect(allMigrationSrc).toContain(
`CREATE INDEX IF NOT EXISTS ${(idx as { name: string }).name}`,
);
},
);
});
@@ -0,0 +1,142 @@
import { Kysely, sql } from 'kysely';
/**
* Indexes that MUST be built with `CREATE INDEX CONCURRENTLY` so an auto-deploy
* migration never takes a `SHARE` lock that blocks writes on a hot table
* (`pages`) for the potentially minutes-long GIN trigram build (#495 item 12).
*
* Kysely runs each migration INSIDE a transaction (Postgres has transactional
* DDL), and `CREATE INDEX CONCURRENTLY` cannot run inside a transaction block, so
* these cannot live in an ordinary migration. Instead {@link ensureConcurrentIndexes}
* builds them out-of-band (no transaction) BEFORE the migrator runs; the matching
* migrations keep a plain `CREATE INDEX IF NOT EXISTS` as a backstop, which then
* no-ops because the index already exists. So:
* - existing prod DB, incremental deploy: the pre-build FIRST redefines
* `f_unaccent` to its index-inlinable 1-arg form (see below), THEN builds each
* index CONCURRENTLY (no write lock); the migration's IF NOT EXISTS no-ops
* the write-blocking build is gone;
* - fresh DB (or a DB whose `pages` / `unaccent` extension does not exist yet):
* the pre-build fails and is swallowed (best-effort), and the migration builds
* the index normally on an empty/small table where the lock is irrelevant.
*
* NOT a strict "worst case = previous behaviour":
* - The f_unaccent form matters. The trigram expressions use `LOWER(f_unaccent(col))`.
* The INDEX-INLINABLE 1-arg `f_unaccent(text)` (`SELECT public.unaccent($1)`)
* is created INSIDE migration 20260705, which runs AFTER this pre-build. On an
* existing tenant the live `f_unaccent` is still the OLD 2-arg form from
* 20250729, which Postgres CANNOT inline into a CONCURRENTLY index expression
* (`function unaccent(unknown, text) does not exist ... during inlining`) the
* build fails outright. So this pre-build redefines `f_unaccent` to the 1-arg
* form FIRST (idempotent, output-identical, in lockstep with 20260705). Without
* that redefine the whole feature is dead-on-arrival for the very case it
* targets.
* - An INTERRUPTED `CREATE INDEX CONCURRENTLY` (killed pod, cancelled query) with
* a working `f_unaccent` leaves an INVALID index behind. A subsequent name-based
* `IF NOT EXISTS` in BOTH this pre-build and the migration backstop sees the
* name and skips, so the invalid index is NEVER repaired automatically and the
* query keeps seq-scanning until an operator `DROP INDEX`es it. The old
* in-transaction build could not leave an invalid index (a failed tx rolled the
* whole index back), so this is a genuinely new failure mode, not "= previous".
* (A future hardening could `DROP` an `indisvalid = false` index before
* rebuilding; not done here.)
*
* The `create` text is the CANONICAL definition it MUST match the migration's
* `IF NOT EXISTS` create expression exactly (same functional expression + opclass)
* or Postgres would treat them as two different indexes.
*/
export const CONCURRENT_INDEXES: ReadonlyArray<{
name: string;
create: string;
}> = [
{
// #348 perf-indexes — pages.title trigram (coalesce-free functional expr).
name: 'idx_pages_title_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_title_trgm ' +
'ON pages USING gin ((LOWER(f_unaccent(title))) gin_trgm_ops)',
},
{
// #348 perf-indexes — users.name trigram (member search-suggest).
name: 'idx_users_name_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_name_trgm ' +
'ON users USING gin ((LOWER(f_unaccent(name))) gin_trgm_ops)',
},
{
// #443 search-lookup-trgm — pages.text_content trigram (the slow, large one).
name: 'idx_pages_text_content_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_text_content_trgm ' +
'ON pages USING gin ((LOWER(f_unaccent(text_content))) gin_trgm_ops)',
},
];
/**
* Idempotent, output-identical redefinition of `f_unaccent` to the INDEX-INLINABLE
* 1-arg form. MUST stay byte-for-byte in lockstep with migration
* `20260705T120000-perf-indexes.ts` (same signature + body): the trigram indexes
* above only build CONCURRENTLY once `f_unaccent(text)` inlines, and on an existing
* tenant it is still the old 2-arg form until that migration runs which is AFTER
* this pre-build.
*/
const F_UNACCENT_REDEF = `
CREATE OR REPLACE FUNCTION f_unaccent(text)
RETURNS text
LANGUAGE sql
IMMUTABLE PARALLEL SAFE STRICT
AS $func$
SELECT public.unaccent($1);
$func$
`;
/**
* Best-effort, non-transactional pre-build of {@link CONCURRENT_INDEXES}. Run
* BEFORE the migrator so the blocking `CREATE INDEX` in the corresponding
* migration becomes an `IF NOT EXISTS` no-op.
*
* `db` MUST be the top-level Kysely instance (NOT a transaction): each statement
* then executes on its own connection with no surrounding `BEGIN`, which is
* required for `CONCURRENTLY`. Every statement is independent and swallowed on
* error: the migration backstop still builds the index, so a failure here is
* never fatal. `onLog` reports progress/failures for the caller to route to its
* logger.
*
* ORDER MATTERS: redefine `f_unaccent` to the inlinable form BEFORE the index
* loop otherwise an existing tenant's old 2-arg `f_unaccent` makes every
* CONCURRENTLY build fail and the feature is a no-op (see the module docstring).
*/
export async function ensureConcurrentIndexes(
db: Kysely<any>,
onLog?: (message: string, error?: unknown) => void,
): Promise<void> {
// Make f_unaccent inlinable FIRST. On a fresh DB (no `unaccent` extension yet)
// this throws harmlessly and is swallowed — the migration builds everything.
try {
await sql.raw(F_UNACCENT_REDEF).execute(db);
onLog?.('f_unaccent redefined to the index-inlinable 1-arg form');
} catch (error) {
onLog?.(
'f_unaccent redefine skipped (fresh DB / no unaccent extension) — ' +
'the migration will define it and build the indexes',
error,
);
}
for (const idx of CONCURRENT_INDEXES) {
try {
await sql.raw(idx.create).execute(db);
onLog?.(`Concurrent index ensured: ${idx.name}`);
} catch (error) {
// Non-fatal by design — the migration's IF NOT EXISTS create is the
// backstop. Benign on a fresh DB (the table/extension does not exist yet).
// NOT benign, but still swallowed, on an existing tenant whose f_unaccent
// could not be redefined above (then the migration builds the index NON-
// concurrently under the SHARE lock this feature meant to avoid).
onLog?.(
`Concurrent index pre-build skipped for ${idx.name} ` +
`(will fall back to the in-migration build)`,
error,
);
}
}
}
@@ -0,0 +1,113 @@
import { executeTx, registerAfterCommit } from './utils';
import { KyselyDB, KyselyTransaction } from './types/kysely.types';
// Post-commit hook contract (#495 item 13): a side effect registered via
// registerAfterCommit must run ONLY AFTER the owning transaction commits, and a
// hook registered against a passed-through existingTrx must fire at the OUTER
// commit boundary — never inside the inner call. We fake the Kysely transaction
// runner so the ordering is observable without a real DB.
/**
* A minimal db whose `.transaction().execute(cb)` records the commit ORDER: it
* runs `cb(trx)`, pushes 'commit' onto `log` (simulating the real commit that
* happens after the callback resolves), then returns the callback's result.
*/
function fakeDb(log: string[]): { db: KyselyDB; trx: KyselyTransaction } {
const trx = { __fakeTrx: true } as unknown as KyselyTransaction;
const db = {
transaction: () => ({
execute: async (cb: (t: KyselyTransaction) => Promise<unknown>) => {
const result = await cb(trx);
log.push('commit');
return result;
},
}),
} as unknown as KyselyDB;
return { db, trx };
}
describe('executeTx post-commit hooks', () => {
it('runs an afterCommit hook only AFTER the transaction commits', async () => {
const log: string[] = [];
const { db } = fakeDb(log);
await executeTx(db, async (trx) => {
log.push('body');
registerAfterCommit(trx, () => {
log.push('hook');
});
// The hook must NOT have run yet — the tx is still open.
expect(log).toEqual(['body']);
});
// Order proves post-commit: body → commit → hook (never body → hook → commit).
expect(log).toEqual(['body', 'commit', 'hook']);
});
it('drains hooks registered against a passed-through existingTrx at the OUTER commit', async () => {
const log: string[] = [];
const { db, trx: outerTrx } = fakeDb(log);
await executeTx(db, async (outer) => {
// Nested executeTx reuses the outer trx: it must NOT commit or drain now.
await executeTx(
db,
async (inner) => {
registerAfterCommit(inner, () => {
log.push('inner-hook');
});
},
outer,
);
// Still inside the outer tx — the inner hook has not fired.
expect(log).toEqual([]);
});
// The single (outer) commit drains the hook registered on the shared trx.
expect(log).toEqual(['commit', 'inner-hook']);
// Sanity: the trx the hooks were registered against is the outer one.
expect(outerTrx).toBeDefined();
});
it('a hook failure does not reject the already-committed executeTx', async () => {
const log: string[] = [];
const { db } = fakeDb(log);
await expect(
executeTx(db, async (trx) => {
registerAfterCommit(trx, () => {
throw new Error('cache del blew up');
});
registerAfterCommit(trx, () => {
log.push('second-hook-still-runs');
});
return 'ok';
}),
).resolves.toBe('ok');
// The throwing hook is swallowed; a later hook still runs.
expect(log).toEqual(['commit', 'second-hook-still-runs']);
});
it('does NOT run afterCommit hooks when the transaction body throws (rollback)', async () => {
// The body rejects -> the fake transaction never pushes 'commit' and
// db.transaction().execute() rejects, mirroring a real rolled-back tx. The
// drain runs only AFTER the awaited (committed) transaction, so a rollback
// must leave every registered hook UN-run — otherwise a cache-bust / event
// would fire for a write that never landed.
const log: string[] = [];
const { db } = fakeDb(log);
const hook = jest.fn();
await expect(
executeTx(db, async (trx) => {
registerAfterCommit(trx, hook);
throw new Error('write failed -> rollback');
}),
).rejects.toThrow('write failed -> rollback');
// No commit happened, and the post-commit hook never ran.
expect(log).toEqual([]); // no 'commit'
expect(hook).not.toHaveBeenCalled();
});
});
@@ -33,16 +33,18 @@ import { type Kysely, sql } from 'kysely';
* - comments: `findPageComments` does WHERE page_id ORDER BY id ASC, but only
* `(page_id)` exists extra sort.
*
* DEPLOY-TIME LOCK WARNING: these are plain (non-CONCURRENT) CREATE INDEX
* statements CONCURRENTLY is impossible because Kysely runs each migration in a
* transaction. They take a SHARE lock that BLOCKS writes (INSERT/UPDATE/DELETE) on
* pages/users/groups/comments/page_history for the duration of the build. The two
* GIN trigram builds on pages.title / users.name are the slow ones and can take
* minutes on a large tenant a write-outage window during the deploy migration.
* For large installations, run this migration in a maintenance window, or build
* the trigram indexes out-of-band with CREATE INDEX CONCURRENTLY before deploying
* (then this migration's `IF NOT EXISTS` is a no-op). Small/typical tenants are
* unaffected.
* DEPLOY-TIME LOCK: these are plain (non-CONCURRENT) CREATE INDEX statements
* CONCURRENTLY is impossible HERE because Kysely runs each migration in a
* transaction. The two GIN trigram builds on pages.title / users.name are the
* slow ones and would take a SHARE lock that BLOCKS writes on pages/users for
* minutes on a large tenant. To avoid that, `ensureConcurrentIndexes`
* (database/concurrent-indexes.ts) now pre-builds BOTH trigram indexes with
* CREATE INDEX CONCURRENTLY (no transaction) BEFORE the migrator runs, so on an
* existing DB the two `IF NOT EXISTS` trigram creates below no-op and no write
* lock is taken. On a fresh DB the pre-build is skipped and they build on an
* empty table. Keep the two trigram creates in lockstep with their CANONICAL
* definitions in CONCURRENT_INDEXES. (The plain b-tree indexes further down are
* fast metadata-only builds; they are not pre-built.)
*/
export async function up(db: Kysely<any>): Promise<void> {
// Index-compatible, output-identical redefinition of f_unaccent (see header).
@@ -26,13 +26,16 @@ import { type Kysely, sql } from 'kysely';
* to update than b-trees); on the small instances this fork targets that cost
* is acceptable and the read win on agent lookups is the priority.
*
* DEPLOY-TIME LOCK WARNING: plain (non-CONCURRENT) CREATE INDEX Kysely runs
* each migration in a transaction, so CONCURRENTLY is impossible. The build takes
* a SHARE lock that BLOCKS writes on `pages` for its duration. The text_content
* GIN build is the slow one and can take minutes on a large tenant. For large
* installations, run this in a maintenance window or build the index out-of-band
* with CREATE INDEX CONCURRENTLY before deploying (then `IF NOT EXISTS` no-ops
* here). Small/typical tenants are unaffected.
* DEPLOY-TIME LOCK: this is a plain (non-CONCURRENT) CREATE INDEX Kysely runs
* each migration in a transaction, so CONCURRENTLY is impossible HERE, and the
* build would take a SHARE lock that BLOCKS writes on `pages` for its duration
* (the text_content GIN build can take minutes on a large tenant). To avoid that,
* `ensureConcurrentIndexes` (database/concurrent-indexes.ts) now pre-builds this
* index with CREATE INDEX CONCURRENTLY (no transaction) BEFORE the migrator runs,
* so on an existing DB the `IF NOT EXISTS` below no-ops and no write lock is taken.
* On a fresh DB the pre-build is skipped and this builds it on an empty table
* where the lock is irrelevant. Keep this create in lockstep with the CANONICAL
* definition in CONCURRENT_INDEXES (same expression + opclass).
*/
export async function up(db: Kysely<any>): Promise<void> {
// The title predicate is served by #348's idx_pages_title_trgm — see header.
@@ -1,5 +1,6 @@
import { Injectable, Logger } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { sql } from 'kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import {
@@ -188,6 +189,144 @@ export class AiChatMessageRepo {
return query.returning(this.baseFields).executeTakeFirst();
}
/**
* #487 OWNER terminal write the streamText terminal callback's finalize. Like
* `update` but CONDITIONAL on `status='streaming' OR metadata.finalizeFailed`:
* the owner writes its real content EITHER when the row is still streaming (the
* normal case) OR when a reconcile stamp already flipped it to a terminal status
* but marked `finalizeFailed:true` the owner's real content OVERWRITES that
* placeholder stamp (owner-write priority, #487). A row that is properly terminal
* (no finalizeFailed) is left untouched (undefined) idempotent. The `patch`
* carries the real metadata WITHOUT finalizeFailed, so a successful write CLEARS
* the flag. Returns the updated row, or undefined when nothing matched.
*/
async finalizeOwner(
id: string,
workspaceId: string,
patch: Partial<{
content: string | null;
toolCalls: unknown;
metadata: unknown;
status: string | null;
}>,
trx?: KyselyTransaction,
): Promise<AiChatMessage | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatMessages')
.set({ ...(patch as Record<string, unknown>), updatedAt: new Date() })
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where((eb) =>
eb.or([
eb('status', '=', 'streaming'),
eb(sql<string>`(metadata->>'finalizeFailed')`, '=', 'true'),
]),
)
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* #487 RECONCILE status-only stamp settle a stuck 'streaming' row to a
* terminal status WITHOUT the owner's real content (which lived only in the
* dead process's memory — a documented loss). CONDITIONAL on `status='streaming'`
* (never touches an already-terminal row) AND it MERGES `finalizeFailed:true`
* into metadata (preserving the partial `parts` already persisted) so a LATER
* owner-write (finalizeOwner) can still OVERWRITE this placeholder with real
* content, and so `isInterruptResume` can EXCLUDE this row (a reconcile stamp is
* not a genuine user interruption). Returns the updated row, or undefined.
*/
async stampTerminalIfStreaming(
id: string,
workspaceId: string,
status: 'aborted' | 'error' | 'completed',
trx?: KyselyTransaction,
): Promise<AiChatMessage | undefined> {
const db = dbOrTx(this.db, trx);
return db
.updateTable('aiChatMessages')
.set({
status,
metadata: sql`coalesce(metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('status', '=', 'streaming')
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* #487 reconcile clause (b): streaming assistant rows whose linked RUN has
* already reached a terminal status an asymmetry ("run settled / message
* streaming forever") the periodic reconcile heals by stamping the message.
* Returns the message id + its run's terminal status, bounded.
*/
async findStreamingWithTerminalRun(
limit = 200,
// #487: scope to ONE chat for the opportunistic per-turn reconcile (removes
// reconcile latency from the user-visible path); omit for the periodic sweep.
chat?: { chatId: string; workspaceId: string },
): Promise<
Array<{ messageId: string; workspaceId: string; runStatus: string }>
> {
let query = this.db
.selectFrom('aiChatMessages as m')
.innerJoin('aiChatRuns as r', 'r.assistantMessageId', 'm.id')
.select([
'm.id as messageId',
'm.workspaceId as workspaceId',
'r.status as runStatus',
])
.where('m.status', '=', 'streaming')
.where('r.status', 'in', ['succeeded', 'failed', 'aborted']);
if (chat) {
query = query
.where('m.chatId', '=', chat.chatId)
.where('m.workspaceId', '=', chat.workspaceId);
}
return query.limit(limit).execute();
}
/**
* #487 reconcile clause (d) historical-row safety: streaming rows older than
* `staleMs` whose chat has NO active run row (double-gated). Settle them to
* 'aborted' + finalizeFailed (so a late owner-write could still overwrite).
* Returns the count. Used ONLY by the periodic reconcile, never at boot.
*/
async sweepStreamingWithoutActiveRun(
staleMs: number,
trx?: KyselyTransaction,
): Promise<number> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - staleMs);
const rows = await db
.updateTable('aiChatMessages as m')
.set({
status: 'aborted',
metadata: sql`coalesce(m.metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('m.status', '=', 'streaming')
.where('m.updatedAt', '<', staleBefore)
.where((eb) =>
eb.not(
eb.exists(
eb
.selectFrom('aiChatRuns as r')
.select('r.id')
.whereRef('r.chatId', '=', 'm.chatId')
.where('r.status', 'in', ['pending', 'running']),
),
),
)
.returning('m.id')
.execute();
return rows.length;
}
/**
* Crash-recovery sweep (#183): flip every assistant row still left in the
* 'streaming' state (a turn that died mid-write before reaching a terminal
@@ -200,13 +339,20 @@ export class AiChatMessageRepo {
* step, so an actively-streaming row never matches; this prevents a fresh
* replica's boot-sweep from aborting a turn another replica is still streaming
* in a multi-instance deploy.
*
* #487: the sweep now ALSO marks `finalizeFailed:true` so a late owner-write can
* overwrite this placeholder with real content (owner-write priority).
*/
async sweepStreaming(trx?: KyselyTransaction): Promise<number> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - SWEEP_STREAMING_STALE_MS);
const rows = await db
.updateTable('aiChatMessages')
.set({ status: 'aborted', updatedAt: new Date() })
.set({
status: 'aborted',
metadata: sql`coalesce(metadata, '{}'::jsonb) || jsonb_build_object('finalizeFailed', true)`,
updatedAt: new Date(),
})
.where('status', '=', 'streaming')
.where('updatedAt', '<', staleBefore)
.returning('id')
@@ -143,6 +143,41 @@ export class AiChatRunRepo {
.executeTakeFirst();
}
/**
* #487: CONDITIONAL terminal finalize flip a run to a terminal status and
* stamp `finished_at` ONLY while it is still active (pending|running), mirroring
* the assistant message's `onlyIfStreaming` guard. A double-settle (a late or
* second writer, a supersede applying a zombie's intended, a reconcile stamp)
* matches NOTHING once the row is terminal and is a benign no-op so a terminal
* status can never be clobbered by a later writer (last-writer-wins is gone).
*
* Returns the updated row when it WAS active (this call wrote it), else
* undefined (the row was already terminal another writer won). The caller
* distinguishes the two to resolve the correct settle outcome.
*/
async finalizeIfActive(
id: string,
workspaceId: string,
patch: { status: string; error: string | null },
trx?: KyselyTransaction,
): Promise<AiChatRun | undefined> {
const db = dbOrTx(this.db, trx);
const now = new Date();
return db
.updateTable('aiChatRuns')
.set({
status: patch.status,
error: patch.error,
finishedAt: now,
updatedAt: now,
})
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.returning(this.baseFields)
.executeTakeFirst();
}
/**
* Mark an EXPLICIT stop request on an active run (distinct from a browser
* disconnect, which never stops a run). Stamps `stop_requested_at` ONLY while
@@ -184,6 +219,31 @@ export class AiChatRunRepo {
* sweeps only runs UNTOUCHED past the window. Phase 1 is single-process, so the
* boot path supplies no window.
*/
/**
* #487 reconcile clause (c): active (pending|running) runs UNTOUCHED past
* `staleMs` candidates for "no live runner" abort. Staleness is measured from
* `updated_at` (the LAST-PROGRESS timestamp recordStep bumps it), NOT
* `started_at`, so a legitimate long-running marathon (1125 min of steady
* progress) is never a candidate. The caller filters these against its in-memory
* `active` / zombie maps ("no entry" is the PRIMARY gate a live entry is never
* aborted) before settling any of them. Bounded.
*/
async findStaleActive(
staleMs: number,
limit = 200,
trx?: KyselyTransaction,
): Promise<Array<{ id: string; workspaceId: string; chatId: string }>> {
const db = dbOrTx(this.db, trx);
const staleBefore = new Date(Date.now() - staleMs);
return db
.selectFrom('aiChatRuns')
.select(['id', 'workspaceId', 'chatId'])
.where('status', 'in', ACTIVE_RUN_STATUSES as unknown as string[])
.where('updatedAt', '<', staleBefore)
.limit(limit)
.execute();
}
async sweepRunning(
opts: { staleMs?: number } = {},
trx?: KyselyTransaction,
@@ -1,7 +1,7 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx, executeTx } from '../../utils';
import { dbOrTx, executeTx, registerAfterCommit } from '../../utils';
import {
InsertablePage,
Page,
@@ -349,14 +349,23 @@ export class PageRepo {
pageId: string,
deletedById: string,
workspaceId: string,
// Optional caller transaction. When passed, the reads + soft-delete run in
// THAT transaction (so a caller holding a `FOR UPDATE` lock on the row — e.g.
// the temporary-note sweeper — can delete under the lock without deadlocking
// on a nested independent transaction) and the PAGE_SOFT_DELETED broadcast is
// deferred to the caller's COMMIT via registerAfterCommit (so a rolled-back
// delete never broadcasts). With no trx the behaviour is unchanged: own
// transaction, broadcast right after it commits.
existingTrx?: KyselyTransaction,
): Promise<void> {
const currentDate = new Date();
const readDb = dbOrTx(this.db, existingTrx);
// Read the root snapshot up front so PAGE_SOFT_DELETED can carry it without
// a post-commit DB read (variant A). Only the root of the deleted subtree is
// needed for the tree broadcast — the client `treeModel.remove` drops all
// descendants, so we don't snapshot/broadcast every descendant.
const rootSnapshot = await this.db
const rootSnapshot = await readDb
.selectFrom('pages')
.select([
'id',
@@ -371,7 +380,7 @@ export class PageRepo {
.where('deletedAt', 'is', null)
.executeTakeFirst();
const descendants = await this.db
const descendants = await readDb
.withRecursive('page_descendants', (db) =>
db
.selectFrom('pages')
@@ -393,39 +402,60 @@ export class PageRepo {
const pageIds = descendants.map((d) => d.id);
if (pageIds.length > 0) {
await executeTx(this.db, async (trx) => {
await trx
.updateTable('pages')
.set({
deletedById: deletedById,
deletedAt: currentDate,
})
.where('id', 'in', pageIds)
.where('deletedAt', 'is', null)
.execute();
// Reuse the caller's transaction when given (executeTx passes it straight
// through), else own a fresh one.
await executeTx(
this.db,
async (trx) => {
await trx
.updateTable('pages')
.set({
deletedById: deletedById,
deletedAt: currentDate,
})
.where('id', 'in', pageIds)
.where('deletedAt', 'is', null)
.execute();
await trx.deleteFrom('shares').where('pageId', 'in', pageIds).execute();
});
await trx
.deleteFrom('shares')
.where('pageId', 'in', pageIds)
.execute();
},
existingTrx,
);
this.eventEmitter.emit(EventName.PAGE_SOFT_DELETED, {
pageIds: pageIds,
workspaceId,
// Root-only snapshot: one `deleteTreeNode` is enough, the client removes
// the whole subtree. Skip if the root vanished between the two reads.
pages: rootSnapshot
? [
{
id: rootSnapshot.id,
slugId: rootSnapshot.slugId,
title: rootSnapshot.title,
icon: rootSnapshot.icon,
position: rootSnapshot.position,
spaceId: rootSnapshot.spaceId,
parentPageId: rootSnapshot.parentPageId,
},
]
: [],
});
const emitSoftDeleted = () => {
this.eventEmitter.emit(EventName.PAGE_SOFT_DELETED, {
pageIds: pageIds,
workspaceId,
// Root-only snapshot: one `deleteTreeNode` is enough, the client
// removes the whole subtree. Skip if the root vanished between reads.
pages: rootSnapshot
? [
{
id: rootSnapshot.id,
slugId: rootSnapshot.slugId,
title: rootSnapshot.title,
icon: rootSnapshot.icon,
position: rootSnapshot.position,
spaceId: rootSnapshot.spaceId,
parentPageId: rootSnapshot.parentPageId,
},
]
: [],
});
};
if (existingTrx) {
// Inside a caller transaction: the delete above is NOT committed yet.
// Defer the tree broadcast to the caller's commit so a rolled-back delete
// never broadcasts a phantom removal.
registerAfterCommit(existingTrx, emitSoftDeleted);
} else {
// Own transaction already committed above — broadcast now.
emitSoftDeleted();
}
}
}
@@ -3,7 +3,7 @@ import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Cache } from 'cache-manager';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import { dbOrTx, registerAfterCommit } from '../../utils';
import {
InsertableWorkspace,
UpdatableWorkspace,
@@ -80,16 +80,30 @@ export class WorkspaceRepo {
*/
private async bustWorkspaceCache(
workspace?: Pick<Workspace, 'hostname'> | undefined,
trx?: KyselyTransaction,
): Promise<void> {
try {
await this.cacheManager.del(CacheKey.WORKSPACE_SELF_HOSTED);
if (workspace?.hostname) {
await this.cacheManager.del(
CacheKey.WORKSPACE_BY_HOST(workspace.hostname),
);
const del = async () => {
try {
await this.cacheManager.del(CacheKey.WORKSPACE_SELF_HOSTED);
if (workspace?.hostname) {
await this.cacheManager.del(
CacheKey.WORKSPACE_BY_HOST(workspace.hostname),
);
}
} catch {
// cache is best-effort; TTL is the backstop
}
} catch {
// cache is best-effort; TTL is the backstop
};
if (trx) {
// Inside a caller transaction the write is NOT yet committed: busting now
// opens a repopulation window (a concurrent reader reloads the cache with
// the pre-commit / stale row, which then survives until TTL). Defer the del
// to the transaction's commit (drained by the owning executeTx) (#495).
registerAfterCommit(trx, del);
} else {
// No transaction: the mutation above already auto-committed, so this del is
// already post-commit.
await del();
}
}
@@ -180,7 +194,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -195,7 +209,7 @@ export class WorkspaceRepo {
.returning(this.baseFields)
.executeTakeFirst();
// Bust the cached "not found" so a fresh install / new tenant is seen at once.
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -249,7 +263,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -271,7 +285,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -326,7 +340,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -354,7 +368,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -376,7 +390,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -398,7 +412,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace);
await this.bustWorkspaceCache(workspace, trx);
return workspace;
}
@@ -4,6 +4,7 @@ import { promises as fs } from 'fs';
import { Migrator, FileMigrationProvider } from 'kysely';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { ensureConcurrentIndexes } from '@docmost/db/concurrent-indexes';
@Injectable()
export class MigrationService {
@@ -12,6 +13,16 @@ export class MigrationService {
constructor(@InjectKysely() private readonly db: KyselyDB) {}
async migrateToLatest(): Promise<void> {
// Build write-blocking trigram indexes CONCURRENTLY (no transaction) BEFORE
// the migrator runs, so the corresponding in-migration `CREATE INDEX IF NOT
// EXISTS` no-ops instead of taking a SHARE lock on `pages` during deploy
// (#495). Best-effort: on a fresh DB (no `pages`/`f_unaccent` yet) this is a
// no-op and the migrations build the index normally.
await ensureConcurrentIndexes(this.db, (message, error) => {
if (error) this.logger.warn(`${message}: ${String(error)}`);
else this.logger.log(message);
});
const migrator = new Migrator({
db: this.db,
provider: new FileMigrationProvider({
+63 -3
View File
@@ -6,16 +6,76 @@ import { KyselyDB, KyselyTransaction } from './types/kysely.types';
* If an existing transaction is provided, it directly executes the callback with it.
* Otherwise, it starts a new transaction using the provided database instance and executes the callback within that transaction.
*/
/**
* Post-commit side-effect hooks, keyed by the transaction they were registered
* against. A WeakMap so an abandoned/never-drained transaction's entry is GC'd
* with the trx object (no leak). Used by {@link registerAfterCommit} /
* {@link executeTx}.
*/
const afterCommitHooks = new WeakMap<
KyselyTransaction,
Array<() => Promise<void> | void>
>();
/**
* Register a side effect to run ONLY AFTER the transaction that owns `trx`
* commits. THE fix for "bust the cache inside the open transaction" bugs: a
* cache-invalidation (or any read-your-write-visible side effect) done while the
* writing transaction is still open opens a window where a concurrent reader
* repopulates the cache with the PRE-COMMIT (stale) row, so after commit the
* cache holds the old value until its TTL. Deferring the effect to post-commit
* closes that window.
*
* The hook is drained by the OUTERMOST {@link executeTx} that actually owns
* (created) this transaction so registering against a passed-through
* `existingTrx` still fires at the real commit boundary, not at the inner call.
* NOTE: a hook registered against a transaction that was NOT created via
* `executeTx` (untracked) will never be drained always create transactions
* through `executeTx` when you rely on post-commit hooks.
*/
export function registerAfterCommit(
trx: KyselyTransaction,
hook: () => Promise<void> | void,
): void {
const existing = afterCommitHooks.get(trx);
if (existing) existing.push(hook);
else afterCommitHooks.set(trx, [hook]);
}
export async function executeTx<T>(
db: KyselyDB,
callback: (trx: KyselyTransaction) => Promise<T>,
existingTrx?: KyselyTransaction,
): Promise<T> {
if (existingTrx) {
return await callback(existingTrx); // Execute callback with existing transaction
} else {
return await db.transaction().execute((trx) => callback(trx)); // Start new transaction and execute callback
// Reuse the caller's transaction. Any post-commit hooks registered here are
// drained by the OUTER executeTx that created `existingTrx`, at the true
// commit boundary — so we must NOT drain them now.
return await callback(existingTrx);
}
// We OWN this transaction: run the body, then (only once it has COMMITTED)
// drain the post-commit hooks registered against it during the body.
let ownTrx: KyselyTransaction | undefined;
const result = await db.transaction().execute((trx) => {
ownTrx = trx;
return callback(trx);
});
if (ownTrx) {
const hooks = afterCommitHooks.get(ownTrx);
if (hooks) {
afterCommitHooks.delete(ownTrx);
for (const hook of hooks) {
// Best-effort: a failed side effect (e.g. a cache del) must not fail the
// already-committed transaction.
try {
await hook();
} catch {
// swallow — the durable write already committed
}
}
}
}
return result;
}
/*
@@ -99,10 +99,12 @@ describe('AiSettingsService.getMasked reindex progress', () => {
// actually pins the progress.total branch rather than coincidentally
// matching the DB fallback. With fix #1 the two sources agree in practice,
// but getMasked must still return progress.total when a record is active.
const startedAt = Date.now();
reindexProgress.get.mockResolvedValue({
total: 500,
done: 120,
startedAt: Date.now(),
startedAt,
runId: 'run-abc',
});
const masked = await service.getMasked(WORKSPACE_ID);
@@ -110,6 +112,10 @@ describe('AiSettingsService.getMasked reindex progress', () => {
expect(masked.indexedPages).toBe(120); // progress.done, not DB 478
expect(masked.totalPages).toBe(500); // progress.total, not DB 478
expect(masked.reindexing).toBe(true);
// The status payload must carry the run identity so the client can key its
// poll on it (a changed runId => a NEW run).
expect(masked.runId).toBe('run-abc');
expect(masked.reindexStartedAt).toBe(startedAt);
});
it('falls back to countIndexedPages when no reindex is active', async () => {
@@ -121,6 +127,10 @@ describe('AiSettingsService.getMasked reindex progress', () => {
expect(masked.indexedPages).toBe(478);
expect(masked.totalPages).toBe(478);
expect(masked.reindexing).toBe(false);
// No active run -> no run identity surfaced (the client keeps its prior
// steady-state behaviour).
expect(masked.runId).toBeUndefined();
expect(masked.reindexStartedAt).toBeUndefined();
});
});
@@ -371,6 +371,12 @@ export class AiSettingsService {
totalPages,
// Optional hint for the client: a reindex run is currently in progress.
reindexing: progress != null,
// Per-run identity so the client can key its poll on a stable run id and
// reset its per-run state when a NEW run starts. Present only while a run
// is active; `runId` may be '' for a legacy/degraded record (the client
// treats that as "no identity").
runId: progress?.runId,
reindexStartedAt: progress?.startedAt,
};
}
@@ -129,6 +129,12 @@ const DEFAULT_MCP_STREAM_TIMEOUT_MS = 60_000;
/** Default total wall-clock cap for ONE external MCP tool call (2 min). */
const DEFAULT_MCP_CALL_TIMEOUT_MS = 120_000;
/**
* Default `bodyTimeout` for the EXTERNAL-MCP SSE transport (10 min) #489.
* Deliberately much LARGER than {@link DEFAULT_MCP_STREAM_TIMEOUT_MS}.
*/
const DEFAULT_MCP_SSE_BODY_TIMEOUT_MS = 600_000;
/**
* SILENCE timeout (ms) for EXTERNAL-MCP transport ONLY. Override with
* `AI_MCP_STREAM_TIMEOUT_MS`; a missing/invalid/non-positive value falls back to
@@ -164,6 +170,26 @@ export function mcpCallTimeoutMs(): number {
return positiveEnv('AI_MCP_CALL_TIMEOUT_MS', DEFAULT_MCP_CALL_TIMEOUT_MS);
}
/**
* `bodyTimeout` (ms) for the EXTERNAL-MCP **SSE** transport ONLY #489. Override
* with `AI_MCP_SSE_BODY_TIMEOUT_MS`; a missing/invalid/non-positive value falls
* back to {@link DEFAULT_MCP_SSE_BODY_TIMEOUT_MS} (10 min).
*
* The SSE transport holds ONE long-lived response body open across many tool
* calls, so undici's `bodyTimeout` (time between body bytes) counts the LEGITIMATE
* silence BETWEEN calls, not just a hung single call. At the tight HTTP silence
* timeout ({@link mcpStreamTimeoutMs}, 1 min) a normal >1-min gap between the
* model's tool calls would break the SSE socket, and the cache would then serve a
* dead client until TTL. So the SSE transport gets its OWN, RAISED bodyTimeout;
* the per-call total cap ({@link mcpCallTimeoutMs}) still bounds a single stuck
* call, and the app-level transport-error retry heals a socket that does break.
* The HTTP (streamable) transport keeps the tight timeout it opens a fresh
* request per call, so idle-between-calls does not apply there.
*/
export function mcpSseBodyTimeoutMs(): number {
return positiveEnv('AI_MCP_SSE_BODY_TIMEOUT_MS', DEFAULT_MCP_SSE_BODY_TIMEOUT_MS);
}
/**
* undici `Agent` options for streaming AI traffic the (generous, finite)
* silence timeouts plus the keep-alive recycle window. Shared by the chat
@@ -0,0 +1,86 @@
// `.provider` alone cannot prove the gemini/ollama chat factories were built
// with the instrumented streaming fetch — a regression dropping it (which drops
// them back to the global undici fetch: no keep-alive recycle, no reset retries,
// unbounded silence timeout; incident classes #140/#175/#310) would still pass.
// So mock the factories and assert the exact fetch argument. jest.mock is
// module-scoped, hence a dedicated file.
const mockGeminiModel = { provider: 'google.generative-ai', modelId: 'm' };
const mockOllamaModel = { provider: 'ollama.chat', modelId: 'm' };
// jest allows `mock`-prefixed vars inside a jest.mock factory.
const mockCreateGoogle = jest.fn((_settings: unknown) => () => mockGeminiModel);
const mockCreateOllama = jest.fn((_settings: unknown) => () => mockOllamaModel);
jest.mock('@ai-sdk/google', () => ({
createGoogleGenerativeAI: (settings: unknown) => mockCreateGoogle(settings),
}));
jest.mock('ai-sdk-ollama', () => ({
createOllama: (settings: unknown) => mockCreateOllama(settings),
}));
import { AiService } from './ai.service';
describe('AiService.getChatModel provider transport fetch (gemini/ollama)', () => {
function serviceWith(cfg: Record<string, unknown>) {
const aiSettings = {
resolve: jest.fn().mockResolvedValue(cfg),
};
return new AiService(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
aiSettings as any,
{ find: jest.fn() } as never,
{ decryptSecret: jest.fn() } as never,
);
}
beforeEach(() => {
mockCreateGoogle.mockClear();
mockCreateOllama.mockClear();
});
it('builds the gemini chat model with the instrumented streaming fetch', async () => {
await serviceWith({
driver: 'gemini',
chatModel: 'gemini-2.5-pro',
apiKey: 'the-key',
}).getChatModel('ws-1');
expect(mockCreateGoogle).toHaveBeenCalledTimes(1);
expect(mockCreateGoogle).toHaveBeenCalledWith(
expect.objectContaining({
apiKey: 'the-key',
fetch: expect.any(Function),
}),
);
});
it('builds the ollama chat model with the instrumented streaming fetch', async () => {
await serviceWith({
driver: 'ollama',
chatModel: 'llama3',
baseUrl: 'http://localhost:11434/api',
}).getChatModel('ws-1');
expect(mockCreateOllama).toHaveBeenCalledTimes(1);
expect(mockCreateOllama).toHaveBeenCalledWith(
expect.objectContaining({
baseURL: 'http://localhost:11434/api',
fetch: expect.any(Function),
}),
);
});
it('reuses ONE service-lifetime fetch instance across both providers', async () => {
const svc = serviceWith({
driver: 'gemini',
chatModel: 'gemini-2.5-pro',
apiKey: 'k',
});
await svc.getChatModel('ws-1');
const geminiFetch = mockCreateGoogle.mock.calls[0][0] as { fetch: unknown };
// Same instance on a second call — the fetch is held for the service
// lifetime to reuse the streaming dispatcher's connection pool.
await svc.getChatModel('ws-1');
const geminiFetch2 = mockCreateGoogle.mock.calls[1][0] as { fetch: unknown };
expect(geminiFetch.fetch).toBe(geminiFetch2.fetch);
});
});
+15 -3
View File
@@ -190,10 +190,22 @@ export class AiService {
}).chat(chatModel);
}
case 'gemini':
return createGoogleGenerativeAI({ apiKey })(chatModel);
// Route gemini through the same instrumented streaming fetch as openai
// (finite silence timeouts + keep-alive recycling + pre-response
// connection-reset retry). Without it the provider ran on the global
// undici fetch — no keep-alive recycle, no reset retries, default
// (unbounded silence) timeout — so incident classes #140/#175/#310 were
// reproducible for gemini too.
return createGoogleGenerativeAI({
apiKey,
fetch: this.aiProviderFetch,
})(chatModel);
case 'ollama':
// Ollama needs no API key.
return createOllama({ baseURL: baseUrl })(chatModel);
// Ollama needs no API key. Same transport hardening as above (#140/#175/#310).
return createOllama({
baseURL: baseUrl,
fetch: this.aiProviderFetch,
})(chatModel);
default:
throw new AiNotConfiguredException();
}
@@ -149,4 +149,14 @@ export interface MaskedAiSettings {
// True while a full workspace reindex is actively running (the counts above
// then reflect the live run progress rather than the steady-state DB count).
reindexing?: boolean;
// Identity of the ACTIVE reindex run (present only while `reindexing`). The
// client keys its poll on `runId`: a changed value means a NEW run (reset the
// per-run poll state it latched), the same value means the run it is already
// watching — removing the "same run or a fresh one?" ambiguity a stale
// pre-reindex snapshot otherwise causes. Absent/empty degrades gracefully.
runId?: string;
// Epoch-ms the active run started (present only while `reindexing`). Paired
// with `runId` so a run that restarts with the same (recycled) id is still
// seen as new.
reindexStartedAt?: number;
}
@@ -48,19 +48,38 @@ describe('EmbeddingReindexProgressService', () => {
}
describe('get', () => {
it('maps a valid hash to a ReindexProgress object', async () => {
it('maps a valid hash to a ReindexProgress object (incl. the run identity)', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({ total: '478', done: '120', startedAt: '1000' });
hgetall.mockResolvedValue({
total: '478',
done: '120',
startedAt: '1000',
runId: 'run-xyz',
});
const service = makeService(redis);
await expect(service.get(WORKSPACE_ID)).resolves.toEqual({
total: 478,
done: 120,
startedAt: 1000,
runId: 'run-xyz',
});
expect(hgetall).toHaveBeenCalledWith(KEY);
});
it('degrades a missing runId to an empty string (legacy/partial record)', async () => {
const { redis, hgetall } = makeRedis();
// A record written before runId existed: get() must still succeed and
// report runId='' so the client treats it as "no identity", never breaks.
hgetall.mockResolvedValue({ total: '10', done: '3', startedAt: '5' });
await expect(makeService(redis).get(WORKSPACE_ID)).resolves.toEqual({
total: 10,
done: 3,
startedAt: 5,
runId: '',
});
});
it('returns null for an empty hash (no record)', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({});
@@ -87,11 +106,17 @@ describe('EmbeddingReindexProgressService', () => {
it('coerces a non-finite startedAt to 0', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({ total: '10', done: '2', startedAt: 'nope' });
hgetall.mockResolvedValue({
total: '10',
done: '2',
startedAt: 'nope',
runId: 'run-1',
});
await expect(makeService(redis).get(WORKSPACE_ID)).resolves.toEqual({
total: 10,
done: 2,
startedAt: 0,
runId: 'run-1',
});
});
@@ -115,6 +140,21 @@ describe('EmbeddingReindexProgressService', () => {
expect(multiObj.exec).toHaveBeenCalledTimes(1);
});
it('mints a fresh non-empty runId into the record on each start', async () => {
const { redis, multiObj } = makeRedis();
const service = makeService(redis);
await service.start(WORKSPACE_ID, 1);
await service.start(WORKSPACE_ID, 1);
const firstRunId = multiObj.hset.mock.calls[0][1].runId;
const secondRunId = multiObj.hset.mock.calls[1][1].runId;
expect(typeof firstRunId).toBe('string');
expect(firstRunId).not.toBe('');
// Each run gets its OWN identity so the client can tell a re-trigger apart
// from the run it is already watching.
expect(secondRunId).not.toBe(firstRunId);
});
it('defaults the expire TTL to the full 1h record TTL', async () => {
const { redis, multiObj } = makeRedis();
await makeService(redis).start(WORKSPACE_ID, 478);
@@ -1,17 +1,28 @@
import { Injectable, Logger } from '@nestjs/common';
import { RedisService } from '@nestjs-labs/nestjs-ioredis';
import { randomUUID } from 'node:crypto';
import type { Redis } from 'ioredis';
/**
* Live progress of an in-flight workspace embeddings reindex run.
* `total` is the number of pages the run will process, `done` how many it has
* already processed (success OR handled failure), `startedAt` the epoch-ms the
* record was created.
* record was created, and `runId` a per-run identity minted at `start()`.
*
* `runId` gives each reindex run a stable identity so a poller can tell "same
* run I've been watching" from "a NEW run started" WITHOUT guessing from the
* progress counters (the ambiguity that a stale pre-reindex snapshot vs a fresh
* run otherwise causes the bug class fixed twice under #262). It is best-
* effort like the rest of this record: a record written before this field
* existed (or a Redis hiccup) yields an empty `runId`, which the client must
* treat as "no identity available" and degrade to its prior behaviour, never
* break.
*/
export interface ReindexProgress {
total: number;
done: number;
startedAt: number;
runId: string;
}
/** Redis key namespace for the per-workspace reindex-progress record. */
@@ -86,12 +97,18 @@ export class EmbeddingReindexProgressService {
): Promise<void> {
const key = this.key(workspaceId);
try {
// A fresh identity per run so the client poll can key on it: a changed
// runId means a genuinely NEW run (reset any latched per-run poll state),
// the same runId means the run the client is already watching. Best-effort
// like the counters — never surfaced to the user, only used to disambiguate.
const runId = randomUUID();
await this.redis
.multi()
.hset(key, {
total: String(total),
done: '0',
startedAt: String(Date.now()),
runId,
})
.expire(key, ttlSeconds)
.exec();
@@ -150,7 +167,15 @@ export class EmbeddingReindexProgressService {
const done = Number(data.done);
const startedAt = Number(data.startedAt);
if (!Number.isFinite(total) || !Number.isFinite(done)) return null;
return { total, done, startedAt: Number.isFinite(startedAt) ? startedAt : 0 };
// `runId` degrades gracefully: a pre-existing record (written before this
// field) or a stripped value reads as '' — the client treats that as "no
// identity" and keeps its prior behaviour rather than breaking the poll.
return {
total,
done,
startedAt: Number.isFinite(startedAt) ? startedAt : 0,
runId: typeof data.runId === 'string' ? data.runId : '',
};
} catch (err) {
this.logger.warn(
`reindex-progress read failed for workspace ${workspaceId}; ` +
@@ -0,0 +1,305 @@
import { Kysely } from 'kysely';
import { AiChatMessageRepo } from '@docmost/db/repos/ai-chat/ai-chat-message.repo';
import { AiChatRunRepo } from '@docmost/db/repos/ai-chat/ai-chat-run.repo';
import { AiChatRunService } from '../../src/core/ai-chat/ai-chat-run.service';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createUser,
createChat,
createMessage,
} from './db';
/**
* #487 commit 4 bidirectional reconcile + owner-write priority, real SQL.
*
* Proves the OBSERVABLE recovery properties against docmost_test:
* - the CONDITIONAL owner-write beats a reconcile stamp, and a stamp never
* clobbers a proper terminal row;
* - a LATE owner-finalize with real content OVERWRITES a reconcile 'aborted'
* stamp (finalizeFailed);
* - each reconcile clause (b message<-run, c stale-run, d historical row) settles
* the stuck row/run, and a LIVE run entry is never touched;
* - the "kill DB on finish" recovery: after the DB comes back, neither the
* message row nor the run row stays stuck.
*/
describe('#487 reconcile + owner-write priority [integration]', () => {
let db: Kysely<any>;
let messageRepo: AiChatMessageRepo;
let runRepo: AiChatRunRepo;
let runService: AiChatRunService;
let workspaceId: string;
let userId: string;
beforeAll(async () => {
db = getTestDb();
messageRepo = new AiChatMessageRepo(db as any);
runRepo = new AiChatRunRepo(db as any);
runService = new AiChatRunService(runRepo, { isCloud: () => false } as never);
workspaceId = (await createWorkspace(db)).id;
userId = (await createUser(db, workspaceId)).id;
});
afterAll(async () => {
await destroyTestDb();
});
const newChat = async () =>
(await createChat(db, { workspaceId, creatorId: userId })).id;
const metaOf = async (id: string): Promise<Record<string, unknown> | null> => {
const row = await messageRepo.findById(id, workspaceId);
return (row?.metadata as Record<string, unknown> | null) ?? null;
};
it('owner finalizeOwner writes a streaming row and CLEARS finalizeFailed', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
const wrote = await messageRepo.finalizeOwner(m.id, workspaceId, {
content: 'final answer',
status: 'completed',
metadata: { parts: [{ type: 'text', text: 'final answer' }] },
} as never);
expect(wrote!.status).toBe('completed');
expect((await metaOf(m.id))?.finalizeFailed).toBeUndefined();
});
it('a reconcile stamp NEVER clobbers a proper terminal row (finalizeOwner is a no-op there)', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'completed',
content: 'real',
metadata: { parts: [] },
});
// The reconcile stamp is onlyIfStreaming -> no-op on a completed row.
const stamped = await messageRepo.stampTerminalIfStreaming(
m.id,
workspaceId,
'aborted',
);
expect(stamped).toBeUndefined();
expect((await messageRepo.findById(m.id, workspaceId))!.status).toBe(
'completed',
);
});
it('LATE owner-finalize with real content OVERWRITES a reconcile aborted stamp', async () => {
const chatId = await newChat();
const m = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [{ type: 'text', text: 'partial' }] },
});
// Reconcile stamps it aborted + finalizeFailed (final text lived only in mem).
const stamped = await messageRepo.stampTerminalIfStreaming(
m.id,
workspaceId,
'aborted',
);
expect(stamped!.status).toBe('aborted');
expect((await metaOf(m.id))?.finalizeFailed).toBe(true);
// A LATE owner-write (finalizeFailed=true satisfies the OR) overwrites it with
// real content, clearing the flag — owner-write priority.
const wrote = await messageRepo.finalizeOwner(m.id, workspaceId, {
content: 'the real final answer',
status: 'completed',
metadata: { parts: [{ type: 'text', text: 'the real final answer' }] },
} as never);
expect(wrote!.status).toBe('completed');
expect(wrote!.content).toBe('the real final answer');
expect((await metaOf(m.id))?.finalizeFailed).toBeUndefined();
});
it('clause (c): a stale active run with NO live entry -> aborted; a LIVE entry is untouched', async () => {
// Stale run, NOT owned by this replica (no entry) -> reconcile aborts it.
const staleChat = await newChat();
const stale = await runRepo.insert({
chatId: staleChat,
workspaceId,
createdBy: userId,
status: 'running',
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', stale.id)
.execute();
// A live run OWNED by this replica (beginRun registers an in-memory entry),
// ALSO backdated stale — the "no entry" primary gate must protect it.
const liveChat = await newChat();
const live = await runService.beginRun({
chatId: liveChat,
workspaceId,
userId,
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', live.runId)
.execute();
const aborted = await runService.reconcileStaleRuns(15 * 60 * 1000);
expect(aborted).toBeGreaterThanOrEqual(1);
expect((await runRepo.findById(stale.id, workspaceId))!.status).toBe(
'aborted',
);
// The live entry is NEVER aborted, however stale its row looks.
expect((await runRepo.findById(live.runId, workspaceId))!.status).toBe(
'running',
);
expect(runService.isLocallyActive(live.runId)).toBe(true);
// cleanup the live run
await runService.finalizeRun(live.runId, workspaceId, 'aborted');
});
it('clause (b): a streaming message whose RUN is terminal is stamped by run status (succeeded -> aborted, NOT completed-empty)', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
// A SUCCEEDED run linked to the still-streaming message (the asymmetry).
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
assistantMessageId: msg.id,
});
await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'succeeded',
error: null,
});
const stuck = await messageRepo.findStreamingWithTerminalRun();
const mine = stuck.find((s) => s.messageId === msg.id);
expect(mine?.runStatus).toBe('succeeded');
// Reconcile clause (b): succeeded run -> message 'aborted' (NOT 'completed'),
// the final text lived only in memory (documented loss), +finalizeFailed.
const status = mine!.runStatus === 'failed' ? 'error' : 'aborted';
await messageRepo.stampTerminalIfStreaming(msg.id, workspaceId, status);
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
it('clause (d): a stale streaming row with NO active run on the chat -> aborted+finalizeFailed', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
const swept = await messageRepo.sweepStreamingWithoutActiveRun(
15 * 60 * 1000,
);
expect(swept).toBeGreaterThanOrEqual(1);
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
it('clause (d) is DOUBLE-GATED: a stale streaming row WITH an active run on the chat is left alone', async () => {
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [] },
});
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
// An ACTIVE run on the same chat -> clause (d) must NOT touch the message.
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
});
await messageRepo.sweepStreamingWithoutActiveRun(15 * 60 * 1000);
expect((await messageRepo.findById(msg.id, workspaceId))!.status).toBe(
'streaming',
);
await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'aborted',
error: null,
});
});
it('"kill DB on finish" recovery: after the DB is back, reconcile leaves NEITHER the row nor the run stuck', async () => {
// Simulate a process that seeded the assistant row + run, then died before
// finalizing EITHER (a mid-turn crash): a streaming message + a running run,
// both stale, with no in-memory entry (fresh service = fresh maps).
const chatId = await newChat();
const msg = await createMessage(db, {
workspaceId,
chatId,
role: 'assistant',
status: 'streaming',
metadata: { parts: [{ type: 'text', text: 'partial' }] },
});
const run = await runRepo.insert({
chatId,
workspaceId,
createdBy: userId,
status: 'running',
assistantMessageId: msg.id,
});
await db
.updateTable('aiChatRuns')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', run.id)
.execute();
await db
.updateTable('aiChatMessages')
.set({ updatedAt: new Date(Date.now() - 60 * 60 * 1000) })
.where('id', '=', msg.id)
.execute();
// Reconcile (as the periodic job would): (c) aborts the orphan run, then
// (b) settles the message from the now-terminal run.
await runService.reconcileStaleRuns(15 * 60 * 1000);
const stuck = await messageRepo.findStreamingWithTerminalRun();
for (const s of stuck) {
const status = s.runStatus === 'failed' ? 'error' : 'aborted';
await messageRepo.stampTerminalIfStreaming(s.messageId, s.workspaceId, status);
}
// Neither is stuck: the run is terminal AND the message is terminal.
expect((await runRepo.findById(run.id, workspaceId))!.status).toBe('aborted');
const row = await messageRepo.findById(msg.id, workspaceId);
expect(row!.status).toBe('aborted');
expect((row!.metadata as Record<string, unknown>).finalizeFailed).toBe(true);
});
});
@@ -281,6 +281,52 @@ describe('AiChatRun durable lifecycle [integration]', () => {
});
});
it('#487 finalizeIfActive is CONDITIONAL: a late terminal write cannot clobber the settled status (real SQL)', async () => {
const c = (await createChat(db, { workspaceId, creatorId: userId })).id;
const run = await runRepo.insert({
chatId: c,
workspaceId,
createdBy: userId,
status: 'running',
});
// First terminal write: the run IS active, so it flips + returns the row.
const first = await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'succeeded',
error: null,
});
expect(first!.status).toBe('succeeded');
expect(first!.finishedAt).toBeTruthy();
// A late/second writer tries to flip it to 'aborted' — the WHERE status IN
// ('pending','running') guard matches NOTHING now, so it is a benign no-op.
const second = await runRepo.finalizeIfActive(run.id, workspaceId, {
status: 'aborted',
error: 'late clobber attempt',
});
expect(second).toBeUndefined();
// The persisted terminal status is UNCHANGED — last-writer-wins is gone.
const row = await runRepo.findById(run.id, workspaceId);
expect(row!.status).toBe('succeeded');
expect(row!.error).toBeNull();
});
it('#487 double-settle through the service collapses to one write at the SQL gate', async () => {
const c = (await createChat(db, { workspaceId, creatorId: userId })).id;
const handle = await service.beginRun({ chatId: c, workspaceId, userId });
// First settle writes 'aborted' via the conditional write.
await service.finalizeRun(handle.runId, workspaceId, 'aborted');
// A late safety-net settle to 'error' is a no-op (row already terminal).
await service.finalizeRun(handle.runId, workspaceId, 'error', 'late');
const row = await runRepo.findById(handle.runId, workspaceId);
expect(row!.status).toBe('aborted');
expect(service.isLocallyActive(handle.runId)).toBe(false);
expect(service.hasZombie(handle.runId)).toBe(false);
});
it('sweepRunning() with NO args (boot sweep / variant C) aborts even a FRESH running run', async () => {
// F1/DECISION C at the SQL level: the unconditional boot sweep has NO
// staleness window, so a run updated just now (a fast restart) is settled too
@@ -33,6 +33,11 @@ describe('share_aliases one-per-page invariant [integration]', () => {
const pageRepo = {
findById: async (id: string) => ({ id, title: `title-${id}` }),
};
// The requester can view the target page (permissive), so the reassign 409 may
// include its title — these tests exercise the one-per-page invariant, not the
// #495 disclosure gate (that is unit-tested in share-alias.service.spec.ts).
const pageAccessService = { validateCanView: async () => {} };
const USER = { id: 'u-int' } as any;
beforeAll(async () => {
db = getTestDb();
@@ -41,6 +46,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
repo as any,
pageRepo as any,
{} as any, // shareService — unused by setAlias
pageAccessService as any,
db as any,
);
wsId = (await createWorkspace(db)).id;
@@ -188,6 +194,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId,
user: USER,
alias: 'te',
});
expect(first.alias).toBe('te');
@@ -196,6 +203,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId,
user: USER,
alias: 'ted',
});
// Same row id — a RENAME, not a new insert.
@@ -217,12 +225,14 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'hello',
});
const again = await service.setAlias({
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'hello',
});
expect(again.id).toBe(inserted.id);
@@ -244,6 +254,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
flakyRepo as any,
pageRepo as any,
{} as any,
pageAccessService as any,
db as any,
);
@@ -252,6 +263,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'rollback-me',
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -275,6 +287,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageA,
creatorId: null as any,
user: USER,
alias: 'shared',
});
@@ -283,6 +296,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared',
}),
).rejects.toBeInstanceOf(ConflictException);
@@ -291,6 +305,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared',
confirmReassign: true,
});
@@ -317,12 +332,14 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageA,
creatorId: null as any,
user: USER,
alias: 'shared-target',
});
await service.setAlias({
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'bee',
});
@@ -330,6 +347,7 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared-target',
confirmReassign: true,
});
+6
View File
@@ -33,6 +33,12 @@ import { TransformsMixin, type ITransformsMixin } from "./client/transforms.js";
export type { DocmostMcpConfig, SandboxPut } from "./client/context.js";
export { formatDocmostAxiosError, assertFullUuid } from "./client/errors.js";
// Branded canonical page-identity type (#435): the internal page UUID is a
// distinct nominal type so an unresolved raw/slug string can't be swapped into
// the seams that require the canonical id (see lib/page-id.ts). Re-exported on
// the package surface for hosts that type against the resolved id.
export type { PageId } from "./lib/page-id.js";
// The full public + shared instance surface of the assembled client. Built by
// INTERSECTING each domain mixin's public interface (each DERIVED from its class
// and enforced by that class's `implements` clause — issue #446, no hand-mirror)
+76 -6
View File
@@ -24,6 +24,7 @@ import {
isCollabAuthFailedError,
} from "../lib/collab-session.js";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import type { PageId } from "../lib/page-id.js";
import { getCollabToken, performLogin } from "../lib/auth-utils.js";
import { formatDocmostAxiosError } from "./errors.js";
import { GetPageConversionCache } from "./getpage-cache.js";
@@ -170,6 +171,43 @@ export abstract class DocmostClientContext {
// cached conversion can never leak across identities. See getpage-cache.ts.
protected getPageCache = new GetPageConversionCache();
// #487: an OPTIONAL abort signal the in-app tool host sets before each tool
// call (a composite of the turn's Stop signal + a per-call wall-clock cap). It
// is checked at safe-points BETWEEN the sequential HTTP calls of a paginated
// read (paginateAll) and just before the atomic collab commit of a write (the
// mutatePage/replacePage/mutateLiveContentUnlocked seams), so a Stop / cap
// stops the NEXT network call from STARTING. An already-started single call may
// still land — a documented limitation (#487).
//
// SINGLE-WRITER by phase-1 assumption: exactly one DocmostClient is built per
// turn and shared by every tool call; the host sets this per call and does NOT
// restore the prior value on unwind (set-and-leave) — a fresh client per turn
// plus overwrite-by-the-next-call keeps it correct, and leaving a settled
// call's signal in place is what makes a discarded race-loser throw on its
// next safe-point. If the model emits PARALLEL in-app
// tool calls they share this one field, so the per-call CAP of one call is not
// guaranteed to bound another's in-flight pagination — but every composite the
// host sets carries the SAME turn Stop signal, so a Stop still aborts whichever
// signal is current. #487.
protected toolAbortSignal: AbortSignal | null = null;
/**
* #487: set (or clear with null) the in-app tool abort signal governing the
* NEXT client call's safe-points. The host wraps each in-app tool call: it sets
* the composite (Stop + per-call cap) here before invoking the tool and leaves
* it in place afterwards (set-and-leave, NOT restored) the next call
* overwrites it, and a fresh client is built per turn. Public so the
* server-side tool wrapper can reach it; harmless (a no-op) when never set.
*/
public setToolAbortSignal(signal: AbortSignal | null): void {
this.toolAbortSignal = signal;
}
/** #487: the abort signal currently governing this client's safe-points. */
public getToolAbortSignal(): AbortSignal | null {
return this.toolAbortSignal;
}
// Two construction forms:
// - new DocmostClient(config) // discriminated union (current)
// - new DocmostClient(baseURL, email, password) // legacy positional creds
@@ -571,6 +609,10 @@ export abstract class DocmostClientContext {
this.onMetricFn?.("collab_connect_timeouts_total", 1),
});
try {
// #487 PRE-COMMIT safe-point (reentrant twin of mutatePageContent): a
// Stop/cap after acquiring the session but before the atomic write skips
// this commit. Same limitation applies (stops the NEXT commit only).
this.toolAbortSignal?.throwIfAborted();
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh.
@@ -602,6 +644,11 @@ export abstract class DocmostClientContext {
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
// #487 safe-point: a Stop (or the in-app tool per-call cap) that fires
// BETWEEN sequential page fetches must stop the NEXT request from starting
// — a read tool that would otherwise paginate for minutes is interrupted
// here. throwIfAborted() rejects with the signal's reason.
this.toolAbortSignal?.throwIfAborted();
const payload: Record<string, any> = {
...basePayload,
limit: clampedLimit,
@@ -669,10 +716,18 @@ export abstract class DocmostClientContext {
* once via getPageRaw and cached (both slugId->uuid and uuid->uuid), so
* repeated edits on the same page add no extra request.
*/
protected async resolvePageId(pageId: string): Promise<string> {
if (isUuid(pageId)) return pageId;
protected async resolvePageId(pageId: string): Promise<PageId> {
// This is the ONE canonicalization seam, so it is where the `PageId` brand
// is minted (#435). The value is validated here — a UUID input by isUuid, a
// resolved id as the server's own page.id — so the downstream write path
// (withPageLock / mutatePageContent) can require the brand and reject any
// unresolved raw id at compile time. The brand is a pure compile-time marker
// applied by cast (no runtime guard): the guarantee is that this seam is the
// only place a `PageId` is produced, so every branded value went through the
// UUID/resolve check above.
if (isUuid(pageId)) return pageId as PageId;
const cached = this.pageIdCache.get(pageId);
if (cached) return cached;
if (cached) return cached as PageId;
const data = await this.getPageRaw(pageId);
const uuid = data?.id;
if (typeof uuid !== "string" || !uuid) {
@@ -681,7 +736,7 @@ export abstract class DocmostClientContext {
);
}
this.pageIdCache.set(pageId, uuid);
return uuid;
return uuid as PageId;
}
@@ -709,7 +764,15 @@ export abstract class DocmostClientContext {
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
mutatePageContent(pageUuid, token, apiUrl, transform),
// #487: thread the in-app tool signal to mutatePageContent's pre-commit
// safe-point so a Stop/cap during the connect/lock window skips the write.
mutatePageContent(
pageUuid,
token,
apiUrl,
transform,
this.toolAbortSignal ?? undefined,
),
);
}
@@ -733,7 +796,14 @@ export abstract class DocmostClientContext {
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
replacePageContent(pageUuid, doc, token, apiUrl),
// #487: same pre-commit safe-point as mutatePage, for full-document writes.
replacePageContent(
pageUuid,
doc,
token,
apiUrl,
this.toolAbortSignal ?? undefined,
),
);
}
+7
View File
@@ -30,6 +30,13 @@ export { destroyAllSessions } from "./lib/collab-session.js";
// internals directly; it goes through loadDocmostMcp()).
export { SHARED_TOOL_SPECS } from "./tool-specs.js";
export type { SharedToolSpec } from "./tool-specs.js";
// #489 — write-class registry consumed by the in-app external-MCP retry gate.
export {
SHARED_TOOL_WRITE_CLASS,
isRetryableWriteClass,
assertEverySpecDeclaresWriteClass,
} from "./tool-specs.js";
export type { ToolWriteClass } from "./tool-specs.js";
// Re-export the build-time REGISTRY_STAMP (issue #447): a deterministic hash of
// the tool-specs registry content, generated into src/registry-stamp.generated.ts
+23 -3
View File
@@ -13,6 +13,7 @@ import { JSDOM } from "jsdom";
import { markdownToProseMirror } from "@docmost/prosemirror-markdown";
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import type { PageId } from "./page-id.js";
import {
sanitizeForYjs,
findUnstorableAttr,
@@ -250,10 +251,19 @@ export function assertYjsEncodable(doc: any): void {
* read->write window, and it never throws (it can NEVER break a write).
*/
export async function mutatePageContent(
pageId: string,
// Canonical UUID only (#260/#435): the brand forces every caller to
// resolvePageId() BEFORE this seam so the lock + CollabSession key can never
// be a raw slugId.
pageId: PageId,
collabToken: string,
baseUrl: string,
transform: (liveDoc: any) => any | null,
// #487: optional abort signal carrying the turn's Stop + the in-app tool
// per-call cap. Checked as the PRE-COMMIT safe-point below (after the session
// is acquired, immediately before the atomic read->write), so a Stop that
// arrives during the connect/lock window stops THIS write from landing. See the
// limitation note at the check.
signal?: AbortSignal,
): Promise<MutationResult> {
return withPageLock(pageId, async () => {
if (process.env.DEBUG) {
@@ -266,6 +276,13 @@ export async function mutatePageContent(
const session = await acquireCollabSession(pageId, collabToken, baseUrl);
try {
// #487 PRE-COMMIT safe-point: if the turn was Stopped (or the in-app tool
// per-call cap fired) after we acquired the collab session but before the
// atomic write, throw NOW so this commit never runs. KNOWN LIMITATION
// (#487): this only stops THIS commit — a write tool that already committed
// an EARLIER call this turn leaves that op applied. Cancel guarantees "no
// NEW commit starts", NOT "the write didn't land".
signal?.throwIfAborted();
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh (this
@@ -287,10 +304,12 @@ export async function mutatePageContent(
* mutatePageContent.
*/
export async function replacePageContent(
pageId: string,
pageId: PageId,
prosemirrorDoc: any,
collabToken: string,
baseUrl: string,
// #487: threaded straight to mutatePageContent's pre-commit safe-point.
signal?: AbortSignal,
): Promise<MutationResult> {
// Fail fast on a bad document instead of deferring the failure into the
// collaboration write (where TiptapTransformer.toYdoc(undefined) used to
@@ -307,6 +326,7 @@ export async function replacePageContent(
collabToken,
baseUrl,
() => prosemirrorDoc,
signal,
);
}
@@ -316,7 +336,7 @@ export async function replacePageContent(
* Tables and :::callout::: blocks survive thanks to the full schema.
*/
export async function updatePageContentRealtime(
pageId: string,
pageId: PageId,
markdownContent: string,
collabToken: string,
baseUrl: string,
+18
View File
@@ -0,0 +1,18 @@
/**
* Branded canonical page-identity type for the MCP client (incident family #435).
*
* A Docmost page has TWO identities that are both plain strings: the internal
* `page.id` (a canonical UUID the server generates as UUIDv7) and the public
* `slugId` (a 10-char nanoid used in URLs). Because both are bare `string`s they
* were passed around interchangeably and silently swapped e.g. locking/keying a
* collab doc by the slugId instead of the UUID (the #260 data-loss).
*
* `PageId` brands the CANONICAL id as a distinct nominal type so a raw/unresolved
* string cannot flow into the seams that REQUIRE the canonical id (resolvePageId's
* result, the per-page lock key, the collab write entrypoints) those become a
* COMPILE error, catching a swap at build time. It is still a `string` at runtime
* (assignable INTO any `string` parameter unchanged), so branding flows outward
* for free; the brand is minted at the single canonicalization seam
* (`resolvePageId`, via `as PageId`).
*/
export type PageId = string & { readonly __brand: "PageId" };
+8 -3
View File
@@ -8,6 +8,8 @@
* failure) before it runs. Different pages never block each other.
*/
import type { PageId } from "./page-id.js";
const chains = new Map<string, Promise<unknown>>();
// Canonical UUID shape (versions 1–8, matching the `uuid` package's `validate`
@@ -28,11 +30,14 @@ export function isUuid(value: string): boolean {
// awaited/handled by the caller; only the internal chaining tail swallows
// errors (purely to gate ordering).
export function withPageLock<T>(
pageId: string,
pageId: PageId,
fn: () => Promise<T>,
): Promise<T> {
// STRUCTURAL INVARIANT (issue #449, "resolve-then-lock"): the mutex key MUST
// be the canonical page UUID, never a raw slugId. The whole write path relies
// STRUCTURAL INVARIANT (issue #449/#435, "resolve-then-lock"): the mutex key
// MUST be the canonical page UUID, never a raw slugId. The `PageId` brand now
// enforces this at COMPILE time (a raw string / slugId no longer type-checks
// as a key); the runtime assert below stays as a backstop for untyped (JS)
// callers and the http/stdio transports. The whole write path relies
// on the lock key AND the CollabSession cache key being the resolved UUID
// (#260) — if a future write method forgot to call resolvePageId and locked
// under a slugId, two writes to the same page would take DIFFERENT mutex keys
+110
View File
@@ -127,6 +127,18 @@ export interface SharedToolSpec {
mcpName: string;
/** camelCase key in the ai-SDK tools object (the in-app layer). */
inAppKey: string;
/**
* Write-class of the tool (#489), declared on EVERY spec (a registration-time
* assert enforces completeness; `satisfies Record<string, SharedToolSpec>`
* makes it a compile error to omit). 'readOnly' = a pure read that mutates
* NOTHING durable, so it is safe to auto-retry once after a transport break.
* 'write' = anything that mutates a page/comment/share/diagram/etc a
* transport error is INDETERMINATE (the server may have applied it before the
* connection reset), so it is NEVER blind-retried (a retry would double-apply,
* the #435 incident class). Consumed by the external-MCP retry path
* (mcp-clients.service.ts) to gate its single auto-retry.
*/
writeClass: 'readOnly' | 'write';
/** Single canonical model-facing description used by both layers. */
description: string;
/**
@@ -240,6 +252,7 @@ export const SHARED_TOOL_SPECS = {
getWorkspace: {
mcpName: 'getWorkspace',
inAppKey: 'getWorkspace',
writeClass: 'readOnly',
description: 'Fetch metadata about the current workspace (name, settings).',
tier: 'core',
catalogLine: 'getWorkspace — fetch current workspace metadata (name, settings).',
@@ -249,6 +262,7 @@ export const SHARED_TOOL_SPECS = {
listSpaces: {
mcpName: 'listSpaces',
inAppKey: 'listSpaces',
writeClass: 'readOnly',
description:
'List the spaces the current user can access. Returns the array of ' +
'spaces (id, name, slug, ...).',
@@ -260,6 +274,7 @@ export const SHARED_TOOL_SPECS = {
listShares: {
mcpName: 'listShares',
inAppKey: 'listShares',
writeClass: 'readOnly',
description:
'List all public shares in the workspace with page titles and public URLs.',
tier: 'deferred',
@@ -272,6 +287,7 @@ export const SHARED_TOOL_SPECS = {
getPageJson: {
mcpName: 'getPageJson',
inAppKey: 'getPageJson',
writeClass: 'readOnly',
description:
'Get page details with the raw ProseMirror JSON content (lossless: ' +
'includes block ids, callouts, tables, link/image attributes) plus the ' +
@@ -289,6 +305,7 @@ export const SHARED_TOOL_SPECS = {
getOutline: {
mcpName: 'getOutline',
inAppKey: 'getOutline',
writeClass: 'readOnly',
description:
"Return a COMPACT outline of a page's top-level blocks ({index, type, " +
'id, level, firstText}; tables add rows/cols/header; lists add item ' +
@@ -309,6 +326,7 @@ export const SHARED_TOOL_SPECS = {
getNode: {
mcpName: 'getNode',
inAppKey: 'getNode',
writeClass: 'readOnly',
description:
"Fetch a single block for editing. `nodeId` is a block id from the page " +
'outline or page-JSON view (works for headings/paragraphs/callouts/images), OR ' +
@@ -350,6 +368,7 @@ export const SHARED_TOOL_SPECS = {
searchInPage: {
mcpName: 'searchInPage',
inAppKey: 'searchInPage',
writeClass: 'readOnly',
description:
'Find every occurrence of a string (or regex) INSIDE one page and get ' +
'WHERE each is — instead of pulling blocks one-by-one with getNode. ' +
@@ -413,6 +432,7 @@ export const SHARED_TOOL_SPECS = {
deleteNode: {
mcpName: 'deleteNode',
inAppKey: 'deleteNode',
writeClass: 'write',
description:
'Remove a single block by its attrs.id (from the page outline or ' +
'page-JSON view) WITHOUT resending the whole document.',
@@ -438,6 +458,7 @@ export const SHARED_TOOL_SPECS = {
patchNode: {
mcpName: 'patchNode',
inAppKey: 'patchNode',
writeClass: 'write',
description:
'Replace a single content block identified by its attrs.id, WITHOUT ' +
'resending the whole document; the replacement keeps the same block id. ' +
@@ -505,6 +526,7 @@ export const SHARED_TOOL_SPECS = {
insertNode: {
mcpName: 'insertNode',
inAppKey: 'insertNode',
writeClass: 'write',
description:
'Insert content before/after another block (by attrs.id or anchor text) ' +
'or append it at the end (top level). For before/after you MUST provide ' +
@@ -597,6 +619,7 @@ export const SHARED_TOOL_SPECS = {
sharePage: {
mcpName: 'sharePage',
inAppKey: 'sharePage',
writeClass: 'write',
// CANONICAL: merges the MCP copy's URL-format + idempotency detail with the
// in-app copy's reversibility note; keeps the security framing both had.
description:
@@ -626,6 +649,7 @@ export const SHARED_TOOL_SPECS = {
unsharePage: {
mcpName: 'unsharePage',
inAppKey: 'unsharePage',
writeClass: 'write',
description: 'Remove the public share of a page (revokes the public URL).',
tier: 'deferred',
catalogLine: "unsharePage — revoke a page's public share (removes the public URL).",
@@ -640,6 +664,7 @@ export const SHARED_TOOL_SPECS = {
diffPageVersions: {
mcpName: 'diffPageVersions',
inAppKey: 'diffPageVersions',
writeClass: 'readOnly',
description:
'Diff two versions of a page and return a Docmost-equivalent change set ' +
'(inserted/deleted text, integrity counts for images/links/tables/' +
@@ -672,6 +697,7 @@ export const SHARED_TOOL_SPECS = {
listPageHistory: {
mcpName: 'listPageHistory',
inAppKey: 'listPageHistory',
writeClass: 'readOnly',
description:
"List a page's saved versions (Docmost auto-snapshots on every save), " +
'newest first, cursor-paginated. Returns { items, nextCursor }; each ' +
@@ -693,6 +719,7 @@ export const SHARED_TOOL_SPECS = {
restorePageVersion: {
mcpName: 'restorePageVersion',
inAppKey: 'restorePageVersion',
writeClass: 'write',
description:
'Restore a page to a saved version: writes that version\'s content back ' +
'as the page\'s current content (Docmost has no restore endpoint, so ' +
@@ -713,6 +740,7 @@ export const SHARED_TOOL_SPECS = {
importPageMarkdown: {
mcpName: 'importPageMarkdown',
inAppKey: 'importPageMarkdown',
writeClass: 'write',
// IN-APP ONLY (issue #411): the external /mcp surface no longer exposes
// importPageMarkdown — the registry loop in index.ts skips inAppOnly specs,
// so this stays available to the in-app agent (round-tripping an EXPORTED
@@ -742,6 +770,7 @@ export const SHARED_TOOL_SPECS = {
copyPageContent: {
mcpName: 'copyPageContent',
inAppKey: 'copyPageContent',
writeClass: 'write',
description:
"Replace targetPageId's content with a copy of sourcePageId's content, " +
'entirely server-side — the document is NOT sent through the model. The ' +
@@ -770,6 +799,7 @@ export const SHARED_TOOL_SPECS = {
editPageText: {
mcpName: 'editPageText',
inAppKey: 'editPageText',
writeClass: 'write',
description:
"Surgical find/replace inside a page's text, preserving all block " +
'ids and marks. A find MAY cross bold/italic/link boundaries; the ' +
@@ -819,6 +849,7 @@ export const SHARED_TOOL_SPECS = {
stashPage: {
mcpName: 'stashPage',
inAppKey: 'stashPage',
writeClass: 'readOnly',
description:
'Serialize a whole page (the full ProseMirror JSON, as getPageJson ' +
'returns) into an ephemeral in-memory blob and return ONLY a short ' +
@@ -880,6 +911,7 @@ export const SHARED_TOOL_SPECS = {
getPage: {
mcpName: 'getPage',
inAppKey: 'getPage',
writeClass: 'readOnly',
description:
'Fetch a single page as Markdown by its id. Returns the page title and ' +
'its Markdown content. The converter is canonical (round-trips text and ' +
@@ -919,6 +951,7 @@ export const SHARED_TOOL_SPECS = {
listPages: {
mcpName: 'listPages',
inAppKey: 'listPages',
writeClass: 'readOnly',
description:
'List the most recent pages (ordered by updatedAt, descending), ' +
'optionally scoped to a single space. Returns a bounded list (default ' +
@@ -965,6 +998,7 @@ export const SHARED_TOOL_SPECS = {
getTree: {
mcpName: 'getTree',
inAppKey: 'getTree',
writeClass: 'readOnly',
description:
"Get a space's page hierarchy (or one subtree) as a nested tree in a " +
'SINGLE request — completely and without loss. Each node is ' +
@@ -1009,6 +1043,7 @@ export const SHARED_TOOL_SPECS = {
getPageContext: {
mcpName: 'getPageContext',
inAppKey: 'getPageContext',
writeClass: 'readOnly',
description:
'Given a pageId, get its LOCATION and immediate surroundings (metadata ' +
'only, no page content) in one call — answers "where am I / what is ' +
@@ -1038,6 +1073,7 @@ export const SHARED_TOOL_SPECS = {
createPage: {
mcpName: 'createPage',
inAppKey: 'createPage',
writeClass: 'write',
description:
'Create a new page with a Markdown body in a space, optionally under a ' +
'parent page (omit parentPageId to create at the space root). Returns ' +
@@ -1088,6 +1124,7 @@ export const SHARED_TOOL_SPECS = {
movePage: {
mcpName: 'movePage',
inAppKey: 'movePage',
writeClass: 'write',
description:
'Move a page under a new parent page, or to the space root when no ' +
'parent is given. Reversible: move it back at any time.',
@@ -1181,6 +1218,7 @@ export const SHARED_TOOL_SPECS = {
renamePage: {
mcpName: 'renamePage',
inAppKey: 'renamePage',
writeClass: 'write',
description:
'Rename a page (change its title only; the body is untouched, never ' +
'resent). Reversible: rename back at any time.',
@@ -1202,6 +1240,7 @@ export const SHARED_TOOL_SPECS = {
deletePage: {
mcpName: 'deletePage',
inAppKey: 'deletePage',
writeClass: 'write',
description:
'Move a page to the trash — SOFT delete only: the page can be restored ' +
'from trash and nothing is ever permanently deleted.',
@@ -1235,6 +1274,7 @@ export const SHARED_TOOL_SPECS = {
updatePageJson: {
mcpName: 'updatePageJson',
inAppKey: 'updatePageJson',
writeClass: 'write',
description:
"Replace a page's content with a raw ProseMirror JSON document (lossless " +
'write: preserves the block ids, callouts, tables and attributes you pass ' +
@@ -1291,6 +1331,7 @@ export const SHARED_TOOL_SPECS = {
updatePageMarkdown: {
mcpName: 'updatePageMarkdown',
inAppKey: 'updatePageMarkdown',
writeClass: 'write',
description:
"Replace a page's body with new Markdown content (and optionally its " +
'title). The whole body is re-imported from the markdown (block ids ' +
@@ -1325,6 +1366,7 @@ export const SHARED_TOOL_SPECS = {
exportPageMarkdown: {
mcpName: 'exportPageMarkdown',
inAppKey: 'exportPageMarkdown',
writeClass: 'readOnly',
// CANONICAL: the MCP copy (a strict superset of the terse in-app wording).
description:
'Export a page to a single self-contained Docmost-flavoured Markdown ' +
@@ -1373,6 +1415,7 @@ export const SHARED_TOOL_SPECS = {
createComment: {
mcpName: 'createComment',
inAppKey: 'createComment',
writeClass: 'write',
// CANONICAL: the in-app copy (the more-maintained one). It keeps the same
// rules as the MCP copy — inline-only, top-level requires a `selection`, no
// page-level comments, replies inherit the anchor, suggestedText must be
@@ -1505,6 +1548,7 @@ export const SHARED_TOOL_SPECS = {
listComments: {
mcpName: 'listComments',
inAppKey: 'listComments',
writeClass: 'readOnly',
// CANONICAL: the two copies are near-identical; the MCP copy is the
// superset (it keeps the "(pagination is handled internally)" note the
// in-app copy dropped), so it is used verbatim.
@@ -1532,6 +1576,7 @@ export const SHARED_TOOL_SPECS = {
resolveComment: {
mcpName: 'resolveComment',
inAppKey: 'resolveComment',
writeClass: 'write',
// CANONICAL: the MCP copy's richer wording, minus its reference
// to `deleteComment` (a sibling tool that does NOT exist in the in-app
// layer) — rephrased transport-neutrally per the registry convention.
@@ -1572,6 +1617,7 @@ export const SHARED_TOOL_SPECS = {
checkNewComments: {
mcpName: 'checkNewComments',
inAppKey: 'checkNewComments',
writeClass: 'readOnly',
// CANONICAL: the MCP copy (the more detailed of the two). The MCP layer's
// execute-side guard that rejects an unparseable `since` timestamp stays in
// its execute body (per-layer logic), not in the shared schema.
@@ -1651,6 +1697,7 @@ export const SHARED_TOOL_SPECS = {
tableInsertRow: {
mcpName: 'tableInsertRow',
inAppKey: 'tableInsertRow',
writeClass: 'write',
description:
'Insert a row of plain-text cells into a table. `table` is `#<index>` ' +
'from the page outline, or a block id inside it. `cells` is the text per ' +
@@ -1684,6 +1731,7 @@ export const SHARED_TOOL_SPECS = {
tableDeleteRow: {
mcpName: 'tableDeleteRow',
inAppKey: 'tableDeleteRow',
writeClass: 'write',
description:
'Delete the row at 0-based `index` from a table (`table` is `#<index>` ' +
'from the page outline, or a block id inside it). Refuses to delete the ' +
@@ -1707,6 +1755,7 @@ export const SHARED_TOOL_SPECS = {
tableUpdateCell: {
mcpName: 'tableUpdateCell',
inAppKey: 'tableUpdateCell',
writeClass: 'write',
description:
'Set the plain-text content of cell [row, col] (0-based) in a table ' +
'(`table` is `#<index>` from the page outline, or a block id inside it). ' +
@@ -1747,6 +1796,7 @@ export const SHARED_TOOL_SPECS = {
insertFootnote: {
mcpName: 'insertFootnote',
inAppKey: 'insertFootnote',
writeClass: 'write',
description:
'Insert an AUTHOR-INLINE footnote: you specify only WHERE (anchorText) ' +
'and WHAT (text). The footnote marker is placed right after anchorText in ' +
@@ -1784,6 +1834,7 @@ export const SHARED_TOOL_SPECS = {
insertImage: {
mcpName: 'insertImage',
inAppKey: 'insertImage',
writeClass: 'write',
description:
'Download an image from a web (http/https) URL and insert it into ' +
'a page in one step. By default ' +
@@ -1828,6 +1879,7 @@ export const SHARED_TOOL_SPECS = {
replaceImage: {
mcpName: 'replaceImage',
inAppKey: 'replaceImage',
writeClass: 'write',
description:
'Replace an existing image on a page with a new image fetched from a web ' +
'(http/https) URL: uploads the new file as a NEW ' +
@@ -1866,6 +1918,7 @@ export const SHARED_TOOL_SPECS = {
drawioGet: {
mcpName: 'drawioGet',
inAppKey: 'drawioGet',
writeClass: 'readOnly',
description:
'Read a draw.io diagram on a page as mxGraph XML (default) or as its raw ' +
'`.drawio.svg`. `node` is the drawio node\'s attrs.id (from getOutline / ' +
@@ -1899,6 +1952,7 @@ export const SHARED_TOOL_SPECS = {
drawioCreate: {
mcpName: 'drawioCreate',
inAppKey: 'drawioCreate',
writeClass: 'write',
description:
'Create a draw.io diagram from mxGraph XML and insert it as a diagram ' +
'block. `xml` is a bare `<mxGraphModel>` OR a list of `<mxCell>` elements ' +
@@ -1969,6 +2023,7 @@ export const SHARED_TOOL_SPECS = {
drawioUpdate: {
mcpName: 'drawioUpdate',
inAppKey: 'drawioUpdate',
writeClass: 'write',
description:
'Replace a draw.io diagram\'s content with new mxGraph XML (same lint ' +
'pipeline as drawioCreate). `baseHash` is MANDATORY: pass the hash from ' +
@@ -2020,6 +2075,7 @@ export const SHARED_TOOL_SPECS = {
drawioEditCells: {
mcpName: 'drawioEditCells',
inAppKey: 'drawioEditCells',
writeClass: 'write',
description:
'Make TARGETED, id-based edits to an existing draw.io diagram instead of ' +
'resending the whole XML (a full-XML diff is fragile — draw.io reorders ' +
@@ -2078,6 +2134,7 @@ export const SHARED_TOOL_SPECS = {
drawioFromGraph: {
mcpName: 'drawioFromGraph',
inAppKey: 'drawioFromGraph',
writeClass: 'write',
description:
'Build a draw.io diagram from a SEMANTIC graph — you describe nodes, groups ' +
'and edges by MEANING and the server picks every coordinate, color and icon ' +
@@ -2202,6 +2259,7 @@ export const SHARED_TOOL_SPECS = {
drawioFromMermaid: {
mcpName: 'drawioFromMermaid',
inAppKey: 'drawioFromMermaid',
writeClass: 'write',
description:
'Convert Mermaid `flowchart` text into an EDITABLE draw.io diagram (LLMs ' +
'write Mermaid reliably). Best for STANDARD flowcharts/decision trees: ' +
@@ -2248,6 +2306,7 @@ export const SHARED_TOOL_SPECS = {
drawioShapes: {
mcpName: 'drawioShapes',
inAppKey: 'drawioShapes',
writeClass: 'readOnly',
description:
'Look up VERIFIED draw.io stencil style-strings so you never guess a ' +
'`shape=mxgraph.*` name (a wrong name renders as an EMPTY BOX). Searches a ' +
@@ -2294,6 +2353,7 @@ export const SHARED_TOOL_SPECS = {
drawioGuide: {
mcpName: 'drawioGuide',
inAppKey: 'drawioGuide',
writeClass: 'readOnly',
description:
'Progressive-disclosure draw.io authoring reference. Call with a `section` ' +
'to pull one focused, <=4KB chapter instead of bloating context: ' +
@@ -2323,3 +2383,53 @@ export const SHARED_TOOL_SPECS = {
inlineBothHosts: true,
},
} satisfies Record<string, SharedToolSpec>;
// --- write-class registry (#489) ------------------------------------------
/** A tool's retry-safety class. 'readOnly' may be auto-retried once after a
* transport break; 'write' is indeterminate and must never be blind-retried. */
export type ToolWriteClass = 'readOnly' | 'write';
/**
* Name write-class map for the shared registry, keyed by mcpName (=== inAppKey).
* The external-MCP retry path (mcp-clients.service.ts) looks a tool up here by its
* RAW (un-namespaced) name to decide whether a transport failure may be retried.
* A tool NOT in this map (a third-party external MCP tool) is treated as 'write'
* by the consumer the safe default (never blind-retry an unknown tool).
*/
export const SHARED_TOOL_WRITE_CLASS: Record<string, ToolWriteClass> =
Object.fromEntries(
Object.values(SHARED_TOOL_SPECS).map((spec) => [spec.mcpName, spec.writeClass]),
);
/** Whether a write-class permits a single automatic retry after a transport
* break. Only a pure read is retry-safe; everything mutating is indeterminate. */
export function isRetryableWriteClass(
writeClass: ToolWriteClass | undefined,
): boolean {
return writeClass === 'readOnly';
}
/**
* Registration-time assert (#489): EVERY spec must declare a valid write-class.
* `satisfies Record<string, SharedToolSpec>` already makes an omission a compile
* error, but this guards a raw/cast construction path and documents the invariant
* at the point of use. Runs once on import both hosts import this module, so
* both get the check. Throws (fails startup) rather than silently mis-gating a
* retry in production.
*/
export function assertEverySpecDeclaresWriteClass(): void {
for (const [key, spec] of Object.entries(SHARED_TOOL_SPECS)) {
const wc = (spec as SharedToolSpec).writeClass;
if (wc !== 'readOnly' && wc !== 'write') {
throw new Error(
`tool-specs: spec "${key}" must declare writeClass ('readOnly' | 'write'), got ${JSON.stringify(
wc,
)}`,
);
}
}
}
// Enforce at module load (registration time) on both hosts.
assertEverySpecDeclaresWriteClass();
@@ -0,0 +1,143 @@
// #487 commit 1 — the in-app tool cancellation safe-point inside paginateAll.
//
// The in-app tool host sets a composite abort signal on the client
// (setToolAbortSignal) before each tool call; paginateAll checks it at a
// safe-point BEFORE every sequential page fetch, so a Stop that lands mid-read
// stops the NEXT HTTP request from STARTING (a read tool can no longer paginate
// for minutes past a Stop). This pins the HONEST observable property against the
// REAL client + a real HTTP server: "after Stop, no NEW request starts".
import { test, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { DocmostClient } from "../../build/client.js";
function readBody(req) {
return new Promise((resolve) => {
let raw = "";
req.on("data", (c) => (raw += c));
req.on("end", () => resolve(raw));
});
}
function sendJson(res, status, obj, extra = {}) {
res.writeHead(status, { "Content-Type": "application/json", ...extra });
res.end(JSON.stringify(obj));
}
const openServers = [];
async function spawn(handler) {
const server = await new Promise((resolve) => {
const s = http.createServer(handler);
s.listen(0, "127.0.0.1", () => resolve(s));
});
openServers.push(server);
const { port } = server.address();
return { baseURL: `http://127.0.0.1:${port}/api` };
}
after(async () => {
await Promise.all(openServers.map((s) => new Promise((r) => s.close(r))));
});
function handleLogin(req, res) {
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
});
return true;
}
return false;
}
// A Stop that lands DURING pagination: the server aborts the client signal as it
// serves page 1 (more pages remain). The loop's next safe-point must throw before
// the page-2 request is sent.
test("paginateAll stops the NEXT request when the signal aborts mid-pagination", async () => {
let requests = 0;
const ac = new AbortController();
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/spaces") {
requests++;
// Simulate a user Stop that lands while page 1 is in flight.
if (requests === 1) ac.abort(new Error("user stop"));
sendJson(res, 200, {
success: true,
data: {
items: [{ id: `p${requests}` }],
meta: { hasNextPage: true, nextCursor: `c${requests}` },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
client.setToolAbortSignal(ac.signal);
await assert.rejects(
() => client.paginateAll("/spaces", {}),
/user stop/,
"the aborted safe-point rejects with the signal's reason",
);
assert.equal(requests, 1, "page 2 never started after the Stop");
});
// A Stop that is already in effect before the read starts: zero requests fire.
test("paginateAll starts no request when the signal is already aborted", async () => {
let requests = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/spaces") {
requests++;
sendJson(res, 200, {
success: true,
data: { items: [], meta: { hasNextPage: false, nextCursor: null } },
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// Warm the auth so ensureAuthenticated does not itself POST after the abort.
await client.ensureAuthenticated();
const ac = new AbortController();
ac.abort(new Error("already stopped"));
client.setToolAbortSignal(ac.signal);
await assert.rejects(() => client.paginateAll("/spaces", {}), /already stopped/);
assert.equal(requests, 0, "no /spaces request started once already aborted");
});
// Without a tool signal (default), pagination is unaffected — the safe-point is a
// pure no-op, so pre-#487 behaviour is byte-identical.
test("paginateAll is unaffected when no tool signal is set", async () => {
let requests = 0;
const PAGES = {
"": { items: [{ id: "a" }], nextCursor: "c1" },
c1: { items: [{ id: "b" }], nextCursor: null },
};
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/spaces") {
requests++;
const body = JSON.parse(raw || "{}");
const page = PAGES[body.cursor ?? ""] ?? { items: [], nextCursor: null };
sendJson(res, 200, {
success: true,
data: {
items: page.items,
meta: { hasNextPage: page.nextCursor != null, nextCursor: page.nextCursor },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.paginateAll("/spaces", {});
assert.equal(requests, 2, "both pages fetched with no signal set");
assert.deepEqual(all.map((p) => p.id), ["a", "b"]);
});
@@ -0,0 +1,164 @@
// #487 F4 — the WRITE-side cancellation safe-point.
//
// Every content-mutating collab write (collaboration.mutatePageContent and the
// reentrant twin client.mutateLiveContentUnlocked used by replaceImage) checks
// the in-app tool abort signal at a PRE-COMMIT safe-point — after the collab
// session is acquired but immediately BEFORE the atomic read->write
// (session.mutate). So a Stop (or the per-call cap) that lands during the
// connect/lock window stops THIS write from landing: no new commit starts once
// aborted. paginate-abort-safepoint.test.mjs pins the READ half; this pins the
// integrity-critical WRITE half — remove the `throwIfAborted()` and the transform
// would run and the doc would be mutated past a Stop.
//
// There is no collab server in the unit env, so we swap the provider factory
// (__setCollabProviderFactory) for a fake that reports an immediate successful
// sync. That makes acquireCollabSession SUCCEED and hand back a live, ready
// session, so the ONLY thing standing between the call and session.mutate is the
// safe-point under test. The transform is instrumented to prove it never runs.
import { test, afterEach } from "node:test";
import assert from "node:assert/strict";
import { mutatePageContent } from "../../build/lib/collaboration.js";
import {
__setCollabProviderFactory,
destroyAllSessions,
} from "../../build/lib/collab-session.js";
import { DocmostClient } from "../../build/client.js";
const BASE_URL = "http://127.0.0.1:1/api";
// mutatePageContent locks via withPageLock, which demands a canonical page UUID
// (resolve-then-lock invariant, #260/#449); the unlocked twin does not.
const PAGE_UUID = "11111111-1111-4111-8111-111111111111";
// A fake HocuspocusProvider that immediately reports a successful initial sync so
// CollabSession.open() resolves to a ready session, and stays "synced" with zero
// unsynced changes so a reached session.mutate() would resolve at once. It speaks
// only the tiny CollabProviderLike surface the session depends on.
function syncedProviderFactory(config) {
// Fire the initial-sync callback so open() settles as ready.
config.onSynced();
return {
synced: true,
unsyncedChanges: 0,
destroy() {},
on() {},
off() {},
};
}
// Disable the session cache so every acquire opens (and the failure path destroys)
// its own ephemeral session — no cross-test session reuse.
process.env.MCP_COLLAB_SESSION_IDLE_MS = "0";
afterEach(() => {
__setCollabProviderFactory(null); // restore the real factory
destroyAllSessions();
});
// --- collaboration.mutatePageContent (the page-locked write path) -------------
test("mutatePageContent rejects at the pre-commit safe-point BEFORE session.mutate when the signal is already aborted", async () => {
__setCollabProviderFactory(syncedProviderFactory);
let transformCalls = 0;
const ac = new AbortController();
ac.abort(new Error("user stop"));
await assert.rejects(
() =>
mutatePageContent(
PAGE_UUID,
"collab-jwt",
BASE_URL,
(liveDoc) => {
transformCalls++;
return liveDoc;
},
ac.signal,
),
/user stop/,
"the aborted safe-point rejects with the signal's reason before committing",
);
assert.equal(
transformCalls,
0,
"the transform (and therefore session.mutate) must NEVER run once aborted",
);
});
test("mutatePageContent (control) DOES reach session.mutate and invoke the transform when the signal is live", async () => {
__setCollabProviderFactory(syncedProviderFactory);
let transformCalls = 0;
const ac = new AbortController(); // never aborted
const result = await mutatePageContent(
PAGE_UUID,
"collab-jwt",
BASE_URL,
(liveDoc) => {
transformCalls++;
return null; // null -> no-op write; still proves the transform was invoked
},
ac.signal,
);
assert.equal(
transformCalls,
1,
"with a live signal the safe-point is a no-op and session.mutate runs the transform",
);
assert.ok(result && result.verify, "a MutationResult is returned");
});
// --- client.mutateLiveContentUnlocked (the reentrant twin, replaceImage) ------
test("mutateLiveContentUnlocked rejects at the pre-commit safe-point BEFORE session.mutate when the tool signal is already aborted", async () => {
__setCollabProviderFactory(syncedProviderFactory);
const client = new DocmostClient({
apiUrl: BASE_URL,
getToken: async () => "access",
getCollabToken: async () => "collab-jwt",
});
let transformCalls = 0;
const ac = new AbortController();
ac.abort(new Error("cap fired"));
client.setToolAbortSignal(ac.signal);
await assert.rejects(
() =>
client.mutateLiveContentUnlocked("page-1", "collab-jwt", (liveDoc) => {
transformCalls++;
return liveDoc;
}),
/cap fired/,
"the aborted safe-point rejects with the signal's reason before committing",
);
assert.equal(
transformCalls,
0,
"the transform (and therefore session.mutate) must NEVER run once aborted",
);
});
test("mutateLiveContentUnlocked (control) DOES reach session.mutate and invoke the transform when the tool signal is live", async () => {
__setCollabProviderFactory(syncedProviderFactory);
const client = new DocmostClient({
apiUrl: BASE_URL,
getToken: async () => "access",
getCollabToken: async () => "collab-jwt",
});
let transformCalls = 0;
client.setToolAbortSignal(new AbortController().signal); // live
const result = await client.mutateLiveContentUnlocked(
"page-1",
"collab-jwt",
(liveDoc) => {
transformCalls++;
return null;
},
);
assert.equal(
transformCalls,
1,
"with a live signal the safe-point is a no-op and session.mutate runs the transform",
);
assert.ok(result && result.verify, "a MutationResult is returned");
});
+41 -1
View File
@@ -2,7 +2,12 @@ import { test } from "node:test";
import assert from "node:assert/strict";
import { z } from "zod";
import { SHARED_TOOL_SPECS } from "../../build/tool-specs.js";
import {
SHARED_TOOL_SPECS,
SHARED_TOOL_WRITE_CLASS,
isRetryableWriteClass,
assertEverySpecDeclaresWriteClass,
} from "../../build/tool-specs.js";
// The shared registry is consumed by BOTH the zod-v3 MCP server and the zod-v4
// in-app AI-SDK service, so every spec must carry the cross-layer wiring
@@ -43,6 +48,41 @@ test("mcpName and inAppKey are each unique across the registry", () => {
}
});
// #489 — every spec must declare its write-class so the external-MCP retry path
// can gate a single auto-retry ONLY on a pure read (a blind retry of a write =
// double-apply). The declaration is enforced at registration time.
test("#489: every spec declares a valid writeClass ('readOnly' | 'write')", () => {
for (const [key, spec] of Object.entries(SHARED_TOOL_SPECS)) {
assert.ok(
spec.writeClass === "readOnly" || spec.writeClass === "write",
`${key}: missing/invalid writeClass: ${JSON.stringify(spec.writeClass)}`,
);
}
// The registration-time assert must not throw for the shipped registry.
assert.doesNotThrow(() => assertEverySpecDeclaresWriteClass());
});
test("#489: SHARED_TOOL_WRITE_CLASS maps every mcpName to its class; helper gates on readOnly", () => {
const specs = Object.values(SHARED_TOOL_SPECS);
assert.equal(Object.keys(SHARED_TOOL_WRITE_CLASS).length, specs.length);
for (const spec of specs) {
assert.equal(SHARED_TOOL_WRITE_CLASS[spec.mcpName], spec.writeClass);
}
// Only a readOnly tool is retry-eligible; a write tool and an unknown tool are not.
assert.equal(isRetryableWriteClass("readOnly"), true);
assert.equal(isRetryableWriteClass("write"), false);
assert.equal(isRetryableWriteClass(undefined), false);
});
test("#489: representative reads are readOnly and representative writes are write", () => {
for (const name of ["getPage", "getTree", "searchInPage", "listComments"]) {
assert.equal(SHARED_TOOL_SPECS[name].writeClass, "readOnly", `${name} should be readOnly`);
}
for (const name of ["patchNode", "createPage", "deletePage", "createComment", "drawioCreate"]) {
assert.equal(SHARED_TOOL_SPECS[name].writeClass, "write", `${name} should be write`);
}
});
test("buildShape (when present) returns a usable ZodRawShape with a real zod", () => {
for (const [key, spec] of Object.entries(SHARED_TOOL_SPECS)) {
if (!spec.buildShape) continue;