Compare commits

...

21 Commits

Author SHA1 Message Date
agent_coder 6bf9358387 fix(#486): ревью — CHANGELOG + AGENTS.md + два теста
Ре-ревью PR #500 (changes-requested, 4 мелких, все doc/test):

F1 [CHANGELOG] CHANGELOG.md [Unreleased]:
- Breaking Changes: metrics-листенер 0.0.0.0→127.0.0.1 — кросс-контейнерный
  скрейп (docmost:9464) молча умрёт без METRICS_BIND=0.0.0.0 + METRICS_TOKEN
  (миграция в .env.example).
- Security: утечка errorText тулов/провайдера анониму (closes #394);
  /metrics под Bearer (METRICS_TOKEN).
- Fixed: ioredis-утечка в /health; ELK вешал event loop; beginRun-призрак →
  честный 503 A_RUN_BEGIN_FAILED; ai drain-hang.

F2 [AGENTS.md] строка про ai-патч: теперь он несёт ДВА фикса (#184 O(n²)
partialOutput И #486 drain-hang), оба тривайра названы.

F3 [test] metrics.server.spec: добавлен кейс токена ТОЙ ЖЕ длины
(Bearer topsecreX) → 401 — пиннит constant-time сравнение (прежние кейсы
коротили на length-guard, до timingSafeEqual не доходили).

F4 [test] output-degeneration.spec: behavior-тест, гоняющий РЕАЛЬНЫЕ
onChunk/onStepFinish из stream() — длинный чистый шаг → граница → свежий
дегенеративный бёрст → ассерт abortSignal.aborted (было хардкодом
resetWatermark=0, ревёрт правки не краснил).

Мутационные пруфы (non-vacuous): F3 — форс compare→true роняет same-length
кейс (401→200); F4 — ревёрт lastDegenerationCheckLen=0 роняет behavior-тест
(aborted true→false). Оба восстановлены, специи зелёные (34/34).
2026-07-11 04:38:16 +03:00
agent_coder ccfbf3343b fix(#486): ревью — run-race контракт под новую политику + timing-safe metrics-токен
Ревью полной ветки #486 нашло два пункта в моих коммитах (3 и 4):

- BLOCKER (коммит 4): ai-chat.service.run-race.spec.ts (#184 F14) пинил СТАРУЮ
  политику «plain begin() failure → swallow + стрим UNTRACKED» (resolves
  toBeUndefined). Коммит 4 её развернул → тест падал (1 failed/6). Кейс
  ИНВЕРТИРОВАН под новую политику: plain begin-failure теперь REJECTS с 503
  A_RUN_BEGIN_FAILED, до первого байта, user-строка не вставлена, streamText не
  вызван — путь остаётся явно запинён, а не удалён.

- NIT (коммит 3): metrics-токен сравнивался наивным !== (единственный слой auth
  эндпоинта) → тайминг-утечка токена. Заменено на crypto.timingSafeEqual с
  length-guard (разная длина → reject), семантика 401/200 без изменений.

Внесено отдельным fixup-коммитом (rebase -i недоступен в окружении; ветка не
запушена). Тесты: run-race 7/7 + metrics.server 7/7 зелёные.
2026-07-11 03:54:50 +03:00
agent_coder b86c747245 chore(mcp): McpService.onModuleDestroy → destroyAllSessions + удаление мёртвого кода (#486)
onModuleDestroy чистил только sweep-таймер; loopback-collab-сессии держали доки
открытыми на collab-сервере до идла — рестарт мог гонять с доком, запиненным
умирающим воркером. Теперь дергает destroyAllSessions() через переопределяемый
шов (для юнит-теста без ESM-пакета), best-effort. Плюс удалён мёртвый код:
неиспользуемый импорт parseNodeArg в mcp/index.ts и мёртвые enum-члены
SEARCH_REMOVE_* в queue.constants (подтверждено grep'ом — ни диспетчера, ни
процессора).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:31:24 +03:00
agent_coder dc96736d44 fix(mcp): пробросить truncated в tree-mode listPages (#486)
listPages(tree:true) деструктурировал только pages из enumerateSpacePages и
возвращал голое дерево, теряя truncated — на неполном дереве (stdio-fallback BFS
упёрся в node-cap) вызывающий не знал, что страницы потеряны. Возвращаем
{ tree, truncated } (по образцу check_new_comments); основной /pages/tree путь
беспредельный, там false.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:28:08 +03:00
agent_coder ef1a2b3960 fix(auth): провенанс для API-key пути (#486)
validateApiKey возвращал результат до резолва провенанса — REST-записи по
is_agent API-ключу не получали маркер 'agent'. Перенести «выше» нельзя: payload
API-ключа не несёт подписанный actor-клейм, а user (с isAgent) неизвестен до
валидации ключа. Резолвим провенанс от возвращённого user: isAgent -> 'agent',
иначе 'user'; aiChatId у API-ключа всегда null (нет ai_chats-строки). Загрузка
EE ApiKeyService вынесена в переопределяемый шов resolveApiKeyService для юнит-
теста без EE-бандла.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:25:56 +03:00
agent_coder 0be44e307e fix(client): flushNext в onFinish гейтится mountedRef (#486)
Финальный onFinish->flushNext() не проверял live-mount флаг. Чистый onFinish
может прийти ПОСЛЕ анмаунта треда (New-chat / переключение чата мид-стрим —
асинхронные attach/resume оседают поздно): flush дергал очередь и re-POST'ил
сообщение из брошенного треда — «призрачные» отправки/чаты-призраки. Остальные
обращения к очереди уже гейтятся mountedRef; закрываем последнюю дыру.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:24:19 +03:00
agent_coder 9e99309096 fix(ai-chat): сброс lastDegenerationCheckLen в onStepFinish (#486)
onStepFinish обнулял inProgressText, но НЕ lastDegenerationCheckLen — а это
смещение В аккумулятор. После первого длинного шага протухшая (большая) отметка
делала throttle-условие отрицательным, и детектор токен-лупов молчал весь
следующий шаг, пока текст не перерастал старую отметку. Обнуляем отметку в
onStepFinish. Throttle-предикат вынесен в output-degeneration
(shouldCheckDegeneration + DEGENERATION_CHECK_STEP), чтобы юнит гонял ту же
логику, что и стрим.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:22:14 +03:00
agent_coder 433bd822b6 fix(mcp): сброс collab-токена на WS-auth-failure с ретраем (#486)
Кэш collab-токена (#435) инвалидировался только на HTTP-401/403 (REST-
интерцептор и login()); отклонённый Hocuspocus-handshake оставлял протухший
токен в кэше — каждая последующая мутация переотправляла тот же битый токен до
истечения TTL (минуты) без self-heal. collab-session помечает ошибку
onAuthenticationFailed маркером; клиентские write-швы (mutatePage/replacePage/
mutateLiveContentUnlocked) обёрнуты в writeWithCollabAuthRetry: на помеченной
ошибке кэш сбрасывается и запись ретраится ровно один раз со свежим токеном —
симметрично HTTP-пути.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:19:15 +03:00
agent_coder ece84924f3 fix(mcp): REGISTRY_STAMP хэширует всё src/**, а не только tool-specs.ts (#486)
Стамп детектил build/src-skew только по tool-specs.ts — правка client.ts,
client/*-модуля, comment-signal или drawio-* без пересборки проходила молча,
и build/ отдавал старый код. Теперь codegen и рантайм-лоадер хэшируют весь
src/**/*.ts (кроме *.generated.ts — иначе цикл через собственный выход),
симметрично: одинаковый обход, POSIX-сортировка, нормализация и sha256.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:16:02 +03:00
agent_coder a14174a726 fix(ai): drain-hang в writeToServerResponse — расширить patches/ai@6.0.134.patch (#486)
Серверная ai@6.0.134 в writeToServerResponse при backpressure (write()===false)
ждала ТОЛЬКО once('drain'). Если клиент отвалился мид-запись, сокет не дренится,
await не резолвится: read-цикл паркуется НАВСЕГДА, finally{response.end()}
недостижим, reader и буферы висят до рестарта. В autonomous ран продолжает лить
вывод после дисконнекта → КАЖДЫЙ дисконнект мид-ран оставляет висящий пайп.
Плюс read() — fire-and-forget с throw → unhandledRejection.

Патч расширен (index.js и index.mjs): Promise.race drain/close/error с гигиеной
once-слушателей (все три снимаются на первом settle — не копятся по одному на
stall); при close/error — reader.cancel() и выход (безопасно для detached-ранов:
независимый дренаж делает consumeStream); rejection read() поглощается с логом.
pnpm-lock patch_hash перегенерён (patch-commit). Выравнивание версии ai —
отдельный коммит (#495), не здесь.

Тест: трипвайр-спека в apps/server (только там резолвится патченная копия) по
образцу ai-sdk-partial-output.patch.spec — дисконнект мид-запись без drain
завершает ответ (end() вызван, reader не висит), read()-throw не даёт
unhandledRejection, оба dist-билда несут маркер PATCH(docmost #486).
2026-07-11 02:54:03 +03:00
agent_coder d822ceeaf7 fix(share): не утекать errorText тулов и провайдера анониму в публичном шэре (closes #394)
SECURITY. В публичном share-чате сырой текст ошибки тула или провайдера утекал
анонимному читателю. Три слоя, все обязательны:

(1) Рендер-гейт: prop showErrors в ToolCallCard (протянут через MessageList/
MessageItem), share-виджет передаёт false — сырой errorText не рисуется. Но
рендер-гейт маскирует только DOM, не байты.

(2) Санитизация на уровне share-тулсета (авторитетно): forShare оборачивает
execute каждого тула catch'ем. Своя ShareToolError (безопасные строки: «page not
available in this share») пробрасывается, ЛЮБАЯ другая ошибка → generic «tool
could not complete», полный текст только в серверный лог. Одно место закрывает
байты (атомарный tool-output-error фрейм v6), рендер и контекст модели; self-
correction сохранена.

(3) Анонимный onError пайпа: ShareToolError → её безопасное сообщение; иначе
describeProviderError (statusCode + тело: внутренний baseUrl/модель) только в лог,
читателю — фиксированная классифицированная строка (rate-limited/unavailable/
provider error).

Тест: интеграционный с РЕАЛЬНЫМ падением тула и провайдера — assert по СЫРЫМ SSE-
БАЙТАМ (не по DOM): секрет/baseUrl/стек отсутствуют, видна безопасная строка,
полный текст провайдера ушёл в серверный лог.
2026-07-11 02:50:19 +03:00
agent_coder 4ac014def2 fix(ai-chat): провал beginRun фейлит ход честным 503 (A_RUN_BEGIN_FAILED) (#486)
Раньше провал beginRun (кроме unique-violation), напр. blip пула БД, логировался
и ход ПРОДОЛЖАЛСЯ без run-строки. В autonomous такой ран никто не абортит: /stop
его не видит, дисконнект не абортит, one-run-гейт пропускает ВТОРОЙ ран — невидимый
неостанавливаемый ран до рестарта.

Теперь провал beginRun (кроме RunAlreadyActiveError → прежний 409) бросает
ServiceUnavailableException с кодом A_RUN_BEGIN_FAILED ДО первого байта и до
вставки user-строки (post-hijack catch контроллера отдаёт честный 503 на raw-
сокет). Без ветвления по режимам — #487 наследует ту же политику. В тело кладём
statusCode: 503 (object-arg исключение его не добавляет), чтобы клиент видел
статус.

Клиентский классификатор: ветка A_RUN_BEGIN_FAILED добавлена СТРОГО ДО generic-
503-матча — иначе показал бы «provider is not configured» вместо «временно,
повторите».

Тесты: unit fail-fast (stream() бросает 503 A_RUN_BEGIN_FAILED, ни байта в сокет,
user-строка не вставлена; RunAlreadyActiveError по-прежнему 409); unit клиентского
классификатора из ПОЛНОГО реального тела ответа с гвардом порядка.
2026-07-11 02:36:43 +03:00
agent_coder 799dcfe7c5 fix(metrics): bind на 127.0.0.1 по умолчанию + METRICS_BIND/METRICS_TOKEN (#486)
/metrics слушал на 0.0.0.0 без какой-либо аутентификации — auth-less
эндпоинт на всех интерфейсах. Теперь дефолтный bind — loopback 127.0.0.1;
env METRICS_BIND переопределяет интерфейс (0.0.0.0 для скрейпера в
отдельном контейнере, docmost:9464); опциональный METRICS_TOKEN включает
Bearer-аутентификацию (запросы без точного токена получают 401). Доки
скрейпа в .env.example обновлены.

Тест: unit на дефолтный bind + env-переопределение + резолв токена;
интеграционный по РЕАЛЬНОМУ сокету — listener забинден на 127.0.0.1,
без токена /metrics отдаётся, с токеном без/с неверным Bearer → 401, с
верным → 200.
2026-07-11 02:32:21 +03:00
agent_coder 14c864f5c6 fix(health): устранить утечку ioredis-клиента в /health-пробе (#486)
pingCheck строил new Redis(...) на КАЖДЫЙ вызов и делал disconnect() только
на success-пути. Пока Redis лежит, каждый тик health-пробы добавлял свежий,
вечно реконнектящийся клиент — неограниченный рост хэндлов/клиентов на всё
время недоступности Redis.

Теперь один долгоживущий probe-клиент, переиспользуемый между тиками:
lazyConnect (конструктор не бросает и не коннектится жадно),
maxRetriesPerRequest: 1 и enableOfflineQueue: false (проба фейлится быстро,
команды не буферизуются), плюс listener на 'error' (иначе unhandled error
роняет процесс). onModuleDestroy закрывает клиент при shutdown.

Тест: интеграционный — N проб при лежащем Redis (реальный refused-порт, не
мок поведения) создают РОВНО ОДИН клиент (на баге было бы N); onModuleDestroy
освобождает клиент, следующая проба лениво строит новый.
2026-07-11 02:29:41 +03:00
agent_coder e292ed5117 fix(mcp): ELK-лейаут в worker_thread — таймаут через terminate() (#486)
elkjs.layout() возвращает Promise, но саму раскладку крутит СИНХРОННО и
блокирует поток целиком. На in-app хосте это был главный event loop:
патологический граф у капа 500 узлов вешал ВСЕ HTTP/SSE/loopback. Прежняя
защита (Promise.race с setTimeout(5s)) была иллюзией — таймер физически не
мог сработать, пока тот же поток заблокирован внутри elkjs (комментарий в
коде это сам признавал).

Теперь elk.layout() исполняется в worker_thread, а таймаут форсится
worker.terminate() — единственный способ прервать синхронный JS. Главный
поток остаётся свободным; на таймауте/ошибке — best-effort откат к
исходной модели, как и раньше. Лживый комментарий «can never wedge the
server» убран.

Тесты: unit на terminate-по-таймауту (крошечный ceiling → hard-kill →
исходная модель нетронута) и бенчмарк-гард на worst-case графе у капа
(500 узлов/~1000 рёбер раскладывается, а главный event loop продолжает
тикать во время раскладки).
2026-07-11 02:25:28 +03:00
vvzvlad 6d7dba970c Merge pull request 'refactor(mcp): структурные инварианты конкурентной записи — UUID-assert, self-resolve seams, no-await-guard (#449)' (#475) from refactor/449-write-invariants into develop
Reviewed-on: #475
2026-07-11 01:32:30 +03:00
vvzvlad 512bcba5f3 Merge pull request 'refactor(mcp): распил client.ts (5206 строк) на доменные модули за тонким фасадом (#450)' (#478) from refactor/450-split-client into refactor/449-write-invariants
Reviewed-on: #478
2026-07-11 01:30:28 +03:00
agent_vscode 5c1ab9c7b5 fix(docker): ship packages/mcp/data into runtime image
drawioFromGraph и каталог фигур читают данные в рантайме по пути
packages/mcp/data/ (относительно build/lib/*.js через import.meta.url).
tsc эмитит только build/, а стадия installer копировала лишь build/ и
package.json — в образе не было ни drawio-presets.json (#425), ни
drawio-shape-index.json.gz (#440), из-за чего drawioFromGraph падал с
ENOENT, а иконки каталога фигур молча не резолвились.

Добавлен COPY packages/mcp/data в стадию installer. .dockerignore /data
заякорен на корень и этот путь не затрагивает.
2026-07-11 01:13:48 +03:00
agent_coder 14d7b21df0 refactor(mcp): распил client.ts (5206 строк) на доменные модули за тонким фасадом (#450)
client.ts был god-object'ом на 5206 строк / ~65 методов / 5 ответственностей —
любая правка рисковала всем write-path. Разнесли на доменные модули;
DocmostClient остаётся ТОНКИМ ФАСАДОМ с прежним внешним контрактом. Чистый
рефакторинг, поведение не меняется. closes #450

- Фасад client.ts (93 строки) композирует 10 миксинов над общим абстрактным
  базовым классом. Паттерн МИКСИНЫ (не context-object): тесты субклассируют
  DocmostClient и переопределяют seam'ы; единая цепочка прототипов сохраняет
  виртуальную диспетчеризацию this.<method> сквозь модули.
- client/context.ts (база): общее состояние (axios, apiUrl, токены, кэши),
  конструктор + оба интерсептора, login/ensureAuthenticated/paginateAll/
  resolvePageId/mutateLiveContentUnlocked + write-seam'ы mutatePage/replacePage.
  private→protected (внутреннее, не в публичном контракте).
- Модули (каждый ≤730): read, pages, nodes-write, media (images/attachments/
  drawio), comments, transforms, tables, stash, doc-validate. errors.ts —
  единый REST error-mapping (довершение #437; тексты сообщений без изменений).
- Внешний контракт СОХРАНЁН (доказано компиляцией с обеих сторон): 53 публичных
  метода через `implements IXMixin` (без ручного зеркала, #446); оба Pick
  (in-app loader + tool-specs) резолвятся; множество async-методов 59==59.
  Нагруженные seam'ы (replaceImage один-лок #425, self-resolve #449, единый
  error-путь) байт-идентичны; ни одного дубля публичного метода.
- zod v3→v4: investigate-only, отложено — SDK 1.29 поддерживает v4, но мажор-
  бамп трогает всю схема-поверхность; отдельным follow-up.

Стоит на #449 (#475). Тесты: mcp node --test 800/800 (== база), tsc чисто в
client/. Ни одной строки кода не потеряно; убраны 52 дублированных JSDoc-блока.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 01:04:42 +03:00
agent_vscode 363f20ab75 docs(agents): add architectural invariants (non‑negotiable rules)
Introduce a new “ARCHITECTURAL INVARIANTS — NON-NEGOTIABLE” section that
lists ten hard constraints derived from past production incidents. These
rules act as non‑negotiable guidelines for future development and code
review.
2026-07-11 01:04:10 +03:00
agent_coder 9a8671c3af refactor(mcp): структурные инварианты конкурентной записи — UUID-assert в page-lock, self-resolve seams, no-await-guard (#449)
Механика конкурентной записи (withPageLock → acquireCollabSession → mutate)
держалась на цепочке конвенций в комментариях. Новый write-метод без знания
правил компилировался и уходил в прод (класс #260/#152/#159). Закрепляем ключевые
инварианты кодом.

- page-lock.ts: экспортированы UUID_RE/isUuid (тот же regex, что resolvePageId,
  UUID v1–8/v7). withPageLock FAIL-FAST кидает при не-UUID ключе ДО любой работы
  (комментарий-инвариант #260/#449) — забытый resolve/slugId больше не даёт тихую
  потерю сериализации под другим ключом. client.ts импортирует isUuid оттуда
  (убран локальный дубль — resolver и assert не разъедутся).
- mutatePage/replacePage seams стали async и сами вызывают resolvePageId — ключ
  лока/кэша канонический даже если вызывающий забыл (для уже-UUID это cached
  no-op; все 7 текущих вызывающих и так резолвят). replaceImage (один внешний
  лок + mutateLiveContentUnlocked) не тронут, deadlock невозможен.
- collab-session.ts: машинно-проверяемые маркеры MUTATE-CRITICAL-WINDOW
  BEGIN/END вокруг синхронного блока fromYdoc→applyDocToFragment (INVARIANT 1).
  Тест no-await-critical-window читает исходник и краснеет на await/yield в окне
  (проверено нейтером). Случайный await больше не тихо клоббит живые правки.
- Документация осознанной позиции: single-instance/sticky-sessions —
  требование деплоя (Dockerfile + README EN/RU + .env.example), т.к. мьютекс и
  stash — per-process. Окно устаревших прав (кэш-сессия пишет под токеном
  момента connect до MCP_COLLAB_SESSION_MAX_AGE_MS=10мин) — задокументированный
  trade-off в .env.example; push-инвалидации нет (осознанно).

Тесты: page-lock fail-fast (slugId/пусто/non-string → throw; канонический UUID
принят), no-away-guard, обновлённые фикстуры на валидные UUID. #449-специфичные
37/37 зелёные; mcp tsc чисто. closes #449.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 23:51:02 +03:00
64 changed files with 9214 additions and 5486 deletions
+48
View File
@@ -124,6 +124,40 @@ MCP_DOCMOST_PASSWORD=
# MCP_TOKEN=
# MCP_SESSION_IDLE_MS=1800000
#
# --- MCP collaboration write path: concurrency + rights-staleness (#449) ------
# MCP content writes (update_page, insert/replace nodes, comments-in-body, etc.)
# go over the collaboration websocket and are serialized PER PAGE by an
# in-process mutex (a module-level Map, one promise-chain per page UUID). This
# guarantees no two MCP writes on the SAME page overlap and clobber each other.
#
# DEPLOY REQUIREMENT — SINGLE INSTANCE or STICKY SESSIONS. The mutex is
# process-local. Behind a multi-replica load balancer WITHOUT sticky sessions,
# two replicas can each "hold" the lock for the same page at the same time and
# serialization is silently lost (concurrent full-document writes race on the
# live Yjs fragment). Run the MCP/app as a SINGLE instance, OR pin a page's
# traffic to one replica (sticky sessions / consistent hashing on page id). The
# same constraint applies to the RAM-only stash_page blob store above. There is
# deliberately no cross-process (e.g. Postgres advisory) lock yet — this is a
# CONSCIOUS documented constraint, not an oversight (#449).
#
# To reduce connect-storms the write path caches ONE live collab session per
# (wsUrl, page, token). Tunables (all optional; defaults are safe):
# MCP_COLLAB_SESSION_IDLE_MS=60000 # idle TTL, reset per op; 0 disables cache
# MCP_COLLAB_SESSION_MAX_ENTRIES=32 # LRU cap on cached sessions
# MCP_COLLAB_TOKEN_TTL_MS=300000 # per-client collab-token cache (5 min)
#
# RIGHTS-STALENESS TRADE-OFF. A cached collab session writes under the token
# captured at CONNECT time, and the collab-token cache reuses a token for its TTL.
# So if a user's access to a page is REVOKED, MCP writes on an already-open
# session may keep succeeding until the session ages out. MCP_COLLAB_SESSION_MAX_AGE_MS
# is the HARD lifetime (checked at each acquire) that BOUNDS this window: after it,
# the session is torn down and the next write re-auths with a fresh token, picking
# up the revocation. Default 10 min. LOWER it to shorten the revocation lag at the
# cost of more reconnects; RAISE it to reduce reconnects at the cost of a longer
# stale-rights window. There is intentionally no push-based cache invalidation on
# a rights change — this bounded window is the accepted trade-off (#449).
# MCP_COLLAB_SESSION_MAX_AGE_MS=600000
#
# BLOB SANDBOX (stash_page). An in-RAM, process-local store that hands large page
# content + images to an external consumer WITHOUT bloating the model context or
# requiring Docmost auth. The stash_page tool serializes a page, mirrors its
@@ -301,6 +335,20 @@ MCP_DOCMOST_PASSWORD=
# VictoriaMetrics/Prometheus reaching it as <host>:<port>/metrics.
# METRICS_PORT=9464
#
# METRICS_BIND — interface the /metrics listener binds to. DEFAULT 127.0.0.1
# (loopback only), so the unauthenticated endpoint is NOT exposed on all
# interfaces. If the scraper runs in a SEPARATE container and reaches this as
# docmost:9464, set METRICS_BIND=0.0.0.0 — but then also set METRICS_TOKEN
# and/or keep the port on a private network, since /metrics is otherwise open.
# METRICS_BIND=127.0.0.1
#
# METRICS_TOKEN — optional Bearer token guarding /metrics. When set, every
# scrape MUST send `Authorization: Bearer <token>` (others get 401). Configure
# the scraper with the same bearer token (e.g. VictoriaMetrics/vmagent
# `bearer_token`, Prometheus `authorization.credentials`). Leave unset only
# when the endpoint is bound to loopback or an otherwise-trusted network.
# METRICS_TOKEN=
#
# 2) CLIENT_TELEMETRY_ENABLED — the public client perf-telemetry sink.
# OFF by default. When true, the unauthenticated POST /api/telemetry/vitals
# endpoint is registered and browsers collect + send web-vitals / editor
+134 -1
View File
@@ -5,6 +5,139 @@ repository. It has two layers: **how to run a task end-to-end** (the
sections below), and **how the codebase is built** (the technical sections
further down, formerly in `CLAUDE.md`).
## ARCHITECTURAL INVARIANTS — NON-NEGOTIABLE
THE TEN RULES BELOW ARE HARD CONSTRAINTS. Each one was paid for with a real
production incident or a multi-PR bug chain in THIS repository (cited inline).
They override convenience, deadlines and "it's just a small feature". A PR that
violates any of them MUST be rejected in review regardless of how good the rest
of it is. If a task genuinely seems to require breaking one — STOP and raise it
with the owner; do not code around it.
### 1. EVERY BUFFER, CACHE, HISTORY AND PAYLOAD HAS AN EXPLICIT SIZE BUDGET
Nothing accumulates unboundedly. A row/item cap is NOT a byte cap. Anything
replayed to a model, buffered in memory, persisted per step, or refetched by a
poll must state its budget in bytes/tokens and enforce it. Rewriting a growing
structure in full on every increment is FORBIDDEN — append or diff instead;
O(n²) write/serialize patterns do not pass review.
(Paid for by: full-row rewrite on every agent step — hundreds of MB of Postgres
writes per 50-step run, with every tool output serialized twice; unbounded
history replay killing long chats on the provider context window; 32 MB replay
buffers per active run.)
### 2. EVERYTHING LONG-RUNNING TERMINATES BY CONSTRUCTION
Every run / row / session / lease / subscriber / queue entry must define AT
DESIGN TIME: its owner; every terminal state; who writes the terminal state on
EVERY path (success, error, abort, disconnect in each phase, process restart);
retries for the terminal write; and a periodic sweeper that does not depend on
a reboot. A best-effort terminal write with no retry and no sweep is FORBIDDEN.
(Paid for by: assistant rows stuck 'streaming' forever; runs stuck 'running'
409-locking their chat until a restart — the #183/#184 follow-up chain.)
### 3. EVERY AWAIT IS CANCELLABLE AND DEADLINED; NEVER BLOCK THE EVENT LOOP
Every async step inside a request or agent turn honors the turn's AbortSignal
AND a wall-clock deadline — including in-app tools, lock queues and pagination
loops, not just external calls. Synchronous CPU work beyond ~50 ms goes to a
worker_thread. Promise.race DOES NOT cancel synchronous work — using it as a
"timeout" for sync computation is forbidden (the timer only fires after the
event loop is free again, i.e. after the damage is done).
(Paid for by: in-app tools ignoring abortSignal and writing pages AFTER Stop;
the synchronous ELK layout freezing every SSE stream in the process; the
step-0 MCP handshake hang — #397.)
### 4. ONE SOURCE OF TRUTH; EVERYTHING ELSE IS A REBUILDABLE CACHE
Postgres is the authoritative state. Every in-memory structure (registries,
caches, client stores) must be reconstructible from the DB and treated as
lossy. The client renders SERVER-DECLARED state — "a run is active" is a server
fact delivered as data, never inferred from side signals (204 vs 2xx, the
flavor of a disconnect). A new feature must name the owner of each piece of
state before implementation starts.
(Paid for by: the strip/restore resume machinery, silently frozen UIs and
ghost sends after unmount — the #381#432#456 chain.)
### 5. STATE MACHINES ARE EXPLICIT — ONE-SHOT FLAGS ARE FORBIDDEN
A complex lifecycle (chat thread, resume/reconnect, run) lives in a named-state
automaton (reducer / enum) where every state has an owner and a rendered
representation — including the failure states. Adding a boolean ref that one
callback arms and another reads-and-clears is FORBIDDEN in the AI-chat client.
New behavior = a new named state + explicit transitions, and the interruption
matrix (disconnect in each phase × restart × stop × supersede) is enumerated at
design time, not discovered one incident at a time.
(Paid for by: 26 one-shot useRef flags in chat-thread.tsx and the drip of
"one more missing transition" across #381#386/#389#432#456.)
### 6. NO NEW MODE FORKS; A FLAG IS FOR ROLLOUT, THEN IT DIES
A behavior flag that forks a code path must ship with a written sunset
condition; stacking a new flag onto the existing matrix without deleting or
scheduling an old one is forbidden. While a temporary fork exists, BOTH sides
must share identical lifecycle handling (abort semantics, error listeners,
concurrency gates) — asymmetric forks are outlawed.
(Paid for by: legacy vs autonomous divergence — the one-active-run gate and
the socket 'error' listener each existing on only ONE side; 2^4 flag
combinations each with different abort semantics.)
### 7. NO HAND-SYNCED MIRRORS — CODEGEN OR A CI PARITY TEST, NOTHING LESS
Two copies of the same knowledge (schema, tool registry, glyph map, probe
body, hash/normalize algorithm, label list) require either generation from a
single source or a CI test that FAILS on drift. A "mirror this change over
there" comment is NOT a guard and does not pass review.
(Paid for by: #293 — three drifting converter copies losing data; #447
REGISTRY_STAMP covering only one of the mirrored files; ~10 still-unguarded
mirrors across the MCP layer.)
### 8. CACHES, HEADERS, BUFFERS AND FSM TRANSITIONS GET AN INTEGRATION TEST OF THE OBSERVABLE PROPERTY
A unit test of a pure helper DOES NOT COUNT for these. Test the real header on
the real HTTP response, the real cache hit under real token sources, the real
transition under a really-killed socket. If the observable property cannot be
tested, the design is wrong — fix the design, not the test.
(Paid for by: #431#439 — a cache keyed on a fresh-per-call JWT, so it NEVER
hit and became prod incident #435 while its unit tests stayed green; and by
the #352#455 immutable-cache header silently overwritten by a framework
default AFTER the unit-tested code ran.)
### 9. CLIENT INPUT IS HOSTILE UNTIL VALIDATED — ALSO BEFORE PERSISTENCE
Anything from the browser (message parts, ids, titles, selections, flags) is
validated/sanitized BEFORE it is persisted into a row that will later be
replayed into a prompt, a converter or another subsystem. A poisoned row must
never be able to permanently brick a chat or a page on every subsequent read.
(Paid for by: unvalidated UIMessage parts persisted verbatim — one bad row
500s the chat on every later turn; #159 client-spoofed page titles; #388
selection re-sanitized server-side for the same reason.)
### 10. FAILURES ARE LOUD AND SPECIFIC; SILENT DEGRADATION IS FORBIDDEN
Extends the error convention below: a fire-and-forget write is allowed ONLY
with a metric or a greppable ERROR log; a degraded mode (dead cached MCP
client, stopped poll, exhausted retries, evicted buffer) must be VISIBLE to
the user or the operator. A feature that can quietly stop working — a frozen
"streaming…" UI, a poll that silently gives up, a cache serving corpses — does
not pass review.
(Paid for by: the degraded poll's silent 10-minute death leaving a forever-
"streaming" answer; dead MCP clients served from cache while every external
tool call failed; #435 being caught in minutes ONLY because metrics — #403
existed.)
## Default skill for feature design
For any feature-design request — the user hands over a raw feature idea, asks
to design or think through a feature, or to draft an issue («спроектируй»,
«продумай фичу», «составь ишью», "design X", "write an issue for X") — invoke
the `orchestrator-feature-designer` skill (Skill tool) BEFORE any other work.
It is the default operating mode for design work in this repository: research
→ design checklist (R1–R10) → forks resolved with the human → adversarial
self-attack → filed PR-sized issues. Do not design features or write issues
ad-hoc while this skill is available. This does not apply to non-design work
(bug fixes, reviews, retrospectives, refactors already specified by an issue).
## Task lifecycle
### 1. Start: sync with develop
@@ -337,7 +470,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- **Errors must never be swallowed or shown as generic messages.** Every caught error MUST (1) be logged in full to the console/logger — error name, message, stack, `cause`, and (for HTTP/provider failures) the status code and response body — and (2) be surfaced to the user with a *specific, human-readable explanation of what actually went wrong*, never a bare generic string like "Something went wrong" / "Could not start recording" / "Transcription failed". Include the real reason (the underlying error/provider message) in the user-facing text. On the server, wrap third-party/provider failures with `describeProviderError` (or equivalent) and rethrow as a meaningful HTTP status + message — never let them collapse into an opaque 500. On the client, `console.error(<context>, err)` the raw error AND show the extracted reason (e.g. `err.response?.data?.message`, or the error `name: message`) in the notification.
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire test: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`) — it MUST be re-created via `pnpm patch` when bumping `ai`.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch carries TWO independent server fixes, each with its own tripwire test: (1) it disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`); (2) it fixes `writeToServerResponse`'s drain-hang — the loop awaited only `"drain"` under backpressure, so a mid-write client disconnect parked the pipe forever and leaked the reader/buffers until restart; it now races `"drain"` against `"close"`/`"error"`, cancels the reader on disconnect, and swallows the fire-and-forget read rejection (#486; tripwire: `apps/server/src/integrations/ai/ai-sdk-drain-hang.patch.spec.ts`). Both tripwires assert BOTH installed dist builds carry their patch marker. The patch MUST be re-created via `pnpm patch` when bumping `ai`.
- **The MCP tool inventory in `SERVER_INSTRUCTIONS` is GENERATED from the registry** (`packages/mcp/src/server-instructions.ts`: `buildToolInventory()` over `SHARED_TOOL_SPECS`) and spliced into the hand-written routing prose (`ROUTING_PROSE`). So adding/renaming/removing a **shared** spec in `packages/mcp/src/tool-specs.ts` auto-updates the `<tool_inventory>` — no manual `SERVER_INSTRUCTIONS` edit needed. Only an **inline** MCP-only tool (those registered via `server.registerTool(...)` in `index.ts`, not through the registry) needs a one-line entry in `INLINE_MCP_INVENTORY`. Enforced by `packages/mcp/test/unit/tool-inventory.test.mjs`, which fails when a registered tool is missing from the generated inventory (there is no `EXCEPTIONS` opt-out anymore — every tool must appear). Update `ROUTING_PROSE` when a tool's *intent guidance* (when-to-use) changes. `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
+63
View File
@@ -115,6 +115,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the old ProseMirror-JSON output. Released together with the `#411`/`#412`
breaking window so external configs break exactly once. (#413)
- **The Prometheus `/metrics` listener now binds to `127.0.0.1` (loopback) by
default instead of `0.0.0.0` (all interfaces).** This closes an unauthenticated
endpoint that was previously reachable on every interface. **DEPLOY MIGRATION —
cross-container scraping breaks silently otherwise:** if your scraper runs in a
SEPARATE container and reaches the app as `docmost:9464` (the exact topology the
old `0.0.0.0` hardcode served), you MUST now set `METRICS_BIND=0.0.0.0` — and,
because that re-exposes the endpoint, also set `METRICS_TOKEN=<secret>` and
configure the scraper with a matching Bearer token. Without `METRICS_BIND`, the
scraper can no longer connect and metrics go dark with no error. See the
`METRICS_BIND` / `METRICS_TOKEN` block in `.env.example` for the migration.
Same-host (loopback) scrapers need no change. (#486)
### Added
- **Place several images side by side in a row.** A new "Inline (side by
@@ -298,6 +310,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`tee()` branch of the stream result — a ~20-step, ~28k-chunk agent run
retained ~1.7 GB and OOM'd the 2 GB JS heap. Streaming granularity is
unchanged; the patch must be re-created if `ai` is ever bumped. (#184)
- **The server no longer leaks a hung stream pipe on every mid-run client
disconnect.** The same `ai@6.0.134` pnpm patch now also fixes the SDK's
`writeToServerResponse`, which awaited only a `"drain"` event under
backpressure: when a client disconnected mid-write the socket never drained, so
the write loop parked forever, `response.end()` was unreachable, and the stream
reader plus buffered chunks were pinned until process restart (every mid-run
disconnect in autonomous mode leaked one). The patch races `"drain"` against
`"close"`/`"error"`, cancels the reader and ends the response on disconnect, and
swallows the fire-and-forget read rejection instead of crashing on an
unhandledRejection. (#486)
- **A failed autonomous agent-run start no longer becomes an unstoppable ghost
run.** When `beginRun` failed for a transient reason (e.g. a DB-pool blip),
the turn previously continued with NO run row — invisible to `/stop`, not
aborted on disconnect, and able to slip a second run past the one-run-per-chat
gate, leaving an unstoppable run until restart. The turn now fails fast with an
honest `503 A_RUN_BEGIN_FAILED` before the first byte (no orphan state), and the
client shows a "temporary — please try again" message instead of a misleading
"provider not configured". (#486)
- **A pathological draw.io graph can no longer wedge the whole server.** The ELK
auto-layout (`layout:"elk"`) ran elkjs synchronously on the main event loop, so
a graph at the node/edge cap blocked ALL HTTP/SSE/loopback traffic while it
churned — and the old `setTimeout` "timeout" could never fire because the same
thread was blocked. Layout now runs in a worker thread with the timeout enforced
by `worker.terminate()`; the main loop stays responsive. (#486)
- **The `/health` Redis probe no longer leaks a client on every tick while Redis
is down.** It built a new `ioredis` client per probe and disconnected it only on
success, so during an outage each health tick added another forever-reconnecting
client (an unbounded handle leak). A single long-lived probe client is now
reused and closed on shutdown. (#486)
- **Internal links in exported Markdown no longer lose their visible text.** A
link whose target page name had no file extension (e.g. a bare title) was
collapsed to empty text during export, producing an unclickable, label-less
@@ -374,6 +419,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
share); any other value now returns the generic "not found" instead of
serving the page. (#218)
- **Tool and provider error text no longer leaks to anonymous readers in the
public-share AI chat.** A failing tool's raw error (which could carry an
internal page title or a stack fragment) and a provider error (which bundles the
provider `statusCode` and response body — potentially the internal baseUrl or
model name) were streamed verbatim to the anonymous reader over SSE. Errors are
now sanitized at the source: the share toolset collapses any unclassified tool
error to a safe generic string (safe, classified tool messages still pass
through for the model's self-correction), and the anonymous stream `onError`
maps provider failures to a fixed set of neutral strings — the full detail goes
only to the server log. A UI render gate is layered on top. (closes #394)
- **The Prometheus `/metrics` endpoint can now require Bearer authentication and
is loopback-bound by default.** Previously it listened on all interfaces with no
auth. Setting `METRICS_TOKEN` requires every scrape to present
`Authorization: Bearer <token>` (compared in constant time), and the listener
defaults to `127.0.0.1` (see the Breaking Changes entry for the cross-container
migration). (#486)
## [0.94.0] - 2026-06-26
This release makes AI chat durable and fast: assistant turns are persisted to
+15
View File
@@ -45,6 +45,11 @@ COPY --from=builder /app/packages/editor-ext/dist /app/packages/editor-ext/dist
COPY --from=builder /app/packages/editor-ext/package.json /app/packages/editor-ext/package.json
COPY --from=builder /app/packages/mcp/build /app/packages/mcp/build
COPY --from=builder /app/packages/mcp/package.json /app/packages/mcp/package.json
# The mcp package reads its data files (drawio-presets.json, drawio-shape-index.json.gz)
# at runtime via `new URL("../../data/…", import.meta.url)` relative to build/lib/*.js,
# i.e. from packages/mcp/data/. tsc emits only build/, so ship data/ explicitly or
# drawioFromGraph and the shape catalog die with ENOENT on packages/mcp/data/*.
COPY --from=builder /app/packages/mcp/data /app/packages/mcp/data
# mcp now depends on @docmost/prosemirror-markdown (workspace:*) and eager-imports
# it at runtime (the in-app ai-chat DocmostClient loads build/index.js -> lib/
# markdown-converter.js). Ship the built package + its manifest, or the prod
@@ -81,4 +86,14 @@ VOLUME ["/app/data/storage"]
EXPOSE 3000
# DEPLOY REQUIREMENT — SINGLE INSTANCE or STICKY SESSIONS (#449).
# MCP content writes are serialized per page by an IN-PROCESS mutex, and the
# stash_page blob store + cached collab sessions are RAM-only and process-local.
# Running MULTIPLE replicas of this image behind a load balancer WITHOUT sticky
# sessions silently breaks per-page write serialization (two replicas can lock
# the same page at once) and makes stash_page blobs unreachable across replicas.
# Run a SINGLE instance, or pin each page's traffic to one replica (sticky
# sessions / consistent hashing on page id). There is deliberately no
# cross-process lock yet — a conscious constraint. See .env.example (the "MCP
# collaboration write path" block) and packages/mcp/README.md for details.
CMD ["pnpm", "start"]
@@ -203,6 +203,52 @@ describe("ChatThread — send now (#198)", () => {
});
});
// #486: the final onFinish -> flushNext() must be gated on the live-mount flag.
// A clean onFinish can land AFTER the thread unmounts (New-chat / chat-switch
// mid-stream — the async attach/resume settles late); flushing then dequeues and
// re-POSTs a queued message from an abandoned thread (a "ghost" send).
describe("ChatThread — onFinish flush gated on mount (#486)", () => {
beforeEach(resetState);
afterEach(cleanup);
it("a clean onFinish WHILE MOUNTED flushes the queued message (control)", () => {
renderThread();
fireEvent.click(screen.getByTestId("queue-btn")); // enqueue "queued text"
expect(h.state.sendMessage).not.toHaveBeenCalled();
act(() => {
h.state.onFinish?.({
message: { id: "a", role: "assistant", parts: [] },
isAbort: false,
isDisconnect: false,
isError: false,
});
});
// Mounted: the queue flushes normally.
expect(h.state.sendMessage).toHaveBeenCalledWith({ text: "queued text" });
});
it("a clean onFinish AFTER unmount does NOT flush (no ghost send)", () => {
const { unmount } = renderThread();
fireEvent.click(screen.getByTestId("queue-btn")); // enqueue "queued text"
h.state.sendMessage.mockClear();
// Chat switched away mid-stream: the streamer unmounts...
unmount();
// ...and a late, clean onFinish lands on the abandoned thread.
act(() => {
h.state.onFinish?.({
message: { id: "a", role: "assistant", parts: [] },
isAbort: false,
isDisconnect: false,
isError: false,
});
});
// Gated on mountedRef: NOTHING is sent from the dead thread.
expect(h.state.sendMessage).not.toHaveBeenCalled();
});
});
// #396: in autonomous mode a live sendNow must additionally request the
// AUTHORITATIVE server stop of the detached run (a local abort is only a client
// disconnect the server ignores) and arm a bounded 409 retry so the re-POST
@@ -659,7 +659,13 @@ export default function ChatThread({
return;
}
if (isAbort || isDisconnect || isError) return;
flushNext();
// Gate the final flush on the live-mount flag (#486): a clean onFinish can
// land AFTER this thread unmounted (a New-chat / chat-switch mid-stream —
// the async attach/resume settles late). Flushing then dequeues and POSTs a
// queued message from an abandoned thread — a "ghost" send / ghost chat.
// Every other queue side effect already guards on mountedRef; this last one
// was the gap.
if (mountedRef.current) flushNext();
},
// `onError` runs in addition to `onFinish` (which ai@6 also calls on error).
// Log the raw failure here for devtools; the UI shows a friendly classified
@@ -47,6 +47,13 @@ interface MessageItemProps {
* agent's raw query/argument text.
*/
showInput?: boolean;
/**
* Forwarded to ToolCallCard: whether a failed tool card renders its raw
* errorText. Defaults to true (internal chat). The public share passes false so
* internal detail in a tool error is never painted (belt to the server-side
* byte sanitization).
*/
showErrors?: boolean;
/**
* Neutralize internal/relative markdown links in the rendered answer (drop
* their href so they become inert text). Defaults to false (internal chat,
@@ -125,6 +132,7 @@ function MessageItem({
message,
showCitations = true,
showInput = true,
showErrors = true,
neutralizeInternalLinks = false,
assistantName,
turnStreaming = false,
@@ -219,6 +227,7 @@ function MessageItem({
part={part as unknown as ToolUiPart}
showCitations={showCitations}
showInput={showInput}
showErrors={showErrors}
/>
);
}
@@ -284,6 +293,7 @@ export function arePropsEqual(
prev.signature === next.signature &&
prev.showCitations === next.showCitations &&
prev.showInput === next.showInput &&
prev.showErrors === next.showErrors &&
prev.neutralizeInternalLinks === next.neutralizeInternalLinks &&
prev.assistantName === next.assistantName &&
// The turn-end flip re-renders every row once (cheap, terminal event) —
@@ -32,6 +32,12 @@ interface MessageListProps {
* doesn't see the agent's raw query/argument text.
*/
showInput?: boolean;
/**
* Forwarded to MessageItem -> ToolCallCard: whether a failed tool card renders
* its raw errorText. Defaults to true (internal chat). The public share passes
* false so internal detail in a tool error is never painted.
*/
showErrors?: boolean;
/**
* Forwarded to MessageItem: neutralize internal/relative markdown links in
* the rendered answers (drop their href so they render as inert text).
@@ -127,6 +133,7 @@ export default function MessageList({
emptyState,
showCitations = true,
showInput = true,
showErrors = true,
neutralizeInternalLinks = false,
assistantName,
}: MessageListProps) {
@@ -217,6 +224,7 @@ export default function MessageList({
signature={messageSignature(message)}
showCitations={showCitations}
showInput={showInput}
showErrors={showErrors}
neutralizeInternalLinks={neutralizeInternalLinks}
assistantName={assistantName}
// Turn-level liveness, gated to the TAIL row: only the tail message
@@ -30,6 +30,16 @@ interface ToolCallCardProps {
* the extra summary line, leaving the card (the action log) intact.
*/
showInput?: boolean;
/**
* Whether to render the tool's raw errorText on a failed call. Defaults to true
* (the internal chat, where the operator may debug). The public share passes
* false: a tool error string can carry internal detail (an internal page title,
* a stack fragment, a provider message). This is the RENDER gate only — the
* authoritative fix also sanitizes the bytes server-side (see
* PublicShareChatToolsService.forShare), so a share reader never receives raw
* error text over the wire, not just never sees it painted (#394).
*/
showErrors?: boolean;
}
/**
@@ -41,6 +51,7 @@ export default function ToolCallCard({
part,
showCitations = true,
showInput = true,
showErrors = true,
}: ToolCallCardProps) {
const { t } = useTranslation();
const toolName = getToolName(part);
@@ -74,7 +85,7 @@ export default function ToolCallCard({
</Text>
)}
{state === "error" && part.errorText && (
{state === "error" && showErrors && part.errorText && (
<Text size="xs" c="red" mt={2}>
{part.errorText}
</Text>
@@ -23,6 +23,25 @@ describe("describeChatError", () => {
});
});
it("classifies an A_RUN_BEGIN_FAILED 503 as a temporary run-start failure, NOT provider-not-configured (#486)", () => {
// The FULL real body the server writes for a beginRun failure: a
// ServiceUnavailableException(object) whose response is serialized verbatim
// onto the raw socket, self-describing statusCode 503 + the run-start code.
const body =
'{"message":"Could not start the agent run. This is usually temporary — please try again.","code":"A_RUN_BEGIN_FAILED","statusCode":503}';
expect(describeChatError(body, t)).toEqual({
title: "Could not start the run",
detail:
"The agent run could not be started. This is usually temporary — please try again.",
});
// ORDER GUARD: even though the body ALSO carries statusCode 503 (which the
// generic branch matches), the A_RUN_BEGIN_FAILED branch runs first, so it is
// never mislabeled "AI provider not configured".
expect(describeChatError(body, t).title).not.toBe(
"AI provider not configured",
);
});
it("classifies a dropped connection (ECONNRESET) as a lost-connection error", () => {
expect(
describeChatError("Cannot connect to API: read ECONNRESET", t).title,
@@ -24,6 +24,21 @@ export function describeChatError(
): ChatErrorView {
const msg = message ?? "";
// Our own "could not start the run" gate (A_RUN_BEGIN_FAILED, #486): a 503
// whose body carries this code is a TEMPORARY server-side failure while
// starting the run (e.g. a DB-pool blip), NOT an unconfigured provider. It MUST
// be matched STRICTLY BEFORE the generic 503 branch below, which would
// otherwise mislabel it "The AI provider is not configured" and tell the user
// to call an admin instead of just retrying.
if (/"code"\s*:\s*"A_RUN_BEGIN_FAILED"/.test(msg)) {
return {
title: t("Could not start the run"),
detail: t(
"The agent run could not be started. This is usually temporary — please try again.",
),
};
}
if (/"statusCode"\s*:\s*403\b/.test(msg)) {
return {
title: t("AI chat is disabled"),
@@ -168,6 +168,10 @@ export default function ShareAiWidget({
// Anonymous reader: suppress the tool-argument summary line so the
// agent's raw query/argument text isn't shown on the public share.
showInput={false}
// Anonymous reader: never paint a tool's raw errorText (it can carry
// internal detail). This is the render gate; the bytes are also
// sanitized server-side in PublicShareChatToolsService.forShare (#394).
showErrors={false}
// Anonymous reader: neutralize internal/relative links in the
// assistant's markdown so internal UUIDs/auth-gated routes don't
// leak as clickable links (external http(s) links are kept).
@@ -0,0 +1,109 @@
import {
ConflictException,
Logger,
ServiceUnavailableException,
} from '@nestjs/common';
import { AiChatService } from './ai-chat.service';
import { RunAlreadyActiveError } from './ai-chat-run.service';
/**
* Fail-fast guard for beginRun failures (#486, commit 4).
*
* When runHooks.begin() rejects for a reason OTHER than RunAlreadyActiveError
* (e.g. a DB-pool blip), the turn must NOT continue untracked. The old code
* logged and streamed anyway, leaving a run with NO run-row: in autonomous mode
* nobody could abort it (/stop can't see it, disconnect doesn't abort it, and the
* one-run gate would admit a SECOND run) — an unstoppable invisible run until
* restart. The fix throws A_RUN_BEGIN_FAILED (503) BEFORE the first byte and
* before the user row is persisted.
*
* We drive `stream()` directly on a prototype instance wired with only the
* collaborators it touches before the throw, so the assertion is on the REAL
* control flow, not a mock of it.
*/
describe('AiChatService beginRun failure (#486)', () => {
function makeService(insertSpy: jest.Mock): AiChatService {
// Bypass the (heavy) DI constructor: exercise the real stream() method on a
// bare prototype instance with just the fields reached before the throw.
// `any` because the private `logger` field makes a typed intersection collapse.
const svc = Object.create(AiChatService.prototype);
svc.aiChatRepo = {
// Existing chat -> no insert path; chatId is kept as-is.
findById: jest.fn().mockResolvedValue({ id: 'chat1' }),
};
svc.aiChatMessageRepo = { insert: insertSpy };
svc.logger = new Logger('test');
return svc as AiChatService;
}
const baseArgs = () => {
const write = jest.fn();
const res = {
raw: { write, writableEnded: false, headersSent: false },
};
return {
user: { id: 'u1' } as never,
workspace: { id: 'w1' } as never,
sessionId: 's1',
// openPage undefined -> resolveOpenPageContext returns null without any DB
// call; chatId present -> the existing-chat path.
body: { chatId: 'chat1', messages: [] } as never,
res: res as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
write,
};
};
it('throws A_RUN_BEGIN_FAILED (503) before the first byte and before persisting the user turn', async () => {
const insertSpy = jest.fn();
const svc = makeService(insertSpy);
const { write, ...args } = baseArgs();
const runHooks = {
begin: jest.fn().mockRejectedValue(new Error('DB pool exhausted')),
} as never;
let caught: unknown;
try {
await svc.stream({ ...args, runHooks });
} catch (e) {
caught = e;
}
expect(caught).toBeInstanceOf(ServiceUnavailableException);
const http = caught as ServiceUnavailableException;
expect(http.getStatus()).toBe(503);
expect(http.getResponse()).toMatchObject({ code: 'A_RUN_BEGIN_FAILED' });
// Fail-fast: nothing was written to the socket and NO user message row was
// persisted, so the turn left no orphan state to clean up.
expect(write).not.toHaveBeenCalled();
expect(insertSpy).not.toHaveBeenCalled();
});
it('still maps a lost-the-race RunAlreadyActiveError to a 409, not A_RUN_BEGIN_FAILED', async () => {
const insertSpy = jest.fn();
const svc = makeService(insertSpy);
const { write, ...args } = baseArgs();
const runHooks = {
begin: jest.fn().mockRejectedValue(new RunAlreadyActiveError('chat1')),
} as never;
let caught: unknown;
try {
await svc.stream({ ...args, runHooks });
} catch (e) {
caught = e;
}
expect(caught).toBeInstanceOf(ConflictException);
expect((caught as ConflictException).getResponse()).toMatchObject({
code: 'A_RUN_ALREADY_ACTIVE',
});
expect(write).not.toHaveBeenCalled();
expect(insertSpy).not.toHaveBeenCalled();
});
});
@@ -1,4 +1,8 @@
import { ConflictException, Logger } from '@nestjs/common';
import {
ConflictException,
Logger,
ServiceUnavailableException,
} from '@nestjs/common';
// Mock the AI SDK so we can PROVE no provider call is made for the turn we are
// about to reject. The race rejection happens at runHooks.begin(), long before
@@ -360,22 +364,22 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
});
/**
* F14 — the begin-failure RESILIENCE branch (the `else` of the run-race guard).
* F14 — the begin-failure branch (the `else` of the run-race guard).
*
* stream() wraps runHooks.begin in try/catch with TWO branches:
* - RunAlreadyActiveError -> 409 ConflictException (pinned above).
* - ANY OTHER begin failure -> SWALLOW + continue UNTRACKED on the socket signal
* (legacy fallback): it logs "...streaming without run tracking", leaves
* `effectiveSignal = signal` (runId undefined) and serves the turn anyway.
* - ANY OTHER begin failure -> throw ServiceUnavailableException(A_RUN_BEGIN_FAILED)
* BEFORE the first byte (#486, commit 4).
*
* The contract: a transient beginRun failure (e.g. a non-unique DB error inserting
* the run row) must STILL serve the user's turn — it must NOT re-throw and must NOT
* be misclassified as a 409. A regression that re-threw here would break EVERY turn
* on a begin failure with nothing to catch it. This branch is otherwise undriven by
* any spec, so it is pinned here SEPARATELY from the 409 path: a plain begin error
* proceeds to streamText with the SOCKET signal and still persists the user turn.
* POLICY CHANGE (#486): the OLD contract here was "SWALLOW + stream the turn
* UNTRACKED on the socket signal". That was reversed: an untracked run is
* invisible to /stop, is not aborted on disconnect, and slips past the one-run
* gate — an unstoppable ghost run in autonomous mode. Now a plain begin failure
* FAILS the turn fast with a 503 A_RUN_BEGIN_FAILED, before any user row is
* persisted and before streamText runs. This case is INVERTED (not deleted) so
* the "plain begin failure" path stays explicitly pinned under the new policy.
*/
describe('AiChatService.stream — begin-failure resilience / legacy fallback (#184 F14)', () => {
describe('AiChatService.stream — begin-failure fails the turn (#184 F14 / #486)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
function makeStreamResult() {
@@ -455,7 +459,7 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
afterEach(() => jest.restoreAllMocks());
it('a PLAIN begin() failure (NOT RunAlreadyActiveError) does NOT 409 — it swallows, logs, and streams the turn UNTRACKED on the socket signal', async () => {
it('a PLAIN begin() failure (NOT RunAlreadyActiveError) FAILS the turn with a 503 A_RUN_BEGIN_FAILED before the first byte — NO untracked stream (#486)', async () => {
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
@@ -487,28 +491,26 @@ describe('AiChatService.stream — begin-failure resilience / legacy fallback (#
} as never,
});
// The turn proceeds: NO throw at all (in particular NOT a 409).
await expect(promise).resolves.toBeUndefined();
// NEW POLICY: the turn is REJECTED with a 503 A_RUN_BEGIN_FAILED (not a 409,
// and NOT swallowed into an untracked stream).
await expect(promise).rejects.toBeInstanceOf(ServiceUnavailableException);
const err = (await promise.catch(
(e) => e,
)) as ServiceUnavailableException;
expect(err.getStatus()).toBe(503);
expect(err.getResponse()).toMatchObject({ code: 'A_RUN_BEGIN_FAILED' });
expect(begin).toHaveBeenCalledTimes(1);
// The resilience branch logged the legacy-fallback warning.
// It logged the fail-the-turn line.
expect(errorSpy).toHaveBeenCalledWith(
expect.stringContaining('streaming without run tracking'),
expect.stringContaining('failing the turn'),
expect.anything(),
);
// The turn really streamed: the user message was persisted and streamText ran.
expect(aiChatMessageRepo.insert).toHaveBeenCalled();
expect(streamTextMock).toHaveBeenCalledTimes(1);
// The decisive wiring: with no run handle, the fallback uses the SOCKET signal
// (effectiveSignal = signal, runId undefined) — not a run-bound signal. #444:
// the signal is unioned with the degeneration controller via AbortSignal.any,
// so assert the socket abort still reaches the turn rather than identity.
const passed = streamTextMock.mock.calls[0][0].abortSignal as AbortSignal;
expect(passed.aborted).toBe(false);
socketController.abort();
expect(passed.aborted).toBe(true);
// Fail-fast: the turn NEVER streamed — no user row persisted, no streamText
// call, so no orphan/untracked run was left behind.
expect(aiChatMessageRepo.insert).not.toHaveBeenCalled();
expect(streamTextMock).not.toHaveBeenCalled();
});
});
@@ -4,6 +4,7 @@ import {
Injectable,
Logger,
OnModuleInit,
ServiceUnavailableException,
} from '@nestjs/common';
import { FastifyReply } from 'fastify';
import {
@@ -55,6 +56,7 @@ import {
import {
isDegenerateOutput,
truncateDegeneratedTail,
shouldCheckDegeneration,
} from './output-degeneration';
// Max agent steps per turn. One step = one model generation; a step that calls
@@ -797,12 +799,32 @@ export class AiChatService implements OnModuleInit {
code: 'A_RUN_ALREADY_ACTIVE',
});
}
// Any OTHER run-start failure must not break the turn — fall back to the
// socket signal (legacy behavior) and stream anyway.
// Any OTHER run-start failure (e.g. a DB-pool blip) must FAIL THE TURN,
// not silently stream without a run-row. The old fallback let the turn
// continue untracked: in autonomous mode nobody could then abort it —
// /stop can't see a run that doesn't exist, a client disconnect doesn't
// abort it, and the one-run-per-chat gate would let a SECOND run in. That
// is an unstoppable, invisible run until process restart. Reject NOW,
// BEFORE the first byte (nothing is written yet, no user row inserted, no
// MCP lease taken), so the controller's post-hijack catch turns this
// HttpException into an honest 503 on the raw socket. Same policy for BOTH
// modes — #487 inherits it (no mode-branching here).
this.logger.error(
`Failed to begin agent run (chat ${chatId}); streaming without run tracking`,
`Failed to begin agent run (chat ${chatId}); failing the turn`,
err as Error,
);
throw new ServiceUnavailableException({
message:
'Could not start the agent run. This is usually temporary — please try again.',
code: 'A_RUN_BEGIN_FAILED',
// Self-describe the status in the body: the controller's post-hijack
// catch writes getResponse() verbatim onto the raw socket, and an
// object-arg HttpException does NOT inject statusCode. Without it the
// client's 503 classifier (which reads the body JSON) could not see the
// status. With it present, the client's A_RUN_BEGIN_FAILED branch (which
// runs strictly before the generic-503 branch) shows "temporary, retry".
statusCode: 503,
});
}
}
@@ -1080,7 +1102,6 @@ export class AiChatService implements OnModuleInit {
const degenerationController = new AbortController();
let degenerationDetected = false;
let lastDegenerationCheckLen = 0;
const DEGENERATION_CHECK_STEP = 2000;
// Step-granular durability (#183): create the assistant row UPFRONT in the
// 'streaming' state (before any token), then UPDATE it as each step finishes
@@ -1254,8 +1275,10 @@ export class AiChatService implements OnModuleInit {
// trigger, abort the run ONCE with a distinguishable reason.
if (
!degenerationDetected &&
inProgressText.length - lastDegenerationCheckLen >=
DEGENERATION_CHECK_STEP
shouldCheckDegeneration(
inProgressText.length,
lastDegenerationCheckLen,
)
) {
lastDegenerationCheckLen = inProgressText.length;
if (isDegenerateOutput(inProgressText)) {
@@ -1275,6 +1298,13 @@ export class AiChatService implements OnModuleInit {
// the in-progress accumulator for the next step.
capturedSteps.push(step as StepLike);
inProgressText = '';
// Reset the degeneration-check watermark too (#486): it tracks a byte
// offset INTO inProgressText, so once that resets to '' a stale (large)
// mark makes `inProgressText.length - lastDegenerationCheckLen` go
// negative and the throttled detector stays silent until a later step's
// text re-grows past the old offset — a whole degenerate step could slip
// through undetected. Zeroing it re-arms the check from the next byte.
lastDegenerationCheckLen = 0;
// Step-granular durability (#183): persist this finished step (its text +
// tool calls + tool RESULTS) the moment it ends, so a process death after
// this point still recovers the step. Not awaited here (never block the
@@ -1,11 +1,24 @@
import { Logger } from '@nestjs/common';
import { streamText } from 'ai';
import {
hasRepeatedLineRun,
hasPeriodicTail,
isDegenerateOutput,
truncateDegeneratedTail,
shouldCheckDegeneration,
DEGENERATION_CHECK_STEP,
REPEATED_LINES_THRESHOLD,
MIN_PERIOD_REPEATS,
} from './output-degeneration';
import { AiChatService } from './ai-chat.service';
// Mock ONLY streamText so we can capture the onChunk/onStepFinish callbacks the
// service registers and drive them by hand; every other `ai` export the service
// uses (convertToModelMessages, stepCountIs, …) stays real.
jest.mock('ai', () => {
const actual = jest.requireActual('ai');
return { ...actual, streamText: jest.fn() };
});
/**
* Unit tests for the token-degeneration detector (#444) — the sole anti-babble
@@ -180,3 +193,188 @@ describe('truncateDegeneratedTail', () => {
expect(truncateDegeneratedTail(text)).toBe(text);
});
});
/**
* Throttle + step-boundary reset (#486). The stream keeps a watermark
* (`lastDegenerationCheckLen`) that is an OFFSET into the accumulated step text.
* On a step boundary the accumulator resets to '', so the watermark MUST reset to
* 0 too — otherwise the throttle goes silent for the whole next step. These tests
* pin the pure decision AND the reset property that ai-chat.service.onStepFinish
* now enforces.
*/
describe('shouldCheckDegeneration (throttle) + step-boundary reset (#486)', () => {
it('fires once the text grows a full DEGENERATION_CHECK_STEP past the mark', () => {
expect(shouldCheckDegeneration(DEGENERATION_CHECK_STEP, 0)).toBe(true);
expect(shouldCheckDegeneration(DEGENERATION_CHECK_STEP - 1, 0)).toBe(false);
expect(shouldCheckDegeneration(5000, 3000)).toBe(true); // grew 2000 since mark
expect(shouldCheckDegeneration(4000, 3000)).toBe(false); // grew only 1000
});
it('BUG (no reset): a stale large watermark silences the next step', () => {
// End of a long step: the watermark sits at 5000. The step ends and the
// accumulator resets to '' — but if the watermark is NOT reset, a fresh short
// degenerate burst (length 2000) never triggers a check: 2000 - 5000 < STEP.
const staleWatermark = 5000;
const nextStepLen = DEGENERATION_CHECK_STEP; // a fresh 2KB burst
expect(shouldCheckDegeneration(nextStepLen, staleWatermark)).toBe(false);
});
it('FIX (reset to 0): the same short degenerate burst IS checked and detected', () => {
// onStepFinish now zeroes the watermark, so the fresh burst re-arms the check.
const resetWatermark = 0;
const degenerateBurst = 'loadTools.\n'.repeat(300); // real degeneration
expect(degenerateBurst.length).toBeGreaterThanOrEqual(DEGENERATION_CHECK_STEP);
// The throttle now fires...
expect(
shouldCheckDegeneration(degenerateBurst.length, resetWatermark),
).toBe(true);
// ...and the detector catches the loop that would otherwise stream unchecked.
expect(isDegenerateOutput(degenerateBurst)).toBe(true);
});
});
/**
* BEHAVIOR guard for the ACTUAL fix (#486, ai-chat.service.onStepFinish resets
* lastDegenerationCheckLen to 0). The pure tests above use a hard-coded
* resetWatermark, so a REVERT of the real `lastDegenerationCheckLen = 0` line
* would not redden any of them. This drives the REAL onChunk/onStepFinish
* closures from stream() end to end and asserts the run is aborted when a fresh
* degenerate burst arrives in the step AFTER a long clean step — which only
* happens if the watermark was actually zeroed on the step boundary.
*/
describe('AiChatService: onStepFinish re-arms the degeneration watermark (#486)', () => {
const streamTextMock = streamText as unknown as jest.Mock;
function makeRes() {
return {
raw: {
writeHead: jest.fn(),
write: jest.fn(),
once: jest.fn(),
on: jest.fn(),
flushHeaders: jest.fn(),
writableEnded: false,
destroyed: false,
},
};
}
function makeService() {
const aiChatRepo = {
findById: jest.fn(async () => ({ id: 'chat-1', workspaceId: 'ws-1' })),
insert: jest.fn(),
};
const aiChatMessageRepo = {
insert: jest.fn(async () => ({ id: 'msg-1' })),
findAllByChat: jest.fn(async () => []),
update: jest.fn(async () => ({ id: 'msg-1' })),
};
const aiSettings = { resolve: jest.fn(async () => ({})) };
const tools = { forUser: jest.fn(async () => ({})) };
const mcpClients = {
toolsFor: jest.fn(async () => ({
tools: {},
clients: [],
outcomes: [],
instructions: [],
})),
};
return new AiChatService(
{} as never, // ai
aiChatRepo as never,
aiChatMessageRepo as never,
{} as never, // aiChatPageSnapshotRepo
aiSettings as never,
tools as never,
mcpClients as never,
{} as never, // aiAgentRoleRepo
{} as never, // pageRepo
{} as never, // pageAccess
{
isAiChatDeferredToolsEnabled: () => false,
// Lockdown OFF -> the degeneration guard is the active anti-babble path.
isAiChatFinalStepLockdownEnabled: () => false,
} as never, // environment
);
}
beforeEach(() => {
streamTextMock.mockReset();
jest.spyOn(Logger.prototype, 'log').mockImplementation(() => undefined as never);
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined as never);
});
afterEach(() => jest.restoreAllMocks());
it('aborts on a fresh degenerate burst in the NEXT step (reverting the reset line reddens this)', async () => {
let captured:
| {
onChunk?: (e: { chunk: { type: string; text: string } }) => void;
onStepFinish?: (step: unknown) => void;
abortSignal?: AbortSignal;
}
| undefined;
streamTextMock.mockImplementation((opts: never) => {
captured = opts;
return {
consumeStream: jest.fn(),
pipeUIMessageStreamToResponse: jest.fn(),
};
});
const svc = makeService();
await svc.stream({
user: { id: 'user-1' } as never,
workspace: { id: 'ws-1' } as never,
sessionId: 'sess-1',
body: {
chatId: 'chat-1',
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'hi' }] },
],
} as never,
res: makeRes() as never,
signal: new AbortController().signal,
model: {} as never,
role: null,
// No runHooks -> legacy path (socket signal), degeneration guard active.
});
expect(streamTextMock).toHaveBeenCalledTimes(1);
const onChunk = captured!.onChunk!;
const onStepFinish = captured!.onStepFinish!;
const abortSignal = captured!.abortSignal!;
expect(abortSignal.aborted).toBe(false);
// STEP 1: a LONG, non-degenerate first step. Distinct lines never trip the
// detector, but they advance the throttle watermark far past the burst size
// that follows (to ~5x the step). This is the stale watermark that, WITHOUT
// the reset, would silence step 2.
let counter = 0;
let accumulated = 0;
while (accumulated < DEGENERATION_CHECK_STEP * 5) {
const line = `unique clean line number ${counter++} with distinct words\n`;
accumulated += line.length;
onChunk({ chunk: { type: 'text-delta', text: line } });
}
expect(abortSignal.aborted).toBe(false); // clean step must not abort
// STEP BOUNDARY: the real onStepFinish resets inProgressText AND (the fix)
// zeroes lastDegenerationCheckLen.
onStepFinish({ text: 'a clean first step', toolCalls: [], toolResults: [] });
// STEP 2: a FRESH, short degenerate burst (~3.3KB). Its length is far below
// the step-1 stale watermark (~10KB), so WITHOUT the reset the throttle stays
// silent and this streams unchecked. WITH the reset (watermark 0) it re-arms,
// the detector fires, and the run aborts.
const burst = 'loadTools.\n'.repeat(300);
expect(burst.length).toBeGreaterThanOrEqual(DEGENERATION_CHECK_STEP);
expect(burst.length).toBeLessThan(DEGENERATION_CHECK_STEP * 5);
onChunk({ chunk: { type: 'text-delta', text: burst } });
// The decisive assertion: the composed abortSignal (unioned with the
// degeneration controller) is now aborted. Reverting `lastDegenerationCheckLen
// = 0` in onStepFinish makes this stay false.
expect(abortSignal.aborted).toBe(true);
});
});
@@ -131,6 +131,32 @@ export function isDegenerateOutput(text: string): boolean {
return hasRepeatedLineRun(text) || hasPeriodicTail(text);
}
/**
* How many bytes the in-progress text must grow before the (amortized) tail
* heuristics are re-run. Shared with ai-chat.service so the throttle the stream
* applies is the SAME one the unit test drives.
*/
export const DEGENERATION_CHECK_STEP = 2000;
/**
* Throttle decision for the degeneration guard (#444/#486). Returns true when
* the accumulated text has grown at least DEGENERATION_CHECK_STEP bytes past the
* last-checked offset, so the pure rules only fire every ~2KB. Pure; the caller
* updates its watermark to `textLen` when this returns true.
*
* The watermark is an offset INTO the accumulator, so when the accumulator is
* reset to '' on a step boundary the caller MUST reset the watermark to 0 too
* (#486). Otherwise `textLen - lastCheckLen` goes negative after the reset and
* this returns false until a later step re-grows past the stale offset — a whole
* degenerate step could stream unchecked.
*/
export function shouldCheckDegeneration(
textLen: number,
lastCheckLen: number,
): boolean {
return textLen - lastCheckLen >= DEGENERATION_CHECK_STEP;
}
/**
* Truncate a degenerated tail before persist so hundreds of KB of garbage never
* reach the DB / replay (#444). Keeps everything up to and including the FIRST
@@ -0,0 +1,241 @@
// Break the editor-ext import chain (share.service -> collaboration.util ->
// @docmost/editor-ext -> @tiptap/core) that is unresolvable in this jest env and
// pre-existingly breaks these specs. jsonToMarkdown is never reached in these
// tests (the tools fail before rendering markdown).
jest.mock('../../collaboration/collaboration.util', () => ({
jsonToMarkdown: () => '',
}));
import { Logger } from '@nestjs/common';
import { MockLanguageModelV3, simulateReadableStream } from 'ai/test';
import { PublicShareChatService } from './public-share-chat.service';
import { PublicShareChatToolsService } from './tools/public-share-chat-tools.service';
/**
* SECURITY integration guard for #394 (commit 5): a tool's or the provider's raw
* error text must NOT leak to an anonymous public-share reader.
*
* The render gate (ToolCallCard showErrors=false) hides the text in the DOM but
* NOT on the wire, so this test asserts on the RAW SSE BYTES the server writes —
* exactly the channel the render gate masks. We drive the real
* PublicShareChatService.stream() with a real share toolset (its underlying
* services mocked to fail) and a mock model, then inspect every byte piped to the
* fake socket.
*/
// A minimal ServerResponse stand-in that records every written chunk.
class FakeSocket {
chunks: string[] = [];
statusCode = 200;
writableEnded = false;
destroyed = false;
headersSent = false;
writeHead(): this {
this.headersSent = true;
return this;
}
setHeader(): void {}
removeHeader(): void {}
getHeader(): undefined {
return undefined;
}
flushHeaders(): void {}
write(chunk: unknown): boolean {
this.chunks.push(
typeof chunk === 'string' ? chunk : Buffer.from(chunk as never).toString('utf8'),
);
return true;
}
end(chunk?: unknown): void {
if (chunk) this.write(chunk);
this.writableEnded = true;
}
on(): this {
return this;
}
once(): this {
return this;
}
get body(): string {
return this.chunks.join('');
}
}
/** Mock model that issues one getSharePage tool call, then finishes with text. */
function toolCallingModel(): MockLanguageModelV3 {
let call = 0;
return new MockLanguageModelV3({
doStream: async () => {
call++;
if (call === 1) {
return {
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'tool-input-start' as const, id: 't1', toolName: 'getSharePage' },
{ type: 'tool-input-end' as const, id: 't1' },
{
type: 'tool-call' as const,
toolCallId: 't1',
toolName: 'getSharePage',
input: '{"pageId":"secret-page"}',
},
{
type: 'finish' as const,
finishReason: { unified: 'tool-calls' as const, raw: 'tool_calls' },
usage: {
inputTokens: { total: 1, noCache: undefined, cacheRead: undefined, cacheWrite: undefined },
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
};
}
return {
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'text-start' as const, id: '1' },
{ type: 'text-delta' as const, id: '1', delta: 'Sorry.' },
{ type: 'text-end' as const, id: '1' },
{
type: 'finish' as const,
finishReason: { unified: 'stop' as const, raw: 'stop' },
usage: {
inputTokens: { total: 1, noCache: undefined, cacheRead: undefined, cacheWrite: undefined },
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
};
},
});
}
/** Mock model whose stream emits a provider error carrying an internal secret. */
function providerErrorModel(secret: string): MockLanguageModelV3 {
return new MockLanguageModelV3({
doStream: async () => ({
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{
type: 'error' as const,
error: {
statusCode: 503,
message: 'Service Unavailable',
responseBody: `upstream ${secret} model=internal-gpt`,
},
},
],
}),
}),
});
}
function makeService(toolsService: PublicShareChatToolsService): {
svc: PublicShareChatService;
logSpy: jest.SpyInstance;
} {
const svc = Object.create(PublicShareChatService.prototype);
const logger = new Logger('test');
const logSpy = jest.spyOn(logger, 'error').mockImplementation(() => undefined);
jest.spyOn(logger, 'warn').mockImplementation(() => undefined);
svc.tools = toolsService;
svc.logger = logger;
svc.tokenBudget = { record: jest.fn().mockResolvedValue(undefined) };
return { svc, logSpy };
}
async function runStream(
svc: PublicShareChatService,
model: MockLanguageModelV3,
): Promise<FakeSocket> {
const socket = new FakeSocket();
await svc.stream({
workspaceId: 'ws1',
shareId: 'share1',
share: { id: 'share1', pageId: 'p1', sharedPage: { id: 'p1', title: 'Docs' } },
openedPage: null,
messages: [
{ id: 'm1', role: 'user', parts: [{ type: 'text', text: 'read the page' }] } as never,
],
res: { raw: socket } as never,
signal: new AbortController().signal,
model: model as never,
role: null,
});
// Let the piped stream drain fully.
await new Promise((r) => setTimeout(r, 300));
return socket;
}
describe('public share chat error leak (#394)', () => {
afterEach(() => jest.restoreAllMocks());
it('does NOT leak a tool\'s raw internal error to the SSE bytes (generic classified string instead)', async () => {
const SECRET = 'INTERNAL_baseUrl_http://provider.internal:8080/v1';
const shareService = {
// The canonical boundary throws a RAW internal error (with a secret).
resolveReadableSharePage: jest
.fn()
.mockRejectedValue(new Error(`db failed at ${SECRET} stack@line42`)),
};
const tools = new PublicShareChatToolsService(
shareService as never,
{} as never,
{} as never,
);
const { svc } = makeService(tools);
const socket = await runStream(svc, toolCallingModel());
// The tool-output-error frame is present on the wire...
expect(socket.body).toContain('tool-output-error');
// ...but it carries ONLY the generic classified string — never the secret,
// the raw driver message, or a stack fragment.
expect(socket.body).toContain('The tool could not complete the request.');
expect(socket.body).not.toContain(SECRET);
expect(socket.body).not.toContain('stack@line42');
expect(socket.body).not.toContain('db failed');
});
it('passes a SAFE ShareToolError message (page not available) through to the bytes', async () => {
const shareService = {
// Not found in this share -> the tool throws the classified SAFE message.
resolveReadableSharePage: jest.fn().mockResolvedValue(null),
};
const tools = new PublicShareChatToolsService(
shareService as never,
{} as never,
{} as never,
);
const { svc } = makeService(tools);
const socket = await runStream(svc, toolCallingModel());
expect(socket.body).toContain('tool-output-error');
expect(socket.body).toContain('not available in this share');
});
it('does NOT leak a provider error (statusCode + response body) to the SSE bytes', async () => {
const SECRET = 'http://provider.internal:8080';
const tools = new PublicShareChatToolsService(
{} as never,
{} as never,
{} as never,
);
const { svc, logSpy } = makeService(tools);
const socket = await runStream(svc, providerErrorModel(SECRET));
// The anon sees a fixed classified string, not the provider body/baseUrl/model.
expect(socket.body).toContain('temporarily unavailable');
expect(socket.body).not.toContain(SECRET);
expect(socket.body).not.toContain('internal-gpt');
// The FULL provider detail is logged server-side only.
const logged = logSpy.mock.calls.map((c) => String(c[0])).join('\n');
expect(logged).toContain(SECRET);
});
});
@@ -12,7 +12,10 @@ import { AiAgentRoleRepo } from '@docmost/db/repos/ai-agent-roles/ai-agent-roles
import { AiAgentRole } from '@docmost/db/types/entity.types';
import { AiService } from '../../integrations/ai/ai.service';
import { AiSettingsService } from '../../integrations/ai/ai-settings.service';
import { PublicShareChatToolsService } from './tools/public-share-chat-tools.service';
import {
PublicShareChatToolsService,
ShareToolError,
} from './tools/public-share-chat-tools.service';
import { buildShareSystemPrompt } from './public-share-chat.prompt';
import { roleModelOverride } from './roles/role-model-config';
import {
@@ -102,6 +105,30 @@ export function filterShareTranscript(messages: UIMessage[]): UIMessage[] {
);
}
/**
* Fixed, classified strings an ANONYMOUS share reader may see when the assistant
* stream fails (#394). These reveal NOTHING about the internal provider, its
* baseUrl, the model name, or the raw response body — unlike describeProviderError
* (which is for the server log / the authenticated operator only). We classify by
* HTTP status where available so the reader still gets a useful hint (retry vs.
* give up) without any internal detail.
*/
export function classifyAnonStreamError(error: unknown): string {
const status =
typeof error === 'object' && error !== null
? (error as { statusCode?: number }).statusCode
: undefined;
if (status === 429) {
return 'The assistant is receiving too many requests right now. Please try again shortly.';
}
if (typeof status === 'number' && status >= 500) {
return 'The assistant is temporarily unavailable. Please try again.';
}
// Any other failure (including a bare connection error with no status): a
// single neutral line. No provider identity, no config, no response body.
return 'The assistant could not complete your request. Please try again.';
}
/**
* Anonymous, read-only AI assistant for a single PUBLIC share tree.
*
@@ -318,11 +345,28 @@ export class PublicShareChatService {
result.pipeUIMessageStreamToResponse(res.raw, {
headers: { 'X-Accel-Buffering': 'no' },
onError: (error: unknown) => {
// Reuse the shared formatter so provider error formatting stays
// unified between the log line and the streamed error message — a
// share reader sees 402/429/503 causes consistently with the
// authenticated path.
return describeProviderError(error, 'AI stream error');
// SECURITY (#394): the string this returns is written verbatim into the
// SSE error frame delivered to an ANONYMOUS reader (for a tool failure
// it becomes the atomic `tool-output-error` frame's errorText; for a
// stream/provider failure, the terminal error frame).
//
// A ShareToolError is already a classified, safe tool message (see
// PublicShareChatToolsService.wrapToolErrors) — pass it through so the
// reader still gets the useful "page not available in this share" hint.
if (error instanceof ShareToolError) {
return error.message;
}
// Anything else is a provider/stream error. describeProviderError
// bundles the provider statusCode AND response body, which can carry the
// internal baseUrl or model name — NEVER expose that to the public. Log
// the full detail server-side only and return a fixed classified string.
this.logger.error(
`Public share chat pipe error: ${describeProviderError(
error,
'AI stream error',
)}`,
);
return classifyAnonStreamError(error);
},
});
@@ -1,7 +1,15 @@
import { createHash } from 'node:crypto';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import {
mkdtempSync,
mkdirSync,
writeFileSync,
rmSync,
readdirSync,
statSync,
readFileSync,
} from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { dirname, join, relative, sep } from 'node:path';
import { computeSrcRegistryStamp } from './docmost-client.loader';
@@ -30,10 +38,14 @@ function assertStaleGuard(
}
}
// Build a throwaway `<pkg>/build/index.js` + optional `<pkg>/src/tool-specs.ts`
// layout so `computeSrcRegistryStamp(<pkg>/build/index.js)` resolves src the same
// way the loader does (dirname(dirname(entry))/src/tool-specs.ts).
function makeFakePackage(toolSpecsSource: string | null): {
// Build a throwaway `<pkg>/build/index.js` + optional `<pkg>/src/` tree so
// `computeSrcRegistryStamp(<pkg>/build/index.js)` resolves src the same way the
// loader does (dirname(dirname(entry))/src). Since #486 the stamp hashes the WHOLE
// src tree, so a fixture is a { relPath: content } map. A bare string is sugar for
// a single `tool-specs.ts`; `null` means "no src tree" (the prod no-op path).
function makeFakePackage(
src: string | Record<string, string> | null,
): {
entry: string;
cleanup: () => void;
} {
@@ -42,10 +54,15 @@ function makeFakePackage(toolSpecsSource: string | null): {
mkdirSync(buildDir, { recursive: true });
const entry = join(buildDir, 'index.js');
writeFileSync(entry, '// fake @docmost/mcp build entry\n', 'utf8');
if (toolSpecsSource !== null) {
if (src !== null) {
const files =
typeof src === 'string' ? { 'tool-specs.ts': src } : src;
const srcDir = join(root, 'src');
mkdirSync(srcDir, { recursive: true });
writeFileSync(join(srcDir, 'tool-specs.ts'), toolSpecsSource, 'utf8');
for (const [rel, content] of Object.entries(files)) {
const full = join(srcDir, rel);
mkdirSync(dirname(full), { recursive: true });
writeFileSync(full, content, 'utf8');
}
}
return { entry, cleanup: () => rmSync(root, { recursive: true, force: true }) };
}
@@ -93,34 +110,109 @@ describe('computeSrcRegistryStamp (#447 stale-build guard)', () => {
}
});
// CROSS-IMPL EQUALITY (covers reviewer suggestion 2). The SAME fixed input and
// #486 CORE (negative): an edit to a NON-tool-specs src file (client.ts) with a
// rebuild NOT run must move the src stamp away from the built REGISTRY_STAMP, so
// the loader's stale-check refuses. Under the old tool-specs.ts-only hash this
// edit was invisible and a stale build/ served the old client.ts silently.
it('a client.ts edit (no rebuild) moves the src stamp -> loader refuses (#486)', () => {
// "Built" state: the package as it was compiled.
const built = makeFakePackage({
'tool-specs.ts': 'export const SPECS = 1;\n',
'client.ts': "export const impl = 'v1';\n",
});
// "Dev edited src, forgot to rebuild": client.ts changed, tool-specs.ts not.
const edited = makeFakePackage({
'tool-specs.ts': 'export const SPECS = 1;\n',
'client.ts': "export const impl = 'v2';\n",
});
try {
const builtStamp = computeSrcRegistryStamp(built.entry);
const editedStamp = computeSrcRegistryStamp(edited.entry);
expect(builtStamp).not.toBeNull();
expect(editedStamp).not.toBe(builtStamp);
// build/ still carries builtStamp; src now hashes to editedStamp -> refuse.
expect(() => assertStaleGuard(editedStamp, builtStamp as string)).toThrow(
STALE_BUILD_MESSAGE,
);
} finally {
built.cleanup();
edited.cleanup();
}
});
// *.generated.ts is excluded (the codegen's own output — a fixed-point cycle
// otherwise): its presence/content must not move the stamp.
it('excludes *.generated.ts from the stamp', () => {
const without = makeFakePackage({ 'tool-specs.ts': 'x\n' });
const withGen = makeFakePackage({
'tool-specs.ts': 'x\n',
'registry-stamp.generated.ts': 'export const REGISTRY_STAMP = "abc";\n',
});
try {
expect(computeSrcRegistryStamp(withGen.entry)).toBe(
computeSrcRegistryStamp(without.entry),
);
} finally {
without.cleanup();
withGen.cleanup();
}
});
// CROSS-IMPL EQUALITY (covers reviewer suggestion 2). The SAME fixed tree and
// EXPECTED hash are asserted in the mcp-side node test
// (packages/mcp/test/unit/registry-stamp.test.mjs) against the codegen's
// `computeRegistryStamp`. Asserting the SAME pair here against the loader's
// `computeSrcRegistryStamp` proves both implementations normalize+hash
// `computeSrcRegistryStamp` proves both implementations enumerate+normalize+hash
// identically; a divergence in EITHER side reddens one of the two tests.
it('matches the documented cross-impl hash for a fixed input', () => {
const FIXED_INPUT = 'line1\r\nline2\n';
const EXPECTED =
'683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83';
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
const CROSS_IMPL_TREE = {
'tool-specs.ts': 'line1\r\nline2\n',
'client/read.ts': 'export const R = 1;\n',
'registry-stamp.generated.ts': 'export const REGISTRY_STAMP="ignored";\n',
};
const CROSS_IMPL_EXPECTED =
'131c1b9e4e2f5a7d6cef91ca8df619822b442f52bc45ebd09474a4c1d6728616';
it('matches the documented cross-impl hash for a fixed tree', () => {
const { entry, cleanup } = makeFakePackage(CROSS_IMPL_TREE);
try {
expect(computeSrcRegistryStamp(entry)).toBe(EXPECTED);
expect(computeSrcRegistryStamp(entry)).toBe(CROSS_IMPL_EXPECTED);
} finally {
cleanup();
}
});
it('the documented EXPECTED is the normalize+sha256 of the fixed input', () => {
// Proves EXPECTED is not a magic constant but the documented computation.
const FIXED_INPUT = 'line1\r\nline2\n';
const normalized = FIXED_INPUT.replace(/\r\n/g, '\n').replace(/\n$/, '');
const expected = createHash('sha256')
.update(normalized, 'utf8')
.digest('hex');
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
it('the documented EXPECTED is the enumerate+normalize+sha256 of the tree', () => {
// Proves EXPECTED is not a magic constant but the documented computation — a
// local re-implementation of the loader's tree walk.
const { entry, cleanup } = makeFakePackage(CROSS_IMPL_TREE);
try {
expect(computeSrcRegistryStamp(entry)).toBe(expected);
const srcDir = join(dirname(dirname(entry)), 'src');
const collect = (dir: string): string[] => {
const out: string[] = [];
for (const e of readdirSync(dir)) {
const f = join(dir, e);
if (statSync(f).isDirectory()) out.push(...collect(f));
else if (e.endsWith('.ts') && !e.endsWith('.generated.ts'))
out.push(f);
}
return out;
};
const files = collect(srcDir)
.map((abs) => ({ rel: relative(srcDir, abs).split(sep).join('/'), abs }))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const h = createHash('sha256');
for (const { rel, abs } of files) {
const n = readFileSync(abs, 'utf8')
.replace(/\r\n/g, '\n')
.replace(/\n$/, '');
h.update(rel, 'utf8');
h.update('\0', 'utf8');
h.update(n, 'utf8');
h.update('\0', 'utf8');
}
const localHash = h.digest('hex');
expect(computeSrcRegistryStamp(entry)).toBe(localHash);
expect(localHash).toBe(CROSS_IMPL_EXPECTED);
} finally {
cleanup();
}
@@ -1,6 +1,6 @@
import { createHash } from 'node:crypto';
import { existsSync, readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs';
import { dirname, join, relative, sep } from 'node:path';
import { pathToFileURL } from 'node:url';
import type { DocmostClient, SharedToolSpec } from '@docmost/mcp';
@@ -191,33 +191,52 @@ interface DocmostMcpModule {
* present. Returns the stamp string, or `null` when the source is absent (a prod
* image ships only build/, no src/). MUST stay byte-for-byte identical to
* packages/mcp/scripts/gen-registry-stamp.mjs's `computeRegistryStamp` so the
* build-time and src-time hashes agree: same input file (src/tool-specs.ts), same
* normalization (CRLF -> LF, strip a single trailing newline), same sha256.
* build-time and src-time hashes agree: same file set (every src/**\/*.ts except
* *.generated.ts), same POSIX-relative sort, same per-file normalization (CRLF ->
* LF, strip a single trailing newline) with the same path+content framing, same
* sha256. Hashing the WHOLE src tree (not just tool-specs.ts) is #486: an edit to
* client.ts / a client/* module / comment-signal / drawio-* without a rebuild
* must also be caught, otherwise build/ silently serves the old code.
*
* DEV vs PROD detection is by FILE EXISTENCE, not NODE_ENV: we resolve the
* package's own directory from `require.resolve('@docmost/mcp')` (which points at
* build/index.js) and look for ../src/tool-specs.ts next to it. In a dev/test
* worktree that file exists; in a prod image (build/ only, src/ stripped) it does
* not, so this returns null and the caller skips the check. Any error (ENOENT, a
* bad resolve) is swallowed to null — the stale-check must NEVER break startup.
* build/index.js) and look for ../src next to it. In a dev/test worktree that
* directory exists; in a prod image (build/ only, src/ stripped) it does not, so
* this returns null and the caller skips the check. Any error (ENOENT, a bad
* resolve) is swallowed to null — the stale-check must NEVER break startup.
*
* Exported for unit testing (docmost-client.loader.spec.ts): the export keyword
* is behaviourally a no-op — the module-internal caller `loadDocmostMcp` is
* unaffected. The test drives the null (no-src) path and asserts this
* normalize+sha256 stays identical to the codegen's `computeRegistryStamp`.
* enumerate+normalize+sha256 stays identical to the codegen's
* `computeRegistryStamp`.
*/
export function computeSrcRegistryStamp(packageEntry: string): string | null {
try {
// packageEntry is <pkg>/build/index.js; the source lives at <pkg>/src/.
const toolSpecsPath = join(
dirname(dirname(packageEntry)),
'src',
'tool-specs.ts',
);
if (!existsSync(toolSpecsPath)) return null; // prod: no src tree -> skip.
const source = readFileSync(toolSpecsPath, 'utf8');
const normalized = source.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
const srcDir = join(dirname(dirname(packageEntry)), 'src');
if (!existsSync(srcDir)) return null; // prod: no src tree -> skip.
// Enumerate every src/**\/*.ts except the codegen's own *.generated.ts
// output (including it would be a fixed-point cycle). Sort by POSIX-relative
// path so ordering is platform-independent, then fold each file's relative
// path + normalized content into one hash — identical to the codegen.
const files = collectStampFiles(srcDir)
.map((abs) => ({
rel: relative(srcDir, abs).split(sep).join('/'),
abs,
}))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const hash = createHash('sha256');
for (const { rel, abs } of files) {
const normalized = readFileSync(abs, 'utf8')
.replace(/\r\n/g, '\n')
.replace(/\n$/, '');
hash.update(rel, 'utf8');
hash.update('\0', 'utf8');
hash.update(normalized, 'utf8');
hash.update('\0', 'utf8');
}
return hash.digest('hex');
} catch {
// Never let a resolution/read hiccup break server startup — treat as "no
// src available" and skip the check (identical to the prod no-op path).
@@ -225,6 +244,24 @@ export function computeSrcRegistryStamp(packageEntry: string): string | null {
}
}
/**
* Recursively enumerate every `*.ts` under `dir`, EXCLUDING `*.generated.ts`.
* Mirror of the codegen's `collectStampFiles` (packages/mcp/scripts/
* gen-registry-stamp.mjs) — keep the two walk/filter rules identical.
*/
function collectStampFiles(dir: string): string[] {
const out: string[] = [];
for (const entry of readdirSync(dir)) {
const full = join(dir, entry);
if (statSync(full).isDirectory()) {
out.push(...collectStampFiles(full));
} else if (entry.endsWith('.ts') && !entry.endsWith('.generated.ts')) {
out.push(full);
}
}
return out;
}
// TS with module:commonjs downlevels a literal `import()` to `require()`, which
// cannot load the ESM-only `@docmost/mcp` package. Indirect through Function so
// the real dynamic `import()` survives compilation and can load ESM from
@@ -7,6 +7,22 @@ import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { jsonToMarkdown } from '../../../collaboration/collaboration.util';
import { modelFriendlyInput } from './model-friendly-input';
/**
* A tool error whose message is DELIBERATELY safe to expose to an anonymous
* share reader (and to the model, for self-correction). Every OTHER thrown error
* is treated as internal and replaced with a generic string by `wrapToolErrors`,
* so a raw exception message — an internal page title, a DB/stack fragment, a
* driver detail — never rides the public UI stream (#394).
*/
export class ShareToolError extends Error {}
// The only two classified strings an anonymous reader may ever see from a tool
// failure. The specific one keeps the model's self-correction useful ("try a
// different page"); the generic one reveals nothing about the internal fault.
const SHARE_TOOL_ERROR_NOT_AVAILABLE =
'The requested page is not available in this share.';
const SHARE_TOOL_ERROR_GENERIC = 'The tool could not complete the request.';
/**
* Isolated, READ-ONLY toolset for the ANONYMOUS public-share assistant.
*
@@ -44,7 +60,7 @@ export class PublicShareChatToolsService {
* are NO write tools, NO comments/history, NO cross-space or external tools.
*/
forShare(shareId: string, workspaceId: string): Record<string, Tool> {
return {
return this.wrapToolErrors({
searchSharePages: tool({
description:
'Search the pages of THIS published documentation share for a ' +
@@ -96,7 +112,7 @@ export class PublicShareChatToolsService {
execute: async ({ pageId }) => {
const id = (pageId ?? '').trim();
if (!id) {
throw new Error('A pageId is required.');
throw new ShareToolError('A pageId is required.');
}
// Resolve via the SINGLE canonical share-access boundary: confirms the
// page resolves to THIS share (recursive CTE up the tree, honouring
@@ -112,7 +128,7 @@ export class PublicShareChatToolsService {
workspaceId,
);
if (!resolved) {
throw new Error('That page is not part of this published share.');
throw new ShareToolError(SHARE_TOOL_ERROR_NOT_AVAILABLE);
}
const { page } = resolved;
@@ -193,6 +209,57 @@ export class PublicShareChatToolsService {
}
},
}),
};
});
}
/**
* Wrap every tool's `execute` so a THROWN error is sanitized in ONE place —
* closing the byte leak, the render, and the model context at once (#394).
*
* The AI SDK surfaces a tool-execution throw as an atomic `tool-output-error`
* frame on the v6 UI stream whose `errorText` is the thrown message; on the
* public share that frame goes straight to an anonymous reader. Unwrapped, a
* raw exception (an internal page title, a DB/stack fragment, a driver detail)
* would ride that frame verbatim. Here we catch it, LOG the full detail
* server-side only, and re-throw a CLASSIFIED, safe error: the tool's own
* intentional ShareToolError messages pass through (they keep the model's
* self-correction useful), everything else collapses to a generic string.
*/
private wrapToolErrors(
tools: Record<string, Tool>,
): Record<string, Tool> {
const wrapped: Record<string, Tool> = {};
for (const [name, t] of Object.entries(tools)) {
const original = t.execute;
if (typeof original !== 'function') {
wrapped[name] = t;
continue;
}
wrapped[name] = {
...t,
execute: async (args: unknown, options: unknown) => {
try {
return await (
original as (a: unknown, o: unknown) => Promise<unknown>
)(args, options);
} catch (err) {
const safe =
err instanceof ShareToolError
? err.message
: SHARE_TOOL_ERROR_GENERIC;
// Full detail to the server log ONLY — never to the anon.
this.logger.warn(
`Public share tool "${name}" failed: ${
err instanceof Error ? err.message : String(err)
}`,
);
// This safe string is ALL that rides the tool-output-error frame,
// becomes model context, and could be rendered — one choke point.
throw new ShareToolError(safe);
}
},
} as Tool;
}
return wrapped;
}
}
@@ -120,3 +120,102 @@ describe('JwtStrategy — provenance derivation', () => {
expect(req.raw.actor).toBeUndefined();
});
});
/**
* Provenance derivation on the API-KEY path (jwt.strategy.validateApiKey, #486).
*
* The access-token path stamped provenance; the API-key path returned early
* WITHOUT it, so an is_agent API key's REST writes recorded no 'agent' marker.
* The API-key payload carries no signed claim, so provenance is resolved from the
* SERVER-SIDE user returned by ApiKeyService.validateApiKey: isAgent -> 'agent',
* otherwise 'user'; aiChatId is always null (an API key has no ai_chats row).
*
* The enterprise ApiKeyService is not bundled in the OSS build, so the strategy
* loads it through an overridable `resolveApiKeyService` seam that we stub here.
*/
describe('JwtStrategy — API-key provenance derivation (#486)', () => {
function makeApiKeyStrategy(validateApiKeyImpl: (p: any) => Promise<any>) {
const userRepo: any = { findById: jest.fn() };
const workspaceRepo: any = { findById: jest.fn() };
const userSessionRepo: any = { findActiveById: jest.fn() };
const sessionActivityService: any = { trackActivity: jest.fn() };
const environmentService: any = { getAppSecret: () => 'test-secret' };
const moduleRef: any = {};
const strategy = new JwtStrategy(
userRepo,
workspaceRepo,
userSessionRepo,
sessionActivityService,
environmentService,
moduleRef,
);
// Stub the EE ApiKeyService seam (the real module is not in the OSS build).
const validateApiKey = jest.fn(validateApiKeyImpl);
jest
.spyOn(strategy as any, 'resolveApiKeyService')
.mockReturnValue({ validateApiKey });
return { strategy, validateApiKey };
}
const makeReq = () => ({ raw: {} as Record<string, any> });
const apiKeyPayload = () => ({
sub: 'svc-1',
workspaceId: 'ws-1',
apiKeyId: 'key-1',
type: JwtType.API_KEY,
});
it("stamps actor='agent' for an is_agent API key (from the validated user)", async () => {
const validated = {
user: { id: 'svc-1', isAgent: true },
workspace: { id: 'ws-1' },
};
const { strategy, validateApiKey } = makeApiKeyStrategy(
async () => validated,
);
const req = makeReq();
const result = await strategy.validate(req, apiKeyPayload() as any);
expect(validateApiKey).toHaveBeenCalledTimes(1);
expect(req.raw.actor).toBe('agent');
// API keys carry no internal ai_chats row -> null.
expect(req.raw.aiChatId).toBeNull();
// The validated auth object is returned unchanged (req.user shape preserved).
expect(result).toBe(validated);
});
it("stamps actor='user' for an ordinary (non-agent) API key", async () => {
const { strategy } = makeApiKeyStrategy(async () => ({
user: { id: 'u-1', isAgent: false },
workspace: { id: 'ws-1' },
}));
const req = makeReq();
await strategy.validate(req, apiKeyPayload() as any);
expect(req.raw.actor).toBe('user');
expect(req.raw.aiChatId).toBeNull();
});
it('throws Unauthorized (and stamps nothing) when the EE module is missing', async () => {
const userRepo: any = { findById: jest.fn() };
const strategy = new JwtStrategy(
userRepo,
{ findById: jest.fn() } as any,
{ findActiveById: jest.fn() } as any,
{ trackActivity: jest.fn() } as any,
{ getAppSecret: () => 'test-secret' } as any,
{} as any,
);
// EE not bundled: the seam returns null.
jest.spyOn(strategy as any, 'resolveApiKeyService').mockReturnValue(null);
const req = makeReq();
await expect(
strategy.validate(req, apiKeyPayload() as any),
).rejects.toThrow(UnauthorizedException);
expect(req.raw.actor).toBeUndefined();
});
});
@@ -102,28 +102,49 @@ export class JwtStrategy extends PassportStrategy(Strategy, 'jwt') {
}
private async validateApiKey(req: any, payload: JwtApiKeyPayload) {
let ApiKeyModule: any;
let isApiKeyModuleReady = false;
const apiKeyService = this.resolveApiKeyService();
if (!apiKeyService) {
throw new UnauthorizedException('Enterprise API Key module missing');
}
const result = await apiKeyService.validateApiKey(payload);
// Stamp the agent-edit provenance for the API-KEY path too (#486). Unlike the
// access-token path above, it CANNOT be resolved before this point: the
// API-key payload carries no signed actor/aiChatId claim, and the user (with
// its isAgent flag) is unknown until the key is validated. Claim semantics for
// API keys: an is_agent API key (an agent service account) stamps 'agent' on
// every REST write; an ordinary API key resolves to 'user'. An API key has no
// internal ai_chats row, so aiChatId is always null. Derived from the
// SERVER-SIDE user (never a client field), so an 'agent' badge is unspoofable
// — mirroring the access-token path. Passing `null` for the claim means the
// actor is decided solely by user.isAgent.
const provenance = resolveProvenance((result as any)?.user, null);
req.raw.actor = provenance.actor;
req.raw.aiChatId = provenance.aiChatId;
return result;
}
/**
* Resolve the enterprise ApiKeyService, or `null` when the EE module is not
* bundled in this build (community build). Extracted as an overridable seam so
* the API-key provenance stamping can be unit-tested without the EE package
* present (docmost is OSS + a separate EE bundle; `require` of the EE path
* throws here). Any load/resolve error is treated as "module missing".
*/
protected resolveApiKeyService(): {
validateApiKey: (payload: JwtApiKeyPayload) => Promise<unknown>;
} | null {
try {
// eslint-disable-next-line @typescript-eslint/no-require-imports
ApiKeyModule = require('./../../../ee/api-key/api-key.service');
isApiKeyModuleReady = true;
const ApiKeyModule = require('./../../../ee/api-key/api-key.service');
return this.moduleRef.get(ApiKeyModule.ApiKeyService, { strict: false });
} catch (err) {
this.logger.debug(
'API Key module requested but enterprise module not bundled in this build',
);
isApiKeyModuleReady = false;
return null;
}
if (isApiKeyModuleReady) {
const ApiKeyService = this.moduleRef.get(ApiKeyModule.ApiKeyService, {
strict: false,
});
return ApiKeyService.validateApiKey(payload);
}
throw new UnauthorizedException('Enterprise API Key module missing');
}
}
@@ -0,0 +1,133 @@
import { readFileSync } from 'fs';
import { EventEmitter } from 'node:events';
import { streamText } from 'ai';
import { MockLanguageModelV3, simulateReadableStream } from 'ai/test';
/**
* Regression tests for the writeToServerResponse drain-hang fix in
* patches/ai@6.0.134.patch (#486, commit 6).
*
* Unpatched ai@6.0.134's writeToServerResponse awaits ONLY `once("drain")` when
* response.write() returns false (backpressure). If the client disconnects
* mid-write the socket never drains, so that await never resolves: the read loop
* parks FOREVER, its `finally { response.end() }` is unreachable, and the stream
* reader + buffered chunks are pinned until process restart. In autonomous mode
* the run keeps producing output after the disconnect, so EVERY mid-run
* disconnect leaks a hung pipe. The patch races drain against close/error, and on
* a terminal socket event cancels the reader and breaks so `finally` always runs.
*
* This drives the REAL patched writeToServerResponse through the public
* pipeUIMessageStreamToResponse API with a response that never drains and closes
* mid-write — exactly the leak scenario.
*/
/** A ServerResponse-like emitter whose first write() stalls (returns false) and
* then "closes" like a disconnecting client — never firing 'drain'. */
class DisconnectingResponse extends EventEmitter {
ended = false;
writeCount = 0;
statusCode = 200;
writableEnded = false;
destroyed = false;
writeHead(): this {
return this;
}
setHeader(): void {}
flushHeaders(): void {}
write(): boolean {
this.writeCount++;
if (this.writeCount === 1) {
// Simulate the client vanishing mid-write: backpressure (false) and then a
// 'close' on the next tick, and CRUCIALLY never a 'drain'. Unpatched, the
// loop would await drain forever here.
setImmediate(() => this.emit('close'));
return false;
}
return true;
}
end(): void {
this.ended = true;
this.writableEnded = true;
this.emit('finish');
}
}
function makeModel() {
return new MockLanguageModelV3({
doStream: async () => ({
stream: simulateReadableStream({
chunks: [
{ type: 'stream-start' as const, warnings: [] },
{ type: 'text-start' as const, id: '1' },
{ type: 'text-delta' as const, id: '1', delta: 'hello ' },
{ type: 'text-delta' as const, id: '1', delta: 'world' },
{ type: 'text-end' as const, id: '1' },
{
type: 'finish' as const,
finishReason: { unified: 'stop' as const, raw: 'stop' },
usage: {
inputTokens: { total: 1, noCache: undefined, cacheRead: undefined, cacheWrite: undefined },
outputTokens: { total: 1, text: 1, reasoning: undefined },
},
},
],
}),
}),
});
}
describe('ai@6.0.134 pnpm patch: writeToServerResponse drain-hang (#486)', () => {
it('ends the response (does NOT hang) when the socket closes mid-write without draining', async () => {
const result = streamText({ model: makeModel(), prompt: 'hi' });
const res = new DisconnectingResponse();
// Drain the SDK stream independently, like the production detached path.
void result.consumeStream({ onError: () => undefined });
result.pipeUIMessageStreamToResponse(res as never);
// TRIPWIRE: the patched loop exits on 'close' and runs finally -> end().
// Unpatched, it awaits 'drain' forever and this never becomes true.
await new Promise<void>((resolve, reject) => {
const started = Date.now();
const poll = setInterval(() => {
if (res.ended) {
clearInterval(poll);
resolve();
} else if (Date.now() - started > 3000) {
clearInterval(poll);
reject(new Error('writeToServerResponse hung: response never ended'));
}
}, 20);
});
expect(res.ended).toBe(true);
});
it('does not emit an unhandledRejection when the fire-and-forget read() throws', async () => {
// The patch swallows read()'s rejection (fire-and-forget) with a log instead
// of letting it surface as a process-killing unhandledRejection.
const rejections: unknown[] = [];
const onUnhandled = (e: unknown) => rejections.push(e);
process.on('unhandledRejection', onUnhandled);
// Silence the patch's diagnostic console.error for the throwing read().
const errSpy = jest.spyOn(console, 'error').mockImplementation(() => undefined);
try {
const result = streamText({ model: makeModel(), prompt: 'hi' });
const res = new DisconnectingResponse();
void result.consumeStream({ onError: () => undefined });
result.pipeUIMessageStreamToResponse(res as never);
await new Promise((r) => setTimeout(r, 300));
} finally {
process.off('unhandledRejection', onUnhandled);
errSpy.mockRestore();
}
expect(rejections).toEqual([]);
});
it('both installed dist builds (CJS and ESM) carry the #486 patch marker', () => {
const cjsPath = require.resolve('ai');
const mjsPath = cjsPath.replace(/index\.js$/, 'index.mjs');
expect(cjsPath).toMatch(/index\.js$/);
expect(readFileSync(cjsPath, 'utf8')).toContain('PATCH(docmost #486)');
expect(readFileSync(mjsPath, 'utf8')).toContain('PATCH(docmost #486)');
});
});
@@ -0,0 +1,95 @@
import type { HealthIndicatorService } from '@nestjs/terminus';
import type { EnvironmentService } from '../environment/environment.service';
/**
* Integration guard for the /health Redis-probe handle leak (#486, commit 2).
*
* The bug: `pingCheck` built `new Redis(...)` per call and only disconnected on
* the SUCCESS path, so when Redis is DOWN every probe tick added ANOTHER
* forever-reconnecting client — an unbounded handle/client leak for the duration
* of the outage. The fix reuses ONE long-lived probe client.
*
* This is an OBSERVABLE-property test, not an assertion on a mocked return value:
* we point the indicator at a REAL, refused TCP endpoint (a dead port) so ioredis
* genuinely fails to connect, run many probes, and assert the number of live
* Redis CLIENTS created stays at exactly ONE. `ioredis` is delegated to its real
* implementation (requireActual) — only the constructor is wrapped to COUNT the
* real clients it creates, which is precisely the leaking resource.
*/
const mockLiveClients: Array<{ status: string; disconnect: () => void }> = [];
jest.mock('ioredis', () => {
const actual = jest.requireActual('ioredis');
const RealRedis = actual.Redis ?? actual.default ?? actual;
class CountingRedis extends RealRedis {
constructor(...args: unknown[]) {
super(...(args as []));
mockLiveClients.push(this as never);
}
}
return { ...actual, Redis: CountingRedis, default: CountingRedis };
});
// Import AFTER the mock is registered so the class picks up the counting client.
import { RedisHealthIndicator } from './redis.health';
describe('RedisHealthIndicator handle leak (#486)', () => {
const indicatorService = {
check: (key: string) => ({
up: () => ({ [key]: { status: 'up' } }),
down: (message: string) => ({ [key]: { status: 'down', message } }),
}),
} as unknown as HealthIndicatorService;
// A port with (almost certainly) nothing listening -> connection refused fast.
const environmentService = {
getRedisUrl: () => 'redis://127.0.0.1:6399/0',
} as unknown as EnvironmentService;
let indicator: RedisHealthIndicator;
beforeEach(() => {
mockLiveClients.length = 0;
indicator = new RedisHealthIndicator(indicatorService, environmentService);
});
afterEach(() => {
indicator.onModuleDestroy();
// Belt-and-braces: tear down anything the test created so ioredis reconnect
// timers do not keep the jest worker alive.
for (const c of mockLiveClients) {
try {
c.disconnect();
} catch {
/* already gone */
}
}
});
it('creates exactly ONE Redis client across many probes while Redis is DOWN', async () => {
const N = 8;
for (let i = 0; i < N; i++) {
const result = await indicator.pingCheck('redis');
// Down endpoint -> every probe reports "down" (not an unhandled crash).
expect(result.redis.status).toBe('down');
}
// THE OBSERVABLE LEAK: on the buggy code this is N (a fresh, never-cleaned
// reconnecting client per probe). The fix reuses one shared client.
expect(mockLiveClients).toHaveLength(1);
});
it('onModuleDestroy releases the probe client (a later probe builds a fresh one)', async () => {
await indicator.pingCheck('redis');
expect(mockLiveClients).toHaveLength(1);
indicator.onModuleDestroy();
// A second destroy is a safe no-op (probeClient was nulled).
indicator.onModuleDestroy();
// After shutdown the indicator lazily builds a NEW client on the next probe,
// proving the old one was truly released rather than reused.
await indicator.pingCheck('redis');
expect(mockLiveClients).toHaveLength(2);
});
});
@@ -2,33 +2,78 @@ import {
HealthIndicatorResult,
HealthIndicatorService,
} from '@nestjs/terminus';
import { Injectable, Logger } from '@nestjs/common';
import { Injectable, Logger, OnModuleDestroy } from '@nestjs/common';
import { EnvironmentService } from '../environment/environment.service';
import { Redis } from 'ioredis';
@Injectable()
export class RedisHealthIndicator {
export class RedisHealthIndicator implements OnModuleDestroy {
private readonly logger = new Logger(RedisHealthIndicator.name);
/**
* ONE long-lived probe connection, reused across every /health tick. The old
* code built `new Redis(...)` per call and only `disconnect()`d on the SUCCESS
* path, so while Redis was DOWN every probe added a fresh, forever-reconnecting
* client — a handle leak that grew without bound for as long as the outage (and
* the health checker keeps polling) lasted. A single shared client keeps at most
* ONE background reconnect loop regardless of how many probes run.
*/
private probeClient: Redis | null = null;
constructor(
private readonly healthIndicatorService: HealthIndicatorService,
private environmentService: EnvironmentService,
) {}
private getProbeClient(): Redis {
if (!this.probeClient) {
this.probeClient = new Redis(this.environmentService.getRedisUrl(), {
// Constructing must never throw or eagerly connect; the first ping opens
// the socket. This lets us build the client once and reuse it.
lazyConnect: true,
// A health probe must fail FAST, not queue behind a stuck reconnect: one
// retry per request, and no offline queue so a ping while disconnected
// rejects immediately instead of buffering commands that pile up in RAM.
maxRetriesPerRequest: 1,
enableOfflineQueue: false,
});
// ioredis emits 'error' on every failed (re)connect; with no listener that
// surfaces as an unhandled 'error' event and can crash the process. Swallow
// it here — pingCheck already reports health — and log at debug so a Redis
// outage does not flood the logs.
this.probeClient.on('error', (err) => {
this.logger.debug(
`Redis probe connection error: ${
err instanceof Error ? err.message : String(err)
}`,
);
});
}
return this.probeClient;
}
async pingCheck(key: string): Promise<HealthIndicatorResult> {
const indicator = this.healthIndicatorService.check(key);
try {
const redis = new Redis(this.environmentService.getRedisUrl(), {
maxRetriesPerRequest: 15,
});
const redis = this.getProbeClient();
await redis.ping();
redis.disconnect();
return indicator.up();
} catch (e) {
this.logger.error(e);
return indicator.down(`${key} is not available`);
}
}
onModuleDestroy(): void {
if (this.probeClient) {
// disconnect() (not quit()) tears the socket + reconnect loop down
// immediately without waiting on a round-trip to a possibly-down server.
// Do NOT removeAllListeners() with no event name — that would also strip
// ioredis' OWN internal listeners and break its teardown; our 'error'
// listener is harmless and dies with the dropped client reference.
this.probeClient.disconnect();
this.probeClient = null;
}
}
}
@@ -16,6 +16,7 @@ import {
} from './mcp-auth.helpers';
import { JwtType } from '../../core/auth/dto/jwt-payload';
import { CREDENTIALS_MISMATCH_MESSAGE } from '../../core/auth/auth.constants';
import { McpService } from './mcp.service';
// The /mcp per-user auth decision logic is tested through the framework-free
// `resolveMcpSessionConfig` helper that McpService delegates to. McpService
@@ -1179,3 +1180,46 @@ describe('mapAuthResultToResponse (handle status/body mapping, refactor R2)', ()
});
});
});
// #486: onModuleDestroy must ALSO tear down the live loopback CollabSessions, not
// just clear the sweep timer — otherwise the embedded MCP's collab sockets keep
// docs pinned open on the collab server past process exit. The teardown goes
// through an overridable seam (destroyAllMcpSessions) so it can be spied without
// loading the ESM-only @docmost/mcp package.
describe('McpService.onModuleDestroy — CollabSession teardown (#486)', () => {
function makeService(): McpService {
// The constructor only stores its deps and starts the (unref'd) sweep timer,
// so bare stubs suffice. onModuleDestroy clears that timer, so no leak.
return new McpService(
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
{} as any,
);
}
it('destroys all sessions AND clears the sweep timer on shutdown', async () => {
const svc = makeService();
const destroy = jest.fn().mockResolvedValue(undefined);
(svc as any).destroyAllMcpSessions = destroy;
const clearSpy = jest.spyOn(global, 'clearInterval');
await svc.onModuleDestroy();
expect(destroy).toHaveBeenCalledTimes(1);
expect(clearSpy).toHaveBeenCalledWith((svc as any).sweepTimer);
clearSpy.mockRestore();
});
it('swallows a teardown failure so shutdown never throws', async () => {
const svc = makeService();
(svc as any).destroyAllMcpSessions = jest
.fn()
.mockRejectedValue(new Error('collab teardown boom'));
await expect(svc.onModuleDestroy()).resolves.toBeUndefined();
});
});
@@ -117,10 +117,42 @@ export class McpService implements OnModuleDestroy {
this.sweepTimer.unref?.();
}
onModuleDestroy(): void {
async onModuleDestroy(): Promise<void> {
clearInterval(this.sweepTimer);
// Tear down any live loopback CollabSession providers at shutdown (#486). The
// embedded MCP (and the in-app AI agent) open Hocuspocus collab sockets against
// THIS process; without an explicit teardown those sessions keep their docs
// "open" on the collab server and hold providers/buffers until they idle out,
// so a restart can race a doc still pinned by the dying worker. Best-effort:
// any failure is logged, never allowed to break shutdown.
try {
await this.destroyAllMcpSessions();
} catch (err) {
this.logger.error(
'MCP CollabSession teardown on shutdown failed',
err as Error,
);
}
}
/**
* Resolve @docmost/mcp's `destroyAllSessions` and invoke it (#486). The live
* CollabSession registry is a module-level singleton in the ESM package, shared
* by every entry (`.`/`./http`), so this tears down ALL sessions regardless of
* which surface opened them. The module is already loaded whenever MCP was used;
* if it was never loaded (or is absent) the import + no-op is harmless.
*
* Held as an overridable field so a unit test can spy the teardown without
* loading the ESM-only package or standing up the DI graph.
*/
private destroyAllMcpSessions: () => Promise<void> = async () => {
const entry = require.resolve('@docmost/mcp');
const mod = (await esmImport(pathToFileURL(entry).href)) as {
destroyAllSessions?: () => void;
};
mod.destroyAllSessions?.();
};
// Service account the embedded MCP uses to talk back to this Docmost
// instance over loopback REST + the collaboration WebSocket. Now OPTIONAL:
// it is only a fallback when no per-user Basic/Bearer credentials are sent.
@@ -0,0 +1,148 @@
import { get as httpGet } from 'node:http';
import { AddressInfo } from 'node:net';
import { createServer } from 'node:http';
// Drive the metrics HTTP server without the load-time METRICS_PORT gate: mock the
// registry so isMetricsEnabled()/getMetricsRegistry() are always satisfied. What
// we assert is observed over a REAL socket (bind address, status codes), not on
// the mock.
jest.mock('./metrics.registry', () => ({
isMetricsEnabled: () => true,
getMetricsRegistry: () => ({
metrics: async () => '# HELP up test\nup 1\n',
contentType: 'text/plain; version=0.0.4',
}),
}));
import {
startMetricsServer,
closeMetricsServer,
resolveMetricsBind,
resolveMetricsToken,
} from './metrics.server';
/** Find a free TCP port (the metrics server requires METRICS_PORT > 0). */
function freePort(): Promise<number> {
return new Promise((resolve, reject) => {
const s = createServer();
s.once('error', reject);
s.listen(0, '127.0.0.1', () => {
const p = (s.address() as AddressInfo).port;
s.close(() => resolve(p));
});
});
}
/** Minimal GET against 127.0.0.1:port with optional Authorization header. */
function req(
port: number,
headers: Record<string, string> = {},
): Promise<{ status: number; body: string }> {
return new Promise((resolve, reject) => {
const r = httpGet(
{ host: '127.0.0.1', port, path: '/metrics', headers },
(res) => {
let body = '';
res.on('data', (c) => (body += c));
res.on('end', () =>
resolve({ status: res.statusCode ?? 0, body }),
);
},
);
r.on('error', reject);
});
}
describe('metrics server bind + auth (#486)', () => {
const saved = {
bind: process.env.METRICS_BIND,
token: process.env.METRICS_TOKEN,
port: process.env.METRICS_PORT,
};
afterEach(async () => {
await closeMetricsServer();
process.env.METRICS_BIND = saved.bind;
process.env.METRICS_TOKEN = saved.token;
process.env.METRICS_PORT = saved.port;
delete process.env.METRICS_BIND;
delete process.env.METRICS_TOKEN;
});
describe('resolveMetricsBind', () => {
it('defaults to loopback 127.0.0.1', () => {
delete process.env.METRICS_BIND;
expect(resolveMetricsBind()).toBe('127.0.0.1');
});
it('honours the METRICS_BIND override', () => {
process.env.METRICS_BIND = '0.0.0.0';
expect(resolveMetricsBind()).toBe('0.0.0.0');
});
it('treats a blank override as unset (loopback)', () => {
process.env.METRICS_BIND = ' ';
expect(resolveMetricsBind()).toBe('127.0.0.1');
});
});
describe('resolveMetricsToken', () => {
it('is null when unset', () => {
delete process.env.METRICS_TOKEN;
expect(resolveMetricsToken()).toBeNull();
});
it('returns the trimmed token when set', () => {
process.env.METRICS_TOKEN = ' s3cret ';
expect(resolveMetricsToken()).toBe('s3cret');
});
});
it('binds to loopback by default and serves /metrics without auth when no token', async () => {
delete process.env.METRICS_BIND;
delete process.env.METRICS_TOKEN;
const port = await freePort();
process.env.METRICS_PORT = String(port);
const server = startMetricsServer();
expect(server).not.toBeNull();
await new Promise<void>((resolve) => {
if (server!.listening) resolve();
else server!.once('listening', () => resolve());
});
// OBSERVABLE: the listener bound to loopback, not 0.0.0.0.
expect((server!.address() as AddressInfo).address).toBe('127.0.0.1');
const res = await req(port);
expect(res.status).toBe(200);
expect(res.body).toContain('up 1');
});
it('rejects unauthenticated scrapes with 401 and accepts the exact Bearer token', async () => {
delete process.env.METRICS_BIND;
process.env.METRICS_TOKEN = 'topsecret';
const port = await freePort();
process.env.METRICS_PORT = String(port);
const server = startMetricsServer();
expect(server).not.toBeNull();
// No auth -> 401.
const noAuth = await req(port);
expect(noAuth.status).toBe(401);
// Wrong token, DIFFERENT length -> 401 (short-circuits on the length guard).
const wrong = await req(port, { authorization: 'Bearer nope' });
expect(wrong.status).toBe(401);
// Wrong token, SAME length -> 401. This drives the timingSafeEqual compare
// itself (the length guard passes: 'Bearer topsecreX' has the same length as
// 'Bearer topsecret'). Pins the constant-time compare: a regression that made
// it return true would let this equal-length wrong token through — the
// different-length case above would NOT catch that.
const sameLen = await req(port, { authorization: 'Bearer topsecreX' });
expect(sameLen.status).toBe(401);
// Correct token -> 200 with the metrics body.
const ok = await req(port, { authorization: 'Bearer topsecret' });
expect(ok.status).toBe(200);
expect(ok.body).toContain('up 1');
});
});
@@ -1,7 +1,27 @@
import { createServer, Server } from 'node:http';
import { timingSafeEqual } from 'node:crypto';
import { Logger } from '@nestjs/common';
import { getMetricsRegistry, isMetricsEnabled } from './metrics.registry';
/**
* Constant-time compare of the presented Authorization header against the
* expected `Bearer <token>`. This is the ONLY auth layer for the metrics
* endpoint, so a naive `!==` would leak the token byte-by-byte via timing.
* timingSafeEqual requires equal-length buffers, so a length mismatch short-
* circuits to "not equal" (its own length is not itself a useful oracle: the
* expected string length is fixed by config, not secret-derived).
*/
function bearerMatches(
presented: string | undefined,
expected: string,
): boolean {
if (typeof presented !== 'string') return false;
const a = Buffer.from(presented);
const b = Buffer.from(expected);
if (a.length !== b.length) return false;
return timingSafeEqual(a, b);
}
/**
* Start the Prometheus scrape endpoint on a SEPARATE port, taken from
* `METRICS_PORT`. There is NO default port: when `METRICS_PORT` is unset the
@@ -16,6 +36,30 @@ import { getMetricsRegistry, isMetricsEnabled } from './metrics.registry';
*/
let metricsServer: Server | null = null;
/**
* Interface the metrics endpoint binds to. Defaults to LOOPBACK (127.0.0.1) so
* the unauthenticated `/metrics` surface is NOT exposed on all interfaces by
* default — the old `0.0.0.0` bind put an auth-less endpoint on every interface.
* Deployments where the scraper runs in a SEPARATE container (and reaches this as
* `docmost:9464`) set `METRICS_BIND=0.0.0.0`, ideally together with METRICS_TOKEN
* and/or a private network so the port is not world-readable.
*/
export function resolveMetricsBind(): string {
const raw = (process.env.METRICS_BIND ?? '').trim();
return raw.length > 0 ? raw : '127.0.0.1';
}
/**
* Optional Bearer token guarding `/metrics`. When `METRICS_TOKEN` is set, every
* scrape must present `Authorization: Bearer <token>`; unset (default) leaves the
* endpoint open (safe when bound to loopback / a trusted network). Returns the
* trimmed token or null when unset/blank.
*/
export function resolveMetricsToken(): string | null {
const raw = (process.env.METRICS_TOKEN ?? '').trim();
return raw.length > 0 ? raw : null;
}
export function startMetricsServer(): Server | null {
if (!isMetricsEnabled()) return null;
@@ -31,8 +75,22 @@ export function startMetricsServer(): Server | null {
return null;
}
const bind = resolveMetricsBind();
const token = resolveMetricsToken();
const server = createServer(async (req, res) => {
if (req.method === 'GET' && req.url === '/metrics') {
// Optional Bearer auth: reject scrapes without the exact token when one is
// configured. This is the auth layer the old all-interfaces bind lacked.
if (token) {
const auth = req.headers['authorization'];
if (!bearerMatches(auth, `Bearer ${token}`)) {
res.statusCode = 401;
res.setHeader('WWW-Authenticate', 'Bearer');
res.end();
return;
}
}
try {
const body = await register.metrics();
res.setHeader('Content-Type', register.contentType);
@@ -48,10 +106,14 @@ export function startMetricsServer(): Server | null {
res.end();
});
// Bind on all interfaces: the scraper (VictoriaMetrics) reaches this from
// another container as docmost:9464. The port is not published to the host.
server.listen(port, '0.0.0.0', () => {
logger.log(`Metrics endpoint listening on :${port}/metrics`);
// Bind to loopback by default so the auth-less endpoint is not exposed on all
// interfaces. Set METRICS_BIND=0.0.0.0 (ideally with METRICS_TOKEN) when the
// scraper runs in a separate container and reaches this as docmost:9464.
server.listen(port, bind, () => {
logger.log(
`Metrics endpoint listening on ${bind}:${port}/metrics` +
(token ? ' (Bearer auth required)' : ''),
);
});
server.on('error', (err) => {
@@ -31,9 +31,6 @@ export enum QueueJob {
IMPORT_TASK = 'import-task',
EXPORT_TASK = 'export-task',
SEARCH_REMOVE_PAGE = 'search-remove-page',
SEARCH_REMOVE_ASSET = 'search-remove-attachment',
SEARCH_REMOVE_FACE = 'search-remove-comment',
TYPESENSE_FLUSH = 'typesense-flush',
PAGE_CREATED = 'page-created',
+22 -2
View File
@@ -293,8 +293,28 @@ so capable clients steer the model automatically.
the debounced REST snapshot), then **reads → transforms → writes synchronously** in one
tick so no remote update can interleave, and **waits for persistence acknowledgement**
before returning.
- **Per-page write serialization.** A per-`pageId` async mutex ensures two MCP writes to
the same page never overlap; different pages never block each other.
- **Per-page write serialization.** A per-`pageId` async mutex (keyed by the resolved
page **UUID**, never a slugId) ensures two MCP writes to the same page never overlap;
different pages never block each other. The lock helper fails fast if it is ever handed
a non-UUID key, so a write path that forgot to resolve the id can never silently lock
under a split key.
**Deploy requirement — single instance or sticky sessions.** This mutex is an
in-process `Map`, and the cached collab sessions and the `stash_page` blob store are
RAM-only and process-local. Behind a **multi-replica** load balancer **without sticky
sessions**, two replicas can each "hold" the lock for the same page at once and per-page
serialization is silently lost. Run the MCP/app as a **single instance**, or pin each
page's traffic to one replica (sticky sessions / consistent hashing on the page id).
There is deliberately no cross-process (e.g. Postgres advisory) lock yet — a conscious
documented constraint. See the `Dockerfile` comment and the `MCP collaboration write
path` block in `.env.example`.
**Rights-staleness window.** A cached collab session writes under the token captured at
connect time (and the collab-token cache reuses a token for its TTL), so a **revoked**
page access can lag by up to `MCP_COLLAB_SESSION_MAX_AGE_MS` (the hard session lifetime,
default 10 min) before the next re-auth picks it up. Lower it to shorten the lag at the
cost of more reconnects. This bounded window is an accepted trade-off; there is no
push-based cache invalidation on a rights change.
- **Transparent re-authentication.** Login uses email/password; expired tokens are
refreshed automatically on the first 401/403 (covering JSON, multipart upload, and the
collaboration-token path), with in-flight login de-duplication so a burst of calls
+21 -3
View File
@@ -302,9 +302,27 @@ Docmost-MCP не сочетают:
правки, которых ещё нет в дебаунс-снапшоте REST), затем **читает → трансформирует →
пишет синхронно** в одном тике, чтобы никакое удалённое обновление не вклинилось, и
**ждёт подтверждения сохранения** до возврата.
- **Сериализация записи по странице.** Асинхронный мьютекс по `pageId` гарантирует, что
две записи MCP в одну страницу никогда не пересекаются; разные страницы друг друга не
блокируют.
- **Сериализация записи по странице.** Асинхронный мьютекс по разрешённому **UUID**
страницы (никогда не по slugId) гарантирует, что две записи MCP в одну страницу никогда
не пересекаются; разные страницы друг друга не блокируют. Хелпер блокировки падает сразу
(fail-fast), если ему передали не-UUID ключ, — путь записи, забывший разрезолвить id, не
сможет молча взять лок под расщеплённым ключом.
**Требование к деплою — один инстанс или sticky-сессии.** Этот мьютекс — процесс-локальный
`Map`, а кэш collab-сессий и хранилище `stashPage` живут только в RAM одного процесса. За
**мультиреплика**-балансировщиком **без sticky-сессий** две реплики могут одновременно
«держать» лок одной страницы, и сериализация по странице молча теряется. Запускайте
MCP/приложение **одним инстансом** либо прибивайте трафик страницы к одной реплике
(sticky-сессии / consistent hashing по id страницы). Кросс-процессной блокировки (например,
Postgres advisory-lock) намеренно пока нет — осознанное задокументированное ограничение.
См. комментарий в `Dockerfile` и блок `MCP collaboration write path` в `.env.example`.
**Окно устаревших прав.** Кэшированная collab-сессия пишет под токеном, захваченным в
момент connect (а кэш collab-токена переиспользует токен в пределах своего TTL), поэтому
**отозванный** доступ к странице может лагать до `MCP_COLLAB_SESSION_MAX_AGE_MS` (жёсткий
срок жизни сессии, по умолчанию 10 мин), пока следующая переавторизация его не подхватит.
Уменьшите значение, чтобы сократить лаг ценой большего числа переподключений. Это
ограниченное окно — принятый trade-off; push-инвалидации кэша при смене прав нет.
- **Прозрачная переавторизация.** Логин по email/паролю; истёкшие токены обновляются
автоматически на первом 401/403 (покрывая JSON, multipart-загрузку и путь токена
коллаборации), с дедупликацией параллельных логинов, так что пачка вызовов вызывает один
+77 -37
View File
@@ -1,60 +1,100 @@
// Codegen: emit src/registry-stamp.generated.ts with a REGISTRY_STAMP hash of
// the tool-specs REGISTRY CONTENT, so a build/ vs src/ skew (issue #447) is
// detectable at runtime.
// the ENTIRE src/ tree, so a build/ vs src/ skew (issue #447) is detectable at
// runtime for ANY source file — not just tool-specs.ts.
//
// WHY hash the raw source text (not extracted structured data):
// SHARED_TOOL_SPECS carries `buildShape` functions (the input SCHEMAS) which are
// NOT serializable. The input schema is exactly one of the things that MUST stay
// in sync between build/ and src/, so we cannot drop it from the hash. Rather
// than probe zod with a fragile shim to reconstruct the schema shape, we hash the
// STABLE, deterministic source TEXT of tool-specs.ts. That text fully captures
// every field that must stay in sync — mcpName, inAppKey, description, tier,
// catalogLine AND the buildShape bodies (input schemas) — with zero probing
// fragility. Any edit to a spec (a renamed tool, a reworded description, a
// changed schema field) changes the text and therefore the stamp.
// WHY hash the whole src tree (not just tool-specs.ts): the runtime tools are
// assembled from far more than the spec registry — client.ts, the client/*
// domain modules, comment-signal.ts and the drawio-* helpers all ship in build/
// and are loaded by the in-app server. Hashing ONLY tool-specs.ts meant an edit
// to any of those (e.g. a behavioural fix in client.ts) left the stamp unchanged,
// so a stale build/ served the OLD code silently (issue #486). Hashing every
// src/**/*.ts closes that gap: any source edit changes the stamp.
//
// DETERMINISM: the hash is computed over the file bytes with line endings
// normalized to LF and a single trailing newline stripped, so a CRLF checkout or
// an editor's trailing-newline habit cannot make build/ and src/ disagree. No
// Date.now / randomness. The loader's dev-only stale-check (docmost-client.loader.ts)
// re-runs THIS SAME normalization + sha256 over src/tool-specs.ts and compares to
// the built REGISTRY_STAMP; the two must compute identically.
// WHY hash the raw source text (not extracted structured data): the tool input
// SCHEMAS live as `buildShape` functions which are NOT serializable, so we cannot
// reduce them to structured data without a fragile zod shim. Hashing the STABLE,
// deterministic source TEXT captures every field that must stay in sync with zero
// probing fragility. Any edit to any source file changes the text → the stamp.
//
// DETERMINISM: files are enumerated recursively, filtered to *.ts EXCLUDING
// *.generated.ts (the codegen's OWN output — including it would create a
// fixed-point cycle), and sorted by their POSIX-normalized path relative to src/
// so the order is platform-independent. Each file contributes its relative path
// AND its content with line endings normalized to LF and a single trailing
// newline stripped, so a CRLF checkout or an editor's trailing-newline habit
// cannot make build/ and src/ disagree. No Date.now / randomness. The loader's
// dev-only stale-check (docmost-client.loader.ts) re-runs THIS SAME enumeration +
// normalization + sha256 and compares to the built REGISTRY_STAMP; the two must
// compute identically.
//
// This script runs from the `build` and `pretest` npm scripts BEFORE tsc, so
// build/ always carries a stamp derived from the tool-specs.ts that was compiled.
// build/ always carries a stamp derived from the src/ tree that was compiled.
import { createHash } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
import { readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { dirname, join, relative, sep } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const SRC_DIR = join(__dirname, '..', 'src');
const TOOL_SPECS_PATH = join(SRC_DIR, 'tool-specs.ts');
const OUT_PATH = join(SRC_DIR, 'registry-stamp.generated.ts');
/**
* Deterministic stamp of the tool-specs registry content. Kept as a plain
* function (exported) so the algorithm has a single home; the loader duplicates
* only the tiny normalize+sha256 steps because it lives in the CJS server build
* and cannot import this ESM script. If you change the normalization here, mirror
* it in apps/server/src/core/ai-chat/tools/docmost-client.loader.ts.
* Recursively enumerate every `*.ts` file under `dir`, EXCLUDING the codegen's
* own `*.generated.ts` output (a self-referential cycle otherwise). Returns
* absolute paths, unsorted (the caller sorts by relative path for determinism).
* Kept as a plain exported function so the algorithm has a single home; the
* loader duplicates it because it lives in the CJS server build and cannot import
* this ESM script. If you change the walk/filter here, mirror it in
* apps/server/src/core/ai-chat/tools/docmost-client.loader.ts.
*/
export function computeRegistryStamp(toolSpecsSource) {
const normalized = toolSpecsSource.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
export function collectStampFiles(dir) {
const out = [];
for (const entry of readdirSync(dir)) {
const full = join(dir, entry);
if (statSync(full).isDirectory()) {
out.push(...collectStampFiles(full));
} else if (entry.endsWith('.ts') && !entry.endsWith('.generated.ts')) {
out.push(full);
}
}
return out;
}
/**
* Deterministic stamp of the whole src/ tree. Enumerate + sort by POSIX-relative
* path, then fold each file's relative path AND normalized content into one
* sha256. MUST stay byte-for-byte identical to the loader's recompute.
*/
export function computeRegistryStamp(srcDir) {
const files = collectStampFiles(srcDir)
.map((abs) => ({
rel: relative(srcDir, abs).split(sep).join('/'),
abs,
}))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const hash = createHash('sha256');
for (const { rel, abs } of files) {
const normalized = readFileSync(abs, 'utf8')
.replace(/\r\n/g, '\n')
.replace(/\n$/, '');
hash.update(rel, 'utf8');
hash.update('\0', 'utf8');
hash.update(normalized, 'utf8');
hash.update('\0', 'utf8');
}
return hash.digest('hex');
}
function main() {
const source = readFileSync(TOOL_SPECS_PATH, 'utf8');
const stamp = computeRegistryStamp(source);
const stamp = computeRegistryStamp(SRC_DIR);
const out =
'// AUTO-GENERATED by scripts/gen-registry-stamp.mjs — DO NOT EDIT BY HAND.\n' +
'// A deterministic hash of src/tool-specs.ts content (tool names, descriptions,\n' +
'// tiers, catalog lines and input schemas). Regenerated on every build/pretest\n' +
'// so build/ always matches the compiled src. The in-app loader recomputes this\n' +
'// from src and refuses to run on a mismatch (issue #447). This file is\n' +
'// gitignored and produced by the build — see .gitignore.\n' +
'// A deterministic hash of the whole src/ tree (every src/**/*.ts except\n' +
'// *.generated.ts). Regenerated on every build/pretest so build/ always\n' +
'// matches the compiled src. The in-app loader recomputes this from src and\n' +
'// refuses to run on a mismatch (issue #447/#486). This file is gitignored\n' +
'// and produced by the build — see .gitignore.\n' +
`export const REGISTRY_STAMP = ${JSON.stringify(stamp)};\n`;
writeFileSync(OUT_PATH, out, 'utf8');
// eslint-disable-next-line no-console
+89 -5191
View File
File diff suppressed because it is too large Load Diff
+704
View File
@@ -0,0 +1,704 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import { assertFullUuid } from "./errors.js";
import {
filterWorkspace,
filterSpace,
filterPage,
filterComment,
filterSearchResult,
} from "../lib/filters.js";
import { convertProseMirrorToMarkdown } from "../lib/markdown-converter.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
applyAnchorInDoc,
countAnchorMatches,
getAnchoredText,
resolveAnchorSelection,
normalizeForMatch,
} from "../lib/comment-anchor.js";
import { closestBlockHint } from "../lib/text-normalize.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of CommentsMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements ICommentsMixin` fails to compile on drift.
export interface ICommentsMixin {
listComments(pageId: string, includeResolved?: boolean): any;
getComment(commentId: string): any;
createComment(pageId: string, content: string, type?: "page" | "inline", selection?: string, parentCommentId?: string, suggestedText?: string): any;
updateComment(commentId: string, content: string): any;
deleteComment(commentId: string): any;
resolveComment(commentId: string, resolved: boolean): any;
checkNewComments(spaceId: string, since: string, parentPageId?: string): any;
}
export function CommentsMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & ICommentsMixin> & TBase {
abstract class CommentsMixin extends Base implements ICommentsMixin {
// --- Comment methods (ported from upstream PR #3 by Max Nikitin) ---
/**
* Normalize a comment's `content` into a ProseMirror doc object before
* markdown conversion. createComment/updateComment send content as a
* JSON.stringify(...) STRING, and the server stores it as-is, so on read it
* comes back as a string. convertProseMirrorToMarkdown returns "" for a
* string, so parse it first (guarded fall back to the raw value on any
* parse failure so a non-JSON legacy value is still handled gracefully).
*/
protected parseCommentContent(content: any): any {
if (typeof content !== "string") return content;
try {
return JSON.parse(content);
} catch {
return content;
}
}
/**
* List comments on a page (cursor-paginated), content as markdown.
*
* DEFAULT (`includeResolved = false`) hides RESOLVED THREADS WHOLESALE so the
* agent sees only active discussions: a top-level comment with `resolvedAt`
* set AND every reply under it (a reply of a closed thread is part of the
* closed thread) are dropped from `items`. `resolvedThreadsHidden` reports how
* many resolved top-level threads were hidden so the agent can re-query with
* `includeResolved: true` to see everything. Active threads always stay.
*
* Returns `{ items, resolvedThreadsHidden }` (NOT a bare array) callers that
* need the full feed (lossless export, transformPage, checkNewComments) pass
* `includeResolved: true` and read `.items`.
*/
async listComments(pageId: string, includeResolved = false) {
await this.ensureAuthenticated();
let allComments: any[] = [];
let cursor: string | null = null;
// Hard ceiling + immovable-cursor guard (mirrors paginateAll): if /comments
// ever stops advancing the cursor (the exact #442 drift scenario) this loop
// would otherwise spin forever accumulating duplicates.
const MAX_PAGES = 50;
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = { pageId, limit: 100 };
if (cursor) payload.cursor = cursor;
const response = await this.client.post("/comments", payload);
const data = response.data.data || response.data;
const items = data.items || [];
allComments = allComments.concat(items);
// Advance strictly via the server-issued cursor. A missing nextCursor or a
// cursor identical to the one we just sent means the end (or a server that
// ignores our pagination param) — stop instead of re-fetching page one.
const next: string | null = data.meta?.nextCursor || null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with a still-advancing cursor means truncation.
if (page === MAX_PAGES - 1) truncated = true;
}
if (truncated) {
console.warn(
`listComments: comments for "${pageId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
const mapped = allComments.map((comment: any) => {
const markdown = comment.content
? convertProseMirrorToMarkdown(
this.parseCommentContent(comment.content),
)
: "";
return filterComment(comment, markdown);
});
if (includeResolved) {
return { items: mapped, resolvedThreadsHidden: 0 };
}
// Ids of RESOLVED top-level threads (a top-level comment has no
// parentCommentId). A whole thread is hidden when its root is resolved.
const resolvedRootIds = new Set(
mapped
.filter((c) => !c.parentCommentId && c.resolvedAt != null)
.map((c) => c.id),
);
const items = mapped.filter((c) => {
// Hide the resolved root itself and every reply anchored to it. A reply's
// own resolvedAt is irrelevant — its membership follows the parent thread.
// ASSUMPTION: Docmost's comment model is FLAT — a reply's parentCommentId
// always points at the thread ROOT (no reply-of-reply nesting), so a single
// level of parent lookup covers a whole thread. If nested replies are ever
// introduced, a deep reply of a resolved thread would need a root-walk here.
if (!c.parentCommentId) return !resolvedRootIds.has(c.id);
return !resolvedRootIds.has(c.parentCommentId);
});
return { items, resolvedThreadsHidden: resolvedRootIds.size };
}
async getComment(commentId: string) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("get_comment", "commentId", commentId);
await this.ensureAuthenticated();
const response = await this.client.post("/comments/info", { commentId });
const comment = response.data.data || response.data;
const markdown = comment.content
? convertProseMirrorToMarkdown(this.parseCommentContent(comment.content))
: "";
return {
data: filterComment(comment, markdown),
success: true,
};
}
/** Plain text of each TOP-LEVEL block of `doc`, for anchor-failure hints. */
protected topLevelBlockTexts(doc: any): string[] {
const content = doc && Array.isArray(doc.content) ? doc.content : [];
return content
.map((b: any) => blockPlainText(b))
.filter((t: string) => t.length > 0);
}
/**
* True when per-block anchoring failed but the (normalized) selection DOES
* appear in the blocks' joined plain text i.e. it straddles a block
* boundary. Blocks are joined with a newline (collapsed to one space by
* normalizeForMatch) so a selection whose parts are separated by a paragraph
* break still matches. Callers only reach here after single-block anchoring
* (incl. the markdown-strip fallback) has already failed.
*/
protected selectionSpansMultipleBlocks(
blockTexts: string[],
selection: string,
): boolean {
const normSel = normalizeForMatch(selection).norm.trim();
if (normSel.length === 0) return false;
const joined = normalizeForMatch(blockTexts.join("\n")).norm;
return joined.indexOf(normSel) !== -1;
}
/**
* Build the actionable error for a createComment anchor MISS, porting
* editPageText's self-correction affordances: an explicit "spans multiple
* blocks" message when the selection straddles a block boundary, otherwise a
* "closest block text" hint quoting the block that holds the selection's
* longest token. `live` switches the wording between the pre-check (reading the
* persisted page) and the post-create live-anchor failure (which rolls back).
*/
protected anchorNotFoundError(
doc: any,
selection: string,
live: boolean,
): Error {
const blockTexts = this.topLevelBlockTexts(doc);
const rolled = live ? " The comment was rolled back." : "";
if (this.selectionSpansMultipleBlocks(blockTexts, selection)) {
return new Error(
"createComment: the selection spans multiple blocks; anchor on a " +
"contiguous fragment within a SINGLE paragraph/block (<=250 chars)." +
rolled,
);
}
const where = live ? "in the live document" : "in the page";
return new Error(
`createComment: could not find the selection text ${where} to anchor ` +
"the comment. Provide the EXACT contiguous text from a single " +
"paragraph/block (<=250 chars)." +
closestBlockHint(blockTexts, selection) +
rolled,
);
}
/**
* Create an inline comment anchored to its `selection` text, or a reply.
*
* Top-level comments (no `parentCommentId`) are ALWAYS inline and MUST carry a
* `selection`: the `type` argument is kept for interface compatibility but the
* effective type is coerced to "inline". The selection has to anchor in the
* document; if it cannot, the comment is rolled back and an error is thrown so
* the caller is forced to supply a proper inline selection rather than leaving
* an orphan, unanchored comment behind. Replies (parentCommentId set) inherit
* their parent's anchor: they take NO selection and are not anchored.
*/
async createComment(
pageId: string,
content: string,
type: "page" | "inline" = "page",
selection?: string,
parentCommentId?: string,
suggestedText?: string,
) {
// Fail fast (#436): a provided parent id must be a full UUID before any
// network call. Validate only when truthy — a falsy parentCommentId means
// "top-level comment" (mirrors the isReply computation below), not a reply.
if (parentCommentId) {
assertFullUuid("createComment", "parentCommentId", parentCommentId);
}
await this.ensureAuthenticated();
const isReply = !!parentCommentId;
const hasSuggestion =
suggestedText !== undefined && suggestedText !== null;
// Defense in depth mirroring the server DTO/service: a suggested edit rewrites
// the exact anchored text, so it is only meaningful on a top-level inline
// comment that carries a selection.
if (hasSuggestion) {
if (isReply) {
throw new Error(
"createComment: a suggested edit (suggestedText) cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !selection.trim()) {
throw new Error(
"createComment: a suggested edit (suggestedText) requires a 'selection' to anchor and rewrite.",
);
}
}
// Only top-level comments are inline-anchored, so they are stored as
// "inline". Replies carry no inline selection, so they keep the historical
// general ("page") type — both backward-compatible and semantically correct.
// The `type` argument is kept for interface compatibility; createComment
// normalizes the effective type internally, so callers may pass "inline".
const effectiveType: "page" | "inline" = isReply ? "page" : "inline";
if (!isReply && (!selection || !selection.trim())) {
throw new Error(
"createComment: an inline 'selection' (exact text to anchor on) is required for a top-level comment",
);
}
// For a SUGGESTION, the value we store as the comment's `selection` must be
// the RAW document substring the mark lands on (typographic quotes/dashes,
// nbsp, collapsed whitespace), NOT the agent's ASCII input. The anchor is
// placed via normalization, so when the doc was auto-converted to
// typographic the raw substring differs from the agent input; apply-time
// compares the stored selection to the marked doc text STRICTLY, so storing
// the raw substring is what makes "Apply" succeed instead of a spurious 409.
// Captured in the pre-check below (which already reads the page) and used as
// payload.selection. Ordinary comments keep sending the raw agent selection.
let anchoredSelection: string | null = null;
// Set when the anchor matched only after stripping markdown from the
// selection (the strip fallback); surfaced as a soft warning like
// editPageText does, so a stale-markdown selection is flagged.
let anchorNormalized = false;
// For a top-level comment, fail BEFORE creating anything when the selection
// is not present in the persisted document — this avoids leaving an orphan
// comment + notification behind. A read failure (network) is non-fatal: the
// live anchor step below still enforces the anchoring invariant.
if (!isReply && selection) {
try {
const page = await this.getPageJson(pageId);
if (hasSuggestion) {
// A suggestion's anchor MUST be unambiguous: applying it rewrites the
// exact anchored text, and ordinary anchoring silently takes the first
// occurrence, so 0 matches -> not found and >=2 -> ambiguous, both
// rejected BEFORE creating the comment.
const matches = countAnchorMatches(page.content, selection);
if (matches === 0) {
throw this.anchorNotFoundError(page.content, selection, false);
}
if (matches >= 2) {
throw new Error(
`createComment: the suggestion's selection is ambiguous — it occurs ${matches} times in the page. ` +
"A suggested edit must anchor to a UNIQUE location; expand the selection with surrounding context " +
"(still <=250 chars) so it appears exactly once.",
);
}
// Exactly one match: capture the RAW anchored substring to store as the
// comment selection (so apply-time equality holds). If this returns
// null despite countAnchorMatches===1 (shouldn't happen), fall back to
// the raw agent selection below rather than crash.
anchoredSelection = getAnchoredText(page.content, selection);
anchorNormalized = resolveAnchorSelection(
page.content,
selection,
).normalized;
} else {
const resolved = resolveAnchorSelection(page.content, selection);
if (!resolved.found) {
throw this.anchorNotFoundError(page.content, selection, false);
}
anchorNormalized = resolved.normalized;
}
} catch (e) {
// Rethrow our own "not found"/"ambiguous"/"spans multiple blocks" errors;
// swallow read/network errors so the live anchor step can still try (and
// enforce) anchoring.
if (
e instanceof Error &&
(e.message.startsWith("createComment: could not find the selection") ||
e.message.startsWith(
"createComment: the selection spans multiple blocks",
) ||
e.message.startsWith(
"createComment: the suggestion's selection is ambiguous",
))
) {
throw e;
}
if (process.env.DEBUG) {
console.error(
"Pre-check getPageJson failed; deferring to live anchor step:",
e,
);
}
}
}
// Convert through the full Docmost schema. Deliberately the NON-canonicalizing
// variant: a comment body may carry a footnote definition with no matching
// reference, and canonicalization would drop it (data loss). See
// markdownToProseMirror vs markdownToProseMirrorCanonical.
const jsonContent = await markdownToProseMirror(content);
const payload: Record<string, any> = {
pageId,
content: JSON.stringify(jsonContent),
type: effectiveType,
};
// For a suggestion, store the RAW anchored substring (anchoredSelection) so
// the stored selection === the text under the mark === apply-time
// expectedText. Ordinary comments (and the null fallback) keep the raw
// agent selection — their selection is only display/anchor and never used
// by apply, so their behavior is unchanged.
if (!isReply && selection)
payload.selection = anchoredSelection ?? selection;
if (parentCommentId) payload.parentCommentId = parentCommentId;
// Only a top-level inline comment (with a selection) may carry a suggestion.
if (!isReply && selection && hasSuggestion) {
payload.suggestedText = suggestedText;
}
const response = await this.client.post("/comments/create", payload);
const comment = response.data.data || response.data;
const markdown = comment.content
? convertProseMirrorToMarkdown(this.parseCommentContent(comment.content))
: content;
const result: any = {
data: filterComment(comment, markdown),
success: true,
};
// Replies inherit the parent's anchor: no selection, no anchoring.
if (isReply) {
return result;
}
// Anchor the comment in the document. The /comments/create API records the
// comment + its `selection` text, but it does NOT insert the comment MARK
// into the page content, so without this the inline comment has no
// highlight/anchor and is not clickable. If anchoring fails the comment is
// rolled back (deleted) and an error is thrown — never an orphan comment.
const newCommentId: string = comment.id;
// Guard: a create response without an id would mean writing a comment mark
// with commentId: undefined and a later delete of a falsy id. We have no id
// to roll back here (nothing was created with an id), so just fail loudly.
if (!newCommentId) {
throw new Error(
"createComment: the server returned no comment id, so the comment could not be anchored",
);
}
let anchored = false;
// Set inside the transform when a suggestion's live anchor is ambiguous
// (>=2 occurrences), so the rollback path can surface the right error.
let ambiguousInLiveDoc = false;
// Captured inside the transform on a not-found abort, so the rollback path
// can surface the closest-block / spans-multiple-blocks hint built from the
// LIVE document (the pre-check page is not in scope there).
let liveNotFoundError: Error | null = null;
try {
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// /comments/create REST call above keeps the agent-supplied id.
const pageUuid = await this.resolvePageId(pageId);
// Route through the mutatePage seam (not the free function) so this
// wrapper's uniqueness gate + rollback can be unit-tested without a live
// Hocuspocus collab socket.
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (hasSuggestion) {
// Authoritative uniqueness check against the LIVE document: a
// suggestion must anchor to EXACTLY ONE occurrence, otherwise
// "Apply" would rewrite the wrong/ambiguous text. If the live doc
// no longer has exactly one occurrence (it changed since the
// pre-check), abort so the just-created comment is rolled back
// rather than mis-anchored to the first occurrence.
const liveCount = countAnchorMatches(doc, selection as string);
if (liveCount !== 1) {
ambiguousInLiveDoc = liveCount >= 2;
if (liveCount === 0) {
liveNotFoundError = this.anchorNotFoundError(
doc,
selection as string,
true,
);
}
return null;
}
}
if (applyAnchorInDoc(doc, selection as string, newCommentId)) {
anchored = true;
return doc;
}
// Selection text not found in the LIVE document: abort the write. The
// rollback + throw below turns this into a hard error.
liveNotFoundError = this.anchorNotFoundError(
doc,
selection as string,
true,
);
return null;
},
);
result.verify = mutation.verify;
} catch (e) {
// The comment record already exists; roll it back so we never leave an
// orphan, then rethrow the original anchoring error.
await this.safeDeleteComment(newCommentId);
throw e;
}
if (!anchored) {
// Mutation aborted because the selection was not found (or, for a
// suggestion, was ambiguous) in the live document. Roll back the comment
// and surface a hard error.
await this.safeDeleteComment(newCommentId);
if (ambiguousInLiveDoc) {
throw new Error(
"createComment: the suggestion's selection is ambiguous in the live document (multiple occurrences); the comment was rolled back. Expand the selection with surrounding context so it is unique.",
);
}
throw (
liveNotFoundError ??
new Error(
"createComment: failed to anchor the comment (selection not found in the live document); the comment was rolled back",
)
);
}
// Soft warning (like editPageText): the selection only matched after
// stripping markdown, so the caller likely quoted a styled fragment.
if (anchorNormalized) {
result.warning =
"The selection matched only after stripping markdown syntax; the comment " +
"was anchored on the document's plain text. Copy the selection verbatim " +
"from getPage / searchInPage output to avoid this.";
}
result.anchored = true;
return result;
}
/**
* Best-effort rollback of a just-created comment. Swallows any delete failure
* (logging under DEBUG) so a failed cleanup never masks the original error.
*/
protected async safeDeleteComment(commentId: string): Promise<void> {
// Defense in depth: never call the delete API with a falsy id — there is
// nothing to roll back, and deleteComment(undefined) would hit a bad route.
if (!commentId) return;
try {
await this.deleteComment(commentId);
} catch (delErr) {
if (process.env.DEBUG) {
console.error(
"Failed to roll back comment after anchoring error:",
delErr,
);
}
}
}
async updateComment(commentId: string, content: string) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("updateComment", "commentId", commentId);
await this.ensureAuthenticated();
// NON-canonicalizing on purpose (comment body — see createComment).
const jsonContent = await markdownToProseMirror(content);
await this.client.post("/comments/update", {
commentId,
content: JSON.stringify(jsonContent),
});
return {
success: true,
commentId,
message: "Comment updated successfully.",
};
}
async deleteComment(commentId: string) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("deleteComment", "commentId", commentId);
await this.ensureAuthenticated();
return this.client
.post("/comments/delete", { commentId })
.then((res) => res.data);
}
/**
* Resolve or reopen a top-level comment thread (reversible `resolved`
* toggles the state). Only top-level comments can be resolved; the server
* rejects resolving a reply. Hits POST /comments/resolve.
*/
async resolveComment(commentId: string, resolved: boolean) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("resolveComment", "commentId", commentId);
await this.ensureAuthenticated();
const response = await this.client.post("/comments/resolve", {
commentId,
resolved,
});
const comment = response.data?.data ?? response.data;
return {
success: true,
commentId,
resolved,
comment,
};
}
/**
* Check for new comments across pages in a space (optionally scoped to a
* subtree): pages updated after `since` are scanned and their comments
* filtered by createdAt > since.
*/
async checkNewComments(
spaceId: string,
since: string,
parentPageId?: string,
) {
await this.ensureAuthenticated();
const sinceDate = new Date(since);
// Reject an unparseable `since`: comparing against an Invalid Date silently
// yields zero new comments (every `>` against NaN is false), which would
// mask a malformed input as "nothing new" instead of erroring.
if (Number.isNaN(sinceDate.getTime())) {
throw new Error(
`checkNewComments: invalid "since" date "${since}"; expected an ISO-8601 timestamp`,
);
}
// 1. Enumerate the FULL set of pages in scope via the page tree (a complete
// page index), NOT the bounded "/pages/recent" feed which caps at ~5000
// recent items and silently misses comments on older pages.
//
// Subtree scope: when parentPageId is given, the scope is that page ITSELF
// plus every descendant. Otherwise the scope is the whole space (all roots
// and their descendants).
//
// NOTE: do NOT pre-filter by page.updatedAt — creating a comment does not
// bump it (verified on a live server), so such a filter silently misses
// comments on pages that were not otherwise edited. The complete tree walk
// already restricts the scope correctly, so no recent-feed allow-list is
// needed any more.
//
// The subtree scope (parentPageId given) already INCLUDES the root node
// itself: /pages/tree seeds getPageAndDescendants with id = parentPageId, so
// no separate getPageRaw fetch for the parent is needed.
const { pages: pagesInScope, truncated } = await this.enumerateSpacePages(
spaceId,
parentPageId,
);
// 2. Fetch comments for each page, keep ones created after since
const results: any[] = [];
for (const page of pagesInScope) {
try {
// Full feed (incl. resolved): a "new comments since" scan reports all
// recent activity; the active-only filter is scoped to listComments.
const comments = (await this.listComments(page.id, true)).items;
const newComments = comments.filter(
(c: any) => new Date(c.createdAt) > sinceDate,
);
if (newComments.length > 0) {
results.push({
pageId: page.id,
pageTitle: page.title,
comments: newComments,
});
}
} catch (e: any) {
// Skip pages with errors (e.g. deleted between calls)
}
}
const totalNewComments = results.reduce(
(sum, r) => sum + r.comments.length,
0,
);
// `truncated` is reported by enumerateSpacePages: it is true ONLY when the
// stdio fallback BFS hit its node cap. The primary /pages/tree path is
// uncapped, so a space with legitimately many pages is not falsely flagged.
return {
since,
scope: parentPageId ? `subtree of ${parentPageId}` : `space ${spaceId}`,
checkedPages: pagesInScope.length,
pagesWithNewComments: results.length,
totalNewComments,
truncated,
comments: results,
};
}
// --- Image upload / embedding ---
/** Map a Content-Type string to a supported MIME type, or null if unsupported. */
}
return CommentsMixin;
}
+737
View File
@@ -0,0 +1,737 @@
// Shared client context + core seams (issue #450). The abstract base of the
// DocmostClient mixin chain: it owns ALL shared instance state (the axios
// client, apiUrl, auth tokens, the resolvePageId cache, the collab-token cache,
// the sandbox/metrics sinks) and the core HTTP/auth/pagination/write seams every
// domain module builds on. Domain modules are mixins layered on top; the final
// DocmostClient (client.ts) assembles them. Extracted VERBATIM from the original
// monolith — only field/seam visibility was widened from `private` to
// `protected` so sibling mixins can reach the shared state through `this`, and
// the cross-module methods that live in other mixins are declared `abstract`
// here so `this.<method>` type-checks. No behaviour changed.
import axios, { AxiosInstance } from "axios";
import FormData from "form-data";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
acquireCollabSession,
isCollabAuthFailedError,
} from "../lib/collab-session.js";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import { getCollabToken, performLogin } from "../lib/auth-utils.js";
import { formatDocmostAxiosError } from "./errors.js";
// A generic mixin base constructor (issue #450). Each domain mixin is a factory
// `<T extends GConstructor<DocmostClientContext>>(Base: T) => class extends Base`
// so the mixins compose into one prototype chain sharing this context.
export type GConstructor<T = {}> = abstract new (...args: any[]) => T;
/**
* Configuration for a DocmostClient / MCP server instance. A discriminated
* union: either service-account credentials (email/password the client calls
* performLogin, powering the external /mcp HTTP endpoint and the stdio CLI) OR
* a token getter (getToken the client uses the returned BARE access JWT as
* the Bearer and never calls performLogin; used for the internal per-user path).
*
* Both branches may ALSO carry an optional `getCollabToken` provider. When set,
* content mutations (which go over the collaboration websocket) use the token it
* returns INSTEAD of calling `POST /auth/collab-token`. The internal per-user
* agent path uses this to hand the client a provenance collab token (signed
* `actor:'agent'`+`aiChatId`), so agent content edits are attributed without a
* spoofable client-side field. When absent the client keeps the original
* `/auth/collab-token` path (service-account/stdio unchanged).
*
* Housed here (not in index.ts) so client.ts has no type dependency on index.ts;
* index.ts re-exports it for the package's public surface.
*/
// Sink the stash tool writes blobs into. The host app binds this to its in-RAM
// SandboxStore and composes the public `uri` (the package never sees the store
// or any env). `put` returns the anonymous read URL plus integrity metadata.
export type SandboxPut = (
buf: Buffer,
mime: string,
) => { uri: string; sha256: string; size: number };
export type DocmostMcpConfig = { apiUrl: string } & (
| { email: string; password: string }
| { getToken: () => Promise<string> } // returns a BARE JWT; the client adds "Bearer "
) & {
// Optional collab-token provider (returns a ready collab JWT). Common to
// both branches; see the type doc above.
getCollabToken?: () => Promise<string>;
// Optional blob sandbox sink. Present only where the stash tool is wired;
// when absent, stashPage throws a clear "not configured" error. The
// optional `has`/`evict` probes let stashPage keep its mirror counts honest
// under the store's FIFO eviction (see stashPage); older sinks omit them.
sandbox?: {
put: SandboxPut;
has?: (uri: string) => boolean;
evict?: (uri: string) => void;
};
// Dependency-neutral metrics sink. When present, the client emits generic
// (name, value, labels) samples; the HOST maps those names onto its own
// metrics registry (the package never depends on prom-client or the server).
// Absent in standalone/stdio mode → the client is a complete no-op here.
onMetric?: (
name: string,
value: number,
labels?: Record<string, string>,
) => void;
};
/**
* Collab-token cache TTL in milliseconds (issue #435). Read fresh from the
* environment on every mint like collab-session.ts readConfig so tests and a
* live rollback can change it without reloading the module.
*
* Why a cache at all: the live CollabSession registry (#400/#431) keys sessions
* on (wsUrl, pageId, collabToken) for identity isolation (invariant 4). But BOTH
* collab-token sources mint a FRESH token per mutation the in-app provider
* re-signs a JWT whose iat/exp (seconds) changes every second, and the external
* MCP POSTs /auth/collab-token each call so the token in the key changed on
* every op and the session was almost never reused (connect-storms, 25s
* timeouts, zombie sessions). Caching the token per-client keeps the key stable
* across a burst of mutations so ONE session is reused.
*
* Default 5 min: well under the 24h collab-token lifetime AND <= the collab
* session max-age (10 min, MCP_COLLAB_SESSION_MAX_AGE_MS), so the
* permission-staleness window is not widened beyond what #431 already accepted.
* The rollback knob is an EXPLICIT 0 (or a negative number): that DISABLES the
* cache an exact fetch-per-call legacy path, mirroring how idleMs<=0 disables
* the session cache. Unset OR unparseable (e.g. a typo like "5min", "abc") falls
* back to the 5-min default with the cache ON parseInt yields NaN, which is
* treated as "not configured", not as "disabled". So to turn the cache off you
* must set the value to exactly 0, not to garbage.
*/
function readCollabTokenTtlMs(): number {
const raw = parseInt(process.env.MCP_COLLAB_TOKEN_TTL_MS ?? "", 10);
return Number.isFinite(raw) ? Math.max(0, raw) : 5 * 60 * 1000;
}
export abstract class DocmostClientContext {
protected client: AxiosInstance;
protected token: string | null = null;
protected apiUrl: string;
// email/password are only set on the service-account (credentials) variant;
// null on the getToken variant (where there are no credentials to log in with).
protected email: string | null = null;
protected password: string | null = null;
// Per-user token provider. When set, login() calls it to obtain a BARE access
// JWT instead of performLogin, and the 401/403 re-auth path re-calls it.
protected getTokenFn: (() => Promise<string>) | null = null;
// Optional collab-token provider. When set, getCollabTokenWithReauth() returns
// its token instead of calling POST /auth/collab-token; on a 401/403 it is
// re-invoked once. Used by the internal agent to carry signed provenance.
protected getCollabTokenFn: (() => Promise<string>) | null = null;
// Optional blob-sandbox sink for the stash tool. Null when not configured.
protected sandboxPut: SandboxPut | null = null;
// Optional probes paired with the sink. `has` lets stashPage detect a blob
// FIFO-evicted by a LATER put in the same stash; `evict` lets it free this
// op's image blobs if the final doc put throws. Null when the sink omits them.
protected sandboxHas: ((uri: string) => boolean) | null = null;
protected sandboxEvict: ((uri: string) => void) | null = null;
// Optional dependency-neutral metrics sink (see DocmostMcpConfig.onMetric).
// Null on the legacy positional form and whenever the host omits it → no-op.
protected onMetricFn:
| ((name: string, value: number, labels?: Record<string, string>) => void)
| null = null;
// In-flight login dedup: when the token expires, the 401 interceptor,
// ensureAuthenticated, getCollabTokenWithReauth and the two multipart retries
// can all call login() at once. Memoizing a single promise collapses that
// thundering herd into ONE /auth/login request that everyone awaits.
protected loginPromise: Promise<void> | null = null;
// Canonical-UUID cache for resolvePageId: maps an agent-supplied slugId to the
// page's canonical UUID, so repeated collab edits on the same page do not
// re-fetch /pages/info. A UUID input short-circuits before this cache (see
// resolvePageId), so only slugId->uuid entries are stored/read here.
protected pageIdCache = new Map<string, string>();
// Collab-token cache (issue #435): the last minted collab token plus the
// wall-clock time it was minted, so a burst of content mutations reuses ONE
// token and therefore ONE live CollabSession (whose registry key includes the
// token — #400 invariant 4). Per-instance: a DocmostClient is built per
// user/per chat request, so a cached token can never leak across identities.
// Reset whenever the client's identity changes (login() / this.token cleared);
// bypassed on a forced refresh (the 401/403 reauth path). null = no token yet.
protected collabTokenCache: { token: string; mintedAt: number } | null = null;
// Two construction forms:
// - new DocmostClient(config) // discriminated union (current)
// - new DocmostClient(baseURL, email, password) // legacy positional creds
// The positional form is retained so existing callers/tests keep working; it
// is exactly equivalent to the credentials branch of the object form.
constructor(config: DocmostMcpConfig);
constructor(baseURL: string, email: string, password: string);
constructor(
configOrBaseURL: DocmostMcpConfig | string,
email?: string,
password?: string,
) {
// Normalize the legacy positional form into the object union.
const config: DocmostMcpConfig =
typeof configOrBaseURL === "string"
? { apiUrl: configOrBaseURL, email: email!, password: password! }
: configOrBaseURL;
this.apiUrl = config.apiUrl;
if ("getToken" in config) {
// Token variant: carry the user's JWT via getToken; no credentials, so
// login() must never call performLogin (there is nothing to log in with).
this.getTokenFn = config.getToken;
} else {
// Service-account variant: behaves exactly as before (performLogin).
this.email = config.email;
this.password = config.password;
}
// Optional, available to both variants. When present, content mutations get
// their collab token from here instead of POST /auth/collab-token.
if (config.getCollabToken) {
this.getCollabTokenFn = config.getCollabToken;
}
if (config.sandbox) {
this.sandboxPut = config.sandbox.put;
this.sandboxHas = config.sandbox.has ?? null;
this.sandboxEvict = config.sandbox.evict ?? null;
}
// Legacy positional form carries no onMetric → null (complete no-op).
this.onMetricFn = config.onMetric ?? null;
this.client = axios.create({
baseURL: this.apiUrl,
// Default request timeout so a hung connection cannot wedge a per-page
// lock or block the server indefinitely. Multipart uploads override this
// with a longer per-request timeout.
timeout: 30000,
headers: {
"Content-Type": "application/json",
},
});
// Re-authenticate transparently on a 401/403 once: the JWT authToken can
// expire while the server is long-running, after which every cached-token
// request would otherwise fail until a manual restart. On such a response,
// clear the stale token, perform a fresh login, and replay the original
// request exactly once (guarded by config._retry to avoid infinite loops;
// the login request itself is never retried).
this.client.interceptors.response.use(
(response) => response,
async (error) => {
const config = error.config;
const status = error.response?.status;
const isAuthError = status === 401 || status === 403;
const isLoginRequest =
typeof config?.url === "string" && config.url.includes("/auth/login");
if (config && isAuthError && !config._retry && !isLoginRequest) {
config._retry = true;
// Drop the stale token + Authorization header before re-login. Also
// clear the collab-token cache (#435): a new identity/login must not
// keep serving a collab token minted under the old one.
this.token = null;
this.collabTokenCache = null;
delete this.client.defaults.headers.common["Authorization"];
try {
await this.login();
} catch (loginError) {
// Re-login failed: surface the original error to the caller.
return Promise.reject(error);
}
// Re-issue the original request with the freshly minted Bearer token.
// Read it from the default header that login() just set, not from
// this.token, to avoid a theoretical "Bearer null" if this.token was
// cleared between login() resolving and this point.
config.headers = config.headers || {};
config.headers["Authorization"] =
this.client.defaults.headers.common["Authorization"];
return this.client.request(config);
}
return Promise.reject(error);
},
);
// Diagnostics interceptor (issue #437). Registered AFTER the re-login
// interceptor so a successful re-login retry (which resolves to a real
// response) is never seen here as an error; only a genuine failure reaches
// this rejection handler. It reformats error.message IN PLACE (see
// formatDocmostAxiosError — kept as a mutation, not a custom Error class, so
// the surrounding axios.isAxiosError / error.response?.status / config._retry
// checks keep working) and re-rejects the SAME error. The _docmostFormatted
// flag makes a re-processed retry-failure a no-op.
this.client.interceptors.response.use(
(response) => response,
(error) => {
formatDocmostAxiosError(error);
return Promise.reject(error);
},
);
}
// --- Cross-module seams (issue #450) -----------------------------------
// A method in one domain mixin sometimes calls a PROTECTED method owned by
// another mixin (e.g. nodes-write -> validateDocUrls in doc-validate). Those
// callees are `protected`, so they cannot be surfaced through the public
// per-mixin interfaces. Declaring them here on the shared base lets `this.<m>`
// type-check across modules. Each is a stub that is ALWAYS overridden by the
// owning mixin (layered above this base in the chain), so the body never runs;
// it throws only to make an impossible mis-wiring loud instead of silent.
// (The PUBLIC cross-module callees — getPage, getPageJson, listComments,
// deleteComment, listPageHistory — arrive via the mixins' public interfaces,
// so they are not restated here.)
protected enumerateSpacePages(
_spaceId: string,
_rootPageId?: string,
): Promise<{ pages: any[]; truncated: boolean }> {
throw new Error("enumerateSpacePages not wired (missing ReadMixin)");
}
protected validateDocUrls(_node: any, _depth?: number): void {
throw new Error("validateDocUrls not wired (missing DocValidateMixin)");
}
protected validateDocStructure(_node: any, _depth?: number): void {
throw new Error("validateDocStructure not wired (missing DocValidateMixin)");
}
protected assertValidNodeShape(_op: string, _node: any): void {
throw new Error("assertValidNodeShape not wired (missing DocValidateMixin)");
}
protected fetchInternalFile(
_src: string,
): Promise<{ buffer: Buffer; mime: string }> {
throw new Error("fetchInternalFile not wired (missing StashMixin)");
}
protected uploadAttachmentBuffer(
_pageId: string,
_buffer: Buffer,
_fileName: string,
_mime: string,
): Promise<{ id: string; fileName: string; fileSize: number }> {
throw new Error("uploadAttachmentBuffer not wired (missing MediaMixin)");
}
protected fetchAttachmentText(_src: string): Promise<string> {
throw new Error("fetchAttachmentText not wired (missing MediaMixin)");
}
// PUBLIC cross-module callees. Declared here too (as always-overridden stubs)
// so a mixin calling e.g. `this.getPageJson` type-checks against the base —
// the mixin's own public interface only covers its own methods. The real
// implementations live in ReadMixin / CommentsMixin / PagesMixin and shadow
// these on the prototype chain.
getPage(_pageId: string): Promise<any> {
throw new Error("getPage not wired (missing ReadMixin)");
}
getPageJson(_pageId: string): Promise<any> {
throw new Error("getPageJson not wired (missing ReadMixin)");
}
listComments(_pageId: string, _includeResolved?: boolean): Promise<any> {
throw new Error("listComments not wired (missing CommentsMixin)");
}
deleteComment(_commentId: string): Promise<any> {
throw new Error("deleteComment not wired (missing CommentsMixin)");
}
listPageHistory(_pageId: string, _cursor?: string): Promise<any> {
throw new Error("listPageHistory not wired (missing PagesMixin)");
}
/** Application base URL (API URL without the /api suffix). */
get appUrl(): string {
return this.apiUrl.replace(/\/api\/?$/, "");
}
async login() {
// Reuse an in-flight login if one is already running so concurrent callers
// share a single token fetch instead of each issuing their own.
if (!this.loginPromise) {
// Token variant: re-fetch a BARE JWT via getToken() (there are no
// credentials to log in with — on a 401/403 the interceptor below calls
// login() again, which re-invokes getToken()). Credentials variant:
// performLogin against /auth/login exactly as before.
const fetchToken = this.getTokenFn
? this.getTokenFn()
: performLogin(this.apiUrl, this.email!, this.password!);
this.loginPromise = fetchToken
.then((token) => {
// Guard against an empty/invalid token (e.g. a getToken provider that
// resolves to "" or null): without this an empty token would set a
// literal "Authorization: Bearer null"/"Bearer " header and every
// request would 401 with a confusing error. Fail loudly instead.
if (typeof token !== "string" || token.length === 0) {
throw new Error("getToken returned an empty token");
}
this.token = token;
// Identity (re)established: drop any collab token minted under a
// previous identity so the #435 cache can never outlive it.
this.collabTokenCache = null;
this.client.defaults.headers.common["Authorization"] =
`Bearer ${token}`;
})
.finally(() => {
this.loginPromise = null;
});
}
return this.loginPromise;
}
async ensureAuthenticated() {
if (!this.token) {
await this.login();
}
}
/**
* Fetch a collaboration token, transparently re-authenticating once on a
* 401/403. getCollabToken() uses bare axios internally, so it is NOT covered
* by this.client's response interceptor; this helper replicates that
* behaviour for collab-token requests: ensure a token, try once, and on an
* expired-token auth error perform a fresh login and retry exactly once.
*
* Collab-token cache (issue #435): both sources the getCollabToken provider
* (in-app agent) AND the REST /auth/collab-token endpoint (external MCP) mint
* a FRESH token per call, whose string therefore changes every op. Since the
* live CollabSession registry keys on the token string (#400/#431 invariant 4),
* that churned the key and defeated session reuse. So we cache the last minted
* token per-client for readCollabTokenTtlMs() and hand it back for a burst of
* mutations, keeping the session key stable. `forceRefresh` bypasses the cache
* (the 401/403 reauth retry uses it, so the retry cannot be handed the same
* stale token that just failed otherwise reauth would be a no-op). TTL 0
* disables the cache: exact fetch-per-call legacy behaviour.
*/
protected async getCollabTokenWithReauth(
forceRefresh = false,
): Promise<string> {
const ttl = readCollabTokenTtlMs();
// Serve the cached collab token while it is still fresh (identity isolation
// is preserved: the cache is a per-instance field on a client built per
// user/per chat request, and it is cleared on every identity change).
if (
!forceRefresh &&
ttl > 0 &&
this.collabTokenCache &&
Date.now() - this.collabTokenCache.mintedAt < ttl
) {
return this.collabTokenCache.token;
}
// Collab-token PROVIDER path: when a getCollabToken provider was supplied
// (the internal agent's provenance collab token), use it instead of the
// REST /auth/collab-token endpoint. Re-invoke it once on a 401/403 (e.g. the
// signed token expired between content mutations in a long agent turn).
if (this.getCollabTokenFn) {
try {
const token = await this.getCollabTokenFn();
if (typeof token !== "string" || token.length === 0) {
throw new Error("getCollabToken returned an empty token");
}
return this.rememberCollabToken(token, ttl);
} catch (e) {
// On an auth error retry EXACTLY once, forcing a refresh so the retry
// re-invokes the provider (bypassing the cache) for a genuinely fresh
// token. `!forceRefresh` bounds it to a single retry (no loop).
if (this.isCollabAuthError(e) && !forceRefresh) {
return this.getCollabTokenWithReauth(true);
}
throw e;
}
}
await this.ensureAuthenticated();
try {
const token = await getCollabToken(this.apiUrl, this.token!);
return this.rememberCollabToken(token, ttl);
} catch (e) {
// getCollabToken wraps the AxiosError in a plain Error but attaches the
// HTTP status as `.status`, so isCollabAuthError detects an auth failure
// via either the raw AxiosError shape OR the attached status.
if (this.isCollabAuthError(e) && !forceRefresh) {
// Fresh login (which clears this.token AND the collab-token cache), then
// retry exactly once with the cache bypassed via forceRefresh.
await this.login();
return this.getCollabTokenWithReauth(true);
}
throw e;
}
}
/**
* Store a freshly minted collab token in the per-client cache (issue #435) and
* return it unchanged. No-op write when the cache is disabled (ttl<=0) or the
* token is empty, so a disabled cache is exact fetch-per-call legacy behaviour
* and a bad token is never cached.
*/
protected rememberCollabToken(token: string, ttl: number): string {
if (ttl > 0 && typeof token === "string" && token.length > 0) {
this.collabTokenCache = { token, mintedAt: Date.now() };
}
return token;
}
/**
* True when an error carries a 401/403 either as a raw AxiosError
* (`error.response.status`) or as the plain-Error `.status` that
* lib/auth-utils.getCollabToken attaches after wrapping the AxiosError.
*/
protected isCollabAuthError(e: unknown): boolean {
const axiosStatus = axios.isAxiosError(e) ? e.response?.status : undefined;
const attachedStatus = (e as any)?.status;
return (
axiosStatus === 401 ||
axiosStatus === 403 ||
attachedStatus === 401 ||
attachedStatus === 403
);
}
/**
* Run a collab write and, on a Hocuspocus HANDSHAKE auth failure, self-heal
* once (#486). Symmetric to the HTTP-401 path in getCollabTokenWithReauth: the
* REST interceptor and login() already drop the cached collab token on a 401/
* 403, but a rejected WEBSOCKET handshake left the stale token in the cache, so
* every subsequent mutation kept re-presenting the same bad token for up to the
* collab-token TTL (minutes) with no self-heal. Here, when the write rejects
* with the tagged collab-auth error, we invalidate the cached token and retry
* the write EXACTLY once with a force-refreshed token. Not a loop: a second
* failure (or any non-auth error) propagates unchanged.
*
* `write` receives the token to use, so the retry can hand it a genuinely fresh
* one rather than re-running with the same stale string.
*/
protected async writeWithCollabAuthRetry<T>(
collabToken: string,
write: (token: string) => Promise<T>,
): Promise<T> {
try {
return await write(collabToken);
} catch (e) {
if (!isCollabAuthFailedError(e)) throw e;
// The WS handshake rejected our token: drop it from the cache so it can't
// be reused for the rest of the TTL, mint a fresh one (forceRefresh bypasses
// the cache and re-invokes the provider/login), and retry the write once.
this.collabTokenCache = null;
const fresh = await this.getCollabTokenWithReauth(true);
return await write(fresh);
}
}
/**
* Connect to the collaboration websocket, read the live doc, apply
* `transform`, write the result, and wait for the server to persist it
* WITHOUT acquiring the per-page lock.
*
* This mirrors collaboration.mutatePageContent EXCEPT that it does not call
* withPageLock. It exists solely so replaceImage can hold ONE withPageLock
* across its scan -> upload -> write sequence: the per-page mutex is NOT
* reentrant, so calling the normal (self-locking) mutatePageContent inside an
* outer withPageLock for the same pageId would deadlock. The caller MUST hold
* the page lock for the whole operation; this helper assumes that invariant.
*
* `transform` receives the live ProseMirror doc and returns the NEW full doc
* to write, or `null` to abort with no write. Errors thrown by `transform`
* propagate to the caller.
*
* Resolves a `MutationResult { doc, verify }` mirroring mutatePageContent, so
* every content mutator (including replaceImage) can return a verifiable
* change report. The report is computed AFTER the atomic read->write and
* never throws.
*/
protected async mutateLiveContentUnlocked(
pageId: string,
collabToken: string,
transform: (liveDoc: any) => any | null,
): Promise<MutationResult> {
// Reuse a live CollabSession for the page (issue #400) instead of opening a
// fresh provider per op. acquireCollabSession does NOT take the per-page
// lock — the caller (replaceImage) already holds ONE withPageLock across its
// scan -> upload -> write sequence, and the mutex is not reentrant, so
// taking it here would deadlock. The synchronous read->write section and the
// unsyncedChanges/connectionLost ack logic live in CollabSession.mutate,
// preserved verbatim from the old inline machine (incl. the #152 structural
// diff that keeps a live editor's cursor anchored).
// Wrap in the collab-auth self-heal (#486): a rejected WS handshake drops the
// cached collab token and retries once with a fresh one (the retry passes the
// refreshed token down to acquireCollabSession via `token`).
return this.writeWithCollabAuthRetry(collabToken, async (token) => {
const session = await acquireCollabSession(pageId, token, this.apiUrl, {
// Only the actual 25s collab connect timeout emits this — the connect-vs-
// unload signal; the other failure paths must NOT emit it.
onConnectTimeout: () =>
this.onMetricFn?.("collab_connect_timeouts_total", 1),
});
try {
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh.
session.destroy("mutate failed");
throw e;
}
});
}
/**
* Generic pagination handler for Docmost API endpoints
*/
async paginateAll<T = any>(
endpoint: string,
basePayload: Record<string, any> = {},
limit: number = 100,
): Promise<T[]> {
await this.ensureAuthenticated();
const clampedLimit = Math.max(1, Math.min(100, limit));
// Hard ceiling on the number of pages to fetch: guards against a server
// that returns a perpetually-true hasNextPage (which would otherwise loop
// forever and accumulate duplicates).
const MAX_PAGES = 50;
let cursor: string | undefined;
let allItems: T[] = [];
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = {
...basePayload,
limit: clampedLimit,
};
if (cursor) payload.cursor = cursor;
const response = await this.client.post(endpoint, payload);
const data = response.data;
const items = data.data?.items || data.items || [];
const meta = data.data?.meta || data.meta;
allItems = allItems.concat(items);
// Advance strictly via the server-issued cursor. A missing nextCursor (or
// hasNextPage false) means we reached the end. A cursor identical to the
// one we just sent means the server did not understand our pagination
// param — stop instead of re-fetching page one forever and duplicating.
const next = meta?.hasNextPage ? meta?.nextCursor : null;
if (!next || next === cursor) {
// If the server still reports more pages but stopped issuing a usable
// cursor at the ceiling, flag the result as truncated below.
if (page === MAX_PAGES - 1 && meta?.hasNextPage) truncated = true;
break;
}
cursor = next;
// Reaching the ceiling with more pages still available means the result
// set is truncated.
if (page === MAX_PAGES - 1) truncated = true;
}
// If the loop stopped because it hit the MAX_PAGES ceiling while the server
// still reported more results, the result set is truncated — warn so the
// caller is not silently handed an incomplete list.
if (truncated) {
console.warn(
`paginateAll: results from "${endpoint}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
return allItems;
}
/** Raw page info including the ProseMirror JSON content and slugId. */
async getPageRaw(pageId: string) {
await this.ensureAuthenticated();
const response = await this.client.post("/pages/info", { pageId });
return response.data?.data ?? response.data;
}
/**
* Resolve an agent-supplied pageId to the page's CANONICAL UUID (`page.id`),
* so every collaboration document the MCP opens is named `page.<uuid>` the
* SAME name the web editor always uses (`page.${page.id}`).
*
* The agent commonly passes a 10-char public slugId (from URLs/listings) as
* the pageId. The web editor opens the collab doc by UUID, but the MCP used to
* pass that slugId straight into the collab doc name (`page.<slugId>`). For one
* DB row that produced TWO independent Yjs documents whose debounced stores
* clobbered each other the agent's edit was silently lost (#260).
*
* A UUID input short-circuits with no network round-trip. A slugId is resolved
* once via getPageRaw and cached (both slugId->uuid and uuid->uuid), so
* repeated edits on the same page add no extra request.
*/
protected async resolvePageId(pageId: string): Promise<string> {
if (isUuid(pageId)) return pageId;
const cached = this.pageIdCache.get(pageId);
if (cached) return cached;
const data = await this.getPageRaw(pageId);
const uuid = data?.id;
if (typeof uuid !== "string" || !uuid) {
throw new Error(
`Could not resolve a canonical page id for "${pageId}"`,
);
}
this.pageIdCache.set(pageId, uuid);
return uuid;
}
/**
* Page-locked write seam over collaboration.mutatePageContent. Production just
* delegates; it exists as an overridable method so the insertFootnote wrapper
* (transform abort-on-not-found + response shaping) can be unit-tested without
* standing up a live Hocuspocus collab socket.
*
* SELF-RESOLVES the pageId to the canonical UUID (issue #449, "resolve-then-
* lock"): every write must lock and key its CollabSession by the UUID, never a
* raw slugId (#260). resolvePageId is cached/idempotent, so a caller that
* already resolved pays no extra round-trip; centralizing it here means a
* caller that reaches this seam with a raw slugId still locks correctly instead
* of silently splitting the mutex key. withPageLock also asserts the key is a
* UUID as a hard backstop.
*/
protected async mutatePage(
pageId: string,
collabToken: string,
apiUrl: string,
transform: (doc: any) => any,
): Promise<{ doc?: any; verify?: any }> {
const pageUuid = await this.resolvePageId(pageId);
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
mutatePageContent(pageUuid, token, apiUrl, transform),
);
}
/**
* Full-document write seam over collaboration.replacePageContent. Production
* just delegates; it exists as an overridable method so the full-doc write
* tools (updatePageJson, copyPageContent) can have their footnote-
* canonicalization binding unit-tested without a live Hocuspocus collab socket.
*
* SELF-RESOLVES the pageId to the canonical UUID (issue #449, "resolve-then-
* lock") for the same reason as mutatePage above the lock/CollabSession key
* is guaranteed canonical here, not left to the caller's discipline.
*/
protected async replacePage(
pageId: string,
doc: any,
collabToken: string,
apiUrl: string,
): Promise<{ doc?: any; verify?: any }> {
const pageUuid = await this.resolvePageId(pageId);
// #486: on a rejected collab-WS handshake, invalidate + refresh the token and
// retry the write once (symmetric to the HTTP-401 reauth path).
return this.writeWithCollabAuthRetry(collabToken, (token) =>
replacePageContent(pageUuid, doc, token, apiUrl),
);
}
/**
* Export a page to a single self-contained Docmost-flavoured markdown file:
* meta block + body (with inline comment anchors + diagrams) + comment
* threads. Lossless round-trip target; see importPageMarkdown for the inverse.
*/
}
+248
View File
@@ -0,0 +1,248 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of DocValidateMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IDocValidateMixin` fails to compile on drift.
export interface IDocValidateMixin {
}
export function DocValidateMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IDocValidateMixin> & TBase {
abstract class DocValidateMixin extends Base implements IDocValidateMixin {
/**
* Validate a URL string against a scheme allowlist for a given context.
*
* The markdown link path enforces safe schemes via TipTap, but the raw
* JSON path (updatePageJson) bypasses that so this is the sanitization
* choke point for ProseMirror JSON written directly by the caller.
*
* - "link": reject javascript:, vbscript:, data: (any scheme that can
* execute or smuggle script when the href is clicked).
* - "src": allow only http(s):, mailto:, /api/files paths, or a
* scheme-less relative/absolute path; reject
* javascript:/vbscript:/data:/file:.
*/
protected isSafeUrl(url: unknown, context: "link" | "src"): boolean {
if (typeof url !== "string") return false;
const trimmed = url.trim();
if (trimmed === "") return true; // empty href/src is harmless
// Extract a leading "scheme:" if present. A scheme must start with a
// letter and contain only letters/digits/+/-/. before the colon. Strip
// whitespace and ASCII control chars first so a tab/newline embedded in
// the scheme cannot smuggle a dangerous scheme past the check.
const cleaned = trimmed.replace(/[\s\x00-\x1f]+/g, "");
const schemeMatch = /^([a-zA-Z][a-zA-Z0-9+.-]*):/.exec(cleaned);
const scheme = schemeMatch ? schemeMatch[1].toLowerCase() : null;
const dangerous = new Set(["javascript", "vbscript", "data", "file"]);
if (context === "link") {
if (scheme === null) return true; // relative/anchor link is fine
// For links, data: is also blocked (can carry script payloads).
return !new Set(["javascript", "vbscript", "data"]).has(scheme);
}
// context === "src"
if (scheme === null) return true; // relative/absolute path (incl. /api/files)
if (dangerous.has(scheme)) return false;
return scheme === "http" || scheme === "https" || scheme === "mailto";
}
/**
* Recursively walk a ProseMirror doc and reject any unsafe URL on a link
* mark href or on a media node's src/url. Media nodes covered: image,
* attachment, video, plus embed (rendered as an iframe), youtube, drawio
* and excalidraw all of which carry a user-controlled URL that Docmost
* renders. Throws a clear error on the first violation. A max-depth guard
* turns an over-deep document into a clean error instead of a RangeError
* stack overflow.
*/
protected validateDocUrls(node: any, depth: number = 0): void {
const MAX_DEPTH = 200;
if (depth > MAX_DEPTH) {
throw new Error(
`document nesting exceeds the maximum depth of ${MAX_DEPTH}`,
);
}
if (!node || typeof node !== "object") return;
// Link marks on text nodes: validate the href.
if (Array.isArray(node.marks)) {
for (const mark of node.marks) {
if (mark && mark.type === "link" && mark.attrs) {
if (!this.isSafeUrl(mark.attrs.href, "link")) {
throw new Error(`unsafe link href rejected: "${mark.attrs.href}"`);
}
}
}
}
// Media nodes: validate src/url against the stricter src allowlist.
// embed renders as an iframe (highest risk); youtube/drawio/excalidraw
// likewise carry a user-controlled URL Docmost renders, so they get the
// same scheme check as image/attachment/video.
if (
node.type === "image" ||
node.type === "attachment" ||
node.type === "video" ||
node.type === "embed" ||
node.type === "youtube" ||
node.type === "drawio" ||
node.type === "excalidraw" ||
node.type === "audio" ||
node.type === "pdf"
) {
const attrs = node.attrs || {};
for (const key of ["src", "url"]) {
if (attrs[key] != null && !this.isSafeUrl(attrs[key], "src")) {
throw new Error(
`unsafe ${node.type} ${key} rejected: "${attrs[key]}"`,
);
}
}
}
if (Array.isArray(node.content)) {
for (const child of node.content) {
this.validateDocUrls(child, depth + 1);
}
}
}
/**
* Recursively validate the STRUCTURE of a ProseMirror node (reuses the
* recursion shape of validateDocUrls). Every node must be an object with a
* string `type`; when present, `content` must be an array, `marks` must be
* an array of objects each with a string `type`, and a text node's `text`
* must be a string. Throws a clear "invalid ProseMirror document" error on
* the first violation. A max-depth guard turns an over-deep document into a
* clean error instead of a RangeError stack overflow.
*/
protected validateDocStructure(node: any, depth: number = 0): void {
const MAX_DEPTH = 200;
if (depth > MAX_DEPTH) {
throw new Error(
`invalid ProseMirror document: nesting exceeds the maximum depth of ${MAX_DEPTH}`,
);
}
if (!node || typeof node !== "object" || typeof node.type !== "string") {
throw new Error(
"invalid ProseMirror document: every node must be an object with a string `type`",
);
}
if (
"text" in node &&
node.type === "text" &&
typeof node.text !== "string"
) {
throw new Error(
"invalid ProseMirror document: a text node must have a string `text`",
);
}
if (node.marks !== undefined) {
if (!Array.isArray(node.marks)) {
throw new Error(
"invalid ProseMirror document: `marks` must be an array",
);
}
for (const mark of node.marks) {
if (
!mark ||
typeof mark !== "object" ||
typeof mark.type !== "string"
) {
throw new Error(
"invalid ProseMirror document: every mark must be an object with a string `type`",
);
}
}
}
if (node.content !== undefined) {
if (!Array.isArray(node.content)) {
throw new Error(
"invalid ProseMirror document: `content` must be an array when present",
);
}
for (const child of node.content) {
this.validateDocStructure(child, depth + 1);
}
}
}
/**
* Pre-write SHAPE gate (#409). Walk the WHOLE node tree with the shared
* `findInvalidNode` and throw a rich, path-anchored error the instant a nested
* node has an absent/unknown `type` (or an unknown mark) the exact shape that
* otherwise surfaces DEEP in the Yjs encode as the cryptic
* `Unknown node type: undefined`, but only AFTER a collab session was opened
* and a page lock taken. Calling this BEFORE `getCollabTokenWithReauth` /
* `mutatePageContent` fails fast: no collab connection, no lock, deterministic
* message. `op` names the tool for the message prefix (e.g. "patchNode").
*
* `findInvalidNode` derives its "known type" set from the very same
* `docmostExtensions` the encode path uses, so a node this gate accepts is one
* the encoder will accept too.
*/
protected assertValidNodeShape(op: string, node: any): void {
const bad = findInvalidNode(node);
if (bad) {
throw new Error(`${op}: invalid node — ${bad.summary}`);
}
}
/**
* Replace page content with a raw ProseMirror JSON document (lossless) and/or
* update its title. Both `doc` and `title` are optional, but at least one must
* be supplied:
* - `doc` provided -> validate + full-overwrite the body (and update the
* title too when `title` is also given).
* - `doc` omitted, `title` given -> title-only update; the body is NOT
* touched/resent (no collab write happens).
* - neither given -> throws (nothing to update).
*/
}
return DocValidateMixin;
}
+707
View File
@@ -0,0 +1,707 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import { parseCells as parseDrawioCells } from "../lib/drawio-xml.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
prepareModel,
decodeDrawioSvg,
buildDrawioSvg,
mxHash,
normalizeXml,
countUserCells,
} from "../lib/drawio-xml.js";
import { renderDiagramShapes } from "../lib/drawio-preview.js";
import { applyElkLayout } from "../lib/drawio-layout.js";
import {
buildFromGraph,
type Graph,
type LayoutMode as GraphLayoutMode,
} from "../lib/drawio-graph.js";
import { applyCellOps, type CellOp } from "../lib/drawio-cell-ops.js";
import { mermaidToGraph } from "../lib/drawio-mermaid.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of DrawioMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IDrawioMixin` fails to compile on drift.
export interface IDrawioMixin {
drawioGet(pageId: string, node: string, format?: "xml" | "svg"): Promise<{ pageId: string; nodeId: string; format: "xml" | "svg"; content: string; meta: { attachmentId: string | null; title: string | null; width: number | null; height: number | null; cellCount: number; hash: string; }; }>;
drawioCreate(pageId: string, where: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }, xml: string, title?: string, layout?: "elk"): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; verify?: any; }>;
drawioUpdate(pageId: string, node: string, xml: string, baseHash: string, layout?: "elk"): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; verify?: any; }>;
drawioEditCells(pageId: string, node: string, operations: CellOp[], baseHash: string): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; verify?: any; }>;
drawioFromGraph(pageId: string, where: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }, graph: Graph, direction?: "LR" | "RL" | "TB" | "BT", preset?: string, layout?: GraphLayoutMode, node?: string): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; iconsResolved: number; iconsMissing: string[]; verify?: any; }>;
drawioFromMermaid(pageId: string, where: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }, mermaid: string, preset?: string): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; iconsResolved: number; iconsMissing: string[]; verify?: any; }>;
}
export function DrawioMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IDrawioMixin> & TBase {
abstract class DrawioMixin extends Base implements IDrawioMixin {
/**
* Resolve a drawio node on a page by `attrs.id` or `#<index>` and return the
* node plus its ref. Throws a clear error if the ref does not resolve to a
* drawio node.
*/
protected async resolveDrawioNode(
pageId: string,
node: string,
): Promise<{ node: any; ref: string }> {
const data = await this.getPageRaw(pageId);
const hit = getNodeByRef(
data.content ?? { type: "doc", content: [] },
node,
);
if (!hit) {
throw new Error(
`drawio: no node found for "${node}" on page ${pageId} (use the drawio node's attrs.id or "#<index>" from getOutline)`,
);
}
if (hit.type !== "drawio") {
throw new Error(
`drawio: node "${node}" on page ${pageId} is a ${hit.type}, not a drawio diagram`,
);
}
return { node: hit.node, ref: node };
}
/**
* Read a drawio diagram as mxGraph XML (default) or as the raw `.drawio.svg`.
* Runs the decode chain (base64/entity content= drawio file nested XML or
* pako-inflated compressed <diagram>). The returned `hash` is the
* optimistic-lock key for drawioUpdate.
*/
async drawioGet(
pageId: string,
node: string,
format: "xml" | "svg" = "xml",
): Promise<{
pageId: string;
nodeId: string;
format: "xml" | "svg";
content: string;
meta: {
attachmentId: string | null;
title: string | null;
width: number | null;
height: number | null;
cellCount: number;
hash: string;
};
}> {
await this.ensureAuthenticated();
const { node: drawio } = await this.resolveDrawioNode(pageId, node);
const attrs = drawio.attrs || {};
const src = attrs.src;
if (!src) {
throw new Error(
`drawio: node "${node}" on page ${pageId} has no src to read`,
);
}
const svg = await this.fetchAttachmentText(src);
const modelXml = decodeDrawioSvg(svg);
const meta = {
attachmentId: attrs.attachmentId ?? null,
title: attrs.title ?? null,
width: attrs.width != null ? Number(attrs.width) : null,
height: attrs.height != null ? Number(attrs.height) : null,
cellCount: countUserCells(modelXml),
hash: mxHash(modelXml),
};
return {
pageId,
nodeId: attrs.id ?? node,
format,
content: format === "svg" ? svg : normalizeXml(modelXml),
meta,
};
}
/**
* Create a drawio diagram from mxGraph XML: lint schematic SVG preview
* (pure TS) build the `.drawio.svg` (createDrawioSvg contract) create the
* attachment insert a `drawio` node before/after an anchor or appended.
* `xml` is a bare `<mxGraphModel>` or a list of `<mxCell>` (the server wraps
* it and adds the id=0/id=1 sentinels).
*/
async drawioCreate(
pageId: string,
where: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
xml: string,
title?: string,
layout?: "elk",
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (
!where ||
(where.position !== "before" &&
where.position !== "after" &&
where.position !== "append")
) {
throw new Error(
'drawioCreate: `where.position` must be one of "before", "after", "append"',
);
}
if (where.position === "before" || where.position === "after") {
const hasId =
typeof where.anchorNodeId === "string" && where.anchorNodeId.length > 0;
const hasText =
typeof where.anchorText === "string" && where.anchorText.length > 0;
if (hasId === hasText) {
throw new Error(
`drawioCreate: position "${where.position}" requires exactly one of anchorNodeId or anchorText`,
);
}
}
// Optional server-side ELK auto-layout: the model declares structure with
// rough coords, ELK computes the pixels (best-effort — returns the input
// unchanged on any layout failure).
const laidOutXml = layout === "elk" ? await applyElkLayout(xml) : xml;
// Pre-write pipeline (throws a structured DrawioLintError on any violation).
const prepared = prepareModel(laidOutXml);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
// NOTE: no `id` attribute is set here. The vendored `drawio` node schema
// (diagramAttributes) declares no `id`, so any block id would be silently
// dropped by PMNode.fromJSON on save and the returned handle would fail to
// resolve. The addressable handle is the node's "#<index>" (like image/table
// nodes), computed after the insert below.
const drawioNode: any = {
type: "drawio",
attrs: {
src: `/api/files/${att.id}/${att.fileName}`,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
align: "center",
},
};
if (title) drawioNode.attrs.title = title;
// Reuse the existing URL trust boundary (rejects unsafe src schemes).
this.validateDocUrls(drawioNode);
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let inserted = false;
let insertedIndex = -1;
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
inserted = false;
insertedIndex = -1;
const { doc: nd, inserted: ins } = insertNodeRelative(
liveDoc,
drawioNode,
where,
);
inserted = ins;
if (!inserted) return null; // anchor not found -> skip the write
// Locate the freshly-inserted node to derive its "#<index>" handle. The
// just-uploaded attachmentId is unique, so it identifies our node.
if (Array.isArray(nd.content)) {
insertedIndex = nd.content.findIndex(
(b: any) =>
b &&
b.type === "drawio" &&
b.attrs &&
b.attrs.attachmentId === att.id,
);
}
return nd;
},
);
if (!inserted) {
const anchorDesc = where.anchorNodeId
? `anchorNodeId "${where.anchorNodeId}"`
: `anchorText "${where.anchorText}"`;
throw new Error(
`drawioCreate: anchor not found (${anchorDesc}) on page ${pageId}. The diagram attachment ${att.id} is now an unreferenced orphan.`,
);
}
if (insertedIndex < 0) {
// The node was inserted nested (e.g. inside a callout/table cell via an
// anchor), where "#<index>" — which addresses only top-level blocks —
// cannot reference it. drawio nodes carry no persisted id, so there is no
// stable handle for a nested diagram.
throw new Error(
`drawioCreate: the diagram was inserted on page ${pageId} but not as a ` +
`top-level block, so it has no addressable "#<index>" handle. Anchor ` +
`on a top-level block (or append) so the diagram can be re-read.`,
);
}
// The returned handle is POSITIONAL ("#<index>"): valid for the immediate
// create -> get/update flow, but re-resolve via getOutline if the document
// structure changes (blocks added/removed before it shift the index).
const nodeId = `#${insertedIndex}`;
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
/**
* Full-replacement update of a drawio diagram. `baseHash` is MANDATORY: it is
* compared against the hash of the diagram's CURRENT XML (from drawioGet);
* any mismatch means a human or another agent edited the diagram after the
* read, so the write is refused with a conflict error. On success the new
* `.drawio.svg` is uploaded as a FRESH attachment (in-place byte overwrite is
* avoided some Docmost versions corrupt an attachment on overwrite, exactly
* as replaceImage documents) and the node is repointed with new dimensions.
*/
async drawioUpdate(
pageId: string,
node: string,
xml: string,
baseHash: string,
layout?: "elk",
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (typeof baseHash !== "string" || baseHash.length === 0) {
throw new Error(
"drawioUpdate: baseHash is mandatory — read the diagram with drawioGet first and pass back its meta.hash",
);
}
// Resolve the node and read the CURRENT diagram to enforce the optimistic
// lock before doing any write or upload.
const { node: drawio, ref } = await this.resolveDrawioNode(pageId, node);
const oldAttrs = drawio.attrs || {};
const oldSrc = oldAttrs.src;
// The returned handle is the caller-supplied reference. drawio nodes carry
// no persisted id, so `ref` (an "#<index>" or a rare legacy attrs.id) is the
// honest identifier to hand back.
const nodeId = oldAttrs.id ?? ref;
if (!oldSrc) {
throw new Error(
`drawioUpdate: node "${node}" on page ${pageId} has no src to compare against`,
);
}
const currentSvg = await this.fetchAttachmentText(oldSrc);
const currentHash = mxHash(decodeDrawioSvg(currentSvg));
if (currentHash !== baseHash) {
throw new Error(
`drawioUpdate: conflict — the diagram changed since it was read ` +
`(baseHash ${baseHash} != current ${currentHash}). Re-read it with drawioGet and retry.`,
);
}
// Optional server-side ELK auto-layout (best-effort; see drawioCreate).
const laidOutXml = layout === "elk" ? await applyElkLayout(xml) : xml;
// Pipeline for the new content (throws a structured DrawioLintError).
const prepared = prepareModel(laidOutXml);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = oldAttrs.title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
const newSrc = `/api/files/${att.id}/${att.fileName}`;
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let repointed = 0;
const repoint = (n: any) => {
n.attrs = {
...n.attrs,
src: newSrc,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
};
repointed++;
};
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
repointed = 0;
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
// Repoint ONLY the resolved node — never every node that happens to
// share this attachmentId (a copied diagram is two nodes with one
// attachmentId; keying on it would clobber both). Re-resolve the same
// handle against the live doc and walk to its exact position.
const hit = getNodeByRef(doc, ref);
if (!hit || hit.type !== "drawio") return null; // vanished/changed -> skip
let target: any = doc;
for (const idx of hit.path) {
if (!target || !Array.isArray(target.content)) {
target = null;
break;
}
target = target.content[idx];
}
if (!target || target.type !== "drawio") return null;
repoint(target);
if (repointed === 0) return null; // node vanished concurrently -> skip
return doc;
},
);
if (repointed === 0) {
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: [
...prepared.warnings,
"target drawio node was removed concurrently; uploaded attachment is unreferenced",
],
verify: mutation.verify,
};
}
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
// --- draw.io high-level semantic tools (issue #425) ---
/**
* ID-based targeted edits of an existing drawio diagram (add / update / delete
* cells) instead of resending the whole XML. Reads the CURRENT diagram, checks
* the optimistic lock (`baseHash` is MANDATORY, exactly as drawioUpdate), applies
* the operations to the parsed model (a `delete` CASCADES to container children
* and to every edge whose source/target is deleted), then runs the SAME #423
* pipeline as drawioUpdate (lint + quality warnings -> preview -> attachment ->
* repoint the node). Ids are stable so diffs stay meaningful across edits.
*/
// --- draw.io high-level semantic tools (issue #425) ---
/**
* ID-based targeted edits of an existing drawio diagram (add / update / delete
* cells) instead of resending the whole XML. Reads the CURRENT diagram, checks
* the optimistic lock (`baseHash` is MANDATORY, exactly as drawioUpdate), applies
* the operations to the parsed model (a `delete` CASCADES to container children
* and to every edge whose source/target is deleted), then runs the SAME #423
* pipeline as drawioUpdate (lint + quality warnings -> preview -> attachment ->
* repoint the node). Ids are stable so diffs stay meaningful across edits.
*/
async drawioEditCells(
pageId: string,
node: string,
operations: CellOp[],
baseHash: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (typeof baseHash !== "string" || baseHash.length === 0) {
throw new Error(
"drawioEditCells: baseHash is mandatory — read the diagram with drawioGet first and pass back its meta.hash",
);
}
if (!Array.isArray(operations) || operations.length === 0) {
throw new Error(
"drawioEditCells: operations must be a non-empty array of { op, ... }",
);
}
const { node: drawio, ref } = await this.resolveDrawioNode(pageId, node);
const oldAttrs = drawio.attrs || {};
const oldSrc = oldAttrs.src;
const nodeId = oldAttrs.id ?? ref;
if (!oldSrc) {
throw new Error(
`drawioEditCells: node "${node}" on page ${pageId} has no src to edit`,
);
}
const currentSvg = await this.fetchAttachmentText(oldSrc);
const currentModel = decodeDrawioSvg(currentSvg);
const currentHash = mxHash(currentModel);
if (currentHash !== baseHash) {
throw new Error(
`drawioEditCells: conflict — the diagram changed since it was read ` +
`(baseHash ${baseHash} != current ${currentHash}). Re-read it with drawioGet and retry.`,
);
}
// Apply the operations to the parsed model, then run the standard pipeline.
const editedModel = applyCellOps(currentModel, operations);
const prepared = prepareModel(editedModel);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = oldAttrs.title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
const newSrc = `/api/files/${att.id}/${att.fileName}`;
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let repointed = 0;
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
repointed = 0;
const doc =
liveDoc && liveDoc.type === "doc" ? liveDoc : { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
const hit = getNodeByRef(doc, ref);
if (!hit || hit.type !== "drawio") return null;
let target: any = doc;
for (const idx of hit.path) {
if (!target || !Array.isArray(target.content)) {
target = null;
break;
}
target = target.content[idx];
}
if (!target || target.type !== "drawio") return null;
target.attrs = {
...target.attrs,
src: newSrc,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
};
repointed++;
return doc;
},
);
if (repointed === 0) {
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: [
...prepared.warnings,
"target drawio node was removed concurrently; uploaded attachment is unreferenced",
],
verify: mutation.verify,
};
}
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
/**
* The main high-level tool: build a diagram from a SEMANTIC graph (nodes with
* a `kind`/`icon`, groups, edges) the model never supplies coordinates or
* style strings. The server resolves icons via the shape catalog (#424),
* assigns palette colors from the preset, runs ELK layered layout (honouring
* `direction` and the `layer`/`sameLayerAs`/`pinned` hints and compound groups),
* and assembles linter-clean XML, then inserts it through the SAME create
* pipeline as drawioCreate. `layout:"incremental"` is only meaningful when a
* target `node` is given (it preserves that diagram's existing coordinates and
* places only new cells); on a fresh insert it behaves like "full".
*/
async drawioFromGraph(
pageId: string,
where: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
graph: Graph,
direction?: "LR" | "RL" | "TB" | "BT",
preset?: string,
layout?: GraphLayoutMode,
node?: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
iconsResolved: number;
iconsMissing: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
// Direction/preset supplied as separate params override the graph fields so
// both the flat tool schema and an inline graph can set them.
const merged: Graph = {
...graph,
direction: direction ?? graph.direction,
preset: preset ?? graph.preset,
};
const mode: GraphLayoutMode = layout ?? "full";
// Incremental into an EXISTING node: read its coords so ELK preserves them,
// and keep the full existing model so incremental MERGES (never drops) any
// cell the new graph doesn't re-list.
let existingCoords: Map<string, { x: number; y: number }> | undefined;
let existingModelXml: string | undefined;
let editExisting = false;
let baseHash: string | undefined;
if (node && (mode === "incremental" || mode === "none")) {
const { node: drawio } = await this.resolveDrawioNode(pageId, node);
const src = (drawio.attrs || {}).src;
if (src) {
const svg = await this.fetchAttachmentText(src);
const model = decodeDrawioSvg(svg);
baseHash = mxHash(model);
existingModelXml = model;
existingCoords = new Map();
for (const c of parseDrawioCells(model)) {
if (c.vertex && c.geometry.x != null && c.geometry.y != null) {
existingCoords.set(c.id, { x: c.geometry.x, y: c.geometry.y });
}
}
editExisting = true;
}
}
const built = await buildFromGraph(
merged,
mode,
existingCoords,
existingModelXml,
);
if (editExisting && node && baseHash) {
// Re-target the existing diagram: replace it with the assembled model.
const res = await this.drawioUpdate(pageId, node, built.modelXml, baseHash);
return {
...res,
iconsResolved: built.iconsResolved,
iconsMissing: built.iconsMissing,
};
}
const res = await this.drawioCreate(pageId, where, built.modelXml);
return {
...res,
iconsResolved: built.iconsResolved,
iconsMissing: built.iconsMissing,
};
}
/**
* Convert a Mermaid `flowchart` to a redactable draw.io diagram via a PURE
* parser (no Electron / draw.io CLI): mermaid text -> graph-JSON -> the
* drawioFromGraph pipeline. Only `flowchart`/`graph` is supported (the most
* common wiki case); other diagram types throw a clear error so the model can
* fall back to drawioFromGraph.
*/
async drawioFromMermaid(
pageId: string,
where: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
mermaid: string,
preset?: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
iconsResolved: number;
iconsMissing: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
const graph = mermaidToGraph(mermaid);
if (preset) graph.preset = preset;
return this.drawioFromGraph(pageId, where, graph, graph.direction, graph.preset);
}
// --- Page history / diff / transform ---
/**
* List the saved versions (history snapshots) of a page, newest first.
* Docmost auto-snapshots on every save. Returns one cursor-paginated page of
* results: `{ items, nextCursor }`. The history record's id field is `id`.
*/
}
return DrawioMixin;
}
+165
View File
@@ -0,0 +1,165 @@
// Central REST error diagnostics (issues #437 + #450). SINGLE place that maps an
// axios error to the model-facing message. Extracted verbatim from client.ts;
// the constructor's response interceptor (see client/context.ts) routes every
// REST call through formatDocmostAxiosError so the whole surface is uniform.
import axios from "axios";
// --- Issue #437: central error diagnostics -------------------------------
// The agent only ever sees the thrown exception's `error.message`, so a failed
// tool must return an ACTIONABLE message (method, path, status, and the
// server's own validation text) instead of the opaque "Request failed with
// status code 400". These helpers + the response interceptor in the
// constructor are the single authoritative place that text is composed.
// Overall cap on the composed diagnostic message so the model context stays
// compact and a (whitelisted) server string can never blow up the text.
const ERROR_MESSAGE_CAP = 300;
// Only attempt to JSON.parse an arraybuffer body under this size: a larger
// binary body is never a JSON error envelope, so parsing it just wastes memory
// (fetchInternalFile uses responseType:"arraybuffer", so a failed file fetch
// carries the JSON error envelope as raw bytes here).
const ERROR_BUFFER_PARSE_CAP = 4096;
// Canonical 36-char UUID (8-4-4-4-12 hex). Deliberately version/variant-
// AGNOSTIC: the ids are UUIDv7 (e.g. 019f499a-9f8c-7d68-...), so only the
// canonical shape/length is enforced, not the version/variant nibble.
const FULL_UUID_RE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
/**
* Throw an actionable error BEFORE any network call when `value` is not a full
* canonical UUID. Absorbs #436: a truncated/short comment id used to reach the
* server and bounce back as an opaque 400/404 the agent could not self-correct;
* failing fast here names the exact fix.
*/
export function assertFullUuid(
tool: string,
param: string,
value: string,
): void {
if (typeof value !== "string" || !FULL_UUID_RE.test(value)) {
throw new Error(
`${tool}: '${param}' must be the FULL comment UUID (36 chars, e.g. ` +
`019f499a-9f8c-7d68-b7be-ce100d7c6c56), got '${value}'. Copy the id ` +
`verbatim from listComments / createComment output.`,
);
}
}
// Keep ONLY the pathname of a request (no host, no query string, no fragment)
// so the message never leaks a host or query params. Resolves a relative
// config.url against config.baseURL, then discards everything but the path.
function requestPath(config: any): string {
const rawUrl = typeof config?.url === "string" ? config.url : "";
const base =
typeof config?.baseURL === "string" ? config.baseURL : undefined;
try {
// A dummy base makes an absolute config.url parse too; its host is dropped.
return new URL(rawUrl, base ?? "http://localhost").pathname;
} catch {
// Malformed url: still strip any query/fragment manually.
return rawUrl.split(/[?#]/)[0] || rawUrl;
}
}
/**
* Compose the server-facing message from `error.response.data`, using ONLY the
* whitelisted `message`/`error` fields or the HTTP statusText. SECURITY: the
* raw response body, headers (Authorization!) and config are NEVER read here
* a string/HTML body (e.g. a proxy's 502 page) is deliberately dropped in
* favour of the statusText.
*/
function extractServerMessage(data: any, statusText: string): string {
// class-validator envelope: { message: string | string[], error?: string }.
if (
data &&
typeof data === "object" &&
!Buffer.isBuffer(data) &&
!(data instanceof ArrayBuffer)
) {
const msg = (data as any).message;
if (Array.isArray(msg)) {
const joined = msg.filter((m) => typeof m === "string").join("; ");
if (joined) return joined;
} else if (typeof msg === "string" && msg) {
return msg;
}
const err = (data as any).error;
if (typeof err === "string" && err) return err;
return statusText;
}
// Buffer / ArrayBuffer body: attempt a size-capped, guarded JSON.parse so a
// failed arraybuffer fetch still surfaces the server's validation text.
if (Buffer.isBuffer(data) || data instanceof ArrayBuffer) {
const buf = Buffer.isBuffer(data) ? data : Buffer.from(data);
if (buf.length > 0 && buf.length <= ERROR_BUFFER_PARSE_CAP) {
try {
return extractServerMessage(JSON.parse(buf.toString("utf8")), statusText);
} catch {
return statusText;
}
}
return statusText;
}
// A raw string / HTML body is never surfaced (may echo server internals).
return statusText;
}
/**
* Reformat an AxiosError's `.message` IN PLACE into an actionable diagnostic:
* `<METHOD> <path> failed (<status> <statusText>): <serverMessage>`
* or, when the request never got a response:
* `<METHOD> <path> failed: <code> (no response from server)`.
*
* Mutates the SAME error object (never a custom subclass) so the live
* axios.isAxiosError / error.response?.status / config._retry checks around the
* client keep working, and sets `_docmostFormatted` as a double-processing
* guard. A no-op on a non-axios or already-formatted error.
*/
export function formatDocmostAxiosError(error: any): void {
if (!error || error._docmostFormatted) return;
if (!axios.isAxiosError(error)) return;
const config: any = error.config ?? {};
const method =
typeof config.method === "string" ? config.method.toUpperCase() : "";
const methodPath = `${method} ${requestPath(config)}`.trim();
const response = error.response;
let message: string;
if (response) {
const statusText =
typeof response.statusText === "string" ? response.statusText : "";
const serverMessage = extractServerMessage(response.data, statusText);
message = `${methodPath} failed (${response.status} ${statusText}): ${serverMessage}`;
// Full body only to stderr under DEBUG (parity with downloadImage).
if (process.env.DEBUG) {
console.error(
"Docmost request failed; response body:",
JSON.stringify(response.data),
);
}
} else {
// No response at all (ECONNREFUSED / ETIMEDOUT / ECONNRESET / DNS / timeout).
// Use ONLY error.code, never the raw error.message: axios network messages
// embed host:port ("connect ECONNREFUSED 127.0.0.1:3000", "getaddrinfo
// ENOTFOUND host") and #437's invariant is that the host never reaches the
// model-visible message. code is set for essentially every real no-response
// error (ECONNREFUSED/ETIMEDOUT/ECONNRESET/ENOTFOUND/ECONNABORTED); the full
// native message still goes to stderr under DEBUG.
const reason = error.code ?? "network error";
message = `${methodPath} failed: ${reason} (no response from server)`;
if (process.env.DEBUG) {
console.error("Docmost request failed; no response:", error.message);
}
}
if (message.length > ERROR_MESSAGE_CAP) {
message = message.slice(0, ERROR_MESSAGE_CAP - 1) + "…";
}
error.message = message;
(error as any)._docmostFormatted = true;
}
+730
View File
@@ -0,0 +1,730 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import FormData from "form-data";
import axios, { AxiosInstance } from "axios";
import { basename, extname } from "path";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import { diffDocs, summarizeChange } from "../lib/diff.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Supported image types, kept as two lookup tables so both a local file
// extension and a remote Content-Type can be mapped to the same canonical set.
const EXT_TO_MIME: Record<string, string> = {
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".gif": "image/gif",
".webp": "image/webp",
".svg": "image/svg+xml",
};
const MIME_TO_EXT: Record<string, string> = {
"image/png": ".png",
"image/jpeg": ".jpg",
"image/gif": ".gif",
"image/webp": ".webp",
"image/svg+xml": ".svg",
};
// Public method surface of MediaMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IMediaMixin` fails to compile on drift.
export interface IMediaMixin {
uploadImage(pageId: string, url: string): any;
insertImage(pageId: string, url: string, opts?: { align?: "left" | "center" | "right"; alt?: string; replaceText?: string; afterText?: string; }): any;
replaceImage(pageId: string, oldAttachmentId: string, url: string, opts?: { align?: "left" | "center" | "right"; alt?: string }): any;
}
export function MediaMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IMediaMixin> & TBase {
abstract class MediaMixin extends Base implements IMediaMixin {
// --- Image upload / embedding ---
/** Map a Content-Type string to a supported MIME type, or null if unsupported. */
protected supportedImageMime(ct: string): string | null {
return MIME_TO_EXT[ct] ? ct : null;
}
/**
* Download a remote image from a caller-supplied URL and resolve its bytes,
* MIME and a filename.
*
* SSRF / RESOURCE TRUST BOUNDARY: the URL comes from the MCP caller and is
* fetched BY THE SERVER, so it must be guarded before and after the request.
* The guards mirror the local-file trust boundary in uploadImage:
* - scheme allowlist (http/https only) rejects file:, data:, ftp:, etc.,
* so the caller cannot use this path to read local files or other schemes;
* - a size cap enforced both via axios maxContentLength/maxBodyLength AND a
* post-download buffer.length re-check (defends against a missing/lying
* Content-Length), so a huge response cannot exhaust memory;
* - a 30s timeout. The timeout matters because replaceImage holds the
* per-page lock across this upload, so a hung download would wedge the
* lock for that page.
* We deliberately do NOT block private IP ranges: the MCP caller is already
* trusted to read arbitrary host files via the filePath path, so the marginal
* trust granted by fetching internal URLs is comparable, and blocking would
* break legitimate internal-image use.
*/
protected async fetchRemoteImage(
url: string,
maxBytes: number,
): Promise<{ buffer: Buffer; mime: string; fileName: string }> {
// Scheme allowlist first — cheapest guard, and rejects non-http(s) schemes
// (file:, data:, ftp:, ...) before any network request is made.
let parsed: URL;
try {
parsed = new URL(url);
} catch (e: any) {
throw new Error(`Invalid image URL "${url}": ${e.message}`);
}
if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
throw new Error(
`unsupported image URL scheme "${parsed.protocol}"; only http and https are allowed`,
);
}
let response;
try {
response = await axios.get(url, {
responseType: "arraybuffer",
timeout: 30000,
maxContentLength: maxBytes,
maxBodyLength: maxBytes,
headers: { Accept: "image/*" },
});
} catch (error) {
// Keep the thrown message free of the raw response body (it may echo
// server internals); surface only status/statusText. The full body is
// logged under DEBUG for diagnostics.
if (axios.isAxiosError(error)) {
if (process.env.DEBUG) {
console.error(
"Image download failed; response body:",
JSON.stringify(error.response?.data),
);
}
throw new Error(
`Image download failed for "${url}": ${error.response?.status ?? ""} ${error.response?.statusText ?? error.message}`.trim(),
);
}
throw error;
}
// axios returns an ArrayBuffer for responseType: "arraybuffer".
const buffer = Buffer.from(response.data);
// Re-check the size: maxContentLength relies on Content-Length, which may be
// absent or lie, so guard against the actual byte count too.
if (buffer.length === 0) {
throw new Error(`Empty image response from "${url}"`);
}
if (buffer.length > maxBytes) {
throw new Error(
`Image too large: ${buffer.length} bytes exceeds the ${maxBytes}-byte cap`,
);
}
// Resolve MIME: prefer the response Content-Type (strip any "; charset=..."
// parameter, lowercase, trim) mapped through the supported set; if the
// header is generic/missing/unsupported, fall back to the URL path
// extension via the existing extension->MIME logic.
const rawCt = response.headers?.["content-type"];
let mime: string | null = null;
if (typeof rawCt === "string" && rawCt.length > 0) {
const ct = rawCt.split(";")[0].trim().toLowerCase();
mime = this.supportedImageMime(ct);
}
if (!mime) {
// Fall back to the URL path extension. Use the pathname so the query
// string never contaminates the extension lookup.
const ext = extname(parsed.pathname).toLowerCase();
mime = EXT_TO_MIME[ext] ?? null;
}
if (!mime) {
throw new Error(
`cannot determine supported image type for "${url}"; supported: png, jpg, jpeg, gif, webp, svg`,
);
}
// Build a filename from the URL path basename (ignore the query string),
// defaulting to "image" when empty, and ensure it ends with the canonical
// extension for the resolved MIME (append it when missing/mismatched).
const canonicalExt = MIME_TO_EXT[mime];
let fileName = basename(parsed.pathname) || "image";
if (extname(fileName).toLowerCase() !== canonicalExt) {
fileName += canonicalExt;
}
return { buffer, mime, fileName };
}
/** Build a Docmost ProseMirror image node from an uploaded attachment. */
protected buildImageNode(
att: { id: string; fileName: string; fileSize?: number },
align?: "left" | "center" | "right",
alt?: string,
): any {
// Clean file URL, matching Docmost's native behaviour. No cache-busting
// query: the server serves the bare URL correctly, and replacement creates
// a new attachment id (a new URL) which busts caches naturally.
const src = `/api/files/${att.id}/${att.fileName}`;
const node: any = {
type: "image",
attrs: {
src,
attachmentId: att.id,
// Default to null when the server omits fileSize so the attr is never
// undefined (undefined would be dropped on serialization / break the
// ProseMirror image schema which expects size present).
size: att.fileSize ?? null,
align: align || "center",
width: null,
},
};
if (alt) node.attrs.alt = alt;
return node;
}
/**
* Download a remote image from an http(s) URL and upload it as an attachment
* of a page, returning the attachment metadata plus a ready-to-insert
* ProseMirror image node. Local file paths are intentionally not supported:
* the MCP caller is a remote AI with no access to this server's filesystem.
*/
async uploadImage(pageId: string, url: string) {
await this.ensureAuthenticated();
const MAX_IMAGE_BYTES = 20 * 1024 * 1024; // 20 MiB
// Fetch + validate the remote image (scheme allowlist, size cap, timeout).
// See fetchRemoteImage for the SSRF / resource trust boundary.
const fetched = await this.fetchRemoteImage(url, MAX_IMAGE_BYTES);
const fileBuffer = fetched.buffer;
const mime = fetched.mime;
const fileName = fetched.fileName;
// Build a FRESH FormData for every send attempt. A FormData body is a
// single-use stream that is CONSUMED on the first send, so it cannot be
// replayed by this.client's response interceptor (replaying a consumed
// stream fails with 'socket hang up'). Multipart re-auth is therefore done
// here with bare axios and an explicit one-shot 401/403 retry that rebuilds
// the body. Field order matters: text fields must precede the file part so
// the server reads them; the server always generates a fresh attachment id.
const buildForm = () => {
const form = new FormData();
form.append("pageId", pageId);
form.append("file", fileBuffer, {
filename: fileName,
contentType: mime,
});
return form;
};
// Local name distinct from the `url` parameter (the source image URL): this
// is the /files/upload endpoint we POST the multipart body to.
const uploadUrl = `${this.apiUrl}/files/upload`;
let response;
try {
// Call buildForm() ONCE per attempt and reuse the instance for both
// getHeaders() and the body so the Content-Type boundary matches the body.
const form = buildForm();
// Read the Authorization header from this.client's defaults (set by
// login(), only ever deleted — never set to null) instead of building
// `Bearer ${this.token}`: a concurrent JSON 401 can null this.token
// mid-flight, which would otherwise produce a literal "Bearer null".
// ensureAuthenticated() above guarantees login() ran, so the default
// header exists here. A 60s timeout keeps a hung upload from wedging the
// per-page lock (replaceImage holds withPageLock across this call).
response = await axios.post(uploadUrl, form, {
headers: {
...form.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} catch (error) {
// On an expired-token auth error, re-login and retry exactly once with a
// freshly-rebuilt FormData (the previous one was already consumed).
if (
axios.isAxiosError(error) &&
(error.response?.status === 401 || error.response?.status === 403)
) {
await this.login();
const form2 = buildForm();
response = await axios.post(uploadUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} else if (axios.isAxiosError(error)) {
// Keep the thrown message free of the raw response body (it may echo
// request data or server internals); surface only status/statusText.
// The full body is logged under DEBUG for diagnostics.
if (process.env.DEBUG) {
console.error(
"Image upload failed; response body:",
JSON.stringify(error.response?.data),
);
}
throw new Error(
`Image upload failed: ${error.response?.status} ${error.response?.statusText}`,
);
} else {
throw error;
}
}
// The attachment may arrive bare or wrapped in a { data } envelope.
const att = response.data?.data ?? response.data;
if (!att?.id || !att?.fileName) {
throw new Error(
"Unexpected /files/upload response: " + JSON.stringify(response.data),
);
}
// Some Docmost versions omit fileSize from the upload response. Fall back
// to the fetched byte length (the bytes we just uploaded) so callers never
// get an undefined size.
const resolvedSize = att.fileSize ?? fileBuffer.length;
return {
attachmentId: att.id,
fileName: att.fileName,
fileSize: resolvedSize,
src: `/api/files/${att.id}/${att.fileName}`,
imageNode: this.buildImageNode({ ...att, fileSize: resolvedSize }),
};
}
/**
* Upload an image from a web (http/https) URL and insert it into a page in
* one step.
* By default the image is appended at the end. With replaceText, the first
* top-level block whose text contains the string is replaced; with afterText,
* the image is inserted right after the first matching block. All other
* block ids are preserved (only one top-level block is added or swapped).
*/
async insertImage(
pageId: string,
url: string,
opts: {
align?: "left" | "center" | "right";
alt?: string;
replaceText?: string;
afterText?: string;
} = {},
) {
const up = await this.uploadImage(pageId, url);
// Reuse the node from uploadImage (clean /api/files/<id>/<file> src), then
// apply align/alt onto a shallow attrs copy.
const node: any = { ...up.imageNode, attrs: { ...up.imageNode.attrs } };
if (opts.align) node.attrs.align = opts.align;
if (opts.alt) node.attrs.alt = opts.alt;
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// uploadImage /files/upload call above keeps the agent-supplied id.
const pageUuid = await this.resolvePageId(pageId);
// Recursively collect the plain text of a top-level block.
const blockText = (n: any): string => {
let out = "";
if (n.type === "text") out += n.text || "";
for (const child of n.content || []) out += blockText(child);
return out;
};
// Insert into the LIVE synced document, not the debounced REST snapshot, so
// concurrent edits/comments/images are preserved and parallel insertImage
// calls (serialized by the per-page lock) each see the previous insertion.
let placement: "replaced" | "after" | "appended" | undefined;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
if (opts.replaceText) {
// Ambiguity guard (mirrors editPageText): count matching top-level
// blocks first, so a non-unique fragment cannot silently replace the
// wrong block (e.g. text that also appears inside a callout/table).
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.replaceText!),
);
if (matches.length === 0) {
throw new Error(`replaceText not found: "${opts.replaceText}"`);
}
if (matches.length > 1) {
throw new Error(
`replaceText "${opts.replaceText}" matches ${matches.length} blocks; use a longer unique fragment`,
);
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.replaceText!),
);
// Data-loss guard: replaceText swaps the WHOLE top-level block, so if
// the fragment only appears nested inside a container (table, callout,
// list, blockquote) the entire structure would be destroyed. Refuse
// when the matched block is a container rather than a leaf
// paragraph/heading and point the caller at a safer tool.
const CONTAINER_TYPES = new Set([
"table",
"callout",
"bulletList",
"orderedList",
"taskList",
"blockquote",
]);
const matchedBlock = doc.content[idx];
if (matchedBlock && CONTAINER_TYPES.has(matchedBlock.type)) {
throw new Error(
`replaceText matched a ${matchedBlock.type} container block; replacing it would destroy the whole structure. ` +
`Use afterText to insert near it, or updatePageJson for surgical edits.`,
);
}
doc.content.splice(idx, 1, node);
placement = "replaced";
} else if (opts.afterText) {
// Ambiguity guard (mirrors editPageText): refuse a non-unique fragment.
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.afterText!),
);
if (matches.length === 0) {
throw new Error(`afterText not found: "${opts.afterText}"`);
}
if (matches.length > 1) {
throw new Error(
`afterText "${opts.afterText}" matches ${matches.length} blocks; use a longer unique fragment`,
);
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.afterText!),
);
doc.content.splice(idx + 1, 0, node);
placement = "after";
} else {
doc.content.push(node);
placement = "appended";
}
return doc;
},
);
return {
success: true,
pageId,
attachmentId: up.attachmentId,
src: up.src,
placement,
verify: mutation.verify,
};
}
/**
* Replace an existing image in a page with a new image fetched from a web
* (http/https) URL. Uploads the new file as a brand-new attachment, which
* yields a fresh clean URL that both renders correctly and busts browser
* caches (the URL changed). Finds every image node
* whose attrs.attachmentId === oldAttachmentId (recursively, incl. nodes nested
* in callouts/tables) and repoints its src/attachmentId/size, preserving
* comments, alignment and alt. Operates on the live collab document so comments
* and concurrent edits are preserved. Throws if no matching image is found.
*
* The OLD attachment is left in place as an unreferenced orphan: Docmost
* exposes NO HTTP API to delete a single content attachment (verified against
* the attachment controller/service and by probing the live API deletion
* happens only by cascade when the page, space or user is removed). This is the
* same outcome as Docmost's own editor when an image is removed/replaced.
* In-place byte overwrite is deliberately NOT used because some Docmost
* versions corrupt the attachment (HTTP 500) when its bytes are overwritten.
*/
async replaceImage(
pageId: string,
oldAttachmentId: string,
url: string,
opts: { align?: "left" | "center" | "right"; alt?: string } = {},
) {
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// page lock must ALSO key on the UUID so this operation serializes against
// other writes to the same page (mutatePageContent now locks by the resolved
// UUID too); locking by the raw slugId here would desync the mutex key and
// reopen the TOCTOU/orphan-attachment window the lock closes. uploadImage
// keeps the agent-supplied id (it hits REST, not the collab doc).
const pageUuid = await this.resolvePageId(pageId);
// Hold ONE per-page lock for the WHOLE operation (scan -> upload -> write).
// Previously the scan and the write were two separate mutatePageContent
// calls, each acquiring + releasing the lock, with the upload happening in
// the UNLOCKED gap between them. A concurrent op could interleave there: it
// could remove the target image so the write pass matches nothing, leaving
// the freshly-uploaded attachment as an un-deletable orphan (Docmost has no
// API to delete a single content attachment). Acquiring the lock once and
// using the non-locking collab helper inside (the per-page mutex is NOT
// reentrant, so the self-locking mutatePageContent would deadlock here)
// closes that TOCTOU window. uploadImage hits /files/upload over plain HTTP
// and does not touch the page lock, so it is safe to call while held.
return withPageLock(pageUuid, async () => {
// STEP 1: read-only live check. Scan the live document for any image node
// matching oldAttachmentId BEFORE uploading anything, so a wrong/stale id
// throws without ever creating an orphan attachment.
let matchFound = false;
const scan = (nodes: any[]) => {
for (const node of nodes) {
if (!node) continue;
if (
node.type === "image" &&
node.attrs &&
node.attrs.attachmentId === oldAttachmentId
) {
matchFound = true;
}
if (Array.isArray(node.content)) scan(node.content);
}
};
await this.mutateLiveContentUnlocked(pageUuid, collabToken, (liveDoc) => {
matchFound = false; // reset per-transform (collab may retry the read).
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (Array.isArray(doc.content)) scan(doc.content);
return null; // read-only: never write on the check pass.
});
if (!matchFound) {
throw new Error(
`replaceImage: no image with attachmentId "${oldAttachmentId}" found on page ${pageId}`,
);
}
// STEP 2: a match exists — upload the new file as a FRESH attachment (new
// id, new clean URL) and repoint every matching node in a second pass.
// Still inside the SAME lock, so no other op can have changed the page
// since the scan.
const up = await this.uploadImage(pageId, url);
let replaced = 0;
// Swap the source of one image node, preserving align/alt/title/geometry.
const repoint = (node: any) => {
node.attrs = {
...node.attrs,
src: up.src,
attachmentId: up.attachmentId,
// Default to null when fileSize is unknown so the attr is never
// undefined.
size: up.fileSize ?? null,
};
if (opts.align) node.attrs.align = opts.align;
if (opts.alt !== undefined) node.attrs.alt = opts.alt;
replaced++;
};
// Recursively repoint every image node (incl. ones nested in callouts/tables).
const walk = (nodes: any[]) => {
for (const node of nodes) {
if (!node) continue;
if (
node.type === "image" &&
node.attrs &&
node.attrs.attachmentId === oldAttachmentId
) {
repoint(node);
}
if (Array.isArray(node.content)) walk(node.content);
}
};
const mutation = await this.mutateLiveContentUnlocked(
pageUuid,
collabToken,
(liveDoc) => {
// Reset per-transform so collab retries recompute cleanly (no double-count).
replaced = 0;
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
walk(doc.content);
if (replaced === 0) return null; // no match -> skip the write entirely
return doc;
},
);
// KNOWN LIMITATION: a same-count image SRC swap (image count unchanged, no
// text/mark change) may still report verify.changed === false, because the
// text+marks+integrity-count model in summarizeChange does not inspect
// image `src`/attachmentId attributes. That is acceptable here — the
// replace is confirmed by `replaced` below, and verify is supplementary.
if (replaced === 0) {
// The pass-1 SCAN found the target (matchFound was true) and we already
// uploaded the new attachment, but pass-2 matched nothing — a concurrent
// editor must have removed the node between the two passes. Do NOT throw
// here (that would leak the just-uploaded attachment AND report failure);
// instead report success with the upload flagged as an unreferenced
// orphan so the caller knows. (The early throw above still covers the
// case where pass-1 finds nothing, before any upload happens.)
return {
success: true,
replaced: 0,
pageId,
oldAttachmentId,
newAttachmentId: up.attachmentId,
src: up.src,
orphanedAttachmentId: up.attachmentId,
warning:
"target image was removed concurrently; uploaded attachment is unreferenced",
verify: mutation.verify,
};
}
return {
success: true,
pageId,
replaced,
oldAttachmentId,
newAttachmentId: up.attachmentId,
src: up.src,
verify: mutation.verify,
};
});
}
// --- draw.io diagrams (issue #423) ---
/**
* Upload a ready-made byte buffer as a page attachment via the same
* multipart /files/upload endpoint uploadImage uses. Split out as its own
* (overridable) seam so drawioCreate/update can upload the generated
* `.drawio.svg` without going through the URL-fetch path, and so tests can
* stub the network. Mirrors uploadImage's fresh-FormData + one-shot 401/403
* re-auth handling (a FormData body is single-use, so it must be rebuilt per
* attempt).
*/
// --- draw.io diagrams (issue #423) ---
/**
* Upload a ready-made byte buffer as a page attachment via the same
* multipart /files/upload endpoint uploadImage uses. Split out as its own
* (overridable) seam so drawioCreate/update can upload the generated
* `.drawio.svg` without going through the URL-fetch path, and so tests can
* stub the network. Mirrors uploadImage's fresh-FormData + one-shot 401/403
* re-auth handling (a FormData body is single-use, so it must be rebuilt per
* attempt).
*/
protected async uploadAttachmentBuffer(
pageId: string,
buffer: Buffer,
fileName: string,
mime: string,
): Promise<{ id: string; fileName: string; fileSize: number }> {
await this.ensureAuthenticated();
const buildForm = () => {
const form = new FormData();
form.append("pageId", pageId);
form.append("file", buffer, { filename: fileName, contentType: mime });
return form;
};
const uploadUrl = `${this.apiUrl}/files/upload`;
let response;
try {
const form = buildForm();
response = await axios.post(uploadUrl, form, {
headers: {
...form.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} catch (error) {
if (
axios.isAxiosError(error) &&
(error.response?.status === 401 || error.response?.status === 403)
) {
await this.login();
const form2 = buildForm();
response = await axios.post(uploadUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} else if (axios.isAxiosError(error)) {
if (process.env.DEBUG) {
console.error(
"Attachment upload failed; response body:",
JSON.stringify(error.response?.data),
);
}
throw new Error(
`Attachment upload failed: ${error.response?.status} ${error.response?.statusText}`,
);
} else {
throw error;
}
}
const att = response.data?.data ?? response.data;
if (!att?.id || !att?.fileName) {
throw new Error(
"Unexpected /files/upload response: " + JSON.stringify(response.data),
);
}
return {
id: att.id,
fileName: att.fileName,
fileSize: att.fileSize ?? buffer.length,
};
}
/**
* Fetch a stored `.drawio.svg` attachment as text. Overridable seam over
* fetchInternalFile (the authed loopback fetch, which also rejects any
* traversal/SSRF src) so drawioGet/update can read the current diagram and
* tests can stub the bytes.
*/
protected async fetchAttachmentText(src: string): Promise<string> {
const { buffer } = await this.fetchInternalFile(src);
return buffer.toString("utf-8");
}
/**
* Resolve a drawio node on a page by `attrs.id` or `#<index>` and return the
* node plus its ref. Throws a clear error if the ref does not resolve to a
* drawio node.
*/
}
return MediaMixin;
}
+700
View File
@@ -0,0 +1,700 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
importMarkdownFragment,
canBeDocChild,
findUnrepresentableTableAttrs,
} from "../lib/markdown-fragment.js";
import {
applyTextEdits,
TextEdit,
TextEditResult,
TextEditFailure,
} from "../lib/json-edit.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
import { normalizeAndMergeFootnotes } from "../lib/footnote-normalize-merge.js";
// Public method surface of NodesWriteMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements INodesWriteMixin` fails to compile on drift.
export interface INodesWriteMixin {
updatePageJson(pageId: string, doc?: any, title?: string): any;
editPageText(pageId: string, edits: TextEdit[]): any;
patchNode(pageId: string, nodeId: string, input: { markdown?: string; node?: any }): any;
insertNode(pageId: string, input: { markdown?: string; node?: any }, opts: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }): any;
deleteNode(pageId: string, nodeId: string): any;
}
export function NodesWriteMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & INodesWriteMixin> & TBase {
abstract class NodesWriteMixin extends Base implements INodesWriteMixin {
/**
* Replace page content with a raw ProseMirror JSON document (lossless) and/or
* update its title. Both `doc` and `title` are optional, but at least one must
* be supplied:
* - `doc` provided -> validate + full-overwrite the body (and update the
* title too when `title` is also given).
* - `doc` omitted, `title` given -> title-only update; the body is NOT
* touched/resent (no collab write happens).
* - neither given -> throws (nothing to update).
*/
async updatePageJson(pageId: string, doc?: any, title?: string) {
await this.ensureAuthenticated();
// Title-only / no-op handling: when no document is supplied, do NOT write
// the body. Update the title if one was given; otherwise there is nothing
// to do, so fail loudly rather than silently no-op.
if (doc == null) {
if (!title) {
throw new Error(
"updatePageJson: nothing to update (provide content and/or title)",
);
}
await this.client.post("/pages/update", { pageId, title });
return {
success: true,
modified: true,
message: "Page title updated (content left unchanged).",
pageId,
};
}
// Validate the document shape before a full overwrite: a malformed doc
// would otherwise silently corrupt the page (full-overwrite is the
// documented behaviour; no optimistic-concurrency is applied here).
if (
typeof doc !== "object" ||
doc.type !== "doc" ||
!Array.isArray(doc.content)
) {
throw new Error(
'content must be a ProseMirror document ({"type":"doc","content":[...]}) ' +
"where content is an array of nodes each having a string `type`",
);
}
// Recurse the WHOLE document so a malformed nested node (e.g. a node with a
// non-string type, a non-array content/marks, or a text node missing its
// string text) is rejected up front rather than silently corrupting the
// page on overwrite.
this.validateDocStructure(doc);
// #409: beyond the string-`type` check above, reject a nested node whose
// `type` is a string but NOT a known Docmost schema node (a typo/unknown
// block) — the same `Unknown node type` the encoder throws — with a rich,
// path-anchored message, still BEFORE any collab connection.
this.assertValidNodeShape("updatePageJson", doc);
// Sanitize URLs before writing. This closes the JSON-path bypass: unlike
// the markdown link path (which TipTap sanitizes), raw JSON could otherwise
// inject javascript:/data: link hrefs or media srcs straight into the doc.
this.validateDocUrls(doc);
// Canonicalize footnotes (idempotent): an agent-authored JSON doc cannot
// leave footnotes out of order, orphaned, or in multiple lists — the bottom
// list + numbering are always derived from reference order. No-op when the
// footnotes are already canonical.
// #419: normalize + merge glyph-forked definitions before canonicalizing.
doc = normalizeAndMergeFootnotes(doc);
doc = canonicalizeFootnotes(doc);
// Write the BODY first, then the title (#159 split-brain): a failed body
// write (e.g. persist timeout) must not leave a new title over the old body.
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
const mutation = await this.replacePage(
pageUuid,
doc,
collabToken,
this.apiUrl,
);
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
return {
success: true,
modified: true,
message: "Page content replaced from ProseMirror JSON.",
pageId,
verify: mutation.verify,
};
}
/**
* AUTHOR-INLINE footnote insertion. The agent supplies only WHERE
* (`anchorText`, a snippet of body text to attach the marker after) and WHAT
* (`text`, the footnote content as markdown). Numbering and the bottom
* `footnotesList` are derived deterministically server-side
* (`insertInlineFootnote` -> `canonicalizeFootnotes`): the agent never sees,
* assigns, or edits a footnote number or the list, so it CANNOT desync.
*
* Content DEDUP: when an existing definition has the same content, its id is
* reused (one number, one definition, several references). The write is atomic
* via `mutatePageContent` (single-writer, page-locked); if the anchor text is
* not found the transform aborts with a clear error and no write happens.
*/
/**
* Surgical text edits: find/replace inside text nodes of the live
* document. Preserves all block ids, marks, callouts and tables.
*/
async editPageText(pageId: string, edits: TextEdit[]) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Apply the edits against the LIVE synced document, not the debounced REST
// snapshot, so concurrent human edits/comments are preserved. applyTextEdits
// records per-edit match problems in `failed` instead of throwing, and
// applies whatever it can; we abort the write only when nothing applied.
let results: TextEditResult[] | undefined;
let failed: TextEditFailure[] | undefined;
// Whether we actually wrote new content. Set inside the transform: a
// degenerate edit (e.g. find === replace, or a batch that nets to no change)
// can "apply" yet leave the document byte-for-byte identical, in which case
// we must NOT write (no spurious history version) and must not claim a write
// happened.
let wrote = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
wrote = false;
const r = applyTextEdits(liveDoc, edits);
results = r.results;
failed = r.failed;
// Nothing applied -> abort the write (mutatePageContent treats a null
// return from the transform as "write nothing").
if (r.results.length === 0) return null;
// Edits "applied" but produced an identical document: skip the write so
// no new history version is created. Stable structural comparison via
// JSON.stringify (both docs come from the same deep-copied source, so
// key order is stable).
if (JSON.stringify(r.doc) === JSON.stringify(liveDoc)) return null;
wrote = true;
return r.doc;
},
);
if ((results?.length ?? 0) === 0 && (failed?.length ?? 0) > 0) {
// No edit applied: surface an aggregated, actionable error so the caller
// does not mistake a no-op for a partial success.
throw new Error(
"editPageText: no edits were applied (nothing written). " +
failed!.map((f) => `"${f.find}": ${f.reason}`).join("; "),
);
}
// Edits matched but produced no content change (identical document): report
// a successful no-op — NOT a failure — and do not falsely claim a write.
if (!wrote) {
return {
success: true,
pageId,
applied: results,
failed,
message: "No changes written (edits produced identical content).",
verify: mutation.verify,
};
}
const result: any = {
success: true,
pageId,
applied: results,
failed,
message:
(failed?.length ?? 0)
? `Applied ${results?.length ?? 0} edit(s); ${failed!.length} failed (see failed[]). Node ids and formatting preserved.`
: "Text edits applied (node ids and formatting preserved).",
verify: mutation.verify,
};
// If any applied edit matched only after stripping markdown (the
// normalized fallback), warn that editPageText preserved existing marks
// and did NOT change formatting — so a caller who intended a formatting
// change is pointed at patchNode.
if (results?.some((r) => r.normalized === true)) {
result.warning =
"Some edits matched only after stripping markdown from your find string; " +
"editPageText preserved existing marks (it did not change bold/strike/etc.). " +
"If you intended a formatting change, use patchNode.";
}
return result;
}
/**
* Replace the block whose attrs.id === nodeId. Operates on the LIVE collab
* document so comments and concurrent edits are preserved.
*
* Exactly one of `input.markdown` / `input.node` (#413):
* - `markdown` (RECOMMENDED): the block is rewritten from a canonical markdown
* fragment. The fragment may import to N blocks (a "1 -> N" splice: rewrite a
* whole section in one call). The FIRST resulting block INHERITS the target's
* `attrs.id` (so an existing comment anchoring the block by id survives); the
* rest get FRESH ids. `^[...]` footnotes in the fragment are first-class:
* their definitions merge into the page's TAIL footnote list (content-key
* dedup + canonicalize), same machinery insertFootnote uses. REJECTED when
* the TARGET block carries a table-cell attribute markdown cannot represent
* (colspan/rowspan/colwidth/background) use the table tools or `node`.
* - `node`: a raw ProseMirror node for precise attr/mark work. The replacement
* keeps the target id (if `node.attrs.id` is missing it is set to nodeId).
*
* #159 ambiguous-id semantics are unchanged: 0 matches -> "no node"; >1 matches
* -> "ambiguous, refused" (nothing written), on BOTH paths the markdown path
* runs a dry `replaceNodeById` count first, so a duplicated id never splices.
*/
async patchNode(
pageId: string,
nodeId: string,
input: { markdown?: string; node?: any },
) {
await this.ensureAuthenticated();
// XOR: exactly one of markdown / node. Both optional in the schema; the
// runtime enforces the recommendation ("markdown for prose, node for fine
// work") without letting an ambiguous both-or-neither call through.
const hasMd =
input != null &&
typeof input.markdown === "string" &&
input.markdown.trim() !== "";
const hasNode = input != null && input.node != null;
if (hasMd === hasNode) {
throw new Error(
"patchNode: provide exactly one of `markdown` (recommended, for prose) " +
"or `node` (a raw ProseMirror node, for precise attr/mark work)",
);
}
if (hasMd) {
return this.patchNodeMarkdown(pageId, nodeId, input.markdown as string);
}
return this.patchNodeJson(pageId, nodeId, input.node);
}
/**
* patchNode with a raw ProseMirror `node` (the pre-#413 behavior). Replaces
* EVERY node whose attrs.id === nodeId; the swapped-in node keeps the target
* id. #159 ambiguity refused. Split out so the markdown path can reuse the
* shared collab/guard plumbing without a giant branch.
*/
protected async patchNodeJson(pageId: string, nodeId: string, node: any) {
if (!node || typeof node !== "object" || typeof node.type !== "string") {
throw new Error(
"patchNode: `node` must be an object with a string `type`",
);
}
// Preserve the block id WITHOUT mutating the caller's object: build a local
// copy whose attrs.id === nodeId (so the swapped-in node keeps the id of the
// node it replaces).
const target = {
...node,
attrs: {
...(node.attrs && typeof node.attrs === "object" ? node.attrs : {}),
},
};
if (target.attrs.id == null) {
target.attrs.id = nodeId;
}
// #409: fail fast on a malformed node SHAPE (a nested child with an
// absent/unknown `type`, e.g. a text leaf written as `{"text":"foo"}` with
// no `"type":"text"`) BEFORE opening a collab session or taking the page
// lock — the root-only `typeof node.type === "string"` check above never
// sees nested children, and the encoder's `Unknown node type: undefined`
// would otherwise only surface after the connection.
this.assertValidNodeShape("patchNode", target);
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track the replacement count in an outer var, reset per-transform, so a
// collab retry recomputes it cleanly (mirrors replaceImage's pattern).
let replaced = 0;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
replaced = 0;
const { doc: nd, replaced: r } = replaceNodeById(
liveDoc,
nodeId,
target,
);
replaced = r;
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS: Docmost
// duplicates block ids on copy/paste (and copyPageContent writes them
// verbatim), so replacing "the node with id X" would silently clobber
// EVERY duplicate (#159). Refuse: skip the write and throw below so the
// model re-targets with a more specific anchor instead of corrupting the
// page. Only an unambiguous single match is written.
if (replaced !== 1) return null;
return nd;
},
);
// 0 -> "no node"; >1 -> "ambiguous, refused" (the transform already skipped
// the write for any count !== 1). Single shared guard (#159, #185 review).
assertUnambiguousMatch("patchNode", "replace", replaced, nodeId, pageId);
return { success: true, replaced, nodeId, verify: mutation.verify };
}
/**
* patchNode with a MARKDOWN fragment (#413). Imports the fragment through the
* canonical importer, then 1 -> N splices the resulting blocks in place of the
* target block on the LIVE collab doc:
* - the FIRST block inherits the target's id; the rest get FRESH ids (minted
* by the importer/id-remap, so neighbour blocks are untouched);
* - `^[...]` footnote definitions merge into the page's tail list;
* - REJECTED when the target block carries a markdown-unrepresentable table
* attr (colspan/rowspan/colwidth/background) guarding against silent loss;
* - #159 ambiguity is enforced by a dry `replaceNodeById` count BEFORE the
* splice, so a duplicated id never writes.
*/
protected async patchNodeMarkdown(
pageId: string,
nodeId: string,
markdown: string,
) {
// Import the fragment up front (network-free, canonical) so a bad fragment
// fails before any collab connection or page lock.
const { blocks, definitions } = await importMarkdownFragment(markdown);
// The first imported block inherits the target id; the rest keep the fresh
// ids the importer assigned. Build the thread now so it is stable across a
// collab retry (the transform below is pure over its inputs).
const threaded = blocks.map((b, i) => {
if (i !== 0) return b;
return {
...b,
attrs: {
...(b && typeof b.attrs === "object" ? b.attrs : {}),
id: nodeId,
},
};
});
// Shape-validate every imported block up front (parity with the JSON path):
// the importer only emits schema nodes, but the check is cheap insurance and
// yields the same rich #409 diagnostics if the schema ever drifts.
for (const b of threaded) {
this.assertValidNodeShape("patchNode", b);
}
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let replaced = 0;
let guardAttrs: string | null = null;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
replaced = 0;
guardAttrs = null;
// #159: count matches with the same recursive walk the JSON path uses;
// only an UNAMBIGUOUS single match may write. A dry count keeps the
// ambiguity semantics identical across both paths.
const { replaced: count } = replaceNodeById(liveDoc, nodeId, {
type: "paragraph",
});
replaced = count;
if (count !== 1) return null;
// Guard against SILENT LOSS: if the target block carries a table-cell
// attribute markdown cannot represent (colspan/rowspan/colwidth/
// background), refuse the markdown rewrite so those attrs are not
// dropped. Simple tables (no such attrs) rewrite fine.
const hit = getNodeByRef(liveDoc, nodeId);
guardAttrs = hit ? findUnrepresentableTableAttrs(hit.node) : null;
if (guardAttrs != null) return null;
// Re-mint any minted block id that collides with an existing page id
// (skip index 0: its id is intentionally the target nodeId, unique by
// the #159 dry-count above), so the 1 -> N splice stays page-wide unique.
reassignCollidingBlockIds(liveDoc, threaded, 0);
// 1 -> N splice, then merge any fragment footnote definitions into the
// page's tail list and re-derive canonical footnote numbering.
const { doc: spliced } = replaceNodeByIdWithMany(
liveDoc,
nodeId,
threaded,
);
return mergeFootnoteDefinitions(spliced, definitions);
},
);
// Surface the guard rejection with an actionable message (nothing written).
if (guardAttrs != null) {
throw new Error(
`patchNode: the target block has table-cell attributes markdown cannot ` +
`represent (${guardAttrs}) — a markdown rewrite would drop them. Use ` +
`the table tools (tableUpdateCell/tableInsertRow) or pass a raw ` +
`ProseMirror \`node\` instead of \`markdown\`.`,
);
}
// 0 -> "no node"; >1 -> "ambiguous, refused" (the transform skipped the write
// for any count !== 1). Shared #159 guard, identical to the JSON path.
assertUnambiguousMatch("patchNode", "replace", replaced, nodeId, pageId);
return {
success: true,
replaced,
nodeId,
blocks: threaded.length,
verify: mutation.verify,
};
}
/**
* Insert content relative to an anchor (or append it at the top level).
* Operates on the LIVE collab document so comments and concurrent edits are
* preserved.
*
* Exactly one of `input.markdown` / `input.node` (#413):
* - `markdown` (RECOMMENDED): a canonical markdown fragment. It may import to
* SEVERAL blocks they are inserted IN ORDER at the anchor. `^[...]`
* footnote definitions merge into the page's tail list (same machinery as
* insertFootnote). Every inserted block gets a fresh id.
* - `node`: a raw ProseMirror node for precise attr/mark work, or to insert
* table structure (a bare tableRow/tableCell/tableHeader NOT expressible in
* markdown, so those stay JSON-only).
*
* opts.position:
* - "append": push the content at the end of the top-level content.
* - "before"/"after": insert as a sibling of the anchor, just before/after it.
* Exactly one of anchorNodeId / anchorText must be given; anchorNodeId
* locates a node anywhere by attrs.id, anchorText matches the first top-level
* block whose plain text includes it.
*
* Throws if the anchor cannot be found.
*/
async insertNode(
pageId: string,
input: { markdown?: string; node?: any },
opts: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
) {
await this.ensureAuthenticated();
// XOR: exactly one of markdown / node (both optional in the schema).
const hasMd =
input != null &&
typeof input.markdown === "string" &&
input.markdown.trim() !== "";
const hasNode = input != null && input.node != null;
if (hasMd === hasNode) {
throw new Error(
"insertNode: provide exactly one of `markdown` (recommended, for prose) " +
"or `node` (a raw ProseMirror node, for precise attr/mark work or table structure)",
);
}
if (
!opts ||
(opts.position !== "before" &&
opts.position !== "after" &&
opts.position !== "append")
) {
throw new Error(
'insertNode: `position` must be one of "before", "after", "append"',
);
}
if (opts.position === "before" || opts.position === "after") {
// before/after require EXACTLY ONE anchor (an id or a text fragment).
const hasId =
typeof opts.anchorNodeId === "string" && opts.anchorNodeId.length > 0;
const hasText =
typeof opts.anchorText === "string" && opts.anchorText.length > 0;
if (hasId === hasText) {
throw new Error(
`insertNode: position "${opts.position}" requires exactly one of anchorNodeId or anchorText`,
);
}
}
// Resolve the ordered list of blocks to insert plus any footnote definitions
// to merge. The markdown path imports canonically (so an inserted block is
// byte-identical to the same content in a full-page import); the node path is
// a single block with no footnote merge (raw JSON `^[...]` is not touched).
let blocks: any[];
let definitions: any[] = [];
if (hasMd) {
const frag = await importMarkdownFragment(input.markdown as string);
blocks = frag.blocks;
definitions = frag.definitions;
} else {
const node = input.node;
if (!node || typeof node !== "object" || typeof node.type !== "string") {
throw new Error(
"insertNode: `node` must be an object with a string `type`",
);
}
blocks = [node];
}
// #409: fail fast on a malformed node SHAPE (a nested child with an
// absent/unknown `type`) BEFORE opening a collab session or taking the page
// lock — the root-only check above never sees nested children.
for (const b of blocks) {
this.assertValidNodeShape("insertNode", b);
}
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track insertion in an outer var, reset per-transform, so a collab retry
// recomputes it cleanly (mirrors replaceImage's pattern).
let inserted = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
inserted = false;
// Re-mint any minted block id that collides with an existing page id
// (all inserted blocks are fresh, no skip) so the splice stays unique.
if (hasMd) reassignCollidingBlockIds(liveDoc, blocks);
// Single-block node path keeps `insertNodeRelative` (it owns the
// structural table-node splicing); the markdown path uses the array
// splice so N blocks land in order at one anchor.
const res = hasMd
? insertNodesRelative(liveDoc, blocks, opts)
: insertNodeRelative(liveDoc, blocks[0], opts);
inserted = res.inserted;
if (!inserted) return null; // anchor not found -> skip the write entirely
// Merge any fragment footnote definitions into the page tail list and
// re-derive canonical numbering (no-op when there are none).
return mergeFootnoteDefinitions(res.doc, definitions);
},
);
if (!inserted) {
const anchorDesc = opts.anchorNodeId
? `anchorNodeId "${opts.anchorNodeId}"`
: `anchorText "${opts.anchorText}"`;
// anchorText is matched against the block's literal RENDERED plain text;
// markdown/emoji are tolerated only as a strip-and-retry fallback, so a
// miss usually means the text differs from what's on the page.
const hint = opts.anchorText
? " anchorText must be the block's literal rendered plain text (no markdown wrappers or emoji); anchorNodeId from getPageJson is more reliable."
: "";
throw new Error(
`insertNode: anchor not found (${anchorDesc}) on page ${pageId}.${hint}`,
);
}
return {
success: true,
inserted: true,
position: opts.position,
blocks: blocks.length,
verify: mutation.verify,
};
}
/**
* Remove EVERY node whose attrs.id === nodeId (recursively, including nodes
* nested in callouts/tables) from its parent content array. Operates on the
* LIVE collab document so comments and concurrent edits are preserved.
* Throws if no node matches.
*/
async deleteNode(pageId: string, nodeId: string) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track the deletion count in an outer var, reset per-transform, so a
// collab retry recomputes it cleanly (mirrors replaceImage's pattern).
let deleted = 0;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
deleted = 0;
const { doc: nd, deleted: d } = deleteNodeById(liveDoc, nodeId);
deleted = d;
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS (block
// ids are duplicated on copy/paste, #159): deleting "the node with id X"
// would silently remove EVERY duplicate. Refuse: skip the write and throw
// below so the model re-targets. Only an unambiguous single match is
// deleted.
if (deleted !== 1) return null;
return nd;
},
);
// 0 -> "no node"; >1 -> "ambiguous, refused" (the transform already skipped
// the write for any count !== 1). Single shared guard (#159, #185 review).
assertUnambiguousMatch("deleteNode", "delete", deleted, nodeId, pageId);
return { success: true, deleted, nodeId, verify: mutation.verify };
}
/** Build the public share URL for a page. */
}
return NodesWriteMixin;
}
+655
View File
@@ -0,0 +1,655 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import FormData from "form-data";
import axios, { AxiosInstance } from "axios";
import { convertProseMirrorToMarkdown } from "../lib/markdown-converter.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { footnoteWarningsField } from "../lib/footnote-analyze.js";
import {
serializeDocmostMarkdown,
parseDocmostMarkdown,
} from "../lib/markdown-document.js";
import { diffDocs, summarizeChange } from "../lib/diff.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
import { normalizeAndMergeFootnotes } from "../lib/footnote-normalize-merge.js";
import vm from "node:vm";
// Public method surface of PagesMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IPagesMixin` fails to compile on drift.
export interface IPagesMixin {
createPage(title: string, content: string, spaceId: string, parentPageId?: string): any;
updatePage(pageId: string, content: string, title?: string): any;
renamePage(pageId: string, title: string): any;
movePage(pageId: string, parentPageId: string | null, position?: string): any;
deletePage(pageId: string): any;
sharePage(pageId: string, searchIndexing?: boolean): any;
listShares(): any;
unsharePage(pageId: string): any;
exportPageMarkdown(pageId: string): Promise<string>;
importPageMarkdown(pageId: string, fullMarkdown: string): Promise<any>;
copyPageContent(sourcePageId: string, targetPageId: string): any;
listPageHistory(pageId: string, cursor?: string): any;
getPageHistory(historyId: string): any;
restorePageVersion(historyId: string): any;
diffPageVersions(pageId: string, from?: string, to?: string): any;
}
export function PagesMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IPagesMixin> & TBase {
abstract class PagesMixin extends Base implements IPagesMixin {
/**
* Create a new page with title and content.
* Uses the /pages/import workaround (the only endpoint accepting content),
* then moves the page and restores the exact title: the import endpoint
* derives the title from the FILENAME and replaces spaces with
* underscores, so we explicitly re-set it via /pages/update afterwards.
*/
async createPage(
title: string,
content: string,
spaceId: string,
parentPageId?: string,
) {
await this.ensureAuthenticated();
if (parentPageId) {
try {
await this.getPage(parentPageId);
} catch (e) {
throw new Error(`Parent page with ID ${parentPageId} not found.`);
}
}
// 1. Create content via Import (using multipart/form-data).
// Build a FRESH FormData per send attempt: a FormData body is a single-use
// stream consumed on the first send, so it cannot be replayed by
// this.client's response interceptor (replay fails with 'socket hang up').
// Multipart re-auth is therefore done here with bare axios and an explicit
// one-shot 401/403 retry that rebuilds the body.
const fileContent = Buffer.from(content, "utf-8");
const buildForm = () => {
const form = new FormData();
form.append("spaceId", spaceId);
form.append("file", fileContent, {
filename: `${title || "import"}.md`,
contentType: "text/markdown",
});
return form;
};
const importUrl = `${this.apiUrl}/pages/import`;
let response;
try {
// Call buildForm() ONCE per attempt and reuse the instance for both
// getHeaders() and the body so the Content-Type boundary matches the body.
const form = buildForm();
// Read the Authorization header from this.client's defaults (set by
// login(), only ever deleted — never set to null) instead of building
// `Bearer ${this.token}`: a concurrent JSON 401 can null this.token
// mid-flight, which would otherwise produce a literal "Bearer null".
// ensureAuthenticated() above guarantees login() ran, so the default
// header exists here.
response = await axios.post(importUrl, form, {
headers: {
...form.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} catch (error) {
// On an expired-token auth error, re-login and retry exactly once with a
// freshly-rebuilt FormData (the previous one was already consumed).
if (
axios.isAxiosError(error) &&
(error.response?.status === 401 || error.response?.status === 403)
) {
await this.login();
const form2 = buildForm();
response = await axios.post(importUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} else {
throw error;
}
}
const newPageId = (response.data?.data ?? response.data).id;
// 2. Move to parent if needed
if (parentPageId) {
await this.movePage(newPageId, parentPageId);
}
// 3. Restore the exact title (import mangles spaces into underscores)
if (title) {
await this.client.post("/pages/update", { pageId: newPageId, title });
}
const page = await this.getPage(newPageId);
// Surface non-fatal footnote problems (dangling refs, empty/duplicate
// definitions, markers in tables) so the agent can fix its markup (#166).
return { ...page, ...footnoteWarningsField(content) };
}
/**
* Update a page's content from markdown and optionally its title.
* NOTE: full re-import block ids regenerate. For surgical changes
* use editPageText / updatePageJson instead.
*/
async updatePage(pageId: string, content: string, title?: string) {
await this.ensureAuthenticated();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// REST /pages/update title write below keeps the agent-supplied id (the
// server resolves a slugId there).
const pageUuid = await this.resolvePageId(pageId);
// Write the BODY first, then the title (#159 split-brain). If the collab
// body write fails (e.g. a persist timeout), the title must be left
// UNTOUCHED so the page never ends up with a new title over its old body.
// A title write failing AFTER a successful body is rarer (REST is fast) and
// leaves correct content under a stale title — the lesser inconsistency.
let collabToken = "";
let mutation;
try {
collabToken = await this.getCollabTokenWithReauth();
mutation = await updatePageContentRealtime(
pageUuid,
content,
collabToken,
this.apiUrl,
);
} catch (error: any) {
// Verbose diagnostics (incl. anything that could expose a token prefix)
// are gated behind DEBUG; the thrown Error below carries no token data.
if (process.env.DEBUG) {
console.error(
"Failed to update page content via realtime collaboration:",
error,
);
const tokenPreview = collabToken
? collabToken.substring(0, 15) + "..."
: "null";
console.error(`Collab token preview: ${tokenPreview}`);
}
throw new Error(`Failed to update page content: ${error.message}`);
}
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
return {
success: true,
modified: true,
message: "Page updated successfully.",
pageId: pageId,
verify: mutation.verify,
// Non-fatal footnote diagnostics (#166); omitted when there are none.
...footnoteWarningsField(content),
};
}
/**
* Validate a URL string against a scheme allowlist for a given context.
*
* The markdown link path enforces safe schemes via TipTap, but the raw
* JSON path (updatePageJson) bypasses that so this is the sanitization
* choke point for ProseMirror JSON written directly by the caller.
*
* - "link": reject javascript:, vbscript:, data: (any scheme that can
* execute or smuggle script when the href is clicked).
* - "src": allow only http(s):, mailto:, /api/files paths, or a
* scheme-less relative/absolute path; reject
* javascript:/vbscript:/data:/file:.
*/
/**
* Rename a page (change its title only) without touching or resending its
* content. The slug is derived from the page record, not the body, so it is
* left intact too.
*/
async renamePage(pageId: string, title: string) {
await this.ensureAuthenticated();
await this.client.post("/pages/update", { pageId, title });
return { success: true, pageId, title };
}
/**
* Copy the WHOLE content of one page onto another, entirely server-side: the
* source's ProseMirror document is read and written verbatim onto the target
* via the live collab path, so the document never passes through the model.
*
* Only the target's BODY is replaced its title and slug live on the page
* record (not in the content), so they are untouched. The source page is not
* modified at all.
*/
async movePage(
pageId: string,
parentPageId: string | null,
position?: string,
) {
await this.ensureAuthenticated();
// Docmost requires position >= 5 chars.
const validPosition = position || "a00000";
return this.client
.post("/pages/move", {
pageId,
parentPageId,
position: validPosition,
})
.then((res) => res.data);
}
async deletePage(pageId: string) {
await this.ensureAuthenticated();
return this.client
.post("/pages/delete", { pageId })
.then((res) => res.data);
}
// --- Comment methods (ported from upstream PR #3 by Max Nikitin) ---
/**
* Normalize a comment's `content` into a ProseMirror doc object before
* markdown conversion. createComment/updateComment send content as a
* JSON.stringify(...) STRING, and the server stores it as-is, so on read it
* comes back as a string. convertProseMirrorToMarkdown returns "" for a
* string, so parse it first (guarded fall back to the raw value on any
* parse failure so a non-JSON legacy value is still handled gracefully).
*/
/** Share a page publicly (idempotent) and return the public URL. */
async sharePage(pageId: string, searchIndexing: boolean = true) {
await this.ensureAuthenticated();
const response = await this.client.post("/shares/create", {
pageId,
includeSubPages: false,
searchIndexing,
});
const share = response.data?.data ?? response.data;
const slugId = share.page?.slugId || (await this.getPageRaw(pageId)).slugId;
return {
shareId: share.id,
key: share.key,
pageId: share.pageId,
publicUrl: this.shareUrl(share.key, slugId),
searchIndexing: share.searchIndexing,
};
}
/** List all public shares in the workspace with their URLs. */
/** Build the public share URL for a page. */
protected shareUrl(shareKey: string, slugId: string): string {
return `${this.appUrl}/share/${shareKey}/p/${slugId}`;
}
/** Share a page publicly (idempotent) and return the public URL. */
/** List all public shares in the workspace with their URLs. */
async listShares() {
const shares = await this.paginateAll("/shares", {});
return shares.map((s: any) => ({
shareId: s.id,
key: s.key,
pageId: s.pageId,
pageTitle: s.page?.title,
publicUrl: s.page?.slugId ? this.shareUrl(s.key, s.page.slugId) : null,
searchIndexing: s.searchIndexing,
createdAt: s.createdAt,
}));
}
/** Remove the public share of a page. */
async unsharePage(pageId: string) {
await this.ensureAuthenticated();
const shares = await this.listShares();
const share = shares.find((s: any) => s.pageId === pageId);
if (!share) {
throw new Error(`Page ${pageId} is not shared.`);
}
await this.client.post("/shares/delete", { shareId: share.shareId });
return { success: true, removedShareId: share.shareId, pageId };
}
/**
* Export a page to a single self-contained Docmost-flavoured markdown file:
* meta block + body (with inline comment anchors + diagrams) + comment
* threads. Lossless round-trip target; see importPageMarkdown for the inverse.
*/
async exportPageMarkdown(pageId: string): Promise<string> {
await this.ensureAuthenticated();
const page = await this.getPageRaw(pageId);
const body = page.content ? convertProseMirrorToMarkdown(page.content) : "";
let comments: any[] = [];
try {
// Lossless export: include RESOLVED threads so the export -> import
// round-trip preserves every comment. This is exactly why the active-only
// filter is an opt-in (default false) on listComments.
comments = (await this.listComments(pageId, true)).items;
} catch (e) {
// A comments fetch failure must not lose the body; export with [] and let
// the caller see the (empty) comments block. Log under DEBUG only.
if (process.env.DEBUG) console.error("export: listComments failed", e);
}
const meta = {
version: 1,
pageId: page.id,
slugId: page.slugId,
title: page.title,
spaceId: page.spaceId,
parentPageId: page.parentPageId ?? null,
};
return serializeDocmostMarkdown(meta, body, comments);
}
/**
* Import a self-contained Docmost markdown file back into a page. Parses out
* the meta + comments metadata blocks, converts the body to ProseMirror
* (restoring comment marks + diagrams from their inline HTML), and replaces
* the page content. Comment THREAD records are NOT written to the server in
* this version they are preserved in the file and the inline marks are
* re-applied so the highlights survive; managing comment records stays with
* the comment tools/UI.
*/
async importPageMarkdown(pageId: string, fullMarkdown: string): Promise<any> {
await this.ensureAuthenticated();
const { meta, body, comments } = parseDocmostMarkdown(fullMarkdown);
// PAGE import: canonicalize footnotes (see markdownToProseMirrorCanonical).
const doc = await markdownToProseMirrorCanonical(body);
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
const mutation = await replacePageContent(
pageUuid,
doc,
collabToken,
this.apiUrl,
);
// Collect distinct comment ids that actually became comment marks in the doc.
const collectCommentIds = (node: any, acc: Set<string>): Set<string> => {
if (!node || typeof node !== "object") return acc;
if (Array.isArray(node.marks)) {
for (const mk of node.marks) {
if (mk && mk.type === "comment" && mk.attrs?.commentId) {
acc.add(mk.attrs.commentId);
}
}
}
if (Array.isArray(node.content)) {
for (const child of node.content) collectCommentIds(child, acc);
}
return acc;
};
// Count reflects the comment marks present in the written document, so an id
// that only appears as inert text (e.g. inside a fenced code block) is not
// counted because it never becomes a comment mark.
const anchoredIds = collectCommentIds(doc, new Set<string>());
const result: any = {
success: true,
pageId,
anchoredCommentCount: anchoredIds.size,
commentsInFile: Array.isArray(comments) ? comments.length : 0,
verify: mutation.verify,
};
// Warn (non-fatal) if the file was exported from a DIFFERENT page.
if (meta?.pageId && meta.pageId !== pageId) {
result.warning = `File was exported from page ${meta.pageId} but is being imported into ${pageId}.`;
}
// Non-fatal footnote diagnostics (#166), analyzed on the BODY (the part after
// the docmost:meta / docmost:comments blocks) — so a `[^x]`-like token inside
// those JSON blocks never produces a false warning, while real markers in the
// body do. `body` comes from parseDocmostMarkdown(fullMarkdown) above.
Object.assign(result, footnoteWarningsField(body));
return result;
}
/**
* Rename a page (change its title only) without touching or resending its
* content. The slug is derived from the page record, not the body, so it is
* left intact too.
*/
/**
* Copy the WHOLE content of one page onto another, entirely server-side: the
* source's ProseMirror document is read and written verbatim onto the target
* via the live collab path, so the document never passes through the model.
*
* Only the target's BODY is replaced its title and slug live on the page
* record (not in the content), so they are untouched. The source page is not
* modified at all.
*/
async copyPageContent(sourcePageId: string, targetPageId: string) {
await this.ensureAuthenticated();
// A self-copy would be a no-op overwrite; reject it explicitly so a caller
// mistake surfaces as a clear error rather than a silent round-trip.
if (sourcePageId === targetPageId) {
throw new Error(
"copyPageContent: sourcePageId and targetPageId are the same page (no-op copy)",
);
}
const source = await this.getPageRaw(sourcePageId);
const content = source?.content;
if (
!content ||
typeof content !== "object" ||
content.type !== "doc" ||
!Array.isArray(content.content)
) {
throw new Error(
`copyPageContent: source page ${sourcePageId} has no usable ProseMirror content to copy`,
);
}
// Defense-in-depth: run the same URL-scheme sanitizer the JSON write path
// uses, so copying never lands a javascript:/data: href/src on the target
// (parity with updatePageJson; harmless for already-stored source content).
this.validateDocUrls(content);
// Defense-in-depth (#228): this is a FULL-document write, so canonicalize
// footnotes before copying — a no-op on already-canonical source content, but
// it guarantees a copy can never propagate a non-canonical footnote topology
// to the target (parity with the other full-doc write paths).
// #419: normalize + merge glyph-forked definitions before canonicalizing.
const canonical = canonicalizeFootnotes(normalizeAndMergeFootnotes(content));
const collabToken = await this.getCollabTokenWithReauth();
// Open the TARGET collab doc by its canonical UUID, never the slugId (#260).
const targetUuid = await this.resolvePageId(targetPageId);
const mutation = await this.replacePage(
targetUuid,
canonical,
collabToken,
this.apiUrl,
);
return {
success: true,
sourcePageId,
targetPageId,
copiedNodes: canonical.content.length,
verify: mutation.verify,
};
}
/**
* Surgical text edits: find/replace inside text nodes of the live
* document. Preserves all block ids, marks, callouts and tables.
*/
// --- Page history / diff / transform ---
/**
* List the saved versions (history snapshots) of a page, newest first.
* Docmost auto-snapshots on every save. Returns one cursor-paginated page of
* results: `{ items, nextCursor }`. The history record's id field is `id`.
*/
async listPageHistory(pageId: string, cursor?: string) {
await this.ensureAuthenticated();
const payload: Record<string, any> = { pageId };
if (cursor) payload.cursor = cursor;
const response = await this.client.post("/pages/history", payload);
const data = response.data?.data ?? response.data;
return {
items: data?.items ?? [],
nextCursor: data?.meta?.nextCursor ?? null,
};
}
/**
* Fetch a single page-history version including its lossless ProseMirror
* `content`. The version also carries pageId/title/createdAt.
*/
async getPageHistory(historyId: string) {
await this.ensureAuthenticated();
const response = await this.client.post("/pages/history/info", {
historyId,
});
return response.data?.data ?? response.data;
}
/**
* "Restore" a version: Docmost has NO restore endpoint, so we take the
* version's `content` and write it as the page's current content via the live
* collab path (which itself creates a new history snapshot). Returns the
* affected pageId and the source historyId.
*/
async restorePageVersion(historyId: string) {
await this.ensureAuthenticated();
const version = await this.getPageHistory(historyId);
if (
!version ||
!version.pageId ||
!version.content ||
typeof version.content !== "object"
) {
throw new Error(
`restorePageVersion: history ${historyId} has no usable content`,
);
}
// Defense-in-depth: sanitize URLs in the restored content (parity with the
// JSON write path) before writing it back.
this.validateDocUrls(version.content);
const collabToken = await this.getCollabTokenWithReauth();
// version.pageId is the page entity id (already a UUID); resolvePageId
// short-circuits a UUID with no round-trip, so this is defensive only (#260).
const pageUuid = await this.resolvePageId(version.pageId);
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
() => version.content,
);
return {
pageId: version.pageId,
restoredFrom: historyId,
verify: mutation.verify,
};
}
/**
* Diff two versions of a page and return a Docmost-equivalent change set.
* `from`/`to` each resolve to a ProseMirror doc:
* - null / undefined / "current" -> the page's CURRENT content;
* - any other string -> that historyId's content.
* Returns the diff plus the resolved version metadata for each side.
*/
async diffPageVersions(pageId: string, from?: string, to?: string) {
await this.ensureAuthenticated();
const isCurrent = (v?: string) => v == null || v === "" || v === "current";
const resolveSide = async (
v?: string,
): Promise<{ doc: any; meta: any }> => {
if (isCurrent(v)) {
const raw = await this.getPageRaw(pageId);
return {
doc: raw.content || { type: "doc", content: [] },
meta: {
kind: "current",
pageId,
title: raw.title,
updatedAt: raw.updatedAt,
},
};
}
const version = await this.getPageHistory(v as string);
return {
doc: version.content || { type: "doc", content: [] },
meta: {
kind: "history",
historyId: version.id,
pageId: version.pageId,
title: version.title,
createdAt: version.createdAt,
},
};
};
const fromSide = await resolveSide(from);
const toSide = await resolveSide(to);
const diff = diffDocs(fromSide.doc, toSide.doc);
return { from: fromSide.meta, to: toSide.meta, diff };
}
/**
* Edit a page by running an arbitrary user-supplied JS transform against the
* live document, with a diff preview + page-history safety net.
*
* The transform string is evaluated as `(doc, ctx) => doc` inside a node:vm
* sandbox: it gets ONLY `{ doc, ctx, structuredClone, console }` as globals,
* a 5s timeout, and NO access to require/process/fs/network. It must return a
* `{ type: "doc" }` node, which is validated structurally before any write.
*
* `ctx` exposes:
* - comments: the page's comments (fetched before the live read);
* - log: an array the transform can push diagnostics to (via console.log);
* - consume(id): mark a comment id as consumed (for deleteComments);
* - helpers: the transforms.ts primitives + commentsToFootnotes.
*
* Footnote convention used by the helpers: footnote markers are plain "[N]"
* text in the body, and the notes are an orderedList under a heading whose
* text is "Примечания переводчика".
*
* dryRun (default true): read the page's current content, run the transform,
* and return `{ pushed:false, diff, log }` WITHOUT opening the collab socket.
* Otherwise the transform runs atomically inside mutatePageContent, optionally
* deletes consumed comments, and returns the new historyId + diff + log.
*/
}
return PagesMixin;
}
+656
View File
@@ -0,0 +1,656 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import axios, { AxiosInstance } from "axios";
import {
filterWorkspace,
filterSpace,
filterPage,
filterComment,
filterSearchResult,
} from "../lib/filters.js";
import { convertProseMirrorToMarkdown } from "../lib/markdown-converter.js";
import {
collectInternalFileNodes,
normalizeFileUrl,
resolveInternalFilePath,
} from "../lib/internal-file-urls.js";
import { buildPageTree } from "../lib/tree.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
importMarkdownFragment,
canBeDocChild,
findUnrepresentableTableAttrs,
} from "../lib/markdown-fragment.js";
import { searchInDoc, SearchOptions } from "../lib/page-search.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of ReadMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IReadMixin` fails to compile on drift.
export interface IReadMixin {
getWorkspace(): any;
getSpaces(): any;
listPages(spaceId?: string, limit?: number, tree?: boolean): any;
getTree(spaceId: string, rootPageId?: string, maxDepth?: number): any;
getPageContext(pageId: string): any;
listSidebarPages(spaceId: string, pageId?: string): any;
getPage(pageId: string): any;
getPageJson(pageId: string): any;
getOutline(pageId: string): any;
getNode(pageId: string, nodeId: string, format?: "markdown" | "json"): any;
searchInPage(pageId: string, query: string, opts?: SearchOptions): any;
getTable(pageId: string, tableRef: string): any;
search(query: string, spaceId?: string, limit?: number, opts?: { parentPageId?: string; titleOnly?: boolean }): any;
}
export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IReadMixin> & TBase {
abstract class ReadMixin extends Base implements IReadMixin {
async getWorkspace() {
await this.ensureAuthenticated();
const response = await this.client.post("/workspace/info", {});
return {
data: filterWorkspace(response.data?.data ?? response.data),
success: response.data.success,
};
}
async getSpaces() {
const spaces = await this.paginateAll("/spaces", {});
return spaces.map((space) => filterSpace(space));
}
/**
* List pages in one of two modes.
*
* Default (`tree` false): most recent pages by updatedAt (descending),
* bounded. Fetching the whole space can exceed MCP response/time limits on
* large instances, so a single bounded page of results is returned (default
* 50, max 100) via the `/pages/recent` feed.
*
* Tree (`tree` true): DEPRECATED prefer `getTree`, which shares this exact
* code path (a single `/pages/tree` request via `enumerateSpacePages` +
* `buildPageTree`) but returns the compact `{pageId, title, children?,
* hasChildren?}` shape and supports `rootPageId`/`maxDepth`. This tree mode is
* kept for backward compatibility; it REQUIRES `spaceId` (a page tree is
* scoped to one space) and IGNORES `limit` the whole hierarchy is returned.
* It fetches the tree via `enumerateSpacePages`, which on the fork server
* resolves to a single `/pages/tree` request returning the whole
* permission-filtered flat page set (soft-deleted pages excluded
* server-side); the cursor-BFS in `enumerateSpacePages` is only a fallback for
* stock upstream servers that lack `/pages/tree`.
*/
async listPages(spaceId?: string, limit: number = 50, tree: boolean = false) {
await this.ensureAuthenticated();
if (tree) {
if (!spaceId) {
throw new Error(
"listPages: tree mode requires a spaceId (a page tree is scoped to one space). Pass spaceId, or omit tree to get the recent-pages list.",
);
}
// #486: propagate `truncated` (same pattern as check_new_comments). The old
// code dropped it, so a caller handed an INCOMPLETE tree (the stdio-fallback
// BFS hit its node cap) had no way to know pages were missing. Return the
// tree alongside the flag; the primary /pages/tree path is uncapped so this
// is false there.
const { pages, truncated } = await this.enumerateSpacePages(spaceId);
return { tree: buildPageTree(pages), truncated };
}
const clampedLimit = Math.max(1, Math.min(100, limit));
const payload: Record<string, any> = { limit: clampedLimit, page: 1 };
if (spaceId) payload.spaceId = spaceId;
const response = await this.client.post("/pages/recent", payload);
const data = response.data;
const items = data.data?.items || data.items || [];
return items.map((page: any) => filterPage(page));
}
/**
* Fetch a space's page hierarchy (or one subtree) as a nested tree in a SINGLE
* request the #443 `getTree` tool. Shares its whole code path with
* `listPages(tree:true)`: `enumerateSpacePages` issues one `POST /pages/tree`
* (with the cursor-BFS only as a fallback for stock upstream servers that lack
* the endpoint), then `buildPageTree` nests the flat, permission-filtered,
* position-ordered list. No second tree fetch, no per-node BFS.
*
* - `rootPageId` restrict to that page's subtree; the server seeds the CTE
* with the page itself, so the result is exactly ONE root (the page and its
* descendants). Omit it for the whole space.
* - `maxDepth` trim the response to that many levels (roots = depth 1) to
* save tokens; the server still returns everything in one request, the cut
* is applied in `buildPageTree` AFTER the full tree is built. A node whose
* children were cut carries `hasChildren: true` (source of truth = the flat
* item's server `hasChildren`) so the caller can descend with a follow-up
* `getTree(spaceId, rootPageId=that node)` call.
*
* Output nodes are `{pageId, title, children?, hasChildren?}` only the UUID
* `pageId` is exposed (never `slugId`/`icon`/`position`). Requires `spaceId`
* (a page tree is scoped to one space).
*/
async getTree(spaceId: string, rootPageId?: string, maxDepth?: number) {
await this.ensureAuthenticated();
if (!spaceId) {
throw new Error(
"getTree: spaceId is required (a page tree is scoped to one space).",
);
}
const { pages } = await this.enumerateSpacePages(spaceId, rootPageId);
return buildPageTree(pages, { shape: "getTree", maxDepth });
}
/**
* "Where am I / what's around" for a single page the #443 `getPageContext`
* tool. Metadata only (no page content), using exactly TWO server requests:
*
* 1. `POST /pages/breadcrumbs` a recursive CTE that walks UP from the page.
* The server returns the chain root->page order (it `.reverse()`s the
* child-first walk before responding), INCLUDING the page itself as the
* LAST element. So the last element is the page and everything before it
* is the ancestor chain root->parent. This carries the page's own title
* and spaceId, so no extra page-info fetch is needed for a UUID input.
* 2. `listSidebarPages(spaceId, pageId)` the page's DIRECT children,
* cursor-paginated (a page with >20 children returns ALL of them, no
* dupes) and in sidebar `position` order, each carrying `hasChildren`.
*
* The input may be a slugId (agents copy them from URLs); it is run through
* `resolvePageId` first, exactly like the other page tools. A UUID input adds
* no request there (short-circuit), keeping the total at two; a slugId input
* adds one unavoidable resolve round-trip.
*
* INVARIANT: only the UUID `pageId` is exposed anywhere server `id` is
* mapped to `pageId` and `slugId` is never leaked. A nonexistent/inaccessible
* pageId makes the server 404/403, which propagates as a clear tool error
* (never a hollow empty object).
*/
async getPageContext(pageId: string) {
await this.ensureAuthenticated();
// Resolve a possibly-slugId input to the canonical UUID (no round-trip for a
// UUID). Errors here (bad/inaccessible id) propagate as a clear tool error.
const pageUuid = await this.resolvePageId(pageId);
// Request 1: the ancestor chain, root->page, page included as the LAST item.
const response = await this.client.post("/pages/breadcrumbs", {
pageId: pageUuid,
});
const chain: any[] = (response.data?.data ?? response.data) ?? [];
if (!Array.isArray(chain) || chain.length === 0) {
// The endpoint always includes the page itself, so an empty chain means
// the page is gone/inaccessible — surface a clear error, not {}.
throw new Error(`getPageContext: page "${pageId}" not found or inaccessible`);
}
// Split: the last element is the page, the rest (root->parent) are the
// breadcrumbs. A root page has no ancestors -> breadcrumbs is [].
const self = chain[chain.length - 1];
const ancestors = chain.slice(0, -1);
const page = {
pageId: self.id,
title: self.title,
spaceId: self.spaceId,
};
const breadcrumbs = ancestors.map((n: any) => ({
pageId: n.id,
title: n.title,
}));
// Request 2: direct children in sidebar order, each with hasChildren.
const childItems = await this.listSidebarPages(self.spaceId, pageUuid);
const children = childItems.map((c: any) => ({
pageId: c.id,
title: c.title,
hasChildren: Boolean(c.hasChildren),
}));
return { page, breadcrumbs, children };
}
/**
* List sidebar pages for a space. With no pageId the request returns the
* space ROOT pages; with a pageId it returns the direct CHILDREN of that
* page. pageId is therefore optional and is only included in the POST body
* when provided (an empty/undefined pageId would otherwise change the
* semantics on the server).
*/
async listSidebarPages(spaceId: string, pageId?: string) {
await this.ensureAuthenticated();
// Paginate via the server-issued cursor. The server switched from OFFSET
// (`page`) to CURSOR (`cursor`/`nextCursor`) pagination, and the global
// ValidationPipe(whitelist:true) SILENTLY STRIPS the obsolete `page` field
// — so the old offset loop got the SAME first page every time (with
// hasNextPage stuck true) and dropped every child beyond the first page.
const MAX_PAGES = 50;
let cursor: string | undefined;
let allItems: any[] = [];
let truncated = false;
for (let i = 0; i < MAX_PAGES; i++) {
// limit: 100 is the server-side Max; cuts request count 5x vs the default 20.
const payload: Record<string, any> = { spaceId, limit: 100 };
// Only send pageId when scoping to a page's children; omit it for roots.
if (pageId) payload.pageId = pageId;
if (cursor) payload.cursor = cursor;
const data = (await this.client.post("/pages/sidebar-pages", payload)).data
?.data;
allItems = allItems.concat(data?.items ?? []);
// Advance strictly via the server-issued cursor; a missing/repeated cursor
// means the protocol drifted again — stop instead of looping on page one.
const next = data?.meta?.hasNextPage ? data?.meta?.nextCursor : null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with more pages still available means the child
// list is truncated (mirrors paginateAll).
if (i === MAX_PAGES - 1) truncated = true;
}
// Warn on real truncation (ceiling hit while the server still had pages) so
// the caller is not silently handed an incomplete child list.
if (truncated) {
console.warn(
`listSidebarPages: children of "${pageId ?? spaceId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
return allItems;
}
/**
* Enumerate EVERY page in a space (or in a subtree, when rootPageId is given).
*
* Primary path (fork server): a SINGLE `POST /pages/tree` returns the whole
* space (or a subtree) as a flat, permission-filtered list in one request, in
* the exact node shape buildPageTree consumes. This replaces the old
* per-node BFS, which issued N sidebar requests and after the server moved
* to cursor pagination silently lost every child past the first sidebar
* page (the obsolete `page` param was stripped by ValidationPipe).
*
* The subtree variant (rootPageId given) INCLUDES the root node itself
* (getPageAndDescendants seeds with id = rootPageId), unlike the old BFS
* which started from the root's children.
*
* Fallback path (stdio mode may target STOCK upstream Docmost, which lacks
* `/pages/tree`): on a 404/405 it falls back to the cursor-based BFS below,
* walking direct children via the fixed cursor listSidebarPages. Safeguards:
* a `visited` Set of page ids prevents re-processing a node (cycles /
* duplicate references), and a hard node cap bounds pathological trees so the
* walk always terminates.
*
* Returns `{ pages, truncated }`. `truncated` is true ONLY when the fallback
* BFS stopped at its MAX_NODES cap the primary /pages/tree path is uncapped
* and always returns the complete set, so it never reports truncation.
*/
protected async enumerateSpacePages(
spaceId: string,
rootPageId?: string,
): Promise<{ pages: any[]; truncated: boolean }> {
await this.ensureAuthenticated();
// Single request replaces the whole BFS: /pages/tree returns the full
// permission-filtered flat page set of a space (or a subtree) at once. This
// path is uncapped, so it is never truncated.
const payload = rootPageId ? { pageId: rootPageId } : { spaceId };
try {
const response = await this.client.post("/pages/tree", payload);
const pages = (response.data?.data ?? response.data)?.items ?? [];
return { pages, truncated: false };
} catch (e: any) {
// Only fall back when the endpoint is absent (stock upstream Docmost);
// any other error is a genuine failure and must propagate.
if (
!axios.isAxiosError(e) ||
(e.response?.status !== 404 && e.response?.status !== 405)
) {
throw e;
}
}
// Fallback: cursor-based breadth-first walk via listSidebarPages.
const MAX_NODES = 10000;
const result: any[] = [];
const visited = new Set<string>();
// Seed with the root node itself when scoping to a subtree, so its own
// comments aren't dropped: the primary /pages/tree seeds
// getPageAndDescendants with id = rootPageId (root included), but
// listSidebarPages(spaceId, rootPageId) returns only the root's CHILDREN.
// The `visited` set below prevents a double-add if the root also appears
// among the children. getPageRaw returns a page whose id/title/spaceId are
// exactly what buildPageTree and checkNewComments consume.
if (rootPageId) {
try {
const root = await this.getPageRaw(rootPageId);
if (root?.id) {
result.push(root);
visited.add(root.id);
}
} catch {
// Non-fatal: if the root can't be read, fall through to children-only.
}
}
// Seed the queue with the starting level (subtree children or roots).
const queue: any[] = await this.listSidebarPages(spaceId, rootPageId);
while (queue.length > 0 && result.length < MAX_NODES) {
const node = queue.shift();
if (!node || typeof node !== "object" || !node.id) continue;
// Skip already-seen ids to guard against cycles / duplicate references.
if (visited.has(node.id)) continue;
visited.add(node.id);
result.push(node);
if (node.hasChildren) {
try {
const children = await this.listSidebarPages(spaceId, node.id);
for (const child of children) queue.push(child);
} catch (e: any) {
// A failure fetching one node's children must not abort the whole
// walk: skip this branch and keep enumerating the rest.
}
}
}
// Truncated only when the cap was hit with the queue still non-empty (real
// truncation, not a natural end at exactly MAX_NODES).
return {
pages: result,
truncated: result.length >= MAX_NODES && queue.length > 0,
};
}
/** Raw page info including the ProseMirror JSON content and slugId. */
async getPage(pageId: string) {
await this.ensureAuthenticated();
const resultData = await this.getPageRaw(pageId);
// Agent read: hide resolved-comment anchors so the agent sees only active
// discussions. Active anchors are kept. (The lossless exportPageMarkdown
// round-trip deliberately does NOT pass this flag — resolved anchors there
// must be preserved.)
let content = resultData.content
? convertProseMirrorToMarkdown(resultData.content, {
dropResolvedCommentAnchors: true,
})
: "";
// Always fetch subpages to provide context to the agent
let subpages: any[] = [];
try {
// `pageId` may be a slugId, but the sidebar-pages endpoint requires the
// UUID; `resultData.id` holds the resolved UUID returned by getPageRaw.
subpages = await this.listSidebarPages(resultData.spaceId, resultData.id);
} catch (e: any) {
console.warn("Failed to fetch subpages:", e);
}
// Resolve subpages if the placeholder exists
if (content && content.includes("{{SUBPAGES}}")) {
if (subpages && subpages.length > 0) {
const list = subpages
.map((p: any) => `- [${p.title}](page:${p.id})`)
.join("\n");
content = content.replace("{{SUBPAGES}}", `### Subpages\n${list}`);
} else {
content = content.replace("{{SUBPAGES}}", "");
}
}
return {
data: filterPage(resultData, content, subpages),
success: true,
};
}
/** Page info + raw ProseMirror JSON content (lossless representation). */
async getPageJson(pageId: string) {
const data = await this.getPageRaw(pageId);
return {
id: data.id,
slugId: data.slugId,
title: data.title,
parentPageId: data.parentPageId,
spaceId: data.spaceId,
updatedAt: data.updatedAt,
content: data.content || { type: "doc", content: [] },
};
}
/**
* Fetch an INTERNAL Docmost file (authed loopback) for sandbox mirroring.
* `src` is normalized to `/api/files/<id>/<file>`; `this.client.baseURL`
* already ends in `/api`, so we strip the leading `/api` and request the
* relative path with the client's Authorization header. Returns the raw bytes
* and the response Content-Type (mime), defaulting to octet-stream.
*
* The fetch is size-bounded (hard 64 MiB ceiling) purely to protect memory;
* the authoritative per-blob cap is enforced by the sandbox `put`. The path is
* resolved via resolveInternalFilePath, which REJECTS (throws) any traversal
* or percent-encoded src that would let an attacker-controlled `attrs.src`
* escape `/api/files/` and reach another internal endpoint (SSRF). That throw
* happens before this.client.get, so a malicious src is counted as a failed
* mirror it never reaches the network.
*/
/**
* Compact outline of a page's top-level blocks (no full document body).
* Cheap way to locate sections/tables and grab block ids before drilling in
* with getNode / patchNode / insertNode.
*/
async getOutline(pageId: string) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
return {
pageId,
slugId: data.slugId,
title: data.title,
outline: buildOutline(data.content ?? { type: "doc", content: [] }),
};
}
/**
* Fetch a single block for editing by reference: a block id (headings/
* paragraphs/callouts/images), or `#<index>` to select a top-level block by its
* outline index (the only way to reach tables/rows/cells, which carry no id).
*
* `format` (#413):
* - `"markdown"` (DEFAULT): serialize the block via the canonical converter
* (`{type:"doc",content:[node]}` -> `convertProseMirrorToMarkdown`) a read
* "for editing": pair it with `patchNode({markdown})` to rewrite the block.
* Comment anchors (`<span data-comment-id>`, INCLUDING resolved ones) are
* NOT stripped here (unlike getPage): losing them on write-back would
* orphan the thread. Returns `{ ..., format:"markdown", markdown }`.
* - `"json"`: return the raw ProseMirror subtree as-is (lossless; the previous
* default). Returns `{ ..., format:"json", node }`.
*
* AUTO fallback: a type that cannot be a document top-level child
* (tableRow/tableCell/tableHeader, addressed by `#<index>`) is NOT expressible
* as a standalone markdown document, so a `"markdown"` request for such a node
* transparently falls back to JSON with an explicit `format:"json"` field. The
* check derives from the schema's `doc` contentMatch, so it tracks the schema.
*/
async getNode(
pageId: string,
nodeId: string,
format: "markdown" | "json" = "markdown",
) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
const hit = getNodeByRef(
data.content ?? { type: "doc", content: [] },
nodeId,
);
if (!hit) {
throw new Error(
`getNode: no node found for "${nodeId}" on page ${pageId} (use a block id from getOutline, or "#<index>" for a top-level block such as a table)`,
);
}
// JSON requested (or a non-top-level type that markdown cannot represent as a
// standalone document): return the subtree verbatim.
if (format === "json" || !canBeDocChild(hit.type)) {
return {
pageId,
ref: nodeId,
path: hit.path,
type: hit.type,
format: "json" as const,
node: hit.node,
};
}
// Markdown: wrap the node as a one-block doc and run the canonical converter.
// Comment anchors are DELIBERATELY preserved (converter default) so a
// getNode(markdown) -> edit -> patchNode(markdown) round trip does not orphan
// a comment thread; this differs from getPage, which strips them.
const markdown = convertProseMirrorToMarkdown({
type: "doc",
content: [hit.node],
});
return {
pageId,
ref: nodeId,
path: hit.path,
type: hit.type,
format: "markdown" as const,
markdown,
};
}
/**
* Find every occurrence of `query` on a page IN MEMORY, over the plain text of
* each text container (reusing the same `getPageRaw` fetch as the other read
* tools) no server search endpoint, no whole-document round-trip through the
* model. Returns `{ total, truncated, matches }`; each match carries a ref for
* getNode/patchNode (the `#<index>` form resolves with getNode but NOT
* patchNode see SearchMatch.nodeId), plus the top-level block index and a
* short context window used to build a unique text `selection` for
* createComment (createComment has no nodeId param). The pure engine
* (`searchInDoc`) owns the traversal, glue, the RE2 ReDoS-safe regex engine
* and the empty-query / invalid-or-unsupported-regex errors.
*/
async searchInPage(pageId: string, query: string, opts: SearchOptions = {}) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
const result = searchInDoc(
data.content ?? { type: "doc", content: [] },
query,
opts,
);
return { pageId, query, ...result };
}
/**
* Read a table as a matrix. `tableRef` is `#<index>` (from getOutline) or a
* block id of any node inside the table. Returns the cell texts plus a
* parallel cellIds matrix (each cell's first paragraph id, or null) so a
* caller can patchNode a cell for rich-formatted edits. Throws when no table
* resolves for the reference.
*/
async getTable(pageId: string, tableRef: string) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
const t = readTable(data.content ?? { type: "doc", content: [] }, tableRef);
if (!t) {
throw new Error(
`tableGet: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
pageId,
table: tableRef,
rows: t.rows,
cols: t.cols,
path: t.path,
cells: t.cells,
cellIds: t.cellIds,
};
}
/**
* Insert a row of plain-text cells into a table on the LIVE collab document.
* `tableRef` is `#<index>` or a block id inside the target table. `cells` is
* padded to the table's column count (more cells than columns throws); `index`
* is a 0-based insert position (omit/out-of-range to append). Throws when no
* table resolves for the reference.
*/
async search(
query: string,
spaceId?: string,
limit?: number,
opts: { parentPageId?: string; titleOnly?: boolean } = {},
) {
await this.ensureAuthenticated();
// Opt into the #443 agent-lookup mode: `substring: true` turns on the hybrid
// substring + FTS branch that returns path + snippet + score. A stock
// upstream server strips these unknown DTO fields (whitelist:true) and
// silently degrades to plain FTS — see the tool-registration comment.
const payload: Record<string, any> = {
query,
spaceId,
substring: true,
};
if (opts.parentPageId) payload.parentPageId = opts.parentPageId;
if (opts.titleOnly) payload.titleOnly = true;
// Clamp an optional caller-supplied limit into the lookup range (1..50)
// before forwarding; omit it when not provided so the server default applies.
if (limit !== undefined) {
payload.limit = Math.max(1, Math.min(50, limit));
}
const response = await this.client.post("/search", payload);
// Normalize both response shapes: bare array and paginated { items: [...] }
const data = response.data?.data;
const items = Array.isArray(data) ? data : data?.items || [];
const filteredItems = items.map((item: any) => filterSearchResult(item));
return {
items: filteredItems,
success: response.data?.success || false,
};
}
}
return ReadMixin;
}
+225
View File
@@ -0,0 +1,225 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
collectInternalFileNodes,
normalizeFileUrl,
resolveInternalFilePath,
} from "../lib/internal-file-urls.js";
// Public method surface of StashMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IStashMixin` fails to compile on drift.
export interface IStashMixin {
stashPage(pageId: string): Promise<{ uri: string; sha256: string; size: number; images: { mirrored: number; failed: number }; }>;
}
export function StashMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IStashMixin> & TBase {
abstract class StashMixin extends Base implements IStashMixin {
/**
* Fetch an INTERNAL Docmost file (authed loopback) for sandbox mirroring.
* `src` is normalized to `/api/files/<id>/<file>`; `this.client.baseURL`
* already ends in `/api`, so we strip the leading `/api` and request the
* relative path with the client's Authorization header. Returns the raw bytes
* and the response Content-Type (mime), defaulting to octet-stream.
*
* The fetch is size-bounded (hard 64 MiB ceiling) purely to protect memory;
* the authoritative per-blob cap is enforced by the sandbox `put`. The path is
* resolved via resolveInternalFilePath, which REJECTS (throws) any traversal
* or percent-encoded src that would let an attacker-controlled `attrs.src`
* escape `/api/files/` and reach another internal endpoint (SSRF). That throw
* happens before this.client.get, so a malicious src is counted as a failed
* mirror it never reaches the network.
*/
protected async fetchInternalFile(
src: string,
): Promise<{ buffer: Buffer; mime: string }> {
const HARD_CEILING = 64 * 1024 * 1024; // 64 MiB memory guard
const relPath = resolveInternalFilePath(src);
const response = await this.client.get(relPath, {
responseType: "arraybuffer",
timeout: 30000,
maxContentLength: HARD_CEILING,
maxBodyLength: HARD_CEILING,
});
const buffer = Buffer.from(response.data);
if (buffer.length === 0) {
throw new Error(`Empty file response from "${src}"`);
}
const rawCt = response.headers?.["content-type"];
const mime =
typeof rawCt === "string" && rawCt.length > 0
? rawCt.split(";")[0].trim().toLowerCase()
: "application/octet-stream";
return { buffer, mime };
}
/**
* Stash a page's full content into the in-RAM blob sandbox and return ONLY a
* short anonymous URL the body never enters the model context (this is the
* whole point: ~30KB+ ProseMirror docs blow the model context if passed as a
* tool argument). Every INTERNAL file/image src (the type-agnostic criterion,
* so drawio/excalidraw/video/file nodes are covered too) is mirrored into the
* sandbox and its `src` rewritten to the sandbox URL, so an external consumer
* can fetch the images anonymously. External http(s) srcs are left untouched.
*
* Blobs live in RAM with a short TTL and are cleared on restart consume the
* URLs within the TTL and one uptime. A failed image fetch never aborts the
* doc: the original src is kept and the failure counted.
*
* Returns { uri, sha256, size, images:{mirrored, failed} }. `uri` and `sha256`
* are for the document blob; `sha256` is also the blob's ETag (integrity).
*/
async stashPage(pageId: string): Promise<{
uri: string;
sha256: string;
size: number;
images: { mirrored: number; failed: number };
}> {
if (!this.sandboxPut) {
throw new Error(
"stashPage is unavailable: the blob sandbox is not configured on this server",
);
}
await this.ensureAuthenticated();
// Stash the SAME shape getPageJson returns (id/title/.../content), with a
// deep clone so the rewrite never mutates anything shared.
const pageJson = await this.getPageJson(pageId);
const cloned: any = structuredClone(pageJson);
// Group internal-file nodes by normalized src so each unique resource is
// fetched + stored ONCE (dedup), and every node sharing that src points at
// the one sandbox blob. Capture each node's ORIGINAL raw src per-node:
// dedup groups nodes whose normalized src is equal even when their raw srcs
// differ (e.g. `/api/files/...` vs the bare `/files/...`), so on a revert we
// must restore each node's own original value, not the group key.
const bySrc = new Map<string, Array<{ node: any; origSrc: string }>>();
for (const node of collectInternalFileNodes(cloned.content)) {
const origSrc = String(node.attrs.src);
const src = normalizeFileUrl(origSrc);
const entry = { node, origSrc };
const group = bySrc.get(src);
if (group) group.push(entry);
else bySrc.set(src, [entry]);
}
let mirrored = 0;
let failed = 0;
// Record every successful mirror so it can be (a) reverted if its blob gets
// FIFO-evicted by a LATER put in this same stash, and (b) freed if the final
// doc put throws.
const mirrors: Array<{
uri: string;
entries: Array<{ node: any; origSrc: string }>;
}> = [];
const MAX_CONCURRENCY = 5;
const groups = [...bySrc.entries()];
for (let i = 0; i < groups.length; i += MAX_CONCURRENCY) {
const batch = groups.slice(i, i + MAX_CONCURRENCY);
await Promise.all(
batch.map(async ([src, entries]) => {
try {
const { buffer, mime } = await this.fetchInternalFile(src);
// put may throw if the blob exceeds the per-blob/total caps.
const stored = this.sandboxPut!(buffer, mime);
for (const entry of entries) entry.node.attrs.src = stored.uri;
mirrors.push({ uri: stored.uri, entries });
mirrored++;
} catch (err) {
// One bad/oversized image (or a rejected traversal src) must not
// abort the document. Logged unconditionally (never the blob body),
// matching the package's ungated console.warn convention.
failed++;
console.warn(
`stashPage: failed to mirror "${src}": ${
err instanceof Error ? err.message : String(err)
}`,
);
}
}),
);
}
// Revert one mirror's nodes to their original internal srcs and re-count it
// as failed (its blob was FIFO-evicted before the doc could reference it
// safely).
const revertMirror = (mirror: {
uri: string;
entries: Array<{ node: any; origSrc: string }>;
}) => {
for (const entry of mirror.entries) entry.node.attrs.src = entry.origSrc;
mirrored--;
failed++;
console.warn(
`stashPage: mirrored blob ${mirror.uri} was evicted before the doc ` +
`could safely reference it; reverted its src and counted it as failed`,
);
};
// Pre-put reconciliation: an image put earlier in THIS stash can FIFO-evict
// an even-earlier image of the same stash. Drop those from the live set
// first so the first serialized doc is already mostly correct.
let liveMirrors = mirrors;
if (this.sandboxHas) {
liveMirrors = [];
for (const mirror of mirrors) {
if (this.sandboxHas(mirror.uri)) liveMirrors.push(mirror);
else revertMirror(mirror);
}
}
// Put the document, then reconcile against eviction caused by the doc put
// ITSELF (the doc is newest, FIFO drops oldest = this stash's images). Each
// iteration reverts >=1 mirror, so the loop terminates (worst case: all
// images reverted and the doc references no sandbox image URLs).
let stored: { uri: string; sha256: string; size: number };
for (;;) {
const docBuf = Buffer.from(JSON.stringify(cloned), "utf8");
let docStored: { uri: string; sha256: string; size: number };
try {
docStored = this.sandboxPut(docBuf, "application/json");
} catch (err) {
// The doc put failed (e.g. doc exceeds the cap). Free this op's image
// blobs instead of leaking them in RAM for the whole TTL, then
// re-throw.
if (this.sandboxEvict) {
for (const mirror of liveMirrors) this.sandboxEvict(mirror.uri);
}
throw err;
}
if (!this.sandboxHas) {
stored = docStored;
break;
}
const evictedNow = liveMirrors.filter((m) => !this.sandboxHas!(m.uri));
if (evictedNow.length === 0) {
stored = docStored;
break;
}
// The doc we just stored references now-dead blobs. Revert those nodes,
// drop the stale doc blob, and loop to re-serialize + re-put the
// corrected doc.
for (const mirror of evictedNow) revertMirror(mirror);
liveMirrors = liveMirrors.filter((m) => this.sandboxHas!(m.uri));
if (this.sandboxEvict) this.sandboxEvict(docStored.uri);
}
return {
uri: stored.uri,
sha256: stored.sha256,
size: stored.size,
images: { mirrored, failed },
};
}
/**
* Compact outline of a page's top-level blocks (no full document body).
* Cheap way to locate sections/tables and grab block ids before drilling in
* with getNode / patchNode / insertNode.
*/
}
return StashMixin;
}
+291
View File
@@ -0,0 +1,291 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of TablesMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements ITablesMixin` fails to compile on drift.
export interface ITablesMixin {
insertFootnote(pageId: string, anchorText: string, text: string): any;
tableInsertRow(pageId: string, tableRef: string, cells: string[], index?: number): any;
tableDeleteRow(pageId: string, tableRef: string, index: number): any;
tableUpdateCell(pageId: string, tableRef: string, row: number, col: number, text: string): any;
}
export function TablesMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & ITablesMixin> & TBase {
abstract class TablesMixin extends Base implements ITablesMixin {
/**
* AUTHOR-INLINE footnote insertion. The agent supplies only WHERE
* (`anchorText`, a snippet of body text to attach the marker after) and WHAT
* (`text`, the footnote content as markdown). Numbering and the bottom
* `footnotesList` are derived deterministically server-side
* (`insertInlineFootnote` -> `canonicalizeFootnotes`): the agent never sees,
* assigns, or edits a footnote number or the list, so it CANNOT desync.
*
* Content DEDUP: when an existing definition has the same content, its id is
* reused (one number, one definition, several references). The write is atomic
* via `mutatePageContent` (single-writer, page-locked); if the anchor text is
* not found the transform aborts with a clear error and no write happens.
*/
async insertFootnote(pageId: string, anchorText: string, text: string) {
await this.ensureAuthenticated();
if (!anchorText || !anchorText.trim()) {
throw new Error("insertFootnote: anchorText is required");
}
if (text == null || `${text}`.trim() === "") {
throw new Error("insertFootnote: text is required");
}
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let result: { footnoteId: string; reused: boolean } | null = null;
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc: any) => {
const r = insertInlineFootnote(liveDoc, { anchorText, text });
if (!r.inserted) {
// Abort the page-locked write by throwing: mutatePageContent does not
// persist when the transform throws, so a missing anchor leaves the
// page untouched (no partial write).
throw new Error(
`insertFootnote: anchor text not found: ${JSON.stringify(
anchorText.slice(0, 80),
)}`,
);
}
result = { footnoteId: r.footnoteId, reused: r.reused };
return r.doc;
},
);
// The not-found path throws inside the transform (aborting mutatePage), so by
// here `result` is always set.
const r = result!;
return {
success: true,
modified: true,
pageId,
footnoteId: r.footnoteId,
reused: r.reused,
message: r.reused
? "Footnote inserted (reused an existing same-content definition)."
: "Footnote inserted.",
verify: mutation.verify,
};
}
/**
* Page-locked write seam over collaboration.mutatePageContent. Production just
* delegates; it exists as an overridable method so the insertFootnote wrapper
* (transform abort-on-not-found + response shaping) can be unit-tested without
* standing up a live Hocuspocus collab socket.
*
* SELF-RESOLVES the pageId to the canonical UUID (issue #449, "resolve-then-
* lock"): every write must lock and key its CollabSession by the UUID, never a
* raw slugId (#260). resolvePageId is cached/idempotent, so a caller that
* already resolved pays no extra round-trip; centralizing it here means a
* caller that reaches this seam with a raw slugId still locks correctly instead
* of silently splitting the mutex key. withPageLock also asserts the key is a
* UUID as a hard backstop.
*/
/**
* Insert a row of plain-text cells into a table on the LIVE collab document.
* `tableRef` is `#<index>` or a block id inside the target table. `cells` is
* padded to the table's column count (more cells than columns throws); `index`
* is a 0-based insert position (omit/out-of-range to append). Throws when no
* table resolves for the reference.
*/
async tableInsertRow(
pageId: string,
tableRef: string,
cells: string[],
index?: number,
) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track insertion in an outer var, reset per-transform, so a collab retry
// recomputes it cleanly (mirrors insertNode's pattern).
let inserted = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
inserted = false;
const { doc: nd, inserted: ins } = insertTableRow(
liveDoc,
tableRef,
cells,
index,
);
inserted = ins;
if (!inserted) return null; // table not found -> skip the write entirely
return nd;
},
);
if (!inserted) {
throw new Error(
`tableInsertRow: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
success: true,
table: tableRef,
inserted: true,
verify: mutation.verify,
};
}
/**
* Delete the row at 0-based `index` from a table on the LIVE collab document.
* `tableRef` is `#<index>` or a block id inside the target table. The helper's
* out-of-range and last-row errors propagate; a missing table throws here.
*/
async tableDeleteRow(pageId: string, tableRef: string, index: number) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let deleted = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
deleted = false;
const { doc: nd, deleted: del } = deleteTableRow(
liveDoc,
tableRef,
index,
);
deleted = del;
if (!deleted) return null; // table not found -> skip the write entirely
return nd;
},
);
if (!deleted) {
throw new Error(
`tableDeleteRow: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
success: true,
table: tableRef,
deleted: true,
verify: mutation.verify,
};
}
/**
* Set the plain-text content of cell `[row, col]` (0-based) in a table on the
* LIVE collab document, replacing the cell's content with a single text
* paragraph (the cell's first-paragraph id is preserved). `tableRef` is
* `#<index>` or a block id inside the target table. The helper's out-of-range
* error propagates; a missing table throws here.
*/
async tableUpdateCell(
pageId: string,
tableRef: string,
row: number,
col: number,
text: string,
) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let updated = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
updated = false;
const { doc: nd, updated: upd } = updateTableCell(
liveDoc,
tableRef,
row,
col,
text,
);
updated = upd;
if (!updated) return null; // table not found -> skip the write entirely
return nd;
},
);
if (!updated) {
throw new Error(
`tableUpdateCell: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
success: true,
table: tableRef,
row,
col,
verify: mutation.verify,
};
}
/**
* Create a new page with title and content.
* Uses the /pages/import workaround (the only endpoint accepting content),
* then moves the page and restores the exact title: the import endpoint
* derives the title from the FILENAME and replaces spaces with
* underscores, so we explicitly re-set it via /pages/update afterwards.
*/
}
return TablesMixin;
}
+232
View File
@@ -0,0 +1,232 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { diffDocs, summarizeChange } from "../lib/diff.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
import { normalizeAndMergeFootnotes } from "../lib/footnote-normalize-merge.js";
import vm from "node:vm";
// Public method surface of TransformsMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements ITransformsMixin` fails to compile on drift.
export interface ITransformsMixin {
transformPage(pageId: string, transformJs: string, opts?: { dryRun?: boolean; deleteComments?: boolean }): any;
}
export function TransformsMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & ITransformsMixin> & TBase {
abstract class TransformsMixin extends Base implements ITransformsMixin {
/**
* Edit a page by running an arbitrary user-supplied JS transform against the
* live document, with a diff preview + page-history safety net.
*
* The transform string is evaluated as `(doc, ctx) => doc` inside a node:vm
* sandbox: it gets ONLY `{ doc, ctx, structuredClone, console }` as globals,
* a 5s timeout, and NO access to require/process/fs/network. It must return a
* `{ type: "doc" }` node, which is validated structurally before any write.
*
* `ctx` exposes:
* - comments: the page's comments (fetched before the live read);
* - log: an array the transform can push diagnostics to (via console.log);
* - consume(id): mark a comment id as consumed (for deleteComments);
* - helpers: the transforms.ts primitives + commentsToFootnotes.
*
* Footnote convention used by the helpers: footnote markers are plain "[N]"
* text in the body, and the notes are an orderedList under a heading whose
* text is "Примечания переводчика".
*
* dryRun (default true): read the page's current content, run the transform,
* and return `{ pushed:false, diff, log }` WITHOUT opening the collab socket.
* Otherwise the transform runs atomically inside mutatePageContent, optionally
* deletes consumed comments, and returns the new historyId + diff + log.
*/
async transformPage(
pageId: string,
transformJs: string,
opts: { dryRun?: boolean; deleteComments?: boolean } = {},
) {
const dryRun = opts.dryRun ?? true;
const deleteComments = opts.deleteComments ?? false;
await this.ensureAuthenticated();
// Full feed (incl. resolved): a page transform (e.g. comments -> footnotes)
// must operate on every comment, so it opts into the unfiltered feed.
const comments = (await this.listComments(pageId, true)).items;
// ctx handed to the sandbox. consume() records ids; helpers are the pure
// transform primitives. log is captured from console.log inside the sandbox.
const ctx = {
comments,
log: [] as string[],
consumed: new Set<string>(),
consume(id: string) {
this.consumed.add(id);
},
helpers: {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
},
};
// Captured oldDoc / newDoc for the diff (set inside runTransform).
let oldDoc: any;
let newDoc: any;
// SYNCHRONOUS transform runner — safe to call inside mutatePageContent's
// onSynced (no await between the live read and the write).
const runTransform = (liveDoc: any): any => {
oldDoc = structuredClone(liveDoc);
const sandbox: Record<string, any> = {
doc: structuredClone(liveDoc),
ctx,
structuredClone,
console: {
log: (...a: any[]) => ctx.log.push(a.map((x) => String(x)).join(" ")),
},
};
// Wrap the provided string in parentheses so both an expression-arrow
// (`(doc, ctx) => {...}`) and a parenthesized function work. Run it in a
// fresh context with no require/process/module so the transform cannot
// touch fs/network/process. 5s wall-clock timeout.
let fn: any;
try {
fn = vm.runInNewContext("(" + transformJs + ")", sandbox, {
timeout: 5000,
});
} catch (e: any) {
throw new Error(`transform did not compile: ${e?.message ?? e}`);
}
if (typeof fn !== "function") {
throw new Error(
"transform must evaluate to a function (doc, ctx) => doc",
);
}
const raw = vm.runInNewContext(
"f(d, c)",
{ f: fn, d: sandbox.doc, c: ctx },
{ timeout: 5000 },
);
if (
!raw ||
typeof raw !== "object" ||
raw.type !== "doc" ||
!Array.isArray(raw.content)
) {
throw new Error(
'transform must return a ProseMirror doc node ({ type:"doc", content:[...] })',
);
}
// Validate the RAW transform output FIRST (structure — including the
// MAX_DEPTH guard — and URLs), mirroring updatePageJson. The canonicalizer
// recurses without a depth limiter, so validating after it would turn a
// too-deep doc into an opaque "Maximum call stack size exceeded" instead of
// the intended "nesting exceeds the maximum depth" error.
this.validateDocStructure(raw);
this.validateDocUrls(raw);
// Auto-canonicalize footnotes after the transform (idempotent): no write
// path can leave footnotes out of order / orphaned / in a raw `[^id]`
// block. In a dryRun preview this may surface footnote edits the script
// author did not write (the canonicalizer tidied them) — that is expected.
// #419: normalize + merge glyph-forked definitions before canonicalizing.
const result = canonicalizeFootnotes(normalizeAndMergeFootnotes(raw));
newDoc = result;
return result;
};
if (dryRun) {
// Preview only: run against the current REST snapshot, never open the
// socket. oldDoc/newDoc are captured by runTransform.
const raw = await this.getPageRaw(pageId);
const current = raw.content || { type: "doc", content: [] };
runTransform(current);
// Run an independent Yjs-encodability check (same sanitize + schema as the
// apply path), so the preview fails with the same descriptive error when
// the doc is not encodable instead of returning a misleadingly-green diff.
assertYjsEncodable(newDoc);
return {
pushed: false,
diff: diffDocs(oldDoc, newDoc),
log: ctx.log,
};
}
// Apply atomically against the live doc.
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
runTransform,
);
// Optionally delete consumed comments (best-effort; a delete failure must
// not undo the successful write).
const deletedComments: string[] = [];
if (deleteComments) {
for (const id of ctx.consumed) {
try {
await this.deleteComment(id);
deletedComments.push(id);
} catch (e) {
if (process.env.DEBUG) {
console.error(`transform: failed to delete comment ${id}:`, e);
}
}
}
}
// Fetch the newest historyId (Docmost snapshots on the write above).
let historyId: string | null = null;
try {
const hist = await this.listPageHistory(pageId);
historyId = hist.items?.[0]?.id ?? null;
} catch (e) {
if (process.env.DEBUG) {
console.error("transform: failed to fetch history id:", e);
}
}
return {
pushed: true,
historyId,
diff: diffDocs(oldDoc, newDoc),
deletedComments,
log: ctx.log,
verify: mutation.verify,
};
}
}
return TransformsMixin;
}
-1
View File
@@ -4,7 +4,6 @@ import { readFileSync } from "fs";
import { fileURLToPath } from "url";
import { dirname, join } from "path";
import { DocmostClient, DocmostMcpConfig } from "./client.js";
import { parseNodeArg } from "@docmost/prosemirror-markdown";
import { searchShapes } from "./lib/drawio-shapes.js";
import { getGuideSection } from "./lib/drawio-guide.js";
import { SHARED_TOOL_SPECS, SharedToolSpec } from "./tool-specs.js";
+34 -4
View File
@@ -37,6 +37,25 @@ const CONNECT_TIMEOUT_MS = 25000;
/** Time we wait for the server to acknowledge our write before giving up. */
const PERSIST_TIMEOUT_MS = 20000;
/**
* Marker property set on the Error thrown when the Hocuspocus handshake REJECTS
* our collab token (onAuthenticationFailed). The client wraps content writes so
* that on this specific failure it invalidates its cached collab token and
* retries once with a fresh one symmetric to the HTTP-401 reauth path (#486).
* A plain message-match would be brittle; a tagged property is unambiguous and
* survives teardown (which rejects pending ops with this SAME error object).
*/
const COLLAB_AUTH_FAILED_MARKER = "collabAuthFailed";
/** True when `e` is the tagged collab-WS auth-failure error (see marker above). */
export function isCollabAuthFailedError(e: unknown): boolean {
return !!(
e &&
typeof e === "object" &&
(e as Record<string, unknown>)[COLLAB_AUTH_FAILED_MARKER] === true
);
}
/**
* Tunables, read fresh from the environment on every acquire so tests (and a
* live rollback) can change them without reloading the module. Mirrors how
@@ -302,10 +321,13 @@ export class CollabSession {
this.openResolve?.();
},
onAuthenticationFailed: () => {
this.teardown(
new Error("Authentication failed for collaboration connection"),
true,
);
// Tag the error so the client can tell a REJECTED collab token apart
// from a generic disconnect and invalidate + refresh it (#486).
const err = new Error(
"Authentication failed for collaboration connection",
) as Error & { [COLLAB_AUTH_FAILED_MARKER]?: boolean };
err[COLLAB_AUTH_FAILED_MARKER] = true;
this.teardown(err, true);
},
});
});
@@ -440,9 +462,16 @@ export class CollabSession {
// must stay synchronous (no await). While the JS event loop is not
// yielded, no incoming remote update can interleave, so any already-synced
// concurrent edits are preserved in liveDoc.
//
// INVARIANT 1 is machine-checked: the BEGIN/END markers below delimit the
// no-await window, and test/unit/no-await-critical-window.test.mjs scans
// this source and FAILS if any `await` (or `for await`/`yield`) appears
// between them. Do NOT add an await inside this block — an accidental
// async boundary here silently reopens the clobber-live-edits race (#152).
let newDoc: any;
let beforeDoc: any;
try {
// === MUTATE-CRITICAL-WINDOW: BEGIN (no await between here and END #449) ===
let liveDoc = TiptapTransformer.fromYdoc(this.ydoc, "default");
if (
!liveDoc ||
@@ -480,6 +509,7 @@ export class CollabSession {
// ids of unchanged nodes, so an open editor's cursor is not yanked to the
// end of the document on every agent write.
applyDocToFragment(this.ydoc, newDoc);
// === MUTATE-CRITICAL-WINDOW: END (#449) ===
} catch (e) {
// Includes errors thrown by transform (e.g. "afterText not found",
// "text not found"): propagate them verbatim to the caller.
+74 -31
View File
@@ -10,7 +10,7 @@
// exactly mxGraph's convention for a child of a container, so they map across
// directly. Container sizes are computed by ELK; leaf sizes are preserved.
import ELK from "elkjs/lib/elk.bundled.js";
import { Worker } from "node:worker_threads";
import { JSDOM } from "jsdom";
import { normalizeInput, parseCells, type DrawioCell } from "./drawio-xml.js";
@@ -18,22 +18,33 @@ import { normalizeInput, parseCells, type DrawioCell } from "./drawio-xml.js";
const DEFAULT_W = 140;
const DEFAULT_H = 60;
// DoS bounds for the in-process ELK layout. The mxGraph XML is LLM-supplied
// (layout:"elk" in drawioCreate/drawioUpdate) and elkjs runs synchronously on
// the MCP server's event loop, so an unbounded graph would block it for
// seconds-to-minutes. A ~1MB XML (well under the stage-1 16MB cap) can carry
// thousands of nodes. We cap the graph size and race the layout against a
// wall-clock timeout; on either bound we fall back to the ORIGINAL model, the
// same best-effort contract the catch already honours.
// DoS bounds for the ELK layout. The mxGraph XML is LLM-supplied (layout:"elk"
// in drawioCreate/drawioUpdate). elkjs' layout() returns a Promise but runs the
// crossing-minimisation SYNCHRONOUSLY — it blocks whatever thread it runs on for
// the whole pass. A ~1MB XML (well under the stage-1 16MB cap) can carry
// thousands of nodes. We (a) cap the graph size before ever calling ELK and
// (b) run the layout in a WORKER THREAD so the main event loop stays free, with
// the wall-clock timeout enforced by terminating that worker. On either bound we
// fall back to the ORIGINAL model, the same best-effort contract the catch honours.
// - 500 nodes lays out in well under a second; beyond that ELK cost climbs
// steeply, so refuse and leave the (already-valid) model untouched.
// - Edges dominate the layered-crossing cost, so allow a bit more headroom
// (1000) than nodes but still bound them.
// - 5s is generous for any graph within the caps yet short enough that a
// pathological input can never wedge the server.
// - The timeout is a HARD kill of the worker thread — the only way to interrupt
// synchronous JS. The in-process setTimeout race we used before was an
// illusion: the timer could never fire while the SAME thread was blocked
// inside elkjs, so it "protected" nothing. Now the timer runs on the main
// thread while ELK runs on the worker, so it can actually fire and terminate.
const ELK_MAX_NODES = 500;
const ELK_MAX_EDGES = 1000;
const ELK_TIMEOUT_MS = 5000;
// Wall-clock ceiling for a single layout pass. Overridable for tests (a tiny
// value forces the terminate-on-timeout path deterministically); a non-positive
// or unparseable override falls back to the default.
const ELK_TIMEOUT_DEFAULT_MS = 5000;
function resolveElkTimeoutMs(): number {
const raw = Number(process.env.DRAWIO_ELK_TIMEOUT_MS);
return Number.isFinite(raw) && raw > 0 ? Math.floor(raw) : ELK_TIMEOUT_DEFAULT_MS;
}
// Spacing is set >=150px on purpose so an ELK layout never trips the linter's
// "gap between adjacent shapes < 150px" quality warning (acceptance #3).
@@ -78,13 +89,57 @@ interface ElkGraph extends ElkNode {
edges?: ElkEdge[];
}
/**
* Run one ELK layered layout on a worker thread and resolve with the laid-out
* graph. The timeout is enforced by `worker.terminate()` a HARD kill, which is
* the only way to interrupt elkjs' synchronous crossing-minimisation once it has
* started. Rejects on timeout, worker error, or an early exit; the caller treats
* any rejection as "keep the original model" (best-effort layout).
*/
function layoutInWorker(graph: ElkGraph, timeoutMs: number): Promise<ElkGraph> {
return new Promise((resolve, reject) => {
const worker = new Worker(
new URL("./drawio-layout.worker.js", import.meta.url),
{ workerData: { graph } },
);
let settled = false;
const finish = (fn: () => void) => {
if (settled) return;
settled = true;
clearTimeout(timer);
// Always tear the worker down: on the happy path so it does not linger,
// on timeout so the blocked synchronous ELK run is actually interrupted.
void worker.terminate();
fn();
};
const timer = setTimeout(
() => finish(() => reject(new Error("ELK layout timed out"))),
timeoutMs,
);
worker.once("message", (msg: { ok?: boolean; laid?: ElkGraph; error?: string }) => {
finish(() =>
msg?.ok
? resolve(msg.laid as ElkGraph)
: reject(new Error(msg?.error ?? "ELK layout failed")),
);
});
worker.once("error", (err) => finish(() => reject(err)));
worker.once("exit", (code) => {
// A clean exit after we already settled is normal (terminate()); only an
// unexpected early exit while still pending is a failure.
if (settled) return;
finish(() => reject(new Error(`ELK worker exited early (code ${code})`)));
});
});
}
/**
* Apply an ELK layered layout to a drawio input and return a full mxGraphModel
* string with rewritten geometry. Accepts the same three input forms as
* drawioCreate (a bare model, an <mxfile>, or a <mxCell> list). Async because
* elkjs' layout() is promise-based. On any layout failure the ORIGINAL
* (normalized) model is returned unchanged layout is best-effort polish, never
* a reason to fail the write.
* the layout runs on a worker thread. On any layout failure (including a
* terminate-on-timeout) the ORIGINAL (normalized) model is returned unchanged
* layout is best-effort polish, never a reason to fail the write.
*/
export async function applyElkLayout(inputXml: string): Promise<string> {
const modelXml = normalizeInput(inputXml);
@@ -150,26 +205,14 @@ export async function applyElkLayout(inputXml: string): Promise<string> {
};
let laid: ElkGraph;
let timer: ReturnType<typeof setTimeout> | undefined;
try {
// elkjs ships a CJS default export whose interop shape varies across
// module systems; resolve the real constructor at runtime, then cast (the
// runtime call is verified — see the layout unit test).
const Ctor: any = (ELK as any).default ?? ELK;
const elk = new Ctor();
// Race the layout against a wall-clock timeout so a graph that is under the
// node/edge caps but still pathologically slow can never wedge the server.
const timeout = new Promise<never>((_, reject) => {
timer = setTimeout(
() => reject(new Error("ELK layout timed out")),
ELK_TIMEOUT_MS,
);
});
laid = (await Promise.race([elk.layout(graph as any), timeout])) as ElkGraph;
// Run the (synchronous-under-the-hood) ELK pass on a worker thread so the
// main event loop is never blocked, and enforce the wall-clock ceiling by
// terminating that worker on timeout. A graph under the node/edge caps but
// still pathologically slow is hard-killed instead of wedging anything.
laid = await layoutInWorker(graph, resolveElkTimeoutMs());
} catch {
return modelXml; // best-effort: keep the model as-is on timeout or ELK failure
} finally {
if (timer) clearTimeout(timer);
}
// Collect computed geometry per node id (coords are parent-relative already).
@@ -0,0 +1,36 @@
// Worker-thread entry for the ELK layered layout (issue #486, commit 1).
//
// elkjs' layout() returns a Promise but runs the actual crossing-minimisation
// SYNCHRONOUSLY — it blocks whatever thread it runs on for the whole pass. On
// the in-app MCP host that thread used to be the main NestJS event loop, so a
// pathological graph at the node/edge cap could wedge ALL HTTP/SSE/loopback
// traffic while it churned. Running it HERE, on a dedicated worker thread, keeps
// the main loop free; the parent enforces the wall-clock timeout by calling
// `worker.terminate()` — the only way to interrupt synchronous JS — since the
// in-process `setTimeout` race the parent used before could never fire while the
// same thread was blocked inside elkjs.
import { parentPort, workerData } from "node:worker_threads";
import ELK from "elkjs/lib/elk.bundled.js";
interface WorkerInput {
graph: unknown;
}
const { graph } = (workerData ?? {}) as WorkerInput;
// elkjs ships a CJS default export whose interop shape varies across module
// systems; resolve the real constructor at runtime (same as the parent did).
const Ctor: any = (ELK as any).default ?? ELK;
const elk = new Ctor();
elk
.layout(graph as any)
.then((laid: unknown) => {
parentPort?.postMessage({ ok: true, laid });
})
.catch((err: unknown) => {
parentPort?.postMessage({
ok: false,
error: err instanceof Error ? err.message : String(err),
});
});
+33
View File
@@ -10,6 +10,20 @@
const chains = new Map<string, Promise<unknown>>();
// Canonical UUID shape (versions 1–8, matching the `uuid` package's `validate`
// that the server's isValidUUID uses). This is the SINGLE source of truth for
// "is this a canonical page UUID?" in the MCP: client.ts's resolvePageId
// imports isUuid from here to decide whether a pageId already IS a UUID (and so
// needs no /pages/info round-trip). page.repo.ts treats any non-UUID pageId as
// a slugId; a 10-char nanoid slugId never contains dashes, so it can never be
// misread as a UUID here.
export const UUID_RE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
export function isUuid(value: string): boolean {
return typeof value === "string" && UUID_RE.test(value);
}
// The returned promise carries the real result/rejection of `fn` and MUST be
// awaited/handled by the caller; only the internal chaining tail swallows
// errors (purely to gate ordering).
@@ -17,6 +31,25 @@ export function withPageLock<T>(
pageId: string,
fn: () => Promise<T>,
): Promise<T> {
// STRUCTURAL INVARIANT (issue #449, "resolve-then-lock"): the mutex key MUST
// be the canonical page UUID, never a raw slugId. The whole write path relies
// on the lock key AND the CollabSession cache key being the resolved UUID
// (#260) — if a future write method forgot to call resolvePageId and locked
// under a slugId, two writes to the same page would take DIFFERENT mutex keys
// and silently lose serialization (clobbering live human edits). This was an
// invariant enforced only by comments/convention; assert it in CODE so the
// violation fails fast and loud at the lock instead of corrupting data in
// prod. The centralizing helper (mutatePageContent/replacePageContent) already
// guards a raw-input caller, but this backstop catches ANY path.
if (!isUuid(pageId)) {
throw new Error(
`withPageLock: key must be a canonical page UUID, got '${pageId}'. ` +
`The write path must resolvePageId(pageId) BEFORE locking so the ` +
`mutex/CollabSession cache key is the UUID (invariant "resolve-then-` +
`lock", #260/#449). A slugId or other non-UUID key would silently lose ` +
`per-page serialization.`,
);
}
// Wait for the previous op on this page; swallow its error so a failure does
// not poison the queue for the next caller.
const prev = (chains.get(pageId) ?? Promise.resolve()).catch(() => {});
@@ -0,0 +1,137 @@
// Unit tests for the collab-token reset on a Hocuspocus WS auth failure (#486).
//
// Before this fix the cached collab token (#435) was dropped ONLY on an HTTP
// 401/403 (the REST interceptor + login()); a rejected collab-WEBSOCKET handshake
// left the stale token in the cache, so every subsequent mutation re-presented
// the SAME bad token for up to the collab-token TTL (minutes) with no self-heal.
//
// The fix wraps collab writes in `writeWithCollabAuthRetry`: when the write
// rejects with the tagged collab-auth error (collab-session.ts's
// onAuthenticationFailed), it invalidates the cached token and retries the write
// ONCE with a force-refreshed token — symmetric to the HTTP-401 path.
//
// writeWithCollabAuthRetry / getCollabTokenWithReauth are protected in TS but
// plain methods on the compiled build, so the tests call them directly (same
// convention as collab-token-cache.test.mjs).
import { test, afterEach } from "node:test";
import assert from "node:assert/strict";
import { DocmostClient } from "../../build/client.js";
const ENV_KEY = "MCP_COLLAB_TOKEN_TTL_MS";
afterEach(() => {
delete process.env[ENV_KEY];
});
// A counting provider that returns a distinct token each call so a cached
// (reused) token is visibly the SAME string while a fresh mint is different.
function countingProvider() {
let n = 0;
const fn = async () => {
n++;
return `provider-token-${n}`;
};
return {
fn,
get calls() {
return n;
},
};
}
// The tagged error collab-session.ts throws on a rejected WS handshake.
function collabAuthError() {
const err = new Error("Authentication failed for collaboration connection");
err.collabAuthFailed = true;
return err;
}
test("a WS auth failure clears the cached token and retries the write with a FRESH one (#486)", async () => {
process.env[ENV_KEY] = "300000"; // 5 min: the cache is warm across the burst.
const p = countingProvider();
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
getCollabToken: p.fn,
});
// Warm the cache the way a real write would (mints provider-token-1).
const initial = await client.getCollabTokenWithReauth();
assert.equal(initial, "provider-token-1");
assert.equal(p.calls, 1);
const tokensSeen = [];
const write = async (token) => {
tokensSeen.push(token);
// The FIRST attempt (with the stale cached token) fails the WS handshake;
// the retry (with a fresh token) succeeds.
if (tokensSeen.length === 1) throw collabAuthError();
return `written-with:${token}`;
};
const result = await client.writeWithCollabAuthRetry(initial, write);
assert.equal(tokensSeen.length, 2, "write attempted exactly twice (one retry)");
assert.equal(tokensSeen[0], "provider-token-1", "first attempt used the stale token");
assert.equal(
tokensSeen[1],
"provider-token-2",
"retry used a FRESH force-refreshed token, not the stale cached one",
);
assert.equal(result, "written-with:provider-token-2", "the retry's result wins");
assert.equal(p.calls, 2, "exactly one extra mint for the retry — no loop");
// The cache now holds the fresh token, so a subsequent op reuses it (proving
// the stale token was evicted and the fresh one cached, not re-minted).
const next = await client.getCollabTokenWithReauth();
assert.equal(next, "provider-token-2", "the fresh token replaced the stale cache");
assert.equal(p.calls, 2, "served from cache — provider not re-invoked");
});
test("a successful write is NOT retried and mints nothing extra", async () => {
process.env[ENV_KEY] = "300000";
const p = countingProvider();
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
getCollabToken: p.fn,
});
const initial = await client.getCollabTokenWithReauth(); // provider-token-1
let attempts = 0;
const result = await client.writeWithCollabAuthRetry(initial, async (token) => {
attempts++;
return `ok:${token}`;
});
assert.equal(attempts, 1, "no retry on success");
assert.equal(result, "ok:provider-token-1");
assert.equal(p.calls, 1, "no extra mint");
});
test("a NON-auth write error propagates unchanged (no reset, no retry)", async () => {
process.env[ENV_KEY] = "300000";
const p = countingProvider();
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
getCollabToken: p.fn,
});
const initial = await client.getCollabTokenWithReauth(); // provider-token-1
let attempts = 0;
await assert.rejects(
client.writeWithCollabAuthRetry(initial, async () => {
attempts++;
throw new Error("collab connection closed before persist"); // NOT tagged.
}),
/closed before persist/,
);
assert.equal(attempts, 1, "a non-auth error is not retried");
assert.equal(p.calls, 1, "the cache is untouched -> no fresh mint");
// Cache still holds the original token (was never invalidated).
const still = await client.getCollabTokenWithReauth();
assert.equal(still, "provider-token-1");
assert.equal(p.calls, 1);
});
@@ -0,0 +1,55 @@
// Unit test: listPages tree mode must propagate the `truncated` flag (#486).
//
// enumerateSpacePages returns { pages, truncated } — truncated is true ONLY when
// the stdio-fallback BFS hit its node cap (the primary /pages/tree path is
// uncapped). The old tree-mode listPages destructured only `pages` and returned a
// bare tree, dropping `truncated`, so a caller handed an INCOMPLETE tree had no
// way to know pages were missing. The fix returns { tree, truncated } (same
// pattern check_new_comments uses).
//
// Reaching the real cap (MAX_NODES = 10000) in a mock is impractical, so we stub
// enumerateSpacePages directly to assert the flag is threaded through verbatim.
import { test } from "node:test";
import assert from "node:assert/strict";
import { DocmostClient } from "../../build/client.js";
function stubClient() {
const client = new DocmostClient({
apiUrl: "http://127.0.0.1:1/api",
getToken: async () => "access",
});
// No network: the tree path only calls ensureAuthenticated + enumerateSpacePages.
client.ensureAuthenticated = async () => {};
return client;
}
const onePage = [{ id: "r1", title: "Root", parentPageId: null }];
test("tree mode carries truncated:true when the enumeration truncated (#486)", async () => {
const client = stubClient();
client.enumerateSpacePages = async () => ({ pages: onePage, truncated: true });
const res = await client.listPages("space-1", 50, true);
assert.equal(res.truncated, true, "the truncated flag is threaded through");
assert.ok(Array.isArray(res.tree), "the built tree rides alongside the flag");
assert.equal(res.tree[0].id, "r1");
});
test("tree mode carries truncated:false for a complete enumeration", async () => {
const client = stubClient();
client.enumerateSpacePages = async () => ({ pages: onePage, truncated: false });
const res = await client.listPages("space-1", 50, true);
assert.equal(res.truncated, false);
assert.equal(res.tree[0].id, "r1");
});
test("tree mode still requires a spaceId", async () => {
const client = stubClient();
await assert.rejects(
client.listPages(undefined, 50, true),
/tree mode requires a spaceId/,
);
});
@@ -184,12 +184,14 @@ test("enumerateSpacePages (via listPages tree) uses one /pages/tree request", as
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// listPages tree:true -> enumerateSpacePages(spaceId) -> buildPageTree.
const tree = await client.listPages("space-1", 50, true);
// listPages tree:true -> enumerateSpacePages(spaceId) -> { tree, truncated }.
const { tree, truncated } = await client.listPages("space-1", 50, true);
assert.equal(treeRequests, 1, "exactly one /pages/tree request for the space");
assert.equal(sidebarRequests, 0, "no per-node sidebar BFS requests");
assert.deepEqual(treeBody, { spaceId: "space-1" }, "space scope posts spaceId only");
// The uncapped /pages/tree path is never truncated (#486).
assert.equal(truncated, false, "primary /pages/tree path is not truncated");
// buildPageTree nests c1 under r1; two roots at the top level.
assert.equal(tree.length, 2, "two root nodes");
const r1 = tree.find((n) => n.id === "r1");
@@ -249,7 +251,7 @@ test("enumerateSpacePages falls back to the cursor BFS on /pages/tree 404", asyn
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const tree = await client.listPages("space-1", 50, true);
const { tree, truncated } = await client.listPages("space-1", 50, true);
assert.ok(treeRequests >= 1, "the tree endpoint was attempted first");
assert.deepEqual(
@@ -257,6 +259,8 @@ test("enumerateSpacePages falls back to the cursor BFS on /pages/tree 404", asyn
["<root>", "r1"],
"fell back to the sidebar BFS: roots then the root's children",
);
// Small fallback walk well under the node cap -> not truncated (#486).
assert.equal(truncated, false, "fallback BFS below the cap is not truncated");
assert.equal(tree.length, 1, "one root in the built tree");
assert.equal(tree[0].children[0].id, "c1", "leaf nested via the BFS");
});
@@ -372,7 +372,11 @@ test("replaceImage opens by the resolved UUID AND keys its page lock by that UUI
// single flush. This proves the flush actually executes queued callbacks, so
// probeRan === false above means "blocked", not "the flush never ran anyone".
let freeRan = false;
const freeDone = withPageLock(`page.free-${UUID}`, async () => {
// A DIFFERENT canonical UUID (unrelated to the page under test). withPageLock
// now asserts its key is a canonical UUID (#449), so the "free" probe key must
// also be a valid — but distinct — UUID, not a synthetic label.
const FREE_UUID = "99999999-9999-4999-8999-999999999999";
const freeDone = withPageLock(FREE_UUID, async () => {
freeRan = true;
});
await new Promise((r) => setImmediate(r));
@@ -323,7 +323,9 @@ test("MCP_COLLAB_SESSION_IDLE_MS=0 disables the cache (legacy provider-per-op)",
});
test("replaceImage-shaped flow: acquire under an EXTERNAL page lock does not deadlock and reuses one session", async () => {
const pageId = "page-lock";
// withPageLock now asserts a canonical UUID key (#449); this flow takes the
// real page lock (mirroring replaceImage), so the key must be a valid UUID.
const pageId = "77777777-7777-4777-8777-777777777777";
// Mirror replaceImage: hold ONE withPageLock across scan (read-only) + write,
// each going through the non-locking acquireCollabSession.
const result = await withPageLock(pageId, async () => {
@@ -101,6 +101,88 @@ test("DoS guard: a graph over the node cap is returned unchanged, quickly", asyn
assert.ok(dt < 2000, `cap path should be fast, took ${dt}ms`);
});
/** Build a layered DAG near the caps: `n` vertices, up to ~2 edges each into the
* next layer of `layerSize`. Used as a real worst-case graph for the benchmark. */
function layeredGraph(n, layerSize) {
let cells = "";
for (let i = 2; i < 2 + n; i++) {
cells +=
`<mxCell id="${i}" value="N${i}" style="rounded=1;html=1;" vertex="1" parent="1">` +
`<mxGeometry x="10" y="10" width="120" height="60" as="geometry"/></mxCell>`;
}
let ei = 0;
for (let i = 2; i < 2 + n; i++) {
for (const off of [layerSize, layerSize + 1]) {
const t = i + off;
if (t < 2 + n) cells += `<mxCell id="e${ei++}" edge="1" parent="1" source="${i}" target="${t}"><mxGeometry relative="1" as="geometry"/></mxCell>`;
}
}
return (
'<mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/>' +
cells +
"</root></mxGraphModel>"
);
}
test("terminate-on-timeout: a layout that exceeds the wall-clock ceiling is hard-killed and the original model is returned (#486)", async () => {
// A 1ms ceiling fires before the worker can even finish loading elkjs, so the
// parent must terminate() the worker and fall back to the ORIGINAL model. On
// the OLD in-process race this timer could never fire while the SAME thread was
// blocked inside elkjs — the fallback path was unreachable; here it works.
const prev = process.env.DRAWIO_ELK_TIMEOUT_MS;
process.env.DRAWIO_ELK_TIMEOUT_MS = "1";
try {
const model = layeredGraph(400, 20);
const t0 = Date.now();
const laid = await applyElkLayout(model);
const dt = Date.now() - t0;
// Original geometry is preserved verbatim: every vertex is still stacked at
// (10,10), proving NO ELK coordinates were applied (the pass was killed).
const verts = parseCells(laid).filter((c) => c.vertex);
assert.equal(verts.length, 400, "all vertices survived the fallback");
for (const v of verts) {
assert.equal(v.geometry.x, 10, "x untouched -> layout was terminated");
assert.equal(v.geometry.y, 10, "y untouched -> layout was terminated");
}
// The kill is prompt: terminate() returns the call well under the natural
// layout time for a 400-node graph.
assert.ok(dt < 2000, `terminate path should be prompt, took ${dt}ms`);
} finally {
if (prev === undefined) delete process.env.DRAWIO_ELK_TIMEOUT_MS;
else process.env.DRAWIO_ELK_TIMEOUT_MS = prev;
}
});
test("benchmark guard: a worst-case graph AT the cap lays out without wedging the main event loop (#486)", async () => {
// ~500 nodes / ~1000 edges — a real worst case at the node/edge caps. The
// layout runs on a WORKER thread, so the MAIN event loop must stay responsive
// throughout: a timer scheduled on the main thread keeps firing while ELK
// churns. On the OLD synchronous-on-main-thread code this counter would be
// pinned at 0 for the whole layout (event loop wedged) — exactly the prod fire.
const model = layeredGraph(500, 20);
let mainLoopTicks = 0;
const iv = setInterval(() => {
mainLoopTicks++;
}, 2);
const t0 = Date.now();
const laid = await applyElkLayout(model);
const dt = Date.now() - t0;
clearInterval(iv);
assert.ok(
mainLoopTicks > 0,
"main event loop must stay responsive while ELK runs on the worker",
);
// Benchmark guard: the worst-case graph actually LAYS OUT within the default
// ceiling (it did not fall back). At least one vertex moved off the stack.
const verts = parseCells(laid).filter((c) => c.vertex);
assert.equal(verts.length, 500, "all vertices survived");
const moved = verts.some((v) => v.geometry.x !== 10 || v.geometry.y !== 10);
assert.ok(moved, "layout was applied (did not time out / fall back)");
// Sanity ceiling well under the 5s wall-clock timeout.
assert.ok(dt < 5000, `worst-case layout should be under the ceiling, took ${dt}ms`);
});
test("layout is best-effort: an empty/degenerate model is returned intact", async () => {
const model =
'<mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/></root></mxGraphModel>';
@@ -0,0 +1,84 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
// Issue #449, invariant 2 ("no-await-окно"): the atomicity of the
// read -> transform -> write section in CollabSession.mutate depends on there
// being NO `await` (nor any other async yield point) between
// `TiptapTransformer.fromYdoc` and `applyDocToFragment`. Yjs applies queued
// remote updates only when the event loop yields, so an accidental await in that
// window would let a concurrent human edit interleave and be clobbered (#152).
//
// That was an invariant enforced only by a comment. This test turns a violation
// RED: it reads the SOURCE of collab-session.ts, extracts the block delimited by
// the machine-readable BEGIN/END markers, and asserts no async boundary appears
// inside it. Introducing an `await` (or `for await`, or `yield`) between the
// markers fails this test.
const here = dirname(fileURLToPath(import.meta.url));
// Scan the .ts SOURCE (not the compiled .js): the markers live in the source and
// transpilation could rewrite/erase them, so the source is the authoritative
// artifact the human edits.
const sourcePath = join(here, "..", "..", "src", "lib", "collab-session.ts");
const source = readFileSync(sourcePath, "utf8");
const BEGIN = "=== MUTATE-CRITICAL-WINDOW: BEGIN";
const END = "=== MUTATE-CRITICAL-WINDOW: END";
test("critical-window markers exist exactly once each", () => {
const begins = source.split(BEGIN).length - 1;
const ends = source.split(END).length - 1;
assert.equal(
begins,
1,
`expected exactly one '${BEGIN}' marker, found ${begins}`,
);
assert.equal(ends, 1, `expected exactly one '${END}' marker, found ${ends}`);
});
test("the read->write critical window contains no async boundary (no await/yield)", () => {
const beginIdx = source.indexOf(BEGIN);
const endIdx = source.indexOf(END);
assert.ok(beginIdx !== -1, "BEGIN marker not found");
assert.ok(endIdx !== -1, "END marker not found");
assert.ok(endIdx > beginIdx, "END marker must come after BEGIN marker");
// The block strictly between the two marker lines. Move past the end of the
// BEGIN marker line so the marker comment text itself is not scanned.
const afterBeginLine = source.indexOf("\n", beginIdx) + 1;
const block = source.slice(afterBeginLine, endIdx);
// Detect any real async yield keyword as a whole word. `\bawait\b` also matches
// inside `for await`, which is exactly what we want to forbid here.
const forbidden = [/\bawait\b/, /\byield\b/];
for (const re of forbidden) {
const m = block.match(re);
assert.equal(
m,
null,
`forbidden async boundary '${m?.[0]}' found inside the no-await critical ` +
`window of CollabSession.mutate. INVARIANT 1 (#449): the block between ` +
`TiptapTransformer.fromYdoc and applyDocToFragment must be fully ` +
`synchronous — an await there reopens the clobber-live-edits race (#152).`,
);
}
});
test("the critical window still spans fromYdoc -> applyDocToFragment", () => {
// Guards the markers from drifting off the code they are meant to protect: if
// someone moves the read/write out of the window, this catches it.
const beginIdx = source.indexOf(BEGIN);
const endIdx = source.indexOf(END);
const afterBeginLine = source.indexOf("\n", beginIdx) + 1;
const block = source.slice(afterBeginLine, endIdx);
assert.ok(
block.includes("TiptapTransformer.fromYdoc"),
"critical window must contain the TiptapTransformer.fromYdoc read",
);
assert.ok(
block.includes("applyDocToFragment"),
"critical window must contain the applyDocToFragment write",
);
});
+74 -8
View File
@@ -1,13 +1,26 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { withPageLock } from "../../build/lib/page-lock.js";
import { withPageLock, isUuid } from "../../build/lib/page-lock.js";
const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
// withPageLock now asserts its key is a canonical UUID (#449, "resolve-then-
// lock"), so the mechanics tests below must lock under real UUIDs, not arbitrary
// labels. Distinct valid UUIDv7-shaped ids for the distinct-page cases.
const U = {
same: "11111111-1111-7111-8111-111111111111",
ordered: "22222222-2222-7222-8222-222222222222",
poison: "33333333-3333-7333-8333-333333333333",
poison2: "44444444-4444-7444-8444-444444444444",
A: "aaaaaaaa-aaaa-7aaa-8aaa-aaaaaaaaaaaa",
B: "bbbbbbbb-bbbb-7bbb-8bbb-bbbbbbbbbbbb",
leak: "55555555-5555-7555-8555-555555555555",
};
test("two ops on the same pageId run strictly sequentially (no overlap)", async () => {
const events = [];
const pageId = "same-page";
const pageId = U.same;
const p1 = withPageLock(pageId, async () => {
events.push("start-1");
@@ -33,7 +46,7 @@ test("two ops on the same pageId run strictly sequentially (no overlap)", async
});
test("same pageId ordering holds for many queued ops", async () => {
const pageId = "ordered-page";
const pageId = U.ordered;
const order = [];
const active = { count: 0, maxConcurrent: 0 };
@@ -60,7 +73,7 @@ test("same pageId ordering holds for many queued ops", async () => {
});
test("a rejecting op does not poison the chain for the same page", async () => {
const pageId = "poison-page";
const pageId = U.poison;
const events = [];
const failing = withPageLock(pageId, async () => {
@@ -87,7 +100,7 @@ test("a rejecting op does not poison the chain for the same page", async () => {
});
test("failing op queued before a success both resolve/reject correctly", async () => {
const pageId = "poison-page-2";
const pageId = U.poison2;
const order = [];
const failing = withPageLock(pageId, async () => {
@@ -111,14 +124,14 @@ test("failing op queued before a success both resolve/reject correctly", async (
test("ops on different pageIds run concurrently (overlap)", async () => {
const events = [];
const pA = withPageLock("page-A", async () => {
const pA = withPageLock(U.A, async () => {
events.push("A-start");
await delay(40);
events.push("A-end");
return "A";
});
const pB = withPageLock("page-B", async () => {
const pB = withPageLock(U.B, async () => {
events.push("B-start");
await delay(10);
events.push("B-end");
@@ -134,7 +147,7 @@ test("ops on different pageIds run concurrently (overlap)", async () => {
});
test("no functional leak: many sequential ops on same page keep working", async () => {
const pageId = "leak-page";
const pageId = U.leak;
// Run a long series of fully sequential ops (each awaited before the next is
// queued) so the internal map entry is created and dropped repeatedly.
@@ -151,3 +164,56 @@ test("no functional leak: many sequential ops on same page keep working", async
const final = await withPageLock(pageId, async () => "still-works");
assert.equal(final, "still-works");
});
// --- Issue #449: fail-fast on a non-canonical lock key ---------------------
// A write method that reaches the lock path with an unresolved slugId (or any
// non-UUID key) must fail IMMEDIATELY and LOUDLY, not lock under a split key and
// silently lose per-page serialization. These assert withPageLock rejects such
// a key before ever running fn.
test("withPageLock throws on a raw 10-char slugId (unresolved key)", () => {
let ran = false;
assert.throws(
() =>
withPageLock("p7Xk29Lm4Q", async () => {
ran = true;
return "should-not-run";
}),
/canonical page UUID|resolve-then-lock/i,
"a slugId key must fail-fast at the lock",
);
// The work must NOT have started: fail-fast means no serialization was
// silently skipped under a bad key.
assert.equal(ran, false, "fn must not run when the key is rejected");
});
test("withPageLock throws on other non-UUID keys (label, empty, non-string)", () => {
for (const bad of ["same-page", "", "not-a-uuid", "1234"]) {
assert.throws(
() => withPageLock(bad, async () => "x"),
/canonical page UUID/i,
`expected withPageLock to reject key ${JSON.stringify(bad)}`,
);
}
// A non-string key is also rejected (guards a mistyped call site).
assert.throws(
() => withPageLock(/** @type {any} */ (undefined), async () => "x"),
/canonical page UUID/i,
);
});
test("withPageLock accepts a canonical UUID key (no false positive)", async () => {
const uuid = "0192f3a4-b5c6-7d8e-9f01-23456789abcd";
assert.equal(isUuid(uuid), true);
const r = await withPageLock(uuid, async () => "ok");
assert.equal(r, "ok");
});
test("isUuid discriminates UUIDs from slugIds (shared predicate)", () => {
// The predicate withPageLock asserts on is the SAME one resolvePageId uses to
// decide whether a pageId is already a UUID (imported from page-lock).
assert.equal(isUuid("0192f3a4-b5c6-7d8e-9f01-23456789abcd"), true);
assert.equal(isUuid("p7Xk29Lm4Q"), false); // 10-char nanoid slugId
assert.equal(isUuid("not-a-uuid"), false);
assert.equal(isUuid(""), false);
});
+189 -70
View File
@@ -1,101 +1,220 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import {
mkdtempSync,
mkdirSync,
writeFileSync,
rmSync,
readdirSync,
statSync,
readFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
import { dirname, join, relative, sep } from "node:path";
import { createHash } from "node:crypto";
import { computeRegistryStamp } from "../../scripts/gen-registry-stamp.mjs";
import { REGISTRY_STAMP } from "../../build/index.js";
// Guard tests for the build/src-skew stamp (issue #447). The codegen script
// exports `computeRegistryStamp(sourceText)` — a sha256 over normalized source
// text (CRLF->LF, single trailing newline stripped). The in-app loader
// (apps/server/.../docmost-client.loader.ts) DUPLICATES that normalize+sha256 to
// recompute the stamp from src and refuse a stale build. These tests pin the
// algorithm's behaviour AND assert the built stamp matches the current src, so a
// stale generated file OR a normalize divergence reddens.
// Guard tests for the build/src-skew stamp (issues #447/#486). The codegen script
// exports `computeRegistryStamp(srcDir)` — a sha256 over the WHOLE src/ tree
// (every src/**/*.ts EXCEPT *.generated.ts), each file folded in as its
// POSIX-relative path + its normalized content (CRLF->LF, single trailing newline
// stripped). Hashing the whole tree (not just tool-specs.ts) is #486: an edit to
// client.ts / a client/* module without a rebuild must ALSO redden. The in-app
// loader (apps/server/.../docmost-client.loader.ts) DUPLICATES this enumerate+
// normalize+sha256 to refuse a stale build. These tests pin the algorithm and
// assert the built stamp matches the current src.
const __dirname = dirname(fileURLToPath(import.meta.url));
const TOOL_SPECS_PATH = join(__dirname, "..", "..", "src", "tool-specs.ts");
const SRC_DIR = join(__dirname, "..", "..", "src");
test("computeRegistryStamp is deterministic: same input -> same hash", () => {
const input = "export const X = 1;\nexport const Y = 2;\n";
assert.equal(computeRegistryStamp(input), computeRegistryStamp(input));
// Build a throwaway src/ tree from a { relPath: content } map and return its dir.
function makeSrcTree(files) {
const root = mkdtempSync(join(tmpdir(), "mcp-stamp-tree-"));
const src = join(root, "src");
for (const [rel, content] of Object.entries(files)) {
const full = join(src, rel);
mkdirSync(dirname(full), { recursive: true });
writeFileSync(full, content, "utf8");
}
return { src, cleanup: () => rmSync(root, { recursive: true, force: true }) };
}
test("computeRegistryStamp is deterministic: same tree -> same hash", () => {
const a = makeSrcTree({ "tool-specs.ts": "export const X = 1;\n" });
const b = makeSrcTree({ "tool-specs.ts": "export const X = 1;\n" });
try {
assert.equal(computeRegistryStamp(a.src), computeRegistryStamp(b.src));
} finally {
a.cleanup();
b.cleanup();
}
});
test("computeRegistryStamp returns a 64-char lowercase hex sha256", () => {
const stamp = computeRegistryStamp("anything");
assert.match(stamp, /^[0-9a-f]{64}$/);
const t = makeSrcTree({ "tool-specs.ts": "anything\n" });
try {
assert.match(computeRegistryStamp(t.src), /^[0-9a-f]{64}$/);
} finally {
t.cleanup();
}
});
test("normalizes CRLF vs LF: the same content hashes equal", () => {
const lf = "line1\nline2\nline3";
const crlf = "line1\r\nline2\r\nline3";
assert.equal(computeRegistryStamp(crlf), computeRegistryStamp(lf));
// #486 CORE: an edit to a NON-tool-specs source file (client.ts) must change the
// stamp. Under the old single-file (tool-specs.ts only) hash this edit was
// invisible and a stale build/ served the old client.ts silently.
test("editing client.ts (not tool-specs.ts) changes the stamp (#486)", () => {
const before = makeSrcTree({
"tool-specs.ts": "export const SPECS = 1;\n",
"client.ts": "export const impl = 'v1';\n",
});
const after = makeSrcTree({
"tool-specs.ts": "export const SPECS = 1;\n",
"client.ts": "export const impl = 'v2';\n",
});
try {
assert.notEqual(
computeRegistryStamp(before.src),
computeRegistryStamp(after.src),
"a client.ts edit with an unchanged tool-specs.ts must move the stamp",
);
} finally {
before.cleanup();
after.cleanup();
}
});
test("normalizes a trailing newline: with/without a final \\n hashes equal", () => {
const noTrailing = "alpha\nbeta";
const trailing = "alpha\nbeta\n";
assert.equal(computeRegistryStamp(trailing), computeRegistryStamp(noTrailing));
test("editing a nested client/* module changes the stamp", () => {
const before = makeSrcTree({
"tool-specs.ts": "x\n",
"client/read.ts": "export const READ = 1;\n",
});
const after = makeSrcTree({
"tool-specs.ts": "x\n",
"client/read.ts": "export const READ = 2;\n",
});
try {
assert.notEqual(
computeRegistryStamp(before.src),
computeRegistryStamp(after.src),
);
} finally {
before.cleanup();
after.cleanup();
}
});
test("a CRLF checkout WITH a trailing CRLF still hashes equal to bare LF", () => {
// A worst-case Windows checkout: CRLF line endings + a trailing CRLF. Both the
// \r\n->\n replace and the trailing-newline strip must apply for parity.
const bare = "alpha\nbeta";
const crlfTrailing = "alpha\r\nbeta\r\n";
assert.equal(
computeRegistryStamp(crlfTrailing),
computeRegistryStamp(bare),
);
// *.generated.ts is EXCLUDED (else the codegen's own output is a fixed-point
// cycle): adding/removing/changing it must not move the stamp.
test("*.generated.ts is excluded from the stamp", () => {
const without = makeSrcTree({ "tool-specs.ts": "x\n" });
const withGen = makeSrcTree({
"tool-specs.ts": "x\n",
"registry-stamp.generated.ts": 'export const REGISTRY_STAMP = "abc";\n',
});
try {
assert.equal(
computeRegistryStamp(without.src),
computeRegistryStamp(withGen.src),
"a *.generated.ts file must not affect the stamp",
);
} finally {
without.cleanup();
withGen.cleanup();
}
});
test("a real content change hashes differently", () => {
const before = "export const description = 'search a page';\n";
const after = "export const description = 'search a PAGE';\n";
assert.notEqual(computeRegistryStamp(before), computeRegistryStamp(after));
test("a CRLF checkout WITH trailing CRLF hashes equal to bare LF", () => {
const bare = makeSrcTree({ "tool-specs.ts": "alpha\nbeta" });
const crlfTrailing = makeSrcTree({ "tool-specs.ts": "alpha\r\nbeta\r\n" });
try {
assert.equal(
computeRegistryStamp(crlfTrailing.src),
computeRegistryStamp(bare.src),
);
} finally {
bare.cleanup();
crlfTrailing.cleanup();
}
});
// Only a SINGLE trailing newline is stripped — a second blank line is content and
// must change the hash. This pins the exact `/\n$/` semantics the loader mirrors.
// Only a SINGLE trailing newline is stripped — a second blank line is content.
test("only ONE trailing newline is stripped (two differ from one)", () => {
assert.notEqual(
computeRegistryStamp("x\n"),
computeRegistryStamp("x\n\n"),
);
const one = makeSrcTree({ "tool-specs.ts": "x\n" });
const two = makeSrcTree({ "tool-specs.ts": "x\n\n" });
try {
assert.notEqual(
computeRegistryStamp(one.src),
computeRegistryStamp(two.src),
);
} finally {
one.cleanup();
two.cleanup();
}
});
// Cross-impl equality against a fixed, documented input. The SAME literal input
// and expected hash are asserted in the server-side jest test
// (docmost-client.loader.spec.ts). If either side's normalize+sha256 ever
// diverges, one of the two tests reddens. Input exercises BOTH normalize steps.
test("fixed-input hash matches the documented cross-impl value", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const EXPECTED =
"683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83";
assert.equal(computeRegistryStamp(FIXED_INPUT), EXPECTED);
// Cross-impl equality against a fixed, documented tree. The SAME literal tree and
// expected hash are asserted in the server-side jest test
// (docmost-client.loader.spec.ts). If either side's enumerate+normalize+sha256
// ever diverges, one of the two tests reddens. The tree exercises: a nested file,
// BOTH normalize steps (tool-specs.ts uses CRLF + trailing \n) and the
// *.generated.ts exclusion.
const CROSS_IMPL_TREE = {
"tool-specs.ts": "line1\r\nline2\n",
"client/read.ts": "export const R = 1;\n",
"registry-stamp.generated.ts": 'export const REGISTRY_STAMP="ignored";\n',
};
const CROSS_IMPL_EXPECTED =
"131c1b9e4e2f5a7d6cef91ca8df619822b442f52bc45ebd09474a4c1d6728616";
test("fixed-tree hash matches the documented cross-impl value", () => {
const t = makeSrcTree(CROSS_IMPL_TREE);
try {
assert.equal(computeRegistryStamp(t.src), CROSS_IMPL_EXPECTED);
} finally {
t.cleanup();
}
});
// DESYNC GUARD (covers reviewer suggestion 2). Recompute the stamp from the
// actual src/tool-specs.ts and assert it equals the REGISTRY_STAMP baked into the
// freshly-built build/index.js. This reddens if the generated file is stale OR if
// the codegen normalize ever diverges from what produced the built stamp.
test("built REGISTRY_STAMP equals the stamp recomputed from src/tool-specs.ts", () => {
const source = readFileSync(TOOL_SPECS_PATH, "utf8");
assert.equal(computeRegistryStamp(source), REGISTRY_STAMP);
// Sanity: the EXPECTED constant is not a magic value but the documented
// enumerate+normalize+sha256 of CROSS_IMPL_TREE (a local re-implementation).
test("the documented EXPECTED is the enumerate+normalize+sha256 of the tree", () => {
const t = makeSrcTree(CROSS_IMPL_TREE);
try {
const collect = (dir) => {
const out = [];
for (const e of readdirSync(dir)) {
const f = join(dir, e);
if (statSync(f).isDirectory()) out.push(...collect(f));
else if (e.endsWith(".ts") && !e.endsWith(".generated.ts")) out.push(f);
}
return out;
};
const files = collect(t.src)
.map((abs) => ({ rel: relative(t.src, abs).split(sep).join("/"), abs }))
.sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
const h = createHash("sha256");
for (const { rel, abs } of files) {
const n = readFileSync(abs, "utf8")
.replace(/\r\n/g, "\n")
.replace(/\n$/, "");
h.update(rel, "utf8");
h.update("\0", "utf8");
h.update(n, "utf8");
h.update("\0", "utf8");
}
assert.equal(h.digest("hex"), CROSS_IMPL_EXPECTED);
} finally {
t.cleanup();
}
});
// Sanity: the fixed-input helper computes the SAME way the codegen does, proving
// the EXPECTED constant above is not an arbitrary magic value but the documented
// normalize+sha256 of FIXED_INPUT. Belt-and-braces so a bad EXPECTED can't hide a
// real regression.
test("the documented EXPECTED constant is the normalize+sha256 of FIXED_INPUT", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const normalized = FIXED_INPUT.replace(/\r\n/g, "\n").replace(/\n$/, "");
const expected = createHash("sha256")
.update(normalized, "utf8")
.digest("hex");
assert.equal(computeRegistryStamp(FIXED_INPUT), expected);
// DESYNC GUARD. Recompute the stamp from the REAL src/ tree and assert it equals
// the REGISTRY_STAMP baked into the freshly-built build/index.js. This reddens if
// the generated file is stale OR if the codegen ever diverges from what produced
// the built stamp.
test("built REGISTRY_STAMP equals the stamp recomputed from src/", () => {
assert.equal(computeRegistryStamp(SRC_DIR), REGISTRY_STAMP);
});
+114 -6
View File
@@ -1,8 +1,62 @@
diff --git a/dist/index.js b/dist/index.js
index ae447a12f7823ec0a00837ee9f0eb809a610d5f8..a3402b2c2d021ef432cfa76e35d370073d525135 100644
index ae447a12f7823ec0a00837ee9f0eb809a610d5f8..210b5a9009e5cf1537cb2007c524007c1a01253c 100644
--- a/dist/index.js
+++ b/dist/index.js
@@ -6578,9 +6578,19 @@ function createOutputTransformStream(output) {
@@ -5036,9 +5036,40 @@ function writeToServerResponse({
break;
const canContinue = response.write(value);
if (!canContinue) {
- await new Promise((resolve3) => {
- response.once("drain", resolve3);
+ // PATCH(docmost #486): race "drain" against "close"/"error". The
+ // original awaited ONLY "drain", so a client that disconnected mid-write
+ // (the socket never drains) parked this loop FOREVER: the finally never
+ // ran, response.end() was unreachable, and the reader + buffered chunks
+ // were held until process restart. On close/error we cancel the reader
+ // and break so the finally always runs (safe for detached runs:
+ // consumeStream drains the SDK stream independently). Listener hygiene:
+ // all three once-listeners are removed on the first settle, so they
+ // cannot pile up one-per-stall.
+ const closed = await new Promise((resolve3) => {
+ function finish(isClosed) {
+ response.removeListener("drain", onDrain);
+ response.removeListener("close", onClose);
+ response.removeListener("error", onError);
+ resolve3(isClosed);
+ }
+ function onDrain() {
+ finish(false);
+ }
+ function onClose() {
+ finish(true);
+ }
+ function onError() {
+ finish(true);
+ }
+ response.once("drain", onDrain);
+ response.once("close", onClose);
+ response.once("error", onError);
});
+ if (closed) {
+ await reader.cancel().catch(() => {
+ });
+ break;
+ }
}
}
} catch (error) {
@@ -5047,7 +5078,9 @@ function writeToServerResponse({
response.end();
}
};
- read();
+ read().catch((error) => {
+ console.error("ai writeToServerResponse read() failed:", error);
+ });
}
// src/text-stream/pipe-text-stream-to-response.ts
@@ -6578,9 +6611,19 @@ function createOutputTransformStream(output) {
controller.enqueue({ part: chunk, partialOutput: void 0 });
return;
}
@@ -23,7 +77,7 @@ index ae447a12f7823ec0a00837ee9f0eb809a610d5f8..a3402b2c2d021ef432cfa76e35d37007
const result = await output.parsePartialOutput({ text: text2 });
if (result !== void 0) {
const currentJson = JSON.stringify(result.partial);
@@ -6959,7 +6969,7 @@ var DefaultStreamTextResult = class {
@@ -6959,7 +7002,7 @@ var DefaultStreamTextResult = class {
})
);
}
@@ -33,10 +87,64 @@ index ae447a12f7823ec0a00837ee9f0eb809a610d5f8..a3402b2c2d021ef432cfa76e35d37007
maxRetries: maxRetriesArg,
abortSignal
diff --git a/dist/index.mjs b/dist/index.mjs
index 663875332e3f9a9bd167c25583c515876f42951b..b840b0502c9894df983e0154805abb80e70e6331 100644
index 663875332e3f9a9bd167c25583c515876f42951b..a51514a390d22811a407edd8b703e21793586cc8 100644
--- a/dist/index.mjs
+++ b/dist/index.mjs
@@ -6501,9 +6501,19 @@ function createOutputTransformStream(output) {
@@ -4957,9 +4957,40 @@ function writeToServerResponse({
break;
const canContinue = response.write(value);
if (!canContinue) {
- await new Promise((resolve3) => {
- response.once("drain", resolve3);
+ // PATCH(docmost #486): race "drain" against "close"/"error". The
+ // original awaited ONLY "drain", so a client that disconnected mid-write
+ // (the socket never drains) parked this loop FOREVER: the finally never
+ // ran, response.end() was unreachable, and the reader + buffered chunks
+ // were held until process restart. On close/error we cancel the reader
+ // and break so the finally always runs (safe for detached runs:
+ // consumeStream drains the SDK stream independently). Listener hygiene:
+ // all three once-listeners are removed on the first settle, so they
+ // cannot pile up one-per-stall.
+ const closed = await new Promise((resolve3) => {
+ function finish(isClosed) {
+ response.removeListener("drain", onDrain);
+ response.removeListener("close", onClose);
+ response.removeListener("error", onError);
+ resolve3(isClosed);
+ }
+ function onDrain() {
+ finish(false);
+ }
+ function onClose() {
+ finish(true);
+ }
+ function onError() {
+ finish(true);
+ }
+ response.once("drain", onDrain);
+ response.once("close", onClose);
+ response.once("error", onError);
});
+ if (closed) {
+ await reader.cancel().catch(() => {
+ });
+ break;
+ }
}
}
} catch (error) {
@@ -4968,7 +4999,9 @@ function writeToServerResponse({
response.end();
}
};
- read();
+ read().catch((error) => {
+ console.error("ai writeToServerResponse read() failed:", error);
+ });
}
// src/text-stream/pipe-text-stream-to-response.ts
@@ -6501,9 +6534,19 @@ function createOutputTransformStream(output) {
controller.enqueue({ part: chunk, partialOutput: void 0 });
return;
}
@@ -57,7 +165,7 @@ index 663875332e3f9a9bd167c25583c515876f42951b..b840b0502c9894df983e0154805abb80
const result = await output.parsePartialOutput({ text: text2 });
if (result !== void 0) {
const currentJson = JSON.stringify(result.partial);
@@ -6882,7 +6892,7 @@ var DefaultStreamTextResult = class {
@@ -6882,7 +6925,7 @@ var DefaultStreamTextResult = class {
})
);
}
+6 -6
View File
@@ -48,7 +48,7 @@ patchedDependencies:
hash: d8dc66e5ec3b9d23a876b979f493b6aa901fd2d965be54729495da5136296a42
path: patches/@hocuspocus__server@3.4.4.patch
ai@6.0.134:
hash: f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9
hash: e8c599b3963eb01b9ed1481683b7b795ce94137aa1a0c951917c20c8b870299b
path: patches/ai@6.0.134.patch
scimmy@1.3.5:
hash: 775d80f86830b2c5dd1a250c9802c10f8fc3da3c7898373de5aa0c23993d1673
@@ -641,10 +641,10 @@ importers:
version: 8.3.0(socket.io-adapter@2.5.4)
ai:
specifier: ^6.0.134
version: 6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6)
version: 6.0.134(patch_hash=e8c599b3963eb01b9ed1481683b7b795ce94137aa1a0c951917c20c8b870299b)(zod@4.3.6)
ai-sdk-ollama:
specifier: ^3.8.1
version: 3.8.1(ai@6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6))(zod@4.3.6)
version: 3.8.1(ai@6.0.134(patch_hash=e8c599b3963eb01b9ed1481683b7b795ce94137aa1a0c951917c20c8b870299b)(zod@4.3.6))(zod@4.3.6)
bcrypt:
specifier: ^6.0.0
version: 6.0.0
@@ -16453,17 +16453,17 @@ snapshots:
agent-base@7.1.4: {}
ai-sdk-ollama@3.8.1(ai@6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6))(zod@4.3.6):
ai-sdk-ollama@3.8.1(ai@6.0.134(patch_hash=e8c599b3963eb01b9ed1481683b7b795ce94137aa1a0c951917c20c8b870299b)(zod@4.3.6))(zod@4.3.6):
dependencies:
'@ai-sdk/provider': 3.0.8
'@ai-sdk/provider-utils': 4.0.21(zod@4.3.6)
ai: 6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6)
ai: 6.0.134(patch_hash=e8c599b3963eb01b9ed1481683b7b795ce94137aa1a0c951917c20c8b870299b)(zod@4.3.6)
jsonrepair: 3.13.3
ollama: 0.6.3
transitivePeerDependencies:
- zod
ai@6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6):
ai@6.0.134(patch_hash=e8c599b3963eb01b9ed1481683b7b795ce94137aa1a0c951917c20c8b870299b)(zod@4.3.6):
dependencies:
'@ai-sdk/gateway': 3.0.77(zod@4.3.6)
'@ai-sdk/provider': 3.0.8