Структурные редакторы (patchNode/insertNode/updatePageJson/transformPage) кидали
опаковый Yjs-крах на агентском JSON с вложенным узлом без/с неизвестным `type`:
«Failed to encode document to Yjs (fromJSON): Unknown node type: undefined» —
ГЛУБОКО в энкодере, уже ПОСЛЕ открытия collab-сессии, а хинт мислейблил это как
проблему атрибута. Агент ретраил вслепую (~34 краха в истории 06-17…07-07).
- findInvalidNode(doc) в prosemirror-markdown/node-ops.ts: DFS по content,
возвращает {path, summary} первого узла с отсутствующим/не-строковым `type`
или типом/маркой вне схемы. Множество имён — из getSchema(docmostExtensions),
ТОГО ЖЕ, из которого энкод-путь строит docmostSchema → «известный тип»
обходчика ровно то, что примет PMNode.fromJSON/toYdoc (сверено на 45 узлах +
12 марках, ни ложных положительных, ни пропуска краш-типа).
- unstorableYjsError: findInvalidNode ПЕРВЫМ (node-shape крах больше не
мислейблится как атрибут), затем findUnstorableAttr, generic-фраза последней.
- assertValidNodeShape(op, node) ДО getCollabTokenWithReauth/mutatePageContent
в patchNode/insertNode/updatePageJson: fail-fast — collab-сессия не
открывается, page-lock не берётся, сообщение детерминировано (mock-тест
ассертит collabTokenFetched===false на битом пути). tableUpdateCell не тронут
(строит абзац из plain text через makeCellParagraph, агентский JSON не глотает).
- Описания patch_node/insert_node/update_page_json: каждый узел, включая
вложенные, несёт строковый `type` из схемы; текст-листы {"type":"text",...}.
sanitizeForYjs (стрип undefined-атрибутов) сохранён — другой класс отказа.
Внутреннее ревью: APPROVE WITH SUGGESTIONS — schema-fidelity/fail-fast/
no-false-positive/precedence подтверждены; замечания необязательны (тест
перечисления схемы, depth-guard безобиден т.к. энкодер падает раньше).
prosemirror-markdown vitest 726/726, mcp node --test 613/613.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Апстрим 78b1c1a4 перевёл серверные эндпоинты на КУРСОРНУЮ пагинацию, а
ValidationPipe({whitelist:true}) молча вырезает неизвестное поле `page`.
MCP-клиент так и слал офсетный `page` → сервер отдавал ту же первую двадцатку
с hasNextPage:true, цикл выкручивался до MAX_PAGES=50 одинаковых запросов, дети
№21+ не выгружались (с поддеревьями). Дедуп `visited` гасил дубли → «дырявое»
дерево без ошибок. Netmap: 20/299 страниц терялось, 160 запросов вместо 62.
- A: enumerateSpacePages → один POST /pages/tree (весь спейс/поддерево разом);
fallback на курсорный BFS при 404/405 (stock upstream). Возврат {pages,
truncated}; truncated честный — true только при реальном упоре fallback-BFS в
MAX_NODES.
- B: listSidebarPages → курсорный цикл, limit:100, guard на неподвижный курсор
(!next || next===cursor → break) — если протокол снова разойдётся, не крутит
дубли молча; warn при упоре в MAX_PAGES.
- C: paginateAll (/spaces, /shares) → та же курсорная миграция + guard.
- D: check_new_comments — /pages/tree поддерева включает корень
(getPageAndDescendants), убран лишний getPageRaw; в fallback корень
засевается явно (иначе его комменты терялись — регрессия того же класса).
- listComments: do/while → for с MAX_PAGES + guard неподвижного курсора
(был безлимитный — тот же сценарий #442 дал бы бесконечный цикл).
Внутренний цикл: 2 прохода. Первый нашёл потерю комментов корня в fallback
поддерева (data-loss) → засев корня; догрёб honest-truncated, warn в
listSidebarPages, guard в listComments. Второй проход — APPROVE, форма возврата
{pages,truncated} распространена на оба вызова без пропусков.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ревью #422: безусловная нормализация переписывала в ASCII и текст inline-code
внутри сносок (кавычки/тире/спецпробелы) — для code-литерала это не типографика,
а изменение смысла (эмпирически на raw-JSON путях: code-нода "a—b «x»" -> "a-b").
normalizeDefinitionText теперь пропускает текст-ноды с маркой code (verbatim), и
краевой trim определения тоже не трогает крайние code-ноды. footnoteMergeKey
читает сырой текст code-нод -> сноски, различающиеся только глифами в коде, НЕ
сливаются, а форки в прозе по-прежнему сливаются. Тесты: code verbatim +
непослияние по code-глифам.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Новый чистый модуль footnote-normalize-merge.ts (normalizeAndMergeFootnotes):
нормализует текст определений сносок (типографские кавычки->ASCII, тире->'-',
NBSP/спецпробелы->пробел, схлопывание пробелов, trim) и сливает определения с
совпавшим нормализованным текстом, перевешивая ссылки на канонический id;
дубли-сироты добивает canonicalizeFootnotes. Ключ слияния attrs-aware
(footnoteMergeKey/stableAttrs) — сноски с одинаковым текстом, но разными attrs
марок (напр. link.href) НЕ сливаются (защита от потери target). Пасс вызывается
строго ПЕРЕД canonicalizeFootnotes на 5 write-путях MCP (markdown-импорт,
updatePageJson, copyPageContent, docmost_transform, insertInlineFootnote).
Глиф-карты продублированы из comment-anchor.ts (там private+завязаны на golden).
Идемпотентен, чистый (deep-clone), scope строго внутри footnoteDefinition.
closes#419
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CollabSession-реестр (#400/#431) ключуется на `wsUrl + " " + pageId + " " +
token`. Клиент чеканил СВЕЖИЙ collab-токен на КАЖДУЮ мутацию (in-app —
пере-подписывает JWT с новым iat/exp раз в секунду; внешний MCP — POST
/auth/collab-token на каждый вызов), поэтому token-компонент ключа менялся
каждый раз и сессия почти никогда не переиспользовалась (connect-штормы, 25s
таймауты, зомби-сессии).
- новое пер-инстансное поле collabTokenCache {token, mintedAt};
- getCollabTokenWithReauth(forceRefresh=false) сначала смотрит кэш (гейт
!forceRefresh && ttl>0 && свежесть), оборачивает ОБЕ ветки чеканки
(provider-путь in-app агента И REST POST /auth/collab-token), обе через
rememberCollabToken (пустой токен не кэшируется);
- readCollabTokenTtlMs() читает env MCP_COLLAB_TOKEN_TTL_MS свежо; дефолт 5
мин (сильно ниже 24h жизни токена и <= max-age сессии, окно устаревания
прав не расширяется сверх #431); ЯВНЫЙ 0/отрицательное отключают кэш
(rollback-knob = точный fetch-per-call), unset/непарсибельное -> дефолт;
- reauth-ретрай (401/403) в обеих ветках рекурсит forceRefresh=true (обход
кэша -> свежий токен), гард !forceRefresh ограничивает ровно одним
повтором;
- кэш сбрасывается на КАЖДОЙ смене идентичности: в login() и в
response-интерцепторе (где this.token зануляется перед пере-логином) —
инвариант-4 (изоляция идентичностей) сохранён, кэш пер-клиентный.
Внутренний цикл: 1 проход внутреннего ревью (APPROVE WITH SUGGESTIONS);
правка по ревью — уточнён docstring readCollabTokenTtlMs (расхождение с
поведением: непарсибельное значение даёт дефолт 5мин с ВКЛючённым кэшем, а
не отключает; отключает только явный 0/отрицательное).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #434 (drawio stage 1) added drawioGet/drawioCreate/drawioUpdate to the
shared tool-spec registry with in-app metadata (inAppKey, deferred tier,
catalogLine) but wired them only in the standalone MCP server, breaking the
contract-parity and phantom-catalog unit tests on develop CI.
- expose drawioGet/drawioCreate/drawioUpdate in forUser() via sharedTool(),
mirroring the MCP transport's argument mapping (format ?? 'xml' default,
flat schema regrouped into the client's `where` object, positional
baseHash pass-through)
- extend the DocmostClientLike hand-mirror with the three client methods
- append the three names to the HOST_CONTRACT_METHODS drift-guard whitelist
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mutate трекает in-flight одним полем inflightReject; наложенный второй вызов
перезаписал бы рехджектор первого -> при disconnect отклонился бы только второй,
первый бы висел до PERSIST_TIMEOUT_MS (20с). В проде безопасно (оба call-site
сериализуют через per-page withPageLock), но это футган на разделяемом примитиве.
Гард в начале mutate (после ready-проверки, до касания inflightReject): наличие
in-flight -> reject нового вызова без порчи состояния первого. Docstring CONCURRENCY.
Последовательные mutate не задеты (localFinish синхронно чистит inflightReject до
резолва). +2 теста: конкурентный второй реджектится, первый цел; последовательные
оба успешны.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Инцидент: агент заполнял большую таблицу десятками table_update_cell, каждый —
полный цикл connect/auth/load/initial-sync/store/unload; часть падала с 25s connect-
timeout, event loop lag до 1.7с. Вариант B: кэшировать живой HocuspocusProvider
per page на серию правок — пока провайдер жив (connections>0), сервер не входит в
store->unload->reload, дебаунс реально коалесцирует записи (N ячеек -> 1-2 store).
Новый модуль collab-session.ts: класс CollabSession (connecting->ready->dead) +
реестр (ключ wsUrl+pageId+token) с idle-TTL/max-age/LRU-evict. mutatePageContent
(collaboration.ts) и mutateLiveContentUnlocked (client.ts, replaceImage) переведены
на acquireCollabSession; one-shot Promise-машина (~360 строк дублирования) удалена.
5 инвариантов (подтверждены внутренним ревью): (1) read->write атомарна — между
fromYdoc и applyDocToFragment нет await; (2) per-edit ack сохранён дословно (гард
connectionLost от false-success при реконнекте); (3) disconnect=смерть сессии (без
авто-реконнекта, in-flight реджектится теми же текстами ошибок); (4) изоляция
identity (токен в ключе); (5) валидация при reuse. replaceImage работает под
внешним page-локом без дедлока (acquire лок не берёт). Тексты ошибок == develop.
Env: MCP_COLLAB_SESSION_IDLE_MS (0=выкл кэш, точное легаси), _MAX_AGE_MS, _MAX_ENTRIES.
Teardown: destroyAllSessions обвязан в stdio (exit/SIGINT/SIGTERM) + реэкспорт из
index для встраивающего хоста.
closes#400
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Аудит при подготовке #413 нашёл дрейфующие дубли между packages/mcp и
packages/prosemirror-markdown. Четыре дедупа (поведение тулов не меняется):
1. node-ops: форк ~960 строк сведён в ОДНУ копию в prosemirror-markdown (живая
mcp-версия — строгое надмножество замороженного #293-seed'а пакета; сверено по
git-истории, новая пакет-копия байт-в-байт == прежней mcp-копии). Barrel-экспорт
полной поверхности; mcp/client.ts/page-search.ts/transforms.ts/collaboration.ts
импортируют из пакета; тесты переехали. node-ops тянет stripInlineMarkdown ->
пакет-локальная text-normalize.ts несёт только этот примитив (mcp-версия —
домен #408; заголовок документирует дубликацию + источник истины).
2. footnote-lex/footnote-analyze (vestigial legacy [^id]: диагностика): сведены к
одному fence-aware предупреждению 'reference-style footnotes -> use ^[...]'
(полезно для класса #410); footnote-lex удалён.
3. footnote-authoring -> примитивы (footnoteContentKey/makeFootnoteDefinition/
generateFootnoteId) перенесены в пакетный footnote.ts, одна реализация конвенции.
4. parse-node-arg -> перенесён в prosemirror-markdown (не mcp: сервер CommonJS не
импортирует ESM-only @docmost/mcp, но нативно импортирует пакет), обе копии
удалены, консьюмеры перенаправлены.
canonicalizeFootnotes/ENFORCEMENT RULE #228 и comment-anchor/json-edit/text-normalize
(mcp) не тронуты. API-поверхность node-ops оставлена чистой для #409/#413.
closes#414
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Комментарии человека находят агента сами: короткая эфемерная строка
'new comments: N on page … — call listComments(pageId)' в результате ЛЮБОГО
tool-вызова, mid-turn. Общий хелпер packages/mcp/src/comment-signal.ts
(createCommentSignalTracker: watermark + per-page debounce + working-set;
buildCommentSignalLine; defangCommentSignalTitle).
- Standalone MCP (index.ts): второй wrapper в choke point registerTool (паттерн
метрик #402) — отдельный {type:'text'} content-элемент, форма результата не
меняется. Источник: rate-limited listComments по working-set, title через
getPageRaw только на hit. State per-session.
- In-app (ai-chat-tools.service.ts): execute ВСЕГДА возвращает сырой результат
(part.output/цитаты не трогаются), сигнал доставляется модели через отдельный
toModelOutput ({type:'content', value:[raw, signal]}) — зеркало MCP; no-signal
ветка точно воспроизводит дефолт SDK. Источник: REST-probe (осознанный форк от
DB-count из ТЗ — чтобы не менять конструктор сервиса и не ломать спеки).
- Инъекционная защита: в сигнал идут только count+pageId+defanged-title, НИКОГДА
текст комментария (untrusted). Per-page watermark (не глобальный) — комментарии
на второй странице не теряются.
closes#417
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
createComment — топ-хотспот ошибок агента (промахи по якорю, слепые ретраи).
Портированы аффордансы самокоррекции из editPageText:
- Closest-block hint: общий хелпер closestBlockHint вынесен в text-normalize.ts
(json-edit.ts теперь тоже его зовёт), подключён во все 3 throw-а createComment.
- Markdown-strip fallback в comment-anchor.ts согласованно по всем 4 функциям
(can/count/apply/get) через единый resolveAnchorSelection: exact-verbatim wins
глобально, stripped — только если raw не якорится нигде; soft warning как в
editPageText. Инвариант уникальности suggestion (0/1/>=2) сохранён: raw-unique
никогда не запускает fallback -> не может стать ambiguous. Хранимый selection
остаётся СЫРОЙ подстрокой документа (strip только для поиска).
- Multi-block detection: явное сообщение 'selection spans multiple blocks' когда
per-block поиск провалился, но выделение есть в объединённом тексте блоков.
- tool-spec createComment: копировать selection дословно из getPage/searchInPage.
Известное мелкое ограничение (нит внутреннего ревью): детектор multi-block
использует raw selection, поэтому markdown-стилизованное выделение через границу
блоков получит generic-подсказку вместо spans-multiple-blocks (редко, guidance
всё равно корректный).
closes#408
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promoting insertFootnote/insertImage/replaceImage into the in-app
DocmostClientLike interface requires their mirror in the drift-guard's
HOST_CONTRACT_METHODS list (the bidirectional deepEqual — 40 vs 43 — was
red, exactly what the guard exists to catch). Added the three names and
corrected the header comment: they were MCP-only but are now in-app-consumed
and tracked; only deleteComment/updateComment remain untracked MCP-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Researcher role wrote 40 literal `^[...]` and zero real footnotes: its
incremental write path (insertNode/editPageText) doesn't parse markdown, and
the footnote-capable tool was MCP-only. Promote three tools from inline
MCP-only to the shared registry so the in-app agent gets them too.
- tool-specs.ts: insertFootnote/insertImage/replaceImage added to
SHARED_TOOL_SPECS (mcpName/schema/description moved VERBATIM from the inline
registrations — MCP names + behaviour unchanged for external clients).
- index.ts: the 3 inline registerTool calls become registerShared; drop the
"MCP-only by design" comments.
- ai-chat-tools.service.ts: register the 3 in-app via sharedTool ->
client.insertFootnote/insertImage/replaceImage (imageUrl->url,
attachmentId->oldAttachmentId mapping).
- tool-tiers.ts: insertFootnote -> core (else the original asymmetry recurs —
footnote tool hidden while editPageText is core); images -> deferred.
- research/{en,ru}.yaml FOOTNOTES: `^[...]` parses ONLY on a whole-markdown
write (create/update/import); for a pinpoint citation to existing text use
insertFootnote; via editPageText/insertNode it stays literal.
- json-edit.ts guardrail: an edit_page_text `replace` containing a `^[...]`
token is refused into failed[] with an insert_footnote hint, mirroring the
existing formatting-marker refusal. (Slightly broader net than that mirror —
a literal `^[a-z]` regex class in a replace is also refused; accepted
defense-in-depth, has a no-false-positive test.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DO-1: add an integration test (test/mock/tool-timing-server.test.mjs) that
constructs a real createDocmostMcpServer with a spy onMetric, links a Client
over InMemoryTransport, invokes get_workspace (no input schema, so the wrapped
handler always runs) and asserts onMetric fired with
("mcp_tool_duration_seconds", <number>, { tool: "get_workspace" }). This locks
that the monkeypatch wraps every tool AND labels with the registration name —
which the isolated timeToolHandler unit test does not. Mutation-verified
(mislabel -> test fails). The tool's expected backend failure (ECONNREFUSED)
is tolerated: the wrapper times in a finally on throw too, so the metric fires.
DO-2: reword the mcp.service.ts "zero overhead when disabled" comment to
"negligible overhead" — the registerTool wrapper still runs performance.now()
+ an async try/finally per tool call when onMetric is undefined (the
`onMetric?.()` short-circuits the label build; cost is immaterial at
tool-call rate), so "zero" was literally inaccurate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
packages/mcp stays free of prom-client/server: it only calls an optional
DocmostMcpConfig.onMetric(name, value, labels?) sink the host provides.
- client.ts: onMetric on the config; fires collab_connect_timeouts_total
once, only inside the 25s connect-timeout callback (the connect-vs-unload
signal). cleanup() clears the timer on every other finish path, so no
double-count.
- index.ts: createDocmostMcpServer monkeypatches server.registerTool (before
registerShared + inline tools are registered) to wrap every handler with
timeToolHandler — times in a finally on success AND throw, re-throws
unchanged, emits mcp_tool_duration_seconds{tool=<registered name>} (bounded
cardinality). Single choke point catches all tools.
- mcp.service.ts: the per-request config resolver injects onMetric ONLY when
isMetricsEnabled(), routing mcp_tool_duration_seconds -> observeMcpTool and
collab_connect_timeouts_total -> incConnectTimeout. Disabled / standalone
(stdio, no onMetric) -> undefined -> zero-overhead no-op.
New node:test unit (tool-timing.test.mjs) covers the wrapper's value/throw
preservation and the standalone no-op. packages/mcp/build/ is gitignored,
not committed (CI rebuilds it).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #351 converter fix (79a461f7) made multi-block list items emit a blank-line
separator between block children (loose list) to stop silent content merging on
re-parse. The prosemirror-markdown goldens were updated in that PR, but the two
mcp unit tests asserting the old tight output were missed — packages/mcp
re-exports the shared converter, so they broke the develop CI run (test job,
`pnpm -r test`, 2/480 failures).
- expect "- Parent\n\n - A..." instead of "- Parent\n - A..."
- expect "1. Parent\n\n - Child" instead of "1. Parent\n - Child"
Full mcp suite: 480/480 green; editor-ext / prosemirror-markdown / client /
git-sync suites green as well.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Maintainer decision on the escalated ReDoS fork: use re2. The regex path
compiled agent-supplied patterns with `new RegExp` and ran them synchronously in
the shared event-loop; a catastrophic-backtracking pattern (e.g. `(a+)+$`) hung
the whole Node backend for all users (the tool is in both transports incl. the
in-app apps/server agent), and size caps do NOT bound backtracking.
Switch the regex engine to re2 (Google RE2, linear-time, no backtracking):
- `new RE2(query, caseSensitive?'g':'gi')`. RE2 extends RegExp, so eachMatch and
the zero-length-match lastIndex guard are unchanged.
- Unsupported patterns are now a CLEAN error, not a hang: RE2 throws on invalid
syntax AND on the backtracking-only features it can't do (lookaround
(?=…)/(?<=…), backreferences \1) — caught at compile and returned as a clear
tool error telling the agent to rewrite without them.
- Removed MAX_CONTAINER_TEXT + the per-container slice (re2 is linear, so it's no
longer a ReDoS defense, and truncating risked silently dropping real matches in
a long container); kept MAX_PATTERN_LENGTH as a cheap query sanity cap.
- Verified: `(a+)+$` over 50k `a` completes in ~4ms; lookaround/backref throw.
- Added re2 (^1.21.0) to packages/mcp; lockfile updated.
Reviewer DO items:
- F1 [doc]: removed the false "pass nodeId as a comment anchor" claim
(create_comment has no nodeId param — it needs a text `selection`). Fixed in
tool-specs.ts + page-search.ts (module + SearchMatch JSDoc) + client.ts; the ref
is for get_node/patch_node, and for a comment you build a unique text selection
from before+match+after.
- F2 [doc]: clarified `#<index>` refs (id-less table/cell) are accepted by get_node
but NOT patch_node (id-only).
- F3 [test]: round-trip — each match's nodeId fed to the real getNodeByRef
(attrs.id node + `#<index>` table-cell) to prove the ref format is consumable.
- F4 [test]: before/after edge-pinning (match in first 40 chars of a long
container; index 0 → before==""; container end → after=="").
- New re2 tests: catastrophic patterns complete fast; lookaround/backref → error.
mcp: tsc clean; node --test 472 passed (+5). apps/server: tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Editorial roles (Corrector/Factchecker) brute-forced `get_node` block-by-block to
find occurrences (unquoted «ё», straight quotes, «т.е.»), burning tokens. New
`search_in_page(pageId, query, {regex?, caseSensitive?, limit?})` reads the page's
ProseMirror JSON via the existing getPageRaw and searches it IN MEMORY — no server
endpoint, no DB/schema change, no touch to the packages/mcp/src/lib schema mirror.
New pure `searchInDoc(doc, query, opts)` (packages/mcp/src/lib/page-search.ts):
recursive descent to each TEXT CONTAINER (paragraph/heading/table-cell paragraph),
glues its inline text via `blockPlainText` (a match survives inline-mark
boundaries — e.g. «т.е.» split across bold/italic), searches literal (indexOf) or
regex, and returns `{ total, truncated, matches:[{ nodeId, blockIndex, type,
before, match, after }] }`. `nodeId` is the container's attrs.id or the
`#<topLevelIndex>` of the enclosing top-level block — the SAME ref format
get_node/patch_node/comment-anchoring accept (verified identical to getNodeByRef),
so the agent goes straight from a hit to a targeted comment; `before`/`after` are
~40-char windows for a unique selection. `total`/`truncated` always reported (never
silent truncation). Lives in the SHARED_TOOL_SPECS registry → exposed in BOTH
transports (external /mcp + in-app AI-chat), with a SERVER_INSTRUCTIONS line and a
DocmostClientLike signature + contract-test entry. Corrector/Factchecker prompts
get a one-line "use search_in_page first" hint (versions bumped, catalog hash lock
refreshed).
Guards: empty/whitespace query → clear error; invalid regex → clear error (not a
generic 500); zero-length regex matches (`\b`, `a*`) skipped with lastIndex
advanced (no loop/flood); MAX_PATTERN_LENGTH=1000, MAX_CONTAINER_TEXT=100k bound
each exec; limit clamped [1,200] (default 50).
Tests: new page-search.test.mjs (17) — literal+regex, case-sensitivity,
mark-boundary glue, nodeId for paragraph/heading (attrs.id) and table-cell
(#<index> fallback), context bounds, limit/total/truncated + clamp, invalid
regex/empty/over-long errors, zero-length skip, empty-doc null-safety.
mcp: tsc clean; node --test 467 passed (+17). apps/server: tsc --noEmit clean
(DocmostClientLike + wiring). catalog check.mjs OK.
Known limitations (from internal review, non-blocking):
- Residual ReDoS: a crafted catastrophic-backtracking pattern (e.g. `(a+)+$`)
against a large single container can hang the event loop — JS regex is not
interruptible, so the length caps bound the base but not the backtracking.
Realistic exposure is low (containers are small; the pattern is supplied by the
authenticated model). Candidate for a follow-up hardening (safe-regex validation
or a worker+timeout) if it matters.
- Case-insensitive LITERAL search folds via toLowerCase; a char whose lowercase
differs in length (e.g. Turkish İ) BEFORE a match could shift the context
window — negligible for the RU/EN editorial scenario.
- On a `#<index>` table-cell fallback, `type` is the inline container ("paragraph")
while nodeId addresses the top-level block — addressing is correct; the field is
documented as the container's type.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The AI agent (MCP + in-app chat) saw ALL comments incl. resolved via two
channels, cluttering its context and breaking fragment search. Default now:
the agent sees only ACTIVE discussions; resolved is opt-in. Active anchors and
threads are always kept.
Channel 1 — resolved comment anchors on agent reads (converter option):
`convertProseMirrorToMarkdown(content, options?)` gains
`options.dropResolvedCommentAnchors` (default false — zero change for every
existing caller incl. git-sync). Both `case "comment"` emitters (top-level and
the raw-HTML inlineToHtml path) emit BARE text (no `<span data-comment-id>`) when
`resolved && the flag`; active anchors keep their wrapper. mcp `getPage` passes
the flag; `export_page_markdown` does NOT (lossless export must preserve resolved
anchors — that is why it is an opt-in option, not unconditional); `get_page_json`
is untouched (lossless PM JSON). Built on the #293 package converter.
Channel 2 — `list_comments` default active-only: `listComments(pageId,
includeResolved=false)` now returns `{ items, resolvedThreadsHidden }` (was a
bare array). By default a RESOLVED top-level thread is hidden wholesale — the
root AND every reply anchored to it (a thread is gated only by its root's
resolvedAt; a resolved reply under an ACTIVE root stays). `resolvedThreadsHidden`
counts hidden threads so the agent knows to re-query. `includeResolved:true`
returns everything. The `includeResolved` param is added to both tool
registrations (MCP index.ts + in-app ai-chat-tools.service.ts); `DocmostClientLike`
signature updated. Server `findPageComments` is NOT touched — the web UI's tabs
depend on the full feed; filtering is only at the mcp-client level. All internal
call sites (export_page_markdown / checkNewComments / transformPage) updated to
`.items` with `includeResolved:true` to keep their full-feed behavior.
The comment model is assumed FLAT (a reply's parentCommentId points at the
thread root) — documented in the filter; a future reply-of-reply model would
need a root-walk there.
Tests: resolved-comment-anchors.test.ts (6 — anchor dropped with flag / kept
without, for BOTH emitters; active always kept); list-comments-resolved.test.mjs
(4 — resolved thread+reply hidden + counter; includeResolved:true returns all;
an ACTIVE thread with a RESOLVED reply is NOT hidden).
package vitest: 664 passed; tsc clean. mcp: node --test 458 passed; tsc clean.
apps/server + git-sync: tsc clean (converter option default-off).
NOTE: based on feat/293-B (#293/#326 STEP 5) — the converter lives in the
package; this PR is stacked on #333 and its base retargets to develop once #333
merges. mcp/build is gitignored (not committed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mcp had its OWN drifted copy of the converter (markdown-converter.ts ~900 lines,
docmost-schema.ts ~1270 lines, markdown-document.ts) — older than the shared
package, missing the git-sync fixes AND the #293 canon. This switches mcp's
converter CORE to @docmost/prosemirror-markdown, so mcp jumps straight to the
canonical format and the drift-generating second copy is gone.
- markdown-converter.ts / markdown-document.ts / docmost-schema.ts become thin
re-export shims of the package (convertProseMirrorToMarkdown, the docmost:meta
envelope, docmostExtensions + docmostSchema=getSchema(docmostExtensions)). The
mcp-only helpers clampCalloutType/sanitizeCssColor are preserved verbatim in
the schema shim (the package doesn't expose them via its barrel). ~2170 lines
of the drifted converter/schema bodies deleted.
- collaboration.ts drops its own ~360-line marked pipeline (preprocessCallouts,
bridgeTaskLists, extractFootnotes, the footnoteRef extension) and re-points to
the package's markdownToProseMirror, keeping markdownToProseMirrorCanonical and
all the yjs/collab write glue. footnote-lex/analyze doc comments updated (they
now describe advisory legacy-syntax diagnostics, not an importer).
Schema parity verified: the package schema is a strict SUPERSET of mcp's old
schema — every node and attr mcp declared is present (the package only adds
status/pageEmbed/transclusion*/subpages.recursive/etc.), so nothing is silently
dropped on the switch. The switch actually FIXES two pre-existing mcp data-loss
bugs its own tests documented: htmlEmbed and pageBreak now round-trip (were
dropped by the old mcp converter).
Footnotes: the package assembles inline ^[body] footnotes on import (sequential
fn-N ids, identical bodies merged), so mcp's canonicalizeFootnotes is now an
idempotent no-op after it (verified). Legacy reference footnotes [^id]/[^id]:
are inert literal text (canon #2 no-backward-compat) — lossless, the text
survives verbatim.
Build hygiene: packages/mcp/build/ is now gitignored and untracked, matching the
git-sync/prosemirror-markdown convention (private package, rebuilt in CI/Docker,
so src and prod can never silently diverge). This also removes a dead untracked
build/_vendored_editor_ext/ artifact that a broad `git add` would otherwise
commit.
Dependency: packages/mcp/package.json gains @docmost/prosemirror-markdown
(workspace:*); pnpm-lock.yaml gets the matching link importer (mirrors git-sync).
mcp tests updated deliberately to the canonical forms (highlight ==, math $…$,
image <!--img-->, drawio/media discriminators, subpages/pageBreak
comments, textAlign, inline ^[…] footnotes) with strict assertions; 4 structural
safety-net round-trip tests added.
mcp: node --test 454 passed; tsc clean. package: 657 passed. git-sync: 268 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The intent-routing guide had rotted: 17 of 41 registered tools were absent
(get_outline, get_node, the whole table_* family, search, stash_page, sharing,
page lifecycle), and two tips were actively harmful — 'read block ids via
get_page_json' told agents to pull the whole ~100KB document when get_outline
exists precisely to grab ids cheaply, and 'table cell -> patch_node by
attrs.id' dead-ends because table nodes carry no attrs.id.
- Rewrite SERVER_INSTRUCTIONS as intent clusters (READ / EDIT / PAGES /
COMMENTS / HISTORY) covering every tool except get_workspace; add safety
notes (share_page = PUBLIC, delete_page = soft) and a comment-anchor
markup warning for get_page.
- delete_page tool description: state SOFT delete / restorable explicitly.
- MAINTENANCE RULE comments at both registration sites (index.ts,
tool-specs.ts) + an AGENTS.md convention bullet: adding/renaming/removing
a tool REQUIRES updating the guide.
- New guard test (test/unit/server-instructions.test.mjs): extracts every
registered tool name from source and fails when one is not mentioned in
the shipped SERVER_INSTRUCTIONS (word-boundary match, so get_page can't
hide behind get_page_json); EXCEPTIONS list is itself validated against
the registry. SERVER_INSTRUCTIONS exported for the test.
Tests: @docmost/mcp 450/450 (448 + 2 new).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
F1 [blocking]: a suggestion whose anchor matched via normalization could never
be applied (spurious 409). The comment mark lands on the doc's ACTUAL text
(Docmost auto-converts to typographic quotes/dashes/nbsp), but the stored
selection — used as expectedText at apply — was the raw ASCII agent input
(+substring(0,250)). So replaceYjsMarkedText's strict joined!==expectedText
always failed and threw "text changed" though nobody edited. Fix: new pure
getAnchoredText(doc, selection) reconstructs the exact raw doc substring the mark
covers (slicing identical to spliceCommentMark); on the suggestion path
client.createComment stores THAT as selection, so expectedText equals the marked
text and apply returns applied:true. Live anchoring still uses the raw agent
selection (normalization still finds the anchor). Truncation raised 250->2000
(+ DTO @MaxLength(2000)) so the anchored substring is never cut below the mark
span. Ordinary comments unchanged. AI-chat shares client.createComment, so
covered. Regression tests: getAnchoredText raw-vs-ASCII; create payload selection
is the typographic substring; apply with typographic expectedText -> applied.
F2 [blocking]: added comment.controller.spec.ts pinning that validateCanEdit runs
before applySuggestion (Forbidden -> applySuggestion never called; happy path ->
called; missing comment -> 404 without authorizing).
MCP 448 pass; server comment+yjs 54 pass. MCP build/ rebuilt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents can attach a suggested replacement when creating an inline comment, via
both the MCP create_comment tool and the AI-chat createComment tool.
Because applying a suggestion edits the EXACT anchored text, an ambiguous anchor
would let Apply corrupt the wrong occurrence. So when suggestedText is set the
selection must occur EXACTLY ONCE:
- new countAnchorMatches(doc, selection) counts occurrences across all blocks
(same normalization/traversal as canAnchorInDoc), counting occurrences (2 in
one block => 2) — stricter than block-count, never under-counting distinct
occurrences (false-unique is the dangerous direction).
- client.createComment gains suggestedText: a pre-check (getPageJson +
countAnchorMatches: 0 => not-found, >=2 => ambiguity error) before create, and
an AUTHORITATIVE live check inside the anchoring mutation that recomputes on the
live doc and, if != 1, aborts and rolls back the just-created comment (reusing
the existing safeDeleteComment "anchor not found" path). Ordinary comments keep
first-occurrence behavior unchanged.
- suggestedText is rejected on a reply or without selection in all three layers
(MCP handler, MCP client, AI-chat tool), mirroring the server DTO/service.
- filterComment surfaces suggestedText/suggestionAppliedAt/suggestionAppliedById.
- DocmostClientLike.createComment signature updated. MCP build/ rebuilt.
Tests: countAnchorMatches (0/1/N, within/across/nested block, span nodes,
quote normalization); createComment (ambiguous refused pre-create, reply and
no-selection rejected, unique succeeds and forwards suggestedText, filterComment
surfaces it); ai-chat schema accepts suggestedText. MCP 443 pass; ai-chat 601 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The same tool metadata (zod schema + model-facing description) was hand-duplicated
between the standalone MCP server and the in-app AI-chat agent, so every tweak had to
land in two places and copies drifted (a materialized parity bug). The shared
transport-agnostic registry (packages/mcp/src/tool-specs.ts) already de-duplicates 14
tools; this migrates two more genuinely-identical ones — patch_node/patchNode and
insert_node/insertNode. The canonical description is a strict SUPERSET of both originals
(keeps MCP's "without resending the whole document" + table-structure/anchor guidance
AND the in-app "reversible via page history" / "exactly one of anchorNodeId or
anchorText" framing — no model-facing guidance dropped); the schema is identical (the
in-app side just gains MCP's .min(1) on ids, a safe tightening). Each transport keeps its
own execute/auth wrapper, and the in-app parseNodeArg node-arg normalization is unchanged.
The three table tools are intentionally NOT merged (a real param-name divergence:
table vs tableRef) — documented on both sides. Other per-transport divergences
(search/share/create_comment/transform/list_pages) are left separate with a short comment
explaining why (the issue asked to flag these as intentional). DocmostClientLike stays a
hand-mirror (the ESM/CJS boundary blocks a compile-time type import; a runtime drift-guard
already pins it). Also fixes a latent contract-spec bug: derive `required` from
`instanceof z.ZodOptional` (matches the emitted JSON schema) instead of `isOptional()`,
which wrongly reported z.any() fields as optional.
Partially addresses #294.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reviewer round 1 on the #260 collab-doc-name fix:
- F1: replaceImage is the one path where the resolved UUID gates BOTH the
collab-doc open AND the per-page mutex key (withPageLock(pageUuid)). Add a
deterministic test to resolve-page-id-collab-doc-name.test.mjs: it gates
/files/upload so replaceImage parks mid-upload holding its lock, asserts the
doc opened as page.<uuid> (never page.<slug>), and probes the SHARED
page-lock chain — a withPageLock(UUID) probe must stay blocked while
replaceImage holds it (with a free-key probe as a non-vacuity guard). The
test fails if the lock key is reverted to the slugId (verified).
- F2: drop the dead `pageIdCache.set(uuid, uuid)` — resolvePageId returns on
the isUuid() short-circuit before the cache is ever read with a uuid key, so
only slugId->uuid entries are stored/read. Comment corrected to match.
MCP suite 430/430, tsc 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real root cause of the silent MCP edit loss: the web editor always opens the
collaboration document by the page UUID (`page.${page.id}`), but the MCP
opened it by the agent-supplied id — usually a slugId — so `page.${pageId}`
became `page.<slugId>`. For one DB page that is TWO independent Yjs documents;
both persist to the same `pages` row (findById/updatePage resolve id or
slugId), so the human tab's debounced store overwrites the agent edit
(last-store-wins) — gone after reload, never shown live. The slugId doc also
made the server's transclusion sync + embedding reindex throw Postgres 22P02.
Fix:
- MCP (primary): resolvePageId(pageId) returns the canonical UUID — a UUID
short-circuits with no network call, a slugId resolves once via getPageRaw
and is cached both ways. Every collab-write path (mutatePageContent /
updatePageContentRealtime / replacePageContent and the mutate/replace/
unlocked seams) now opens by the resolved UUID, so the MCP and the editor
share ONE Yjs doc. replaceImage's whole-operation page lock also keys on the
UUID so it serializes against the other (now-UUID-keyed) writes.
- Server (defense + kills the 22P02 noise): onStoreDocument passes the resolved
page.id — not the raw doc-name id — to syncTransclusion, the embedding queue,
the mention-notification job, addContributors, and the in-tx history read.
Content store and the empty-guard are untouched.
Tests: a new MCP test stands up a real Hocuspocus server and asserts a slugId
input opens `page.<uuid>` (never `page.<slugId>`), with UUID short-circuit and
single-resolve caching; the server spec asserts the side-effects receive the
UUID for a `page.<slugId>` doc. closes#260
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stashPage is declared in the server's DocmostClientLike interface and
shipped as the stash_page MCP tool (client.ts, tool-specs.ts, index.ts),
but the hand-maintained HOST_CONTRACT_METHODS mirror in the contract test
was never updated — so the drift-guard test failed and broke CI's
unit-test job. Add the missing name; both directions now agree.
F1 (data loss): packages/mcp keeps its own copy of the document schema
(AGENTS.md), and the spoiler mark was only added to editor-ext + the server
tiptapExtensions, so a doc with a spoiler silently lost the mark through /mcp.
Add a local Spoiler mark to docmostExtensions (span[data-spoiler] parse,
data-spoiler="true"+class render) and a case "spoiler" in markdown-converter
emitting the same <span data-spoiler="true">…</span> as the editor-ext turndown
rule; add an MCP json->md->json round-trip test. Regenerated build/lib output.
F2: add the #259 inline-spoiler entry to CHANGELOG [Unreleased] Added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uploadImage is internal to client.ts (called by insertImage/replaceImage);
the MCP transport (index.ts) does not call it directly. Remove it from the
comment's list of transport-called methods.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract pure extractAuthTokenFromSetCookie from performLogin (behavior-identical)
so cookie parsing is unit-testable without a network login. Add round-trip
coverage for media attrs (width/height/align/drawio/escaping) the existing
suite omitted; applyAnchorInDoc selection/ambiguity/atom-break cases; and a
cross-copy drift guard proving the vendored editor-ext recreate-transform and
the @fellow npm copy used by diff.ts emit identical steps (apply(diff)==target).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DocmostClientLike mirror covers only methods the in-app adapter consumes;
the standalone MCP transport calls additional client methods not tracked here
(covered by its own typecheck). Fixes the misleading 'superset' wording (F2).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A captioned image in a column is emitted via the imageToHtml helper, a
separate path from the top-level image case whose data-caption branch was
untested. Add a round-trip test with special chars (Tom & "Jerry") that
fails if the imageToHtml caption branch breaks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Architect-review hardening of the bidirectional DocmostClientLike <->
HOST_CONTRACT_METHODS guard (test-only, no production change):
- Interface method-name regex now accepts full TS identifiers
(digits/_/$) and generic signatures (method<T>(), avoiding a future
benign false-FAIL.
- Skip /* ... */ block comments in the interface body so a `name(` line
inside one is not falsely parsed as a method.
- Wrap the cross-package readFileSync with a clear "expected monorepo
layout" error instead of a bare ENOENT when run outside the monorepo.
- Narrow the guard's comments/error to state plainly it checks the
method-NAME set only; signature parity remains the deferred staged-plan
item.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add PM -> markdown -> PM round-trip assertions for image caption
(plain and special-char), which fail without F1 and pass with it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The contract test only checked one direction (each name in
HOST_CONTRACT_METHODS exists on the real DocmostClient). But
HOST_CONTRACT_METHODS is itself a hand-copy of the server's
DocmostClientLike interface (docmost-client.loader.ts), and that
list<->interface link was untested: a method added to the interface +
consumed by the adapter but forgotten in the list (or removed from the
interface but left in the list) would escape both the server typecheck
(the pkg emits no .d.ts) and the existing test (name not in the list) ->
a runtime "x is not a function" in a tool call.
Parse the method names from the DocmostClientLike interface body (read
the .ts source via import.meta.url, scan member-signature lines) and
assert.deepEqual them against HOST_CONTRACT_METHODS BOTH ways. Lists are
currently identical (39=39), so this is a coverage hole closed, not a
live bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mandatory (test-coverage):
- internal-file-urls.test: pin the SSRF/traversal ACCEPT path of
resolveInternalFilePath (the sole guard for content-controlled `src`): an
absolute/protocol-relative URL has its foreign host dropped and only an
/api/files/ pathname survives (http://evil.com/api/files/x/y.png -> /files/x/y.png),
while a host-dropped path that escapes /api/files/ (https://evil.com/api/auth/whoami)
or a backslash-traversal (/api/files\..\auth\whoami) is rejected. Locks the
behavior so a future prefix-only refactor cannot silently open a bypass.
Suggestions:
- index.ts: the stash_page MCP tool now returns structuredContent
{ uri, sha256, size, images } alongside the resource_link, so the MCP output
matches the documented shape (clients get the blob's sha256/ETag and the
mirror counts, not just the link). No outputSchema registered. Rebuilt build/.
- new stash-page-mcp-result.test: server round-trip via InMemoryTransport asserts
both the resource_link and the structuredContent mirror.
- internal-file-urls.test: cover the new URL parse-failure catch branch
(http://[ -> "Invalid internal file src").
- environment.service.spec: assert getPositiveIntEnv warns once per key and
independently across keys (the invalidPositiveIntWarned dedup).
Tests: packages/mcp 383 pass; apps/server sandbox/environment/mcp 235 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security (must-fix):
- sandbox.controller: the anonymous GET /api/sb/:id response now sets
X-Content-Type-Options: nosniff, a restrictive CSP, and Content-Disposition=
attachment for any mime outside a raster-image allowlist (png/jpeg/gif/webp/
avif). entry.mime is attacker-controlled, so an evil.svg/evil.html could
otherwise execute script inline on the Docmost origin (stored XSS). Mirrors
the public attachment route's hardening.
Stability:
- client.stashPage: reconcile mirrors AFTER the final document put, not only
before it. The doc blob is the newest entry and FIFO eviction drops the
oldest = this stash's own images, so the stored doc could reference an
evicted blob (consumer 404) and over-report images.mirrored. A bounded loop
now reverts doc-put-evicted mirrors, drops the stale doc blob, and re-puts
until stable. Regenerated packages/mcp/build/.
- sandbox.controller: emit Cache-Control on the 304 branch too (ttlSeconds is
computed before the conditional check).
Docs:
- Bump the MCP tool count 39 -> 40 across all READMEs and AGENTS.md (the
registry now exposes exactly 40 tools).
Refactor:
- SandboxStore.asSink() centralizes the {put,has,evict} sink + uri<->id
mapping; the embedded-MCP and in-app agent-tools wiring sites share it.
Tests:
- security headers (inline vs attachment, nosniff, CSP), 304 Cache-Control,
putAndLink URL form, has()/remove(), asSink() round-trip, getSandboxPublicUrl
(trailing-slash trim + APP_URL fallback), and a stash test where the doc put
itself evicts a mirrored image.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security:
- stash_page: reject path-traversal / percent-encoded srcs before the authed
loopback fetch (resolveInternalFilePath), closing an SSRF/exfiltration hole
where a crafted node.attrs.src could read an arbitrary internal GET endpoint
into the anonymous sandbox.
Stability:
- stash_page: revert + recount mirrors FIFO-evicted by a later put in the same
stash (no dangling sandbox refs, honest images.mirrored/failed); free image
blobs if the final document put throws.
- Reject/clamp non-positive SANDBOX_TTL_MS to the 1h default (warn once).
- Log mirror failures unconditionally (console.warn, no blob bodies).
Cleanup / architecture:
- Remove dead expiresAt from SandboxPutResult.
- Centralize the /api/sb route in SANDBOX_ROUTE_SEGMENT/SANDBOX_API_PATH and
move URL composition into SandboxStore.putAndLink; drop the duplicated sink
closures and the now-unused EnvironmentService injection from McpService and
AiChatToolsService.
- Un-export isInternalFileUrl; document the process-local (instance-bound)
sandbox limitation in the tool description and .env.example.
Docs/tests:
- README/README.ru: 38 -> 39 tools + stash_page entry.
- Add traversal/normalize/recursion unit tests, stash self-eviction +
doc-put-throw + empty/octet-stream mock tests, controller If-None-Match
(wildcard/weak/list) + Cache-Control tests, and SANDBOX_TTL_MS validation
tests. Regenerate packages/mcp/build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
agent_coder
vvzvlad
agent_vscode
claude code agent 227
agent_coder
