Правки по внутреннему ревью #413.
1. Orphan-сноска (нарушение «no second canon»). mergeFootnoteDefinitions
раньше делал ранний return при пустых definitions, пропуская
canonicalizeFootnotes. Если markdown-фрагмент БЕЗ сносок заменял/удалял
блок, бывший последним referrer'ом существующей сноски, её определение
оставалось orphan в хвостовом списке (полный ре-импорт его бы убрал).
Теперь fast-path (возврат doc по ссылке без clone) только когда
defs.length===0 И !hasFootnoteArtifacts(doc); иначе клон → append (no-op
при пустых) → normalizeAndMergeFootnotes → canonicalizeFootnotes, как при
полном импорте. Новый предикат hasFootnoteArtifacts обходит дерево на
любой footnotesList/footnoteReference (существующие walk/isObject).
Идемпотентно: несвязанный plain-патч на странице со сносками не трогает
их топологию (canonicalizeFootnotes шаг 6 возвращает как есть).
2. Block-id disjoint от страницы. Новый экспорт reassignCollidingBlockIds
(liveDoc, blocks, skipIndex?) в prosemirror-markdown/node-ops (переиспользует
collectIds/makeFreshId) + barrel. Зовётся перед сплайсами в client.ts:
patch — (liveDoc, threaded, 0) (skip 0 = блок, унаследовавший id цели);
insert — (liveDoc, blocks) без skip. used-аккумулятор ловит и коллизию со
страницей, и внутрифрагментную. freshBlockId/freshId без изменений.
Тесты (+6, markdown-patch-insert): orphan-repro (0 defs/0 list/0 refs +
docsCanonicallyEqual полному импорту), fast-path (footnote-free patch не трогает
топологию, соседи byte-identical), insert канонизирует при пустых defs со
страничной сноской, уникальность top-level block-id для patch 1→N и insert N.
mcp node --test 702/702, pmd vitest 736, tsc чисто.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Канонический конвертер (#293/#345/#351) зрел для блочного уровня: markdown
становится дефолтом чтения/записи ОДНОГО блока, PM JSON — опция «для тонких
работ». Новых тулов нет, поверхность не растёт. Имена уже camelCase (стоит на #412).
- getNode(pageId, nodeId, format='markdown'): дефолт markdown — обёртка
{type:doc,content:[node]} → convertProseMirrorToMarkdown, comment-якоря
(ВКЛЮЧАЯ resolved) сохранены (это чтение под редактирование, не getPage).
format:'json' — сырой сабтри. Авто-фолбэк не-топ-левел типов (tableRow/Cell/
Header по #<index>) в JSON через canBeDocChild = docmostSchema.nodes.doc.
contentMatch.matchType (не рукописный список); поле format на каждом ответе.
- patchNode/insertNode: XOR-вход {markdown?|node?} (оба optional в схеме, XOR
на рантайме). markdown → импорт фрагмента → 1→N сплайс: первый блок наследует
id цели, остальные свежие; dry replaceNodeById для #159-ambiguity ДО сплайса;
соседние блоки byte-identical. Guard findUnrepresentableTableAttrs: цель со
span/colwidth/backgroundColor → отказ с указанием на table-тулы/node-JSON.
insertNode — insertNodesRelative (N блоков по порядку); голый tableRow/Cell
JSON-only.
- Сноски: ^[...] во фрагменте → канон-импортёр; importMarkdownFragment делит
блоки от footnotesList, РЕМАПИТ id сносок фрагмента в свежие uuid (fn-1
фрагмента не коллизит с fn-1 страницы), mergeFootnoteDefinitions добавляет
через ту же машинерию appendDefinition→normalizeAndMergeFootnotes→
canonicalizeFootnotes, что insertFootnote. Сырые JSON-пути не тронуты.
- node-ops: replaceNodeByIdWithMany/insertNodesRelative (сплайс массива) в
prosemirror-markdown/node-ops (канон после #414) + barrel.
- ROUTING_PROSE (READ/EDIT), compile-time client-call contract, CHANGELOG
(getNode default→markdown, breaking для внешних клиентов, в окне #411/#412).
Тесты: сходимость (patchNode(markdown) блок docsCanonicallyEqual полному
импорту — нет «второго канона»); id-нить 1→N (первый наследует, остальные
свежие, соседи byte-identical); XOR; getNode markdown/json/non-top-level-fallback;
сохранение comment-якорей (active+resolved); ^[...]→хвостовой список+перенумерация;
guard. mcp node --test 697/697; pmd vitest 736; tsc чисто; server jest 273.
Третий линк breaking-окна, стоит на #412 (#411→#412→ЭТОТ→#415).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Один логический тул жил под двумя именами: внешний MCP snake_case
(edit_page_text), in-app camelCase (editPageText) — дублирование доков, путаница
при переносе промптов/скиллов, помеха шарингу спек (#294). Решение владельца:
единый camelCase везде, включая внешний MCP. После этого mcpName === inAppKey.
- tool-specs.ts: mcpName ВЫВЕДЕН из ключа спеки (mcpName == inAppKey) для всех
43 shared-спек — раньше divergent snake, теперь равен ключу (проверено: mcpName
читается только структурно — цикл регистрации, генератор <tool_inventory>,
TOOL_FAMILY). +5 inline-регистраций (tableGet/updateComment/deleteComment/
docmostTransform; search без изменений). Рантайм: 47 тулов, все camelCase, ноль
подчёркиваний.
- Контракт-конвенция ИНВЕРТИРОВАНА: shared-tool-specs.contract.spec
`mcpName === toSnake(inAppKey)` → `mcpName === inAppKey`; tool-specs.test
и tool-inventory.test обновлены.
- ROUTING_PROSE/TOOL_FAMILY/INLINE_MCP_INVENTORY (server-instructions.ts) →
camelCase (105 замен). ai-chat.prompt/guard уже на in-app camelCase-ключах —
без изменений (guard прошёл). comment-signal EXCLUDED_TOOLS схлопнут с
дублей snake+camel до camelCase.
- Некоторое неочевидное: assertUnambiguousMatch(op: "patch_node"|"delete_node")
в prosemirror-markdown/node-ops — op интерполируется в model-facing ошибку;
литерал-юнион + call-sites → "patchNode"|"deleteNode".
- Все snake-имена в описаниях/error-строках/комментах/тестах/доках → camelCase
(whole-token, longest-match-first). CHANGELOG: BREAKING-таблица 46 строк +
миграция (allowlists mcp__gitmost-*__get_node→__getNode, промпты/скиллы,
.mcp.json, метрики по tool-label); релизится вместе с #411.
Внутренние имена методов (PageService.updatePageContent и т.п.) НЕ тронуты —
переименованы только ИМЕНА ТУЛОВ.
Гейт: mcp node --test 677/677; tsc -p apps/server чисто; jest ai-chat-tools.
service + shared-tool-specs.contract + tool-tiers + ai-chat.prompt +
comment-signal-inapp → 323. Второй линк breaking-окна (#411→ЭТОТ→#413→#415).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
applyElkLayout крутит elkjs СИНХРОННО в процессе на mxGraph-XML от LLM
(layout:'elk' в drawio_create/update) без лимита размера и без таймаута —
большой граф (тысячи узлов, ~1МБ XML проходит stage-1 cap 16МБ) блокирует
event-loop MCP-сервера на секунды-минуты. try/catch ловил только брошенную
ошибку, но не зависание.
- кап ДО построения графа: >500 узлов или >1000 рёбер → вернуть исходный XML
(best-effort, как существующий catch); синхронный elkjs → кап и есть
реальная защита;
- Promise.race с 5s-таймаутом (defense-in-depth на случай async-elkjs); таймер
гасится в finally → нет утечки хендла и unhandled-rejection (проигравший
timeout остаётся pending с погашенным таймером);
- тест: 600-узловой граф возвращается без изменений и быстро (<2s) — кап-путь.
79/79 drawio-тестов зелёные.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Структурные редакторы (patchNode/insertNode/updatePageJson/transformPage) кидали
опаковый Yjs-крах на агентском JSON с вложенным узлом без/с неизвестным `type`:
«Failed to encode document to Yjs (fromJSON): Unknown node type: undefined» —
ГЛУБОКО в энкодере, уже ПОСЛЕ открытия collab-сессии, а хинт мислейблил это как
проблему атрибута. Агент ретраил вслепую (~34 краха в истории 06-17…07-07).
- findInvalidNode(doc) в prosemirror-markdown/node-ops.ts: DFS по content,
возвращает {path, summary} первого узла с отсутствующим/не-строковым `type`
или типом/маркой вне схемы. Множество имён — из getSchema(docmostExtensions),
ТОГО ЖЕ, из которого энкод-путь строит docmostSchema → «известный тип»
обходчика ровно то, что примет PMNode.fromJSON/toYdoc (сверено на 45 узлах +
12 марках, ни ложных положительных, ни пропуска краш-типа).
- unstorableYjsError: findInvalidNode ПЕРВЫМ (node-shape крах больше не
мислейблится как атрибут), затем findUnstorableAttr, generic-фраза последней.
- assertValidNodeShape(op, node) ДО getCollabTokenWithReauth/mutatePageContent
в patchNode/insertNode/updatePageJson: fail-fast — collab-сессия не
открывается, page-lock не берётся, сообщение детерминировано (mock-тест
ассертит collabTokenFetched===false на битом пути). tableUpdateCell не тронут
(строит абзац из plain text через makeCellParagraph, агентский JSON не глотает).
- Описания patch_node/insert_node/update_page_json: каждый узел, включая
вложенные, несёт строковый `type` из схемы; текст-листы {"type":"text",...}.
sanitizeForYjs (стрип undefined-атрибутов) сохранён — другой класс отказа.
Внутреннее ревью: APPROVE WITH SUGGESTIONS — schema-fidelity/fail-fast/
no-false-positive/precedence подтверждены; замечания необязательны (тест
перечисления схемы, depth-guard безобиден т.к. энкодер падает раньше).
prosemirror-markdown vitest 726/726, mcp node --test 613/613.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#429 (дедуп node-ops) перенёс footnoteContentKey в @docmost/prosemirror-markdown
и удалил footnote-authoring.ts, но два docstring-комментария в
footnote-normalize-merge.ts всё ещё ссылались на старое имя файла. Только
комментарии, на сборку/поведение не влияет.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append `(pageId <id>; transient — retry once; persistent failures mean the
collab server is unreachable/overloaded)` to the connect-timeout, persist-timeout
and connection-closed error texts in CollabSession, so the agent can self-correct
instead of blind-looping. The Yjs-encode error is left untouched (it already names
the offending attribute).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ревью #422: безусловная нормализация переписывала в ASCII и текст inline-code
внутри сносок (кавычки/тире/спецпробелы) — для code-литерала это не типографика,
а изменение смысла (эмпирически на raw-JSON путях: code-нода "a—b «x»" -> "a-b").
normalizeDefinitionText теперь пропускает текст-ноды с маркой code (verbatim), и
краевой trim определения тоже не трогает крайние code-ноды. footnoteMergeKey
читает сырой текст code-нод -> сноски, различающиеся только глифами в коде, НЕ
сливаются, а форки в прозе по-прежнему сливаются. Тесты: code verbatim +
непослияние по code-глифам.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Новый чистый модуль footnote-normalize-merge.ts (normalizeAndMergeFootnotes):
нормализует текст определений сносок (типографские кавычки->ASCII, тире->'-',
NBSP/спецпробелы->пробел, схлопывание пробелов, trim) и сливает определения с
совпавшим нормализованным текстом, перевешивая ссылки на канонический id;
дубли-сироты добивает canonicalizeFootnotes. Ключ слияния attrs-aware
(footnoteMergeKey/stableAttrs) — сноски с одинаковым текстом, но разными attrs
марок (напр. link.href) НЕ сливаются (защита от потери target). Пасс вызывается
строго ПЕРЕД canonicalizeFootnotes на 5 write-путях MCP (markdown-импорт,
updatePageJson, copyPageContent, docmost_transform, insertInlineFootnote).
Глиф-карты продублированы из comment-anchor.ts (там private+завязаны на golden).
Идемпотентен, чистый (deep-clone), scope строго внутри footnoteDefinition.
closes#419
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mutate трекает in-flight одним полем inflightReject; наложенный второй вызов
перезаписал бы рехджектор первого -> при disconnect отклонился бы только второй,
первый бы висел до PERSIST_TIMEOUT_MS (20с). В проде безопасно (оба call-site
сериализуют через per-page withPageLock), но это футган на разделяемом примитиве.
Гард в начале mutate (после ready-проверки, до касания inflightReject): наличие
in-flight -> reject нового вызова без порчи состояния первого. Docstring CONCURRENCY.
Последовательные mutate не задеты (localFinish синхронно чистит inflightReject до
резолва). +2 теста: конкурентный второй реджектится, первый цел; последовательные
оба успешны.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Инцидент: агент заполнял большую таблицу десятками table_update_cell, каждый —
полный цикл connect/auth/load/initial-sync/store/unload; часть падала с 25s connect-
timeout, event loop lag до 1.7с. Вариант B: кэшировать живой HocuspocusProvider
per page на серию правок — пока провайдер жив (connections>0), сервер не входит в
store->unload->reload, дебаунс реально коалесцирует записи (N ячеек -> 1-2 store).
Новый модуль collab-session.ts: класс CollabSession (connecting->ready->dead) +
реестр (ключ wsUrl+pageId+token) с idle-TTL/max-age/LRU-evict. mutatePageContent
(collaboration.ts) и mutateLiveContentUnlocked (client.ts, replaceImage) переведены
на acquireCollabSession; one-shot Promise-машина (~360 строк дублирования) удалена.
5 инвариантов (подтверждены внутренним ревью): (1) read->write атомарна — между
fromYdoc и applyDocToFragment нет await; (2) per-edit ack сохранён дословно (гард
connectionLost от false-success при реконнекте); (3) disconnect=смерть сессии (без
авто-реконнекта, in-flight реджектится теми же текстами ошибок); (4) изоляция
identity (токен в ключе); (5) валидация при reuse. replaceImage работает под
внешним page-локом без дедлока (acquire лок не берёт). Тексты ошибок == develop.
Env: MCP_COLLAB_SESSION_IDLE_MS (0=выкл кэш, точное легаси), _MAX_AGE_MS, _MAX_ENTRIES.
Teardown: destroyAllSessions обвязан в stdio (exit/SIGINT/SIGTERM) + реэкспорт из
index для встраивающего хоста.
closes#400
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Аудит при подготовке #413 нашёл дрейфующие дубли между packages/mcp и
packages/prosemirror-markdown. Четыре дедупа (поведение тулов не меняется):
1. node-ops: форк ~960 строк сведён в ОДНУ копию в prosemirror-markdown (живая
mcp-версия — строгое надмножество замороженного #293-seed'а пакета; сверено по
git-истории, новая пакет-копия байт-в-байт == прежней mcp-копии). Barrel-экспорт
полной поверхности; mcp/client.ts/page-search.ts/transforms.ts/collaboration.ts
импортируют из пакета; тесты переехали. node-ops тянет stripInlineMarkdown ->
пакет-локальная text-normalize.ts несёт только этот примитив (mcp-версия —
домен #408; заголовок документирует дубликацию + источник истины).
2. footnote-lex/footnote-analyze (vestigial legacy [^id]: диагностика): сведены к
одному fence-aware предупреждению 'reference-style footnotes -> use ^[...]'
(полезно для класса #410); footnote-lex удалён.
3. footnote-authoring -> примитивы (footnoteContentKey/makeFootnoteDefinition/
generateFootnoteId) перенесены в пакетный footnote.ts, одна реализация конвенции.
4. parse-node-arg -> перенесён в prosemirror-markdown (не mcp: сервер CommonJS не
импортирует ESM-only @docmost/mcp, но нативно импортирует пакет), обе копии
удалены, консьюмеры перенаправлены.
canonicalizeFootnotes/ENFORCEMENT RULE #228 и comment-anchor/json-edit/text-normalize
(mcp) не тронуты. API-поверхность node-ops оставлена чистой для #409/#413.
closes#414
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- text-normalize.ts: closestBlockHint был вставлен между docstring'ом
stripInlineMarkdown и его определением -> docstring осиротел. closestBlockHint
перенесён ПОСЛЕ stripInlineMarkdown, каждый docstring снова примыкает к своей
функции. Поведение не менялось (только порядок объявлений).
- comment-anchor.ts: header-коммент завышал маршрутизацию — countAnchorMatches НЕ
зовёт resolveAnchorSelection, у него своя параллельная реализация exact-wins.
Коммент уточнён: can/get/apply идут через resolveAnchorSelection, count держит
свой счётчик-примитив, синхронный с ним; обе реализации exact-wins должны
держаться в синхроне при правках.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
createComment — топ-хотспот ошибок агента (промахи по якорю, слепые ретраи).
Портированы аффордансы самокоррекции из editPageText:
- Closest-block hint: общий хелпер closestBlockHint вынесен в text-normalize.ts
(json-edit.ts теперь тоже его зовёт), подключён во все 3 throw-а createComment.
- Markdown-strip fallback в comment-anchor.ts согласованно по всем 4 функциям
(can/count/apply/get) через единый resolveAnchorSelection: exact-verbatim wins
глобально, stripped — только если raw не якорится нигде; soft warning как в
editPageText. Инвариант уникальности suggestion (0/1/>=2) сохранён: raw-unique
никогда не запускает fallback -> не может стать ambiguous. Хранимый selection
остаётся СЫРОЙ подстрокой документа (strip только для поиска).
- Multi-block detection: явное сообщение 'selection spans multiple blocks' когда
per-block поиск провалился, но выделение есть в объединённом тексте блоков.
- tool-spec createComment: копировать selection дословно из getPage/searchInPage.
Известное мелкое ограничение (нит внутреннего ревью): детектор multi-block
использует raw selection, поэтому markdown-стилизованное выделение через границу
блоков получит generic-подсказку вместо spans-multiple-blocks (редко, guidance
всё равно корректный).
closes#408
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Researcher role wrote 40 literal `^[...]` and zero real footnotes: its
incremental write path (insertNode/editPageText) doesn't parse markdown, and
the footnote-capable tool was MCP-only. Promote three tools from inline
MCP-only to the shared registry so the in-app agent gets them too.
- tool-specs.ts: insertFootnote/insertImage/replaceImage added to
SHARED_TOOL_SPECS (mcpName/schema/description moved VERBATIM from the inline
registrations — MCP names + behaviour unchanged for external clients).
- index.ts: the 3 inline registerTool calls become registerShared; drop the
"MCP-only by design" comments.
- ai-chat-tools.service.ts: register the 3 in-app via sharedTool ->
client.insertFootnote/insertImage/replaceImage (imageUrl->url,
attachmentId->oldAttachmentId mapping).
- tool-tiers.ts: insertFootnote -> core (else the original asymmetry recurs —
footnote tool hidden while editPageText is core); images -> deferred.
- research/{en,ru}.yaml FOOTNOTES: `^[...]` parses ONLY on a whole-markdown
write (create/update/import); for a pinpoint citation to existing text use
insertFootnote; via editPageText/insertNode it stays literal.
- json-edit.ts guardrail: an edit_page_text `replace` containing a `^[...]`
token is refused into failed[] with an insert_footnote hint, mirroring the
existing formatting-marker refusal. (Slightly broader net than that mirror —
a literal `^[a-z]` regex class in a replace is also refused; accepted
defense-in-depth, has a no-false-positive test.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Maintainer decision on the escalated ReDoS fork: use re2. The regex path
compiled agent-supplied patterns with `new RegExp` and ran them synchronously in
the shared event-loop; a catastrophic-backtracking pattern (e.g. `(a+)+$`) hung
the whole Node backend for all users (the tool is in both transports incl. the
in-app apps/server agent), and size caps do NOT bound backtracking.
Switch the regex engine to re2 (Google RE2, linear-time, no backtracking):
- `new RE2(query, caseSensitive?'g':'gi')`. RE2 extends RegExp, so eachMatch and
the zero-length-match lastIndex guard are unchanged.
- Unsupported patterns are now a CLEAN error, not a hang: RE2 throws on invalid
syntax AND on the backtracking-only features it can't do (lookaround
(?=…)/(?<=…), backreferences \1) — caught at compile and returned as a clear
tool error telling the agent to rewrite without them.
- Removed MAX_CONTAINER_TEXT + the per-container slice (re2 is linear, so it's no
longer a ReDoS defense, and truncating risked silently dropping real matches in
a long container); kept MAX_PATTERN_LENGTH as a cheap query sanity cap.
- Verified: `(a+)+$` over 50k `a` completes in ~4ms; lookaround/backref throw.
- Added re2 (^1.21.0) to packages/mcp; lockfile updated.
Reviewer DO items:
- F1 [doc]: removed the false "pass nodeId as a comment anchor" claim
(create_comment has no nodeId param — it needs a text `selection`). Fixed in
tool-specs.ts + page-search.ts (module + SearchMatch JSDoc) + client.ts; the ref
is for get_node/patch_node, and for a comment you build a unique text selection
from before+match+after.
- F2 [doc]: clarified `#<index>` refs (id-less table/cell) are accepted by get_node
but NOT patch_node (id-only).
- F3 [test]: round-trip — each match's nodeId fed to the real getNodeByRef
(attrs.id node + `#<index>` table-cell) to prove the ref format is consumable.
- F4 [test]: before/after edge-pinning (match in first 40 chars of a long
container; index 0 → before==""; container end → after=="").
- New re2 tests: catastrophic patterns complete fast; lookaround/backref → error.
mcp: tsc clean; node --test 472 passed (+5). apps/server: tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Editorial roles (Corrector/Factchecker) brute-forced `get_node` block-by-block to
find occurrences (unquoted «ё», straight quotes, «т.е.»), burning tokens. New
`search_in_page(pageId, query, {regex?, caseSensitive?, limit?})` reads the page's
ProseMirror JSON via the existing getPageRaw and searches it IN MEMORY — no server
endpoint, no DB/schema change, no touch to the packages/mcp/src/lib schema mirror.
New pure `searchInDoc(doc, query, opts)` (packages/mcp/src/lib/page-search.ts):
recursive descent to each TEXT CONTAINER (paragraph/heading/table-cell paragraph),
glues its inline text via `blockPlainText` (a match survives inline-mark
boundaries — e.g. «т.е.» split across bold/italic), searches literal (indexOf) or
regex, and returns `{ total, truncated, matches:[{ nodeId, blockIndex, type,
before, match, after }] }`. `nodeId` is the container's attrs.id or the
`#<topLevelIndex>` of the enclosing top-level block — the SAME ref format
get_node/patch_node/comment-anchoring accept (verified identical to getNodeByRef),
so the agent goes straight from a hit to a targeted comment; `before`/`after` are
~40-char windows for a unique selection. `total`/`truncated` always reported (never
silent truncation). Lives in the SHARED_TOOL_SPECS registry → exposed in BOTH
transports (external /mcp + in-app AI-chat), with a SERVER_INSTRUCTIONS line and a
DocmostClientLike signature + contract-test entry. Corrector/Factchecker prompts
get a one-line "use search_in_page first" hint (versions bumped, catalog hash lock
refreshed).
Guards: empty/whitespace query → clear error; invalid regex → clear error (not a
generic 500); zero-length regex matches (`\b`, `a*`) skipped with lastIndex
advanced (no loop/flood); MAX_PATTERN_LENGTH=1000, MAX_CONTAINER_TEXT=100k bound
each exec; limit clamped [1,200] (default 50).
Tests: new page-search.test.mjs (17) — literal+regex, case-sensitivity,
mark-boundary glue, nodeId for paragraph/heading (attrs.id) and table-cell
(#<index> fallback), context bounds, limit/total/truncated + clamp, invalid
regex/empty/over-long errors, zero-length skip, empty-doc null-safety.
mcp: tsc clean; node --test 467 passed (+17). apps/server: tsc --noEmit clean
(DocmostClientLike + wiring). catalog check.mjs OK.
Known limitations (from internal review, non-blocking):
- Residual ReDoS: a crafted catastrophic-backtracking pattern (e.g. `(a+)+$`)
against a large single container can hang the event loop — JS regex is not
interruptible, so the length caps bound the base but not the backtracking.
Realistic exposure is low (containers are small; the pattern is supplied by the
authenticated model). Candidate for a follow-up hardening (safe-regex validation
or a worker+timeout) if it matters.
- Case-insensitive LITERAL search folds via toLowerCase; a char whose lowercase
differs in length (e.g. Turkish İ) BEFORE a match could shift the context
window — negligible for the RU/EN editorial scenario.
- On a `#<index>` table-cell fallback, `type` is the inline container ("paragraph")
while nodeId addresses the top-level block — addressing is correct; the field is
documented as the container's type.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The AI agent (MCP + in-app chat) saw ALL comments incl. resolved via two
channels, cluttering its context and breaking fragment search. Default now:
the agent sees only ACTIVE discussions; resolved is opt-in. Active anchors and
threads are always kept.
Channel 1 — resolved comment anchors on agent reads (converter option):
`convertProseMirrorToMarkdown(content, options?)` gains
`options.dropResolvedCommentAnchors` (default false — zero change for every
existing caller incl. git-sync). Both `case "comment"` emitters (top-level and
the raw-HTML inlineToHtml path) emit BARE text (no `<span data-comment-id>`) when
`resolved && the flag`; active anchors keep their wrapper. mcp `getPage` passes
the flag; `export_page_markdown` does NOT (lossless export must preserve resolved
anchors — that is why it is an opt-in option, not unconditional); `get_page_json`
is untouched (lossless PM JSON). Built on the #293 package converter.
Channel 2 — `list_comments` default active-only: `listComments(pageId,
includeResolved=false)` now returns `{ items, resolvedThreadsHidden }` (was a
bare array). By default a RESOLVED top-level thread is hidden wholesale — the
root AND every reply anchored to it (a thread is gated only by its root's
resolvedAt; a resolved reply under an ACTIVE root stays). `resolvedThreadsHidden`
counts hidden threads so the agent knows to re-query. `includeResolved:true`
returns everything. The `includeResolved` param is added to both tool
registrations (MCP index.ts + in-app ai-chat-tools.service.ts); `DocmostClientLike`
signature updated. Server `findPageComments` is NOT touched — the web UI's tabs
depend on the full feed; filtering is only at the mcp-client level. All internal
call sites (export_page_markdown / checkNewComments / transformPage) updated to
`.items` with `includeResolved:true` to keep their full-feed behavior.
The comment model is assumed FLAT (a reply's parentCommentId points at the
thread root) — documented in the filter; a future reply-of-reply model would
need a root-walk there.
Tests: resolved-comment-anchors.test.ts (6 — anchor dropped with flag / kept
without, for BOTH emitters; active always kept); list-comments-resolved.test.mjs
(4 — resolved thread+reply hidden + counter; includeResolved:true returns all;
an ACTIVE thread with a RESOLVED reply is NOT hidden).
package vitest: 664 passed; tsc clean. mcp: node --test 458 passed; tsc clean.
apps/server + git-sync: tsc clean (converter option default-off).
NOTE: based on feat/293-B (#293/#326 STEP 5) — the converter lives in the
package; this PR is stacked on #333 and its base retargets to develop once #333
merges. mcp/build is gitignored (not committed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
F1 [critical, data-loss] — escape the image alt in ``. Canon #4 moved
the top-level image off the lossless <img> form onto markdown ``, but
the alt was inserted raw; the importer re-parses the `![alt]` label as CommonMark
inline, so a markdown-active char in a realistic description ("Figure [1]", "the
*new* logo", "a]b[c") broke the round-trip — the image node vanished or emphasis
collapsed. Now `escapeLinkText(imgAttrs.alt ?? "")`, exactly as the link-form
media (attachment/pdf/embed) already escape their visible text. Regression test
added: six active-punctuation alts round-trip byte-stable with the node intact.
F2 [drift] — re-export `clampCalloutType` / `sanitizeCssColor` from the package
barrel and drop the verbatim copies in the mcp schema shim. The copies had
already drifted (the mcp `clampCalloutType` lost the callout-type alias mapping
the package applies), which is exactly the schema drift #293 exists to kill. The
sanitizers now live only in the package; mcp `schema.test.mjs` exercises the
single alias-aware implementation.
F3 [docs] — AGENTS.md:296 said `packages/mcp/build/` is committed; this branch
gitignored it (git-sync/prosemirror-markdown convention). Updated the line to say
it is gitignored and rebuilt in CI/Docker via `pnpm build`.
F4 [cleanup] — removed the dead `test.typecheck` block from the package
vitest.config.ts and deleted tsconfig.vitest.json. Both were copied verbatim from
git-sync; this package has zero `*.test-d.ts` files, and the ported comments
referenced git-sync-only entities. Kept the `docmost-client` resolve alias
(22 tests use it) and the runtime include/environment.
package vitest: 658 passed (+1 F1 regression); tsc clean. git-sync: 268 passed.
mcp: node --test 454 passed; tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mcp had its OWN drifted copy of the converter (markdown-converter.ts ~900 lines,
docmost-schema.ts ~1270 lines, markdown-document.ts) — older than the shared
package, missing the git-sync fixes AND the #293 canon. This switches mcp's
converter CORE to @docmost/prosemirror-markdown, so mcp jumps straight to the
canonical format and the drift-generating second copy is gone.
- markdown-converter.ts / markdown-document.ts / docmost-schema.ts become thin
re-export shims of the package (convertProseMirrorToMarkdown, the docmost:meta
envelope, docmostExtensions + docmostSchema=getSchema(docmostExtensions)). The
mcp-only helpers clampCalloutType/sanitizeCssColor are preserved verbatim in
the schema shim (the package doesn't expose them via its barrel). ~2170 lines
of the drifted converter/schema bodies deleted.
- collaboration.ts drops its own ~360-line marked pipeline (preprocessCallouts,
bridgeTaskLists, extractFootnotes, the footnoteRef extension) and re-points to
the package's markdownToProseMirror, keeping markdownToProseMirrorCanonical and
all the yjs/collab write glue. footnote-lex/analyze doc comments updated (they
now describe advisory legacy-syntax diagnostics, not an importer).
Schema parity verified: the package schema is a strict SUPERSET of mcp's old
schema — every node and attr mcp declared is present (the package only adds
status/pageEmbed/transclusion*/subpages.recursive/etc.), so nothing is silently
dropped on the switch. The switch actually FIXES two pre-existing mcp data-loss
bugs its own tests documented: htmlEmbed and pageBreak now round-trip (were
dropped by the old mcp converter).
Footnotes: the package assembles inline ^[body] footnotes on import (sequential
fn-N ids, identical bodies merged), so mcp's canonicalizeFootnotes is now an
idempotent no-op after it (verified). Legacy reference footnotes [^id]/[^id]:
are inert literal text (canon #2 no-backward-compat) — lossless, the text
survives verbatim.
Build hygiene: packages/mcp/build/ is now gitignored and untracked, matching the
git-sync/prosemirror-markdown convention (private package, rebuilt in CI/Docker,
so src and prod can never silently diverge). This also removes a dead untracked
build/_vendored_editor_ext/ artifact that a broad `git add` would otherwise
commit.
Dependency: packages/mcp/package.json gains @docmost/prosemirror-markdown
(workspace:*); pnpm-lock.yaml gets the matching link importer (mirrors git-sync).
mcp tests updated deliberately to the canonical forms (highlight ==, math $…$,
image <!--img-->, drawio/media discriminators, subpages/pageBreak
comments, textAlign, inline ^[…] footnotes) with strict assertions; 4 structural
safety-net round-trip tests added.
mcp: node --test 454 passed; tsc clean. package: 657 passed. git-sync: 268 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
F1 [blocking]: a suggestion whose anchor matched via normalization could never
be applied (spurious 409). The comment mark lands on the doc's ACTUAL text
(Docmost auto-converts to typographic quotes/dashes/nbsp), but the stored
selection — used as expectedText at apply — was the raw ASCII agent input
(+substring(0,250)). So replaceYjsMarkedText's strict joined!==expectedText
always failed and threw "text changed" though nobody edited. Fix: new pure
getAnchoredText(doc, selection) reconstructs the exact raw doc substring the mark
covers (slicing identical to spliceCommentMark); on the suggestion path
client.createComment stores THAT as selection, so expectedText equals the marked
text and apply returns applied:true. Live anchoring still uses the raw agent
selection (normalization still finds the anchor). Truncation raised 250->2000
(+ DTO @MaxLength(2000)) so the anchored substring is never cut below the mark
span. Ordinary comments unchanged. AI-chat shares client.createComment, so
covered. Regression tests: getAnchoredText raw-vs-ASCII; create payload selection
is the typographic substring; apply with typographic expectedText -> applied.
F2 [blocking]: added comment.controller.spec.ts pinning that validateCanEdit runs
before applySuggestion (Forbidden -> applySuggestion never called; happy path ->
called; missing comment -> 404 without authorizing).
MCP 448 pass; server comment+yjs 54 pass. MCP build/ rebuilt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents can attach a suggested replacement when creating an inline comment, via
both the MCP create_comment tool and the AI-chat createComment tool.
Because applying a suggestion edits the EXACT anchored text, an ambiguous anchor
would let Apply corrupt the wrong occurrence. So when suggestedText is set the
selection must occur EXACTLY ONCE:
- new countAnchorMatches(doc, selection) counts occurrences across all blocks
(same normalization/traversal as canAnchorInDoc), counting occurrences (2 in
one block => 2) — stricter than block-count, never under-counting distinct
occurrences (false-unique is the dangerous direction).
- client.createComment gains suggestedText: a pre-check (getPageJson +
countAnchorMatches: 0 => not-found, >=2 => ambiguity error) before create, and
an AUTHORITATIVE live check inside the anchoring mutation that recomputes on the
live doc and, if != 1, aborts and rolls back the just-created comment (reusing
the existing safeDeleteComment "anchor not found" path). Ordinary comments keep
first-occurrence behavior unchanged.
- suggestedText is rejected on a reply or without selection in all three layers
(MCP handler, MCP client, AI-chat tool), mirroring the server DTO/service.
- filterComment surfaces suggestedText/suggestionAppliedAt/suggestionAppliedById.
- DocmostClientLike.createComment signature updated. MCP build/ rebuilt.
Tests: countAnchorMatches (0/1/N, within/across/nested block, span nodes,
quote normalization); createComment (ambiguous refused pre-create, reply and
no-selection rejected, unique succeeds and forwards suggestedText, filterComment
surfaces it); ai-chat schema accepts suggestedText. MCP 443 pass; ai-chat 601 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
F1 (data loss): packages/mcp keeps its own copy of the document schema
(AGENTS.md), and the spoiler mark was only added to editor-ext + the server
tiptapExtensions, so a doc with a spoiler silently lost the mark through /mcp.
Add a local Spoiler mark to docmostExtensions (span[data-spoiler] parse,
data-spoiler="true"+class render) and a case "spoiler" in markdown-converter
emitting the same <span data-spoiler="true">…</span> as the editor-ext turndown
rule; add an MCP json->md->json round-trip test. Regenerated build/lib output.
F2: add the #259 inline-spoiler entry to CHANGELOG [Unreleased] Added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract pure extractAuthTokenFromSetCookie from performLogin (behavior-identical)
so cookie parsing is unit-testable without a network login. Add round-trip
coverage for media attrs (width/height/align/drawio/escaping) the existing
suite omitted; applyAnchorInDoc selection/ambiguity/atom-break cases; and a
cross-copy drift guard proving the vendored editor-ext recreate-transform and
the @fellow npm copy used by diff.ts emit identical steps (apply(diff)==target).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The comment referenced markdownToHtml, which does not exist in the mcp
package; the import path is marked.parse + generateJSON (which runs the
image extension's parseHTML). Describe the actual step and regenerate the
build artifact in sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stock @tiptap/extension-image carries no caption attribute, so
markdownToProseMirror through docmostExtensions dropped the
data-caption the client emits, breaking the lossless claim. Extend the
Image node (mirroring editor-ext image.ts and the nearby Highlight
extend) to parse/render data-caption. Rebuilt build/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Security:
- stash_page: reject path-traversal / percent-encoded srcs before the authed
loopback fetch (resolveInternalFilePath), closing an SSRF/exfiltration hole
where a crafted node.attrs.src could read an arbitrary internal GET endpoint
into the anonymous sandbox.
Stability:
- stash_page: revert + recount mirrors FIFO-evicted by a later put in the same
stash (no dangling sandbox refs, honest images.mirrored/failed); free image
blobs if the final document put throws.
- Reject/clamp non-positive SANDBOX_TTL_MS to the 1h default (warn once).
- Log mirror failures unconditionally (console.warn, no blob bodies).
Cleanup / architecture:
- Remove dead expiresAt from SandboxPutResult.
- Centralize the /api/sb route in SANDBOX_ROUTE_SEGMENT/SANDBOX_API_PATH and
move URL composition into SandboxStore.putAndLink; drop the duplicated sink
closures and the now-unused EnvironmentService injection from McpService and
AiChatToolsService.
- Un-export isInternalFileUrl; document the process-local (instance-bound)
sandbox limitation in the tool description and .env.example.
Docs/tests:
- README/README.ru: 38 -> 39 tools + stash_page entry.
- Add traversal/normalize/recursion unit tests, stash self-eviction +
doc-put-throw + empty/octet-stream mock tests, controller If-None-Match
(wildcard/weak/list) + Cache-Control tests, and SANDBOX_TTL_MS validation
tests. Regenerate packages/mcp/build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an ephemeral, process-local blob store so the in-app agent (and the
embedded MCP) can hand a large page document and its images to an external
consumer WITHOUT routing the bytes through the model context or Docmost auth.
- SandboxStore (@Injectable singleton): Map<uuid,{buf,mime,sha256,expiresAt}>
in RAM only. put() picks a per-blob cap by mime (image vs doc), enforces a
total-bytes RAM guard with oldest-first eviction, and stamps a TTL; get()
lazily expires. sha256 computed at put() doubles as the strong ETag. An
unref'd sweep interval clears expired entries and is cleared on destroy.
- GET /api/sb/:uuid anonymous controller: serves raw bytes with Content-Type,
Content-Length and ETag=sha256; 404 on missing/expired/non-UUID (anti-
traversal), 304 on a matching If-None-Match. No tokens, no 401 — the
capability is the unguessable UUID + short TTL + TLS. Auth-exempt the same
way as /api/files/public (no JwtAuthGuard) plus an /api/sb entry in main.ts's
workspace-resolution preHandler so a remote consumer with no workspace host
is not rejected.
- stash_page tool in both layers (MCP resource_link + in-app {uri,size,sha256,
images}). client.stashPage serializes the get_page_json shape, mirrors every
INTERNAL file/image src (type-agnostic, covers drawio/excalidraw/video/file)
into the sandbox under Docmost auth and rewrites src to the sandbox URL;
external http(s) srcs are left untouched; dedup by src; a failed image fetch
is counted, never aborts the doc.
- SANDBOX_PUBLIC_URL / SANDBOX_TTL_MS / SANDBOX_MAX_BYTES /
SANDBOX_MAX_IMAGE_BYTES / SANDBOX_MAX_TOTAL_BYTES wired through the
environment service + validation + .env.example.
- SandboxModule (@Global) provides the shared store to the controller,
McpService and AiChatToolsService (same instance for put and get).
Tests: SandboxStore (round-trip, sha256, TTL lazy + sweep, caps, eviction),
SandboxController (200+ETag+CT+CL, 404 missing/expired/non-UUID, 304), and a
mock-HTTP stashPage test (mirror+rewrite internal, keep external, dedup, failed
image counted, returns only a link). Interoperates with the vvzvlad/habr-mcp
consumer's anonymous-GET + sha256-ETag + resource_link contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a visible caption (<figcaption>) under images, editable from the
image bubble-menu and persisted across all formats: native Yjs/JSON,
HTML export, and Markdown.
- image node: new plain-text `caption` attribute (parse/render
`data-caption` on <img>, emitted only when set) + `setImageCaption`
command. The node stays an atom; the schema shape is unchanged, so the
server's generateHTML/generateJSON path round-trips it for free.
- resize node-view: re-parent the resizable wrapper into a <figure> and
render the caption in a <figcaption> BELOW it, outside nodeView.wrapper
(so onCommit's offsetHeight measurement and the left/right resize
handles still cover the image only). This path also drives read-only /
share rendering. React placeholder view renders the caption too.
- bubble-menu: new useCaptionControl panel modeled on useAltTextControl
(own icon, Caption strings, softer sanitizer, ~500 char limit).
- markdown lossless round-trip: a captioned image is emitted as a raw
<img data-caption> wrapped in a block <div> (same trick as <video>) in
both the editor-ext turndown rule and the MCP converter; caption-less
images stay clean . Import restores the caption via the
shared markdownToHtml + parseHTML.
- styles + i18n keys; tests for the schema attr round-trip, markdown
round-trip (editor-ext) and the MCP converter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review #6 (approve-with-comments) follow-ups:
1. canonicalize step 7 now strips bare footnoteDefinitions at ANY depth
(stripFootnoteDefinitionsDeep), not just footnotesList, in BOTH copies. A
definition hand-authored outside a list (e.g. nested in a callout via a
raw-JSON write path) was left in place while a copy was also added to the
rebuilt list -> duplicate, idempotent, self-perpetuating. Runs only in the
rebuild path (after the lists are stripped); the fast-path / placement-keep
branch is untouched. Added a shared-corpus case (bare def nested in a callout)
to pin it in both mirrors.
2. markdown-clipboard: removed the dead top-level footnoteReference check in
canonicalizePastedFootnotes (an inline atom is never a top-level slice child;
only the descendants scan can find it).
Test coverage:
4. New MCP binding tests (full-doc-write-canonicalize.test.mjs): update_page_json
and copy_page_content canonicalize the persisted full doc, asserted via a new
`replacePage` seam (symmetric to the existing `mutatePage` seam) so no live
collab socket is needed. Routed both writers through the seam.
5. New server spec (file-import-task.service.footnote-canonicalize.spec.ts): the
zip-import path (processGenericImport) canonicalizes footnotes — real
markdown->HTML->JSON via a real ImportService over a temp-dir .md file, DB trx
stubbed to capture the persisted page content. FileImportTaskService had no
spec before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix:
- insertInlineFootnote could glue a footnoteReference inside an EXISTING
definition (nested footnotesList, or a bare footnoteDefinition with no list
wrapper), which canonicalize then dropped as an orphan — silently losing the
definition's prose. Now: (a) the body/notes boundary is computed from the first
top-level block that IS or CONTAINS (recursively) a footnotesList/
footnoteDefinition, not just a top-level list; and (b) the insertNodesAfterAnchor
core skips footnotesList/footnoteDefinition subtrees entirely (skipSubtreeTypes),
so an anchor whose only match is inside a definition -> inserted:false (clean
abort, no write). Added tests: nested-definition, bare-definition, and
body-before-nested-list-still-inserts.
- editor-ext footnote-canonicalize header listed `markdownToProseMirror` among the
canonicalizing MCP paths; it is the NON-canonicalizing primitive. Replaced with
`markdownToProseMirrorCanonical` (+ note that the plain primitive is for comment
bodies) and added copy_page_content.
- Client paste: canonicalizePastedFootnotes now skips a definitions-ONLY paste
(no footnoteReference anywhere) — canonicalizing it would strip the
reference-less list and yield an EMPTY paste. Added a test.
Suggestions:
- docmost_transform now runs validateDocStructure/validateDocUrls on the RAW
transform output BEFORE canonicalizeFootnotes (mirrors updatePageJson), so a
too-deep doc gives the intended max-depth error instead of a stack overflow.
- docmost_transform tool description now states the RESULT is footnote-canonical
(dryRun diff may show tidy-ups; idempotent after first run).
- insertFootnote: dropped the dead `result ? … : undefined` ternaries and the
`as any` casts (result is always set by the time we return; the not-found path
throws and aborts mutatePage). `const r = result!;`.
Tests / architecture:
- Added a LIVE-plugin golden case: the real footnoteSyncPlugin leaves a list with
non-empty content after it in place, and canonicalize agrees (placement parity
is now a driven property, not a hand-set expected).
- Added generateFootnoteId uuidv7 shape + uniqueness test.
- Item 9: added the ENFORCEMENT-RULE comments at the server parseProsemirrorContent
and the MCP canonicalizer header (any NEW full-doc persist path MUST canonicalize;
fragments/append/prepend and comment bodies MUST NOT). Kept per-call-site over a
brittle grep CI test (the replace-vs-fragment + comment-vs-page nuance makes a
single wrapper unsafe).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix (REAL DATA LOSS):
- markdownToProseMirror is reused for COMMENT bodies (createComment/updateComment).
It unconditionally canonicalized, so a comment carrying a standalone footnote
definition ([^1]: text with no matching reference) had its whole footnotesList
stripped (referenceIds.length===0 -> stripFootnotesListsDeep) — the text
vanished. Fix: markdownToProseMirror no longer canonicalizes (content-preserving
primitive); a new markdownToProseMirrorCanonical wraps it for the PAGE write
paths (markdown import via importPageMarkdown, update_page markdown via
updatePageContentRealtime). Comment callers keep the non-canonicalizing
primitive. Updated the now-false header comment and added create/update-comment
inline notes. Added collaboration tests: comment path PRESERVES a reference-less
definition; page path still drops it AND still reorders real footnotes. Updated
the page-import canonicalization test to use the canonical variant.
Suggestions / architecture:
- #2: collapsed transforms.footnoteDefinition onto the shared
makeFootnoteDefinition factory (adds only the inner paragraph block id); kept
the dependency direction transforms -> footnote-authoring (no circular import,
mirror stays pure).
- #3: confirmed docmost_transform auto-canonicalization is documented (inline
comment, tool description, CHANGELOG) — no code change.
- #4: copyPageContent is a FULL-document write (replacePageContent of a
type:"doc"); added a defensive canonicalizeFootnotes pass (no-op on
already-canonical source).
- CHANGELOG entry refined to list the FULL-document write paths (incl.
copy_page_content) and to state canonicalization is NOT applied to comment
bodies.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix:
- REAL BUG: insertInlineFootnote could splice a footnoteReference (inline atom)
into a codeBlock or an existing footnoteDefinition, persisting a schema-invalid
doc (insert_footnote skips validateDocStructure). Now the search is bounded to
the BODY (before the first footnotesList) and the insertNodesAfterAnchor core
refuses textblocks that can't hold the atom (codeBlock); when the only match is
in such a place the insert returns inserted:false and the write aborts cleanly.
Reachable via docmost_transform too. Added codeBlock / definition / fall-through
tests.
- Fixed the deepEqualJson doc comment in both copies: arrays are order-SENSITIVE
(correctness depends on it), only object keys are order-insensitive.
- README.ru.md MCP tool count 38 -> 39 (lines 36/47/63), matching README.md/AGENTS.
- CHANGELOG [Unreleased] Added entry for insert_footnote + server-side footnote
canonicalization on non-editor write paths (#228).
Suggestions:
- canonicalize step 5/7 now strips footnotesList at ANY depth (both copies), so a
schema-valid list nested in a callout/blockquote can't leave duplicate defs.
- Exclude the test-only footnote-corpus.ts fixture from the editor-ext build
(tsconfig), so it no longer ships in dist/.
- Removed the duplicate manual canonicalize cases from the MCP unit test (the
shared corpus covers them via full deepEqual); kept idempotence + immutability.
- insertInlineFootnote dedup key now keys off the inline array directly
(footnoteContentKey({ content: inline })) instead of a throwaway node.
Tests / architecture:
- New client-wrapper test (#9): overrides a small mutatePage seam to assert the
not-found path throws and persists NOTHING, and the success path shapes
footnoteId/reused/message/verify and writes the right content. Fixed the
misleading comment in footnote-write.test.mjs.
- B: cross-copy corpus parity guard test (loads both corpora, asserts deep-equal)
so a typo in one copy can't pass both suites green.
- A: declined — the full-vs-fragment decision lives at the call site, so a
prepareDocForPersist wrapper would be a bare alias for canonicalizeFootnotes;
kept the existing per-call-site comments instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix:
- Move canonicalizeFootnotes OUT of parseProsemirrorContent. It now runs only
on FULL writes (createPage, updatePageContent operation==='replace'), never on
an append/prepend fragment (a fragment would lose definition-only footnotes or
synthesize a bogus empty list). Add a server binding spec.
- Match the live plugin's list PLACEMENT: a single already-canonical
footnotesList is left exactly where it sits (the plugin never repositions a
sole correct list), so the first write no longer reorders content that follows
the list. Applied to BOTH the editor-ext copy and the MCP mirror; pinned by a
shared golden corpus case with content after the list.
- Fix MCP tool count 38 -> 39 (README x3, AGENTS.md) and the transformJs param
help (add canonicalizeFootnotes/insertInlineFootnote).
Simplifications:
- Remove the dead duplicate re-id mechanism (deriveFootnoteId/suffix/occurrence)
from the PURE canonicalizer in both copies — references are never renamed, so
the derived ids were never requested; first-wins-drop is the real behaviour.
This also makes the editor-ext footnote-util note about "no cross-package copy"
true again.
- Remove the sentinel round-trip in insertInlineFootnote: a generalized
insertNodesAfterAnchor core inserts the footnoteReference node directly.
- Drop the redundant per-definition deep clone in step 4 (shallow id-normalizing
copy; out is already deep-cloned).
Docs / architecture:
- Correct the editor-ext copy's "It exists because…" header to its real
consumers (server import, page.service create/update, client paste).
- Note markdownToProseMirror reuse for create/update comment in collaboration.ts.
- A: shared golden JSON corpus exercised by BOTH the editor-ext copy and the MCP
mirror (footnote-corpus.ts / .mjs) so "the two copies behave identically" is
checkable.
- C: split the MCP canonicalizer into a pure mirror + footnote-authoring.ts.
- B: import services persist via a different path, so left one-line consolidation
comments at the call sites rather than folding (does not fall out cleanly).
Tests: insertFootnote wrapper guards + docmost_transform dryRun auto-canonicalize
(MCP mock), page.service create/update + append/prepend binding (server jest),
shared corpus incl. nested-container reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make footnotes author-inline: the agent/tool inserts a footnote at its point
of use (anchor + text) and the numbering plus the bottom list are DERIVED
deterministically server-side. The agent has no access to footnotesList and
cannot desync — out-of-order lists, orphan definitions, and raw trailing
[^id] blocks become structurally impossible.
editor-ext:
- canonicalizeFootnotes(docJSON) -> docJSON: a pure, EditorView-free port of
footnoteSyncPlugin's end-state. Distinct reference ids in document order are
the source of truth; exactly one trailing footnotesList holds one definition
per referenced id in reference order (reusing the existing node or
synthesizing an empty one); orphans dropped; duplicate definitions resolved
deterministically (first wins, never lost); idempotent.
- Unit tests + a golden parity suite: on every editor-reachable steady state
the live footnoteSyncPlugin's JSON is a canonicalize no-op (byte-for-byte
parity), and the canonicalizer additionally repairs the out-of-order list a
non-editor write produces.
mcp:
- footnote-canonicalize.ts: behavioural mirror of the editor-ext canonicalizer
(the MCP package is intentionally decoupled from the editor barrel, like
footnote-lex/docmost-schema), plus footnoteContentKey for content dedup.
- Auto-canonicalize on EVERY write path: markdownToProseMirror (fixes import
ordering), update_page_json, and after every docmost_transform. Idempotent,
so it is a no-op when footnotes are already canonical.
- insert_footnote tool + insertInlineFootnote: anchor + markdown text -> a
mark-safe footnoteReference and a content-dedup'd definition; the list and
numbering are derived. Same-content footnotes reuse one number/definition.
- canonicalizeFootnotes + insertInlineFootnote exposed as docmost_transform
sandbox helpers.
Tests: editor-ext 157 green; MCP 325 green; server + client tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8-point multi-aspect review of the batch PR; security/regressions were clean.
1. Lease leak: the #180 reorder moved `toolsFor` (which leases external MCP
clients, refCount+1) ahead of buildSystemPrompt + forUser, but the only
release (closeExternalClients) was bound to the streamText callbacks. A throw
in between leaked the lease (refCount stuck, undici sockets held until
restart). Define closeExternalClients right after the lease and wrap
buildSystemPrompt+forUser in try/catch that closes-then-rethrows.
2. Cover the patch_node/delete_node dup-id refusal (#159#6): extract the guard
into a pure `assertUnambiguousMatch` (node-ops) and unit-test 0/1/>1.
3. Regress the body-before-title order (#159#10): mock-HTTP test (collab fails
fast against a server with no WS upgrade) asserts /pages/update (title) is
NEVER posted when the body write fails — for updatePage AND updatePageJson.
4. CHANGELOG [Unreleased]: #180, #168 (Added); #163 (Fixed).
5. Add the missing en-US i18n keys (Back to references / {{label}}).
6. Drop the duplicate content/empty/blank cases in ai-chat.prompt.spec.ts (they
repeat the buildMcpToolingBlock unit tests); keep only sandwich placement +
both-safety-copies.
7. CI Postgres pg16 -> pg18 (match docker-compose).
8. jsonb decode seam: shared `parseJsonbValue(value, guard)` in database/utils.ts
holds the legacy double-encoding self-heal in one place; parseToolAllowlist /
parseModelConfig keep only a type-guard.
Verified: server build + 124 unit + 15 integration; mcp 311; prettier clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the merged #166/#169. Addresses the second review pass (comment
1227):
- footnoteWarnings plumbing: extract a single `footnoteWarningsField(markdown)`
helper (footnote-analyze) and use it at all three call sites (create_page,
update_page, import_page_markdown) so the field is attached identically.
- New unit test footnote-warnings-import.test.mjs pins the contract that was
uncovered: the field is present on problems / omitted on clean input, and the
IMPORT path analyzes the BODY after the docmost:meta / docmost:comments blocks
(a footnote-like token inside those JSON blocks must NOT warn; a real body
marker must). Tested via the same pure composition the importer uses
(footnoteWarningsField(parseDocmostMarkdown(full).body)) — no collab socket
needed; a regression that analyzed fullMarkdown or skipped the body split would
now go red.
- footnote.marked.ts: correct the stale module header — it claimed "only
definitions that have a matching reference are emitted", which was never true
(orphan defs are emitted; the editor sync plugin reconciles). Now describes
first-wins + reuse + sync reconciliation.
- derive-id golden test: rename the describe from "(cross-package drift guard)"
to "(deterministic-scheme pin)" — there is no second package to drift against.
editor-ext 129, MCP 304 (+3), client+server tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- footnote-sync: remove the now-dead `refReids` (CollisionPlan field, local,
return, the 6a consumer loop) — references are never re-id'd under reuse, so it
was dead structure on the hot reconciliation path. Rewrite the stale comments
(plugin header, step 0, refOccurrences field) that still described the old
"duplicates re-id'd so both survive" model to the reuse model.
- Shared footnote lexer: new packages/mcp/src/lib/footnote-lex.ts
(lexFootnoteLines + forEachFootnoteReference). extractFootnotes (collaboration)
and analyzeFootnotes now consume the SAME fence-aware lexer, so "the analyzer
sees exactly what the importer keeps/strips" is structural, not comment-kept.
Removed the duplicated DEF_RE/fence machine from both consumers.
- Tests: new mock test for the footnoteWarnings plumbing on createPage (problems
-> field present; clean -> omitted); new paste-reuse case for TWO colliding
pasted definitions (reservation -> distinct ids). Updated the derive-id golden
test header (no MCP copy / parity test anymore).
- CHANGELOG: [Unreleased] entries for footnote reuse (Changed, supersedes 0.93.0)
and footnoteWarnings (Added).
editor-ext 129, MCP 301, server roundtrip 2; client+server tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Footnotes were strict 1:1: a repeated `[^a]` reference was treated as a
collision and re-id'd to `a__2`, and a reference with no definition synthesized
its own empty one — so an agent-authored article with reused labels produced
dozens of empty `kowiki__N` footnotes. Move to Pandoc REUSE semantics and add
non-fatal import diagnostics.
Reuse (core):
- resolveCollisions (footnote-sync): repeated references sharing an id are REUSE
(recorded once in document order, never re-id'd) — one number, one shared
definition. Only a duplicate DEFINITION is re-id'd deterministically and, with
no matching reference, dropped by the existing orphan policy (first-wins).
CollisionPlan.refReids is now always empty (harmless no-op downstream).
- extractFootnoteDefinitions (marked) and extractFootnotes (MCP): duplicate
definition ids are FIRST-WINS (keep first, drop rest); reference markers are
never rewritten. Removed the marker-rewriting and the now-dead deriveFootnoteId
mirror + helpers from the MCP path.
Import diagnostics:
- New analyzeFootnotes() (MCP): fence-aware pure scan reporting dangling
references, empty/duplicate definitions and `[^id]` markers inside table rows.
- createPage / updatePage / importPageMarkdown now attach `footnoteWarnings`
(only when non-empty) so an agent can fix its markup; the page is still created.
Paste-reuse:
- footnotePastePlugin remaps only ids the pasted slice DEFINES (a colliding
definition); a pasted lone reference to an existing id keeps it (reuse).
Tests: reuse/first-wins rewrites of footnote.test, footnote-markdown.test,
footnote.marked.orphan.test and the MCP footnotes.test; new footnote-paste.test
(editor-ext) and footnote-analyze.test (MCP). Deleted derive-id-parity.test.mjs
(the MCP no longer derives ids; editor-ext's deriveFootnoteId keeps its own
golden test). editor-ext 128, MCP 299, server roundtrip 2, client views 3,
client+server tsc clean.
Two review suggestions applied: corrected a stale "duplicated in MCP" comment and
the dangling-reference warning wording.
Note: the multi-backlink editor UI (a reused definition linking back to each of
its references) is deferred to a follow-up — this PR delivers the data-integrity
core (reuse + warnings + paste-reuse). Forward links and numbering already reuse
correctly; the backlink currently targets the first reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the approve-with-comments review on PR #154:
- applyDocToFragment: hydrate PMNode.fromJSON in its OWN try so a hydration
failure (e.g. an unknown node type) is labelled "fromJSON" — the stage that
actually threw — instead of the misleading "updateYFragment". The diagnostic
comment on unstorableYjsError ("label names the stage that failed") is now
truthful.
- assertYjsEncodable: also rehearse PMNode.fromJSON(docmostSchema, …) so a doc
that would only fail in apply's hydration step is rejected at preview time too,
narrowing the preview/apply gap (review suggestion B). Still cheap — no live
fragment, no updateYFragment.
- Tests: relabel the diagnostic test to (fromJSON); add structural-diff edge
cases — neighbour deletion keeps the unchanged node's cursor anchor, doc->empty
clears the fragment without throwing, top-level node-type change diffs in
place — plus a preview-gate test for the new fromJSON rehearsal. 297/297 green.
build/ rebuilt for the changed lib module only (build/client.js left untouched
to avoid pulling in pre-existing unrelated src/build drift).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agent_coder
vvzvlad
claude code agent 227
a