Compare commits

...

57 Commits

Author SHA1 Message Date
agent_coder f6fc914c95 test(client): doc-changed-guard тесты вставки — сделать нехолостыми (#347, ревью F3)
Ревьюер мутационно доказал: 3 теста doc-changed-guard были ВХОЛОСТУЮ — зелёные даже
при обоих гардах `if(false)`. Причина: вставка в ПУСТОЙ курсор (from==to==1) +
mutateDoc только РАСТИТ док → протухший нулевой диапазон всегда валидная точка
вставки: replaceRange(1,1,…) не затирает, растущий док не выводит `to` за границы,
RangeError не бросается. Гард-код корректен — вхолостую были ТЕСТЫ.

Переписаны по рецепту ревьюера:
- success: вставка поверх НЕПУСТОГО выделения ("AAAABBBB", selection {1,5}); mid-flight
  вставить "MARKER" в pos 1 + курсор в конец. Рабочий гард → замена в живой (конечной)
  selection, MARKER цел; сломанный → stale {1,5} стирает голову MARKER.
- fail-open: захваченный `to` выводится ЗА ГРАНИЦЫ — после захвата {1,9} и провала
  конверсии док СЖИМАЕТСЯ до пустого параграфа. Рабочий гард → raw-текст в живую
  selection; сломанный → insertText(md,1,9) на size-2 доке → RangeError, ничего не
  ложится.
- two-pastes оставлен (пинит «ни один payload не потерян», не гард).

Мутационно проверено: оба гарда→if(false) → тесты 1 и 2 КРАСНЕЮТ (stale-range
clobber; stale-`to` RangeError), 3-й и не-гард-тесты зелёные; реальный гард
восстановлен → 6/6. Тесты теперь отличают рабочий гард от сломанного.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:34:43 +03:00
agent_coder 1d89cc2058 fix(client): ревью #347 — Reasoning-panel отступы, диагностика вставки, тест guard'а, комменты, доки (#347, ревью)
Правки по 5 находкам ревью #498.

F1 (регрессия отступов списков в Reasoning-панели): добавлен `.reasoningText li p
{margin:0}` (зеркало существующего `.markdown li p`). Reasoning рендерит через тот
же renderChatMarkdown (теперь всегда <li><p>…</p></li>), но под .reasoningText, где
`.reasoningText p{margin:0 0 4px}` давал 4px на пункт. Обе поверхности покрыты.

F2 (глухой catch): `.catch` вставки был `()=>{}` → теперь `(err)=>console.error(
"markdown paste conversion failed, inserting raw text", err)` — тихая деградация в
raw-текст больше не невидима (покрывает и конвертер, и тело success-.then, напр.
PMNode.fromJSON при дрейфе схемы).

F3 (нет теста doc-changed guard): +3 теста в markdown-clipboard.paste.test.ts:
success-ветка при mid-flight изменении дока → вставка в живую selection (маркер
цел, без клоббера/throw); fail-open ветка при mid-flight + провале конверсии →
raw-текст в живую selection без RangeError; две вставки в полёте → инвариант «ни
один payload не потерян».

F4 (устаревшие комменты): исправлены ссылки на удалённый md-слой в markdown-
clipboard.ts, footnote-sync/util(+test), docmost-schema, foreign-markdown,
footnote-canonicalize → на @docmost/prosemirror-markdown / локальные символы.

F5 (внешние доки): AGENTS.md (apps/client как потребитель через browser-entry,
jsdom только в Node, удалён marked/turndown-слой); prosemirror-markdown/README
(секция Node vs browser entry, markdownToProseMirrorSync); CHANGELOG.

Тесты: client paste+canonicalize+ai-chat 61; pmd 744; editor-ext 196; клиентская
сборка успешна, grep бандла на JSDOM/parse5/happy-dom/turndown — 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:02:08 +03:00
agent_coder 5d8083f8ff refactor(client): вставка markdown через канонический @docmost/prosemirror-markdown + удаление md-слоя editor-ext (#347)
Третий шаг #345: клиентская вставка markdown переезжает с marked-слоя editor-ext
(не знал канона — ^[…], <!--img {…}-->, <!--subpages--> при вставке не
распознавались) на канонический пакет. По завершении слой удалён целиком. closes #347

- Browser-entry пакета (инъекция DOM-парсера): jsdom только в Node-пути.
  dom-parser.ts — 2 слота инъекции (HtmlDocumentParser для HTML→Document,
  GenerateJsonFn для @tiptap/html), без импорта DOM. dom-parser.node.ts
  регистрирует jsdom + @tiptap/html/server; dom-parser.browser.ts — нативный
  DOMParser + @tiptap/html (browser), экспонирован через exports-условие
  "browser" + сабпас "./browser". markdown-to-prosemirror.ts: убраны статический
  импорт jsdom и module-level global.window-шим. Клиент ВСЕГДА импортирует явный
  сабпас /browser — не полагается на порядок условий. Node-потребители (mcp/
  server) идут по "." → default → index.js → jsdom, не затронуты.
- markdown-clipboard.ts: конвертация через browser-entry (markdownToProseMirror
  → PM-JSON → HTML через живую схему редактора DOMSerializer → НЕИЗМЕНЁННЫЙ
  downstream-шов normalizeTableColumnWidths→parseSlice→canonicalizePastedFootnotes
  →dispatch). Эвристики/fragment-insertion не тронуты. Конвертер async → handle
  Paste захватывает диапазон, забирает событие, диспатчит на резолве; и success,
  и fail-open ветки защищены guard'ом doc!==startDoc (не диспатчить по устаревшему
  диапазону). clipboardTextSerializer (copy PM→md) — через convertProseMirror
  ToMarkdown.
- Удалён packages/editor-ext/src/lib/markdown/ целиком (+ marked из package.json).
  Мигрированы ВСЕ потребители markdownToHtml/htmlToMarkdown: ai-chat/utils/
  markdown.ts (→ новый markdownToProseMirrorSync + DOMSerializer), use-generate-
  page-title.ts / page-header-menu.tsx (→ convertProseMirrorToMarkdown(getJSON)),
  серверный spec. Grep: осиротевших импортов нет, editor-ext = только схема/
  расширения. Turndown ушёл из бандла (был в старом htmlToMarkdown).
- AI-чат теперь рендерит markdown через схему редактора (li в <p>); добавлен
  .markdown li p{margin:0} (CSS-модуль, скоуп только чата) — визуально плотно.

Проверка: pmd tsc + vitest 744; client build УСПЕШЕН, grep бандла на
JSDOM/parse5/happy-dom/turndown — 0 (утечки нет); клиентский suite + paste-тесты
зелёные (34); editor-ext 196; node-потребители (mcp/server/git-sync) зелёные.
Юнит-тесты: dual-path parity (jsdom==DOMParser), канон-формы == серверный импорт,
негативы ($5/==/[^1] не корёжатся), async-paste (claim→convert→dispatch, fail-open).

Ручная paste-QA полного клиентского round-trip (footnote/callout/math/image-comment;
вставка из VSCode/Obsidian/GitHub в список/таблицу/callout) и Docker-сборка клиента
(#333-класс) — за пределами автостенда, оставлено на ручную проверку.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 02:11:19 +03:00
vvzvlad 6d7dba970c Merge pull request 'refactor(mcp): структурные инварианты конкурентной записи — UUID-assert, self-resolve seams, no-await-guard (#449)' (#475) from refactor/449-write-invariants into develop
Reviewed-on: #475
2026-07-11 01:32:30 +03:00
vvzvlad 512bcba5f3 Merge pull request 'refactor(mcp): распил client.ts (5206 строк) на доменные модули за тонким фасадом (#450)' (#478) from refactor/450-split-client into refactor/449-write-invariants
Reviewed-on: #478
2026-07-11 01:30:28 +03:00
agent_vscode 5c1ab9c7b5 fix(docker): ship packages/mcp/data into runtime image
drawioFromGraph и каталог фигур читают данные в рантайме по пути
packages/mcp/data/ (относительно build/lib/*.js через import.meta.url).
tsc эмитит только build/, а стадия installer копировала лишь build/ и
package.json — в образе не было ни drawio-presets.json (#425), ни
drawio-shape-index.json.gz (#440), из-за чего drawioFromGraph падал с
ENOENT, а иконки каталога фигур молча не резолвились.

Добавлен COPY packages/mcp/data в стадию installer. .dockerignore /data
заякорен на корень и этот путь не затрагивает.
2026-07-11 01:13:48 +03:00
agent_coder 14d7b21df0 refactor(mcp): распил client.ts (5206 строк) на доменные модули за тонким фасадом (#450)
client.ts был god-object'ом на 5206 строк / ~65 методов / 5 ответственностей —
любая правка рисковала всем write-path. Разнесли на доменные модули;
DocmostClient остаётся ТОНКИМ ФАСАДОМ с прежним внешним контрактом. Чистый
рефакторинг, поведение не меняется. closes #450

- Фасад client.ts (93 строки) композирует 10 миксинов над общим абстрактным
  базовым классом. Паттерн МИКСИНЫ (не context-object): тесты субклассируют
  DocmostClient и переопределяют seam'ы; единая цепочка прототипов сохраняет
  виртуальную диспетчеризацию this.<method> сквозь модули.
- client/context.ts (база): общее состояние (axios, apiUrl, токены, кэши),
  конструктор + оба интерсептора, login/ensureAuthenticated/paginateAll/
  resolvePageId/mutateLiveContentUnlocked + write-seam'ы mutatePage/replacePage.
  private→protected (внутреннее, не в публичном контракте).
- Модули (каждый ≤730): read, pages, nodes-write, media (images/attachments/
  drawio), comments, transforms, tables, stash, doc-validate. errors.ts —
  единый REST error-mapping (довершение #437; тексты сообщений без изменений).
- Внешний контракт СОХРАНЁН (доказано компиляцией с обеих сторон): 53 публичных
  метода через `implements IXMixin` (без ручного зеркала, #446); оба Pick
  (in-app loader + tool-specs) резолвятся; множество async-методов 59==59.
  Нагруженные seam'ы (replaceImage один-лок #425, self-resolve #449, единый
  error-путь) байт-идентичны; ни одного дубля публичного метода.
- zod v3→v4: investigate-only, отложено — SDK 1.29 поддерживает v4, но мажор-
  бамп трогает всю схема-поверхность; отдельным follow-up.

Стоит на #449 (#475). Тесты: mcp node --test 800/800 (== база), tsc чисто в
client/. Ни одной строки кода не потеряно; убраны 52 дублированных JSDoc-блока.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 01:04:42 +03:00
agent_vscode 363f20ab75 docs(agents): add architectural invariants (non‑negotiable rules)
Introduce a new “ARCHITECTURAL INVARIANTS — NON-NEGOTIABLE” section that
lists ten hard constraints derived from past production incidents. These
rules act as non‑negotiable guidelines for future development and code
review.
2026-07-11 01:04:10 +03:00
agent_vscode 3411bda2d1 fix(mcp): adapt e2e node-ops calls to the #413 XOR input of patchNode/insertNode
#413 changed patchNode/insertNode to take { markdown? | node? } (exactly
one), but test-e2e.mjs still passed the raw ProseMirror node directly,
so the e2e-mcp CI job died with the XOR guard error right after the
node_ops seed step. Wrap both call sites in { node: ... }.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 00:08:00 +03:00
agent_vscode e670f7498a fix(ai-chat): sync CORE_TOOL_KEYS with #443 core tools — restore mcp-server-parity
The #443 merge added getTree and getPageContext to the shared registry
(packages/mcp/src/tool-specs.ts) with tier 'core', but the server-side
authoritative core list CORE_TOOL_KEYS was never updated. Two guard tests
in tool-tiers.spec.ts failed on develop (CI job test / mcp-server-parity):
tier agreement SHARED_TOOL_SPECS <-> CORE_TOOL_SET, and the live-toolset
<-> deferred-catalog partition (both tools were live, non-core and absent
from the catalog).

- add 'getTree' and 'getPageContext' to CORE_TOOL_KEYS (kept core, not
  demoted: cheap single-call navigation tools; core listPages description
  itself points to getTree)
- update the CORE_TOOL_KEYS JSDoc accordingly
- pin the first tool-tiers spec to the new 17-entry list with membership
  assertions for both #443 tools

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 00:03:34 +03:00
agent_coder 9a8671c3af refactor(mcp): структурные инварианты конкурентной записи — UUID-assert в page-lock, self-resolve seams, no-await-guard (#449)
Механика конкурентной записи (withPageLock → acquireCollabSession → mutate)
держалась на цепочке конвенций в комментариях. Новый write-метод без знания
правил компилировался и уходил в прод (класс #260/#152/#159). Закрепляем ключевые
инварианты кодом.

- page-lock.ts: экспортированы UUID_RE/isUuid (тот же regex, что resolvePageId,
  UUID v1–8/v7). withPageLock FAIL-FAST кидает при не-UUID ключе ДО любой работы
  (комментарий-инвариант #260/#449) — забытый resolve/slugId больше не даёт тихую
  потерю сериализации под другим ключом. client.ts импортирует isUuid оттуда
  (убран локальный дубль — resolver и assert не разъедутся).
- mutatePage/replacePage seams стали async и сами вызывают resolvePageId — ключ
  лока/кэша канонический даже если вызывающий забыл (для уже-UUID это cached
  no-op; все 7 текущих вызывающих и так резолвят). replaceImage (один внешний
  лок + mutateLiveContentUnlocked) не тронут, deadlock невозможен.
- collab-session.ts: машинно-проверяемые маркеры MUTATE-CRITICAL-WINDOW
  BEGIN/END вокруг синхронного блока fromYdoc→applyDocToFragment (INVARIANT 1).
  Тест no-await-critical-window читает исходник и краснеет на await/yield в окне
  (проверено нейтером). Случайный await больше не тихо клоббит живые правки.
- Документация осознанной позиции: single-instance/sticky-sessions —
  требование деплоя (Dockerfile + README EN/RU + .env.example), т.к. мьютекс и
  stash — per-process. Окно устаревших прав (кэш-сессия пишет под токеном
  момента connect до MCP_COLLAB_SESSION_MAX_AGE_MS=10мин) — задокументированный
  trade-off в .env.example; push-инвалидации нет (осознанно).

Тесты: page-lock fail-fast (slugId/пусто/non-string → throw; канонический UUID
принят), no-away-guard, обновлённые фикстуры на валидные UUID. #449-специфичные
37/37 зелёные; mcp tsc чисто. closes #449.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 23:51:02 +03:00
vvzvlad bec2156e96 Merge pull request 'recovery: хвост стопки breaking-окна — #413, #415, #443 (3 части), #425 в develop' (#474) from feat/443-get-page-context into develop
Reviewed-on: #474
2026-07-10 23:49:29 +03:00
vvzvlad acc705de19 Merge pull request 'feat(mcp): drawio стадия 3 — семантические тулы fromGraph/fromMermaid/editCells (#425)' (#472) from feat/425-drawio-graph into feat/443-get-page-context
Reviewed-on: #472
2026-07-10 23:45:45 +03:00
vvzvlad a49872444e Merge pull request 'refactor(mcp)!: BREAKING — все имена MCP-тулов snake_case → camelCase (#412)' (#463) from refactor/412-camelcase-tool-names into develop
Reviewed-on: #463
2026-07-10 23:44:42 +03:00
vvzvlad ffc38ea2ca Merge pull request 'fix(mcp): pre-flight size-guard в diffDocs против CPU-DoS на больших правках (#464)' (#465) from fix/464-diffdocs-cpu-guard into develop
Reviewed-on: #465
2026-07-10 23:44:20 +03:00
agent_coder 2a951df096 feat(mcp): drawio стадия 3 — семантические тулы drawioFromGraph/FromMermaid/EditCells (#425)
Сырой XML остаётся escape-hatch, но для 90% случаев модель не видит ни координат,
ни style-строк — весь класс ошибок лейаута и иконок уходит by construction: эти
решения принимает сервер, а не LLM. Стоит на #443, переиспользует конвейер #423 и
shape-index/elkjs/линтер #424.

- drawioFromGraph(pageId, where, graph, direction?, preset?, layout?): граф
  узлов/групп/связей → резолв иконок (shape-index #424; неизвестная → generic по
  kind с подписью, не пустой квадрат), стили из пресета (kind→палитра),
  elkjs-layered с compound-группами, ассемблер XML (линтер-чистый by construction:
  зазоры >=150, прозрачные контейнеры, относительные координаты детей,
  cross-container рёбра parent=1, эскейп меток). Хинты pinned/sameLayerAs/layer и
  layout none/full/incremental — детерминированным post-pass'ом (ELK-констрейнты
  оказались ненадёжны). Пресеты default/dark/colorblind-safe (Okabe-Ito) — данные.
- drawioFromMermaid(pageId, where, mermaid): чистый парсер flowchart (без
  браузера/CLI) → graph → тот же конвейер. Формы/направление/пунктир=async/
  subgraph→группы/цепочки; не-flowchart отвергает внятно.
- drawioEditCells(pageId, node, operations, baseHash): ID-based add/update/delete,
  delete каскадит на детей контейнера и связанные рёбра; baseHash обязателен
  (optimistic lock как drawioUpdate); сентинелы 0/1 от delete защищены.

DoS-границы (LLM-вход): MAX_GRAPH_EDGES=1000/GROUPS=500 в validateGraph (узлы уже
500), mermaid MAX_CHARS=200k/LINES=20k/GROUPS=500, chain 500 — все с быстрым
throw ДО лейаута/ассемблера (ассемблер и маппер вне ELK-таймаута, иначе OOM
воркера). incremental сохраняет неперечисленные существующие ячейки (mergeExisting
Cells) — «добавь узел» не стирает ручную расстановку. sameLayerAs/layer после
снапа раскладываются по перпендикулярной оси с зазором >=150 → 0 quality-warnings;
pinned — точные пользовательские координаты (clamp>=0, могут дать warning, гарантия
«0 by construction» относится к авто-лейауту).

Регистрация: 3 shared-spec на оба хоста (camelCase, execute-in-spec, inlineBoth
Hosts не понадобился), DocmostClientLike/Method += 3, contract, routing-проза
(fromGraph→архитектуры/облака, fromMermaid→стандартные flowchart, raw xml→экзотика).

Тесты: mcp node --test 782/782 (57 новых) — приёмка (15+ узлов/2 вложенные группы/
AWS-иконки→0 lint/0 warnings/иконки резолвятся, hints, incremental без сдвига,
edit_cells update/cascade/stale-baseHash, снапшоты пресетов + colorblind-safe,
mermaid ветвление+subgraph) + регрессии на DoS-границы. tsc чисто; server jest
(contract + ai-chat) 290/290. closes #425.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 23:16:50 +03:00
agent_coder 5dc7a2703f feat(mcp): getPageContext — «где я / что вокруг» по pageId одним вызовом (#443, часть 3/3)
Финальная из трёх частей #443. Есть pageId (из поиска, из ссылки) — нужно понять
местоположение и окружение. Раньше это get_page + полное дерево. Только чтение,
только метаданные (без контента). closes #443.

- client.ts getPageContext(pageId): resolvePageId (slugId→uuid, для uuid без
  round-trip) → POST /pages/breadcrumbs + listSidebarPages(spaceId, uuid). Два
  запроса для uuid-входа (spaceId берётся из ответа breadcrumbs), +1 для slugId.
  Выход {page:{pageId,title,spaceId}, breadcrumbs:[{pageId,title}] (root→parent;
  [] для корня), children:[{pageId,title,hasChildren}]}.
- Порядок breadcrumbs выверен по серверному источнику (getPageBreadCrumbs: CTE
  сидится на childPageId, идёт вверх, .reverse() → root→page; страница —
  ПОСЛЕДНИЙ элемент). page = chain[last], breadcrumbs = chain.slice(0,-1); корень
  → []. Проекция явная — наружу только pageId (UUID), slugId/icon/position не
  утекают. Пустой chain / 404 / 403 → явная ошибка, не пустой объект.
- children — прямые дети через listSidebarPages (cursor-пагинация #442/#451,
  страница с >20 детьми отдаёт всех без дублей, порядок по position, hasChildren).
- Общий реестр: getPageContext на ОБОИХ хостах через цикл спеков (mcpName===
  inAppKey==="getPageContext", camelCase); DocmostClientLike/DocmostClientMethod
  += it; compile-time contract; TOOL_FAMILY READ + routing-проза.

Тесты: get-page-context.test.mjs +6 (3-й уровень split, no-slugId-leak + 2
запроса, корень→[], slugId-resolve, >20 детей без cap/дублей, плохой id→ошибка,
пустой chain→throw), tool-specs +1. mcp node --test 725/725, tsc чисто; server
jest (contract + ai-chat) 265/265.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 22:19:25 +03:00
agent_coder bfb4c8d8d0 feat(mcp): getTree — иерархия пространства/поддерева одним вызовом (#443, часть 2/3)
Раньше единственный способ получить дерево — listPages tree:true: BFS по
sidebar-эндпоинту, десятки-сотни HTTP-вызовов и молчаливая потеря страниц
(#442). Бэкенд форка уже отдаёт всё одним POST /pages/tree (getSidebarPagesTree,
merged #442/#451), так что это MCP-сторона + общий реестр. Только чтение.

- lib/tree.ts: buildPageTree(nodes) аддитивно расширен до buildPageTree(nodes,
  options?) с {shape?: "lean"|"getTree"; maxDepth?}. Дефолт/{} — байт-идентичный
  lean {id,slugId,title,children?} (существующие вызыватели listPages tree:true
  и subtree-BFS не тронуты, есть регрессионный тест на точный набор полей).
  shape:"getTree" проецирует {pageId, title, children?, hasChildren?} (id→pageId;
  slugId/icon/position/parentPageId не утекают ни на одной глубине).
- client.ts: getTree(spaceId, rootPageId?, maxDepth?) — тот же код-путь, что
  listPages tree:true: один enumerateSpacePages(spaceId, rootPageId) (единичный
  /pages/tree + cursor-BFS фолбэк для stock upstream) → buildPageTree(pages,
  {shape:"getTree", maxDepth}). listPages tree:true помечен DEPRECATED в JSDoc
  (BFS-фолбэк на месте, поведение не тронуто).
- maxDepth/hasChildren: полное дерево строится, потом обрезается. Корни = глубина
  1; maxDepth:1 → только корни; hasChildren:true ТОЛЬКО на срезанном узле с
  серверным hasChildren (source of truth), опущен у листьев и раскрытых узлов.
  Схема тула min(1) отбраковывает 0/отрицательные.
- tool-specs.ts: shared-спек getTree (оба хоста через цикл реестра),
  DocmostClientLike Pick += getTree, listPages desc/catalogLine — про депрекацию.
  server-instructions.ts: getTree:"READ" + routing-проза. Loader
  DocmostClientMethod += getTree, compile-time client-call contract += getTree.

Циклы/self-ref безопасны: project рекурсирует только от корней, циклический
компонент недостижим из корней (нет бесконечной рекурсии). Orphan (родитель
отфильтрован пермишенами) всплывает как корень.

Тесты: tree.test.mjs +9 (nesting+порядок+no-leak, maxDepth:1/2 + hasChildren,
orphan→root, seeded rootPageId, ≤0/нефинитный = без среза, lean-байт-идентичность),
tool-specs.test.mjs (схема getTree + депрекация listPages). mcp node --test
718/718, tsc чисто; server jest (shared-tool-specs.contract + ai-chat) 263/263.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 21:58:32 +03:00
agent_coder f794ac6d6c fix(search): оживить trgm-индекс — убрать coalesce из LIKE-предикатов (#443, ревью)
Ревьюер поймал EXPLAIN'ом: lookup-query фильтровал по
LOWER(f_unaccent(coalesce(col,''))), а GIN-trgm-индексы — на coalesce-free
LOWER(f_unaccent(col)). Postgres берёт функциональный индекс только при точном
совпадении выражения, так что обёртка coalesce отключала индекс → Seq Scan по
pages на КАЖДЫЙ lookup (MCP-клиент всегда шлёт substring:true). Зелёный
int-гейт это не ловил (проверял корректность на крошечном датасете, не
использование индекса).

- search.service.ts: убрал coalesce из двух index-driving LIKE-предикатов
  (title + text_content). Семантически эквивалентно: NULL LIKE '%q%' = NULL
  (falsy), NULL-страница просто не матчится — как пустая строка не матчит %q%.
  SELECT/ORDER BY оставлены с coalesce (индекс не выбирают, Node гардит NULL).
- Миграция: убран избыточный ре-ассерт idx_pages_title_trgm — его создаёт #348
  на том же coalesce-free выражении, теперь title-предикат его использует.
  idx_pages_text_content_trgm без изменений (уже coalesce-free), поправлены
  вводящие в заблуждение комментарии. down() дропает только text-индекс.
- Новый тест search-lookup-explain.int-spec.ts (3): title→idx_pages_title_trgm,
  text→idx_pages_text_content_trgm (оба ассертят отсутствие Seq Scan on pages),
  + негативный контроль (coalesce-обёртка не может использовать индекс) против
  тихой ре-регрессии. Дискриминатор enable_seqscan=off.
- CHANGELOG: две записи (opt-in substring lookup mode + смена shape MCP search).

EXPLAIN на реальном PG: обе fixed-ветки → Bitmap Index Scan on trgm; buggy
coalesce → Seq Scan (Disabled:true). Гейт: server jest src/core/search 27/27
(16 исходных int без изменений + 3 EXPLAIN + 8 unit), mcp node --test 708/708,
tsc чисто.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 21:57:16 +03:00
agent_coder e4e788f151 feat(search): агентский lookup-режим — substring по точкам/дефисам/цифрам, path, snippet, scope (#443, часть 1/3)
MCP-сервером пользуются LLM-агенты; основной паттерн — lookup: найти страницу по
обрывку технической строки и сразу понять, где она лежит и что в ней. Раньше на
такой вопрос уходило 3–5 вызовов. Эта часть закрывает `search` (часть 1 из 3;
get_tree и get_page_context — следующими PR стопкой). Только чтение.

Серверная часть — opt-in, веб-UI байт-в-байт неизменен:
- SearchDTO: опциональные substring/parentPageId/titleOnly (default-off; без
  флагов путь FTS не тронут — guard `if (substring) return searchPageLookup`).
- searchPageLookup: один скан pages, WHERE = title LIKE '%q%' OR (если не
  titleOnly) text LIKE '%q%' OR (если tsquery непустой) tsv @@ ... . LIKE-
  метасимволы %/_/\ экранируются (escapeLikePattern, ESCAPE '\') — `%`/`_` не
  матчат всё; substring-ветка работает даже при пустом tsquery (кейс 10.0.12).
- Ранжирование тирами (TITLE_EXACT > TITLE_SUBSTRING > TEXT), вторичный сигнал
  ts_rank / позиция; score∈(0,1] только для сортировки одной выдачи (формула в
  комментарии). 200-cap упорядочен по SQL-прокси тира ДО среза (иначе Postgres
  отдаёт произвольные 200 и сильный хит мог выпасть). Пермишен-фильтр к
  merged-набору ДО limit. path — одна рекурсивная CTE на все хиты (не N+1).
- snippet оконный в SQL (~500 символов вокруг первого совпадения). Позиция и
  срез в ОДНОМ пространстве LOWER(f_unaccent(...)) — f_unaccent не length-
  preserving (ß→ss, лигатуры, …→...), иначе окно смещалось/пустело. titleOnly →
  пустой snippet. Компромисс задокументирован.
- Миграция: GIN gin_trgm_ops по LOWER(f_unaccent(text_content)); title-trgm
  индекс #348 переиспользован (IF NOT EXISTS), down() дропает только новый.

MCP: схема search (spaceId/parentPageId/titleOnly/limit 1–50, default 10),
client.search прокидывает substring:true, filterSearchResult → {pageId, title,
path, snippet, score}. Инвариант: наружу только pageId (UUID), slugId/id
никогда. Комментарии про намеренное расхождение с in-app hybrid-RRF (не тронут)
и про деградацию на stock-upstream/Typesense (substring→plain FTS, без path/
snippet).

Проверка на реальном Postgres: server integration 16/16 (вся acceptance-таблица
#443 + регрессии на смещение snippet и cap-200), server unit 27/27, mcp
node --test 708/708, tsc чисто.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 21:34:53 +03:00
agent_coder 4be4a75fa3 docs(mcp): точные описания потерь getPage/exportPageMarkdown — конвертер канонический (#415)
Описания двух тулов устарели после #345 (канонический конвертер, #293/#351):
«LOSSY…approximated» у getPage и «lossless» у exportPageMarkdown искажали
роутинг агентов и молчаливо скрывали реальную потерю данных. Четвёртый линк
breaking-окна, стоит на #413.

- tool-specs.ts: getPage — вместо «LOSSY…approximated» точный закрытый список
  потерь (canonical for text; теряет лишь id блоков, resolved-якоря
  комментариев и фиксированный набор ACCEPTED-атрибутов без markdown-
  представления: спаны/colwidth/фон ячеек таблиц, indent, callout.icon,
  orderedList.type, link internal/target/rel/class). exportPageMarkdown —
  убрано «lossless»: round-trip перегенерирует id блоков и молча отбрасывает
  тот же набор (в первую очередь merge-спаны ячеек); держать в page-JSON, если
  нужны. Список выведен из источника истины (ATTR_VALUE_FUZZ_ALLOWLIST +
  MARK_ATTR_ALLOWLIST); opaque carried-verbatim токены (attachmentId, mime,
  slugId и пр.) намеренно НЕ в списке потерь — они round-trip'ятся.
- server-instructions.ts: READ-строка ROUTING_PROSE приведена к тому же
  точному списку, без противоречия markdown-default роутингу #413.
- Исправлена вторая латентная неточность: getPage описывался как
  сохраняющий resolved-якоря — но client.ts передаёт dropResolvedCommentAnchors:
  true, resolved-якоря скрыты (getNode, наоборот, их сохраняет — другой тул).
- README пакета (EN + RU-зеркало) приведены в соответствие passage-for-passage:
  убраны безоговорочные «lossless/lossy» для Markdown round-trip; genuinely-
  lossless ссылки на raw-JSON (getPageJson/getNode) оставлены.

Логику не трогает. server-instructions.test.mjs удалён и заменён
tool-inventory.test.mjs (без пинов на старую формулировку). mcp tsc чисто,
node --test 702/702, tool-inventory 5/5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 20:55:28 +03:00
agent_coder e6171a1810 fix(mcp): сходимость orphan-сноски + дедуп block-id в markdown-splice (#413, ревью)
Правки по внутреннему ревью #413.

1. Orphan-сноска (нарушение «no second canon»). mergeFootnoteDefinitions
   раньше делал ранний return при пустых definitions, пропуская
   canonicalizeFootnotes. Если markdown-фрагмент БЕЗ сносок заменял/удалял
   блок, бывший последним referrer'ом существующей сноски, её определение
   оставалось orphan в хвостовом списке (полный ре-импорт его бы убрал).
   Теперь fast-path (возврат doc по ссылке без clone) только когда
   defs.length===0 И !hasFootnoteArtifacts(doc); иначе клон → append (no-op
   при пустых) → normalizeAndMergeFootnotes → canonicalizeFootnotes, как при
   полном импорте. Новый предикат hasFootnoteArtifacts обходит дерево на
   любой footnotesList/footnoteReference (существующие walk/isObject).
   Идемпотентно: несвязанный plain-патч на странице со сносками не трогает
   их топологию (canonicalizeFootnotes шаг 6 возвращает как есть).

2. Block-id disjoint от страницы. Новый экспорт reassignCollidingBlockIds
   (liveDoc, blocks, skipIndex?) в prosemirror-markdown/node-ops (переиспользует
   collectIds/makeFreshId) + barrel. Зовётся перед сплайсами в client.ts:
   patch — (liveDoc, threaded, 0) (skip 0 = блок, унаследовавший id цели);
   insert — (liveDoc, blocks) без skip. used-аккумулятор ловит и коллизию со
   страницей, и внутрифрагментную. freshBlockId/freshId без изменений.

Тесты (+6, markdown-patch-insert): orphan-repro (0 defs/0 list/0 refs +
docsCanonicallyEqual полному импорту), fast-path (footnote-free patch не трогает
топологию, соседи byte-identical), insert канонизирует при пустых defs со
страничной сноской, уникальность top-level block-id для patch 1→N и insert N.
mcp node --test 702/702, pmd vitest 736, tsc чисто.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 20:38:06 +03:00
agent_coder 2f23cf4b65 fix(mcp): pre-flight size-guard в diffDocs против CPU-DoS на больших правках (#464)
Боевой инцидент: diffDocs синхронно зовёт recreateTransform (rfc6902), чей
array-diff — O(n·m) по числу нод и O(w²) по длине прогонов слов, и который
НИКОГДА не бросает. Поэтому существующий catch→coarseDiff не спасает event loop:
большая агентская правка блокирует воркер на секунды→минуты (бенч: 401 нода→1.3с,
801→5.5с, 1601→OOM; отдельная ось — байт-тяжёлый вход: 309КБ→OOM), а это душит
все остальные запросы воркера. Отсюда живой прод-хэнг.

Фикс — pre-flight size-guard ПЕРЕД дорогим вызовом:
- readPositiveIntEnv(name, dflt): парс env каждый вызов; мусор/пусто/≤0/не-finite →
  дефолт, так что guard нельзя случайно отключить.
- exceedsDiffSizeGuard(old, new): срабатывает по max(old,new) на ОБЕИХ осях —
  countNodes и JSON.stringify().length — так что асимметричная пара (крошечный
  new / огромный old и наоборот) тоже триггерит. Дёшево: один обход + один
  stringify на документ.
- Дефолты MCP_DIFF_MAX_NODES=150, MCP_DIFF_MAX_BYTES=12288 (12 КиБ),
  env-переопределяемы. Подобраны бенчмарком под бюджет ~200мс на ЛЮБОЙ форме
  входа на границе cap'а (исходные догадки тикета 3000-5000 нод / 512КБ-1МБ были
  в 20-30× завышены — пропускали бы многоминутные блоки).
- Рефактор: выделены coarseDiffTally (единый источник формы fellBack:true) и
  preciseDiffTally. Новый поток diffDocs: computeIntegrity (без изменений, для
  обоих путей) → если exceedsDiffSizeGuard → coarseDiffTally(fellBack=true) →
  иначе try preciseDiffTally / catch → coarseDiffTally. Обе деградации дают
  идентичную форму результата.

Fail-closed: единственный путь к recreateTransform — ветка else, достижимая
только когда guard НЕ сработал; огромный документ физически до transform не
доходит. computeIntegrity (корректностный сигнал) считается ВСЕГДА. Все три
call-site потребляют DiffResult как информационный preview, не гейтят запись —
coarse-фоллбэк контракт-сохраняющий.

Тесты (11): over/under-порог, обе асимметрии, байт-ось (нод-мало/байт-много),
env-override низким cap'ом, garbage-env→дефолт (guard не отключается), integrity
корректен на tripped-документе; behavioral-proxy что transform пропущен над
cap'ом (guarded ~5мс vs caps-raised ~3.3с, ≥5× + <200мс бюджет). node --test
688/688. tsc --noEmit чисто. Только packages/mcp, внешний симлинк не тронут.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 20:32:50 +03:00
agent_coder d269bd9efe feat(mcp): markdown — формат по умолчанию для блочных getNode/patchNode/insertNode (#413)
Канонический конвертер (#293/#345/#351) зрел для блочного уровня: markdown
становится дефолтом чтения/записи ОДНОГО блока, PM JSON — опция «для тонких
работ». Новых тулов нет, поверхность не растёт. Имена уже camelCase (стоит на #412).

- getNode(pageId, nodeId, format='markdown'): дефолт markdown — обёртка
  {type:doc,content:[node]} → convertProseMirrorToMarkdown, comment-якоря
  (ВКЛЮЧАЯ resolved) сохранены (это чтение под редактирование, не getPage).
  format:'json' — сырой сабтри. Авто-фолбэк не-топ-левел типов (tableRow/Cell/
  Header по #<index>) в JSON через canBeDocChild = docmostSchema.nodes.doc.
  contentMatch.matchType (не рукописный список); поле format на каждом ответе.
- patchNode/insertNode: XOR-вход {markdown?|node?} (оба optional в схеме, XOR
  на рантайме). markdown → импорт фрагмента → 1→N сплайс: первый блок наследует
  id цели, остальные свежие; dry replaceNodeById для #159-ambiguity ДО сплайса;
  соседние блоки byte-identical. Guard findUnrepresentableTableAttrs: цель со
  span/colwidth/backgroundColor → отказ с указанием на table-тулы/node-JSON.
  insertNode — insertNodesRelative (N блоков по порядку); голый tableRow/Cell
  JSON-only.
- Сноски: ^[...] во фрагменте → канон-импортёр; importMarkdownFragment делит
  блоки от footnotesList, РЕМАПИТ id сносок фрагмента в свежие uuid (fn-1
  фрагмента не коллизит с fn-1 страницы), mergeFootnoteDefinitions добавляет
  через ту же машинерию appendDefinition→normalizeAndMergeFootnotes→
  canonicalizeFootnotes, что insertFootnote. Сырые JSON-пути не тронуты.
- node-ops: replaceNodeByIdWithMany/insertNodesRelative (сплайс массива) в
  prosemirror-markdown/node-ops (канон после #414) + barrel.
- ROUTING_PROSE (READ/EDIT), compile-time client-call contract, CHANGELOG
  (getNode default→markdown, breaking для внешних клиентов, в окне #411/#412).

Тесты: сходимость (patchNode(markdown) блок docsCanonicallyEqual полному
импорту — нет «второго канона»); id-нить 1→N (первый наследует, остальные
свежие, соседи byte-identical); XOR; getNode markdown/json/non-top-level-fallback;
сохранение comment-якорей (active+resolved); ^[...]→хвостовой список+перенумерация;
guard. mcp node --test 697/697; pmd vitest 736; tsc чисто; server jest 273.
Третий линк breaking-окна, стоит на #412 (#411→#412→ЭТОТ→#415).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 20:10:38 +03:00
agent_vscode d287c15db4 fix(ci,ai-chat): восстановить зелёный CI на develop
Чинит два падения прогона Develop (#29104398390):

A. Интеграционные тесты сервера (10 failed) падали с
   `TypeError: this.environment.isAiChatFinalStepLockdownEnabled is not a function`.
   Мерж #444 добавил вызов этого метода в AiChatService.stream, но два
   интеграционных фикстура не обновили env-стаб. Добавлен
   `isAiChatFinalStepLockdownEnabled: () => false` в 4 стаба
   (ai-chat-attach.int-spec.ts, ai-chat-stream.int-spec.ts).

B. Job e2e-server не компилировал app.e2e-spec.ts:
   `TS2307: Cannot find module '@docmost/mcp'`. Рефактор #446 добавил
   тип-импорт из @docmost/mcp в docmost-client.loader.ts, но job не
   собирал этот пакет (его build/ в gitignore). Добавлен шаг
   `Build mcp` в e2e-server (по образцу e2e-mcp / mcp-server-parity).

Только тесты и CI-конфиг; продакшн-логика не менялась.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 19:42:16 +03:00
agent_coder 7cb3199d09 refactor(mcp)!: BREAKING — все имена MCP-тулов snake_case → camelCase (унификация с in-app) (#412)
Один логический тул жил под двумя именами: внешний MCP snake_case
(edit_page_text), in-app camelCase (editPageText) — дублирование доков, путаница
при переносе промптов/скиллов, помеха шарингу спек (#294). Решение владельца:
единый camelCase везде, включая внешний MCP. После этого mcpName === inAppKey.

- tool-specs.ts: mcpName ВЫВЕДЕН из ключа спеки (mcpName == inAppKey) для всех
  43 shared-спек — раньше divergent snake, теперь равен ключу (проверено: mcpName
  читается только структурно — цикл регистрации, генератор <tool_inventory>,
  TOOL_FAMILY). +5 inline-регистраций (tableGet/updateComment/deleteComment/
  docmostTransform; search без изменений). Рантайм: 47 тулов, все camelCase, ноль
  подчёркиваний.
- Контракт-конвенция ИНВЕРТИРОВАНА: shared-tool-specs.contract.spec
  `mcpName === toSnake(inAppKey)` → `mcpName === inAppKey`; tool-specs.test
  и tool-inventory.test обновлены.
- ROUTING_PROSE/TOOL_FAMILY/INLINE_MCP_INVENTORY (server-instructions.ts) →
  camelCase (105 замен). ai-chat.prompt/guard уже на in-app camelCase-ключах —
  без изменений (guard прошёл). comment-signal EXCLUDED_TOOLS схлопнут с
  дублей snake+camel до camelCase.
- Некоторое неочевидное: assertUnambiguousMatch(op: "patch_node"|"delete_node")
  в prosemirror-markdown/node-ops — op интерполируется в model-facing ошибку;
  литерал-юнион + call-sites → "patchNode"|"deleteNode".
- Все snake-имена в описаниях/error-строках/комментах/тестах/доках → camelCase
  (whole-token, longest-match-first). CHANGELOG: BREAKING-таблица 46 строк +
  миграция (allowlists mcp__gitmost-*__get_node→__getNode, промпты/скиллы,
  .mcp.json, метрики по tool-label); релизится вместе с #411.
  Внутренние имена методов (PageService.updatePageContent и т.п.) НЕ тронуты —
  переименованы только ИМЕНА ТУЛОВ.

Гейт: mcp node --test 677/677; tsc -p apps/server чисто; jest ai-chat-tools.
service + shared-tool-specs.contract + tool-tiers + ai-chat.prompt +
comment-signal-inapp → 323. Второй линк breaking-окна (#411→ЭТОТ→#413→#415).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 19:10:38 +03:00
vvzvlad e19275e96e Merge pull request 'refactor(tools): updatePageContent → updatePageMarkdown; −import_page_markdown с MCP (#411)' (#462) from refactor/411-update-page-markdown into develop
Reviewed-on: #462
2026-07-10 18:36:44 +03:00
agent_coder 3a521ada4d refactor(tools): updatePageContent → updatePageMarkdown; внешний MCP: +updatePageMarkdown, −import_page_markdown (#411)
Поверхности записи «целым телом» были несимметричны: у in-app агента полная
замена тела markdown называлась updatePageContent (имя не про формат, тогда как
парный updatePageJson — про JSON), а у внешнего MCP голого plain-body-replace
не было вовсе (только import_page_markdown — на деле парсер round-trip к
export_page_markdown, не plain-replace). Пара должна быть updatePageMarkdown /
updatePageJson.

Пост-Фаза-1б архитектура (реестр + циклы по обоим хостам):
- новая shared-спека updatePageMarkdown (mcpName update_page_markdown, inAppKey
  updatePageMarkdown, tier как у updatePageJson) с execute (client, {pageId,
  content, title}) => client.updatePage(...) — тот же путь updatePageContentRealtime
  → markdownToProseMirrorCanonical, ^[...]-сноски парсятся. Реестровый цикл
  регистрирует её на ОБОИХ хостах автоматически. Добавлен 'updatePage' в
  Pick DocmostClientLike.
- import_page_markdown убран с внешнего MCP через inAppOnly:true у спеки
  importPageMarkdown — MCP-цикл и генератор инвентаря её пропускают, in-app
  агент сохраняет importPageMarkdown; спека и client-метод НЕ удалены.
- удалён inline in-app updatePageContent tool (теперь из реестра под inAppKey
  updatePageMarkdown) + его INLINE_TOOL_TIERS-энтри.
- ROUTING_PROSE: bulk-rewrite ссылается на update_page_markdown|update_page_json;
  убрано упоминание import_page_markdown; инвентарь генерируется из catalogLine.
- лейбл-мапы chat-markdown.util (en/ru), человекочитаемые метки не тронуты.
- НЕ тронуты одноимённые внутренности: PageService.updatePageContent,
  updatePageContentRealtime, collaboration.handler — переименовано только имя тула.

Тесты: updatePageMarkdown на обеих поверхностях с идентичной схемой, forward в
client.updatePage; import_page_markdown ОТСУТСТВУЕТ на MCP, присутствует in-app;
^[...]→сноски покрыт через collaboration.test. CHANGELOG BREAKING + миграция;
README/README.ru пакета обновлены. Гейт: mcp node --test 646/646, server jest
259, tsc чисто. Первый линк breaking-окна #416 (#411→#412→#413→#415).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:33:20 +03:00
vvzvlad 576db3c8f9 Merge pull request 'feat(mcp): drawio стадия 2 — guide, каталог фигур, ELK-лейаут, quality-warnings (#424)' (#440) from feat/424-drawio-rules into develop
Reviewed-on: #440
2026-07-10 18:29:46 +03:00
agent_coder ddb37376a4 refactor(mcp): вписать drawio-тулы в реестровый цикл (Фаза 1б смержена) (#440)
Фаза 1б (#445/#446/#447/#448) влилась ПЕРЕД этим PR, поэтому явная проводка
drawio через registerShared/sharedTool конфликтовала с реестровыми циклами.
Фолд:
- drawioGet/Create/Update → канонический execute в спеке (это client-методы):
  execute возвращает сырой результат, цикл оборачивает jsonContent на MCP и
  отдаёт как есть in-app — байт-в-байт как старые тела, overrides не нужны.
  layout сохранён: добавлен в buildShape create/update + 5-м аргументом
  (layout as 'elk'|undefined) в обоих execute.
- drawioShapes/drawioGuide → остаются inline на ОБОИХ хостах. Гайд архитектора
  «execute в спеке с импортом searchShapes/getGuideSection в tool-specs.ts»
  оказался невозможен: drawio-shapes.ts использует import.meta.url, а
  tool-specs.ts тайпчекается из исходника под module:commonjs (contract-спека)
  → TS1343 на статический value-import, а import.meta не индиректится. Введён
  флаг спеки inlineBothHosts: спеки остаются в SHARED_TOOL_SPECS (contract
  пинит имя/описание/схему), но без execute; ОБА цикла их пропускают (добавлен
  симметричный guard в MCP-цикл), каждый хост регистрирует их inline через
  чистые хелперы — поведение байт-в-байт как до ребейза.
- docmost-client.loader: взята develop-форма DocmostClientLike = Pick<
  DocmostClient> (#446); ручное зеркало убрано, паритет layout наследуется из
  реальной сигнатуры client.
- ROUTING_PROSE: drawio intent-подсказки (shapes-first, guide, layout:elk);
  инвентарные строки убраны (генерируются из catalogLine).

Гейт: mcp node --test 674/674; server jest ai-chat-tools.service + contract
(211, все 5 drawio на обоих хостах, идентичная схема) + tool-tiers → 264;
tsc чисто; layout-passthrough тест 3/3. ELK DoS-кап и layout-фикс из round 3/4
сохранены.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:34:54 +03:00
agent_coder dc58974b31 fix(ai-chat): пробросить layout:elk в in-app drawioCreate/drawioUpdate — паритет с MCP (ревью #440)
In-app хендлеры drawio_create/drawio_update деструктурировали args БЕЗ layout и
не передавали его клиенту 5-м аргументом → layout:"elk" (схема его принимает —
общий buildShape) ТИХО терялся, ELK-автолейаут работал только по MCP-хосту.
Корень: ручное зеркало DocmostClientLike (loader) отстало от реального client.ts
— у его drawioCreate/drawioUpdate не было параметра layout (то, что #446 чинит
деривацией типа, но #446 ещё не влит). Добавил layout?:'elk' в обе сигнатуры
зеркала + проброс в обоих хендлерах.

Тест (пропущенный зелёным гейтом пробел — не было теста на in-app passthrough):
in-app drawioCreate/drawioUpdate с layout:'elk' → фейк-клиент получает layout
5-м позиционным аргументом; omit-кейс → undefined. Мутационно: убрать проброс
в drawioCreate → layout-create-тест краснеет.

Гейт: mcp build чисто; tsc -p apps/server без новых ошибок; jest
ai-chat-tools.service (35) + shared-tool-specs.contract + tool-tiers +
comment-signal-inapp → 273 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:16:19 +03:00
agent_coder c917dcc3c1 fix(mcp): ограничить ELK-лейаут (кап узлов/рёбер + таймаут) — untrusted-граф DoS (ревью #440)
applyElkLayout крутит elkjs СИНХРОННО в процессе на mxGraph-XML от LLM
(layout:'elk' в drawio_create/update) без лимита размера и без таймаута —
большой граф (тысячи узлов, ~1МБ XML проходит stage-1 cap 16МБ) блокирует
event-loop MCP-сервера на секунды-минуты. try/catch ловил только брошенную
ошибку, но не зависание.

- кап ДО построения графа: >500 узлов или >1000 рёбер → вернуть исходный XML
  (best-effort, как существующий catch); синхронный elkjs → кап и есть
  реальная защита;
- Promise.race с 5s-таймаутом (defense-in-depth на случай async-elkjs); таймер
  гасится в finally → нет утечки хендла и unhandled-rejection (проигравший
  timeout остаётся pending с погашенным таймером);
- тест: 600-узловой граф возвращается без изменений и быстро (<2s) — кап-путь.

79/79 drawio-тестов зелёные.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:15:06 +03:00
agent_coder eddc3b5c33 feat(ai-chat): пробросить drawio_shapes/drawio_guide in-app — восстановить SHARED_TOOL_SPECS-паритет (#424)
Стадия-1 (#434) уже была довяжена in-app в develop (f46d89ea, agent_vscode)
для CRUD-тулов; два новых чистых read-only хелпера стадии-2 остались
незаброшенными → contract-parity спека падала 6 ассертами (по 3 на
drawio_shapes/drawio_guide). В отличие от CRUD-тулов это ЧИСТЫЕ функции без
сетевого вызова, поэтому НЕ client-методы:

- реэкспорт searchShapes / getGuideSection (+ тип SearchShapesOptions) из
  entry пакета @docmost/mcp; loadDocmostMcp() пробрасывает их так же, как
  sharedToolSpecs (типы SearchShapesFn/GetGuideSectionFn);
- две записи sharedTool(...) в forUser() после drawioUpdate: drawioShapes
  повторяет серверный вызов searchShapes(query,{category,limit}) и форму
  { query, count, results }; drawioGuide — getGuideSection(section)
  (omit section -> index); голый объект без jsonContent-envelope, как у
  соседних in-app хендлеров;
- DocmostClientLike и HOST_CONTRACT_METHODS-вайтлист НЕ тронуты (это не
  методы клиента);
- три тест-мока (contract/service/tool-tiers) получили type-only no-op
  заглушки под расширенный тип loadDocmostMcp() — тела инструментов в этих
  тестах не исполняются, contract-спека реально гоняет настоящий
  SHARED_TOOL_SPECS.

Внутреннее ревью обвязки: APPROVE, 0 находок. shared-tool-specs.contract:
211/211 (было 6 падений); client-host-contract drift-guard 3/0; tsc EXIT 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:15:06 +03:00
agent_coder e454fe189c feat(mcp): drawio стадия 2 — правила качества, каталог фигур, guide, ELK-лейаут, warnings (#424)
Надстройка над стадией-1 (сырой mxGraph XML) — помогает агенту рисовать
корректные диаграммы без бэкенд-рендеринга:

- hard-rules в описаниях drawio_create/drawio_update (геометрия, parent-
  relative координаты, стили);
- drawio_guide (5 секций: skeleton / layout / containers / icons-aws /
  icons-azure, каждая ≤4KB) — по требованию, не раздувает контекст;
- drawio_shapes — реальный jgraph shape-index (10446 фигур, gzip 437KB,
  ленивый node:zlib gunzip) + курируемый оверлей (service-level паттерны
  AWS/Azure, note-подсказки на пустые resIcon, палитра категорий);
  ранжирование aws4>aws3; escapeRe в score (не ReDoS);
- layout:"elk" через elkjs (чистый JS, dependencies:{}) — compound-nesting,
  best-effort (на сбое ELK возвращает нормализованный вход), 73→0 warnings
  на 12-узловом графе;
- 6 типов quality-warnings в линтере (overlap, out-of-bounds, edge-cross,
  и т.п.), геометрия Liang-Barsky; warnings НИКОГДА не блокируют write.

Оба новых инструмента в SHARED_TOOL_SPECS (tier:deferred) + SERVER_
INSTRUCTIONS; drift-guards зелёные. elkjs ^0.11.1 — единственный новый
рантайм-деп; lockfile синхронизирован (--frozen-lockfile --offline EXIT 0).
data/ едет с воркспейсом (.gitignore-негация !packages/mcp/data/).

Внутренний цикл: 1 проход внутреннего ревью (APPROVE WITH SUGGESTIONS);
все 59 профильных тестов зелёные.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:12:37 +03:00
vvzvlad ea99d4fe63 Merge pull request 'feat(mcp): внятная диагностика ошибок тулов + fail-fast валидация comment-id (#437, #436)' (#441) from feat/437-error-diagnostics into develop
Reviewed-on: #441
2026-07-10 16:03:44 +03:00
vvzvlad 23966ce51c Merge pull request 'fix(ai-chat): «send now» во время detached-run — серверный stop + ограниченный ретрай 409 (#396)' (#456) from fix/396-sendnow-detached-run into develop
Reviewed-on: #456
2026-07-10 16:03:18 +03:00
vvzvlad a53b2f454e Merge pull request 'fix(delivery): immutable-кэш ассетов — отключить дефолтный cacheControl @fastify/static (#452)' (#455) from fix/452-immutable-cache-control into develop
Reviewed-on: #455
2026-07-10 16:03:03 +03:00
vvzvlad d219eb7525 Merge pull request 'fix(ai-chat): защита от петель агента — lockdown под тоггл + детектор деградации + бюджет шагов (#444)' (#454) from fix/444-agent-loop-guards into develop
Reviewed-on: #454
2026-07-10 16:02:48 +03:00
vvzvlad 93d244478e Merge pull request 'refactor(tools): генерировать инвентарь SERVER_INSTRUCTIONS из реестра + guard имён тулов (#448)' (#460) from refactor/448-generate-inventory into develop
Reviewed-on: #460
2026-07-10 16:02:17 +03:00
vvzvlad 791f709c18 Merge pull request 'refactor(tools): execute-маппинг в SHARED_TOOL_SPECS + автопроводка обоих хостов (#445)' (#459) from refactor/445-execute-mapping into develop
Reviewed-on: #459
2026-07-10 16:02:01 +03:00
vvzvlad f8a27cba91 Merge pull request 'refactor(mcp): вывести DocmostClientLike/SharedToolSpec из реального типа — убить ручные зеркала (#446)' (#458) from refactor/446-derive-client-types into develop
Reviewed-on: #458
2026-07-10 16:01:48 +03:00
vvzvlad 61dc9b50c1 Merge pull request 'fix(ci,mcp): REGISTRY_STAMP в билде + кросс-пакетный CI — закрыть skew build/vs/src (#447)' (#457) from fix/447-registry-stamp-ci into develop
Reviewed-on: #457
2026-07-10 16:01:38 +03:00
vvzvlad cebb1cca87 Merge pull request 'fix(mcp): pre-validate node JSON против схемы + путь битого узла (#409)' (#461) from fix/409-invalid-node-validation into develop
Reviewed-on: #461
2026-07-10 16:01:25 +03:00
vvzvlad 605c0f3dda Merge pull request 'docs(mcp): поправить устаревшую ссылку footnote-authoring.ts в комментариях' (#453) from docs/footnote-authoring-comment-cleanup into develop
Reviewed-on: #453
2026-07-10 15:49:35 +03:00
agent_coder 0f5f048ca2 fix(mcp): pre-validate node JSON против схемы + путь битого узла (#409, остаток Фазы 1)
Структурные редакторы (patchNode/insertNode/updatePageJson/transformPage) кидали
опаковый Yjs-крах на агентском JSON с вложенным узлом без/с неизвестным `type`:
«Failed to encode document to Yjs (fromJSON): Unknown node type: undefined» —
ГЛУБОКО в энкодере, уже ПОСЛЕ открытия collab-сессии, а хинт мислейблил это как
проблему атрибута. Агент ретраил вслепую (~34 краха в истории 06-17…07-07).

- findInvalidNode(doc) в prosemirror-markdown/node-ops.ts: DFS по content,
  возвращает {path, summary} первого узла с отсутствующим/не-строковым `type`
  или типом/маркой вне схемы. Множество имён — из getSchema(docmostExtensions),
  ТОГО ЖЕ, из которого энкод-путь строит docmostSchema → «известный тип»
  обходчика ровно то, что примет PMNode.fromJSON/toYdoc (сверено на 45 узлах +
  12 марках, ни ложных положительных, ни пропуска краш-типа).
- unstorableYjsError: findInvalidNode ПЕРВЫМ (node-shape крах больше не
  мислейблится как атрибут), затем findUnstorableAttr, generic-фраза последней.
- assertValidNodeShape(op, node) ДО getCollabTokenWithReauth/mutatePageContent
  в patchNode/insertNode/updatePageJson: fail-fast — collab-сессия не
  открывается, page-lock не берётся, сообщение детерминировано (mock-тест
  ассертит collabTokenFetched===false на битом пути). tableUpdateCell не тронут
  (строит абзац из plain text через makeCellParagraph, агентский JSON не глотает).
- Описания patch_node/insert_node/update_page_json: каждый узел, включая
  вложенные, несёт строковый `type` из схемы; текст-листы {"type":"text",...}.

sanitizeForYjs (стрип undefined-атрибутов) сохранён — другой класс отказа.
Внутреннее ревью: APPROVE WITH SUGGESTIONS — schema-fidelity/fail-fast/
no-false-positive/precedence подтверждены; замечания необязательны (тест
перечисления схемы, depth-guard безобиден т.к. энкодер падает раньше).
prosemirror-markdown vitest 726/726, mcp node --test 613/613.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 10:49:51 +03:00
agent_coder 4cb762b039 docs(mcp): обновить AGENTS.md + коммент под генерируемый инвентарь (ревью #460)
Две доковые правки по ревью: (1) AGENTS.md-буллет описывал ДО-#448 мир (ручная
правка SERVER_INSTRUCTIONS, enforced server-instructions.test.mjs, EXCEPTIONS) —
переписан: shared-спеки авто-обновляют генерируемый <tool_inventory>, только
inline-тул требует строки в INLINE_MCP_INVENTORY, enforced tool-inventory.test.mjs,
EXCEPTIONS больше нет; (2) коммент в server-instructions.ts называл гард окольно
('tool-specs.test.mjs's sibling test') → назван tool-inventory.test.mjs напрямую.
Единственная оставшаяся ссылка на удалённый тест устранена. Только доки/комменты.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 09:55:30 +03:00
agent_coder d0f99052cf refactor(tools): генерировать инвентарь SERVER_INSTRUCTIONS из реестра + guard имён тулов в промпте (#448)
Финальный линк Фазы 1б. Инвентарь тулов жил в 4 рукописных прозаических копиях
(SERVER_INSTRUCTIONS под regex-тестом; <tool_catalog>; имена в ai-chat.prompt.ts
без гарда; README) — роадмап #416 планировал 4 последовательных ручных правки
этого текста (#411/#412/#413/#415).

- SERVER_INSTRUCTIONS разбит (новый модуль server-instructions.ts): ROUTING_
  PROSE (рукописные intent-подсказки «когда что» — осмысленно ручные, перенесены
  ДОСЛОВНО со всеми предостережениями: <=250 у create_comment, soft-delete у
  delete_page, baseHash у drawio_update, PUBLIC у share_page) + buildToolInventory()
  — генерирует <tool_inventory> из реестра (mcpName + purpose из catalogLine,
  группировка по TOOL_FAMILY, бакет OTHER ловит незамаппленное → тул нельзя
  тихо потерять) + 5 inline MCP-only (INLINE_MCP_INVENTORY). Детерминирован
  (семейства FAMILY_ORDER, имена localeCompare). regex-тест server-instructions
  удалён; структурные гарантии — в новом tool-inventory.test.mjs (точное
  членство множества сильнее старого \b-скрейпа).
- Имена тулов в ai-chat.prompt.ts → через экспорт PROMPT_TOOL_NAMES; новый гард
  ai-chat.prompt.tool-names.spec.ts: каждое имя — реальный тул реестра, скан
  guidance-нот на camelCase-токены падает на несуществующем (escape-
  нейтрализация против ложных nThe-токенов).
- INLINE_TOOL_TIERS уже содержал ровно 8 genuinely-inline тулов (после #445) —
  сжатие не потребовалось.

Критерий: добавление/переименование спека меняет инвентарь БЕЗ правки прозы.
Внутреннее ревью: APPROVE — фактическим прогоном подтверждено, что НИ ОДИН тул
из старого SERVER_INSTRUCTIONS не выпал (диф старый-vs-новый пуст; добавился
get_workspace, раньше прятавшийся в EXCEPTIONS); проза дословна; инвентарь
полон/детерминирован/без фантомов; гард краснеет на обеих ветках провала.
613 node + 289 jest зелёные. Стоит на #445 — мержить последним в стопке 1б.

README-каталоги вне обязательного скоупа (docs-скрипт) — в чек-лист #412.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 09:21:28 +03:00
agent_coder 8c74659d91 refactor(tools): execute-маппинг в SHARED_TOOL_SPECS + автопроводка обоих хостов (#445)
Ядро Фазы 1б. Реестр (#294) шарил только метаданные (имя/схема/описание/tier),
но НЕ execute-логику — у каждого shared-тула было ДВА рукописных execute-тела
с копией маппинга аргументов (MCP registerShared в index.ts; in-app sharedTool
в ai-chat-tools.service, зеркалящий MCP-транспорт вручную). Корень
повторяющихся parity-багов (f46d89ea drawio, f8d26420 stashPage, fc9088b7
node-args): добавление одного тула = 7-9 согласованных ручных правок в двух
пакетах.

- SharedToolSpec расширен: канонический execute(client, args) (чистый JS —
  свободно пересекает zod-мажорную границу v3/v4) + оверрайды
  mcpExecute/inAppExecute/mcpOnly/inAppOnly для ОСОЗНАННЫХ per-layer различий.
  client: DocmostClientLike (Pick из #446). Канон возвращает СЫРЬЁ, каждый хост
  накладывает свой конверт (MCP jsonContent, in-app как есть); override владеет
  результатом хоста целиком.
- Оба хоста → циклы по Object.values(SHARED_TOOL_SPECS): index.ts registerShared
  39→0 (цикл), ai-chat-tools.service sharedTool ~40→1 (цикл). Добавление спека
  автоматически регистрирует тул в ОБОИХ хостах — сценарий PR #434 невозможен
  по построению.
- Осознанные различия через overrides (ни одно не сплющено к одному хосту):
  оба mcpExecute+inAppExecute — createPage/movePage/deletePage/
  exportPageMarkdown/createComment (guardrails, конверты, проекции, тексты
  ошибок); execute+inAppExecute — getPage/renamePage/resolveComment;
  execute+mcpExecute — stashPage (resource_link+structuredContent),
  checkNewComments (since-guard только на MCP).
- Оставлены inline (по делу): update_comment/delete_comment (MCP-only, in-app
  не даёт хард-правку/удаление комментов), search/transformPage (per-transport
  дивергенция — hybrid RRF / без deleteComments), table_get (noun-vs-verb
  naming clash — уедет после camelCase #412), getCurrentPage/updatePageContent/
  listSidebarPages/getComment/getPageHistory (in-app-only, per-request state).
- Guard-тесты (contract-parity, phantom-catalog) сохранены — теперь инварианты,
  не «последняя линия».

Внутреннее ревью: APPROVE, построчная BEFORE/AFTER-сверка по каждому shared-тулу
на обоих хостах — ни одного изменённого per-host поведения (порядок/дефолты
аргументов, guard'ы, конверты, проекции сохранены), множества тулов побайтово
совпадают (48 in-app, 45 MCP), кросс-zod-граница чистая (нет z. в execute),
611 mcp + 260 server тестов зелёные. Ядро Фазы 1б, стоит на #446 — мержить после.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 08:59:40 +03:00
agent_coder fe5b6ecd8c refactor(mcp): вывести DocmostClientLike/SharedToolSpec из реального типа клиента — убить ручные зеркала (#446)
Восстановленный отложенный долг #294: @docmost/mcp не отдавал .d.ts, поэтому в
сервере жили ТРИ дрейфующие ручные копии одних и тех же имён/сигнатур
(DocmostClientLike ~230 строк, копия SharedToolSpec, name-only HOST_CONTRACT_
METHODS-тест). In-app execute-тела зовут клиент ПОЗИЦИОННО, так что перестановка
параметра в client.ts доезжала до прода рантайм-ошибкой без сигнала на компиляции.

- declaration:true (+declarationMap) в packages/mcp/tsconfig.json; types-экспорт
  в package.json (exports → conditional {types, default} для . и ./http;
  require.resolve/dynamic-import резолвят default → build/index.js, рантайм не
  тронут). build/index.d.ts эмитится, реэкспортит DocmostClient + SharedToolSpec.
  Правок исходников пакета для эмита НЕ потребовалось.
- DocmostClientLike → Pick<DocmostClient, 48 методов> из type-only import
  (стёрт на компиляции, ESM/CJS-границу не задевает); ручное зеркало удалено.
- SharedToolSpec → type-only реэкспорт из пакета; ручная копия удалена.
- client-host-contract.test.mjs удалён целиком — имена И сигнатуры теперь
  проверяет tsc.
- Позиционная безопасность: never-called __assertClientCallContract(client:
  DocmostClientLike) воспроизводит каждый позиционный вызов с типизированными
  плейсхолдерами (AI-SDK стирает вход execute-замыканий в any, иначе позиционные
  вызовы не проверялись). Перестановка параметров client.ts → ошибка компиляции
  сервера ровно тут. Loose as-касты в ai-chat-tools.service не потребовали
  правок; as any не добавлялся.

Внутреннее ревью: APPROVE. Runtime resolution через conditional exports не сломан
(разобрано для прод-инсталляции, не только symlink); покрытие
__assertClientCallContract полное (48 call-sites == union == assert, сверено
программно); Pick полон; демонстрация reorder → TS2345 в assert. Единственная
находка (Promise<any> в части возвратов) предсуществующая в client.ts, вне
цели PR. Стоит на #447 (закрытие skew build/vs/src) — мержить после него.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 08:29:46 +03:00
agent_coder 2e6f1c3de5 fix(ci,mcp): REGISTRY_STAMP в билд-артефакте + кросс-пакетный CI — закрыть skew build/ vs src/ (#447)
Два структурных слепых пятна: (1) сервер грузит СКОМПИЛИРОВАННЫЙ
packages/mcp/build/index.js, а серверные guard-тесты читают src/tool-specs.ts —
правка src без пересборки оставляет тесты зелёными, но рантайм расходится со
спеками; (2) спеки добавляют в packages/mcp, а parity-тесты живут в jest-сьюте
apps/server — PR, трогающий только пакет, проходит зелёным, сломанная in-app-
проводка всплывает уже на develop (кейс f46d89ea).

- REGISTRY_STAMP: codegen (scripts/gen-registry-stamp.mjs) считает sha256 от
  нормализованного (CRLF→LF, один хвостовой \n снят) сырого текста
  src/tool-specs.ts, пишет src/registry-stamp.generated.ts (gitignored),
  index.ts реэкспортит → попадает в build/. Вшит в build/pretest/watch ДО tsc.
- Loader (dev/test): computeSrcRegistryStamp пересчитывает стамп из src рядом с
  build/index.js (dev-vs-prod по existsSync, любая ошибка → null), сверяет с
  build-стампом → при рассинхроне бросает «build is stale — rebuild». В prod
  (src нет) и на pre-#447 билдах (нет REGISTRY_STAMP) — чистый no-op.
- CI: job mcp-server-parity собирает shared-deps+mcp (регенерит стамп) и гоняет
  ОБА сьюта вместе (mcp node:test + server guard-спеки) — именованный гейт, его
  нельзя случайно расщепить.
- AGENTS.md: правка спеков требует ребилда @docmost/mcp.

Тесты (20): mcp-сайд (детерминизм, нормализация, desync-гард стамп-vs-билд) +
server-сайд (null при отсутствии src = prod no-op; mismatch → throw точного
сообщения; pre-#447 no-op). Кросс-импл equality-гард: один фиксированный вход →
один хэш на ОБЕИХ сторонах, ловит рассинхрон двух нормализаций. Внутреннее
ревью: APPROVE WITH SUGGESTIONS (обе — покрытие guard'а — закрыты этим тестом).
Мутационно: любой из двух normalize-имплов расходится → equality-тест краснеет.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:58:37 +03:00
agent_coder 3cba551800 fix(ai-chat): «send now» во время detached-run — авторитетный серверный stop + ограниченный ретрай 409 (#396)
В автономном режиме «Interrupt and send now» во время живого detached-run делал
только локальный stop() (abort SSE), который сервер игнорирует (run живёт по
дизайну #184/#234), поэтому onFinish→flush новый POST упирался в гейт «один
активный run на чат» → 409 A_RUN_ALREADY_ACTIVE, новый turn не стартовал.
handleStop делает правильно (доп. onServerStop), sendNow — нет. Вариант A
(клиентский, горячий путь сервера не тронут):

- sendNow в автономном режиме дополнительно зовёт onServerStop(chatId) (или
  откладывает через stopPendingRef, если chatId ещё не усыновлён — как
  handleStop) и взводит one-shot supersedeRetryRef ДО stop();
- транспорт-fetch на supersede-отправке (и только на ней) ретраит РОВНО 409 с
  body.code===A_RUN_ALREADY_ACTIVE до 4 попыток с бэкоффом 150/300/600ms;
  onServerStop гарантирует осадку run → ретрай сходится. Обычная отправка (не
  взведён флаг) падает на 409 мгновенно. isRunAlreadyActive читает
  response.clone() → тело оригинала возвращается потребителю нетронутым.
  409 всегда до записи user-строки (pre-check/beginRun раньше insert) → повтор
  POST безопасен, дублей нет.

Внутренний цикл: 2 прохода. Ревью нашло CRITICAL: supersedeRetryRef застревал
взведённым, когда sendNow взвёл, но POST не ушёл (promoted head удалён →
flushNext() false; либо wasResumed-return) — следующая обычная отправка молча
ретраила настоящий 409. Починка: разоружать флаг симметрично остальным one-shot
(в ветке !flushNext() и в isStreaming-defuse-эффекте); транспорт read-and-clear
на входе каждой отправки. Мутационно: убрать disarm → strand-тест краснеет
(4 вызова вместо 1). Легаси-режим не тронут (регресс-гард).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:50:04 +03:00
agent_coder 15a9eba562 fix(delivery): immutable-кэш ассетов — отключить дефолтный cacheControl @fastify/static (#452)
Хэшированные ассеты отдавали 'cache-control: public, max-age=0' вместо
'public, max-age=31536000, immutable' → повторные заходы ревалидировали каждый
ассет (десятки 304 × мобильный RTT), главный выигрыш #346 не реализовывался.
Причина: у @fastify/static опция cacheControl:true по умолчанию пишет свой
Cache-Control (из maxAge, дефолт 0) ПОСЛЕ setHeaders-колбэка, затирая immutable-
заголовок из resolveStaticAssetHeaders. Фикс — cacheControl:false, колбэк
владеет заголовком. preCompressed не конфликтовал, потому баг был только в
заголовках.

Крайние случаи проверены: locales/vad/иконки получают только vary (без
cache-control → браузер ревалидирует по etag — ок); index.html отдаётся
отдельным wildcard-роутом со своим no-cache (не затронут); preCompressed .br
получает путь с /assets/ → маппинг матчит, immutable ставится.

Тест: bare-fastify + inject() — /assets/<hashed>.js содержит immutable+
max-age=31536000, /locales/en.json — нет. Мутационно: cacheControl:true роняет
ассерт immutable. jest static.module → 5/5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:30:44 +03:00
agent_coder 76af4f692e docs(mcp): поправить устаревшую ссылку footnote-authoring.ts -> @docmost/prosemirror-markdown
#429 (дедуп node-ops) перенёс footnoteContentKey в @docmost/prosemirror-markdown
и удалил footnote-authoring.ts, но два docstring-комментария в
footnote-normalize-merge.ts всё ещё ссылались на старое имя файла. Только
комментарии, на сборку/поведение не влияет.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:26:42 +03:00
agent_coder 4809348457 test(mcp): ассертить collab-hint (pageId + transient/retry) в reject-текстах (ревью #441)
Enrichment collab-ошибок из Phase C (#437) дописывает
'(pageId …; transient — retry once; …)' к connect-timeout/connection-closed,
но reject-регекспы матчили только базовый текст → проходили и С hint, и БЕЗ
(vacuous). Ужесточил два ассерта (connection-closed, connect-timeout) до
'<база> (pageId page-1; transient' — теперь рефактор, убравший hint(), их
роняет. Мутационно: hint()->'' → ровно эти 2 теста краснеют (18->16).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:14:39 +03:00
agent_coder d3d32d637b fix(mcp): no-response диагностика — не отдавать сырой error.message (утечка host в модель) (внутр. ревью #437)
no-response ветка формата использовала error.code ?? error.message: при
отсутствии code axios-сообщения сетевых ошибок содержат host:port
('connect ECONNREFUSED 127.0.0.1:3000', 'getaddrinfo ENOTFOUND host'), что
нарушает инвариант #437 «host никогда не попадает в видимое модели сообщение».
Теперь только error.code (?? 'network error'); полный нативный текст уходит в
stderr под DEBUG. code проставлен фактически для всех реальных no-response
ошибок. Тест обновлён: сырое host-содержащее сообщение -> нейтральный reason,
плюс ассерт что host в сообщении отсутствует.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:12:37 +03:00
agent_coder 9a435201b8 feat(mcp): enrich collab connect/persist/closed error texts with pageId + retry hint (#437)
Append `(pageId <id>; transient — retry once; persistent failures mean the
collab server is unreachable/overloaded)` to the connect-timeout, persist-timeout
and connection-closed error texts in CollabSession, so the agent can self-correct
instead of blind-looping. The Yjs-encode error is left untouched (it already names
the offending attribute).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:12:37 +03:00
agent_coder d6827b9210 feat(mcp): actionable tool errors — central axios diagnostics + fail-fast comment-id guard (#437)
Phase A: add ONE response interceptor on DocmostClient's axios instance,
registered AFTER the re-login interceptor, that reformats a failed request's
error.message IN PLACE (never a custom Error subclass, so the live
axios.isAxiosError / error.response?.status / config._retry checks keep working)
into `<METHOD> <path> failed (<status> <statusText>): <serverMessage>`, or the
no-response variant. serverMessage is built ONLY from the whitelisted
message/error fields or statusText — raw string/HTML bodies, headers and config
never appear; an arraybuffer body is size-capped JSON.parsed; the full body goes
to stderr only under DEBUG (parity with downloadImage). A _docmostFormatted flag
guards against double-processing.

Phase B (#436): assertFullUuid throws an actionable error BEFORE any network
call at all five commentId sites (resolve/update/delete/get_comment and
create_comment's parentCommentId when provided), so a truncated id can no longer
loop as an opaque 400/404.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:12:37 +03:00
186 changed files with 20976 additions and 8716 deletions
+34
View File
@@ -124,6 +124,40 @@ MCP_DOCMOST_PASSWORD=
# MCP_TOKEN=
# MCP_SESSION_IDLE_MS=1800000
#
# --- MCP collaboration write path: concurrency + rights-staleness (#449) ------
# MCP content writes (update_page, insert/replace nodes, comments-in-body, etc.)
# go over the collaboration websocket and are serialized PER PAGE by an
# in-process mutex (a module-level Map, one promise-chain per page UUID). This
# guarantees no two MCP writes on the SAME page overlap and clobber each other.
#
# DEPLOY REQUIREMENT — SINGLE INSTANCE or STICKY SESSIONS. The mutex is
# process-local. Behind a multi-replica load balancer WITHOUT sticky sessions,
# two replicas can each "hold" the lock for the same page at the same time and
# serialization is silently lost (concurrent full-document writes race on the
# live Yjs fragment). Run the MCP/app as a SINGLE instance, OR pin a page's
# traffic to one replica (sticky sessions / consistent hashing on page id). The
# same constraint applies to the RAM-only stash_page blob store above. There is
# deliberately no cross-process (e.g. Postgres advisory) lock yet — this is a
# CONSCIOUS documented constraint, not an oversight (#449).
#
# To reduce connect-storms the write path caches ONE live collab session per
# (wsUrl, page, token). Tunables (all optional; defaults are safe):
# MCP_COLLAB_SESSION_IDLE_MS=60000 # idle TTL, reset per op; 0 disables cache
# MCP_COLLAB_SESSION_MAX_ENTRIES=32 # LRU cap on cached sessions
# MCP_COLLAB_TOKEN_TTL_MS=300000 # per-client collab-token cache (5 min)
#
# RIGHTS-STALENESS TRADE-OFF. A cached collab session writes under the token
# captured at CONNECT time, and the collab-token cache reuses a token for its TTL.
# So if a user's access to a page is REVOKED, MCP writes on an already-open
# session may keep succeeding until the session ages out. MCP_COLLAB_SESSION_MAX_AGE_MS
# is the HARD lifetime (checked at each acquire) that BOUNDS this window: after it,
# the session is torn down and the next write re-auths with a fresh token, picking
# up the revocation. Default 10 min. LOWER it to shorten the revocation lag at the
# cost of more reconnects; RAISE it to reduce reconnects at the cost of a longer
# stale-rights window. There is intentionally no push-based cache invalidation on
# a rights change — this bounded window is the accepted trade-off (#449).
# MCP_COLLAB_SESSION_MAX_AGE_MS=600000
#
# BLOB SANDBOX (stash_page). An in-RAM, process-local store that hands large page
# content + images to an external consumer WITHOUT bloating the model context or
# requiring Docmost auth. The stash_page tool serializes a page, mirrors its
+6
View File
@@ -157,6 +157,12 @@ jobs:
- name: Build prosemirror-markdown
run: pnpm --filter @docmost/prosemirror-markdown build
# docmost-client.loader.ts type-imports from @docmost/mcp (issue #446); its
# build/ is gitignored and `test:e2e` type-checks, so build it here or tsc
# fails with TS2307 (mirrors the e2e-mcp / mcp-server-parity jobs).
- name: Build mcp
run: pnpm --filter @docmost/mcp build
- name: Run migrations
run: pnpm --filter ./apps/server migration:latest
+58
View File
@@ -1,5 +1,13 @@
name: Test
# NO `paths:` filter on purpose (issue #447). The tool-spec REGISTRY is split
# across two packages that MUST stay in sync: the specs live in `packages/mcp`
# but the parity/tier guard tests that read them live in the `apps/server` jest
# suite. A PR touching only `packages/mcp/**` must therefore still run the SERVER
# suite (and vice-versa), or an in-app wiring break slips through green and only
# surfaces on develop after merge. The `test` job below runs BOTH suites via
# `pnpm -r test` on every PR; the dedicated `mcp-server-parity` job makes that
# cross-package gate explicit and fast. Do not add a `paths:` filter here.
on:
pull_request:
workflow_call:
@@ -132,3 +140,53 @@ jobs:
# isolated `docmost_test` DB and migrates it to latest.
- name: Run server integration tests
run: pnpm --filter server test:int
# Cross-package tool-spec parity gate (issue #447). The tool-spec registry lives
# in `packages/mcp` but its parity/tier guard tests live in the `apps/server`
# jest suite, so a PR touching ONLY one of the two packages must still run BOTH
# sides — otherwise an in-app wiring break (e.g. PR #434 drawio) passes the mcp
# suite green and only surfaces on develop after merge. The `test` job already
# runs everything via `pnpm -r test`; this job is a fast, explicitly-named guard
# that runs the mcp `node --test` suite AND the server tool-guard jest specs
# together, so the coupling is visible and can never be accidentally split by a
# path filter. No Postgres/Redis needed: these specs mock the DB/loader.
mcp-server-parity:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up pnpm
uses: pnpm/action-setup@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Shared deps first (build/ dirs are gitignored; see test.yml build order).
- name: Build editor-ext
run: pnpm --filter @docmost/editor-ext build
- name: Build prosemirror-markdown
run: pnpm --filter @docmost/prosemirror-markdown build
# Build the mcp package so build/ carries a FRESH REGISTRY_STAMP (#447): the
# build runs gen-registry-stamp.mjs before tsc, so a build/ vs src/ skew
# cannot slip into the tests that exercise the loader's stale-check.
- name: Build mcp (regenerates REGISTRY_STAMP)
run: pnpm --filter @docmost/mcp build
# mcp side: the standalone MCP server's own tool-spec / instructions guards.
- name: Run mcp tool-spec suite
run: pnpm --filter @docmost/mcp test
# server side: the parity + tier guards that read packages/mcp/src/tool-specs
# and assert the in-app AI-chat wiring matches it.
- name: Run server tool-spec guard specs
run: pnpm --filter server exec jest shared-tool-specs.contract tool-tiers ai-chat-tools.service --runInBand
+10
View File
@@ -2,6 +2,11 @@
.env.dev
.env.prod
data
# Exception: the committed draw.io shape catalog (issue #424) lives in a `data/`
# dir, but the bare `data` ignore above is meant for runtime state, not this
# bundled build asset. Re-include the directory and its contents.
!packages/mcp/data/
!packages/mcp/data/**
# compiled output
/dist
node_modules
@@ -19,6 +24,11 @@ packages/prosemirror-markdown/build/
# markdown convention; the package is private and rebuilt at deploy.
packages/mcp/build/
# mcp REGISTRY_STAMP codegen output (issue #447). Regenerated into src/ by
# scripts/gen-registry-stamp.mjs on every `build`/`pretest` (before tsc), so it
# is a build artifact like build/ — never committed, always fresh.
packages/mcp/src/registry-stamp.generated.ts
# Logs
logs
*.log
+152 -3
View File
@@ -5,6 +5,139 @@ repository. It has two layers: **how to run a task end-to-end** (the
sections below), and **how the codebase is built** (the technical sections
further down, formerly in `CLAUDE.md`).
## ARCHITECTURAL INVARIANTS — NON-NEGOTIABLE
THE TEN RULES BELOW ARE HARD CONSTRAINTS. Each one was paid for with a real
production incident or a multi-PR bug chain in THIS repository (cited inline).
They override convenience, deadlines and "it's just a small feature". A PR that
violates any of them MUST be rejected in review regardless of how good the rest
of it is. If a task genuinely seems to require breaking one — STOP and raise it
with the owner; do not code around it.
### 1. EVERY BUFFER, CACHE, HISTORY AND PAYLOAD HAS AN EXPLICIT SIZE BUDGET
Nothing accumulates unboundedly. A row/item cap is NOT a byte cap. Anything
replayed to a model, buffered in memory, persisted per step, or refetched by a
poll must state its budget in bytes/tokens and enforce it. Rewriting a growing
structure in full on every increment is FORBIDDEN — append or diff instead;
O(n²) write/serialize patterns do not pass review.
(Paid for by: full-row rewrite on every agent step — hundreds of MB of Postgres
writes per 50-step run, with every tool output serialized twice; unbounded
history replay killing long chats on the provider context window; 32 MB replay
buffers per active run.)
### 2. EVERYTHING LONG-RUNNING TERMINATES BY CONSTRUCTION
Every run / row / session / lease / subscriber / queue entry must define AT
DESIGN TIME: its owner; every terminal state; who writes the terminal state on
EVERY path (success, error, abort, disconnect in each phase, process restart);
retries for the terminal write; and a periodic sweeper that does not depend on
a reboot. A best-effort terminal write with no retry and no sweep is FORBIDDEN.
(Paid for by: assistant rows stuck 'streaming' forever; runs stuck 'running'
409-locking their chat until a restart — the #183/#184 follow-up chain.)
### 3. EVERY AWAIT IS CANCELLABLE AND DEADLINED; NEVER BLOCK THE EVENT LOOP
Every async step inside a request or agent turn honors the turn's AbortSignal
AND a wall-clock deadline — including in-app tools, lock queues and pagination
loops, not just external calls. Synchronous CPU work beyond ~50 ms goes to a
worker_thread. Promise.race DOES NOT cancel synchronous work — using it as a
"timeout" for sync computation is forbidden (the timer only fires after the
event loop is free again, i.e. after the damage is done).
(Paid for by: in-app tools ignoring abortSignal and writing pages AFTER Stop;
the synchronous ELK layout freezing every SSE stream in the process; the
step-0 MCP handshake hang — #397.)
### 4. ONE SOURCE OF TRUTH; EVERYTHING ELSE IS A REBUILDABLE CACHE
Postgres is the authoritative state. Every in-memory structure (registries,
caches, client stores) must be reconstructible from the DB and treated as
lossy. The client renders SERVER-DECLARED state — "a run is active" is a server
fact delivered as data, never inferred from side signals (204 vs 2xx, the
flavor of a disconnect). A new feature must name the owner of each piece of
state before implementation starts.
(Paid for by: the strip/restore resume machinery, silently frozen UIs and
ghost sends after unmount — the #381#432#456 chain.)
### 5. STATE MACHINES ARE EXPLICIT — ONE-SHOT FLAGS ARE FORBIDDEN
A complex lifecycle (chat thread, resume/reconnect, run) lives in a named-state
automaton (reducer / enum) where every state has an owner and a rendered
representation — including the failure states. Adding a boolean ref that one
callback arms and another reads-and-clears is FORBIDDEN in the AI-chat client.
New behavior = a new named state + explicit transitions, and the interruption
matrix (disconnect in each phase × restart × stop × supersede) is enumerated at
design time, not discovered one incident at a time.
(Paid for by: 26 one-shot useRef flags in chat-thread.tsx and the drip of
"one more missing transition" across #381#386/#389#432#456.)
### 6. NO NEW MODE FORKS; A FLAG IS FOR ROLLOUT, THEN IT DIES
A behavior flag that forks a code path must ship with a written sunset
condition; stacking a new flag onto the existing matrix without deleting or
scheduling an old one is forbidden. While a temporary fork exists, BOTH sides
must share identical lifecycle handling (abort semantics, error listeners,
concurrency gates) — asymmetric forks are outlawed.
(Paid for by: legacy vs autonomous divergence — the one-active-run gate and
the socket 'error' listener each existing on only ONE side; 2^4 flag
combinations each with different abort semantics.)
### 7. NO HAND-SYNCED MIRRORS — CODEGEN OR A CI PARITY TEST, NOTHING LESS
Two copies of the same knowledge (schema, tool registry, glyph map, probe
body, hash/normalize algorithm, label list) require either generation from a
single source or a CI test that FAILS on drift. A "mirror this change over
there" comment is NOT a guard and does not pass review.
(Paid for by: #293 — three drifting converter copies losing data; #447
REGISTRY_STAMP covering only one of the mirrored files; ~10 still-unguarded
mirrors across the MCP layer.)
### 8. CACHES, HEADERS, BUFFERS AND FSM TRANSITIONS GET AN INTEGRATION TEST OF THE OBSERVABLE PROPERTY
A unit test of a pure helper DOES NOT COUNT for these. Test the real header on
the real HTTP response, the real cache hit under real token sources, the real
transition under a really-killed socket. If the observable property cannot be
tested, the design is wrong — fix the design, not the test.
(Paid for by: #431#439 — a cache keyed on a fresh-per-call JWT, so it NEVER
hit and became prod incident #435 while its unit tests stayed green; and by
the #352#455 immutable-cache header silently overwritten by a framework
default AFTER the unit-tested code ran.)
### 9. CLIENT INPUT IS HOSTILE UNTIL VALIDATED — ALSO BEFORE PERSISTENCE
Anything from the browser (message parts, ids, titles, selections, flags) is
validated/sanitized BEFORE it is persisted into a row that will later be
replayed into a prompt, a converter or another subsystem. A poisoned row must
never be able to permanently brick a chat or a page on every subsequent read.
(Paid for by: unvalidated UIMessage parts persisted verbatim — one bad row
500s the chat on every later turn; #159 client-spoofed page titles; #388
selection re-sanitized server-side for the same reason.)
### 10. FAILURES ARE LOUD AND SPECIFIC; SILENT DEGRADATION IS FORBIDDEN
Extends the error convention below: a fire-and-forget write is allowed ONLY
with a metric or a greppable ERROR log; a degraded mode (dead cached MCP
client, stopped poll, exhausted retries, evicted buffer) must be VISIBLE to
the user or the operator. A feature that can quietly stop working — a frozen
"streaming…" UI, a poll that silently gives up, a cache serving corpses — does
not pass review.
(Paid for by: the degraded poll's silent 10-minute death leaving a forever-
"streaming" answer; dead MCP clients served from cache while every external
tool call failed; #435 being caught in minutes ONLY because metrics — #403
existed.)
## Default skill for feature design
For any feature-design request — the user hands over a raw feature idea, asks
to design or think through a feature, or to draft an issue («спроектируй»,
«продумай фичу», «составь ишью», "design X", "write an issue for X") — invoke
the `orchestrator-feature-designer` skill (Skill tool) BEFORE any other work.
It is the default operating mode for design work in this repository: research
→ design checklist (R1–R10) → forks resolved with the human → adversarial
self-attack → filed PR-sized issues. Do not design features or write issues
ad-hoc while this skill is available. This does not apply to non-design work
(bug fixes, reviews, retrospectives, refactors already specified by an issue).
## Task lifecycle
### 1. Start: sync with develop
@@ -201,7 +334,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked; jsdom (Node only) | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, `apps/server` (server-side markdown import/export, #345), AND `apps/client` (markdown paste/copy + AI-chat render, via the `browser` entry — native `DOMParser`, no jsdom in the client bundle, #347); there is exactly ONE copy of the converter now |
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
@@ -248,6 +381,22 @@ pnpm collab:dev # run the collaboration server process standalone (
> that order). Reach for it whenever you run a consumer package's checks on their
> own rather than through the full `pnpm build`.
> **Editing an MCP tool spec requires a rebuild (issue #447).** The running
> server loads the **compiled** `packages/mcp/build/` of `@docmost/mcp` (via the
> runtime loader in `apps/server/src/core/ai-chat/tools/docmost-client.loader.ts`),
> but the parity/tier guard tests read `packages/mcp/src/tool-specs.ts`. So if you
> edit `tool-specs.ts` (any tool name, description, tier, catalog line, or input
> schema) **without rebuilding**, `build/` and `src/` silently diverge — the tests
> stay green while the server serves the OLD tools. To close that gap, the build
> emits a `REGISTRY_STAMP` (a deterministic hash of the tool-specs content, via
> `scripts/gen-registry-stamp.mjs` before `tsc`); on dev/test startup the loader
> recomputes it from `src/` and **refuses to start with a "@docmost/mcp build is
> stale …" error** on a mismatch (a pure no-op in prod, where only `build/` ships).
> After editing tool specs, rebuild:
> ```bash
> pnpm --filter @docmost/mcp build # or: pnpm --filter @docmost/mcp watch
> ```
**Lint** (per package — there is no root lint script):
```bash
pnpm --filter server lint # eslint --fix on server .ts
@@ -311,7 +460,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, `apps/server` (#345), and `apps/client` (#347) — do NOT reintroduce a per-package copy. The client uses the package's `browser` entry (`@docmost/prosemirror-markdown/browser`): markdown paste (`markdown-clipboard.ts`), copy-as-markdown, and AI-chat rendering now all go through the canonical converter, so the hand-written `marked`/`turndown` markdown layer that used to live in `editor-ext` was deleted (#347). The browser entry runs the HTML→DOM stage on the native `DOMParser`, so jsdom stays out of the client bundle. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
@@ -322,7 +471,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire test: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`) — it MUST be re-created via `pnpm patch` when bumping `ai`.
- **Adding/renaming/removing an MCP tool requires updating `SERVER_INSTRUCTIONS`** in `packages/mcp/src/index.ts` — the intent-routing guide MCP clients receive on initialize. This applies both to inline `server.registerTool(...)` calls in `index.ts` and to specs in `packages/mcp/src/tool-specs.ts`. Enforced by `packages/mcp/test/unit/server-instructions.test.mjs`, which fails when a registered tool is not mentioned in the guide (deliberate opt-outs go into its `EXCEPTIONS` list). `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
- **The MCP tool inventory in `SERVER_INSTRUCTIONS` is GENERATED from the registry** (`packages/mcp/src/server-instructions.ts`: `buildToolInventory()` over `SHARED_TOOL_SPECS`) and spliced into the hand-written routing prose (`ROUTING_PROSE`). So adding/renaming/removing a **shared** spec in `packages/mcp/src/tool-specs.ts` auto-updates the `<tool_inventory>` — no manual `SERVER_INSTRUCTIONS` edit needed. Only an **inline** MCP-only tool (those registered via `server.registerTool(...)` in `index.ts`, not through the registry) needs a one-line entry in `INLINE_MCP_INVENTORY`. Enforced by `packages/mcp/test/unit/tool-inventory.test.mjs`, which fails when a registered tool is missing from the generated inventory (there is no `EXCEPTIONS` opt-out anymore — every tool must appear). Update `ROUTING_PROSE` when a tool's *intent guidance* (when-to-use) changes. `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
+133
View File
@@ -10,6 +10,111 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Breaking Changes
- **External MCP tool names are now camelCase (all renamed).** Every tool on the
external `/mcp` surface was renamed from `snake_case` to `camelCase`, so the
external MCP name now matches the in-app tool name exactly (one logical tool,
one name everywhere). For example `get_node``getNode`, `edit_page_text`
`editPageText`, `patch_node``patchNode`. The tools' behaviour, inputs and
outputs are unchanged — only the names change. The single-word `search`
keeps its name.
*Migration (external MCP clients only — the in-app AI agent already used these
names and is unaffected):* update anything that refers to a tool by its
string name — permission allowlists (`mcp__gitmost-*__get_node`
`mcp__gitmost-*__getNode`), saved prompts/skills, `.mcp.json` tool filters,
and metrics dashboards that group by the `tool` label — and roll it out in
lockstep with this deploy, because the old snake_case names stop resolving.
Released together with the `import_page_markdown`/`update_page_markdown`
change below so external configs break exactly once.
Full mapping (old → new):
| Old (snake_case) | New (camelCase) |
| --- | --- |
| `check_new_comments` | `checkNewComments` |
| `copy_page_content` | `copyPageContent` |
| `create_comment` | `createComment` |
| `create_page` | `createPage` |
| `delete_comment` | `deleteComment` |
| `delete_node` | `deleteNode` |
| `delete_page` | `deletePage` |
| `diff_page_versions` | `diffPageVersions` |
| `docmost_transform` | `docmostTransform` |
| `drawio_create` | `drawioCreate` |
| `drawio_get` | `drawioGet` |
| `drawio_guide` | `drawioGuide` |
| `drawio_shapes` | `drawioShapes` |
| `drawio_update` | `drawioUpdate` |
| `edit_page_text` | `editPageText` |
| `export_page_markdown` | `exportPageMarkdown` |
| `get_node` | `getNode` |
| `get_outline` | `getOutline` |
| `get_page` | `getPage` |
| `get_page_json` | `getPageJson` |
| `get_workspace` | `getWorkspace` |
| `insert_footnote` | `insertFootnote` |
| `insert_image` | `insertImage` |
| `insert_node` | `insertNode` |
| `list_comments` | `listComments` |
| `list_page_history` | `listPageHistory` |
| `list_pages` | `listPages` |
| `list_shares` | `listShares` |
| `list_spaces` | `listSpaces` |
| `move_page` | `movePage` |
| `patch_node` | `patchNode` |
| `rename_page` | `renamePage` |
| `replace_image` | `replaceImage` |
| `resolve_comment` | `resolveComment` |
| `restore_page_version` | `restorePageVersion` |
| `search` | `search` (unchanged) |
| `search_in_page` | `searchInPage` |
| `share_page` | `sharePage` |
| `stash_page` | `stashPage` |
| `table_delete_row` | `tableDeleteRow` |
| `table_get` | `tableGet` |
| `table_insert_row` | `tableInsertRow` |
| `table_update_cell` | `tableUpdateCell` |
| `unshare_page` | `unsharePage` |
| `update_comment` | `updateComment` |
| `update_page_json` | `updatePageJson` |
| `update_page_markdown` | `updatePageMarkdown` |
(#412)
- **External MCP: `import_page_markdown` removed, `update_page_markdown` added.**
The external `/mcp` surface no longer exposes `importPageMarkdown` (the
round-trip parser for a self-contained *exported* Docmost-Markdown file). In
its place it now exposes **`updatePageMarkdown`** — a plain-Markdown
full-body replace (`{pageId, content, title?}`) that pairs with
`updatePageJson`, re-imports the whole body (block ids regenerate) and
parses Docmost-flavoured markdown including `^[...]` inline footnotes.
*Migration:* MCP clients that called `importPageMarkdown` to overwrite a
page's body from Markdown should call `updatePageMarkdown` instead (pass the
markdown as `content`). Round-tripping an exported Docmost-Markdown file with
comment anchors/diagrams is no longer available on the external MCP surface;
export remains via `exportPageMarkdown`. The in-app AI agent is unaffected —
it keeps both `importPageMarkdown` and the renamed `updatePageMarkdown` (was
`updatePageContent`). The total MCP tool count is unchanged (−1 / +1). The
external names shown here are the post-#412 camelCase names. (#411)
- **`getNode` now returns Markdown by default (was ProseMirror JSON).** The
block-level read/write tools default to Markdown so a block round trip is
`getNode` (markdown) → edit → `patchNode` (markdown). `getNode` now returns
`{ …, format: "markdown", markdown }` unless you pass `format: "json"` (which
restores the previous `{ …, node }` ProseMirror subtree); comment anchors —
including resolved ones — are preserved in the markdown so a write-back never
orphans a thread, and a node that cannot be a document top-level block
(`tableRow`/`tableCell`/`tableHeader` addressed via `#<index>`) auto-falls back
to JSON with `format: "json"` in the response. `patchNode`/`insertNode` gain a
`markdown` input alongside `node` (provide exactly one): the markdown fragment
may rewrite/insert several blocks at once and supports `^[...]` footnotes.
*Migration (external MCP clients only):* a client that consumed `getNode`'s
`node` field must now either read `markdown`, or pass `format: "json"` to keep
the old ProseMirror-JSON output. Released together with the `#411`/`#412`
breaking window so external configs break exactly once. (#413)
### Added
- **Place several images side by side in a row.** A new "Inline (side by
@@ -146,9 +251,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
by physical key position and matched against the commands; genuine Cyrillic
search terms keep priority over remapped candidates, and short wrong-layout
prefixes match by command title. (#283, #285, #287)
- **Opt-in substring "lookup" search mode for agents.** `/api/search` gains an
additive, opt-in mode (guarded by a new `substring` flag) that matches literal
substrings of page titles and body text — so technical tokens the full-text
tokenizer mangles (`backup-srv.local`, `10.0.12.5`, `WB-MGE-30D86B`) are found
even when the FTS query is empty. It returns a location `path`, a windowed
`snippet` and a per-response relevance `score`, supports `titleOnly` and a
`parentPageId` subtree scope, and applies the page-level permission filter
before the limit. The web UI never sets `substring`, so its full-text search
behaviour is byte-for-byte unchanged. The leading-wildcard `LIKE` predicates
are backed by GIN trigram indexes on `LOWER(f_unaccent(title))` and
`LOWER(f_unaccent(text_content))` so lookups use a bitmap index scan instead of
a sequential scan. (#443)
- **MCP `search` tool returns richer, agent-oriented results.** The external MCP
`search` response shape changes for the agent surface: each hit now carries
`pageId` (renamed from `id`), plus `path`, `snippet` and `score`; the
UI-oriented `spaceId`, `rank` and `highlight` fields are dropped. (#443)
### Changed
- **Client markdown paste/copy and AI-chat rendering now go through the canonical
converter.** Pasting markdown into the editor, "Copy as markdown", the AI title
generator, and the AI-chat markdown renderer all now use
`@docmost/prosemirror-markdown` (via its new `browser` entry — native
`DOMParser`, no jsdom in the client bundle) instead of the hand-written
`marked`/`turndown` markdown layer in `editor-ext`, which was **deleted**. As a
result, pasting canonical markdown (`^[…]` footnotes, `<!--img …-->`,
`> [!type]` callouts, `$…$` math, `==…==` highlight, standalone `<!--subpages-->`
comments) now produces the SAME nodes the server import produces for the same
text. Chat/reasoning markdown now renders through the editor schema (list items
are wrapped in `<p>`; CSS keeps them tight). (#347)
- **Enabling a public share no longer auto-shares the whole sub-tree.** Turning
a page "Shared to web" now defaults to the page alone; descendant pages become
public only when you explicitly turn on the dedicated "Include sub-pages"
+15
View File
@@ -45,6 +45,11 @@ COPY --from=builder /app/packages/editor-ext/dist /app/packages/editor-ext/dist
COPY --from=builder /app/packages/editor-ext/package.json /app/packages/editor-ext/package.json
COPY --from=builder /app/packages/mcp/build /app/packages/mcp/build
COPY --from=builder /app/packages/mcp/package.json /app/packages/mcp/package.json
# The mcp package reads its data files (drawio-presets.json, drawio-shape-index.json.gz)
# at runtime via `new URL("../../data/…", import.meta.url)` relative to build/lib/*.js,
# i.e. from packages/mcp/data/. tsc emits only build/, so ship data/ explicitly or
# drawioFromGraph and the shape catalog die with ENOENT on packages/mcp/data/*.
COPY --from=builder /app/packages/mcp/data /app/packages/mcp/data
# mcp now depends on @docmost/prosemirror-markdown (workspace:*) and eager-imports
# it at runtime (the in-app ai-chat DocmostClient loads build/index.js -> lib/
# markdown-converter.js). Ship the built package + its manifest, or the prod
@@ -81,4 +86,14 @@ VOLUME ["/app/data/storage"]
EXPOSE 3000
# DEPLOY REQUIREMENT — SINGLE INSTANCE or STICKY SESSIONS (#449).
# MCP content writes are serialized per page by an IN-PROCESS mutex, and the
# stash_page blob store + cached collab sessions are RAM-only and process-local.
# Running MULTIPLE replicas of this image behind a load balancer WITHOUT sticky
# sessions silently breaks per-page write serialization (two replicas can lock
# the same page at once) and makes stash_page blobs unreachable across replicas.
# Run a SINGLE instance, or pin each page's traffic to one replica (sticky
# sessions / consistent hashing on page id). There is deliberately no
# cross-process lock yet — a conscious constraint. See .env.example (the "MCP
# collaboration write path" block) and packages/mcp/README.md for details.
CMD ["pnpm", "start"]
+1
View File
@@ -21,6 +21,7 @@
"@atlaskit/pragmatic-drag-and-drop-live-region": "1.3.4",
"@casl/react": "5.0.1",
"@docmost/editor-ext": "workspace:*",
"@docmost/prosemirror-markdown": "workspace:*",
"@excalidraw/excalidraw": "0.18.0-3a5ef40",
"@mantine/core": "8.3.18",
"@mantine/dates": "8.3.18",
@@ -55,6 +55,15 @@
padding-inline-start: 1.4em;
}
/* The canonical converter renders list items through the editor schema, which
wraps each item's content in a <p> (listItem content is `paragraph+`). Drop
that paragraph's block margin so list items render TIGHT (no extra vertical
gap), matching the previous marked output — same rule already applied to
table cells above (issue #347). */
.markdown li p {
margin: 0;
}
/* GFM tables in assistant markdown. The chat lives in a NARROW side panel, so a
wide LLM table must scroll horizontally instead of collapsing its columns:
`.markdown` sets `word-break: break-word`, which (with the default table
@@ -172,6 +181,14 @@
margin: 0 0 4px;
}
/* Same as `.markdown li p` above: the canonical converter wraps every list
item's content in a <p>, so without this each reasoning-panel list item would
pick up `.reasoningText p`'s 4px bottom margin and render too loose. Drop it
so Reasoning-panel lists stay tight, mirroring the pre-#347 marked output. */
.reasoningText li p {
margin: 0;
}
.inputWrapper {
flex: 0 0 auto;
padding-top: var(--mantine-spacing-xs);
@@ -28,7 +28,10 @@ const h = vi.hoisted(() => ({
body: Record<string, unknown>;
}) => { body: Record<string, unknown> };
prepareReconnectToStreamRequest?: () => { api?: string };
fetch?: (input: unknown, init?: { method?: string }) => Promise<unknown>;
fetch?: (
input: unknown,
init?: { method?: string; body?: unknown },
) => Promise<unknown>;
},
},
}));
@@ -200,6 +203,244 @@ describe("ChatThread — send now (#198)", () => {
});
});
// #396: in autonomous mode a live sendNow must additionally request the
// AUTHORITATIVE server stop of the detached run (a local abort is only a client
// disconnect the server ignores) and arm a bounded 409 retry so the re-POST
// converges once the one-active-run slot frees. Legacy mode is unchanged.
describe("ChatThread — send now server-stop + supersede retry (#396)", () => {
beforeEach(resetState);
afterEach(cleanup);
// A settled assistant tail => no mount resume (attemptResumeRef false), so the
// "Send now" button is visible for the NEW local streaming turn while
// autonomous runs are enabled.
const settledTail = () => [
row("u1", "user", undefined, "hi"),
row("a1", "assistant", "succeeded", "done"),
];
it("autonomous: sendNow during a live stream calls onServerStop with the chat id", () => {
const { onServerStop } = renderThread({
autonomousRunsEnabled: true,
initialRows: settledTail(),
});
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
expect(h.state.stop).toHaveBeenCalledTimes(1);
expect(onServerStop).toHaveBeenCalledWith("c1");
});
it("legacy (autonomous off): sendNow does NOT call onServerStop and does NOT retry the send", async () => {
const { onServerStop } = renderThread({
autonomousRunsEnabled: false,
initialRows: settledTail(),
});
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
expect(onServerStop).not.toHaveBeenCalled();
// The supersede retry must NOT be armed: a POST that 409s is returned as-is
// (single fetch, no retry).
const fetchMock = vi
.fn()
.mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
it("armed supersede send retries 409 A_RUN_ALREADY_ACTIVE and succeeds once the slot frees", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
// Arm the retry by performing a live sendNow (autonomous branch sets the ref).
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
const fetchMock = vi
.fn()
// First POST: the old detached run still holds the slot -> 409.
.mockResolvedValueOnce(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
)
// Retry: the server stop settled the old run -> 200.
.mockResolvedValueOnce(new Response("ok", { status: 200 }));
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(2);
expect(res.status).toBe(200);
});
it("supersede retry is one-shot: a later send (ref cleared) does NOT retry a 409", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now")); // arms the one-shot
// First armed send: immediately succeeds, consuming the arm.
let fetchMock = vi
.fn()
.mockResolvedValue(new Response("ok", { status: 200 }));
vi.stubGlobal("fetch", fetchMock);
await act(async () => {
await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
});
});
expect(fetchMock).toHaveBeenCalledTimes(1);
// A subsequent send is NOT armed -> a 409 is returned as-is (no retry).
fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
it("supersede retry is bounded: exhaustion surfaces the 409 error", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
// Every attempt 409s -> after 4 attempts the last 409 surfaces.
const fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
// 4 attempts total (1 immediate + 3 backoff retries), then give up.
expect(fetchMock).toHaveBeenCalledTimes(4);
expect(res.status).toBe(409);
});
it("armed supersede send does NOT retry a non-409 status", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
const fetchMock = vi
.fn()
.mockResolvedValue(new Response("boom", { status: 500 }));
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(500);
});
// Strand-path regression: sendNow arms the supersede retry, but if the promoted
// head is removed before the abort's onFinish lands, flushNext() sends nothing
// (returns false) and NO re-POST consumes the arm. The arm must be disarmed on
// that no-send branch so the NEXT unrelated NORMAL send does not inherit it and
// silently retry a genuine 409 (e.g. a legitimate two-tab conflict) 4x instead
// of surfacing it immediately.
it("strand-path: a stranded supersede arm (flushNext no-send) does NOT retry a later normal 409", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
// Arm the retry via a live autonomous sendNow (promotes the head + arms).
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
// Remove the promoted head BEFORE the abort lands, so flushNext() returns
// false (no POST) and the arm would strand without the disarm fix.
fireEvent.click(screen.getByLabelText("Remove queued message"));
// The abort's onFinish now takes the flushOnAbortRef branch, calls flushNext()
// which finds an empty queue and returns false -> the no-send disarm must run.
act(() => {
h.state.onFinish?.({
message: { id: "a1", role: "assistant", parts: [] },
isAbort: true,
isDisconnect: false,
isError: false,
});
});
// No re-POST was sent (nothing to flush).
expect(h.state.sendMessage).not.toHaveBeenCalled();
// A subsequent NORMAL send that 409s must be returned as-is (exactly 1 fetch):
// the stranded arm must NOT cause the genuine 409 to be retried.
const fetchMock = vi
.fn()
.mockResolvedValue(
new Response(JSON.stringify({ code: "A_RUN_ALREADY_ACTIVE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
it("armed supersede send does NOT retry a 409 with a different (non-A_RUN_ALREADY_ACTIVE) body", async () => {
renderThread({ autonomousRunsEnabled: true, initialRows: settledTail() });
fireEvent.click(screen.getByTestId("queue-btn"));
fireEvent.click(screen.getByLabelText("Send now"));
const fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ code: "SOMETHING_ELSE" }), {
status: 409,
}),
);
vi.stubGlobal("fetch", fetchMock);
let res!: Response;
await act(async () => {
res = (await h.state.transport!.fetch!("http://x", {
method: "POST",
body: "{}",
})) as Response;
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(res.status).toBe(409);
});
});
// #388: the editor selection is snapshotted at send time and nested inside
// openPage on the wire. The getter is read live from a ref, so each send ships a
// fresh snapshot.
@@ -70,6 +70,36 @@ const RECONNECT_MAX_ATTEMPTS = 5;
// Backoff before attempt N (1-based): 1s, 2s, 4s, 8s, 16s.
const RECONNECT_BASE_DELAY_MS = 1000;
// #396: bounded retry for the "Interrupt and send now" re-send when it races the
// authoritative server stop of the just-superseded detached run. The re-POST can
// arrive before the old run has released the one-active-run slot, so the server
// returns 409 A_RUN_ALREADY_ACTIVE. The server stop guarantees the slot frees, so
// a few short backoffs converge. 4 total attempts: attempt 1 fires immediately,
// then these are the waits BEFORE attempts 2, 3 and 4 (150ms, 300ms, 600ms). If
// all 4 attempts 409, the last 409 surfaces (the banner) — acceptable per #396.
const SUPERSEDE_RETRY_DELAYS_MS = [150, 300, 600];
// The server error code that means "another run is already active for this chat".
const A_RUN_ALREADY_ACTIVE = "A_RUN_ALREADY_ACTIVE";
/**
* #396: defensively decide whether a 409 response is the one-active-run gate
* rejection (code A_RUN_ALREADY_ACTIVE) vs. some other 409. Reads a CLONE so the
* original response body stays intact for the caller when it is returned as-is.
* Any parse failure or unexpected shape => false (do NOT retry).
*/
async function isRunAlreadyActive(response: Response): Promise<boolean> {
try {
const body = (await response.clone().json()) as unknown;
return (
typeof body === "object" &&
body !== null &&
(body as { code?: unknown }).code === A_RUN_ALREADY_ACTIVE
);
} catch {
return false;
}
}
/** The page the user is currently viewing, sent as chat context. */
export interface OpenPageContext {
id: string;
@@ -326,6 +356,26 @@ export default function ChatThread({
const flushOnAbortRef = useRef(false);
const interruptNextSendRef = useRef(false);
// #396: one-shot arm for the bounded 409 A_RUN_ALREADY_ACTIVE retry on the
// "Interrupt and send now" re-send in autonomous mode. sendNow triggers the
// authoritative server stop of the detached run, but that stop and the
// onFinish->flushNext re-POST race: the new POST can hit the one-active-run
// gate before the old detached run has settled, yielding a spurious 409. When
// this ref is armed, the transport's send path retries that 409 with a short
// bounded backoff (the server stop guarantees convergence). A normal send (ref
// not armed) must STILL fail a 409 instantly (e.g. a genuine two-tab conflict).
//
// INVARIANT: sendNow arms this only to be consumed by the ONE re-POST that
// flushNext fires from onFinish. But that re-POST does not always happen (the
// promoted head may be gone, the finish may be a resumed turn, or the arm may
// race a stale finish). To keep the arm strictly one-shot it is disarmed on
// EVERY path where the paired interrupt one-shots (flushOnAbortRef /
// interruptNextSendRef) are cleared without a POST: the transport POST branch
// consumes it (read-and-clear), the onFinish `!flushNext()` no-send branch
// clears it, and the isStreaming-defuse effect clears it symmetrically. So it
// can never leak into a later, unrelated send and retry that send's genuine 409.
const supersedeRetryRef = useRef(false);
// #234 F5: the user pressed Stop while streaming a BRAND-NEW chat whose server
// chat id has not been adopted yet (the `start` chunk carrying it hadn't landed
// when Stop was pressed). A local SSE abort alone does NOT stop the DETACHED
@@ -382,7 +432,43 @@ export default function ChatThread({
}`,
}),
fetch: async (input: RequestInfo | URL, init: RequestInit = {}) => {
if ((init.method ?? "GET") !== "GET") return fetch(input, init); // send path untouched
if ((init.method ?? "GET") !== "GET") {
// Send path (POST). #396: read-and-clear the one-shot supersede arm
// here so it is strictly scoped to THIS send. When unarmed, behave
// exactly as before — a single fetch, a 409 surfaces instantly (a
// genuine two-tab conflict must NOT be retried).
const supersede = supersedeRetryRef.current;
supersedeRetryRef.current = false;
if (!supersede) return fetch(input, init);
// Buffer a ReadableStream body once so each retry can replay it.
// DefaultChatTransport sends the body as a JSON STRING (replayable as
// is), but guard defensively in case a future SDK streams it.
let sendInit = init;
if (init.body instanceof ReadableStream) {
const buffered = await new Response(init.body).arrayBuffer();
sendInit = { ...init, body: buffered };
}
// Bounded retry: attempt 1 fires immediately, then wait between
// attempts per SUPERSEDE_RETRY_DELAYS_MS. Retry ONLY on a real
// 409 A_RUN_ALREADY_ACTIVE; any other status/body is returned as-is.
for (let attempt = 0; ; attempt++) {
const response = await fetch(input, sendInit);
if (
response.status !== 409 ||
attempt >= SUPERSEDE_RETRY_DELAYS_MS.length ||
!(await isRunAlreadyActive(response))
) {
return response;
}
// The old detached run has not released the one-active-run slot
// yet; the server stop we requested guarantees it will, so back off
// and re-POST (the 409 fired before the user message was persisted,
// so re-POSTing is safe — no duplicate rows).
await new Promise((r) =>
setTimeout(r, SUPERSEDE_RETRY_DELAYS_MS[attempt]),
);
}
}
// Reconnect GET: the SDK passes no AbortSignal, so wire our own controller
// for observer Stop / unmount abort.
const controller = new AbortController();
@@ -562,9 +648,14 @@ export default function ChatThread({
setStopNotice(null);
// If the promoted head vanished (e.g. the user removed it before the
// abort landed) flushNext sends nothing — clear the one-shot interrupt
// tag so it can't leak onto the next unrelated send. On a real send the
// tag is consumed by prepareSendMessagesRequest and stays untouched.
if (!flushNext()) interruptNextSendRef.current = false;
// tag AND the #396 supersede arm so neither can leak onto the next
// unrelated send (no re-POST will consume the arm here). On a real send
// the tag is consumed by prepareSendMessagesRequest and the arm by the
// transport POST branch, so both stay untouched then.
if (!flushNext()) {
interruptNextSendRef.current = false;
supersedeRetryRef.current = false;
}
return;
}
if (isAbort || isDisconnect || isError) return;
@@ -873,6 +964,30 @@ export default function ChatThread({
setQueue(promoteToHead(queuedRef.current, id));
flushOnAbortRef.current = true;
interruptNextSendRef.current = true;
// #396: in autonomous mode the turn is a DETACHED run — a local stop()
// is only a client disconnect the server ignores, so the run keeps going.
// The onFinish->flushNext re-POST would then hit the one-active-run gate
// and get a spurious 409 A_RUN_ALREADY_ACTIVE. Mirror handleStop: request
// the AUTHORITATIVE server stop so the detached run settles, and arm the
// one-shot bounded 409 retry BEFORE stop() so the re-send converges once
// the slot frees. Read chatId live from chatIdRef (adopted at the `start`
// chunk). If it is not known yet (brand-new chat, first moment of its
// first turn), defer the server stop via stopPendingRef exactly as
// handleStop does — the onServerChatId adoption effect fires it once the
// id lands; the retry stays armed so the re-send still converges then.
if (autonomousRunsEnabled) {
supersedeRetryRef.current = true; // arm the bounded 409 retry
if (chatIdRef.current) {
onServerStop?.(chatIdRef.current);
} else {
// Same #234-F5 sub-window limitation documented in handleStop: if the
// local abort below cancels the reader before the `start` chunk lands,
// the adoption effect never runs and the deferred stop never fires. Not
// a regression; at minimum we don't strand refs (the isStreaming effect
// defuses stopPendingRef on the next turn start).
stopPendingRef.current = true;
}
}
stop(); // -> onFinish({ isAbort: true }) flushes the promoted head
} else {
// Nothing to interrupt: just send it now (no interrupt note).
@@ -884,7 +999,7 @@ export default function ChatThread({
sendMessageRef.current?.({ text: msg.text });
}
},
[setQueue, stop, setResumedTurnPair],
[setQueue, stop, setResumedTurnPair, autonomousRunsEnabled, onServerStop],
);
// Stop the current turn. ALWAYS abort the local SSE (`stop()`) so the composer
@@ -944,6 +1059,13 @@ export default function ChatThread({
setStopNotice(null);
flushOnAbortRef.current = false;
interruptNextSendRef.current = false;
// #396: symmetric with the other one-shot interrupt flags — defuse a stale
// supersede arm that was set but whose expected re-POST never fired (the
// turn finished in the same tick as the click, or the promoted head was
// gone), so it can never leak into this (or a later) turn's send and retry
// that send's genuine 409. A legit arm is consumed by the transport POST
// branch before this new turn streams, so this does not clobber it.
supersedeRetryRef.current = false;
// #234 F5: a new turn is starting — drop any pending deferred-stop from a
// previous turn that never adopted an id, so it can never fire against this
// (or a later) unrelated turn's run. A deferred stop for the CURRENT turn is
@@ -33,29 +33,44 @@ describe("collapseBlankLines", () => {
});
});
describe("collapseBlankLines + renderChatMarkdown (tight reasoning rendering)", () => {
it("renders a blank-line-separated list as a TIGHT list (no <li><p>)", () => {
describe("collapseBlankLines + renderChatMarkdown (canonical converter)", () => {
// Chat markdown now renders through @docmost/prosemirror-markdown (issue #347):
// the SAME converter the editor/import use. Its list items are schema-shaped —
// each <li>'s content is wrapped in a <p> (listItem content is `paragraph+`) —
// so the HTML always carries `<li><p>…</p></li>` regardless of blank-line
// looseness in the source (the converter has no tight/loose distinction). The
// visual tightness that `collapseBlankLines` used to buy is now provided by
// CSS (`.markdown li p { margin: 0 }`), not the HTML shape.
it("renders a blank-line-separated bullet list as a real <ul> list", () => {
const loose =
"Intro paragraph.\n\n- item one\n\n- item two\n\n- item three";
const html = renderChatMarkdown(collapseBlankLines(loose), {});
// Tight list: each <li> holds the text directly, not wrapped in a <p>.
expect(html).toContain("<li>item one</li>");
expect(html).not.toContain("<li><p>");
// The list still parses as a list after the paragraph (not a paragraph+<br>).
// Clean, un-namespaced HTML (DOMSerializer, not XMLSerializer) — no xmlns.
expect(html).toContain("<ul>");
expect(html).not.toMatch(/<ul[^>]*xmlns/);
// The item text is present (inside the schema's <li><p> wrapper).
expect(html).toContain("item one");
// The intro paragraph renders as its own paragraph before the list.
expect(html).toContain("<p>Intro paragraph.</p>");
});
it("renders an ordered list (1. 2.) as tight after collapsing", () => {
it("renders an ordered list (1. 2.) as a real <ol> list", () => {
const loose = "Intro.\n\n1. first\n\n2. second";
const html = renderChatMarkdown(collapseBlankLines(loose), {});
expect(html).toContain("<ol>");
expect(html).toContain("<li>first</li>");
expect(html).not.toContain("<li><p>");
expect(html).not.toMatch(/<ol[^>]*xmlns/);
expect(html).toContain("first");
expect(html).toContain("second");
});
it("the loose source WOULD render <li><p> without collapsing (control)", () => {
it("wraps list-item content in <p> (schema shape; tightness is CSS)", () => {
// The canonical converter always wraps a list item's content in a paragraph,
// whether or not the source had blank lines between items.
const loose = "- a\n\n- b";
expect(renderChatMarkdown(loose, {})).toContain("<li><p>");
// And a "tight" source produces the identical wrapping (no distinction).
expect(renderChatMarkdown(collapseBlankLines(loose), {})).toContain(
"<li><p>",
);
});
});
@@ -1,6 +1,37 @@
import { markdownToHtml } from "@docmost/editor-ext";
import {
markdownToProseMirrorSync,
docmostExtensions,
} from "@docmost/prosemirror-markdown/browser";
import { getSchema } from "@tiptap/core";
import { Node as PMNode, DOMSerializer } from "@tiptap/pm/model";
import DOMPurify from "dompurify";
// The Docmost editor schema, built once. Chat markdown is rendered through the
// SAME schema the editor/import use (issue #347), so chat output matches how the
// page would render the same markdown.
const chatSchema = getSchema(docmostExtensions);
/**
* Markdown -> HTML for chat display, via the canonical converter. We serialize
* the ProseMirror doc with `DOMSerializer` into a real element and read its
* `innerHTML` (rather than `@tiptap/html`'s `generateHTML`, whose browser path
* uses `XMLSerializer` and stamps a `xmlns` on every block) so the markup is
* clean HTML. `li > p` wrapping is inherent to the schema (listItem content is
* `paragraph+`); the chat CSS zeroes those paragraph margins so lists still
* render tight.
*/
function markdownToChatHtml(markdown: string): string {
const doc = markdownToProseMirrorSync(markdown);
const node = PMNode.fromJSON(chatSchema, doc);
const div = document.createElement("div");
DOMSerializer.fromSchema(chatSchema).serializeFragment(
node.content,
{ document },
div,
);
return div.innerHTML;
}
export interface RenderChatMarkdownOptions {
/**
* Neutralize INTERNAL links so they render as inert text (no `href`/`target`).
@@ -63,22 +94,32 @@ function neutralizeInternalLinksHook(node: Element): void {
/**
* Render AI markdown to sanitized HTML for read-only display. We reuse the
* app's `markdownToHtml` (the same `marked` pipeline used for paste/import) so
* chat output matches the editor's markdown flavor, then sanitize with
* DOMPurify — LLM output is untrusted, so it must never reach the DOM unsanitized.
* canonical converter (issue #347): markdown -> ProseMirror JSON (the SAME
* `markdownToProseMirrorSync` the editor paste/import path uses, so chat output
* matches the editor's markdown flavor) -> HTML via `markdownToChatHtml`
* (DOMSerializer), then sanitize with DOMPurify — LLM output is untrusted, so it
* must never reach the DOM unsanitized.
*
* `markdownToHtml` can return `string | Promise<string>` (it has async marked
* extensions registered). In practice plain chat markdown resolves
* synchronously, but we guard the Promise case by returning a safe empty string
* for that branch (the caller renders the raw text fallback instead).
* Stays SYNCHRONOUS: both callers render inside React (a memo and a useMemo),
* so the whole pipeline must resolve without awaiting. The converter's sync
* entry makes that possible; on any conversion error we return "" so the caller
* falls back to raw text (the same fallback the old Promise-guard produced).
*/
export function renderChatMarkdown(
markdown: string,
options: RenderChatMarkdownOptions = {},
): string {
if (!markdown) return "";
const html = markdownToHtml(markdown);
if (typeof html !== "string") return "";
let html: string;
try {
// markdown -> canonical PM JSON -> HTML (native DOMParser in the browser;
// jsdom is never bundled — see @docmost/prosemirror-markdown/browser).
html = markdownToChatHtml(markdown);
} catch {
// Malformed/unsupported markdown must not crash the chat render; fall back
// to raw text (empty return -> caller shows the plain-text branch).
return "";
}
if (!options.neutralizeInternalLinks) {
// Internal chat: unchanged behavior, no hook registered.
@@ -0,0 +1,206 @@
import { describe, it, expect } from "vitest";
import { Editor } from "@tiptap/core";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Bold } from "@tiptap/extension-bold";
import { Italic } from "@tiptap/extension-italic";
import { MarkdownClipboard } from "./markdown-clipboard";
/**
* Integration coverage for the async `handlePaste` seam (issue #347). The paste
* conversion moved to `@docmost/prosemirror-markdown`'s browser entry, whose
* `markdownToProseMirror` is async — so `handlePaste` captures the range, claims
* the event (returns true), and dispatches the insert on the next microtask.
* These tests drive that path end to end on a minimal schema (a plain-markdown
* paste whose converted nodes fit paragraph/text/bold/italic), asserting the
* text lands with the right marks and that the raw markdown syntax is consumed
* (recognized as markdown, not inserted literally).
*/
function makeEditor() {
const element = document.createElement("div");
document.body.appendChild(element);
return new Editor({
element,
extensions: [
Document,
Paragraph,
Text,
Bold,
Italic,
MarkdownClipboard.configure({ transformPastedText: true }),
],
content: { type: "doc", content: [{ type: "paragraph" }] },
});
}
// Locate the markdownClipboard plugin and invoke its handlePaste directly with a
// synthetic clipboard event (jsdom has no real paste pipeline). The plugin's
// handlePaste closes over the extension `this`, so calling it off the plugin
// props preserves `this.editor`/`this.options`.
function paste(editor: Editor, text: string): boolean {
const view = editor.view;
const plugin = view.state.plugins.find(
(p: any) => p.props && p.spec?.key,
) as any;
const event = {
clipboardData: {
getData: (type: string) => (type === "text/plain" ? text : ""),
},
} as unknown as ClipboardEvent;
// Find the specific handlePaste that belongs to the markdown clipboard plugin.
const md = view.state.plugins.find(
(p: any) => typeof p.props?.handlePaste === "function",
) as any;
return md.props.handlePaste(view, event, view.state.selection.content());
}
// Flush the microtask queue so the async .then() dispatch runs.
const flush = () => new Promise((r) => setTimeout(r, 0));
describe("MarkdownClipboard handlePaste (async md -> PM)", () => {
it("converts a plain-markdown paste with bold/italic into marked text", async () => {
const editor = makeEditor();
const claimed = paste(editor, "hello **bold** and *italic*");
// The paste is claimed synchronously (async insert follows).
expect(claimed).toBe(true);
await flush();
const json = editor.getJSON();
const text = JSON.stringify(json);
// The raw markdown asterisks are consumed (recognized), not inserted literally.
expect(editor.getText()).not.toContain("**");
expect(editor.getText()).toContain("bold");
expect(editor.getText()).toContain("italic");
// The bold/italic marks materialized.
expect(text).toContain('"bold"');
expect(text).toContain('"italic"');
editor.destroy();
});
it("recognizes a bullet list paste as list structure (not literal '-')", async () => {
// A bullet list is not representable in this minimal schema, so the converter
// output would fail PMNode.fromJSON and the catch inserts raw text. Use a
// paste whose nodes DO fit the schema to assert the happy path instead: two
// paragraphs separated by a blank line.
const editor = makeEditor();
paste(editor, "first para\n\nsecond para");
await flush();
const json = editor.getJSON() as any;
const paras = (json.content || []).filter(
(n: any) => n.type === "paragraph",
);
// Two paragraphs materialized from the blank-line-separated markdown.
expect(paras.length).toBeGreaterThanOrEqual(2);
expect(editor.getText()).toContain("first para");
expect(editor.getText()).toContain("second para");
editor.destroy();
});
it("falls back to raw text when conversion yields nodes the schema lacks", async () => {
// `# heading` converts to a `heading` node absent from this minimal schema,
// so PMNode.fromJSON throws and the catch re-inserts the raw text — the user
// never loses their clipboard content.
const editor = makeEditor();
paste(editor, "# a heading line");
await flush();
// Content is preserved (either as heading text or literal), never dropped.
expect(editor.getText()).toContain("a heading line");
editor.destroy();
});
});
// The async seam captures the target range synchronously, then replaces on the
// next microtask. If the document changed under it between capture and resolve
// (impossible in prod — same microtask — but pinned here), BOTH the success
// (replaceRange) and the fail-open (insertText) branches must fall back to the
// LIVE selection rather than a stale absolute range, so neither clobbers content
// nor throws a RangeError. We force the mid-flight change by dispatching a
// doc-mutating transaction AFTER the synchronous claim but BEFORE flushing the
// microtask that runs the `.then`/`.catch`.
describe("MarkdownClipboard handlePaste — doc-changed-mid-flight guard", () => {
// Replace the whole doc with one paragraph of `text` (synchronous dispatch).
// An empty string yields an empty paragraph (a text node may not be empty).
function seedContent(editor: Editor, text: string) {
editor.commands.setContent({
type: "doc",
content: [
text
? { type: "paragraph", content: [{ type: "text", text }] }
: { type: "paragraph" },
],
});
}
it("success branch: mid-flight doc change routes the paste to the LIVE selection, never the stale range (clobber-proving)", async () => {
// The paste captures a NON-EMPTY range {1,5} (over "AAAA"). Then, before the
// async resolve, the doc GROWS ("MARKER" inserted at the start) and the cursor
// is parked at the doc END. The captured {1,5} is now stale and points INTO
// "MARKER". A WORKING guard replaces at the live (end) selection → MARKER is
// untouched. A BROKEN guard replaces the stale {1,5} → it erases the first
// characters of MARKER (this is what a zero-width `from==to` range could never
// reveal, which is why the earlier version was vacuous).
const editor = makeEditor();
seedContent(editor, "AAAABBBB");
editor.commands.setTextSelection({ from: 1, to: 5 }); // captured range = {1,5}
const claimed = paste(editor, "hello **bold**");
expect(claimed).toBe(true);
// Mid-flight: grow the doc and move the cursor to a KNOWN-safe end position.
editor.view.dispatch(editor.view.state.tr.insertText("MARKER", 1));
const end = editor.state.doc.content.size;
editor.commands.setTextSelection({ from: end, to: end });
await flush();
const text = editor.getText();
// MARKER intact only if the guard used the live selection, not the stale range.
expect(text).toContain("MARKER");
expect(text).toContain("bold");
expect(text).not.toContain("**");
editor.destroy();
});
it("fail-open branch: a mid-flight doc SHRINK makes the stale `to` out of bounds — the guard must avoid a RangeError (throw-proving)", async () => {
// The paste captures a range {1,9} over an 8-char paragraph, then the
// conversion FAILS (`# heading` -> a heading node the minimal schema lacks,
// so PMNode.fromJSON throws -> the fail-open catch runs). Before the reject,
// the doc is SHRUNK to an empty paragraph, so the captured `to` (9) is now far
// past the doc's end. A WORKING guard inserts the raw text at the live (valid)
// selection → "raw heading" lands. A BROKEN guard does insertText(md, 1, 9) on
// a size-2 doc → RangeError, so the dispatch never runs and "raw heading" is
// absent (the assertion reddens). A zero-width/growing-doc setup could never
// push `to` out of bounds, which is why the earlier version was vacuous.
const editor = makeEditor();
seedContent(editor, "AAAABBBB");
editor.commands.setTextSelection({ from: 1, to: 9 }); // captured range = {1,9}
paste(editor, "# raw heading");
// Mid-flight: shrink the doc so the captured `to` = 9 is now out of bounds.
seedContent(editor, "");
await flush();
const text = editor.getText();
// Raw text lands (via the live selection) only if the guard avoided the
// stale, now-out-of-bounds range.
expect(text).toContain("raw heading");
editor.destroy();
});
it("two pastes in flight: neither payload is lost (no data loss)", async () => {
// Prod-unreachable (two paste events are separate macrotasks, and each
// conversion resolves on a microtask before the next), but pinned here: when
// both resolve back-to-back, the second sees the changed doc and inserts at
// the live selection the first left — so the two payloads may INTERLEAVE, but
// neither is dropped. We assert no data loss, not contiguity.
const editor = makeEditor();
paste(editor, "alphaword");
paste(editor, "betaword");
await flush();
const text = editor.getText();
// Neither payload fully dropped (interleaving may split one of them).
expect(text).toContain("alpha");
expect(text).toContain("beta");
editor.destroy();
});
});
@@ -1,5 +1,12 @@
import { describe, it, expect } from "vitest";
import { htmlToMarkdown } from "@docmost/editor-ext";
// Markdown conversion now goes through the canonical package's BROWSER entry
// (issue #347): the same converter the server import/export uses, resolved via
// the `browser` exports condition so it runs on the native `DOMParser` (the
// client jsdom vitest env provides one) with jsdom never bundled.
import {
convertProseMirrorToMarkdown,
markdownToProseMirrorSync,
} from "@docmost/prosemirror-markdown/browser";
import {
normalizeTableColumnWidths,
classifyClipboardSelection,
@@ -175,10 +182,13 @@ describe("classifyClipboardSelection", () => {
// Output-level tests for the table clipboard regression: copying a table must
// yield a real GFM pipe table, NOT one-value-per-line concatenated cells.
// These exercise the actual markdown produced by htmlToMarkdown (the same
// serializer step the clipboardTextSerializer runs), so they pin the OUTPUT
// shape that the classifier-flag tests above do not cover.
describe("table clipboard markdown output (htmlToMarkdown)", () => {
// These exercise the actual markdown produced by convertProseMirrorToMarkdown
// the same serializer step the clipboardTextSerializer now runs (issue #347) —
// so they pin the OUTPUT shape that the classifier-flag tests above do not cover.
// Input is ProseMirror JSON (what the copied slice serializes to), matching the
// clipboardTextSerializer's new call: it wraps the slice content in a synthetic
// `doc` (and the bare-rows case in a `table`) and calls the converter.
describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () => {
// Trim each line and drop blanks so structural assertions are whitespace-robust.
function lines(md: string): string[] {
return md
@@ -188,10 +198,10 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
}
// A GFM separator row like "| --- | --- |" (any number of columns), tolerant
// of the padding turndown emits.
// of the padding the serializer emits.
function isSeparatorRow(line: string): boolean {
const compact = line.replace(/\s+/g, "");
return /^\|(?:-{3,}\|)+$/.test(compact);
return /^\|(?::?-{2,}:?\|)+$/.test(compact);
}
// Split a pipe-delimited row into trimmed cell values.
@@ -203,42 +213,33 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
.map((c) => c.trim());
}
it("serializes a header-less partial cell selection (bare rows) as a valid GFM pipe table", () => {
// Mirror the serializer's `wrapBareRows` branch exactly: bare <tr> nodes are
// wrapped in <table><tbody> and htmlToMarkdown(div.innerHTML) is called.
// See markdown-clipboard.ts clipboardTextSerializer:
// const table = document.createElement("table");
// const tbody = document.createElement("tbody");
// tbody.appendChild(fragment); table.appendChild(tbody);
// div.appendChild(table);
// return htmlToMarkdown(div.innerHTML);
const div = document.createElement("div");
const table = document.createElement("table");
const tbody = document.createElement("tbody");
for (const [c1, c2] of [
["a", "b"],
["c", "d"],
]) {
const tr = document.createElement("tr");
const td1 = document.createElement("td");
td1.textContent = c1;
const td2 = document.createElement("td");
td2.textContent = c2;
tr.appendChild(td1);
tr.appendChild(td2);
tbody.appendChild(tr);
}
table.appendChild(tbody);
div.appendChild(table);
const cell = (t: string) => ({
type: "tableCell",
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
});
const headerCell = (t: string) => ({
type: "tableHeader",
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
});
const row = (nodes: any[]) => ({ type: "tableRow", content: nodes });
const md = htmlToMarkdown(div.innerHTML);
it("serializes a header-less partial cell selection (bare rows) as a valid GFM pipe table", () => {
// Mirror the serializer's `wrapBareRows` branch: bare tableRow nodes are
// wrapped in a synthetic `table` and convertProseMirrorToMarkdown is called
// (see markdown-clipboard.ts clipboardTextSerializer).
const rows = [
row([cell("a"), cell("b")]),
row([cell("c"), cell("d")]),
];
const md = convertProseMirrorToMarkdown({
type: "doc",
content: [{ type: "table", content: rows }],
});
const ls = lines(md);
// Valid GFM: a header/data separator row is present (an empty header is
// synthesized by the GFM turndown plugin for a header-less table — fine).
// Valid GFM: a header/data separator row is present.
expect(ls.some(isSeparatorRow)).toBe(true);
// NOT the old broken "one value per line" shape: every line is pipe-delimited
// and no line is a bare cell value on its own.
// NOT the old broken "one value per line" shape: every line is pipe-delimited.
expect(ls.every((l) => l.includes("|"))).toBe(true);
expect(md).not.toMatch(/^\s*(a|b|c|d)\s*$/m);
// The cell values land in real pipe-delimited data rows.
@@ -248,39 +249,21 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
});
it("serializes a whole table with a header row as a proper GFM table (headline regression)", () => {
// Mirror the serializer's non-wrap branch: the full <table> node is appended
// directly (div.appendChild(fragment)) and htmlToMarkdown(div.innerHTML) runs.
const div = document.createElement("div");
const table = document.createElement("table");
const thead = document.createElement("thead");
const headerRow = document.createElement("tr");
for (const h of ["Name", "Age"]) {
const th = document.createElement("th");
th.textContent = h;
headerRow.appendChild(th);
}
thead.appendChild(headerRow);
table.appendChild(thead);
const tbody = document.createElement("tbody");
for (const [name, age] of [
["Alice", "30"],
["Bob", "25"],
]) {
const tr = document.createElement("tr");
const td1 = document.createElement("td");
td1.textContent = name;
const td2 = document.createElement("td");
td2.textContent = age;
tr.appendChild(td1);
tr.appendChild(td2);
tbody.appendChild(tr);
}
table.appendChild(tbody);
div.appendChild(table);
const md = htmlToMarkdown(div.innerHTML);
// Mirror the serializer's non-wrap branch: the full `table` node is the
// slice content and convertProseMirrorToMarkdown runs on it.
const md = convertProseMirrorToMarkdown({
type: "doc",
content: [
{
type: "table",
content: [
row([headerCell("Name"), headerCell("Age")]),
row([cell("Alice"), cell("30")]),
row([cell("Bob"), cell("25")]),
],
},
],
});
const ls = lines(md);
// Proper GFM structure: separator row + all rows pipe-delimited.
@@ -296,3 +279,146 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
expect(md).not.toMatch(/^\s*(Name|Age|Alice|Bob|30|25)\s*$/m);
});
});
// #347 acceptance: pasting CANONICAL markdown yields the SAME nodes the server
// import produces for the same text. The paste path calls markdownToProseMirror
// (the package browser entry) — the identical converter the server import uses —
// so asserting the converter (via the browser entry, on the native DOMParser)
// recognizes each canon form pins the paste-parity guarantee. These forms were
// NOT recognized by the old editor-ext marked layer the paste used before.
describe("canonical markdown paste recognition (browser entry parity)", () => {
// Collect every node type present in a doc (recursively).
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
if (!n || typeof n !== "object") return set;
if (n.type) set.add(n.type);
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
return set;
};
const findNode = (n: any, type: string): any => {
if (!n || typeof n !== "object") return undefined;
if (n.type === type) return n;
if (Array.isArray(n.content)) {
for (const c of n.content) {
const hit = findNode(c, type);
if (hit) return hit;
}
}
return undefined;
};
const allText = (n: any): string => {
if (!n || typeof n !== "object") return "";
if (typeof n.text === "string") return n.text;
if (Array.isArray(n.content)) return n.content.map(allText).join("");
return "";
};
it("^[…] inline footnote -> footnoteReference + footnotesList", () => {
const doc = markdownToProseMirrorSync("Body^[a note here].");
const types = collectTypes(doc);
expect(types.has("footnoteReference")).toBe(true);
expect(types.has("footnotesList")).toBe(true);
expect(types.has("footnoteDefinition")).toBe(true);
});
it('<!--img {…}--> attached image comment -> image with align', () => {
const doc = markdownToProseMirrorSync(
'![alt](/files/x.png) <!--img {"align":"left"}-->',
);
const img = findNode(doc, "image");
expect(img).toBeTruthy();
expect(img.attrs?.align).toBe("left");
expect(img.attrs?.src).toBe("/files/x.png");
});
it("> [!type] Obsidian callout -> callout node with type", () => {
const doc = markdownToProseMirrorSync("> [!warning]\n> be careful");
const callout = findNode(doc, "callout");
expect(callout).toBeTruthy();
expect(callout.attrs?.type).toBe("warning");
expect(allText(callout)).toContain("be careful");
});
it("$…$ inline math -> mathInline node", () => {
const doc = markdownToProseMirrorSync("Euler: $e^{i\\pi}+1=0$ done");
const math = findNode(doc, "mathInline");
expect(math).toBeTruthy();
expect(math.attrs?.text).toContain("e^{i\\pi}");
});
it("==…== highlight -> highlight mark", () => {
const doc = markdownToProseMirrorSync("A ==marked== word");
const marked = findNode(doc, "text");
// The highlighted run carries a `highlight` mark somewhere in the doc.
const hasHighlight = (n: any): boolean => {
if (!n || typeof n !== "object") return false;
if (
n.type === "text" &&
(n.marks || []).some((m: any) => m.type === "highlight")
)
return true;
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
};
expect(marked).toBeTruthy();
expect(hasHighlight(doc)).toBe(true);
});
it("<!--subpages--> standalone comment -> subpages node", () => {
const doc = markdownToProseMirrorSync("intro\n\n<!--subpages-->\n\nafter");
expect(collectTypes(doc).has("subpages")).toBe(true);
});
});
// #347 negatives: plain text carrying markdown-LIKE punctuation must NOT be
// silently converted/mangled (currency, bare `==`, a `[^1]` reference form).
describe("plain-text paste negatives (no phantom conversion)", () => {
const findNode = (n: any, type: string): any => {
if (!n || typeof n !== "object") return undefined;
if (n.type === type) return n;
if (Array.isArray(n.content)) {
for (const c of n.content) {
const hit = findNode(c, type);
if (hit) return hit;
}
}
return undefined;
};
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
if (!n || typeof n !== "object") return set;
if (n.type) set.add(n.type);
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
return set;
};
const allText = (n: any): string => {
if (!n || typeof n !== "object") return "";
if (typeof n.text === "string") return n.text;
if (Array.isArray(n.content)) return n.content.map(allText).join("");
return "";
};
it("currency `$5 and $10` is NOT turned into math", () => {
const doc = markdownToProseMirrorSync("It costs $5 and $10 total");
expect(findNode(doc, "mathInline")).toBeFalsy();
expect(allText(doc)).toContain("$5 and $10");
});
it("a lone `==` is NOT turned into a highlight", () => {
const doc = markdownToProseMirrorSync("compare a == b in code");
const hasHighlight = (n: any): boolean => {
if (!n || typeof n !== "object") return false;
if (
n.type === "text" &&
(n.marks || []).some((m: any) => m.type === "highlight")
)
return true;
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
};
expect(hasHighlight(doc)).toBe(false);
expect(allText(doc)).toContain("== b");
});
it("a `[^1]` reference form (no `^[`) is NOT turned into a footnote", () => {
const doc = markdownToProseMirrorSync("see note [^1] for details");
expect(collectTypes(doc).has("footnoteReference")).toBe(false);
expect(allText(doc)).toContain("[^1]");
});
});
@@ -1,15 +1,23 @@
// adapted from: https://github.com/aguingand/tiptap-markdown/blob/main/src/extensions/tiptap/clipboard.js - MIT
import { Extension } from "@tiptap/core";
import { Plugin, PluginKey, TextSelection } from "@tiptap/pm/state";
import { DOMParser, DOMSerializer, Fragment, Slice } from "@tiptap/pm/model";
import { DOMParser, DOMSerializer, Fragment, Slice, Node as PMNode } from "@tiptap/pm/model";
import { find } from "linkifyjs";
import {
markdownToHtml,
htmlToMarkdown,
canonicalizeFootnotes,
FOOTNOTES_LIST_NAME,
FOOTNOTE_REFERENCE_NAME,
} from "@docmost/editor-ext";
// Markdown <-> ProseMirror conversion now lives ONLY in the canonical
// `@docmost/prosemirror-markdown` package (issue #347). The BROWSER entry uses
// the native `DOMParser` for its HTML->DOM stage (jsdom stays out of the client
// bundle) while producing the SAME nodes the server import does — so a paste of
// canonical markdown (`^[…]`, `<!--img …-->`, `> [!type]`, `$…$`, `==…==`,
// standalone comments) is recognized identically to import.
import {
markdownToProseMirror,
convertProseMirrorToMarkdown,
} from "@docmost/prosemirror-markdown/browser";
import type { Schema } from "@tiptap/pm/model";
export const MarkdownClipboard = Extension.create({
@@ -39,25 +47,24 @@ export const MarkdownClipboard = Extension.create({
classifyClipboardSelection(topLevelNodes);
if (!asMarkdown) return null;
const div = document.createElement("div");
const serializer = DOMSerializer.fromSchema(this.editor.schema);
const fragment = serializer.serializeFragment(slice.content);
// Convert the copied selection to Markdown through the canonical
// package (issue #347), the SAME serializer the server export uses,
// so a copied table/list matches the on-disk markdown form. The
// converter takes a ProseMirror `doc` JSON, so wrap the slice's
// top-level content in a synthetic doc.
const content = slice.content.toJSON() as any[];
if (wrapBareRows) {
// A partial table cell-selection serializes to bare <tr> nodes
// (prosemirror-tables returns the whole `table` node only when the
// entire table is selected). Bare <tr> would be foster-parented
// away by the HTML parser inside htmlToMarkdown, so wrap them in
// <table><tbody> first for the GFM turndown rule to detect them.
const table = document.createElement("table");
const tbody = document.createElement("tbody");
tbody.appendChild(fragment);
table.appendChild(tbody);
div.appendChild(table);
} else {
div.appendChild(fragment);
// A partial table cell-selection serializes to bare `tableRow`
// nodes (prosemirror-tables yields the whole `table` node only for
// a full-table selection). The converter's table case expects a
// `table` wrapper, so wrap the bare rows in one — mirroring the old
// <table><tbody> wrap that the HTML->markdown step needed.
return convertProseMirrorToMarkdown({
type: "doc",
content: [{ type: "table", content }],
});
}
return htmlToMarkdown(div.innerHTML);
return convertProseMirrorToMarkdown({ type: "doc", content });
},
handlePaste: (view, event, slice) => {
if (!event.clipboardData) {
@@ -95,37 +102,115 @@ export const MarkdownClipboard = Extension.create({
}
}
const { tr } = view.state;
const { from, to } = view.state.selection;
const schema = this.editor.schema;
// Capture the target range NOW. markdownToProseMirror RETURNS A
// PROMISE (kept async only for the Node consumers' contract; the
// conversion pipeline itself is synchronous), so the actual replace
// happens on the next microtask. No user input can interleave a
// microtask, so the state is unchanged when we dispatch — but we
// still re-read the live state before replacing and, if the doc did
// change under us, fall back to the live selection rather than the
// captured (now-stale) range.
const from = view.state.selection.from;
const to = view.state.selection.to;
const startDoc = view.state.doc;
const md = text.replace(/\n+$/, "");
const parsed = markdownToHtml(text.replace(/\n+$/, ""));
const body = elementFromString(parsed);
normalizeTableColumnWidths(body);
void markdownToProseMirror(md)
.then((doc) => {
if (view.isDestroyed) return;
// Canonical PM-JSON -> HTML via the LIVE editor schema, then
// reuse the UNCHANGED downstream seam (normalizeTableColumnWidths
// + parseSlice + canonicalizePastedFootnotes). The JSON->HTML->
// JSON hop is lossless (same schema both directions); it lets the
// existing paste-insertion logic stay byte-identical — only the
// SOURCE of the markdown conversion changed (issue #347 guardrail:
// no converter logic in the client, only a call into the package).
const node = PMNode.fromJSON(schema, doc);
const div = document.createElement("div");
DOMSerializer.fromSchema(schema).serializeFragment(
node.content,
{ document },
div,
);
const parsedSlice = DOMParser.fromSchema(
this.editor.schema,
).parseSlice(body, {
preserveWhitespace: true,
});
const body = elementFromString(div.innerHTML);
normalizeTableColumnWidths(body);
// A markdown paste builds its ProseMirror fragment directly (DOM ->
// parseSlice), bypassing the editor's footnoteSyncPlugin, which never
// reorders an existing list. So a pasted markdown block whose footnote
// definitions are out of order (or contains orphan defs) would be
// stored out of order. Canonicalize the self-contained pasted block so
// its footnotes come out reference-ordered, deduped and orphan-free
// (issue #228). See canonicalizePastedFootnotes for why this is scoped
// to whole-block pastes that carry their own footnotesList.
const contentNodes = canonicalizePastedFootnotes(
parsedSlice,
this.editor.schema,
);
const parsedSlice = DOMParser.fromSchema(schema).parseSlice(
body,
{ preserveWhitespace: true },
);
tr.replaceRange(from, to, contentNodes);
const insertEnd = tr.mapping.map(from, 1);
tr.setSelection(TextSelection.near(tr.doc.resolve(Math.max(from, insertEnd - 2)), -1));
tr.setMeta('paste', true)
view.dispatch(tr);
// A markdown paste builds its ProseMirror fragment directly (DOM
// -> parseSlice), bypassing the editor's footnoteSyncPlugin, which
// never reorders an existing list. So a pasted markdown block whose
// footnote definitions are out of order (or contains orphan defs)
// would be stored out of order. Canonicalize the self-contained
// pasted block so its footnotes come out reference-ordered, deduped
// and orphan-free (issue #228). See canonicalizePastedFootnotes for
// why this is scoped to whole-block pastes that carry their own
// footnotesList.
const contentNodes = canonicalizePastedFootnotes(
parsedSlice,
schema,
);
// Target the captured range (normally still valid — same
// microtask). If the doc changed under us since capture, the
// captured absolute from/to are stale, so fall back to the live
// selection rather than StepMap-mapping the old range.
const tr = view.state.tr;
let mappedFrom = from;
let mappedTo = to;
if (view.state.doc !== startDoc) {
// Defensive: if the doc changed under us, fall back to the
// current selection rather than a stale absolute range.
mappedFrom = view.state.selection.from;
mappedTo = view.state.selection.to;
}
tr.replaceRange(mappedFrom, mappedTo, contentNodes);
const insertEnd = tr.mapping.map(mappedFrom, 1);
tr.setSelection(
TextSelection.near(
tr.doc.resolve(Math.max(mappedFrom, insertEnd - 2)),
-1,
),
);
tr.setMeta("paste", true);
view.dispatch(tr);
})
.catch((err) => {
// Fail-open: a conversion error must not swallow the paste
// silently in a way that loses the text. We already claimed the
// event (returned true), so re-insert the raw text as a plain
// paragraph so the user never loses their clipboard content.
// Log it: this catch covers BOTH the converter and the success
// `.then` body (e.g. PMNode.fromJSON throwing on a schema drift
// between the canonical package and the live editor schema), so a
// silent degrade to raw text would otherwise be an invisible,
// non-reproducible regression ("my table pasted as text").
console.error(
"markdown paste conversion failed, inserting raw text",
err,
);
if (view.isDestroyed) return;
const tr = view.state.tr;
// Same guard the success path uses: if the doc changed under us
// since the range was captured (normally never — same microtask),
// the captured absolute from/to are stale and would throw a
// RangeError here (an unhandled rejection on a hot paste path).
// Fall back to the live selection instead of a stale range.
if (view.state.doc !== startDoc) {
const sel = view.state.selection;
tr.insertText(md, sel.from, sel.to);
} else {
tr.insertText(md, from, to);
}
tr.setMeta("paste", true);
view.dispatch(tr);
});
// Claim the paste: we insert asynchronously above.
return true;
},
// Strip trailing whitespace-only paragraphs from pasted content.
@@ -33,10 +33,11 @@ vi.mock("@/lib/local-emitter.ts", () => ({
default: { emit: (...args: unknown[]) => localEmitMock(...args) },
}));
// htmlToMarkdown just echoes the editor HTML so each test controls the markdown
// purely via the fake page editor's getHTML().
vi.mock("@docmost/editor-ext", () => ({
htmlToMarkdown: (html: string) => html,
// convertProseMirrorToMarkdown echoes a marker carried on the fake editor's
// getJSON() doc, so each test controls the markdown purely via the fake page
// editor (issue #347: the hook now serializes editor JSON through the package).
vi.mock("@docmost/prosemirror-markdown/browser", () => ({
convertProseMirrorToMarkdown: (doc: { __md?: string }) => doc?.__md ?? "",
}));
const notificationsShowMock = vi.fn();
@@ -53,10 +54,12 @@ import { useGeneratePageTitle } from "./use-generate-page-title.ts";
// --- Test helpers -------------------------------------------------------------
function makePageEditor(pageId: string, html = "<p>content</p>"): Editor {
function makePageEditor(pageId: string, md = "content"): Editor {
return {
isDestroyed: false,
getHTML: () => html,
// The mocked convertProseMirrorToMarkdown reads `__md` back off this doc,
// so `md` is exactly the markdown the hook will send to the title service.
getJSON: () => ({ type: "doc", __md: md }),
storage: { pageId },
} as unknown as Editor;
}
@@ -3,7 +3,7 @@ import { useMutation } from "@tanstack/react-query";
import { useAtomValue } from "jotai";
import { notifications } from "@mantine/notifications";
import { useTranslation } from "react-i18next";
import { htmlToMarkdown } from "@docmost/editor-ext";
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser";
import {
pageEditorAtom,
titleEditorAtom,
@@ -49,7 +49,9 @@ export function useGeneratePageTitle(pageId: string) {
mutationFn: async () => {
if (!pageEditor || pageEditor.isDestroyed) return;
const markdown = htmlToMarkdown(pageEditor.getHTML()).trim();
// Serialize the live editor content to markdown through the canonical
// converter (issue #347), matching the on-disk/export markdown form.
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON()).trim();
if (!markdown) {
notifications.show({ message: t("The note is empty"), color: "yellow" });
return;
@@ -37,7 +37,7 @@ import { useTreeMutation } from "@/features/page/tree/hooks/use-tree-mutation.ts
import { PageWidthToggle } from "@/features/user/components/page-width-pref.tsx";
import { Trans, useTranslation } from "react-i18next";
import ExportModal from "@/components/common/export-modal";
import { htmlToMarkdown } from "@docmost/editor-ext";
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser";
import {
pageEditorAtom,
yjsConnectionStatusAtom,
@@ -199,8 +199,9 @@ function PageActionMenu({ readOnly }: PageActionMenuProps) {
const handleCopyAsMarkdown = () => {
if (!pageEditor) return;
const html = pageEditor.getHTML();
const markdown = htmlToMarkdown(html);
// Copy the page as canonical markdown through the shared converter (issue
// #347), so "Copy as markdown" matches the server export byte-for-byte.
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON());
const title = page?.title ? `# ${page.title}\n\n` : "";
clipboard.copy(`${title}${markdown}`);
notifications.show({ message: t("Copied") });
@@ -1,4 +1,5 @@
import { markdownToHtml, encodeHtmlEmbedSource } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { encodeHtmlEmbedSource } from '@docmost/editor-ext';
import { htmlToJson } from '../../../collaboration/collaboration.util';
import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
@@ -10,13 +11,12 @@ import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
*
* The block renders inside a sandboxed iframe, so this is not an XSS surface;
* this exercises the REAL server import conversion path that ImportService uses
* (`markdownToHtml` then `htmlToJson`; `processHTML` adds only a cheerio
* link/iframe normalize pass which does not touch htmlEmbed divs) and asserts
* that such a node is DETECTED and STRIPPABLE — so the share read path's
* (`markdownToProseMirror`, the canonical converter — issue #345/#347) and
* asserts that such a node is DETECTED and STRIPPABLE — so the share read path's
* master-toggle strip can remove it when the workspace toggle is OFF.
*/
describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTML', () => {
it('round-trips through markdownToHtml -> htmlToJson and is DETECTED (base64 data-source)', async () => {
it('round-trips through markdownToProseMirror and is DETECTED (base64 data-source)', async () => {
const source = '<script>steal()</script>';
const encoded = encodeHtmlEmbedSource(source);
const md = [
@@ -27,12 +27,9 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
'World',
].join('\n');
const html = await markdownToHtml(md);
// marked preserves the raw block-level div verbatim.
expect(html).toContain('data-type="htmlEmbed"');
const json = htmlToJson(html);
// The div parses into a real htmlEmbed node carrying the decoded source.
// The canonical importer parses the raw block-level div into a real
// htmlEmbed node carrying the decoded source.
const json = await markdownToProseMirror(md);
expect(hasHtmlEmbedNode(json)).toBe(true);
// Because it is detected, the share master-toggle strip can remove it.
@@ -59,8 +56,7 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
// therefore stripping) does not depend on the source being well-formed, so
// the bypass cannot be hidden by sending a malformed data-source.
const md = `<div data-type="htmlEmbed" data-source="&lt;script&gt;x&lt;/script&gt;"></div>`;
const html = await markdownToHtml(md);
const json = htmlToJson(html);
const json = await markdownToProseMirror(md);
expect(hasHtmlEmbedNode(json)).toBe(true);
expect(hasHtmlEmbedNode(stripHtmlEmbedNodes(json))).toBe(false);
});
@@ -0,0 +1,125 @@
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import { PROMPT_TOOL_NAMES } from './ai-chat.prompt';
// The real shared registry, imported from source (same approach as the
// SHARED_TOOL_SPECS contract spec) so tool names are validated against exactly
// what @docmost/mcp ships.
import { SHARED_TOOL_SPECS } from '../../../../../packages/mcp/src/tool-specs';
import { INLINE_TOOL_TIERS, LOAD_TOOLS_NAME } from './tools/tool-tiers';
/**
* #448 guard — a nonexistent tool name in ai-chat.prompt.ts must fail a test.
*
* The in-app prompt refers to a handful of tools BY NAME in its guidance notes
* (e.g. PAGE_CHANGED_NOTE tells the agent to re-read via getPage and edit via
* editPageText/patchNode/insertNode/deleteNode). Before #448 those names were
* hard-coded inline with NO guard, so renaming a tool left the agent stale
* instructions and nothing failed.
*
* APPROACH — substitution + a precise source scan:
* 1. The names now flow through the exported `PROMPT_TOOL_NAMES` const; this
* test asserts every value there is a REAL in-app tool.
* 2. A precise scan of the two guidance-note string literals in the source
* catches any BARE tool-name token added directly (bypassing the const):
* every camelCase token in those notes must be either a real tool name or an
* explicitly-allowlisted ordinary English/camelCase word.
*
* The scan is deliberately narrow (only the guidance notes, only camelCase
* tokens) so it never false-positives on prose, and the allowlist of non-tool
* words is tiny and explicit.
*/
// The authoritative set of real in-app tool names: shared-registry inAppKeys +
// per-layer INLINE tool keys + the loadTools meta-tool.
const VALID_TOOL_NAMES = new Set<string>([
...Object.values(SHARED_TOOL_SPECS).map((s) => s.inAppKey),
...Object.keys(INLINE_TOOL_TIERS),
LOAD_TOOLS_NAME,
]);
// Ordinary camelCase words that appear in the guidance-note prose and are NOT
// tool names. Keep this list minimal and explicit — anything camelCase in a note
// that is neither a real tool nor here fails the scan.
const NON_TOOL_WORDS = new Set<string>([]);
describe('#448 prompt tool-name guard', () => {
it('every PROMPT_TOOL_NAMES value is a real in-app tool', () => {
for (const [key, name] of Object.entries(PROMPT_TOOL_NAMES)) {
expect(typeof name).toBe('string');
expect(VALID_TOOL_NAMES.has(name)).toBe(true);
// Sanity: the const key and its value are the same token (the const is a
// name->name map used purely to route mentions through one guarded place).
expect(key).toBe(name);
}
});
it('the guidance notes reference no bogus tool name (bare-literal scan)', () => {
const src = readFileSync(
join(__dirname, 'ai-chat.prompt.ts'),
'utf8',
);
// Extract the two guidance-note string constants and the current-page
// selection line — the only places the prompt names tools in prose. Each is
// a `const NAME =` ... `;` block; we scan their raw text for camelCase
// tokens. (Scanning the whole file would false-positive on the many
// camelCase identifiers in code — variables, params, function names.)
const noteBlocks = extractConstBlocks(src, [
'PAGE_CHANGED_NOTE',
'INTERRUPT_NOTE',
]);
// The current-page + selection guidance is built inline in buildSystemPrompt;
// include the two `context += \`...\`` template lines that mention tools.
const contextLines = src
.split('\n')
.filter((l) => l.includes('context +=') && l.includes('getCurrentPage'))
.join('\n');
// Neutralize string-literal escape sequences (\n, \t, ...) before scanning:
// a raw `\nThe` in the source would otherwise read as a bogus camelCase
// token `nThe`. Replace any backslash-escape with a space.
const scanText = (noteBlocks + '\n' + contextLines).replace(/\\./g, ' ');
expect(scanText.length).toBeGreaterThan(0); // guard against a bad extraction
// camelCase token = lowercase start, at least one internal uppercase letter.
const tokens = new Set(scanText.match(/\b[a-z][a-zA-Z0-9]*[A-Z][a-zA-Z0-9]*\b/g) ?? []);
const offenders = [...tokens].filter(
(t) => !VALID_TOOL_NAMES.has(t) && !NON_TOOL_WORDS.has(t),
);
expect(offenders).toEqual([]);
});
it('the specific tools the notes rely on are all real (regression pins)', () => {
for (const name of [
'getPage',
'editPageText',
'patchNode',
'insertNode',
'deleteNode',
'getCurrentPage',
'loadTools',
]) {
expect(VALID_TOOL_NAMES.has(name)).toBe(true);
}
});
});
/**
* Extract the raw text of one or more top-level `const NAME = ... ;` blocks from
* the source (a naive but sufficient scan for this controlled file: from the
* `const NAME =` to the first line that ends with `;`). Returns the blocks
* concatenated.
*/
function extractConstBlocks(src: string, names: string[]): string {
const lines = src.split('\n');
const out: string[] = [];
for (const name of names) {
const start = lines.findIndex((l) => l.trimStart().startsWith(`const ${name} =`));
if (start < 0) continue;
for (let i = start; i < lines.length; i++) {
out.push(lines[i]);
if (lines[i].trimEnd().endsWith(';')) break;
}
}
return out.join('\n');
}
+29 -5
View File
@@ -2,6 +2,30 @@ import { Workspace } from '@docmost/db/types/entity.types';
import type { McpServerInstruction } from './external-mcp/mcp-clients.service';
import { CORE_TOOL_KEYS, type ToolCatalogEntry } from './tools/tool-tiers';
/**
* The in-app tool names this prompt refers to BY NAME in its guidance notes
* (issue #448). Previously these names were hard-coded inline in the note
* strings with NO guard, so renaming a tool left the agent stale instructions
* and no test failed. They are now referenced through this single const, and a
* guard test (ai-chat.prompt.tool-names.spec.ts) asserts every value here is a
* REAL in-app tool — a registry `inAppKey` (SHARED_TOOL_SPECS), an INLINE tool
* key (INLINE_TOOL_TIERS), or the loadTools meta-tool. Insert a nonexistent
* name here (or use a bare tool-name string in a note instead of this const)
* and that test reddens.
*
* `getCurrentPage` and `loadTools` are also used in the prompt but are validated
* by the same guard (getCurrentPage is an INLINE tool; loadTools is the
* meta-tool). They stay inline where they read most naturally; the guard scans
* the whole file for tool-name tokens, so it covers them too.
*/
export const PROMPT_TOOL_NAMES = {
getPage: 'getPage',
editPageText: 'editPageText',
patchNode: 'patchNode',
insertNode: 'insertNode',
deleteNode: 'deleteNode',
} as const;
/**
* Default agent persona used when the admin has not configured a custom system
* prompt (`settings.ai.provider.systemPrompt`).
@@ -91,15 +115,15 @@ const PAGE_CHANGED_NOTE =
'NOTE: The user edited the open page AFTER your last response in this ' +
'conversation, so any copy of that page you produced or remember from earlier ' +
'is now STALE and must not be reused. Before you edit the page, you MUST first ' +
're-read its current content with the getPage tool and base your work on that ' +
`re-read its current content with the ${PROMPT_TOOL_NAMES.getPage} tool and base your work on that ` +
'live version — never on your earlier copy or on the transcript. The unified ' +
'diff below shows exactly what the user changed since you last spoke (lines ' +
'starting with "-" were removed, "+" were added) and is the source of truth. ' +
'Preserve every one of the user\'s edits: make the smallest change that ' +
'satisfies the request using the targeted edit tools (editPageText, patchNode, ' +
'insertNode, deleteNode) rather than replacing the whole page, and do not ' +
'revert, drop, or overwrite anything the user changed. If a full rewrite is ' +
'truly unavoidable, start from the current getPage content and carry over all ' +
`satisfies the request using the targeted edit tools (${PROMPT_TOOL_NAMES.editPageText}, ${PROMPT_TOOL_NAMES.patchNode}, ` +
`${PROMPT_TOOL_NAMES.insertNode}, ${PROMPT_TOOL_NAMES.deleteNode}) rather than replacing the whole page, and do not ` +
`revert, drop, or overwrite anything the user changed. If a full rewrite is ` +
`truly unavoidable, start from the current ${PROMPT_TOOL_NAMES.getPage} content and carry over all ` +
'of the user\'s edits.';
/**
@@ -75,7 +75,7 @@ const LABELS: Record<
searchPages: 'Searched pages',
getPage: 'Read page',
createPage: 'Created page',
updatePageContent: 'Updated page',
updatePageMarkdown: 'Updated page',
renamePage: 'Renamed page',
movePage: 'Moved page',
deletePage: 'Deleted page (to trash)',
@@ -96,7 +96,7 @@ const LABELS: Record<
searchPages: 'Искал по страницам',
getPage: 'Прочитал страницу',
createPage: 'Создал страницу',
updatePageContent: 'Обновил страницу',
updatePageMarkdown: 'Обновил страницу',
renamePage: 'Переименовал страницу',
movePage: 'Переместил страницу',
deletePage: 'Удалил страницу (в корзину)',
@@ -1,6 +1,17 @@
import { AiChatToolsService } from './ai-chat-tools.service';
import * as loader from './docmost-client.loader';
import type { DocmostClientLike } from './docmost-client.loader';
// Test-double type for the loopback client. `DocmostClientLike` is now DERIVED
// from the real `DocmostClient` (issue #446), so its method RETURN types are the
// concrete client shapes. These stubs deliberately return minimal recording
// shapes (e.g. `{ ok: true }`), which no longer satisfy those concrete returns —
// so the doubles are typed with the same method NAMES but loose async returns.
// Each is still cast to `DocmostClientLike` at the (return-erased) mock site, so
// the positional-call type-safety on the PRODUCTION client is unaffected.
type FakeDocmostClient = Partial<
Record<keyof DocmostClientLike, (...args: any[]) => Promise<any>>
>;
// The real zod-agnostic shared tool-spec registry. It has no runtime deps, so
// importing the TS source directly keeps these mocks honest: the service builds
// the shared tools from exactly the specs the package ships, not a hand-stub.
@@ -12,7 +23,15 @@ import { SHARED_TOOL_SPECS } from '../../../../../../packages/mcp/src/tool-specs
// sync.
const mockLoaded = (DocmostClient: loader.DocmostClientCtor) => ({
DocmostClient,
sharedToolSpecs: SHARED_TOOL_SPECS as Record<string, loader.SharedToolSpec>,
sharedToolSpecs: SHARED_TOOL_SPECS as unknown as Record<string, loader.SharedToolSpec>,
// Pure no-network draw.io helpers (#424). Type-correct stubs: these tests
// never execute the drawioShapes / drawioGuide tool bodies.
searchShapes: (() => []) as unknown as loader.SearchShapesFn,
getGuideSection: (() => ({
section: 'index',
content: '',
sections: [],
})) as unknown as loader.GetGuideSectionFn,
});
/**
@@ -31,7 +50,7 @@ describe('AiChatToolsService deletePage guardrail (H4)', () => {
// Minimal fake DocmostClient: only the write methods the tools touch need to
// exist; deletePage records its args. No network, no ESM import.
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
deletePage: (...args: unknown[]) => {
deletePageCalls.push(args);
return Promise.resolve({ success: true });
@@ -160,7 +179,7 @@ describe('AiChatToolsService deletePage guardrail (H4)', () => {
describe('AiChatToolsService expanded toolset guardrails', () => {
// No client method is invoked here — every assertion is on tool presence /
// input schema — so an empty fake client is sufficient.
const fakeClient: Partial<DocmostClientLike> = {};
const fakeClient: FakeDocmostClient = {};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
@@ -264,8 +283,9 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
const patchNodeCalls: unknown[][] = [];
const insertNodeCalls: unknown[][] = [];
const updatePageJsonCalls: unknown[][] = [];
const updatePageCalls: unknown[][] = [];
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
patchNode: (...args: unknown[]) => {
patchNodeCalls.push(args);
return Promise.resolve({ ok: true });
@@ -278,6 +298,11 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
updatePageJsonCalls.push(args);
return Promise.resolve({ ok: true });
},
// Backs the plain-Markdown full-body-replace tool updatePageMarkdown (#411).
updatePage: (...args: unknown[]) => {
updatePageCalls.push(args);
return Promise.resolve({ success: true });
},
};
const tokenServiceStub = {
@@ -291,6 +316,7 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
patchNodeCalls.length = 0;
insertNodeCalls.length = 0;
updatePageJsonCalls.length = 0;
updatePageCalls.length = 0;
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue(
mockLoaded(function () {
return fakeClient as DocmostClientLike;
@@ -329,23 +355,32 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
content: [{ type: 'text', text: 'Hello' }],
};
it('patchNode parses a JSON-string node and forwards it as an object', async () => {
it('patchNode parses a JSON-string node and forwards it as { node } (object)', async () => {
const tools = await buildTools();
await tools.patchNode.execute(
{ pageId: 'p1', nodeId: 'n1', node: JSON.stringify(NODE_OBJ) } as never,
{} as never,
);
expect(patchNodeCalls).toHaveLength(1);
expect(patchNodeCalls[0]).toEqual(['p1', 'n1', NODE_OBJ]);
// #413: the 3rd arg is now the XOR input { markdown?, node? }.
expect(patchNodeCalls[0]).toEqual([
'p1',
'n1',
{ markdown: undefined, node: NODE_OBJ },
]);
});
it('patchNode passes an object node through unchanged', async () => {
it('patchNode passes an object node through unchanged inside { node }', async () => {
const tools = await buildTools();
await tools.patchNode.execute(
{ pageId: 'p1', nodeId: 'n1', node: NODE_OBJ } as never,
{} as never,
);
expect(patchNodeCalls[0]).toEqual(['p1', 'n1', NODE_OBJ]);
expect(patchNodeCalls[0]).toEqual([
'p1',
'n1',
{ markdown: undefined, node: NODE_OBJ },
]);
});
it('patchNode throws the documented message on invalid JSON string', async () => {
@@ -359,7 +394,7 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
expect(patchNodeCalls).toHaveLength(0);
});
it('insertNode parses a JSON-string node and forwards it as an object', async () => {
it('insertNode parses a JSON-string node and forwards it inside { node }', async () => {
const tools = await buildTools();
await tools.insertNode.execute(
{
@@ -370,9 +405,15 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
{} as never,
);
expect(insertNodeCalls).toHaveLength(1);
const [pageId, node] = insertNodeCalls[0];
// #413: the 2nd arg is the XOR input { markdown?, node? }, the 3rd is opts.
const [pageId, input, opts] = insertNodeCalls[0] as [
string,
{ markdown?: unknown; node?: unknown },
{ position?: string },
];
expect(pageId).toBe('p1');
expect(node).toEqual(NODE_OBJ);
expect(input).toEqual({ markdown: undefined, node: NODE_OBJ });
expect(opts.position).toBe('append');
});
it('insertNode throws the documented message on invalid JSON string', async () => {
@@ -426,6 +467,54 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
).rejects.toThrow('content was a string but not valid JSON');
expect(updatePageJsonCalls).toHaveLength(0);
});
// #411: the plain-Markdown full-body-replace tool is now the shared
// `updatePageMarkdown` (was inline `updatePageContent`). It forwards to
// client.updatePage(pageId, content, title) -> updatePageContentRealtime ->
// markdownToProseMirrorCanonical, so `^[...]` footnotes materialize.
it('updatePageMarkdown forwards { pageId, content, title } to client.updatePage', async () => {
const tools = await buildTools();
await tools.updatePageMarkdown.execute(
{ pageId: 'p1', content: 'Body^[a note]', title: 'New title' } as never,
{} as never,
);
expect(updatePageCalls).toHaveLength(1);
expect(updatePageCalls[0]).toEqual(['p1', 'Body^[a note]', 'New title']);
});
it('updatePageMarkdown returns the RAW client result in-app (deliberate #411 shape change, documented on the spec)', async () => {
const tools = await buildTools();
// Registry canonical execute returns client.updatePage's result verbatim.
// The old inline tool projected to { pageId, updated }; the rename now
// surfaces the raw result (nothing reads the removed `.updated`; the raw
// shape carries footnote/verify warnings and matches the on-both-hosts
// registry convention). fakeClient.updatePage resolves { success: true }.
const result = await tools.updatePageMarkdown.execute(
{ pageId: 'p1', content: '# Hi' } as never,
{} as never,
);
expect(result).toEqual({ success: true });
});
it('updatePageMarkdown forwards title=undefined when omitted', async () => {
const tools = await buildTools();
await tools.updatePageMarkdown.execute(
{ pageId: 'p1', content: '# Hi' } as never,
{} as never,
);
expect(updatePageCalls[0]).toEqual(['p1', '# Hi', undefined]);
});
// #411 surface split: the plain-Markdown replace tool exists in-app under the
// new key; the OLD inline updatePageContent key is gone; importPageMarkdown is
// still present IN-APP (only the external MCP surface drops it — asserted in
// packages/mcp/test/unit/tool-inventory.test.mjs).
it('exposes updatePageMarkdown in-app, no legacy updatePageContent, keeps importPageMarkdown', async () => {
const tools = await buildTools();
expect(tools.updatePageMarkdown).toBeDefined();
expect((tools as Record<string, unknown>).updatePageContent).toBeUndefined();
expect(tools.importPageMarkdown).toBeDefined();
});
});
/**
@@ -439,7 +528,7 @@ describe('AiChatToolsService node-arg JSON-string coercion', () => {
* getOutline) are exercised here end-to-end through forUser().
*/
describe('AiChatToolsService model-friendly input validation (#190)', () => {
const fakeClient: Partial<DocmostClientLike> = {};
const fakeClient: FakeDocmostClient = {};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
generateCollabToken: jest.fn().mockResolvedValue('collab-token'),
@@ -557,7 +646,7 @@ describe('AiChatToolsService #294 changed execute wirings', () => {
tableDeleteRow: [],
tableUpdateCell: [],
};
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
movePage: (...args: unknown[]) => {
calls.movePage.push(args);
return Promise.resolve({ success: true });
@@ -666,7 +755,7 @@ describe('AiChatToolsService #410 footnote + image tools', () => {
insertImage: [],
replaceImage: [],
};
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
insertFootnote: (...args: unknown[]) => {
calls.insertFootnote.push(args);
return Promise.resolve({ success: true, footnoteId: 'fn1', reused: false });
@@ -836,3 +925,109 @@ describe('AiChatToolsService getCurrentPage selection (#388)', () => {
);
});
});
/**
* #440 review: the in-app drawioCreate / drawioUpdate handlers must forward
* the optional `layout:"elk"` param to the client (5th positional arg), exactly
* like the MCP host. It was silently dropped, so ELK auto-layout worked only via
* the standalone MCP server, not in-app. These tests pin per-host parity.
*/
describe('AiChatToolsService drawio layout passthrough (#440)', () => {
const createCalls: unknown[][] = [];
const updateCalls: unknown[][] = [];
// FakeDocmostClient (not Partial<DocmostClientLike>): since #446 derived
// DocmostClientLike from the real client, its drawioCreate/drawioUpdate return
// the concrete result shape, so a minimal stub object would not be assignable.
// FakeDocmostClient types every method as (...args) => Promise<any>, which is
// exactly what these arg-capturing doubles need.
const fakeClient: FakeDocmostClient = {
drawioCreate: (...args: unknown[]) => {
createCalls.push(args);
return Promise.resolve({ success: true, nodeId: '#0' });
},
drawioUpdate: (...args: unknown[]) => {
updateCalls.push(args);
return Promise.resolve({ success: true, nodeId: '#0' });
},
};
const tokenServiceStub = {
generateAccessToken: jest.fn().mockResolvedValue('access-token'),
generateCollabToken: jest.fn().mockResolvedValue('collab-token'),
};
let service: AiChatToolsService;
beforeEach(() => {
createCalls.length = 0;
updateCalls.length = 0;
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue(
mockLoaded(function () {
return fakeClient as DocmostClientLike;
} as unknown as loader.DocmostClientCtor),
);
service = new AiChatToolsService(
tokenServiceStub as never,
{} as never,
{} as never,
{} as never,
{} as never,
{
asSink: () => ({ put: jest.fn(), has: jest.fn(), evict: jest.fn() }),
} as never,
);
});
afterEach(() => jest.restoreAllMocks());
const buildTools = () =>
service.forUser(
{ id: 'user-1', email: 'u@example.com', workspaceId: 'ws-1' } as never,
'session-1',
'ws-1',
'chat-1',
);
it('forwards layout:"elk" to client.drawioCreate as the 5th positional arg', async () => {
const tools = await buildTools();
await tools.drawioCreate.execute(
{
pageId: 'p-1',
xml: '<mxGraphModel/>',
position: 'append',
layout: 'elk',
} as never,
{} as never,
);
expect(createCalls).toHaveLength(1);
// drawioCreate(pageId, where, xml, title, layout) — layout is args[4].
expect(createCalls[0][4]).toBe('elk');
});
it('forwards layout:"elk" to client.drawioUpdate as the 5th positional arg', async () => {
const tools = await buildTools();
await tools.drawioUpdate.execute(
{
pageId: 'p-1',
node: '#0',
xml: '<mxGraphModel/>',
baseHash: 'h',
layout: 'elk',
} as never,
{} as never,
);
expect(updateCalls).toHaveLength(1);
// drawioUpdate(pageId, node, xml, baseHash, layout) — layout is args[4].
expect(updateCalls[0][4]).toBe('elk');
});
it('omits layout (undefined 5th arg) when not requested', async () => {
const tools = await buildTools();
await tools.drawioCreate.execute(
{ pageId: 'p-1', xml: '<mxGraphModel/>', position: 'append' } as never,
{} as never,
);
expect(createCalls[0][4]).toBeUndefined();
});
});
@@ -26,6 +26,125 @@ import {
type ToolCatalogEntry,
} from './tool-tiers';
/**
* Compile-time contract (issue #446): the in-app tool `execute` closures below
* call the loopback `DocmostClient` POSITIONALLY (e.g.
* `client.drawioGet(pageId, node, format ?? 'xml')`). Those closures receive an
* AI-SDK-erased (`any`) input, so a positional call inside them is NOT checked
* against the real signature — a parameter reorder/type-change in
* `packages/mcp/src/client.ts` would otherwise reach production as a runtime
* "wrong argument" tool failure with zero compile signal (the restored #294
* debt). This never-called function reproduces every positional call with
* correctly-typed placeholder arguments against the DERIVED `DocmostClientLike`
* (a `Pick` of the real `DocmostClient`), so any such reorder/rename becomes a
* SERVER COMPILE ERROR here. It emits nothing (types only) and is never invoked;
* keep each call in lockstep with the matching `execute` body below.
*/
// eslint-disable-next-line @typescript-eslint/no-unused-vars
function __assertClientCallContract(client: DocmostClientLike): void {
// Placeholders standing in for the AI-SDK-erased execute inputs. Their types
// are deliberately concrete so the positional calls are checked end-to-end.
const s = '' as string;
const n = 0 as number;
const node: unknown = null;
const edits: Array<{ find: string; replace: string; replaceAll?: boolean }> =
[];
const cells: string[] = [];
const align = undefined as 'left' | 'center' | 'right' | undefined;
// --- read ---
void client.search(s, undefined, n);
void client.getPage(s);
void client.getPageRaw(s);
void client.getWorkspace();
void client.getSpaces();
void client.listPages(s, n, true);
void client.getTree(s, s, n);
void client.getPageContext(s);
void client.listSidebarPages(s, s);
void client.getOutline(s);
void client.getPageJson(s);
void client.getNode(s, s, 'markdown');
void client.searchInPage(s, s, {
regex: true,
caseSensitive: true,
limit: n,
});
void client.getTable(s, s);
void client.listComments(s, true);
void client.getComment(s);
void client.checkNewComments(s, s, s);
void client.listShares();
void client.listPageHistory(s, s);
void client.getPageHistory(s);
void client.diffPageVersions(s, s, s);
void client.exportPageMarkdown(s);
// --- write (page) ---
void client.createPage(s, s, s, s);
void client.updatePage(s, s, s);
void client.renamePage(s, s);
void client.movePage(s, s, s);
void client.deletePage(s);
void client.editPageText(s, edits);
void client.patchNode(s, s, { markdown: s, node });
void client.insertNode(
s,
{ markdown: s, node },
{
position: 'append',
anchorNodeId: s,
anchorText: s,
},
);
void client.deleteNode(s, s);
void client.updatePageJson(s, node, s);
void client.tableInsertRow(s, s, cells, n);
void client.tableDeleteRow(s, s, n);
void client.tableUpdateCell(s, s, n, n, s);
void client.copyPageContent(s, s);
void client.importPageMarkdown(s, s);
void client.sharePage(s, true);
void client.unsharePage(s);
void client.restorePageVersion(s);
void client.transformPage(s, s, { dryRun: true });
void client.stashPage(s);
// --- write (image / footnote), in-app since #410 ---
void client.insertFootnote(s, s, s);
void client.insertImage(s, s, {
align,
alt: s,
replaceText: s,
afterText: s,
});
void client.replaceImage(s, s, s, { align, alt: s });
// --- draw.io diagrams (#423 stage 1, #424 stage 2) ---
// The 5th `layout` arg (#424) is exercised so this parity assertion fails if the
// client signature drops it — it must reach the client from the shared execute.
void client.drawioGet(s, s, 'xml');
void client.drawioCreate(s, { position: 'append', anchorNodeId: s }, s, s, 'elk');
void client.drawioUpdate(s, s, s, s, 'elk');
// --- draw.io high-level semantic tools (#425 stage 3) ---
void client.drawioEditCells(s, s, [{ op: 'delete', cellId: s }], s);
void client.drawioFromGraph(
s,
{ position: 'append', anchorNodeId: s },
{ nodes: [{ id: s, label: s }] },
'LR',
s,
'full',
s,
);
void client.drawioFromMermaid(
s,
{ position: 'append', anchorNodeId: s },
s,
s,
);
// --- write (comment) ---
void client.createComment(s, s, 'inline', s, s, s);
void client.resolveComment(s, true);
}
/**
* Per-user, per-request adapter that exposes Docmost READ operations to the
* agent as AI SDK tools (STAGE A = read only).
@@ -169,8 +288,19 @@ export class AiChatToolsService {
// provenance tokens) and load the shared tool-spec registry. Client
// construction is shared with the page-change detection path (#274) via
// buildDocmostClient so both go over the exact same authenticated route.
const { sharedToolSpecs, createCommentSignalTracker } =
await loadDocmostMcp();
// searchShapes / getGuideSection (#424) are the PURE, no-network helpers
// backing drawioShapes / drawioGuide. They are `inlineBothHosts` specs (no
// canonical execute — their catalog loader uses import.meta and can't be
// value-imported into the zod-agnostic tool-specs.ts under the server's
// commonjs type-check), so the shared registry loop below SKIPS them and this
// service wires them inline (see drawioShapes/drawioGuide entries), mirroring
// how index.ts registers them on the standalone MCP host.
const {
sharedToolSpecs,
createCommentSignalTracker,
searchShapes,
getGuideSection,
} = await loadDocmostMcp();
const client = await this.buildDocmostClient(
user,
sessionId,
@@ -198,6 +328,18 @@ export class AiChatToolsService {
execute,
});
// The in-app toolset. It starts with the tools kept INLINE here for a
// documented per-layer reason: an intentional behaviour/schema divergence from
// the standalone MCP surface (searchPages' hybrid RRF,
// transformPage's guardrailed shorter schema), a name clash the shared
// registry forbids (in-app `getTable` verb-first vs the MCP noun-first
// `tableGet` — the registry requires mcpName === inAppKey), per-request
// state the registry loop cannot provide
// (getCurrentPage reads the resolved openedPage; searchPages closes over the
// per-request user/embedding deps), or a tool with no MCP twin
// (listSidebarPages/getComment/getPageHistory). Every SHARED tool is then added
// by the registry loop below (see it), so there is exactly one arg-mapping per
// shared tool and it can never drift from the MCP host again (#445).
const tools: Record<string, Tool> = {
// INTENTIONAL per-transport divergence (not in the shared registry): this
// in-app search runs a semantic + keyword hybrid (RRF) with in-process
@@ -332,180 +474,14 @@ export class AiChatToolsService {
execute: async () => resolveCurrentPageResult(openedPage),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The execute body keeps this layer's { title, markdown } projection.
getPage: sharedTool(sharedToolSpecs.getPage, async ({ pageId }) => {
// getPage(pageId) -> { data: filterPage(page, markdown), success }.
const result = await client.getPage(pageId);
const data = (result?.data ?? {}) as {
title?: string;
content?: string;
};
return {
title: data.title ?? '',
markdown: typeof data.content === 'string' ? data.content : '',
};
}),
// --- WRITE tools (all reversible — history/trash; §6.5 / D3) ---
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
createPage: sharedTool(
sharedToolSpecs.createPage,
async ({ title, content, spaceId, parentPageId }) => {
// createPage(title, content, spaceId, parentPageId?) ->
// { data: filterPage(page, markdown), success }.
const result = await client.createPage(
title,
content ?? '',
spaceId,
parentPageId,
);
const data = (result?.data ?? {}) as {
id?: string;
slugId?: string;
title?: string;
};
return { id: data.id ?? data.slugId, title: data.title ?? title };
},
),
updatePageContent: tool({
description:
"Replace a page's body with new Markdown content (and optionally its " +
'title). Reversible: the previous version is kept in page history.',
inputSchema: modelFriendlyInput({
pageId: z.string().describe('The id of the page to update.'),
content: z.string().describe('The new page body as Markdown.'),
title: z
.string()
.optional()
.describe('Optional new title for the page.'),
}),
execute: async ({ pageId, content, title }) => {
// updatePage mutates the live collab doc -> provenance flows from the
// collab-token provider. Returns { success, modified, message, pageId }.
const result = (await client.updatePage(pageId, content, title)) as {
success?: boolean;
};
return { pageId, updated: result?.success ?? true };
},
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
renamePage: sharedTool(
sharedToolSpecs.renamePage,
async ({ pageId, title }) => {
// renamePage(pageId, title) -> { success, pageId, title }.
await client.renamePage(pageId, title);
return { pageId, title };
},
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The shared schema adds the optional `position` field this layer lacked
// before; the execute now forwards it (the client already accepted it).
movePage: sharedTool(
sharedToolSpecs.movePage,
async ({ pageId, parentPageId, position }) => {
// movePage(pageId, parentPageId, position?) -> raw move response.
await client.movePage(pageId, parentPageId ?? null, position);
return { pageId, parentPageId: parentPageId ?? null, moved: true };
},
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// GUARDRAIL (§14 H4) preserved: the shared schema exposes ONLY pageId, so
// permanentlyDelete/forceDelete are never part of the input and can never
// be forwarded — the agent physically cannot permanently delete a page.
deletePage: sharedTool(sharedToolSpecs.deletePage, async ({ pageId }) => {
// deletePage(pageId) hits POST /pages/delete with { pageId } only,
// which is the soft-delete (trash) path on the server.
await client.deletePage(pageId);
return { pageId, trashed: true };
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// This layer keeps only its own execute-side guards (require a selection
// for a top-level comment; reject suggestedText on a reply / without a
// selection) — the schema+description are shared.
createComment: sharedTool(
sharedToolSpecs.createComment,
async ({
pageId,
content,
selection,
parentCommentId,
suggestedText,
}) => {
// createComment(pageId, content, type, selection?, parentCommentId?,
// suggestedText?). Top-level comments are inline and must carry a
// selection to anchor on; replies inherit the parent's anchor (no
// selection). Throwing here surfaces a tool error to the model (Vercel
// `ai` SDK) so the agent retries with a better selection — do not
// catch/suppress it.
if (!parentCommentId && (!selection || !selection.trim())) {
throw new Error(
"createComment requires a 'selection' (exact text to anchor on) for a new top-level comment.",
);
}
if (suggestedText !== undefined) {
if (parentCommentId) {
throw new Error(
"createComment: 'suggestedText' cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !selection.trim()) {
throw new Error(
"createComment: 'suggestedText' requires a 'selection' to anchor and rewrite.",
);
}
}
const result = await client.createComment(
pageId,
content,
'inline',
selection,
parentCommentId,
suggestedText,
);
const data = (result?.data ?? {}) as { id?: string };
return { commentId: data.id, pageId };
},
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
resolveComment: sharedTool(
sharedToolSpecs.resolveComment,
async ({ commentId, resolved }) => {
// resolveComment(commentId, resolved) -> { success, commentId, resolved }.
await client.resolveComment(commentId, resolved);
return { commentId, resolved };
},
),
// --- READ tools (added) ---
getWorkspace: sharedTool(
sharedToolSpecs.getWorkspace,
async () => await client.getWorkspace(),
),
listSpaces: sharedTool(
sharedToolSpecs.listSpaces,
async () => await client.getSpaces(),
),
// INTENTIONAL per-transport divergence (not shared): keeps the `tree:true`
// hierarchy mode but is worded for the in-app agent; the standalone MCP
// `list_pages` carries its own wording. Kept per-layer so each side tunes
// its own guidance.
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
listPages: sharedTool(
sharedToolSpecs.listPages,
async ({ spaceId, limit, tree }) =>
await client.listPages(spaceId, limit, tree),
),
//
// NOTE (issue #411): the plain-Markdown full-body-replace tool is no longer
// inline here — it moved to @docmost/mcp's SHARED_TOOL_SPECS as
// `updatePageMarkdown` (was inline `updatePageContent`) so it registers on
// BOTH the external MCP and the in-app agent. The registry loop below adds
// it under its inAppKey. importPageMarkdown stays a shared spec too (now
// inAppOnly — dropped from the external MCP surface, kept in-app).
listSidebarPages: tool({
description:
@@ -525,34 +501,9 @@ export class AiChatToolsService {
await client.listSidebarPages(spaceId, pageId),
}),
getOutline: sharedTool(
sharedToolSpecs.getOutline,
async ({ pageId }) => await client.getOutline(pageId),
),
getPageJson: sharedTool(
sharedToolSpecs.getPageJson,
async ({ pageId }) => await client.getPageJson(pageId),
),
getNode: sharedTool(
sharedToolSpecs.getNode,
async ({ pageId, nodeId }) => await client.getNode(pageId, nodeId),
),
searchInPage: sharedTool(
sharedToolSpecs.searchInPage,
async ({ pageId, query, regex, caseSensitive, limit }) =>
await client.searchInPage(pageId, query, {
regex,
caseSensitive,
limit,
}),
),
// NOT shared (kept inline): the MCP tool name `table_get` is noun-first
// while this key is `getTable` (verb-first), breaking the
// snake_case(inAppKey) convention the shared registry enforces. Its
// NOT shared (kept inline): the MCP tool name `tableGet` is noun-first
// while this key is `getTable` (verb-first), so it cannot satisfy the
// shared registry's `mcpName === inAppKey` convention (#412). Its
// reference parameter is still named `table` (was `tableRef`) so it matches
// the migrated table row/cell tools below.
getTable: tool({
@@ -572,13 +523,6 @@ export class AiChatToolsService {
await client.getTable(pageId, table),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
listComments: sharedTool(
sharedToolSpecs.listComments,
async ({ pageId, includeResolved }) =>
await client.listComments(pageId, includeResolved),
),
getComment: tool({
description: 'Fetch a single comment by id (content as Markdown).',
inputSchema: modelFriendlyInput({
@@ -587,24 +531,6 @@ export class AiChatToolsService {
execute: async ({ commentId }) => await client.getComment(commentId),
}),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
checkNewComments: sharedTool(
sharedToolSpecs.checkNewComments,
async ({ spaceId, since, parentPageId }) =>
await client.checkNewComments(spaceId, since, parentPageId),
),
listShares: sharedTool(
sharedToolSpecs.listShares,
async () => await client.listShares(),
),
listPageHistory: sharedTool(
sharedToolSpecs.listPageHistory,
async ({ pageId, cursor }) =>
await client.listPageHistory(pageId, cursor),
),
getPageHistory: tool({
description:
'Fetch a single page-history version including its lossless ' +
@@ -616,206 +542,11 @@ export class AiChatToolsService {
await client.getPageHistory(historyId),
}),
diffPageVersions: sharedTool(
sharedToolSpecs.diffPageVersions,
async ({ pageId, from, to }) =>
await client.diffPageVersions(pageId, from, to),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
exportPageMarkdown: sharedTool(
sharedToolSpecs.exportPageMarkdown,
async ({ pageId }) => {
const markdown = await client.exportPageMarkdown(pageId);
return { markdown };
},
),
// --- WRITE tools (added; reversible via page history/trash) ---
editPageText: sharedTool(
sharedToolSpecs.editPageText,
async ({ pageId, edits }) => await client.editPageText(pageId, edits),
),
// Returns ONLY the short link object — never the document body — so a
// large page can be handed to an external consumer without bloating
// context.
stashPage: sharedTool(
sharedToolSpecs.stashPage,
async ({ pageId }) => await client.stashPage(pageId),
),
// Schema + description from the shared registry (identical across both
// transports). The execute body keeps its OWN parseNodeArg normalization:
// the model sometimes serializes the node as a JSON string, and we parse it
// before the client's typeof-object guard rejects it (parity with the
// standalone MCP server, index.ts patch_node).
patchNode: sharedTool(
sharedToolSpecs.patchNode,
async ({ pageId, nodeId, node }) => {
const parsedNode = parseNodeArg(node);
return await client.patchNode(pageId, nodeId, parsedNode);
},
),
// Shared registry schema + description; execute retains parseNodeArg on the
// incoming node (parity with the standalone MCP server, index.ts
// insert_node).
insertNode: sharedTool(
sharedToolSpecs.insertNode,
async ({ pageId, node, position, anchorNodeId, anchorText }) => {
const parsedNode = parseNodeArg(node);
return await client.insertNode(pageId, parsedNode, {
position,
anchorNodeId,
anchorText,
});
},
),
deleteNode: sharedTool(
sharedToolSpecs.deleteNode,
async ({ pageId, nodeId }) => await client.deleteNode(pageId, nodeId),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The execute body keeps this layer's content normalization (parity with
// the standalone MCP server, index.ts update_page_json).
updatePageJson: sharedTool(
sharedToolSpecs.updatePageJson,
async ({ pageId, content, title }) => {
// undefined/null pass through as undefined (title-only / no-op); any
// string is JSON.parsed (so an empty string "" throws, matching the
// MCP server); an object is passed through unchanged.
let doc;
if (content === undefined || content === null) {
doc = undefined;
} else {
// String -> JSON.parse (throwing on invalid); object passes through.
doc = parseNodeArg(content, 'content was a string but not valid JSON');
}
return await client.updatePageJson(pageId, doc, title);
},
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
// Promoted from MCP-only so the in-app agent can attach a REAL footnote to
// already-written text instead of leaving a literal `^[...]` string.
insertFootnote: sharedTool(
sharedToolSpecs.insertFootnote,
async ({ pageId, anchorText, text }) =>
await client.insertFootnote(pageId, anchorText, text),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
// The schema field is `imageUrl`; the client method takes it positionally.
insertImage: sharedTool(
sharedToolSpecs.insertImage,
async ({ pageId, imageUrl, align, alt, replaceText, afterText }) =>
await client.insertImage(pageId, imageUrl, {
align,
alt,
replaceText,
afterText,
}),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#410).
replaceImage: sharedTool(
sharedToolSpecs.replaceImage,
async ({ pageId, attachmentId, imageUrl, align, alt }) =>
await client.replaceImage(pageId, attachmentId, imageUrl, {
align,
alt,
}),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#423).
// meta.hash in the result is the baseHash drawioUpdate requires.
drawioGet: sharedTool(
sharedToolSpecs.drawioGet,
async ({ pageId, node, format }) =>
await client.drawioGet(pageId, node, format ?? 'xml'),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#423).
// The flat schema fields are regrouped into the client's `where` object.
drawioCreate: sharedTool(
sharedToolSpecs.drawioCreate,
async ({ pageId, xml, position, anchorNodeId, anchorText, title }) =>
await client.drawioCreate(
pageId,
{ position, anchorNodeId, anchorText },
xml,
title,
),
),
// Schema + description live in @docmost/mcp's SHARED_TOOL_SPECS (#423).
// baseHash is the optimistic lock: mismatch => structured conflict error.
drawioUpdate: sharedTool(
sharedToolSpecs.drawioUpdate,
async ({ pageId, node, xml, baseHash }) =>
await client.drawioUpdate(pageId, node, xml, baseHash),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// The table reference parameter was unified to `table` (was `tableRef`).
tableInsertRow: sharedTool(
sharedToolSpecs.tableInsertRow,
async ({ pageId, table, cells, index }) =>
await client.tableInsertRow(pageId, table, cells, index),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
tableDeleteRow: sharedTool(
sharedToolSpecs.tableDeleteRow,
async ({ pageId, table, index }) =>
await client.tableDeleteRow(pageId, table, index),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
tableUpdateCell: sharedTool(
sharedToolSpecs.tableUpdateCell,
async ({ pageId, table, row, col, text }) =>
await client.tableUpdateCell(pageId, table, row, col, text),
),
copyPageContent: sharedTool(
sharedToolSpecs.copyPageContent,
async ({ sourcePageId, targetPageId }) =>
await client.copyPageContent(sourcePageId, targetPageId),
),
importPageMarkdown: sharedTool(
sharedToolSpecs.importPageMarkdown,
async ({ pageId, markdown }) =>
await client.importPageMarkdown(pageId, markdown),
),
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294).
// Both layers already carried the security-confirmation framing, so there
// was no real divergence to preserve — only wording drift.
sharePage: sharedTool(
sharedToolSpecs.sharePage,
async ({ pageId, searchIndexing }) =>
await client.sharePage(pageId, searchIndexing),
),
unsharePage: sharedTool(
sharedToolSpecs.unsharePage,
async ({ pageId }) => await client.unsharePage(pageId),
),
restorePageVersion: sharedTool(
sharedToolSpecs.restorePageVersion,
async ({ historyId }) => await client.restorePageVersion(historyId),
),
// INTENTIONAL per-transport divergence (not shared): deliberately omits the
// `deleteComments` schema field (comment-deletion guardrail) and carries a
// much shorter description; the standalone MCP `docmost_transform` exposes
// much shorter description; the standalone MCP `docmostTransform` exposes
// the full helper catalogue. Different schema, so kept per-layer.
transformPage: tool({
description:
@@ -841,6 +572,51 @@ export class AiChatToolsService {
}),
};
// Add EVERY shared tool from the zod-agnostic registry in one loop (#445).
// The spec owns the canonical arg->client mapping; this host only decides
// WHICH mapping to run and returns its value directly (no envelope). For each
// spec:
// - skip `mcpOnly` specs (they belong to the standalone MCP host only);
// - skip `inlineBothHosts` specs (drawioShapes / drawioGuide): they carry
// no execute and are wired INLINE just below, calling the pure helpers;
// - use `inAppExecute` when the spec declares a DELIBERATE per-layer
// difference (a projected result shape, a different guardrail message);
// - otherwise use the canonical `execute` (raw client result, identical to
// the MCP host's before it wraps it as JSON).
// The execute receives the AI-SDK-validated, type-erased input; the spec reads
// the same fields its buildShape declares. This is the SINGLE place the in-app
// arg mapping lives — it can no longer silently drift from the MCP host.
for (const spec of Object.values(sharedToolSpecs)) {
if (spec.mcpOnly) continue;
if (spec.inlineBothHosts) continue;
const run = spec.inAppExecute ?? spec.execute;
if (!run) continue; // defensive: a shared spec always carries one of them.
tools[spec.inAppKey] = sharedTool(
spec,
(async (args) =>
run(client, args as Record<string, unknown>)) as Tool['execute'],
);
}
// drawioShapes / drawioGuide (#424): `inlineBothHosts` registry specs wired
// here with the SAME schema+description the shared spec pins, but calling the
// pure searchShapes / getGuideSection helpers off the loaded @docmost/mcp
// module — they are not client methods and their catalog loader uses
// import.meta, so they cannot live in the zod-agnostic shared execute. The raw
// result is identical to the MCP host's (which wraps it as JSON text); here
// the in-app host returns it plain, exactly like every other shared tool.
tools[sharedToolSpecs.drawioShapes.inAppKey] = sharedTool(
sharedToolSpecs.drawioShapes,
async ({ query, category, limit }) => {
const results = searchShapes(query, { category, limit });
return { query, count: results.length, results };
},
);
tools[sharedToolSpecs.drawioGuide.inAppKey] = sharedTool(
sharedToolSpecs.drawioGuide,
async ({ section }) => getGuideSection(section),
);
// Passive "new comments: N" signal (#417). PER-TURN state (forUser runs once
// per turn), so the watermark starts now and only comments a human leaves
// WHILE this turn runs are signalled — exactly the mid-turn loop; between-turn
@@ -7,6 +7,16 @@ import type {
DocmostClientLike,
CommentSignalTrackerLike,
} from './docmost-client.loader';
// Test-double type for the loopback client. `DocmostClientLike` is now DERIVED
// from the real `DocmostClient` (issue #446), so its method RETURN types are the
// concrete client shapes. These probe stubs deliberately return minimal shapes
// (e.g. `getPageRaw` yielding only `{ title }`), so the doubles use the same
// method NAMES but loose async returns; each is cast to `DocmostClientLike` at
// the (return-erased) mock site, leaving production positional-call safety intact.
type FakeDocmostClient = Partial<
Record<keyof DocmostClientLike, (...args: any[]) => Promise<any>>
>;
import { SHARED_TOOL_SPECS } from '../../../../../../packages/mcp/src/tool-specs';
// The REAL shared tracker factory, imported from source (same cross-boundary
// approach the tool-specs spec uses) so the in-app wiring is exercised against
@@ -268,15 +278,23 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
// seeded at forUser time).
const future = new Date(Date.now() + 3_600_000).toISOString();
function buildService(fakeClient: Partial<DocmostClientLike>) {
function buildService(fakeClient: FakeDocmostClient) {
jest.spyOn(loader, 'loadDocmostMcp').mockResolvedValue({
DocmostClient: function () {
return fakeClient as DocmostClientLike;
} as unknown as loader.DocmostClientCtor,
sharedToolSpecs: SHARED_TOOL_SPECS as Record<string, loader.SharedToolSpec>,
sharedToolSpecs: SHARED_TOOL_SPECS as unknown as Record<string, loader.SharedToolSpec>,
// Wire the REAL factory so the in-app path is exercised end to end.
createCommentSignalTracker:
createCommentSignalTracker as unknown as loader.CommentSignalTrackerFactory,
// Pure no-network draw.io helpers (#424) — required on the loader return;
// this comment-signal test doesn't exercise them, so no-op stubs suffice.
searchShapes: (() => []) as unknown as loader.SearchShapesFn,
getGuideSection: (() => ({
section: '',
content: '',
sections: [],
})) as unknown as loader.GetGuideSectionFn,
});
return new AiChatToolsService(
tokenServiceStub as never,
@@ -317,7 +335,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
afterEach(() => jest.restoreAllMocks());
it('emits the signal (model-only) on a non-comment tool when a new comment exists', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
getPage: async () => ({
data: { title: 'Иранские языки', content: 'body' },
success: true,
@@ -342,7 +360,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
});
it('does NOT add the signal to the listComments tool itself (tautological)', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
listComments: async () => ({
items: [{ createdAt: future }],
resolvedThreadsHidden: 0,
@@ -356,7 +374,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
});
it('no new comments => tool output is byte-identical AND the model sees no signal', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
getPage: async () => ({
data: { title: 'T', content: 'body' },
success: true,
@@ -372,7 +390,7 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
});
it('injection-safety: a malicious page title cannot forge a second signal', async () => {
const fakeClient: Partial<DocmostClientLike> = {
const fakeClient: FakeDocmostClient = {
getPage: async () => ({
data: { title: 'body-title', content: 'body' },
success: true,
@@ -0,0 +1,173 @@
import { createHash } from 'node:crypto';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { computeSrcRegistryStamp } from './docmost-client.loader';
// The exact message the loader throws on a build/src skew (issue #447). Kept as a
// literal here so a reworded prod message reddens this test (the message is a
// developer-facing contract: it tells them how to fix it).
const STALE_BUILD_MESSAGE =
'@docmost/mcp build is stale (tool-specs changed since last build) — run: pnpm --filter @docmost/mcp build';
// Replica of the loader's inline stale-check predicate + throw from
// `loadDocmostMcp`. That guard is not independently exported (it lives inside the
// dynamic-import IIFE, wired to a fixed `require.resolve('@docmost/mcp')`), so we
// exercise the exact same three-condition logic against a stamp produced by the
// REAL `computeSrcRegistryStamp`. This documents and locks the throw/no-throw
// behaviour; if the prod predicate changes, this replica must change with it.
function assertStaleGuard(
srcStamp: string | null,
registryStamp: string | undefined,
): void {
if (
srcStamp !== null &&
typeof registryStamp === 'string' &&
srcStamp !== registryStamp
) {
throw new Error(STALE_BUILD_MESSAGE);
}
}
// Build a throwaway `<pkg>/build/index.js` + optional `<pkg>/src/tool-specs.ts`
// layout so `computeSrcRegistryStamp(<pkg>/build/index.js)` resolves src the same
// way the loader does (dirname(dirname(entry))/src/tool-specs.ts).
function makeFakePackage(toolSpecsSource: string | null): {
entry: string;
cleanup: () => void;
} {
const root = mkdtempSync(join(tmpdir(), 'mcp-stamp-'));
const buildDir = join(root, 'build');
mkdirSync(buildDir, { recursive: true });
const entry = join(buildDir, 'index.js');
writeFileSync(entry, '// fake @docmost/mcp build entry\n', 'utf8');
if (toolSpecsSource !== null) {
const srcDir = join(root, 'src');
mkdirSync(srcDir, { recursive: true });
writeFileSync(join(srcDir, 'tool-specs.ts'), toolSpecsSource, 'utf8');
}
return { entry, cleanup: () => rmSync(root, { recursive: true, force: true }) };
}
describe('computeSrcRegistryStamp (#447 stale-build guard)', () => {
it('returns null when src/tool-specs.ts is absent (prod no-op path)', () => {
// A prod image ships only build/, no src/ — the guard must be a silent no-op.
const { entry, cleanup } = makeFakePackage(null);
try {
expect(computeSrcRegistryStamp(entry)).toBeNull();
} finally {
cleanup();
}
});
it('returns null for a bogus package entry (swallowed error path)', () => {
// A resolution/read hiccup must NEVER break startup — it resolves to null.
expect(
computeSrcRegistryStamp('/no/such/pkg/build/index.js'),
).toBeNull();
});
it('computes a 64-char sha256 hex when src/tool-specs.ts exists', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const stamp = computeSrcRegistryStamp(entry);
expect(stamp).toMatch(/^[0-9a-f]{64}$/);
} finally {
cleanup();
}
});
it('normalizes CRLF->LF and strips a single trailing newline', () => {
// A CRLF+trailing-newline variant of the same content hashes identically to
// the bare-LF form — the guard must not fire on a checkout-style difference.
const bare = makeFakePackage('alpha\nbeta');
const crlfTrailing = makeFakePackage('alpha\r\nbeta\r\n');
try {
expect(computeSrcRegistryStamp(crlfTrailing.entry)).toBe(
computeSrcRegistryStamp(bare.entry),
);
} finally {
bare.cleanup();
crlfTrailing.cleanup();
}
});
// CROSS-IMPL EQUALITY (covers reviewer suggestion 2). The SAME fixed input and
// EXPECTED hash are asserted in the mcp-side node test
// (packages/mcp/test/unit/registry-stamp.test.mjs) against the codegen's
// `computeRegistryStamp`. Asserting the SAME pair here against the loader's
// `computeSrcRegistryStamp` proves both implementations normalize+hash
// identically; a divergence in EITHER side reddens one of the two tests.
it('matches the documented cross-impl hash for a fixed input', () => {
const FIXED_INPUT = 'line1\r\nline2\n';
const EXPECTED =
'683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83';
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
try {
expect(computeSrcRegistryStamp(entry)).toBe(EXPECTED);
} finally {
cleanup();
}
});
it('the documented EXPECTED is the normalize+sha256 of the fixed input', () => {
// Proves EXPECTED is not a magic constant but the documented computation.
const FIXED_INPUT = 'line1\r\nline2\n';
const normalized = FIXED_INPUT.replace(/\r\n/g, '\n').replace(/\n$/, '');
const expected = createHash('sha256')
.update(normalized, 'utf8')
.digest('hex');
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
try {
expect(computeSrcRegistryStamp(entry)).toBe(expected);
} finally {
cleanup();
}
});
});
describe('loadDocmostMcp stale-check predicate (#447)', () => {
it('THROWS the exact stale message when src stamp != built REGISTRY_STAMP', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
expect(srcStamp).not.toBeNull();
// Simulate a stale build: build/ carries a DIFFERENT stamp than src.
expect(() => assertStaleGuard(srcStamp, 'a'.repeat(64))).toThrow(
STALE_BUILD_MESSAGE,
);
} finally {
cleanup();
}
});
it('does NOT throw when src stamp equals the built REGISTRY_STAMP', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
// Fresh build: build/ stamp == src stamp -> guard is a no-op.
expect(() => assertStaleGuard(srcStamp, srcStamp as string)).not.toThrow();
} finally {
cleanup();
}
});
it('does NOT throw when src is absent (prod: srcStamp === null)', () => {
// Even against a present-but-mismatched REGISTRY_STAMP, a null src stamp
// (prod image with build/ only) must skip the check entirely.
expect(() => assertStaleGuard(null, 'a'.repeat(64))).not.toThrow();
});
it('does NOT throw when REGISTRY_STAMP is absent (pre-#447 build)', () => {
// An older @docmost/mcp build has no REGISTRY_STAMP export; the guard must be
// a no-op so an out-of-date build never wrongly blocks startup.
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
expect(() => assertStaleGuard(srcStamp, undefined)).not.toThrow();
} finally {
cleanup();
}
});
});
@@ -1,264 +1,102 @@
import { createHash } from 'node:crypto';
import { existsSync, readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { pathToFileURL } from 'node:url';
import type { DocmostClient, SharedToolSpec } from '@docmost/mcp';
// Re-export SharedToolSpec so downstream server modules keep a single import
// path (they import it from this loader). The shape is DERIVED from the package
// entry, not re-declared here — see the import above (issue #446).
export type { SharedToolSpec } from '@docmost/mcp';
/**
* Minimal structural type for the `DocmostClient` class we consume from the
* ESM-only `@docmost/mcp` package. We only need the constructor + the read/write
* methods used by the per-user tool adapter; the full client surface lives in
* `packages/mcp/src/client.ts`. Signatures here mirror that file exactly.
*
* DRIFT GUARD: the method NAMES below are runtime-checked against the real
* `DocmostClient` by `packages/mcp/test/unit/client-host-contract.test.mjs`
* (which can import the ESM class directly). If you rename/remove a method here
* or in client.ts, that test fails — so a stale mirror cannot silently ship a
* runtime "x is not a function" into an agent tool call. Keep the two in sync.
*
* STAGED PLAN — full derivation `DocmostClientLike = <real DocmostClient type>`
* (issue #193, layer 3) is intentionally NOT done; it stays a hand-mirror for
* now because of two verified blockers across the ESM(mcp)/CJS(server) boundary:
* 1. `@docmost/mcp` emits NO declaration files (its tsconfig has no
* `declaration`, package.json has no `types`/types-export) and the server
* tsconfig has no path mapping for it — the server only loads it via the
* runtime `import()` trick below, so there is no type to import today.
* 2. The real client methods have inferred, CONCRETE return types; the in-app
* tool adapter reads results through loose `Record<string,unknown>` returns
* + `as` casts (e.g. `(result?.data ?? {}) as { title?: string }`).
* Deriving the exact type would make those casts non-overlapping ("may be a
* mistake") and break the build, and `Partial<DocmostClientLike>` test stubs
* would have to satisfy the full concrete surface.
* To do it safely later (incrementally): (a) turn on `declaration: true` in
* packages/mcp/tsconfig.json + add a `types` export condition and commit the
* emitted `.d.ts`; (b) `import type { DocmostClient } from '@docmost/mcp'` here
* and replace this interface with a `Pick<DocmostClient, ...>` of the consumed
* methods; (c) audit every `as` cast in ai-chat-tools.service.ts against the now
* concrete return types (double-cast through `unknown` only where genuinely
* needed); (d) keep the runtime guard test as a belt-and-braces check. Until
* then the guard test above is the cheap, behaviour-neutral protection.
* The exact set of `DocmostClient` methods the per-user in-app tool adapter
* consumes. This is the AUTHORITATIVE list of the client surface the server
* depends on; the adapter calls these methods POSITIONALLY, so this set is what
* the derived type below type-checks against the real class (issue #446).
*/
export interface DocmostClientLike {
type DocmostClientMethod =
// --- read ---
search(
query: string,
spaceId?: string,
limit?: number,
): Promise<{ items: unknown[]; success: boolean }>;
getPage(
pageId: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
// Light raw page info (`/pages/info`): title + slugId + ProseMirror content,
// WITHOUT the Markdown render / subpage expansion getPage does. Used by the
// comment-signal probe to read just the page title on a hit.
getPageRaw(pageId: string): Promise<Record<string, unknown> | null>;
getWorkspace(): Promise<{ data: Record<string, unknown>; success: boolean }>;
getSpaces(): Promise<unknown[]>;
listPages(
spaceId?: string,
limit?: number,
tree?: boolean,
): Promise<unknown[]>;
listSidebarPages(spaceId: string, pageId?: string): Promise<unknown[]>;
getOutline(pageId: string): Promise<Record<string, unknown>>;
getPageJson(pageId: string): Promise<Record<string, unknown>>;
getNode(pageId: string, nodeId: string): Promise<Record<string, unknown>>;
searchInPage(
pageId: string,
query: string,
opts?: { regex?: boolean; caseSensitive?: boolean; limit?: number },
): Promise<Record<string, unknown>>;
getTable(pageId: string, tableRef: string): Promise<Record<string, unknown>>;
// Returns `{ items, resolvedThreadsHidden }`. DEFAULT (includeResolved unset/
// false) hides resolved threads wholesale; pass true for the full feed.
listComments(
pageId: string,
includeResolved?: boolean,
): Promise<{ items: unknown[]; resolvedThreadsHidden: number }>;
getComment(
commentId: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
checkNewComments(
spaceId: string,
since: string,
parentPageId?: string,
): Promise<unknown>;
listShares(): Promise<unknown[]>;
listPageHistory(
pageId: string,
cursor?: string,
): Promise<{ items: unknown[]; nextCursor: string | null }>;
getPageHistory(historyId: string): Promise<Record<string, unknown>>;
diffPageVersions(
pageId: string,
from?: string,
to?: string,
): Promise<Record<string, unknown>>;
exportPageMarkdown(pageId: string): Promise<string>;
| 'search'
| 'getPage'
| 'getPageRaw'
| 'getWorkspace'
| 'getSpaces'
| 'listPages'
| 'getTree'
| 'getPageContext'
| 'listSidebarPages'
| 'getOutline'
| 'getPageJson'
| 'getNode'
| 'searchInPage'
| 'getTable'
| 'listComments'
| 'getComment'
| 'checkNewComments'
| 'listShares'
| 'listPageHistory'
| 'getPageHistory'
| 'diffPageVersions'
| 'exportPageMarkdown'
// --- write (page) ---
createPage(
title: string,
content: string,
spaceId: string,
parentPageId?: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
// Markdown content update via the collab path (carries provenance via the
// collab-token provider). Optionally also updates the title.
updatePage(
pageId: string,
content: string,
title?: string,
): Promise<Record<string, unknown>>;
// Title-only rename via REST.
renamePage(
pageId: string,
title: string,
): Promise<Record<string, unknown>>;
// Move via REST. parentPageId null => move to space root.
movePage(
pageId: string,
parentPageId: string | null,
position?: string,
): Promise<unknown>;
// SOFT delete only (POST /pages/delete with { pageId }). NEVER permanent.
deletePage(pageId: string): Promise<unknown>;
editPageText(
pageId: string,
edits: Array<{ find: string; replace: string; replaceAll?: boolean }>,
): Promise<Record<string, unknown>>;
patchNode(
pageId: string,
nodeId: string,
node: unknown,
): Promise<Record<string, unknown>>;
insertNode(
pageId: string,
node: unknown,
opts: {
position: 'before' | 'after' | 'append';
anchorNodeId?: string;
anchorText?: string;
},
): Promise<Record<string, unknown>>;
deleteNode(
pageId: string,
nodeId: string,
): Promise<Record<string, unknown>>;
updatePageJson(
pageId: string,
doc?: unknown,
title?: string,
): Promise<Record<string, unknown>>;
// Attach an author-inline footnote after the first occurrence of anchorText;
// numbering + the footnotes list are derived server-side.
insertFootnote(
pageId: string,
anchorText: string,
text: string,
): Promise<Record<string, unknown>>;
// Download a web image and insert it into the page (append, or replace/after a
// text anchor). `url` is the image http(s) URL.
insertImage(
pageId: string,
url: string,
opts?: {
align?: 'left' | 'center' | 'right';
alt?: string;
replaceText?: string;
afterText?: string;
},
): Promise<Record<string, unknown>>;
// Swap an existing image (by its attachmentId) for a new one fetched from a web
// URL, repointing every reference in the live document.
replaceImage(
pageId: string,
oldAttachmentId: string,
url: string,
opts?: { align?: 'left' | 'center' | 'right'; alt?: string },
): Promise<Record<string, unknown>>;
// --- draw.io diagrams (#423, stage 1) ---
// Read a diagram as decoded mxGraph XML (default) or the raw .drawio.svg.
// meta.hash is the optimistic-lock key drawioUpdate expects as baseHash.
drawioGet(
pageId: string,
node: string,
format?: 'xml' | 'svg',
): Promise<Record<string, unknown>>;
// Lint mxGraph XML, build the .drawio.svg attachment and insert a drawio node.
drawioCreate(
pageId: string,
where: {
position: 'before' | 'after' | 'append';
anchorNodeId?: string;
anchorText?: string;
},
xml: string,
title?: string,
): Promise<Record<string, unknown>>;
// Optimistic-locked full replacement of a diagram (baseHash from drawioGet).
drawioUpdate(
pageId: string,
node: string,
xml: string,
baseHash: string,
): Promise<Record<string, unknown>>;
tableInsertRow(
pageId: string,
tableRef: string,
cells: string[],
index?: number,
): Promise<Record<string, unknown>>;
tableDeleteRow(
pageId: string,
tableRef: string,
index: number,
): Promise<Record<string, unknown>>;
tableUpdateCell(
pageId: string,
tableRef: string,
row: number,
col: number,
text: string,
): Promise<Record<string, unknown>>;
copyPageContent(
sourcePageId: string,
targetPageId: string,
): Promise<Record<string, unknown>>;
importPageMarkdown(
pageId: string,
fullMarkdown: string,
): Promise<Record<string, unknown>>;
sharePage(
pageId: string,
searchIndexing?: boolean,
): Promise<Record<string, unknown>>;
unsharePage(pageId: string): Promise<Record<string, unknown>>;
restorePageVersion(historyId: string): Promise<Record<string, unknown>>;
// The opts type declares deleteComments? to match the real client signature,
// but the agent tool NEVER sets it (comment deletion stays unreachable).
transformPage(
pageId: string,
transformJs: string,
opts?: { dryRun?: boolean; deleteComments?: boolean },
): Promise<Record<string, unknown>>;
| 'createPage'
| 'updatePage'
| 'renamePage'
| 'movePage'
| 'deletePage'
| 'editPageText'
| 'patchNode'
| 'insertNode'
| 'deleteNode'
| 'updatePageJson'
| 'tableInsertRow'
| 'tableDeleteRow'
| 'tableUpdateCell'
| 'copyPageContent'
| 'importPageMarkdown'
| 'sharePage'
| 'unsharePage'
| 'restorePageVersion'
| 'transformPage'
| 'stashPage'
// --- write (image / footnote), in-app since #410 ---
| 'insertImage'
| 'replaceImage'
| 'insertFootnote'
// --- draw.io diagrams (#423 stage 1, #424 stage 2) ---
// DERIVED from the real DocmostClient (#446): drawioCreate/drawioUpdate carry
// the optional layout:"elk" 5th arg in the real signature, so the layout parity
// (#440) is inherited automatically — no hand-written mirror to keep in sync.
| 'drawioGet'
| 'drawioCreate'
| 'drawioUpdate'
// --- draw.io high-level semantic tools (#425 stage 3) ---
| 'drawioEditCells'
| 'drawioFromGraph'
| 'drawioFromMermaid'
// --- write (comment) ---
createComment(
pageId: string,
content: string,
type?: 'page' | 'inline',
selection?: string,
parentCommentId?: string,
suggestedText?: string,
): Promise<{ data: Record<string, unknown>; success: boolean }>;
resolveComment(
commentId: string,
resolved: boolean,
): Promise<Record<string, unknown>>;
// Serialize a page + mirror its internal images into the blob sandbox; returns
// ONLY a short anonymous URL (the body never enters the model context).
stashPage(pageId: string): Promise<{
uri: string;
sha256: string;
size: number;
images: { mirrored: number; failed: number };
}>;
}
| 'createComment'
| 'resolveComment';
/**
* The client surface the per-user tool adapter consumes, DERIVED from the real
* `DocmostClient` type in `@docmost/mcp` (issue #446, restored #294 debt). This
* replaces the former hand-mirror of ~45 method signatures.
*
* `import type` (above) is fully ERASED at compile time, so nothing is actually
* imported from the ESM-only package at runtime — the server still loads the
* class through the dynamic `import()` trick in `loadDocmostMcp` below; this is
* purely a compile-time type. Deriving via `Pick` means a parameter reorder or a
* type change to any of these methods in `client.ts` now becomes a SERVER
* COMPILE ERROR at the positional call sites in ai-chat-tools.service.ts,
* instead of a silent runtime "wrong argument" failure inside an agent tool.
*
* This made the old name-only drift-guard test
* (packages/mcp/test/unit/client-host-contract.test.mjs) redundant — tsc now
* enforces both names AND signatures — so that test was removed.
*/
export type DocmostClientLike = Pick<DocmostClient, DocmostClientMethod>;
export type DocmostClientConfig = {
apiUrl: string;
@@ -280,32 +118,7 @@ export type DocmostClientConfig = {
};
export interface DocmostClientCtor {
new (config: DocmostClientConfig): DocmostClientLike;
}
/**
* Local hand-mirror of the `SharedToolSpec` shape exported from
* `@docmost/mcp` (packages/mcp/src/tool-specs.ts). Same approach as
* `DocmostClientLike`: we do not import the ESM package's types directly across
* the CJS/ESM boundary. The registry itself has no runtime deps, but keeping the
* type local avoids coupling the server build to the package's type surface.
*
* `buildShape` is intentionally zod-agnostic: it returns a plain ZodRawShape
* built with whatever zod namespace the caller passes (the server passes its own
* zod v4; the MCP package passes its zod v3). See the registry module comment.
*/
export interface SharedToolSpec {
mcpName: string;
inAppKey: string;
description: string;
// Deferred-tool metadata (#332). Optional in this mirror so an older/stale
// @docmost/mcp build (pre-#332) still type-checks; the in-app catalog builder
// reads them defensively. The external /mcp server ignores both fields.
tier?: 'core' | 'deferred';
catalogLine?: string;
// Loose `z` on purpose: the registry is zod-agnostic so the server can pass
// its own zod (v4) and the MCP package its own (v3) into the same builder.
buildShape?: (z: any) => Record<string, unknown>;
new (config: DocmostClientConfig): DocmostClient;
}
/**
@@ -337,6 +150,19 @@ export type CommentSignalTrackerFactory = (options: {
debounceMs?: number;
}) => CommentSignalTrackerLike;
// Pure, no-network draw.io helpers (#424). These are plain functions on the
// module (NOT DocmostClient methods) — the in-app AI-SDK service calls them
// directly to wire drawioShapes / drawioGuide, mirroring the MCP server.
export type SearchShapesFn = (
query: string,
opts?: { category?: string; limit?: number },
) => Array<Record<string, unknown>>;
export type GetGuideSectionFn = (section?: string) => {
section: string;
content: string;
sections: string[];
};
interface DocmostMcpModule {
DocmostClient: DocmostClientCtor;
SHARED_TOOL_SPECS: Record<string, SharedToolSpec>;
@@ -344,6 +170,59 @@ interface DocmostMcpModule {
// loader in unit tests. The in-app layer treats an absent factory as "signal
// disabled" — a pure no-op that leaves tool results byte-identical.
createCommentSignalTracker?: CommentSignalTrackerFactory;
// Optional (#447): a deterministic hash of the tool-specs registry content,
// generated into build/ by the package's build. Absent on a pre-#447 build (or
// the mocked loader in unit tests) — the stale-check below is a NO-OP when it
// is missing, so an older build never wrongly fails startup.
REGISTRY_STAMP?: string;
// Pure, no-network draw.io helpers (#424) backing drawioShapes / drawioGuide.
// Those two specs are `inlineBothHosts` (they stay in SHARED_TOOL_SPECS for the
// shared contract but carry no execute — their catalog loader uses import.meta
// and can't be value-imported into the zod-agnostic tool-specs.ts), so the
// in-app service wires them INLINE off these helpers, mirroring the standalone
// MCP host. Exposed off the loaded module so the service and its test mocks can
// reach them.
searchShapes: SearchShapesFn;
getGuideSection: GetGuideSectionFn;
}
/**
* Recompute the REGISTRY_STAMP (#447) from the @docmost/mcp source tree, if it is
* present. Returns the stamp string, or `null` when the source is absent (a prod
* image ships only build/, no src/). MUST stay byte-for-byte identical to
* packages/mcp/scripts/gen-registry-stamp.mjs's `computeRegistryStamp` so the
* build-time and src-time hashes agree: same input file (src/tool-specs.ts), same
* normalization (CRLF -> LF, strip a single trailing newline), same sha256.
*
* DEV vs PROD detection is by FILE EXISTENCE, not NODE_ENV: we resolve the
* package's own directory from `require.resolve('@docmost/mcp')` (which points at
* build/index.js) and look for ../src/tool-specs.ts next to it. In a dev/test
* worktree that file exists; in a prod image (build/ only, src/ stripped) it does
* not, so this returns null and the caller skips the check. Any error (ENOENT, a
* bad resolve) is swallowed to null — the stale-check must NEVER break startup.
*
* Exported for unit testing (docmost-client.loader.spec.ts): the export keyword
* is behaviourally a no-op — the module-internal caller `loadDocmostMcp` is
* unaffected. The test drives the null (no-src) path and asserts this
* normalize+sha256 stays identical to the codegen's `computeRegistryStamp`.
*/
export function computeSrcRegistryStamp(packageEntry: string): string | null {
try {
// packageEntry is <pkg>/build/index.js; the source lives at <pkg>/src/.
const toolSpecsPath = join(
dirname(dirname(packageEntry)),
'src',
'tool-specs.ts',
);
if (!existsSync(toolSpecsPath)) return null; // prod: no src tree -> skip.
const source = readFileSync(toolSpecsPath, 'utf8');
const normalized = source.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
} catch {
// Never let a resolution/read hiccup break server startup — treat as "no
// src available" and skip the check (identical to the prod no-op path).
return null;
}
}
// TS with module:commonjs downlevels a literal `import()` to `require()`, which
@@ -368,6 +247,8 @@ export async function loadDocmostMcp(): Promise<{
DocmostClient: DocmostClientCtor;
sharedToolSpecs: Record<string, SharedToolSpec>;
createCommentSignalTracker?: CommentSignalTrackerFactory;
searchShapes: SearchShapesFn;
getGuideSection: GetGuideSectionFn;
}> {
if (!modulePromise) {
modulePromise = (async () => {
@@ -375,6 +256,23 @@ export async function loadDocmostMcp(): Promise<{
const mod = (await esmImport(
pathToFileURL(entry).href,
)) as DocmostMcpModule;
// #447 stale-build guard (dev/test only). The server loads the COMPILED
// build/ of @docmost/mcp, but the parity/tier guard tests read src/. If a
// tool spec is edited in src without rebuilding the package, build/ and src/
// silently diverge and the running server serves the OLD tools. Here we
// recompute the stamp from src/tool-specs.ts and compare it to the stamp
// baked into build/. In PROD the src tree is absent (image ships build/
// only), so computeSrcRegistryStamp returns null and this is a pure no-op.
const srcStamp = computeSrcRegistryStamp(entry);
if (
srcStamp !== null &&
typeof mod.REGISTRY_STAMP === 'string' &&
srcStamp !== mod.REGISTRY_STAMP
) {
throw new Error(
'@docmost/mcp build is stale (tool-specs changed since last build) — run: pnpm --filter @docmost/mcp build',
);
}
return mod;
})().catch((err) => {
// Do not cache a rejected import — allow the next call to retry.
@@ -396,5 +294,8 @@ export async function loadDocmostMcp(): Promise<{
// Optional: forwarded when present so the in-app layer can build the passive
// comment signal (#417); undefined on a stale build => signal disabled.
createCommentSignalTracker: mod.createCommentSignalTracker,
// Pure no-network draw.io helpers (#424); not client methods.
searchShapes: mod.searchShapes,
getGuideSection: mod.getGuideSection,
};
}
@@ -17,8 +17,10 @@ import { SHARED_TOOL_SPECS } from '../../../../../../packages/mcp/src/tool-specs
* This test fails the build if a spec is added to the registry but never wired
* in-app, if an `inAppKey` is renamed without updating the service, if the
* description drifts between the registry and the exposed tool, if the
* snake_case `mcpName` <-> camelCase `inAppKey` convention is broken, or if the
* exposed tool's input-schema keys diverge from the spec's `buildShape`.
* `mcpName === inAppKey` convention is broken (issue #412 unified the external
* MCP tool name with the in-app key — both are the same camelCase identifier),
* or if the exposed tool's input-schema keys diverge from the spec's
* `buildShape`.
*
* It does NOT need @docmost/mcp built: the registry is imported from TS source,
* and the ESM loader is mocked so `forUser()` never dynamically imports the
@@ -45,6 +47,16 @@ describe('SHARED_TOOL_SPECS contract parity', () => {
string,
loader.SharedToolSpec
>,
// Pure no-network draw.io helpers (#424). The contract test never executes
// a tool body, so type-correct stubs suffice (the real functions can't be
// imported here — drawio-shapes.ts uses import.meta, incompatible with the
// CommonJS jest transform).
searchShapes: (() => []) as unknown as loader.SearchShapesFn,
getGuideSection: (() => ({
section: 'index',
content: '',
sections: [],
})) as unknown as loader.GetGuideSectionFn,
});
const service = new AiChatToolsService(
tokenServiceStub as never,
@@ -64,13 +76,9 @@ describe('SHARED_TOOL_SPECS contract parity', () => {
afterAll(() => jest.restoreAllMocks());
// camelCase -> snake_case, matching the registry's mcpName convention.
const toSnake = (s: string) =>
s.replace(/[A-Z]/g, (c) => `_${c.toLowerCase()}`);
// Type as the (optional-buildShape) SharedToolSpec; the `satisfies` literal
// above otherwise narrows to a union where some members lack buildShape.
const specEntries = Object.entries(SHARED_TOOL_SPECS) as Array<
const specEntries = Object.entries(SHARED_TOOL_SPECS) as unknown as Array<
[string, loader.SharedToolSpec]
>;
@@ -86,8 +94,8 @@ describe('SHARED_TOOL_SPECS contract parity', () => {
expect(spec.inAppKey).toBe(registryKey);
});
it('mcpName is the snake_case form of inAppKey', () => {
expect(spec.mcpName).toBe(toSnake(spec.inAppKey));
it('mcpName equals inAppKey (unified camelCase name, #412)', () => {
expect(spec.mcpName).toBe(spec.inAppKey);
});
it('is exposed in-app under its inAppKey', () => {
@@ -27,16 +27,18 @@ import type { DocmostClientLike } from './docmost-client.loader';
*/
describe('tool tier metadata (#332)', () => {
it('core set is the documented 13 + searchInPage + insertFootnote (15)', () => {
expect(CORE_TOOL_KEYS).toHaveLength(15);
it('core set is the documented 13 + searchInPage + insertFootnote + getTree + getPageContext (17, #443)', () => {
expect(CORE_TOOL_KEYS).toHaveLength(17);
expect(CORE_TOOL_SET.has('searchInPage')).toBe(true); // #330, promoted to core
expect(CORE_TOOL_SET.has('insertFootnote')).toBe(true); // #410, promoted to core
expect(CORE_TOOL_SET.has('getTree')).toBe(true); // #443, promoted to core
expect(CORE_TOOL_SET.has('getPageContext')).toBe(true); // #443, promoted to core
// loadTools is a meta-tool, not a normal core key.
expect(CORE_TOOL_SET.has(LOAD_TOOLS_NAME)).toBe(false);
});
it('#410 image tools are DEFERRED, footnote tool is CORE', () => {
// insert_footnote is core (symmetric with editPageText); the image tools stay
// insertFootnote is core (symmetric with editPageText); the image tools stay
// deferred (rare, fat — loaded on demand). Assert both the spec tier and the
// CORE_TOOL_SET membership so a future tier edit that desyncs them fails here.
expect(SHARED_TOOL_SPECS.insertFootnote.tier).toBe('core');
@@ -123,7 +125,14 @@ describe('deferred catalog ↔ live forUser() toolset partition (#332, F3)', ()
DocmostClient: function () {
return {} as DocmostClientLike;
} as unknown as loader.DocmostClientCtor,
sharedToolSpecs: SHARED_TOOL_SPECS as Record<string, loader.SharedToolSpec>,
sharedToolSpecs: SHARED_TOOL_SPECS as unknown as Record<string, loader.SharedToolSpec>,
// Pure no-network draw.io helpers (#424); tool bodies are never executed here.
searchShapes: (() => []) as unknown as loader.SearchShapesFn,
getGuideSection: (() => ({
section: 'index',
content: '',
sections: [],
})) as unknown as loader.GetGuideSectionFn,
});
const service = new AiChatToolsService(
{
@@ -39,12 +39,14 @@ export interface ToolCatalogEntry {
/**
* CORE (always-active) in-app tool keys — 13 frequent/tiny tools + `searchInPage`
* (#330) + `insertFootnote` (#410). `searchInPage` is core because it is frequent
* for the editorial roles this feature targets; `insertFootnote` is core so the
* footnote tool is NOT hidden while its natural sibling `editPageText` is always
* active (that asymmetry is exactly what pushed the agent to write literal
* `^[...]`). `loadTools` is active too but is not a normal tool key (it is added
* to activeTools separately).
* (#330) + `insertFootnote` (#410) + `getTree`/`getPageContext` (#443).
* `searchInPage` is core because it is frequent for the editorial roles this
* feature targets; `insertFootnote` is core so the footnote tool is NOT hidden
* while its natural sibling `editPageText` is always active (that asymmetry is
* exactly what pushed the agent to write literal `^[...]`). `getTree` and
* `getPageContext` are the single-call navigation/lookup tools — core so the
* agent never has to loadTools just to orient itself. `loadTools` is active too
* but is not a normal tool key (it is added to activeTools separately).
*/
export const CORE_TOOL_KEYS = [
'searchPages',
@@ -60,12 +62,17 @@ export const CORE_TOOL_KEYS = [
'listComments',
'resolveComment',
'editPageText',
// #330 search_in_page — frequent for editorial sweeps; core despite predating
// #330 searchInPage — frequent for editorial sweeps; core despite predating
// the issue's tier list.
'searchInPage',
// #410 insert_footnote — core so pinpoint citations to already-written text
// #410 insertFootnote — core so pinpoint citations to already-written text
// don't degrade into literal `^[...]`; kept symmetric with editPageText.
'insertFootnote',
// #443 getTree + getPageContext — cheap single-call navigation/lookup tools
// (the core listPages even points to getTree); core so the agent never has
// to loadTools just to orient itself.
'getTree',
'getPageContext',
] as const;
/** O(1) membership test for the core tier. */
@@ -124,12 +131,9 @@ export const INLINE_TOOL_TIERS: Record<
// --- deferred inline ---
// NOTE: createPage, renamePage, movePage, deletePage, updatePageJson and
// exportPageMarkdown moved to @docmost/mcp's SHARED_TOOL_SPECS (#294); they
// carry their own deferred tier + catalogLine there.
updatePageContent: {
tier: 'deferred',
catalogLine:
"updatePageContent — replace a page's body (and optionally title) with new Markdown.",
},
// carry their own deferred tier + catalogLine there. updatePageContent moved
// there too as updatePageMarkdown (#411) — a shared registry spec now, so it
// is no longer an inline tier entry.
listSidebarPages: {
tier: 'deferred',
catalogLine:
@@ -141,7 +145,7 @@ export const INLINE_TOOL_TIERS: Record<
},
// NOTE: tableInsertRow, tableDeleteRow and tableUpdateCell moved to
// @docmost/mcp's SHARED_TOOL_SPECS (#294); they carry their own deferred tier +
// catalogLine there. getTable stays inline (its MCP name table_get breaks the
// catalogLine there. getTable stays inline (its MCP name tableGet breaks the
// snake_case(inAppKey) convention, so it has no shared spec).
// NOTE: checkNewComments moved to @docmost/mcp's SHARED_TOOL_SPECS (#294);
// it carries its own deferred tier + catalogLine there.
@@ -153,7 +157,7 @@ export const INLINE_TOOL_TIERS: Record<
// NOTE: sharePage moved to @docmost/mcp's SHARED_TOOL_SPECS (#294); it carries
// its own deferred tier + catalogLine there. transformPage stays inline (its
// schema deliberately diverges — it omits the deleteComments field the MCP
// docmost_transform exposes, a comment-deletion guardrail).
// docmostTransform exposes, a comment-deletion guardrail).
transformPage: {
tier: 'deferred',
catalogLine: "transformPage — run a sandboxed JS transform over a page's document.",
@@ -12,3 +12,22 @@ export class SearchResponseDto {
updatedAt: Date;
space: Partial<Space>;
}
// Response shape for the opt-in agent-lookup mode (#443, `substring: true`).
// Additive to the FTS response: carries the location (`path`), a windowed
// `snippet` around the first match and a per-response sort `score`. The MCP
// layer maps `id → pageId`; `slugId` is never exposed.
export class SearchLookupResponseDto {
id: string;
slugId: string;
title: string;
parentPageId: string | null;
// Ancestor titles from the space root down to the direct parent; [] for a
// root page.
path: string[];
// ~300–500 chars around the first match (or a leading text window / extended
// ts_headline fallback).
snippet: string;
// 0..1 float, meaningful ONLY for sorting within one response.
score: number;
}
@@ -30,6 +30,31 @@ export class SearchDTO {
@IsOptional()
@IsNumber()
offset?: number;
// --- Opt-in agent-lookup mode (#443). ------------------------------------
// These fields are ADDITIVE and default-off: a web client that sends none of
// them gets byte-identical FTS behaviour and result shape. They are only read
// by the substring/path/snippet code path in SearchService.searchPage.
//
// NOTE (standalone stdio vs stock upstream): stock upstream validates this DTO
// with `whitelist: true`, so an older server silently strips these unknown
// fields and the request degrades gracefully to the plain FTS behaviour.
// Enables the hybrid substring branch (title + text_content LIKE) merged with
// the existing FTS branch, plus tiered ranking, path and windowed snippet.
@IsOptional()
@IsBoolean()
substring?: boolean;
// Restrict the search to a page and all of its descendants (inclusive).
@IsOptional()
@IsString()
parentPageId?: string;
// Match titles only; do not scan text_content.
@IsOptional()
@IsBoolean()
titleOnly?: boolean;
}
export class SearchShareDTO extends SearchDTO {
@@ -60,6 +60,12 @@ export class SearchController {
}
}
// #443 graceful degradation: on EE/Typesense instances the request routes to
// the Typesense backend, which does NOT implement the opt-in agent-lookup
// mode. The `substring`/`parentPageId`/`titleOnly` fields are silently ignored
// and the response carries no `path`/`snippet`/`score` and no substring/tier
// ranking — it degrades to plain Typesense FTS. The native lookup mode below
// is Postgres-search-driver only.
if (this.environmentService.getSearchDriver() === 'typesense') {
return this.searchTypesense(searchDto, {
userId: user.id,
@@ -0,0 +1,95 @@
import {
computeLookupScore,
escapeLikePattern,
SearchLookupTier,
} from './search.service';
/**
* Pure-function coverage for the #443 agent-lookup helpers:
* - escapeLikePattern: LIKE-metacharacter escaping so `%`/`_`/`\` are literals
* (the acceptance-table requirement that a query of `%` or `_` does NOT match
* everything);
* - computeLookupScore: the tiered 0..1 ranking score, where a stronger tier
* always outranks a weaker one regardless of the in-tier secondary signal.
*
* The DB-touching branch (substring UNION FTS, path CTE, snippet window) is
* covered by the integration spec against the real schema.
*/
describe('escapeLikePattern', () => {
it('escapes the LIKE metacharacters % _ and \\', () => {
expect(escapeLikePattern('%')).toBe('\\%');
expect(escapeLikePattern('_')).toBe('\\_');
expect(escapeLikePattern('\\')).toBe('\\\\');
});
it('escapes the backslash FIRST so it does not double-escape %/_', () => {
// Input `\%` must become `\\` + `\%` = `\\\%`, not `\\%`.
expect(escapeLikePattern('\\%')).toBe('\\\\\\%');
});
it('leaves ordinary technical chars (. - / digits) untouched', () => {
expect(escapeLikePattern('backup-srv.local')).toBe('backup-srv.local');
expect(escapeLikePattern('10.0.12')).toBe('10.0.12');
expect(escapeLikePattern('WB-MGE-30D86B')).toBe('WB-MGE-30D86B');
expect(escapeLikePattern('a/b')).toBe('a/b');
});
it('escapes only the metacharacters in a mixed string', () => {
expect(escapeLikePattern('50%_off.zip')).toBe('50\\%\\_off.zip');
});
it('is null/undefined-safe', () => {
expect(escapeLikePattern(undefined as any)).toBe('');
expect(escapeLikePattern(null as any)).toBe('');
});
});
describe('computeLookupScore', () => {
it('keeps every score within (0, 1]', () => {
for (const tier of [
SearchLookupTier.TITLE_EXACT,
SearchLookupTier.TITLE_SUBSTRING,
SearchLookupTier.TEXT,
]) {
for (const secondary of [0, 0.001, 1, 100, 1e6]) {
const s = computeLookupScore({ tier, secondary });
expect(s).toBeGreaterThan(0);
expect(s).toBeLessThanOrEqual(1);
}
}
});
it('a stronger tier ALWAYS outranks a weaker tier, whatever the secondary', () => {
// Weak tier with a huge secondary must still lose to a strong tier with a
// tiny secondary — tiers dominate.
const strongLowSecondary = computeLookupScore({
tier: SearchLookupTier.TITLE_EXACT,
secondary: 0,
});
const weakHighSecondary = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 1e9,
});
expect(strongLowSecondary).toBeGreaterThan(weakHighSecondary);
});
it('within a tier a larger secondary sorts higher', () => {
const lo = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 0.1,
});
const hi = computeLookupScore({
tier: SearchLookupTier.TEXT,
secondary: 5,
});
expect(hi).toBeGreaterThan(lo);
});
it('treats a negative/absent secondary as 0', () => {
const zero = computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: 0 });
expect(computeLookupScore({ tier: SearchLookupTier.TEXT })).toBe(zero);
expect(
computeLookupScore({ tier: SearchLookupTier.TEXT, secondary: -5 }),
).toBe(zero);
});
});
+401 -2
View File
@@ -1,6 +1,9 @@
import { Injectable } from '@nestjs/common';
import { SearchDTO, SearchSuggestionDTO } from './dto/search.dto';
import { SearchResponseDto } from './dto/search-response.dto';
import {
SearchLookupResponseDto,
SearchResponseDto,
} from './dto/search-response.dto';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { sql } from 'kysely';
@@ -34,6 +37,53 @@ export function buildTsQuery(raw: string): string {
return tsquery(cleaned + '*');
}
// Escape the LIKE metacharacters (`%`, `_`, `\`) in a raw user query so every
// character — including `.`, `-`, `_`, `%`, `/` — is matched LITERALLY by a
// `col LIKE '%' || q || '%'` predicate. Without this, a query of `%` or `_`
// would match every row (see the #443 acceptance table). The backslash is the
// escape char (Postgres LIKE default), so it must be escaped first.
export function escapeLikePattern(raw: string): string {
return (raw ?? '')
.replace(/\\/g, '\\\\')
.replace(/%/g, '\\%')
.replace(/_/g, '\\_');
}
// Ranking tiers for the agent-lookup mode (#443), highest first. A hit's tier
// is the strongest way it matched; ties inside a tier break on a secondary
// signal (FTS rank, or first-match position). The numeric `score` returned to
// the caller is derived from (tier, secondary) and is meaningful ONLY for
// ordering within a single response.
export enum SearchLookupTier {
// Title equals the query, case-insensitively.
TITLE_EXACT = 3,
// Query is a substring of the title.
TITLE_SUBSTRING = 2,
// Query matched in the text (substring or FTS).
TEXT = 1,
}
export interface RankableHit {
tier: SearchLookupTier;
// Secondary in-tier signal, higher = better (e.g. ts_rank, or a
// position-derived closeness score). Defaults to 0.
secondary?: number;
}
// Map (tier, secondary) → a 0..1 float used ONLY to sort one response.
//
// Formula: score = (tier + squash(secondary)) / (maxTier + 1), where
// squash(x) = x / (1 + x) maps any non-negative secondary into [0, 1)
// so a stronger tier ALWAYS outranks a weaker one regardless of the secondary
// value, and within a tier a larger secondary sorts higher. maxTier is the top
// enum value (TITLE_EXACT = 3), so the divisor keeps the result in (0, 1].
export function computeLookupScore(hit: RankableHit): number {
const maxTier = SearchLookupTier.TITLE_EXACT;
const secondary = Math.max(0, hit.secondary ?? 0);
const squashed = secondary / (1 + secondary);
return (hit.tier + squashed) / (maxTier + 1);
}
@Injectable()
export class SearchService {
constructor(
@@ -50,12 +100,19 @@ export class SearchService {
userId?: string;
workspaceId: string;
},
): Promise<{ items: SearchResponseDto[] }> {
): Promise<{ items: SearchResponseDto[] | SearchLookupResponseDto[] }> {
const { query } = searchParams;
if (query.length < 1) {
return { items: [] };
}
// Opt-in agent-lookup mode (#443). Guarded by the `substring` flag so the
// web-UI (which never sets it) keeps byte-identical FTS behaviour below.
if (searchParams.substring) {
return this.searchPageLookup(searchParams, opts);
}
const searchQuery = buildTsQuery(query);
let queryResults = this.db
@@ -175,6 +232,348 @@ export class SearchService {
return { items: searchResults };
}
/**
* Agent-lookup search (#443, opt-in via `SearchDTO.substring`).
*
* ADDITIVE to the FTS path: runs a substring branch (title + optionally
* text_content, LIKE with metacharacters escaped) MERGED with the existing
* FTS branch, so technical tokens that the `english` tokenizer mangles
* (`backup-srv.local`, `10.0.12.5`, `WB-MGE-30D86B`) are still found — even
* when `buildTsQuery()` returns '' for a dotted/numeric query. Results carry a
* location (`path`), a windowed `snippet` and a per-response `score`.
*
* The whole method is only reached when `substring: true`; the web-UI never
* sets it, so its behaviour is unchanged.
*/
private async searchPageLookup(
searchParams: SearchDTO,
opts: { userId?: string; workspaceId: string },
): Promise<{ items: SearchLookupResponseDto[] }> {
const rawQuery = searchParams.query.trim();
if (!rawQuery) {
return { items: [] };
}
const limit = Math.min(Math.max(searchParams.limit || 10, 1), 50);
// Normalize the query the same way as the FTS / suggest path: f_unaccent +
// lower, done in SQL. `q` is the escaped LIKE pattern body (literal chars).
const likeBody = escapeLikePattern(rawQuery);
// Compare against `LOWER(f_unaccent(col))`; unaccent+lower the needle too.
const needle = sql<string>`LOWER(f_unaccent(${rawQuery}))`;
const likePattern = sql<string>`LOWER(f_unaccent(${'%' + likeBody + '%'}))`;
const tsQuery = buildTsQuery(rawQuery);
const hasTsQuery = tsQuery.length > 0;
// --- Resolve the space scope. ---------------------------------------------
// Mirrors searchPage: explicit spaceId, else the authenticated user's member
// spaces. The share path is not exposed to this opt-in mode.
let spaceIds: string[] = [];
if (searchParams.spaceId) {
spaceIds = [searchParams.spaceId];
} else if (opts.userId) {
spaceIds = await this.spaceMemberRepo.getUserSpaceIds(opts.userId);
} else {
return { items: [] };
}
if (spaceIds.length === 0) {
return { items: [] };
}
// --- Optional parentPageId subtree scope (inclusive). ---------------------
// Reuse the same recursive-descendants pattern used for share-scope.
let descendantIds: string[] | null = null;
if (searchParams.parentPageId) {
const descendants = await this.pageRepo.getPageAndDescendants(
searchParams.parentPageId,
{ includeContent: false },
);
descendantIds = descendants.map((p: any) => p.id);
if (descendantIds.length === 0) {
return { items: [] };
}
}
// --- Candidate query: substring (title + text) UNION FTS. -----------------
// We compute everything the ranker needs in SQL and pull only small columns
// (never the whole text_content) into Node:
// - titleExact / titleSub: tier signals
// - textMatchPos: 1-based position of the first text match (0 = none)
// - ftsRank: ts_rank for the FTS secondary signal (0 when no tsquery)
// - snippet: windowed ~500 chars around the first text match, or a leading
// text window (title-only hit), or an extended ts_headline fallback.
const N_BEFORE = 60; // chars of context before the first match
const SNIPPET_LEN = 500;
let candidates = this.db
.selectFrom('pages')
.select([
'pages.id as id',
'pages.slugId as slugId',
'pages.title as title',
'pages.parentPageId as parentPageId',
// Tier signals.
sql<boolean>`LOWER(f_unaccent(coalesce(pages.title, ''))) = ${needle}`.as(
'titleExact',
),
sql<boolean>`LOWER(f_unaccent(coalesce(pages.title, ''))) LIKE ${likePattern} ESCAPE '\\'`.as(
'titleSub',
),
// 1-based position of the first text match (0 = no text match).
sql<number>`strpos(LOWER(f_unaccent(coalesce(pages.text_content, ''))), ${needle})`.as(
'textMatchPos',
),
// FTS secondary signal (0 when the tsquery is empty).
hasTsQuery
? sql<number>`ts_rank(pages.tsv, to_tsquery('english', f_unaccent(${tsQuery})))`.as(
'ftsRank',
)
: sql<number>`0`.as('ftsRank'),
// Windowed snippet, computed entirely in SQL. Priority:
// 1. window around the first text match;
// 2. otherwise (titleOnly: no snippet; else) a leading window of the
// page text (title-only hit);
// 3. otherwise an extended ts_headline for pure-FTS hits.
//
// #443 snippet-position fix: the match position (`strpos`) is computed in
// the LOWER(f_unaccent(...)) space, but f_unaccent is NOT length-
// preserving (ß→ss, æ→ae, …→..., ½→ 1/2, full-width forms), so slicing
// the ORIGINAL text at that position was misaligned — a single expanding
// char before the match shifted the window (or ran it past end → empty).
// We now slice from the SAME LOWER(f_unaccent(...)) string so position
// and slice share one coordinate space. DELIBERATE trade-off: the snippet
// loses original case/diacritics — acceptable for an agent-facing snippet
// (position accuracy over original-glyph fidelity). The ts_headline branch
// matches over the ORIGINAL text itself, so it is unaffected and kept as-is.
searchParams.titleOnly
? sql<string>`''`.as('snippet')
: sql<string>`
coalesce(
case
when strpos(LOWER(f_unaccent(coalesce(pages.text_content, ''))), ${needle}) > 0
then substring(
LOWER(f_unaccent(coalesce(pages.text_content, '')))
from greatest(1, strpos(LOWER(f_unaccent(coalesce(pages.text_content, ''))), ${needle}) - ${N_BEFORE})
for ${SNIPPET_LEN}
)
when coalesce(pages.text_content, '') <> ''
then substring(LOWER(f_unaccent(pages.text_content)) from 1 for 300)
${
hasTsQuery
? sql`else ts_headline('english', coalesce(pages.text_content, ''), to_tsquery('english', f_unaccent(${tsQuery})), 'MinWords=25, MaxWords=40, MaxFragments=3')`
: sql``
}
end,
''
)
`.as('snippet'),
])
.where('pages.deletedAt', 'is', null)
.where('pages.spaceId', 'in', spaceIds);
if (descendantIds) {
candidates = candidates.where('pages.id', 'in', descendantIds);
}
// Match predicate: title substring OR (unless titleOnly) text substring OR
// (unless titleOnly) FTS. The substring branch runs even when the tsquery is
// empty — that is the dotted/numeric-token case the FTS path misses.
//
// #443 dead-index fix: these two LIKE predicates MUST match the GIN trgm
// index expressions EXACTLY for Postgres to use them. The indexes are on the
// coalesce-FREE expressions `LOWER(f_unaccent(title))` (#348's
// idx_pages_title_trgm) and `LOWER(f_unaccent(text_content))` (this PR's
// idx_pages_text_content_trgm). A `coalesce(col,'')` wrapper here would make
// the query expression differ from the index expression and force a Seq Scan
// on pages for every lookup. Dropping coalesce is SEMANTICALLY EQUIVALENT:
// `NULL LIKE '%q%'` is NULL (falsy), so a NULL title/text simply doesn't
// match — exactly as an empty string wouldn't match `%q%`.
candidates = candidates.where((eb) => {
const ors = [
eb(
sql`LOWER(f_unaccent(pages.title))`,
'like',
sql`${likePattern} ESCAPE '\\'`,
),
];
if (!searchParams.titleOnly) {
ors.push(
eb(
sql`LOWER(f_unaccent(pages.text_content))`,
'like',
sql`${likePattern} ESCAPE '\\'`,
),
);
if (hasTsQuery) {
ors.push(
sql<boolean>`pages.tsv @@ to_tsquery('english', f_unaccent(${tsQuery}))` as any,
);
}
}
return eb.or(ors);
});
// Pull a generous candidate set (before permission filtering + limit).
// Cap it so a pathological match set cannot blow up memory; 200 >> limit
// (max 50) leaves ample headroom for the post-permission truncation.
//
// #443 cap-ordering fix: the 200-cap MUST be deterministic and relevance-
// biased. Without an ORDER BY, Postgres returns an ARBITRARY 200 rows, so on
// a broad match set (common word / short substring) a strong TITLE_EXACT hit
// could be among the dropped rows while 200 low-tier TEXT hits fill the cap.
// We order by the SAME SQL tier proxies the Node ranker uses — title-exact,
// then title-substring, then fts-rank (nulls last), then earliest text-match
// position — so the cap keeps the strongest candidates. The Node-side final
// tier sort + slice(0, limit) below still runs and stays authoritative; this
// ORDER BY only decides WHICH candidates survive the 200-cap.
// NB: a BARE integer literal in ORDER BY is read by Postgres as an ordinal
// column position (`ORDER BY 0` → "position 0 is not in select list"), so the
// no-tsquery fallback is `0::float`, not `0`.
const ftsRankExpr = hasTsQuery
? sql`ts_rank(pages.tsv, to_tsquery('english', f_unaccent(${tsQuery})))`
: sql`0::float`;
const candidatesCapped = candidates
// Raw-SQL ORDER BY expressions: pass the full `<expr> <dir>` as ONE arg
// (the two-arg form treats a raw-SQL second arg as an ORDER BY position).
.orderBy(
sql`(LOWER(f_unaccent(coalesce(pages.title, ''))) = ${needle}) desc`,
)
.orderBy(
sql`(LOWER(f_unaccent(coalesce(pages.title, ''))) LIKE ${likePattern} ESCAPE '\\') desc`,
)
.orderBy(sql`${ftsRankExpr} desc nulls last`)
// Earlier text match first; strpos returns 0 for "no match", which would
// sort BEFORE a real (>=1) position under plain ASC, so push 0 to the end.
.orderBy(
sql`case when strpos(LOWER(f_unaccent(coalesce(pages.text_content, ''))), ${needle}) = 0 then 2147483647 else strpos(LOWER(f_unaccent(coalesce(pages.text_content, ''))), ${needle}) end asc`,
);
let rows: any[] = await candidatesCapped.limit(200).execute();
if (rows.length === 0) {
return { items: [] };
}
// --- Permissions BEFORE limit. --------------------------------------------
// Apply the existing page-level post-filter to the MERGED set, then rank and
// only THEN truncate to `limit` — never lose the permission filter.
if (opts.userId) {
const accessibleIds =
await this.pagePermissionRepo.filterAccessiblePageIds({
pageIds: rows.map((r) => r.id),
userId: opts.userId,
spaceId: searchParams.spaceId,
workspaceId: opts.workspaceId,
});
const accessibleSet = new Set(accessibleIds);
rows = rows.filter((r) => accessibleSet.has(r.id));
}
if (rows.length === 0) {
return { items: [] };
}
// --- Tiered ranking + dedup. ----------------------------------------------
// Rows are already unique by id (single pages scan), so no cross-branch
// dedup is needed here; the tier captures the strongest match reason.
const ranked = rows.map((r) => {
let tier: SearchLookupTier;
let secondary: number;
if (r.titleExact) {
tier = SearchLookupTier.TITLE_EXACT;
secondary = Number(r.ftsRank) || 0;
} else if (r.titleSub) {
tier = SearchLookupTier.TITLE_SUBSTRING;
secondary = Number(r.ftsRank) || 0;
} else {
tier = SearchLookupTier.TEXT;
// Prefer earlier text matches; map position → closeness in (0, 1].
const pos = Number(r.textMatchPos) || 0;
secondary =
pos > 0 ? 1 / (1 + (pos - 1) / 100) : Number(r.ftsRank) || 0;
}
return { row: r, tier, score: computeLookupScore({ tier, secondary }) };
});
ranked.sort((a, b) => b.score - a.score);
const top = ranked.slice(0, limit);
// --- Batch ancestor path (ONE recursive CTE, not N+1). --------------------
const pathById = await this.buildAncestorPaths(top.map((t) => t.row.id));
const items: SearchLookupResponseDto[] = top.map((t) => ({
id: t.row.id,
slugId: t.row.slugId,
title: t.row.title,
parentPageId: t.row.parentPageId ?? null,
path: pathById.get(t.row.id) ?? [],
snippet: (t.row.snippet ?? '')
.replace(/\r\n|\r|\n/g, ' ')
.replace(/\s+/g, ' ')
.trim(),
score: t.score,
}));
return { items };
}
/**
* Batch ancestor-titles helper (#443): ONE recursive CTE seeded with ALL hit
* ids, walking UP parentPageId. Returns a map hitId → ancestor titles ordered
* root → direct parent (the hit's own title is excluded). Root pages map to
* an empty array. Avoids the N+1 of a per-page breadcrumb call.
*/
private async buildAncestorPaths(
hitIds: string[],
): Promise<Map<string, string[]>> {
const result = new Map<string, string[]>();
if (hitIds.length === 0) return result;
// ancestry(hit_id, page_id, title, parent_page_id, depth): seed one row per
// hit at depth 0 (the hit itself), then walk to parents (increasing depth).
const rows = await this.db
.withRecursive('ancestry', (db) =>
db
.selectFrom('pages')
.select([
'pages.id as hitId',
'pages.id as pageId',
'pages.title as title',
'pages.parentPageId as parentPageId',
sql<number>`0`.as('depth'),
])
.where('pages.id', 'in', hitIds)
.unionAll((exp) =>
exp
.selectFrom('pages as p')
.innerJoin('ancestry as a', 'p.id', 'a.parentPageId')
.select([
'a.hitId as hitId',
'p.id as pageId',
'p.title as title',
'p.parentPageId as parentPageId',
sql<number>`a.depth + 1`.as('depth'),
]),
),
)
.selectFrom('ancestry')
.select(['hitId', 'title', 'depth'])
// depth 0 is the hit itself — excluded from the path.
.where('depth', '>', 0)
.orderBy('hitId')
// Larger depth = closer to the space root. Ordering DESC gives
// root → parent once collected.
.orderBy('depth', 'desc')
.execute();
for (const r of rows as any[]) {
const list = result.get(r.hitId) ?? [];
list.push(r.title);
result.set(r.hitId, list);
}
return result;
}
async searchSuggestions(
suggestion: SearchSuggestionDTO,
userId: string,
@@ -0,0 +1,55 @@
import { type Kysely, sql } from 'kysely';
/**
* #443 — trigram indexes for the opt-in agent-lookup search mode.
*
* The lookup mode adds a substring branch that runs leading-wildcard
* `LOWER(f_unaccent(col)) LIKE '%q%'` predicates on pages.title and
* pages.text_content. A leading wildcard cannot use a b-tree index, so without a
* GIN trigram index each such predicate is a sequential scan.
*
* - TITLE: the lookup-mode title predicate is `LOWER(f_unaccent(title)) LIKE
* '%q%'` (coalesce-free, so it can use a functional index), which is IDENTICAL
* to the one added for /search/suggest (#348). #348's perf-indexes migration
* already created `idx_pages_title_trgm` on `(LOWER(f_unaccent(title)))
* gin_trgm_ops`, so the title predicate is already covered — we do NOT
* re-create that index here (it would be redundant).
*
* - TEXT_CONTENT: NEW. The substring branch scans text_content when the query
* is not titleOnly. text_content is the large column, so a GIN trigram index
* on it is the meaningful acceleration for the lookup mode. The lookup search
* is ALWAYS space-scoped (spaceId or the user's member spaces), so on small
* instances a per-space sequential scan is tolerable — but the index turns the
* `%q%` text predicate into a Bitmap Index Scan and removes the only
* unbounded-per-space cost of the feature. We add it. The trade-off is disk +
* write amplification on page edits (GIN trigram indexes are larger and slower
* to update than b-trees); on the small instances this fork targets that cost
* is acceptable and the read win on agent lookups is the priority.
*
* DEPLOY-TIME LOCK WARNING: plain (non-CONCURRENT) CREATE INDEX — Kysely runs
* each migration in a transaction, so CONCURRENTLY is impossible. The build takes
* a SHARE lock that BLOCKS writes on `pages` for its duration. The text_content
* GIN build is the slow one and can take minutes on a large tenant. For large
* installations, run this in a maintenance window or build the index out-of-band
* with CREATE INDEX CONCURRENTLY before deploying (then `IF NOT EXISTS` no-ops
* here). Small/typical tenants are unaffected.
*/
export async function up(db: Kysely<any>): Promise<void> {
// The title predicate is served by #348's idx_pages_title_trgm — see header.
// Only the text_content index is introduced here.
// text_content trigram index. Its expression is coalesce-free —
// `LOWER(f_unaccent(text_content))` — to EXACTLY match the coalesce-free
// lookup-mode text substring predicate in search.service.ts, so Postgres can
// use it (a `coalesce(...)` mismatch would silently fall back to a Seq Scan).
await sql`
CREATE INDEX IF NOT EXISTS idx_pages_text_content_trgm
ON pages USING gin ((LOWER(f_unaccent(text_content))) gin_trgm_ops)
`.execute(db);
}
export async function down(db: Kysely<any>): Promise<void> {
// Only drop the index this migration introduced. idx_pages_title_trgm is owned
// by the #348 perf-indexes migration, so leave it for that migration's down().
await sql`DROP INDEX IF EXISTS idx_pages_text_content_trgm`.execute(db);
}
@@ -238,8 +238,9 @@ function convertReferenceFootnotes(markdown: string): string {
*
* LINE-ANCHORED (the same shape the canonical parser uses in
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
* very start and closes only on a `\n---` line. The retired `markdownToHtml`
* strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
* very start and closes only on a `\n---` line. The retired editor-ext
* `markdownToHtml` front-matter strip (removed in #347) closed on the FIRST
* `---` ANYWHERE (an unanchored close), so a value
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
* and leaked the rest into the body. An optional leading BOM is tolerated.
*/
@@ -143,7 +143,7 @@ export type DocmostMcpConfig = (
buf: Buffer,
mime: string,
) => { uri: string; sha256: string; size: number };
// Optional live/evict probes the package uses to keep stash_page's mirror
// Optional live/evict probes the package uses to keep stashPage's mirror
// counts honest under the store's FIFO eviction (mirror of the package's
// sink type); older bindings omit them.
has?: (uri: string) => boolean;
@@ -332,7 +332,7 @@ export class McpService implements OnModuleDestroy {
// Should never happen: handle() always stashes before delegating.
throw new UnauthorizedException('MCP authentication missing.');
}
// Inject the blob-sandbox sink after the auth decision so stash_page
// Inject the blob-sandbox sink after the auth decision so stashPage
// can store blobs in the shared in-RAM store regardless of which
// credential variant resolved. The sink (put/has/evict + uri↔id
// mapping) is owned by SandboxStore.asSink().
@@ -1,3 +1,8 @@
import * as fs from 'node:fs';
import * as os from 'node:os';
import { join } from 'node:path';
import Fastify, { FastifyInstance } from 'fastify';
import fastifyStatic from '@fastify/static';
import { resolveStaticAssetHeaders } from './static.module';
// Unit tests for the static-asset cache classifier extracted from the
@@ -33,3 +38,69 @@ describe('resolveStaticAssetHeaders', () => {
expect(headers['vary']).toBe('Accept-Encoding');
});
});
// Integration test proving the ACTUAL response header emitted by @fastify/static
// with the exact registration options StaticModule uses. This is the regression
// guard for #452: without `cacheControl: false`, @fastify/static writes its own
// `Cache-Control: public, max-age=0` AFTER the setHeaders callback, overwriting
// the immutable header — the /assets/ assertion below would then fail.
describe('static.module @fastify/static registration (integration)', () => {
let app: FastifyInstance;
let tmpDir: string;
beforeAll(async () => {
tmpDir = fs.mkdtempSync(join(os.tmpdir(), 'static-module-spec-'));
fs.mkdirSync(join(tmpDir, 'assets'), { recursive: true });
fs.mkdirSync(join(tmpDir, 'locales'), { recursive: true });
fs.writeFileSync(
join(tmpDir, 'assets', 'index-a1b2c3.js'),
'console.log(1);',
);
fs.writeFileSync(join(tmpDir, 'locales', 'en.json'), '{"hello":"world"}');
app = Fastify();
// Mirror StaticModule.onModuleInit's registration options exactly.
await app.register(fastifyStatic, {
root: tmpDir,
wildcard: false,
preCompressed: true,
cacheControl: false,
setHeaders: (res, filePath) => {
for (const [name, value] of Object.entries(
resolveStaticAssetHeaders(filePath),
)) {
res.setHeader(name, value);
}
},
});
await app.ready();
});
afterAll(async () => {
await app.close();
fs.rmSync(tmpDir, { recursive: true, force: true });
});
it('serves a hashed /assets/ file with an immutable, 1-year cache-control', async () => {
const res = await app.inject({
method: 'GET',
url: '/assets/index-a1b2c3.js',
});
expect(res.statusCode).toBe(200);
const cacheControl = res.headers['cache-control'];
expect(cacheControl).toContain('immutable');
expect(cacheControl).toContain('max-age=31536000');
});
it('serves a non-hashed /locales/ file WITHOUT an immutable cache-control', async () => {
const res = await app.inject({ method: 'GET', url: '/locales/en.json' });
expect(res.statusCode).toBe(200);
// resolveStaticAssetHeaders sets no cache-control here and cacheControl:false
// stops @fastify/static from adding one, so the browser revalidates by
// etag/last-modified — either an absent header or one without `immutable`.
const cacheControl = res.headers['cache-control'];
if (cacheControl !== undefined) {
expect(cacheControl).not.toContain('immutable');
}
});
});
@@ -115,6 +115,11 @@ export class StaticModule implements OnModuleInit {
// Serve the build-time .br/.gz neighbour when the client accepts it
// (see vite-plugin-compression2 in apps/client/vite.config.ts).
preCompressed: true,
// @fastify/static's default cacheControl:true writes its own
// Cache-Control (from maxAge, default 0) AFTER the setHeaders callback,
// silently overwriting the immutable header that resolveStaticAssetHeaders
// sets — disable it so setHeaders/resolveStaticAssetHeaders own the header.
cacheControl: false,
setHeaders: (res, filePath) => {
for (const [name, value] of Object.entries(
resolveStaticAssetHeaders(filePath),
@@ -182,6 +182,7 @@ describe('AiChatService run-stream attach [integration]', () => {
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatResumableStreamEnabled: () => true,
isAiChatFinalStepLockdownEnabled: () => false,
} as any,
registry,
);
@@ -499,6 +500,7 @@ describe('AiChatService run-stream attach [integration]', () => {
{
isAiChatDeferredToolsEnabled: () => false,
isAiChatResumableStreamEnabled: () => true,
isAiChatFinalStepLockdownEnabled: () => false,
} as any,
registry,
);
@@ -150,7 +150,7 @@ describe('AiChatService.stream [integration]', () => {
{} as any, // pageAccess (idem)
// environment (#332): keep deferred tool loading OFF for this lifecycle
// harness so the toolset/behavior is exactly as before.
{ isAiChatDeferredToolsEnabled: () => false } as any,
{ isAiChatDeferredToolsEnabled: () => false, isAiChatFinalStepLockdownEnabled: () => false } as any,
);
}
@@ -378,7 +378,7 @@ describe('AiChatService.stream [integration]', () => {
{} as any,
{} as any,
// #332: deferred tool loading ON — the property under test.
{ isAiChatDeferredToolsEnabled: () => true } as any,
{ isAiChatDeferredToolsEnabled: () => true, isAiChatFinalStepLockdownEnabled: () => false } as any,
);
}
@@ -0,0 +1,123 @@
import { randomUUID } from 'node:crypto';
import { Kysely, sql } from 'kysely';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
} from './db';
/**
* #443 dead-index guard — EXPLAIN on the REAL DB.
*
* The lookup mode's substring predicates run a leading-wildcard
* `LOWER(f_unaccent(col)) LIKE '%q%'`. Those are only fast when Postgres uses
* the GIN trigram indexes:
* - idx_pages_title_trgm on (LOWER(f_unaccent(title))) [#348]
* - idx_pages_text_content_trgm on (LOWER(f_unaccent(text_content))) [#443]
*
* Postgres uses a functional index ONLY when the query expression matches the
* index expression EXACTLY. The original lookup query wrapped the columns in
* `coalesce(col,'')`, which differs from the coalesce-FREE index expression and
* silently forced a Seq Scan on pages for EVERY lookup (the MCP client always
* sends substring:true). This test locks that in.
*
* Discriminator: `SET enable_seqscan = off` asks the planner "CAN this predicate
* use the index at all?" — which is exactly what the coalesce bug breaks. With
* seqscan disabled:
* - the coalesce-FREE (fixed) predicate plans a Bitmap Index Scan on the trgm
* index (no Seq Scan on pages);
* - the coalesce-WRAPPED (buggy) predicate cannot use the index and falls back
* to a Seq Scan on pages even though seqscan is disabled.
* We assert both to prove the fix and to keep the regression from silently
* returning.
*/
describe('SearchService agent-lookup EXPLAIN — trgm index is live [integration]', () => {
let db: Kysely<any>;
let workspaceId: string;
let spaceId: string;
async function insertPage(title: string, textContent: string): Promise<void> {
const id = randomUUID();
await db
.insertInto('pages')
.values({
id,
slugId: `slug-${id.slice(0, 12)}`,
title,
textContent,
spaceId,
workspaceId,
})
.execute();
}
// Run EXPLAIN (no ANALYZE — we only inspect the chosen plan) and return the
// concatenated plan text.
async function explain(query: string): Promise<string> {
const rows = await sql<{ 'QUERY PLAN': string }>`EXPLAIN ${sql.raw(query)}`.execute(
db,
);
return (rows.rows as any[]).map((r) => r['QUERY PLAN']).join('\n');
}
beforeAll(async () => {
db = getTestDb();
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
// Seed enough rows that a trigram index is a plausible plan. The content is
// varied so the '%needle%' pattern is selective.
for (let i = 0; i < 200; i++) {
await insertPage(
`seed-title-${i}`,
`seed body content number ${i} lorem ipsum dolor sit amet ${i}`,
);
}
await insertPage('backup-srv.local', 'the needle-token-xyz lives here');
// Keep the trgm indexes' stats fresh so the planner costs them correctly.
await sql`ANALYZE pages`.execute(db);
});
afterAll(async () => {
await destroyTestDb();
});
// Force the planner to answer "can the index be used?" rather than "is it
// cheaper than a seq scan on this size?". Restored after each test.
beforeEach(async () => {
await sql`SET enable_seqscan = off`.execute(db);
});
afterEach(async () => {
await sql`RESET enable_seqscan`.execute(db);
});
it('title predicate (coalesce-FREE, as fixed) uses idx_pages_title_trgm, not a Seq Scan', async () => {
const plan = await explain(
`SELECT id FROM pages WHERE LOWER(f_unaccent(title)) LIKE '%srv.local%'`,
);
expect(plan).toContain('idx_pages_title_trgm');
expect(plan).not.toMatch(/Seq Scan on pages/i);
});
it('text_content predicate (coalesce-FREE, as fixed) uses idx_pages_text_content_trgm, not a Seq Scan', async () => {
const plan = await explain(
`SELECT id FROM pages WHERE LOWER(f_unaccent(text_content)) LIKE '%needle-token%'`,
);
expect(plan).toContain('idx_pages_text_content_trgm');
expect(plan).not.toMatch(/Seq Scan on pages/i);
});
// Negative control: the OLD coalesce-wrapped predicate must NOT be able to use
// the index — even with seqscan disabled it can only Seq Scan pages. If this
// ever stops seq-scanning, the coalesce/index expressions have re-aligned and
// the guard above is no longer meaningful.
it('coalesce-WRAPPED text predicate (the bug) cannot use the index — falls to Seq Scan', async () => {
const plan = await explain(
`SELECT id FROM pages WHERE LOWER(f_unaccent(coalesce(text_content,''))) LIKE '%needle-token%'`,
);
expect(plan).not.toContain('idx_pages_text_content_trgm');
expect(plan).toMatch(/Seq Scan on pages/i);
});
});
@@ -0,0 +1,462 @@
import { randomUUID } from 'node:crypto';
import { Kysely } from 'kysely';
import { SearchService } from 'src/core/search/search.service';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import {
getTestDb,
destroyTestDb,
createWorkspace,
createSpace,
} from './db';
/**
* #443 — agent-lookup search mode, acceptance on the REAL DB schema.
*
* Exercises SearchService.searchPage(..., { substring: true }) against a
* migrated Postgres: substring matching of technical tokens the FTS tokenizer
* mangles (backup-srv.local, 10.0.12.5, WB-MGE-30D86B, "Теги: Docker"), the
* populated path + snippet, parentPageId subtree scoping, titleOnly, the empty
* result, LIKE-metacharacter escaping (`%`/`_` must NOT match everything), the
* permission post-filter applied BEFORE the limit, and the web-UI path staying
* on the legacy FTS shape when `substring` is absent.
*
* The tsv column is populated by the pages_tsvector_trigger on insert, so the
* FTS branch is exercised too.
*/
describe('SearchService agent-lookup mode [integration]', () => {
let db: Kysely<any>;
let service: SearchService;
let workspaceId: string;
let spaceId: string;
// Direct page insert (the shared createPage seeder omits text_content /
// parent_page_id, both of which this mode depends on). Returns the id.
async function insertPage(args: {
title: string;
textContent?: string;
parentPageId?: string | null;
spaceId?: string;
}): Promise<string> {
const id = randomUUID();
await db
.insertInto('pages')
.values({
id,
slugId: `slug-${id.slice(0, 12)}`,
title: args.title,
textContent: args.textContent ?? null,
parentPageId: args.parentPageId ?? null,
spaceId: args.spaceId ?? spaceId,
workspaceId,
})
.execute();
return id;
}
// Build a SearchService wired to the real DB + a real PageRepo (only its
// recursive-descendants method is used by this mode, and it needs only `db`),
// with lightweight stubs for the space-membership and permission repos so a
// test can drive scope + the permission post-filter explicitly.
function buildService(opts?: {
userSpaceIds?: string[];
// ids to KEEP after the permission post-filter; undefined = keep all.
accessibleIds?: string[];
}): SearchService {
const pageRepo = new PageRepo(db as any, null as any, null as any);
const spaceMemberRepo = {
getUserSpaceIds: async () => opts?.userSpaceIds ?? [spaceId],
};
const pagePermissionRepo = {
filterAccessiblePageIds: async ({ pageIds }: { pageIds: string[] }) =>
opts?.accessibleIds
? pageIds.filter((id) => opts.accessibleIds!.includes(id))
: pageIds,
};
return new SearchService(
db as any,
pageRepo as any,
{} as any, // shareRepo — unused by the lookup path
spaceMemberRepo as any,
pagePermissionRepo as any,
);
}
beforeAll(async () => {
db = getTestDb();
workspaceId = (await createWorkspace(db)).id;
spaceId = (await createSpace(db, workspaceId)).id;
service = buildService();
});
afterAll(async () => {
await destroyTestDb();
});
it('finds `backup-srv.local` by the fragment `srv.local`', async () => {
const pageId = await insertPage({
title: 'backup-srv.local',
textContent: 'A backup server node.',
});
const { items } = (await service.searchPage(
{ query: 'srv.local', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
expect(items.map((i: any) => i.id)).toContain(pageId);
const hit = items.find((i: any) => i.id === pageId);
expect(hit.title).toBe('backup-srv.local');
// slugId must never be part of the server response shape.
expect('slugId' in hit).toBe(true); // server carries it; MCP strips it
});
it('finds a page whose TEXT contains `10.0.12.5` by the fragment `10.0.12` (empty-tsquery case)', async () => {
const pageId = await insertPage({
title: 'Server inventory',
textContent: 'The backup box lives at IP: 10.0.12.5. Debian 12, backups.',
});
const { items } = (await service.searchPage(
{ query: '10.0.12', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// The windowed snippet must include the matched text.
expect(hit.snippet).toContain('10.0.12.5');
});
it('finds `WB-MGE-30D86B` (alphanumeric token with dashes) by title', async () => {
const pageId = await insertPage({
title: 'WB-MGE-30D86B',
textContent: 'Device page.',
});
const { items } = (await service.searchPage(
{ query: 'WB-MGE-30D86B', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// Exact title match → top tier (TITLE_EXACT=3) → score in [0.75, 1].
expect(hit.score).toBeGreaterThanOrEqual(0.75);
// And it is the top-ranked hit of its own result set.
expect(items[0].id).toBe(pageId);
});
it('finds every page whose text literally contains `Теги: Docker`', async () => {
const a = await insertPage({
title: 'Container host A',
textContent: 'Some notes.\nТеги: Docker, compose\nmore.',
});
const b = await insertPage({
title: 'Container host B',
textContent: 'Prelude.\nТеги: Docker\nepilogue.',
});
const noise = await insertPage({
title: 'Unrelated',
textContent: 'Теги: Kubernetes',
});
const { items } = (await service.searchPage(
{ query: 'Теги: Docker', spaceId, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(a);
expect(ids).toContain(b);
expect(ids).not.toContain(noise);
});
it('populates a non-empty `path` for a nested hit and `[]` for a root hit', async () => {
const root = await insertPage({ title: 'Infrastructure' });
const mid = await insertPage({ title: 'Datacenter A', parentPageId: root });
const leaf = await insertPage({
title: 'unique-nested-host',
parentPageId: mid,
});
const { items } = (await service.searchPage(
{ query: 'unique-nested-host', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === leaf);
expect(hit.path).toEqual(['Infrastructure', 'Datacenter A']);
const rootHits = (await service.searchPage(
{ query: 'Infrastructure', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const rootHit = rootHits.items.find((i: any) => i.id === root);
expect(rootHit.path).toEqual([]);
});
it('scopes to a subtree with parentPageId (cutting off sibling branches)', async () => {
const branchA = await insertPage({ title: 'BranchA-root' });
const inA = await insertPage({
title: 'scoped-target-xyz',
parentPageId: branchA,
});
const branchB = await insertPage({ title: 'BranchB-root' });
const inB = await insertPage({
title: 'scoped-target-xyz',
parentPageId: branchB,
});
const { items } = (await service.searchPage(
{
query: 'scoped-target-xyz',
spaceId,
substring: true,
parentPageId: branchA,
} as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(inA);
expect(ids).not.toContain(inB);
});
it('includes the parent page itself in the parentPageId subtree', async () => {
const parent = await insertPage({ title: 'self-included-parent' });
await insertPage({ title: 'child-of-self', parentPageId: parent });
const { items } = (await service.searchPage(
{
query: 'self-included-parent',
spaceId,
substring: true,
parentPageId: parent,
} as any,
{ workspaceId },
)) as any;
expect(items.map((i: any) => i.id)).toContain(parent);
});
it('titleOnly does NOT match on text_content', async () => {
const pageId = await insertPage({
title: 'Plain title',
textContent: 'body mentions the-secret-token here',
});
const withText = (await service.searchPage(
{ query: 'the-secret-token', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
expect(withText.items.map((i: any) => i.id)).toContain(pageId);
const titleOnly = (await service.searchPage(
{
query: 'the-secret-token',
spaceId,
substring: true,
titleOnly: true,
} as any,
{ workspaceId },
)) as any;
expect(titleOnly.items.map((i: any) => i.id)).not.toContain(pageId);
});
// #443 Fix #1 regression: f_unaccent is NOT length-preserving, so an
// expanding char (ß→ss, …→...) BEFORE the match shifted the strpos position
// relative to the ORIGINAL text and the snippet slice ran past end → empty.
// The position and the slice now share the LOWER(f_unaccent(...)) space, so
// the window is aligned and always contains the matched (unaccented) token.
it('returns a populated snippet when an unaccent-EXPANDING char precedes the match', async () => {
// 300 × `ß` (each f_unaccent-expands to `ss`) before the needle. Under the
// old code strpos returned a position ~593 in the expanded space but the
// slice ran over the ORIGINAL (~360 char) text → empty snippet, match lost.
const prefix = 'ß'.repeat(300);
const pageId = await insertPage({
title: 'Expanding-unaccent page',
textContent: `${prefix} needle-token-xyz trailing.`,
});
const { items } = (await service.searchPage(
{ query: 'needle-token-xyz', spaceId, substring: true } as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// Snippet must be non-empty AND contain the matched token (unaccented form).
expect(hit.snippet.length).toBeGreaterThan(0);
expect(hit.snippet).toContain('needle-token-xyz');
});
// #443 Fix #2 regression: >200 matching pages for a broad substring, with
// exactly ONE exact-title hit. Without an ORDER BY on the 200-cap the exact
// hit could be among the arbitrarily-dropped rows; the ORDER BY keeps the
// strongest candidates so it must survive the cap and rank at the top.
it('keeps an exact-title hit through the 200-cap on a >200-row match set', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
// 250 low-tier TEXT hits: the shared substring `capword` appears only in the
// body, never the title, so each is a TEXT-tier match (weakest tier).
for (let i = 0; i < 250; i++) {
await insertPage({
title: `filler-page-${i}`,
textContent: `body contains capword here #${i}`,
spaceId: isoSpace,
});
}
// Exactly one EXACT-title hit for the same query token.
const exact = await insertPage({
title: 'capword',
textContent: 'unrelated body text',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: 'capword', spaceId: isoSpace, substring: true, limit: 10 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
// The exact-title hit must survive the 200-cap and appear in the top `limit`.
expect(ids).toContain(exact);
// And, being TITLE_EXACT, it must be the single strongest hit.
expect(items[0].id).toBe(exact);
});
// #443 Fix #3: titleOnly matches only the title, so it must not leak the page
// body as the snippet (the old "first 300 chars of text_content" fallback).
it('titleOnly does NOT return a text-body snippet', async () => {
const pageId = await insertPage({
title: 'titleonly-snippet-page',
textContent: 'SECRET-BODY-CONTENT-NOT-IN-TITLE that must not leak.',
});
const { items } = (await service.searchPage(
{
query: 'titleonly-snippet-page',
spaceId,
substring: true,
titleOnly: true,
} as any,
{ workspaceId },
)) as any;
const hit = items.find((i: any) => i.id === pageId);
expect(hit).toBeDefined();
// The body text must not appear in the snippet; titleOnly → empty snippet.
expect(hit.snippet).not.toContain('SECRET-BODY-CONTENT-NOT-IN-TITLE');
expect(hit.snippet).toBe('');
});
it('returns [] (not an error) for a query that matches nothing', async () => {
const { items } = (await service.searchPage(
{
query: 'zzz-no-such-string-anywhere-42',
spaceId,
substring: true,
} as any,
{ workspaceId },
)) as any;
expect(items).toEqual([]);
});
it('a `%` query does NOT match everything (LIKE metacharacter escaped)', async () => {
// Fresh space so we can assert on total counts without cross-test noise.
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
await insertPage({ title: 'alpha', spaceId: isoSpace });
await insertPage({ title: 'beta', spaceId: isoSpace });
const literal = await insertPage({
title: '100%-coverage',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: '%', spaceId: isoSpace, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
// `%` is a literal → matches only the page that actually contains '%'.
expect(ids).toContain(literal);
expect(ids).not.toContain(
items.find((i: any) => i.title === 'alpha')?.id,
);
expect(items.length).toBe(1);
});
it('an `_` query does NOT match everything (LIKE metacharacter escaped)', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const svc = buildService({ userSpaceIds: [isoSpace] });
await insertPage({ title: 'gamma', spaceId: isoSpace });
const literal = await insertPage({
title: 'snake_case_name',
spaceId: isoSpace,
});
const { items } = (await svc.searchPage(
{ query: '_', spaceId: isoSpace, substring: true, limit: 50 } as any,
{ workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(literal);
expect(items.length).toBe(1);
});
it('applies the permission post-filter to the MERGED set BEFORE the limit', async () => {
const isoSpace = (await createSpace(db, workspaceId)).id;
const keep = await insertPage({
title: 'perm-visible-target',
spaceId: isoSpace,
});
const hidden = await insertPage({
title: 'perm-hidden-target',
spaceId: isoSpace,
});
// Authenticated (userId set) so the permission filter runs; only `keep` is
// accessible. limit 1 must NOT be able to select `hidden`.
const svc = buildService({
userSpaceIds: [isoSpace],
accessibleIds: [keep],
});
const { items } = (await svc.searchPage(
{
query: 'perm-',
spaceId: isoSpace,
substring: true,
limit: 1,
} as any,
{ userId: 'user-1', workspaceId },
)) as any;
const ids = items.map((i: any) => i.id);
expect(ids).toContain(keep);
expect(ids).not.toContain(hidden);
});
it('web-UI path (no `substring` flag) keeps the legacy FTS response shape', async () => {
await insertPage({
title: 'legacy shape page',
textContent: 'searchable legacyword content',
});
const { items } = (await service.searchPage(
{ query: 'legacyword', spaceId } as any,
{ userId: 'user-1', workspaceId },
)) as any;
// Legacy hits carry rank + highlight + space, and NO path/snippet/score.
const hit = items[0];
expect(hit).toBeDefined();
expect('rank' in hit).toBe(true);
expect('highlight' in hit).toBe(true);
expect('path' in hit).toBe(false);
expect('snippet' in hit).toBe(false);
expect('score' in hit).toBe(false);
});
});
-3
View File
@@ -11,9 +11,6 @@
"main": "dist/index.js",
"module": "./src/index.ts",
"types": "dist/index.d.ts",
"dependencies": {
"marked": "17.0.5"
},
"devDependencies": {
"@vitest/coverage-v8": "4.1.6",
"vitest": "4.1.6"
-1
View File
@@ -18,7 +18,6 @@ export * from "./lib/excalidraw";
export * from "./lib/embed";
export * from "./lib/html-embed/html-embed";
export * from "./lib/mention";
export * from "./lib/markdown";
export * from "./lib/search-and-replace";
export * from "./lib/embed-provider";
export * from "./lib/subpages";
@@ -14,7 +14,8 @@ import {
* ProseMirror JSON directly (never running the editor's plugins), so the
* canonical footnote topology was never enforced on those writes. The consumers
* of this editor-ext copy are: the server markdown/HTML import
* (`markdownToHtml -> htmlToJson` in import.service / file-import-task.service),
* (`markdownToProseMirror` from @docmost/prosemirror-markdown in import.service /
* file-import-task.service),
* `PageService` create/update (`parseProsemirrorContent` for the JSON/markdown/
* HTML REST write paths), and the client markdown PASTE path
* (`markdown-clipboard.ts`). (The MCP package mirrors this canonicalizer in
@@ -1,131 +0,0 @@
import { describe, it, expect } from "vitest";
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
import { markdownToHtml } from "../markdown/utils/marked.utils";
import { extractFootnoteDefinitions } from "../markdown/utils/footnote.marked";
// HTML the editor-ext nodes render (sup[data-footnote-ref], section/div).
const HTML =
`<p>Water<sup data-footnote-ref data-id="fn1"></sup> and clay<sup data-footnote-ref data-id="fn2"></sup>.</p>` +
`<section data-footnotes>` +
`<div data-footnote-def data-id="fn1"><p>First note.</p></div>` +
`<div data-footnote-def data-id="fn2"><p>Second note.</p></div>` +
`</section>`;
describe("footnote markdown round-trip", () => {
it("HTML -> Markdown produces pandoc footnote syntax", () => {
const md = htmlToMarkdown(HTML);
expect(md).toContain("[^fn1]");
expect(md).toContain("[^fn2]");
expect(md).toContain("[^fn1]: First note.");
expect(md).toContain("[^fn2]: Second note.");
});
it("Markdown -> HTML rebuilds the footnote nodes' HTML", async () => {
const md = htmlToMarkdown(HTML);
const html = await markdownToHtml(md);
expect(html).toContain('data-footnote-ref data-id="fn1"');
expect(html).toContain('data-footnote-ref data-id="fn2"');
expect(html).toContain("data-footnotes");
expect(html).toContain('data-footnote-def data-id="fn1"');
expect(html).toContain("First note.");
expect(html).toContain("Second note.");
});
it("preserves a [^id]: line shown inside a fenced code block (not a definition)", async () => {
// A document that DOCUMENTS footnote syntax inside a code fence. The
// `[^demo]: ...` line is example text, not a real definition, and must
// survive the Markdown -> HTML conversion verbatim.
const md = [
"Here is how footnotes look:",
"",
"```markdown",
"Some text[^demo]",
"",
"[^demo]: this is the definition",
"```",
"",
"End of doc.",
].join("\n");
const html = await markdownToHtml(md);
// The example definition line is kept inside the rendered code block.
expect(html).toContain("[^demo]: this is the definition");
// It did NOT get pulled out into a real footnotes section.
expect(html).not.toContain("data-footnotes");
expect(html).not.toContain("data-footnote-def");
});
it("extractFootnoteDefinitions keeps the FIRST duplicate definition and reuses markers", () => {
// Two definitions share id `d`, and the body has two `[^d]` markers. Under
// the import model (#166) duplicate definition ids are FIRST-WINS: only the
// first definition is kept; markers are NEVER rewritten, so the two `[^d]`
// references reuse the single footnote.
const md = [
"See here[^d] and there[^d].",
"",
"[^d]: first",
"[^d]: second",
].join("\n");
const { body, section } = extractFootnoteDefinitions(md);
const defIds = Array.from(
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
expect(defIds).toEqual(["d"]); // first-wins: one definition
expect(section).toContain("first");
expect(section).not.toContain("second"); // duplicate dropped
// Both markers stay `[^d]` (reuse) — no `d__2` minting.
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
(m) => m[1],
);
expect(refIds).toEqual(["d", "d"]);
});
it("extractFootnoteDefinitions is DETERMINISTIC and stable (same input -> same output)", () => {
// The output must be a pure function of the input markdown so importing the
// same source twice (or via the editor and the MCP mirror) is identical.
const md = [
"See[^d] one[^d] two[^d].",
"",
"[^d]: first",
"[^d]: second",
"[^d]: third",
].join("\n");
const run = () => {
const { body, section } = extractFootnoteDefinitions(md);
const defIds = Array.from(
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
(m) => m[1],
);
return { defIds, refIds };
};
const a = run();
const b = run();
expect(a).toEqual(b);
// First-wins: one kept definition `d`; all three reuse markers stay `d`.
expect(a.defIds).toEqual(["d"]);
expect(a.refIds).toEqual(["d", "d", "d"]);
});
it("markdownToHtml with a reused id renders ONE shared footnote def", async () => {
const md = [
"See here[^d] and there[^d].",
"",
"[^d]: first",
"[^d]: second",
].join("\n");
const html = await markdownToHtml(md);
const defIds = Array.from(
html.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
expect(defIds).toEqual(["d"]); // one shared definition
expect(html).toContain("first");
expect(html).not.toContain("second");
});
});
@@ -103,8 +103,9 @@ interface CollisionPlan {
* `X__2`, `X__3`, collision-bumped) so it survives as a distinct footnote — which,
* having no matching reference, then falls under the normal orphan policy. It is
* only ever dropped for lacking a reference, never for colliding. The IMPORT
* paths (footnote.marked.ts / MCP extractFootnotes) instead apply first-wins +
* drop + warn for duplicate definitions; that divergence is intentional — import
* paths (@docmost/prosemirror-markdown / MCP extractFootnotes) instead apply
* first-wins + drop + warn for duplicate definitions; that divergence is
* intentional — import
* is an agent-authored artifact we sanitize, the editor is live user data we must
* not lose.
*
@@ -6,8 +6,9 @@ import { deriveFootnoteId } from "./footnote-util";
*
* `deriveFootnoteId` lives ONLY in editor-ext now — it is used by
* `resolveCollisions` (re-id of a duplicate definition) and `footnotePastePlugin`
* (re-id of a pasted colliding definition). The MCP/marked import paths no longer
* derive ids (duplicate definitions there are first-wins-dropped, #166), so there
* (re-id of a pasted colliding definition). The MCP / @docmost/prosemirror-markdown
* import paths no longer derive ids (duplicate definitions there are
* first-wins-dropped, #166), so there
* is no cross-package copy and no parity test to keep in sync. This table pins the
* deterministic scheme so a future change to it is a conscious one.
*/
@@ -63,8 +63,9 @@ export function generateFootnoteId(): string {
* its own seen-set before requesting the next derived id.
*
* Used only inside editor-ext now (resolveCollisions for a re-id'd duplicate
* DEFINITION, and footnotePastePlugin). The MCP/marked import paths no longer
* derive ids — duplicate definitions there are first-wins-dropped (#166) — so
* DEFINITION, and footnotePastePlugin). The MCP / @docmost/prosemirror-markdown
* import paths no longer derive ids — duplicate definitions there are
* first-wins-dropped (#166) — so
* there is no cross-package copy to keep in sync. The golden table in
* footnote-util.derive-id.test.ts pins the scheme.
*/
@@ -1,68 +0,0 @@
import { describe, it, expect } from "vitest";
import { generateJSON } from "@tiptap/html";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
import { markdownToHtml } from "../markdown/utils/marked.utils";
import { TiptapImage } from "./image";
// Minimal schema for parsing markdownToHtml output back to JSON (mirrors
// image.spec.ts), so we can assert the recovered caption EXACTLY.
const parseExtensions = [Document, Paragraph, Text, TiptapImage];
// Lossless markdown round-trip for image captions (issue #221). An image WITH a
// caption can't be expressed as `![alt](src)`, so it is emitted as a raw <img>
// (carrying data-caption) wrapped in a block <div>, the same trick the <video>
// rule uses. marked passes the raw HTML through, so markdownToHtml keeps the
// data-caption, and the image extension's parseHTML restores the attribute.
describe("image caption markdown round-trip", () => {
it("HTML -> Markdown emits a raw <img data-caption> for captioned images", () => {
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
const md = htmlToMarkdown(html);
expect(md).toContain("data-caption=\"A grey cat\"");
expect(md).toContain('src="/files/a.png"');
expect(md).toContain('alt="cat"');
// It must NOT degrade to the lossy ![]() form.
expect(md).not.toContain("![cat]");
});
it("Markdown -> HTML restores data-caption on the <img>", async () => {
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
const md = htmlToMarkdown(html);
const back = await markdownToHtml(md);
expect(back).toContain('data-caption="A grey cat"');
expect(back).toContain('src="/files/a.png"');
});
it("special characters in the caption survive the round-trip (escaped)", async () => {
// The source caption is the decoded string `Tom & "Jerry"` (both an `&` and
// a `"`). escapeHtmlAttr must encode `&` -> `&amp;` and `"` -> `&quot;`.
const html = `<p><img src="/files/a.png" data-caption='Tom &amp; &quot;Jerry&quot;'></p>`;
const md = htmlToMarkdown(html);
// (a) The intermediate Markdown must carry the EXACT escaped attribute. This
// fails if escapeHtmlAttr stopped escaping `"` (attribute break-out:
// data-caption="Tom & "Jerry"") or double-encoded `&` (`&amp;amp;`).
expect(md).toContain('data-caption="Tom &amp; &quot;Jerry&quot;"');
const back = await markdownToHtml(md);
expect(back).toContain("data-caption=");
expect(back).toContain("Jerry");
expect(back).toContain("Tom");
// (b) Re-parse the rendered HTML through the image extension's parseHTML and
// assert the recovered caption is EXACTLY the original (no corruption, loss,
// or double-encoding).
const json = generateJSON(back, parseExtensions);
expect(json.content?.[0]?.attrs?.caption).toBe('Tom & "Jerry"');
});
it("caption-less images stay a clean ![alt](src) with no raw HTML", () => {
const html = `<p><img src="/files/a.png" alt="cat"></p>`;
const md = htmlToMarkdown(html);
expect(md).toContain("![cat](/files/a.png)");
expect(md).not.toContain("data-caption");
expect(md).not.toContain("<img");
});
});
@@ -1,105 +0,0 @@
import { describe, expect, it } from "vitest";
import { htmlEmbedExtension } from "./utils/html-embed.marked";
import { markdownToHtml } from "./index";
import { encodeHtmlEmbedSource } from "../html-embed/html-embed";
// CONTRACT tests for the marked block tokenizer that rebuilds an htmlEmbed node
// from the `<!--html-embed:BASE64-->` marker (html-embed.marked.ts), plus the
// observable round-trip through markdownToHtml.
//
// These pin the REAL tokenizer behaviour the import path depends on:
// - the tokenizer rule is anchored (^) and only accepts the base64 alphabet
// [A-Za-z0-9+/=], so a marker with non-base64 chars is NOT tokenized and
// survives as a literal HTML comment (not silently turned into something the
// server's strip no longer recognizes);
// - start() reports the correct index of the next marker so marked invokes the
// tokenizer at the right offset when a marker sits mid-document / after text;
// - a marker with surrounding text on the SAME line is split out into its own
// embed div while the surrounding text becomes ordinary paragraphs.
//
// The contract is asserted against the actual exported extension and pipeline —
// no behaviour is invented; the expectations were read off the real tokenizer.
const SAMPLE = "<b>x</b>";
const ENC = encodeHtmlEmbedSource(SAMPLE);
describe("htmlEmbed marked tokenizer — start()", () => {
it("returns the index of a marker that sits mid-document", () => {
const src = `hello world <!--html-embed:${ENC}-->`;
expect(htmlEmbedExtension.start(src)).toBe(src.indexOf("<!--html-embed:"));
});
it("returns 0 when the marker is at the very start", () => {
expect(htmlEmbedExtension.start(`<!--html-embed:${ENC}-->`)).toBe(0);
});
it("returns -1 when there is no marker", () => {
expect(htmlEmbedExtension.start("no marker here")).toBe(-1);
});
});
describe("htmlEmbed marked tokenizer — tokenizer()", () => {
it("tokenizes a marker at the start of the input, capturing the base64 payload", () => {
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`);
expect(token).toBeTruthy();
expect(token!.type).toBe("htmlEmbed");
expect(token!.raw).toBe(`<!--html-embed:${ENC}-->`);
expect(token!.encoded).toBe(ENC);
});
it("tokenizes an EMPTY marker (the [A-Za-z0-9+/=]* class allows zero chars)", () => {
const token = htmlEmbedExtension.tokenizer("<!--html-embed:-->");
expect(token).toBeTruthy();
expect(token!.encoded).toBe("");
expect(token!.raw).toBe("<!--html-embed:-->");
});
it("does NOT tokenize when text precedes the marker (rule is anchored ^)", () => {
// marked relies on start() to advance to the marker; the tokenizer itself
// only matches at offset 0, so a non-anchored call returns undefined.
expect(
htmlEmbedExtension.tokenizer(`hello <!--html-embed:${ENC}-->`),
).toBeUndefined();
});
it("does NOT tokenize a marker containing a non-base64 char ('$')", () => {
expect(
htmlEmbedExtension.tokenizer("<!--html-embed:ab$cd-->"),
).toBeUndefined();
});
it("does NOT tokenize a marker containing a space", () => {
expect(
htmlEmbedExtension.tokenizer("<!--html-embed:ab cd-->"),
).toBeUndefined();
});
it("renderer emits the embed div the node's parseHTML recognizes", () => {
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`)!;
const html = htmlEmbedExtension.renderer(token as any);
expect(html).toBe(
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
);
});
});
describe("htmlEmbed marked tokenizer — markdownToHtml round-trip", () => {
it("splits a marker out of surrounding same-line text into its own embed div", async () => {
const html = await markdownToHtml(`before <!--html-embed:${ENC}--> after`);
// The marker became the embed div...
expect(html).toContain(
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
);
// ...and the surrounding text survived as ordinary paragraph content.
expect(html).toContain("before");
expect(html).toContain("after");
});
it("leaves a marker with non-base64 chars as a literal comment (NOT an embed div)", async () => {
const html = await markdownToHtml("<!--html-embed:ab$cd-->");
// It is NOT tokenized into an embed div the server would strip...
expect(html).not.toContain('data-type="htmlEmbed"');
// ...it passes through unchanged as a literal HTML comment.
expect(html).toContain("<!--html-embed:ab$cd-->");
});
});
@@ -1,2 +0,0 @@
export * from "./utils/marked.utils";
export * from "./utils/turndown.utils";
@@ -1,112 +0,0 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml, htmlToMarkdown } from "./index";
import {
encodeHtmlEmbedSource,
decodeHtmlEmbedSource,
} from "../html-embed/html-embed";
// SECURITY (Variant C admin gate, import attack surface).
//
// The markdown import path is the only write path where an htmlEmbed reaches
// the server purely from file bytes (no editor / collab socket). The marked
// tokenizer in `html-embed.marked.ts` and the turndown rule in
// `turndown.utils.ts` are what materialize the `<!--html-embed:BASE64-->`
// marker into the `<div data-type="htmlEmbed" data-source="BASE64">` element
// that the server then parses into an htmlEmbed node and the admin gate strips.
//
// If either the tokenizer regex or the turndown rule shape drifts, the marker
// would either (a) stop becoming an htmlEmbed node (silently dropping admin
// content) or (b) become some OTHER tag the server's `hasHtmlEmbedNode` no
// longer recognizes (a strip bypass). These tests pin the marker <-> embed-div
// contract that the server-side strip relies on. editor-ext had ZERO tests
// before this file; this adds the runner + the round-trip coverage.
// The server parses the embed div by matching `data-type="htmlEmbed"` and
// decoding `data-source`; mirror that here so the assertion is exactly what the
// real `htmlToJson` -> htmlEmbed node parse depends on (the node's parseHTML in
// html-embed.ts uses the same selector + decodeHtmlEmbedSource).
const EMBED_DIV_RE = /<div[^>]*\bdata-type="htmlEmbed"[^>]*>/;
function extractEmbedSource(html: string): string | undefined {
const div = EMBED_DIV_RE.exec(html);
if (!div) return undefined;
const enc = /data-source="([^"]*)"/.exec(div[0]);
if (!enc) return undefined;
return decodeHtmlEmbedSource(enc[1]);
}
// Replicates the server's `hasHtmlEmbedNode` decision against the embed *div*
// (the HTML form the server immediately converts to JSON). If this matches, the
// server's JSON-level `hasHtmlEmbedNode` will too, because htmlToJson maps this
// exact div to an htmlEmbed node.
function htmlHasHtmlEmbed(html: string): boolean {
return EMBED_DIV_RE.test(html);
}
describe("markdown <!--html-embed--> import round-trip", () => {
const source = "<script>x</script>";
it("markdownToHtml turns the marker into an htmlEmbed div carrying the source", async () => {
const md = "<!--html-embed:" + encodeHtmlEmbedSource(source) + "-->";
const html = await markdownToHtml(md);
// The marker became the embed div the server recognizes as an htmlEmbed
// node (so the server's hasHtmlEmbedNode would match it after htmlToJson).
expect(htmlHasHtmlEmbed(html)).toBe(true);
// The decoded source is the original script, intact.
expect(extractEmbedSource(html)).toBe(source);
// The raw script is NOT inlined into the HTML — it stays base64 in the
// attribute (the marker itself must not be a direct injection vector).
expect(html).not.toContain("<script>x</script>");
});
it("preserves UTF-8 / special chars in the embedded source", async () => {
const utf8 = '<script>console.log("héllo → 世界")</script>';
const md = "<!--html-embed:" + encodeHtmlEmbedSource(utf8) + "-->";
const html = await markdownToHtml(md);
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe(utf8);
});
it("an empty marker still produces an htmlEmbed div (empty source)", async () => {
const html = await markdownToHtml("<!--html-embed:-->");
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe("");
});
it("round-trips htmlToMarkdown -> markdownToHtml preserving the embed marker", async () => {
const encoded = encodeHtmlEmbedSource(source);
// NOTE: turndown drops a *blank* (childless) element before any custom rule
// runs, and the htmlEmbed div is normally childless. The export pipeline
// therefore must give the rule a non-blank div to fire on; we add an inert
// text child here to exercise the real turndown htmlEmbed rule. (A blank
// embed div serializing to "" is asserted separately below as a documented
// edge so this contract drift is visible.)
const startHtml = `<div data-type="htmlEmbed" data-source="${encoded}">x</div>`;
// Export to markdown: the turndown rule emits the <!--html-embed:..-->
// marker (lossless, inert in plain markdown viewers).
const md = htmlToMarkdown(startHtml);
expect(md).toContain("<!--html-embed:" + encoded + "-->");
// Re-import: the marker round-trips back into an embed div with the same
// decoded source — this is the marker <-> embed-div contract the server's
// import strip depends on.
const html = await markdownToHtml(md);
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe(source);
});
it("documents that a BLANK embed div serializes to empty markdown (turndown drops childless blocks)", () => {
const encoded = encodeHtmlEmbedSource(source);
const blank = `<div data-type="htmlEmbed" data-source="${encoded}"></div>`;
// This pins current behavior so a future change to the turndown rule (e.g.
// making it fire on blank nodes) is caught rather than silently shipping.
expect(htmlToMarkdown(blank)).toBe("");
});
it("the base64 codec itself round-trips (no '<' leaks into the attribute)", () => {
const encoded = encodeHtmlEmbedSource(source);
expect(encoded).not.toContain("<");
expect(decodeHtmlEmbedSource(encoded)).toBe(source);
});
});
@@ -1,29 +0,0 @@
/**
* Flexible `basename` implementation for node and the browser
* @see https://stackoverflow.com/a/59907288/2228771
*/
export function getBasename(path: string) {
// make sure the basename is not empty, if string ends with separator
let end = path.length - 1;
while (path[end] === '/' || path[end] === '\\') {
--end;
}
// support mixing of Win + Unix path separators
const i1 = path.lastIndexOf('/', end);
const i2 = path.lastIndexOf('\\', end);
let start: number;
if (i1 === -1) {
if (i2 === -1) {
// no separator in the whole thing
return path;
}
start = i2;
} else if (i2 === -1) {
start = i1;
} else {
start = Math.max(i1, i2);
}
return path.substring(start + 1, end + 1);
}
@@ -1,33 +0,0 @@
/**
* Shared pieces for the two callout tokenizers — `callout.marked.ts` (the
* `:::type` fenced form) and `github-callout.marked.ts` (the `> [!type]` GitHub
* alert form). Both emit the SAME callout node, so the banner type dictionary
* and the HTML renderer live here once instead of drifting apart in two files.
* The tokenizers themselves stay separate (different syntaxes / source matching).
*/
/** The four callout banner types the editor schema supports. */
export const CALLOUT_TYPES = ['info', 'success', 'warning', 'danger'] as const;
export type CalloutType = (typeof CALLOUT_TYPES)[number];
/**
* Coerce an arbitrary type name onto a supported banner type, defaulting to
* `info` for anything unrecognized (the shared fallback both tokenizers use).
*/
export function normalizeCalloutType(type: string): CalloutType {
return (CALLOUT_TYPES as readonly string[]).includes(type)
? (type as CalloutType)
: 'info';
}
/**
* Render a callout node to the editor's HTML shape. `body` is the already
* markdown-parsed inner content (marked may hand back a string synchronously).
*/
export function renderCalloutHtml(
type: string,
body: string | Promise<string>,
): string {
return `<div data-type="callout" data-callout-type="${type}">${body}</div>`;
}
@@ -1,37 +0,0 @@
import { Token, marked } from 'marked';
import { normalizeCalloutType, renderCalloutHtml } from './callout-common.marked';
interface CalloutToken {
type: 'callout';
calloutType: string;
text: string;
raw: string;
}
export const calloutExtension = {
name: 'callout',
level: 'block',
start(src: string) {
return src.match(/:::/)?.index ?? -1;
},
tokenizer(src: string): CalloutToken | undefined {
const rule = /^:::([a-zA-Z0-9]+)\s+([\s\S]+?):::/;
const match = rule.exec(src);
if (match) {
return {
type: 'callout',
calloutType: normalizeCalloutType(match[1]),
raw: match[0],
text: match[2].trim(),
};
}
},
renderer(token: Token) {
const calloutToken = token as CalloutToken;
return renderCalloutHtml(
calloutToken.calloutType,
marked.parse(calloutToken.text),
);
},
};
@@ -1,72 +0,0 @@
import { describe, it, expect } from "vitest";
import { extractFootnoteDefinitions } from "./footnote.marked";
/** Pull the ordered list of `data-footnote-def` ids out of the rendered section. */
function defIds(section: string): string[] {
return [...section.matchAll(/data-footnote-def data-id="([^"]+)"/g)].map(
(m) => m[1],
);
}
/** Pull the ordered list of `[^id]` markers that remain in the body. */
function bodyMarkers(body: string): string[] {
return [...body.matchAll(/\[\^([^\]\s]+)\]/g)].map((m) => m[1]);
}
describe("extractFootnoteDefinitions: duplicate definition ids (first-wins)", () => {
// Body has ONE `[^d]` reference but THREE `[^d]:` definitions. Under the
// import model (#166) a duplicate definition id is FIRST-WINS: only the first
// definition is kept; the rest are DROPPED (and surfaced by analyzeFootnotes,
// not silently re-id'd into orphan footnotes as before). Reference markers are
// never rewritten, so repeated references would reuse the single footnote.
const md = ["See[^d].", "", "[^d]: a", "[^d]: b", "[^d]: c"].join("\n");
it("keeps only the FIRST definition for the id (first-wins)", () => {
const { section } = extractFootnoteDefinitions(md);
const ids = defIds(section);
expect(ids).toEqual(["d"]);
});
it("keeps the first definition's text and drops the duplicates", () => {
const { section } = extractFootnoteDefinitions(md);
expect(section).toContain('data-footnote-def data-id="d"><p>a</p>');
// No derived `d__2` / `d__3` ids are emitted anymore.
expect(section).not.toContain("d__2");
expect(section).not.toContain("d__3");
// The dropped duplicate texts are not in the section.
expect(section).not.toContain("<p>b</p>");
expect(section).not.toContain("<p>c</p>");
});
it("leaves the SINGLE body marker as [^d] (markers are never rewritten)", () => {
const { body } = extractFootnoteDefinitions(md);
expect(bodyMarkers(body)).toEqual(["d"]);
expect(body).toContain("See[^d].");
// The definition lines themselves were pulled OUT of the body.
expect(body).not.toContain("[^d]: a");
expect(body).not.toContain("[^d]: b");
expect(body).not.toContain("[^d]: c");
});
it("does not crash and produces a well-formed footnotes section", () => {
const { section } = extractFootnoteDefinitions(md);
expect(section.startsWith("<section data-footnotes>")).toBe(true);
expect(section.endsWith("</section>")).toBe(true);
// Exactly one definition div (first-wins).
expect([...section.matchAll(/<div data-footnote-def/g)]).toHaveLength(1);
});
});
describe("extractFootnoteDefinitions: reuse (repeated references, one definition)", () => {
// Pandoc semantics: many `[^a]` references + one `[^a]:` definition = one
// footnote, shared. Markers are left intact so the editor numbers them as one.
const md = ["A[^a] B[^a] C[^a].", "", "[^a]: shared note"].join("\n");
it("emits exactly one definition and leaves every reference marker as [^a]", () => {
const { section, body } = extractFootnoteDefinitions(md);
expect(defIds(section)).toEqual(["a"]);
expect(section).toContain('data-footnote-def data-id="a"><p>shared note</p>');
// All three reference markers stay `a` (no `a__2`/`a__3` minting).
expect(bodyMarkers(body)).toEqual(["a", "a", "a"]);
});
});
@@ -1,131 +0,0 @@
import { marked } from "marked";
/**
* Pandoc/GFM footnote support for the marked (Markdown -> HTML) pipeline.
*
* Two pieces:
* - an INLINE tokenizer for `[^id]` references -> <sup data-footnote-ref
* data-id="id"> (matches the editor-ext FootnoteReference renderHTML);
* - a document hook (`preprocess`/`walkTokens` is awkward for collecting +
* removing definitions, so we use a regex preprocessing step instead) that
* pulls every `[^id]: text` definition line out of the body and appends a
* single <section data-footnotes> with one <div data-footnote-def> per
* definition, so the round-trip rebuilds footnotesList + footnoteDefinition.
*
* Every FIRST definition line is emitted — duplicate ids are first-wins (the
* rest are dropped, and surfaced via analyzeFootnotes), and reference markers are
* left untouched so repeated `[^a]` references reuse the one footnote (#166).
* Orphan definitions (no matching reference) are still emitted here; the editor's
* sync plugin reconciles the final reference/definition set (drops orphans,
* synthesizes a single empty definition for a reference that lacks one).
*/
const DEFINITION_RE = /^\[\^([^\]\s]+)\]:[ \t]*(.*)$/;
const REFERENCE_RE = /\[\^([^\]\s]+)\]/;
interface FootnoteRefToken {
type: "footnoteRef";
raw: string;
id: string;
}
export const footnoteReferenceExtension = {
name: "footnoteRef",
level: "inline" as const,
start(src: string) {
return src.match(/\[\^/)?.index ?? -1;
},
tokenizer(src: string): FootnoteRefToken | undefined {
const match = REFERENCE_RE.exec(src);
// Only match at the very start of the remaining inline source.
if (match && match.index === 0) {
return {
type: "footnoteRef",
raw: match[0],
id: match[1],
};
}
return undefined;
},
renderer(token: FootnoteRefToken) {
return `<sup data-footnote-ref data-id="${escapeAttr(token.id)}"></sup>`;
},
};
function escapeAttr(value: string): string {
return String(value).replace(/&/g, "&amp;").replace(/"/g, "&quot;");
}
/**
* Extract `[^id]: text` definition lines from the markdown body, returning the
* cleaned body plus a rendered <section data-footnotes> (empty string when no
* definitions). Call this BEFORE marked.parse and append the section to the
* resulting HTML.
*/
export function extractFootnoteDefinitions(markdown: string): {
body: string;
section: string;
} {
const lines = markdown.split("\n");
const bodyLines: string[] = [];
const definitions: Array<{ id: string; text: string }> = [];
// Track fenced-code state so a `[^id]: ...` line that merely SHOWS footnote
// syntax inside a ``` / ~~~ code block is left in the body verbatim and not
// mistaken for a real definition.
let fence: string | null = null;
for (const line of lines) {
const fenceMatch = /^(\s*)(`{3,}|~{3,})/.exec(line);
if (fenceMatch) {
const marker = fenceMatch[2][0];
if (fence === null) {
fence = marker; // opening fence
} else if (marker === fence) {
fence = null; // closing fence (matching delimiter type)
}
bodyLines.push(line);
continue;
}
const m = fence === null ? DEFINITION_RE.exec(line) : null;
if (m) {
definitions.push({ id: m[1], text: m[2] });
} else {
bodyLines.push(line);
}
}
if (definitions.length === 0) {
return { body: markdown, section: "" };
}
// Duplicate definition ids (e.g. `[^d]: first` / `[^d]: second`): FIRST WINS,
// the rest are DROPPED. Reference markers are left UNTOUCHED so repeated `[^a]`
// references reuse the single footnote (Pandoc semantics, #166). This differs
// from the live editor's never-lose policy (resolveCollisions re-ids a
// duplicate definition into an orphan) on purpose: an import is an
// agent-authored artifact we sanitize, and the dropped duplicate is surfaced
// to the caller via analyzeFootnotes' `duplicateDefinitions` warning instead.
const firstById = new Map<string, string>(); // id -> first definition text
for (const def of definitions) {
if (!firstById.has(def.id)) firstById.set(def.id, def.text);
}
const defsHtml = [...firstById.entries()]
.map(([id, text]) => {
// Render the definition text as inline markdown so emphasis/links inside
// a footnote survive the round-trip; wrap in a paragraph (the node's
// content is paragraph+).
const inner = marked.parseInline(text || "");
return `<div data-footnote-def data-id="${escapeAttr(
id,
)}"><p>${inner}</p></div>`;
})
.join("");
return {
body: bodyLines.join("\n"),
section: `<section data-footnotes>${defsHtml}</section>`,
};
}
@@ -1,54 +0,0 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml } from "./marked.utils";
/**
* Regression for issue #192: pasting a GitHub-style `> [!type]` alert produced a
* literal `<blockquote>` containing `[!info]` instead of a callout node, because
* only the `:::type` form was tokenized. The editor paste path runs the same
* `markdownToHtml`, so these assertions pin the conversion at the source.
*/
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
describe("markdownToHtml: GitHub `> [!type]` callouts", () => {
it("converts `> [!info]` to a callout node, not a literal blockquote", () => {
const out = html("> [!info]\n> Callout body text here");
expect(out).toContain('data-type="callout"');
expect(out).toContain('data-callout-type="info"');
expect(out).toContain("Callout body text here");
expect(out).not.toContain("[!info]");
expect(out).not.toContain("<blockquote");
});
it("maps GitHub alert aliases onto the supported banner types", () => {
expect(html("> [!NOTE]\n> x")).toContain('data-callout-type="info"');
expect(html("> [!TIP]\n> x")).toContain('data-callout-type="success"');
expect(html("> [!WARNING]\n> x")).toContain('data-callout-type="warning"');
expect(html("> [!CAUTION]\n> x")).toContain('data-callout-type="danger"');
});
it("accepts the editor's own type names directly", () => {
expect(html("> [!success]\n> x")).toContain('data-callout-type="success"');
expect(html("> [!danger]\n> x")).toContain('data-callout-type="danger"');
});
it("falls back to info for an unknown type", () => {
expect(html("> [!bogus]\n> x")).toContain('data-callout-type="info"');
});
it("preserves multi-line callout bodies", () => {
const out = html("> [!warning]\n> line one\n> line two");
expect(out).toContain('data-callout-type="warning"');
expect(out).toContain("line one");
expect(out).toContain("line two");
});
it("still converts the `:::type` form", () => {
const out = html(":::info\nbody\n:::");
expect(out).toContain('data-type="callout"');
expect(out).toContain('data-callout-type="info"');
});
});
@@ -1,81 +0,0 @@
import { Token, marked } from 'marked';
import { renderCalloutHtml } from './callout-common.marked';
interface GithubCalloutToken {
type: 'githubCallout';
calloutType: string;
text: string;
raw: string;
}
/**
* Map GitHub "alert" blockquote markers (`> [!NOTE]`, `> [!WARNING]`, …) onto
* the four callout banner types the editor schema supports. The editor's own
* type names (`info`/`success`/`warning`/`danger`) are also accepted directly,
* because users paste both forms. Anything unrecognized falls back to `info`,
* matching the `:::type` callout tokenizer.
*/
const GITHUB_ALERT_TYPE_MAP: Record<string, string> = {
note: 'info',
tip: 'success',
important: 'info',
warning: 'warning',
caution: 'danger',
info: 'info',
success: 'success',
danger: 'danger',
};
/**
* Tokenizer for GitHub-flavored alert callouts written as a blockquote whose
* first line is `[!type]`:
*
* > [!info]
* > body line one
* > body line two
*
* Without this, the default blockquote tokenizer wins and the marker renders as
* a literal `[!info]` inside a `<blockquote>`. The editor's paste path runs the
* same `markdownToHtml`, so registering this here also fixes pasting the syntax
* into the editor (issue #192), not just markdown import.
*/
export const githubCalloutExtension = {
name: 'githubCallout',
level: 'block' as const,
start(src: string) {
return src.match(/^ {0,3}>[ \t]*\[!/m)?.index ?? -1;
},
tokenizer(src: string): GithubCalloutToken | undefined {
const rule =
/^ {0,3}>[ \t]*\[!([a-zA-Z]+)\][^\n]*(?:\n {0,3}>[^\n]*)*(?:\n|$)/;
const match = rule.exec(src);
if (!match) return undefined;
const rawType = match[1].toLowerCase();
const calloutType = GITHUB_ALERT_TYPE_MAP[rawType] ?? 'info';
const text = match[0]
.replace(/\n+$/, '')
.split('\n')
// Strip the blockquote marker (`>` + optional space) from every line.
.map((line) => line.replace(/^ {0,3}>[ \t]?/, ''))
// Drop the `[!type]` marker that opens the first line.
.map((line, i) => (i === 0 ? line.replace(/^\[![a-zA-Z]+\][ \t]*/, '') : line))
.join('\n')
.trim();
return {
type: 'githubCallout',
calloutType,
raw: match[0],
text,
};
},
renderer(token: Token) {
const calloutToken = token as GithubCalloutToken;
return renderCalloutHtml(
calloutToken.calloutType,
marked.parse(calloutToken.text),
);
},
};
@@ -1,41 +0,0 @@
import { Token } from "marked";
interface HtmlEmbedToken {
type: "htmlEmbed";
raw: string;
encoded: string;
}
/**
* Marked extension that rebuilds an `htmlEmbed` node from the HTML comment
* marker produced by the turndown rule (`<!--html-embed:<base64>-->`).
*
* It emits the same marker div the node's `parseHTML` recognizes, so the
* pipeline MD -> HTML -> ProseMirror JSON restores the node (and its
* base64 `data-source`) exactly. We do NOT expand the raw markup here; the
* source stays base64-encoded in the attribute and is only executed by the
* client NodeView.
*/
export const htmlEmbedExtension = {
name: "htmlEmbed",
level: "block" as const,
start(src: string) {
return src.indexOf("<!--html-embed:");
},
tokenizer(src: string): HtmlEmbedToken | undefined {
const rule = /^<!--html-embed:([A-Za-z0-9+/=]*)-->/;
const match = rule.exec(src);
if (match) {
return {
type: "htmlEmbed",
raw: match[0],
encoded: match[1] ?? "",
};
}
},
renderer(token: Token) {
const htmlEmbedToken = token as HtmlEmbedToken;
return `<div data-type="htmlEmbed" data-source="${htmlEmbedToken.encoded}"></div>`;
},
};
@@ -1,76 +0,0 @@
import { marked } from "marked";
import { calloutExtension } from "./callout.marked";
import { githubCalloutExtension } from "./github-callout.marked";
import { mathBlockExtension } from "./math-block.marked";
import { mathInlineExtension } from "./math-inline.marked";
import {
footnoteReferenceExtension,
extractFootnoteDefinitions,
} from "./footnote.marked";
import { htmlEmbedExtension } from "./html-embed.marked";
marked.use({
renderer: {
list({ ordered, start, items }) {
let body = "";
for (const item of items) {
body += this.listitem(item);
}
if (ordered) {
const startAttr = start !== 1 ? ` start="${start}"` : "";
return `<ol${startAttr}>\n${body}</ol>\n`;
}
const isTaskList = items.some((item) => item.task);
const dataType = isTaskList ? ' data-type="taskList"' : "";
return `<ul${dataType}>\n${body}</ul>\n`;
},
listitem({ tokens, task: isTask, checked: isChecked }) {
const text = this.parser.parse(tokens);
if (!isTask) {
return `<li>${text}</li>\n`;
}
const checkedAttr = isChecked
? 'data-checked="true"'
: 'data-checked="false"';
return `<li data-type="taskItem" ${checkedAttr}>${text}</li>\n`;
},
},
});
marked.use({
extensions: [
calloutExtension,
githubCalloutExtension,
mathBlockExtension,
mathInlineExtension,
footnoteReferenceExtension,
htmlEmbedExtension,
],
});
marked.setOptions({ breaks: true });
export function markdownToHtml(
markdownInput: string,
): string | Promise<string> {
const YAML_FONT_MATTER_REGEX = /^\s*---[\s\S]*?---\s*/;
const markdown = markdownInput
.replace(YAML_FONT_MATTER_REGEX, "")
.trimStart();
// Pull `[^id]: ...` definition lines out of the body, render the body, then
// append a single <section data-footnotes> so the round-trip rebuilds the
// footnotesList + footnoteDefinition nodes.
const { body, section } = extractFootnoteDefinitions(markdown);
const parsed = marked.parse(body);
if (!section) return parsed;
if (typeof parsed === "string") {
return parsed + section;
}
return parsed.then((html) => html + section);
}
@@ -1,37 +0,0 @@
import { Token, marked } from 'marked';
interface MathBlockToken {
type: 'mathBlock';
text: string;
raw: string;
}
export const mathBlockExtension = {
name: 'mathBlock',
level: 'block',
start(src: string) {
return src.match(/\$\$/)?.index ?? -1;
},
tokenizer(src: string): MathBlockToken | undefined {
const rule = /^\$\$(?!(\$))([\s\S]+?)\$\$/;
const match = rule.exec(src);
if (match) {
return {
type: 'mathBlock',
raw: match[0],
text: match[2]?.trim(),
};
}
},
renderer(token: Token) {
const mathBlockToken = token as MathBlockToken;
// parse to prevent escaping slashes
const latex = marked
.parse(mathBlockToken.text)
.toString()
.replace(/<(\/)?p>/g, '');
return `<div data-type="${mathBlockToken.type}" data-katex="true">${latex}</div>`;
},
};
@@ -1,50 +0,0 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml } from "./marked.utils";
/**
* Data-integrity regression (issue #204, Phase 2): plain prose that mentions
* prices like `$5 and $6` must NOT be misread as inline math. The inline-math
* tokenizer mutates a global `marked` singleton at import time
* (`marked.utils.ts`), so math behaviour can only be exercised safely through
* the public `markdownToHtml`; importing the tokenizer in isolation would give
* a different, non-representative result. These assertions therefore drive the
* real conversion path.
*/
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
const MATH_MARKERS = ['data-type="mathInline"', 'data-katex="true"'];
function hasInlineMath(out: string): boolean {
return MATH_MARKERS.some((m) => out.includes(m));
}
describe("markdownToHtml: inline-math false positives", () => {
it("does not treat prices `$5 and $6` as inline math", () => {
const out = html("It costs $5 and $6 today.");
expect(hasInlineMath(out)).toBe(false);
// The text survives verbatim (no katex span swallowing it).
expect(out).toContain("$5 and $6");
});
it("does not treat a single trailing price `$5` as inline math", () => {
const out = html("Lunch was $5.");
expect(hasInlineMath(out)).toBe(false);
expect(out).toContain("$5");
});
it("does not treat `$5, $6, $7` (multiple prices) as inline math", () => {
const out = html("Choose $5, $6, $7 plans.");
expect(hasInlineMath(out)).toBe(false);
});
it("STILL converts a genuine inline-math expression `$x + y$`", () => {
// Guard the positive path so the false-positive guard above can't be
// satisfied by simply disabling math entirely.
const out = html("The sum $x + y$ is shown.");
expect(hasInlineMath(out)).toBe(true);
});
});
@@ -1,55 +0,0 @@
import { Token, marked } from 'marked';
interface MathInlineToken {
type: 'mathInline';
text: string;
raw: string;
}
const inlineMathRegex = /^\$(?!\s)(.+?)(?<!\s)\$(?!\d)/;
export const mathInlineExtension = {
name: 'mathInline',
level: 'inline',
start(src: string) {
let index: number;
let indexSrc = src;
while (indexSrc) {
index = indexSrc.indexOf('$');
if (index === -1) {
return;
}
const f = index === 0 || indexSrc.charAt(index - 1) === ' ';
if (f) {
const possibleKatex = indexSrc.substring(index);
if (possibleKatex.match(inlineMathRegex)) {
return index;
}
}
indexSrc = indexSrc.substring(index + 1).replace(/^\$+/, '');
}
},
tokenizer(src: string): MathInlineToken | undefined {
const match = inlineMathRegex.exec(src);
if (match) {
return {
type: 'mathInline',
raw: match[0],
text: match[1]?.trim(),
};
}
},
renderer(token: Token) {
const mathInlineToken = token as MathInlineToken;
// parse to prevent escaping slashes
const latex = marked
.parse(mathInlineToken.text)
.toString()
.replace(/<(\/)?p>/g, '');
return `<span data-type="${mathInlineToken.type}" data-katex="true">${latex}</span>`;
},
};
@@ -1,128 +0,0 @@
import { describe, it, expect } from "vitest";
import { getSchema } from "@tiptap/core";
import { generateHTML, generateJSON } from "@tiptap/html";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Bold } from "@tiptap/extension-bold";
import { htmlToMarkdown } from "./turndown.utils";
import { markdownToHtml } from "./marked.utils";
import { Spoiler } from "../../spoiler/spoiler";
// The spoiler mark has no native Markdown syntax, so it is preserved losslessly
// as raw inline HTML (`<span data-spoiler="true">…</span>`), the same approach
// htmlEmbed uses. This test drives the full editor round-trip:
// JSON -> HTML -> Markdown -> HTML -> JSON
// and asserts the `spoiler` mark survives end to end. We use the same
// getSchema + @tiptap/html generateHTML/generateJSON utilities the other
// editor-ext schema tests use.
const extensions = [Document, Paragraph, Text, Bold, Spoiler];
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
// Count text nodes carrying a `spoiler` mark anywhere in a ProseMirror JSON doc.
function countSpoilerMarks(doc: any): number {
let count = 0;
const walk = (node: any) => {
if (!node || typeof node !== "object") return;
if (Array.isArray(node.marks)) {
for (const mark of node.marks) {
if (mark?.type === "spoiler") count++;
}
}
if (Array.isArray(node.content)) node.content.forEach(walk);
};
walk(doc);
return count;
}
describe("Spoiler mark schema", () => {
it("registers the spoiler mark in the schema", () => {
const schema = getSchema(extensions);
expect(schema.marks.spoiler).toBeTruthy();
});
it("recovers the spoiler mark from span[data-spoiler] (HTML -> JSON)", () => {
const json = generateJSON(
'<p>before <span data-spoiler="true">hidden</span> after</p>',
extensions,
);
expect(countSpoilerMarks(json)).toBe(1);
});
it("emits data-spoiler + class on render (JSON -> HTML)", () => {
const doc = {
type: "doc",
content: [
{
type: "paragraph",
content: [
{
type: "text",
text: "hidden",
marks: [{ type: "spoiler" }],
},
],
},
],
};
const out = generateHTML(doc, extensions);
expect(out).toContain('data-spoiler="true"');
expect(out).toContain('class="spoiler"');
});
});
describe("Spoiler Markdown round-trip is lossless", () => {
const docWith = (textNode: any) => ({
type: "doc",
content: [
{
type: "paragraph",
content: [{ type: "text", text: "before " }, textNode, { type: "text", text: " after" }],
},
],
});
it("preserves the spoiler mark through JSON -> MD -> HTML -> JSON", () => {
const startDoc = docWith({
type: "text",
text: "hidden",
marks: [{ type: "spoiler" }],
});
// JSON -> HTML
const html1 = generateHTML(startDoc, extensions);
expect(html1).toContain('data-spoiler="true"');
// HTML -> Markdown (raw inline HTML, lossless)
const md = htmlToMarkdown(html1);
expect(md).toContain('<span data-spoiler="true">hidden</span>');
// MD -> HTML -> JSON (mark restored via parseHTML)
const endJson = generateJSON(html(md), extensions);
expect(countSpoilerMarks(endJson)).toBe(1);
// The visible text survives.
expect(JSON.stringify(endJson)).toContain("hidden");
});
it("keeps the spoiler intact when it intersects a bold mark", () => {
const startDoc = docWith({
type: "text",
text: "secret",
marks: [{ type: "bold" }, { type: "spoiler" }],
});
const md = htmlToMarkdown(generateHTML(startDoc, extensions));
expect(md).toContain("data-spoiler=\"true\"");
const endJson = generateJSON(html(md), extensions);
expect(countSpoilerMarks(endJson)).toBe(1);
// Bold survives alongside the spoiler.
expect(JSON.stringify(endJson)).toContain('"bold"');
});
});
@@ -1,12 +0,0 @@
// Map @joplin/turndown types to @types/turndown
declare module "@joplin/turndown" {
import TurndownService from "turndown";
export = TurndownService;
}
declare module "@joplin/turndown-plugin-gfm" {
import TurndownService from "turndown";
export const tables: TurndownService.Plugin;
export const strikethrough: TurndownService.Plugin;
export const highlightedCodeBlock: TurndownService.Plugin;
}
@@ -1,147 +0,0 @@
import { describe, it, expect } from "vitest";
import { htmlToMarkdown } from "./turndown.utils";
import { markdownToHtml } from "./marked.utils";
/**
* #206 mdrt-2 — Markdown export must never SILENTLY drop a block. (FIXED)
*
* `htmlToMarkdown` (turndown) historically only registered rules for a fixed
* set of custom nodes (callout, taskItem, details, math, iframe, htmlEmbed,
* image, video, footnote). Any other custom node — `transclusionReference`,
* `pageBreak`, `mention`, `status` — fell through to turndown's default
* handling: an empty wrapper is "blank" and removed, so the block disappeared
* from the exported Markdown with no trace, and `mention`/`status` collapsed to
* bare text, losing their identity (data-id / data-color). The invariant
* "never silently lose a block" was broken.
*
* The fix adds lossless turndown rules that re-emit each of these nodes as raw
* HTML carrying every `data-*` attribute. Plain-Markdown viewers ignore the
* inert tag; the import path round-trips it (`markdownToHtml` passes the raw
* HTML through and each node's `parseHTML` rebuilds the ProseMirror node). These
* tests assert the surviving contract (the block is preserved AND its identity
* round-trips back through import).
*/
describe("htmlToMarkdown — custom nodes are preserved losslessly (#206 mdrt-2)", () => {
const wrap = (inner: string) => `<p>before</p>${inner}<p>after</p>`;
it("preserves a pageBreak block on Markdown export", () => {
const md = htmlToMarkdown(
wrap('<div data-type="pageBreak" class="page-break"></div>'),
);
expect(md).toContain("before");
expect(md).toContain("after");
// The break survives as an inert raw-HTML tag, not silently dropped.
expect(md).toMatch(/data-type="pageBreak"/);
expect(md).toMatch(/page-?break/i);
});
it("preserves a transclusionReference's identity on Markdown export", () => {
const md = htmlToMarkdown(
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
);
expect(md).toContain("before");
expect(md).toContain("after");
// The data-id (the only thing that gives the reference identity) survives.
expect(md).toContain("abc");
expect(md).toMatch(/data-type="transclusionReference"/);
});
it("preserves a mention's data-id (stable identity) on Markdown export", () => {
const md = htmlToMarkdown(
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
);
// The mention keeps its stable identity (data-id), not just the text.
expect(md).toContain("u1");
expect(md).toContain("Bob");
expect(md).toMatch(/data-type="mention"/);
});
it("preserves a status chip's color on Markdown export", () => {
const md = htmlToMarkdown(
'<p>s <span data-type="status" data-color="green">Done</span></p>',
);
// The chip's color (its identity) survives, not just the visible text.
expect(md).toContain("green");
expect(md).toContain("Done");
expect(md).toMatch(/data-type="status"/);
});
// The export form is only lossless if the import path can rebuild it. These
// assert the full MD -> HTML round-trip restores the node + its attributes,
// which is the marker <-> node contract each `parseHTML` relies on.
describe("import round-trip (markdownToHtml restores the node)", () => {
it("round-trips a pageBreak through export + import", async () => {
const md = htmlToMarkdown(
wrap('<div data-type="pageBreak" class="page-break"></div>'),
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<div[^>]*data-type="pageBreak"[^>]*>/);
expect(html).toContain("before");
expect(html).toContain("after");
});
it("round-trips a transclusionReference (keeps data-id)", async () => {
const md = htmlToMarkdown(
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<div[^>]*data-type="transclusionReference"[^>]*>/);
expect(html).toContain("abc");
});
it("round-trips a mention (keeps data-id + data-label)", async () => {
const md = htmlToMarkdown(
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<span[^>]*data-type="mention"[^>]*>/);
expect(html).toContain("u1");
expect(html).toContain("Bob");
});
it("round-trips a status chip (keeps data-color)", async () => {
const md = htmlToMarkdown(
'<p>s <span data-type="status" data-color="green">Done</span></p>',
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<span[^>]*data-type="status"[^>]*>/);
expect(html).toContain("green");
});
// HTML special chars in an attribute value or in a node's text must be
// ESCAPED when re-emitted as raw HTML, otherwise the exported tag is
// malformed and `markdownToHtml`'s parser cannot restore the original value
// (the same silent data loss this PR fixes). Dropping `<`/`>` escaping is the
// dangerous regression: a stray `<` or `>` corrupts the tag (or injects new
// markup), so the test data carries ALL of `&`, `"`, `<`, `>` in BOTH the
// data-label attribute and the visible text. That fully exercises
// escapeHtmlAttr's `&,",<,>` branches and escapeHtmlText's `&,<,>` branches
// (escapeHtmlText leaves `"` literal); the alphanumeric-only cases above hit
// none of them.
it("escapes HTML special chars (& \" < >) in attrs + text and round-trips them", async () => {
const md = htmlToMarkdown(
`<p>hi <span data-type="mention" data-id="u1" data-label="A &amp; &lt;B&gt; &quot;C&quot;">@A &amp; &lt;B&gt; "C"</span> there</p>`,
);
// (a) The exported Markdown carries a WELL-FORMED, correctly-escaped tag:
// the attribute escapes `&`, `<`, `>` AND `"`; the text escapes `&`, `<`,
// `>` (a `"` inside text content is legal, so it stays literal).
expect(md).toContain('data-label="A &amp; &lt;B&gt; &quot;C&quot;"');
expect(md).toContain('>@A &amp; &lt;B&gt; "C"</span>');
// And explicitly NOT the raw, tag-corrupting forms: a literal `<B>` (would
// mean `<`/`>` escaping was dropped in either the attr or the text)...
expect(md).not.toContain("<B>");
// ...nor the malformed attribute that an unescaped `"` would produce.
expect(md).not.toContain('data-label="A &amp; &lt;B&gt; "C""');
// (b) Import restores the ORIGINAL (unescaped) values, attribute and text.
const html = await markdownToHtml(md);
const dom = new DOMParser().parseFromString(html as string, "text/html");
const span = dom.querySelector('span[data-type="mention"]');
expect(span).not.toBeNull();
expect(span!.getAttribute("data-id")).toBe("u1");
expect(span!.getAttribute("data-label")).toBe('A & <B> "C"');
expect(span!.textContent).toBe('@A & <B> "C"');
});
});
});
@@ -1,488 +0,0 @@
import * as _TurndownService from '@joplin/turndown';
import * as TurndownPluginGfm from '@joplin/turndown-plugin-gfm';
import { getBasename } from './basename';
// CJS/ESM interop: .default exists in Vite, not in NestJS
const TurndownService = (_TurndownService as any).default || _TurndownService;
function sanitizeMdLinkText(value: string): string {
return value
.replace(/\\/g, '\\\\')
.replace(/([\[\]!])/g, '\\$1')
.replace(/[\r\n]+/g, ' ');
}
// Tags turndown treats as void (self-closing). Footnote references render as an
// empty <sup data-footnote-ref> whose meaning lives entirely in its data-id;
// without marking it void, turndown's blank-node removal drops it before our
// rule runs, losing the `[^id]` marker. Mirrors turndown's built-in list.
const TURNDOWN_VOID_ELEMENTS = [
'AREA', 'BASE', 'BR', 'COL', 'COMMAND', 'EMBED', 'HR', 'IMG', 'INPUT',
'KEYGEN', 'LINK', 'META', 'PARAM', 'SOURCE', 'TRACK', 'WBR',
];
function isVoidNode(node: any): boolean {
const name = node?.nodeName?.toUpperCase?.();
if (!name) return false;
if (name === 'SUP' && node.hasAttribute?.('data-footnote-ref')) {
return true;
}
return TURNDOWN_VOID_ELEMENTS.indexOf(name) !== -1;
}
/**
* An empty <sup data-footnote-ref> is "blank" to turndown, which removes blank
* inline nodes (RootNode/Node use a module-level isVoid the options cannot
* override). To survive, inject the id as text content so the node is non-blank;
* the footnoteReference rule then reads data-id and emits `[^id]`.
*/
function fillEmptyFootnoteRefs(html: string): string {
return html.replace(
/<sup\b([^>]*\bdata-footnote-ref\b[^>]*)>\s*<\/sup>/gi,
(_m, attrs) => `<sup${attrs}>​</sup>`,
);
}
/**
* `pageBreak` and `transclusionReference` are childless atom <div>s. Like an
* empty footnote ref (see above), turndown treats a childless block as "blank"
* and replaces it with the blankRule BEFORE any custom rule can fire — so the
* node disappears from the export with no trace (#206 mdrt-2). Inject a
* zero-width space so the node is non-blank and our lossless rule runs; the
* rule rebuilds the tag from the element's attributes, so the injected char
* never reaches the output.
*/
function fillEmptyAtomBlocks(html: string): string {
return html.replace(
/<div\b([^>]*\bdata-type="(?:pageBreak|transclusionReference)"[^>]*)>\s*<\/div>/gi,
(_m, attrs) => `<div${attrs}>​</div>`,
);
}
/** HTML-escape an attribute value so a re-emitted raw-HTML tag is well-formed. */
function escapeHtmlAttr(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
/** HTML-escape text placed inside a re-emitted raw-HTML element. */
function escapeHtmlText(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
/**
* Serialize ALL of an element's attributes back to a raw-HTML attribute string
* (leading space included). Generic on purpose: a custom node's identity lives
* entirely in its `data-*` attributes (data-id, data-color, data-source-page-id,
* data-transclusion-id, …), and serializing every attribute keeps the export
* lossless regardless of which attributes a given node carries.
*/
function serializeAttrs(node: any): string {
const attrs = node?.attributes;
if (!attrs) return '';
return Array.from(attrs as ArrayLike<{ name: string; value: string }>)
.map((attr) => ` ${attr.name}="${escapeHtmlAttr(attr.value ?? '')}"`)
.join('');
}
export function htmlToMarkdown(html: string): string {
const turndownService = new TurndownService({
headingStyle: 'atx',
codeBlockStyle: 'fenced',
hr: '---',
bulletListMarker: '-',
isVoid: isVoidNode,
});
turndownService.use([
TurndownPluginGfm.tables,
TurndownPluginGfm.strikethrough,
TurndownPluginGfm.highlightedCodeBlock,
taskList,
callout,
preserveDetail,
listParagraph,
orderedListItem,
mathInline,
mathBlock,
iframeEmbed,
htmlEmbed,
spoiler,
image,
video,
footnoteReference,
footnotesList,
pageBreak,
transclusionReference,
mention,
status,
]);
return turndownService
.turndown(fillEmptyAtomBlocks(fillEmptyFootnoteRefs(html)))
.replaceAll('<br>', ' ');
}
/**
* Lossless export rules for custom nodes that have NO native Markdown syntax
* (#206 mdrt-2). Markdown cannot represent a page break, a transclusion
* reference, a mention's stable id, or a status chip's color — so rather than
* letting turndown silently drop them, each rule re-emits the node as raw HTML
* carrying every `data-*` attribute. Plain-Markdown viewers ignore the inert
* tag, and the import path round-trips it: `markdownToHtml` passes raw HTML
* through and each node's `parseHTML` (`div[data-type="…"]`, `span[…]`) rebuilds
* the ProseMirror node with its attributes intact.
*/
function pageBreak(turndownService: _TurndownService) {
turndownService.addRule('pageBreak', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'pageBreak'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
},
});
}
function transclusionReference(turndownService: _TurndownService) {
turndownService.addRule('transclusionReference', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'transclusionReference'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
},
});
}
function mention(turndownService: _TurndownService) {
turndownService.addRule('mention', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-type') === 'mention'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const text = escapeHtmlText(node.textContent || '');
return `<span${serializeAttrs(node)}>${text}</span>`;
},
});
}
function status(turndownService: _TurndownService) {
turndownService.addRule('status', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' && node.getAttribute('data-type') === 'status'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const text = escapeHtmlText(node.textContent || '');
return `<span${serializeAttrs(node)}>${text}</span>`;
},
});
}
/**
* Serialize the `htmlEmbed` node to Markdown.
*
* Markdown has no native representation for an arbitrary-HTML block, so we
* preserve the node losslessly as an HTML comment carrying the base64-encoded
* source (the same `data-source` payload the node stores). `markdownToHtml`
* recognizes the same marker and rebuilds the node, so the round-trip
* MD -> HTML -> JSON keeps the source intact. The comment also keeps the raw
* markup inert in the exported `.md` file (it does not render in plain Markdown
* viewers).
*/
function htmlEmbed(turndownService: _TurndownService) {
turndownService.addRule('htmlEmbed', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'htmlEmbed'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const encoded = node.getAttribute('data-source') || '';
return `\n\n<!--html-embed:${encoded}-->\n\n`;
},
});
}
/**
* Serialize the `spoiler` inline mark to lossless raw inline HTML.
*
* Markdown has no native spoiler syntax, so we emit the same `<span
* data-spoiler="true">…</span>` the mark renders. `marked` passes inline raw HTML
* through untouched, and `generateJSON` restores the mark via its parseHTML, so
* the round-trip MD -> HTML -> JSON keeps the spoiler intact. The UI-only
* `is-revealed` state is never serialized.
*/
function spoiler(turndownService: _TurndownService) {
turndownService.addRule('spoiler', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-spoiler') === 'true'
);
},
replacement: function (content: string) {
return `<span data-spoiler="true">${content}</span>`;
},
});
}
function listParagraph(turndownService: _TurndownService) {
turndownService.addRule('paragraph', {
filter: ['p'],
replacement: (content: string, node: HTMLInputElement) => {
if (node.parentElement?.nodeName === 'LI') {
return content;
}
return `\n\n${content}\n\n`;
},
});
}
function orderedListItem(turndownService: _TurndownService) {
turndownService.addRule('orderedListItem', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'LI' && node.getAttribute('data-type') !== 'taskItem';
},
replacement: (content: string, node: HTMLInputElement, options: any) => {
const parent = node.parentNode as HTMLElement;
if (parent.nodeName !== 'OL' && parent.nodeName !== 'UL') {
return content;
}
content = content
.replace(/^\n+/, '')
.replace(/\n+$/, '\n')
.replace(/\n/gm, '\n ');
let prefix: string;
if (parent.nodeName === 'OL') {
const start = parseInt(parent.getAttribute('start') || '1', 10);
const index = Array.prototype.indexOf.call(parent.children, node);
prefix = `${start + index}. `;
} else {
prefix = `${options.bulletListMarker} `;
}
return (
prefix +
content +
(node.nextSibling && !/\n$/.test(content) ? '\n' : '')
);
},
});
}
function callout(turndownService: _TurndownService) {
turndownService.addRule('callout', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' && node.getAttribute('data-type') === 'callout'
);
},
replacement: function (content: string, node: HTMLInputElement) {
const calloutType = node.getAttribute('data-callout-type');
return `\n\n:::${calloutType}\n${content.trim()}\n:::\n\n`;
},
});
}
function taskList(turndownService: _TurndownService) {
turndownService.addRule('taskListItem', {
filter: function (node: HTMLInputElement) {
return (
node.getAttribute('data-type') === 'taskItem' &&
node.parentNode.nodeName === 'UL'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const isChecked = node.getAttribute('data-checked') === 'true';
const div = node.querySelector('div');
const text = div ? div.textContent.trim() : node.textContent.trim();
const prefix = `- ${isChecked ? '[x]' : '[ ]'} `;
return (
prefix +
text +
(node.nextSibling && !/\n$/.test(text) ? '\n' : '')
);
},
});
}
function preserveDetail(turndownService: _TurndownService) {
turndownService.addRule('preserveDetail', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'DETAILS';
},
replacement: function (_content: string, node: HTMLInputElement) {
const summary = node.querySelector(':scope > summary');
let detailSummary = '';
if (summary) {
detailSummary = `<summary>${turndownService.turndown(summary.innerHTML)}</summary>`;
}
const detailsContent = Array.from(node.childNodes)
.filter((child) => child.nodeName !== 'SUMMARY')
.map((child) =>
child.nodeType === 1
? turndownService.turndown((child as HTMLElement).outerHTML)
: child.textContent,
)
.join('');
return `\n<details>\n${detailSummary}\n\n${detailsContent}\n\n</details>\n`;
},
});
}
function mathInline(turndownService: _TurndownService) {
turndownService.addRule('mathInline', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-type') === 'mathInline'
);
},
replacement: function (content: string) {
return `$${content}$`;
},
});
}
function mathBlock(turndownService: _TurndownService) {
turndownService.addRule('mathBlock', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'mathBlock'
);
},
replacement: function (content: string) {
return `\n$$\n${content}\n$$\n`;
},
});
}
function iframeEmbed(turndownService: _TurndownService) {
turndownService.addRule('iframeEmbed', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'IFRAME';
},
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src');
return '[' + src + '](' + src + ')';
},
});
}
function image(turndownService: _TurndownService) {
turndownService.addRule('image', {
filter: 'img',
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src') || '';
if (!src) return '';
const caption = node.getAttribute('data-caption') || '';
if (caption) {
// ![]() can't carry a caption, so emit a raw <img> wrapped in a block
// <div>. marked passes it through and the image extension's parseHTML
// restores the caption from data-caption.
const parts = [`src="${escapeHtmlAttr(src)}"`];
const alt = node.getAttribute('alt') || '';
if (alt) parts.push(`alt="${escapeHtmlAttr(alt)}"`);
parts.push(`data-caption="${escapeHtmlAttr(caption)}"`);
return `<div><img ${parts.join(' ')}></div>`;
}
const alt = sanitizeMdLinkText(node.getAttribute('alt') || '');
const title = node.getAttribute('title') || '';
const titlePart = title ? ' "' + title.replace(/"/g, '\\"') + '"' : '';
return '![' + alt + '](' + src + titlePart + ')';
},
});
}
/**
* Footnote reference (inline atom) -> pandoc/GFM marker `[^id]`.
* The visible number is derived (not stored), so the id is the stable anchor.
*/
function footnoteReference(turndownService: _TurndownService) {
turndownService.addRule('footnoteReference', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SUP' && node.hasAttribute('data-footnote-ref')
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const id = node.getAttribute('data-id') || '';
return id ? `[^${id}]` : '';
},
});
}
/**
* Footnotes container -> the list of `[^id]: text` definitions at the end of
* the document (one per line). Each footnoteDefinition inside emits its own
* `[^id]: ...` line; turndown joins them with the surrounding block spacing.
*/
function footnotesList(turndownService: _TurndownService) {
turndownService.addRule('footnoteDefinition', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' && node.hasAttribute('data-footnote-def')
);
},
replacement: function (content: string, node: HTMLInputElement) {
const id = node.getAttribute('data-id') || '';
// Collapse internal newlines so the definition stays a single MD line;
// continuation lines are a v2 refinement.
const text = content.replace(/\s*\n+\s*/g, ' ').trim();
return id ? `\n[^${id}]: ${text}\n` : '';
},
});
turndownService.addRule('footnotesList', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SECTION' && node.hasAttribute('data-footnotes')
);
},
replacement: function (content: string) {
return `\n\n${content.trim()}\n`;
},
});
}
function video(turndownService: _TurndownService) {
turndownService.addRule('video', {
filter: function (node: HTMLInputElement) {
return node.tagName === 'VIDEO';
},
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src') || '';
const ariaLabel = node.getAttribute('aria-label');
const name = sanitizeMdLinkText(
ariaLabel || getBasename(src) || src,
);
return '[' + name + '](' + src + ')';
},
});
}
+6 -1
View File
@@ -14,10 +14,15 @@ export default defineConfig({
provider: "v8",
reporter: ["text-summary", "text"],
all: false,
// functions lowered 60 -> 57 after issue #347 removed the editor-ext
// markdown layer (src/lib/markdown) and its image/footnote round-trip
// specs: that markdown behavior now lives in — and is tested by —
// @docmost/prosemirror-markdown, so the editor-ext baseline shifts down.
// Still a real gate (a few points below the post-removal measured level).
thresholds: {
statements: 54,
branches: 44,
functions: 60,
functions: 57,
lines: 54,
},
},
+120 -88
View File
@@ -12,7 +12,7 @@ license.
> better at *writing a small function that fixes the text* than at re-reading and
> re-emitting a whole document. So this server is built around the way a model actually
> wants to edit: address a block by id, run a find/replace, or hand it a
> `(doc, ctx) => doc` transform and let it *program* the change. `docmost_transform` is
> `(doc, ctx) => doc` transform and let it *program* the change. `docmostTransform` is
> that interface. Other Docmost MCPs are human-shaped — they expose "open the page" and
> "replace the page"; this one exposes the editing primitives a model is good at.
@@ -40,7 +40,7 @@ There are several Docmost MCPs. Here is a capability-by-capability comparison.
| **Enterprise license required** | **No** | **Yes** | No | No | No |
| Authentication | email + password, **auto re-auth** | API key | email + password | cookie `authToken` (copy from DevTools) | Docmost API / **direct PostgreSQL** |
| Read page as Markdown | ✅ | ✅ | ✅ | ✅ | ✅ (read-only) |
| **Lossless Markdown round-trip** (export / import, keeps comment anchors) | ✅ | — | — | — | — |
| **Markdown round-trip** (export / import, keeps comment anchors) | ✅ | — | — | — | — |
| Read **lossless ProseMirror JSON** (with block ids) | ✅ | — | — | — | — |
| **Compact page outline** (cheap block-id lookup) | ✅ | — | — | — | — |
| **Fetch a single block** (by id or index) | ✅ | — | — | — | — |
@@ -69,9 +69,9 @@ There are several Docmost MCPs. Here is a capability-by-capability comparison.
- **Token-efficient editing.** Most Docmost MCPs (and the official one) only offer
"replace the whole page" writes — the agent must download the entire document, mutate
it, and upload it back, paying for the full document **twice** on every tiny fix.
This server lets the agent change exactly one block (`patch_node` / `insert_node` /
`delete_node`), do a structure-preserving find/replace (`edit_page_text`), or copy a
whole page server-side (`copy_page_content`) — **without the document ever passing
This server lets the agent change exactly one block (`patchNode` / `insertNode` /
`deleteNode`), do a structure-preserving find/replace (`editPageText`), or copy a
whole page server-side (`copyPageContent`) — **without the document ever passing
through the model**.
- **Writes that don't fight the editor.** Naive REST writes race with whatever a human
@@ -85,12 +85,12 @@ There are several Docmost MCPs. Here is a capability-by-capability comparison.
- **Agent-native editing model.** Human-facing servers expose "open the page" and "replace
the page", because that mirrors how a person works. A model edits better by *programming*
the change — addressing blocks by id, running a find/replace, or supplying a
`(doc, ctx) => doc` transform (`docmost_transform`, with a dry-run diff before it
`(doc, ctx) => doc` transform (`docmostTransform`, with a dry-run diff before it
commits). This server is shaped around that, which is why it has editing primitives the
others simply don't.
- **An editing safety net the others lack.** `list_page_history``diff_page_versions`
`restore_page_version` give an agent (and you) a full view-and-undo loop. The diff
- **An editing safety net the others lack.** `listPageHistory``diffPageVersions`
`restorePageVersion` give an agent (and you) a full view-and-undo loop. The diff
uses the *same* `recreateTransform → ChangeSet → simplifyChanges` pipeline Docmost's
own history viewer uses, so what you see matches the product.
@@ -110,52 +110,58 @@ All 41 tools, grouped by what you'd reach for them.
### Exploration & retrieval
- **`get_workspace`** — Information about the current Docmost workspace.
- **`list_spaces`** — All spaces in the workspace.
- **`list_pages`** — Recent pages in a space, ordered by `updatedAt` desc (default 50,
- **`getWorkspace`** — Information about the current Docmost workspace.
- **`listSpaces`** — All spaces in the workspace.
- **`listPages`** — Recent pages in a space, ordered by `updatedAt` desc (default 50,
max 100). Use `search` for lookups in large spaces.
- **`search`** — Full-text search across pages and content (bounded by `limit`, max 100).
- **`get_page`** — A page's content as clean **Markdown** (convenient, but a *lossy*
view — block ids and exact table/callout structure are approximated).
- **`get_page_json`** — A page's **lossless ProseMirror/TipTap JSON**, including every
- **`getPage`** — A page's content as clean **Markdown** (canonical for text; drops only
block ids, resolved-comment anchors, and a fixed no-Markdown-representation attr set —
table spans/colwidth/background, indent, `callout.icon`, `orderedList.type`, and link
`internal`/`target`/`rel`/`class`; use `getPageJson` when you need those).
- **`getPageJson`** — A page's **lossless ProseMirror/TipTap JSON**, including every
block's `attrs.id` and the `slugId` used in URLs. This is what the per-block editing
tools consume.
- **`get_outline`** — A compact outline of a page's top-level blocks (`{index, type, id,
- **`getOutline`** — A compact outline of a page's top-level blocks (`{index, type, id,
level, firstText}`; tables add row/column counts and their header-cell texts, lists add
item counts) **without** the document body. The cheap way to locate a section or table
and grab its block id before
`get_node` / `patch_node` / `insert_node`.
- **`get_node`** — Fetch a single block's full ProseMirror subtree (lossless) without
pulling the whole page. Address it by a block id (from `get_outline` / `get_page_json`),
`getNode` / `patchNode` / `insertNode`.
- **`getNode`** — Fetch a single block's full ProseMirror subtree (lossless) without
pulling the whole page. Address it by a block id (from `getOutline` / `getPageJson`),
or by `#<index>` for a top-level block — use the `#<index>` form for tables/rows/cells,
which carry no id.
### Page lifecycle
- **`create_page`** — Create a page from Markdown and place it in the hierarchy (optional
- **`createPage`** — Create a page from Markdown and place it in the hierarchy (optional
`parentPageId`) in one call. Uses Docmost's import API for clean Markdown→ProseMirror.
- **`rename_page`** — Change a page's title only, without touching or resending content.
- **`move_page`** — Re-parent a page (nest it, or move to root); supports fractional-index
- **`renamePage`** — Change a page's title only, without touching or resending content.
- **`movePage`** — Re-parent a page (nest it, or move to root); supports fractional-index
positioning. Returns only on a *positively confirmed* success.
- **`delete_page`** — Delete a single page.
- **`copy_page_content`** — Replace one page's body with a copy of another's, **entirely
- **`deletePage`** — Delete a single page.
- **`copyPageContent`** — Replace one page's body with a copy of another's, **entirely
server-side** — the document never passes through the model. The target keeps its own
title and slug (so its URL is preserved).
### Editing
- **`edit_page_text`** — Surgical find/replace inside a page's text. Preserves **all**
- **`editPageText`** — Surgical find/replace inside a page's text. Preserves **all**
structure: block ids, marks, links, callouts, tables. The preferred tool for fixing
wording, typos, numbers and names.
- **`patch_node`** — Replace a single block addressed by its `attrs.id` (from
`get_page_json`), without resending the document.
- **`insert_node`** — Insert a block before/after another (by `attrs.id` or anchor text),
- **`patchNode`** — Replace a single block addressed by its `attrs.id` (from
`getPageJson`), without resending the document.
- **`insertNode`** — Insert a block before/after another (by `attrs.id` or anchor text),
or append at the end.
- **`delete_node`** — Remove a single block by its `attrs.id`.
- **`update_page_json`** — Replace a page's entire content with a ProseMirror document
- **`deleteNode`** — Remove a single block by its `attrs.id`.
- **`updatePageJson`** — Replace a page's entire content with a ProseMirror document
(bulk rewrites, or when nodes lack ids). `content` is optional — omit it to update only
the title. Keeps the block ids you pass in, so heading anchors and history stay stable.
- **`docmost_transform`** — The agent-native editing interface: instead of retyping a
- **`updatePageMarkdown`** — Replace a page's body (and optionally its title) with new
**plain Markdown**. The whole body is re-imported (block ids regenerate — for surgical or
id-preserving edits prefer `editPageText` / `patchNode` / `updatePageJson`).
Docmost-flavoured markdown is parsed, including `^[...]` inline footnotes.
- **`docmostTransform`** — The agent-native editing interface: instead of retyping a
document, the agent **writes a function that fixes it**. Edit a page by running an
arbitrary **`(doc, ctx) => doc` JavaScript transform** against its *live* ProseMirror
document. Runs **sandboxed**
@@ -168,42 +174,46 @@ All 41 tools, grouped by what you'd reach for them.
### Tables
- **`table_get`** — Read a table as a matrix: `{rows, cols, cells (text[][]), cellIds}`
- **`tableGet`** — Read a table as a matrix: `{rows, cols, cells (text[][]), cellIds}`
(a paragraph id per cell, or `null`). Address the table by `#<index>` (from
`get_outline`) or any block id inside it. Use `cellIds` with `patch_node` for
`getOutline`) or any block id inside it. Use `cellIds` with `patchNode` for
rich-formatted cell edits.
- **`table_insert_row`** — Insert a row of plain-text cells, padded to the table's column
- **`tableInsertRow`** — Insert a row of plain-text cells, padded to the table's column
count (passing more cells than columns is an error). `index` is the 0-based insert
position (0 inserts before the header); omit it to append at the end.
- **`table_delete_row`** — Delete the row at a 0-based `index`. Refuses to delete a table's
- **`tableDeleteRow`** — Delete the row at a 0-based `index`. Refuses to delete a table's
only row; deleting row 0 promotes the next row to header.
- **`table_update_cell`** — Set the plain-text content of cell `[row, col]` (0-based). For
rich formatting, `patch_node` the cell's paragraph id from `table_get`.
- **`tableUpdateCell`** — Set the plain-text content of cell `[row, col]` (0-based). For
rich formatting, `patchNode` the cell's paragraph id from `tableGet`.
### Markdown round-trip
- **`export_page_markdown`** — Export a page to a single self-contained, **lossless
Docmost-flavoured Markdown** file: a meta header, the body with inline comment anchors
and diagrams, and a trailing comments-thread block. Built for a download → edit body
`import_page_markdown` round-trip that preserves everything, including comment highlights.
- **`import_page_markdown`** — Replace a page's content from a Docmost-flavoured Markdown
file produced by `export_page_markdown`, restoring comment-highlight anchors and diagrams
from their inline HTML. (Comment *threads* in the file are not re-created on the server —
only the page body and inline comment marks are written; manage threads via the comment
tools/UI.)
- **`exportPageMarkdown`** — Export a page to a single self-contained
**Docmost-flavoured Markdown** file: a meta header, the body with inline comment anchors
and diagrams, and a trailing comments-thread block. The download → edit → import
round-trip regenerates block ids and **silently drops** the no-Markdown-representation
attr set (table merge spans/colwidth/background, indent, `callout.icon`,
`orderedList.type`, link `internal`/`target`/`rel`/`class`); keep those in ProseMirror
JSON if they must survive. To replace a page's body from plain authoring Markdown, use
`updatePageMarkdown`.
> **Removed in this release:** `importPageMarkdown` (the round-trip parser for an
> exported Docmost-Markdown file) is **no longer exposed on the external MCP surface**.
> To replace a page's body from Markdown, use **`updatePageMarkdown`** (plain Markdown
> body replace). See the CHANGELOG for the migration note.
### Images
- **`insert_image`** — Download an image from a web (http/https) URL and insert it in one
- **`insertImage`** — Download an image from a web (http/https) URL and insert it in one
step: append it, drop it in place of a text placeholder (`replaceText`), or put it after
a given block (`afterText`). Preserves all other block ids.
- **`replace_image`** — Swap an existing image for one fetched from a web (http/https) URL.
- **`replaceImage`** — Swap an existing image for one fetched from a web (http/https) URL.
Uploads the new file as a **fresh
attachment** (clean URL that renders and busts browser caches), then re-points every
node referencing the old attachment (recursively, including callouts/tables) via the
live document, preserving comments, alignment and alt text. (In-place overwrite is
deliberately avoided — some Docmost versions corrupt the attachment on overwrite.)
- **`stash_page`** — Serialize a whole page (its full ProseMirror JSON) into an ephemeral
- **`stashPage`** — Serialize a whole page (its full ProseMirror JSON) into an ephemeral
in-RAM blob and return ONLY a short anonymous URL — the body never enters the model
context, so it is the way to hand a large page (and its images) to an external consumer
without truncation. Every internal file/image attachment is mirrored into the same
@@ -214,35 +224,35 @@ All 41 tools, grouped by what you'd reach for them.
### Comments
- **`create_comment`** — Add a page comment, optionally **anchored inline** to an exact
- **`createComment`** — Add a page comment, optionally **anchored inline** to an exact
span of text (the first occurrence is wrapped in a comment mark).
- **`list_comments`** — List a page's comments (content returned as Markdown).
- **`update_comment`** — Edit an existing comment.
- **`delete_comment`** — Delete a comment.
- **`resolve_comment`** — Resolve (close) or reopen a comment thread (reversible). Only top-level
comments can be resolved; the thread and its replies are kept, unlike `delete_comment`.
- **`check_new_comments`** — Find comments created after a given ISO-8601 timestamp across
- **`listComments`** — List a page's comments (content returned as Markdown).
- **`updateComment`** — Edit an existing comment.
- **`deleteComment`** — Delete a comment.
- **`resolveComment`** — Resolve (close) or reopen a comment thread (reversible). Only top-level
comments can be resolved; the thread and its replies are kept, unlike `deleteComment`.
- **`checkNewComments`** — Find comments created after a given ISO-8601 timestamp across
a space, optionally scoped to a page subtree — ideal for an agent that watches a doc for
feedback.
### Versioning & history
- **`list_page_history`** — A page's saved versions (Docmost auto-snapshots on save),
- **`listPageHistory`** — A page's saved versions (Docmost auto-snapshots on save),
newest first, cursor-paginated. Each item's id is the `historyId`.
- **`diff_page_versions`** — Diff two versions (or a version against the live page).
- **`diffPageVersions`** — Diff two versions (or a version against the live page).
Returns inserted/deleted text, integrity counts (images, links, tables, callouts,
footnote markers), and a human-readable Markdown summary — computed with the same
pipeline Docmost's own history viewer uses.
- **`restore_page_version`** — Write a saved version back as the current content. Docmost
- **`restorePageVersion`** — Write a saved version back as the current content. Docmost
has no restore endpoint, so this creates a **new** snapshot — the restore is itself
revertible.
### Sharing
- **`share_page`** — Make a page publicly accessible (idempotent) and return its public
- **`sharePage`** — Make a page publicly accessible (idempotent) and return its public
URL (`<app>/share/<key>/p/<slugId>`); optional search-engine indexing.
- **`unshare_page`** — Revoke a page's public share.
- **`list_shares`** — All public shares in the workspace, with titles and public URLs.
- **`unsharePage`** — Revoke a page's public share.
- **`listShares`** — All public shares in the workspace, with titles and public URLs.
---
@@ -251,26 +261,27 @@ All 41 tools, grouped by what you'd reach for them.
This same guidance is also delivered at runtime via the MCP server `instructions` field,
so capable clients steer the model automatically.
- **Text fixes** (wording, typos, numbers): `edit_page_text`.
- **One block** (paragraph/heading/callout/table cell): `patch_node` / `insert_node` /
`delete_node`, addressing the node by its `attrs.id` from `get_page_json`.
- **Images**: `insert_image` / `replace_image`.
- **A new page**: `create_page`.
- **Bulk rewrite, or nodes without ids**: `update_page_json`.
- **Text fixes** (wording, typos, numbers): `editPageText`.
- **One block** (paragraph/heading/callout/table cell): `patchNode` / `insertNode` /
`deleteNode`, addressing the node by its `attrs.id` from `getPageJson`.
- **Images**: `insertImage` / `replaceImage`.
- **A new page**: `createPage`.
- **Bulk rewrite, or nodes without ids**: `updatePageJson` (ProseMirror) or
`updatePageMarkdown` (plain Markdown body replace).
- **Multi-step / scripted rewrite** (renumbering, footnotes, coordinated edits):
`docmost_transform` — preview with `dryRun`, then apply.
- **Copy a whole page's content from another page** (server-side): `copy_page_content`.
- **Rename a page** (title only): `rename_page`.
- **Reads**: `get_page` (Markdown) / `get_page_json` (lossless ProseMirror with ids).
- **Review changes**: `list_page_history` → `diff_page_versions` → `restore_page_version`.
- **Comments**: `create_comment` (with optional inline anchoring) / `list_comments` /
`update_comment` / `resolve_comment` / `delete_comment` / `check_new_comments`.
- **Navigate a page cheaply** (find a section/table, grab a block id): `get_outline` →
`get_node`.
- **Tables** (add/remove a row, set a cell): `table_get` / `table_insert_row` /
`table_delete_row` / `table_update_cell`.
- **Round-trip a page as Markdown** (download, edit, re-upload losslessly with comments):
`export_page_markdown` / `import_page_markdown`.
`docmostTransform` — preview with `dryRun`, then apply.
- **Copy a whole page's content from another page** (server-side): `copyPageContent`.
- **Rename a page** (title only): `renamePage`.
- **Reads**: `getPage` (Markdown) / `getPageJson` (lossless ProseMirror with ids).
- **Review changes**: `listPageHistory` → `diffPageVersions` → `restorePageVersion`.
- **Comments**: `createComment` (with optional inline anchoring) / `listComments` /
`updateComment` / `resolveComment` / `deleteComment` / `checkNewComments`.
- **Navigate a page cheaply** (find a section/table, grab a block id): `getOutline` →
`getNode`.
- **Tables** (add/remove a row, set a cell): `tableGet` / `tableInsertRow` /
`tableDeleteRow` / `tableUpdateCell`.
- **Export a page as self-contained Markdown** (with comment anchors): `exportPageMarkdown`.
- **Replace a page's body from Markdown**: `updatePageMarkdown`.
---
@@ -282,25 +293,46 @@ so capable clients steer the model automatically.
the debounced REST snapshot), then **reads → transforms → writes synchronously** in one
tick so no remote update can interleave, and **waits for persistence acknowledgement**
before returning.
- **Per-page write serialization.** A per-`pageId` async mutex ensures two MCP writes to
the same page never overlap; different pages never block each other.
- **Per-page write serialization.** A per-`pageId` async mutex (keyed by the resolved
page **UUID**, never a slugId) ensures two MCP writes to the same page never overlap;
different pages never block each other. The lock helper fails fast if it is ever handed
a non-UUID key, so a write path that forgot to resolve the id can never silently lock
under a split key.
**Deploy requirement — single instance or sticky sessions.** This mutex is an
in-process `Map`, and the cached collab sessions and the `stash_page` blob store are
RAM-only and process-local. Behind a **multi-replica** load balancer **without sticky
sessions**, two replicas can each "hold" the lock for the same page at once and per-page
serialization is silently lost. Run the MCP/app as a **single instance**, or pin each
page's traffic to one replica (sticky sessions / consistent hashing on the page id).
There is deliberately no cross-process (e.g. Postgres advisory) lock yet — a conscious
documented constraint. See the `Dockerfile` comment and the `MCP collaboration write
path` block in `.env.example`.
**Rights-staleness window.** A cached collab session writes under the token captured at
connect time (and the collab-token cache reuses a token for its TTL), so a **revoked**
page access can lag by up to `MCP_COLLAB_SESSION_MAX_AGE_MS` (the hard session lifetime,
default 10 min) before the next re-auth picks it up. Lower it to shorten the lag at the
cost of more reconnects. This bounded window is an accepted trade-off; there is no
push-based cache invalidation on a rights change.
- **Transparent re-authentication.** Login uses email/password; expired tokens are
refreshed automatically on the first 401/403 (covering JSON, multipart upload, and the
collaboration-token path), with in-flight login de-duplication so a burst of calls
triggers a single re-login.
- **Lossless and lossy reads.** `get_page_json` returns the exact ProseMirror tree with
block ids; `get_page` returns clean Markdown for convenience.
- **Precise reads.** `getPageJson` returns the exact ProseMirror tree with block ids;
`getPage` returns canonical Markdown that drops only a fixed, documented attr set.
- **Full Docmost schema.** Markdown↔ProseMirror conversion supports callouts (including
nested), task lists (bullet *and* numbered checklists), tables, math blocks, embeds,
highlights, sub/superscript and more, with defensive caps against pathological input.
- **Structured tables & lossless Markdown round-trip.** Tables can be edited as a matrix
- **Structured tables & Markdown round-trip.** Tables can be edited as a matrix
(read, insert/delete rows, set cells by `[row,col]`) without resending the document, and
a page can be exported to and re-imported from a self-contained Docmost-flavoured
Markdown file that preserves inline comment anchors and diagrams.
Markdown file that preserves inline comment anchors and diagrams (block ids regenerate
and a fixed no-Markdown-representation attr set is dropped — see `exportPageMarkdown`).
- **Token-optimized responses.** API responses are filtered down to the fields agents
actually need, and large collections (spaces, pages, comments, history) are paginated.
- **Hardened runtime.** Global handlers keep a stray socket error from tearing down the
stdio server; `move_page` requires a positively confirmed success; the diff engine
stdio server; `movePage` requires a positively confirmed success; the diff engine
falls back to a coarse block diff rather than hard-failing on a pathological document.
---
@@ -358,7 +390,7 @@ npm run test:e2e
This project began as a fork of [MrMartiniMo/docmost-mcp](https://github.com/MrMartiniMo/docmost-mcp)
(by Moritz Krause) and extends it substantially — adding per-block node editing,
surgical text edits, the sandboxed `docmost_transform`, version history / diff / restore,
surgical text edits, the sandboxed `docmostTransform`, version history / diff / restore,
comments, image insert/replace, public sharing, server-side page copy, dual
JSON/Markdown reads, transparent re-authentication and significant hardening. The comment
tools were ported from upstream PR #3 by Max Nikitin. Thanks to both.
+122 -90
View File
@@ -12,7 +12,7 @@
> небольшую функцию, которая чинит текст*, чем перечитывать и заново выдавать весь
> документ. Поэтому сервер построен вокруг того, как модели на самом деле удобно
> редактировать: адресовать блок по id, сделать find/replace или передать трансформ
> `(doc, ctx) => doc` и позволить модели *запрограммировать* правку. `docmost_transform`
> `(doc, ctx) => doc` и позволить модели *запрограммировать* правку. `docmostTransform`
> это и есть такой интерфейс. Другие Docmost-MCP «заточены под человека» — они дают
> «открыть страницу» и «заменить страницу»; этот даёт примитивы редактирования, в которых
> модель сильна.
@@ -43,7 +43,7 @@ Docmost-MCP не сочетают:
| **Нужна enterprise-лицензия** | **Нет** | **Да** | Нет | Нет | Нет |
| Аутентификация | email + пароль, **авто-переавторизация** | API-ключ | email + пароль | cookie `authToken` (копировать из DevTools) | API Docmost / **напрямую PostgreSQL** |
| Чтение страницы как Markdown | ✅ | ✅ | ✅ | ✅ | ✅ (только чтение) |
| **Lossless Markdown round-trip** (экспорт/импорт, сохраняет якоря комментариев) | ✅ | — | — | — | — |
| **Markdown round-trip** (экспорт/импорт, сохраняет якоря комментариев) | ✅ | — | — | — | — |
| Чтение **lossless ProseMirror JSON** (с id блоков) | ✅ | — | — | — | — |
| **Компактная структура страницы** (дешёвый поиск id блока) | ✅ | — | — | — | — |
| **Получение одного блока** (по id или индексу) | ✅ | — | — | — | — |
@@ -71,9 +71,9 @@ Docmost-MCP не сочетают:
- **Экономия токенов при редактировании.** Большинство Docmost-MCP (и официальный)
предлагают только запись «заменить всю страницу» — агент вынужден скачать весь документ,
изменить и загрузить обратно, оплачивая весь документ **дважды** на каждой мелкой
правке. Этот сервер позволяет агенту изменить ровно один блок (`patch_node` /
`insert_node` / `delete_node`), сделать find/replace с сохранением структуры
(`edit_page_text`) или скопировать страницу на стороне сервера (`copy_page_content`) —
правке. Этот сервер позволяет агенту изменить ровно один блок (`patchNode` /
`insertNode` / `deleteNode`), сделать find/replace с сохранением структуры
(`editPageText`) или скопировать страницу на стороне сервера (`copyPageContent`) —
**причём документ ни разу не проходит через модель**.
- **Записи, которые не воюют с редактором.** Наивная запись через REST конфликтует с тем,
@@ -87,12 +87,12 @@ Docmost-MCP не сочетают:
- **Агентоориентированная модель редактирования.** Серверы «под человека» дают «открыть
страницу» и «заменить страницу», потому что это отражает то, как работает человек. Модель
редактирует лучше, *программируя* правку — адресуя блоки по id, делая find/replace или
передавая трансформ `(doc, ctx) => doc` (`docmost_transform`, с dry-run диффом перед
передавая трансформ `(doc, ctx) => doc` (`docmostTransform`, с dry-run диффом перед
коммитом). Этот сервер построен вокруг этого — поэтому у него есть примитивы
редактирования, которых у остальных просто нет.
- **Страховка при редактировании, которой нет у других.** `list_page_history`
`diff_page_versions``restore_page_version` дают агенту (и вам) полный цикл «посмотреть
- **Страховка при редактировании, которой нет у других.** `listPageHistory`
`diffPageVersions``restorePageVersion` дают агенту (и вам) полный цикл «посмотреть
и откатить». Дифф использует *тот же* конвейер `recreateTransform → ChangeSet →
simplifyChanges`, что и встроенный просмотр истории Docmost, так что результат совпадает
с продуктом.
@@ -113,55 +113,62 @@ Docmost-MCP не сочетают:
### Чтение и поиск
- **`get_workspace`** — Информация о текущем воркспейсе Docmost.
- **`list_spaces`** — Все пространства воркспейса.
- **`list_pages`** — Недавние страницы пространства, по убыванию `updatedAt` (по умолчанию
- **`getWorkspace`** — Информация о текущем воркспейсе Docmost.
- **`listSpaces`** — Все пространства воркспейса.
- **`listPages`** — Недавние страницы пространства, по убыванию `updatedAt` (по умолчанию
50, максимум 100). Для поиска в больших пространствах используйте `search`.
- **`search`** — Полнотекстовый поиск по страницам и контенту (ограничен `limit`, максимум
100).
- **`get_page`** — Контент страницы как чистый **Markdown** (удобно, но это
*lossy*-представление — id блоков и точная структура таблиц/коллаутов аппроксимируются).
- **`get_page_json`** — **Lossless ProseMirror/TipTap JSON** страницы, включая `attrs.id`
- **`getPage`** — Контент страницы как чистый **Markdown** (канонично для текста; теряет
лишь id блоков, якоря разрешённых комментариев и фиксированный набор атрибутов без
markdown-представления — спаны/colwidth/фон ячеек таблиц, отступы (indent),
`callout.icon`, `orderedList.type` и `internal`/`target`/`rel`/`class` у ссылок;
используйте `getPageJson`, когда они нужны).
- **`getPageJson`** — **Lossless ProseMirror/TipTap JSON** страницы, включая `attrs.id`
каждого блока и `slugId`, используемый в URL. Именно его потребляют инструменты
поблочного редактирования.
- **`get_outline`** — Компактная структура страницы из блоков верхнего уровня (`{index,
- **`getOutline`** — Компактная структура страницы из блоков верхнего уровня (`{index,
type, id, level, firstText}`; для таблиц добавляются число строк/столбцов и тексты ячеек
заголовка, для списков — число пунктов) **без** тела документа. Дешёвый способ найти раздел или таблицу и получить
id блока перед `get_node` / `patch_node` / `insert_node`.
- **`get_node`** — Получить полное ProseMirror-поддерево одного блока (lossless), не
вытягивая всю страницу. Адресуйте его по id блока (из `get_outline` / `get_page_json`)
id блока перед `getNode` / `patchNode` / `insertNode`.
- **`getNode`** — Получить полное ProseMirror-поддерево одного блока (lossless), не
вытягивая всю страницу. Адресуйте его по id блока (из `getOutline` / `getPageJson`)
или формой `#<index>` для блока верхнего уровня — используйте `#<index>` для
таблиц/строк/ячеек, у которых нет id.
### Жизненный цикл страниц
- **`create_page`** — Создать страницу из Markdown и поместить в иерархию (опционально
- **`createPage`** — Создать страницу из Markdown и поместить в иерархию (опционально
`parentPageId`) одним вызовом. Использует import API Docmost для чистой конвертации
Markdown→ProseMirror.
- **`rename_page`** — Изменить только заголовок страницы, не трогая и не пересылая контент.
- **`move_page`** — Сменить родителя страницы (вложить или вынести в корень); поддерживает
- **`renamePage`** — Изменить только заголовок страницы, не трогая и не пересылая контент.
- **`movePage`** — Сменить родителя страницы (вложить или вынести в корень); поддерживает
позиционирование по fractional-index. Возвращает успех только при *положительно
подтверждённом* результате.
- **`delete_page`** — Удалить одну страницу.
- **`copy_page_content`** — Заменить тело одной страницы копией тела другой, **полностью на
- **`deletePage`** — Удалить одну страницу.
- **`copyPageContent`** — Заменить тело одной страницы копией тела другой, **полностью на
стороне сервера** — документ не проходит через модель. У целевой страницы сохраняются
собственные заголовок и slug (URL не меняется).
### Редактирование
- **`edit_page_text`** — Хирургический find/replace внутри текста страницы. Сохраняет
- **`editPageText`** — Хирургический find/replace внутри текста страницы. Сохраняет
**всю** структуру: id блоков, marks, ссылки, коллауты, таблицы. Предпочтительный
инструмент для правки формулировок, опечаток, чисел и имён.
- **`patch_node`** — Заменить один блок, адресованный по `attrs.id` (из `get_page_json`),
- **`patchNode`** — Заменить один блок, адресованный по `attrs.id` (из `getPageJson`),
без пересылки документа.
- **`insert_node`** — Вставить блок до/после другого (по `attrs.id` или по якорному тексту)
- **`insertNode`** — Вставить блок до/после другого (по `attrs.id` или по якорному тексту)
либо добавить в конец.
- **`delete_node`** — Удалить один блок по его `attrs.id`.
- **`update_page_json`** — Заменить весь контент страницы документом ProseMirror (массовые
- **`deleteNode`** — Удалить один блок по его `attrs.id`.
- **`updatePageJson`** — Заменить весь контент страницы документом ProseMirror (массовые
перезаписи или когда у узлов нет id). `content` опционален — опустите его, чтобы изменить
только заголовок. Сохраняет переданные id блоков, поэтому якоря заголовков и история
остаются стабильными.
- **`docmost_transform`** — Агентоориентированный интерфейс редактирования: вместо
- **`updatePageMarkdown`** — Заменить тело страницы (и опционально заголовок) новым
**обычным Markdown**. Всё тело переимпортируется (id блоков перегенерируются — для
хирургических правок или сохранения id используйте `editPageText` / `patchNode` /
`updatePageJson`). Markdown в диалекте Docmost разбирается, включая inline-сноски `^[...]`.
- **`docmostTransform`** — Агентоориентированный интерфейс редактирования: вместо
перепечатывания документа агент **пишет функцию, которая его чинит**. Редактирует
страницу, запуская произвольный **JS-трансформ `(doc, ctx) => doc`** на её *живом*
документе ProseMirror. Работает в **песочнице** (без `require`/`process`/`fs`/сети,
@@ -173,43 +180,46 @@ Docmost-MCP не сочетают:
### Таблицы
- **`table_get`** — Прочитать таблицу как матрицу: `{rows, cols, cells (text[][]),
- **`tableGet`** — Прочитать таблицу как матрицу: `{rows, cols, cells (text[][]),
cellIds}` (id абзаца на ячейку или `null`). Адресуйте таблицу через `#<index>` (из
`get_outline`) или любой id блока внутри неё. Используйте `cellIds` вместе с `patch_node`
`getOutline`) или любой id блока внутри неё. Используйте `cellIds` вместе с `patchNode`
для правок ячеек с форматированием.
- **`table_insert_row`** — Вставить строку из текстовых ячеек, дополненную до числа
- **`tableInsertRow`** — Вставить строку из текстовых ячеек, дополненную до числа
столбцов таблицы (передать ячеек больше числа столбцов — ошибка). `index` — 0-based
позиция вставки (0 вставляет перед заголовком); опустите, чтобы добавить в конец.
- **`table_delete_row`** — Удалить строку по 0-based `index`. Отказывается удалять
- **`tableDeleteRow`** — Удалить строку по 0-based `index`. Отказывается удалять
единственную строку таблицы; удаление строки 0 делает заголовком следующую строку.
- **`table_update_cell`** — Задать текстовое содержимое ячейки `[row, col]` (0-based). Для
форматирования используйте `patch_node` по id абзаца ячейки из `table_get`.
- **`tableUpdateCell`** — Задать текстовое содержимое ячейки `[row, col]` (0-based). Для
форматирования используйте `patchNode` по id абзаца ячейки из `tableGet`.
### Markdown: экспорт и импорт
- **`export_page_markdown`** — Экспортировать страницу в один самодостаточный, **lossless
Markdown в диалекте Docmost**: мета-заголовок, тело с inline-якорями комментариев и
диаграммами и завершающий блок тредов комментариев. Рассчитан на цикл «скачать →
отредактировать тело → `import_page_markdown`», сохраняющий всё, включая выделения
комментариев.
- **`import_page_markdown`** — Заменить контент страницы из Markdown-файла в диалекте
Docmost, созданного `export_page_markdown`, восстанавливая якоря-выделения комментариев и
диаграммы из их inline-HTML. (Треды комментариев из файла не пересоздаются на сервере —
записываются только тело страницы и inline-марки комментариев; тредами управляйте через
инструменты/UI комментариев.)
- **`exportPageMarkdown`** — Экспортировать страницу в один самодостаточный
**Markdown в диалекте Docmost**: мета-заголовок, тело с inline-якорями комментариев и
диаграммами и завершающий блок тредов комментариев. Round-trip скачать → отредактировать →
импортировать перегенерирует id блоков и **молча отбрасывает** набор атрибутов без
markdown-представления (спаны/colwidth/фон ячеек таблиц, отступы (indent), `callout.icon`,
`orderedList.type`, `internal`/`target`/`rel`/`class` у ссылок); держите их в ProseMirror
JSON, если они должны выжить. Чтобы заменить тело страницы из обычного авторского Markdown,
используйте `updatePageMarkdown`.
> **Удалено в этом релизе:** `importPageMarkdown` (парсер round-trip для
> экспортированного Docmost-Markdown-файла) **больше не отдаётся на внешней MCP-поверхности**.
> Чтобы заменить тело страницы из Markdown, используйте **`updatePageMarkdown`** (замена
> тела обычным Markdown). См. заметку о миграции в CHANGELOG.
### Изображения
- **`insert_image`** — Загрузить локальное изображение и вставить за один шаг: добавить в
- **`insertImage`** — Загрузить локальное изображение и вставить за один шаг: добавить в
конец, поставить вместо текстового плейсхолдера (`replaceText`) или после заданного блока
(`afterText`). Сохраняет id всех остальных блоков.
- **`replace_image`** — Заменить существующее изображение. Загружает новый файл как **новое
- **`replaceImage`** — Заменить существующее изображение. Загружает новый файл как **новое
вложение** (чистый URL, который рендерится и сбрасывает кэш браузера), затем
перенаправляет все узлы, ссылавшиеся на старое вложение (рекурсивно, включая
коллауты/таблицы), через живой документ, сохраняя комментарии, выравнивание и alt-текст.
(Перезапись «по месту» намеренно не используется — некоторые версии Docmost портят
вложение при перезаписи.)
- **`stash_page`** — Сериализовать страницу целиком (её полный ProseMirror JSON) в
- **`stashPage`** — Сериализовать страницу целиком (её полный ProseMirror JSON) в
эфемерный blob в оперативной памяти и вернуть ТОЛЬКО короткий анонимный URL — тело
никогда не попадает в контекст модели, поэтому это способ передать большую страницу
(вместе с её изображениями) внешнему потребителю без усечения. Каждое внутреннее
@@ -221,35 +231,35 @@ Docmost-MCP не сочетают:
### Комментарии
- **`create_comment`** — Добавить комментарий к странице, опционально **привязав inline** к
- **`createComment`** — Добавить комментарий к странице, опционально **привязав inline** к
точному фрагменту текста (первое вхождение оборачивается comment-маркой).
- **`list_comments`** — Список комментариев страницы (контент возвращается как Markdown).
- **`update_comment`** — Изменить существующий комментарий.
- **`delete_comment`** — Удалить комментарий.
- **`resolve_comment`** — Закрыть (resolve) или переоткрыть тред комментария (обратимо). Resolve
доступен только для корневых комментариев; тред и ответы сохраняются, в отличие от `delete_comment`.
- **`check_new_comments`** — Найти комментарии, созданные после заданной метки времени
- **`listComments`** — Список комментариев страницы (контент возвращается как Markdown).
- **`updateComment`** — Изменить существующий комментарий.
- **`deleteComment`** — Удалить комментарий.
- **`resolveComment`** — Закрыть (resolve) или переоткрыть тред комментария (обратимо). Resolve
доступен только для корневых комментариев; тред и ответы сохраняются, в отличие от `deleteComment`.
- **`checkNewComments`** — Найти комментарии, созданные после заданной метки времени
ISO-8601, по пространству, опционально в рамках поддерева страниц — идеально для агента,
который следит за обратной связью в документе.
### Версии и история
- **`list_page_history`** — Сохранённые версии страницы (Docmost авто-снапшотит при каждом
- **`listPageHistory`** — Сохранённые версии страницы (Docmost авто-снапшотит при каждом
сохранении), новые сверху, курсорная пагинация. id каждого элемента — это `historyId`.
- **`diff_page_versions`** — Дифф двух версий (или версии против живой страницы).
- **`diffPageVersions`** — Дифф двух версий (или версии против живой страницы).
Возвращает вставленный/удалённый текст, счётчики целостности (изображения, ссылки,
таблицы, коллауты, маркеры сносок) и человекочитаемую Markdown-сводку — посчитано тем же
конвейером, что использует встроенный просмотр истории Docmost.
- **`restore_page_version`** — Записать сохранённую версию обратно как текущий контент. У
- **`restorePageVersion`** — Записать сохранённую версию обратно как текущий контент. У
Docmost нет эндпоинта восстановления, поэтому создаётся **новый** снапшот — само
восстановление тоже обратимо.
### Публикация
- **`share_page`** — Сделать страницу публично доступной (идемпотентно) и вернуть её
- **`sharePage`** — Сделать страницу публично доступной (идемпотентно) и вернуть её
публичный URL (`<app>/share/<key>/p/<slugId>`); опционально индексирование поисковиками.
- **`unshare_page`** — Отозвать публичный доступ к странице.
- **`list_shares`** — Все публичные ссылки воркспейса с заголовками и публичными URL.
- **`unsharePage`** — Отозвать публичный доступ к странице.
- **`listShares`** — Все публичные ссылки воркспейса с заголовками и публичными URL.
---
@@ -258,28 +268,29 @@ Docmost-MCP не сочетают:
Та же подсказка отдаётся в рантайме через поле `instructions` MCP-сервера, так что
подходящие клиенты направляют модель автоматически.
- **Правки текста** (формулировки, опечатки, числа): `edit_page_text`.
- **Один блок** (абзац/заголовок/коллаут/ячейка таблицы): `patch_node` / `insert_node` /
`delete_node`, адресуя узел по его `attrs.id` из `get_page_json`.
- **Изображения**: `insert_image` / `replace_image`.
- **Новая страница**: `create_page`.
- **Массовая перезапись или узлы без id**: `update_page_json`.
- **Правки текста** (формулировки, опечатки, числа): `editPageText`.
- **Один блок** (абзац/заголовок/коллаут/ячейка таблицы): `patchNode` / `insertNode` /
`deleteNode`, адресуя узел по его `attrs.id` из `getPageJson`.
- **Изображения**: `insertImage` / `replaceImage`.
- **Новая страница**: `createPage`.
- **Массовая перезапись или узлы без id**: `updatePageJson` (ProseMirror) или
`updatePageMarkdown` (замена тела обычным Markdown).
- **Многошаговая / скриптовая перезапись** (перенумерация, сноски, согласованные правки):
`docmost_transform` — предпросмотр через `dryRun`, затем применение.
`docmostTransform` — предпросмотр через `dryRun`, затем применение.
- **Скопировать контент целой страницы из другой** (на стороне сервера):
`copy_page_content`.
- **Переименовать страницу** (только заголовок): `rename_page`.
- **Чтение**: `get_page` (Markdown) / `get_page_json` (lossless ProseMirror с id).
- **Просмотр изменений**: `list_page_history``diff_page_versions`
`restore_page_version`.
- **Комментарии**: `create_comment` (с опциональной inline-привязкой) / `list_comments` /
`update_comment` / `resolve_comment` / `delete_comment` / `check_new_comments`.
- **Дешёвая навигация по странице** (найти раздел/таблицу, получить id блока): `get_outline`
`get_node`.
- **Таблицы** (добавить/удалить строку, задать ячейку): `table_get` / `table_insert_row` /
`table_delete_row` / `table_update_cell`.
- **Round-trip страницы через Markdown** (скачать, отредактировать, залить обратно без
потерь, с комментариями): `export_page_markdown` / `import_page_markdown`.
`copyPageContent`.
- **Переименовать страницу** (только заголовок): `renamePage`.
- **Чтение**: `getPage` (Markdown) / `getPageJson` (lossless ProseMirror с id).
- **Просмотр изменений**: `listPageHistory``diffPageVersions`
`restorePageVersion`.
- **Комментарии**: `createComment` (с опциональной inline-привязкой) / `listComments` /
`updateComment` / `resolveComment` / `deleteComment` / `checkNewComments`.
- **Дешёвая навигация по странице** (найти раздел/таблицу, получить id блока): `getOutline`
`getNode`.
- **Таблицы** (добавить/удалить строку, задать ячейку): `tableGet` / `tableInsertRow` /
`tableDeleteRow` / `tableUpdateCell`.
- **Экспорт страницы в самодостаточный Markdown** (с якорями комментариев): `exportPageMarkdown`.
- **Заменить тело страницы из Markdown**: `updatePageMarkdown`.
---
@@ -291,28 +302,49 @@ Docmost-MCP не сочетают:
правки, которых ещё нет в дебаунс-снапшоте REST), затем **читает → трансформирует →
пишет синхронно** в одном тике, чтобы никакое удалённое обновление не вклинилось, и
**ждёт подтверждения сохранения** до возврата.
- **Сериализация записи по странице.** Асинхронный мьютекс по `pageId` гарантирует, что
две записи MCP в одну страницу никогда не пересекаются; разные страницы друг друга не
блокируют.
- **Сериализация записи по странице.** Асинхронный мьютекс по разрешённому **UUID**
страницы (никогда не по slugId) гарантирует, что две записи MCP в одну страницу никогда
не пересекаются; разные страницы друг друга не блокируют. Хелпер блокировки падает сразу
(fail-fast), если ему передали не-UUID ключ, — путь записи, забывший разрезолвить id, не
сможет молча взять лок под расщеплённым ключом.
**Требование к деплою — один инстанс или sticky-сессии.** Этот мьютекс — процесс-локальный
`Map`, а кэш collab-сессий и хранилище `stashPage` живут только в RAM одного процесса. За
**мультиреплика**-балансировщиком **без sticky-сессий** две реплики могут одновременно
«держать» лок одной страницы, и сериализация по странице молча теряется. Запускайте
MCP/приложение **одним инстансом** либо прибивайте трафик страницы к одной реплике
(sticky-сессии / consistent hashing по id страницы). Кросс-процессной блокировки (например,
Postgres advisory-lock) намеренно пока нет — осознанное задокументированное ограничение.
См. комментарий в `Dockerfile` и блок `MCP collaboration write path` в `.env.example`.
**Окно устаревших прав.** Кэшированная collab-сессия пишет под токеном, захваченным в
момент connect (а кэш collab-токена переиспользует токен в пределах своего TTL), поэтому
**отозванный** доступ к странице может лагать до `MCP_COLLAB_SESSION_MAX_AGE_MS` (жёсткий
срок жизни сессии, по умолчанию 10 мин), пока следующая переавторизация его не подхватит.
Уменьшите значение, чтобы сократить лаг ценой большего числа переподключений. Это
ограниченное окно — принятый trade-off; push-инвалидации кэша при смене прав нет.
- **Прозрачная переавторизация.** Логин по email/паролю; истёкшие токены обновляются
автоматически на первом 401/403 (покрывая JSON, multipart-загрузку и путь токена
коллаборации), с дедупликацией параллельных логинов, так что пачка вызовов вызывает один
повторный логин.
- **Lossless- и lossy-чтение.** `get_page_json` возвращает точное дерево ProseMirror с id
блоков; `get_page` возвращает чистый Markdown для удобства.
- **Точные чтения.** `getPageJson` возвращает точное дерево ProseMirror с id блоков;
`getPage` возвращает канонический Markdown, теряющий лишь фиксированный, документированный
набор атрибутов.
- **Полная схема Docmost.** Конвертация Markdown↔ProseMirror поддерживает коллауты
(включая вложенные), списки задач (маркированные *и* нумерованные чек-листы), таблицы,
блоки формул, эмбеды, выделение, под/надстрочный текст и прочее, с защитными лимитами
против патологического ввода.
- **Структурные таблицы и lossless Markdown round-trip.** Таблицы можно редактировать как
- **Структурные таблицы и Markdown round-trip.** Таблицы можно редактировать как
матрицу (чтение, вставка/удаление строк, задание ячеек по `[row, col]`) без пересылки
документа, а страницу — экспортировать и заново импортировать как самодостаточный
Markdown-файл в диалекте Docmost, сохраняющий inline-якоря комментариев и диаграммы.
Markdown-файл в диалекте Docmost, сохраняющий inline-якоря комментариев и диаграммы
(id блоков перегенерируются, а фиксированный набор атрибутов без markdown-представления
отбрасывается — см. `exportPageMarkdown`).
- **Ответы, оптимизированные по токенам.** Ответы API урезаются до полей, действительно
нужных агентам, а большие коллекции (пространства, страницы, комментарии, история)
пагинируются.
- **Закалённый рантайм.** Глобальные обработчики не дают случайной ошибке сокета уронить
stdio-сервер; `move_page` требует положительно подтверждённого успеха; движок диффа
stdio-сервер; `movePage` требует положительно подтверждённого успеха; движок диффа
откатывается к грубому поблочному диффу, а не падает на патологическом документе.
---
@@ -372,7 +404,7 @@ npm run test:e2e
Проект начинался как форк
[MrMartiniMo/docmost-mcp](https://github.com/MrMartiniMo/docmost-mcp) (автор Moritz Krause)
и существенно его расширяет — добавлены поблочное редактирование узлов, хирургические
правки текста, песочница `docmost_transform`, история версий / дифф / восстановление,
правки текста, песочница `docmostTransform`, история версий / дифф / восстановление,
комментарии, вставка/замена изображений, публичные ссылки, серверное копирование страниц,
двойное чтение JSON/Markdown, прозрачная переавторизация и значительное упрочнение.
Инструменты комментариев портированы из upstream PR #3 от Max Nikitin. Спасибо обоим.
+10 -10
View File
@@ -22,14 +22,14 @@ are debounced server-side, so the script waits ~16 s before reading back via RES
| # | Tool / path | What is checked | Expected |
|---|-------------|-----------------|----------|
| 1 | `create_page` | title with spaces, slugId returned | page created, title intact |
| 1 | `createPage` | title with spaces, slugId returned | page created, title intact |
| 2 | `update_page` (markdown) | headings, **bold**/*italic*/~~strike~~/`code`/link, nested bullet + ordered lists, blockquote, code block, `:::callout:::`, table | all structures survive re-import |
| 3 | `get_page_json` | lossless ProseMirror, block ids, callout/table nodes | present (note: reads the **debounced** REST snapshot — recent collab writes may lag a few seconds) |
| 4 | `edit_page_text` | surgical replace; block ids + marks preserved; ambiguous match rejected; missing match reported | edits applied, ids stable, errors correct |
| 5 | `update_page_json` | full lossless write; custom block ids preserved; existing content (text edits, images, callout, table) not lost | round-trips intact |
| 3 | `getPageJson` | lossless ProseMirror, block ids, callout/table nodes | present (note: reads the **debounced** REST snapshot — recent collab writes may lag a few seconds) |
| 4 | `editPageText` | surgical replace; block ids + marks preserved; ambiguous match rejected; missing match reported | edits applied, ids stable, errors correct |
| 5 | `updatePageJson` | full lossless write; custom block ids preserved; existing content (text edits, images, callout, table) not lost | round-trips intact |
| 6 | `upload_image` | uploads attachment, returns node | src is a **clean** `/api/files/<id>/<file>` URL, served `200 image/*` |
| 7 | `insert_image` (append / `replaceText` / `afterText`) | three placements | image lands in the right place, all other block ids preserved |
| 8 | **`replace_image`** | swap an existing figure for new bytes; comments/align/alt preserved; **the new URL must actually serve the image** | new image renders (`200`), old node repointed |
| 7 | `insertImage` (append / `replaceText` / `afterText`) | three placements | image lands in the right place, all other block ids preserved |
| 8 | **`replaceImage`** | swap an existing figure for new bytes; comments/align/alt preserved; **the new URL must actually serve the image** | new image renders (`200`), old node repointed |
## Image-specific assertions (the recurring bug area)
@@ -39,7 +39,7 @@ For every uploaded/inserted/replaced image, assert at the HTTP level that the
* `GET <src>``200`, `Content-Type: image/*`, body starts with the image magic
(`89 50 4E 47` for PNG, etc.).
* `src` does **not** contain a `?v=` query (see "Known pitfalls").
* After `replace_image`: the returned `newAttachmentId` **differs** from the old
* After `replaceImage`: the returned `newAttachmentId` **differs** from the old
one (replacement uses a fresh attachment → fresh URL), and `GET <new src>``200`.
* The old image node on the page is repointed to the new attachmentId.
@@ -64,7 +64,7 @@ broken/empty figure.
Uploading with an existing `attachmentId` (`POST /files/upload` + `attachmentId`)
overwrites the bytes in place. On this Docmost the attachment then returns
**500 for every URL** (clean, `?v=`, any filename) → broken image. Therefore
`replace_image` must upload a **new** attachment and repoint the nodes; the new
`replaceImage` must upload a **new** attachment and repoint the nodes; the new
id yields a new URL that both renders and busts the browser cache. The old
attachment is left as an unreferenced orphan: Docmost exposes **no HTTP API to
delete a single content attachment** (verified against the attachment
@@ -80,9 +80,9 @@ broken/empty figure.
from `?v=`. Image `src` is kept clean (`/api/files/<id>/<file>`); cache-busting
on replace is achieved by the new attachment id.
3. **REST snapshot lag.** `get_page_json` reads the debounced DB snapshot, so a
3. **REST snapshot lag.** `getPageJson` reads the debounced DB snapshot, so a
write made moments earlier may not be visible yet. Wait (~16 s) before reading
back, and never feed a possibly-stale snapshot straight into `update_page_json`.
back, and never feed a possibly-stale snapshot straight into `updatePageJson`.
4. **Callout type narrowing (minor, open).** A `:::warning` callout is imported as
`type: "info"` — the markdown→callout conversion does not carry non-`info`
+63
View File
@@ -0,0 +1,63 @@
{
"$comment": "Semantic palettes for drawioFromGraph (issue #425). DATA, not code: node `kind` -> fill/stroke slot, edge `kind` -> line-style props, per preset. The `default` node palette is the issue's base table; `dark` keeps the same hues on a dark canvas with lighter strokes/font; `colorblind-safe` maps every slot onto the Okabe-Ito qualitative palette (8 colours proven distinguishable for all common colour-vision deficiencies) so no two adjacent kinds collide. `fontColor`/`fillColor`/`strokeColor` are exact draw.io values. `edgeDefault` is the fallback line style; `group` is the (always-transparent) container stroke per preset.",
"presets": {
"default": {
"canvasDark": false,
"nodes": {
"service": { "fillColor": "#dae8fc", "strokeColor": "#6c8ebf", "fontColor": "#000000" },
"db": { "fillColor": "#d5e8d4", "strokeColor": "#82b366", "fontColor": "#000000" },
"queue": { "fillColor": "#fff2cc", "strokeColor": "#d6b656", "fontColor": "#000000" },
"gateway": { "fillColor": "#ffe6cc", "strokeColor": "#d79b00", "fontColor": "#000000" },
"error": { "fillColor": "#f8cecc", "strokeColor": "#b85450", "fontColor": "#000000" },
"external": { "fillColor": "#f5f5f5", "strokeColor": "#666666", "fontColor": "#333333" },
"security": { "fillColor": "#e1d5e7", "strokeColor": "#9673a6", "fontColor": "#000000" }
},
"edges": {
"sync": { "props": "" },
"async": { "props": "dashed=1;" },
"error": { "props": "dashed=1;strokeColor=#DD344C;" }
},
"edgeDefault": { "strokeColor": "#333333", "fontColor": "#333333" },
"group": { "strokeColor": "#666666", "fontColor": "#333333" }
},
"dark": {
"canvasDark": true,
"nodes": {
"service": { "fillColor": "#1a2a44", "strokeColor": "#7ea6e0", "fontColor": "#dae8fc" },
"db": { "fillColor": "#1f331e", "strokeColor": "#97d077", "fontColor": "#d5e8d4" },
"queue": { "fillColor": "#3a3218", "strokeColor": "#e5c15a", "fontColor": "#fff2cc" },
"gateway": { "fillColor": "#3a2812", "strokeColor": "#ffb570", "fontColor": "#ffe6cc" },
"error": { "fillColor": "#3a1c1b", "strokeColor": "#e08e8b", "fontColor": "#f8cecc" },
"external": { "fillColor": "#2b2b2b", "strokeColor": "#999999", "fontColor": "#e0e0e0" },
"security": { "fillColor": "#2c2338", "strokeColor": "#b39ddb", "fontColor": "#e1d5e7" }
},
"edges": {
"sync": { "props": "" },
"async": { "props": "dashed=1;" },
"error": { "props": "dashed=1;strokeColor=#ff6b6b;" }
},
"edgeDefault": { "strokeColor": "#cccccc", "fontColor": "#e0e0e0" },
"group": { "strokeColor": "#aaaaaa", "fontColor": "#e0e0e0" }
},
"colorblind-safe": {
"canvasDark": false,
"okabeIto": ["#000000", "#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7"],
"nodes": {
"service": { "fillColor": "#D6E9F5", "strokeColor": "#0072B2", "fontColor": "#000000" },
"db": { "fillColor": "#D6EFE4", "strokeColor": "#009E73", "fontColor": "#000000" },
"queue": { "fillColor": "#FCF8CC", "strokeColor": "#F0E442", "fontColor": "#000000" },
"gateway": { "fillColor": "#FBEBD0", "strokeColor": "#E69F00", "fontColor": "#000000" },
"error": { "fillColor": "#F7DDCC", "strokeColor": "#D55E00", "fontColor": "#000000" },
"external": { "fillColor": "#EDEDED", "strokeColor": "#000000", "fontColor": "#000000" },
"security": { "fillColor": "#F3DEEB", "strokeColor": "#CC79A7", "fontColor": "#000000" }
},
"edges": {
"sync": { "props": "" },
"async": { "props": "dashed=1;" },
"error": { "props": "dashed=1;strokeColor=#D55E00;" }
},
"edgeDefault": { "strokeColor": "#000000", "fontColor": "#000000" },
"group": { "strokeColor": "#000000", "fontColor": "#000000" }
}
}
}
Binary file not shown.
+14 -5
View File
@@ -5,18 +5,26 @@
"private": true,
"type": "module",
"main": "./build/index.js",
"types": "./build/index.d.ts",
"exports": {
".": "./build/index.js",
"./http": "./build/http.js"
".": {
"types": "./build/index.d.ts",
"default": "./build/index.js"
},
"./http": {
"types": "./build/http.d.ts",
"default": "./build/http.js"
}
},
"bin": {
"docmost-mcp": "./build/stdio.js"
},
"scripts": {
"build": "tsc",
"gen:stamp": "node scripts/gen-registry-stamp.mjs",
"build": "node scripts/gen-registry-stamp.mjs && tsc",
"start": "node build/stdio.js",
"watch": "tsc --watch",
"pretest": "tsc",
"watch": "node scripts/gen-registry-stamp.mjs && tsc --watch",
"pretest": "node scripts/gen-registry-stamp.mjs && tsc",
"test": "node --test \"test/unit/*.test.mjs\" \"test/mock/*.test.mjs\"",
"test:unit": "node --test \"test/unit/*.test.mjs\"",
"test:mock": "node --test \"test/mock/*.test.mjs\"",
@@ -49,6 +57,7 @@
"@tiptap/starter-kit": "3.20.4",
"@types/jsdom": "^27.0.0",
"axios": "^1.6.0",
"elkjs": "^0.11.1",
"form-data": "^4.0.0",
"jsdom": "^27.4.0",
"marked": "^17.0.1",
@@ -0,0 +1,67 @@
// Codegen: emit src/registry-stamp.generated.ts with a REGISTRY_STAMP hash of
// the tool-specs REGISTRY CONTENT, so a build/ vs src/ skew (issue #447) is
// detectable at runtime.
//
// WHY hash the raw source text (not extracted structured data):
// SHARED_TOOL_SPECS carries `buildShape` functions (the input SCHEMAS) which are
// NOT serializable. The input schema is exactly one of the things that MUST stay
// in sync between build/ and src/, so we cannot drop it from the hash. Rather
// than probe zod with a fragile shim to reconstruct the schema shape, we hash the
// STABLE, deterministic source TEXT of tool-specs.ts. That text fully captures
// every field that must stay in sync — mcpName, inAppKey, description, tier,
// catalogLine AND the buildShape bodies (input schemas) — with zero probing
// fragility. Any edit to a spec (a renamed tool, a reworded description, a
// changed schema field) changes the text and therefore the stamp.
//
// DETERMINISM: the hash is computed over the file bytes with line endings
// normalized to LF and a single trailing newline stripped, so a CRLF checkout or
// an editor's trailing-newline habit cannot make build/ and src/ disagree. No
// Date.now / randomness. The loader's dev-only stale-check (docmost-client.loader.ts)
// re-runs THIS SAME normalization + sha256 over src/tool-specs.ts and compares to
// the built REGISTRY_STAMP; the two must compute identically.
//
// This script runs from the `build` and `pretest` npm scripts BEFORE tsc, so
// build/ always carries a stamp derived from the tool-specs.ts that was compiled.
import { createHash } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const SRC_DIR = join(__dirname, '..', 'src');
const TOOL_SPECS_PATH = join(SRC_DIR, 'tool-specs.ts');
const OUT_PATH = join(SRC_DIR, 'registry-stamp.generated.ts');
/**
* Deterministic stamp of the tool-specs registry content. Kept as a plain
* function (exported) so the algorithm has a single home; the loader duplicates
* only the tiny normalize+sha256 steps because it lives in the CJS server build
* and cannot import this ESM script. If you change the normalization here, mirror
* it in apps/server/src/core/ai-chat/tools/docmost-client.loader.ts.
*/
export function computeRegistryStamp(toolSpecsSource) {
const normalized = toolSpecsSource.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
}
function main() {
const source = readFileSync(TOOL_SPECS_PATH, 'utf8');
const stamp = computeRegistryStamp(source);
const out =
'// AUTO-GENERATED by scripts/gen-registry-stamp.mjs — DO NOT EDIT BY HAND.\n' +
'// A deterministic hash of src/tool-specs.ts content (tool names, descriptions,\n' +
'// tiers, catalog lines and input schemas). Regenerated on every build/pretest\n' +
'// so build/ always matches the compiled src. The in-app loader recomputes this\n' +
'// from src and refuses to run on a mismatch (issue #447). This file is\n' +
'// gitignored and produced by the build — see .gitignore.\n' +
`export const REGISTRY_STAMP = ${JSON.stringify(stamp)};\n`;
writeFileSync(OUT_PATH, out, 'utf8');
// eslint-disable-next-line no-console
console.log(`gen-registry-stamp: wrote ${OUT_PATH} (${stamp.slice(0, 12)}…)`);
}
// Only run when invoked directly (not when imported for computeRegistryStamp).
if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
main();
}
+89 -4322
View File
File diff suppressed because it is too large Load Diff
+704
View File
@@ -0,0 +1,704 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import { assertFullUuid } from "./errors.js";
import {
filterWorkspace,
filterSpace,
filterPage,
filterComment,
filterSearchResult,
} from "../lib/filters.js";
import { convertProseMirrorToMarkdown } from "../lib/markdown-converter.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
applyAnchorInDoc,
countAnchorMatches,
getAnchoredText,
resolveAnchorSelection,
normalizeForMatch,
} from "../lib/comment-anchor.js";
import { closestBlockHint } from "../lib/text-normalize.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of CommentsMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements ICommentsMixin` fails to compile on drift.
export interface ICommentsMixin {
listComments(pageId: string, includeResolved?: boolean): any;
getComment(commentId: string): any;
createComment(pageId: string, content: string, type?: "page" | "inline", selection?: string, parentCommentId?: string, suggestedText?: string): any;
updateComment(commentId: string, content: string): any;
deleteComment(commentId: string): any;
resolveComment(commentId: string, resolved: boolean): any;
checkNewComments(spaceId: string, since: string, parentPageId?: string): any;
}
export function CommentsMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & ICommentsMixin> & TBase {
abstract class CommentsMixin extends Base implements ICommentsMixin {
// --- Comment methods (ported from upstream PR #3 by Max Nikitin) ---
/**
* Normalize a comment's `content` into a ProseMirror doc object before
* markdown conversion. createComment/updateComment send content as a
* JSON.stringify(...) STRING, and the server stores it as-is, so on read it
* comes back as a string. convertProseMirrorToMarkdown returns "" for a
* string, so parse it first (guarded fall back to the raw value on any
* parse failure so a non-JSON legacy value is still handled gracefully).
*/
protected parseCommentContent(content: any): any {
if (typeof content !== "string") return content;
try {
return JSON.parse(content);
} catch {
return content;
}
}
/**
* List comments on a page (cursor-paginated), content as markdown.
*
* DEFAULT (`includeResolved = false`) hides RESOLVED THREADS WHOLESALE so the
* agent sees only active discussions: a top-level comment with `resolvedAt`
* set AND every reply under it (a reply of a closed thread is part of the
* closed thread) are dropped from `items`. `resolvedThreadsHidden` reports how
* many resolved top-level threads were hidden so the agent can re-query with
* `includeResolved: true` to see everything. Active threads always stay.
*
* Returns `{ items, resolvedThreadsHidden }` (NOT a bare array) callers that
* need the full feed (lossless export, transformPage, checkNewComments) pass
* `includeResolved: true` and read `.items`.
*/
async listComments(pageId: string, includeResolved = false) {
await this.ensureAuthenticated();
let allComments: any[] = [];
let cursor: string | null = null;
// Hard ceiling + immovable-cursor guard (mirrors paginateAll): if /comments
// ever stops advancing the cursor (the exact #442 drift scenario) this loop
// would otherwise spin forever accumulating duplicates.
const MAX_PAGES = 50;
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = { pageId, limit: 100 };
if (cursor) payload.cursor = cursor;
const response = await this.client.post("/comments", payload);
const data = response.data.data || response.data;
const items = data.items || [];
allComments = allComments.concat(items);
// Advance strictly via the server-issued cursor. A missing nextCursor or a
// cursor identical to the one we just sent means the end (or a server that
// ignores our pagination param) — stop instead of re-fetching page one.
const next: string | null = data.meta?.nextCursor || null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with a still-advancing cursor means truncation.
if (page === MAX_PAGES - 1) truncated = true;
}
if (truncated) {
console.warn(
`listComments: comments for "${pageId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
const mapped = allComments.map((comment: any) => {
const markdown = comment.content
? convertProseMirrorToMarkdown(
this.parseCommentContent(comment.content),
)
: "";
return filterComment(comment, markdown);
});
if (includeResolved) {
return { items: mapped, resolvedThreadsHidden: 0 };
}
// Ids of RESOLVED top-level threads (a top-level comment has no
// parentCommentId). A whole thread is hidden when its root is resolved.
const resolvedRootIds = new Set(
mapped
.filter((c) => !c.parentCommentId && c.resolvedAt != null)
.map((c) => c.id),
);
const items = mapped.filter((c) => {
// Hide the resolved root itself and every reply anchored to it. A reply's
// own resolvedAt is irrelevant — its membership follows the parent thread.
// ASSUMPTION: Docmost's comment model is FLAT — a reply's parentCommentId
// always points at the thread ROOT (no reply-of-reply nesting), so a single
// level of parent lookup covers a whole thread. If nested replies are ever
// introduced, a deep reply of a resolved thread would need a root-walk here.
if (!c.parentCommentId) return !resolvedRootIds.has(c.id);
return !resolvedRootIds.has(c.parentCommentId);
});
return { items, resolvedThreadsHidden: resolvedRootIds.size };
}
async getComment(commentId: string) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("get_comment", "commentId", commentId);
await this.ensureAuthenticated();
const response = await this.client.post("/comments/info", { commentId });
const comment = response.data.data || response.data;
const markdown = comment.content
? convertProseMirrorToMarkdown(this.parseCommentContent(comment.content))
: "";
return {
data: filterComment(comment, markdown),
success: true,
};
}
/** Plain text of each TOP-LEVEL block of `doc`, for anchor-failure hints. */
protected topLevelBlockTexts(doc: any): string[] {
const content = doc && Array.isArray(doc.content) ? doc.content : [];
return content
.map((b: any) => blockPlainText(b))
.filter((t: string) => t.length > 0);
}
/**
* True when per-block anchoring failed but the (normalized) selection DOES
* appear in the blocks' joined plain text i.e. it straddles a block
* boundary. Blocks are joined with a newline (collapsed to one space by
* normalizeForMatch) so a selection whose parts are separated by a paragraph
* break still matches. Callers only reach here after single-block anchoring
* (incl. the markdown-strip fallback) has already failed.
*/
protected selectionSpansMultipleBlocks(
blockTexts: string[],
selection: string,
): boolean {
const normSel = normalizeForMatch(selection).norm.trim();
if (normSel.length === 0) return false;
const joined = normalizeForMatch(blockTexts.join("\n")).norm;
return joined.indexOf(normSel) !== -1;
}
/**
* Build the actionable error for a createComment anchor MISS, porting
* editPageText's self-correction affordances: an explicit "spans multiple
* blocks" message when the selection straddles a block boundary, otherwise a
* "closest block text" hint quoting the block that holds the selection's
* longest token. `live` switches the wording between the pre-check (reading the
* persisted page) and the post-create live-anchor failure (which rolls back).
*/
protected anchorNotFoundError(
doc: any,
selection: string,
live: boolean,
): Error {
const blockTexts = this.topLevelBlockTexts(doc);
const rolled = live ? " The comment was rolled back." : "";
if (this.selectionSpansMultipleBlocks(blockTexts, selection)) {
return new Error(
"createComment: the selection spans multiple blocks; anchor on a " +
"contiguous fragment within a SINGLE paragraph/block (<=250 chars)." +
rolled,
);
}
const where = live ? "in the live document" : "in the page";
return new Error(
`createComment: could not find the selection text ${where} to anchor ` +
"the comment. Provide the EXACT contiguous text from a single " +
"paragraph/block (<=250 chars)." +
closestBlockHint(blockTexts, selection) +
rolled,
);
}
/**
* Create an inline comment anchored to its `selection` text, or a reply.
*
* Top-level comments (no `parentCommentId`) are ALWAYS inline and MUST carry a
* `selection`: the `type` argument is kept for interface compatibility but the
* effective type is coerced to "inline". The selection has to anchor in the
* document; if it cannot, the comment is rolled back and an error is thrown so
* the caller is forced to supply a proper inline selection rather than leaving
* an orphan, unanchored comment behind. Replies (parentCommentId set) inherit
* their parent's anchor: they take NO selection and are not anchored.
*/
async createComment(
pageId: string,
content: string,
type: "page" | "inline" = "page",
selection?: string,
parentCommentId?: string,
suggestedText?: string,
) {
// Fail fast (#436): a provided parent id must be a full UUID before any
// network call. Validate only when truthy — a falsy parentCommentId means
// "top-level comment" (mirrors the isReply computation below), not a reply.
if (parentCommentId) {
assertFullUuid("createComment", "parentCommentId", parentCommentId);
}
await this.ensureAuthenticated();
const isReply = !!parentCommentId;
const hasSuggestion =
suggestedText !== undefined && suggestedText !== null;
// Defense in depth mirroring the server DTO/service: a suggested edit rewrites
// the exact anchored text, so it is only meaningful on a top-level inline
// comment that carries a selection.
if (hasSuggestion) {
if (isReply) {
throw new Error(
"createComment: a suggested edit (suggestedText) cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !selection.trim()) {
throw new Error(
"createComment: a suggested edit (suggestedText) requires a 'selection' to anchor and rewrite.",
);
}
}
// Only top-level comments are inline-anchored, so they are stored as
// "inline". Replies carry no inline selection, so they keep the historical
// general ("page") type — both backward-compatible and semantically correct.
// The `type` argument is kept for interface compatibility; createComment
// normalizes the effective type internally, so callers may pass "inline".
const effectiveType: "page" | "inline" = isReply ? "page" : "inline";
if (!isReply && (!selection || !selection.trim())) {
throw new Error(
"createComment: an inline 'selection' (exact text to anchor on) is required for a top-level comment",
);
}
// For a SUGGESTION, the value we store as the comment's `selection` must be
// the RAW document substring the mark lands on (typographic quotes/dashes,
// nbsp, collapsed whitespace), NOT the agent's ASCII input. The anchor is
// placed via normalization, so when the doc was auto-converted to
// typographic the raw substring differs from the agent input; apply-time
// compares the stored selection to the marked doc text STRICTLY, so storing
// the raw substring is what makes "Apply" succeed instead of a spurious 409.
// Captured in the pre-check below (which already reads the page) and used as
// payload.selection. Ordinary comments keep sending the raw agent selection.
let anchoredSelection: string | null = null;
// Set when the anchor matched only after stripping markdown from the
// selection (the strip fallback); surfaced as a soft warning like
// editPageText does, so a stale-markdown selection is flagged.
let anchorNormalized = false;
// For a top-level comment, fail BEFORE creating anything when the selection
// is not present in the persisted document — this avoids leaving an orphan
// comment + notification behind. A read failure (network) is non-fatal: the
// live anchor step below still enforces the anchoring invariant.
if (!isReply && selection) {
try {
const page = await this.getPageJson(pageId);
if (hasSuggestion) {
// A suggestion's anchor MUST be unambiguous: applying it rewrites the
// exact anchored text, and ordinary anchoring silently takes the first
// occurrence, so 0 matches -> not found and >=2 -> ambiguous, both
// rejected BEFORE creating the comment.
const matches = countAnchorMatches(page.content, selection);
if (matches === 0) {
throw this.anchorNotFoundError(page.content, selection, false);
}
if (matches >= 2) {
throw new Error(
`createComment: the suggestion's selection is ambiguous — it occurs ${matches} times in the page. ` +
"A suggested edit must anchor to a UNIQUE location; expand the selection with surrounding context " +
"(still <=250 chars) so it appears exactly once.",
);
}
// Exactly one match: capture the RAW anchored substring to store as the
// comment selection (so apply-time equality holds). If this returns
// null despite countAnchorMatches===1 (shouldn't happen), fall back to
// the raw agent selection below rather than crash.
anchoredSelection = getAnchoredText(page.content, selection);
anchorNormalized = resolveAnchorSelection(
page.content,
selection,
).normalized;
} else {
const resolved = resolveAnchorSelection(page.content, selection);
if (!resolved.found) {
throw this.anchorNotFoundError(page.content, selection, false);
}
anchorNormalized = resolved.normalized;
}
} catch (e) {
// Rethrow our own "not found"/"ambiguous"/"spans multiple blocks" errors;
// swallow read/network errors so the live anchor step can still try (and
// enforce) anchoring.
if (
e instanceof Error &&
(e.message.startsWith("createComment: could not find the selection") ||
e.message.startsWith(
"createComment: the selection spans multiple blocks",
) ||
e.message.startsWith(
"createComment: the suggestion's selection is ambiguous",
))
) {
throw e;
}
if (process.env.DEBUG) {
console.error(
"Pre-check getPageJson failed; deferring to live anchor step:",
e,
);
}
}
}
// Convert through the full Docmost schema. Deliberately the NON-canonicalizing
// variant: a comment body may carry a footnote definition with no matching
// reference, and canonicalization would drop it (data loss). See
// markdownToProseMirror vs markdownToProseMirrorCanonical.
const jsonContent = await markdownToProseMirror(content);
const payload: Record<string, any> = {
pageId,
content: JSON.stringify(jsonContent),
type: effectiveType,
};
// For a suggestion, store the RAW anchored substring (anchoredSelection) so
// the stored selection === the text under the mark === apply-time
// expectedText. Ordinary comments (and the null fallback) keep the raw
// agent selection — their selection is only display/anchor and never used
// by apply, so their behavior is unchanged.
if (!isReply && selection)
payload.selection = anchoredSelection ?? selection;
if (parentCommentId) payload.parentCommentId = parentCommentId;
// Only a top-level inline comment (with a selection) may carry a suggestion.
if (!isReply && selection && hasSuggestion) {
payload.suggestedText = suggestedText;
}
const response = await this.client.post("/comments/create", payload);
const comment = response.data.data || response.data;
const markdown = comment.content
? convertProseMirrorToMarkdown(this.parseCommentContent(comment.content))
: content;
const result: any = {
data: filterComment(comment, markdown),
success: true,
};
// Replies inherit the parent's anchor: no selection, no anchoring.
if (isReply) {
return result;
}
// Anchor the comment in the document. The /comments/create API records the
// comment + its `selection` text, but it does NOT insert the comment MARK
// into the page content, so without this the inline comment has no
// highlight/anchor and is not clickable. If anchoring fails the comment is
// rolled back (deleted) and an error is thrown — never an orphan comment.
const newCommentId: string = comment.id;
// Guard: a create response without an id would mean writing a comment mark
// with commentId: undefined and a later delete of a falsy id. We have no id
// to roll back here (nothing was created with an id), so just fail loudly.
if (!newCommentId) {
throw new Error(
"createComment: the server returned no comment id, so the comment could not be anchored",
);
}
let anchored = false;
// Set inside the transform when a suggestion's live anchor is ambiguous
// (>=2 occurrences), so the rollback path can surface the right error.
let ambiguousInLiveDoc = false;
// Captured inside the transform on a not-found abort, so the rollback path
// can surface the closest-block / spans-multiple-blocks hint built from the
// LIVE document (the pre-check page is not in scope there).
let liveNotFoundError: Error | null = null;
try {
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// /comments/create REST call above keeps the agent-supplied id.
const pageUuid = await this.resolvePageId(pageId);
// Route through the mutatePage seam (not the free function) so this
// wrapper's uniqueness gate + rollback can be unit-tested without a live
// Hocuspocus collab socket.
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (hasSuggestion) {
// Authoritative uniqueness check against the LIVE document: a
// suggestion must anchor to EXACTLY ONE occurrence, otherwise
// "Apply" would rewrite the wrong/ambiguous text. If the live doc
// no longer has exactly one occurrence (it changed since the
// pre-check), abort so the just-created comment is rolled back
// rather than mis-anchored to the first occurrence.
const liveCount = countAnchorMatches(doc, selection as string);
if (liveCount !== 1) {
ambiguousInLiveDoc = liveCount >= 2;
if (liveCount === 0) {
liveNotFoundError = this.anchorNotFoundError(
doc,
selection as string,
true,
);
}
return null;
}
}
if (applyAnchorInDoc(doc, selection as string, newCommentId)) {
anchored = true;
return doc;
}
// Selection text not found in the LIVE document: abort the write. The
// rollback + throw below turns this into a hard error.
liveNotFoundError = this.anchorNotFoundError(
doc,
selection as string,
true,
);
return null;
},
);
result.verify = mutation.verify;
} catch (e) {
// The comment record already exists; roll it back so we never leave an
// orphan, then rethrow the original anchoring error.
await this.safeDeleteComment(newCommentId);
throw e;
}
if (!anchored) {
// Mutation aborted because the selection was not found (or, for a
// suggestion, was ambiguous) in the live document. Roll back the comment
// and surface a hard error.
await this.safeDeleteComment(newCommentId);
if (ambiguousInLiveDoc) {
throw new Error(
"createComment: the suggestion's selection is ambiguous in the live document (multiple occurrences); the comment was rolled back. Expand the selection with surrounding context so it is unique.",
);
}
throw (
liveNotFoundError ??
new Error(
"createComment: failed to anchor the comment (selection not found in the live document); the comment was rolled back",
)
);
}
// Soft warning (like editPageText): the selection only matched after
// stripping markdown, so the caller likely quoted a styled fragment.
if (anchorNormalized) {
result.warning =
"The selection matched only after stripping markdown syntax; the comment " +
"was anchored on the document's plain text. Copy the selection verbatim " +
"from getPage / searchInPage output to avoid this.";
}
result.anchored = true;
return result;
}
/**
* Best-effort rollback of a just-created comment. Swallows any delete failure
* (logging under DEBUG) so a failed cleanup never masks the original error.
*/
protected async safeDeleteComment(commentId: string): Promise<void> {
// Defense in depth: never call the delete API with a falsy id — there is
// nothing to roll back, and deleteComment(undefined) would hit a bad route.
if (!commentId) return;
try {
await this.deleteComment(commentId);
} catch (delErr) {
if (process.env.DEBUG) {
console.error(
"Failed to roll back comment after anchoring error:",
delErr,
);
}
}
}
async updateComment(commentId: string, content: string) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("updateComment", "commentId", commentId);
await this.ensureAuthenticated();
// NON-canonicalizing on purpose (comment body — see createComment).
const jsonContent = await markdownToProseMirror(content);
await this.client.post("/comments/update", {
commentId,
content: JSON.stringify(jsonContent),
});
return {
success: true,
commentId,
message: "Comment updated successfully.",
};
}
async deleteComment(commentId: string) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("deleteComment", "commentId", commentId);
await this.ensureAuthenticated();
return this.client
.post("/comments/delete", { commentId })
.then((res) => res.data);
}
/**
* Resolve or reopen a top-level comment thread (reversible `resolved`
* toggles the state). Only top-level comments can be resolved; the server
* rejects resolving a reply. Hits POST /comments/resolve.
*/
async resolveComment(commentId: string, resolved: boolean) {
// Fail fast (#436): reject a truncated id before any network call.
assertFullUuid("resolveComment", "commentId", commentId);
await this.ensureAuthenticated();
const response = await this.client.post("/comments/resolve", {
commentId,
resolved,
});
const comment = response.data?.data ?? response.data;
return {
success: true,
commentId,
resolved,
comment,
};
}
/**
* Check for new comments across pages in a space (optionally scoped to a
* subtree): pages updated after `since` are scanned and their comments
* filtered by createdAt > since.
*/
async checkNewComments(
spaceId: string,
since: string,
parentPageId?: string,
) {
await this.ensureAuthenticated();
const sinceDate = new Date(since);
// Reject an unparseable `since`: comparing against an Invalid Date silently
// yields zero new comments (every `>` against NaN is false), which would
// mask a malformed input as "nothing new" instead of erroring.
if (Number.isNaN(sinceDate.getTime())) {
throw new Error(
`checkNewComments: invalid "since" date "${since}"; expected an ISO-8601 timestamp`,
);
}
// 1. Enumerate the FULL set of pages in scope via the page tree (a complete
// page index), NOT the bounded "/pages/recent" feed which caps at ~5000
// recent items and silently misses comments on older pages.
//
// Subtree scope: when parentPageId is given, the scope is that page ITSELF
// plus every descendant. Otherwise the scope is the whole space (all roots
// and their descendants).
//
// NOTE: do NOT pre-filter by page.updatedAt — creating a comment does not
// bump it (verified on a live server), so such a filter silently misses
// comments on pages that were not otherwise edited. The complete tree walk
// already restricts the scope correctly, so no recent-feed allow-list is
// needed any more.
//
// The subtree scope (parentPageId given) already INCLUDES the root node
// itself: /pages/tree seeds getPageAndDescendants with id = parentPageId, so
// no separate getPageRaw fetch for the parent is needed.
const { pages: pagesInScope, truncated } = await this.enumerateSpacePages(
spaceId,
parentPageId,
);
// 2. Fetch comments for each page, keep ones created after since
const results: any[] = [];
for (const page of pagesInScope) {
try {
// Full feed (incl. resolved): a "new comments since" scan reports all
// recent activity; the active-only filter is scoped to listComments.
const comments = (await this.listComments(page.id, true)).items;
const newComments = comments.filter(
(c: any) => new Date(c.createdAt) > sinceDate,
);
if (newComments.length > 0) {
results.push({
pageId: page.id,
pageTitle: page.title,
comments: newComments,
});
}
} catch (e: any) {
// Skip pages with errors (e.g. deleted between calls)
}
}
const totalNewComments = results.reduce(
(sum, r) => sum + r.comments.length,
0,
);
// `truncated` is reported by enumerateSpacePages: it is true ONLY when the
// stdio fallback BFS hit its node cap. The primary /pages/tree path is
// uncapped, so a space with legitimately many pages is not falsely flagged.
return {
since,
scope: parentPageId ? `subtree of ${parentPageId}` : `space ${spaceId}`,
checkedPages: pagesInScope.length,
pagesWithNewComments: results.length,
totalNewComments,
truncated,
comments: results,
};
}
// --- Image upload / embedding ---
/** Map a Content-Type string to a supported MIME type, or null if unsupported. */
}
return CommentsMixin;
}
+690
View File
@@ -0,0 +1,690 @@
// Shared client context + core seams (issue #450). The abstract base of the
// DocmostClient mixin chain: it owns ALL shared instance state (the axios
// client, apiUrl, auth tokens, the resolvePageId cache, the collab-token cache,
// the sandbox/metrics sinks) and the core HTTP/auth/pagination/write seams every
// domain module builds on. Domain modules are mixins layered on top; the final
// DocmostClient (client.ts) assembles them. Extracted VERBATIM from the original
// monolith — only field/seam visibility was widened from `private` to
// `protected` so sibling mixins can reach the shared state through `this`, and
// the cross-module methods that live in other mixins are declared `abstract`
// here so `this.<method>` type-checks. No behaviour changed.
import axios, { AxiosInstance } from "axios";
import FormData from "form-data";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { acquireCollabSession } from "../lib/collab-session.js";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import { getCollabToken, performLogin } from "../lib/auth-utils.js";
import { formatDocmostAxiosError } from "./errors.js";
// A generic mixin base constructor (issue #450). Each domain mixin is a factory
// `<T extends GConstructor<DocmostClientContext>>(Base: T) => class extends Base`
// so the mixins compose into one prototype chain sharing this context.
export type GConstructor<T = {}> = abstract new (...args: any[]) => T;
/**
* Configuration for a DocmostClient / MCP server instance. A discriminated
* union: either service-account credentials (email/password the client calls
* performLogin, powering the external /mcp HTTP endpoint and the stdio CLI) OR
* a token getter (getToken the client uses the returned BARE access JWT as
* the Bearer and never calls performLogin; used for the internal per-user path).
*
* Both branches may ALSO carry an optional `getCollabToken` provider. When set,
* content mutations (which go over the collaboration websocket) use the token it
* returns INSTEAD of calling `POST /auth/collab-token`. The internal per-user
* agent path uses this to hand the client a provenance collab token (signed
* `actor:'agent'`+`aiChatId`), so agent content edits are attributed without a
* spoofable client-side field. When absent the client keeps the original
* `/auth/collab-token` path (service-account/stdio unchanged).
*
* Housed here (not in index.ts) so client.ts has no type dependency on index.ts;
* index.ts re-exports it for the package's public surface.
*/
// Sink the stash tool writes blobs into. The host app binds this to its in-RAM
// SandboxStore and composes the public `uri` (the package never sees the store
// or any env). `put` returns the anonymous read URL plus integrity metadata.
export type SandboxPut = (
buf: Buffer,
mime: string,
) => { uri: string; sha256: string; size: number };
export type DocmostMcpConfig = { apiUrl: string } & (
| { email: string; password: string }
| { getToken: () => Promise<string> } // returns a BARE JWT; the client adds "Bearer "
) & {
// Optional collab-token provider (returns a ready collab JWT). Common to
// both branches; see the type doc above.
getCollabToken?: () => Promise<string>;
// Optional blob sandbox sink. Present only where the stash tool is wired;
// when absent, stashPage throws a clear "not configured" error. The
// optional `has`/`evict` probes let stashPage keep its mirror counts honest
// under the store's FIFO eviction (see stashPage); older sinks omit them.
sandbox?: {
put: SandboxPut;
has?: (uri: string) => boolean;
evict?: (uri: string) => void;
};
// Dependency-neutral metrics sink. When present, the client emits generic
// (name, value, labels) samples; the HOST maps those names onto its own
// metrics registry (the package never depends on prom-client or the server).
// Absent in standalone/stdio mode → the client is a complete no-op here.
onMetric?: (
name: string,
value: number,
labels?: Record<string, string>,
) => void;
};
/**
* Collab-token cache TTL in milliseconds (issue #435). Read fresh from the
* environment on every mint like collab-session.ts readConfig so tests and a
* live rollback can change it without reloading the module.
*
* Why a cache at all: the live CollabSession registry (#400/#431) keys sessions
* on (wsUrl, pageId, collabToken) for identity isolation (invariant 4). But BOTH
* collab-token sources mint a FRESH token per mutation the in-app provider
* re-signs a JWT whose iat/exp (seconds) changes every second, and the external
* MCP POSTs /auth/collab-token each call so the token in the key changed on
* every op and the session was almost never reused (connect-storms, 25s
* timeouts, zombie sessions). Caching the token per-client keeps the key stable
* across a burst of mutations so ONE session is reused.
*
* Default 5 min: well under the 24h collab-token lifetime AND <= the collab
* session max-age (10 min, MCP_COLLAB_SESSION_MAX_AGE_MS), so the
* permission-staleness window is not widened beyond what #431 already accepted.
* The rollback knob is an EXPLICIT 0 (or a negative number): that DISABLES the
* cache an exact fetch-per-call legacy path, mirroring how idleMs<=0 disables
* the session cache. Unset OR unparseable (e.g. a typo like "5min", "abc") falls
* back to the 5-min default with the cache ON parseInt yields NaN, which is
* treated as "not configured", not as "disabled". So to turn the cache off you
* must set the value to exactly 0, not to garbage.
*/
function readCollabTokenTtlMs(): number {
const raw = parseInt(process.env.MCP_COLLAB_TOKEN_TTL_MS ?? "", 10);
return Number.isFinite(raw) ? Math.max(0, raw) : 5 * 60 * 1000;
}
export abstract class DocmostClientContext {
protected client: AxiosInstance;
protected token: string | null = null;
protected apiUrl: string;
// email/password are only set on the service-account (credentials) variant;
// null on the getToken variant (where there are no credentials to log in with).
protected email: string | null = null;
protected password: string | null = null;
// Per-user token provider. When set, login() calls it to obtain a BARE access
// JWT instead of performLogin, and the 401/403 re-auth path re-calls it.
protected getTokenFn: (() => Promise<string>) | null = null;
// Optional collab-token provider. When set, getCollabTokenWithReauth() returns
// its token instead of calling POST /auth/collab-token; on a 401/403 it is
// re-invoked once. Used by the internal agent to carry signed provenance.
protected getCollabTokenFn: (() => Promise<string>) | null = null;
// Optional blob-sandbox sink for the stash tool. Null when not configured.
protected sandboxPut: SandboxPut | null = null;
// Optional probes paired with the sink. `has` lets stashPage detect a blob
// FIFO-evicted by a LATER put in the same stash; `evict` lets it free this
// op's image blobs if the final doc put throws. Null when the sink omits them.
protected sandboxHas: ((uri: string) => boolean) | null = null;
protected sandboxEvict: ((uri: string) => void) | null = null;
// Optional dependency-neutral metrics sink (see DocmostMcpConfig.onMetric).
// Null on the legacy positional form and whenever the host omits it → no-op.
protected onMetricFn:
| ((name: string, value: number, labels?: Record<string, string>) => void)
| null = null;
// In-flight login dedup: when the token expires, the 401 interceptor,
// ensureAuthenticated, getCollabTokenWithReauth and the two multipart retries
// can all call login() at once. Memoizing a single promise collapses that
// thundering herd into ONE /auth/login request that everyone awaits.
protected loginPromise: Promise<void> | null = null;
// Canonical-UUID cache for resolvePageId: maps an agent-supplied slugId to the
// page's canonical UUID, so repeated collab edits on the same page do not
// re-fetch /pages/info. A UUID input short-circuits before this cache (see
// resolvePageId), so only slugId->uuid entries are stored/read here.
protected pageIdCache = new Map<string, string>();
// Collab-token cache (issue #435): the last minted collab token plus the
// wall-clock time it was minted, so a burst of content mutations reuses ONE
// token and therefore ONE live CollabSession (whose registry key includes the
// token — #400 invariant 4). Per-instance: a DocmostClient is built per
// user/per chat request, so a cached token can never leak across identities.
// Reset whenever the client's identity changes (login() / this.token cleared);
// bypassed on a forced refresh (the 401/403 reauth path). null = no token yet.
protected collabTokenCache: { token: string; mintedAt: number } | null = null;
// Two construction forms:
// - new DocmostClient(config) // discriminated union (current)
// - new DocmostClient(baseURL, email, password) // legacy positional creds
// The positional form is retained so existing callers/tests keep working; it
// is exactly equivalent to the credentials branch of the object form.
constructor(config: DocmostMcpConfig);
constructor(baseURL: string, email: string, password: string);
constructor(
configOrBaseURL: DocmostMcpConfig | string,
email?: string,
password?: string,
) {
// Normalize the legacy positional form into the object union.
const config: DocmostMcpConfig =
typeof configOrBaseURL === "string"
? { apiUrl: configOrBaseURL, email: email!, password: password! }
: configOrBaseURL;
this.apiUrl = config.apiUrl;
if ("getToken" in config) {
// Token variant: carry the user's JWT via getToken; no credentials, so
// login() must never call performLogin (there is nothing to log in with).
this.getTokenFn = config.getToken;
} else {
// Service-account variant: behaves exactly as before (performLogin).
this.email = config.email;
this.password = config.password;
}
// Optional, available to both variants. When present, content mutations get
// their collab token from here instead of POST /auth/collab-token.
if (config.getCollabToken) {
this.getCollabTokenFn = config.getCollabToken;
}
if (config.sandbox) {
this.sandboxPut = config.sandbox.put;
this.sandboxHas = config.sandbox.has ?? null;
this.sandboxEvict = config.sandbox.evict ?? null;
}
// Legacy positional form carries no onMetric → null (complete no-op).
this.onMetricFn = config.onMetric ?? null;
this.client = axios.create({
baseURL: this.apiUrl,
// Default request timeout so a hung connection cannot wedge a per-page
// lock or block the server indefinitely. Multipart uploads override this
// with a longer per-request timeout.
timeout: 30000,
headers: {
"Content-Type": "application/json",
},
});
// Re-authenticate transparently on a 401/403 once: the JWT authToken can
// expire while the server is long-running, after which every cached-token
// request would otherwise fail until a manual restart. On such a response,
// clear the stale token, perform a fresh login, and replay the original
// request exactly once (guarded by config._retry to avoid infinite loops;
// the login request itself is never retried).
this.client.interceptors.response.use(
(response) => response,
async (error) => {
const config = error.config;
const status = error.response?.status;
const isAuthError = status === 401 || status === 403;
const isLoginRequest =
typeof config?.url === "string" && config.url.includes("/auth/login");
if (config && isAuthError && !config._retry && !isLoginRequest) {
config._retry = true;
// Drop the stale token + Authorization header before re-login. Also
// clear the collab-token cache (#435): a new identity/login must not
// keep serving a collab token minted under the old one.
this.token = null;
this.collabTokenCache = null;
delete this.client.defaults.headers.common["Authorization"];
try {
await this.login();
} catch (loginError) {
// Re-login failed: surface the original error to the caller.
return Promise.reject(error);
}
// Re-issue the original request with the freshly minted Bearer token.
// Read it from the default header that login() just set, not from
// this.token, to avoid a theoretical "Bearer null" if this.token was
// cleared between login() resolving and this point.
config.headers = config.headers || {};
config.headers["Authorization"] =
this.client.defaults.headers.common["Authorization"];
return this.client.request(config);
}
return Promise.reject(error);
},
);
// Diagnostics interceptor (issue #437). Registered AFTER the re-login
// interceptor so a successful re-login retry (which resolves to a real
// response) is never seen here as an error; only a genuine failure reaches
// this rejection handler. It reformats error.message IN PLACE (see
// formatDocmostAxiosError — kept as a mutation, not a custom Error class, so
// the surrounding axios.isAxiosError / error.response?.status / config._retry
// checks keep working) and re-rejects the SAME error. The _docmostFormatted
// flag makes a re-processed retry-failure a no-op.
this.client.interceptors.response.use(
(response) => response,
(error) => {
formatDocmostAxiosError(error);
return Promise.reject(error);
},
);
}
// --- Cross-module seams (issue #450) -----------------------------------
// A method in one domain mixin sometimes calls a PROTECTED method owned by
// another mixin (e.g. nodes-write -> validateDocUrls in doc-validate). Those
// callees are `protected`, so they cannot be surfaced through the public
// per-mixin interfaces. Declaring them here on the shared base lets `this.<m>`
// type-check across modules. Each is a stub that is ALWAYS overridden by the
// owning mixin (layered above this base in the chain), so the body never runs;
// it throws only to make an impossible mis-wiring loud instead of silent.
// (The PUBLIC cross-module callees — getPage, getPageJson, listComments,
// deleteComment, listPageHistory — arrive via the mixins' public interfaces,
// so they are not restated here.)
protected enumerateSpacePages(
_spaceId: string,
_rootPageId?: string,
): Promise<{ pages: any[]; truncated: boolean }> {
throw new Error("enumerateSpacePages not wired (missing ReadMixin)");
}
protected validateDocUrls(_node: any, _depth?: number): void {
throw new Error("validateDocUrls not wired (missing DocValidateMixin)");
}
protected validateDocStructure(_node: any, _depth?: number): void {
throw new Error("validateDocStructure not wired (missing DocValidateMixin)");
}
protected assertValidNodeShape(_op: string, _node: any): void {
throw new Error("assertValidNodeShape not wired (missing DocValidateMixin)");
}
protected fetchInternalFile(
_src: string,
): Promise<{ buffer: Buffer; mime: string }> {
throw new Error("fetchInternalFile not wired (missing StashMixin)");
}
protected uploadAttachmentBuffer(
_pageId: string,
_buffer: Buffer,
_fileName: string,
_mime: string,
): Promise<{ id: string; fileName: string; fileSize: number }> {
throw new Error("uploadAttachmentBuffer not wired (missing MediaMixin)");
}
protected fetchAttachmentText(_src: string): Promise<string> {
throw new Error("fetchAttachmentText not wired (missing MediaMixin)");
}
// PUBLIC cross-module callees. Declared here too (as always-overridden stubs)
// so a mixin calling e.g. `this.getPageJson` type-checks against the base —
// the mixin's own public interface only covers its own methods. The real
// implementations live in ReadMixin / CommentsMixin / PagesMixin and shadow
// these on the prototype chain.
getPage(_pageId: string): Promise<any> {
throw new Error("getPage not wired (missing ReadMixin)");
}
getPageJson(_pageId: string): Promise<any> {
throw new Error("getPageJson not wired (missing ReadMixin)");
}
listComments(_pageId: string, _includeResolved?: boolean): Promise<any> {
throw new Error("listComments not wired (missing CommentsMixin)");
}
deleteComment(_commentId: string): Promise<any> {
throw new Error("deleteComment not wired (missing CommentsMixin)");
}
listPageHistory(_pageId: string, _cursor?: string): Promise<any> {
throw new Error("listPageHistory not wired (missing PagesMixin)");
}
/** Application base URL (API URL without the /api suffix). */
get appUrl(): string {
return this.apiUrl.replace(/\/api\/?$/, "");
}
async login() {
// Reuse an in-flight login if one is already running so concurrent callers
// share a single token fetch instead of each issuing their own.
if (!this.loginPromise) {
// Token variant: re-fetch a BARE JWT via getToken() (there are no
// credentials to log in with — on a 401/403 the interceptor below calls
// login() again, which re-invokes getToken()). Credentials variant:
// performLogin against /auth/login exactly as before.
const fetchToken = this.getTokenFn
? this.getTokenFn()
: performLogin(this.apiUrl, this.email!, this.password!);
this.loginPromise = fetchToken
.then((token) => {
// Guard against an empty/invalid token (e.g. a getToken provider that
// resolves to "" or null): without this an empty token would set a
// literal "Authorization: Bearer null"/"Bearer " header and every
// request would 401 with a confusing error. Fail loudly instead.
if (typeof token !== "string" || token.length === 0) {
throw new Error("getToken returned an empty token");
}
this.token = token;
// Identity (re)established: drop any collab token minted under a
// previous identity so the #435 cache can never outlive it.
this.collabTokenCache = null;
this.client.defaults.headers.common["Authorization"] =
`Bearer ${token}`;
})
.finally(() => {
this.loginPromise = null;
});
}
return this.loginPromise;
}
async ensureAuthenticated() {
if (!this.token) {
await this.login();
}
}
/**
* Fetch a collaboration token, transparently re-authenticating once on a
* 401/403. getCollabToken() uses bare axios internally, so it is NOT covered
* by this.client's response interceptor; this helper replicates that
* behaviour for collab-token requests: ensure a token, try once, and on an
* expired-token auth error perform a fresh login and retry exactly once.
*
* Collab-token cache (issue #435): both sources the getCollabToken provider
* (in-app agent) AND the REST /auth/collab-token endpoint (external MCP) mint
* a FRESH token per call, whose string therefore changes every op. Since the
* live CollabSession registry keys on the token string (#400/#431 invariant 4),
* that churned the key and defeated session reuse. So we cache the last minted
* token per-client for readCollabTokenTtlMs() and hand it back for a burst of
* mutations, keeping the session key stable. `forceRefresh` bypasses the cache
* (the 401/403 reauth retry uses it, so the retry cannot be handed the same
* stale token that just failed otherwise reauth would be a no-op). TTL 0
* disables the cache: exact fetch-per-call legacy behaviour.
*/
protected async getCollabTokenWithReauth(
forceRefresh = false,
): Promise<string> {
const ttl = readCollabTokenTtlMs();
// Serve the cached collab token while it is still fresh (identity isolation
// is preserved: the cache is a per-instance field on a client built per
// user/per chat request, and it is cleared on every identity change).
if (
!forceRefresh &&
ttl > 0 &&
this.collabTokenCache &&
Date.now() - this.collabTokenCache.mintedAt < ttl
) {
return this.collabTokenCache.token;
}
// Collab-token PROVIDER path: when a getCollabToken provider was supplied
// (the internal agent's provenance collab token), use it instead of the
// REST /auth/collab-token endpoint. Re-invoke it once on a 401/403 (e.g. the
// signed token expired between content mutations in a long agent turn).
if (this.getCollabTokenFn) {
try {
const token = await this.getCollabTokenFn();
if (typeof token !== "string" || token.length === 0) {
throw new Error("getCollabToken returned an empty token");
}
return this.rememberCollabToken(token, ttl);
} catch (e) {
// On an auth error retry EXACTLY once, forcing a refresh so the retry
// re-invokes the provider (bypassing the cache) for a genuinely fresh
// token. `!forceRefresh` bounds it to a single retry (no loop).
if (this.isCollabAuthError(e) && !forceRefresh) {
return this.getCollabTokenWithReauth(true);
}
throw e;
}
}
await this.ensureAuthenticated();
try {
const token = await getCollabToken(this.apiUrl, this.token!);
return this.rememberCollabToken(token, ttl);
} catch (e) {
// getCollabToken wraps the AxiosError in a plain Error but attaches the
// HTTP status as `.status`, so isCollabAuthError detects an auth failure
// via either the raw AxiosError shape OR the attached status.
if (this.isCollabAuthError(e) && !forceRefresh) {
// Fresh login (which clears this.token AND the collab-token cache), then
// retry exactly once with the cache bypassed via forceRefresh.
await this.login();
return this.getCollabTokenWithReauth(true);
}
throw e;
}
}
/**
* Store a freshly minted collab token in the per-client cache (issue #435) and
* return it unchanged. No-op write when the cache is disabled (ttl<=0) or the
* token is empty, so a disabled cache is exact fetch-per-call legacy behaviour
* and a bad token is never cached.
*/
protected rememberCollabToken(token: string, ttl: number): string {
if (ttl > 0 && typeof token === "string" && token.length > 0) {
this.collabTokenCache = { token, mintedAt: Date.now() };
}
return token;
}
/**
* True when an error carries a 401/403 either as a raw AxiosError
* (`error.response.status`) or as the plain-Error `.status` that
* lib/auth-utils.getCollabToken attaches after wrapping the AxiosError.
*/
protected isCollabAuthError(e: unknown): boolean {
const axiosStatus = axios.isAxiosError(e) ? e.response?.status : undefined;
const attachedStatus = (e as any)?.status;
return (
axiosStatus === 401 ||
axiosStatus === 403 ||
attachedStatus === 401 ||
attachedStatus === 403
);
}
/**
* Connect to the collaboration websocket, read the live doc, apply
* `transform`, write the result, and wait for the server to persist it
* WITHOUT acquiring the per-page lock.
*
* This mirrors collaboration.mutatePageContent EXCEPT that it does not call
* withPageLock. It exists solely so replaceImage can hold ONE withPageLock
* across its scan -> upload -> write sequence: the per-page mutex is NOT
* reentrant, so calling the normal (self-locking) mutatePageContent inside an
* outer withPageLock for the same pageId would deadlock. The caller MUST hold
* the page lock for the whole operation; this helper assumes that invariant.
*
* `transform` receives the live ProseMirror doc and returns the NEW full doc
* to write, or `null` to abort with no write. Errors thrown by `transform`
* propagate to the caller.
*
* Resolves a `MutationResult { doc, verify }` mirroring mutatePageContent, so
* every content mutator (including replaceImage) can return a verifiable
* change report. The report is computed AFTER the atomic read->write and
* never throws.
*/
protected async mutateLiveContentUnlocked(
pageId: string,
collabToken: string,
transform: (liveDoc: any) => any | null,
): Promise<MutationResult> {
// Reuse a live CollabSession for the page (issue #400) instead of opening a
// fresh provider per op. acquireCollabSession does NOT take the per-page
// lock — the caller (replaceImage) already holds ONE withPageLock across its
// scan -> upload -> write sequence, and the mutex is not reentrant, so
// taking it here would deadlock. The synchronous read->write section and the
// unsyncedChanges/connectionLost ack logic live in CollabSession.mutate,
// preserved verbatim from the old inline machine (incl. the #152 structural
// diff that keeps a live editor's cursor anchored).
const session = await acquireCollabSession(pageId, collabToken, this.apiUrl, {
// Only the actual 25s collab connect timeout emits this — the connect-vs-
// unload signal; the other failure paths must NOT emit it.
onConnectTimeout: () =>
this.onMetricFn?.("collab_connect_timeouts_total", 1),
});
try {
return await session.mutate(transform);
} catch (e) {
// Drop the session on any failure so the next call reconnects fresh.
session.destroy("mutate failed");
throw e;
}
}
/**
* Generic pagination handler for Docmost API endpoints
*/
async paginateAll<T = any>(
endpoint: string,
basePayload: Record<string, any> = {},
limit: number = 100,
): Promise<T[]> {
await this.ensureAuthenticated();
const clampedLimit = Math.max(1, Math.min(100, limit));
// Hard ceiling on the number of pages to fetch: guards against a server
// that returns a perpetually-true hasNextPage (which would otherwise loop
// forever and accumulate duplicates).
const MAX_PAGES = 50;
let cursor: string | undefined;
let allItems: T[] = [];
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = {
...basePayload,
limit: clampedLimit,
};
if (cursor) payload.cursor = cursor;
const response = await this.client.post(endpoint, payload);
const data = response.data;
const items = data.data?.items || data.items || [];
const meta = data.data?.meta || data.meta;
allItems = allItems.concat(items);
// Advance strictly via the server-issued cursor. A missing nextCursor (or
// hasNextPage false) means we reached the end. A cursor identical to the
// one we just sent means the server did not understand our pagination
// param — stop instead of re-fetching page one forever and duplicating.
const next = meta?.hasNextPage ? meta?.nextCursor : null;
if (!next || next === cursor) {
// If the server still reports more pages but stopped issuing a usable
// cursor at the ceiling, flag the result as truncated below.
if (page === MAX_PAGES - 1 && meta?.hasNextPage) truncated = true;
break;
}
cursor = next;
// Reaching the ceiling with more pages still available means the result
// set is truncated.
if (page === MAX_PAGES - 1) truncated = true;
}
// If the loop stopped because it hit the MAX_PAGES ceiling while the server
// still reported more results, the result set is truncated — warn so the
// caller is not silently handed an incomplete list.
if (truncated) {
console.warn(
`paginateAll: results from "${endpoint}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
return allItems;
}
/** Raw page info including the ProseMirror JSON content and slugId. */
async getPageRaw(pageId: string) {
await this.ensureAuthenticated();
const response = await this.client.post("/pages/info", { pageId });
return response.data?.data ?? response.data;
}
/**
* Resolve an agent-supplied pageId to the page's CANONICAL UUID (`page.id`),
* so every collaboration document the MCP opens is named `page.<uuid>` the
* SAME name the web editor always uses (`page.${page.id}`).
*
* The agent commonly passes a 10-char public slugId (from URLs/listings) as
* the pageId. The web editor opens the collab doc by UUID, but the MCP used to
* pass that slugId straight into the collab doc name (`page.<slugId>`). For one
* DB row that produced TWO independent Yjs documents whose debounced stores
* clobbered each other the agent's edit was silently lost (#260).
*
* A UUID input short-circuits with no network round-trip. A slugId is resolved
* once via getPageRaw and cached (both slugId->uuid and uuid->uuid), so
* repeated edits on the same page add no extra request.
*/
protected async resolvePageId(pageId: string): Promise<string> {
if (isUuid(pageId)) return pageId;
const cached = this.pageIdCache.get(pageId);
if (cached) return cached;
const data = await this.getPageRaw(pageId);
const uuid = data?.id;
if (typeof uuid !== "string" || !uuid) {
throw new Error(
`Could not resolve a canonical page id for "${pageId}"`,
);
}
this.pageIdCache.set(pageId, uuid);
return uuid;
}
/**
* Page-locked write seam over collaboration.mutatePageContent. Production just
* delegates; it exists as an overridable method so the insertFootnote wrapper
* (transform abort-on-not-found + response shaping) can be unit-tested without
* standing up a live Hocuspocus collab socket.
*
* SELF-RESOLVES the pageId to the canonical UUID (issue #449, "resolve-then-
* lock"): every write must lock and key its CollabSession by the UUID, never a
* raw slugId (#260). resolvePageId is cached/idempotent, so a caller that
* already resolved pays no extra round-trip; centralizing it here means a
* caller that reaches this seam with a raw slugId still locks correctly instead
* of silently splitting the mutex key. withPageLock also asserts the key is a
* UUID as a hard backstop.
*/
protected async mutatePage(
pageId: string,
collabToken: string,
apiUrl: string,
transform: (doc: any) => any,
): Promise<{ doc?: any; verify?: any }> {
const pageUuid = await this.resolvePageId(pageId);
return mutatePageContent(pageUuid, collabToken, apiUrl, transform);
}
/**
* Full-document write seam over collaboration.replacePageContent. Production
* just delegates; it exists as an overridable method so the full-doc write
* tools (updatePageJson, copyPageContent) can have their footnote-
* canonicalization binding unit-tested without a live Hocuspocus collab socket.
*
* SELF-RESOLVES the pageId to the canonical UUID (issue #449, "resolve-then-
* lock") for the same reason as mutatePage above the lock/CollabSession key
* is guaranteed canonical here, not left to the caller's discipline.
*/
protected async replacePage(
pageId: string,
doc: any,
collabToken: string,
apiUrl: string,
): Promise<{ doc?: any; verify?: any }> {
const pageUuid = await this.resolvePageId(pageId);
return replacePageContent(pageUuid, doc, collabToken, apiUrl);
}
/**
* Export a page to a single self-contained Docmost-flavoured markdown file:
* meta block + body (with inline comment anchors + diagrams) + comment
* threads. Lossless round-trip target; see importPageMarkdown for the inverse.
*/
}
+248
View File
@@ -0,0 +1,248 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of DocValidateMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IDocValidateMixin` fails to compile on drift.
export interface IDocValidateMixin {
}
export function DocValidateMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IDocValidateMixin> & TBase {
abstract class DocValidateMixin extends Base implements IDocValidateMixin {
/**
* Validate a URL string against a scheme allowlist for a given context.
*
* The markdown link path enforces safe schemes via TipTap, but the raw
* JSON path (updatePageJson) bypasses that so this is the sanitization
* choke point for ProseMirror JSON written directly by the caller.
*
* - "link": reject javascript:, vbscript:, data: (any scheme that can
* execute or smuggle script when the href is clicked).
* - "src": allow only http(s):, mailto:, /api/files paths, or a
* scheme-less relative/absolute path; reject
* javascript:/vbscript:/data:/file:.
*/
protected isSafeUrl(url: unknown, context: "link" | "src"): boolean {
if (typeof url !== "string") return false;
const trimmed = url.trim();
if (trimmed === "") return true; // empty href/src is harmless
// Extract a leading "scheme:" if present. A scheme must start with a
// letter and contain only letters/digits/+/-/. before the colon. Strip
// whitespace and ASCII control chars first so a tab/newline embedded in
// the scheme cannot smuggle a dangerous scheme past the check.
const cleaned = trimmed.replace(/[\s\x00-\x1f]+/g, "");
const schemeMatch = /^([a-zA-Z][a-zA-Z0-9+.-]*):/.exec(cleaned);
const scheme = schemeMatch ? schemeMatch[1].toLowerCase() : null;
const dangerous = new Set(["javascript", "vbscript", "data", "file"]);
if (context === "link") {
if (scheme === null) return true; // relative/anchor link is fine
// For links, data: is also blocked (can carry script payloads).
return !new Set(["javascript", "vbscript", "data"]).has(scheme);
}
// context === "src"
if (scheme === null) return true; // relative/absolute path (incl. /api/files)
if (dangerous.has(scheme)) return false;
return scheme === "http" || scheme === "https" || scheme === "mailto";
}
/**
* Recursively walk a ProseMirror doc and reject any unsafe URL on a link
* mark href or on a media node's src/url. Media nodes covered: image,
* attachment, video, plus embed (rendered as an iframe), youtube, drawio
* and excalidraw all of which carry a user-controlled URL that Docmost
* renders. Throws a clear error on the first violation. A max-depth guard
* turns an over-deep document into a clean error instead of a RangeError
* stack overflow.
*/
protected validateDocUrls(node: any, depth: number = 0): void {
const MAX_DEPTH = 200;
if (depth > MAX_DEPTH) {
throw new Error(
`document nesting exceeds the maximum depth of ${MAX_DEPTH}`,
);
}
if (!node || typeof node !== "object") return;
// Link marks on text nodes: validate the href.
if (Array.isArray(node.marks)) {
for (const mark of node.marks) {
if (mark && mark.type === "link" && mark.attrs) {
if (!this.isSafeUrl(mark.attrs.href, "link")) {
throw new Error(`unsafe link href rejected: "${mark.attrs.href}"`);
}
}
}
}
// Media nodes: validate src/url against the stricter src allowlist.
// embed renders as an iframe (highest risk); youtube/drawio/excalidraw
// likewise carry a user-controlled URL Docmost renders, so they get the
// same scheme check as image/attachment/video.
if (
node.type === "image" ||
node.type === "attachment" ||
node.type === "video" ||
node.type === "embed" ||
node.type === "youtube" ||
node.type === "drawio" ||
node.type === "excalidraw" ||
node.type === "audio" ||
node.type === "pdf"
) {
const attrs = node.attrs || {};
for (const key of ["src", "url"]) {
if (attrs[key] != null && !this.isSafeUrl(attrs[key], "src")) {
throw new Error(
`unsafe ${node.type} ${key} rejected: "${attrs[key]}"`,
);
}
}
}
if (Array.isArray(node.content)) {
for (const child of node.content) {
this.validateDocUrls(child, depth + 1);
}
}
}
/**
* Recursively validate the STRUCTURE of a ProseMirror node (reuses the
* recursion shape of validateDocUrls). Every node must be an object with a
* string `type`; when present, `content` must be an array, `marks` must be
* an array of objects each with a string `type`, and a text node's `text`
* must be a string. Throws a clear "invalid ProseMirror document" error on
* the first violation. A max-depth guard turns an over-deep document into a
* clean error instead of a RangeError stack overflow.
*/
protected validateDocStructure(node: any, depth: number = 0): void {
const MAX_DEPTH = 200;
if (depth > MAX_DEPTH) {
throw new Error(
`invalid ProseMirror document: nesting exceeds the maximum depth of ${MAX_DEPTH}`,
);
}
if (!node || typeof node !== "object" || typeof node.type !== "string") {
throw new Error(
"invalid ProseMirror document: every node must be an object with a string `type`",
);
}
if (
"text" in node &&
node.type === "text" &&
typeof node.text !== "string"
) {
throw new Error(
"invalid ProseMirror document: a text node must have a string `text`",
);
}
if (node.marks !== undefined) {
if (!Array.isArray(node.marks)) {
throw new Error(
"invalid ProseMirror document: `marks` must be an array",
);
}
for (const mark of node.marks) {
if (
!mark ||
typeof mark !== "object" ||
typeof mark.type !== "string"
) {
throw new Error(
"invalid ProseMirror document: every mark must be an object with a string `type`",
);
}
}
}
if (node.content !== undefined) {
if (!Array.isArray(node.content)) {
throw new Error(
"invalid ProseMirror document: `content` must be an array when present",
);
}
for (const child of node.content) {
this.validateDocStructure(child, depth + 1);
}
}
}
/**
* Pre-write SHAPE gate (#409). Walk the WHOLE node tree with the shared
* `findInvalidNode` and throw a rich, path-anchored error the instant a nested
* node has an absent/unknown `type` (or an unknown mark) the exact shape that
* otherwise surfaces DEEP in the Yjs encode as the cryptic
* `Unknown node type: undefined`, but only AFTER a collab session was opened
* and a page lock taken. Calling this BEFORE `getCollabTokenWithReauth` /
* `mutatePageContent` fails fast: no collab connection, no lock, deterministic
* message. `op` names the tool for the message prefix (e.g. "patchNode").
*
* `findInvalidNode` derives its "known type" set from the very same
* `docmostExtensions` the encode path uses, so a node this gate accepts is one
* the encoder will accept too.
*/
protected assertValidNodeShape(op: string, node: any): void {
const bad = findInvalidNode(node);
if (bad) {
throw new Error(`${op}: invalid node — ${bad.summary}`);
}
}
/**
* Replace page content with a raw ProseMirror JSON document (lossless) and/or
* update its title. Both `doc` and `title` are optional, but at least one must
* be supplied:
* - `doc` provided -> validate + full-overwrite the body (and update the
* title too when `title` is also given).
* - `doc` omitted, `title` given -> title-only update; the body is NOT
* touched/resent (no collab write happens).
* - neither given -> throws (nothing to update).
*/
}
return DocValidateMixin;
}
+707
View File
@@ -0,0 +1,707 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import { parseCells as parseDrawioCells } from "../lib/drawio-xml.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
prepareModel,
decodeDrawioSvg,
buildDrawioSvg,
mxHash,
normalizeXml,
countUserCells,
} from "../lib/drawio-xml.js";
import { renderDiagramShapes } from "../lib/drawio-preview.js";
import { applyElkLayout } from "../lib/drawio-layout.js";
import {
buildFromGraph,
type Graph,
type LayoutMode as GraphLayoutMode,
} from "../lib/drawio-graph.js";
import { applyCellOps, type CellOp } from "../lib/drawio-cell-ops.js";
import { mermaidToGraph } from "../lib/drawio-mermaid.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of DrawioMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IDrawioMixin` fails to compile on drift.
export interface IDrawioMixin {
drawioGet(pageId: string, node: string, format?: "xml" | "svg"): Promise<{ pageId: string; nodeId: string; format: "xml" | "svg"; content: string; meta: { attachmentId: string | null; title: string | null; width: number | null; height: number | null; cellCount: number; hash: string; }; }>;
drawioCreate(pageId: string, where: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }, xml: string, title?: string, layout?: "elk"): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; verify?: any; }>;
drawioUpdate(pageId: string, node: string, xml: string, baseHash: string, layout?: "elk"): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; verify?: any; }>;
drawioEditCells(pageId: string, node: string, operations: CellOp[], baseHash: string): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; verify?: any; }>;
drawioFromGraph(pageId: string, where: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }, graph: Graph, direction?: "LR" | "RL" | "TB" | "BT", preset?: string, layout?: GraphLayoutMode, node?: string): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; iconsResolved: number; iconsMissing: string[]; verify?: any; }>;
drawioFromMermaid(pageId: string, where: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }, mermaid: string, preset?: string): Promise<{ success: boolean; nodeId: string; attachmentId: string; warnings: string[]; iconsResolved: number; iconsMissing: string[]; verify?: any; }>;
}
export function DrawioMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IDrawioMixin> & TBase {
abstract class DrawioMixin extends Base implements IDrawioMixin {
/**
* Resolve a drawio node on a page by `attrs.id` or `#<index>` and return the
* node plus its ref. Throws a clear error if the ref does not resolve to a
* drawio node.
*/
protected async resolveDrawioNode(
pageId: string,
node: string,
): Promise<{ node: any; ref: string }> {
const data = await this.getPageRaw(pageId);
const hit = getNodeByRef(
data.content ?? { type: "doc", content: [] },
node,
);
if (!hit) {
throw new Error(
`drawio: no node found for "${node}" on page ${pageId} (use the drawio node's attrs.id or "#<index>" from getOutline)`,
);
}
if (hit.type !== "drawio") {
throw new Error(
`drawio: node "${node}" on page ${pageId} is a ${hit.type}, not a drawio diagram`,
);
}
return { node: hit.node, ref: node };
}
/**
* Read a drawio diagram as mxGraph XML (default) or as the raw `.drawio.svg`.
* Runs the decode chain (base64/entity content= drawio file nested XML or
* pako-inflated compressed <diagram>). The returned `hash` is the
* optimistic-lock key for drawioUpdate.
*/
async drawioGet(
pageId: string,
node: string,
format: "xml" | "svg" = "xml",
): Promise<{
pageId: string;
nodeId: string;
format: "xml" | "svg";
content: string;
meta: {
attachmentId: string | null;
title: string | null;
width: number | null;
height: number | null;
cellCount: number;
hash: string;
};
}> {
await this.ensureAuthenticated();
const { node: drawio } = await this.resolveDrawioNode(pageId, node);
const attrs = drawio.attrs || {};
const src = attrs.src;
if (!src) {
throw new Error(
`drawio: node "${node}" on page ${pageId} has no src to read`,
);
}
const svg = await this.fetchAttachmentText(src);
const modelXml = decodeDrawioSvg(svg);
const meta = {
attachmentId: attrs.attachmentId ?? null,
title: attrs.title ?? null,
width: attrs.width != null ? Number(attrs.width) : null,
height: attrs.height != null ? Number(attrs.height) : null,
cellCount: countUserCells(modelXml),
hash: mxHash(modelXml),
};
return {
pageId,
nodeId: attrs.id ?? node,
format,
content: format === "svg" ? svg : normalizeXml(modelXml),
meta,
};
}
/**
* Create a drawio diagram from mxGraph XML: lint schematic SVG preview
* (pure TS) build the `.drawio.svg` (createDrawioSvg contract) create the
* attachment insert a `drawio` node before/after an anchor or appended.
* `xml` is a bare `<mxGraphModel>` or a list of `<mxCell>` (the server wraps
* it and adds the id=0/id=1 sentinels).
*/
async drawioCreate(
pageId: string,
where: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
xml: string,
title?: string,
layout?: "elk",
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (
!where ||
(where.position !== "before" &&
where.position !== "after" &&
where.position !== "append")
) {
throw new Error(
'drawioCreate: `where.position` must be one of "before", "after", "append"',
);
}
if (where.position === "before" || where.position === "after") {
const hasId =
typeof where.anchorNodeId === "string" && where.anchorNodeId.length > 0;
const hasText =
typeof where.anchorText === "string" && where.anchorText.length > 0;
if (hasId === hasText) {
throw new Error(
`drawioCreate: position "${where.position}" requires exactly one of anchorNodeId or anchorText`,
);
}
}
// Optional server-side ELK auto-layout: the model declares structure with
// rough coords, ELK computes the pixels (best-effort — returns the input
// unchanged on any layout failure).
const laidOutXml = layout === "elk" ? await applyElkLayout(xml) : xml;
// Pre-write pipeline (throws a structured DrawioLintError on any violation).
const prepared = prepareModel(laidOutXml);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
// NOTE: no `id` attribute is set here. The vendored `drawio` node schema
// (diagramAttributes) declares no `id`, so any block id would be silently
// dropped by PMNode.fromJSON on save and the returned handle would fail to
// resolve. The addressable handle is the node's "#<index>" (like image/table
// nodes), computed after the insert below.
const drawioNode: any = {
type: "drawio",
attrs: {
src: `/api/files/${att.id}/${att.fileName}`,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
align: "center",
},
};
if (title) drawioNode.attrs.title = title;
// Reuse the existing URL trust boundary (rejects unsafe src schemes).
this.validateDocUrls(drawioNode);
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let inserted = false;
let insertedIndex = -1;
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
inserted = false;
insertedIndex = -1;
const { doc: nd, inserted: ins } = insertNodeRelative(
liveDoc,
drawioNode,
where,
);
inserted = ins;
if (!inserted) return null; // anchor not found -> skip the write
// Locate the freshly-inserted node to derive its "#<index>" handle. The
// just-uploaded attachmentId is unique, so it identifies our node.
if (Array.isArray(nd.content)) {
insertedIndex = nd.content.findIndex(
(b: any) =>
b &&
b.type === "drawio" &&
b.attrs &&
b.attrs.attachmentId === att.id,
);
}
return nd;
},
);
if (!inserted) {
const anchorDesc = where.anchorNodeId
? `anchorNodeId "${where.anchorNodeId}"`
: `anchorText "${where.anchorText}"`;
throw new Error(
`drawioCreate: anchor not found (${anchorDesc}) on page ${pageId}. The diagram attachment ${att.id} is now an unreferenced orphan.`,
);
}
if (insertedIndex < 0) {
// The node was inserted nested (e.g. inside a callout/table cell via an
// anchor), where "#<index>" — which addresses only top-level blocks —
// cannot reference it. drawio nodes carry no persisted id, so there is no
// stable handle for a nested diagram.
throw new Error(
`drawioCreate: the diagram was inserted on page ${pageId} but not as a ` +
`top-level block, so it has no addressable "#<index>" handle. Anchor ` +
`on a top-level block (or append) so the diagram can be re-read.`,
);
}
// The returned handle is POSITIONAL ("#<index>"): valid for the immediate
// create -> get/update flow, but re-resolve via getOutline if the document
// structure changes (blocks added/removed before it shift the index).
const nodeId = `#${insertedIndex}`;
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
/**
* Full-replacement update of a drawio diagram. `baseHash` is MANDATORY: it is
* compared against the hash of the diagram's CURRENT XML (from drawioGet);
* any mismatch means a human or another agent edited the diagram after the
* read, so the write is refused with a conflict error. On success the new
* `.drawio.svg` is uploaded as a FRESH attachment (in-place byte overwrite is
* avoided some Docmost versions corrupt an attachment on overwrite, exactly
* as replaceImage documents) and the node is repointed with new dimensions.
*/
async drawioUpdate(
pageId: string,
node: string,
xml: string,
baseHash: string,
layout?: "elk",
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (typeof baseHash !== "string" || baseHash.length === 0) {
throw new Error(
"drawioUpdate: baseHash is mandatory — read the diagram with drawioGet first and pass back its meta.hash",
);
}
// Resolve the node and read the CURRENT diagram to enforce the optimistic
// lock before doing any write or upload.
const { node: drawio, ref } = await this.resolveDrawioNode(pageId, node);
const oldAttrs = drawio.attrs || {};
const oldSrc = oldAttrs.src;
// The returned handle is the caller-supplied reference. drawio nodes carry
// no persisted id, so `ref` (an "#<index>" or a rare legacy attrs.id) is the
// honest identifier to hand back.
const nodeId = oldAttrs.id ?? ref;
if (!oldSrc) {
throw new Error(
`drawioUpdate: node "${node}" on page ${pageId} has no src to compare against`,
);
}
const currentSvg = await this.fetchAttachmentText(oldSrc);
const currentHash = mxHash(decodeDrawioSvg(currentSvg));
if (currentHash !== baseHash) {
throw new Error(
`drawioUpdate: conflict — the diagram changed since it was read ` +
`(baseHash ${baseHash} != current ${currentHash}). Re-read it with drawioGet and retry.`,
);
}
// Optional server-side ELK auto-layout (best-effort; see drawioCreate).
const laidOutXml = layout === "elk" ? await applyElkLayout(xml) : xml;
// Pipeline for the new content (throws a structured DrawioLintError).
const prepared = prepareModel(laidOutXml);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = oldAttrs.title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
const newSrc = `/api/files/${att.id}/${att.fileName}`;
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let repointed = 0;
const repoint = (n: any) => {
n.attrs = {
...n.attrs,
src: newSrc,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
};
repointed++;
};
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
repointed = 0;
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
// Repoint ONLY the resolved node — never every node that happens to
// share this attachmentId (a copied diagram is two nodes with one
// attachmentId; keying on it would clobber both). Re-resolve the same
// handle against the live doc and walk to its exact position.
const hit = getNodeByRef(doc, ref);
if (!hit || hit.type !== "drawio") return null; // vanished/changed -> skip
let target: any = doc;
for (const idx of hit.path) {
if (!target || !Array.isArray(target.content)) {
target = null;
break;
}
target = target.content[idx];
}
if (!target || target.type !== "drawio") return null;
repoint(target);
if (repointed === 0) return null; // node vanished concurrently -> skip
return doc;
},
);
if (repointed === 0) {
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: [
...prepared.warnings,
"target drawio node was removed concurrently; uploaded attachment is unreferenced",
],
verify: mutation.verify,
};
}
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
// --- draw.io high-level semantic tools (issue #425) ---
/**
* ID-based targeted edits of an existing drawio diagram (add / update / delete
* cells) instead of resending the whole XML. Reads the CURRENT diagram, checks
* the optimistic lock (`baseHash` is MANDATORY, exactly as drawioUpdate), applies
* the operations to the parsed model (a `delete` CASCADES to container children
* and to every edge whose source/target is deleted), then runs the SAME #423
* pipeline as drawioUpdate (lint + quality warnings -> preview -> attachment ->
* repoint the node). Ids are stable so diffs stay meaningful across edits.
*/
// --- draw.io high-level semantic tools (issue #425) ---
/**
* ID-based targeted edits of an existing drawio diagram (add / update / delete
* cells) instead of resending the whole XML. Reads the CURRENT diagram, checks
* the optimistic lock (`baseHash` is MANDATORY, exactly as drawioUpdate), applies
* the operations to the parsed model (a `delete` CASCADES to container children
* and to every edge whose source/target is deleted), then runs the SAME #423
* pipeline as drawioUpdate (lint + quality warnings -> preview -> attachment ->
* repoint the node). Ids are stable so diffs stay meaningful across edits.
*/
async drawioEditCells(
pageId: string,
node: string,
operations: CellOp[],
baseHash: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
if (typeof baseHash !== "string" || baseHash.length === 0) {
throw new Error(
"drawioEditCells: baseHash is mandatory — read the diagram with drawioGet first and pass back its meta.hash",
);
}
if (!Array.isArray(operations) || operations.length === 0) {
throw new Error(
"drawioEditCells: operations must be a non-empty array of { op, ... }",
);
}
const { node: drawio, ref } = await this.resolveDrawioNode(pageId, node);
const oldAttrs = drawio.attrs || {};
const oldSrc = oldAttrs.src;
const nodeId = oldAttrs.id ?? ref;
if (!oldSrc) {
throw new Error(
`drawioEditCells: node "${node}" on page ${pageId} has no src to edit`,
);
}
const currentSvg = await this.fetchAttachmentText(oldSrc);
const currentModel = decodeDrawioSvg(currentSvg);
const currentHash = mxHash(currentModel);
if (currentHash !== baseHash) {
throw new Error(
`drawioEditCells: conflict — the diagram changed since it was read ` +
`(baseHash ${baseHash} != current ${currentHash}). Re-read it with drawioGet and retry.`,
);
}
// Apply the operations to the parsed model, then run the standard pipeline.
const editedModel = applyCellOps(currentModel, operations);
const prepared = prepareModel(editedModel);
const inner = renderDiagramShapes(prepared.cells, prepared.bbox);
const diagramTitle = oldAttrs.title || "Page-1";
const svg = buildDrawioSvg(prepared.modelXml, inner, prepared.bbox, diagramTitle);
const att = await this.uploadAttachmentBuffer(
pageId,
Buffer.from(svg, "utf-8"),
"diagram.drawio.svg",
"image/svg+xml",
);
const newSrc = `/api/files/${att.id}/${att.fileName}`;
const collabToken = await this.getCollabTokenWithReauth();
const pageUuid = await this.resolvePageId(pageId);
let repointed = 0;
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
repointed = 0;
const doc =
liveDoc && liveDoc.type === "doc" ? liveDoc : { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
const hit = getNodeByRef(doc, ref);
if (!hit || hit.type !== "drawio") return null;
let target: any = doc;
for (const idx of hit.path) {
if (!target || !Array.isArray(target.content)) {
target = null;
break;
}
target = target.content[idx];
}
if (!target || target.type !== "drawio") return null;
target.attrs = {
...target.attrs,
src: newSrc,
attachmentId: att.id,
width: prepared.bbox.width,
height: prepared.bbox.height,
};
repointed++;
return doc;
},
);
if (repointed === 0) {
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: [
...prepared.warnings,
"target drawio node was removed concurrently; uploaded attachment is unreferenced",
],
verify: mutation.verify,
};
}
return {
success: true,
nodeId,
attachmentId: att.id,
warnings: prepared.warnings,
verify: mutation.verify,
};
}
/**
* The main high-level tool: build a diagram from a SEMANTIC graph (nodes with
* a `kind`/`icon`, groups, edges) the model never supplies coordinates or
* style strings. The server resolves icons via the shape catalog (#424),
* assigns palette colors from the preset, runs ELK layered layout (honouring
* `direction` and the `layer`/`sameLayerAs`/`pinned` hints and compound groups),
* and assembles linter-clean XML, then inserts it through the SAME create
* pipeline as drawioCreate. `layout:"incremental"` is only meaningful when a
* target `node` is given (it preserves that diagram's existing coordinates and
* places only new cells); on a fresh insert it behaves like "full".
*/
async drawioFromGraph(
pageId: string,
where: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
graph: Graph,
direction?: "LR" | "RL" | "TB" | "BT",
preset?: string,
layout?: GraphLayoutMode,
node?: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
iconsResolved: number;
iconsMissing: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
// Direction/preset supplied as separate params override the graph fields so
// both the flat tool schema and an inline graph can set them.
const merged: Graph = {
...graph,
direction: direction ?? graph.direction,
preset: preset ?? graph.preset,
};
const mode: GraphLayoutMode = layout ?? "full";
// Incremental into an EXISTING node: read its coords so ELK preserves them,
// and keep the full existing model so incremental MERGES (never drops) any
// cell the new graph doesn't re-list.
let existingCoords: Map<string, { x: number; y: number }> | undefined;
let existingModelXml: string | undefined;
let editExisting = false;
let baseHash: string | undefined;
if (node && (mode === "incremental" || mode === "none")) {
const { node: drawio } = await this.resolveDrawioNode(pageId, node);
const src = (drawio.attrs || {}).src;
if (src) {
const svg = await this.fetchAttachmentText(src);
const model = decodeDrawioSvg(svg);
baseHash = mxHash(model);
existingModelXml = model;
existingCoords = new Map();
for (const c of parseDrawioCells(model)) {
if (c.vertex && c.geometry.x != null && c.geometry.y != null) {
existingCoords.set(c.id, { x: c.geometry.x, y: c.geometry.y });
}
}
editExisting = true;
}
}
const built = await buildFromGraph(
merged,
mode,
existingCoords,
existingModelXml,
);
if (editExisting && node && baseHash) {
// Re-target the existing diagram: replace it with the assembled model.
const res = await this.drawioUpdate(pageId, node, built.modelXml, baseHash);
return {
...res,
iconsResolved: built.iconsResolved,
iconsMissing: built.iconsMissing,
};
}
const res = await this.drawioCreate(pageId, where, built.modelXml);
return {
...res,
iconsResolved: built.iconsResolved,
iconsMissing: built.iconsMissing,
};
}
/**
* Convert a Mermaid `flowchart` to a redactable draw.io diagram via a PURE
* parser (no Electron / draw.io CLI): mermaid text -> graph-JSON -> the
* drawioFromGraph pipeline. Only `flowchart`/`graph` is supported (the most
* common wiki case); other diagram types throw a clear error so the model can
* fall back to drawioFromGraph.
*/
async drawioFromMermaid(
pageId: string,
where: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
mermaid: string,
preset?: string,
): Promise<{
success: boolean;
nodeId: string;
attachmentId: string;
warnings: string[];
iconsResolved: number;
iconsMissing: string[];
verify?: any;
}> {
await this.ensureAuthenticated();
const graph = mermaidToGraph(mermaid);
if (preset) graph.preset = preset;
return this.drawioFromGraph(pageId, where, graph, graph.direction, graph.preset);
}
// --- Page history / diff / transform ---
/**
* List the saved versions (history snapshots) of a page, newest first.
* Docmost auto-snapshots on every save. Returns one cursor-paginated page of
* results: `{ items, nextCursor }`. The history record's id field is `id`.
*/
}
return DrawioMixin;
}
+165
View File
@@ -0,0 +1,165 @@
// Central REST error diagnostics (issues #437 + #450). SINGLE place that maps an
// axios error to the model-facing message. Extracted verbatim from client.ts;
// the constructor's response interceptor (see client/context.ts) routes every
// REST call through formatDocmostAxiosError so the whole surface is uniform.
import axios from "axios";
// --- Issue #437: central error diagnostics -------------------------------
// The agent only ever sees the thrown exception's `error.message`, so a failed
// tool must return an ACTIONABLE message (method, path, status, and the
// server's own validation text) instead of the opaque "Request failed with
// status code 400". These helpers + the response interceptor in the
// constructor are the single authoritative place that text is composed.
// Overall cap on the composed diagnostic message so the model context stays
// compact and a (whitelisted) server string can never blow up the text.
const ERROR_MESSAGE_CAP = 300;
// Only attempt to JSON.parse an arraybuffer body under this size: a larger
// binary body is never a JSON error envelope, so parsing it just wastes memory
// (fetchInternalFile uses responseType:"arraybuffer", so a failed file fetch
// carries the JSON error envelope as raw bytes here).
const ERROR_BUFFER_PARSE_CAP = 4096;
// Canonical 36-char UUID (8-4-4-4-12 hex). Deliberately version/variant-
// AGNOSTIC: the ids are UUIDv7 (e.g. 019f499a-9f8c-7d68-...), so only the
// canonical shape/length is enforced, not the version/variant nibble.
const FULL_UUID_RE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
/**
* Throw an actionable error BEFORE any network call when `value` is not a full
* canonical UUID. Absorbs #436: a truncated/short comment id used to reach the
* server and bounce back as an opaque 400/404 the agent could not self-correct;
* failing fast here names the exact fix.
*/
export function assertFullUuid(
tool: string,
param: string,
value: string,
): void {
if (typeof value !== "string" || !FULL_UUID_RE.test(value)) {
throw new Error(
`${tool}: '${param}' must be the FULL comment UUID (36 chars, e.g. ` +
`019f499a-9f8c-7d68-b7be-ce100d7c6c56), got '${value}'. Copy the id ` +
`verbatim from listComments / createComment output.`,
);
}
}
// Keep ONLY the pathname of a request (no host, no query string, no fragment)
// so the message never leaks a host or query params. Resolves a relative
// config.url against config.baseURL, then discards everything but the path.
function requestPath(config: any): string {
const rawUrl = typeof config?.url === "string" ? config.url : "";
const base =
typeof config?.baseURL === "string" ? config.baseURL : undefined;
try {
// A dummy base makes an absolute config.url parse too; its host is dropped.
return new URL(rawUrl, base ?? "http://localhost").pathname;
} catch {
// Malformed url: still strip any query/fragment manually.
return rawUrl.split(/[?#]/)[0] || rawUrl;
}
}
/**
* Compose the server-facing message from `error.response.data`, using ONLY the
* whitelisted `message`/`error` fields or the HTTP statusText. SECURITY: the
* raw response body, headers (Authorization!) and config are NEVER read here
* a string/HTML body (e.g. a proxy's 502 page) is deliberately dropped in
* favour of the statusText.
*/
function extractServerMessage(data: any, statusText: string): string {
// class-validator envelope: { message: string | string[], error?: string }.
if (
data &&
typeof data === "object" &&
!Buffer.isBuffer(data) &&
!(data instanceof ArrayBuffer)
) {
const msg = (data as any).message;
if (Array.isArray(msg)) {
const joined = msg.filter((m) => typeof m === "string").join("; ");
if (joined) return joined;
} else if (typeof msg === "string" && msg) {
return msg;
}
const err = (data as any).error;
if (typeof err === "string" && err) return err;
return statusText;
}
// Buffer / ArrayBuffer body: attempt a size-capped, guarded JSON.parse so a
// failed arraybuffer fetch still surfaces the server's validation text.
if (Buffer.isBuffer(data) || data instanceof ArrayBuffer) {
const buf = Buffer.isBuffer(data) ? data : Buffer.from(data);
if (buf.length > 0 && buf.length <= ERROR_BUFFER_PARSE_CAP) {
try {
return extractServerMessage(JSON.parse(buf.toString("utf8")), statusText);
} catch {
return statusText;
}
}
return statusText;
}
// A raw string / HTML body is never surfaced (may echo server internals).
return statusText;
}
/**
* Reformat an AxiosError's `.message` IN PLACE into an actionable diagnostic:
* `<METHOD> <path> failed (<status> <statusText>): <serverMessage>`
* or, when the request never got a response:
* `<METHOD> <path> failed: <code> (no response from server)`.
*
* Mutates the SAME error object (never a custom subclass) so the live
* axios.isAxiosError / error.response?.status / config._retry checks around the
* client keep working, and sets `_docmostFormatted` as a double-processing
* guard. A no-op on a non-axios or already-formatted error.
*/
export function formatDocmostAxiosError(error: any): void {
if (!error || error._docmostFormatted) return;
if (!axios.isAxiosError(error)) return;
const config: any = error.config ?? {};
const method =
typeof config.method === "string" ? config.method.toUpperCase() : "";
const methodPath = `${method} ${requestPath(config)}`.trim();
const response = error.response;
let message: string;
if (response) {
const statusText =
typeof response.statusText === "string" ? response.statusText : "";
const serverMessage = extractServerMessage(response.data, statusText);
message = `${methodPath} failed (${response.status} ${statusText}): ${serverMessage}`;
// Full body only to stderr under DEBUG (parity with downloadImage).
if (process.env.DEBUG) {
console.error(
"Docmost request failed; response body:",
JSON.stringify(response.data),
);
}
} else {
// No response at all (ECONNREFUSED / ETIMEDOUT / ECONNRESET / DNS / timeout).
// Use ONLY error.code, never the raw error.message: axios network messages
// embed host:port ("connect ECONNREFUSED 127.0.0.1:3000", "getaddrinfo
// ENOTFOUND host") and #437's invariant is that the host never reaches the
// model-visible message. code is set for essentially every real no-response
// error (ECONNREFUSED/ETIMEDOUT/ECONNRESET/ENOTFOUND/ECONNABORTED); the full
// native message still goes to stderr under DEBUG.
const reason = error.code ?? "network error";
message = `${methodPath} failed: ${reason} (no response from server)`;
if (process.env.DEBUG) {
console.error("Docmost request failed; no response:", error.message);
}
}
if (message.length > ERROR_MESSAGE_CAP) {
message = message.slice(0, ERROR_MESSAGE_CAP - 1) + "…";
}
error.message = message;
(error as any)._docmostFormatted = true;
}
+730
View File
@@ -0,0 +1,730 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import FormData from "form-data";
import axios, { AxiosInstance } from "axios";
import { basename, extname } from "path";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import { diffDocs, summarizeChange } from "../lib/diff.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Supported image types, kept as two lookup tables so both a local file
// extension and a remote Content-Type can be mapped to the same canonical set.
const EXT_TO_MIME: Record<string, string> = {
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".gif": "image/gif",
".webp": "image/webp",
".svg": "image/svg+xml",
};
const MIME_TO_EXT: Record<string, string> = {
"image/png": ".png",
"image/jpeg": ".jpg",
"image/gif": ".gif",
"image/webp": ".webp",
"image/svg+xml": ".svg",
};
// Public method surface of MediaMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IMediaMixin` fails to compile on drift.
export interface IMediaMixin {
uploadImage(pageId: string, url: string): any;
insertImage(pageId: string, url: string, opts?: { align?: "left" | "center" | "right"; alt?: string; replaceText?: string; afterText?: string; }): any;
replaceImage(pageId: string, oldAttachmentId: string, url: string, opts?: { align?: "left" | "center" | "right"; alt?: string }): any;
}
export function MediaMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IMediaMixin> & TBase {
abstract class MediaMixin extends Base implements IMediaMixin {
// --- Image upload / embedding ---
/** Map a Content-Type string to a supported MIME type, or null if unsupported. */
protected supportedImageMime(ct: string): string | null {
return MIME_TO_EXT[ct] ? ct : null;
}
/**
* Download a remote image from a caller-supplied URL and resolve its bytes,
* MIME and a filename.
*
* SSRF / RESOURCE TRUST BOUNDARY: the URL comes from the MCP caller and is
* fetched BY THE SERVER, so it must be guarded before and after the request.
* The guards mirror the local-file trust boundary in uploadImage:
* - scheme allowlist (http/https only) rejects file:, data:, ftp:, etc.,
* so the caller cannot use this path to read local files or other schemes;
* - a size cap enforced both via axios maxContentLength/maxBodyLength AND a
* post-download buffer.length re-check (defends against a missing/lying
* Content-Length), so a huge response cannot exhaust memory;
* - a 30s timeout. The timeout matters because replaceImage holds the
* per-page lock across this upload, so a hung download would wedge the
* lock for that page.
* We deliberately do NOT block private IP ranges: the MCP caller is already
* trusted to read arbitrary host files via the filePath path, so the marginal
* trust granted by fetching internal URLs is comparable, and blocking would
* break legitimate internal-image use.
*/
protected async fetchRemoteImage(
url: string,
maxBytes: number,
): Promise<{ buffer: Buffer; mime: string; fileName: string }> {
// Scheme allowlist first — cheapest guard, and rejects non-http(s) schemes
// (file:, data:, ftp:, ...) before any network request is made.
let parsed: URL;
try {
parsed = new URL(url);
} catch (e: any) {
throw new Error(`Invalid image URL "${url}": ${e.message}`);
}
if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
throw new Error(
`unsupported image URL scheme "${parsed.protocol}"; only http and https are allowed`,
);
}
let response;
try {
response = await axios.get(url, {
responseType: "arraybuffer",
timeout: 30000,
maxContentLength: maxBytes,
maxBodyLength: maxBytes,
headers: { Accept: "image/*" },
});
} catch (error) {
// Keep the thrown message free of the raw response body (it may echo
// server internals); surface only status/statusText. The full body is
// logged under DEBUG for diagnostics.
if (axios.isAxiosError(error)) {
if (process.env.DEBUG) {
console.error(
"Image download failed; response body:",
JSON.stringify(error.response?.data),
);
}
throw new Error(
`Image download failed for "${url}": ${error.response?.status ?? ""} ${error.response?.statusText ?? error.message}`.trim(),
);
}
throw error;
}
// axios returns an ArrayBuffer for responseType: "arraybuffer".
const buffer = Buffer.from(response.data);
// Re-check the size: maxContentLength relies on Content-Length, which may be
// absent or lie, so guard against the actual byte count too.
if (buffer.length === 0) {
throw new Error(`Empty image response from "${url}"`);
}
if (buffer.length > maxBytes) {
throw new Error(
`Image too large: ${buffer.length} bytes exceeds the ${maxBytes}-byte cap`,
);
}
// Resolve MIME: prefer the response Content-Type (strip any "; charset=..."
// parameter, lowercase, trim) mapped through the supported set; if the
// header is generic/missing/unsupported, fall back to the URL path
// extension via the existing extension->MIME logic.
const rawCt = response.headers?.["content-type"];
let mime: string | null = null;
if (typeof rawCt === "string" && rawCt.length > 0) {
const ct = rawCt.split(";")[0].trim().toLowerCase();
mime = this.supportedImageMime(ct);
}
if (!mime) {
// Fall back to the URL path extension. Use the pathname so the query
// string never contaminates the extension lookup.
const ext = extname(parsed.pathname).toLowerCase();
mime = EXT_TO_MIME[ext] ?? null;
}
if (!mime) {
throw new Error(
`cannot determine supported image type for "${url}"; supported: png, jpg, jpeg, gif, webp, svg`,
);
}
// Build a filename from the URL path basename (ignore the query string),
// defaulting to "image" when empty, and ensure it ends with the canonical
// extension for the resolved MIME (append it when missing/mismatched).
const canonicalExt = MIME_TO_EXT[mime];
let fileName = basename(parsed.pathname) || "image";
if (extname(fileName).toLowerCase() !== canonicalExt) {
fileName += canonicalExt;
}
return { buffer, mime, fileName };
}
/** Build a Docmost ProseMirror image node from an uploaded attachment. */
protected buildImageNode(
att: { id: string; fileName: string; fileSize?: number },
align?: "left" | "center" | "right",
alt?: string,
): any {
// Clean file URL, matching Docmost's native behaviour. No cache-busting
// query: the server serves the bare URL correctly, and replacement creates
// a new attachment id (a new URL) which busts caches naturally.
const src = `/api/files/${att.id}/${att.fileName}`;
const node: any = {
type: "image",
attrs: {
src,
attachmentId: att.id,
// Default to null when the server omits fileSize so the attr is never
// undefined (undefined would be dropped on serialization / break the
// ProseMirror image schema which expects size present).
size: att.fileSize ?? null,
align: align || "center",
width: null,
},
};
if (alt) node.attrs.alt = alt;
return node;
}
/**
* Download a remote image from an http(s) URL and upload it as an attachment
* of a page, returning the attachment metadata plus a ready-to-insert
* ProseMirror image node. Local file paths are intentionally not supported:
* the MCP caller is a remote AI with no access to this server's filesystem.
*/
async uploadImage(pageId: string, url: string) {
await this.ensureAuthenticated();
const MAX_IMAGE_BYTES = 20 * 1024 * 1024; // 20 MiB
// Fetch + validate the remote image (scheme allowlist, size cap, timeout).
// See fetchRemoteImage for the SSRF / resource trust boundary.
const fetched = await this.fetchRemoteImage(url, MAX_IMAGE_BYTES);
const fileBuffer = fetched.buffer;
const mime = fetched.mime;
const fileName = fetched.fileName;
// Build a FRESH FormData for every send attempt. A FormData body is a
// single-use stream that is CONSUMED on the first send, so it cannot be
// replayed by this.client's response interceptor (replaying a consumed
// stream fails with 'socket hang up'). Multipart re-auth is therefore done
// here with bare axios and an explicit one-shot 401/403 retry that rebuilds
// the body. Field order matters: text fields must precede the file part so
// the server reads them; the server always generates a fresh attachment id.
const buildForm = () => {
const form = new FormData();
form.append("pageId", pageId);
form.append("file", fileBuffer, {
filename: fileName,
contentType: mime,
});
return form;
};
// Local name distinct from the `url` parameter (the source image URL): this
// is the /files/upload endpoint we POST the multipart body to.
const uploadUrl = `${this.apiUrl}/files/upload`;
let response;
try {
// Call buildForm() ONCE per attempt and reuse the instance for both
// getHeaders() and the body so the Content-Type boundary matches the body.
const form = buildForm();
// Read the Authorization header from this.client's defaults (set by
// login(), only ever deleted — never set to null) instead of building
// `Bearer ${this.token}`: a concurrent JSON 401 can null this.token
// mid-flight, which would otherwise produce a literal "Bearer null".
// ensureAuthenticated() above guarantees login() ran, so the default
// header exists here. A 60s timeout keeps a hung upload from wedging the
// per-page lock (replaceImage holds withPageLock across this call).
response = await axios.post(uploadUrl, form, {
headers: {
...form.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} catch (error) {
// On an expired-token auth error, re-login and retry exactly once with a
// freshly-rebuilt FormData (the previous one was already consumed).
if (
axios.isAxiosError(error) &&
(error.response?.status === 401 || error.response?.status === 403)
) {
await this.login();
const form2 = buildForm();
response = await axios.post(uploadUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} else if (axios.isAxiosError(error)) {
// Keep the thrown message free of the raw response body (it may echo
// request data or server internals); surface only status/statusText.
// The full body is logged under DEBUG for diagnostics.
if (process.env.DEBUG) {
console.error(
"Image upload failed; response body:",
JSON.stringify(error.response?.data),
);
}
throw new Error(
`Image upload failed: ${error.response?.status} ${error.response?.statusText}`,
);
} else {
throw error;
}
}
// The attachment may arrive bare or wrapped in a { data } envelope.
const att = response.data?.data ?? response.data;
if (!att?.id || !att?.fileName) {
throw new Error(
"Unexpected /files/upload response: " + JSON.stringify(response.data),
);
}
// Some Docmost versions omit fileSize from the upload response. Fall back
// to the fetched byte length (the bytes we just uploaded) so callers never
// get an undefined size.
const resolvedSize = att.fileSize ?? fileBuffer.length;
return {
attachmentId: att.id,
fileName: att.fileName,
fileSize: resolvedSize,
src: `/api/files/${att.id}/${att.fileName}`,
imageNode: this.buildImageNode({ ...att, fileSize: resolvedSize }),
};
}
/**
* Upload an image from a web (http/https) URL and insert it into a page in
* one step.
* By default the image is appended at the end. With replaceText, the first
* top-level block whose text contains the string is replaced; with afterText,
* the image is inserted right after the first matching block. All other
* block ids are preserved (only one top-level block is added or swapped).
*/
async insertImage(
pageId: string,
url: string,
opts: {
align?: "left" | "center" | "right";
alt?: string;
replaceText?: string;
afterText?: string;
} = {},
) {
const up = await this.uploadImage(pageId, url);
// Reuse the node from uploadImage (clean /api/files/<id>/<file> src), then
// apply align/alt onto a shallow attrs copy.
const node: any = { ...up.imageNode, attrs: { ...up.imageNode.attrs } };
if (opts.align) node.attrs.align = opts.align;
if (opts.alt) node.attrs.alt = opts.alt;
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// uploadImage /files/upload call above keeps the agent-supplied id.
const pageUuid = await this.resolvePageId(pageId);
// Recursively collect the plain text of a top-level block.
const blockText = (n: any): string => {
let out = "";
if (n.type === "text") out += n.text || "";
for (const child of n.content || []) out += blockText(child);
return out;
};
// Insert into the LIVE synced document, not the debounced REST snapshot, so
// concurrent edits/comments/images are preserved and parallel insertImage
// calls (serialized by the per-page lock) each see the previous insertion.
let placement: "replaced" | "after" | "appended" | undefined;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
if (opts.replaceText) {
// Ambiguity guard (mirrors editPageText): count matching top-level
// blocks first, so a non-unique fragment cannot silently replace the
// wrong block (e.g. text that also appears inside a callout/table).
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.replaceText!),
);
if (matches.length === 0) {
throw new Error(`replaceText not found: "${opts.replaceText}"`);
}
if (matches.length > 1) {
throw new Error(
`replaceText "${opts.replaceText}" matches ${matches.length} blocks; use a longer unique fragment`,
);
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.replaceText!),
);
// Data-loss guard: replaceText swaps the WHOLE top-level block, so if
// the fragment only appears nested inside a container (table, callout,
// list, blockquote) the entire structure would be destroyed. Refuse
// when the matched block is a container rather than a leaf
// paragraph/heading and point the caller at a safer tool.
const CONTAINER_TYPES = new Set([
"table",
"callout",
"bulletList",
"orderedList",
"taskList",
"blockquote",
]);
const matchedBlock = doc.content[idx];
if (matchedBlock && CONTAINER_TYPES.has(matchedBlock.type)) {
throw new Error(
`replaceText matched a ${matchedBlock.type} container block; replacing it would destroy the whole structure. ` +
`Use afterText to insert near it, or updatePageJson for surgical edits.`,
);
}
doc.content.splice(idx, 1, node);
placement = "replaced";
} else if (opts.afterText) {
// Ambiguity guard (mirrors editPageText): refuse a non-unique fragment.
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.afterText!),
);
if (matches.length === 0) {
throw new Error(`afterText not found: "${opts.afterText}"`);
}
if (matches.length > 1) {
throw new Error(
`afterText "${opts.afterText}" matches ${matches.length} blocks; use a longer unique fragment`,
);
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.afterText!),
);
doc.content.splice(idx + 1, 0, node);
placement = "after";
} else {
doc.content.push(node);
placement = "appended";
}
return doc;
},
);
return {
success: true,
pageId,
attachmentId: up.attachmentId,
src: up.src,
placement,
verify: mutation.verify,
};
}
/**
* Replace an existing image in a page with a new image fetched from a web
* (http/https) URL. Uploads the new file as a brand-new attachment, which
* yields a fresh clean URL that both renders correctly and busts browser
* caches (the URL changed). Finds every image node
* whose attrs.attachmentId === oldAttachmentId (recursively, incl. nodes nested
* in callouts/tables) and repoints its src/attachmentId/size, preserving
* comments, alignment and alt. Operates on the live collab document so comments
* and concurrent edits are preserved. Throws if no matching image is found.
*
* The OLD attachment is left in place as an unreferenced orphan: Docmost
* exposes NO HTTP API to delete a single content attachment (verified against
* the attachment controller/service and by probing the live API deletion
* happens only by cascade when the page, space or user is removed). This is the
* same outcome as Docmost's own editor when an image is removed/replaced.
* In-place byte overwrite is deliberately NOT used because some Docmost
* versions corrupt the attachment (HTTP 500) when its bytes are overwritten.
*/
async replaceImage(
pageId: string,
oldAttachmentId: string,
url: string,
opts: { align?: "left" | "center" | "right"; alt?: string } = {},
) {
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// page lock must ALSO key on the UUID so this operation serializes against
// other writes to the same page (mutatePageContent now locks by the resolved
// UUID too); locking by the raw slugId here would desync the mutex key and
// reopen the TOCTOU/orphan-attachment window the lock closes. uploadImage
// keeps the agent-supplied id (it hits REST, not the collab doc).
const pageUuid = await this.resolvePageId(pageId);
// Hold ONE per-page lock for the WHOLE operation (scan -> upload -> write).
// Previously the scan and the write were two separate mutatePageContent
// calls, each acquiring + releasing the lock, with the upload happening in
// the UNLOCKED gap between them. A concurrent op could interleave there: it
// could remove the target image so the write pass matches nothing, leaving
// the freshly-uploaded attachment as an un-deletable orphan (Docmost has no
// API to delete a single content attachment). Acquiring the lock once and
// using the non-locking collab helper inside (the per-page mutex is NOT
// reentrant, so the self-locking mutatePageContent would deadlock here)
// closes that TOCTOU window. uploadImage hits /files/upload over plain HTTP
// and does not touch the page lock, so it is safe to call while held.
return withPageLock(pageUuid, async () => {
// STEP 1: read-only live check. Scan the live document for any image node
// matching oldAttachmentId BEFORE uploading anything, so a wrong/stale id
// throws without ever creating an orphan attachment.
let matchFound = false;
const scan = (nodes: any[]) => {
for (const node of nodes) {
if (!node) continue;
if (
node.type === "image" &&
node.attrs &&
node.attrs.attachmentId === oldAttachmentId
) {
matchFound = true;
}
if (Array.isArray(node.content)) scan(node.content);
}
};
await this.mutateLiveContentUnlocked(pageUuid, collabToken, (liveDoc) => {
matchFound = false; // reset per-transform (collab may retry the read).
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (Array.isArray(doc.content)) scan(doc.content);
return null; // read-only: never write on the check pass.
});
if (!matchFound) {
throw new Error(
`replaceImage: no image with attachmentId "${oldAttachmentId}" found on page ${pageId}`,
);
}
// STEP 2: a match exists — upload the new file as a FRESH attachment (new
// id, new clean URL) and repoint every matching node in a second pass.
// Still inside the SAME lock, so no other op can have changed the page
// since the scan.
const up = await this.uploadImage(pageId, url);
let replaced = 0;
// Swap the source of one image node, preserving align/alt/title/geometry.
const repoint = (node: any) => {
node.attrs = {
...node.attrs,
src: up.src,
attachmentId: up.attachmentId,
// Default to null when fileSize is unknown so the attr is never
// undefined.
size: up.fileSize ?? null,
};
if (opts.align) node.attrs.align = opts.align;
if (opts.alt !== undefined) node.attrs.alt = opts.alt;
replaced++;
};
// Recursively repoint every image node (incl. ones nested in callouts/tables).
const walk = (nodes: any[]) => {
for (const node of nodes) {
if (!node) continue;
if (
node.type === "image" &&
node.attrs &&
node.attrs.attachmentId === oldAttachmentId
) {
repoint(node);
}
if (Array.isArray(node.content)) walk(node.content);
}
};
const mutation = await this.mutateLiveContentUnlocked(
pageUuid,
collabToken,
(liveDoc) => {
// Reset per-transform so collab retries recompute cleanly (no double-count).
replaced = 0;
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
walk(doc.content);
if (replaced === 0) return null; // no match -> skip the write entirely
return doc;
},
);
// KNOWN LIMITATION: a same-count image SRC swap (image count unchanged, no
// text/mark change) may still report verify.changed === false, because the
// text+marks+integrity-count model in summarizeChange does not inspect
// image `src`/attachmentId attributes. That is acceptable here — the
// replace is confirmed by `replaced` below, and verify is supplementary.
if (replaced === 0) {
// The pass-1 SCAN found the target (matchFound was true) and we already
// uploaded the new attachment, but pass-2 matched nothing — a concurrent
// editor must have removed the node between the two passes. Do NOT throw
// here (that would leak the just-uploaded attachment AND report failure);
// instead report success with the upload flagged as an unreferenced
// orphan so the caller knows. (The early throw above still covers the
// case where pass-1 finds nothing, before any upload happens.)
return {
success: true,
replaced: 0,
pageId,
oldAttachmentId,
newAttachmentId: up.attachmentId,
src: up.src,
orphanedAttachmentId: up.attachmentId,
warning:
"target image was removed concurrently; uploaded attachment is unreferenced",
verify: mutation.verify,
};
}
return {
success: true,
pageId,
replaced,
oldAttachmentId,
newAttachmentId: up.attachmentId,
src: up.src,
verify: mutation.verify,
};
});
}
// --- draw.io diagrams (issue #423) ---
/**
* Upload a ready-made byte buffer as a page attachment via the same
* multipart /files/upload endpoint uploadImage uses. Split out as its own
* (overridable) seam so drawioCreate/update can upload the generated
* `.drawio.svg` without going through the URL-fetch path, and so tests can
* stub the network. Mirrors uploadImage's fresh-FormData + one-shot 401/403
* re-auth handling (a FormData body is single-use, so it must be rebuilt per
* attempt).
*/
// --- draw.io diagrams (issue #423) ---
/**
* Upload a ready-made byte buffer as a page attachment via the same
* multipart /files/upload endpoint uploadImage uses. Split out as its own
* (overridable) seam so drawioCreate/update can upload the generated
* `.drawio.svg` without going through the URL-fetch path, and so tests can
* stub the network. Mirrors uploadImage's fresh-FormData + one-shot 401/403
* re-auth handling (a FormData body is single-use, so it must be rebuilt per
* attempt).
*/
protected async uploadAttachmentBuffer(
pageId: string,
buffer: Buffer,
fileName: string,
mime: string,
): Promise<{ id: string; fileName: string; fileSize: number }> {
await this.ensureAuthenticated();
const buildForm = () => {
const form = new FormData();
form.append("pageId", pageId);
form.append("file", buffer, { filename: fileName, contentType: mime });
return form;
};
const uploadUrl = `${this.apiUrl}/files/upload`;
let response;
try {
const form = buildForm();
response = await axios.post(uploadUrl, form, {
headers: {
...form.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} catch (error) {
if (
axios.isAxiosError(error) &&
(error.response?.status === 401 || error.response?.status === 403)
) {
await this.login();
const form2 = buildForm();
response = await axios.post(uploadUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} else if (axios.isAxiosError(error)) {
if (process.env.DEBUG) {
console.error(
"Attachment upload failed; response body:",
JSON.stringify(error.response?.data),
);
}
throw new Error(
`Attachment upload failed: ${error.response?.status} ${error.response?.statusText}`,
);
} else {
throw error;
}
}
const att = response.data?.data ?? response.data;
if (!att?.id || !att?.fileName) {
throw new Error(
"Unexpected /files/upload response: " + JSON.stringify(response.data),
);
}
return {
id: att.id,
fileName: att.fileName,
fileSize: att.fileSize ?? buffer.length,
};
}
/**
* Fetch a stored `.drawio.svg` attachment as text. Overridable seam over
* fetchInternalFile (the authed loopback fetch, which also rejects any
* traversal/SSRF src) so drawioGet/update can read the current diagram and
* tests can stub the bytes.
*/
protected async fetchAttachmentText(src: string): Promise<string> {
const { buffer } = await this.fetchInternalFile(src);
return buffer.toString("utf-8");
}
/**
* Resolve a drawio node on a page by `attrs.id` or `#<index>` and return the
* node plus its ref. Throws a clear error if the ref does not resolve to a
* drawio node.
*/
}
return MediaMixin;
}
+700
View File
@@ -0,0 +1,700 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
importMarkdownFragment,
canBeDocChild,
findUnrepresentableTableAttrs,
} from "../lib/markdown-fragment.js";
import {
applyTextEdits,
TextEdit,
TextEditResult,
TextEditFailure,
} from "../lib/json-edit.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
import { normalizeAndMergeFootnotes } from "../lib/footnote-normalize-merge.js";
// Public method surface of NodesWriteMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements INodesWriteMixin` fails to compile on drift.
export interface INodesWriteMixin {
updatePageJson(pageId: string, doc?: any, title?: string): any;
editPageText(pageId: string, edits: TextEdit[]): any;
patchNode(pageId: string, nodeId: string, input: { markdown?: string; node?: any }): any;
insertNode(pageId: string, input: { markdown?: string; node?: any }, opts: { position: "before" | "after" | "append"; anchorNodeId?: string; anchorText?: string; }): any;
deleteNode(pageId: string, nodeId: string): any;
}
export function NodesWriteMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & INodesWriteMixin> & TBase {
abstract class NodesWriteMixin extends Base implements INodesWriteMixin {
/**
* Replace page content with a raw ProseMirror JSON document (lossless) and/or
* update its title. Both `doc` and `title` are optional, but at least one must
* be supplied:
* - `doc` provided -> validate + full-overwrite the body (and update the
* title too when `title` is also given).
* - `doc` omitted, `title` given -> title-only update; the body is NOT
* touched/resent (no collab write happens).
* - neither given -> throws (nothing to update).
*/
async updatePageJson(pageId: string, doc?: any, title?: string) {
await this.ensureAuthenticated();
// Title-only / no-op handling: when no document is supplied, do NOT write
// the body. Update the title if one was given; otherwise there is nothing
// to do, so fail loudly rather than silently no-op.
if (doc == null) {
if (!title) {
throw new Error(
"updatePageJson: nothing to update (provide content and/or title)",
);
}
await this.client.post("/pages/update", { pageId, title });
return {
success: true,
modified: true,
message: "Page title updated (content left unchanged).",
pageId,
};
}
// Validate the document shape before a full overwrite: a malformed doc
// would otherwise silently corrupt the page (full-overwrite is the
// documented behaviour; no optimistic-concurrency is applied here).
if (
typeof doc !== "object" ||
doc.type !== "doc" ||
!Array.isArray(doc.content)
) {
throw new Error(
'content must be a ProseMirror document ({"type":"doc","content":[...]}) ' +
"where content is an array of nodes each having a string `type`",
);
}
// Recurse the WHOLE document so a malformed nested node (e.g. a node with a
// non-string type, a non-array content/marks, or a text node missing its
// string text) is rejected up front rather than silently corrupting the
// page on overwrite.
this.validateDocStructure(doc);
// #409: beyond the string-`type` check above, reject a nested node whose
// `type` is a string but NOT a known Docmost schema node (a typo/unknown
// block) — the same `Unknown node type` the encoder throws — with a rich,
// path-anchored message, still BEFORE any collab connection.
this.assertValidNodeShape("updatePageJson", doc);
// Sanitize URLs before writing. This closes the JSON-path bypass: unlike
// the markdown link path (which TipTap sanitizes), raw JSON could otherwise
// inject javascript:/data: link hrefs or media srcs straight into the doc.
this.validateDocUrls(doc);
// Canonicalize footnotes (idempotent): an agent-authored JSON doc cannot
// leave footnotes out of order, orphaned, or in multiple lists — the bottom
// list + numbering are always derived from reference order. No-op when the
// footnotes are already canonical.
// #419: normalize + merge glyph-forked definitions before canonicalizing.
doc = normalizeAndMergeFootnotes(doc);
doc = canonicalizeFootnotes(doc);
// Write the BODY first, then the title (#159 split-brain): a failed body
// write (e.g. persist timeout) must not leave a new title over the old body.
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
const mutation = await this.replacePage(
pageUuid,
doc,
collabToken,
this.apiUrl,
);
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
return {
success: true,
modified: true,
message: "Page content replaced from ProseMirror JSON.",
pageId,
verify: mutation.verify,
};
}
/**
* AUTHOR-INLINE footnote insertion. The agent supplies only WHERE
* (`anchorText`, a snippet of body text to attach the marker after) and WHAT
* (`text`, the footnote content as markdown). Numbering and the bottom
* `footnotesList` are derived deterministically server-side
* (`insertInlineFootnote` -> `canonicalizeFootnotes`): the agent never sees,
* assigns, or edits a footnote number or the list, so it CANNOT desync.
*
* Content DEDUP: when an existing definition has the same content, its id is
* reused (one number, one definition, several references). The write is atomic
* via `mutatePageContent` (single-writer, page-locked); if the anchor text is
* not found the transform aborts with a clear error and no write happens.
*/
/**
* Surgical text edits: find/replace inside text nodes of the live
* document. Preserves all block ids, marks, callouts and tables.
*/
async editPageText(pageId: string, edits: TextEdit[]) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Apply the edits against the LIVE synced document, not the debounced REST
// snapshot, so concurrent human edits/comments are preserved. applyTextEdits
// records per-edit match problems in `failed` instead of throwing, and
// applies whatever it can; we abort the write only when nothing applied.
let results: TextEditResult[] | undefined;
let failed: TextEditFailure[] | undefined;
// Whether we actually wrote new content. Set inside the transform: a
// degenerate edit (e.g. find === replace, or a batch that nets to no change)
// can "apply" yet leave the document byte-for-byte identical, in which case
// we must NOT write (no spurious history version) and must not claim a write
// happened.
let wrote = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
wrote = false;
const r = applyTextEdits(liveDoc, edits);
results = r.results;
failed = r.failed;
// Nothing applied -> abort the write (mutatePageContent treats a null
// return from the transform as "write nothing").
if (r.results.length === 0) return null;
// Edits "applied" but produced an identical document: skip the write so
// no new history version is created. Stable structural comparison via
// JSON.stringify (both docs come from the same deep-copied source, so
// key order is stable).
if (JSON.stringify(r.doc) === JSON.stringify(liveDoc)) return null;
wrote = true;
return r.doc;
},
);
if ((results?.length ?? 0) === 0 && (failed?.length ?? 0) > 0) {
// No edit applied: surface an aggregated, actionable error so the caller
// does not mistake a no-op for a partial success.
throw new Error(
"editPageText: no edits were applied (nothing written). " +
failed!.map((f) => `"${f.find}": ${f.reason}`).join("; "),
);
}
// Edits matched but produced no content change (identical document): report
// a successful no-op — NOT a failure — and do not falsely claim a write.
if (!wrote) {
return {
success: true,
pageId,
applied: results,
failed,
message: "No changes written (edits produced identical content).",
verify: mutation.verify,
};
}
const result: any = {
success: true,
pageId,
applied: results,
failed,
message:
(failed?.length ?? 0)
? `Applied ${results?.length ?? 0} edit(s); ${failed!.length} failed (see failed[]). Node ids and formatting preserved.`
: "Text edits applied (node ids and formatting preserved).",
verify: mutation.verify,
};
// If any applied edit matched only after stripping markdown (the
// normalized fallback), warn that editPageText preserved existing marks
// and did NOT change formatting — so a caller who intended a formatting
// change is pointed at patchNode.
if (results?.some((r) => r.normalized === true)) {
result.warning =
"Some edits matched only after stripping markdown from your find string; " +
"editPageText preserved existing marks (it did not change bold/strike/etc.). " +
"If you intended a formatting change, use patchNode.";
}
return result;
}
/**
* Replace the block whose attrs.id === nodeId. Operates on the LIVE collab
* document so comments and concurrent edits are preserved.
*
* Exactly one of `input.markdown` / `input.node` (#413):
* - `markdown` (RECOMMENDED): the block is rewritten from a canonical markdown
* fragment. The fragment may import to N blocks (a "1 -> N" splice: rewrite a
* whole section in one call). The FIRST resulting block INHERITS the target's
* `attrs.id` (so an existing comment anchoring the block by id survives); the
* rest get FRESH ids. `^[...]` footnotes in the fragment are first-class:
* their definitions merge into the page's TAIL footnote list (content-key
* dedup + canonicalize), same machinery insertFootnote uses. REJECTED when
* the TARGET block carries a table-cell attribute markdown cannot represent
* (colspan/rowspan/colwidth/background) use the table tools or `node`.
* - `node`: a raw ProseMirror node for precise attr/mark work. The replacement
* keeps the target id (if `node.attrs.id` is missing it is set to nodeId).
*
* #159 ambiguous-id semantics are unchanged: 0 matches -> "no node"; >1 matches
* -> "ambiguous, refused" (nothing written), on BOTH paths the markdown path
* runs a dry `replaceNodeById` count first, so a duplicated id never splices.
*/
async patchNode(
pageId: string,
nodeId: string,
input: { markdown?: string; node?: any },
) {
await this.ensureAuthenticated();
// XOR: exactly one of markdown / node. Both optional in the schema; the
// runtime enforces the recommendation ("markdown for prose, node for fine
// work") without letting an ambiguous both-or-neither call through.
const hasMd =
input != null &&
typeof input.markdown === "string" &&
input.markdown.trim() !== "";
const hasNode = input != null && input.node != null;
if (hasMd === hasNode) {
throw new Error(
"patchNode: provide exactly one of `markdown` (recommended, for prose) " +
"or `node` (a raw ProseMirror node, for precise attr/mark work)",
);
}
if (hasMd) {
return this.patchNodeMarkdown(pageId, nodeId, input.markdown as string);
}
return this.patchNodeJson(pageId, nodeId, input.node);
}
/**
* patchNode with a raw ProseMirror `node` (the pre-#413 behavior). Replaces
* EVERY node whose attrs.id === nodeId; the swapped-in node keeps the target
* id. #159 ambiguity refused. Split out so the markdown path can reuse the
* shared collab/guard plumbing without a giant branch.
*/
protected async patchNodeJson(pageId: string, nodeId: string, node: any) {
if (!node || typeof node !== "object" || typeof node.type !== "string") {
throw new Error(
"patchNode: `node` must be an object with a string `type`",
);
}
// Preserve the block id WITHOUT mutating the caller's object: build a local
// copy whose attrs.id === nodeId (so the swapped-in node keeps the id of the
// node it replaces).
const target = {
...node,
attrs: {
...(node.attrs && typeof node.attrs === "object" ? node.attrs : {}),
},
};
if (target.attrs.id == null) {
target.attrs.id = nodeId;
}
// #409: fail fast on a malformed node SHAPE (a nested child with an
// absent/unknown `type`, e.g. a text leaf written as `{"text":"foo"}` with
// no `"type":"text"`) BEFORE opening a collab session or taking the page
// lock — the root-only `typeof node.type === "string"` check above never
// sees nested children, and the encoder's `Unknown node type: undefined`
// would otherwise only surface after the connection.
this.assertValidNodeShape("patchNode", target);
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track the replacement count in an outer var, reset per-transform, so a
// collab retry recomputes it cleanly (mirrors replaceImage's pattern).
let replaced = 0;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
replaced = 0;
const { doc: nd, replaced: r } = replaceNodeById(
liveDoc,
nodeId,
target,
);
replaced = r;
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS: Docmost
// duplicates block ids on copy/paste (and copyPageContent writes them
// verbatim), so replacing "the node with id X" would silently clobber
// EVERY duplicate (#159). Refuse: skip the write and throw below so the
// model re-targets with a more specific anchor instead of corrupting the
// page. Only an unambiguous single match is written.
if (replaced !== 1) return null;
return nd;
},
);
// 0 -> "no node"; >1 -> "ambiguous, refused" (the transform already skipped
// the write for any count !== 1). Single shared guard (#159, #185 review).
assertUnambiguousMatch("patchNode", "replace", replaced, nodeId, pageId);
return { success: true, replaced, nodeId, verify: mutation.verify };
}
/**
* patchNode with a MARKDOWN fragment (#413). Imports the fragment through the
* canonical importer, then 1 -> N splices the resulting blocks in place of the
* target block on the LIVE collab doc:
* - the FIRST block inherits the target's id; the rest get FRESH ids (minted
* by the importer/id-remap, so neighbour blocks are untouched);
* - `^[...]` footnote definitions merge into the page's tail list;
* - REJECTED when the target block carries a markdown-unrepresentable table
* attr (colspan/rowspan/colwidth/background) guarding against silent loss;
* - #159 ambiguity is enforced by a dry `replaceNodeById` count BEFORE the
* splice, so a duplicated id never writes.
*/
protected async patchNodeMarkdown(
pageId: string,
nodeId: string,
markdown: string,
) {
// Import the fragment up front (network-free, canonical) so a bad fragment
// fails before any collab connection or page lock.
const { blocks, definitions } = await importMarkdownFragment(markdown);
// The first imported block inherits the target id; the rest keep the fresh
// ids the importer assigned. Build the thread now so it is stable across a
// collab retry (the transform below is pure over its inputs).
const threaded = blocks.map((b, i) => {
if (i !== 0) return b;
return {
...b,
attrs: {
...(b && typeof b.attrs === "object" ? b.attrs : {}),
id: nodeId,
},
};
});
// Shape-validate every imported block up front (parity with the JSON path):
// the importer only emits schema nodes, but the check is cheap insurance and
// yields the same rich #409 diagnostics if the schema ever drifts.
for (const b of threaded) {
this.assertValidNodeShape("patchNode", b);
}
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let replaced = 0;
let guardAttrs: string | null = null;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
replaced = 0;
guardAttrs = null;
// #159: count matches with the same recursive walk the JSON path uses;
// only an UNAMBIGUOUS single match may write. A dry count keeps the
// ambiguity semantics identical across both paths.
const { replaced: count } = replaceNodeById(liveDoc, nodeId, {
type: "paragraph",
});
replaced = count;
if (count !== 1) return null;
// Guard against SILENT LOSS: if the target block carries a table-cell
// attribute markdown cannot represent (colspan/rowspan/colwidth/
// background), refuse the markdown rewrite so those attrs are not
// dropped. Simple tables (no such attrs) rewrite fine.
const hit = getNodeByRef(liveDoc, nodeId);
guardAttrs = hit ? findUnrepresentableTableAttrs(hit.node) : null;
if (guardAttrs != null) return null;
// Re-mint any minted block id that collides with an existing page id
// (skip index 0: its id is intentionally the target nodeId, unique by
// the #159 dry-count above), so the 1 -> N splice stays page-wide unique.
reassignCollidingBlockIds(liveDoc, threaded, 0);
// 1 -> N splice, then merge any fragment footnote definitions into the
// page's tail list and re-derive canonical footnote numbering.
const { doc: spliced } = replaceNodeByIdWithMany(
liveDoc,
nodeId,
threaded,
);
return mergeFootnoteDefinitions(spliced, definitions);
},
);
// Surface the guard rejection with an actionable message (nothing written).
if (guardAttrs != null) {
throw new Error(
`patchNode: the target block has table-cell attributes markdown cannot ` +
`represent (${guardAttrs}) — a markdown rewrite would drop them. Use ` +
`the table tools (tableUpdateCell/tableInsertRow) or pass a raw ` +
`ProseMirror \`node\` instead of \`markdown\`.`,
);
}
// 0 -> "no node"; >1 -> "ambiguous, refused" (the transform skipped the write
// for any count !== 1). Shared #159 guard, identical to the JSON path.
assertUnambiguousMatch("patchNode", "replace", replaced, nodeId, pageId);
return {
success: true,
replaced,
nodeId,
blocks: threaded.length,
verify: mutation.verify,
};
}
/**
* Insert content relative to an anchor (or append it at the top level).
* Operates on the LIVE collab document so comments and concurrent edits are
* preserved.
*
* Exactly one of `input.markdown` / `input.node` (#413):
* - `markdown` (RECOMMENDED): a canonical markdown fragment. It may import to
* SEVERAL blocks they are inserted IN ORDER at the anchor. `^[...]`
* footnote definitions merge into the page's tail list (same machinery as
* insertFootnote). Every inserted block gets a fresh id.
* - `node`: a raw ProseMirror node for precise attr/mark work, or to insert
* table structure (a bare tableRow/tableCell/tableHeader NOT expressible in
* markdown, so those stay JSON-only).
*
* opts.position:
* - "append": push the content at the end of the top-level content.
* - "before"/"after": insert as a sibling of the anchor, just before/after it.
* Exactly one of anchorNodeId / anchorText must be given; anchorNodeId
* locates a node anywhere by attrs.id, anchorText matches the first top-level
* block whose plain text includes it.
*
* Throws if the anchor cannot be found.
*/
async insertNode(
pageId: string,
input: { markdown?: string; node?: any },
opts: {
position: "before" | "after" | "append";
anchorNodeId?: string;
anchorText?: string;
},
) {
await this.ensureAuthenticated();
// XOR: exactly one of markdown / node (both optional in the schema).
const hasMd =
input != null &&
typeof input.markdown === "string" &&
input.markdown.trim() !== "";
const hasNode = input != null && input.node != null;
if (hasMd === hasNode) {
throw new Error(
"insertNode: provide exactly one of `markdown` (recommended, for prose) " +
"or `node` (a raw ProseMirror node, for precise attr/mark work or table structure)",
);
}
if (
!opts ||
(opts.position !== "before" &&
opts.position !== "after" &&
opts.position !== "append")
) {
throw new Error(
'insertNode: `position` must be one of "before", "after", "append"',
);
}
if (opts.position === "before" || opts.position === "after") {
// before/after require EXACTLY ONE anchor (an id or a text fragment).
const hasId =
typeof opts.anchorNodeId === "string" && opts.anchorNodeId.length > 0;
const hasText =
typeof opts.anchorText === "string" && opts.anchorText.length > 0;
if (hasId === hasText) {
throw new Error(
`insertNode: position "${opts.position}" requires exactly one of anchorNodeId or anchorText`,
);
}
}
// Resolve the ordered list of blocks to insert plus any footnote definitions
// to merge. The markdown path imports canonically (so an inserted block is
// byte-identical to the same content in a full-page import); the node path is
// a single block with no footnote merge (raw JSON `^[...]` is not touched).
let blocks: any[];
let definitions: any[] = [];
if (hasMd) {
const frag = await importMarkdownFragment(input.markdown as string);
blocks = frag.blocks;
definitions = frag.definitions;
} else {
const node = input.node;
if (!node || typeof node !== "object" || typeof node.type !== "string") {
throw new Error(
"insertNode: `node` must be an object with a string `type`",
);
}
blocks = [node];
}
// #409: fail fast on a malformed node SHAPE (a nested child with an
// absent/unknown `type`) BEFORE opening a collab session or taking the page
// lock — the root-only check above never sees nested children.
for (const b of blocks) {
this.assertValidNodeShape("insertNode", b);
}
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track insertion in an outer var, reset per-transform, so a collab retry
// recomputes it cleanly (mirrors replaceImage's pattern).
let inserted = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
inserted = false;
// Re-mint any minted block id that collides with an existing page id
// (all inserted blocks are fresh, no skip) so the splice stays unique.
if (hasMd) reassignCollidingBlockIds(liveDoc, blocks);
// Single-block node path keeps `insertNodeRelative` (it owns the
// structural table-node splicing); the markdown path uses the array
// splice so N blocks land in order at one anchor.
const res = hasMd
? insertNodesRelative(liveDoc, blocks, opts)
: insertNodeRelative(liveDoc, blocks[0], opts);
inserted = res.inserted;
if (!inserted) return null; // anchor not found -> skip the write entirely
// Merge any fragment footnote definitions into the page tail list and
// re-derive canonical numbering (no-op when there are none).
return mergeFootnoteDefinitions(res.doc, definitions);
},
);
if (!inserted) {
const anchorDesc = opts.anchorNodeId
? `anchorNodeId "${opts.anchorNodeId}"`
: `anchorText "${opts.anchorText}"`;
// anchorText is matched against the block's literal RENDERED plain text;
// markdown/emoji are tolerated only as a strip-and-retry fallback, so a
// miss usually means the text differs from what's on the page.
const hint = opts.anchorText
? " anchorText must be the block's literal rendered plain text (no markdown wrappers or emoji); anchorNodeId from getPageJson is more reliable."
: "";
throw new Error(
`insertNode: anchor not found (${anchorDesc}) on page ${pageId}.${hint}`,
);
}
return {
success: true,
inserted: true,
position: opts.position,
blocks: blocks.length,
verify: mutation.verify,
};
}
/**
* Remove EVERY node whose attrs.id === nodeId (recursively, including nodes
* nested in callouts/tables) from its parent content array. Operates on the
* LIVE collab document so comments and concurrent edits are preserved.
* Throws if no node matches.
*/
async deleteNode(pageId: string, nodeId: string) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track the deletion count in an outer var, reset per-transform, so a
// collab retry recomputes it cleanly (mirrors replaceImage's pattern).
let deleted = 0;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
deleted = 0;
const { doc: nd, deleted: d } = deleteNodeById(liveDoc, nodeId);
deleted = d;
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS (block
// ids are duplicated on copy/paste, #159): deleting "the node with id X"
// would silently remove EVERY duplicate. Refuse: skip the write and throw
// below so the model re-targets. Only an unambiguous single match is
// deleted.
if (deleted !== 1) return null;
return nd;
},
);
// 0 -> "no node"; >1 -> "ambiguous, refused" (the transform already skipped
// the write for any count !== 1). Single shared guard (#159, #185 review).
assertUnambiguousMatch("deleteNode", "delete", deleted, nodeId, pageId);
return { success: true, deleted, nodeId, verify: mutation.verify };
}
/** Build the public share URL for a page. */
}
return NodesWriteMixin;
}
+655
View File
@@ -0,0 +1,655 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import FormData from "form-data";
import axios, { AxiosInstance } from "axios";
import { convertProseMirrorToMarkdown } from "../lib/markdown-converter.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { footnoteWarningsField } from "../lib/footnote-analyze.js";
import {
serializeDocmostMarkdown,
parseDocmostMarkdown,
} from "../lib/markdown-document.js";
import { diffDocs, summarizeChange } from "../lib/diff.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
import { normalizeAndMergeFootnotes } from "../lib/footnote-normalize-merge.js";
import vm from "node:vm";
// Public method surface of PagesMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IPagesMixin` fails to compile on drift.
export interface IPagesMixin {
createPage(title: string, content: string, spaceId: string, parentPageId?: string): any;
updatePage(pageId: string, content: string, title?: string): any;
renamePage(pageId: string, title: string): any;
movePage(pageId: string, parentPageId: string | null, position?: string): any;
deletePage(pageId: string): any;
sharePage(pageId: string, searchIndexing?: boolean): any;
listShares(): any;
unsharePage(pageId: string): any;
exportPageMarkdown(pageId: string): Promise<string>;
importPageMarkdown(pageId: string, fullMarkdown: string): Promise<any>;
copyPageContent(sourcePageId: string, targetPageId: string): any;
listPageHistory(pageId: string, cursor?: string): any;
getPageHistory(historyId: string): any;
restorePageVersion(historyId: string): any;
diffPageVersions(pageId: string, from?: string, to?: string): any;
}
export function PagesMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IPagesMixin> & TBase {
abstract class PagesMixin extends Base implements IPagesMixin {
/**
* Create a new page with title and content.
* Uses the /pages/import workaround (the only endpoint accepting content),
* then moves the page and restores the exact title: the import endpoint
* derives the title from the FILENAME and replaces spaces with
* underscores, so we explicitly re-set it via /pages/update afterwards.
*/
async createPage(
title: string,
content: string,
spaceId: string,
parentPageId?: string,
) {
await this.ensureAuthenticated();
if (parentPageId) {
try {
await this.getPage(parentPageId);
} catch (e) {
throw new Error(`Parent page with ID ${parentPageId} not found.`);
}
}
// 1. Create content via Import (using multipart/form-data).
// Build a FRESH FormData per send attempt: a FormData body is a single-use
// stream consumed on the first send, so it cannot be replayed by
// this.client's response interceptor (replay fails with 'socket hang up').
// Multipart re-auth is therefore done here with bare axios and an explicit
// one-shot 401/403 retry that rebuilds the body.
const fileContent = Buffer.from(content, "utf-8");
const buildForm = () => {
const form = new FormData();
form.append("spaceId", spaceId);
form.append("file", fileContent, {
filename: `${title || "import"}.md`,
contentType: "text/markdown",
});
return form;
};
const importUrl = `${this.apiUrl}/pages/import`;
let response;
try {
// Call buildForm() ONCE per attempt and reuse the instance for both
// getHeaders() and the body so the Content-Type boundary matches the body.
const form = buildForm();
// Read the Authorization header from this.client's defaults (set by
// login(), only ever deleted — never set to null) instead of building
// `Bearer ${this.token}`: a concurrent JSON 401 can null this.token
// mid-flight, which would otherwise produce a literal "Bearer null".
// ensureAuthenticated() above guarantees login() ran, so the default
// header exists here.
response = await axios.post(importUrl, form, {
headers: {
...form.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} catch (error) {
// On an expired-token auth error, re-login and retry exactly once with a
// freshly-rebuilt FormData (the previous one was already consumed).
if (
axios.isAxiosError(error) &&
(error.response?.status === 401 || error.response?.status === 403)
) {
await this.login();
const form2 = buildForm();
response = await axios.post(importUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
} else {
throw error;
}
}
const newPageId = (response.data?.data ?? response.data).id;
// 2. Move to parent if needed
if (parentPageId) {
await this.movePage(newPageId, parentPageId);
}
// 3. Restore the exact title (import mangles spaces into underscores)
if (title) {
await this.client.post("/pages/update", { pageId: newPageId, title });
}
const page = await this.getPage(newPageId);
// Surface non-fatal footnote problems (dangling refs, empty/duplicate
// definitions, markers in tables) so the agent can fix its markup (#166).
return { ...page, ...footnoteWarningsField(content) };
}
/**
* Update a page's content from markdown and optionally its title.
* NOTE: full re-import block ids regenerate. For surgical changes
* use editPageText / updatePageJson instead.
*/
async updatePage(pageId: string, content: string, title?: string) {
await this.ensureAuthenticated();
// Open the collab doc by the canonical UUID, never the slugId (#260). The
// REST /pages/update title write below keeps the agent-supplied id (the
// server resolves a slugId there).
const pageUuid = await this.resolvePageId(pageId);
// Write the BODY first, then the title (#159 split-brain). If the collab
// body write fails (e.g. a persist timeout), the title must be left
// UNTOUCHED so the page never ends up with a new title over its old body.
// A title write failing AFTER a successful body is rarer (REST is fast) and
// leaves correct content under a stale title — the lesser inconsistency.
let collabToken = "";
let mutation;
try {
collabToken = await this.getCollabTokenWithReauth();
mutation = await updatePageContentRealtime(
pageUuid,
content,
collabToken,
this.apiUrl,
);
} catch (error: any) {
// Verbose diagnostics (incl. anything that could expose a token prefix)
// are gated behind DEBUG; the thrown Error below carries no token data.
if (process.env.DEBUG) {
console.error(
"Failed to update page content via realtime collaboration:",
error,
);
const tokenPreview = collabToken
? collabToken.substring(0, 15) + "..."
: "null";
console.error(`Collab token preview: ${tokenPreview}`);
}
throw new Error(`Failed to update page content: ${error.message}`);
}
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
return {
success: true,
modified: true,
message: "Page updated successfully.",
pageId: pageId,
verify: mutation.verify,
// Non-fatal footnote diagnostics (#166); omitted when there are none.
...footnoteWarningsField(content),
};
}
/**
* Validate a URL string against a scheme allowlist for a given context.
*
* The markdown link path enforces safe schemes via TipTap, but the raw
* JSON path (updatePageJson) bypasses that so this is the sanitization
* choke point for ProseMirror JSON written directly by the caller.
*
* - "link": reject javascript:, vbscript:, data: (any scheme that can
* execute or smuggle script when the href is clicked).
* - "src": allow only http(s):, mailto:, /api/files paths, or a
* scheme-less relative/absolute path; reject
* javascript:/vbscript:/data:/file:.
*/
/**
* Rename a page (change its title only) without touching or resending its
* content. The slug is derived from the page record, not the body, so it is
* left intact too.
*/
async renamePage(pageId: string, title: string) {
await this.ensureAuthenticated();
await this.client.post("/pages/update", { pageId, title });
return { success: true, pageId, title };
}
/**
* Copy the WHOLE content of one page onto another, entirely server-side: the
* source's ProseMirror document is read and written verbatim onto the target
* via the live collab path, so the document never passes through the model.
*
* Only the target's BODY is replaced its title and slug live on the page
* record (not in the content), so they are untouched. The source page is not
* modified at all.
*/
async movePage(
pageId: string,
parentPageId: string | null,
position?: string,
) {
await this.ensureAuthenticated();
// Docmost requires position >= 5 chars.
const validPosition = position || "a00000";
return this.client
.post("/pages/move", {
pageId,
parentPageId,
position: validPosition,
})
.then((res) => res.data);
}
async deletePage(pageId: string) {
await this.ensureAuthenticated();
return this.client
.post("/pages/delete", { pageId })
.then((res) => res.data);
}
// --- Comment methods (ported from upstream PR #3 by Max Nikitin) ---
/**
* Normalize a comment's `content` into a ProseMirror doc object before
* markdown conversion. createComment/updateComment send content as a
* JSON.stringify(...) STRING, and the server stores it as-is, so on read it
* comes back as a string. convertProseMirrorToMarkdown returns "" for a
* string, so parse it first (guarded fall back to the raw value on any
* parse failure so a non-JSON legacy value is still handled gracefully).
*/
/** Share a page publicly (idempotent) and return the public URL. */
async sharePage(pageId: string, searchIndexing: boolean = true) {
await this.ensureAuthenticated();
const response = await this.client.post("/shares/create", {
pageId,
includeSubPages: false,
searchIndexing,
});
const share = response.data?.data ?? response.data;
const slugId = share.page?.slugId || (await this.getPageRaw(pageId)).slugId;
return {
shareId: share.id,
key: share.key,
pageId: share.pageId,
publicUrl: this.shareUrl(share.key, slugId),
searchIndexing: share.searchIndexing,
};
}
/** List all public shares in the workspace with their URLs. */
/** Build the public share URL for a page. */
protected shareUrl(shareKey: string, slugId: string): string {
return `${this.appUrl}/share/${shareKey}/p/${slugId}`;
}
/** Share a page publicly (idempotent) and return the public URL. */
/** List all public shares in the workspace with their URLs. */
async listShares() {
const shares = await this.paginateAll("/shares", {});
return shares.map((s: any) => ({
shareId: s.id,
key: s.key,
pageId: s.pageId,
pageTitle: s.page?.title,
publicUrl: s.page?.slugId ? this.shareUrl(s.key, s.page.slugId) : null,
searchIndexing: s.searchIndexing,
createdAt: s.createdAt,
}));
}
/** Remove the public share of a page. */
async unsharePage(pageId: string) {
await this.ensureAuthenticated();
const shares = await this.listShares();
const share = shares.find((s: any) => s.pageId === pageId);
if (!share) {
throw new Error(`Page ${pageId} is not shared.`);
}
await this.client.post("/shares/delete", { shareId: share.shareId });
return { success: true, removedShareId: share.shareId, pageId };
}
/**
* Export a page to a single self-contained Docmost-flavoured markdown file:
* meta block + body (with inline comment anchors + diagrams) + comment
* threads. Lossless round-trip target; see importPageMarkdown for the inverse.
*/
async exportPageMarkdown(pageId: string): Promise<string> {
await this.ensureAuthenticated();
const page = await this.getPageRaw(pageId);
const body = page.content ? convertProseMirrorToMarkdown(page.content) : "";
let comments: any[] = [];
try {
// Lossless export: include RESOLVED threads so the export -> import
// round-trip preserves every comment. This is exactly why the active-only
// filter is an opt-in (default false) on listComments.
comments = (await this.listComments(pageId, true)).items;
} catch (e) {
// A comments fetch failure must not lose the body; export with [] and let
// the caller see the (empty) comments block. Log under DEBUG only.
if (process.env.DEBUG) console.error("export: listComments failed", e);
}
const meta = {
version: 1,
pageId: page.id,
slugId: page.slugId,
title: page.title,
spaceId: page.spaceId,
parentPageId: page.parentPageId ?? null,
};
return serializeDocmostMarkdown(meta, body, comments);
}
/**
* Import a self-contained Docmost markdown file back into a page. Parses out
* the meta + comments metadata blocks, converts the body to ProseMirror
* (restoring comment marks + diagrams from their inline HTML), and replaces
* the page content. Comment THREAD records are NOT written to the server in
* this version they are preserved in the file and the inline marks are
* re-applied so the highlights survive; managing comment records stays with
* the comment tools/UI.
*/
async importPageMarkdown(pageId: string, fullMarkdown: string): Promise<any> {
await this.ensureAuthenticated();
const { meta, body, comments } = parseDocmostMarkdown(fullMarkdown);
// PAGE import: canonicalize footnotes (see markdownToProseMirrorCanonical).
const doc = await markdownToProseMirrorCanonical(body);
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
const mutation = await replacePageContent(
pageUuid,
doc,
collabToken,
this.apiUrl,
);
// Collect distinct comment ids that actually became comment marks in the doc.
const collectCommentIds = (node: any, acc: Set<string>): Set<string> => {
if (!node || typeof node !== "object") return acc;
if (Array.isArray(node.marks)) {
for (const mk of node.marks) {
if (mk && mk.type === "comment" && mk.attrs?.commentId) {
acc.add(mk.attrs.commentId);
}
}
}
if (Array.isArray(node.content)) {
for (const child of node.content) collectCommentIds(child, acc);
}
return acc;
};
// Count reflects the comment marks present in the written document, so an id
// that only appears as inert text (e.g. inside a fenced code block) is not
// counted because it never becomes a comment mark.
const anchoredIds = collectCommentIds(doc, new Set<string>());
const result: any = {
success: true,
pageId,
anchoredCommentCount: anchoredIds.size,
commentsInFile: Array.isArray(comments) ? comments.length : 0,
verify: mutation.verify,
};
// Warn (non-fatal) if the file was exported from a DIFFERENT page.
if (meta?.pageId && meta.pageId !== pageId) {
result.warning = `File was exported from page ${meta.pageId} but is being imported into ${pageId}.`;
}
// Non-fatal footnote diagnostics (#166), analyzed on the BODY (the part after
// the docmost:meta / docmost:comments blocks) — so a `[^x]`-like token inside
// those JSON blocks never produces a false warning, while real markers in the
// body do. `body` comes from parseDocmostMarkdown(fullMarkdown) above.
Object.assign(result, footnoteWarningsField(body));
return result;
}
/**
* Rename a page (change its title only) without touching or resending its
* content. The slug is derived from the page record, not the body, so it is
* left intact too.
*/
/**
* Copy the WHOLE content of one page onto another, entirely server-side: the
* source's ProseMirror document is read and written verbatim onto the target
* via the live collab path, so the document never passes through the model.
*
* Only the target's BODY is replaced its title and slug live on the page
* record (not in the content), so they are untouched. The source page is not
* modified at all.
*/
async copyPageContent(sourcePageId: string, targetPageId: string) {
await this.ensureAuthenticated();
// A self-copy would be a no-op overwrite; reject it explicitly so a caller
// mistake surfaces as a clear error rather than a silent round-trip.
if (sourcePageId === targetPageId) {
throw new Error(
"copyPageContent: sourcePageId and targetPageId are the same page (no-op copy)",
);
}
const source = await this.getPageRaw(sourcePageId);
const content = source?.content;
if (
!content ||
typeof content !== "object" ||
content.type !== "doc" ||
!Array.isArray(content.content)
) {
throw new Error(
`copyPageContent: source page ${sourcePageId} has no usable ProseMirror content to copy`,
);
}
// Defense-in-depth: run the same URL-scheme sanitizer the JSON write path
// uses, so copying never lands a javascript:/data: href/src on the target
// (parity with updatePageJson; harmless for already-stored source content).
this.validateDocUrls(content);
// Defense-in-depth (#228): this is a FULL-document write, so canonicalize
// footnotes before copying — a no-op on already-canonical source content, but
// it guarantees a copy can never propagate a non-canonical footnote topology
// to the target (parity with the other full-doc write paths).
// #419: normalize + merge glyph-forked definitions before canonicalizing.
const canonical = canonicalizeFootnotes(normalizeAndMergeFootnotes(content));
const collabToken = await this.getCollabTokenWithReauth();
// Open the TARGET collab doc by its canonical UUID, never the slugId (#260).
const targetUuid = await this.resolvePageId(targetPageId);
const mutation = await this.replacePage(
targetUuid,
canonical,
collabToken,
this.apiUrl,
);
return {
success: true,
sourcePageId,
targetPageId,
copiedNodes: canonical.content.length,
verify: mutation.verify,
};
}
/**
* Surgical text edits: find/replace inside text nodes of the live
* document. Preserves all block ids, marks, callouts and tables.
*/
// --- Page history / diff / transform ---
/**
* List the saved versions (history snapshots) of a page, newest first.
* Docmost auto-snapshots on every save. Returns one cursor-paginated page of
* results: `{ items, nextCursor }`. The history record's id field is `id`.
*/
async listPageHistory(pageId: string, cursor?: string) {
await this.ensureAuthenticated();
const payload: Record<string, any> = { pageId };
if (cursor) payload.cursor = cursor;
const response = await this.client.post("/pages/history", payload);
const data = response.data?.data ?? response.data;
return {
items: data?.items ?? [],
nextCursor: data?.meta?.nextCursor ?? null,
};
}
/**
* Fetch a single page-history version including its lossless ProseMirror
* `content`. The version also carries pageId/title/createdAt.
*/
async getPageHistory(historyId: string) {
await this.ensureAuthenticated();
const response = await this.client.post("/pages/history/info", {
historyId,
});
return response.data?.data ?? response.data;
}
/**
* "Restore" a version: Docmost has NO restore endpoint, so we take the
* version's `content` and write it as the page's current content via the live
* collab path (which itself creates a new history snapshot). Returns the
* affected pageId and the source historyId.
*/
async restorePageVersion(historyId: string) {
await this.ensureAuthenticated();
const version = await this.getPageHistory(historyId);
if (
!version ||
!version.pageId ||
!version.content ||
typeof version.content !== "object"
) {
throw new Error(
`restorePageVersion: history ${historyId} has no usable content`,
);
}
// Defense-in-depth: sanitize URLs in the restored content (parity with the
// JSON write path) before writing it back.
this.validateDocUrls(version.content);
const collabToken = await this.getCollabTokenWithReauth();
// version.pageId is the page entity id (already a UUID); resolvePageId
// short-circuits a UUID with no round-trip, so this is defensive only (#260).
const pageUuid = await this.resolvePageId(version.pageId);
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
() => version.content,
);
return {
pageId: version.pageId,
restoredFrom: historyId,
verify: mutation.verify,
};
}
/**
* Diff two versions of a page and return a Docmost-equivalent change set.
* `from`/`to` each resolve to a ProseMirror doc:
* - null / undefined / "current" -> the page's CURRENT content;
* - any other string -> that historyId's content.
* Returns the diff plus the resolved version metadata for each side.
*/
async diffPageVersions(pageId: string, from?: string, to?: string) {
await this.ensureAuthenticated();
const isCurrent = (v?: string) => v == null || v === "" || v === "current";
const resolveSide = async (
v?: string,
): Promise<{ doc: any; meta: any }> => {
if (isCurrent(v)) {
const raw = await this.getPageRaw(pageId);
return {
doc: raw.content || { type: "doc", content: [] },
meta: {
kind: "current",
pageId,
title: raw.title,
updatedAt: raw.updatedAt,
},
};
}
const version = await this.getPageHistory(v as string);
return {
doc: version.content || { type: "doc", content: [] },
meta: {
kind: "history",
historyId: version.id,
pageId: version.pageId,
title: version.title,
createdAt: version.createdAt,
},
};
};
const fromSide = await resolveSide(from);
const toSide = await resolveSide(to);
const diff = diffDocs(fromSide.doc, toSide.doc);
return { from: fromSide.meta, to: toSide.meta, diff };
}
/**
* Edit a page by running an arbitrary user-supplied JS transform against the
* live document, with a diff preview + page-history safety net.
*
* The transform string is evaluated as `(doc, ctx) => doc` inside a node:vm
* sandbox: it gets ONLY `{ doc, ctx, structuredClone, console }` as globals,
* a 5s timeout, and NO access to require/process/fs/network. It must return a
* `{ type: "doc" }` node, which is validated structurally before any write.
*
* `ctx` exposes:
* - comments: the page's comments (fetched before the live read);
* - log: an array the transform can push diagnostics to (via console.log);
* - consume(id): mark a comment id as consumed (for deleteComments);
* - helpers: the transforms.ts primitives + commentsToFootnotes.
*
* Footnote convention used by the helpers: footnote markers are plain "[N]"
* text in the body, and the notes are an orderedList under a heading whose
* text is "Примечания переводчика".
*
* dryRun (default true): read the page's current content, run the transform,
* and return `{ pushed:false, diff, log }` WITHOUT opening the collab socket.
* Otherwise the transform runs atomically inside mutatePageContent, optionally
* deletes consumed comments, and returns the new historyId + diff + log.
*/
}
return PagesMixin;
}
+651
View File
@@ -0,0 +1,651 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import axios, { AxiosInstance } from "axios";
import {
filterWorkspace,
filterSpace,
filterPage,
filterComment,
filterSearchResult,
} from "../lib/filters.js";
import { convertProseMirrorToMarkdown } from "../lib/markdown-converter.js";
import {
collectInternalFileNodes,
normalizeFileUrl,
resolveInternalFilePath,
} from "../lib/internal-file-urls.js";
import { buildPageTree } from "../lib/tree.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import {
importMarkdownFragment,
canBeDocChild,
findUnrepresentableTableAttrs,
} from "../lib/markdown-fragment.js";
import { searchInDoc, SearchOptions } from "../lib/page-search.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of ReadMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IReadMixin` fails to compile on drift.
export interface IReadMixin {
getWorkspace(): any;
getSpaces(): any;
listPages(spaceId?: string, limit?: number, tree?: boolean): any;
getTree(spaceId: string, rootPageId?: string, maxDepth?: number): any;
getPageContext(pageId: string): any;
listSidebarPages(spaceId: string, pageId?: string): any;
getPage(pageId: string): any;
getPageJson(pageId: string): any;
getOutline(pageId: string): any;
getNode(pageId: string, nodeId: string, format?: "markdown" | "json"): any;
searchInPage(pageId: string, query: string, opts?: SearchOptions): any;
getTable(pageId: string, tableRef: string): any;
search(query: string, spaceId?: string, limit?: number, opts?: { parentPageId?: string; titleOnly?: boolean }): any;
}
export function ReadMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IReadMixin> & TBase {
abstract class ReadMixin extends Base implements IReadMixin {
async getWorkspace() {
await this.ensureAuthenticated();
const response = await this.client.post("/workspace/info", {});
return {
data: filterWorkspace(response.data?.data ?? response.data),
success: response.data.success,
};
}
async getSpaces() {
const spaces = await this.paginateAll("/spaces", {});
return spaces.map((space) => filterSpace(space));
}
/**
* List pages in one of two modes.
*
* Default (`tree` false): most recent pages by updatedAt (descending),
* bounded. Fetching the whole space can exceed MCP response/time limits on
* large instances, so a single bounded page of results is returned (default
* 50, max 100) via the `/pages/recent` feed.
*
* Tree (`tree` true): DEPRECATED prefer `getTree`, which shares this exact
* code path (a single `/pages/tree` request via `enumerateSpacePages` +
* `buildPageTree`) but returns the compact `{pageId, title, children?,
* hasChildren?}` shape and supports `rootPageId`/`maxDepth`. This tree mode is
* kept for backward compatibility; it REQUIRES `spaceId` (a page tree is
* scoped to one space) and IGNORES `limit` the whole hierarchy is returned.
* It fetches the tree via `enumerateSpacePages`, which on the fork server
* resolves to a single `/pages/tree` request returning the whole
* permission-filtered flat page set (soft-deleted pages excluded
* server-side); the cursor-BFS in `enumerateSpacePages` is only a fallback for
* stock upstream servers that lack `/pages/tree`.
*/
async listPages(spaceId?: string, limit: number = 50, tree: boolean = false) {
await this.ensureAuthenticated();
if (tree) {
if (!spaceId) {
throw new Error(
"listPages: tree mode requires a spaceId (a page tree is scoped to one space). Pass spaceId, or omit tree to get the recent-pages list.",
);
}
const { pages } = await this.enumerateSpacePages(spaceId);
return buildPageTree(pages);
}
const clampedLimit = Math.max(1, Math.min(100, limit));
const payload: Record<string, any> = { limit: clampedLimit, page: 1 };
if (spaceId) payload.spaceId = spaceId;
const response = await this.client.post("/pages/recent", payload);
const data = response.data;
const items = data.data?.items || data.items || [];
return items.map((page: any) => filterPage(page));
}
/**
* Fetch a space's page hierarchy (or one subtree) as a nested tree in a SINGLE
* request the #443 `getTree` tool. Shares its whole code path with
* `listPages(tree:true)`: `enumerateSpacePages` issues one `POST /pages/tree`
* (with the cursor-BFS only as a fallback for stock upstream servers that lack
* the endpoint), then `buildPageTree` nests the flat, permission-filtered,
* position-ordered list. No second tree fetch, no per-node BFS.
*
* - `rootPageId` restrict to that page's subtree; the server seeds the CTE
* with the page itself, so the result is exactly ONE root (the page and its
* descendants). Omit it for the whole space.
* - `maxDepth` trim the response to that many levels (roots = depth 1) to
* save tokens; the server still returns everything in one request, the cut
* is applied in `buildPageTree` AFTER the full tree is built. A node whose
* children were cut carries `hasChildren: true` (source of truth = the flat
* item's server `hasChildren`) so the caller can descend with a follow-up
* `getTree(spaceId, rootPageId=that node)` call.
*
* Output nodes are `{pageId, title, children?, hasChildren?}` only the UUID
* `pageId` is exposed (never `slugId`/`icon`/`position`). Requires `spaceId`
* (a page tree is scoped to one space).
*/
async getTree(spaceId: string, rootPageId?: string, maxDepth?: number) {
await this.ensureAuthenticated();
if (!spaceId) {
throw new Error(
"getTree: spaceId is required (a page tree is scoped to one space).",
);
}
const { pages } = await this.enumerateSpacePages(spaceId, rootPageId);
return buildPageTree(pages, { shape: "getTree", maxDepth });
}
/**
* "Where am I / what's around" for a single page the #443 `getPageContext`
* tool. Metadata only (no page content), using exactly TWO server requests:
*
* 1. `POST /pages/breadcrumbs` a recursive CTE that walks UP from the page.
* The server returns the chain root->page order (it `.reverse()`s the
* child-first walk before responding), INCLUDING the page itself as the
* LAST element. So the last element is the page and everything before it
* is the ancestor chain root->parent. This carries the page's own title
* and spaceId, so no extra page-info fetch is needed for a UUID input.
* 2. `listSidebarPages(spaceId, pageId)` the page's DIRECT children,
* cursor-paginated (a page with >20 children returns ALL of them, no
* dupes) and in sidebar `position` order, each carrying `hasChildren`.
*
* The input may be a slugId (agents copy them from URLs); it is run through
* `resolvePageId` first, exactly like the other page tools. A UUID input adds
* no request there (short-circuit), keeping the total at two; a slugId input
* adds one unavoidable resolve round-trip.
*
* INVARIANT: only the UUID `pageId` is exposed anywhere server `id` is
* mapped to `pageId` and `slugId` is never leaked. A nonexistent/inaccessible
* pageId makes the server 404/403, which propagates as a clear tool error
* (never a hollow empty object).
*/
async getPageContext(pageId: string) {
await this.ensureAuthenticated();
// Resolve a possibly-slugId input to the canonical UUID (no round-trip for a
// UUID). Errors here (bad/inaccessible id) propagate as a clear tool error.
const pageUuid = await this.resolvePageId(pageId);
// Request 1: the ancestor chain, root->page, page included as the LAST item.
const response = await this.client.post("/pages/breadcrumbs", {
pageId: pageUuid,
});
const chain: any[] = (response.data?.data ?? response.data) ?? [];
if (!Array.isArray(chain) || chain.length === 0) {
// The endpoint always includes the page itself, so an empty chain means
// the page is gone/inaccessible — surface a clear error, not {}.
throw new Error(`getPageContext: page "${pageId}" not found or inaccessible`);
}
// Split: the last element is the page, the rest (root->parent) are the
// breadcrumbs. A root page has no ancestors -> breadcrumbs is [].
const self = chain[chain.length - 1];
const ancestors = chain.slice(0, -1);
const page = {
pageId: self.id,
title: self.title,
spaceId: self.spaceId,
};
const breadcrumbs = ancestors.map((n: any) => ({
pageId: n.id,
title: n.title,
}));
// Request 2: direct children in sidebar order, each with hasChildren.
const childItems = await this.listSidebarPages(self.spaceId, pageUuid);
const children = childItems.map((c: any) => ({
pageId: c.id,
title: c.title,
hasChildren: Boolean(c.hasChildren),
}));
return { page, breadcrumbs, children };
}
/**
* List sidebar pages for a space. With no pageId the request returns the
* space ROOT pages; with a pageId it returns the direct CHILDREN of that
* page. pageId is therefore optional and is only included in the POST body
* when provided (an empty/undefined pageId would otherwise change the
* semantics on the server).
*/
async listSidebarPages(spaceId: string, pageId?: string) {
await this.ensureAuthenticated();
// Paginate via the server-issued cursor. The server switched from OFFSET
// (`page`) to CURSOR (`cursor`/`nextCursor`) pagination, and the global
// ValidationPipe(whitelist:true) SILENTLY STRIPS the obsolete `page` field
// — so the old offset loop got the SAME first page every time (with
// hasNextPage stuck true) and dropped every child beyond the first page.
const MAX_PAGES = 50;
let cursor: string | undefined;
let allItems: any[] = [];
let truncated = false;
for (let i = 0; i < MAX_PAGES; i++) {
// limit: 100 is the server-side Max; cuts request count 5x vs the default 20.
const payload: Record<string, any> = { spaceId, limit: 100 };
// Only send pageId when scoping to a page's children; omit it for roots.
if (pageId) payload.pageId = pageId;
if (cursor) payload.cursor = cursor;
const data = (await this.client.post("/pages/sidebar-pages", payload)).data
?.data;
allItems = allItems.concat(data?.items ?? []);
// Advance strictly via the server-issued cursor; a missing/repeated cursor
// means the protocol drifted again — stop instead of looping on page one.
const next = data?.meta?.hasNextPage ? data?.meta?.nextCursor : null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with more pages still available means the child
// list is truncated (mirrors paginateAll).
if (i === MAX_PAGES - 1) truncated = true;
}
// Warn on real truncation (ceiling hit while the server still had pages) so
// the caller is not silently handed an incomplete child list.
if (truncated) {
console.warn(
`listSidebarPages: children of "${pageId ?? spaceId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
return allItems;
}
/**
* Enumerate EVERY page in a space (or in a subtree, when rootPageId is given).
*
* Primary path (fork server): a SINGLE `POST /pages/tree` returns the whole
* space (or a subtree) as a flat, permission-filtered list in one request, in
* the exact node shape buildPageTree consumes. This replaces the old
* per-node BFS, which issued N sidebar requests and after the server moved
* to cursor pagination silently lost every child past the first sidebar
* page (the obsolete `page` param was stripped by ValidationPipe).
*
* The subtree variant (rootPageId given) INCLUDES the root node itself
* (getPageAndDescendants seeds with id = rootPageId), unlike the old BFS
* which started from the root's children.
*
* Fallback path (stdio mode may target STOCK upstream Docmost, which lacks
* `/pages/tree`): on a 404/405 it falls back to the cursor-based BFS below,
* walking direct children via the fixed cursor listSidebarPages. Safeguards:
* a `visited` Set of page ids prevents re-processing a node (cycles /
* duplicate references), and a hard node cap bounds pathological trees so the
* walk always terminates.
*
* Returns `{ pages, truncated }`. `truncated` is true ONLY when the fallback
* BFS stopped at its MAX_NODES cap the primary /pages/tree path is uncapped
* and always returns the complete set, so it never reports truncation.
*/
protected async enumerateSpacePages(
spaceId: string,
rootPageId?: string,
): Promise<{ pages: any[]; truncated: boolean }> {
await this.ensureAuthenticated();
// Single request replaces the whole BFS: /pages/tree returns the full
// permission-filtered flat page set of a space (or a subtree) at once. This
// path is uncapped, so it is never truncated.
const payload = rootPageId ? { pageId: rootPageId } : { spaceId };
try {
const response = await this.client.post("/pages/tree", payload);
const pages = (response.data?.data ?? response.data)?.items ?? [];
return { pages, truncated: false };
} catch (e: any) {
// Only fall back when the endpoint is absent (stock upstream Docmost);
// any other error is a genuine failure and must propagate.
if (
!axios.isAxiosError(e) ||
(e.response?.status !== 404 && e.response?.status !== 405)
) {
throw e;
}
}
// Fallback: cursor-based breadth-first walk via listSidebarPages.
const MAX_NODES = 10000;
const result: any[] = [];
const visited = new Set<string>();
// Seed with the root node itself when scoping to a subtree, so its own
// comments aren't dropped: the primary /pages/tree seeds
// getPageAndDescendants with id = rootPageId (root included), but
// listSidebarPages(spaceId, rootPageId) returns only the root's CHILDREN.
// The `visited` set below prevents a double-add if the root also appears
// among the children. getPageRaw returns a page whose id/title/spaceId are
// exactly what buildPageTree and checkNewComments consume.
if (rootPageId) {
try {
const root = await this.getPageRaw(rootPageId);
if (root?.id) {
result.push(root);
visited.add(root.id);
}
} catch {
// Non-fatal: if the root can't be read, fall through to children-only.
}
}
// Seed the queue with the starting level (subtree children or roots).
const queue: any[] = await this.listSidebarPages(spaceId, rootPageId);
while (queue.length > 0 && result.length < MAX_NODES) {
const node = queue.shift();
if (!node || typeof node !== "object" || !node.id) continue;
// Skip already-seen ids to guard against cycles / duplicate references.
if (visited.has(node.id)) continue;
visited.add(node.id);
result.push(node);
if (node.hasChildren) {
try {
const children = await this.listSidebarPages(spaceId, node.id);
for (const child of children) queue.push(child);
} catch (e: any) {
// A failure fetching one node's children must not abort the whole
// walk: skip this branch and keep enumerating the rest.
}
}
}
// Truncated only when the cap was hit with the queue still non-empty (real
// truncation, not a natural end at exactly MAX_NODES).
return {
pages: result,
truncated: result.length >= MAX_NODES && queue.length > 0,
};
}
/** Raw page info including the ProseMirror JSON content and slugId. */
async getPage(pageId: string) {
await this.ensureAuthenticated();
const resultData = await this.getPageRaw(pageId);
// Agent read: hide resolved-comment anchors so the agent sees only active
// discussions. Active anchors are kept. (The lossless exportPageMarkdown
// round-trip deliberately does NOT pass this flag — resolved anchors there
// must be preserved.)
let content = resultData.content
? convertProseMirrorToMarkdown(resultData.content, {
dropResolvedCommentAnchors: true,
})
: "";
// Always fetch subpages to provide context to the agent
let subpages: any[] = [];
try {
// `pageId` may be a slugId, but the sidebar-pages endpoint requires the
// UUID; `resultData.id` holds the resolved UUID returned by getPageRaw.
subpages = await this.listSidebarPages(resultData.spaceId, resultData.id);
} catch (e: any) {
console.warn("Failed to fetch subpages:", e);
}
// Resolve subpages if the placeholder exists
if (content && content.includes("{{SUBPAGES}}")) {
if (subpages && subpages.length > 0) {
const list = subpages
.map((p: any) => `- [${p.title}](page:${p.id})`)
.join("\n");
content = content.replace("{{SUBPAGES}}", `### Subpages\n${list}`);
} else {
content = content.replace("{{SUBPAGES}}", "");
}
}
return {
data: filterPage(resultData, content, subpages),
success: true,
};
}
/** Page info + raw ProseMirror JSON content (lossless representation). */
async getPageJson(pageId: string) {
const data = await this.getPageRaw(pageId);
return {
id: data.id,
slugId: data.slugId,
title: data.title,
parentPageId: data.parentPageId,
spaceId: data.spaceId,
updatedAt: data.updatedAt,
content: data.content || { type: "doc", content: [] },
};
}
/**
* Fetch an INTERNAL Docmost file (authed loopback) for sandbox mirroring.
* `src` is normalized to `/api/files/<id>/<file>`; `this.client.baseURL`
* already ends in `/api`, so we strip the leading `/api` and request the
* relative path with the client's Authorization header. Returns the raw bytes
* and the response Content-Type (mime), defaulting to octet-stream.
*
* The fetch is size-bounded (hard 64 MiB ceiling) purely to protect memory;
* the authoritative per-blob cap is enforced by the sandbox `put`. The path is
* resolved via resolveInternalFilePath, which REJECTS (throws) any traversal
* or percent-encoded src that would let an attacker-controlled `attrs.src`
* escape `/api/files/` and reach another internal endpoint (SSRF). That throw
* happens before this.client.get, so a malicious src is counted as a failed
* mirror it never reaches the network.
*/
/**
* Compact outline of a page's top-level blocks (no full document body).
* Cheap way to locate sections/tables and grab block ids before drilling in
* with getNode / patchNode / insertNode.
*/
async getOutline(pageId: string) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
return {
pageId,
slugId: data.slugId,
title: data.title,
outline: buildOutline(data.content ?? { type: "doc", content: [] }),
};
}
/**
* Fetch a single block for editing by reference: a block id (headings/
* paragraphs/callouts/images), or `#<index>` to select a top-level block by its
* outline index (the only way to reach tables/rows/cells, which carry no id).
*
* `format` (#413):
* - `"markdown"` (DEFAULT): serialize the block via the canonical converter
* (`{type:"doc",content:[node]}` -> `convertProseMirrorToMarkdown`) a read
* "for editing": pair it with `patchNode({markdown})` to rewrite the block.
* Comment anchors (`<span data-comment-id>`, INCLUDING resolved ones) are
* NOT stripped here (unlike getPage): losing them on write-back would
* orphan the thread. Returns `{ ..., format:"markdown", markdown }`.
* - `"json"`: return the raw ProseMirror subtree as-is (lossless; the previous
* default). Returns `{ ..., format:"json", node }`.
*
* AUTO fallback: a type that cannot be a document top-level child
* (tableRow/tableCell/tableHeader, addressed by `#<index>`) is NOT expressible
* as a standalone markdown document, so a `"markdown"` request for such a node
* transparently falls back to JSON with an explicit `format:"json"` field. The
* check derives from the schema's `doc` contentMatch, so it tracks the schema.
*/
async getNode(
pageId: string,
nodeId: string,
format: "markdown" | "json" = "markdown",
) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
const hit = getNodeByRef(
data.content ?? { type: "doc", content: [] },
nodeId,
);
if (!hit) {
throw new Error(
`getNode: no node found for "${nodeId}" on page ${pageId} (use a block id from getOutline, or "#<index>" for a top-level block such as a table)`,
);
}
// JSON requested (or a non-top-level type that markdown cannot represent as a
// standalone document): return the subtree verbatim.
if (format === "json" || !canBeDocChild(hit.type)) {
return {
pageId,
ref: nodeId,
path: hit.path,
type: hit.type,
format: "json" as const,
node: hit.node,
};
}
// Markdown: wrap the node as a one-block doc and run the canonical converter.
// Comment anchors are DELIBERATELY preserved (converter default) so a
// getNode(markdown) -> edit -> patchNode(markdown) round trip does not orphan
// a comment thread; this differs from getPage, which strips them.
const markdown = convertProseMirrorToMarkdown({
type: "doc",
content: [hit.node],
});
return {
pageId,
ref: nodeId,
path: hit.path,
type: hit.type,
format: "markdown" as const,
markdown,
};
}
/**
* Find every occurrence of `query` on a page IN MEMORY, over the plain text of
* each text container (reusing the same `getPageRaw` fetch as the other read
* tools) no server search endpoint, no whole-document round-trip through the
* model. Returns `{ total, truncated, matches }`; each match carries a ref for
* getNode/patchNode (the `#<index>` form resolves with getNode but NOT
* patchNode see SearchMatch.nodeId), plus the top-level block index and a
* short context window used to build a unique text `selection` for
* createComment (createComment has no nodeId param). The pure engine
* (`searchInDoc`) owns the traversal, glue, the RE2 ReDoS-safe regex engine
* and the empty-query / invalid-or-unsupported-regex errors.
*/
async searchInPage(pageId: string, query: string, opts: SearchOptions = {}) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
const result = searchInDoc(
data.content ?? { type: "doc", content: [] },
query,
opts,
);
return { pageId, query, ...result };
}
/**
* Read a table as a matrix. `tableRef` is `#<index>` (from getOutline) or a
* block id of any node inside the table. Returns the cell texts plus a
* parallel cellIds matrix (each cell's first paragraph id, or null) so a
* caller can patchNode a cell for rich-formatted edits. Throws when no table
* resolves for the reference.
*/
async getTable(pageId: string, tableRef: string) {
await this.ensureAuthenticated();
const data = await this.getPageRaw(pageId);
const t = readTable(data.content ?? { type: "doc", content: [] }, tableRef);
if (!t) {
throw new Error(
`tableGet: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
pageId,
table: tableRef,
rows: t.rows,
cols: t.cols,
path: t.path,
cells: t.cells,
cellIds: t.cellIds,
};
}
/**
* Insert a row of plain-text cells into a table on the LIVE collab document.
* `tableRef` is `#<index>` or a block id inside the target table. `cells` is
* padded to the table's column count (more cells than columns throws); `index`
* is a 0-based insert position (omit/out-of-range to append). Throws when no
* table resolves for the reference.
*/
async search(
query: string,
spaceId?: string,
limit?: number,
opts: { parentPageId?: string; titleOnly?: boolean } = {},
) {
await this.ensureAuthenticated();
// Opt into the #443 agent-lookup mode: `substring: true` turns on the hybrid
// substring + FTS branch that returns path + snippet + score. A stock
// upstream server strips these unknown DTO fields (whitelist:true) and
// silently degrades to plain FTS — see the tool-registration comment.
const payload: Record<string, any> = {
query,
spaceId,
substring: true,
};
if (opts.parentPageId) payload.parentPageId = opts.parentPageId;
if (opts.titleOnly) payload.titleOnly = true;
// Clamp an optional caller-supplied limit into the lookup range (1..50)
// before forwarding; omit it when not provided so the server default applies.
if (limit !== undefined) {
payload.limit = Math.max(1, Math.min(50, limit));
}
const response = await this.client.post("/search", payload);
// Normalize both response shapes: bare array and paginated { items: [...] }
const data = response.data?.data;
const items = Array.isArray(data) ? data : data?.items || [];
const filteredItems = items.map((item: any) => filterSearchResult(item));
return {
items: filteredItems,
success: response.data?.success || false,
};
}
}
return ReadMixin;
}
+225
View File
@@ -0,0 +1,225 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
collectInternalFileNodes,
normalizeFileUrl,
resolveInternalFilePath,
} from "../lib/internal-file-urls.js";
// Public method surface of StashMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements IStashMixin` fails to compile on drift.
export interface IStashMixin {
stashPage(pageId: string): Promise<{ uri: string; sha256: string; size: number; images: { mirrored: number; failed: number }; }>;
}
export function StashMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & IStashMixin> & TBase {
abstract class StashMixin extends Base implements IStashMixin {
/**
* Fetch an INTERNAL Docmost file (authed loopback) for sandbox mirroring.
* `src` is normalized to `/api/files/<id>/<file>`; `this.client.baseURL`
* already ends in `/api`, so we strip the leading `/api` and request the
* relative path with the client's Authorization header. Returns the raw bytes
* and the response Content-Type (mime), defaulting to octet-stream.
*
* The fetch is size-bounded (hard 64 MiB ceiling) purely to protect memory;
* the authoritative per-blob cap is enforced by the sandbox `put`. The path is
* resolved via resolveInternalFilePath, which REJECTS (throws) any traversal
* or percent-encoded src that would let an attacker-controlled `attrs.src`
* escape `/api/files/` and reach another internal endpoint (SSRF). That throw
* happens before this.client.get, so a malicious src is counted as a failed
* mirror it never reaches the network.
*/
protected async fetchInternalFile(
src: string,
): Promise<{ buffer: Buffer; mime: string }> {
const HARD_CEILING = 64 * 1024 * 1024; // 64 MiB memory guard
const relPath = resolveInternalFilePath(src);
const response = await this.client.get(relPath, {
responseType: "arraybuffer",
timeout: 30000,
maxContentLength: HARD_CEILING,
maxBodyLength: HARD_CEILING,
});
const buffer = Buffer.from(response.data);
if (buffer.length === 0) {
throw new Error(`Empty file response from "${src}"`);
}
const rawCt = response.headers?.["content-type"];
const mime =
typeof rawCt === "string" && rawCt.length > 0
? rawCt.split(";")[0].trim().toLowerCase()
: "application/octet-stream";
return { buffer, mime };
}
/**
* Stash a page's full content into the in-RAM blob sandbox and return ONLY a
* short anonymous URL the body never enters the model context (this is the
* whole point: ~30KB+ ProseMirror docs blow the model context if passed as a
* tool argument). Every INTERNAL file/image src (the type-agnostic criterion,
* so drawio/excalidraw/video/file nodes are covered too) is mirrored into the
* sandbox and its `src` rewritten to the sandbox URL, so an external consumer
* can fetch the images anonymously. External http(s) srcs are left untouched.
*
* Blobs live in RAM with a short TTL and are cleared on restart consume the
* URLs within the TTL and one uptime. A failed image fetch never aborts the
* doc: the original src is kept and the failure counted.
*
* Returns { uri, sha256, size, images:{mirrored, failed} }. `uri` and `sha256`
* are for the document blob; `sha256` is also the blob's ETag (integrity).
*/
async stashPage(pageId: string): Promise<{
uri: string;
sha256: string;
size: number;
images: { mirrored: number; failed: number };
}> {
if (!this.sandboxPut) {
throw new Error(
"stashPage is unavailable: the blob sandbox is not configured on this server",
);
}
await this.ensureAuthenticated();
// Stash the SAME shape getPageJson returns (id/title/.../content), with a
// deep clone so the rewrite never mutates anything shared.
const pageJson = await this.getPageJson(pageId);
const cloned: any = structuredClone(pageJson);
// Group internal-file nodes by normalized src so each unique resource is
// fetched + stored ONCE (dedup), and every node sharing that src points at
// the one sandbox blob. Capture each node's ORIGINAL raw src per-node:
// dedup groups nodes whose normalized src is equal even when their raw srcs
// differ (e.g. `/api/files/...` vs the bare `/files/...`), so on a revert we
// must restore each node's own original value, not the group key.
const bySrc = new Map<string, Array<{ node: any; origSrc: string }>>();
for (const node of collectInternalFileNodes(cloned.content)) {
const origSrc = String(node.attrs.src);
const src = normalizeFileUrl(origSrc);
const entry = { node, origSrc };
const group = bySrc.get(src);
if (group) group.push(entry);
else bySrc.set(src, [entry]);
}
let mirrored = 0;
let failed = 0;
// Record every successful mirror so it can be (a) reverted if its blob gets
// FIFO-evicted by a LATER put in this same stash, and (b) freed if the final
// doc put throws.
const mirrors: Array<{
uri: string;
entries: Array<{ node: any; origSrc: string }>;
}> = [];
const MAX_CONCURRENCY = 5;
const groups = [...bySrc.entries()];
for (let i = 0; i < groups.length; i += MAX_CONCURRENCY) {
const batch = groups.slice(i, i + MAX_CONCURRENCY);
await Promise.all(
batch.map(async ([src, entries]) => {
try {
const { buffer, mime } = await this.fetchInternalFile(src);
// put may throw if the blob exceeds the per-blob/total caps.
const stored = this.sandboxPut!(buffer, mime);
for (const entry of entries) entry.node.attrs.src = stored.uri;
mirrors.push({ uri: stored.uri, entries });
mirrored++;
} catch (err) {
// One bad/oversized image (or a rejected traversal src) must not
// abort the document. Logged unconditionally (never the blob body),
// matching the package's ungated console.warn convention.
failed++;
console.warn(
`stashPage: failed to mirror "${src}": ${
err instanceof Error ? err.message : String(err)
}`,
);
}
}),
);
}
// Revert one mirror's nodes to their original internal srcs and re-count it
// as failed (its blob was FIFO-evicted before the doc could reference it
// safely).
const revertMirror = (mirror: {
uri: string;
entries: Array<{ node: any; origSrc: string }>;
}) => {
for (const entry of mirror.entries) entry.node.attrs.src = entry.origSrc;
mirrored--;
failed++;
console.warn(
`stashPage: mirrored blob ${mirror.uri} was evicted before the doc ` +
`could safely reference it; reverted its src and counted it as failed`,
);
};
// Pre-put reconciliation: an image put earlier in THIS stash can FIFO-evict
// an even-earlier image of the same stash. Drop those from the live set
// first so the first serialized doc is already mostly correct.
let liveMirrors = mirrors;
if (this.sandboxHas) {
liveMirrors = [];
for (const mirror of mirrors) {
if (this.sandboxHas(mirror.uri)) liveMirrors.push(mirror);
else revertMirror(mirror);
}
}
// Put the document, then reconcile against eviction caused by the doc put
// ITSELF (the doc is newest, FIFO drops oldest = this stash's images). Each
// iteration reverts >=1 mirror, so the loop terminates (worst case: all
// images reverted and the doc references no sandbox image URLs).
let stored: { uri: string; sha256: string; size: number };
for (;;) {
const docBuf = Buffer.from(JSON.stringify(cloned), "utf8");
let docStored: { uri: string; sha256: string; size: number };
try {
docStored = this.sandboxPut(docBuf, "application/json");
} catch (err) {
// The doc put failed (e.g. doc exceeds the cap). Free this op's image
// blobs instead of leaking them in RAM for the whole TTL, then
// re-throw.
if (this.sandboxEvict) {
for (const mirror of liveMirrors) this.sandboxEvict(mirror.uri);
}
throw err;
}
if (!this.sandboxHas) {
stored = docStored;
break;
}
const evictedNow = liveMirrors.filter((m) => !this.sandboxHas!(m.uri));
if (evictedNow.length === 0) {
stored = docStored;
break;
}
// The doc we just stored references now-dead blobs. Revert those nodes,
// drop the stale doc blob, and loop to re-serialize + re-put the
// corrected doc.
for (const mirror of evictedNow) revertMirror(mirror);
liveMirrors = liveMirrors.filter((m) => this.sandboxHas!(m.uri));
if (this.sandboxEvict) this.sandboxEvict(docStored.uri);
}
return {
uri: stored.uri,
sha256: stored.sha256,
size: stored.size,
images: { mirrored, failed },
};
}
/**
* Compact outline of a page's top-level blocks (no full document body).
* Cheap way to locate sections/tables and grab block ids before drilling in
* with getNode / patchNode / insertNode.
*/
}
return StashMixin;
}
+291
View File
@@ -0,0 +1,291 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import {
replaceNodeById,
replaceNodeByIdWithMany,
reassignCollidingBlockIds,
deleteNodeById,
assertUnambiguousMatch,
insertNodeRelative,
insertNodesRelative,
blockPlainText,
buildOutline,
getNodeByRef,
readTable,
insertTableRow,
deleteTableRow,
updateTableCell,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import { withPageLock, isUuid } from "../lib/page-lock.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
// Public method surface of TablesMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements ITablesMixin` fails to compile on drift.
export interface ITablesMixin {
insertFootnote(pageId: string, anchorText: string, text: string): any;
tableInsertRow(pageId: string, tableRef: string, cells: string[], index?: number): any;
tableDeleteRow(pageId: string, tableRef: string, index: number): any;
tableUpdateCell(pageId: string, tableRef: string, row: number, col: number, text: string): any;
}
export function TablesMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & ITablesMixin> & TBase {
abstract class TablesMixin extends Base implements ITablesMixin {
/**
* AUTHOR-INLINE footnote insertion. The agent supplies only WHERE
* (`anchorText`, a snippet of body text to attach the marker after) and WHAT
* (`text`, the footnote content as markdown). Numbering and the bottom
* `footnotesList` are derived deterministically server-side
* (`insertInlineFootnote` -> `canonicalizeFootnotes`): the agent never sees,
* assigns, or edits a footnote number or the list, so it CANNOT desync.
*
* Content DEDUP: when an existing definition has the same content, its id is
* reused (one number, one definition, several references). The write is atomic
* via `mutatePageContent` (single-writer, page-locked); if the anchor text is
* not found the transform aborts with a clear error and no write happens.
*/
async insertFootnote(pageId: string, anchorText: string, text: string) {
await this.ensureAuthenticated();
if (!anchorText || !anchorText.trim()) {
throw new Error("insertFootnote: anchorText is required");
}
if (text == null || `${text}`.trim() === "") {
throw new Error("insertFootnote: text is required");
}
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let result: { footnoteId: string; reused: boolean } | null = null;
const mutation = await this.mutatePage(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc: any) => {
const r = insertInlineFootnote(liveDoc, { anchorText, text });
if (!r.inserted) {
// Abort the page-locked write by throwing: mutatePageContent does not
// persist when the transform throws, so a missing anchor leaves the
// page untouched (no partial write).
throw new Error(
`insertFootnote: anchor text not found: ${JSON.stringify(
anchorText.slice(0, 80),
)}`,
);
}
result = { footnoteId: r.footnoteId, reused: r.reused };
return r.doc;
},
);
// The not-found path throws inside the transform (aborting mutatePage), so by
// here `result` is always set.
const r = result!;
return {
success: true,
modified: true,
pageId,
footnoteId: r.footnoteId,
reused: r.reused,
message: r.reused
? "Footnote inserted (reused an existing same-content definition)."
: "Footnote inserted.",
verify: mutation.verify,
};
}
/**
* Page-locked write seam over collaboration.mutatePageContent. Production just
* delegates; it exists as an overridable method so the insertFootnote wrapper
* (transform abort-on-not-found + response shaping) can be unit-tested without
* standing up a live Hocuspocus collab socket.
*
* SELF-RESOLVES the pageId to the canonical UUID (issue #449, "resolve-then-
* lock"): every write must lock and key its CollabSession by the UUID, never a
* raw slugId (#260). resolvePageId is cached/idempotent, so a caller that
* already resolved pays no extra round-trip; centralizing it here means a
* caller that reaches this seam with a raw slugId still locks correctly instead
* of silently splitting the mutex key. withPageLock also asserts the key is a
* UUID as a hard backstop.
*/
/**
* Insert a row of plain-text cells into a table on the LIVE collab document.
* `tableRef` is `#<index>` or a block id inside the target table. `cells` is
* padded to the table's column count (more cells than columns throws); `index`
* is a 0-based insert position (omit/out-of-range to append). Throws when no
* table resolves for the reference.
*/
async tableInsertRow(
pageId: string,
tableRef: string,
cells: string[],
index?: number,
) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
// Track insertion in an outer var, reset per-transform, so a collab retry
// recomputes it cleanly (mirrors insertNode's pattern).
let inserted = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
inserted = false;
const { doc: nd, inserted: ins } = insertTableRow(
liveDoc,
tableRef,
cells,
index,
);
inserted = ins;
if (!inserted) return null; // table not found -> skip the write entirely
return nd;
},
);
if (!inserted) {
throw new Error(
`tableInsertRow: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
success: true,
table: tableRef,
inserted: true,
verify: mutation.verify,
};
}
/**
* Delete the row at 0-based `index` from a table on the LIVE collab document.
* `tableRef` is `#<index>` or a block id inside the target table. The helper's
* out-of-range and last-row errors propagate; a missing table throws here.
*/
async tableDeleteRow(pageId: string, tableRef: string, index: number) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let deleted = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
deleted = false;
const { doc: nd, deleted: del } = deleteTableRow(
liveDoc,
tableRef,
index,
);
deleted = del;
if (!deleted) return null; // table not found -> skip the write entirely
return nd;
},
);
if (!deleted) {
throw new Error(
`tableDeleteRow: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
success: true,
table: tableRef,
deleted: true,
verify: mutation.verify,
};
}
/**
* Set the plain-text content of cell `[row, col]` (0-based) in a table on the
* LIVE collab document, replacing the cell's content with a single text
* paragraph (the cell's first-paragraph id is preserved). `tableRef` is
* `#<index>` or a block id inside the target table. The helper's out-of-range
* error propagates; a missing table throws here.
*/
async tableUpdateCell(
pageId: string,
tableRef: string,
row: number,
col: number,
text: string,
) {
await this.ensureAuthenticated();
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
let updated = false;
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
(liveDoc) => {
updated = false;
const { doc: nd, updated: upd } = updateTableCell(
liveDoc,
tableRef,
row,
col,
text,
);
updated = upd;
if (!updated) return null; // table not found -> skip the write entirely
return nd;
},
);
if (!updated) {
throw new Error(
`tableUpdateCell: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from getOutline, or a block id inside the table)`,
);
}
return {
success: true,
table: tableRef,
row,
col,
verify: mutation.verify,
};
}
/**
* Create a new page with title and content.
* Uses the /pages/import workaround (the only endpoint accepting content),
* then moves the page and restores the exact title: the import endpoint
* derives the title from the FILENAME and replaces spaces with
* underscores, so we explicitly re-set it via /pages/update afterwards.
*/
}
return TablesMixin;
}
+232
View File
@@ -0,0 +1,232 @@
// Auto-split from client.ts (issue #450). Mixin over the shared client context.
// Bodies are VERBATIM from the original DocmostClient; only the enclosing class
// changed to a mixin factory. See client/context.ts for the shared base.
import type { GConstructor, DocmostClientContext } from "./context.js";
import {
updatePageContentRealtime,
replacePageContent,
markdownToProseMirror,
markdownToProseMirrorCanonical,
mutatePageContent,
assertYjsEncodable,
MutationResult,
} from "../lib/collaboration.js";
import { diffDocs, summarizeChange } from "../lib/diff.js";
import {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
mergeFootnoteDefinitions,
} from "../lib/transforms.js";
import { normalizeAndMergeFootnotes } from "../lib/footnote-normalize-merge.js";
import vm from "node:vm";
// Public method surface of TransformsMixin (issue #450) — a NAMED type so the factory
// return type is expressible in the emitted .d.ts (the anonymous mixin class
// carries the base's protected shared state, which would otherwise trip TS4094).
// Derived from the class below; `implements ITransformsMixin` fails to compile on drift.
export interface ITransformsMixin {
transformPage(pageId: string, transformJs: string, opts?: { dryRun?: boolean; deleteComments?: boolean }): any;
}
export function TransformsMixin<TBase extends GConstructor<DocmostClientContext>>(Base: TBase): GConstructor<DocmostClientContext & ITransformsMixin> & TBase {
abstract class TransformsMixin extends Base implements ITransformsMixin {
/**
* Edit a page by running an arbitrary user-supplied JS transform against the
* live document, with a diff preview + page-history safety net.
*
* The transform string is evaluated as `(doc, ctx) => doc` inside a node:vm
* sandbox: it gets ONLY `{ doc, ctx, structuredClone, console }` as globals,
* a 5s timeout, and NO access to require/process/fs/network. It must return a
* `{ type: "doc" }` node, which is validated structurally before any write.
*
* `ctx` exposes:
* - comments: the page's comments (fetched before the live read);
* - log: an array the transform can push diagnostics to (via console.log);
* - consume(id): mark a comment id as consumed (for deleteComments);
* - helpers: the transforms.ts primitives + commentsToFootnotes.
*
* Footnote convention used by the helpers: footnote markers are plain "[N]"
* text in the body, and the notes are an orderedList under a heading whose
* text is "Примечания переводчика".
*
* dryRun (default true): read the page's current content, run the transform,
* and return `{ pushed:false, diff, log }` WITHOUT opening the collab socket.
* Otherwise the transform runs atomically inside mutatePageContent, optionally
* deletes consumed comments, and returns the new historyId + diff + log.
*/
async transformPage(
pageId: string,
transformJs: string,
opts: { dryRun?: boolean; deleteComments?: boolean } = {},
) {
const dryRun = opts.dryRun ?? true;
const deleteComments = opts.deleteComments ?? false;
await this.ensureAuthenticated();
// Full feed (incl. resolved): a page transform (e.g. comments -> footnotes)
// must operate on every comment, so it opts into the unfiltered feed.
const comments = (await this.listComments(pageId, true)).items;
// ctx handed to the sandbox. consume() records ids; helpers are the pure
// transform primitives. log is captured from console.log inside the sandbox.
const ctx = {
comments,
log: [] as string[],
consumed: new Set<string>(),
consume(id: string) {
this.consumed.add(id);
},
helpers: {
blockText,
walk,
getList,
insertMarkerAfter,
setCalloutRange,
noteItem,
mdToInlineNodes,
commentsToFootnotes,
canonicalizeFootnotes,
insertInlineFootnote,
},
};
// Captured oldDoc / newDoc for the diff (set inside runTransform).
let oldDoc: any;
let newDoc: any;
// SYNCHRONOUS transform runner — safe to call inside mutatePageContent's
// onSynced (no await between the live read and the write).
const runTransform = (liveDoc: any): any => {
oldDoc = structuredClone(liveDoc);
const sandbox: Record<string, any> = {
doc: structuredClone(liveDoc),
ctx,
structuredClone,
console: {
log: (...a: any[]) => ctx.log.push(a.map((x) => String(x)).join(" ")),
},
};
// Wrap the provided string in parentheses so both an expression-arrow
// (`(doc, ctx) => {...}`) and a parenthesized function work. Run it in a
// fresh context with no require/process/module so the transform cannot
// touch fs/network/process. 5s wall-clock timeout.
let fn: any;
try {
fn = vm.runInNewContext("(" + transformJs + ")", sandbox, {
timeout: 5000,
});
} catch (e: any) {
throw new Error(`transform did not compile: ${e?.message ?? e}`);
}
if (typeof fn !== "function") {
throw new Error(
"transform must evaluate to a function (doc, ctx) => doc",
);
}
const raw = vm.runInNewContext(
"f(d, c)",
{ f: fn, d: sandbox.doc, c: ctx },
{ timeout: 5000 },
);
if (
!raw ||
typeof raw !== "object" ||
raw.type !== "doc" ||
!Array.isArray(raw.content)
) {
throw new Error(
'transform must return a ProseMirror doc node ({ type:"doc", content:[...] })',
);
}
// Validate the RAW transform output FIRST (structure — including the
// MAX_DEPTH guard — and URLs), mirroring updatePageJson. The canonicalizer
// recurses without a depth limiter, so validating after it would turn a
// too-deep doc into an opaque "Maximum call stack size exceeded" instead of
// the intended "nesting exceeds the maximum depth" error.
this.validateDocStructure(raw);
this.validateDocUrls(raw);
// Auto-canonicalize footnotes after the transform (idempotent): no write
// path can leave footnotes out of order / orphaned / in a raw `[^id]`
// block. In a dryRun preview this may surface footnote edits the script
// author did not write (the canonicalizer tidied them) — that is expected.
// #419: normalize + merge glyph-forked definitions before canonicalizing.
const result = canonicalizeFootnotes(normalizeAndMergeFootnotes(raw));
newDoc = result;
return result;
};
if (dryRun) {
// Preview only: run against the current REST snapshot, never open the
// socket. oldDoc/newDoc are captured by runTransform.
const raw = await this.getPageRaw(pageId);
const current = raw.content || { type: "doc", content: [] };
runTransform(current);
// Run an independent Yjs-encodability check (same sanitize + schema as the
// apply path), so the preview fails with the same descriptive error when
// the doc is not encodable instead of returning a misleadingly-green diff.
assertYjsEncodable(newDoc);
return {
pushed: false,
diff: diffDocs(oldDoc, newDoc),
log: ctx.log,
};
}
// Apply atomically against the live doc.
const collabToken = await this.getCollabTokenWithReauth();
// Open the collab doc by the canonical UUID, never the slugId (#260).
const pageUuid = await this.resolvePageId(pageId);
const mutation = await mutatePageContent(
pageUuid,
collabToken,
this.apiUrl,
runTransform,
);
// Optionally delete consumed comments (best-effort; a delete failure must
// not undo the successful write).
const deletedComments: string[] = [];
if (deleteComments) {
for (const id of ctx.consumed) {
try {
await this.deleteComment(id);
deletedComments.push(id);
} catch (e) {
if (process.env.DEBUG) {
console.error(`transform: failed to delete comment ${id}:`, e);
}
}
}
}
// Fetch the newest historyId (Docmost snapshots on the write above).
let historyId: string | null = null;
try {
const hist = await this.listPageHistory(pageId);
historyId = hist.items?.[0]?.id ?? null;
} catch (e) {
if (process.env.DEBUG) {
console.error("transform: failed to fetch history id:", e);
}
}
return {
pushed: true,
historyId,
diff: diffDocs(oldDoc, newDoc),
deletedComments,
log: ctx.log,
verify: mutation.verify,
};
}
}
return TransformsMixin;
}
+4 -7
View File
@@ -57,17 +57,14 @@ export const DEFAULT_COMMENT_SIGNAL_DEBOUNCE_MS = 20_000;
/**
* Tools whose OWN result must NOT carry the signal it would be tautological
* (the agent is already looking at comments) and noisy. Listed in BOTH the
* standalone MCP snake_case names AND the in-app camelCase keys so a single set
* covers both surfaces (the signal text itself uses the camelCase `listComments`
* per roadmap #412). `getComment` (single fetch) is intentionally NOT excluded.
* (the agent is already looking at comments) and noisy. Since issue #412 both
* the standalone MCP surface and the in-app agent use the same camelCase tool
* names, so a single set of camelCase names covers both surfaces. `getComment`
* (single fetch) is intentionally NOT excluded.
*/
export const COMMENT_SIGNAL_EXCLUDED_TOOLS: ReadonlySet<string> = new Set([
"list_comments",
"listComments",
"check_new_comments",
"checkNewComments",
"create_comment",
"createComment",
]);
+172 -563
View File
@@ -5,7 +5,10 @@ import { fileURLToPath } from "url";
import { dirname, join } from "path";
import { DocmostClient, DocmostMcpConfig } from "./client.js";
import { parseNodeArg } from "@docmost/prosemirror-markdown";
import { searchShapes } from "./lib/drawio-shapes.js";
import { getGuideSection } from "./lib/drawio-guide.js";
import { SHARED_TOOL_SPECS, SharedToolSpec } from "./tool-specs.js";
import { SERVER_INSTRUCTIONS } from "./server-instructions.js";
import {
createCommentSignalTracker,
CommentSignalTracker,
@@ -29,6 +32,14 @@ export { destroyAllSessions } from "./lib/collab-session.js";
export { SHARED_TOOL_SPECS } from "./tool-specs.js";
export type { SharedToolSpec } from "./tool-specs.js";
// Re-export the build-time REGISTRY_STAMP (issue #447): a deterministic hash of
// the tool-specs registry content, generated into src/registry-stamp.generated.ts
// by scripts/gen-registry-stamp.mjs BEFORE tsc, so it lands in build/. The in-app
// loader recomputes the same hash from src/tool-specs.ts (dev/test only) and
// refuses to run on a mismatch, catching a build/ vs src/ skew (a spec edited in
// src without rebuilding the package the server actually loads from build/).
export { REGISTRY_STAMP } from "./registry-stamp.generated.js";
// Re-export the shared "new comments: N" signal helper (#417) so the in-app
// layer reads the SAME watermark/debounce/injection-safe line builder off the
// loaded module (same pattern as SHARED_TOOL_SPECS). Both surfaces then differ
@@ -46,6 +57,13 @@ export type {
CommentSignalProbeResult,
CommentSignalTrackerOptions,
} from "./comment-signal.js";
// Re-export the pure, no-network draw.io helpers (#424) so the in-app AI-SDK
// service can wire drawioShapes / drawioGuide off the loaded module. These are
// NOT client methods (no page/backend hit) — the in-app handler calls them
// directly, mirroring how the standalone MCP server wires them here.
export { searchShapes } from "./lib/drawio-shapes.js";
export type { SearchShapesOptions } from "./lib/drawio-shapes.js";
export { getGuideSection } from "./lib/drawio-guide.js";
// Read version from package.json
const __filename = fileURLToPath(import.meta.url);
@@ -66,19 +84,17 @@ const VERSION = packageJson.version;
// Editing guide surfaced to MCP clients in the initialize result so they can
// pick the right tool by intent and avoid resending whole documents.
//
// MAINTENANCE RULE: when you ADD, RENAME, or REMOVE a tool (either an inline
// server.registerTool(...) here or a spec in tool-specs.ts), you MUST update
// this guide so the new tool is routed by intent. This is enforced by
// test/unit/server-instructions.test.mjs, which fails when a registered tool
// name is not mentioned below (see its EXCEPTIONS list for the rare opt-outs).
// Exported for that test.
export const SERVER_INSTRUCTIONS =
"Docmost editing guide — choose the tool by intent.\n" +
"READ: find a page -> search (workspace-wide full-text); list -> list_pages / list_spaces. Locate blocks and their ids CHEAPLY -> get_outline (compact top-level map; start here, not get_page_json). One block's subtree -> get_node (by attrs.id, or \"#<index>\" for tables, which carry no id). Find every occurrence of a string/regex ON a page (and where each is) -> search_in_page, NOT block-by-block get_node — it returns each hit's node ref + block index + context for a targeted comment. Whole page -> get_page (Markdown, lossy; inline <span data-comment-id> tags are comment anchors — markup, not text) or get_page_json (lossless ProseMirror with block ids). Hand a huge page (with images) to an external consumer without pulling it through the model context -> stash_page (returns a short-lived anonymous URL).\n" +
"EDIT: fix wording/typos/numbers -> edit_page_text (find/replace inside blocks, no node id needed). Change ONE block (paragraph/heading/callout/etc.) structurally -> patch_node (by attrs.id from get_outline). Add a block -> insert_node (before/after a block by attrs.id or by anchor text, or append). Remove a block -> delete_node (by attrs.id). Tables -> table_get / table_update_cell / table_insert_row / table_delete_row (address by \"#<index>\" from get_outline; table nodes have no attrs.id). Images -> insert_image (add from a web URL) / replace_image (swap an existing image). Draw.io diagrams -> drawio_create (create from mxGraph XML and insert), drawio_get (read a diagram as mxGraph XML + a hash), drawio_update (replace a diagram; pass the hash from drawio_get as baseHash for optimistic locking). Footnotes -> insert_footnote. Bulk/structural rewrite -> update_page_json (full ProseMirror replace; prefer the granular tools above to avoid resending the whole ~100KB+ document). Complex/scripted rewrite (multiple coordinated edits, renumbering) -> docmost_transform: write a JS `(doc, ctx) => doc` transform, preview the diff with dryRun (default), then apply with dryRun:false; ctx.helpers includes commentsToFootnotes for turning inline comments into numbered footnotes.\n" +
"PAGES: new -> create_page (Markdown). Rename (title only) -> rename_page. Move -> move_page. Delete -> delete_page (SOFT delete — the page goes to trash and is restorable; nothing is permanent). Copy/replace a page's whole content from another page (server-side, no document through the model) -> copy_page_content. Sharing -> share_page / unshare_page / list_shares; share_page makes the page PUBLICLY accessible — do it only when explicitly asked.\n" +
"COMMENTS: create_comment is always inline and requires an EXACT selection — contiguous text from a single block, <=250 chars (fails rather than leaving an unanchored comment); reply to a thread via parentCommentId. Propose a concrete text fix for one-click human approval -> create_comment with suggestedText (the exact plain-text replacement for the selection; the selection must then be UNIQUE in the page — extend it with context if needed); prefer this over editing directly when the change is subjective or needs the author's sign-off. Manage -> list_comments, update_comment, resolve_comment (resolve/reopen, reversible — prefer over delete to close), delete_comment, check_new_comments.\n" +
"HISTORY: review what changed -> diff_page_versions (a historyId vs current, or two versions). List saved versions -> list_page_history. Undo a bad edit -> restore_page_version (writes a past version back as current; itself revertible). Lossless markdown round-trip (download, edit, re-upload, incl. comment anchors) -> export_page_markdown / import_page_markdown.";
// The guide is now SPLIT (issue #448): the hand-written routing prose lives in
// server-instructions.ts and the tool INVENTORY is GENERATED from the registry
// (SHARED_TOOL_SPECS + INLINE_MCP_INVENTORY), so it can no longer drift out of
// sync with the registered tools. Re-exported here (its old home) so existing
// importers are unaffected; the composition lives in server-instructions.ts.
// The drawioShapes / drawioGuide tools (#424) stay in SHARED_TOOL_SPECS (so the
// generated <tool_inventory> picks them up from their catalogLine automatically)
// but are flagged `inlineBothHosts` and registered inline below (their pure
// helpers can't cross into tool-specs.ts); only the hand-written routing prose in
// server-instructions.ts is updated to mention them.
export { SERVER_INSTRUCTIONS };
// Helper to format JSON responses
const jsonContent = (data: any) => ({
@@ -194,10 +210,10 @@ export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
{ instructions: SERVER_INSTRUCTIONS },
);
// Single choke point for MCP tool timing. Both `registerShared` (below) and
// the inline `server.registerTool(...)` calls funnel through this one method,
// so monkeypatching it HERE — before any tool is registered and before
// `registerShared` captures a reference to it — times every tool with no
// Single choke point for MCP tool timing. Both `registerSharedFromSpec` (below)
// and the inline `server.registerTool(...)` calls funnel through this one
// method, so monkeypatching it HERE — before any tool is registered and before
// the registry loop captures a reference to it — times every tool with no
// per-tool boilerplate. The wrapped handler records wall-clock duration and,
// in a `finally`, feeds the host's dependency-neutral sink
// `config.onMetric("mcp_tool_duration_seconds", seconds, { tool })`. The tool
@@ -253,104 +269,112 @@ export function createDocmostMcpServer(config: DocmostMcpConfig): McpServer {
return originalRegisterTool(...args.slice(0, -1), signalledHandler);
};
// Register a tool from the shared, zod-agnostic spec registry. The spec owns
// the canonical name + model-facing description + (optional) schema builder;
// only the execute body is supplied per call. buildShape is invoked with THIS
// package's zod (v3); the in-app layer passes its own zod (v4).
//
// The spec's schema builder returns a plain ZodRawShape (Record<string,
// unknown> in the shared module since it must stay zod-agnostic), so the
// McpServer.registerTool overloads cannot infer the execute arg's shape from
// it. We type `execute` loosely and cast the call through `any`; runtime
// behaviour is unchanged — each execute body destructures the same fields the
// builder declares.
const registerShared = (
spec: SharedToolSpec,
execute: (args: any) => Promise<{ content: { type: "text"; text: string }[] }>,
) =>
(server.registerTool as any)(
// Register EVERY shared tool from the zod-agnostic registry in one loop (#445).
// The spec owns the canonical name + description + (optional) schema builder AND
// the canonical execute mapping; the host only supplies the RESULT ENVELOPE. For
// each spec:
// - skip `inAppOnly` specs (they belong to the in-app host only);
// - if the spec has an `mcpExecute` override (a deliberate per-layer
// difference — a guardrail, an omitted param, or a non-JSON envelope like a
// resource_link/bare success line), the override OWNS the full MCP content
// result and is used VERBATIM;
// - otherwise the canonical `execute` returns RAW data and this host wraps it
// in the standard JSON text envelope (jsonContent), exactly as the old inline
// bodies did.
// buildShape is invoked with THIS package's zod (v3); the in-app layer passes its
// own zod (v4). The registry's execute returns `unknown` (it is zod-agnostic), so
// the wrapping is typed loosely and cast — runtime behaviour is unchanged.
const registerSharedFromSpec = (spec: SharedToolSpec) => {
if (spec.inAppOnly) return;
// `inlineBothHosts` specs (drawioShapes / drawioGuide) carry no execute —
// their pure helper cannot cross into the zod-agnostic tool-specs.ts, so they
// are registered INLINE below (searchShapes / getGuideSection). Skip them here
// so the loop never dereferences a missing `execute`.
if (spec.inlineBothHosts) return;
const handler = async (args: any) => {
if (spec.mcpExecute) {
// The override owns the full MCP result envelope (not re-wrapped).
return (await spec.mcpExecute(docmostClient, args)) as {
content: { type: "text"; text: string }[];
};
}
// Canonical execute returns raw data; wrap it as JSON text content.
const raw = await spec.execute!(docmostClient, args);
return jsonContent(raw);
};
return (server.registerTool as any)(
spec.mcpName,
spec.buildShape
? { description: spec.description, inputSchema: spec.buildShape(z) }
: { description: spec.description },
execute,
handler,
);
};
// Tool: get_workspace
registerShared(SHARED_TOOL_SPECS.getWorkspace, async () => {
const workspace = await docmostClient.getWorkspace();
return jsonContent(workspace);
});
for (const spec of Object.values(SHARED_TOOL_SPECS)) {
registerSharedFromSpec(spec as SharedToolSpec);
}
// Tool: list_spaces
registerShared(SHARED_TOOL_SPECS.listSpaces, async () => {
const spaces = await docmostClient.getSpaces();
return jsonContent(spaces);
});
// --- INLINE drawio helper tools (IN the shared registry, but inlineBothHosts) ---
// drawioShapes / drawioGuide (#424) live in SHARED_TOOL_SPECS (so the shared
// contract pins their name/description/schema across both hosts) but carry the
// `inlineBothHosts` flag and NO execute: their pure backing helpers
// (searchShapes / getGuideSection) cannot be value-imported into the
// zod-agnostic tool-specs.ts without breaking the in-app server's commonjs
// type-check (searchShapes' catalog loader uses import.meta). So both hosts wire
// them directly. Here on the MCP host they reuse the spec's name/description/
// schema and wrap the raw helper result as JSON text content — byte-identical to
// what the registry loop would have produced. The in-app host mirrors this in
// ai-chat-tools.service.ts.
{
// Cast registerTool like the loop's registerSharedFromSpec does: the spec's
// buildShape returns the loose zod-agnostic ZodRawShape (Record<string,
// unknown>) and the handler args are the SDK-validated, type-erased input.
const registerInline = server.registerTool as any;
const shapesSpec = SHARED_TOOL_SPECS.drawioShapes as SharedToolSpec;
registerInline(
shapesSpec.mcpName,
{
description: shapesSpec.description,
inputSchema: shapesSpec.buildShape!(z),
},
async ({ query, category, limit }: any) => {
const results = searchShapes(query, { category, limit });
return jsonContent({ query, count: results.length, results });
},
);
const guideSpec = SHARED_TOOL_SPECS.drawioGuide as SharedToolSpec;
registerInline(
guideSpec.mcpName,
{
description: guideSpec.description,
inputSchema: guideSpec.buildShape!(z),
},
async ({ section }: any) => jsonContent(getGuideSection(section)),
);
}
// Tool: list_pages
// INTENTIONAL per-transport divergence (not in the shared registry): this
// transport exposes a `tree:true` mode that returns the full nested hierarchy;
// the in-app copy keeps the same tree option but is worded for the in-app agent.
// Kept per-layer so each side can tune its own guidance.
// Schema + description now live in @docmost/mcp's SHARED_TOOL_SPECS (#294). This
// transport keeps applying its own defaults (limit=50, tree=false) in execute.
registerShared(SHARED_TOOL_SPECS.listPages, async ({ spaceId, limit, tree }) => {
const result = await docmostClient.listPages(spaceId, limit ?? 50, tree ?? false);
return jsonContent(result);
});
// --- INLINE tools kept per-transport (NOT in the shared registry) ---
// Each stays inline for a documented reason: a snake_case/camelCase naming
// clash the registry convention forbids (tableGet), an intentional
// per-transport behaviour/schema divergence (search, docmostTransform), or a
// tool that exists ONLY on this standalone MCP surface (updateComment,
// deleteComment — the in-app agent deliberately exposes no hard comment
// edit/delete tool).
// Tool: get_page
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.getPage, async ({ pageId }) => {
const page = await docmostClient.getPage(pageId);
return jsonContent(page);
});
// Tool: get_page_json
registerShared(SHARED_TOOL_SPECS.getPageJson, async ({ pageId }) => {
const page = await docmostClient.getPageJson(pageId);
return jsonContent(page);
});
// Tool: get_outline
registerShared(SHARED_TOOL_SPECS.getOutline, async ({ pageId }) => {
const result = await docmostClient.getOutline(pageId);
return jsonContent(result);
});
// Tool: get_node
registerShared(SHARED_TOOL_SPECS.getNode, async ({ pageId, nodeId }) => {
const result = await docmostClient.getNode(pageId, nodeId);
return jsonContent(result);
});
// Tool: search_in_page
registerShared(
SHARED_TOOL_SPECS.searchInPage,
async ({ pageId, query, regex, caseSensitive, limit }) => {
const result = await docmostClient.searchInPage(pageId, query, {
regex,
caseSensitive,
limit,
});
return jsonContent(result);
},
);
// Tool: table_get
// NOT in the shared registry: the MCP tool name `table_get` is noun-first while
// Tool: tableGet
// NOT in the shared registry: the MCP tool name `tableGet` is noun-first while
// the in-app key is `getTable` (verb-first), breaking the snake_case(inAppKey)
// convention the shared registry enforces (shared-tool-specs.contract.spec.ts).
// Renaming the public MCP tool would break external clients, so it stays inline.
server.registerTool(
"table_get",
"tableGet",
{
description:
"Read a table as a matrix. Returns {rows, cols, cells (text[][]), " +
"cellIds (paragraph id per cell, or null)}. `table` = `#<index>` from " +
"get_outline, or any block id inside the table. Use cellIds with " +
"patch_node for rich-formatted cell edits. `cols` is the FIRST row's " +
"getOutline, or any block id inside the table. Use cellIds with " +
"patchNode for rich-formatted cell edits. `cols` is the FIRST row's " +
"width; ragged tables may vary per row, so use the per-row length of " +
"`cells` for each row.",
inputSchema: {
@@ -364,386 +388,9 @@ server.registerTool(
},
);
// Tool: table_insert_row
// Schema + description now live in the shared registry (#294); the `table`
// parameter name is the canonical one (the in-app layer was unified to it).
registerShared(
SHARED_TOOL_SPECS.tableInsertRow,
async ({ pageId, table, cells, index }) => {
const result = await docmostClient.tableInsertRow(
pageId,
table,
cells,
index,
);
return jsonContent(result);
},
);
// Tool: table_delete_row
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.tableDeleteRow,
async ({ pageId, table, index }) => {
const result = await docmostClient.tableDeleteRow(pageId, table, index);
return jsonContent(result);
},
);
// Tool: table_update_cell
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.tableUpdateCell,
async ({ pageId, table, row, col, text }) => {
const result = await docmostClient.tableUpdateCell(
pageId,
table,
row,
col,
text,
);
return jsonContent(result);
},
);
// Tool: create_page
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.createPage,
async ({ title, content, spaceId, parentPageId }) => {
const result = await docmostClient.createPage(
title,
content,
spaceId,
parentPageId,
);
return jsonContent(result);
},
);
// Tool: update_page_json
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's content normalization (parse a JSON-string content,
// pass undefined/null through for a title-only/no-op update).
registerShared(
SHARED_TOOL_SPECS.updatePageJson,
async ({ pageId, content, title }) => {
// Only parse/validate the document when it was actually supplied; when it
// is omitted, pass it straight through so the client performs a title-only
// (or no-op) update.
let doc;
if (content === undefined || content === null) {
doc = undefined;
} else {
// String -> JSON.parse (throwing on invalid); object passes through.
doc = parseNodeArg(content, "content was a string but not valid JSON");
}
const result = await docmostClient.updatePageJson(pageId, doc, title);
return jsonContent(result);
},
);
// Tool: export_page_markdown
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.exportPageMarkdown, async ({ pageId }) => {
const md = await docmostClient.exportPageMarkdown(pageId);
return { content: [{ type: "text" as const, text: md }] };
});
// Tool: import_page_markdown
registerShared(
SHARED_TOOL_SPECS.importPageMarkdown,
async ({ pageId, markdown }) => {
const res = await docmostClient.importPageMarkdown(pageId, markdown);
return jsonContent(res);
},
);
// Tool: copy_page_content
registerShared(
SHARED_TOOL_SPECS.copyPageContent,
async ({ sourcePageId, targetPageId }) => {
const result = await docmostClient.copyPageContent(
sourcePageId,
targetPageId,
);
return jsonContent(result);
},
);
// Tool: rename_page
// Schema + description now live in the shared registry (#294).
registerShared(SHARED_TOOL_SPECS.renamePage, async ({ pageId, title }) => {
const result = await docmostClient.renamePage(pageId, title);
return jsonContent(result);
});
// Tool: edit_page_text
registerShared(SHARED_TOOL_SPECS.editPageText, async ({ pageId, edits }) => {
const result = await docmostClient.editPageText(pageId, edits);
return jsonContent(result);
});
// Tool: stash_page — returns a resource_link (NOT embedded text) so the doc
// body never enters the model context. Registered directly (not via
// registerShared) because that helper only emits text content. Also returns
// `structuredContent` carrying the full documented `{uri, sha256, size, images}`
// shape alongside the resource_link, so MCP clients receive the blob's sha256
// (its ETag, for integrity) and mirror counts, not just the link.
// Tool: updateComment
server.registerTool(
SHARED_TOOL_SPECS.stashPage.mcpName,
{
description: SHARED_TOOL_SPECS.stashPage.description,
inputSchema: SHARED_TOOL_SPECS.stashPage.buildShape!(z),
},
async ({ pageId }: { pageId: string }) => {
const result = await docmostClient.stashPage(pageId);
return {
content: [
{
type: "resource_link" as const,
uri: result.uri,
name: "page.json",
mimeType: "application/json",
size: result.size,
},
],
// Mirror the full documented result shape ({ uri, size, sha256, images })
// as structuredContent so MCP clients get the blob's sha256 (its ETag, for
// integrity) and the mirror counts, not just the resource_link.
structuredContent: {
uri: result.uri,
sha256: result.sha256,
size: result.size,
images: result.images,
},
};
},
);
// Tool: patch_node — schema + description from the shared registry (identical
// across both transports). The execute body keeps its own parseNodeArg
// normalization (the model sometimes serializes `node` as a JSON string).
registerShared(
SHARED_TOOL_SPECS.patchNode,
async ({ pageId, nodeId, node }) => {
const parsedNode = parseNodeArg(node);
const result = await docmostClient.patchNode(pageId, nodeId, parsedNode);
return jsonContent(result);
},
);
// Tool: insert_node — schema + description from the shared registry. As with
// patch_node, the execute body retains parseNodeArg on the incoming node.
registerShared(
SHARED_TOOL_SPECS.insertNode,
async ({ pageId, node, position, anchorNodeId, anchorText }) => {
const parsedNode = parseNodeArg(node);
const result = await docmostClient.insertNode(pageId, parsedNode, {
position,
anchorNodeId,
anchorText,
});
return jsonContent(result);
},
);
// Tool: delete_node
registerShared(SHARED_TOOL_SPECS.deleteNode, async ({ pageId, nodeId }) => {
const result = await docmostClient.deleteNode(pageId, nodeId);
return jsonContent(result);
});
// Tool: insert_image
// Schema + description now live in the shared registry (#410) so BOTH this MCP
// server and the in-app AI-chat agent expose it. The execute body is unchanged.
registerShared(
SHARED_TOOL_SPECS.insertImage,
async ({ pageId, imageUrl, align, alt, replaceText, afterText }) => {
const result = await docmostClient.insertImage(pageId, imageUrl, {
align,
alt,
replaceText,
afterText,
});
return jsonContent(result);
},
);
// Tool: replace_image
// Schema + description now live in the shared registry (#410).
registerShared(
SHARED_TOOL_SPECS.replaceImage,
async ({ pageId, attachmentId, imageUrl, align, alt }) => {
const result = await docmostClient.replaceImage(
pageId,
attachmentId,
imageUrl,
{
align,
alt,
},
);
return jsonContent(result);
},
);
// Tool: drawio_get — read a draw.io diagram as mxGraph XML (or the raw SVG).
registerShared(
SHARED_TOOL_SPECS.drawioGet,
async ({ pageId, node, format }) => {
const result = await docmostClient.drawioGet(pageId, node, format ?? "xml");
return jsonContent(result);
},
);
// Tool: drawio_create — lint mxGraph XML, build the .drawio.svg, insert a node.
registerShared(
SHARED_TOOL_SPECS.drawioCreate,
async ({ pageId, xml, position, anchorNodeId, anchorText, title }) => {
const result = await docmostClient.drawioCreate(
pageId,
{ position, anchorNodeId, anchorText },
xml,
title,
);
return jsonContent(result);
},
);
// Tool: drawio_update — optimistic-locked full replacement of a diagram.
registerShared(
SHARED_TOOL_SPECS.drawioUpdate,
async ({ pageId, node, xml, baseHash }) => {
const result = await docmostClient.drawioUpdate(pageId, node, xml, baseHash);
return jsonContent(result);
},
);
// Tool: share_page
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own `searchIndexing ?? true` default.
registerShared(
SHARED_TOOL_SPECS.sharePage,
async ({ pageId, searchIndexing }) => {
const result = await docmostClient.sharePage(pageId, searchIndexing ?? true);
return jsonContent(result);
},
);
// Tool: unshare_page
registerShared(SHARED_TOOL_SPECS.unsharePage, async ({ pageId }) => {
const result = await docmostClient.unsharePage(pageId);
return jsonContent(result);
});
// Tool: list_shares
registerShared(SHARED_TOOL_SPECS.listShares, async () => {
const result = await docmostClient.listShares();
return jsonContent(result);
});
// Tool: move_page
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's cycle guard, its 'null'/'' -> null string coercion, and
// its positive-confirmation check on the move response.
registerShared(
SHARED_TOOL_SPECS.movePage,
async ({ pageId, parentPageId, position }) => {
const finalParentId =
parentPageId === "" || parentPageId === "null" ? null : parentPageId;
// Cheap cycle guard: a page cannot be moved directly under itself.
// (Deeper descendant-cycle detection is intentionally out of scope.)
if (finalParentId !== null && finalParentId === pageId) {
throw new Error("cannot move a page under itself");
}
const result = await docmostClient.movePage(
pageId,
finalParentId || null,
position,
);
// Require POSITIVE confirmation: the live /pages/move success shape is
// exactly { success: true, status: 200 }. An empty body, a 204, or any odd
// shape lacking success === true must NOT be reported as a successful move,
// so we surface the raw API result instead of declaring success.
if (!(result && typeof result === "object" && result.success === true)) {
throw new Error(
`Failed to move page ${pageId}: ${JSON.stringify(result)}`,
);
}
return jsonContent({
message: `Successfully moved page ${pageId} to parent ${finalParentId || "root"}`,
result,
});
},
);
// Tool: delete_page
// Schema + description now live in the shared registry (#294). The shared schema
// exposes ONLY pageId, so no permanent/force-delete flag can reach the client.
registerShared(SHARED_TOOL_SPECS.deletePage, async ({ pageId }) => {
await docmostClient.deletePage(pageId);
return {
content: [
{ type: "text" as const, text: `Successfully deleted page ${pageId}` },
],
};
});
// --- Comment tools (ported from upstream PR #3 by Max Nikitin) ---
// Tool: list_comments
registerShared(
SHARED_TOOL_SPECS.listComments,
async ({ pageId, includeResolved }) => {
const comments = await docmostClient.listComments(pageId, includeResolved);
return jsonContent(comments);
},
);
// Tool: create_comment
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own guards (require a selection for a top-level
// comment; reject suggestedText on a reply / without a selection).
registerShared(
SHARED_TOOL_SPECS.createComment,
async ({ pageId, content, selection, parentCommentId, suggestedText }) => {
if (!parentCommentId && (!selection || !selection.trim())) {
throw new Error(
"create_comment: a 'selection' (exact text to anchor on) is required for a top-level comment; omit it only when replying via parentCommentId.",
);
}
if (suggestedText !== undefined) {
if (parentCommentId) {
throw new Error(
"create_comment: 'suggestedText' cannot be attached to a reply; it applies only to a top-level inline comment.",
);
}
if (!selection || !selection.trim()) {
throw new Error(
"create_comment: 'suggestedText' requires a 'selection' to anchor and rewrite.",
);
}
}
const result = await docmostClient.createComment(
pageId,
content,
"inline",
selection,
parentCommentId,
suggestedText,
);
return jsonContent(result);
},
);
// Tool: update_comment
server.registerTool(
"update_comment",
"updateComment",
{
description:
"Update an existing comment's content. Only the comment creator can " +
@@ -762,9 +409,9 @@ server.registerTool(
},
);
// Tool: delete_comment
// Tool: deleteComment
server.registerTool(
"delete_comment",
"deleteComment",
{
description:
"Delete a comment. Only the comment creator or space admin can delete it.",
@@ -785,77 +432,77 @@ server.registerTool(
},
);
// Tool: resolve_comment
// Schema + description now live in the shared registry (#294).
registerShared(
SHARED_TOOL_SPECS.resolveComment,
async ({ commentId, resolved }) => {
const result = await docmostClient.resolveComment(commentId, resolved);
return jsonContent(result);
},
);
// Tool: check_new_comments
// Schema + description now live in the shared registry (#294). The execute body
// keeps this transport's own guard rejecting an unparseable `since` timestamp.
registerShared(
SHARED_TOOL_SPECS.checkNewComments,
async ({ spaceId, since, parentPageId }) => {
// Reject an unparseable timestamp up front: otherwise the comparison
// against NaN silently treats every comment as "not new" and the tool
// returns zero results without signalling the bad input.
if (Number.isNaN(Date.parse(since))) {
throw new Error(
`Invalid 'since' timestamp: ${JSON.stringify(since)} — expected an ISO 8601 date (e.g. '2026-03-10T00:00:00Z')`,
);
}
const result = await docmostClient.checkNewComments(
spaceId,
since,
parentPageId,
);
return jsonContent(result);
},
);
// Tool: search
// INTENTIONAL per-transport divergence (not shared): the in-app `searchPages`
// runs a semantic + keyword hybrid (RRF) with in-process access control and a
// different schema (limit 1-20); this transport is a plain REST full-text search
// (limit up to 100). Different behaviour AND schema, so kept per-layer.
// different schema; this transport is the #443 agent-lookup search — a hybrid
// substring + full-text search that also returns each hit's location (`path`)
// and a windowed `snippet`, so one call answers "where is it and what's in it".
// The in-app hybrid-RRF search is deliberately NOT touched. Different behaviour
// AND schema, so kept per-layer.
//
// STANDALONE-vs-STOCK-UPSTREAM: the client sends the opt-in `substring`/
// `parentPageId`/`titleOnly` DTO fields. A stock upstream server validates the
// DTO with `whitelist: true` and silently strips these unknown fields, so the
// request degrades gracefully to plain FTS (no path/snippet, current shape).
//
// EE/TYPESENSE DEGRADATION (#443): on an instance whose SEARCH_DRIVER is
// `typesense`, the server routes this request to the Typesense backend, which
// does NOT implement agent-lookup — the substring/path/snippet/tiering is
// ignored and the response degrades to plain Typesense FTS. The rich lookup
// shape is only produced by the native Postgres search driver.
server.registerTool(
"search",
{
description:
"Full-text search for pages and content across the whole workspace. " +
"Results are bounded by `limit` (1-100; when omitted the server applies " +
"its own default).",
"Find pages by a fragment of a technical string (hostnames, IPs, IDs " +
"like `srv.local`, `10.0.12`, `WB-MGE-30D86B`) — one call returns each " +
"hit's location (`path`: ancestor titles root→parent) and a `snippet` " +
"around the first match, so you rarely need a follow-up get_page. " +
"Matches substrings literally (dots/dashes/digits are not tokenized) as " +
"well as full-text. Returns `{ pageId, title, path, snippet, score }` " +
"sorted by `score` (a per-response relevance float).",
inputSchema: {
query: z.string().min(1).describe("Search query"),
spaceId: z
.string()
.optional()
.describe("Restrict the search to a single space"),
parentPageId: z
.string()
.optional()
.describe(
"Restrict to a page and all its descendants (the page itself included)",
),
titleOnly: z
.boolean()
.optional()
.describe("Match page titles only; skip page text"),
limit: z
.number()
.int()
.min(1)
.max(100)
.max(50)
.optional()
.describe("Max results to return (max 100)"),
.describe("Max results to return (1-50, default 10)"),
},
},
async ({ query, limit }) => {
// The tool exposes no spaceId filter, so pass undefined for the client's
// optional spaceId parameter and forward limit into its correct slot.
const result = await docmostClient.search(query, undefined, limit);
async ({ query, spaceId, parentPageId, titleOnly, limit }) => {
const result = await docmostClient.search(query, spaceId, limit, {
parentPageId,
titleOnly,
});
return jsonContent(result);
},
);
// Tool: docmost_transform
// Tool: docmostTransform
// INTENTIONAL per-transport divergence (not shared): the in-app `transformPage`
// deliberately omits the `deleteComments` schema field (comment-deletion
// guardrail) and carries a much shorter description; this transport exposes the
// full helper catalogue. Different schema, so kept per-layer.
server.registerTool(
"docmost_transform",
"docmostTransform",
{
description:
"Edit a page by running an arbitrary JS transform `(doc, ctx) => doc` " +
@@ -927,43 +574,5 @@ server.registerTool(
},
);
// Tool: insert_footnote
// Schema + description now live in the shared registry (#410) so the in-app
// AI-chat agent exposes it too. The execute body is unchanged.
registerShared(
SHARED_TOOL_SPECS.insertFootnote,
async ({ pageId, anchorText, text }) => {
const result = await docmostClient.insertFootnote(pageId, anchorText, text);
return jsonContent(result);
},
);
// Tool: diff_page_versions
registerShared(
SHARED_TOOL_SPECS.diffPageVersions,
async ({ pageId, from, to }) => {
const result = await docmostClient.diffPageVersions(pageId, from, to);
return jsonContent(result);
},
);
// Tool: list_page_history
registerShared(
SHARED_TOOL_SPECS.listPageHistory,
async ({ pageId, cursor }) => {
const result = await docmostClient.listPageHistory(pageId, cursor);
return jsonContent(result);
},
);
// Tool: restore_page_version
registerShared(
SHARED_TOOL_SPECS.restorePageVersion,
async ({ historyId }) => {
const result = await docmostClient.restorePageVersion(historyId);
return jsonContent(result);
},
);
return server;
}
+29 -4
View File
@@ -202,6 +202,21 @@ export class CollabSession {
this.ydoc = new Y.Doc();
}
/**
* Shared diagnostic suffix (issue #437) appended to the connect-timeout,
* persist-timeout and connection-closed error texts: names the offending
* pageId and tells the agent this class of failure is transient (retry once)
* vs. a persistent collab-server outage, so it can self-correct instead of
* blind-looping. The Yjs-encode error is deliberately NOT touched it
* already names the offending attribute.
*/
private hint(): string {
return (
`(pageId ${this.pageId}; transient — retry once; persistent failures ` +
`mean the collab server is unreachable/overloaded)`
);
}
/**
* A cached session may be reused only when it is fully ready, still synced,
* has not lost its connection, and has not exceeded its max age (invariant 5
@@ -232,7 +247,9 @@ export class CollabSession {
// The 25s connect timeout: the collab connection never became ready.
this.opts?.onConnectTimeout?.();
this.teardown(
new Error("Connection timeout to collaboration server"),
new Error(
`Connection timeout to collaboration server ${this.hint()}`,
),
false,
);
}, CONNECT_TIMEOUT_MS);
@@ -259,7 +276,7 @@ export class CollabSession {
if (process.env.DEBUG) console.error("WS Disconnect");
this.teardown(
new Error(
"Collaboration connection closed before the update was persisted/synced",
`Collaboration connection closed before the update was persisted/synced ${this.hint()}`,
),
true,
);
@@ -268,7 +285,7 @@ export class CollabSession {
if (process.env.DEBUG) console.error("WS Close");
this.teardown(
new Error(
"Collaboration connection closed before the update was persisted/synced",
`Collaboration connection closed before the update was persisted/synced ${this.hint()}`,
),
true,
);
@@ -403,7 +420,7 @@ export class CollabSession {
persistTimer = setTimeout(() => {
localFinish(
new Error(
"Timeout waiting for collaboration server to persist the update",
`Timeout waiting for collaboration server to persist the update ${this.hint()}`,
),
);
}, PERSIST_TIMEOUT_MS);
@@ -423,9 +440,16 @@ export class CollabSession {
// must stay synchronous (no await). While the JS event loop is not
// yielded, no incoming remote update can interleave, so any already-synced
// concurrent edits are preserved in liveDoc.
//
// INVARIANT 1 is machine-checked: the BEGIN/END markers below delimit the
// no-await window, and test/unit/no-await-critical-window.test.mjs scans
// this source and FAILS if any `await` (or `for await`/`yield`) appears
// between them. Do NOT add an await inside this block — an accidental
// async boundary here silently reopens the clobber-live-edits race (#152).
let newDoc: any;
let beforeDoc: any;
try {
// === MUTATE-CRITICAL-WINDOW: BEGIN (no await between here and END #449) ===
let liveDoc = TiptapTransformer.fromYdoc(this.ydoc, "default");
if (
!liveDoc ||
@@ -463,6 +487,7 @@ export class CollabSession {
// ids of unchanged nodes, so an open editor's cursor is not yanked to the
// end of the document on every agent write.
applyDocToFragment(this.ydoc, newDoc);
// === MUTATE-CRITICAL-WINDOW: END (#449) ===
} catch (e) {
// Includes errors thrown by transform (e.g. "afterText not found",
// "text not found"): propagate them verbatim to the caller.
+23 -5
View File
@@ -13,7 +13,11 @@ import { JSDOM } from "jsdom";
import { markdownToProseMirror } from "@docmost/prosemirror-markdown";
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import { sanitizeForYjs, findUnstorableAttr } from "@docmost/prosemirror-markdown";
import {
sanitizeForYjs,
findUnstorableAttr,
findInvalidNode,
} from "@docmost/prosemirror-markdown";
import { canonicalizeFootnotes } from "./footnote-canonicalize.js";
import { normalizeAndMergeFootnotes } from "./footnote-normalize-merge.js";
import { VerifyReport } from "./diff.js";
@@ -28,11 +32,25 @@ export { markdownToProseMirror };
* place. `label` names the stage that failed (diagnostic). `sanitizeForYjs`
* already stripped `undefined` attrs, so a remaining failure is pinpointed via
* `findUnstorableAttr`.
*
* Diagnostics precedence (#409): the dominant crash here is
* `Unknown node type: undefined` a nested node with an absent/unknown `type`
* (a SHAPE problem, e.g. `{"text":"foo"}` missing `"type":"text"`). That points
* at the node, not an attribute, so `findInvalidNode` is consulted FIRST and,
* on a hit, yields a path-anchored node-shape message. Only when the document
* shape is sound do we fall back to `findUnstorableAttr` (undefined/function/
* symbol/bigint attr values); the generic "attribute likely holds a value Yjs
* cannot store" sentence is the last resort.
*/
function unstorableYjsError(safe: any, label: string, e: unknown): Error {
const base = `Failed to encode document to Yjs (${label}): ${e instanceof Error ? e.message : String(e)}.`;
const badNode = findInvalidNode(safe);
if (badNode) {
return new Error(`${base} Invalid node: ${badNode.summary}`);
}
const bad = findUnstorableAttr(safe);
return new Error(
`Failed to encode document to Yjs (${label}): ${e instanceof Error ? e.message : String(e)}.${bad ? ` Offending attribute: ${bad}.` : " A node/mark attribute likely holds a value Yjs cannot store (e.g. undefined)."}`,
`${base}${bad ? ` Offending attribute: ${bad}.` : " A node/mark attribute likely holds a value Yjs cannot store (e.g. undefined)."}`,
);
}
@@ -69,8 +87,8 @@ global.WebSocket = WebSocket;
* bodies merged. So the import output is ALREADY in canonical footnote
* topology.
* - `canonicalizeFootnotes` runs AFTER as the mcp write-path invariant shared
* with every other full-document persist path (`update_page_json`,
* `docmost_transform`, `insert_footnote`, ). Because the package output is
* with every other full-document persist path (`updatePageJson`,
* `docmostTransform`, `insertFootnote`, ). Because the package output is
* already canonical, this layer is a no-op here (idempotent) it exists so
* the page-write contract is enforced uniformly regardless of how the PM doc
* was produced, not because the import needs fixing.
@@ -264,7 +282,7 @@ export async function mutatePageContent(
* it was produced from markdown (ids regenerate) or edited in place
* (existing block ids preserved).
*
* This is an intentional full replace (used by update_page / update_page_json),
* This is an intentional full replace (used by update_page / updatePageJson),
* but now runs under the per-page lock and waits for server persistence via
* mutatePageContent.
*/
+1 -1
View File
@@ -20,7 +20,7 @@
*
* MARKDOWN-STRIP FALLBACK: when the agent copies a selection that still carries
* inline markdown (`**bold**`, `` `code` ``, `[t](u)`), the raw locator will not
* match the document's plain text. Exactly like edit_page_text's json-edit
* match the document's plain text. Exactly like editPageText's json-edit
* fallback, we first try the verbatim selection and, ONLY if it anchors nowhere
* in the whole document, retry with `stripInlineMarkdown` applied. `canAnchorInDoc`,
* `getAnchoredText` and `applyAnchorInDoc` share this decision via
+163 -53
View File
@@ -14,7 +14,19 @@
* signature.
*
* If recreateTransform / the changeset throws on a pathological document pair,
* we fall back to a coarse block-level text diff so the tool never hard-fails.
* OR the pair is too large to diff cheaply (see the size guard below), we fall
* back to a coarse block-level text diff so the tool never hard-fails and never
* pins the event loop.
*
* SIZE GUARD (issue #464 prod CPU-DoS). recreateTransform computes its diff via
* rfc6902.createPatch, whose array diff is O(n·m) Levenshtein per array pair and
* whose per-run word diff is O(w²); on a large/heavily-changed doc this runs for
* seconds-to-hours and starves the whole process (BullMQ, Redis lock renewals,
* embeddings). It never THROWS it just never finishes so the try/catch below
* cannot save us. Because diffDocs runs on EVERY in-app/MCP content edit's verify
* report, we PRE-FLIGHT the doc size and route anything above a cheap cap straight
* to the coarse fallback (the same shape the catch produces). Same cap+fallback
* pattern as the ELK-layout DoS fix (#440 / c917dcc3).
*/
import { Node } from "@tiptap/pm/model";
@@ -72,6 +84,56 @@ function countNodes(doc: any, pred: (node: any) => boolean): number {
return n;
}
// --- Issue #464: pre-flight size guard for the precise diff ------------------
// Defaults are BENCHMARK-derived on the recreateTransform(complexSteps:false,
// wordDiffs:true, simplifyDiff:true) pipeline, chosen so the WORST case (a fully
// re-written doc — the adversarial shape that drove the incident) keeps the
// synchronous block under ~200ms REGARDLESS of input:
// - 150 total nodes: worst-case pair ~176ms; the O(node²) array diff crosses
// 200ms at ~170 nodes and then explodes super-linearly (400 nodes ~1.3s,
// 800 ~5.5s), so cap just below the crossover.
// - 12 KiB serialized JSON: an independent axis, because the per-run word diff
// is O(words²) — a FEW nodes with very long text runs is dangerous even at a
// low node count (17 nodes / ~11 KiB ~176ms, / ~14 KiB ~290ms). A node-light
// but byte-heavy doc is still refused.
// Either metric over its cap routes to the coarse fallback. Both are env-tunable
// for operators who accept more CPU in exchange for exact diffs on larger docs.
const DEFAULT_MAX_NODES = 150;
const DEFAULT_MAX_BYTES = 12 * 1024;
/**
* Read a positive-integer env override, falling back to `dflt`. Garbage / unset /
* non-finite / non-positive all fall back (so the guard can never be accidentally
* disabled by a malformed value). Read fresh on every call so a test / operator
* can flip the knob without a restart.
*/
function readPositiveIntEnv(name: string, dflt: number): number {
const raw = parseInt(process.env[name] ?? "", 10);
return Number.isFinite(raw) && raw > 0 ? raw : dflt;
}
/**
* True when the pair is too large for the precise (recreateTransform) diff and
* must degrade to the coarse fallback. Takes the MAX of the two docs on each
* metric so an ASYMMETRIC pair (a small new doc vs a huge old doc, or vice
* versa) which still explodes rfc6902 is caught. Cheap: one node walk +
* one JSON.stringify per doc, both O(size).
*/
function exceedsDiffSizeGuard(oldDoc: any, newDoc: any): boolean {
const maxNodes = readPositiveIntEnv("MCP_DIFF_MAX_NODES", DEFAULT_MAX_NODES);
const maxBytes = readPositiveIntEnv("MCP_DIFF_MAX_BYTES", DEFAULT_MAX_BYTES);
const nodes = Math.max(
countNodes(oldDoc, () => true),
countNodes(newDoc, () => true),
);
if (nodes > maxNodes) return true;
const bytes = Math.max(
JSON.stringify(oldDoc)?.length ?? 0,
JSON.stringify(newDoc)?.length ?? 0,
);
return bytes > maxBytes;
}
/**
* Count UNIQUE links in a JSON doc by their `href`. A single link can be split
* across several adjacent text runs (e.g. a "link+bold" run followed by a "link"
@@ -226,6 +288,81 @@ function coarseDiff(oldDoc: any, newDoc: any): DiffChange[] {
return changes;
}
/** Accumulated textual changes plus their derived char/block tallies. */
interface DiffTally {
changes: DiffChange[];
inserted: number;
deleted: number;
changedBlocks: Set<string>;
}
/**
* Produce the coarse-fallback tally for a pair. This is the SINGLE source of the
* `fellBack:true` result shape, shared by BOTH degrade paths in diffDocs (the
* pre-flight size guard and the recreateTransform catch) so they behave and
* report identically.
*/
function coarseDiffTally(oldDoc: any, newDoc: any): DiffTally {
const changes = coarseDiff(oldDoc, newDoc);
let inserted = 0;
let deleted = 0;
const changedBlocks = new Set<string>();
for (const c of changes) {
if (c.op === "insert") inserted += c.text.length;
else deleted += c.text.length;
if (c.block) changedBlocks.add(c.op[0] + ":" + c.block);
}
return { changes, inserted, deleted, changedBlocks };
}
/**
* Compute the PRECISE tally via the recreateTransform pipeline. Callers MUST
* gate this behind the size guard (it can block the event loop for a large pair)
* and wrap it in try/catch (a pathological pair can throw); on either the guard
* or a throw, use `coarseDiffTally` instead. Kept as a sibling of
* `coarseDiffTally` so both produce the same `DiffTally` shape.
*/
function preciseDiffTally(oldDocJson: any, newDocJson: any): DiffTally {
const oldNode = Node.fromJSON(docmostSchema, oldDocJson);
const newNode = Node.fromJSON(docmostSchema, newDocJson);
const tr = recreateTransform(oldNode, newNode, {
complexSteps: false,
wordDiffs: true,
simplifyDiff: true,
});
const changeSet = ChangeSet.create(oldNode).addSteps(tr.doc, tr.mapping.maps, []);
const simplified = simplifyChanges(changeSet.changes, newNode);
const changes: DiffChange[] = [];
let inserted = 0;
let deleted = 0;
const changedBlocks = new Set<string>();
for (const change of simplified) {
// Deleted text lives in the OLD doc coordinate range [fromA, toA).
if (change.toA > change.fromA) {
const text = oldNode.textBetween(change.fromA, change.toA, "\n", " ");
if (text.length > 0) {
deleted += text.length;
const block = blockContextAt(oldNode, change.fromA);
changes.push({ op: "delete", block, text });
if (block) changedBlocks.add("d:" + block);
}
}
// Inserted text lives in the NEW doc coordinate range [fromB, toB).
if (change.toB > change.fromB) {
const text = newNode.textBetween(change.fromB, change.toB, "\n", " ");
if (text.length > 0) {
inserted += text.length;
const block = blockContextAt(newNode, change.fromB);
changes.push({ op: "insert", block, text });
if (block) changedBlocks.add("i:" + block);
}
}
}
return { changes, inserted, deleted, changedBlocks };
}
/** Build the human-readable unified-ish markdown summary. */
function renderMarkdown(
result: Omit<DiffResult, "markdown">,
@@ -276,66 +413,39 @@ export function diffDocs(
newDocJson: any,
notesHeading: string = "Примечания переводчика",
): DiffResult {
// computeIntegrity is cheap (linear node walks) and its counts are needed in
// BOTH the precise and coarse paths, so it always runs first.
const integrity = computeIntegrity(oldDocJson, newDocJson, notesHeading);
let changes: DiffChange[] = [];
let inserted = 0;
let deleted = 0;
let fellBack = false;
const changedBlocks = new Set<string>();
let tally: DiffTally;
try {
const oldNode = Node.fromJSON(docmostSchema, oldDocJson);
const newNode = Node.fromJSON(docmostSchema, newDocJson);
const tr = recreateTransform(oldNode, newNode, {
complexSteps: false,
wordDiffs: true,
simplifyDiff: true,
});
const changeSet = ChangeSet.create(oldNode).addSteps(
tr.doc,
tr.mapping.maps,
[],
);
const simplified = simplifyChanges(changeSet.changes, newNode);
for (const change of simplified) {
// Deleted text lives in the OLD doc coordinate range [fromA, toA).
if (change.toA > change.fromA) {
const text = oldNode.textBetween(change.fromA, change.toA, "\n", " ");
if (text.length > 0) {
deleted += text.length;
const block = blockContextAt(oldNode, change.fromA);
changes.push({ op: "delete", block, text });
if (block) changedBlocks.add("d:" + block);
}
}
// Inserted text lives in the NEW doc coordinate range [fromB, toB).
if (change.toB > change.fromB) {
const text = newNode.textBetween(change.fromB, change.toB, "\n", " ");
if (text.length > 0) {
inserted += text.length;
const block = blockContextAt(newNode, change.fromB);
changes.push({ op: "insert", block, text });
if (block) changedBlocks.add("i:" + block);
}
}
}
} catch {
// Pathological pair: degrade to a coarse block-level diff so we never throw.
// Pre-flight size guard (#464): a too-large pair would make recreateTransform
// block the event loop for seconds-to-hours WITHOUT throwing, so route it to
// the coarse fallback BEFORE calling recreateTransform at all. Both this path
// and the catch below go through coarseDiffTally for an identical `fellBack`
// result shape.
if (exceedsDiffSizeGuard(oldDocJson, newDocJson)) {
fellBack = true;
changes = coarseDiff(oldDocJson, newDocJson);
for (const c of changes) {
if (c.op === "insert") inserted += c.text.length;
else deleted += c.text.length;
if (c.block) changedBlocks.add(c.op[0] + ":" + c.block);
tally = coarseDiffTally(oldDocJson, newDocJson);
} else {
try {
tally = preciseDiffTally(oldDocJson, newDocJson);
} catch {
// Pathological pair: degrade to a coarse block-level diff so we never throw.
fellBack = true;
tally = coarseDiffTally(oldDocJson, newDocJson);
}
}
const partial: Omit<DiffResult, "markdown"> = {
summary: { inserted, deleted, blocksChanged: changedBlocks.size },
summary: {
inserted: tally.inserted,
deleted: tally.deleted,
blocksChanged: tally.changedBlocks.size,
},
integrity,
changes,
changes: tally.changes,
};
return { ...partial, markdown: renderMarkdown(partial, fellBack) };
}
@@ -380,7 +490,7 @@ export interface VerifyReport {
/**
* ONLY structural integrity types whose count changed, as [before, after]
* (images/links/tables/callouts). Surfaces structural mutations that touch
* neither text nor marks (e.g. insert_image, deleting a table) which diffDocs
* neither text nor marks (e.g. insertImage, deleting a table) which diffDocs
* being TEXT-only would otherwise report as "no content change".
*/
structure?: Record<string, [number, number]>;
@@ -400,7 +510,7 @@ export interface VerifyReport {
*
* The structural integrity delta (from diffDocs's `integrity` tuples) is what
* makes `changed` true for an image/table/callout/link count change that diffs
* to zero text closing a verify blind spot for insert_image, delete_node on a
* to zero text closing a verify blind spot for insertImage, deleteNode on a
* table, etc.
*/
export function summarizeChange(before: any, after: any): VerifyReport {

Some files were not shown because too many files have changed in this diff Show More