Compare commits

..

12 Commits

Author SHA1 Message Date
agent_coder 0a53be9e81 docs(converter): correct #515 mark-order comment — imported order depends on extension, not fixed
The inlineToHtml comment falsely claimed import ALWAYS yields the code mark
last; import yields code last for bold/italic/strike but code FIRST for the
==-highlight extension. Reworded to state the real invariant (wrap <code>
innermost regardless of imported order) and de-universalized the parallel
phrase in text-arbitraries.ts. Comment-only, no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 02:44:54 +03:00
agent_coder 95c0d813b0 fix(converter): round-trip code+эмфаза — code внутри, склейка соседних пробегов, HTML-fallback (#515)
Теперь узел может нести [code, bold], и сериализатор PM→Markdown должен это
корректно выгружать в обе стороны (git-sync — data-loss-critical, требуется
байт-стабильность md2===md1).

case "text": убран ранний return для code-рана. Backtick-спан оборачивается
ПЕРВЫМ (самая внутренняя марка), затем прочие марки в том же порядке массива —
`**`code`**`. Для НЕ-code ранов вывод байт-идентичен прежнему (вынесены хелперы
escapeInlineText и applyInlineMark, поведение сохранено).

renderInlineChildren: собирает максимальный пробег подряд идущих text-узлов с
голой-делимитерной эмфазис-маркой (bold/italic/strike/uncolored-highlight),
содержащий хотя бы один code-узел. ОДНОРОДНЫЙ пробег (у всех идентичное
множество не-code марок) и безопасные границы → общие марки выносятся наружу
ОДИН раз: `**`aaa` + `bbb`**`, `**`code4` tail**`. НЕОДНОРОДНЫЙ (`[code,bold]`
рядом с `[italic]`) ИЛИ граница упирается в словесный символ (делимитер `**`
перед backtick не был бы flanking → эмфаза потерялась бы) → весь пробег через
lossless inlineToHtml (схема-HTML). НЕ-code вывод байт-идентичен, кроме
редкого случая голой-делимитерной эмфазы вплотную к code+эмфазе. Узлы, где
code раньше вообще не мог нести эмфазу, — новая территория, существующие
страницы не затрагиваются.

inlineToHtml: `<code>` тоже оборачивается ВНУТРЕННИМ (импорт отдаёт code
последним в массиве — `[emphasis, code]`; порядок-зависимый цикл переворачивал
бы `<strong><code>` в `<code><strong>` на реэкспорте и ломал байт-фикспойнт).

Тесты: перевёрнуты ассерты, фиксировавшие старое (по CommonMark неверное)
поведение (code+bold/strike/link → теперь `**`x`**`/`~~`x`~~`/`[`x`](…)`);
генераторы расширены на code+{bold,italic,strike,highlight} в каноническом
порядке импорта; добавлены явные round-trip пины 5 кейсов репорта (импорт→марки
и md→pm→md идемпотентность) + усиление code-combo свойства (обе марки выживают).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 23:41:07 +03:00
agent_coder 9004de60e3 fix(schema): снять excludes:"_" у марки code — она комбинируется со всеми марками (#515)
По CommonMark `**` рядом с инлайн-кодом дают `<strong><code>` — узел с
составом марок [code, bold]. Марка `code` из tiptap несёт `excludes: "_"`
(исключает все прочие инлайн-марки), и ProseMirror на HTML→PM импорте
(`generateJSON`) выкидывает сосуществующую bold — так `**`--flag`**` терял
жирный. Ставим `excludes: ""` (не исключает ничего) во всех четырёх местах,
где марка конфигурируется независимо:

- Единый источник: новая каноническая марка `Code = TiptapCode.extend({
  excludes: "" })` в @docmost/editor-ext, экспортируется из barrel.
- Живой редактор (extensions.ts): базовая `Code` теперь из editor-ext,
  поверх сохранены client-only addInputRules/addKeyboardShortcuts.
- Collab-сервер + серверный HTML-парс/экспорт (collaboration.util.ts):
  StarterKit code:false + общая `Code`; @docmost/editor-ext объявлен в
  apps/server/package.json (использовался, но не был задекларирован).
- Редактор комментариев (comment-editor.tsx): StarterKit code:false + `Code`.
- Вендор-зеркало docmostExtensions (docmost-schema.ts) — сознательно отдельная
  копия, не тянущая editor-ext в node-рантайм: `excludes:""` объявлен локально
  (StarterKit code:false + `Code.extend({ excludes:"" })`), держится в
  синхроне с editor-ext паритет-тестом.

Паритет-гард: тест в пакете сверяет excludes марки code вендор-зеркала с
канонической editor-ext Code (обе ""); vitest резолвит @docmost/editor-ext на
sibling-исходник, чтобы гард был герметичным.

`excludes:""` делает code «перекрывающейся» маркой в y-prosemirror (как comment):
она пишется в Yjs под хешированным ключом `code--<hash>` и распаковывается
обратно в `code` штатным декодером — правим mcp-тест-хелпер fragmentToJson,
чтобы он снимал хеш ровно как yattr2markname (иначе overlapping-code утёк бы
в сравнение как `code--<hash>`).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 23:41:07 +03:00
agent_coder 3a344626db fix(converter): block-escape закрывает setext-подчёркивание -- и одиночный = (#514 ревью)
CRITICAL из ревью: escapeLeadingBlockTrigger не покрывал setext-underline
из ровно двух дефисов (--) и одиночного = → строка-продолжение после
hardBreak, равная -- или =, репарсилась как setext-heading, а текст
предыдущей строки терялся. Тот же класс потери данных, что PR и чинит.

Добавлена setext-рука после тематической: целая строка ^-+[ \t]*$ или
^=+[ \t]*$ экранирует ведущий символ. Заякорено на всю строку → mid-content
-/= не задевается; после тематической руки → ---/---- не двойно-экранируются;
идемпотентно против уже-экранированного \=\= (инлайн-escape отрабатывает
раньше). Пины --/----/=/==== + генеративный hardBreakThenSetextArb, оба
mutation-verified. CHANGELOG.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 23:40:26 +03:00
agent_coder 31f51eaa47 fix(mcp): agent-write НЕ срезает ведущий ---…--- (front-matter strip — только импорт)
Ревью #493 (MEDIUM): вшив normalizeForeignMarkdown первым шагом в
markdownToProseMirrorCanonical, коммит 4 распространил срез YAML front-matter
(YAML_FRONT_MATTER_RE) на КАЖДЫЙ agent-write путь. convertProseMirrorToMarkdown
эмитит `---` для horizontalRule, поэтому страница, начинающаяся с
horizontalRule и содержащая второй `---`, при полном agent-write теряла всё до
второго `---` — молчаливая потеря ранее сохранённого контента.

Правка: разделил нормализацию. normalizeForeignMarkdown (серверный file-import
boundary) по-прежнему срезает front-matter. Новый normalizeAgentMarkdown
(agent-write, markdownToProseMirrorCanonical) делает ТОЛЬКО CRLF-нормализацию +
rewrite GFM reference-сносок (тот самый drift, ради которого коммит 4) и НЕ
трогает ведущий `---…---`. На каноническом сериализованном контенте rewrite —
no-op (он не эмитит `[^id]:`-строк).

Тесты: agent-write horizontalRule-led дока со вторым `---` сохраняет весь
контент (round-trip); file-import с реальным YAML front-matter его по-прежнему
срезает; agent-write всё ещё канонизирует GFM reference-сноски.
Mutation-verify: strip обратно на agent-write → тесты потери контента краснеют.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:54:38 +03:00
agent_coder b66929714f fix(converter): block-escape КАЖДОЙ строки параграфа (не только первой)
Ревью #493 (HIGH): escapeLeadingBlockTrigger применялся к всему результату
renderInlineChildren один раз, а регэкспы якорены на ^ без флага m → защищалась
только ПЕРВАЯ строка. hardBreak сериализуется как `  \n`, поэтому триггер на
строке-продолжении не экранировался и на git-sync round-trip re-парсился в
другой блок; для setext/thematic `---` текст строки терялся ЦЕЛИКОМ
([text "a", hardBreak, text "---"] → heading, "---" пропадал).

Правка: параграф теперь бьётся на `\n`-строки и escapeLeadingBlockTrigger
применяется к КАЖДОЙ. Фаззер расширен: hardBreakThenTriggerArb вставляет
триггер ПОСЛЕ hardBreak в inlineContentArb, так что P1/P2/P3 структурно
покрывают подслучай (раньше триггер стоял только первым run'ом). Добавлен
детерминированный пин на continuation-line триггеры, включая `a  \n---`
(текст «---» сохраняется, не setext).

Mutation-verify: со старым single-line escape новый пин + P1/P2 краснеют.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:54:38 +03:00
agent_coder a09935aa29 fix(mcp): resolved-якоря переживают полный markdown-write
Read-путь прячет resolved comment-анкоры (#337), поэтому markdown, который
агент шлёт в updatePageMarkdown, их уже не содержит — наивный full-write
стирал ВСЕ resolved comment-марки (потеря данных). Активные комментарии
переживают round-trip сами (read отдаёт их <span data-comment-id>), а
resolved — нет.

Правка: в write-пути (updatePageContentRealtime) пере-прививаем resolved-
марки из ЖИВОГО дока на совпадающие текстовые диапазоны свежеимпортированного
тела, механикой анкоринга comment-anchor. spliceCommentMark обобщён на
произвольную марку; добавлены applyCommentMarkInDoc (сохраняет resolved:true
+ attrs), collectResolvedCommentSpans и regraftResolvedComments (чистая, не
мутирует входы). Спан, чей текст агент изменил/удалил, просто не
переанкорится и отбрасывается (он и так resolved). first-occurrence-семантика
как у остального анкоринга.

Проверено: 6 новых наблюдаемых тестов + весь MCP unit-suite (691) зелёные.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:19:10 +03:00
agent_coder 047433595e refactor(mcp): дедуп stripInlineMarkdown — единый источник в каноническом пакете
Локаторная нормализация markdown (stripInlineMarkdown + примитив
stripWrappersAndLinks с WRAPPER_PATTERNS/LINK_IMAGE_RE) была ФОРКНУТА один-в-
один в packages/mcp/src/lib/text-normalize.ts и в каноническом
@docmost/prosemirror-markdown (где ей пользуется node-ops). MCP теперь
импортирует оба примитива из пакета (mcp и так от него зависит — цикла нет) и
держит на них лишь свои тонкие надстройки stripBalancedWrappers/
closestBlockHint. ~60 строк дубля удалено, дрейф закрыт.

Проверено: пакет node-ops (103) + весь MCP unit-suite (685) зелёные.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:14:39 +03:00
agent_coder 9e95412695 refactor(converter): normalizeForeignMarkdown -> в пакет, единый import-boundary
Нормализация чужого markdown (GFM reference-сноски [^id] -> инлайн ^[body],
срез ведущего YAML front-matter) жила только в apps/server, поэтому MCP-путь
записи страницы (updatePageMarkdown -> markdownToProseMirrorCanonical) тот же
ввод обрабатывал ИНАЧЕ, чем серверный импорт: front-matter и [^id] утекали
как литеральный текст / битая ссылка.

Перенёс normalizeForeignMarkdown в @docmost/prosemirror-markdown и вызвал его
первым шагом в MCP markdownToProseMirrorCanonical — теперь агентский
updatePageMarkdown нормализуется точно как серверный импорт. Серверные
импортёры (import.service, file-import-task.service, page.service) берут
функцию из пакета. Тест-корпус перенесён в пакет (foreign-markdown.test.ts).

Проверено: пакет (17 тестов corpus) + весь MCP unit-suite (685) зелёные,
включая reference-footnote/fence-кейсы на canonical-пути.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 18:05:47 +03:00
agent_coder 2fa86e2a33 fix(converter): warnings вместо тихой потери незнакомых нод/марок
Незнакомый тип ноды (default-ветка switch) молча схлопывался в свои дети, а
незнакомая марка молча выбрасывалась — тихая потеря данных. Теперь
сериализатор РЕПОРТИТ потерю:
- по умолчанию поведение байт-в-байт прежнее (graceful degrade), но при
  переданном options.warnings в сток кладётся по одному сообщению на
  незамапленный тип (дедуп по типу) — потеря наблюдаема;
- options.strict бросает ConverterLossError на ПЕРВОМ незнакомом типе
  (warning = ошибка).

git-sync (lossless-путь) включает strict в stabilizePageBody: тип без
серизализующей ветки падает громко на записи, а не пишет lossy .md. Валидный
контент не затронут — у всех текущих типов схемы есть ветка.

Покрытие: converter-loss-warnings.test.ts (нода/марка × strict/non-strict,
дедуп, чистый контент) и strict-пин в git-sync stabilize.test.ts — всё через
реальный конвертер.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 17:56:11 +03:00
agent_coder e3eece78c3 test(converter): атрибутный contract-тест схем editor-ext <-> mirror
Name-level контракт ловил пропажу целой ноды/марки, но не дрейф АТРИБУТОВ
внутри вендоренной ноды — класс, из-за которого молча потерялся
subpages.recursive. Добавлен атрибутный контракт: для каждой ноды/марки
@docmost/editor-ext сравниваются её СОБСТВЕННЫЕ объявленные атрибуты (имена
+ дефолты, читаются из config.addAttributes) с spec.attrs собранной схемы
зеркала.

Направление editor-ext -> mirror: зеркало намеренно надмножество (глобальные
id/textAlign/indent, нормализация части дефолтов в null), поэтому обратное
сравнение — ложный дрейф. Значимый провал — атрибут, который зеркало РОНЯЕТ
(имя) или чей дефолт молча меняет. Два blessed вида расхождений вынесены в
обоснованные allowlist'ы (highlight.colorName — нет md-формы;
image.src/link.internal/pdf.width/height — null-нормализация непереносимых
атрибутов), оба со stale-guard, чтобы список не сгнил.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 17:49:03 +03:00
agent_coder e1b8ef5b8b fix(converter): block-escape начала параграфа — закрытие класса потерь данных
Строка параграфа, начинающаяся с блочного триггера (`#`/`-`/`*`/`+`/`>`,
упорядоченного `N.`/`N)`, фенса ```/~~~, таблицы `|` или тематического
разрыва `---`/`***`/`___`), на round-trip doc->markdown->doc молча
превращалась в heading/list/quote/code block/table/horizontalRule. Худший
случай — тематический разрыв: horizontalRule не несёт текста, и строка
теряла его целиком.

Сериализатор параграфа теперь backslash-экранирует ведущий блочный триггер
(escapeLeadingBlockTrigger): экранируется только ПЕРВЫЙ значащий символ,
токенизатор CommonMark декодирует `\` обратно в литерал И снимает блочную
интерпретацию, так что строка round-trip'ится байт-в-байт как параграф.
Emphasis `**x**`, inline-code и обычная проза триггерами не являются и не
трогаются (нет мусорных backslash).

Класс раньше не чинили, а ОБХОДИЛИ; обход убран у обоих потребителей:
- клиентский мост (gitmost-recording.ts) больше не подставляет ZWSP-хак;
- генеративный корпус (text-arbitraries.ts) снял самоцензуру — добавлен
  blockTriggerLeadRunArb, параграф теперь МОЖЕТ открываться триггером, и
  P1/P2/P3 сами доказывают закрытие класса.

Пины на каждый триггер — детерминированные round-trip через реальный
конвертер (gitmost-transcript-neutralization.test.ts). Обновлён
документировавший старую потерю gap-тест (spec 13).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 17:43:24 +03:00
51 changed files with 1704 additions and 1216 deletions
-63
View File
@@ -62,38 +62,6 @@ jobs:
needs: [test, e2e-server, e2e-mcp, build]
runs-on: ubuntu-latest
timeout-minutes: 30
# Image boot-smoke (issue #476): every other job tests code from the working
# tree, but the :develop IMAGE that watchtower pulls was never actually
# started anywhere (incident classes #353/#452/#361-boot: startup-migrator
# crash-loop, runtime module missing from the image, wrong static-asset
# headers). The services below back a smoke boot of the exact image right
# before it is pushed; a smoke failure blocks the push.
services:
postgres:
# via mirror.gcr.io (Docker Hub pull-through cache; avoids Hub anonymous
# pull rate-limit that randomly fails on shared GitHub runner IPs).
image: mirror.gcr.io/pgvector/pgvector:pg18
env:
POSTGRES_DB: docmost
POSTGRES_USER: docmost
POSTGRES_PASSWORD: docmost
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U docmost"
--health-interval 5s
--health-timeout 5s
--health-retries 20
redis:
# via mirror.gcr.io (see postgres note above).
image: mirror.gcr.io/library/redis:7
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 5s
--health-timeout 5s
--health-retries 20
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -114,37 +82,6 @@ jobs:
id: version
run: echo "value=$(git describe --tags --always)" >> "$GITHUB_OUTPUT"
# Load the image into the local docker daemon so it can be booted (the
# push step below exports straight to the registry and leaves nothing
# runnable locally). CONVENTION: build-args here must stay TEXTUALLY
# IDENTICAL to the push step's build-args — same cache scope + same args
# means the layers are reused and the image we smoke IS the image we push.
- name: Build image for smoke (load, no push)
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64
build-args: |
APP_VERSION=${{ steps.version.outputs.value }}
AI_AGENT_ROLES_CATALOG_URL=https://raw.githubusercontent.com/vvzvlad/gitmost/develop/agent-roles-catalog
load: true
push: false
tags: gitmost:smoke
cache-from: type=gha,scope=develop-amd64
# Boot-smoke the exact image against the job services (see the comment on
# `services:` above): health (startup migrator), auth/setup, client dist
# served, immutable + brotli asset headers. Fails the job (and therefore
# the push) on any miss.
- name: Smoke the built image
run: bash scripts/ci/image-smoke.sh gitmost:smoke
# The smoke script leaves the container running on failure precisely so
# the boot error (migration mismatch, stack trace) is diagnosable here.
- name: Dump smoke container log on failure
if: failure()
run: docker logs gitmost-smoke 2>&1 | tail -200 || true
- name: Build and push develop image
uses: docker/build-push-action@v6
with:
+13 -41
View File
@@ -25,65 +25,37 @@ jobs:
# filename sorts BEFORE migrations already applied on the target branch (and
# thus in prod). The Kysely startup migrator rejects that as "corrupted
# migrations" and crash-loops the app on boot (incident #361). This gate fails
# the PR so the migration is renamed to a current timestamp before merge.
# Runs for pull_request (diff against the base branch) AND for push (#476
# retrospective: a DIRECT push to develop used to bypass this PR-only gate
# entirely — now the push is diffed against its `before` SHA; workflow_call
# from develop.yml inherits the caller's push event). workflow_dispatch has
# nothing to diff against and still skips the job.
# the PR so the migration is renamed to a current timestamp before merge. Only
# runs for pull_request events (needs a base branch to diff against).
migration-order:
if: github.event_name == 'pull_request' || github.event_name == 'push'
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Checkout (full history for the base diff)
- name: Checkout (full history for the base-branch diff)
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Added migrations must sort after the newest on the base
- name: Added migrations must sort after the newest on the base branch
env:
TARGET_BRANCH: ${{ github.base_ref }}
BEFORE_SHA: ${{ github.event.before }}
run: |
set -euo pipefail
MIG_DIR="apps/server/src/database/migrations"
if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then
# checkout above already did fetch-depth:0 (full history). Fetch the base
# WITHOUT --depth (a shallow graft would truncate the base history and
# break the merge-base when the base has moved ahead of the PR merge —
# exactly the long-branch-vs-moving-base case this gate guards, #361).
git fetch --no-tags origin "$TARGET_BRANCH"
BASE="origin/${TARGET_BRANCH}"
else
# push event: compare against the pre-push tip of the branch.
if [ "$BEFORE_SHA" = "0000000000000000000000000000000000000000" ]; then
echo "::notice::branch creation push — nothing to compare"
exit 0
fi
if ! git cat-file -e "${BEFORE_SHA}^{commit}" 2>/dev/null; then
# The before-SHA is not in the clone (a force-push rewrote history).
# One recovery attempt — refresh every remote head (cheap: the
# checkout is already fetch-depth:0); a fetch failure aborts via
# `set -e`, which is fail-closed too.
git fetch --no-tags origin '+refs/heads/*:refs/remotes/origin/*'
fi
if ! git cat-file -e "${BEFORE_SHA}^{commit}" 2>/dev/null; then
# FAIL-CLOSED: without the before-SHA there is no base to prove the
# ordering against, and a gate whose job is to BLOCK must not guess.
echo "::error::force-push detected — verify migration order manually, then re-run via workflow_dispatch"
exit 1
fi
BASE="$BEFORE_SHA"
fi
newest_on_target=$(git ls-tree -r --name-only "$BASE" "$MIG_DIR" | sort | tail -1)
# checkout above already did fetch-depth:0 (full history). Fetch the base
# WITHOUT --depth (a shallow graft would truncate the base history and
# break the merge-base when the base has moved ahead of the PR merge —
# exactly the long-branch-vs-moving-base case this gate guards, #361).
git fetch --no-tags origin "$TARGET_BRANCH"
newest_on_target=$(git ls-tree -r --name-only "origin/${TARGET_BRANCH}" "$MIG_DIR" | sort | tail -1)
# NO `|| true`: a diff failure (e.g. an unresolved merge-base) must fail
# the job CLOSED — a gate whose job is to BLOCK must never pass on error.
# `set -e` above already aborts on a non-zero diff exit.
added=$(git diff --diff-filter=A --name-only "${BASE}...HEAD" -- "$MIG_DIR")
added=$(git diff --diff-filter=A --name-only "origin/${TARGET_BRANCH}...HEAD" -- "$MIG_DIR")
bad=0
for f in $added; do
if [[ "$f" < "$newest_on_target" || "$f" == "$newest_on_target" ]]; then
echo "::error::Migration $f sorts at or before the newest on the base ($newest_on_target) — rename it with a CURRENT timestamp before merge (do not change its contents). See incident #361."
echo "::error::Migration $f sorts at or before the newest on ${TARGET_BRANCH} ($newest_on_target) — rename it with a CURRENT timestamp before merge (do not change its contents). See incident #361."
bad=1
fi
done
+9 -13
View File
@@ -302,6 +302,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **Markdown round-trips no longer silently drop a line that opens with a block
trigger.** When a document is exported to Markdown and re-imported (git-sync
stabilize, agent writes), a paragraph or continuation line (after a hard break)
that begins with a block marker — an ATX heading `#`, a blockquote/callout `>`,
a list marker (`-`/`*`/`+`/`N.`/`N)`), a code fence, a table `|`, a thematic
break (`---`), or a setext underline (`--`, `----`, or a lone `=`) — is now
backslash-escaped so it round-trips as text instead of being re-parsed into a
heading/list/quote/rule and losing its content. Front-matter stripping is
scoped to the import path only. (#493)
- **The server no longer runs out of heap during long autonomous agent runs.** A
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
snapshot of the ENTIRE turn text on every streamed text-delta when no output
@@ -385,19 +394,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
through that exact share (its own share or an ancestor `includeSubPages`
share); any other value now returns the generic "not found" instead of
serving the page. (#218)
- **MCP tool-allowlist semantics flipped: an empty `[]` now means deny-all
(previously it was coerced to "no restrictions").** For an external MCP server,
a stored `tool_allowlist` of `[]` now denies **every** tool of that server
(zero tools reach the agent) instead of being treated as an empty/unset filter
that allowed all of them. A corrupt or non-array stored value now **fails
closed** to deny-all rather than silently allowing everything. The admin form
no longer silently widens an existing deny-all server: leaving its tag field
empty preserves `[]` (deny-all) on save instead of NULL-ing the column to
allow-all, so a routine rename/toggle can no longer grant the agent every tool.
"No restrictions" is still expressible — a genuinely unrestricted server stores
NULL, and clearing the field on such a server keeps it NULL. Operationally
significant: audit any server that was created or left with a literal `[]`, as
it now exposes no tools until an explicit allowlist (or NULL) is set. (#476)
## [0.94.0] - 2026-06-26
@@ -1,7 +1,7 @@
import { EditorContent, ReactNodeViewRenderer, useEditor } from "@tiptap/react";
import { Placeholder } from "@tiptap/extension-placeholder";
import { StarterKit } from "@tiptap/starter-kit";
import { Mention, LinkExtension } from "@docmost/editor-ext";
import { Mention, LinkExtension, Code } from "@docmost/editor-ext";
import classes from "./comment.module.css";
import { useFocusWithin } from "@mantine/hooks";
import clsx from "clsx";
@@ -44,7 +44,12 @@ const CommentEditor = forwardRef(
gapcursor: false,
dropcursor: false,
link: false,
// #515: use the shared editor-ext `Code` (excludes: "") instead of
// StarterKit's excluding one, so inline code in a comment can carry
// other marks and does not drop them when the comment is edited.
code: false,
}),
Code,
Placeholder.configure({
placeholder: placeholder || t("Reply..."),
}),
@@ -1,6 +1,5 @@
import { markInputRule } from "@tiptap/core";
import { StarterKit } from "@tiptap/starter-kit";
import { Code } from "@tiptap/extension-code";
import { TextAlign } from "@tiptap/extension-text-align";
import { TaskList, TaskItem } from "@tiptap/extension-list";
import { Placeholder, CharacterCount, UndoRedo } from "@tiptap/extensions";
@@ -67,6 +66,7 @@ import {
FootnoteReference,
FootnotesList,
FootnoteDefinition,
Code,
} from "@docmost/editor-ext";
import {
randomElement,
@@ -153,6 +153,10 @@ export const mainExtensions = [
codeBlock: false,
code: false,
}),
// Base `Code` comes from @docmost/editor-ext, which overrides `excludes: ""`
// (#515) so inline code can co-occur with bold/italic/… — the SINGLE shared
// source also used by the collab server and comment editor. Here we keep the
// existing client-only behavior on top of it:
// Override TipTap's Code extension to fix the inline code input rule.
// The upstream regex /(^|[^`])`([^`]+)`(?!`)$/ captures the character
// before the opening backtick as part of the match, causing markInputRule
@@ -9,7 +9,7 @@ import { Italic } from "@tiptap/extension-italic";
import { Link } from "@tiptap/extension-link";
import { gitmostInsertTranscriptIntoEditor } from "./gitmost-recording.ts";
const ZWSP = "​"; // U+200B, the helper's block-trigger neutralizer
const ZWSP = "​"; // U+200B — asserted ABSENT (the block-escape lives in the serializer now)
/**
* #377 — the web-side bridge must append the native host's transcript below the
@@ -18,8 +18,9 @@ const ZWSP = "​"; // U+200B, the helper's block-trigger neutralizer
* regression would be caught), asserting the resulting document rather than
* mocking the editor: transcript present -> "Transcript" heading + one paragraph
* per non-empty line; content is inserted as LITERAL TEXT (no HTML/markdown
* parsing); col-0 markdown block triggers are neutralized so git-sync keeps them
* paragraphs; absent/empty/non-string -> no-op.
* parsing); col-0 markdown block triggers are stored verbatim (the git-sync
* serializer block-escapes them, so no client-side ZWSP is needed);
* absent/empty/non-string -> no-op.
*/
describe("gitmostInsertTranscriptIntoEditor", () => {
const makeEditor = () =>
@@ -91,19 +92,22 @@ describe("gitmostInsertTranscriptIntoEditor", () => {
editor.destroy();
});
it("neutralizes col-0 markdown block triggers with a leading ZWSP (git-sync safety)", () => {
it("inserts col-0 markdown block triggers as verbatim paragraph text (no ZWSP workaround)", () => {
const editor = makeEditor();
// Trigger lines (some with a leaked indent) + a normal prefixed line.
// Trigger lines (some with a leaked indent) + a normal prefixed line. The
// git-sync serializer now block-escapes a leading trigger itself, so the
// bridge inserts each line's TEXT byte-exact (only the leaked indent is
// trimmed) — no invisible ZWSP is prepended anymore.
const inserted = gitmostInsertTranscriptIntoEditor(
editor,
[
"- dash",
" > quote", // leading indent must be trimmed then neutralized
" > quote", // leading indent is trimmed, text otherwise verbatim
"# hash",
"1. one",
"> [!info] note",
"```js",
"---", // solid thematic break -> horizontalRule (text-losing) if unneutralized
"---",
"***",
"___",
"You: normal line",
@@ -116,20 +120,23 @@ describe("gitmostInsertTranscriptIntoEditor", () => {
.map((n: any) => n.content?.[0]?.text)
.filter((t: any) => typeof t === "string") as string[];
// Every block-trigger line is prefixed with the invisible ZWSP (indent
// trimmed first); the normal `You:` line is left byte-exact.
// Each trigger line is stored as its own byte-exact text (indent trimmed);
// the git-sync round-trip keeps it a paragraph via the serializer's
// block-escape, so no ZWSP is needed here.
expect(texts).toEqual([
ZWSP + "- dash",
ZWSP + "> quote",
ZWSP + "# hash",
ZWSP + "1. one",
ZWSP + "> [!info] note",
ZWSP + "```js",
ZWSP + "---",
ZWSP + "***",
ZWSP + "___",
"- dash",
"> quote",
"# hash",
"1. one",
"> [!info] note",
"```js",
"---",
"***",
"___",
"You: normal line",
]);
// Guard: no invisible ZWSP leaked into any inserted line.
for (const t of texts) expect(t).not.toContain(ZWSP);
editor.destroy();
});
@@ -240,45 +240,22 @@ export async function gitmostUploadFileToEditor(
}
}
// Zero-width space (U+200B). Prepended to a transcript line that begins with a
// markdown BLOCK trigger: it is invisible in the rendered doc but shifts the
// trigger off column 0, so the git-sync doc->markdown->doc round-trip keeps the
// line a plain paragraph (see GITMOST_MD_BLOCK_TRIGGER_RE).
const GITMOST_ZWSP = "​";
// A markdown BLOCK-level construct that, sitting at column 0 of a paragraph
// line, the git-sync markdown serializer (packages/prosemirror-markdown
// markdown-converter.ts, `case "paragraph"`) would re-parse into a NON-paragraph
// block on the doc->markdown->doc cycle. That serializer emits paragraph text
// verbatim with NO block-escape (the pre-existing root cause), so a leading
// `#`/`-`/`*`/`+`/`>`, an ordered-list `N.`/`N)`, a code fence ```/~~~, a table
// `|`, or a `> [!info]` callout opener would silently become a heading / list /
// quote / code block / table / callout. The final alternative matches a WHOLE-
// LINE thematic break — solid `---`/`***`/`___` or spaced `- - -`/`_ _ _` (3+ of
// the same `-`/`*`/`_`) — which round-trips into a `horizontalRule`; because
// that node carries NO text, an un-neutralized separator line would LOSE its
// text entirely (worse than the list/quote case). This matches a TRIMMED line's
// start; the transcript's own `You:` / `Speaker N:` prefix begins with a letter
// and never matches, so prefixed lines are left byte-exact.
const GITMOST_MD_BLOCK_TRIGGER_RE =
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
// Append a transcript block BELOW the recording's audio node in a live editor:
// a "Transcript" heading followed by one paragraph per non-empty transcript
// line. The transcript is plain text, `\n`-separated, each line already
// formatted as `You: ...` / `Speaker N: ...` by the native host — line text is
// inserted as a TEXT node (never HTML/markdown), so there is no injection or
// mark-parsing surface. Each kept line is trimmed (drops an indent that would
// both leak into the display and, at col 0, form a markdown block trigger) and,
// if it still begins with a col-0 markdown block trigger, gets an invisible
// zero-width space prepended so the git-sync round-trip cannot turn it into a
// list/quote/heading/callout/code/table (defensive boundary against the
// serializer's missing block-escape). This is best-effort and meant to run
// AFTER the audio has already been inserted; the caller must guard against a
// throw so a transcript failure never fails the (already successful) recording.
// Returns true when a block was inserted, false when there was nothing to
// insert (transcript undefined/empty/not-a-string). A non-string value is a
// no-op, not an error.
// leak into the display). A line that begins with a col-0 markdown block
// trigger (`#`/`-`/`>`/`1.`/fence/`---`/…) needs no client-side workaround: the
// git-sync serializer (packages/prosemirror-markdown, `case "paragraph"`) now
// block-escapes such a leading trigger, so the doc->markdown->doc round-trip
// keeps the line a paragraph on its own — the former invisible-ZWSP defense is
// gone. This is best-effort and meant to run AFTER the audio has already been
// inserted; the caller must guard against a throw so a transcript failure never
// fails the (already successful) recording. Returns true when a block was
// inserted, false when there was nothing to insert (transcript
// undefined/empty/not-a-string). A non-string value is a no-op, not an error.
export function gitmostInsertTranscriptIntoEditor(
editor: Editor,
transcript: unknown,
@@ -288,13 +265,7 @@ export function gitmostInsertTranscriptIntoEditor(
.split("\n")
// Trim each line and drop blank (whitespace-only) ones.
.map((line) => line.trim())
.filter((line) => line.length > 0)
// Neutralize a col-0 markdown block trigger with an invisible ZWSP so the
// git-sync round-trip keeps the line a paragraph. Host lines (`You:` /
// `Speaker N:`) never match and stay byte-exact.
.map((line) =>
GITMOST_MD_BLOCK_TRIGGER_RE.test(line) ? GITMOST_ZWSP + line : line,
);
.filter((line) => line.length > 0);
if (lines.length === 0) return false;
const content = [
@@ -28,7 +28,6 @@ import {
IAiMcpServerCreate,
IAiMcpServerUpdate,
} from "@/features/workspace/services/ai-mcp-server-service.ts";
import { resolveToolAllowlist } from "./ai-mcp-server-form.utils.ts";
const formSchema = z.object({
name: z.string().min(1),
@@ -122,20 +121,13 @@ export default function AiMcpServerForm({
async function handleSubmit(values: FormValues) {
const headers = resolveHeaders();
// An empty tag field means "no restriction" (sent as null) — since #476 the
// server persists a literal `[]` as deny-all (zero tools). But a server that
// was ALREADY deny-all loads into an empty field too; sending null there
// would silently widen it to allow-all on a routine edit, so preserve `[]`.
// See resolveToolAllowlist for the full rationale.
const toolAllowlist = resolveToolAllowlist(values.toolAllowlist, server);
if (isEdit && server) {
const payload: IAiMcpServerUpdate = {
id: server.id,
name: values.name,
transport: values.transport,
url: values.url,
toolAllowlist,
toolAllowlist: values.toolAllowlist,
// Always sent: a blank value clears the stored guidance (server -> null).
instructions: values.instructions,
enabled: values.enabled,
@@ -148,7 +140,7 @@ export default function AiMcpServerForm({
name: values.name,
transport: values.transport,
url: values.url,
toolAllowlist,
toolAllowlist: values.toolAllowlist,
// Blank => server stores null (no guidance).
instructions: values.instructions,
enabled: values.enabled,
@@ -1,29 +0,0 @@
import { describe, it, expect } from "vitest";
import { resolveToolAllowlist } from "./ai-mcp-server-form.utils.ts";
describe("resolveToolAllowlist", () => {
it("sends the typed tools when the field is non-empty", () => {
expect(resolveToolAllowlist(["a", "b"], { toolAllowlist: null })).toEqual([
"a",
"b",
]);
});
it("creates as null (unrestricted) when empty and there is no server", () => {
expect(resolveToolAllowlist([], undefined)).toBeNull();
});
it("sends null for an empty field on a previously-unrestricted server", () => {
expect(resolveToolAllowlist([], { toolAllowlist: null })).toBeNull();
});
it("preserves deny-all: an empty field on a `[]` server stays `[]`, not null", () => {
// The core #476/#477 guard: editing a deny-all server (rename/toggle) with
// an empty tag field must NOT silently widen it to allow-all.
expect(resolveToolAllowlist([], { toolAllowlist: [] })).toEqual([]);
});
it("still sends explicit tools even if the server was deny-all", () => {
expect(resolveToolAllowlist(["x"], { toolAllowlist: [] })).toEqual(["x"]);
});
});
@@ -1,22 +0,0 @@
import { IAiMcpServer } from "@/features/workspace/services/ai-mcp-server-service.ts";
// Resolve the tool allowlist value to persist from the form field.
//
// An empty tag field normally means "no restriction" and is sent as null so
// the server drops the column (all tools allowed). But a server that was
// ALREADY deny-all (a stored literal `[]`, meaning zero tools — creatable via
// the API) loads into the form as an empty field too. Coercing that empty
// field to null on submit would SILENTLY widen a deny-all server to allow-all
// on any routine edit (rename, toggle) — the exact silent-widen class #476
// closed on the read side. So when the edited server was deny-all, preserve
// `[]` (deny-all); only a genuinely-unrestricted server (stored null/absent)
// stays null.
export function resolveToolAllowlist(
fieldValue: string[],
server?: Pick<IAiMcpServer, "toolAllowlist">,
): string[] | null {
if (fieldValue.length > 0) return fieldValue;
const wasDenyAll =
Array.isArray(server?.toolAllowlist) && server.toolAllowlist.length === 0;
return wasDenyAll ? [] : null;
}
@@ -27,9 +27,7 @@ export interface IAiMcpServerCreate {
// Auth headers map (e.g. { Authorization: 'Bearer ...' }). Encrypted on save;
// never returned.
headers?: Record<string, string>;
// Omit/null => no restriction; `[]` is persisted verbatim and means
// deny-all (zero tools) since #476.
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Blank => stored as null.
instructions?: string;
enabled?: boolean;
@@ -45,9 +43,7 @@ export interface IAiMcpServerUpdate {
transport?: McpTransport;
url?: string;
headers?: Record<string, string>;
// Absent => unchanged; null => no restriction; `[]` is persisted verbatim
// and means deny-all (zero tools) since #476.
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Absent => unchanged; blank => cleared.
instructions?: string;
enabled?: boolean;
+1
View File
@@ -41,6 +41,7 @@
"@aws-sdk/s3-request-presigner": "3.1050.0",
"@azure/storage-blob": "12.31.0",
"@clickhouse/client": "^1.18.2",
"@docmost/editor-ext": "workspace:*",
"@docmost/mcp": "workspace:*",
"@docmost/pdf-inspector": "1.9.6",
"@docmost/prosemirror-markdown": "workspace:*",
@@ -49,6 +49,7 @@ import {
FootnotesList,
FootnoteDefinition,
PageEmbed,
Code,
} from '@docmost/editor-ext';
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
import { generateText, getSchema, JSONContent } from '@tiptap/core';
@@ -67,7 +68,12 @@ export const tiptapExtensions = [
link: false,
trailingNode: false,
heading: false,
// #515: replace StarterKit's bundled inline `code` (which inherits tiptap's
// `excludes: "_"`) with the shared editor-ext `Code` below, so the server's
// HTML->PM parse/export keeps code co-occurring with other marks.
code: false,
}),
Code,
Heading,
UniqueID.configure({
types: ['heading', 'paragraph', 'transclusionSource'],
@@ -37,13 +37,10 @@ export class CreateMcpServerDto {
@IsObject()
headers?: Record<string, string>;
// Omit/null => no restriction; `[]` is persisted verbatim and means deny-all
// (zero tools) since #476. @IsOptional() skips validation for null as well,
// so an explicit null is accepted.
@IsOptional()
@IsArray()
@IsString({ each: true })
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored guidance ("how/when to use this server's tools") injected
// into the agent system prompt next to the tool descriptions (#180). Trusted,
@@ -38,13 +38,10 @@ export class UpdateMcpServerDto {
@IsObject()
headers?: Record<string, string>;
// Absent => unchanged; null => no restriction; `[]` is persisted verbatim
// and means deny-all (zero tools) since #476. @IsOptional() skips validation
// for null as well, so an explicit null is accepted.
@IsOptional()
@IsArray()
@IsString({ each: true })
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Absent => unchanged; blank => cleared
// (stored as null by the repo). Capped to bound prompt/token size.
@@ -1,169 +0,0 @@
import { type Tool } from 'ai';
import { McpClientsService } from './mcp-clients.service';
/**
* Tool-allowlist filtering semantics on the merged external toolset (#476).
*
* COVERAGE CHOICE (documented per issue #476): the full corrupt-row chain
* (DB value -> repo normalizeRow -> toolsFor filter) is covered on TWO levels
* instead of one live-stub-MCP-server integration test:
* (a) apps/server/test/integration/ai-mcp-server-repo.int-spec.ts pins the
* repo read/write semantics against a real Postgres — `[]` round-trips
* as jsonb `[]`, a present-but-corrupt value fails CLOSED to `[]` with
* an error log;
* (b) THIS spec pins what the toolset builder does with the repo's output —
* null = unrestricted, `['alpha']` = only alpha, `[]` (including the
* corrupt-row fallback) = ZERO tools.
* Together they prove the end-to-end property "corrupt/empty allowlist can
* never widen to all tools" without a live stub HTTP MCP server.
*
* The drive path mirrors mcp-namespacing.spec.ts: stub the repo's listEnabled,
* spy the private `connect` to return a fake client, inspect the merged keys.
*/
function fakeTool(): Tool {
return { description: 'x', inputSchema: undefined } as unknown as Tool;
}
interface FakeServer {
id: string;
name: string;
transport: string;
url: string;
headersEnc: string | null;
toolAllowlist: string[] | null;
}
function server(
over: Partial<FakeServer> & { id: string; name: string },
): FakeServer {
return {
transport: 'http',
url: 'https://example.com/mcp',
headersEnc: null,
toolAllowlist: null,
...over,
};
}
/**
* Build a service whose repo returns `servers` and whose fake clients expose
* `rawTools` from tools(). Returns the merged tool keys produced by toolsFor.
*/
async function mergedKeysFor(
servers: FakeServer[],
rawTools: Record<string, Tool>,
): Promise<string[]> {
const repoStub = {
listEnabled: jest.fn().mockResolvedValue(servers),
};
const service = new McpClientsService(repoStub as never, {} as never);
jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => unknown },
'connect',
)
.mockImplementation(() =>
Promise.resolve({
tools: () => Promise.resolve(rawTools),
close: () => Promise.resolve(),
}),
);
const toolset = await service.toolsFor('ws-1');
// Release the lease so the service does not hold the fake clients open.
await Promise.all(toolset.clients.map((c) => c.close()));
return Object.keys(toolset.tools);
}
describe('external MCP tool-allowlist filtering (via toolsFor, #476)', () => {
afterEach(() => jest.restoreAllMocks());
const RAW = () => ({
alpha: fakeTool(),
beta: fakeTool(),
gamma: fakeTool(),
});
it("['alpha'] lets ONLY alpha through", async () => {
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: ['alpha'] })],
RAW(),
);
expect(keys).toEqual(['srv_alpha']);
});
it('null (no restriction) lets every tool through', async () => {
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: null })],
RAW(),
);
expect(keys.sort()).toEqual(['srv_alpha', 'srv_beta', 'srv_gamma']);
});
it('[] (deny-all) yields ZERO tools — an empty array is authoritative, not falsy (#476)', async () => {
// This is the regression the #476 change guards: `[]` used to fall through
// the old `allow.length > 0` check and expose ALL tools. It must expose NONE.
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: [] })],
RAW(),
);
expect(keys).toEqual([]);
});
it('the corrupt-row fallback ([] from the repo) also yields ZERO tools (#476)', async () => {
// The repo turns a present-but-corrupt tool_allowlist into `[]` (fail-closed,
// see normalizeRow in ai-mcp-server.repo.ts + the int-spec); this pins that
// the toolset builder honours that fallback as deny-all rather than allow-all.
const corruptFallback: string[] = [];
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: corruptFallback })],
RAW(),
);
expect(keys).toEqual([]);
});
it('allowlisted names not exposed by the server are ignored (no phantom tools)', async () => {
const keys = await mergedKeysFor(
[
server({
id: 'id-1',
name: 'srv',
toolAllowlist: ['alpha', 'does-not-exist'],
}),
],
RAW(),
);
expect(keys).toEqual(['srv_alpha']);
});
it('a deny-all server contributes no prompt instructions (0 tools merged)', async () => {
const repoStub = {
listEnabled: jest.fn().mockResolvedValue([
{
...server({ id: 'id-1', name: 'srv', toolAllowlist: [] }),
instructions: 'use the tools wisely',
},
]),
};
const service = new McpClientsService(repoStub as never, {} as never);
jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => unknown },
'connect',
)
.mockImplementation(() =>
Promise.resolve({
tools: () => Promise.resolve(RAW()),
close: () => Promise.resolve(),
}),
);
const toolset = await service.toolsFor('ws-1');
await Promise.all(toolset.clients.map((c) => c.close()));
expect(Object.keys(toolset.tools)).toEqual([]);
// mergeNamespaced reported 0 contributed tools, so no guidance is attached.
expect(toolset.instructions).toEqual([]);
});
});
@@ -285,13 +285,9 @@ export class McpClientsService {
try {
client = await this.connectWithTimeout(server, CONNECT_TIMEOUT_MS);
const raw = await withTimeout(client.tools(), CONNECT_TIMEOUT_MS);
// Allowlist semantics (#476): null/absent = no restriction (all tools);
// ANY array — including `[]` — is authoritative, so an EMPTY allowlist
// yields ZERO tools (deny-all). Do NOT add a `.length > 0` escape here:
// that read `[]` as falsy and silently widened deny-all to allow-all
// (the repo also fails corrupt rows closed to `[]` for the same reason).
const allow = server.toolAllowlist;
const picked = Array.isArray(allow) ? pick(raw, allow) : raw;
const picked =
Array.isArray(allow) && allow.length > 0 ? pick(raw, allow) : raw;
// Bound each tool's execute with a per-call total-timeout guard before
// merging, so a single chatty-but-stuck call is aborted after the cap.
const guarded = wrapToolsWithCallTimeout(picked, callTimeoutMs);
@@ -100,8 +100,7 @@ export class McpServersService {
transport: dto.transport,
url: dto.url,
headersEnc,
// undefined => unchanged; null => no restriction; `[]` is persisted
// verbatim and means deny-all (#476).
// undefined => unchanged; [] / value handled by repo (empty => null).
toolAllowlist: dto.toolAllowlist,
// undefined => unchanged; blank => cleared (null) by the repo.
instructions: dto.instructions,
@@ -53,8 +53,10 @@ import {
extractPageSlugId,
} from '../../../integrations/export/utils';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../../../integrations/import/utils/foreign-markdown';
import {
markdownToProseMirror,
normalizeForeignMarkdown,
} from '@docmost/prosemirror-markdown';
import { WatcherService } from '../../watcher/watcher.service';
import { sql } from 'kysely';
import { TransclusionService } from '../transclusion/transclusion.service';
@@ -35,25 +35,4 @@ describe('jsonbBind', () => {
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
// preserveEmpty (#476): opts a column OUT of the empty-to-null collapse so an
// empty container is persisted verbatim (e.g. `[]` = deny-all for
// tool_allowlist). null stays null regardless of the flag.
describe('preserveEmpty', () => {
it('returns a (non-null) bind for an empty array', () => {
const out = jsonbBind([], { preserveEmpty: true });
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
it('returns a (non-null) bind for an empty object', () => {
const out = jsonbBind({}, { preserveEmpty: true });
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
it('still returns null for null (null means null, flag or not)', () => {
expect(jsonbBind(null, { preserveEmpty: true })).toBeNull();
});
});
});
@@ -78,9 +78,7 @@ export class AiMcpServerRepo {
headersEnc: values.headersEnc ?? null,
// jsonb column: the postgres driver would otherwise encode a JS array as
// a Postgres array literal. Bind the JSON text and cast it to jsonb.
// preserveEmpty (#476): `[]` is a real value here (deny-all), distinct
// from null ("no restriction") — it must round-trip as `[]`, not null.
toolAllowlist: jsonbBind(values.toolAllowlist, { preserveEmpty: true }),
toolAllowlist: jsonbBind(values.toolAllowlist),
// Plain text column: blank/whitespace-only guidance is stored as null.
instructions: blankToNull(values.instructions),
enabled: values.enabled ?? true,
@@ -113,10 +111,7 @@ export class AiMcpServerRepo {
if (patch.url !== undefined) set.url = patch.url;
if (patch.headersEnc !== undefined) set.headersEnc = patch.headersEnc;
if (patch.toolAllowlist !== undefined) {
// preserveEmpty (#476): see insert — `[]` (deny-all) must not become null.
set.toolAllowlist = jsonbBind(patch.toolAllowlist, {
preserveEmpty: true,
});
set.toolAllowlist = jsonbBind(patch.toolAllowlist);
}
if (patch.instructions !== undefined) {
// Blank/whitespace-only guidance clears the column (stored as null).
@@ -163,9 +158,7 @@ export function blankToNull(value: string | null | undefined): string | null {
* fix), so the driver hands back a string like `'["a","b"]'` rather than an
* array. Be tolerant: normalize a JSON string to its value, then accept it only
* if it is an array of strings; null / a non-array / unparseable value / an
* array with a non-string element all become null. NOTE: null here only means
* "could not parse" the null-vs-deny-all policy decision lives in
* normalizeRow (#476: present-but-corrupt fails CLOSED to `[]`).
* array with a non-string element all become null (unrestricted).
*/
export function parseToolAllowlist(value: unknown): string[] | null {
// Shape guard only; the legacy double-encoding self-heal lives in
@@ -180,20 +173,17 @@ export function parseToolAllowlist(value: unknown): string[] | null {
/**
* Normalize a DB row so `toolAllowlist` is always `string[] | null`.
*
* FAIL-CLOSED (#476): a stored value that is PRESENT but cannot be parsed into
* a string[] (corrupt JSON, a non-array, non-string elements) degrades to `[]`
* = deny-all, so a corrupted allowlist can never silently widen to "the agent
* gets ALL of the server's tools" (the old fail-open null). An error line is
* logged (server id only, never the contents) so the admin can repair the row.
* A column that is truly NULL/absent stays `null` = "no restriction".
* FAIL-OPEN logging: a stored value that is present but cannot be parsed into a
* string[] (corrupt JSON, a non-array, non-string elements) degrades to `null` =
* "no restriction", so the agent silently gets ALL of the server's tools. Log
* one line (server id only, never the contents) so that widening is not silent.
*/
function normalizeRow(row: AiMcpServer): AiMcpServer {
const parsed = parseToolAllowlist(row.toolAllowlist);
if (parsed === null && row.toolAllowlist != null) {
logger.error(
`Corrupt tool_allowlist for MCP server ${row.id}; failing closed (NO tools allowed) — re-save the server's allowlist to repair it`,
logger.warn(
`Corrupt tool_allowlist for MCP server ${row.id}; ignoring it (no tool restriction applied)`,
);
return { ...row, toolAllowlist: [] };
}
return { ...row, toolAllowlist: parsed };
}
+7 -19
View File
@@ -78,30 +78,18 @@ export function violatedConstraint(err: unknown): string | undefined {
* verbatim); `::jsonb` then parses it into a real array/object. Read-side
* parsers repair rows written the old buggy way without a migration.
*
* Returns `null` for null/undefined. By default it ALSO returns `null` for
* "empty" values (an empty array, or an object with no own enumerable keys)
* most callers treat empty as "clear/unset", so an empty config never
* round-trips as `[]`/`{}`.
*
* `preserveEmpty` (issue #476) opts a column OUT of that empty-to-null
* normalization so `[]`/`{}` are persisted as real jsonb values. Needed where
* empty and null mean DIFFERENT things: an empty `tool_allowlist` is
* deny-all ("zero tools allowed"), while null is "no restriction" collapsing
* `[]` to null silently widened deny-all to allow-all. Deliberately an opt-in
* flag, NOT a global change: the other jsonb callers (model_config, source)
* keep the empty-means-unset contract.
* Returns `null` for null/undefined and for "empty" values (an empty array, or
* an object with no own enumerable keys) callers treat empty as "clear/unset",
* so an empty allowlist/config never round-trips as `[]`/`{}`.
*/
export function jsonbBind<T>(
value: T | null | undefined,
opts?: { preserveEmpty?: boolean },
): RawBuilder<T> | null {
if (value === null || value === undefined) return null;
if (!opts?.preserveEmpty) {
if (Array.isArray(value)) {
if (value.length === 0) return null;
} else if (typeof value === 'object') {
if (Object.keys(value as object).length === 0) return null;
}
if (Array.isArray(value)) {
if (value.length === 0) return null;
} else if (typeof value === 'object') {
if (Object.keys(value as object).length === 0) return null;
}
return sql<T>`${JSON.stringify(value)}::text::jsonb`;
}
@@ -22,10 +22,12 @@ import { v7 } from 'uuid';
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import {
markdownToProseMirror,
normalizeForeignMarkdown,
} from '@docmost/prosemirror-markdown';
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
import { formatImportHtml } from '../utils/import-formatter';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import {
buildAttachmentCandidates,
collectMarkdownAndHtmlFiles,
@@ -18,8 +18,10 @@ import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { TiptapTransformer } from '@hocuspocus/transformer';
import * as Y from 'yjs';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import {
markdownToProseMirror,
normalizeForeignMarkdown,
} from '@docmost/prosemirror-markdown';
import {
FileTaskStatus,
FileTaskType,
@@ -1,6 +1,5 @@
import { Kysely, sql } from 'kysely';
import { randomUUID } from 'node:crypto';
import { Logger } from '@nestjs/common';
import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo';
import { getTestDb, destroyTestDb, createWorkspace } from './db';
@@ -55,11 +54,7 @@ describe('AiMcpServerRepo tool_allowlist jsonb round-trip [integration]', () =>
expect(Array.isArray(found?.toolAllowlist)).toBe(true);
});
// #476 (deliberate behaviour change): an empty allowlist used to be
// normalized to SQL NULL, which downstream means "no restriction" — so an
// admin's deny-all `[]` silently became allow-all. It must now round-trip as
// a real jsonb `[]` (deny-all), distinct from NULL.
it('an empty allowlist round-trips as jsonb [] (deny-all), not null (#476)', async () => {
it('an empty allowlist is normalized to null (no restriction), not []', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
@@ -67,27 +62,7 @@ describe('AiMcpServerRepo tool_allowlist jsonb round-trip [integration]', () =>
url: 'https://example.com/mcp',
toolAllowlist: [],
});
// The column holds a real (empty) jsonb ARRAY, not SQL NULL.
expect(await jsonbTypeof(row.id)).toBe('array');
expect((await repo.findById(row.id, ws))?.toolAllowlist).toEqual([]);
});
it('update to [] persists jsonb [] and update to null clears to SQL NULL (#476)', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
toolAllowlist: ['search'],
});
// Deny-all via update: [] must survive as a real jsonb array.
await repo.update(row.id, ws, { toolAllowlist: [] });
expect(await jsonbTypeof(row.id)).toBe('array');
expect((await repo.findById(row.id, ws))?.toolAllowlist).toEqual([]);
// Explicit clear (null) still means "no restriction" = SQL NULL.
await repo.update(row.id, ws, { toolAllowlist: null });
// The column is SQL NULL, so jsonb_typeof returns SQL NULL (JS null).
expect(await jsonbTypeof(row.id)).toBeNull();
expect((await repo.findById(row.id, ws))?.toolAllowlist).toBeNull();
});
@@ -117,60 +92,23 @@ describe('AiMcpServerRepo tool_allowlist jsonb round-trip [integration]', () =>
expect(healed?.toolAllowlist).toEqual(['alpha', 'beta']);
});
// #476 (deliberate behaviour change, replaces the old FAIL-OPEN pin): a
// present-but-corrupt tool_allowlist used to degrade to null ("no
// restriction"), silently handing the agent ALL of the server's tools. It
// must now FAIL CLOSED to `[]` (deny-all) and log an error.
it('FAIL-CLOSED: a present-but-corrupt tool_allowlist reads back as [] (deny-all) + error log (#476)', async () => {
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined);
try {
// The column is PRESENT but does not parse into a string[] — a jsonb
// string scalar holding unparseable text (a truncated legacy write).
const id = randomUUID();
await sql`
INSERT INTO ai_mcp_servers (id, workspace_id, name, transport, url, tool_allowlist)
VALUES (
${id}, ${ws}, ${`srv-${id}`}, 'http', 'https://example.com/mcp',
to_jsonb(${'{oops'}::text)
)
`.execute(db);
// Sanity: the column is present (a jsonb string scalar), not SQL NULL.
expect(await jsonbTypeof(id)).toBe('string');
// ...and the read degrades to [] (fail-closed deny-all), not null.
expect((await repo.findById(id, ws))?.toolAllowlist).toEqual([]);
// The narrowing is not silent: an error names the server id (never the
// corrupt contents).
expect(errorSpy).toHaveBeenCalledWith(
expect.stringContaining(`Corrupt tool_allowlist for MCP server ${id}`),
);
expect(
errorSpy.mock.calls.some((c) => String(c[0]).includes('{oops')),
).toBe(false);
} finally {
errorSpy.mockRestore();
}
});
it('FAIL-CLOSED: corrupt non-array JSON (an object) also reads back as [] (#476)', async () => {
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined);
try {
const id = randomUUID();
await sql`
INSERT INTO ai_mcp_servers (id, workspace_id, name, transport, url, tool_allowlist)
VALUES (
${id}, ${ws}, ${`srv-${id}`}, 'http', 'https://example.com/mcp',
to_jsonb(${'{"not":"an array"}'}::text)
)
`.execute(db);
expect(await jsonbTypeof(id)).toBe('string');
expect((await repo.findById(id, ws))?.toolAllowlist).toEqual([]);
} finally {
errorSpy.mockRestore();
}
it('FAIL-OPEN: a present-but-corrupt tool_allowlist reads back as null (no restriction)', async () => {
// #185 re-review pt 8: normalizeRow's fail-open branch — the column is
// PRESENT but does not parse into a string[] (here a jsonb string scalar
// holding non-array JSON). The read must degrade to `null` ("no restriction"),
// not crash. (A warn is logged with the server id; not asserted here.)
const id = randomUUID();
await sql`
INSERT INTO ai_mcp_servers (id, workspace_id, name, transport, url, tool_allowlist)
VALUES (
${id}, ${ws}, ${`srv-${id}`}, 'http', 'https://example.com/mcp',
to_jsonb(${'{"not":"an array"}'}::text)
)
`.execute(db);
// Sanity: the column is present (a jsonb string scalar), not SQL NULL.
expect(await jsonbTypeof(id)).toBe('string');
// ...yet the read degrades to null (fail-open).
expect((await repo.findById(id, ws))?.toolAllowlist).toBeNull();
});
});
+1
View File
@@ -1,4 +1,5 @@
export * from "./lib/trailing-node";
export * from "./lib/code";
export * from "./lib/comment/comment";
export * from "./lib/utils";
export * from "./lib/math";
+20
View File
@@ -0,0 +1,20 @@
import { Code as TiptapCode } from "@tiptap/extension-code";
// #515: canonical inline `code` mark for Docmost.
//
// Tiptap's stock Code mark (via StarterKit) declares `excludes: "_"`, which
// makes it exclude EVERY other inline mark: applying `code` drops any co-
// occurring bold/italic/… on both the HTML->PM import and editor transactions.
// That silently stripped emphasis adjacent to inline code (`` **`--flag`** ``
// lost its bold on markdown import). CommonMark nests them (`<strong><code>`),
// so Docmost lets `code` combine with all marks by overriding `excludes` to the
// empty string (excludes nothing).
//
// This is the SINGLE shared source imported by the live editor, the collab
// server and the comment editor schemas. The markdown-import mirror in
// @docmost/prosemirror-markdown re-declares the same override locally (it must
// not pull this React-aware package into its node runtime) and a parity test
// keeps the two in lockstep.
export const Code = TiptapCode.extend({
excludes: "",
});
+8 -2
View File
@@ -72,7 +72,13 @@ export async function stabilizePageFile(
* keeps re-pulls of an unchanged page byte-identical (no churn, loop-guard).
*/
export async function stabilizePageBody(content: unknown): Promise<string> {
const md1 = convertProseMirrorToMarkdown(content);
// git-sync is the LOSSLESS mirror path, so run the serializer in `strict`
// mode: a node/mark type the converter has no case for (e.g. one added to the
// schema without a matching serializer arm) throws a ConverterLossError here
// rather than silently degrading — surfacing the loss loudly at write time
// instead of committing a lossy file. Valid content (every current schema type
// has a case) is unaffected.
const md1 = convertProseMirrorToMarkdown(content, { strict: true });
const doc2 = await markdownToProseMirror(md1);
return convertProseMirrorToMarkdown(doc2);
return convertProseMirrorToMarkdown(doc2, { strict: true });
}
+18
View File
@@ -4,6 +4,7 @@ import { stabilizePageFile, type PageMeta } from '../src/engine/stabilize.js';
// global DOM via jsdom at module load time (required for @tiptap/html under Node).
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { parseDocmostMarkdown } from '@docmost/prosemirror-markdown';
import { ConverterLossError } from '@docmost/prosemirror-markdown';
// stabilize.ts (SPEC §11 normalize-on-write) was 0% covered (only the gated e2e
// touched it). stabilizePageFile is import-testable: build a small ProseMirror
@@ -66,6 +67,23 @@ describe('stabilizePageFile — normalize-on-write fixpoint (SPEC §11)', () =>
expect(body1).toContain('data-src="/d.drawio"');
});
it('runs the serializer in STRICT mode — an unmappable node throws, not a lossy write (#493)', async () => {
// git-sync is the lossless mirror path: a node type the converter has no
// case for (here a fabricated one, standing in for a schema type added
// without a matching serializer arm) must surface loudly at write time
// rather than being silently flattened into a lossy .md file.
const content = {
type: 'doc',
content: [
{ type: 'paragraph', content: [{ type: 'text', text: 'ok' }] },
{ type: 'quantumWidget', content: [{ type: 'text', text: 'lost?' }] },
],
};
await expect(stabilizePageFile(content, meta)).rejects.toBeInstanceOf(
ConverterLossError,
);
});
it('already-stable content is unchanged by the pass (idempotent)', async () => {
// Plain prose is already a fixpoint; stabilizing it once and twice agree.
const content = {
+24 -3
View File
@@ -10,7 +10,10 @@ import { JSDOM } from "jsdom";
// handled there). MCP consumes it directly instead of maintaining its own
// drifted marked pipeline; only the collab/yjs write glue and the footnote
// canonicalization wrapper stay mcp-side.
import { markdownToProseMirror } from "@docmost/prosemirror-markdown";
import {
markdownToProseMirror,
normalizeAgentMarkdown,
} from "@docmost/prosemirror-markdown";
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
import { withPageLock } from "./page-lock.js";
import {
@@ -20,6 +23,7 @@ import {
} from "@docmost/prosemirror-markdown";
import { canonicalizeFootnotes } from "./footnote-canonicalize.js";
import { normalizeAndMergeFootnotes } from "./footnote-normalize-merge.js";
import { regraftResolvedComments } from "./comment-anchor.js";
import { VerifyReport } from "./diff.js";
import { acquireCollabSession } from "./collab-session.js";
@@ -97,6 +101,15 @@ global.WebSocket = WebSocket;
* plain `markdownToProseMirror` (no canonicalization) safe now because inline
* `^[body]` footnotes carry their body at the reference point, so a comment can
* no longer produce a reference-less footnote definition to be dropped.
*
* #493: `normalizeAgentMarkdown` runs FIRST, so an agent's `updatePageMarkdown`
* body gets the SAME GFM `[^id]` reference-footnote -> inline `^[body]` rewrite as
* the server import path (instead of the reference leaking as literal text / a
* bogus link). It DELIBERATELY does NOT strip a leading YAML front-matter block:
* a full-body agent rewrite that opens with a `---…---` is (almost) always a
* horizontalRule the serializer emitted, and stripping it would silently drop the
* page's leading content (#493 review). The front-matter strip stays on the
* server FILE-import boundary only (`normalizeForeignMarkdown`).
*/
export async function markdownToProseMirrorCanonical(
markdownContent: string,
@@ -105,7 +118,9 @@ export async function markdownToProseMirrorCanonical(
// canonicalizing, so the canonicalizer re-hangs references and drops the
// now-orphaned duplicate definitions.
return canonicalizeFootnotes(
normalizeAndMergeFootnotes(await markdownToProseMirror(markdownContent)),
normalizeAndMergeFootnotes(
await markdownToProseMirror(normalizeAgentMarkdown(markdownContent)),
),
);
}
@@ -328,6 +343,12 @@ export async function updatePageContentRealtime(
pageId,
collabToken,
baseUrl,
() => tiptapJson,
// #493: an agent read HIDES resolved-comment anchors (#337), so the markdown
// it sends here no longer carries them — a naive full rewrite would erase
// every resolved comment mark. Re-graft the resolved marks from the LIVE doc
// onto the matching text in the freshly-imported body. Active comments are
// untouched (they ride through the markdown themselves); a resolved span whose
// text the agent changed simply does not re-anchor and is dropped.
(liveDoc) => regraftResolvedComments(liveDoc, tiptapJson),
);
}
+112 -3
View File
@@ -312,10 +312,9 @@ export function canAnchorInDoc(doc: any, selection: string): boolean {
function spliceCommentMark(
blockContent: any[],
match: AnchorMatch,
commentId: string,
commentMark: any,
): void {
const { startChild, startOffset, endChild, endOffset } = match;
const commentMark = makeCommentMark(commentId);
const fragments: any[] = [];
for (let k = startChild; k <= endChild; k++) {
@@ -451,6 +450,22 @@ export function applyAnchorInDoc(
doc: any,
selection: string,
commentId: string,
): boolean {
return applyCommentMarkInDoc(doc, selection, makeCommentMark(commentId));
}
/**
* Core of {@link applyAnchorInDoc}, but splices an ARBITRARY comment mark object
* (not just a fresh `{ commentId, resolved:false }`) across the first matching
* range. This lets a caller re-apply a mark that carries `resolved:true` and any
* other stored attrs. Depth-first (same order as canAnchorInDoc); mutates in
* place on the first matching block and returns true, else returns false without
* mutating.
*/
export function applyCommentMarkInDoc(
doc: any,
selection: string,
commentMark: any,
): boolean {
const { selection: effective, found } = resolveAnchorSelection(doc, selection);
if (!found) return false;
@@ -459,7 +474,7 @@ export function applyAnchorInDoc(
if (!Array.isArray(node.content)) return false;
const match = findAnchorInBlock(node.content, effective);
if (match) {
spliceCommentMark(node.content, match, commentId);
spliceCommentMark(node.content, match, commentMark);
return true;
}
for (const child of node.content) {
@@ -471,3 +486,97 @@ export function applyAnchorInDoc(
};
return visit(doc, 0);
}
/** A resolved inline-comment span lifted from a doc: its mark + anchored text. */
export interface ResolvedCommentSpan {
commentId: string;
/** The full comment mark (carrying `resolved:true` + any stored attrs). */
mark: any;
/** The concatenated raw text the mark spans — used as the re-anchor selection. */
text: string;
}
/** True when a text node carries a RESOLVED comment mark; returns that mark. */
function resolvedCommentMarkOf(node: any): any | null {
if (!node || node.type !== "text" || !Array.isArray(node.marks)) return null;
return (
node.marks.find(
(m: any) =>
m && m.type === "comment" && m.attrs?.resolved === true && m.attrs?.commentId,
) || null
);
}
/**
* Collect every RESOLVED inline-comment span in `doc`, in document order. Within
* each block's direct content, a maximal run of consecutive text nodes sharing
* the same resolved `commentId` is ONE span; its concatenated raw text is the
* selection used to re-anchor it elsewhere. Active (unresolved) comment marks are
* ignored they survive a markdown round-trip on their own (a page read emits
* their `<span data-comment-id>` wrapper), whereas resolved anchors are hidden
* from agent reads (#337) and would be erased by a full-body markdown rewrite.
*/
export function collectResolvedCommentSpans(doc: any): ResolvedCommentSpan[] {
const spans: ResolvedCommentSpan[] = [];
const visit = (node: any, depth: number): void => {
if (depth > MAX_DEPTH || !node || typeof node !== "object") return;
if (!Array.isArray(node.content)) return;
const content = node.content;
let i = 0;
while (i < content.length) {
const mark = resolvedCommentMarkOf(content[i]);
if (mark) {
const commentId = mark.attrs.commentId;
let text = "";
let j = i;
while (j < content.length) {
const mj = resolvedCommentMarkOf(content[j]);
if (!mj || mj.attrs.commentId !== commentId) break;
text += typeof content[j].text === "string" ? content[j].text : "";
j++;
}
if (text.length > 0) spans.push({ commentId, mark, text });
i = j > i ? j : i + 1;
} else {
i++;
}
}
for (const child of content) {
if (child && typeof child === "object" && Array.isArray(child.content)) {
visit(child, depth + 1);
}
}
};
visit(doc, 0);
return spans;
}
/**
* Re-graft RESOLVED comment marks from `oldDoc` onto matching text ranges in
* `newDoc`, returning a NEW doc (never mutates the inputs).
*
* WHY (#493): an agent read hides resolved-comment anchors (#337), so the
* markdown it sends to a FULL-body rewrite (`updatePageMarkdown`) no longer
* carries them a naive full write would erase every resolved comment mark.
* This restores them: each resolved span from the previous document is re-anchored
* onto the SAME text in the newly-imported body (first occurrence, using the
* shared anchoring / markdown-strip fallback), preserving `resolved:true` and the
* stored attrs. A span whose text the agent changed or deleted simply does not
* re-anchor and is dropped (its anchor is gone; it was already resolved). Active
* comments are untouched they ride through the markdown themselves.
*/
export function regraftResolvedComments<T = any>(oldDoc: any, newDoc: T): T {
if (!newDoc || typeof newDoc !== "object") return newDoc;
const spans = collectResolvedCommentSpans(oldDoc);
if (spans.length === 0) return newDoc;
const out =
typeof structuredClone === "function"
? structuredClone(newDoc)
: (JSON.parse(JSON.stringify(newDoc)) as T);
for (const span of spans) {
// Clone the mark so the new document never shares a mark object with oldDoc.
const markClone = { type: "comment", attrs: { ...span.mark.attrs } };
applyCommentMarkInDoc(out, span.text, markClone);
}
return out;
}
+21 -93
View File
@@ -1,64 +1,30 @@
/**
* Locator normalization: strip inline markdown wrappers and trailing
* decoration from a LOCATOR string so a find/anchor that the model wrote with
* markdown (or a stray emoji) can still match the document's plain text.
* Locator normalization helpers for mcp. The two PRIMITIVES
* `stripInlineMarkdown` (lenient locator normalizer) and `stripWrappersAndLinks`
* (strict balanced-wrapper/link collapse) live in the canonical package
* `@docmost/prosemirror-markdown` (#493 dedup: they used to be forked verbatim
* here). This module now only re-exports `stripInlineMarkdown` and adds the two
* mcp-only helpers built on top: `stripBalancedWrappers` and `closestBlockHint`.
*
* This is used ONLY as a fallback for LOCATING (after an exact match fails);
* it is never applied to replacement text or inserted node content, so no
* formatting is ever lost.
* They are used ONLY as a fallback for LOCATING (after an exact match fails) and
* for formatting-vs-plain intent detection; never applied to replacement text or
* inserted node content, so no formatting is ever lost.
*/
import {
stripInlineMarkdown,
stripWrappersAndLinks,
} from "@docmost/prosemirror-markdown";
/** Maximum unwrap passes, so pathological/nested input cannot loop forever. */
const MAX_PASSES = 8;
// Re-export the canonical locator normalizer so mcp call sites keep importing it
// from `./text-normalize.js` unchanged.
export { stripInlineMarkdown };
/**
* Inline emphasis/code/strikethrough wrappers, strong BEFORE emphasis so
* `**x**` collapses to `x` rather than leaving a stray `*x*`. Each pattern is
* non-greedy and capture group 1 is the inner text. Applied repeatedly until
* the string stops changing (nested wrappers like `**_x_**`).
*/
const WRAPPER_PATTERNS: RegExp[] = [
/\*\*([^*]+?)\*\*/g, // **x**
/__([^_]+?)__/g, // __x__
/~~([^~]+?)~~/g, // ~~x~~
/\*([^*]+?)\*/g, // *x*
/_([^_]+?)_/g, // _x_
/``([^`]+?)``/g, // ``x``
/`([^`]+?)`/g, // `x`
];
/** Links/images -> their visible text. `!?` covers both `[t](u)` and `![a](s)`. */
const LINK_IMAGE_RE = /!?\[([^\]]*)\]\([^)]*\)/g;
/**
* Apply ONLY the two balanced/link passes shared by both normalizers: first
* collapse links/images to their visible text, then collapse balanced inline
* wrappers repeatedly until stable. Does NOT trim decoration, does NOT guard
* against an empty result it returns exactly the transformed string.
*/
function stripWrappersAndLinks(s: string): string {
// 1. Links/images -> their visible text.
let out = s.replace(LINK_IMAGE_RE, "$1");
// 2. Strip balanced wrappers, repeating until the string is stable so nested
// wrappers (`**_x_**`) and adjacent runs both collapse.
for (let pass = 0; pass < MAX_PASSES; pass++) {
const before = out;
for (const re of WRAPPER_PATTERNS) {
out = out.replace(re, "$1");
}
if (out === before) break;
}
return out;
}
/**
* STRICT formatting detector distinct from the lenient locator
* normalization below. It strips ONLY what unambiguously is markdown markup:
* 1. links/images `[text](url)` -> `text`, `![alt](src)` -> `alt`, and
* 2. balanced inline `**`/`__`/`~~`/`*`/`_`/`` ` `` wrappers (repeat-until-stable),
* and DELIBERATELY does NOT trim leading/trailing whitespace, emoji, or lone
* marker chars (the lenient extras `stripInlineMarkdown` does in its step 3).
* STRICT formatting detector distinct from the lenient locator normalization.
* It strips ONLY what unambiguously is markdown markup (links/images to visible
* text, and balanced inline `**`/`__`/`~~`/`*`/`_`/`` ` `` wrappers) and
* DELIBERATELY does NOT trim leading/trailing whitespace, emoji, or lone marker
* chars (the lenient extras `stripInlineMarkdown` does).
*
* It exists ONLY to recognize formatting-vs-plain INTENT in `applyTextEdits`
* (deciding whether find/replace differ purely by markdown markers). Because it
@@ -77,44 +43,6 @@ export function stripBalancedWrappers(s: string): string {
return stripWrappersAndLinks(s);
}
/**
* Conservatively strip inline markdown from a locator string.
*
* Deterministic, order-fixed steps:
* 1. Links/images: `[text](url)` -> `text`, `![alt](src)` -> `alt`.
* 2. Balanced inline wrappers (strong before emphasis, code, strikethrough),
* applied repeatedly until stable for nested cases.
* 3. Trim leading/trailing decoration only: whitespace, leftover marker chars
* (`* _ ~ \``) and emoji. Letters/digits and sentence punctuation (`.`/`,`
* etc.) are NEVER trimmed.
*
* If the result is empty (e.g. the input was only markers like `***`), the
* ORIGINAL string is returned so a locator can never normalize down to "" and
* match everything.
*/
export function stripInlineMarkdown(s: string): string {
if (typeof s !== "string" || s.length === 0) return s;
// 1 + 2. Shared link/image and balanced-wrapper passes.
let out = stripWrappersAndLinks(s);
// 3. Trim leading/trailing decoration: whitespace, leftover markdown markers,
// and emoji (Extended_Pictographic plus the VS16 / ZWJ joiners, plus the
// regional-indicator range U+1F1E6–U+1F1FF for flag emoji, which are NOT
// Extended_Pictographic). The `u` flag enables the Unicode property escape.
// Anchored runs only — interior text and sentence punctuation are untouched.
const DECORATION =
"[\\s*_~\\x60\\p{Extended_Pictographic}\\u{1F1E6}-\\u{1F1FF}\\u{FE0F}\\u{200D}]+";
out = out
.replace(new RegExp("^" + DECORATION, "u"), "")
.replace(new RegExp(DECORATION + "$", "u"), "");
// 4. Never normalize a locator down to nothing.
if (out.length === 0) return s;
return out;
}
/**
* Build a bounded "closest text" hint for an anchor/find MISS, shared by
* editPageText (json-edit) and createComment (client) so both surface the
-65
View File
@@ -434,71 +434,6 @@ async function main() {
}
}
// 6h. markdown converter fixpoint (#476): pins the converter fixpoint
// THROUGH the live server/collab path, not just the package tests. The
// unit corpus (docmost-md-roundtrip) proves the converter alone is a
// fixpoint; this asserts the property survives the real pipeline — export
// (REST read, PM -> MD) -> import (MD -> PM -> collab replace -> server
// persistence) -> export — where the server schema, the Yjs structural
// diff or the collab write path could still mangle the doc while every
// unit test stays green. importPageMarkdown is the designed inverse of
// exportPageMarkdown (the self-contained envelope with meta/comments
// blocks); updatePageMarkdown (client.updatePage) takes plain authoring
// markdown and would re-import the envelope blocks as literal content.
{
const FIXMD = [
"# Fixpoint heading",
"",
"Paragraph with **bold**, *italic* and a [link](https://example.com).",
"",
"## Second level",
"",
"- bullet one",
"- bullet two",
"",
"1. ordered one",
"2. ordered two",
"",
"```js",
"const answer = 42; // code block must survive byte-identically",
"```",
"",
"| A | B |",
"| --- | --- |",
"| one | two |",
"",
":::info",
"Callout body.",
":::",
].join("\n");
const fx = await client.createPage("E2E md fixpoint " + Date.now(), FIXMD, spaceId);
const fxid = fx.data.id;
try {
const md1 = await client.exportPageMarkdown(fxid);
await client.importPageMarkdown(fxid, md1);
await new Promise((r) => setTimeout(r, 16000)); // wait for server persistence
const md2 = await client.exportPageMarkdown(fxid);
// On failure, name the first diverging line of the two exports.
const firstDiff = (a, b) => {
const al = a.split("\n");
const bl = b.split("\n");
for (let i = 0; i < Math.max(al.length, bl.length); i++) {
if (al[i] !== bl[i]) {
return `first diff at line ${i + 1}: ${JSON.stringify(al[i] ?? "<EOF>")} -> ${JSON.stringify(bl[i] ?? "<EOF>")}`;
}
}
return "same lines, different bytes (line endings?)";
};
check(
"markdown fixpoint: export -> import -> export is byte-identical",
md1 === md2,
md1 === md2 ? "" : firstDiff(md1, md2),
);
} finally {
try { await client.deletePage(fxid); } catch {}
}
}
// 7. shares: create (idempotent), public access, list, unshare
const share = await client.sharePage(pageId);
check("sharePage: returns public URL", share.publicUrl?.startsWith(`${APP}/share/`), share.publicUrl);
@@ -108,6 +108,17 @@ async function spawnCollabStack(seedDoc) {
return { state, baseURL };
}
// y-prosemirror stores an OVERLAPPING mark (one whose type does not exclude
// itself — e.g. `comment`, and since #515 `code` with `excludes: ""`) under a
// HASHED Yjs attribute key `name--<8-char hash>` so several may coexist on a
// range. The real read path (yDocToProsemirrorJSON) strips that suffix back to
// the bare mark name via this exact regex; mirror it here so this minimal decoder
// reports the same mark names Docmost actually returns (without it an overlapping
// `code` would leak as `code--<hash>`).
const hashedMarkNameRegex = /(.*)(--[a-zA-Z0-9+/=]{8})$/;
const yattr2markname = (attrName) =>
hashedMarkNameRegex.exec(attrName)?.[1] ?? attrName;
// Minimal XmlFragment -> ProseMirror JSON decode, mirroring the shape Docmost
// stores. Reads element name as node type, attributes as attrs, and recurses into
// children; text nodes carry their string.
@@ -121,8 +132,8 @@ function fragmentToJson(frag) {
if (d.attributes && Object.keys(d.attributes).length) {
node.marks = Object.entries(d.attributes).map(([type, attrs]) =>
attrs && typeof attrs === "object" && Object.keys(attrs).length
? { type, attrs }
: { type },
? { type: yattr2markname(type), attrs }
: { type: yattr2markname(type) },
);
}
return node;
@@ -9,7 +9,6 @@ import {
__sessionCountForTests,
} from "../../build/lib/collab-session.js";
import { withPageLock } from "../../build/lib/page-lock.js";
import { DocmostClient } from "../../build/client.js";
// A stand-in for HocuspocusProvider: it shares the ydoc (so the real yjs
// read/transform/write in CollabSession.mutate runs unchanged), auto-completes
@@ -90,7 +89,6 @@ const ENV_KEYS = [
"MCP_COLLAB_SESSION_IDLE_MS",
"MCP_COLLAB_SESSION_MAX_AGE_MS",
"MCP_COLLAB_SESSION_MAX_ENTRIES",
"MCP_COLLAB_TOKEN_TTL_MS",
];
let savedEnv;
@@ -347,86 +345,6 @@ test("replaceImage-shaped flow: acquire under an EXTERNAL page lock does not dea
);
});
// --- #439: the collab-token cache is what makes the session cache ACTUALLY hit ---
//
// WHY these two tests exist (the #435 incident): the session registry keys on
// (wsUrl, pageId, token) for identity isolation, but BOTH production token
// sources mint a FRESH JWT on every call (the in-app provider re-signs a JWT
// whose iat/exp changes every second; the external MCP POSTs /auth/collab-token
// per call). The fresh token per call made the session-registry key unstable,
// so the prod hit-rate was 0% — connect storms, 25s timeouts, zombie sessions —
// while every other test in this file stayed green because they pass a FIXED
// "tok" string. The #439 fix is the per-client collab-token cache
// (DocmostClient.getCollabTokenWithReauth + MCP_COLLAB_TOKEN_TTL_MS); these
// tests drive the token through it with a source that returns a DIFFERENT
// fresh JWT per mint, exactly like prod, so a regression in EITHER the token
// cache or the registry keying turns them red.
//
// getCollabTokenWithReauth is TS-private, but the compiled JS exposes it; the
// tests call it directly because that is exactly the per-op composition of the
// production call sites (updatePage etc.: mint the token, then acquire).
test("#439 token cache ON: fresh-JWT-per-mint source, two ops => ONE connect (session cache hits)", async () => {
process.env.MCP_COLLAB_TOKEN_TTL_MS = "300000"; // cache ON (explicit, not default-dependent)
let mints = 0;
const client = new DocmostClient({
apiUrl: "http://h/api",
getToken: async () => "user-jwt",
// Like both prod sources: a DIFFERENT fresh JWT on every mint.
getCollabToken: async () => `fresh-jwt-${++mints}`,
});
// Op 1: mint the collab token through the client, then acquire + mutate.
const tok1 = await client.getCollabTokenWithReauth();
const s1 = await acquireCollabSession("page-1", tok1, "http://h/api");
await s1.mutate(() => docWith("one"));
// Op 2: the same identity mints again — the cache must serve the SAME token.
const tok2 = await client.getCollabTokenWithReauth();
const s2 = await acquireCollabSession("page-1", tok2, "http://h/api");
await s2.mutate(() => docWith("two"));
assert.equal(mints, 1, "the second op is served from the token cache");
assert.equal(tok2, tok1, "stable token => stable session-registry key");
assert.equal(s2, s1, "the live session is reused");
assert.equal(
FakeProvider.connectCount,
1,
"two mutations over one identity must cost exactly ONE real connect",
);
assert.equal(__sessionCountForTests(), 1);
});
test("#439 negative control: token cache OFF (TTL=0) reproduces the #435 churn — two ops => TWO connects", async () => {
process.env.MCP_COLLAB_TOKEN_TTL_MS = "0"; // explicit 0 disables the cache (fetch-per-call legacy)
let mints = 0;
const client = new DocmostClient({
apiUrl: "http://h/api",
getToken: async () => "user-jwt",
getCollabToken: async () => `fresh-jwt-${++mints}`,
});
const tok1 = await client.getCollabTokenWithReauth();
const s1 = await acquireCollabSession("page-1", tok1, "http://h/api");
await s1.mutate(() => docWith("one"));
const tok2 = await client.getCollabTokenWithReauth();
const s2 = await acquireCollabSession("page-1", tok2, "http://h/api");
await s2.mutate(() => docWith("two"));
assert.equal(mints, 2, "without the cache every op mints its own token");
assert.notEqual(tok2, tok1, "unstable token => unstable session-registry key");
assert.notEqual(s2, s1, "no session reuse");
assert.equal(
FakeProvider.connectCount,
2,
"a full reconnect per op — the #435 storm in miniature",
);
// The first session lingers under its now-unreachable key until its idle
// TTL — the zombie-session symptom of the incident.
assert.equal(__sessionCountForTests(), 2);
});
test("destroyAllSessions tears down every cached session", async () => {
await acquireCollabSession("page-1", "tok", "http://h/api");
await acquireCollabSession("page-2", "tok", "http://h/api");
@@ -0,0 +1,107 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import {
collectResolvedCommentSpans,
regraftResolvedComments,
applyCommentMarkInDoc,
} from "../../build/lib/comment-anchor.js";
/**
* #493 commit 6 resolved-comment anchors must survive a full markdown rewrite
* (updatePageMarkdown). An agent read HIDES resolved anchors (#337), so its
* markdown drops them; a naive full write would erase the resolved comment marks.
* `regraftResolvedComments(oldDoc, newDoc)` re-anchors them onto the matching
* text. These exercise the real anchoring (no mock).
*/
const doc = (...content) => ({ type: "doc", content });
const para = (...content) => ({ type: "paragraph", content });
const text = (t, marks) => (marks ? { type: "text", text: t, marks } : { type: "text", text: t });
const resolvedComment = (commentId) => ({ type: "comment", attrs: { commentId, resolved: true } });
const activeComment = (commentId) => ({ type: "comment", attrs: { commentId, resolved: false } });
/** The comment mark on a text node, or null. */
function commentMarkOf(node) {
const marks = Array.isArray(node?.marks) ? node.marks : [];
return marks.find((m) => m && m.type === "comment") || null;
}
/** Flatten every text node in a doc (deep). */
function textNodes(node, out = []) {
if (!node || typeof node !== "object") return out;
if (node.type === "text") out.push(node);
if (Array.isArray(node.content)) for (const c of node.content) textNodes(c, out);
return out;
}
test("collectResolvedCommentSpans: only resolved marks, concatenated across a run", () => {
const old = doc(
para(
text("keep "),
text("resolved bit", [resolvedComment("r1")]),
text(" and "),
text("active bit", [activeComment("a1")]),
),
);
const spans = collectResolvedCommentSpans(old);
assert.equal(spans.length, 1);
assert.equal(spans[0].commentId, "r1");
assert.equal(spans[0].text, "resolved bit");
assert.equal(spans[0].mark.attrs.resolved, true);
});
test("regraft restores a resolved mark the agent's markdown dropped", () => {
// OLD doc has a resolved comment on "important note".
const old = doc(para(text("An "), text("important note", [resolvedComment("r1")]), text(" here.")));
// NEW doc (re-imported from the agent's markdown) has the SAME text but NO
// comment mark — the resolved anchor was hidden on read.
const fresh = doc(para(text("An important note here.")));
const out = regraftResolvedComments(old, fresh);
// Inputs are not mutated.
assert.equal(commentMarkOf(textNodes(fresh)[0]), null);
// The resolved mark is back on exactly "important note".
const marked = textNodes(out).filter((n) => commentMarkOf(n));
assert.equal(marked.length, 1);
assert.equal(marked[0].text, "important note");
assert.equal(commentMarkOf(marked[0]).attrs.commentId, "r1");
assert.equal(commentMarkOf(marked[0]).attrs.resolved, true);
});
test("a resolved span whose text the agent changed is dropped (no re-anchor)", () => {
const old = doc(para(text("stale text", [resolvedComment("r1")])));
const fresh = doc(para(text("completely rewritten body")));
const out = regraftResolvedComments(old, fresh);
assert.equal(textNodes(out).filter((n) => commentMarkOf(n)).length, 0);
});
test("regraft is a no-op when the old doc has no resolved comments", () => {
const old = doc(para(text("plain "), text("active", [activeComment("a1")])));
const fresh = doc(para(text("plain active")));
const out = regraftResolvedComments(old, fresh);
assert.equal(textNodes(out).filter((n) => commentMarkOf(n)).length, 0);
});
test("multiple distinct resolved comments are all restored", () => {
const old = doc(
para(text("first", [resolvedComment("r1")]), text(" middle "), text("second", [resolvedComment("r2")])),
);
const fresh = doc(para(text("first middle second")));
const out = regraftResolvedComments(old, fresh);
const byId = Object.fromEntries(
textNodes(out)
.filter((n) => commentMarkOf(n))
.map((n) => [commentMarkOf(n).attrs.commentId, n.text]),
);
assert.equal(byId["r1"], "first");
assert.equal(byId["r2"], "second");
});
test("applyCommentMarkInDoc preserves an arbitrary mark's attrs (resolved:true)", () => {
const d = doc(para(text("anchor me somewhere")));
const ok = applyCommentMarkInDoc(d, "anchor me", { type: "comment", attrs: { commentId: "x9", resolved: true } });
assert.equal(ok, true);
const marked = textNodes(d).filter((n) => commentMarkOf(n));
assert.equal(marked[0].text, "anchor me");
assert.equal(commentMarkOf(marked[0]).attrs.resolved, true);
});
@@ -16,6 +16,7 @@
* `@docmost/editor-ext` before updating the snapshot.
*/
import StarterKit from "@tiptap/starter-kit";
import { Code } from "@tiptap/extension-code";
import Image from "@tiptap/extension-image";
import TaskList from "@tiptap/extension-task-list";
import TaskItem from "@tiptap/extension-task-item";
@@ -1481,7 +1482,20 @@ export const docmostExtensions = [
codeBlock: {},
heading: {},
link: { openOnClick: false },
// #515: disable StarterKit's bundled inline `code` mark so it can be replaced
// by the local override below. StarterKit's `code` inherits tiptap's
// `excludes: "_"`, which strips every co-occurring mark on HTML->PM import
// (`generateJSON`) — so `` **`--flag`** `` lost its bold. This mirror is a
// DELIBERATE standalone copy (it must not pull @docmost/editor-ext into the
// node import runtime — that would drag in React/node-views; see #293), so
// the `excludes: ""` override is declared LOCALLY here and kept in lockstep
// with the canonical `Code` in @docmost/editor-ext by a parity test.
code: false,
}),
// #515: inline code that COMBINES with other marks (CommonMark-consistent).
// `excludes: ""` means the mark excludes nothing, so bold/italic/strike/… may
// co-occur with `code` and survive import.
Code.extend({ excludes: "" }),
// Preserve image width/height as the AUTHORED string. Without an explicit
// parseHTML the stock Image node attribute falls back to tiptap core's
// `fromString`, which coerces a numeric width like "320" into the number 320
@@ -1,7 +1,14 @@
/**
* Foreign-markdown normalizer an input-liberal / output-canonical adapter that
* runs at the IMPORT boundary, BEFORE the canonical parser
* (`markdownToProseMirror` from `@docmost/prosemirror-markdown`).
* (`markdownToProseMirror`, this package).
*
* OWNED BY THIS PACKAGE (#493): the normalizer used to live only in
* apps/server's import path, so the MCP page-write path (`updatePageMarkdown` ->
* `markdownToProseMirrorCanonical`) handled the SAME foreign input differently
* (no front-matter strip, no `[^id]` reference-footnote rewrite) than the server
* importer. Moving it here and calling it from `markdownToProseMirrorCanonical`
* makes every canonical import boundary treat foreign markdown identically.
*
* The canonical parser is deliberately STRICT: it only understands Docmost's
* canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
@@ -247,11 +254,18 @@ function convertReferenceFootnotes(markdown: string): string {
const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
/**
* Normalize a foreign markdown string into Docmost's canonical markdown surface
* so the strict canonical parser accepts it losslessly: normalize line endings,
* strip a leading YAML front-matter block, then rewrite GFM reference footnotes
* into inline footnotes. Add further fixture-driven foreign-surface cases here as
* they are found.
* Normalize a foreign markdown string from a FILE IMPORT into Docmost's canonical
* markdown surface so the strict canonical parser accepts it losslessly: normalize
* line endings, strip a leading YAML front-matter block, then rewrite GFM reference
* footnotes into inline footnotes. Add further fixture-driven foreign-surface cases
* here as they are found.
*
* FRONT-MATTER STRIP IS IMPORT-ONLY (#493 review): use this ONLY at the server
* file-import boundary, where a `.md` file really can open with an Obsidian/Hugo
* YAML header. Do NOT use it on the canonical AGENT-WRITE path see
* {@link normalizeAgentMarkdown} for why a full-body agent rewrite must NOT strip
* a leading `---…---` (it is normally a horizontalRule the serializer emitted, and
* stripping it would silently drop the page's leading content).
*/
export function normalizeForeignMarkdown(markdown: string): string {
if (!markdown) return markdown;
@@ -264,3 +278,26 @@ export function normalizeForeignMarkdown(markdown: string): string {
const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
return convertReferenceFootnotes(withoutFrontMatter);
}
/**
* Canonical AGENT-WRITE normalization: normalize line endings and rewrite GFM
* `[^id]` reference footnotes to inline `^[body]` but DELIBERATELY NOT strip a
* leading YAML front-matter block.
*
* WHY the split (#493 review): the reference-footnote rewrite is the drift the
* MCP page-write path (`updatePageMarkdown` -> `markdownToProseMirrorCanonical`)
* needed unified with the server import (an agent may paste GFM footnotes). The
* front-matter strip, however, is a FILE-import concern: on a full-body agent
* rewrite a leading `---…---` is (almost) always a `horizontalRule` the
* serializer emitted plus a later rule/heading NOT a foreign YAML header so
* `YAML_FRONT_MATTER_RE` would match it and SILENTLY DELETE the page's leading
* content (a page that starts with a horizontal rule and contains a second `---`
* lost everything up to it). Agent writes must never lose already-stored content,
* so this variant skips the strip. It IS a no-op on canonical serialized content
* (which never emits `[^id]:` reference-definition lines).
*/
export function normalizeAgentMarkdown(markdown: string): string {
if (!markdown) return markdown;
const src = markdown.replace(/\r\n/g, '\n');
return convertReferenceFootnotes(src);
}
+28 -1
View File
@@ -15,7 +15,10 @@ export {
} from "./markdown-document.js";
export type { DocmostMdMeta } from "./markdown-document.js";
export { convertProseMirrorToMarkdown } from "./markdown-converter.js";
export {
convertProseMirrorToMarkdown,
ConverterLossError,
} from "./markdown-converter.js";
export type { ConvertProseMirrorToMarkdownOptions } from "./markdown-converter.js";
export {
@@ -23,6 +26,19 @@ export {
markdownToProseMirrorSync,
} from "./markdown-to-prosemirror.js";
// Foreign-markdown normalizer (#493): the input-liberal pre-pass that rewrites
// GFM `[^id]` reference footnotes to canonical inline `^[body]`. Two variants:
// `normalizeForeignMarkdown` (server FILE-import boundary) ALSO strips a leading
// YAML front-matter block; `normalizeAgentMarkdown` (canonical AGENT-WRITE path,
// mcp `markdownToProseMirrorCanonical`) does NOT — a full-body agent rewrite must
// not lose a leading `---…---` horizontalRule to the front-matter strip (#493
// review). The reference-footnote rewrite is shared so agent + import stay unified
// where it matters, without the content-losing strip on the write path.
export {
normalizeForeignMarkdown,
normalizeAgentMarkdown,
} from "./foreign-markdown.js";
// The Docmost tiptap schema mirror. Exposed so consumers (and the sync
// engine's schema-validity regression tests) can build the exact ProseMirror
// schema the converter targets.
@@ -76,6 +92,17 @@ export type { OutlineEntry } from "./node-ops.js";
// string (#414: single copy shared by mcp and the CommonJS server app).
export { parseNodeArg } from "./parse-node-arg.js";
// Locator markdown-stripping (#493 dedup): the single canonical copy of the
// markdown-tolerant anchor-normalization primitives, imported by mcp's
// text-normalize.ts instead of a forked duplicate. `stripInlineMarkdown` is the
// lenient locator normalizer (trims stray decoration); `stripWrappersAndLinks`
// is the strict balanced-wrapper/link primitive mcp builds `stripBalancedWrappers`
// on top of.
export {
stripInlineMarkdown,
stripWrappersAndLinks,
} from "./text-normalize.js";
// Inline-footnote authoring convention (#414: single copy, formerly the mcp
// `footnote-authoring.ts` fork), shared with the importer's `assembleFootnotes`.
export {
@@ -33,6 +33,26 @@ import {
*/
const MAX_NODE_DEPTH = 400;
/**
* Thrown by {@link convertProseMirrorToMarkdown} in `strict` mode when it hits a
* node or mark type it has no lossless markdown form for (the serializer would
* otherwise silently degrade it drop an unknown mark, flatten an unknown node
* to its children). Carries the offending kind/name so a caller (git-sync) can
* surface exactly what would have been lost.
*/
export class ConverterLossError extends Error {
readonly kind: "node" | "mark";
readonly typeName: string;
constructor(kind: "node" | "mark", typeName: string) {
super(
`convertProseMirrorToMarkdown: unknown ${kind} type "${typeName}" has no lossless markdown representation (strict mode)`,
);
this.name = "ConverterLossError";
this.kind = kind;
this.typeName = typeName;
}
}
/**
* Options for {@link convertProseMirrorToMarkdown}.
*/
@@ -46,6 +66,23 @@ export interface ConvertProseMirrorToMarkdownOptions {
* path where resolved anchors MUST be preserved for round-tripping.
*/
dropResolvedCommentAnchors?: boolean;
/**
* Optional sink for LOSS warnings. When the serializer reaches a node or mark
* type it has no dedicated case for, it degrades gracefully (flattens an
* unknown node to its children, drops an unknown mark) historically a SILENT
* data loss. When this array is provided, one human-readable message per such
* event is pushed here so the caller can observe (and log) what was degraded.
* Not provided by default -> behavior is byte-identical to before for existing
* callers.
*/
warnings?: string[];
/**
* When true, THROW a {@link ConverterLossError} on the FIRST unknown node/mark
* instead of degrading silently a warning becomes a hard error. Used by the
* lossless git-sync export path and the converter tests, where an unmapped
* type is a bug to surface, not data to quietly drop.
*/
strict?: boolean;
}
/**
@@ -63,6 +100,70 @@ export interface ConvertProseMirrorToMarkdownOptions {
* separator is emitted for any other join, so non-list output is unchanged.
*/
const LIST_MARKER_SEPARATOR = "<!-- -->";
/**
* Backslash-escape a leading markdown BLOCK trigger so a serialized paragraph
* line re-parses as a PARAGRAPH, not another block. Without this, a paragraph
* whose text begins at column 0 with an ATX heading `#`, a blockquote/callout
* `>`, a bullet marker `-`/`*`/`+`, an ordered marker `N.`/`N)`, a code fence
* (```` ``` ````/`~~~`), a table `|`, or a thematic break (`---`/`***`/`___`,
* solid or spaced) silently becomes a heading/list/quote/code block/table/rule
* on the next markdown -> ProseMirror import a known data-loss class (the
* thematic-break case drops the text entirely, since a horizontalRule carries
* none). CommonMark's escape tokenizer decodes the inserted `\` back to the
* literal character on import AND stops the block interpretation, so the line
* round-trips byte-exact as paragraph text. Only the FIRST offending character
* is escaped (the minimum needed to break block recognition); a line that does
* NOT open a block emphasis `**x**`, an inline code span, ordinary prose is
* returned verbatim, so there is no backslash churn for the common case.
*
* Applied ONLY to paragraph text, once per `\n`-separated LINE (the paragraph
* case splits on `\n` each hardBreak emits ` \n` so a trigger on a
* continuation line is escaped too): headings/lists/blockquotes legitimately
* open with these markers and render them from their own cases. This is the
* single, canonical fix for the class the client bridge worked around with a
* ZWSP (`gitmost-recording.ts`) and the generative suite self-censored around
* (`text-arbitraries.ts`) both now removed.
*/
function escapeLeadingBlockTrigger(line: string): string {
// ATX heading: 1..6 `#` then whitespace/EOL.
if (/^#{1,6}(?:\s|$)/.test(line)) return "\\" + line;
// Blockquote / Docmost callout opener (`>` or `> [!info]`).
if (line.startsWith(">")) return "\\" + line;
// Bullet list marker then whitespace/EOL. Emphasis (`*x*`, `**x**`) has no
// space after the leading marker and is intentionally left verbatim.
if (/^[-*+](?:\s|$)/.test(line)) return "\\" + line;
// Ordered list marker `N.` / `N)`: escape the DELIMITER so the digits stay
// literal (`1. x` -> `1\. x`, which imports back as the text `1. x`).
const ordered = line.match(/^(\d+)[.)](?:\s|$)/);
if (ordered) {
const digits = ordered[1].length;
return line.slice(0, digits) + "\\" + line.slice(digits);
}
// Fenced code block: 3+ backticks or tildes. A single/double backtick is an
// inline code span and is left verbatim.
if (/^(?:`{3,}|~{3,})/.test(line)) return "\\" + line;
// Thematic break: a WHOLE line of 3+ identical `-`/`*`/`_`, optionally spaced.
if (/^([-*_])(?:\s*\1){2,}\s*$/.test(line)) return "\\" + line;
// Setext underline: a continuation line (after a hardBreak) that is ONLY `-`
// or ONLY `=` (any count, trailing spaces allowed). Under a paragraph line
// such a line re-parses as a SETEXT HEADING and SILENTLY DROPS its own text
// (`a\n--` -> heading "a", the `--` is LOST; `a\n=` -> heading "a", `=` LOST).
// The bullet arm above catches a lone `-` (via its `$`) and the thematic arm
// catches 3+ dashes, but exactly TWO dashes (`--`) fall through both; and no
// arm covers a lone `=` at all (a `==` pair is neutralized earlier by the
// inline `==`->`\=\=` escape, so only a single `=` line reaches here). Escaping
// the leading char (`\--`, `\=`) breaks the setext interpretation so the line
// round-trips as paragraph text. The WHOLE line must be the marker (anchored
// `^-+`/`^=+` to EOL), so a mid-content `-`/`=` is never spuriously escaped;
// and a `---`/`----` already handled by the thematic arm never reaches here,
// so there is no double-escape.
if (/^-+[ \t]*$/.test(line) || /^=+[ \t]*$/.test(line)) return "\\" + line;
// GFM table row opener.
if (line.startsWith("|")) return "\\" + line;
return line;
}
function listMarkerFamily(type: string | undefined): "ul" | "ol" | null {
if (type === "bulletList" || type === "taskList") return "ul";
if (type === "orderedList") return "ol";
@@ -109,6 +210,26 @@ export function convertProseMirrorToMarkdown(
// callers (mcp getPage / in-app AI chat) pass it true.
const dropResolvedCommentAnchors = options.dropResolvedCommentAnchors === true;
// Loss reporting for node/mark types with no dedicated serializer case. In
// `strict` mode the FIRST such type throws (git-sync, tests); otherwise the
// serializer degrades gracefully (as it always has) but records one warning
// per unmapped type into the optional sink so the loss is observable, not
// silent. Deduped per type so a document with many unknown nodes of one type
// produces one message.
const strict = options.strict === true;
const warningsSink = options.warnings;
const seenLossTypes = new Set<string>();
const warnLoss = (kind: "node" | "mark", typeName: string): void => {
if (strict) throw new ConverterLossError(kind, typeName);
if (!warningsSink) return;
const key = `${kind}:${typeName}`;
if (seenLossTypes.has(key)) return;
seenLossTypes.add(key);
warningsSink.push(
`Unknown ${kind} type "${typeName}" has no lossless markdown form; it was degraded on export.`,
);
};
// Escape a value interpolated into an HTML double-quoted attribute value
// (textAlign, colors, image src, math `text`, all data-* attrs, etc.). In the
// ATTRIBUTE context only the quote that delimits the value and the ampersand
@@ -362,6 +483,99 @@ export function convertProseMirrorToMarkdown(
return `<table><tbody>${htmlRows}</tbody></table>`;
};
// Layer the intentional inline escapes onto a NON-code text run BEFORE its
// marks are applied. Extracted so both `case "text"` and the #515 code-emphasis
// run factoring (renderInlineChildren) escape the inner text identically. NEVER
// called on code content (a code span is literal — see the gating in the text
// case and the run helper). Order is load-bearing: the footnote raw-backslash
// doubling MUST precede the `==`/`$`/`^[` escapes (see inFootnoteBody).
const escapeInlineText = (text: string): string => {
let t = text;
if (inFootnoteBody) t = t.replace(/\\/g, "\\\\");
t = t.replace(/==/g, "\\=\\=");
t = escapeProseMath(t);
t = t.replace(/\^\[/g, "^\\[");
return t;
};
// Wrap `text` with the markdown/HTML form of a SINGLE inline mark. Extracted
// from `case "text"` so the same per-mark emission is reused when the #515
// run factoring layers a shared outer mark over a code-emphasis run. `code` is
// handled by the callers (wrapped innermost, before this runs), so this branch
// is defensive only. For any non-code mark the output is byte-identical to the
// pre-#515 inline switch.
const applyInlineMark = (text: string, mark: any): string => {
switch (mark.type) {
case "bold":
return `**${text}**`;
case "italic":
return `*${text}*`;
case "code":
// Callers wrap the code span innermost themselves; reached only if a
// mark list is applied through here directly. Emit the backtick span.
return `\`${text}\``;
case "link": {
const href = mark.attrs?.href || "";
const title = mark.attrs?.title;
if (title) {
// Emit the optional markdown link title; escape an embedded double-
// quote so it cannot terminate the title string early.
const safeTitle = String(title).replace(/"/g, '\\"');
return `[${text}](${href} "${safeTitle}")`;
}
return `[${text}](${href})`;
}
case "strike":
return `~~${text}~~`;
case "underline":
return `<u>${text}</u>`;
case "subscript":
return `<sub>${text}</sub>`;
case "superscript":
return `<sup>${text}</sup>`;
case "highlight": {
// #293 canon #7: a highlight WITHOUT a color serializes as the
// Obsidian/GFM `==text==` syntax; a colored highlight keeps the `<mark
// style>` HTML form. The inner text already had any literal `==`
// backslash-escaped upstream.
const color = mark.attrs?.color;
return color
? `<mark style="background-color: ${escapeAttr(color)}">${text}</mark>`
: `==${text}==`;
}
case "textStyle":
if (mark.attrs?.color) {
return `<span style="color: ${escapeAttr(mark.attrs.color)}">${text}</span>`;
}
return text;
case "spoiler":
// Markdown has no native spoiler syntax, so emit the same raw inline HTML
// the editor-ext/MCP stack uses (span[data-spoiler] round-trips).
return `<span data-spoiler="true">${text}</span>`;
case "comment": {
// Inline comment anchor (span[data-comment-id]); resolved anchors are
// optionally dropped for agent reads, keeping only the bare text.
const cid = mark.attrs?.commentId;
if (cid) {
if (mark.attrs?.resolved && dropResolvedCommentAnchors) {
return text;
}
const resolvedAttr = mark.attrs?.resolved
? ` data-resolved="true"`
: "";
return `<span data-comment-id="${escapeAttr(cid)}"${resolvedAttr}>${text}</span>`;
}
return text;
}
default:
// Unknown mark: no dedicated case, so it has no markdown form and is
// dropped from the run. Report the loss (throws in strict mode) then
// leave the text unwrapped — the historical behavior.
warnLoss("mark", String(mark.type));
return text;
}
};
const processNode = (node: any): string => {
if (nodeDepth >= MAX_NODE_DEPTH) {
// Bail out of deeper recursion without throwing. A text node still has
@@ -412,7 +626,17 @@ export function convertProseMirrorToMarkdown(
}
case "paragraph": {
const text = renderInlineChildren(nodeContent);
// Escape a leading block trigger on EVERY line of the paragraph, not
// just the first: a hardBreak serializes as ` \n`, so a `#`/`-`/`>`/
// `1.`/`|`/fence/`---` at the start of a CONTINUATION line would also
// re-parse into another block on the next import (a heading/list/table/
// setext-`---`), and for the text-less thematic/setext case would LOSE
// that line's text entirely. Escaping each `\n`-separated line closes
// the class for multi-line paragraphs too.
const text = renderInlineChildren(nodeContent)
.split("\n")
.map(escapeLeadingBlockTrigger)
.join("\n");
const align = node.attrs?.textAlign;
// Non-default alignment round-trips as an ATTACHED HTML comment at the
// END of the block line (#293 canon #9):
@@ -451,154 +675,38 @@ export function convertProseMirrorToMarkdown(
return headingLine;
}
case "text":
case "text": {
let textContent = node.text || "";
// #293 canon #7: `==` is now a LIVE inline highlight syntax on import (a
// marked inline extension turns `==text==` into a color-less highlight
// mark). A LITERAL `==` in a text run would therefore be misparsed as a
// highlight on the next import, so backslash-escape each `=` of a `==`
// pair; marked's escape tokenizer decodes `\=` back to a literal `=`, so
// a literal `==` round-trips as text (never materializes a phantom mark).
// This runs for BOTH unmarked text and marked non-code runs, but NOT for
// an inline code span (a run carrying the `code` mark returns a backtick
// span below with `==` verbatim, matching `` `a == b` `` staying code).
// A highlight run's own `==` delimiters are appended AFTER this in the
// marks loop, so they are never escaped; only the run's inner text is.
if (!(node.marks || []).some((m: any) => m.type === "code")) {
// #293 canon #2 (F2): inside a footnote body, DOUBLE every RAW user
// backslash FIRST, so it survives `^[…]` (the import tokenizer treats
// `\<char>` as an escape when balancing brackets, and `parseInline`
// decodes escapes). Doing it before the intentional escapes below keeps
// the serializer's own single escapes (`\=` `\$` `^\[`, and the `\[`/
// `\]` balanceBrackets adds) single; only genuine user backslashes are
// doubled. Skipped for code runs (a code span's content is NOT decoded
// by parseInline, so its backslashes must stay verbatim).
if (inFootnoteBody) {
textContent = textContent.replace(/\\/g, "\\\\");
}
textContent = textContent.replace(/==/g, "\\=\\=");
// #293 canon #6: escape a would-be inline-math `$…$` span so it stays
// literal text on re-import (currency `$5` is left clean — see
// escapeProseMath). Runs on the SAME non-code runs as the `==` escape
// above; an inline `code` run returns verbatim below, matching the
// codeBlock path (a `$…$` inside code must stay code, never math).
textContent = escapeProseMath(textContent);
// #293 canon #2: `^[` opens a LIVE inline-footnote span on import
// (`^[text]` -> a footnote reference). A LITERAL `^[` in prose text
// would therefore materialize a phantom footnote on the next import, so
// backslash-escape the bracket (`^[` -> `^\[`); marked's escape
// tokenizer decodes `\[` back to `[`, so a literal `^[…]` round-trips
// as text and never opens a footnote. Only the OPENING `^[` needs
// breaking (the tokenizer requires it), so this is a minimal, idempotent
// escape. A real footnoteReference node emits `^[body]` from its own
// case, never through here.
textContent = textContent.replace(/\^\[/g, "^\\[");
// #515: `code` is no longer exclusive (`excludes: ""`), so a run may
// carry `code` TOGETHER with other marks. The inner escapes below apply
// ONLY to a NON-code run (a code span's content is literal — `==`, `$…$`,
// `^[` must stay verbatim, matching `` `a == b` `` staying code). See
// #293 canon #2/#6/#7 for why each escape exists (extracted into
// escapeInlineText). A code run's `==`/`$`/`^[` are protected by the
// backticks, so they are never misparsed on re-import.
const hasCode = (node.marks || []).some((m: any) => m.type === "code");
if (!hasCode) {
textContent = escapeInlineText(textContent);
}
// Apply marks (bold, italic, code, etc.)
if (node.marks) {
// The schema's `code` mark declares `excludes: "_"` — it excludes every
// other inline mark — so the editor can NEVER produce a text run that
// carries `code` together with another mark, and on import any
// co-occurring mark is always dropped (the run comes back as code-only).
// The lossless, byte-stable behavior is therefore: when a run has the
// `code` mark, emit ONLY the backtick code span and ignore every other
// mark, so md1 is already code-only and md2 === md1. Runs WITHOUT a code
// mark are rendered exactly as before.
const markTypes = node.marks.map((m: any) => m.type);
const hasCode = markTypes.includes("code");
// #515: wrap the backtick code span FIRST (innermost mark), then layer
// the REMAINING marks in array order. For a run WITHOUT a code mark the
// loop applies every mark exactly as the pre-#515 switch did, so the
// output is byte-identical. For a code+emphasis run the code span sits
// inside the emphasis delimiters (`` **`code`** ``), matching CommonMark.
// The shared-mark grouping across ADJACENT nodes (`` **`a` + `b`** ``)
// lives in renderInlineChildren; this direct path handles a lone run
// and the table/`default` callers that invoke processNode per node.
if (hasCode) {
textContent = `\`${textContent}\``;
return textContent;
}
for (const mark of node.marks) {
switch (mark.type) {
case "bold":
textContent = `**${textContent}**`;
break;
case "italic":
textContent = `*${textContent}*`;
break;
case "code":
// A `code` run already returned above (hasCode early return), so
// this branch is only reached for a non-code run that somehow
// still lists `code`; emit the plain backtick span.
textContent = `\`${textContent}\``;
break;
case "link": {
const href = mark.attrs?.href || "";
const title = mark.attrs?.title;
if (title) {
// Emit the optional markdown link title; escape an embedded
// double-quote so it cannot terminate the title string early.
const safeTitle = String(title).replace(/"/g, '\\"');
textContent = `[${textContent}](${href} "${safeTitle}")`;
} else {
textContent = `[${textContent}](${href})`;
}
break;
}
case "strike":
textContent = `~~${textContent}~~`;
break;
case "underline":
textContent = `<u>${textContent}</u>`;
break;
case "subscript":
textContent = `<sub>${textContent}</sub>`;
break;
case "superscript":
textContent = `<sup>${textContent}</sup>`;
break;
case "highlight": {
// #293 canon #7: a highlight WITHOUT a color serializes as the
// Obsidian/GFM `==text==` syntax (the importer's marked inline
// `==` extension parses it back to a color-less highlight mark).
// A highlight WITH a color keeps the `<mark style="background-
// color: …">` HTML form (the condition is deterministic on the
// `color` attr), so a colored highlight is not flattened. The
// inner textContent already had any literal `==` backslash-
// escaped above, so a highlight over text containing `==` still
// round-trips.
const color = mark.attrs?.color;
textContent = color
? `<mark style="background-color: ${escapeAttr(color)}">${textContent}</mark>`
: `==${textContent}==`;
break;
}
case "textStyle":
if (mark.attrs?.color) {
textContent = `<span style="color: ${escapeAttr(mark.attrs.color)}">${textContent}</span>`;
}
break;
case "spoiler":
// Markdown has no native spoiler syntax, so emit the same raw
// inline HTML the editor-ext/MCP stack uses. The schema's Spoiler
// mark parses span[data-spoiler] back on import, so the mark
// survives the PM -> MD -> PM round-trip.
textContent = `<span data-spoiler="true">${textContent}</span>`;
break;
case "comment": {
// Emit the inline comment anchor so highlights round-trip. The
// schema's Comment mark parses span[data-comment-id] (attrs
// commentId/resolved).
const cid = mark.attrs?.commentId;
if (cid) {
// Hide resolved anchors from agent reads: drop the wrapper and
// keep only the bare text. Active anchors keep their wrapper.
if (mark.attrs?.resolved && dropResolvedCommentAnchors) {
break;
}
const resolvedAttr = mark.attrs?.resolved
? ` data-resolved="true"`
: "";
textContent = `<span data-comment-id="${escapeAttr(cid)}"${resolvedAttr}>${textContent}</span>`;
}
break;
}
}
if (mark.type === "code") continue; // wrapped innermost above
textContent = applyInlineMark(textContent, mark);
}
}
return textContent;
}
case "codeBlock":
const language = node.attrs?.language || "";
@@ -1173,7 +1281,11 @@ export function convertProseMirrorToMarkdown(
}
default:
// Fallback: process children
// Unknown node type: no dedicated case, so the node's identity + attrs
// have no lossless markdown form. Report the loss (throws in strict
// mode) then degrade by flattening to its children — the historical
// graceful fallback.
warnLoss("node", String(type));
return nodeContent.map(processNode).join("");
}
};
@@ -1186,18 +1298,165 @@ export function convertProseMirrorToMarkdown(
// For that node ONLY we fall back to the lossless schema-HTML `<span>` form.
// Every other inline node is rendered exactly as processNode would, so output
// is unchanged whenever no math sits directly before a digit.
// #515: a "bare-delimiter" emphasis mark is one that serializes as a naked
// markdown delimiter run (`**` `*` `~~` `==`) — bold / italic / strike /
// UNCOLORED highlight. These delimiters COLLIDE with the backtick-flanking
// delimiters emitted around a code+emphasis run: rendering `[code,bold]` next
// to `[italic]` node-by-node would produce `` **`a`***b* `` (a `***` run that
// re-imports wrong). Every OTHER mark (underline/sub/sup/spoiler/comment/
// textStyle/colored-highlight/link) emits an HTML/bracket form whose boundaries
// do NOT collapse, so those neighbors never join a run.
const isBareEmphasisMark = (mark: any): boolean => {
switch (mark?.type) {
case "bold":
case "italic":
case "strike":
return true;
case "highlight":
return !mark.attrs?.color; // colored highlight emits <mark>, not `==`
default:
return false;
}
};
// A text node participates in a code-emphasis run iff it carries at least one
// bare-delimiter emphasis mark. A code-ONLY node (no emphasis) does NOT — so a
// plain `` `code` `` next to `**bold**` keeps its clean, byte-identical
// markdown (they share no colliding delimiter). Existing pages, where a code
// node could never carry emphasis, therefore serialize exactly as before.
const isEmphasisMember = (node: any): boolean =>
node?.type === "text" &&
(node.marks || []).some((m: any) => isBareEmphasisMark(m));
// The run's non-code marks (order preserved) — the candidate marks to factor.
const nonCodeMarks = (node: any): any[] =>
(node.marks || []).filter((m: any) => m.type !== "code");
// Deep structural equality of two marks (type + full attrs). Two `link` marks
// are equal only when EVERY attr matches (class/href/internal/rel/target/title
// — not just href), so a homogeneous run never merges links that differ.
const marksEqual = (a: any, b: any): boolean =>
a.type === b.type &&
JSON.stringify(a.attrs ?? null) === JSON.stringify(b.attrs ?? null);
// Two non-code mark lists are equal AS SETS (a run is homogeneous when every
// node shares the identical non-code mark set — order-independent).
const markSetsEqual = (a: any[], b: any[]): boolean =>
a.length === b.length &&
a.every((ma) => b.some((mb) => marksEqual(ma, mb))) &&
b.every((mb) => a.some((ma) => marksEqual(mb, ma)));
// Serialize one node's INNER form for a homogeneous run: the factored marks are
// applied by the caller, so here a code node emits only its literal backtick
// span and a non-code node emits only its (escaped) text.
const renderRunInner = (node: any): string => {
const text = node.text || "";
if ((node.marks || []).some((m: any) => m.type === "code")) {
return `\`${text}\``; // code content is literal
}
return escapeInlineText(text);
};
// A markdown emphasis delimiter (`**`/`*`/`~~`/`==`) wrapping a code span opens
// with the delimiter immediately followed by a backtick and closes immediately
// preceded by one. A backtick is CommonMark punctuation, so such a delimiter is
// only left/right-flanking — able to open/close emphasis — when the character
// on its OUTER side is start/end, whitespace or punctuation. If a run boundary
// abuts a word character, the delimiter would NOT flank (`a**` `code` `**`
// never opens) and the emphasis silently degrades on re-import. This checks the
// outer boundary char conservatively: ASCII whitespace or ASCII punctuation (or
// the string edge) is safe; anything else (a letter/number, unicode letter or
// emoji) is treated as unsafe so the run takes the lossless HTML fallback.
const SAFE_BOUNDARY = /[\s!-/:-@[-`{-~]/;
const isSafeBoundary = (c: string): boolean => c === "" || SAFE_BOUNDARY.test(c);
// Serialize a maximal run of adjacent emphasis-member text nodes that contains
// at least one `code` node (#515). HOMOGENEOUS (all share the identical
// non-code mark set) AND flank-safe on both boundaries: factor the common marks
// ONCE around the concatenated inner spans — `` **`aaa` + `bbb`** ``, code
// innermost. Otherwise — HETEROGENEOUS (non-code sets differ, e.g. `[code,bold]`
// next to `[italic]`) OR a boundary abuts a word char — emit the whole run as
// schema-HTML via the lossless inlineToHtml fallback, avoiding a colliding
// `***` delimiter run or a non-flanking `a**` that would drop the emphasis.
const renderCodeEmphasisRun = (
run: any[],
prevChar: string,
nextChar: string,
): string => {
const firstNonCode = nonCodeMarks(run[0]);
const homogeneous = run.every((n) =>
markSetsEqual(nonCodeMarks(n), firstNonCode),
);
if (!homogeneous || !isSafeBoundary(prevChar) || !isSafeBoundary(nextChar)) {
return inlineToHtml(run);
}
let out = run.map(renderRunInner).join("");
// Apply the common non-code marks in the FIRST node's array order (code is
// already innermost inside each span).
for (const mark of firstNonCode) out = applyInlineMark(out, mark);
return out;
};
const renderInlineChildren = (nodes: any[]): string => {
const parts = nodes.map(processNode);
for (let i = 0; i < nodes.length - 1; i++) {
if (
nodes[i]?.type === "mathInline" &&
parts[i].startsWith("$") &&
/^[0-9]/.test(parts[i + 1] || "")
) {
parts[i] = mathInlineHtml(nodes[i].attrs?.text || "");
// Pass 1: segment the nodes. Each segment is either an already-rendered
// non-run node / pure-emphasis node (byte-identical to the pre-#515 output),
// or a DEFERRED code-emphasis run (a maximal block of consecutive
// emphasis-member text nodes containing a code node) — its markdown-vs-HTML
// choice needs the neighbor boundary chars, resolved in pass 2.
type Seg = { firstNode: any; text?: string; run?: any[] };
const segs: Seg[] = [];
let i = 0;
while (i < nodes.length) {
const node = nodes[i];
if (isEmphasisMember(node)) {
let j = i;
while (j < nodes.length && isEmphasisMember(nodes[j])) j++;
const run = nodes.slice(i, j);
const hasCode = run.some((n: any) =>
(n.marks || []).some((m: any) => m.type === "code"),
);
if (hasCode) {
segs.push({ firstNode: run[0], run });
} else {
// Pure-emphasis run (no code): render each node as before.
for (const n of run) segs.push({ firstNode: n, text: processNode(n) });
}
i = j;
} else {
segs.push({ firstNode: node, text: processNode(node) });
i++;
}
}
return parts.join("");
// A deferred run always emits either a delimiter/backtick (markdown) or `<`
// (HTML) first — both punctuation — so a following run counts as a safe
// boundary for the current one without resolving it first.
const firstCharOf = (seg: Seg): string =>
seg.text !== undefined ? seg.text[0] || "" : "*";
// Pass 2: resolve deferred runs left-to-right, tracking the previous emitted
// char (for the opening boundary) and peeking the next segment (for closing).
let prevChar = "";
for (let k = 0; k < segs.length; k++) {
const seg = segs[k];
if (seg.text === undefined) {
const nextChar = k + 1 < segs.length ? firstCharOf(segs[k + 1]) : "";
seg.text = renderCodeEmphasisRun(seg.run!, prevChar, nextChar);
}
if (seg.text.length > 0) prevChar = seg.text[seg.text.length - 1];
}
// Preserve the mathInline-before-digit guard: a `$…$` immediately followed by
// a digit-leading segment would re-tokenize as a longer math span, so emit
// that math node as HTML instead. A code-emphasis run never starts with a
// digit (it opens with a delimiter or `<`), so segment granularity is safe.
for (let k = 0; k < segs.length - 1; k++) {
if (
segs[k].firstNode?.type === "mathInline" &&
(segs[k].text || "").startsWith("$") &&
/^[0-9]/.test(segs[k + 1].text || "")
) {
segs[k].text = mathInlineHtml(segs[k].firstNode.attrs?.text || "");
}
}
return segs.map((s) => s.text).join("");
};
// Render inline content (text runs + their marks) to HTML. Used by the raw
@@ -1232,7 +1491,22 @@ export function convertProseMirrorToMarkdown(
return processNode(n);
}
let t = escapeHtmlText(n.text || "");
// #515: wrap `<code>` INNERMOST first (before the array-order mark loop),
// then skip `code` in the loop. The imported mark order is NOT fixed — it
// DEPENDS on the emphasis extension: import (`generateJSON`) yields code
// LAST for bold/italic/strike (`[emphasis, code]`) but code FIRST for the
// `==`-highlight extension (`[code, highlight]`). So we cannot rely on a
// fixed array position; the invariant is instead "wrap `<code>` innermost
// regardless of the imported order". That keeps `<code>` nested inside the
// emphasis tag both directions (preserving the byte fixpoint — an order-
// sensitive loop would flip `<strong><code>`↔`<code><strong>` depending on
// which order it happened to see) and matches the markdown path (case
// "text" / run factoring).
if ((n.marks || []).some((m: any) => m.type === "code")) {
t = `<code>${t}</code>`;
}
for (const mark of n.marks || []) {
if (mark.type === "code") continue; // wrapped innermost above
switch (mark.type) {
case "bold":
t = `<strong>${t}</strong>`;
@@ -1297,6 +1571,12 @@ export function convertProseMirrorToMarkdown(
t = `<span data-comment-id="${escapeAttr(mark.attrs.commentId)}"${r}>${t}</span>`;
}
break;
default:
// Unknown mark on the raw-HTML path: dropped (no HTML form). Report
// the loss (throws in strict mode) — same policy as the markdown
// path's marks loop above.
warnLoss("mark", String(mark.type));
break;
}
}
return t;
@@ -7,13 +7,12 @@
* it is never applied to replacement text or inserted node content, so no
* formatting is ever lost.
*
* Scope note (#414): this package-local copy exists so `node-ops.ts` which
* lives here now (the single canonical copy) can resolve its markdown-tolerant
* anchor fallback without a circular dependency back on `@docmost/mcp`. It
* intentionally carries ONLY `stripInlineMarkdown` (the primitive `node-ops`
* needs); the mcp-side `text-normalize.ts` (which additionally serves
* `json-edit.ts` via `stripBalancedWrappers`) is the subject of a separate
* dedup task and is left untouched here.
* CANONICAL HOME (#414/#493): this is the single source of truth for locator
* markdown-stripping. `node-ops.ts` (which lives here) uses it directly, and the
* mcp-side `text-normalize.ts` now IMPORTS `stripInlineMarkdown` and the shared
* `stripWrappersAndLinks` primitive from here (via `@docmost/prosemirror-markdown`)
* instead of keeping a drifting copy mcp only adds its own thin
* `stripBalancedWrappers`/`closestBlockHint` on top.
*/
/** Maximum unwrap passes, so pathological/nested input cannot loop forever. */
@@ -44,7 +43,7 @@ const LINK_IMAGE_RE = /!?\[([^\]]*)\]\([^)]*\)/g;
* Does NOT trim decoration, does NOT guard against an empty result it returns
* exactly the transformed string.
*/
function stripWrappersAndLinks(s: string): string {
export function stripWrappersAndLinks(s: string): string {
// 1. Links/images -> their visible text.
let out = s.replace(LINK_IMAGE_RE, "$1");
@@ -0,0 +1,112 @@
import { describe, expect, it } from "vitest";
import {
convertProseMirrorToMarkdown,
ConverterLossError,
} from "../src/lib/markdown-converter.js";
/**
* #493 commit 3 a node/mark type the serializer has no dedicated case for used
* to be degraded SILENTLY (an unknown node flattened to its children, an unknown
* mark dropped from the run). The serializer now REPORTS the loss:
* - default (non-strict): unchanged graceful degradation, but one warning per
* unmapped type is pushed into an optional `warnings` sink so callers can
* observe it;
* - strict: the FIRST unmapped type throws a ConverterLossError (git-sync +
* tests), turning a silent loss into a hard, surfaced error.
*
* Exercised through the REAL converter (no mock): the observable properties are
* the emitted markdown, the warnings collected, and the thrown error.
*/
const doc = (...nodes: any[]) => ({ type: "doc", content: nodes });
describe("converter loss reporting — unknown node types", () => {
const unknownNode = doc({
type: "quantumWidget",
content: [{ type: "text", text: "inner text" }],
});
it("degrades to children AND records a warning (non-strict, sink provided)", () => {
const warnings: string[] = [];
const md = convertProseMirrorToMarkdown(unknownNode, { warnings });
// Graceful degrade: the child text still survives (historical behavior).
expect(md).toContain("inner text");
// The loss is now observable.
expect(warnings).toHaveLength(1);
expect(warnings[0]).toContain("quantumWidget");
expect(warnings[0]).toContain("node");
});
it("stays byte-identical for callers that pass no sink (zero behavior change)", () => {
const withSink: string[] = [];
const a = convertProseMirrorToMarkdown(unknownNode, { warnings: withSink });
const b = convertProseMirrorToMarkdown(unknownNode);
expect(b).toBe(a); // the sink does not alter the produced markdown
});
it("throws ConverterLossError in strict mode", () => {
try {
convertProseMirrorToMarkdown(unknownNode, { strict: true });
expect.unreachable("strict mode must throw on an unknown node");
} catch (e) {
expect(e).toBeInstanceOf(ConverterLossError);
expect((e as ConverterLossError).kind).toBe("node");
expect((e as ConverterLossError).typeName).toBe("quantumWidget");
}
});
it("dedupes the warning per type (many unknown nodes -> one message)", () => {
const warnings: string[] = [];
convertProseMirrorToMarkdown(
doc(
{ type: "quantumWidget", content: [{ type: "text", text: "a" }] },
{ type: "quantumWidget", content: [{ type: "text", text: "b" }] },
),
{ warnings },
);
expect(warnings).toHaveLength(1);
});
});
describe("converter loss reporting — unknown mark types", () => {
const unknownMark = doc({
type: "paragraph",
content: [{ type: "text", text: "glowing", marks: [{ type: "glow" }] }],
});
it("drops the mark but keeps the text AND records a warning (non-strict)", () => {
const warnings: string[] = [];
const md = convertProseMirrorToMarkdown(unknownMark, { warnings });
expect(md).toBe("glowing"); // text survives, mark silently had no form
expect(warnings).toHaveLength(1);
expect(warnings[0]).toContain("glow");
expect(warnings[0]).toContain("mark");
});
it("throws ConverterLossError in strict mode", () => {
expect(() =>
convertProseMirrorToMarkdown(unknownMark, { strict: true }),
).toThrow(ConverterLossError);
});
});
describe("converter loss reporting — known content is never flagged", () => {
it("a fully-mapped document produces no warnings and does not throw in strict mode", () => {
const d = doc(
{ type: "heading", attrs: { level: 2 }, content: [{ type: "text", text: "Title" }] },
{
type: "paragraph",
content: [
{ type: "text", text: "bold", marks: [{ type: "bold" }] },
{ type: "text", text: " and " },
{ type: "text", text: "link", marks: [{ type: "link", attrs: { href: "https://x.y" } }] },
],
},
{ type: "bulletList", content: [{ type: "listItem", content: [{ type: "paragraph", content: [{ type: "text", text: "item" }] }] }] },
);
const warnings: string[] = [];
const md = convertProseMirrorToMarkdown(d, { warnings, strict: true });
expect(warnings).toEqual([]);
expect(md).toContain("## Title");
});
});
@@ -1,12 +1,15 @@
import { describe, it, expect } from 'vitest';
import { convertProseMirrorToMarkdown } from '../src/lib/markdown-converter.js';
import { markdownToProseMirror } from '../src/lib/markdown-to-prosemirror.js';
import {
convertProseMirrorToMarkdown,
markdownToProseMirror,
} from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from './foreign-markdown';
normalizeForeignMarkdown,
normalizeAgentMarkdown,
} from '../src/lib/foreign-markdown.js';
/**
* STEP 2 goldens for issue #345: the foreign-markdown normalizer that runs at the
* import boundary BEFORE the strict canonical parser (`markdownToProseMirror`).
* STEP 2 goldens for issue #345 (moved into the package with the normalizer in
* #493): the foreign-markdown normalizer that runs at the import boundary BEFORE
* the strict canonical parser (`markdownToProseMirror`).
*
* Two layers:
* 1. PURE stringstring cases pinning the normalizer's own behavior (GFM
@@ -216,3 +219,53 @@ describe('foreign markdown import acceptance (normalizer + canonical parser)', (
).toHaveLength(1);
});
});
describe('normalizeAgentMarkdown vs normalizeForeignMarkdown — front-matter strip is IMPORT-only (#493 review)', () => {
// A page that OPENS with a horizontalRule and contains a later `---` serializes
// to a `---…---`-shaped body. On a full-body AGENT rewrite this must NOT be
// mistaken for YAML front-matter and stripped — that silently dropped the
// page's leading content.
const rulePage = '---\n\nIntro\n\nMore\n\n---\n\nRest';
it('normalizeAgentMarkdown does NOT strip a leading ---…--- (no content loss)', () => {
expect(normalizeAgentMarkdown(rulePage)).toBe(rulePage);
});
it('normalizeForeignMarkdown (file import) STILL strips a real leading YAML front-matter block', () => {
const withYaml = '---\ntitle: My Page\ntags: [a, b]\n---\n\nBody here.';
const out = normalizeForeignMarkdown(withYaml);
expect(out).toBe('Body here.');
// And the horizontalRule-shaped body IS stripped on the import path (its
// documented file-import behavior) — the two variants differ ONLY here.
expect(normalizeForeignMarkdown(rulePage)).not.toContain('Intro');
});
it('agent-write round-trip keeps a horizontalRule-led doc with a second rule intact', async () => {
// Simulate the serializer output for [horizontalRule, para, para, horizontalRule, para].
const doc = {
type: 'doc',
content: [
{ type: 'horizontalRule' },
{ type: 'paragraph', content: [{ type: 'text', text: 'Intro' }] },
{ type: 'paragraph', content: [{ type: 'text', text: 'More' }] },
{ type: 'horizontalRule' },
{ type: 'paragraph', content: [{ type: 'text', text: 'Rest' }] },
],
};
const body = convertProseMirrorToMarkdown(doc);
// The agent-write normalization must NOT eat the head; re-import keeps every
// paragraph's text.
const back = await markdownToProseMirror(normalizeAgentMarkdown(body));
const texts = JSON.stringify(back);
for (const t of ['Intro', 'More', 'Rest']) expect(texts).toContain(t);
// Both horizontal rules survive.
expect(back.content.filter((n: any) => n.type === 'horizontalRule')).toHaveLength(2);
});
it('agent-write STILL rewrites GFM reference footnotes (the shared drift-fix)', () => {
const gfm = 'See[^1].\n\n[^1]: the note.';
const out = normalizeAgentMarkdown(gfm);
expect(out).toContain('^[the note.]');
expect(out).not.toMatch(/\[\^1\]:/);
});
});
@@ -11,9 +11,11 @@
*
* The corpus deliberately spans the CommonMark / canon hostile alphabet
* (`* _ [ ] ( ) { } | < > & # ! ~ = + -`), unicode / emoji / RTL, and the legal
* mark combinations on runs (including the `code` mark, which the schema's
* `excludes: "_"` makes suppress every co-occurring mark so it is never
* combined with another mark in the byte-stable space).
* mark combinations on runs. As of #515 the `code` mark no longer excludes other
* marks (`excludes: ""`), so the corpus ALSO combines `code` with bold / italic /
* strike / highlight exercising both the HOMOGENEOUS run factoring (adjacent
* code+bold spans -> `` **`a` `b`** ``) and the HETEROGENEOUS anti-collision
* fallback (`[code,bold]` next to `[italic]` -> schema-HTML, never `` `a`***b* ``).
*/
import fc from 'fast-check';
@@ -106,16 +108,16 @@ export const urlArb: fc.Arbitrary<string> = fc
/**
* A text run with an OPTIONAL single non-code formatting mark (bold/italic/
* strike/underline/superscript/subscript/spoiler), or a SOLE `code` mark, or a
* link, or an inline comment anchor. `code` is NEVER combined with another mark
* in the byte-stable space (that combination is a documented converter
* limitation the schema's `code` mark declares `excludes: "_"`). Marks wrap
* `safeTextArb`, which stays stable even when it contains isolated specials.
* `code` mark COMBINED with a bare-delimiter emphasis mark (#515), or a link, or
* an inline comment anchor. Marks wrap `safeTextArb`, which stays stable even
* when it contains isolated specials.
*
* The mark set here is broadened past the sibling test's {bold,italic,strike}
* to also cover underline / superscript / subscript / spoiler / textStyle /
* highlight (all single, non-code marks), so the marks-on-text generator
* exercises every mark the schema declares except the deliberately-excluded
* `code`+other combination.
* The mark set here is broadened past the sibling test's {bold,italic,strike} to
* also cover underline / superscript / subscript / spoiler / textStyle /
* highlight (all single, non-code marks). As of #515 it ALSO emits `code`
* combined with bold/italic/strike, so the assembled inline content exercises the
* converter's code-emphasis run detection (adjacent combos -> homogeneous
* factoring or heterogeneous HTML fallback, both lossless).
*/
export const markedTextRunArb: fc.Arbitrary<any> = fc.oneof(
// Plain text.
@@ -138,6 +140,25 @@ export const markedTextRunArb: fc.Arbitrary<any> = fc.oneof(
// Sole code mark (backtick span). safeTextArb is backtick-free, so the span
// content cannot contain an inner backtick.
safeTextArb.map((t) => ({ type: 'text', text: t, marks: [{ type: 'code' }] })),
// #515: code COMBINED with a bare-delimiter emphasis mark. The converter nests
// the backtick span inside the emphasis delimiters (`` **`x`** ``) and, when
// such runs sit adjacent, factors a shared mark or falls back to schema-HTML.
// Mark order here is `[emphasis, code]` — the order the HTML->PM import yields
// for bold/italic/strike specifically (code last). This is NOT universal: the
// `==`-highlight case below imports code FIRST — so match each case to its own
// imported order for the order-exact P1 round-trip (do not assume a fixed order).
fc
.tuple(safeTextArb, fc.constantFrom('bold', 'italic', 'strike'))
.map(([t, m]) => ({ type: 'text', text: t, marks: [{ type: m }, { type: 'code' }] })),
// #515: code combined with an UNCOLORED highlight (also a bare-delimiter mark,
// `==…==`), so the highlight+code delimiter interaction is covered too. Import
// yields `[code, highlight]` here (the `==` inline extension nests code first),
// so the generator matches that order for the order-exact P1 round-trip.
safeTextArb.map((t) => ({
type: 'text',
text: t,
marks: [{ type: 'code' }, { type: 'highlight' }],
})),
// Link with safe text, a paren/space-free href, optionally a letter-bearing
// title (a purely numeric title is coerced to a number and dropped).
fc
@@ -212,25 +233,93 @@ export function normalizeInline(nodes: any[]): any[] {
return out;
}
/**
* #493 commit 1: a plain-text run whose text DELIBERATELY OPENS with a markdown
* BLOCK trigger ATX heading `#`, bullet `-`/`*`/`+`, blockquote `>`, ordered
* `N.`/`N)`, or a table `|` followed by safe text. Pre-#493 the corpus
* self-censored these away (safeTextArb's leading-word guarantee); the paragraph
* serializer now BLOCK-ESCAPES a leading trigger, so the generative round-trip
* itself proves the data-loss class is closed rather than avoiding it.
*
* DELIBERATELY excludes the code-fence (backtick) trigger the backtick is a
* code-span delimiter that re-pairs globally (see specialCharArb's note), an
* instability UNRELATED to block-escape and the whole-line thematic break
* (`---`), which only triggers when the line is ONLY dashes; both are covered by
* the deterministic pin (gitmost-transcript-neutralization.test.ts). Each still
* ENDS in a word (safeTextArb) so adjacent-run concatenation stays safe.
*/
export const blockTriggerLeadRunArb: fc.Arbitrary<any> = fc
.tuple(
fc.constantFrom('# ', '## ', '- ', '* ', '+ ', '> ', '1. ', '1) ', '| '),
safeTextArb,
)
.map(([trigger, rest]) => ({ type: 'text', text: trigger + rest }));
/**
* A hardBreak IMMEDIATELY followed by a block-trigger-leading run a two-node
* segment. Because a hardBreak serializes as ` \n`, the trigger then sits at
* the START of a CONTINUATION line, exercising the serializer's PER-LINE block
* escape (not just the first line). #493 review: without this the fuzzer never
* placed a trigger after a hardBreak, so a single-line-only escape passed P1P3.
*/
export const hardBreakThenTriggerArb: fc.Arbitrary<any[]> = fc
.tuple(hardBreakArb, blockTriggerLeadRunArb)
.map(([hb, trigger]) => [hb, trigger]);
/**
* #493 (setext data-loss): a WHOLE-LINE setext underline landing on a
* continuation line. A setext underline is a line of ONLY `-` (any count) or
* ONLY `=` (any count) that FOLLOWS a paragraph line; on re-parse it turns the
* preceding line into a heading and DROPS its own text. The block-escape must
* neutralize it. Unlike blockTriggerLeadRunArb, the underline must occupy the
* whole line, so we sandwich it between two hardBreaks (underline on its own
* line, preceded by earlier paragraph content, followed by a trailing word so
* the closing hardBreak is not dropped by normalizeInline). Covers underlines
* of every length: `--` (the two-dash case the bullet/thematic arms miss), a
* lone `=`, `==`/`====` (neutralized by the inline `==` escape), and `---`/
* `----` (regression for the existing thematic case).
*/
export const hardBreakThenSetextArb: fc.Arbitrary<any[]> = fc
.tuple(
fc.constantFrom('--', '=', '==', '====', '---', '----'),
safeTextArb,
)
.map(([underline, rest]) => [
{ type: 'hardBreak' },
{ type: 'text', text: underline },
{ type: 'hardBreak' },
{ type: 'text', text: rest },
]);
/**
* Inline content for a paragraph: at least one marked text run, optionally with
* inline atoms (math/mention) and hard breaks interspersed. Always starts with a
* text run so the paragraph never opens with a block trigger. (Ported.)
* inline atoms (math/mention) and hard breaks interspersed. The FIRST run is
* usually an ordinary marked run, but sometimes a block-trigger-leading run
* (blockTriggerLeadRunArb) so the paragraph OPENS with a markdown block trigger;
* and a `hardBreak + trigger` segment can appear anywhere in the rest, so a
* trigger also lands at the start of a CONTINUATION line both exercising the
* serializer's per-line block-escape end-to-end. (Ported, with the #493
* leading-trigger + post-hardBreak dimensions added.)
*/
export const inlineContentArb: fc.Arbitrary<any[]> = fc
.tuple(
markedTextRunArb,
fc.oneof(
{ weight: 5, arbitrary: markedTextRunArb },
{ weight: 1, arbitrary: blockTriggerLeadRunArb },
),
fc.array(
fc.oneof(
{ weight: 5, arbitrary: markedTextRunArb },
{ weight: 1, arbitrary: mathInlineArb },
{ weight: 1, arbitrary: mentionArb },
{ weight: 1, arbitrary: hardBreakArb },
{ weight: 5, arbitrary: markedTextRunArb.map((n) => [n]) },
{ weight: 1, arbitrary: mathInlineArb.map((n) => [n]) },
{ weight: 1, arbitrary: mentionArb.map((n) => [n]) },
{ weight: 1, arbitrary: hardBreakArb.map((n) => [n]) },
{ weight: 2, arbitrary: hardBreakThenTriggerArb },
{ weight: 2, arbitrary: hardBreakThenSetextArb },
),
{ minLength: 0, maxLength: 4 },
),
)
.map(([first, rest]) => normalizeInline([first, ...rest]));
.map(([first, rest]) => normalizeInline([first, ...rest.flat()]));
/**
* Inline content for a HEADING identical to a paragraph's, but WITHOUT hard
@@ -5,32 +5,21 @@ import { convertProseMirrorToMarkdown } from "../src/lib/markdown-converter.js";
import { markdownToProseMirror } from "../src/lib/markdown-to-prosemirror.js";
/**
* gitmost #377 (round-1 review, finding #1) proof, against the REAL
* converter, that the transcript-insert boundary defense survives git-sync.
* #493 commit 1 the paragraph serializer's leading-block-escape closes the
* data-loss class where a paragraph whose text opens at column 0 with a markdown
* block trigger (`#`/`-`/`*`/`+`/`>`, an ordered `N.`/`N)`, a code fence, a
* table `|`, a callout opener, or a thematic break) silently re-parsed into a
* heading / list / quote / code block / table / horizontalRule on the git-sync
* doc -> markdown -> doc cycle. The thematic-break case was the worst: a
* horizontalRule carries NO text, so the line's text was lost entirely.
*
* The web bridge (apps/client .../gitmost/gitmost-recording.ts,
* `gitmostInsertTranscriptIntoEditor`) appends each transcript line as a
* PARAGRAPH text node. The paragraph serializer here (`case "paragraph"`) emits
* that text VERBATIM with no block-escape, so a line whose text begins with a
* col-0 markdown block trigger would, on the doc -> markdown -> doc git-sync
* cycle, silently re-parse into a heading / list / quote / callout / code block.
* That missing block-escape is the pre-existing root cause; the bridge's
* boundary defense prepends an invisible zero-width space (U+200B) to a line
* that begins with such a trigger, shifting it off column 0.
*
* This test keeps a COPY of the bridge's trigger regex (the bridge is in a
* different package and can't be imported here) and asserts:
* 1. bare trigger lines DO corrupt (documents the root cause), and
* 2. the ZWSP-neutralized form round-trips as a single PARAGRAPH with the
* text byte-preserved.
* This is the deterministic PIN, one assertion per trigger, exercised through
* the REAL converter round-trip (not a mock): each bare trigger line now
* round-trips as a SINGLE paragraph with its text byte-preserved proving the
* class is closed WITHOUT the former client-side ZWSP workaround (removed) or
* the generative suite's leading-word self-censorship (removed).
*/
const ZWSP = "​"; // U+200B
// MUST stay in sync with GITMOST_MD_BLOCK_TRIGGER_RE in the client bridge.
const MD_BLOCK_TRIGGER_RE =
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
const doc = (...nodes: any[]) => ({ type: "doc", content: nodes });
const para = (t: string) => ({
type: "paragraph",
@@ -43,78 +32,117 @@ const roundtrip = async (text: string) => {
return back.content as any[];
};
describe("gitmost transcript neutralization (git-sync round-trip)", () => {
// Lines that, at column 0, the serializer's missing block-escape would let
// git-sync re-parse into a non-paragraph block.
describe("paragraph block-escape (git-sync round-trip)", () => {
// Every line here, at column 0, WOULD (pre-fix) re-parse into a non-paragraph
// block. Each is now block-escaped by the serializer and round-trips clean.
const triggerLines = [
"- dash",
"* star",
"+ plus",
"> quote",
"# hash",
"## two hash",
"###### six hash",
"1. one",
"1) one",
"> [!info] note",
"```js",
"~~~",
// Solid + spaced thematic breaks — these re-parse into a `horizontalRule`,
// which carries NO text, so a bare separator line LOSES its text entirely
// (round-2 finding). `_` also only forms a block via this construct.
"| a | b |",
// Solid + spaced thematic breaks — the text-LOSING case pre-fix.
"---",
"***",
"___",
"- - -", // spaced dash break (solid form is caught by [-*+]\s too, but this is the break)
"- - -",
"_ _ _",
];
it("BARE trigger lines corrupt into non-paragraph blocks (root cause)", async () => {
it("every bare trigger line round-trips as a single paragraph, text byte-preserved", async () => {
for (const line of triggerLines) {
const blocks = await roundtrip(line);
// At least one produced block is NOT a paragraph — i.e. corruption.
const allParagraphs = blocks.every((b) => b.type === "paragraph");
expect(
allParagraphs,
`expected "${line}" to corrupt when inserted bare`,
).toBe(false);
}
});
it("BARE solid thematic breaks corrupt into a text-LOSING horizontalRule", async () => {
// The severe case: no text node survives. Documents why neutralization
// matters more here than for list/quote (where the text survived).
for (const line of ["---", "***", "___"]) {
const blocks = await roundtrip(line);
expect(blocks.map((b) => b.type)).toContain("horizontalRule");
// No block carries the original text anywhere.
const flat = JSON.stringify(blocks);
expect(flat).not.toContain(line);
}
});
it("ZWSP-neutralized trigger lines round-trip as a single paragraph, text preserved", async () => {
for (const line of triggerLines) {
// The regex must actually classify each as a trigger.
expect(MD_BLOCK_TRIGGER_RE.test(line), `regex missed "${line}"`).toBe(
true,
expect(blocks, `"${line}" should be one block`).toHaveLength(1);
expect(blocks[0].type, `"${line}" should stay a paragraph`).toBe(
"paragraph",
);
const neutralized = ZWSP + line;
const blocks = await roundtrip(neutralized);
expect(blocks).toHaveLength(1);
expect(blocks[0].type).toBe("paragraph");
// Text is byte-preserved (ZWSP + original line), so the display is the
// original line with only an invisible leading character.
expect(blocks[0].content[0].text).toBe(neutralized);
expect(
blocks[0].content?.[0]?.text,
`"${line}" text should survive byte-exact`,
).toBe(line);
}
});
it("normal host-prefixed lines never match the trigger regex and round-trip byte-exact", async () => {
it("emphasis / inline-code paragraphs are NOT escaped (no backslash churn)", async () => {
// These open with `*`/`` ` `` but are NOT block triggers; the serialized
// markdown must not gain a stray leading backslash, and they round-trip.
for (const [text, mark] of [
["bold", "bold"],
["italic", "italic"],
["code", "code"],
] as const) {
const node = doc({
type: "paragraph",
content: [{ type: "text", text, marks: [{ type: mark }] }],
});
const md = convertProseMirrorToMarkdown(node);
expect(md.startsWith("\\"), `${mark} must not be block-escaped`).toBe(
false,
);
const back = await markdownToProseMirror(md);
expect(back.content[0].type).toBe("paragraph");
expect(back.content[0].content[0].text).toBe(text);
expect(back.content[0].content[0].marks?.[0]?.type).toBe(mark);
}
});
it("a block trigger on a CONTINUATION line (after a hardBreak) is escaped too", async () => {
// A hardBreak serializes as ` \n`, so a trigger on the second line would,
// without a per-line escape, re-parse into another block. The worst case is
// `---`: a setext underline would turn the first line into a heading and LOSE
// the `---` text entirely. Each pair round-trips as ONE paragraph with the
// hardBreak and both texts preserved.
for (const [first, second] of [
["a", "# b"],
["a", "- b"],
["a", "> b"],
["a", "1. b"],
["a", "| b |"],
["a", "---"], // setext / thematic (3 dashes) — the text-losing case
["a", "--"], // setext underline, EXACTLY two dashes (bullet/thematic miss it)
["a", "----"], // setext / thematic (4 dashes)
["a", "="], // setext H1 underline, a lone `=` (no other arm covers it)
["a", "===="], // setext H1 underline, run of `=`
]) {
const d = doc({
type: "paragraph",
content: [
{ type: "text", text: first },
{ type: "hardBreak" },
{ type: "text", text: second },
],
});
const back = await markdownToProseMirror(convertProseMirrorToMarkdown(d));
expect(back.content, `"${first}${second}" should be one block`).toHaveLength(1);
expect(back.content[0].type).toBe("paragraph");
const texts = (back.content[0].content as any[])
.filter((n) => n.type === "text")
.map((n) => n.text);
const hasBreak = (back.content[0].content as any[]).some(
(n) => n.type === "hardBreak",
);
expect(hasBreak, `"${first}${second}" should keep the hardBreak`).toBe(true);
expect(texts, `"${first}${second}" should preserve both line texts`).toEqual([
first,
second,
]);
}
});
it("normal host-prefixed lines round-trip byte-exact (unaffected)", async () => {
for (const line of [
"You: hello there",
"Speaker 1: - and then a dash mid-line",
"Speaker 2: 1. not a list",
]) {
expect(MD_BLOCK_TRIGGER_RE.test(line)).toBe(false);
const blocks = await roundtrip(line);
expect(blocks).toHaveLength(1);
expect(blocks[0].type).toBe("paragraph");
@@ -294,10 +294,11 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
);
});
// 5. code + link co-occur: the schema's `code` mark excludes all other marks
// (including link), so the link cannot survive import. The lossless,
// byte-stable behavior is to emit ONLY the backtick code span (code wins).
it('a code+link run emits the backtick code form (code wins, link dropped)', () => {
// 5. code + link co-occur (#515): `code` no longer excludes other marks, so a
// link can wrap inline code. The code span is emitted innermost and the link
// wraps it — CommonMark allows inline code inside link text, so it survives
// the round trip.
it('a code+link run nests the backtick span inside the link (#515)', () => {
const out = convertProseMirrorToMarkdown(
doc(
para({
@@ -310,7 +311,7 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
}),
),
);
expect(out).toBe('`x`');
expect(out).toBe('[`x`](http://a?b&c"d)');
});
// 6. hardBreak inside a heading: prefix applied once, " \n" between a and b.
@@ -430,7 +431,7 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
});
});
describe('converter gap coverage — documented round-trip data loss (specs 12–14)', () => {
describe('converter gap coverage — formerly-lossy round-trips, now closed (specs 12–14)', () => {
// 12. A 3-backtick fence inside a codeBlock body is now lengthened: the outer
// fence widens to (longest inner run + 1) backticks per CommonMark, so the
// inner ``` is treated as content and the block survives as ONE node.
@@ -460,25 +461,24 @@ describe('converter gap coverage — documented round-trip data loss (specs 12
expect(docsCanonicallyEqual(d, doc2)).toBe(false);
});
// 13. A leading ordered-list marker in paragraph text is NOT escaped, so a
// plain paragraph silently becomes an orderedList on re-import.
it('a paragraph starting with "1. " is promoted to an orderedList on re-import', async () => {
// 13. #493 commit 1: a leading ordered-list marker in paragraph text is now
// BLOCK-ESCAPED, so the paragraph round-trips as a paragraph instead of
// silently becoming an orderedList (was documented data loss, now closed).
it('a paragraph starting with "1. " is block-escaped and stays a paragraph', async () => {
const d = doc({
type: 'paragraph',
content: [{ type: 'text', text: '1. not a list' }],
});
const md1 = convertProseMirrorToMarkdown(d);
expect(md1).toBe('1. not a list'); // no backslash escape
expect(md1).toBe('1\\. not a list'); // the ordered-list delimiter is escaped
const doc2 = await markdownToProseMirror(md1);
expect(doc2.content?.[0]?.type).toBe('orderedList');
const li = doc2.content[0].content?.[0];
expect(li?.type).toBe('listItem');
expect(li.content?.[0]?.content?.[0]).toMatchObject({
expect(doc2.content?.[0]?.type).toBe('paragraph');
expect(doc2.content[0].content?.[0]).toMatchObject({
type: 'text',
text: 'not a list', // the "1. " was consumed as a list marker
text: '1. not a list', // the escape decodes back to the literal text
});
expect(docsCanonicallyEqual(d, doc2)).toBe(false);
expect(docsCanonicallyEqual(d, doc2)).toBe(true);
});
// 14. #293 canon #4: the image title now round-trips via the attached
@@ -59,22 +59,21 @@ describe('convertProseMirrorToMarkdown', () => {
).toBe('`x`');
});
it('code + another mark emits the backtick code form (code wins)', () => {
// The schema's `code` mark excludes all other marks, so the editor can
// never produce code+bold on one run and import always drops the co-mark.
// The lossless, byte-stable behavior is to emit ONLY the backtick code
// span and ignore the co-occurring mark.
it('code + bold nests the backtick span inside the emphasis (#515)', () => {
// #515: the `code` mark no longer excludes other marks (`excludes: ""`), so
// a run can carry code+bold. CommonMark nests them (`<strong><code>`), so
// the code span is emitted innermost and the bold delimiters wrap it.
const out = convertProseMirrorToMarkdown(
doc(para(text('x', [{ type: 'bold' }, { type: 'code' }]))),
);
expect(out).toBe('`x`');
expect(out).toBe('**`x`**');
});
it('code + strike combo emits the backtick code form (code wins)', () => {
it('code + strike nests the backtick span inside the emphasis (#515)', () => {
const out = convertProseMirrorToMarkdown(
doc(para(text('x', [{ type: 'strike' }, { type: 'code' }]))),
);
expect(out).toBe('`x`');
expect(out).toBe('~~`x`~~');
});
});
@@ -80,13 +80,7 @@ import { stripBlockIds } from './roundtrip-helpers.js';
// `it.fails` blocks below (so the suite stays green only because they are marked
// expected-to-fail, never by hiding them):
//
// 1. The `code` mark COMBINED with any other mark. The converter emits nested
// HTML (`<strong><code>x</code></strong>`), but the schema's `code` mark
// declares `excludes: "_"`, so on import every co-occurring mark is dropped
// and the run comes back as `code` only -> md2 == "`x`". Acknowledged in
// markdown-converter.ts (the long comment above the marks switch);
// impossible to round-trip both while `code` excludes them.
// 2. A BLOCK-level `image` placed BETWEEN other blocks. The Docmost image node
// 1. A BLOCK-level `image` placed BETWEEN other blocks. The Docmost image node
// is block-level but `![](url)` is inline; marked wraps it in a <p>, the
// schema hoists the <img> out and leaves an empty paragraph sibling, which
// injects an extra blank gap on the second export. An image IS byte-stable
@@ -625,7 +619,7 @@ describe('markdown <-> ProseMirror round-trip (property-based)', () => {
});
// -------------------------------------------------------------------------
// KNOWN, DOCUMENTED non-roundtrip bug #2 (kept honest as it.fails).
// KNOWN, DOCUMENTED non-roundtrip bug #1 (kept honest as it.fails).
//
// BUG: a block-level `image` placed BETWEEN other blocks is not byte-stable.
// The Docmost image node is BLOCK-level but its markdown form `![](url)` is
@@ -655,23 +649,18 @@ describe('markdown <-> ProseMirror round-trip (property-based)', () => {
});
// -------------------------------------------------------------------------
// KNOWN, DOCUMENTED non-roundtrip bug #1 (kept honest as it.fails).
// #515 ROUND-TRIP PIN: `code` combined with another mark.
//
// BUG: the `code` mark combined with ANY other mark does NOT round-trip.
// The converter emits nested HTML so the output is well-formed, e.g.
// marks [code, bold] -> md1 = "<strong><code>x</code></strong>"
// but the schema's `code` mark declares `excludes: "_"`, so on import the
// co-occurring mark is dropped and the run comes back as code-only:
// md2 = "`x`" (=> md2 !== md1).
// Minimal repro doc:
// { type:'doc', content:[ { type:'paragraph', content:[
// { type:'text', text:'x', marks:[{type:'code'},{type:'bold'}] } ] } ] }
// This is acknowledged in markdown-converter.ts (the long comment above the
// marks switch): preserving both marks is impossible while `code` excludes
// them. Documented here, not "fixed", because the source must not change.
// Before #515 the `code` mark declared `excludes: "_"`, dropping every co-
// occurring mark on import so `` **`x`** `` came back as code-only. Now
// `excludes: ""` lets code combine with all marks (CommonMark nests them,
// `<strong><code>x</code></strong>`), so the run BOTH round-trips byte-stably
// AND preserves the co-occurring mark. This asserts the observable property in
// both directions: md2 === md1 (idempotent export) and the imported doc still
// carries [code, other].
// -------------------------------------------------------------------------
it(
'code mark combined with another mark is byte-stable',
'code combined with another mark round-trips and keeps both marks (#515)',
async () => {
const codeComboArb = fc
.tuple(safeTextArb, fc.constantFrom('bold', 'italic', 'strike'))
@@ -688,11 +677,90 @@ describe('markdown <-> ProseMirror round-trip (property-based)', () => {
}));
await fc.assert(
fc.asyncProperty(codeComboArb, async (doc) => {
const { md1, md2 } = await roundTrip(doc);
const { md1, md2, doc2 } = await roundTrip(doc);
expect(md2).toBe(md1);
// The re-imported run carries BOTH code and the co-occurring mark.
const run = doc2?.content?.[0]?.content?.[0];
const markTypes = (run?.marks || []).map((m: any) => m.type).sort();
expect(markTypes).toContain('code');
expect(markTypes.length).toBe(2);
}),
{ numRuns: 20, seed: SEED },
);
},
);
// -------------------------------------------------------------------------
// #515 REPRO CASES: the five markdown inputs from the issue must import to a
// code+bold node (import correctness) AND re-export byte-stably with no
// dangling `**` (export correctness). Import direction is checked against the
// real markdown->PM bridge; export direction via the md->pm->md fixpoint.
// -------------------------------------------------------------------------
it('the five #515 repro cases import to [code,bold] and round-trip clean', async () => {
// Collect every inline text run in a doc with its mark type set.
const runs = (node: any): { text: string; marks: string[] }[] => {
if (node?.type === 'text') {
return [{ text: node.text || '', marks: (node.marks || []).map((m: any) => m.type) }];
}
return (node?.content || []).flatMap(runs);
};
const findRun = (doc: any, text: string) =>
runs(doc).find((r) => r.text === text);
// Case 1: **`code1`** -> code1 = [code, bold].
{
const md = '**`code1`**';
const pm = await markdownToProseMirror(md);
const r = findRun(pm, 'code1');
expect(r?.marks.sort()).toEqual(['bold', 'code']);
const md2 = convertProseMirrorToMarkdown(pm);
expect(md2).toBe('**`code1`**');
// md -> pm -> md fixpoint.
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
}
// Case 2: **`aaa` + `bbb`** -> aaa,bbb = [code,bold], "+" carries bold; no
// dangling `**` on export.
{
const md = '**`aaa` + `bbb`**';
const pm = await markdownToProseMirror(md);
expect(findRun(pm, 'aaa')?.marks.sort()).toEqual(['bold', 'code']);
expect(findRun(pm, 'bbb')?.marks.sort()).toEqual(['bold', 'code']);
const md2 = convertProseMirrorToMarkdown(pm);
expect(md2).toBe('**`aaa` + `bbb`**');
// NOT the old broken export with the bold delimiters split onto each span.
expect(md2).not.toBe('`aaa`** + **`bbb`');
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
}
// Case 3 (control): **bold3** and `code3` -> bold and code stay SEPARATE.
{
const md = '**bold3** and `code3`';
const pm = await markdownToProseMirror(md);
expect(findRun(pm, 'bold3')?.marks).toEqual(['bold']);
expect(findRun(pm, 'code3')?.marks).toEqual(['code']);
const md2 = convertProseMirrorToMarkdown(pm);
expect(md2).toBe('**bold3** and `code3`');
}
// Case 4: **`code4` tail** -> code4 = [code,bold], " tail" = [bold].
{
const md = '**`code4` tail**';
const pm = await markdownToProseMirror(md);
expect(findRun(pm, 'code4')?.marks.sort()).toEqual(['bold', 'code']);
const md2 = convertProseMirrorToMarkdown(pm);
expect(md2).toBe('**`code4` tail**');
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
}
// Case 5: pre **`code5`** post -> code5 = [code,bold], surroundings plain.
{
const md = 'pre **`code5`** post';
const pm = await markdownToProseMirror(md);
expect(findRun(pm, 'code5')?.marks.sort()).toEqual(['bold', 'code']);
const md2 = convertProseMirrorToMarkdown(pm);
expect(md2).toBe('pre **`code5`** post');
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
}
});
});
@@ -16,14 +16,16 @@ import * as editorExt from "@docmost/editor-ext";
// or mark added upstream that the mirror forgets to vendor fails CI loudly
// (otherwise it is silently dropped on the markdown <-> ProseMirror round-trip).
//
// LIMITATION (intentional, see schema-surface-snapshot.test.ts): this is a
// NAME-LEVEL contract only, not a full attribute-level structural compare.
// editor-ext's Tiptap representation (node views, commands, suggestion plugins,
// addGlobalAttributes spread across separate extensions) differs from this
// minimal mirror, so a mechanical attribute-by-attribute equality would be
// fragile and produce false drift. Attribute parity is guarded by the inline
// surface snapshot (reviewed in every diff); this test guards that no canonical
// node/mark TYPE goes unmirrored. StarterKit-provided types (paragraph, bold,
// This file now holds TWO contracts (see the two describe blocks): the original
// NAME-LEVEL type contract (no canonical node/mark TYPE goes unmirrored) AND, as
// of #493, an ATTRIBUTE-LEVEL contract that compares each editor-ext node/mark's
// OWN declared attributes (names + defaults) against the mirror's built schema.
// A full mechanical attribute-by-attribute EQUALITY would be fragile (the mirror
// is a deliberate superset: it injects the global id/textAlign/indent attrs and
// normalizes some editor-ext defaults to null), so the attribute contract is
// asymmetric — editor-ext -> mirror — with a small, reasoned, stale-guarded
// allowlist for the two blessed divergence kinds (non-round-trippable omissions
// and null-normalized defaults). StarterKit-provided types (paragraph, bold,
// heading, …) are contributed by @tiptap/starter-kit in the mirror rather than
// by editor-ext, so they are naturally covered by the mirror's superset.
//
@@ -85,3 +87,224 @@ describe("docmost schema vs @docmost/editor-ext (name-level contract)", () => {
expect(missing).toEqual([]);
});
});
// ── #515 CODE-MARK `excludes` PARITY (data-loss-sensitive) ──────────────────
//
// The `code` mark's `excludes` field decides whether inline code can co-occur
// with other marks. #515 sets it to "" (excludes nothing) in the canonical
// `Code` exported by @docmost/editor-ext AND, because the vendored markdown
// mirror must NOT pull that React-aware package into its node runtime, RE-DECLARES
// the same override locally in docmost-schema.ts. If the two drift, markdown
// import would silently strip bold/italic adjacent to inline code again. Guard it
// mechanically: the mirror's built `code` mark and the canonical editor-ext
// `Code` must agree on `excludes` (both ""). getSchema surfaces the resolved
// value on the mark spec.
describe("docmost schema vs @docmost/editor-ext (#515 code excludes parity)", () => {
it("keeps the vendored `code` mark's excludes in lockstep with editor-ext Code", () => {
// Mirror side: the value the mirror's BUILT schema resolves for `code`.
const mirrorExcludes = getSchema(docmostExtensions as never).marks.code.spec
.excludes;
// Canonical side: the `excludes` DECLARED on the editor-ext `Code` extension
// (read from its config — getSchema needs a full node set, so a lone mark
// can't be built into a schema here).
const canonicalCode = (
editorExt as unknown as { Code?: { config?: { excludes?: unknown } } }
).Code;
const canonicalExcludes = canonicalCode?.config?.excludes;
// Both must be the empty string: `code` excludes NOTHING, so bold/italic/…
// survive alongside inline code (#515). A drift here would silently strip
// marks adjacent to code on markdown import again.
expect(canonicalCode).toBeDefined();
expect(mirrorExcludes).toBe("");
expect(canonicalExcludes).toBe("");
expect(mirrorExcludes).toBe(canonicalExcludes);
});
});
// ── ATTRIBUTE-LEVEL CONTRACT (#493 commit 2) ────────────────────────────────
//
// The name-level contract above catches a WHOLE node/mark type going unmirrored,
// but not ATTRIBUTE drift within a vendored type — the exact class that silently
// dropped `subpages.recursive`: editor-ext grew an attribute the hand-synced
// mirror forgot, so documents using it lost that attribute on a git-sync
// round-trip while CI stayed green. This closes that gap by comparing each
// editor-ext node/mark's OWN declared attributes (names + defaults) against the
// mirror's built ProseMirror schema `spec.attrs`.
//
// DIRECTION: editor-ext -> mirror. The mirror is deliberately a SUPERSET (it
// injects the global `id`/`textAlign`/`indent` attributes and normalizes some
// editor-ext "required" attrs to a `null` default), so a reverse compare would
// be pure false drift; the meaningful failure is an editor-ext attribute the
// mirror DROPS (name) or whose DEFAULT it silently changes. Both directions of
// staleness are guarded so the allowlists cannot rot.
/**
* The attributes an editor-ext Tiptap Node/Mark DECLARES itself, read from its
* `config.addAttributes()`. Global attributes injected by separate extensions
* (unique-id, indent, textAlign) are NOT included here they are the mirror's
* superset and are not part of a per-type declaration so this isolates each
* type's own contribution. A declared attribute with no explicit `default` is a
* required attr (Tiptap default `undefined`); we surface that as-is so the
* default compare can skip it (the mirror makes such attrs optional/`null`).
*/
function editorExtOwnAttrs(): Map<
string,
{ kind: "node" | "mark"; attrs: Record<string, unknown> }
> {
const out = new Map<
string,
{ kind: "node" | "mark"; attrs: Record<string, unknown> }
>();
for (const value of Object.values(editorExt)) {
if (!isTiptapNodeOrMark(value)) continue;
const ext = value as unknown as {
name: string;
type: "node" | "mark";
options?: unknown;
storage?: unknown;
config?: { addAttributes?: () => Record<string, { default?: unknown }> };
};
const fn = ext.config?.addAttributes;
// addAttributes reads `this.options`/`this.name`; bind a minimal context
// (verified sufficient for every editor-ext extension — none reach for
// `this.editor` here). A type with no addAttributes contributes no attrs.
const declared =
typeof fn === "function"
? fn.call({
options: ext.options ?? {},
name: ext.name,
parent: undefined,
storage: ext.storage ?? {},
} as never)
: {};
const attrs: Record<string, unknown> = {};
for (const [attr, spec] of Object.entries(declared || {})) {
// `undefined` marks a required (no-default) attr; keep it so the default
// compare can distinguish "no default declared" from "default is null".
attrs[attr] = (spec as { default?: unknown })?.default;
}
out.set(ext.name, { kind: ext.type, attrs });
}
return out;
}
/** The mirror's built-schema `spec.attrs` for a type: attr name -> default. */
function mirrorAttrs(
name: string,
kind: "node" | "mark",
): Record<string, unknown> | null {
const schema = getSchema(docmostExtensions as never);
const spec = kind === "node" ? schema.nodes[name]?.spec : schema.marks[name]?.spec;
if (!spec) return null;
const out: Record<string, unknown> = {};
for (const [attr, def] of Object.entries(spec.attrs || {})) {
out[attr] = (def as { default?: unknown }).default;
}
return out;
}
// An editor-ext attribute the mirror deliberately does NOT vendor because it has
// NO markdown round-trip representation — dropping it loses nothing on the
// git-sync cycle (the same rationale the flat-roundtrip property suite uses to
// allowlist e.g. `tableCell.backgroundColorName`). Blessed by the hand-curated
// surface snapshot (schema-surface-snapshot.test.ts), reviewed in every diff.
const ACCEPTED_ATTR_OMISSIONS = new Set<string>([
"highlight.colorName", // only `highlight.color` round-trips (==text==); the
// secondary palette-name is presentational and has no markdown form.
]);
// An editor-ext attribute the mirror vendors but with a DIFFERENT default: the
// mirror normalizes an "absent" value to `null` (its uniform optional-attr
// convention) rather than editor-ext's UI-oriented default. None of these attrs
// is emitted on the markdown surface (the converter round-trips only the
// serializable ones), so the default never round-trips and the divergence is
// inert — but pinned here so a NEW default change on either side forces review.
const ACCEPTED_DEFAULT_DIVERGENCE = new Set<string>([
"image.src", // mirror null vs editor "" (an image is never emitted src-less)
"link.internal", // mirror null vs editor false (routing attr, not in md link)
"pdf.width", // mirror null vs editor 800 (presentational sizing, not in md)
"pdf.height", // mirror null vs editor 600 (presentational sizing, not in md)
]);
describe("docmost schema vs @docmost/editor-ext (attribute-level contract)", () => {
it("vendors every editor-ext attribute (name) of every shared type — no silently-dropped attrs", () => {
const dropped: string[] = [];
for (const [name, { kind, attrs }] of editorExtOwnAttrs()) {
const mirror = mirrorAttrs(name, kind);
if (!mirror) continue; // whole-type omission is the name-level test's job
for (const attr of Object.keys(attrs)) {
const key = `${name}.${attr}`;
if (!(attr in mirror) && !ACCEPTED_ATTR_OMISSIONS.has(key)) {
dropped.push(key);
}
}
}
// Any entry here exists on the editor-ext node/mark but NOT in the mirror
// (and is not a blessed non-round-trippable omission): documents using it
// lose that attribute on a git-sync round-trip — the subpages.recursive
// class. Re-sync src/lib/docmost-schema.ts (and the surface snapshot) or add
// a reasoned ACCEPTED_ATTR_OMISSIONS entry before clearing.
expect(dropped.sort()).toEqual([]);
});
it("keeps every editor-ext attribute DEFAULT in sync — no silent default drift", () => {
const drift: string[] = [];
for (const [name, { kind, attrs }] of editorExtOwnAttrs()) {
const mirror = mirrorAttrs(name, kind);
if (!mirror) continue;
for (const [attr, extDefault] of Object.entries(attrs)) {
const key = `${name}.${attr}`;
// Skip attrs editor-ext declares WITHOUT a default (required attrs):
// the mirror deliberately makes them optional (`null`), a safe superset.
if (extDefault === undefined) continue;
if (!(attr in mirror)) continue; // a drop, reported by the name test
if (
JSON.stringify(mirror[attr]) !== JSON.stringify(extDefault) &&
!ACCEPTED_DEFAULT_DIVERGENCE.has(key)
) {
drift.push(
`${key}: mirror=${JSON.stringify(mirror[attr])} editor-ext=${JSON.stringify(extDefault)}`,
);
}
}
}
expect(drift.sort()).toEqual([]);
});
it("the attribute allowlists have no stale rows (each is really omitted / divergent)", () => {
const ext = editorExtOwnAttrs();
const staleOmission: string[] = [];
for (const key of ACCEPTED_ATTR_OMISSIONS) {
const [name, attr] = key.split(".");
const entry = ext.get(name);
const mirror = entry ? mirrorAttrs(name, entry.kind) : null;
// Stale if editor-ext no longer declares it, or the mirror now DOES vendor
// it (so it should be removed from the omission allowlist).
if (!entry || !(attr in entry.attrs) || (mirror && attr in mirror)) {
staleOmission.push(key);
}
}
expect(staleOmission, "stale ACCEPTED_ATTR_OMISSIONS rows").toEqual([]);
const staleDivergence: string[] = [];
for (const key of ACCEPTED_DEFAULT_DIVERGENCE) {
const [name, attr] = key.split(".");
const entry = ext.get(name);
const mirror = entry ? mirrorAttrs(name, entry.kind) : null;
const extDefault = entry?.attrs[attr];
// Stale if the divergence no longer exists (attr gone, or defaults now
// agree) — the row should be dropped so the allowlist stays honest.
if (
!entry ||
!mirror ||
!(attr in entry.attrs) ||
!(attr in mirror) ||
extDefault === undefined ||
JSON.stringify(mirror[attr]) === JSON.stringify(extDefault)
) {
staleDivergence.push(key);
}
}
expect(staleDivergence, "stale ACCEPTED_DEFAULT_DIVERGENCE rows").toEqual([]);
});
});
@@ -9,11 +9,18 @@ import { defineConfig } from 'vitest/config';
// envelope, markdownToProseMirror) is re-exported there.
const here = path.dirname(fileURLToPath(import.meta.url));
const libBarrel = path.resolve(here, 'src/lib/index.ts');
// Resolve the cross-package `@docmost/editor-ext` specifier to the SIBLING
// workspace SOURCE. In a normal checkout this is what pnpm's workspace link +
// the package's `module` field already yield; pinning it here makes the schema
// contract tests (incl. the #515 code-excludes parity) hermetic and independent
// of node_modules layout (e.g. a shared/hoisted store in a git worktree).
const editorExtBarrel = path.resolve(here, '../editor-ext/src/index.ts');
export default defineConfig({
resolve: {
alias: {
'docmost-client': libBarrel,
'@docmost/editor-ext': editorExtBarrel,
},
},
test: {
-79
View File
@@ -1,79 +0,0 @@
#!/usr/bin/env bash
# Boot-smoke for the exact Docker image that is about to be pushed (issue #476).
#
# Retrospective class "local logic is right, the integration property was never
# checked" (#353/#452/#361): every other CI job builds and tests code from the
# working tree, but the IMAGE watchtower pulls was never actually started
# anywhere before this gate. This script boots the built image against the
# publish job's postgres/redis services and asserts four integration
# properties end-to-end:
# S1 the app boots and /api/health answers (startup migrator + boot)
# S2 the first-run workspace setup endpoint works (API + DB writes)
# S3 the client dist is inside the image and served
# S4 hashed assets are served immutable (#452) with the precompressed
# brotli copy shipped in the image
set -euo pipefail
IMAGE="${1:?usage: image-smoke.sh <image>}"
fail() { echo "FAIL: $*"; exit 1; }
# Boot the exact image that will be pushed, wired to the job services via host
# network (postgres on localhost:5432, redis on localhost:6379). The container
# is deliberately NOT removed on failure so the workflow's dump-on-failure step
# can read `docker logs gitmost-smoke`.
docker run -d --name gitmost-smoke --network host \
-e DATABASE_URL=postgresql://docmost:docmost@localhost:5432/docmost \
-e REDIS_URL=redis://localhost:6379 \
-e APP_SECRET=ci-smoke-secret-change-me-min-32-characters \
-e APP_URL=http://localhost:3000 \
"$IMAGE"
# S1: wait for /api/health — covers the startup migrator + boot inside the
# shipped image (#361-boot, #353 runtime class): a migration the Kysely startup
# migrator rejects, or a runtime module missing from the image, dies right here.
healthy=0
for _ in $(seq 1 60); do
if curl -fsS http://localhost:3000/api/health > /dev/null 2>&1; then
healthy=1
break
fi
sleep 2
done
[ "$healthy" -eq 1 ] || fail "S1: /api/health did not answer within 120s (boot or startup migration failed)"
echo "OK S1: image booted and /api/health answers"
# S2: the first-run workspace setup works end-to-end (controller -> service ->
# DB write chain inside the shipped image, not just a static health probe).
curl -fsS -X POST http://localhost:3000/api/auth/setup \
-H "Content-Type: application/json" \
-d '{"name":"Smoke","email":"smoke@example.com","password":"SmokePassword123","workspaceName":"Smoke"}' \
> /dev/null || fail "S2: POST /api/auth/setup failed"
echo "OK S2: workspace setup succeeded"
# S3: the client dist is actually inside the image and served — the SPA HTML
# must reference hashed /assets/ bundles (a broken client COPY in the
# Dockerfile would serve an empty shell that every other job stays green on).
HTML=$(curl -fsS http://localhost:3000/) || fail "S3: fetching / failed"
grep -q '/assets/' <<<"$HTML" || fail "S3: served HTML references no /assets/ bundle (client dist missing from the image?)"
echo "OK S3: client dist served (HTML references /assets/)"
# S4: hashed /assets/ files must be served with an immutable cache-control
# (#452 class: static.module.ts resolveStaticAssetHeaders owns the header) AND
# with the precompressed brotli neighbour. Both checks are mandatory — verified
# against the code: resolveStaticAssetHeaders marks every /assets/ path
# immutable, and the client build (vite-plugin-compression2, include covers
# .js) emits a .br copy next to every bundle that the Dockerfile ships and
# @fastify/static serves via preCompressed:true.
ASSET=$(grep -oE '/assets/[A-Za-z0-9._@/-]+\.js' <<<"$HTML" | head -1 || true)
[ -n "$ASSET" ] || fail "S4: no /assets/*.js path found in the served HTML"
HDRS=$(curl -fsSI -H 'Accept-Encoding: br' "http://localhost:3000$ASSET") || fail "S4: HEAD $ASSET failed"
grep -qi '^cache-control:.*immutable' <<<"$HDRS" || fail "S4: $ASSET served without an immutable cache-control (#452)"
echo "OK S4: hashed asset served with immutable cache-control"
grep -qi '^content-encoding:.*br' <<<"$HDRS" || fail "S4: $ASSET not served brotli-precompressed (content-encoding: br missing)"
echo "OK S4: hashed asset served with the precompressed brotli copy"
# Remove the container ONLY on success, so the failure path keeps it around for
# the workflow's "Dump smoke container log on failure" step.
docker rm -f gitmost-smoke > /dev/null
echo "OK image smoke passed"