vvzvlad/docmost-sync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sync): resolve §11 idempotency via canonical comparison + corpus harness

Browse Source 
Close Задача №0 (SPEC §11) with the spec-sanctioned option (b): compare a
canonicalized ProseMirror form instead of raw bytes.

- canonicalize.ts: canonicalizeContent/docsCanonicallyEqual — strip node attrs.id,
  drop null/undefined attrs, and drop attrs equal to their type's known non-null
  schema default (KNOWN_DEFAULTS: link target/rel, comment.resolved, orderedList.start,
  diagram/media align) so "absent" ≡ "default"; comment anchors + meaningful attrs kept
- roundtrip.ts: assert markdown byte-stability AND canonical stability; add --corpus
  mode and mutually-exclusive-flag warning
- synthetic corpus (headings, marks, lists, table, callout, code w/ trailing \n,
  diagrams, textStyle/mention) + canonicalize/corpus tests (558 green)
- known converter asymmetries (block image after paragraph; embed width/height
  coercion) converge to a fixpoint after one export->import pass -> handled by
  normalize-on-write at vault-write time; isolated under it.fails
- SPEC §11: record the resolution and normalize-on-write strategy
This commit is contained in:
vvzvlad
2026-06-16 23:23:32 +03:00
parent 90d8f86fda
commit 4b34f4d30a
15 changed files with 1258 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
SPEC.md
View File

@@ -256,6 +256,20 @@ git диффает побайтово. Если export недетерминир
Это **Задача №0** перед Фазой 2.
### Резолюция (реализовано)
- **Выбран вариант (б):** сравнение по канонизированной форме — `canonicalizeContent`
снимает node-`attrs.id` и сводит к default'у/убирает дефолтные атрибуты (включая
non-null дефолты схемы: `link.target/rel`, `comment.resolved`, `orderedList.start`,
`*.align`), comment-якоря и значимые атрибуты сохраняются (§3). Состояния синка
сравниваем через `docsCanonicallyEqual`, не побайтово.
- **Markdown byte-stable** на синтетическом корпусе (заголовки, марки, списки,
таблицы, callout'ы, код с хвостовым `\n`, диаграммы) — harness `--corpus`.
- **Известные асимметрии конвертера** (блочная картинка после абзаца добавляет
пустой абзац; диаграмма материализует `data-align`) **сходятся к фикспойнту за
один проход** `export→import→export`. Лечатся **normalize-on-write**: при записи
в vault прогоняем один такой проход, дальше форма стабильна. Глубокий фикс
конвертера не требуется.
---
## 12. Безопасность и эксплуатация

7
packages/docmost-client/src/index.ts
View File

@@ -21,3 +21,10 @@ export type { DocmostMdMeta } from "./lib/markdown-document.js";
export { convertProseMirrorToMarkdown } from "./lib/markdown-converter.js";
export { markdownToProseMirror } from "./lib/collaboration.js";
// docmost-sync addition: semantic canonicalization for the Phase-0 round-trip
// idempotency check (SPEC §11).
export {
canonicalizeContent,
docsCanonicallyEqual,
} from "./lib/canonicalize.js";

250
packages/docmost-client/src/lib/canonicalize.ts Normal file
View File

@@ -0,0 +1,250 @@
/**
* docmost-sync ADDITION (not present in docmost-mcp).
*
* Semantic canonicalization of ProseMirror/TipTap documents for the Phase-0
* round-trip idempotency check (SPEC §11, "Задача №0", option (б): compare a
* CANONICALIZED form rather than raw bytes).
*
* `markdownToProseMirror` reconstructs schema DEFAULT attributes (e.g.
* `indent: null` where the source omitted it) and regenerates per-block ids on
* every import. A raw deep-equal of the source doc against the re-imported doc
* therefore diverges even when the two are semantically identical. This module
* normalizes a document so that two semantically-equal docs compare deep-equal
* regardless of block ids and absent-vs-explicit-default-null attributes.
*
* This file is intentionally a NEW, self-contained module so it is trivial to
* backport into docmost-mcp without touching existing code.
*/
/**
* Known NON-NULL schema defaults that `markdownToProseMirror` materializes on
* import, keyed by node/mark type → { attr: defaultValue }.
*
* Why this exists: `canonicalizeAttrs` already treats an absent attr as
* equivalent to an explicit `null`/`undefined`. But several Docmost schema
* attributes default to a NON-null value, so import fills them in even when the
* source omitted them — making "attr absent" diverge from "attr at its default
* value" under a raw deep-equal. To keep "absent ≡ explicit-default", we ALSO
* drop any attr whose value equals its known schema default. A non-default
* value (e.g. `orderedList.start: 5`) is NOT a default, so it is KEPT.
*
* Every entry below was read from `packages/docmost-client/src/lib/
* docmost-schema.ts` (the line refs are the exact `default:` declarations) and
* confirmed to be materialized by an export→import→export round-trip:
* - mark `link` target / rel — DocmostAttributes + StarterKit link.
* StarterKit's link extension defaults `target: "_blank"` and
* `rel: "noopener noreferrer nofollow"`; both materialize on import
* (empirically confirmed) even when the source had only `href`.
* - mark `comment` resolved — docmost-schema.ts L213-214 (`default: false`).
* - node `orderedList` start — provided by StarterKit's orderedList
* (`default: 1`); materializes on import (empirically confirmed).
* - node `drawio`/`excalidraw`/`video`/`youtube`/`embed` align — the diagram
* attribute set and the media nodes declare `align: { default: "center" }`
* (docmost-schema.ts L745-750 diagramAttributes; L564 video; L626 youtube;
* L667 embed). The diagram `align` is the one the round-trip materializes
* (docmost-schema.ts L745); the media/embed entries normalize the SAME
* `align` default for consistency. Note: this only normalizes `align` —
* full canonical stability of `embed` is separately limited by the
* converter coercing numeric `width`/`height` to strings, which is outside
* canonicalize's scope.
*
* NOTE: `image` has NO non-null align default — its `align` defaults to `null`
* (docmost-schema.ts L174), so it is already handled by the null-drop rule and
* is intentionally NOT listed here.
*/
const KNOWN_DEFAULTS: Record<string, Record<string, unknown>> = {
// mark types
link: {
target: "_blank",
rel: "noopener noreferrer nofollow",
},
comment: {
resolved: false,
},
// node types
orderedList: {
start: 1,
},
drawio: {
align: "center",
},
excalidraw: {
align: "center",
},
video: {
align: "center",
},
youtube: {
align: "center",
},
embed: {
align: "center",
},
};
/**
* Prune an `attrs` object in place on a fresh copy: drop keys whose value is
* `null` or `undefined` (an absent attribute and an explicit default of `null`
* are semantically equivalent here). Optionally also drop a node-level `id`
* (block ids are regenerated on import, SPEC §11). ALSO drop any attr whose
* value equals the node/mark `type`'s known NON-null schema default
* (`KNOWN_DEFAULTS`), so "attr absent" ≡ "attr at its default value" — without
* this, the import-materialized `link.target`/`comment.resolved`/
* `orderedList.start`/diagram `align` defaults would be a phantom diff. Every
* non-default attribute value is KEPT (level, language, src, href, commentId,
* width, a non-default `start`/`align`, ...).
*
* Returns the pruned attrs object, or `undefined` if nothing meaningful is
* left (so the caller can drop the `attrs` key entirely: `{attrs:{}}` ≡ no
* attrs).
*/
function canonicalizeAttrs(
attrs: Record<string, unknown>,
dropId: boolean,
type: string | undefined,
): Record<string, unknown> | undefined {
const defaults = type ? KNOWN_DEFAULTS[type] : undefined;
const out: Record<string, unknown> = {};
// Stable key order so a JSON.stringify of the canonical form is comparable
// regardless of the input's key order.
for (const key of Object.keys(attrs).sort()) {
// Block ids are regenerated on import; drop them on NODE attrs only.
if (dropId && key === "id") continue;
const value = attrs[key];
// Absent ≡ explicit-default-null/undefined.
if (value === null || value === undefined) continue;
// Absent ≡ explicit known non-null default (e.g. link.target="_blank").
// A non-default value (e.g. orderedList.start=5) does NOT match, so it is
// kept. The `comment` mark's `commentId` is never a default, so it always
// survives (SPEC §3); only its `resolved: false` default is normalized away.
if (defaults && key in defaults && value === defaults[key]) continue;
out[key] = value;
}
return Object.keys(out).length > 0 ? out : undefined;
}
/**
* Return a DEEP COPY of a ProseMirror node tree, canonicalized so that two
* semantically-equal documents compare deep-equal. Rules (applied recursively
* to the node, its `content`, and its `marks`):
*
* 1. Remove node-level `attrs.id` (regenerated on import). Mark attrs are NOT
* touched for `id` (marks carry no block id; only their meaningful attrs).
* 2. In any `attrs` object (node OR mark) drop keys whose value is `null`/
* `undefined` (absent ≡ explicit default null) OR equals that node/mark
* type's known non-null schema default (absent ≡ explicit default).
* Keep every non-default value. The type is passed into the attrs
* normalizer so it can look up `KNOWN_DEFAULTS`.
* 3. If an `attrs` object becomes empty after pruning, drop the `attrs` key.
* 4. Preserve `marks` (including the `comment` mark and its `commentId` — a
* meaningful anchor per SPEC §3; never strip it).
* 5. Preserve `text`, `type`, and `content` order exactly.
* 6. Never mutate the input.
*/
export function canonicalizeContent(node: any): any {
if (Array.isArray(node)) {
return node.map((child) => canonicalizeContent(child));
}
if (node === null || typeof node !== "object") {
// Primitive leaf (string/number/boolean/null): returned as-is.
return node;
}
// A node is a mark when it has a `type` but never carries block `content`
// and lives inside a `marks` array. We cannot tell from the node alone, so
// we distinguish at the recursion site: node `attrs` drop `id`, mark `attrs`
// do not. This is handled by passing a `dropId` flag down for the `attrs`
// key specifically (nodes) vs the `marks[].attrs` path (marks).
const out: Record<string, unknown> = {};
for (const key of Object.keys(node)) {
if (key === "attrs" && node.attrs && typeof node.attrs === "object") {
// Node-level attrs: drop the block id, null/undefined attrs, and any
// attr at this node type's known non-null schema default.
const canon = canonicalizeAttrs(
node.attrs as Record<string, unknown>,
true,
typeof node.type === "string" ? node.type : undefined,
);
if (canon !== undefined) out.attrs = canon;
// else: drop the `attrs` key entirely (rule 3).
} else if (key === "marks" && Array.isArray(node.marks)) {
// Marks: keep them all (incl. comment); canonicalize their attrs but do
// NOT drop `id` (a mark's `id` would be a meaningful attr, not a block
// id). An empty marks array is dropped so `marks:[]` ≡ no marks.
const marks = (node.marks as any[]).map((mark) => canonicalizeMark(mark));
if (marks.length > 0) out.marks = marks;
} else {
out[key] = canonicalizeContent(node[key]);
}
}
return out;
}
/**
* Canonicalize a single mark: keep `type`, prune its `attrs` (null/undefined
* AND known non-null defaults dropped, empty attrs removed) but NEVER drop a
* mark's attribute as a "block id" — marks have no block id, only meaningful
* attrs (href, commentId, color, level, ...). Meaningful NON-default attrs
* survive (the `comment` mark's `commentId` is never a default, so it always
* survives — SPEC §3); only known defaults like `link.target="_blank"`,
* `link.rel="noopener…"` and `comment.resolved=false` are normalized away.
*/
function canonicalizeMark(mark: any): any {
if (mark === null || typeof mark !== "object") return mark;
const out: Record<string, unknown> = {};
for (const key of Object.keys(mark)) {
if (key === "attrs" && mark.attrs && typeof mark.attrs === "object") {
const canon = canonicalizeAttrs(
mark.attrs as Record<string, unknown>,
false,
typeof mark.type === "string" ? mark.type : undefined,
);
if (canon !== undefined) out.attrs = canon;
} else {
out[key] = canonicalizeContent(mark[key]);
}
}
return out;
}
/**
* Deep structural equality of two values that is key-order-insensitive.
* Used to compare canonical forms. (`canonicalizeContent` already emits
* `attrs` in a stable key order, but the top-level node keys preserve input
* order, so we compare structurally rather than by string.)
*/
function deepEqual(a: any, b: any): boolean {
if (a === b) return true;
if (typeof a !== typeof b) return false;
if (a === null || b === null) return a === b;
if (typeof a !== "object") return false;
const aIsArr = Array.isArray(a);
const bIsArr = Array.isArray(b);
if (aIsArr !== bIsArr) return false;
if (aIsArr) {
if (a.length !== b.length) return false;
for (let i = 0; i < a.length; i++) {
if (!deepEqual(a[i], b[i])) return false;
}
return true;
}
const aKeys = Object.keys(a);
const bKeys = Object.keys(b);
if (aKeys.length !== bKeys.length) return false;
for (const k of aKeys) {
if (!Object.prototype.hasOwnProperty.call(b, k)) return false;
if (!deepEqual(a[k], b[k])) return false;
}
return true;
}
/**
* True when two ProseMirror documents are semantically equal: equal after
* canonicalization (block ids stripped, absent-vs-default-null normalized).
*/
export function docsCanonicallyEqual(a: any, b: any): boolean {
return deepEqual(canonicalizeContent(a), canonicalizeContent(b));
}

225
src/roundtrip.ts
View File

@@ -12,23 +12,33 @@
* Usage (live — needs a .env with real Docmost creds):
* node build/roundtrip.js --page <pageId>
*
* Exit code: 0 when the markdown is byte-stable, 1 on any markdown mismatch (so
* it is CI-able). A non-empty document-level divergence (after stripping block
* ids) is reported but does NOT fail the run — the converter reconstructs
* schema default attrs, a KNOWN finding per SPEC §11.
* Usage (corpus — run every *.json fixture in a directory through the cycle):
* node build/roundtrip.js --corpus test/fixtures/corpus
*
* Exit code: 0 only when BOTH the markdown is byte-stable (md1 === md2) AND the
* document is canonically stable (semantically equal after stripping block ids
* and normalizing absent-vs-default attrs, SPEC §11 option (б)); 1 otherwise.
* The prior `indent: null` style divergence disappears under canonicalization,
* so it no longer fails the run.
*/
import { readFile } from "node:fs/promises";
import { readFile, readdir } from "node:fs/promises";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import {
DocmostClient,
convertProseMirrorToMarkdown,
markdownToProseMirror,
canonicalizeContent,
docsCanonicallyEqual,
} from "docmost-client";
import { loadSettings } from "./settings.js";
// Default fixture used when no --fixture/--page is given (offline CI path).
const DEFAULT_FIXTURE = "test/fixtures/sample-doc.json";
// Default corpus directory scanned by --corpus when no path is supplied.
const DEFAULT_CORPUS_DIR = "test/fixtures/corpus";
/**
* Recursively strip every `attrs.id` from a ProseMirror node tree. Block ids
* are regenerated by `markdownToProseMirror` (SPEC §11), so they must be
@@ -59,6 +69,7 @@ export function stripBlockIds(node: any): any {
interface ParsedArgs {
fixture?: string;
page?: string;
corpus?: string;
}
function parseArgs(argv: string[]): ParsedArgs {
@@ -69,6 +80,14 @@ function parseArgs(argv: string[]): ParsedArgs {
args.fixture = argv[++i];
} else if (a === "--page") {
args.page = argv[++i];
} else if (a === "--corpus") {
// The directory is optional; default applies when omitted.
const next = argv[i + 1];
if (next && !next.startsWith("--")) {
args.corpus = argv[++i];
} else {
args.corpus = DEFAULT_CORPUS_DIR;
}
}
}
return args;
@@ -137,8 +156,76 @@ export function firstDivergence(
return null;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
/** Result of running one document through the export -> import -> export cycle. */
interface RoundTripResult {
/** md1 === md2 (the byte-stable property git needs). */
markdownStable: boolean;
/** docsCanonicallyEqual(doc, doc2) (SPEC §11 option (б)). */
canonicalStable: boolean;
md1: string;
md2: string;
/** First canonical divergence (run on the CANONICALIZED docs), or null. */
canonicalDivergence: { path: string; a: any; b: any } | null;
}
/** Run a single document through export -> import -> export and measure both
* stability properties. */
async function runRoundTrip(doc: any): Promise<RoundTripResult> {
// export -> import -> export
const md1 = convertProseMirrorToMarkdown(doc);
const doc2 = await markdownToProseMirror(md1);
const md2 = convertProseMirrorToMarkdown(doc2);
const markdownStable = md1 === md2;
// Semantic comparison of the CANONICALIZED documents: block ids stripped and
// absent-vs-default-null attrs normalized (SPEC §11). This makes the known
// `indent: null` reconstruction noise disappear.
const canonicalStable = docsCanonicallyEqual(doc, doc2);
const canonicalDivergence = canonicalStable
? null
: firstDivergence(canonicalizeContent(doc), canonicalizeContent(doc2));
return { markdownStable, canonicalStable, md1, md2, canonicalDivergence };
}
/** Print the concrete markdown divergence (first differing line). */
function printMarkdownDivergence(md1: string, md2: string): void {
const lines1 = md1.split("\n");
const lines2 = md2.split("\n");
const max = Math.max(lines1.length, lines2.length);
let firstLine = -1;
for (let i = 0; i < max; i++) {
if (lines1[i] !== lines2[i]) {
firstLine = i;
break;
}
}
console.log("");
console.log("--- markdown divergence ---");
console.log(`first differing line: ${firstLine + 1}`);
if (firstLine >= 0) {
console.log(` export #1: ${JSON.stringify(lines1[firstLine])}`);
console.log(` export #2: ${JSON.stringify(lines2[firstLine])}`);
}
console.log(` md1 length: ${md1.length}, md2 length: ${md2.length}`);
}
/** Print the first canonical document divergence. */
function printCanonicalDivergence(d: {
path: string;
a: any;
b: any;
}): void {
console.log("");
console.log("--- document divergence (canonicalized) ---");
console.log(` path: ${d.path}`);
console.log(` doc: ${JSON.stringify(d.a)}`);
console.log(` doc2: ${JSON.stringify(d.b)}`);
}
/** Single-document mode (--fixture / --page / default fixture). */
async function runSingle(args: ParsedArgs): Promise<number> {
const doc = await loadDoc(args);
const source = args.page
@@ -146,65 +233,85 @@ async function main(): Promise<void> {
: `fixture ${args.fixture ?? DEFAULT_FIXTURE}`;
console.log(`Round-trip harness — source: ${source}`);
// export -> import -> export
const md1 = convertProseMirrorToMarkdown(doc);
const doc2 = await markdownToProseMirror(md1);
const md2 = convertProseMirrorToMarkdown(doc2);
// 1) The byte-stable markdown property git actually needs.
const markdownStable = md1 === md2;
// 2) Semantic comparison of the documents with block ids stripped (they are
// regenerated on import, per SPEC §11).
const normDoc = stripBlockIds(doc);
const normDoc2 = stripBlockIds(doc2);
const docDivergence = firstDivergence(normDoc, normDoc2);
const semanticStable = docDivergence === null;
const r = await runRoundTrip(doc);
console.log("");
console.log(`markdown byte-stable (md1 === md2): ${markdownStable}`);
console.log(`document semantically stable (ids stripped): ${semanticStable}`);
console.log(`markdown byte-stable (md1 === md2): ${r.markdownStable}`);
console.log(`document canonically stable (SPEC §11): ${r.canonicalStable}`);
if (!markdownStable) {
// Show the first differing line so the divergence is concrete.
const lines1 = md1.split("\n");
const lines2 = md2.split("\n");
const max = Math.max(lines1.length, lines2.length);
let firstLine = -1;
for (let i = 0; i < max; i++) {
if (lines1[i] !== lines2[i]) {
firstLine = i;
break;
}
}
console.log("");
console.log("--- markdown divergence ---");
console.log(`first differing line: ${firstLine + 1}`);
if (firstLine >= 0) {
console.log(` export #1: ${JSON.stringify(lines1[firstLine])}`);
console.log(` export #2: ${JSON.stringify(lines2[firstLine])}`);
}
console.log(` md1 length: ${md1.length}, md2 length: ${md2.length}`);
if (!r.markdownStable) printMarkdownDivergence(r.md1, r.md2);
if (!r.canonicalStable && r.canonicalDivergence)
printCanonicalDivergence(r.canonicalDivergence);
// Exit 0 only if BOTH properties hold.
const ok = r.markdownStable && r.canonicalStable;
console.log("");
console.log(ok ? "RESULT: STABLE" : "RESULT: NOT STABLE");
return ok ? 0 : 1;
}
/** Corpus mode (--corpus <dir>): run every *.json file through the cycle. */
async function runCorpus(dir: string): Promise<number> {
console.log(`Round-trip harness — corpus: ${dir}`);
console.log("");
const entries = (await readdir(dir))
.filter((name) => name.endsWith(".json"))
.sort();
if (entries.length === 0) {
console.log(`No *.json fixtures found in ${dir}`);
return 1;
}
if (!semanticStable && docDivergence) {
console.log("");
console.log("--- document divergence (ids stripped) ---");
console.log(` path: ${docDivergence.path}`);
console.log(` doc: ${JSON.stringify(docDivergence.a)}`);
console.log(` doc2: ${JSON.stringify(docDivergence.b)}`);
console.log(
" (EXPECTED per SPEC §11: the converter reconstructs schema default" +
" attrs; does not affect markdown byte-stability)",
let anyFailed = false;
for (const name of entries) {
const path = join(dir, name);
const doc = JSON.parse(await readFile(path, "utf8"));
const r = await runRoundTrip(doc);
const pass = r.markdownStable && r.canonicalStable;
if (!pass) anyFailed = true;
const flags = `md=${r.markdownStable ? "ok" : "FAIL"} canon=${
r.canonicalStable ? "ok" : "FAIL"
}`;
console.log(`${pass ? "PASS" : "FAIL"} ${name} (${flags})`);
if (!pass) {
if (!r.markdownStable) printMarkdownDivergence(r.md1, r.md2);
if (!r.canonicalStable && r.canonicalDivergence)
printCanonicalDivergence(r.canonicalDivergence);
}
}
console.log("");
console.log(
anyFailed ? "RESULT: CORPUS HAS FAILURES" : "RESULT: CORPUS STABLE",
);
return anyFailed ? 1 : 0;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
// The modes are mutually exclusive. If --corpus is combined with a
// single-document flag, --corpus takes precedence and the others are ignored;
// warn clearly so the caller is not surprised by silently dropped flags.
if (args.corpus && (args.fixture || args.page)) {
const ignored = [
args.fixture ? "--fixture" : null,
args.page ? "--page" : null,
]
.filter(Boolean)
.join(", ");
console.warn(
`WARNING: --corpus is mutually exclusive with ${ignored}; ` +
`running --corpus and ignoring ${ignored}.`,
);
}
// The CI-relevant invariant is markdown byte-stability. A doc-level
// divergence after id-stripping is a KNOWN SPEC §11 finding and does not
// fail the harness.
console.log("");
console.log(markdownStable ? "RESULT: MARKDOWN STABLE" : "RESULT: NOT STABLE");
process.exit(markdownStable ? 0 : 1);
const code = args.corpus
? await runCorpus(args.corpus)
: await runSingle(args);
process.exit(code);
}
// Only auto-run when invoked directly as the CLI entrypoint, not when this

302
test/canonicalize.test.ts Normal file
View File

@@ -0,0 +1,302 @@
import { describe, expect, it } from 'vitest';
// Import via the package barrel to also assert the symbols are re-exported.
import { canonicalizeContent, docsCanonicallyEqual } from 'docmost-client';
describe('canonicalizeContent', () => {
it('strips node-level attrs.id, recursively', () => {
const input = {
type: 'doc',
content: [
{
type: 'heading',
attrs: { id: 'h-1', level: 2 },
content: [{ type: 'text', text: 'Title' }],
},
],
};
const out = canonicalizeContent(input);
expect(out.content[0].attrs).toEqual({ level: 2 });
// No `id` survives anywhere in the canonical tree.
expect(JSON.stringify(out)).not.toContain('"id"');
});
it('drops null/undefined attrs but keeps every non-null attr', () => {
const out = canonicalizeContent({
type: 'paragraph',
attrs: {
id: 'p-1',
indent: null,
textAlign: undefined,
level: 0,
keep: 'yes',
},
content: [],
});
// null/undefined gone; non-null values (incl. 0 and false) kept.
expect(out.attrs).toEqual({ keep: 'yes', level: 0 });
});
it('removes an attrs object that becomes empty after pruning', () => {
const out = canonicalizeContent({
type: 'paragraph',
attrs: { id: 'p-1', indent: null, textAlign: null },
content: [{ type: 'text', text: 'x' }],
});
// attrs had only an id + null defaults -> the whole attrs key is dropped.
expect('attrs' in out).toBe(false);
expect(out).toEqual({
type: 'paragraph',
content: [{ type: 'text', text: 'x' }],
});
});
it('treats {attrs:{}} as equivalent to no attrs', () => {
const withEmpty = canonicalizeContent({ type: 'paragraph', attrs: {} });
const without = canonicalizeContent({ type: 'paragraph' });
expect(withEmpty).toEqual(without);
});
it('keeps comment marks + commentId but normalizes resolved:false default (SPEC §3 anchor)', () => {
const out = canonicalizeContent({
type: 'text',
text: 'anchored',
marks: [
{ type: 'comment', attrs: { commentId: 'cmt-1', resolved: false } },
],
});
// The comment mark is preserved; commentId (a meaningful anchor) survives,
// but the `resolved: false` schema default is normalized away.
expect(out.marks).toEqual([
{ type: 'comment', attrs: { commentId: 'cmt-1' } },
]);
});
it('drops known non-null schema defaults (link target/rel, comment resolved)', () => {
const out = canonicalizeContent({
type: 'text',
text: 'a link',
marks: [
{
type: 'link',
attrs: {
href: 'https://example.com/page',
target: '_blank',
rel: 'noopener noreferrer nofollow',
},
},
],
});
// href (non-default) kept; target/rel (schema defaults) dropped.
expect(out.marks).toEqual([
{ type: 'link', attrs: { href: 'https://example.com/page' } },
]);
});
it('keeps a NON-default value that happens to share an attr name (orderedList start:5)', () => {
const out = canonicalizeContent({
type: 'orderedList',
attrs: { id: 'ol-1', start: 5 },
content: [],
});
// start:5 is NOT the default (1), so it must survive.
expect(out.attrs).toEqual({ start: 5 });
});
it('keeps meaningful node/mark attrs (level, language, href, src, width)', () => {
const out = canonicalizeContent({
type: 'doc',
content: [
{
type: 'codeBlock',
attrs: { id: 'c-1', language: 'js' },
content: [{ type: 'text', text: 'x' }],
},
{
type: 'image',
attrs: { id: 'i-1', src: '/a.png', width: 100, height: null },
},
{
type: 'paragraph',
content: [
{
type: 'text',
text: 'link',
marks: [{ type: 'link', attrs: { href: 'https://e.com' } }],
},
],
},
],
});
expect(out.content[0].attrs).toEqual({ language: 'js' });
expect(out.content[1].attrs).toEqual({ src: '/a.png', width: 100 });
expect(out.content[2].content[0].marks[0].attrs).toEqual({
href: 'https://e.com',
});
});
it('preserves text, type and content order exactly', () => {
const input = {
type: 'paragraph',
content: [
{ type: 'text', text: 'one' },
{ type: 'text', text: 'two', marks: [{ type: 'bold' }] },
{ type: 'text', text: 'three' },
],
};
const out = canonicalizeContent(input);
expect(out.content.map((n: any) => n.text)).toEqual([
'one',
'two',
'three',
]);
expect(out.content[1].marks).toEqual([{ type: 'bold' }]);
});
it('drops an empty marks array (marks:[] === no marks)', () => {
const out = canonicalizeContent({ type: 'text', text: 'x', marks: [] });
expect('marks' in out).toBe(false);
});
it('does not mutate its input (frozen tree passes through unchanged)', () => {
const input = Object.freeze({
type: 'doc',
content: Object.freeze([
Object.freeze({
type: 'paragraph',
attrs: Object.freeze({ id: 'p-1', indent: null }),
content: Object.freeze([Object.freeze({ type: 'text', text: 'x' })]),
}),
]),
});
const before = JSON.stringify(input);
const out = canonicalizeContent(input);
// Input is structurally identical after the call.
expect(JSON.stringify(input)).toBe(before);
// A fresh tree is returned.
expect(out).not.toBe(input);
expect('attrs' in out.content[0]).toBe(false);
});
});
describe('docsCanonicallyEqual', () => {
it('is true when docs differ only by block ids', () => {
const a = {
type: 'doc',
content: [
{ type: 'heading', attrs: { id: 'h-1', level: 1 }, content: [] },
],
};
const b = {
type: 'doc',
content: [
{ type: 'heading', attrs: { id: 'h-DIFFERENT', level: 1 }, content: [] },
],
};
expect(docsCanonicallyEqual(a, b)).toBe(true);
});
it('is true when one side omits an attr the other sets to default null', () => {
const a = {
type: 'paragraph',
attrs: { id: 'p-1' },
content: [{ type: 'text', text: 'x' }],
};
const b = {
type: 'paragraph',
attrs: { id: 'p-2', indent: null, textAlign: null },
content: [{ type: 'text', text: 'x' }],
};
expect(docsCanonicallyEqual(a, b)).toBe(true);
});
it('is key-order-insensitive for attrs', () => {
const a = { type: 'image', attrs: { src: '/a.png', width: 10 } };
const b = { type: 'image', attrs: { width: 10, src: '/a.png' } };
expect(docsCanonicallyEqual(a, b)).toBe(true);
});
it('is false for a real text difference', () => {
const a = { type: 'text', text: 'hello' };
const b = { type: 'text', text: 'world' };
expect(docsCanonicallyEqual(a, b)).toBe(false);
});
it('is false for a real attr difference (different level)', () => {
const a = { type: 'heading', attrs: { id: 'x', level: 1 } };
const b = { type: 'heading', attrs: { id: 'y', level: 2 } };
expect(docsCanonicallyEqual(a, b)).toBe(false);
});
it('is false when a meaningful mark attr differs (commentId)', () => {
const a = {
type: 'text',
text: 'x',
marks: [{ type: 'comment', attrs: { commentId: 'cmt-1' } }],
};
const b = {
type: 'text',
text: 'x',
marks: [{ type: 'comment', attrs: { commentId: 'cmt-2' } }],
};
expect(docsCanonicallyEqual(a, b)).toBe(false);
});
it('is true when a link has only href vs one with the schema-default target/rel', () => {
const a = {
type: 'text',
text: 'link',
marks: [{ type: 'link', attrs: { href: 'https://example.com' } }],
};
const b = {
type: 'text',
text: 'link',
marks: [
{
type: 'link',
attrs: {
href: 'https://example.com',
target: '_blank',
rel: 'noopener noreferrer nofollow',
},
},
],
};
expect(docsCanonicallyEqual(a, b)).toBe(true);
});
it('is true when an orderedList omits start vs one with the default start:1', () => {
const a = { type: 'orderedList', content: [] };
const b = { type: 'orderedList', attrs: { start: 1 }, content: [] };
expect(docsCanonicallyEqual(a, b)).toBe(true);
});
it('is false when an orderedList has a non-default start (5 vs absent)', () => {
const a = { type: 'orderedList', content: [] };
const b = { type: 'orderedList', attrs: { start: 5 }, content: [] };
expect(docsCanonicallyEqual(a, b)).toBe(false);
});
it('is true when a comment mark omits resolved vs one with the default false', () => {
const a = {
type: 'text',
text: 'x',
marks: [{ type: 'comment', attrs: { commentId: 'cmt-1' } }],
};
const b = {
type: 'text',
text: 'x',
marks: [{ type: 'comment', attrs: { commentId: 'cmt-1', resolved: false } }],
};
expect(docsCanonicallyEqual(a, b)).toBe(true);
});
it('is false when a comment mark is dropped entirely', () => {
const a = {
type: 'text',
text: 'x',
marks: [{ type: 'comment', attrs: { commentId: 'cmt-1' } }],
};
const b = { type: 'text', text: 'x' };
expect(docsCanonicallyEqual(a, b)).toBe(false);
});
});

36
test/fixtures/corpus/01-headings-paragraphs.json vendored Normal file
View File

@@ -0,0 +1,36 @@
{
"type": "doc",
"content": [
{
"type": "heading",
"attrs": { "level": 1 },
"content": [{ "type": "text", "text": "Level one heading" }]
},
{
"type": "paragraph",
"content": [{ "type": "text", "text": "A plain paragraph of text." }]
},
{
"type": "heading",
"attrs": { "level": 2 },
"content": [{ "type": "text", "text": "Level two heading" }]
},
{
"type": "paragraph",
"content": [
{ "type": "text", "text": "First line of a paragraph" },
{ "type": "hardBreak" },
{ "type": "text", "text": "second line after a hard break." }
]
},
{
"type": "heading",
"attrs": { "level": 3 },
"content": [{ "type": "text", "text": "Level three heading" }]
},
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Closing paragraph." }]
}
]
}

62
test/fixtures/corpus/02-inline-marks.json vendored Normal file
View File

@@ -0,0 +1,62 @@
{
"type": "doc",
"content": [
{
"type": "paragraph",
"content": [
{ "type": "text", "marks": [{ "type": "bold" }], "text": "bold" },
{ "type": "text", "text": " " },
{ "type": "text", "marks": [{ "type": "italic" }], "text": "italic" },
{ "type": "text", "text": " " },
{ "type": "text", "marks": [{ "type": "code" }], "text": "code" },
{ "type": "text", "text": " " },
{ "type": "text", "marks": [{ "type": "strike" }], "text": "strike" }
]
},
{
"type": "paragraph",
"content": [
{
"type": "text",
"marks": [
{
"type": "link",
"attrs": {
"href": "https://example.com/page"
}
}
],
"text": "a link"
},
{ "type": "text", "text": ", " },
{
"type": "text",
"marks": [{ "type": "highlight" }],
"text": "highlighted"
},
{ "type": "text", "text": ", base" },
{ "type": "text", "marks": [{ "type": "subscript" }], "text": "sub" },
{ "type": "text", "text": " and base" },
{ "type": "text", "marks": [{ "type": "superscript" }], "text": "sup" },
{ "type": "text", "text": "." }
]
},
{
"type": "paragraph",
"content": [
{ "type": "text", "text": "Here is a " },
{
"type": "text",
"marks": [
{
"type": "comment",
"attrs": { "commentId": "cmt-xyz789" }
}
],
"text": "commented anchor span"
},
{ "type": "text", "text": " that must survive (SPEC §3)." }
]
}
]
}

113
test/fixtures/corpus/03-lists.json vendored Normal file
View File

@@ -0,0 +1,113 @@
{
"type": "doc",
"content": [
{
"type": "bulletList",
"content": [
{
"type": "listItem",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "First bullet" }]
}
]
},
{
"type": "listItem",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Second bullet with a nested list" }]
},
{
"type": "bulletList",
"content": [
{
"type": "listItem",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Nested bullet A" }]
}
]
},
{
"type": "listItem",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Nested bullet B" }]
}
]
}
]
}
]
}
]
},
{
"type": "orderedList",
"content": [
{
"type": "listItem",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "First ordered item" }]
}
]
},
{
"type": "listItem",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Second ordered item" }]
},
{
"type": "orderedList",
"content": [
{
"type": "listItem",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Nested ordered one" }]
}
]
}
]
}
]
}
]
},
{
"type": "taskList",
"content": [
{
"type": "taskItem",
"attrs": { "checked": true },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Done task" }]
}
]
},
{
"type": "taskItem",
"attrs": { "checked": false },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Pending task" }]
}
]
}
]
}
]
}

38
test/fixtures/corpus/04-blocks.json vendored Normal file
View File

@@ -0,0 +1,38 @@
{
"type": "doc",
"content": [
{
"type": "blockquote",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "A quoted line." }]
},
{
"type": "paragraph",
"content": [{ "type": "text", "text": "A second quoted paragraph." }]
}
]
},
{
"type": "horizontalRule"
},
{
"type": "codeBlock",
"attrs": { "language": "js" },
"content": [
{ "type": "text", "text": "const a = 1;\nconsole.log(a);\n" }
]
},
{
"type": "callout",
"attrs": { "type": "warning" },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "This is a warning callout." }]
}
]
}
]
}

85
test/fixtures/corpus/05-table.json vendored Normal file
View File

@@ -0,0 +1,85 @@
{
"type": "doc",
"content": [
{
"type": "table",
"content": [
{
"type": "tableRow",
"content": [
{
"type": "tableHeader",
"attrs": { "colspan": 1, "rowspan": 1 },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Name" }]
}
]
},
{
"type": "tableHeader",
"attrs": { "colspan": 1, "rowspan": 1 },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "Value" }]
}
]
}
]
},
{
"type": "tableRow",
"content": [
{
"type": "tableCell",
"attrs": { "colspan": 1, "rowspan": 1 },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "alpha" }]
}
]
},
{
"type": "tableCell",
"attrs": { "colspan": 1, "rowspan": 1 },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "1" }]
}
]
}
]
},
{
"type": "tableRow",
"content": [
{
"type": "tableCell",
"attrs": { "colspan": 1, "rowspan": 1 },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "beta" }]
}
]
},
{
"type": "tableCell",
"attrs": { "colspan": 1, "rowspan": 1 },
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "2" }]
}
]
}
]
}
]
}
]
}

17
test/fixtures/corpus/06-diagrams.json vendored Normal file
View File

@@ -0,0 +1,17 @@
{
"type": "doc",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "A drawio and an excalidraw diagram follow." }]
},
{
"type": "drawio",
"attrs": { "src": "/api/files/def/flow.drawio", "align": "center", "attachmentId": "att-1" }
},
{
"type": "excalidraw",
"attrs": { "src": "/api/files/ghi/sketch.excalidraw", "align": "center", "attachmentId": "att-2" }
}
]
}

35
test/fixtures/corpus/07-textstyle-mention.json vendored Normal file
View File

@@ -0,0 +1,35 @@
{
"type": "doc",
"content": [
{
"type": "paragraph",
"content": [
{ "type": "text", "text": "Some " },
{
"type": "text",
"marks": [{ "type": "textStyle", "attrs": { "color": "#ff0000" } }],
"text": "red colored"
},
{ "type": "text", "text": " text." }
]
},
{
"type": "paragraph",
"content": [
{ "type": "text", "text": "Ping " },
{
"type": "mention",
"attrs": {
"id": "m-1",
"label": "Alice",
"entityType": "user",
"entityId": "u-1",
"slugId": "s-1",
"creatorId": "c-1"
}
},
{ "type": "text", "text": " please." }
]
}
]
}

21
test/fixtures/known-limitations/image-diagrams.json vendored Normal file
View File

@@ -0,0 +1,21 @@
{
"type": "doc",
"content": [
{
"type": "paragraph",
"content": [{ "type": "text", "text": "An image followed by two diagrams." }]
},
{
"type": "image",
"attrs": { "src": "/api/files/abc/diagram.png", "alt": "A picture" }
},
{
"type": "drawio",
"attrs": { "src": "/api/files/def/flow.drawio", "attachmentId": "att-1" }
},
{
"type": "excalidraw",
"attrs": { "src": "/api/files/ghi/sketch.excalidraw", "attachmentId": "att-2" }
}
]
}

9
test/fixtures/sample-doc.json vendored
View File

@@ -17,7 +17,14 @@
{ "type": "text", "text": " and a " },
{
"type": "text",
"marks": [{ "type": "link", "attrs": { "href": "https://example.com" } }],
"marks": [
{
"type": "link",
"attrs": {
"href": "https://example.com"
}
}
],
"text": "link"
},
{ "type": "text", "text": "." }

104
test/roundtrip-corpus.test.ts Normal file
View File

@@ -0,0 +1,104 @@
import { readFile } from 'node:fs/promises';
import { readdirSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { describe, expect, it } from 'vitest';
import {
convertProseMirrorToMarkdown,
markdownToProseMirror,
docsCanonicallyEqual,
} from 'docmost-client';
// Resolve fixtures relative to this test file so the test is CWD-independent.
const here = dirname(fileURLToPath(import.meta.url));
const CORPUS_DIR = join(here, 'fixtures', 'corpus');
const KNOWN_LIMITATIONS_DIR = join(here, 'fixtures', 'known-limitations');
/** Run a single document through export -> import -> export. */
async function roundTrip(doc: any) {
const md1 = convertProseMirrorToMarkdown(doc);
const doc2 = await markdownToProseMirror(md1);
const md2 = convertProseMirrorToMarkdown(doc2);
return { md1, md2, doc2 };
}
describe('round-trip corpus (SPEC §11)', () => {
// Discover the corpus synchronously at collection time so each fixture gets
// its own `it` with the file name in the test title.
const files = readdirSync(CORPUS_DIR)
.filter((name) => name.endsWith('.json'))
.sort();
it('has a non-empty corpus', () => {
expect(files.length).toBeGreaterThan(0);
});
for (const name of files) {
it(`${name}: markdown byte-stable AND canonically stable`, async () => {
const doc = JSON.parse(await readFile(join(CORPUS_DIR, name), 'utf8'));
const { md1, md2, doc2 } = await roundTrip(doc);
// 1) The byte-stable markdown property git actually needs.
expect(md2, `${name}: markdown not byte-stable`).toBe(md1);
// 2) Semantic stability (block ids stripped, default-null normalized).
expect(
docsCanonicallyEqual(doc, doc2),
`${name}: document not canonically stable`,
).toBe(true);
});
}
});
// ---------------------------------------------------------------------------
// KNOWN CONVERTER LIMITATIONS (isolated so they do NOT make CI red).
//
// SPEC §11 explicitly flags images and diagrams as high round-trip risk. These
// fixtures are kept OUT of the green corpus above and asserted with `it.fails`
// so the documented divergence is locked in (the test FAILS if the converter
// ever starts round-tripping them — at which point promote the fixture into
// the corpus). The precise divergences for `image-diagrams.json` are:
//
// * A BLOCK-LEVEL image preceded by a paragraph is NOT byte-stable on the
// FIRST re-export. The HTML re-parser hoists the block <img> out of its
// line and leaves an empty paragraph behind, so `paragraph` + `![..](..)`
// re-imports as paragraph + empty-paragraph + image; the empty paragraph
// adds one blank line, so export #2 grows by a one-time "\n\n" (md1 !== md2).
// This is NOT non-convergence: the growth happens exactly ONCE. The doc
// CONVERGES to a fixpoint after one extra `export→import→export` pass — the
// empty paragraph is already present after the first import, so export #2
// and export #3 are byte-identical (md2 === md3, verified).
//
// * drawio / excalidraw diagrams gain `data-align="center"` on the second
// export: the schema's diagram `align` attribute has a NON-null default of
// "center", which materializes on import; the converter only emits
// data-align when set, so it appears on export #2 but not #1. Like the
// image case, this is one-time and converges after one extra pass.
//
// * A STANDALONE block image (no preceding paragraph) IS byte-stable from
// export #1 (md1 === md2) — but it is still NOT canonically stable: on
// import the bare <img> is wrapped, gaining a leading EMPTY paragraph, so
// the canonical doc differs by that spurious paragraph node even though the
// markdown bytes match.
//
// Resolution (SPEC §11, "normalize-on-write"): rather than deep-fixing the
// converter, the engine runs ONE `export→import→export` pass when writing into
// the vault; from that fixpoint onward the form is byte-stable, so git sees no
// phantom diff. The green corpus above avoids these one-time asymmetries by
// pre-authoring the materialized defaults (e.g. `align: "center"` on the
// diagrams in 06-diagrams.json) so a single pass is already at the fixpoint.
// ---------------------------------------------------------------------------
describe('round-trip KNOWN LIMITATIONS (SPEC §11 image/diagram risk)', () => {
it.fails(
'image-diagrams.json is NOT byte-stable on export #1 (block image hoist + diagram align default; converges after one extra pass — SPEC §11 normalize-on-write)',
async () => {
const doc = JSON.parse(
await readFile(join(KNOWN_LIMITATIONS_DIR, 'image-diagrams.json'), 'utf8'),
);
const { md1, md2 } = await roundTrip(doc);
// This assertion FAILS today (documented divergence). `it.fails` turns a
// failing body into a PASS; if the converter is fixed this flips and the
// test goes red, prompting promotion into the green corpus.
expect(md2).toBe(md1);
},
);
});