Compare commits

...

7 Commits

Author SHA1 Message Date
vvzvlad da7bb95d4f Merge pull request 'test(int): убрать избыточный forceExit — bounded teardown уже расхангивает test:int (#382)' (#383) from fix/382-int-open-handles into develop
Reviewed-on: #383
2026-07-06 16:08:22 +03:00
vvzvlad 6ab2e989b9 Merge pull request 'test(converter): вложенный variant-A генератор + P4 + фикс всех round-trip багов (#351)' (#385) from test/351-nested-generator into develop
Reviewed-on: #385
2026-07-06 16:07:48 +03:00
vvzvlad 169e34d766 Merge pull request 'docs(agents): build shared packages before a consumer's tsc/tests in isolation' (#384) from docs/agents-workspace-build-order into develop
Reviewed-on: #384
2026-07-06 16:06:34 +03:00
agent_coder ccee32cb0b test(git-sync): assert stable drawio markers, not the omittable center default (#351 review)
stabilize.test.ts used `data-align="center"` as proof the convergence pass
materialized the drawio node. Since this PR correctly stops emitting the
schema-default center align in the media builders, that marker is no longer a
reliable convergence proof. Assert on the stable canonical markers instead —
`data-type="drawio"` + `data-src="/d.drawio"` — which are always materialized
regardless of the align default. The fixpoint assertion (file2 === file1) is
unchanged; round-trip stays byte-stable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 06:44:58 +03:00
agent_coder 79a461f79d test(converter): вложенный variant-A генератор + P4-фазз, и фикс всех найденных round-trip багов (#351)
Финальная ступень #351: генератор ЦЕЛЫХ вложенных документов (random walk
по ContentMatch схемы, min-depth fixpoint для терминирования, глубина/размер
ограничены) + инвариант P4 (фазз парсера: для ЛЮБОЙ строки markdownToProseMirror
не бросает и результат валиден по схеме). Инварианты P1 (семантический
round-trip), P2 (байтовый fixpoint со 2-го прохода), P3 (тотальность) — строгие.

Генератор сразу нашёл классы реальных багов конвертера; все починены (инварианты
НЕ ослаблялись — чинился конвертер):

- loose (много-блочные) контейнеры (listItem/taskItem/callout/detailsContent)
  склеивали блоки при реимпорте — ТИХАЯ ПОТЕРЯ ДАННЫХ; теперь blank-line
  разделитель между блок-детьми (по образцу blockquote).
- соседние sibling-списки одного marker-family (task+bullet и т.п.) сливались
  в один список с ПОТЕРЕЙ чекбокса — теперь между ними эмитится инертный
  `<!-- -->` разделитель (byte-stable round-trip).
- paragraph textAlign терялся во вложенных li/td/th.
- вложенный codeBlock терял хвостовой перевод строки.
- pageBreak/pageEmbed/subpages/transclusion дропались во вложении
  (blockquote/callout/details/li).
- медиа в columns: number→string ширины/высоты и лишний data-align (churn).
- callout `> [!type]`, вложенный в список/цитату, парсился неверно
  (prefix-aware regex).

Golden-обновления (6) — прямые следствия loose-container фикса, каждое
round-trip'ится. 2100+ сгенерированных документов (3 seed) — 0 падений P1/P2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 06:29:58 +03:00
claude_code 24946ad820 docs(agents): build shared packages before a consumer's tsc/tests in isolation
Document the TS2307 trap: the gitignored build/ of @docmost/prosemirror-markdown,
@docmost/git-sync and @docmost/mcp is not honoured by a single-package
pnpm --filter <pkg> test/tsc or a bare pnpm -r test (Nx dependsOn ^build is only
applied by nx run-many), so a consumer's typecheck fails with
Cannot find module '@docmost/...' until those packages are built first.
Mirrors the order .github/workflows/test.yml already uses.
2026-07-06 05:50:34 +03:00
agent_coder 84334a1f34 test(int): drop redundant forceExit — bounded teardown already un-hangs test:int (#382)
The int suite could not self-exit after the ESM fix (8e125799) unmasked 4
specs, and was patched with two things: forceExit:true AND a bounded
destroyTestDb (sql.end({ timeout: 5 })). The bounded teardown is the real
fix — postgres.js .end() without a timeout blocks indefinitely on a stuck
pooled connection (the CI-observed "Jest did not exit"); the { timeout: 5 }
grace drains then force-closes sockets so teardown always completes.

forceExit was redundant belt-and-suspenders that also HID whether the
process truly exits on its own. Removing it: every handle-creating spec is
verified to close its handle — ai-chat-stream closes its http.createServer
in a finally, public-share-workspace-limiter closes its ioredis via
redis.quit() in afterAll, and the shared DB pools close via the bounded
destroyTestDb. --detectOpenHandles is clean and the suite self-exits.

Kept: the bounded destroyTestDb (defense for a genuinely stuck connection).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 04:07:44 +03:00
10 changed files with 876 additions and 96 deletions
+18
View File
@@ -230,6 +230,24 @@ pnpm build # nx run-many -t build (all packages)
pnpm collab:dev # run the collaboration server process standalone (see "Two server processes") pnpm collab:dev # run the collaboration server process standalone (see "Two server processes")
``` ```
> **Build the shared packages before running a consumer's `tsc`/tests in
> isolation.** The `build/` dirs of `@docmost/prosemirror-markdown`,
> `@docmost/git-sync`, and `@docmost/mcp` are **gitignored** (not committed), and
> a single-package `pnpm --filter <pkg> test` / `tsc` or a bare `pnpm -r test`
> does **NOT** honour the Nx `dependsOn: ["^build"]` ordering. So a consumer — the
> server's `tsc`, `git-sync`'s vitest typecheck, `mcp`'s `pretest: tsc` — fails
> with `error TS2307: Cannot find module '@docmost/…'` until those packages are
> built first:
> ```bash
> pnpm --filter @docmost/prosemirror-markdown build
> pnpm --filter @docmost/editor-ext build
> pnpm --filter @docmost/git-sync build && pnpm --filter @docmost/mcp build
> ```
> `pnpm build` (nx run-many) does this for you; CI does it explicitly in
> `.github/workflows/test.yml` (prosemirror-markdown → git-sync/mcp → server, in
> that order). Reach for it whenever you run a consumer package's checks on their
> own rather than through the full `pnpm build`.
**Lint** (per package — there is no root lint script): **Lint** (per package — there is no root lint script):
```bash ```bash
pnpm --filter server lint # eslint --fix on server .ts pnpm --filter server lint # eslint --fix on server .ts
-1
View File
@@ -16,7 +16,6 @@
"testEnvironment": "node", "testEnvironment": "node",
"testTimeout": 60000, "testTimeout": 60000,
"maxWorkers": 1, "maxWorkers": 1,
"forceExit": true,
"globalSetup": "<rootDir>/test/integration/global-setup.ts", "globalSetup": "<rootDir>/test/integration/global-setup.ts",
"globalTeardown": "<rootDir>/test/integration/global-teardown.ts", "globalTeardown": "<rootDir>/test/integration/global-teardown.ts",
"moduleNameMapper": { "moduleNameMapper": {
+9 -4
View File
@@ -55,10 +55,15 @@ describe('stabilizePageFile — normalize-on-write fixpoint (SPEC §11)', () =>
const file2 = await stabilizePageFile(doc2, meta); const file2 = await stabilizePageFile(doc2, meta);
expect(file2).toBe(file1); expect(file2).toBe(file1);
// The materialized diagram default is present in the stabilized body (proof // The drawio node was materialized to its canonical HTML form by the
// that the convergence pass actually ran, not just that two naive exports // convergence pass — a bare `{ src }` doc node becomes the full
// happened to match). // `<div data-type="drawio" data-src=...>` — proof the pass actually ran, not
expect(body1).toContain('data-align="center"'); // just two naive exports happening to match. Assert on the stable canonical
// markers rather than `data-align="center"`: center is a schema default the
// converter may omit (see prosemirror-markdown media-html.ts), so it is not
// a reliable convergence proof.
expect(body1).toContain('data-type="drawio"');
expect(body1).toContain('data-src="/d.drawio"');
}); });
it('already-stable content is unchanged by the pass (idempotent)', async () => { it('already-stable content is unchanged by the pass (idempotent)', async () => {
@@ -48,6 +48,52 @@ export interface ConvertProseMirrorToMarkdownOptions {
dropResolvedCommentAnchors?: boolean; dropResolvedCommentAnchors?: boolean;
} }
/**
* Adjacent sibling lists that share a markdown MARKER FAMILY re-parse as ONE
* merged list — bulletList and taskList both emit `- ` markers (→ a single
* `<ul>`), and two orderedLists both emit `1.` markers (→ a single `<ol>`). The
* cross-type case is real data loss the editor CAN produce (e.g. a taskList
* followed by a bulletList: the merged `<ul>` has a mix of checkbox and plain
* items, so `bridgeTaskLists` refuses to convert it and every taskItem loses its
* checkbox). Between two such adjacent list children we emit an empty HTML comment
* `<!-- -->`: marked renders it as its own HTML block that interrupts the list, so
* the two lists stay distinct; on import the comment is inert (parseAttachedComment
* → null) and dropped by generateJSON, and re-export re-inserts it, so the marker
* is byte-stable. It fires ONLY between two adjacent same-family list nodes — no
* separator is emitted for any other join, so non-list output is unchanged.
*/
const LIST_MARKER_SEPARATOR = "<!-- -->";
function listMarkerFamily(type: string | undefined): "ul" | "ol" | null {
if (type === "bulletList" || type === "taskList") return "ul";
if (type === "orderedList") return "ol";
return null;
}
function adjacentListsMerge(
prevType: string | undefined,
curType: string | undefined,
): boolean {
const a = listMarkerFamily(prevType);
return a !== null && a === listMarkerFamily(curType);
}
/**
* Render each block child, inserting a `<!-- -->` separator entry between any two
* adjacent same-marker-family list nodes (see LIST_MARKER_SEPARATOR). Callers
* join the returned strings with their own context separator.
*/
function renderBlockChildren(
children: any[],
render: (n: any) => string,
): string[] {
const out: string[] = [];
let prevType: string | undefined;
for (const child of children) {
if (adjacentListsMerge(prevType, child?.type)) out.push(LIST_MARKER_SEPARATOR);
out.push(render(child));
prevType = child?.type;
}
return out;
}
/** /**
* Convert ProseMirror/TipTap JSON content to Markdown * Convert ProseMirror/TipTap JSON content to Markdown
* Supports all Docmost-specific node types and extensions * Supports all Docmost-specific node types and extensions
@@ -347,9 +393,15 @@ export function convertProseMirrorToMarkdown(
// lossless (the body survives) and byte-stable (it re-exports identically), // lossless (the body survives) and byte-stable (it re-exports identically),
// so it is deliberately not treated as data loss. // so it is deliberately not treated as data loss.
const parts: string[] = []; const parts: string[] = [];
let prevDocType: string | undefined;
for (const child of nodeContent) { for (const child of nodeContent) {
if (child?.type === "footnotesList") continue; if (child?.type === "footnotesList") continue;
// Keep adjacent same-family sibling lists distinct (see renderBlockChildren).
if (adjacentListsMerge(prevDocType, child?.type)) {
parts.push(LIST_MARKER_SEPARATOR);
}
parts.push(processNode(child)); parts.push(processNode(child));
prevDocType = child?.type;
} }
for (const [id, def] of footnoteDefs) { for (const [id, def] of footnoteDefs) {
if (!referencedFootnoteIds.has(id)) { if (!referencedFootnoteIds.has(id)) {
@@ -599,20 +651,34 @@ export function convertProseMirrorToMarkdown(
return processTaskItem(node); return processTaskItem(node);
case "listItem": case "listItem":
return nodeContent.map(processNode).join("\n"); // Direct-listItem path (lists normally render via processListItem, which
// handles the marker + indentation). Blank line between block children so
// multiple paragraphs do not merge on re-parse; a `<!-- -->` entry
// (renderBlockChildren) separates adjacent sibling lists.
return renderBlockChildren(nodeContent, processNode).join("\n\n");
case "blockquote": case "blockquote": {
// Prefix EVERY line of EVERY child with "> " and separate block-level // Prefix EVERY line of EVERY child with "> " and separate block-level
// children with a blank ">" line so code blocks / multi-paragraph // children with a blank ">" line so code blocks / multi-paragraph
// quotes round-trip correctly. // quotes round-trip correctly. A `> <!-- -->` separator line is inserted
return nodeContent // between two adjacent same-family sibling lists (renderBlockChildren)
.map((n: any) => // so they stay distinct inside the quote.
const bqParts: string[] = [];
let prevBqType: string | undefined;
for (const n of nodeContent) {
if (adjacentListsMerge(prevBqType, n?.type)) {
bqParts.push(`> ${LIST_MARKER_SEPARATOR}`);
}
bqParts.push(
processNode(n) processNode(n)
.split("\n") .split("\n")
.map((line: string) => (line.length ? `> ${line}` : ">")) .map((line: string) => (line.length ? `> ${line}` : ">"))
.join("\n"), .join("\n"),
) );
.join("\n>\n"); prevBqType = n?.type;
}
return bqParts.join("\n>\n");
}
case "horizontalRule": case "horizontalRule":
return "---"; return "---";
@@ -787,9 +853,12 @@ export function convertProseMirrorToMarkdown(
// blockquote-prefixed; a blank line becomes a bare `>` so the callout is // blockquote-prefixed; a blank line becomes a bare `>` so the callout is
// not split. // not split.
const calloutType = (node.attrs?.type || "info").toLowerCase(); const calloutType = (node.attrs?.type || "info").toLowerCase();
const calloutBody = nodeContent const calloutBody = renderBlockChildren(nodeContent, processNode)
.map(processNode) // Blank line between block children (rendered as a bare `>` after the
.join("\n") // prefix pass below) so multiple paragraphs stay separate nodes instead
// of merging on re-parse — same rule blockquote already uses. A
// `<!-- -->` entry (renderBlockChildren) separates adjacent sibling lists.
.join("\n\n")
.split("\n") .split("\n")
.map((l: string) => (l.length ? `> ${l}` : ">")) .map((l: string) => (l.length ? `> ${l}` : ">"))
.join("\n"); .join("\n");
@@ -809,7 +878,10 @@ export function convertProseMirrorToMarkdown(
return `<summary>${renderInlineChildren(nodeContent)}</summary>\n\n`; return `<summary>${renderInlineChildren(nodeContent)}</summary>\n\n`;
case "detailsContent": case "detailsContent":
return `${nodeContent.map(processNode).join("\n")}\n`; // Blank line between block children so multiple paragraphs in a details
// body survive as separate nodes (a single "\n" merges them on re-parse);
// a `<!-- -->` entry (renderBlockChildren) separates adjacent sibling lists.
return `${renderBlockChildren(nodeContent, processNode).join("\n\n")}\n`;
case "mathInline": { case "mathInline": {
// #293 canon #6: inline math serializes as Obsidian-native `$LaTeX$` // #293 canon #6: inline math serializes as Obsidian-native `$LaTeX$`
@@ -1338,11 +1410,19 @@ export function convertProseMirrorToMarkdown(
// The code itself is element TEXT content (between <code> tags), so it // The code itself is element TEXT content (between <code> tags), so it
// must escape < > & — NOT the attribute escaper. The language rides in // must escape < > & — NOT the attribute escaper. The language rides in
// a class ATTRIBUTE, so it uses escapeAttr. // a class ATTRIBUTE, so it uses escapeAttr.
//
// Read the child text RAW (as `case "codeBlock"` does) and keep it
// VERBATIM — do NOT strip the trailing newline. Unlike the markdown fence
// path (which strips then relies on marked re-adding one `\n`), the schema
// codeBlock parseHTML reads the `<code>` text content back byte-for-byte,
// so stripping here would drop a trailing newline the node legitimately
// carries and break the round trip inside a column/cell.
const code = escapeHtmlText( const code = escapeHtmlText(
children children
.map(processNode) .map((child: any) =>
.join("") typeof child?.text === "string" ? child.text : "",
.replace(/\n+$/, ""), )
.join(""),
); );
const cls = lang ? ` class="language-${escapeAttr(lang)}"` : ""; const cls = lang ? ` class="language-${escapeAttr(lang)}"` : "";
return `<pre><code${cls}>${code}</code></pre>`; return `<pre><code${cls}>${code}</code></pre>`;
@@ -1484,6 +1564,13 @@ export function convertProseMirrorToMarkdown(
const indent = " ".repeat(indentWidth); const indent = " ".repeat(indentWidth);
const lines: string[] = []; const lines: string[] = [];
childStrings.forEach((child, childIndex) => { childStrings.forEach((child, childIndex) => {
// Separate consecutive block children with a BLANK line so the item is a
// CommonMark "loose" list item and each block stays its own node. Without
// it, a second paragraph (`- a\n b`) is re-parsed as a lazy continuation
// of the first and the two merge into one paragraph — silent data loss.
// The blank line still sits INSIDE the item (the following block keeps the
// continuation indent), so nested lists/code blocks remain nested.
if (childIndex > 0) lines.push("");
child.split("\n").forEach((line, lineIndex) => { child.split("\n").forEach((line, lineIndex) => {
if (childIndex === 0 && lineIndex === 0) { if (childIndex === 0 && lineIndex === 0) {
// First physical line of the first block gets the marker. // First physical line of the first block gets the marker.
@@ -1500,7 +1587,9 @@ export function convertProseMirrorToMarkdown(
const processListItem = (item: any, prefix: string): string => { const processListItem = (item: any, prefix: string): string => {
const itemContent = item.content || []; const itemContent = item.content || [];
const childStrings = itemContent.map(processNode); // A `<!-- -->` entry separates two adjacent same-family sublists inside the
// item so they do not merge on re-parse (see renderBlockChildren).
const childStrings = renderBlockChildren(itemContent, processNode);
if (childStrings.length === 0) return prefix; if (childStrings.length === 0) return prefix;
// The rendered marker is `${prefix} ` (prefix + one space), so its width — // The rendered marker is `${prefix} ` (prefix + one space), so its width —
// and thus the continuation indent — is prefix.length + 1. This is correct // and thus the continuation indent — is prefix.length + 1. This is correct
@@ -1514,7 +1603,9 @@ export function convertProseMirrorToMarkdown(
const checkbox = checked ? "[x]" : "[ ]"; const checkbox = checked ? "[x]" : "[ ]";
const prefix = `- ${checkbox}`; const prefix = `- ${checkbox}`;
const itemContent = item.content || []; const itemContent = item.content || [];
const childStrings = itemContent.map(processNode); // A `<!-- -->` entry separates two adjacent same-family sublists inside the
// item so they do not merge on re-parse (see renderBlockChildren).
const childStrings = renderBlockChildren(itemContent, processNode);
// An empty task item still needs its checkbox marker; without this guard // An empty task item still needs its checkbox marker; without this guard
// the indent below produces "" and the "- [ ]"/"- [x]" row disappears. // the indent below produces "" and the "- [ ]"/"- [x]" row disappears.
if (childStrings.length === 0) return prefix; if (childStrings.length === 0) return prefix;
@@ -270,9 +270,10 @@ const CALLOUT_CLOSE_RE = /^:::\s*$/;
* optional title after the type is allowed but ignored (the Docmost callout * optional title after the type is allowed but ignored (the Docmost callout
* schema has no title). The body is the following contiguous blockquote lines. * schema has no title). The body is the following contiguous blockquote lines.
*/ */
const CALLOUT_BQ_OPEN_RE = /^>\s*\[!(\w+)\]/; // The callout's own `>` marker may be preceded by an ENCLOSING container prefix:
/** Matches any blockquote continuation line (`>` … ). */ // list-item indentation (` `) and/or blockquote markers (`> `). Group 1 captures
const BLOCKQUOTE_LINE_RE = /^>/; // that prefix (lazily, so the LAST `>` before `[!type]` is the callout's own).
const CALLOUT_BQ_OPEN_RE = /^([>\s]*?)>\s*\[!(\w+)\]/;
/** Matches the start/end of a code fence (``` or ~~~), capturing the marker. */ /** Matches the start/end of a code fence (``` or ~~~), capturing the marker. */
const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/; const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
@@ -402,20 +403,47 @@ async function preprocessCallouts(markdown: string): Promise<string> {
// recurse so nested callouts (`> > [!type]`) are handled, then emit the same // recurse so nested callouts (`> > [!type]`) are handled, then emit the same
// callout div the `:::` path produces. A normal blockquote (no `[!type]` on // callout div the `:::` path produces. A normal blockquote (no `[!type]` on
// its first line) does not match and stays a blockquote. // its first line) does not match and stays a blockquote.
//
// PREFIX-aware: a callout nested inside a list item and/or a blockquote is
// serialized with the enclosing container prefix in front of its own `>`
// marker — ` > [!type]` (list indent) or `> > [!type]` (blockquote). We
// capture that prefix, take only continuation lines carrying `prefix>`, strip
// it, and re-apply the prefix to the emitted HTML block so the callout div
// stays WITHIN its container (an unprefixed div would escape and re-parse as
// a top-level callout / plain blockquote — silent structure loss).
const bqOpen = line.match(CALLOUT_BQ_OPEN_RE); const bqOpen = line.match(CALLOUT_BQ_OPEN_RE);
if (bqOpen) { if (bqOpen) {
const type = bqOpen[1].toLowerCase(); const prefix = bqOpen[1];
const type = bqOpen[2].toLowerCase();
const cont = prefix + ">"; // a body line = prefix + the callout's own `>`
const bodyLines: string[] = []; const bodyLines: string[] = [];
let j = i + 1; let j = i + 1;
for (; j < lines.length; j++) { for (; j < lines.length; j++) {
if (!BLOCKQUOTE_LINE_RE.test(lines[j])) break; if (!lines[j].startsWith(cont)) break;
bodyLines.push(lines[j].replace(/^>\s?/, "")); // Drop the prefix + `>` + one optional space, leaving the body content.
bodyLines.push(lines[j].slice(prefix.length).replace(/^>\s?/, ""));
} }
const inner = await transform(bodyLines); const inner = await transform(bodyLines);
const renderedInner = await markedInstance.parse(inner); const renderedInner = await markedInstance.parse(inner);
out.push( const block = `<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>`;
`\n<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>\n`, if (prefix.length === 0) {
); // Top-level callout: blank lines isolate the HTML block.
out.push(`\n${block}\n`);
} else if (prefix.includes(">")) {
// Enclosing BLOCKQUOTE: prefix every line and add NO surrounding blank
// lines — a blank line would terminate the blockquote and split the
// callout out of it.
out.push(block.split("\n").map((l) => prefix + l).join("\n"));
} else {
// Pure LIST-ITEM indentation: re-indent and keep the blank-line
// separators (a loose list item), so the div sits at the marker column.
out.push(
`\n${block
.split("\n")
.map((l) => (l.length ? prefix + l : l))
.join("\n")}\n`,
);
}
i = j; i = j;
continue; continue;
} }
@@ -597,6 +625,40 @@ function bridgeTaskLists(html: string): string {
* (null from parseAttachedComment), an unknown name, a wrong-position comment, or * (null from parseAttachedComment), an unknown name, a wrong-position comment, or
* an unknown/empty attr value is ignored. * an unknown/empty attr value is ignored.
*/ */
/**
* A directive comment is in ATTACHED position when it sits inside a `<p>`/`<hN>`
* textblock — bound to that block's text (the `attrs`/`img` conventions). Every
* other parent (body, document level, a block container like blockquote/details/
* li/column div) is STANDALONE position, where a lone-block directive
* (subpages/pagebreak/pageembed/transclusion) is materialized. Broadening
* standalone beyond body/document is what lets these nodes survive NESTED inside
* a blockquote/callout/details/list item (previously dropped -> silent data loss).
*/
function isAttachedPosition(tag: string): boolean {
return tag === "p" || /^h[1-6]$/.test(tag);
}
/**
* Place a materialized standalone-directive element in the DOM: replace the
* comment IN PLACE when it has a real element parent inside <body> (body itself
* or a nested block container), preserving document order; queue it as a leading
* div only when the comment is at document level (no parentElement) or directly
* under `<html>` (outside <body>, which `document.body.innerHTML` would drop).
*/
function placeStandalone(
comment: any,
el: any,
tag: string,
leadingDivs: any[],
): void {
if (comment.parentElement && tag !== "html") {
comment.replaceWith(el);
} else {
comment.remove();
leadingDivs.push(el);
}
}
function applyCommentDirectives(html: string): string { function applyCommentDirectives(html: string): string {
// Cheap early-out: no comments at all -> nothing to intercept. // Cheap early-out: no comments at all -> nothing to intercept.
if (!html.includes("<!--")) return html; if (!html.includes("<!--")) return html;
@@ -664,13 +726,13 @@ function applyCommentDirectives(html: string): string {
if (parsed.name === "subpages" || parsed.name === "pagebreak") { if (parsed.name === "subpages" || parsed.name === "pagebreak") {
// #293 canon #5 STANDALONE machinery. A lone comment line is rendered by // #293 canon #5 STANDALONE machinery. A lone comment line is rendered by
// marked as an HTML block; the parser places it either directly under // marked as its own HTML block; the parser places it under <body>, at
// <body> (when other content surrounds it) or at document level (when it // document level (leading), or — when the directive is NESTED — inside a
// leads the output). Both are STANDALONE position. A `subpages`/`pagebreak` // block CONTAINER (`<blockquote>` for blockquote/callout, `<details>`,
// comment sitting inside a `<p>`/`<hN>` (or any other element) is attached // `<li>`, a column `<div>`, …). All of those are STANDALONE position. Only a
// position -> INERT. // comment ATTACHED inside a `<p>`/`<hN>` (bound to that block's text) is
const standalone = tag === "" || tag === "body" || tag === "html"; // attached position -> INERT.
if (!standalone) continue; // wrong position -> inert if (isAttachedPosition(tag)) continue; // wrong position -> inert
const div = document.createElement("div"); const div = document.createElement("div");
if (parsed.name === "pagebreak") { if (parsed.name === "pagebreak") {
div.setAttribute("data-type", "pageBreak"); div.setAttribute("data-type", "pageBreak");
@@ -680,26 +742,18 @@ function applyCommentDirectives(html: string): string {
div.setAttribute("data-recursive", "true"); div.setAttribute("data-recursive", "true");
} }
} }
if (tag === "body") { placeStandalone(comment, div, tag, leadingDivs);
// In-body: replace in place so surrounding content keeps its order.
comment.replaceWith(div);
} else {
// Document-level (leading): drop the stray comment and queue the div to
// be prepended into body below.
comment.remove();
leadingDivs.push(div);
}
continue; continue;
} }
if (parsed.name === "pageembed" || parsed.name === "transclusion") { if (parsed.name === "pageembed" || parsed.name === "transclusion") {
// #293 canon #8 STANDALONE media. Like subpages/pagebreak: a lone comment // #293 canon #8 STANDALONE media. Like subpages/pagebreak: a lone comment
// line placed under <body> or at document level (leading). An attached- // line placed under <body>, at document level (leading), or NESTED inside a
// position comment (inside a <p>/<hN> with a sibling) is INERT. We rebuild // block container (blockquote/callout/details/li/column). An ATTACHED-
// the schema div the raw-HTML path emits (media-html.ts) from the decoded // position comment (inside a `<p>`/`<hN>`) is INERT. We rebuild the schema
// attrs so serialize/parse stay in sync. // div the raw-HTML path emits (media-html.ts) from the decoded attrs so
const standalone = tag === "" || tag === "body" || tag === "html"; // serialize/parse stay in sync.
if (!standalone) continue; // wrong position -> inert if (isAttachedPosition(tag)) continue; // wrong position -> inert
const el = buildElement( const el = buildElement(
parsed.name === "pageembed" parsed.name === "pageembed"
? pageEmbedToHtml({ sourcePageId: parsed.attrs.sourcePageId }) ? pageEmbedToHtml({ sourcePageId: parsed.attrs.sourcePageId })
@@ -709,12 +763,7 @@ function applyCommentDirectives(html: string): string {
}), }),
); );
if (!el) continue; // defensive: builder always yields an element if (!el) continue; // defensive: builder always yields an element
if (tag === "body") { placeStandalone(comment, el, tag, leadingDivs);
comment.replaceWith(el);
} else {
comment.remove();
leadingDivs.push(el);
}
continue; continue;
} }
@@ -806,18 +855,36 @@ function applyCommentDirectives(html: string): string {
if (!parent) continue; // attrs comment must have an element parent if (!parent) continue; // attrs comment must have an element parent
if (parsed.name !== "attrs") continue; // unknown name -> inert if (parsed.name !== "attrs") continue; // unknown name -> inert
// #293 canon #9 ATTACHED attrs: honored only in attached position.
const isBlock = tag === "p" || /^h[1-6]$/.test(tag);
if (!isBlock) continue; // misplaced comment -> inert
const align = parsed.attrs.textAlign; const align = parsed.attrs.textAlign;
if (typeof align === "string" && align) { // #293 canon #9 ATTACHED attrs: honored only in attached position.
// Re-express as an inline style; the schema's textAlign parseHTML reads if (tag === "p" || /^h[1-6]$/.test(tag)) {
// `el.style.textAlign` back onto the paragraph/heading node. // A real <p>/<hN> host (loose list item, top-level block, …): re-express as
parent.style.textAlign = align; // an inline style; the schema's textAlign parseHTML reads `el.style.textAlign`
// back onto the paragraph/heading node.
if (typeof align === "string" && align) parent.style.textAlign = align;
comment.remove();
} else if (tag === "li" || tag === "td" || tag === "th") {
// TIGHT list item / GFM table cell: marked emits the paragraph's inline
// content DIRECTLY inside the <li>/<td>/<th> with NO <p> wrapper, so there
// is no element to carry the style — generateJSON materializes the
// paragraph later. Wrap the host's LEADING inline content (everything up to
// the comment; any trailing block child such as a nested list stays put) in
// a <p> carrying the alignment, so the materialized paragraph re-reads it.
if (typeof align === "string" && align) {
const p = document.createElement("p");
p.style.textAlign = align;
while (parent.firstChild && parent.firstChild !== comment) {
p.appendChild(parent.firstChild);
}
parent.insertBefore(p, comment);
}
comment.remove();
} else {
// Misplaced `attrs` comment (not a textblock/li/cell host): inert. Consume
// it anyway so no attached marker ever survives into the parsed body
// (matches the pre-existing "consume regardless" behaviour).
comment.remove();
} }
// Consume the marker regardless (unknown keys are simply ignored) so no
// attached comment ever survives into the parsed body.
comment.remove();
} }
// Prepend any document-level (leading) standalone divs into body, preserving // Prepend any document-level (leading) standalone divs into body, preserving
// their document order relative to each other and ahead of existing content. // their document order relative to each other and ahead of existing content.
@@ -46,7 +46,11 @@ export function videoToHtml(attrs: Record<string, any>): string {
if (attrs.width != null) parts.push(`width="${escapeAttr(attrs.width)}"`); if (attrs.width != null) parts.push(`width="${escapeAttr(attrs.width)}"`);
if (attrs.height != null) parts.push(`height="${escapeAttr(attrs.height)}"`); if (attrs.height != null) parts.push(`height="${escapeAttr(attrs.height)}"`);
if (attrs.size != null) parts.push(`data-size="${escapeAttr(attrs.size)}"`); if (attrs.size != null) parts.push(`data-size="${escapeAttr(attrs.size)}"`);
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`); // align default is "center" (schema): OMIT it so a bare/center node stays
// clean and parse's re-materialized "center" default is not a P2 churn — only
// a genuinely non-default left/right emits data-align (mirrors imageToHtml).
if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
if (attrs.aspectRatio != null) if (attrs.aspectRatio != null)
parts.push(`data-aspect-ratio="${escapeAttr(attrs.aspectRatio)}"`); parts.push(`data-aspect-ratio="${escapeAttr(attrs.aspectRatio)}"`);
return `<div><video ${parts.join(" ")}></video></div>`; return `<div><video ${parts.join(" ")}></video></div>`;
@@ -62,7 +66,9 @@ export function youtubeToHtml(attrs: Record<string, any>): string {
parts.push(`data-width="${escapeAttr(attrs.width)}"`); parts.push(`data-width="${escapeAttr(attrs.width)}"`);
if (attrs.height != null) if (attrs.height != null)
parts.push(`data-height="${escapeAttr(attrs.height)}"`); parts.push(`data-height="${escapeAttr(attrs.height)}"`);
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`); // "center" is the schema default -> omit (see videoToHtml rationale).
if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
return `<div ${parts.join(" ")}></div>`; return `<div ${parts.join(" ")}></div>`;
} }
@@ -97,7 +103,9 @@ export function diagramToHtml(
if (attrs.size != null) parts.push(`data-size="${escapeAttr(attrs.size)}"`); if (attrs.size != null) parts.push(`data-size="${escapeAttr(attrs.size)}"`);
if (attrs.aspectRatio != null) if (attrs.aspectRatio != null)
parts.push(`data-aspect-ratio="${escapeAttr(attrs.aspectRatio)}"`); parts.push(`data-aspect-ratio="${escapeAttr(attrs.aspectRatio)}"`);
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`); // "center" is the schema default -> omit (see videoToHtml rationale).
if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
if (attrs.attachmentId) if (attrs.attachmentId)
parts.push(`data-attachment-id="${escapeAttr(attrs.attachmentId)}"`); parts.push(`data-attachment-id="${escapeAttr(attrs.attachmentId)}"`);
return `<div ${parts.join(" ")}></div>`; return `<div ${parts.join(" ")}></div>`;
@@ -110,10 +118,18 @@ export function embedToHtml(attrs: Record<string, any>): string {
`data-src="${escapeAttr(attrs.src ?? "")}"`, `data-src="${escapeAttr(attrs.src ?? "")}"`,
`data-provider="${escapeAttr(attrs.provider ?? "")}"`, `data-provider="${escapeAttr(attrs.provider ?? "")}"`,
]; ];
if (attrs.align) parts.push(`data-align="${escapeAttr(attrs.align)}"`); // "center" is the schema default -> omit (see videoToHtml rationale).
if (attrs.width != null) if (attrs.align && attrs.align !== "center")
parts.push(`data-align="${escapeAttr(attrs.align)}"`);
// embed width/height default to the NUMBERS 800/600 (schema). Getting a data-
// attribute back always yields a STRING, so emitting the default here would
// round-trip 800 -> "800" (a number->string P1 divergence canonicalize does
// NOT normalize). OMIT the defaults so parse re-materializes the numeric
// default instead — mirrors the top-level embed path (markdown-converter.ts),
// which also emits width/height only when they differ from 800/600.
if (attrs.width != null && attrs.width !== 800)
parts.push(`data-width="${escapeAttr(attrs.width)}"`); parts.push(`data-width="${escapeAttr(attrs.width)}"`);
if (attrs.height != null) if (attrs.height != null && attrs.height !== 600)
parts.push(`data-height="${escapeAttr(attrs.height)}"`); parts.push(`data-height="${escapeAttr(attrs.height)}"`);
return `<div ${parts.join(" ")}></div>`; return `<div ${parts.join(" ")}></div>`;
} }
@@ -0,0 +1,392 @@
/**
* Nested whole-document generator (#351, PR 2) — "variant A": a random walk over
* the schema's ContentMatch automaton.
*
* Where the FLAT generator (node-generators.ts) emits `{doc:[ <one target> ]}`,
* this module produces arbitrarily DEEP, valid ProseMirror documents. Validity is
* NOT hand-asserted: it comes straight from the schema. To fill a container's
* block content we start at `nodeType.contentMatch` and walk the automaton —
* enumerate the legal next node types (`match.edge(i).type`), let fast-check pick
* one (or STOP once `match.validEnd`), generate that child RECURSIVELY, then
* advance the automaton via `match.matchType(childType)`. A bad walk therefore
* cannot emit a structurally-invalid doc (the `generator validity` test guards
* this with `schema.nodeFromJSON(json).check()`).
*
* ── What the walk drives, and what it delegates ──────────────────────────────
* The walk owns BLOCK STRUCTURE (which blocks nest inside which containers, in
* what order and how deep). Two things it deliberately delegates, to avoid
* REGRESSING the byte-stable space the flat suite already proved empirically:
*
* - INLINE content of textblocks (paragraph/heading/codeBlock/detailsSummary)
* is filled from text-arbitraries.ts — the exact hostile-but-byte-stable
* inline corpus the flat suite established. Walking the inline ContentMatch
* instead would re-derive (and re-fail) the text-space limitations the flat
* suite already pins, which is not this generator's job.
* - Node ATTRS come from nodeAttrsArb(type, 'p1') — the round-trip-safe
* attribute space. Attribute-degenerate fuzzing (P2/P3 over 'fuzz') is the
* flat suite's concern; the nested suite isolates STRUCTURAL round-trip, so
* it stays in 'p1' and does not re-litigate frozen/pinned attributes.
*
* ── Coordination the automaton cannot express ────────────────────────────────
* A ContentMatch guarantees a child SEQUENCE is legal, but not cross-sibling
* invariants. Two nodes need coordination the walk injects by hand:
* - `table`: GFM needs a RECTANGULAR grid with column-consistent alignment.
* The automaton happily allows ragged rows / per-cell align, which are not a
* converter bug — just a malformed table. So a table is generated ATOMICALLY
* (same shape the flat suite proved) rather than walked.
* - `columns`: the `layout` attr must agree with the column COUNT. We pick the
* layout, derive the count, then walk each column's block body normally — so
* columns still gain real nested content, only the count is coordinated.
*
* ── Excluded from the nested walk ────────────────────────────────────────────
* Footnote nodes (footnoteReference / footnotesList / footnoteDefinition) need a
* DOCUMENT-GLOBAL id match between a reference and its definition. That
* coordination is owned by the flat suite's `footnotes` generator; placing them
* independently here would fabricate id mismatches that look like converter bugs
* but are generator defects. They are filtered out of the walk (documented in
* EXCLUDED). The completeness contract lives in the FLAT suite and is unaffected.
*
* ── Termination / budgets ────────────────────────────────────────────────────
* Two bounds keep every doc finite and the suite fast:
* - MAX_DEPTH — a hard cap on block-nesting depth. A precomputed `minDepth`
* fixpoint (the minimum extra nesting a subtree of each type needs to be
* valid) lets the walk pick a CONTAINER child only when there is depth
* headroom to complete it — so the walk can never paint itself into a corner
* where a required child cannot fit (no invalid docs, guaranteed termination).
* - NODE_BUDGET — a soft cap on total nodes; as it runs low the walk biases
* toward STOP (when validEnd) or toward cheap terminating children.
*/
import fc from 'fast-check';
import { getSchema } from '@tiptap/core';
import { docmostExtensions } from '../../src/lib/docmost-schema.js';
import { nodeAttrsArb } from './attr-arbitraries.js';
import {
inlineContentArb,
headingInlineContentArb,
plainInlineContentArb,
phraseArb,
} from './text-arbitraries.js';
/** The exact ProseMirror schema the converter targets (built per the issue). */
export const schema = getSchema(docmostExtensions as never);
/** Hard cap on block-nesting depth (doc = depth 0). Kept in the issue's 4–5 band. */
export const MAX_DEPTH = 4;
/**
* Soft cap on total node count per generated document. Kept moderate: every P1/P2
* run parses the emitted markdown through jsdom (heavy), so 100+ node docs across
* hundreds of runs exhaust the worker heap. 60 still yields deeply-nested docs
* (depth 4) while keeping the suite within memory.
*/
export const NODE_BUDGET = 60;
/**
* Nodes kept OUT of the nested walk: footnote nodes need a doc-global id match a
* local walk cannot coordinate (owned by the flat suite's `footnotes` generator).
*/
const EXCLUDED = new Set<string>([
'footnoteReference',
'footnotesList',
'footnoteDefinition',
]);
/**
* Structural-only children that carry `group: "block"` in the schema and so leak
* into EVERY block container's ContentMatch, even though the editor only ever
* places them inside their one true parent. Choosing them freely (e.g. a bare
* `column` at the document root) fabricates documents no editor produces and that
* the converter is not designed to round-trip — a GENERATOR artifact, not a
* converter bug. They are admitted ONLY when the container being filled is their
* dedicated parent. (`column` is in fact always built inside columnsArb, so this
* just double-guards it.)
*/
const DEDICATED_PARENT: Record<string, string> = {
column: 'columns',
detailsSummary: 'details',
detailsContent: 'details',
};
/** Is child type `t` legal as a freely-chosen child of container `parentType`? */
function childAllowedUnder(t: string, parentType: string): boolean {
const dedicated = DEDICATED_PARENT[t];
return dedicated === undefined || dedicated === parentType;
}
/** Textblock (inlineContent) types — filled from the proven inline corpus. */
function isTextblock(typeName: string): boolean {
return !!schema.nodes[typeName]?.isTextblock;
}
/** Leaf/atom types — no content, only generated attrs. */
function isLeaf(typeName: string): boolean {
return !!schema.nodes[typeName]?.isLeaf;
}
// ---------------------------------------------------------------------------
// minDepth fixpoint: the minimum EXTRA block-nesting depth a valid subtree
// rooted at each node type requires. Leaves and textblocks need 0 (a textblock
// is satisfied by inline content, no block recursion). A container needs
// 1 + the cheapest way to satisfy its ContentMatch. Computed as a min–max path
// to `validEnd` over the automaton, iterated to a fixpoint over node types.
// ---------------------------------------------------------------------------
/** Cheapest (min over reachable validEnd of max child minDepth) to complete a match. */
function minCompletion(
match: any,
md: Record<string, number>,
seen: Set<any>,
parentType: string,
): number {
let best = match.validEnd ? 0 : Infinity;
if (seen.has(match)) return best; // a cycle never completes more cheaply
seen.add(match);
for (let i = 0; i < match.edgeCount; i++) {
const edge = match.edge(i);
const t = edge.type.name;
if (EXCLUDED.has(t) || t === 'text') continue;
if (!childAllowedUnder(t, parentType)) continue;
const childCost = md[t];
if (childCost === undefined || childCost === Infinity) continue;
const rest = minCompletion(edge.next, md, seen, parentType);
if (rest === Infinity) continue;
best = Math.min(best, Math.max(childCost, rest));
}
seen.delete(match);
return best;
}
function computeMinDepth(): Record<string, number> {
const md: Record<string, number> = {};
for (const name of Object.keys(schema.nodes)) {
md[name] = isLeaf(name) || isTextblock(name) ? 0 : Infinity;
}
let changed = true;
while (changed) {
changed = false;
for (const name of Object.keys(schema.nodes)) {
if (md[name] === 0) continue; // leaves/textblocks fixed at 0
const nt: any = schema.nodes[name];
const completion = minCompletion(nt.contentMatch, md, new Set(), name);
const next = completion === Infinity ? Infinity : 1 + completion;
if (next < md[name]) {
md[name] = next;
changed = true;
}
}
}
return md;
}
const MIN_DEPTH = computeMinDepth();
// ---------------------------------------------------------------------------
// Leaf / textblock builders (attrs from 'p1', inline from the proven corpus).
// ---------------------------------------------------------------------------
function attachAttrs(typeName: string, base: Record<string, unknown> = {}) {
return nodeAttrsArb(typeName, 'p1', base).map((attrs) => {
const node: any = { type: typeName };
if (Object.keys(attrs).length) node.attrs = attrs;
return node;
});
}
/** A leaf/atom block: attrs only, no content. */
function leafArb(typeName: string): fc.Arbitrary<any> {
return attachAttrs(typeName);
}
/** A textblock, inline content taken from the byte-stable flat corpus. */
function textblockArb(typeName: string): fc.Arbitrary<any> {
if (typeName === 'codeBlock') {
return fc
.tuple(
nodeAttrsArb('codeBlock', 'p1'),
// Fenced code re-imports with a TRAILING NEWLINE (flat suite finding);
// author it so the doc is already at the round-trip fixpoint.
fc.array(phraseArb, { minLength: 1, maxLength: 3 }).map((l) => l.join('\n') + '\n'),
)
.map(([attrs, code]) => ({
type: 'codeBlock',
...(Object.keys(attrs).length ? { attrs } : {}),
content: [{ type: 'text', text: code }],
}));
}
const inline =
typeName === 'heading'
? headingInlineContentArb
: typeName === 'detailsSummary'
? plainInlineContentArb
: inlineContentArb;
return fc
.tuple(nodeAttrsArb(typeName, 'p1'), inline)
.map(([attrs, content]) => ({
type: typeName,
...(Object.keys(attrs).length ? { attrs } : {}),
content,
}));
}
// ---------------------------------------------------------------------------
// Coordinated builders: table (atomic, rectangular, column-consistent align)
// and columns (layout coupled to count, bodies walked).
// ---------------------------------------------------------------------------
/** A rectangular GFM-safe table (mirrors the flat suite's proven shape). */
function tableArb(): fc.Arbitrary<any> {
return fc.integer({ min: 1, max: 3 }).chain((cols) => {
// One alignment per COLUMN, identical on header + every body cell, so the
// second export cannot re-align and churn.
const alignsArb = fc.array(fc.constantFrom(undefined, 'left', 'center', 'right'), {
minLength: cols,
maxLength: cols,
});
const cell = (header: boolean, align?: string) =>
phraseArb.map((t) => ({
type: header ? 'tableHeader' : 'tableCell',
attrs: { colspan: 1, rowspan: 1, ...(align ? { align } : {}) },
content: [{ type: 'paragraph', content: [{ type: 'text', text: t }] }],
}));
return alignsArb.chain((aligns) => {
const headerRow = fc
.tuple(...aligns.map((a) => cell(true, a)))
.map((cells) => ({ type: 'tableRow', content: cells }));
const bodyRow = fc
.tuple(...aligns.map((a) => cell(false, a)))
.map((cells) => ({ type: 'tableRow', content: cells }));
return fc
.tuple(headerRow, fc.array(bodyRow, { minLength: 1, maxLength: 2 }))
.map(([h, body]) => ({ type: 'table', content: [h, ...body] }));
});
});
}
/** A columns block: layout ↔ count coupled, each column body walked as blocks. */
function columnsArb(depth: number, budget: number): fc.Arbitrary<any> {
return fc
.constantFrom('two_equal', 'three_equal', 'left_sidebar', 'right_sidebar')
.chain((layout) => {
const count = layout === 'three_equal' ? 3 : 2;
const columnType: any = schema.nodes.column;
// Split the remaining budget across the fixed number of columns.
const per = Math.max(2, Math.floor((budget - 1) / count));
return nodeAttrsArb('columns', 'p1', { layout, widthMode: 'normal' }).chain((attrs) =>
fc
.tuple(
...Array.from({ length: count }, () =>
fillMatch(columnType.contentMatch, depth + 1, per, 'column').map(({ children }) => ({
type: 'column',
content: children,
})),
),
)
.map((cols) => ({ type: 'columns', attrs, content: cols })),
);
});
}
// ---------------------------------------------------------------------------
// The ContentMatch walk.
// ---------------------------------------------------------------------------
/** Count every node in a subtree (block + inline), for budget accounting. */
function countNodes(node: any): number {
let n = 1;
for (const c of node.content ?? []) n += countNodes(c);
return n;
}
/** Build a single child node of a given type at `depth`, within `budget`. */
function blockNode(typeName: string, depth: number, budget: number): fc.Arbitrary<any> {
if (typeName === 'table') return tableArb();
if (typeName === 'columns') return columnsArb(depth, budget);
if (isTextblock(typeName)) return textblockArb(typeName);
if (isLeaf(typeName)) return leafArb(typeName);
// Generic container: attrs from 'p1', block content from the automaton walk.
const nt: any = schema.nodes[typeName];
return nodeAttrsArb(typeName, 'p1').chain((attrs) =>
fillMatch(nt.contentMatch, depth, budget - 1, typeName).map(({ children }) => {
const node: any = { type: typeName };
if (Object.keys(attrs).length) node.attrs = attrs;
if (children.length) node.content = children;
return node;
}),
);
}
/**
* Fill a container's block content by walking its ContentMatch automaton from
* `match`. Returns the children array plus the budget left after them.
*/
function fillMatch(
match: any,
depth: number,
budget: number,
parentType: string,
): fc.Arbitrary<{ children: any[]; budget: number }> {
const canStop = match.validEnd;
// A child lives at depth+1; only pick it if its subtree can complete within
// MAX_DEPTH. This headroom rule is what makes the walk deadlock-free.
const headroom = MAX_DEPTH - (depth + 1);
const edges: { t: string; next: any }[] = [];
if (headroom >= 0) {
for (let i = 0; i < match.edgeCount; i++) {
const edge = match.edge(i);
const t = edge.type.name;
if (EXCLUDED.has(t) || t === 'text') continue;
if (!childAllowedUnder(t, parentType)) continue;
if ((MIN_DEPTH[t] ?? Infinity) > headroom) continue;
edges.push({ t, next: edge.next });
}
}
// Decide the next action: STOP (if allowed) or extend with one more child.
// Bias toward stopping when the budget is spent; force a child only when the
// match is not yet at a valid end.
const pool: { weight: number; arbitrary: fc.Arbitrary<{ t: string; next: any } | null> }[] = [];
const canGo = edges.length > 0 && (budget > 0 || !canStop);
if (canStop) {
// Stop is weighted higher when the budget is low so docs stay bounded.
pool.push({ weight: budget > 0 ? 2 : 5, arbitrary: fc.constant(null) });
}
if (canGo && !(canStop && budget <= 0)) {
pool.push({ weight: 3, arbitrary: fc.constantFrom(...edges) });
}
// Forced continuation: not a valid end yet and (budget exhausted) — must place
// a mandatory child regardless of budget.
if (pool.length === 0) {
if (edges.length > 0) {
pool.push({ weight: 1, arbitrary: fc.constantFrom(...edges) });
} else {
// No legal child and not required to place one: stop with what we have.
return fc.constant({ children: [], budget });
}
}
return fc.oneof(...pool).chain((choice) => {
if (choice === null) return fc.constant({ children: [], budget });
return blockNode(choice.t, depth + 1, budget).chain((node) => {
const cost = countNodes(node);
return fillMatch(choice.next, depth, budget - cost, parentType).map(
({ children, budget: left }) => ({
children: [node, ...children],
budget: left,
}),
);
});
});
}
/**
* The nested-document arbitrary: a valid, arbitrarily-deep ProseMirror doc built
* by walking the schema from the document root. Attrs stay in the round-trip-safe
* 'p1' space; inline content reuses the byte-stable flat corpus.
*/
export const docArb: fc.Arbitrary<any> = fillMatch(
schema.nodes.doc.contentMatch,
0,
NODE_BUDGET,
'doc',
).map(({ children }) => ({ type: 'doc', content: children }));
/** The precomputed minDepth table, exported for inspection/debugging. */
export { MIN_DEPTH };
@@ -0,0 +1,185 @@
import { describe, expect, it, vi } from 'vitest';
import fc from 'fast-check';
// Real converters. Importing markdownToProseMirror (transitively, via index)
// mutates the global DOM via jsdom at module load — expected, required for
// @tiptap/html's generateJSON under Node (same as the flat sibling suite).
import {
convertProseMirrorToMarkdown,
markdownToProseMirror,
docsCanonicallyEqual,
canonicalizeContent,
} from '../../src/lib/index.js';
import { firstDivergence } from '../roundtrip-helpers.js';
import { schema, docArb } from './doc-generator.js';
// Each run does a real convert + jsdom parse; give ample headroom so the suite
// is deterministic under parallel worker load (matching the flat sibling suite).
vi.setConfig({ testTimeout: 60000 });
// ---------------------------------------------------------------------------
// #351 PR 2 — GENERATIVE round-trip over NESTED (whole-document) docs produced
// by the ContentMatch random walk (doc-generator.ts). The invariants mirror the
// flat suite, plus a parser-fuzz totality property (P4):
//
// P1 — semantic round-trip: docsCanonicallyEqual(mdToPm(pmToMd(d)), d)
// P2 — byte fixpoint (2nd pass): pmToMd(mdToPm(pmToMd(d))) === pmToMd(d)
// (the FIRST pass may normalize once; the SECOND pass must be a fixpoint)
// P3 — totality: neither converter throws; bounded.
// P4 — parser fuzz totality: for ANY string, markdownToProseMirror does NOT
// throw and returns a SCHEMA-VALID document.
//
// GUARDRAIL: a P1/P2/P3/P4 failure means the generator FOUND A REAL CONVERTER
// BUG. These invariants are kept STRICT — no it.fails / skip / weakening. A
// failure prints the shrunk minimal counterexample for triage.
// ---------------------------------------------------------------------------
const SEED = 20250705;
// The nested walk builds far heavier docs than the flat suite (each P1/P2 run
// parses the emitted markdown through jsdom), so keep the run count moderate to
// hold runtime and worker memory in budget while still exercising deep
// structures. P4 (cheap string parsing) runs at a higher count below.
const NUM_RUNS = 100;
const pmToMd = (doc: unknown): string => convertProseMirrorToMarkdown(doc);
const mdToPm = (md: string): Promise<any> => markdownToProseMirror(md);
async function roundTrip(doc: unknown): Promise<{ md1: string; md2: string; doc2: any }> {
const md1 = pmToMd(doc);
const doc2 = await mdToPm(md1);
const md2 = pmToMd(doc2);
return { md1, md2, doc2 };
}
describe('#351 nested generative round-trip — generator validity', () => {
it('every generated nested doc passes schema.nodeFromJSON(...).check()', () => {
// A nested generator that emits an invalid ProseMirror document is a
// GENERATOR bug — the ContentMatch walk must only produce schema-valid docs.
fc.assert(
fc.property(docArb, (doc) => {
schema.nodeFromJSON(doc).check(); // throws on an invalid doc
return true;
}),
{ numRuns: NUM_RUNS * 2, seed: SEED },
);
});
});
// ── STATUS: P1/P2/P3/P4 all GREEN. The nested generator originally surfaced a
// batch of real converter bugs; all were fixed in the serializer/parser (see the
// #351 hand-off). For the record, the classes it found and that are now fixed:
// • Loose (multi-block) list items / task items / callouts / details bodies were
// joined with a single "\n", so every block after the first merged into the
// first paragraph on re-parse (silent content loss) — now blank-line separated.
// • Paragraph `textAlign` was dropped inside a TIGHT list item (no <p> host).
// • A nested codeBlock lost its trailing newline on the raw-HTML path.
// • Media (embed/video/youtube/drawio/excalidraw) inside `columns` churned a
// default `data-align` and coerced embed's numeric width/height to strings.
// • pageBreak / pageEmbed / subpages / transclusion were dropped when nested in
// blockquote / callout / details / list item (standalone-comment position).
// • Callouts nested in a list item or a blockquote (` > [!type]` / `> > [!type]`)
// were re-parsed as plain blockquotes (prefix-unaware callout preprocessor).
// • Two adjacent sibling lists sharing a marker family (bulletList/taskList →
// `<ul>`; orderedList → `<ol>`) merged into one list on re-parse — and for the
// cross-type case (taskList beside bulletList) the merged `<ul>` LOST every
// taskItem checkbox. The serializer now emits a `<!-- -->` separator between
// such adjacent lists (markdown-converter.ts renderBlockChildren), so they stay
// distinct and round-trip; the generator therefore emits them freely again.
describe('#351 nested generative round-trip — properties', () => {
it('P1 — semantic round-trip: docsCanonicallyEqual(mdToPm(pmToMd(d)), d)', async () => {
await fc.assert(
fc.asyncProperty(docArb, async (doc) => {
const { doc2 } = await roundTrip(doc);
if (!docsCanonicallyEqual(doc2, doc)) {
const div = firstDivergence(
JSON.parse(JSON.stringify(canonicalizeContent(doc2))),
JSON.parse(JSON.stringify(canonicalizeContent(doc))),
);
throw new Error(
`P1 divergence @ ${div?.path}: got=${JSON.stringify(div?.a)} want=${JSON.stringify(div?.b)}`,
);
}
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
it('P2 — byte fixpoint: pmToMd(mdToPm(pmToMd(d))) === pmToMd(d)', async () => {
await fc.assert(
fc.asyncProperty(docArb, async (doc) => {
const { md1, md2 } = await roundTrip(doc);
expect(md2).toBe(md1);
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
it('P3 — totality: neither converter throws', async () => {
await fc.assert(
fc.asyncProperty(docArb, async (doc) => {
await roundTrip(doc);
}),
{ numRuns: NUM_RUNS, seed: SEED },
);
});
});
// ---------------------------------------------------------------------------
// P4 — parser fuzz. Independent of the doc generator: for ANY input string the
// PARSER (markdownToProseMirror) must be TOTAL — never throw — and must always
// return a schema-valid document. The corpus mixes raw unicode strings with
// strings assembled from markdown-significant fragments (headings, list bullets,
// fences, pipes, thematic breaks, HTML-ish snippets) to probe the block/inline
// parsers on hostile but plausible input.
// ---------------------------------------------------------------------------
const mdFragmentArb: fc.Arbitrary<string> = fc.constantFrom(
'# ', '## ', '### ###', '- ', '* ', '+ ', '1. ', '> ', '>> ',
'```', '```js', '~~~', '---', '***', '___', '| a | b |', '|---|---|',
'[link](http://x)', '![img](http://x)', '**', '__', '~~', '`code`',
'<div>', '</div>', '<b>', '<!-- c -->', '<table>', '<br>', '&amp;',
'\t', '\n', ' ', '\\', '^[fn]', '[^1]:', '- [ ] ', '- [x] ',
'$$', '$x$', ':::', '{.class}', '\u0000', '\uFEFF', '😀', 'مرحبا',
);
// Full-unicode strings (fast-check v4 replaced fullUnicodeString with the
// `unit: 'binary'` string option, which draws over the whole code-point range).
const fullUnicodeStringArb = (max?: number) =>
fc.string({ unit: 'binary', ...(max !== undefined ? { maxLength: max } : {}) });
const assembledMarkdownArb: fc.Arbitrary<string> = fc
.array(fc.oneof(mdFragmentArb, fc.string(), fullUnicodeStringArb(8)), {
minLength: 1,
maxLength: 12,
})
.map((parts) => parts.join(''));
const parserInputArb: fc.Arbitrary<string> = fc.oneof(
{ weight: 2, arbitrary: fc.string() },
{ weight: 2, arbitrary: fullUnicodeStringArb() },
{ weight: 3, arbitrary: assembledMarkdownArb },
{ weight: 1, arbitrary: fc.array(mdFragmentArb, { minLength: 1, maxLength: 8 }).map((p) => p.join('\n')) },
);
describe('#351 parser fuzz — totality on arbitrary input (P4)', () => {
it('P4 — markdownToProseMirror never throws and always returns a schema-valid doc', async () => {
await fc.assert(
fc.asyncProperty(parserInputArb, async (s) => {
let result: any;
try {
result = await mdToPm(s);
} catch (e: any) {
throw new Error(`P4 parser THREW on input ${JSON.stringify(s)}: ${e?.message ?? e}`);
}
try {
schema.nodeFromJSON(result).check();
} catch (e: any) {
throw new Error(
`P4 parser produced an INVALID doc for input ${JSON.stringify(s)}: ${e?.message ?? e}\n` +
`doc=${JSON.stringify(result).slice(0, 600)}`,
);
}
}),
{ numRuns: NUM_RUNS * 2, seed: SEED },
);
});
});
@@ -374,7 +374,9 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
], ],
}), }),
); );
expect(out).toBe('- [ ] top\n - child'); // Block children of a task item are blank-line separated (loose list) per the
// #351 fix; the sublist stays at the fixed 2-column continuation indent.
expect(out).toBe('- [ ] top\n\n - child');
}); });
// 10. A bulletList inside a blockquote: each list line independently prefixed. // 10. A bulletList inside a blockquote: each list line independently prefixed.
@@ -198,7 +198,11 @@ describe('convertProseMirrorToMarkdown', () => {
}), }),
); );
// First line carries the marker; the nested list is indented 2 columns. // First line carries the marker; the nested list is indented 2 columns.
expect(out).toBe('- parent\n - child'); // Block children of a list item are separated by a BLANK line (loose list):
// this is the #351 fix — a single "\n" let a following block merge into the
// first paragraph on re-parse (silent content loss). The blank line stays
// inside the item, so the sublist remains nested at the 2-col marker column.
expect(out).toBe('- parent\n\n - child');
}); });
it('nested ordered list indents by the wider 3-col marker width', () => { it('nested ordered list indents by the wider 3-col marker width', () => {
@@ -219,8 +223,9 @@ describe('convertProseMirrorToMarkdown', () => {
], ],
}), }),
); );
// "1. " is 3 columns wide, so the continuation indent is 3 spaces. // "1. " is 3 columns wide, so the continuation indent is 3 spaces. Block
expect(out).toBe('1. parent\n 1. child'); // children are blank-line separated (loose list) per the #351 fix.
expect(out).toBe('1. parent\n\n 1. child');
}); });
}); });
@@ -539,11 +544,12 @@ describe('convertProseMirrorToMarkdown', () => {
); );
// The 10th marker is the 4-column "10. "; the nested sublist line must be // The 10th marker is the 4-column "10. "; the nested sublist line must be
// indented exactly 4 spaces (prefix.length 3 + 1), NOT 3. // indented exactly 4 spaces (prefix.length 3 + 1), NOT 3. Block children are
expect(out).toContain('10. j\n 1. x'); // blank-line separated (loose list) per the #351 fix.
expect(out).toContain('10. j\n\n 1. x');
// Guard against the off-by-one (3-space) regression that would re-parse // Guard against the off-by-one (3-space) regression that would re-parse
// the sublist as loose/sibling content on import. // the sublist as loose/sibling content on import.
expect(out).not.toContain('10. j\n 1. x'); expect(out).not.toContain('10. j\n\n 1. x');
// And the single-digit items keep the narrower 3-column marker (no body // And the single-digit items keep the narrower 3-column marker (no body
// continuation here, but the marker itself must stay "1. ".."9. "). // continuation here, but the marker itself must stay "1. ".."9. ").
expect(out.startsWith('1. a\n2. b\n')).toBe(true); expect(out.startsWith('1. a\n2. b\n')).toBe(true);
@@ -580,17 +586,18 @@ describe('convertProseMirrorToMarkdown', () => {
content: [para(text('line1')), para(text('line2'))], content: [para(text('line1')), para(text('line2'))],
}), }),
); );
// NOTE(review): the spec predicted ':::warning\nline1\n\nline2\n:::' (a // The converter emits an Obsidian-native callout: a `> [!type]` opener plus
// The converter joins the callout's rendered children with a single '\n' // one `>`-prefixed body line per content line. Block children are separated
// and emits an Obsidian-native callout: a `> [!type]` opener plus one // by a blank `>` line (#351 fix): a single '\n' let the two paragraphs merge
// `>`-prefixed body line per content line. We pin the lowercasing // into one on re-parse. We pin the lowercasing (WARNING -> warning) and the
// (WARNING -> warning) and the multi-child join. // blank-line-separated multi-child join.
expect(out).toBe('> [!warning]\n> line1\n> line2'); expect(out).toBe('> [!warning]\n> line1\n>\n> line2');
// The type is lowercased (an uppercase `[!WARNING]` would not re-import). // The type is lowercased (an uppercase `[!WARNING]` would not re-import).
expect(out.startsWith('> [!warning]\n')).toBe(true); expect(out.startsWith('> [!warning]\n')).toBe(true);
expect(out).not.toContain('[!WARNING]'); expect(out).not.toContain('[!WARNING]');
// Both paragraph children are present, each blockquote-prefixed. // Both paragraph children are present, each blockquote-prefixed, blank-`>`
expect(out).toContain('> line1\n> line2'); // separated so they stay distinct paragraphs on re-parse.
expect(out).toContain('> line1\n>\n> line2');
}); });
// Spec 4 — blockquote per-line prefixer over a multi-line nested callout. // Spec 4 — blockquote per-line prefixer over a multi-line nested callout.
@@ -607,13 +614,11 @@ describe('convertProseMirrorToMarkdown', () => {
], ],
}), }),
); );
// NOTE(review): the spec predicted '> :::info\n> a\n>\n> b\n> :::', // The nested callout renders as an Obsidian callout '> [!info]\n> a\n>\n> b'
// assuming the nested callout body contains a blank line between 'a' and // (blank-`>` separated children per the #351 fix). The outer blockquote
// The nested callout renders as an Obsidian callout '> [!info]\n> a\n> b' // prefixer then prefixes each of those lines with '> ' again, yielding a
// (single-'\n' join, no blank line). The outer blockquote prefixer then // doubly-nested blockquote — the per-line-prefix loop over a multi-line child.
// prefixes each of those lines with '> ' again, yielding a doubly-nested expect(out).toBe('> > [!info]\n> > a\n> >\n> > b');
// blockquote — the realistic per-line-prefix loop over a multi-line child.
expect(out).toBe('> > [!info]\n> > a\n> > b');
// Every produced line carries the '> ' prefix (no line escapes to col 0). // Every produced line carries the '> ' prefix (no line escapes to col 0).
for (const line of out.split('\n')) { for (const line of out.split('\n')) {
expect(line.startsWith('>')).toBe(true); expect(line.startsWith('>')).toBe(true);