Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0a53be9e81 | |||
| 95c0d813b0 | |||
| 9004de60e3 | |||
| 3a344626db | |||
| 31f51eaa47 | |||
| b66929714f | |||
| a09935aa29 | |||
| 047433595e | |||
| 9e95412695 | |||
| 2fa86e2a33 | |||
| e3eece78c3 | |||
| e1b8ef5b8b | |||
| fe5bd159c4 | |||
| f12b685698 | |||
| f6fc914c95 | |||
| 1d89cc2058 | |||
| 5d8083f8ff |
@@ -334,7 +334,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
|
|||||||
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
|
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
|
||||||
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
|
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
|
||||||
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
|
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
|
||||||
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
|
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked; jsdom (Node only) | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, `apps/server` (server-side markdown import/export, #345), AND `apps/client` (markdown paste/copy + AI-chat render, via the `browser` entry — native `DOMParser`, no jsdom in the client bundle, #347); there is exactly ONE copy of the converter now |
|
||||||
|
|
||||||
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
|
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
|
||||||
|
|
||||||
@@ -460,7 +460,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
|
|||||||
### Client structure
|
### Client structure
|
||||||
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
|
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
|
||||||
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
|
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
|
||||||
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
|
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, `apps/server` (#345), and `apps/client` (#347) — do NOT reintroduce a per-package copy. The client uses the package's `browser` entry (`@docmost/prosemirror-markdown/browser`): markdown paste (`markdown-clipboard.ts`), copy-as-markdown, and AI-chat rendering now all go through the canonical converter, so the hand-written `marked`/`turndown` markdown layer that used to live in `editor-ext` was deleted (#347). The browser entry runs the HTML→DOM stage on the native `DOMParser`, so jsdom stays out of the client bundle. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
|
||||||
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
|
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
|
||||||
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
|
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
|
||||||
|
|
||||||
|
|||||||
@@ -270,6 +270,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
|
- **Client markdown paste/copy and AI-chat rendering now go through the canonical
|
||||||
|
converter.** Pasting markdown into the editor, "Copy as markdown", the AI title
|
||||||
|
generator, and the AI-chat markdown renderer all now use
|
||||||
|
`@docmost/prosemirror-markdown` (via its new `browser` entry — native
|
||||||
|
`DOMParser`, no jsdom in the client bundle) instead of the hand-written
|
||||||
|
`marked`/`turndown` markdown layer in `editor-ext`, which was **deleted**. As a
|
||||||
|
result, pasting canonical markdown (`^[…]` footnotes, `<!--img …-->`,
|
||||||
|
`> [!type]` callouts, `$…$` math, `==…==` highlight, standalone `<!--subpages-->`
|
||||||
|
comments) now produces the SAME nodes the server import produces for the same
|
||||||
|
text. Chat/reasoning markdown now renders through the editor schema (list items
|
||||||
|
are wrapped in `<p>`; CSS keeps them tight). (#347)
|
||||||
|
|
||||||
- **Enabling a public share no longer auto-shares the whole sub-tree.** Turning
|
- **Enabling a public share no longer auto-shares the whole sub-tree.** Turning
|
||||||
a page "Shared to web" now defaults to the page alone; descendant pages become
|
a page "Shared to web" now defaults to the page alone; descendant pages become
|
||||||
public only when you explicitly turn on the dedicated "Include sub-pages"
|
public only when you explicitly turn on the dedicated "Include sub-pages"
|
||||||
@@ -290,6 +302,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
- **Markdown round-trips no longer silently drop a line that opens with a block
|
||||||
|
trigger.** When a document is exported to Markdown and re-imported (git-sync
|
||||||
|
stabilize, agent writes), a paragraph or continuation line (after a hard break)
|
||||||
|
that begins with a block marker — an ATX heading `#`, a blockquote/callout `>`,
|
||||||
|
a list marker (`-`/`*`/`+`/`N.`/`N)`), a code fence, a table `|`, a thematic
|
||||||
|
break (`---`), or a setext underline (`--`, `----`, or a lone `=`) — is now
|
||||||
|
backslash-escaped so it round-trips as text instead of being re-parsed into a
|
||||||
|
heading/list/quote/rule and losing its content. Front-matter stripping is
|
||||||
|
scoped to the import path only. (#493)
|
||||||
- **The server no longer runs out of heap during long autonomous agent runs.** A
|
- **The server no longer runs out of heap during long autonomous agent runs.** A
|
||||||
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
|
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
|
||||||
snapshot of the ENTIRE turn text on every streamed text-delta when no output
|
snapshot of the ENTIRE turn text on every streamed text-delta when no output
|
||||||
|
|||||||
@@ -21,6 +21,7 @@
|
|||||||
"@atlaskit/pragmatic-drag-and-drop-live-region": "1.3.4",
|
"@atlaskit/pragmatic-drag-and-drop-live-region": "1.3.4",
|
||||||
"@casl/react": "5.0.1",
|
"@casl/react": "5.0.1",
|
||||||
"@docmost/editor-ext": "workspace:*",
|
"@docmost/editor-ext": "workspace:*",
|
||||||
|
"@docmost/prosemirror-markdown": "workspace:*",
|
||||||
"@excalidraw/excalidraw": "0.18.0-3a5ef40",
|
"@excalidraw/excalidraw": "0.18.0-3a5ef40",
|
||||||
"@mantine/core": "8.3.18",
|
"@mantine/core": "8.3.18",
|
||||||
"@mantine/dates": "8.3.18",
|
"@mantine/dates": "8.3.18",
|
||||||
|
|||||||
@@ -55,6 +55,15 @@
|
|||||||
padding-inline-start: 1.4em;
|
padding-inline-start: 1.4em;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The canonical converter renders list items through the editor schema, which
|
||||||
|
wraps each item's content in a <p> (listItem content is `paragraph+`). Drop
|
||||||
|
that paragraph's block margin so list items render TIGHT (no extra vertical
|
||||||
|
gap), matching the previous marked output — same rule already applied to
|
||||||
|
table cells above (issue #347). */
|
||||||
|
.markdown li p {
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
/* GFM tables in assistant markdown. The chat lives in a NARROW side panel, so a
|
/* GFM tables in assistant markdown. The chat lives in a NARROW side panel, so a
|
||||||
wide LLM table must scroll horizontally instead of collapsing its columns:
|
wide LLM table must scroll horizontally instead of collapsing its columns:
|
||||||
`.markdown` sets `word-break: break-word`, which (with the default table
|
`.markdown` sets `word-break: break-word`, which (with the default table
|
||||||
@@ -172,6 +181,14 @@
|
|||||||
margin: 0 0 4px;
|
margin: 0 0 4px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Same as `.markdown li p` above: the canonical converter wraps every list
|
||||||
|
item's content in a <p>, so without this each reasoning-panel list item would
|
||||||
|
pick up `.reasoningText p`'s 4px bottom margin and render too loose. Drop it
|
||||||
|
so Reasoning-panel lists stay tight, mirroring the pre-#347 marked output. */
|
||||||
|
.reasoningText li p {
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
.inputWrapper {
|
.inputWrapper {
|
||||||
flex: 0 0 auto;
|
flex: 0 0 auto;
|
||||||
padding-top: var(--mantine-spacing-xs);
|
padding-top: var(--mantine-spacing-xs);
|
||||||
|
|||||||
@@ -33,29 +33,44 @@ describe("collapseBlankLines", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("collapseBlankLines + renderChatMarkdown (tight reasoning rendering)", () => {
|
describe("collapseBlankLines + renderChatMarkdown (canonical converter)", () => {
|
||||||
it("renders a blank-line-separated list as a TIGHT list (no <li><p>)", () => {
|
// Chat markdown now renders through @docmost/prosemirror-markdown (issue #347):
|
||||||
|
// the SAME converter the editor/import use. Its list items are schema-shaped —
|
||||||
|
// each <li>'s content is wrapped in a <p> (listItem content is `paragraph+`) —
|
||||||
|
// so the HTML always carries `<li><p>…</p></li>` regardless of blank-line
|
||||||
|
// looseness in the source (the converter has no tight/loose distinction). The
|
||||||
|
// visual tightness that `collapseBlankLines` used to buy is now provided by
|
||||||
|
// CSS (`.markdown li p { margin: 0 }`), not the HTML shape.
|
||||||
|
it("renders a blank-line-separated bullet list as a real <ul> list", () => {
|
||||||
const loose =
|
const loose =
|
||||||
"Intro paragraph.\n\n- item one\n\n- item two\n\n- item three";
|
"Intro paragraph.\n\n- item one\n\n- item two\n\n- item three";
|
||||||
const html = renderChatMarkdown(collapseBlankLines(loose), {});
|
const html = renderChatMarkdown(collapseBlankLines(loose), {});
|
||||||
// Tight list: each <li> holds the text directly, not wrapped in a <p>.
|
// Clean, un-namespaced HTML (DOMSerializer, not XMLSerializer) — no xmlns.
|
||||||
expect(html).toContain("<li>item one</li>");
|
|
||||||
expect(html).not.toContain("<li><p>");
|
|
||||||
// The list still parses as a list after the paragraph (not a paragraph+<br>).
|
|
||||||
expect(html).toContain("<ul>");
|
expect(html).toContain("<ul>");
|
||||||
|
expect(html).not.toMatch(/<ul[^>]*xmlns/);
|
||||||
|
// The item text is present (inside the schema's <li><p> wrapper).
|
||||||
|
expect(html).toContain("item one");
|
||||||
|
// The intro paragraph renders as its own paragraph before the list.
|
||||||
expect(html).toContain("<p>Intro paragraph.</p>");
|
expect(html).toContain("<p>Intro paragraph.</p>");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("renders an ordered list (1. 2.) as tight after collapsing", () => {
|
it("renders an ordered list (1. 2.) as a real <ol> list", () => {
|
||||||
const loose = "Intro.\n\n1. first\n\n2. second";
|
const loose = "Intro.\n\n1. first\n\n2. second";
|
||||||
const html = renderChatMarkdown(collapseBlankLines(loose), {});
|
const html = renderChatMarkdown(collapseBlankLines(loose), {});
|
||||||
expect(html).toContain("<ol>");
|
expect(html).toContain("<ol>");
|
||||||
expect(html).toContain("<li>first</li>");
|
expect(html).not.toMatch(/<ol[^>]*xmlns/);
|
||||||
expect(html).not.toContain("<li><p>");
|
expect(html).toContain("first");
|
||||||
|
expect(html).toContain("second");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("the loose source WOULD render <li><p> without collapsing (control)", () => {
|
it("wraps list-item content in <p> (schema shape; tightness is CSS)", () => {
|
||||||
|
// The canonical converter always wraps a list item's content in a paragraph,
|
||||||
|
// whether or not the source had blank lines between items.
|
||||||
const loose = "- a\n\n- b";
|
const loose = "- a\n\n- b";
|
||||||
expect(renderChatMarkdown(loose, {})).toContain("<li><p>");
|
expect(renderChatMarkdown(loose, {})).toContain("<li><p>");
|
||||||
|
// And a "tight" source produces the identical wrapping (no distinction).
|
||||||
|
expect(renderChatMarkdown(collapseBlankLines(loose), {})).toContain(
|
||||||
|
"<li><p>",
|
||||||
|
);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,6 +1,37 @@
|
|||||||
import { markdownToHtml } from "@docmost/editor-ext";
|
import {
|
||||||
|
markdownToProseMirrorSync,
|
||||||
|
docmostExtensions,
|
||||||
|
} from "@docmost/prosemirror-markdown/browser";
|
||||||
|
import { getSchema } from "@tiptap/core";
|
||||||
|
import { Node as PMNode, DOMSerializer } from "@tiptap/pm/model";
|
||||||
import DOMPurify from "dompurify";
|
import DOMPurify from "dompurify";
|
||||||
|
|
||||||
|
// The Docmost editor schema, built once. Chat markdown is rendered through the
|
||||||
|
// SAME schema the editor/import use (issue #347), so chat output matches how the
|
||||||
|
// page would render the same markdown.
|
||||||
|
const chatSchema = getSchema(docmostExtensions);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Markdown -> HTML for chat display, via the canonical converter. We serialize
|
||||||
|
* the ProseMirror doc with `DOMSerializer` into a real element and read its
|
||||||
|
* `innerHTML` (rather than `@tiptap/html`'s `generateHTML`, whose browser path
|
||||||
|
* uses `XMLSerializer` and stamps a `xmlns` on every block) so the markup is
|
||||||
|
* clean HTML. `li > p` wrapping is inherent to the schema (listItem content is
|
||||||
|
* `paragraph+`); the chat CSS zeroes those paragraph margins so lists still
|
||||||
|
* render tight.
|
||||||
|
*/
|
||||||
|
function markdownToChatHtml(markdown: string): string {
|
||||||
|
const doc = markdownToProseMirrorSync(markdown);
|
||||||
|
const node = PMNode.fromJSON(chatSchema, doc);
|
||||||
|
const div = document.createElement("div");
|
||||||
|
DOMSerializer.fromSchema(chatSchema).serializeFragment(
|
||||||
|
node.content,
|
||||||
|
{ document },
|
||||||
|
div,
|
||||||
|
);
|
||||||
|
return div.innerHTML;
|
||||||
|
}
|
||||||
|
|
||||||
export interface RenderChatMarkdownOptions {
|
export interface RenderChatMarkdownOptions {
|
||||||
/**
|
/**
|
||||||
* Neutralize INTERNAL links so they render as inert text (no `href`/`target`).
|
* Neutralize INTERNAL links so they render as inert text (no `href`/`target`).
|
||||||
@@ -63,22 +94,32 @@ function neutralizeInternalLinksHook(node: Element): void {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Render AI markdown to sanitized HTML for read-only display. We reuse the
|
* Render AI markdown to sanitized HTML for read-only display. We reuse the
|
||||||
* app's `markdownToHtml` (the same `marked` pipeline used for paste/import) so
|
* canonical converter (issue #347): markdown -> ProseMirror JSON (the SAME
|
||||||
* chat output matches the editor's markdown flavor, then sanitize with
|
* `markdownToProseMirrorSync` the editor paste/import path uses, so chat output
|
||||||
* DOMPurify — LLM output is untrusted, so it must never reach the DOM unsanitized.
|
* matches the editor's markdown flavor) -> HTML via `markdownToChatHtml`
|
||||||
|
* (DOMSerializer), then sanitize with DOMPurify — LLM output is untrusted, so it
|
||||||
|
* must never reach the DOM unsanitized.
|
||||||
*
|
*
|
||||||
* `markdownToHtml` can return `string | Promise<string>` (it has async marked
|
* Stays SYNCHRONOUS: both callers render inside React (a memo and a useMemo),
|
||||||
* extensions registered). In practice plain chat markdown resolves
|
* so the whole pipeline must resolve without awaiting. The converter's sync
|
||||||
* synchronously, but we guard the Promise case by returning a safe empty string
|
* entry makes that possible; on any conversion error we return "" so the caller
|
||||||
* for that branch (the caller renders the raw text fallback instead).
|
* falls back to raw text (the same fallback the old Promise-guard produced).
|
||||||
*/
|
*/
|
||||||
export function renderChatMarkdown(
|
export function renderChatMarkdown(
|
||||||
markdown: string,
|
markdown: string,
|
||||||
options: RenderChatMarkdownOptions = {},
|
options: RenderChatMarkdownOptions = {},
|
||||||
): string {
|
): string {
|
||||||
if (!markdown) return "";
|
if (!markdown) return "";
|
||||||
const html = markdownToHtml(markdown);
|
let html: string;
|
||||||
if (typeof html !== "string") return "";
|
try {
|
||||||
|
// markdown -> canonical PM JSON -> HTML (native DOMParser in the browser;
|
||||||
|
// jsdom is never bundled — see @docmost/prosemirror-markdown/browser).
|
||||||
|
html = markdownToChatHtml(markdown);
|
||||||
|
} catch {
|
||||||
|
// Malformed/unsupported markdown must not crash the chat render; fall back
|
||||||
|
// to raw text (empty return -> caller shows the plain-text branch).
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
|
||||||
if (!options.neutralizeInternalLinks) {
|
if (!options.neutralizeInternalLinks) {
|
||||||
// Internal chat: unchanged behavior, no hook registered.
|
// Internal chat: unchanged behavior, no hook registered.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import { EditorContent, ReactNodeViewRenderer, useEditor } from "@tiptap/react";
|
import { EditorContent, ReactNodeViewRenderer, useEditor } from "@tiptap/react";
|
||||||
import { Placeholder } from "@tiptap/extension-placeholder";
|
import { Placeholder } from "@tiptap/extension-placeholder";
|
||||||
import { StarterKit } from "@tiptap/starter-kit";
|
import { StarterKit } from "@tiptap/starter-kit";
|
||||||
import { Mention, LinkExtension } from "@docmost/editor-ext";
|
import { Mention, LinkExtension, Code } from "@docmost/editor-ext";
|
||||||
import classes from "./comment.module.css";
|
import classes from "./comment.module.css";
|
||||||
import { useFocusWithin } from "@mantine/hooks";
|
import { useFocusWithin } from "@mantine/hooks";
|
||||||
import clsx from "clsx";
|
import clsx from "clsx";
|
||||||
@@ -44,7 +44,12 @@ const CommentEditor = forwardRef(
|
|||||||
gapcursor: false,
|
gapcursor: false,
|
||||||
dropcursor: false,
|
dropcursor: false,
|
||||||
link: false,
|
link: false,
|
||||||
|
// #515: use the shared editor-ext `Code` (excludes: "") instead of
|
||||||
|
// StarterKit's excluding one, so inline code in a comment can carry
|
||||||
|
// other marks and does not drop them when the comment is edited.
|
||||||
|
code: false,
|
||||||
}),
|
}),
|
||||||
|
Code,
|
||||||
Placeholder.configure({
|
Placeholder.configure({
|
||||||
placeholder: placeholder || t("Reply..."),
|
placeholder: placeholder || t("Reply..."),
|
||||||
}),
|
}),
|
||||||
|
|||||||
@@ -1,6 +1,5 @@
|
|||||||
import { markInputRule } from "@tiptap/core";
|
import { markInputRule } from "@tiptap/core";
|
||||||
import { StarterKit } from "@tiptap/starter-kit";
|
import { StarterKit } from "@tiptap/starter-kit";
|
||||||
import { Code } from "@tiptap/extension-code";
|
|
||||||
import { TextAlign } from "@tiptap/extension-text-align";
|
import { TextAlign } from "@tiptap/extension-text-align";
|
||||||
import { TaskList, TaskItem } from "@tiptap/extension-list";
|
import { TaskList, TaskItem } from "@tiptap/extension-list";
|
||||||
import { Placeholder, CharacterCount, UndoRedo } from "@tiptap/extensions";
|
import { Placeholder, CharacterCount, UndoRedo } from "@tiptap/extensions";
|
||||||
@@ -67,6 +66,7 @@ import {
|
|||||||
FootnoteReference,
|
FootnoteReference,
|
||||||
FootnotesList,
|
FootnotesList,
|
||||||
FootnoteDefinition,
|
FootnoteDefinition,
|
||||||
|
Code,
|
||||||
} from "@docmost/editor-ext";
|
} from "@docmost/editor-ext";
|
||||||
import {
|
import {
|
||||||
randomElement,
|
randomElement,
|
||||||
@@ -153,6 +153,10 @@ export const mainExtensions = [
|
|||||||
codeBlock: false,
|
codeBlock: false,
|
||||||
code: false,
|
code: false,
|
||||||
}),
|
}),
|
||||||
|
// Base `Code` comes from @docmost/editor-ext, which overrides `excludes: ""`
|
||||||
|
// (#515) so inline code can co-occur with bold/italic/… — the SINGLE shared
|
||||||
|
// source also used by the collab server and comment editor. Here we keep the
|
||||||
|
// existing client-only behavior on top of it:
|
||||||
// Override TipTap's Code extension to fix the inline code input rule.
|
// Override TipTap's Code extension to fix the inline code input rule.
|
||||||
// The upstream regex /(^|[^`])`([^`]+)`(?!`)$/ captures the character
|
// The upstream regex /(^|[^`])`([^`]+)`(?!`)$/ captures the character
|
||||||
// before the opening backtick as part of the match, causing markInputRule
|
// before the opening backtick as part of the match, causing markInputRule
|
||||||
|
|||||||
@@ -0,0 +1,206 @@
|
|||||||
|
import { describe, it, expect } from "vitest";
|
||||||
|
import { Editor } from "@tiptap/core";
|
||||||
|
import { Document } from "@tiptap/extension-document";
|
||||||
|
import { Paragraph } from "@tiptap/extension-paragraph";
|
||||||
|
import { Text } from "@tiptap/extension-text";
|
||||||
|
import { Bold } from "@tiptap/extension-bold";
|
||||||
|
import { Italic } from "@tiptap/extension-italic";
|
||||||
|
import { MarkdownClipboard } from "./markdown-clipboard";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Integration coverage for the async `handlePaste` seam (issue #347). The paste
|
||||||
|
* conversion moved to `@docmost/prosemirror-markdown`'s browser entry, whose
|
||||||
|
* `markdownToProseMirror` is async — so `handlePaste` captures the range, claims
|
||||||
|
* the event (returns true), and dispatches the insert on the next microtask.
|
||||||
|
* These tests drive that path end to end on a minimal schema (a plain-markdown
|
||||||
|
* paste whose converted nodes fit paragraph/text/bold/italic), asserting the
|
||||||
|
* text lands with the right marks and that the raw markdown syntax is consumed
|
||||||
|
* (recognized as markdown, not inserted literally).
|
||||||
|
*/
|
||||||
|
|
||||||
|
function makeEditor() {
|
||||||
|
const element = document.createElement("div");
|
||||||
|
document.body.appendChild(element);
|
||||||
|
return new Editor({
|
||||||
|
element,
|
||||||
|
extensions: [
|
||||||
|
Document,
|
||||||
|
Paragraph,
|
||||||
|
Text,
|
||||||
|
Bold,
|
||||||
|
Italic,
|
||||||
|
MarkdownClipboard.configure({ transformPastedText: true }),
|
||||||
|
],
|
||||||
|
content: { type: "doc", content: [{ type: "paragraph" }] },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Locate the markdownClipboard plugin and invoke its handlePaste directly with a
|
||||||
|
// synthetic clipboard event (jsdom has no real paste pipeline). The plugin's
|
||||||
|
// handlePaste closes over the extension `this`, so calling it off the plugin
|
||||||
|
// props preserves `this.editor`/`this.options`.
|
||||||
|
function paste(editor: Editor, text: string): boolean {
|
||||||
|
const view = editor.view;
|
||||||
|
const plugin = view.state.plugins.find(
|
||||||
|
(p: any) => p.props && p.spec?.key,
|
||||||
|
) as any;
|
||||||
|
const event = {
|
||||||
|
clipboardData: {
|
||||||
|
getData: (type: string) => (type === "text/plain" ? text : ""),
|
||||||
|
},
|
||||||
|
} as unknown as ClipboardEvent;
|
||||||
|
// Find the specific handlePaste that belongs to the markdown clipboard plugin.
|
||||||
|
const md = view.state.plugins.find(
|
||||||
|
(p: any) => typeof p.props?.handlePaste === "function",
|
||||||
|
) as any;
|
||||||
|
return md.props.handlePaste(view, event, view.state.selection.content());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flush the microtask queue so the async .then() dispatch runs.
|
||||||
|
const flush = () => new Promise((r) => setTimeout(r, 0));
|
||||||
|
|
||||||
|
describe("MarkdownClipboard handlePaste (async md -> PM)", () => {
|
||||||
|
it("converts a plain-markdown paste with bold/italic into marked text", async () => {
|
||||||
|
const editor = makeEditor();
|
||||||
|
const claimed = paste(editor, "hello **bold** and *italic*");
|
||||||
|
// The paste is claimed synchronously (async insert follows).
|
||||||
|
expect(claimed).toBe(true);
|
||||||
|
await flush();
|
||||||
|
|
||||||
|
const json = editor.getJSON();
|
||||||
|
const text = JSON.stringify(json);
|
||||||
|
// The raw markdown asterisks are consumed (recognized), not inserted literally.
|
||||||
|
expect(editor.getText()).not.toContain("**");
|
||||||
|
expect(editor.getText()).toContain("bold");
|
||||||
|
expect(editor.getText()).toContain("italic");
|
||||||
|
// The bold/italic marks materialized.
|
||||||
|
expect(text).toContain('"bold"');
|
||||||
|
expect(text).toContain('"italic"');
|
||||||
|
editor.destroy();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("recognizes a bullet list paste as list structure (not literal '-')", async () => {
|
||||||
|
// A bullet list is not representable in this minimal schema, so the converter
|
||||||
|
// output would fail PMNode.fromJSON and the catch inserts raw text. Use a
|
||||||
|
// paste whose nodes DO fit the schema to assert the happy path instead: two
|
||||||
|
// paragraphs separated by a blank line.
|
||||||
|
const editor = makeEditor();
|
||||||
|
paste(editor, "first para\n\nsecond para");
|
||||||
|
await flush();
|
||||||
|
const json = editor.getJSON() as any;
|
||||||
|
const paras = (json.content || []).filter(
|
||||||
|
(n: any) => n.type === "paragraph",
|
||||||
|
);
|
||||||
|
// Two paragraphs materialized from the blank-line-separated markdown.
|
||||||
|
expect(paras.length).toBeGreaterThanOrEqual(2);
|
||||||
|
expect(editor.getText()).toContain("first para");
|
||||||
|
expect(editor.getText()).toContain("second para");
|
||||||
|
editor.destroy();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("falls back to raw text when conversion yields nodes the schema lacks", async () => {
|
||||||
|
// `# heading` converts to a `heading` node absent from this minimal schema,
|
||||||
|
// so PMNode.fromJSON throws and the catch re-inserts the raw text — the user
|
||||||
|
// never loses their clipboard content.
|
||||||
|
const editor = makeEditor();
|
||||||
|
paste(editor, "# a heading line");
|
||||||
|
await flush();
|
||||||
|
// Content is preserved (either as heading text or literal), never dropped.
|
||||||
|
expect(editor.getText()).toContain("a heading line");
|
||||||
|
editor.destroy();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// The async seam captures the target range synchronously, then replaces on the
|
||||||
|
// next microtask. If the document changed under it between capture and resolve
|
||||||
|
// (impossible in prod — same microtask — but pinned here), BOTH the success
|
||||||
|
// (replaceRange) and the fail-open (insertText) branches must fall back to the
|
||||||
|
// LIVE selection rather than a stale absolute range, so neither clobbers content
|
||||||
|
// nor throws a RangeError. We force the mid-flight change by dispatching a
|
||||||
|
// doc-mutating transaction AFTER the synchronous claim but BEFORE flushing the
|
||||||
|
// microtask that runs the `.then`/`.catch`.
|
||||||
|
describe("MarkdownClipboard handlePaste — doc-changed-mid-flight guard", () => {
|
||||||
|
// Replace the whole doc with one paragraph of `text` (synchronous dispatch).
|
||||||
|
// An empty string yields an empty paragraph (a text node may not be empty).
|
||||||
|
function seedContent(editor: Editor, text: string) {
|
||||||
|
editor.commands.setContent({
|
||||||
|
type: "doc",
|
||||||
|
content: [
|
||||||
|
text
|
||||||
|
? { type: "paragraph", content: [{ type: "text", text }] }
|
||||||
|
: { type: "paragraph" },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
it("success branch: mid-flight doc change routes the paste to the LIVE selection, never the stale range (clobber-proving)", async () => {
|
||||||
|
// The paste captures a NON-EMPTY range {1,5} (over "AAAA"). Then, before the
|
||||||
|
// async resolve, the doc GROWS ("MARKER" inserted at the start) and the cursor
|
||||||
|
// is parked at the doc END. The captured {1,5} is now stale and points INTO
|
||||||
|
// "MARKER". A WORKING guard replaces at the live (end) selection → MARKER is
|
||||||
|
// untouched. A BROKEN guard replaces the stale {1,5} → it erases the first
|
||||||
|
// characters of MARKER (this is what a zero-width `from==to` range could never
|
||||||
|
// reveal, which is why the earlier version was vacuous).
|
||||||
|
const editor = makeEditor();
|
||||||
|
seedContent(editor, "AAAABBBB");
|
||||||
|
editor.commands.setTextSelection({ from: 1, to: 5 }); // captured range = {1,5}
|
||||||
|
const claimed = paste(editor, "hello **bold**");
|
||||||
|
expect(claimed).toBe(true);
|
||||||
|
|
||||||
|
// Mid-flight: grow the doc and move the cursor to a KNOWN-safe end position.
|
||||||
|
editor.view.dispatch(editor.view.state.tr.insertText("MARKER", 1));
|
||||||
|
const end = editor.state.doc.content.size;
|
||||||
|
editor.commands.setTextSelection({ from: end, to: end });
|
||||||
|
await flush();
|
||||||
|
|
||||||
|
const text = editor.getText();
|
||||||
|
// MARKER intact only if the guard used the live selection, not the stale range.
|
||||||
|
expect(text).toContain("MARKER");
|
||||||
|
expect(text).toContain("bold");
|
||||||
|
expect(text).not.toContain("**");
|
||||||
|
editor.destroy();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fail-open branch: a mid-flight doc SHRINK makes the stale `to` out of bounds — the guard must avoid a RangeError (throw-proving)", async () => {
|
||||||
|
// The paste captures a range {1,9} over an 8-char paragraph, then the
|
||||||
|
// conversion FAILS (`# heading` -> a heading node the minimal schema lacks,
|
||||||
|
// so PMNode.fromJSON throws -> the fail-open catch runs). Before the reject,
|
||||||
|
// the doc is SHRUNK to an empty paragraph, so the captured `to` (9) is now far
|
||||||
|
// past the doc's end. A WORKING guard inserts the raw text at the live (valid)
|
||||||
|
// selection → "raw heading" lands. A BROKEN guard does insertText(md, 1, 9) on
|
||||||
|
// a size-2 doc → RangeError, so the dispatch never runs and "raw heading" is
|
||||||
|
// absent (the assertion reddens). A zero-width/growing-doc setup could never
|
||||||
|
// push `to` out of bounds, which is why the earlier version was vacuous.
|
||||||
|
const editor = makeEditor();
|
||||||
|
seedContent(editor, "AAAABBBB");
|
||||||
|
editor.commands.setTextSelection({ from: 1, to: 9 }); // captured range = {1,9}
|
||||||
|
paste(editor, "# raw heading");
|
||||||
|
|
||||||
|
// Mid-flight: shrink the doc so the captured `to` = 9 is now out of bounds.
|
||||||
|
seedContent(editor, "");
|
||||||
|
await flush();
|
||||||
|
|
||||||
|
const text = editor.getText();
|
||||||
|
// Raw text lands (via the live selection) only if the guard avoided the
|
||||||
|
// stale, now-out-of-bounds range.
|
||||||
|
expect(text).toContain("raw heading");
|
||||||
|
editor.destroy();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("two pastes in flight: neither payload is lost (no data loss)", async () => {
|
||||||
|
// Prod-unreachable (two paste events are separate macrotasks, and each
|
||||||
|
// conversion resolves on a microtask before the next), but pinned here: when
|
||||||
|
// both resolve back-to-back, the second sees the changed doc and inserts at
|
||||||
|
// the live selection the first left — so the two payloads may INTERLEAVE, but
|
||||||
|
// neither is dropped. We assert no data loss, not contiguity.
|
||||||
|
const editor = makeEditor();
|
||||||
|
paste(editor, "alphaword");
|
||||||
|
paste(editor, "betaword");
|
||||||
|
await flush();
|
||||||
|
const text = editor.getText();
|
||||||
|
// Neither payload fully dropped (interleaving may split one of them).
|
||||||
|
expect(text).toContain("alpha");
|
||||||
|
expect(text).toContain("beta");
|
||||||
|
editor.destroy();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1,5 +1,12 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
import { describe, it, expect } from "vitest";
|
||||||
import { htmlToMarkdown } from "@docmost/editor-ext";
|
// Markdown conversion now goes through the canonical package's BROWSER entry
|
||||||
|
// (issue #347): the same converter the server import/export uses, resolved via
|
||||||
|
// the `browser` exports condition so it runs on the native `DOMParser` (the
|
||||||
|
// client jsdom vitest env provides one) with jsdom never bundled.
|
||||||
|
import {
|
||||||
|
convertProseMirrorToMarkdown,
|
||||||
|
markdownToProseMirrorSync,
|
||||||
|
} from "@docmost/prosemirror-markdown/browser";
|
||||||
import {
|
import {
|
||||||
normalizeTableColumnWidths,
|
normalizeTableColumnWidths,
|
||||||
classifyClipboardSelection,
|
classifyClipboardSelection,
|
||||||
@@ -175,10 +182,13 @@ describe("classifyClipboardSelection", () => {
|
|||||||
|
|
||||||
// Output-level tests for the table clipboard regression: copying a table must
|
// Output-level tests for the table clipboard regression: copying a table must
|
||||||
// yield a real GFM pipe table, NOT one-value-per-line concatenated cells.
|
// yield a real GFM pipe table, NOT one-value-per-line concatenated cells.
|
||||||
// These exercise the actual markdown produced by htmlToMarkdown (the same
|
// These exercise the actual markdown produced by convertProseMirrorToMarkdown —
|
||||||
// serializer step the clipboardTextSerializer runs), so they pin the OUTPUT
|
// the same serializer step the clipboardTextSerializer now runs (issue #347) —
|
||||||
// shape that the classifier-flag tests above do not cover.
|
// so they pin the OUTPUT shape that the classifier-flag tests above do not cover.
|
||||||
describe("table clipboard markdown output (htmlToMarkdown)", () => {
|
// Input is ProseMirror JSON (what the copied slice serializes to), matching the
|
||||||
|
// clipboardTextSerializer's new call: it wraps the slice content in a synthetic
|
||||||
|
// `doc` (and the bare-rows case in a `table`) and calls the converter.
|
||||||
|
describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () => {
|
||||||
// Trim each line and drop blanks so structural assertions are whitespace-robust.
|
// Trim each line and drop blanks so structural assertions are whitespace-robust.
|
||||||
function lines(md: string): string[] {
|
function lines(md: string): string[] {
|
||||||
return md
|
return md
|
||||||
@@ -188,10 +198,10 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// A GFM separator row like "| --- | --- |" (any number of columns), tolerant
|
// A GFM separator row like "| --- | --- |" (any number of columns), tolerant
|
||||||
// of the padding turndown emits.
|
// of the padding the serializer emits.
|
||||||
function isSeparatorRow(line: string): boolean {
|
function isSeparatorRow(line: string): boolean {
|
||||||
const compact = line.replace(/\s+/g, "");
|
const compact = line.replace(/\s+/g, "");
|
||||||
return /^\|(?:-{3,}\|)+$/.test(compact);
|
return /^\|(?::?-{2,}:?\|)+$/.test(compact);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Split a pipe-delimited row into trimmed cell values.
|
// Split a pipe-delimited row into trimmed cell values.
|
||||||
@@ -203,42 +213,33 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
|
|||||||
.map((c) => c.trim());
|
.map((c) => c.trim());
|
||||||
}
|
}
|
||||||
|
|
||||||
it("serializes a header-less partial cell selection (bare rows) as a valid GFM pipe table", () => {
|
const cell = (t: string) => ({
|
||||||
// Mirror the serializer's `wrapBareRows` branch exactly: bare <tr> nodes are
|
type: "tableCell",
|
||||||
// wrapped in <table><tbody> and htmlToMarkdown(div.innerHTML) is called.
|
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
|
||||||
// See markdown-clipboard.ts clipboardTextSerializer:
|
});
|
||||||
// const table = document.createElement("table");
|
const headerCell = (t: string) => ({
|
||||||
// const tbody = document.createElement("tbody");
|
type: "tableHeader",
|
||||||
// tbody.appendChild(fragment); table.appendChild(tbody);
|
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
|
||||||
// div.appendChild(table);
|
});
|
||||||
// return htmlToMarkdown(div.innerHTML);
|
const row = (nodes: any[]) => ({ type: "tableRow", content: nodes });
|
||||||
const div = document.createElement("div");
|
|
||||||
const table = document.createElement("table");
|
|
||||||
const tbody = document.createElement("tbody");
|
|
||||||
for (const [c1, c2] of [
|
|
||||||
["a", "b"],
|
|
||||||
["c", "d"],
|
|
||||||
]) {
|
|
||||||
const tr = document.createElement("tr");
|
|
||||||
const td1 = document.createElement("td");
|
|
||||||
td1.textContent = c1;
|
|
||||||
const td2 = document.createElement("td");
|
|
||||||
td2.textContent = c2;
|
|
||||||
tr.appendChild(td1);
|
|
||||||
tr.appendChild(td2);
|
|
||||||
tbody.appendChild(tr);
|
|
||||||
}
|
|
||||||
table.appendChild(tbody);
|
|
||||||
div.appendChild(table);
|
|
||||||
|
|
||||||
const md = htmlToMarkdown(div.innerHTML);
|
it("serializes a header-less partial cell selection (bare rows) as a valid GFM pipe table", () => {
|
||||||
|
// Mirror the serializer's `wrapBareRows` branch: bare tableRow nodes are
|
||||||
|
// wrapped in a synthetic `table` and convertProseMirrorToMarkdown is called
|
||||||
|
// (see markdown-clipboard.ts clipboardTextSerializer).
|
||||||
|
const rows = [
|
||||||
|
row([cell("a"), cell("b")]),
|
||||||
|
row([cell("c"), cell("d")]),
|
||||||
|
];
|
||||||
|
const md = convertProseMirrorToMarkdown({
|
||||||
|
type: "doc",
|
||||||
|
content: [{ type: "table", content: rows }],
|
||||||
|
});
|
||||||
const ls = lines(md);
|
const ls = lines(md);
|
||||||
|
|
||||||
// Valid GFM: a header/data separator row is present (an empty header is
|
// Valid GFM: a header/data separator row is present.
|
||||||
// synthesized by the GFM turndown plugin for a header-less table — fine).
|
|
||||||
expect(ls.some(isSeparatorRow)).toBe(true);
|
expect(ls.some(isSeparatorRow)).toBe(true);
|
||||||
// NOT the old broken "one value per line" shape: every line is pipe-delimited
|
// NOT the old broken "one value per line" shape: every line is pipe-delimited.
|
||||||
// and no line is a bare cell value on its own.
|
|
||||||
expect(ls.every((l) => l.includes("|"))).toBe(true);
|
expect(ls.every((l) => l.includes("|"))).toBe(true);
|
||||||
expect(md).not.toMatch(/^\s*(a|b|c|d)\s*$/m);
|
expect(md).not.toMatch(/^\s*(a|b|c|d)\s*$/m);
|
||||||
// The cell values land in real pipe-delimited data rows.
|
// The cell values land in real pipe-delimited data rows.
|
||||||
@@ -248,39 +249,21 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it("serializes a whole table with a header row as a proper GFM table (headline regression)", () => {
|
it("serializes a whole table with a header row as a proper GFM table (headline regression)", () => {
|
||||||
// Mirror the serializer's non-wrap branch: the full <table> node is appended
|
// Mirror the serializer's non-wrap branch: the full `table` node is the
|
||||||
// directly (div.appendChild(fragment)) and htmlToMarkdown(div.innerHTML) runs.
|
// slice content and convertProseMirrorToMarkdown runs on it.
|
||||||
const div = document.createElement("div");
|
const md = convertProseMirrorToMarkdown({
|
||||||
const table = document.createElement("table");
|
type: "doc",
|
||||||
|
content: [
|
||||||
const thead = document.createElement("thead");
|
{
|
||||||
const headerRow = document.createElement("tr");
|
type: "table",
|
||||||
for (const h of ["Name", "Age"]) {
|
content: [
|
||||||
const th = document.createElement("th");
|
row([headerCell("Name"), headerCell("Age")]),
|
||||||
th.textContent = h;
|
row([cell("Alice"), cell("30")]),
|
||||||
headerRow.appendChild(th);
|
row([cell("Bob"), cell("25")]),
|
||||||
}
|
],
|
||||||
thead.appendChild(headerRow);
|
},
|
||||||
table.appendChild(thead);
|
],
|
||||||
|
});
|
||||||
const tbody = document.createElement("tbody");
|
|
||||||
for (const [name, age] of [
|
|
||||||
["Alice", "30"],
|
|
||||||
["Bob", "25"],
|
|
||||||
]) {
|
|
||||||
const tr = document.createElement("tr");
|
|
||||||
const td1 = document.createElement("td");
|
|
||||||
td1.textContent = name;
|
|
||||||
const td2 = document.createElement("td");
|
|
||||||
td2.textContent = age;
|
|
||||||
tr.appendChild(td1);
|
|
||||||
tr.appendChild(td2);
|
|
||||||
tbody.appendChild(tr);
|
|
||||||
}
|
|
||||||
table.appendChild(tbody);
|
|
||||||
div.appendChild(table);
|
|
||||||
|
|
||||||
const md = htmlToMarkdown(div.innerHTML);
|
|
||||||
const ls = lines(md);
|
const ls = lines(md);
|
||||||
|
|
||||||
// Proper GFM structure: separator row + all rows pipe-delimited.
|
// Proper GFM structure: separator row + all rows pipe-delimited.
|
||||||
@@ -296,3 +279,146 @@ describe("table clipboard markdown output (htmlToMarkdown)", () => {
|
|||||||
expect(md).not.toMatch(/^\s*(Name|Age|Alice|Bob|30|25)\s*$/m);
|
expect(md).not.toMatch(/^\s*(Name|Age|Alice|Bob|30|25)\s*$/m);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// #347 acceptance: pasting CANONICAL markdown yields the SAME nodes the server
|
||||||
|
// import produces for the same text. The paste path calls markdownToProseMirror
|
||||||
|
// (the package browser entry) — the identical converter the server import uses —
|
||||||
|
// so asserting the converter (via the browser entry, on the native DOMParser)
|
||||||
|
// recognizes each canon form pins the paste-parity guarantee. These forms were
|
||||||
|
// NOT recognized by the old editor-ext marked layer the paste used before.
|
||||||
|
describe("canonical markdown paste recognition (browser entry parity)", () => {
|
||||||
|
// Collect every node type present in a doc (recursively).
|
||||||
|
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
|
||||||
|
if (!n || typeof n !== "object") return set;
|
||||||
|
if (n.type) set.add(n.type);
|
||||||
|
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
|
||||||
|
return set;
|
||||||
|
};
|
||||||
|
const findNode = (n: any, type: string): any => {
|
||||||
|
if (!n || typeof n !== "object") return undefined;
|
||||||
|
if (n.type === type) return n;
|
||||||
|
if (Array.isArray(n.content)) {
|
||||||
|
for (const c of n.content) {
|
||||||
|
const hit = findNode(c, type);
|
||||||
|
if (hit) return hit;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
};
|
||||||
|
const allText = (n: any): string => {
|
||||||
|
if (!n || typeof n !== "object") return "";
|
||||||
|
if (typeof n.text === "string") return n.text;
|
||||||
|
if (Array.isArray(n.content)) return n.content.map(allText).join("");
|
||||||
|
return "";
|
||||||
|
};
|
||||||
|
|
||||||
|
it("^[…] inline footnote -> footnoteReference + footnotesList", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("Body^[a note here].");
|
||||||
|
const types = collectTypes(doc);
|
||||||
|
expect(types.has("footnoteReference")).toBe(true);
|
||||||
|
expect(types.has("footnotesList")).toBe(true);
|
||||||
|
expect(types.has("footnoteDefinition")).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('<!--img {…}--> attached image comment -> image with align', () => {
|
||||||
|
const doc = markdownToProseMirrorSync(
|
||||||
|
' <!--img {"align":"left"}-->',
|
||||||
|
);
|
||||||
|
const img = findNode(doc, "image");
|
||||||
|
expect(img).toBeTruthy();
|
||||||
|
expect(img.attrs?.align).toBe("left");
|
||||||
|
expect(img.attrs?.src).toBe("/files/x.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("> [!type] Obsidian callout -> callout node with type", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("> [!warning]\n> be careful");
|
||||||
|
const callout = findNode(doc, "callout");
|
||||||
|
expect(callout).toBeTruthy();
|
||||||
|
expect(callout.attrs?.type).toBe("warning");
|
||||||
|
expect(allText(callout)).toContain("be careful");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("$…$ inline math -> mathInline node", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("Euler: $e^{i\\pi}+1=0$ done");
|
||||||
|
const math = findNode(doc, "mathInline");
|
||||||
|
expect(math).toBeTruthy();
|
||||||
|
expect(math.attrs?.text).toContain("e^{i\\pi}");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("==…== highlight -> highlight mark", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("A ==marked== word");
|
||||||
|
const marked = findNode(doc, "text");
|
||||||
|
// The highlighted run carries a `highlight` mark somewhere in the doc.
|
||||||
|
const hasHighlight = (n: any): boolean => {
|
||||||
|
if (!n || typeof n !== "object") return false;
|
||||||
|
if (
|
||||||
|
n.type === "text" &&
|
||||||
|
(n.marks || []).some((m: any) => m.type === "highlight")
|
||||||
|
)
|
||||||
|
return true;
|
||||||
|
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
|
||||||
|
};
|
||||||
|
expect(marked).toBeTruthy();
|
||||||
|
expect(hasHighlight(doc)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("<!--subpages--> standalone comment -> subpages node", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("intro\n\n<!--subpages-->\n\nafter");
|
||||||
|
expect(collectTypes(doc).has("subpages")).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// #347 negatives: plain text carrying markdown-LIKE punctuation must NOT be
|
||||||
|
// silently converted/mangled (currency, bare `==`, a `[^1]` reference form).
|
||||||
|
describe("plain-text paste negatives (no phantom conversion)", () => {
|
||||||
|
const findNode = (n: any, type: string): any => {
|
||||||
|
if (!n || typeof n !== "object") return undefined;
|
||||||
|
if (n.type === type) return n;
|
||||||
|
if (Array.isArray(n.content)) {
|
||||||
|
for (const c of n.content) {
|
||||||
|
const hit = findNode(c, type);
|
||||||
|
if (hit) return hit;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
};
|
||||||
|
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
|
||||||
|
if (!n || typeof n !== "object") return set;
|
||||||
|
if (n.type) set.add(n.type);
|
||||||
|
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
|
||||||
|
return set;
|
||||||
|
};
|
||||||
|
const allText = (n: any): string => {
|
||||||
|
if (!n || typeof n !== "object") return "";
|
||||||
|
if (typeof n.text === "string") return n.text;
|
||||||
|
if (Array.isArray(n.content)) return n.content.map(allText).join("");
|
||||||
|
return "";
|
||||||
|
};
|
||||||
|
|
||||||
|
it("currency `$5 and $10` is NOT turned into math", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("It costs $5 and $10 total");
|
||||||
|
expect(findNode(doc, "mathInline")).toBeFalsy();
|
||||||
|
expect(allText(doc)).toContain("$5 and $10");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a lone `==` is NOT turned into a highlight", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("compare a == b in code");
|
||||||
|
const hasHighlight = (n: any): boolean => {
|
||||||
|
if (!n || typeof n !== "object") return false;
|
||||||
|
if (
|
||||||
|
n.type === "text" &&
|
||||||
|
(n.marks || []).some((m: any) => m.type === "highlight")
|
||||||
|
)
|
||||||
|
return true;
|
||||||
|
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
|
||||||
|
};
|
||||||
|
expect(hasHighlight(doc)).toBe(false);
|
||||||
|
expect(allText(doc)).toContain("== b");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a `[^1]` reference form (no `^[`) is NOT turned into a footnote", () => {
|
||||||
|
const doc = markdownToProseMirrorSync("see note [^1] for details");
|
||||||
|
expect(collectTypes(doc).has("footnoteReference")).toBe(false);
|
||||||
|
expect(allText(doc)).toContain("[^1]");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,15 +1,23 @@
|
|||||||
// adapted from: https://github.com/aguingand/tiptap-markdown/blob/main/src/extensions/tiptap/clipboard.js - MIT
|
// adapted from: https://github.com/aguingand/tiptap-markdown/blob/main/src/extensions/tiptap/clipboard.js - MIT
|
||||||
import { Extension } from "@tiptap/core";
|
import { Extension } from "@tiptap/core";
|
||||||
import { Plugin, PluginKey, TextSelection } from "@tiptap/pm/state";
|
import { Plugin, PluginKey, TextSelection } from "@tiptap/pm/state";
|
||||||
import { DOMParser, DOMSerializer, Fragment, Slice } from "@tiptap/pm/model";
|
import { DOMParser, DOMSerializer, Fragment, Slice, Node as PMNode } from "@tiptap/pm/model";
|
||||||
import { find } from "linkifyjs";
|
import { find } from "linkifyjs";
|
||||||
import {
|
import {
|
||||||
markdownToHtml,
|
|
||||||
htmlToMarkdown,
|
|
||||||
canonicalizeFootnotes,
|
canonicalizeFootnotes,
|
||||||
FOOTNOTES_LIST_NAME,
|
FOOTNOTES_LIST_NAME,
|
||||||
FOOTNOTE_REFERENCE_NAME,
|
FOOTNOTE_REFERENCE_NAME,
|
||||||
} from "@docmost/editor-ext";
|
} from "@docmost/editor-ext";
|
||||||
|
// Markdown <-> ProseMirror conversion now lives ONLY in the canonical
|
||||||
|
// `@docmost/prosemirror-markdown` package (issue #347). The BROWSER entry uses
|
||||||
|
// the native `DOMParser` for its HTML->DOM stage (jsdom stays out of the client
|
||||||
|
// bundle) while producing the SAME nodes the server import does — so a paste of
|
||||||
|
// canonical markdown (`^[…]`, `<!--img …-->`, `> [!type]`, `$…$`, `==…==`,
|
||||||
|
// standalone comments) is recognized identically to import.
|
||||||
|
import {
|
||||||
|
markdownToProseMirror,
|
||||||
|
convertProseMirrorToMarkdown,
|
||||||
|
} from "@docmost/prosemirror-markdown/browser";
|
||||||
import type { Schema } from "@tiptap/pm/model";
|
import type { Schema } from "@tiptap/pm/model";
|
||||||
|
|
||||||
export const MarkdownClipboard = Extension.create({
|
export const MarkdownClipboard = Extension.create({
|
||||||
@@ -39,25 +47,24 @@ export const MarkdownClipboard = Extension.create({
|
|||||||
classifyClipboardSelection(topLevelNodes);
|
classifyClipboardSelection(topLevelNodes);
|
||||||
if (!asMarkdown) return null;
|
if (!asMarkdown) return null;
|
||||||
|
|
||||||
const div = document.createElement("div");
|
// Convert the copied selection to Markdown through the canonical
|
||||||
const serializer = DOMSerializer.fromSchema(this.editor.schema);
|
// package (issue #347), the SAME serializer the server export uses,
|
||||||
const fragment = serializer.serializeFragment(slice.content);
|
// so a copied table/list matches the on-disk markdown form. The
|
||||||
|
// converter takes a ProseMirror `doc` JSON, so wrap the slice's
|
||||||
|
// top-level content in a synthetic doc.
|
||||||
|
const content = slice.content.toJSON() as any[];
|
||||||
if (wrapBareRows) {
|
if (wrapBareRows) {
|
||||||
// A partial table cell-selection serializes to bare <tr> nodes
|
// A partial table cell-selection serializes to bare `tableRow`
|
||||||
// (prosemirror-tables returns the whole `table` node only when the
|
// nodes (prosemirror-tables yields the whole `table` node only for
|
||||||
// entire table is selected). Bare <tr> would be foster-parented
|
// a full-table selection). The converter's table case expects a
|
||||||
// away by the HTML parser inside htmlToMarkdown, so wrap them in
|
// `table` wrapper, so wrap the bare rows in one — mirroring the old
|
||||||
// <table><tbody> first for the GFM turndown rule to detect them.
|
// <table><tbody> wrap that the HTML->markdown step needed.
|
||||||
const table = document.createElement("table");
|
return convertProseMirrorToMarkdown({
|
||||||
const tbody = document.createElement("tbody");
|
type: "doc",
|
||||||
tbody.appendChild(fragment);
|
content: [{ type: "table", content }],
|
||||||
table.appendChild(tbody);
|
});
|
||||||
div.appendChild(table);
|
|
||||||
} else {
|
|
||||||
div.appendChild(fragment);
|
|
||||||
}
|
}
|
||||||
return htmlToMarkdown(div.innerHTML);
|
return convertProseMirrorToMarkdown({ type: "doc", content });
|
||||||
},
|
},
|
||||||
handlePaste: (view, event, slice) => {
|
handlePaste: (view, event, slice) => {
|
||||||
if (!event.clipboardData) {
|
if (!event.clipboardData) {
|
||||||
@@ -95,37 +102,115 @@ export const MarkdownClipboard = Extension.create({
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const { tr } = view.state;
|
const schema = this.editor.schema;
|
||||||
const { from, to } = view.state.selection;
|
// Capture the target range NOW. markdownToProseMirror RETURNS A
|
||||||
|
// PROMISE (kept async only for the Node consumers' contract; the
|
||||||
|
// conversion pipeline itself is synchronous), so the actual replace
|
||||||
|
// happens on the next microtask. No user input can interleave a
|
||||||
|
// microtask, so the state is unchanged when we dispatch — but we
|
||||||
|
// still re-read the live state before replacing and, if the doc did
|
||||||
|
// change under us, fall back to the live selection rather than the
|
||||||
|
// captured (now-stale) range.
|
||||||
|
const from = view.state.selection.from;
|
||||||
|
const to = view.state.selection.to;
|
||||||
|
const startDoc = view.state.doc;
|
||||||
|
const md = text.replace(/\n+$/, "");
|
||||||
|
|
||||||
const parsed = markdownToHtml(text.replace(/\n+$/, ""));
|
void markdownToProseMirror(md)
|
||||||
const body = elementFromString(parsed);
|
.then((doc) => {
|
||||||
normalizeTableColumnWidths(body);
|
if (view.isDestroyed) return;
|
||||||
|
// Canonical PM-JSON -> HTML via the LIVE editor schema, then
|
||||||
|
// reuse the UNCHANGED downstream seam (normalizeTableColumnWidths
|
||||||
|
// + parseSlice + canonicalizePastedFootnotes). The JSON->HTML->
|
||||||
|
// JSON hop is lossless (same schema both directions); it lets the
|
||||||
|
// existing paste-insertion logic stay byte-identical — only the
|
||||||
|
// SOURCE of the markdown conversion changed (issue #347 guardrail:
|
||||||
|
// no converter logic in the client, only a call into the package).
|
||||||
|
const node = PMNode.fromJSON(schema, doc);
|
||||||
|
const div = document.createElement("div");
|
||||||
|
DOMSerializer.fromSchema(schema).serializeFragment(
|
||||||
|
node.content,
|
||||||
|
{ document },
|
||||||
|
div,
|
||||||
|
);
|
||||||
|
|
||||||
const parsedSlice = DOMParser.fromSchema(
|
const body = elementFromString(div.innerHTML);
|
||||||
this.editor.schema,
|
normalizeTableColumnWidths(body);
|
||||||
).parseSlice(body, {
|
|
||||||
preserveWhitespace: true,
|
|
||||||
});
|
|
||||||
|
|
||||||
// A markdown paste builds its ProseMirror fragment directly (DOM ->
|
const parsedSlice = DOMParser.fromSchema(schema).parseSlice(
|
||||||
// parseSlice), bypassing the editor's footnoteSyncPlugin, which never
|
body,
|
||||||
// reorders an existing list. So a pasted markdown block whose footnote
|
{ preserveWhitespace: true },
|
||||||
// definitions are out of order (or contains orphan defs) would be
|
);
|
||||||
// stored out of order. Canonicalize the self-contained pasted block so
|
|
||||||
// its footnotes come out reference-ordered, deduped and orphan-free
|
|
||||||
// (issue #228). See canonicalizePastedFootnotes for why this is scoped
|
|
||||||
// to whole-block pastes that carry their own footnotesList.
|
|
||||||
const contentNodes = canonicalizePastedFootnotes(
|
|
||||||
parsedSlice,
|
|
||||||
this.editor.schema,
|
|
||||||
);
|
|
||||||
|
|
||||||
tr.replaceRange(from, to, contentNodes);
|
// A markdown paste builds its ProseMirror fragment directly (DOM
|
||||||
const insertEnd = tr.mapping.map(from, 1);
|
// -> parseSlice), bypassing the editor's footnoteSyncPlugin, which
|
||||||
tr.setSelection(TextSelection.near(tr.doc.resolve(Math.max(from, insertEnd - 2)), -1));
|
// never reorders an existing list. So a pasted markdown block whose
|
||||||
tr.setMeta('paste', true)
|
// footnote definitions are out of order (or contains orphan defs)
|
||||||
view.dispatch(tr);
|
// would be stored out of order. Canonicalize the self-contained
|
||||||
|
// pasted block so its footnotes come out reference-ordered, deduped
|
||||||
|
// and orphan-free (issue #228). See canonicalizePastedFootnotes for
|
||||||
|
// why this is scoped to whole-block pastes that carry their own
|
||||||
|
// footnotesList.
|
||||||
|
const contentNodes = canonicalizePastedFootnotes(
|
||||||
|
parsedSlice,
|
||||||
|
schema,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Target the captured range (normally still valid — same
|
||||||
|
// microtask). If the doc changed under us since capture, the
|
||||||
|
// captured absolute from/to are stale, so fall back to the live
|
||||||
|
// selection rather than StepMap-mapping the old range.
|
||||||
|
const tr = view.state.tr;
|
||||||
|
let mappedFrom = from;
|
||||||
|
let mappedTo = to;
|
||||||
|
if (view.state.doc !== startDoc) {
|
||||||
|
// Defensive: if the doc changed under us, fall back to the
|
||||||
|
// current selection rather than a stale absolute range.
|
||||||
|
mappedFrom = view.state.selection.from;
|
||||||
|
mappedTo = view.state.selection.to;
|
||||||
|
}
|
||||||
|
tr.replaceRange(mappedFrom, mappedTo, contentNodes);
|
||||||
|
const insertEnd = tr.mapping.map(mappedFrom, 1);
|
||||||
|
tr.setSelection(
|
||||||
|
TextSelection.near(
|
||||||
|
tr.doc.resolve(Math.max(mappedFrom, insertEnd - 2)),
|
||||||
|
-1,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
tr.setMeta("paste", true);
|
||||||
|
view.dispatch(tr);
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
// Fail-open: a conversion error must not swallow the paste
|
||||||
|
// silently in a way that loses the text. We already claimed the
|
||||||
|
// event (returned true), so re-insert the raw text as a plain
|
||||||
|
// paragraph so the user never loses their clipboard content.
|
||||||
|
// Log it: this catch covers BOTH the converter and the success
|
||||||
|
// `.then` body (e.g. PMNode.fromJSON throwing on a schema drift
|
||||||
|
// between the canonical package and the live editor schema), so a
|
||||||
|
// silent degrade to raw text would otherwise be an invisible,
|
||||||
|
// non-reproducible regression ("my table pasted as text").
|
||||||
|
console.error(
|
||||||
|
"markdown paste conversion failed, inserting raw text",
|
||||||
|
err,
|
||||||
|
);
|
||||||
|
if (view.isDestroyed) return;
|
||||||
|
const tr = view.state.tr;
|
||||||
|
// Same guard the success path uses: if the doc changed under us
|
||||||
|
// since the range was captured (normally never — same microtask),
|
||||||
|
// the captured absolute from/to are stale and would throw a
|
||||||
|
// RangeError here (an unhandled rejection on a hot paste path).
|
||||||
|
// Fall back to the live selection instead of a stale range.
|
||||||
|
if (view.state.doc !== startDoc) {
|
||||||
|
const sel = view.state.selection;
|
||||||
|
tr.insertText(md, sel.from, sel.to);
|
||||||
|
} else {
|
||||||
|
tr.insertText(md, from, to);
|
||||||
|
}
|
||||||
|
tr.setMeta("paste", true);
|
||||||
|
view.dispatch(tr);
|
||||||
|
});
|
||||||
|
// Claim the paste: we insert asynchronously above.
|
||||||
return true;
|
return true;
|
||||||
},
|
},
|
||||||
// Strip trailing whitespace-only paragraphs from pasted content.
|
// Strip trailing whitespace-only paragraphs from pasted content.
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import { Italic } from "@tiptap/extension-italic";
|
|||||||
import { Link } from "@tiptap/extension-link";
|
import { Link } from "@tiptap/extension-link";
|
||||||
import { gitmostInsertTranscriptIntoEditor } from "./gitmost-recording.ts";
|
import { gitmostInsertTranscriptIntoEditor } from "./gitmost-recording.ts";
|
||||||
|
|
||||||
const ZWSP = ""; // U+200B, the helper's block-trigger neutralizer
|
const ZWSP = ""; // U+200B — asserted ABSENT (the block-escape lives in the serializer now)
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* #377 — the web-side bridge must append the native host's transcript below the
|
* #377 — the web-side bridge must append the native host's transcript below the
|
||||||
@@ -18,8 +18,9 @@ const ZWSP = ""; // U+200B, the helper's block-trigger neutralizer
|
|||||||
* regression would be caught), asserting the resulting document rather than
|
* regression would be caught), asserting the resulting document rather than
|
||||||
* mocking the editor: transcript present -> "Transcript" heading + one paragraph
|
* mocking the editor: transcript present -> "Transcript" heading + one paragraph
|
||||||
* per non-empty line; content is inserted as LITERAL TEXT (no HTML/markdown
|
* per non-empty line; content is inserted as LITERAL TEXT (no HTML/markdown
|
||||||
* parsing); col-0 markdown block triggers are neutralized so git-sync keeps them
|
* parsing); col-0 markdown block triggers are stored verbatim (the git-sync
|
||||||
* paragraphs; absent/empty/non-string -> no-op.
|
* serializer block-escapes them, so no client-side ZWSP is needed);
|
||||||
|
* absent/empty/non-string -> no-op.
|
||||||
*/
|
*/
|
||||||
describe("gitmostInsertTranscriptIntoEditor", () => {
|
describe("gitmostInsertTranscriptIntoEditor", () => {
|
||||||
const makeEditor = () =>
|
const makeEditor = () =>
|
||||||
@@ -91,19 +92,22 @@ describe("gitmostInsertTranscriptIntoEditor", () => {
|
|||||||
editor.destroy();
|
editor.destroy();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("neutralizes col-0 markdown block triggers with a leading ZWSP (git-sync safety)", () => {
|
it("inserts col-0 markdown block triggers as verbatim paragraph text (no ZWSP workaround)", () => {
|
||||||
const editor = makeEditor();
|
const editor = makeEditor();
|
||||||
// Trigger lines (some with a leaked indent) + a normal prefixed line.
|
// Trigger lines (some with a leaked indent) + a normal prefixed line. The
|
||||||
|
// git-sync serializer now block-escapes a leading trigger itself, so the
|
||||||
|
// bridge inserts each line's TEXT byte-exact (only the leaked indent is
|
||||||
|
// trimmed) — no invisible ZWSP is prepended anymore.
|
||||||
const inserted = gitmostInsertTranscriptIntoEditor(
|
const inserted = gitmostInsertTranscriptIntoEditor(
|
||||||
editor,
|
editor,
|
||||||
[
|
[
|
||||||
"- dash",
|
"- dash",
|
||||||
" > quote", // leading indent must be trimmed then neutralized
|
" > quote", // leading indent is trimmed, text otherwise verbatim
|
||||||
"# hash",
|
"# hash",
|
||||||
"1. one",
|
"1. one",
|
||||||
"> [!info] note",
|
"> [!info] note",
|
||||||
"```js",
|
"```js",
|
||||||
"---", // solid thematic break -> horizontalRule (text-losing) if unneutralized
|
"---",
|
||||||
"***",
|
"***",
|
||||||
"___",
|
"___",
|
||||||
"You: normal line",
|
"You: normal line",
|
||||||
@@ -116,20 +120,23 @@ describe("gitmostInsertTranscriptIntoEditor", () => {
|
|||||||
.map((n: any) => n.content?.[0]?.text)
|
.map((n: any) => n.content?.[0]?.text)
|
||||||
.filter((t: any) => typeof t === "string") as string[];
|
.filter((t: any) => typeof t === "string") as string[];
|
||||||
|
|
||||||
// Every block-trigger line is prefixed with the invisible ZWSP (indent
|
// Each trigger line is stored as its own byte-exact text (indent trimmed);
|
||||||
// trimmed first); the normal `You:` line is left byte-exact.
|
// the git-sync round-trip keeps it a paragraph via the serializer's
|
||||||
|
// block-escape, so no ZWSP is needed here.
|
||||||
expect(texts).toEqual([
|
expect(texts).toEqual([
|
||||||
ZWSP + "- dash",
|
"- dash",
|
||||||
ZWSP + "> quote",
|
"> quote",
|
||||||
ZWSP + "# hash",
|
"# hash",
|
||||||
ZWSP + "1. one",
|
"1. one",
|
||||||
ZWSP + "> [!info] note",
|
"> [!info] note",
|
||||||
ZWSP + "```js",
|
"```js",
|
||||||
ZWSP + "---",
|
"---",
|
||||||
ZWSP + "***",
|
"***",
|
||||||
ZWSP + "___",
|
"___",
|
||||||
"You: normal line",
|
"You: normal line",
|
||||||
]);
|
]);
|
||||||
|
// Guard: no invisible ZWSP leaked into any inserted line.
|
||||||
|
for (const t of texts) expect(t).not.toContain(ZWSP);
|
||||||
|
|
||||||
editor.destroy();
|
editor.destroy();
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -240,45 +240,22 @@ export async function gitmostUploadFileToEditor(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Zero-width space (U+200B). Prepended to a transcript line that begins with a
|
|
||||||
// markdown BLOCK trigger: it is invisible in the rendered doc but shifts the
|
|
||||||
// trigger off column 0, so the git-sync doc->markdown->doc round-trip keeps the
|
|
||||||
// line a plain paragraph (see GITMOST_MD_BLOCK_TRIGGER_RE).
|
|
||||||
const GITMOST_ZWSP = "";
|
|
||||||
|
|
||||||
// A markdown BLOCK-level construct that, sitting at column 0 of a paragraph
|
|
||||||
// line, the git-sync markdown serializer (packages/prosemirror-markdown
|
|
||||||
// markdown-converter.ts, `case "paragraph"`) would re-parse into a NON-paragraph
|
|
||||||
// block on the doc->markdown->doc cycle. That serializer emits paragraph text
|
|
||||||
// verbatim with NO block-escape (the pre-existing root cause), so a leading
|
|
||||||
// `#`/`-`/`*`/`+`/`>`, an ordered-list `N.`/`N)`, a code fence ```/~~~, a table
|
|
||||||
// `|`, or a `> [!info]` callout opener would silently become a heading / list /
|
|
||||||
// quote / code block / table / callout. The final alternative matches a WHOLE-
|
|
||||||
// LINE thematic break — solid `---`/`***`/`___` or spaced `- - -`/`_ _ _` (3+ of
|
|
||||||
// the same `-`/`*`/`_`) — which round-trips into a `horizontalRule`; because
|
|
||||||
// that node carries NO text, an un-neutralized separator line would LOSE its
|
|
||||||
// text entirely (worse than the list/quote case). This matches a TRIMMED line's
|
|
||||||
// start; the transcript's own `You:` / `Speaker N:` prefix begins with a letter
|
|
||||||
// and never matches, so prefixed lines are left byte-exact.
|
|
||||||
const GITMOST_MD_BLOCK_TRIGGER_RE =
|
|
||||||
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
|
|
||||||
|
|
||||||
// Append a transcript block BELOW the recording's audio node in a live editor:
|
// Append a transcript block BELOW the recording's audio node in a live editor:
|
||||||
// a "Transcript" heading followed by one paragraph per non-empty transcript
|
// a "Transcript" heading followed by one paragraph per non-empty transcript
|
||||||
// line. The transcript is plain text, `\n`-separated, each line already
|
// line. The transcript is plain text, `\n`-separated, each line already
|
||||||
// formatted as `You: ...` / `Speaker N: ...` by the native host — line text is
|
// formatted as `You: ...` / `Speaker N: ...` by the native host — line text is
|
||||||
// inserted as a TEXT node (never HTML/markdown), so there is no injection or
|
// inserted as a TEXT node (never HTML/markdown), so there is no injection or
|
||||||
// mark-parsing surface. Each kept line is trimmed (drops an indent that would
|
// mark-parsing surface. Each kept line is trimmed (drops an indent that would
|
||||||
// both leak into the display and, at col 0, form a markdown block trigger) and,
|
// leak into the display). A line that begins with a col-0 markdown block
|
||||||
// if it still begins with a col-0 markdown block trigger, gets an invisible
|
// trigger (`#`/`-`/`>`/`1.`/fence/`---`/…) needs no client-side workaround: the
|
||||||
// zero-width space prepended so the git-sync round-trip cannot turn it into a
|
// git-sync serializer (packages/prosemirror-markdown, `case "paragraph"`) now
|
||||||
// list/quote/heading/callout/code/table (defensive boundary against the
|
// block-escapes such a leading trigger, so the doc->markdown->doc round-trip
|
||||||
// serializer's missing block-escape). This is best-effort and meant to run
|
// keeps the line a paragraph on its own — the former invisible-ZWSP defense is
|
||||||
// AFTER the audio has already been inserted; the caller must guard against a
|
// gone. This is best-effort and meant to run AFTER the audio has already been
|
||||||
// throw so a transcript failure never fails the (already successful) recording.
|
// inserted; the caller must guard against a throw so a transcript failure never
|
||||||
// Returns true when a block was inserted, false when there was nothing to
|
// fails the (already successful) recording. Returns true when a block was
|
||||||
// insert (transcript undefined/empty/not-a-string). A non-string value is a
|
// inserted, false when there was nothing to insert (transcript
|
||||||
// no-op, not an error.
|
// undefined/empty/not-a-string). A non-string value is a no-op, not an error.
|
||||||
export function gitmostInsertTranscriptIntoEditor(
|
export function gitmostInsertTranscriptIntoEditor(
|
||||||
editor: Editor,
|
editor: Editor,
|
||||||
transcript: unknown,
|
transcript: unknown,
|
||||||
@@ -288,13 +265,7 @@ export function gitmostInsertTranscriptIntoEditor(
|
|||||||
.split("\n")
|
.split("\n")
|
||||||
// Trim each line and drop blank (whitespace-only) ones.
|
// Trim each line and drop blank (whitespace-only) ones.
|
||||||
.map((line) => line.trim())
|
.map((line) => line.trim())
|
||||||
.filter((line) => line.length > 0)
|
.filter((line) => line.length > 0);
|
||||||
// Neutralize a col-0 markdown block trigger with an invisible ZWSP so the
|
|
||||||
// git-sync round-trip keeps the line a paragraph. Host lines (`You:` /
|
|
||||||
// `Speaker N:`) never match and stay byte-exact.
|
|
||||||
.map((line) =>
|
|
||||||
GITMOST_MD_BLOCK_TRIGGER_RE.test(line) ? GITMOST_ZWSP + line : line,
|
|
||||||
);
|
|
||||||
if (lines.length === 0) return false;
|
if (lines.length === 0) return false;
|
||||||
|
|
||||||
const content = [
|
const content = [
|
||||||
|
|||||||
@@ -33,10 +33,11 @@ vi.mock("@/lib/local-emitter.ts", () => ({
|
|||||||
default: { emit: (...args: unknown[]) => localEmitMock(...args) },
|
default: { emit: (...args: unknown[]) => localEmitMock(...args) },
|
||||||
}));
|
}));
|
||||||
|
|
||||||
// htmlToMarkdown just echoes the editor HTML so each test controls the markdown
|
// convertProseMirrorToMarkdown echoes a marker carried on the fake editor's
|
||||||
// purely via the fake page editor's getHTML().
|
// getJSON() doc, so each test controls the markdown purely via the fake page
|
||||||
vi.mock("@docmost/editor-ext", () => ({
|
// editor (issue #347: the hook now serializes editor JSON through the package).
|
||||||
htmlToMarkdown: (html: string) => html,
|
vi.mock("@docmost/prosemirror-markdown/browser", () => ({
|
||||||
|
convertProseMirrorToMarkdown: (doc: { __md?: string }) => doc?.__md ?? "",
|
||||||
}));
|
}));
|
||||||
|
|
||||||
const notificationsShowMock = vi.fn();
|
const notificationsShowMock = vi.fn();
|
||||||
@@ -53,10 +54,12 @@ import { useGeneratePageTitle } from "./use-generate-page-title.ts";
|
|||||||
|
|
||||||
// --- Test helpers -------------------------------------------------------------
|
// --- Test helpers -------------------------------------------------------------
|
||||||
|
|
||||||
function makePageEditor(pageId: string, html = "<p>content</p>"): Editor {
|
function makePageEditor(pageId: string, md = "content"): Editor {
|
||||||
return {
|
return {
|
||||||
isDestroyed: false,
|
isDestroyed: false,
|
||||||
getHTML: () => html,
|
// The mocked convertProseMirrorToMarkdown reads `__md` back off this doc,
|
||||||
|
// so `md` is exactly the markdown the hook will send to the title service.
|
||||||
|
getJSON: () => ({ type: "doc", __md: md }),
|
||||||
storage: { pageId },
|
storage: { pageId },
|
||||||
} as unknown as Editor;
|
} as unknown as Editor;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ import { useMutation } from "@tanstack/react-query";
|
|||||||
import { useAtomValue } from "jotai";
|
import { useAtomValue } from "jotai";
|
||||||
import { notifications } from "@mantine/notifications";
|
import { notifications } from "@mantine/notifications";
|
||||||
import { useTranslation } from "react-i18next";
|
import { useTranslation } from "react-i18next";
|
||||||
import { htmlToMarkdown } from "@docmost/editor-ext";
|
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser";
|
||||||
import {
|
import {
|
||||||
pageEditorAtom,
|
pageEditorAtom,
|
||||||
titleEditorAtom,
|
titleEditorAtom,
|
||||||
@@ -49,7 +49,9 @@ export function useGeneratePageTitle(pageId: string) {
|
|||||||
mutationFn: async () => {
|
mutationFn: async () => {
|
||||||
if (!pageEditor || pageEditor.isDestroyed) return;
|
if (!pageEditor || pageEditor.isDestroyed) return;
|
||||||
|
|
||||||
const markdown = htmlToMarkdown(pageEditor.getHTML()).trim();
|
// Serialize the live editor content to markdown through the canonical
|
||||||
|
// converter (issue #347), matching the on-disk/export markdown form.
|
||||||
|
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON()).trim();
|
||||||
if (!markdown) {
|
if (!markdown) {
|
||||||
notifications.show({ message: t("The note is empty"), color: "yellow" });
|
notifications.show({ message: t("The note is empty"), color: "yellow" });
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ import { useTreeMutation } from "@/features/page/tree/hooks/use-tree-mutation.ts
|
|||||||
import { PageWidthToggle } from "@/features/user/components/page-width-pref.tsx";
|
import { PageWidthToggle } from "@/features/user/components/page-width-pref.tsx";
|
||||||
import { Trans, useTranslation } from "react-i18next";
|
import { Trans, useTranslation } from "react-i18next";
|
||||||
import ExportModal from "@/components/common/export-modal";
|
import ExportModal from "@/components/common/export-modal";
|
||||||
import { htmlToMarkdown } from "@docmost/editor-ext";
|
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser";
|
||||||
import {
|
import {
|
||||||
pageEditorAtom,
|
pageEditorAtom,
|
||||||
yjsConnectionStatusAtom,
|
yjsConnectionStatusAtom,
|
||||||
@@ -199,8 +199,9 @@ function PageActionMenu({ readOnly }: PageActionMenuProps) {
|
|||||||
|
|
||||||
const handleCopyAsMarkdown = () => {
|
const handleCopyAsMarkdown = () => {
|
||||||
if (!pageEditor) return;
|
if (!pageEditor) return;
|
||||||
const html = pageEditor.getHTML();
|
// Copy the page as canonical markdown through the shared converter (issue
|
||||||
const markdown = htmlToMarkdown(html);
|
// #347), so "Copy as markdown" matches the server export byte-for-byte.
|
||||||
|
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON());
|
||||||
const title = page?.title ? `# ${page.title}\n\n` : "";
|
const title = page?.title ? `# ${page.title}\n\n` : "";
|
||||||
clipboard.copy(`${title}${markdown}`);
|
clipboard.copy(`${title}${markdown}`);
|
||||||
notifications.show({ message: t("Copied") });
|
notifications.show({ message: t("Copied") });
|
||||||
|
|||||||
@@ -41,6 +41,7 @@
|
|||||||
"@aws-sdk/s3-request-presigner": "3.1050.0",
|
"@aws-sdk/s3-request-presigner": "3.1050.0",
|
||||||
"@azure/storage-blob": "12.31.0",
|
"@azure/storage-blob": "12.31.0",
|
||||||
"@clickhouse/client": "^1.18.2",
|
"@clickhouse/client": "^1.18.2",
|
||||||
|
"@docmost/editor-ext": "workspace:*",
|
||||||
"@docmost/mcp": "workspace:*",
|
"@docmost/mcp": "workspace:*",
|
||||||
"@docmost/pdf-inspector": "1.9.6",
|
"@docmost/pdf-inspector": "1.9.6",
|
||||||
"@docmost/prosemirror-markdown": "workspace:*",
|
"@docmost/prosemirror-markdown": "workspace:*",
|
||||||
|
|||||||
@@ -49,6 +49,7 @@ import {
|
|||||||
FootnotesList,
|
FootnotesList,
|
||||||
FootnoteDefinition,
|
FootnoteDefinition,
|
||||||
PageEmbed,
|
PageEmbed,
|
||||||
|
Code,
|
||||||
} from '@docmost/editor-ext';
|
} from '@docmost/editor-ext';
|
||||||
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
|
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
|
||||||
import { generateText, getSchema, JSONContent } from '@tiptap/core';
|
import { generateText, getSchema, JSONContent } from '@tiptap/core';
|
||||||
@@ -67,7 +68,12 @@ export const tiptapExtensions = [
|
|||||||
link: false,
|
link: false,
|
||||||
trailingNode: false,
|
trailingNode: false,
|
||||||
heading: false,
|
heading: false,
|
||||||
|
// #515: replace StarterKit's bundled inline `code` (which inherits tiptap's
|
||||||
|
// `excludes: "_"`) with the shared editor-ext `Code` below, so the server's
|
||||||
|
// HTML->PM parse/export keeps code co-occurring with other marks.
|
||||||
|
code: false,
|
||||||
}),
|
}),
|
||||||
|
Code,
|
||||||
Heading,
|
Heading,
|
||||||
UniqueID.configure({
|
UniqueID.configure({
|
||||||
types: ['heading', 'paragraph', 'transclusionSource'],
|
types: ['heading', 'paragraph', 'transclusionSource'],
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import { markdownToHtml, encodeHtmlEmbedSource } from '@docmost/editor-ext';
|
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
||||||
|
import { encodeHtmlEmbedSource } from '@docmost/editor-ext';
|
||||||
import { htmlToJson } from '../../../collaboration/collaboration.util';
|
import { htmlToJson } from '../../../collaboration/collaboration.util';
|
||||||
import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
|
import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
|
||||||
|
|
||||||
@@ -10,13 +11,12 @@ import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
|
|||||||
*
|
*
|
||||||
* The block renders inside a sandboxed iframe, so this is not an XSS surface;
|
* The block renders inside a sandboxed iframe, so this is not an XSS surface;
|
||||||
* this exercises the REAL server import conversion path that ImportService uses
|
* this exercises the REAL server import conversion path that ImportService uses
|
||||||
* (`markdownToHtml` then `htmlToJson`; `processHTML` adds only a cheerio
|
* (`markdownToProseMirror`, the canonical converter — issue #345/#347) and
|
||||||
* link/iframe normalize pass which does not touch htmlEmbed divs) and asserts
|
* asserts that such a node is DETECTED and STRIPPABLE — so the share read path's
|
||||||
* that such a node is DETECTED and STRIPPABLE — so the share read path's
|
|
||||||
* master-toggle strip can remove it when the workspace toggle is OFF.
|
* master-toggle strip can remove it when the workspace toggle is OFF.
|
||||||
*/
|
*/
|
||||||
describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTML', () => {
|
describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTML', () => {
|
||||||
it('round-trips through markdownToHtml -> htmlToJson and is DETECTED (base64 data-source)', async () => {
|
it('round-trips through markdownToProseMirror and is DETECTED (base64 data-source)', async () => {
|
||||||
const source = '<script>steal()</script>';
|
const source = '<script>steal()</script>';
|
||||||
const encoded = encodeHtmlEmbedSource(source);
|
const encoded = encodeHtmlEmbedSource(source);
|
||||||
const md = [
|
const md = [
|
||||||
@@ -27,12 +27,9 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
|
|||||||
'World',
|
'World',
|
||||||
].join('\n');
|
].join('\n');
|
||||||
|
|
||||||
const html = await markdownToHtml(md);
|
// The canonical importer parses the raw block-level div into a real
|
||||||
// marked preserves the raw block-level div verbatim.
|
// htmlEmbed node carrying the decoded source.
|
||||||
expect(html).toContain('data-type="htmlEmbed"');
|
const json = await markdownToProseMirror(md);
|
||||||
|
|
||||||
const json = htmlToJson(html);
|
|
||||||
// The div parses into a real htmlEmbed node carrying the decoded source.
|
|
||||||
expect(hasHtmlEmbedNode(json)).toBe(true);
|
expect(hasHtmlEmbedNode(json)).toBe(true);
|
||||||
|
|
||||||
// Because it is detected, the share master-toggle strip can remove it.
|
// Because it is detected, the share master-toggle strip can remove it.
|
||||||
@@ -59,8 +56,7 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
|
|||||||
// therefore stripping) does not depend on the source being well-formed, so
|
// therefore stripping) does not depend on the source being well-formed, so
|
||||||
// the bypass cannot be hidden by sending a malformed data-source.
|
// the bypass cannot be hidden by sending a malformed data-source.
|
||||||
const md = `<div data-type="htmlEmbed" data-source="<script>x</script>"></div>`;
|
const md = `<div data-type="htmlEmbed" data-source="<script>x</script>"></div>`;
|
||||||
const html = await markdownToHtml(md);
|
const json = await markdownToProseMirror(md);
|
||||||
const json = htmlToJson(html);
|
|
||||||
expect(hasHtmlEmbedNode(json)).toBe(true);
|
expect(hasHtmlEmbedNode(json)).toBe(true);
|
||||||
expect(hasHtmlEmbedNode(stripHtmlEmbedNodes(json))).toBe(false);
|
expect(hasHtmlEmbedNode(stripHtmlEmbedNodes(json))).toBe(false);
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -53,8 +53,10 @@ import {
|
|||||||
extractPageSlugId,
|
extractPageSlugId,
|
||||||
} from '../../../integrations/export/utils';
|
} from '../../../integrations/export/utils';
|
||||||
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
import {
|
||||||
import { normalizeForeignMarkdown } from '../../../integrations/import/utils/foreign-markdown';
|
markdownToProseMirror,
|
||||||
|
normalizeForeignMarkdown,
|
||||||
|
} from '@docmost/prosemirror-markdown';
|
||||||
import { WatcherService } from '../../watcher/watcher.service';
|
import { WatcherService } from '../../watcher/watcher.service';
|
||||||
import { sql } from 'kysely';
|
import { sql } from 'kysely';
|
||||||
import { TransclusionService } from '../transclusion/transclusion.service';
|
import { TransclusionService } from '../transclusion/transclusion.service';
|
||||||
|
|||||||
@@ -22,10 +22,12 @@ import { v7 } from 'uuid';
|
|||||||
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
|
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
|
||||||
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
|
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
|
||||||
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
import {
|
||||||
|
markdownToProseMirror,
|
||||||
|
normalizeForeignMarkdown,
|
||||||
|
} from '@docmost/prosemirror-markdown';
|
||||||
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
|
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
|
||||||
import { formatImportHtml } from '../utils/import-formatter';
|
import { formatImportHtml } from '../utils/import-formatter';
|
||||||
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
|
|
||||||
import {
|
import {
|
||||||
buildAttachmentCandidates,
|
buildAttachmentCandidates,
|
||||||
collectMarkdownAndHtmlFiles,
|
collectMarkdownAndHtmlFiles,
|
||||||
|
|||||||
@@ -18,8 +18,10 @@ import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
|
|||||||
import { TiptapTransformer } from '@hocuspocus/transformer';
|
import { TiptapTransformer } from '@hocuspocus/transformer';
|
||||||
import * as Y from 'yjs';
|
import * as Y from 'yjs';
|
||||||
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
import {
|
||||||
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
|
markdownToProseMirror,
|
||||||
|
normalizeForeignMarkdown,
|
||||||
|
} from '@docmost/prosemirror-markdown';
|
||||||
import {
|
import {
|
||||||
FileTaskStatus,
|
FileTaskStatus,
|
||||||
FileTaskType,
|
FileTaskType,
|
||||||
|
|||||||
@@ -11,9 +11,6 @@
|
|||||||
"main": "dist/index.js",
|
"main": "dist/index.js",
|
||||||
"module": "./src/index.ts",
|
"module": "./src/index.ts",
|
||||||
"types": "dist/index.d.ts",
|
"types": "dist/index.d.ts",
|
||||||
"dependencies": {
|
|
||||||
"marked": "17.0.5"
|
|
||||||
},
|
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@vitest/coverage-v8": "4.1.6",
|
"@vitest/coverage-v8": "4.1.6",
|
||||||
"vitest": "4.1.6"
|
"vitest": "4.1.6"
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
export * from "./lib/trailing-node";
|
export * from "./lib/trailing-node";
|
||||||
|
export * from "./lib/code";
|
||||||
export * from "./lib/comment/comment";
|
export * from "./lib/comment/comment";
|
||||||
export * from "./lib/utils";
|
export * from "./lib/utils";
|
||||||
export * from "./lib/math";
|
export * from "./lib/math";
|
||||||
@@ -18,7 +19,6 @@ export * from "./lib/excalidraw";
|
|||||||
export * from "./lib/embed";
|
export * from "./lib/embed";
|
||||||
export * from "./lib/html-embed/html-embed";
|
export * from "./lib/html-embed/html-embed";
|
||||||
export * from "./lib/mention";
|
export * from "./lib/mention";
|
||||||
export * from "./lib/markdown";
|
|
||||||
export * from "./lib/search-and-replace";
|
export * from "./lib/search-and-replace";
|
||||||
export * from "./lib/embed-provider";
|
export * from "./lib/embed-provider";
|
||||||
export * from "./lib/subpages";
|
export * from "./lib/subpages";
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
import { Code as TiptapCode } from "@tiptap/extension-code";
|
||||||
|
|
||||||
|
// #515: canonical inline `code` mark for Docmost.
|
||||||
|
//
|
||||||
|
// Tiptap's stock Code mark (via StarterKit) declares `excludes: "_"`, which
|
||||||
|
// makes it exclude EVERY other inline mark: applying `code` drops any co-
|
||||||
|
// occurring bold/italic/… on both the HTML->PM import and editor transactions.
|
||||||
|
// That silently stripped emphasis adjacent to inline code (`` **`--flag`** ``
|
||||||
|
// lost its bold on markdown import). CommonMark nests them (`<strong><code>`),
|
||||||
|
// so Docmost lets `code` combine with all marks by overriding `excludes` to the
|
||||||
|
// empty string (excludes nothing).
|
||||||
|
//
|
||||||
|
// This is the SINGLE shared source imported by the live editor, the collab
|
||||||
|
// server and the comment editor schemas. The markdown-import mirror in
|
||||||
|
// @docmost/prosemirror-markdown re-declares the same override locally (it must
|
||||||
|
// not pull this React-aware package into its node runtime) and a parity test
|
||||||
|
// keeps the two in lockstep.
|
||||||
|
export const Code = TiptapCode.extend({
|
||||||
|
excludes: "",
|
||||||
|
});
|
||||||
@@ -14,7 +14,8 @@ import {
|
|||||||
* ProseMirror JSON directly (never running the editor's plugins), so the
|
* ProseMirror JSON directly (never running the editor's plugins), so the
|
||||||
* canonical footnote topology was never enforced on those writes. The consumers
|
* canonical footnote topology was never enforced on those writes. The consumers
|
||||||
* of this editor-ext copy are: the server markdown/HTML import
|
* of this editor-ext copy are: the server markdown/HTML import
|
||||||
* (`markdownToHtml -> htmlToJson` in import.service / file-import-task.service),
|
* (`markdownToProseMirror` from @docmost/prosemirror-markdown in import.service /
|
||||||
|
* file-import-task.service),
|
||||||
* `PageService` create/update (`parseProsemirrorContent` for the JSON/markdown/
|
* `PageService` create/update (`parseProsemirrorContent` for the JSON/markdown/
|
||||||
* HTML REST write paths), and the client markdown PASTE path
|
* HTML REST write paths), and the client markdown PASTE path
|
||||||
* (`markdown-clipboard.ts`). (The MCP package mirrors this canonicalizer in
|
* (`markdown-clipboard.ts`). (The MCP package mirrors this canonicalizer in
|
||||||
|
|||||||
@@ -1,131 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
|
|
||||||
import { markdownToHtml } from "../markdown/utils/marked.utils";
|
|
||||||
import { extractFootnoteDefinitions } from "../markdown/utils/footnote.marked";
|
|
||||||
|
|
||||||
// HTML the editor-ext nodes render (sup[data-footnote-ref], section/div).
|
|
||||||
const HTML =
|
|
||||||
`<p>Water<sup data-footnote-ref data-id="fn1"></sup> and clay<sup data-footnote-ref data-id="fn2"></sup>.</p>` +
|
|
||||||
`<section data-footnotes>` +
|
|
||||||
`<div data-footnote-def data-id="fn1"><p>First note.</p></div>` +
|
|
||||||
`<div data-footnote-def data-id="fn2"><p>Second note.</p></div>` +
|
|
||||||
`</section>`;
|
|
||||||
|
|
||||||
describe("footnote markdown round-trip", () => {
|
|
||||||
it("HTML -> Markdown produces pandoc footnote syntax", () => {
|
|
||||||
const md = htmlToMarkdown(HTML);
|
|
||||||
expect(md).toContain("[^fn1]");
|
|
||||||
expect(md).toContain("[^fn2]");
|
|
||||||
expect(md).toContain("[^fn1]: First note.");
|
|
||||||
expect(md).toContain("[^fn2]: Second note.");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("Markdown -> HTML rebuilds the footnote nodes' HTML", async () => {
|
|
||||||
const md = htmlToMarkdown(HTML);
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
expect(html).toContain('data-footnote-ref data-id="fn1"');
|
|
||||||
expect(html).toContain('data-footnote-ref data-id="fn2"');
|
|
||||||
expect(html).toContain("data-footnotes");
|
|
||||||
expect(html).toContain('data-footnote-def data-id="fn1"');
|
|
||||||
expect(html).toContain("First note.");
|
|
||||||
expect(html).toContain("Second note.");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("preserves a [^id]: line shown inside a fenced code block (not a definition)", async () => {
|
|
||||||
// A document that DOCUMENTS footnote syntax inside a code fence. The
|
|
||||||
// `[^demo]: ...` line is example text, not a real definition, and must
|
|
||||||
// survive the Markdown -> HTML conversion verbatim.
|
|
||||||
const md = [
|
|
||||||
"Here is how footnotes look:",
|
|
||||||
"",
|
|
||||||
"```markdown",
|
|
||||||
"Some text[^demo]",
|
|
||||||
"",
|
|
||||||
"[^demo]: this is the definition",
|
|
||||||
"```",
|
|
||||||
"",
|
|
||||||
"End of doc.",
|
|
||||||
].join("\n");
|
|
||||||
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
// The example definition line is kept inside the rendered code block.
|
|
||||||
expect(html).toContain("[^demo]: this is the definition");
|
|
||||||
// It did NOT get pulled out into a real footnotes section.
|
|
||||||
expect(html).not.toContain("data-footnotes");
|
|
||||||
expect(html).not.toContain("data-footnote-def");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("extractFootnoteDefinitions keeps the FIRST duplicate definition and reuses markers", () => {
|
|
||||||
// Two definitions share id `d`, and the body has two `[^d]` markers. Under
|
|
||||||
// the import model (#166) duplicate definition ids are FIRST-WINS: only the
|
|
||||||
// first definition is kept; markers are NEVER rewritten, so the two `[^d]`
|
|
||||||
// references reuse the single footnote.
|
|
||||||
const md = [
|
|
||||||
"See here[^d] and there[^d].",
|
|
||||||
"",
|
|
||||||
"[^d]: first",
|
|
||||||
"[^d]: second",
|
|
||||||
].join("\n");
|
|
||||||
|
|
||||||
const { body, section } = extractFootnoteDefinitions(md);
|
|
||||||
|
|
||||||
const defIds = Array.from(
|
|
||||||
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
|
|
||||||
).map((m) => m[1]);
|
|
||||||
expect(defIds).toEqual(["d"]); // first-wins: one definition
|
|
||||||
expect(section).toContain("first");
|
|
||||||
expect(section).not.toContain("second"); // duplicate dropped
|
|
||||||
|
|
||||||
// Both markers stay `[^d]` (reuse) — no `d__2` minting.
|
|
||||||
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
|
|
||||||
(m) => m[1],
|
|
||||||
);
|
|
||||||
expect(refIds).toEqual(["d", "d"]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("extractFootnoteDefinitions is DETERMINISTIC and stable (same input -> same output)", () => {
|
|
||||||
// The output must be a pure function of the input markdown so importing the
|
|
||||||
// same source twice (or via the editor and the MCP mirror) is identical.
|
|
||||||
const md = [
|
|
||||||
"See[^d] one[^d] two[^d].",
|
|
||||||
"",
|
|
||||||
"[^d]: first",
|
|
||||||
"[^d]: second",
|
|
||||||
"[^d]: third",
|
|
||||||
].join("\n");
|
|
||||||
|
|
||||||
const run = () => {
|
|
||||||
const { body, section } = extractFootnoteDefinitions(md);
|
|
||||||
const defIds = Array.from(
|
|
||||||
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
|
|
||||||
).map((m) => m[1]);
|
|
||||||
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
|
|
||||||
(m) => m[1],
|
|
||||||
);
|
|
||||||
return { defIds, refIds };
|
|
||||||
};
|
|
||||||
|
|
||||||
const a = run();
|
|
||||||
const b = run();
|
|
||||||
expect(a).toEqual(b);
|
|
||||||
// First-wins: one kept definition `d`; all three reuse markers stay `d`.
|
|
||||||
expect(a.defIds).toEqual(["d"]);
|
|
||||||
expect(a.refIds).toEqual(["d", "d", "d"]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("markdownToHtml with a reused id renders ONE shared footnote def", async () => {
|
|
||||||
const md = [
|
|
||||||
"See here[^d] and there[^d].",
|
|
||||||
"",
|
|
||||||
"[^d]: first",
|
|
||||||
"[^d]: second",
|
|
||||||
].join("\n");
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
const defIds = Array.from(
|
|
||||||
html.matchAll(/data-footnote-def data-id="([^"]+)"/g),
|
|
||||||
).map((m) => m[1]);
|
|
||||||
expect(defIds).toEqual(["d"]); // one shared definition
|
|
||||||
expect(html).toContain("first");
|
|
||||||
expect(html).not.toContain("second");
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -103,8 +103,9 @@ interface CollisionPlan {
|
|||||||
* `X__2`, `X__3`, collision-bumped) so it survives as a distinct footnote — which,
|
* `X__2`, `X__3`, collision-bumped) so it survives as a distinct footnote — which,
|
||||||
* having no matching reference, then falls under the normal orphan policy. It is
|
* having no matching reference, then falls under the normal orphan policy. It is
|
||||||
* only ever dropped for lacking a reference, never for colliding. The IMPORT
|
* only ever dropped for lacking a reference, never for colliding. The IMPORT
|
||||||
* paths (footnote.marked.ts / MCP extractFootnotes) instead apply first-wins +
|
* paths (@docmost/prosemirror-markdown / MCP extractFootnotes) instead apply
|
||||||
* drop + warn for duplicate definitions; that divergence is intentional — import
|
* first-wins + drop + warn for duplicate definitions; that divergence is
|
||||||
|
* intentional — import
|
||||||
* is an agent-authored artifact we sanitize, the editor is live user data we must
|
* is an agent-authored artifact we sanitize, the editor is live user data we must
|
||||||
* not lose.
|
* not lose.
|
||||||
*
|
*
|
||||||
|
|||||||
@@ -6,8 +6,9 @@ import { deriveFootnoteId } from "./footnote-util";
|
|||||||
*
|
*
|
||||||
* `deriveFootnoteId` lives ONLY in editor-ext now — it is used by
|
* `deriveFootnoteId` lives ONLY in editor-ext now — it is used by
|
||||||
* `resolveCollisions` (re-id of a duplicate definition) and `footnotePastePlugin`
|
* `resolveCollisions` (re-id of a duplicate definition) and `footnotePastePlugin`
|
||||||
* (re-id of a pasted colliding definition). The MCP/marked import paths no longer
|
* (re-id of a pasted colliding definition). The MCP / @docmost/prosemirror-markdown
|
||||||
* derive ids (duplicate definitions there are first-wins-dropped, #166), so there
|
* import paths no longer derive ids (duplicate definitions there are
|
||||||
|
* first-wins-dropped, #166), so there
|
||||||
* is no cross-package copy and no parity test to keep in sync. This table pins the
|
* is no cross-package copy and no parity test to keep in sync. This table pins the
|
||||||
* deterministic scheme so a future change to it is a conscious one.
|
* deterministic scheme so a future change to it is a conscious one.
|
||||||
*/
|
*/
|
||||||
|
|||||||
@@ -63,8 +63,9 @@ export function generateFootnoteId(): string {
|
|||||||
* its own seen-set before requesting the next derived id.
|
* its own seen-set before requesting the next derived id.
|
||||||
*
|
*
|
||||||
* Used only inside editor-ext now (resolveCollisions for a re-id'd duplicate
|
* Used only inside editor-ext now (resolveCollisions for a re-id'd duplicate
|
||||||
* DEFINITION, and footnotePastePlugin). The MCP/marked import paths no longer
|
* DEFINITION, and footnotePastePlugin). The MCP / @docmost/prosemirror-markdown
|
||||||
* derive ids — duplicate definitions there are first-wins-dropped (#166) — so
|
* import paths no longer derive ids — duplicate definitions there are
|
||||||
|
* first-wins-dropped (#166) — so
|
||||||
* there is no cross-package copy to keep in sync. The golden table in
|
* there is no cross-package copy to keep in sync. The golden table in
|
||||||
* footnote-util.derive-id.test.ts pins the scheme.
|
* footnote-util.derive-id.test.ts pins the scheme.
|
||||||
*/
|
*/
|
||||||
|
|||||||
@@ -1,68 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { generateJSON } from "@tiptap/html";
|
|
||||||
import { Document } from "@tiptap/extension-document";
|
|
||||||
import { Paragraph } from "@tiptap/extension-paragraph";
|
|
||||||
import { Text } from "@tiptap/extension-text";
|
|
||||||
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
|
|
||||||
import { markdownToHtml } from "../markdown/utils/marked.utils";
|
|
||||||
import { TiptapImage } from "./image";
|
|
||||||
|
|
||||||
// Minimal schema for parsing markdownToHtml output back to JSON (mirrors
|
|
||||||
// image.spec.ts), so we can assert the recovered caption EXACTLY.
|
|
||||||
const parseExtensions = [Document, Paragraph, Text, TiptapImage];
|
|
||||||
|
|
||||||
// Lossless markdown round-trip for image captions (issue #221). An image WITH a
|
|
||||||
// caption can't be expressed as ``, so it is emitted as a raw <img>
|
|
||||||
// (carrying data-caption) wrapped in a block <div>, the same trick the <video>
|
|
||||||
// rule uses. marked passes the raw HTML through, so markdownToHtml keeps the
|
|
||||||
// data-caption, and the image extension's parseHTML restores the attribute.
|
|
||||||
describe("image caption markdown round-trip", () => {
|
|
||||||
it("HTML -> Markdown emits a raw <img data-caption> for captioned images", () => {
|
|
||||||
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
|
|
||||||
const md = htmlToMarkdown(html);
|
|
||||||
expect(md).toContain("data-caption=\"A grey cat\"");
|
|
||||||
expect(md).toContain('src="/files/a.png"');
|
|
||||||
expect(md).toContain('alt="cat"');
|
|
||||||
// It must NOT degrade to the lossy ![]() form.
|
|
||||||
expect(md).not.toContain("![cat]");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("Markdown -> HTML restores data-caption on the <img>", async () => {
|
|
||||||
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
|
|
||||||
const md = htmlToMarkdown(html);
|
|
||||||
const back = await markdownToHtml(md);
|
|
||||||
expect(back).toContain('data-caption="A grey cat"');
|
|
||||||
expect(back).toContain('src="/files/a.png"');
|
|
||||||
});
|
|
||||||
|
|
||||||
it("special characters in the caption survive the round-trip (escaped)", async () => {
|
|
||||||
// The source caption is the decoded string `Tom & "Jerry"` (both an `&` and
|
|
||||||
// a `"`). escapeHtmlAttr must encode `&` -> `&` and `"` -> `"`.
|
|
||||||
const html = `<p><img src="/files/a.png" data-caption='Tom & "Jerry"'></p>`;
|
|
||||||
const md = htmlToMarkdown(html);
|
|
||||||
|
|
||||||
// (a) The intermediate Markdown must carry the EXACT escaped attribute. This
|
|
||||||
// fails if escapeHtmlAttr stopped escaping `"` (attribute break-out:
|
|
||||||
// data-caption="Tom & "Jerry"") or double-encoded `&` (`&amp;`).
|
|
||||||
expect(md).toContain('data-caption="Tom & "Jerry""');
|
|
||||||
|
|
||||||
const back = await markdownToHtml(md);
|
|
||||||
expect(back).toContain("data-caption=");
|
|
||||||
expect(back).toContain("Jerry");
|
|
||||||
expect(back).toContain("Tom");
|
|
||||||
|
|
||||||
// (b) Re-parse the rendered HTML through the image extension's parseHTML and
|
|
||||||
// assert the recovered caption is EXACTLY the original (no corruption, loss,
|
|
||||||
// or double-encoding).
|
|
||||||
const json = generateJSON(back, parseExtensions);
|
|
||||||
expect(json.content?.[0]?.attrs?.caption).toBe('Tom & "Jerry"');
|
|
||||||
});
|
|
||||||
|
|
||||||
it("caption-less images stay a clean  with no raw HTML", () => {
|
|
||||||
const html = `<p><img src="/files/a.png" alt="cat"></p>`;
|
|
||||||
const md = htmlToMarkdown(html);
|
|
||||||
expect(md).toContain("");
|
|
||||||
expect(md).not.toContain("data-caption");
|
|
||||||
expect(md).not.toContain("<img");
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,105 +0,0 @@
|
|||||||
import { describe, expect, it } from "vitest";
|
|
||||||
import { htmlEmbedExtension } from "./utils/html-embed.marked";
|
|
||||||
import { markdownToHtml } from "./index";
|
|
||||||
import { encodeHtmlEmbedSource } from "../html-embed/html-embed";
|
|
||||||
|
|
||||||
// CONTRACT tests for the marked block tokenizer that rebuilds an htmlEmbed node
|
|
||||||
// from the `<!--html-embed:BASE64-->` marker (html-embed.marked.ts), plus the
|
|
||||||
// observable round-trip through markdownToHtml.
|
|
||||||
//
|
|
||||||
// These pin the REAL tokenizer behaviour the import path depends on:
|
|
||||||
// - the tokenizer rule is anchored (^) and only accepts the base64 alphabet
|
|
||||||
// [A-Za-z0-9+/=], so a marker with non-base64 chars is NOT tokenized and
|
|
||||||
// survives as a literal HTML comment (not silently turned into something the
|
|
||||||
// server's strip no longer recognizes);
|
|
||||||
// - start() reports the correct index of the next marker so marked invokes the
|
|
||||||
// tokenizer at the right offset when a marker sits mid-document / after text;
|
|
||||||
// - a marker with surrounding text on the SAME line is split out into its own
|
|
||||||
// embed div while the surrounding text becomes ordinary paragraphs.
|
|
||||||
//
|
|
||||||
// The contract is asserted against the actual exported extension and pipeline —
|
|
||||||
// no behaviour is invented; the expectations were read off the real tokenizer.
|
|
||||||
|
|
||||||
const SAMPLE = "<b>x</b>";
|
|
||||||
const ENC = encodeHtmlEmbedSource(SAMPLE);
|
|
||||||
|
|
||||||
describe("htmlEmbed marked tokenizer — start()", () => {
|
|
||||||
it("returns the index of a marker that sits mid-document", () => {
|
|
||||||
const src = `hello world <!--html-embed:${ENC}-->`;
|
|
||||||
expect(htmlEmbedExtension.start(src)).toBe(src.indexOf("<!--html-embed:"));
|
|
||||||
});
|
|
||||||
|
|
||||||
it("returns 0 when the marker is at the very start", () => {
|
|
||||||
expect(htmlEmbedExtension.start(`<!--html-embed:${ENC}-->`)).toBe(0);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("returns -1 when there is no marker", () => {
|
|
||||||
expect(htmlEmbedExtension.start("no marker here")).toBe(-1);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("htmlEmbed marked tokenizer — tokenizer()", () => {
|
|
||||||
it("tokenizes a marker at the start of the input, capturing the base64 payload", () => {
|
|
||||||
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`);
|
|
||||||
expect(token).toBeTruthy();
|
|
||||||
expect(token!.type).toBe("htmlEmbed");
|
|
||||||
expect(token!.raw).toBe(`<!--html-embed:${ENC}-->`);
|
|
||||||
expect(token!.encoded).toBe(ENC);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("tokenizes an EMPTY marker (the [A-Za-z0-9+/=]* class allows zero chars)", () => {
|
|
||||||
const token = htmlEmbedExtension.tokenizer("<!--html-embed:-->");
|
|
||||||
expect(token).toBeTruthy();
|
|
||||||
expect(token!.encoded).toBe("");
|
|
||||||
expect(token!.raw).toBe("<!--html-embed:-->");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("does NOT tokenize when text precedes the marker (rule is anchored ^)", () => {
|
|
||||||
// marked relies on start() to advance to the marker; the tokenizer itself
|
|
||||||
// only matches at offset 0, so a non-anchored call returns undefined.
|
|
||||||
expect(
|
|
||||||
htmlEmbedExtension.tokenizer(`hello <!--html-embed:${ENC}-->`),
|
|
||||||
).toBeUndefined();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("does NOT tokenize a marker containing a non-base64 char ('$')", () => {
|
|
||||||
expect(
|
|
||||||
htmlEmbedExtension.tokenizer("<!--html-embed:ab$cd-->"),
|
|
||||||
).toBeUndefined();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("does NOT tokenize a marker containing a space", () => {
|
|
||||||
expect(
|
|
||||||
htmlEmbedExtension.tokenizer("<!--html-embed:ab cd-->"),
|
|
||||||
).toBeUndefined();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("renderer emits the embed div the node's parseHTML recognizes", () => {
|
|
||||||
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`)!;
|
|
||||||
const html = htmlEmbedExtension.renderer(token as any);
|
|
||||||
expect(html).toBe(
|
|
||||||
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
|
|
||||||
);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("htmlEmbed marked tokenizer — markdownToHtml round-trip", () => {
|
|
||||||
it("splits a marker out of surrounding same-line text into its own embed div", async () => {
|
|
||||||
const html = await markdownToHtml(`before <!--html-embed:${ENC}--> after`);
|
|
||||||
// The marker became the embed div...
|
|
||||||
expect(html).toContain(
|
|
||||||
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
|
|
||||||
);
|
|
||||||
// ...and the surrounding text survived as ordinary paragraph content.
|
|
||||||
expect(html).toContain("before");
|
|
||||||
expect(html).toContain("after");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("leaves a marker with non-base64 chars as a literal comment (NOT an embed div)", async () => {
|
|
||||||
const html = await markdownToHtml("<!--html-embed:ab$cd-->");
|
|
||||||
// It is NOT tokenized into an embed div the server would strip...
|
|
||||||
expect(html).not.toContain('data-type="htmlEmbed"');
|
|
||||||
// ...it passes through unchanged as a literal HTML comment.
|
|
||||||
expect(html).toContain("<!--html-embed:ab$cd-->");
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,2 +0,0 @@
|
|||||||
export * from "./utils/marked.utils";
|
|
||||||
export * from "./utils/turndown.utils";
|
|
||||||
@@ -1,112 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { markdownToHtml, htmlToMarkdown } from "./index";
|
|
||||||
import {
|
|
||||||
encodeHtmlEmbedSource,
|
|
||||||
decodeHtmlEmbedSource,
|
|
||||||
} from "../html-embed/html-embed";
|
|
||||||
|
|
||||||
// SECURITY (Variant C admin gate, import attack surface).
|
|
||||||
//
|
|
||||||
// The markdown import path is the only write path where an htmlEmbed reaches
|
|
||||||
// the server purely from file bytes (no editor / collab socket). The marked
|
|
||||||
// tokenizer in `html-embed.marked.ts` and the turndown rule in
|
|
||||||
// `turndown.utils.ts` are what materialize the `<!--html-embed:BASE64-->`
|
|
||||||
// marker into the `<div data-type="htmlEmbed" data-source="BASE64">` element
|
|
||||||
// that the server then parses into an htmlEmbed node and the admin gate strips.
|
|
||||||
//
|
|
||||||
// If either the tokenizer regex or the turndown rule shape drifts, the marker
|
|
||||||
// would either (a) stop becoming an htmlEmbed node (silently dropping admin
|
|
||||||
// content) or (b) become some OTHER tag the server's `hasHtmlEmbedNode` no
|
|
||||||
// longer recognizes (a strip bypass). These tests pin the marker <-> embed-div
|
|
||||||
// contract that the server-side strip relies on. editor-ext had ZERO tests
|
|
||||||
// before this file; this adds the runner + the round-trip coverage.
|
|
||||||
|
|
||||||
// The server parses the embed div by matching `data-type="htmlEmbed"` and
|
|
||||||
// decoding `data-source`; mirror that here so the assertion is exactly what the
|
|
||||||
// real `htmlToJson` -> htmlEmbed node parse depends on (the node's parseHTML in
|
|
||||||
// html-embed.ts uses the same selector + decodeHtmlEmbedSource).
|
|
||||||
const EMBED_DIV_RE = /<div[^>]*\bdata-type="htmlEmbed"[^>]*>/;
|
|
||||||
function extractEmbedSource(html: string): string | undefined {
|
|
||||||
const div = EMBED_DIV_RE.exec(html);
|
|
||||||
if (!div) return undefined;
|
|
||||||
const enc = /data-source="([^"]*)"/.exec(div[0]);
|
|
||||||
if (!enc) return undefined;
|
|
||||||
return decodeHtmlEmbedSource(enc[1]);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Replicates the server's `hasHtmlEmbedNode` decision against the embed *div*
|
|
||||||
// (the HTML form the server immediately converts to JSON). If this matches, the
|
|
||||||
// server's JSON-level `hasHtmlEmbedNode` will too, because htmlToJson maps this
|
|
||||||
// exact div to an htmlEmbed node.
|
|
||||||
function htmlHasHtmlEmbed(html: string): boolean {
|
|
||||||
return EMBED_DIV_RE.test(html);
|
|
||||||
}
|
|
||||||
|
|
||||||
describe("markdown <!--html-embed--> import round-trip", () => {
|
|
||||||
const source = "<script>x</script>";
|
|
||||||
|
|
||||||
it("markdownToHtml turns the marker into an htmlEmbed div carrying the source", async () => {
|
|
||||||
const md = "<!--html-embed:" + encodeHtmlEmbedSource(source) + "-->";
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
|
|
||||||
// The marker became the embed div the server recognizes as an htmlEmbed
|
|
||||||
// node (so the server's hasHtmlEmbedNode would match it after htmlToJson).
|
|
||||||
expect(htmlHasHtmlEmbed(html)).toBe(true);
|
|
||||||
// The decoded source is the original script, intact.
|
|
||||||
expect(extractEmbedSource(html)).toBe(source);
|
|
||||||
// The raw script is NOT inlined into the HTML — it stays base64 in the
|
|
||||||
// attribute (the marker itself must not be a direct injection vector).
|
|
||||||
expect(html).not.toContain("<script>x</script>");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("preserves UTF-8 / special chars in the embedded source", async () => {
|
|
||||||
const utf8 = '<script>console.log("héllo → 世界")</script>';
|
|
||||||
const md = "<!--html-embed:" + encodeHtmlEmbedSource(utf8) + "-->";
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
expect(htmlHasHtmlEmbed(html)).toBe(true);
|
|
||||||
expect(extractEmbedSource(html)).toBe(utf8);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("an empty marker still produces an htmlEmbed div (empty source)", async () => {
|
|
||||||
const html = await markdownToHtml("<!--html-embed:-->");
|
|
||||||
expect(htmlHasHtmlEmbed(html)).toBe(true);
|
|
||||||
expect(extractEmbedSource(html)).toBe("");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("round-trips htmlToMarkdown -> markdownToHtml preserving the embed marker", async () => {
|
|
||||||
const encoded = encodeHtmlEmbedSource(source);
|
|
||||||
// NOTE: turndown drops a *blank* (childless) element before any custom rule
|
|
||||||
// runs, and the htmlEmbed div is normally childless. The export pipeline
|
|
||||||
// therefore must give the rule a non-blank div to fire on; we add an inert
|
|
||||||
// text child here to exercise the real turndown htmlEmbed rule. (A blank
|
|
||||||
// embed div serializing to "" is asserted separately below as a documented
|
|
||||||
// edge so this contract drift is visible.)
|
|
||||||
const startHtml = `<div data-type="htmlEmbed" data-source="${encoded}">x</div>`;
|
|
||||||
|
|
||||||
// Export to markdown: the turndown rule emits the <!--html-embed:..-->
|
|
||||||
// marker (lossless, inert in plain markdown viewers).
|
|
||||||
const md = htmlToMarkdown(startHtml);
|
|
||||||
expect(md).toContain("<!--html-embed:" + encoded + "-->");
|
|
||||||
|
|
||||||
// Re-import: the marker round-trips back into an embed div with the same
|
|
||||||
// decoded source — this is the marker <-> embed-div contract the server's
|
|
||||||
// import strip depends on.
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
expect(htmlHasHtmlEmbed(html)).toBe(true);
|
|
||||||
expect(extractEmbedSource(html)).toBe(source);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("documents that a BLANK embed div serializes to empty markdown (turndown drops childless blocks)", () => {
|
|
||||||
const encoded = encodeHtmlEmbedSource(source);
|
|
||||||
const blank = `<div data-type="htmlEmbed" data-source="${encoded}"></div>`;
|
|
||||||
// This pins current behavior so a future change to the turndown rule (e.g.
|
|
||||||
// making it fire on blank nodes) is caught rather than silently shipping.
|
|
||||||
expect(htmlToMarkdown(blank)).toBe("");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("the base64 codec itself round-trips (no '<' leaks into the attribute)", () => {
|
|
||||||
const encoded = encodeHtmlEmbedSource(source);
|
|
||||||
expect(encoded).not.toContain("<");
|
|
||||||
expect(decodeHtmlEmbedSource(encoded)).toBe(source);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
/**
|
|
||||||
* Flexible `basename` implementation for node and the browser
|
|
||||||
* @see https://stackoverflow.com/a/59907288/2228771
|
|
||||||
*/
|
|
||||||
export function getBasename(path: string) {
|
|
||||||
// make sure the basename is not empty, if string ends with separator
|
|
||||||
let end = path.length - 1;
|
|
||||||
while (path[end] === '/' || path[end] === '\\') {
|
|
||||||
--end;
|
|
||||||
}
|
|
||||||
|
|
||||||
// support mixing of Win + Unix path separators
|
|
||||||
const i1 = path.lastIndexOf('/', end);
|
|
||||||
const i2 = path.lastIndexOf('\\', end);
|
|
||||||
|
|
||||||
let start: number;
|
|
||||||
if (i1 === -1) {
|
|
||||||
if (i2 === -1) {
|
|
||||||
// no separator in the whole thing
|
|
||||||
return path;
|
|
||||||
}
|
|
||||||
start = i2;
|
|
||||||
} else if (i2 === -1) {
|
|
||||||
start = i1;
|
|
||||||
} else {
|
|
||||||
start = Math.max(i1, i2);
|
|
||||||
}
|
|
||||||
return path.substring(start + 1, end + 1);
|
|
||||||
}
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
/**
|
|
||||||
* Shared pieces for the two callout tokenizers — `callout.marked.ts` (the
|
|
||||||
* `:::type` fenced form) and `github-callout.marked.ts` (the `> [!type]` GitHub
|
|
||||||
* alert form). Both emit the SAME callout node, so the banner type dictionary
|
|
||||||
* and the HTML renderer live here once instead of drifting apart in two files.
|
|
||||||
* The tokenizers themselves stay separate (different syntaxes / source matching).
|
|
||||||
*/
|
|
||||||
|
|
||||||
/** The four callout banner types the editor schema supports. */
|
|
||||||
export const CALLOUT_TYPES = ['info', 'success', 'warning', 'danger'] as const;
|
|
||||||
|
|
||||||
export type CalloutType = (typeof CALLOUT_TYPES)[number];
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Coerce an arbitrary type name onto a supported banner type, defaulting to
|
|
||||||
* `info` for anything unrecognized (the shared fallback both tokenizers use).
|
|
||||||
*/
|
|
||||||
export function normalizeCalloutType(type: string): CalloutType {
|
|
||||||
return (CALLOUT_TYPES as readonly string[]).includes(type)
|
|
||||||
? (type as CalloutType)
|
|
||||||
: 'info';
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Render a callout node to the editor's HTML shape. `body` is the already
|
|
||||||
* markdown-parsed inner content (marked may hand back a string synchronously).
|
|
||||||
*/
|
|
||||||
export function renderCalloutHtml(
|
|
||||||
type: string,
|
|
||||||
body: string | Promise<string>,
|
|
||||||
): string {
|
|
||||||
return `<div data-type="callout" data-callout-type="${type}">${body}</div>`;
|
|
||||||
}
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
import { Token, marked } from 'marked';
|
|
||||||
import { normalizeCalloutType, renderCalloutHtml } from './callout-common.marked';
|
|
||||||
|
|
||||||
interface CalloutToken {
|
|
||||||
type: 'callout';
|
|
||||||
calloutType: string;
|
|
||||||
text: string;
|
|
||||||
raw: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
export const calloutExtension = {
|
|
||||||
name: 'callout',
|
|
||||||
level: 'block',
|
|
||||||
start(src: string) {
|
|
||||||
return src.match(/:::/)?.index ?? -1;
|
|
||||||
},
|
|
||||||
tokenizer(src: string): CalloutToken | undefined {
|
|
||||||
const rule = /^:::([a-zA-Z0-9]+)\s+([\s\S]+?):::/;
|
|
||||||
const match = rule.exec(src);
|
|
||||||
|
|
||||||
if (match) {
|
|
||||||
return {
|
|
||||||
type: 'callout',
|
|
||||||
calloutType: normalizeCalloutType(match[1]),
|
|
||||||
raw: match[0],
|
|
||||||
text: match[2].trim(),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
},
|
|
||||||
renderer(token: Token) {
|
|
||||||
const calloutToken = token as CalloutToken;
|
|
||||||
return renderCalloutHtml(
|
|
||||||
calloutToken.calloutType,
|
|
||||||
marked.parse(calloutToken.text),
|
|
||||||
);
|
|
||||||
},
|
|
||||||
};
|
|
||||||
@@ -1,72 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { extractFootnoteDefinitions } from "./footnote.marked";
|
|
||||||
|
|
||||||
/** Pull the ordered list of `data-footnote-def` ids out of the rendered section. */
|
|
||||||
function defIds(section: string): string[] {
|
|
||||||
return [...section.matchAll(/data-footnote-def data-id="([^"]+)"/g)].map(
|
|
||||||
(m) => m[1],
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Pull the ordered list of `[^id]` markers that remain in the body. */
|
|
||||||
function bodyMarkers(body: string): string[] {
|
|
||||||
return [...body.matchAll(/\[\^([^\]\s]+)\]/g)].map((m) => m[1]);
|
|
||||||
}
|
|
||||||
|
|
||||||
describe("extractFootnoteDefinitions: duplicate definition ids (first-wins)", () => {
|
|
||||||
// Body has ONE `[^d]` reference but THREE `[^d]:` definitions. Under the
|
|
||||||
// import model (#166) a duplicate definition id is FIRST-WINS: only the first
|
|
||||||
// definition is kept; the rest are DROPPED (and surfaced by analyzeFootnotes,
|
|
||||||
// not silently re-id'd into orphan footnotes as before). Reference markers are
|
|
||||||
// never rewritten, so repeated references would reuse the single footnote.
|
|
||||||
const md = ["See[^d].", "", "[^d]: a", "[^d]: b", "[^d]: c"].join("\n");
|
|
||||||
|
|
||||||
it("keeps only the FIRST definition for the id (first-wins)", () => {
|
|
||||||
const { section } = extractFootnoteDefinitions(md);
|
|
||||||
const ids = defIds(section);
|
|
||||||
expect(ids).toEqual(["d"]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("keeps the first definition's text and drops the duplicates", () => {
|
|
||||||
const { section } = extractFootnoteDefinitions(md);
|
|
||||||
expect(section).toContain('data-footnote-def data-id="d"><p>a</p>');
|
|
||||||
// No derived `d__2` / `d__3` ids are emitted anymore.
|
|
||||||
expect(section).not.toContain("d__2");
|
|
||||||
expect(section).not.toContain("d__3");
|
|
||||||
// The dropped duplicate texts are not in the section.
|
|
||||||
expect(section).not.toContain("<p>b</p>");
|
|
||||||
expect(section).not.toContain("<p>c</p>");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("leaves the SINGLE body marker as [^d] (markers are never rewritten)", () => {
|
|
||||||
const { body } = extractFootnoteDefinitions(md);
|
|
||||||
expect(bodyMarkers(body)).toEqual(["d"]);
|
|
||||||
expect(body).toContain("See[^d].");
|
|
||||||
// The definition lines themselves were pulled OUT of the body.
|
|
||||||
expect(body).not.toContain("[^d]: a");
|
|
||||||
expect(body).not.toContain("[^d]: b");
|
|
||||||
expect(body).not.toContain("[^d]: c");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("does not crash and produces a well-formed footnotes section", () => {
|
|
||||||
const { section } = extractFootnoteDefinitions(md);
|
|
||||||
expect(section.startsWith("<section data-footnotes>")).toBe(true);
|
|
||||||
expect(section.endsWith("</section>")).toBe(true);
|
|
||||||
// Exactly one definition div (first-wins).
|
|
||||||
expect([...section.matchAll(/<div data-footnote-def/g)]).toHaveLength(1);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("extractFootnoteDefinitions: reuse (repeated references, one definition)", () => {
|
|
||||||
// Pandoc semantics: many `[^a]` references + one `[^a]:` definition = one
|
|
||||||
// footnote, shared. Markers are left intact so the editor numbers them as one.
|
|
||||||
const md = ["A[^a] B[^a] C[^a].", "", "[^a]: shared note"].join("\n");
|
|
||||||
|
|
||||||
it("emits exactly one definition and leaves every reference marker as [^a]", () => {
|
|
||||||
const { section, body } = extractFootnoteDefinitions(md);
|
|
||||||
expect(defIds(section)).toEqual(["a"]);
|
|
||||||
expect(section).toContain('data-footnote-def data-id="a"><p>shared note</p>');
|
|
||||||
// All three reference markers stay `a` (no `a__2`/`a__3` minting).
|
|
||||||
expect(bodyMarkers(body)).toEqual(["a", "a", "a"]);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,131 +0,0 @@
|
|||||||
import { marked } from "marked";
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Pandoc/GFM footnote support for the marked (Markdown -> HTML) pipeline.
|
|
||||||
*
|
|
||||||
* Two pieces:
|
|
||||||
* - an INLINE tokenizer for `[^id]` references -> <sup data-footnote-ref
|
|
||||||
* data-id="id"> (matches the editor-ext FootnoteReference renderHTML);
|
|
||||||
* - a document hook (`preprocess`/`walkTokens` is awkward for collecting +
|
|
||||||
* removing definitions, so we use a regex preprocessing step instead) that
|
|
||||||
* pulls every `[^id]: text` definition line out of the body and appends a
|
|
||||||
* single <section data-footnotes> with one <div data-footnote-def> per
|
|
||||||
* definition, so the round-trip rebuilds footnotesList + footnoteDefinition.
|
|
||||||
*
|
|
||||||
* Every FIRST definition line is emitted — duplicate ids are first-wins (the
|
|
||||||
* rest are dropped, and surfaced via analyzeFootnotes), and reference markers are
|
|
||||||
* left untouched so repeated `[^a]` references reuse the one footnote (#166).
|
|
||||||
* Orphan definitions (no matching reference) are still emitted here; the editor's
|
|
||||||
* sync plugin reconciles the final reference/definition set (drops orphans,
|
|
||||||
* synthesizes a single empty definition for a reference that lacks one).
|
|
||||||
*/
|
|
||||||
|
|
||||||
const DEFINITION_RE = /^\[\^([^\]\s]+)\]:[ \t]*(.*)$/;
|
|
||||||
const REFERENCE_RE = /\[\^([^\]\s]+)\]/;
|
|
||||||
|
|
||||||
interface FootnoteRefToken {
|
|
||||||
type: "footnoteRef";
|
|
||||||
raw: string;
|
|
||||||
id: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
export const footnoteReferenceExtension = {
|
|
||||||
name: "footnoteRef",
|
|
||||||
level: "inline" as const,
|
|
||||||
start(src: string) {
|
|
||||||
return src.match(/\[\^/)?.index ?? -1;
|
|
||||||
},
|
|
||||||
tokenizer(src: string): FootnoteRefToken | undefined {
|
|
||||||
const match = REFERENCE_RE.exec(src);
|
|
||||||
// Only match at the very start of the remaining inline source.
|
|
||||||
if (match && match.index === 0) {
|
|
||||||
return {
|
|
||||||
type: "footnoteRef",
|
|
||||||
raw: match[0],
|
|
||||||
id: match[1],
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return undefined;
|
|
||||||
},
|
|
||||||
renderer(token: FootnoteRefToken) {
|
|
||||||
return `<sup data-footnote-ref data-id="${escapeAttr(token.id)}"></sup>`;
|
|
||||||
},
|
|
||||||
};
|
|
||||||
|
|
||||||
function escapeAttr(value: string): string {
|
|
||||||
return String(value).replace(/&/g, "&").replace(/"/g, """);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Extract `[^id]: text` definition lines from the markdown body, returning the
|
|
||||||
* cleaned body plus a rendered <section data-footnotes> (empty string when no
|
|
||||||
* definitions). Call this BEFORE marked.parse and append the section to the
|
|
||||||
* resulting HTML.
|
|
||||||
*/
|
|
||||||
export function extractFootnoteDefinitions(markdown: string): {
|
|
||||||
body: string;
|
|
||||||
section: string;
|
|
||||||
} {
|
|
||||||
const lines = markdown.split("\n");
|
|
||||||
const bodyLines: string[] = [];
|
|
||||||
const definitions: Array<{ id: string; text: string }> = [];
|
|
||||||
|
|
||||||
// Track fenced-code state so a `[^id]: ...` line that merely SHOWS footnote
|
|
||||||
// syntax inside a ``` / ~~~ code block is left in the body verbatim and not
|
|
||||||
// mistaken for a real definition.
|
|
||||||
let fence: string | null = null;
|
|
||||||
|
|
||||||
for (const line of lines) {
|
|
||||||
const fenceMatch = /^(\s*)(`{3,}|~{3,})/.exec(line);
|
|
||||||
if (fenceMatch) {
|
|
||||||
const marker = fenceMatch[2][0];
|
|
||||||
if (fence === null) {
|
|
||||||
fence = marker; // opening fence
|
|
||||||
} else if (marker === fence) {
|
|
||||||
fence = null; // closing fence (matching delimiter type)
|
|
||||||
}
|
|
||||||
bodyLines.push(line);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
const m = fence === null ? DEFINITION_RE.exec(line) : null;
|
|
||||||
if (m) {
|
|
||||||
definitions.push({ id: m[1], text: m[2] });
|
|
||||||
} else {
|
|
||||||
bodyLines.push(line);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (definitions.length === 0) {
|
|
||||||
return { body: markdown, section: "" };
|
|
||||||
}
|
|
||||||
|
|
||||||
// Duplicate definition ids (e.g. `[^d]: first` / `[^d]: second`): FIRST WINS,
|
|
||||||
// the rest are DROPPED. Reference markers are left UNTOUCHED so repeated `[^a]`
|
|
||||||
// references reuse the single footnote (Pandoc semantics, #166). This differs
|
|
||||||
// from the live editor's never-lose policy (resolveCollisions re-ids a
|
|
||||||
// duplicate definition into an orphan) on purpose: an import is an
|
|
||||||
// agent-authored artifact we sanitize, and the dropped duplicate is surfaced
|
|
||||||
// to the caller via analyzeFootnotes' `duplicateDefinitions` warning instead.
|
|
||||||
const firstById = new Map<string, string>(); // id -> first definition text
|
|
||||||
for (const def of definitions) {
|
|
||||||
if (!firstById.has(def.id)) firstById.set(def.id, def.text);
|
|
||||||
}
|
|
||||||
|
|
||||||
const defsHtml = [...firstById.entries()]
|
|
||||||
.map(([id, text]) => {
|
|
||||||
// Render the definition text as inline markdown so emphasis/links inside
|
|
||||||
// a footnote survive the round-trip; wrap in a paragraph (the node's
|
|
||||||
// content is paragraph+).
|
|
||||||
const inner = marked.parseInline(text || "");
|
|
||||||
return `<div data-footnote-def data-id="${escapeAttr(
|
|
||||||
id,
|
|
||||||
)}"><p>${inner}</p></div>`;
|
|
||||||
})
|
|
||||||
.join("");
|
|
||||||
|
|
||||||
return {
|
|
||||||
body: bodyLines.join("\n"),
|
|
||||||
section: `<section data-footnotes>${defsHtml}</section>`,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { markdownToHtml } from "./marked.utils";
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Regression for issue #192: pasting a GitHub-style `> [!type]` alert produced a
|
|
||||||
* literal `<blockquote>` containing `[!info]` instead of a callout node, because
|
|
||||||
* only the `:::type` form was tokenized. The editor paste path runs the same
|
|
||||||
* `markdownToHtml`, so these assertions pin the conversion at the source.
|
|
||||||
*/
|
|
||||||
function html(md: string): string {
|
|
||||||
const out = markdownToHtml(md);
|
|
||||||
if (typeof out !== "string") throw new Error("expected sync string output");
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
describe("markdownToHtml: GitHub `> [!type]` callouts", () => {
|
|
||||||
it("converts `> [!info]` to a callout node, not a literal blockquote", () => {
|
|
||||||
const out = html("> [!info]\n> Callout body text here");
|
|
||||||
expect(out).toContain('data-type="callout"');
|
|
||||||
expect(out).toContain('data-callout-type="info"');
|
|
||||||
expect(out).toContain("Callout body text here");
|
|
||||||
expect(out).not.toContain("[!info]");
|
|
||||||
expect(out).not.toContain("<blockquote");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("maps GitHub alert aliases onto the supported banner types", () => {
|
|
||||||
expect(html("> [!NOTE]\n> x")).toContain('data-callout-type="info"');
|
|
||||||
expect(html("> [!TIP]\n> x")).toContain('data-callout-type="success"');
|
|
||||||
expect(html("> [!WARNING]\n> x")).toContain('data-callout-type="warning"');
|
|
||||||
expect(html("> [!CAUTION]\n> x")).toContain('data-callout-type="danger"');
|
|
||||||
});
|
|
||||||
|
|
||||||
it("accepts the editor's own type names directly", () => {
|
|
||||||
expect(html("> [!success]\n> x")).toContain('data-callout-type="success"');
|
|
||||||
expect(html("> [!danger]\n> x")).toContain('data-callout-type="danger"');
|
|
||||||
});
|
|
||||||
|
|
||||||
it("falls back to info for an unknown type", () => {
|
|
||||||
expect(html("> [!bogus]\n> x")).toContain('data-callout-type="info"');
|
|
||||||
});
|
|
||||||
|
|
||||||
it("preserves multi-line callout bodies", () => {
|
|
||||||
const out = html("> [!warning]\n> line one\n> line two");
|
|
||||||
expect(out).toContain('data-callout-type="warning"');
|
|
||||||
expect(out).toContain("line one");
|
|
||||||
expect(out).toContain("line two");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("still converts the `:::type` form", () => {
|
|
||||||
const out = html(":::info\nbody\n:::");
|
|
||||||
expect(out).toContain('data-type="callout"');
|
|
||||||
expect(out).toContain('data-callout-type="info"');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
import { Token, marked } from 'marked';
|
|
||||||
import { renderCalloutHtml } from './callout-common.marked';
|
|
||||||
|
|
||||||
interface GithubCalloutToken {
|
|
||||||
type: 'githubCallout';
|
|
||||||
calloutType: string;
|
|
||||||
text: string;
|
|
||||||
raw: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Map GitHub "alert" blockquote markers (`> [!NOTE]`, `> [!WARNING]`, …) onto
|
|
||||||
* the four callout banner types the editor schema supports. The editor's own
|
|
||||||
* type names (`info`/`success`/`warning`/`danger`) are also accepted directly,
|
|
||||||
* because users paste both forms. Anything unrecognized falls back to `info`,
|
|
||||||
* matching the `:::type` callout tokenizer.
|
|
||||||
*/
|
|
||||||
const GITHUB_ALERT_TYPE_MAP: Record<string, string> = {
|
|
||||||
note: 'info',
|
|
||||||
tip: 'success',
|
|
||||||
important: 'info',
|
|
||||||
warning: 'warning',
|
|
||||||
caution: 'danger',
|
|
||||||
info: 'info',
|
|
||||||
success: 'success',
|
|
||||||
danger: 'danger',
|
|
||||||
};
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Tokenizer for GitHub-flavored alert callouts written as a blockquote whose
|
|
||||||
* first line is `[!type]`:
|
|
||||||
*
|
|
||||||
* > [!info]
|
|
||||||
* > body line one
|
|
||||||
* > body line two
|
|
||||||
*
|
|
||||||
* Without this, the default blockquote tokenizer wins and the marker renders as
|
|
||||||
* a literal `[!info]` inside a `<blockquote>`. The editor's paste path runs the
|
|
||||||
* same `markdownToHtml`, so registering this here also fixes pasting the syntax
|
|
||||||
* into the editor (issue #192), not just markdown import.
|
|
||||||
*/
|
|
||||||
export const githubCalloutExtension = {
|
|
||||||
name: 'githubCallout',
|
|
||||||
level: 'block' as const,
|
|
||||||
start(src: string) {
|
|
||||||
return src.match(/^ {0,3}>[ \t]*\[!/m)?.index ?? -1;
|
|
||||||
},
|
|
||||||
tokenizer(src: string): GithubCalloutToken | undefined {
|
|
||||||
const rule =
|
|
||||||
/^ {0,3}>[ \t]*\[!([a-zA-Z]+)\][^\n]*(?:\n {0,3}>[^\n]*)*(?:\n|$)/;
|
|
||||||
const match = rule.exec(src);
|
|
||||||
if (!match) return undefined;
|
|
||||||
|
|
||||||
const rawType = match[1].toLowerCase();
|
|
||||||
const calloutType = GITHUB_ALERT_TYPE_MAP[rawType] ?? 'info';
|
|
||||||
|
|
||||||
const text = match[0]
|
|
||||||
.replace(/\n+$/, '')
|
|
||||||
.split('\n')
|
|
||||||
// Strip the blockquote marker (`>` + optional space) from every line.
|
|
||||||
.map((line) => line.replace(/^ {0,3}>[ \t]?/, ''))
|
|
||||||
// Drop the `[!type]` marker that opens the first line.
|
|
||||||
.map((line, i) => (i === 0 ? line.replace(/^\[![a-zA-Z]+\][ \t]*/, '') : line))
|
|
||||||
.join('\n')
|
|
||||||
.trim();
|
|
||||||
|
|
||||||
return {
|
|
||||||
type: 'githubCallout',
|
|
||||||
calloutType,
|
|
||||||
raw: match[0],
|
|
||||||
text,
|
|
||||||
};
|
|
||||||
},
|
|
||||||
renderer(token: Token) {
|
|
||||||
const calloutToken = token as GithubCalloutToken;
|
|
||||||
return renderCalloutHtml(
|
|
||||||
calloutToken.calloutType,
|
|
||||||
marked.parse(calloutToken.text),
|
|
||||||
);
|
|
||||||
},
|
|
||||||
};
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
import { Token } from "marked";
|
|
||||||
|
|
||||||
interface HtmlEmbedToken {
|
|
||||||
type: "htmlEmbed";
|
|
||||||
raw: string;
|
|
||||||
encoded: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Marked extension that rebuilds an `htmlEmbed` node from the HTML comment
|
|
||||||
* marker produced by the turndown rule (`<!--html-embed:<base64>-->`).
|
|
||||||
*
|
|
||||||
* It emits the same marker div the node's `parseHTML` recognizes, so the
|
|
||||||
* pipeline MD -> HTML -> ProseMirror JSON restores the node (and its
|
|
||||||
* base64 `data-source`) exactly. We do NOT expand the raw markup here; the
|
|
||||||
* source stays base64-encoded in the attribute and is only executed by the
|
|
||||||
* client NodeView.
|
|
||||||
*/
|
|
||||||
export const htmlEmbedExtension = {
|
|
||||||
name: "htmlEmbed",
|
|
||||||
level: "block" as const,
|
|
||||||
start(src: string) {
|
|
||||||
return src.indexOf("<!--html-embed:");
|
|
||||||
},
|
|
||||||
tokenizer(src: string): HtmlEmbedToken | undefined {
|
|
||||||
const rule = /^<!--html-embed:([A-Za-z0-9+/=]*)-->/;
|
|
||||||
const match = rule.exec(src);
|
|
||||||
|
|
||||||
if (match) {
|
|
||||||
return {
|
|
||||||
type: "htmlEmbed",
|
|
||||||
raw: match[0],
|
|
||||||
encoded: match[1] ?? "",
|
|
||||||
};
|
|
||||||
}
|
|
||||||
},
|
|
||||||
renderer(token: Token) {
|
|
||||||
const htmlEmbedToken = token as HtmlEmbedToken;
|
|
||||||
return `<div data-type="htmlEmbed" data-source="${htmlEmbedToken.encoded}"></div>`;
|
|
||||||
},
|
|
||||||
};
|
|
||||||
@@ -1,76 +0,0 @@
|
|||||||
import { marked } from "marked";
|
|
||||||
import { calloutExtension } from "./callout.marked";
|
|
||||||
import { githubCalloutExtension } from "./github-callout.marked";
|
|
||||||
import { mathBlockExtension } from "./math-block.marked";
|
|
||||||
import { mathInlineExtension } from "./math-inline.marked";
|
|
||||||
import {
|
|
||||||
footnoteReferenceExtension,
|
|
||||||
extractFootnoteDefinitions,
|
|
||||||
} from "./footnote.marked";
|
|
||||||
import { htmlEmbedExtension } from "./html-embed.marked";
|
|
||||||
|
|
||||||
marked.use({
|
|
||||||
renderer: {
|
|
||||||
list({ ordered, start, items }) {
|
|
||||||
let body = "";
|
|
||||||
for (const item of items) {
|
|
||||||
body += this.listitem(item);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (ordered) {
|
|
||||||
const startAttr = start !== 1 ? ` start="${start}"` : "";
|
|
||||||
return `<ol${startAttr}>\n${body}</ol>\n`;
|
|
||||||
}
|
|
||||||
|
|
||||||
const isTaskList = items.some((item) => item.task);
|
|
||||||
const dataType = isTaskList ? ' data-type="taskList"' : "";
|
|
||||||
return `<ul${dataType}>\n${body}</ul>\n`;
|
|
||||||
},
|
|
||||||
listitem({ tokens, task: isTask, checked: isChecked }) {
|
|
||||||
const text = this.parser.parse(tokens);
|
|
||||||
if (!isTask) {
|
|
||||||
return `<li>${text}</li>\n`;
|
|
||||||
}
|
|
||||||
const checkedAttr = isChecked
|
|
||||||
? 'data-checked="true"'
|
|
||||||
: 'data-checked="false"';
|
|
||||||
return `<li data-type="taskItem" ${checkedAttr}>${text}</li>\n`;
|
|
||||||
},
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
marked.use({
|
|
||||||
extensions: [
|
|
||||||
calloutExtension,
|
|
||||||
githubCalloutExtension,
|
|
||||||
mathBlockExtension,
|
|
||||||
mathInlineExtension,
|
|
||||||
footnoteReferenceExtension,
|
|
||||||
htmlEmbedExtension,
|
|
||||||
],
|
|
||||||
});
|
|
||||||
|
|
||||||
marked.setOptions({ breaks: true });
|
|
||||||
|
|
||||||
export function markdownToHtml(
|
|
||||||
markdownInput: string,
|
|
||||||
): string | Promise<string> {
|
|
||||||
const YAML_FONT_MATTER_REGEX = /^\s*---[\s\S]*?---\s*/;
|
|
||||||
|
|
||||||
const markdown = markdownInput
|
|
||||||
.replace(YAML_FONT_MATTER_REGEX, "")
|
|
||||||
.trimStart();
|
|
||||||
|
|
||||||
// Pull `[^id]: ...` definition lines out of the body, render the body, then
|
|
||||||
// append a single <section data-footnotes> so the round-trip rebuilds the
|
|
||||||
// footnotesList + footnoteDefinition nodes.
|
|
||||||
const { body, section } = extractFootnoteDefinitions(markdown);
|
|
||||||
|
|
||||||
const parsed = marked.parse(body);
|
|
||||||
if (!section) return parsed;
|
|
||||||
|
|
||||||
if (typeof parsed === "string") {
|
|
||||||
return parsed + section;
|
|
||||||
}
|
|
||||||
return parsed.then((html) => html + section);
|
|
||||||
}
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
import { Token, marked } from 'marked';
|
|
||||||
|
|
||||||
interface MathBlockToken {
|
|
||||||
type: 'mathBlock';
|
|
||||||
text: string;
|
|
||||||
raw: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
export const mathBlockExtension = {
|
|
||||||
name: 'mathBlock',
|
|
||||||
level: 'block',
|
|
||||||
start(src: string) {
|
|
||||||
return src.match(/\$\$/)?.index ?? -1;
|
|
||||||
},
|
|
||||||
tokenizer(src: string): MathBlockToken | undefined {
|
|
||||||
const rule = /^\$\$(?!(\$))([\s\S]+?)\$\$/;
|
|
||||||
const match = rule.exec(src);
|
|
||||||
|
|
||||||
if (match) {
|
|
||||||
return {
|
|
||||||
type: 'mathBlock',
|
|
||||||
raw: match[0],
|
|
||||||
text: match[2]?.trim(),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
},
|
|
||||||
renderer(token: Token) {
|
|
||||||
const mathBlockToken = token as MathBlockToken;
|
|
||||||
// parse to prevent escaping slashes
|
|
||||||
const latex = marked
|
|
||||||
.parse(mathBlockToken.text)
|
|
||||||
.toString()
|
|
||||||
.replace(/<(\/)?p>/g, '');
|
|
||||||
|
|
||||||
return `<div data-type="${mathBlockToken.type}" data-katex="true">${latex}</div>`;
|
|
||||||
},
|
|
||||||
};
|
|
||||||
@@ -1,50 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { markdownToHtml } from "./marked.utils";
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Data-integrity regression (issue #204, Phase 2): plain prose that mentions
|
|
||||||
* prices like `$5 and $6` must NOT be misread as inline math. The inline-math
|
|
||||||
* tokenizer mutates a global `marked` singleton at import time
|
|
||||||
* (`marked.utils.ts`), so math behaviour can only be exercised safely through
|
|
||||||
* the public `markdownToHtml`; importing the tokenizer in isolation would give
|
|
||||||
* a different, non-representative result. These assertions therefore drive the
|
|
||||||
* real conversion path.
|
|
||||||
*/
|
|
||||||
function html(md: string): string {
|
|
||||||
const out = markdownToHtml(md);
|
|
||||||
if (typeof out !== "string") throw new Error("expected sync string output");
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
const MATH_MARKERS = ['data-type="mathInline"', 'data-katex="true"'];
|
|
||||||
|
|
||||||
function hasInlineMath(out: string): boolean {
|
|
||||||
return MATH_MARKERS.some((m) => out.includes(m));
|
|
||||||
}
|
|
||||||
|
|
||||||
describe("markdownToHtml: inline-math false positives", () => {
|
|
||||||
it("does not treat prices `$5 and $6` as inline math", () => {
|
|
||||||
const out = html("It costs $5 and $6 today.");
|
|
||||||
expect(hasInlineMath(out)).toBe(false);
|
|
||||||
// The text survives verbatim (no katex span swallowing it).
|
|
||||||
expect(out).toContain("$5 and $6");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("does not treat a single trailing price `$5` as inline math", () => {
|
|
||||||
const out = html("Lunch was $5.");
|
|
||||||
expect(hasInlineMath(out)).toBe(false);
|
|
||||||
expect(out).toContain("$5");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("does not treat `$5, $6, $7` (multiple prices) as inline math", () => {
|
|
||||||
const out = html("Choose $5, $6, $7 plans.");
|
|
||||||
expect(hasInlineMath(out)).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("STILL converts a genuine inline-math expression `$x + y$`", () => {
|
|
||||||
// Guard the positive path so the false-positive guard above can't be
|
|
||||||
// satisfied by simply disabling math entirely.
|
|
||||||
const out = html("The sum $x + y$ is shown.");
|
|
||||||
expect(hasInlineMath(out)).toBe(true);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,55 +0,0 @@
|
|||||||
import { Token, marked } from 'marked';
|
|
||||||
|
|
||||||
interface MathInlineToken {
|
|
||||||
type: 'mathInline';
|
|
||||||
text: string;
|
|
||||||
raw: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
const inlineMathRegex = /^\$(?!\s)(.+?)(?<!\s)\$(?!\d)/;
|
|
||||||
|
|
||||||
export const mathInlineExtension = {
|
|
||||||
name: 'mathInline',
|
|
||||||
level: 'inline',
|
|
||||||
start(src: string) {
|
|
||||||
let index: number;
|
|
||||||
let indexSrc = src;
|
|
||||||
|
|
||||||
while (indexSrc) {
|
|
||||||
index = indexSrc.indexOf('$');
|
|
||||||
if (index === -1) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const f = index === 0 || indexSrc.charAt(index - 1) === ' ';
|
|
||||||
if (f) {
|
|
||||||
const possibleKatex = indexSrc.substring(index);
|
|
||||||
if (possibleKatex.match(inlineMathRegex)) {
|
|
||||||
return index;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
indexSrc = indexSrc.substring(index + 1).replace(/^\$+/, '');
|
|
||||||
}
|
|
||||||
},
|
|
||||||
tokenizer(src: string): MathInlineToken | undefined {
|
|
||||||
const match = inlineMathRegex.exec(src);
|
|
||||||
|
|
||||||
if (match) {
|
|
||||||
return {
|
|
||||||
type: 'mathInline',
|
|
||||||
raw: match[0],
|
|
||||||
text: match[1]?.trim(),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
},
|
|
||||||
renderer(token: Token) {
|
|
||||||
const mathInlineToken = token as MathInlineToken;
|
|
||||||
// parse to prevent escaping slashes
|
|
||||||
const latex = marked
|
|
||||||
.parse(mathInlineToken.text)
|
|
||||||
.toString()
|
|
||||||
.replace(/<(\/)?p>/g, '');
|
|
||||||
|
|
||||||
return `<span data-type="${mathInlineToken.type}" data-katex="true">${latex}</span>`;
|
|
||||||
},
|
|
||||||
};
|
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { getSchema } from "@tiptap/core";
|
|
||||||
import { generateHTML, generateJSON } from "@tiptap/html";
|
|
||||||
import { Document } from "@tiptap/extension-document";
|
|
||||||
import { Paragraph } from "@tiptap/extension-paragraph";
|
|
||||||
import { Text } from "@tiptap/extension-text";
|
|
||||||
import { Bold } from "@tiptap/extension-bold";
|
|
||||||
import { htmlToMarkdown } from "./turndown.utils";
|
|
||||||
import { markdownToHtml } from "./marked.utils";
|
|
||||||
import { Spoiler } from "../../spoiler/spoiler";
|
|
||||||
|
|
||||||
// The spoiler mark has no native Markdown syntax, so it is preserved losslessly
|
|
||||||
// as raw inline HTML (`<span data-spoiler="true">…</span>`), the same approach
|
|
||||||
// htmlEmbed uses. This test drives the full editor round-trip:
|
|
||||||
// JSON -> HTML -> Markdown -> HTML -> JSON
|
|
||||||
// and asserts the `spoiler` mark survives end to end. We use the same
|
|
||||||
// getSchema + @tiptap/html generateHTML/generateJSON utilities the other
|
|
||||||
// editor-ext schema tests use.
|
|
||||||
|
|
||||||
const extensions = [Document, Paragraph, Text, Bold, Spoiler];
|
|
||||||
|
|
||||||
function html(md: string): string {
|
|
||||||
const out = markdownToHtml(md);
|
|
||||||
if (typeof out !== "string") throw new Error("expected sync string output");
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Count text nodes carrying a `spoiler` mark anywhere in a ProseMirror JSON doc.
|
|
||||||
function countSpoilerMarks(doc: any): number {
|
|
||||||
let count = 0;
|
|
||||||
const walk = (node: any) => {
|
|
||||||
if (!node || typeof node !== "object") return;
|
|
||||||
if (Array.isArray(node.marks)) {
|
|
||||||
for (const mark of node.marks) {
|
|
||||||
if (mark?.type === "spoiler") count++;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (Array.isArray(node.content)) node.content.forEach(walk);
|
|
||||||
};
|
|
||||||
walk(doc);
|
|
||||||
return count;
|
|
||||||
}
|
|
||||||
|
|
||||||
describe("Spoiler mark schema", () => {
|
|
||||||
it("registers the spoiler mark in the schema", () => {
|
|
||||||
const schema = getSchema(extensions);
|
|
||||||
expect(schema.marks.spoiler).toBeTruthy();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("recovers the spoiler mark from span[data-spoiler] (HTML -> JSON)", () => {
|
|
||||||
const json = generateJSON(
|
|
||||||
'<p>before <span data-spoiler="true">hidden</span> after</p>',
|
|
||||||
extensions,
|
|
||||||
);
|
|
||||||
expect(countSpoilerMarks(json)).toBe(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("emits data-spoiler + class on render (JSON -> HTML)", () => {
|
|
||||||
const doc = {
|
|
||||||
type: "doc",
|
|
||||||
content: [
|
|
||||||
{
|
|
||||||
type: "paragraph",
|
|
||||||
content: [
|
|
||||||
{
|
|
||||||
type: "text",
|
|
||||||
text: "hidden",
|
|
||||||
marks: [{ type: "spoiler" }],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
};
|
|
||||||
const out = generateHTML(doc, extensions);
|
|
||||||
expect(out).toContain('data-spoiler="true"');
|
|
||||||
expect(out).toContain('class="spoiler"');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("Spoiler Markdown round-trip is lossless", () => {
|
|
||||||
const docWith = (textNode: any) => ({
|
|
||||||
type: "doc",
|
|
||||||
content: [
|
|
||||||
{
|
|
||||||
type: "paragraph",
|
|
||||||
content: [{ type: "text", text: "before " }, textNode, { type: "text", text: " after" }],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
});
|
|
||||||
|
|
||||||
it("preserves the spoiler mark through JSON -> MD -> HTML -> JSON", () => {
|
|
||||||
const startDoc = docWith({
|
|
||||||
type: "text",
|
|
||||||
text: "hidden",
|
|
||||||
marks: [{ type: "spoiler" }],
|
|
||||||
});
|
|
||||||
|
|
||||||
// JSON -> HTML
|
|
||||||
const html1 = generateHTML(startDoc, extensions);
|
|
||||||
expect(html1).toContain('data-spoiler="true"');
|
|
||||||
|
|
||||||
// HTML -> Markdown (raw inline HTML, lossless)
|
|
||||||
const md = htmlToMarkdown(html1);
|
|
||||||
expect(md).toContain('<span data-spoiler="true">hidden</span>');
|
|
||||||
|
|
||||||
// MD -> HTML -> JSON (mark restored via parseHTML)
|
|
||||||
const endJson = generateJSON(html(md), extensions);
|
|
||||||
expect(countSpoilerMarks(endJson)).toBe(1);
|
|
||||||
// The visible text survives.
|
|
||||||
expect(JSON.stringify(endJson)).toContain("hidden");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("keeps the spoiler intact when it intersects a bold mark", () => {
|
|
||||||
const startDoc = docWith({
|
|
||||||
type: "text",
|
|
||||||
text: "secret",
|
|
||||||
marks: [{ type: "bold" }, { type: "spoiler" }],
|
|
||||||
});
|
|
||||||
|
|
||||||
const md = htmlToMarkdown(generateHTML(startDoc, extensions));
|
|
||||||
expect(md).toContain("data-spoiler=\"true\"");
|
|
||||||
|
|
||||||
const endJson = generateJSON(html(md), extensions);
|
|
||||||
expect(countSpoilerMarks(endJson)).toBe(1);
|
|
||||||
// Bold survives alongside the spoiler.
|
|
||||||
expect(JSON.stringify(endJson)).toContain('"bold"');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
// Map @joplin/turndown types to @types/turndown
|
|
||||||
declare module "@joplin/turndown" {
|
|
||||||
import TurndownService from "turndown";
|
|
||||||
export = TurndownService;
|
|
||||||
}
|
|
||||||
|
|
||||||
declare module "@joplin/turndown-plugin-gfm" {
|
|
||||||
import TurndownService from "turndown";
|
|
||||||
export const tables: TurndownService.Plugin;
|
|
||||||
export const strikethrough: TurndownService.Plugin;
|
|
||||||
export const highlightedCodeBlock: TurndownService.Plugin;
|
|
||||||
}
|
|
||||||
@@ -1,147 +0,0 @@
|
|||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import { htmlToMarkdown } from "./turndown.utils";
|
|
||||||
import { markdownToHtml } from "./marked.utils";
|
|
||||||
|
|
||||||
/**
|
|
||||||
* #206 mdrt-2 — Markdown export must never SILENTLY drop a block. (FIXED)
|
|
||||||
*
|
|
||||||
* `htmlToMarkdown` (turndown) historically only registered rules for a fixed
|
|
||||||
* set of custom nodes (callout, taskItem, details, math, iframe, htmlEmbed,
|
|
||||||
* image, video, footnote). Any other custom node — `transclusionReference`,
|
|
||||||
* `pageBreak`, `mention`, `status` — fell through to turndown's default
|
|
||||||
* handling: an empty wrapper is "blank" and removed, so the block disappeared
|
|
||||||
* from the exported Markdown with no trace, and `mention`/`status` collapsed to
|
|
||||||
* bare text, losing their identity (data-id / data-color). The invariant
|
|
||||||
* "never silently lose a block" was broken.
|
|
||||||
*
|
|
||||||
* The fix adds lossless turndown rules that re-emit each of these nodes as raw
|
|
||||||
* HTML carrying every `data-*` attribute. Plain-Markdown viewers ignore the
|
|
||||||
* inert tag; the import path round-trips it (`markdownToHtml` passes the raw
|
|
||||||
* HTML through and each node's `parseHTML` rebuilds the ProseMirror node). These
|
|
||||||
* tests assert the surviving contract (the block is preserved AND its identity
|
|
||||||
* round-trips back through import).
|
|
||||||
*/
|
|
||||||
describe("htmlToMarkdown — custom nodes are preserved losslessly (#206 mdrt-2)", () => {
|
|
||||||
const wrap = (inner: string) => `<p>before</p>${inner}<p>after</p>`;
|
|
||||||
|
|
||||||
it("preserves a pageBreak block on Markdown export", () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
wrap('<div data-type="pageBreak" class="page-break"></div>'),
|
|
||||||
);
|
|
||||||
expect(md).toContain("before");
|
|
||||||
expect(md).toContain("after");
|
|
||||||
// The break survives as an inert raw-HTML tag, not silently dropped.
|
|
||||||
expect(md).toMatch(/data-type="pageBreak"/);
|
|
||||||
expect(md).toMatch(/page-?break/i);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("preserves a transclusionReference's identity on Markdown export", () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
|
|
||||||
);
|
|
||||||
expect(md).toContain("before");
|
|
||||||
expect(md).toContain("after");
|
|
||||||
// The data-id (the only thing that gives the reference identity) survives.
|
|
||||||
expect(md).toContain("abc");
|
|
||||||
expect(md).toMatch(/data-type="transclusionReference"/);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("preserves a mention's data-id (stable identity) on Markdown export", () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
|
|
||||||
);
|
|
||||||
// The mention keeps its stable identity (data-id), not just the text.
|
|
||||||
expect(md).toContain("u1");
|
|
||||||
expect(md).toContain("Bob");
|
|
||||||
expect(md).toMatch(/data-type="mention"/);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("preserves a status chip's color on Markdown export", () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
'<p>s <span data-type="status" data-color="green">Done</span></p>',
|
|
||||||
);
|
|
||||||
// The chip's color (its identity) survives, not just the visible text.
|
|
||||||
expect(md).toContain("green");
|
|
||||||
expect(md).toContain("Done");
|
|
||||||
expect(md).toMatch(/data-type="status"/);
|
|
||||||
});
|
|
||||||
|
|
||||||
// The export form is only lossless if the import path can rebuild it. These
|
|
||||||
// assert the full MD -> HTML round-trip restores the node + its attributes,
|
|
||||||
// which is the marker <-> node contract each `parseHTML` relies on.
|
|
||||||
describe("import round-trip (markdownToHtml restores the node)", () => {
|
|
||||||
it("round-trips a pageBreak through export + import", async () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
wrap('<div data-type="pageBreak" class="page-break"></div>'),
|
|
||||||
);
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
expect(html).toMatch(/<div[^>]*data-type="pageBreak"[^>]*>/);
|
|
||||||
expect(html).toContain("before");
|
|
||||||
expect(html).toContain("after");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("round-trips a transclusionReference (keeps data-id)", async () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
|
|
||||||
);
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
expect(html).toMatch(/<div[^>]*data-type="transclusionReference"[^>]*>/);
|
|
||||||
expect(html).toContain("abc");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("round-trips a mention (keeps data-id + data-label)", async () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
|
|
||||||
);
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
expect(html).toMatch(/<span[^>]*data-type="mention"[^>]*>/);
|
|
||||||
expect(html).toContain("u1");
|
|
||||||
expect(html).toContain("Bob");
|
|
||||||
});
|
|
||||||
|
|
||||||
it("round-trips a status chip (keeps data-color)", async () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
'<p>s <span data-type="status" data-color="green">Done</span></p>',
|
|
||||||
);
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
expect(html).toMatch(/<span[^>]*data-type="status"[^>]*>/);
|
|
||||||
expect(html).toContain("green");
|
|
||||||
});
|
|
||||||
|
|
||||||
// HTML special chars in an attribute value or in a node's text must be
|
|
||||||
// ESCAPED when re-emitted as raw HTML, otherwise the exported tag is
|
|
||||||
// malformed and `markdownToHtml`'s parser cannot restore the original value
|
|
||||||
// (the same silent data loss this PR fixes). Dropping `<`/`>` escaping is the
|
|
||||||
// dangerous regression: a stray `<` or `>` corrupts the tag (or injects new
|
|
||||||
// markup), so the test data carries ALL of `&`, `"`, `<`, `>` in BOTH the
|
|
||||||
// data-label attribute and the visible text. That fully exercises
|
|
||||||
// escapeHtmlAttr's `&,",<,>` branches and escapeHtmlText's `&,<,>` branches
|
|
||||||
// (escapeHtmlText leaves `"` literal); the alphanumeric-only cases above hit
|
|
||||||
// none of them.
|
|
||||||
it("escapes HTML special chars (& \" < >) in attrs + text and round-trips them", async () => {
|
|
||||||
const md = htmlToMarkdown(
|
|
||||||
`<p>hi <span data-type="mention" data-id="u1" data-label="A & <B> "C"">@A & <B> "C"</span> there</p>`,
|
|
||||||
);
|
|
||||||
|
|
||||||
// (a) The exported Markdown carries a WELL-FORMED, correctly-escaped tag:
|
|
||||||
// the attribute escapes `&`, `<`, `>` AND `"`; the text escapes `&`, `<`,
|
|
||||||
// `>` (a `"` inside text content is legal, so it stays literal).
|
|
||||||
expect(md).toContain('data-label="A & <B> "C""');
|
|
||||||
expect(md).toContain('>@A & <B> "C"</span>');
|
|
||||||
// And explicitly NOT the raw, tag-corrupting forms: a literal `<B>` (would
|
|
||||||
// mean `<`/`>` escaping was dropped in either the attr or the text)...
|
|
||||||
expect(md).not.toContain("<B>");
|
|
||||||
// ...nor the malformed attribute that an unescaped `"` would produce.
|
|
||||||
expect(md).not.toContain('data-label="A & <B> "C""');
|
|
||||||
|
|
||||||
// (b) Import restores the ORIGINAL (unescaped) values, attribute and text.
|
|
||||||
const html = await markdownToHtml(md);
|
|
||||||
const dom = new DOMParser().parseFromString(html as string, "text/html");
|
|
||||||
const span = dom.querySelector('span[data-type="mention"]');
|
|
||||||
expect(span).not.toBeNull();
|
|
||||||
expect(span!.getAttribute("data-id")).toBe("u1");
|
|
||||||
expect(span!.getAttribute("data-label")).toBe('A & <B> "C"');
|
|
||||||
expect(span!.textContent).toBe('@A & <B> "C"');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -1,488 +0,0 @@
|
|||||||
import * as _TurndownService from '@joplin/turndown';
|
|
||||||
import * as TurndownPluginGfm from '@joplin/turndown-plugin-gfm';
|
|
||||||
import { getBasename } from './basename';
|
|
||||||
|
|
||||||
// CJS/ESM interop: .default exists in Vite, not in NestJS
|
|
||||||
const TurndownService = (_TurndownService as any).default || _TurndownService;
|
|
||||||
|
|
||||||
function sanitizeMdLinkText(value: string): string {
|
|
||||||
return value
|
|
||||||
.replace(/\\/g, '\\\\')
|
|
||||||
.replace(/([\[\]!])/g, '\\$1')
|
|
||||||
.replace(/[\r\n]+/g, ' ');
|
|
||||||
}
|
|
||||||
|
|
||||||
// Tags turndown treats as void (self-closing). Footnote references render as an
|
|
||||||
// empty <sup data-footnote-ref> whose meaning lives entirely in its data-id;
|
|
||||||
// without marking it void, turndown's blank-node removal drops it before our
|
|
||||||
// rule runs, losing the `[^id]` marker. Mirrors turndown's built-in list.
|
|
||||||
const TURNDOWN_VOID_ELEMENTS = [
|
|
||||||
'AREA', 'BASE', 'BR', 'COL', 'COMMAND', 'EMBED', 'HR', 'IMG', 'INPUT',
|
|
||||||
'KEYGEN', 'LINK', 'META', 'PARAM', 'SOURCE', 'TRACK', 'WBR',
|
|
||||||
];
|
|
||||||
|
|
||||||
function isVoidNode(node: any): boolean {
|
|
||||||
const name = node?.nodeName?.toUpperCase?.();
|
|
||||||
if (!name) return false;
|
|
||||||
if (name === 'SUP' && node.hasAttribute?.('data-footnote-ref')) {
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
return TURNDOWN_VOID_ELEMENTS.indexOf(name) !== -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* An empty <sup data-footnote-ref> is "blank" to turndown, which removes blank
|
|
||||||
* inline nodes (RootNode/Node use a module-level isVoid the options cannot
|
|
||||||
* override). To survive, inject the id as text content so the node is non-blank;
|
|
||||||
* the footnoteReference rule then reads data-id and emits `[^id]`.
|
|
||||||
*/
|
|
||||||
function fillEmptyFootnoteRefs(html: string): string {
|
|
||||||
return html.replace(
|
|
||||||
/<sup\b([^>]*\bdata-footnote-ref\b[^>]*)>\s*<\/sup>/gi,
|
|
||||||
(_m, attrs) => `<sup${attrs}></sup>`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* `pageBreak` and `transclusionReference` are childless atom <div>s. Like an
|
|
||||||
* empty footnote ref (see above), turndown treats a childless block as "blank"
|
|
||||||
* and replaces it with the blankRule BEFORE any custom rule can fire — so the
|
|
||||||
* node disappears from the export with no trace (#206 mdrt-2). Inject a
|
|
||||||
* zero-width space so the node is non-blank and our lossless rule runs; the
|
|
||||||
* rule rebuilds the tag from the element's attributes, so the injected char
|
|
||||||
* never reaches the output.
|
|
||||||
*/
|
|
||||||
function fillEmptyAtomBlocks(html: string): string {
|
|
||||||
return html.replace(
|
|
||||||
/<div\b([^>]*\bdata-type="(?:pageBreak|transclusionReference)"[^>]*)>\s*<\/div>/gi,
|
|
||||||
(_m, attrs) => `<div${attrs}></div>`,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** HTML-escape an attribute value so a re-emitted raw-HTML tag is well-formed. */
|
|
||||||
function escapeHtmlAttr(value: string): string {
|
|
||||||
return value
|
|
||||||
.replace(/&/g, '&')
|
|
||||||
.replace(/"/g, '"')
|
|
||||||
.replace(/</g, '<')
|
|
||||||
.replace(/>/g, '>');
|
|
||||||
}
|
|
||||||
|
|
||||||
/** HTML-escape text placed inside a re-emitted raw-HTML element. */
|
|
||||||
function escapeHtmlText(value: string): string {
|
|
||||||
return value
|
|
||||||
.replace(/&/g, '&')
|
|
||||||
.replace(/</g, '<')
|
|
||||||
.replace(/>/g, '>');
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Serialize ALL of an element's attributes back to a raw-HTML attribute string
|
|
||||||
* (leading space included). Generic on purpose: a custom node's identity lives
|
|
||||||
* entirely in its `data-*` attributes (data-id, data-color, data-source-page-id,
|
|
||||||
* data-transclusion-id, …), and serializing every attribute keeps the export
|
|
||||||
* lossless regardless of which attributes a given node carries.
|
|
||||||
*/
|
|
||||||
function serializeAttrs(node: any): string {
|
|
||||||
const attrs = node?.attributes;
|
|
||||||
if (!attrs) return '';
|
|
||||||
return Array.from(attrs as ArrayLike<{ name: string; value: string }>)
|
|
||||||
.map((attr) => ` ${attr.name}="${escapeHtmlAttr(attr.value ?? '')}"`)
|
|
||||||
.join('');
|
|
||||||
}
|
|
||||||
|
|
||||||
export function htmlToMarkdown(html: string): string {
|
|
||||||
const turndownService = new TurndownService({
|
|
||||||
headingStyle: 'atx',
|
|
||||||
codeBlockStyle: 'fenced',
|
|
||||||
hr: '---',
|
|
||||||
bulletListMarker: '-',
|
|
||||||
isVoid: isVoidNode,
|
|
||||||
});
|
|
||||||
|
|
||||||
turndownService.use([
|
|
||||||
TurndownPluginGfm.tables,
|
|
||||||
TurndownPluginGfm.strikethrough,
|
|
||||||
TurndownPluginGfm.highlightedCodeBlock,
|
|
||||||
taskList,
|
|
||||||
callout,
|
|
||||||
preserveDetail,
|
|
||||||
listParagraph,
|
|
||||||
orderedListItem,
|
|
||||||
mathInline,
|
|
||||||
mathBlock,
|
|
||||||
iframeEmbed,
|
|
||||||
htmlEmbed,
|
|
||||||
spoiler,
|
|
||||||
image,
|
|
||||||
video,
|
|
||||||
footnoteReference,
|
|
||||||
footnotesList,
|
|
||||||
pageBreak,
|
|
||||||
transclusionReference,
|
|
||||||
mention,
|
|
||||||
status,
|
|
||||||
]);
|
|
||||||
return turndownService
|
|
||||||
.turndown(fillEmptyAtomBlocks(fillEmptyFootnoteRefs(html)))
|
|
||||||
.replaceAll('<br>', ' ');
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Lossless export rules for custom nodes that have NO native Markdown syntax
|
|
||||||
* (#206 mdrt-2). Markdown cannot represent a page break, a transclusion
|
|
||||||
* reference, a mention's stable id, or a status chip's color — so rather than
|
|
||||||
* letting turndown silently drop them, each rule re-emits the node as raw HTML
|
|
||||||
* carrying every `data-*` attribute. Plain-Markdown viewers ignore the inert
|
|
||||||
* tag, and the import path round-trips it: `markdownToHtml` passes raw HTML
|
|
||||||
* through and each node's `parseHTML` (`div[data-type="…"]`, `span[…]`) rebuilds
|
|
||||||
* the ProseMirror node with its attributes intact.
|
|
||||||
*/
|
|
||||||
function pageBreak(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('pageBreak', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'DIV' &&
|
|
||||||
node.getAttribute('data-type') === 'pageBreak'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function transclusionReference(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('transclusionReference', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'DIV' &&
|
|
||||||
node.getAttribute('data-type') === 'transclusionReference'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function mention(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('mention', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'SPAN' &&
|
|
||||||
node.getAttribute('data-type') === 'mention'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const text = escapeHtmlText(node.textContent || '');
|
|
||||||
return `<span${serializeAttrs(node)}>${text}</span>`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function status(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('status', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'SPAN' && node.getAttribute('data-type') === 'status'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const text = escapeHtmlText(node.textContent || '');
|
|
||||||
return `<span${serializeAttrs(node)}>${text}</span>`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Serialize the `htmlEmbed` node to Markdown.
|
|
||||||
*
|
|
||||||
* Markdown has no native representation for an arbitrary-HTML block, so we
|
|
||||||
* preserve the node losslessly as an HTML comment carrying the base64-encoded
|
|
||||||
* source (the same `data-source` payload the node stores). `markdownToHtml`
|
|
||||||
* recognizes the same marker and rebuilds the node, so the round-trip
|
|
||||||
* MD -> HTML -> JSON keeps the source intact. The comment also keeps the raw
|
|
||||||
* markup inert in the exported `.md` file (it does not render in plain Markdown
|
|
||||||
* viewers).
|
|
||||||
*/
|
|
||||||
function htmlEmbed(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('htmlEmbed', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'DIV' &&
|
|
||||||
node.getAttribute('data-type') === 'htmlEmbed'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const encoded = node.getAttribute('data-source') || '';
|
|
||||||
return `\n\n<!--html-embed:${encoded}-->\n\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Serialize the `spoiler` inline mark to lossless raw inline HTML.
|
|
||||||
*
|
|
||||||
* Markdown has no native spoiler syntax, so we emit the same `<span
|
|
||||||
* data-spoiler="true">…</span>` the mark renders. `marked` passes inline raw HTML
|
|
||||||
* through untouched, and `generateJSON` restores the mark via its parseHTML, so
|
|
||||||
* the round-trip MD -> HTML -> JSON keeps the spoiler intact. The UI-only
|
|
||||||
* `is-revealed` state is never serialized.
|
|
||||||
*/
|
|
||||||
function spoiler(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('spoiler', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'SPAN' &&
|
|
||||||
node.getAttribute('data-spoiler') === 'true'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (content: string) {
|
|
||||||
return `<span data-spoiler="true">${content}</span>`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function listParagraph(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('paragraph', {
|
|
||||||
filter: ['p'],
|
|
||||||
replacement: (content: string, node: HTMLInputElement) => {
|
|
||||||
if (node.parentElement?.nodeName === 'LI') {
|
|
||||||
return content;
|
|
||||||
}
|
|
||||||
return `\n\n${content}\n\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function orderedListItem(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('orderedListItem', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return node.nodeName === 'LI' && node.getAttribute('data-type') !== 'taskItem';
|
|
||||||
},
|
|
||||||
replacement: (content: string, node: HTMLInputElement, options: any) => {
|
|
||||||
const parent = node.parentNode as HTMLElement;
|
|
||||||
if (parent.nodeName !== 'OL' && parent.nodeName !== 'UL') {
|
|
||||||
return content;
|
|
||||||
}
|
|
||||||
|
|
||||||
content = content
|
|
||||||
.replace(/^\n+/, '')
|
|
||||||
.replace(/\n+$/, '\n')
|
|
||||||
.replace(/\n/gm, '\n ');
|
|
||||||
|
|
||||||
let prefix: string;
|
|
||||||
if (parent.nodeName === 'OL') {
|
|
||||||
const start = parseInt(parent.getAttribute('start') || '1', 10);
|
|
||||||
const index = Array.prototype.indexOf.call(parent.children, node);
|
|
||||||
prefix = `${start + index}. `;
|
|
||||||
} else {
|
|
||||||
prefix = `${options.bulletListMarker} `;
|
|
||||||
}
|
|
||||||
|
|
||||||
return (
|
|
||||||
prefix +
|
|
||||||
content +
|
|
||||||
(node.nextSibling && !/\n$/.test(content) ? '\n' : '')
|
|
||||||
);
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function callout(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('callout', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'DIV' && node.getAttribute('data-type') === 'callout'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (content: string, node: HTMLInputElement) {
|
|
||||||
const calloutType = node.getAttribute('data-callout-type');
|
|
||||||
return `\n\n:::${calloutType}\n${content.trim()}\n:::\n\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function taskList(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('taskListItem', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.getAttribute('data-type') === 'taskItem' &&
|
|
||||||
node.parentNode.nodeName === 'UL'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const isChecked = node.getAttribute('data-checked') === 'true';
|
|
||||||
const div = node.querySelector('div');
|
|
||||||
const text = div ? div.textContent.trim() : node.textContent.trim();
|
|
||||||
|
|
||||||
const prefix = `- ${isChecked ? '[x]' : '[ ]'} `;
|
|
||||||
|
|
||||||
return (
|
|
||||||
prefix +
|
|
||||||
text +
|
|
||||||
(node.nextSibling && !/\n$/.test(text) ? '\n' : '')
|
|
||||||
);
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function preserveDetail(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('preserveDetail', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return node.nodeName === 'DETAILS';
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const summary = node.querySelector(':scope > summary');
|
|
||||||
let detailSummary = '';
|
|
||||||
|
|
||||||
if (summary) {
|
|
||||||
detailSummary = `<summary>${turndownService.turndown(summary.innerHTML)}</summary>`;
|
|
||||||
}
|
|
||||||
|
|
||||||
const detailsContent = Array.from(node.childNodes)
|
|
||||||
.filter((child) => child.nodeName !== 'SUMMARY')
|
|
||||||
.map((child) =>
|
|
||||||
child.nodeType === 1
|
|
||||||
? turndownService.turndown((child as HTMLElement).outerHTML)
|
|
||||||
: child.textContent,
|
|
||||||
)
|
|
||||||
.join('');
|
|
||||||
|
|
||||||
return `\n<details>\n${detailSummary}\n\n${detailsContent}\n\n</details>\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function mathInline(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('mathInline', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'SPAN' &&
|
|
||||||
node.getAttribute('data-type') === 'mathInline'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (content: string) {
|
|
||||||
return `$${content}$`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function mathBlock(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('mathBlock', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'DIV' &&
|
|
||||||
node.getAttribute('data-type') === 'mathBlock'
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (content: string) {
|
|
||||||
return `\n$$\n${content}\n$$\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function iframeEmbed(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('iframeEmbed', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return node.nodeName === 'IFRAME';
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const src = node.getAttribute('src');
|
|
||||||
return '[' + src + '](' + src + ')';
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function image(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('image', {
|
|
||||||
filter: 'img',
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const src = node.getAttribute('src') || '';
|
|
||||||
if (!src) return '';
|
|
||||||
const caption = node.getAttribute('data-caption') || '';
|
|
||||||
if (caption) {
|
|
||||||
// ![]() can't carry a caption, so emit a raw <img> wrapped in a block
|
|
||||||
// <div>. marked passes it through and the image extension's parseHTML
|
|
||||||
// restores the caption from data-caption.
|
|
||||||
const parts = [`src="${escapeHtmlAttr(src)}"`];
|
|
||||||
const alt = node.getAttribute('alt') || '';
|
|
||||||
if (alt) parts.push(`alt="${escapeHtmlAttr(alt)}"`);
|
|
||||||
parts.push(`data-caption="${escapeHtmlAttr(caption)}"`);
|
|
||||||
return `<div><img ${parts.join(' ')}></div>`;
|
|
||||||
}
|
|
||||||
const alt = sanitizeMdLinkText(node.getAttribute('alt') || '');
|
|
||||||
const title = node.getAttribute('title') || '';
|
|
||||||
const titlePart = title ? ' "' + title.replace(/"/g, '\\"') + '"' : '';
|
|
||||||
return '';
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Footnote reference (inline atom) -> pandoc/GFM marker `[^id]`.
|
|
||||||
* The visible number is derived (not stored), so the id is the stable anchor.
|
|
||||||
*/
|
|
||||||
function footnoteReference(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('footnoteReference', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'SUP' && node.hasAttribute('data-footnote-ref')
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const id = node.getAttribute('data-id') || '';
|
|
||||||
return id ? `[^${id}]` : '';
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Footnotes container -> the list of `[^id]: text` definitions at the end of
|
|
||||||
* the document (one per line). Each footnoteDefinition inside emits its own
|
|
||||||
* `[^id]: ...` line; turndown joins them with the surrounding block spacing.
|
|
||||||
*/
|
|
||||||
function footnotesList(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('footnoteDefinition', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'DIV' && node.hasAttribute('data-footnote-def')
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (content: string, node: HTMLInputElement) {
|
|
||||||
const id = node.getAttribute('data-id') || '';
|
|
||||||
// Collapse internal newlines so the definition stays a single MD line;
|
|
||||||
// continuation lines are a v2 refinement.
|
|
||||||
const text = content.replace(/\s*\n+\s*/g, ' ').trim();
|
|
||||||
return id ? `\n[^${id}]: ${text}\n` : '';
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
turndownService.addRule('footnotesList', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return (
|
|
||||||
node.nodeName === 'SECTION' && node.hasAttribute('data-footnotes')
|
|
||||||
);
|
|
||||||
},
|
|
||||||
replacement: function (content: string) {
|
|
||||||
return `\n\n${content.trim()}\n`;
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function video(turndownService: _TurndownService) {
|
|
||||||
turndownService.addRule('video', {
|
|
||||||
filter: function (node: HTMLInputElement) {
|
|
||||||
return node.tagName === 'VIDEO';
|
|
||||||
},
|
|
||||||
replacement: function (_content: string, node: HTMLInputElement) {
|
|
||||||
const src = node.getAttribute('src') || '';
|
|
||||||
const ariaLabel = node.getAttribute('aria-label');
|
|
||||||
const name = sanitizeMdLinkText(
|
|
||||||
ariaLabel || getBasename(src) || src,
|
|
||||||
);
|
|
||||||
return '[' + name + '](' + src + ')';
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
@@ -14,10 +14,15 @@ export default defineConfig({
|
|||||||
provider: "v8",
|
provider: "v8",
|
||||||
reporter: ["text-summary", "text"],
|
reporter: ["text-summary", "text"],
|
||||||
all: false,
|
all: false,
|
||||||
|
// functions lowered 60 -> 57 after issue #347 removed the editor-ext
|
||||||
|
// markdown layer (src/lib/markdown) and its image/footnote round-trip
|
||||||
|
// specs: that markdown behavior now lives in — and is tested by —
|
||||||
|
// @docmost/prosemirror-markdown, so the editor-ext baseline shifts down.
|
||||||
|
// Still a real gate (a few points below the post-removal measured level).
|
||||||
thresholds: {
|
thresholds: {
|
||||||
statements: 54,
|
statements: 54,
|
||||||
branches: 44,
|
branches: 44,
|
||||||
functions: 60,
|
functions: 57,
|
||||||
lines: 54,
|
lines: 54,
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -72,7 +72,13 @@ export async function stabilizePageFile(
|
|||||||
* keeps re-pulls of an unchanged page byte-identical (no churn, loop-guard).
|
* keeps re-pulls of an unchanged page byte-identical (no churn, loop-guard).
|
||||||
*/
|
*/
|
||||||
export async function stabilizePageBody(content: unknown): Promise<string> {
|
export async function stabilizePageBody(content: unknown): Promise<string> {
|
||||||
const md1 = convertProseMirrorToMarkdown(content);
|
// git-sync is the LOSSLESS mirror path, so run the serializer in `strict`
|
||||||
|
// mode: a node/mark type the converter has no case for (e.g. one added to the
|
||||||
|
// schema without a matching serializer arm) throws a ConverterLossError here
|
||||||
|
// rather than silently degrading — surfacing the loss loudly at write time
|
||||||
|
// instead of committing a lossy file. Valid content (every current schema type
|
||||||
|
// has a case) is unaffected.
|
||||||
|
const md1 = convertProseMirrorToMarkdown(content, { strict: true });
|
||||||
const doc2 = await markdownToProseMirror(md1);
|
const doc2 = await markdownToProseMirror(md1);
|
||||||
return convertProseMirrorToMarkdown(doc2);
|
return convertProseMirrorToMarkdown(doc2, { strict: true });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import { stabilizePageFile, type PageMeta } from '../src/engine/stabilize.js';
|
|||||||
// global DOM via jsdom at module load time (required for @tiptap/html under Node).
|
// global DOM via jsdom at module load time (required for @tiptap/html under Node).
|
||||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
||||||
import { parseDocmostMarkdown } from '@docmost/prosemirror-markdown';
|
import { parseDocmostMarkdown } from '@docmost/prosemirror-markdown';
|
||||||
|
import { ConverterLossError } from '@docmost/prosemirror-markdown';
|
||||||
|
|
||||||
// stabilize.ts (SPEC §11 normalize-on-write) was 0% covered (only the gated e2e
|
// stabilize.ts (SPEC §11 normalize-on-write) was 0% covered (only the gated e2e
|
||||||
// touched it). stabilizePageFile is import-testable: build a small ProseMirror
|
// touched it). stabilizePageFile is import-testable: build a small ProseMirror
|
||||||
@@ -66,6 +67,23 @@ describe('stabilizePageFile — normalize-on-write fixpoint (SPEC §11)', () =>
|
|||||||
expect(body1).toContain('data-src="/d.drawio"');
|
expect(body1).toContain('data-src="/d.drawio"');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('runs the serializer in STRICT mode — an unmappable node throws, not a lossy write (#493)', async () => {
|
||||||
|
// git-sync is the lossless mirror path: a node type the converter has no
|
||||||
|
// case for (here a fabricated one, standing in for a schema type added
|
||||||
|
// without a matching serializer arm) must surface loudly at write time
|
||||||
|
// rather than being silently flattened into a lossy .md file.
|
||||||
|
const content = {
|
||||||
|
type: 'doc',
|
||||||
|
content: [
|
||||||
|
{ type: 'paragraph', content: [{ type: 'text', text: 'ok' }] },
|
||||||
|
{ type: 'quantumWidget', content: [{ type: 'text', text: 'lost?' }] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
await expect(stabilizePageFile(content, meta)).rejects.toBeInstanceOf(
|
||||||
|
ConverterLossError,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it('already-stable content is unchanged by the pass (idempotent)', async () => {
|
it('already-stable content is unchanged by the pass (idempotent)', async () => {
|
||||||
// Plain prose is already a fixpoint; stabilizing it once and twice agree.
|
// Plain prose is already a fixpoint; stabilizing it once and twice agree.
|
||||||
const content = {
|
const content = {
|
||||||
|
|||||||
@@ -10,7 +10,10 @@ import { JSDOM } from "jsdom";
|
|||||||
// handled there). MCP consumes it directly instead of maintaining its own
|
// handled there). MCP consumes it directly instead of maintaining its own
|
||||||
// drifted marked pipeline; only the collab/yjs write glue and the footnote
|
// drifted marked pipeline; only the collab/yjs write glue and the footnote
|
||||||
// canonicalization wrapper stay mcp-side.
|
// canonicalization wrapper stay mcp-side.
|
||||||
import { markdownToProseMirror } from "@docmost/prosemirror-markdown";
|
import {
|
||||||
|
markdownToProseMirror,
|
||||||
|
normalizeAgentMarkdown,
|
||||||
|
} from "@docmost/prosemirror-markdown";
|
||||||
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
|
import { docmostExtensions, docmostSchema } from "./docmost-schema.js";
|
||||||
import { withPageLock } from "./page-lock.js";
|
import { withPageLock } from "./page-lock.js";
|
||||||
import {
|
import {
|
||||||
@@ -20,6 +23,7 @@ import {
|
|||||||
} from "@docmost/prosemirror-markdown";
|
} from "@docmost/prosemirror-markdown";
|
||||||
import { canonicalizeFootnotes } from "./footnote-canonicalize.js";
|
import { canonicalizeFootnotes } from "./footnote-canonicalize.js";
|
||||||
import { normalizeAndMergeFootnotes } from "./footnote-normalize-merge.js";
|
import { normalizeAndMergeFootnotes } from "./footnote-normalize-merge.js";
|
||||||
|
import { regraftResolvedComments } from "./comment-anchor.js";
|
||||||
import { VerifyReport } from "./diff.js";
|
import { VerifyReport } from "./diff.js";
|
||||||
import { acquireCollabSession } from "./collab-session.js";
|
import { acquireCollabSession } from "./collab-session.js";
|
||||||
|
|
||||||
@@ -97,6 +101,15 @@ global.WebSocket = WebSocket;
|
|||||||
* plain `markdownToProseMirror` (no canonicalization) — safe now because inline
|
* plain `markdownToProseMirror` (no canonicalization) — safe now because inline
|
||||||
* `^[body]` footnotes carry their body at the reference point, so a comment can
|
* `^[body]` footnotes carry their body at the reference point, so a comment can
|
||||||
* no longer produce a reference-less footnote definition to be dropped.
|
* no longer produce a reference-less footnote definition to be dropped.
|
||||||
|
*
|
||||||
|
* #493: `normalizeAgentMarkdown` runs FIRST, so an agent's `updatePageMarkdown`
|
||||||
|
* body gets the SAME GFM `[^id]` reference-footnote -> inline `^[body]` rewrite as
|
||||||
|
* the server import path (instead of the reference leaking as literal text / a
|
||||||
|
* bogus link). It DELIBERATELY does NOT strip a leading YAML front-matter block:
|
||||||
|
* a full-body agent rewrite that opens with a `---…---` is (almost) always a
|
||||||
|
* horizontalRule the serializer emitted, and stripping it would silently drop the
|
||||||
|
* page's leading content (#493 review). The front-matter strip stays on the
|
||||||
|
* server FILE-import boundary only (`normalizeForeignMarkdown`).
|
||||||
*/
|
*/
|
||||||
export async function markdownToProseMirrorCanonical(
|
export async function markdownToProseMirrorCanonical(
|
||||||
markdownContent: string,
|
markdownContent: string,
|
||||||
@@ -105,7 +118,9 @@ export async function markdownToProseMirrorCanonical(
|
|||||||
// canonicalizing, so the canonicalizer re-hangs references and drops the
|
// canonicalizing, so the canonicalizer re-hangs references and drops the
|
||||||
// now-orphaned duplicate definitions.
|
// now-orphaned duplicate definitions.
|
||||||
return canonicalizeFootnotes(
|
return canonicalizeFootnotes(
|
||||||
normalizeAndMergeFootnotes(await markdownToProseMirror(markdownContent)),
|
normalizeAndMergeFootnotes(
|
||||||
|
await markdownToProseMirror(normalizeAgentMarkdown(markdownContent)),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -328,6 +343,12 @@ export async function updatePageContentRealtime(
|
|||||||
pageId,
|
pageId,
|
||||||
collabToken,
|
collabToken,
|
||||||
baseUrl,
|
baseUrl,
|
||||||
() => tiptapJson,
|
// #493: an agent read HIDES resolved-comment anchors (#337), so the markdown
|
||||||
|
// it sends here no longer carries them — a naive full rewrite would erase
|
||||||
|
// every resolved comment mark. Re-graft the resolved marks from the LIVE doc
|
||||||
|
// onto the matching text in the freshly-imported body. Active comments are
|
||||||
|
// untouched (they ride through the markdown themselves); a resolved span whose
|
||||||
|
// text the agent changed simply does not re-anchor and is dropped.
|
||||||
|
(liveDoc) => regraftResolvedComments(liveDoc, tiptapJson),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -312,10 +312,9 @@ export function canAnchorInDoc(doc: any, selection: string): boolean {
|
|||||||
function spliceCommentMark(
|
function spliceCommentMark(
|
||||||
blockContent: any[],
|
blockContent: any[],
|
||||||
match: AnchorMatch,
|
match: AnchorMatch,
|
||||||
commentId: string,
|
commentMark: any,
|
||||||
): void {
|
): void {
|
||||||
const { startChild, startOffset, endChild, endOffset } = match;
|
const { startChild, startOffset, endChild, endOffset } = match;
|
||||||
const commentMark = makeCommentMark(commentId);
|
|
||||||
const fragments: any[] = [];
|
const fragments: any[] = [];
|
||||||
|
|
||||||
for (let k = startChild; k <= endChild; k++) {
|
for (let k = startChild; k <= endChild; k++) {
|
||||||
@@ -451,6 +450,22 @@ export function applyAnchorInDoc(
|
|||||||
doc: any,
|
doc: any,
|
||||||
selection: string,
|
selection: string,
|
||||||
commentId: string,
|
commentId: string,
|
||||||
|
): boolean {
|
||||||
|
return applyCommentMarkInDoc(doc, selection, makeCommentMark(commentId));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Core of {@link applyAnchorInDoc}, but splices an ARBITRARY comment mark object
|
||||||
|
* (not just a fresh `{ commentId, resolved:false }`) across the first matching
|
||||||
|
* range. This lets a caller re-apply a mark that carries `resolved:true` and any
|
||||||
|
* other stored attrs. Depth-first (same order as canAnchorInDoc); mutates in
|
||||||
|
* place on the first matching block and returns true, else returns false without
|
||||||
|
* mutating.
|
||||||
|
*/
|
||||||
|
export function applyCommentMarkInDoc(
|
||||||
|
doc: any,
|
||||||
|
selection: string,
|
||||||
|
commentMark: any,
|
||||||
): boolean {
|
): boolean {
|
||||||
const { selection: effective, found } = resolveAnchorSelection(doc, selection);
|
const { selection: effective, found } = resolveAnchorSelection(doc, selection);
|
||||||
if (!found) return false;
|
if (!found) return false;
|
||||||
@@ -459,7 +474,7 @@ export function applyAnchorInDoc(
|
|||||||
if (!Array.isArray(node.content)) return false;
|
if (!Array.isArray(node.content)) return false;
|
||||||
const match = findAnchorInBlock(node.content, effective);
|
const match = findAnchorInBlock(node.content, effective);
|
||||||
if (match) {
|
if (match) {
|
||||||
spliceCommentMark(node.content, match, commentId);
|
spliceCommentMark(node.content, match, commentMark);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
for (const child of node.content) {
|
for (const child of node.content) {
|
||||||
@@ -471,3 +486,97 @@ export function applyAnchorInDoc(
|
|||||||
};
|
};
|
||||||
return visit(doc, 0);
|
return visit(doc, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** A resolved inline-comment span lifted from a doc: its mark + anchored text. */
|
||||||
|
export interface ResolvedCommentSpan {
|
||||||
|
commentId: string;
|
||||||
|
/** The full comment mark (carrying `resolved:true` + any stored attrs). */
|
||||||
|
mark: any;
|
||||||
|
/** The concatenated raw text the mark spans — used as the re-anchor selection. */
|
||||||
|
text: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** True when a text node carries a RESOLVED comment mark; returns that mark. */
|
||||||
|
function resolvedCommentMarkOf(node: any): any | null {
|
||||||
|
if (!node || node.type !== "text" || !Array.isArray(node.marks)) return null;
|
||||||
|
return (
|
||||||
|
node.marks.find(
|
||||||
|
(m: any) =>
|
||||||
|
m && m.type === "comment" && m.attrs?.resolved === true && m.attrs?.commentId,
|
||||||
|
) || null
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Collect every RESOLVED inline-comment span in `doc`, in document order. Within
|
||||||
|
* each block's direct content, a maximal run of consecutive text nodes sharing
|
||||||
|
* the same resolved `commentId` is ONE span; its concatenated raw text is the
|
||||||
|
* selection used to re-anchor it elsewhere. Active (unresolved) comment marks are
|
||||||
|
* ignored — they survive a markdown round-trip on their own (a page read emits
|
||||||
|
* their `<span data-comment-id>` wrapper), whereas resolved anchors are hidden
|
||||||
|
* from agent reads (#337) and would be erased by a full-body markdown rewrite.
|
||||||
|
*/
|
||||||
|
export function collectResolvedCommentSpans(doc: any): ResolvedCommentSpan[] {
|
||||||
|
const spans: ResolvedCommentSpan[] = [];
|
||||||
|
const visit = (node: any, depth: number): void => {
|
||||||
|
if (depth > MAX_DEPTH || !node || typeof node !== "object") return;
|
||||||
|
if (!Array.isArray(node.content)) return;
|
||||||
|
const content = node.content;
|
||||||
|
let i = 0;
|
||||||
|
while (i < content.length) {
|
||||||
|
const mark = resolvedCommentMarkOf(content[i]);
|
||||||
|
if (mark) {
|
||||||
|
const commentId = mark.attrs.commentId;
|
||||||
|
let text = "";
|
||||||
|
let j = i;
|
||||||
|
while (j < content.length) {
|
||||||
|
const mj = resolvedCommentMarkOf(content[j]);
|
||||||
|
if (!mj || mj.attrs.commentId !== commentId) break;
|
||||||
|
text += typeof content[j].text === "string" ? content[j].text : "";
|
||||||
|
j++;
|
||||||
|
}
|
||||||
|
if (text.length > 0) spans.push({ commentId, mark, text });
|
||||||
|
i = j > i ? j : i + 1;
|
||||||
|
} else {
|
||||||
|
i++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const child of content) {
|
||||||
|
if (child && typeof child === "object" && Array.isArray(child.content)) {
|
||||||
|
visit(child, depth + 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
visit(doc, 0);
|
||||||
|
return spans;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Re-graft RESOLVED comment marks from `oldDoc` onto matching text ranges in
|
||||||
|
* `newDoc`, returning a NEW doc (never mutates the inputs).
|
||||||
|
*
|
||||||
|
* WHY (#493): an agent read hides resolved-comment anchors (#337), so the
|
||||||
|
* markdown it sends to a FULL-body rewrite (`updatePageMarkdown`) no longer
|
||||||
|
* carries them — a naive full write would erase every resolved comment mark.
|
||||||
|
* This restores them: each resolved span from the previous document is re-anchored
|
||||||
|
* onto the SAME text in the newly-imported body (first occurrence, using the
|
||||||
|
* shared anchoring / markdown-strip fallback), preserving `resolved:true` and the
|
||||||
|
* stored attrs. A span whose text the agent changed or deleted simply does not
|
||||||
|
* re-anchor and is dropped (its anchor is gone; it was already resolved). Active
|
||||||
|
* comments are untouched — they ride through the markdown themselves.
|
||||||
|
*/
|
||||||
|
export function regraftResolvedComments<T = any>(oldDoc: any, newDoc: T): T {
|
||||||
|
if (!newDoc || typeof newDoc !== "object") return newDoc;
|
||||||
|
const spans = collectResolvedCommentSpans(oldDoc);
|
||||||
|
if (spans.length === 0) return newDoc;
|
||||||
|
const out =
|
||||||
|
typeof structuredClone === "function"
|
||||||
|
? structuredClone(newDoc)
|
||||||
|
: (JSON.parse(JSON.stringify(newDoc)) as T);
|
||||||
|
for (const span of spans) {
|
||||||
|
// Clone the mark so the new document never shares a mark object with oldDoc.
|
||||||
|
const markClone = { type: "comment", attrs: { ...span.mark.attrs } };
|
||||||
|
applyCommentMarkInDoc(out, span.text, markClone);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,64 +1,30 @@
|
|||||||
/**
|
/**
|
||||||
* Locator normalization: strip inline markdown wrappers and trailing
|
* Locator normalization helpers for mcp. The two PRIMITIVES —
|
||||||
* decoration from a LOCATOR string so a find/anchor that the model wrote with
|
* `stripInlineMarkdown` (lenient locator normalizer) and `stripWrappersAndLinks`
|
||||||
* markdown (or a stray emoji) can still match the document's plain text.
|
* (strict balanced-wrapper/link collapse) — live in the canonical package
|
||||||
|
* `@docmost/prosemirror-markdown` (#493 dedup: they used to be forked verbatim
|
||||||
|
* here). This module now only re-exports `stripInlineMarkdown` and adds the two
|
||||||
|
* mcp-only helpers built on top: `stripBalancedWrappers` and `closestBlockHint`.
|
||||||
*
|
*
|
||||||
* This is used ONLY as a fallback for LOCATING (after an exact match fails);
|
* They are used ONLY as a fallback for LOCATING (after an exact match fails) and
|
||||||
* it is never applied to replacement text or inserted node content, so no
|
* for formatting-vs-plain intent detection; never applied to replacement text or
|
||||||
* formatting is ever lost.
|
* inserted node content, so no formatting is ever lost.
|
||||||
*/
|
*/
|
||||||
|
import {
|
||||||
|
stripInlineMarkdown,
|
||||||
|
stripWrappersAndLinks,
|
||||||
|
} from "@docmost/prosemirror-markdown";
|
||||||
|
|
||||||
/** Maximum unwrap passes, so pathological/nested input cannot loop forever. */
|
// Re-export the canonical locator normalizer so mcp call sites keep importing it
|
||||||
const MAX_PASSES = 8;
|
// from `./text-normalize.js` unchanged.
|
||||||
|
export { stripInlineMarkdown };
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Inline emphasis/code/strikethrough wrappers, strong BEFORE emphasis so
|
* STRICT formatting detector — distinct from the lenient locator normalization.
|
||||||
* `**x**` collapses to `x` rather than leaving a stray `*x*`. Each pattern is
|
* It strips ONLY what unambiguously is markdown markup (links/images to visible
|
||||||
* non-greedy and capture group 1 is the inner text. Applied repeatedly until
|
* text, and balanced inline `**`/`__`/`~~`/`*`/`_`/`` ` `` wrappers) and
|
||||||
* the string stops changing (nested wrappers like `**_x_**`).
|
* DELIBERATELY does NOT trim leading/trailing whitespace, emoji, or lone marker
|
||||||
*/
|
* chars (the lenient extras `stripInlineMarkdown` does).
|
||||||
const WRAPPER_PATTERNS: RegExp[] = [
|
|
||||||
/\*\*([^*]+?)\*\*/g, // **x**
|
|
||||||
/__([^_]+?)__/g, // __x__
|
|
||||||
/~~([^~]+?)~~/g, // ~~x~~
|
|
||||||
/\*([^*]+?)\*/g, // *x*
|
|
||||||
/_([^_]+?)_/g, // _x_
|
|
||||||
/``([^`]+?)``/g, // ``x``
|
|
||||||
/`([^`]+?)`/g, // `x`
|
|
||||||
];
|
|
||||||
|
|
||||||
/** Links/images -> their visible text. `!?` covers both `[t](u)` and ``. */
|
|
||||||
const LINK_IMAGE_RE = /!?\[([^\]]*)\]\([^)]*\)/g;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Apply ONLY the two balanced/link passes shared by both normalizers: first
|
|
||||||
* collapse links/images to their visible text, then collapse balanced inline
|
|
||||||
* wrappers repeatedly until stable. Does NOT trim decoration, does NOT guard
|
|
||||||
* against an empty result — it returns exactly the transformed string.
|
|
||||||
*/
|
|
||||||
function stripWrappersAndLinks(s: string): string {
|
|
||||||
// 1. Links/images -> their visible text.
|
|
||||||
let out = s.replace(LINK_IMAGE_RE, "$1");
|
|
||||||
|
|
||||||
// 2. Strip balanced wrappers, repeating until the string is stable so nested
|
|
||||||
// wrappers (`**_x_**`) and adjacent runs both collapse.
|
|
||||||
for (let pass = 0; pass < MAX_PASSES; pass++) {
|
|
||||||
const before = out;
|
|
||||||
for (const re of WRAPPER_PATTERNS) {
|
|
||||||
out = out.replace(re, "$1");
|
|
||||||
}
|
|
||||||
if (out === before) break;
|
|
||||||
}
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* STRICT formatting detector — distinct from the lenient locator
|
|
||||||
* normalization below. It strips ONLY what unambiguously is markdown markup:
|
|
||||||
* 1. links/images `[text](url)` -> `text`, `` -> `alt`, and
|
|
||||||
* 2. balanced inline `**`/`__`/`~~`/`*`/`_`/`` ` `` wrappers (repeat-until-stable),
|
|
||||||
* and DELIBERATELY does NOT trim leading/trailing whitespace, emoji, or lone
|
|
||||||
* marker chars (the lenient extras `stripInlineMarkdown` does in its step 3).
|
|
||||||
*
|
*
|
||||||
* It exists ONLY to recognize formatting-vs-plain INTENT in `applyTextEdits`
|
* It exists ONLY to recognize formatting-vs-plain INTENT in `applyTextEdits`
|
||||||
* (deciding whether find/replace differ purely by markdown markers). Because it
|
* (deciding whether find/replace differ purely by markdown markers). Because it
|
||||||
@@ -77,44 +43,6 @@ export function stripBalancedWrappers(s: string): string {
|
|||||||
return stripWrappersAndLinks(s);
|
return stripWrappersAndLinks(s);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Conservatively strip inline markdown from a locator string.
|
|
||||||
*
|
|
||||||
* Deterministic, order-fixed steps:
|
|
||||||
* 1. Links/images: `[text](url)` -> `text`, `` -> `alt`.
|
|
||||||
* 2. Balanced inline wrappers (strong before emphasis, code, strikethrough),
|
|
||||||
* applied repeatedly until stable for nested cases.
|
|
||||||
* 3. Trim leading/trailing decoration only: whitespace, leftover marker chars
|
|
||||||
* (`* _ ~ \``) and emoji. Letters/digits and sentence punctuation (`.`/`,`
|
|
||||||
* etc.) are NEVER trimmed.
|
|
||||||
*
|
|
||||||
* If the result is empty (e.g. the input was only markers like `***`), the
|
|
||||||
* ORIGINAL string is returned so a locator can never normalize down to "" and
|
|
||||||
* match everything.
|
|
||||||
*/
|
|
||||||
export function stripInlineMarkdown(s: string): string {
|
|
||||||
if (typeof s !== "string" || s.length === 0) return s;
|
|
||||||
|
|
||||||
// 1 + 2. Shared link/image and balanced-wrapper passes.
|
|
||||||
let out = stripWrappersAndLinks(s);
|
|
||||||
|
|
||||||
// 3. Trim leading/trailing decoration: whitespace, leftover markdown markers,
|
|
||||||
// and emoji (Extended_Pictographic plus the VS16 / ZWJ joiners, plus the
|
|
||||||
// regional-indicator range U+1F1E6–U+1F1FF for flag emoji, which are NOT
|
|
||||||
// Extended_Pictographic). The `u` flag enables the Unicode property escape.
|
|
||||||
// Anchored runs only — interior text and sentence punctuation are untouched.
|
|
||||||
const DECORATION =
|
|
||||||
"[\\s*_~\\x60\\p{Extended_Pictographic}\\u{1F1E6}-\\u{1F1FF}\\u{FE0F}\\u{200D}]+";
|
|
||||||
out = out
|
|
||||||
.replace(new RegExp("^" + DECORATION, "u"), "")
|
|
||||||
.replace(new RegExp(DECORATION + "$", "u"), "");
|
|
||||||
|
|
||||||
// 4. Never normalize a locator down to nothing.
|
|
||||||
if (out.length === 0) return s;
|
|
||||||
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Build a bounded "closest text" hint for an anchor/find MISS, shared by
|
* Build a bounded "closest text" hint for an anchor/find MISS, shared by
|
||||||
* editPageText (json-edit) and createComment (client) so both surface the
|
* editPageText (json-edit) and createComment (client) so both surface the
|
||||||
|
|||||||
@@ -108,6 +108,17 @@ async function spawnCollabStack(seedDoc) {
|
|||||||
return { state, baseURL };
|
return { state, baseURL };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// y-prosemirror stores an OVERLAPPING mark (one whose type does not exclude
|
||||||
|
// itself — e.g. `comment`, and since #515 `code` with `excludes: ""`) under a
|
||||||
|
// HASHED Yjs attribute key `name--<8-char hash>` so several may coexist on a
|
||||||
|
// range. The real read path (yDocToProsemirrorJSON) strips that suffix back to
|
||||||
|
// the bare mark name via this exact regex; mirror it here so this minimal decoder
|
||||||
|
// reports the same mark names Docmost actually returns (without it an overlapping
|
||||||
|
// `code` would leak as `code--<hash>`).
|
||||||
|
const hashedMarkNameRegex = /(.*)(--[a-zA-Z0-9+/=]{8})$/;
|
||||||
|
const yattr2markname = (attrName) =>
|
||||||
|
hashedMarkNameRegex.exec(attrName)?.[1] ?? attrName;
|
||||||
|
|
||||||
// Minimal XmlFragment -> ProseMirror JSON decode, mirroring the shape Docmost
|
// Minimal XmlFragment -> ProseMirror JSON decode, mirroring the shape Docmost
|
||||||
// stores. Reads element name as node type, attributes as attrs, and recurses into
|
// stores. Reads element name as node type, attributes as attrs, and recurses into
|
||||||
// children; text nodes carry their string.
|
// children; text nodes carry their string.
|
||||||
@@ -121,8 +132,8 @@ function fragmentToJson(frag) {
|
|||||||
if (d.attributes && Object.keys(d.attributes).length) {
|
if (d.attributes && Object.keys(d.attributes).length) {
|
||||||
node.marks = Object.entries(d.attributes).map(([type, attrs]) =>
|
node.marks = Object.entries(d.attributes).map(([type, attrs]) =>
|
||||||
attrs && typeof attrs === "object" && Object.keys(attrs).length
|
attrs && typeof attrs === "object" && Object.keys(attrs).length
|
||||||
? { type, attrs }
|
? { type: yattr2markname(type), attrs }
|
||||||
: { type },
|
: { type: yattr2markname(type) },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
return node;
|
return node;
|
||||||
|
|||||||
@@ -0,0 +1,107 @@
|
|||||||
|
import { test } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
|
||||||
|
import {
|
||||||
|
collectResolvedCommentSpans,
|
||||||
|
regraftResolvedComments,
|
||||||
|
applyCommentMarkInDoc,
|
||||||
|
} from "../../build/lib/comment-anchor.js";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* #493 commit 6 — resolved-comment anchors must survive a full markdown rewrite
|
||||||
|
* (updatePageMarkdown). An agent read HIDES resolved anchors (#337), so its
|
||||||
|
* markdown drops them; a naive full write would erase the resolved comment marks.
|
||||||
|
* `regraftResolvedComments(oldDoc, newDoc)` re-anchors them onto the matching
|
||||||
|
* text. These exercise the real anchoring (no mock).
|
||||||
|
*/
|
||||||
|
|
||||||
|
const doc = (...content) => ({ type: "doc", content });
|
||||||
|
const para = (...content) => ({ type: "paragraph", content });
|
||||||
|
const text = (t, marks) => (marks ? { type: "text", text: t, marks } : { type: "text", text: t });
|
||||||
|
const resolvedComment = (commentId) => ({ type: "comment", attrs: { commentId, resolved: true } });
|
||||||
|
const activeComment = (commentId) => ({ type: "comment", attrs: { commentId, resolved: false } });
|
||||||
|
|
||||||
|
/** The comment mark on a text node, or null. */
|
||||||
|
function commentMarkOf(node) {
|
||||||
|
const marks = Array.isArray(node?.marks) ? node.marks : [];
|
||||||
|
return marks.find((m) => m && m.type === "comment") || null;
|
||||||
|
}
|
||||||
|
/** Flatten every text node in a doc (deep). */
|
||||||
|
function textNodes(node, out = []) {
|
||||||
|
if (!node || typeof node !== "object") return out;
|
||||||
|
if (node.type === "text") out.push(node);
|
||||||
|
if (Array.isArray(node.content)) for (const c of node.content) textNodes(c, out);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
test("collectResolvedCommentSpans: only resolved marks, concatenated across a run", () => {
|
||||||
|
const old = doc(
|
||||||
|
para(
|
||||||
|
text("keep "),
|
||||||
|
text("resolved bit", [resolvedComment("r1")]),
|
||||||
|
text(" and "),
|
||||||
|
text("active bit", [activeComment("a1")]),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
const spans = collectResolvedCommentSpans(old);
|
||||||
|
assert.equal(spans.length, 1);
|
||||||
|
assert.equal(spans[0].commentId, "r1");
|
||||||
|
assert.equal(spans[0].text, "resolved bit");
|
||||||
|
assert.equal(spans[0].mark.attrs.resolved, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("regraft restores a resolved mark the agent's markdown dropped", () => {
|
||||||
|
// OLD doc has a resolved comment on "important note".
|
||||||
|
const old = doc(para(text("An "), text("important note", [resolvedComment("r1")]), text(" here.")));
|
||||||
|
// NEW doc (re-imported from the agent's markdown) has the SAME text but NO
|
||||||
|
// comment mark — the resolved anchor was hidden on read.
|
||||||
|
const fresh = doc(para(text("An important note here.")));
|
||||||
|
|
||||||
|
const out = regraftResolvedComments(old, fresh);
|
||||||
|
// Inputs are not mutated.
|
||||||
|
assert.equal(commentMarkOf(textNodes(fresh)[0]), null);
|
||||||
|
// The resolved mark is back on exactly "important note".
|
||||||
|
const marked = textNodes(out).filter((n) => commentMarkOf(n));
|
||||||
|
assert.equal(marked.length, 1);
|
||||||
|
assert.equal(marked[0].text, "important note");
|
||||||
|
assert.equal(commentMarkOf(marked[0]).attrs.commentId, "r1");
|
||||||
|
assert.equal(commentMarkOf(marked[0]).attrs.resolved, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a resolved span whose text the agent changed is dropped (no re-anchor)", () => {
|
||||||
|
const old = doc(para(text("stale text", [resolvedComment("r1")])));
|
||||||
|
const fresh = doc(para(text("completely rewritten body")));
|
||||||
|
const out = regraftResolvedComments(old, fresh);
|
||||||
|
assert.equal(textNodes(out).filter((n) => commentMarkOf(n)).length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("regraft is a no-op when the old doc has no resolved comments", () => {
|
||||||
|
const old = doc(para(text("plain "), text("active", [activeComment("a1")])));
|
||||||
|
const fresh = doc(para(text("plain active")));
|
||||||
|
const out = regraftResolvedComments(old, fresh);
|
||||||
|
assert.equal(textNodes(out).filter((n) => commentMarkOf(n)).length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("multiple distinct resolved comments are all restored", () => {
|
||||||
|
const old = doc(
|
||||||
|
para(text("first", [resolvedComment("r1")]), text(" middle "), text("second", [resolvedComment("r2")])),
|
||||||
|
);
|
||||||
|
const fresh = doc(para(text("first middle second")));
|
||||||
|
const out = regraftResolvedComments(old, fresh);
|
||||||
|
const byId = Object.fromEntries(
|
||||||
|
textNodes(out)
|
||||||
|
.filter((n) => commentMarkOf(n))
|
||||||
|
.map((n) => [commentMarkOf(n).attrs.commentId, n.text]),
|
||||||
|
);
|
||||||
|
assert.equal(byId["r1"], "first");
|
||||||
|
assert.equal(byId["r2"], "second");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("applyCommentMarkInDoc preserves an arbitrary mark's attrs (resolved:true)", () => {
|
||||||
|
const d = doc(para(text("anchor me somewhere")));
|
||||||
|
const ok = applyCommentMarkInDoc(d, "anchor me", { type: "comment", attrs: { commentId: "x9", resolved: true } });
|
||||||
|
assert.equal(ok, true);
|
||||||
|
const marked = textNodes(d).filter((n) => commentMarkOf(n));
|
||||||
|
assert.equal(marked[0].text, "anchor me");
|
||||||
|
assert.equal(commentMarkOf(marked[0]).attrs.resolved, true);
|
||||||
|
});
|
||||||
@@ -1,12 +1,31 @@
|
|||||||
# @docmost/prosemirror-markdown
|
# @docmost/prosemirror-markdown
|
||||||
|
|
||||||
The single, canonical **ProseMirror ↔ Markdown converter** plus the Docmost
|
The single, canonical **ProseMirror ↔ Markdown converter** plus the Docmost
|
||||||
schema mirror (#293/#345). Headless and framework-free: no React, no browser
|
schema mirror (#293/#345/#347). Headless and framework-free: no React. There is
|
||||||
runtime. There is exactly ONE copy of this converter in the repo, consumed by:
|
exactly ONE copy of this converter in the repo, consumed by:
|
||||||
|
|
||||||
- `packages/mcp` (the MCP server),
|
- `packages/mcp` (the MCP server),
|
||||||
- `packages/git-sync` (two-way Git sync),
|
- `packages/git-sync` (two-way Git sync),
|
||||||
- `apps/server` (server-side markdown import/export, #345).
|
- `apps/server` (server-side markdown import/export, #345),
|
||||||
|
- `apps/client` (markdown paste/copy + AI-chat render, #347).
|
||||||
|
|
||||||
|
### Node vs browser entry
|
||||||
|
|
||||||
|
The HTML→DOM stage of markdown import runs on `jsdom` in Node and the native
|
||||||
|
`DOMParser` in the browser, injected per environment so **jsdom never enters a
|
||||||
|
client bundle**:
|
||||||
|
|
||||||
|
- default entry (`@docmost/prosemirror-markdown`) — Node: registers jsdom +
|
||||||
|
`@tiptap/html`'s happy-dom `server` `generateJSON`. Used by mcp / git-sync /
|
||||||
|
apps/server.
|
||||||
|
- `browser` entry (`@docmost/prosemirror-markdown/browser`, via the `"browser"`
|
||||||
|
exports condition) — registers the native `DOMParser` + `@tiptap/html`'s
|
||||||
|
browser `generateJSON`. Used by `apps/client`; carries no jsdom/happy-dom.
|
||||||
|
|
||||||
|
Both entries expose the identical converter surface; only the injected
|
||||||
|
DOM/`generateJSON` implementations differ (`src/lib/dom-parser.ts`). A
|
||||||
|
`markdownToProseMirrorSync` variant exists for callers that cannot await (the
|
||||||
|
client's synchronous chat renderer).
|
||||||
|
|
||||||
`src/lib/docmost-schema.ts` **mirrors** the upstream Tiptap schema that lives in
|
`src/lib/docmost-schema.ts` **mirrors** the upstream Tiptap schema that lives in
|
||||||
`packages/editor-ext`. The mirror is not free-floating: `serializer-contract.test.ts`
|
`packages/editor-ext`. The mirror is not free-floating: `serializer-contract.test.ts`
|
||||||
|
|||||||
@@ -9,7 +9,12 @@
|
|||||||
"exports": {
|
"exports": {
|
||||||
".": {
|
".": {
|
||||||
"types": "./build/index.d.ts",
|
"types": "./build/index.d.ts",
|
||||||
|
"browser": "./build/browser.js",
|
||||||
"default": "./build/index.js"
|
"default": "./build/index.js"
|
||||||
|
},
|
||||||
|
"./browser": {
|
||||||
|
"types": "./build/browser.d.ts",
|
||||||
|
"default": "./build/browser.js"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"scripts": {
|
"scripts": {
|
||||||
@@ -30,6 +35,7 @@
|
|||||||
"@tiptap/html": "3.20.4",
|
"@tiptap/html": "3.20.4",
|
||||||
"@tiptap/pm": "3.20.4",
|
"@tiptap/pm": "3.20.4",
|
||||||
"@tiptap/starter-kit": "3.20.4",
|
"@tiptap/starter-kit": "3.20.4",
|
||||||
|
"happy-dom": "20.8.9",
|
||||||
"jsdom": "25.0.0",
|
"jsdom": "25.0.0",
|
||||||
"marked": "17.0.5",
|
"marked": "17.0.5",
|
||||||
"zod": "4.3.6"
|
"zod": "4.3.6"
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
/**
|
||||||
|
* BROWSER entry of `@docmost/prosemirror-markdown`.
|
||||||
|
*
|
||||||
|
* Selected via the package's `"browser"` exports condition (bundlers) so the
|
||||||
|
* client gets the SAME public converter surface as the Node entry, but the
|
||||||
|
* markdown-import DOM passes run on the native `window.DOMParser` instead of
|
||||||
|
* jsdom. This module installs the native parser as a side effect BEFORE
|
||||||
|
* re-exporting, and imports NO jsdom — so a client bundle that resolves this
|
||||||
|
* entry carries no `jsdom` (nor any transitive jsdom import).
|
||||||
|
*
|
||||||
|
* The re-exported surface is identical to the default (Node) entry — the only
|
||||||
|
* difference is which HTML-DOM parser is registered — so a browser consumer can
|
||||||
|
* call `markdownToProseMirror` (and everything else) exactly as the server does.
|
||||||
|
*/
|
||||||
|
import "./lib/dom-parser.browser.js";
|
||||||
|
|
||||||
|
export * from "./lib/index.js";
|
||||||
@@ -6,4 +6,11 @@
|
|||||||
* this top-level barrel simply re-exports that surface so the package entry is
|
* this top-level barrel simply re-exports that surface so the package entry is
|
||||||
* the converter surface.
|
* the converter surface.
|
||||||
*/
|
*/
|
||||||
|
// DEFAULT (Node) entry: install the jsdom-backed HTML parser as a side effect
|
||||||
|
// BEFORE re-exporting the converter surface, so every Node consumer (server,
|
||||||
|
// mcp, git-sync) keeps the identical jsdom import behaviour with no code change.
|
||||||
|
// The browser entry (`./browser.js`) installs the native-`DOMParser` parser
|
||||||
|
// instead and never loads this module, so jsdom stays out of client bundles.
|
||||||
|
import "./lib/dom-parser.node.js";
|
||||||
|
|
||||||
export * from "./lib/index.js";
|
export * from "./lib/index.js";
|
||||||
|
|||||||
@@ -16,6 +16,7 @@
|
|||||||
* `@docmost/editor-ext` before updating the snapshot.
|
* `@docmost/editor-ext` before updating the snapshot.
|
||||||
*/
|
*/
|
||||||
import StarterKit from "@tiptap/starter-kit";
|
import StarterKit from "@tiptap/starter-kit";
|
||||||
|
import { Code } from "@tiptap/extension-code";
|
||||||
import Image from "@tiptap/extension-image";
|
import Image from "@tiptap/extension-image";
|
||||||
import TaskList from "@tiptap/extension-task-list";
|
import TaskList from "@tiptap/extension-task-list";
|
||||||
import TaskItem from "@tiptap/extension-task-item";
|
import TaskItem from "@tiptap/extension-task-item";
|
||||||
@@ -63,10 +64,9 @@ function getStyleProperty(element: HTMLElement, propertyName: string): string |
|
|||||||
* The editor SCHEMA genuinely only supports these six banner types — there is no
|
* The editor SCHEMA genuinely only supports these six banner types — there is no
|
||||||
* `tip`/`caution`/`important`/`question` callout node. So those are NOT first-
|
* `tip`/`caution`/`important`/`question` callout node. So those are NOT first-
|
||||||
* class types we can round-trip literally; they are INPUT ALIASES (GitHub/Obsidian
|
* class types we can round-trip literally; they are INPUT ALIASES (GitHub/Obsidian
|
||||||
* alert syntax). The editor's own paste/import path maps them onto the supported
|
* alert syntax). This package's own `> [!type]` import path maps them onto the
|
||||||
* set (see `GITHUB_ALERT_TYPE_MAP` in
|
* supported set (see `CALLOUT_TYPE_ALIASES` below: tip -> success, caution ->
|
||||||
* `@docmost/editor-ext` markdown/utils/github-callout.marked.ts:
|
* danger, important -> info). We apply that aliasing
|
||||||
* tip -> success, caution -> danger, important -> info). We mirror that aliasing
|
|
||||||
* here so an ingested `> [!tip]` / `> [!caution]` lands on the closest real banner
|
* here so an ingested `> [!tip]` / `> [!caution]` lands on the closest real banner
|
||||||
* (success / danger) instead of flatly collapsing to `info` — matching exactly how
|
* (success / danger) instead of flatly collapsing to `info` — matching exactly how
|
||||||
* the editor itself would interpret the same alias. A schema type always maps to
|
* the editor itself would interpret the same alias. A schema type always maps to
|
||||||
@@ -75,11 +75,11 @@ function getStyleProperty(element: HTMLElement, propertyName: string): string |
|
|||||||
*/
|
*/
|
||||||
const CALLOUT_TYPES = ["default", "info", "note", "success", "warning", "danger"];
|
const CALLOUT_TYPES = ["default", "info", "note", "success", "warning", "danger"];
|
||||||
/**
|
/**
|
||||||
* NON-schema callout aliases -> their closest supported banner. Mirrors the
|
* NON-schema callout aliases -> their closest supported banner, for the names
|
||||||
* editor's `GITHUB_ALERT_TYPE_MAP` for the names that are NOT already schema
|
* that are NOT already schema types (a schema type is preserved as-is and never
|
||||||
* types (a schema type is preserved as-is and never consulted here). Keeping
|
* consulted here). This is the single canonical alias map now that the editor's
|
||||||
* these in lockstep means git-sync ingest and an editor paste interpret the same
|
* old marked layer is gone; git-sync ingest and an editor paste both go through
|
||||||
* `> [!alias]` identically.
|
* this package, so they interpret the same `> [!alias]` identically.
|
||||||
*/
|
*/
|
||||||
const CALLOUT_TYPE_ALIASES: Record<string, string> = {
|
const CALLOUT_TYPE_ALIASES: Record<string, string> = {
|
||||||
tip: "success",
|
tip: "success",
|
||||||
@@ -1482,7 +1482,20 @@ export const docmostExtensions = [
|
|||||||
codeBlock: {},
|
codeBlock: {},
|
||||||
heading: {},
|
heading: {},
|
||||||
link: { openOnClick: false },
|
link: { openOnClick: false },
|
||||||
|
// #515: disable StarterKit's bundled inline `code` mark so it can be replaced
|
||||||
|
// by the local override below. StarterKit's `code` inherits tiptap's
|
||||||
|
// `excludes: "_"`, which strips every co-occurring mark on HTML->PM import
|
||||||
|
// (`generateJSON`) — so `` **`--flag`** `` lost its bold. This mirror is a
|
||||||
|
// DELIBERATE standalone copy (it must not pull @docmost/editor-ext into the
|
||||||
|
// node import runtime — that would drag in React/node-views; see #293), so
|
||||||
|
// the `excludes: ""` override is declared LOCALLY here and kept in lockstep
|
||||||
|
// with the canonical `Code` in @docmost/editor-ext by a parity test.
|
||||||
|
code: false,
|
||||||
}),
|
}),
|
||||||
|
// #515: inline code that COMBINES with other marks (CommonMark-consistent).
|
||||||
|
// `excludes: ""` means the mark excludes nothing, so bold/italic/strike/… may
|
||||||
|
// co-occur with `code` and survive import.
|
||||||
|
Code.extend({ excludes: "" }),
|
||||||
// Preserve image width/height as the AUTHORED string. Without an explicit
|
// Preserve image width/height as the AUTHORED string. Without an explicit
|
||||||
// parseHTML the stock Image node attribute falls back to tiptap core's
|
// parseHTML the stock Image node attribute falls back to tiptap core's
|
||||||
// `fromString`, which coerces a numeric width like "320" into the number 320
|
// `fromString`, which coerces a numeric width like "320" into the number 320
|
||||||
|
|||||||
@@ -0,0 +1,27 @@
|
|||||||
|
/**
|
||||||
|
* BROWSER registration of the injectable HTML parser (native `DOMParser`).
|
||||||
|
*
|
||||||
|
* Importing this module for its SIDE EFFECT installs a `window.DOMParser`-backed
|
||||||
|
* {@link HtmlDocumentParser}. It is loaded by the package's `browser.ts` barrel
|
||||||
|
* (selected via the `"browser"` exports condition). It imports NO `jsdom`, so a
|
||||||
|
* client bundle that resolves the browser entry never pulls jsdom in.
|
||||||
|
*
|
||||||
|
* The returned `Document` is a real browser document, so it exposes exactly the
|
||||||
|
* same query/mutation surface the import passes use (`querySelector(All)`,
|
||||||
|
* `createElement`, `createTreeWalker`/`NodeFilter` via `defaultView`,
|
||||||
|
* `body.innerHTML`) as jsdom did on the Node path.
|
||||||
|
*/
|
||||||
|
// @tiptap/html's default (browser) entry: its `generateJSON` uses the native
|
||||||
|
// `window.DOMParser`, so it carries NO jsdom/happy-dom — keeping the client
|
||||||
|
// bundle free of Node-only DOM libs.
|
||||||
|
import { generateJSON } from "@tiptap/html";
|
||||||
|
import { setHtmlDocumentParser, setGenerateJson } from "./dom-parser.js";
|
||||||
|
|
||||||
|
setHtmlDocumentParser((html: string): Document => {
|
||||||
|
// Native, always available in a browser (and in a jsdom/happy-dom test
|
||||||
|
// environment, which is what the client vitest suite runs under). `text/html`
|
||||||
|
// parsing matches jsdom's `new JSDOM(html)` behaviour for our fragments.
|
||||||
|
return new DOMParser().parseFromString(html, "text/html");
|
||||||
|
});
|
||||||
|
|
||||||
|
setGenerateJson(generateJSON);
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
/**
|
||||||
|
* NODE registration of the injectable HTML parser (jsdom-backed).
|
||||||
|
*
|
||||||
|
* Importing this module for its SIDE EFFECT installs a jsdom-backed
|
||||||
|
* {@link HtmlDocumentParser}. It is loaded by the package's default entry
|
||||||
|
* (`index.ts`) so every existing Node consumer (server, mcp, git-sync) keeps
|
||||||
|
* the identical jsdom behaviour with no code change. This is the ONLY module in
|
||||||
|
* the import chain that imports `jsdom`; the browser entry never loads it, so
|
||||||
|
* `jsdom` cannot reach the client bundle.
|
||||||
|
*/
|
||||||
|
import { JSDOM } from "jsdom";
|
||||||
|
// Use @tiptap/html's EXPLICIT server entry (happy-dom backed): it builds its own
|
||||||
|
// DOM internally and needs NO ambient global `window`, so the Node path never
|
||||||
|
// depends on which of @tiptap/html's conditional exports a resolver picks (Jest
|
||||||
|
// selects the browser entry, which would throw without a global window). This
|
||||||
|
// avoids the old module-level `global.window` jsdom shim entirely — that shim
|
||||||
|
// was timing-fragile (it had to be installed AFTER prosemirror-view's
|
||||||
|
// import-time env detection, or prosemirror-view reads an undefined `navigator`).
|
||||||
|
import { generateJSON } from "@tiptap/html/server";
|
||||||
|
import { setHtmlDocumentParser, setGenerateJson } from "./dom-parser.js";
|
||||||
|
|
||||||
|
setHtmlDocumentParser((html: string): Document => {
|
||||||
|
// A fresh JSDOM per call mirrors the previous `new JSDOM(html)` usage in each
|
||||||
|
// import pass — no shared mutable document between conversions, so concurrent
|
||||||
|
// conversions never interfere.
|
||||||
|
const dom = new JSDOM(html);
|
||||||
|
return dom.window.document as unknown as Document;
|
||||||
|
});
|
||||||
|
|
||||||
|
setGenerateJson(generateJSON);
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
/**
|
||||||
|
* Injectable HTML-string -> DOM parser for the markdown import path.
|
||||||
|
*
|
||||||
|
* The markdown -> ProseMirror chain (`markdown-to-prosemirror.ts`) does three
|
||||||
|
* post-`marked` DOM passes (task-list bridge, comment directives, footnote
|
||||||
|
* assembly) that need a real DOM to query/mutate an HTML fragment. On the NODE
|
||||||
|
* path that DOM comes from `jsdom`; in the BROWSER the platform already provides
|
||||||
|
* a native `DOMParser` and `document`, and `jsdom` must NOT be bundled (size +
|
||||||
|
* it is Node-only). So the concrete parser is INJECTED per environment rather
|
||||||
|
* than imported statically here — this module carries no `jsdom` (or any DOM)
|
||||||
|
* import, so nothing on the browser code path can transitively pull `jsdom` in.
|
||||||
|
*
|
||||||
|
* The Node entry (`./dom-parser.node.js`, loaded by the package's default
|
||||||
|
* `index.js`) registers a jsdom-backed parser; the browser entry
|
||||||
|
* (`./dom-parser.browser.js`, loaded by the `browser.js` barrel) registers a
|
||||||
|
* `window.DOMParser`-backed one. A consumer that forgets to load an entry (e.g.
|
||||||
|
* a raw deep import) gets a clear error instead of a silent wrong-environment
|
||||||
|
* crash.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse an HTML string into a `Document`. The returned document must support the
|
||||||
|
* standard query/mutation surface the import passes use: `querySelector(All)`,
|
||||||
|
* `createElement`, `createTreeWalker` + `NodeFilter` (read off the document's
|
||||||
|
* `defaultView`), and `body.innerHTML`.
|
||||||
|
*/
|
||||||
|
export type HtmlDocumentParser = (html: string) => Document;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert an HTML string to a ProseMirror JSON doc against the given TipTap
|
||||||
|
* extension set. This is `@tiptap/html`'s `generateJSON`, injected per
|
||||||
|
* environment so the Node path binds its happy-dom `server` entry and the
|
||||||
|
* browser path its native-`DOMParser` entry — neither leaking the other's DOM
|
||||||
|
* lib into the wrong bundle.
|
||||||
|
*/
|
||||||
|
export type GenerateJsonFn = (html: string, extensions: any[]) => any;
|
||||||
|
|
||||||
|
let injectedParser: HtmlDocumentParser | null = null;
|
||||||
|
let injectedGenerateJson: GenerateJsonFn | null = null;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Register the environment's HTML parser. Called ONCE at import time by the
|
||||||
|
* Node or browser entry module. Idempotent-friendly: the last registration
|
||||||
|
* wins, so a test harness can override it.
|
||||||
|
*/
|
||||||
|
export function setHtmlDocumentParser(parser: HtmlDocumentParser): void {
|
||||||
|
injectedParser = parser;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse `html` into a `Document` using the registered parser. Throws a clear
|
||||||
|
* error when no environment entry has registered one (the caller imported the
|
||||||
|
* converter without going through the Node/browser barrel).
|
||||||
|
*/
|
||||||
|
export function parseHtmlDocument(html: string): Document {
|
||||||
|
if (!injectedParser) {
|
||||||
|
throw new Error(
|
||||||
|
"No HTML DOM parser registered. Import `@docmost/prosemirror-markdown` " +
|
||||||
|
"(Node) or `@docmost/prosemirror-markdown/browser` (browser) so the " +
|
||||||
|
"environment's DOM parser is installed before calling the converter.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return injectedParser(html);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Register the environment's `generateJSON` (HTML -> ProseMirror JSON). Called
|
||||||
|
* ONCE at import time by the Node or browser entry module.
|
||||||
|
*/
|
||||||
|
export function setGenerateJson(fn: GenerateJsonFn): void {
|
||||||
|
injectedGenerateJson = fn;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run the registered `generateJSON`. Throws a clear error when no environment
|
||||||
|
* entry has registered one (same cause as {@link parseHtmlDocument}).
|
||||||
|
*/
|
||||||
|
export function generateJsonWith(html: string, extensions: any[]): any {
|
||||||
|
if (!injectedGenerateJson) {
|
||||||
|
throw new Error(
|
||||||
|
"No generateJSON registered. Import `@docmost/prosemirror-markdown` " +
|
||||||
|
"(Node) or `@docmost/prosemirror-markdown/browser` (browser) so the " +
|
||||||
|
"environment's generateJSON is installed before calling the converter.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return injectedGenerateJson(html, extensions);
|
||||||
|
}
|
||||||
+46
-8
@@ -1,7 +1,14 @@
|
|||||||
/**
|
/**
|
||||||
* Foreign-markdown normalizer — an input-liberal / output-canonical adapter that
|
* Foreign-markdown normalizer — an input-liberal / output-canonical adapter that
|
||||||
* runs at the IMPORT boundary, BEFORE the canonical parser
|
* runs at the IMPORT boundary, BEFORE the canonical parser
|
||||||
* (`markdownToProseMirror` from `@docmost/prosemirror-markdown`).
|
* (`markdownToProseMirror`, this package).
|
||||||
|
*
|
||||||
|
* OWNED BY THIS PACKAGE (#493): the normalizer used to live only in
|
||||||
|
* apps/server's import path, so the MCP page-write path (`updatePageMarkdown` ->
|
||||||
|
* `markdownToProseMirrorCanonical`) handled the SAME foreign input differently
|
||||||
|
* (no front-matter strip, no `[^id]` reference-footnote rewrite) than the server
|
||||||
|
* importer. Moving it here — and calling it from `markdownToProseMirrorCanonical`
|
||||||
|
* — makes every canonical import boundary treat foreign markdown identically.
|
||||||
*
|
*
|
||||||
* The canonical parser is deliberately STRICT: it only understands Docmost's
|
* The canonical parser is deliberately STRICT: it only understands Docmost's
|
||||||
* canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
|
* canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
|
||||||
@@ -238,19 +245,27 @@ function convertReferenceFootnotes(markdown: string): string {
|
|||||||
*
|
*
|
||||||
* LINE-ANCHORED (the same shape the canonical parser uses in
|
* LINE-ANCHORED (the same shape the canonical parser uses in
|
||||||
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
|
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
|
||||||
* very start and closes only on a `\n---` line. The retired `markdownToHtml`
|
* very start and closes only on a `\n---` line. The retired editor-ext
|
||||||
* strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
|
* `markdownToHtml` front-matter strip (removed in #347) closed on the FIRST
|
||||||
|
* `---` ANYWHERE (an unanchored close), so a value
|
||||||
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
|
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
|
||||||
* and leaked the rest into the body. An optional leading BOM is tolerated.
|
* and leaked the rest into the body. An optional leading BOM is tolerated.
|
||||||
*/
|
*/
|
||||||
const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
|
const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Normalize a foreign markdown string into Docmost's canonical markdown surface
|
* Normalize a foreign markdown string from a FILE IMPORT into Docmost's canonical
|
||||||
* so the strict canonical parser accepts it losslessly: normalize line endings,
|
* markdown surface so the strict canonical parser accepts it losslessly: normalize
|
||||||
* strip a leading YAML front-matter block, then rewrite GFM reference footnotes
|
* line endings, strip a leading YAML front-matter block, then rewrite GFM reference
|
||||||
* into inline footnotes. Add further fixture-driven foreign-surface cases here as
|
* footnotes into inline footnotes. Add further fixture-driven foreign-surface cases
|
||||||
* they are found.
|
* here as they are found.
|
||||||
|
*
|
||||||
|
* FRONT-MATTER STRIP IS IMPORT-ONLY (#493 review): use this ONLY at the server
|
||||||
|
* file-import boundary, where a `.md` file really can open with an Obsidian/Hugo
|
||||||
|
* YAML header. Do NOT use it on the canonical AGENT-WRITE path — see
|
||||||
|
* {@link normalizeAgentMarkdown} for why a full-body agent rewrite must NOT strip
|
||||||
|
* a leading `---…---` (it is normally a horizontalRule the serializer emitted, and
|
||||||
|
* stripping it would silently drop the page's leading content).
|
||||||
*/
|
*/
|
||||||
export function normalizeForeignMarkdown(markdown: string): string {
|
export function normalizeForeignMarkdown(markdown: string): string {
|
||||||
if (!markdown) return markdown;
|
if (!markdown) return markdown;
|
||||||
@@ -263,3 +278,26 @@ export function normalizeForeignMarkdown(markdown: string): string {
|
|||||||
const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
|
const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
|
||||||
return convertReferenceFootnotes(withoutFrontMatter);
|
return convertReferenceFootnotes(withoutFrontMatter);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Canonical AGENT-WRITE normalization: normalize line endings and rewrite GFM
|
||||||
|
* `[^id]` reference footnotes to inline `^[body]` — but DELIBERATELY NOT strip a
|
||||||
|
* leading YAML front-matter block.
|
||||||
|
*
|
||||||
|
* WHY the split (#493 review): the reference-footnote rewrite is the drift the
|
||||||
|
* MCP page-write path (`updatePageMarkdown` -> `markdownToProseMirrorCanonical`)
|
||||||
|
* needed unified with the server import (an agent may paste GFM footnotes). The
|
||||||
|
* front-matter strip, however, is a FILE-import concern: on a full-body agent
|
||||||
|
* rewrite a leading `---…---` is (almost) always a `horizontalRule` the
|
||||||
|
* serializer emitted plus a later rule/heading — NOT a foreign YAML header — so
|
||||||
|
* `YAML_FRONT_MATTER_RE` would match it and SILENTLY DELETE the page's leading
|
||||||
|
* content (a page that starts with a horizontal rule and contains a second `---`
|
||||||
|
* lost everything up to it). Agent writes must never lose already-stored content,
|
||||||
|
* so this variant skips the strip. It IS a no-op on canonical serialized content
|
||||||
|
* (which never emits `[^id]:` reference-definition lines).
|
||||||
|
*/
|
||||||
|
export function normalizeAgentMarkdown(markdown: string): string {
|
||||||
|
if (!markdown) return markdown;
|
||||||
|
const src = markdown.replace(/\r\n/g, '\n');
|
||||||
|
return convertReferenceFootnotes(src);
|
||||||
|
}
|
||||||
@@ -15,10 +15,29 @@ export {
|
|||||||
} from "./markdown-document.js";
|
} from "./markdown-document.js";
|
||||||
export type { DocmostMdMeta } from "./markdown-document.js";
|
export type { DocmostMdMeta } from "./markdown-document.js";
|
||||||
|
|
||||||
export { convertProseMirrorToMarkdown } from "./markdown-converter.js";
|
export {
|
||||||
|
convertProseMirrorToMarkdown,
|
||||||
|
ConverterLossError,
|
||||||
|
} from "./markdown-converter.js";
|
||||||
export type { ConvertProseMirrorToMarkdownOptions } from "./markdown-converter.js";
|
export type { ConvertProseMirrorToMarkdownOptions } from "./markdown-converter.js";
|
||||||
|
|
||||||
export { markdownToProseMirror } from "./markdown-to-prosemirror.js";
|
export {
|
||||||
|
markdownToProseMirror,
|
||||||
|
markdownToProseMirrorSync,
|
||||||
|
} from "./markdown-to-prosemirror.js";
|
||||||
|
|
||||||
|
// Foreign-markdown normalizer (#493): the input-liberal pre-pass that rewrites
|
||||||
|
// GFM `[^id]` reference footnotes to canonical inline `^[body]`. Two variants:
|
||||||
|
// `normalizeForeignMarkdown` (server FILE-import boundary) ALSO strips a leading
|
||||||
|
// YAML front-matter block; `normalizeAgentMarkdown` (canonical AGENT-WRITE path,
|
||||||
|
// mcp `markdownToProseMirrorCanonical`) does NOT — a full-body agent rewrite must
|
||||||
|
// not lose a leading `---…---` horizontalRule to the front-matter strip (#493
|
||||||
|
// review). The reference-footnote rewrite is shared so agent + import stay unified
|
||||||
|
// where it matters, without the content-losing strip on the write path.
|
||||||
|
export {
|
||||||
|
normalizeForeignMarkdown,
|
||||||
|
normalizeAgentMarkdown,
|
||||||
|
} from "./foreign-markdown.js";
|
||||||
|
|
||||||
// The Docmost tiptap schema mirror. Exposed so consumers (and the sync
|
// The Docmost tiptap schema mirror. Exposed so consumers (and the sync
|
||||||
// engine's schema-validity regression tests) can build the exact ProseMirror
|
// engine's schema-validity regression tests) can build the exact ProseMirror
|
||||||
@@ -73,6 +92,17 @@ export type { OutlineEntry } from "./node-ops.js";
|
|||||||
// string (#414: single copy shared by mcp and the CommonJS server app).
|
// string (#414: single copy shared by mcp and the CommonJS server app).
|
||||||
export { parseNodeArg } from "./parse-node-arg.js";
|
export { parseNodeArg } from "./parse-node-arg.js";
|
||||||
|
|
||||||
|
// Locator markdown-stripping (#493 dedup): the single canonical copy of the
|
||||||
|
// markdown-tolerant anchor-normalization primitives, imported by mcp's
|
||||||
|
// text-normalize.ts instead of a forked duplicate. `stripInlineMarkdown` is the
|
||||||
|
// lenient locator normalizer (trims stray decoration); `stripWrappersAndLinks`
|
||||||
|
// is the strict balanced-wrapper/link primitive mcp builds `stripBalancedWrappers`
|
||||||
|
// on top of.
|
||||||
|
export {
|
||||||
|
stripInlineMarkdown,
|
||||||
|
stripWrappersAndLinks,
|
||||||
|
} from "./text-normalize.js";
|
||||||
|
|
||||||
// Inline-footnote authoring convention (#414: single copy, formerly the mcp
|
// Inline-footnote authoring convention (#414: single copy, formerly the mcp
|
||||||
// `footnote-authoring.ts` fork), shared with the importer's `assembleFootnotes`.
|
// `footnote-authoring.ts` fork), shared with the importer's `assembleFootnotes`.
|
||||||
export {
|
export {
|
||||||
|
|||||||
@@ -33,6 +33,26 @@ import {
|
|||||||
*/
|
*/
|
||||||
const MAX_NODE_DEPTH = 400;
|
const MAX_NODE_DEPTH = 400;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Thrown by {@link convertProseMirrorToMarkdown} in `strict` mode when it hits a
|
||||||
|
* node or mark type it has no lossless markdown form for (the serializer would
|
||||||
|
* otherwise silently degrade it — drop an unknown mark, flatten an unknown node
|
||||||
|
* to its children). Carries the offending kind/name so a caller (git-sync) can
|
||||||
|
* surface exactly what would have been lost.
|
||||||
|
*/
|
||||||
|
export class ConverterLossError extends Error {
|
||||||
|
readonly kind: "node" | "mark";
|
||||||
|
readonly typeName: string;
|
||||||
|
constructor(kind: "node" | "mark", typeName: string) {
|
||||||
|
super(
|
||||||
|
`convertProseMirrorToMarkdown: unknown ${kind} type "${typeName}" has no lossless markdown representation (strict mode)`,
|
||||||
|
);
|
||||||
|
this.name = "ConverterLossError";
|
||||||
|
this.kind = kind;
|
||||||
|
this.typeName = typeName;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Options for {@link convertProseMirrorToMarkdown}.
|
* Options for {@link convertProseMirrorToMarkdown}.
|
||||||
*/
|
*/
|
||||||
@@ -46,6 +66,23 @@ export interface ConvertProseMirrorToMarkdownOptions {
|
|||||||
* path where resolved anchors MUST be preserved for round-tripping.
|
* path where resolved anchors MUST be preserved for round-tripping.
|
||||||
*/
|
*/
|
||||||
dropResolvedCommentAnchors?: boolean;
|
dropResolvedCommentAnchors?: boolean;
|
||||||
|
/**
|
||||||
|
* Optional sink for LOSS warnings. When the serializer reaches a node or mark
|
||||||
|
* type it has no dedicated case for, it degrades gracefully (flattens an
|
||||||
|
* unknown node to its children, drops an unknown mark) — historically a SILENT
|
||||||
|
* data loss. When this array is provided, one human-readable message per such
|
||||||
|
* event is pushed here so the caller can observe (and log) what was degraded.
|
||||||
|
* Not provided by default -> behavior is byte-identical to before for existing
|
||||||
|
* callers.
|
||||||
|
*/
|
||||||
|
warnings?: string[];
|
||||||
|
/**
|
||||||
|
* When true, THROW a {@link ConverterLossError} on the FIRST unknown node/mark
|
||||||
|
* instead of degrading silently — a warning becomes a hard error. Used by the
|
||||||
|
* lossless git-sync export path and the converter tests, where an unmapped
|
||||||
|
* type is a bug to surface, not data to quietly drop.
|
||||||
|
*/
|
||||||
|
strict?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -63,6 +100,70 @@ export interface ConvertProseMirrorToMarkdownOptions {
|
|||||||
* separator is emitted for any other join, so non-list output is unchanged.
|
* separator is emitted for any other join, so non-list output is unchanged.
|
||||||
*/
|
*/
|
||||||
const LIST_MARKER_SEPARATOR = "<!-- -->";
|
const LIST_MARKER_SEPARATOR = "<!-- -->";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Backslash-escape a leading markdown BLOCK trigger so a serialized paragraph
|
||||||
|
* line re-parses as a PARAGRAPH, not another block. Without this, a paragraph
|
||||||
|
* whose text begins at column 0 with an ATX heading `#`, a blockquote/callout
|
||||||
|
* `>`, a bullet marker `-`/`*`/`+`, an ordered marker `N.`/`N)`, a code fence
|
||||||
|
* (```` ``` ````/`~~~`), a table `|`, or a thematic break (`---`/`***`/`___`,
|
||||||
|
* solid or spaced) silently becomes a heading/list/quote/code block/table/rule
|
||||||
|
* on the next markdown -> ProseMirror import — a known data-loss class (the
|
||||||
|
* thematic-break case drops the text entirely, since a horizontalRule carries
|
||||||
|
* none). CommonMark's escape tokenizer decodes the inserted `\` back to the
|
||||||
|
* literal character on import AND stops the block interpretation, so the line
|
||||||
|
* round-trips byte-exact as paragraph text. Only the FIRST offending character
|
||||||
|
* is escaped (the minimum needed to break block recognition); a line that does
|
||||||
|
* NOT open a block — emphasis `**x**`, an inline code span, ordinary prose — is
|
||||||
|
* returned verbatim, so there is no backslash churn for the common case.
|
||||||
|
*
|
||||||
|
* Applied ONLY to paragraph text, once per `\n`-separated LINE (the paragraph
|
||||||
|
* case splits on `\n` — each hardBreak emits ` \n` — so a trigger on a
|
||||||
|
* continuation line is escaped too): headings/lists/blockquotes legitimately
|
||||||
|
* open with these markers and render them from their own cases. This is the
|
||||||
|
* single, canonical fix for the class the client bridge worked around with a
|
||||||
|
* ZWSP (`gitmost-recording.ts`) and the generative suite self-censored around
|
||||||
|
* (`text-arbitraries.ts`) — both now removed.
|
||||||
|
*/
|
||||||
|
function escapeLeadingBlockTrigger(line: string): string {
|
||||||
|
// ATX heading: 1..6 `#` then whitespace/EOL.
|
||||||
|
if (/^#{1,6}(?:\s|$)/.test(line)) return "\\" + line;
|
||||||
|
// Blockquote / Docmost callout opener (`>` or `> [!info]`).
|
||||||
|
if (line.startsWith(">")) return "\\" + line;
|
||||||
|
// Bullet list marker then whitespace/EOL. Emphasis (`*x*`, `**x**`) has no
|
||||||
|
// space after the leading marker and is intentionally left verbatim.
|
||||||
|
if (/^[-*+](?:\s|$)/.test(line)) return "\\" + line;
|
||||||
|
// Ordered list marker `N.` / `N)`: escape the DELIMITER so the digits stay
|
||||||
|
// literal (`1. x` -> `1\. x`, which imports back as the text `1. x`).
|
||||||
|
const ordered = line.match(/^(\d+)[.)](?:\s|$)/);
|
||||||
|
if (ordered) {
|
||||||
|
const digits = ordered[1].length;
|
||||||
|
return line.slice(0, digits) + "\\" + line.slice(digits);
|
||||||
|
}
|
||||||
|
// Fenced code block: 3+ backticks or tildes. A single/double backtick is an
|
||||||
|
// inline code span and is left verbatim.
|
||||||
|
if (/^(?:`{3,}|~{3,})/.test(line)) return "\\" + line;
|
||||||
|
// Thematic break: a WHOLE line of 3+ identical `-`/`*`/`_`, optionally spaced.
|
||||||
|
if (/^([-*_])(?:\s*\1){2,}\s*$/.test(line)) return "\\" + line;
|
||||||
|
// Setext underline: a continuation line (after a hardBreak) that is ONLY `-`
|
||||||
|
// or ONLY `=` (any count, trailing spaces allowed). Under a paragraph line
|
||||||
|
// such a line re-parses as a SETEXT HEADING and SILENTLY DROPS its own text
|
||||||
|
// (`a\n--` -> heading "a", the `--` is LOST; `a\n=` -> heading "a", `=` LOST).
|
||||||
|
// The bullet arm above catches a lone `-` (via its `$`) and the thematic arm
|
||||||
|
// catches 3+ dashes, but exactly TWO dashes (`--`) fall through both; and no
|
||||||
|
// arm covers a lone `=` at all (a `==` pair is neutralized earlier by the
|
||||||
|
// inline `==`->`\=\=` escape, so only a single `=` line reaches here). Escaping
|
||||||
|
// the leading char (`\--`, `\=`) breaks the setext interpretation so the line
|
||||||
|
// round-trips as paragraph text. The WHOLE line must be the marker (anchored
|
||||||
|
// `^-+`/`^=+` to EOL), so a mid-content `-`/`=` is never spuriously escaped;
|
||||||
|
// and a `---`/`----` already handled by the thematic arm never reaches here,
|
||||||
|
// so there is no double-escape.
|
||||||
|
if (/^-+[ \t]*$/.test(line) || /^=+[ \t]*$/.test(line)) return "\\" + line;
|
||||||
|
// GFM table row opener.
|
||||||
|
if (line.startsWith("|")) return "\\" + line;
|
||||||
|
return line;
|
||||||
|
}
|
||||||
|
|
||||||
function listMarkerFamily(type: string | undefined): "ul" | "ol" | null {
|
function listMarkerFamily(type: string | undefined): "ul" | "ol" | null {
|
||||||
if (type === "bulletList" || type === "taskList") return "ul";
|
if (type === "bulletList" || type === "taskList") return "ul";
|
||||||
if (type === "orderedList") return "ol";
|
if (type === "orderedList") return "ol";
|
||||||
@@ -109,6 +210,26 @@ export function convertProseMirrorToMarkdown(
|
|||||||
// callers (mcp getPage / in-app AI chat) pass it true.
|
// callers (mcp getPage / in-app AI chat) pass it true.
|
||||||
const dropResolvedCommentAnchors = options.dropResolvedCommentAnchors === true;
|
const dropResolvedCommentAnchors = options.dropResolvedCommentAnchors === true;
|
||||||
|
|
||||||
|
// Loss reporting for node/mark types with no dedicated serializer case. In
|
||||||
|
// `strict` mode the FIRST such type throws (git-sync, tests); otherwise the
|
||||||
|
// serializer degrades gracefully (as it always has) but records one warning
|
||||||
|
// per unmapped type into the optional sink so the loss is observable, not
|
||||||
|
// silent. Deduped per type so a document with many unknown nodes of one type
|
||||||
|
// produces one message.
|
||||||
|
const strict = options.strict === true;
|
||||||
|
const warningsSink = options.warnings;
|
||||||
|
const seenLossTypes = new Set<string>();
|
||||||
|
const warnLoss = (kind: "node" | "mark", typeName: string): void => {
|
||||||
|
if (strict) throw new ConverterLossError(kind, typeName);
|
||||||
|
if (!warningsSink) return;
|
||||||
|
const key = `${kind}:${typeName}`;
|
||||||
|
if (seenLossTypes.has(key)) return;
|
||||||
|
seenLossTypes.add(key);
|
||||||
|
warningsSink.push(
|
||||||
|
`Unknown ${kind} type "${typeName}" has no lossless markdown form; it was degraded on export.`,
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
// Escape a value interpolated into an HTML double-quoted attribute value
|
// Escape a value interpolated into an HTML double-quoted attribute value
|
||||||
// (textAlign, colors, image src, math `text`, all data-* attrs, etc.). In the
|
// (textAlign, colors, image src, math `text`, all data-* attrs, etc.). In the
|
||||||
// ATTRIBUTE context only the quote that delimits the value and the ampersand
|
// ATTRIBUTE context only the quote that delimits the value and the ampersand
|
||||||
@@ -362,6 +483,99 @@ export function convertProseMirrorToMarkdown(
|
|||||||
return `<table><tbody>${htmlRows}</tbody></table>`;
|
return `<table><tbody>${htmlRows}</tbody></table>`;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// Layer the intentional inline escapes onto a NON-code text run BEFORE its
|
||||||
|
// marks are applied. Extracted so both `case "text"` and the #515 code-emphasis
|
||||||
|
// run factoring (renderInlineChildren) escape the inner text identically. NEVER
|
||||||
|
// called on code content (a code span is literal — see the gating in the text
|
||||||
|
// case and the run helper). Order is load-bearing: the footnote raw-backslash
|
||||||
|
// doubling MUST precede the `==`/`$`/`^[` escapes (see inFootnoteBody).
|
||||||
|
const escapeInlineText = (text: string): string => {
|
||||||
|
let t = text;
|
||||||
|
if (inFootnoteBody) t = t.replace(/\\/g, "\\\\");
|
||||||
|
t = t.replace(/==/g, "\\=\\=");
|
||||||
|
t = escapeProseMath(t);
|
||||||
|
t = t.replace(/\^\[/g, "^\\[");
|
||||||
|
return t;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Wrap `text` with the markdown/HTML form of a SINGLE inline mark. Extracted
|
||||||
|
// from `case "text"` so the same per-mark emission is reused when the #515
|
||||||
|
// run factoring layers a shared outer mark over a code-emphasis run. `code` is
|
||||||
|
// handled by the callers (wrapped innermost, before this runs), so this branch
|
||||||
|
// is defensive only. For any non-code mark the output is byte-identical to the
|
||||||
|
// pre-#515 inline switch.
|
||||||
|
const applyInlineMark = (text: string, mark: any): string => {
|
||||||
|
switch (mark.type) {
|
||||||
|
case "bold":
|
||||||
|
return `**${text}**`;
|
||||||
|
case "italic":
|
||||||
|
return `*${text}*`;
|
||||||
|
case "code":
|
||||||
|
// Callers wrap the code span innermost themselves; reached only if a
|
||||||
|
// mark list is applied through here directly. Emit the backtick span.
|
||||||
|
return `\`${text}\``;
|
||||||
|
case "link": {
|
||||||
|
const href = mark.attrs?.href || "";
|
||||||
|
const title = mark.attrs?.title;
|
||||||
|
if (title) {
|
||||||
|
// Emit the optional markdown link title; escape an embedded double-
|
||||||
|
// quote so it cannot terminate the title string early.
|
||||||
|
const safeTitle = String(title).replace(/"/g, '\\"');
|
||||||
|
return `[${text}](${href} "${safeTitle}")`;
|
||||||
|
}
|
||||||
|
return `[${text}](${href})`;
|
||||||
|
}
|
||||||
|
case "strike":
|
||||||
|
return `~~${text}~~`;
|
||||||
|
case "underline":
|
||||||
|
return `<u>${text}</u>`;
|
||||||
|
case "subscript":
|
||||||
|
return `<sub>${text}</sub>`;
|
||||||
|
case "superscript":
|
||||||
|
return `<sup>${text}</sup>`;
|
||||||
|
case "highlight": {
|
||||||
|
// #293 canon #7: a highlight WITHOUT a color serializes as the
|
||||||
|
// Obsidian/GFM `==text==` syntax; a colored highlight keeps the `<mark
|
||||||
|
// style>` HTML form. The inner text already had any literal `==`
|
||||||
|
// backslash-escaped upstream.
|
||||||
|
const color = mark.attrs?.color;
|
||||||
|
return color
|
||||||
|
? `<mark style="background-color: ${escapeAttr(color)}">${text}</mark>`
|
||||||
|
: `==${text}==`;
|
||||||
|
}
|
||||||
|
case "textStyle":
|
||||||
|
if (mark.attrs?.color) {
|
||||||
|
return `<span style="color: ${escapeAttr(mark.attrs.color)}">${text}</span>`;
|
||||||
|
}
|
||||||
|
return text;
|
||||||
|
case "spoiler":
|
||||||
|
// Markdown has no native spoiler syntax, so emit the same raw inline HTML
|
||||||
|
// the editor-ext/MCP stack uses (span[data-spoiler] round-trips).
|
||||||
|
return `<span data-spoiler="true">${text}</span>`;
|
||||||
|
case "comment": {
|
||||||
|
// Inline comment anchor (span[data-comment-id]); resolved anchors are
|
||||||
|
// optionally dropped for agent reads, keeping only the bare text.
|
||||||
|
const cid = mark.attrs?.commentId;
|
||||||
|
if (cid) {
|
||||||
|
if (mark.attrs?.resolved && dropResolvedCommentAnchors) {
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
const resolvedAttr = mark.attrs?.resolved
|
||||||
|
? ` data-resolved="true"`
|
||||||
|
: "";
|
||||||
|
return `<span data-comment-id="${escapeAttr(cid)}"${resolvedAttr}>${text}</span>`;
|
||||||
|
}
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
// Unknown mark: no dedicated case, so it has no markdown form and is
|
||||||
|
// dropped from the run. Report the loss (throws in strict mode) then
|
||||||
|
// leave the text unwrapped — the historical behavior.
|
||||||
|
warnLoss("mark", String(mark.type));
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
const processNode = (node: any): string => {
|
const processNode = (node: any): string => {
|
||||||
if (nodeDepth >= MAX_NODE_DEPTH) {
|
if (nodeDepth >= MAX_NODE_DEPTH) {
|
||||||
// Bail out of deeper recursion without throwing. A text node still has
|
// Bail out of deeper recursion without throwing. A text node still has
|
||||||
@@ -412,7 +626,17 @@ export function convertProseMirrorToMarkdown(
|
|||||||
}
|
}
|
||||||
|
|
||||||
case "paragraph": {
|
case "paragraph": {
|
||||||
const text = renderInlineChildren(nodeContent);
|
// Escape a leading block trigger on EVERY line of the paragraph, not
|
||||||
|
// just the first: a hardBreak serializes as ` \n`, so a `#`/`-`/`>`/
|
||||||
|
// `1.`/`|`/fence/`---` at the start of a CONTINUATION line would also
|
||||||
|
// re-parse into another block on the next import (a heading/list/table/
|
||||||
|
// setext-`---`), and for the text-less thematic/setext case would LOSE
|
||||||
|
// that line's text entirely. Escaping each `\n`-separated line closes
|
||||||
|
// the class for multi-line paragraphs too.
|
||||||
|
const text = renderInlineChildren(nodeContent)
|
||||||
|
.split("\n")
|
||||||
|
.map(escapeLeadingBlockTrigger)
|
||||||
|
.join("\n");
|
||||||
const align = node.attrs?.textAlign;
|
const align = node.attrs?.textAlign;
|
||||||
// Non-default alignment round-trips as an ATTACHED HTML comment at the
|
// Non-default alignment round-trips as an ATTACHED HTML comment at the
|
||||||
// END of the block line (#293 canon #9):
|
// END of the block line (#293 canon #9):
|
||||||
@@ -451,154 +675,38 @@ export function convertProseMirrorToMarkdown(
|
|||||||
return headingLine;
|
return headingLine;
|
||||||
}
|
}
|
||||||
|
|
||||||
case "text":
|
case "text": {
|
||||||
let textContent = node.text || "";
|
let textContent = node.text || "";
|
||||||
// #293 canon #7: `==` is now a LIVE inline highlight syntax on import (a
|
// #515: `code` is no longer exclusive (`excludes: ""`), so a run may
|
||||||
// marked inline extension turns `==text==` into a color-less highlight
|
// carry `code` TOGETHER with other marks. The inner escapes below apply
|
||||||
// mark). A LITERAL `==` in a text run would therefore be misparsed as a
|
// ONLY to a NON-code run (a code span's content is literal — `==`, `$…$`,
|
||||||
// highlight on the next import, so backslash-escape each `=` of a `==`
|
// `^[` must stay verbatim, matching `` `a == b` `` staying code). See
|
||||||
// pair; marked's escape tokenizer decodes `\=` back to a literal `=`, so
|
// #293 canon #2/#6/#7 for why each escape exists (extracted into
|
||||||
// a literal `==` round-trips as text (never materializes a phantom mark).
|
// escapeInlineText). A code run's `==`/`$`/`^[` are protected by the
|
||||||
// This runs for BOTH unmarked text and marked non-code runs, but NOT for
|
// backticks, so they are never misparsed on re-import.
|
||||||
// an inline code span (a run carrying the `code` mark returns a backtick
|
const hasCode = (node.marks || []).some((m: any) => m.type === "code");
|
||||||
// span below with `==` verbatim, matching `` `a == b` `` staying code).
|
if (!hasCode) {
|
||||||
// A highlight run's own `==` delimiters are appended AFTER this in the
|
textContent = escapeInlineText(textContent);
|
||||||
// marks loop, so they are never escaped; only the run's inner text is.
|
|
||||||
if (!(node.marks || []).some((m: any) => m.type === "code")) {
|
|
||||||
// #293 canon #2 (F2): inside a footnote body, DOUBLE every RAW user
|
|
||||||
// backslash FIRST, so it survives `^[…]` (the import tokenizer treats
|
|
||||||
// `\<char>` as an escape when balancing brackets, and `parseInline`
|
|
||||||
// decodes escapes). Doing it before the intentional escapes below keeps
|
|
||||||
// the serializer's own single escapes (`\=` `\$` `^\[`, and the `\[`/
|
|
||||||
// `\]` balanceBrackets adds) single; only genuine user backslashes are
|
|
||||||
// doubled. Skipped for code runs (a code span's content is NOT decoded
|
|
||||||
// by parseInline, so its backslashes must stay verbatim).
|
|
||||||
if (inFootnoteBody) {
|
|
||||||
textContent = textContent.replace(/\\/g, "\\\\");
|
|
||||||
}
|
|
||||||
textContent = textContent.replace(/==/g, "\\=\\=");
|
|
||||||
// #293 canon #6: escape a would-be inline-math `$…$` span so it stays
|
|
||||||
// literal text on re-import (currency `$5` is left clean — see
|
|
||||||
// escapeProseMath). Runs on the SAME non-code runs as the `==` escape
|
|
||||||
// above; an inline `code` run returns verbatim below, matching the
|
|
||||||
// codeBlock path (a `$…$` inside code must stay code, never math).
|
|
||||||
textContent = escapeProseMath(textContent);
|
|
||||||
// #293 canon #2: `^[` opens a LIVE inline-footnote span on import
|
|
||||||
// (`^[text]` -> a footnote reference). A LITERAL `^[` in prose text
|
|
||||||
// would therefore materialize a phantom footnote on the next import, so
|
|
||||||
// backslash-escape the bracket (`^[` -> `^\[`); marked's escape
|
|
||||||
// tokenizer decodes `\[` back to `[`, so a literal `^[…]` round-trips
|
|
||||||
// as text and never opens a footnote. Only the OPENING `^[` needs
|
|
||||||
// breaking (the tokenizer requires it), so this is a minimal, idempotent
|
|
||||||
// escape. A real footnoteReference node emits `^[body]` from its own
|
|
||||||
// case, never through here.
|
|
||||||
textContent = textContent.replace(/\^\[/g, "^\\[");
|
|
||||||
}
|
}
|
||||||
// Apply marks (bold, italic, code, etc.)
|
|
||||||
if (node.marks) {
|
if (node.marks) {
|
||||||
// The schema's `code` mark declares `excludes: "_"` — it excludes every
|
// #515: wrap the backtick code span FIRST (innermost mark), then layer
|
||||||
// other inline mark — so the editor can NEVER produce a text run that
|
// the REMAINING marks in array order. For a run WITHOUT a code mark the
|
||||||
// carries `code` together with another mark, and on import any
|
// loop applies every mark exactly as the pre-#515 switch did, so the
|
||||||
// co-occurring mark is always dropped (the run comes back as code-only).
|
// output is byte-identical. For a code+emphasis run the code span sits
|
||||||
// The lossless, byte-stable behavior is therefore: when a run has the
|
// inside the emphasis delimiters (`` **`code`** ``), matching CommonMark.
|
||||||
// `code` mark, emit ONLY the backtick code span and ignore every other
|
// The shared-mark grouping across ADJACENT nodes (`` **`a` + `b`** ``)
|
||||||
// mark, so md1 is already code-only and md2 === md1. Runs WITHOUT a code
|
// lives in renderInlineChildren; this direct path handles a lone run
|
||||||
// mark are rendered exactly as before.
|
// and the table/`default` callers that invoke processNode per node.
|
||||||
const markTypes = node.marks.map((m: any) => m.type);
|
|
||||||
const hasCode = markTypes.includes("code");
|
|
||||||
if (hasCode) {
|
if (hasCode) {
|
||||||
textContent = `\`${textContent}\``;
|
textContent = `\`${textContent}\``;
|
||||||
return textContent;
|
|
||||||
}
|
}
|
||||||
for (const mark of node.marks) {
|
for (const mark of node.marks) {
|
||||||
switch (mark.type) {
|
if (mark.type === "code") continue; // wrapped innermost above
|
||||||
case "bold":
|
textContent = applyInlineMark(textContent, mark);
|
||||||
textContent = `**${textContent}**`;
|
|
||||||
break;
|
|
||||||
case "italic":
|
|
||||||
textContent = `*${textContent}*`;
|
|
||||||
break;
|
|
||||||
case "code":
|
|
||||||
// A `code` run already returned above (hasCode early return), so
|
|
||||||
// this branch is only reached for a non-code run that somehow
|
|
||||||
// still lists `code`; emit the plain backtick span.
|
|
||||||
textContent = `\`${textContent}\``;
|
|
||||||
break;
|
|
||||||
case "link": {
|
|
||||||
const href = mark.attrs?.href || "";
|
|
||||||
const title = mark.attrs?.title;
|
|
||||||
if (title) {
|
|
||||||
// Emit the optional markdown link title; escape an embedded
|
|
||||||
// double-quote so it cannot terminate the title string early.
|
|
||||||
const safeTitle = String(title).replace(/"/g, '\\"');
|
|
||||||
textContent = `[${textContent}](${href} "${safeTitle}")`;
|
|
||||||
} else {
|
|
||||||
textContent = `[${textContent}](${href})`;
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
case "strike":
|
|
||||||
textContent = `~~${textContent}~~`;
|
|
||||||
break;
|
|
||||||
case "underline":
|
|
||||||
textContent = `<u>${textContent}</u>`;
|
|
||||||
break;
|
|
||||||
case "subscript":
|
|
||||||
textContent = `<sub>${textContent}</sub>`;
|
|
||||||
break;
|
|
||||||
case "superscript":
|
|
||||||
textContent = `<sup>${textContent}</sup>`;
|
|
||||||
break;
|
|
||||||
case "highlight": {
|
|
||||||
// #293 canon #7: a highlight WITHOUT a color serializes as the
|
|
||||||
// Obsidian/GFM `==text==` syntax (the importer's marked inline
|
|
||||||
// `==` extension parses it back to a color-less highlight mark).
|
|
||||||
// A highlight WITH a color keeps the `<mark style="background-
|
|
||||||
// color: …">` HTML form (the condition is deterministic on the
|
|
||||||
// `color` attr), so a colored highlight is not flattened. The
|
|
||||||
// inner textContent already had any literal `==` backslash-
|
|
||||||
// escaped above, so a highlight over text containing `==` still
|
|
||||||
// round-trips.
|
|
||||||
const color = mark.attrs?.color;
|
|
||||||
textContent = color
|
|
||||||
? `<mark style="background-color: ${escapeAttr(color)}">${textContent}</mark>`
|
|
||||||
: `==${textContent}==`;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
case "textStyle":
|
|
||||||
if (mark.attrs?.color) {
|
|
||||||
textContent = `<span style="color: ${escapeAttr(mark.attrs.color)}">${textContent}</span>`;
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
case "spoiler":
|
|
||||||
// Markdown has no native spoiler syntax, so emit the same raw
|
|
||||||
// inline HTML the editor-ext/MCP stack uses. The schema's Spoiler
|
|
||||||
// mark parses span[data-spoiler] back on import, so the mark
|
|
||||||
// survives the PM -> MD -> PM round-trip.
|
|
||||||
textContent = `<span data-spoiler="true">${textContent}</span>`;
|
|
||||||
break;
|
|
||||||
case "comment": {
|
|
||||||
// Emit the inline comment anchor so highlights round-trip. The
|
|
||||||
// schema's Comment mark parses span[data-comment-id] (attrs
|
|
||||||
// commentId/resolved).
|
|
||||||
const cid = mark.attrs?.commentId;
|
|
||||||
if (cid) {
|
|
||||||
// Hide resolved anchors from agent reads: drop the wrapper and
|
|
||||||
// keep only the bare text. Active anchors keep their wrapper.
|
|
||||||
if (mark.attrs?.resolved && dropResolvedCommentAnchors) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
const resolvedAttr = mark.attrs?.resolved
|
|
||||||
? ` data-resolved="true"`
|
|
||||||
: "";
|
|
||||||
textContent = `<span data-comment-id="${escapeAttr(cid)}"${resolvedAttr}>${textContent}</span>`;
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return textContent;
|
return textContent;
|
||||||
|
}
|
||||||
|
|
||||||
case "codeBlock":
|
case "codeBlock":
|
||||||
const language = node.attrs?.language || "";
|
const language = node.attrs?.language || "";
|
||||||
@@ -1173,7 +1281,11 @@ export function convertProseMirrorToMarkdown(
|
|||||||
}
|
}
|
||||||
|
|
||||||
default:
|
default:
|
||||||
// Fallback: process children
|
// Unknown node type: no dedicated case, so the node's identity + attrs
|
||||||
|
// have no lossless markdown form. Report the loss (throws in strict
|
||||||
|
// mode) then degrade by flattening to its children — the historical
|
||||||
|
// graceful fallback.
|
||||||
|
warnLoss("node", String(type));
|
||||||
return nodeContent.map(processNode).join("");
|
return nodeContent.map(processNode).join("");
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
@@ -1186,18 +1298,165 @@ export function convertProseMirrorToMarkdown(
|
|||||||
// For that node ONLY we fall back to the lossless schema-HTML `<span>` form.
|
// For that node ONLY we fall back to the lossless schema-HTML `<span>` form.
|
||||||
// Every other inline node is rendered exactly as processNode would, so output
|
// Every other inline node is rendered exactly as processNode would, so output
|
||||||
// is unchanged whenever no math sits directly before a digit.
|
// is unchanged whenever no math sits directly before a digit.
|
||||||
|
// #515: a "bare-delimiter" emphasis mark is one that serializes as a naked
|
||||||
|
// markdown delimiter run (`**` `*` `~~` `==`) — bold / italic / strike /
|
||||||
|
// UNCOLORED highlight. These delimiters COLLIDE with the backtick-flanking
|
||||||
|
// delimiters emitted around a code+emphasis run: rendering `[code,bold]` next
|
||||||
|
// to `[italic]` node-by-node would produce `` **`a`***b* `` (a `***` run that
|
||||||
|
// re-imports wrong). Every OTHER mark (underline/sub/sup/spoiler/comment/
|
||||||
|
// textStyle/colored-highlight/link) emits an HTML/bracket form whose boundaries
|
||||||
|
// do NOT collapse, so those neighbors never join a run.
|
||||||
|
const isBareEmphasisMark = (mark: any): boolean => {
|
||||||
|
switch (mark?.type) {
|
||||||
|
case "bold":
|
||||||
|
case "italic":
|
||||||
|
case "strike":
|
||||||
|
return true;
|
||||||
|
case "highlight":
|
||||||
|
return !mark.attrs?.color; // colored highlight emits <mark>, not `==`
|
||||||
|
default:
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// A text node participates in a code-emphasis run iff it carries at least one
|
||||||
|
// bare-delimiter emphasis mark. A code-ONLY node (no emphasis) does NOT — so a
|
||||||
|
// plain `` `code` `` next to `**bold**` keeps its clean, byte-identical
|
||||||
|
// markdown (they share no colliding delimiter). Existing pages, where a code
|
||||||
|
// node could never carry emphasis, therefore serialize exactly as before.
|
||||||
|
const isEmphasisMember = (node: any): boolean =>
|
||||||
|
node?.type === "text" &&
|
||||||
|
(node.marks || []).some((m: any) => isBareEmphasisMark(m));
|
||||||
|
|
||||||
|
// The run's non-code marks (order preserved) — the candidate marks to factor.
|
||||||
|
const nonCodeMarks = (node: any): any[] =>
|
||||||
|
(node.marks || []).filter((m: any) => m.type !== "code");
|
||||||
|
|
||||||
|
// Deep structural equality of two marks (type + full attrs). Two `link` marks
|
||||||
|
// are equal only when EVERY attr matches (class/href/internal/rel/target/title
|
||||||
|
// — not just href), so a homogeneous run never merges links that differ.
|
||||||
|
const marksEqual = (a: any, b: any): boolean =>
|
||||||
|
a.type === b.type &&
|
||||||
|
JSON.stringify(a.attrs ?? null) === JSON.stringify(b.attrs ?? null);
|
||||||
|
|
||||||
|
// Two non-code mark lists are equal AS SETS (a run is homogeneous when every
|
||||||
|
// node shares the identical non-code mark set — order-independent).
|
||||||
|
const markSetsEqual = (a: any[], b: any[]): boolean =>
|
||||||
|
a.length === b.length &&
|
||||||
|
a.every((ma) => b.some((mb) => marksEqual(ma, mb))) &&
|
||||||
|
b.every((mb) => a.some((ma) => marksEqual(mb, ma)));
|
||||||
|
|
||||||
|
// Serialize one node's INNER form for a homogeneous run: the factored marks are
|
||||||
|
// applied by the caller, so here a code node emits only its literal backtick
|
||||||
|
// span and a non-code node emits only its (escaped) text.
|
||||||
|
const renderRunInner = (node: any): string => {
|
||||||
|
const text = node.text || "";
|
||||||
|
if ((node.marks || []).some((m: any) => m.type === "code")) {
|
||||||
|
return `\`${text}\``; // code content is literal
|
||||||
|
}
|
||||||
|
return escapeInlineText(text);
|
||||||
|
};
|
||||||
|
|
||||||
|
// A markdown emphasis delimiter (`**`/`*`/`~~`/`==`) wrapping a code span opens
|
||||||
|
// with the delimiter immediately followed by a backtick and closes immediately
|
||||||
|
// preceded by one. A backtick is CommonMark punctuation, so such a delimiter is
|
||||||
|
// only left/right-flanking — able to open/close emphasis — when the character
|
||||||
|
// on its OUTER side is start/end, whitespace or punctuation. If a run boundary
|
||||||
|
// abuts a word character, the delimiter would NOT flank (`a**` `code` `**`
|
||||||
|
// never opens) and the emphasis silently degrades on re-import. This checks the
|
||||||
|
// outer boundary char conservatively: ASCII whitespace or ASCII punctuation (or
|
||||||
|
// the string edge) is safe; anything else (a letter/number, unicode letter or
|
||||||
|
// emoji) is treated as unsafe so the run takes the lossless HTML fallback.
|
||||||
|
const SAFE_BOUNDARY = /[\s!-/:-@[-`{-~]/;
|
||||||
|
const isSafeBoundary = (c: string): boolean => c === "" || SAFE_BOUNDARY.test(c);
|
||||||
|
|
||||||
|
// Serialize a maximal run of adjacent emphasis-member text nodes that contains
|
||||||
|
// at least one `code` node (#515). HOMOGENEOUS (all share the identical
|
||||||
|
// non-code mark set) AND flank-safe on both boundaries: factor the common marks
|
||||||
|
// ONCE around the concatenated inner spans — `` **`aaa` + `bbb`** ``, code
|
||||||
|
// innermost. Otherwise — HETEROGENEOUS (non-code sets differ, e.g. `[code,bold]`
|
||||||
|
// next to `[italic]`) OR a boundary abuts a word char — emit the whole run as
|
||||||
|
// schema-HTML via the lossless inlineToHtml fallback, avoiding a colliding
|
||||||
|
// `***` delimiter run or a non-flanking `a**` that would drop the emphasis.
|
||||||
|
const renderCodeEmphasisRun = (
|
||||||
|
run: any[],
|
||||||
|
prevChar: string,
|
||||||
|
nextChar: string,
|
||||||
|
): string => {
|
||||||
|
const firstNonCode = nonCodeMarks(run[0]);
|
||||||
|
const homogeneous = run.every((n) =>
|
||||||
|
markSetsEqual(nonCodeMarks(n), firstNonCode),
|
||||||
|
);
|
||||||
|
if (!homogeneous || !isSafeBoundary(prevChar) || !isSafeBoundary(nextChar)) {
|
||||||
|
return inlineToHtml(run);
|
||||||
|
}
|
||||||
|
let out = run.map(renderRunInner).join("");
|
||||||
|
// Apply the common non-code marks in the FIRST node's array order (code is
|
||||||
|
// already innermost inside each span).
|
||||||
|
for (const mark of firstNonCode) out = applyInlineMark(out, mark);
|
||||||
|
return out;
|
||||||
|
};
|
||||||
|
|
||||||
const renderInlineChildren = (nodes: any[]): string => {
|
const renderInlineChildren = (nodes: any[]): string => {
|
||||||
const parts = nodes.map(processNode);
|
// Pass 1: segment the nodes. Each segment is either an already-rendered
|
||||||
for (let i = 0; i < nodes.length - 1; i++) {
|
// non-run node / pure-emphasis node (byte-identical to the pre-#515 output),
|
||||||
if (
|
// or a DEFERRED code-emphasis run (a maximal block of consecutive
|
||||||
nodes[i]?.type === "mathInline" &&
|
// emphasis-member text nodes containing a code node) — its markdown-vs-HTML
|
||||||
parts[i].startsWith("$") &&
|
// choice needs the neighbor boundary chars, resolved in pass 2.
|
||||||
/^[0-9]/.test(parts[i + 1] || "")
|
type Seg = { firstNode: any; text?: string; run?: any[] };
|
||||||
) {
|
const segs: Seg[] = [];
|
||||||
parts[i] = mathInlineHtml(nodes[i].attrs?.text || "");
|
let i = 0;
|
||||||
|
while (i < nodes.length) {
|
||||||
|
const node = nodes[i];
|
||||||
|
if (isEmphasisMember(node)) {
|
||||||
|
let j = i;
|
||||||
|
while (j < nodes.length && isEmphasisMember(nodes[j])) j++;
|
||||||
|
const run = nodes.slice(i, j);
|
||||||
|
const hasCode = run.some((n: any) =>
|
||||||
|
(n.marks || []).some((m: any) => m.type === "code"),
|
||||||
|
);
|
||||||
|
if (hasCode) {
|
||||||
|
segs.push({ firstNode: run[0], run });
|
||||||
|
} else {
|
||||||
|
// Pure-emphasis run (no code): render each node as before.
|
||||||
|
for (const n of run) segs.push({ firstNode: n, text: processNode(n) });
|
||||||
|
}
|
||||||
|
i = j;
|
||||||
|
} else {
|
||||||
|
segs.push({ firstNode: node, text: processNode(node) });
|
||||||
|
i++;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return parts.join("");
|
// A deferred run always emits either a delimiter/backtick (markdown) or `<`
|
||||||
|
// (HTML) first — both punctuation — so a following run counts as a safe
|
||||||
|
// boundary for the current one without resolving it first.
|
||||||
|
const firstCharOf = (seg: Seg): string =>
|
||||||
|
seg.text !== undefined ? seg.text[0] || "" : "*";
|
||||||
|
// Pass 2: resolve deferred runs left-to-right, tracking the previous emitted
|
||||||
|
// char (for the opening boundary) and peeking the next segment (for closing).
|
||||||
|
let prevChar = "";
|
||||||
|
for (let k = 0; k < segs.length; k++) {
|
||||||
|
const seg = segs[k];
|
||||||
|
if (seg.text === undefined) {
|
||||||
|
const nextChar = k + 1 < segs.length ? firstCharOf(segs[k + 1]) : "";
|
||||||
|
seg.text = renderCodeEmphasisRun(seg.run!, prevChar, nextChar);
|
||||||
|
}
|
||||||
|
if (seg.text.length > 0) prevChar = seg.text[seg.text.length - 1];
|
||||||
|
}
|
||||||
|
// Preserve the mathInline-before-digit guard: a `$…$` immediately followed by
|
||||||
|
// a digit-leading segment would re-tokenize as a longer math span, so emit
|
||||||
|
// that math node as HTML instead. A code-emphasis run never starts with a
|
||||||
|
// digit (it opens with a delimiter or `<`), so segment granularity is safe.
|
||||||
|
for (let k = 0; k < segs.length - 1; k++) {
|
||||||
|
if (
|
||||||
|
segs[k].firstNode?.type === "mathInline" &&
|
||||||
|
(segs[k].text || "").startsWith("$") &&
|
||||||
|
/^[0-9]/.test(segs[k + 1].text || "")
|
||||||
|
) {
|
||||||
|
segs[k].text = mathInlineHtml(segs[k].firstNode.attrs?.text || "");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return segs.map((s) => s.text).join("");
|
||||||
};
|
};
|
||||||
|
|
||||||
// Render inline content (text runs + their marks) to HTML. Used by the raw
|
// Render inline content (text runs + their marks) to HTML. Used by the raw
|
||||||
@@ -1232,7 +1491,22 @@ export function convertProseMirrorToMarkdown(
|
|||||||
return processNode(n);
|
return processNode(n);
|
||||||
}
|
}
|
||||||
let t = escapeHtmlText(n.text || "");
|
let t = escapeHtmlText(n.text || "");
|
||||||
|
// #515: wrap `<code>` INNERMOST first (before the array-order mark loop),
|
||||||
|
// then skip `code` in the loop. The imported mark order is NOT fixed — it
|
||||||
|
// DEPENDS on the emphasis extension: import (`generateJSON`) yields code
|
||||||
|
// LAST for bold/italic/strike (`[emphasis, code]`) but code FIRST for the
|
||||||
|
// `==`-highlight extension (`[code, highlight]`). So we cannot rely on a
|
||||||
|
// fixed array position; the invariant is instead "wrap `<code>` innermost
|
||||||
|
// regardless of the imported order". That keeps `<code>` nested inside the
|
||||||
|
// emphasis tag both directions (preserving the byte fixpoint — an order-
|
||||||
|
// sensitive loop would flip `<strong><code>`↔`<code><strong>` depending on
|
||||||
|
// which order it happened to see) and matches the markdown path (case
|
||||||
|
// "text" / run factoring).
|
||||||
|
if ((n.marks || []).some((m: any) => m.type === "code")) {
|
||||||
|
t = `<code>${t}</code>`;
|
||||||
|
}
|
||||||
for (const mark of n.marks || []) {
|
for (const mark of n.marks || []) {
|
||||||
|
if (mark.type === "code") continue; // wrapped innermost above
|
||||||
switch (mark.type) {
|
switch (mark.type) {
|
||||||
case "bold":
|
case "bold":
|
||||||
t = `<strong>${t}</strong>`;
|
t = `<strong>${t}</strong>`;
|
||||||
@@ -1297,6 +1571,12 @@ export function convertProseMirrorToMarkdown(
|
|||||||
t = `<span data-comment-id="${escapeAttr(mark.attrs.commentId)}"${r}>${t}</span>`;
|
t = `<span data-comment-id="${escapeAttr(mark.attrs.commentId)}"${r}>${t}</span>`;
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
|
default:
|
||||||
|
// Unknown mark on the raw-HTML path: dropped (no HTML form). Report
|
||||||
|
// the loss (throws in strict mode) — same policy as the markdown
|
||||||
|
// path's marks loop above.
|
||||||
|
warnLoss("mark", String(mark.type));
|
||||||
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return t;
|
return t;
|
||||||
|
|||||||
@@ -7,9 +7,8 @@
|
|||||||
* natively through the collab gateway, so no websocket/Yjs write-path lives
|
* natively through the collab gateway, so no websocket/Yjs write-path lives
|
||||||
* here.
|
* here.
|
||||||
*/
|
*/
|
||||||
import { generateJSON } from "@tiptap/html";
|
|
||||||
import { JSDOM } from "jsdom";
|
|
||||||
import { Marked } from "marked";
|
import { Marked } from "marked";
|
||||||
|
import { parseHtmlDocument, generateJsonWith } from "./dom-parser.js";
|
||||||
import type { TokenizerExtension, RendererExtension } from "marked";
|
import type { TokenizerExtension, RendererExtension } from "marked";
|
||||||
import { docmostExtensions } from "./docmost-schema.js";
|
import { docmostExtensions } from "./docmost-schema.js";
|
||||||
import { parseAttachedComment } from "./attached-comment.js";
|
import { parseAttachedComment } from "./attached-comment.js";
|
||||||
@@ -245,12 +244,13 @@ const markedInstance = new Marked().use({
|
|||||||
],
|
],
|
||||||
});
|
});
|
||||||
|
|
||||||
// Setup DOM environment for Tiptap HTML parsing in Node.js
|
// NOTE: this module no longer installs a module-level `global.window`/`document`
|
||||||
const dom = new JSDOM("<!DOCTYPE html><html><body></body></html>");
|
// jsdom shim. The HTML->DOM passes below (bridgeTaskLists / applyCommentDirectives
|
||||||
global.window = dom.window as any;
|
// / assembleFootnotes) parse via the INJECTED `parseHtmlDocument` (jsdom on the
|
||||||
global.document = dom.window.document;
|
// Node entry, native `DOMParser` on the browser entry), and `@tiptap/html`'s v3
|
||||||
// @ts-ignore
|
// `generateJSON` supplies its OWN DOM per environment (happy-dom in Node, native
|
||||||
global.Element = dom.window.Element;
|
// `DOMParser` in the browser) — so no ambient global DOM is needed here, and
|
||||||
|
// nothing on the browser code path statically imports jsdom.
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Hard ceiling above which we skip callout preprocessing entirely. The linear
|
* Hard ceiling above which we skip callout preprocessing entirely. The linear
|
||||||
@@ -295,7 +295,13 @@ const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
|
|||||||
* - emits the same `<div data-type="callout" data-callout-type="TYPE">` output
|
* - emits the same `<div data-type="callout" data-callout-type="TYPE">` output
|
||||||
* (inner rendered through marked) as the previous regex implementation.
|
* (inner rendered through marked) as the previous regex implementation.
|
||||||
*/
|
*/
|
||||||
async function preprocessCallouts(markdown: string): Promise<string> {
|
// SYNCHRONOUS by construction: the only formerly-awaited call is
|
||||||
|
// `markedInstance.parse`, which returns a string synchronously for this
|
||||||
|
// instance (no async marked extensions are registered), so the whole callout
|
||||||
|
// preprocess is sync. Keeping it sync lets a sync converter entry
|
||||||
|
// (`markdownToProseMirrorSync`, used by the client's chat renderer which must
|
||||||
|
// stay synchronous) share this exact logic with the async entry.
|
||||||
|
function preprocessCallouts(markdown: string): string {
|
||||||
// Defensive cap: skip preprocessing for pathologically large inputs.
|
// Defensive cap: skip preprocessing for pathologically large inputs.
|
||||||
if (markdown.length > MAX_CALLOUT_PREPROCESS_BYTES) {
|
if (markdown.length > MAX_CALLOUT_PREPROCESS_BYTES) {
|
||||||
return markdown;
|
return markdown;
|
||||||
@@ -304,7 +310,7 @@ async function preprocessCallouts(markdown: string): Promise<string> {
|
|||||||
// Recursively transform a slice of lines, converting top-level callouts in
|
// Recursively transform a slice of lines, converting top-level callouts in
|
||||||
// that slice into <div> blocks and rendering their inner content (which may
|
// that slice into <div> blocks and rendering their inner content (which may
|
||||||
// itself contain nested callouts) through this same function.
|
// itself contain nested callouts) through this same function.
|
||||||
const transform = async (lines: string[]): Promise<string> => {
|
const transform = (lines: string[]): string => {
|
||||||
const out: string[] = [];
|
const out: string[] = [];
|
||||||
let inCodeFence = false;
|
let inCodeFence = false;
|
||||||
let codeFenceMarker = ""; // the exact run of backticks/tildes that opened it
|
let codeFenceMarker = ""; // the exact run of backticks/tildes that opened it
|
||||||
@@ -383,8 +389,8 @@ async function preprocessCallouts(markdown: string): Promise<string> {
|
|||||||
if (j < lines.length) {
|
if (j < lines.length) {
|
||||||
// Found the matching closing fence: render the body (recursively, so
|
// Found the matching closing fence: render the body (recursively, so
|
||||||
// nested callouts are handled) and emit the callout div.
|
// nested callouts are handled) and emit the callout div.
|
||||||
const inner = await transform(bodyLines);
|
const inner = transform(bodyLines);
|
||||||
const renderedInner = await markedInstance.parse(inner);
|
const renderedInner = markedInstance.parse(inner) as string;
|
||||||
out.push(
|
out.push(
|
||||||
`\n<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>\n`,
|
`\n<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>\n`,
|
||||||
);
|
);
|
||||||
@@ -423,8 +429,8 @@ async function preprocessCallouts(markdown: string): Promise<string> {
|
|||||||
// Drop the prefix + `>` + one optional space, leaving the body content.
|
// Drop the prefix + `>` + one optional space, leaving the body content.
|
||||||
bodyLines.push(lines[j].slice(prefix.length).replace(/^>\s?/, ""));
|
bodyLines.push(lines[j].slice(prefix.length).replace(/^>\s?/, ""));
|
||||||
}
|
}
|
||||||
const inner = await transform(bodyLines);
|
const inner = transform(bodyLines);
|
||||||
const renderedInner = await markedInstance.parse(inner);
|
const renderedInner = markedInstance.parse(inner) as string;
|
||||||
const block = `<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>`;
|
const block = `<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>`;
|
||||||
if (prefix.length === 0) {
|
if (prefix.length === 0) {
|
||||||
// Top-level callout: blank lines isolate the HTML block.
|
// Top-level callout: blank lines isolate the HTML block.
|
||||||
@@ -491,8 +497,7 @@ function bridgeTaskLists(html: string): string {
|
|||||||
if (html.length > MAX_CALLOUT_PREPROCESS_BYTES) {
|
if (html.length > MAX_CALLOUT_PREPROCESS_BYTES) {
|
||||||
return html;
|
return html;
|
||||||
}
|
}
|
||||||
const dom = new JSDOM(html);
|
const document = parseHtmlDocument(html);
|
||||||
const document = dom.window.document;
|
|
||||||
// Collect the checkbox(es) that belong to THIS <li> directly: either direct
|
// Collect the checkbox(es) that belong to THIS <li> directly: either direct
|
||||||
// child <input type="checkbox"> elements or ones inside the <li>'s direct <p>
|
// child <input type="checkbox"> elements or ones inside the <li>'s direct <p>
|
||||||
// child (the shape marked emits: `<li><p><input type="checkbox"> text</p></li>`).
|
// child (the shape marked emits: `<li><p><input type="checkbox"> text</p></li>`).
|
||||||
@@ -662,15 +667,23 @@ function placeStandalone(
|
|||||||
function applyCommentDirectives(html: string): string {
|
function applyCommentDirectives(html: string): string {
|
||||||
// Cheap early-out: no comments at all -> nothing to intercept.
|
// Cheap early-out: no comments at all -> nothing to intercept.
|
||||||
if (!html.includes("<!--")) return html;
|
if (!html.includes("<!--")) return html;
|
||||||
const dom = new JSDOM(html);
|
const document = parseHtmlDocument(html);
|
||||||
const document = dom.window.document;
|
// `SHOW_COMMENT` (128) is a stable DOM constant. Read it from whichever holder
|
||||||
const nodeFilter = dom.window.NodeFilter;
|
// exists — the document's window (jsdom: `defaultView`), the ambient global
|
||||||
|
// `NodeFilter` (browsers / test envs), else the literal — because a document
|
||||||
|
// produced by `DOMParser.parseFromString` has NO browsing context, so its
|
||||||
|
// `defaultView` is `null` (unlike a jsdom `new JSDOM(html).window.document`).
|
||||||
|
// `createTreeWalker` takes the numeric `whatToShow` mask directly.
|
||||||
|
const SHOW_COMMENT =
|
||||||
|
(document.defaultView as any)?.NodeFilter?.SHOW_COMMENT ??
|
||||||
|
(globalThis as any).NodeFilter?.SHOW_COMMENT ??
|
||||||
|
0x80;
|
||||||
// Walk the WHOLE document, not just <body>: when a standalone machinery
|
// Walk the WHOLE document, not just <body>: when a standalone machinery
|
||||||
// comment is the FIRST thing in the output (before any body content), the
|
// comment is the FIRST thing in the output (before any body content), the
|
||||||
// HTML parser places it at document level (a child of `#document`, before
|
// HTML parser places it at document level (a child of `#document`, before
|
||||||
// `<html>`), where it is outside `document.body` and would be lost. Attached
|
// `<html>`), where it is outside `document.body` and would be lost. Attached
|
||||||
// attrs comments always live inside body, so this wider walk still finds them.
|
// attrs comments always live inside body, so this wider walk still finds them.
|
||||||
const walker = document.createTreeWalker(document, nodeFilter.SHOW_COMMENT);
|
const walker = document.createTreeWalker(document, SHOW_COMMENT);
|
||||||
const comments: any[] = [];
|
const comments: any[] = [];
|
||||||
let current: any;
|
let current: any;
|
||||||
while ((current = walker.nextNode())) comments.push(current);
|
while ((current = walker.nextNode())) comments.push(current);
|
||||||
@@ -945,8 +958,7 @@ const MAX_FOOTNOTE_ROUNDS = 10000;
|
|||||||
function assembleFootnotes(html: string): string {
|
function assembleFootnotes(html: string): string {
|
||||||
// Cheap early-out: nothing carries a footnote body -> nothing to assemble.
|
// Cheap early-out: nothing carries a footnote body -> nothing to assemble.
|
||||||
if (!html.includes("data-fn-text")) return html;
|
if (!html.includes("data-fn-text")) return html;
|
||||||
const dom = new JSDOM(html);
|
const document = parseHtmlDocument(html);
|
||||||
const document = dom.window.document;
|
|
||||||
if (document.querySelector("sup[data-footnote-ref][data-fn-text]") == null) {
|
if (document.querySelector("sup[data-footnote-ref][data-fn-text]") == null) {
|
||||||
return html;
|
return html;
|
||||||
}
|
}
|
||||||
@@ -1060,12 +1072,18 @@ function stripEmptyParagraphs(node: any): any {
|
|||||||
return { ...node, content: cleaned };
|
return { ...node, content: cleaned };
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Convert markdown to a ProseMirror doc using the full Docmost schema. */
|
/**
|
||||||
export async function markdownToProseMirror(
|
* Convert markdown to a ProseMirror doc using the full Docmost schema
|
||||||
markdownContent: string,
|
* (SYNCHRONOUS core). Every stage — callout preprocess, `marked` parse, the
|
||||||
): Promise<any> {
|
* three DOM passes, and generateJSON — is synchronous for this configuration
|
||||||
const withCallouts = await preprocessCallouts(markdownContent);
|
* (no async marked extensions), so the conversion needs no `await`. The async
|
||||||
const html = await markedInstance.parse(withCallouts);
|
* `markdownToProseMirror` below delegates here (its Promise return is preserved
|
||||||
|
* for every existing Node consumer). A sync entry is REQUIRED by the client's
|
||||||
|
* chat renderer, which runs inside a React render/useMemo and cannot await.
|
||||||
|
*/
|
||||||
|
export function markdownToProseMirrorSync(markdownContent: string): any {
|
||||||
|
const withCallouts = preprocessCallouts(markdownContent);
|
||||||
|
const html = markedInstance.parse(withCallouts) as string;
|
||||||
// Materialize comment directives (#293 #9 attached textAlign; #5 standalone
|
// Materialize comment directives (#293 #9 attached textAlign; #5 standalone
|
||||||
// subpages/pageBreak) while the comment nodes still exist, before generateJSON
|
// subpages/pageBreak) while the comment nodes still exist, before generateJSON
|
||||||
// drops them.
|
// drops them.
|
||||||
@@ -1075,6 +1093,17 @@ export async function markdownToProseMirror(
|
|||||||
// generateJSON, so references + definitions materialize into the schema model.
|
// generateJSON, so references + definitions materialize into the schema model.
|
||||||
const withFootnotes = assembleFootnotes(withAttrs);
|
const withFootnotes = assembleFootnotes(withAttrs);
|
||||||
const bridged = bridgeTaskLists(withFootnotes);
|
const bridged = bridgeTaskLists(withFootnotes);
|
||||||
const doc = generateJSON(bridged, docmostExtensions);
|
const doc = generateJsonWith(bridged, docmostExtensions);
|
||||||
return stripEmptyParagraphs(doc);
|
return stripEmptyParagraphs(doc);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert markdown to a ProseMirror doc (async entry, unchanged contract). Kept
|
||||||
|
* async so every existing Node consumer (server, mcp, git-sync) that `await`s
|
||||||
|
* it is untouched; it simply delegates to the synchronous core.
|
||||||
|
*/
|
||||||
|
export async function markdownToProseMirror(
|
||||||
|
markdownContent: string,
|
||||||
|
): Promise<any> {
|
||||||
|
return markdownToProseMirrorSync(markdownContent);
|
||||||
|
}
|
||||||
|
|||||||
@@ -7,13 +7,12 @@
|
|||||||
* it is never applied to replacement text or inserted node content, so no
|
* it is never applied to replacement text or inserted node content, so no
|
||||||
* formatting is ever lost.
|
* formatting is ever lost.
|
||||||
*
|
*
|
||||||
* Scope note (#414): this package-local copy exists so `node-ops.ts` — which
|
* CANONICAL HOME (#414/#493): this is the single source of truth for locator
|
||||||
* lives here now (the single canonical copy) — can resolve its markdown-tolerant
|
* markdown-stripping. `node-ops.ts` (which lives here) uses it directly, and the
|
||||||
* anchor fallback without a circular dependency back on `@docmost/mcp`. It
|
* mcp-side `text-normalize.ts` now IMPORTS `stripInlineMarkdown` and the shared
|
||||||
* intentionally carries ONLY `stripInlineMarkdown` (the primitive `node-ops`
|
* `stripWrappersAndLinks` primitive from here (via `@docmost/prosemirror-markdown`)
|
||||||
* needs); the mcp-side `text-normalize.ts` (which additionally serves
|
* instead of keeping a drifting copy — mcp only adds its own thin
|
||||||
* `json-edit.ts` via `stripBalancedWrappers`) is the subject of a separate
|
* `stripBalancedWrappers`/`closestBlockHint` on top.
|
||||||
* dedup task and is left untouched here.
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
/** Maximum unwrap passes, so pathological/nested input cannot loop forever. */
|
/** Maximum unwrap passes, so pathological/nested input cannot loop forever. */
|
||||||
@@ -44,7 +43,7 @@ const LINK_IMAGE_RE = /!?\[([^\]]*)\]\([^)]*\)/g;
|
|||||||
* Does NOT trim decoration, does NOT guard against an empty result — it returns
|
* Does NOT trim decoration, does NOT guard against an empty result — it returns
|
||||||
* exactly the transformed string.
|
* exactly the transformed string.
|
||||||
*/
|
*/
|
||||||
function stripWrappersAndLinks(s: string): string {
|
export function stripWrappersAndLinks(s: string): string {
|
||||||
// 1. Links/images -> their visible text.
|
// 1. Links/images -> their visible text.
|
||||||
let out = s.replace(LINK_IMAGE_RE, "$1");
|
let out = s.replace(LINK_IMAGE_RE, "$1");
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,112 @@
|
|||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import {
|
||||||
|
convertProseMirrorToMarkdown,
|
||||||
|
ConverterLossError,
|
||||||
|
} from "../src/lib/markdown-converter.js";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* #493 commit 3 — a node/mark type the serializer has no dedicated case for used
|
||||||
|
* to be degraded SILENTLY (an unknown node flattened to its children, an unknown
|
||||||
|
* mark dropped from the run). The serializer now REPORTS the loss:
|
||||||
|
* - default (non-strict): unchanged graceful degradation, but one warning per
|
||||||
|
* unmapped type is pushed into an optional `warnings` sink so callers can
|
||||||
|
* observe it;
|
||||||
|
* - strict: the FIRST unmapped type throws a ConverterLossError (git-sync +
|
||||||
|
* tests), turning a silent loss into a hard, surfaced error.
|
||||||
|
*
|
||||||
|
* Exercised through the REAL converter (no mock): the observable properties are
|
||||||
|
* the emitted markdown, the warnings collected, and the thrown error.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const doc = (...nodes: any[]) => ({ type: "doc", content: nodes });
|
||||||
|
|
||||||
|
describe("converter loss reporting — unknown node types", () => {
|
||||||
|
const unknownNode = doc({
|
||||||
|
type: "quantumWidget",
|
||||||
|
content: [{ type: "text", text: "inner text" }],
|
||||||
|
});
|
||||||
|
|
||||||
|
it("degrades to children AND records a warning (non-strict, sink provided)", () => {
|
||||||
|
const warnings: string[] = [];
|
||||||
|
const md = convertProseMirrorToMarkdown(unknownNode, { warnings });
|
||||||
|
// Graceful degrade: the child text still survives (historical behavior).
|
||||||
|
expect(md).toContain("inner text");
|
||||||
|
// The loss is now observable.
|
||||||
|
expect(warnings).toHaveLength(1);
|
||||||
|
expect(warnings[0]).toContain("quantumWidget");
|
||||||
|
expect(warnings[0]).toContain("node");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("stays byte-identical for callers that pass no sink (zero behavior change)", () => {
|
||||||
|
const withSink: string[] = [];
|
||||||
|
const a = convertProseMirrorToMarkdown(unknownNode, { warnings: withSink });
|
||||||
|
const b = convertProseMirrorToMarkdown(unknownNode);
|
||||||
|
expect(b).toBe(a); // the sink does not alter the produced markdown
|
||||||
|
});
|
||||||
|
|
||||||
|
it("throws ConverterLossError in strict mode", () => {
|
||||||
|
try {
|
||||||
|
convertProseMirrorToMarkdown(unknownNode, { strict: true });
|
||||||
|
expect.unreachable("strict mode must throw on an unknown node");
|
||||||
|
} catch (e) {
|
||||||
|
expect(e).toBeInstanceOf(ConverterLossError);
|
||||||
|
expect((e as ConverterLossError).kind).toBe("node");
|
||||||
|
expect((e as ConverterLossError).typeName).toBe("quantumWidget");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("dedupes the warning per type (many unknown nodes -> one message)", () => {
|
||||||
|
const warnings: string[] = [];
|
||||||
|
convertProseMirrorToMarkdown(
|
||||||
|
doc(
|
||||||
|
{ type: "quantumWidget", content: [{ type: "text", text: "a" }] },
|
||||||
|
{ type: "quantumWidget", content: [{ type: "text", text: "b" }] },
|
||||||
|
),
|
||||||
|
{ warnings },
|
||||||
|
);
|
||||||
|
expect(warnings).toHaveLength(1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("converter loss reporting — unknown mark types", () => {
|
||||||
|
const unknownMark = doc({
|
||||||
|
type: "paragraph",
|
||||||
|
content: [{ type: "text", text: "glowing", marks: [{ type: "glow" }] }],
|
||||||
|
});
|
||||||
|
|
||||||
|
it("drops the mark but keeps the text AND records a warning (non-strict)", () => {
|
||||||
|
const warnings: string[] = [];
|
||||||
|
const md = convertProseMirrorToMarkdown(unknownMark, { warnings });
|
||||||
|
expect(md).toBe("glowing"); // text survives, mark silently had no form
|
||||||
|
expect(warnings).toHaveLength(1);
|
||||||
|
expect(warnings[0]).toContain("glow");
|
||||||
|
expect(warnings[0]).toContain("mark");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("throws ConverterLossError in strict mode", () => {
|
||||||
|
expect(() =>
|
||||||
|
convertProseMirrorToMarkdown(unknownMark, { strict: true }),
|
||||||
|
).toThrow(ConverterLossError);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("converter loss reporting — known content is never flagged", () => {
|
||||||
|
it("a fully-mapped document produces no warnings and does not throw in strict mode", () => {
|
||||||
|
const d = doc(
|
||||||
|
{ type: "heading", attrs: { level: 2 }, content: [{ type: "text", text: "Title" }] },
|
||||||
|
{
|
||||||
|
type: "paragraph",
|
||||||
|
content: [
|
||||||
|
{ type: "text", text: "bold", marks: [{ type: "bold" }] },
|
||||||
|
{ type: "text", text: " and " },
|
||||||
|
{ type: "text", text: "link", marks: [{ type: "link", attrs: { href: "https://x.y" } }] },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{ type: "bulletList", content: [{ type: "listItem", content: [{ type: "paragraph", content: [{ type: "text", text: "item" }] }] }] },
|
||||||
|
);
|
||||||
|
const warnings: string[] = [];
|
||||||
|
const md = convertProseMirrorToMarkdown(d, { warnings, strict: true });
|
||||||
|
expect(warnings).toEqual([]);
|
||||||
|
expect(md).toContain("## Title");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||||
|
import { JSDOM } from "jsdom";
|
||||||
|
import {
|
||||||
|
setHtmlDocumentParser,
|
||||||
|
parseHtmlDocument,
|
||||||
|
} from "../src/lib/dom-parser.js";
|
||||||
|
import { markdownToProseMirror } from "../src/lib/markdown-to-prosemirror.js";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The markdown import path parses its post-`marked` HTML through an INJECTED
|
||||||
|
* DOM parser: jsdom on the Node entry, native `DOMParser` on the browser entry
|
||||||
|
* (see dom-parser.node.ts / dom-parser.browser.ts). These tests exercise BOTH
|
||||||
|
* registrations against the same canonical inputs — the three DOM passes
|
||||||
|
* (task-list bridge, comment directives, footnote assembly) — and assert the
|
||||||
|
* converter produces the IDENTICAL ProseMirror doc regardless of which DOM
|
||||||
|
* parser is installed. That is the guarantee the client paste path relies on:
|
||||||
|
* pasting in the browser must yield the same nodes the server import produces.
|
||||||
|
*/
|
||||||
|
|
||||||
|
// A jsdom-backed parser (the Node entry's registration).
|
||||||
|
const jsdomParser = (html: string): Document =>
|
||||||
|
new JSDOM(html).window.document as unknown as Document;
|
||||||
|
|
||||||
|
// A `DOMParser`-backed parser standing in for the BROWSER entry's registration.
|
||||||
|
// We drive a real `DOMParser` from a jsdom window (jsdom exposes the same
|
||||||
|
// `new DOMParser().parseFromString(html, "text/html")` API the browser and the
|
||||||
|
// client's jsdom vitest environment provide), so this path uses the exact code
|
||||||
|
// dom-parser.browser.ts runs — a native `DOMParser`, no `JSDOM` document glue.
|
||||||
|
const browserWindow = new JSDOM("").window;
|
||||||
|
const domParserBackedParser = (html: string): Document =>
|
||||||
|
new browserWindow.DOMParser().parseFromString(
|
||||||
|
html,
|
||||||
|
"text/html",
|
||||||
|
) as unknown as Document;
|
||||||
|
|
||||||
|
// Canonical inputs, each hitting a different post-marked DOM pass.
|
||||||
|
const CASES: Record<string, string> = {
|
||||||
|
// footnote assembly (assembleFootnotes): `^[…]` -> sup + section/def
|
||||||
|
"inline footnote ^[…]": "Body^[a note].",
|
||||||
|
// task-list bridge (bridgeTaskLists): checkbox list -> taskList/taskItem
|
||||||
|
"task list": "- [x] done\n- [ ] todo",
|
||||||
|
// comment directives (applyCommentDirectives): standalone machinery comment
|
||||||
|
"standalone subpages comment": "text\n\n<!--subpages-->\n\ntext2",
|
||||||
|
// attached image comment (applyCommentDirectives img form)
|
||||||
|
"attached image comment": ' <!--img {"align":"left"}-->',
|
||||||
|
// github callout (preprocessCallouts bq path) + comment pass
|
||||||
|
"obsidian callout": "> [!info]\n> hello",
|
||||||
|
// highlight + math (marked extensions; still parsed through the DOM stage)
|
||||||
|
"highlight + math": "A ==mark== and $x^2$ end",
|
||||||
|
};
|
||||||
|
|
||||||
|
async function convertWith(
|
||||||
|
parser: (html: string) => Document,
|
||||||
|
md: string,
|
||||||
|
): Promise<any> {
|
||||||
|
setHtmlDocumentParser(parser);
|
||||||
|
return markdownToProseMirror(md);
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("markdown import: Node (jsdom) and browser (DOMParser) DOM paths agree", () => {
|
||||||
|
afterEach(() => {
|
||||||
|
// Restore the jsdom parser the suite-wide setup file installs, so later
|
||||||
|
// tests in the run are unaffected by our per-case swaps.
|
||||||
|
setHtmlDocumentParser(jsdomParser);
|
||||||
|
});
|
||||||
|
|
||||||
|
for (const [name, md] of Object.entries(CASES)) {
|
||||||
|
it(`produces identical nodes for: ${name}`, async () => {
|
||||||
|
const viaJsdom = await convertWith(jsdomParser, md);
|
||||||
|
const viaDomParser = await convertWith(domParserBackedParser, md);
|
||||||
|
expect(viaDomParser).toEqual(viaJsdom);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("dom-parser injection contract", () => {
|
||||||
|
let saved: (html: string) => Document;
|
||||||
|
beforeEach(() => {
|
||||||
|
saved = jsdomParser;
|
||||||
|
});
|
||||||
|
afterEach(() => {
|
||||||
|
setHtmlDocumentParser(saved);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("throws a clear error when no parser is registered", () => {
|
||||||
|
// Install a thrower to simulate the unregistered state, then assert
|
||||||
|
// parseHtmlDocument surfaces the guidance error (we cannot un-set the
|
||||||
|
// module singleton, so we assert via a registration that throws the same).
|
||||||
|
setHtmlDocumentParser(() => {
|
||||||
|
throw new Error("No HTML DOM parser registered.");
|
||||||
|
});
|
||||||
|
expect(() => parseHtmlDocument("<p>x</p>")).toThrow(/No HTML DOM parser/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("uses the most-recently registered parser", () => {
|
||||||
|
const marker = new JSDOM("<!DOCTYPE html><body><b>marker</b></body>").window
|
||||||
|
.document as unknown as Document;
|
||||||
|
setHtmlDocumentParser(() => marker);
|
||||||
|
expect(parseHtmlDocument("<i>ignored</i>")).toBe(marker);
|
||||||
|
});
|
||||||
|
});
|
||||||
+59
-6
@@ -1,12 +1,15 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { convertProseMirrorToMarkdown } from '../src/lib/markdown-converter.js';
|
||||||
|
import { markdownToProseMirror } from '../src/lib/markdown-to-prosemirror.js';
|
||||||
import {
|
import {
|
||||||
convertProseMirrorToMarkdown,
|
normalizeForeignMarkdown,
|
||||||
markdownToProseMirror,
|
normalizeAgentMarkdown,
|
||||||
} from '@docmost/prosemirror-markdown';
|
} from '../src/lib/foreign-markdown.js';
|
||||||
import { normalizeForeignMarkdown } from './foreign-markdown';
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* STEP 2 goldens for issue #345: the foreign-markdown normalizer that runs at the
|
* STEP 2 goldens for issue #345 (moved into the package with the normalizer in
|
||||||
* import boundary BEFORE the strict canonical parser (`markdownToProseMirror`).
|
* #493): the foreign-markdown normalizer that runs at the import boundary BEFORE
|
||||||
|
* the strict canonical parser (`markdownToProseMirror`).
|
||||||
*
|
*
|
||||||
* Two layers:
|
* Two layers:
|
||||||
* 1. PURE string→string cases pinning the normalizer's own behavior (GFM
|
* 1. PURE string→string cases pinning the normalizer's own behavior (GFM
|
||||||
@@ -216,3 +219,53 @@ describe('foreign markdown import acceptance (normalizer + canonical parser)', (
|
|||||||
).toHaveLength(1);
|
).toHaveLength(1);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('normalizeAgentMarkdown vs normalizeForeignMarkdown — front-matter strip is IMPORT-only (#493 review)', () => {
|
||||||
|
// A page that OPENS with a horizontalRule and contains a later `---` serializes
|
||||||
|
// to a `---…---`-shaped body. On a full-body AGENT rewrite this must NOT be
|
||||||
|
// mistaken for YAML front-matter and stripped — that silently dropped the
|
||||||
|
// page's leading content.
|
||||||
|
const rulePage = '---\n\nIntro\n\nMore\n\n---\n\nRest';
|
||||||
|
|
||||||
|
it('normalizeAgentMarkdown does NOT strip a leading ---…--- (no content loss)', () => {
|
||||||
|
expect(normalizeAgentMarkdown(rulePage)).toBe(rulePage);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('normalizeForeignMarkdown (file import) STILL strips a real leading YAML front-matter block', () => {
|
||||||
|
const withYaml = '---\ntitle: My Page\ntags: [a, b]\n---\n\nBody here.';
|
||||||
|
const out = normalizeForeignMarkdown(withYaml);
|
||||||
|
expect(out).toBe('Body here.');
|
||||||
|
// And the horizontalRule-shaped body IS stripped on the import path (its
|
||||||
|
// documented file-import behavior) — the two variants differ ONLY here.
|
||||||
|
expect(normalizeForeignMarkdown(rulePage)).not.toContain('Intro');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('agent-write round-trip keeps a horizontalRule-led doc with a second rule intact', async () => {
|
||||||
|
// Simulate the serializer output for [horizontalRule, para, para, horizontalRule, para].
|
||||||
|
const doc = {
|
||||||
|
type: 'doc',
|
||||||
|
content: [
|
||||||
|
{ type: 'horizontalRule' },
|
||||||
|
{ type: 'paragraph', content: [{ type: 'text', text: 'Intro' }] },
|
||||||
|
{ type: 'paragraph', content: [{ type: 'text', text: 'More' }] },
|
||||||
|
{ type: 'horizontalRule' },
|
||||||
|
{ type: 'paragraph', content: [{ type: 'text', text: 'Rest' }] },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
const body = convertProseMirrorToMarkdown(doc);
|
||||||
|
// The agent-write normalization must NOT eat the head; re-import keeps every
|
||||||
|
// paragraph's text.
|
||||||
|
const back = await markdownToProseMirror(normalizeAgentMarkdown(body));
|
||||||
|
const texts = JSON.stringify(back);
|
||||||
|
for (const t of ['Intro', 'More', 'Rest']) expect(texts).toContain(t);
|
||||||
|
// Both horizontal rules survive.
|
||||||
|
expect(back.content.filter((n: any) => n.type === 'horizontalRule')).toHaveLength(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('agent-write STILL rewrites GFM reference footnotes (the shared drift-fix)', () => {
|
||||||
|
const gfm = 'See[^1].\n\n[^1]: the note.';
|
||||||
|
const out = normalizeAgentMarkdown(gfm);
|
||||||
|
expect(out).toContain('^[the note.]');
|
||||||
|
expect(out).not.toMatch(/\[\^1\]:/);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -11,9 +11,11 @@
|
|||||||
*
|
*
|
||||||
* The corpus deliberately spans the CommonMark / canon hostile alphabet
|
* The corpus deliberately spans the CommonMark / canon hostile alphabet
|
||||||
* (`* _ [ ] ( ) { } | < > & # ! ~ = + -`), unicode / emoji / RTL, and the legal
|
* (`* _ [ ] ( ) { } | < > & # ! ~ = + -`), unicode / emoji / RTL, and the legal
|
||||||
* mark combinations on runs (including the `code` mark, which the schema's
|
* mark combinations on runs. As of #515 the `code` mark no longer excludes other
|
||||||
* `excludes: "_"` makes suppress every co-occurring mark — so it is never
|
* marks (`excludes: ""`), so the corpus ALSO combines `code` with bold / italic /
|
||||||
* combined with another mark in the byte-stable space).
|
* strike / highlight — exercising both the HOMOGENEOUS run factoring (adjacent
|
||||||
|
* code+bold spans -> `` **`a` `b`** ``) and the HETEROGENEOUS anti-collision
|
||||||
|
* fallback (`[code,bold]` next to `[italic]` -> schema-HTML, never `` `a`***b* ``).
|
||||||
*/
|
*/
|
||||||
import fc from 'fast-check';
|
import fc from 'fast-check';
|
||||||
|
|
||||||
@@ -106,16 +108,16 @@ export const urlArb: fc.Arbitrary<string> = fc
|
|||||||
/**
|
/**
|
||||||
* A text run with an OPTIONAL single non-code formatting mark (bold/italic/
|
* A text run with an OPTIONAL single non-code formatting mark (bold/italic/
|
||||||
* strike/underline/superscript/subscript/spoiler), or a SOLE `code` mark, or a
|
* strike/underline/superscript/subscript/spoiler), or a SOLE `code` mark, or a
|
||||||
* link, or an inline comment anchor. `code` is NEVER combined with another mark
|
* `code` mark COMBINED with a bare-delimiter emphasis mark (#515), or a link, or
|
||||||
* in the byte-stable space (that combination is a documented converter
|
* an inline comment anchor. Marks wrap `safeTextArb`, which stays stable even
|
||||||
* limitation — the schema's `code` mark declares `excludes: "_"`). Marks wrap
|
* when it contains isolated specials.
|
||||||
* `safeTextArb`, which stays stable even when it contains isolated specials.
|
|
||||||
*
|
*
|
||||||
* The mark set here is broadened past the sibling test's {bold,italic,strike}
|
* The mark set here is broadened past the sibling test's {bold,italic,strike} to
|
||||||
* to also cover underline / superscript / subscript / spoiler / textStyle /
|
* also cover underline / superscript / subscript / spoiler / textStyle /
|
||||||
* highlight (all single, non-code marks), so the marks-on-text generator
|
* highlight (all single, non-code marks). As of #515 it ALSO emits `code`
|
||||||
* exercises every mark the schema declares except the deliberately-excluded
|
* combined with bold/italic/strike, so the assembled inline content exercises the
|
||||||
* `code`+other combination.
|
* converter's code-emphasis run detection (adjacent combos -> homogeneous
|
||||||
|
* factoring or heterogeneous HTML fallback, both lossless).
|
||||||
*/
|
*/
|
||||||
export const markedTextRunArb: fc.Arbitrary<any> = fc.oneof(
|
export const markedTextRunArb: fc.Arbitrary<any> = fc.oneof(
|
||||||
// Plain text.
|
// Plain text.
|
||||||
@@ -138,6 +140,25 @@ export const markedTextRunArb: fc.Arbitrary<any> = fc.oneof(
|
|||||||
// Sole code mark (backtick span). safeTextArb is backtick-free, so the span
|
// Sole code mark (backtick span). safeTextArb is backtick-free, so the span
|
||||||
// content cannot contain an inner backtick.
|
// content cannot contain an inner backtick.
|
||||||
safeTextArb.map((t) => ({ type: 'text', text: t, marks: [{ type: 'code' }] })),
|
safeTextArb.map((t) => ({ type: 'text', text: t, marks: [{ type: 'code' }] })),
|
||||||
|
// #515: code COMBINED with a bare-delimiter emphasis mark. The converter nests
|
||||||
|
// the backtick span inside the emphasis delimiters (`` **`x`** ``) and, when
|
||||||
|
// such runs sit adjacent, factors a shared mark or falls back to schema-HTML.
|
||||||
|
// Mark order here is `[emphasis, code]` — the order the HTML->PM import yields
|
||||||
|
// for bold/italic/strike specifically (code last). This is NOT universal: the
|
||||||
|
// `==`-highlight case below imports code FIRST — so match each case to its own
|
||||||
|
// imported order for the order-exact P1 round-trip (do not assume a fixed order).
|
||||||
|
fc
|
||||||
|
.tuple(safeTextArb, fc.constantFrom('bold', 'italic', 'strike'))
|
||||||
|
.map(([t, m]) => ({ type: 'text', text: t, marks: [{ type: m }, { type: 'code' }] })),
|
||||||
|
// #515: code combined with an UNCOLORED highlight (also a bare-delimiter mark,
|
||||||
|
// `==…==`), so the highlight+code delimiter interaction is covered too. Import
|
||||||
|
// yields `[code, highlight]` here (the `==` inline extension nests code first),
|
||||||
|
// so the generator matches that order for the order-exact P1 round-trip.
|
||||||
|
safeTextArb.map((t) => ({
|
||||||
|
type: 'text',
|
||||||
|
text: t,
|
||||||
|
marks: [{ type: 'code' }, { type: 'highlight' }],
|
||||||
|
})),
|
||||||
// Link with safe text, a paren/space-free href, optionally a letter-bearing
|
// Link with safe text, a paren/space-free href, optionally a letter-bearing
|
||||||
// title (a purely numeric title is coerced to a number and dropped).
|
// title (a purely numeric title is coerced to a number and dropped).
|
||||||
fc
|
fc
|
||||||
@@ -212,25 +233,93 @@ export function normalizeInline(nodes: any[]): any[] {
|
|||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* #493 commit 1: a plain-text run whose text DELIBERATELY OPENS with a markdown
|
||||||
|
* BLOCK trigger — ATX heading `#`, bullet `-`/`*`/`+`, blockquote `>`, ordered
|
||||||
|
* `N.`/`N)`, or a table `|` — followed by safe text. Pre-#493 the corpus
|
||||||
|
* self-censored these away (safeTextArb's leading-word guarantee); the paragraph
|
||||||
|
* serializer now BLOCK-ESCAPES a leading trigger, so the generative round-trip
|
||||||
|
* itself proves the data-loss class is closed rather than avoiding it.
|
||||||
|
*
|
||||||
|
* DELIBERATELY excludes the code-fence (backtick) trigger — the backtick is a
|
||||||
|
* code-span delimiter that re-pairs globally (see specialCharArb's note), an
|
||||||
|
* instability UNRELATED to block-escape — and the whole-line thematic break
|
||||||
|
* (`---`), which only triggers when the line is ONLY dashes; both are covered by
|
||||||
|
* the deterministic pin (gitmost-transcript-neutralization.test.ts). Each still
|
||||||
|
* ENDS in a word (safeTextArb) so adjacent-run concatenation stays safe.
|
||||||
|
*/
|
||||||
|
export const blockTriggerLeadRunArb: fc.Arbitrary<any> = fc
|
||||||
|
.tuple(
|
||||||
|
fc.constantFrom('# ', '## ', '- ', '* ', '+ ', '> ', '1. ', '1) ', '| '),
|
||||||
|
safeTextArb,
|
||||||
|
)
|
||||||
|
.map(([trigger, rest]) => ({ type: 'text', text: trigger + rest }));
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A hardBreak IMMEDIATELY followed by a block-trigger-leading run — a two-node
|
||||||
|
* segment. Because a hardBreak serializes as ` \n`, the trigger then sits at
|
||||||
|
* the START of a CONTINUATION line, exercising the serializer's PER-LINE block
|
||||||
|
* escape (not just the first line). #493 review: without this the fuzzer never
|
||||||
|
* placed a trigger after a hardBreak, so a single-line-only escape passed P1–P3.
|
||||||
|
*/
|
||||||
|
export const hardBreakThenTriggerArb: fc.Arbitrary<any[]> = fc
|
||||||
|
.tuple(hardBreakArb, blockTriggerLeadRunArb)
|
||||||
|
.map(([hb, trigger]) => [hb, trigger]);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* #493 (setext data-loss): a WHOLE-LINE setext underline landing on a
|
||||||
|
* continuation line. A setext underline is a line of ONLY `-` (any count) or
|
||||||
|
* ONLY `=` (any count) that FOLLOWS a paragraph line; on re-parse it turns the
|
||||||
|
* preceding line into a heading and DROPS its own text. The block-escape must
|
||||||
|
* neutralize it. Unlike blockTriggerLeadRunArb, the underline must occupy the
|
||||||
|
* whole line, so we sandwich it between two hardBreaks (underline on its own
|
||||||
|
* line, preceded by earlier paragraph content, followed by a trailing word so
|
||||||
|
* the closing hardBreak is not dropped by normalizeInline). Covers underlines
|
||||||
|
* of every length: `--` (the two-dash case the bullet/thematic arms miss), a
|
||||||
|
* lone `=`, `==`/`====` (neutralized by the inline `==` escape), and `---`/
|
||||||
|
* `----` (regression for the existing thematic case).
|
||||||
|
*/
|
||||||
|
export const hardBreakThenSetextArb: fc.Arbitrary<any[]> = fc
|
||||||
|
.tuple(
|
||||||
|
fc.constantFrom('--', '=', '==', '====', '---', '----'),
|
||||||
|
safeTextArb,
|
||||||
|
)
|
||||||
|
.map(([underline, rest]) => [
|
||||||
|
{ type: 'hardBreak' },
|
||||||
|
{ type: 'text', text: underline },
|
||||||
|
{ type: 'hardBreak' },
|
||||||
|
{ type: 'text', text: rest },
|
||||||
|
]);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Inline content for a paragraph: at least one marked text run, optionally with
|
* Inline content for a paragraph: at least one marked text run, optionally with
|
||||||
* inline atoms (math/mention) and hard breaks interspersed. Always starts with a
|
* inline atoms (math/mention) and hard breaks interspersed. The FIRST run is
|
||||||
* text run so the paragraph never opens with a block trigger. (Ported.)
|
* usually an ordinary marked run, but sometimes a block-trigger-leading run
|
||||||
|
* (blockTriggerLeadRunArb) so the paragraph OPENS with a markdown block trigger;
|
||||||
|
* and a `hardBreak + trigger` segment can appear anywhere in the rest, so a
|
||||||
|
* trigger also lands at the start of a CONTINUATION line — both exercising the
|
||||||
|
* serializer's per-line block-escape end-to-end. (Ported, with the #493
|
||||||
|
* leading-trigger + post-hardBreak dimensions added.)
|
||||||
*/
|
*/
|
||||||
export const inlineContentArb: fc.Arbitrary<any[]> = fc
|
export const inlineContentArb: fc.Arbitrary<any[]> = fc
|
||||||
.tuple(
|
.tuple(
|
||||||
markedTextRunArb,
|
fc.oneof(
|
||||||
|
{ weight: 5, arbitrary: markedTextRunArb },
|
||||||
|
{ weight: 1, arbitrary: blockTriggerLeadRunArb },
|
||||||
|
),
|
||||||
fc.array(
|
fc.array(
|
||||||
fc.oneof(
|
fc.oneof(
|
||||||
{ weight: 5, arbitrary: markedTextRunArb },
|
{ weight: 5, arbitrary: markedTextRunArb.map((n) => [n]) },
|
||||||
{ weight: 1, arbitrary: mathInlineArb },
|
{ weight: 1, arbitrary: mathInlineArb.map((n) => [n]) },
|
||||||
{ weight: 1, arbitrary: mentionArb },
|
{ weight: 1, arbitrary: mentionArb.map((n) => [n]) },
|
||||||
{ weight: 1, arbitrary: hardBreakArb },
|
{ weight: 1, arbitrary: hardBreakArb.map((n) => [n]) },
|
||||||
|
{ weight: 2, arbitrary: hardBreakThenTriggerArb },
|
||||||
|
{ weight: 2, arbitrary: hardBreakThenSetextArb },
|
||||||
),
|
),
|
||||||
{ minLength: 0, maxLength: 4 },
|
{ minLength: 0, maxLength: 4 },
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
.map(([first, rest]) => normalizeInline([first, ...rest]));
|
.map(([first, rest]) => normalizeInline([first, ...rest.flat()]));
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Inline content for a HEADING — identical to a paragraph's, but WITHOUT hard
|
* Inline content for a HEADING — identical to a paragraph's, but WITHOUT hard
|
||||||
|
|||||||
@@ -5,32 +5,21 @@ import { convertProseMirrorToMarkdown } from "../src/lib/markdown-converter.js";
|
|||||||
import { markdownToProseMirror } from "../src/lib/markdown-to-prosemirror.js";
|
import { markdownToProseMirror } from "../src/lib/markdown-to-prosemirror.js";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* gitmost #377 (round-1 review, finding #1) — proof, against the REAL
|
* #493 commit 1 — the paragraph serializer's leading-block-escape closes the
|
||||||
* converter, that the transcript-insert boundary defense survives git-sync.
|
* data-loss class where a paragraph whose text opens at column 0 with a markdown
|
||||||
|
* block trigger (`#`/`-`/`*`/`+`/`>`, an ordered `N.`/`N)`, a code fence, a
|
||||||
|
* table `|`, a callout opener, or a thematic break) silently re-parsed into a
|
||||||
|
* heading / list / quote / code block / table / horizontalRule on the git-sync
|
||||||
|
* doc -> markdown -> doc cycle. The thematic-break case was the worst: a
|
||||||
|
* horizontalRule carries NO text, so the line's text was lost entirely.
|
||||||
*
|
*
|
||||||
* The web bridge (apps/client .../gitmost/gitmost-recording.ts,
|
* This is the deterministic PIN, one assertion per trigger, exercised through
|
||||||
* `gitmostInsertTranscriptIntoEditor`) appends each transcript line as a
|
* the REAL converter round-trip (not a mock): each bare trigger line now
|
||||||
* PARAGRAPH text node. The paragraph serializer here (`case "paragraph"`) emits
|
* round-trips as a SINGLE paragraph with its text byte-preserved — proving the
|
||||||
* that text VERBATIM with no block-escape, so a line whose text begins with a
|
* class is closed WITHOUT the former client-side ZWSP workaround (removed) or
|
||||||
* col-0 markdown block trigger would, on the doc -> markdown -> doc git-sync
|
* the generative suite's leading-word self-censorship (removed).
|
||||||
* cycle, silently re-parse into a heading / list / quote / callout / code block.
|
|
||||||
* That missing block-escape is the pre-existing root cause; the bridge's
|
|
||||||
* boundary defense prepends an invisible zero-width space (U+200B) to a line
|
|
||||||
* that begins with such a trigger, shifting it off column 0.
|
|
||||||
*
|
|
||||||
* This test keeps a COPY of the bridge's trigger regex (the bridge is in a
|
|
||||||
* different package and can't be imported here) and asserts:
|
|
||||||
* 1. bare trigger lines DO corrupt (documents the root cause), and
|
|
||||||
* 2. the ZWSP-neutralized form round-trips as a single PARAGRAPH with the
|
|
||||||
* text byte-preserved.
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
const ZWSP = ""; // U+200B
|
|
||||||
|
|
||||||
// MUST stay in sync with GITMOST_MD_BLOCK_TRIGGER_RE in the client bridge.
|
|
||||||
const MD_BLOCK_TRIGGER_RE =
|
|
||||||
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
|
|
||||||
|
|
||||||
const doc = (...nodes: any[]) => ({ type: "doc", content: nodes });
|
const doc = (...nodes: any[]) => ({ type: "doc", content: nodes });
|
||||||
const para = (t: string) => ({
|
const para = (t: string) => ({
|
||||||
type: "paragraph",
|
type: "paragraph",
|
||||||
@@ -43,78 +32,117 @@ const roundtrip = async (text: string) => {
|
|||||||
return back.content as any[];
|
return back.content as any[];
|
||||||
};
|
};
|
||||||
|
|
||||||
describe("gitmost transcript neutralization (git-sync round-trip)", () => {
|
describe("paragraph block-escape (git-sync round-trip)", () => {
|
||||||
// Lines that, at column 0, the serializer's missing block-escape would let
|
// Every line here, at column 0, WOULD (pre-fix) re-parse into a non-paragraph
|
||||||
// git-sync re-parse into a non-paragraph block.
|
// block. Each is now block-escaped by the serializer and round-trips clean.
|
||||||
const triggerLines = [
|
const triggerLines = [
|
||||||
"- dash",
|
"- dash",
|
||||||
"* star",
|
"* star",
|
||||||
"+ plus",
|
"+ plus",
|
||||||
"> quote",
|
"> quote",
|
||||||
"# hash",
|
"# hash",
|
||||||
|
"## two hash",
|
||||||
|
"###### six hash",
|
||||||
"1. one",
|
"1. one",
|
||||||
"1) one",
|
"1) one",
|
||||||
"> [!info] note",
|
"> [!info] note",
|
||||||
"```js",
|
"```js",
|
||||||
"~~~",
|
"~~~",
|
||||||
// Solid + spaced thematic breaks — these re-parse into a `horizontalRule`,
|
"| a | b |",
|
||||||
// which carries NO text, so a bare separator line LOSES its text entirely
|
// Solid + spaced thematic breaks — the text-LOSING case pre-fix.
|
||||||
// (round-2 finding). `_` also only forms a block via this construct.
|
|
||||||
"---",
|
"---",
|
||||||
"***",
|
"***",
|
||||||
"___",
|
"___",
|
||||||
"- - -", // spaced dash break (solid form is caught by [-*+]\s too, but this is the break)
|
"- - -",
|
||||||
"_ _ _",
|
"_ _ _",
|
||||||
];
|
];
|
||||||
|
|
||||||
it("BARE trigger lines corrupt into non-paragraph blocks (root cause)", async () => {
|
it("every bare trigger line round-trips as a single paragraph, text byte-preserved", async () => {
|
||||||
for (const line of triggerLines) {
|
for (const line of triggerLines) {
|
||||||
const blocks = await roundtrip(line);
|
const blocks = await roundtrip(line);
|
||||||
// At least one produced block is NOT a paragraph — i.e. corruption.
|
expect(blocks, `"${line}" should be one block`).toHaveLength(1);
|
||||||
const allParagraphs = blocks.every((b) => b.type === "paragraph");
|
expect(blocks[0].type, `"${line}" should stay a paragraph`).toBe(
|
||||||
expect(
|
"paragraph",
|
||||||
allParagraphs,
|
|
||||||
`expected "${line}" to corrupt when inserted bare`,
|
|
||||||
).toBe(false);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
it("BARE solid thematic breaks corrupt into a text-LOSING horizontalRule", async () => {
|
|
||||||
// The severe case: no text node survives. Documents why neutralization
|
|
||||||
// matters more here than for list/quote (where the text survived).
|
|
||||||
for (const line of ["---", "***", "___"]) {
|
|
||||||
const blocks = await roundtrip(line);
|
|
||||||
expect(blocks.map((b) => b.type)).toContain("horizontalRule");
|
|
||||||
// No block carries the original text anywhere.
|
|
||||||
const flat = JSON.stringify(blocks);
|
|
||||||
expect(flat).not.toContain(line);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
it("ZWSP-neutralized trigger lines round-trip as a single paragraph, text preserved", async () => {
|
|
||||||
for (const line of triggerLines) {
|
|
||||||
// The regex must actually classify each as a trigger.
|
|
||||||
expect(MD_BLOCK_TRIGGER_RE.test(line), `regex missed "${line}"`).toBe(
|
|
||||||
true,
|
|
||||||
);
|
);
|
||||||
const neutralized = ZWSP + line;
|
expect(
|
||||||
const blocks = await roundtrip(neutralized);
|
blocks[0].content?.[0]?.text,
|
||||||
|
`"${line}" text should survive byte-exact`,
|
||||||
expect(blocks).toHaveLength(1);
|
).toBe(line);
|
||||||
expect(blocks[0].type).toBe("paragraph");
|
|
||||||
// Text is byte-preserved (ZWSP + original line), so the display is the
|
|
||||||
// original line with only an invisible leading character.
|
|
||||||
expect(blocks[0].content[0].text).toBe(neutralized);
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
it("normal host-prefixed lines never match the trigger regex and round-trip byte-exact", async () => {
|
it("emphasis / inline-code paragraphs are NOT escaped (no backslash churn)", async () => {
|
||||||
|
// These open with `*`/`` ` `` but are NOT block triggers; the serialized
|
||||||
|
// markdown must not gain a stray leading backslash, and they round-trip.
|
||||||
|
for (const [text, mark] of [
|
||||||
|
["bold", "bold"],
|
||||||
|
["italic", "italic"],
|
||||||
|
["code", "code"],
|
||||||
|
] as const) {
|
||||||
|
const node = doc({
|
||||||
|
type: "paragraph",
|
||||||
|
content: [{ type: "text", text, marks: [{ type: mark }] }],
|
||||||
|
});
|
||||||
|
const md = convertProseMirrorToMarkdown(node);
|
||||||
|
expect(md.startsWith("\\"), `${mark} must not be block-escaped`).toBe(
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
const back = await markdownToProseMirror(md);
|
||||||
|
expect(back.content[0].type).toBe("paragraph");
|
||||||
|
expect(back.content[0].content[0].text).toBe(text);
|
||||||
|
expect(back.content[0].content[0].marks?.[0]?.type).toBe(mark);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("a block trigger on a CONTINUATION line (after a hardBreak) is escaped too", async () => {
|
||||||
|
// A hardBreak serializes as ` \n`, so a trigger on the second line would,
|
||||||
|
// without a per-line escape, re-parse into another block. The worst case is
|
||||||
|
// `---`: a setext underline would turn the first line into a heading and LOSE
|
||||||
|
// the `---` text entirely. Each pair round-trips as ONE paragraph with the
|
||||||
|
// hardBreak and both texts preserved.
|
||||||
|
for (const [first, second] of [
|
||||||
|
["a", "# b"],
|
||||||
|
["a", "- b"],
|
||||||
|
["a", "> b"],
|
||||||
|
["a", "1. b"],
|
||||||
|
["a", "| b |"],
|
||||||
|
["a", "---"], // setext / thematic (3 dashes) — the text-losing case
|
||||||
|
["a", "--"], // setext underline, EXACTLY two dashes (bullet/thematic miss it)
|
||||||
|
["a", "----"], // setext / thematic (4 dashes)
|
||||||
|
["a", "="], // setext H1 underline, a lone `=` (no other arm covers it)
|
||||||
|
["a", "===="], // setext H1 underline, run of `=`
|
||||||
|
]) {
|
||||||
|
const d = doc({
|
||||||
|
type: "paragraph",
|
||||||
|
content: [
|
||||||
|
{ type: "text", text: first },
|
||||||
|
{ type: "hardBreak" },
|
||||||
|
{ type: "text", text: second },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
const back = await markdownToProseMirror(convertProseMirrorToMarkdown(d));
|
||||||
|
expect(back.content, `"${first}⏎${second}" should be one block`).toHaveLength(1);
|
||||||
|
expect(back.content[0].type).toBe("paragraph");
|
||||||
|
const texts = (back.content[0].content as any[])
|
||||||
|
.filter((n) => n.type === "text")
|
||||||
|
.map((n) => n.text);
|
||||||
|
const hasBreak = (back.content[0].content as any[]).some(
|
||||||
|
(n) => n.type === "hardBreak",
|
||||||
|
);
|
||||||
|
expect(hasBreak, `"${first}⏎${second}" should keep the hardBreak`).toBe(true);
|
||||||
|
expect(texts, `"${first}⏎${second}" should preserve both line texts`).toEqual([
|
||||||
|
first,
|
||||||
|
second,
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("normal host-prefixed lines round-trip byte-exact (unaffected)", async () => {
|
||||||
for (const line of [
|
for (const line of [
|
||||||
"You: hello there",
|
"You: hello there",
|
||||||
"Speaker 1: - and then a dash mid-line",
|
"Speaker 1: - and then a dash mid-line",
|
||||||
"Speaker 2: 1. not a list",
|
"Speaker 2: 1. not a list",
|
||||||
]) {
|
]) {
|
||||||
expect(MD_BLOCK_TRIGGER_RE.test(line)).toBe(false);
|
|
||||||
const blocks = await roundtrip(line);
|
const blocks = await roundtrip(line);
|
||||||
expect(blocks).toHaveLength(1);
|
expect(blocks).toHaveLength(1);
|
||||||
expect(blocks[0].type).toBe("paragraph");
|
expect(blocks[0].type).toBe("paragraph");
|
||||||
|
|||||||
@@ -294,10 +294,11 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
// 5. code + link co-occur: the schema's `code` mark excludes all other marks
|
// 5. code + link co-occur (#515): `code` no longer excludes other marks, so a
|
||||||
// (including link), so the link cannot survive import. The lossless,
|
// link can wrap inline code. The code span is emitted innermost and the link
|
||||||
// byte-stable behavior is to emit ONLY the backtick code span (code wins).
|
// wraps it — CommonMark allows inline code inside link text, so it survives
|
||||||
it('a code+link run emits the backtick code form (code wins, link dropped)', () => {
|
// the round trip.
|
||||||
|
it('a code+link run nests the backtick span inside the link (#515)', () => {
|
||||||
const out = convertProseMirrorToMarkdown(
|
const out = convertProseMirrorToMarkdown(
|
||||||
doc(
|
doc(
|
||||||
para({
|
para({
|
||||||
@@ -310,7 +311,7 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
|
|||||||
}),
|
}),
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
expect(out).toBe('`x`');
|
expect(out).toBe('[`x`](http://a?b&c"d)');
|
||||||
});
|
});
|
||||||
|
|
||||||
// 6. hardBreak inside a heading: prefix applied once, " \n" between a and b.
|
// 6. hardBreak inside a heading: prefix applied once, " \n" between a and b.
|
||||||
@@ -430,7 +431,7 @@ describe('converter gap coverage — emission branches (specs 1–11)', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('converter gap coverage — documented round-trip data loss (specs 12–14)', () => {
|
describe('converter gap coverage — formerly-lossy round-trips, now closed (specs 12–14)', () => {
|
||||||
// 12. A 3-backtick fence inside a codeBlock body is now lengthened: the outer
|
// 12. A 3-backtick fence inside a codeBlock body is now lengthened: the outer
|
||||||
// fence widens to (longest inner run + 1) backticks per CommonMark, so the
|
// fence widens to (longest inner run + 1) backticks per CommonMark, so the
|
||||||
// inner ``` is treated as content and the block survives as ONE node.
|
// inner ``` is treated as content and the block survives as ONE node.
|
||||||
@@ -460,25 +461,24 @@ describe('converter gap coverage — documented round-trip data loss (specs 12
|
|||||||
expect(docsCanonicallyEqual(d, doc2)).toBe(false);
|
expect(docsCanonicallyEqual(d, doc2)).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
// 13. A leading ordered-list marker in paragraph text is NOT escaped, so a
|
// 13. #493 commit 1: a leading ordered-list marker in paragraph text is now
|
||||||
// plain paragraph silently becomes an orderedList on re-import.
|
// BLOCK-ESCAPED, so the paragraph round-trips as a paragraph instead of
|
||||||
it('a paragraph starting with "1. " is promoted to an orderedList on re-import', async () => {
|
// silently becoming an orderedList (was documented data loss, now closed).
|
||||||
|
it('a paragraph starting with "1. " is block-escaped and stays a paragraph', async () => {
|
||||||
const d = doc({
|
const d = doc({
|
||||||
type: 'paragraph',
|
type: 'paragraph',
|
||||||
content: [{ type: 'text', text: '1. not a list' }],
|
content: [{ type: 'text', text: '1. not a list' }],
|
||||||
});
|
});
|
||||||
const md1 = convertProseMirrorToMarkdown(d);
|
const md1 = convertProseMirrorToMarkdown(d);
|
||||||
expect(md1).toBe('1. not a list'); // no backslash escape
|
expect(md1).toBe('1\\. not a list'); // the ordered-list delimiter is escaped
|
||||||
|
|
||||||
const doc2 = await markdownToProseMirror(md1);
|
const doc2 = await markdownToProseMirror(md1);
|
||||||
expect(doc2.content?.[0]?.type).toBe('orderedList');
|
expect(doc2.content?.[0]?.type).toBe('paragraph');
|
||||||
const li = doc2.content[0].content?.[0];
|
expect(doc2.content[0].content?.[0]).toMatchObject({
|
||||||
expect(li?.type).toBe('listItem');
|
|
||||||
expect(li.content?.[0]?.content?.[0]).toMatchObject({
|
|
||||||
type: 'text',
|
type: 'text',
|
||||||
text: 'not a list', // the "1. " was consumed as a list marker
|
text: '1. not a list', // the escape decodes back to the literal text
|
||||||
});
|
});
|
||||||
expect(docsCanonicallyEqual(d, doc2)).toBe(false);
|
expect(docsCanonicallyEqual(d, doc2)).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
// 14. #293 canon #4: the image title now round-trips via the attached
|
// 14. #293 canon #4: the image title now round-trips via the attached
|
||||||
|
|||||||
@@ -59,22 +59,21 @@ describe('convertProseMirrorToMarkdown', () => {
|
|||||||
).toBe('`x`');
|
).toBe('`x`');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('code + another mark emits the backtick code form (code wins)', () => {
|
it('code + bold nests the backtick span inside the emphasis (#515)', () => {
|
||||||
// The schema's `code` mark excludes all other marks, so the editor can
|
// #515: the `code` mark no longer excludes other marks (`excludes: ""`), so
|
||||||
// never produce code+bold on one run and import always drops the co-mark.
|
// a run can carry code+bold. CommonMark nests them (`<strong><code>`), so
|
||||||
// The lossless, byte-stable behavior is to emit ONLY the backtick code
|
// the code span is emitted innermost and the bold delimiters wrap it.
|
||||||
// span and ignore the co-occurring mark.
|
|
||||||
const out = convertProseMirrorToMarkdown(
|
const out = convertProseMirrorToMarkdown(
|
||||||
doc(para(text('x', [{ type: 'bold' }, { type: 'code' }]))),
|
doc(para(text('x', [{ type: 'bold' }, { type: 'code' }]))),
|
||||||
);
|
);
|
||||||
expect(out).toBe('`x`');
|
expect(out).toBe('**`x`**');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('code + strike combo emits the backtick code form (code wins)', () => {
|
it('code + strike nests the backtick span inside the emphasis (#515)', () => {
|
||||||
const out = convertProseMirrorToMarkdown(
|
const out = convertProseMirrorToMarkdown(
|
||||||
doc(para(text('x', [{ type: 'strike' }, { type: 'code' }]))),
|
doc(para(text('x', [{ type: 'strike' }, { type: 'code' }]))),
|
||||||
);
|
);
|
||||||
expect(out).toBe('`x`');
|
expect(out).toBe('~~`x`~~');
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -80,13 +80,7 @@ import { stripBlockIds } from './roundtrip-helpers.js';
|
|||||||
// `it.fails` blocks below (so the suite stays green only because they are marked
|
// `it.fails` blocks below (so the suite stays green only because they are marked
|
||||||
// expected-to-fail, never by hiding them):
|
// expected-to-fail, never by hiding them):
|
||||||
//
|
//
|
||||||
// 1. The `code` mark COMBINED with any other mark. The converter emits nested
|
// 1. A BLOCK-level `image` placed BETWEEN other blocks. The Docmost image node
|
||||||
// HTML (`<strong><code>x</code></strong>`), but the schema's `code` mark
|
|
||||||
// declares `excludes: "_"`, so on import every co-occurring mark is dropped
|
|
||||||
// and the run comes back as `code` only -> md2 == "`x`". Acknowledged in
|
|
||||||
// markdown-converter.ts (the long comment above the marks switch);
|
|
||||||
// impossible to round-trip both while `code` excludes them.
|
|
||||||
// 2. A BLOCK-level `image` placed BETWEEN other blocks. The Docmost image node
|
|
||||||
// is block-level but `` is inline; marked wraps it in a <p>, the
|
// is block-level but `` is inline; marked wraps it in a <p>, the
|
||||||
// schema hoists the <img> out and leaves an empty paragraph sibling, which
|
// schema hoists the <img> out and leaves an empty paragraph sibling, which
|
||||||
// injects an extra blank gap on the second export. An image IS byte-stable
|
// injects an extra blank gap on the second export. An image IS byte-stable
|
||||||
@@ -625,7 +619,7 @@ describe('markdown <-> ProseMirror round-trip (property-based)', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
// KNOWN, DOCUMENTED non-roundtrip bug #2 (kept honest as it.fails).
|
// KNOWN, DOCUMENTED non-roundtrip bug #1 (kept honest as it.fails).
|
||||||
//
|
//
|
||||||
// BUG: a block-level `image` placed BETWEEN other blocks is not byte-stable.
|
// BUG: a block-level `image` placed BETWEEN other blocks is not byte-stable.
|
||||||
// The Docmost image node is BLOCK-level but its markdown form `` is
|
// The Docmost image node is BLOCK-level but its markdown form `` is
|
||||||
@@ -655,23 +649,18 @@ describe('markdown <-> ProseMirror round-trip (property-based)', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
// KNOWN, DOCUMENTED non-roundtrip bug #1 (kept honest as it.fails).
|
// #515 ROUND-TRIP PIN: `code` combined with another mark.
|
||||||
//
|
//
|
||||||
// BUG: the `code` mark combined with ANY other mark does NOT round-trip.
|
// Before #515 the `code` mark declared `excludes: "_"`, dropping every co-
|
||||||
// The converter emits nested HTML so the output is well-formed, e.g.
|
// occurring mark on import so `` **`x`** `` came back as code-only. Now
|
||||||
// marks [code, bold] -> md1 = "<strong><code>x</code></strong>"
|
// `excludes: ""` lets code combine with all marks (CommonMark nests them,
|
||||||
// but the schema's `code` mark declares `excludes: "_"`, so on import the
|
// `<strong><code>x</code></strong>`), so the run BOTH round-trips byte-stably
|
||||||
// co-occurring mark is dropped and the run comes back as code-only:
|
// AND preserves the co-occurring mark. This asserts the observable property in
|
||||||
// md2 = "`x`" (=> md2 !== md1).
|
// both directions: md2 === md1 (idempotent export) and the imported doc still
|
||||||
// Minimal repro doc:
|
// carries [code, other].
|
||||||
// { type:'doc', content:[ { type:'paragraph', content:[
|
|
||||||
// { type:'text', text:'x', marks:[{type:'code'},{type:'bold'}] } ] } ] }
|
|
||||||
// This is acknowledged in markdown-converter.ts (the long comment above the
|
|
||||||
// marks switch): preserving both marks is impossible while `code` excludes
|
|
||||||
// them. Documented here, not "fixed", because the source must not change.
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
it(
|
it(
|
||||||
'code mark combined with another mark is byte-stable',
|
'code combined with another mark round-trips and keeps both marks (#515)',
|
||||||
async () => {
|
async () => {
|
||||||
const codeComboArb = fc
|
const codeComboArb = fc
|
||||||
.tuple(safeTextArb, fc.constantFrom('bold', 'italic', 'strike'))
|
.tuple(safeTextArb, fc.constantFrom('bold', 'italic', 'strike'))
|
||||||
@@ -688,11 +677,90 @@ describe('markdown <-> ProseMirror round-trip (property-based)', () => {
|
|||||||
}));
|
}));
|
||||||
await fc.assert(
|
await fc.assert(
|
||||||
fc.asyncProperty(codeComboArb, async (doc) => {
|
fc.asyncProperty(codeComboArb, async (doc) => {
|
||||||
const { md1, md2 } = await roundTrip(doc);
|
const { md1, md2, doc2 } = await roundTrip(doc);
|
||||||
expect(md2).toBe(md1);
|
expect(md2).toBe(md1);
|
||||||
|
// The re-imported run carries BOTH code and the co-occurring mark.
|
||||||
|
const run = doc2?.content?.[0]?.content?.[0];
|
||||||
|
const markTypes = (run?.marks || []).map((m: any) => m.type).sort();
|
||||||
|
expect(markTypes).toContain('code');
|
||||||
|
expect(markTypes.length).toBe(2);
|
||||||
}),
|
}),
|
||||||
{ numRuns: 20, seed: SEED },
|
{ numRuns: 20, seed: SEED },
|
||||||
);
|
);
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// #515 REPRO CASES: the five markdown inputs from the issue must import to a
|
||||||
|
// code+bold node (import correctness) AND re-export byte-stably with no
|
||||||
|
// dangling `**` (export correctness). Import direction is checked against the
|
||||||
|
// real markdown->PM bridge; export direction via the md->pm->md fixpoint.
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
it('the five #515 repro cases import to [code,bold] and round-trip clean', async () => {
|
||||||
|
// Collect every inline text run in a doc with its mark type set.
|
||||||
|
const runs = (node: any): { text: string; marks: string[] }[] => {
|
||||||
|
if (node?.type === 'text') {
|
||||||
|
return [{ text: node.text || '', marks: (node.marks || []).map((m: any) => m.type) }];
|
||||||
|
}
|
||||||
|
return (node?.content || []).flatMap(runs);
|
||||||
|
};
|
||||||
|
const findRun = (doc: any, text: string) =>
|
||||||
|
runs(doc).find((r) => r.text === text);
|
||||||
|
|
||||||
|
// Case 1: **`code1`** -> code1 = [code, bold].
|
||||||
|
{
|
||||||
|
const md = '**`code1`**';
|
||||||
|
const pm = await markdownToProseMirror(md);
|
||||||
|
const r = findRun(pm, 'code1');
|
||||||
|
expect(r?.marks.sort()).toEqual(['bold', 'code']);
|
||||||
|
const md2 = convertProseMirrorToMarkdown(pm);
|
||||||
|
expect(md2).toBe('**`code1`**');
|
||||||
|
// md -> pm -> md fixpoint.
|
||||||
|
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Case 2: **`aaa` + `bbb`** -> aaa,bbb = [code,bold], "+" carries bold; no
|
||||||
|
// dangling `**` on export.
|
||||||
|
{
|
||||||
|
const md = '**`aaa` + `bbb`**';
|
||||||
|
const pm = await markdownToProseMirror(md);
|
||||||
|
expect(findRun(pm, 'aaa')?.marks.sort()).toEqual(['bold', 'code']);
|
||||||
|
expect(findRun(pm, 'bbb')?.marks.sort()).toEqual(['bold', 'code']);
|
||||||
|
const md2 = convertProseMirrorToMarkdown(pm);
|
||||||
|
expect(md2).toBe('**`aaa` + `bbb`**');
|
||||||
|
// NOT the old broken export with the bold delimiters split onto each span.
|
||||||
|
expect(md2).not.toBe('`aaa`** + **`bbb`');
|
||||||
|
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Case 3 (control): **bold3** and `code3` -> bold and code stay SEPARATE.
|
||||||
|
{
|
||||||
|
const md = '**bold3** and `code3`';
|
||||||
|
const pm = await markdownToProseMirror(md);
|
||||||
|
expect(findRun(pm, 'bold3')?.marks).toEqual(['bold']);
|
||||||
|
expect(findRun(pm, 'code3')?.marks).toEqual(['code']);
|
||||||
|
const md2 = convertProseMirrorToMarkdown(pm);
|
||||||
|
expect(md2).toBe('**bold3** and `code3`');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Case 4: **`code4` tail** -> code4 = [code,bold], " tail" = [bold].
|
||||||
|
{
|
||||||
|
const md = '**`code4` tail**';
|
||||||
|
const pm = await markdownToProseMirror(md);
|
||||||
|
expect(findRun(pm, 'code4')?.marks.sort()).toEqual(['bold', 'code']);
|
||||||
|
const md2 = convertProseMirrorToMarkdown(pm);
|
||||||
|
expect(md2).toBe('**`code4` tail**');
|
||||||
|
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Case 5: pre **`code5`** post -> code5 = [code,bold], surroundings plain.
|
||||||
|
{
|
||||||
|
const md = 'pre **`code5`** post';
|
||||||
|
const pm = await markdownToProseMirror(md);
|
||||||
|
expect(findRun(pm, 'code5')?.marks.sort()).toEqual(['bold', 'code']);
|
||||||
|
const md2 = convertProseMirrorToMarkdown(pm);
|
||||||
|
expect(md2).toBe('pre **`code5`** post');
|
||||||
|
expect(convertProseMirrorToMarkdown(await markdownToProseMirror(md2))).toBe(md2);
|
||||||
|
}
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -16,14 +16,16 @@ import * as editorExt from "@docmost/editor-ext";
|
|||||||
// or mark added upstream that the mirror forgets to vendor fails CI loudly
|
// or mark added upstream that the mirror forgets to vendor fails CI loudly
|
||||||
// (otherwise it is silently dropped on the markdown <-> ProseMirror round-trip).
|
// (otherwise it is silently dropped on the markdown <-> ProseMirror round-trip).
|
||||||
//
|
//
|
||||||
// LIMITATION (intentional, see schema-surface-snapshot.test.ts): this is a
|
// This file now holds TWO contracts (see the two describe blocks): the original
|
||||||
// NAME-LEVEL contract only, not a full attribute-level structural compare.
|
// NAME-LEVEL type contract (no canonical node/mark TYPE goes unmirrored) AND, as
|
||||||
// editor-ext's Tiptap representation (node views, commands, suggestion plugins,
|
// of #493, an ATTRIBUTE-LEVEL contract that compares each editor-ext node/mark's
|
||||||
// addGlobalAttributes spread across separate extensions) differs from this
|
// OWN declared attributes (names + defaults) against the mirror's built schema.
|
||||||
// minimal mirror, so a mechanical attribute-by-attribute equality would be
|
// A full mechanical attribute-by-attribute EQUALITY would be fragile (the mirror
|
||||||
// fragile and produce false drift. Attribute parity is guarded by the inline
|
// is a deliberate superset: it injects the global id/textAlign/indent attrs and
|
||||||
// surface snapshot (reviewed in every diff); this test guards that no canonical
|
// normalizes some editor-ext defaults to null), so the attribute contract is
|
||||||
// node/mark TYPE goes unmirrored. StarterKit-provided types (paragraph, bold,
|
// asymmetric — editor-ext -> mirror — with a small, reasoned, stale-guarded
|
||||||
|
// allowlist for the two blessed divergence kinds (non-round-trippable omissions
|
||||||
|
// and null-normalized defaults). StarterKit-provided types (paragraph, bold,
|
||||||
// heading, …) are contributed by @tiptap/starter-kit in the mirror rather than
|
// heading, …) are contributed by @tiptap/starter-kit in the mirror rather than
|
||||||
// by editor-ext, so they are naturally covered by the mirror's superset.
|
// by editor-ext, so they are naturally covered by the mirror's superset.
|
||||||
//
|
//
|
||||||
@@ -85,3 +87,224 @@ describe("docmost schema vs @docmost/editor-ext (name-level contract)", () => {
|
|||||||
expect(missing).toEqual([]);
|
expect(missing).toEqual([]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ── #515 CODE-MARK `excludes` PARITY (data-loss-sensitive) ──────────────────
|
||||||
|
//
|
||||||
|
// The `code` mark's `excludes` field decides whether inline code can co-occur
|
||||||
|
// with other marks. #515 sets it to "" (excludes nothing) in the canonical
|
||||||
|
// `Code` exported by @docmost/editor-ext AND, because the vendored markdown
|
||||||
|
// mirror must NOT pull that React-aware package into its node runtime, RE-DECLARES
|
||||||
|
// the same override locally in docmost-schema.ts. If the two drift, markdown
|
||||||
|
// import would silently strip bold/italic adjacent to inline code again. Guard it
|
||||||
|
// mechanically: the mirror's built `code` mark and the canonical editor-ext
|
||||||
|
// `Code` must agree on `excludes` (both ""). getSchema surfaces the resolved
|
||||||
|
// value on the mark spec.
|
||||||
|
describe("docmost schema vs @docmost/editor-ext (#515 code excludes parity)", () => {
|
||||||
|
it("keeps the vendored `code` mark's excludes in lockstep with editor-ext Code", () => {
|
||||||
|
// Mirror side: the value the mirror's BUILT schema resolves for `code`.
|
||||||
|
const mirrorExcludes = getSchema(docmostExtensions as never).marks.code.spec
|
||||||
|
.excludes;
|
||||||
|
// Canonical side: the `excludes` DECLARED on the editor-ext `Code` extension
|
||||||
|
// (read from its config — getSchema needs a full node set, so a lone mark
|
||||||
|
// can't be built into a schema here).
|
||||||
|
const canonicalCode = (
|
||||||
|
editorExt as unknown as { Code?: { config?: { excludes?: unknown } } }
|
||||||
|
).Code;
|
||||||
|
const canonicalExcludes = canonicalCode?.config?.excludes;
|
||||||
|
// Both must be the empty string: `code` excludes NOTHING, so bold/italic/…
|
||||||
|
// survive alongside inline code (#515). A drift here would silently strip
|
||||||
|
// marks adjacent to code on markdown import again.
|
||||||
|
expect(canonicalCode).toBeDefined();
|
||||||
|
expect(mirrorExcludes).toBe("");
|
||||||
|
expect(canonicalExcludes).toBe("");
|
||||||
|
expect(mirrorExcludes).toBe(canonicalExcludes);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── ATTRIBUTE-LEVEL CONTRACT (#493 commit 2) ────────────────────────────────
|
||||||
|
//
|
||||||
|
// The name-level contract above catches a WHOLE node/mark type going unmirrored,
|
||||||
|
// but not ATTRIBUTE drift within a vendored type — the exact class that silently
|
||||||
|
// dropped `subpages.recursive`: editor-ext grew an attribute the hand-synced
|
||||||
|
// mirror forgot, so documents using it lost that attribute on a git-sync
|
||||||
|
// round-trip while CI stayed green. This closes that gap by comparing each
|
||||||
|
// editor-ext node/mark's OWN declared attributes (names + defaults) against the
|
||||||
|
// mirror's built ProseMirror schema `spec.attrs`.
|
||||||
|
//
|
||||||
|
// DIRECTION: editor-ext -> mirror. The mirror is deliberately a SUPERSET (it
|
||||||
|
// injects the global `id`/`textAlign`/`indent` attributes and normalizes some
|
||||||
|
// editor-ext "required" attrs to a `null` default), so a reverse compare would
|
||||||
|
// be pure false drift; the meaningful failure is an editor-ext attribute the
|
||||||
|
// mirror DROPS (name) or whose DEFAULT it silently changes. Both directions of
|
||||||
|
// staleness are guarded so the allowlists cannot rot.
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The attributes an editor-ext Tiptap Node/Mark DECLARES itself, read from its
|
||||||
|
* `config.addAttributes()`. Global attributes injected by separate extensions
|
||||||
|
* (unique-id, indent, textAlign) are NOT included here — they are the mirror's
|
||||||
|
* superset and are not part of a per-type declaration — so this isolates each
|
||||||
|
* type's own contribution. A declared attribute with no explicit `default` is a
|
||||||
|
* required attr (Tiptap default `undefined`); we surface that as-is so the
|
||||||
|
* default compare can skip it (the mirror makes such attrs optional/`null`).
|
||||||
|
*/
|
||||||
|
function editorExtOwnAttrs(): Map<
|
||||||
|
string,
|
||||||
|
{ kind: "node" | "mark"; attrs: Record<string, unknown> }
|
||||||
|
> {
|
||||||
|
const out = new Map<
|
||||||
|
string,
|
||||||
|
{ kind: "node" | "mark"; attrs: Record<string, unknown> }
|
||||||
|
>();
|
||||||
|
for (const value of Object.values(editorExt)) {
|
||||||
|
if (!isTiptapNodeOrMark(value)) continue;
|
||||||
|
const ext = value as unknown as {
|
||||||
|
name: string;
|
||||||
|
type: "node" | "mark";
|
||||||
|
options?: unknown;
|
||||||
|
storage?: unknown;
|
||||||
|
config?: { addAttributes?: () => Record<string, { default?: unknown }> };
|
||||||
|
};
|
||||||
|
const fn = ext.config?.addAttributes;
|
||||||
|
// addAttributes reads `this.options`/`this.name`; bind a minimal context
|
||||||
|
// (verified sufficient for every editor-ext extension — none reach for
|
||||||
|
// `this.editor` here). A type with no addAttributes contributes no attrs.
|
||||||
|
const declared =
|
||||||
|
typeof fn === "function"
|
||||||
|
? fn.call({
|
||||||
|
options: ext.options ?? {},
|
||||||
|
name: ext.name,
|
||||||
|
parent: undefined,
|
||||||
|
storage: ext.storage ?? {},
|
||||||
|
} as never)
|
||||||
|
: {};
|
||||||
|
const attrs: Record<string, unknown> = {};
|
||||||
|
for (const [attr, spec] of Object.entries(declared || {})) {
|
||||||
|
// `undefined` marks a required (no-default) attr; keep it so the default
|
||||||
|
// compare can distinguish "no default declared" from "default is null".
|
||||||
|
attrs[attr] = (spec as { default?: unknown })?.default;
|
||||||
|
}
|
||||||
|
out.set(ext.name, { kind: ext.type, attrs });
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The mirror's built-schema `spec.attrs` for a type: attr name -> default. */
|
||||||
|
function mirrorAttrs(
|
||||||
|
name: string,
|
||||||
|
kind: "node" | "mark",
|
||||||
|
): Record<string, unknown> | null {
|
||||||
|
const schema = getSchema(docmostExtensions as never);
|
||||||
|
const spec = kind === "node" ? schema.nodes[name]?.spec : schema.marks[name]?.spec;
|
||||||
|
if (!spec) return null;
|
||||||
|
const out: Record<string, unknown> = {};
|
||||||
|
for (const [attr, def] of Object.entries(spec.attrs || {})) {
|
||||||
|
out[attr] = (def as { default?: unknown }).default;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// An editor-ext attribute the mirror deliberately does NOT vendor because it has
|
||||||
|
// NO markdown round-trip representation — dropping it loses nothing on the
|
||||||
|
// git-sync cycle (the same rationale the flat-roundtrip property suite uses to
|
||||||
|
// allowlist e.g. `tableCell.backgroundColorName`). Blessed by the hand-curated
|
||||||
|
// surface snapshot (schema-surface-snapshot.test.ts), reviewed in every diff.
|
||||||
|
const ACCEPTED_ATTR_OMISSIONS = new Set<string>([
|
||||||
|
"highlight.colorName", // only `highlight.color` round-trips (==text==); the
|
||||||
|
// secondary palette-name is presentational and has no markdown form.
|
||||||
|
]);
|
||||||
|
|
||||||
|
// An editor-ext attribute the mirror vendors but with a DIFFERENT default: the
|
||||||
|
// mirror normalizes an "absent" value to `null` (its uniform optional-attr
|
||||||
|
// convention) rather than editor-ext's UI-oriented default. None of these attrs
|
||||||
|
// is emitted on the markdown surface (the converter round-trips only the
|
||||||
|
// serializable ones), so the default never round-trips and the divergence is
|
||||||
|
// inert — but pinned here so a NEW default change on either side forces review.
|
||||||
|
const ACCEPTED_DEFAULT_DIVERGENCE = new Set<string>([
|
||||||
|
"image.src", // mirror null vs editor "" (an image is never emitted src-less)
|
||||||
|
"link.internal", // mirror null vs editor false (routing attr, not in md link)
|
||||||
|
"pdf.width", // mirror null vs editor 800 (presentational sizing, not in md)
|
||||||
|
"pdf.height", // mirror null vs editor 600 (presentational sizing, not in md)
|
||||||
|
]);
|
||||||
|
|
||||||
|
describe("docmost schema vs @docmost/editor-ext (attribute-level contract)", () => {
|
||||||
|
it("vendors every editor-ext attribute (name) of every shared type — no silently-dropped attrs", () => {
|
||||||
|
const dropped: string[] = [];
|
||||||
|
for (const [name, { kind, attrs }] of editorExtOwnAttrs()) {
|
||||||
|
const mirror = mirrorAttrs(name, kind);
|
||||||
|
if (!mirror) continue; // whole-type omission is the name-level test's job
|
||||||
|
for (const attr of Object.keys(attrs)) {
|
||||||
|
const key = `${name}.${attr}`;
|
||||||
|
if (!(attr in mirror) && !ACCEPTED_ATTR_OMISSIONS.has(key)) {
|
||||||
|
dropped.push(key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Any entry here exists on the editor-ext node/mark but NOT in the mirror
|
||||||
|
// (and is not a blessed non-round-trippable omission): documents using it
|
||||||
|
// lose that attribute on a git-sync round-trip — the subpages.recursive
|
||||||
|
// class. Re-sync src/lib/docmost-schema.ts (and the surface snapshot) or add
|
||||||
|
// a reasoned ACCEPTED_ATTR_OMISSIONS entry before clearing.
|
||||||
|
expect(dropped.sort()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps every editor-ext attribute DEFAULT in sync — no silent default drift", () => {
|
||||||
|
const drift: string[] = [];
|
||||||
|
for (const [name, { kind, attrs }] of editorExtOwnAttrs()) {
|
||||||
|
const mirror = mirrorAttrs(name, kind);
|
||||||
|
if (!mirror) continue;
|
||||||
|
for (const [attr, extDefault] of Object.entries(attrs)) {
|
||||||
|
const key = `${name}.${attr}`;
|
||||||
|
// Skip attrs editor-ext declares WITHOUT a default (required attrs):
|
||||||
|
// the mirror deliberately makes them optional (`null`), a safe superset.
|
||||||
|
if (extDefault === undefined) continue;
|
||||||
|
if (!(attr in mirror)) continue; // a drop, reported by the name test
|
||||||
|
if (
|
||||||
|
JSON.stringify(mirror[attr]) !== JSON.stringify(extDefault) &&
|
||||||
|
!ACCEPTED_DEFAULT_DIVERGENCE.has(key)
|
||||||
|
) {
|
||||||
|
drift.push(
|
||||||
|
`${key}: mirror=${JSON.stringify(mirror[attr])} editor-ext=${JSON.stringify(extDefault)}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expect(drift.sort()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("the attribute allowlists have no stale rows (each is really omitted / divergent)", () => {
|
||||||
|
const ext = editorExtOwnAttrs();
|
||||||
|
const staleOmission: string[] = [];
|
||||||
|
for (const key of ACCEPTED_ATTR_OMISSIONS) {
|
||||||
|
const [name, attr] = key.split(".");
|
||||||
|
const entry = ext.get(name);
|
||||||
|
const mirror = entry ? mirrorAttrs(name, entry.kind) : null;
|
||||||
|
// Stale if editor-ext no longer declares it, or the mirror now DOES vendor
|
||||||
|
// it (so it should be removed from the omission allowlist).
|
||||||
|
if (!entry || !(attr in entry.attrs) || (mirror && attr in mirror)) {
|
||||||
|
staleOmission.push(key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expect(staleOmission, "stale ACCEPTED_ATTR_OMISSIONS rows").toEqual([]);
|
||||||
|
|
||||||
|
const staleDivergence: string[] = [];
|
||||||
|
for (const key of ACCEPTED_DEFAULT_DIVERGENCE) {
|
||||||
|
const [name, attr] = key.split(".");
|
||||||
|
const entry = ext.get(name);
|
||||||
|
const mirror = entry ? mirrorAttrs(name, entry.kind) : null;
|
||||||
|
const extDefault = entry?.attrs[attr];
|
||||||
|
// Stale if the divergence no longer exists (attr gone, or defaults now
|
||||||
|
// agree) — the row should be dropped so the allowlist stays honest.
|
||||||
|
if (
|
||||||
|
!entry ||
|
||||||
|
!mirror ||
|
||||||
|
!(attr in entry.attrs) ||
|
||||||
|
!(attr in mirror) ||
|
||||||
|
extDefault === undefined ||
|
||||||
|
JSON.stringify(mirror[attr]) === JSON.stringify(extDefault)
|
||||||
|
) {
|
||||||
|
staleDivergence.push(key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expect(staleDivergence, "stale ACCEPTED_DEFAULT_DIVERGENCE rows").toEqual([]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -0,0 +1,13 @@
|
|||||||
|
/**
|
||||||
|
* Vitest setup: register the Node (jsdom) HTML parser for the whole suite.
|
||||||
|
*
|
||||||
|
* The package's tests import the converter through the RELATIVE `src/lib/*`
|
||||||
|
* modules (or the `docmost-client`/`src/lib/index` barrel), NOT through the
|
||||||
|
* top-level `index.ts` entry that a real Node consumer imports — so the entry's
|
||||||
|
* side-effect registration of the jsdom parser never runs here. This setup file
|
||||||
|
* performs the SAME registration the Node entry does, so `markdownToProseMirror`
|
||||||
|
* has a DOM parser in the (node-environment) tests, matching production Node
|
||||||
|
* behaviour. The browser path is covered separately by the client paste tests
|
||||||
|
* (jsdom vitest env + native DOMParser) and the dedicated dom-parser test.
|
||||||
|
*/
|
||||||
|
import "../src/lib/dom-parser.node.js";
|
||||||
@@ -9,15 +9,26 @@ import { defineConfig } from 'vitest/config';
|
|||||||
// envelope, markdownToProseMirror) is re-exported there.
|
// envelope, markdownToProseMirror) is re-exported there.
|
||||||
const here = path.dirname(fileURLToPath(import.meta.url));
|
const here = path.dirname(fileURLToPath(import.meta.url));
|
||||||
const libBarrel = path.resolve(here, 'src/lib/index.ts');
|
const libBarrel = path.resolve(here, 'src/lib/index.ts');
|
||||||
|
// Resolve the cross-package `@docmost/editor-ext` specifier to the SIBLING
|
||||||
|
// workspace SOURCE. In a normal checkout this is what pnpm's workspace link +
|
||||||
|
// the package's `module` field already yield; pinning it here makes the schema
|
||||||
|
// contract tests (incl. the #515 code-excludes parity) hermetic and independent
|
||||||
|
// of node_modules layout (e.g. a shared/hoisted store in a git worktree).
|
||||||
|
const editorExtBarrel = path.resolve(here, '../editor-ext/src/index.ts');
|
||||||
|
|
||||||
export default defineConfig({
|
export default defineConfig({
|
||||||
resolve: {
|
resolve: {
|
||||||
alias: {
|
alias: {
|
||||||
'docmost-client': libBarrel,
|
'docmost-client': libBarrel,
|
||||||
|
'@docmost/editor-ext': editorExtBarrel,
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
test: {
|
test: {
|
||||||
environment: 'node',
|
environment: 'node',
|
||||||
include: ['test/**/*.test.ts'],
|
include: ['test/**/*.test.ts'],
|
||||||
|
// Register the Node (jsdom) HTML parser before any test runs. Tests import
|
||||||
|
// the converter via relative src/lib modules, bypassing the top-level entry
|
||||||
|
// that normally installs the parser as a side effect (see setup file).
|
||||||
|
setupFiles: ['test/setup.dom-parser.ts'],
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|||||||
Generated
+7
-5
@@ -284,6 +284,9 @@ importers:
|
|||||||
'@docmost/editor-ext':
|
'@docmost/editor-ext':
|
||||||
specifier: workspace:*
|
specifier: workspace:*
|
||||||
version: link:../../packages/editor-ext
|
version: link:../../packages/editor-ext
|
||||||
|
'@docmost/prosemirror-markdown':
|
||||||
|
specifier: workspace:*
|
||||||
|
version: link:../../packages/prosemirror-markdown
|
||||||
'@excalidraw/excalidraw':
|
'@excalidraw/excalidraw':
|
||||||
specifier: 0.18.0-3a5ef40
|
specifier: 0.18.0-3a5ef40
|
||||||
version: 0.18.0-3a5ef40(@types/react-dom@18.3.1)(@types/react@18.3.12)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)
|
version: 0.18.0-3a5ef40(@types/react-dom@18.3.1)(@types/react@18.3.12)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)
|
||||||
@@ -912,10 +915,6 @@ importers:
|
|||||||
version: 8.57.1(eslint@9.39.4(jiti@2.4.2))(typescript@5.9.3)
|
version: 8.57.1(eslint@9.39.4(jiti@2.4.2))(typescript@5.9.3)
|
||||||
|
|
||||||
packages/editor-ext:
|
packages/editor-ext:
|
||||||
dependencies:
|
|
||||||
marked:
|
|
||||||
specifier: 17.0.5
|
|
||||||
version: 17.0.5
|
|
||||||
devDependencies:
|
devDependencies:
|
||||||
'@vitest/coverage-v8':
|
'@vitest/coverage-v8':
|
||||||
specifier: 4.1.6
|
specifier: 4.1.6
|
||||||
@@ -1117,6 +1116,9 @@ importers:
|
|||||||
'@tiptap/starter-kit':
|
'@tiptap/starter-kit':
|
||||||
specifier: 3.20.4
|
specifier: 3.20.4
|
||||||
version: 3.20.4
|
version: 3.20.4
|
||||||
|
happy-dom:
|
||||||
|
specifier: 20.8.9
|
||||||
|
version: 20.8.9
|
||||||
jsdom:
|
jsdom:
|
||||||
specifier: 25.0.0
|
specifier: 25.0.0
|
||||||
version: 25.0.0
|
version: 25.0.0
|
||||||
@@ -18490,7 +18492,7 @@ snapshots:
|
|||||||
|
|
||||||
happy-dom@20.8.9:
|
happy-dom@20.8.9:
|
||||||
dependencies:
|
dependencies:
|
||||||
'@types/node': 22.19.1
|
'@types/node': 25.5.0
|
||||||
'@types/whatwg-mimetype': 3.0.2
|
'@types/whatwg-mimetype': 3.0.2
|
||||||
'@types/ws': 8.18.1
|
'@types/ws': 8.18.1
|
||||||
entities: 7.0.1
|
entities: 7.0.1
|
||||||
|
|||||||
Reference in New Issue
Block a user