Compare commits

..

3 Commits

Author SHA1 Message Date
agent_coder 03c91dfb0b docs(deploy): поправить устаревший коммент про auto-reload скрытой вкладки под вариант C (#481, ревью F1)
r1-коммент утверждал «auto-reload on a hidden tab», но вариант C (r2) убрал
авто-reload скрытой вкладки ради защиты несохранённого ввода. Переписал под
фактическое поведение: баннер + отложенный reload на следующей in-app навигации.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 04:10:22 +03:00
agent_coder 83f792ca39 fix(deploy): variant C — reload по in-app навигации вместо visibilitychange + лог-след + CHANGELOG (#481, ревью)
Правки по ревью #499. Владелец выбрал вариант C по вопросу потери несохранённого
ввода.

F1 (reload по навигации): убрал ветку visibilitychange→reload И немедленный reload
скрытой-при-получении вкладки — visibility больше не влияет на авто-reload вообще.
На реальном mismatch: баннер (кнопка «Обновить» — немедленный reload под тем же
one-shot гардом) + модульный флаг pendingNavReload. Хук useVersionReloadOnNavigation
(useLocation + useEffect по location.key, пропуск первого рендера через useRef;
react-router v7 BrowserRouter компонентный, объекта роутера нет — подписка изнутри
дерева) на СЛЕДУЮЩЕЙ навигации зовёт consumeNavigationReload → performAutoReload
(mark-перед-reload, при spent/непишущемся storage — только баннер). Скрытая вкладка
идёт тем же путём (C единообразно). chunk-load-error-boundary reload не тронут —
остаётся бэкапом для не-навигирующей вкладки. Так reload в безопасной точке (юзер
и так уходит со страницы), а не когда свернул с недописанным комментарием.

F2 (лог-след): общий recordReloadBreadcrumb/takeReloadBreadcrumb (sessionStorage,
переживает reload). performAutoReload пишет breadcrumb {path:proactive,server,
client} + console.warn перед reload; chunk-boundary пишет {path:chunk-boundary};
surfacePreviousReloadBreadcrumb на маунте UserProvider логирует прошлый след.

F3 (CHANGELOG + AGENTS.md): user-facing запись про version-coherence + строка про
артефакт client/dist/version.json.

Тесты: vitest version-coherence+reload-guard+guarded-reload 23 (переписан на
MemoryRouter+useNavigate: reload РОВНО раз на первой навигации после mismatch, НЕ
на 2-й навигации, НЕ на 2-м mismatch one-shot, НЕ от ухода в фон; скрытая вкладка
— только на навигации; Update — сразу; banner-only/write-fail — никогда). Шире:
features/user 34.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 03:01:07 +03:00
agent_coder 1e4925007d feat(deploy): version-coherence — WS-анонс версии сервера + guarded reload устаревших вкладок (#481)
SPA — долгоживущий клиент: вкладку держат часами, сервер за это время
передеплоивают. Старый index.html ссылается на content-hashed чанки, которых в
новом образе нет → ленивый import() → catch-all отдаёт index.html как text/html →
WebKit «not a valid JavaScript MIME type» (наблюдали в проде). Реактивный
предохранитель (chunk-error-boundary) ловит УЖЕ случившуюся ошибку. Делаем сигнал
ПРОАКТИВНЫМ: сервер сообщает версию по существующему WS, клиент сравнивает и
делает guarded reload ДО битого чанка. closes #481

- Единый источник версии: vite.config.ts вычисляет resolveAppVersion РОВНО раз →
  и в define.APP_VERSION, и в плагине, пишущем dist/version.json. Бандл и файл
  тождественны by construction. Сервер читает client/dist/version.json при старте
  (client-version.ts: resolveClientDistPath вынесен из static.module — единый
  источник пути; readClientBuildVersion → версия или '' на любой ошибке,
  fail-safe). НИКАКИХ Docker/ENV-изменений.
- WS: отдельное событие app-version {version} (НЕ в message/WebSocketEvent —
  член без operation сломал бы union). ws.gateway читает версию в onModuleInit +
  boot-лог ACTIVE/DISABLED; emit в handleConnection (success-ветка, после join).
  ТОЛЬКО per-connect анонс, БЕЗ broadcast (broadcast в кластере → thundering-herd
  reload флота; естественный реконнект покрывает и single-container, и кластер).
- Клиент: listener навешан СИНХРОННО в том же useEffect, что создаёт сокет (до
  connect — иначе гонка с немедленным emit сервера). Чистая decideVersionAction
  (reload|banner|noop; пустая версия → noop fail-safe). Грязная оболочка
  triggerGuardedReload: модульный latch, hidden→немедленный reload, visible→баннер
  + reload-on-hidden {once}, фикс-id закрываемый баннер с кнопкой «Обновить».
- Общий one-shot флаг: reload-guard.ts (hasAutoReloaded/markAutoReloaded над
  существующим chunk-reload-attempted); chunk-load-error-boundary переведён на
  него. Проактивный + реактивный reload делят ОДИН счётчик → максимум одна
  авто-перезагрузка за сессию суммарно; permanent skew / oscillation / disabled
  storage → после первого reload только баннер, петли нет (проверено трассировкой).

Проверка: server jest client-version 7/7; client vitest version-coherence+
reload-guard+guarded-reload 16/16 (coverage 97.9%). Build-proof единого источника:
APP_VERSION=test-A → dist/version.json {"version":"test-A"} + вшито в бандл.
Полный staging-приём (docker build-arg, WS-кадры в DevTools, мульти-таб) — вне
автостенда, шаги приёмки #481 (крит.1-4) на ручную проверку.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 02:14:12 +03:00
72 changed files with 2992 additions and 1105 deletions
+3 -2
View File
@@ -334,7 +334,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend | | `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server | | `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy | | `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked; jsdom (Node only) | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, `apps/server` (server-side markdown import/export, #345), AND `apps/client` (markdown paste/copy + AI-chat render, via the `browser` entry — native `DOMParser`, no jsdom in the client bundle, #347); there is exactly ONE copy of the converter now | | `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`. `build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
@@ -460,9 +460,10 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
### Client structure ### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions: Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI. - **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, `apps/server` (#345), and `apps/client` (#347) — do NOT reintroduce a per-package copy. The client uses the package's `browser` entry (`@docmost/prosemirror-markdown/browser`): markdown paste (`markdown-clipboard.ts`), copy-as-markdown, and AI-chat rendering now all go through the canonical converter, so the hand-written `marked`/`turndown` markdown layer that used to live in `editor-ext` was deleted (#347). The browser entry runs the HTML→DOM stage on the native `DOMParser`, so jsdom stays out of the client bundle. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`. - The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence. For the converter's property-testing and counterexample→fixture process (P1–P4 invariants, the `PROPERTY_SEED`/`PROPERTY_NUM_RUNS` knobs, and the nightly fuzz workflow), see `packages/prosemirror-markdown/README.md`.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`. - API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`. - Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
- The build also emits `client/dist/version.json` (`{"version": …}`) from a small `vite.config.ts` plugin using the **same** `appVersion` that feeds `define.APP_VERSION`, so the file and the baked-in bundle version are identical by construction. The server reads it at startup (`ws.gateway.ts` via `readClientBuildVersion`/`resolveClientDistPath`) and announces it to each socket on connect (`app-version` event) so a tab left open across a redeploy can guard-reload before hitting a stale chunk (version-coherence). No runtime env / Dockerfile change — the file already ships in `client/dist`; missing/empty file ⇒ feature inert.
## Conventions ## Conventions
+12 -12
View File
@@ -117,6 +117,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added ### Added
- **Open tabs pick up a new deploy on their own.** After the server is
redeployed while a tab is left open for hours, the tab now learns the new
build version over the existing WebSocket (announced per-connect, so a natural
reconnect delivers it) and shows a "A new version is available" banner with an
Update button. To avoid dropping a half-written comment or form, the tab is
not reloaded when you merely switch away from it; instead it auto-reloads at
the next safe point — the next in-app navigation (or immediately if you click
Update) — before it can hit a stale lazy-loaded chunk. At most one automatic
reload happens per browser session (shared with the existing chunk-load
recovery), so a permanent version skew degrades to the banner rather than a
reload loop. When the build carries no version info the feature stays inert.
- **Place several images side by side in a row.** A new "Inline (side by - **Place several images side by side in a row.** A new "Inline (side by
side)" alignment mode in the image bubble menu renders consecutive inline side)" alignment mode in the image bubble menu renders consecutive inline
images as a row that wraps onto the next line on narrow screens. The row is images as a row that wraps onto the next line on narrow screens. The row is
@@ -270,18 +282,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed ### Changed
- **Client markdown paste/copy and AI-chat rendering now go through the canonical
converter.** Pasting markdown into the editor, "Copy as markdown", the AI title
generator, and the AI-chat markdown renderer all now use
`@docmost/prosemirror-markdown` (via its new `browser` entry — native
`DOMParser`, no jsdom in the client bundle) instead of the hand-written
`marked`/`turndown` markdown layer in `editor-ext`, which was **deleted**. As a
result, pasting canonical markdown (`^[…]` footnotes, `<!--img …-->`,
`> [!type]` callouts, `$…$` math, `==…==` highlight, standalone `<!--subpages-->`
comments) now produces the SAME nodes the server import produces for the same
text. Chat/reasoning markdown now renders through the editor schema (list items
are wrapped in `<p>`; CSS keeps them tight). (#347)
- **Enabling a public share no longer auto-shares the whole sub-tree.** Turning - **Enabling a public share no longer auto-shares the whole sub-tree.** Turning
a page "Shared to web" now defaults to the page alone; descendant pages become a page "Shared to web" now defaults to the page alone; descendant pages become
public only when you explicitly turn on the dedicated "Include sub-pages" public only when you explicitly turn on the dedicated "Include sub-pages"
-1
View File
@@ -21,7 +21,6 @@
"@atlaskit/pragmatic-drag-and-drop-live-region": "1.3.4", "@atlaskit/pragmatic-drag-and-drop-live-region": "1.3.4",
"@casl/react": "5.0.1", "@casl/react": "5.0.1",
"@docmost/editor-ext": "workspace:*", "@docmost/editor-ext": "workspace:*",
"@docmost/prosemirror-markdown": "workspace:*",
"@excalidraw/excalidraw": "0.18.0-3a5ef40", "@excalidraw/excalidraw": "0.18.0-3a5ef40",
"@mantine/core": "8.3.18", "@mantine/core": "8.3.18",
"@mantine/dates": "8.3.18", "@mantine/dates": "8.3.18",
@@ -1,4 +1,5 @@
{ {
"A new version is available": "A new version is available",
"Account": "Account", "Account": "Account",
"Active": "Active", "Active": "Active",
"Add": "Add", "Add": "Add",
@@ -1,4 +1,5 @@
{ {
"A new version is available": "Доступна новая версия",
"Account": "Аккаунт", "Account": "Аккаунт",
"Active": "Активный", "Active": "Активный",
"Add": "Добавить", "Add": "Добавить",
@@ -1,8 +1,11 @@
import { ReactNode } from "react"; import { ReactNode } from "react";
import { ErrorBoundary } from "react-error-boundary"; import { ErrorBoundary } from "react-error-boundary";
import { Button, Center, Stack, Text } from "@mantine/core"; import { Button, Center, Stack, Text } from "@mantine/core";
import {
const RELOAD_FLAG = "chunk-reload-attempted"; hasAutoReloaded,
markAutoReloaded,
recordReloadBreadcrumb,
} from "@/lib/reload-guard";
// Heuristic detection of a failed dynamic import. Since the code-splitting work, // Heuristic detection of a failed dynamic import. Since the code-splitting work,
// every route (plus Aside / AiChatWindow) is React.lazy: when a new deploy // every route (plus Aside / AiChatWindow) is React.lazy: when a new deploy
@@ -25,16 +28,15 @@ function handleError(error: unknown) {
if (!isChunkLoadError(error)) return; if (!isChunkLoadError(error)) return;
// A stale-chunk 404 is cured by a full reload that re-fetches index.html and // A stale-chunk 404 is cured by a full reload that re-fetches index.html and
// the new chunk manifest. Auto-reload once, guarding against a reload loop // the new chunk manifest. Auto-reload once, guarding against a reload loop
// (e.g. a genuinely missing chunk) with a one-shot sessionStorage flag. If the // (e.g. a genuinely missing chunk) with the shared one-shot session flag
// flag is already set we fall through to the manual recovery UI below. // (see @/lib/reload-guard — shared with the proactive version-coherence
try { // path). If it is already set, or the write fails (storage unavailable), we
if (sessionStorage.getItem(RELOAD_FLAG)) return; // fall through to the manual recovery UI below rather than risk a loop.
sessionStorage.setItem(RELOAD_FLAG, "1"); if (hasAutoReloaded()) return;
} catch { if (!markAutoReloaded()) return;
// sessionStorage unavailable (private mode / disabled): skip the automatic // Trace before the reload clears the console (same diagnostic breadcrumb the
// reload rather than risk an unguarded loop; the fallback UI still recovers. // proactive version-coherence path writes, tagged with this path).
return; recordReloadBreadcrumb({ path: "chunk-boundary" });
}
window.location.reload(); window.location.reload();
} }
@@ -55,15 +55,6 @@
padding-inline-start: 1.4em; padding-inline-start: 1.4em;
} }
/* The canonical converter renders list items through the editor schema, which
wraps each item's content in a <p> (listItem content is `paragraph+`). Drop
that paragraph's block margin so list items render TIGHT (no extra vertical
gap), matching the previous marked output same rule already applied to
table cells above (issue #347). */
.markdown li p {
margin: 0;
}
/* GFM tables in assistant markdown. The chat lives in a NARROW side panel, so a /* GFM tables in assistant markdown. The chat lives in a NARROW side panel, so a
wide LLM table must scroll horizontally instead of collapsing its columns: wide LLM table must scroll horizontally instead of collapsing its columns:
`.markdown` sets `word-break: break-word`, which (with the default table `.markdown` sets `word-break: break-word`, which (with the default table
@@ -181,14 +172,6 @@
margin: 0 0 4px; margin: 0 0 4px;
} }
/* Same as `.markdown li p` above: the canonical converter wraps every list
item's content in a <p>, so without this each reasoning-panel list item would
pick up `.reasoningText p`'s 4px bottom margin and render too loose. Drop it
so Reasoning-panel lists stay tight, mirroring the pre-#347 marked output. */
.reasoningText li p {
margin: 0;
}
.inputWrapper { .inputWrapper {
flex: 0 0 auto; flex: 0 0 auto;
padding-top: var(--mantine-spacing-xs); padding-top: var(--mantine-spacing-xs);
@@ -33,44 +33,29 @@ describe("collapseBlankLines", () => {
}); });
}); });
describe("collapseBlankLines + renderChatMarkdown (canonical converter)", () => { describe("collapseBlankLines + renderChatMarkdown (tight reasoning rendering)", () => {
// Chat markdown now renders through @docmost/prosemirror-markdown (issue #347): it("renders a blank-line-separated list as a TIGHT list (no <li><p>)", () => {
// the SAME converter the editor/import use. Its list items are schema-shaped —
// each <li>'s content is wrapped in a <p> (listItem content is `paragraph+`) —
// so the HTML always carries `<li><p>…</p></li>` regardless of blank-line
// looseness in the source (the converter has no tight/loose distinction). The
// visual tightness that `collapseBlankLines` used to buy is now provided by
// CSS (`.markdown li p { margin: 0 }`), not the HTML shape.
it("renders a blank-line-separated bullet list as a real <ul> list", () => {
const loose = const loose =
"Intro paragraph.\n\n- item one\n\n- item two\n\n- item three"; "Intro paragraph.\n\n- item one\n\n- item two\n\n- item three";
const html = renderChatMarkdown(collapseBlankLines(loose), {}); const html = renderChatMarkdown(collapseBlankLines(loose), {});
// Clean, un-namespaced HTML (DOMSerializer, not XMLSerializer) — no xmlns. // Tight list: each <li> holds the text directly, not wrapped in a <p>.
expect(html).toContain("<li>item one</li>");
expect(html).not.toContain("<li><p>");
// The list still parses as a list after the paragraph (not a paragraph+<br>).
expect(html).toContain("<ul>"); expect(html).toContain("<ul>");
expect(html).not.toMatch(/<ul[^>]*xmlns/);
// The item text is present (inside the schema's <li><p> wrapper).
expect(html).toContain("item one");
// The intro paragraph renders as its own paragraph before the list.
expect(html).toContain("<p>Intro paragraph.</p>"); expect(html).toContain("<p>Intro paragraph.</p>");
}); });
it("renders an ordered list (1. 2.) as a real <ol> list", () => { it("renders an ordered list (1. 2.) as tight after collapsing", () => {
const loose = "Intro.\n\n1. first\n\n2. second"; const loose = "Intro.\n\n1. first\n\n2. second";
const html = renderChatMarkdown(collapseBlankLines(loose), {}); const html = renderChatMarkdown(collapseBlankLines(loose), {});
expect(html).toContain("<ol>"); expect(html).toContain("<ol>");
expect(html).not.toMatch(/<ol[^>]*xmlns/); expect(html).toContain("<li>first</li>");
expect(html).toContain("first"); expect(html).not.toContain("<li><p>");
expect(html).toContain("second");
}); });
it("wraps list-item content in <p> (schema shape; tightness is CSS)", () => { it("the loose source WOULD render <li><p> without collapsing (control)", () => {
// The canonical converter always wraps a list item's content in a paragraph,
// whether or not the source had blank lines between items.
const loose = "- a\n\n- b"; const loose = "- a\n\n- b";
expect(renderChatMarkdown(loose, {})).toContain("<li><p>"); expect(renderChatMarkdown(loose, {})).toContain("<li><p>");
// And a "tight" source produces the identical wrapping (no distinction).
expect(renderChatMarkdown(collapseBlankLines(loose), {})).toContain(
"<li><p>",
);
}); });
}); });
@@ -1,37 +1,6 @@
import { import { markdownToHtml } from "@docmost/editor-ext";
markdownToProseMirrorSync,
docmostExtensions,
} from "@docmost/prosemirror-markdown/browser";
import { getSchema } from "@tiptap/core";
import { Node as PMNode, DOMSerializer } from "@tiptap/pm/model";
import DOMPurify from "dompurify"; import DOMPurify from "dompurify";
// The Docmost editor schema, built once. Chat markdown is rendered through the
// SAME schema the editor/import use (issue #347), so chat output matches how the
// page would render the same markdown.
const chatSchema = getSchema(docmostExtensions);
/**
* Markdown -> HTML for chat display, via the canonical converter. We serialize
* the ProseMirror doc with `DOMSerializer` into a real element and read its
* `innerHTML` (rather than `@tiptap/html`'s `generateHTML`, whose browser path
* uses `XMLSerializer` and stamps a `xmlns` on every block) so the markup is
* clean HTML. `li > p` wrapping is inherent to the schema (listItem content is
* `paragraph+`); the chat CSS zeroes those paragraph margins so lists still
* render tight.
*/
function markdownToChatHtml(markdown: string): string {
const doc = markdownToProseMirrorSync(markdown);
const node = PMNode.fromJSON(chatSchema, doc);
const div = document.createElement("div");
DOMSerializer.fromSchema(chatSchema).serializeFragment(
node.content,
{ document },
div,
);
return div.innerHTML;
}
export interface RenderChatMarkdownOptions { export interface RenderChatMarkdownOptions {
/** /**
* Neutralize INTERNAL links so they render as inert text (no `href`/`target`). * Neutralize INTERNAL links so they render as inert text (no `href`/`target`).
@@ -94,32 +63,22 @@ function neutralizeInternalLinksHook(node: Element): void {
/** /**
* Render AI markdown to sanitized HTML for read-only display. We reuse the * Render AI markdown to sanitized HTML for read-only display. We reuse the
* canonical converter (issue #347): markdown -> ProseMirror JSON (the SAME * app's `markdownToHtml` (the same `marked` pipeline used for paste/import) so
* `markdownToProseMirrorSync` the editor paste/import path uses, so chat output * chat output matches the editor's markdown flavor, then sanitize with
* matches the editor's markdown flavor) -> HTML via `markdownToChatHtml` * DOMPurify LLM output is untrusted, so it must never reach the DOM unsanitized.
* (DOMSerializer), then sanitize with DOMPurify LLM output is untrusted, so it
* must never reach the DOM unsanitized.
* *
* Stays SYNCHRONOUS: both callers render inside React (a memo and a useMemo), * `markdownToHtml` can return `string | Promise<string>` (it has async marked
* so the whole pipeline must resolve without awaiting. The converter's sync * extensions registered). In practice plain chat markdown resolves
* entry makes that possible; on any conversion error we return "" so the caller * synchronously, but we guard the Promise case by returning a safe empty string
* falls back to raw text (the same fallback the old Promise-guard produced). * for that branch (the caller renders the raw text fallback instead).
*/ */
export function renderChatMarkdown( export function renderChatMarkdown(
markdown: string, markdown: string,
options: RenderChatMarkdownOptions = {}, options: RenderChatMarkdownOptions = {},
): string { ): string {
if (!markdown) return ""; if (!markdown) return "";
let html: string; const html = markdownToHtml(markdown);
try { if (typeof html !== "string") return "";
// markdown -> canonical PM JSON -> HTML (native DOMParser in the browser;
// jsdom is never bundled — see @docmost/prosemirror-markdown/browser).
html = markdownToChatHtml(markdown);
} catch {
// Malformed/unsupported markdown must not crash the chat render; fall back
// to raw text (empty return -> caller shows the plain-text branch).
return "";
}
if (!options.neutralizeInternalLinks) { if (!options.neutralizeInternalLinks) {
// Internal chat: unchanged behavior, no hook registered. // Internal chat: unchanged behavior, no hook registered.
@@ -1,206 +0,0 @@
import { describe, it, expect } from "vitest";
import { Editor } from "@tiptap/core";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Bold } from "@tiptap/extension-bold";
import { Italic } from "@tiptap/extension-italic";
import { MarkdownClipboard } from "./markdown-clipboard";
/**
* Integration coverage for the async `handlePaste` seam (issue #347). The paste
* conversion moved to `@docmost/prosemirror-markdown`'s browser entry, whose
* `markdownToProseMirror` is async so `handlePaste` captures the range, claims
* the event (returns true), and dispatches the insert on the next microtask.
* These tests drive that path end to end on a minimal schema (a plain-markdown
* paste whose converted nodes fit paragraph/text/bold/italic), asserting the
* text lands with the right marks and that the raw markdown syntax is consumed
* (recognized as markdown, not inserted literally).
*/
function makeEditor() {
const element = document.createElement("div");
document.body.appendChild(element);
return new Editor({
element,
extensions: [
Document,
Paragraph,
Text,
Bold,
Italic,
MarkdownClipboard.configure({ transformPastedText: true }),
],
content: { type: "doc", content: [{ type: "paragraph" }] },
});
}
// Locate the markdownClipboard plugin and invoke its handlePaste directly with a
// synthetic clipboard event (jsdom has no real paste pipeline). The plugin's
// handlePaste closes over the extension `this`, so calling it off the plugin
// props preserves `this.editor`/`this.options`.
function paste(editor: Editor, text: string): boolean {
const view = editor.view;
const plugin = view.state.plugins.find(
(p: any) => p.props && p.spec?.key,
) as any;
const event = {
clipboardData: {
getData: (type: string) => (type === "text/plain" ? text : ""),
},
} as unknown as ClipboardEvent;
// Find the specific handlePaste that belongs to the markdown clipboard plugin.
const md = view.state.plugins.find(
(p: any) => typeof p.props?.handlePaste === "function",
) as any;
return md.props.handlePaste(view, event, view.state.selection.content());
}
// Flush the microtask queue so the async .then() dispatch runs.
const flush = () => new Promise((r) => setTimeout(r, 0));
describe("MarkdownClipboard handlePaste (async md -> PM)", () => {
it("converts a plain-markdown paste with bold/italic into marked text", async () => {
const editor = makeEditor();
const claimed = paste(editor, "hello **bold** and *italic*");
// The paste is claimed synchronously (async insert follows).
expect(claimed).toBe(true);
await flush();
const json = editor.getJSON();
const text = JSON.stringify(json);
// The raw markdown asterisks are consumed (recognized), not inserted literally.
expect(editor.getText()).not.toContain("**");
expect(editor.getText()).toContain("bold");
expect(editor.getText()).toContain("italic");
// The bold/italic marks materialized.
expect(text).toContain('"bold"');
expect(text).toContain('"italic"');
editor.destroy();
});
it("recognizes a bullet list paste as list structure (not literal '-')", async () => {
// A bullet list is not representable in this minimal schema, so the converter
// output would fail PMNode.fromJSON and the catch inserts raw text. Use a
// paste whose nodes DO fit the schema to assert the happy path instead: two
// paragraphs separated by a blank line.
const editor = makeEditor();
paste(editor, "first para\n\nsecond para");
await flush();
const json = editor.getJSON() as any;
const paras = (json.content || []).filter(
(n: any) => n.type === "paragraph",
);
// Two paragraphs materialized from the blank-line-separated markdown.
expect(paras.length).toBeGreaterThanOrEqual(2);
expect(editor.getText()).toContain("first para");
expect(editor.getText()).toContain("second para");
editor.destroy();
});
it("falls back to raw text when conversion yields nodes the schema lacks", async () => {
// `# heading` converts to a `heading` node absent from this minimal schema,
// so PMNode.fromJSON throws and the catch re-inserts the raw text — the user
// never loses their clipboard content.
const editor = makeEditor();
paste(editor, "# a heading line");
await flush();
// Content is preserved (either as heading text or literal), never dropped.
expect(editor.getText()).toContain("a heading line");
editor.destroy();
});
});
// The async seam captures the target range synchronously, then replaces on the
// next microtask. If the document changed under it between capture and resolve
// (impossible in prod — same microtask — but pinned here), BOTH the success
// (replaceRange) and the fail-open (insertText) branches must fall back to the
// LIVE selection rather than a stale absolute range, so neither clobbers content
// nor throws a RangeError. We force the mid-flight change by dispatching a
// doc-mutating transaction AFTER the synchronous claim but BEFORE flushing the
// microtask that runs the `.then`/`.catch`.
describe("MarkdownClipboard handlePaste — doc-changed-mid-flight guard", () => {
// Replace the whole doc with one paragraph of `text` (synchronous dispatch).
// An empty string yields an empty paragraph (a text node may not be empty).
function seedContent(editor: Editor, text: string) {
editor.commands.setContent({
type: "doc",
content: [
text
? { type: "paragraph", content: [{ type: "text", text }] }
: { type: "paragraph" },
],
});
}
it("success branch: mid-flight doc change routes the paste to the LIVE selection, never the stale range (clobber-proving)", async () => {
// The paste captures a NON-EMPTY range {1,5} (over "AAAA"). Then, before the
// async resolve, the doc GROWS ("MARKER" inserted at the start) and the cursor
// is parked at the doc END. The captured {1,5} is now stale and points INTO
// "MARKER". A WORKING guard replaces at the live (end) selection → MARKER is
// untouched. A BROKEN guard replaces the stale {1,5} → it erases the first
// characters of MARKER (this is what a zero-width `from==to` range could never
// reveal, which is why the earlier version was vacuous).
const editor = makeEditor();
seedContent(editor, "AAAABBBB");
editor.commands.setTextSelection({ from: 1, to: 5 }); // captured range = {1,5}
const claimed = paste(editor, "hello **bold**");
expect(claimed).toBe(true);
// Mid-flight: grow the doc and move the cursor to a KNOWN-safe end position.
editor.view.dispatch(editor.view.state.tr.insertText("MARKER", 1));
const end = editor.state.doc.content.size;
editor.commands.setTextSelection({ from: end, to: end });
await flush();
const text = editor.getText();
// MARKER intact only if the guard used the live selection, not the stale range.
expect(text).toContain("MARKER");
expect(text).toContain("bold");
expect(text).not.toContain("**");
editor.destroy();
});
it("fail-open branch: a mid-flight doc SHRINK makes the stale `to` out of bounds — the guard must avoid a RangeError (throw-proving)", async () => {
// The paste captures a range {1,9} over an 8-char paragraph, then the
// conversion FAILS (`# heading` -> a heading node the minimal schema lacks,
// so PMNode.fromJSON throws -> the fail-open catch runs). Before the reject,
// the doc is SHRUNK to an empty paragraph, so the captured `to` (9) is now far
// past the doc's end. A WORKING guard inserts the raw text at the live (valid)
// selection → "raw heading" lands. A BROKEN guard does insertText(md, 1, 9) on
// a size-2 doc → RangeError, so the dispatch never runs and "raw heading" is
// absent (the assertion reddens). A zero-width/growing-doc setup could never
// push `to` out of bounds, which is why the earlier version was vacuous.
const editor = makeEditor();
seedContent(editor, "AAAABBBB");
editor.commands.setTextSelection({ from: 1, to: 9 }); // captured range = {1,9}
paste(editor, "# raw heading");
// Mid-flight: shrink the doc so the captured `to` = 9 is now out of bounds.
seedContent(editor, "");
await flush();
const text = editor.getText();
// Raw text lands (via the live selection) only if the guard avoided the
// stale, now-out-of-bounds range.
expect(text).toContain("raw heading");
editor.destroy();
});
it("two pastes in flight: neither payload is lost (no data loss)", async () => {
// Prod-unreachable (two paste events are separate macrotasks, and each
// conversion resolves on a microtask before the next), but pinned here: when
// both resolve back-to-back, the second sees the changed doc and inserts at
// the live selection the first left — so the two payloads may INTERLEAVE, but
// neither is dropped. We assert no data loss, not contiguity.
const editor = makeEditor();
paste(editor, "alphaword");
paste(editor, "betaword");
await flush();
const text = editor.getText();
// Neither payload fully dropped (interleaving may split one of them).
expect(text).toContain("alpha");
expect(text).toContain("beta");
editor.destroy();
});
});
@@ -1,12 +1,5 @@
import { describe, it, expect } from "vitest"; import { describe, it, expect } from "vitest";
// Markdown conversion now goes through the canonical package's BROWSER entry import { htmlToMarkdown } from "@docmost/editor-ext";
// (issue #347): the same converter the server import/export uses, resolved via
// the `browser` exports condition so it runs on the native `DOMParser` (the
// client jsdom vitest env provides one) with jsdom never bundled.
import {
convertProseMirrorToMarkdown,
markdownToProseMirrorSync,
} from "@docmost/prosemirror-markdown/browser";
import { import {
normalizeTableColumnWidths, normalizeTableColumnWidths,
classifyClipboardSelection, classifyClipboardSelection,
@@ -182,13 +175,10 @@ describe("classifyClipboardSelection", () => {
// Output-level tests for the table clipboard regression: copying a table must // Output-level tests for the table clipboard regression: copying a table must
// yield a real GFM pipe table, NOT one-value-per-line concatenated cells. // yield a real GFM pipe table, NOT one-value-per-line concatenated cells.
// These exercise the actual markdown produced by convertProseMirrorToMarkdown // These exercise the actual markdown produced by htmlToMarkdown (the same
// the same serializer step the clipboardTextSerializer now runs (issue #347) — // serializer step the clipboardTextSerializer runs), so they pin the OUTPUT
// so they pin the OUTPUT shape that the classifier-flag tests above do not cover. // shape that the classifier-flag tests above do not cover.
// Input is ProseMirror JSON (what the copied slice serializes to), matching the describe("table clipboard markdown output (htmlToMarkdown)", () => {
// clipboardTextSerializer's new call: it wraps the slice content in a synthetic
// `doc` (and the bare-rows case in a `table`) and calls the converter.
describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () => {
// Trim each line and drop blanks so structural assertions are whitespace-robust. // Trim each line and drop blanks so structural assertions are whitespace-robust.
function lines(md: string): string[] { function lines(md: string): string[] {
return md return md
@@ -198,10 +188,10 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
} }
// A GFM separator row like "| --- | --- |" (any number of columns), tolerant // A GFM separator row like "| --- | --- |" (any number of columns), tolerant
// of the padding the serializer emits. // of the padding turndown emits.
function isSeparatorRow(line: string): boolean { function isSeparatorRow(line: string): boolean {
const compact = line.replace(/\s+/g, ""); const compact = line.replace(/\s+/g, "");
return /^\|(?::?-{2,}:?\|)+$/.test(compact); return /^\|(?:-{3,}\|)+$/.test(compact);
} }
// Split a pipe-delimited row into trimmed cell values. // Split a pipe-delimited row into trimmed cell values.
@@ -213,33 +203,42 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
.map((c) => c.trim()); .map((c) => c.trim());
} }
const cell = (t: string) => ({
type: "tableCell",
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
});
const headerCell = (t: string) => ({
type: "tableHeader",
content: [{ type: "paragraph", content: [{ type: "text", text: t }] }],
});
const row = (nodes: any[]) => ({ type: "tableRow", content: nodes });
it("serializes a header-less partial cell selection (bare rows) as a valid GFM pipe table", () => { it("serializes a header-less partial cell selection (bare rows) as a valid GFM pipe table", () => {
// Mirror the serializer's `wrapBareRows` branch: bare tableRow nodes are // Mirror the serializer's `wrapBareRows` branch exactly: bare <tr> nodes are
// wrapped in a synthetic `table` and convertProseMirrorToMarkdown is called // wrapped in <table><tbody> and htmlToMarkdown(div.innerHTML) is called.
// (see markdown-clipboard.ts clipboardTextSerializer). // See markdown-clipboard.ts clipboardTextSerializer:
const rows = [ // const table = document.createElement("table");
row([cell("a"), cell("b")]), // const tbody = document.createElement("tbody");
row([cell("c"), cell("d")]), // tbody.appendChild(fragment); table.appendChild(tbody);
]; // div.appendChild(table);
const md = convertProseMirrorToMarkdown({ // return htmlToMarkdown(div.innerHTML);
type: "doc", const div = document.createElement("div");
content: [{ type: "table", content: rows }], const table = document.createElement("table");
}); const tbody = document.createElement("tbody");
for (const [c1, c2] of [
["a", "b"],
["c", "d"],
]) {
const tr = document.createElement("tr");
const td1 = document.createElement("td");
td1.textContent = c1;
const td2 = document.createElement("td");
td2.textContent = c2;
tr.appendChild(td1);
tr.appendChild(td2);
tbody.appendChild(tr);
}
table.appendChild(tbody);
div.appendChild(table);
const md = htmlToMarkdown(div.innerHTML);
const ls = lines(md); const ls = lines(md);
// Valid GFM: a header/data separator row is present. // Valid GFM: a header/data separator row is present (an empty header is
// synthesized by the GFM turndown plugin for a header-less table — fine).
expect(ls.some(isSeparatorRow)).toBe(true); expect(ls.some(isSeparatorRow)).toBe(true);
// NOT the old broken "one value per line" shape: every line is pipe-delimited. // NOT the old broken "one value per line" shape: every line is pipe-delimited
// and no line is a bare cell value on its own.
expect(ls.every((l) => l.includes("|"))).toBe(true); expect(ls.every((l) => l.includes("|"))).toBe(true);
expect(md).not.toMatch(/^\s*(a|b|c|d)\s*$/m); expect(md).not.toMatch(/^\s*(a|b|c|d)\s*$/m);
// The cell values land in real pipe-delimited data rows. // The cell values land in real pipe-delimited data rows.
@@ -249,21 +248,39 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
}); });
it("serializes a whole table with a header row as a proper GFM table (headline regression)", () => { it("serializes a whole table with a header row as a proper GFM table (headline regression)", () => {
// Mirror the serializer's non-wrap branch: the full `table` node is the // Mirror the serializer's non-wrap branch: the full <table> node is appended
// slice content and convertProseMirrorToMarkdown runs on it. // directly (div.appendChild(fragment)) and htmlToMarkdown(div.innerHTML) runs.
const md = convertProseMirrorToMarkdown({ const div = document.createElement("div");
type: "doc", const table = document.createElement("table");
content: [
{ const thead = document.createElement("thead");
type: "table", const headerRow = document.createElement("tr");
content: [ for (const h of ["Name", "Age"]) {
row([headerCell("Name"), headerCell("Age")]), const th = document.createElement("th");
row([cell("Alice"), cell("30")]), th.textContent = h;
row([cell("Bob"), cell("25")]), headerRow.appendChild(th);
], }
}, thead.appendChild(headerRow);
], table.appendChild(thead);
});
const tbody = document.createElement("tbody");
for (const [name, age] of [
["Alice", "30"],
["Bob", "25"],
]) {
const tr = document.createElement("tr");
const td1 = document.createElement("td");
td1.textContent = name;
const td2 = document.createElement("td");
td2.textContent = age;
tr.appendChild(td1);
tr.appendChild(td2);
tbody.appendChild(tr);
}
table.appendChild(tbody);
div.appendChild(table);
const md = htmlToMarkdown(div.innerHTML);
const ls = lines(md); const ls = lines(md);
// Proper GFM structure: separator row + all rows pipe-delimited. // Proper GFM structure: separator row + all rows pipe-delimited.
@@ -279,146 +296,3 @@ describe("table clipboard markdown output (convertProseMirrorToMarkdown)", () =>
expect(md).not.toMatch(/^\s*(Name|Age|Alice|Bob|30|25)\s*$/m); expect(md).not.toMatch(/^\s*(Name|Age|Alice|Bob|30|25)\s*$/m);
}); });
}); });
// #347 acceptance: pasting CANONICAL markdown yields the SAME nodes the server
// import produces for the same text. The paste path calls markdownToProseMirror
// (the package browser entry) — the identical converter the server import uses —
// so asserting the converter (via the browser entry, on the native DOMParser)
// recognizes each canon form pins the paste-parity guarantee. These forms were
// NOT recognized by the old editor-ext marked layer the paste used before.
describe("canonical markdown paste recognition (browser entry parity)", () => {
// Collect every node type present in a doc (recursively).
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
if (!n || typeof n !== "object") return set;
if (n.type) set.add(n.type);
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
return set;
};
const findNode = (n: any, type: string): any => {
if (!n || typeof n !== "object") return undefined;
if (n.type === type) return n;
if (Array.isArray(n.content)) {
for (const c of n.content) {
const hit = findNode(c, type);
if (hit) return hit;
}
}
return undefined;
};
const allText = (n: any): string => {
if (!n || typeof n !== "object") return "";
if (typeof n.text === "string") return n.text;
if (Array.isArray(n.content)) return n.content.map(allText).join("");
return "";
};
it("^[…] inline footnote -> footnoteReference + footnotesList", () => {
const doc = markdownToProseMirrorSync("Body^[a note here].");
const types = collectTypes(doc);
expect(types.has("footnoteReference")).toBe(true);
expect(types.has("footnotesList")).toBe(true);
expect(types.has("footnoteDefinition")).toBe(true);
});
it('<!--img {…}--> attached image comment -> image with align', () => {
const doc = markdownToProseMirrorSync(
'![alt](/files/x.png) <!--img {"align":"left"}-->',
);
const img = findNode(doc, "image");
expect(img).toBeTruthy();
expect(img.attrs?.align).toBe("left");
expect(img.attrs?.src).toBe("/files/x.png");
});
it("> [!type] Obsidian callout -> callout node with type", () => {
const doc = markdownToProseMirrorSync("> [!warning]\n> be careful");
const callout = findNode(doc, "callout");
expect(callout).toBeTruthy();
expect(callout.attrs?.type).toBe("warning");
expect(allText(callout)).toContain("be careful");
});
it("$…$ inline math -> mathInline node", () => {
const doc = markdownToProseMirrorSync("Euler: $e^{i\\pi}+1=0$ done");
const math = findNode(doc, "mathInline");
expect(math).toBeTruthy();
expect(math.attrs?.text).toContain("e^{i\\pi}");
});
it("==…== highlight -> highlight mark", () => {
const doc = markdownToProseMirrorSync("A ==marked== word");
const marked = findNode(doc, "text");
// The highlighted run carries a `highlight` mark somewhere in the doc.
const hasHighlight = (n: any): boolean => {
if (!n || typeof n !== "object") return false;
if (
n.type === "text" &&
(n.marks || []).some((m: any) => m.type === "highlight")
)
return true;
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
};
expect(marked).toBeTruthy();
expect(hasHighlight(doc)).toBe(true);
});
it("<!--subpages--> standalone comment -> subpages node", () => {
const doc = markdownToProseMirrorSync("intro\n\n<!--subpages-->\n\nafter");
expect(collectTypes(doc).has("subpages")).toBe(true);
});
});
// #347 negatives: plain text carrying markdown-LIKE punctuation must NOT be
// silently converted/mangled (currency, bare `==`, a `[^1]` reference form).
describe("plain-text paste negatives (no phantom conversion)", () => {
const findNode = (n: any, type: string): any => {
if (!n || typeof n !== "object") return undefined;
if (n.type === type) return n;
if (Array.isArray(n.content)) {
for (const c of n.content) {
const hit = findNode(c, type);
if (hit) return hit;
}
}
return undefined;
};
const collectTypes = (n: any, set = new Set<string>()): Set<string> => {
if (!n || typeof n !== "object") return set;
if (n.type) set.add(n.type);
if (Array.isArray(n.content)) n.content.forEach((c) => collectTypes(c, set));
return set;
};
const allText = (n: any): string => {
if (!n || typeof n !== "object") return "";
if (typeof n.text === "string") return n.text;
if (Array.isArray(n.content)) return n.content.map(allText).join("");
return "";
};
it("currency `$5 and $10` is NOT turned into math", () => {
const doc = markdownToProseMirrorSync("It costs $5 and $10 total");
expect(findNode(doc, "mathInline")).toBeFalsy();
expect(allText(doc)).toContain("$5 and $10");
});
it("a lone `==` is NOT turned into a highlight", () => {
const doc = markdownToProseMirrorSync("compare a == b in code");
const hasHighlight = (n: any): boolean => {
if (!n || typeof n !== "object") return false;
if (
n.type === "text" &&
(n.marks || []).some((m: any) => m.type === "highlight")
)
return true;
return Array.isArray(n.content) ? n.content.some(hasHighlight) : false;
};
expect(hasHighlight(doc)).toBe(false);
expect(allText(doc)).toContain("== b");
});
it("a `[^1]` reference form (no `^[`) is NOT turned into a footnote", () => {
const doc = markdownToProseMirrorSync("see note [^1] for details");
expect(collectTypes(doc).has("footnoteReference")).toBe(false);
expect(allText(doc)).toContain("[^1]");
});
});
@@ -1,23 +1,15 @@
// adapted from: https://github.com/aguingand/tiptap-markdown/blob/main/src/extensions/tiptap/clipboard.js - MIT // adapted from: https://github.com/aguingand/tiptap-markdown/blob/main/src/extensions/tiptap/clipboard.js - MIT
import { Extension } from "@tiptap/core"; import { Extension } from "@tiptap/core";
import { Plugin, PluginKey, TextSelection } from "@tiptap/pm/state"; import { Plugin, PluginKey, TextSelection } from "@tiptap/pm/state";
import { DOMParser, DOMSerializer, Fragment, Slice, Node as PMNode } from "@tiptap/pm/model"; import { DOMParser, DOMSerializer, Fragment, Slice } from "@tiptap/pm/model";
import { find } from "linkifyjs"; import { find } from "linkifyjs";
import { import {
markdownToHtml,
htmlToMarkdown,
canonicalizeFootnotes, canonicalizeFootnotes,
FOOTNOTES_LIST_NAME, FOOTNOTES_LIST_NAME,
FOOTNOTE_REFERENCE_NAME, FOOTNOTE_REFERENCE_NAME,
} from "@docmost/editor-ext"; } from "@docmost/editor-ext";
// Markdown <-> ProseMirror conversion now lives ONLY in the canonical
// `@docmost/prosemirror-markdown` package (issue #347). The BROWSER entry uses
// the native `DOMParser` for its HTML->DOM stage (jsdom stays out of the client
// bundle) while producing the SAME nodes the server import does — so a paste of
// canonical markdown (`^[…]`, `<!--img …-->`, `> [!type]`, `$…$`, `==…==`,
// standalone comments) is recognized identically to import.
import {
markdownToProseMirror,
convertProseMirrorToMarkdown,
} from "@docmost/prosemirror-markdown/browser";
import type { Schema } from "@tiptap/pm/model"; import type { Schema } from "@tiptap/pm/model";
export const MarkdownClipboard = Extension.create({ export const MarkdownClipboard = Extension.create({
@@ -47,24 +39,25 @@ export const MarkdownClipboard = Extension.create({
classifyClipboardSelection(topLevelNodes); classifyClipboardSelection(topLevelNodes);
if (!asMarkdown) return null; if (!asMarkdown) return null;
// Convert the copied selection to Markdown through the canonical const div = document.createElement("div");
// package (issue #347), the SAME serializer the server export uses, const serializer = DOMSerializer.fromSchema(this.editor.schema);
// so a copied table/list matches the on-disk markdown form. The const fragment = serializer.serializeFragment(slice.content);
// converter takes a ProseMirror `doc` JSON, so wrap the slice's
// top-level content in a synthetic doc.
const content = slice.content.toJSON() as any[];
if (wrapBareRows) { if (wrapBareRows) {
// A partial table cell-selection serializes to bare `tableRow` // A partial table cell-selection serializes to bare <tr> nodes
// nodes (prosemirror-tables yields the whole `table` node only for // (prosemirror-tables returns the whole `table` node only when the
// a full-table selection). The converter's table case expects a // entire table is selected). Bare <tr> would be foster-parented
// `table` wrapper, so wrap the bare rows in one — mirroring the old // away by the HTML parser inside htmlToMarkdown, so wrap them in
// <table><tbody> wrap that the HTML->markdown step needed. // <table><tbody> first for the GFM turndown rule to detect them.
return convertProseMirrorToMarkdown({ const table = document.createElement("table");
type: "doc", const tbody = document.createElement("tbody");
content: [{ type: "table", content }], tbody.appendChild(fragment);
}); table.appendChild(tbody);
div.appendChild(table);
} else {
div.appendChild(fragment);
} }
return convertProseMirrorToMarkdown({ type: "doc", content }); return htmlToMarkdown(div.innerHTML);
}, },
handlePaste: (view, event, slice) => { handlePaste: (view, event, slice) => {
if (!event.clipboardData) { if (!event.clipboardData) {
@@ -102,115 +95,37 @@ export const MarkdownClipboard = Extension.create({
} }
} }
const schema = this.editor.schema; const { tr } = view.state;
// Capture the target range NOW. markdownToProseMirror RETURNS A const { from, to } = view.state.selection;
// PROMISE (kept async only for the Node consumers' contract; the
// conversion pipeline itself is synchronous), so the actual replace
// happens on the next microtask. No user input can interleave a
// microtask, so the state is unchanged when we dispatch — but we
// still re-read the live state before replacing and, if the doc did
// change under us, fall back to the live selection rather than the
// captured (now-stale) range.
const from = view.state.selection.from;
const to = view.state.selection.to;
const startDoc = view.state.doc;
const md = text.replace(/\n+$/, "");
void markdownToProseMirror(md) const parsed = markdownToHtml(text.replace(/\n+$/, ""));
.then((doc) => { const body = elementFromString(parsed);
if (view.isDestroyed) return; normalizeTableColumnWidths(body);
// Canonical PM-JSON -> HTML via the LIVE editor schema, then
// reuse the UNCHANGED downstream seam (normalizeTableColumnWidths
// + parseSlice + canonicalizePastedFootnotes). The JSON->HTML->
// JSON hop is lossless (same schema both directions); it lets the
// existing paste-insertion logic stay byte-identical — only the
// SOURCE of the markdown conversion changed (issue #347 guardrail:
// no converter logic in the client, only a call into the package).
const node = PMNode.fromJSON(schema, doc);
const div = document.createElement("div");
DOMSerializer.fromSchema(schema).serializeFragment(
node.content,
{ document },
div,
);
const body = elementFromString(div.innerHTML); const parsedSlice = DOMParser.fromSchema(
normalizeTableColumnWidths(body); this.editor.schema,
).parseSlice(body, {
preserveWhitespace: true,
});
const parsedSlice = DOMParser.fromSchema(schema).parseSlice( // A markdown paste builds its ProseMirror fragment directly (DOM ->
body, // parseSlice), bypassing the editor's footnoteSyncPlugin, which never
{ preserveWhitespace: true }, // reorders an existing list. So a pasted markdown block whose footnote
); // definitions are out of order (or contains orphan defs) would be
// stored out of order. Canonicalize the self-contained pasted block so
// its footnotes come out reference-ordered, deduped and orphan-free
// (issue #228). See canonicalizePastedFootnotes for why this is scoped
// to whole-block pastes that carry their own footnotesList.
const contentNodes = canonicalizePastedFootnotes(
parsedSlice,
this.editor.schema,
);
// A markdown paste builds its ProseMirror fragment directly (DOM tr.replaceRange(from, to, contentNodes);
// -> parseSlice), bypassing the editor's footnoteSyncPlugin, which const insertEnd = tr.mapping.map(from, 1);
// never reorders an existing list. So a pasted markdown block whose tr.setSelection(TextSelection.near(tr.doc.resolve(Math.max(from, insertEnd - 2)), -1));
// footnote definitions are out of order (or contains orphan defs) tr.setMeta('paste', true)
// would be stored out of order. Canonicalize the self-contained view.dispatch(tr);
// pasted block so its footnotes come out reference-ordered, deduped
// and orphan-free (issue #228). See canonicalizePastedFootnotes for
// why this is scoped to whole-block pastes that carry their own
// footnotesList.
const contentNodes = canonicalizePastedFootnotes(
parsedSlice,
schema,
);
// Target the captured range (normally still valid — same
// microtask). If the doc changed under us since capture, the
// captured absolute from/to are stale, so fall back to the live
// selection rather than StepMap-mapping the old range.
const tr = view.state.tr;
let mappedFrom = from;
let mappedTo = to;
if (view.state.doc !== startDoc) {
// Defensive: if the doc changed under us, fall back to the
// current selection rather than a stale absolute range.
mappedFrom = view.state.selection.from;
mappedTo = view.state.selection.to;
}
tr.replaceRange(mappedFrom, mappedTo, contentNodes);
const insertEnd = tr.mapping.map(mappedFrom, 1);
tr.setSelection(
TextSelection.near(
tr.doc.resolve(Math.max(mappedFrom, insertEnd - 2)),
-1,
),
);
tr.setMeta("paste", true);
view.dispatch(tr);
})
.catch((err) => {
// Fail-open: a conversion error must not swallow the paste
// silently in a way that loses the text. We already claimed the
// event (returned true), so re-insert the raw text as a plain
// paragraph so the user never loses their clipboard content.
// Log it: this catch covers BOTH the converter and the success
// `.then` body (e.g. PMNode.fromJSON throwing on a schema drift
// between the canonical package and the live editor schema), so a
// silent degrade to raw text would otherwise be an invisible,
// non-reproducible regression ("my table pasted as text").
console.error(
"markdown paste conversion failed, inserting raw text",
err,
);
if (view.isDestroyed) return;
const tr = view.state.tr;
// Same guard the success path uses: if the doc changed under us
// since the range was captured (normally never — same microtask),
// the captured absolute from/to are stale and would throw a
// RangeError here (an unhandled rejection on a hot paste path).
// Fall back to the live selection instead of a stale range.
if (view.state.doc !== startDoc) {
const sel = view.state.selection;
tr.insertText(md, sel.from, sel.to);
} else {
tr.insertText(md, from, to);
}
tr.setMeta("paste", true);
view.dispatch(tr);
});
// Claim the paste: we insert asynchronously above.
return true; return true;
}, },
// Strip trailing whitespace-only paragraphs from pasted content. // Strip trailing whitespace-only paragraphs from pasted content.
@@ -33,11 +33,10 @@ vi.mock("@/lib/local-emitter.ts", () => ({
default: { emit: (...args: unknown[]) => localEmitMock(...args) }, default: { emit: (...args: unknown[]) => localEmitMock(...args) },
})); }));
// convertProseMirrorToMarkdown echoes a marker carried on the fake editor's // htmlToMarkdown just echoes the editor HTML so each test controls the markdown
// getJSON() doc, so each test controls the markdown purely via the fake page // purely via the fake page editor's getHTML().
// editor (issue #347: the hook now serializes editor JSON through the package). vi.mock("@docmost/editor-ext", () => ({
vi.mock("@docmost/prosemirror-markdown/browser", () => ({ htmlToMarkdown: (html: string) => html,
convertProseMirrorToMarkdown: (doc: { __md?: string }) => doc?.__md ?? "",
})); }));
const notificationsShowMock = vi.fn(); const notificationsShowMock = vi.fn();
@@ -54,12 +53,10 @@ import { useGeneratePageTitle } from "./use-generate-page-title.ts";
// --- Test helpers ------------------------------------------------------------- // --- Test helpers -------------------------------------------------------------
function makePageEditor(pageId: string, md = "content"): Editor { function makePageEditor(pageId: string, html = "<p>content</p>"): Editor {
return { return {
isDestroyed: false, isDestroyed: false,
// The mocked convertProseMirrorToMarkdown reads `__md` back off this doc, getHTML: () => html,
// so `md` is exactly the markdown the hook will send to the title service.
getJSON: () => ({ type: "doc", __md: md }),
storage: { pageId }, storage: { pageId },
} as unknown as Editor; } as unknown as Editor;
} }
@@ -3,7 +3,7 @@ import { useMutation } from "@tanstack/react-query";
import { useAtomValue } from "jotai"; import { useAtomValue } from "jotai";
import { notifications } from "@mantine/notifications"; import { notifications } from "@mantine/notifications";
import { useTranslation } from "react-i18next"; import { useTranslation } from "react-i18next";
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser"; import { htmlToMarkdown } from "@docmost/editor-ext";
import { import {
pageEditorAtom, pageEditorAtom,
titleEditorAtom, titleEditorAtom,
@@ -49,9 +49,7 @@ export function useGeneratePageTitle(pageId: string) {
mutationFn: async () => { mutationFn: async () => {
if (!pageEditor || pageEditor.isDestroyed) return; if (!pageEditor || pageEditor.isDestroyed) return;
// Serialize the live editor content to markdown through the canonical const markdown = htmlToMarkdown(pageEditor.getHTML()).trim();
// converter (issue #347), matching the on-disk/export markdown form.
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON()).trim();
if (!markdown) { if (!markdown) {
notifications.show({ message: t("The note is empty"), color: "yellow" }); notifications.show({ message: t("The note is empty"), color: "yellow" });
return; return;
@@ -37,7 +37,7 @@ import { useTreeMutation } from "@/features/page/tree/hooks/use-tree-mutation.ts
import { PageWidthToggle } from "@/features/user/components/page-width-pref.tsx"; import { PageWidthToggle } from "@/features/user/components/page-width-pref.tsx";
import { Trans, useTranslation } from "react-i18next"; import { Trans, useTranslation } from "react-i18next";
import ExportModal from "@/components/common/export-modal"; import ExportModal from "@/components/common/export-modal";
import { convertProseMirrorToMarkdown } from "@docmost/prosemirror-markdown/browser"; import { htmlToMarkdown } from "@docmost/editor-ext";
import { import {
pageEditorAtom, pageEditorAtom,
yjsConnectionStatusAtom, yjsConnectionStatusAtom,
@@ -199,9 +199,8 @@ function PageActionMenu({ readOnly }: PageActionMenuProps) {
const handleCopyAsMarkdown = () => { const handleCopyAsMarkdown = () => {
if (!pageEditor) return; if (!pageEditor) return;
// Copy the page as canonical markdown through the shared converter (issue const html = pageEditor.getHTML();
// #347), so "Copy as markdown" matches the server export byte-for-byte. const markdown = htmlToMarkdown(html);
const markdown = convertProseMirrorToMarkdown(pageEditor.getJSON());
const title = page?.title ? `# ${page.title}\n\n` : ""; const title = page?.title ? `# ${page.title}\n\n` : "";
clipboard.copy(`${title}${markdown}`); clipboard.copy(`${title}${markdown}`);
notifications.show({ message: t("Copied") }); notifications.show({ message: t("Copied") });
@@ -0,0 +1,195 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { render, act, cleanup } from "@testing-library/react";
import { MemoryRouter, useNavigate } from "react-router-dom";
// Mocks for the dirty shell's side-effecting collaborators.
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
vi.mock("@/i18n.ts", () => ({ default: { t: (k: string) => k } }));
vi.mock("@/lib/reload-guard", () => ({
hasAutoReloaded: vi.fn(() => false),
markAutoReloaded: vi.fn(() => true),
recordReloadBreadcrumb: vi.fn(),
takeReloadBreadcrumb: vi.fn(() => null),
}));
import { notifications } from "@mantine/notifications";
import { hasAutoReloaded, markAutoReloaded } from "@/lib/reload-guard";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
__resetGuardedReloadForTests,
} from "./guarded-reload";
const show = notifications.show as unknown as ReturnType<typeof vi.fn>;
const mockHasAutoReloaded = hasAutoReloaded as unknown as ReturnType<
typeof vi.fn
>;
const mockMarkAutoReloaded = markAutoReloaded as unknown as ReturnType<
typeof vi.fn
>;
let reload: ReturnType<typeof vi.fn>;
let visibility: DocumentVisibilityState;
// Test harness mounted inside a router: it installs the navigation hook and
// exposes `navigate` so a test can drive an in-app router navigation.
let doNavigate: (to: string) => void;
function Harness() {
useVersionReloadOnNavigation();
const navigate = useNavigate();
doNavigate = navigate;
return null;
}
function mountHarness() {
render(
<MemoryRouter initialEntries={["/start"]}>
<Harness />
</MemoryRouter>,
);
}
function navigateTo(path: string) {
act(() => {
doNavigate(path);
});
}
beforeEach(() => {
__resetGuardedReloadForTests();
vi.clearAllMocks();
mockHasAutoReloaded.mockReturnValue(false);
mockMarkAutoReloaded.mockReturnValue(true);
vi.stubGlobal("APP_VERSION", "test-A");
reload = vi.fn();
Object.defineProperty(window, "location", {
configurable: true,
value: { reload },
});
visibility = "visible";
Object.defineProperty(document, "visibilityState", {
configurable: true,
get: () => visibility,
});
});
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
});
describe("triggerGuardedReload (variant C)", () => {
it("noop when versions match: no banner, no reload", () => {
triggerGuardedReload("test-A");
expect(show).not.toHaveBeenCalled();
expect(reload).not.toHaveBeenCalled();
});
it("noop when the server version is empty (fail-safe)", () => {
triggerGuardedReload("");
triggerGuardedReload(undefined);
expect(show).not.toHaveBeenCalled();
expect(reload).not.toHaveBeenCalled();
});
it("real mismatch shows the banner but does NOT reload immediately", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(show).toHaveBeenCalledTimes(1);
expect(show.mock.calls[0][0]).toMatchObject({
id: "app-version-reload",
autoClose: false,
withCloseButton: true,
});
expect(reload).not.toHaveBeenCalled();
});
it("reloads EXACTLY ONCE on the first in-app navigation after a mismatch", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
// A second navigation must NOT reload again (one-shot was consumed).
navigateTo("/again");
expect(reload).toHaveBeenCalledTimes(1);
});
it("does NOT reload on a 2nd mismatch after the first was armed/consumed (one-shot)", () => {
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
// Another app-version mismatch arrives (reconnect): must not re-arm.
triggerGuardedReload("test-C");
navigateTo("/again");
expect(reload).toHaveBeenCalledTimes(1);
});
it("does NOT reload merely from the tab going to the background", () => {
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
visibility = "hidden";
act(() => {
document.dispatchEvent(new Event("visibilitychange"));
});
expect(reload).not.toHaveBeenCalled();
});
it("a hidden-at-receipt tab is also NOT reloaded immediately (variant C uniform); reloads on next navigation", () => {
visibility = "hidden";
mountHarness();
triggerGuardedReload("test-B");
expect(reload).not.toHaveBeenCalled();
expect(show).toHaveBeenCalledTimes(1);
navigateTo("/next");
expect(reload).toHaveBeenCalledTimes(1);
});
it("the banner's Update button reloads immediately", () => {
triggerGuardedReload("test-B");
const message = show.mock.calls[0][0].message as {
props: { onClick: () => void };
};
message.props.onClick();
expect(reload).toHaveBeenCalledTimes(1);
});
it("banner-only (auto-reload already spent): banner, never auto-reload on navigation", () => {
mockHasAutoReloaded.mockReturnValue(true);
mountHarness();
triggerGuardedReload("test-B");
expect(show).toHaveBeenCalledTimes(1);
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
});
it("does NOT reload when the flag write fails; falls back to the banner", () => {
mockMarkAutoReloaded.mockReturnValue(false);
mountHarness();
triggerGuardedReload("test-B");
navigateTo("/next");
expect(reload).not.toHaveBeenCalled();
// performAutoReload falls back to showing the banner (initial + fallback).
expect(show).toHaveBeenCalled();
});
it("is idempotent within a tab-load: repeated emits do not stack banners", () => {
triggerGuardedReload("test-B");
triggerGuardedReload("test-B");
triggerGuardedReload("test-C");
expect(show).toHaveBeenCalledTimes(1);
});
});
@@ -0,0 +1,187 @@
import { useEffect, useRef } from "react";
import { useLocation } from "react-router-dom";
import { Button } from "@mantine/core";
import { notifications } from "@mantine/notifications";
import i18n from "@/i18n.ts";
import {
hasAutoReloaded,
markAutoReloaded,
recordReloadBreadcrumb,
takeReloadBreadcrumb,
} from "@/lib/reload-guard";
import { decideVersionAction } from "@/features/user/version-coherence";
// Dirty shell around the pure `decideVersionAction`: it reads globals
// (APP_VERSION), touches sessionStorage via the shared reload-guard, drives the
// Mantine notification, and arms the router-navigation reload hook. Kept
// separate from the pure module so the decision stays unit-testable without a
// DOM.
// One fixed id so repeated app-version signals (e.g. every reconnect) update a
// single banner instead of stacking a new one each time.
const BANNER_ID = "app-version-reload";
// Module-level idempotency for the current tab-load: once a mismatch has been
// handled we don't re-arm the navigation reload or re-show the banner on
// subsequent app-version emits.
let handled = false;
// Variant C: on a real mismatch we do NOT reload the tab when it merely goes to
// the background (that would silently drop a half-written comment/form). Instead
// we arm a one-shot reload for the NEXT in-app router navigation — a point where
// the user is already leaving the current page, so an in-app navigation would
// discard that unsaved component-state anyway and the reload adds no extra loss.
let pendingNavReload = false;
// Remembered from the last detected mismatch for the pre-reload breadcrumb and
// the (already-visible) banner.
let lastServerVersion = "";
let lastClientVersion = "";
// Read the build version baked into THIS bundle. The `typeof` guard avoids a
// ReferenceError where the `APP_VERSION` global is absent (e.g. under vitest,
// where Vite's `define` did not run) — an unknown client version makes the
// pure decision no-op (fail-safe).
function readClientVersion(): string {
return (typeof APP_VERSION !== "undefined" ? APP_VERSION : "").trim();
}
// Perform the actual reload — but only after the shared one-shot flag is
// persisted. If the write fails (storage unavailable) we must NOT reload
// (mirrors the reactive chunk-load boundary's `catch → return`), and fall back
// to the manual banner so the user can still recover.
function performAutoReload(): void {
if (!markAutoReloaded()) {
showReloadBanner();
return;
}
// Trace right before the reload (which clears the console): a persistent
// breadcrumb + a log line so the auto-reload is observable in a field report.
recordReloadBreadcrumb({
path: "proactive",
serverVersion: lastServerVersion,
clientVersion: lastClientVersion,
});
console.warn(
`[version-coherence] auto-reloading: client=${lastClientVersion} -> server=${lastServerVersion}`,
);
window.location.reload();
}
function showReloadBanner(): void {
notifications.show({
id: BANNER_ID,
title: i18n.t("A new version is available"),
message: (
<Button size="xs" mt="xs" onClick={() => performAutoReload()}>
{i18n.t("Update")}
</Button>
),
autoClose: false,
withCloseButton: true,
});
}
/**
* Handle a server `app-version` announcement: compare it to this bundle's
* version and, on a real mismatch, show the banner and arm a guarded reload for
* the next in-app navigation (variant C).
*
* - real mismatch (first this session) banner + arm navigation reload. The
* banner's "Update" button reloads immediately (same one-shot guard). The tab
* is NOT reloaded on visibility change.
* - auto-reload already used / storage error banner only (no arm), so there is
* at most one automatic reload per session (loop safety).
* - in sync / unknown version noop (fail-safe).
*/
export function triggerGuardedReload(
rawServerVersion: string | undefined | null,
): void {
const serverVersion = (rawServerVersion ?? "").trim();
const clientVersion = readClientVersion();
// A storage read error surfaces as autoReloadUsed=true → fail toward NOT
// reloading (banner only).
const autoReloadUsed = hasAutoReloaded();
const action = decideVersionAction({
serverVersion,
clientVersion,
autoReloadUsed,
});
if (action === "noop") return;
// Idempotent per tab-load: don't re-arm or re-stack the banner across repeated
// emits (reconnects) once we've already acted.
if (handled) return;
handled = true;
lastServerVersion = serverVersion;
lastClientVersion = clientVersion;
if (action === "banner") {
// Entered banner-only (permanent skew, node oscillation, or spent
// auto-reload). Log for diagnosability; show the manual banner.
console.warn(
`[version-coherence] server=${serverVersion} client=${clientVersion}: ` +
"auto-reload already spent this session — showing manual banner",
);
showReloadBanner();
return;
}
// action === "reload" (variant C): show the banner and defer the auto-reload
// to the next in-app navigation instead of reloading now / on visibility.
showReloadBanner();
pendingNavReload = true;
}
/**
* Consume the armed one-shot navigation reload, if any. Called by
* `useVersionReloadOnNavigation` on each in-app router navigation.
*/
export function consumeNavigationReload(): void {
if (!pendingNavReload) return;
pendingNavReload = false;
performAutoReload();
}
/**
* Hook (mounted inside the Router) that fires the armed one-shot reload on the
* NEXT in-app router navigation after a version mismatch. Skips the initial
* render so it only reacts to real navigations, not the first location.
*/
export function useVersionReloadOnNavigation(): void {
const location = useLocation();
const firstRender = useRef(true);
useEffect(() => {
if (firstRender.current) {
firstRender.current = false;
return;
}
consumeNavigationReload();
}, [location.key]);
}
/**
* Surface (log once) the breadcrumb left by an auto-reload in the previous page
* load the reload cleared the console, so this makes a "tab reloaded itself"
* report diagnosable. Call once on app startup.
*/
export function surfacePreviousReloadBreadcrumb(): void {
const crumb = takeReloadBreadcrumb();
if (!crumb) return;
console.info(
`[version-coherence] previous auto-reload: path=${crumb.path} ` +
`client=${crumb.clientVersion ?? ""} -> server=${crumb.serverVersion ?? ""} ` +
`at=${new Date(crumb.at).toISOString()}`,
);
}
// Test-only: reset module-level latches between cases.
export function __resetGuardedReloadForTests(): void {
handled = false;
pendingNavReload = false;
lastServerVersion = "";
lastClientVersion = "";
}
@@ -13,6 +13,12 @@ import { useCollabToken } from "@/features/auth/queries/auth-query.tsx";
import { Error404 } from "@/components/ui/error-404.tsx"; import { Error404 } from "@/components/ui/error-404.tsx";
import { queryClient } from "@/main.tsx"; import { queryClient } from "@/main.tsx";
import { makeConnectHandler } from "@/features/user/connect-resync.ts"; import { makeConnectHandler } from "@/features/user/connect-resync.ts";
import {
triggerGuardedReload,
useVersionReloadOnNavigation,
surfacePreviousReloadBreadcrumb,
} from "@/features/user/guarded-reload.tsx";
import type { AppVersionSocketPayload } from "@/features/user/version-coherence.ts";
export function UserProvider({ children }: React.PropsWithChildren) { export function UserProvider({ children }: React.PropsWithChildren) {
const [, setCurrentUser] = useAtom(currentUserAtom); const [, setCurrentUser] = useAtom(currentUserAtom);
@@ -22,6 +28,16 @@ export function UserProvider({ children }: React.PropsWithChildren) {
// fetch collab token on load // fetch collab token on load
const { data: collab } = useCollabToken(); const { data: collab } = useCollabToken();
// version-coherence: fire the armed one-shot reload on the next in-app
// navigation (variant C — a safe point, not on tab backgrounding).
useVersionReloadOnNavigation();
// Surface any breadcrumb left by an auto-reload in the previous page load
// (the reload cleared the console) so a field report stays diagnosable.
useEffect(() => {
surfacePreviousReloadBreadcrumb();
}, []);
useEffect(() => { useEffect(() => {
if (isLoading || isError) { if (isLoading || isError) {
return; return;
@@ -47,6 +63,16 @@ export function UserProvider({ children }: React.PropsWithChildren) {
handleConnect(); handleConnect();
}); });
// Register the version-coherence listener SYNCHRONOUSLY, before the socket
// connects: the server emits `app-version` immediately in handleConnection,
// so a listener attached after connect would miss it on a fast localhost
// connect. On a version mismatch the client shows a banner and defers the
// auto-reload to the next in-app navigation (variant C — avoids reloading a
// backgrounded tab that may hold unsaved input) before it hits a stale chunk.
newSocket.on("app-version", (payload?: AppVersionSocketPayload) => {
triggerGuardedReload(payload?.version);
});
return () => { return () => {
console.log("ws disconnected"); console.log("ws disconnected");
newSocket.disconnect(); newSocket.disconnect();
@@ -0,0 +1,64 @@
import { describe, it, expect } from "vitest";
import { decideVersionAction } from "./version-coherence";
describe("decideVersionAction", () => {
it("noop when the server version is empty (fail-safe)", () => {
expect(
decideVersionAction({
serverVersion: "",
clientVersion: "v1",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("noop when the client version is empty (fail-safe)", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("noop when versions are equal (in sync)", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "v1",
autoReloadUsed: false,
}),
).toBe("noop");
});
it("reload on a real mismatch the first time this session", () => {
expect(
decideVersionAction({
serverVersion: "test-B",
clientVersion: "test-A",
autoReloadUsed: false,
}),
).toBe("reload");
});
it("banner on a mismatch once the session auto-reload is spent", () => {
expect(
decideVersionAction({
serverVersion: "test-B",
clientVersion: "test-A",
autoReloadUsed: true,
}),
).toBe("banner");
});
it("equal versions stay noop even if auto-reload was already used", () => {
expect(
decideVersionAction({
serverVersion: "v1",
clientVersion: "v1",
autoReloadUsed: true,
}),
).toBe("noop");
});
});
@@ -0,0 +1,32 @@
// Payload of the per-connect `app-version` socket.io event announced by the
// server (ws.gateway.ts) after a successful auth. A dedicated event — NOT a
// member of the room-scoped `WebSocketEvent` union (which is discriminated by
// `operation`), so it never touches use-query-subscription.
export type AppVersionSocketPayload = { version: string };
/**
* Pure decision for the version-coherence guard.
*
* All inputs are injected (no globals, no side effects) so it is unit-testable
* without a DOM or the build-time `APP_VERSION` global (undefined under vitest).
*
* - `autoReloadUsed` = a session-wide automatic reload has already happened,
* so we must not auto-reload again (loop safety, shared with the reactive
* chunk-load boundary).
*
* Returns:
* - "noop" do nothing (unknown version on either side, or already in sync).
* - "banner" show the manual "update available" banner only (no auto-reload).
* - "reload" real first-time mismatch: eligible for a guarded auto-reload.
*/
export function decideVersionAction(args: {
serverVersion: string;
clientVersion: string;
autoReloadUsed: boolean;
}): "reload" | "banner" | "noop" {
const { serverVersion, clientVersion, autoReloadUsed } = args;
if (!serverVersion || !clientVersion) return "noop"; // fail-safe: unknown version → never act
if (serverVersion === clientVersion) return "noop"; // in sync
if (autoReloadUsed) return "banner"; // one auto-reload per session already spent
return "reload"; // real mismatch, first time this session
}
+97
View File
@@ -0,0 +1,97 @@
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import {
hasAutoReloaded,
markAutoReloaded,
recordReloadBreadcrumb,
takeReloadBreadcrumb,
} from "./reload-guard";
const FLAG = "chunk-reload-attempted";
describe("reload-guard", () => {
beforeEach(() => {
sessionStorage.clear();
vi.restoreAllMocks();
});
afterEach(() => {
sessionStorage.clear();
vi.restoreAllMocks();
});
it("hasAutoReloaded is false before any reload, true after mark", () => {
expect(hasAutoReloaded()).toBe(false);
expect(markAutoReloaded()).toBe(true);
expect(hasAutoReloaded()).toBe(true);
// Uses the same key the reactive chunk-load boundary reads.
expect(sessionStorage.getItem(FLAG)).toBe("1");
});
it("hasAutoReloaded returns true when reading storage throws (fail toward not reloading)", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => {
throw new Error("storage disabled");
},
setItem: () => {
throw new Error("storage disabled");
},
});
try {
expect(hasAutoReloaded()).toBe(true);
} finally {
vi.unstubAllGlobals();
}
});
it("markAutoReloaded returns false when writing storage throws", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("storage disabled");
},
});
try {
expect(markAutoReloaded()).toBe(false);
} finally {
vi.unstubAllGlobals();
}
});
it("records and then takes a breadcrumb once (cleared on read)", () => {
recordReloadBreadcrumb({
path: "proactive",
serverVersion: "test-B",
clientVersion: "test-A",
});
const crumb = takeReloadBreadcrumb();
expect(crumb).toMatchObject({
path: "proactive",
serverVersion: "test-B",
clientVersion: "test-A",
});
expect(typeof crumb?.at).toBe("number");
// Cleared on read → a second take returns null.
expect(takeReloadBreadcrumb()).toBeNull();
});
it("takeReloadBreadcrumb returns null when nothing was recorded", () => {
expect(takeReloadBreadcrumb()).toBeNull();
});
it("recordReloadBreadcrumb swallows a storage-write error (diagnostics only)", () => {
vi.stubGlobal("sessionStorage", {
getItem: () => null,
setItem: () => {
throw new Error("storage disabled");
},
removeItem: () => {},
});
try {
expect(() =>
recordReloadBreadcrumb({ path: "chunk-boundary" }),
).not.toThrow();
} finally {
vi.unstubAllGlobals();
}
});
});
+90
View File
@@ -0,0 +1,90 @@
// Shared one-shot auto-reload guard.
//
// Both auto-reload paths — the reactive chunk-load-error-boundary (recovers
// AFTER a stale lazy chunk 404s) and the proactive version-coherence feature
// (reloads BEFORE the tab hits a stale chunk) — go through these functions so
// they share ONE session-scoped flag. Net guarantee: at most a single
// automatic reload per browser session across both paths. Once the flag is set
// (or sessionStorage is unavailable), every further mismatch degrades to a
// manual banner/UI — no reload loop under permanent skew, node oscillation, or
// a disabled storage.
const RELOAD_FLAG = "chunk-reload-attempted";
/**
* Has an automatic reload already been performed (or attempted) this session?
*
* A storage read error (private mode / disabled) is reported as `true` so the
* caller fails toward NOT reloading an unguarded loop is worse than a stale
* tab the user can reload manually.
*/
export function hasAutoReloaded(): boolean {
try {
return sessionStorage.getItem(RELOAD_FLAG) !== null;
} catch {
return true;
}
}
/**
* Record that an automatic reload is being performed this session.
*
* Returns whether the write succeeded. A `false` return (storage unavailable)
* means the caller MUST NOT reload otherwise the flag would never stick and
* the reload could loop.
*/
export function markAutoReloaded(): boolean {
try {
sessionStorage.setItem(RELOAD_FLAG, "1");
return true;
} catch {
return false;
}
}
// Diagnostic breadcrumb for an automatic reload. Written right before
// window.location.reload() (which clears the console) and read back on the next
// page load, so a "the tab reloaded itself / it's looping" field report is
// diagnosable: which path fired (proactive version-coherence vs the reactive
// chunk-load boundary) and which version pair triggered it. sessionStorage
// survives a same-tab reload, unlike the console.
const RELOAD_BREADCRUMB_KEY = "reload-breadcrumb";
export type ReloadBreadcrumb = {
path: "proactive" | "chunk-boundary";
serverVersion?: string;
clientVersion?: string;
at: number;
};
/**
* Persist a best-effort breadcrumb just before an automatic reload. Failures
* (storage unavailable) are swallowed this is diagnostics only and must never
* block or alter the reload decision.
*/
export function recordReloadBreadcrumb(
entry: Omit<ReloadBreadcrumb, "at">,
): void {
try {
sessionStorage.setItem(
RELOAD_BREADCRUMB_KEY,
JSON.stringify({ ...entry, at: Date.now() }),
);
} catch {
// best-effort diagnostics only
}
}
/**
* Read and clear the breadcrumb left by an auto-reload in the previous page
* load. Cleared on read so it surfaces exactly once per reload.
*/
export function takeReloadBreadcrumb(): ReloadBreadcrumb | null {
try {
const raw = sessionStorage.getItem(RELOAD_BREADCRUMB_KEY);
if (!raw) return null;
sessionStorage.removeItem(RELOAD_BREADCRUMB_KEY);
return JSON.parse(raw) as ReloadBreadcrumb;
} catch {
return null;
}
}
+29 -2
View File
@@ -1,7 +1,8 @@
import { defineConfig, loadEnv } from "vite"; import { defineConfig, loadEnv, type Plugin } from "vite";
import react from "@vitejs/plugin-react"; import react from "@vitejs/plugin-react";
import { compression } from "vite-plugin-compression2"; import { compression } from "vite-plugin-compression2";
import * as path from "path"; import * as path from "path";
import * as fs from "node:fs";
import { execSync } from "node:child_process"; import { execSync } from "node:child_process";
const envPath = path.resolve(process.cwd(), "..", ".."); const envPath = path.resolve(process.cwd(), "..", "..");
@@ -24,7 +25,32 @@ function resolveAppVersion(cwd: string): string {
} }
} }
// Emit <outDir>/version.json = { "version": appVersion } so the server can read
// the exact same build id the bundle was compiled with. The value is the SAME
// `appVersion` fed into `define.APP_VERSION`, so version.json and the baked-in
// global are identical by construction — the single source of truth (no
// runtime-env second copy that could drift and cause a false version mismatch).
function versionJsonPlugin(version: string): Plugin {
let outDir = "dist";
return {
name: "emit-version-json",
apply: "build",
configResolved(config) {
outDir = config.build.outDir;
},
writeBundle() {
const root = path.resolve(process.cwd(), outDir);
fs.mkdirSync(root, { recursive: true });
fs.writeFileSync(
path.join(root, "version.json"),
JSON.stringify({ version }),
);
},
};
}
export default defineConfig(({ mode }) => { export default defineConfig(({ mode }) => {
const appVersion = resolveAppVersion(envPath);
const { const {
APP_URL, APP_URL,
FILE_UPLOAD_SIZE_LIMIT, FILE_UPLOAD_SIZE_LIMIT,
@@ -52,10 +78,11 @@ export default defineConfig(({ mode }) => {
POSTHOG_HOST, POSTHOG_HOST,
POSTHOG_KEY, POSTHOG_KEY,
}, },
APP_VERSION: JSON.stringify(resolveAppVersion(envPath)), APP_VERSION: JSON.stringify(appVersion),
}, },
plugins: [ plugins: [
react(), react(),
versionJsonPlugin(appVersion),
// Emit .br and .gz next to every built asset so the server can serve the // Emit .br and .gz next to every built asset so the server can serve the
// precompressed copy (see @fastify/static preCompressed in static.module.ts). // precompressed copy (see @fastify/static preCompressed in static.module.ts).
compression({ compression({
@@ -0,0 +1,52 @@
import { join } from 'path';
import * as fs from 'node:fs';
import * as os from 'node:os';
import { readClientBuildVersion } from './client-version';
describe('readClientBuildVersion', () => {
let dir: string;
beforeEach(() => {
dir = fs.mkdtempSync(join(os.tmpdir(), 'client-version-'));
});
afterEach(() => {
fs.rmSync(dir, { recursive: true, force: true });
});
const writeVersionJson = (content: string) =>
fs.writeFileSync(join(dir, 'version.json'), content);
it('returns the version from a valid version.json', () => {
writeVersionJson(JSON.stringify({ version: 'test-A' }));
expect(readClientBuildVersion(dir)).toBe('test-A');
});
it('trims surrounding whitespace in the version', () => {
writeVersionJson(JSON.stringify({ version: ' v1.2.3 ' }));
expect(readClientBuildVersion(dir)).toBe('v1.2.3');
});
it('returns "" when version.json is missing', () => {
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" on malformed JSON', () => {
writeVersionJson('{ not json');
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the version field is absent', () => {
writeVersionJson(JSON.stringify({ notVersion: 'x' }));
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the version field is not a string', () => {
writeVersionJson(JSON.stringify({ version: 123 }));
expect(readClientBuildVersion(dir)).toBe('');
});
it('returns "" when the path does not exist at all', () => {
expect(readClientBuildVersion(join(dir, 'nope'))).toBe('');
});
});
@@ -0,0 +1,36 @@
import { join } from 'path';
import * as fs from 'node:fs';
/**
* Resolve the absolute path to the built client bundle directory
* (`apps/client/dist`) shipped into the runtime image.
*
* The `../` depth is anchored on THIS module's compiled location
* (`dist/common/helpers`). `integrations/static` sits at the same depth under
* the compiled root, so both callers (StaticModule and readClientBuildVersion)
* MUST share this single helper rather than duplicating the depth a copy in a
* module at a different depth would silently resolve to the wrong directory.
*/
export function resolveClientDistPath(): string {
return join(__dirname, '..', '..', '..', '..', 'client/dist');
}
/**
* Read the build version the client bundle was compiled with, from
* `<clientDistPath>/version.json` (written by the Vite build the single
* source of truth shared by the baked-in `APP_VERSION` global and this file).
*
* Fail-safe: any error (missing file, unreadable, bad JSON, non-string
* version) yields `''`. The caller treats an empty version as "unknown" and
* the whole version-coherence feature stays silently inert existing deploys
* without the file keep working unchanged.
*/
export function readClientBuildVersion(clientDistPath: string): string {
try {
const raw = fs.readFileSync(join(clientDistPath, 'version.json'), 'utf8');
const version = (JSON.parse(raw) as { version?: unknown }).version;
return typeof version === 'string' ? version.trim() : '';
} catch {
return '';
}
}
+1
View File
@@ -3,3 +3,4 @@ export * from './nanoid.utils';
export * from './file.helper'; export * from './file.helper';
export * from './constants'; export * from './constants';
export * from './security-headers'; export * from './security-headers';
export * from './client-version';
@@ -1,5 +1,4 @@
import { markdownToProseMirror } from '@docmost/prosemirror-markdown'; import { markdownToHtml, encodeHtmlEmbedSource } from '@docmost/editor-ext';
import { encodeHtmlEmbedSource } from '@docmost/editor-ext';
import { htmlToJson } from '../../../collaboration/collaboration.util'; import { htmlToJson } from '../../../collaboration/collaboration.util';
import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util'; import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
@@ -11,12 +10,13 @@ import { hasHtmlEmbedNode, stripHtmlEmbedNodes } from './html-embed.util';
* *
* The block renders inside a sandboxed iframe, so this is not an XSS surface; * The block renders inside a sandboxed iframe, so this is not an XSS surface;
* this exercises the REAL server import conversion path that ImportService uses * this exercises the REAL server import conversion path that ImportService uses
* (`markdownToProseMirror`, the canonical converter issue #345/#347) and * (`markdownToHtml` then `htmlToJson`; `processHTML` adds only a cheerio
* asserts that such a node is DETECTED and STRIPPABLE so the share read path's * link/iframe normalize pass which does not touch htmlEmbed divs) and asserts
* that such a node is DETECTED and STRIPPABLE so the share read path's
* master-toggle strip can remove it when the workspace toggle is OFF. * master-toggle strip can remove it when the workspace toggle is OFF.
*/ */
describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTML', () => { describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTML', () => {
it('round-trips through markdownToProseMirror and is DETECTED (base64 data-source)', async () => { it('round-trips through markdownToHtml -> htmlToJson and is DETECTED (base64 data-source)', async () => {
const source = '<script>steal()</script>'; const source = '<script>steal()</script>';
const encoded = encodeHtmlEmbedSource(source); const encoded = encodeHtmlEmbedSource(source);
const md = [ const md = [
@@ -27,9 +27,12 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
'World', 'World',
].join('\n'); ].join('\n');
// The canonical importer parses the raw block-level div into a real const html = await markdownToHtml(md);
// htmlEmbed node carrying the decoded source. // marked preserves the raw block-level div verbatim.
const json = await markdownToProseMirror(md); expect(html).toContain('data-type="htmlEmbed"');
const json = htmlToJson(html);
// The div parses into a real htmlEmbed node carrying the decoded source.
expect(hasHtmlEmbedNode(json)).toBe(true); expect(hasHtmlEmbedNode(json)).toBe(true);
// Because it is detected, the share master-toggle strip can remove it. // Because it is detected, the share master-toggle strip can remove it.
@@ -56,7 +59,8 @@ describe('htmlEmbed smuggled via the raw serialized div in imported markdown/HTM
// therefore stripping) does not depend on the source being well-formed, so // therefore stripping) does not depend on the source being well-formed, so
// the bypass cannot be hidden by sending a malformed data-source. // the bypass cannot be hidden by sending a malformed data-source.
const md = `<div data-type="htmlEmbed" data-source="&lt;script&gt;x&lt;/script&gt;"></div>`; const md = `<div data-type="htmlEmbed" data-source="&lt;script&gt;x&lt;/script&gt;"></div>`;
const json = await markdownToProseMirror(md); const html = await markdownToHtml(md);
const json = htmlToJson(html);
expect(hasHtmlEmbedNode(json)).toBe(true); expect(hasHtmlEmbedNode(json)).toBe(true);
expect(hasHtmlEmbedNode(stripHtmlEmbedNodes(json))).toBe(false); expect(hasHtmlEmbedNode(stripHtmlEmbedNodes(json))).toBe(false);
}); });
@@ -238,9 +238,8 @@ function convertReferenceFootnotes(markdown: string): string {
* *
* LINE-ANCHORED (the same shape the canonical parser uses in * LINE-ANCHORED (the same shape the canonical parser uses in
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the * prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
* very start and closes only on a `\n---` line. The retired editor-ext * very start and closes only on a `\n---` line. The retired `markdownToHtml`
* `markdownToHtml` front-matter strip (removed in #347) closed on the FIRST * strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
* `---` ANYWHERE (an unanchored close), so a value
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter * containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
* and leaked the rest into the body. An optional leading BOM is tolerated. * and leaked the rest into the body. An optional leading BOM is tolerated.
*/ */
@@ -4,6 +4,7 @@ import { join } from 'path';
import * as fs from 'node:fs'; import * as fs from 'node:fs';
import fastifyStatic from '@fastify/static'; import fastifyStatic from '@fastify/static';
import { EnvironmentService } from '../environment/environment.service'; import { EnvironmentService } from '../environment/environment.service';
import { resolveClientDistPath } from '../../common/helpers/client-version';
/** /**
* Resolve the response headers for a statically served client asset. * Resolve the response headers for a statically served client asset.
@@ -56,14 +57,7 @@ export class StaticModule implements OnModuleInit {
const httpAdapter = this.httpAdapterHost.httpAdapter; const httpAdapter = this.httpAdapterHost.httpAdapter;
const app = httpAdapter.getInstance(); const app = httpAdapter.getInstance();
const clientDistPath = join( const clientDistPath = resolveClientDistPath();
__dirname,
'..',
'..',
'..',
'..',
'client/dist',
);
const indexFilePath = join(clientDistPath, 'index.html'); const indexFilePath = join(clientDistPath, 'index.html');
+37 -2
View File
@@ -9,10 +9,14 @@ import {
import { Server, Socket } from 'socket.io'; import { Server, Socket } from 'socket.io';
import { TokenService } from '../core/auth/services/token.service'; import { TokenService } from '../core/auth/services/token.service';
import { JwtPayload, JwtType } from '../core/auth/dto/jwt-payload'; import { JwtPayload, JwtType } from '../core/auth/dto/jwt-payload';
import { OnModuleDestroy } from '@nestjs/common'; import { Logger, OnModuleDestroy, OnModuleInit } from '@nestjs/common';
import { SpaceMemberRepo } from '@docmost/db/repos/space/space-member.repo'; import { SpaceMemberRepo } from '@docmost/db/repos/space/space-member.repo';
import { WsService } from './ws.service'; import { WsService } from './ws.service';
import { getSpaceRoomName, getUserRoomName } from './ws.utils'; import { getSpaceRoomName, getUserRoomName } from './ws.utils';
import {
readClientBuildVersion,
resolveClientDistPath,
} from '../common/helpers/client-version';
import * as cookie from 'cookie'; import * as cookie from 'cookie';
@WebSocketGateway({ @WebSocketGateway({
@@ -20,17 +24,40 @@ import * as cookie from 'cookie';
transports: ['websocket'], transports: ['websocket'],
}) })
export class WsGateway export class WsGateway
implements OnGatewayConnection, OnGatewayInit, OnModuleDestroy implements
OnGatewayConnection,
OnGatewayInit,
OnModuleInit,
OnModuleDestroy
{ {
@WebSocketServer() @WebSocketServer()
server: Server; server: Server;
private readonly logger = new Logger(WsGateway.name);
// The build version of the client bundle shipped in this image, read once at
// startup from client/dist/version.json (single source of truth, same value
// baked into the client's APP_VERSION). Empty string => version.json missing
// or empty => the proactive version-coherence reload feature stays inert.
private appVersion = '';
constructor( constructor(
private tokenService: TokenService, private tokenService: TokenService,
private spaceMemberRepo: SpaceMemberRepo, private spaceMemberRepo: SpaceMemberRepo,
private wsService: WsService, private wsService: WsService,
) {} ) {}
onModuleInit(): void {
this.appVersion = readClientBuildVersion(resolveClientDistPath());
if (this.appVersion) {
this.logger.log(`app-version reload: ACTIVE (v=${this.appVersion})`);
} else {
this.logger.log(
'app-version reload: DISABLED (version.json missing/empty)',
);
}
}
afterInit(server: Server): void { afterInit(server: Server): void {
this.wsService.setServer(server); this.wsService.setServer(server);
} }
@@ -55,6 +82,14 @@ export class WsGateway
const spaceRooms = userSpaceIds.map((id) => getSpaceRoomName(id)); const spaceRooms = userSpaceIds.map((id) => getSpaceRoomName(id));
client.join([userRoom, workspaceRoom, ...spaceRooms]); client.join([userRoom, workspaceRoom, ...spaceRooms]);
// Announce this container's client build version to the freshly
// authenticated socket. On a redeploy the client reconnects to the new
// container and receives the new version here, letting it guard-reload
// before it hits a stale lazy chunk. Per-connect only (no broadcast):
// natural reconnect covers both single-container and cluster without a
// thundering-herd fleet reload.
client.emit('app-version', { version: this.appVersion });
} catch (err) { } catch (err) {
client.emit('Unauthorized'); client.emit('Unauthorized');
client.disconnect(); client.disconnect();
+3
View File
@@ -11,6 +11,9 @@
"main": "dist/index.js", "main": "dist/index.js",
"module": "./src/index.ts", "module": "./src/index.ts",
"types": "dist/index.d.ts", "types": "dist/index.d.ts",
"dependencies": {
"marked": "17.0.5"
},
"devDependencies": { "devDependencies": {
"@vitest/coverage-v8": "4.1.6", "@vitest/coverage-v8": "4.1.6",
"vitest": "4.1.6" "vitest": "4.1.6"
+1
View File
@@ -18,6 +18,7 @@ export * from "./lib/excalidraw";
export * from "./lib/embed"; export * from "./lib/embed";
export * from "./lib/html-embed/html-embed"; export * from "./lib/html-embed/html-embed";
export * from "./lib/mention"; export * from "./lib/mention";
export * from "./lib/markdown";
export * from "./lib/search-and-replace"; export * from "./lib/search-and-replace";
export * from "./lib/embed-provider"; export * from "./lib/embed-provider";
export * from "./lib/subpages"; export * from "./lib/subpages";
@@ -14,8 +14,7 @@ import {
* ProseMirror JSON directly (never running the editor's plugins), so the * ProseMirror JSON directly (never running the editor's plugins), so the
* canonical footnote topology was never enforced on those writes. The consumers * canonical footnote topology was never enforced on those writes. The consumers
* of this editor-ext copy are: the server markdown/HTML import * of this editor-ext copy are: the server markdown/HTML import
* (`markdownToProseMirror` from @docmost/prosemirror-markdown in import.service / * (`markdownToHtml -> htmlToJson` in import.service / file-import-task.service),
* file-import-task.service),
* `PageService` create/update (`parseProsemirrorContent` for the JSON/markdown/ * `PageService` create/update (`parseProsemirrorContent` for the JSON/markdown/
* HTML REST write paths), and the client markdown PASTE path * HTML REST write paths), and the client markdown PASTE path
* (`markdown-clipboard.ts`). (The MCP package mirrors this canonicalizer in * (`markdown-clipboard.ts`). (The MCP package mirrors this canonicalizer in
@@ -0,0 +1,131 @@
import { describe, it, expect } from "vitest";
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
import { markdownToHtml } from "../markdown/utils/marked.utils";
import { extractFootnoteDefinitions } from "../markdown/utils/footnote.marked";
// HTML the editor-ext nodes render (sup[data-footnote-ref], section/div).
const HTML =
`<p>Water<sup data-footnote-ref data-id="fn1"></sup> and clay<sup data-footnote-ref data-id="fn2"></sup>.</p>` +
`<section data-footnotes>` +
`<div data-footnote-def data-id="fn1"><p>First note.</p></div>` +
`<div data-footnote-def data-id="fn2"><p>Second note.</p></div>` +
`</section>`;
describe("footnote markdown round-trip", () => {
it("HTML -> Markdown produces pandoc footnote syntax", () => {
const md = htmlToMarkdown(HTML);
expect(md).toContain("[^fn1]");
expect(md).toContain("[^fn2]");
expect(md).toContain("[^fn1]: First note.");
expect(md).toContain("[^fn2]: Second note.");
});
it("Markdown -> HTML rebuilds the footnote nodes' HTML", async () => {
const md = htmlToMarkdown(HTML);
const html = await markdownToHtml(md);
expect(html).toContain('data-footnote-ref data-id="fn1"');
expect(html).toContain('data-footnote-ref data-id="fn2"');
expect(html).toContain("data-footnotes");
expect(html).toContain('data-footnote-def data-id="fn1"');
expect(html).toContain("First note.");
expect(html).toContain("Second note.");
});
it("preserves a [^id]: line shown inside a fenced code block (not a definition)", async () => {
// A document that DOCUMENTS footnote syntax inside a code fence. The
// `[^demo]: ...` line is example text, not a real definition, and must
// survive the Markdown -> HTML conversion verbatim.
const md = [
"Here is how footnotes look:",
"",
"```markdown",
"Some text[^demo]",
"",
"[^demo]: this is the definition",
"```",
"",
"End of doc.",
].join("\n");
const html = await markdownToHtml(md);
// The example definition line is kept inside the rendered code block.
expect(html).toContain("[^demo]: this is the definition");
// It did NOT get pulled out into a real footnotes section.
expect(html).not.toContain("data-footnotes");
expect(html).not.toContain("data-footnote-def");
});
it("extractFootnoteDefinitions keeps the FIRST duplicate definition and reuses markers", () => {
// Two definitions share id `d`, and the body has two `[^d]` markers. Under
// the import model (#166) duplicate definition ids are FIRST-WINS: only the
// first definition is kept; markers are NEVER rewritten, so the two `[^d]`
// references reuse the single footnote.
const md = [
"See here[^d] and there[^d].",
"",
"[^d]: first",
"[^d]: second",
].join("\n");
const { body, section } = extractFootnoteDefinitions(md);
const defIds = Array.from(
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
expect(defIds).toEqual(["d"]); // first-wins: one definition
expect(section).toContain("first");
expect(section).not.toContain("second"); // duplicate dropped
// Both markers stay `[^d]` (reuse) — no `d__2` minting.
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
(m) => m[1],
);
expect(refIds).toEqual(["d", "d"]);
});
it("extractFootnoteDefinitions is DETERMINISTIC and stable (same input -> same output)", () => {
// The output must be a pure function of the input markdown so importing the
// same source twice (or via the editor and the MCP mirror) is identical.
const md = [
"See[^d] one[^d] two[^d].",
"",
"[^d]: first",
"[^d]: second",
"[^d]: third",
].join("\n");
const run = () => {
const { body, section } = extractFootnoteDefinitions(md);
const defIds = Array.from(
section.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
const refIds = Array.from(body.matchAll(/\[\^([^\]\s]+)\]/g)).map(
(m) => m[1],
);
return { defIds, refIds };
};
const a = run();
const b = run();
expect(a).toEqual(b);
// First-wins: one kept definition `d`; all three reuse markers stay `d`.
expect(a.defIds).toEqual(["d"]);
expect(a.refIds).toEqual(["d", "d", "d"]);
});
it("markdownToHtml with a reused id renders ONE shared footnote def", async () => {
const md = [
"See here[^d] and there[^d].",
"",
"[^d]: first",
"[^d]: second",
].join("\n");
const html = await markdownToHtml(md);
const defIds = Array.from(
html.matchAll(/data-footnote-def data-id="([^"]+)"/g),
).map((m) => m[1]);
expect(defIds).toEqual(["d"]); // one shared definition
expect(html).toContain("first");
expect(html).not.toContain("second");
});
});
@@ -103,9 +103,8 @@ interface CollisionPlan {
* `X__2`, `X__3`, collision-bumped) so it survives as a distinct footnote which, * `X__2`, `X__3`, collision-bumped) so it survives as a distinct footnote which,
* having no matching reference, then falls under the normal orphan policy. It is * having no matching reference, then falls under the normal orphan policy. It is
* only ever dropped for lacking a reference, never for colliding. The IMPORT * only ever dropped for lacking a reference, never for colliding. The IMPORT
* paths (@docmost/prosemirror-markdown / MCP extractFootnotes) instead apply * paths (footnote.marked.ts / MCP extractFootnotes) instead apply first-wins +
* first-wins + drop + warn for duplicate definitions; that divergence is * drop + warn for duplicate definitions; that divergence is intentional import
* intentional import
* is an agent-authored artifact we sanitize, the editor is live user data we must * is an agent-authored artifact we sanitize, the editor is live user data we must
* not lose. * not lose.
* *
@@ -6,9 +6,8 @@ import { deriveFootnoteId } from "./footnote-util";
* *
* `deriveFootnoteId` lives ONLY in editor-ext now it is used by * `deriveFootnoteId` lives ONLY in editor-ext now it is used by
* `resolveCollisions` (re-id of a duplicate definition) and `footnotePastePlugin` * `resolveCollisions` (re-id of a duplicate definition) and `footnotePastePlugin`
* (re-id of a pasted colliding definition). The MCP / @docmost/prosemirror-markdown * (re-id of a pasted colliding definition). The MCP/marked import paths no longer
* import paths no longer derive ids (duplicate definitions there are * derive ids (duplicate definitions there are first-wins-dropped, #166), so there
* first-wins-dropped, #166), so there
* is no cross-package copy and no parity test to keep in sync. This table pins the * is no cross-package copy and no parity test to keep in sync. This table pins the
* deterministic scheme so a future change to it is a conscious one. * deterministic scheme so a future change to it is a conscious one.
*/ */
@@ -63,9 +63,8 @@ export function generateFootnoteId(): string {
* its own seen-set before requesting the next derived id. * its own seen-set before requesting the next derived id.
* *
* Used only inside editor-ext now (resolveCollisions for a re-id'd duplicate * Used only inside editor-ext now (resolveCollisions for a re-id'd duplicate
* DEFINITION, and footnotePastePlugin). The MCP / @docmost/prosemirror-markdown * DEFINITION, and footnotePastePlugin). The MCP/marked import paths no longer
* import paths no longer derive ids duplicate definitions there are * derive ids duplicate definitions there are first-wins-dropped (#166) so
* first-wins-dropped (#166) so
* there is no cross-package copy to keep in sync. The golden table in * there is no cross-package copy to keep in sync. The golden table in
* footnote-util.derive-id.test.ts pins the scheme. * footnote-util.derive-id.test.ts pins the scheme.
*/ */
@@ -0,0 +1,68 @@
import { describe, it, expect } from "vitest";
import { generateJSON } from "@tiptap/html";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { htmlToMarkdown } from "../markdown/utils/turndown.utils";
import { markdownToHtml } from "../markdown/utils/marked.utils";
import { TiptapImage } from "./image";
// Minimal schema for parsing markdownToHtml output back to JSON (mirrors
// image.spec.ts), so we can assert the recovered caption EXACTLY.
const parseExtensions = [Document, Paragraph, Text, TiptapImage];
// Lossless markdown round-trip for image captions (issue #221). An image WITH a
// caption can't be expressed as `![alt](src)`, so it is emitted as a raw <img>
// (carrying data-caption) wrapped in a block <div>, the same trick the <video>
// rule uses. marked passes the raw HTML through, so markdownToHtml keeps the
// data-caption, and the image extension's parseHTML restores the attribute.
describe("image caption markdown round-trip", () => {
it("HTML -> Markdown emits a raw <img data-caption> for captioned images", () => {
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
const md = htmlToMarkdown(html);
expect(md).toContain("data-caption=\"A grey cat\"");
expect(md).toContain('src="/files/a.png"');
expect(md).toContain('alt="cat"');
// It must NOT degrade to the lossy ![]() form.
expect(md).not.toContain("![cat]");
});
it("Markdown -> HTML restores data-caption on the <img>", async () => {
const html = `<p><img src="/files/a.png" alt="cat" data-caption="A grey cat"></p>`;
const md = htmlToMarkdown(html);
const back = await markdownToHtml(md);
expect(back).toContain('data-caption="A grey cat"');
expect(back).toContain('src="/files/a.png"');
});
it("special characters in the caption survive the round-trip (escaped)", async () => {
// The source caption is the decoded string `Tom & "Jerry"` (both an `&` and
// a `"`). escapeHtmlAttr must encode `&` -> `&amp;` and `"` -> `&quot;`.
const html = `<p><img src="/files/a.png" data-caption='Tom &amp; &quot;Jerry&quot;'></p>`;
const md = htmlToMarkdown(html);
// (a) The intermediate Markdown must carry the EXACT escaped attribute. This
// fails if escapeHtmlAttr stopped escaping `"` (attribute break-out:
// data-caption="Tom & "Jerry"") or double-encoded `&` (`&amp;amp;`).
expect(md).toContain('data-caption="Tom &amp; &quot;Jerry&quot;"');
const back = await markdownToHtml(md);
expect(back).toContain("data-caption=");
expect(back).toContain("Jerry");
expect(back).toContain("Tom");
// (b) Re-parse the rendered HTML through the image extension's parseHTML and
// assert the recovered caption is EXACTLY the original (no corruption, loss,
// or double-encoding).
const json = generateJSON(back, parseExtensions);
expect(json.content?.[0]?.attrs?.caption).toBe('Tom & "Jerry"');
});
it("caption-less images stay a clean ![alt](src) with no raw HTML", () => {
const html = `<p><img src="/files/a.png" alt="cat"></p>`;
const md = htmlToMarkdown(html);
expect(md).toContain("![cat](/files/a.png)");
expect(md).not.toContain("data-caption");
expect(md).not.toContain("<img");
});
});
@@ -0,0 +1,105 @@
import { describe, expect, it } from "vitest";
import { htmlEmbedExtension } from "./utils/html-embed.marked";
import { markdownToHtml } from "./index";
import { encodeHtmlEmbedSource } from "../html-embed/html-embed";
// CONTRACT tests for the marked block tokenizer that rebuilds an htmlEmbed node
// from the `<!--html-embed:BASE64-->` marker (html-embed.marked.ts), plus the
// observable round-trip through markdownToHtml.
//
// These pin the REAL tokenizer behaviour the import path depends on:
// - the tokenizer rule is anchored (^) and only accepts the base64 alphabet
// [A-Za-z0-9+/=], so a marker with non-base64 chars is NOT tokenized and
// survives as a literal HTML comment (not silently turned into something the
// server's strip no longer recognizes);
// - start() reports the correct index of the next marker so marked invokes the
// tokenizer at the right offset when a marker sits mid-document / after text;
// - a marker with surrounding text on the SAME line is split out into its own
// embed div while the surrounding text becomes ordinary paragraphs.
//
// The contract is asserted against the actual exported extension and pipeline —
// no behaviour is invented; the expectations were read off the real tokenizer.
const SAMPLE = "<b>x</b>";
const ENC = encodeHtmlEmbedSource(SAMPLE);
describe("htmlEmbed marked tokenizer — start()", () => {
it("returns the index of a marker that sits mid-document", () => {
const src = `hello world <!--html-embed:${ENC}-->`;
expect(htmlEmbedExtension.start(src)).toBe(src.indexOf("<!--html-embed:"));
});
it("returns 0 when the marker is at the very start", () => {
expect(htmlEmbedExtension.start(`<!--html-embed:${ENC}-->`)).toBe(0);
});
it("returns -1 when there is no marker", () => {
expect(htmlEmbedExtension.start("no marker here")).toBe(-1);
});
});
describe("htmlEmbed marked tokenizer — tokenizer()", () => {
it("tokenizes a marker at the start of the input, capturing the base64 payload", () => {
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`);
expect(token).toBeTruthy();
expect(token!.type).toBe("htmlEmbed");
expect(token!.raw).toBe(`<!--html-embed:${ENC}-->`);
expect(token!.encoded).toBe(ENC);
});
it("tokenizes an EMPTY marker (the [A-Za-z0-9+/=]* class allows zero chars)", () => {
const token = htmlEmbedExtension.tokenizer("<!--html-embed:-->");
expect(token).toBeTruthy();
expect(token!.encoded).toBe("");
expect(token!.raw).toBe("<!--html-embed:-->");
});
it("does NOT tokenize when text precedes the marker (rule is anchored ^)", () => {
// marked relies on start() to advance to the marker; the tokenizer itself
// only matches at offset 0, so a non-anchored call returns undefined.
expect(
htmlEmbedExtension.tokenizer(`hello <!--html-embed:${ENC}-->`),
).toBeUndefined();
});
it("does NOT tokenize a marker containing a non-base64 char ('$')", () => {
expect(
htmlEmbedExtension.tokenizer("<!--html-embed:ab$cd-->"),
).toBeUndefined();
});
it("does NOT tokenize a marker containing a space", () => {
expect(
htmlEmbedExtension.tokenizer("<!--html-embed:ab cd-->"),
).toBeUndefined();
});
it("renderer emits the embed div the node's parseHTML recognizes", () => {
const token = htmlEmbedExtension.tokenizer(`<!--html-embed:${ENC}-->`)!;
const html = htmlEmbedExtension.renderer(token as any);
expect(html).toBe(
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
);
});
});
describe("htmlEmbed marked tokenizer — markdownToHtml round-trip", () => {
it("splits a marker out of surrounding same-line text into its own embed div", async () => {
const html = await markdownToHtml(`before <!--html-embed:${ENC}--> after`);
// The marker became the embed div...
expect(html).toContain(
`<div data-type="htmlEmbed" data-source="${ENC}"></div>`,
);
// ...and the surrounding text survived as ordinary paragraph content.
expect(html).toContain("before");
expect(html).toContain("after");
});
it("leaves a marker with non-base64 chars as a literal comment (NOT an embed div)", async () => {
const html = await markdownToHtml("<!--html-embed:ab$cd-->");
// It is NOT tokenized into an embed div the server would strip...
expect(html).not.toContain('data-type="htmlEmbed"');
// ...it passes through unchanged as a literal HTML comment.
expect(html).toContain("<!--html-embed:ab$cd-->");
});
});
@@ -0,0 +1,2 @@
export * from "./utils/marked.utils";
export * from "./utils/turndown.utils";
@@ -0,0 +1,112 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml, htmlToMarkdown } from "./index";
import {
encodeHtmlEmbedSource,
decodeHtmlEmbedSource,
} from "../html-embed/html-embed";
// SECURITY (Variant C admin gate, import attack surface).
//
// The markdown import path is the only write path where an htmlEmbed reaches
// the server purely from file bytes (no editor / collab socket). The marked
// tokenizer in `html-embed.marked.ts` and the turndown rule in
// `turndown.utils.ts` are what materialize the `<!--html-embed:BASE64-->`
// marker into the `<div data-type="htmlEmbed" data-source="BASE64">` element
// that the server then parses into an htmlEmbed node and the admin gate strips.
//
// If either the tokenizer regex or the turndown rule shape drifts, the marker
// would either (a) stop becoming an htmlEmbed node (silently dropping admin
// content) or (b) become some OTHER tag the server's `hasHtmlEmbedNode` no
// longer recognizes (a strip bypass). These tests pin the marker <-> embed-div
// contract that the server-side strip relies on. editor-ext had ZERO tests
// before this file; this adds the runner + the round-trip coverage.
// The server parses the embed div by matching `data-type="htmlEmbed"` and
// decoding `data-source`; mirror that here so the assertion is exactly what the
// real `htmlToJson` -> htmlEmbed node parse depends on (the node's parseHTML in
// html-embed.ts uses the same selector + decodeHtmlEmbedSource).
const EMBED_DIV_RE = /<div[^>]*\bdata-type="htmlEmbed"[^>]*>/;
function extractEmbedSource(html: string): string | undefined {
const div = EMBED_DIV_RE.exec(html);
if (!div) return undefined;
const enc = /data-source="([^"]*)"/.exec(div[0]);
if (!enc) return undefined;
return decodeHtmlEmbedSource(enc[1]);
}
// Replicates the server's `hasHtmlEmbedNode` decision against the embed *div*
// (the HTML form the server immediately converts to JSON). If this matches, the
// server's JSON-level `hasHtmlEmbedNode` will too, because htmlToJson maps this
// exact div to an htmlEmbed node.
function htmlHasHtmlEmbed(html: string): boolean {
return EMBED_DIV_RE.test(html);
}
describe("markdown <!--html-embed--> import round-trip", () => {
const source = "<script>x</script>";
it("markdownToHtml turns the marker into an htmlEmbed div carrying the source", async () => {
const md = "<!--html-embed:" + encodeHtmlEmbedSource(source) + "-->";
const html = await markdownToHtml(md);
// The marker became the embed div the server recognizes as an htmlEmbed
// node (so the server's hasHtmlEmbedNode would match it after htmlToJson).
expect(htmlHasHtmlEmbed(html)).toBe(true);
// The decoded source is the original script, intact.
expect(extractEmbedSource(html)).toBe(source);
// The raw script is NOT inlined into the HTML — it stays base64 in the
// attribute (the marker itself must not be a direct injection vector).
expect(html).not.toContain("<script>x</script>");
});
it("preserves UTF-8 / special chars in the embedded source", async () => {
const utf8 = '<script>console.log("héllo → 世界")</script>';
const md = "<!--html-embed:" + encodeHtmlEmbedSource(utf8) + "-->";
const html = await markdownToHtml(md);
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe(utf8);
});
it("an empty marker still produces an htmlEmbed div (empty source)", async () => {
const html = await markdownToHtml("<!--html-embed:-->");
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe("");
});
it("round-trips htmlToMarkdown -> markdownToHtml preserving the embed marker", async () => {
const encoded = encodeHtmlEmbedSource(source);
// NOTE: turndown drops a *blank* (childless) element before any custom rule
// runs, and the htmlEmbed div is normally childless. The export pipeline
// therefore must give the rule a non-blank div to fire on; we add an inert
// text child here to exercise the real turndown htmlEmbed rule. (A blank
// embed div serializing to "" is asserted separately below as a documented
// edge so this contract drift is visible.)
const startHtml = `<div data-type="htmlEmbed" data-source="${encoded}">x</div>`;
// Export to markdown: the turndown rule emits the <!--html-embed:..-->
// marker (lossless, inert in plain markdown viewers).
const md = htmlToMarkdown(startHtml);
expect(md).toContain("<!--html-embed:" + encoded + "-->");
// Re-import: the marker round-trips back into an embed div with the same
// decoded source — this is the marker <-> embed-div contract the server's
// import strip depends on.
const html = await markdownToHtml(md);
expect(htmlHasHtmlEmbed(html)).toBe(true);
expect(extractEmbedSource(html)).toBe(source);
});
it("documents that a BLANK embed div serializes to empty markdown (turndown drops childless blocks)", () => {
const encoded = encodeHtmlEmbedSource(source);
const blank = `<div data-type="htmlEmbed" data-source="${encoded}"></div>`;
// This pins current behavior so a future change to the turndown rule (e.g.
// making it fire on blank nodes) is caught rather than silently shipping.
expect(htmlToMarkdown(blank)).toBe("");
});
it("the base64 codec itself round-trips (no '<' leaks into the attribute)", () => {
const encoded = encodeHtmlEmbedSource(source);
expect(encoded).not.toContain("<");
expect(decodeHtmlEmbedSource(encoded)).toBe(source);
});
});
@@ -0,0 +1,29 @@
/**
* Flexible `basename` implementation for node and the browser
* @see https://stackoverflow.com/a/59907288/2228771
*/
export function getBasename(path: string) {
// make sure the basename is not empty, if string ends with separator
let end = path.length - 1;
while (path[end] === '/' || path[end] === '\\') {
--end;
}
// support mixing of Win + Unix path separators
const i1 = path.lastIndexOf('/', end);
const i2 = path.lastIndexOf('\\', end);
let start: number;
if (i1 === -1) {
if (i2 === -1) {
// no separator in the whole thing
return path;
}
start = i2;
} else if (i2 === -1) {
start = i1;
} else {
start = Math.max(i1, i2);
}
return path.substring(start + 1, end + 1);
}
@@ -0,0 +1,33 @@
/**
* Shared pieces for the two callout tokenizers `callout.marked.ts` (the
* `:::type` fenced form) and `github-callout.marked.ts` (the `> [!type]` GitHub
* alert form). Both emit the SAME callout node, so the banner type dictionary
* and the HTML renderer live here once instead of drifting apart in two files.
* The tokenizers themselves stay separate (different syntaxes / source matching).
*/
/** The four callout banner types the editor schema supports. */
export const CALLOUT_TYPES = ['info', 'success', 'warning', 'danger'] as const;
export type CalloutType = (typeof CALLOUT_TYPES)[number];
/**
* Coerce an arbitrary type name onto a supported banner type, defaulting to
* `info` for anything unrecognized (the shared fallback both tokenizers use).
*/
export function normalizeCalloutType(type: string): CalloutType {
return (CALLOUT_TYPES as readonly string[]).includes(type)
? (type as CalloutType)
: 'info';
}
/**
* Render a callout node to the editor's HTML shape. `body` is the already
* markdown-parsed inner content (marked may hand back a string synchronously).
*/
export function renderCalloutHtml(
type: string,
body: string | Promise<string>,
): string {
return `<div data-type="callout" data-callout-type="${type}">${body}</div>`;
}
@@ -0,0 +1,37 @@
import { Token, marked } from 'marked';
import { normalizeCalloutType, renderCalloutHtml } from './callout-common.marked';
interface CalloutToken {
type: 'callout';
calloutType: string;
text: string;
raw: string;
}
export const calloutExtension = {
name: 'callout',
level: 'block',
start(src: string) {
return src.match(/:::/)?.index ?? -1;
},
tokenizer(src: string): CalloutToken | undefined {
const rule = /^:::([a-zA-Z0-9]+)\s+([\s\S]+?):::/;
const match = rule.exec(src);
if (match) {
return {
type: 'callout',
calloutType: normalizeCalloutType(match[1]),
raw: match[0],
text: match[2].trim(),
};
}
},
renderer(token: Token) {
const calloutToken = token as CalloutToken;
return renderCalloutHtml(
calloutToken.calloutType,
marked.parse(calloutToken.text),
);
},
};
@@ -0,0 +1,72 @@
import { describe, it, expect } from "vitest";
import { extractFootnoteDefinitions } from "./footnote.marked";
/** Pull the ordered list of `data-footnote-def` ids out of the rendered section. */
function defIds(section: string): string[] {
return [...section.matchAll(/data-footnote-def data-id="([^"]+)"/g)].map(
(m) => m[1],
);
}
/** Pull the ordered list of `[^id]` markers that remain in the body. */
function bodyMarkers(body: string): string[] {
return [...body.matchAll(/\[\^([^\]\s]+)\]/g)].map((m) => m[1]);
}
describe("extractFootnoteDefinitions: duplicate definition ids (first-wins)", () => {
// Body has ONE `[^d]` reference but THREE `[^d]:` definitions. Under the
// import model (#166) a duplicate definition id is FIRST-WINS: only the first
// definition is kept; the rest are DROPPED (and surfaced by analyzeFootnotes,
// not silently re-id'd into orphan footnotes as before). Reference markers are
// never rewritten, so repeated references would reuse the single footnote.
const md = ["See[^d].", "", "[^d]: a", "[^d]: b", "[^d]: c"].join("\n");
it("keeps only the FIRST definition for the id (first-wins)", () => {
const { section } = extractFootnoteDefinitions(md);
const ids = defIds(section);
expect(ids).toEqual(["d"]);
});
it("keeps the first definition's text and drops the duplicates", () => {
const { section } = extractFootnoteDefinitions(md);
expect(section).toContain('data-footnote-def data-id="d"><p>a</p>');
// No derived `d__2` / `d__3` ids are emitted anymore.
expect(section).not.toContain("d__2");
expect(section).not.toContain("d__3");
// The dropped duplicate texts are not in the section.
expect(section).not.toContain("<p>b</p>");
expect(section).not.toContain("<p>c</p>");
});
it("leaves the SINGLE body marker as [^d] (markers are never rewritten)", () => {
const { body } = extractFootnoteDefinitions(md);
expect(bodyMarkers(body)).toEqual(["d"]);
expect(body).toContain("See[^d].");
// The definition lines themselves were pulled OUT of the body.
expect(body).not.toContain("[^d]: a");
expect(body).not.toContain("[^d]: b");
expect(body).not.toContain("[^d]: c");
});
it("does not crash and produces a well-formed footnotes section", () => {
const { section } = extractFootnoteDefinitions(md);
expect(section.startsWith("<section data-footnotes>")).toBe(true);
expect(section.endsWith("</section>")).toBe(true);
// Exactly one definition div (first-wins).
expect([...section.matchAll(/<div data-footnote-def/g)]).toHaveLength(1);
});
});
describe("extractFootnoteDefinitions: reuse (repeated references, one definition)", () => {
// Pandoc semantics: many `[^a]` references + one `[^a]:` definition = one
// footnote, shared. Markers are left intact so the editor numbers them as one.
const md = ["A[^a] B[^a] C[^a].", "", "[^a]: shared note"].join("\n");
it("emits exactly one definition and leaves every reference marker as [^a]", () => {
const { section, body } = extractFootnoteDefinitions(md);
expect(defIds(section)).toEqual(["a"]);
expect(section).toContain('data-footnote-def data-id="a"><p>shared note</p>');
// All three reference markers stay `a` (no `a__2`/`a__3` minting).
expect(bodyMarkers(body)).toEqual(["a", "a", "a"]);
});
});
@@ -0,0 +1,131 @@
import { marked } from "marked";
/**
* Pandoc/GFM footnote support for the marked (Markdown -> HTML) pipeline.
*
* Two pieces:
* - an INLINE tokenizer for `[^id]` references -> <sup data-footnote-ref
* data-id="id"> (matches the editor-ext FootnoteReference renderHTML);
* - a document hook (`preprocess`/`walkTokens` is awkward for collecting +
* removing definitions, so we use a regex preprocessing step instead) that
* pulls every `[^id]: text` definition line out of the body and appends a
* single <section data-footnotes> with one <div data-footnote-def> per
* definition, so the round-trip rebuilds footnotesList + footnoteDefinition.
*
* Every FIRST definition line is emitted duplicate ids are first-wins (the
* rest are dropped, and surfaced via analyzeFootnotes), and reference markers are
* left untouched so repeated `[^a]` references reuse the one footnote (#166).
* Orphan definitions (no matching reference) are still emitted here; the editor's
* sync plugin reconciles the final reference/definition set (drops orphans,
* synthesizes a single empty definition for a reference that lacks one).
*/
const DEFINITION_RE = /^\[\^([^\]\s]+)\]:[ \t]*(.*)$/;
const REFERENCE_RE = /\[\^([^\]\s]+)\]/;
interface FootnoteRefToken {
type: "footnoteRef";
raw: string;
id: string;
}
export const footnoteReferenceExtension = {
name: "footnoteRef",
level: "inline" as const,
start(src: string) {
return src.match(/\[\^/)?.index ?? -1;
},
tokenizer(src: string): FootnoteRefToken | undefined {
const match = REFERENCE_RE.exec(src);
// Only match at the very start of the remaining inline source.
if (match && match.index === 0) {
return {
type: "footnoteRef",
raw: match[0],
id: match[1],
};
}
return undefined;
},
renderer(token: FootnoteRefToken) {
return `<sup data-footnote-ref data-id="${escapeAttr(token.id)}"></sup>`;
},
};
function escapeAttr(value: string): string {
return String(value).replace(/&/g, "&amp;").replace(/"/g, "&quot;");
}
/**
* Extract `[^id]: text` definition lines from the markdown body, returning the
* cleaned body plus a rendered <section data-footnotes> (empty string when no
* definitions). Call this BEFORE marked.parse and append the section to the
* resulting HTML.
*/
export function extractFootnoteDefinitions(markdown: string): {
body: string;
section: string;
} {
const lines = markdown.split("\n");
const bodyLines: string[] = [];
const definitions: Array<{ id: string; text: string }> = [];
// Track fenced-code state so a `[^id]: ...` line that merely SHOWS footnote
// syntax inside a ``` / ~~~ code block is left in the body verbatim and not
// mistaken for a real definition.
let fence: string | null = null;
for (const line of lines) {
const fenceMatch = /^(\s*)(`{3,}|~{3,})/.exec(line);
if (fenceMatch) {
const marker = fenceMatch[2][0];
if (fence === null) {
fence = marker; // opening fence
} else if (marker === fence) {
fence = null; // closing fence (matching delimiter type)
}
bodyLines.push(line);
continue;
}
const m = fence === null ? DEFINITION_RE.exec(line) : null;
if (m) {
definitions.push({ id: m[1], text: m[2] });
} else {
bodyLines.push(line);
}
}
if (definitions.length === 0) {
return { body: markdown, section: "" };
}
// Duplicate definition ids (e.g. `[^d]: first` / `[^d]: second`): FIRST WINS,
// the rest are DROPPED. Reference markers are left UNTOUCHED so repeated `[^a]`
// references reuse the single footnote (Pandoc semantics, #166). This differs
// from the live editor's never-lose policy (resolveCollisions re-ids a
// duplicate definition into an orphan) on purpose: an import is an
// agent-authored artifact we sanitize, and the dropped duplicate is surfaced
// to the caller via analyzeFootnotes' `duplicateDefinitions` warning instead.
const firstById = new Map<string, string>(); // id -> first definition text
for (const def of definitions) {
if (!firstById.has(def.id)) firstById.set(def.id, def.text);
}
const defsHtml = [...firstById.entries()]
.map(([id, text]) => {
// Render the definition text as inline markdown so emphasis/links inside
// a footnote survive the round-trip; wrap in a paragraph (the node's
// content is paragraph+).
const inner = marked.parseInline(text || "");
return `<div data-footnote-def data-id="${escapeAttr(
id,
)}"><p>${inner}</p></div>`;
})
.join("");
return {
body: bodyLines.join("\n"),
section: `<section data-footnotes>${defsHtml}</section>`,
};
}
@@ -0,0 +1,54 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml } from "./marked.utils";
/**
* Regression for issue #192: pasting a GitHub-style `> [!type]` alert produced a
* literal `<blockquote>` containing `[!info]` instead of a callout node, because
* only the `:::type` form was tokenized. The editor paste path runs the same
* `markdownToHtml`, so these assertions pin the conversion at the source.
*/
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
describe("markdownToHtml: GitHub `> [!type]` callouts", () => {
it("converts `> [!info]` to a callout node, not a literal blockquote", () => {
const out = html("> [!info]\n> Callout body text here");
expect(out).toContain('data-type="callout"');
expect(out).toContain('data-callout-type="info"');
expect(out).toContain("Callout body text here");
expect(out).not.toContain("[!info]");
expect(out).not.toContain("<blockquote");
});
it("maps GitHub alert aliases onto the supported banner types", () => {
expect(html("> [!NOTE]\n> x")).toContain('data-callout-type="info"');
expect(html("> [!TIP]\n> x")).toContain('data-callout-type="success"');
expect(html("> [!WARNING]\n> x")).toContain('data-callout-type="warning"');
expect(html("> [!CAUTION]\n> x")).toContain('data-callout-type="danger"');
});
it("accepts the editor's own type names directly", () => {
expect(html("> [!success]\n> x")).toContain('data-callout-type="success"');
expect(html("> [!danger]\n> x")).toContain('data-callout-type="danger"');
});
it("falls back to info for an unknown type", () => {
expect(html("> [!bogus]\n> x")).toContain('data-callout-type="info"');
});
it("preserves multi-line callout bodies", () => {
const out = html("> [!warning]\n> line one\n> line two");
expect(out).toContain('data-callout-type="warning"');
expect(out).toContain("line one");
expect(out).toContain("line two");
});
it("still converts the `:::type` form", () => {
const out = html(":::info\nbody\n:::");
expect(out).toContain('data-type="callout"');
expect(out).toContain('data-callout-type="info"');
});
});
@@ -0,0 +1,81 @@
import { Token, marked } from 'marked';
import { renderCalloutHtml } from './callout-common.marked';
interface GithubCalloutToken {
type: 'githubCallout';
calloutType: string;
text: string;
raw: string;
}
/**
* Map GitHub "alert" blockquote markers (`> [!NOTE]`, `> [!WARNING]`, ) onto
* the four callout banner types the editor schema supports. The editor's own
* type names (`info`/`success`/`warning`/`danger`) are also accepted directly,
* because users paste both forms. Anything unrecognized falls back to `info`,
* matching the `:::type` callout tokenizer.
*/
const GITHUB_ALERT_TYPE_MAP: Record<string, string> = {
note: 'info',
tip: 'success',
important: 'info',
warning: 'warning',
caution: 'danger',
info: 'info',
success: 'success',
danger: 'danger',
};
/**
* Tokenizer for GitHub-flavored alert callouts written as a blockquote whose
* first line is `[!type]`:
*
* > [!info]
* > body line one
* > body line two
*
* Without this, the default blockquote tokenizer wins and the marker renders as
* a literal `[!info]` inside a `<blockquote>`. The editor's paste path runs the
* same `markdownToHtml`, so registering this here also fixes pasting the syntax
* into the editor (issue #192), not just markdown import.
*/
export const githubCalloutExtension = {
name: 'githubCallout',
level: 'block' as const,
start(src: string) {
return src.match(/^ {0,3}>[ \t]*\[!/m)?.index ?? -1;
},
tokenizer(src: string): GithubCalloutToken | undefined {
const rule =
/^ {0,3}>[ \t]*\[!([a-zA-Z]+)\][^\n]*(?:\n {0,3}>[^\n]*)*(?:\n|$)/;
const match = rule.exec(src);
if (!match) return undefined;
const rawType = match[1].toLowerCase();
const calloutType = GITHUB_ALERT_TYPE_MAP[rawType] ?? 'info';
const text = match[0]
.replace(/\n+$/, '')
.split('\n')
// Strip the blockquote marker (`>` + optional space) from every line.
.map((line) => line.replace(/^ {0,3}>[ \t]?/, ''))
// Drop the `[!type]` marker that opens the first line.
.map((line, i) => (i === 0 ? line.replace(/^\[![a-zA-Z]+\][ \t]*/, '') : line))
.join('\n')
.trim();
return {
type: 'githubCallout',
calloutType,
raw: match[0],
text,
};
},
renderer(token: Token) {
const calloutToken = token as GithubCalloutToken;
return renderCalloutHtml(
calloutToken.calloutType,
marked.parse(calloutToken.text),
);
},
};
@@ -0,0 +1,41 @@
import { Token } from "marked";
interface HtmlEmbedToken {
type: "htmlEmbed";
raw: string;
encoded: string;
}
/**
* Marked extension that rebuilds an `htmlEmbed` node from the HTML comment
* marker produced by the turndown rule (`<!--html-embed:<base64>-->`).
*
* It emits the same marker div the node's `parseHTML` recognizes, so the
* pipeline MD -> HTML -> ProseMirror JSON restores the node (and its
* base64 `data-source`) exactly. We do NOT expand the raw markup here; the
* source stays base64-encoded in the attribute and is only executed by the
* client NodeView.
*/
export const htmlEmbedExtension = {
name: "htmlEmbed",
level: "block" as const,
start(src: string) {
return src.indexOf("<!--html-embed:");
},
tokenizer(src: string): HtmlEmbedToken | undefined {
const rule = /^<!--html-embed:([A-Za-z0-9+/=]*)-->/;
const match = rule.exec(src);
if (match) {
return {
type: "htmlEmbed",
raw: match[0],
encoded: match[1] ?? "",
};
}
},
renderer(token: Token) {
const htmlEmbedToken = token as HtmlEmbedToken;
return `<div data-type="htmlEmbed" data-source="${htmlEmbedToken.encoded}"></div>`;
},
};
@@ -0,0 +1,76 @@
import { marked } from "marked";
import { calloutExtension } from "./callout.marked";
import { githubCalloutExtension } from "./github-callout.marked";
import { mathBlockExtension } from "./math-block.marked";
import { mathInlineExtension } from "./math-inline.marked";
import {
footnoteReferenceExtension,
extractFootnoteDefinitions,
} from "./footnote.marked";
import { htmlEmbedExtension } from "./html-embed.marked";
marked.use({
renderer: {
list({ ordered, start, items }) {
let body = "";
for (const item of items) {
body += this.listitem(item);
}
if (ordered) {
const startAttr = start !== 1 ? ` start="${start}"` : "";
return `<ol${startAttr}>\n${body}</ol>\n`;
}
const isTaskList = items.some((item) => item.task);
const dataType = isTaskList ? ' data-type="taskList"' : "";
return `<ul${dataType}>\n${body}</ul>\n`;
},
listitem({ tokens, task: isTask, checked: isChecked }) {
const text = this.parser.parse(tokens);
if (!isTask) {
return `<li>${text}</li>\n`;
}
const checkedAttr = isChecked
? 'data-checked="true"'
: 'data-checked="false"';
return `<li data-type="taskItem" ${checkedAttr}>${text}</li>\n`;
},
},
});
marked.use({
extensions: [
calloutExtension,
githubCalloutExtension,
mathBlockExtension,
mathInlineExtension,
footnoteReferenceExtension,
htmlEmbedExtension,
],
});
marked.setOptions({ breaks: true });
export function markdownToHtml(
markdownInput: string,
): string | Promise<string> {
const YAML_FONT_MATTER_REGEX = /^\s*---[\s\S]*?---\s*/;
const markdown = markdownInput
.replace(YAML_FONT_MATTER_REGEX, "")
.trimStart();
// Pull `[^id]: ...` definition lines out of the body, render the body, then
// append a single <section data-footnotes> so the round-trip rebuilds the
// footnotesList + footnoteDefinition nodes.
const { body, section } = extractFootnoteDefinitions(markdown);
const parsed = marked.parse(body);
if (!section) return parsed;
if (typeof parsed === "string") {
return parsed + section;
}
return parsed.then((html) => html + section);
}
@@ -0,0 +1,37 @@
import { Token, marked } from 'marked';
interface MathBlockToken {
type: 'mathBlock';
text: string;
raw: string;
}
export const mathBlockExtension = {
name: 'mathBlock',
level: 'block',
start(src: string) {
return src.match(/\$\$/)?.index ?? -1;
},
tokenizer(src: string): MathBlockToken | undefined {
const rule = /^\$\$(?!(\$))([\s\S]+?)\$\$/;
const match = rule.exec(src);
if (match) {
return {
type: 'mathBlock',
raw: match[0],
text: match[2]?.trim(),
};
}
},
renderer(token: Token) {
const mathBlockToken = token as MathBlockToken;
// parse to prevent escaping slashes
const latex = marked
.parse(mathBlockToken.text)
.toString()
.replace(/<(\/)?p>/g, '');
return `<div data-type="${mathBlockToken.type}" data-katex="true">${latex}</div>`;
},
};
@@ -0,0 +1,50 @@
import { describe, it, expect } from "vitest";
import { markdownToHtml } from "./marked.utils";
/**
* Data-integrity regression (issue #204, Phase 2): plain prose that mentions
* prices like `$5 and $6` must NOT be misread as inline math. The inline-math
* tokenizer mutates a global `marked` singleton at import time
* (`marked.utils.ts`), so math behaviour can only be exercised safely through
* the public `markdownToHtml`; importing the tokenizer in isolation would give
* a different, non-representative result. These assertions therefore drive the
* real conversion path.
*/
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
const MATH_MARKERS = ['data-type="mathInline"', 'data-katex="true"'];
function hasInlineMath(out: string): boolean {
return MATH_MARKERS.some((m) => out.includes(m));
}
describe("markdownToHtml: inline-math false positives", () => {
it("does not treat prices `$5 and $6` as inline math", () => {
const out = html("It costs $5 and $6 today.");
expect(hasInlineMath(out)).toBe(false);
// The text survives verbatim (no katex span swallowing it).
expect(out).toContain("$5 and $6");
});
it("does not treat a single trailing price `$5` as inline math", () => {
const out = html("Lunch was $5.");
expect(hasInlineMath(out)).toBe(false);
expect(out).toContain("$5");
});
it("does not treat `$5, $6, $7` (multiple prices) as inline math", () => {
const out = html("Choose $5, $6, $7 plans.");
expect(hasInlineMath(out)).toBe(false);
});
it("STILL converts a genuine inline-math expression `$x + y$`", () => {
// Guard the positive path so the false-positive guard above can't be
// satisfied by simply disabling math entirely.
const out = html("The sum $x + y$ is shown.");
expect(hasInlineMath(out)).toBe(true);
});
});
@@ -0,0 +1,55 @@
import { Token, marked } from 'marked';
interface MathInlineToken {
type: 'mathInline';
text: string;
raw: string;
}
const inlineMathRegex = /^\$(?!\s)(.+?)(?<!\s)\$(?!\d)/;
export const mathInlineExtension = {
name: 'mathInline',
level: 'inline',
start(src: string) {
let index: number;
let indexSrc = src;
while (indexSrc) {
index = indexSrc.indexOf('$');
if (index === -1) {
return;
}
const f = index === 0 || indexSrc.charAt(index - 1) === ' ';
if (f) {
const possibleKatex = indexSrc.substring(index);
if (possibleKatex.match(inlineMathRegex)) {
return index;
}
}
indexSrc = indexSrc.substring(index + 1).replace(/^\$+/, '');
}
},
tokenizer(src: string): MathInlineToken | undefined {
const match = inlineMathRegex.exec(src);
if (match) {
return {
type: 'mathInline',
raw: match[0],
text: match[1]?.trim(),
};
}
},
renderer(token: Token) {
const mathInlineToken = token as MathInlineToken;
// parse to prevent escaping slashes
const latex = marked
.parse(mathInlineToken.text)
.toString()
.replace(/<(\/)?p>/g, '');
return `<span data-type="${mathInlineToken.type}" data-katex="true">${latex}</span>`;
},
};
@@ -0,0 +1,128 @@
import { describe, it, expect } from "vitest";
import { getSchema } from "@tiptap/core";
import { generateHTML, generateJSON } from "@tiptap/html";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Bold } from "@tiptap/extension-bold";
import { htmlToMarkdown } from "./turndown.utils";
import { markdownToHtml } from "./marked.utils";
import { Spoiler } from "../../spoiler/spoiler";
// The spoiler mark has no native Markdown syntax, so it is preserved losslessly
// as raw inline HTML (`<span data-spoiler="true">…</span>`), the same approach
// htmlEmbed uses. This test drives the full editor round-trip:
// JSON -> HTML -> Markdown -> HTML -> JSON
// and asserts the `spoiler` mark survives end to end. We use the same
// getSchema + @tiptap/html generateHTML/generateJSON utilities the other
// editor-ext schema tests use.
const extensions = [Document, Paragraph, Text, Bold, Spoiler];
function html(md: string): string {
const out = markdownToHtml(md);
if (typeof out !== "string") throw new Error("expected sync string output");
return out;
}
// Count text nodes carrying a `spoiler` mark anywhere in a ProseMirror JSON doc.
function countSpoilerMarks(doc: any): number {
let count = 0;
const walk = (node: any) => {
if (!node || typeof node !== "object") return;
if (Array.isArray(node.marks)) {
for (const mark of node.marks) {
if (mark?.type === "spoiler") count++;
}
}
if (Array.isArray(node.content)) node.content.forEach(walk);
};
walk(doc);
return count;
}
describe("Spoiler mark schema", () => {
it("registers the spoiler mark in the schema", () => {
const schema = getSchema(extensions);
expect(schema.marks.spoiler).toBeTruthy();
});
it("recovers the spoiler mark from span[data-spoiler] (HTML -> JSON)", () => {
const json = generateJSON(
'<p>before <span data-spoiler="true">hidden</span> after</p>',
extensions,
);
expect(countSpoilerMarks(json)).toBe(1);
});
it("emits data-spoiler + class on render (JSON -> HTML)", () => {
const doc = {
type: "doc",
content: [
{
type: "paragraph",
content: [
{
type: "text",
text: "hidden",
marks: [{ type: "spoiler" }],
},
],
},
],
};
const out = generateHTML(doc, extensions);
expect(out).toContain('data-spoiler="true"');
expect(out).toContain('class="spoiler"');
});
});
describe("Spoiler Markdown round-trip is lossless", () => {
const docWith = (textNode: any) => ({
type: "doc",
content: [
{
type: "paragraph",
content: [{ type: "text", text: "before " }, textNode, { type: "text", text: " after" }],
},
],
});
it("preserves the spoiler mark through JSON -> MD -> HTML -> JSON", () => {
const startDoc = docWith({
type: "text",
text: "hidden",
marks: [{ type: "spoiler" }],
});
// JSON -> HTML
const html1 = generateHTML(startDoc, extensions);
expect(html1).toContain('data-spoiler="true"');
// HTML -> Markdown (raw inline HTML, lossless)
const md = htmlToMarkdown(html1);
expect(md).toContain('<span data-spoiler="true">hidden</span>');
// MD -> HTML -> JSON (mark restored via parseHTML)
const endJson = generateJSON(html(md), extensions);
expect(countSpoilerMarks(endJson)).toBe(1);
// The visible text survives.
expect(JSON.stringify(endJson)).toContain("hidden");
});
it("keeps the spoiler intact when it intersects a bold mark", () => {
const startDoc = docWith({
type: "text",
text: "secret",
marks: [{ type: "bold" }, { type: "spoiler" }],
});
const md = htmlToMarkdown(generateHTML(startDoc, extensions));
expect(md).toContain("data-spoiler=\"true\"");
const endJson = generateJSON(html(md), extensions);
expect(countSpoilerMarks(endJson)).toBe(1);
// Bold survives alongside the spoiler.
expect(JSON.stringify(endJson)).toContain('"bold"');
});
});
@@ -0,0 +1,12 @@
// Map @joplin/turndown types to @types/turndown
declare module "@joplin/turndown" {
import TurndownService from "turndown";
export = TurndownService;
}
declare module "@joplin/turndown-plugin-gfm" {
import TurndownService from "turndown";
export const tables: TurndownService.Plugin;
export const strikethrough: TurndownService.Plugin;
export const highlightedCodeBlock: TurndownService.Plugin;
}
@@ -0,0 +1,147 @@
import { describe, it, expect } from "vitest";
import { htmlToMarkdown } from "./turndown.utils";
import { markdownToHtml } from "./marked.utils";
/**
* #206 mdrt-2 Markdown export must never SILENTLY drop a block. (FIXED)
*
* `htmlToMarkdown` (turndown) historically only registered rules for a fixed
* set of custom nodes (callout, taskItem, details, math, iframe, htmlEmbed,
* image, video, footnote). Any other custom node `transclusionReference`,
* `pageBreak`, `mention`, `status` fell through to turndown's default
* handling: an empty wrapper is "blank" and removed, so the block disappeared
* from the exported Markdown with no trace, and `mention`/`status` collapsed to
* bare text, losing their identity (data-id / data-color). The invariant
* "never silently lose a block" was broken.
*
* The fix adds lossless turndown rules that re-emit each of these nodes as raw
* HTML carrying every `data-*` attribute. Plain-Markdown viewers ignore the
* inert tag; the import path round-trips it (`markdownToHtml` passes the raw
* HTML through and each node's `parseHTML` rebuilds the ProseMirror node). These
* tests assert the surviving contract (the block is preserved AND its identity
* round-trips back through import).
*/
describe("htmlToMarkdown — custom nodes are preserved losslessly (#206 mdrt-2)", () => {
const wrap = (inner: string) => `<p>before</p>${inner}<p>after</p>`;
it("preserves a pageBreak block on Markdown export", () => {
const md = htmlToMarkdown(
wrap('<div data-type="pageBreak" class="page-break"></div>'),
);
expect(md).toContain("before");
expect(md).toContain("after");
// The break survives as an inert raw-HTML tag, not silently dropped.
expect(md).toMatch(/data-type="pageBreak"/);
expect(md).toMatch(/page-?break/i);
});
it("preserves a transclusionReference's identity on Markdown export", () => {
const md = htmlToMarkdown(
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
);
expect(md).toContain("before");
expect(md).toContain("after");
// The data-id (the only thing that gives the reference identity) survives.
expect(md).toContain("abc");
expect(md).toMatch(/data-type="transclusionReference"/);
});
it("preserves a mention's data-id (stable identity) on Markdown export", () => {
const md = htmlToMarkdown(
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
);
// The mention keeps its stable identity (data-id), not just the text.
expect(md).toContain("u1");
expect(md).toContain("Bob");
expect(md).toMatch(/data-type="mention"/);
});
it("preserves a status chip's color on Markdown export", () => {
const md = htmlToMarkdown(
'<p>s <span data-type="status" data-color="green">Done</span></p>',
);
// The chip's color (its identity) survives, not just the visible text.
expect(md).toContain("green");
expect(md).toContain("Done");
expect(md).toMatch(/data-type="status"/);
});
// The export form is only lossless if the import path can rebuild it. These
// assert the full MD -> HTML round-trip restores the node + its attributes,
// which is the marker <-> node contract each `parseHTML` relies on.
describe("import round-trip (markdownToHtml restores the node)", () => {
it("round-trips a pageBreak through export + import", async () => {
const md = htmlToMarkdown(
wrap('<div data-type="pageBreak" class="page-break"></div>'),
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<div[^>]*data-type="pageBreak"[^>]*>/);
expect(html).toContain("before");
expect(html).toContain("after");
});
it("round-trips a transclusionReference (keeps data-id)", async () => {
const md = htmlToMarkdown(
wrap('<div data-type="transclusionReference" data-id="abc"></div>'),
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<div[^>]*data-type="transclusionReference"[^>]*>/);
expect(html).toContain("abc");
});
it("round-trips a mention (keeps data-id + data-label)", async () => {
const md = htmlToMarkdown(
'<p>hi <span data-type="mention" data-id="u1" data-label="Bob">@Bob</span> there</p>',
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<span[^>]*data-type="mention"[^>]*>/);
expect(html).toContain("u1");
expect(html).toContain("Bob");
});
it("round-trips a status chip (keeps data-color)", async () => {
const md = htmlToMarkdown(
'<p>s <span data-type="status" data-color="green">Done</span></p>',
);
const html = await markdownToHtml(md);
expect(html).toMatch(/<span[^>]*data-type="status"[^>]*>/);
expect(html).toContain("green");
});
// HTML special chars in an attribute value or in a node's text must be
// ESCAPED when re-emitted as raw HTML, otherwise the exported tag is
// malformed and `markdownToHtml`'s parser cannot restore the original value
// (the same silent data loss this PR fixes). Dropping `<`/`>` escaping is the
// dangerous regression: a stray `<` or `>` corrupts the tag (or injects new
// markup), so the test data carries ALL of `&`, `"`, `<`, `>` in BOTH the
// data-label attribute and the visible text. That fully exercises
// escapeHtmlAttr's `&,",<,>` branches and escapeHtmlText's `&,<,>` branches
// (escapeHtmlText leaves `"` literal); the alphanumeric-only cases above hit
// none of them.
it("escapes HTML special chars (& \" < >) in attrs + text and round-trips them", async () => {
const md = htmlToMarkdown(
`<p>hi <span data-type="mention" data-id="u1" data-label="A &amp; &lt;B&gt; &quot;C&quot;">@A &amp; &lt;B&gt; "C"</span> there</p>`,
);
// (a) The exported Markdown carries a WELL-FORMED, correctly-escaped tag:
// the attribute escapes `&`, `<`, `>` AND `"`; the text escapes `&`, `<`,
// `>` (a `"` inside text content is legal, so it stays literal).
expect(md).toContain('data-label="A &amp; &lt;B&gt; &quot;C&quot;"');
expect(md).toContain('>@A &amp; &lt;B&gt; "C"</span>');
// And explicitly NOT the raw, tag-corrupting forms: a literal `<B>` (would
// mean `<`/`>` escaping was dropped in either the attr or the text)...
expect(md).not.toContain("<B>");
// ...nor the malformed attribute that an unescaped `"` would produce.
expect(md).not.toContain('data-label="A &amp; &lt;B&gt; "C""');
// (b) Import restores the ORIGINAL (unescaped) values, attribute and text.
const html = await markdownToHtml(md);
const dom = new DOMParser().parseFromString(html as string, "text/html");
const span = dom.querySelector('span[data-type="mention"]');
expect(span).not.toBeNull();
expect(span!.getAttribute("data-id")).toBe("u1");
expect(span!.getAttribute("data-label")).toBe('A & <B> "C"');
expect(span!.textContent).toBe('@A & <B> "C"');
});
});
});
@@ -0,0 +1,488 @@
import * as _TurndownService from '@joplin/turndown';
import * as TurndownPluginGfm from '@joplin/turndown-plugin-gfm';
import { getBasename } from './basename';
// CJS/ESM interop: .default exists in Vite, not in NestJS
const TurndownService = (_TurndownService as any).default || _TurndownService;
function sanitizeMdLinkText(value: string): string {
return value
.replace(/\\/g, '\\\\')
.replace(/([\[\]!])/g, '\\$1')
.replace(/[\r\n]+/g, ' ');
}
// Tags turndown treats as void (self-closing). Footnote references render as an
// empty <sup data-footnote-ref> whose meaning lives entirely in its data-id;
// without marking it void, turndown's blank-node removal drops it before our
// rule runs, losing the `[^id]` marker. Mirrors turndown's built-in list.
const TURNDOWN_VOID_ELEMENTS = [
'AREA', 'BASE', 'BR', 'COL', 'COMMAND', 'EMBED', 'HR', 'IMG', 'INPUT',
'KEYGEN', 'LINK', 'META', 'PARAM', 'SOURCE', 'TRACK', 'WBR',
];
function isVoidNode(node: any): boolean {
const name = node?.nodeName?.toUpperCase?.();
if (!name) return false;
if (name === 'SUP' && node.hasAttribute?.('data-footnote-ref')) {
return true;
}
return TURNDOWN_VOID_ELEMENTS.indexOf(name) !== -1;
}
/**
* An empty <sup data-footnote-ref> is "blank" to turndown, which removes blank
* inline nodes (RootNode/Node use a module-level isVoid the options cannot
* override). To survive, inject the id as text content so the node is non-blank;
* the footnoteReference rule then reads data-id and emits `[^id]`.
*/
function fillEmptyFootnoteRefs(html: string): string {
return html.replace(
/<sup\b([^>]*\bdata-footnote-ref\b[^>]*)>\s*<\/sup>/gi,
(_m, attrs) => `<sup${attrs}>​</sup>`,
);
}
/**
* `pageBreak` and `transclusionReference` are childless atom <div>s. Like an
* empty footnote ref (see above), turndown treats a childless block as "blank"
* and replaces it with the blankRule BEFORE any custom rule can fire so the
* node disappears from the export with no trace (#206 mdrt-2). Inject a
* zero-width space so the node is non-blank and our lossless rule runs; the
* rule rebuilds the tag from the element's attributes, so the injected char
* never reaches the output.
*/
function fillEmptyAtomBlocks(html: string): string {
return html.replace(
/<div\b([^>]*\bdata-type="(?:pageBreak|transclusionReference)"[^>]*)>\s*<\/div>/gi,
(_m, attrs) => `<div${attrs}>​</div>`,
);
}
/** HTML-escape an attribute value so a re-emitted raw-HTML tag is well-formed. */
function escapeHtmlAttr(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
/** HTML-escape text placed inside a re-emitted raw-HTML element. */
function escapeHtmlText(value: string): string {
return value
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
/**
* Serialize ALL of an element's attributes back to a raw-HTML attribute string
* (leading space included). Generic on purpose: a custom node's identity lives
* entirely in its `data-*` attributes (data-id, data-color, data-source-page-id,
* data-transclusion-id, ), and serializing every attribute keeps the export
* lossless regardless of which attributes a given node carries.
*/
function serializeAttrs(node: any): string {
const attrs = node?.attributes;
if (!attrs) return '';
return Array.from(attrs as ArrayLike<{ name: string; value: string }>)
.map((attr) => ` ${attr.name}="${escapeHtmlAttr(attr.value ?? '')}"`)
.join('');
}
export function htmlToMarkdown(html: string): string {
const turndownService = new TurndownService({
headingStyle: 'atx',
codeBlockStyle: 'fenced',
hr: '---',
bulletListMarker: '-',
isVoid: isVoidNode,
});
turndownService.use([
TurndownPluginGfm.tables,
TurndownPluginGfm.strikethrough,
TurndownPluginGfm.highlightedCodeBlock,
taskList,
callout,
preserveDetail,
listParagraph,
orderedListItem,
mathInline,
mathBlock,
iframeEmbed,
htmlEmbed,
spoiler,
image,
video,
footnoteReference,
footnotesList,
pageBreak,
transclusionReference,
mention,
status,
]);
return turndownService
.turndown(fillEmptyAtomBlocks(fillEmptyFootnoteRefs(html)))
.replaceAll('<br>', ' ');
}
/**
* Lossless export rules for custom nodes that have NO native Markdown syntax
* (#206 mdrt-2). Markdown cannot represent a page break, a transclusion
* reference, a mention's stable id, or a status chip's color so rather than
* letting turndown silently drop them, each rule re-emits the node as raw HTML
* carrying every `data-*` attribute. Plain-Markdown viewers ignore the inert
* tag, and the import path round-trips it: `markdownToHtml` passes raw HTML
* through and each node's `parseHTML` (`div[data-type="…"]`, `span[…]`) rebuilds
* the ProseMirror node with its attributes intact.
*/
function pageBreak(turndownService: _TurndownService) {
turndownService.addRule('pageBreak', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'pageBreak'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
},
});
}
function transclusionReference(turndownService: _TurndownService) {
turndownService.addRule('transclusionReference', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'transclusionReference'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
return `\n\n<div${serializeAttrs(node)}></div>\n\n`;
},
});
}
function mention(turndownService: _TurndownService) {
turndownService.addRule('mention', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-type') === 'mention'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const text = escapeHtmlText(node.textContent || '');
return `<span${serializeAttrs(node)}>${text}</span>`;
},
});
}
function status(turndownService: _TurndownService) {
turndownService.addRule('status', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' && node.getAttribute('data-type') === 'status'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const text = escapeHtmlText(node.textContent || '');
return `<span${serializeAttrs(node)}>${text}</span>`;
},
});
}
/**
* Serialize the `htmlEmbed` node to Markdown.
*
* Markdown has no native representation for an arbitrary-HTML block, so we
* preserve the node losslessly as an HTML comment carrying the base64-encoded
* source (the same `data-source` payload the node stores). `markdownToHtml`
* recognizes the same marker and rebuilds the node, so the round-trip
* MD -> HTML -> JSON keeps the source intact. The comment also keeps the raw
* markup inert in the exported `.md` file (it does not render in plain Markdown
* viewers).
*/
function htmlEmbed(turndownService: _TurndownService) {
turndownService.addRule('htmlEmbed', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'htmlEmbed'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const encoded = node.getAttribute('data-source') || '';
return `\n\n<!--html-embed:${encoded}-->\n\n`;
},
});
}
/**
* Serialize the `spoiler` inline mark to lossless raw inline HTML.
*
* Markdown has no native spoiler syntax, so we emit the same `<span
* data-spoiler="true"></span>` the mark renders. `marked` passes inline raw HTML
* through untouched, and `generateJSON` restores the mark via its parseHTML, so
* the round-trip MD -> HTML -> JSON keeps the spoiler intact. The UI-only
* `is-revealed` state is never serialized.
*/
function spoiler(turndownService: _TurndownService) {
turndownService.addRule('spoiler', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-spoiler') === 'true'
);
},
replacement: function (content: string) {
return `<span data-spoiler="true">${content}</span>`;
},
});
}
function listParagraph(turndownService: _TurndownService) {
turndownService.addRule('paragraph', {
filter: ['p'],
replacement: (content: string, node: HTMLInputElement) => {
if (node.parentElement?.nodeName === 'LI') {
return content;
}
return `\n\n${content}\n\n`;
},
});
}
function orderedListItem(turndownService: _TurndownService) {
turndownService.addRule('orderedListItem', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'LI' && node.getAttribute('data-type') !== 'taskItem';
},
replacement: (content: string, node: HTMLInputElement, options: any) => {
const parent = node.parentNode as HTMLElement;
if (parent.nodeName !== 'OL' && parent.nodeName !== 'UL') {
return content;
}
content = content
.replace(/^\n+/, '')
.replace(/\n+$/, '\n')
.replace(/\n/gm, '\n ');
let prefix: string;
if (parent.nodeName === 'OL') {
const start = parseInt(parent.getAttribute('start') || '1', 10);
const index = Array.prototype.indexOf.call(parent.children, node);
prefix = `${start + index}. `;
} else {
prefix = `${options.bulletListMarker} `;
}
return (
prefix +
content +
(node.nextSibling && !/\n$/.test(content) ? '\n' : '')
);
},
});
}
function callout(turndownService: _TurndownService) {
turndownService.addRule('callout', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' && node.getAttribute('data-type') === 'callout'
);
},
replacement: function (content: string, node: HTMLInputElement) {
const calloutType = node.getAttribute('data-callout-type');
return `\n\n:::${calloutType}\n${content.trim()}\n:::\n\n`;
},
});
}
function taskList(turndownService: _TurndownService) {
turndownService.addRule('taskListItem', {
filter: function (node: HTMLInputElement) {
return (
node.getAttribute('data-type') === 'taskItem' &&
node.parentNode.nodeName === 'UL'
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const isChecked = node.getAttribute('data-checked') === 'true';
const div = node.querySelector('div');
const text = div ? div.textContent.trim() : node.textContent.trim();
const prefix = `- ${isChecked ? '[x]' : '[ ]'} `;
return (
prefix +
text +
(node.nextSibling && !/\n$/.test(text) ? '\n' : '')
);
},
});
}
function preserveDetail(turndownService: _TurndownService) {
turndownService.addRule('preserveDetail', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'DETAILS';
},
replacement: function (_content: string, node: HTMLInputElement) {
const summary = node.querySelector(':scope > summary');
let detailSummary = '';
if (summary) {
detailSummary = `<summary>${turndownService.turndown(summary.innerHTML)}</summary>`;
}
const detailsContent = Array.from(node.childNodes)
.filter((child) => child.nodeName !== 'SUMMARY')
.map((child) =>
child.nodeType === 1
? turndownService.turndown((child as HTMLElement).outerHTML)
: child.textContent,
)
.join('');
return `\n<details>\n${detailSummary}\n\n${detailsContent}\n\n</details>\n`;
},
});
}
function mathInline(turndownService: _TurndownService) {
turndownService.addRule('mathInline', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SPAN' &&
node.getAttribute('data-type') === 'mathInline'
);
},
replacement: function (content: string) {
return `$${content}$`;
},
});
}
function mathBlock(turndownService: _TurndownService) {
turndownService.addRule('mathBlock', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' &&
node.getAttribute('data-type') === 'mathBlock'
);
},
replacement: function (content: string) {
return `\n$$\n${content}\n$$\n`;
},
});
}
function iframeEmbed(turndownService: _TurndownService) {
turndownService.addRule('iframeEmbed', {
filter: function (node: HTMLInputElement) {
return node.nodeName === 'IFRAME';
},
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src');
return '[' + src + '](' + src + ')';
},
});
}
function image(turndownService: _TurndownService) {
turndownService.addRule('image', {
filter: 'img',
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src') || '';
if (!src) return '';
const caption = node.getAttribute('data-caption') || '';
if (caption) {
// ![]() can't carry a caption, so emit a raw <img> wrapped in a block
// <div>. marked passes it through and the image extension's parseHTML
// restores the caption from data-caption.
const parts = [`src="${escapeHtmlAttr(src)}"`];
const alt = node.getAttribute('alt') || '';
if (alt) parts.push(`alt="${escapeHtmlAttr(alt)}"`);
parts.push(`data-caption="${escapeHtmlAttr(caption)}"`);
return `<div><img ${parts.join(' ')}></div>`;
}
const alt = sanitizeMdLinkText(node.getAttribute('alt') || '');
const title = node.getAttribute('title') || '';
const titlePart = title ? ' "' + title.replace(/"/g, '\\"') + '"' : '';
return '![' + alt + '](' + src + titlePart + ')';
},
});
}
/**
* Footnote reference (inline atom) -> pandoc/GFM marker `[^id]`.
* The visible number is derived (not stored), so the id is the stable anchor.
*/
function footnoteReference(turndownService: _TurndownService) {
turndownService.addRule('footnoteReference', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SUP' && node.hasAttribute('data-footnote-ref')
);
},
replacement: function (_content: string, node: HTMLInputElement) {
const id = node.getAttribute('data-id') || '';
return id ? `[^${id}]` : '';
},
});
}
/**
* Footnotes container -> the list of `[^id]: text` definitions at the end of
* the document (one per line). Each footnoteDefinition inside emits its own
* `[^id]: ...` line; turndown joins them with the surrounding block spacing.
*/
function footnotesList(turndownService: _TurndownService) {
turndownService.addRule('footnoteDefinition', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'DIV' && node.hasAttribute('data-footnote-def')
);
},
replacement: function (content: string, node: HTMLInputElement) {
const id = node.getAttribute('data-id') || '';
// Collapse internal newlines so the definition stays a single MD line;
// continuation lines are a v2 refinement.
const text = content.replace(/\s*\n+\s*/g, ' ').trim();
return id ? `\n[^${id}]: ${text}\n` : '';
},
});
turndownService.addRule('footnotesList', {
filter: function (node: HTMLInputElement) {
return (
node.nodeName === 'SECTION' && node.hasAttribute('data-footnotes')
);
},
replacement: function (content: string) {
return `\n\n${content.trim()}\n`;
},
});
}
function video(turndownService: _TurndownService) {
turndownService.addRule('video', {
filter: function (node: HTMLInputElement) {
return node.tagName === 'VIDEO';
},
replacement: function (_content: string, node: HTMLInputElement) {
const src = node.getAttribute('src') || '';
const ariaLabel = node.getAttribute('aria-label');
const name = sanitizeMdLinkText(
ariaLabel || getBasename(src) || src,
);
return '[' + name + '](' + src + ')';
},
});
}
+1 -6
View File
@@ -14,15 +14,10 @@ export default defineConfig({
provider: "v8", provider: "v8",
reporter: ["text-summary", "text"], reporter: ["text-summary", "text"],
all: false, all: false,
// functions lowered 60 -> 57 after issue #347 removed the editor-ext
// markdown layer (src/lib/markdown) and its image/footnote round-trip
// specs: that markdown behavior now lives in — and is tested by —
// @docmost/prosemirror-markdown, so the editor-ext baseline shifts down.
// Still a real gate (a few points below the post-removal measured level).
thresholds: { thresholds: {
statements: 54, statements: 54,
branches: 44, branches: 44,
functions: 57, functions: 60,
lines: 54, lines: 54,
}, },
}, },
+3 -22
View File
@@ -1,31 +1,12 @@
# @docmost/prosemirror-markdown # @docmost/prosemirror-markdown
The single, canonical **ProseMirror ↔ Markdown converter** plus the Docmost The single, canonical **ProseMirror ↔ Markdown converter** plus the Docmost
schema mirror (#293/#345/#347). Headless and framework-free: no React. There is schema mirror (#293/#345). Headless and framework-free: no React, no browser
exactly ONE copy of this converter in the repo, consumed by: runtime. There is exactly ONE copy of this converter in the repo, consumed by:
- `packages/mcp` (the MCP server), - `packages/mcp` (the MCP server),
- `packages/git-sync` (two-way Git sync), - `packages/git-sync` (two-way Git sync),
- `apps/server` (server-side markdown import/export, #345), - `apps/server` (server-side markdown import/export, #345).
- `apps/client` (markdown paste/copy + AI-chat render, #347).
### Node vs browser entry
The HTML→DOM stage of markdown import runs on `jsdom` in Node and the native
`DOMParser` in the browser, injected per environment so **jsdom never enters a
client bundle**:
- default entry (`@docmost/prosemirror-markdown`) — Node: registers jsdom +
`@tiptap/html`'s happy-dom `server` `generateJSON`. Used by mcp / git-sync /
apps/server.
- `browser` entry (`@docmost/prosemirror-markdown/browser`, via the `"browser"`
exports condition) — registers the native `DOMParser` + `@tiptap/html`'s
browser `generateJSON`. Used by `apps/client`; carries no jsdom/happy-dom.
Both entries expose the identical converter surface; only the injected
DOM/`generateJSON` implementations differ (`src/lib/dom-parser.ts`). A
`markdownToProseMirrorSync` variant exists for callers that cannot await (the
client's synchronous chat renderer).
`src/lib/docmost-schema.ts` **mirrors** the upstream Tiptap schema that lives in `src/lib/docmost-schema.ts` **mirrors** the upstream Tiptap schema that lives in
`packages/editor-ext`. The mirror is not free-floating: `serializer-contract.test.ts` `packages/editor-ext`. The mirror is not free-floating: `serializer-contract.test.ts`
@@ -9,12 +9,7 @@
"exports": { "exports": {
".": { ".": {
"types": "./build/index.d.ts", "types": "./build/index.d.ts",
"browser": "./build/browser.js",
"default": "./build/index.js" "default": "./build/index.js"
},
"./browser": {
"types": "./build/browser.d.ts",
"default": "./build/browser.js"
} }
}, },
"scripts": { "scripts": {
@@ -35,7 +30,6 @@
"@tiptap/html": "3.20.4", "@tiptap/html": "3.20.4",
"@tiptap/pm": "3.20.4", "@tiptap/pm": "3.20.4",
"@tiptap/starter-kit": "3.20.4", "@tiptap/starter-kit": "3.20.4",
"happy-dom": "20.8.9",
"jsdom": "25.0.0", "jsdom": "25.0.0",
"marked": "17.0.5", "marked": "17.0.5",
"zod": "4.3.6" "zod": "4.3.6"
@@ -1,17 +0,0 @@
/**
* BROWSER entry of `@docmost/prosemirror-markdown`.
*
* Selected via the package's `"browser"` exports condition (bundlers) so the
* client gets the SAME public converter surface as the Node entry, but the
* markdown-import DOM passes run on the native `window.DOMParser` instead of
* jsdom. This module installs the native parser as a side effect BEFORE
* re-exporting, and imports NO jsdom so a client bundle that resolves this
* entry carries no `jsdom` (nor any transitive jsdom import).
*
* The re-exported surface is identical to the default (Node) entry the only
* difference is which HTML-DOM parser is registered so a browser consumer can
* call `markdownToProseMirror` (and everything else) exactly as the server does.
*/
import "./lib/dom-parser.browser.js";
export * from "./lib/index.js";
@@ -6,11 +6,4 @@
* this top-level barrel simply re-exports that surface so the package entry is * this top-level barrel simply re-exports that surface so the package entry is
* the converter surface. * the converter surface.
*/ */
// DEFAULT (Node) entry: install the jsdom-backed HTML parser as a side effect
// BEFORE re-exporting the converter surface, so every Node consumer (server,
// mcp, git-sync) keeps the identical jsdom import behaviour with no code change.
// The browser entry (`./browser.js`) installs the native-`DOMParser` parser
// instead and never loads this module, so jsdom stays out of client bundles.
import "./lib/dom-parser.node.js";
export * from "./lib/index.js"; export * from "./lib/index.js";
@@ -63,9 +63,10 @@ function getStyleProperty(element: HTMLElement, propertyName: string): string |
* The editor SCHEMA genuinely only supports these six banner types there is no * The editor SCHEMA genuinely only supports these six banner types there is no
* `tip`/`caution`/`important`/`question` callout node. So those are NOT first- * `tip`/`caution`/`important`/`question` callout node. So those are NOT first-
* class types we can round-trip literally; they are INPUT ALIASES (GitHub/Obsidian * class types we can round-trip literally; they are INPUT ALIASES (GitHub/Obsidian
* alert syntax). This package's own `> [!type]` import path maps them onto the * alert syntax). The editor's own paste/import path maps them onto the supported
* supported set (see `CALLOUT_TYPE_ALIASES` below: tip -> success, caution -> * set (see `GITHUB_ALERT_TYPE_MAP` in
* danger, important -> info). We apply that aliasing * `@docmost/editor-ext` markdown/utils/github-callout.marked.ts:
* tip -> success, caution -> danger, important -> info). We mirror that aliasing
* here so an ingested `> [!tip]` / `> [!caution]` lands on the closest real banner * here so an ingested `> [!tip]` / `> [!caution]` lands on the closest real banner
* (success / danger) instead of flatly collapsing to `info` matching exactly how * (success / danger) instead of flatly collapsing to `info` matching exactly how
* the editor itself would interpret the same alias. A schema type always maps to * the editor itself would interpret the same alias. A schema type always maps to
@@ -74,11 +75,11 @@ function getStyleProperty(element: HTMLElement, propertyName: string): string |
*/ */
const CALLOUT_TYPES = ["default", "info", "note", "success", "warning", "danger"]; const CALLOUT_TYPES = ["default", "info", "note", "success", "warning", "danger"];
/** /**
* NON-schema callout aliases -> their closest supported banner, for the names * NON-schema callout aliases -> their closest supported banner. Mirrors the
* that are NOT already schema types (a schema type is preserved as-is and never * editor's `GITHUB_ALERT_TYPE_MAP` for the names that are NOT already schema
* consulted here). This is the single canonical alias map now that the editor's * types (a schema type is preserved as-is and never consulted here). Keeping
* old marked layer is gone; git-sync ingest and an editor paste both go through * these in lockstep means git-sync ingest and an editor paste interpret the same
* this package, so they interpret the same `> [!alias]` identically. * `> [!alias]` identically.
*/ */
const CALLOUT_TYPE_ALIASES: Record<string, string> = { const CALLOUT_TYPE_ALIASES: Record<string, string> = {
tip: "success", tip: "success",
@@ -1,27 +0,0 @@
/**
* BROWSER registration of the injectable HTML parser (native `DOMParser`).
*
* Importing this module for its SIDE EFFECT installs a `window.DOMParser`-backed
* {@link HtmlDocumentParser}. It is loaded by the package's `browser.ts` barrel
* (selected via the `"browser"` exports condition). It imports NO `jsdom`, so a
* client bundle that resolves the browser entry never pulls jsdom in.
*
* The returned `Document` is a real browser document, so it exposes exactly the
* same query/mutation surface the import passes use (`querySelector(All)`,
* `createElement`, `createTreeWalker`/`NodeFilter` via `defaultView`,
* `body.innerHTML`) as jsdom did on the Node path.
*/
// @tiptap/html's default (browser) entry: its `generateJSON` uses the native
// `window.DOMParser`, so it carries NO jsdom/happy-dom — keeping the client
// bundle free of Node-only DOM libs.
import { generateJSON } from "@tiptap/html";
import { setHtmlDocumentParser, setGenerateJson } from "./dom-parser.js";
setHtmlDocumentParser((html: string): Document => {
// Native, always available in a browser (and in a jsdom/happy-dom test
// environment, which is what the client vitest suite runs under). `text/html`
// parsing matches jsdom's `new JSDOM(html)` behaviour for our fragments.
return new DOMParser().parseFromString(html, "text/html");
});
setGenerateJson(generateJSON);
@@ -1,30 +0,0 @@
/**
* NODE registration of the injectable HTML parser (jsdom-backed).
*
* Importing this module for its SIDE EFFECT installs a jsdom-backed
* {@link HtmlDocumentParser}. It is loaded by the package's default entry
* (`index.ts`) so every existing Node consumer (server, mcp, git-sync) keeps
* the identical jsdom behaviour with no code change. This is the ONLY module in
* the import chain that imports `jsdom`; the browser entry never loads it, so
* `jsdom` cannot reach the client bundle.
*/
import { JSDOM } from "jsdom";
// Use @tiptap/html's EXPLICIT server entry (happy-dom backed): it builds its own
// DOM internally and needs NO ambient global `window`, so the Node path never
// depends on which of @tiptap/html's conditional exports a resolver picks (Jest
// selects the browser entry, which would throw without a global window). This
// avoids the old module-level `global.window` jsdom shim entirely — that shim
// was timing-fragile (it had to be installed AFTER prosemirror-view's
// import-time env detection, or prosemirror-view reads an undefined `navigator`).
import { generateJSON } from "@tiptap/html/server";
import { setHtmlDocumentParser, setGenerateJson } from "./dom-parser.js";
setHtmlDocumentParser((html: string): Document => {
// A fresh JSDOM per call mirrors the previous `new JSDOM(html)` usage in each
// import pass — no shared mutable document between conversions, so concurrent
// conversions never interfere.
const dom = new JSDOM(html);
return dom.window.document as unknown as Document;
});
setGenerateJson(generateJSON);
@@ -1,87 +0,0 @@
/**
* Injectable HTML-string -> DOM parser for the markdown import path.
*
* The markdown -> ProseMirror chain (`markdown-to-prosemirror.ts`) does three
* post-`marked` DOM passes (task-list bridge, comment directives, footnote
* assembly) that need a real DOM to query/mutate an HTML fragment. On the NODE
* path that DOM comes from `jsdom`; in the BROWSER the platform already provides
* a native `DOMParser` and `document`, and `jsdom` must NOT be bundled (size +
* it is Node-only). So the concrete parser is INJECTED per environment rather
* than imported statically here this module carries no `jsdom` (or any DOM)
* import, so nothing on the browser code path can transitively pull `jsdom` in.
*
* The Node entry (`./dom-parser.node.js`, loaded by the package's default
* `index.js`) registers a jsdom-backed parser; the browser entry
* (`./dom-parser.browser.js`, loaded by the `browser.js` barrel) registers a
* `window.DOMParser`-backed one. A consumer that forgets to load an entry (e.g.
* a raw deep import) gets a clear error instead of a silent wrong-environment
* crash.
*/
/**
* Parse an HTML string into a `Document`. The returned document must support the
* standard query/mutation surface the import passes use: `querySelector(All)`,
* `createElement`, `createTreeWalker` + `NodeFilter` (read off the document's
* `defaultView`), and `body.innerHTML`.
*/
export type HtmlDocumentParser = (html: string) => Document;
/**
* Convert an HTML string to a ProseMirror JSON doc against the given TipTap
* extension set. This is `@tiptap/html`'s `generateJSON`, injected per
* environment so the Node path binds its happy-dom `server` entry and the
* browser path its native-`DOMParser` entry neither leaking the other's DOM
* lib into the wrong bundle.
*/
export type GenerateJsonFn = (html: string, extensions: any[]) => any;
let injectedParser: HtmlDocumentParser | null = null;
let injectedGenerateJson: GenerateJsonFn | null = null;
/**
* Register the environment's HTML parser. Called ONCE at import time by the
* Node or browser entry module. Idempotent-friendly: the last registration
* wins, so a test harness can override it.
*/
export function setHtmlDocumentParser(parser: HtmlDocumentParser): void {
injectedParser = parser;
}
/**
* Parse `html` into a `Document` using the registered parser. Throws a clear
* error when no environment entry has registered one (the caller imported the
* converter without going through the Node/browser barrel).
*/
export function parseHtmlDocument(html: string): Document {
if (!injectedParser) {
throw new Error(
"No HTML DOM parser registered. Import `@docmost/prosemirror-markdown` " +
"(Node) or `@docmost/prosemirror-markdown/browser` (browser) so the " +
"environment's DOM parser is installed before calling the converter.",
);
}
return injectedParser(html);
}
/**
* Register the environment's `generateJSON` (HTML -> ProseMirror JSON). Called
* ONCE at import time by the Node or browser entry module.
*/
export function setGenerateJson(fn: GenerateJsonFn): void {
injectedGenerateJson = fn;
}
/**
* Run the registered `generateJSON`. Throws a clear error when no environment
* entry has registered one (same cause as {@link parseHtmlDocument}).
*/
export function generateJsonWith(html: string, extensions: any[]): any {
if (!injectedGenerateJson) {
throw new Error(
"No generateJSON registered. Import `@docmost/prosemirror-markdown` " +
"(Node) or `@docmost/prosemirror-markdown/browser` (browser) so the " +
"environment's generateJSON is installed before calling the converter.",
);
}
return injectedGenerateJson(html, extensions);
}
@@ -18,10 +18,7 @@ export type { DocmostMdMeta } from "./markdown-document.js";
export { convertProseMirrorToMarkdown } from "./markdown-converter.js"; export { convertProseMirrorToMarkdown } from "./markdown-converter.js";
export type { ConvertProseMirrorToMarkdownOptions } from "./markdown-converter.js"; export type { ConvertProseMirrorToMarkdownOptions } from "./markdown-converter.js";
export { export { markdownToProseMirror } from "./markdown-to-prosemirror.js";
markdownToProseMirror,
markdownToProseMirrorSync,
} from "./markdown-to-prosemirror.js";
// The Docmost tiptap schema mirror. Exposed so consumers (and the sync // The Docmost tiptap schema mirror. Exposed so consumers (and the sync
// engine's schema-validity regression tests) can build the exact ProseMirror // engine's schema-validity regression tests) can build the exact ProseMirror
@@ -7,8 +7,9 @@
* natively through the collab gateway, so no websocket/Yjs write-path lives * natively through the collab gateway, so no websocket/Yjs write-path lives
* here. * here.
*/ */
import { generateJSON } from "@tiptap/html";
import { JSDOM } from "jsdom";
import { Marked } from "marked"; import { Marked } from "marked";
import { parseHtmlDocument, generateJsonWith } from "./dom-parser.js";
import type { TokenizerExtension, RendererExtension } from "marked"; import type { TokenizerExtension, RendererExtension } from "marked";
import { docmostExtensions } from "./docmost-schema.js"; import { docmostExtensions } from "./docmost-schema.js";
import { parseAttachedComment } from "./attached-comment.js"; import { parseAttachedComment } from "./attached-comment.js";
@@ -244,13 +245,12 @@ const markedInstance = new Marked().use({
], ],
}); });
// NOTE: this module no longer installs a module-level `global.window`/`document` // Setup DOM environment for Tiptap HTML parsing in Node.js
// jsdom shim. The HTML->DOM passes below (bridgeTaskLists / applyCommentDirectives const dom = new JSDOM("<!DOCTYPE html><html><body></body></html>");
// / assembleFootnotes) parse via the INJECTED `parseHtmlDocument` (jsdom on the global.window = dom.window as any;
// Node entry, native `DOMParser` on the browser entry), and `@tiptap/html`'s v3 global.document = dom.window.document;
// `generateJSON` supplies its OWN DOM per environment (happy-dom in Node, native // @ts-ignore
// `DOMParser` in the browser) — so no ambient global DOM is needed here, and global.Element = dom.window.Element;
// nothing on the browser code path statically imports jsdom.
/** /**
* Hard ceiling above which we skip callout preprocessing entirely. The linear * Hard ceiling above which we skip callout preprocessing entirely. The linear
@@ -295,13 +295,7 @@ const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
* - emits the same `<div data-type="callout" data-callout-type="TYPE">` output * - emits the same `<div data-type="callout" data-callout-type="TYPE">` output
* (inner rendered through marked) as the previous regex implementation. * (inner rendered through marked) as the previous regex implementation.
*/ */
// SYNCHRONOUS by construction: the only formerly-awaited call is async function preprocessCallouts(markdown: string): Promise<string> {
// `markedInstance.parse`, which returns a string synchronously for this
// instance (no async marked extensions are registered), so the whole callout
// preprocess is sync. Keeping it sync lets a sync converter entry
// (`markdownToProseMirrorSync`, used by the client's chat renderer which must
// stay synchronous) share this exact logic with the async entry.
function preprocessCallouts(markdown: string): string {
// Defensive cap: skip preprocessing for pathologically large inputs. // Defensive cap: skip preprocessing for pathologically large inputs.
if (markdown.length > MAX_CALLOUT_PREPROCESS_BYTES) { if (markdown.length > MAX_CALLOUT_PREPROCESS_BYTES) {
return markdown; return markdown;
@@ -310,7 +304,7 @@ function preprocessCallouts(markdown: string): string {
// Recursively transform a slice of lines, converting top-level callouts in // Recursively transform a slice of lines, converting top-level callouts in
// that slice into <div> blocks and rendering their inner content (which may // that slice into <div> blocks and rendering their inner content (which may
// itself contain nested callouts) through this same function. // itself contain nested callouts) through this same function.
const transform = (lines: string[]): string => { const transform = async (lines: string[]): Promise<string> => {
const out: string[] = []; const out: string[] = [];
let inCodeFence = false; let inCodeFence = false;
let codeFenceMarker = ""; // the exact run of backticks/tildes that opened it let codeFenceMarker = ""; // the exact run of backticks/tildes that opened it
@@ -389,8 +383,8 @@ function preprocessCallouts(markdown: string): string {
if (j < lines.length) { if (j < lines.length) {
// Found the matching closing fence: render the body (recursively, so // Found the matching closing fence: render the body (recursively, so
// nested callouts are handled) and emit the callout div. // nested callouts are handled) and emit the callout div.
const inner = transform(bodyLines); const inner = await transform(bodyLines);
const renderedInner = markedInstance.parse(inner) as string; const renderedInner = await markedInstance.parse(inner);
out.push( out.push(
`\n<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>\n`, `\n<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>\n`,
); );
@@ -429,8 +423,8 @@ function preprocessCallouts(markdown: string): string {
// Drop the prefix + `>` + one optional space, leaving the body content. // Drop the prefix + `>` + one optional space, leaving the body content.
bodyLines.push(lines[j].slice(prefix.length).replace(/^>\s?/, "")); bodyLines.push(lines[j].slice(prefix.length).replace(/^>\s?/, ""));
} }
const inner = transform(bodyLines); const inner = await transform(bodyLines);
const renderedInner = markedInstance.parse(inner) as string; const renderedInner = await markedInstance.parse(inner);
const block = `<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>`; const block = `<div data-type="callout" data-callout-type="${type}">${renderedInner}</div>`;
if (prefix.length === 0) { if (prefix.length === 0) {
// Top-level callout: blank lines isolate the HTML block. // Top-level callout: blank lines isolate the HTML block.
@@ -497,7 +491,8 @@ function bridgeTaskLists(html: string): string {
if (html.length > MAX_CALLOUT_PREPROCESS_BYTES) { if (html.length > MAX_CALLOUT_PREPROCESS_BYTES) {
return html; return html;
} }
const document = parseHtmlDocument(html); const dom = new JSDOM(html);
const document = dom.window.document;
// Collect the checkbox(es) that belong to THIS <li> directly: either direct // Collect the checkbox(es) that belong to THIS <li> directly: either direct
// child <input type="checkbox"> elements or ones inside the <li>'s direct <p> // child <input type="checkbox"> elements or ones inside the <li>'s direct <p>
// child (the shape marked emits: `<li><p><input type="checkbox"> text</p></li>`). // child (the shape marked emits: `<li><p><input type="checkbox"> text</p></li>`).
@@ -667,23 +662,15 @@ function placeStandalone(
function applyCommentDirectives(html: string): string { function applyCommentDirectives(html: string): string {
// Cheap early-out: no comments at all -> nothing to intercept. // Cheap early-out: no comments at all -> nothing to intercept.
if (!html.includes("<!--")) return html; if (!html.includes("<!--")) return html;
const document = parseHtmlDocument(html); const dom = new JSDOM(html);
// `SHOW_COMMENT` (128) is a stable DOM constant. Read it from whichever holder const document = dom.window.document;
// exists — the document's window (jsdom: `defaultView`), the ambient global const nodeFilter = dom.window.NodeFilter;
// `NodeFilter` (browsers / test envs), else the literal — because a document
// produced by `DOMParser.parseFromString` has NO browsing context, so its
// `defaultView` is `null` (unlike a jsdom `new JSDOM(html).window.document`).
// `createTreeWalker` takes the numeric `whatToShow` mask directly.
const SHOW_COMMENT =
(document.defaultView as any)?.NodeFilter?.SHOW_COMMENT ??
(globalThis as any).NodeFilter?.SHOW_COMMENT ??
0x80;
// Walk the WHOLE document, not just <body>: when a standalone machinery // Walk the WHOLE document, not just <body>: when a standalone machinery
// comment is the FIRST thing in the output (before any body content), the // comment is the FIRST thing in the output (before any body content), the
// HTML parser places it at document level (a child of `#document`, before // HTML parser places it at document level (a child of `#document`, before
// `<html>`), where it is outside `document.body` and would be lost. Attached // `<html>`), where it is outside `document.body` and would be lost. Attached
// attrs comments always live inside body, so this wider walk still finds them. // attrs comments always live inside body, so this wider walk still finds them.
const walker = document.createTreeWalker(document, SHOW_COMMENT); const walker = document.createTreeWalker(document, nodeFilter.SHOW_COMMENT);
const comments: any[] = []; const comments: any[] = [];
let current: any; let current: any;
while ((current = walker.nextNode())) comments.push(current); while ((current = walker.nextNode())) comments.push(current);
@@ -958,7 +945,8 @@ const MAX_FOOTNOTE_ROUNDS = 10000;
function assembleFootnotes(html: string): string { function assembleFootnotes(html: string): string {
// Cheap early-out: nothing carries a footnote body -> nothing to assemble. // Cheap early-out: nothing carries a footnote body -> nothing to assemble.
if (!html.includes("data-fn-text")) return html; if (!html.includes("data-fn-text")) return html;
const document = parseHtmlDocument(html); const dom = new JSDOM(html);
const document = dom.window.document;
if (document.querySelector("sup[data-footnote-ref][data-fn-text]") == null) { if (document.querySelector("sup[data-footnote-ref][data-fn-text]") == null) {
return html; return html;
} }
@@ -1072,18 +1060,12 @@ function stripEmptyParagraphs(node: any): any {
return { ...node, content: cleaned }; return { ...node, content: cleaned };
} }
/** /** Convert markdown to a ProseMirror doc using the full Docmost schema. */
* Convert markdown to a ProseMirror doc using the full Docmost schema export async function markdownToProseMirror(
* (SYNCHRONOUS core). Every stage callout preprocess, `marked` parse, the markdownContent: string,
* three DOM passes, and generateJSON is synchronous for this configuration ): Promise<any> {
* (no async marked extensions), so the conversion needs no `await`. The async const withCallouts = await preprocessCallouts(markdownContent);
* `markdownToProseMirror` below delegates here (its Promise return is preserved const html = await markedInstance.parse(withCallouts);
* for every existing Node consumer). A sync entry is REQUIRED by the client's
* chat renderer, which runs inside a React render/useMemo and cannot await.
*/
export function markdownToProseMirrorSync(markdownContent: string): any {
const withCallouts = preprocessCallouts(markdownContent);
const html = markedInstance.parse(withCallouts) as string;
// Materialize comment directives (#293 #9 attached textAlign; #5 standalone // Materialize comment directives (#293 #9 attached textAlign; #5 standalone
// subpages/pageBreak) while the comment nodes still exist, before generateJSON // subpages/pageBreak) while the comment nodes still exist, before generateJSON
// drops them. // drops them.
@@ -1093,17 +1075,6 @@ export function markdownToProseMirrorSync(markdownContent: string): any {
// generateJSON, so references + definitions materialize into the schema model. // generateJSON, so references + definitions materialize into the schema model.
const withFootnotes = assembleFootnotes(withAttrs); const withFootnotes = assembleFootnotes(withAttrs);
const bridged = bridgeTaskLists(withFootnotes); const bridged = bridgeTaskLists(withFootnotes);
const doc = generateJsonWith(bridged, docmostExtensions); const doc = generateJSON(bridged, docmostExtensions);
return stripEmptyParagraphs(doc); return stripEmptyParagraphs(doc);
} }
/**
* Convert markdown to a ProseMirror doc (async entry, unchanged contract). Kept
* async so every existing Node consumer (server, mcp, git-sync) that `await`s
* it is untouched; it simply delegates to the synchronous core.
*/
export async function markdownToProseMirror(
markdownContent: string,
): Promise<any> {
return markdownToProseMirrorSync(markdownContent);
}
@@ -1,101 +0,0 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { JSDOM } from "jsdom";
import {
setHtmlDocumentParser,
parseHtmlDocument,
} from "../src/lib/dom-parser.js";
import { markdownToProseMirror } from "../src/lib/markdown-to-prosemirror.js";
/**
* The markdown import path parses its post-`marked` HTML through an INJECTED
* DOM parser: jsdom on the Node entry, native `DOMParser` on the browser entry
* (see dom-parser.node.ts / dom-parser.browser.ts). These tests exercise BOTH
* registrations against the same canonical inputs the three DOM passes
* (task-list bridge, comment directives, footnote assembly) and assert the
* converter produces the IDENTICAL ProseMirror doc regardless of which DOM
* parser is installed. That is the guarantee the client paste path relies on:
* pasting in the browser must yield the same nodes the server import produces.
*/
// A jsdom-backed parser (the Node entry's registration).
const jsdomParser = (html: string): Document =>
new JSDOM(html).window.document as unknown as Document;
// A `DOMParser`-backed parser standing in for the BROWSER entry's registration.
// We drive a real `DOMParser` from a jsdom window (jsdom exposes the same
// `new DOMParser().parseFromString(html, "text/html")` API the browser and the
// client's jsdom vitest environment provide), so this path uses the exact code
// dom-parser.browser.ts runs — a native `DOMParser`, no `JSDOM` document glue.
const browserWindow = new JSDOM("").window;
const domParserBackedParser = (html: string): Document =>
new browserWindow.DOMParser().parseFromString(
html,
"text/html",
) as unknown as Document;
// Canonical inputs, each hitting a different post-marked DOM pass.
const CASES: Record<string, string> = {
// footnote assembly (assembleFootnotes): `^[…]` -> sup + section/def
"inline footnote ^[…]": "Body^[a note].",
// task-list bridge (bridgeTaskLists): checkbox list -> taskList/taskItem
"task list": "- [x] done\n- [ ] todo",
// comment directives (applyCommentDirectives): standalone machinery comment
"standalone subpages comment": "text\n\n<!--subpages-->\n\ntext2",
// attached image comment (applyCommentDirectives img form)
"attached image comment": '![alt](img.png) <!--img {"align":"left"}-->',
// github callout (preprocessCallouts bq path) + comment pass
"obsidian callout": "> [!info]\n> hello",
// highlight + math (marked extensions; still parsed through the DOM stage)
"highlight + math": "A ==mark== and $x^2$ end",
};
async function convertWith(
parser: (html: string) => Document,
md: string,
): Promise<any> {
setHtmlDocumentParser(parser);
return markdownToProseMirror(md);
}
describe("markdown import: Node (jsdom) and browser (DOMParser) DOM paths agree", () => {
afterEach(() => {
// Restore the jsdom parser the suite-wide setup file installs, so later
// tests in the run are unaffected by our per-case swaps.
setHtmlDocumentParser(jsdomParser);
});
for (const [name, md] of Object.entries(CASES)) {
it(`produces identical nodes for: ${name}`, async () => {
const viaJsdom = await convertWith(jsdomParser, md);
const viaDomParser = await convertWith(domParserBackedParser, md);
expect(viaDomParser).toEqual(viaJsdom);
});
}
});
describe("dom-parser injection contract", () => {
let saved: (html: string) => Document;
beforeEach(() => {
saved = jsdomParser;
});
afterEach(() => {
setHtmlDocumentParser(saved);
});
it("throws a clear error when no parser is registered", () => {
// Install a thrower to simulate the unregistered state, then assert
// parseHtmlDocument surfaces the guidance error (we cannot un-set the
// module singleton, so we assert via a registration that throws the same).
setHtmlDocumentParser(() => {
throw new Error("No HTML DOM parser registered.");
});
expect(() => parseHtmlDocument("<p>x</p>")).toThrow(/No HTML DOM parser/);
});
it("uses the most-recently registered parser", () => {
const marker = new JSDOM("<!DOCTYPE html><body><b>marker</b></body>").window
.document as unknown as Document;
setHtmlDocumentParser(() => marker);
expect(parseHtmlDocument("<i>ignored</i>")).toBe(marker);
});
});
@@ -1,13 +0,0 @@
/**
* Vitest setup: register the Node (jsdom) HTML parser for the whole suite.
*
* The package's tests import the converter through the RELATIVE `src/lib/*`
* modules (or the `docmost-client`/`src/lib/index` barrel), NOT through the
* top-level `index.ts` entry that a real Node consumer imports so the entry's
* side-effect registration of the jsdom parser never runs here. This setup file
* performs the SAME registration the Node entry does, so `markdownToProseMirror`
* has a DOM parser in the (node-environment) tests, matching production Node
* behaviour. The browser path is covered separately by the client paste tests
* (jsdom vitest env + native DOMParser) and the dedicated dom-parser test.
*/
import "../src/lib/dom-parser.node.js";
@@ -19,9 +19,5 @@ export default defineConfig({
test: { test: {
environment: 'node', environment: 'node',
include: ['test/**/*.test.ts'], include: ['test/**/*.test.ts'],
// Register the Node (jsdom) HTML parser before any test runs. Tests import
// the converter via relative src/lib modules, bypassing the top-level entry
// that normally installs the parser as a side effect (see setup file).
setupFiles: ['test/setup.dom-parser.ts'],
}, },
}); });
+5 -7
View File
@@ -284,9 +284,6 @@ importers:
'@docmost/editor-ext': '@docmost/editor-ext':
specifier: workspace:* specifier: workspace:*
version: link:../../packages/editor-ext version: link:../../packages/editor-ext
'@docmost/prosemirror-markdown':
specifier: workspace:*
version: link:../../packages/prosemirror-markdown
'@excalidraw/excalidraw': '@excalidraw/excalidraw':
specifier: 0.18.0-3a5ef40 specifier: 0.18.0-3a5ef40
version: 0.18.0-3a5ef40(@types/react-dom@18.3.1)(@types/react@18.3.12)(react-dom@18.3.1(react@18.3.1))(react@18.3.1) version: 0.18.0-3a5ef40(@types/react-dom@18.3.1)(@types/react@18.3.12)(react-dom@18.3.1(react@18.3.1))(react@18.3.1)
@@ -915,6 +912,10 @@ importers:
version: 8.57.1(eslint@9.39.4(jiti@2.4.2))(typescript@5.9.3) version: 8.57.1(eslint@9.39.4(jiti@2.4.2))(typescript@5.9.3)
packages/editor-ext: packages/editor-ext:
dependencies:
marked:
specifier: 17.0.5
version: 17.0.5
devDependencies: devDependencies:
'@vitest/coverage-v8': '@vitest/coverage-v8':
specifier: 4.1.6 specifier: 4.1.6
@@ -1116,9 +1117,6 @@ importers:
'@tiptap/starter-kit': '@tiptap/starter-kit':
specifier: 3.20.4 specifier: 3.20.4
version: 3.20.4 version: 3.20.4
happy-dom:
specifier: 20.8.9
version: 20.8.9
jsdom: jsdom:
specifier: 25.0.0 specifier: 25.0.0
version: 25.0.0 version: 25.0.0
@@ -18492,7 +18490,7 @@ snapshots:
happy-dom@20.8.9: happy-dom@20.8.9:
dependencies: dependencies:
'@types/node': 25.5.0 '@types/node': 22.19.1
'@types/whatwg-mimetype': 3.0.2 '@types/whatwg-mimetype': 3.0.2
'@types/ws': 8.18.1 '@types/ws': 8.18.1
entities: 7.0.1 entities: 7.0.1