fix(#345 ): normalize CRLF before front-matter strip (review round 3)

F9 [WARNING] The line-anchored front-matter regex from round 2 requires a bare LF after the opening `---`, so a Windows/CRLF foreign file (`---\r\n...`) slips past the strip and leaks its front-matter into the body (where `title: Foo` renders as a setext heading that title extraction hijacks). The canonical parser whose regex shape this copied (page-file.ts) normalizes CRLF -> LF BEFORE its FRONTMATTER_RE; the import path copied the regex but missed the normalization. normalizeForeignMarkdown now replaces CRLF with LF first (which also makes convertReferenceFootnotes' split('\n') consistent). Adds a CRLF fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(#345 ): replace id-alternation regex with a fixed generic scanner + line-anchor frontmatter (review round 2)
2026-07-05 05:38:07 +03:00 · 2026-07-05 05:18:44 +03:00 · 2026-07-05 04:54:07 +03:00 · 2026-07-05 03:39:01 +03:00 · 2026-07-05 03:27:01 +03:00 · 2026-07-05 03:21:07 +03:00
24 changed files with 1184 additions and 1783 deletions
@@ -201,7 +201,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
 | `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
 | `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
 | `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
-| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp` and `git-sync`; there is exactly ONE copy of the converter now |
+| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
 `build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
@@ -284,7 +284,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
 ### Client structure
 Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
 - **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, import/export) — editor schema changes often need to be made in `editor-ext`, not just the client. The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by both `mcp` and `git-sync` — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
+- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
 - API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
 - Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
@@ -1373,39 +1373,6 @@
  "The role catalog is unavailable": "The role catalog is unavailable",
  "Please try again later.": "Please try again later.",
  "No bundles available": "No bundles available",
  "Content": "Content",
  "Content language of the roles": "Content language of the roles",
  "{{count}} updates available in {{bundles}} bundles": "{{count}} updates available in {{bundles}} bundles",
  "Update all ({{count}})": "Update all ({{count}})",
  "Updating {{current}}/{{total}}…": "Updating {{current}}/{{total}}…",
  "{{count}} roles are installed in another language. A different language installs separately and appears as new.": "{{count}} roles are installed in another language. A different language installs separately and appears as new.",
  "{{count}} roles": "{{count}} roles",
  "{{count}} new — none installed": "{{count}} new — none installed",
  "All installed · up to date": "All installed · up to date",
  "{{count}} updates · {{installed}} up to date": "{{count}} updates · {{installed}} up to date",
  "{{count}} new": "{{count}} new",
  "{{count}} installed": "{{count}} installed",
  "{{count}} updates": "{{count}} updates",
  "Install bundle": "Install bundle",
  "Install {{count}} selected": "Install {{count}} selected",
  "Install bundle ({{count}})": "Install bundle ({{count}})",
  "{{selected}} of {{total}} selected": "{{selected}} of {{total}} selected",
  "Select all": "Select all",
  "Deselect all": "Deselect all",
  "Skipped": "Skipped",
  "v{{version}}": "v{{version}}",
  "{{count}} roles installed": "{{count}} roles installed",
  "{{count}} roles installed · {{renamed}} renamed": "{{count}} roles installed · {{renamed}} renamed",
  "{{count}} roles updated": "{{count}} roles updated",
  "Installed {{installed}} · {{skipped}} skipped": "Installed {{installed}} · {{skipped}} skipped",
  "A role named \"{{name}}\" already exists in this workspace.": "A role named \"{{name}}\" already exists in this workspace.",
  "\"{{name}}\" is already installed.": "\"{{name}}\" is already installed.",
  "Rename & install": "Rename & install",
  "Couldn’t load the catalog": "Couldn’t load the catalog",
  "Check your connection and try again. Installed roles are not affected.": "Check your connection and try again. Installed roles are not affected.",
  "Retry": "Retry",
  "The catalog is empty": "The catalog is empty",
  "No role bundles are published for this language yet. Try switching the content language.": "No role bundles are published for this language yet. Try switching the content language.",
  "Already up to date": "Already up to date",
  "Updated to the latest version": "Updated to the latest version",
  "This role is no longer in the catalog": "This role is no longer in the catalog",
@@ -1235,39 +1235,6 @@
  "The role catalog is unavailable": "Каталог ролей недоступен",
  "Please try again later.": "Попробуйте позже.",
  "No bundles available": "Наборы недоступны",
  "Content": "Язык контента",
  "Content language of the roles": "Язык контента ролей",
  "{{count}} updates available in {{bundles}} bundles": "Доступно обновлений: {{count}} в наборах: {{bundles}}",
  "Update all ({{count}})": "Обновить все ({{count}})",
  "Updating {{current}}/{{total}}…": "Обновление {{current}}/{{total}}…",
  "{{count}} roles are installed in another language. A different language installs separately and appears as new.": "Ролей установлено на другом языке: {{count}}. Другой язык устанавливается отдельно и отображается как новый.",
  "{{count}} roles": "ролей: {{count}}",
  "{{count}} new — none installed": "новых: {{count}} — ничего не установлено",
  "All installed · up to date": "Все установлены · актуальны",
  "{{count}} updates · {{installed}} up to date": "обновлений: {{count}} · актуальны: {{installed}}",
  "{{count}} new": "новых: {{count}}",
  "{{count}} installed": "установлено: {{count}}",
  "{{count}} updates": "обновлений: {{count}}",
  "Install bundle": "Установить набор",
  "Install {{count}} selected": "Установить выбранные ({{count}})",
  "Install bundle ({{count}})": "Установить набор ({{count}})",
  "{{selected}} of {{total}} selected": "выбрано {{selected}} из {{total}}",
  "Select all": "Выбрать все",
  "Deselect all": "Снять выбор",
  "Skipped": "Пропущено",
  "v{{version}}": "v{{version}}",
  "{{count}} roles installed": "Установлено ролей: {{count}}",
  "{{count}} roles installed · {{renamed}} renamed": "Установлено ролей: {{count}} · переименовано: {{renamed}}",
  "{{count}} roles updated": "Обновлено ролей: {{count}}",
  "Installed {{installed}} · {{skipped}} skipped": "Установлено: {{installed}} · пропущено: {{skipped}}",
  "A role named \"{{name}}\" already exists in this workspace.": "Роль с именем «{{name}}» уже существует в этом рабочем пространстве.",
  "\"{{name}}\" is already installed.": "«{{name}}» уже установлена.",
  "Rename & install": "Переименовать и установить",
  "Couldn’t load the catalog": "Не удалось загрузить каталог",
  "Check your connection and try again. Installed roles are not affected.": "Проверьте подключение и попробуйте снова. Установленные роли не затронуты.",
  "Retry": "Повторить",
  "The catalog is empty": "Каталог пуст",
  "No role bundles are published for this language yet. Try switching the content language.": "Для этого языка ещё не опубликовано ни одного набора ролей. Попробуйте сменить язык контента.",
  "No roles configured": "Роли не настроены",
  "Already up to date": "Уже актуальна",
  "Updated to the latest version": "Обновлено до последней версии",
@@ -1,7 +1,6 @@
 import {
  useInfiniteQuery,
  useMutation,
  useQueries,
  useQuery,
  useQueryClient,
 } from "@tanstack/react-query";
@@ -308,29 +307,6 @@ export function useAiRoleCatalogBundleQuery(
  });
 }
 /**
 * Eagerly open EVERY listed bundle's content in parallel for one language. The
 * redesigned catalog shows each bundle's status summary in its COLLAPSED header,
 * which needs every role's install state up front — so contents can no longer be
 * lazy-loaded on expand. The catalog is small, so a fan-out of `useQueries` (one
 * cached read per bundle, sharing the same cache keys as
 * `useAiRoleCatalogBundleQuery`) is cheap. Gated by `enabled` (modal open + a
 * resolved language) so nothing fetches while the modal is closed.
 */
 export function useAiRoleCatalogBundlesQueries(
  bundleIds: string[],
  language: string,
  enabled: boolean,
 ) {
  return useQueries({
    queries: bundleIds.map((bundleId) => ({
      queryKey: AI_ROLE_CATALOG_BUNDLE_RQ_KEY(bundleId, language),
      queryFn: () => getAiRoleCatalogBundle(bundleId, language),
      enabled: enabled && !!language,
    })),
  });
 }
 export function useImportAiRolesFromCatalogMutation() {
  const queryClient = useQueryClient();
  const { t } = useTranslation();
@@ -77,14 +77,7 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
  });
  it("errors:[] -> only the summary notification (counts interpolated)", async () => {
-    await runMutation({
+    await runMutation({ created: 3, renamed: 1, skipped: 2, errors: [] });
      created: 3,
      renamed: 1,
      skipped: 2,
      errors: [],
      createdRoles: [],
      skippedRoles: [],
    });
    expect(notificationsShowMock).toHaveBeenCalledTimes(1);
    expect(notificationsShowMock).toHaveBeenCalledWith({
      message: "Imported 3, renamed 1, skipped 2",
@@ -100,8 +93,6 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
        { slug: "a", message: "name taken" },
        { slug: "b", message: "name taken" },
      ],
      createdRoles: [{ slug: "ok", name: "Ok" }],
      skippedRoles: [],
    });
    expect(notificationsShowMock).toHaveBeenCalledTimes(2);
    expect(notificationsShowMock).toHaveBeenNthCalledWith(1, {
@@ -108,25 +108,12 @@ export interface IAiRoleImportPayload {
  conflict: "skip" | "rename";
 }
-/**
+/** Import result counts (mirrors `importFromCatalog()`). */
 * Import result (mirrors `importFromCatalog()`). The counters (`created`,
 * `skipped`, `renamed`) drive the summary notification; the per-role lists
 * (`createdRoles`, `skippedRoles`) drive the redesigned catalog modal's inline
 * result plaque — which roles were installed (and any rename) and which were
 * skipped and why (so the plaque can name the conflicting role and offer
 * "Rename & install").
 */
 export interface IAiRoleImportResult {
  created: number;
  skipped: number;
  renamed: number;
  errors: { slug: string; message: string }[];
  createdRoles: { slug: string; name: string; renamedTo?: string }[];
  skippedRoles: {
    slug: string;
    name: string;
    reason: "name-conflict" | "already-installed";
  }[];
 }
 /**
@@ -1,234 +0,0 @@
 import { describe, it, expect } from "vitest";
 import {
  bundleCounts,
  bundlePhase,
  installedLangForRole,
  mapBundleRolesToView,
  mapCatalogRoleToView,
  nameConflictSlugs,
  partialOffersRename,
  type CatalogViewRole,
 } from "./catalog-bundle-model.ts";
 import type {
  IAiRole,
  IAiRoleCatalogRole,
 } from "@/features/ai-chat/types/ai-chat.types.ts";
 function installedRole(
  source: { slug: string; language: string; version: number },
  overrides: Partial<IAiRole> = {},
 ): IAiRole {
  return {
    id: `role-${source.slug}-${source.language}`,
    name: source.slug,
    emoji: null,
    description: null,
    enabled: true,
    autoStart: true,
    launchMessage: null,
    source,
    ...overrides,
  };
 }
 function catalogRole(
  overrides: Partial<IAiRoleCatalogRole> = {},
 ): IAiRoleCatalogRole {
  return {
    slug: "writer",
    emoji: "✍️",
    name: "Writer",
    description: "Drafts copy.",
    instructions: "be a writer",
    autoStart: true,
    launchMessage: null,
    version: 3,
    ...overrides,
  };
 }
 // Build a minimal view role for bundlePhase tests.
 function viewRole(status: CatalogViewRole["status"]): CatalogViewRole {
  return { slug: `s-${status}`, name: status, description: "", version: 1, status };
 }
 describe("bundlePhase", () => {
  it("empty bundle -> empty", () => {
    expect(bundlePhase([])).toBe("empty");
  });
  it("all importable, none installed -> allNew", () => {
    expect(bundlePhase([viewRole("import"), viewRole("import")])).toBe(
      "allNew",
    );
  });
  it("nothing to import or update -> allInstalled", () => {
    expect(bundlePhase([viewRole("installed"), viewRole("installed")])).toBe(
      "allInstalled",
    );
  });
  it("updates present, nothing to import -> updates", () => {
    expect(bundlePhase([viewRole("update"), viewRole("installed")])).toBe(
      "updates",
    );
  });
  it("import + installed (no updates) -> mixed", () => {
    expect(bundlePhase([viewRole("import"), viewRole("installed")])).toBe(
      "mixed",
    );
  });
  it("import + update -> mixed", () => {
    expect(bundlePhase([viewRole("import"), viewRole("update")])).toBe("mixed");
  });
  it("a skipped role with nothing installed -> mixed (NOT allInstalled)", () => {
    // F1: a bundle whose only non-installed role was skipped has 0 installed for
    // it, so the collapsed 'All installed · up to date' header would contradict
    // the open 'Installed 0 · 1 skipped' plaque. It must be mixed until resolved.
    expect(bundlePhase([viewRole("skipped")])).toBe("mixed");
  });
  it("installed + a skipped role -> mixed (partial success is not allInstalled)", () => {
    expect(bundlePhase([viewRole("installed"), viewRole("skipped")])).toBe(
      "mixed",
    );
  });
 });
 describe("bundleCounts", () => {
  it("tallies each status once", () => {
    expect(
      bundleCounts([
        viewRole("import"),
        viewRole("import"),
        viewRole("installed"),
        viewRole("update"),
        viewRole("skipped"),
      ]),
    ).toEqual({ importable: 2, installed: 1, update: 1, skipped: 1 });
  });
 });
 describe("nameConflictSlugs / partialOffersRename (reason -> action)", () => {
  it("only name-conflict skips become the transient overlay / offer rename", () => {
    const skipped = [
      { slug: "writer", name: "Writer", reason: "name-conflict" as const },
      { slug: "editor", name: "Editor", reason: "already-installed" as const },
    ];
    expect(nameConflictSlugs(skipped)).toEqual(["writer"]);
    expect(partialOffersRename(skipped)).toBe(true);
  });
  it("an already-installed-only skip is informational: no overlay, no rename", () => {
    const skipped = [
      { slug: "editor", name: "Editor", reason: "already-installed" as const },
    ];
    expect(nameConflictSlugs(skipped)).toEqual([]);
    expect(partialOffersRename(skipped)).toBe(false);
  });
 });
 describe("installedLangForRole", () => {
  it("returns the other language when the same slug is installed elsewhere", () => {
    const roles = [installedRole({ slug: "writer", language: "ru", version: 2 })];
    expect(installedLangForRole("writer", roles, "en")).toBe("ru");
  });
  it("returns undefined when the same slug is installed in the SAME language", () => {
    const roles = [installedRole({ slug: "writer", language: "en", version: 2 })];
    expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
  });
  it("returns undefined when no install of the slug exists", () => {
    expect(installedLangForRole("writer", [], "en")).toBeUndefined();
  });
  it("ignores manually-created roles (no source)", () => {
    const roles = [
      installedRole({ slug: "writer", language: "ru", version: 2 }, {
        source: null,
      }),
    ];
    expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
  });
 });
 describe("mapCatalogRoleToView", () => {
  it("no install -> import status, catalog version, emoji preserved", () => {
    const view = mapCatalogRoleToView(catalogRole(), [], "en");
    expect(view).toMatchObject({
      slug: "writer",
      emoji: "✍️",
      name: "Writer",
      description: "Drafts copy.",
      status: "import",
      version: 3,
    });
    expect(view.installedRoleId).toBeUndefined();
    expect(view.installedLang).toBeUndefined();
  });
  it("import with the slug installed in another language -> installedLang set", () => {
    const roles = [installedRole({ slug: "writer", language: "ru", version: 9 })];
    const view = mapCatalogRoleToView(catalogRole(), roles, "en");
    expect(view.status).toBe("import");
    expect(view.installedLang).toBe("ru");
  });
  it("installed (up to date) -> installed status, catalog version, installedRoleId", () => {
    const installed = installedRole({
      slug: "writer",
      language: "en",
      version: 3,
    });
    const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
    expect(view).toMatchObject({
      status: "installed",
      version: 3,
      installedRoleId: installed.id,
    });
  });
  it("update -> version=from, newVersion=to, installedRoleId", () => {
    const installed = installedRole({
      slug: "writer",
      language: "en",
      version: 1,
    });
    const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
    expect(view).toMatchObject({
      status: "update",
      version: 1,
      newVersion: 3,
      installedRoleId: installed.id,
    });
  });
  it("missing emoji -> emoji undefined; null description -> empty string", () => {
    const view = mapCatalogRoleToView(
      catalogRole({ emoji: null, description: null }),
      [],
      "en",
    );
    expect(view.emoji).toBeUndefined();
    expect(view.description).toBe("");
  });
 });
 describe("mapBundleRolesToView", () => {
  it("maps a bundle's roles preserving order", () => {
    const roles = [
      catalogRole({ slug: "a", name: "A", version: 1 }),
      catalogRole({ slug: "b", name: "B", version: 1 }),
    ];
    const installed = [installedRole({ slug: "a", language: "en", version: 1 })];
    const view = mapBundleRolesToView(roles, installed, "en");
    expect(view.map((r) => r.slug)).toEqual(["a", "b"]);
    expect(view[0].status).toBe("installed");
    expect(view[1].status).toBe("import");
  });
 });
@@ -1,206 +0,0 @@
 import type {
  IAiRole,
  IAiRoleCatalogRole,
 } from "@/features/ai-chat/types/ai-chat.types.ts";
 import { catalogRoleInstallState } from "@/features/ai-chat/utils/catalog-role-install-state.ts";
 /**
 * The redesigned catalog modal renders bundles as cards with a summary status
 * (readable without expanding) and a single primary action. The per-role and
 * per-bundle view model that drives that UI is derived here as PURE functions so
 * the mapping, the "installed in another language" hint, and the bundle-phase
 * computation are unit-testable without mounting the component (mirrors the
 * `catalogRoleInstallState` precedent).
 */
 /**
 * A role's status in the catalog view model.
 *  - `import`    — not installed in the current content language.
 *  - `installed` — installed and up to date.
 *  - `update`    — installed, but the catalog ships a newer version.
 *  - `skipped`   — TRANSIENT client-only status set after a conflicted import
 *                  (a name collision under `conflict:'skip'`); never from the
 *                  backend.
 */
 export type RoleStatus = "import" | "installed" | "update" | "skipped";
 /** A catalog role mapped into the modal's view model. */
 export interface CatalogViewRole {
  // Slug is the stable identity within a bundle; used as the row key and as the
  // `slugs[]` payload for import.
  slug: string;
  // Optional in the catalog — the row reserves space and renders nothing when
  // absent.
  emoji?: string;
  name: string;
  description: string;
  // For `installed`/`import`: the catalog version. For `update`: the installed
  // (from) version, with `newVersion` holding the catalog (to) version.
  version: number;
  newVersion?: number;
  status: RoleStatus;
  // The language a same-slug role is installed under, when it differs from the
  // current content language (drives the Р5 hint). Only set for `import` roles.
  installedLang?: string;
  // The workspace role id, present for `installed`/`update` — needed to call the
  // update-from-catalog mutation.
  installedRoleId?: string;
 }
 /**
 * The summary phase of a bundle, derived from its roles' statuses. Determines
 * the collapsed-header summary and the bundle's single primary action.
 *  - `empty`        — the bundle has no roles.
 *  - `allNew`       — everything is importable, nothing installed.
 *  - `allInstalled` — everything installed & up to date; nothing else pending.
 *  - `updates`      — updates available and nothing left to import.
 *  - `mixed`        — any other combination.
 */
 export type BundlePhase =
  | "empty"
  | "allNew"
  | "allInstalled"
  | "updates"
  | "mixed";
 /** Per-status tallies for a bundle's roles (the single source of truth). */
 export interface BundleCounts {
  importable: number;
  installed: number;
  update: number;
  skipped: number;
 }
 /**
 * Count a bundle's roles by status ONCE. Both `bundlePhase` and the panel derive
 * from this, so the tally logic lives in exactly one place (no rescans / drift).
 */
 export function bundleCounts(roles: CatalogViewRole[]): BundleCounts {
  const counts: BundleCounts = {
    importable: 0,
    installed: 0,
    update: 0,
    skipped: 0,
  };
  for (const r of roles) {
    if (r.status === "import") counts.importable += 1;
    else if (r.status === "installed") counts.installed += 1;
    else if (r.status === "update") counts.update += 1;
    else if (r.status === "skipped") counts.skipped += 1;
  }
  return counts;
 }
 export function bundlePhase(roles: CatalogViewRole[]): BundlePhase {
  if (roles.length === 0) return "empty";
  const { importable, installed, update, skipped } = bundleCounts(roles);
  // A `skipped` role is a pending post-import conflict (0 installed for it), so a
  // bundle that has ANY skipped role is NOT "all installed & up to date" — that
  // would make the collapsed green "up to date" header contradict the open
  // panel's "Installed 0 · 1 skipped" plaque. It is `mixed` until resolved.
  if (importable === 0 && update === 0 && skipped === 0) return "allInstalled";
  if (update > 0 && importable === 0 && skipped === 0) return "updates";
  if (importable > 0 && installed === 0 && update === 0 && skipped === 0)
    return "allNew";
  return "mixed";
 }
 /**
 * The subset of a skip result that should be shown as a TRANSIENT `skipped`
 * overlay in the bundle (so the row offers a re-import path). Only NAME-CONFLICT
 * skips qualify: an `already-installed` skip (a concurrent-import race) has
 * nothing to act on — re-importing the same slug would just skip again — so it
 * must NOT be overlaid (else the row shows a misleading "Rename & install" that
 * self-heals into a false "installed"). Pure so both reason branches are tested.
 */
 export function nameConflictSlugs(
  skipped: { slug: string; reason: "name-conflict" | "already-installed" }[],
 ): string[] {
  return skipped
    .filter((s) => s.reason === "name-conflict")
    .map((s) => s.slug);
 }
 /**
 * Whether a partial-import result should offer the "Rename & install" action:
 * only when at least one skip is a name conflict (renameable). An
 * `already-installed`-only partial is informational.
 */
 export function partialOffersRename(
  skipped: { reason: "name-conflict" | "already-installed" }[],
 ): boolean {
  return skipped.some((s) => s.reason === "name-conflict");
 }
 /**
 * For a role NOT installed in the current `language`, find a workspace role with
 * the same catalog `slug` installed under a DIFFERENT language, and return that
 * language. Drives the "installed in another language" hint (Р5): a different
 * language of the same slug is a separate install and appears as `import`.
 */
 export function installedLangForRole(
  slug: string,
  workspaceRoles: IAiRole[],
  language: string,
 ): string | undefined {
  const other = workspaceRoles.find(
    (r) =>
      r.source?.slug === slug &&
      !!r.source?.language &&
      r.source.language !== language,
  );
  return other?.source?.language;
 }
 /**
 * Map one catalog role to the view model, computing its install status against
 * the workspace roles (via `catalogRoleInstallState`) and, for importable roles,
 * the other-language hint.
 */
 export function mapCatalogRoleToView(
  role: IAiRoleCatalogRole,
  workspaceRoles: IAiRole[],
  language: string,
 ): CatalogViewRole {
  const state = catalogRoleInstallState(role, workspaceRoles, language);
  const base = {
    slug: role.slug,
    emoji: role.emoji ?? undefined,
    name: role.name,
    description: role.description ?? "",
  };
  if (state.state === "update") {
    return {
      ...base,
      status: "update",
      version: state.fromVersion,
      newVersion: state.toVersion,
      installedRoleId: state.installed.id,
    };
  }
  if (state.state === "installed") {
    return {
      ...base,
      status: "installed",
      version: role.version,
      installedRoleId: state.installed.id,
    };
  }
  return {
    ...base,
    status: "import",
    version: role.version,
    installedLang: installedLangForRole(role.slug, workspaceRoles, language),
  };
 }
 /**
 * Map a whole bundle's catalog roles to the view model, preserving order.
 */
 export function mapBundleRolesToView(
  roles: IAiRoleCatalogRole[],
  workspaceRoles: IAiRole[],
  language: string,
 ): CatalogViewRole[] {
  return roles.map((r) => mapCatalogRoleToView(r, workspaceRoles, language));
 }
@@ -23,7 +23,7 @@
    "migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
    "migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
-    "pretest": "pnpm --filter @docmost/editor-ext build",
+    "pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build",
    "test": "jest",
    "test:int": "jest --config test/jest-integration.json",
    "test:watch": "jest --watch",
@@ -43,6 +43,7 @@
    "@clickhouse/client": "^1.18.2",
    "@docmost/mcp": "workspace:*",
    "@docmost/pdf-inspector": "1.9.6",
    "@docmost/prosemirror-markdown": "workspace:*",
    "@fastify/cookie": "^11.0.2",
    "@fastify/multipart": "^10.0.0",
    "@fastify/static": "^9.1.3",
@@ -175,7 +176,7 @@
      "/node_modules/"
    ],
    "transform": {
-      "happy-dom.+\\.js$": [
+      "(happy-dom.+|prosemirror-markdown/build/.+)\\.js$": [
        "babel-jest",
        {
          "presets": [
@@ -193,7 +194,7 @@
      "^.+\\.(t|j)sx?$": "ts-jest"
    },
    "transformIgnorePatterns": [
-      "/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0)(@|/))"
+      "/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@docmost/prosemirror-markdown)(@|/))"
    ],
    "collectCoverageFrom": [
      "**/*.(t|j)s"
@@ -204,7 +205,8 @@
      "^@docmost/db/(.*)$": "<rootDir>/database/$1",
      "^@docmost/transactional/(.*)$": "<rootDir>/integrations/transactional/$1",
      "^@docmost/ee/(.*)$": "<rootDir>/ee/$1",
-      "^src/(.*)$": "<rootDir>/$1"
+      "^src/(.*)$": "<rootDir>/$1",
      "^@tiptap/react$": "<rootDir>/../test/stubs/tiptap-react.js"
    }
  }
 }
@@ -43,7 +43,6 @@ import {
  Column,
  Status,
  addUniqueIdsToDoc,
  htmlToMarkdown,
  TransclusionSource,
  TransclusionReference,
  FootnoteReference,
@@ -51,6 +50,7 @@ import {
  FootnoteDefinition,
  PageEmbed,
 } from '@docmost/editor-ext';
 import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
 import { generateText, getSchema, JSONContent } from '@tiptap/core';
 import { generateHTML, generateJSON } from '../common/helpers/prosemirror/html';
 // @tiptap/html library works best for generating prosemirror json state but not HTML
@@ -239,6 +239,10 @@ export function prosemirrorNodeToYElement(node: any): Y.XmlElement | Y.XmlText {
 }
 export function jsonToMarkdown(tiptapJson: any): string {
-  const html = jsonToHtml(tiptapJson);
+  // Direct ProseMirror JSON -> Markdown via the canonical converter
-  return htmlToMarkdown(html);
+  // (`@docmost/prosemirror-markdown`) — no HTML intermediate, no second
  // editor-ext markdown layer. Same serializer as the page/space export and the
  // git-sync vault writer, so every server PM->MD path emits identical canonical
  // markdown (issue #345).
  return convertProseMirrorToMarkdown(tiptapJson);
 }
@@ -610,63 +610,6 @@ describe('AiAgentRolesService guards', () => {
      expect(repo.insert.mock.calls[0][0].name).toBe('Researcher (2)');
    });
    it('createdRoles lists the installed role (no renamedTo when not renamed)', async () => {
      const { service } = makeImportService({});
      const res = await service.importFromCatalog('ws-1', 'u1', dto());
      expect(res.createdRoles).toEqual([
        { slug: 'researcher', name: 'Researcher' },
      ]);
      expect(res.skippedRoles).toEqual([]);
    });
    it('createdRoles carries renamedTo on a rename', async () => {
      const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
      const { service } = makeImportService({ existing });
      const res = await service.importFromCatalog(
        'ws-1',
        'u1',
        dto({ conflict: 'rename' }),
      );
      expect(res.createdRoles).toEqual([
        { slug: 'researcher', name: 'Researcher', renamedTo: 'Researcher (2)' },
      ]);
      expect(res.skippedRoles).toEqual([]);
    });
    it('skippedRoles: already-installed slug carries reason "already-installed"', async () => {
      const existing = [
        makeRow({
          id: 'r-existing',
          name: 'Old researcher',
          source: { slug: 'researcher', language: 'en', version: 1 } as never,
        }),
      ];
      const { service } = makeImportService({ existing });
      const res = await service.importFromCatalog('ws-1', 'u1', dto());
      expect(res.skippedRoles).toEqual([
        {
          slug: 'researcher',
          name: 'Researcher',
          reason: 'already-installed',
        },
      ]);
      expect(res.createdRoles).toEqual([]);
    });
    it('skippedRoles: a name collision under conflict:skip carries reason "name-conflict"', async () => {
      const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
      const { service } = makeImportService({ existing });
      const res = await service.importFromCatalog(
        'ws-1',
        'u1',
        dto({ conflict: 'skip' }),
      );
      expect(res.skippedRoles).toEqual([
        { slug: 'researcher', name: 'Researcher', reason: 'name-conflict' },
      ]);
      expect(res.createdRoles).toEqual([]);
    });
    it('dto.slugs filters; an unknown slug becomes an error entry', async () => {
      const { service, repo } = makeImportService({
        bundleRoles: [catalogRole()],
@@ -734,15 +677,6 @@ describe('AiAgentRolesService guards', () => {
      // 'a' converged on the concurrent install (skip); 'b' imported; no errors.
      expect(res).toMatchObject({ created: 1, skipped: 1, renamed: 0 });
      expect(res.errors).toEqual([]);
      // The per-role list records 'a' as an already-installed skip (the UI reads
      // skippedRoles, not the counter, to render its plaque — assert the array,
      // not just the count).
      expect(res.skippedRoles).toContainEqual({
        slug: 'a',
        name: 'A',
        reason: 'already-installed',
      });
      expect(res.createdRoles.map((r) => r.slug)).toEqual(['b']);
      // Both inserts were attempted (the batch did not abort on the 23505).
      expect(repo.insert).toHaveBeenCalledTimes(2);
    });
@@ -305,16 +305,6 @@ export class AiAgentRolesService {
    skipped: number;
    renamed: number;
    errors: { slug: string; message: string }[];
    // Per-role lists alongside the counters (kept for back-compat). The redesigned
    // catalog UI needs the actual roles — which were created (and any rename) and
    // which were skipped and why — to render an inline result plaque with the
    // conflicting role's name and a "Rename & install" affordance.
    createdRoles: { slug: string; name: string; renamedTo?: string }[];
    skippedRoles: {
      slug: string;
      name: string;
      reason: 'name-conflict' | 'already-installed';
    }[];
  }> {
    const { file, versions } = await this.loadBundleById(
      dto.bundleId,
@@ -322,13 +312,6 @@ export class AiAgentRolesService {
    );
    const errors: { slug: string; message: string }[] = [];
    const createdRoles: { slug: string; name: string; renamedTo?: string }[] =
      [];
    const skippedRoles: {
      slug: string;
      name: string;
      reason: 'name-conflict' | 'already-installed';
    }[] = [];
    // Resolve the selected catalog roles (honor dto.slugs; flag unknown ones).
    let selected = file.roles;
@@ -368,27 +351,16 @@ export class AiAgentRolesService {
      // Already installed from the catalog in THIS language => skip (use
      // update-from-catalog). A different language of the same slug still imports.
      const installKey = `${role.slug}:${dto.language}`;
      const originalName = role.name.trim();
      if (installedKeys.has(installKey)) {
        skipped++;
        skippedRoles.push({
          slug: role.slug,
          name: originalName,
          reason: 'already-installed',
        });
        continue;
      }
-      let name = originalName;
+      let name = role.name.trim();
      let didRename = false;
      if (takenNames.has(name.toLowerCase())) {
        if (dto.conflict === 'skip') {
          skipped++;
          skippedRoles.push({
            slug: role.slug,
            name: originalName,
            reason: 'name-conflict',
          });
          continue;
        }
        // conflict === 'rename': find a free " (N)" suffix.
@@ -408,11 +380,6 @@ export class AiAgentRolesService {
        });
        created++;
        if (didRename) renamed++;
        createdRoles.push({
          slug: role.slug,
          name: originalName,
          ...(didRename ? { renamedTo: name } : {}),
        });
        takenNames.add(name.toLowerCase());
        installedKeys.add(installKey);
      } catch (err) {
@@ -424,11 +391,6 @@ export class AiAgentRolesService {
        // skipped (already installed) and continue; do NOT abort or error.
        if (isSourceUniqueViolation(err)) {
          skipped++;
          skippedRoles.push({
            slug: role.slug,
            name: originalName,
            reason: 'already-installed',
          });
          installedKeys.add(installKey);
          continue;
        }
@@ -445,7 +407,7 @@ export class AiAgentRolesService {
      }
    }
-    return { created, skipped, renamed, errors, createdRoles, skippedRoles };
+    return { created, skipped, renamed, errors };
  }
  /**
@@ -52,7 +52,9 @@ import {
  INTERNAL_LINK_REGEX,
  extractPageSlugId,
 } from '../../../integrations/export/utils';
-import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
+import { canonicalizeFootnotes } from '@docmost/editor-ext';
 import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
 import { normalizeForeignMarkdown } from '../../../integrations/import/utils/foreign-markdown';
 import { WatcherService } from '../../watcher/watcher.service';
 import { sql } from 'kysely';
 import { TransclusionService } from '../transclusion/transclusion.service';
@@ -1301,8 +1303,14 @@ export class PageService {
    switch (format) {
      case 'markdown': {
-        const html = await markdownToHtml(content as string);
+        // Canonical markdown -> ProseMirror JSON directly via
-        prosemirrorJson = htmlToJson(html as string);
+        // `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate,
        // no editor-ext markdown layer. Foreign markdown surfaces the strict
        // parser rejects (GFM `[^id]` reference footnotes) are normalized to the
        // canonical inline form first.
        prosemirrorJson = await markdownToProseMirror(
          normalizeForeignMarkdown(content as string),
        );
        break;
      }
      case 'html': {
@@ -0,0 +1,145 @@
 // export.service.ts imports the ESM-only @sindresorhus/slugify (not in jest's
 // transform allowlist). It is irrelevant to the markdown-serialization path under
 // test (only used for page-mention link slugs on the DB path), so it is mocked
 // out to keep the module graph loadable under ts-jest (mirrors the import specs).
 jest.mock('@sindresorhus/slugify', () => ({
  __esModule: true,
  default: (input: string) => String(input),
 }));
 import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
 import { ExportService } from './export.service';
 import { ExportFormat } from './dto/export-dto';
 /**
 * STEP 1 golden test for issue #345: server MARKDOWN export runs DIRECTLY through
 * the canonical converter (`convertProseMirrorToMarkdown`) — no HTML intermediate
 * and no `@docmost/editor-ext` markdown layer — so the emitted markdown is in the
 * canonical package forms and is byte-identical to the git-sync vault body.
 *
 * These are the goldens the swap has to satisfy: they assert the CANONICAL
 * surface (callout `> [!type]`, inline footnote `^[…]`, lossless image
 * `<!--img …-->`) rather than the old editor-ext forms (`:::type`, `[^id]`,
 * lossy `![alt](src)`).
 *
 * `exportPage(..., singlePage=false)` takes no DB path (no mention rewriting), so
 * the service is constructed with null collaborators and only the pure
 * PM -> Markdown path is exercised.
 */
 function makeService(): ExportService {
  return new ExportService(
    null as any, // pageRepo
    null as any, // pagePermissionRepo
    null as any, // db
    null as any, // storageService
    null as any, // environmentService
    null as any, // domainService
  );
 }
 // A representative page exercising the node types whose canonical markdown form
 // changed with the move off the editor-ext layer: callout, inline footnote, and a
 // lossless image carrying width/align attrs that the old layer dropped.
 const REPRESENTATIVE_DOC = {
  type: 'doc',
  content: [
    {
      type: 'paragraph',
      content: [
        { type: 'text', text: 'Body ' },
        { type: 'footnoteReference', attrs: { id: 'fn-1' } },
        { type: 'text', text: ' end.' },
      ],
    },
    {
      type: 'callout',
      attrs: { type: 'info', icon: null },
      content: [
        {
          type: 'paragraph',
          content: [{ type: 'text', text: 'Heads up' }],
        },
      ],
    },
    {
      type: 'image',
      attrs: {
        src: '/files/pic.png',
        alt: 'Pic',
        width: 320,
        align: 'left',
      },
    },
    {
      type: 'footnotesList',
      content: [
        {
          type: 'footnoteDefinition',
          attrs: { id: 'fn-1' },
          content: [
            {
              type: 'paragraph',
              content: [{ type: 'text', text: 'the note' }],
            },
          ],
        },
      ],
    },
  ],
 };
 describe('ExportService — markdown export via the canonical converter (#345)', () => {
  it('emits canonical callout, inline footnote and lossless image forms', async () => {
    const service = makeService();
    const md = (await service.exportPage(ExportFormat.Markdown, {
      title: '',
      content: REPRESENTATIVE_DOC,
    } as any)) as string;
    // Callout: Obsidian `> [!type]`, NOT the legacy `:::type`.
    expect(md).toContain('> [!info]');
    expect(md).not.toContain(':::');
    // Inline footnote: `^[…]`, NOT the reference `[^id]` form.
    expect(md).toContain('^[the note]');
    expect(md).not.toMatch(/\[\^/);
    // Lossless image: trailing `<!--img …-->` carrying the dropped attrs.
    expect(md).toContain('![Pic](/files/pic.png)');
    expect(md).toContain('<!--img');
    expect(md).toContain('"width":"320"');
    expect(md).toContain('"align":"left"');
  });
  it('export body is byte-identical to the git-sync vault serializer (export == vault)', async () => {
    const service = makeService();
    // A title-less page: exportPage prepends NO heading, so the whole output is
    // the page BODY — exactly what git-sync serializes (git-sync stores the title
    // in frontmatter / the filename, never as an in-body H1).
    const exported = (await service.exportPage(ExportFormat.Markdown, {
      title: '',
      content: REPRESENTATIVE_DOC,
    } as any)) as string;
    // The git-sync vault writer feeds this SAME converter (git-sync
    // `stabilizePageBody` = convertProseMirrorToMarkdown(content) at the
    // fixpoint). For an already-stable doc the single pass IS the fixpoint, so
    // the two are byte-identical by construction — assert it.
    const vaultBody = convertProseMirrorToMarkdown(REPRESENTATIVE_DOC);
    expect(exported).toBe(vaultBody);
  });
  it('prepends the page title as an H1 heading (the one documented export/vault delta)', async () => {
    const service = makeService();
    const md = (await service.exportPage(ExportFormat.Markdown, {
      title: 'My Page',
      content: { type: 'doc', content: [] },
    } as any)) as string;
    // Export makes standalone files, so it prepends the title as an H1. This is
    // the ONE deliberate difference from the vault body (which carries the title
    // in frontmatter). The body below the heading still serializes canonically.
    expect(md.startsWith('# My Page')).toBe(true);
  });
 });
@@ -37,7 +37,7 @@ import {
  getAttachmentIds,
  getProsemirrorContent,
 } from '../../common/helpers/prosemirror/utils';
-import { htmlToMarkdown } from '@docmost/editor-ext';
+import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
 type AllowedAttachment = { id: string; fileName: string; filePath: string };
@@ -79,9 +79,8 @@ export class ExportService {
      prosemirrorJson.content.unshift(titleNode);
    }
    const pageHtml = jsonToHtml(prosemirrorJson);
    if (format === ExportFormat.HTML) {
      const pageHtml = jsonToHtml(prosemirrorJson);
      return `<!DOCTYPE html>
      <html>
        <head>
@@ -92,11 +91,14 @@ export class ExportService {
    }
    if (format === ExportFormat.Markdown) {
-      const newPageHtml = pageHtml.replace(
+      // Direct ProseMirror JSON -> Markdown via the canonical converter
-        /<colgroup[^>]*>[\s\S]*?<\/colgroup>/gim,
+      // (`@docmost/prosemirror-markdown`). This is the SAME serializer the
-        '',
+      // git-sync vault writer feeds (see git-sync `stabilizePageBody`), so an
-      );
+      // exported page body is byte-identical to its vault representation — no
-      return htmlToMarkdown(newPageHtml);
+      // HTML intermediate, no second markdown layer, no format drift (issue
      // #345). The old `<colgroup>` scrub is gone with the HTML step: the
      // converter emits GFM tables directly and never produces `<colgroup>`.
      return convertProseMirrorToMarkdown(prosemirrorJson);
    }
    return;
@@ -17,6 +17,22 @@ jest.mock('image-dimensions', () => ({
  __esModule: true,
  imageDimensionsFromData: () => undefined,
 }));
 // FileImportTaskService -> PageService -> collaboration.gateway ->
 // metrics.registry imports `prom-client`, which is not resolvable in this
 // workspace's node_modules (types-only stub, no runtime entry). Metrics are
 // disabled on this path, so a virtual no-op mock keeps the module graph loadable.
 jest.mock(
  'prom-client',
  () => ({
    collectDefaultMetrics: () => undefined,
    Registry: class {},
    Histogram: class {},
    Gauge: class {},
    Counter: class {},
    Summary: class {},
  }),
  { virtual: true },
 );
 import { promises as fs } from 'fs';
 import * as os from 'os';
@@ -26,14 +42,17 @@ import { ImportService } from './import.service';
 /**
 * Binding test for issue #228 / review #5: FileImportTaskService.processGenericImport
- * is a NON-editor write path (markdownToHtml -> processHTML -> JSON, never runs
+ * is a NON-editor write path, so a zip-imported `.md` page ends up with canonical
- * footnoteSyncPlugin), so it canonicalizes footnotes before persisting. This pins
+ * footnotes before persisting: ordered by first reference, reused refs deduped,
- * that binding — the same one import.service has a spec for — which previously had
+ * orphan definitions dropped.
 * NO spec at all.
 *
- * The markdown -> HTML -> ProseMirror conversion is REAL (a real ImportService,
+ * Since #345 the `.md` parse runs `normalizeForeignMarkdown` ->
- * its createYdoc stubbed); the filesystem is a real temp dir with one .md file;
+ * `markdownToProseMirror` -> `jsonToHtml` (feeding the shared HTML attachment /
- * the DB transaction is stubbed to capture the persisted page content.
+ * link pipeline) -> `processHTML` -> `canonicalizeFootnotes`. The parser assigns
 * fresh `fn-*` ids, so we assert by definition BODY order rather than the source
 * labels. The conversion is REAL (a real ImportService, its createYdoc stubbed);
 * the filesystem is a real temp dir with one .md file; the DB transaction is
 * stubbed to capture the persisted page content.
 */
 // Out-of-order references (c, a, b), a REUSED reference ([^a] twice), and an
@@ -49,13 +68,14 @@ const MARKDOWN = [
  '[^z]: orphan note',
 ].join('\n');
-function footnoteListIds(content: any): string[] {
+/** Definition body texts of the (single) footnotesList, in list order. */
 function footnoteListBodies(content: any): string[] {
  const list = (content?.content ?? []).find(
    (n: any) => n.type === 'footnotesList',
  );
  return (list?.content ?? [])
    .filter((n: any) => n.type === 'footnoteDefinition')
-    .map((n: any) => n.attrs?.id);
+    .map((n: any) => n.content?.[0]?.content?.[0]?.text);
 }
 // A permissive chainable stub for the spaces lookup (selectFrom(...).select(...)
@@ -71,12 +91,17 @@ function chainable(result: any): any {
  return proxy;
 }
-describe('FileImportTaskService.processGenericImport — footnote canonicalization (#228)', () => {
+/**
-  it('orders footnotes by first reference, dedupes reuse, and drops orphans on zip import', async () => {
+ * Run one markdown file through the REAL zip-import pipeline
 * (`processGenericImport` -> `markdownToProseMirror` -> `jsonToHtml` ->
 * `processHTML`/`htmlToJson`) and return the persisted page `content`. This is
 * the server-specific PM->HTML->PM hop that the package's own PM<->MD tests do
 * NOT cover.
 */
 async function runZipImport(markdown: string): Promise<any> {
  const extractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fit-canon-'));
-    await fs.writeFile(path.join(extractDir, 'note.md'), MARKDOWN, 'utf-8');
+  await fs.writeFile(path.join(extractDir, 'note.md'), markdown, 'utf-8');
    // Real ImportService for the html -> JSON conversion; stub the yjs encode.
  const importService = new ImportService(
    {} as any,
    {} as any,
@@ -104,21 +129,15 @@ describe('FileImportTaskService.processGenericImport — footnote canonicalizati
  const importAttachmentService = {
    processAttachments: async ({ html }: any) => html,
  };
    const backlinkRepo = { insertBacklink: jest.fn() };
    const eventEmitter = { emit: jest.fn() };
    const auditService = { logBatchWithContext: jest.fn() };
    const pageService = { nextPagePosition: async () => 'a0' };
  const service = new FileImportTaskService(
    {} as any, // storageService
    importService as any,
-      pageService as any,
+    { nextPagePosition: async () => 'a0' } as any,
-      backlinkRepo as any,
+    { insertBacklink: jest.fn() } as any,
    db,
    importAttachmentService as any,
-      eventEmitter as any,
+    { emit: jest.fn() } as any,
-      auditService as any,
+    { logBatchWithContext: jest.fn() } as any,
  );
  const fileTask: any = {
@@ -131,20 +150,68 @@ describe('FileImportTaskService.processGenericImport — footnote canonicalizati
  try {
    await service.processGenericImport({ extractDir, fileTask });
    expect(captured).toBeTruthy();
-      const content = captured.content;
+    return captured.content;
-      // Reference order is c, a, b (NOT the markdown definition order a, b, c).
+  } finally {
-      expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']);
+    await fs.rm(extractDir, { recursive: true, force: true });
  }
 }
 /** Find the first node of a given type anywhere in a PM content tree. */
 function findFirst(node: any, type: string): any {
  if (!node || typeof node !== 'object') return null;
  if (node.type === type) return node;
  for (const child of node.content ?? []) {
    const hit = findFirst(child, type);
    if (hit) return hit;
  }
  return null;
 }
 describe('FileImportTaskService.processGenericImport — footnote canonicalization (#228)', () => {
  it('orders footnotes by first reference, dedupes reuse, and drops orphans on zip import', async () => {
    const content = await runZipImport(MARKDOWN);
    // Definitions ordered by FIRST REFERENCE (C, A, B), NOT the markdown
    // definition order (A, B, C). Ids are the parser's fresh `fn-*`, so pin
    // the BODIES.
    expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
    // Orphan [^z] dropped; reused [^a] collapses to one definition; one list.
-      expect(footnoteListIds(content)).not.toContain('z');
+    expect(footnoteListBodies(content)).not.toContain('orphan note');
    const lists = (content.content ?? []).filter(
      (n: any) => n.type === 'footnotesList',
    );
    expect(lists).toHaveLength(1);
-      expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1);
+    expect(
-    } finally {
+      footnoteListBodies(content).filter((b) => b === 'note A'),
-      await fs.rm(extractDir, { recursive: true, force: true });
+    ).toHaveLength(1);
-    }
+  });
  // #345 F4: the zip path routes markdown through jsonToHtml -> processHTML ->
  // htmlToJson (the shared HTML attachment pipeline). #345's headline is LOSSLESS
  // image width/align via the `<!--img {...}-->` comment; a callout carries its
  // `type`. This asserts those survive the PM->HTML->PM hop — the one hop the
  // package's PM<->MD suite does not exercise.
  it('preserves image width/align and callout type through the PM->HTML->PM hop', async () => {
    const md = [
      '# Doc',
      '',
      '![a picture](https://example.com/i.png) <!--img {"width":"320","align":"left"}-->',
      '',
      ':::warning',
      'Careful now.',
      ':::',
    ].join('\n');
    const content = await runZipImport(md);
    const image = findFirst(content, 'image');
    expect(image).toBeTruthy();
    // The lossless sizing/alignment must survive the HTML hop.
    expect(String(image.attrs?.width)).toBe('320');
    expect(image.attrs?.align).toBe('left');
    const callout = findFirst(content, 'callout');
    expect(callout).toBeTruthy();
    expect(callout.attrs?.type).toBe('warning');
  });
 });
@@ -1,6 +1,9 @@
 import { Inject, Injectable, Logger } from '@nestjs/common';
 import * as path from 'path';
-import { jsonToText } from '../../../collaboration/collaboration.util';
+import {
  jsonToHtml,
  jsonToText,
 } from '../../../collaboration/collaboration.util';
 import { InjectKysely } from 'nestjs-kysely';
 import { KyselyDB } from '@docmost/db/types/kysely.types';
 import {
@@ -18,9 +21,11 @@ import { generateSlugId } from '../../../common/helpers';
 import { v7 } from 'uuid';
 import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
 import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
-import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
+import { canonicalizeFootnotes } from '@docmost/editor-ext';
 import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
 import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
 import { formatImportHtml } from '../utils/import-formatter';
 import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
 import {
  buildAttachmentCandidates,
  collectMarkdownAndHtmlFiles,
@@ -461,7 +466,18 @@ export class FileImportTaskService {
              content = await fs.readFile(absPath, 'utf-8');
              if (page.fileExtension.toLowerCase() === '.md') {
-                content = await markdownToHtml(content);
+                // Parse markdown with the single canonical converter
                // (`@docmost/prosemirror-markdown`), after normalizing foreign
                // reference footnotes, then serialize to HTML so the shared HTML
                // pipeline below (processAttachments + formatImportHtml +
                // processHTML) keeps handling `.md` and `.html` imports
                // uniformly. The markdown PARSE no longer goes through the
                // editor-ext markdown layer (issue #345) — the drift source is
                // gone. The PM -> HTML -> PM hop that follows is lossless
                // plumbing for attachment/link resolution, NOT a second parse.
                content = jsonToHtml(
                  await markdownToProseMirror(normalizeForeignMarkdown(content)),
                );
              }
            } catch (err: any) {
              if (err?.code === 'ENOENT') {
@@ -500,10 +516,12 @@ export class FileImportTaskService {
              this.importService.extractTitleAndRemoveHeading(pmState);
            // Canonicalize footnote topology on this non-editor write path
-            // (markdownToHtml/processHTML never runs footnoteSyncPlugin), so a
+            // (the HTML pipeline's processHTML never runs footnoteSyncPlugin), so
-            // zip-imported page's footnotes are reference-ordered, deduped, and
+            // a zip-imported page's footnotes are reference-ordered, deduped, and
            // orphan-free like the editor's invariant (issue #228). Pure +
-            // idempotent + shape-safe; a footnote-free doc is unchanged.
+            // idempotent + shape-safe; a footnote-free doc is unchanged. (For a
            // `.md` file the package parser already yields canonical footnotes,
            // so this is a no-op there.)
            // (Future consolidation, architecture B: like import.service, this
            // path persists directly rather than via PageService — a shared
            // "prepare JSON for persist" helper would centralize this call.)
@@ -12,13 +12,19 @@ import { canonicalizeFootnotes } from '@docmost/editor-ext';
 /**
 * Integration-ish test for the USER-FACING markdown import path
- * (`ImportService.importPage`). It exercises the REAL markdown -> HTML -> JSON
+ * (`ImportService.importPage`). It exercises the REAL markdown -> ProseMirror
- * conversion and asserts that the stored page content has its footnotes
+ * conversion and asserts the stored page's footnotes are canonical: ordered by
- * canonicalized — the gap that issue #228 fixes: the import path builds
+ * FIRST REFERENCE (not markdown definition order), reused references deduped to a
- * ProseMirror JSON directly (never running the editor's footnoteSyncPlugin), so
+ * single definition, and orphan definitions dropped.
- * before this wiring the stored footnotes kept the markdown's physical
+ *
- * definition order (out of order vs. references), retained orphan definitions,
+ * Since #345 the markdown parse runs through the canonical package
- * and did not collapse reused references.
+ * (`normalizeForeignMarkdown` -> `markdownToProseMirror`), which owns this
 * canonicalization: the input's GFM `[^id]` reference footnotes are normalized to
 * inline `^[…]`, and the parser assigns fresh sequential ids (`fn-*`) in
 * reference order while merging identical bodies — so we assert by definition
 * BODY order, not by the source labels. `canonicalizeFootnotes` remains wired as
 * an idempotent safety net (issue #228) and is a no-op on this already-canonical
 * output.
 *
 * The DB/ydoc side-effects are stubbed: `getNewPagePosition` (DB query) and
 * `createYdoc` (Yjs encode) are spied, and `pageRepo.insertPage` captures the
@@ -67,24 +73,14 @@ function makeService() {
 }
 /** List the footnote-definition ids of the (single) footnotesList, in order. */
-function footnoteListIds(content: any): string[] {
+/** Definition body texts of the (single) footnotesList, in list order. */
 function footnoteListBodies(content: any): string[] {
  const list = (content.content ?? []).find(
    (n: any) => n.type === 'footnotesList',
  );
-  if (!list) return [];
+  return (list?.content ?? [])
  return (list.content ?? [])
    .filter((n: any) => n.type === 'footnoteDefinition')
-    .map((n: any) => n.attrs?.id);
+    .map((n: any) => n.content?.[0]?.content?.[0]?.text);
 }
 function definitionText(content: any, id: string): string | undefined {
  const list = (content.content ?? []).find(
    (n: any) => n.type === 'footnotesList',
  );
  const def = (list?.content ?? []).find(
    (n: any) => n.type === 'footnoteDefinition' && n.attrs?.id === id,
  );
  return def?.content?.[0]?.content?.[0]?.text;
 }
 describe('ImportService.importPage — footnote canonicalization (#228)', () => {
@@ -101,23 +97,23 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
    const content = getCaptured().content;
    expect(content).toBeTruthy();
-    // Reference order is c, a, b (NOT the markdown definition order a, b, c).
+    // Definitions ordered by FIRST REFERENCE (C, A, B) — NOT the markdown
-    expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']);
+    // definition order (A, B, C) — with the orphan [^z] dropped and the reused
-
+    // [^a] collapsed to a single definition. (Ids are the parser's fresh `fn-*`,
-    // Definitions preserved and attached to the right ids.
+    // so we pin the BODIES.)
-    expect(definitionText(content, 'c')).toBe('note C');
+    expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
    expect(definitionText(content, 'a')).toBe('note A');
    expect(definitionText(content, 'b')).toBe('note B');
    // Orphan definition [^z] is dropped.
-    expect(footnoteListIds(content)).not.toContain('z');
+    expect(footnoteListBodies(content)).not.toContain('orphan note');
    // Reused [^a] yields exactly ONE definition, and exactly one list.
    const lists = (content.content ?? []).filter(
      (n: any) => n.type === 'footnotesList',
    );
    expect(lists).toHaveLength(1);
-    expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1);
+    expect(
      footnoteListBodies(content).filter((b) => b === 'note A'),
    ).toHaveLength(1);
  });
  it('is idempotent: canonicalizing the stored output again is a no-op', async () => {
@@ -134,6 +130,6 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
    // time must not change it (safe to wire into every write path).
    const second = canonicalizeFootnotes(stored);
    expect(second).toEqual(stored);
-    expect(footnoteListIds(second)).toEqual(['c', 'a', 'b']);
+    expect(footnoteListBodies(second)).toEqual(['note C', 'note A', 'note B']);
  });
 });
@@ -17,7 +17,9 @@ import {
 import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
 import { TiptapTransformer } from '@hocuspocus/transformer';
 import * as Y from 'yjs';
-import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
+import { canonicalizeFootnotes } from '@docmost/editor-ext';
 import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
 import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
 import {
  FileTaskStatus,
  FileTaskType,
@@ -85,11 +87,13 @@ export class ImportService {
    const extracted = this.extractTitleAndRemoveHeading(prosemirrorState);
    const title = extracted.title;
-    // Imported markdown/HTML is built via markdownToHtml -> htmlToJson, which
+    // The markdown path now canonicalizes footnotes itself (the package parser),
-    // never runs the editor's footnoteSyncPlugin, so the footnote topology keeps
+    // but the HTML path (processHTML -> htmlToJson) does NOT run the editor's
-    // the source's PHYSICAL definition order (out of order vs. references),
+    // footnoteSyncPlugin, so an imported HTML doc can keep its source's PHYSICAL
-    // retains orphan definitions, and is not deduped. Canonicalize before
+    // definition order (out of order vs. references), retain orphan definitions,
-    // persisting so the stored page matches the editor's invariant (issue #228).
+    // and not be deduped. Canonicalize before persisting so the stored page
    // matches the editor's invariant (issue #228); it is an idempotent no-op on
    // the already-canonical markdown output.
    // Pure + idempotent + shape-safe: a doc with no footnotes is unchanged.
    // (Future consolidation, architecture B: this import path persists directly
    // via pageRepo.insertPage rather than through PageService.createPage, so the
@@ -133,12 +137,15 @@ export class ImportService {
  }
  async processMarkdown(markdownInput: string): Promise<any> {
-    try {
+    // Canonical markdown -> ProseMirror JSON directly via
-      const html = await markdownToHtml(markdownInput);
+    // `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate and no
-      return this.processHTML(html);
+    // second editor-ext markdown layer. Foreign markdown surfaces the strict
-    } catch (err) {
+    // canonical parser does not accept (GFM `[^id]` reference footnotes) are
-      throw err;
+    // rewritten to the canonical inline form by `normalizeForeignMarkdown` first.
-    }
+    // The HTML-cleanup pass (`normalizeImportHtml`) is intentionally skipped here:
    // it targets foreign *HTML* (Notion/XWiki), which only ever arrives on the
    // `.html` path (`processHTML`), never as canonical markdown.
    return markdownToProseMirror(normalizeForeignMarkdown(markdownInput));
  }
  async processHTML(htmlInput: string): Promise<any> {
@@ -0,0 +1,218 @@
 import {
  convertProseMirrorToMarkdown,
  markdownToProseMirror,
 } from '@docmost/prosemirror-markdown';
 import { normalizeForeignMarkdown } from './foreign-markdown';
 /**
 * STEP 2 goldens for issue #345: the foreign-markdown normalizer that runs at the
 * import boundary BEFORE the strict canonical parser (`markdownToProseMirror`).
 *
 * Two layers:
 *  1. PURE string→string cases pinning the normalizer's own behavior (GFM
 *     reference footnotes → inline `^[…]`).
 *  2. END-TO-END acceptance: for a foreign corpus, `normalizeForeignMarkdown`
 *     then `markdownToProseMirror` then `convertProseMirrorToMarkdown` must leave
 *     NO literal `[^id]` / `:::` garbage in the document and must re-export in the
 *     canonical forms.
 */
 describe('normalizeForeignMarkdown — GFM reference footnotes', () => {
  it('inlines a single-line reference footnote and drops its definition', () => {
    const out = normalizeForeignMarkdown(
      'A note[^1] here.\n\n[^1]: The definition.',
    );
    expect(out).toBe('A note^[The definition.] here.\n');
  });
  it('inlines every reference to a reused id (downstream dedups)', () => {
    const out = normalizeForeignMarkdown(
      'X[^a] and Y[^a].\n\n[^a]: shared.',
    );
    expect(out).toBe('X^[shared.] and Y^[shared.].\n');
  });
  it('joins indented continuation lines of a definition with a space', () => {
    const out = normalizeForeignMarkdown(
      'See[^n].\n\n[^n]: line one\n    line two',
    );
    expect(out).toBe('See^[line one line two].\n');
  });
  it('never rewrites a reference inside a fenced code block', () => {
    const out = normalizeForeignMarkdown(
      '```\ncode[^1] here\n```\n\n[^1]: def.',
    );
    expect(out).toContain('code[^1] here');
    // The (now orphaned) definition line is still removed.
    expect(out).not.toContain('[^1]: def.');
  });
  it('never rewrites a reference inside an INLINE-code span (backticks)', () => {
    // The `[^1]` inside backticks is literal code and must survive verbatim;
    // the one outside is rewritten. (Bug #1: only fenced blocks were protected.)
    const out = normalizeForeignMarkdown(
      'Use `arr[^1]` in code but note[^1] in prose.\n\n[^1]: def.',
    );
    expect(out).toBe('Use `arr[^1]` in code but note^[def.] in prose.\n');
  });
  it('escapes brackets in a body so an unbalanced ] cannot truncate the footnote', () => {
    // A foreign definition body with a stray `]` would, unescaped, close the
    // canonical `^[...]` early and leak the tail as text (bug #2). The body's
    // brackets are backslash-escaped so the footnote stays whole.
    const out = normalizeForeignMarkdown(
      'Ref[^1] here.\n\n[^1]: see item ] and [more] later',
    );
    expect(out).toBe('Ref^[see item \\] and \\[more\\] later] here.\n');
    // The tokenizer must see exactly one unescaped closing bracket (our own).
    expect(out.match(/(?<!\\)\]/g)).toHaveLength(1);
  });
  it('leaves a reference with no matching definition literal (no body to inline)', () => {
    const out = normalizeForeignMarkdown('Dangling[^x] ref.');
    expect(out).toBe('Dangling[^x] ref.');
  });
  it('returns the input unchanged when there are no reference footnotes', () => {
    const md = '# Title\n\nJust text with `inline code` and a [link](/x).';
    expect(normalizeForeignMarkdown(md)).toBe(md);
  });
  it('does NOT touch callout surfaces — the canonical parser handles them', () => {
    const callouts = ':::info\nHi\n:::\n\n> [!warning]\n> Careful';
    expect(normalizeForeignMarkdown(callouts)).toBe(callouts);
  });
  it('strips a leading YAML front-matter block (Obsidian/Hugo/git-sync files)', () => {
    const out = normalizeForeignMarkdown(
      '---\ntitle: My Page\ntags: [a, b]\n---\n\n# Heading\n\nBody.',
    );
    expect(out).toBe('# Heading\n\nBody.');
    // The front-matter must not leak into the body as a setext heading.
    expect(out).not.toContain('title: My Page');
    expect(out).not.toContain('---');
  });
  it('does not strip a horizontal rule that is not leading front-matter', () => {
    const md = 'Intro paragraph.\n\n---\n\nAfter the rule.';
    expect(normalizeForeignMarkdown(md)).toBe(md);
  });
  it('is linear on a document with thousands of definitions (no quadratic blowup)', () => {
    // F2(a): the pass-2 rewrite must be O(text), not O(text × defs). Build a
    // pathological doc (many defs + many plain text lines) and assert it
    // completes well under a second — a quadratic implementation took ~14s.
    const N = 4000;
    const refs = Array.from({ length: N }, (_, i) => `line ${i} plain text`).join('\n');
    const defs = Array.from({ length: N }, (_, i) => `[^n${i}]: def ${i}`).join('\n');
    const doc = `start[^n0] and[^n${N - 1}] end\n\n${refs}\n\n${defs}`;
    const t0 = Date.now();
    const out = normalizeForeignMarkdown(doc);
    const elapsed = Date.now() - t0;
    expect(elapsed).toBeLessThan(2000);
    // Sanity: the two real references were still inlined.
    expect(out).toContain('^[def 0]');
    expect(out).toContain(`^[def ${N - 1}]`);
  });
  it('is bounded on a long unclosed backtick run (no inline-split ReDoS)', () => {
    // F2(b): a huge unterminated backtick run must not cause quadratic
    // backtracking in the inline-code split. Oversized lines skip the split
    // entirely (left untouched), so this returns promptly.
    const line = 'x' + '`'.repeat(200000);
    const doc = `${line}\n\n[^1]: def`;
    const t0 = Date.now();
    normalizeForeignMarkdown(doc);
    expect(Date.now() - t0).toBeLessThan(2000);
  });
  it('does not crash or slow down on thousands of prefix-chain definition ids', () => {
    // F7: the rewrite must use a FIXED generic scanner, not an alternation built
    // from the ids. A `(a|aa|aaa|…)` alternation over prefix-chain ids blows the
    // V8 regex compiler (FATAL RegExpCompiler Allocation failed — uncatchable,
    // kills the process). A fixed scanner has no id-dependent compilation cost.
    const N = 4000;
    const ids = Array.from({ length: N }, (_, i) => 'a'.repeat(i + 1));
    const defs = ids.map((id) => `[^${id}]: body ${id.length}`).join('\n');
    const doc = `ref[^${ids[0]}] and[^${ids[N - 1]}] end\n\n${defs}`;
    const t0 = Date.now();
    const out = normalizeForeignMarkdown(doc);
    expect(Date.now() - t0).toBeLessThan(2000);
    // Prefix disambiguation is correct: [^a] and [^aaaa...] inline their OWN body.
    expect(out).toContain('^[body 1]');
    expect(out).toContain(`^[body ${N}]`);
  });
  it('strips a CRLF (Windows) front-matter block, not just LF', () => {
    // F9: the line-anchored regex needs LF after the opening `---`, so a Windows
    // file (`---\r\n…`) would slip past the strip and leak the front-matter into
    // the body. normalizeForeignMarkdown normalizes CRLF -> LF first.
    const out = normalizeForeignMarkdown(
      '---\r\ntitle: Foo\r\ntags: [a]\r\n---\r\n\r\n# Heading\r\n\r\nBody.',
    );
    expect(out).toBe('# Heading\n\nBody.');
    expect(out).not.toContain('title: Foo');
    expect(out).not.toContain('---');
  });
  it('strips front-matter whose value contains a triple-dash (line-anchored)', () => {
    // F8: the block must close only on a `\n---` LINE, not the first inline
    // `---`. A value like `title: Q1 --- Q2` must not truncate the front-matter
    // and leak the rest (author/closing ---) into the body.
    const out = normalizeForeignMarkdown(
      '---\ntitle: Q1 --- Q2 results\nauthor: bob\n---\n\nReal body.',
    );
    expect(out).toBe('Real body.');
    expect(out).not.toContain('author: bob');
    expect(out).not.toContain('Q2 results');
  });
 });
 describe('foreign markdown import acceptance (normalizer + canonical parser)', () => {
  const FOREIGN = [
    '# Doc',
    '',
    'Body refs [^c] and [^a] and [^b] and again [^a].',
    '',
    ':::info',
    'A legacy callout.',
    ':::',
    '',
    '| h1 | h2 |',
    '| --- | --- |',
    '| 1 | 2 |',
    '',
    '[^a]: note A',
    '[^b]: note B',
    '[^c]: note C',
    '[^z]: orphan note',
  ].join('\n');
  it('leaves no literal [^id] or ::: in the imported doc and re-exports canonically', async () => {
    const normalized = normalizeForeignMarkdown(FOREIGN);
    const doc = await markdownToProseMirror(normalized);
    const reexport = convertProseMirrorToMarkdown(doc);
    // No foreign garbage leaks into the document.
    expect(reexport).not.toMatch(/\[\^/); // no reference footnote refs/defs
    expect(reexport).not.toContain(':::'); // no legacy callout fences
    // Canonical forms are present.
    expect(reexport).toContain('^[note C]');
    expect(reexport).toContain('> [!info]');
    expect(reexport).toContain('| h1 | h2 |');
    // Footnotes: ordered by first reference (C, A, B), reused [^a] deduped to one,
    // orphan [^z] dropped (it had no reference after normalization).
    const list = doc.content.find((n: any) => n.type === 'footnotesList');
    const bodies = list.content.map(
      (d: any) => d.content[0].content[0].text,
    );
    expect(bodies).toEqual(['note C', 'note A', 'note B']);
    expect(bodies).not.toContain('orphan note');
    expect(
      doc.content.filter((n: any) => n.type === 'footnotesList'),
    ).toHaveLength(1);
  });
 });
@@ -0,0 +1,265 @@
 /**
 * Foreign-markdown normalizer — an input-liberal / output-canonical adapter that
 * runs at the IMPORT boundary, BEFORE the canonical parser
 * (`markdownToProseMirror` from `@docmost/prosemirror-markdown`).
 *
 * The canonical parser is deliberately STRICT: it only understands Docmost's
 * canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
 * inline footnotes `^[body]`, lossless `![alt](src) <!--img {...}-->` images, …).
 * Import, however, ingests FOREIGN files (GitHub/GFM, Notion, old Docmost
 * exports). Those use surfaces the canonical parser does not accept, most notably
 * GitHub-flavoured *reference* footnotes:
 *
 *     Text with a note[^1] and another[^long].
 *
 *     [^1]: The first definition.
 *     [^long]: A second one.
 *
 * Left untouched, the parser does NOT recognise `[^id]` (it only parses `^[body]`),
 * so the reference leaks as literal text — and worse, the trailing `[^id]: def`
 * line is a valid CommonMark *link-reference definition*, so `[^id]` is silently
 * rendered as a bogus link. This normalizer rewrites reference footnotes into the
 * canonical inline form so the parser materialises real footnote nodes.
 *
 * This is a TEXT pre-pass, NOT a second parser fork: it does not re-implement any
 * converter logic. Callout surfaces (`:::type` and `> [!type]`) are intentionally
 * NOT touched here — the canonical parser already accepts BOTH natively (its
 * `preprocessCallouts` pass), so normalizing them would be redundant and would
 * only risk degrading the parser's nesting/code-fence-aware handling.
 */
 /** Matches a fenced code block delimiter (``` or ~~~), capturing the marker run. */
 const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
 /**
 * Matches a GFM footnote DEFINITION line: `[^id]: body`. The id is any run of
 * non-`]` characters; the body is the remainder of the line (possibly empty).
 */
 const FOOTNOTE_DEF_RE = /^\[\^([^\]]+)\]:[ \t]?(.*)$/;
 /** True when a line is a code-fence delimiter that toggles fenced-code state. */
 function fenceMarker(line: string): string | null {
  const m = line.match(CODE_FENCE_RE);
  return m ? m[2] : null;
 }
 /** True when a line is indented (leading space/tab) and not blank — a continuation. */
 function isIndentedContinuation(line: string): boolean {
  return /^[ \t]+\S/.test(line);
 }
 function escapeRegExp(value: string): string {
  return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
 }
 /**
 * Backslash-escape any square bracket in a footnote body before it is wrapped in
 * `^[...]`. The canonical inline-footnote tokenizer scans the body with bracket
 * balancing and closes on the first UNMATCHED `]`, so an unbalanced bracket in a
 * foreign definition (e.g. `[^1]: see item ] later`) would otherwise truncate the
 * footnote and leak the tail as literal text. Escaping every `[`/`]` makes the
 * body an inert run of characters — the tokenizer then closes only on our own
 * closing `]`. (A balanced `[link](url)` inside a body still round-trips because
 * the escaped form renders the literal brackets, which is the safe reading for a
 * footnote body; the alternative — brittle balance tracking — risks worse.)
 */
 function escapeFootnoteBody(body: string): string {
  return body.replace(/[[\]]/g, '\\$&');
 }
 /**
 * Rewrite every `[^id]` reference on a line to its `^[body]` form, but ONLY in the
 * text OUTSIDE inline-code spans. A `[^id]` inside backticks is literal code
 * content and must be preserved verbatim (a footnote ref never lives inside code).
 * We split the line on inline-code spans (paired backtick runs) and rewrite only
 * the non-code segments.
 */
 // Above this length a single line is not split into inline-code spans (see
 // below). A genuine markdown line carrying a footnote reference is never tens of
 // KB; the cap only bypasses the inline-code protection for pathological lines.
 const INLINE_SPLIT_MAX_LINE = 8192;
 function rewriteRefsOutsideInlineCode(
  line: string,
  replace: (text: string) => string,
 ): string {
  // The inline-code split alternation `(`+)(?:(?!\1)[\s\S])*\1` backtracks
  // quadratically on a long UNCLOSED backtick run (its middle can consume the
  // rest of the line, then fail to find a closing run and retry from each
  // position). On an untrusted import this is a request-thread ReDoS. A real
  // footnote line is short, so for an oversized line we skip the inline-code
  // protection entirely and leave the line UNTOUCHED (rewriting it wholesale
  // could corrupt a `[^id]` that legitimately lives inside inline code). This is
  // a conservative bypass: an over-8KB line simply does not get its reference
  // footnotes inlined — acceptable for a pathological input.
  if (line.length > INLINE_SPLIT_MAX_LINE) return line;
  // Alternation: an inline-code span (one or more backticks, then anything up to
  // the SAME run of backticks) OR a run of non-backtick text. Unterminated
  // backticks fall through as ordinary text (matched by the second branch on the
  // leftover), so a stray backtick never swallows the rest of the line.
  const parts = line.match(/(`+)(?:(?!\1)[\s\S])*\1|[^`]+|`+/g);
  if (!parts) return line;
  return parts
    .map((seg) => (seg.startsWith('`') ? seg : replace(seg)))
    .join('');
 }
 /**
 * Convert GFM reference footnotes (`[^id]` + `[^id]: def`) into canonical inline
 * footnotes (`^[def]`).
 *
 * - Definitions are collected first (a leading `[^id]: text` line plus any
 *   immediately-following indented continuation lines, joined with a space) and
 *   removed from the output.
 * - Each in-text reference `[^id]` for which a definition was found is replaced by
 *   `^[def]`. References with no matching definition are left literal (there is no
 *   body to inline; the parser fails them open the same way).
 * - Code is respected on both passes: `[^id]` inside a fenced ``` / ~~~ block is
 *   never rewritten and a `[^id]:` line inside a fence is never a definition; and
 *   on the rewrite pass a `[^id]` inside an INLINE-code span (backticks) is left
 *   literal too.
 * - The inlined body is bracket-escaped so an unbalanced `[`/`]` in a foreign
 *   definition cannot truncate the resulting `^[...]` footnote.
 *
 * Deduplication / reference-ordering / orphan-dropping of the resulting footnotes
 * is handled downstream by the canonical parser (`assembleFootnotes`); this pass
 * only changes the surface syntax.
 */
 function convertReferenceFootnotes(markdown: string): string {
  const lines = markdown.split('\n');
  // Pass 1: collect definitions and mark their lines for removal.
  const defs = new Map<string, string>();
  const dropped = new Array<boolean>(lines.length).fill(false);
  let inFence = false;
  let fence = '';
  for (let i = 0; i < lines.length; i++) {
    const line = lines[i];
    const marker = fenceMarker(line);
    if (inFence) {
      if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
        inFence = false;
        fence = '';
      }
      continue;
    }
    if (marker) {
      inFence = true;
      fence = marker;
      continue;
    }
    const def = line.match(FOOTNOTE_DEF_RE);
    if (!def) continue;
    const id = def[1];
    const body: string[] = [def[2].trim()];
    dropped[i] = true;
    // Consume immediately-following indented continuation lines (GFM lazy
    // continuation is not supported by design — keep it simple and predictable).
    let j = i + 1;
    while (j < lines.length && isIndentedContinuation(lines[j])) {
      body.push(lines[j].trim());
      dropped[j] = true;
      j++;
    }
    i = j - 1;
    // Last definition wins for a duplicated id (matches CommonMark link-ref
    // semantics closely enough for a foreign-input adapter).
    defs.set(id, body.filter((s) => s.length > 0).join(' '));
  }
  if (defs.size === 0) {
    return markdown;
  }
  // ONE fixed, generic scanner regex — NOT one built from the definition ids.
  // It matches ANY `[^id]` shape, and the replacer decides per match via a map
  // lookup whether that id is a real definition (replace) or not (leave as-is).
  // This is genuinely O(total text) with no per-document regex compilation.
  //
  // Do NOT rebuild this as an alternation over `[...defs.keys()]`: a giant
  // `(id1|id2|...)` alternation over thousands of ids can blow the V8 regex
  // compiler's stack — a fatal, UNCATCHABLE "RegExpCompiler Allocation failed"
  // on prefix-chain ids (`a`, `aa`, `aaa`, ...) that kills the whole process
  // (worse than the earlier per-def thread-hang). A fixed scanner has no
  // id-dependent compilation cost and cannot blow up.
  const refRe = /\[\^([^\]]+)\]/g;
  const rewriteSegment = (segment: string): string =>
    segment.replace(refRe, (whole, id: string) => {
      const body = defs.get(id);
      // Only real definitions are inlined; an unknown id is left literal (same as
      // the old per-def loop, which simply never matched it).
      return body === undefined ? whole : `^[${escapeFootnoteBody(body)}]`;
    });
  // Pass 2: rewrite in-text references, skipping fenced code and dropped lines.
  const out: string[] = [];
  inFence = false;
  fence = '';
  for (let i = 0; i < lines.length; i++) {
    if (dropped[i]) continue;
    let line = lines[i];
    const marker = fenceMarker(line);
    if (inFence) {
      out.push(line);
      if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
        inFence = false;
        fence = '';
      }
      continue;
    }
    if (marker) {
      inFence = true;
      fence = marker;
      out.push(line);
      continue;
    }
    line = rewriteRefsOutsideInlineCode(line, rewriteSegment);
    out.push(line);
  }
  return out.join('\n');
 }
 /**
 * Strip a single leading YAML front-matter block (`---\n…\n---`). Foreign files
 * from Obsidian / Hugo / Jekyll / Notion — and Docmost's OWN git-sync page files
 * — open with front-matter that the canonical parser does not consume, so
 * without this it leaks into the body (and `title: Foo` above the closing `---`
 * renders as a setext `<h2>` that `extractTitleAndRemoveHeading` can hijack as
 * the page title). It is a no-op for front-matter-free input.
 *
 * LINE-ANCHORED (the same shape the canonical parser uses in
 * prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
 * very start and closes only on a `\n---` line. The retired `markdownToHtml`
 * strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
 * containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
 * and leaked the rest into the body. An optional leading BOM is tolerated.
 */
 const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
 /**
 * Normalize a foreign markdown string into Docmost's canonical markdown surface
 * so the strict canonical parser accepts it losslessly: normalize line endings,
 * strip a leading YAML front-matter block, then rewrite GFM reference footnotes
 * into inline footnotes. Add further fixture-driven foreign-surface cases here as
 * they are found.
 */
 export function normalizeForeignMarkdown(markdown: string): string {
  if (!markdown) return markdown;
  // Normalize CRLF -> LF FIRST. The line-anchored front-matter regex requires a
  // bare `\n` after the opening `---`, and convertReferenceFootnotes splits on
  // `\n`; a Windows/CRLF foreign file (`---\r\n…`) would otherwise slip past the
  // front-matter strip and leak into the body. The canonical parser
  // (page-file.ts parsePageFile) normalizes the same way before its FRONTMATTER_RE.
  const src = markdown.replace(/\r\n/g, '\n');
  const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
  return convertReferenceFootnotes(withoutFrontMatter);
 }
@@ -0,0 +1,23 @@
 // Jest stub for @tiptap/react.
 //
 // The server export/import code paths transitively import editor-ext, whose node
 // extensions import from `@tiptap/react`. The real module re-exports all of
 // `@tiptap/core` (headless, safe under node) AND adds React view helpers
 // (`ReactNodeViewRenderer`, …) that eagerly pull in react-dom — which throws
 // `navigator is not defined` under jest's node environment.
 //
 // So this stub DELEGATES to the real `@tiptap/core` (keeping `mergeAttributes`,
 // `Node`, `Mark`, `nodeInputRule`, … working — they are used by
 // `jsonToHtml`/`htmlToJson` on the server) and overrides ONLY the React view
 // helpers with no-ops. Those helpers are referenced solely inside `addNodeView()`
 // — code that runs only in a live browser editor, never on the server; if any
 // were actually invoked here it would (correctly) surface as a test failure.
 const core = require('@tiptap/core');
 module.exports = {
  ...core,
  ReactNodeViewRenderer: () => () => ({}),
  NodeViewWrapper: () => null,
  NodeViewContent: () => null,
  ReactRenderer: class {},
 };
@@ -543,6 +543,9 @@ importers:
      '@docmost/pdf-inspector':
        specifier: 1.9.6
        version: 1.9.6
      '@docmost/prosemirror-markdown':
        specifier: workspace:*
        version: link:../../packages/prosemirror-markdown
      '@fastify/cookie':
        specifier: ^11.0.2
        version: 11.0.2
Author	SHA1	Message	Date
agent_coder	c5bff2d84a	fix(#345 ): normalize CRLF before front-matter strip (review round 3) F9 [WARNING] The line-anchored front-matter regex from round 2 requires a bare LF after the opening `---`, so a Windows/CRLF foreign file (`---\r\n...`) slips past the strip and leaks its front-matter into the body (where `title: Foo` renders as a setext heading that title extraction hijacks). The canonical parser whose regex shape this copied (page-file.ts) normalizes CRLF -> LF BEFORE its FRONTMATTER_RE; the import path copied the regex but missed the normalization. normalizeForeignMarkdown now replaces CRLF with LF first (which also makes convertReferenceFootnotes' split('\n') consistent). Adds a CRLF fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 05:38:07 +03:00
agent_coder	80fc30633b	fix(#345 ): replace id-alternation regex with a fixed generic scanner + line-anchor frontmatter (review round 2) F7 [CRITICAL] The round-1 F2(a) fix built ONE alternation regex over all definition ids (`(id1\|id2\|...)`). On prefix-chain ids (a, aa, aaa, ...) V8's regex compiler blows its stack with a fatal, UNCATCHABLE 'RegExpCompiler Allocation failed' that kills the whole process — strictly worse than the original per-def thread-hang, and its match cost was still O(text x defs). Replaced with a single FIXED generic scanner `/\[\^([^\]]+)\]/g` plus a map lookup in the replacer: genuinely O(total text), no per-document regex compilation, cannot blow up. Output is identical (only real def ids are inlined). F8 [WARNING] The frontmatter strip regex was not line-anchored: it closed on the FIRST `---` anywhere, so a value containing a triple-dash (e.g. 'title: Q1 --- Q2') truncated the frontmatter and leaked the rest into the body. Replaced with the line-anchored shape the canonical parser already uses (page-file.ts): open on `---\n`, close on a `\n---` line. Adds tests: 4000 prefix-chain ids do not crash and stay fast; a frontmatter value containing '---' is stripped whole. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 05:18:44 +03:00
agent_coder	e17d5bc060	fix(#345 ): restore prom-client, harden normalizer against ReDoS, strip frontmatter (review round 1) Addresses the round-1 review of #369: F1 [CRITICAL] Restore prom-client. The prior commit removed it as a 'stray dep', but metrics.registry.ts imports it unconditionally at startup (main.ts boot), so a clean frozen install had no prom-client -> server tsc TS2307 + boot crash. It was surviving only via hoisting from a warm store. Restored to apps/server dependencies + regenerated the lock (prom-client/tdigest/bintrees return), keeping the @docmost/prosemirror-markdown dep. Verified: clean frozen install -> require.resolve('prom-client') ok, server tsc EXIT 0. F2 [HIGH] Two quadratic ReDoS vectors in foreign-markdown.ts on untrusted import (runs synchronously on the request thread, 30MB cap): (a) pass-2 was O(lines x defs) — a per-def RegExp rebuilt and run over every line. Replaced with ONE precompiled alternation regex over all def ids, built once per document, with an id->body lookup in the replacer: O(text). (b) the inline-code split alternation backtracks quadratically on a long UNCLOSED backtick run. Lines over 8KB now skip the split (left untouched) — a real footnote line is never that long. F3 [WARNING] Restore the leading YAML front-matter strip that the retired markdownToHtml layer did. Without it, Obsidian/Hugo/Jekyll/git-sync files leak their front-matter into the body (and 'title:' renders as a setext heading that title extraction can hijack). F4 [WARNING] Extend the zip-import spec with an image (width+align) + callout fidelity assertion through the PM->HTML->PM hop (the one hop the package suite does not cover). F5/F6 Update AGENTS.md (apps/server is now a prosemirror-markdown consumer) and make the server pretest build prosemirror-markdown too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 04:54:07 +03:00
agent_coder	2c2d60a5dc	fix(#345 ): protect inline-code refs and escape footnote-body brackets The foreign-markdown import normalizer rewrote GFM reference footnotes (`[^id]` + `[^id]: def`) into canonical inline `^[def]` footnotes, but two edge cases corrupted content: 1. A `[^id]` inside an inline-code span (backticks) was rewritten like prose text — only fenced code blocks were protected. Now the rewrite pass splits each line on inline-code spans and only touches the text outside them. 2. An unbalanced `]` in a definition body truncated the resulting `^[...]` footnote at the canonical tokenizer, leaking the tail as literal text. The body's square brackets are now backslash-escaped before wrapping. Adds golden cases for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:39:01 +03:00
agent_coder	1417209915	fix(#345 ): drop stray prom-client dep + add prosemirror-markdown to the lock The step-1 package.json declared the new @docmost/prosemirror-markdown workspace dep but the lock was not regenerated (CI frozen install would fail), and it also added a stray prom-client dep (a coder env-workaround for a pre-existing hoisted import, unrelated to #345 — removed). Regenerated the lock with only the prosemirror-markdown dep; faithful frozen install now passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:27:01 +03:00
agent_coder	f555fc87da	refactor(#345 step 2): server markdown IMPORT via canonical parser + normalizer Move every SERVER Markdown->ProseMirror path off the editor-ext markdown layer (`markdownToHtml`, a second marked-based parser) onto the canonical `@docmost/prosemirror-markdown` package, and add a foreign-markdown normalizer at the import boundary. Code: - `ImportService.processMarkdown` (single `.md` upload) now parses `markdownToProseMirror(normalizeForeignMarkdown(md))` directly — no HTML hop. - `PageService.parseProsemirrorContent` markdown case (page create/update with `format: 'markdown'`) same. - `FileImportTaskService` (zip import) parses markdown with the package, then serializes to HTML (`jsonToHtml`) so the SHARED HTML attachment / internal-link pipeline (processAttachments + formatImportHtml + processHTML) keeps handling `.md` and `.html` imports uniformly. The markdown PARSE — the drift source — no longer goes through editor-ext; the PM->HTML->PM hop that follows is lossless plumbing for attachment resolution, not a second parse. - `canonicalizeFootnotes` stays as an idempotent #228 safety net for the HTML path (a no-op on the already-canonical markdown output). Normalizer (`integrations/import/utils/foreign-markdown.ts`): a TEXT pre-pass, NOT a parser fork. The strict canonical parser does not accept GFM `[^id]` reference footnotes (and would misread `[^id]: def` as a CommonMark link-ref definition, silently corrupting the ref into a bogus link), so the normalizer rewrites reference footnotes into canonical inline `^[def]` before parsing. Callout surfaces (`:::type` and `> [!type]`) are intentionally NOT touched — the canonical parser already accepts BOTH natively, so normalizing them would be redundant and risk degrading its nesting/code-fence-aware handling. Fixtures-first: foreign-markdown.spec pins the normalizer and the end-to-end acceptance (no literal `[^id]`/`:::` leaks; re-export is canonical). The two footnote-canonicalize specs are updated to the canonical output — the parser assigns fresh `fn-*` ids, so they now assert by definition BODY order (still reference-ordered, deduped, orphan-free). FINAL CHECK: `grep -rn "htmlToMarkdown\\|markdownToHtml" apps/server/src` (non -test) is now empty — both editor-ext markdown-layer functions are gone from the server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:21:07 +03:00
agent_coder	d6d1195abd	refactor(#345 step 1): server markdown EXPORT via canonical converter Move every SERVER ProseMirror->Markdown path off the editor-ext markdown layer (`htmlToMarkdown`, a second turndown-based converter) onto the canonical `@docmost/prosemirror-markdown` package. - `ExportService.exportPage` (page/space markdown export) and `collaboration.util.jsonToMarkdown` (used by page.controller's markdown responses and the AI public-share chat tool) now serialize DIRECTLY from ProseMirror JSON via `convertProseMirrorToMarkdown` — no HTML intermediate, no `<colgroup>` scrub (the converter emits GFM tables directly). This is the SAME serializer the git-sync vault writer feeds, so an exported page BODY is byte-identical to its vault representation: no more export-md vs vault-md drift. The HTML export path is unchanged (still `jsonToHtml`). Emitted markdown moves to the canonical forms: callouts `> [!type]` (not `:::type`), inline footnotes `^[…]` (not `[^id]`), lossless images `![alt](src) <!--img {…}-->` (editor-ext dropped width/height/align). Fixtures-first: export-markdown.spec asserts those canonical forms and the export==vault-by-construction equality (both call the package converter). The one deliberate export/vault delta — export prepends the page title as an H1 while the vault carries it in frontmatter — is pinned by a test. Test infra: declare the `@docmost/prosemirror-markdown` workspace dep; teach jest to load its ESM build (babel-jest) and stub `@tiptap/react` (server code imports editor-ext, whose node views reference React renderers only used in a live browser editor — never on the server). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:20:25 +03:00