Compare commits

..

2 Commits

Author SHA1 Message Date
agent_coder 8b36294b7b test(editor): pin remote-delete cursor contract + correct scope docblock (#196 review round 1)
- F2: add a remap test for a REMOTE (meta-less) delete of a cursor's own range,
  pinning the collapse-not-drop contract — the deleted-over cursor collapses to a
  zero-width caret at the deletion point and stays in the set; others keep their
  occurrence (this is the riskiest remap path, previously covered only for insert).
- F1: reword the out-of-scope docblock — occurrences inside tables/code-blocks/
  callouts ARE matched and edited as plain text (no schema-aware cursor); they
  were never a 'deliberately not built' boundary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 04:41:06 +03:00
agent_coder 0fa2f9fb91 feat(editor): multi-cursor editing MVP — select all occurrences + type into all (#196)
Variant A of #196: VS Code-style multi-cursor limited to "select all
occurrences of a word (or the selection) and type into all at once", built on
top of the existing SearchAndReplace mass-transaction machinery.

- New `MultiCursor` Tiptap extension (packages/editor-ext/src/lib/multi-cursor/):
  Cmd/Ctrl+Shift+L selects all occurrences, Cmd/Ctrl+D adds the next, typing /
  Backspace / Delete apply to every cursor in ONE reverse-order transaction (so a
  single undo reverts the whole multi-edit), Esc / click / navigation collapse.
- Cursors live in plugin state and are remapped on every docChanged — covering
  remote Yjs edits (applied as ordinary transactions) with no collab-specific code.
- Extracted a shared `findOccurrences` util so SearchAndReplace and MultiCursor
  no longer duplicate the occurrence walk (behaviour-preserving).
- Conscious v1 out-of-scope boundaries (Variant B) documented in the extension.

Registered in mainExtensions; carets styled distinctly from collaborative carets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 04:24:34 +03:00
24 changed files with 1287 additions and 950 deletions
+2 -2
View File
@@ -201,7 +201,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp` and `git-sync`; there is exactly ONE copy of the converter now |
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
@@ -284,7 +284,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
### Client structure
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, import/export) — editor schema changes often need to be made in `editor-ext`, not just the client. The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by both `mcp` and `git-sync` — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
@@ -45,6 +45,7 @@ import {
TiptapPdf,
PageBreak,
SearchAndReplace,
MultiCursor,
Mention,
TableDndExtension,
TableHandleCommandsExtension,
@@ -447,6 +448,10 @@ export const mainExtensions = [
};
},
}).configure(),
// Multi-cursor editing (MVP / Variant A): select-all-occurrences + type into
// all at once. Does not depend on collaboration, so it lives in mainExtensions
// (available in both the plain and collaborative editors).
MultiCursor,
Columns,
Column,
AutoJoiner.configure({
@@ -1,5 +1,6 @@
@import "./core.css";
@import "./collaboration.css";
@import "./multi-cursor.css";
@import "./task-list.css";
@import "./placeholder.css";
@import "./drag-handle.css";
@@ -0,0 +1,60 @@
/*
* Multi-cursor (issue #196). Deliberately DISTINCT from the collaboration
* carets (collaboration.css) so a user never confuses their own multi-cursors
* with a co-author's caret: solid accent-blue carets + a translucent blue
* range highlight, versus the thin dark collaboration caret with a name label.
*/
/* A secondary caret rendered as a Decoration.widget at each cursor position. */
.multi-cursor__caret {
position: relative;
display: inline-block;
width: 0;
height: 1em;
vertical-align: text-bottom;
pointer-events: none;
}
.multi-cursor__caret::after {
content: "";
position: absolute;
left: -1px;
top: 0;
bottom: 0;
width: 2px;
background: #2b6cb0;
animation: multi-cursor-blink 1s steps(1) infinite;
}
/* Optional label class reserved for future per-cursor annotations. */
.multi-cursor__label {
position: absolute;
top: -1.4em;
left: -1px;
font-size: 0.7rem;
line-height: normal;
padding: 0.05rem 0.25rem;
border-radius: 3px 3px 3px 0;
background: #2b6cb0;
color: #fff;
white-space: nowrap;
user-select: none;
pointer-events: none;
}
/* Inline highlight for a multi-cursor RANGE (from < to). */
.multi-cursor__selection {
background: rgba(43, 108, 176, 0.28);
border-radius: 2px;
}
@keyframes multi-cursor-blink {
0%,
50% {
opacity: 1;
}
51%,
100% {
opacity: 0;
}
}
+4 -6
View File
@@ -23,7 +23,7 @@
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build",
"pretest": "pnpm --filter @docmost/editor-ext build",
"test": "jest",
"test:int": "jest --config test/jest-integration.json",
"test:watch": "jest --watch",
@@ -43,7 +43,6 @@
"@clickhouse/client": "^1.18.2",
"@docmost/mcp": "workspace:*",
"@docmost/pdf-inspector": "1.9.6",
"@docmost/prosemirror-markdown": "workspace:*",
"@fastify/cookie": "^11.0.2",
"@fastify/multipart": "^10.0.0",
"@fastify/static": "^9.1.3",
@@ -176,7 +175,7 @@
"/node_modules/"
],
"transform": {
"(happy-dom.+|prosemirror-markdown/build/.+)\\.js$": [
"happy-dom.+\\.js$": [
"babel-jest",
{
"presets": [
@@ -194,7 +193,7 @@
"^.+\\.(t|j)sx?$": "ts-jest"
},
"transformIgnorePatterns": [
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@docmost/prosemirror-markdown)(@|/))"
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0)(@|/))"
],
"collectCoverageFrom": [
"**/*.(t|j)s"
@@ -205,8 +204,7 @@
"^@docmost/db/(.*)$": "<rootDir>/database/$1",
"^@docmost/transactional/(.*)$": "<rootDir>/integrations/transactional/$1",
"^@docmost/ee/(.*)$": "<rootDir>/ee/$1",
"^src/(.*)$": "<rootDir>/$1",
"^@tiptap/react$": "<rootDir>/../test/stubs/tiptap-react.js"
"^src/(.*)$": "<rootDir>/$1"
}
}
}
@@ -43,6 +43,7 @@ import {
Column,
Status,
addUniqueIdsToDoc,
htmlToMarkdown,
TransclusionSource,
TransclusionReference,
FootnoteReference,
@@ -50,7 +51,6 @@ import {
FootnoteDefinition,
PageEmbed,
} from '@docmost/editor-ext';
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
import { generateText, getSchema, JSONContent } from '@tiptap/core';
import { generateHTML, generateJSON } from '../common/helpers/prosemirror/html';
// @tiptap/html library works best for generating prosemirror json state but not HTML
@@ -239,10 +239,6 @@ export function prosemirrorNodeToYElement(node: any): Y.XmlElement | Y.XmlText {
}
export function jsonToMarkdown(tiptapJson: any): string {
// Direct ProseMirror JSON -> Markdown via the canonical converter
// (`@docmost/prosemirror-markdown`) — no HTML intermediate, no second
// editor-ext markdown layer. Same serializer as the page/space export and the
// git-sync vault writer, so every server PM->MD path emits identical canonical
// markdown (issue #345).
return convertProseMirrorToMarkdown(tiptapJson);
const html = jsonToHtml(tiptapJson);
return htmlToMarkdown(html);
}
@@ -52,9 +52,7 @@ import {
INTERNAL_LINK_REGEX,
extractPageSlugId,
} from '../../../integrations/export/utils';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../../../integrations/import/utils/foreign-markdown';
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
import { WatcherService } from '../../watcher/watcher.service';
import { sql } from 'kysely';
import { TransclusionService } from '../transclusion/transclusion.service';
@@ -1303,14 +1301,8 @@ export class PageService {
switch (format) {
case 'markdown': {
// Canonical markdown -> ProseMirror JSON directly via
// `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate,
// no editor-ext markdown layer. Foreign markdown surfaces the strict
// parser rejects (GFM `[^id]` reference footnotes) are normalized to the
// canonical inline form first.
prosemirrorJson = await markdownToProseMirror(
normalizeForeignMarkdown(content as string),
);
const html = await markdownToHtml(content as string);
prosemirrorJson = htmlToJson(html as string);
break;
}
case 'html': {
@@ -1,145 +0,0 @@
// export.service.ts imports the ESM-only @sindresorhus/slugify (not in jest's
// transform allowlist). It is irrelevant to the markdown-serialization path under
// test (only used for page-mention link slugs on the DB path), so it is mocked
// out to keep the module graph loadable under ts-jest (mirrors the import specs).
jest.mock('@sindresorhus/slugify', () => ({
__esModule: true,
default: (input: string) => String(input),
}));
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
import { ExportService } from './export.service';
import { ExportFormat } from './dto/export-dto';
/**
* STEP 1 golden test for issue #345: server MARKDOWN export runs DIRECTLY through
* the canonical converter (`convertProseMirrorToMarkdown`) — no HTML intermediate
* and no `@docmost/editor-ext` markdown layer — so the emitted markdown is in the
* canonical package forms and is byte-identical to the git-sync vault body.
*
* These are the goldens the swap has to satisfy: they assert the CANONICAL
* surface (callout `> [!type]`, inline footnote `^[…]`, lossless image
* `<!--img …-->`) rather than the old editor-ext forms (`:::type`, `[^id]`,
* lossy `![alt](src)`).
*
* `exportPage(..., singlePage=false)` takes no DB path (no mention rewriting), so
* the service is constructed with null collaborators and only the pure
* PM -> Markdown path is exercised.
*/
function makeService(): ExportService {
return new ExportService(
null as any, // pageRepo
null as any, // pagePermissionRepo
null as any, // db
null as any, // storageService
null as any, // environmentService
null as any, // domainService
);
}
// A representative page exercising the node types whose canonical markdown form
// changed with the move off the editor-ext layer: callout, inline footnote, and a
// lossless image carrying width/align attrs that the old layer dropped.
const REPRESENTATIVE_DOC = {
type: 'doc',
content: [
{
type: 'paragraph',
content: [
{ type: 'text', text: 'Body ' },
{ type: 'footnoteReference', attrs: { id: 'fn-1' } },
{ type: 'text', text: ' end.' },
],
},
{
type: 'callout',
attrs: { type: 'info', icon: null },
content: [
{
type: 'paragraph',
content: [{ type: 'text', text: 'Heads up' }],
},
],
},
{
type: 'image',
attrs: {
src: '/files/pic.png',
alt: 'Pic',
width: 320,
align: 'left',
},
},
{
type: 'footnotesList',
content: [
{
type: 'footnoteDefinition',
attrs: { id: 'fn-1' },
content: [
{
type: 'paragraph',
content: [{ type: 'text', text: 'the note' }],
},
],
},
],
},
],
};
describe('ExportService — markdown export via the canonical converter (#345)', () => {
it('emits canonical callout, inline footnote and lossless image forms', async () => {
const service = makeService();
const md = (await service.exportPage(ExportFormat.Markdown, {
title: '',
content: REPRESENTATIVE_DOC,
} as any)) as string;
// Callout: Obsidian `> [!type]`, NOT the legacy `:::type`.
expect(md).toContain('> [!info]');
expect(md).not.toContain(':::');
// Inline footnote: `^[…]`, NOT the reference `[^id]` form.
expect(md).toContain('^[the note]');
expect(md).not.toMatch(/\[\^/);
// Lossless image: trailing `<!--img …-->` carrying the dropped attrs.
expect(md).toContain('![Pic](/files/pic.png)');
expect(md).toContain('<!--img');
expect(md).toContain('"width":"320"');
expect(md).toContain('"align":"left"');
});
it('export body is byte-identical to the git-sync vault serializer (export == vault)', async () => {
const service = makeService();
// A title-less page: exportPage prepends NO heading, so the whole output is
// the page BODY — exactly what git-sync serializes (git-sync stores the title
// in frontmatter / the filename, never as an in-body H1).
const exported = (await service.exportPage(ExportFormat.Markdown, {
title: '',
content: REPRESENTATIVE_DOC,
} as any)) as string;
// The git-sync vault writer feeds this SAME converter (git-sync
// `stabilizePageBody` = convertProseMirrorToMarkdown(content) at the
// fixpoint). For an already-stable doc the single pass IS the fixpoint, so
// the two are byte-identical by construction — assert it.
const vaultBody = convertProseMirrorToMarkdown(REPRESENTATIVE_DOC);
expect(exported).toBe(vaultBody);
});
it('prepends the page title as an H1 heading (the one documented export/vault delta)', async () => {
const service = makeService();
const md = (await service.exportPage(ExportFormat.Markdown, {
title: 'My Page',
content: { type: 'doc', content: [] },
} as any)) as string;
// Export makes standalone files, so it prepends the title as an H1. This is
// the ONE deliberate difference from the vault body (which carries the title
// in frontmatter). The body below the heading still serializes canonically.
expect(md.startsWith('# My Page')).toBe(true);
});
});
@@ -37,7 +37,7 @@ import {
getAttachmentIds,
getProsemirrorContent,
} from '../../common/helpers/prosemirror/utils';
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
import { htmlToMarkdown } from '@docmost/editor-ext';
type AllowedAttachment = { id: string; fileName: string; filePath: string };
@@ -79,8 +79,9 @@ export class ExportService {
prosemirrorJson.content.unshift(titleNode);
}
const pageHtml = jsonToHtml(prosemirrorJson);
if (format === ExportFormat.HTML) {
const pageHtml = jsonToHtml(prosemirrorJson);
return `<!DOCTYPE html>
<html>
<head>
@@ -91,14 +92,11 @@ export class ExportService {
}
if (format === ExportFormat.Markdown) {
// Direct ProseMirror JSON -> Markdown via the canonical converter
// (`@docmost/prosemirror-markdown`). This is the SAME serializer the
// git-sync vault writer feeds (see git-sync `stabilizePageBody`), so an
// exported page body is byte-identical to its vault representation — no
// HTML intermediate, no second markdown layer, no format drift (issue
// #345). The old `<colgroup>` scrub is gone with the HTML step: the
// converter emits GFM tables directly and never produces `<colgroup>`.
return convertProseMirrorToMarkdown(prosemirrorJson);
const newPageHtml = pageHtml.replace(
/<colgroup[^>]*>[\s\S]*?<\/colgroup>/gim,
'',
);
return htmlToMarkdown(newPageHtml);
}
return;
@@ -17,22 +17,6 @@ jest.mock('image-dimensions', () => ({
__esModule: true,
imageDimensionsFromData: () => undefined,
}));
// FileImportTaskService -> PageService -> collaboration.gateway ->
// metrics.registry imports `prom-client`, which is not resolvable in this
// workspace's node_modules (types-only stub, no runtime entry). Metrics are
// disabled on this path, so a virtual no-op mock keeps the module graph loadable.
jest.mock(
'prom-client',
() => ({
collectDefaultMetrics: () => undefined,
Registry: class {},
Histogram: class {},
Gauge: class {},
Counter: class {},
Summary: class {},
}),
{ virtual: true },
);
import { promises as fs } from 'fs';
import * as os from 'os';
@@ -42,17 +26,14 @@ import { ImportService } from './import.service';
/**
* Binding test for issue #228 / review #5: FileImportTaskService.processGenericImport
* is a NON-editor write path, so a zip-imported `.md` page ends up with canonical
* footnotes before persisting: ordered by first reference, reused refs deduped,
* orphan definitions dropped.
* is a NON-editor write path (markdownToHtml -> processHTML -> JSON, never runs
* footnoteSyncPlugin), so it canonicalizes footnotes before persisting. This pins
* that binding — the same one import.service has a spec for — which previously had
* NO spec at all.
*
* Since #345 the `.md` parse runs `normalizeForeignMarkdown` ->
* `markdownToProseMirror` -> `jsonToHtml` (feeding the shared HTML attachment /
* link pipeline) -> `processHTML` -> `canonicalizeFootnotes`. The parser assigns
* fresh `fn-*` ids, so we assert by definition BODY order rather than the source
* labels. The conversion is REAL (a real ImportService, its createYdoc stubbed);
* the filesystem is a real temp dir with one .md file; the DB transaction is
* stubbed to capture the persisted page content.
* The markdown -> HTML -> ProseMirror conversion is REAL (a real ImportService,
* its createYdoc stubbed); the filesystem is a real temp dir with one .md file;
* the DB transaction is stubbed to capture the persisted page content.
*/
// Out-of-order references (c, a, b), a REUSED reference ([^a] twice), and an
@@ -68,14 +49,13 @@ const MARKDOWN = [
'[^z]: orphan note',
].join('\n');
/** Definition body texts of the (single) footnotesList, in list order. */
function footnoteListBodies(content: any): string[] {
function footnoteListIds(content: any): string[] {
const list = (content?.content ?? []).find(
(n: any) => n.type === 'footnotesList',
);
return (list?.content ?? [])
.filter((n: any) => n.type === 'footnoteDefinition')
.map((n: any) => n.content?.[0]?.content?.[0]?.text);
.map((n: any) => n.attrs?.id);
}
// A permissive chainable stub for the spaces lookup (selectFrom(...).select(...)
@@ -91,127 +71,80 @@ function chainable(result: any): any {
return proxy;
}
/**
* Run one markdown file through the REAL zip-import pipeline
* (`processGenericImport` -> `markdownToProseMirror` -> `jsonToHtml` ->
* `processHTML`/`htmlToJson`) and return the persisted page `content`. This is
* the server-specific PM->HTML->PM hop that the package's own PM<->MD tests do
* NOT cover.
*/
async function runZipImport(markdown: string): Promise<any> {
const extractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fit-canon-'));
await fs.writeFile(path.join(extractDir, 'note.md'), markdown, 'utf-8');
const importService = new ImportService(
{} as any,
{} as any,
{} as any,
{} as any,
);
jest
.spyOn(importService as any, 'createYdoc')
.mockResolvedValue(Buffer.from([]) as any);
let captured: any = null;
const trx = {
insertInto: (table: string) => ({
values: (v: any) => {
if (table === 'pages') captured = v;
return { execute: async () => {} };
},
}),
};
const db: any = {
selectFrom: () => chainable({ slug: 'space-slug' }),
transaction: () => ({ execute: (fn: any) => fn(trx) }),
};
const importAttachmentService = {
processAttachments: async ({ html }: any) => html,
};
const service = new FileImportTaskService(
{} as any, // storageService
importService as any,
{ nextPagePosition: async () => 'a0' } as any,
{ insertBacklink: jest.fn() } as any,
db,
importAttachmentService as any,
{ emit: jest.fn() } as any,
{ logBatchWithContext: jest.fn() } as any,
);
const fileTask: any = {
id: 'task-1',
source: 'generic',
spaceId: 'space-1',
workspaceId: 'ws-1',
creatorId: 'user-1',
};
try {
await service.processGenericImport({ extractDir, fileTask });
expect(captured).toBeTruthy();
return captured.content;
} finally {
await fs.rm(extractDir, { recursive: true, force: true });
}
}
/** Find the first node of a given type anywhere in a PM content tree. */
function findFirst(node: any, type: string): any {
if (!node || typeof node !== 'object') return null;
if (node.type === type) return node;
for (const child of node.content ?? []) {
const hit = findFirst(child, type);
if (hit) return hit;
}
return null;
}
describe('FileImportTaskService.processGenericImport — footnote canonicalization (#228)', () => {
it('orders footnotes by first reference, dedupes reuse, and drops orphans on zip import', async () => {
const content = await runZipImport(MARKDOWN);
// Definitions ordered by FIRST REFERENCE (C, A, B), NOT the markdown
// definition order (A, B, C). Ids are the parser's fresh `fn-*`, so pin
// the BODIES.
expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
// Orphan [^z] dropped; reused [^a] collapses to one definition; one list.
expect(footnoteListBodies(content)).not.toContain('orphan note');
const lists = (content.content ?? []).filter(
(n: any) => n.type === 'footnotesList',
const extractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fit-canon-'));
await fs.writeFile(path.join(extractDir, 'note.md'), MARKDOWN, 'utf-8');
// Real ImportService for the html -> JSON conversion; stub the yjs encode.
const importService = new ImportService(
{} as any,
{} as any,
{} as any,
{} as any,
);
expect(lists).toHaveLength(1);
expect(
footnoteListBodies(content).filter((b) => b === 'note A'),
).toHaveLength(1);
});
jest
.spyOn(importService as any, 'createYdoc')
.mockResolvedValue(Buffer.from([]) as any);
// #345 F4: the zip path routes markdown through jsonToHtml -> processHTML ->
// htmlToJson (the shared HTML attachment pipeline). #345's headline is LOSSLESS
// image width/align via the `<!--img {...}-->` comment; a callout carries its
// `type`. This asserts those survive the PM->HTML->PM hop — the one hop the
// package's PM<->MD suite does not exercise.
it('preserves image width/align and callout type through the PM->HTML->PM hop', async () => {
const md = [
'# Doc',
'',
'![a picture](https://example.com/i.png) <!--img {"width":"320","align":"left"}-->',
'',
':::warning',
'Careful now.',
':::',
].join('\n');
let captured: any = null;
const trx = {
insertInto: (table: string) => ({
values: (v: any) => {
if (table === 'pages') captured = v;
return { execute: async () => {} };
},
}),
};
const db: any = {
selectFrom: () => chainable({ slug: 'space-slug' }),
transaction: () => ({ execute: (fn: any) => fn(trx) }),
};
const content = await runZipImport(md);
const importAttachmentService = {
processAttachments: async ({ html }: any) => html,
};
const backlinkRepo = { insertBacklink: jest.fn() };
const eventEmitter = { emit: jest.fn() };
const auditService = { logBatchWithContext: jest.fn() };
const image = findFirst(content, 'image');
expect(image).toBeTruthy();
// The lossless sizing/alignment must survive the HTML hop.
expect(String(image.attrs?.width)).toBe('320');
expect(image.attrs?.align).toBe('left');
const pageService = { nextPagePosition: async () => 'a0' };
const callout = findFirst(content, 'callout');
expect(callout).toBeTruthy();
expect(callout.attrs?.type).toBe('warning');
const service = new FileImportTaskService(
{} as any, // storageService
importService as any,
pageService as any,
backlinkRepo as any,
db,
importAttachmentService as any,
eventEmitter as any,
auditService as any,
);
const fileTask: any = {
id: 'task-1',
source: 'generic',
spaceId: 'space-1',
workspaceId: 'ws-1',
creatorId: 'user-1',
};
try {
await service.processGenericImport({ extractDir, fileTask });
expect(captured).toBeTruthy();
const content = captured.content;
// Reference order is c, a, b (NOT the markdown definition order a, b, c).
expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']);
// Orphan [^z] dropped; reused [^a] collapses to one definition; one list.
expect(footnoteListIds(content)).not.toContain('z');
const lists = (content.content ?? []).filter(
(n: any) => n.type === 'footnotesList',
);
expect(lists).toHaveLength(1);
expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1);
} finally {
await fs.rm(extractDir, { recursive: true, force: true });
}
});
});
@@ -1,9 +1,6 @@
import { Inject, Injectable, Logger } from '@nestjs/common';
import * as path from 'path';
import {
jsonToHtml,
jsonToText,
} from '../../../collaboration/collaboration.util';
import { jsonToText } from '../../../collaboration/collaboration.util';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import {
@@ -21,11 +18,9 @@ import { generateSlugId } from '../../../common/helpers';
import { v7 } from 'uuid';
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
import { formatImportHtml } from '../utils/import-formatter';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import {
buildAttachmentCandidates,
collectMarkdownAndHtmlFiles,
@@ -466,18 +461,7 @@ export class FileImportTaskService {
content = await fs.readFile(absPath, 'utf-8');
if (page.fileExtension.toLowerCase() === '.md') {
// Parse markdown with the single canonical converter
// (`@docmost/prosemirror-markdown`), after normalizing foreign
// reference footnotes, then serialize to HTML so the shared HTML
// pipeline below (processAttachments + formatImportHtml +
// processHTML) keeps handling `.md` and `.html` imports
// uniformly. The markdown PARSE no longer goes through the
// editor-ext markdown layer (issue #345) — the drift source is
// gone. The PM -> HTML -> PM hop that follows is lossless
// plumbing for attachment/link resolution, NOT a second parse.
content = jsonToHtml(
await markdownToProseMirror(normalizeForeignMarkdown(content)),
);
content = await markdownToHtml(content);
}
} catch (err: any) {
if (err?.code === 'ENOENT') {
@@ -516,12 +500,10 @@ export class FileImportTaskService {
this.importService.extractTitleAndRemoveHeading(pmState);
// Canonicalize footnote topology on this non-editor write path
// (the HTML pipeline's processHTML never runs footnoteSyncPlugin), so
// a zip-imported page's footnotes are reference-ordered, deduped, and
// (markdownToHtml/processHTML never runs footnoteSyncPlugin), so a
// zip-imported page's footnotes are reference-ordered, deduped, and
// orphan-free like the editor's invariant (issue #228). Pure +
// idempotent + shape-safe; a footnote-free doc is unchanged. (For a
// `.md` file the package parser already yields canonical footnotes,
// so this is a no-op there.)
// idempotent + shape-safe; a footnote-free doc is unchanged.
// (Future consolidation, architecture B: like import.service, this
// path persists directly rather than via PageService — a shared
// "prepare JSON for persist" helper would centralize this call.)
@@ -12,19 +12,13 @@ import { canonicalizeFootnotes } from '@docmost/editor-ext';
/**
* Integration-ish test for the USER-FACING markdown import path
* (`ImportService.importPage`). It exercises the REAL markdown -> ProseMirror
* conversion and asserts the stored page's footnotes are canonical: ordered by
* FIRST REFERENCE (not markdown definition order), reused references deduped to a
* single definition, and orphan definitions dropped.
*
* Since #345 the markdown parse runs through the canonical package
* (`normalizeForeignMarkdown` -> `markdownToProseMirror`), which owns this
* canonicalization: the input's GFM `[^id]` reference footnotes are normalized to
* inline `^[…]`, and the parser assigns fresh sequential ids (`fn-*`) in
* reference order while merging identical bodies — so we assert by definition
* BODY order, not by the source labels. `canonicalizeFootnotes` remains wired as
* an idempotent safety net (issue #228) and is a no-op on this already-canonical
* output.
* (`ImportService.importPage`). It exercises the REAL markdown -> HTML -> JSON
* conversion and asserts that the stored page content has its footnotes
* canonicalized — the gap that issue #228 fixes: the import path builds
* ProseMirror JSON directly (never running the editor's footnoteSyncPlugin), so
* before this wiring the stored footnotes kept the markdown's physical
* definition order (out of order vs. references), retained orphan definitions,
* and did not collapse reused references.
*
* The DB/ydoc side-effects are stubbed: `getNewPagePosition` (DB query) and
* `createYdoc` (Yjs encode) are spied, and `pageRepo.insertPage` captures the
@@ -73,14 +67,24 @@ function makeService() {
}
/** List the footnote-definition ids of the (single) footnotesList, in order. */
/** Definition body texts of the (single) footnotesList, in list order. */
function footnoteListBodies(content: any): string[] {
function footnoteListIds(content: any): string[] {
const list = (content.content ?? []).find(
(n: any) => n.type === 'footnotesList',
);
return (list?.content ?? [])
if (!list) return [];
return (list.content ?? [])
.filter((n: any) => n.type === 'footnoteDefinition')
.map((n: any) => n.content?.[0]?.content?.[0]?.text);
.map((n: any) => n.attrs?.id);
}
function definitionText(content: any, id: string): string | undefined {
const list = (content.content ?? []).find(
(n: any) => n.type === 'footnotesList',
);
const def = (list?.content ?? []).find(
(n: any) => n.type === 'footnoteDefinition' && n.attrs?.id === id,
);
return def?.content?.[0]?.content?.[0]?.text;
}
describe('ImportService.importPage — footnote canonicalization (#228)', () => {
@@ -97,23 +101,23 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
const content = getCaptured().content;
expect(content).toBeTruthy();
// Definitions ordered by FIRST REFERENCE (C, A, B) — NOT the markdown
// definition order (A, B, C) — with the orphan [^z] dropped and the reused
// [^a] collapsed to a single definition. (Ids are the parser's fresh `fn-*`,
// so we pin the BODIES.)
expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
// Reference order is c, a, b (NOT the markdown definition order a, b, c).
expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']);
// Definitions preserved and attached to the right ids.
expect(definitionText(content, 'c')).toBe('note C');
expect(definitionText(content, 'a')).toBe('note A');
expect(definitionText(content, 'b')).toBe('note B');
// Orphan definition [^z] is dropped.
expect(footnoteListBodies(content)).not.toContain('orphan note');
expect(footnoteListIds(content)).not.toContain('z');
// Reused [^a] yields exactly ONE definition, and exactly one list.
const lists = (content.content ?? []).filter(
(n: any) => n.type === 'footnotesList',
);
expect(lists).toHaveLength(1);
expect(
footnoteListBodies(content).filter((b) => b === 'note A'),
).toHaveLength(1);
expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1);
});
it('is idempotent: canonicalizing the stored output again is a no-op', async () => {
@@ -130,6 +134,6 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
// time must not change it (safe to wire into every write path).
const second = canonicalizeFootnotes(stored);
expect(second).toEqual(stored);
expect(footnoteListBodies(second)).toEqual(['note C', 'note A', 'note B']);
expect(footnoteListIds(second)).toEqual(['c', 'a', 'b']);
});
});
@@ -17,9 +17,7 @@ import {
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { TiptapTransformer } from '@hocuspocus/transformer';
import * as Y from 'yjs';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
import {
FileTaskStatus,
FileTaskType,
@@ -87,13 +85,11 @@ export class ImportService {
const extracted = this.extractTitleAndRemoveHeading(prosemirrorState);
const title = extracted.title;
// The markdown path now canonicalizes footnotes itself (the package parser),
// but the HTML path (processHTML -> htmlToJson) does NOT run the editor's
// footnoteSyncPlugin, so an imported HTML doc can keep its source's PHYSICAL
// definition order (out of order vs. references), retain orphan definitions,
// and not be deduped. Canonicalize before persisting so the stored page
// matches the editor's invariant (issue #228); it is an idempotent no-op on
// the already-canonical markdown output.
// Imported markdown/HTML is built via markdownToHtml -> htmlToJson, which
// never runs the editor's footnoteSyncPlugin, so the footnote topology keeps
// the source's PHYSICAL definition order (out of order vs. references),
// retains orphan definitions, and is not deduped. Canonicalize before
// persisting so the stored page matches the editor's invariant (issue #228).
// Pure + idempotent + shape-safe: a doc with no footnotes is unchanged.
// (Future consolidation, architecture B: this import path persists directly
// via pageRepo.insertPage rather than through PageService.createPage, so the
@@ -137,15 +133,12 @@ export class ImportService {
}
async processMarkdown(markdownInput: string): Promise<any> {
// Canonical markdown -> ProseMirror JSON directly via
// `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate and no
// second editor-ext markdown layer. Foreign markdown surfaces the strict
// canonical parser does not accept (GFM `[^id]` reference footnotes) are
// rewritten to the canonical inline form by `normalizeForeignMarkdown` first.
// The HTML-cleanup pass (`normalizeImportHtml`) is intentionally skipped here:
// it targets foreign *HTML* (Notion/XWiki), which only ever arrives on the
// `.html` path (`processHTML`), never as canonical markdown.
return markdownToProseMirror(normalizeForeignMarkdown(markdownInput));
try {
const html = await markdownToHtml(markdownInput);
return this.processHTML(html);
} catch (err) {
throw err;
}
}
async processHTML(htmlInput: string): Promise<any> {
@@ -1,218 +0,0 @@
import {
convertProseMirrorToMarkdown,
markdownToProseMirror,
} from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from './foreign-markdown';
/**
* STEP 2 goldens for issue #345: the foreign-markdown normalizer that runs at the
* import boundary BEFORE the strict canonical parser (`markdownToProseMirror`).
*
* Two layers:
* 1. PURE stringstring cases pinning the normalizer's own behavior (GFM
* reference footnotes inline `^[…]`).
* 2. END-TO-END acceptance: for a foreign corpus, `normalizeForeignMarkdown`
* then `markdownToProseMirror` then `convertProseMirrorToMarkdown` must leave
* NO literal `[^id]` / `:::` garbage in the document and must re-export in the
* canonical forms.
*/
describe('normalizeForeignMarkdown — GFM reference footnotes', () => {
it('inlines a single-line reference footnote and drops its definition', () => {
const out = normalizeForeignMarkdown(
'A note[^1] here.\n\n[^1]: The definition.',
);
expect(out).toBe('A note^[The definition.] here.\n');
});
it('inlines every reference to a reused id (downstream dedups)', () => {
const out = normalizeForeignMarkdown(
'X[^a] and Y[^a].\n\n[^a]: shared.',
);
expect(out).toBe('X^[shared.] and Y^[shared.].\n');
});
it('joins indented continuation lines of a definition with a space', () => {
const out = normalizeForeignMarkdown(
'See[^n].\n\n[^n]: line one\n line two',
);
expect(out).toBe('See^[line one line two].\n');
});
it('never rewrites a reference inside a fenced code block', () => {
const out = normalizeForeignMarkdown(
'```\ncode[^1] here\n```\n\n[^1]: def.',
);
expect(out).toContain('code[^1] here');
// The (now orphaned) definition line is still removed.
expect(out).not.toContain('[^1]: def.');
});
it('never rewrites a reference inside an INLINE-code span (backticks)', () => {
// The `[^1]` inside backticks is literal code and must survive verbatim;
// the one outside is rewritten. (Bug #1: only fenced blocks were protected.)
const out = normalizeForeignMarkdown(
'Use `arr[^1]` in code but note[^1] in prose.\n\n[^1]: def.',
);
expect(out).toBe('Use `arr[^1]` in code but note^[def.] in prose.\n');
});
it('escapes brackets in a body so an unbalanced ] cannot truncate the footnote', () => {
// A foreign definition body with a stray `]` would, unescaped, close the
// canonical `^[...]` early and leak the tail as text (bug #2). The body's
// brackets are backslash-escaped so the footnote stays whole.
const out = normalizeForeignMarkdown(
'Ref[^1] here.\n\n[^1]: see item ] and [more] later',
);
expect(out).toBe('Ref^[see item \\] and \\[more\\] later] here.\n');
// The tokenizer must see exactly one unescaped closing bracket (our own).
expect(out.match(/(?<!\\)\]/g)).toHaveLength(1);
});
it('leaves a reference with no matching definition literal (no body to inline)', () => {
const out = normalizeForeignMarkdown('Dangling[^x] ref.');
expect(out).toBe('Dangling[^x] ref.');
});
it('returns the input unchanged when there are no reference footnotes', () => {
const md = '# Title\n\nJust text with `inline code` and a [link](/x).';
expect(normalizeForeignMarkdown(md)).toBe(md);
});
it('does NOT touch callout surfaces — the canonical parser handles them', () => {
const callouts = ':::info\nHi\n:::\n\n> [!warning]\n> Careful';
expect(normalizeForeignMarkdown(callouts)).toBe(callouts);
});
it('strips a leading YAML front-matter block (Obsidian/Hugo/git-sync files)', () => {
const out = normalizeForeignMarkdown(
'---\ntitle: My Page\ntags: [a, b]\n---\n\n# Heading\n\nBody.',
);
expect(out).toBe('# Heading\n\nBody.');
// The front-matter must not leak into the body as a setext heading.
expect(out).not.toContain('title: My Page');
expect(out).not.toContain('---');
});
it('does not strip a horizontal rule that is not leading front-matter', () => {
const md = 'Intro paragraph.\n\n---\n\nAfter the rule.';
expect(normalizeForeignMarkdown(md)).toBe(md);
});
it('is linear on a document with thousands of definitions (no quadratic blowup)', () => {
// F2(a): the pass-2 rewrite must be O(text), not O(text × defs). Build a
// pathological doc (many defs + many plain text lines) and assert it
// completes well under a second — a quadratic implementation took ~14s.
const N = 4000;
const refs = Array.from({ length: N }, (_, i) => `line ${i} plain text`).join('\n');
const defs = Array.from({ length: N }, (_, i) => `[^n${i}]: def ${i}`).join('\n');
const doc = `start[^n0] and[^n${N - 1}] end\n\n${refs}\n\n${defs}`;
const t0 = Date.now();
const out = normalizeForeignMarkdown(doc);
const elapsed = Date.now() - t0;
expect(elapsed).toBeLessThan(2000);
// Sanity: the two real references were still inlined.
expect(out).toContain('^[def 0]');
expect(out).toContain(`^[def ${N - 1}]`);
});
it('is bounded on a long unclosed backtick run (no inline-split ReDoS)', () => {
// F2(b): a huge unterminated backtick run must not cause quadratic
// backtracking in the inline-code split. Oversized lines skip the split
// entirely (left untouched), so this returns promptly.
const line = 'x' + '`'.repeat(200000);
const doc = `${line}\n\n[^1]: def`;
const t0 = Date.now();
normalizeForeignMarkdown(doc);
expect(Date.now() - t0).toBeLessThan(2000);
});
it('does not crash or slow down on thousands of prefix-chain definition ids', () => {
// F7: the rewrite must use a FIXED generic scanner, not an alternation built
// from the ids. A `(a|aa|aaa|…)` alternation over prefix-chain ids blows the
// V8 regex compiler (FATAL RegExpCompiler Allocation failed — uncatchable,
// kills the process). A fixed scanner has no id-dependent compilation cost.
const N = 4000;
const ids = Array.from({ length: N }, (_, i) => 'a'.repeat(i + 1));
const defs = ids.map((id) => `[^${id}]: body ${id.length}`).join('\n');
const doc = `ref[^${ids[0]}] and[^${ids[N - 1]}] end\n\n${defs}`;
const t0 = Date.now();
const out = normalizeForeignMarkdown(doc);
expect(Date.now() - t0).toBeLessThan(2000);
// Prefix disambiguation is correct: [^a] and [^aaaa...] inline their OWN body.
expect(out).toContain('^[body 1]');
expect(out).toContain(`^[body ${N}]`);
});
it('strips a CRLF (Windows) front-matter block, not just LF', () => {
// F9: the line-anchored regex needs LF after the opening `---`, so a Windows
// file (`---\r\n…`) would slip past the strip and leak the front-matter into
// the body. normalizeForeignMarkdown normalizes CRLF -> LF first.
const out = normalizeForeignMarkdown(
'---\r\ntitle: Foo\r\ntags: [a]\r\n---\r\n\r\n# Heading\r\n\r\nBody.',
);
expect(out).toBe('# Heading\n\nBody.');
expect(out).not.toContain('title: Foo');
expect(out).not.toContain('---');
});
it('strips front-matter whose value contains a triple-dash (line-anchored)', () => {
// F8: the block must close only on a `\n---` LINE, not the first inline
// `---`. A value like `title: Q1 --- Q2` must not truncate the front-matter
// and leak the rest (author/closing ---) into the body.
const out = normalizeForeignMarkdown(
'---\ntitle: Q1 --- Q2 results\nauthor: bob\n---\n\nReal body.',
);
expect(out).toBe('Real body.');
expect(out).not.toContain('author: bob');
expect(out).not.toContain('Q2 results');
});
});
describe('foreign markdown import acceptance (normalizer + canonical parser)', () => {
const FOREIGN = [
'# Doc',
'',
'Body refs [^c] and [^a] and [^b] and again [^a].',
'',
':::info',
'A legacy callout.',
':::',
'',
'| h1 | h2 |',
'| --- | --- |',
'| 1 | 2 |',
'',
'[^a]: note A',
'[^b]: note B',
'[^c]: note C',
'[^z]: orphan note',
].join('\n');
it('leaves no literal [^id] or ::: in the imported doc and re-exports canonically', async () => {
const normalized = normalizeForeignMarkdown(FOREIGN);
const doc = await markdownToProseMirror(normalized);
const reexport = convertProseMirrorToMarkdown(doc);
// No foreign garbage leaks into the document.
expect(reexport).not.toMatch(/\[\^/); // no reference footnote refs/defs
expect(reexport).not.toContain(':::'); // no legacy callout fences
// Canonical forms are present.
expect(reexport).toContain('^[note C]');
expect(reexport).toContain('> [!info]');
expect(reexport).toContain('| h1 | h2 |');
// Footnotes: ordered by first reference (C, A, B), reused [^a] deduped to one,
// orphan [^z] dropped (it had no reference after normalization).
const list = doc.content.find((n: any) => n.type === 'footnotesList');
const bodies = list.content.map(
(d: any) => d.content[0].content[0].text,
);
expect(bodies).toEqual(['note C', 'note A', 'note B']);
expect(bodies).not.toContain('orphan note');
expect(
doc.content.filter((n: any) => n.type === 'footnotesList'),
).toHaveLength(1);
});
});
@@ -1,265 +0,0 @@
/**
* Foreign-markdown normalizer an input-liberal / output-canonical adapter that
* runs at the IMPORT boundary, BEFORE the canonical parser
* (`markdownToProseMirror` from `@docmost/prosemirror-markdown`).
*
* The canonical parser is deliberately STRICT: it only understands Docmost's
* canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
* inline footnotes `^[body]`, lossless `![alt](src) <!--img {...}-->` images, ).
* Import, however, ingests FOREIGN files (GitHub/GFM, Notion, old Docmost
* exports). Those use surfaces the canonical parser does not accept, most notably
* GitHub-flavoured *reference* footnotes:
*
* Text with a note[^1] and another[^long].
*
* [^1]: The first definition.
* [^long]: A second one.
*
* Left untouched, the parser does NOT recognise `[^id]` (it only parses `^[body]`),
* so the reference leaks as literal text and worse, the trailing `[^id]: def`
* line is a valid CommonMark *link-reference definition*, so `[^id]` is silently
* rendered as a bogus link. This normalizer rewrites reference footnotes into the
* canonical inline form so the parser materialises real footnote nodes.
*
* This is a TEXT pre-pass, NOT a second parser fork: it does not re-implement any
* converter logic. Callout surfaces (`:::type` and `> [!type]`) are intentionally
* NOT touched here the canonical parser already accepts BOTH natively (its
* `preprocessCallouts` pass), so normalizing them would be redundant and would
* only risk degrading the parser's nesting/code-fence-aware handling.
*/
/** Matches a fenced code block delimiter (``` or ~~~), capturing the marker run. */
const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
/**
* Matches a GFM footnote DEFINITION line: `[^id]: body`. The id is any run of
* non-`]` characters; the body is the remainder of the line (possibly empty).
*/
const FOOTNOTE_DEF_RE = /^\[\^([^\]]+)\]:[ \t]?(.*)$/;
/** True when a line is a code-fence delimiter that toggles fenced-code state. */
function fenceMarker(line: string): string | null {
const m = line.match(CODE_FENCE_RE);
return m ? m[2] : null;
}
/** True when a line is indented (leading space/tab) and not blank — a continuation. */
function isIndentedContinuation(line: string): boolean {
return /^[ \t]+\S/.test(line);
}
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
/**
* Backslash-escape any square bracket in a footnote body before it is wrapped in
* `^[...]`. The canonical inline-footnote tokenizer scans the body with bracket
* balancing and closes on the first UNMATCHED `]`, so an unbalanced bracket in a
* foreign definition (e.g. `[^1]: see item ] later`) would otherwise truncate the
* footnote and leak the tail as literal text. Escaping every `[`/`]` makes the
* body an inert run of characters the tokenizer then closes only on our own
* closing `]`. (A balanced `[link](url)` inside a body still round-trips because
* the escaped form renders the literal brackets, which is the safe reading for a
* footnote body; the alternative brittle balance tracking risks worse.)
*/
function escapeFootnoteBody(body: string): string {
return body.replace(/[[\]]/g, '\\$&');
}
/**
* Rewrite every `[^id]` reference on a line to its `^[body]` form, but ONLY in the
* text OUTSIDE inline-code spans. A `[^id]` inside backticks is literal code
* content and must be preserved verbatim (a footnote ref never lives inside code).
* We split the line on inline-code spans (paired backtick runs) and rewrite only
* the non-code segments.
*/
// Above this length a single line is not split into inline-code spans (see
// below). A genuine markdown line carrying a footnote reference is never tens of
// KB; the cap only bypasses the inline-code protection for pathological lines.
const INLINE_SPLIT_MAX_LINE = 8192;
function rewriteRefsOutsideInlineCode(
line: string,
replace: (text: string) => string,
): string {
// The inline-code split alternation `(`+)(?:(?!\1)[\s\S])*\1` backtracks
// quadratically on a long UNCLOSED backtick run (its middle can consume the
// rest of the line, then fail to find a closing run and retry from each
// position). On an untrusted import this is a request-thread ReDoS. A real
// footnote line is short, so for an oversized line we skip the inline-code
// protection entirely and leave the line UNTOUCHED (rewriting it wholesale
// could corrupt a `[^id]` that legitimately lives inside inline code). This is
// a conservative bypass: an over-8KB line simply does not get its reference
// footnotes inlined — acceptable for a pathological input.
if (line.length > INLINE_SPLIT_MAX_LINE) return line;
// Alternation: an inline-code span (one or more backticks, then anything up to
// the SAME run of backticks) OR a run of non-backtick text. Unterminated
// backticks fall through as ordinary text (matched by the second branch on the
// leftover), so a stray backtick never swallows the rest of the line.
const parts = line.match(/(`+)(?:(?!\1)[\s\S])*\1|[^`]+|`+/g);
if (!parts) return line;
return parts
.map((seg) => (seg.startsWith('`') ? seg : replace(seg)))
.join('');
}
/**
* Convert GFM reference footnotes (`[^id]` + `[^id]: def`) into canonical inline
* footnotes (`^[def]`).
*
* - Definitions are collected first (a leading `[^id]: text` line plus any
* immediately-following indented continuation lines, joined with a space) and
* removed from the output.
* - Each in-text reference `[^id]` for which a definition was found is replaced by
* `^[def]`. References with no matching definition are left literal (there is no
* body to inline; the parser fails them open the same way).
* - Code is respected on both passes: `[^id]` inside a fenced ``` / ~~~ block is
* never rewritten and a `[^id]:` line inside a fence is never a definition; and
* on the rewrite pass a `[^id]` inside an INLINE-code span (backticks) is left
* literal too.
* - The inlined body is bracket-escaped so an unbalanced `[`/`]` in a foreign
* definition cannot truncate the resulting `^[...]` footnote.
*
* Deduplication / reference-ordering / orphan-dropping of the resulting footnotes
* is handled downstream by the canonical parser (`assembleFootnotes`); this pass
* only changes the surface syntax.
*/
function convertReferenceFootnotes(markdown: string): string {
const lines = markdown.split('\n');
// Pass 1: collect definitions and mark their lines for removal.
const defs = new Map<string, string>();
const dropped = new Array<boolean>(lines.length).fill(false);
let inFence = false;
let fence = '';
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
const marker = fenceMarker(line);
if (inFence) {
if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
inFence = false;
fence = '';
}
continue;
}
if (marker) {
inFence = true;
fence = marker;
continue;
}
const def = line.match(FOOTNOTE_DEF_RE);
if (!def) continue;
const id = def[1];
const body: string[] = [def[2].trim()];
dropped[i] = true;
// Consume immediately-following indented continuation lines (GFM lazy
// continuation is not supported by design — keep it simple and predictable).
let j = i + 1;
while (j < lines.length && isIndentedContinuation(lines[j])) {
body.push(lines[j].trim());
dropped[j] = true;
j++;
}
i = j - 1;
// Last definition wins for a duplicated id (matches CommonMark link-ref
// semantics closely enough for a foreign-input adapter).
defs.set(id, body.filter((s) => s.length > 0).join(' '));
}
if (defs.size === 0) {
return markdown;
}
// ONE fixed, generic scanner regex — NOT one built from the definition ids.
// It matches ANY `[^id]` shape, and the replacer decides per match via a map
// lookup whether that id is a real definition (replace) or not (leave as-is).
// This is genuinely O(total text) with no per-document regex compilation.
//
// Do NOT rebuild this as an alternation over `[...defs.keys()]`: a giant
// `(id1|id2|...)` alternation over thousands of ids can blow the V8 regex
// compiler's stack — a fatal, UNCATCHABLE "RegExpCompiler Allocation failed"
// on prefix-chain ids (`a`, `aa`, `aaa`, ...) that kills the whole process
// (worse than the earlier per-def thread-hang). A fixed scanner has no
// id-dependent compilation cost and cannot blow up.
const refRe = /\[\^([^\]]+)\]/g;
const rewriteSegment = (segment: string): string =>
segment.replace(refRe, (whole, id: string) => {
const body = defs.get(id);
// Only real definitions are inlined; an unknown id is left literal (same as
// the old per-def loop, which simply never matched it).
return body === undefined ? whole : `^[${escapeFootnoteBody(body)}]`;
});
// Pass 2: rewrite in-text references, skipping fenced code and dropped lines.
const out: string[] = [];
inFence = false;
fence = '';
for (let i = 0; i < lines.length; i++) {
if (dropped[i]) continue;
let line = lines[i];
const marker = fenceMarker(line);
if (inFence) {
out.push(line);
if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
inFence = false;
fence = '';
}
continue;
}
if (marker) {
inFence = true;
fence = marker;
out.push(line);
continue;
}
line = rewriteRefsOutsideInlineCode(line, rewriteSegment);
out.push(line);
}
return out.join('\n');
}
/**
* Strip a single leading YAML front-matter block (`---\n…\n---`). Foreign files
* from Obsidian / Hugo / Jekyll / Notion and Docmost's OWN git-sync page files
* open with front-matter that the canonical parser does not consume, so
* without this it leaks into the body (and `title: Foo` above the closing `---`
* renders as a setext `<h2>` that `extractTitleAndRemoveHeading` can hijack as
* the page title). It is a no-op for front-matter-free input.
*
* LINE-ANCHORED (the same shape the canonical parser uses in
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
* very start and closes only on a `\n---` line. The retired `markdownToHtml`
* strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
* and leaked the rest into the body. An optional leading BOM is tolerated.
*/
const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
/**
* Normalize a foreign markdown string into Docmost's canonical markdown surface
* so the strict canonical parser accepts it losslessly: normalize line endings,
* strip a leading YAML front-matter block, then rewrite GFM reference footnotes
* into inline footnotes. Add further fixture-driven foreign-surface cases here as
* they are found.
*/
export function normalizeForeignMarkdown(markdown: string): string {
if (!markdown) return markdown;
// Normalize CRLF -> LF FIRST. The line-anchored front-matter regex requires a
// bare `\n` after the opening `---`, and convertReferenceFootnotes splits on
// `\n`; a Windows/CRLF foreign file (`---\r\n…`) would otherwise slip past the
// front-matter strip and leak into the body. The canonical parser
// (page-file.ts parsePageFile) normalizes the same way before its FRONTMATTER_RE.
const src = markdown.replace(/\r\n/g, '\n');
const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
return convertReferenceFootnotes(withoutFrontMatter);
}
-23
View File
@@ -1,23 +0,0 @@
// Jest stub for @tiptap/react.
//
// The server export/import code paths transitively import editor-ext, whose node
// extensions import from `@tiptap/react`. The real module re-exports all of
// `@tiptap/core` (headless, safe under node) AND adds React view helpers
// (`ReactNodeViewRenderer`, …) that eagerly pull in react-dom — which throws
// `navigator is not defined` under jest's node environment.
//
// So this stub DELEGATES to the real `@tiptap/core` (keeping `mergeAttributes`,
// `Node`, `Mark`, `nodeInputRule`, … working — they are used by
// `jsonToHtml`/`htmlToJson` on the server) and overrides ONLY the React view
// helpers with no-ops. Those helpers are referenced solely inside `addNodeView()`
// — code that runs only in a live browser editor, never on the server; if any
// were actually invoked here it would (correctly) surface as a test failure.
const core = require('@tiptap/core');
module.exports = {
...core,
ReactNodeViewRenderer: () => () => ({}),
NodeViewWrapper: () => null,
NodeViewContent: () => null,
ReactRenderer: class {},
};
+1
View File
@@ -20,6 +20,7 @@ export * from "./lib/html-embed/html-embed";
export * from "./lib/mention";
export * from "./lib/markdown";
export * from "./lib/search-and-replace";
export * from "./lib/multi-cursor";
export * from "./lib/embed-provider";
export * from "./lib/subpages";
export * from "./lib/transclusion";
@@ -0,0 +1,3 @@
import { MultiCursor } from "./multi-cursor";
export * from "./multi-cursor";
export default MultiCursor;
@@ -0,0 +1,453 @@
import { describe, it, expect } from "vitest";
import { Editor } from "@tiptap/core";
import { Document } from "@tiptap/extension-document";
import { Paragraph } from "@tiptap/extension-paragraph";
import { Text } from "@tiptap/extension-text";
import { Bold } from "@tiptap/extension-bold";
import { Node as PMNode } from "@tiptap/pm/model";
import { MultiCursor, multiCursorPluginKey, MAX_CURSORS } from "./multi-cursor";
import { findOccurrences } from "../search-and-replace/find-occurrences";
const extensions = [Document, Paragraph, Text, Bold, MultiCursor];
function makeEditor(content?: any) {
return new Editor({
extensions,
content: content ?? { type: "doc", content: [{ type: "paragraph" }] },
});
}
function doc(...paragraphs: string[]) {
return {
type: "doc",
content: paragraphs.map((text) => ({
type: "paragraph",
content: text ? [{ type: "text", text }] : [],
})),
};
}
function paraTexts(d: PMNode): string[] {
const out: string[] = [];
d.forEach((node) => {
if (node.type.name === "paragraph") out.push(node.textContent);
});
return out;
}
function cursors(editor: Editor) {
return multiCursorPluginKey.getState(editor.state)!.cursors;
}
// Simulate typing a character through the real handleTextInput routing (the
// browser path). someMethod-equivalent: dispatch a DOM-ish text input by calling
// the view's input handler directly.
function typeText(editor: Editor, text: string) {
const { from, to } = editor.state.selection;
// props.handleTextInput is what ProseMirror calls on beforeinput/keypress.
const handled = editor.view.someProp(
"handleTextInput",
(fn) => fn(editor.view, from, to, text) || false,
);
if (!handled) {
// Fall back to a normal insertion (no active multi-cursor set).
editor.view.dispatch(editor.state.tr.insertText(text, from, to));
}
}
function pressKey(editor: Editor, key: string) {
editor.view.someProp("handleKeyDown", (fn) =>
fn(editor.view, new KeyboardEvent("keydown", { key })),
);
}
describe("multi-cursor: selectAllOccurrences", () => {
it("finds EVERY occurrence of a repeated word under the cursor", () => {
const editor = makeEditor(doc("foo bar foo baz foo"));
// Cursor inside the first "foo".
editor.commands.setTextSelection(2);
expect(editor.commands.selectAllOccurrences()).toBe(true);
const cs = cursors(editor);
expect(cs.length).toBe(3);
// Every cursor spans a "foo".
for (const c of cs) {
expect(editor.state.doc.textBetween(c.from, c.to)).toBe("foo");
}
editor.destroy();
});
it("uses the current non-empty selection as the term", () => {
const editor = makeEditor(doc("ab abc ab abcd ab"));
// Select the first "ab".
editor.commands.setTextSelection({ from: 1, to: 3 });
expect(editor.state.doc.textBetween(1, 3)).toBe("ab");
editor.commands.selectAllOccurrences();
// Literal substring match (selection is not whole-word), so every "ab"
// including those inside "abc"/"abcd" is matched: 5 total.
const cs = cursors(editor);
expect(cs.length).toBe(5);
editor.destroy();
});
it("whole-word matching from a word cursor does not match substrings", () => {
const editor = makeEditor(doc("cat category cat scatter cat"));
editor.commands.setTextSelection(2); // inside first "cat"
editor.commands.selectAllOccurrences();
// Only the three standalone "cat" words, not "category"/"scatter".
expect(cursors(editor).length).toBe(3);
editor.destroy();
});
});
describe("multi-cursor: mass typing (single transaction)", () => {
it("types text into N carets at once", () => {
const editor = makeEditor(doc("foo foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(3);
// Typing replaces each selected "foo" with "X".
typeText(editor, "X");
expect(paraTexts(editor.state.doc)).toEqual(["X X X"]);
// The cursors are now carets right after each inserted "X".
const cs = cursors(editor);
expect(cs.length).toBe(3);
for (const c of cs) expect(c.from).toBe(c.to);
editor.destroy();
});
it("continues typing at the resulting carets (append semantics)", () => {
const editor = makeEditor(doc("a a a"));
editor.commands.setTextSelection(1);
editor.commands.selectAllOccurrences();
typeText(editor, "b"); // each "a" -> "b"
typeText(editor, "c"); // append at each caret -> "bc"
expect(paraTexts(editor.state.doc)).toEqual(["bc bc bc"]);
editor.destroy();
});
it("applies the whole multi-edit in a SINGLE transaction (one undo step)", () => {
// "One Cmd/Ctrl+Z undoes the whole multi-edit" holds iff the N edits land in
// ONE transaction (history groups by transaction). @tiptap/extension-history
// is not a dependency here, so rather than exercise undo we assert the
// property that guarantees it: typing into N cursors is exactly ONE dispatch.
const editor = makeEditor(doc("foo foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(3);
const orig = editor.view.dispatch.bind(editor.view);
let dispatches = 0;
editor.view.dispatch = (tr) => {
dispatches += 1;
return orig(tr);
};
typeText(editor, "Z");
editor.view.dispatch = orig;
expect(dispatches).toBe(1); // all three edits share one transaction
expect(paraTexts(editor.state.doc)).toEqual(["Z Z Z"]);
editor.destroy();
});
it("off-by-one guard: reverse-order iteration keeps every position valid", () => {
// If the mass edit iterated FORWARD, inserting at an earlier cursor would
// shift every later cursor and corrupt the result. Different-length
// replacement makes such a bug visible.
const editor = makeEditor(doc("x x x x"));
editor.commands.setTextSelection(1);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(4);
typeText(editor, "LONG");
expect(paraTexts(editor.state.doc)).toEqual(["LONG LONG LONG LONG"]);
editor.destroy();
});
});
describe("multi-cursor: mass Backspace / Delete", () => {
it("Backspace removes one char before each caret", () => {
const editor = makeEditor(doc("foo foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
// Collapse selections to carets at the END of each "foo" by typing then
// removing is complex; instead type to convert ranges into carets first.
typeText(editor, "ab"); // each "foo" -> "ab", carets after "ab"
expect(paraTexts(editor.state.doc)).toEqual(["ab ab ab"]);
pressKey(editor, "Backspace"); // remove the trailing "b" at each caret
expect(paraTexts(editor.state.doc)).toEqual(["a a a"]);
editor.destroy();
});
it("Delete removes one char after each caret", () => {
const editor = makeEditor(doc("fooX fooX"));
// Literal (selection) match of "foo" -> both occurrences inside "fooX".
editor.commands.setTextSelection({ from: 1, to: 4 }); // first "foo"
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(2);
typeText(editor, "foo"); // rewrite "foo", carets now sit before each "X"
expect(paraTexts(editor.state.doc)).toEqual(["fooX fooX"]);
pressKey(editor, "Delete"); // remove the "X" after each caret
expect(paraTexts(editor.state.doc)).toEqual(["foo foo"]);
editor.destroy();
});
it("Backspace at a block-start caret is a no-op for that cursor", () => {
const editor = makeEditor(doc("ab", "ab"));
// Select both "ab" then convert to carets at start by replacing with "".
editor.commands.setTextSelection({ from: 1, to: 3 }); // first "ab"
editor.commands.selectAllOccurrences();
// Move carets to block start: type "" is not possible; instead delete range.
pressKey(editor, "Backspace"); // deletes each selected "ab"
expect(paraTexts(editor.state.doc)).toEqual(["", ""]);
// Carets are now at each block start; another Backspace must not throw and
// must not merge blocks (still two empty paragraphs).
pressKey(editor, "Backspace");
expect(paraTexts(editor.state.doc)).toEqual(["", ""]);
editor.destroy();
});
});
describe("multi-cursor: addNextOccurrence (Cmd/Ctrl+D)", () => {
it("first press selects the current word, next press adds the next", () => {
const editor = makeEditor(doc("go go go"));
editor.commands.setTextSelection(2); // inside first "go"
editor.commands.addNextOccurrence();
expect(cursors(editor).length).toBe(1);
editor.commands.addNextOccurrence();
expect(cursors(editor).length).toBe(2);
editor.commands.addNextOccurrence();
expect(cursors(editor).length).toBe(3);
// Nothing left to add — stays at 3.
editor.commands.addNextOccurrence();
expect(cursors(editor).length).toBe(3);
for (const c of cursors(editor)) {
expect(editor.state.doc.textBetween(c.from, c.to)).toBe("go");
}
editor.destroy();
});
});
describe("multi-cursor: position remapping", () => {
it("remaps cursors after a LOCAL edit before them", () => {
const editor = makeEditor(doc("foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
const before = cursors(editor).map((c) => ({ ...c }));
// Insert unrelated text at the very start (pos 1), shifting everything +5.
editor.view.dispatch(editor.state.tr.insertText("HELLO", 1));
const after = cursors(editor);
expect(after.length).toBe(before.length);
for (let i = 0; i < after.length; i += 1) {
expect(after[i].from).toBe(before[i].from + 5);
expect(after[i].to).toBe(before[i].to + 5);
// And they still point at "foo".
expect(editor.state.doc.textBetween(after[i].from, after[i].to)).toBe(
"foo",
);
}
editor.destroy();
});
it("remaps cursors after a simulated REMOTE edit (ordinary transaction)", () => {
const editor = makeEditor(doc("foo bar foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
const before = cursors(editor).map((c) => ({ ...c }));
expect(before.length).toBe(2);
// y-prosemirror applies remote changes as ordinary transactions. Emulate a
// remote insertion between the two "foo"s (inside "bar", pos 6) with a tr
// that carries NO multi-cursor meta — exactly like a collaborator's edit.
const tr = editor.state.tr.insertText("ZZ", 6);
editor.view.dispatch(tr);
const after = cursors(editor);
// The first "foo" (before the insertion) is unchanged; the second shifts +2.
expect(after[0].from).toBe(before[0].from);
expect(after[1].from).toBe(before[1].from + 2);
for (const c of after) {
expect(editor.state.doc.textBetween(c.from, c.to)).toBe("foo");
}
editor.destroy();
});
it("a REMOTE delete UNDER a cursor collapses it to a caret (not drop), leaving others intact", () => {
// The riskiest remap path: a collaborator deletes the very text one cursor
// spans. Both edges map with assoc +1 and there is no drop logic, so the
// deleted-over cursor CONTRACT is: it collapses to a zero-width caret at the
// deletion point (from === to) and STAYS in the set — it is not removed.
// Untouched cursors keep spanning their occurrence. Pinning this makes the
// collapse-not-drop choice explicit (review #372 F2).
const editor = makeEditor(doc("foo bar foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
const before = cursors(editor).map((c) => ({ ...c }));
expect(before.length).toBe(2);
// Remote (no multi-cursor meta) delete of the FIRST "foo" range.
const tr = editor.state.tr.delete(before[0].from, before[0].to);
editor.view.dispatch(tr);
const after = cursors(editor);
// Still two cursors — the deleted-over one is NOT dropped.
expect(after.length).toBe(2);
// The first collapsed to a caret at the deletion point.
expect(after[0].from).toBe(after[0].to);
expect(after[0].from).toBe(before[0].from);
// The second still spans "foo" (shifted left by the 3 removed chars).
expect(after[1].from).toBe(before[1].from - 3);
expect(editor.state.doc.textBetween(after[1].from, after[1].to)).toBe("foo");
// Sanity: the document now reads " bar foo".
expect(paraTexts(editor.state.doc)).toEqual([" bar foo"]);
editor.destroy();
});
});
describe("multi-cursor: collapse / exit", () => {
it("exitMultiCursor clears the set", () => {
const editor = makeEditor(doc("foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(2);
editor.commands.exitMultiCursor();
expect(cursors(editor).length).toBe(0);
editor.destroy();
});
it("an arrow key collapses the set", () => {
const editor = makeEditor(doc("foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(2);
pressKey(editor, "ArrowRight");
expect(cursors(editor).length).toBe(0);
editor.destroy();
});
});
describe("multi-cursor: collapse on composition / mousedown", () => {
// Invoke a plugin handleDOMEvents handler through the real prop plumbing.
function fireDOM(editor: Editor, name: string): void {
editor.view.someProp("handleDOMEvents", (handlers: any) => {
const h = handlers && handlers[name];
if (h) h(editor.view, new Event(name));
return false;
});
}
it("collapses the set on compositionstart (IME) — MVP does not multi-IME", () => {
const editor = makeEditor(doc("foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(2);
fireDOM(editor, "compositionstart");
expect(cursors(editor).length).toBe(0);
editor.destroy();
});
it("collapses the set on a plain mousedown (VS Code behaviour)", () => {
const editor = makeEditor(doc("foo foo"));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(2);
fireDOM(editor, "mousedown");
expect(cursors(editor).length).toBe(0);
editor.destroy();
});
});
describe("multi-cursor: hard cap", () => {
it("never activates more than MAX_CURSORS cursors", () => {
const many = new Array(MAX_CURSORS + 20).fill("w").join(" ");
const editor = makeEditor(doc(many));
editor.commands.setTextSelection(2);
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(MAX_CURSORS);
editor.destroy();
});
});
describe("multi-cursor: marks are carried across a mass edit", () => {
it("preserves marks spanning each replaced range", () => {
const editor = makeEditor({
type: "doc",
content: [
{
type: "paragraph",
content: [
{ type: "text", text: "a " },
{ type: "text", marks: [{ type: "bold" }], text: "key" },
{ type: "text", text: " b " },
{ type: "text", marks: [{ type: "bold" }], text: "key" },
],
},
],
});
editor.commands.setTextSelection(3); // inside first bold "key"
editor.commands.selectAllOccurrences();
expect(cursors(editor).length).toBe(2);
typeText(editor, "NEW");
// Both replacements keep the bold mark.
let boldRuns = 0;
editor.state.doc.descendants((node) => {
if (
node.isText &&
node.text === "NEW" &&
node.marks.some((m) => m.type.name === "bold")
) {
boldRuns += 1;
}
});
expect(boldRuns).toBe(2);
editor.destroy();
});
});
// The extracted find-occurrences util must return the SAME occurrences that the
// old inline walk produced (and that search-and-replace still relies on).
describe("find-occurrences util", () => {
it("finds all matches of a literal regex across text nodes", () => {
const editor = makeEditor(doc("foo foofoo foo"));
const results = findOccurrences(editor.state.doc, /foo/gu);
// 4 occurrences: two standalone + two inside "foofoo".
expect(results.length).toBe(4);
for (const r of results) {
expect(editor.state.doc.textBetween(r.from, r.to)).toBe("foo");
}
editor.destroy();
});
it("ignores whitespace-only matches and empty regex", () => {
const editor = makeEditor(doc("a b c"));
expect(findOccurrences(editor.state.doc, null as any).length).toBe(0);
// A whitespace regex yields no results (matches are trimmed away).
expect(findOccurrences(editor.state.doc, /\s/gu).length).toBe(0);
editor.destroy();
});
it("finds a match spanning two differently-marked contiguous text nodes", () => {
const editor = makeEditor({
type: "doc",
content: [
{
type: "paragraph",
content: [
{ type: "text", text: "wo" },
{ type: "text", marks: [{ type: "bold" }], text: "rd" },
],
},
],
});
const results = findOccurrences(editor.state.doc, /word/gu);
expect(results.length).toBe(1);
expect(editor.state.doc.textBetween(results[0].from, results[0].to)).toBe(
"word",
);
editor.destroy();
});
});
@@ -0,0 +1,545 @@
import { Extension, Range } from "@tiptap/core";
import { Decoration, DecorationSet, EditorView } from "@tiptap/pm/view";
import {
Plugin,
PluginKey,
TextSelection,
type EditorState,
} from "@tiptap/pm/state";
import { Mark } from "@tiptap/pm/model";
import { findOccurrences } from "../search-and-replace/find-occurrences";
/**
* Multi-cursor editing MVP (issue #196, "Variant A").
*
* VS Code-style multi-cursor limited to "select all occurrences of a word (or
* the current selection) and type into all of them at once", built ON TOP OF
* the search-and-replace mass-transaction machinery:
*
* - Cmd/Ctrl+Shift+L (selectAllOccurrences): the word under the cursor (or the
* current non-empty selection) -> ALL its occurrences become active cursors.
* - Cmd/Ctrl+D (addNextOccurrence): add the NEXT occurrence of the term.
* - Typing / Backspace / Delete apply to EVERY active cursor in ONE
* transaction (so a single Cmd/Ctrl+Z undoes the whole multi-edit).
* - Esc (exitMultiCursor): collapse back to a single cursor.
*
* The single-transaction, reverse-order edit mechanic mirrors `replaceAll` in
* search-and-replace.ts: we iterate cursors from the END of the document to the
* START so an earlier edit never invalidates a later position, carrying the
* marks that span each range.
*
* CONSCIOUS v1 OUT-OF-SCOPE BOUNDARIES (these are "Variant B", deliberately NOT
* built here):
* - Alt+Click arbitrary carets and Alt+drag column selection.
* - Cmd/Ctrl+Alt+Up/Down "add cursor on the adjacent line".
* - Simultaneous IME / composition input into multiple positions on
* `compositionstart` we collapse back to a single cursor.
* - Cursors spanning different schema nodes in one edit.
*
* NOT out of scope, but worth stating precisely: there is NO schema-aware or
* structural cursor. Occurrences are found by a plain text-node walk
* (`findOccurrences`), so a term that appears inside a table cell, code block or
* callout DOES get a cursor there and IS edited as plain text, exactly like
* `replaceAll`. There is no special table/code handling; the per-cursor try/catch
* only SKIPS a cursor whose edit would violate the schema (never applied
* half-way), it does not exclude those node types from matching.
*/
interface MultiCursorState {
// Each active cursor: a caret when from === to, a range when from < to.
cursors: Range[];
}
export const multiCursorPluginKey = new PluginKey<MultiCursorState>(
"multiCursor",
);
// Hard safety cap on simultaneously-active cursors — stop adding past it.
export const MAX_CURSORS = 100;
export interface MultiCursorStorage {
// Whether the active term matches whole words only. Set to true when the set
// was seeded from a bare cursor (word under caret), false when seeded from an
// explicit selection (literal substring match, like VS Code). Remembered so
// addNextOccurrence keeps matching the same way as selectAllOccurrences.
wholeWord: boolean;
}
declare module "@tiptap/core" {
interface Storage {
multiCursor: MultiCursorStorage;
}
interface Commands<ReturnType> {
multiCursor: {
/** Select all occurrences of the word/selection as active cursors. */
selectAllOccurrences: () => ReturnType;
/** Add the next occurrence of the current term to the cursor set. */
addNextOccurrence: () => ReturnType;
/** Collapse the multi-cursor set back to a single cursor. */
exitMultiCursor: () => ReturnType;
};
}
}
// ---------------------------------------------------------------------------
// Term helpers
// ---------------------------------------------------------------------------
function escapeRegExp(s: string): string {
return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
// A "word" is a run of letters/numbers/underscore; those get whole-word
// matching (\b…\b) so a term never matches inside a larger word. Anything else
// (punctuation, phrases) is matched literally. Case-sensitive, like VS Code.
function isWordTerm(s: string): boolean {
return /^[\p{L}\p{N}_]+$/u.test(s);
}
// wholeWord uses \b…\b so the term never matches inside a larger word; it only
// applies to word-like terms (a term containing punctuation cannot be
// whole-word-bounded meaningfully). Otherwise the term is matched literally.
function buildTermRegex(term: string, wholeWord: boolean): RegExp {
const esc = escapeRegExp(term);
return wholeWord && isWordTerm(term)
? new RegExp(`\\b${esc}\\b`, "gu")
: new RegExp(esc, "gu");
}
// Word under a position: returns the exact { from, to } range and its text, or
// null if the position is not inside a word in a textblock.
function getWordAt(
state: EditorState,
pos: number,
): { from: number; to: number; text: string } | null {
const $pos = state.doc.resolve(pos);
const parent = $pos.parent;
if (!parent.isTextblock) return null;
const text = parent.textContent;
const offset = $pos.parentOffset;
const start = $pos.start();
const wordRe = /[\p{L}\p{N}_]+/gu;
let m: RegExpExecArray | null;
while ((m = wordRe.exec(text)) !== null) {
const s = m.index;
const e = m.index + m[0].length;
if (offset >= s && offset <= e) {
return { from: start + s, to: start + e, text: m[0] };
}
}
return null;
}
// ---------------------------------------------------------------------------
// Plugin-state access
// ---------------------------------------------------------------------------
function getCursors(state: EditorState): Range[] {
const st = multiCursorPluginKey.getState(state);
return st ? st.cursors : [];
}
function setCursors(view: EditorView, cursors: Range[]): void {
view.dispatch(view.state.tr.setMeta(multiCursorPluginKey, cursors));
}
function collapse(view: EditorView): void {
setCursors(view, []);
}
// ---------------------------------------------------------------------------
// The single-transaction, reverse-order mass edit (mirrors replaceAll)
// ---------------------------------------------------------------------------
interface EditOp {
from: number;
to: number;
// Text to insert at `from` after deleting [from, to); "" for a pure delete.
text: string;
}
/**
* Apply one edit per cursor in ONE transaction. Ops are processed from the END
* of the document to the START so an earlier edit never shifts a later position
* (mirrors `replaceAll`). Each cursor is wrapped independently: a schema
* violation SKIPS that one cursor instead of throwing away the whole
* transaction, so the document is never left half-applied.
*
* After building the transaction the new cursor positions are recomputed by
* mapping each op's original anchor through `tr.mapping` (which also remaps any
* concurrent changes), so carets land right after their inserted text.
*/
function dispatchMassEdit(view: EditorView, ops: EditOp[]): boolean {
if (!ops.length) return false;
const { state } = view;
const tr = state.tr;
const schema = state.schema;
// Ascending by `from`; iterate reverse so earlier positions stay valid.
const sorted = [...ops].sort((a, b) => a.from - b.from);
const appliedLen: number[] = new Array(sorted.length).fill(0);
for (let i = sorted.length - 1; i >= 0; i -= 1) {
const { from, to, text } = sorted[i];
try {
let marks: readonly Mark[] = [];
if (text) {
if (to > from) {
// Carry all marks spanning the replaced range.
const set = new Set<Mark>();
tr.doc.nodesBetween(from, to, (node) => {
if (node.isText && node.marks) {
node.marks.forEach((mk) => set.add(mk));
}
});
marks = Array.from(set);
} else {
// Caret: continue the marks active at the insertion point.
marks = state.storedMarks || state.doc.resolve(from).marks();
}
}
// ONE atomic step per cursor: replaceWith covers both insert (from === to)
// and replace (to > from); a pure delete (empty text) uses delete. This
// can never leave a cursor half-applied (deleted but not re-inserted) the
// way a separate delete-then-insert pair could if the insert step threw.
if (text) {
tr.replaceWith(from, to, schema.text(text, marks as Mark[]));
} else if (to > from) {
tr.delete(from, to);
}
appliedLen[i] = text.length;
} catch {
// Per-cursor backstop (text-only MVP): drop this cursor's edit, keep the
// rest of the transaction intact.
appliedLen[i] = 0;
}
}
if (!tr.docChanged) return false;
// Recompute cursor carets from the ORIGINAL op anchors through the full map.
const newCursors: Range[] = sorted.map((op, i) => {
const start = tr.mapping.map(op.from, -1);
const caret = start + appliedLen[i];
return { from: caret, to: caret };
});
tr.setMeta(multiCursorPluginKey, newCursors);
// Park the native selection on the last caret so the browser draws exactly
// one real caret; the rest are our decoration widgets.
const last = newCursors[newCursors.length - 1];
tr.setSelection(TextSelection.create(tr.doc, last.from));
view.dispatch(tr);
return true;
}
function buildDeleteOps(
state: EditorState,
cursors: Range[],
forward: boolean,
): EditOp[] {
return cursors.map((c) => {
// A selected range: Backspace/Delete removes the whole range.
if (c.to > c.from) return { from: c.from, to: c.to, text: "" };
const $pos = state.doc.resolve(c.from);
if (forward) {
// Delete: at the end of a textblock there is nothing to remove (a no-op;
// MVP does not merge blocks across a multi-cursor set).
if ($pos.parentOffset >= $pos.parent.content.size) {
return { from: c.from, to: c.from, text: "" };
}
return { from: c.from, to: c.from + 1, text: "" };
}
// Backspace: at the start of a textblock there is nothing to remove.
if ($pos.parentOffset <= 0) {
return { from: c.from, to: c.from, text: "" };
}
return { from: c.from - 1, to: c.from, text: "" };
});
}
// ---------------------------------------------------------------------------
// Extension
// ---------------------------------------------------------------------------
export const MultiCursor = Extension.create<unknown, MultiCursorStorage>({
name: "multiCursor",
addStorage() {
return { wholeWord: true };
},
addCommands() {
return {
selectAllOccurrences:
() =>
({ editor, state, tr, dispatch }) => {
let term: string;
// A bare cursor expands to the whole word; an explicit selection is
// matched literally (VS Code semantics).
const wholeWord = state.selection.empty;
if (wholeWord) {
const word = getWordAt(state, state.selection.from);
if (!word) return false;
term = word.text;
} else {
term = state.doc.textBetween(
state.selection.from,
state.selection.to,
);
}
if (!term.trim()) return false;
editor.storage.multiCursor.wholeWord = wholeWord;
const results = findOccurrences(
state.doc,
buildTermRegex(term, wholeWord),
).slice(0, MAX_CURSORS);
if (!results.length) return false;
if (dispatch) {
tr.setMeta(multiCursorPluginKey, results);
const last = results[results.length - 1];
tr.setSelection(TextSelection.create(tr.doc, last.from, last.to));
dispatch(tr);
}
return true;
},
addNextOccurrence:
() =>
({ editor, state, tr, dispatch }) => {
const existing = getCursors(state);
let cursors: Range[];
if (!existing.length) {
// First press: turn the current word/selection into the one cursor.
let range: Range;
const wholeWord = state.selection.empty;
if (wholeWord) {
const word = getWordAt(state, state.selection.from);
if (!word) return false;
range = { from: word.from, to: word.to };
} else {
range = { from: state.selection.from, to: state.selection.to };
}
editor.storage.multiCursor.wholeWord = wholeWord;
cursors = [range];
} else {
// Subsequent press: add the next unselected occurrence of the term,
// matched the SAME way (whole-word vs literal) the set was seeded.
if (existing.length >= MAX_CURSORS) return true;
const first = existing[0];
const term = state.doc.textBetween(first.from, first.to);
if (!term.trim()) return false;
const results = findOccurrences(
state.doc,
buildTermRegex(term, editor.storage.multiCursor.wholeWord),
);
const keys = new Set(existing.map((c) => `${c.from}:${c.to}`));
const notSelected = results.filter(
(r) => !keys.has(`${r.from}:${r.to}`),
);
if (!notSelected.length) return true; // all occurrences selected
const maxTo = Math.max(...existing.map((c) => c.to));
const next =
notSelected.find((r) => r.from >= maxTo) || notSelected[0];
cursors = [...existing, next];
}
if (dispatch) {
tr.setMeta(multiCursorPluginKey, cursors);
const last = cursors[cursors.length - 1];
tr.setSelection(TextSelection.create(tr.doc, last.from, last.to));
dispatch(tr);
}
return true;
},
exitMultiCursor:
() =>
({ tr, dispatch }) => {
if (dispatch) {
tr.setMeta(multiCursorPluginKey, []);
dispatch(tr);
}
return true;
},
};
},
addKeyboardShortcuts() {
return {
"Mod-Shift-l": () => {
this.editor.commands.selectAllOccurrences();
// Always consume so the browser's default is prevented.
return true;
},
"Mod-d": () => {
this.editor.commands.addNextOccurrence();
// Consume unconditionally to prevent the browser's Cmd/Ctrl+D bookmark.
return true;
},
Escape: () => {
// Only swallow Escape while a multi-cursor set is active; otherwise let
// Escape keep its other behaviours (e.g. closing dialogs).
if (!getCursors(this.editor.state).length) return false;
return this.editor.commands.exitMultiCursor();
},
};
},
addProseMirrorPlugins() {
return [
new Plugin<MultiCursorState>({
key: multiCursorPluginKey,
state: {
init: () => ({ cursors: [] }),
apply(tr, value): MultiCursorState {
// A command (or a mass edit) can set/clear the cursor set directly.
// Its cursors are already in the post-transaction coordinate space,
// so they take priority over remapping.
const meta = tr.getMeta(multiCursorPluginKey) as
| Range[]
| undefined;
if (meta !== undefined) {
return { cursors: meta.slice(0, MAX_CURSORS) };
}
if (!value.cursors.length) return value;
// Remap surviving cursors across ANY doc change — this covers both
// local edits and REMOTE Yjs edits (y-prosemirror applies remote
// changes as ordinary transactions, so mapping them here keeps every
// multi-cursor correctly positioned without special-casing collab).
if (tr.docChanged) {
// Map both edges with the SAME association (+1) so content
// inserted at a boundary shifts the whole cursor right and a caret
// (from === to) can never invert into a range.
const cursors = value.cursors.map((c) => ({
from: tr.mapping.map(c.from, 1),
to: tr.mapping.map(c.to, 1),
}));
return { cursors };
}
return value;
},
},
props: {
decorations(state) {
const st = multiCursorPluginKey.getState(state);
if (!st || !st.cursors.length) return DecorationSet.empty;
const decorations: Decoration[] = [];
st.cursors.forEach((c, i) => {
if (c.from === c.to) {
decorations.push(
Decoration.widget(
c.from,
() => {
const el = document.createElement("span");
el.className = "multi-cursor__caret";
return el;
},
{ side: 0, key: `mc-caret-${i}` },
),
);
} else {
decorations.push(
Decoration.inline(c.from, c.to, {
class: "multi-cursor__selection",
}),
);
}
});
return DecorationSet.create(state.doc, decorations);
},
handleTextInput(view, _from, _to, text) {
const cursors = getCursors(view.state);
if (!cursors.length) return false;
// Insert `text` at EVERY cursor in one transaction. Returning true
// prevents ProseMirror's own single-position insert at the native
// selection, so there is no double-insert there.
const ops = cursors.map((c) => ({
from: c.from,
to: c.to,
text,
}));
return dispatchMassEdit(view, ops);
},
handleKeyDown(view, event) {
const cursors = getCursors(view.state);
if (!cursors.length) return false;
if (event.key === "Backspace") {
dispatchMassEdit(view, buildDeleteOps(view.state, cursors, false));
return true;
}
if (event.key === "Delete") {
dispatchMassEdit(view, buildDeleteOps(view.state, cursors, true));
return true;
}
// Let modifier combinations (our own shortcuts, copy, etc.) through
// WITHOUT collapsing the set.
if (event.metaKey || event.ctrlKey || event.altKey) return false;
// Navigation / block keys collapse back to a single cursor, then let
// ProseMirror handle the movement on the native selection.
const COLLAPSE_KEYS = [
"ArrowLeft",
"ArrowRight",
"ArrowUp",
"ArrowDown",
"Home",
"End",
"PageUp",
"PageDown",
"Enter",
"Tab",
];
if (COLLAPSE_KEYS.includes(event.key)) {
collapse(view);
return false;
}
return false;
},
handleDOMEvents: {
// A plain click exits multi-cursor (VS Code behaviour).
mousedown: (view) => {
if (getCursors(view.state).length) collapse(view);
return false;
},
// MVP does not drive multi-position IME — collapse on composition.
compositionstart: (view) => {
if (getCursors(view.state).length) collapse(view);
return false;
},
},
},
}),
];
},
});
export default MultiCursor;
@@ -0,0 +1,69 @@
import { Range } from "@tiptap/core";
import { Node as PMNode } from "@tiptap/pm/model";
interface TextNodesWithPosition {
text: string;
pos: number;
}
/**
* Shared "find all occurrences of a term in the doc" primitive.
*
* Walks every text node of the document and returns each regex match as a
* `{ from, to }` range. Contiguous text nodes (which may differ only by marks)
* are concatenated into a single run, so a match that spans e.g. "wo" + bold
* "rd" is still found; runs are split by any non-text node, so a match never
* crosses a node boundary. Whitespace-only matches are ignored.
*
* This is used by BOTH search-and-replace (highlight/replace) and multi-cursor
* (turn occurrences into active cursors) so the two stay behaviourally in sync.
* Extracted verbatim from the original `processSearches` walk.
*/
export function findOccurrences(doc: PMNode, searchTerm: RegExp): Range[] {
const results: Range[] = [];
if (!searchTerm) return results;
let textNodesWithPosition: TextNodesWithPosition[] = [];
let index = 0;
doc?.descendants((node, pos) => {
if (node.isText) {
if (textNodesWithPosition[index]) {
textNodesWithPosition[index] = {
text: textNodesWithPosition[index].text + node.text,
pos: textNodesWithPosition[index].pos,
};
} else {
textNodesWithPosition[index] = {
text: `${node.text}`,
pos,
};
}
} else {
index += 1;
}
});
textNodesWithPosition = textNodesWithPosition.filter(Boolean);
for (const element of textNodesWithPosition) {
const { text, pos } = element;
const matches = Array.from(text.matchAll(searchTerm)).filter(
([matchText]) => matchText.trim(),
);
for (const m of matches) {
if (m[0] === "") break;
if (m.index !== undefined) {
results.push({
from: pos + m.index,
to: pos + m.index + m[0].length,
});
}
}
}
return results;
}
@@ -1,3 +1,4 @@
import { SearchAndReplace } from './search-and-replace'
export * from './search-and-replace'
export * from './find-occurrences'
export default SearchAndReplace
@@ -29,6 +29,7 @@ import {
type Transaction,
} from "@tiptap/pm/state";
import { Node as PMNode, Mark } from "@tiptap/pm/model";
import { findOccurrences } from "./find-occurrences";
declare module "@tiptap/core" {
interface Storage {
@@ -76,11 +77,6 @@ declare module "@tiptap/core" {
}
}
interface TextNodesWithPosition {
text: string;
pos: number;
}
const getRegex = (
s: string,
disableRegex: boolean,
@@ -104,10 +100,6 @@ function processSearches(
resultIndex: number,
): ProcessedSearches {
const decorations: Decoration[] = [];
const results: Range[] = [];
let textNodesWithPosition: TextNodesWithPosition[] = [];
let index = 0;
if (!searchTerm) {
return {
@@ -116,43 +108,8 @@ function processSearches(
};
}
doc?.descendants((node, pos) => {
if (node.isText) {
if (textNodesWithPosition[index]) {
textNodesWithPosition[index] = {
text: textNodesWithPosition[index].text + node.text,
pos: textNodesWithPosition[index].pos,
};
} else {
textNodesWithPosition[index] = {
text: `${node.text}`,
pos,
};
}
} else {
index += 1;
}
});
textNodesWithPosition = textNodesWithPosition.filter(Boolean);
for (const element of textNodesWithPosition) {
const { text, pos } = element;
const matches = Array.from(text.matchAll(searchTerm)).filter(
([matchText]) => matchText.trim(),
);
for (const m of matches) {
if (m[0] === "") break;
if (m.index !== undefined) {
results.push({
from: pos + m.index,
to: pos + m.index + m[0].length,
});
}
}
}
// Shared find-all-occurrences primitive (also used by multi-cursor).
const results: Range[] = findOccurrences(doc, searchTerm);
for (let i = 0; i < results.length; i += 1) {
const r = results[i];
-3
View File
@@ -543,9 +543,6 @@ importers:
'@docmost/pdf-inspector':
specifier: 1.9.6
version: 1.9.6
'@docmost/prosemirror-markdown':
specifier: workspace:*
version: link:../../packages/prosemirror-markdown
'@fastify/cookie':
specifier: ^11.0.2
version: 11.0.2