refactor(#345 step 2): server markdown IMPORT via canonical parser + normalizer
Move every SERVER Markdown->ProseMirror path off the editor-ext markdown layer (`markdownToHtml`, a second marked-based parser) onto the canonical `@docmost/prosemirror-markdown` package, and add a foreign-markdown normalizer at the import boundary. Code: - `ImportService.processMarkdown` (single `.md` upload) now parses `markdownToProseMirror(normalizeForeignMarkdown(md))` directly — no HTML hop. - `PageService.parseProsemirrorContent` markdown case (page create/update with `format: 'markdown'`) same. - `FileImportTaskService` (zip import) parses markdown with the package, then serializes to HTML (`jsonToHtml`) so the SHARED HTML attachment / internal-link pipeline (processAttachments + formatImportHtml + processHTML) keeps handling `.md` and `.html` imports uniformly. The markdown PARSE — the drift source — no longer goes through editor-ext; the PM->HTML->PM hop that follows is lossless plumbing for attachment resolution, not a second parse. - `canonicalizeFootnotes` stays as an idempotent #228 safety net for the HTML path (a no-op on the already-canonical markdown output). Normalizer (`integrations/import/utils/foreign-markdown.ts`): a TEXT pre-pass, NOT a parser fork. The strict canonical parser does not accept GFM `[^id]` reference footnotes (and would misread `[^id]: def` as a CommonMark link-ref definition, silently corrupting the ref into a bogus link), so the normalizer rewrites reference footnotes into canonical inline `^[def]` before parsing. Callout surfaces (`:::type` and `> [!type]`) are intentionally NOT touched — the canonical parser already accepts BOTH natively, so normalizing them would be redundant and risk degrading its nesting/code-fence-aware handling. Fixtures-first: foreign-markdown.spec pins the normalizer and the end-to-end acceptance (no literal `[^id]`/`:::` leaks; re-export is canonical). The two footnote-canonicalize specs are updated to the canonical output — the parser assigns fresh `fn-*` ids, so they now assert by definition BODY order (still reference-ordered, deduped, orphan-free). FINAL CHECK: `grep -rn "htmlToMarkdown\|markdownToHtml" apps/server/src` (non -test) is now empty — both editor-ext markdown-layer functions are gone from the server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,9 @@
|
||||
import { Inject, Injectable, Logger } from '@nestjs/common';
|
||||
import * as path from 'path';
|
||||
import { jsonToText } from '../../../collaboration/collaboration.util';
|
||||
import {
|
||||
jsonToHtml,
|
||||
jsonToText,
|
||||
} from '../../../collaboration/collaboration.util';
|
||||
import { InjectKysely } from 'nestjs-kysely';
|
||||
import { KyselyDB } from '@docmost/db/types/kysely.types';
|
||||
import {
|
||||
@@ -18,9 +21,11 @@ import { generateSlugId } from '../../../common/helpers';
|
||||
import { v7 } from 'uuid';
|
||||
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
|
||||
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
|
||||
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
||||
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
|
||||
import { formatImportHtml } from '../utils/import-formatter';
|
||||
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
|
||||
import {
|
||||
buildAttachmentCandidates,
|
||||
collectMarkdownAndHtmlFiles,
|
||||
@@ -461,7 +466,18 @@ export class FileImportTaskService {
|
||||
content = await fs.readFile(absPath, 'utf-8');
|
||||
|
||||
if (page.fileExtension.toLowerCase() === '.md') {
|
||||
content = await markdownToHtml(content);
|
||||
// Parse markdown with the single canonical converter
|
||||
// (`@docmost/prosemirror-markdown`), after normalizing foreign
|
||||
// reference footnotes, then serialize to HTML so the shared HTML
|
||||
// pipeline below (processAttachments + formatImportHtml +
|
||||
// processHTML) keeps handling `.md` and `.html` imports
|
||||
// uniformly. The markdown PARSE no longer goes through the
|
||||
// editor-ext markdown layer (issue #345) — the drift source is
|
||||
// gone. The PM -> HTML -> PM hop that follows is lossless
|
||||
// plumbing for attachment/link resolution, NOT a second parse.
|
||||
content = jsonToHtml(
|
||||
await markdownToProseMirror(normalizeForeignMarkdown(content)),
|
||||
);
|
||||
}
|
||||
} catch (err: any) {
|
||||
if (err?.code === 'ENOENT') {
|
||||
@@ -500,10 +516,12 @@ export class FileImportTaskService {
|
||||
this.importService.extractTitleAndRemoveHeading(pmState);
|
||||
|
||||
// Canonicalize footnote topology on this non-editor write path
|
||||
// (markdownToHtml/processHTML never runs footnoteSyncPlugin), so a
|
||||
// zip-imported page's footnotes are reference-ordered, deduped, and
|
||||
// (the HTML pipeline's processHTML never runs footnoteSyncPlugin), so
|
||||
// a zip-imported page's footnotes are reference-ordered, deduped, and
|
||||
// orphan-free like the editor's invariant (issue #228). Pure +
|
||||
// idempotent + shape-safe; a footnote-free doc is unchanged.
|
||||
// idempotent + shape-safe; a footnote-free doc is unchanged. (For a
|
||||
// `.md` file the package parser already yields canonical footnotes,
|
||||
// so this is a no-op there.)
|
||||
// (Future consolidation, architecture B: like import.service, this
|
||||
// path persists directly rather than via PageService — a shared
|
||||
// "prepare JSON for persist" helper would centralize this call.)
|
||||
|
||||
Reference in New Issue
Block a user