vvzvlad/gitmost

refactor(#345): серверный экспорт/импорт markdown через @docmost/prosemirror-markdown #369

Open

agent_coder wants to merge 7 commits from refactor/345-server-converter into develop

Author	SHA1	Message	Date
agent_coder	c5bff2d84a	fix(#345 ): normalize CRLF before front-matter strip (review round 3) F9 [WARNING] The line-anchored front-matter regex from round 2 requires a bare LF after the opening `---`, so a Windows/CRLF foreign file (`---\r\n...`) slips past the strip and leaks its front-matter into the body (where `title: Foo` renders as a setext heading that title extraction hijacks). The canonical parser whose regex shape this copied (page-file.ts) normalizes CRLF -> LF BEFORE its FRONTMATTER_RE; the import path copied the regex but missed the normalization. normalizeForeignMarkdown now replaces CRLF with LF first (which also makes convertReferenceFootnotes' split('\n') consistent). Adds a CRLF fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 05:38:07 +03:00
agent_coder	80fc30633b	fix(#345 ): replace id-alternation regex with a fixed generic scanner + line-anchor frontmatter (review round 2) F7 [CRITICAL] The round-1 F2(a) fix built ONE alternation regex over all definition ids (`(id1\|id2\|...)`). On prefix-chain ids (a, aa, aaa, ...) V8's regex compiler blows its stack with a fatal, UNCATCHABLE 'RegExpCompiler Allocation failed' that kills the whole process — strictly worse than the original per-def thread-hang, and its match cost was still O(text x defs). Replaced with a single FIXED generic scanner `/\[\^([^\]]+)\]/g` plus a map lookup in the replacer: genuinely O(total text), no per-document regex compilation, cannot blow up. Output is identical (only real def ids are inlined). F8 [WARNING] The frontmatter strip regex was not line-anchored: it closed on the FIRST `---` anywhere, so a value containing a triple-dash (e.g. 'title: Q1 --- Q2') truncated the frontmatter and leaked the rest into the body. Replaced with the line-anchored shape the canonical parser already uses (page-file.ts): open on `---\n`, close on a `\n---` line. Adds tests: 4000 prefix-chain ids do not crash and stay fast; a frontmatter value containing '---' is stripped whole. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 05:18:44 +03:00
agent_coder	e17d5bc060	fix(#345 ): restore prom-client, harden normalizer against ReDoS, strip frontmatter (review round 1) Addresses the round-1 review of #369: F1 [CRITICAL] Restore prom-client. The prior commit removed it as a 'stray dep', but metrics.registry.ts imports it unconditionally at startup (main.ts boot), so a clean frozen install had no prom-client -> server tsc TS2307 + boot crash. It was surviving only via hoisting from a warm store. Restored to apps/server dependencies + regenerated the lock (prom-client/tdigest/bintrees return), keeping the @docmost/prosemirror-markdown dep. Verified: clean frozen install -> require.resolve('prom-client') ok, server tsc EXIT 0. F2 [HIGH] Two quadratic ReDoS vectors in foreign-markdown.ts on untrusted import (runs synchronously on the request thread, 30MB cap): (a) pass-2 was O(lines x defs) — a per-def RegExp rebuilt and run over every line. Replaced with ONE precompiled alternation regex over all def ids, built once per document, with an id->body lookup in the replacer: O(text). (b) the inline-code split alternation backtracks quadratically on a long UNCLOSED backtick run. Lines over 8KB now skip the split (left untouched) — a real footnote line is never that long. F3 [WARNING] Restore the leading YAML front-matter strip that the retired markdownToHtml layer did. Without it, Obsidian/Hugo/Jekyll/git-sync files leak their front-matter into the body (and 'title:' renders as a setext heading that title extraction can hijack). F4 [WARNING] Extend the zip-import spec with an image (width+align) + callout fidelity assertion through the PM->HTML->PM hop (the one hop the package suite does not cover). F5/F6 Update AGENTS.md (apps/server is now a prosemirror-markdown consumer) and make the server pretest build prosemirror-markdown too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 04:54:07 +03:00
agent_coder	2c2d60a5dc	fix(#345 ): protect inline-code refs and escape footnote-body brackets The foreign-markdown import normalizer rewrote GFM reference footnotes (`[^id]` + `[^id]: def`) into canonical inline `^[def]` footnotes, but two edge cases corrupted content: 1. A `[^id]` inside an inline-code span (backticks) was rewritten like prose text — only fenced code blocks were protected. Now the rewrite pass splits each line on inline-code spans and only touches the text outside them. 2. An unbalanced `]` in a definition body truncated the resulting `^[...]` footnote at the canonical tokenizer, leaking the tail as literal text. The body's square brackets are now backslash-escaped before wrapping. Adds golden cases for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:39:01 +03:00
agent_coder	1417209915	fix(#345 ): drop stray prom-client dep + add prosemirror-markdown to the lock The step-1 package.json declared the new @docmost/prosemirror-markdown workspace dep but the lock was not regenerated (CI frozen install would fail), and it also added a stray prom-client dep (a coder env-workaround for a pre-existing hoisted import, unrelated to #345 — removed). Regenerated the lock with only the prosemirror-markdown dep; faithful frozen install now passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:27:01 +03:00
agent_coder	f555fc87da	refactor(#345 step 2): server markdown IMPORT via canonical parser + normalizer Move every SERVER Markdown->ProseMirror path off the editor-ext markdown layer (`markdownToHtml`, a second marked-based parser) onto the canonical `@docmost/prosemirror-markdown` package, and add a foreign-markdown normalizer at the import boundary. Code: - `ImportService.processMarkdown` (single `.md` upload) now parses `markdownToProseMirror(normalizeForeignMarkdown(md))` directly — no HTML hop. - `PageService.parseProsemirrorContent` markdown case (page create/update with `format: 'markdown'`) same. - `FileImportTaskService` (zip import) parses markdown with the package, then serializes to HTML (`jsonToHtml`) so the SHARED HTML attachment / internal-link pipeline (processAttachments + formatImportHtml + processHTML) keeps handling `.md` and `.html` imports uniformly. The markdown PARSE — the drift source — no longer goes through editor-ext; the PM->HTML->PM hop that follows is lossless plumbing for attachment resolution, not a second parse. - `canonicalizeFootnotes` stays as an idempotent #228 safety net for the HTML path (a no-op on the already-canonical markdown output). Normalizer (`integrations/import/utils/foreign-markdown.ts`): a TEXT pre-pass, NOT a parser fork. The strict canonical parser does not accept GFM `[^id]` reference footnotes (and would misread `[^id]: def` as a CommonMark link-ref definition, silently corrupting the ref into a bogus link), so the normalizer rewrites reference footnotes into canonical inline `^[def]` before parsing. Callout surfaces (`:::type` and `> [!type]`) are intentionally NOT touched — the canonical parser already accepts BOTH natively, so normalizing them would be redundant and risk degrading its nesting/code-fence-aware handling. Fixtures-first: foreign-markdown.spec pins the normalizer and the end-to-end acceptance (no literal `[^id]`/`:::` leaks; re-export is canonical). The two footnote-canonicalize specs are updated to the canonical output — the parser assigns fresh `fn-*` ids, so they now assert by definition BODY order (still reference-ordered, deduped, orphan-free). FINAL CHECK: `grep -rn "htmlToMarkdown\\|markdownToHtml" apps/server/src` (non -test) is now empty — both editor-ext markdown-layer functions are gone from the server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:21:07 +03:00
agent_coder	d6d1195abd	refactor(#345 step 1): server markdown EXPORT via canonical converter Move every SERVER ProseMirror->Markdown path off the editor-ext markdown layer (`htmlToMarkdown`, a second turndown-based converter) onto the canonical `@docmost/prosemirror-markdown` package. - `ExportService.exportPage` (page/space markdown export) and `collaboration.util.jsonToMarkdown` (used by page.controller's markdown responses and the AI public-share chat tool) now serialize DIRECTLY from ProseMirror JSON via `convertProseMirrorToMarkdown` — no HTML intermediate, no `<colgroup>` scrub (the converter emits GFM tables directly). This is the SAME serializer the git-sync vault writer feeds, so an exported page BODY is byte-identical to its vault representation: no more export-md vs vault-md drift. The HTML export path is unchanged (still `jsonToHtml`). Emitted markdown moves to the canonical forms: callouts `> [!type]` (not `:::type`), inline footnotes `^[…]` (not `[^id]`), lossless images `![alt](src) <!--img {…}-->` (editor-ext dropped width/height/align). Fixtures-first: export-markdown.spec asserts those canonical forms and the export==vault-by-construction equality (both call the package converter). The one deliberate export/vault delta — export prepends the page title as an H1 while the vault carries it in frontmatter — is pinned by a test. Test infra: declare the `@docmost/prosemirror-markdown` workspace dep; teach jest to load its ESM build (babel-jest) and stub `@tiptap/react` (server code imports editor-ext, whose node views reference React renderers only used in a live browser editor — never on the server). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 03:20:25 +03:00