Closes the architecture item from the #119 review: drop the "vendored from
docmost-sync" framing and the CJS↔ESM `Function('import()')` bridge so the engine
is a normal first-class gitmost package.
Part 1 — vendoring markers removed (prose only, zero behavior change): reworded
"VENDORED into gitmost" / "vendored from docmost-sync" / "Engine LOGIC is
byte-identical" / "it's a port" comments across the engine. Behavior-bearing
strings are untouched: BOT_AUTHOR_NAME/EMAIL and the `Docmost-Sync-Source:`
provenance trailers (changing them would break git authorship + the loop-guard).
Part 2 — the package is now ESM (matching the sibling @docmost/mcp): `type: module`,
tsconfig Node16, `.js` extensions on relative imports, and a static
`import { marked }` replacing the `new Function('return import(...)')` /
`loadMarked` hack — the bridge is GONE from the package. The CommonJS NestJS
server loads the now-ESM engine via a new `git-sync.loader.ts` that mirrors the
existing `docmost-client.loader.ts` mcp loader exactly (Function-indirected
dynamic import + cached promise + retry-on-reject). The 4 server consumers
(orchestrator/datasource/vault-registry/git-http-backend) call `await loadGitSync()`
for value exports; types stay `import type` (erased). The converter-gate spec —
which needs the real converter — loads the package's TS source via a jest
moduleNameMapper + isolatedModules (documented in that spec); the other git-sync
specs mock the loader.
Verified: engine builds pure ESM (no Function/require leftover), vitest 614,
editor-ext build, server + client tsc, full server jest 1397/0. Live stand
smoke-test: server starts clean on the ESM engine (no ERR_REQUIRE_ESM), a real
sync cycle runs through the loader, and the basic e2e suite is 12/12 (clone via
git-http-backend, push, pull, delete, 3-way merge — all through the new loader).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-review of phase 3 caught a data-corruption regression: nativeMeta always
supplies the run's spaceId, so the planner's 'create-without-spaceId' skip — which
had doubled as the only filter for non-page files — went dead. An ADDED
.obsidian/*.json, attachment, or dotfile (committed to the vault, no .gitignore)
would then be classified as a CREATE: a junk Docmost page, plus a gitmost_id
frontmatter written INTO the file, corrupting it.
Fix: isPageFile(path) — a .md file with NO dot-segment anywhere — and filter the
diff to page files at the very top of computePushActions, BEFORE any
classification, so non-page A/M/D/R are ignored (design §Адопция). 2 unit tests
pin it (.obsidian/json, attachment, dotfile, dot-segment, .md dotfile all ignored;
real pages still created). 614 engine tests green.
Also: refreshed stale docmost:meta comments to gitmost_id (review SUGGESTION), and
documented the deferred adoption frontmatter-preservation gap (review WARNING) in
page-file.ts + the design doc (do NOT roll native onto a real vault with Obsidian
properties until phase 4 round-trips them).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PUSH now consumes the native-Obsidian format end-to-end:
- identity from the gitmost_id frontmatter (parsePageFile), not docmost:meta;
- title from the FILENAME, parentPageId from the enclosing folder's folder-note
(parentFolderFile is now FOLDER-NOTE aware: a child's parent is dir/dir.md, and
a folder-note's own parent is one level up), spaceId from the run (every vault
file belongs to the vault's space);
- CREATE derives title/parent/space from path + run and writes the assigned
pageId back as gitmost_id frontmatter (serializePageFile);
- UPDATE pushes the STRIPPED body (current + 3-way-merge base), so the frontmatter
never leaks into Docmost content; the loop-guard hashes the body.
The PURE delete-sensitive classifier (computePushActions/classifyRenameMoves) is
UNCHANGED — only the injected IO resolvers (metaAt, parent, create write-back)
switched source. nativeMeta always carries the run spaceId, so the legacy
'create-without-spaceId' skip no longer fires through runPush.
Tests rewritten to native fixtures + folder-note parent paths; the noop case is
now a child under a renamed parent folder (filename=title, so a path-only-noop
needs an ancestor rename). parentFolderFile tests cover leaf/folder-note/nested/
dotted. 612 engine tests green; engine rebuilt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PULL now serializes each page as the native-Obsidian format (serializePageFile:
a minimal gitmost_id frontmatter + the fixpoint markdown body) instead of the
heavy docmost:meta envelope. title/parent/space are derived (filename / folder /
repo), so only the pageId is persisted. readExisting recovers identity from the
gitmost_id frontmatter (parsePageFile) instead of docmost:meta.
Extracted stabilizePageBody() (the export->import->export fixpoint, no meta) so
the native writer and the legacy serializer share the same deterministic body —
re-pulls of an unchanged page stay byte-identical (loop-guard).
Tests: read-existing fixtures rewritten to gitmost_id; apply-pull asserts the
written text is native frontmatter and carries NO docmost:meta (regression
guard). 611 engine tests green.
NOTE: PUSH still reads docmost:meta — the end-to-end cycle is intentionally NOT
runnable until phase 3 (PUSH reads frontmatter + derives title/parent from path)
lands; no vault is wiped/deployed until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Native-Obsidian structure: a page WITH children now lives at its folder-note
<name>/<name>.md (LostPaul Folder Notes convention) with its children alongside;
a leaf stays <name>.md. Folder-notes claim their canonical path before a
same-named child, so the child (a leaf) is the one disambiguated, never the
folder-note — a folder X/ always contains its own note X.
Format-agnostic and safe in isolation: only the destination PATH changes, the
file content/serialization is untouched, so an existing parent relocates via the
move-by-id path (no delete). The frontmatter format flip (pull+push) is next.
6 new layout unit tests (leaf / parent / nested / child-named-as-parent /
twin-parents / childless). 611 engine tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per owner: test data, no migration. parsePageFile no longer reads the old
docmost:meta block — a file without a gitmost_id frontmatter is simply un-tracked
(adopt). Vaults are a cache: rm -rf on the transition, rebuilt native from
Docmost. Simplifies the format work (no fallback). Doc updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pivot the thin-meta design to "the vault IS a native Obsidian vault": clean
markdown + a minimal YAML frontmatter `gitmost_id:` (the durable pageId, travels
with the file so identity survives any move); folders mirror the page tree with
the parent's body as a folder-note `<Folder>/<Folder>.md` (LostPaul Folder Notes
convention); links as `[[wikilinks]]` (basename-resolved → reparent never breaks a
link, only retitle does); collisions disambiguated Obsidian-style; `.obsidian/`
and non-page files left untouched (no .gitignore). Verified the conventions
against the Obsidian/Folder-Notes docs.
Replaces the abandoned `.gitmost/index.json` sidecar (path-keyed → fragile to
git-undetected renames; the in-file id is self-sufficient): removes vault-index.ts.
Adds lib/page-file.ts — parsePageFile/serializePageFile (frontmatter id + clean
body) with a LEGACY `docmost:meta` fallback for migration. 6 unit tests; engine
suite green. Not yet wired into pull/push — no behavior change. Design doc
rewritten to the native-Obsidian format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure read/write/lookup for the vault sidecar index that will hold page identity
(pageId) + collision token (slugId) keyed by file path, so the .md files can be
clean markdown. parseVaultIndex is tolerant (missing/garbage/bad entries degrade
to empty/skipped — never crashes a cycle); serializeVaultIndex is deterministic
(sorted keys -> stable diffs, no churn). Lookups (pageIdAt, pathForPageId reverse,
trackedPageIds) + mutations (set/remove/move). NOT wired into pull/push yet — no
behavior change. 5 unit tests; engine suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to 4376c5a6, found by a real BROWSER e2e (the flow the in-diff fix
missed). When the layout reshuffle's two halves land in SEPARATE sync cycles, the
later cycle's diff has only the DELETE of the old path — the matching add was
already pushed — so in-diff D+A coalescing can't see it, and the live page was
still trashed.
Robust fix on the identity invariant the reviewer (and the user) called out: a
page EXISTS iff its pageId is in the vault, regardless of filename. runPush now
collects the pageIds present at ANY path in the current `main` tree and passes
them to computePushActions; a deleted file whose pageId is still tracked
elsewhere is a MOVE, never a deletion. (Built only when the diff has deletes.)
Adds apps/server/test/git-sync-browser-e2e.cjs — a Playwright test that drives the
REAL Docmost web UI: log in, create several untitled pages, type a title, sync,
assert NOTHING is trashed. Reproduced the data loss before this fix; 5/5 green and
stable after. Engine suite 600 green (+2 computePushActions cases:
pageId-still-present -> skip; pageId-gone -> real delete).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CRITICAL data-loss bug: creating pages in Docmost (which start UNTITLED) and then
typing a title could soft-delete OTHER pages. Untitled pages all serialize to the
`_` fallback filename; the layout disambiguates them (`_.md`, `_ ~slug.md`).
Retitling one frees the bare `_` and another untitled page's file relocates into
it. git's rename detection (`-M`) can't see the move (the tiny meta-only files are
too dissimilar), so `git diff` reports it as DELETE(old) + ADD/MODIFY(new). The
push took the DELETE literally and trashed a live page.
Root cause is that the push trusted git's path-level rename heuristic for page
IDENTITY. Identity is the pageId. Fix: before emitting any delete, coalesce by
pageId — a pageId that is BOTH deleted (pre-image) AND present on the surviving
side (current meta of an ADD or a MODIFY, since a relocation into an occupied path
shows as M) is one page that MOVED, classified as a rename/move and NEVER a delete.
Reproduced + verified on a live stand: 4 untitled pages + retitle one trashed a
different page before; after the fix, retitling one (and stress-retitling all)
trashes nothing. Engine suite 598 green; 3 new computePushActions cases (ghost
D+A move -> rename; real delete still deletes; unrelated D+A stay delete+update).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconcile choreography (ensureRepo -> merge-check -> ensureBranch ->
checkout('docmost') -> pull -> push) was hand-rolled in the app orchestrator's
driveCycle, duplicating an order the vendored engine owns and could drift from on
upgrade — the failure mode is data clobber. Lift it into @docmost/git-sync as a
single entry point, `runCycle(deps)`. The orchestrator now calls runCycle and
keeps only the lock (its caller) and the gitmost-specific delete-cap POLICY,
injected as the `resolveApplyClient` hook (the engine does the dry-run, hands the
hook the planned delete count — Infinity if planning failed — and uses whatever
client it returns for the apply). driveCycle drops from ~150 lines to ~30.
Tests:
- engine test/cycle.test.ts: composition (merge-in-progress short-circuit;
ensureRepo->ensureBranch->checkout staging order before the pull; the cap hook
is consulted with the planned count; no dry-run when no hook).
- engine test/cycle-roundtrip.test.ts: runCycle against a REAL VaultGit in a temp
repo with a faked Docmost client — a git-originated CREATE flows pull->push and
the assigned pageId is written back; an unresolved merge short-circuits before
any client call.
- orchestrator spec rewired to mock runCycle and assert the wiring + the
resolveApplyClient cap policy (the engine-internal cycle-order/merge tests moved
to the engine).
Validated end to end on a live stand (real Postgres/Redis + server): a git clone
-> edit -> push over the /git remote round-trips the change into the Docmost page
through the refactored cycle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The implementation spec docs/git-sync-plan.md was removed as completed, but ~44
code comments still cited it as "plan §N". Strip those citations (comments only),
keeping each comment grammatical. The vendored engine's own "SPEC §N" references
point at a different, still-present spec and are left untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same hygiene fix as git-sync (review #2), applied to packages/mcp which had the
identical pre-existing problem: committed build/ (20 files) + node_modules (28,
pnpm symlinks with a baked /home/claude store path).
- git rm --cached packages/mcp/{build,node_modules}.
- .gitignore: add packages/mcp/build/ (packages/*/node_modules/ already covers it).
- Build where consumed: apps/server `pretest` and the CI Test workflow now build
@docmost/mcp too. The Dockerfile builder already runs `pnpm build` (nx builds
mcp) and already COPYs packages/mcp/build into the runtime image.
Verified: wiped build/, rebuilt via `pnpm --filter @docmost/mcp build`; the mcp
server suites (96 tests) pass against the freshly-built, non-committed output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Upgrades the 2-way body merge to a real diff3 three-way merge (review #5), so a
block ONLY the human changed is KEPT when git changed a DIFFERENT block — the
2-way merge would revert it to git's stale version.
Engine: the push update loop reads the last-synced pre-image
(`git.showFileAtRef(refs/docmost/last-pushed, path)`) and passes it as the
optional `baseMarkdown` to `client.importPageMarkdown` (the common ancestor).
Server: gitmost-datasource converts base+incoming, and writeBody runs a block-
level diff3 (new three-way-merge.ts `diff3Plan`): live-only change -> keep live,
git-only change -> take git, both-changed -> git wins (conflict policy), inserts/
deletes from either side preserved. Without a base (createPage) it falls back to
the 2-way merge. Crash-safety unchanged (docs built before the connection opens).
Tests: three-way-merge.spec.ts (14 — every diff3 case incl. the cross-block
preservation and conflict policy), yjs-body-merge 3-way (real Y.Docs: human's
block instance preserved while git's block is applied), plus an engine test that
the base is forwarded from showFileAtRef. Existing push assertions updated for the
new base arg. git-sync 589 pass; server merge/datasource/gate 62 pass; typecheck
clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review finding #2: packages/git-sync/build/ (the COMPILED engine) and the
package's node_modules/ were committed. Prod executed the committed build/ while
CI/tests ran src/ and never rebuilt it — so a fix in src/ could pass tests while
stale compiled code shipped (a silent src/prod skew). The committed node_modules
were pnpm symlinks with a baked machine-local store path (/home/claude/...),
useless and misleading for everyone else.
- git rm --cached packages/git-sync/{build,node_modules} (42 + 31 files).
- .gitignore: ignore packages/*/node_modules/ and packages/git-sync/build/.
- Build the package where it is actually consumed: apps/server `pretest` now
builds @docmost/git-sync (its suite imports the built build/index.js), and the
CI Test workflow gains an explicit "Build git-sync" step. The Dockerfile builder
already runs `pnpm build` (nx builds the package) and now COPYs the fresh build/.
Verified: wiped build/, rebuilt via `pnpm --filter @docmost/git-sync build`, then
the server converter gate (26/26, imports the rebuilt package) and the git-sync
suite (588 passed) both pass against the freshly-built, non-committed output.
NOTE: packages/mcp/ has the same committed-build/node_modules pattern (pre-existing,
out of this PR's scope) and should get the same treatment in a follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Coder↔reviewer design loop (9 rounds, reviewer verdict: exhaustive) produced
92 specs; implemented +123 tests (465 -> 588 passing). The new round-trip
coverage exposed three genuine data-loss bugs in the Markdown<->ProseMirror
converter, all now FIXED (round-trip is lossless for these):
1. pageBreak was lost on export (no converter case -> rendered to "" and the
node vanished). Now emits <div data-type="pageBreak"></div>, which the schema
parses back -> round-trips.
2. A block image between blocks left an empty <p> artifact after import-hoisting,
producing a phantom blank-gap diff on every sync. markdownToProseMirror now
strips content-less paragraphs after generateJSON — with a schema-validity
guard that keeps the obligatory single empty paragraph in `content: "block+"`
containers (tableCell/tableHeader/blockquote/column/callout/doc), so empty
cells/quotes never become an invalid `content: []`.
3. The `code` mark combined with another mark was not byte-stable (emitted nested
HTML that the schema's `code` `excludes:"_"` collapsed on import). The
converter now emits code-only when `code` co-occurs, matching the editor.
New coverage spans media/diagram/details/columns/math/mention attribute
round-trips, converter emission branches, git error paths, and engine decision
branches. A dedicated test pins the empty-container schema validity (the review
catch on the bug-2 fix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make @docmost/git-sync natively consumable by the CommonJS server (and jest):
build to CommonJS (tsconfig module CommonJS, drop type:module, strip .js from
relative imports), and lazy-load the only ESM-only dep (marked) via the dynamic
Function('import()') trick (mirrors docmost-client.loader.ts) with a require()
fallback so vitest's evaluator works too. git-sync tests stay green (314 pass,
3 expected fail).
Add the §13.1 idempotency gate (apps/server .../git-sync-converter-gate.spec.ts):
13 editor-ext docs (paragraphs/headings, marks, links, bullet/ordered/task lists,
blockquote, callouts, code block, hr, table, nested mix) round-trip
content(editor-ext) -> convertProseMirrorToMarkdown -> markdownToProseMirror ->
TiptapTransformer.toYdoc/fromYdoc(tiptapExtensions) -> canonicalize and assert
docsCanonicallyEqual. All green => the vendored converter's docmost-schema is
schema-compatible with editor-ext (no node/mark/attr loss), which the plan §13.1
requires before Phase B. The one intrinsic markdown-image lossiness (width/height
/align can't ride plain ) is isolated in a KNOWN DIVERGENCE block, not
hidden. Server tsc clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First step of docs/git-sync-plan.md. New workspace package @docmost/git-sync
vendoring the PURE parts from docmost-sync (HEAD b03eb35):
- lib: markdown-converter, markdown-document, canonicalize, docmost-schema,
node-ops, diff, and an extracted markdown-to-prosemirror (only the pure
marked->HTML->generateJSON path from upstream collaboration.ts; no websocket).
- engine (pure, no IO): reconcile, layout, sanitize, stabilize, loop-guard.
Ported the upstream pure-module + round-trip corpus tests (vitest): 314 pass,
3 expected upstream known-limitation fails. tsc clean. No server wiring yet.
docmost-schema inlines getStyleProperty (as packages/mcp does — @tiptap/core
3.20.4 doesn't export it). IO engine (pull/push/git/settings) deferred to later
Phase A/B steps; the editor-ext idempotency gate (plan §13.1) is the next step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the image fix in place, the mcp e2e ran through every section and
failed only at the last one (comments): create_comment was hardened to
require an inline "selection" (exact text to anchor on) for a top-level
comment, but the test created one without a selection ("an inline
'selection' ... is required for a top-level comment").
Pass an inline selection ("Добавленный абзац.", a plain paragraph
re-imported in section 5 and still present at the comments stage). The
reply is unchanged: it carries a parentCommentId, so it is a reply and
needs no selection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the first e2e fix: with nanoid/editRes.edits resolved, the
suites failed one layer deeper. Both layers were never green since the
e2e jobs were added (non-blocking in CI), so the failures had stacked up.
server e2e (jest-e2e.json) — align module resolution/transform with the
working unit/integration jest configs so AppModule's full import graph
loads:
- moduleFileExtensions: add "tsx" (React-Email .tsx templates are pulled
in via the auth controller chain).
- transform: ^.+\.(t|j)s$ -> ^.+\.(t|j)sx?$ so .tsx is transformed.
- moduleNameMapper: add ^src/(.*)$ -> <rootDir>/../src/$1 (code imports
via the absolute 'src/...' alias). Verified locally: the module graph
now fully resolves (only env vars, supplied by CI, remain).
mcp e2e (test-e2e.mjs) — insert_image/replace_image accept only http(s)
URLs the server fetches; the test passed local file paths and died with
"Invalid image URL". Serve the PNG bytes over a throwaway 127.0.0.1 HTTP
server (the Docmost server runs on the same CI host) and pass URLs. The
featPng negative test is untouched: replaceImage checks the attachmentId
and throws before fetching, so its local path is never validated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two unrelated CI failures on the 0.94.0 release PR:
- server e2e: jest-e2e.json lacked transformIgnorePatterns, so the
ESM-only nanoid@5 package was loaded as CommonJS and crashed with
"Cannot use import statement outside a module". Add the same
node_modules whitelist already present in the unit and integration
jest configs (nanoid|uuid|image-dimensions|marked|happy-dom|lib0).
- mcp e2e: test-e2e.mjs read editRes.edits, but editPageText() returns
the per-edit results under `applied` (not `edits`), so editRes.edits
was undefined and .every() threw. Read editRes.applied instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A colliding transclusionSource id is deliberately NOT reassigned (its id is a
cross-reference key), while a missing id is still filled. Add coverage for both:
two sources sharing an id keep it (red if the NO_REASSIGN guard is removed), and
a source with no id gets a fresh one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
editor-pm-7: addUniqueIdsToDoc only FILLED missing ids and never deduplicated
existing ones, so a copy/paste or bulk-JSON duplicate that kept its attrs.id
produced two nodes sharing an id. MCP addressed edits (patch_node /
delete_node "before/after id") then hit the wrong node or both.
Walk the configured-type nodes in document order: the first occurrence of an
id keeps it (stable anchor), later duplicates are reassigned a fresh id.
transclusionSource ids are cross-reference keys (references resolve a source by
this id), so they are only filled-when-missing, never reassigned, to avoid
orphaning their references.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Approve-with-comments auto-review (8 axes); no blockers. Closes the two flagged
test gaps; the two forward-looking dedup suggestions (reconcileHasChildren helper;
unifying reconcileChildren/mergeRootTrees) are non-blocking architecture notes and
left for a follow-up (as with #186's forward-looking point).
1. Ambiguous-id refusal end-to-end (#159): the patch_node/delete_node guard
`if (replaced/deleted !== 1) return null` was only covered in pieces — the
replaceNodeById/deleteNodeById counts and assertUnambiguousMatch in isolation —
so loosening the guard would not have failed a test. New mock test stands up a
REAL Hocuspocus collab server seeded (via buildYDoc, same docmost extensions)
with a two-blocks-one-id document and drives the real client methods: both must
reject with /ambiguous/ AND never write to collab. Tracked via Hocuspocus
onChange (fires synchronously per update, unlike the debounced onStoreDocument)
so a clobbering write is actually observed — verified the test FAILS when the
guard is loosened to `< 1`.
2. scrollToReference zero-match bail: the branch "non-empty id but querySelectorAll
returns 0 -> matches[index] ?? matches[0] is undefined -> return false" (the real
desync: definition present, inline ref removed from the DOM) was uncovered. Added
a footnote.test.ts case: a definition for 'ghost' with no rendered ref -> false,
no scroll.
Verified: 313 mcp tests + 24 editor-ext footnote tests; prettier clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The re-review's blocking/structural points (lease leak, dup-id guard test,
body-before-title test, CHANGELOG, pg18, shared jsonb decoder) were already
addressed in commit 24264ef; this adds the 4 genuinely-new coverage requests:
- pt 6: `scrollToReference(id, index?)` exercised against a live editor DOM —
selects the index-th `sup[data-footnote-ref][data-id]` occurrence, falls back
to the first for out-of-range, returns false for an empty id (scrollIntoView
stubbed). (#168)
- pt 7: export `backlinkLabel` and pin the base-26 carry boundary
(25->z, 26->aa, 27->ab, 51->az, 52->ba). (#168)
- pt 8: integration fail-open — a PRESENT-but-corrupt tool_allowlist (jsonb
string scalar holding non-array JSON) reads back as null ("no restriction"),
covering normalizeRow's degrade branch. (#159 #172/#173)
- pt 9: getFootnoteRefCount cache invalidation — adding a `[^a]` reference bumps
the cached count 2 -> 3. (#168)
Verified: editor-ext footnote 23; client structure 7 + tsc; server int 8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8-point multi-aspect review of the batch PR; security/regressions were clean.
1. Lease leak: the #180 reorder moved `toolsFor` (which leases external MCP
clients, refCount+1) ahead of buildSystemPrompt + forUser, but the only
release (closeExternalClients) was bound to the streamText callbacks. A throw
in between leaked the lease (refCount stuck, undici sockets held until
restart). Define closeExternalClients right after the lease and wrap
buildSystemPrompt+forUser in try/catch that closes-then-rethrows.
2. Cover the patch_node/delete_node dup-id refusal (#159#6): extract the guard
into a pure `assertUnambiguousMatch` (node-ops) and unit-test 0/1/>1.
3. Regress the body-before-title order (#159#10): mock-HTTP test (collab fails
fast against a server with no WS upgrade) asserts /pages/update (title) is
NEVER posted when the body write fails — for updatePage AND updatePageJson.
4. CHANGELOG [Unreleased]: #180, #168 (Added); #163 (Fixed).
5. Add the missing en-US i18n keys (Back to references / {{label}}).
6. Drop the duplicate content/empty/blank cases in ai-chat.prompt.spec.ts (they
repeat the buildMcpToolingBlock unit tests); keep only sandwich placement +
both-safety-copies.
7. CI Postgres pg16 -> pg18 (match docker-compose).
8. jsonb decode seam: shared `parseJsonbValue(value, guard)` in database/utils.ts
holds the legacy double-encoding self-heal in one place; parseToolAllowlist /
parseModelConfig keep only a type-guard.
Verified: server build + 124 unit + 15 integration; mcp 311; prettier clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
updatePage (markdown) and updatePageJson wrote the title via REST FIRST, then
the body via collab. If the body write failed (e.g. a collab persist timeout),
the page was left with the NEW title over its OLD body — a split-brain the tool
reported as an error but never repaired (red-team finding #10).
Reorder both: write the body first, and only set the title after the body has
persisted. Now a body-write failure leaves the title untouched (no split-brain).
A title write failing after a successful body is rarer (REST is fast) and leaves
correct content under a stale title — the strictly lesser inconsistency — which
is the same trade-off the issue's "atomic, or roll back the title" intends,
without the fragility of a rollback write that could itself fail.
No unit test: both paths require a live collab provider and the suite has no
provider mock; the change is a pure reordering. All 306 mcp tests still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Docmost duplicates block ids on copy/paste, and copyPageContent writes the
source document verbatim with the same ids. `patchNode`/`deleteNode` address a
block by `attrs.id` via replaceNodeById/deleteNodeById, which act on EVERY node
sharing the id — so a single patch_node/delete_node could silently
replace/remove multiple unrelated blocks with no signal to the model
(red-team finding #6).
Guard both write paths: when more than one node matches the id, skip the write
entirely (the transform returns null -> no mutation) and throw a clear
"ambiguous id — N nodes share it" error so the model re-targets with a more
specific anchor. Only an unambiguous single match is written; the 0-match and
1-match behavior is unchanged.
The duplicate-count basis is covered by node-ops.test.mjs (replaceNodeById /
deleteNodeById report count===2 for a 2-duplicate doc). The end-to-end guard
is not unit-tested because patchNode/deleteNode require a live collab provider
and the test suite has no provider mock.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After #166 a repeated `[^a]` is one footnote (reuse): one number, one
definition, N forward links. But the definition's ↩ only returned to the
FIRST reference. Now a definition with N references shows ↩ a b c …, each
backlink scrolling to its own occurrence (Pandoc/Wikipedia convention); a
single-reference footnote keeps the plain ↩ unchanged.
- editor-ext: `computeFootnoteRefCounts(doc)` (id -> occurrence count) cached
alongside the number map in the numbering plugin state; `getFootnoteRefCount`
getter (O(1), no per-render doc walk). `scrollToReference(id, index?)` picks
the index-th `sup[data-footnote-ref][data-id]` occurrence (document order),
falling back to the first.
- client: FootnoteDefinitionView renders one lettered link (a, b, c, … aa …)
per occurrence when refCount > 1; the chrome stays after the contentDOM so
the #146 caret invariant holds. i18n keys (ru) added.
Tests: computeFootnoteRefCounts + getFootnoteRefCount (reuse counts, unknown
id => 0); structure test gains 3 cases (N lettered links render, click jumps
to the n-th occorrence, single ref => one ↩). NOTE: the visual layout of the
backlink row needs a real browser to verify (jsdom can't); the structural and
behavioral contract is covered headless.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`mutateLiveContentUnlocked` — the write path used by `replaceImage` — still
did the pre-#152 destructive write (delete the whole fragment + applyUpdate a
fresh Y.Doc), discarding every Yjs node id. y-prosemirror anchors the editor
selection to those ids, so an open editor's cursor snapped to the document
end on every image swap, exactly the #152 jump that the main write path no
longer causes.
Switch it to the same `applyDocToFragment(ydoc, newDoc)` structural diff
(updateYFragment) as the main path, so unchanged nodes keep their ids and the
live cursor stays put. It runs its own atomic transact, so the old explicit
transact/delete is gone; the now-unused docmostExtensions import is dropped.
Regression tests (cursor-stability suite): a sibling paragraph's
RelativePosition survives a top-level image src/attachmentId swap, and an
image nested in a callout, matching the shapes replaceImage produces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the merged #166/#169. Addresses the second review pass (comment
1227):
- footnoteWarnings plumbing: extract a single `footnoteWarningsField(markdown)`
helper (footnote-analyze) and use it at all three call sites (create_page,
update_page, import_page_markdown) so the field is attached identically.
- New unit test footnote-warnings-import.test.mjs pins the contract that was
uncovered: the field is present on problems / omitted on clean input, and the
IMPORT path analyzes the BODY after the docmost:meta / docmost:comments blocks
(a footnote-like token inside those JSON blocks must NOT warn; a real body
marker must). Tested via the same pure composition the importer uses
(footnoteWarningsField(parseDocmostMarkdown(full).body)) — no collab socket
needed; a regression that analyzed fullMarkdown or skipped the body split would
now go red.
- footnote.marked.ts: correct the stale module header — it claimed "only
definitions that have a matching reference are emitted", which was never true
(orphan defs are emitted; the editor sync plugin reconciles). Now describes
first-wins + reuse + sync reconciliation.
- derive-id golden test: rename the describe from "(cross-package drift guard)"
to "(deterministic-scheme pin)" — there is no second package to drift against.
editor-ext 129, MCP 304 (+3), client+server tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- footnote-sync: remove the now-dead `refReids` (CollisionPlan field, local,
return, the 6a consumer loop) — references are never re-id'd under reuse, so it
was dead structure on the hot reconciliation path. Rewrite the stale comments
(plugin header, step 0, refOccurrences field) that still described the old
"duplicates re-id'd so both survive" model to the reuse model.
- Shared footnote lexer: new packages/mcp/src/lib/footnote-lex.ts
(lexFootnoteLines + forEachFootnoteReference). extractFootnotes (collaboration)
and analyzeFootnotes now consume the SAME fence-aware lexer, so "the analyzer
sees exactly what the importer keeps/strips" is structural, not comment-kept.
Removed the duplicated DEF_RE/fence machine from both consumers.
- Tests: new mock test for the footnoteWarnings plumbing on createPage (problems
-> field present; clean -> omitted); new paste-reuse case for TWO colliding
pasted definitions (reservation -> distinct ids). Updated the derive-id golden
test header (no MCP copy / parity test anymore).
- CHANGELOG: [Unreleased] entries for footnote reuse (Changed, supersedes 0.93.0)
and footnoteWarnings (Added).
editor-ext 129, MCP 301, server roundtrip 2; client+server tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Footnotes were strict 1:1: a repeated `[^a]` reference was treated as a
collision and re-id'd to `a__2`, and a reference with no definition synthesized
its own empty one — so an agent-authored article with reused labels produced
dozens of empty `kowiki__N` footnotes. Move to Pandoc REUSE semantics and add
non-fatal import diagnostics.
Reuse (core):
- resolveCollisions (footnote-sync): repeated references sharing an id are REUSE
(recorded once in document order, never re-id'd) — one number, one shared
definition. Only a duplicate DEFINITION is re-id'd deterministically and, with
no matching reference, dropped by the existing orphan policy (first-wins).
CollisionPlan.refReids is now always empty (harmless no-op downstream).
- extractFootnoteDefinitions (marked) and extractFootnotes (MCP): duplicate
definition ids are FIRST-WINS (keep first, drop rest); reference markers are
never rewritten. Removed the marker-rewriting and the now-dead deriveFootnoteId
mirror + helpers from the MCP path.
Import diagnostics:
- New analyzeFootnotes() (MCP): fence-aware pure scan reporting dangling
references, empty/duplicate definitions and `[^id]` markers inside table rows.
- createPage / updatePage / importPageMarkdown now attach `footnoteWarnings`
(only when non-empty) so an agent can fix its markup; the page is still created.
Paste-reuse:
- footnotePastePlugin remaps only ids the pasted slice DEFINES (a colliding
definition); a pasted lone reference to an existing id keeps it (reuse).
Tests: reuse/first-wins rewrites of footnote.test, footnote-markdown.test,
footnote.marked.orphan.test and the MCP footnotes.test; new footnote-paste.test
(editor-ext) and footnote-analyze.test (MCP). Deleted derive-id-parity.test.mjs
(the MCP no longer derives ids; editor-ext's deriveFootnoteId keeps its own
golden test). editor-ext 128, MCP 299, server roundtrip 2, client views 3,
client+server tsc clean.
Two review suggestions applied: corrected a stale "duplicated in MCP" comment and
the dangling-reference warning wording.
Note: the multi-backlink editor UI (a reused definition linking back to each of
its references) is deferred to a follow-up — this PR delivers the data-integrity
core (reuse + warnings + paste-reuse). Forward links and numbering already reuse
correctly; the backlink currently targets the first reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the approve-with-comments review on PR #154:
- applyDocToFragment: hydrate PMNode.fromJSON in its OWN try so a hydration
failure (e.g. an unknown node type) is labelled "fromJSON" — the stage that
actually threw — instead of the misleading "updateYFragment". The diagnostic
comment on unstorableYjsError ("label names the stage that failed") is now
truthful.
- assertYjsEncodable: also rehearse PMNode.fromJSON(docmostSchema, …) so a doc
that would only fail in apply's hydration step is rejected at preview time too,
narrowing the preview/apply gap (review suggestion B). Still cheap — no live
fragment, no updateYFragment.
- Tests: relabel the diagnostic test to (fromJSON); add structural-diff edge
cases — neighbour deletion keeps the unchanged node's cursor anchor, doc->empty
clears the fragment without throwing, top-level node-type change diffs in
place — plus a preview-gate test for the new fromJSON rehearsal. 297/297 green.
build/ rebuilt for the changed lib module only (build/client.js left untouched
to avoid pulling in pre-existing unrelated src/build drift).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `subpages` node showed only one level of direct children. Add a `recursive`
attribute that renders the FULL descendant tree of the current page — fully
expanded, unlimited depth. Default `false`, so every previously-inserted node
stays flat (backward compatible). No backend changes: `POST /pages/tree` (via the
`getSpaceTree` wrapper) already returns the whole subtree as a flat `IPage[]`
(recursive CTE, permission-filtered); the nested tree is built on the client by
`parentPageId`.
- editor-ext `subpages.ts`: `recursive` attribute (parse/render `data-recursive`),
shared by client + server so the collab ProseMirror schema keeps the attribute.
- `getSpaceTree`: arg loosened to `{ spaceId?; pageId? }` (the endpoint accepts
either); new `useGetPageTreeQuery(pageId)` react-query hook.
- `subpages-view.tsx`: split into `FlatSubpages` (unchanged) and
`RecursiveSubpages`; `buildSubtree` assembles the nested tree (cycle/self-parent
guard, `sortPositionKeys` per level, root excluded) and a recursive `TreeNode`
renders it (16px indent per depth, soft "showing N" note past 300 — data never
capped). Shared/public context reads the already-nested shared tree, no
`/pages/tree` request.
- toggles: bubble-menu flat⇄tree button + a second slash-menu item "Page tree".
Review follow-ups folded in: invalidate `["page-tree"]` from the create / update /
move / delete cache helpers so an open recursive tree refreshes (no stale data);
mode icon made reactive on editor transactions; `t` threaded into `TreeNode`
(no per-node useTranslation); shared-subtree hook deduped to a thin alias.
editor-ext build + client `tsc --noEmit` both clean. Backend untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review of #154 (Request changes) — all clean follow-ups, no defect in the fix:
1. Single source of the ProseMirror schema: export `docmostSchema` from
docmost-schema.ts (next to docmostExtensions); diff.ts and collaboration.ts
import it instead of each calling getSchema(docmostExtensions) — the schema
can no longer drift between call sites. Removed both local builds + the now
unused getSchema imports.
2. Doc fix: assertYjsEncodable's docstring and the client.ts comment no longer
claim "the same encoder as apply" — apply uses updateYFragment, the dry-run
uses toYdoc; both reject the same unstorable attrs but are NOT byte-identical.
Reworded to "independent encodability gate".
3+4+5. Extracted `unstorableYjsError(safe, label, e)` — buildYDoc and
applyDocToFragment now share one message template (label kept for diagnostics:
toYdoc vs updateYFragment), so the wording can't drift between dry-run/apply.
6. Test for applyDocToFragment's catch branch: an unknown node type makes the
schema-validated PMNode.fromJSON throw, and the function must re-throw it
wrapped with the (updateYFragment) diagnostic.
build/ rebuilt for the three changed lib modules; 293 package tests green.
(Left build/client.js untouched: rebuilding it would pull in a pre-existing,
unrelated src/build drift — a listSidebarPages slugId fix never rebuilt on
develop — and my client.ts change there is comment-only.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review of #157 (Request changes) caught two blockers:
1. DEAD responsive CSS: the `@media (max-width:600px)` float-reset was added to
`image-resize.module.css`, which is imported NOWHERE — the image container's
classes come from `common/node-resize.module.css` (via buildResizeClasses).
So on mobile a floated image kept its px width + float and crushed the text,
exactly the failure the rule promised to prevent. Moved the rule to
`common/node-resize.module.css` (the module actually imported by the resize
node views); its `:global([data-image-align=...])` selectors are data-attr
based, so they work unchanged. Reverted the dead addition from the (pre-existing,
orphaned) image-resize.module.css.
2. `applyAlignment` was untested. Exported it and added `image.spec.ts` (vitest/
jsdom) covering all five align values, the data-image-align mirror, and the
floatLeft -> left reset-then-apply (the guard against a leaked float).
Switched the float writes to the canonical CSSOM `cssFloat` property (portable:
browsers + jsdom; behavior identical to the `.float` alias).
editor-ext build + client tsc clean; 6 image.spec tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds floatLeft / floatRight image alignment so text wraps beside the image,
beyond the existing block left/center/right. Ported from Forkmost PR #7 /
upstream Docmost PR #1132 (fuscodev), adapted to gitmost's imperative image
node-view (the upstream uses a React styled component; ours styles the node-view
container directly via applyAlignment).
- editor-ext image.ts: `setImageAlign` accepts `floatLeft`/`floatRight`;
`applyAlignment` resets float/padding then, for a float mode, sets
`float:left|right` + side padding on the (shrink-to-fit) container so text
flows beside it (the inner <img> already has max-width:100%). The resolved
align is mirrored onto the container as `data-image-align` for the responsive
rule. `data-align` already round-trips the value through parse/renderHTML, so
float survives serialization / collab / history with no schema change.
- image-menu.tsx: Float-left / Float-right bubble-menu buttons (IconFloatLeft/
Right) with active state.
- image-resize.module.css: on narrow screens (<=600px) a floated image collapses
to full width and drops the float (`!important`, keyed on data-image-align) —
the upstream "100% width on small screen" follow-up.
- i18n: en-US + ru-RU strings.
editor-ext build + client tsc --noEmit clean. Visual wrap behavior is best
confirmed in-browser (logic/serialization verified by build + types).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mutatePageContent wrote agent edits back by DELETING the whole Yjs fragment and
re-applying a fresh Y.Doc. Yjs is a CRDT — the editor anchors its selection to
node ids — so wiping every id made an open editor's cursor lose its anchor and
snap to the end of the document on every agent write. It was most visible on
comment anchoring (issue #152): a comment changes no text, yet the cursor jumped.
(Before commit 4201f0a3 the anchoring silently no-op'd, so the destructive write
never ran for comments — hence the regression.)
Fix: write via `updateYFragment` (y-prosemirror) — the same routine the editor
uses to sync its own edits into Yjs. It structurally diffs the new doc against
the live fragment and touches only changed nodes, preserving the ids of unchanged
ones, so the cursor stays put. This improves ALL agent write tools (text edits,
node ops, comments, replace) — minimal diff instead of full replace: less collab
noise, stable block-ids, other users' cursors no longer disrupted.
- collaboration.ts: new `applyDocToFragment` (sanitize -> PMNode.fromJSON against
a memoized docmost schema -> updateYFragment in one transact), keeping the
`findUnstorableAttr` encode diagnostic; swap the destructive write-back for it.
- package.json: `y-prosemirror` promoted to a direct dependency (was transitive).
- test: comment-cursor-stability.test.mjs — a Yjs RelativePosition (the cursor
anchor) survives both a sibling edit and a comment-mark anchoring (the old
full-replace tombstoned it -> null). 292 package tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds co-located unit tests for ten targets (client → vitest *.test.ts(x),
server → jest *.spec.ts), plus minimal behavior-preserving extractions/exports
where the issue required a pure function to test:
- encode-wav: WAV header + PCM16 clamping
- editor-ext embed-provider / utils (sanitizeUrl, isInternalFileUrl) / indent
(export clampIndent)
- label.dto @Matches regex
- move-page.dto vs generateJitteredKeyBetween parity (bug locked via test.failing)
- new-note-button canCreatePage (extracted to can-create-page.ts)
- history-editor diff (extracted pure computeHistoryDiff into history-diff.ts)
- notification getTypesForTab + repo contract (direct-tab divergence locked via
test.failing)
- search buildTsQuery (extracted + sanitizes operator inputs so adversarial
queries no longer risk a to_tsquery 500)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a stable native-host JS-API on window.gitmost so the gitmost.app
WKWebView wrapper can hand a recorded audio file to the current page as an
audio block without depending on editor internals (atoms/Tiptap/Yjs).
- page-editor.tsx: register/tear down window.gitmost only while an editable
page editor is mounted (ready=true, version=1). insertRecording validates
mime/size, decodes base64 in 4-char-aligned chunks, rejects oversized
payloads before decoding (too-large), reuses the existing uploadAudioAction
pipeline, resolves machine-readable error codes
(no-editor/bad-type/too-large/insert-failed) and never throws. Cleanup is
identity-guarded so an unmounting PageEditor cannot disable a newer live
registration.
- editor-ext audio-upload: return the uploaded attachment from the upload fn
so the bridge can report success + attachmentId. Backward compatible:
existing fire-and-forget callers ignore the return value; the error path
still swallows and returns undefined (no re-throw).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
createPage always failed with "generateJSON can only be used in a Node
environment". Root cause: the MCP module (packages/mcp/.../collaboration.ts)
sets `global.window = dom.window` (jsdom) at load time and is imported
in-process by the server's AI-chat tools, leaking a global `window` into the
Node process. The server's self-contained ProseMirror helpers guarded with
`if (typeof window !== 'undefined') throw`, which then became a false positive
and broke POST /pages/import (the endpoint createPage calls).
- server: drop the vestigial `typeof window` guard in generateJSON.ts and
generateHTML.ts; both helpers create their own happy-dom Window and never
read the global one. Replace it with an explanatory comment.
- mcp: in DocmostClient.getPage, pass the resolved UUID (resultData.id) to
listSidebarPages instead of the original pageId, which may be a slugId and
triggered a Postgres "invalid input syntax for type uuid" (and a silent
empty subpages list).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The in-app AI chat hardcoded type='page' and the shared createComment
swallowed anchoring failures silently, so agent comments never got a
text anchor/highlight.
- Forbid page-type comments for the agent: top-level comments are always
inline and require an exact `selection`; replies inherit the parent
anchor (stored as the historical `page` type).
- Throw and roll back the just-created comment when the selection cannot
be anchored, instead of leaving an orphan unanchored comment.
- Add comment-anchor module: text normalization (smart quotes, dashes,
nbsp, collapsed whitespace) and matching across adjacent text nodes
within a block, so selections crossing inline-code/bold/link anchor.
- Update create_comment (MCP) and createComment (ai-chat) tool schemas
and descriptions; add unit + mock-HTTP orchestration tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Shared zod-agnostic tool-spec registry for the 14 identical AI tools across
the standalone MCP server and the in-app AI-SDK chat (keeps execute/auth and
the ~17 intentionally-divergent guardrail tools per-layer), folds in the
edit_page_text drift-bug fix, and formalizes the integration-test db factory.
Implements two architecture follow-ups from the multi-aspect review.
1. Shared, zod-agnostic tool-spec registry (packages/mcp/src/tool-specs.ts)
for the 14 AI tools whose name + schema + model-facing description are
genuinely identical across the standalone MCP server and the in-app
AI-SDK chat. Both layers consume it (registerShared in index.ts;
sharedTool in ai-chat-tools.service.ts) and keep their own execute/auth.
- Zod-agnostic builders (z) => ZodRawShape bridge the zod v3 (mcp) vs
zod v4 (server) split; the registry imports no zod.
- Folds in the documented edit_page_text drift-bug fix: the stale
"strip-and-retry tolerated" claim is gone; canonical wording states a
formatting-only change is refused into failed[].
- Sibling-tool references in shared descriptions are transport-neutral so
one description is correct for both snake_case (MCP) and camelCase
(in-app) tool names.
- Loader fail-fast guard for a stale @docmost/mcp build.
- The ~17 intentionally-divergent tools (security guardrails, tuned UX)
stay per-layer, untouched.
- Rebuilt committed mcp artifacts (also regenerates a previously stale
build/lib/docmost-schema.js to match its already-committed source).
2. Formalize apps/server/test/integration/db.ts as the canonical
integration-test seed factory (module doc + a shortId helper); the
hand-written minimal seeders are kept on purpose, decoupled from the
app service-layer side effects.
Verified: server tsc + lint clean, mcp build clean; mcp unit tests 261 pass,
ai-chat-tools.service 16 pass, public-share-chat-tools 8 pass, ai-chat suite
224 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
