PULL now serializes each page as the native-Obsidian format (serializePageFile:
a minimal gitmost_id frontmatter + the fixpoint markdown body) instead of the
heavy docmost:meta envelope. title/parent/space are derived (filename / folder /
repo), so only the pageId is persisted. readExisting recovers identity from the
gitmost_id frontmatter (parsePageFile) instead of docmost:meta.
Extracted stabilizePageBody() (the export->import->export fixpoint, no meta) so
the native writer and the legacy serializer share the same deterministic body —
re-pulls of an unchanged page stay byte-identical (loop-guard).
Tests: read-existing fixtures rewritten to gitmost_id; apply-pull asserts the
written text is native frontmatter and carries NO docmost:meta (regression
guard). 611 engine tests green.
NOTE: PUSH still reads docmost:meta — the end-to-end cycle is intentionally NOT
runnable until phase 3 (PUSH reads frontmatter + derives title/parent from path)
lands; no vault is wiped/deployed until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Native-Obsidian structure: a page WITH children now lives at its folder-note
<name>/<name>.md (LostPaul Folder Notes convention) with its children alongside;
a leaf stays <name>.md. Folder-notes claim their canonical path before a
same-named child, so the child (a leaf) is the one disambiguated, never the
folder-note — a folder X/ always contains its own note X.
Format-agnostic and safe in isolation: only the destination PATH changes, the
file content/serialization is untouched, so an existing parent relocates via the
move-by-id path (no delete). The frontmatter format flip (pull+push) is next.
6 new layout unit tests (leaf / parent / nested / child-named-as-parent /
twin-parents / childless). 611 engine tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per owner: test data, no migration. parsePageFile no longer reads the old
docmost:meta block — a file without a gitmost_id frontmatter is simply un-tracked
(adopt). Vaults are a cache: rm -rf on the transition, rebuilt native from
Docmost. Simplifies the format work (no fallback). Doc updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pivot the thin-meta design to "the vault IS a native Obsidian vault": clean
markdown + a minimal YAML frontmatter `gitmost_id:` (the durable pageId, travels
with the file so identity survives any move); folders mirror the page tree with
the parent's body as a folder-note `<Folder>/<Folder>.md` (LostPaul Folder Notes
convention); links as `[[wikilinks]]` (basename-resolved → reparent never breaks a
link, only retitle does); collisions disambiguated Obsidian-style; `.obsidian/`
and non-page files left untouched (no .gitignore). Verified the conventions
against the Obsidian/Folder-Notes docs.
Replaces the abandoned `.gitmost/index.json` sidecar (path-keyed → fragile to
git-undetected renames; the in-file id is self-sufficient): removes vault-index.ts.
Adds lib/page-file.ts — parsePageFile/serializePageFile (frontmatter id + clean
body) with a LEGACY `docmost:meta` fallback for migration. 6 unit tests; engine
suite green. Not yet wired into pull/push — no behavior change. Design doc
rewritten to the native-Obsidian format.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure read/write/lookup for the vault sidecar index that will hold page identity
(pageId) + collision token (slugId) keyed by file path, so the .md files can be
clean markdown. parseVaultIndex is tolerant (missing/garbage/bad entries degrade
to empty/skipped — never crashes a cycle); serializeVaultIndex is deterministic
(sorted keys -> stable diffs, no churn). Lookups (pageIdAt, pathForPageId reverse,
trackedPageIds) + mutations (set/remove/move). NOT wired into pull/push yet — no
behavior change. 5 unit tests; engine suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to 4376c5a6, found by a real BROWSER e2e (the flow the in-diff fix
missed). When the layout reshuffle's two halves land in SEPARATE sync cycles, the
later cycle's diff has only the DELETE of the old path — the matching add was
already pushed — so in-diff D+A coalescing can't see it, and the live page was
still trashed.
Robust fix on the identity invariant the reviewer (and the user) called out: a
page EXISTS iff its pageId is in the vault, regardless of filename. runPush now
collects the pageIds present at ANY path in the current `main` tree and passes
them to computePushActions; a deleted file whose pageId is still tracked
elsewhere is a MOVE, never a deletion. (Built only when the diff has deletes.)
Adds apps/server/test/git-sync-browser-e2e.cjs — a Playwright test that drives the
REAL Docmost web UI: log in, create several untitled pages, type a title, sync,
assert NOTHING is trashed. Reproduced the data loss before this fix; 5/5 green and
stable after. Engine suite 600 green (+2 computePushActions cases:
pageId-still-present -> skip; pageId-gone -> real delete).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CRITICAL data-loss bug: creating pages in Docmost (which start UNTITLED) and then
typing a title could soft-delete OTHER pages. Untitled pages all serialize to the
`_` fallback filename; the layout disambiguates them (`_.md`, `_ ~slug.md`).
Retitling one frees the bare `_` and another untitled page's file relocates into
it. git's rename detection (`-M`) can't see the move (the tiny meta-only files are
too dissimilar), so `git diff` reports it as DELETE(old) + ADD/MODIFY(new). The
push took the DELETE literally and trashed a live page.
Root cause is that the push trusted git's path-level rename heuristic for page
IDENTITY. Identity is the pageId. Fix: before emitting any delete, coalesce by
pageId — a pageId that is BOTH deleted (pre-image) AND present on the surviving
side (current meta of an ADD or a MODIFY, since a relocation into an occupied path
shows as M) is one page that MOVED, classified as a rename/move and NEVER a delete.
Reproduced + verified on a live stand: 4 untitled pages + retitle one trashed a
different page before; after the fix, retitling one (and stress-retitling all)
trashes nothing. Engine suite 598 green; 3 new computePushActions cases (ghost
D+A move -> rename; real delete still deletes; unrelated D+A stay delete+update).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconcile choreography (ensureRepo -> merge-check -> ensureBranch ->
checkout('docmost') -> pull -> push) was hand-rolled in the app orchestrator's
driveCycle, duplicating an order the vendored engine owns and could drift from on
upgrade — the failure mode is data clobber. Lift it into @docmost/git-sync as a
single entry point, `runCycle(deps)`. The orchestrator now calls runCycle and
keeps only the lock (its caller) and the gitmost-specific delete-cap POLICY,
injected as the `resolveApplyClient` hook (the engine does the dry-run, hands the
hook the planned delete count — Infinity if planning failed — and uses whatever
client it returns for the apply). driveCycle drops from ~150 lines to ~30.
Tests:
- engine test/cycle.test.ts: composition (merge-in-progress short-circuit;
ensureRepo->ensureBranch->checkout staging order before the pull; the cap hook
is consulted with the planned count; no dry-run when no hook).
- engine test/cycle-roundtrip.test.ts: runCycle against a REAL VaultGit in a temp
repo with a faked Docmost client — a git-originated CREATE flows pull->push and
the assigned pageId is written back; an unresolved merge short-circuits before
any client call.
- orchestrator spec rewired to mock runCycle and assert the wiring + the
resolveApplyClient cap policy (the engine-internal cycle-order/merge tests moved
to the engine).
Validated end to end on a live stand (real Postgres/Redis + server): a git clone
-> edit -> push over the /git remote round-trips the change into the Docmost page
through the refactored cycle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The implementation spec docs/git-sync-plan.md was removed as completed, but ~44
code comments still cited it as "plan §N". Strip those citations (comments only),
keeping each comment grammatical. The vendored engine's own "SPEC §N" references
point at a different, still-present spec and are left untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same hygiene fix as git-sync (review #2), applied to packages/mcp which had the
identical pre-existing problem: committed build/ (20 files) + node_modules (28,
pnpm symlinks with a baked /home/claude store path).
- git rm --cached packages/mcp/{build,node_modules}.
- .gitignore: add packages/mcp/build/ (packages/*/node_modules/ already covers it).
- Build where consumed: apps/server `pretest` and the CI Test workflow now build
@docmost/mcp too. The Dockerfile builder already runs `pnpm build` (nx builds
mcp) and already COPYs packages/mcp/build into the runtime image.
Verified: wiped build/, rebuilt via `pnpm --filter @docmost/mcp build`; the mcp
server suites (96 tests) pass against the freshly-built, non-committed output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Upgrades the 2-way body merge to a real diff3 three-way merge (review #5), so a
block ONLY the human changed is KEPT when git changed a DIFFERENT block — the
2-way merge would revert it to git's stale version.
Engine: the push update loop reads the last-synced pre-image
(`git.showFileAtRef(refs/docmost/last-pushed, path)`) and passes it as the
optional `baseMarkdown` to `client.importPageMarkdown` (the common ancestor).
Server: gitmost-datasource converts base+incoming, and writeBody runs a block-
level diff3 (new three-way-merge.ts `diff3Plan`): live-only change -> keep live,
git-only change -> take git, both-changed -> git wins (conflict policy), inserts/
deletes from either side preserved. Without a base (createPage) it falls back to
the 2-way merge. Crash-safety unchanged (docs built before the connection opens).
Tests: three-way-merge.spec.ts (14 — every diff3 case incl. the cross-block
preservation and conflict policy), yjs-body-merge 3-way (real Y.Docs: human's
block instance preserved while git's block is applied), plus an engine test that
the base is forwarded from showFileAtRef. Existing push assertions updated for the
new base arg. git-sync 589 pass; server merge/datasource/gate 62 pass; typecheck
clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review finding #2: packages/git-sync/build/ (the COMPILED engine) and the
package's node_modules/ were committed. Prod executed the committed build/ while
CI/tests ran src/ and never rebuilt it — so a fix in src/ could pass tests while
stale compiled code shipped (a silent src/prod skew). The committed node_modules
were pnpm symlinks with a baked machine-local store path (/home/claude/...),
useless and misleading for everyone else.
- git rm --cached packages/git-sync/{build,node_modules} (42 + 31 files).
- .gitignore: ignore packages/*/node_modules/ and packages/git-sync/build/.
- Build the package where it is actually consumed: apps/server `pretest` now
builds @docmost/git-sync (its suite imports the built build/index.js), and the
CI Test workflow gains an explicit "Build git-sync" step. The Dockerfile builder
already runs `pnpm build` (nx builds the package) and now COPYs the fresh build/.
Verified: wiped build/, rebuilt via `pnpm --filter @docmost/git-sync build`, then
the server converter gate (26/26, imports the rebuilt package) and the git-sync
suite (588 passed) both pass against the freshly-built, non-committed output.
NOTE: packages/mcp/ has the same committed-build/node_modules pattern (pre-existing,
out of this PR's scope) and should get the same treatment in a follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This branch commits packages/git-sync/build/ and the server/Docker consume it,
so the stale build/ would otherwise ship WITHOUT the round-trip data-loss fixes
in 7d39c16b. Rebuilt via tsc (only the two changed modules). NOTE: not committing
build/ at all (review finding #2) is the proper fix, pending the CI/Docker
build-orchestration change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Coder↔reviewer design loop (9 rounds, reviewer verdict: exhaustive) produced
92 specs; implemented +123 tests (465 -> 588 passing). The new round-trip
coverage exposed three genuine data-loss bugs in the Markdown<->ProseMirror
converter, all now FIXED (round-trip is lossless for these):
1. pageBreak was lost on export (no converter case -> rendered to "" and the
node vanished). Now emits <div data-type="pageBreak"></div>, which the schema
parses back -> round-trips.
2. A block image between blocks left an empty <p> artifact after import-hoisting,
producing a phantom blank-gap diff on every sync. markdownToProseMirror now
strips content-less paragraphs after generateJSON — with a schema-validity
guard that keeps the obligatory single empty paragraph in `content: "block+"`
containers (tableCell/tableHeader/blockquote/column/callout/doc), so empty
cells/quotes never become an invalid `content: []`.
3. The `code` mark combined with another mark was not byte-stable (emitted nested
HTML that the schema's `code` `excludes:"_"` collapsed on import). The
converter now emits code-only when `code` co-occurs, matching the editor.
New coverage spans media/diagram/details/columns/math/mention attribute
round-trips, converter emission branches, git error paths, and engine decision
branches. A dedicated test pins the empty-container schema validity (the review
catch on the bug-2 fix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make @docmost/git-sync natively consumable by the CommonJS server (and jest):
build to CommonJS (tsconfig module CommonJS, drop type:module, strip .js from
relative imports), and lazy-load the only ESM-only dep (marked) via the dynamic
Function('import()') trick (mirrors docmost-client.loader.ts) with a require()
fallback so vitest's evaluator works too. git-sync tests stay green (314 pass,
3 expected fail).
Add the §13.1 idempotency gate (apps/server .../git-sync-converter-gate.spec.ts):
13 editor-ext docs (paragraphs/headings, marks, links, bullet/ordered/task lists,
blockquote, callouts, code block, hr, table, nested mix) round-trip
content(editor-ext) -> convertProseMirrorToMarkdown -> markdownToProseMirror ->
TiptapTransformer.toYdoc/fromYdoc(tiptapExtensions) -> canonicalize and assert
docsCanonicallyEqual. All green => the vendored converter's docmost-schema is
schema-compatible with editor-ext (no node/mark/attr loss), which the plan §13.1
requires before Phase B. The one intrinsic markdown-image lossiness (width/height
/align can't ride plain ) is isolated in a KNOWN DIVERGENCE block, not
hidden. Server tsc clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First step of docs/git-sync-plan.md. New workspace package @docmost/git-sync
vendoring the PURE parts from docmost-sync (HEAD b03eb35):
- lib: markdown-converter, markdown-document, canonicalize, docmost-schema,
node-ops, diff, and an extracted markdown-to-prosemirror (only the pure
marked->HTML->generateJSON path from upstream collaboration.ts; no websocket).
- engine (pure, no IO): reconcile, layout, sanitize, stabilize, loop-guard.
Ported the upstream pure-module + round-trip corpus tests (vitest): 314 pass,
3 expected upstream known-limitation fails. tsc clean. No server wiring yet.
docmost-schema inlines getStyleProperty (as packages/mcp does — @tiptap/core
3.20.4 doesn't export it). IO engine (pull/push/git/settings) deferred to later
Phase A/B steps; the editor-ext idempotency gate (plan §13.1) is the next step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The in-app AI chat hardcoded type='page' and the shared createComment
swallowed anchoring failures silently, so agent comments never got a
text anchor/highlight.
- Forbid page-type comments for the agent: top-level comments are always
inline and require an exact `selection`; replies inherit the parent
anchor (stored as the historical `page` type).
- Throw and roll back the just-created comment when the selection cannot
be anchored, instead of leaving an orphan unanchored comment.
- Add comment-anchor module: text normalization (smart quotes, dashes,
nbsp, collapsed whitespace) and matching across adjacent text nodes
within a block, so selections crossing inline-code/bold/link anchor.
- Update create_comment (MCP) and createComment (ai-chat) tool schemas
and descriptions; add unit + mock-HTTP orchestration tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Shared zod-agnostic tool-spec registry for the 14 identical AI tools across
the standalone MCP server and the in-app AI-SDK chat (keeps execute/auth and
the ~17 intentionally-divergent guardrail tools per-layer), folds in the
edit_page_text drift-bug fix, and formalizes the integration-test db factory.
Implements two architecture follow-ups from the multi-aspect review.
1. Shared, zod-agnostic tool-spec registry (packages/mcp/src/tool-specs.ts)
for the 14 AI tools whose name + schema + model-facing description are
genuinely identical across the standalone MCP server and the in-app
AI-SDK chat. Both layers consume it (registerShared in index.ts;
sharedTool in ai-chat-tools.service.ts) and keep their own execute/auth.
- Zod-agnostic builders (z) => ZodRawShape bridge the zod v3 (mcp) vs
zod v4 (server) split; the registry imports no zod.
- Folds in the documented edit_page_text drift-bug fix: the stale
"strip-and-retry tolerated" claim is gone; canonical wording states a
formatting-only change is refused into failed[].
- Sibling-tool references in shared descriptions are transport-neutral so
one description is correct for both snake_case (MCP) and camelCase
(in-app) tool names.
- Loader fail-fast guard for a stale @docmost/mcp build.
- The ~17 intentionally-divergent tools (security guardrails, tuned UX)
stay per-layer, untouched.
- Rebuilt committed mcp artifacts (also regenerates a previously stale
build/lib/docmost-schema.js to match its already-committed source).
2. Formalize apps/server/test/integration/db.ts as the canonical
integration-test seed factory (module doc + a shortId helper); the
hand-written minimal seeders are kept on purpose, decoupled from the
app service-layer side effects.
Verified: server tsc + lint clean, mcp build clean; mcp unit tests 261 pass,
ai-chat-tools.service 16 pass, public-share-chat-tools 8 pass, ai-chat suite
224 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First, safe step of docs/backlog/ai-chat-tool-definitions-duplicated.md: the
"node may be a JSON object OR a JSON string" quirk was hand-copied at 6 tool
sites. Extract it into a single parseNodeArg() helper per package and call it at
every site. Behavior-preserving — each site's throw message is byte-identical
(patch/insert: 'node was a string but not valid JSON'; update_page_json: 'content
was a string but not valid JSON'); no tool name/description/schema changed.
Two helper copies (packages/mcp/src/lib/parse-node-arg.ts and
apps/server/src/core/ai-chat/tools/parse-node-arg.ts) are intentional: the
ESM-only @docmost/mcp cannot be imported by the CommonJS server (it is loaded at
runtime via the Function('import()') trick), so runtime code cannot cross that
boundary by a normal import. Each copy is now the single source within its
package (6 inline copies -> 2 helpers). packages/mcp/build rebuilt in sync.
Tests: parse-node-arg.spec.ts (server, Jest) + parse-node-arg.test.mjs (mcp,
node:test) — object passthrough, valid-string parse, invalid-string throw with
the right message. Server tsc clean; mcp suite 254 pass; agent structural-edit
path verified live in-browser (agent inserted a node, persisted to the doc).
Deferred (documented for the record, since the backlog doc is removed with this
commit): the FULL transport-agnostic tool-spec registry (one name+schema+
description per tool shared by both transports) and deriving DocmostClientLike
from the real client type. Both are blocked by the current architecture, not by
effort: (1) @docmost/mcp ships no type declarations and is ESM-only, so a
type-only derivation needs declaration emission + tsconfig path wiring, and the
real client's precise return types break the in-app tool test stubs (attempted,
reverted to keep tsc green); (2) the two transports intentionally DIVERGE in tool
NAMES (snake_case x38 vs camelCase x41), membership (in-app adds getCurrentPage/
listSidebarPages, omits delete_comment/image tools) and model-facing
DESCRIPTIONS, so a unified registry would change behavior on BOTH the agent and
external MCP clients and needs its own verification pass. This is forward-looking
debt (the code is correct today), to be done incrementally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract parse/renderHtmlEmbedHeight and test: '300'->300, absent->null,
'abc'->null (pins the NaN guard), '120px'->120; render 120->data-height, null/0->{}.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add htmlEmbed to the schema toYdoc/fromYdoc acceptance cases, asserting source +
height survive, so removing the passthrough node (which prevents 'Unknown node
type: htmlEmbed' on MCP/AI edits of an embed page) fails CI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up fixes to the htmlEmbed-sandbox / trackerHead change:
- share-seo: inject trackerHead via a function replacer so `$`-sequences
($&, $', $`, $$) in the admin snippet are inserted literally instead of
being treated as String.replace substitution patterns; warn when the
</head> marker is absent instead of silently skipping injection.
- mcp: register a passthrough `htmlEmbed` node in the schema mirror so an
AI/MCP edit of a page containing an embed no longer throws
"Unknown node type: htmlEmbed" in TiptapTransformer.toYdoc.
- editor-ext + client: treat a non-finite `data-height` as auto (null) so a
crafted/corrupted height cannot disable auto-resize or yield a NaN iframe
height; extract a shared clampHeight helper.
- client: rename render-raw-html.{ts,test.ts} -> html-embed-sandbox.{...} and
shouldExecute -> shouldRender so the seam name matches the sandbox model.
- client: i18n the iframe title; surface the real error reason in
tracker-settings (console.error + err.response.data.message).
- docs: note hasHtmlEmbedNode is now a test-only helper; add an Unreleased
CHANGELOG entry; drop the dangling "arbitrary HTML embed" planning-doc ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The encode catch comment promised 'fall back to raw' but the code returns '';
returning raw source wouldn't help anyway (un-encoded markup can't be atob-decoded
downstream, so decode would yield '' regardless), and a raw value in data-source
breaks the inert-storage guarantee. '' is the correct decode-symmetric failure —
fix the misleading comment to say so. Adds a codec test for the encode-throw path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the htmlEmbed node from same-origin raw-HTML execution to a sandboxed
iframe (sandbox="allow-scripts allow-popups allow-forms", no allow-same-origin,
srcdoc) with postMessage auto-resize (validated by event.source) and an optional
manual height attr. The block now runs in an opaque origin and cannot reach the
viewer's cookies/session/API, so it is safe for any member.
Because the block is now harmless, remove the entire admin/role gating apparatus:
drop htmlEmbedAllowed/canAuthorHtmlEmbed/stripDisallowedHtmlEmbedNodes/
collectHtmlEmbedSources and every role-based strip on the write paths (collab
REST/MCP + socket, page create/duplicate, import x2, transclusion unsync), along
with the now-unused WorkspaceRepo/UserRepo injections and the PageService.create
callerRole param. Keep one strip: prepareContentForShare still removes htmlEmbed
on the anonymous public-share read path when the workspace master toggle is OFF.
The workspace settings.htmlEmbed toggle is now a plain feature switch (gates the
slash-menu and share rendering); when ON the block is available to all members.
Add settings.trackerHead: an admin-only raw HTML/JS analytics snippet injected
verbatim into the <head> of public share pages only (ShareSeoController), for
trackers that genuinely need same-origin. Admin-gated via the existing CASL
Manage/Settings ability; never injected into the authenticated app shell.
Closes security-review findings #1, #2, #4, #5, #10 (and #3 as a security issue).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ~330 tests across server (Jest), client (Vitest), editor-ext (Vitest)
and packages/mcp (node:test) for the gitmost features added since
053a9c0d: AI chat, AI agent roles, public-share assistant, MCP per-user
auth, HTML embed, page templates/embed, realtime tree, tree
expand/collapse, and the AI-settings UI.
Test-tooling fixes (prerequisite, were silently hiding coverage):
- Repair 3 page-template specs broken by the 11-arg TransclusionService
constructor; they never compiled, so template access-control / content
-leak / unsync-strip coverage was fictitious.
- Build @docmost/editor-ext before server tests via a `pretest` hook;
the stale dist omitted the new HtmlEmbed/PageEmbed exports (TS2305).
- Let jest resolve the .tsx email templates: add `tsx` to
moduleFileExtensions and widen the ts-jest transform to (t|j)sx?.
Behaviour-preserving "extract pure core" refactors that the tests drive:
- server: resolveShareAssistantRequest + uiMessageTextLength
(public-share controller), decideBasicGate + mapAuthResultToResponse
(mcp), buildErrorAssistantRecord (ai-chat), jsonbObject export (roles).
- client: render-raw-html + shouldExecute/canEdit, decide-embed-state,
page-embed picker utils, tree-socket reducers, open/close branch maps,
isEndpointConfigured/resolveKeyField; buildTreeWithChildren now treats
a permission-trimmed orphan as a root instead of crashing.
Deferred (need a test DB or HTTP harness, documented in the specs):
repo-level Postgres integration tests and the public-share XFF E2E.
Pre-existing DI/lib0-ESM suite failures are untouched and out of scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve conflicts at shared registration points by unioning both features
(footnotes + the already-merged html-embed / page-embed work):
- slash-menu/menu-items.ts, editor extensions.ts: keep both imports + configures
- collaboration.util.ts: register footnote nodes and pageEmbed
- editor-ext marked.utils.ts: register footnote + html-embed markdown extensions
- editor-ext package.json/tsconfig.json/vitest.config.ts: union of test config
(jsdom env for footnote DOM tests + combined test/spec include glob)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle review found two hardening gaps:
- The sync plugin deleted+rebuilt the WHOLE footnotesList on any reorder/orphan,
replacing every definition's Yjs subtree -> a collaborator typing in a
definition could lose in-flight characters on merge. Rework to targeted,
minimal mutations: attr-only setNodeMarkup for collision re-ids, delete only
genuine orphans, insert only genuinely-missing definitions (at the list end,
not shifting existing subtrees), and consolidate multiple lists only in the
abnormal paste/merge case. An unchanged (correct id, referenced) definition is
left completely untouched. Numbering is decoration-only, so physical list order
may drift after a reorder (accepted) while displayed numbers stay correct.
Invariants preserved (reviewed + tested): one SYNC_META transaction, null when
canonical (terminates), deterministic deriveFootnoteId, remote-skip -> no
re-introduced freeze or divergence.
- computeFootnoteNumbers ran per-NodeView-render (O(n^2)/keystroke in big docs).
The numbering plugin now caches the number map in its state (computed once per
docChanged); NodeViews read it O(1) via getFootnoteNumber.
Tests: no-rebuild-on-reorder asserts unchanged definition node subtrees are
identity-preserved; isRemoteTransaction skip; enableSync:false read-only; cache
correctness. Browser re-smoke: insert (no freeze), number, persist across reload,
cascade delete all pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle test audit: the strip boundary was tested only via a stand-in
helper re-implemented in the spec, so a deleted/misplaced guard kept CI green
(the missing create() guard was proof). Replace it with tests against real code:
- persistence.extension.onStoreDocument: real ydoc from a rich doc (columns/
table/mention/htmlEmbed) -> non-admin strip removes only htmlEmbed, every other
node preserved (data-loss guard); admin keeps; empty fragment no-throw.
- collaboration.handler.updatePageContent: real path, user?.role gate, decoded
ydoc embed-free for non-admin, kept for admin.
- transclusion unsync: member stripped, admin preserved.
- editor-ext gains a vitest setup (was zero tests) + a markdown round-trip:
the <!--html-embed:BASE64--> marker -> htmlEmbed node with decoded source, and
hasHtmlEmbedNode matches it — pinning the marked/turndown shape the import
strip relies on. tsconfig now excludes specs from the shipped dist.
- Fail-closed identity: source-pinned contracts that the gate keys on
fileTask.creatorId (zip) / request userId (single) / callerRole (create) /
authUser.role (duplicate), and missing-user -> strip (services can't load under
jest's ESM graph; helpers replay the exact predicate).
Adds the verified-safe ^src/ jest moduleNameMapper (identical fail set).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle red-team found two same-id footnoteDefinition nodes (trivially
produced by markdown import [^d]: first / [^d]: second, or paste/duplicate)
caused silent data loss: scan() used a last-wins Map and the sync rebuild
(addToHistory:false, propagated via Yjs, un-undoable) dropped all but the last.
Fix resolves collisions so BOTH survive, with a DETERMINISTIC id scheme so
collaborators converge:
- deriveFootnoteId(originalId, occurrence, taken): the k-th (k>=2) occurrence of
id X becomes X__k, bumped with a deterministic alpha suffix only against the
doc's own id set — a pure function of document state. No Math.random/Date.now
on the sync or import paths (random uuid stays only in setFootnote, where a
single user originates a brand-new id).
- footnote-sync.resolveCollisions walks refs+defs in document order, re-ids
duplicate references via setNodeMarkup and pairs them 1:1 with definitions;
single SYNC_META-tagged transaction, returns null when canonical (terminates).
- Markdown import (footnote.marked) + MCP mirror (collaboration.ts) dedup with
the same deterministic scheme + marker rewrite; packages/mcp/build regenerated.
- Paste plugin remaps colliding pasted ids against the current doc.
Tests: two independent editors resolving the same duplicate-id doc produce
IDENTICAL ids (the cross-client determinism guard that the random version would
fail); both definitions survive the first edit; import dedup is deterministic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds footnotes: a superscript marker in the text linked to an editable
definition in a Footnotes section at the end of the page, with auto-numbering
and a read-only hover popover. Chose the reference+definitions model (3 plain
nodes) over an inline atom with a sub-editor specifically for collaboration
safety.
editor-ext (packages/editor-ext/src/lib/footnote/):
- footnoteReference (inline atom, id), footnotesList (block, last child),
footnoteDefinition (paragraph+, id). renderHTML emits sup[data-footnote-ref]
/ section[data-footnotes] / div[data-footnote-def]; parse-rule priority makes
the empty reference win over the Superscript mark (else it is dropped on the
server save).
- numbering: a decoration-only plugin (pure function of doc order) -> every
client computes identical numbers, no document mutation, Yjs-safe.
- sync plugin: single-pass, always SYNC_META-tagged and skipping remote txns
(terminates, no loop), idempotent; canonicalizes to one trailing footnotesList
(merging duplicates), creates missing definitions, drops orphans, and
coexists with TrailingNode. Disabled in read-only.
- commands setFootnote (one tx: reference + definition at the matching index +
focus) / removeFootnote (cascade, one undo) / scrollTo*. slash /footnote.
client: superscript NodeView + floating-ui read-only popover; bottom-list and
definition NodeViews; registered in mainExtensions.
server: the three nodes registered in tiptapExtensions so collab/save/export
keep them. Round-trip regression spec guards the Superscript parse-priority.
markdown: turndown/marked round-trip to pandoc/GFM [^id] (+ a code-fence guard
so footnote-like lines inside code blocks are not extracted).
MCP mirror: schema + markdown-converter + commentsToFootnotes rewritten to real
footnote nodes + diff marker counting; NUL sentinels written as \u0000 escapes.
v2 follow-ups (per plan): definition reordering on reference move, id-collision
regeneration on paste, multiple references to one footnote.
Implements docs/footnotes-plan.md (variant B).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Embed another page's LIVE content into a host page (it updates when the source
changes, not a static copy). A page can be flagged a template for discovery in
the picker; any accessible page can be embedded.
Server:
- migrations: pages.is_template (+ partial index) and page_template_references
(whole-page back-refs); db.d.ts/entity types hand-merged (db.d.ts is curated).
- POST /pages/toggle-template (CASL Edit) flips is_template; is_template is
returned by findById + the sidebar tree select so the tree menu label
reflects state. Search suggestions gain an onlyTemplates filter for the picker.
- POST /pages/template/lookup ({sourcePageIds[]}, <=50): returns each accessible
source's {title, icon, slugId, content, sourceUpdatedAt} with comment marks
stripped (same access path as transclusion: filterViewerAccessiblePageIds;
inaccessible -> no_access, missing -> not_found; error path -> not_found, never
raw content).
- reference sync (collectPageEmbedsFromPmJson + syncPageTemplateReferences) on
the Yjs save hook; duplicatePage remaps pageEmbed.sourcePageId + inserts refs.
Known MVP gap: REST content updates don't resync refs (lookup uses in-doc ids).
Client:
- pageEmbed node (editor-ext, registered in BOTH client + server schemas);
read-only NodeView with a batching lookup; '/Embed page' slash + template
picker (self-embed prevented); 'Make/Unset template' in the tree node menu.
- Cycle guard: an ancestry-chain context + depth cap (5) render a 'circular
embed' placeholder instead of recursing.
- Public shares show a placeholder (no public lookup in MVP).
MVP excludes (follow-ups): public-share lookup, unsync->static copy, server-side
expansion for export/RAG, MCP schema mirror, point-in-time snapshots.
Implements docs/page-templates-plan.md (MVP, variant A).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an htmlEmbed block node that renders and executes raw HTML/CSS/JS in the
wiki origin (e.g. an analytics tracker) — the owner-chosen variant C. Because
this is stored-XSS by design, only workspace admins/owners may get such a node
persisted; everyone executes it when reading.
- Node (editor-ext): htmlEmbed atom/isolating block; source stored base64 in
data-source for lossless HTML<->JSON round-trip. renderHTML emits only the
encoded marker (never inlines raw markup), so generateHTML/export/search are
not themselves injection vectors. Registered in BOTH client extensions and
server tiptapExtensions. Markdown round-trip via an <!--html-embed:b64-->
comment (turndown) + a marked rule.
- Client NodeView: injects source and re-creates <script> elements so they
actually run; edit modal; renders in read-only/share too. Slash item is
admin-gated (adminOnly filtered by the user's workspace role).
- SERVER ENFORCEMENT (the real control — UI gating alone is insufficient):
stripHtmlEmbedNodes() removes htmlEmbed from any document persisted by a
non-admin, applied at every write path that introduces content from an
untrusted author: collab onStoreDocument, REST/MCP/AI updatePageContent,
single-file import, zip/multi-file import, page duplication, and transclusion
unsync. Page restore introduces no new content. Public share/readonly viewers
render fetched (already-stripped) content and do NOT open a collab socket, so
the only residual is a transient broadcast window to concurrent authenticated
editors (documented).
Implements docs/arbitrary-html-embed-plan.md (variant C).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The embedded MCP server acted as a single service account; now each /mcp
session authenticates as the current user, so tools run under that user's
CASL and edits attribute to them.
- HTTP Basic (chosen path): Authorization: Basic email:password, validated
server-side via AuthService; the session carries the issued user JWT (not
the raw password). Password may contain ':' (split on first only).
- Bearer fallback: Authorization: Bearer <access JWT>, verified as ACCESS and
additionally checked for an active session + non-disabled user (matching
JwtStrategy), so revoked/disabled users are rejected.
- Service account stays as an optional fallback (no creds + env configured).
- packages/mcp createMcpHttpHandler accepts a per-request config resolver
(back-compat: static config / stdio unchanged); identity is bound to the
mcp-session-id at init and re-validated from the caller's own credentials on
every request (anti session-fixation: a guessed session id can't be reused
without matching creds).
- A full login (session + audit) happens only once at session init; later
requests re-verify credentials via a new non-side-effecting
AuthService.verifyUserCredentials (no session/audit spam).
- Failed-login limiter (5/60s, keyed per-IP, per-IP+email, and per-email so IP
rotation can't brute one account) since direct login bypasses the controller
throttler. Only real credential failures count.
- MCP_TOKEN shared guard moved off Authorization to an X-MCP-Token header
(timing-safe compare); credsConfigured 503 gate replaced by a clear 401.
- No secrets logged; all auth resolved before res.hijack() so failures return
clean 401 JSON. .env.example marks the service account optional.
Implements docs/backlog/mcp-per-user-auth.md (variant L).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
list_pages gains an opt-in `tree` parameter on both surfaces (the
@docmost/mcp server tool and the AI-chat agent tool), which share the
same DocmostClient.listPages. Default behavior (recent-by-updatedAt flat
list) is unchanged.
- client.ts: listPages(spaceId?, limit=50, tree=false); when tree is
true it requires spaceId (throws a specific error otherwise), walks the
sidebar tree via the existing bounded/cycle-safe enumerateSpacePages,
and returns a nested tree; limit is ignored in tree mode.
- lib/tree.ts: new pure buildPageTree() — lean nodes { id, slugId, title,
children? }, children sorted by position (code-unit order), orphans
promoted to roots, cycle-safe.
- index.ts + ai-chat-tools.service.ts: expose `tree` in the tool schemas
and descriptions; docmost-client.loader.ts: mirror the new signature.
- tests: add packages/mcp/test/unit/tree.test.mjs (nesting, ordering,
lean shape, orphan promotion, cycle/self-reference safety).
- rebuild @docmost/mcp (build/ is tracked and loaded at runtime).
edit_page_text reported "success" when asked to change formatting (e.g. remove
strikethrough): the markdown-strip fallback matched the bare text, the replace
preserved marks, and the tool returned success — so the agent believed it had
fixed something that never changed.
Two fixes, both in the shared @docmost/mcp DocmostClient so they reach BOTH the
standalone MCP server and the in-app AI chat (which loads @docmost/mcp):
- Verifiable result for every content mutator: mutatePageContent now computes a
`verify` change-report (text inserted/deleted, blocks changed, per-mark-type
delta, integrity/structure delta) via summarizeChange() and returns it on all
mutators (incl. replaceImage via mutateLiveContentUnlocked). diffDocs is
text-only, so the mark/structure delta is what surfaces formatting changes.
- edit_page_text hard-refuses formatting edits: applyTextEdits rejects an edit
whose find/replace differ only in markdown markers (via stripBalancedWrappers,
which strips balanced wrappers/links without trimming whitespace/emoji, so
plain-text edits like trailing-space trims, snake_case, math are NOT refused).
A fully-refused batch errors instead of silently succeeding.
Also updated the model-facing edit_page_text descriptions in BOTH tool layers
(packages/mcp/src/index.ts and ai-chat-tools.service.ts) to drop the misleading
"strip-and-retry tolerated" wording and point formatting changes to patch_node.
New unit tests: test/unit/diff-verify.test.mjs, test/unit/json-edit-refuse.test.mjs.
The insert_image and replace_image MCP tools previously uploaded only
local files (filePath), which an AI MCP client cannot provide — it has
no access to the server filesystem. Replace filePath with a required
imageUrl and download the image over http(s).
- client.ts: add fetchRemoteImage(url, maxBytes) — http/https-only scheme
allowlist, 20 MiB cap (maxContentLength + post-download length recheck),
30s timeout, Content-Type→MIME resolution with URL-extension fallback,
filename derivation with canonical extension
- client.ts: rewrite uploadImage(pageId, url) as URL-only; drop the
local-file branch, imageMimeFromPath and the fs import; insertImage/
replaceImage now take a url
- index.ts: drop filePath, add required imageUrl to both tools; update
tool descriptions and SERVER_INSTRUCTIONS
- README: document the web-URL behaviour
Locators (edit_page_text `find`, insert_node `anchorText`) are matched
against the document's plain text, so a model-supplied locator carrying
markdown wrappers (**bold**, *italic*, `code`, [t](url)) or trailing emoji
never matched and the edit/insert failed. Add stripInlineMarkdown() and a
fallback: try the locator verbatim first (exact match wins, so literal
asterisks/underscores still work), and only on zero matches retry with a
markdown-stripped form. The ambiguity guard runs on the post-fallback count,
and `replace` / inserted node content are never stripped, so no formatting is
lost. Failed edits gain an atom-aware reason plus a bounded "closest block
text" hint; the insert_node "anchor not found" error now points at plain-text
anchors / anchorNodeId.
New packages/mcp/src/lib/text-normalize.ts (+ unit tests); wired into
json-edit.ts and node-ops.ts; tool descriptions updated. Tests: 212 pass.
edit_page_text (applyTextEdits) now matches at the inline-block level instead of
per text node, so a find/replace may cross bold/italic/link boundaries; the
replacement inherits marks from the unchanged common prefix/suffix via a diff
splice. Atom (non-text inline) slots can never be part of a match, making the
U+FFFC placeholder collision-safe, and inserted text never inherits an atom's
marks.
The edit batch is no longer all-or-nothing: applyTextEdits returns
{ doc, results, failed } and applies what it can; editPageText writes only on a
real change (no spurious history version for a no-op) and throws an aggregated,
actionable error only when nothing applied.
The AI-chat insert_node / patch_node / update_page_json tools now JSON.parse a
node/content argument that arrives as a string, matching the standalone MCP
server (this is what made insert_node fail under OpenAI tool calls).
Tool descriptions gain concrete ProseMirror examples and reflect the new
edit_page_text behavior. Adds/updates json-edit unit tests (183 pass).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- openai provider: use .chat() (Chat Completions) instead of the default callable
(Responses API), which gateways reject on multi-turn -> 400.
- updateAiProviderSettings: assemble settings.ai.provider via jsonb_build_object
with ::text-cast bound params + jsonb_typeof self-heal (postgres.js was
double-encoding it into an array; the ::text cast avoids 'could not determine
data type of parameter').
- chat agent: drop the hard maxOutputTokens cap (truncated complex tool calls);
keep a tiny cap only on the test-connection ping.
- testConnection + chat stream: surface the real provider error (statusCode+message)
to logs and the UI instead of generic masks; never log the API key.
- chat UI: typing indicator, incremental streaming render, tool 'running' status, Stop.
Also bundled (prior uncommitted ai-chat work):
- history 'AI agent' provenance badge; vector RAG (pgvector image + page_embeddings
+ AI_QUEUE indexer + space-scoped semanticSearch); external MCP servers backend
(@ai-sdk/mcp client, SSRF IP-pinning, encrypted headers, admin CRUD/Test);
yjs duplicate-instance fix via pnpm patch (single CJS instance server-side).
- Add reversible write tools to the per-user agent toolset (page create/update/
move/soft-delete; comment reply + resolve), exposed under the user's JWT and
enforced by Docmost CASL; no permanent/force delete (D3).
- Non-spoofable agent provenance: sign actor/aiChatId into the access and collab
tokens (TokenService), propagate via jwt.strategy onto the request, and set
pages.last_updated_source/last_updated_ai_chat_id on REST create/update/move and
comments.created_source/resolved_source/ai_chat_id.
- packages/mcp: add an optional getCollabToken provider (content-edit provenance)
and guard against empty tokens; service-account /mcp path unchanged.
Frontend:
- Admin 'AI / Models' settings section: provider/model/embedding/base URL, a
write-only API key field, system prompt, and Test connection.
- AI chat panel (useChat + DefaultChatTransport): conversation list, streamed
messages, tool-call action log and page citations; header entry point gated on
settings.ai.chat.
Compile-verified (server nest build + client tsc/vite); not yet live-tested.
Known gaps: history 'AI agent' badge (C3), vector RAG (D), external MCP (E);
chat tool-card citation links pending a fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WIP checkpoint of the gitmost AI-chat backend (plan stages A + B1 + B3a).
The agent acts under the requesting user's JWT (Docmost CASL enforces page
access); the external service-account /mcp endpoint is untouched.
LLM provider config (A2-A4):
- integrations/crypto: AES-256-GCM SecretBoxService (key derived from APP_SECRET,
per-record salt/iv; clear error on rotation instead of crashing).
- ai_provider_credentials table/repo/types: encrypted API key stored outside
workspace settings/baseFields, write-only (never returned by any endpoint).
- integrations/ai: per-workspace AI SDK v6 provider driver (openai/gemini/ollama),
admin-gated GET(masked)/PATCH(write-only key)/Test endpoints; settings.ai.provider
holds non-secret config incl. systemPrompt. Removed unused AI_* env getters (DB is
the single source of truth).
Chat module (A1, A5-A8):
- ai_chats/ai_chat_messages repos (workspace-scoped, soft-delete, tsv never selected).
- core/ai-chat: CRUD + POST /ai-chat/stream (Fastify hijack + AI SDK v6
pipeUIMessageStreamToResponse, abort on disconnect, persist user/assistant msgs).
- Agent loop: streamText + stepCountIs(8); read tools searchPages/getPage via a
per-request DocmostClient over loopback REST under the user's minted access token.
- Gate settings.ai.chat (+ 503 when provider unconfigured); buildSystemPrompt with a
non-removable safety/anti-prompt-injection framework. Per-user rate limit.
Per-user auth (B1):
- @docmost/mcp DocmostClient gains an additive getToken variant (carry a user JWT,
re-fetch on 401) and exports DocmostClient; the email/password service-account path
(external /mcp, stdio) is unchanged.
Agent-edit provenance backbone (B3a):
- Migration: pages/page_history (last_updated_source, last_updated_ai_chat_id) and
comments (created_source, ai_chat_id, resolved_source).
- Signed actor/aiChatId claim in the collab token; onAuthenticate propagates it,
onStoreDocument writes it with a sticky agent marker, saveHistory copies it.
Migrations auto-run on boot (additive). Write tools, frontend, RAG and external MCP
servers are not in this checkpoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the removed enterprise EE MCP (private apps/server/src/ee submodule,
license-gated /mcp route) with our docmost-mcp, vendored as an isolated ESM
workspace package and served by the server over HTTP — no enterprise license.
Backend:
- Add packages/mcp (@docmost/mcp): vendored docmost-mcp refactored into a
side-effect-free createDocmostMcpServer() factory (38 tools preserved),
stdio entry kept in stdio.ts, Streamable-HTTP session manager in http.ts.
- Add apps/server McpModule: @Post/@Get/@Delete('mcp') (served at /mcp via the
existing global-prefix exclude), @SkipTransform + reply.hijack to bridge raw
Fastify req/res into the SDK transport. The module dynamically imports the
ESM-only package from CommonJS via a Function-indirected import resolved with
require.resolve + file:// URL. Gated by the workspace ai.mcp toggle, a
service-account (MCP_DOCMOST_EMAIL/PASSWORD/API_URL) and optional MCP_TOKEN;
per-session idle eviction (MCP_SESSION_IDLE_MS).
- Drop the enterprise license check on mcpEnabled in workspace.service.
- Dockerfile: copy packages/mcp into the production image.
- .env.example: document MCP_DOCMOST_*, MCP_TOKEN, MCP_SESSION_IDLE_MS.
Frontend:
- Recreate the community "AI & MCP" workspace-settings panel (mcp-settings.tsx):
admin-only toggle on settings.ai.mcp with optimistic update, copyable
${APP_URL}/mcp URL; wired into workspace-settings page. Reuses existing i18n.
Fixes:
- Pin packages/mcp tiptap deps to 3.20.4 (matching the client) and inline
getStyleProperty, preventing a duplicate @tiptap/core@3.26.1 from leaking into
the client editor via pnpm shamefully-hoist (was breaking apps/client tsc).
