mcp had its OWN drifted copy of the converter (markdown-converter.ts ~900 lines,
docmost-schema.ts ~1270 lines, markdown-document.ts) — older than the shared
package, missing the git-sync fixes AND the #293 canon. This switches mcp's
converter CORE to @docmost/prosemirror-markdown, so mcp jumps straight to the
canonical format and the drift-generating second copy is gone.
- markdown-converter.ts / markdown-document.ts / docmost-schema.ts become thin
re-export shims of the package (convertProseMirrorToMarkdown, the docmost:meta
envelope, docmostExtensions + docmostSchema=getSchema(docmostExtensions)). The
mcp-only helpers clampCalloutType/sanitizeCssColor are preserved verbatim in
the schema shim (the package doesn't expose them via its barrel). ~2170 lines
of the drifted converter/schema bodies deleted.
- collaboration.ts drops its own ~360-line marked pipeline (preprocessCallouts,
bridgeTaskLists, extractFootnotes, the footnoteRef extension) and re-points to
the package's markdownToProseMirror, keeping markdownToProseMirrorCanonical and
all the yjs/collab write glue. footnote-lex/analyze doc comments updated (they
now describe advisory legacy-syntax diagnostics, not an importer).
Schema parity verified: the package schema is a strict SUPERSET of mcp's old
schema — every node and attr mcp declared is present (the package only adds
status/pageEmbed/transclusion*/subpages.recursive/etc.), so nothing is silently
dropped on the switch. The switch actually FIXES two pre-existing mcp data-loss
bugs its own tests documented: htmlEmbed and pageBreak now round-trip (were
dropped by the old mcp converter).
Footnotes: the package assembles inline ^[body] footnotes on import (sequential
fn-N ids, identical bodies merged), so mcp's canonicalizeFootnotes is now an
idempotent no-op after it (verified). Legacy reference footnotes [^id]/[^id]:
are inert literal text (canon #2 no-backward-compat) — lossless, the text
survives verbatim.
Build hygiene: packages/mcp/build/ is now gitignored and untracked, matching the
git-sync/prosemirror-markdown convention (private package, rebuilt in CI/Docker,
so src and prod can never silently diverge). This also removes a dead untracked
build/_vendored_editor_ext/ artifact that a broad `git add` would otherwise
commit.
Dependency: packages/mcp/package.json gains @docmost/prosemirror-markdown
(workspace:*); pnpm-lock.yaml gets the matching link importer (mirrors git-sync).
mcp tests updated deliberately to the canonical forms (highlight ==, math $…$,
image <!--img-->, drawio/media discriminators, subpages/pageBreak
comments, textAlign, inline ^[…] footnotes) with strict assertions; 4 structural
safety-net round-trip tests added.
mcp: node --test 454 passed; tsc clean. package: 657 passed. git-sync: 268 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit of all 41 tool descriptions against the actual implementation found
factually wrong or misleading texts:
- list_comments claimed '(paginated)' — it takes only pageId and returns ALL
comments in one call (internal pagination); now also states that RESOLVED
threads are included and how to filter them. In-app twin synced.
- search claimed the limit default is 'applied by the client' — the client
deliberately omits it so the SERVER applies its default.
- create_page's '(automatically moves it to the correct hierarchy)' said
nothing useful — now documents parentPageId nesting semantics; move_page
drops the stale 'essential for organizing pages created via create_page'.
- share_page now warns the page becomes accessible to ANYONE with the URL.
- get_page (both transports) now explains inline <span data-comment-id> tags
are comment anchors (incl. resolved) — markup, not page text.
- patch_node/delete_node/insert_node pointed only at the expensive page-JSON
view for block ids — now route through the cheap page outline first.
- docmost_transform marks 'Примечания переводчика' as the DEFAULT
notesHeading, overridable for non-Russian pages.
Checks: @docmost/mcp tests 450/450 (incl. the server-instructions guard);
server ai-chat-tools spec 20/20; mcp build/ artifacts rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The intent-routing guide had rotted: 17 of 41 registered tools were absent
(get_outline, get_node, the whole table_* family, search, stash_page, sharing,
page lifecycle), and two tips were actively harmful — 'read block ids via
get_page_json' told agents to pull the whole ~100KB document when get_outline
exists precisely to grab ids cheaply, and 'table cell -> patch_node by
attrs.id' dead-ends because table nodes carry no attrs.id.
- Rewrite SERVER_INSTRUCTIONS as intent clusters (READ / EDIT / PAGES /
COMMENTS / HISTORY) covering every tool except get_workspace; add safety
notes (share_page = PUBLIC, delete_page = soft) and a comment-anchor
markup warning for get_page.
- delete_page tool description: state SOFT delete / restorable explicitly.
- MAINTENANCE RULE comments at both registration sites (index.ts,
tool-specs.ts) + an AGENTS.md convention bullet: adding/renaming/removing
a tool REQUIRES updating the guide.
- New guard test (test/unit/server-instructions.test.mjs): extracts every
registered tool name from source and fails when one is not mentioned in
the shipped SERVER_INSTRUCTIONS (word-boundary match, so get_page can't
hide behind get_page_json); EXCEPTIONS list is itself validated against
the registry. SERVER_INSTRUCTIONS exported for the test.
Tests: @docmost/mcp 450/450 (448 + 2 new).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- editorial roles (ru/en): proofreader and line editor attach suggestedText
replacements to targeted fixes; fact-checker ALWAYS attaches the ready
correction for [Incorrect] verdicts; structural editor and narrator get a
light-touch rule for in-place rewordings; role versions bumped and the
content-hash lock refreshed
- MCP SERVER_INSTRUCTIONS: route 'propose a concrete text fix for one-click
human approval' to create_comment with suggestedText (unique-selection
reminder); build/ artifacts rebuilt
- AI-chat SAFETY_FRAMEWORK: mention the comment-suggestion capability so the
default assistant offers ready fixes instead of only describing changes
Checks: catalog check.mjs OK; @docmost/mcp tests 448/448; server
ai-chat.prompt spec 28/28.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
F1 [blocking]: a suggestion whose anchor matched via normalization could never
be applied (spurious 409). The comment mark lands on the doc's ACTUAL text
(Docmost auto-converts to typographic quotes/dashes/nbsp), but the stored
selection — used as expectedText at apply — was the raw ASCII agent input
(+substring(0,250)). So replaceYjsMarkedText's strict joined!==expectedText
always failed and threw "text changed" though nobody edited. Fix: new pure
getAnchoredText(doc, selection) reconstructs the exact raw doc substring the mark
covers (slicing identical to spliceCommentMark); on the suggestion path
client.createComment stores THAT as selection, so expectedText equals the marked
text and apply returns applied:true. Live anchoring still uses the raw agent
selection (normalization still finds the anchor). Truncation raised 250->2000
(+ DTO @MaxLength(2000)) so the anchored substring is never cut below the mark
span. Ordinary comments unchanged. AI-chat shares client.createComment, so
covered. Regression tests: getAnchoredText raw-vs-ASCII; create payload selection
is the typographic substring; apply with typographic expectedText -> applied.
F2 [blocking]: added comment.controller.spec.ts pinning that validateCanEdit runs
before applySuggestion (Forbidden -> applySuggestion never called; happy path ->
called; missing comment -> 404 without authorizing).
MCP 448 pass; server comment+yjs 54 pass. MCP build/ rebuilt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents can attach a suggested replacement when creating an inline comment, via
both the MCP create_comment tool and the AI-chat createComment tool.
Because applying a suggestion edits the EXACT anchored text, an ambiguous anchor
would let Apply corrupt the wrong occurrence. So when suggestedText is set the
selection must occur EXACTLY ONCE:
- new countAnchorMatches(doc, selection) counts occurrences across all blocks
(same normalization/traversal as canAnchorInDoc), counting occurrences (2 in
one block => 2) — stricter than block-count, never under-counting distinct
occurrences (false-unique is the dangerous direction).
- client.createComment gains suggestedText: a pre-check (getPageJson +
countAnchorMatches: 0 => not-found, >=2 => ambiguity error) before create, and
an AUTHORITATIVE live check inside the anchoring mutation that recomputes on the
live doc and, if != 1, aborts and rolls back the just-created comment (reusing
the existing safeDeleteComment "anchor not found" path). Ordinary comments keep
first-occurrence behavior unchanged.
- suggestedText is rejected on a reply or without selection in all three layers
(MCP handler, MCP client, AI-chat tool), mirroring the server DTO/service.
- filterComment surfaces suggestedText/suggestionAppliedAt/suggestionAppliedById.
- DocmostClientLike.createComment signature updated. MCP build/ rebuilt.
Tests: countAnchorMatches (0/1/N, within/across/nested block, span nodes,
quote normalization); createComment (ambiguous refused pre-create, reply and
no-selection rejected, unique succeeds and forwards suggestedText, filterComment
surfaces it); ai-chat schema accepts suggestedText. MCP 443 pass; ai-chat 601 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The same tool metadata (zod schema + model-facing description) was hand-duplicated
between the standalone MCP server and the in-app AI-chat agent, so every tweak had to
land in two places and copies drifted (a materialized parity bug). The shared
transport-agnostic registry (packages/mcp/src/tool-specs.ts) already de-duplicates 14
tools; this migrates two more genuinely-identical ones — patch_node/patchNode and
insert_node/insertNode. The canonical description is a strict SUPERSET of both originals
(keeps MCP's "without resending the whole document" + table-structure/anchor guidance
AND the in-app "reversible via page history" / "exactly one of anchorNodeId or
anchorText" framing — no model-facing guidance dropped); the schema is identical (the
in-app side just gains MCP's .min(1) on ids, a safe tightening). Each transport keeps its
own execute/auth wrapper, and the in-app parseNodeArg node-arg normalization is unchanged.
The three table tools are intentionally NOT merged (a real param-name divergence:
table vs tableRef) — documented on both sides. Other per-transport divergences
(search/share/create_comment/transform/list_pages) are left separate with a short comment
explaining why (the issue asked to flag these as intentional). DocmostClientLike stays a
hand-mirror (the ESM/CJS boundary blocks a compile-time type import; a runtime drift-guard
already pins it). Also fixes a latent contract-spec bug: derive `required` from
`instanceof z.ZodOptional` (matches the emitted JSON schema) instead of `isOptional()`,
which wrongly reported z.any() fields as optional.
Partially addresses #294.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reviewer round 1 on the #260 collab-doc-name fix:
- F1: replaceImage is the one path where the resolved UUID gates BOTH the
collab-doc open AND the per-page mutex key (withPageLock(pageUuid)). Add a
deterministic test to resolve-page-id-collab-doc-name.test.mjs: it gates
/files/upload so replaceImage parks mid-upload holding its lock, asserts the
doc opened as page.<uuid> (never page.<slug>), and probes the SHARED
page-lock chain — a withPageLock(UUID) probe must stay blocked while
replaceImage holds it (with a free-key probe as a non-vacuity guard). The
test fails if the lock key is reverted to the slugId (verified).
- F2: drop the dead `pageIdCache.set(uuid, uuid)` — resolvePageId returns on
the isUuid() short-circuit before the cache is ever read with a uuid key, so
only slugId->uuid entries are stored/read. Comment corrected to match.
MCP suite 430/430, tsc 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real root cause of the silent MCP edit loss: the web editor always opens the
collaboration document by the page UUID (`page.${page.id}`), but the MCP
opened it by the agent-supplied id — usually a slugId — so `page.${pageId}`
became `page.<slugId>`. For one DB page that is TWO independent Yjs documents;
both persist to the same `pages` row (findById/updatePage resolve id or
slugId), so the human tab's debounced store overwrites the agent edit
(last-store-wins) — gone after reload, never shown live. The slugId doc also
made the server's transclusion sync + embedding reindex throw Postgres 22P02.
Fix:
- MCP (primary): resolvePageId(pageId) returns the canonical UUID — a UUID
short-circuits with no network call, a slugId resolves once via getPageRaw
and is cached both ways. Every collab-write path (mutatePageContent /
updatePageContentRealtime / replacePageContent and the mutate/replace/
unlocked seams) now opens by the resolved UUID, so the MCP and the editor
share ONE Yjs doc. replaceImage's whole-operation page lock also keys on the
UUID so it serializes against the other (now-UUID-keyed) writes.
- Server (defense + kills the 22P02 noise): onStoreDocument passes the resolved
page.id — not the raw doc-name id — to syncTransclusion, the embedding queue,
the mention-notification job, addContributors, and the in-tx history read.
Content store and the empty-guard are untouched.
Tests: a new MCP test stands up a real Hocuspocus server and asserts a slugId
input opens `page.<uuid>` (never `page.<slugId>`), with UUID short-circuit and
single-resolve caching; the server spec asserts the side-effects receive the
UUID for a `page.<slugId>` doc. closes#260
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
F1 (data loss): packages/mcp keeps its own copy of the document schema
(AGENTS.md), and the spoiler mark was only added to editor-ext + the server
tiptapExtensions, so a doc with a spoiler silently lost the mark through /mcp.
Add a local Spoiler mark to docmostExtensions (span[data-spoiler] parse,
data-spoiler="true"+class render) and a case "spoiler" in markdown-converter
emitting the same <span data-spoiler="true">…</span> as the editor-ext turndown
rule; add an MCP json->md->json round-trip test. Regenerated build/lib output.
F2: add the #259 inline-spoiler entry to CHANGELOG [Unreleased] Added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract pure extractAuthTokenFromSetCookie from performLogin (behavior-identical)
so cookie parsing is unit-testable without a network login. Add round-trip
coverage for media attrs (width/height/align/drawio/escaping) the existing
suite omitted; applyAnchorInDoc selection/ambiguity/atom-break cases; and a
cross-copy drift guard proving the vendored editor-ext recreate-transform and
the @fellow npm copy used by diff.ts emit identical steps (apply(diff)==target).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The comment referenced markdownToHtml, which does not exist in the mcp
package; the import path is marked.parse + generateJSON (which runs the
image extension's parseHTML). Describe the actual step and regenerate the
build artifact in sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stock @tiptap/extension-image carries no caption attribute, so
markdownToProseMirror through docmostExtensions dropped the
data-caption the client emits, breaking the lossless claim. Extend the
Image node (mirroring editor-ext image.ts and the nearby Highlight
extend) to parse/render data-caption. Rebuilt build/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mandatory (test-coverage):
- internal-file-urls.test: pin the SSRF/traversal ACCEPT path of
resolveInternalFilePath (the sole guard for content-controlled `src`): an
absolute/protocol-relative URL has its foreign host dropped and only an
/api/files/ pathname survives (http://evil.com/api/files/x/y.png -> /files/x/y.png),
while a host-dropped path that escapes /api/files/ (https://evil.com/api/auth/whoami)
or a backslash-traversal (/api/files\..\auth\whoami) is rejected. Locks the
behavior so a future prefix-only refactor cannot silently open a bypass.
Suggestions:
- index.ts: the stash_page MCP tool now returns structuredContent
{ uri, sha256, size, images } alongside the resource_link, so the MCP output
matches the documented shape (clients get the blob's sha256/ETag and the
mirror counts, not just the link). No outputSchema registered. Rebuilt build/.
- new stash-page-mcp-result.test: server round-trip via InMemoryTransport asserts
both the resource_link and the structuredContent mirror.
- internal-file-urls.test: cover the new URL parse-failure catch branch
(http://[ -> "Invalid internal file src").
- environment.service.spec: assert getPositiveIntEnv warns once per key and
independently across keys (the invalidPositiveIntWarned dedup).
Tests: packages/mcp 383 pass; apps/server sandbox/environment/mcp 235 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security (must-fix):
- sandbox.controller: the anonymous GET /api/sb/:id response now sets
X-Content-Type-Options: nosniff, a restrictive CSP, and Content-Disposition=
attachment for any mime outside a raster-image allowlist (png/jpeg/gif/webp/
avif). entry.mime is attacker-controlled, so an evil.svg/evil.html could
otherwise execute script inline on the Docmost origin (stored XSS). Mirrors
the public attachment route's hardening.
Stability:
- client.stashPage: reconcile mirrors AFTER the final document put, not only
before it. The doc blob is the newest entry and FIFO eviction drops the
oldest = this stash's own images, so the stored doc could reference an
evicted blob (consumer 404) and over-report images.mirrored. A bounded loop
now reverts doc-put-evicted mirrors, drops the stale doc blob, and re-puts
until stable. Regenerated packages/mcp/build/.
- sandbox.controller: emit Cache-Control on the 304 branch too (ttlSeconds is
computed before the conditional check).
Docs:
- Bump the MCP tool count 39 -> 40 across all READMEs and AGENTS.md (the
registry now exposes exactly 40 tools).
Refactor:
- SandboxStore.asSink() centralizes the {put,has,evict} sink + uri<->id
mapping; the embedded-MCP and in-app agent-tools wiring sites share it.
Tests:
- security headers (inline vs attachment, nosniff, CSP), 304 Cache-Control,
putAndLink URL form, has()/remove(), asSink() round-trip, getSandboxPublicUrl
(trailing-slash trim + APP_URL fallback), and a stash test where the doc put
itself evicts a mirrored image.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security:
- stash_page: reject path-traversal / percent-encoded srcs before the authed
loopback fetch (resolveInternalFilePath), closing an SSRF/exfiltration hole
where a crafted node.attrs.src could read an arbitrary internal GET endpoint
into the anonymous sandbox.
Stability:
- stash_page: revert + recount mirrors FIFO-evicted by a later put in the same
stash (no dangling sandbox refs, honest images.mirrored/failed); free image
blobs if the final document put throws.
- Reject/clamp non-positive SANDBOX_TTL_MS to the 1h default (warn once).
- Log mirror failures unconditionally (console.warn, no blob bodies).
Cleanup / architecture:
- Remove dead expiresAt from SandboxPutResult.
- Centralize the /api/sb route in SANDBOX_ROUTE_SEGMENT/SANDBOX_API_PATH and
move URL composition into SandboxStore.putAndLink; drop the duplicated sink
closures and the now-unused EnvironmentService injection from McpService and
AiChatToolsService.
- Un-export isInternalFileUrl; document the process-local (instance-bound)
sandbox limitation in the tool description and .env.example.
Docs/tests:
- README/README.ru: 38 -> 39 tools + stash_page entry.
- Add traversal/normalize/recursion unit tests, stash self-eviction +
doc-put-throw + empty/octet-stream mock tests, controller If-None-Match
(wildcard/weak/list) + Cache-Control tests, and SANDBOX_TTL_MS validation
tests. Regenerate packages/mcp/build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an ephemeral, process-local blob store so the in-app agent (and the
embedded MCP) can hand a large page document and its images to an external
consumer WITHOUT routing the bytes through the model context or Docmost auth.
- SandboxStore (@Injectable singleton): Map<uuid,{buf,mime,sha256,expiresAt}>
in RAM only. put() picks a per-blob cap by mime (image vs doc), enforces a
total-bytes RAM guard with oldest-first eviction, and stamps a TTL; get()
lazily expires. sha256 computed at put() doubles as the strong ETag. An
unref'd sweep interval clears expired entries and is cleared on destroy.
- GET /api/sb/:uuid anonymous controller: serves raw bytes with Content-Type,
Content-Length and ETag=sha256; 404 on missing/expired/non-UUID (anti-
traversal), 304 on a matching If-None-Match. No tokens, no 401 — the
capability is the unguessable UUID + short TTL + TLS. Auth-exempt the same
way as /api/files/public (no JwtAuthGuard) plus an /api/sb entry in main.ts's
workspace-resolution preHandler so a remote consumer with no workspace host
is not rejected.
- stash_page tool in both layers (MCP resource_link + in-app {uri,size,sha256,
images}). client.stashPage serializes the get_page_json shape, mirrors every
INTERNAL file/image src (type-agnostic, covers drawio/excalidraw/video/file)
into the sandbox under Docmost auth and rewrites src to the sandbox URL;
external http(s) srcs are left untouched; dedup by src; a failed image fetch
is counted, never aborts the doc.
- SANDBOX_PUBLIC_URL / SANDBOX_TTL_MS / SANDBOX_MAX_BYTES /
SANDBOX_MAX_IMAGE_BYTES / SANDBOX_MAX_TOTAL_BYTES wired through the
environment service + validation + .env.example.
- SandboxModule (@Global) provides the shared store to the controller,
McpService and AiChatToolsService (same instance for put and get).
Tests: SandboxStore (round-trip, sha256, TTL lazy + sweep, caps, eviction),
SandboxController (200+ETag+CT+CL, 404 missing/expired/non-UUID, 304), and a
mock-HTTP stashPage test (mirror+rewrite internal, keep external, dedup, failed
image counted, returns only a link). Interoperates with the vvzvlad/habr-mcp
consumer's anonymous-GET + sha256-ETag + resource_link contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a visible caption (<figcaption>) under images, editable from the
image bubble-menu and persisted across all formats: native Yjs/JSON,
HTML export, and Markdown.
- image node: new plain-text `caption` attribute (parse/render
`data-caption` on <img>, emitted only when set) + `setImageCaption`
command. The node stays an atom; the schema shape is unchanged, so the
server's generateHTML/generateJSON path round-trips it for free.
- resize node-view: re-parent the resizable wrapper into a <figure> and
render the caption in a <figcaption> BELOW it, outside nodeView.wrapper
(so onCommit's offsetHeight measurement and the left/right resize
handles still cover the image only). This path also drives read-only /
share rendering. React placeholder view renders the caption too.
- bubble-menu: new useCaptionControl panel modeled on useAltTextControl
(own icon, Caption strings, softer sanitizer, ~500 char limit).
- markdown lossless round-trip: a captioned image is emitted as a raw
<img data-caption> wrapped in a block <div> (same trick as <video>) in
both the editor-ext turndown rule and the MCP converter; caption-less
images stay clean . Import restores the caption via the
shared markdownToHtml + parseHTML.
- styles + i18n keys; tests for the schema attr round-trip, markdown
round-trip (editor-ext) and the MCP converter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review #6 (approve-with-comments) follow-ups:
1. canonicalize step 7 now strips bare footnoteDefinitions at ANY depth
(stripFootnoteDefinitionsDeep), not just footnotesList, in BOTH copies. A
definition hand-authored outside a list (e.g. nested in a callout via a
raw-JSON write path) was left in place while a copy was also added to the
rebuilt list -> duplicate, idempotent, self-perpetuating. Runs only in the
rebuild path (after the lists are stripped); the fast-path / placement-keep
branch is untouched. Added a shared-corpus case (bare def nested in a callout)
to pin it in both mirrors.
2. markdown-clipboard: removed the dead top-level footnoteReference check in
canonicalizePastedFootnotes (an inline atom is never a top-level slice child;
only the descendants scan can find it).
Test coverage:
4. New MCP binding tests (full-doc-write-canonicalize.test.mjs): update_page_json
and copy_page_content canonicalize the persisted full doc, asserted via a new
`replacePage` seam (symmetric to the existing `mutatePage` seam) so no live
collab socket is needed. Routed both writers through the seam.
5. New server spec (file-import-task.service.footnote-canonicalize.spec.ts): the
zip-import path (processGenericImport) canonicalizes footnotes — real
markdown->HTML->JSON via a real ImportService over a temp-dir .md file, DB trx
stubbed to capture the persisted page content. FileImportTaskService had no
spec before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix:
- insertInlineFootnote could glue a footnoteReference inside an EXISTING
definition (nested footnotesList, or a bare footnoteDefinition with no list
wrapper), which canonicalize then dropped as an orphan — silently losing the
definition's prose. Now: (a) the body/notes boundary is computed from the first
top-level block that IS or CONTAINS (recursively) a footnotesList/
footnoteDefinition, not just a top-level list; and (b) the insertNodesAfterAnchor
core skips footnotesList/footnoteDefinition subtrees entirely (skipSubtreeTypes),
so an anchor whose only match is inside a definition -> inserted:false (clean
abort, no write). Added tests: nested-definition, bare-definition, and
body-before-nested-list-still-inserts.
- editor-ext footnote-canonicalize header listed `markdownToProseMirror` among the
canonicalizing MCP paths; it is the NON-canonicalizing primitive. Replaced with
`markdownToProseMirrorCanonical` (+ note that the plain primitive is for comment
bodies) and added copy_page_content.
- Client paste: canonicalizePastedFootnotes now skips a definitions-ONLY paste
(no footnoteReference anywhere) — canonicalizing it would strip the
reference-less list and yield an EMPTY paste. Added a test.
Suggestions:
- docmost_transform now runs validateDocStructure/validateDocUrls on the RAW
transform output BEFORE canonicalizeFootnotes (mirrors updatePageJson), so a
too-deep doc gives the intended max-depth error instead of a stack overflow.
- docmost_transform tool description now states the RESULT is footnote-canonical
(dryRun diff may show tidy-ups; idempotent after first run).
- insertFootnote: dropped the dead `result ? … : undefined` ternaries and the
`as any` casts (result is always set by the time we return; the not-found path
throws and aborts mutatePage). `const r = result!;`.
Tests / architecture:
- Added a LIVE-plugin golden case: the real footnoteSyncPlugin leaves a list with
non-empty content after it in place, and canonicalize agrees (placement parity
is now a driven property, not a hand-set expected).
- Added generateFootnoteId uuidv7 shape + uniqueness test.
- Item 9: added the ENFORCEMENT-RULE comments at the server parseProsemirrorContent
and the MCP canonicalizer header (any NEW full-doc persist path MUST canonicalize;
fragments/append/prepend and comment bodies MUST NOT). Kept per-call-site over a
brittle grep CI test (the replace-vs-fragment + comment-vs-page nuance makes a
single wrapper unsafe).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix (REAL DATA LOSS):
- markdownToProseMirror is reused for COMMENT bodies (createComment/updateComment).
It unconditionally canonicalized, so a comment carrying a standalone footnote
definition ([^1]: text with no matching reference) had its whole footnotesList
stripped (referenceIds.length===0 -> stripFootnotesListsDeep) — the text
vanished. Fix: markdownToProseMirror no longer canonicalizes (content-preserving
primitive); a new markdownToProseMirrorCanonical wraps it for the PAGE write
paths (markdown import via importPageMarkdown, update_page markdown via
updatePageContentRealtime). Comment callers keep the non-canonicalizing
primitive. Updated the now-false header comment and added create/update-comment
inline notes. Added collaboration tests: comment path PRESERVES a reference-less
definition; page path still drops it AND still reorders real footnotes. Updated
the page-import canonicalization test to use the canonical variant.
Suggestions / architecture:
- #2: collapsed transforms.footnoteDefinition onto the shared
makeFootnoteDefinition factory (adds only the inner paragraph block id); kept
the dependency direction transforms -> footnote-authoring (no circular import,
mirror stays pure).
- #3: confirmed docmost_transform auto-canonicalization is documented (inline
comment, tool description, CHANGELOG) — no code change.
- #4: copyPageContent is a FULL-document write (replacePageContent of a
type:"doc"); added a defensive canonicalizeFootnotes pass (no-op on
already-canonical source).
- CHANGELOG entry refined to list the FULL-document write paths (incl.
copy_page_content) and to state canonicalization is NOT applied to comment
bodies.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix:
- REAL BUG: insertInlineFootnote could splice a footnoteReference (inline atom)
into a codeBlock or an existing footnoteDefinition, persisting a schema-invalid
doc (insert_footnote skips validateDocStructure). Now the search is bounded to
the BODY (before the first footnotesList) and the insertNodesAfterAnchor core
refuses textblocks that can't hold the atom (codeBlock); when the only match is
in such a place the insert returns inserted:false and the write aborts cleanly.
Reachable via docmost_transform too. Added codeBlock / definition / fall-through
tests.
- Fixed the deepEqualJson doc comment in both copies: arrays are order-SENSITIVE
(correctness depends on it), only object keys are order-insensitive.
- README.ru.md MCP tool count 38 -> 39 (lines 36/47/63), matching README.md/AGENTS.
- CHANGELOG [Unreleased] Added entry for insert_footnote + server-side footnote
canonicalization on non-editor write paths (#228).
Suggestions:
- canonicalize step 5/7 now strips footnotesList at ANY depth (both copies), so a
schema-valid list nested in a callout/blockquote can't leave duplicate defs.
- Exclude the test-only footnote-corpus.ts fixture from the editor-ext build
(tsconfig), so it no longer ships in dist/.
- Removed the duplicate manual canonicalize cases from the MCP unit test (the
shared corpus covers them via full deepEqual); kept idempotence + immutability.
- insertInlineFootnote dedup key now keys off the inline array directly
(footnoteContentKey({ content: inline })) instead of a throwaway node.
Tests / architecture:
- New client-wrapper test (#9): overrides a small mutatePage seam to assert the
not-found path throws and persists NOTHING, and the success path shapes
footnoteId/reused/message/verify and writes the right content. Fixed the
misleading comment in footnote-write.test.mjs.
- B: cross-copy corpus parity guard test (loads both corpora, asserts deep-equal)
so a typo in one copy can't pass both suites green.
- A: declined — the full-vs-fragment decision lives at the call site, so a
prepareDocForPersist wrapper would be a bare alias for canonicalizeFootnotes;
kept the existing per-call-site comments instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Must-fix:
- Move canonicalizeFootnotes OUT of parseProsemirrorContent. It now runs only
on FULL writes (createPage, updatePageContent operation==='replace'), never on
an append/prepend fragment (a fragment would lose definition-only footnotes or
synthesize a bogus empty list). Add a server binding spec.
- Match the live plugin's list PLACEMENT: a single already-canonical
footnotesList is left exactly where it sits (the plugin never repositions a
sole correct list), so the first write no longer reorders content that follows
the list. Applied to BOTH the editor-ext copy and the MCP mirror; pinned by a
shared golden corpus case with content after the list.
- Fix MCP tool count 38 -> 39 (README x3, AGENTS.md) and the transformJs param
help (add canonicalizeFootnotes/insertInlineFootnote).
Simplifications:
- Remove the dead duplicate re-id mechanism (deriveFootnoteId/suffix/occurrence)
from the PURE canonicalizer in both copies — references are never renamed, so
the derived ids were never requested; first-wins-drop is the real behaviour.
This also makes the editor-ext footnote-util note about "no cross-package copy"
true again.
- Remove the sentinel round-trip in insertInlineFootnote: a generalized
insertNodesAfterAnchor core inserts the footnoteReference node directly.
- Drop the redundant per-definition deep clone in step 4 (shallow id-normalizing
copy; out is already deep-cloned).
Docs / architecture:
- Correct the editor-ext copy's "It exists because…" header to its real
consumers (server import, page.service create/update, client paste).
- Note markdownToProseMirror reuse for create/update comment in collaboration.ts.
- A: shared golden JSON corpus exercised by BOTH the editor-ext copy and the MCP
mirror (footnote-corpus.ts / .mjs) so "the two copies behave identically" is
checkable.
- C: split the MCP canonicalizer into a pure mirror + footnote-authoring.ts.
- B: import services persist via a different path, so left one-line consolidation
comments at the call sites rather than folding (does not fall out cleanly).
Tests: insertFootnote wrapper guards + docmost_transform dryRun auto-canonicalize
(MCP mock), page.service create/update + append/prepend binding (server jest),
shared corpus incl. nested-container reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make footnotes author-inline: the agent/tool inserts a footnote at its point
of use (anchor + text) and the numbering plus the bottom list are DERIVED
deterministically server-side. The agent has no access to footnotesList and
cannot desync — out-of-order lists, orphan definitions, and raw trailing
[^id] blocks become structurally impossible.
editor-ext:
- canonicalizeFootnotes(docJSON) -> docJSON: a pure, EditorView-free port of
footnoteSyncPlugin's end-state. Distinct reference ids in document order are
the source of truth; exactly one trailing footnotesList holds one definition
per referenced id in reference order (reusing the existing node or
synthesizing an empty one); orphans dropped; duplicate definitions resolved
deterministically (first wins, never lost); idempotent.
- Unit tests + a golden parity suite: on every editor-reachable steady state
the live footnoteSyncPlugin's JSON is a canonicalize no-op (byte-for-byte
parity), and the canonicalizer additionally repairs the out-of-order list a
non-editor write produces.
mcp:
- footnote-canonicalize.ts: behavioural mirror of the editor-ext canonicalizer
(the MCP package is intentionally decoupled from the editor barrel, like
footnote-lex/docmost-schema), plus footnoteContentKey for content dedup.
- Auto-canonicalize on EVERY write path: markdownToProseMirror (fixes import
ordering), update_page_json, and after every docmost_transform. Idempotent,
so it is a no-op when footnotes are already canonical.
- insert_footnote tool + insertInlineFootnote: anchor + markdown text -> a
mark-safe footnoteReference and a content-dedup'd definition; the list and
numbering are derived. Same-content footnotes reuse one number/definition.
- canonicalizeFootnotes + insertInlineFootnote exposed as docmost_transform
sandbox helpers.
Tests: editor-ext 157 green; MCP 325 green; server + client tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8-point multi-aspect review of the batch PR; security/regressions were clean.
1. Lease leak: the #180 reorder moved `toolsFor` (which leases external MCP
clients, refCount+1) ahead of buildSystemPrompt + forUser, but the only
release (closeExternalClients) was bound to the streamText callbacks. A throw
in between leaked the lease (refCount stuck, undici sockets held until
restart). Define closeExternalClients right after the lease and wrap
buildSystemPrompt+forUser in try/catch that closes-then-rethrows.
2. Cover the patch_node/delete_node dup-id refusal (#159#6): extract the guard
into a pure `assertUnambiguousMatch` (node-ops) and unit-test 0/1/>1.
3. Regress the body-before-title order (#159#10): mock-HTTP test (collab fails
fast against a server with no WS upgrade) asserts /pages/update (title) is
NEVER posted when the body write fails — for updatePage AND updatePageJson.
4. CHANGELOG [Unreleased]: #180, #168 (Added); #163 (Fixed).
5. Add the missing en-US i18n keys (Back to references / {{label}}).
6. Drop the duplicate content/empty/blank cases in ai-chat.prompt.spec.ts (they
repeat the buildMcpToolingBlock unit tests); keep only sandwich placement +
both-safety-copies.
7. CI Postgres pg16 -> pg18 (match docker-compose).
8. jsonb decode seam: shared `parseJsonbValue(value, guard)` in database/utils.ts
holds the legacy double-encoding self-heal in one place; parseToolAllowlist /
parseModelConfig keep only a type-guard.
Verified: server build + 124 unit + 15 integration; mcp 311; prettier clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
updatePage (markdown) and updatePageJson wrote the title via REST FIRST, then
the body via collab. If the body write failed (e.g. a collab persist timeout),
the page was left with the NEW title over its OLD body — a split-brain the tool
reported as an error but never repaired (red-team finding #10).
Reorder both: write the body first, and only set the title after the body has
persisted. Now a body-write failure leaves the title untouched (no split-brain).
A title write failing after a successful body is rarer (REST is fast) and leaves
correct content under a stale title — the strictly lesser inconsistency — which
is the same trade-off the issue's "atomic, or roll back the title" intends,
without the fragility of a rollback write that could itself fail.
No unit test: both paths require a live collab provider and the suite has no
provider mock; the change is a pure reordering. All 306 mcp tests still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Docmost duplicates block ids on copy/paste, and copyPageContent writes the
source document verbatim with the same ids. `patchNode`/`deleteNode` address a
block by `attrs.id` via replaceNodeById/deleteNodeById, which act on EVERY node
sharing the id — so a single patch_node/delete_node could silently
replace/remove multiple unrelated blocks with no signal to the model
(red-team finding #6).
Guard both write paths: when more than one node matches the id, skip the write
entirely (the transform returns null -> no mutation) and throw a clear
"ambiguous id — N nodes share it" error so the model re-targets with a more
specific anchor. Only an unambiguous single match is written; the 0-match and
1-match behavior is unchanged.
The duplicate-count basis is covered by node-ops.test.mjs (replaceNodeById /
deleteNodeById report count===2 for a 2-duplicate doc). The end-to-end guard
is not unit-tested because patchNode/deleteNode require a live collab provider
and the test suite has no provider mock.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`mutateLiveContentUnlocked` — the write path used by `replaceImage` — still
did the pre-#152 destructive write (delete the whole fragment + applyUpdate a
fresh Y.Doc), discarding every Yjs node id. y-prosemirror anchors the editor
selection to those ids, so an open editor's cursor snapped to the document
end on every image swap, exactly the #152 jump that the main write path no
longer causes.
Switch it to the same `applyDocToFragment(ydoc, newDoc)` structural diff
(updateYFragment) as the main path, so unchanged nodes keep their ids and the
live cursor stays put. It runs its own atomic transact, so the old explicit
transact/delete is gone; the now-unused docmostExtensions import is dropped.
Regression tests (cursor-stability suite): a sibling paragraph's
RelativePosition survives a top-level image src/attachmentId swap, and an
image nested in a callout, matching the shapes replaceImage produces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the merged #166/#169. Addresses the second review pass (comment
1227):
- footnoteWarnings plumbing: extract a single `footnoteWarningsField(markdown)`
helper (footnote-analyze) and use it at all three call sites (create_page,
update_page, import_page_markdown) so the field is attached identically.
- New unit test footnote-warnings-import.test.mjs pins the contract that was
uncovered: the field is present on problems / omitted on clean input, and the
IMPORT path analyzes the BODY after the docmost:meta / docmost:comments blocks
(a footnote-like token inside those JSON blocks must NOT warn; a real body
marker must). Tested via the same pure composition the importer uses
(footnoteWarningsField(parseDocmostMarkdown(full).body)) — no collab socket
needed; a regression that analyzed fullMarkdown or skipped the body split would
now go red.
- footnote.marked.ts: correct the stale module header — it claimed "only
definitions that have a matching reference are emitted", which was never true
(orphan defs are emitted; the editor sync plugin reconciles). Now describes
first-wins + reuse + sync reconciliation.
- derive-id golden test: rename the describe from "(cross-package drift guard)"
to "(deterministic-scheme pin)" — there is no second package to drift against.
editor-ext 129, MCP 304 (+3), client+server tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- footnote-sync: remove the now-dead `refReids` (CollisionPlan field, local,
return, the 6a consumer loop) — references are never re-id'd under reuse, so it
was dead structure on the hot reconciliation path. Rewrite the stale comments
(plugin header, step 0, refOccurrences field) that still described the old
"duplicates re-id'd so both survive" model to the reuse model.
- Shared footnote lexer: new packages/mcp/src/lib/footnote-lex.ts
(lexFootnoteLines + forEachFootnoteReference). extractFootnotes (collaboration)
and analyzeFootnotes now consume the SAME fence-aware lexer, so "the analyzer
sees exactly what the importer keeps/strips" is structural, not comment-kept.
Removed the duplicated DEF_RE/fence machine from both consumers.
- Tests: new mock test for the footnoteWarnings plumbing on createPage (problems
-> field present; clean -> omitted); new paste-reuse case for TWO colliding
pasted definitions (reservation -> distinct ids). Updated the derive-id golden
test header (no MCP copy / parity test anymore).
- CHANGELOG: [Unreleased] entries for footnote reuse (Changed, supersedes 0.93.0)
and footnoteWarnings (Added).
editor-ext 129, MCP 301, server roundtrip 2; client+server tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Footnotes were strict 1:1: a repeated `[^a]` reference was treated as a
collision and re-id'd to `a__2`, and a reference with no definition synthesized
its own empty one — so an agent-authored article with reused labels produced
dozens of empty `kowiki__N` footnotes. Move to Pandoc REUSE semantics and add
non-fatal import diagnostics.
Reuse (core):
- resolveCollisions (footnote-sync): repeated references sharing an id are REUSE
(recorded once in document order, never re-id'd) — one number, one shared
definition. Only a duplicate DEFINITION is re-id'd deterministically and, with
no matching reference, dropped by the existing orphan policy (first-wins).
CollisionPlan.refReids is now always empty (harmless no-op downstream).
- extractFootnoteDefinitions (marked) and extractFootnotes (MCP): duplicate
definition ids are FIRST-WINS (keep first, drop rest); reference markers are
never rewritten. Removed the marker-rewriting and the now-dead deriveFootnoteId
mirror + helpers from the MCP path.
Import diagnostics:
- New analyzeFootnotes() (MCP): fence-aware pure scan reporting dangling
references, empty/duplicate definitions and `[^id]` markers inside table rows.
- createPage / updatePage / importPageMarkdown now attach `footnoteWarnings`
(only when non-empty) so an agent can fix its markup; the page is still created.
Paste-reuse:
- footnotePastePlugin remaps only ids the pasted slice DEFINES (a colliding
definition); a pasted lone reference to an existing id keeps it (reuse).
Tests: reuse/first-wins rewrites of footnote.test, footnote-markdown.test,
footnote.marked.orphan.test and the MCP footnotes.test; new footnote-paste.test
(editor-ext) and footnote-analyze.test (MCP). Deleted derive-id-parity.test.mjs
(the MCP no longer derives ids; editor-ext's deriveFootnoteId keeps its own
golden test). editor-ext 128, MCP 299, server roundtrip 2, client views 3,
client+server tsc clean.
Two review suggestions applied: corrected a stale "duplicated in MCP" comment and
the dangling-reference warning wording.
Note: the multi-backlink editor UI (a reused definition linking back to each of
its references) is deferred to a follow-up — this PR delivers the data-integrity
core (reuse + warnings + paste-reuse). Forward links and numbering already reuse
correctly; the backlink currently targets the first reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the approve-with-comments review on PR #154:
- applyDocToFragment: hydrate PMNode.fromJSON in its OWN try so a hydration
failure (e.g. an unknown node type) is labelled "fromJSON" — the stage that
actually threw — instead of the misleading "updateYFragment". The diagnostic
comment on unstorableYjsError ("label names the stage that failed") is now
truthful.
- assertYjsEncodable: also rehearse PMNode.fromJSON(docmostSchema, …) so a doc
that would only fail in apply's hydration step is rejected at preview time too,
narrowing the preview/apply gap (review suggestion B). Still cheap — no live
fragment, no updateYFragment.
- Tests: relabel the diagnostic test to (fromJSON); add structural-diff edge
cases — neighbour deletion keeps the unchanged node's cursor anchor, doc->empty
clears the fragment without throwing, top-level node-type change diffs in
place — plus a preview-gate test for the new fromJSON rehearsal. 297/297 green.
build/ rebuilt for the changed lib module only (build/client.js left untouched
to avoid pulling in pre-existing unrelated src/build drift).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review of #154 (Request changes) — all clean follow-ups, no defect in the fix:
1. Single source of the ProseMirror schema: export `docmostSchema` from
docmost-schema.ts (next to docmostExtensions); diff.ts and collaboration.ts
import it instead of each calling getSchema(docmostExtensions) — the schema
can no longer drift between call sites. Removed both local builds + the now
unused getSchema imports.
2. Doc fix: assertYjsEncodable's docstring and the client.ts comment no longer
claim "the same encoder as apply" — apply uses updateYFragment, the dry-run
uses toYdoc; both reject the same unstorable attrs but are NOT byte-identical.
Reworded to "independent encodability gate".
3+4+5. Extracted `unstorableYjsError(safe, label, e)` — buildYDoc and
applyDocToFragment now share one message template (label kept for diagnostics:
toYdoc vs updateYFragment), so the wording can't drift between dry-run/apply.
6. Test for applyDocToFragment's catch branch: an unknown node type makes the
schema-validated PMNode.fromJSON throw, and the function must re-throw it
wrapped with the (updateYFragment) diagnostic.
build/ rebuilt for the three changed lib modules; 293 package tests green.
(Left build/client.js untouched: rebuilding it would pull in a pre-existing,
unrelated src/build drift — a listSidebarPages slugId fix never rebuilt on
develop — and my client.ts change there is comment-only.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mutatePageContent wrote agent edits back by DELETING the whole Yjs fragment and
re-applying a fresh Y.Doc. Yjs is a CRDT — the editor anchors its selection to
node ids — so wiping every id made an open editor's cursor lose its anchor and
snap to the end of the document on every agent write. It was most visible on
comment anchoring (issue #152): a comment changes no text, yet the cursor jumped.
(Before commit 4201f0a3 the anchoring silently no-op'd, so the destructive write
never ran for comments — hence the regression.)
Fix: write via `updateYFragment` (y-prosemirror) — the same routine the editor
uses to sync its own edits into Yjs. It structurally diffs the new doc against
the live fragment and touches only changed nodes, preserving the ids of unchanged
ones, so the cursor stays put. This improves ALL agent write tools (text edits,
node ops, comments, replace) — minimal diff instead of full replace: less collab
noise, stable block-ids, other users' cursors no longer disrupted.
- collaboration.ts: new `applyDocToFragment` (sanitize -> PMNode.fromJSON against
a memoized docmost schema -> updateYFragment in one transact), keeping the
`findUnstorableAttr` encode diagnostic; swap the destructive write-back for it.
- package.json: `y-prosemirror` promoted to a direct dependency (was transitive).
- test: comment-cursor-stability.test.mjs — a Yjs RelativePosition (the cursor
anchor) survives both a sibling edit and a comment-mark anchoring (the old
full-replace tombstoned it -> null). 292 package tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The in-app AI chat hardcoded type='page' and the shared createComment
swallowed anchoring failures silently, so agent comments never got a
text anchor/highlight.
- Forbid page-type comments for the agent: top-level comments are always
inline and require an exact `selection`; replies inherit the parent
anchor (stored as the historical `page` type).
- Throw and roll back the just-created comment when the selection cannot
be anchored, instead of leaving an orphan unanchored comment.
- Add comment-anchor module: text normalization (smart quotes, dashes,
nbsp, collapsed whitespace) and matching across adjacent text nodes
within a block, so selections crossing inline-code/bold/link anchor.
- Update create_comment (MCP) and createComment (ai-chat) tool schemas
and descriptions; add unit + mock-HTTP orchestration tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements two architecture follow-ups from the multi-aspect review.
1. Shared, zod-agnostic tool-spec registry (packages/mcp/src/tool-specs.ts)
for the 14 AI tools whose name + schema + model-facing description are
genuinely identical across the standalone MCP server and the in-app
AI-SDK chat. Both layers consume it (registerShared in index.ts;
sharedTool in ai-chat-tools.service.ts) and keep their own execute/auth.
- Zod-agnostic builders (z) => ZodRawShape bridge the zod v3 (mcp) vs
zod v4 (server) split; the registry imports no zod.
- Folds in the documented edit_page_text drift-bug fix: the stale
"strip-and-retry tolerated" claim is gone; canonical wording states a
formatting-only change is refused into failed[].
- Sibling-tool references in shared descriptions are transport-neutral so
one description is correct for both snake_case (MCP) and camelCase
(in-app) tool names.
- Loader fail-fast guard for a stale @docmost/mcp build.
- The ~17 intentionally-divergent tools (security guardrails, tuned UX)
stay per-layer, untouched.
- Rebuilt committed mcp artifacts (also regenerates a previously stale
build/lib/docmost-schema.js to match its already-committed source).
2. Formalize apps/server/test/integration/db.ts as the canonical
integration-test seed factory (module doc + a shortId helper); the
hand-written minimal seeders are kept on purpose, decoupled from the
app service-layer side effects.
Verified: server tsc + lint clean, mcp build clean; mcp unit tests 261 pass,
ai-chat-tools.service 16 pass, public-share-chat-tools 8 pass, ai-chat suite
224 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First, safe step of docs/backlog/ai-chat-tool-definitions-duplicated.md: the
"node may be a JSON object OR a JSON string" quirk was hand-copied at 6 tool
sites. Extract it into a single parseNodeArg() helper per package and call it at
every site. Behavior-preserving — each site's throw message is byte-identical
(patch/insert: 'node was a string but not valid JSON'; update_page_json: 'content
was a string but not valid JSON'); no tool name/description/schema changed.
Two helper copies (packages/mcp/src/lib/parse-node-arg.ts and
apps/server/src/core/ai-chat/tools/parse-node-arg.ts) are intentional: the
ESM-only @docmost/mcp cannot be imported by the CommonJS server (it is loaded at
runtime via the Function('import()') trick), so runtime code cannot cross that
boundary by a normal import. Each copy is now the single source within its
package (6 inline copies -> 2 helpers). packages/mcp/build rebuilt in sync.
Tests: parse-node-arg.spec.ts (server, Jest) + parse-node-arg.test.mjs (mcp,
node:test) — object passthrough, valid-string parse, invalid-string throw with
the right message. Server tsc clean; mcp suite 254 pass; agent structural-edit
path verified live in-browser (agent inserted a node, persisted to the doc).
Deferred (documented for the record, since the backlog doc is removed with this
commit): the FULL transport-agnostic tool-spec registry (one name+schema+
description per tool shared by both transports) and deriving DocmostClientLike
from the real client type. Both are blocked by the current architecture, not by
effort: (1) @docmost/mcp ships no type declarations and is ESM-only, so a
type-only derivation needs declaration emission + tsconfig path wiring, and the
real client's precise return types break the in-app tool test stubs (attempted,
reverted to keep tsc green); (2) the two transports intentionally DIVERGE in tool
NAMES (snake_case x38 vs camelCase x41), membership (in-app adds getCurrentPage/
listSidebarPages, omits delete_comment/image tools) and model-facing
DESCRIPTIONS, so a unified registry would change behavior on BOTH the agent and
external MCP clients and needs its own verification pass. This is forward-looking
debt (the code is correct today), to be done incrementally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve conflicts at shared registration points by unioning both features
(footnotes + the already-merged html-embed / page-embed work):
- slash-menu/menu-items.ts, editor extensions.ts: keep both imports + configures
- collaboration.util.ts: register footnote nodes and pageEmbed
- editor-ext marked.utils.ts: register footnote + html-embed markdown extensions
- editor-ext package.json/tsconfig.json/vitest.config.ts: union of test config
(jsdom env for footnote DOM tests + combined test/spec include glob)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle red-team found two same-id footnoteDefinition nodes (trivially
produced by markdown import [^d]: first / [^d]: second, or paste/duplicate)
caused silent data loss: scan() used a last-wins Map and the sync rebuild
(addToHistory:false, propagated via Yjs, un-undoable) dropped all but the last.
Fix resolves collisions so BOTH survive, with a DETERMINISTIC id scheme so
collaborators converge:
- deriveFootnoteId(originalId, occurrence, taken): the k-th (k>=2) occurrence of
id X becomes X__k, bumped with a deterministic alpha suffix only against the
doc's own id set — a pure function of document state. No Math.random/Date.now
on the sync or import paths (random uuid stays only in setFootnote, where a
single user originates a brand-new id).
- footnote-sync.resolveCollisions walks refs+defs in document order, re-ids
duplicate references via setNodeMarkup and pairs them 1:1 with definitions;
single SYNC_META-tagged transaction, returns null when canonical (terminates).
- Markdown import (footnote.marked) + MCP mirror (collaboration.ts) dedup with
the same deterministic scheme + marker rewrite; packages/mcp/build regenerated.
- Paste plugin remaps colliding pasted ids against the current doc.
Tests: two independent editors resolving the same duplicate-id doc produce
IDENTICAL ids (the cross-client determinism guard that the random version would
fail); both definitions survive the first edit; import dedup is deterministic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds footnotes: a superscript marker in the text linked to an editable
definition in a Footnotes section at the end of the page, with auto-numbering
and a read-only hover popover. Chose the reference+definitions model (3 plain
nodes) over an inline atom with a sub-editor specifically for collaboration
safety.
editor-ext (packages/editor-ext/src/lib/footnote/):
- footnoteReference (inline atom, id), footnotesList (block, last child),
footnoteDefinition (paragraph+, id). renderHTML emits sup[data-footnote-ref]
/ section[data-footnotes] / div[data-footnote-def]; parse-rule priority makes
the empty reference win over the Superscript mark (else it is dropped on the
server save).
- numbering: a decoration-only plugin (pure function of doc order) -> every
client computes identical numbers, no document mutation, Yjs-safe.
- sync plugin: single-pass, always SYNC_META-tagged and skipping remote txns
(terminates, no loop), idempotent; canonicalizes to one trailing footnotesList
(merging duplicates), creates missing definitions, drops orphans, and
coexists with TrailingNode. Disabled in read-only.
- commands setFootnote (one tx: reference + definition at the matching index +
focus) / removeFootnote (cascade, one undo) / scrollTo*. slash /footnote.
client: superscript NodeView + floating-ui read-only popover; bottom-list and
definition NodeViews; registered in mainExtensions.
server: the three nodes registered in tiptapExtensions so collab/save/export
keep them. Round-trip regression spec guards the Superscript parse-priority.
markdown: turndown/marked round-trip to pandoc/GFM [^id] (+ a code-fence guard
so footnote-like lines inside code blocks are not extracted).
MCP mirror: schema + markdown-converter + commentsToFootnotes rewritten to real
footnote nodes + diff marker counting; NUL sentinels written as \u0000 escapes.
v2 follow-ups (per plan): definition reordering on reference move, id-collision
regeneration on paste, multiple references to one footnote.
Implements docs/footnotes-plan.md (variant B).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The embedded MCP server acted as a single service account; now each /mcp
session authenticates as the current user, so tools run under that user's
CASL and edits attribute to them.
- HTTP Basic (chosen path): Authorization: Basic email:password, validated
server-side via AuthService; the session carries the issued user JWT (not
the raw password). Password may contain ':' (split on first only).
- Bearer fallback: Authorization: Bearer <access JWT>, verified as ACCESS and
additionally checked for an active session + non-disabled user (matching
JwtStrategy), so revoked/disabled users are rejected.
- Service account stays as an optional fallback (no creds + env configured).
- packages/mcp createMcpHttpHandler accepts a per-request config resolver
(back-compat: static config / stdio unchanged); identity is bound to the
mcp-session-id at init and re-validated from the caller's own credentials on
every request (anti session-fixation: a guessed session id can't be reused
without matching creds).
- A full login (session + audit) happens only once at session init; later
requests re-verify credentials via a new non-side-effecting
AuthService.verifyUserCredentials (no session/audit spam).
- Failed-login limiter (5/60s, keyed per-IP, per-IP+email, and per-email so IP
rotation can't brute one account) since direct login bypasses the controller
throttler. Only real credential failures count.
- MCP_TOKEN shared guard moved off Authorization to an X-MCP-Token header
(timing-safe compare); credsConfigured 503 gate replaced by a clear 401.
- No secrets logged; all auth resolved before res.hijack() so failures return
clean 401 JSON. .env.example marks the service account optional.
Implements docs/backlog/mcp-per-user-auth.md (variant L).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
list_pages gains an opt-in `tree` parameter on both surfaces (the
@docmost/mcp server tool and the AI-chat agent tool), which share the
same DocmostClient.listPages. Default behavior (recent-by-updatedAt flat
list) is unchanged.
- client.ts: listPages(spaceId?, limit=50, tree=false); when tree is
true it requires spaceId (throws a specific error otherwise), walks the
sidebar tree via the existing bounded/cycle-safe enumerateSpacePages,
and returns a nested tree; limit is ignored in tree mode.
- lib/tree.ts: new pure buildPageTree() — lean nodes { id, slugId, title,
children? }, children sorted by position (code-unit order), orphans
promoted to roots, cycle-safe.
- index.ts + ai-chat-tools.service.ts: expose `tree` in the tool schemas
and descriptions; docmost-client.loader.ts: mirror the new signature.
- tests: add packages/mcp/test/unit/tree.test.mjs (nesting, ordering,
lean shape, orphan promotion, cycle/self-reference safety).
- rebuild @docmost/mcp (build/ is tracked and loaded at runtime).
edit_page_text reported "success" when asked to change formatting (e.g. remove
strikethrough): the markdown-strip fallback matched the bare text, the replace
preserved marks, and the tool returned success — so the agent believed it had
fixed something that never changed.
Two fixes, both in the shared @docmost/mcp DocmostClient so they reach BOTH the
standalone MCP server and the in-app AI chat (which loads @docmost/mcp):
- Verifiable result for every content mutator: mutatePageContent now computes a
`verify` change-report (text inserted/deleted, blocks changed, per-mark-type
delta, integrity/structure delta) via summarizeChange() and returns it on all
mutators (incl. replaceImage via mutateLiveContentUnlocked). diffDocs is
text-only, so the mark/structure delta is what surfaces formatting changes.
- edit_page_text hard-refuses formatting edits: applyTextEdits rejects an edit
whose find/replace differ only in markdown markers (via stripBalancedWrappers,
which strips balanced wrappers/links without trimming whitespace/emoji, so
plain-text edits like trailing-space trims, snake_case, math are NOT refused).
A fully-refused batch errors instead of silently succeeding.
Also updated the model-facing edit_page_text descriptions in BOTH tool layers
(packages/mcp/src/index.ts and ai-chat-tools.service.ts) to drop the misleading
"strip-and-retry tolerated" wording and point formatting changes to patch_node.
New unit tests: test/unit/diff-verify.test.mjs, test/unit/json-edit-refuse.test.mjs.
The insert_image and replace_image MCP tools previously uploaded only
local files (filePath), which an AI MCP client cannot provide — it has
no access to the server filesystem. Replace filePath with a required
imageUrl and download the image over http(s).
- client.ts: add fetchRemoteImage(url, maxBytes) — http/https-only scheme
allowlist, 20 MiB cap (maxContentLength + post-download length recheck),
30s timeout, Content-Type→MIME resolution with URL-extension fallback,
filename derivation with canonical extension
- client.ts: rewrite uploadImage(pageId, url) as URL-only; drop the
local-file branch, imageMimeFromPath and the fs import; insertImage/
replaceImage now take a url
- index.ts: drop filePath, add required imageUrl to both tools; update
tool descriptions and SERVER_INSTRUCTIONS
- README: document the web-URL behaviour
Locators (edit_page_text `find`, insert_node `anchorText`) are matched
against the document's plain text, so a model-supplied locator carrying
markdown wrappers (**bold**, *italic*, `code`, [t](url)) or trailing emoji
never matched and the edit/insert failed. Add stripInlineMarkdown() and a
fallback: try the locator verbatim first (exact match wins, so literal
asterisks/underscores still work), and only on zero matches retry with a
markdown-stripped form. The ambiguity guard runs on the post-fallback count,
and `replace` / inserted node content are never stripped, so no formatting is
lost. Failed edits gain an atom-aware reason plus a bounded "closest block
text" hint; the insert_node "anchor not found" error now points at plain-text
anchors / anchorNodeId.
New packages/mcp/src/lib/text-normalize.ts (+ unit tests); wired into
json-edit.ts and node-ops.ts; tool descriptions updated. Tests: 212 pass.
edit_page_text (applyTextEdits) now matches at the inline-block level instead of
per text node, so a find/replace may cross bold/italic/link boundaries; the
replacement inherits marks from the unchanged common prefix/suffix via a diff
splice. Atom (non-text inline) slots can never be part of a match, making the
U+FFFC placeholder collision-safe, and inserted text never inherits an atom's
marks.
The edit batch is no longer all-or-nothing: applyTextEdits returns
{ doc, results, failed } and applies what it can; editPageText writes only on a
real change (no spurious history version for a no-op) and throws an aggregated,
actionable error only when nothing applied.
The AI-chat insert_node / patch_node / update_page_json tools now JSON.parse a
node/content argument that arrives as a string, matching the standalone MCP
server (this is what made insert_node fail under OpenAI tool calls).
Tool descriptions gain concrete ProseMirror examples and reflect the new
edit_page_text behavior. Adds/updates json-edit unit tests (183 pass).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
claude code agent 227
agent_vscode
agent_coder
vvzvlad
a
vvzvlad
