The '(shareId,pageId) -> usable non-restricted page in THIS share' boundary was
written as 3 must-be-identical async sequences. They weren't: the chat funnel
omitted an explicit page.deletedAt check (latently safe via getShareForPage's
CTE) and layered isSharingAllowed separately. Add ShareService.resolveReadable-
SharePage(shareId,pageId,workspaceId) running the single canonical sequence
(getShareForPage -> id match (skipped when null) -> findById -> !deletedAt ->
!hasRestrictedAncestor) returning {share,page}|null; getSharedPage, the funnel,
and the getSharePage tool all use it. hasRestrictedAncestor now lives in the one
method no caller can skip; the funnel still returns uniform 404s and keeps
isSharingAllowed. Adds a direct security-invariant test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolveRoleForRequest and resolveShareRole duplicated the security invariant
'role exists, not soft-deleted, enabled, workspace-scoped, else null'. Move it to
AiAgentRoleRepo.findLiveEnabled(id, workspaceId) (deletedAt IS NULL + enabled +
workspace scope) and have both services call it, preserving each one's roleId
derivation + null handling. (describeProviderError half of #95 was done earlier.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
16 suites were disabled via testPathIgnorePatterns due to two root causes: lib0
ESM not transformed (the @hocuspocus/server -> lib0/decoding.js chain) and stock
'should be defined' specs built via Test.createTestingModule without providers.
Add lib0 to transformIgnorePatterns; convert the 14 DI placeholders to direct
new X(...) instantiation with stub deps (keeping a real construct smoke test);
re-enable the suites. Also updates the public-share limiter test to assert the
fail-closed behavior from #62 (surfaced now that the suite runs). Full server
suite: 67 passed, 689 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The concurrent-soft-delete guard was already covered; add the missing assertion
that update() returns toView(updated) from the post-update re-fetch (full
AgentRoleView shape, distinct second findById row), so a regression returning the
stale pre-update view or leaking columns is caught.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The anonymous share page-fetch tool's positive branch (sanitize via
updatePublicAttachments then jsonToMarkdown before returning to the model) was
untested, so a dropped/reordered sanitizer would ship a comment-mark/raw-
attachment leak with green tests. Add a positive-branch test pinning the
sanitizer call + that markdown derives from sanitized content, and a soft-deleted
test asserting a generic error with no content fetch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chatModel was a free string accepted with empty/garbage values, failing only at
runtime as a provider 503; tighten it (trim + non-empty + max 200). Driver was
already @IsIn(AI_DRIVERS). Collapse the client driver list to one AI_DRIVER_VALUES
source and add a contract test that reads the server AI_DRIVERS and fails on
client/server drift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A custom AI-role's text preceded the only SAFETY_FRAMEWORK block and replaced
the persona, so a jailbreak in the role text sat before the safety rules.
buildSystemPrompt now emits SAFETY both before AND after the persona, with the
role/persona delimited as lower-trust (<role_persona note=...>); the default
persona is sandwiched too. Context (currently-viewing-page) preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#60: streamText had no maxOutputTokens, so one anonymous request could run up
the provider bill. Add maxOutputTokens (env SHARE_AI_MAX_OUTPUT_TOKENS, default
512) via resolveShareAiMaxOutputTokens().
#95: the anonymous path hand-built error strings, diverging from the unified
describeProviderError format used on the authenticated path; both onError blocks
now call describeProviderError so a share reader sees 402/429/503 causes in the
same form (and the stack is still logged). Both changes are in this one file and
share hunks, hence one commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MAX_SHARE_MESSAGE_CHARS only counted text parts, so a forged non-text part
(tool-result/file/data) bypassed the cap and bloated the model input
(token-DoS); convertToModelMessages would also expand a forged tool-result. The
anonymous path runs no tools, so a client non-text part is never legitimate —
reject any message with a non-text part (isTextUIPart) before the size check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-workspace anonymous share-AI cost cap failed OPEN on a Redis error
(return true => admit), so a Redis outage removed the cap entirely (unmetered
billable anonymous calls). The feature is optional, so unavailability is
harmless: fail CLOSED (return false => controller 429s) instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Harden the anonymous public-share AI assistant against token-cost abuse
before exposing it to the internet:
- Add an env-tunable per-request output ceiling (maxOutputTokens) to the
public-share streamText call so one anonymous request cannot run up the
provider bill even if the per-IP throttle is evaded. New
resolveShareAiMaxOutputTokens() / SHARE_AI_MAX_OUTPUT_TOKENS_DEFAULT
(env SHARE_AI_MAX_OUTPUT_TOKENS, default 512), mirroring
resolveShareAiWorkspaceMax().
- Flip the per-workspace cost limiter to FAIL CLOSED on Redis failure
(was fail-open): if Redis is unavailable we cannot prove the workspace is
under its cap, so deny rather than admit an unmetered, billable call.
- Update the limiter spec (fail-open -> fail-closed) and add resolver tests;
document both knobs in .env.example.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ~330 tests across server (Jest), client (Vitest), editor-ext (Vitest)
and packages/mcp (node:test) for the gitmost features added since
053a9c0d: AI chat, AI agent roles, public-share assistant, MCP per-user
auth, HTML embed, page templates/embed, realtime tree, tree
expand/collapse, and the AI-settings UI.
Test-tooling fixes (prerequisite, were silently hiding coverage):
- Repair 3 page-template specs broken by the 11-arg TransclusionService
constructor; they never compiled, so template access-control / content
-leak / unsync-strip coverage was fictitious.
- Build @docmost/editor-ext before server tests via a `pretest` hook;
the stale dist omitted the new HtmlEmbed/PageEmbed exports (TS2305).
- Let jest resolve the .tsx email templates: add `tsx` to
moduleFileExtensions and widen the ts-jest transform to (t|j)sx?.
Behaviour-preserving "extract pure core" refactors that the tests drive:
- server: resolveShareAssistantRequest + uiMessageTextLength
(public-share controller), decideBasicGate + mapAuthResultToResponse
(mcp), buildErrorAssistantRecord (ai-chat), jsonbObject export (roles).
- client: render-raw-html + shouldExecute/canEdit, decide-embed-state,
page-embed picker utils, tree-socket reducers, open/close branch maps,
isEndpointConfigured/resolveKeyField; buildTreeWithChildren now treats
a permission-trimmed orphan as a root instead of crashing.
Deferred (need a test DB or HTTP harness, documented in the specs):
repo-level Postgres integration tests and the public-share XFF E2E.
Pre-existing DI/lib0-ESM suite failures are untouched and out of scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The current page id was only injected as text in the system prompt, which a
proxy (CLIProxyAPI) can rewrite/truncate, so the agent could lose track of 'this
page'. Add a getCurrentPage tool the model can call to read the open page (id +
title) from the server-side request context (forUser now takes openedPage,
threaded from body.openPage — the same value used for the system prompt). The
inline system-prompt line is kept as belt-and-suspenders. Reads/writes still go
through the CASL-enforced page tools by id, so this is strictly not worse than
the existing prompt hint — just delivered over a channel the proxy can't mangle.
User-approved on the issue. Completes #43 together with the hardness-1 fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Anonymous public-share AI assistant:
- Add a workspace setting `publicShareAssistantRoleId` so an admin can pick which
agent role (identity/persona) the anonymous assistant adopts. The role's
instructions REPLACE the built-in persona while the immutable safety framework
is still always appended; the role's optional model override takes precedence
over the cheap publicShareChatModel. Resolved server-authoritatively
(workspace-scoped, soft-delete aware; disabled/missing roles fall back to the
built-in persona, so the tool scope remains the real security boundary).
- Plumb the field through the update DTO, ai-settings service, the workspace.repo
ALLOWED whitelist, resolve()/getMasked(), stream-time role resolution and the
prompt/model, plus the settings UI: a new "Assistant identity" Select listing
enabled roles (and surfacing a saved-but-disabled role explicitly).
Public-share branding / floating icon:
- Fix the AI assistant FAB overlapping the "Powered by ..." button (both were
Affixed bottom-right): stack the FAB above the bottom-right branding.
- Rename "Powered by Docmost" -> "Powered by Gitmost" and point the link at the
gitmost repo.
Tests: extend public-share-chat.spec (role persona replacement still appends the
safety framework, resolveShareRole edge cases, model-override precedence).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve conflicts with the independently-merged ai-agent-roles feature:
- ai-chat.module.ts: keep BOTH AiAgentRolesModule and the public-share
wiring (Share/Search modules, PublicShareChatController, services).
- ai.service.ts: take develop's getChatModel ChatModelOverride superset,
which already covers the public-share model-id-only override.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up fixes on the agent-roles feature:
- ai.service: a cross-driver override to the ollama driver (when the
workspace driver is not ollama) now fails with an explicit 503 instead
of silently reusing the workspace base URL, which belongs to a different
provider. Same-driver ollama and openai/gemini overrides are unchanged.
- migration: add a partial unique index on (workspace_id, name) WHERE
deleted_at IS NULL so role names are unique per workspace without
soft-deleted rows blocking re-creation; map Postgres 23505 to a 409
ConflictException on create/update.
- dto: validate the role id as @IsUUID instead of @IsString.
- roles list: do not expose instructions/modelConfig to non-admin members.
The list endpoint now returns a picker view (id/name/emoji/description/
enabled) to members and the full view only to admins (same gate as the
CRUD endpoints). Client IAiRole fields made optional accordingly.
Adds tests for the cross-driver-ollama throw, the 23505->409 mapping, and
the non-admin picker-view security invariant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Integrate the already-merged step-limit work from develop. Only conflict was
ai-chat.service.spec.ts: both sides appended a describe block and edited the
import line. Resolved as a union — keep compactToolOutput + the assistantParts/
serializeSteps/rowToUiMessage suites (this branch) AND the prepareAgentStep
suite (develop), importing all symbols from ai-chat.service.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds explicit isIpAllowed cases for the CGNAT, ULA (fd00::/8) and IPv4-mapped
IPv6 loopback (::ffff:127.0.0.1) sample addresses from the parallel
safety-coverage branch. The mapped-loopback case is genuinely new (the existing
table only covered the mapped *private* variant); CGNAT and ULA ranges were
already covered with other samples and are kept here as explicit regression
guards for these specific addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Behaviour change (split out of the test commit per review, and now covered).
Both the stream onError log line and the error text streamed to the client were
formatted by separate inline blocks that only emitted "<status>: <message>".
Route both through the shared describeProviderError() so formatting stays in one
place.
BEHAVIOUR CHANGE: describeProviderError additionally appends a single-line,
300-char-truncated snippet of the provider responseBody/text. So the log line
AND the user-facing stream error now include that snippet (e.g. the HTML error
page from a misconfigured endpoint), which previously neither did. This is
intentional — it makes a misconfigured external endpoint diagnosable — and is
safe: the API key travels in the Authorization header and is never echoed in
the response body (see the util's docstring). A `fallback` param is added so
each call site keeps its own default ('AI stream error' for the stream).
Adds ai-error.util.spec.ts covering the formatter, including the appended /
truncated body snippet, so this behaviour is no longer untested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unit tests for the safety-critical paths: crypto secret-box (round-trip,
tamper detection, wrong key), the SSRF guard (blocked ranges + DNS-rebinding),
the ai-chat tools service, the page-embedding repo, and the
assistant-parts/serialization helpers. Those server helpers (assistantParts,
rowToUiMessage, serializeSteps) are exported ONLY for the tests — no runtime
change.
Also: keyboard a11y on the chat history header and conversation rows
(role/tabIndex/Enter+Space), and DRY refactors that move shared logic into one
place (isToolPart -> tool-parts util; buildInitialValues in the MCP form).
The behaviour-changing edits that previously rode along in this commit are
split out into the following two commits, per review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port two refinements from the GLM variant onto the Claude base:
- prepareAgentStep: add a comment note that AI SDK v7 renames the per-step
`system` field to `instructions` (v6 ^6.0.134 still uses `system`), so it
gets updated correctly on the next SDK bump.
- ai-chat.service.spec: add an explicit off-by-one boundary test for
prepareAgentStep, expressed via MAX_AGENT_STEPS instead of a hardcoded 18/19
so it tracks the constant if the cap changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle review: the per-workspace cost cap was fixed-window + per-instance
(allowed ~2x at a window boundary and K*cap behind K instances) on an anonymous
endpoint that spends the owner's provider budget. Rewrite it as a sliding-window,
CLUSTER-WIDE Redis limiter: one atomic Lua EVAL does ZREMRANGEBYSCORE (age out)
-> ZCARD -> ZADD with PEXPIRE, so concurrent instances share one budget and the
true rate over any trailing window is <= cap. Fails OPEN on a Redis error (logged)
— it's a cost backstop, not access control (the funnel gates + per-IP throttle
still apply), so a Redis blip must not take the assistant offline. Per-IP @Throttle
kept; commented that it needs an XFF-rewriting trusted proxy to be meaningful.
Extract deriveShareAccess (resolvedShareId===requestedShareId + isSharingAllowed +
!restricted, equality-only, never widening) and filterShareTranscript into pure
helpers, and add tests: limiter sliding-window + boundary-burst + fail-open;
access derivation; and red-team boundary locks (cross-share/cross-workspace swap
rejected, forged shareId can't widen tool scope, transcript injection filtered).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle test audit found the role feature's security-critical paths
untested. Adds real unit tests (against the actual functions):
- resolveRoleForRequest invariants: role comes from chat.roleId not body.roleId
(no per-turn swap), lookup scoped to workspace.id, disabled/soft-deleted role
-> null, new-chat uses body.roleId, stale chatId falls back.
- CASL admin gate: non-admin create/update/delete -> Forbidden and service not
called; admin delegates with workspace.id; list() is member-reachable.
- roleModelOverride: unknown driver dropped (never reaches getChatModel's
throwing default), valid override passes through, blanks ignored.
- getChatModel override success path (cross-driver fetch + decrypt; chatModel-
only reuse), and service update/remove cross-workspace 'not found' guards +
modelConfig tri-state.
Tiny fix: findByCreator badge left-join now also requires enabled=true, so a
disabled role (downgraded to universal by resolveRoleForRequest) no longer shows
a misleading chat-list badge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle review: update() re-read the role via findById (filters
deleted_at IS NULL) and passed it straight to toView(updated as AiAgentRole).
A concurrent soft-delete between the UPDATE and the re-fetch makes findById
return undefined, and toView(undefined) dereferences row.id -> opaque 500. Add
the same 'Role not found' guard remove() already uses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release-cycle red-team found getShareForPage joins only the shares table, so it
does not exclude restricted descendants. The public share VIEW (getSharedPage)
compensates with hasRestrictedAncestor, but the assistant's getSharePage tool
and the controller funnel did not — so an anonymous caller could read a
restricted descendant's content (tool) or surface its title into the system
prompt (funnel) within an includeSubPages share.
- getSharePage: after the share-membership check and before returning content,
reject with the generic 'not part of this published share' message when
hasRestrictedAncestor(page.id) is true (page.id is the resolved UUID, so
slugId inputs work). Inject PagePermissionRepo.
- funnel: resolve the OPENED page to its UUID and treat a restricted opened page
as not-in-share (same uniform 404, fail closed if unresolvable) so its title
never reaches buildShareSystemPrompt.
search/list already exclude restricted subtrees (getPageAndDescendantsExcludingRestricted),
so these were the only two bypasses. Generic messages keep restricted
indistinguishable from not-in-share.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lets an unauthenticated viewer of a published share ask an AI scoped strictly
to that share's page tree. The authenticated agent is untouched; the security
boundary is the tool scope (no identity), and nothing is persisted.
Server:
- workspace toggle settings.ai.publicShareAssistant (default off) +
optional settings.ai.provider.publicShareChatModel (cheap model id; reuses
the chat driver/baseUrl/key). getChatModel(workspaceId, override) substitutes
only the model id, falling back to chatModel.
- POST /api/shares/ai/stream (@Public, SSE). Guardrail funnel, each failing
before streaming: toggle off -> 404; share missing/wrong-workspace/sharing
off -> 404; pageId not in share tree -> 404; provider unconfigured -> 503;
per-IP (5/min) and per-workspace (300/h, IP-independent) rate limits -> 429.
Uniform 404s never confirm a private page's existence.
- forShare read-only in-process toolset: searchSharePages (existing shareId
FTS branch, no spaceId/userId), getSharePage (getShareForPage gate +
share.id check, content via the public sanitizer), listSharePages. No write/
comment/history/cross-space/external-MCP tools.
- Locked share system prompt + immutable safety block; stepCountIs(5).
- /shares/page-info exposes an aiAssistant flag (gated behind isSharingAllowed).
Client: an ephemeral, text-only Ask-AI widget on the public shared page,
shown only when the flag is set; useChat -> /api/shares/ai/stream,
credentials omit. Admin toggle + model field in Settings -> AI.
Also adds a jest moduleNameMapper for src/-rooted imports (fixes pre-existing
unresolvable specs; additive).
Implements docs/public-share-assistant-plan.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reusable, workspace-shared agent roles for the built-in AI chat. A role is
a named persona (system-prompt instructions) + optional model override; a
chat is bound to a role at creation and applies it every turn.
Backend:
- migration 20260620T120000: ai_agent_roles table + ai_chats.role_id
(FK ON DELETE SET NULL); hand-merged types into db.d.ts/entity.types.ts
(db.d.ts is hand-curated here, full codegen would clobber it).
- core/ai-chat/roles: CRUD module. list = any workspace member; create/
update/delete = admin (Manage Settings ability, like ai-settings/mcp).
All repo queries scoped by workspace_id; soft-delete (deleted_at).
- buildSystemPrompt gains roleInstructions: role REPLACES the persona base
(admin prompt / DEFAULT_PROMPT) but SAFETY_FRAMEWORK + context are always
still appended.
- stream(): role resolved from ai_chats.role_id for existing chats (never
the request body -> no per-turn role swap); body.roleId only on creation.
Disabled (enabled=false) and soft-deleted roles fall back to universal.
- getChatModel(workspaceId, override): role model_config can swap model id /
driver; a driver without configured creds throws 503 with a clear message
naming the driver+role, resolved BEFORE response hijack.
Client:
- new-chat role picker (enabled roles only, default Universal assistant),
roleId sent only on the first message; role badge (emoji+name) in the chat
header and conversation list; admin Agent-roles management section in
Settings -> AI (add/edit/delete, MCP-form pattern).
Tests: ai-chat.prompt.spec (role layering + safety always present, incl.
jailbreak); ai.service.spec (override on unconfigured driver -> 503).
Implements docs/ai-agent-roles-plan.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A narrow research question could burn all 8 steps on tool calls and end the
turn with no assistant text (empty turn). Two changes:
- MAX_AGENT_STEPS = 20 (was a magic stepCountIs(8)) so multi-search turns
aren't cut off mid-investigation.
- prepareStep reserves the LAST allowed step for a text-only synthesis:
toolChoice 'none' + a FINAL_STEP_INSTRUCTION appended to (not replacing)
the system prompt, so a tool-heavy turn always ends with a real answer.
Logic extracted into the pure, exported prepareAgentStep(stepNumber, system)
for unit testing; earlier steps return undefined (default behavior).
Implements docs/backlog/ai-chat-step-limit-and-forced-final-answer.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
list_pages gains an opt-in `tree` parameter on both surfaces (the
@docmost/mcp server tool and the AI-chat agent tool), which share the
same DocmostClient.listPages. Default behavior (recent-by-updatedAt flat
list) is unchanged.
- client.ts: listPages(spaceId?, limit=50, tree=false); when tree is
true it requires spaceId (throws a specific error otherwise), walks the
sidebar tree via the existing bounded/cycle-safe enumerateSpacePages,
and returns a nested tree; limit is ignored in tree mode.
- lib/tree.ts: new pure buildPageTree() — lean nodes { id, slugId, title,
children? }, children sorted by position (code-unit order), orphans
promoted to roots, cycle-safe.
- index.ts + ai-chat-tools.service.ts: expose `tree` in the tool schemas
and descriptions; docmost-client.loader.ts: mirror the new signature.
- tests: add packages/mcp/test/unit/tree.test.mjs (nesting, ordering,
lean shape, orphan promotion, cycle/self-reference safety).
- rebuild @docmost/mcp (build/ is tracked and loaded at runtime).
The floating AI-chat header badge summed metadata.usage (AI SDK
totalUsage, all steps) across every assistant row, showing the
cumulative tokens SPENT — which grows each turn as history is re-sent.
Replace it with the conversation's CURRENT context size.
- server: persist metadata.contextTokens in streamText onFinish from the
final-step `usage` (inputTokens + outputTokens ≈ current context
window occupancy); keep usage: totalUsage for back-compat/fallback
- client: derive the badge from the most recent assistant row's
contextTokens (fallback to that row's usage total for older chats)
instead of summing all rows
- types: add metadata.contextTokens to IAiChatMessageRow
- i18n: rename badge label "Tokens used in this chat" -> "Current
context size" (en-US)
No DB migration needed (metadata is a JSON column).
- ai.service: route *.openrouter.ai STT to its JSON+base64
/audio/transcriptions API; keep the OpenAI multipart path (AI SDK) for
OpenAI/self-hosted whisper. Unify transcription behind transcribe().
- /transcribe controller: surface the real provider/transport reason
(describeProviderError) instead of an opaque 500; preserve HttpException.
- testConnection: add an 'stt' capability (silent-WAV probe) + DTO; client
gets a Test endpoint button and status dot on the Voice/STT card.
- useDictation: log full errors to the console and show the real reason
(mic start + transcription paths); handle NotReadable/Abort and missing
mediaDevices.
- docs(CLAUDE.md): require full error logging + specific user-facing messages.
Add push-to-talk voice dictation that transcribes recorded audio on the
server via the workspace's OpenAI-compatible AI provider (Whisper /
gpt-4o-transcribe / self-hosted whisper), then inserts the text.
Backend:
- New `stt_api_key_enc` column + migration; STT creds parity with chat/
embeddings (sttModel/sttBaseUrl/sttApiKey, write-only key, fallbacks to
chat baseUrl/key). Both provider whitelists updated (service + repo).
- AiService.getTranscriptionModel + AiTranscriptionService.
- Gated POST /ai-chat/transcribe (dictation flag → 403, JWT + workspace
scope + throttle, 25MB cap, MIME whitelist, never logs audio/key).
- New `settings.ai.dictation` workspace flag (DTO + service + audit).
Frontend:
- Wire up the Voice/STT settings card (model/base URL/key) and the
Voice-dictation toggle.
- New `features/dictation`: useDictation (MediaRecorder state machine),
MicButton, transcribe service; integrated into the chat composer and a
new editor-toolbar dictation group, both gated by ai.dictation.
edit_page_text reported "success" when asked to change formatting (e.g. remove
strikethrough): the markdown-strip fallback matched the bare text, the replace
preserved marks, and the tool returned success — so the agent believed it had
fixed something that never changed.
Two fixes, both in the shared @docmost/mcp DocmostClient so they reach BOTH the
standalone MCP server and the in-app AI chat (which loads @docmost/mcp):
- Verifiable result for every content mutator: mutatePageContent now computes a
`verify` change-report (text inserted/deleted, blocks changed, per-mark-type
delta, integrity/structure delta) via summarizeChange() and returns it on all
mutators (incl. replaceImage via mutateLiveContentUnlocked). diffDocs is
text-only, so the mark/structure delta is what surfaces formatting changes.
- edit_page_text hard-refuses formatting edits: applyTextEdits rejects an edit
whose find/replace differ only in markdown markers (via stripBalancedWrappers,
which strips balanced wrappers/links without trimming whitespace/emoji, so
plain-text edits like trailing-space trims, snake_case, math are NOT refused).
A fully-refused batch errors instead of silently succeeding.
Also updated the model-facing edit_page_text descriptions in BOTH tool layers
(packages/mcp/src/index.ts and ai-chat-tools.service.ts) to drop the misleading
"strip-and-retry tolerated" wording and point formatting changes to patch_node.
New unit tests: test/unit/diff-verify.test.mjs, test/unit/json-edit-refuse.test.mjs.
Improve agent RAG quality with three changes, plus a roadmap doc for the rest.
- Indexer: prefix each chunk with its heading path ("Page > H1 > H2"), built by
walking the ProseMirror JSON (heading nodes) so a `#` inside a fenced code block
is never mistaken for a heading. Falls back to plain-text chunking on any error.
buildChunkRows: drop indexOf-against-source offsets (breadcrumb prefixes break
verbatim matching) for a cumulative cursor — offsets are provenance-only.
- Hybrid search: new migration adds a generated `fts` tsvector column + GIN index
to page_embeddings (same english+f_unaccent config as pages.tsv). New
PageEmbeddingRepo.hybridSearch fuses cosine + full-text rankings via Reciprocal
Rank Fusion (k=60, equal weights) in one SQL query at chunk granularity.
- Tools: collapse semanticSearch + searchPages into one hybrid `searchPages` tool
with a query-rewrite-oriented description; gracefully falls back to the REST
full-text path when embeddings are unconfigured. Access control (space scope +
page-permission post-filter) preserved. Add a query-rewrite hint to the default
system prompt.
- docs/rag-improvements-plan.md: record what shipped and the deferred backlog
(reranker, attachment indexing, eval harness, tuning).
Note: requires a corpus reindex to populate breadcrumbs on existing pages.
The bulk embedding reindex could hang on a single page forever
("Indexed 27 of 34 pages") with zero log output:
- all progress logs were debug-level, suppressed in production (pino info);
- embedMany() had no timeout, so a slow/hung embeddings endpoint blocked
the sequential per-page loop indefinitely.
Changes:
- ai.service.embedTexts: bound embedMany with AbortSignal.timeout
(configurable via AI_EMBEDDING_TIMEOUT_MS, default 120000ms); on timeout
throw a clear, greppable message, classified by both signal.aborted and
the error name (TimeoutError/AbortError/ResponseAborted) so a real
provider error racing the timer keeps its diagnostics.
- embedding-indexer.reindexWorkspace: promote lifecycle/progress logs to
info; log "[i/N] indexing page <id>" BEFORE the await so a hang names the
stuck page; warn on slow pages (>30s); add timing + final summary.
- .env.example: document AI_EMBEDDING_TIMEOUT_MS.
A misconfigured embeddings endpoint failed the RAG indexer with an opaque
"Invalid JSON response" and was not caught by "Test connection" (which only
probed the chat model), so it only surfaced silently during background
indexing.
- add describeProviderError(): formats AI SDK errors as
"<statusCode>: <message> | response body: <truncated one-line snippet>"
(statusCode/message/responseBody never carry the API key)
- use it in the bulk-reindex catch and the embedding processor's formatter so
the real cause (e.g. an HTML 404 from a wrong base URL) is visible in logs
- testConnection now probes chat AND embeddings independently: skips a probe
when that capability is unconfigured, returns ok:false with a Chat:/Embeddings:
prefix on real failure, "not configured" when neither is set
The WORKSPACE_CREATE_EMBEDDINGS / WORKSPACE_DELETE_EMBEDDINGS jobs were
enqueued (on AI Search enable/disable) but had no AI_QUEUE handler, so
existing pages were never indexed ("Indexed 0 of N pages") and disabling
never purged embeddings.
- EmbeddingProcessor: handle WORKSPACE_CREATE_EMBEDDINGS (bulk reindex all
live pages) and WORKSPACE_DELETE_EMBEDDINGS (purge workspace embeddings)
- EmbeddingIndexerService: add reindexWorkspace() (skips when embeddings
unconfigured; per-page error isolation) and removeWorkspace()
- PageRepo.getIdsByWorkspace(), PageEmbeddingRepo.deleteByWorkspace()
- AiSettingsService.reindex() + admin-only POST /workspace/ai-settings/reindex
- Frontend: "Reindex now" button, service call and mutation
- Stable per-workspace jobId with remove-before-add so a stale job can't
block future reindexes; cancel the delayed purge on enable/reindex so it
can't wipe freshly-built embeddings
Read tools (getPage, getPageJson, getNode, diffPageVersions,
exportPageMarkdown) return whole pages with no size cap. Their outputs
were stored verbatim in metadata.parts and the tool_calls column, and
metadata.parts is replayed to the provider on every later turn via
convertToModelMessages. After reading a couple of large pages the prompt
grew by full page bodies each turn — rising token cost, latency and DB
row size.
Add compactToolOutput(): a pure, recursive, size-bounded compactor used
in assistantParts() and serializeSteps(). It preserves the value's kind
and small scalar fields (id/title/pageId, which the client reads to build
citations on reload) while truncating long strings, capping long arrays
with a marker, and collapsing subtrees past a depth limit. Small outputs
are returned unchanged by identity. Tool inputs are left intact so
replayed tool_use arguments keep their object shape.
Compaction runs only at persistence time (onFinish/onAbort), so the live
stream and the current turn's multi-step reasoning still see full bodies.
Add unit tests for compactToolOutput.
The default system prompt introduced the assistant as embedded in
"Docmost". Update DEFAULT_PROMPT to say "Gitmost" so the agent
presents itself with the correct product name.
Locators (edit_page_text `find`, insert_node `anchorText`) are matched
against the document's plain text, so a model-supplied locator carrying
markdown wrappers (**bold**, *italic*, `code`, [t](url)) or trailing emoji
never matched and the edit/insert failed. Add stripInlineMarkdown() and a
fallback: try the locator verbatim first (exact match wins, so literal
asterisks/underscores still work), and only on zero matches retry with a
markdown-stripped form. The ambiguity guard runs on the post-fallback count,
and `replace` / inserted node content are never stripped, so no formatting is
lost. Failed edits gain an atom-aware reason plus a bounded "closest block
text" hint; the insert_node "anchor not found" error now points at plain-text
anchors / anchorNodeId.
New packages/mcp/src/lib/text-normalize.ts (+ unit tests); wired into
json-edit.ts and node-ops.ts; tool descriptions updated. Tests: 212 pass.
edit_page_text (applyTextEdits) now matches at the inline-block level instead of
per text node, so a find/replace may cross bold/italic/link boundaries; the
replacement inherits marks from the unchanged common prefix/suffix via a diff
splice. Atom (non-text inline) slots can never be part of a match, making the
U+FFFC placeholder collision-safe, and inserted text never inherits an atom's
marks.
The edit batch is no longer all-or-nothing: applyTextEdits returns
{ doc, results, failed } and applies what it can; editPageText writes only on a
real change (no spurious history version for a no-op) and throws an aggregated,
actionable error only when nothing applied.
The AI-chat insert_node / patch_node / update_page_json tools now JSON.parse a
node/content argument that arrives as a string, matching the standalone MCP
server (this is what made insert_node fail under OpenAI tool calls).
Tool descriptions gain concrete ProseMirror examples and reflect the new
edit_page_text behavior. Adds/updates json-edit unit tests (183 pass).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Editing an existing comment's text is irreversible (not version-tracked),
which breaks the agent's "only reversible operations" invariant. Remove the
updateComment tool that was added in the toolset-expansion change, leaving the
agent at 40 tools (comments: create/resolve only).
- Remove the updateComment tool from forUser().
- Remove updateComment from the DocmostClientLike interface.
- Reword SAFETY_FRAMEWORK: comments are create/resolve only; drop the
comment-text-edit exception (keep the public-sharing one); keep the
no-permanent-deletion guarantee and anti-prompt-injection rules.
- Tests: assert updateComment is NOT exposed (mirrors the deleteComment guard).
- docs(ai-agent-chat-plan): move updateComment to the "not exposed" list.
Grow the agent tool registry in forUser() from 10 to 41 tools, wiring all
remaining @docmost/mcp client capabilities: reads (workspace/spaces/pages/
sidebar/outline/json/node/table/comments/shares/history/diff/export) and
reversible writes (editPageText, patch/insert/delete node, updatePageJson,
table ops, copy/import content, share/unshare, restorePageVersion,
updateComment, transformPage).
Deliberately NOT exposed: deleteComment (irreversible hard delete) and the
filePath-based image tools (uploadImage/insertImage/replaceImage — useless
and unsafe for a server-side agent). transformPage omits the deleteComments
option from its schema and never passes it, so the comment-deletion path is
unreachable from the agent.
- Extend DocmostClientLike with the new method signatures.
- Update SAFETY_FRAMEWORK to describe the broader toolset while keeping the
no-permanent-deletion guarantee and anti-prompt-injection rules; flag that
comment-text edits are not version-tracked and sharing is public.
- Add guardrail tests: no deleteComment tool; transformPage schema rejects
deleteComments.
- docs(ai-agent-chat-plan): record the toolset expansion and a backlog item
to support image insertion by URL via the existing SSRF guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
so the v6 hook stops re-creating its store every render on a new chat
(which wiped the optimistic user message + streamed deltas, so nothing
showed until the turn finished). Also send X-Accel-Buffering:no + flushHeaders.
- context: client sends the currently-open page {id,title}; the system prompt
tells the agent which page 'this page' refers to (it reads it via its
CASL-scoped getPage tool; id is prompt-context only, no server-side fetch).
- embeddings: make page_embeddings.embedding dimension-agnostic (drop the
HNSW index + ALTER to vector), remove the hard 1536 guard, filter search by
model_dimensions — so 3072-dim (and any) models index instead of being
skipped. Seq-scan <=> search (wiki scale); existing pages reindex on next edit.
- openai provider: use .chat() (Chat Completions) instead of the default callable
(Responses API), which gateways reject on multi-turn -> 400.
- updateAiProviderSettings: assemble settings.ai.provider via jsonb_build_object
with ::text-cast bound params + jsonb_typeof self-heal (postgres.js was
double-encoding it into an array; the ::text cast avoids 'could not determine
data type of parameter').
- chat agent: drop the hard maxOutputTokens cap (truncated complex tool calls);
keep a tiny cap only on the test-connection ping.
- testConnection + chat stream: surface the real provider error (statusCode+message)
to logs and the UI instead of generic masks; never log the API key.
- chat UI: typing indicator, incremental streaming render, tool 'running' status, Stop.
Also bundled (prior uncommitted ai-chat work):
- history 'AI agent' provenance badge; vector RAG (pgvector image + page_embeddings
+ AI_QUEUE indexer + space-scoped semanticSearch); external MCP servers backend
(@ai-sdk/mcp client, SSRF IP-pinning, encrypted headers, admin CRUD/Test);
yjs duplicate-instance fix via pnpm patch (single CJS instance server-side).
- Add reversible write tools to the per-user agent toolset (page create/update/
move/soft-delete; comment reply + resolve), exposed under the user's JWT and
enforced by Docmost CASL; no permanent/force delete (D3).
- Non-spoofable agent provenance: sign actor/aiChatId into the access and collab
tokens (TokenService), propagate via jwt.strategy onto the request, and set
pages.last_updated_source/last_updated_ai_chat_id on REST create/update/move and
comments.created_source/resolved_source/ai_chat_id.
- packages/mcp: add an optional getCollabToken provider (content-edit provenance)
and guard against empty tokens; service-account /mcp path unchanged.
Frontend:
- Admin 'AI / Models' settings section: provider/model/embedding/base URL, a
write-only API key field, system prompt, and Test connection.
- AI chat panel (useChat + DefaultChatTransport): conversation list, streamed
messages, tool-call action log and page citations; header entry point gated on
settings.ai.chat.
Compile-verified (server nest build + client tsc/vite); not yet live-tested.
Known gaps: history 'AI agent' badge (C3), vector RAG (D), external MCP (E);
chat tool-card citation links pending a fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
