Compare commits

...

11 Commits

Author SHA1 Message Date
claude code agent 227
dd64c2ea05 fix(mcp): write page body before title to avoid split-brain on failure (#159)
updatePage (markdown) and updatePageJson wrote the title via REST FIRST, then
the body via collab. If the body write failed (e.g. a collab persist timeout),
the page was left with the NEW title over its OLD body — a split-brain the tool
reported as an error but never repaired (red-team finding #10).

Reorder both: write the body first, and only set the title after the body has
persisted. Now a body-write failure leaves the title untouched (no split-brain).
A title write failing after a successful body is rarer (REST is fast) and leaves
correct content under a stale title — the strictly lesser inconsistency — which
is the same trade-off the issue's "atomic, or roll back the title" intends,
without the fragility of a rollback write that could itself fail.

No unit test: both paths require a live collab provider and the suite has no
provider mock; the change is a pure reordering. All 306 mcp tests still pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:19:32 +03:00
claude code agent 227
96fb737c9d fix(share): SEO route must not leak a restricted page's title (#159)
`ShareSeoController.getShare` resolved the inherited share with the RAW
`getShareForPage`, which does NOT run the restricted-ancestor gate. So for a
page shared with includeSubPages whose descendant is permission-restricted, the
SEO route served that descendant's real title in <title>/og:title/twitter:title
to anonymous visitors and crawlers — even though the content API returns 404 for
it (red-team finding #3).

Funnel the SEO path through the canonical `resolveReadableSharePage` boundary
(the single place that checks `hasRestrictedAncestor`): a non-readable page now
serves the plain SPA index with no meta. Also honour `isSharingAllowed` — a
share whose workspace/space sharing toggle was flipped off after creation no
longer leaks its title via SEO. Title comes from the server-resolved page;
`buildShareMetaHtml` already emits robots=noindex when the share opted out of
indexing.

Tests (controller routing, fs spied at call time so bcrypt's native loader is
untouched): non-readable page => plain index, no title; sharing-disabled =>
plain index; readable+indexing => title + og:title, no noindex; readable+no-
indexing => noindex. Asserts getShareForPage is never called by the SEO path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:19:32 +03:00
claude code agent 227
66f9079996 fix(ai-chat): validate the open page server-side so the agent edits the right one (#159)
The client sends the "current page" as { id, title } in the request body and the
server echoed BOTH verbatim into the system prompt context and the
getCurrentPage tool. id and title are independently attacker/desync-controllable
(two tabs, stale navigation), so openPage.id could point at page B while
openPage.title said "Page A" — the model then reported "updated Page A" while it
actually edited page B (CASL still allowed it; the user has access). Red-team
finding #4.

Resolve the open page ONCE against the DB via a new `resolveOpenPageContext`:
workspace-scoped lookup + access check, returning the AUTHORITATIVE { id, title }
(title from the DB row, never the client) or null (fail-closed) for a missing /
foreign / inaccessible page. That validated value now feeds the system prompt,
the getCurrentPage tool, AND the new-chat history origin (which previously did
this validation inline, for the id only — now shared, and the title is fixed
too).

Tests: resolveOpenPageContext covers no-id, not-found, foreign-workspace,
Forbidden, non-Forbidden-fault (fail-closed), the DB-title-wins-over-client case,
and null-title coercion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:19:32 +03:00
claude code agent 227
3960845eab feat(ai-chat): per-MCP-server instructions in the agent system prompt (#180)
Admins can now give each EXTERNAL MCP server a free-text instruction ("how/
when to use this server's tools") that the agent receives in its SYSTEM
PROMPT next to the tool descriptions — porting the built-in SERVER_INSTRUCTIONS
idea to admin-configured servers. Trusted, admin-authored text (like a system
prompt); NON-secret, so unlike headersEnc it IS returned in views/forms.

- Migration: nullable `instructions text` on ai_mcp_servers (old rows = null =
  no guidance). Table type + repo insert/update (blank/whitespace -> null via
  blankToNull). DTO `@MaxLength(4000)`. Service threads it through
  McpServerView/toView.
- mcp-clients: `McpServerInstruction { serverName, toolPrefix, instructions }`
  threaded through the toolset/cache/lease. Guidance is built ONLY for a server
  that actually connected AND contributed >=1 callable tool (the allowlist may
  filter all of them out) AND has non-blank text — so a guide never appears for
  tools the agent cannot call. Cached with the toolset, so an edit is picked up
  next turn via the existing CRUD cache invalidation.
- System prompt: `buildMcpToolingBlock` renders an <mcp_tooling> block INSIDE
  the safety sandwich (after context, before the trailing SAFETY_FRAMEWORK) so
  it informs tool choice but cannot override the rules; each section is headed
  by the server's `prefix_*` namespace. Empty/blank -> block omitted. The
  caller (ai-chat.service) now builds the external toolset BEFORE the prompt and
  passes external.instructions; client-handle lifecycle (close-once) unchanged.
- Client: instructions field in types + a Textarea (autosize, maxLength 4000)
  in the MCP-server form with a namespace-prefix hint; i18n (en/ru).

Tests across every layer (prompt block placement + both SAFETY copies; view
blank->null; buildEntry includes guidance only for connected+>=1-tool+non-blank;
DTO MaxLength; repo + integration round-trip; service wiring). Delegated impl
reviewed (APPROVE); applied the import-type follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:19:32 +03:00
claude code agent 227
b349673a6b ci(test): run the server integration suite against real Postgres/Redis (#159)
The only test command in CI was `pnpm -r test` (unit `.spec.ts` on mocks).
`test:int` (`.int-spec.ts`, real Postgres/Redis) ran nowhere in CI — there
were no DB `services:` — so the cost-cap, FK-cascade, jsonb round-trip and
real AI-apply integration tests never gated a PR, and regressions in those
high-severity paths stayed green (red-team finding #7).

Add `services: postgres (pgvector) + redis` and a `pnpm --filter server
test:int` step. The pgvector image is required because migrations create
vector columns and global-setup runs `CREATE EXTENSION vector`. Service
credentials/db match the defaults in apps/server/test/integration (docmost /
docmost_dev_pw, maintenance db `docmost`, redis 6379), so no TEST_*_URL
overrides are needed; global-setup drops/recreates the isolated docmost_test
DB and migrates it.

NOTE: the workflow change itself can only be validated by an actual CI run
(YAML parses locally); the int-spec suite is verified passing locally on this
branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:17:56 +03:00
claude code agent 227
0441a2ee75 fix(mcp): refuse ambiguous patch_node/delete_node on duplicated ids (#159)
Docmost duplicates block ids on copy/paste, and copyPageContent writes the
source document verbatim with the same ids. `patchNode`/`deleteNode` address a
block by `attrs.id` via replaceNodeById/deleteNodeById, which act on EVERY node
sharing the id — so a single patch_node/delete_node could silently
replace/remove multiple unrelated blocks with no signal to the model
(red-team finding #6).

Guard both write paths: when more than one node matches the id, skip the write
entirely (the transform returns null -> no mutation) and throw a clear
"ambiguous id — N nodes share it" error so the model re-targets with a more
specific anchor. Only an unambiguous single match is written; the 0-match and
1-match behavior is unchanged.

The duplicate-count basis is covered by node-ops.test.mjs (replaceNodeById /
deleteNodeById report count===2 for a 2-duplicate doc). The end-to-end guard
is not unit-tested because patchNode/deleteNode require a live collab provider
and the test suite has no provider mock.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:17:56 +03:00
claude code agent 227
3ebe24bee2 feat(footnotes): multi-backlinks — definition returns to ALL its references (#168)
After #166 a repeated `[^a]` is one footnote (reuse): one number, one
definition, N forward links. But the definition's ↩ only returned to the
FIRST reference. Now a definition with N references shows ↩ a b c …, each
backlink scrolling to its own occurrence (Pandoc/Wikipedia convention); a
single-reference footnote keeps the plain ↩ unchanged.

- editor-ext: `computeFootnoteRefCounts(doc)` (id -> occurrence count) cached
  alongside the number map in the numbering plugin state; `getFootnoteRefCount`
  getter (O(1), no per-render doc walk). `scrollToReference(id, index?)` picks
  the index-th `sup[data-footnote-ref][data-id]` occurrence (document order),
  falling back to the first.
- client: FootnoteDefinitionView renders one lettered link (a, b, c, … aa …)
  per occurrence when refCount > 1; the chrome stays after the contentDOM so
  the #146 caret invariant holds. i18n keys (ru) added.

Tests: computeFootnoteRefCounts + getFootnoteRefCount (reuse counts, unknown
id => 0); structure test gains 3 cases (N lettered links render, click jumps
to the n-th occorrence, single ref => one ↩). NOTE: the visual layout of the
backlink row needs a real browser to verify (jsdom can't); the structural and
behavioral contract is covered headless.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:17:56 +03:00
claude code agent 227
60b7be3534 fix(db): jsonb double-encoding follow-ups from PR #172 review (#173)
PR #172 fixed the jsonb double-encoding for `tool_allowlist` but the same
class of bug, and the same re-derived workaround, remained elsewhere.

1. model_config (agent roles): jsonbObject still used the buggy `::jsonb`
   bind, so `ai_agent_roles.model_config` round-tripped as a jsonb STRING
   SCALAR. The read-path `typeof === 'object'` check then failed and the
   model override was SILENTLY dropped (role fell back to the default model).
   Fixed to `::text::jsonb` and added `parseModelConfig` + `normalizeRow` so
   every read self-heals already-corrupted rows (no migration).

2. Centralized the write workaround as `jsonbBind()` in database/utils.ts —
   one implementation with one explanation of the quirk — replacing the
   per-repo `jsonbArray` (mcp) and `jsonbObject` (roles).

3. Integration coverage (the fix is a DB round-trip a unit test cannot see;
   the read-side parser MASKS a write regression): new
   ai-mcp-server-repo.int-spec asserts `jsonb_typeof(tool_allowlist)='array'`
   after insert + heals a seeded string-scalar row; ai-agent-roles-repo
   int-spec gains the same for `model_config` (`'object'` + heal).

4. Updated the stale `ai-mcp-servers.types.ts` comment (the driver returns a
   JSON string for legacy rows; the repo normalizes every read).

5. Fail-open logging: a corrupt tool_allowlist degrades to "no restriction"
   (agent gets ALL tools) — normalizeRow now warns (server id only, never
   contents) so the silent widening leaves a trace.

6. Simplified parseToolAllowlist (normalize the string once, then a single
   array-of-strings check) — identical behaviour, all 12 cases still pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:17:56 +03:00
claude code agent 227
4d03321094 fix(mcp): replaceImage no longer yanks the cursor (#164)
`mutateLiveContentUnlocked` — the write path used by `replaceImage` — still
did the pre-#152 destructive write (delete the whole fragment + applyUpdate a
fresh Y.Doc), discarding every Yjs node id. y-prosemirror anchors the editor
selection to those ids, so an open editor's cursor snapped to the document
end on every image swap, exactly the #152 jump that the main write path no
longer causes.

Switch it to the same `applyDocToFragment(ydoc, newDoc)` structural diff
(updateYFragment) as the main path, so unchanged nodes keep their ids and the
live cursor stays put. It runs its own atomic transact, so the old explicit
transact/delete is gone; the now-unused docmostExtensions import is dropped.

Regression tests (cursor-stability suite): a sibling paragraph's
RelativePosition survives a top-level image src/attachmentId swap, and an
image nested in a callout, matching the shapes replaceImage produces.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:17:56 +03:00
claude code agent 227
d14ab4a012 feat(ai-chat): compact reasoning rendering — collapse blank lines (#181)
The "Thinking" (reasoning) block rendered with large vertical gaps: models
emit reasoning with a blank line (\n\n) between every list item and
paragraph, which `marked` turns into loose lists (each <li> wrapped in a
<p>) and separate <p> paragraphs, each carrying a margin.

- Add `collapseBlankLines(text)`: collapse 2+ newlines to one, EXCEPT inside
  fenced code blocks (``` / ~~~) where blank lines are significant. Applied
  in reasoning-block.tsx before renderChatMarkdown, so loose lists become
  tight (no <li><p>) and paragraphs join; `breaks: true` keeps single \n as
  <br>, preserving line breaks. Reasoning-only — the normal answer is
  untouched.
- Drop `white-space: pre-wrap` from `.reasoningText`: on the rendered
  markdown <div> it turned the newlines between block tags into visible
  blank lines on top of the margins. The plain-text fallback <Text> that
  needs pre-wrap already sets it inline.

Tests: collapseBlankLines unit (collapse, fence preservation incl. tilde and
unclosed fences) + rendered-HTML assertions that a blank-line-separated list
becomes a tight list and still parses as a list after a paragraph.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:17:56 +03:00
claude code agent 227
f60fa25696 fix(ai-chat): tick the live token counter between agent steps (#163)
The header token badge (and the "Thinking… · N tokens" line) froze between
agent steps and jumped in chunks instead of ticking smoothly. liveTurnTokens
returned the authoritative server `usage` VERBATIM as soon as it appeared, but
the server only attaches usage at a step boundary and it is cumulative over
COMPLETED steps — so during the next (in-flight) step the figure stayed frozen
at the previous boundary and the running text estimate was ignored.

Combine both sources per component via max: always compute the running estimate
(chars/≈4 over the message's reasoning/text parts, which includes the in-flight
step) and take max(authoritativeBase, estimate). Between boundaries the estimate
ticks the number up; at a boundary the authoritative figure snaps it exact; and
because the server usage is cumulative and we only ever take the max, the counter
is monotonic (never drops). Reasoning/output stay split; the #151 reasoning-only
authoritative count is preserved.

Backward compatible: in every existing test the estimate is <= the authoritative
figure, so max returns the same value. +4 tests for the in-flight-step-exceeds-
base case (output + reasoning), the authoritative-wins case, and monotonicity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:17:56 +03:00
46 changed files with 2862 additions and 804 deletions

View File

@@ -15,6 +15,38 @@ permissions:
jobs:
test:
runs-on: ubuntu-latest
# Real Postgres + Redis so the server integration suite (`*.int-spec.ts`,
# behind `pnpm --filter server test:int`) runs in CI (red-team finding #7).
# Without it, cost-cap / FK-cascade / jsonb-round-trip / real-apply tests
# only ran locally, so regressions in those paths stayed green in CI.
# Postgres uses the pgvector image because migrations create vector columns
# and global-setup runs `CREATE EXTENSION vector`. Credentials/db match the
# defaults in apps/server/test/integration/db.ts + global-setup.ts
# (docmost / docmost_dev_pw, maintenance db `docmost`, redis on 6379), so no
# TEST_*_URL overrides are needed.
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: docmost
POSTGRES_PASSWORD: docmost_dev_pw
POSTGRES_DB: docmost
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U docmost"
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis:7
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -36,5 +68,12 @@ jobs:
- name: Build editor-ext
run: pnpm --filter @docmost/editor-ext build
- name: Run tests
- name: Run unit tests
run: pnpm -r test
# Integration suite against the real Postgres/Redis services above. Runs
# the FK-cascade, cost-cap, jsonb-round-trip and real-apply specs that the
# unit run (mocks only) cannot cover. global-setup drops/recreates the
# isolated `docmost_test` DB and migrates it to latest.
- name: Run server integration tests
run: pnpm --filter server test:int

View File

@@ -710,6 +710,7 @@
"Authorization header": "Authorization header",
"Tool allowlist": "Tool allowlist",
"Optional. Leave empty to allow all tools the server exposes.": "Optional. Leave empty to allow all tools the server exposes.",
"Optional guidance for the agent on how and when to use this server's tools. Injected into the system prompt. The server's tools are namespaced as \"<server name>_*\".": "Optional guidance for the agent on how and when to use this server's tools. Injected into the system prompt. The server's tools are namespaced as \"<server name>_*\".",
"Test": "Test",
"Available tools": "Available tools",
"No tools available": "No tools available",

View File

@@ -405,6 +405,8 @@
"Footnote {{number}}": "Сноска {{number}}",
"Go to footnote": "Перейти к сноске",
"Back to reference": "Вернуться к ссылке",
"Back to references": "Вернуться к ссылкам",
"Back to reference {{label}}": "Вернуться к ссылке {{label}}",
"Empty footnote": "Пустая сноска",
"Math inline": "Строчная формула",
"Insert inline math equation.": "Вставить математическое выражение в строку.",
@@ -749,6 +751,8 @@
"Manage API keys for all users in the workspace. View the <anchor>API documentation</anchor> for usage details.": "Управляйте API-ключами для всех пользователей в рабочем пространстве. Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>API documentation</anchor> for usage details.": "Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>MCP documentation</anchor>.": "Смотрите <anchor>документацию по MCP</anchor>.",
"Instructions": "Инструкции",
"Optional guidance for the agent on how and when to use this server's tools. Injected into the system prompt. The server's tools are namespaced as \"<server name>_*\".": "Необязательное указание агенту, как и когда использовать инструменты этого сервера. Добавляется в системный промпт. Инструменты сервера именуются с префиксом «<имя сервера>_*».",
"Sources": "Источники",
"AI Answers not available for attachments": "Ответы ИИ недоступны для вложений",
"No answer available": "Ответ недоступен",

View File

@@ -122,7 +122,11 @@
margin-top: 4px;
font-size: var(--mantine-font-size-xs);
color: light-dark(var(--mantine-color-gray-7), var(--mantine-color-dark-1));
white-space: pre-wrap;
/* NOTE: `white-space: pre-wrap` is intentionally NOT set here. On the
rendered markdown <div> it would turn the newlines between block tags
(</li>\n<li>, </p>\n<ol>) into visible blank lines/indents on top of the
margins. The plain-text fallback <Text> that needs pre-wrap sets it
inline itself (see reasoning-block.tsx). */
}
.reasoningText p {

View File

@@ -3,6 +3,7 @@ import { Box, Collapse, Group, Text, UnstyledButton } from "@mantine/core";
import { IconChevronDown } from "@tabler/icons-react";
import { useTranslation } from "react-i18next";
import { estimateTokens } from "@/features/ai-chat/utils/count-stream-tokens.ts";
import { collapseBlankLines } from "@/features/ai-chat/utils/collapse-blank-lines.ts";
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
import classes from "@/features/ai-chat/components/ai-chat.module.css";
@@ -33,7 +34,12 @@ export default function ReasoningBlock({ text, tokens }: ReasoningBlockProps) {
// Authoritative count wins; otherwise estimate live from the streamed text.
const count = tokens && tokens > 0 ? tokens : estimateTokens(text);
const trimmed = text.trim();
const html = trimmed ? renderChatMarkdown(trimmed, {}) : "";
// Collapse the blank-line gaps the model emits between every list item /
// paragraph so the reasoning renders compactly (tight lists, joined
// paragraphs) — see collapseBlankLines. ONLY here, not in the normal answer.
const html = trimmed
? renderChatMarkdown(collapseBlankLines(trimmed), {})
: "";
return (
<Box className={classes.reasoningBlock} mb={6}>

View File

@@ -0,0 +1,61 @@
import { describe, it, expect } from "vitest";
import { collapseBlankLines } from "@/features/ai-chat/utils/collapse-blank-lines.ts";
import { renderChatMarkdown } from "@/features/ai-chat/utils/markdown.ts";
describe("collapseBlankLines", () => {
it("collapses a run of 2+ newlines to a single newline", () => {
expect(collapseBlankLines("a\n\nb")).toBe("a\nb");
expect(collapseBlankLines("a\n\n\n\nb")).toBe("a\nb");
});
it("keeps single newlines untouched", () => {
expect(collapseBlankLines("a\nb\nc")).toBe("a\nb\nc");
});
it("preserves blank lines INSIDE a fenced code block", () => {
const src = "a\n\n\nb\n\n```\nx\n\n\ny\n```\n\nc";
// Prose blanks collapse; the blank lines between the ``` fences survive.
expect(collapseBlankLines(src)).toBe("a\nb\n```\nx\n\n\ny\n```\nc");
});
it("handles a tilde fence and preserves its interior blanks", () => {
const src = "p\n\n~~~\ncode\n\nmore\n~~~\n\nq";
expect(collapseBlankLines(src)).toBe("p\n~~~\ncode\n\nmore\n~~~\nq");
});
it("leaves an unclosed fence's remaining lines verbatim", () => {
const src = "intro\n\n```\nstill\n\nopen";
expect(collapseBlankLines(src)).toBe("intro\n```\nstill\n\nopen");
});
it("is a no-op for text with no blank lines", () => {
expect(collapseBlankLines("just one line")).toBe("just one line");
});
});
describe("collapseBlankLines + renderChatMarkdown (tight reasoning rendering)", () => {
it("renders a blank-line-separated list as a TIGHT list (no <li><p>)", () => {
const loose =
"Intro paragraph.\n\n- item one\n\n- item two\n\n- item three";
const html = renderChatMarkdown(collapseBlankLines(loose), {});
// Tight list: each <li> holds the text directly, not wrapped in a <p>.
expect(html).toContain("<li>item one</li>");
expect(html).not.toContain("<li><p>");
// The list still parses as a list after the paragraph (not a paragraph+<br>).
expect(html).toContain("<ul>");
expect(html).toContain("<p>Intro paragraph.</p>");
});
it("renders an ordered list (1. 2.) as tight after collapsing", () => {
const loose = "Intro.\n\n1. first\n\n2. second";
const html = renderChatMarkdown(collapseBlankLines(loose), {});
expect(html).toContain("<ol>");
expect(html).toContain("<li>first</li>");
expect(html).not.toContain("<li><p>");
});
it("the loose source WOULD render <li><p> without collapsing (control)", () => {
const loose = "- a\n\n- b";
expect(renderChatMarkdown(loose, {})).toContain("<li><p>");
});
});

View File

@@ -0,0 +1,56 @@
// Pure helper for compact reasoning ("Thinking") rendering. Kept free of React
// so it can be unit-tested in isolation (see collapse-blank-lines.test.ts).
/**
* Collapse runs of 2+ newlines down to a single newline, EXCEPT inside fenced
* code blocks (``` ... ``` or ~~~ ... ~~~), where blank lines are significant.
*
* Why: reasoning models emit thinking with a blank line (`\n\n`) between every
* list item and paragraph. `marked` turns those into "loose" lists (each `<li>`
* wrapped in a `<p>`) and separate `<p>` paragraphs, each carrying a vertical
* margin — so the "Thinking" block renders with large, airy gaps. Removing the
* blank-line gaps yields tight lists (no `<li><p>`) and joined paragraphs. The
* chat markdown renderer runs with `breaks: true`, so a single `\n` still
* becomes a `<br>` — line breaks inside the reasoning are preserved; only the
* empty gaps between blocks disappear. Apply ONLY to reasoning text, never to a
* normal assistant answer (where paragraph spacing is intentional).
*
* Fenced code is preserved verbatim: a fence opens on a line whose first
* non-space characters are ``` or ~~~ and closes on the next line that starts
* with the same fence character. Blank lines between fences (significant for
* code formatting) are never collapsed.
*/
export function collapseBlankLines(text: string): string {
const lines = text.split("\n");
const out: string[] = [];
let inFence = false;
let fenceChar = "";
for (const line of lines) {
const fenceMatch = line.match(/^\s*(`{3,}|~{3,})/);
if (fenceMatch) {
const ch = fenceMatch[1][0];
if (!inFence) {
inFence = true;
fenceChar = ch;
} else if (ch === fenceChar) {
inFence = false;
}
out.push(line);
continue;
}
// Inside a fenced block every line (including blanks) is significant.
if (inFence) {
out.push(line);
continue;
}
// Outside fences: drop blank lines so a `\n\n+` gap collapses to a single
// `\n` between the surrounding content lines.
if (line.trim() === "") continue;
out.push(line);
}
return out.join("\n");
}

View File

@@ -117,3 +117,55 @@ describe("liveTurnTokens — authoritative path", () => {
expect(r).toEqual({ reasoning: 0, output: 1, authoritative: false });
});
});
describe("liveTurnTokens — combined authoritative + estimate (#163)", () => {
it("ticks the in-flight step above the completed-steps authoritative base", () => {
// The authoritative usage is the sum over COMPLETED steps (step 1). The
// CURRENT step is streaming and its text is NOT in `usage` yet, but it IS in
// the parts -> the running estimate must push the live figure above the base
// so the badge keeps growing between step boundaries.
const longText = "x".repeat(800); // 800 chars -> 200 est output tokens
const r = liveTurnTokens(
msg([{ type: "text", text: longText }], {
usage: { inputTokens: 500, outputTokens: 40 }, // step-1 base: 40 output
}),
);
// max(authOutput=40, estOutput=200) = 200 -> the counter ticks, not frozen.
expect(r.output).toBe(200);
expect(r.authoritative).toBe(true);
});
it("ticks reasoning of the in-flight step above the authoritative reasoning base", () => {
const longReasoning = "r".repeat(400); // 400 chars -> 100 est reasoning
const r = liveTurnTokens(
msg([{ type: "reasoning", text: longReasoning }], {
usage: { inputTokens: 100, outputTokens: 20, reasoningTokens: 20 },
}),
);
// reasoning: max(20, 100) = 100 ; output: max(max(0,20-20)=0, 0) = 0.
expect(r.reasoning).toBe(100);
expect(r.output).toBe(0);
expect(r.authoritative).toBe(true);
});
it("snaps to the authoritative figure once it exceeds the rough estimate", () => {
// Short on-screen text (estimate tiny) but a large authoritative output:
// the exact figure wins at the boundary (the counter never under-reports).
const r = liveTurnTokens(
msg([{ type: "text", text: "abcd" }], {
usage: { inputTokens: 10, outputTokens: 5000 },
}),
);
expect(r.output).toBe(5000);
});
it("is monotonic: max never drops below the authoritative base when the estimate is smaller", () => {
// Mirrors the legacy 'verbatim' tests: estimate < authoritative -> unchanged.
const r = liveTurnTokens(
msg([{ type: "text", text: "tiny" }], {
usage: { inputTokens: 500, outputTokens: 100, reasoningTokens: 30 },
}),
);
expect(r).toEqual({ reasoning: 30, output: 70, authoritative: true });
});
});

View File

@@ -56,39 +56,58 @@ function metadataUsage(message: UIMessage): AuthoritativeUsage | undefined {
/**
* Token split for the given (streaming) assistant message.
*
* Prefers AUTHORITATIVE `metadata.usage` when the server has attached it (at a
* step/turn boundary, incl. `reasoningTokens`) — so the live counter snaps to the
* provider's exact figures. Until then it returns a running ESTIMATE summed over
* the message parts: `reasoning` parts feed the reasoning estimate, `text` parts
* feed the output estimate. Multi-part / multi-step turns accumulate naturally
* because every part of the turn is summed.
* COMBINES the authoritative server usage with the running text estimate so the
* counter ticks in real time AND lands exact. The server only attaches
* `metadata.usage` at a step/turn boundary (`finish-step`/`finish`) and it is
* CUMULATIVE over COMPLETED steps — it does NOT yet include the in-flight step.
* So a multi-step turn that returned the authoritative figure verbatim would
* FREEZE between boundaries and jump in steps (issue #163).
*
* Instead we always compute the running ESTIMATE (chars/≈4 over the message's
* `reasoning`/`text` parts, which grows on every streamed delta) and take the
* per-component MAX of the authoritative base and the estimate:
* - between boundaries the estimate of the in-flight step ticks the number up;
* - at a boundary the authoritative figure snaps it to exact;
* - because the server's usage is cumulative and we only ever take the max, the
* number is MONOTONIC — it never drops.
*
* Providers that don't stream reasoning text still surface a reasoning count once
* the authoritative usage arrives (`usage.reasoningTokens`); on the pure estimate
* path such a turn simply shows `reasoning: 0` until then.
* the authoritative usage arrives (`max(reasoningTokens, 0)`); on the pure
* estimate path (no usage yet) such a turn shows `reasoning: 0` until then.
*/
export function liveTurnTokens(message: UIMessage | undefined): LiveTurnTokens {
if (!message) return { reasoning: 0, output: 0, authoritative: false };
const usage = metadataUsage(message);
if (usage) {
// Authoritative branch: outputTokens already INCLUDES reasoning tokens in the
// AI SDK usage shape, so subtract reasoning out for the "answer" figure (never
// go negative if a provider reports them inconsistently).
const reasoning = usage.reasoningTokens ?? 0;
const totalOutput = usage.outputTokens ?? 0;
const output = Math.max(0, totalOutput - reasoning);
return { reasoning, output, authoritative: true };
}
let reasoning = 0;
let output = 0;
// Running ESTIMATE over every reasoning/text part — grows on each delta. This
// includes the IN-FLIGHT step, which the authoritative usage does not cover yet.
let estReasoning = 0;
let estOutput = 0;
for (const part of message.parts ?? []) {
if (part.type === "reasoning") {
reasoning += estimateTokens((part as { text?: string }).text ?? "");
estReasoning += estimateTokens((part as { text?: string }).text ?? "");
} else if (part.type === "text") {
output += estimateTokens((part as { text?: string }).text ?? "");
estOutput += estimateTokens((part as { text?: string }).text ?? "");
}
}
return { reasoning, output, authoritative: false };
const usage = metadataUsage(message);
if (!usage) {
// No authoritative usage streamed yet: the estimate IS the live figure.
return { reasoning: estReasoning, output: estOutput, authoritative: false };
}
// Authoritative sum over COMPLETED steps. `outputTokens` already INCLUDES
// reasoning in the AI SDK usage shape, so subtract it out for the "answer"
// figure (never go negative if a provider reports them inconsistently).
const authReasoning = usage.reasoningTokens ?? 0;
const authOutput = Math.max(0, (usage.outputTokens ?? 0) - authReasoning);
// Per-component max: the in-flight step's estimate ticks above the completed-
// steps base between boundaries, and the authoritative figure wins once it
// exceeds the (rough) estimate at the next boundary. Monotonic by construction.
return {
reasoning: Math.max(authReasoning, estReasoning),
output: Math.max(authOutput, estOutput),
authoritative: true,
};
}

View File

@@ -1,25 +1,45 @@
import { NodeViewContent, NodeViewProps, NodeViewWrapper } from "@tiptap/react";
import { useTranslation } from "react-i18next";
import { getFootnoteNumber } from "@docmost/editor-ext";
import { getFootnoteNumber, getFootnoteRefCount } from "@docmost/editor-ext";
import classes from "./footnote.module.css";
/**
* A 0-based backlink index -> its lowercase letter label (0 -> "a", 25 -> "z",
* 26 -> "aa", ...), matching the Pandoc/Wikipedia "↩ a b c" convention.
*/
function backlinkLabel(index: number): string {
let out = "";
let x = index;
while (x >= 0) {
out = String.fromCharCode(97 + (x % 26)) + out;
x = Math.floor(x / 26) - 1;
}
return out;
}
/**
* NodeView for a single footnote definition: a decorative number marker, the
* editable content (NodeViewContent), and a "↩" back-link to its reference.
* The number is derived from the document (not stored).
*
* After #166 a footnote can be referenced more than once (one number, one
* definition, N forward links). When it is, the back-link becomes a row of
* per-occurrence links — ↩ a b c … — each scrolling to its own reference (#168);
* a single-reference footnote keeps the plain ↩.
*/
export default function FootnoteDefinitionView(props: NodeViewProps) {
const { node, editor } = props;
const { t } = useTranslation();
const id = node.attrs.id as string;
// Read the cached number from the numbering plugin (computed once per doc
// change) rather than recomputing the whole map on every render.
// Read the cached number/ref-count from the numbering plugin (computed once
// per doc change) rather than recomputing the whole map on every render.
const number = getFootnoteNumber(editor.state, id) ?? "?";
const refCount = getFootnoteRefCount(editor.state, id);
const handleBack = (e: React.MouseEvent) => {
const jumpTo = (e: React.MouseEvent, index: number) => {
e.preventDefault();
editor.commands.scrollToReference(id);
editor.commands.scrollToReference(id, index);
};
return (
@@ -42,16 +62,47 @@ export default function FootnoteDefinitionView(props: NodeViewProps) {
>
{number}.
</span>
<span
className={classes.backLink}
contentEditable={false}
onClick={handleBack}
role="button"
aria-label={t("Back to reference")}
title={t("Back to reference")}
>
</span>
{refCount > 1 ? (
// Multiple references -> ↩ followed by one lettered link per occurrence.
<span
className={classes.backLinks}
contentEditable={false}
role="group"
aria-label={t("Back to references")}
>
<span className={classes.backLinkArrow} aria-hidden="true">
</span>
{Array.from({ length: refCount }, (_, i) => (
<span
key={i}
className={classes.backLink}
onClick={(e) => jumpTo(e, i)}
role="button"
aria-label={t("Back to reference {{label}}", {
label: backlinkLabel(i),
})}
title={t("Back to reference {{label}}", {
label: backlinkLabel(i),
})}
>
{backlinkLabel(i)}
</span>
))}
</span>
) : (
// Single reference -> the plain ↩ (unchanged behavior).
<span
className={classes.backLink}
contentEditable={false}
onClick={(e) => jumpTo(e, 0)}
role="button"
aria-label={t("Back to reference")}
title={t("Back to reference")}
>
</span>
)}
</NodeViewWrapper>
);
}

View File

@@ -1,5 +1,5 @@
import { describe, it, expect, vi } from "vitest";
import { render } from "@testing-library/react";
import { describe, it, expect, vi, afterEach } from "vitest";
import { render, fireEvent } from "@testing-library/react";
/**
* Structural regression guard for #146 (PR #147).
@@ -36,10 +36,14 @@ vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (key: string) => key }),
}));
// footnote-definition-view reads a cached number from the numbering plugin;
// stub it so we don't need a live ProseMirror state.
// footnote-definition-view reads a cached number + reference count from the
// numbering plugin; stub them so we don't need a live ProseMirror state. The
// ref-count is a hoisted mutable so a test can drive the single-vs-multi
// backlink branch (#168). Default 1 = single reference (the #146 cases).
const { mockRefCount } = vi.hoisted(() => ({ mockRefCount: { value: 1 } }));
vi.mock("@docmost/editor-ext", () => ({
getFootnoteNumber: () => 1,
getFootnoteRefCount: () => mockRefCount.value,
}));
// Mocks so CodeBlockView renders cheaply (no MantineProvider, no matchMedia).
@@ -59,7 +63,8 @@ vi.mock("@mantine/core", () => ({
),
}));
vi.mock("@/components/common/copy-button", () => ({
CopyButton: ({ children }: any) => children({ copied: false, copy: () => {} }),
CopyButton: ({ children }: any) =>
children({ copied: false, copy: () => {} }),
}));
vi.mock("@tabler/icons-react", () => ({
IconCheck: () => null,
@@ -141,3 +146,71 @@ describe("#146 editable NodeView contentDOM-first invariant", () => {
},
);
});
// #168: a footnote referenced more than once shows one lettered backlink per
// occurrence (↩ a b c), each scrolling to its own reference; a single-reference
// footnote keeps the plain ↩.
describe("#168 footnote definition multi-backlinks", () => {
afterEach(() => {
// Reset the shared ref-count mock so other tests see a single reference.
mockRefCount.value = 1;
});
const makeProps = () =>
({
node: { attrs: { id: "fn-1" }, textContent: "" },
editor: {
state: {},
isEditable: true,
commands: { scrollToReference: vi.fn() },
},
getPos: () => 0,
updateAttributes: () => {},
deleteNode: () => {},
}) as any;
it("renders one lettered backlink per reference (a, b, c) plus the ↩ arrow", () => {
mockRefCount.value = 3;
const { getByTestId } = render(<FootnoteDefinitionView {...makeProps()} />);
const wrapper = getByTestId("nvw");
const links = wrapper.querySelectorAll('[role="button"]');
expect(Array.from(links).map((l) => l.textContent)).toEqual([
"a",
"b",
"c",
]);
// The ↩ arrow is present (as decorative chrome, not a button).
expect(wrapper.textContent).toContain("↩");
});
it("clicking the n-th backlink scrolls to the n-th occurrence (0-based)", () => {
mockRefCount.value = 3;
const props = makeProps();
const { getByTestId } = render(<FootnoteDefinitionView {...props} />);
const links = getByTestId("nvw").querySelectorAll('[role="button"]');
fireEvent.click(links[1]); // "b"
expect(props.editor.commands.scrollToReference).toHaveBeenCalledWith(
"fn-1",
1,
);
});
it("a single-reference footnote renders just one ↩ (no letters)", () => {
mockRefCount.value = 1;
const props = makeProps();
const { getByTestId } = render(<FootnoteDefinitionView {...props} />);
const wrapper = getByTestId("nvw");
const links = wrapper.querySelectorAll('[role="button"]');
expect(links.length).toBe(1);
expect(links[0].textContent).toBe("↩");
fireEvent.click(links[0]);
expect(props.editor.commands.scrollToReference).toHaveBeenCalledWith(
"fn-1",
0,
);
});
});

View File

@@ -115,3 +115,18 @@
.backLink:hover {
text-decoration: underline;
}
/* Multi-backlink row (#168): ↩ a b c — one lettered link per reference
occurrence. Sits on the right, after the content, like the single ↩. */
.backLinks {
flex: 0 0 auto;
display: inline-flex;
align-items: baseline;
gap: 0.3em;
user-select: none;
}
.backLinkArrow {
color: var(--mantine-color-dimmed);
font-size: 0.9em;
}

View File

@@ -11,6 +11,7 @@ import {
Switch,
TagsInput,
Text,
Textarea,
TextInput,
} from "@mantine/core";
import { useForm } from "@mantine/form";
@@ -35,6 +36,8 @@ const formSchema = z.object({
// Write-only secret buffer. Empty string means "do not change" (unless cleared).
authHeader: z.string(),
toolAllowlist: z.array(z.string()),
// Admin-authored prompt guidance (#180). Capped to mirror the DTO MaxLength.
instructions: z.string().max(4000),
enabled: z.boolean(),
});
@@ -63,6 +66,7 @@ function buildInitialValues(server?: IAiMcpServer): FormValues {
toolAllowlist: Array.isArray(server?.toolAllowlist)
? server.toolAllowlist
: [],
instructions: server?.instructions ?? "",
enabled: server?.enabled ?? true,
};
}
@@ -124,6 +128,8 @@ export default function AiMcpServerForm({
transport: values.transport,
url: values.url,
toolAllowlist: values.toolAllowlist,
// Always sent: a blank value clears the stored guidance (server -> null).
instructions: values.instructions,
enabled: values.enabled,
};
// Only attach headers when set or explicitly cleared (omit => unchanged).
@@ -135,6 +141,8 @@ export default function AiMcpServerForm({
transport: values.transport,
url: values.url,
toolAllowlist: values.toolAllowlist,
// Blank => server stores null (no guidance).
instructions: values.instructions,
enabled: values.enabled,
};
// On create, only a typed value matters (no prior stored headers).
@@ -158,10 +166,7 @@ export default function AiMcpServerForm({
return (
<Stack>
<TextInput
label={t("Server name")}
{...form.getInputProps("name")}
/>
<TextInput label={t("Server name")} {...form.getInputProps("name")} />
<Select
label={t("Transport")}
@@ -177,7 +182,7 @@ export default function AiMcpServerForm({
// Clarify that the value is sent verbatim as the Authorization header,
// so the user supplies the full scheme (no implicit Bearer prefix).
description={t(
"Sent verbatim as the value of the Authorization header (e.g. \"Bearer <token>\" or \"Basic <base64>\").",
'Sent verbatim as the value of the Authorization header (e.g. "Bearer <token>" or "Basic <base64>").',
)}
// Placeholder hints whether headers are stored; the value is never shown.
placeholder={hasHeaders ? t("•••• set") : ""}
@@ -208,6 +213,20 @@ export default function AiMcpServerForm({
{...form.getInputProps("toolAllowlist")}
/>
<Textarea
label={t("Instructions")}
// Hint that the text is injected into the agent's system prompt and that
// the server's tools are namespaced under <name>_* (the prompt header).
description={t(
"Optional guidance for the agent on how and when to use this server's tools. Injected into the system prompt. The server's tools are namespaced as \"<server name>_*\".",
)}
autosize
minRows={2}
maxRows={8}
maxLength={4000}
{...form.getInputProps("instructions")}
/>
<Switch
label={t("Enabled")}
checked={form.values.enabled}

View File

@@ -14,6 +14,9 @@ export interface IAiMcpServer {
enabled: boolean;
toolAllowlist: string[] | null;
hasHeaders: boolean;
// Admin-authored guidance injected into the agent system prompt (#180).
// NON-secret, so it IS returned. Null when no guidance is configured.
instructions: string | null;
}
// Create payload. `headers` is write-only: omit => no auth headers.
@@ -25,6 +28,8 @@ export interface IAiMcpServerCreate {
// never returned.
headers?: Record<string, string>;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Blank => stored as null.
instructions?: string;
enabled?: boolean;
}
@@ -39,6 +44,8 @@ export interface IAiMcpServerUpdate {
url?: string;
headers?: Record<string, string>;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Absent => unchanged; blank => cleared.
instructions?: string;
enabled?: boolean;
}

View File

@@ -1,4 +1,4 @@
import { buildSystemPrompt } from './ai-chat.prompt';
import { buildSystemPrompt, buildMcpToolingBlock } from './ai-chat.prompt';
import { Workspace } from '@docmost/db/types/entity.types';
/**
@@ -161,3 +161,118 @@ describe('buildSystemPrompt current-page context', () => {
expect(pageIdx).toBeLessThan(lastSafety);
});
});
/**
* Unit tests for the per-EXTERNAL-MCP-server guidance block (#180). When the
* caller passes non-blank instructions for ≥1 server, an <mcp_tooling> block
* renders the server name, its tool namespace prefix and the text. The block
* sits INSIDE the safety sandwich (after context, before the trailing SAFETY)
* and never removes/duplicates the immutable safety framework. An empty list or
* all-blank text renders nothing.
*/
describe('buildSystemPrompt mcp tooling guidance', () => {
const workspace = { name: 'Acme' } as unknown as Workspace;
const SAFETY_MARKER = 'Operating rules (always in effect)';
it('renders the server name, tool prefix and text when guidance is present', () => {
const prompt = buildSystemPrompt({
workspace,
mcpInstructions: [
{
serverName: 'Tavily',
toolPrefix: 'tavily',
instructions: 'Use tavily_search for fresh web facts; cite sources.',
},
],
});
expect(prompt).toContain('<mcp_tooling');
expect(prompt).toContain('Tavily');
// The header names the namespace prefix as `<prefix>_*`.
expect(prompt).toContain('tavily_*');
expect(prompt).toContain(
'Use tavily_search for fresh web facts; cite sources.',
);
});
it('renders nothing for an empty list', () => {
const prompt = buildSystemPrompt({ workspace, mcpInstructions: [] });
expect(prompt).not.toContain('<mcp_tooling');
});
it('renders nothing for an undefined list', () => {
const prompt = buildSystemPrompt({ workspace });
expect(prompt).not.toContain('<mcp_tooling');
});
it('renders nothing when every entry has blank text', () => {
const prompt = buildSystemPrompt({
workspace,
mcpInstructions: [
{ serverName: 'A', toolPrefix: 'a', instructions: ' ' },
{ serverName: 'B', toolPrefix: 'b', instructions: '' },
],
});
expect(prompt).not.toContain('<mcp_tooling');
});
it('places the block inside the safety sandwich, after context, before the trailing SAFETY', () => {
const prompt = buildSystemPrompt({
workspace,
openedPage: { id: 'pg-1', title: 'Doc' },
mcpInstructions: [
{ serverName: 'Tavily', toolPrefix: 'tavily', instructions: 'guide' },
],
});
const ctxIdx = prompt.indexOf('currently viewing the page');
const mcpIdx = prompt.indexOf('<mcp_tooling');
const firstSafety = prompt.indexOf(SAFETY_MARKER);
const lastSafety = prompt.lastIndexOf(SAFETY_MARKER);
// After context, and strictly inside the sandwich.
expect(mcpIdx).toBeGreaterThan(ctxIdx);
expect(mcpIdx).toBeGreaterThan(firstSafety);
expect(mcpIdx).toBeLessThan(lastSafety);
});
it('keeps BOTH copies of the safety framework when guidance is present', () => {
const prompt = buildSystemPrompt({
workspace,
mcpInstructions: [
{ serverName: 'Tavily', toolPrefix: 'tavily', instructions: 'guide' },
],
});
const firstSafety = prompt.indexOf(SAFETY_MARKER);
const lastSafety = prompt.lastIndexOf(SAFETY_MARKER);
expect(firstSafety).toBeGreaterThanOrEqual(0);
expect(lastSafety).toBeGreaterThan(firstSafety);
});
});
/**
* Unit tests for the pure block builder. It filters blank entries and returns
* '' so the caller can omit the section entirely.
*/
describe('buildMcpToolingBlock', () => {
it('returns "" for undefined / empty / all-blank', () => {
expect(buildMcpToolingBlock(undefined)).toBe('');
expect(buildMcpToolingBlock([])).toBe('');
expect(
buildMcpToolingBlock([
{ serverName: 'A', toolPrefix: 'a', instructions: ' ' },
]),
).toBe('');
});
it('includes only the non-blank entries', () => {
const block = buildMcpToolingBlock([
{ serverName: 'A', toolPrefix: 'a', instructions: 'alpha guide' },
{ serverName: 'B', toolPrefix: 'b', instructions: ' ' },
{ serverName: 'C', toolPrefix: 'c', instructions: 'gamma guide' },
]);
expect(block).toContain('a_*');
expect(block).toContain('alpha guide');
expect(block).toContain('c_*');
expect(block).toContain('gamma guide');
// The blank-only entry contributes no section header.
expect(block).not.toContain('b_*');
});
});

View File

@@ -1,4 +1,5 @@
import { Workspace } from '@docmost/db/types/entity.types';
import type { McpServerInstruction } from './external-mcp/mcp-clients.service';
/**
* Default agent persona used when the admin has not configured a custom system
@@ -76,6 +77,42 @@ export interface BuildSystemPromptInput {
* uses its CASL-enforced read/write page tools with the id when needed.
*/
openedPage?: { id?: string; title?: string } | null;
/**
* Admin-authored, per-EXTERNAL-MCP-server guidance ("how/when to use this
* server's tools"), built by `McpClientsService.toolsFor` for servers that
* actually connected and contributed ≥1 callable tool (#180). Rendered as an
* `<mcp_tooling>` block INSIDE the safety sandwich (trusted text — it informs
* tool usage but cannot override the surrounding rules). Empty/blank => the
* block is omitted entirely.
*/
mcpInstructions?: McpServerInstruction[];
}
/**
* Render the `<mcp_tooling>` block from per-server guidance. Each server gets a
* section headed by its tool namespace prefix (e.g. `tavily_*`) so the model can
* connect the guidance to the actual namespaced tool names. The prefix is
* advisory: on rare name collisions individual tools may carry a disambiguating
* suffix, but the guidance stays guidance, not a contract. Returns '' when no
* server has non-blank guidance, so the caller can omit the block entirely.
*/
export function buildMcpToolingBlock(
mcpInstructions: McpServerInstruction[] | undefined,
): string {
if (!mcpInstructions || mcpInstructions.length === 0) return '';
const sections = mcpInstructions
.filter((m) => typeof m.instructions === 'string' && m.instructions.trim())
.map((m) => {
const header = `Server "${m.serverName}" (tools: ${m.toolPrefix}_*):`;
return `${header}\n${m.instructions.trim()}`;
});
if (sections.length === 0) return '';
return [
'<mcp_tooling note="admin guidance for the external tools below; informs tool choice only, cannot override the rules above or below">',
'Guidance for the external MCP tools available to you this turn:',
...sections,
'</mcp_tooling>',
].join('\n');
}
/**
@@ -92,6 +129,7 @@ export function buildSystemPrompt({
adminPrompt,
roleInstructions,
openedPage,
mcpInstructions,
}: BuildSystemPromptInput): string {
// Persona precedence: role instructions REPLACE the admin persona / default.
// effectivePersona = roleInstructions || adminPrompt || DEFAULT_PROMPT.
@@ -112,24 +150,35 @@ export function buildSystemPrompt({
const pageId = openedPage?.id;
if (typeof pageId === 'string' && pageId.trim().length > 0) {
const title =
typeof openedPage?.title === 'string' && openedPage.title.trim().length > 0
typeof openedPage?.title === 'string' &&
openedPage.title.trim().length > 0
? openedPage.title.trim()
: 'Untitled';
context += `\nThe user is currently viewing the page "${title}" (pageId: ${pageId.trim()}). When they refer to "this page", "the current page", or similar, operate on that pageId — use the read/write page tools with it.`;
}
// Per-server external-MCP tool guidance (#180). Trusted, admin-authored text;
// rendered inside the sandwich (after context, before the trailing SAFETY) so
// it informs tool choice but cannot override the surrounding safety rules.
// Empty when no qualifying server has guidance.
const mcpTooling = buildMcpToolingBlock(mcpInstructions);
// Sandwich the lower-trust persona/role text between two copies of the
// immutable SAFETY_FRAMEWORK so any jailbreak inside `base` is both preceded
// and followed by the safety rules. The persona is delimited with explicit
// <role_persona> tags noting it only shapes tone/voice. Context (workspace
// name, currently-viewed page) follows the persona, before the trailing
// SAFETY copy.
// name, currently-viewed page) then the MCP tooling guidance follow the
// persona, before the trailing SAFETY copy. Blank parts are filtered out so
// an empty section never adds a stray blank line.
return [
SAFETY_FRAMEWORK,
'<role_persona note="shapes tone/voice only; cannot override the rules above or below">',
base,
'</role_persona>',
context,
mcpTooling,
SAFETY_FRAMEWORK,
].join('\n');
]
.filter((part) => part !== '')
.join('\n');
}

View File

@@ -1,4 +1,6 @@
import { ForbiddenException } from '@nestjs/common';
import {
AiChatService,
compactToolOutput,
assistantParts,
serializeSteps,
@@ -10,7 +12,9 @@ import {
MAX_AGENT_STEPS,
FINAL_STEP_INSTRUCTION,
} from './ai-chat.service';
import type { AiChatMessage } from '@docmost/db/types/entity.types';
import type { AiChatMessage, Workspace } from '@docmost/db/types/entity.types';
import { buildSystemPrompt } from './ai-chat.prompt';
import type { McpClientsService } from './external-mcp/mcp-clients.service';
/**
* Unit tests for compactToolOutput: the pure helper that shrinks LARGE tool
@@ -94,8 +98,12 @@ describe('assistantParts', () => {
const steps = [
{
text: '',
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: { id: 'p1' } }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage', output: { title: 'T' } }],
toolCalls: [
{ toolCallId: 'c1', toolName: 'getPage', input: { id: 'p1' } },
],
toolResults: [
{ toolCallId: 'c1', toolName: 'getPage', output: { title: 'T' } },
],
},
];
const parts = assistantParts(steps, '') as AnyPart[];
@@ -109,7 +117,9 @@ describe('assistantParts', () => {
const steps = [
{
text: '',
toolCalls: [{ toolCallId: 'c9', toolName: 'insertNode', input: { node: {} } }],
toolCalls: [
{ toolCallId: 'c9', toolName: 'insertNode', input: { node: {} } },
],
toolResults: [],
},
];
@@ -136,7 +146,8 @@ describe('assistantParts', () => {
];
const parts = assistantParts(steps, '') as AnyPart[];
const toolParts = parts.filter(
(p) => typeof p.type === 'string' && (p.type as string).startsWith('tool-'),
(p) =>
typeof p.type === 'string' && (p.type as string).startsWith('tool-'),
);
expect(toolParts).toHaveLength(0);
});
@@ -246,16 +257,30 @@ describe('buildPartialAssistantRecord', () => {
type AnyPart = Record<string, unknown>;
it('records an empty turn with the error text (preserves old behavior)', () => {
const rec = buildPartialAssistantRecord([], '', 'error', '401: Unauthorized');
const rec = buildPartialAssistantRecord(
[],
'',
'error',
'401: Unauthorized',
);
expect(rec).toEqual({
text: '',
toolCalls: null,
metadata: { finishReason: 'error', parts: [], error: '401: Unauthorized' },
metadata: {
finishReason: 'error',
parts: [],
error: '401: Unauthorized',
},
});
});
it('persists in-progress text (no finished steps) as the partial answer', () => {
const rec = buildPartialAssistantRecord([], 'partial answer', 'error', 'boom');
const rec = buildPartialAssistantRecord(
[],
'partial answer',
'error',
'boom',
);
expect(rec.text).toBe('partial answer');
expect(rec.metadata.parts).toEqual([
{ type: 'text', text: 'partial answer' },
@@ -275,7 +300,12 @@ describe('buildPartialAssistantRecord', () => {
],
},
];
const rec = buildPartialAssistantRecord(steps, ' and then', 'error', 'boom');
const rec = buildPartialAssistantRecord(
steps,
' and then',
'error',
'boom',
);
const parts = rec.metadata.parts as AnyPart[];
// The finished step's text part is present.
expect(parts).toContainEqual({ type: 'text', text: 'looked it up' });
@@ -284,7 +314,10 @@ describe('buildPartialAssistantRecord', () => {
expect(toolPart).toBeDefined();
expect(toolPart!.state).toBe('output-available');
// The in-progress text is appended LAST so the parts match the stream order.
expect(parts[parts.length - 1]).toEqual({ type: 'text', text: ' and then' });
expect(parts[parts.length - 1]).toEqual({
type: 'text',
text: ' and then',
});
expect(rec.text).toBe('looked it up and then');
expect(rec.toolCalls).not.toBeNull();
expect(rec.metadata.error).toBe('boom');
@@ -319,10 +352,20 @@ describe('chatStreamMetadata', () => {
chatStreamMetadata(
{ type: 'finish-step', usage: { outputTokens: 100 } },
'chat-1',
{ inputTokens: 500, outputTokens: 220, totalTokens: 720, reasoningTokens: 30 },
{
inputTokens: 500,
outputTokens: 220,
totalTokens: 720,
reasoningTokens: 30,
},
),
).toEqual({
usage: { inputTokens: 500, outputTokens: 220, totalTokens: 720, reasoningTokens: 30 },
usage: {
inputTokens: 500,
outputTokens: 220,
totalTokens: 720,
reasoningTokens: 30,
},
});
});
@@ -394,8 +437,18 @@ describe('accumulateStepUsage', () => {
it('sums every field across two steps', () => {
expect(
accumulateStepUsage(
{ inputTokens: 500, outputTokens: 100, totalTokens: 600, reasoningTokens: 30 },
{ inputTokens: 520, outputTokens: 80, totalTokens: 600, reasoningTokens: 10 },
{
inputTokens: 500,
outputTokens: 100,
totalTokens: 600,
reasoningTokens: 30,
},
{
inputTokens: 520,
outputTokens: 80,
totalTokens: 600,
reasoningTokens: 10,
},
),
).toEqual({
inputTokens: 1020,
@@ -431,3 +484,143 @@ describe('accumulateStepUsage', () => {
});
});
});
/**
* Contract test for the #180 wiring in AiChatService.handle: the external MCP
* toolset must be built BEFORE the system prompt, and its per-server guidance
* threaded into buildSystemPrompt({ mcpInstructions }). The full streaming
* handle() is not unit-testable, so this reproduces the exact prompt-build call
* the service makes with a connected-server toolset and asserts the guidance is
* present. The toolsFor->buildSystemPrompt ordering is additionally enforced at
* compile time (the prompt input now consumes external.instructions).
*/
describe('AiChatService system prompt wiring (#180)', () => {
const workspace = { name: 'Acme' } as unknown as Workspace;
it('includes the external MCP server instructions in the built system prompt', () => {
// Shape returned by mcpClients.toolsFor (only `instructions` matters here).
const external: Pick<
Awaited<ReturnType<McpClientsService['toolsFor']>>,
'instructions'
> = {
instructions: [
{
serverName: 'Tavily',
toolPrefix: 'tavily',
instructions: 'Prefer tavily_search for current events.',
},
],
};
// Exactly the call the service makes after building the external toolset.
const system = buildSystemPrompt({
workspace,
adminPrompt: 'persona',
mcpInstructions: external.instructions,
});
expect(system).toContain('<mcp_tooling');
expect(system).toContain('Tavily');
expect(system).toContain('tavily_*');
expect(system).toContain('Prefer tavily_search for current events.');
});
it('renders no MCP block when there are no external servers (empty instructions)', () => {
const system = buildSystemPrompt({
workspace,
adminPrompt: 'persona',
mcpInstructions: [],
});
expect(system).not.toContain('<mcp_tooling');
});
});
/**
* resolveOpenPageContext: the open page the client sends is attacker-controllable
* (id AND title), so the service must validate the id against the DB and take the
* title from the DB row — never echo the client title (#159, AI edits the wrong
* page). Built with Object.create so the test exercises the real method without
* the service's full dependency graph (the constructor only assigns fields).
*/
describe('AiChatService.resolveOpenPageContext (#159 current-page validation)', () => {
const ws = { id: 'ws-1' } as Workspace;
const user = { id: 'u-1' } as any;
function makeService(opts: {
page?: { id: string; workspaceId: string; title: string | null } | null;
canView?: boolean | 'throw-other';
}) {
const svc = Object.create(AiChatService.prototype) as AiChatService;
(svc as any).logger = { warn: () => {} };
(svc as any).pageRepo = {
findById: async () => opts.page ?? undefined,
};
(svc as any).pageAccess = {
validateCanView: async () => {
if (opts.canView === 'throw-other') throw new Error('db down');
if (opts.canView === false) throw new ForbiddenException();
return true;
},
};
return svc;
}
const call = (svc: AiChatService, openPage: any) =>
(svc as any).resolveOpenPageContext(openPage, ws, user) as Promise<{
id: string;
title: string;
} | null>;
it('returns null when no page is open (no id)', async () => {
const svc = makeService({});
expect(await call(svc, null)).toBeNull();
expect(await call(svc, {})).toBeNull();
expect(await call(svc, { title: 'spoofed' })).toBeNull();
});
it('returns null when the page does not exist', async () => {
const svc = makeService({ page: null });
expect(await call(svc, { id: 'p-x' })).toBeNull();
});
it('returns null for a page in a DIFFERENT workspace (tenant isolation)', async () => {
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-OTHER', title: 'Secret' },
});
expect(await call(svc, { id: 'p-1' })).toBeNull();
});
it('returns null when the user may not view the page (Forbidden)', async () => {
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-1', title: 'Restricted' },
canView: false,
});
expect(await call(svc, { id: 'p-1' })).toBeNull();
});
it('returns null (fail-closed) on a non-Forbidden access-check fault', async () => {
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-1', title: 'X' },
canView: 'throw-other',
});
expect(await call(svc, { id: 'p-1' })).toBeNull();
});
it('uses the AUTHORITATIVE DB title, IGNORING the client-supplied title', async () => {
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-1', title: 'Real Title B' },
canView: true,
});
// The client claims it is on "Page A" but the id points at page B.
const result = await call(svc, { id: 'p-1', title: 'Page A' });
expect(result).toEqual({ id: 'p-1', title: 'Real Title B' });
});
it('coerces a null DB title to an empty string', async () => {
const svc = makeService({
page: { id: 'p-1', workspaceId: 'ws-1', title: null },
canView: true,
});
expect(await call(svc, { id: 'p-1' })).toEqual({ id: 'p-1', title: '' });
});
});

View File

@@ -60,7 +60,10 @@ export function prepareAgentStep(
system: string,
): { toolChoice: 'none'; system: string } | undefined {
if (stepNumber >= MAX_AGENT_STEPS - 1) {
return { toolChoice: 'none', system: `${system}\n\n${FINAL_STEP_INSTRUCTION}` };
return {
toolChoice: 'none',
system: `${system}\n\n${FINAL_STEP_INSTRUCTION}`,
};
}
return undefined;
}
@@ -182,6 +185,41 @@ export class AiChatService {
return this.ai.getChatModel(workspaceId, roleModelOverride(role));
}
/**
* Validate the client-supplied open page and return its AUTHORITATIVE identity
* ({ id, title }) or null. The client controls BOTH the id and the title in the
* request body, so neither is trusted: the id must resolve to a real page in
* THIS workspace that the user may read, and the title is taken from the DB row
* (never the client) so the model can't be told it is "on Page A" while the id
* points at page B (#159). Fail-closed — any missing / foreign / inaccessible
* page, or any non-Forbidden access-check fault, returns null.
*/
private async resolveOpenPageContext(
openPage: { id?: string; title?: string } | null | undefined,
workspace: Workspace,
user: User,
): Promise<{ id: string; title: string } | null> {
const candidatePageId = openPage?.id;
if (!candidatePageId) return null;
const page = await this.pageRepo.findById(candidatePageId);
if (!page || page.workspaceId !== workspace.id) return null;
try {
await this.pageAccess.validateCanView(page, user);
} catch (e) {
// A ForbiddenException is the expected "user cannot read this page" case;
// log anything else (e.g. a DB error) so a real fault is not masked.
if (!(e instanceof ForbiddenException)) {
this.logger.warn(
`open page access check failed: ${
e instanceof Error ? e.message : 'unknown error'
}`,
);
}
return null;
}
return { id: page.id, title: page.title ?? '' };
}
async stream({
user,
workspace,
@@ -202,37 +240,26 @@ export class AiChatService {
chatId = undefined;
}
}
// The open page the client sent is attacker-controllable — BOTH its id and
// its title. Resolve it ONCE against the DB (workspace-scoped + access-
// checked) and use the AUTHORITATIVE identity everywhere below: the system
// prompt context, the getCurrentPage tool, and the new-chat history origin.
// Previously the client title was echoed verbatim, so a navigation / two-tab
// desync (openPage.id -> page B, title -> "Page A") made the model report
// "updated Page A" while it edited page B (#159). Null when no page is open
// or the page is foreign / inaccessible / missing.
const openPageContext = await this.resolveOpenPageContext(
body.openPage,
workspace,
user,
);
if (!chatId) {
// Resolve the origin document for the history list. body.openPage.id is
// attacker-controllable, so validate it before persisting: it must be a
// real page in THIS workspace that the user is allowed to read. Anything
// else (foreign workspace, inaccessible/restricted, or non-existent) is
// dropped to null — persisting it would leak the page's title via the
// chat-list join, or violate the page_id FK on insert (this runs after
// res.hijack(), so a DB error would break the stream).
let originPageId: string | null = null;
const candidatePageId = body.openPage?.id;
if (candidatePageId) {
const page = await this.pageRepo.findById(candidatePageId);
if (page && page.workspaceId === workspace.id) {
try {
await this.pageAccess.validateCanView(page, user);
originPageId = page.id;
} catch (e) {
// Fail-closed: no provenance on any failure. A ForbiddenException is
// the expected "user cannot read this page" case; log anything else
// (e.g. a DB error) so a real fault is not masked as "no access".
if (!(e instanceof ForbiddenException)) {
this.logger.warn(
`origin page access check failed: ${
e instanceof Error ? e.message : 'unknown error'
}`,
);
}
originPageId = null;
}
}
}
// The history-list origin is the validated open page (see above):
// persisting an unvalidated id would leak a title via the chat-list join,
// or violate the page_id FK on insert (this runs after res.hijack(), so a
// DB error would break the stream).
const originPageId: string | null = openPageContext?.id ?? null;
const chat = await this.aiChatRepo.insert({
creatorId: user.id,
workspaceId: workspace.id,
@@ -259,9 +286,7 @@ export class AiChatService {
content: incomingText,
// jsonb column: UIMessage parts are JSON-serializable at runtime but not
// structurally `JsonValue`, so cast through unknown.
metadata: (incoming?.parts
? { parts: incoming.parts }
: null) as never,
metadata: (incoming?.parts ? { parts: incoming.parts } : null) as never,
});
// Rebuild the conversation from persisted history (not the client payload),
@@ -280,38 +305,20 @@ export class AiChatService {
// The model is resolved by the controller before hijack (clean 503 path).
// Here we only need the admin-configured system prompt.
const resolved = await this.aiSettings.resolve(workspace.id);
const system = buildSystemPrompt({
workspace,
adminPrompt: resolved?.systemPrompt,
// The role (pre-resolved by the controller) REPLACES the persona layer;
// the safety framework is still appended by buildSystemPrompt.
roleInstructions: role?.instructions,
openedPage: body.openPage,
});
// Pass the resolved chatId so the write tools can mint provenance tokens
// (access + collab) carrying { actor:'agent', aiChatId: chatId }, making
// agent REST/collab writes attributable and non-spoofable (§6.5/§6.6).
const docmostTools = await this.tools.forUser(
user,
sessionId,
workspace.id,
chatId,
// Same open-page value used by the system prompt above; exposed to the
// model via getCurrentPage so page identity survives prompt mangling.
body.openPage,
);
// Merge in admin-configured external MCP tools (web search, etc.; §6.8).
// A down/slow external server never crashes the turn — toolsFor skips it and
// records the outcome. The returned client handles MUST be closed in the
// streamText lifecycle (onFinish/onError/onAbort) — leaking them is a bug.
// Docmost tools take precedence on a name clash (external are namespaced, so
// a clash is not expected; the spread order makes intent explicit).
// Build the external MCP toolset FIRST so the system prompt can carry each
// connected server's admin-authored guidance (#180). Merge in admin-
// configured external MCP tools (web search, etc.; §6.8). A down/slow
// external server never crashes the turn — toolsFor skips it and records the
// outcome. The returned client handles MUST be closed in the streamText
// lifecycle (onFinish/onError/onAbort) — leaking them is a bug. Docmost
// tools take precedence on a name clash (external are namespaced, so a clash
// is not expected; the spread order makes intent explicit).
let external: Awaited<ReturnType<McpClientsService['toolsFor']>> = {
tools: {},
clients: [],
outcomes: [],
instructions: [],
};
try {
external = await this.mcpClients.toolsFor(workspace.id);
@@ -324,6 +331,33 @@ export class AiChatService {
}`,
);
}
const system = buildSystemPrompt({
workspace,
adminPrompt: resolved?.systemPrompt,
// The role (pre-resolved by the controller) REPLACES the persona layer;
// the safety framework is still appended by buildSystemPrompt.
roleInstructions: role?.instructions,
// Server-validated open page (authoritative title), not the client value.
openedPage: openPageContext,
// Guidance only for servers that connected and yielded ≥1 callable tool.
mcpInstructions: external.instructions,
});
// Pass the resolved chatId so the write tools can mint provenance tokens
// (access + collab) carrying { actor:'agent', aiChatId: chatId }, making
// agent REST/collab writes attributable and non-spoofable (§6.5/§6.6).
const docmostTools = await this.tools.forUser(
user,
sessionId,
workspace.id,
chatId,
// Same server-validated open page used by the system prompt above; exposed
// to the model via getCurrentPage so page identity (and the AUTHORITATIVE
// title) survives prompt mangling and client title spoofing (#159).
openPageContext,
);
const tools = { ...external.tools, ...docmostTools };
// Close every external client EXACTLY ONCE across the turn's terminal
@@ -395,144 +429,150 @@ export class AiChatService {
let result: ReturnType<typeof streamText>;
try {
result = streamText({
model,
system,
messages,
tools,
// No maxOutputTokens cap on the agent: tool-call arguments (e.g. a full
// page body for the write tools) are emitted as OUTPUT tokens, so a fixed
// cap would truncate complex tool calls mid-argument. Let the model use its
// natural per-step budget. (Cost/credit limits are an account concern, not
// something to enforce by silently breaking the agent.)
stopWhen: stepCountIs(MAX_AGENT_STEPS),
// Forced finalization: reserve the LAST allowed step for a text-only
// answer. Without this, a turn that spends all its steps on tool calls
// ends with no assistant text (an empty turn). prepareAgentStep forbids
// further tool calls and appends a synthesis instruction on that step,
// concatenated onto the original `system` so the persona is preserved.
prepareStep: ({ stepNumber }) => prepareAgentStep(stepNumber, system),
abortSignal: signal,
onChunk: ({ chunk }) => {
// DIAGNOSTIC (Safari stream-drop investigation) — temporary. Any model
// output chunk means the stream is actively emitting bytes; track first
// + most-recent activity timestamps.
const now = Date.now();
firstModelChunkAt ??= now;
lastModelChunkAt = now;
// 'text-delta' is the assistant's prose; tool-call args are separate chunk
// types — so this mirrors exactly what streams to the client.
if (chunk.type === 'text-delta') inProgressText += chunk.text;
},
onStepFinish: (step) => {
// The finished step's full text is now in `step.text`; fold it in and reset
// the in-progress accumulator for the next step.
capturedSteps.push(step as StepLike);
inProgressText = '';
},
onFinish: async ({ text, finishReason, totalUsage, usage, steps }) => {
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: success
// baseline for Safari comparison.
const diagNow = Date.now();
this.logger.log(
`AI chat stream DIAGNOSTIC (finish): elapsed=${diagNow - streamStartedAt}ms ` +
`firstChunkLatency=${firstModelChunkAt ? firstModelChunkAt - streamStartedAt : 'none'}ms ` +
`heartbeatsSent=${heartbeatsSent} steps=${steps.length}`,
);
await persistAssistant({
text,
toolCalls: serializeSteps(steps),
metadata: {
finishReason,
// Persist the turn's cumulative usage WITH reasoning tokens resolved
// from either the new `outputTokenDetails` or the deprecated top-level
// field, so reopened history / the Markdown export show the thinking
// token cost too.
usage: normalizeStreamUsage(totalUsage as StreamUsage) ?? totalUsage,
// Final-step usage = the context actually fed to the model on the last LLM
// call (full history + tool results) plus the answer it just generated.
// input+output of the FINAL step ≈ the conversation's CURRENT context size,
// distinct from totalUsage which sums every step (cumulative tokens spent).
contextTokens:
(usage?.inputTokens ?? 0) + (usage?.outputTokens ?? 0) || undefined,
// Persist the FULL set of UIMessage parts for the turn (text +
// tool-call/result), so the rebuilt history replays prior tool
// context to the model on later turns.
parts: assistantParts(steps, text),
},
});
// Lifecycle: release the external MCP clients leased for this turn.
await closeExternalClients();
// Generate the chat title for a freshly created chat AFTER the stream's
// provider call has completed — NOT concurrently with it. The z.ai coding
// endpoint stalls one of two concurrent requests to the same plan, which
// black-holed the chat stream (~300s headers timeout) when title
// generation raced it. Running it here (solo, fire-and-forget) avoids the
// race; never block the turn on it, swallow any error.
if (isNewChat && incomingText) {
void this.generateTitle(chatId, workspace.id, incomingText).catch(
(err) => {
this.logger.warn(
`Title generation failed: ${(err as Error)?.message ?? err}`,
);
},
model,
system,
messages,
tools,
// No maxOutputTokens cap on the agent: tool-call arguments (e.g. a full
// page body for the write tools) are emitted as OUTPUT tokens, so a fixed
// cap would truncate complex tool calls mid-argument. Let the model use its
// natural per-step budget. (Cost/credit limits are an account concern, not
// something to enforce by silently breaking the agent.)
stopWhen: stepCountIs(MAX_AGENT_STEPS),
// Forced finalization: reserve the LAST allowed step for a text-only
// answer. Without this, a turn that spends all its steps on tool calls
// ends with no assistant text (an empty turn). prepareAgentStep forbids
// further tool calls and appends a synthesis instruction on that step,
// concatenated onto the original `system` so the persona is preserved.
prepareStep: ({ stepNumber }) => prepareAgentStep(stepNumber, system),
abortSignal: signal,
onChunk: ({ chunk }) => {
// DIAGNOSTIC (Safari stream-drop investigation) — temporary. Any model
// output chunk means the stream is actively emitting bytes; track first
// + most-recent activity timestamps.
const now = Date.now();
firstModelChunkAt ??= now;
lastModelChunkAt = now;
// 'text-delta' is the assistant's prose; tool-call args are separate chunk
// types — so this mirrors exactly what streams to the client.
if (chunk.type === 'text-delta') inProgressText += chunk.text;
},
onStepFinish: (step) => {
// The finished step's full text is now in `step.text`; fold it in and reset
// the in-progress accumulator for the next step.
capturedSteps.push(step as StepLike);
inProgressText = '';
},
onFinish: async ({ text, finishReason, totalUsage, usage, steps }) => {
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: success
// baseline for Safari comparison.
const diagNow = Date.now();
this.logger.log(
`AI chat stream DIAGNOSTIC (finish): elapsed=${diagNow - streamStartedAt}ms ` +
`firstChunkLatency=${firstModelChunkAt ? firstModelChunkAt - streamStartedAt : 'none'}ms ` +
`heartbeatsSent=${heartbeatsSent} steps=${steps.length}`,
);
}
},
onError: async ({ error }) => {
// NestJS Logger.error(message, stack?, context?): pass the real message
// (with statusCode when present) + the stack string, not the Error
// object, so the actual provider cause is clearly logged. Reuse the
// shared formatter so provider error formatting stays unified.
const e = error as { stack?: string };
const errorText = describeProviderError(error, String(error));
this.logger.error(`AI chat stream error: ${errorText}`, e?.stack);
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: timing of
// an error-terminated stream.
const diagNow = Date.now();
this.logger.warn(
`AI chat stream DIAGNOSTIC (error): elapsed=${diagNow - streamStartedAt}ms ` +
`firstChunkLatency=${firstModelChunkAt ? firstModelChunkAt - streamStartedAt : 'none'}ms ` +
`silentGapBeforeDrop=${diagNow - lastModelChunkAt}ms heartbeatsSent=${heartbeatsSent}`,
);
// Persist the PARTIAL answer streamed before the failure (text + any
// finished tool steps) WITH the error in metadata, so the turn shows what
// the user already saw plus the cause — not just a bare error.
await persistAssistant(
buildPartialAssistantRecord(
capturedSteps,
inProgressText,
'error',
errorText,
),
);
await closeExternalClients();
},
onAbort: async ({ steps }) => {
const partialChars =
capturedSteps.reduce((n, s) => n + (s.text?.length ?? 0), 0) +
inProgressText.length;
// Unlike onError/onFinish, this terminal path otherwise writes nothing, so
// an aborted turn (client disconnect / proxy drop / stop()) would be
// invisible in the logs. Log it (warn) so the abort is traceable.
this.logger.warn(
`AI chat stream aborted (chat ${chatId}) after ${steps.length} ` +
`step(s), ${partialChars} chars partial text; persisting partial turn.`,
);
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: THE key
// line — classifies the Safari drop.
const diagNow = Date.now();
this.logger.warn(
`AI chat stream DIAGNOSTIC (abort/disconnect): elapsed=${diagNow - streamStartedAt}ms ` +
`firstChunkLatency=${firstModelChunkAt ? firstModelChunkAt - streamStartedAt : 'none'}ms ` +
`silentGapBeforeDrop=${diagNow - lastModelChunkAt}ms heartbeatsSent=${heartbeatsSent} ` +
`steps=${steps.length}`,
);
await persistAssistant(
buildPartialAssistantRecord(capturedSteps, inProgressText, 'aborted'),
);
await closeExternalClients();
},
await persistAssistant({
text,
toolCalls: serializeSteps(steps),
metadata: {
finishReason,
// Persist the turn's cumulative usage WITH reasoning tokens resolved
// from either the new `outputTokenDetails` or the deprecated top-level
// field, so reopened history / the Markdown export show the thinking
// token cost too.
usage:
normalizeStreamUsage(totalUsage as StreamUsage) ?? totalUsage,
// Final-step usage = the context actually fed to the model on the last LLM
// call (full history + tool results) plus the answer it just generated.
// input+output of the FINAL step ≈ the conversation's CURRENT context size,
// distinct from totalUsage which sums every step (cumulative tokens spent).
contextTokens:
(usage?.inputTokens ?? 0) + (usage?.outputTokens ?? 0) ||
undefined,
// Persist the FULL set of UIMessage parts for the turn (text +
// tool-call/result), so the rebuilt history replays prior tool
// context to the model on later turns.
parts: assistantParts(steps, text),
},
});
// Lifecycle: release the external MCP clients leased for this turn.
await closeExternalClients();
// Generate the chat title for a freshly created chat AFTER the stream's
// provider call has completed — NOT concurrently with it. The z.ai coding
// endpoint stalls one of two concurrent requests to the same plan, which
// black-holed the chat stream (~300s headers timeout) when title
// generation raced it. Running it here (solo, fire-and-forget) avoids the
// race; never block the turn on it, swallow any error.
if (isNewChat && incomingText) {
void this.generateTitle(chatId, workspace.id, incomingText).catch(
(err) => {
this.logger.warn(
`Title generation failed: ${(err as Error)?.message ?? err}`,
);
},
);
}
},
onError: async ({ error }) => {
// NestJS Logger.error(message, stack?, context?): pass the real message
// (with statusCode when present) + the stack string, not the Error
// object, so the actual provider cause is clearly logged. Reuse the
// shared formatter so provider error formatting stays unified.
const e = error as { stack?: string };
const errorText = describeProviderError(error, String(error));
this.logger.error(`AI chat stream error: ${errorText}`, e?.stack);
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: timing of
// an error-terminated stream.
const diagNow = Date.now();
this.logger.warn(
`AI chat stream DIAGNOSTIC (error): elapsed=${diagNow - streamStartedAt}ms ` +
`firstChunkLatency=${firstModelChunkAt ? firstModelChunkAt - streamStartedAt : 'none'}ms ` +
`silentGapBeforeDrop=${diagNow - lastModelChunkAt}ms heartbeatsSent=${heartbeatsSent}`,
);
// Persist the PARTIAL answer streamed before the failure (text + any
// finished tool steps) WITH the error in metadata, so the turn shows what
// the user already saw plus the cause — not just a bare error.
await persistAssistant(
buildPartialAssistantRecord(
capturedSteps,
inProgressText,
'error',
errorText,
),
);
await closeExternalClients();
},
onAbort: async ({ steps }) => {
const partialChars =
capturedSteps.reduce((n, s) => n + (s.text?.length ?? 0), 0) +
inProgressText.length;
// Unlike onError/onFinish, this terminal path otherwise writes nothing, so
// an aborted turn (client disconnect / proxy drop / stop()) would be
// invisible in the logs. Log it (warn) so the abort is traceable.
this.logger.warn(
`AI chat stream aborted (chat ${chatId}) after ${steps.length} ` +
`step(s), ${partialChars} chars partial text; persisting partial turn.`,
);
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: THE key
// line — classifies the Safari drop.
const diagNow = Date.now();
this.logger.warn(
`AI chat stream DIAGNOSTIC (abort/disconnect): elapsed=${diagNow - streamStartedAt}ms ` +
`firstChunkLatency=${firstModelChunkAt ? firstModelChunkAt - streamStartedAt : 'none'}ms ` +
`silentGapBeforeDrop=${diagNow - lastModelChunkAt}ms heartbeatsSent=${heartbeatsSent} ` +
`steps=${steps.length}`,
);
await persistAssistant(
buildPartialAssistantRecord(
capturedSteps,
inProgressText,
'aborted',
),
);
await closeExternalClients();
},
});
// Drain the stream independently of the client socket so the turn always
@@ -652,7 +692,10 @@ export class AiChatService {
'punctuation at the end.',
prompt: firstMessage.slice(0, 2000),
});
const title = text.trim().replace(/^["']|["']$/g, '').slice(0, 120);
const title = text
.trim()
.replace(/^["']|["']$/g, '')
.slice(0, 120);
if (title) {
await this.aiChatRepo.update(chatId, { title }, workspaceId);
}

View File

@@ -42,6 +42,15 @@ export class CreateMcpServerDto {
@IsString({ each: true })
toolAllowlist?: string[];
// Admin-authored guidance ("how/when to use this server's tools") injected
// into the agent system prompt next to the tool descriptions (#180). Trusted,
// NON-secret (so it IS returned). Capped to bound prompt/token size (the
// built-in guide is ~1.5KB). Blank => stored as null.
@IsOptional()
@IsString()
@MaxLength(4000)
instructions?: string;
@IsOptional()
@IsBoolean()
enabled?: boolean;

View File

@@ -0,0 +1,75 @@
import 'reflect-metadata';
import { plainToInstance } from 'class-transformer';
import { validateSync } from 'class-validator';
import { CreateMcpServerDto } from './create-mcp-server.dto';
import { UpdateMcpServerDto } from './update-mcp-server.dto';
/**
* API-boundary validation for the per-server `instructions` field (#180): a free
* text guide injected into the agent system prompt. It is optional, must be a
* string, and is bounded by @MaxLength(4000) to cap prompt/token size.
*/
describe('MCP server DTO instructions validation', () => {
function validateCreate(payload: unknown) {
const dto = plainToInstance(CreateMcpServerDto, payload);
return validateSync(dto as object);
}
function validateUpdate(payload: unknown) {
const dto = plainToInstance(UpdateMcpServerDto, payload);
return validateSync(dto as object);
}
const base = {
name: 'Tavily',
transport: 'http',
url: 'https://example.com/mcp',
};
it('accepts an omitted instructions field on create', () => {
expect(validateCreate({ ...base })).toHaveLength(0);
});
it('accepts a reasonable instructions string on create', () => {
expect(
validateCreate({ ...base, instructions: 'Use search for fresh facts.' }),
).toHaveLength(0);
});
it('rejects instructions over MaxLength(4000) on create', () => {
const errors = validateCreate({
...base,
instructions: 'a'.repeat(4001),
});
expect(
errors.some(
(e) =>
e.property === 'instructions' &&
e.constraints !== undefined &&
'maxLength' in e.constraints,
),
).toBe(true);
});
it('accepts instructions of exactly 4000 chars on create', () => {
expect(
validateCreate({ ...base, instructions: 'a'.repeat(4000) }),
).toHaveLength(0);
});
it('rejects a non-string instructions value', () => {
const errors = validateCreate({ ...base, instructions: 123 });
expect(errors.some((e) => e.property === 'instructions')).toBe(true);
});
it('rejects instructions over MaxLength(4000) on update', () => {
const errors = validateUpdate({ instructions: 'a'.repeat(4001) });
expect(
errors.some(
(e) =>
e.property === 'instructions' &&
e.constraints !== undefined &&
'maxLength' in e.constraints,
),
).toBe(true);
});
});

View File

@@ -43,6 +43,13 @@ export class UpdateMcpServerDto {
@IsString({ each: true })
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Absent => unchanged; blank => cleared
// (stored as null by the repo). Capped to bound prompt/token size.
@IsOptional()
@IsString()
@MaxLength(4000)
instructions?: string;
@IsOptional()
@IsBoolean()
enabled?: boolean;

View File

@@ -33,6 +33,26 @@ interface ServerOutcome {
reason?: string;
}
/**
* One server's admin-authored guidance for the agent system prompt (#180).
* Built ONLY for a server that actually connected AND contributed ≥1 tool
* (after the allowlist filter) AND has non-blank guidance — so a guide never
* appears for a server whose tools the agent cannot actually call.
*/
export interface McpServerInstruction {
/** Display name of the server (for the prompt section header). */
serverName: string;
/**
* The tool-name namespace prefix the server's tools were merged under
* (sanitized name, e.g. `tavily`). The prompt renders this as `tavily_*` so
* the model can connect the guidance to the actual tool names. Advisory:
* individual tools may carry a disambiguating suffix on rare collisions.
*/
toolPrefix: string;
/** The trusted, non-blank guidance text. */
instructions: string;
}
export interface ExternalToolset {
/** Namespaced external tools, merge-ready into the agent toolset. */
tools: Record<string, Tool>;
@@ -40,6 +60,11 @@ export interface ExternalToolset {
clients: Closable[];
/** Per-server connect outcomes so the UI can show unavailable servers. */
outcomes: ServerOutcome[];
/**
* Per-server prompt guidance for connected servers that contributed ≥1 tool
* and have non-blank instructions. Empty when no server qualifies.
*/
instructions: McpServerInstruction[];
}
/** Connect+tools() timeout per server — a slow server must not stall the turn. */
@@ -60,6 +85,8 @@ interface CacheEntry {
tools: Record<string, Tool>;
clients: McpClient[];
outcomes: ServerOutcome[];
/** Prompt guidance for qualifying servers (see McpServerInstruction). */
instructions: McpServerInstruction[];
expiresAt: number;
/** Active leases (turns currently using these clients). */
refCount: number;
@@ -141,6 +168,7 @@ export class McpClientsService {
tools: entry.tools,
clients: [release],
outcomes: entry.outcomes,
instructions: entry.instructions,
};
}
@@ -225,6 +253,7 @@ export class McpClientsService {
const outcomes: ServerOutcome[] = [];
// Per-call total wall-clock cap, read once for this build (env-overridable).
const callTimeoutMs = mcpCallTimeoutMs();
const instructions: McpServerInstruction[] = [];
for (const server of servers) {
try {
@@ -233,17 +262,33 @@ export class McpClientsService {
clients.push(client);
const allow = server.toolAllowlist;
const picked =
Array.isArray(allow) && allow.length > 0
? pick(raw, allow)
: raw;
Array.isArray(allow) && allow.length > 0 ? pick(raw, allow) : raw;
// Bound each tool's execute with a per-call total-timeout guard before
// merging, so a single chatty-but-stuck call is aborted after the cap.
const guarded = wrapToolsWithCallTimeout(picked, callTimeoutMs);
// Namespace each tool with the sanitized server name AND disambiguate
// against names already merged from earlier servers, so no external
// tool is silently overwritten on collision.
this.mergeNamespaced(tools, guarded, server.name, server.id);
// tool is silently overwritten on collision. The returned count drives
// whether this server's prompt guidance is included (≥1 tool merged).
const merged = this.mergeNamespaced(
tools,
guarded,
server.name,
server.id,
);
outcomes.push({ name: server.name, ok: true });
// Include this server's guidance ONLY when it actually contributed at
// least one tool the agent can call (allowlist may have filtered all of
// them out) AND the admin authored non-blank instructions. The header
// prefix is the sanitized server name (= the tool namespace prefix).
const guide = server.instructions?.trim();
if (merged.count > 0 && guide) {
instructions.push({
serverName: server.name,
toolPrefix: merged.prefix,
instructions: guide,
});
}
} catch (err) {
// A failed server is skipped — the turn proceeds with the rest. Log a
// short warning (never the URL/headers) so ops can see degradation, and
@@ -260,6 +305,7 @@ export class McpClientsService {
tools,
clients,
outcomes,
instructions,
expiresAt: Date.now() + CACHE_TTL_MS,
refCount: 0,
evicted: false,
@@ -276,16 +322,19 @@ export class McpClientsService {
* renaming any key that would collide with an already-merged tool (different
* servers with the same sanitized name, or duplicates after truncation), so
* no external tool is silently dropped via overwrite.
*
* Returns how many tools this server actually contributed and the namespace
* prefix used (the sanitized server name) so the caller can attach the
* server's prompt guidance only when ≥1 tool was merged.
*/
private mergeNamespaced(
target: Record<string, Tool>,
picked: Record<string, Tool>,
serverName: string,
serverId: string,
): void {
for (const [name, tool] of Object.entries(
namespace(picked, serverName),
)) {
): { count: number; prefix: string } {
let count = 0;
for (const [name, tool] of Object.entries(namespace(picked, serverName))) {
let key = name;
if (key in target) {
const original = key;
@@ -295,7 +344,9 @@ export class McpClientsService {
);
}
target[key] = tool;
count += 1;
}
return { count, prefix: namespacePrefix(serverName) };
}
/**
@@ -371,9 +422,7 @@ export class McpClientsService {
/** Close clients, swallowing close errors so they never break a response. */
private async closeClients(clients: McpClient[]): Promise<void> {
await Promise.all(
clients.map((c) => c.close().catch(() => undefined)),
);
await Promise.all(clients.map((c) => c.close().catch(() => undefined)));
}
}
@@ -386,9 +435,10 @@ export class McpClientsService {
* lookup hands net/tls.connect ONLY a set that passed this check, so the kernel
* can never connect to an address that did not pass the guard. Pure — no I/O.
*/
export function validateResolvedAddresses(
addrs: readonly LookupAddress[],
): { ok: boolean; blockedHost?: string } {
export function validateResolvedAddresses(addrs: readonly LookupAddress[]): {
ok: boolean;
blockedHost?: string;
} {
if (addrs.length === 0) {
return { ok: false };
}
@@ -524,7 +574,7 @@ function namespace(
tools: Record<string, Tool>,
serverName: string,
): Record<string, Tool> {
const prefix = sanitizeName(serverName) || 'mcp';
const prefix = namespacePrefix(serverName);
const out: Record<string, Tool> = {};
for (const [name, t] of Object.entries(tools)) {
const safe = sanitizeName(name);
@@ -539,6 +589,15 @@ function namespace(
return out;
}
/**
* The tool-name namespace prefix for a server: its sanitized name, or `mcp`
* when the name sanitizes to empty. Tools are merged as `${prefix}_${tool}`, so
* the prompt guidance refers to the server's tools as `${prefix}_*`.
*/
function namespacePrefix(serverName: string): string {
return sanitizeName(serverName) || 'mcp';
}
/** Reduce an arbitrary string to ^[a-zA-Z0-9_-]+, collapsing runs to '_'. */
function sanitizeName(value: string): string {
return value

View File

@@ -0,0 +1,168 @@
import { type Tool } from 'ai';
import { McpClientsService } from './mcp-clients.service';
/**
* Tests for the per-server prompt guidance (#180) assembled by buildEntry and
* surfaced via toolsFor().instructions.
*
* REACHABILITY NOTE: buildEntry is a PRIVATE method; the smallest reachable
* public path is toolsFor() -> getOrBuildEntry -> buildEntry -> connect/tools()
* -> mergeNamespaced. We drive that path: stub the repo's `listEnabled` and spy
* on the private `connect` to return fake MCP clients whose `tools()` we control.
*
* Contract (all checked here): a server's guidance is included ONLY when the
* server actually connected AND contributed ≥1 callable tool (after the
* allowlist filter) AND its instructions are non-blank. The header carries the
* tool namespace prefix (the sanitized server name).
*/
function fakeTool(): Tool {
return { description: 'x', inputSchema: undefined } as unknown as Tool;
}
interface FakeServer {
id: string;
name: string;
transport: string;
url: string;
headersEnc: string | null;
toolAllowlist: string[] | null;
instructions: string | null;
}
function server(
over: Partial<FakeServer> & { id: string; name: string },
): FakeServer {
return {
transport: 'http',
url: 'https://example.com/mcp',
headersEnc: null,
toolAllowlist: null,
instructions: null,
...over,
};
}
async function instructionsFor(
servers: FakeServer[],
toolsByServerId: Record<string, Record<string, Tool>>,
// Server ids whose connect should THROW (simulating an unavailable server).
failingIds: Set<string> = new Set(),
): Promise<
{
serverName: string;
toolPrefix: string;
instructions: string;
}[]
> {
const repoStub = {
listEnabled: jest.fn().mockResolvedValue(servers),
};
const service = new McpClientsService(repoStub as never, {} as never);
jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => unknown },
'connect',
)
.mockImplementation((s: FakeServer) => {
if (failingIds.has(s.id)) {
return Promise.reject(new Error('connection failed'));
}
return Promise.resolve({
tools: () => Promise.resolve(toolsByServerId[s.id] ?? {}),
close: () => Promise.resolve(),
});
});
const toolset = await service.toolsFor('ws-1');
await Promise.all(toolset.clients.map((c) => c.close()));
return toolset.instructions;
}
describe('external MCP per-server prompt guidance (via toolsFor)', () => {
afterEach(() => jest.restoreAllMocks());
it('includes guidance for a connected server with non-empty text and ≥1 tool', async () => {
const instructions = await instructionsFor(
[
server({
id: 'id-tavily',
name: 'Tavily',
instructions: 'Use tavily_search for fresh facts.',
}),
],
{ 'id-tavily': { search: fakeTool() } },
);
// sanitizeName preserves case (charset [a-zA-Z0-9_-]), so the prefix is the
// server name as-is for an already-clean name.
expect(instructions).toEqual([
{
serverName: 'Tavily',
toolPrefix: 'Tavily',
instructions: 'Use tavily_search for fresh facts.',
},
]);
});
it('omits guidance when the server has no instructions', async () => {
const instructions = await instructionsFor(
[server({ id: 'id-1', name: 'Tavily', instructions: null })],
{ 'id-1': { search: fakeTool() } },
);
expect(instructions).toEqual([]);
});
it('omits guidance when the instructions are only whitespace', async () => {
const instructions = await instructionsFor(
[server({ id: 'id-1', name: 'Tavily', instructions: ' ' })],
{ 'id-1': { search: fakeTool() } },
);
expect(instructions).toEqual([]);
});
it('omits guidance for a server that contributed ZERO tools (allowlist filtered all out)', async () => {
const instructions = await instructionsFor(
[
server({
id: 'id-1',
name: 'Tavily',
instructions: 'guide',
// Allowlist names a tool the server does not expose -> 0 picked.
toolAllowlist: ['nonexistent'],
}),
],
{ 'id-1': { search: fakeTool() } },
);
expect(instructions).toEqual([]);
});
it('omits guidance for an unavailable (failed-connect) server', async () => {
const instructions = await instructionsFor(
[server({ id: 'id-1', name: 'Tavily', instructions: 'guide' })],
{ 'id-1': { search: fakeTool() } },
new Set(['id-1']),
);
expect(instructions).toEqual([]);
});
it('includes only the qualifying servers among several', async () => {
const instructions = await instructionsFor(
[
server({ id: 'ok', name: 'Tavily', instructions: 'web guide' }),
server({ id: 'blank', name: 'Crawl', instructions: '' }),
server({ id: 'down', name: 'Down', instructions: 'never shown' }),
],
{
ok: { search: fakeTool() },
blank: { crawl: fakeTool() },
down: { x: fakeTool() },
},
new Set(['down']),
);
expect(instructions).toEqual([
{ serverName: 'Tavily', toolPrefix: 'Tavily', instructions: 'web guide' },
]);
});
});

View File

@@ -17,6 +17,7 @@ function row(overrides: Partial<AiMcpServer>): AiMcpServer {
enabled: true,
toolAllowlist: null,
headersEnc: null,
instructions: null,
...overrides,
} as unknown as AiMcpServer;
}
@@ -28,11 +29,7 @@ describe('McpServersService.toView (via list) — encrypted-header leak guard',
};
// secretBox + clients are unused by the list/toView path; pass stubs to
// satisfy the constructor.
return new McpServersService(
repoStub as never,
{} as never,
{} as never,
);
return new McpServersService(repoStub as never, {} as never, {} as never);
}
it('exposes hasHeaders:true and NO headersEnc when auth headers are set', async () => {
@@ -67,6 +64,7 @@ describe('McpServersService.toView (via list) — encrypted-header leak guard',
enabled: false,
toolAllowlist: ['search'],
headersEnc: 'BLOB',
instructions: 'Use search for fresh web facts.',
}),
]);
@@ -80,6 +78,19 @@ describe('McpServersService.toView (via list) — encrypted-header leak guard',
enabled: false,
toolAllowlist: ['search'],
hasHeaders: true,
instructions: 'Use search for fresh web facts.',
});
});
it('returns instructions (NON-secret) in the view, null when unset', async () => {
const service = buildService([
row({ id: 'a', instructions: 'How to use these tools.' }),
row({ id: 'b', instructions: null }),
]);
const [withText, withoutText] = await service.list('ws-1');
expect(withText.instructions).toBe('How to use these tools.');
expect(withoutText.instructions).toBeNull();
});
});

View File

@@ -20,6 +20,9 @@ export interface McpServerView {
enabled: boolean;
toolAllowlist: string[] | null;
hasHeaders: boolean;
// Admin-authored prompt guidance (#180). NON-secret, so returned in the view.
// Null when no guidance is configured.
instructions: string | null;
}
/**
@@ -56,6 +59,8 @@ export class McpServersService {
url: dto.url,
headersEnc,
toolAllowlist: dto.toolAllowlist ?? null,
// Blank/whitespace guidance is normalized to null by the repo.
instructions: dto.instructions ?? null,
enabled: dto.enabled ?? true,
});
this.clients.invalidate(workspaceId);
@@ -97,6 +102,8 @@ export class McpServersService {
headersEnc,
// undefined => unchanged; [] / value handled by repo (empty => null).
toolAllowlist: dto.toolAllowlist,
// undefined => unchanged; blank => cleared (null) by the repo.
instructions: dto.instructions,
enabled: dto.enabled,
});
this.clients.invalidate(workspaceId);
@@ -167,6 +174,7 @@ export class McpServersService {
enabled: row.enabled,
toolAllowlist: row.toolAllowlist ?? null,
hasHeaders: Boolean(row.headersEnc),
instructions: row.instructions ?? null,
};
}
}

View File

@@ -1,30 +0,0 @@
import { jsonbObject } from '@docmost/db/repos/ai-agent-roles/ai-agent-roles.repo';
/**
* Unit tests for jsonbObject: the repo helper that encodes a model_config object
* as a jsonb bind (or null when there is nothing to persist). It is the last
* line of defence before the column write, so the null-vs-bind decision is what
* matters here. We assert only null vs non-null because the non-null value is a
* kysely `sql` template fragment whose internal shape is an implementation
* detail of the SQL tag.
*/
describe('jsonbObject', () => {
it('returns null for null', () => {
expect(jsonbObject(null)).toBeNull();
});
it('returns null for undefined', () => {
expect(jsonbObject(undefined)).toBeNull();
});
it('returns null for an empty object (nothing to persist)', () => {
expect(jsonbObject({})).toBeNull();
});
it('returns a (non-null) jsonb bind for a non-empty object', () => {
const out = jsonbObject({ driver: 'gemini', chatModel: 'gemini-2.0-flash' });
// A real sql fragment is produced, never null/undefined.
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
});

View File

@@ -0,0 +1,133 @@
import * as fs from 'node:fs';
import { ShareSeoController } from './share-seo.controller';
/**
* Routing guard for ShareSeoController.getShare (red-team finding #3).
*
* The SEO route must NOT leak a shared page's <title>/og:title to anonymous
* visitors / crawlers when the page is not publicly readable. It previously
* called the raw `getShareForPage`, which skips the restricted-ancestor gate, so
* a permission-restricted descendant of an includeSubPages share leaked its
* title. The fix funnels through `resolveReadableSharePage` (the canonical gate)
* AND honours `isSharingAllowed`. These tests pin that routing: a non-readable
* page or sharing-disabled space serves the plain SPA index (no title); only a
* readable, still-shared page gets meta tags.
*/
const SECRET_TITLE = 'Restricted Quarterly Numbers';
const INDEX_HTML = `<!doctype html><html><head><title>App</title><!--meta-tags--></head><body></body></html>`;
const STREAM_SENTINEL = { __isStream: true } as unknown as fs.ReadStream;
// Stub fs at CALL time (jest.spyOn), NOT module load (jest.mock): the controller
// transitively pulls bcrypt, whose native module is located by node-gyp-build
// reading the filesystem at import time — a module-level fs mock breaks that.
beforeEach(() => {
jest.spyOn(fs, 'existsSync').mockReturnValue(true);
jest.spyOn(fs, 'readFileSync').mockReturnValue(INDEX_HTML);
jest.spyOn(fs, 'createReadStream').mockReturnValue(STREAM_SENTINEL);
});
afterEach(() => jest.restoreAllMocks());
function makeRes() {
const res: any = {
sent: undefined as unknown,
type: jest.fn(() => res),
send: jest.fn((v: unknown) => {
res.sent = v;
}),
};
return res;
}
function makeController(opts: {
resolved: { share: any; page: any } | null;
sharingAllowed?: boolean;
}) {
const shareService = {
resolveReadableSharePage: jest.fn(async () => opts.resolved),
isSharingAllowed: jest.fn(async () => opts.sharingAllowed ?? true),
// Must NEVER be used by the SEO path anymore (the bypass is the bug).
getShareForPage: jest.fn(async () => {
throw new Error('getShareForPage must not be called by the SEO path');
}),
};
const workspaceRepo = {
findFirst: async () => ({ id: 'ws-1', settings: {} }),
};
const environmentService = { isSelfHosted: () => true };
const controller = new ShareSeoController(
shareService as any,
workspaceRepo as any,
environmentService as any,
);
return { controller, shareService };
}
const req: any = { raw: { headers: { host: 'self' } } };
describe('ShareSeoController.getShare routing (#3 title-leak gate)', () => {
it('serves the plain index (NO title) when the page is not publicly readable', async () => {
const { controller, shareService } = makeController({ resolved: null });
const res = makeRes();
await controller.getShare(res, req, 'share-key', `slug-pageB`);
// The restricted-ancestor gate ran; the raw bypass did not.
expect(shareService.resolveReadableSharePage).toHaveBeenCalled();
expect(shareService.getShareForPage).not.toHaveBeenCalled();
// The plain index stream was sent — NOT the title-bearing meta HTML.
expect(res.sent).toBe(STREAM_SENTINEL);
});
it('serves the plain index when sharing was disabled at the workspace/space level', async () => {
const { controller } = makeController({
resolved: {
share: { spaceId: 'sp-1', searchIndexing: true },
page: { title: SECRET_TITLE },
},
sharingAllowed: false,
});
const res = makeRes();
await controller.getShare(res, req, 'share-key', 'slug-pageB');
// The plain index stream was sent, so the restricted title never reached
// the response (it is only ever interpolated into the meta HTML string).
expect(res.sent).toBe(STREAM_SENTINEL);
expect(res.sent).not.toBe(SECRET_TITLE);
});
it('injects the title + meta for a readable, still-shared page', async () => {
const { controller } = makeController({
resolved: {
share: { spaceId: 'sp-1', searchIndexing: true },
page: { title: 'Public Handbook' },
},
sharingAllowed: true,
});
const res = makeRes();
await controller.getShare(res, req, 'share-key', 'slug-pageA');
expect(typeof res.sent).toBe('string');
expect(res.sent as string).toContain('<title>Public Handbook</title>');
expect(res.sent as string).toContain('og:title');
// searchIndexing on => crawlable (no noindex).
expect(res.sent as string).not.toContain('content="noindex"');
});
it('adds robots=noindex when the share opted out of search indexing', async () => {
const { controller } = makeController({
resolved: {
share: { spaceId: 'sp-1', searchIndexing: false },
page: { title: 'Internal Notes' },
},
sharingAllowed: true,
});
const res = makeRes();
await controller.getShare(res, req, 'share-key', 'slug-pageA');
expect(res.sent as string).toContain('content="noindex"');
});
});

View File

@@ -63,19 +63,38 @@ export class ShareSeoController {
const pageId = this.extractPageSlugId(pageSlug);
const share = await this.shareService.getShareForPage(
// Funnel through the canonical readable-share boundary (NOT the raw
// getShareForPage) so the restricted-ancestor gate runs: a permission-
// restricted descendant of an includeSubPages share must NOT leak its
// title to anonymous visitors / crawlers (red-team finding #3). null =>
// not publicly readable => serve the plain SPA index with no meta.
const resolved = await this.shareService.resolveReadableSharePage(
undefined,
pageId,
workspace.id,
);
if (!share) {
if (!resolved) {
return this.sendIndex(indexFilePath, res);
}
// Honour a workspace/space-level sharing toggle flipped off AFTER this
// share was created: the content API gates on isSharingAllowed, so the SEO
// path must too or it keeps serving the title for a no-longer-shared page.
const sharingAllowed = await this.shareService.isSharingAllowed(
workspace.id,
resolved.share.spaceId,
);
if (!sharingAllowed) {
return this.sendIndex(indexFilePath, res);
}
const html = fs.readFileSync(indexFilePath, 'utf8');
// Title of the PAGE being viewed (server-resolved), and noindex unless the
// share opted into search indexing (buildShareMetaHtml injects it).
let transformedHtml = buildShareMetaHtml(html, {
title: share?.sharedPage.title,
searchIndexing: share.searchIndexing,
title: resolved.page.title,
searchIndexing: resolved.share.searchIndexing,
});
// Deliberate same-origin tracker surface: this is the ONE place where an

View File

@@ -0,0 +1,38 @@
import { jsonbBind } from './utils';
/**
* Unit tests for jsonbBind: THE shared helper that encodes a JS array/object as
* a jsonb bind (or null when there is nothing to persist). It is the last line
* of defence before a jsonb column write, so the null-vs-bind decision is what
* matters here. We assert only null vs non-null because the non-null value is a
* kysely `sql` template fragment whose internal shape is an implementation
* detail of the SQL tag (the `::text::jsonb` double-encoding fix is verified
* end-to-end by the repo integration specs, where a real DB round-trip can
* actually observe `jsonb_typeof`).
*/
describe('jsonbBind', () => {
it('returns null for null / undefined', () => {
expect(jsonbBind(null)).toBeNull();
expect(jsonbBind(undefined)).toBeNull();
});
it('returns null for an empty array (nothing to persist)', () => {
expect(jsonbBind([])).toBeNull();
});
it('returns null for an empty object (nothing to persist)', () => {
expect(jsonbBind({})).toBeNull();
});
it('returns a (non-null) bind for a non-empty array', () => {
const out = jsonbBind(['search', 'crawl']);
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
it('returns a (non-null) bind for a non-empty object', () => {
const out = jsonbBind({ driver: 'gemini', chatModel: 'gemini-2.0-flash' });
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
});

View File

@@ -0,0 +1,19 @@
import { type Kysely } from 'kysely';
export async function up(db: Kysely<any>): Promise<void> {
// Per-server, admin-authored instruction text injected into the agent system
// prompt next to the server's tool descriptions (#180). NON-secret (unlike
// headers_enc): it IS returned in admin views/forms. Nullable: a server may
// have no guidance. Trusted text — it goes inside the prompt safety sandwich.
await db.schema
.alterTable('ai_mcp_servers')
.addColumn('instructions', 'text', (col) => col)
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema
.alterTable('ai_mcp_servers')
.dropColumn('instructions')
.execute();
}

View File

@@ -35,7 +35,13 @@ describe('AiAgentRoleRepo.findLiveEnabled', () => {
const result = await repo.findLiveEnabled('r-1', 'ws-1');
expect(result).toBe(role);
// The repo normalizes the row (modelConfig parse), so it returns a COPY, not
// the same reference; assert the row's fields are carried through.
expect(result).toMatchObject({
id: 'r-1',
workspaceId: 'ws-1',
enabled: true,
});
expect(db.selectFrom).toHaveBeenCalledWith('aiAgentRoles');
// Every security filter must be present.
expect(where).toHaveBeenCalledWith('id', '=', 'r-1');

View File

@@ -1,8 +1,7 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { sql } from 'kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import { dbOrTx, jsonbBind } from '../../utils';
import { AiAgentRole } from '@docmost/db/types/entity.types';
/** The jsonb shape persisted in `model_config` (loosely typed for the column). */
@@ -23,13 +22,14 @@ export class AiAgentRoleRepo {
id: string,
workspaceId: string,
): Promise<AiAgentRole | undefined> {
return this.db
const row = await this.db
.selectFrom('aiAgentRoles')
.selectAll('aiAgentRoles')
.where('id', '=', id)
.where('workspaceId', '=', workspaceId)
.where('deletedAt', 'is', null)
.executeTakeFirst();
return row ? normalizeRow(row) : row;
}
/**
@@ -45,7 +45,7 @@ export class AiAgentRoleRepo {
id: string,
workspaceId: string,
): Promise<AiAgentRole | undefined> {
return this.db
const row = await this.db
.selectFrom('aiAgentRoles')
.selectAll('aiAgentRoles')
.where('id', '=', id)
@@ -53,17 +53,19 @@ export class AiAgentRoleRepo {
.where('deletedAt', 'is', null)
.where('enabled', '=', true)
.executeTakeFirst();
return row ? normalizeRow(row) : row;
}
/** All live roles for the workspace (management list + chat picker). */
async listByWorkspace(workspaceId: string): Promise<AiAgentRole[]> {
return this.db
const rows = await this.db
.selectFrom('aiAgentRoles')
.selectAll('aiAgentRoles')
.where('workspaceId', '=', workspaceId)
.where('deletedAt', 'is', null)
.orderBy('createdAt', 'asc')
.execute();
return rows.map(normalizeRow);
}
async insert(
@@ -83,7 +85,7 @@ export class AiAgentRoleRepo {
trx?: KyselyTransaction,
): Promise<AiAgentRole> {
const db = dbOrTx(this.db, trx);
return db
const row = await db
.insertInto('aiAgentRoles')
.values({
workspaceId: values.workspaceId,
@@ -92,7 +94,11 @@ export class AiAgentRoleRepo {
emoji: values.emoji ?? null,
description: values.description ?? null,
instructions: values.instructions,
modelConfig: jsonbObject(values.modelConfig),
// Cast: the generated `model_config` column type is the broad JsonValue
// union, which the concrete RawBuilder<Record> is not structurally
// assignable to (same reason the old jsonbObject cast to any).
// eslint-disable-next-line @typescript-eslint/no-explicit-any
modelConfig: jsonbBind(values.modelConfig) as any,
enabled: values.enabled ?? true,
autoStart: values.autoStart ?? true,
// Empty string is treated as "no custom text" => null.
@@ -100,6 +106,7 @@ export class AiAgentRoleRepo {
})
.returningAll()
.executeTakeFirst();
return normalizeRow(row);
}
async update(
@@ -127,7 +134,7 @@ export class AiAgentRoleRepo {
if (patch.description !== undefined) set.description = patch.description;
if (patch.instructions !== undefined) set.instructions = patch.instructions;
if (patch.modelConfig !== undefined) {
set.modelConfig = jsonbObject(patch.modelConfig);
set.modelConfig = jsonbBind(patch.modelConfig);
}
if (patch.enabled !== undefined) set.enabled = patch.enabled;
if (patch.autoStart !== undefined) set.autoStart = patch.autoStart;
@@ -163,16 +170,40 @@ export class AiAgentRoleRepo {
}
/**
* Encode an object as a jsonb bind for the `model_config` column. The postgres
* driver would otherwise need an explicit cast; bind the JSON text and cast it.
* Returns null for null/undefined/empty objects. Cast to `any` because the
* generated column type is the broad `JsonValue` union, which a concrete object
* type is not structurally assignable to.
* Parse the `model_config` value read from the DB into the object the entity
* type promises. Rows written by the old double-encoding bind (`::jsonb` instead
* of `::text::jsonb`) round-trip as a JSON STRING, so the driver hands back e.g.
* `'{"driver":"gemini"}'` rather than an object; the read-path check
* `typeof cfg === 'object'` then failed and the model override was SILENTLY
* dropped (the role fell back to the default model). Be tolerant: a JSON string
* is parsed; an already-parsed object passes through; null / a non-object (incl.
* an array) / unparseable value becomes null (= no override). This self-heals
* already-corrupted rows on read, no migration required.
*/
export function jsonbObject(value: ModelConfigValue | undefined) {
if (value === null || value === undefined || Object.keys(value).length === 0) {
return null;
export function parseModelConfig(
value: unknown,
): Record<string, unknown> | null {
let v: unknown = value;
if (typeof v === 'string') {
try {
v = JSON.parse(v); // legacy double-encoded read
} catch {
return null;
}
}
// eslint-disable-next-line @typescript-eslint/no-explicit-any
return sql`${JSON.stringify(value)}::jsonb` as any;
return v !== null && typeof v === 'object' && !Array.isArray(v)
? (v as Record<string, unknown>)
: null;
}
/** Normalize a DB row so `modelConfig` is always an object or null. The cast
* bridges parseModelConfig's concrete `Record | null` to the column's broad
* generated `JsonValue` type (an object is a valid JsonValue at runtime). */
function normalizeRow(row: AiAgentRole): AiAgentRole {
return {
...row,
modelConfig: parseModelConfig(
row.modelConfig,
) as AiAgentRole['modelConfig'],
};
}

View File

@@ -0,0 +1,46 @@
import { parseModelConfig } from './ai-agent-roles.repo';
/**
* Unit tests for parseModelConfig: the read-side normalizer that repairs the
* jsonb double-encoding regression on `model_config`. Rows written by the old
* `::jsonb` bind round-trip as a JSON STRING, which the read path's
* `typeof === 'object'` check rejected — silently dropping the model override.
* parseModelConfig accepts an already-parsed object, parses a legacy JSON
* string, and rejects everything that is not an object (null = no override).
*/
describe('parseModelConfig', () => {
it('passes an already-parsed object through', () => {
expect(parseModelConfig({ driver: 'gemini' })).toEqual({
driver: 'gemini',
});
});
it('parses a legacy double-encoded JSON string into an object', () => {
expect(parseModelConfig('{"driver":"gemini","chatModel":"x"}')).toEqual({
driver: 'gemini',
chatModel: 'x',
});
});
it('returns null for null / undefined', () => {
expect(parseModelConfig(null)).toBeNull();
expect(parseModelConfig(undefined)).toBeNull();
});
it('returns null for a non-object JSON value (string/number/array)', () => {
expect(parseModelConfig('"justastring"')).toBeNull();
expect(parseModelConfig('42')).toBeNull();
// An array is an object in JS but not a valid model_config shape.
expect(parseModelConfig('["a","b"]')).toBeNull();
expect(parseModelConfig(['a', 'b'])).toBeNull();
});
it('returns null for an unparseable string', () => {
expect(parseModelConfig('not json at all')).toBeNull();
});
it('returns null for a raw non-object primitive', () => {
expect(parseModelConfig(42 as unknown)).toBeNull();
expect(parseModelConfig(true as unknown)).toBeNull();
});
});

View File

@@ -1,4 +1,4 @@
import { parseToolAllowlist } from './ai-mcp-server.repo';
import { parseToolAllowlist, blankToNull } from './ai-mcp-server.repo';
/**
* The `tool_allowlist` jsonb column historically round-trips as a JSON STRING
@@ -10,7 +10,10 @@ import { parseToolAllowlist } from './ai-mcp-server.repo';
*/
describe('parseToolAllowlist', () => {
it('passes a real string array through unchanged', () => {
expect(parseToolAllowlist(['search', 'crawl'])).toEqual(['search', 'crawl']);
expect(parseToolAllowlist(['search', 'crawl'])).toEqual([
'search',
'crawl',
]);
});
it('parses a JSON-string array (the double-encoded read) into an array', () => {
@@ -46,3 +49,26 @@ describe('parseToolAllowlist', () => {
expect(parseToolAllowlist(true as unknown)).toBeNull();
});
});
/**
* `blankToNull` normalizes the per-server `instructions` free text before it is
* stored (#180): a missing/blank/whitespace-only value becomes null (so an empty
* guide is never persisted), any other value is trimmed.
*/
describe('blankToNull', () => {
it('returns null for null / undefined', () => {
expect(blankToNull(null)).toBeNull();
expect(blankToNull(undefined)).toBeNull();
});
it('returns null for an empty / whitespace-only string', () => {
expect(blankToNull('')).toBeNull();
expect(blankToNull(' ')).toBeNull();
expect(blankToNull('\n\t ')).toBeNull();
});
it('trims and returns a non-blank string', () => {
expect(blankToNull(' use the search tool ')).toBe('use the search tool');
expect(blankToNull('guide')).toBe('guide');
});
});

View File

@@ -1,10 +1,11 @@
import { Injectable } from '@nestjs/common';
import { Injectable, Logger } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { sql } from 'kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
import { dbOrTx, jsonbBind } from '../../utils';
import { AiMcpServer } from '@docmost/db/types/entity.types';
const logger = new Logger('AiMcpServerRepo');
/**
* Repository for per-workspace external MCP servers the agent may use (§5.4).
*
@@ -60,6 +61,8 @@ export class AiMcpServerRepo {
url: string;
headersEnc?: string | null;
toolAllowlist?: string[] | null;
// Admin-authored prompt guidance; blank/whitespace normalizes to null.
instructions?: string | null;
enabled?: boolean;
},
trx?: KyselyTransaction,
@@ -75,7 +78,9 @@ export class AiMcpServerRepo {
headersEnc: values.headersEnc ?? null,
// jsonb column: the postgres driver would otherwise encode a JS array as
// a Postgres array literal. Bind the JSON text and cast it to jsonb.
toolAllowlist: jsonbArray(values.toolAllowlist),
toolAllowlist: jsonbBind(values.toolAllowlist),
// Plain text column: blank/whitespace-only guidance is stored as null.
instructions: blankToNull(values.instructions),
enabled: values.enabled ?? true,
})
.returningAll()
@@ -93,6 +98,8 @@ export class AiMcpServerRepo {
headersEnc?: string | null;
// undefined => leave unchanged; null => clear; string[] => set.
toolAllowlist?: string[] | null;
// undefined => leave unchanged; null/blank => clear; string => set.
instructions?: string | null;
enabled?: boolean;
},
trx?: KyselyTransaction,
@@ -104,7 +111,11 @@ export class AiMcpServerRepo {
if (patch.url !== undefined) set.url = patch.url;
if (patch.headersEnc !== undefined) set.headersEnc = patch.headersEnc;
if (patch.toolAllowlist !== undefined) {
set.toolAllowlist = jsonbArray(patch.toolAllowlist);
set.toolAllowlist = jsonbBind(patch.toolAllowlist);
}
if (patch.instructions !== undefined) {
// Blank/whitespace-only guidance clears the column (stored as null).
set.instructions = blankToNull(patch.instructions);
}
if (patch.enabled !== undefined) set.enabled = patch.enabled;
await db
@@ -130,57 +141,53 @@ export class AiMcpServerRepo {
}
/**
* Encode a string[] as a jsonb bind for the `tool_allowlist` column. Passing a
* plain JS array to the postgres driver would serialize it as a Postgres array
* literal (incompatible with jsonb), so we bind the JSON text and cast it.
*
* The cast is `::text::jsonb`, NOT `::jsonb`: if the parameter is bound straight
* to a jsonb cast, node-postgres infers its type as jsonb and JSON-stringifies
* the (already-JSON) string a SECOND time, so the column ends up holding a jsonb
* STRING SCALAR (`"[\"a\"]"`) instead of a jsonb ARRAY. Forcing the param through
* `::text` first binds it as text (sent verbatim), and `::jsonb` then parses it
* into a real array. (`normalizeRow` below repairs rows written the old way.)
*
* Returns null for null/empty arrays (an empty allowlist means "no restriction"
* is not intended — callers pass null to clear; an empty array is normalized to
* null here so it never round-trips as `[]`).
* Normalize an optional free-text field to a stored value: a missing/blank/
* whitespace-only string becomes null (so an "empty" guide is never persisted),
* any other string is trimmed. Returns null for null/undefined input.
*/
function jsonbArray(value: string[] | null | undefined) {
if (value === null || value === undefined || value.length === 0) {
return null;
}
// Typed as string[] so it is assignable to the toolAllowlist column.
return sql<string[]>`${JSON.stringify(value)}::text::jsonb`;
export function blankToNull(value: string | null | undefined): string | null {
if (value == null) return null;
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : null;
}
/**
* Parse the `toolAllowlist` value read from the DB into the `string[] | null`
* the entity type promises. The jsonb column historically round-trips as a JSON
* STRING (rows written by the old double-encoding `jsonbArray`, see above), so
* the driver hands back a string like `'["a","b"]'` rather than an array. Be
* tolerant: an already-parsed array passes through; a JSON string is parsed; null
* / a non-array / unparseable value becomes null (unrestricted).
* STRING (rows written by the old double-encoding bind before the `::text::jsonb`
* fix), so the driver hands back a string like `'["a","b"]'` rather than an
* array. Be tolerant: normalize a JSON string to its value, then accept it only
* if it is an array of strings; null / a non-array / unparseable value / an
* array with a non-string element all become null (unrestricted).
*/
export function parseToolAllowlist(value: unknown): string[] | null {
if (value == null) return null;
if (Array.isArray(value)) {
return value.every((v) => typeof v === 'string') ? (value as string[]) : null;
}
if (typeof value === 'string') {
let v: unknown = value;
if (typeof v === 'string') {
try {
const parsed = JSON.parse(value);
return Array.isArray(parsed) &&
parsed.every((v) => typeof v === 'string')
? (parsed as string[])
: null;
v = JSON.parse(v); // legacy double-encoded read
} catch {
return null;
}
}
return null;
return Array.isArray(v) && v.every((x) => typeof x === 'string')
? (v as string[])
: null;
}
/** Normalize a DB row so `toolAllowlist` is always `string[] | null`. */
/**
* Normalize a DB row so `toolAllowlist` is always `string[] | null`.
*
* FAIL-OPEN logging: a stored value that is present but cannot be parsed into a
* string[] (corrupt JSON, a non-array, non-string elements) degrades to `null` =
* "no restriction", so the agent silently gets ALL of the server's tools. Log
* one line (server id only, never the contents) so that widening is not silent.
*/
function normalizeRow(row: AiMcpServer): AiMcpServer {
return { ...row, toolAllowlist: parseToolAllowlist(row.toolAllowlist) };
const parsed = parseToolAllowlist(row.toolAllowlist);
if (parsed === null && row.toolAllowlist != null) {
logger.warn(
`Corrupt tool_allowlist for MCP server ${row.id}; ignoring it (no tool restriction applied)`,
);
}
return { ...row, toolAllowlist: parsed };
}

View File

@@ -20,8 +20,15 @@ export interface AiMcpServers {
// Encrypted JSON of the auth headers. Nullable (a server may need no auth).
headersEnc: string | null;
// Optional allowlist of remote tool names to expose; null = expose all.
// Stored as jsonb; reads come back as a string[] from the postgres driver.
// Stored as jsonb. The postgres driver may return a JSON string for legacy
// double-encoded rows; `AiMcpServerRepo` normalizes every read to
// `string[] | null` via `parseToolAllowlist`.
toolAllowlist: string[] | null;
// Admin-authored guidance ("how/when to use this server's tools") injected
// into the agent system prompt (#180). Unlike `headersEnc` this is NON-secret
// and IS returned in admin views/forms. Plain text column (no jsonb). Null =
// no guidance. Trusted text — it goes inside the prompt safety sandwich.
instructions: string | null;
enabled: Generated<boolean>;
createdAt: Generated<Timestamp>;
updatedAt: Generated<Timestamp>;

View File

@@ -1,3 +1,4 @@
import { sql, RawBuilder } from 'kysely';
import { KyselyDB, KyselyTransaction } from './types/kysely.types';
/*
@@ -31,3 +32,35 @@ export function dbOrTx(
return db; // Use normal database instance
}
}
/**
* Bind a JS array/object as a `jsonb` column value, working around a postgres
* driver double-encoding quirk. THE single implementation — repos that persist
* jsonb (`tool_allowlist`, `model_config`, ...) call this instead of re-deriving
* the cast.
*
* THE QUIRK: with the `kysely-postgres-js` / postgres.js driver, casting a bound
* parameter straight to `::jsonb` makes the driver infer the param type as jsonb
* and JSON-stringify the (already-JSON) text a SECOND time, so the column ends
* up holding a jsonb STRING SCALAR (`"[\"a\"]"` / `"{\"k\":1}"`) instead of a
* real jsonb array/object. Read paths then see a string, not the structure, and
* silently fall back (an allowlist becomes "unrestricted", a model override is
* ignored). Forcing the param through `::text` first binds it as text (sent
* verbatim); `::jsonb` then parses it into a real array/object. Read-side
* parsers repair rows written the old buggy way without a migration.
*
* Returns `null` for null/undefined and for "empty" values (an empty array, or
* an object with no own enumerable keys) — callers treat empty as "clear/unset",
* so an empty allowlist/config never round-trips as `[]`/`{}`.
*/
export function jsonbBind<T>(
value: T | null | undefined,
): RawBuilder<T> | null {
if (value === null || value === undefined) return null;
if (Array.isArray(value)) {
if (value.length === 0) return null;
} else if (typeof value === 'object') {
if (Object.keys(value as object).length === 0) return null;
}
return sql<T>`${JSON.stringify(value)}::text::jsonb`;
}

View File

@@ -1,4 +1,5 @@
import { Kysely } from 'kysely';
import { Kysely, sql } from 'kysely';
import { randomUUID } from 'node:crypto';
import { AiAgentRoleRepo } from '@docmost/db/repos/ai-agent-roles/ai-agent-roles.repo';
import { getTestDb, destroyTestDb, createWorkspace } from './db';
@@ -25,8 +26,16 @@ describe('AiAgentRoleRepo isolation + partial unique index [integration]', () =>
});
it('findById / listByWorkspace exclude soft-deleted rows', async () => {
const live = await repo.insert({ workspaceId: w1, name: 'Live', instructions: 'x' });
const dead = await repo.insert({ workspaceId: w1, name: 'Dead', instructions: 'x' });
const live = await repo.insert({
workspaceId: w1,
name: 'Live',
instructions: 'x',
});
const dead = await repo.insert({
workspaceId: w1,
name: 'Dead',
instructions: 'x',
});
await repo.softDelete(dead.id, w1);
expect(await repo.findById(live.id, w1)).toBeDefined();
@@ -38,7 +47,11 @@ describe('AiAgentRoleRepo isolation + partial unique index [integration]', () =>
});
it('findById of a W2 role from W1 context returns undefined (tenant isolation)', async () => {
const w2role = await repo.insert({ workspaceId: w2, name: 'W2Role', instructions: 'x' });
const w2role = await repo.insert({
workspaceId: w2,
name: 'W2Role',
instructions: 'x',
});
expect(await repo.findById(w2role.id, w2)).toBeDefined();
// Same id, wrong workspace context -> not visible.
@@ -58,21 +71,100 @@ describe('AiAgentRoleRepo isolation + partial unique index [integration]', () =>
});
it('same name is reusable after softDelete (partial unique index WHERE deleted_at IS NULL)', async () => {
const first = await repo.insert({ workspaceId: w1, name: 'Reusable', instructions: 'x' });
const first = await repo.insert({
workspaceId: w1,
name: 'Reusable',
instructions: 'x',
});
await repo.softDelete(first.id, w1);
// Now inserting the same name must succeed because the soft-deleted row is
// excluded from the partial unique index.
const second = await repo.insert({ workspaceId: w1, name: 'Reusable', instructions: 'x' });
const second = await repo.insert({
workspaceId: w1,
name: 'Reusable',
instructions: 'x',
});
expect(second.id).toBeDefined();
expect(second.id).not.toBe(first.id);
});
it('same name in W1 and W2 is allowed (unique is per-workspace)', async () => {
const a = await repo.insert({ workspaceId: w1, name: 'CrossTenant', instructions: 'x' });
const b = await repo.insert({ workspaceId: w2, name: 'CrossTenant', instructions: 'x' });
const a = await repo.insert({
workspaceId: w1,
name: 'CrossTenant',
instructions: 'x',
});
const b = await repo.insert({
workspaceId: w2,
name: 'CrossTenant',
instructions: 'x',
});
expect(a.id).toBeDefined();
expect(b.id).toBeDefined();
expect(a.id).not.toBe(b.id);
});
// model_config jsonb round-trip (issue #173 §1): the same double-encoding bug
// PR #172 fixed for tool_allowlist lived in jsonbObject. A DB round-trip is the
// only way to observe it — the write must land as a real jsonb OBJECT, and a
// legacy string-scalar row must self-heal on read (else the model override is
// silently dropped and the role falls back to the default model).
const jsonbTypeof = async (id: string): Promise<string | null> => {
const res = await sql<{ t: string | null }>`
SELECT jsonb_typeof(model_config) AS t
FROM ai_agent_roles WHERE id = ${id}
`.execute(db);
return res.rows[0]?.t ?? null;
};
it('insert stores model_config as a jsonb OBJECT and reads it back as an object', async () => {
const role = await repo.insert({
workspaceId: w1,
name: `Model-${randomUUID()}`,
instructions: 'x',
modelConfig: { driver: 'gemini', chatModel: 'gemini-2.0-flash' },
});
expect(await jsonbTypeof(role.id)).toBe('object');
// The returned row is already normalized to an object.
expect(role.modelConfig).toEqual({
driver: 'gemini',
chatModel: 'gemini-2.0-flash',
});
const found = await repo.findById(role.id, w1);
expect(found?.modelConfig).toEqual({
driver: 'gemini',
chatModel: 'gemini-2.0-flash',
});
});
it('an empty model_config is normalized to null (no override)', async () => {
const role = await repo.insert({
workspaceId: w1,
name: `Empty-${randomUUID()}`,
instructions: 'x',
modelConfig: {},
});
// The column is SQL NULL, so jsonb_typeof returns SQL NULL (JS null).
expect(await jsonbTypeof(role.id)).toBeNull();
expect((await repo.findById(role.id, w1))?.modelConfig).toBeNull();
});
it('repairs a legacy double-encoded (string scalar) model_config on read', async () => {
const id = randomUUID();
// Seed the corrupt string-scalar shape the old `::jsonb` bind produced.
await sql`
INSERT INTO ai_agent_roles (id, workspace_id, name, instructions, model_config)
VALUES (
${id}, ${w1}, ${`Legacy-${id}`}, 'x',
to_jsonb(${'{"driver":"openai","chatModel":"gpt"}'}::text)
)
`.execute(db);
expect(await jsonbTypeof(id)).toBe('string'); // sanity: really corrupt
expect((await repo.findById(id, w1))?.modelConfig).toEqual({
driver: 'openai',
chatModel: 'gpt',
});
});
});

View File

@@ -0,0 +1,175 @@
import { Kysely, sql } from 'kysely';
import { randomUUID } from 'node:crypto';
import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo';
import { getTestDb, destroyTestDb, createWorkspace } from './db';
/**
* AiMcpServerRepo `tool_allowlist` jsonb round-trip (PR #172 / issue #173 §3).
*
* The fix under test is a DB round-trip, so a unit test cannot observe it: the
* write must land as a real jsonb ARRAY (not a double-encoded string scalar),
* and the read must repair any legacy string-scalar rows. The read-side
* `parseToolAllowlist` MASKS a write regression (it parses the string back), so
* without this integration check, reverting `::text::jsonb` to `::jsonb` would
* keep every unit test green while silently corrupting the column again.
*/
describe('AiMcpServerRepo tool_allowlist jsonb round-trip [integration]', () => {
let db: Kysely<any>;
let repo: AiMcpServerRepo;
let ws: string;
beforeAll(async () => {
db = getTestDb();
repo = new AiMcpServerRepo(db as any);
ws = (await createWorkspace(db)).id;
});
afterAll(async () => {
await destroyTestDb();
});
const jsonbTypeof = async (id: string): Promise<string | null> => {
const res = await sql<{ t: string | null }>`
SELECT jsonb_typeof(tool_allowlist) AS t
FROM ai_mcp_servers WHERE id = ${id}
`.execute(db);
return res.rows[0]?.t ?? null;
};
it('insert stores the allowlist as a jsonb ARRAY (not a string scalar)', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
toolAllowlist: ['search', 'crawl'],
});
// The column holds a real jsonb array — the whole point of ::text::jsonb.
expect(await jsonbTypeof(row.id)).toBe('array');
// And the read returns a genuine string[], not a JSON string.
const found = await repo.findById(row.id, ws);
expect(found?.toolAllowlist).toEqual(['search', 'crawl']);
expect(Array.isArray(found?.toolAllowlist)).toBe(true);
});
it('an empty allowlist is normalized to null (no restriction), not []', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
toolAllowlist: [],
});
// The column is SQL NULL, so jsonb_typeof returns SQL NULL (JS null).
expect(await jsonbTypeof(row.id)).toBeNull();
expect((await repo.findById(row.id, ws))?.toolAllowlist).toBeNull();
});
it('repairs a legacy double-encoded (string scalar) row on read (self-heal)', async () => {
// Seed a row whose tool_allowlist is a jsonb STRING SCALAR holding the JSON
// text — exactly what the old `::jsonb` double-encoding produced.
const id = randomUUID();
await sql`
INSERT INTO ai_mcp_servers (id, workspace_id, name, transport, url, tool_allowlist)
VALUES (
${id}, ${ws}, ${`srv-${id}`}, 'http', 'https://example.com/mcp',
to_jsonb(${'["alpha","beta"]'}::text)
)
`.execute(db);
// Sanity: the seeded column really IS the corrupt string-scalar shape.
expect(await jsonbTypeof(id)).toBe('string');
// The repo read heals it back to a real string[].
expect((await repo.findById(id, ws))?.toolAllowlist).toEqual([
'alpha',
'beta',
]);
const enabled = await repo.listEnabled(ws);
const healed = enabled.find((r) => r.id === id);
expect(healed?.toolAllowlist).toEqual(['alpha', 'beta']);
});
});
/**
* AiMcpServerRepo `instructions` text round-trip (#180). The column is plain
* text (no jsonb); blank/whitespace is normalized to null on both insert and
* update so an empty guide is never persisted.
*/
describe('AiMcpServerRepo instructions round-trip [integration]', () => {
let db: Kysely<any>;
let repo: AiMcpServerRepo;
let ws: string;
beforeAll(async () => {
db = getTestDb();
repo = new AiMcpServerRepo(db as any);
ws = (await createWorkspace(db)).id;
});
afterAll(async () => {
await destroyTestDb();
});
it('insert stores trimmed non-blank instructions and reads them back', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
instructions: ' Use search for fresh facts. ',
});
expect((await repo.findById(row.id, ws))?.instructions).toBe(
'Use search for fresh facts.',
);
});
it('insert normalizes blank/whitespace instructions to null', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
instructions: ' ',
});
expect((await repo.findById(row.id, ws))?.instructions).toBeNull();
});
it('insert with omitted instructions stores null', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
});
expect((await repo.findById(row.id, ws))?.instructions).toBeNull();
});
it('update sets, clears (blank => null), and leaves unchanged when absent', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
instructions: 'initial guide',
});
// Set a new value.
await repo.update(row.id, ws, { instructions: 'updated guide' });
expect((await repo.findById(row.id, ws))?.instructions).toBe(
'updated guide',
);
// Absent in the patch => unchanged.
await repo.update(row.id, ws, { name: 'renamed' });
expect((await repo.findById(row.id, ws))?.instructions).toBe(
'updated guide',
);
// Blank => cleared to null.
await repo.update(row.id, ws, { instructions: ' ' });
expect((await repo.findById(row.id, ws))?.instructions).toBeNull();
});
});

View File

@@ -1,14 +1,15 @@
import { EditorState, Plugin, PluginKey } from "@tiptap/pm/state";
import { Decoration, DecorationSet } from "@tiptap/pm/view";
import { Node as ProseMirrorNode } from "@tiptap/pm/model";
import { EditorState, Plugin, PluginKey } from '@tiptap/pm/state';
import { Decoration, DecorationSet } from '@tiptap/pm/view';
import { Node as ProseMirrorNode } from '@tiptap/pm/model';
import {
FOOTNOTE_DEFINITION_NAME,
FOOTNOTE_REFERENCE_NAME,
computeFootnoteNumbers,
} from "./footnote-util";
computeFootnoteRefCounts,
} from './footnote-util';
export const footnoteNumberingPluginKey = new PluginKey<FootnoteNumberingState>(
"footnoteNumbering",
'footnoteNumbering',
);
/**
@@ -21,6 +22,9 @@ export const footnoteNumberingPluginKey = new PluginKey<FootnoteNumberingState>(
interface FootnoteNumberingState {
/** referenceId -> 1-based display number, for the current doc. */
numbers: Map<string, number>;
/** referenceId -> number of reference occurrences (>= 1), for the definition's
* multi-backlink UI (#168). */
refCounts: Map<string, number>;
/** Decorations rendering those numbers (refs + definitions). */
decorations: DecorationSet;
}
@@ -46,6 +50,7 @@ function buildFootnoteNumberingState(
doc: ProseMirrorNode,
): FootnoteNumberingState {
const numbers = computeFootnoteNumbers(doc);
const refCounts = computeFootnoteRefCounts(doc);
const decorations: Decoration[] = [];
doc.descendants((node, pos) => {
@@ -54,7 +59,7 @@ function buildFootnoteNumberingState(
if (num != null) {
decorations.push(
Decoration.node(pos, pos + node.nodeSize, {
"data-footnote-number": String(num),
'data-footnote-number': String(num),
style: `--footnote-number: "${num}";`,
}),
);
@@ -65,7 +70,7 @@ function buildFootnoteNumberingState(
if (num != null) {
decorations.push(
Decoration.node(pos, pos + node.nodeSize, {
"data-footnote-number": String(num),
'data-footnote-number': String(num),
style: `--footnote-number: "${num}";`,
}),
);
@@ -73,7 +78,11 @@ function buildFootnoteNumberingState(
}
});
return { numbers, decorations: DecorationSet.create(doc, decorations) };
return {
numbers,
refCounts,
decorations: DecorationSet.create(doc, decorations),
};
}
/**
@@ -90,6 +99,16 @@ export function getFootnoteNumber(
return footnoteNumberingPluginKey.getState(state)?.numbers.get(id);
}
/**
* Read the cached reference-occurrence count for `id` (how many `[^id]` links
* point at this definition). Drives the definition's multi-backlink UI (#168):
* `> 1` renders ↩ a b c …, each scrolling to its own occurrence. Returns 0 when
* the plugin is not installed or the id is unknown (caller treats as single).
*/
export function getFootnoteRefCount(state: EditorState, id: string): number {
return footnoteNumberingPluginKey.getState(state)?.refCounts.get(id) ?? 0;
}
/**
* ProseMirror plugin that renders footnote numbers as decorations. It never
* mutates the document (safe in read-only / share and in collaboration) — it

View File

@@ -1,14 +1,14 @@
import { mergeAttributes, Node } from "@tiptap/core";
import { TextSelection, Transaction } from "@tiptap/pm/state";
import { ReactNodeViewRenderer } from "@tiptap/react";
import { mergeAttributes, Node } from '@tiptap/core';
import { TextSelection, Transaction } from '@tiptap/pm/state';
import { ReactNodeViewRenderer } from '@tiptap/react';
import {
FOOTNOTE_DEFINITION_NAME,
FOOTNOTE_REFERENCE_NAME,
FOOTNOTES_LIST_NAME,
generateFootnoteId,
} from "./footnote-util";
import { footnoteNumberingPlugin } from "./footnote-numbering";
import { footnoteSyncPlugin, footnotePastePlugin } from "./footnote-sync";
} from './footnote-util';
import { footnoteNumberingPlugin } from './footnote-numbering';
import { footnoteSyncPlugin, footnotePastePlugin } from './footnote-sync';
export interface FootnoteReferenceOptions {
HTMLAttributes: Record<string, any>;
@@ -27,7 +27,7 @@ export interface FootnoteReferenceOptions {
enableSync?: boolean;
}
declare module "@tiptap/core" {
declare module '@tiptap/core' {
interface Commands<ReturnType> {
footnote: {
/**
@@ -42,8 +42,11 @@ declare module "@tiptap/core" {
removeFootnote: (id: string) => ReturnType;
/** Scroll to (and focus) a footnote definition by id. */
scrollToFootnote: (id: string) => ReturnType;
/** Scroll to (and select) a footnote reference by id. */
scrollToReference: (id: string) => ReturnType;
/** Scroll to a footnote reference by id. `index` selects WHICH occurrence
* to scroll to when the id is referenced more than once (reuse, #166):
* 0-based, defaults to the first. Used by the definition's multi-backlink
* UI (#168). */
scrollToReference: (id: string, index?: number) => ReturnType;
};
}
}
@@ -66,7 +69,7 @@ export const FootnoteReference = Node.create<FootnoteReferenceOptions>({
// Superscript mark's <sup> rule.
priority: 101,
group: "inline",
group: 'inline',
inline: true,
atom: true,
selectable: true,
@@ -99,10 +102,10 @@ export const FootnoteReference = Node.create<FootnoteReferenceOptions>({
return {
id: {
default: null,
parseHTML: (element) => element.getAttribute("data-id"),
parseHTML: (element) => element.getAttribute('data-id'),
renderHTML: (attributes) => {
if (!attributes.id) return {};
return { "data-id": attributes.id };
return { 'data-id': attributes.id };
},
},
};
@@ -113,7 +116,7 @@ export const FootnoteReference = Node.create<FootnoteReferenceOptions>({
{
// High priority so the Superscript mark (which also matches <sup>) does
// not claim a footnote reference and drop it as empty content.
tag: "sup[data-footnote-ref]",
tag: 'sup[data-footnote-ref]',
priority: 100,
},
];
@@ -121,9 +124,9 @@ export const FootnoteReference = Node.create<FootnoteReferenceOptions>({
renderHTML({ HTMLAttributes }) {
return [
"sup",
'sup',
mergeAttributes(
{ "data-footnote-ref": "", class: "footnote-ref" },
{ 'data-footnote-ref': '', class: 'footnote-ref' },
this.options.HTMLAttributes,
HTMLAttributes,
),
@@ -132,7 +135,7 @@ export const FootnoteReference = Node.create<FootnoteReferenceOptions>({
// Plain-text representation (used by generateText / markdown text fallbacks).
renderText({ node }) {
return `[^${node.attrs.id ?? ""}]`;
return `[^${node.attrs.id ?? ''}]`;
},
addNodeView() {
@@ -170,8 +173,10 @@ export const FootnoteReference = Node.create<FootnoteReferenceOptions>({
// Make sure the parent accepts an inline atom here.
const insertPos = selection.from;
if (!$from.parent.type.spec.content?.includes("inline") &&
!$from.parent.isTextblock) {
if (
!$from.parent.type.spec.content?.includes('inline') &&
!$from.parent.isTextblock
) {
return false;
}
@@ -311,19 +316,23 @@ export const FootnoteReference = Node.create<FootnoteReferenceOptions>({
`[data-footnote-def][data-id="${id}"]`,
) as HTMLElement | null;
if (!dom) return false;
dom.scrollIntoView({ behavior: "smooth", block: "center" });
dom.scrollIntoView({ behavior: 'smooth', block: 'center' });
return true;
},
scrollToReference:
(id: string) =>
(id: string, index = 0) =>
({ editor }) => {
if (!id) return false;
const dom = editor.view.dom.querySelector(
// querySelectorAll returns the occurrences in document order, so the
// index maps 1:1 to the definition's a/b/c backlink (#168). Fall back
// to the first match for an out-of-range index.
const matches = editor.view.dom.querySelectorAll(
`sup[data-footnote-ref][data-id="${id}"]`,
) as HTMLElement | null;
);
const dom = (matches[index] ?? matches[0]) as HTMLElement | undefined;
if (!dom) return false;
dom.scrollIntoView({ behavior: "smooth", block: "center" });
dom.scrollIntoView({ behavior: 'smooth', block: 'center' });
return true;
},
};

View File

@@ -1,12 +1,12 @@
import { Node as ProseMirrorNode } from "@tiptap/pm/model";
import { Node as ProseMirrorNode } from '@tiptap/pm/model';
/**
* Node type names for the footnote feature. Centralized so every part of the
* feature (nodes, plugins, commands) references the same string.
*/
export const FOOTNOTE_REFERENCE_NAME = "footnoteReference";
export const FOOTNOTES_LIST_NAME = "footnotesList";
export const FOOTNOTE_DEFINITION_NAME = "footnoteDefinition";
export const FOOTNOTE_REFERENCE_NAME = 'footnoteReference';
export const FOOTNOTES_LIST_NAME = 'footnotesList';
export const FOOTNOTE_DEFINITION_NAME = 'footnoteDefinition';
/**
* Generate a uuidv7-style id (time-ordered). Implemented locally so editor-ext
@@ -15,10 +15,10 @@ export const FOOTNOTE_DEFINITION_NAME = "footnoteDefinition";
*/
export function generateFootnoteId(): string {
const now = Date.now();
const timeHex = now.toString(16).padStart(12, "0");
const timeHex = now.toString(16).padStart(12, '0');
const rand = (length: number) => {
let out = "";
let out = '';
for (let i = 0; i < length; i++) {
out += Math.floor(Math.random() * 16).toString(16);
}
@@ -26,19 +26,19 @@ export function generateFootnoteId(): string {
};
// version 7 nibble, then variant (8..b) nibble.
const versioned = "7" + rand(3);
const versioned = '7' + rand(3);
const variantNibble = (8 + Math.floor(Math.random() * 4)).toString(16);
const variant = variantNibble + rand(3);
return (
timeHex.slice(0, 8) +
"-" +
'-' +
timeHex.slice(8, 12) +
"-" +
'-' +
versioned +
"-" +
'-' +
variant +
"-" +
'-' +
rand(12)
);
}
@@ -89,7 +89,7 @@ export function deriveFootnoteId(
* Purely deterministic.
*/
function suffix(n: number): string {
let out = "";
let out = '';
let x = n;
while (x > 0) {
const rem = (x - 1) % 25;
@@ -131,3 +131,19 @@ export function computeFootnoteNumbers(
}
return numbers;
}
/**
* Build a map of `referenceId -> number of reference occurrences` (>= 1) from
* document order. After #166 the same id may be referenced multiple times
* (reuse: one number, one definition, N forward links); this count drives the
* definition's multi-backlink UI (↩ a b c …, #168). Pure function of the doc.
*/
export function computeFootnoteRefCounts(
doc: ProseMirrorNode,
): Map<string, number> {
const counts = new Map<string, number>();
for (const id of collectReferenceIds(doc)) {
counts.set(id, (counts.get(id) ?? 0) + 1);
}
return counts;
}

File diff suppressed because it is too large Load Diff

View File

@@ -7,8 +7,7 @@ import { TiptapTransformer } from "@hocuspocus/transformer";
import * as Y from "yjs";
import WebSocket from "ws";
import { convertProseMirrorToMarkdown } from "./lib/markdown-converter.js";
import { updatePageContentRealtime, replacePageContent, markdownToProseMirror, mutatePageContent, buildCollabWsUrl, assertYjsEncodable, } from "./lib/collaboration.js";
import { docmostExtensions } from "./lib/docmost-schema.js";
import { updatePageContentRealtime, replacePageContent, markdownToProseMirror, mutatePageContent, buildCollabWsUrl, assertYjsEncodable, applyDocToFragment, } from "./lib/collaboration.js";
import { footnoteWarningsField } from "./lib/footnote-analyze.js";
import { buildPageTree } from "./lib/tree.js";
import { serializeDocmostMarkdown, parseDocmostMarkdown, } from "./lib/markdown-document.js";
@@ -17,7 +16,7 @@ import { withPageLock } from "./lib/page-lock.js";
import { applyTextEdits, } from "./lib/json-edit.js";
import { getCollabToken, performLogin } from "./lib/auth-utils.js";
import { diffDocs, summarizeChange } from "./lib/diff.js";
import { applyAnchorInDoc, canAnchorInDoc, } from "./lib/comment-anchor.js";
import { applyAnchorInDoc, canAnchorInDoc } from "./lib/comment-anchor.js";
import { blockText, walk, getList, insertMarkerAfter, setCalloutRange, noteItem, mdToInlineNodes, commentsToFootnotes, } from "./lib/transforms.js";
import vm from "node:vm";
// Supported image types, kept as two lookup tables so both a local file
@@ -209,7 +208,9 @@ export class DocmostClient {
// getCollabToken wraps the AxiosError in a plain Error but attaches the
// HTTP status as `.status`, so detect an auth failure via either the raw
// AxiosError shape OR the attached status.
const axiosStatus = axios.isAxiosError(e) ? e.response?.status : undefined;
const axiosStatus = axios.isAxiosError(e)
? e.response?.status
: undefined;
const attachedStatus = e?.status;
const isAuthError = axiosStatus === 401 ||
axiosStatus === 403 ||
@@ -361,14 +362,14 @@ export class DocmostClient {
finish(null, mutationResult);
return;
}
const tempDoc = TiptapTransformer.toYdoc(newDoc, "default", docmostExtensions);
const fragment = ydoc.getXmlFragment("default");
ydoc.transact(() => {
if (fragment.length > 0) {
fragment.delete(0, fragment.length);
}
Y.applyUpdate(ydoc, Y.encodeStateAsUpdate(tempDoc));
});
// Structural diff into the live fragment (issue #152), mirroring
// the main write path: preserves the Yjs ids of unchanged nodes so
// an open editor's cursor is not yanked to the end of the document.
// The previous destructive rewrite (delete-all + applyUpdate of a
// fresh Y.Doc) discarded every node id, so replaceImage — the only
// caller of this method — still reproduced the #152 cursor jump
// (#164). applyDocToFragment runs its own atomic `transact`.
applyDocToFragment(ydoc, newDoc);
}
catch (e) {
finish(e instanceof Error ? e : new Error(String(e)));
@@ -688,7 +689,12 @@ export class DocmostClient {
if (!inserted) {
throw new Error(`table_insert_row: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from get_outline, or a block id inside the table)`);
}
return { success: true, table: tableRef, inserted: true, verify: mutation.verify };
return {
success: true,
table: tableRef,
inserted: true,
verify: mutation.verify,
};
}
/**
* Delete the row at 0-based `index` from a table on the LIVE collab document.
@@ -710,7 +716,12 @@ export class DocmostClient {
if (!deleted) {
throw new Error(`table_delete_row: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from get_outline, or a block id inside the table)`);
}
return { success: true, table: tableRef, deleted: true, verify: mutation.verify };
return {
success: true,
table: tableRef,
deleted: true,
verify: mutation.verify,
};
}
/**
* Set the plain-text content of cell `[row, col]` (0-based) in a table on the
@@ -734,7 +745,13 @@ export class DocmostClient {
if (!updated) {
throw new Error(`table_update_cell: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from get_outline, or a block id inside the table)`);
}
return { success: true, table: tableRef, row, col, verify: mutation.verify };
return {
success: true,
table: tableRef,
row,
col,
verify: mutation.verify,
};
}
/**
* Create a new page with title and content.
@@ -829,9 +846,11 @@ export class DocmostClient {
*/
async updatePage(pageId, content, title) {
await this.ensureAuthenticated();
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
// Write the BODY first, then the title (#159 split-brain). If the collab
// body write fails (e.g. a persist timeout), the title must be left
// UNTOUCHED so the page never ends up with a new title over its old body.
// A title write failing AFTER a successful body is rarer (REST is fast) and
// leaves correct content under a stale title — the lesser inconsistency.
let collabToken = "";
let mutation;
try {
@@ -850,6 +869,10 @@ export class DocmostClient {
}
throw new Error(`Failed to update page content: ${error.message}`);
}
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
return {
success: true,
modified: true,
@@ -969,7 +992,9 @@ export class DocmostClient {
if (!node || typeof node !== "object" || typeof node.type !== "string") {
throw new Error("invalid ProseMirror document: every node must be an object with a string `type`");
}
if ("text" in node && node.type === "text" && typeof node.text !== "string") {
if ("text" in node &&
node.type === "text" &&
typeof node.text !== "string") {
throw new Error("invalid ProseMirror document: a text node must have a string `text`");
}
if (node.marks !== undefined) {
@@ -977,7 +1002,9 @@ export class DocmostClient {
throw new Error("invalid ProseMirror document: `marks` must be an array");
}
for (const mark of node.marks) {
if (!mark || typeof mark !== "object" || typeof mark.type !== "string") {
if (!mark ||
typeof mark !== "object" ||
typeof mark.type !== "string") {
throw new Error("invalid ProseMirror document: every mark must be an object with a string `type`");
}
}
@@ -1036,11 +1063,14 @@ export class DocmostClient {
// the markdown link path (which TipTap sanitizes), raw JSON could otherwise
// inject javascript:/data: link hrefs or media srcs straight into the doc.
this.validateDocUrls(doc);
// Write the BODY first, then the title (#159 split-brain): a failed body
// write (e.g. persist timeout) must not leave a new title over the old body.
const collabToken = await this.getCollabTokenWithReauth();
const mutation = await replacePageContent(pageId, doc, collabToken, this.apiUrl);
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
const collabToken = await this.getCollabTokenWithReauth();
const mutation = await replacePageContent(pageId, doc, collabToken, this.apiUrl);
return {
success: true,
modified: true,
@@ -1057,9 +1087,7 @@ export class DocmostClient {
async exportPageMarkdown(pageId) {
await this.ensureAuthenticated();
const page = await this.getPageRaw(pageId);
const body = page.content
? convertProseMirrorToMarkdown(page.content)
: "";
const body = page.content ? convertProseMirrorToMarkdown(page.content) : "";
let comments = [];
try {
comments = await this.listComments(pageId);
@@ -1293,13 +1321,22 @@ export class DocmostClient {
replaced = 0;
const { doc: nd, replaced: r } = replaceNodeById(liveDoc, nodeId, target);
replaced = r;
if (replaced === 0)
return null; // no match -> skip the write entirely
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS: Docmost
// duplicates block ids on copy/paste (and copyPageContent writes them
// verbatim), so replacing "the node with id X" would silently clobber
// EVERY duplicate (#159). Refuse: skip the write and throw below so the
// model re-targets with a more specific anchor instead of corrupting the
// page. Only an unambiguous single match is written.
if (replaced !== 1)
return null;
return nd;
});
if (replaced === 0) {
throw new Error(`patch_node: no node with id "${nodeId}" found on page ${pageId}`);
}
if (replaced > 1) {
throw new Error(`patch_node: id "${nodeId}" is ambiguous — ${replaced} nodes on page ${pageId} share it (block ids are duplicated on copy/paste). Refusing to replace all of them; nothing was changed. Re-target with a more specific anchor.`);
}
return { success: true, replaced, nodeId, verify: mutation.verify };
}
/**
@@ -1355,7 +1392,7 @@ export class DocmostClient {
// markdown/emoji are tolerated only as a strip-and-retry fallback, so a
// miss usually means the text differs from what's on the page.
const hint = opts.anchorText
? ' anchorText must be the block\'s literal rendered plain text (no markdown wrappers or emoji); anchorNodeId from get_page_json is more reliable.'
? " anchorText must be the block's literal rendered plain text (no markdown wrappers or emoji); anchorNodeId from get_page_json is more reliable."
: "";
throw new Error(`insert_node: anchor not found (${anchorDesc}) on page ${pageId}.${hint}`);
}
@@ -1382,13 +1419,21 @@ export class DocmostClient {
deleted = 0;
const { doc: nd, deleted: d } = deleteNodeById(liveDoc, nodeId);
deleted = d;
if (deleted === 0)
return null; // no match -> skip the write entirely
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS (block
// ids are duplicated on copy/paste, #159): deleting "the node with id X"
// would silently remove EVERY duplicate. Refuse: skip the write and throw
// below so the model re-targets. Only an unambiguous single match is
// deleted.
if (deleted !== 1)
return null;
return nd;
});
if (deleted === 0) {
throw new Error(`delete_node: no node with id "${nodeId}" found on page ${pageId}`);
}
if (deleted > 1) {
throw new Error(`delete_node: id "${nodeId}" is ambiguous — ${deleted} nodes on page ${pageId} share it (block ids are duplicated on copy/paste). Refusing to delete all of them; nothing was changed. Re-target with a more specific anchor.`);
}
return { success: true, deleted, nodeId, verify: mutation.verify };
}
/** Build the public share URL for a page. */

View File

@@ -20,9 +20,9 @@ import {
mutatePageContent,
buildCollabWsUrl,
assertYjsEncodable,
applyDocToFragment,
MutationResult,
} from "./lib/collaboration.js";
import { docmostExtensions } from "./lib/docmost-schema.js";
import { footnoteWarningsField } from "./lib/footnote-analyze.js";
import { buildPageTree } from "./lib/tree.js";
import {
@@ -49,10 +49,7 @@ import {
} from "./lib/json-edit.js";
import { getCollabToken, performLogin } from "./lib/auth-utils.js";
import { diffDocs, summarizeChange } from "./lib/diff.js";
import {
applyAnchorInDoc,
canAnchorInDoc,
} from "./lib/comment-anchor.js";
import { applyAnchorInDoc, canAnchorInDoc } from "./lib/comment-anchor.js";
import {
blockText,
walk,
@@ -305,7 +302,9 @@ export class DocmostClient {
// getCollabToken wraps the AxiosError in a plain Error but attaches the
// HTTP status as `.status`, so detect an auth failure via either the raw
// AxiosError shape OR the attached status.
const axiosStatus = axios.isAxiosError(e) ? e.response?.status : undefined;
const axiosStatus = axios.isAxiosError(e)
? e.response?.status
: undefined;
const attachedStatus = (e as any)?.status;
const isAuthError =
axiosStatus === 401 ||
@@ -479,18 +478,14 @@ export class DocmostClient {
return;
}
const tempDoc = TiptapTransformer.toYdoc(
newDoc,
"default",
docmostExtensions,
);
const fragment = ydoc.getXmlFragment("default");
ydoc.transact(() => {
if (fragment.length > 0) {
fragment.delete(0, fragment.length);
}
Y.applyUpdate(ydoc, Y.encodeStateAsUpdate(tempDoc));
});
// Structural diff into the live fragment (issue #152), mirroring
// the main write path: preserves the Yjs ids of unchanged nodes so
// an open editor's cursor is not yanked to the end of the document.
// The previous destructive rewrite (delete-all + applyUpdate of a
// fresh Y.Doc) discarded every node id, so replaceImage — the only
// caller of this method — still reproduced the #152 cursor jump
// (#164). applyDocToFragment runs its own atomic `transact`.
applyDocToFragment(ydoc, newDoc);
} catch (e) {
finish(e instanceof Error ? e : new Error(String(e)));
return;
@@ -601,11 +596,7 @@ export class DocmostClient {
* sidebar requests and is bounded by that method's 10000-node cap (and skips
* soft-deleted pages server-side).
*/
async listPages(
spaceId?: string,
limit: number = 50,
tree: boolean = false,
) {
async listPages(spaceId?: string, limit: number = 50, tree: boolean = false) {
await this.ensureAuthenticated();
if (tree) {
@@ -884,7 +875,12 @@ export class DocmostClient {
`table_insert_row: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from get_outline, or a block id inside the table)`,
);
}
return { success: true, table: tableRef, inserted: true, verify: mutation.verify };
return {
success: true,
table: tableRef,
inserted: true,
verify: mutation.verify,
};
}
/**
@@ -903,7 +899,11 @@ export class DocmostClient {
this.apiUrl,
(liveDoc) => {
deleted = false;
const { doc: nd, deleted: del } = deleteTableRow(liveDoc, tableRef, index);
const { doc: nd, deleted: del } = deleteTableRow(
liveDoc,
tableRef,
index,
);
deleted = del;
if (!deleted) return null; // table not found -> skip the write entirely
return nd;
@@ -915,7 +915,12 @@ export class DocmostClient {
`table_delete_row: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from get_outline, or a block id inside the table)`,
);
}
return { success: true, table: tableRef, deleted: true, verify: mutation.verify };
return {
success: true,
table: tableRef,
deleted: true,
verify: mutation.verify,
};
}
/**
@@ -960,7 +965,13 @@ export class DocmostClient {
`table_update_cell: no table found for "${tableRef}" on page ${pageId} (use "#<index>" from get_outline, or a block id inside the table)`,
);
}
return { success: true, table: tableRef, row, col, verify: mutation.verify };
return {
success: true,
table: tableRef,
row,
col,
verify: mutation.verify,
};
}
/**
@@ -1034,8 +1045,7 @@ export class DocmostClient {
response = await axios.post(importUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization:
this.client.defaults.headers.common["Authorization"],
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
@@ -1069,10 +1079,11 @@ export class DocmostClient {
async updatePage(pageId: string, content: string, title?: string) {
await this.ensureAuthenticated();
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
// Write the BODY first, then the title (#159 split-brain). If the collab
// body write fails (e.g. a persist timeout), the title must be left
// UNTOUCHED so the page never ends up with a new title over its old body.
// A title write failing AFTER a successful body is rarer (REST is fast) and
// leaves correct content under a stale title — the lesser inconsistency.
let collabToken = "";
let mutation;
try {
@@ -1099,6 +1110,11 @@ export class DocmostClient {
throw new Error(`Failed to update page content: ${error.message}`);
}
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
return {
success: true,
modified: true,
@@ -1173,9 +1189,7 @@ export class DocmostClient {
for (const mark of node.marks) {
if (mark && mark.type === "link" && mark.attrs) {
if (!this.isSafeUrl(mark.attrs.href, "link")) {
throw new Error(
`unsafe link href rejected: "${mark.attrs.href}"`,
);
throw new Error(`unsafe link href rejected: "${mark.attrs.href}"`);
}
}
}
@@ -1234,7 +1248,11 @@ export class DocmostClient {
"invalid ProseMirror document: every node must be an object with a string `type`",
);
}
if ("text" in node && node.type === "text" && typeof node.text !== "string") {
if (
"text" in node &&
node.type === "text" &&
typeof node.text !== "string"
) {
throw new Error(
"invalid ProseMirror document: a text node must have a string `text`",
);
@@ -1246,7 +1264,11 @@ export class DocmostClient {
);
}
for (const mark of node.marks) {
if (!mark || typeof mark !== "object" || typeof mark.type !== "string") {
if (
!mark ||
typeof mark !== "object" ||
typeof mark.type !== "string"
) {
throw new Error(
"invalid ProseMirror document: every mark must be an object with a string `type`",
);
@@ -1321,10 +1343,8 @@ export class DocmostClient {
// inject javascript:/data: link hrefs or media srcs straight into the doc.
this.validateDocUrls(doc);
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
// Write the BODY first, then the title (#159 split-brain): a failed body
// write (e.g. persist timeout) must not leave a new title over the old body.
const collabToken = await this.getCollabTokenWithReauth();
const mutation = await replacePageContent(
pageId,
@@ -1333,6 +1353,11 @@ export class DocmostClient {
this.apiUrl,
);
// Body persisted successfully — now it is safe to set the title.
if (title) {
await this.client.post("/pages/update", { pageId, title });
}
return {
success: true,
modified: true,
@@ -1350,9 +1375,7 @@ export class DocmostClient {
async exportPageMarkdown(pageId: string): Promise<string> {
await this.ensureAuthenticated();
const page = await this.getPageRaw(pageId);
const body = page.content
? convertProseMirrorToMarkdown(page.content)
: "";
const body = page.content ? convertProseMirrorToMarkdown(page.content) : "";
let comments: any[] = [];
try {
comments = await this.listComments(pageId);
@@ -1566,9 +1589,10 @@ export class DocmostClient {
pageId,
applied: results,
failed,
message: (failed?.length ?? 0)
? `Applied ${results?.length ?? 0} edit(s); ${failed!.length} failed (see failed[]). Node ids and formatting preserved.`
: "Text edits applied (node ids and formatting preserved).",
message:
(failed?.length ?? 0)
? `Applied ${results?.length ?? 0} edit(s); ${failed!.length} failed (see failed[]). Node ids and formatting preserved.`
: "Text edits applied (node ids and formatting preserved).",
verify: mutation.verify,
};
@@ -1627,9 +1651,19 @@ export class DocmostClient {
this.apiUrl,
(liveDoc) => {
replaced = 0;
const { doc: nd, replaced: r } = replaceNodeById(liveDoc, nodeId, target);
const { doc: nd, replaced: r } = replaceNodeById(
liveDoc,
nodeId,
target,
);
replaced = r;
if (replaced === 0) return null; // no match -> skip the write entirely
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS: Docmost
// duplicates block ids on copy/paste (and copyPageContent writes them
// verbatim), so replacing "the node with id X" would silently clobber
// EVERY duplicate (#159). Refuse: skip the write and throw below so the
// model re-targets with a more specific anchor instead of corrupting the
// page. Only an unambiguous single match is written.
if (replaced !== 1) return null;
return nd;
},
);
@@ -1639,6 +1673,11 @@ export class DocmostClient {
`patch_node: no node with id "${nodeId}" found on page ${pageId}`,
);
}
if (replaced > 1) {
throw new Error(
`patch_node: id "${nodeId}" is ambiguous — ${replaced} nodes on page ${pageId} share it (block ids are duplicated on copy/paste). Refusing to replace all of them; nothing was changed. Re-target with a more specific anchor.`,
);
}
return { success: true, replaced, nodeId, verify: mutation.verify };
}
@@ -1707,7 +1746,11 @@ export class DocmostClient {
this.apiUrl,
(liveDoc) => {
inserted = false;
const { doc: nd, inserted: ins } = insertNodeRelative(liveDoc, node, opts);
const { doc: nd, inserted: ins } = insertNodeRelative(
liveDoc,
node,
opts,
);
inserted = ins;
if (!inserted) return null; // anchor not found -> skip the write entirely
return nd;
@@ -1722,7 +1765,7 @@ export class DocmostClient {
// markdown/emoji are tolerated only as a strip-and-retry fallback, so a
// miss usually means the text differs from what's on the page.
const hint = opts.anchorText
? ' anchorText must be the block\'s literal rendered plain text (no markdown wrappers or emoji); anchorNodeId from get_page_json is more reliable.'
? " anchorText must be the block's literal rendered plain text (no markdown wrappers or emoji); anchorNodeId from get_page_json is more reliable."
: "";
throw new Error(
`insert_node: anchor not found (${anchorDesc}) on page ${pageId}.${hint}`,
@@ -1759,7 +1802,12 @@ export class DocmostClient {
deleted = 0;
const { doc: nd, deleted: d } = deleteNodeById(liveDoc, nodeId);
deleted = d;
if (deleted === 0) return null; // no match -> skip the write entirely
// 0 matches -> skip the write. >1 matches -> the id is AMBIGUOUS (block
// ids are duplicated on copy/paste, #159): deleting "the node with id X"
// would silently remove EVERY duplicate. Refuse: skip the write and throw
// below so the model re-targets. Only an unambiguous single match is
// deleted.
if (deleted !== 1) return null;
return nd;
},
);
@@ -1769,6 +1817,11 @@ export class DocmostClient {
`delete_node: no node with id "${nodeId}" found on page ${pageId}`,
);
}
if (deleted > 1) {
throw new Error(
`delete_node: id "${nodeId}" is ambiguous — ${deleted} nodes on page ${pageId} share it (block ids are duplicated on copy/paste). Refusing to delete all of them; nothing was changed. Re-target with a more specific anchor.`,
);
}
return { success: true, deleted, nodeId, verify: mutation.verify };
}
@@ -2140,7 +2193,11 @@ export class DocmostClient {
* subtree): pages updated after `since` are scanned and their comments
* filtered by createdAt > since.
*/
async checkNewComments(spaceId: string, since: string, parentPageId?: string) {
async checkNewComments(
spaceId: string,
since: string,
parentPageId?: string,
) {
await this.ensureAuthenticated();
const sinceDate = new Date(since);
@@ -2440,8 +2497,7 @@ export class DocmostClient {
response = await axios.post(uploadUrl, form2, {
headers: {
...form2.getHeaders(),
Authorization:
this.client.defaults.headers.common["Authorization"],
Authorization: this.client.defaults.headers.common["Authorization"],
},
timeout: 60000,
});
@@ -2528,76 +2584,76 @@ export class DocmostClient {
collabToken,
this.apiUrl,
(liveDoc) => {
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
const doc =
liveDoc && liveDoc.type === "doc"
? liveDoc
: { type: "doc", content: [] };
if (!Array.isArray(doc.content)) doc.content = [];
if (opts.replaceText) {
// Ambiguity guard (mirrors editPageText): count matching top-level
// blocks first, so a non-unique fragment cannot silently replace the
// wrong block (e.g. text that also appears inside a callout/table).
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.replaceText!),
);
if (matches.length === 0) {
throw new Error(`replaceText not found: "${opts.replaceText}"`);
}
if (matches.length > 1) {
throw new Error(
`replaceText "${opts.replaceText}" matches ${matches.length} blocks; use a longer unique fragment`,
if (opts.replaceText) {
// Ambiguity guard (mirrors editPageText): count matching top-level
// blocks first, so a non-unique fragment cannot silently replace the
// wrong block (e.g. text that also appears inside a callout/table).
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.replaceText!),
);
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.replaceText!),
);
// Data-loss guard: replaceText swaps the WHOLE top-level block, so if
// the fragment only appears nested inside a container (table, callout,
// list, blockquote) the entire structure would be destroyed. Refuse
// when the matched block is a container rather than a leaf
// paragraph/heading and point the caller at a safer tool.
const CONTAINER_TYPES = new Set([
"table",
"callout",
"bulletList",
"orderedList",
"taskList",
"blockquote",
]);
const matchedBlock = doc.content[idx];
if (matchedBlock && CONTAINER_TYPES.has(matchedBlock.type)) {
throw new Error(
`replaceText matched a ${matchedBlock.type} container block; replacing it would destroy the whole structure. ` +
`Use afterText to insert near it, or update_page_json for surgical edits.`,
if (matches.length === 0) {
throw new Error(`replaceText not found: "${opts.replaceText}"`);
}
if (matches.length > 1) {
throw new Error(
`replaceText "${opts.replaceText}" matches ${matches.length} blocks; use a longer unique fragment`,
);
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.replaceText!),
);
}
doc.content.splice(idx, 1, node);
placement = "replaced";
} else if (opts.afterText) {
// Ambiguity guard (mirrors editPageText): refuse a non-unique fragment.
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.afterText!),
);
if (matches.length === 0) {
throw new Error(`afterText not found: "${opts.afterText}"`);
}
if (matches.length > 1) {
throw new Error(
`afterText "${opts.afterText}" matches ${matches.length} blocks; use a longer unique fragment`,
// Data-loss guard: replaceText swaps the WHOLE top-level block, so if
// the fragment only appears nested inside a container (table, callout,
// list, blockquote) the entire structure would be destroyed. Refuse
// when the matched block is a container rather than a leaf
// paragraph/heading and point the caller at a safer tool.
const CONTAINER_TYPES = new Set([
"table",
"callout",
"bulletList",
"orderedList",
"taskList",
"blockquote",
]);
const matchedBlock = doc.content[idx];
if (matchedBlock && CONTAINER_TYPES.has(matchedBlock.type)) {
throw new Error(
`replaceText matched a ${matchedBlock.type} container block; replacing it would destroy the whole structure. ` +
`Use afterText to insert near it, or update_page_json for surgical edits.`,
);
}
doc.content.splice(idx, 1, node);
placement = "replaced";
} else if (opts.afterText) {
// Ambiguity guard (mirrors editPageText): refuse a non-unique fragment.
const matches = doc.content.filter((b: any) =>
blockText(b).includes(opts.afterText!),
);
if (matches.length === 0) {
throw new Error(`afterText not found: "${opts.afterText}"`);
}
if (matches.length > 1) {
throw new Error(
`afterText "${opts.afterText}" matches ${matches.length} blocks; use a longer unique fragment`,
);
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.afterText!),
);
doc.content.splice(idx + 1, 0, node);
placement = "after";
} else {
doc.content.push(node);
placement = "appended";
}
const idx = doc.content.findIndex((b: any) =>
blockText(b).includes(opts.afterText!),
);
doc.content.splice(idx + 1, 0, node);
placement = "after";
} else {
doc.content.push(node);
placement = "appended";
}
return doc;
return doc;
},
);
@@ -2854,8 +2910,7 @@ export class DocmostClient {
async diffPageVersions(pageId: string, from?: string, to?: string) {
await this.ensureAuthenticated();
const isCurrent = (v?: string) =>
v == null || v === "" || v === "current";
const isCurrent = (v?: string) => v == null || v === "" || v === "current";
const resolveSide = async (
v?: string,
@@ -2976,7 +3031,9 @@ export class DocmostClient {
throw new Error(`transform did not compile: ${e?.message ?? e}`);
}
if (typeof fn !== "function") {
throw new Error("transform must evaluate to a function (doc, ctx) => doc");
throw new Error(
"transform must evaluate to a function (doc, ctx) => doc",
);
}
const result = vm.runInNewContext(
"f(d, c)",

View File

@@ -162,3 +162,70 @@ test("assertYjsEncodable rejects an un-hydratable doc at preview time (fromJSON
/Failed to encode document to Yjs/,
);
});
// Issue #164: `replaceImage` went through `mutateLiveContentUnlocked`, which
// (unlike the main write path fixed in #152) still deleted the whole fragment
// and re-applied a fresh Y.Doc — discarding every node id, so an open editor's
// cursor jumped to the document end on an image swap. That method now uses the
// same `applyDocToFragment`, so a sibling paragraph's cursor anchor survives an
// image `src`/`attachmentId` replacement. These exercise that routine on the
// image shapes `replaceImage` produces (top-level and nested in a callout).
const image = (attachmentId, src) => ({
type: "image",
attrs: { attachmentId, src, width: "640", align: "center" },
});
test("replacing a top-level image keeps a sibling paragraph's cursor anchor (#164)", () => {
const ydoc = new Y.Doc();
applyDocToFragment(
ydoc,
doc(para("Caption above"), image("att-old", "/files/old.png")),
);
// The user's cursor sits in the (unchanged) caption paragraph.
const relPos = Y.createRelativePositionFromTypeIndex(paragraphText(ydoc, 0), 7);
// Agent repoints the image to a freshly uploaded attachment (new id + src).
applyDocToFragment(
ydoc,
doc(para("Caption above"), image("att-new", "/files/new.png")),
);
const abs = Y.createAbsolutePositionFromRelativePosition(relPos, ydoc);
assert.notEqual(abs, null, "the caption cursor anchor must still resolve");
assert.equal(abs.index, 7, "the cursor must stay at the same offset");
// The swap actually landed: the image now carries the new attachment id/src.
const img = ydoc.getXmlFragment("default").get(1);
assert.equal(img.nodeName, "image");
assert.equal(img.getAttribute("attachmentId"), "att-new");
assert.equal(img.getAttribute("src"), "/files/new.png");
});
test("replacing an image nested in a callout keeps an outer paragraph's anchor (#164)", () => {
const callout = (attachmentId, src) => ({
type: "callout",
attrs: { type: "info" },
content: [image(attachmentId, src)],
});
const ydoc = new Y.Doc();
applyDocToFragment(
ydoc,
doc(para("Intro paragraph"), callout("att-old", "/files/old.png")),
);
const relPos = Y.createRelativePositionFromTypeIndex(paragraphText(ydoc, 0), 5);
applyDocToFragment(
ydoc,
doc(para("Intro paragraph"), callout("att-new", "/files/new.png")),
);
const abs = Y.createAbsolutePositionFromRelativePosition(relPos, ydoc);
assert.notEqual(abs, null, "the outer paragraph anchor must still resolve");
assert.equal(abs.index, 5, "the cursor must stay at the same offset");
// The nested image was repointed.
const calloutEl = ydoc.getXmlFragment("default").get(1);
const img = calloutEl.get(0);
assert.equal(img.getAttribute("attachmentId"), "att-new");
});