Compare commits

..

3 Commits

Author SHA1 Message Date
agent_coder 07f82d8fee Merge remote-tracking branch 'gitea/develop' into fix/522-internal-links 2026-07-12 03:47:45 +03:00
agent_coder 201b0c560d test(#522): make the internal-link subset invariant a real cross-package drift guard
Reviewer follow-up on the #522 internal-link import fix. The fix itself is
confirmed correct; these three changes harden its load-bearing subset invariant
and fix a stale doc.

Finding 1 (drift guard): the in-package test only compared isInternalPagePath
against a HAND-COPIED server regex, so a later narrowing of the real server
INTERNAL_LINK_REGEX would leave the test green while the client silently became
WIDER than the server. Add apps/server/.../export/internal-link-parity.spec.ts:
the top layer already depends on @docmost/prosemirror-markdown and exports
isInternalPagePath, so this spec imports the LIVE server INTERNAL_LINK_REGEX AND
the LIVE isInternalPagePath and asserts subset over an accept-corpus — a server
narrowing now reddens CI. The in-package manual copy is demoted from its
"drift guard" role (comment updated to say so; it stays as local documentation).

Finding 2 (charset non-vacuity): the REJECT list was purely structural, so the
mutation widening the slug charset [a-zA-Z0-9-] -> [a-zA-Z0-9-.] survived the
whole suite. Add REJECT cases whose ONLY defect is a forbidden slug character
(abc.def / abc_def / abc%20 / abc~x); assert both isInternalPagePath and the
server regex reject them. The mutation now reddens the new reject test.

Finding 3 (stale JSDoc): canonicalize.ts KNOWN_DEFAULTS module blurb claimed
every entry was read from docmost-schema and import-materialized. The new
link.internal default breaks both claims (source is editor-ext/src/lib/link.ts;
external links leave internal absent/null, never false). Add a bullet documenting
the one editor-sourced, non-materialized default (false ≡ absent/null).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 03:40:18 +03:00
agent_coder 7ea030cc9e fix(prosemirror-markdown): помечать внутренние ссылки на страницы как internal при импорте markdown
При импорте markdown ссылка вида `[t](/s/<space>/p/<slug>)` сохранялась как
внешняя (link-mark получал `internal:null`, `target:"_blank"`,
`rel:"noopener noreferrer nofollow"`): открывалась новой вкладкой, без
hover-превью и без участия в backlinks. Единственный путь к нативной внутренней
ссылке был ручной JSON-патч.

Чиню в общем конвертере пакета, через который проходят ВСЕ markdown-пути (MCP
updatePageMarkdown/importPageMarkdown, patch/insertNode, тела комментариев,
серверный REST create/update, single/zip-импорт, ai-chat, git-sync pull,
вставка markdown в редактор) — все они получают исправление разом.

- Новый модуль `internal-links.ts`: чистый `isInternalPagePath(href)` —
  якорный `^/s/<space>/p/<slug>/?$`, СТРОГОЕ подмножество серверного
  `INTERNAL_LINK_REGEX` (apps/server/.../export/utils.ts), поэтому всё
  помеченное гарантированно бэклинкуется и переписывается при экспорте.
  Fail-toward-external: любая неоднозначность (scheme/host, `#`, `?`,
  бесспейсовый `/p/<slug>`, лишний сегмент, не-строка) остаётся внешней.
- `markInternalLinks(doc)`: пост-обход готового ProseMirror-документа,
  помечает КАЖДУЮ text-ноду, покрытую внутренней ссылкой (включая случай
  вложенных bold/italic внутри ссылки) → `{internal:true, target:null,
  rel:null}`. Чистая и идемпотентная.
- Разводка в `markdownToProseMirrorSync` после stripEmptyParagraphs.
- §11 (обязательная сопутствующая правка): `internal:false` добавлен в
  `KNOWN_DEFAULTS.link` канонизатора — редакторные внешние ссылки хранят
  `internal:false`, теперь он канонизируется как absent/null/external, а
  load-bearing `internal:true` (не дефолт) переживает канонизацию.

Тесты: accept/reject-пиннинг матчера (edge: trailing slash, `#`, `?`,
`/p/x` без спейса, `https://h/p/x`, `/api/...`, uppercase, empty-space),
subset-инвариант против серверного регэкспа, мультинодовая ссылка со вложенными
марками, идемпотентность, полный конвертер (internal помечен / external нетронут /
бэклинк-извлекаемость), комментарий-тело через общий путь, canonicalize §11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 00:36:34 +03:00
142 changed files with 1618 additions and 7367 deletions
-63
View File
@@ -62,38 +62,6 @@ jobs:
needs: [test, e2e-server, e2e-mcp, build]
runs-on: ubuntu-latest
timeout-minutes: 30
# Image boot-smoke (issue #476): every other job tests code from the working
# tree, but the :develop IMAGE that watchtower pulls was never actually
# started anywhere (incident classes #353/#452/#361-boot: startup-migrator
# crash-loop, runtime module missing from the image, wrong static-asset
# headers). The services below back a smoke boot of the exact image right
# before it is pushed; a smoke failure blocks the push.
services:
postgres:
# via mirror.gcr.io (Docker Hub pull-through cache; avoids Hub anonymous
# pull rate-limit that randomly fails on shared GitHub runner IPs).
image: mirror.gcr.io/pgvector/pgvector:pg18
env:
POSTGRES_DB: docmost
POSTGRES_USER: docmost
POSTGRES_PASSWORD: docmost
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U docmost"
--health-interval 5s
--health-timeout 5s
--health-retries 20
redis:
# via mirror.gcr.io (see postgres note above).
image: mirror.gcr.io/library/redis:7
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 5s
--health-timeout 5s
--health-retries 20
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -114,37 +82,6 @@ jobs:
id: version
run: echo "value=$(git describe --tags --always)" >> "$GITHUB_OUTPUT"
# Load the image into the local docker daemon so it can be booted (the
# push step below exports straight to the registry and leaves nothing
# runnable locally). CONVENTION: build-args here must stay TEXTUALLY
# IDENTICAL to the push step's build-args — same cache scope + same args
# means the layers are reused and the image we smoke IS the image we push.
- name: Build image for smoke (load, no push)
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64
build-args: |
APP_VERSION=${{ steps.version.outputs.value }}
AI_AGENT_ROLES_CATALOG_URL=https://raw.githubusercontent.com/vvzvlad/gitmost/develop/agent-roles-catalog
load: true
push: false
tags: gitmost:smoke
cache-from: type=gha,scope=develop-amd64
# Boot-smoke the exact image against the job services (see the comment on
# `services:` above): health (startup migrator), auth/setup, client dist
# served, immutable + brotli asset headers. Fails the job (and therefore
# the push) on any miss.
- name: Smoke the built image
run: bash scripts/ci/image-smoke.sh gitmost:smoke
# The smoke script leaves the container running on failure precisely so
# the boot error (migration mismatch, stack trace) is diagnosable here.
- name: Dump smoke container log on failure
if: failure()
run: docker logs gitmost-smoke 2>&1 | tail -200 || true
- name: Build and push develop image
uses: docker/build-push-action@v6
with:
+10 -41
View File
@@ -124,17 +124,9 @@ jobs:
exit "$FAILED"
# A GENUINE counterexample: fast-check printed a shrunk minimal case and its
# reproducing seed into property-output.txt. File a dedup-guarded issue.
#
# Dedup is keyed on a HASH of the SHRUNK COUNTEREXAMPLE (the minimal failing
# input), NOT on the issue title prefix. Keying on the prefix would let a
# single open issue swallow every OTHER counterexample (a different bug B whose
# title shares the prefix would be treated as a duplicate and stay silent until
# the first issue is closed). Hashing the shrunk example instead means two
# DIFFERENT counterexamples get two DIFFERENT issues, while a re-find of the
# SAME counterexample still dedupes onto the existing one. The infra-failure
# step (below) still keys on its own distinct title, so it can never poison
# this dedup either.
# reproducing seed into property-output.txt. File a dedup-guarded issue whose
# title prefix is UNIQUE to counterexamples, so an infra failure (handled by
# the next step under a different title) can never poison this dedup.
- name: File counterexample issue
# always() is REQUIRED: the fuzz step exits nonzero on a failing shard,
# so a bare `if:` (implicitly success() && ...) would skip this step
@@ -154,48 +146,25 @@ jobs:
echo "No fast-check counterexample signature — infra failure, handled by the next step."
exit 0
fi
# Extract the SHRUNK counterexample block: the "Counterexample:" line(s)
# up to (but excluding) the "Shrunk N time(s)" / "Got error" line. This is
# the minimal failing INPUT and is STABLE across the different seeds/paths
# that reach the same bug — unlike the seed, path, or shrink count (which
# precede/follow this block and vary run-to-run) and unlike the whole
# output (which embeds those varying parts). Hashing THIS is what makes the
# dedup identity the bug itself rather than an incidental run detail.
CE_TEXT=$(awk '/Counterexample:/{c=1} /Shrunk [0-9]+ time|Got error/{c=0} c{print}' property-output.txt)
if [ -z "$CE_TEXT" ]; then
# No parseable shrunk block (unexpected — the signature check above
# already confirmed fast-check output). Fall back to the reproducing
# seed so we still emit a stable identity instead of silently deduping.
CE_TEXT="seed:${FAIL_SEED}"
fi
# Stable short id: first 12 hex chars of sha256 over the counterexample.
CE_HASH=$(printf '%s' "$CE_TEXT" | sha256sum | cut -c1-12)
# Machine-readable marker embedded in the issue body; the open-issue search
# below matches on it (and on the hash in the title) so identity travels
# with the issue regardless of any human title edits.
CE_MARKER="<!-- counterexample-hash: ${CE_HASH} -->"
export CE_HASH CE_MARKER
TITLE="${TITLE_PREFIX} [${CE_HASH}] (seed=${FAIL_SEED})"
TITLE="${TITLE_PREFIX} (seed=${FAIL_SEED})"
# Dedup on the counterexample hash: skip only if an OPEN issue already
# carries this exact hash (in its title or its body marker). A different
# counterexample has a different hash and is NOT deduped. A failure of this
# check must NOT block creation.
# Best-effort dedup: skip if an open issue with the counterexample title
# prefix already exists. A failure of this check must NOT block creation.
EXISTING=""
if EXISTING=$(curl -sS \
-H "Authorization: token ${GITHUB_TOKEN}" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/issues?state=open&limit=100"); then
if printf '%s' "$EXISTING" \
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const h=process.env.CE_HASH,m=process.env.CE_MARKER;process.exit(a.some(i=>(typeof i.title==="string"&&i.title.includes(h))||(typeof i.body==="string"&&i.body.includes(m)))?0:1)})'; then
echo "An open issue for counterexample ${CE_HASH} already exists — skipping creation."
| node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{let a;try{a=JSON.parse(s)}catch{process.exit(1)}if(!Array.isArray(a))process.exit(1);const p=process.env.TITLE_PREFIX;process.exit(a.some(i=>typeof i.title==="string"&&i.title.startsWith(p))?0:1)})'; then
echo "An open '${TITLE_PREFIX}' issue already exists — skipping creation."
exit 0
fi
fi
# Build the JSON body with the test output SAFELY escaped (never hand-
# interpolate the counterexample into JSON).
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed with a fast-check counterexample.\n\n- counterexample hash: `%s`\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nReproduce locally:\n\n```\nPROPERTY_SEED=%s PROPERTY_NUM_RUNS=%s pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/\n```\n\nfast-check shrinks the failure to a minimal counterexample. Commit it as a permanent fixture under `packages/prosemirror-markdown/test/fixtures/counterexamples/` + a case in `counterexamples.test.ts`, then FIX the converter (do not weaken a property). See `packages/prosemirror-markdown/README.md`.\n\nTail of the test output (contains the shrunk counterexample):\n\n```\n%s\n```\n\n%s\n' \
"$CE_HASH" "$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$FAIL_SEED" "$NUM_RUNS" "$(tail -n 120 property-output.txt)" "$CE_MARKER")
BODY_TEXT=$(printf 'A nightly property fuzz SHARD failed with a fast-check counterexample.\n\n- failing shard seed: `%s`\n- NUM_RUNS (per shard): `%s`\n- run: %s\n\nReproduce locally:\n\n```\nPROPERTY_SEED=%s PROPERTY_NUM_RUNS=%s pnpm --filter @docmost/prosemirror-markdown exec vitest run test/generative/\n```\n\nfast-check shrinks the failure to a minimal counterexample. Commit it as a permanent fixture under `packages/prosemirror-markdown/test/fixtures/counterexamples/` + a case in `counterexamples.test.ts`, then FIX the converter (do not weaken a property). See `packages/prosemirror-markdown/README.md`.\n\nTail of the test output (contains the shrunk counterexample):\n\n```\n%s\n```\n' \
"$FAIL_SEED" "$NUM_RUNS" "$RUN_URL" "$FAIL_SEED" "$NUM_RUNS" "$(tail -n 120 property-output.txt)")
jq -n --arg title "$TITLE" --arg body "$BODY_TEXT" \
'{title: $title, body: $body}' > payload.json
+13 -41
View File
@@ -25,65 +25,37 @@ jobs:
# filename sorts BEFORE migrations already applied on the target branch (and
# thus in prod). The Kysely startup migrator rejects that as "corrupted
# migrations" and crash-loops the app on boot (incident #361). This gate fails
# the PR so the migration is renamed to a current timestamp before merge.
# Runs for pull_request (diff against the base branch) AND for push (#476
# retrospective: a DIRECT push to develop used to bypass this PR-only gate
# entirely — now the push is diffed against its `before` SHA; workflow_call
# from develop.yml inherits the caller's push event). workflow_dispatch has
# nothing to diff against and still skips the job.
# the PR so the migration is renamed to a current timestamp before merge. Only
# runs for pull_request events (needs a base branch to diff against).
migration-order:
if: github.event_name == 'pull_request' || github.event_name == 'push'
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Checkout (full history for the base diff)
- name: Checkout (full history for the base-branch diff)
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Added migrations must sort after the newest on the base
- name: Added migrations must sort after the newest on the base branch
env:
TARGET_BRANCH: ${{ github.base_ref }}
BEFORE_SHA: ${{ github.event.before }}
run: |
set -euo pipefail
MIG_DIR="apps/server/src/database/migrations"
if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then
# checkout above already did fetch-depth:0 (full history). Fetch the base
# WITHOUT --depth (a shallow graft would truncate the base history and
# break the merge-base when the base has moved ahead of the PR merge —
# exactly the long-branch-vs-moving-base case this gate guards, #361).
git fetch --no-tags origin "$TARGET_BRANCH"
BASE="origin/${TARGET_BRANCH}"
else
# push event: compare against the pre-push tip of the branch.
if [ "$BEFORE_SHA" = "0000000000000000000000000000000000000000" ]; then
echo "::notice::branch creation push — nothing to compare"
exit 0
fi
if ! git cat-file -e "${BEFORE_SHA}^{commit}" 2>/dev/null; then
# The before-SHA is not in the clone (a force-push rewrote history).
# One recovery attempt — refresh every remote head (cheap: the
# checkout is already fetch-depth:0); a fetch failure aborts via
# `set -e`, which is fail-closed too.
git fetch --no-tags origin '+refs/heads/*:refs/remotes/origin/*'
fi
if ! git cat-file -e "${BEFORE_SHA}^{commit}" 2>/dev/null; then
# FAIL-CLOSED: without the before-SHA there is no base to prove the
# ordering against, and a gate whose job is to BLOCK must not guess.
echo "::error::force-push detected — verify migration order manually, then re-run via workflow_dispatch"
exit 1
fi
BASE="$BEFORE_SHA"
fi
newest_on_target=$(git ls-tree -r --name-only "$BASE" "$MIG_DIR" | sort | tail -1)
# checkout above already did fetch-depth:0 (full history). Fetch the base
# WITHOUT --depth (a shallow graft would truncate the base history and
# break the merge-base when the base has moved ahead of the PR merge —
# exactly the long-branch-vs-moving-base case this gate guards, #361).
git fetch --no-tags origin "$TARGET_BRANCH"
newest_on_target=$(git ls-tree -r --name-only "origin/${TARGET_BRANCH}" "$MIG_DIR" | sort | tail -1)
# NO `|| true`: a diff failure (e.g. an unresolved merge-base) must fail
# the job CLOSED — a gate whose job is to BLOCK must never pass on error.
# `set -e` above already aborts on a non-zero diff exit.
added=$(git diff --diff-filter=A --name-only "${BASE}...HEAD" -- "$MIG_DIR")
added=$(git diff --diff-filter=A --name-only "origin/${TARGET_BRANCH}...HEAD" -- "$MIG_DIR")
bad=0
for f in $added; do
if [[ "$f" < "$newest_on_target" || "$f" == "$newest_on_target" ]]; then
echo "::error::Migration $f sorts at or before the newest on the base ($newest_on_target) — rename it with a CURRENT timestamp before merge (do not change its contents). See incident #361."
echo "::error::Migration $f sorts at or before the newest on ${TARGET_BRANCH} ($newest_on_target) — rename it with a CURRENT timestamp before merge (do not change its contents). See incident #361."
bad=1
fi
done
-4
View File
@@ -29,10 +29,6 @@ packages/mcp/build/
# is a build artifact like build/ — never committed, always fresh.
packages/mcp/src/registry-stamp.generated.ts
# token-estimate compiled output (#490; built in CI/Docker via `pnpm build` /
# the server `pretest`, never committed, so src/ and prod can never diverge).
packages/token-estimate/dist/
# Logs
logs
*.log
-2
View File
@@ -471,8 +471,6 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch carries TWO independent server fixes, each with its own tripwire test: (1) it disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`); (2) it fixes `writeToServerResponse`'s drain-hang — the loop awaited only `"drain"` under backpressure, so a mid-write client disconnect parked the pipe forever and leaked the reader/buffers until restart; it now races `"drain"` against `"close"`/`"error"`, cancels the reader on disconnect, and swallows the fire-and-forget read rejection (#486; tripwire: `apps/server/src/integrations/ai/ai-sdk-drain-hang.patch.spec.ts`). Both tripwires assert BOTH installed dist builds carry their patch marker. The patch MUST be re-created via `pnpm patch` when bumping `ai`.
- **Upstream tracking (report the analysis upstream, don't just carry it):** both `ai` fixes and the hocuspocus one are candidates for upstreaming so we can eventually drop the local patch — the analysis is already written up in each patch's `PATCH(...)` header comments. File (a) an upstream **issue** on `vercel/ai` for the O(n²) cumulative `partialOutput` accumulation (heap OOM), (b) an upstream **issue** on `vercel/ai` for the `writeToServerResponse` drain-hang, and (c) an upstream **PR** on `@hocuspocus/server` for the connect-vs-unload race (local marker `PATCH(gitmost #401)` in `patches/@hocuspocus__server@3.4.4.patch`). Do NOT edit the patch files to add links — the patch bytes feed `patch_hash` in `pnpm-lock.yaml` (`ai@6.0.134``e8c599b3…`), so any content change there desyncs the lockfile pin and breaks `pnpm install`; keep upstream references here instead.
- **`ai` version is split across the monorepo and MUST be aligned deliberately, NOT casually:** the server pins `ai@6.0.134` (patched, exact — the `patchedDependencies` key forces that version), while the client declares `ai@6.0.207` (unpatched — the server-side `writeToServerResponse`/`partialOutput` fixes are dead code in the browser, so the mismatch is currently benign but is real drift). Alignment is a **planned, install-gated step**, never a bare `package.json` edit: (1) choose the target version; (2) re-create ALL THREE patch hunks (partialOutput publish-each, the `DefaultStreamTextResult` lazy-`output` wiring, and the drain-hang race) against the target dist via `pnpm patch` — the line offsets shift between versions, so the current patch WILL fail to apply as-is; (3) run a full `pnpm install` so the lockfile + new `patch_hash` regenerate together; (4) confirm both tripwire specs still find their markers. `pnpm install` FAILS HARD on an unapplied patch — that failure is the guardrail, so treat the port as a deliberate plan rather than discovering it as a deploy-time surprise.
- **The MCP tool inventory in `SERVER_INSTRUCTIONS` is GENERATED from the registry** (`packages/mcp/src/server-instructions.ts`: `buildToolInventory()` over `SHARED_TOOL_SPECS`) and spliced into the hand-written routing prose (`ROUTING_PROSE`). So adding/renaming/removing a **shared** spec in `packages/mcp/src/tool-specs.ts` auto-updates the `<tool_inventory>` — no manual `SERVER_INSTRUCTIONS` edit needed. Only an **inline** MCP-only tool (those registered via `server.registerTool(...)` in `index.ts`, not through the registry) needs a one-line entry in `INLINE_MCP_INVENTORY`. Enforced by `packages/mcp/test/unit/tool-inventory.test.mjs`, which fails when a registered tool is missing from the generated inventory (there is no `EXCEPTIONS` opt-out anymore — every tool must appear). Update `ROUTING_PROSE` when a tool's *intent guidance* (when-to-use) changes. `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
## CI / release
-64
View File
@@ -311,14 +311,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
longer controls whether a turn is a run — it now governs **only** the
browser-disconnect semantics (ON = detached/survives a disconnect; OFF = a
disconnect stops the run). (#487)
- **Vendor `ai` patch: upstream-tracking + version-alignment plan documented.**
The two local `ai@6.0.134` fixes (O(n²) `partialOutput` heap-OOM; the
`writeToServerResponse` drain-hang) and the hocuspocus connect-vs-unload race
now have explicit upstream-reporting and `ai`-version-alignment steps recorded
in `AGENTS.md` (client `ai@6.0.207` vs server `ai@6.0.134`-patched drift). The
patch bytes are unchanged — they feed the lockfile `patch_hash`, so the
alignment is called out as an install-gated plan rather than a bare version
bump. No runtime change.
- **Client markdown paste/copy and AI-chat rendering now go through the canonical
converter.** Pasting markdown into the editor, "Copy as markdown", the AI title
generator, and the AI-chat markdown renderer all now use
@@ -351,40 +343,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **MCP write tools no longer report a false failure that provokes a duplicate
write.** `drawioCreate` used to throw when the diagram landed as a NESTED block
(anchored inside a callout or table cell) because there is no `#<index>` handle
for it — but the diagram was already written, so a retry-prone agent re-created
it and produced a duplicate. It now returns success with `nodeId: null` plus a
warning that explains the write landed and how to re-read it (via
`getOutline` / `getPageJson` by `attachmentId`). Separately, when the live
collaboration-session cache hits its LRU entry cap, evicting a session whose
write is still in flight no longer rejects that write as a hard failure — it is
reported as INDETERMINATE ("the update may already have persisted; verify
before retry") so the agent re-reads instead of blind-retrying, and a
still-connecting session is no longer picked as an idle eviction victim by a
parallel acquire. (#494)
- **A long AI chat no longer bricks on the model's context window, and each turn
stops re-persisting the whole tool-output history.** Tool outputs are now
stored ONCE, in `metadata.parts`; the `tool_calls` trace keeps only per-step
outcome flags (a v2 trace shape), ending the O(N²) write amplification that
re-wrote every prior output on every step (measured on a live Postgres via the
`pg_current_wal_lsn()` delta: the trace column shrank ~3200×, the full
assistant row ~51%). The persisted record is unchanged in content — the full
history still lives in `metadata.parts`. At REPLAY time only, the history sent
to the provider is now bounded by a deterministic, prompt-cache-friendly token
budget: `floor(0.7 × chatContextWindow)` when a window is configured (no cap —
anti-brick protection, not a cost limiter), a flat 100k fallback for installs
with no window set (exactly the ones that hit terminal overflow), or off when
the window is explicitly `0`. Trimming truncates old tool outputs first, then
mechanically collapses the oldest turns, always keeping the recent turns full
and the tool-call/result pairing balanced. A provider context-overflow 400 is
now classified and used as a reactive signal: the row is stamped so the NEXT
turn re-trims aggressively (0.5×), which un-bricks a chat that just 400'd. The
client token badge and the server budgeter now share one estimator (new
`@docmost/token-estimate` package) so they can never diverge. Deferred-tool
activation is also cached in the chat metadata to avoid re-resolving it each
turn. (#490)
- **A chat with one malformed message part no longer 500s on every turn, and a
failed send no longer duplicates the user's message.** Incoming client parts
are now whitelisted to `text` (a forged tool-result part can no longer reach
@@ -415,15 +373,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
anchor mark, silently dropping bold/italic/code/link on the affected run; the
prevailing formatting of the replaced run is now carried onto the applied
text. (#496)
- **Markdown round-trips no longer silently drop a line that opens with a block
trigger.** When a document is exported to Markdown and re-imported (git-sync
stabilize, agent writes), a paragraph or continuation line (after a hard break)
that begins with a block marker — an ATX heading `#`, a blockquote/callout `>`,
a list marker (`-`/`*`/`+`/`N.`/`N)`), a code fence, a table `|`, a thematic
break (`---`), or a setext underline (`--`, `----`, or a lone `=`) — is now
backslash-escaped so it round-trips as text instead of being re-parsed into a
heading/list/quote/rule and losing its content. Front-matter stripping is
scoped to the import path only. (#493)
- **The server no longer runs out of heap during long autonomous agent runs.** A
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
snapshot of the ENTIRE turn text on every streamed text-delta when no output
@@ -540,19 +489,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
through that exact share (its own share or an ancestor `includeSubPages`
share); any other value now returns the generic "not found" instead of
serving the page. (#218)
- **MCP tool-allowlist semantics flipped: an empty `[]` now means deny-all
(previously it was coerced to "no restrictions").** For an external MCP server,
a stored `tool_allowlist` of `[]` now denies **every** tool of that server
(zero tools reach the agent) instead of being treated as an empty/unset filter
that allowed all of them. A corrupt or non-array stored value now **fails
closed** to deny-all rather than silently allowing everything. The admin form
no longer silently widens an existing deny-all server: leaving its tag field
empty preserves `[]` (deny-all) on save instead of NULL-ing the column to
allow-all, so a routine rename/toggle can no longer grant the agent every tool.
"No restrictions" is still expressible — a genuinely unrestricted server stores
NULL, and clearing the field on such a server keeps it NULL. Operationally
significant: audit any server that was created or left with a literal `[]`, as
it now exposes no tools until an explicit allowlist (or NULL) is set. (#476)
- **Tool and provider error text no longer leaks to anonymous readers in the
public-share AI chat.** A failing tool's raw error (which could carry an
-1
View File
@@ -22,7 +22,6 @@
"@casl/react": "5.0.1",
"@docmost/editor-ext": "workspace:*",
"@docmost/prosemirror-markdown": "workspace:*",
"@docmost/token-estimate": "workspace:*",
"@excalidraw/excalidraw": "0.18.0-3a5ef40",
"@mantine/core": "8.3.18",
"@mantine/dates": "8.3.18",
+78 -230
View File
@@ -256,9 +256,6 @@
"Invite link": "Ссылка для приглашения",
"Copy": "Копировать",
"Copy to space": "Копировать в пространство",
"Copy chat": "Копировать чат",
"Dock to sidebar": "Закрепить в боковой панели",
"Undock": "Открепить",
"Copied": "Скопировано",
"Failed to export chat": "Не удалось экспортировать чат",
"Duplicate": "Дублировать",
@@ -288,9 +285,6 @@
"Alt text": "Альтернативный текст",
"Describe this for accessibility.": "Опишите это для специальных возможностей.",
"Add a description": "Добавить описание",
"Caption": "Подпись",
"Add a caption": "Добавить подпись",
"Shown below the image.": "Отображается под изображением.",
"Justify": "По ширине",
"Merge cells": "Объединить ячейки",
"Split cell": "Разделить ячейку",
@@ -394,6 +388,22 @@
"Quote": "Цитата",
"Image": "Изображение",
"Audio": "Аудио",
"Transcribe": "Транскрибировать",
"Transcribing…": "Транскрибация…",
"No speech detected": "Речь не распознана",
"Transcription failed": "Не удалось распознать речь",
"Voice dictation is not configured": "Голосовой ввод не настроен",
"Start dictation": "Начать диктовку",
"Stop recording": "Остановить запись",
"Microphone access denied": "Доступ к микрофону запрещён",
"No microphone found": "Микрофон не найден",
"Microphone is unavailable or already in use": "Микрофон недоступен или уже используется",
"Could not start recording": "Не удалось начать запись",
"Audio recording is not available in this browser/context": "Запись аудио недоступна в этом браузере/контексте",
"Dictation": "Диктовка",
"Dictation becomes available once the page finishes connecting": "Диктовка станет доступна после подключения к документу",
"No connection to the collaboration server — dictation unavailable": "Нет связи с сервером совместного редактирования — диктовка недоступна",
"This page is read-only": "Страница открыта только для чтения",
"Embed PDF": "Встроить PDF",
"Upload and embed a PDF file.": "Загрузите и встроите PDF-файл.",
"Embed as PDF": "Встроить как PDF",
@@ -409,6 +419,9 @@
"Footnote {{number}}": "Сноска {{number}}",
"Go to footnote": "Перейти к сноске",
"Back to reference": "Вернуться к ссылке",
"Back to references": "Вернуться к ссылкам",
"Back to reference {{label}}": "Вернуться к ссылке {{label}}",
"Empty footnote": "Пустая сноска",
"Math inline": "Строчная формула",
"Insert inline math equation.": "Вставить математическое выражение в строку.",
"Math block": "Блок формулы",
@@ -434,9 +447,6 @@
"{{count}} command available_other": "Доступно {{count}} команд",
"{{count}} result available_one": "Доступен 1 результат",
"{{count}} result available_other": "Доступно {{count}} результатов",
"{{count}} result found_one": "Найден {{count}} результат",
"{{count}} result found_few": "Найдено {{count}} результата",
"{{count}} result found_other": "Найдено {{count}} результатов",
"Equal columns": "Равные столбцы",
"Left sidebar": "Левая боковая панель",
"Right sidebar": "Правая боковая панель",
@@ -446,7 +456,6 @@
"Names do not match": "Названия не совпадают",
"Today, {{time}}": "Сегодня, {{time}}",
"Yesterday, {{time}}": "Вчера, {{time}}",
"now": "сейчас",
"Space created successfully": "Пространство успешно создано",
"Space updated successfully": "Пространство успешно обновлено",
"Space deleted successfully": "Пространство успешно удалено",
@@ -550,7 +559,6 @@
"Add 2FA method": "Добавить метод 2FA",
"Backup codes": "Резервные коды",
"Disable": "Отключить",
"disabled": "отключено",
"Invalid verification code": "Недействительный код подтверждения",
"New backup codes have been generated": "Новые резервные коды сгенерированы",
"Failed to regenerate backup codes": "Не удалось заново сгенерировать резервные коды",
@@ -694,6 +702,62 @@
"AI search": "Поиск ИИ",
"AI Answer": "Ответ ИИ",
"Ask AI": "Спросить ИИ",
"AI agent": "AI-агент",
"Take a look at the current document": "Посмотри текущий документ",
"Start automatically": "Запускать автоматически",
"When on, picking this role sends a launch message and starts the chat. When off, the role is selected and you type the first message yourself.": "Когда включено, выбор этой роли отправляет стартовое сообщение и начинает чат. Когда выключено, роль выбирается, а первое сообщение вы вводите сами.",
"Launch message": "Стартовое сообщение",
"Sent automatically when this role is picked. Leave empty to use the default text. Ignored when “Start automatically” is off.": "Отправляется автоматически при выборе этой роли. Оставьте пустым, чтобы использовать текст по умолчанию. Игнорируется, когда «Запускать автоматически» выключено.",
"AI agent is typing…": "AI-агент печатает…",
"{{name}} is typing…": "{{name}} печатает…",
"Thinking…": "Думаю…",
"Thinking… · {{count}} tokens": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_one": "Думаю… · {{count}} токен",
"Thinking… · {{count}} tokens_few": "Думаю… · {{count}} токена",
"Thinking… · {{count}} tokens_many": "Думаю… · {{count}} токенов",
"Thinking · {{count}} tokens": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_one": "Размышления · {{count}} токен",
"Thinking · {{count}} tokens_few": "Размышления · {{count}} токена",
"Thinking · {{count}} tokens_many": "Размышления · {{count}} токенов",
"Agent role": "Роль агента",
"AI chat": "AI-чат",
"AI chat is disabled for this workspace.": "AI-чат отключён для этого рабочего пространства.",
"Ask a question about this documentation.": "Задайте вопрос об этой документации.",
"Ask a question…": "Задайте вопрос…",
"Ask the AI agent anything about your workspace.": "Спросите AI-агента о чём угодно по вашему рабочему пространству.",
"Ask the AI agent…": "Спросите AI-агента…",
"Copy chat": "Копировать чат",
"Dock to sidebar": "Закрепить в боковой панели",
"Undock": "Открепить",
"Created successfully": "Успешно создано",
"Context size / model limit": "Размер контекста / лимит модели",
"Context window (tokens)": "Окно контекста (токены)",
"Shown as used / total in the chat header. Leave empty to hide the limit.": "Показывается в шапке чата как использовано / всего. Пусто — лимит скрыт.",
"Delete this chat?": "Удалить этот чат?",
"Deleted successfully": "Успешно удалено",
"AI agent «{{role}}» on behalf of {{person}}": "AI-агент «{{role}}» от имени {{person}}",
"AI agent {{name}}": "AI-агент {{name}}",
"Failed to delete chat": "Не удалось удалить чат",
"Failed to rename chat": "Не удалось переименовать чат",
"Failed": "Ошибка",
"OK · {{n}}": "OK · {{n}}",
"Test": "Тест",
"No tools available": "Инструменты недоступны",
"Available tools": "Доступные инструменты",
"Minimize": "Свернуть",
"No chats yet.": "Чатов пока нет.",
"Send": "Отправить",
"Send when the agent finishes": "Отправить, когда агент закончит",
"Queue message": "Поставить в очередь",
"Remove queued message": "Убрать из очереди",
"Send now": "Отправить сейчас",
"Interrupt and send now": "Прервать и отправить сейчас",
"Something went wrong": "Что-то пошло не так",
"Stop": "Стоп",
"The AI agent could not respond. Please try again.": "AI-агент не смог ответить. Попробуйте ещё раз.",
"The AI provider is not configured. Ask an administrator to set it up.": "AI-провайдер не настроен. Попросите администратора настроить его.",
"Universal assistant": "Универсальный ассистент",
"You": "Вы",
"AI is thinking...": "ИИ обрабатывает запрос...",
"Thinking": "Думаю",
"Ask a question...": "Задайте вопрос...",
@@ -720,40 +784,8 @@
"Manage API keys for all users in the workspace. View the <anchor>API documentation</anchor> for usage details.": "Управляйте API-ключами для всех пользователей в рабочем пространстве. Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>API documentation</anchor> for usage details.": "Смотрите <anchor>документацию по API</anchor> для получения информации об использовании.",
"View the <anchor>MCP documentation</anchor>.": "Смотрите <anchor>документацию по MCP</anchor>.",
"AI / Models": И / Модели",
"AI / External tools (MCP)": "ИИ / Внешние инструменты (MCP)",
"Add server": "Добавить сервер",
"Edit server": "Изменить сервер",
"Delete server": "Удалить сервер",
"Are you sure you want to delete this MCP server?": "Вы уверены, что хотите удалить этот MCP-сервер?",
"No external servers configured": "Внешние серверы не настроены",
"Server name": "Имя сервера",
"Transport": "Транспорт",
"URL": "URL",
"Authorization header": "Заголовок авторизации",
"Tool allowlist": "Список разрешённых инструментов",
"Optional. Leave empty to allow all tools the server exposes.": "Необязательно. Оставьте пустым, чтобы разрешить все инструменты, которые предоставляет сервер.",
"Instructions": нструкции",
"Optional guidance for the agent on how and when to use this server's tools. Injected into the system prompt. The server's tools are namespaced as \"<server name>_*\".": "Необязательное указание агенту, как и когда использовать инструменты этого сервера. Добавляется в системный промпт. Инструменты сервера именуются с префиксом «<имя сервера>_*».",
"Test": "Тест",
"Available tools": "Доступные инструменты",
"No tools available": "Инструменты недоступны",
"Failed": "Ошибка",
"OK · {{n}}": "OK · {{n}}",
"Created successfully": "Успешно создано",
"Deleted successfully": "Успешно удалено",
"Clear": "Очистить",
"Provider": "Провайдер",
"•••• set": "•••• задан",
"Clear key": "Очистить ключ",
"Base URL": "Базовый URL",
"Chat model": "Модель чата",
"Embedding model": "Модель эмбеддингов",
"System message": "Системное сообщение",
"A built-in safety framework is always appended.": "Встроенный набор правил безопасности всегда добавляется автоматически.",
"Test connection": "Проверить соединение",
"Connection successful": "Соединение установлено",
"Connection failed": "Не удалось установить соединение",
"Only workspace admins can manage AI provider settings.": "Управлять настройками провайдера ИИ могут только администраторы рабочего пространства.",
"Sources": "Источники",
"AI Answers not available for attachments": "Ответы ИИ недоступны для вложений",
"No answer available": "Ответ недоступен",
@@ -981,7 +1013,6 @@
"Try again": "Попробовать снова",
"Untitled chat": "Чат без названия",
"No document": "Без документа",
"You": "Вы",
"What can I help you with?": "Чем я могу вам помочь?",
"Are you sure you want to revoke this {{credential}}": "Вы уверены, что хотите отозвать этот {{credential}}",
"Automatically provision users and groups from your identity provider via SCIM.": "Автоматически предоставляйте доступ пользователям и группам из вашего провайдера удостоверений через SCIM.",
@@ -1010,9 +1041,6 @@
"Page menu": "Меню страницы",
"Expand": "Развернуть",
"Collapse": "Свернуть",
"Expand all": "Развернуть все",
"Collapse all": "Свернуть все",
"Couldn't expand the tree: {{reason}}": "Не удалось развернуть дерево: {{reason}}",
"Comment menu": "Меню комментария",
"Group menu": "Меню группы",
"Show hidden breadcrumbs": "Показать скрытые хлебные крошки",
@@ -1049,7 +1077,7 @@
"Search pages and spaces...": "Поиск страниц и пространств...",
"No results found": "Результаты не найдены",
"You don't have permission to create pages here": "У вас нет прав на создание страниц здесь",
"Chat menu for {{title}}": "Меню чата для {{title}}",
"Chat menu": "Меню чата",
"API key menu": "Меню API-ключа",
"Jump to comment selection": "Перейти к выбору комментария",
"Slash commands": "Команды со слешем",
@@ -1103,9 +1131,6 @@
"Undo": "Отменить",
"Redo": "Повторить",
"Backlinks": "Обратные ссылки",
"Back to references": "Вернуться к ссылкам",
"Back to reference {{label}}": "Вернуться к ссылке {{label}}",
"Empty footnote": "Пустая сноска",
"Last updated by": "Последний изменивший",
"Last updated": "Последнее обновление",
"Stats": "Статистика",
@@ -1139,7 +1164,6 @@
"Page title": "Заголовок страницы",
"Page content": "Содержимое страницы",
"Member actions": "Действия с участником",
"Member actions for {{name}}": "Действия с участником {{name}}",
"Toggle password visibility": "Переключить видимость пароля",
"Send comment": "Отправить комментарий",
"Token actions": "Действия с токеном",
@@ -1159,187 +1183,11 @@
"Removed from favorites": "Удалено из избранного",
"Added {{name}} to favorites": "{{name}} добавлено в избранное",
"Removed {{name}} from favorites": "{{name}} удалено из избранного",
"Label added": "Метка добавлена",
"Label removed": "Метка удалена",
"Image updated": "Изображение обновлено",
"Unsupported image type": "Неподдерживаемый тип изображения",
"Member deactivated": "Участник деактивирован",
"Member activated": "Участник активирован",
"Name is required": "Укажите имя",
"Name must be 40 characters or fewer": "Имя должно содержать не более 40 символов",
"Group name must be at least 2 characters": "Название группы должно содержать не менее 2 символов",
"Group name must be 100 characters or fewer": "Название группы должно содержать не более 100 символов",
"Description must be 500 characters or fewer": "Описание должно содержать не более 500 символов",
"Invalid invitation link": "Недействительная ссылка-приглашение",
"Page menu for {{name}}": "Меню страницы для {{name}}",
"Create subpage of {{name}}": "Создать подстраницу для {{name}}",
"AI chat": "AI-чат",
"Ask a question about this documentation.": "Задайте вопрос об этой документации.",
"Ask a question…": "Задайте вопрос…",
"Thinking…": "Думаю…",
"Thinking… · {{count}} tokens": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_one": "Думаю… · {{count}} токен",
"Thinking… · {{count}} tokens_few": "Думаю… · {{count}} токена",
"Thinking… · {{count}} tokens_many": "Думаю… · {{count}} токенов",
"Thinking… · {{count}} tokens_other": "Думаю… · {{count}} токенов",
"Thinking · {{count}} tokens": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_one": "Размышления · {{count}} токен",
"Thinking · {{count}} tokens_few": "Размышления · {{count}} токена",
"Thinking · {{count}} tokens_many": "Размышления · {{count}} токенов",
"Thinking · {{count}} tokens_other": "Размышления · {{count}} токенов",
"The assistant is unavailable right now. Please try again.": "Ассистент сейчас недоступен. Попробуйте ещё раз.",
"Public share assistant": "Ассистент публичного доступа",
"Let anonymous visitors of public shares ask an AI assistant scoped to that share's pages. You pay for the tokens.": "Позвольте анонимным посетителям публичных ссылок обращаться к ИИ-ассистенту в рамках страниц этой публикации. Токены оплачиваете вы.",
"Public assistant model": "Модель публичного ассистента",
"Defaults to the chat model": "По умолчанию используется модель чата",
"Optional cheaper model id for the public assistant. Empty uses the chat model above.": "Необязательный более дешёвый идентификатор модели для публичного ассистента. Если пусто, используется модель чата выше.",
"Assistant identity": "Личность ассистента",
"Pick an agent role whose persona the public assistant adopts. The safety rules always still apply.": "Выберите роль агента, чью личность примет публичный ассистент. Правила безопасности всегда остаются в силе.",
"Built-in assistant persona": "Встроенная личность ассистента",
"Minimize": "Свернуть",
"Context size / model limit": "Размер контекста / лимит модели",
"Context window (tokens)": "Окно контекста (токены)",
"Shown as used / total in the chat header. Leave empty to hide the limit.": "Показывается в шапке чата как использовано / всего. Пусто — лимит скрыт.",
"AI agent": "AI-агент",
"Take a look at the current document": "Посмотри текущий документ",
"AI agent is typing…": "AI-агент печатает…",
"{{name}} is typing…": "{{name}} печатает…",
"Send": "Отправить",
"Send when the agent finishes": "Отправить, когда агент закончит",
"Queue message": "Поставить в очередь",
"Remove queued message": "Убрать из очереди",
"Send now": "Отправить сейчас",
"Interrupt and send now": "Прервать и отправить сейчас",
"Stop": "Стоп",
"Response stopped.": "Ответ остановлен.",
"Connection lost — the answer was interrupted.": "Соединение потеряно — ответ был прерван.",
"Response stopped (manually or the connection dropped).": "Ответ остановлен (вручную или из-за разрыва соединения).",
"Chat menu": "Меню чата",
"No chats yet.": "Чатов пока нет.",
"Delete this chat?": "Удалить этот чат?",
"Ask the AI agent…": "Спросите AI-агента…",
"Ask the AI agent anything about your workspace.": "Спросите AI-агента о чём угодно по вашему рабочему пространству.",
"Failed to rename chat": "Не удалось переименовать чат",
"Failed to delete chat": "Не удалось удалить чат",
"Something went wrong": "Что-то пошло не так",
"AI chat is disabled for this workspace.": "AI-чат отключён для этого рабочего пространства.",
"The AI provider is not configured. Ask an administrator to set it up.": "AI-провайдер не настроен. Попросите администратора настроить его.",
"The AI agent could not respond. Please try again.": "AI-агент не смог ответить. Попробуйте ещё раз.",
"Searched pages": "Поиск по страницам",
"Read page": "Прочитана страница",
"Created page": "Создана страница",
"Updated page": "Обновлена страница",
"Renamed page": "Переименована страница",
"Moved page": "Перемещена страница",
"Deleted page (to trash)": "Удалена страница (в корзину)",
"Commented": "Добавлен комментарий",
"Resolved comment": "Комментарий решён",
"Ran tool {{name}}": "Выполнен инструмент {{name}}",
"AI agent «{{role}}» on behalf of {{person}}": "AI-агент «{{role}}» от имени {{person}}",
"AI agent {{name}}": "AI-агент {{name}}",
"Endpoints": "Эндпоинты",
"where we fetch models": "откуда мы получаем модели",
"All endpoints are OpenAI-compatible. Point the Base URL at OpenAI, OpenRouter, a local Ollama, or any self-hosted server.": "Все эндпоинты совместимы с OpenAI. Укажите в базовом URL адрес OpenAI, OpenRouter, локального Ollama или любого self-hosted сервера.",
"Chat / LLM": "Чат / LLM",
"root": "корневой",
"Semantic search": "Семантический поиск",
"Voice / STT": "Голос / STT",
"Voice dictation": "Голосовой ввод",
"Streaming dictation": "Потоковый голосовой ввод",
"Transcribe as you speak, cutting on pauses": "Транскрибирование по мере речи, с разбивкой на паузах",
"Voice dictation is not available yet.": "Голосовой ввод пока недоступен.",
"Test endpoint": "Проверить эндпоинт",
"Save and test": "Сохранить и проверить",
"Save endpoints": "Сохранить эндпоинты",
"Configured and enabled": "Настроено и включено",
"Configured but disabled": "Настроено, но отключено",
"Enabled but not configured": "Включено, но не настроено",
"Not configured": "Не настроено",
"External tools": "Внешние инструменты",
"Gitmost as MCP client": "Gitmost как MCP-клиент",
"Servers the agent calls out to.": "Серверы, к которым обращается агент.",
"MCP server": "MCP-сервер",
"expose the workspace": "открыть доступ к рабочему пространству",
"Enable MCP server": "Включить MCP-сервер",
"Exposes the workspace as an MCP server at /mcp — this provides a capability, it doesn't consume a model.": "Открывает рабочее пространство как MCP-сервер по адресу /mcp — это предоставляет возможность, а не потребляет модель.",
"Resolves to {{url}}": "Разрешается в {{url}}",
"Model": "Модель",
"Done": "Готово",
"shared prompt · safety framework appended automatically": "общий промпт · правила безопасности добавляются автоматически",
"/v1/chat/completions · root endpoint — Embeddings and Voice inherit its URL and key": "/v1/chat/completions · корневой эндпоинт — Эмбеддинги и Голос наследуют его URL и ключ",
"/v1/embeddings · embeds pages so semantic search can find them": "/v1/embeddings · создаёт эмбеддинги страниц, чтобы их находил семантический поиск",
"/v1/audio/transcriptions · works with local whisper (speaches / faster-whisper-server)": "/v1/audio/transcriptions · работает с локальным whisper (speaches / faster-whisper-server)",
"Vector search · requires pgvector": "Векторный поиск · требуется pgvector",
"Embedding API key": "API-ключ для эмбеддингов",
"Embeddings": "Эмбеддинги",
"Leave empty to use the chat API key": "Оставьте пустым, чтобы использовать API-ключ чата",
"Leave empty to use the chat base URL": "Оставьте пустым, чтобы использовать базовый URL чата",
"Reindex now": "Переиндексировать сейчас",
"Start dictation": "Начать диктовку",
"Stop recording": "Остановить запись",
"Transcribing…": "Транскрибация…",
"Microphone access denied": "Доступ к микрофону запрещён",
"No microphone found": "Микрофон не найден",
"Could not start recording": "Не удалось начать запись",
"Transcription failed": "Не удалось распознать речь",
"Transcribe": "Транскрибировать",
"No speech detected": "Речь не распознана",
"Voice dictation is not configured": "Голосовой ввод не настроен",
"Microphone is unavailable or already in use": "Микрофон недоступен или уже используется",
"Audio recording is not available in this browser/context": "Запись аудио недоступна в этом браузере/контексте",
"Dictation": "Диктовка",
"Dictation becomes available once the page finishes connecting": "Диктовка станет доступна после подключения к документу",
"No connection to the collaboration server — dictation unavailable": "Нет связи с сервером совместного редактирования — диктовка недоступна",
"This page is read-only": "Страница открыта только для чтения",
"Request format": "Формат запроса",
"How transcription requests are sent to the endpoint": "Как запросы на транскрибирование отправляются на эндпоинт",
"OpenAI-compatible (multipart/form-data)": "Совместимо с OpenAI (multipart/form-data)",
"OpenRouter (JSON, base64 audio)": "OpenRouter (JSON, аудио в base64)",
"Dictation language": "Язык диктовки",
"Auto-detect": "Автоопределение",
"Spoken language hint sent to the transcription model. Auto-detect lets the model decide.": "Подсказка языка речи для модели транскрипции. «Автоопределение» оставляет выбор за моделью.",
"Agent role": "Роль агента",
"Universal assistant": "Универсальный ассистент",
"Add role": "Добавить роль",
"Edit role": "Изменить роль",
"Role name": "Название роли",
"e.g. Proofreader": "напр. Корректор",
"Optional. Shown as the chat badge.": "Необязательно. Отображается как значок чата.",
"Optional. A short note about what this role does.": "Необязательно. Краткое описание того, что делает эта роль.",
"Instructions": "Инструкции",
"The built-in safety framework is always added automatically.": "Встроенный набор правил безопасности всегда добавляется автоматически.",
"Model provider override": "Переопределение провайдера модели",
"Optional. Defaults to the workspace provider.": "Необязательно. По умолчанию используется провайдер рабочего пространства.",
"Model override": "Переопределение модели",
"Optional. Defaults to the workspace model.": "Необязательно. По умолчанию используется модель рабочего пространства.",
"e.g. gpt-4o-mini": "напр. gpt-4o-mini",
"If you choose a different provider, it must already be configured in AI settings.": "Если вы выбираете другого провайдера, он уже должен быть настроен в настройках ИИ.",
"Start automatically": "Запускать автоматически",
"When on, picking this role sends a launch message and starts the chat. When off, the role is selected and you type the first message yourself.": "Когда включено, выбор этой роли отправляет стартовое сообщение и начинает чат. Когда выключено, роль выбирается, а первое сообщение вы вводите сами.",
"Launch message": "Стартовое сообщение",
"Sent automatically when this role is picked. Leave empty to use the default text. Ignored when “Start automatically” is off.": "Отправляется автоматически при выборе этой роли. Оставьте пустым, чтобы использовать текст по умолчанию. Игнорируется, когда «Запускать автоматически» выключено.",
"Agent roles": "Роли агента",
"Reusable presets that shape the agent's behavior (and optionally its model). Picked when starting a new chat.": "Многоразовые пресеты, определяющие поведение агента (и, при желании, его модель). Выбираются при запуске нового чата.",
"No roles configured": "Роли не настроены",
"Delete role": "Удалить роль",
"Are you sure you want to delete this role?": "Вы уверены, что хотите удалить эту роль?",
"HTML embed": "HTML-вставка",
"Edit HTML embed": "Изменить HTML-вставку",
"HTML embed is disabled in this workspace": "HTML-вставки отключены в этом рабочем пространстве",
"Click to add HTML / CSS / JS": "Нажмите, чтобы добавить HTML / CSS / JS",
"This HTML/CSS/JS runs in a sandboxed frame and cannot access the viewer's session, cookies, or API.": "Этот HTML/CSS/JS выполняется в изолированном фрейме и не имеет доступа к сессии, cookie или API просматривающего.",
"<script>...</script>": "<script>...</script>",
"Height (px, blank = auto)": "Высота (px, пусто = авто)",
"advanced": "дополнительно",
"Enable HTML embed": "Включить HTML-вставки",
"Allow members to insert raw HTML/CSS/JavaScript blocks. The block renders in a sandboxed frame and cannot access the viewer's session, cookies, or API. Off by default.": "Разрешить участникам вставлять блоки с необработанным HTML/CSS/JavaScript. Блок отображается в изолированном фрейме и не имеет доступа к сессии, cookie или API просматривающего. По умолчанию выключено.",
"When enabled, any member can insert an HTML embed block. The toggle just enables or disables the block type workspace-wide.": "Когда включено, любой участник может вставить блок HTML-вставки. Переключатель просто включает или отключает этот тип блока во всём рабочем пространстве.",
"Embeds run inside a sandboxed iframe with a separate origin, so they cannot read or modify the page they are embedded in.": "Вставки выполняются в изолированном iframe с отдельным источником, поэтому они не могут читать или изменять страницу, в которую встроены.",
"Turning this off hides existing embeds (they render as a disabled placeholder) and stops serving them on public share pages.": "Отключение этой опции скрывает существующие вставки (они отображаются как отключённая заглушка) и прекращает их показ на публичных страницах.",
"Analytics / tracker": "Аналитика / трекер",
"Injected verbatim into the <head> of PUBLIC SHARE pages only (same-origin). For analytics snippets (Google Analytics, Yandex.Metrika, etc.). Admin only.": "Вставляется дословно в <head> только ПУБЛИЧНЫХ страниц (тот же источник). Для сниппетов аналитики (Google Analytics, Яндекс.Метрика и т. п.). Только для администраторов.",
"Go to login page": "Перейти на страницу входа",
"Move to space": "Переместить в пространство",
"Float left (wrap text)": "Обтекание слева",
"Float right (wrap text)": "Обтекание справа",
"Inline (side by side)": "В ряд",
@@ -1351,7 +1199,6 @@
"Showing {{count}} subpages_one": "Показано {{count}} подстраница",
"Showing {{count}} subpages_few": "Показано {{count}} подстраницы",
"Showing {{count}} subpages_many": "Показано {{count}} подстраниц",
"Showing {{count}} subpages_other": "Показано {{count}} подстраниц",
"Protocol": "Протокол",
"How chat requests are sent and how reasoning is surfaced": "Как отправляются запросы чата и как показывается reasoning",
"OpenAI-compatible (surfaces reasoning)": "OpenAI-совместимый (показывает reasoning)",
@@ -1421,6 +1268,7 @@
"Retry": "Повторить",
"The catalog is empty": "Каталог пуст",
"No role bundles are published for this language yet. Try switching the content language.": "Для этого языка ещё не опубликовано ни одного набора ролей. Попробуйте сменить язык контента.",
"No roles configured": "Роли не настроены",
"Already up to date": "Уже актуальна",
"Updated to the latest version": "Обновлено до последней версии",
"This role is no longer in the catalog": "Эта роль больше не представлена в каталоге",
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { isChunkLoadError, shouldAutoReload } from "./chunk-load-error-boundary";
import { isChunkLoadError } from "./chunk-load-error-boundary";
// The detector decides whether a caught render error is a stale-deploy chunk-404
// (→ auto-reload to fetch the new manifest) vs a genuine app error (→ generic
@@ -35,31 +35,3 @@ describe("isChunkLoadError", () => {
expect(isChunkLoadError(err)).toBe(false);
});
});
// The window gate replaces the old one-shot flag: it must permit recovery across
// several deploys in one tab (each > window apart) while still stopping an infinite
// reload loop when a lazy chunk is permanently broken (a second failure < window).
describe("shouldAutoReload", () => {
const WINDOW = 5 * 60 * 1000;
const NOW = 1_000_000_000_000;
it("allows a reload when we have never auto-reloaded", () => {
expect(shouldAutoReload(NOW, null, WINDOW)).toBe(true);
});
it("allows a reload when the last one was 6 minutes ago (outside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 6 * 60 * 1000, WINDOW)).toBe(true);
});
it("blocks a reload when the last one was 1 minute ago (inside the window)", () => {
expect(shouldAutoReload(NOW, NOW - 1 * 60 * 1000, WINDOW)).toBe(false);
});
it("blocks a reload exactly at the window boundary (not strictly older)", () => {
expect(shouldAutoReload(NOW, NOW - WINDOW, WINDOW)).toBe(false);
});
it("allows a reload when the stored timestamp is unparseable (NaN)", () => {
expect(shouldAutoReload(NOW, NaN, WINDOW)).toBe(true);
});
});
@@ -2,25 +2,7 @@ import { ReactNode } from "react";
import { ErrorBoundary } from "react-error-boundary";
import { Button, Center, Stack, Text } from "@mantine/core";
// sessionStorage key holding the epoch-ms timestamp of the last automatic reload.
const RELOAD_AT_KEY = "chunk-reload-at";
// Allow at most one automatic reload per this window. A stale-deploy 404 is cured
// by a single reload, so anything inside the window is treated as a reload loop
// (permanently-broken chunk) and falls through to the manual UI. A window (rather
// than a one-shot flag) lets a SECOND deploy in the same tab's lifetime recover too.
const RELOAD_WINDOW_MS = 5 * 60 * 1000;
// Pure window decision, unit-tested in isolation: auto-reload only if we have never
// auto-reloaded (lastReloadAt null/NaN) or the last one was strictly older than the
// window. Anything inside the window is suppressed to break an infinite reload loop.
export function shouldAutoReload(
now: number,
lastReloadAt: number | null,
windowMs: number,
): boolean {
if (lastReloadAt === null || Number.isNaN(lastReloadAt)) return true;
return now - lastReloadAt > windowMs;
}
const RELOAD_FLAG = "chunk-reload-attempted";
// Heuristic detection of a failed dynamic import. Since the code-splitting work,
// every route (plus Aside / AiChatWindow) is React.lazy: when a new deploy
@@ -42,16 +24,12 @@ export function isChunkLoadError(error: unknown): boolean {
function handleError(error: unknown) {
if (!isChunkLoadError(error)) return;
// A stale-chunk 404 is cured by a full reload that re-fetches index.html and
// the new chunk manifest. Auto-reload at most once per RELOAD_WINDOW_MS: this
// recovers across multiple deploys in a single tab's lifetime, yet a
// permanently-broken lazy chunk (which would loop) is stopped after the first
// reload and falls through to the manual recovery UI below.
// the new chunk manifest. Auto-reload once, guarding against a reload loop
// (e.g. a genuinely missing chunk) with a one-shot sessionStorage flag. If the
// flag is already set we fall through to the manual recovery UI below.
try {
const raw = sessionStorage.getItem(RELOAD_AT_KEY);
const lastReloadAt = raw === null ? null : Number.parseInt(raw, 10);
const now = Date.now();
if (!shouldAutoReload(now, lastReloadAt, RELOAD_WINDOW_MS)) return;
sessionStorage.setItem(RELOAD_AT_KEY, String(now));
if (sessionStorage.getItem(RELOAD_FLAG)) return;
sessionStorage.setItem(RELOAD_FLAG, "1");
} catch {
// sessionStorage unavailable (private mode / disabled): skip the automatic
// reload rather than risk an unguarded loop; the fallback UI still recovers.
@@ -6,13 +6,10 @@ describe("estimateTokens", () => {
expect(estimateTokens("")).toBe(0);
});
// #490: migrated onto the shared @docmost/token-estimate module (chars/2.5, up
// from the old client-only chars/4) so the client counter and the server replay
// budgeter can never diverge.
it("ceils chars/2.5 so any non-empty text is at least 1 token", () => {
it("ceils chars/4 so any non-empty text is at least 1 token", () => {
expect(estimateTokens("a")).toBe(1);
expect(estimateTokens("ab")).toBe(1);
expect(estimateTokens("abcde")).toBe(2); // 5 / 2.5 = 2
expect(estimateTokens("x".repeat(10))).toBe(4); // 10 / 2.5 = 4
expect(estimateTokens("abcd")).toBe(1);
expect(estimateTokens("abcde")).toBe(2);
expect(estimateTokens("12345678")).toBe(2);
});
});
@@ -2,10 +2,18 @@
* Rough client-side token estimation for AI-chat UI affordances.
*
* No provider streams exact per-token usage mid-stream, so any in-flight figure
* is a CLIENT ESTIMATE. This re-exports the SHARED estimator from
* `@docmost/token-estimate` (chars/2.5) so the in-body counter and the server's
* replay budgeter use the SAME heuristic two divergent estimators would mean
* "the badge shows 60%" while "the budgeter already trimmed" (#490). Used by the
* in-body reasoning counter ("Thinking · N tokens").
* is a CLIENT ESTIMATE (chars/4 heuristic). Pure + unit-testable: it never runs
* a real BPE tokenizer (that would be O(n²) on the hot path, bloat the bundle,
* and be wrong for Gemini/Ollama anyway). Used by the in-body reasoning counter
* ("Thinking · N tokens").
*/
export { estimateTokens } from "@docmost/token-estimate";
/**
* Rough token estimate for a piece of text using the standard chars/4 heuristic.
* Returns 0 for empty/whitespace-free-of-content input, and ceils so any
* non-empty text counts as at least one token.
*/
export function estimateTokens(text: string): number {
if (!text) return 0;
return Math.ceil(text.length / 4);
}
@@ -89,23 +89,6 @@ describe("describeChatError", () => {
expect(view.title).not.toBe("AI provider not configured");
});
it("classifies a token-degeneration abort under the SAME 'Response stopped.' marker the live view shows (#495)", () => {
// The exact reason the server persists in metadata.error on a degeneration
// abort (ai-chat.service OUTPUT_DEGENERATION_ERROR). Live, this event shows
// the neutral "Response stopped." notice; the persisted banner MUST match it
// so live and refetch never disagree.
const view = describeChatError(
"Output degeneration detected (repeated token loop)",
t,
);
expect(view.title).toBe("Response stopped.");
expect(view.detail).toBe(
"The answer was stopped automatically because the model fell into a repeated output loop.",
);
// Regression guard: it must NOT fall through to the generic heading.
expect(view.title).not.toBe("Something went wrong");
});
it("classifies a dropped connection (ECONNRESET) as a lost-connection error", () => {
expect(
describeChatError("Cannot connect to API: read ECONNRESET", t).title,
@@ -77,22 +77,6 @@ export function describeChatError(
};
}
// Our own token-degeneration abort (#444): the server aborts a runaway
// repetition loop and persists this exact reason in metadata.error. LIVE, the
// same abort surfaces as the neutral "Response stopped." notice (the client
// cannot tell it from a manual Stop mid-stream), so the persisted banner must
// read the SAME "Response stopped." marker — otherwise the live view and a
// later refetch show two different texts for one event. The detail explains the
// loop-guard cause without contradicting the shared heading.
if (/output degeneration detected|repeated token loop/i.test(msg)) {
return {
title: t("Response stopped."),
detail: t(
"The answer was stopped automatically because the model fell into a repeated output loop.",
),
};
}
if (/"statusCode"\s*:\s*403\b/.test(msg)) {
return {
title: t("AI chat is disabled"),
@@ -1,65 +0,0 @@
import { describe, it, expect } from "vitest";
import * as Y from "yjs";
import { yHistoryAvailability } from "./use-toolbar-state.ts";
// Undo/redo availability is derived from the Yjs UndoManager's PRIVATE
// `undoStack` / `redoStack` fields (see use-toolbar-state.ts for why we read the
// stack lengths directly instead of the expensive `editor.can().undo()` dry-run).
// These tests lock in the behavior AND pin the library shape so a yjs / y-undo
// upgrade that renames/restructures those internals fails loudly here rather than
// silently enabling/disabling the toolbar buttons in production.
describe("yHistoryAvailability", () => {
it("reports availability from the stack lengths", () => {
expect(yHistoryAvailability({ undoStack: [], redoStack: [] })).toEqual({
canUndo: false,
canRedo: false,
});
expect(
yHistoryAvailability({ undoStack: [{}], redoStack: [] }),
).toEqual({ canUndo: true, canRedo: false });
expect(
yHistoryAvailability({ undoStack: [{}], redoStack: [{}, {}] }),
).toEqual({ canUndo: true, canRedo: true });
});
it("returns null when the private stack shape is unrecognized (upgrade guard)", () => {
// Simulates a yjs / y-undo upgrade that renames or restructures the private
// fields: the caller then falls back to the safe prosemirror-history default
// instead of throwing on `.length` of undefined or reading garbage.
expect(yHistoryAvailability(undefined)).toBeNull();
expect(yHistoryAvailability(null)).toBeNull();
expect(yHistoryAvailability({})).toBeNull();
expect(yHistoryAvailability({ undoStack: 5, redoStack: 5 })).toBeNull();
// Only one stack present (partial rename) is still not trusted.
expect(yHistoryAvailability({ undoStack: [] })).toBeNull();
});
it("pin-test: a real yjs UndoManager still exposes undoStack/redoStack arrays", () => {
const doc = new Y.Doc();
const text = doc.getText("prosemirror");
const undoManager = new Y.UndoManager(text);
// Fresh manager: both stacks empty -> nothing to undo/redo.
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: false,
canRedo: false,
});
// A tracked edit must push onto the private undoStack. If a future yjs
// renames these fields, yHistoryAvailability(undoManager) returns null and
// the expectation below fails loudly.
text.insert(0, "hello");
undoManager.stopCapturing();
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: true,
canRedo: false,
});
// Undoing moves the item to the redoStack -> redo becomes available.
undoManager.undo();
expect(yHistoryAvailability(undoManager)).toEqual({
canUndo: false,
canRedo: true,
});
});
});
@@ -35,30 +35,6 @@ export interface ToolbarState {
// When neither history backend is installed (the pre-sync static editor —
// mainExtensions only, undoRedo disabled), both fall through to 0 -> false,
// matching the previous `safeCan` behavior.
// Reads the Yjs UndoManager's undo/redo availability from its stack lengths.
//
// `undoStack` / `redoStack` are PRIVATE y-undo / yjs internals, so we touch them
// defensively: a yjs or y-undo upgrade that renames or restructures these fields
// must not silently mis-drive the toolbar buttons (nor throw on `.length` of
// `undefined`). We only trust them when they are actually arrays; otherwise this
// returns null and the caller falls back to a safe default. The pin-test in
// use-toolbar-state.test.ts asserts the current library shape, so an upgrade that
// breaks this contract fails loudly there instead of failing silently in the UI.
export function yHistoryAvailability(
undoManager: unknown,
): { canUndo: boolean; canRedo: boolean } | null {
if (!undoManager || typeof undoManager !== "object") return null;
const { undoStack, redoStack } = undoManager as {
undoStack?: unknown;
redoStack?: unknown;
};
if (!Array.isArray(undoStack) || !Array.isArray(redoStack)) return null;
return {
canUndo: undoStack.length > 0,
canRedo: redoStack.length > 0,
};
}
function historyAvailability(editor: Editor): {
canUndo: boolean;
canRedo: boolean;
@@ -67,14 +43,16 @@ function historyAvailability(editor: Editor): {
// Collaboration history (Yjs) takes precedence when present.
const yState = yUndoPluginKey.getState(state) as
| { undoManager?: unknown }
| { undoManager?: { undoStack: unknown[]; redoStack: unknown[] } }
| undefined;
const yAvail = yHistoryAvailability(yState?.undoManager);
if (yAvail) return yAvail;
if (yState?.undoManager) {
return {
canUndo: yState.undoManager.undoStack.length > 0,
canRedo: yState.undoManager.redoStack.length > 0,
};
}
// Plain prosemirror-history (returns 0 when the history plugin is absent).
// This is also the safe default when a Yjs UndoManager is present but its
// private stack shape is no longer recognized (yHistoryAvailability -> null).
return {
canUndo: undoDepth(state) > 0,
canRedo: redoDepth(state) > 0,
@@ -9,7 +9,7 @@ import { Italic } from "@tiptap/extension-italic";
import { Link } from "@tiptap/extension-link";
import { gitmostInsertTranscriptIntoEditor } from "./gitmost-recording.ts";
const ZWSP = "​"; // U+200B — asserted ABSENT (the block-escape lives in the serializer now)
const ZWSP = "​"; // U+200B, the helper's block-trigger neutralizer
/**
* #377 the web-side bridge must append the native host's transcript below the
@@ -18,9 +18,8 @@ const ZWSP = "​"; // U+200B — asserted ABSENT (the block-escape lives in the
* regression would be caught), asserting the resulting document rather than
* mocking the editor: transcript present -> "Transcript" heading + one paragraph
* per non-empty line; content is inserted as LITERAL TEXT (no HTML/markdown
* parsing); col-0 markdown block triggers are stored verbatim (the git-sync
* serializer block-escapes them, so no client-side ZWSP is needed);
* absent/empty/non-string -> no-op.
* parsing); col-0 markdown block triggers are neutralized so git-sync keeps them
* paragraphs; absent/empty/non-string -> no-op.
*/
describe("gitmostInsertTranscriptIntoEditor", () => {
const makeEditor = () =>
@@ -92,22 +91,19 @@ describe("gitmostInsertTranscriptIntoEditor", () => {
editor.destroy();
});
it("inserts col-0 markdown block triggers as verbatim paragraph text (no ZWSP workaround)", () => {
it("neutralizes col-0 markdown block triggers with a leading ZWSP (git-sync safety)", () => {
const editor = makeEditor();
// Trigger lines (some with a leaked indent) + a normal prefixed line. The
// git-sync serializer now block-escapes a leading trigger itself, so the
// bridge inserts each line's TEXT byte-exact (only the leaked indent is
// trimmed) — no invisible ZWSP is prepended anymore.
// Trigger lines (some with a leaked indent) + a normal prefixed line.
const inserted = gitmostInsertTranscriptIntoEditor(
editor,
[
"- dash",
" > quote", // leading indent is trimmed, text otherwise verbatim
" > quote", // leading indent must be trimmed then neutralized
"# hash",
"1. one",
"> [!info] note",
"```js",
"---",
"---", // solid thematic break -> horizontalRule (text-losing) if unneutralized
"***",
"___",
"You: normal line",
@@ -120,23 +116,20 @@ describe("gitmostInsertTranscriptIntoEditor", () => {
.map((n: any) => n.content?.[0]?.text)
.filter((t: any) => typeof t === "string") as string[];
// Each trigger line is stored as its own byte-exact text (indent trimmed);
// the git-sync round-trip keeps it a paragraph via the serializer's
// block-escape, so no ZWSP is needed here.
// Every block-trigger line is prefixed with the invisible ZWSP (indent
// trimmed first); the normal `You:` line is left byte-exact.
expect(texts).toEqual([
"- dash",
"> quote",
"# hash",
"1. one",
"> [!info] note",
"```js",
"---",
"***",
"___",
ZWSP + "- dash",
ZWSP + "> quote",
ZWSP + "# hash",
ZWSP + "1. one",
ZWSP + "> [!info] note",
ZWSP + "```js",
ZWSP + "---",
ZWSP + "***",
ZWSP + "___",
"You: normal line",
]);
// Guard: no invisible ZWSP leaked into any inserted line.
for (const t of texts) expect(t).not.toContain(ZWSP);
editor.destroy();
});
@@ -240,22 +240,45 @@ export async function gitmostUploadFileToEditor(
}
}
// Zero-width space (U+200B). Prepended to a transcript line that begins with a
// markdown BLOCK trigger: it is invisible in the rendered doc but shifts the
// trigger off column 0, so the git-sync doc->markdown->doc round-trip keeps the
// line a plain paragraph (see GITMOST_MD_BLOCK_TRIGGER_RE).
const GITMOST_ZWSP = "​";
// A markdown BLOCK-level construct that, sitting at column 0 of a paragraph
// line, the git-sync markdown serializer (packages/prosemirror-markdown
// markdown-converter.ts, `case "paragraph"`) would re-parse into a NON-paragraph
// block on the doc->markdown->doc cycle. That serializer emits paragraph text
// verbatim with NO block-escape (the pre-existing root cause), so a leading
// `#`/`-`/`*`/`+`/`>`, an ordered-list `N.`/`N)`, a code fence ```/~~~, a table
// `|`, or a `> [!info]` callout opener would silently become a heading / list /
// quote / code block / table / callout. The final alternative matches a WHOLE-
// LINE thematic break — solid `---`/`***`/`___` or spaced `- - -`/`_ _ _` (3+ of
// the same `-`/`*`/`_`) — which round-trips into a `horizontalRule`; because
// that node carries NO text, an un-neutralized separator line would LOSE its
// text entirely (worse than the list/quote case). This matches a TRIMMED line's
// start; the transcript's own `You:` / `Speaker N:` prefix begins with a letter
// and never matches, so prefixed lines are left byte-exact.
const GITMOST_MD_BLOCK_TRIGGER_RE =
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
// Append a transcript block BELOW the recording's audio node in a live editor:
// a "Transcript" heading followed by one paragraph per non-empty transcript
// line. The transcript is plain text, `\n`-separated, each line already
// formatted as `You: ...` / `Speaker N: ...` by the native host — line text is
// inserted as a TEXT node (never HTML/markdown), so there is no injection or
// mark-parsing surface. Each kept line is trimmed (drops an indent that would
// leak into the display). A line that begins with a col-0 markdown block
// trigger (`#`/`-`/`>`/`1.`/fence/`---`/…) needs no client-side workaround: the
// git-sync serializer (packages/prosemirror-markdown, `case "paragraph"`) now
// block-escapes such a leading trigger, so the doc->markdown->doc round-trip
// keeps the line a paragraph on its own — the former invisible-ZWSP defense is
// gone. This is best-effort and meant to run AFTER the audio has already been
// inserted; the caller must guard against a throw so a transcript failure never
// fails the (already successful) recording. Returns true when a block was
// inserted, false when there was nothing to insert (transcript
// undefined/empty/not-a-string). A non-string value is a no-op, not an error.
// both leak into the display and, at col 0, form a markdown block trigger) and,
// if it still begins with a col-0 markdown block trigger, gets an invisible
// zero-width space prepended so the git-sync round-trip cannot turn it into a
// list/quote/heading/callout/code/table (defensive boundary against the
// serializer's missing block-escape). This is best-effort and meant to run
// AFTER the audio has already been inserted; the caller must guard against a
// throw so a transcript failure never fails the (already successful) recording.
// Returns true when a block was inserted, false when there was nothing to
// insert (transcript undefined/empty/not-a-string). A non-string value is a
// no-op, not an error.
export function gitmostInsertTranscriptIntoEditor(
editor: Editor,
transcript: unknown,
@@ -265,7 +288,13 @@ export function gitmostInsertTranscriptIntoEditor(
.split("\n")
// Trim each line and drop blank (whitespace-only) ones.
.map((line) => line.trim())
.filter((line) => line.length > 0);
.filter((line) => line.length > 0)
// Neutralize a col-0 markdown block trigger with an invisible ZWSP so the
// git-sync round-trip keeps the line a paragraph. Host lines (`You:` /
// `Speaker N:`) never match and stay byte-exact.
.map((line) =>
GITMOST_MD_BLOCK_TRIGGER_RE.test(line) ? GITMOST_ZWSP + line : line,
);
if (lines.length === 0) return false;
const content = [
@@ -281,12 +281,10 @@ const SpaceTree = forwardRef<SpaceTreeApi, SpaceTreeProps>(function SpaceTree(
setOpenTreeNodes((prev) => ({ ...prev, [id]: isOpen }));
if (isOpen) {
const node = treeModel.find(data, id) as SpaceTreeNode | null;
// Same "unloaded branch" predicate the realtime insert paths use
// (`isUnloadedBranch`) so the lazy-load gate and the realtime inserts
// (`insertByPosition` / `placeByPosition`) can never disagree about what
// counts as unloaded (#525). Note: local raw `insert` (DnD/create-page)
// does not yet route through it — see #525 follow-up.
if (treeModel.isUnloadedBranch(node)) {
if (
node?.hasChildren &&
(!node.children || node.children.length === 0)
) {
const fetched = await fetchAllAncestorChildren({
pageId: id,
spaceId: node.spaceId,
@@ -74,48 +74,6 @@ describe("treeModel.isDescendant", () => {
});
});
// #525: the single "is this branch unloaded?" predicate shared by the lazy-load
// gate and the insert paths. Unloaded == server says hasChildren but none are
// present locally (canonical form `children: []`, also `undefined`). A parent
// without hasChildren is genuinely empty, not unloaded.
describe("treeModel.isUnloadedBranch", () => {
type PH = TreeNode<{ name: string; hasChildren?: boolean }>;
it("true for hasChildren + empty array (canonical unloaded form)", () => {
const n: PH = { id: "p", name: "P", hasChildren: true, children: [] };
expect(treeModel.isUnloadedBranch(n)).toBe(true);
});
it("true for hasChildren + undefined children", () => {
const n: PH = { id: "p", name: "P", hasChildren: true };
expect(treeModel.isUnloadedBranch(n)).toBe(true);
});
it("false for hasChildren + already-loaded children", () => {
const n: PH = {
id: "p",
name: "P",
hasChildren: true,
children: [{ id: "c", name: "C" }],
};
expect(treeModel.isUnloadedBranch(n)).toBe(false);
});
it("false for a genuinely-empty parent (no hasChildren)", () => {
expect(
treeModel.isUnloadedBranch({
id: "p",
name: "P",
hasChildren: false,
children: [],
} as PH),
).toBe(false);
expect(
treeModel.isUnloadedBranch({ id: "p", name: "P" } as PH),
).toBe(false);
});
it("false for null/undefined", () => {
expect(treeModel.isUnloadedBranch(null)).toBe(false);
expect(treeModel.isUnloadedBranch(undefined)).toBe(false);
});
});
describe("treeModel.visible", () => {
it("returns only root nodes when no openIds", () => {
const v = treeModel.visible(fixture, new Set());
@@ -239,64 +197,43 @@ describe("treeModel.insertByPosition", () => {
]);
});
type PH = TreeNode<{
name: string;
position?: string;
hasChildren?: boolean;
}>;
// #159 #1 / #525: inserting/moving a node under an UNLOADED parent must NOT
// materialize a partial `[node]` list — that would defeat the lazy-load gate and
// hide the parent's other real children. The canonical unloaded form here is
// `children: []` + `hasChildren: true` (from `pageToTreeNode` /
// `pruneCollapsedChildren`), which the pre-#525 `=== undefined` guard MISSED.
// The node is left to be lazy-loaded; the chevron stays enabled.
it("does NOT materialize a child under an UNLOADED parent (children: [], hasChildren: true)", () => {
const tree: PH[] = [
{ id: "p", name: "P", position: "a0", hasChildren: true, children: [] },
];
const node: PH = { id: "x", name: "X", position: "a1" };
const t = treeModel.insertByPosition(tree, "p", node);
const parent = treeModel.find(t, "p");
// The node was NOT inserted (children stay unloaded -> lazy-load fetches the
// full set, including this node, on expand). MUTATION: the pre-#525 predicate
// `children === undefined` does not fire for `[]`, so it would insert `[x]`
// here and reredden this expectation.
expect(parent?.children).toEqual([]);
expect(treeModel.find(t, "x")).toBeNull();
// ...and the chevron stays enabled so the user can expand to load it.
expect((parent as PH).hasChildren).toBe(true);
});
it("does NOT materialize a child under an UNLOADED parent (children undefined, hasChildren: true)", () => {
const tree: PH[] = [
{ id: "p", name: "P", position: "a0", hasChildren: true }, // children: undefined
];
const node: PH = { id: "x", name: "X", position: "a1" };
const t = treeModel.insertByPosition(tree, "p", node);
const parent = treeModel.find(t, "p");
expect(parent?.children).toBeUndefined();
expect(treeModel.find(t, "x")).toBeNull();
expect((parent as PH).hasChildren).toBe(true);
});
it("DOES insert under a genuinely-empty parent (children: [], hasChildren: false)", () => {
const tree: PH[] = [
{ id: "p", name: "P", position: "a0", hasChildren: false, children: [] },
];
const node: PH = { id: "x", name: "X", position: "a1" };
const t = treeModel.insertByPosition(tree, "p", node);
// No server children (`hasChildren: false`), so materializing the first child
// is correct — nothing is hidden.
expect(treeModel.find(t, "p")?.children?.map((n) => n.id)).toEqual(["x"]);
});
it("DOES insert under a genuinely-empty parent (children undefined, hasChildren: false)", () => {
// #159 #1: inserting/moving a node under a parent whose children are NOT
// loaded (`children === undefined`, e.g. a collapsed page) must NOT materialize
// a partial `[node]` list — that would defeat the lazy-load gate and hide the
// parent's other real children. The node is left to be lazy-loaded; only
// `hasChildren` is flagged so the chevron appears.
it("does NOT materialize a child under an UNLOADED parent (children undefined)", () => {
type PH = TreeNode<{
name: string;
position?: string;
hasChildren?: boolean;
}>;
const tree: PH[] = [
{ id: "p", name: "P", position: "a0", hasChildren: false }, // children: undefined
];
const node: PH = { id: "x", name: "X", position: "a1" };
const t = treeModel.insertByPosition(tree, "p", node);
const parent = treeModel.find(t, "p");
// The node was NOT inserted (children stay unloaded -> lazy-load fetches the
// full set, including this node, on expand).
expect(parent?.children).toBeUndefined();
expect(treeModel.find(t, "x")).toBeNull();
// ...but the chevron is enabled so the user can expand to load it.
expect((parent as PH).hasChildren).toBe(true);
});
it("DOES insert under a LOADED-but-empty parent (children: [])", () => {
type PH = TreeNode<{
name: string;
position?: string;
hasChildren?: boolean;
}>;
const tree: PH[] = [
{ id: "p", name: "P", position: "a0", hasChildren: false, children: [] },
];
const node: PH = { id: "x", name: "X", position: "a1" };
const t = treeModel.insertByPosition(tree, "p", node);
// A loaded (empty) child list is complete, so the node IS inserted.
expect(treeModel.find(t, "p")?.children?.map((n) => n.id)).toEqual(["x"]);
});
@@ -43,26 +43,6 @@ export const treeModel = {
};
},
// A branch is "unloaded" when the server says it HAS children (`hasChildren`)
// but none are present locally. The canonical unloaded form in this codebase
// is `children: []` (produced by `pageToTreeNode` and by `pruneCollapsedChildren`
// resetting collapsed branches), NOT `children: undefined` — so a predicate that
// only checks `=== undefined` misses the real case and materializes a misleading
// partial list (#525). This is the SINGLE source of truth for "should a
// fetch/materialize be deferred?", shared by the lazy-load gate (`handleToggle`)
// and the realtime insert paths (`insertByPosition` / `placeByPosition`), so they
// can never drift apart again. (The local raw `insert` primitive and its DnD/
// create-page callers do NOT yet route through this predicate — see #525
// follow-up.) A parent WITHOUT `hasChildren` is genuinely empty
// (no server children) — inserting its first child is correct, not deferred.
isUnloadedBranch<T extends object>(
node: TreeNode<T> | null | undefined,
): boolean {
if (!node) return false;
const hasChildren = (node as { hasChildren?: boolean }).hasChildren === true;
return hasChildren && (node.children == null || node.children.length === 0);
},
isDescendant<T extends object>(
tree: TreeNode<T>[],
ancestorId: string,
@@ -147,15 +127,14 @@ export const treeModel = {
}
const parent = treeModel.find(tree, parentId);
// The parent is in the tree but its children have NOT been lazy-loaded yet
// (`hasChildren` set + children absent/empty — see `isUnloadedBranch`; the
// canonical unloaded form is `children: []`, NOT just `undefined`). Inserting
// (`children === undefined`, distinct from a loaded-but-empty `[]`). Inserting
// here would MATERIALIZE a misleading partial child list (`[node]`) that
// defeats the lazy-load gate — which fetches only when children are
// absent/empty — so the parent's OTHER real children would never load and the
// moved/added node would be the only one shown (a silent data loss, #159 #1).
// Instead, leave the children unloaded and just flag `hasChildren` so the
// chevron appears; expanding fetches the FULL set (including this node).
if (parent && treeModel.isUnloadedBranch(parent)) {
if (parent && parent.children === undefined) {
return treeModel.update(
tree,
parentId,
@@ -13,7 +13,8 @@ let currentAlias: IShareAlias | null = null;
let availabilityResult: {
valid: boolean;
available: boolean;
} = { valid: true, available: true };
currentPageId: string | null;
} = { valid: true, available: true, currentPageId: null };
vi.mock("@/features/share/queries/share-query.ts", () => ({
useShareAliasForPageQuery: () => ({ data: currentAlias }),
@@ -55,7 +56,7 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
beforeEach(() => {
setMutateAsync.mockReset();
currentAlias = null;
availabilityResult = { valid: true, available: true };
availabilityResult = { valid: true, available: true, currentPageId: null };
});
it("shows a 'will move it here' HINT (not a terminal error) when the name belongs to another page, and keeps Save enabled", async () => {
@@ -64,6 +65,7 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
availabilityResult = {
valid: true,
available: false,
currentPageId: "page-X",
};
renderSection("page-Y");
@@ -95,6 +97,7 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
availabilityResult = {
valid: true,
available: false,
currentPageId: "page-X",
};
// The server rejects the un-confirmed save asking the client to confirm.
setMutateAsync.mockRejectedValueOnce({
@@ -103,6 +106,7 @@ describe("ShareAliasSection — taken-name handling is never a dead end", () =>
status: 409,
data: {
code: "ALIAS_REASSIGN_REQUIRED",
currentPageId: "page-X",
currentPageTitle: "Alias Test Page X",
},
},
@@ -48,6 +48,7 @@ export default function ShareAliasSection({
const [availability, setAvailability] = useState<{
valid: boolean;
available: boolean;
currentPageId: string | null;
} | null>(null);
const [reassign, setReassign] = useState<{
alias: string;
@@ -75,6 +76,7 @@ export default function ShareAliasSection({
setAvailability({
valid: res.valid,
available: res.available,
currentPageId: res.currentPageId,
});
} catch {
setAvailability(null);
@@ -108,6 +108,7 @@ export interface IShareAliasAvailability {
alias: string;
valid: boolean;
available: boolean;
currentPageId: string | null;
}
export interface ISharedPageTree {
@@ -82,19 +82,17 @@ describe("applyMoveTreeNode", () => {
]);
});
it("does NOT create a partial child list when the destination is loaded-but-collapsed (children unloaded) — keeps it lazy-loadable (#159 #525)", () => {
// `dstCollapsed` is in the tree but its children were never lazy-loaded. The
// CANONICAL unloaded form here is `hasChildren: true` + `children: []` (from
// `pageToTreeNode` / `pruneCollapsedChildren`), NOT `children: undefined`.
// The pre-#525 predicate (`children === undefined`) missed this form and
// inserted `src` as the ONLY child ([src]), defeating the lazy-load gate and
// HIDING the parent's other real children. Now the move leaves children
// unloaded (so expanding fetches the FULL set, including src).
it("does NOT create a partial child list when the destination is loaded-but-collapsed (children unloaded) — keeps it lazy-loadable (#159)", () => {
// `dstCollapsed` is in the tree but its children were never lazy-loaded
// (children === undefined). The OLD behavior inserted `src` as the ONLY
// child ([src]), which defeated the lazy-load gate and HID the parent's
// other real children. Now the move leaves children unloaded (so expanding
// fetches the FULL set, including src) and just flags hasChildren.
const tree: SpaceTreeNode[] = [
node("dstCollapsed", {
position: "a0",
hasChildren: true,
children: [],
hasChildren: false,
children: undefined as unknown as SpaceTreeNode[],
}),
node("src", { position: "a9" }),
];
@@ -107,10 +105,9 @@ describe("applyMoveTreeNode", () => {
pageData: {},
});
const dst = treeModel.find(next, "dstCollapsed");
// Children stay unloaded ([] not materialized to [src]) -> the lazy-load gate
// fetches the FULL set (incl. src) on expand. MUTATION: the pre-#525
// `=== undefined` predicate would insert [src] here and redden this.
expect(dst?.children).toEqual([]);
// Children stay unloaded -> the lazy-load gate fetches the FULL set (incl.
// src) on expand, rather than showing a misleading partial [src] list.
expect(dst?.children).toBeUndefined();
expect(dst?.hasChildren).toBe(true);
// src moved away from its old root slot (it lives under dstCollapsed
// server-side and reappears when the parent is expanded/loaded).
@@ -28,7 +28,6 @@ import {
IAiMcpServerCreate,
IAiMcpServerUpdate,
} from "@/features/workspace/services/ai-mcp-server-service.ts";
import { resolveToolAllowlist } from "./ai-mcp-server-form.utils.ts";
const formSchema = z.object({
name: z.string().min(1),
@@ -122,20 +121,13 @@ export default function AiMcpServerForm({
async function handleSubmit(values: FormValues) {
const headers = resolveHeaders();
// An empty tag field means "no restriction" (sent as null) — since #476 the
// server persists a literal `[]` as deny-all (zero tools). But a server that
// was ALREADY deny-all loads into an empty field too; sending null there
// would silently widen it to allow-all on a routine edit, so preserve `[]`.
// See resolveToolAllowlist for the full rationale.
const toolAllowlist = resolveToolAllowlist(values.toolAllowlist, server);
if (isEdit && server) {
const payload: IAiMcpServerUpdate = {
id: server.id,
name: values.name,
transport: values.transport,
url: values.url,
toolAllowlist,
toolAllowlist: values.toolAllowlist,
// Always sent: a blank value clears the stored guidance (server -> null).
instructions: values.instructions,
enabled: values.enabled,
@@ -148,7 +140,7 @@ export default function AiMcpServerForm({
name: values.name,
transport: values.transport,
url: values.url,
toolAllowlist,
toolAllowlist: values.toolAllowlist,
// Blank => server stores null (no guidance).
instructions: values.instructions,
enabled: values.enabled,
@@ -1,29 +0,0 @@
import { describe, it, expect } from "vitest";
import { resolveToolAllowlist } from "./ai-mcp-server-form.utils.ts";
describe("resolveToolAllowlist", () => {
it("sends the typed tools when the field is non-empty", () => {
expect(resolveToolAllowlist(["a", "b"], { toolAllowlist: null })).toEqual([
"a",
"b",
]);
});
it("creates as null (unrestricted) when empty and there is no server", () => {
expect(resolveToolAllowlist([], undefined)).toBeNull();
});
it("sends null for an empty field on a previously-unrestricted server", () => {
expect(resolveToolAllowlist([], { toolAllowlist: null })).toBeNull();
});
it("preserves deny-all: an empty field on a `[]` server stays `[]`, not null", () => {
// The core #476/#477 guard: editing a deny-all server (rename/toggle) with
// an empty tag field must NOT silently widen it to allow-all.
expect(resolveToolAllowlist([], { toolAllowlist: [] })).toEqual([]);
});
it("still sends explicit tools even if the server was deny-all", () => {
expect(resolveToolAllowlist(["x"], { toolAllowlist: [] })).toEqual(["x"]);
});
});
@@ -1,22 +0,0 @@
import { IAiMcpServer } from "@/features/workspace/services/ai-mcp-server-service.ts";
// Resolve the tool allowlist value to persist from the form field.
//
// An empty tag field normally means "no restriction" and is sent as null so
// the server drops the column (all tools allowed). But a server that was
// ALREADY deny-all (a stored literal `[]`, meaning zero tools — creatable via
// the API) loads into the form as an empty field too. Coercing that empty
// field to null on submit would SILENTLY widen a deny-all server to allow-all
// on any routine edit (rename, toggle) — the exact silent-widen class #476
// closed on the read side. So when the edited server was deny-all, preserve
// `[]` (deny-all); only a genuinely-unrestricted server (stored null/absent)
// stays null.
export function resolveToolAllowlist(
fieldValue: string[],
server?: Pick<IAiMcpServer, "toolAllowlist">,
): string[] | null {
if (fieldValue.length > 0) return fieldValue;
const wasDenyAll =
Array.isArray(server?.toolAllowlist) && server.toolAllowlist.length === 0;
return wasDenyAll ? [] : null;
}
@@ -6,8 +6,6 @@ import {
nextReindexPollInterval,
isReindexComplete,
isReindexButtonLoading,
reindexRunKey,
isNewReindexRun,
} from './ai-provider-settings';
describe('resolveCardStatus', () => {
@@ -223,128 +221,6 @@ describe('isReindexComplete', () => {
});
});
describe('reindexRunKey', () => {
it('is null when the status carries no run identity', () => {
expect(reindexRunKey(undefined)).toBeNull();
expect(
reindexRunKey({ reindexing: false, indexedPages: 5, totalPages: 5 }),
).toBeNull();
});
it('is null for a legacy/degraded record with an empty runId', () => {
// The server sends runId='' for a record written before the field existed;
// the client must treat that as "no identity" (fall back to prior behaviour).
expect(
reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: '',
reindexStartedAt: 1000,
}),
).toBeNull();
});
it('folds runId and startedAt into one stable key', () => {
expect(
reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
}),
).toBe('run-a:1000');
});
it('changes when the runId changes for the same startedAt', () => {
const a = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
});
const b = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-b',
reindexStartedAt: 1000,
});
expect(a).not.toBe(b);
});
it('changes when the same runId restarts at a new startedAt', () => {
const a = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 1000,
});
const b = reindexRunKey({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId: 'run-a',
reindexStartedAt: 2000,
});
expect(a).not.toBe(b);
});
});
describe('isNewReindexRun (poll keying on runId)', () => {
// Derive the status shape from the helper itself so the test needs no export
// of the component-internal ReindexStatus type.
type ReindexStatusLike = NonNullable<Parameters<typeof reindexRunKey>[0]>;
const run = (runId: string, startedAt: number): ReindexStatusLike => ({
reindexing: true,
indexedPages: 0,
totalPages: 10,
runId,
reindexStartedAt: startedAt,
});
it('first identity after none latched is a NEW run', () => {
expect(isNewReindexRun(null, run('run-a', 1000))).toBe(true);
});
it('the SAME identity is not a new run (same run being watched)', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(isNewReindexRun(key, run('run-a', 1000))).toBe(false);
});
it('a DIFFERENT runId is a new run (reset per-run poll state)', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(isNewReindexRun(key, run('run-b', 1000))).toBe(true);
});
it('an identity-less poll (no runId / cleared record) is never a new run', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(
isNewReindexRun(key, {
reindexing: false,
indexedPages: 10,
totalPages: 10,
}),
).toBe(false);
});
it('a legacy empty-runId poll does not spuriously reset a latched run', () => {
const key = reindexRunKey(run('run-a', 1000));
expect(
isNewReindexRun(key, {
reindexing: true,
indexedPages: 3,
totalPages: 10,
runId: '',
reindexStartedAt: 1000,
}),
).toBe(false);
});
});
describe('isReindexButtonLoading', () => {
it('loads while the POST mutation is pending', () => {
expect(
@@ -173,43 +173,9 @@ export function resolveKeyField(
// Subset of the status payload that drives the reindex poll decisions.
type ReindexStatus = Pick<
IAiSettings,
"reindexing" | "indexedPages" | "totalPages" | "runId" | "reindexStartedAt"
"reindexing" | "indexedPages" | "totalPages"
>;
/**
* A stable per-RUN key for the reindex poll: `runId:startedAt`, or `null` when
* the status carries no run identity (no active run, or a legacy/degraded
* server record with an empty runId). Two polls of the SAME run share a key; a
* new run mints a fresh runId and so a different key.
*
* This is the single place the client turns the server's run identity into the
* value it keys on it removes the "is this the same run I've been watching or
* a brand-new one?" ambiguity that made a class of reindex-status bugs (a stale
* pre-reindex snapshot vs a fresh run) get fixed twice (#262). `startedAt` is
* folded in so a run that somehow reuses a runId but restarted is still new.
*/
export function reindexRunKey(status: ReindexStatus | undefined): string | null {
const runId = status?.runId;
if (!runId) return null;
return `${runId}:${status?.reindexStartedAt ?? ""}`;
}
/**
* Decide whether the latest poll represents a NEW reindex run relative to the
* run key the client last latched (`prevKey`, `null` if none yet). True only
* when the status carries an identity AND it differs from the latched one the
* signal to reset any per-run poll state (the "seen active" latch / progress the
* UI held). The same identity (or no identity) is NOT a new run, so an unchanged
* or identity-less poll never resets mid-run.
*/
export function isNewReindexRun(
prevKey: string | null,
status: ReindexStatus | undefined,
): boolean {
const key = reindexRunKey(status);
return key !== null && key !== prevKey;
}
/**
* Decide the TanStack Query `refetchInterval` while a reindex may be running.
* Returns the poll interval (ms) to keep polling, or `false` to stop.
@@ -354,13 +320,6 @@ export default function AiProviderSettings() {
// counter at 0 until a manual reload. A ref (not state) because it must not
// trigger a render and is only ever read where `reindexing` is already false.
const reindexSeenActiveRef = useRef(false);
// The run identity (runId:startedAt) the current poll window is keyed on. When
// a poll reports a DIFFERENT runId the server has started a NEW run, so we
// re-latch to it and reset `reindexSeenActiveRef` — a fresh run must never
// inherit the previous run's "seen active"/completion state (which would stop
// polling immediately or read the old run's counters as this run's). null =
// no run keyed yet (steady state, or a legacy record without a runId).
const reindexRunKeyRef = useRef<string | null>(null);
// Only admins may read the (masked) AI settings; the server enforces this too.
const { data: settings, isLoading } = useAiSettingsQuery(isAdmin, (query) =>
@@ -377,14 +336,6 @@ export default function AiProviderSettings() {
// unmount because the deadline state goes away with the component.
useEffect(() => {
if (reindexDeadline === null) return;
// Key the poll on the run identity: if this poll carries a runId different
// from the one we latched, the server started a NEW run, so adopt it and
// drop the per-run "seen active" latch (a fresh run must not inherit the
// previous run's completion state). Same runId => same run, leave it alone.
if (isNewReindexRun(reindexRunKeyRef.current, settings)) {
reindexRunKeyRef.current = reindexRunKey(settings);
reindexSeenActiveRef.current = false;
}
// Latch "we have seen the active run" the moment a poll reports it, so the
// completion check below (and the refetchInterval's) only fires once the run
// has genuinely started — never on the stale pre-reindex snapshot.
@@ -1269,10 +1220,6 @@ export default function AiProviderSettings() {
// immediately.
onSuccess: () => {
reindexSeenActiveRef.current = false;
// Forget the previous run's identity so the first poll of
// this window (carrying the new run's runId) is recognized
// as a new run and keyed afresh.
reindexRunKeyRef.current = null;
setReindexDeadline(Date.now() + REINDEX_POLL_CAP_MS);
},
})
@@ -27,9 +27,7 @@ export interface IAiMcpServerCreate {
// Auth headers map (e.g. { Authorization: 'Bearer ...' }). Encrypted on save;
// never returned.
headers?: Record<string, string>;
// Omit/null => no restriction; `[]` is persisted verbatim and means
// deny-all (zero tools) since #476.
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Blank => stored as null.
instructions?: string;
enabled?: boolean;
@@ -45,9 +43,7 @@ export interface IAiMcpServerUpdate {
transport?: McpTransport;
url?: string;
headers?: Record<string, string>;
// Absent => unchanged; null => no restriction; `[]` is persisted verbatim
// and means deny-all (zero tools) since #476.
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Absent => unchanged; blank => cleared.
instructions?: string;
enabled?: boolean;
@@ -51,14 +51,6 @@ export interface IAiSettings {
// True while a full workspace reindex is actively running; the counts above
// then reflect the live run progress (done climbs 0 -> total).
reindexing?: boolean;
// Identity of the ACTIVE reindex run (present only while `reindexing`). The
// poll keys on `runId`: a changed value means a NEW run (reset the per-run
// poll state the UI latched), the same value is the run already being watched.
// Absent/empty ('') => no identity available; the client keeps prior behaviour.
runId?: string;
// Epoch-ms the active run started; paired with `runId` so a restart with a
// recycled id is still detected as a new run.
reindexStartedAt?: number;
}
// Update payload. Key semantics (same for `apiKey` and `embeddingApiKey`):
@@ -1,5 +1,5 @@
import { describe, it, expect } from "vitest";
import { templateRoute, KNOWN_ROUTE_TEMPLATES } from "./route-template";
import { templateRoute } from "./route-template";
describe("templateRoute", () => {
it("templates a space page path (never leaks slugs)", () => {
@@ -32,30 +32,4 @@ describe("templateRoute", () => {
expect(templateRoute("/weird/unknown/thing")).toBe("other");
expect(templateRoute("/s/team/p/slug/extra/segments")).toBe("other");
});
// The server's /api/telemetry/vitals mirror (ALLOWED_ROUTE_TEMPLATES) drops any
// route outside KNOWN_ROUTE_TEMPLATES, so templateRoute must NEVER emit a label
// that is not in that dictionary — otherwise legit client metrics get dropped.
it("only ever emits labels contained in KNOWN_ROUTE_TEMPLATES (#495)", () => {
const samples = [
"/",
"/home",
"/settings/members",
"/settings/groups/g-1",
"/s/team",
"/s/team/trash",
"/s/team/p/slug",
"/p/slug",
"/share/abc",
"/share/abc/p/slug",
"/share/p/slug",
"/labels/urgent",
"/invites/inv-1",
"/weird/unknown/thing", // -> "other"
"/deep/unmatched/x/y/z", // -> "other"
];
for (const path of samples) {
expect(KNOWN_ROUTE_TEMPLATES.has(templateRoute(path))).toBe(true);
}
});
});
@@ -44,22 +44,6 @@ const STATIC_ROUTES = new Set<string>([
'/settings/sharing',
]);
/**
* The COMPLETE, finite vocabulary `templateRoute` can ever emit: the two
* synthetic labels (`/` and `other`), the static routes, and the dynamic
* templates. Exported so the public `/api/telemetry/vitals` endpoint can reject
* any `route` outside this dictionary server-side (the endpoint is anonymous, so
* an un-checked `route` is a free-text write surface). The server keeps a mirror
* (`ALLOWED_ROUTE_TEMPLATES` in client-metrics.constants.ts) this is the
* canonical source; keep them in lockstep.
*/
export const KNOWN_ROUTE_TEMPLATES: ReadonlySet<string> = new Set<string>([
'/',
'other',
...STATIC_ROUTES,
...ROUTE_PATTERNS.map((p) => p.template),
]);
export function templateRoute(pathname: string): string {
// Normalise a trailing slash (except root).
const path =
+1 -10
View File
@@ -3,7 +3,6 @@ import "@mantine/spotlight/styles.css";
import "@mantine/notifications/styles.css";
import '@mantine/dates/styles.css';
import "@/styles/a11y-overrides.css";
import "@/styles/notification-overrides.css";
import ReactDOM from "react-dom/client";
import App from "./App.tsx";
@@ -48,15 +47,7 @@ function renderApp() {
<MantineProvider theme={theme} cssVariablesResolver={mantineCssResolver}>
<ModalsProvider>
<QueryClientProvider client={queryClient}>
{/* top-center: toasts sit in the top of the viewport, in the line
of sight, and no longer cover centered content (e.g. "Load
more"). The below-chrome vertical offset is applied via a
position-scoped CSS rule in notification-overrides.css (NOT an
inline `style`): Mantine renders all six position containers at
once and an inline root style would land on every one, giving the
bottom-* containers both top+bottom full-viewport transparent
overlays that swallow clicks. */}
<Notifications position="top-center" limit={3} zIndex={10000} />
<Notifications position="bottom-center" limit={3} zIndex={10000} />
<HelmetProvider>
{/* Root boundary above every lazy route's Suspense: a stale-chunk
404 after a deploy is caught and recovered here instead of
@@ -1,64 +0,0 @@
/*
* Toast (Mantine Notification) visibility overrides.
* Mantine renders colorless toasts on --mantine-color-body (== the page
* background: white in light mode) with a faint shadow, so on white pages the
* card has no visible edge. These rules give every toast a type-tinted
* background, a WCAG-checked border and a stronger shadow so it separates from
* the page. The [data-mantine-color-scheme] + static-class selector (0,2,0)
* beats Mantine's own (0,1,0) rules regardless of stylesheet order (Mantine's
* bg/border rules wrap the scheme attribute in :where(), so they stay (0,1,0)).
* --notification-color is defined on the same element (defaults to primary,
* set per `color` prop), so tint/border follow the toast type. This also covers
* the loading/import toast (no accent bar, since the spinner takes the icon
* slot): its visibility comes from tone + border + shadow + the colored spinner.
*/
/*
* Push the top-anchored toast containers below the top chrome (fixed 45px
* header + optional 45px format toolbar + ~6px gap) so a toast (z-index 10000)
* neither covers nor intercepts clicks on the header/toolbar (both z-index 99).
*
* Scoped to [data-position^='top'] on purpose. Mantine renders ALL SIX position
* containers simultaneously (`position` only routes toasts into one via the
* store); the root `style` prop would be applied to every one of them by
* getStyles("root"). A blanket `top` would land on the bottom-* containers too
* (which carry `bottom:16px`) position:fixed + both edges + height:auto makes
* them stretch the full viewport height, and the container root has neither
* pointer-events:none nor a background, so those transparent z-10000 overlays
* would swallow clicks across the whole page. Restricting to top-* leaves the
* bottom containers at height:0.
*
* Specificity: `.mantine-Notifications-root[data-position^='top']` is (0,2,0)
* (class + attribute) and beats Mantine's own top rule
* `.m_b37d9ac7:where([data-position='top-center']){top:16px}` which is (0,1,0)
* (the :where() contributes 0), regardless of stylesheet order.
*/
.mantine-Notifications-root[data-position^='top'] {
top: 96px;
}
[data-mantine-color-scheme='light'] .mantine-Notification-root {
/* ~10% type color over white: clearly off-white, text contrast preserved */
background-color: color-mix(in srgb, var(--notification-color) 10%, var(--mantine-color-white));
/* Border must clear WCAG 3:1 non-text contrast on white. The repo rejects
gray-4 for this (a11y-overrides.css); gray-6 base (~3.32:1) darkened by the
type color stays >= 3:1. */
border: 1px solid color-mix(in srgb, var(--notification-color) 45%, var(--mantine-color-gray-6));
box-shadow: var(--mantine-shadow-xl);
}
[data-mantine-color-scheme='dark'] .mantine-Notification-root {
/* Dark page (dark-7/8) vs toast (dark-6) already separate a little; border +
shadow carry the type cue here (a 7% dark tint was near-invisible). */
background-color: color-mix(in srgb, var(--notification-color) 14%, var(--mantine-color-dark-6));
border: 1px solid color-mix(in srgb, var(--notification-color) 45%, var(--mantine-color-dark-3));
box-shadow: var(--mantine-shadow-xl);
}
/* Mantine's message-with-title color is gray-6 (#868e96, already only ~3.32:1
on white below AA 4.5:1); the new tint pushes it lower. Bump to gray-7 to
keep multi-line colored toasts readable, consistent with the repo's existing
WCAG tuning (theme.ts already bumps this same gray-6 up elsewhere). */
[data-mantine-color-scheme='light'] .mantine-Notification-description[data-with-title] {
color: var(--mantine-color-gray-7);
}
+1 -3
View File
@@ -23,7 +23,7 @@
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build && pnpm --filter @docmost/token-estimate build",
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build",
"test": "jest",
"test:int": "jest --config test/jest-integration.json",
"test:watch": "jest --watch",
@@ -44,7 +44,6 @@
"@docmost/mcp": "workspace:*",
"@docmost/pdf-inspector": "1.9.6",
"@docmost/prosemirror-markdown": "workspace:*",
"@docmost/token-estimate": "workspace:*",
"@fastify/compress": "^9.0.0",
"@fastify/cookie": "^11.0.2",
"@fastify/multipart": "^10.0.0",
@@ -207,7 +206,6 @@
"^@docmost/db/(.*)$": "<rootDir>/database/$1",
"^@docmost/transactional/(.*)$": "<rootDir>/integrations/transactional/$1",
"^@docmost/ee/(.*)$": "<rootDir>/ee/$1",
"^@docmost/token-estimate$": "<rootDir>/../../../packages/token-estimate/src/index.ts",
"^src/(.*)$": "<rootDir>/$1",
"^@tiptap/react$": "<rootDir>/../test/stubs/tiptap-react.js"
}
@@ -181,7 +181,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
{} as never, // pageAccess
{ isAiChatDeferredToolsEnabled: () => false, isAiChatFinalStepLockdownEnabled: () => false } as never, // environment
);
return { svc, aiChatMessageRepo };
return { svc };
}
const body = {
@@ -287,7 +287,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
// Drive stream() to the point streamText is called, capturing the options object
// (which carries onStepFinish/onFinish/onError/onAbort) and the run hooks.
async function captureStreamCallbacks() {
const { svc, aiChatMessageRepo } = makeService();
const { svc } = makeService();
let capturedOpts: any;
streamTextMock.mockImplementation((opts: any) => {
capturedOpts = opts;
@@ -314,7 +314,7 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
runHooks: runHooks as never,
});
expect(capturedOpts).toBeDefined();
return { capturedOpts, runHooks, aiChatMessageRepo };
return { capturedOpts, runHooks };
}
it('F9: onStepFinish bumps the run step count, onFinish settles the run "completed" (the dominant autonomous-run path)', async () => {
@@ -369,51 +369,6 @@ describe('AiChatService.stream — abortSignal wiring (#184 F3)', () => {
expect.stringContaining('provider exploded'),
);
});
// #490 reactive branch: a provider CONTEXT-OVERFLOW 400 in onError is classified,
// records a distinguishable cause, and stamps metadata.replayOverflow so the NEXT
// turn's budgeter trims aggressively (the recovery that un-bricks the chat).
it('#490: a context-overflow 400 stamps replayOverflow on the finalized row', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as never);
const { capturedOpts, aiChatMessageRepo } = await captureStreamCallbacks();
const overflow = Object.assign(new Error('too large'), {
statusCode: 400,
message:
"This model's maximum context length is 128000 tokens. However, your messages resulted in 214000 tokens. Please reduce the length.",
});
await capturedOpts.onError({ error: overflow });
// The seed row exists (finalizeOwner is the owner-write path).
expect(aiChatMessageRepo.finalizeOwner).toHaveBeenCalled();
const calls = aiChatMessageRepo.finalizeOwner.mock.calls as any[][];
const patch = calls[calls.length - 1][2] as {
status: string;
metadata: Record<string, unknown>;
};
expect(patch.status).toBe('error');
expect(patch.metadata.replayOverflow).toBe(true);
expect(patch.metadata.error).toContain('контекстное окно');
});
it('#490: a non-overflow error does NOT stamp replayOverflow', async () => {
jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined as never);
const { capturedOpts, aiChatMessageRepo } = await captureStreamCallbacks();
await capturedOpts.onError({ error: new Error('network reset') });
const calls = aiChatMessageRepo.finalizeOwner.mock.calls as any[][];
const patch = calls[calls.length - 1][2] as {
status: string;
metadata: Record<string, unknown>;
};
expect('replayOverflow' in patch.metadata).toBe(false);
});
});
/**
@@ -13,7 +13,6 @@ import {
compactToolOutput,
assistantParts,
serializeSteps,
type StepPartsCache,
rowToUiMessage,
prepareAgentStep,
stepBudgetWarning,
@@ -29,14 +28,10 @@ import {
FINAL_STEP_NUDGE,
STEP_LIMIT_NO_ANSWER_MARKER,
OUTPUT_DEGENERATION_ERROR,
lastAssistantContextTokens,
lastAssistantReplayOverflow,
seedActivatedTools,
} from './ai-chat.service';
import type { AiChatMessage, Workspace } from '@docmost/db/types/entity.types';
import { buildSystemPrompt } from './ai-chat.prompt';
import type { McpClientsService } from './external-mcp/mcp-clients.service';
import { resolveEffectiveReplayThreshold } from './history-budget';
/**
* Unit tests for compactToolOutput: the pure helper that shrinks tool outputs
@@ -119,54 +114,6 @@ describe('compactToolOutput', () => {
describe('assistantParts', () => {
type AnyPart = Record<string, unknown>;
// #490 memoization: assistantParts builds each step's parts once and caches
// them by the step OBJECT's identity, so a mid-stream flush does not
// re-stringify every prior step's (large) output. Observable property: with a
// shared cache, the second call over the SAME step object returns the cached
// (identical) part array even if the step's underlying output was swapped —
// proving the work was memoized, not redone.
it('memoizes a step by identity (shared cache => one build per step)', () => {
const cache: StepPartsCache = new WeakMap();
const step = {
text: 'x',
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: {} }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage', output: { v: 1 } }],
};
const first = assistantParts([step], '', cache) as AnyPart[];
expect((first.find((p) => p.type === 'tool-getPage')!.output as any).v).toBe(
1,
);
// Swap the output for a NEW value; a re-build would pick it up, a cache hit
// keeps the first result.
step.toolResults[0] = {
toolCallId: 'c1',
toolName: 'getPage',
output: { v: 2 },
};
const second = assistantParts([step], '', cache) as AnyPart[];
expect((second.find((p) => p.type === 'tool-getPage')!.output as any).v).toBe(
1,
);
// Same cached part objects are reused.
expect(second.find((p) => p.type === 'tool-getPage')).toBe(
first.find((p) => p.type === 'tool-getPage'),
);
});
it('without a cache, each call rebuilds (no stale memo)', () => {
const step = {
text: 'x',
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: {} }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage', output: { v: 1 } }],
};
const first = assistantParts([step], '') as AnyPart[];
step.toolResults[0].output = { v: 2 };
const second = assistantParts([step], '') as AnyPart[];
expect((second.find((p) => p.type === 'tool-getPage')!.output as any).v).toBe(
2,
);
});
it('emits output-available for a tool-call WITH a paired result', () => {
const steps = [
{
@@ -284,320 +231,61 @@ describe('assistantParts', () => {
});
});
// #490 trace format v2: per call the trace stores { input } for the call and an
// OUTCOME element — { ok: true } on success, { error, kind: 'thrown' } on a
// thrown tool-error, { error, kind: 'interrupted' } on a mid-step abort. The tool
// OUTPUT is no longer duplicated here (it lives once in metadata.parts).
describe('serializeSteps (trace v2)', () => {
describe('serializeSteps', () => {
it('returns null when there are no calls or results', () => {
expect(serializeSteps([])).toBeNull();
});
it('pairs a successful call with an { ok: true } outcome and NO output', () => {
it('flattens calls and results into a compact trace', () => {
const trace = serializeSteps([
{
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: { id: 'p1' } }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage' }],
toolCalls: [{ toolName: 'getPage', input: { id: 'p1' } }],
toolResults: [{ toolName: 'getPage', output: { title: 'T' } }],
},
]) as Array<Record<string, unknown>>;
expect(trace).toHaveLength(2);
expect(trace[0]).toEqual({ toolName: 'getPage', input: { id: 'p1' } });
expect(trace[1]).toEqual({ toolName: 'getPage', ok: true });
// The output is NOT stored in the trace any more (dedup: it lives in parts).
expect(trace.some((e) => 'output' in e)).toBe(false);
expect(trace[1]).toEqual({ toolName: 'getPage', output: { title: 'T' } });
});
it('records a THROWN failure with { error, kind: "thrown" }', () => {
it('records a THROWN tool failure (tool-error part) with its error message', () => {
const trace = serializeSteps([
{
toolCalls: [
{ toolCallId: 'c1', toolName: 'editPageText', input: { id: 'p1' } },
],
toolCalls: [{ toolName: 'editPageText', input: { id: 'p1' } }],
toolResults: [],
content: [
{
type: 'tool-error',
toolCallId: 'c1',
toolName: 'editPageText',
error: new Error('page is locked'),
},
],
},
]) as Array<Record<string, unknown>>;
// The call element is followed by a paired error element (mirroring how a
// successful result is appended), so the failure survives in the trace.
expect(trace).toHaveLength(2);
expect(trace[0]).toEqual({ toolName: 'editPageText', input: { id: 'p1' } });
expect(trace[1]).toEqual({
toolName: 'editPageText',
error: 'page is locked',
kind: 'thrown',
});
});
it('marks an interrupted call (no result, no throw) with kind "interrupted"', () => {
const trace = serializeSteps([
{
toolCalls: [
{ toolCallId: 'c1', toolName: 'createComment', input: { x: 1 } },
],
toolResults: [],
content: [],
},
]) as Array<Record<string, unknown>>;
expect(trace).toHaveLength(2);
expect(trace[1]).toEqual({
toolName: 'createComment',
error: 'Tool call did not complete.',
kind: 'interrupted',
});
// Structurally distinct from a thrown hard-fail so it never inflates an
// error-rate scan.
expect((trace[1] as { kind: string }).kind).not.toBe('thrown');
});
it('truncates a very long thrown-error message to the tool-output limit', () => {
it('truncates a very long tool-error message to the tool-output limit', () => {
const long = 'x'.repeat(5000);
const trace = serializeSteps([
{
toolCalls: [{ toolCallId: 'c1', toolName: 'editPageText', input: {} }],
toolCalls: [{ toolName: 'editPageText', input: {} }],
toolResults: [],
content: [
{
type: 'tool-error',
toolCallId: 'c1',
toolName: 'editPageText',
error: long,
},
],
content: [{ type: 'tool-error', toolName: 'editPageText', error: long }],
},
]) as Array<Record<string, unknown>>;
const errorText = trace[1].error as string;
// Truncated (not the full 5000 chars) and carries the omission marker.
expect(errorText.length).toBeLessThan(long.length);
expect(errorText).toContain('chars omitted');
});
it('pairs parallel calls in one step with their outcomes by id', () => {
const trace = serializeSteps([
{
toolCalls: [
{ toolCallId: 'a', toolName: 'getPage', input: {} },
{ toolCallId: 'b', toolName: 'searchPages', input: {} },
],
toolResults: [{ toolCallId: 'b', toolName: 'searchPages' }],
content: [
{ type: 'tool-error', toolCallId: 'a', toolName: 'getPage', error: 'nope' },
],
},
]) as Array<Record<string, unknown>>;
// call a, outcome a (thrown), call b, outcome b (ok)
expect(trace).toHaveLength(4);
expect(trace[1]).toEqual({ toolName: 'getPage', error: 'nope', kind: 'thrown' });
expect(trace[3]).toEqual({ toolName: 'searchPages', ok: true });
});
});
// #490: every assistant row flushAssistant writes carries the v2 era marker so a
// dual-shape diagnostic query can branch on the trace shape without inspecting it.
describe('toolTraceVersion era marker (#490)', () => {
it('stamps metadata.toolTraceVersion = 2 on every flushed row', () => {
const seed = flushAssistant([], '', 'streaming');
expect(seed.metadata.toolTraceVersion).toBe(2);
const done = flushAssistant(
[
{
text: 'ok',
toolCalls: [{ toolCallId: 'c1', toolName: 'getPage', input: {} }],
toolResults: [{ toolCallId: 'c1', toolName: 'getPage' }],
},
],
'',
'completed',
{ finishReason: 'stop' },
);
expect(done.metadata.toolTraceVersion).toBe(2);
});
});
// #490 replay-budget signal helpers over persisted history.
describe('lastAssistantContextTokens', () => {
const row = (
role: string,
metadata: Record<string, unknown> | null,
): AiChatMessage => ({ role, metadata }) as unknown as AiChatMessage;
it('reads the most recent assistant turn contextTokens (provider fact)', () => {
const hist = [
row('user', null),
row('assistant', { contextTokens: 12000 }),
row('user', null),
row('assistant', { contextTokens: 41000 }),
];
expect(lastAssistantContextTokens(hist)).toBe(41000);
});
it('returns undefined when the last assistant turn recorded no usage', () => {
const hist = [row('assistant', { error: 'boom' }), row('user', null)];
expect(lastAssistantContextTokens(hist)).toBeUndefined();
expect(lastAssistantContextTokens([])).toBeUndefined();
});
});
// #490 snapshotOpenPage fast-path: skip the full Markdown export + upsert when a
// snapshot already exists at the page's CURRENT version (same updated_at instant).
describe('snapshotOpenPage fast-path (#490)', () => {
function makeSvc(existingSnapshot: unknown, pageUpdatedAt: Date) {
const exportPageMarkdown = jest.fn(async () => '# md');
const upsert = jest.fn(async () => undefined);
const findByChatPage = jest.fn(async () => existingSnapshot);
const pageRepo = {
findById: jest.fn(async () => ({
id: 'p1',
workspaceId: 'ws1',
updatedAt: pageUpdatedAt,
})),
};
const svc = new AiChatService(
{} as never, // ai
{} as never, // aiChatRepo
{} as never, // aiChatMessageRepo
{ findByChatPage, upsert } as never, // aiChatPageSnapshotRepo
{} as never, // aiSettings
{ exportPageMarkdown } as never, // tools
{} as never, // mcpClients
{} as never, // aiAgentRoleRepo
pageRepo as never, // pageRepo
{} as never, // pageAccess
{} as never, // environment
);
return { svc, exportPageMarkdown, upsert, findByChatPage };
}
const args = () =>
[
'chat1',
'p1',
{ id: 'ws1' } as never,
{ id: 'u1' } as never,
'sess',
] as const;
it('skips export + upsert when the snapshot is already at this page version', async () => {
const t = new Date('2026-07-07T10:00:00Z');
const { svc, exportPageMarkdown, upsert } = makeSvc(
{ pageUpdatedAt: t, contentMd: '# md' },
t,
);
await (svc as unknown as { snapshotOpenPage: (...a: unknown[]) => Promise<void> })
.snapshotOpenPage(...args());
expect(exportPageMarkdown).not.toHaveBeenCalled();
expect(upsert).not.toHaveBeenCalled();
});
it('exports + upserts when the page advanced since the snapshot', async () => {
const { svc, exportPageMarkdown, upsert } = makeSvc(
{ pageUpdatedAt: new Date('2026-07-07T10:00:00Z'), contentMd: 'old' },
new Date('2026-07-07T11:00:00Z'),
);
await (svc as unknown as { snapshotOpenPage: (...a: unknown[]) => Promise<void> })
.snapshotOpenPage(...args());
expect(exportPageMarkdown).toHaveBeenCalledTimes(1);
expect(upsert).toHaveBeenCalledTimes(1);
});
it('seeds (exports + upserts) on the first turn (no snapshot yet)', async () => {
const { svc, exportPageMarkdown, upsert } = makeSvc(
undefined,
new Date('2026-07-07T10:00:00Z'),
);
await (svc as unknown as { snapshotOpenPage: (...a: unknown[]) => Promise<void> })
.snapshotOpenPage(...args());
expect(exportPageMarkdown).toHaveBeenCalledTimes(1);
expect(upsert).toHaveBeenCalledTimes(1);
});
});
// #490 deferred-tool activation persisted across turns.
describe('seedActivatedTools', () => {
const valid = new Set(['Search_web', 'getPageJson', 'diffPageVersions']);
it('seeds from persisted metadata, intersected with current valid names', () => {
expect(
seedActivatedTools(
{ activatedTools: ['Search_web', 'getPageJson'] },
valid,
),
).toEqual(['Search_web', 'getPageJson']);
});
it('drops a stored tool that is no longer valid (allowlist/role changed)', () => {
// 'Habr_publish' was activated before but is not in the current allowlist.
expect(
seedActivatedTools({ activatedTools: ['Search_web', 'Habr_publish'] }, valid),
).toEqual(['Search_web']);
});
it('is empty/robust for missing, non-array, or unknown-shaped metadata', () => {
expect(seedActivatedTools(undefined, valid)).toEqual([]);
expect(seedActivatedTools({}, valid)).toEqual([]);
expect(seedActivatedTools({ activatedTools: 'nope' }, valid)).toEqual([]);
expect(
seedActivatedTools({ activatedTools: [1, 'getPageJson', null] }, valid),
).toEqual(['getPageJson']);
});
it('de-duplicates stored names', () => {
expect(
seedActivatedTools(
{ activatedTools: ['getPageJson', 'getPageJson'] },
valid,
),
).toEqual(['getPageJson']);
});
});
describe('lastAssistantReplayOverflow', () => {
const row = (
role: string,
metadata: Record<string, unknown> | null,
): AiChatMessage => ({ role, metadata }) as unknown as AiChatMessage;
it('is true only when the LAST assistant turn overflowed', () => {
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
row('user', null),
]),
).toBe(true);
// A recovered (later, non-overflow) assistant turn clears it.
expect(
lastAssistantReplayOverflow([
row('assistant', { replayOverflow: true }),
row('user', null),
row('assistant', { contextTokens: 5 }),
]),
).toBe(false);
expect(lastAssistantReplayOverflow([])).toBe(false);
});
// #490 reactive recovery: a prior turn stamped `replayOverflow` must make the
// NEXT turn's effective budget the AGGRESSIVE 0.5x cut — that harder trim is
// what un-bricks a chat that just 400'd on the context window. This exercises
// the exact wiring the service uses: read the stamp, then scale the threshold.
it('#490: a prior replayOverflow drives the next turn to the 0.5x aggressive budget', () => {
const history = [
row('assistant', { replayOverflow: true }),
row('user', null),
];
const priorOverflowed = lastAssistantReplayOverflow(history);
expect(priorOverflowed).toBe(true);
// Base budget 100k -> aggressive recovery halves it to 50k this turn.
expect(resolveEffectiveReplayThreshold(100_000, priorOverflowed)).toBe(50_000);
// Odd base floors, not rounds.
expect(resolveEffectiveReplayThreshold(99_999, true)).toBe(49_999);
// No prior overflow -> the base budget is used verbatim (no aggressive cut).
expect(resolveEffectiveReplayThreshold(100_000, false)).toBe(100_000);
// An explicit off-switch (null) is never overridden, even on recovery.
expect(resolveEffectiveReplayThreshold(null, true)).toBeNull();
});
});
describe('rowToUiMessage', () => {
@@ -930,23 +618,6 @@ describe('flushAssistant', () => {
expect(flushed.metadata.error).toBe('boom');
});
// #490 observability: the replay budgeter's decision is stamped on the turn.
it('records replayTrimmedToTokens + replayOverflow when provided', () => {
const f = flushAssistant([], '', 'error', {
error: 'ctx',
replayTrimmedToTokens: 42_000,
replayOverflow: true,
});
expect(f.metadata.replayTrimmedToTokens).toBe(42_000);
expect(f.metadata.replayOverflow).toBe(true);
});
it('omits the replay metadata when not provided', () => {
const f = flushAssistant([], '', 'completed', { finishReason: 'stop' });
expect('replayTrimmedToTokens' in f.metadata).toBe(false);
expect('replayOverflow' in f.metadata).toBe(false);
});
// #274 observability: the page-change diff the agent saw this turn is persisted
// to metadata.pageChanged when a non-empty diff was injected, and omitted when
// the diff is empty/whitespace or the arg is not supplied.
+88 -423
View File
@@ -55,12 +55,6 @@ import {
type SelectionContext,
} from './tools/current-page.util';
import { roleModelOverride } from './roles/role-model-config';
import {
resolveReplayBudget,
resolveEffectiveReplayThreshold,
isContextOverflowError,
trimHistoryForReplay,
} from './history-budget';
import {
startSseHeartbeat,
stripStreamingHopByHopHeaders,
@@ -123,14 +117,9 @@ const FINAL_STEP_NUDGE =
// NO text at all (#444, mitigates the "empty turn" the lockdown used to prevent
// when the toggle is OFF). Makes the exhausted-without-answer state explicit to
// the user and, on replay, to the model on the next turn.
// The persisted content is the app's base locale (en-US) — which is ALSO the
// i18n key the client localizes through `t()` — instead of a hardcoded Russian
// string (it used to render Russian for every locale, and fed Russian back to
// the model on replay). Keep it a plain, model-readable English sentence so the
// next turn's replay reads cleanly; the client resolves the locale.
const STEP_LIMIT_NO_ANSWER_MARKER =
'(Step limit reached — no final answer was produced; the work may be ' +
'unfinished. Reply "continue" to let the agent carry on.)';
'(Достигнут лимит шагов — итоговый ответ не сформулирован; работа могла ' +
'остаться незавершённой. Напишите «продолжай», чтобы агент продолжил.)';
// Reason recorded in ai_chat_runs.error / the assistant row when the token-
// degeneration detector (#444) aborts a run. Distinct from a user Stop (no error)
@@ -138,15 +127,6 @@ const STEP_LIMIT_NO_ANSWER_MARKER =
const OUTPUT_DEGENERATION_ERROR =
'Output degeneration detected (repeated token loop)';
// Prefix recorded on the assistant row when the provider rejected the turn for
// CONTEXT OVERFLOW (#490): the replayed history exceeded the model's window. The
// row is ALSO stamped `metadata.replayOverflow` so the NEXT turn's budgeter trims
// aggressively (the reactive recovery — the overflowing turn had no usage signal
// to trigger preventive trimming, so the classified 400 is what un-bricks it).
export const CONTEXT_OVERFLOW_ERROR_PREFIX =
'Диалог превысил контекстное окно модели; история будет агрессивно ' +
'сокращена на следующем ходу.';
/**
* Compute the step-budget warning text (#444), or '' when this step is outside
* the warning band. The warning fires on steps
@@ -902,21 +882,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
const freshPage = await this.pageRepo.findById(pageId);
// Page deleted during the turn (or somehow foreign) => don't write.
if (!freshPage || freshPage.workspaceId !== workspace.id) return;
// Fast-path (#490): if a snapshot already exists at THIS page version
// (same updated_at instant), its content is already current — skip the full
// Markdown export + upsert entirely. A turn that did NOT touch the open page
// (the common case) thus does no snapshot work. This mirrors the read-side
// fast path in detectPageChange (sameInstant): both trust that a page edit
// bumps updated_at. When the agent (or a human) DID edit the page this turn,
// updated_at advanced, so this does not match and we re-export as before.
const existing = await this.aiChatPageSnapshotRepo.findByChatPage(
chatId,
pageId,
workspace.id,
);
if (existing && sameInstant(existing.pageUpdatedAt, freshPage.updatedAt)) {
return;
}
const currentMd = await this.tools.exportPageMarkdown(
user,
sessionId,
@@ -956,17 +921,10 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// supplied or the supplied one does not belong to this workspace.
let isNewChat = false;
let chatId = body.chatId;
// Persisted chat-level metadata bag (#490): read once here so the deferred-tool
// activation set can be seeded from the previous turn. Undefined for a new chat.
let chatMetadata: Record<string, unknown> | undefined;
if (chatId) {
const existing = await this.aiChatRepo.findById(chatId, workspace.id);
if (!existing) {
chatId = undefined;
} else {
chatMetadata = (existing.metadata ?? undefined) as
| Record<string, unknown>
| undefined;
}
}
// The open page the client sent is attacker-controllable — BOTH its id and
@@ -1128,7 +1086,7 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// per-row conversion and degraded to plain text with a "[tool context
// omitted]" marker rather than 500-ing the whole turn (silent loss of tool
// context is not acceptable — the model must see the truncation).
let messages = await convertHistoryResilient(uiMessages, (index, err) =>
const messages = await convertHistoryResilient(uiMessages, (index, err) =>
this.logger.warn(
`Degraded unconvertible history row ${index} on chat ${chatId} to text: ${
err instanceof Error ? err.message : 'unknown error'
@@ -1177,56 +1135,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// Here we only need the admin-configured system prompt.
const resolved = await this.aiSettings.resolve(workspace.id);
// History-replay token budget (#490). The full conversation is replayed to
// the provider every turn, so a long chat eventually 400s on the context
// window — forever. Bound the REPLAYED history (never the persisted rows).
// PRIMARY signal is the provider's own fact: the last turn's contextTokens.
const replayBudget = resolveReplayBudget(resolved?.chatContextWindowRaw);
if (replayBudget.usedDefault) {
// The default fires precisely for installs with NO configured window —
// the ones that hit terminal overflow. Warn so it is observable.
this.logger.warn(
`AI chat (chat ${chatId}): no chatContextWindow configured; ` +
`applying the default replay budget (${replayBudget.thresholdTokens} tokens).`,
);
}
// Last turn's provider-reported context size (authoritative when present).
const priorContextTokens = lastAssistantContextTokens(oldHistory);
// Reactive recovery (#490): if the LAST turn was rejected for context
// overflow (stamped by onError), trim AGGRESSIVELY this turn — the
// overflowing turn produced no usage signal, so a normal-threshold trim may
// not shrink enough to fit. This is what un-bricks a chat that just 400'd.
const priorOverflowed = lastAssistantReplayOverflow(oldHistory);
const effectiveThreshold = resolveEffectiveReplayThreshold(
replayBudget.thresholdTokens,
priorOverflowed,
);
if (priorOverflowed) {
this.logger.warn(
`AI chat (chat ${chatId}): previous turn hit context overflow; ` +
`applying aggressive replay budget (${effectiveThreshold} tokens).`,
);
}
const preTrim = trimHistoryForReplay(
messages,
effectiveThreshold,
// A prior OVERFLOW means the provider count is stale/absent — force the
// char-estimate path by ignoring priorContextTokens on recovery.
priorOverflowed ? undefined : priorContextTokens,
);
messages = preTrim.messages;
// Observability (#490): record the budgeter's decision on the turn so the UI
// can surface "replay truncated at N tokens". Threaded into flushAssistant.
let replayTrimmedToTokens: number | undefined = preTrim.trimmed
? preTrim.estimatedTokens
: undefined;
if (preTrim.trimmed) {
this.logger.log(
`AI chat (chat ${chatId}): replay history trimmed to ~${preTrim.estimatedTokens} ` +
`tokens (budget ${replayBudget.thresholdTokens}).`,
);
}
// Build the external MCP toolset FIRST so the system prompt can carry each
// connected server's admin-authored guidance (#180). Merge in admin-
// configured external MCP tools (web search, etc.; §6.8). A down/slow
@@ -1418,19 +1326,10 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// tools + ALL external MCP tools), computed from the ACTUAL toolset so an
// external tool is loadable by its namespaced name. loadTools rejects any
// name outside this set.
const activatedTools = new Set<string>();
const validDeferredNames = new Set<string>(
Object.keys(baseTools).filter((k) => !CORE_TOOL_SET.has(k)),
);
// #490: seed the activation set from the chat's PERSISTED set so the model
// does not re-run loadTools every turn to re-activate the same tools. Only
// when deferred loading is enabled, and ALWAYS intersected with the CURRENT
// valid deferred names — an allowlist/role change must never resurrect a tool
// that no longer exists (prepareAgentStep would get a phantom active name).
const activatedTools = new Set<string>(
deferredEnabled
? seedActivatedTools(chatMetadata, validDeferredNames)
: [],
);
// Add the loadTools meta-tool ONLY when the feature is enabled; when off the
// toolset and behavior are exactly as before.
const tools = deferredEnabled
@@ -1440,39 +1339,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
}
: baseTools;
// #490: persist the (deterministically ordered) activation set back onto the
// chat metadata at turn end, so the NEXT turn seeds from it. Once-guarded and
// skipped when nothing new was activated (the set equals its seed) so an
// ordinary turn adds no extra write. Preserves other metadata keys.
let activatedToolsPersisted = false;
const persistActivatedTools = async (): Promise<void> => {
if (!deferredEnabled || activatedToolsPersisted || !chatId) return;
activatedToolsPersisted = true;
const current = [...activatedTools].sort();
const seeded = seedActivatedTools(chatMetadata, validDeferredNames).sort();
if (current.length === 0 || current.join('') === seeded.join('')) {
return; // nothing new activated -> no write
}
try {
await this.aiChatRepo.update(
chatId,
{
metadata: {
...(chatMetadata ?? {}),
activatedTools: current,
},
} as never,
workspace.id,
);
} catch (err) {
this.logger.warn(
`Failed to persist activated tools (chat ${chatId}): ${
err instanceof Error ? err.message : 'unknown error'
}`,
);
}
};
// Accumulate the turn's streamed output so a provider error / disconnect can
// persist the PARTIAL answer the user already saw — the SDK's onError/onAbort
// callbacks don't hand us the in-progress text. `capturedSteps` holds finished
@@ -1481,11 +1347,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
const capturedSteps: StepLike[] = [];
let inProgressText = '';
// Per-turn step->parts memo (#490): shared across every flushAssistant call
// this turn so each finished step's (large) output is JSON-stringified ONCE,
// not re-stringified on every subsequent onStepFinish flush (was O(N²)).
const partsCache: StepPartsCache = new WeakMap();
// Token-degeneration guard (#444). When the final-step lockdown is OFF, a
// runaway repetition loop (the 255KB "loadTools." incident) is aborted via
// this internal controller, unioned with the run/socket signal below. The
@@ -1555,10 +1416,7 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
await this.aiChatMessageRepo.update(
assistantId,
workspace.id,
flushAssistant(capturedSteps, '', 'streaming', {
pageChanged,
partsCache,
}),
flushAssistant(capturedSteps, '', 'streaming', { pageChanged }),
{ onlyIfStreaming: true },
);
} catch (err) {
@@ -1656,13 +1514,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
system,
messages,
tools,
// Pin the AI SDK per-request retry budget explicitly instead of relying
// on its default (which is also 2). Connection arithmetic per turn:
// (1 + maxRetries=2) × (1 + AI_STREAM_PRE_RESPONSE_RETRIES) network
// connects worst-case — the two retry layers compose, so making the SDK
// side explicit keeps that ceiling visible and pinned against SDK-default
// drift.
maxRetries: 2,
// No maxOutputTokens cap on the agent: tool-call arguments (e.g. a full
// page body for the write tools) are emitted as OUTPUT tokens, so a fixed
// cap would truncate complex tool calls mid-argument. Let the model use its
@@ -1798,8 +1649,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// closure scope here). Omitted/0 = no limit.
maxContextTokens: resolved?.chatContextWindow,
pageChanged,
partsCache,
replayTrimmedToTokens,
}),
);
// #184/#487: the RUN is finalized ALWAYS (never gated on the message).
@@ -1824,8 +1673,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// own edits are baked in — and this also SEEDS the snapshot on the first
// turn. Runs once across every terminal path (see snapshotTurnEnd).
await snapshotTurnEnd();
// #490: persist the deferred-tool activation set for the next turn.
await persistActivatedTools();
// Generate the chat title for a freshly created chat AFTER the stream's
// provider call has completed — NOT concurrently with it. The z.ai coding
@@ -1849,16 +1696,7 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// object, so the actual provider cause is clearly logged. Reuse the
// shared formatter so provider error formatting stays unified.
const e = error as { stack?: string };
// #490 reactive branch: classify a CONTEXT-OVERFLOW rejection (the
// replayed history exceeded the model window). The overflowing turn had
// no prior usage to trigger preventive trimming, so we record a clear,
// distinguishable cause AND stamp the row so the NEXT turn's budgeter
// trims aggressively — the reactive recovery that un-bricks the chat.
const overflow = isContextOverflowError(error);
const providerError = describeProviderError(error, String(error));
const errorText = overflow
? `${CONTEXT_OVERFLOW_ERROR_PREFIX} (${providerError})`
: providerError;
const errorText = describeProviderError(error, String(error));
this.logger.error(`AI chat stream error: ${errorText}`, e?.stack);
// DIAGNOSTIC (Safari stream-drop investigation) — temporary: timing of
// an error-terminated stream.
@@ -1876,9 +1714,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
flushAssistant(capturedSteps, inProgressText, 'error', {
error: errorText,
pageChanged,
partsCache,
replayTrimmedToTokens,
replayOverflow: overflow || undefined,
}),
);
// #184: settle the RUN as failed, carrying the provider/transport cause.
@@ -1888,8 +1723,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// committed before the error must be baked into the snapshot, or the
// next turn would mis-report it as a user edit.
await snapshotTurnEnd();
// #490: persist the deferred-tool activation set for the next turn.
await persistActivatedTools();
},
onAbort: async ({ steps }) => {
// #444: distinguish a degeneration abort (our internal controller) from
@@ -1904,7 +1737,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
flushAssistant(capturedSteps, truncated, 'error', {
error: OUTPUT_DEGENERATION_ERROR,
pageChanged,
partsCache,
}),
);
if (runId)
@@ -1915,8 +1747,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
);
await closeExternalClients();
await snapshotTurnEnd();
// #490: persist the deferred-tool activation set for the next turn.
await persistActivatedTools();
return;
}
const partialChars =
@@ -1941,7 +1771,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
await finalizeAssistant(
flushAssistant(capturedSteps, inProgressText, 'aborted', {
pageChanged,
partsCache,
}),
);
// #184: settle the RUN as aborted (an explicit user stop reached the
@@ -1952,8 +1781,6 @@ export class AiChatService implements OnModuleInit, OnModuleDestroy {
// committed before the client disconnect / stop() must be baked into the
// snapshot, or the next turn would mis-report it as a user edit.
await snapshotTurnEnd();
// #490: persist the deferred-tool activation set for the next turn.
await persistActivatedTools();
},
});
@@ -2264,70 +2091,6 @@ export function chatStreamMetadata(
return undefined;
}
/**
* The provider-reported context size of the most recent assistant turn, read from
* its persisted `metadata.contextTokens` (#490 replay budgeter's PRIMARY signal
* the provider's own fact, not an estimate). Returns undefined for a chat with no
* assistant turn yet, or one whose last turn recorded no usage (e.g. it errored),
* in which case the budgeter falls back to the char-estimate.
*/
export function lastAssistantContextTokens(
history: ReadonlyArray<AiChatMessage>,
): number | undefined {
for (let i = history.length - 1; i >= 0; i--) {
const row = history[i];
if (row.role !== 'assistant') continue;
const meta = (row.metadata ?? {}) as { contextTokens?: unknown };
const n = meta.contextTokens;
return typeof n === 'number' && Number.isFinite(n) && n > 0 ? n : undefined;
}
return undefined;
}
/**
* Seed the per-turn deferred-tool activation set from a chat's persisted metadata
* (#490), INTERSECTED with the current valid deferred names. Persisting the set
* across turns saves the model re-running loadTools every turn to re-activate the
* same tools; intersecting on load means a changed allowlist / role can never
* resurrect a tool that no longer exists (which would hand prepareAgentStep a
* phantom active name). Tolerant of any stored shape a non-array is ignored.
*/
export function seedActivatedTools(
metadata: Record<string, unknown> | undefined,
validDeferredNames: ReadonlySet<string>,
): string[] {
const stored = metadata?.activatedTools;
if (!Array.isArray(stored)) return [];
const seen = new Set<string>();
const out: string[] = [];
for (const name of stored) {
if (typeof name === 'string' && validDeferredNames.has(name) && !seen.has(name)) {
seen.add(name);
out.push(name);
}
}
return out;
}
/**
* Whether the most recent assistant turn was rejected for CONTEXT OVERFLOW
* (#490): its row carries `metadata.replayOverflow` (stamped by the stream's
* onError). The next turn's budgeter reads this to trim aggressively the
* reactive recovery. Only the LAST assistant turn matters (an older overflow was
* already recovered), so we stop at the first assistant row scanning backwards.
*/
export function lastAssistantReplayOverflow(
history: ReadonlyArray<AiChatMessage>,
): boolean {
for (let i = history.length - 1; i >= 0; i--) {
const row = history[i];
if (row.role !== 'assistant') continue;
const meta = (row.metadata ?? {}) as { replayOverflow?: unknown };
return meta.replayOverflow === true;
}
return false;
}
/** The last message with role 'user' from a useChat payload, if any. */
function lastUserMessage(
messages: UIMessage[] | undefined,
@@ -2389,15 +2152,6 @@ export function sanitizeUserParts(
/** Marker for a history row whose tool parts could not be replayed (#489). */
export const TOOL_CONTEXT_OMITTED_MARKER = '[tool context omitted]';
/**
* Synthetic error text for a tool call that neither returned a result nor threw
* a `tool-error` i.e. it was interrupted mid-step (an abort / server restart).
* Shared by `assistantParts` (the replayed `output-error` part) and
* `serializeSteps` (the `{ kind: 'interrupted' }` trace element) so the replay
* text and the trace stay in lockstep (#490).
*/
export const TOOL_CALL_INCOMPLETE_TEXT = 'Tool call did not complete.';
/**
* Convert persisted UI history to model messages, tolerating a single poisoned
* row (#489). `convertToModelMessages` over the WHOLE array throws if ANY row is
@@ -2605,97 +2359,71 @@ function normalizeToolError(error: unknown): string {
*/
// Exported only so the unit tests can import these pure helpers; exporting
// them does not change runtime behavior.
/**
* Per-turn memo for {@link assistantParts}: a step's rebuilt parts keyed by the
* step OBJECT's identity (#490). A finished step in `capturedSteps` keeps a stable
* reference across every mid-stream flush, and `compactToolOutput` inside it does a
* `JSON.stringify` of the whole (often 50200 KB) output so without a memo each
* `onStepFinish` re-stringifies EVERY prior step's output (O(N²) stringify over a
* turn). Keyed by step identity => one stringify per step per turn. WeakMap so a
* turn's steps are GC'd with the turn.
*/
export type StepPartsCache = WeakMap<object, Array<Record<string, unknown>>>;
/** Build the parts for ONE step (text + a part per tool call). Pure. */
function buildStepParts(step: StepLike): Array<Record<string, unknown>> {
const parts: Array<Record<string, unknown>> = [];
if (step.text) {
parts.push({ type: 'text', text: step.text });
}
// Index this step's results by tool call id to pair calls with outputs.
const resultsById = new Map<string, unknown>();
for (const r of step.toolResults ?? []) {
if (r.toolCallId) resultsById.set(r.toolCallId, r.output);
}
// Index this step's THROWN tool failures (ai@6 `tool-error` content parts)
// by tool call id, so a call that failed replays with its real error text.
const errorsById = new Map<string, unknown>();
for (const part of step.content ?? []) {
if (part.type === 'tool-error' && part.toolCallId) {
errorsById.set(part.toolCallId, part.error);
}
}
for (const call of step.toolCalls ?? []) {
if (!call.toolName || !call.toolCallId) continue;
const hasResult = resultsById.has(call.toolCallId);
if (hasResult) {
// output-available: the tool returned; the next turn replays its result.
parts.push({
type: `tool-${call.toolName}`,
toolCallId: call.toolCallId,
state: 'output-available',
input: call.input,
output: compactToolOutput(resultsById.get(call.toolCallId)),
});
} else if (errorsById.has(call.toolCallId)) {
// The tool THREW: replay the REAL error so the model on the next turn
// knows WHY the call failed (and does not blindly repeat it). An
// output-error round-trips through convertToModelMessages as a balanced
// tool-call + tool-result, keeping the rebuilt history valid.
parts.push({
type: `tool-${call.toolName}`,
toolCallId: call.toolCallId,
state: 'output-error',
input: call.input,
errorText: normalizeToolError(errorsById.get(call.toolCallId)),
});
} else {
// No paired result AND no tool-error (e.g. aborted mid-step). Persisting
// a bare tool-call (input-available) would replay as an unpaired call and
// throw MissingToolResultsError on the next turn (convertToModelMessages
// emits no tool-result for it). Emit a SYNTHETIC paired result instead:
// an output-error round-trips through convertToModelMessages as a
// balanced tool-call + tool-result, keeping the rebuilt history valid.
parts.push({
type: `tool-${call.toolName}`,
toolCallId: call.toolCallId,
state: 'output-error',
input: call.input,
errorText: TOOL_CALL_INCOMPLETE_TEXT,
});
}
}
return parts;
}
export function assistantParts(
steps: ReadonlyArray<StepLike> | undefined,
fallbackText: string,
cache?: StepPartsCache,
): UIMessage['parts'] {
const parts: Array<Record<string, unknown>> = [];
let sawText = false;
for (const step of steps ?? []) {
// Memoize per step object (#490): a finished step is immutable and keeps its
// reference across flushes, so its parts (and the costly output stringify) are
// built exactly once per turn. A cache miss (or no cache) just rebuilds.
let stepParts = cache?.get(step as object);
if (!stepParts) {
stepParts = buildStepParts(step);
cache?.set(step as object, stepParts);
if (step.text) {
parts.push({ type: 'text', text: step.text });
sawText = true;
}
// Index this step's results by tool call id to pair calls with outputs.
const resultsById = new Map<string, unknown>();
for (const r of step.toolResults ?? []) {
if (r.toolCallId) resultsById.set(r.toolCallId, r.output);
}
// Index this step's THROWN tool failures (ai@6 `tool-error` content parts)
// by tool call id, so a call that failed replays with its real error text.
const errorsById = new Map<string, unknown>();
for (const part of step.content ?? []) {
if (part.type === 'tool-error' && part.toolCallId) {
errorsById.set(part.toolCallId, part.error);
}
}
for (const call of step.toolCalls ?? []) {
if (!call.toolName || !call.toolCallId) continue;
const hasResult = resultsById.has(call.toolCallId);
if (hasResult) {
// output-available: the tool returned; the next turn replays its result.
parts.push({
type: `tool-${call.toolName}`,
toolCallId: call.toolCallId,
state: 'output-available',
input: call.input,
output: compactToolOutput(resultsById.get(call.toolCallId)),
});
} else if (errorsById.has(call.toolCallId)) {
// The tool THREW: replay the REAL error so the model on the next turn
// knows WHY the call failed (and does not blindly repeat it). An
// output-error round-trips through convertToModelMessages as a balanced
// tool-call + tool-result, keeping the rebuilt history valid.
parts.push({
type: `tool-${call.toolName}`,
toolCallId: call.toolCallId,
state: 'output-error',
input: call.input,
errorText: normalizeToolError(errorsById.get(call.toolCallId)),
});
} else {
// No paired result AND no tool-error (e.g. aborted mid-step). Persisting
// a bare tool-call (input-available) would replay as an unpaired call and
// throw MissingToolResultsError on the next turn (convertToModelMessages
// emits no tool-result for it). Emit a SYNTHETIC paired result instead:
// an output-error round-trips through convertToModelMessages as a
// balanced tool-call + tool-result, keeping the rebuilt history valid.
parts.push({
type: `tool-${call.toolName}`,
toolCallId: call.toolCallId,
state: 'output-error',
input: call.input,
errorText: 'Tool call did not complete.',
});
}
}
parts.push(...stepParts);
}
const sawText = parts.some((p) => p.type === 'text');
if (!sawText && fallbackText) {
// No per-step text (e.g. a single final block): append the final text after
// any tool parts so the natural call -> result -> answer order is preserved.
@@ -2858,16 +2586,6 @@ export function flushAssistant(
maxContextTokens?: number;
error?: string;
pageChanged?: { title: string; diff: string } | null;
// Per-turn step->parts memo (#490): pass the SAME cache on every flush of a
// turn so each finished step's output is stringified once, not once per flush.
partsCache?: StepPartsCache;
// #490 observability: when the replay budgeter trimmed this turn's history,
// the (estimated) token size it trimmed to — the UI can show "replay truncated
// at N tokens". Omitted when nothing was trimmed.
replayTrimmedToTokens?: number;
// #490 reactive branch: set when the provider rejected this turn for context
// overflow. Stamped into metadata so the NEXT turn's budgeter trims aggressively.
replayOverflow?: boolean;
},
): AssistantFlush {
const finished = capturedSteps ?? [];
@@ -2877,20 +2595,13 @@ export function flushAssistant(
// in-progress step's text (the partial answer cut off by an error/abort, or
// simply not yet flushed mid-stream) as the last text part so the persisted
// parts match what streamed to the client.
const parts = assistantParts(
finished,
'',
extra?.partsCache,
) as unknown as Array<Record<string, unknown>>;
const parts = assistantParts(finished, '') as unknown as Array<
Record<string, unknown>
>;
if (trailing) parts.push({ type: 'text', text: trailing });
const metadata: Record<string, unknown> = {
parts: parts as unknown as UIMessage['parts'],
// Era marker for the `tool_calls` trace shape (#490): v2 stores outcome flags
// ({ ok } / { error, kind }) and NO tool output (the output lives once in
// `parts`). Old rows have no marker and the legacy { output } shape; a
// dual-shape query branches on this. Old rows are deliberately NOT migrated.
toolTraceVersion: 2,
};
// finishReason: prefer an explicit one; else derive a sensible value from the
// terminal status (so onError/onAbort records keep their historical reason).
@@ -2906,9 +2617,6 @@ export function flushAssistant(
if (extra?.contextTokens) metadata.contextTokens = extra.contextTokens;
if (extra?.maxContextTokens)
metadata.maxContextTokens = extra.maxContextTokens;
if (extra?.replayTrimmedToTokens)
metadata.replayTrimmedToTokens = extra.replayTrimmedToTokens;
if (extra?.replayOverflow) metadata.replayOverflow = true;
if (extra?.error) metadata.error = extra.error;
// Persist the page-change diff the agent saw this turn (#274 observability),
// so history / the Markdown export can show what the user changed. Only when
@@ -2934,85 +2642,42 @@ export function flushAssistant(
/**
* Reduce SDK step objects to a compact, JSON-serializable trace for the
* `tool_calls` column trace format **v2** (#490).
*
* v2 stores, per call, ONLY the metadata a queryable trace needs never the
* tool OUTPUT. Before #490 each output was persisted TWICE: once here (compacted)
* and once in `metadata.parts` (via `assistantParts`), so a 50-step run with
* 50200 KB outputs wrote hundreds of MB per turn (each `onStepFinish` rewrote
* the whole row). The parts copy is the one the model replays and the UI/Markdown
* export render, so the trace copy of the output was pure duplication. v2 keeps
* the output ONLY in parts and reduces the trace to outcome flags.
*
* Element shapes (paired per call, in order):
* - `{ toolName, input }` the call
* - `{ toolName, ok: true }` it returned a result (success)
* - `{ toolName, error, kind: 'thrown' }` it threw a `tool-error`
* - `{ toolName, error, kind: 'interrupted' }` no result and no throw (an
* abort / server restart mid-step). `kind` is MANDATORY: without it a
* synthetic "Tool call did not complete." is indistinguishable from a real
* hard-fail and pollutes any error-rate scan. The distinction is STRUCTURAL
* (an `errorsById` hit vs the synthetic fallback branch), NOT a per-tool
* classifier soft failures stay OUT of the trace (they live in
* `metadata.parts` outputs; a per-tool mirror would persist its own bugs).
*
* Rows carry `metadata.toolTraceVersion: 2` (set by {@link flushAssistant}) so a
* dual-shape query can branch on the era. Old rows are NOT migrated (rewriting
* giant jsonb is the very WAL churn this removes); see docs/reading-ai-logs.md.
* `tool_calls` column. Stores only what the UI action-log and history need
* never raw provider payloads or keys.
*/
export function serializeSteps(
steps: ReadonlyArray<{
toolCalls?: ReadonlyArray<{
toolCallId?: string;
toolName?: string;
input?: unknown;
}>;
toolResults?: ReadonlyArray<{ toolCallId?: string; toolName?: string }>;
toolCalls?: ReadonlyArray<{ toolName?: string; input?: unknown }>;
toolResults?: ReadonlyArray<{ toolName?: string; output?: unknown }>;
content?: ReadonlyArray<{
type?: string;
toolCallId?: string;
toolName?: string;
error?: unknown;
}>;
}>,
): unknown {
const calls: Array<
| { toolName?: string; input?: unknown }
| { toolName?: string; ok: true }
| { toolName?: string; error: string; kind: 'thrown' | 'interrupted' }
> = [];
const calls: Array<{
toolName?: string;
input?: unknown;
output?: unknown;
error?: string;
}> = [];
for (const step of steps ?? []) {
// Index this step's results + thrown errors by tool call id, so each call is
// paired with its outcome (mirrors assistantParts' pairing exactly).
const resultIds = new Set<string>();
for (const r of step.toolResults ?? []) {
if (r.toolCallId) resultIds.add(r.toolCallId);
}
const errorsById = new Map<string, unknown>();
for (const part of step.content ?? []) {
if (part.type === 'tool-error' && part.toolCallId) {
errorsById.set(part.toolCallId, part.error);
}
}
for (const call of step.toolCalls ?? []) {
calls.push({ toolName: call.toolName, input: call.input });
if (call.toolCallId && resultIds.has(call.toolCallId)) {
// Success: the output itself lives in metadata.parts, not here.
calls.push({ toolName: call.toolName, ok: true });
} else if (call.toolCallId && errorsById.has(call.toolCallId)) {
// Hard fail: the tool threw. Persist the real (bounded) reason.
}
for (const r of step.toolResults ?? []) {
calls.push({ toolName: r.toolName, output: compactToolOutput(r.output) });
}
// ai@6 surfaces a THROWN tool failure as a `tool-error` content part, NOT as
// a `toolResults` entry. Record it as its own paired element (mirroring how a
// successful result is appended) so the failure and its reason survive in the
// trace instead of leaving an orphaned call with no result.
for (const part of step.content ?? []) {
if (part.type === 'tool-error') {
calls.push({
toolName: call.toolName,
error: normalizeToolError(errorsById.get(call.toolCallId)),
kind: 'thrown',
});
} else {
// Neither a result nor a throw: interrupted mid-step (abort/restart).
// Marked structurally so it never inflates a thrown-error count.
calls.push({
toolName: call.toolName,
error: TOOL_CALL_INCOMPLETE_TEXT,
kind: 'interrupted',
toolName: part.toolName,
error: normalizeToolError(part.error),
});
}
}
@@ -1,209 +0,0 @@
import { randomBytes } from 'crypto';
import { Client } from 'pg';
import { flushAssistant, serializeSteps } from './ai-chat.service';
/**
* #490 write-volume regression an OBSERVABLE-PROPERTY test on a LIVE Postgres,
* not "bytes through a mock repo" (a mock measures exactly the thing that does not
* hurt). It drives a realistic 50-step run where each step returns a ~100 KB tool
* output and, at every `onStepFinish`, UPDATEs the assistant row the way the
* service does then reads the REAL write volume via the `pg_current_wal_lsn()`
* delta around the run.
*
* The property proven: v2 stores each tool OUTPUT only in `metadata.parts`, no
* longer ALSO in the `tool_calls` trace. So:
* 1. the trace (`tool_calls`) column's write volume is now O(Σ steps) tiny,
* linear outcome flags vs the pre-#490 O(N²) that re-persisted every prior
* output on every step; and
* 2. the FULL-row write volume drops sharply (the duplicated output copy is gone).
*
* Connects to the local gitmost test Postgres (docker `gitmost-test-pg` on :5432);
* SKIPS cleanly when that DB is not reachable so it never breaks a DB-less CI.
*/
const CONN =
process.env.WAL_TEST_DATABASE_URL ??
'postgresql://docmost:docmost_dev_pw@localhost:5432/docmost';
// A step whose tool output is ~100 KB (a page read), in the SDK StepLike shape.
// The body is INCOMPRESSIBLE random text — a `'x'.repeat()` filler would TOAST-
// compress to nothing and hide the real write volume (a page body does not).
function makeStep(i: number, outputBytes = 100_000) {
const body = randomBytes(Math.ceil(outputBytes * 0.75)).toString('base64');
return {
text: `step ${i} reasoning`,
toolCalls: [{ toolCallId: `c${i}`, toolName: 'getPage', input: { id: `p${i}` } }],
toolResults: [
{
toolCallId: `c${i}`,
toolName: 'getPage',
output: { id: `p${i}`, title: `Page ${i}`, body },
},
],
};
}
// The pre-#490 (v1) trace: outputs stored a SECOND time in `tool_calls`
// (the duplication #490 removed). Mirrors the OLD serializeSteps shape.
function v1Trace(steps: ReturnType<typeof makeStep>[]): unknown {
const calls: unknown[] = [];
for (const s of steps) {
for (const c of s.toolCalls) calls.push({ toolName: c.toolName, input: c.input });
for (const r of s.toolResults)
calls.push({ toolName: r.toolName, output: r.output });
}
return calls;
}
async function walDelta(
client: Client,
fn: () => Promise<void>,
): Promise<number> {
const before = (await client.query('SELECT pg_current_wal_lsn() AS l')).rows[0]
.l as string;
await fn();
// NOTE: do NOT pg_switch_wal() here — a segment switch pads the LSN to the next
// 16 MB boundary and would swamp the actual write delta. The raw LSN advances by
// the bytes of WAL emitted, which is exactly what we want to measure.
const after = (await client.query('SELECT pg_current_wal_lsn() AS l')).rows[0]
.l as string;
return Number(
(await client.query('SELECT pg_wal_lsn_diff($1,$2) AS d', [after, before]))
.rows[0].d,
);
}
describe('#490 write-volume on a live Postgres (pg_current_wal_lsn delta)', () => {
let client: Client | undefined;
let available = false;
beforeAll(async () => {
try {
client = new Client(CONN);
await client.connect();
await client.query('SELECT pg_current_wal_lsn()');
available = true;
} catch {
available = false;
client = undefined;
}
});
afterAll(async () => {
await client?.end().catch(() => undefined);
});
const STEPS = 50;
it('v2 trace write volume is O(Σ steps) — a tiny fraction of the v1 duplicate', async () => {
if (!available || !client) {
console.warn('SKIP: gitmost-test-pg not reachable; skipping WAL test.');
return;
}
const c = client;
// Isolated table so we measure only the tool_calls (trace) column's writes.
await c.query('DROP TABLE IF EXISTS _wal_trace');
await c.query('CREATE TABLE _wal_trace(id int primary key, tool_calls jsonb)');
await c.query("INSERT INTO _wal_trace VALUES (1, '[]'::jsonb)");
const steps: ReturnType<typeof makeStep>[] = [];
// v1: each step re-persists ALL prior outputs into the trace (the O(N²) churn).
const v1 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
await c.query('UPDATE _wal_trace SET tool_calls=$1 WHERE id=1', [
JSON.stringify(v1Trace(acc)),
]);
}
steps.push(...acc);
});
await c.query("UPDATE _wal_trace SET tool_calls='[]'::jsonb WHERE id=1");
// v2: the REAL serializeSteps — outcome flags only, NO outputs.
const v2 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
await c.query('UPDATE _wal_trace SET tool_calls=$1 WHERE id=1', [
JSON.stringify(serializeSteps(acc)),
]);
}
});
await c.query('DROP TABLE IF EXISTS _wal_trace');
// eslint-disable-next-line no-console
console.log(
`[#490 WAL] trace column over ${STEPS} steps: v1=${(v1 / 1e6).toFixed(1)}MB ` +
`v2=${(v2 / 1e6).toFixed(2)}MB (${(v1 / v2).toFixed(0)}x smaller)`,
);
// The trace no longer carries outputs: v2 is a tiny fraction of v1's WAL.
expect(v2).toBeLessThan(v1 * 0.1);
// And v2's trace WAL is small in absolute terms — O(Σ steps) of flags, not
// O(N² × output). 50 steps of ~40-byte flags is well under a few MB of WAL.
expect(v2).toBeLessThan(5_000_000);
// v1's duplicate alone is huge (≈ the 100 KB output re-written N² times).
expect(v1).toBeGreaterThan(50_000_000);
}, 120_000);
it('the full assistant row write drops sharply once the duplicate is gone', async () => {
if (!available || !client) return;
const c = client;
await c.query('DROP TABLE IF EXISTS _wal_full');
await c.query(
'CREATE TABLE _wal_full(id int primary key, content text, tool_calls jsonb, metadata jsonb, status text)',
);
await c.query("INSERT INTO _wal_full VALUES (1, '', '[]'::jsonb, '{}'::jsonb, 'streaming')");
const writeRow = async (patch: {
content: string;
toolCalls: unknown;
metadata: unknown;
status: string;
}) =>
c.query(
'UPDATE _wal_full SET content=$1, tool_calls=$2, metadata=$3, status=$4 WHERE id=1',
[
patch.content,
JSON.stringify(patch.toolCalls ?? null),
JSON.stringify(patch.metadata),
patch.status,
],
);
// v2 (real flushAssistant): outputs live once, in metadata.parts.
const v2 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
await writeRow(flushAssistant(acc as never, '', 'streaming'));
}
});
await c.query("UPDATE _wal_full SET content='', tool_calls='[]'::jsonb, metadata='{}'::jsonb WHERE id=1");
// v1: same row PLUS the duplicated outputs in the trace column.
const v1 = await walDelta(c, async () => {
const acc: ReturnType<typeof makeStep>[] = [];
for (let i = 0; i < STEPS; i++) {
acc.push(makeStep(i));
const f = flushAssistant(acc as never, '', 'streaming');
await writeRow({ ...f, toolCalls: v1Trace(acc) });
}
});
await c.query('DROP TABLE IF EXISTS _wal_full');
// eslint-disable-next-line no-console
console.log(
`[#490 WAL] full row over ${STEPS} steps: v1=${(v1 / 1e6).toFixed(1)}MB ` +
`v2=${(v2 / 1e6).toFixed(1)}MB (saved ${((1 - v2 / v1) * 100).toFixed(0)}%)`,
);
// Removing the duplicated trace copy is a large, real write-volume reduction.
expect(v2).toBeLessThan(v1 * 0.75);
}, 120_000);
});
@@ -1,10 +1,5 @@
import {
buildChatMarkdown,
normalizeLang,
labelledToolNames,
} from './chat-markdown.util';
import { buildChatMarkdown, normalizeLang } from './chat-markdown.util';
import type { AiChatMessage } from '@docmost/db/types/entity.types';
import { SHARED_TOOL_SPECS } from '../../../../../packages/mcp/src/tool-specs';
/**
* normalizeLang: the client sends `i18n.language` a FULL locale tag like
@@ -460,43 +455,3 @@ describe('buildChatMarkdown (server) — structure', () => {
expect(md).toContain('````');
});
});
/**
* #494 REVERSE drift-guard for the export's friendly tool labels. A label keyed
* by a tool name that no longer exists silently degrades to the generic
* "Ran tool <name>" line; nothing reddened before. This asserts every labelled
* name is a real in-app tool and that both languages label the same set.
*/
describe('tool-label parity (#494)', () => {
// In-app tool names come from the shared registry (inAppKey, excluding
// mcpOnly specs) PLUS the inline in-app-only tools that carry a friendly label.
// The only labelled inline tool is the hybrid semantic search.
const INLINE_INAPP_LABELLED = new Set(['searchPages']);
function validInAppToolNames(): Set<string> {
const names = new Set<string>(INLINE_INAPP_LABELLED);
for (const spec of Object.values(SHARED_TOOL_SPECS)) {
if ((spec as { mcpOnly?: boolean }).mcpOnly) continue;
names.add((spec as { inAppKey: string }).inAppKey);
}
return names;
}
it('en and ru label the SAME set of tools', () => {
expect(labelledToolNames('en').sort()).toEqual(
labelledToolNames('ru').sort(),
);
});
it('every labelled tool name is a real in-app tool', () => {
const valid = validInAppToolNames();
const dead = labelledToolNames('en').filter((n) => !valid.has(n));
expect(dead).toEqual([]);
});
it('the guard REDDENS for an unknown label key (mutation check)', () => {
const valid = validInAppToolNames();
// A hypothetical renamed-away label must be caught.
expect(valid.has('getPageRenamedAway')).toBe(false);
});
});
@@ -154,17 +154,6 @@ function toolLabel(name: string, lang: ExportLang): string {
return LABELS[lang].tools[name] ?? LABELS[lang].ranTool(name);
}
/**
* The tool names that carry a hand-written friendly export label, per language.
* Exported for the drift-guard (#494): a label keyed by a tool name that no
* longer exists is a DEAD entry (the tool was renamed and now silently falls back
* to the generic `ranTool(name)` line). The guard asserts every key here is a
* real in-app tool AND that the two languages label the SAME set of tools.
*/
export function labelledToolNames(lang: ExportLang): string[] {
return Object.keys(LABELS[lang].tools);
}
/**
* Stringify an arbitrary tool input/output value for a fenced block. Strings
* pass through as-is; everything else is pretty-printed JSON, falling back to
@@ -37,13 +37,10 @@ export class CreateMcpServerDto {
@IsObject()
headers?: Record<string, string>;
// Omit/null => no restriction; `[]` is persisted verbatim and means deny-all
// (zero tools) since #476. @IsOptional() skips validation for null as well,
// so an explicit null is accepted.
@IsOptional()
@IsArray()
@IsString({ each: true })
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored guidance ("how/when to use this server's tools") injected
// into the agent system prompt next to the tool descriptions (#180). Trusted,
@@ -38,13 +38,10 @@ export class UpdateMcpServerDto {
@IsObject()
headers?: Record<string, string>;
// Absent => unchanged; null => no restriction; `[]` is persisted verbatim
// and means deny-all (zero tools) since #476. @IsOptional() skips validation
// for null as well, so an explicit null is accepted.
@IsOptional()
@IsArray()
@IsString({ each: true })
toolAllowlist?: string[] | null;
toolAllowlist?: string[];
// Admin-authored prompt guidance (#180). Absent => unchanged; blank => cleared
// (stored as null by the repo). Capped to bound prompt/token size.
@@ -1,169 +0,0 @@
import { type Tool } from 'ai';
import { McpClientsService } from './mcp-clients.service';
/**
* Tool-allowlist filtering semantics on the merged external toolset (#476).
*
* COVERAGE CHOICE (documented per issue #476): the full corrupt-row chain
* (DB value -> repo normalizeRow -> toolsFor filter) is covered on TWO levels
* instead of one live-stub-MCP-server integration test:
* (a) apps/server/test/integration/ai-mcp-server-repo.int-spec.ts pins the
* repo read/write semantics against a real Postgres `[]` round-trips
* as jsonb `[]`, a present-but-corrupt value fails CLOSED to `[]` with
* an error log;
* (b) THIS spec pins what the toolset builder does with the repo's output
* null = unrestricted, `['alpha']` = only alpha, `[]` (including the
* corrupt-row fallback) = ZERO tools.
* Together they prove the end-to-end property "corrupt/empty allowlist can
* never widen to all tools" without a live stub HTTP MCP server.
*
* The drive path mirrors mcp-namespacing.spec.ts: stub the repo's listEnabled,
* spy the private `connect` to return a fake client, inspect the merged keys.
*/
function fakeTool(): Tool {
return { description: 'x', inputSchema: undefined } as unknown as Tool;
}
interface FakeServer {
id: string;
name: string;
transport: string;
url: string;
headersEnc: string | null;
toolAllowlist: string[] | null;
}
function server(
over: Partial<FakeServer> & { id: string; name: string },
): FakeServer {
return {
transport: 'http',
url: 'https://example.com/mcp',
headersEnc: null,
toolAllowlist: null,
...over,
};
}
/**
* Build a service whose repo returns `servers` and whose fake clients expose
* `rawTools` from tools(). Returns the merged tool keys produced by toolsFor.
*/
async function mergedKeysFor(
servers: FakeServer[],
rawTools: Record<string, Tool>,
): Promise<string[]> {
const repoStub = {
listEnabled: jest.fn().mockResolvedValue(servers),
};
const service = new McpClientsService(repoStub as never, {} as never);
jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => unknown },
'connect',
)
.mockImplementation(() =>
Promise.resolve({
tools: () => Promise.resolve(rawTools),
close: () => Promise.resolve(),
}),
);
const toolset = await service.toolsFor('ws-1');
// Release the lease so the service does not hold the fake clients open.
await Promise.all(toolset.clients.map((c) => c.close()));
return Object.keys(toolset.tools);
}
describe('external MCP tool-allowlist filtering (via toolsFor, #476)', () => {
afterEach(() => jest.restoreAllMocks());
const RAW = () => ({
alpha: fakeTool(),
beta: fakeTool(),
gamma: fakeTool(),
});
it("['alpha'] lets ONLY alpha through", async () => {
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: ['alpha'] })],
RAW(),
);
expect(keys).toEqual(['srv_alpha']);
});
it('null (no restriction) lets every tool through', async () => {
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: null })],
RAW(),
);
expect(keys.sort()).toEqual(['srv_alpha', 'srv_beta', 'srv_gamma']);
});
it('[] (deny-all) yields ZERO tools — an empty array is authoritative, not falsy (#476)', async () => {
// This is the regression the #476 change guards: `[]` used to fall through
// the old `allow.length > 0` check and expose ALL tools. It must expose NONE.
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: [] })],
RAW(),
);
expect(keys).toEqual([]);
});
it('the corrupt-row fallback ([] from the repo) also yields ZERO tools (#476)', async () => {
// The repo turns a present-but-corrupt tool_allowlist into `[]` (fail-closed,
// see normalizeRow in ai-mcp-server.repo.ts + the int-spec); this pins that
// the toolset builder honours that fallback as deny-all rather than allow-all.
const corruptFallback: string[] = [];
const keys = await mergedKeysFor(
[server({ id: 'id-1', name: 'srv', toolAllowlist: corruptFallback })],
RAW(),
);
expect(keys).toEqual([]);
});
it('allowlisted names not exposed by the server are ignored (no phantom tools)', async () => {
const keys = await mergedKeysFor(
[
server({
id: 'id-1',
name: 'srv',
toolAllowlist: ['alpha', 'does-not-exist'],
}),
],
RAW(),
);
expect(keys).toEqual(['srv_alpha']);
});
it('a deny-all server contributes no prompt instructions (0 tools merged)', async () => {
const repoStub = {
listEnabled: jest.fn().mockResolvedValue([
{
...server({ id: 'id-1', name: 'srv', toolAllowlist: [] }),
instructions: 'use the tools wisely',
},
]),
};
const service = new McpClientsService(repoStub as never, {} as never);
jest
.spyOn(
service as unknown as { connect: (s: FakeServer) => unknown },
'connect',
)
.mockImplementation(() =>
Promise.resolve({
tools: () => Promise.resolve(RAW()),
close: () => Promise.resolve(),
}),
);
const toolset = await service.toolsFor('ws-1');
await Promise.all(toolset.clients.map((c) => c.close()));
expect(Object.keys(toolset.tools)).toEqual([]);
// mergeNamespaced reported 0 contributed tools, so no guidance is attached.
expect(toolset.instructions).toEqual([]);
});
});
@@ -444,13 +444,9 @@ export class McpClientsService {
try {
client = await this.connectWithTimeout(server, CONNECT_TIMEOUT_MS);
const raw = await withTimeout(client.tools(), CONNECT_TIMEOUT_MS);
// Allowlist semantics (#476): null/absent = no restriction (all tools);
// ANY array — including `[]` — is authoritative, so an EMPTY allowlist
// yields ZERO tools (deny-all). Do NOT add a `.length > 0` escape here:
// that read `[]` as falsy and silently widened deny-all to allow-all
// (the repo also fails corrupt rows closed to `[]` for the same reason).
const allow = server.toolAllowlist;
const picked = Array.isArray(allow) ? pick(raw, allow) : raw;
const picked =
Array.isArray(allow) && allow.length > 0 ? pick(raw, allow) : raw;
// Bound each tool's execute with a per-call total-timeout guard before
// merging, so a single chatty-but-stuck call is aborted after the cap.
const guarded = wrapToolsWithCallTimeout(picked, callTimeoutMs);
@@ -100,8 +100,7 @@ export class McpServersService {
transport: dto.transport,
url: dto.url,
headersEnc,
// undefined => unchanged; null => no restriction; `[]` is persisted
// verbatim and means deny-all (#476).
// undefined => unchanged; [] / value handled by repo (empty => null).
toolAllowlist: dto.toolAllowlist,
// undefined => unchanged; blank => cleared (null) by the repo.
instructions: dto.instructions,
@@ -1,266 +0,0 @@
import type { ModelMessage } from 'ai';
import {
resolveReplayBudget,
isContextOverflowError,
estimateMessagesTokens,
trimHistoryForReplay,
REPLAY_BUDGET_DEFAULT_TOKENS,
REPLAY_TRUNCATION_MARKER,
REPLAY_TURN_COLLAPSED_MARKER,
} from './history-budget';
describe('resolveReplayBudget', () => {
it('uses floor(0.7 x window) for a configured window (no cap)', () => {
// 0.7 x 60k = 42k
expect(resolveReplayBudget(60_000)).toEqual({
thresholdTokens: 42_000,
usedDefault: false,
});
// 0.7 x 1M = 700k — NOT capped (anti-brick vs the window, not a cost limiter).
expect(resolveReplayBudget(1_000_000)).toEqual({
thresholdTokens: 700_000,
usedDefault: false,
});
});
it('accepts the raw ::text stored form', () => {
expect(resolveReplayBudget('60000').thresholdTokens).toBe(42_000);
});
// The crux (#490): a chat with NO context window configured must STILL be
// budgeted — those are exactly the installs that hit terminal overflow.
it('applies the flat default when the window is unset/empty', () => {
expect(resolveReplayBudget(undefined)).toEqual({
thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS,
usedDefault: true,
});
expect(resolveReplayBudget('')).toEqual({
thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS,
usedDefault: true,
});
expect(resolveReplayBudget(' ')).toEqual({
thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS,
usedDefault: true,
});
});
it('treats an explicit 0 as the off-switch (distinct from unset)', () => {
expect(resolveReplayBudget(0)).toEqual({
thresholdTokens: null,
usedDefault: false,
});
expect(resolveReplayBudget('0')).toEqual({
thresholdTokens: null,
usedDefault: false,
});
});
it('falls back to the default on a negative/garbage value', () => {
expect(resolveReplayBudget(-5).usedDefault).toBe(true);
expect(resolveReplayBudget('abc').usedDefault).toBe(true);
});
});
describe('isContextOverflowError', () => {
it('classifies a real provider 400 context-overflow shape', () => {
// OpenAI-compatible shape.
expect(
isContextOverflowError({
statusCode: 400,
message:
"This model's maximum context length is 128000 tokens. However, your messages resulted in 214000 tokens. Please reduce the length of the messages.",
}),
).toBe(true);
// Anthropic-style wording.
expect(
isContextOverflowError({
status: 400,
message: 'prompt is too long: 250000 tokens > 200000 maximum',
}),
).toBe(true);
// Nested body + string status.
expect(
isContextOverflowError({
response: { status: '400' },
message: 'input is too long for the requested model',
}),
).toBe(true);
// Error instance with the cause carrying the body.
const e = new Error('Bad request');
(e as any).statusCode = 400;
(e as any).cause = new Error('maximum context window exceeded');
expect(isContextOverflowError(e)).toBe(true);
});
it('does NOT classify unrelated 400s or auth/rate-limit errors', () => {
expect(
isContextOverflowError({ statusCode: 400, message: 'invalid tool schema' }),
).toBe(false);
expect(
isContextOverflowError({
statusCode: 429,
message: 'context length exceeded but rate limited',
}),
).toBe(false);
expect(isContextOverflowError({ statusCode: 500, message: 'server error' })).toBe(
false,
);
expect(isContextOverflowError(undefined)).toBe(false);
expect(isContextOverflowError('some random string')).toBe(false);
});
});
// Helpers to build ModelMessage fixtures in the ai@6 shape.
const userMsg = (text: string): ModelMessage =>
({ role: 'user', content: [{ type: 'text', text }] }) as ModelMessage;
const assistantMsg = (
text: string,
toolCallId?: string,
toolName?: string,
): ModelMessage =>
({
role: 'assistant',
content: [
{ type: 'text', text },
...(toolCallId
? [{ type: 'tool-call', toolCallId, toolName, input: {} }]
: []),
],
}) as ModelMessage;
const toolMsg = (
toolCallId: string,
toolName: string,
value: unknown,
): ModelMessage =>
({
role: 'tool',
content: [
{ type: 'tool-result', toolCallId, toolName, output: { type: 'json', value } },
],
}) as ModelMessage;
describe('trimHistoryForReplay', () => {
it('null budget disables trimming (returns the same reference)', () => {
const msgs = [userMsg('hi'), assistantMsg('yo')];
const r = trimHistoryForReplay(msgs, null);
expect(r.trimmed).toBe(false);
expect(r.messages).toBe(msgs);
});
it('leaves history under budget untouched (same reference)', () => {
const msgs = [userMsg('hi'), assistantMsg('a short answer')];
const r = trimHistoryForReplay(msgs, 100_000);
expect(r.trimmed).toBe(false);
expect(r.messages).toBe(msgs);
});
it('truncates OLD tool outputs but keeps recent turns full', () => {
const big = 'X'.repeat(40_000); // ~16k tokens on its own
const msgs: ModelMessage[] = [];
// 6 OLD turns (indices 0..5), each with a huge tool output.
for (let i = 0; i < 6; i++) {
msgs.push(userMsg(`old q${i}`));
msgs.push(assistantMsg('looking', `c${i}`, 'getPage'));
msgs.push(toolMsg(`c${i}`, 'getPage', { body: big }));
msgs.push(assistantMsg(`old a${i}`));
}
// 3 small recent turns, then the CURRENT turn with its own huge tool output.
// With REPLAY_KEEP_RECENT_TURNS=4 the last 4 user-turns stay full, so only
// these small recent turns + the current big one are kept full; the 6 old
// turns above fall in the trim region.
for (let i = 0; i < 3; i++) {
msgs.push(userMsg(`recent q${i}`));
msgs.push(assistantMsg(`recent a${i}`));
}
msgs.push(userMsg('current q'));
msgs.push(assistantMsg('looking', 'cR', 'getPage'));
msgs.push(toolMsg('cR', 'getPage', { body: big }));
msgs.push(assistantMsg('current a'));
// Budget large enough that phase-1 tool truncation alone brings it under.
const r = trimHistoryForReplay(msgs, 30_000);
expect(r.trimmed).toBe(true);
const flat = JSON.stringify(r.messages);
// The CURRENT turn's tool output survives in full.
expect(flat).toContain(big);
// Old outputs were truncated with the marker.
expect(flat).toContain(REPLAY_TRUNCATION_MARKER);
// Phase 1 sufficed: the oldest turns were NOT collapsed.
expect(flat).not.toContain(REPLAY_TURN_COLLAPSED_MARKER);
expect(estimateMessagesTokens(r.messages)).toBeLessThan(
estimateMessagesTokens(msgs),
);
});
it('collapses the oldest turns when tool truncation is not enough', () => {
// Many turns with LARGE assistant TEXT (not tool output) so phase 1 can't help.
const bigText = 'слово '.repeat(8_000); // large Cyrillic text per turn
const msgs: ModelMessage[] = [];
for (let i = 0; i < 12; i++) {
msgs.push(userMsg(`q${i}`));
msgs.push(assistantMsg(bigText));
}
const r = trimHistoryForReplay(msgs, 30_000);
expect(r.trimmed).toBe(true);
// Oldest turns collapsed; result fits (best-effort) and is much smaller.
expect(estimateMessagesTokens(r.messages)).toBeLessThan(
estimateMessagesTokens(msgs),
);
// The LAST turn's text is preserved in full (recent turns stay full).
expect(JSON.stringify(r.messages[r.messages.length - 1])).toContain(bigText);
});
it('is deterministic / byte-stable for identical inputs', () => {
const big = 'Y'.repeat(30_000);
const build = (): ModelMessage[] => {
const m: ModelMessage[] = [];
for (let i = 0; i < 10; i++) {
m.push(userMsg(`q${i}`));
m.push(assistantMsg('t', `c${i}`, 'getPage'));
m.push(toolMsg(`c${i}`, 'getPage', { body: big }));
}
return m;
};
const a = trimHistoryForReplay(build(), 15_000);
const b = trimHistoryForReplay(build(), 15_000);
expect(JSON.stringify(a.messages)).toBe(JSON.stringify(b.messages));
});
it('never leaves an unpaired tool-call after collapsing (balanced history)', () => {
const big = 'Z'.repeat(40_000);
const msgs: ModelMessage[] = [];
for (let i = 0; i < 10; i++) {
msgs.push(userMsg(`q${i}`));
msgs.push(assistantMsg('t', `c${i}`, 'getPage'));
msgs.push(toolMsg(`c${i}`, 'getPage', { body: big }));
}
const r = trimHistoryForReplay(msgs, 8_000);
// Count tool-call vs tool-result parts in the trimmed output.
let calls = 0;
let results = 0;
for (const m of r.messages) {
if (!Array.isArray(m.content)) continue;
for (const p of m.content as Array<{ type?: string }>) {
if (p.type === 'tool-call') calls++;
if (p.type === 'tool-result' || p.type === 'tool-error') results++;
}
}
// Every surviving tool-call has a surviving result (collapsing drops BOTH).
expect(calls).toBe(results);
// Collapsed turns carry the marker.
expect(JSON.stringify(r.messages)).toContain(REPLAY_TURN_COLLAPSED_MARKER);
});
it('respects the provider fact: under-budget contextTokens skips trimming', () => {
const big = 'W'.repeat(60_000);
const msgs = [
userMsg('q'),
assistantMsg('t', 'c1', 'getPage'),
toolMsg('c1', 'getPage', { body: big }),
];
// char-estimate is high, but the provider says we are well under budget.
const r = trimHistoryForReplay(msgs, 100_000, 5_000);
expect(r.trimmed).toBe(false);
expect(r.messages).toBe(msgs);
});
});
@@ -1,375 +0,0 @@
/**
* History-replay token budget (#490).
*
* The whole persisted conversation is replayed to the provider on EVERY turn, so
* a long chat eventually exceeds the model's context window and the provider 400s
* on every turn terminally (the chat "bricks"). This module bounds the replayed
* history at REPLAY TIME only: it never mutates what is persisted (the DB stays
* the full record), and its output is a deterministic, byte-stable function of its
* input so the trimmed prefix is identical turn to turn (provider prompt-cache
* friendliness real money on long chats).
*
* The PRIMARY signal is the provider's own fact: `metadata.contextTokens` from the
* last turn. The chars-based {@link estimateTokens} (shared with the client) is
* used only for the DELTA of not-yet-sent messages, to decide WHAT to trim, and as
* the fallback for chats with no usage yet.
*/
import type { ModelMessage } from 'ai';
import { estimateTokens } from '@docmost/token-estimate';
/** Flat default budget when no context window is configured (tokens). */
export const REPLAY_BUDGET_DEFAULT_TOKENS = 100_000;
/** Fraction of a configured context window used as the budget. */
export const REPLAY_BUDGET_WINDOW_FRACTION = 0.7;
/**
* Fraction of the normal budget used for the REACTIVE re-trim after a provider
* context-overflow 400 the preventive estimate under-counted, so cut harder.
*/
export const REPLAY_AGGRESSIVE_FRACTION = 0.5;
/**
* Turns (a user message + its assistant/tool replies) kept FULL at the tail,
* including the current one never trimmed. Older turns are compacted first.
*/
export const REPLAY_KEEP_RECENT_TURNS = 4;
/** Leading chars kept from a truncated old tool output. */
export const REPLAY_TOOL_OUTPUT_HEAD = 800;
/** Trailing chars kept from a truncated old tool output. */
export const REPLAY_TOOL_OUTPUT_TAIL = 300;
/** Marker inserted where an old tool output was truncated for replay. */
export const REPLAY_TRUNCATION_MARKER =
'[…truncated for replay; call the tool again to read the full output]';
/** Marker for a whole old turn collapsed to its text. */
export const REPLAY_TURN_COLLAPSED_MARKER =
'[earlier tool activity omitted for replay]';
export interface ReplayBudget {
/** Token threshold above which replay history is trimmed; `null` = OFF. */
thresholdTokens: number | null;
/** True when the flat default was used (no context window configured). */
usedDefault: boolean;
}
/**
* Resolve the replay budget from the RAW stored `chatContextWindow` (text/number).
* - a positive value -> `floor(fraction × window)` (NO cap the budgeter is
* anti-brick protection against the window itself, not a cost/economy limiter,
* exactly as the codebase already treats maxOutputTokens; the reactive branch
* still guarantees anti-brick regardless of how high this budget is)
* - explicit `0` -> OFF (admin opt-out; `null` threshold)
* - unset/empty/invalid-> the flat default (still protects the installations
* that hit terminal overflow are exactly the ones that never set a window)
*
* Note the raw value is needed because the parsed `chatContextWindow` collapses
* both `0` and unset to `undefined`, which would erase the explicit off-switch.
*/
export function resolveReplayBudget(rawContextWindow: unknown): ReplayBudget {
let n: number | undefined;
if (typeof rawContextWindow === 'number') {
n = rawContextWindow;
} else if (typeof rawContextWindow === 'string') {
const t = rawContextWindow.trim();
n = t === '' ? undefined : Number(t);
}
// Unset / empty / non-numeric / negative -> flat default (the protective case).
if (n === undefined || !Number.isFinite(n) || n < 0) {
return { thresholdTokens: REPLAY_BUDGET_DEFAULT_TOKENS, usedDefault: true };
}
// Explicit 0 -> off-switch.
if (n === 0) {
return { thresholdTokens: null, usedDefault: false };
}
return {
thresholdTokens: Math.floor(REPLAY_BUDGET_WINDOW_FRACTION * n),
usedDefault: false,
};
}
/**
* The effective replay threshold for THIS turn, given the base budget and whether
* the PREVIOUS turn hit a context-overflow 400 (the reactive-recovery signal,
* `metadata.replayOverflow`). On recovery the base budget is scaled down by
* {@link REPLAY_AGGRESSIVE_FRACTION}: the overflowing turn produced no usage
* signal, so the preventive estimate under-counted and a normal-threshold trim may
* not shrink enough to fit this harder cut is what un-bricks the chat.
*
* A `null` base budget (trimming OFF) is passed through unchanged: an explicit
* off-switch is never overridden by the recovery path.
*/
export function resolveEffectiveReplayThreshold(
thresholdTokens: number | null,
priorOverflowed: boolean,
): number | null {
if (!priorOverflowed || thresholdTokens == null) return thresholdTokens;
return Math.floor(thresholdTokens * REPLAY_AGGRESSIVE_FRACTION);
}
/**
* True when a provider error is a CONTEXT-OVERFLOW rejection (the prompt exceeds
* the model's window). Providers surface this as an HTTP 400 with a recognizable
* message; match both the status and the message patterns robustly across
* OpenAI-compatible / Anthropic / Gemini wordings, since the exact shape varies.
*/
export function isContextOverflowError(error: unknown): boolean {
const status = extractStatus(error);
const msg = extractMessage(error).toLowerCase();
// Message patterns seen across providers for "prompt too long".
const overflowPattern =
/context (?:length|window)|maximum context|too many tokens|too large for|reduce the length|prompt is too long|input (?:is )?too long|exceeds? the (?:maximum )?(?:context|token)|maximum.*tokens|string too long/;
if (!overflowPattern.test(msg)) return false;
// A 400/413 with an overflow-shaped message is an overflow. Some providers
// omit/rewrite the status, so accept the message match when the status is
// unknown, but reject it for auth/rate-limit statuses that never mean overflow.
if (status === 400 || status === 413) return true;
if (status === 401 || status === 403 || status === 429) return false;
return true;
}
function extractStatus(error: unknown): number | undefined {
if (!error || typeof error !== 'object') return undefined;
const e = error as Record<string, unknown>;
for (const k of ['statusCode', 'status']) {
const v = e[k];
if (typeof v === 'number') return v;
if (typeof v === 'string' && /^\d+$/.test(v)) return Number(v);
}
// Nested (e.g. { response: { status } } / { cause: { statusCode } }).
for (const k of ['response', 'cause', 'data']) {
const nested = e[k];
if (nested && typeof nested === 'object') {
const s = extractStatus(nested);
if (s !== undefined) return s;
}
}
return undefined;
}
function extractMessage(error: unknown): string {
if (error == null) return '';
if (typeof error === 'string') return error;
if (error instanceof Error) {
// Include nested causes (provider libs wrap the real body in `cause`).
const cause = (error as { cause?: unknown }).cause;
return `${error.message} ${cause ? extractMessage(cause) : ''}`;
}
if (typeof error === 'object') {
const e = error as Record<string, unknown>;
const parts: string[] = [];
for (const k of ['message', 'error', 'body', 'responseBody', 'data']) {
const v = e[k];
if (typeof v === 'string') parts.push(v);
else if (v && typeof v === 'object') parts.push(extractMessage(v));
}
return parts.join(' ');
}
return String(error);
}
/** Rough token size of a ModelMessage array via the shared chars estimator. */
export function estimateMessagesTokens(
messages: ReadonlyArray<ModelMessage>,
): number {
let total = 0;
for (const m of messages) {
total += estimateTokens(serializeContent(m.content));
}
return total;
}
function serializeContent(content: unknown): string {
if (typeof content === 'string') return content;
try {
return JSON.stringify(content) ?? '';
} catch {
return '';
}
}
/** Deep JSON string of an arbitrary value, bounded so estimation never throws. */
function stringifyValue(value: unknown): string {
if (typeof value === 'string') return value;
try {
return JSON.stringify(value) ?? String(value);
} catch {
return String(value);
}
}
export interface TrimResult {
messages: ModelMessage[];
/** Whether any trimming was applied. */
trimmed: boolean;
/** Estimated tokens of the returned messages (chars-based). */
estimatedTokens: number;
}
/**
* Bound the replayed history to `budgetTokens`, deterministically. Returns the
* SAME array reference (no copy) when nothing needs trimming, so the common case
* is free and byte-identical. Trimming order (spec #490):
* 1. truncate OLD turns' tool outputs (head+tail + marker) the bulk of the size
* 2. mechanically collapse the OLDEST turns to their text (concatenation, no LLM)
* 3. the current + last {@link REPLAY_KEEP_RECENT_TURNS} turns stay FULL
*
* `budgetTokens === null` disables trimming. `priorContextTokens` (the provider's
* fact from last turn) short-circuits the decision: when it is known and already
* under budget we skip trimming even if the char-estimate is higher (the provider
* count is authoritative). The char-estimate drives WHAT to cut.
*/
export function trimHistoryForReplay(
messages: ModelMessage[],
budgetTokens: number | null,
priorContextTokens?: number,
): TrimResult {
if (budgetTokens == null) {
return { messages, trimmed: false, estimatedTokens: 0 };
}
const estimated = estimateMessagesTokens(messages);
// Decision signal: prefer the provider's fact (last turn's contextTokens) plus
// the estimated delta of the messages appended since; fall back to the pure
// char-estimate for a chat with no usage yet.
const projected =
priorContextTokens != null
? Math.max(priorContextTokens, estimated)
: estimated;
if (projected <= budgetTokens) {
return { messages, trimmed: false, estimatedTokens: estimated };
}
// The tail we always keep full: from the Nth-from-last user message onward.
const boundary = recentBoundaryIndex(messages, REPLAY_KEEP_RECENT_TURNS);
const tail = messages.slice(boundary);
let head = messages.slice(0, boundary).map(cloneMessage);
// Phase 1: truncate old tool outputs.
for (const m of head) {
if (m.role === 'tool') truncateToolMessage(m);
}
let out = [...head, ...tail];
let est = estimateMessagesTokens(out);
if (est <= budgetTokens) {
return { messages: out, trimmed: true, estimatedTokens: est };
}
// Phase 2: collapse the oldest turns (in `head`) to their text, one at a time,
// from the oldest, until we fit or the whole head is collapsed.
const turns = splitTurns(head);
const collapsed: ModelMessage[] = [];
let i = 0;
for (; i < turns.length; i++) {
if (est <= budgetTokens) break;
collapsed.push(...collapseTurn(turns[i]));
// Re-estimate the whole prospective output.
const remaining = turns.slice(i + 1).flat();
out = [...collapsed, ...remaining, ...tail];
est = estimateMessagesTokens(out);
}
// Include any turns we didn't need to collapse.
const remaining = turns.slice(i).flat();
out = [...collapsed, ...remaining, ...tail];
est = estimateMessagesTokens(out);
return { messages: out, trimmed: true, estimatedTokens: est };
}
/** Index of the first message of the Nth-from-last user turn (0 if fewer). */
function recentBoundaryIndex(
messages: ReadonlyArray<ModelMessage>,
keepTurns: number,
): number {
const userIdx: number[] = [];
for (let i = 0; i < messages.length; i++) {
if (messages[i].role === 'user') userIdx.push(i);
}
if (userIdx.length <= keepTurns) return 0;
return userIdx[userIdx.length - keepTurns];
}
/** Split a message list into turns; each turn starts at a `user` message. */
function splitTurns(messages: ModelMessage[]): ModelMessage[][] {
const turns: ModelMessage[][] = [];
for (const m of messages) {
if (m.role === 'user' || turns.length === 0) turns.push([m]);
else turns[turns.length - 1].push(m);
}
return turns;
}
/**
* Collapse a whole turn to its plain text (mechanical concatenation, not an LLM
* summary). Keeps the user message; replaces the assistant/tool messages with a
* single assistant text message = the assistant's concatenated text + a marker
* when tool activity was dropped. Dropping BOTH the tool-call and tool-result
* parts together keeps the rebuilt history balanced (no unpaired calls).
*/
function collapseTurn(turn: ModelMessage[]): ModelMessage[] {
const out: ModelMessage[] = [];
let assistantText = '';
let hadTools = false;
for (const m of turn) {
if (m.role === 'user') {
out.push(m);
} else if (m.role === 'assistant') {
const { text, tools } = extractAssistantText(m.content);
assistantText += text;
hadTools = hadTools || tools;
} else if (m.role === 'tool') {
hadTools = true;
} else {
out.push(m);
}
}
const note =
(assistantText ? assistantText.trimEnd() : '') +
(hadTools
? `${assistantText ? '\n\n' : ''}${REPLAY_TURN_COLLAPSED_MARKER}`
: '');
if (note) out.push({ role: 'assistant', content: note } as ModelMessage);
return out;
}
function extractAssistantText(content: unknown): {
text: string;
tools: boolean;
} {
if (typeof content === 'string') return { text: content, tools: false };
if (!Array.isArray(content)) return { text: '', tools: false };
let text = '';
let tools = false;
for (const part of content) {
const type = (part as { type?: string })?.type;
if (type === 'text') text += (part as { text?: string }).text ?? '';
else if (type === 'tool-call') tools = true;
}
return { text, tools };
}
/** Truncate every tool-result output in a `tool` message to head+tail+marker. */
function truncateToolMessage(message: ModelMessage): void {
const content = message.content;
if (!Array.isArray(content)) return;
for (const part of content) {
const p = part as { type?: string; output?: { type?: string; value?: unknown } };
if (p.type !== 'tool-result' && p.type !== 'tool-error') continue;
if (!p.output) continue;
const raw = stringifyValue(p.output.value);
const budget = REPLAY_TOOL_OUTPUT_HEAD + REPLAY_TOOL_OUTPUT_TAIL;
if (raw.length <= budget + REPLAY_TRUNCATION_MARKER.length) continue;
const truncated =
raw.slice(0, REPLAY_TOOL_OUTPUT_HEAD) +
`\n${REPLAY_TRUNCATION_MARKER}\n` +
raw.slice(raw.length - REPLAY_TOOL_OUTPUT_TAIL);
// Represent the shrunk output as a text output (a valid tool-result output).
p.output = { type: 'text', value: truncated };
}
}
/** Shallow-ish clone so trimming never mutates the caller's (persisted-derived)
* message objects only the OLD region is cloned before it is edited. */
function cloneMessage(m: ModelMessage): ModelMessage {
if (typeof m.content === 'string') return { ...m };
return {
...m,
content: (m.content as unknown[]).map((p) =>
p && typeof p === 'object' ? { ...(p as object) } : p,
),
} as ModelMessage;
}
@@ -9,73 +9,9 @@ import {
DEGENERATION_CHECK_STEP,
REPEATED_LINES_THRESHOLD,
MIN_PERIOD_REPEATS,
degenerationThresholds,
} from './output-degeneration';
import { AiChatService } from './ai-chat.service';
// Part A (#495 iter10): the detector thresholds are env-tunable. These drive the
// resolver against real repeat-count shapes and mutation-verify that the env
// override actually changes the trigger point (not a vacuous read).
describe('degeneration thresholds are env-configurable', () => {
const VARS = [
'AI_CHAT_DEGENERATION_REPEATED_LINES',
'AI_CHAT_DEGENERATION_PERIOD_MAX_LEN',
'AI_CHAT_DEGENERATION_PERIOD_MIN_REPEATS',
'AI_CHAT_DEGENERATION_CHECK_STEP',
];
const saved: Record<string, string | undefined> = {};
beforeEach(() => {
for (const v of VARS) saved[v] = process.env[v];
});
afterEach(() => {
for (const v of VARS) {
if (saved[v] === undefined) delete process.env[v];
else process.env[v] = saved[v];
}
});
it('defaults to the compiled constants when unset', () => {
for (const v of VARS) delete process.env[v];
expect(degenerationThresholds()).toEqual({
repeatedLines: REPEATED_LINES_THRESHOLD,
maxPeriodLen: 150,
minPeriodRepeats: MIN_PERIOD_REPEATS,
checkStep: DEGENERATION_CHECK_STEP,
});
});
it('falls back to the default on blank / invalid / non-positive values', () => {
for (const bad of ['', ' ', 'abc', '0', '-3', '1.5']) {
process.env.AI_CHAT_DEGENERATION_REPEATED_LINES = bad;
// '1.5' floors to 1 (still ≥1, valid); every other bad value → default.
const expected = bad === '1.5' ? 1 : REPEATED_LINES_THRESHOLD;
expect(degenerationThresholds().repeatedLines).toBe(expected);
}
});
it('a RAISED check-step suppresses a burst the default would have flagged', () => {
// A ~3.3KB periodic burst is periodic-degenerate, but shouldCheckDegeneration
// is the throttle gate. Default checkStep=2000 arms on it; raising the step
// above the burst size means the throttle never re-fires for it.
const burstLen = 'loadTools.\n'.repeat(300).length; // ~3300
delete process.env.AI_CHAT_DEGENERATION_CHECK_STEP;
expect(shouldCheckDegeneration(burstLen, 0)).toBe(true); // default 2000
process.env.AI_CHAT_DEGENERATION_CHECK_STEP = String(burstLen + 1);
expect(shouldCheckDegeneration(burstLen, 0)).toBe(false); // raised gate
});
it('a LOWERED repeated-lines threshold trips on a shorter identical-line run', () => {
// 8 identical lines: below the default 25 (rule 1) and below the periodic
// rule's 20 repeats — so isDegenerateOutput is false by default.
const shortRun = 'x\n'.repeat(8);
delete process.env.AI_CHAT_DEGENERATION_REPEATED_LINES;
expect(isDegenerateOutput(shortRun)).toBe(false);
// Lower rule 1 to 5 → the 8-line run now trips.
process.env.AI_CHAT_DEGENERATION_REPEATED_LINES = '5';
expect(isDegenerateOutput(shortRun)).toBe(true);
});
});
// Mock ONLY streamText so we can capture the onChunk/onStepFinish callbacks the
// service registers and drive them by hand; every other `ai` export the service
// uses (convertToModelMessages, stepCountIs, …) stays real.
@@ -23,54 +23,6 @@ export const MAX_PERIOD_LEN = 150;
/** Rule 2: minimum number of consecutive block repeats to trigger. */
export const MIN_PERIOD_REPEATS = 20;
/**
* Read a positive-integer threshold from an env var, falling back to `fallback`
* on unset/blank/invalid/non-positive. Mirrors the `AI_STREAM_PRE_RESPONSE_RETRIES`
* resolver in `ai-streaming-fetch.ts`: read the RAW string first so a blank value
* is treated as "unset" ( fallback) rather than coercing to 0. Thresholds must
* stay 1 a 0/negative would make the detector fire on any text (or never), so
* a bad value degrades to the safe compiled default instead. Env-tunable so an
* operator can retune the anti-babble guard (#444) without a redeploy, following
* the `AI_CHAT_FINAL_STEP_LOCKDOWN` toggle convention.
*/
function envThreshold(name: string, fallback: number): number {
const rawStr = process.env[name];
if (rawStr === undefined || rawStr.trim() === '') return fallback;
const raw = Number(rawStr);
return Number.isFinite(raw) && raw >= 1 ? Math.floor(raw) : fallback;
}
/**
* Resolve the degeneration-detector thresholds from the environment, each
* defaulting to the compiled constant above. Read fresh per call (not cached at
* import) so a test or a runtime env change takes effect deterministically.
*/
export function degenerationThresholds(): {
repeatedLines: number;
maxPeriodLen: number;
minPeriodRepeats: number;
checkStep: number;
} {
return {
repeatedLines: envThreshold(
'AI_CHAT_DEGENERATION_REPEATED_LINES',
REPEATED_LINES_THRESHOLD,
),
maxPeriodLen: envThreshold(
'AI_CHAT_DEGENERATION_PERIOD_MAX_LEN',
MAX_PERIOD_LEN,
),
minPeriodRepeats: envThreshold(
'AI_CHAT_DEGENERATION_PERIOD_MIN_REPEATS',
MIN_PERIOD_REPEATS,
),
checkStep: envThreshold(
'AI_CHAT_DEGENERATION_CHECK_STEP',
DEGENERATION_CHECK_STEP,
),
};
}
/**
* Rule 1 `REPEATED_LINES_THRESHOLD` consecutive IDENTICAL non-empty lines at
* the tail. Catches the classic newline-delimited loop ("loadTools.\n" ×N).
@@ -176,11 +128,7 @@ export function hasPeriodicTail(
* Pure the caller owns the abort side effect.
*/
export function isDegenerateOutput(text: string): boolean {
const cfg = degenerationThresholds();
return (
hasRepeatedLineRun(text, cfg.repeatedLines) ||
hasPeriodicTail(text, cfg.maxPeriodLen, cfg.minPeriodRepeats)
);
return hasRepeatedLineRun(text) || hasPeriodicTail(text);
}
/**
@@ -206,7 +154,7 @@ export function shouldCheckDegeneration(
textLen: number,
lastCheckLen: number,
): boolean {
return textLen - lastCheckLen >= degenerationThresholds().checkStep;
return textLen - lastCheckLen >= DEGENERATION_CHECK_STEP;
}
/**
@@ -307,10 +307,6 @@ export class PublicShareChatService {
system,
messages: modelMessages,
tools,
// Pin the AI SDK per-request retry budget explicitly (matches the SDK
// default of 2). Connection arithmetic: (1 + maxRetries) × (1 +
// AI_STREAM_PRE_RESPONSE_RETRIES) worst-case connects per turn.
maxRetries: 2,
// Bound the agent loop for anonymous callers.
stopWhen: stepCountIs(5),
// Cap per-request output so one anonymous call cannot run up the provider
@@ -426,7 +426,6 @@ export class AiChatToolsService {
const {
sharedToolSpecs,
createCommentSignalTracker,
createListCommentsProbe,
searchShapes,
getGuideSection,
} = await loadDocmostMcp();
@@ -719,11 +718,7 @@ export class AiChatToolsService {
if (spec.mcpOnly) continue;
if (spec.inlineBothHosts) continue;
const run = spec.inAppExecute ?? spec.execute;
// Guaranteed present by assertEverySpecIsRegisterable() (#494), which runs
// at tool-specs module load and throws if a non-inline spec the in-app host
// registers carries neither inAppExecute nor execute — so this can no longer
// silently drop a mis-declared tool. Kept as a type-narrowing guard.
if (!run) continue;
if (!run) continue; // defensive: a shared spec always carries one of them.
tools[spec.inAppKey] = sharedTool(
spec,
(async (args) =>
@@ -769,21 +764,35 @@ export class AiChatToolsService {
// wrapper below) so the race governs the whole call. The client carries the
// per-call composite signal via setToolAbortSignal.
const capMs = inAppToolCallCapMs();
// The signal needs BOTH the tracker factory AND the shared count-source probe
// factory (#494). Either being absent (a stale @docmost/mcp build or a mocked
// loader) => signal disabled, tool results byte-identical.
if (!createCommentSignalTracker || !createListCommentsProbe) {
if (!createCommentSignalTracker) {
return wrapInAppToolsWithCap(tools, client, capMs);
}
// Shared probe (#494): the SAME factory the standalone MCP host uses, so the
// in-app probe body is no longer a hand-mirror that could drift (counting the
// full feed newer than the watermark, labelling a hit with the light page
// title). `client` supplies the loopback listComments/getPageRaw reads.
const tracker = createCommentSignalTracker({
probe: createListCommentsProbe(
client as unknown as Parameters<typeof createListCommentsProbe>[0],
),
probe: async (pageId: string, sinceMs: number) => {
const { items } = await client.listComments(pageId, true);
const count = (items as Array<{ createdAt?: string }>).filter((c) => {
const created = c?.createdAt ? new Date(c.createdAt).getTime() : NaN;
return Number.isFinite(created) && created > sinceMs;
}).length;
let title: string | undefined;
if (count > 0) {
// Title labels the signal; untrusted, defanged by the shared builder.
// Fetched only on a hit so the no-signal path never pays for it. Uses
// the LIGHT raw page info (title only) — mirroring the standalone MCP
// probe's getPageRaw — instead of the heavy getPage (which also renders
// Markdown + subpages) just to read one field.
try {
const res = (await client.getPageRaw(pageId)) as {
title?: string;
} | null;
title = res?.title ?? undefined;
} catch {
// Title is optional — omit it when the page can't be fetched.
}
}
return { count, title };
},
});
return wrapInAppToolsWithCap(
@@ -21,10 +21,7 @@ import { SHARED_TOOL_SPECS } from '../../../../../../packages/mcp/src/tool-specs
// The REAL shared tracker factory, imported from source (same cross-boundary
// approach the tool-specs spec uses) so the in-app wiring is exercised against
// exactly the watermark/debounce/injection-safe logic the package ships.
import {
createCommentSignalTracker,
createListCommentsProbe,
} from '../../../../../../packages/mcp/src/comment-signal';
import { createCommentSignalTracker } from '../../../../../../packages/mcp/src/comment-signal';
// The REAL client-side citation extractor: proves that the passive signal does
// NOT strip a tool's citations (the #417 in-app regression this spec guards).
import { toolCitations } from '../../../../../../apps/client/src/features/ai-chat/utils/tool-parts';
@@ -287,13 +284,9 @@ describe('AiChatToolsService forUser + comment signal (real tracker)', () => {
return fakeClient as DocmostClientLike;
} as unknown as loader.DocmostClientCtor,
sharedToolSpecs: SHARED_TOOL_SPECS as unknown as Record<string, loader.SharedToolSpec>,
// Wire the REAL factories so the in-app path is exercised end to end
// including the shared count-source probe (#494) the service now builds the
// tracker's `probe` from.
// Wire the REAL factory so the in-app path is exercised end to end.
createCommentSignalTracker:
createCommentSignalTracker as unknown as loader.CommentSignalTrackerFactory,
createListCommentsProbe:
createListCommentsProbe as unknown as loader.CreateListCommentsProbeFn,
// Pure no-network draw.io helpers (#424) — required on the loader return;
// this comment-signal test doesn't exercise them, so no-op stubs suffice.
searchShapes: (() => []) as unknown as loader.SearchShapesFn,
@@ -150,27 +150,6 @@ export type CommentSignalTrackerFactory = (options: {
debounceMs?: number;
}) => CommentSignalTrackerLike;
/**
* Local mirror of `@docmost/mcp`'s `createListCommentsProbe` (#494): the SHARED
* count-source probe both hosts use, so the in-app probe body is no longer a
* hand-copy of the standalone MCP one. Given a client with the light comment feed
* + raw-page-title reads, it returns the tracker's `probe` (count comments newer
* than the watermark, label a hit with the page title). Loosely typed at this
* cross-package boundary, like the rest of this loader.
*/
export type CreateListCommentsProbeFn = (client: {
listComments(
pageId: string,
includeResolved: boolean,
): Promise<{ items: Array<{ createdAt?: string | null }> }>;
getPageRaw(
pageId: string,
): Promise<{ title?: string | null } | null | undefined>;
}) => (
pageId: string,
sinceMs: number,
) => Promise<CommentSignalProbeResultLike>;
// Pure, no-network draw.io helpers (#424). These are plain functions on the
// module (NOT DocmostClient methods) — the in-app AI-SDK service calls them
// directly to wire drawioShapes / drawioGuide, mirroring the MCP server.
@@ -191,10 +170,6 @@ interface DocmostMcpModule {
// loader in unit tests. The in-app layer treats an absent factory as "signal
// disabled" — a pure no-op that leaves tool results byte-identical.
createCommentSignalTracker?: CommentSignalTrackerFactory;
// Optional (#494): the shared count-source probe factory. Absent on a pre-#494
// build or a mocked loader; the in-app layer only builds a probe when the
// signal factory above is also present.
createListCommentsProbe?: CreateListCommentsProbeFn;
// Optional (#447): a deterministic hash of the tool-specs registry content,
// generated into build/ by the package's build. Absent on a pre-#447 build (or
// the mocked loader in unit tests) — the stale-check below is a NO-OP when it
@@ -309,7 +284,6 @@ export async function loadDocmostMcp(): Promise<{
DocmostClient: DocmostClientCtor;
sharedToolSpecs: Record<string, SharedToolSpec>;
createCommentSignalTracker?: CommentSignalTrackerFactory;
createListCommentsProbe?: CreateListCommentsProbeFn;
searchShapes: SearchShapesFn;
getGuideSection: GetGuideSectionFn;
}> {
@@ -357,9 +331,6 @@ export async function loadDocmostMcp(): Promise<{
// Optional: forwarded when present so the in-app layer can build the passive
// comment signal (#417); undefined on a stale build => signal disabled.
createCommentSignalTracker: mod.createCommentSignalTracker,
// Optional (#494): the shared count-source probe factory; undefined on a
// stale build => the in-app layer falls back to no signal.
createListCommentsProbe: mod.createListCommentsProbe,
// Pure no-network draw.io helpers (#424); not client methods.
searchShapes: mod.searchShapes,
getGuideSection: mod.getGuideSection,
@@ -17,10 +17,9 @@ import { MovePageDto } from './move-page.dto';
// a valid ordering key the server itself generated would be refused on move.
//
// The tests below assert the CORRECT contract: any key the generator can produce
// must satisfy the DTO. FIXED (#495 item 9): the DTO now validates `position` by
// CHARSET ([0-9A-Za-z], the generator's base-62 alphabet) instead of the wrong
// @MaxLength(12) length bound, so dense between-inserts are accepted; the former
// `test.failing` bug-lock is now a passing assertion.
// must satisfy the DTO. The genuinely-failing case is marked `test.failing` so the
// suite stays green while locking the bug; it flips red (alerting us) once the DTO
// bounds are widened to cover the generator's real range.
function constraintErrors(position: unknown) {
const dto = plainToInstance(MovePageDto, {
@@ -48,33 +47,24 @@ describe('MovePageDto.position vs generateJitteredKeyBetween parity', () => {
expect(hasError(errors, 'position')).toBe(false);
});
// FIXED: dense between-inserts produce keys longer than 12 chars, which the old
// MaxLength(12) rejected even though they are valid ordering keys. Now accepted.
it('accepts dense between-inserted keys longer than 12 chars', async () => {
let lo = generateJitteredKeyBetween(null, null);
let hi = generateJitteredKeyBetween(lo, null);
// Repeatedly insert just above `lo`, shrinking the gap so the key grows. The
// generator is JITTERED (random), so use enough iterations that the longest
// key reliably clears the old 12-char bound: measured min-over-50-trials is
// ~11 at 40 iterations (flaky) but ~36 at 200 (robust margin).
let longest = lo;
for (let i = 0; i < 200; i++) {
const mid = generateJitteredKeyBetween(lo, hi);
if (mid.length > longest.length) longest = mid;
hi = mid;
}
expect(longest.length).toBeGreaterThan(12); // sanity: we produced a long key
const errors = await constraintErrors(longest);
expect(hasError(errors, 'position')).toBe(false);
});
// The charset guard replaces the length bound: reject anything outside the
// generator's [0-9A-Za-z] alphabet (control chars, separators, injection) and
// the empty string, while still accepting every real key.
it('rejects a position with characters outside the fractional-index alphabet', async () => {
for (const bad of ['a0/b', 'a b', 'a\n0', 'a.b', '', "a';--"]) {
const errors = await constraintErrors(bad);
expect(hasError(errors, 'position')).toBe(true);
}
});
// BUG LOCK: dense between-inserts produce keys longer than 12 chars, which
// MaxLength(12) rejects even though they are valid ordering keys. This SHOULD
// pass; it currently fails. Flips green when the DTO bound is fixed.
test.failing(
'accepts dense between-inserted keys (currently rejected by MaxLength(12))',
async () => {
let lo = generateJitteredKeyBetween(null, null);
let hi = generateJitteredKeyBetween(lo, null);
// Repeatedly insert just above `lo`, shrinking the gap so the key grows.
let longest = lo;
for (let i = 0; i < 40; i++) {
const mid = generateJitteredKeyBetween(lo, hi);
if (mid.length > longest.length) longest = mid;
hi = mid;
}
expect(longest.length).toBeGreaterThan(12); // sanity: we produced a long key
const errors = await constraintErrors(longest);
expect(hasError(errors, 'position')).toBe(false);
},
);
});
+3 -13
View File
@@ -1,8 +1,8 @@
import {
IsString,
IsOptional,
MinLength,
MaxLength,
Matches,
IsNotEmpty,
} from 'class-validator';
@@ -10,19 +10,9 @@ export class MovePageDto {
@IsString()
pageId: string;
// `position` is a fractional-indexing key from `generateJitteredKeyBetween`
// (the SAME generator page.service uses). Validate by CHARSET, not length: the
// generator's default base-62 alphabet is [0-9A-Za-z], and DENSE between-inserts
// legitimately grow a key well past a dozen chars (measured >40), so the old
// @MinLength(5)/@MaxLength(12) bounds wrongly 400'd valid ordering keys the
// server itself produced (Gitea #139 item 6). The charset regex rejects control
// chars / separators / injection, and a generous MaxLength stays only as a
// DoS guard — far above any realistic key, so it never rejects a real move.
@IsString()
@Matches(/^[0-9A-Za-z]+$/, {
message: 'position must be a fractional-index key ([0-9A-Za-z])',
})
@MaxLength(256)
@MinLength(5)
@MaxLength(12)
position: string;
@IsOptional()
@@ -1,49 +0,0 @@
import 'reflect-metadata';
import { plainToInstance } from 'class-transformer';
import { validate } from 'class-validator';
import { PageIdDto } from './page.dto';
// #435: PageIdDto.pageId carries a page's DOUBLE identity (internal UUID OR
// public 10-char slugId), both as bare strings. The DTO must accept exactly
// those two FORMATS and reject a malformed / swapped identity at the boundary.
async function pageIdErrors(pageId: unknown) {
const dto = plainToInstance(PageIdDto, { pageId });
const errors = await validate(dto as object);
return errors.some((e) => e.property === 'pageId');
}
const UUID = '019f499a-9f8c-7d68-b7be-ce100d7c6c56';
const SLUG = 'aB3xQ7kR2p';
describe('PageIdDto pageId format validation', () => {
it('accepts a canonical page UUID', async () => {
expect(await pageIdErrors(UUID)).toBe(false);
});
it('accepts a 10-char slugId', async () => {
expect(await pageIdErrors(SLUG)).toBe(false);
});
it('rejects a truncated / wrong-length slug', async () => {
expect(await pageIdErrors('aB3xQ7kR2')).toBe(true); // 9 chars
expect(await pageIdErrors('aB3xQ7kR2pX')).toBe(true); // 11 chars
});
it('rejects a slug with an illegal character', async () => {
expect(await pageIdErrors('aB3xQ7kR2!')).toBe(true);
});
it('rejects a full URL / path-shaped identity (not the bare id)', async () => {
expect(await pageIdErrors(`my-page-title-${SLUG}`)).toBe(true);
expect(await pageIdErrors(`https://x/p/${SLUG}`)).toBe(true);
});
it('rejects a malformed UUID', async () => {
expect(await pageIdErrors('019f499a-9f8c-7d68-b7be')).toBe(true);
expect(await pageIdErrors('not-a-uuid-at-all-really')).toBe(true);
});
it('rejects an empty string', async () => {
expect(await pageIdErrors('')).toBe(true);
});
});
@@ -1,39 +0,0 @@
import { applyDecorators } from '@nestjs/common';
import { Matches, ValidationOptions } from 'class-validator';
/**
* A page identity at the API boundary is EITHER the internal page UUID or the
* public 10-char slugId (page.repo.findById matches a non-UUID input as a
* slugId). Both arrive as bare strings, which is exactly how the two got swapped
* silently (incident family #435). This regex pins the two accepted FORMATS so a
* malformed / cross-wired identity (a truncated slug, a full URL, an email, an
* id from another entity kind that isn't even shaped like either) is rejected at
* the boundary instead of falling through to a confusing 404.
*
* - UUID: canonical 8-4-4-4-12 hex (version-agnostic page ids are UUIDv7,
* so only the shape/length is enforced, matching the MCP's UUID_RE
* and the server's isValidUUID acceptance).
* - slugId: exactly 10 chars over [0-9A-Za-z] (nanoid `generateSlugId`).
*
* The two are disjoint (a UUID is 36 chars WITH dashes, a slugId 10 chars
* WITHOUT), so a value can only satisfy one branch.
*/
export const PAGE_ID_OR_SLUG_ID_REGEX =
/^(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9A-Za-z]{10})$/i;
/**
* Validate that a DTO string field is a well-formed page identity (page UUID OR
* 10-char slugId). Composed decorator so the same format rule is applied
* consistently wherever a DTO accepts a `pageId` that may be either form.
*/
export function IsPageIdOrSlugId(
validationOptions?: ValidationOptions,
): PropertyDecorator {
return applyDecorators(
Matches(PAGE_ID_OR_SLUG_ID_REGEX, {
message:
'pageId must be a page UUID or a 10-character slugId',
...validationOptions,
}),
);
}
@@ -9,16 +9,10 @@ import {
import { Transform } from 'class-transformer';
import { ContentFormat } from './create-page.dto';
import { IsPageIdOrSlugId } from './page-identity.validator';
export class PageIdDto {
@IsString()
@IsNotEmpty()
// Format-validate the double identity (#435): accept only a page UUID or a
// 10-char slugId so a malformed / swapped identity is rejected at the boundary
// rather than passed to the repo as a bare string. Base for PageInfoDto,
// DeletePageDto, BacklinksListDto, AddLabelsDto/RemoveLabelDto, etc.
@IsPageIdOrSlugId()
pageId: string;
}
@@ -53,10 +53,8 @@ import {
extractPageSlugId,
} from '../../../integrations/export/utils';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import {
markdownToProseMirror,
normalizeForeignMarkdown,
} from '@docmost/prosemirror-markdown';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../../../integrations/import/utils/foreign-markdown';
import { WatcherService } from '../../watcher/watcher.service';
import { sql } from 'kysely';
import { TransclusionService } from '../transclusion/transclusion.service';
@@ -1,38 +1,19 @@
import { TemporaryNoteCleanupService } from '../temporary-note-cleanup.service';
/**
* Chainable Kysely stub for the temporary-note sweep.
*
* `this.db` serves the non-locking candidate SELECT (selectFrom/select/where/
* limit/execute -> the configured expired rows) AND `.transaction().execute(cb)`,
* which runs `cb` with a separate `trx` builder. The `trx` builder serves the
* per-row LOCKED re-check (selectFrom/select/where/forUpdate/skipLocked/
* executeTakeFirst). `lockedRows` drives what that locked re-check returns per
* candidate an id/creator/workspace row means "still expired, delete it";
* `undefined` means the predicate no longer matched (made permanent / re-armed /
* already trashed) or the row was SKIP-LOCKED by another worker, so it is skipped.
* Chainable Kysely stub that records every `.where(...)` call so the test can
* assert the sweep only selects armed, expired, not-yet-trashed notes. The
* terminal `.execute()` resolves the configured expired rows (the batch SELECT);
* `.executeTakeFirst()` resolves the per-row deadline re-read done just before
* each `removePage`. By default the re-read reports the note as still armed and
* still expired (epoch deadline < now), so the sweep proceeds to delete it;
* tests override `reReadFirst` to simulate a concurrent "Make permanent".
*/
function makeDbStub(expiredRows: any[], lockedRows?: any[]) {
function makeDbStub(expiredRows: any[]) {
const whereCalls: any[][] = [];
const locked = [
...(lockedRows ??
expiredRows.map((r) => ({
id: r.id,
creatorId: r.creatorId,
workspaceId: r.workspaceId,
}))),
];
const lockedTakeFirst = jest.fn(() => Promise.resolve(locked.shift()));
const forUpdate = jest.fn(() => trxBuilder);
const skipLocked = jest.fn(() => trxBuilder);
const trxBuilder: any = {
selectFrom: jest.fn(() => trxBuilder),
select: jest.fn(() => trxBuilder),
where: jest.fn(() => trxBuilder),
forUpdate,
skipLocked,
executeTakeFirst: lockedTakeFirst,
};
const reReadFirst = jest
.fn()
.mockResolvedValue({ temporaryExpiresAt: new Date(0), deletedAt: null });
const builder: any = {
selectFrom: jest.fn(() => builder),
select: jest.fn(() => builder),
@@ -42,11 +23,9 @@ function makeDbStub(expiredRows: any[], lockedRows?: any[]) {
}),
limit: jest.fn(() => builder),
execute: jest.fn().mockResolvedValue(expiredRows),
transaction: jest.fn(() => ({
execute: (cb: (trx: any) => Promise<any>) => Promise.resolve(cb(trxBuilder)),
})),
executeTakeFirst: reReadFirst,
};
return { builder, whereCalls, lockedTakeFirst, forUpdate, skipLocked };
return { builder, whereCalls, reReadFirst };
}
describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
@@ -73,36 +52,20 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
expect(builder.limit.mock.calls[0][0]).toBeGreaterThan(0);
});
it('soft-deletes each expired note via removePage under a row lock, attributed to its creator', async () => {
it('soft-deletes each expired note via removePage, attributed to its creator', async () => {
const expired = [
{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' },
{ id: 'p2', creatorId: 'u2', workspaceId: 'w1' },
];
const { builder, forUpdate, skipLocked } = makeDbStub(expired);
const { builder } = makeDbStub(expired);
const pageRepo = { removePage: jest.fn().mockResolvedValue(undefined) } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).toHaveBeenCalledTimes(2);
// The 4th arg is the locking transaction — the delete runs inside it.
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
1,
'p1',
'u1',
'w1',
expect.anything(),
);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
2,
'p2',
'u2',
'w1',
expect.anything(),
);
// The re-check acquired a FOR UPDATE SKIP LOCKED lock (once per candidate).
expect(forUpdate).toHaveBeenCalledTimes(2);
expect(skipLocked).toHaveBeenCalledTimes(2);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(1, 'p1', 'u1', 'w1');
expect(pageRepo.removePage).toHaveBeenNthCalledWith(2, 'p2', 'u2', 'w1');
});
it('continues past a failing note (one bad removePage does not abort the sweep)', async () => {
@@ -123,29 +86,60 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
service.sweepExpiredTemporaryNotes(),
).resolves.toBeUndefined();
expect(pageRepo.removePage).toHaveBeenCalledTimes(2);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(
2,
'good',
'u2',
'w1',
expect.anything(),
);
expect(pageRepo.removePage).toHaveBeenNthCalledWith(2, 'good', 'u2', 'w1');
});
it('does NOT trash a note made permanent / re-armed / already trashed (locked re-check returns nothing)', async () => {
// The batch SELECT saw the note as expired, but by the time the LOCKED
// re-check runs the row no longer matches the still-armed+expired+not-trashed
// predicate (make-permanent, re-arm to a future deadline, or already trashed),
// OR another worker holds the row (SKIP LOCKED). In every case the locked
// SELECT returns nothing and the delete is skipped so the keep/other worker wins.
it('does NOT trash a note made permanent in the race window', async () => {
// The batch SELECT saw the note as expired, but before its turn in the loop
// the user clicked "Make permanent" (temporary_expires_at -> null). The
// deadline re-read must catch this and skip the delete so the keep wins.
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, lockedTakeFirst } = makeDbStub(expired, [undefined]);
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: null,
deletedAt: null,
});
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(lockedTakeFirst).toHaveBeenCalledTimes(1);
expect(reReadFirst).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('skips a note already trashed since the batch SELECT', async () => {
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: new Date(0),
deletedAt: new Date(),
});
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('does NOT trash a note re-armed to a future deadline in the race window', async () => {
// The batch SELECT saw the note as expired, but before its turn in the loop
// the user disarmed it and re-armed it to a fresh, still-future deadline
// (temporary_expires_at -> now + 1h). The deadline re-read must catch that
// the note is no longer expired and skip the delete so the keep wins.
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder, reReadFirst } = makeDbStub(expired);
reReadFirst.mockResolvedValueOnce({
temporaryExpiresAt: new Date(Date.now() + 60 * 60 * 1000),
deletedAt: null,
});
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.sweepExpiredTemporaryNotes();
expect(reReadFirst).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
@@ -157,31 +151,4 @@ describe('TemporaryNoteCleanupService.sweepExpiredTemporaryNotes', () => {
await service.sweepExpiredTemporaryNotes();
expect(pageRepo.removePage).not.toHaveBeenCalled();
});
it('sweeps once on application bootstrap (catches notes expired during downtime)', async () => {
const expired = [{ id: 'p1', creatorId: 'u1', workspaceId: 'w1' }];
const { builder } = makeDbStub(expired);
const pageRepo = { removePage: jest.fn().mockResolvedValue(undefined) } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await service.onApplicationBootstrap();
expect(pageRepo.removePage).toHaveBeenCalledTimes(1);
expect(pageRepo.removePage).toHaveBeenCalledWith(
'p1',
'u1',
'w1',
expect.anything(),
);
});
it('a startup-sweep failure never blocks application boot', async () => {
const { builder } = makeDbStub([]);
// Make the candidate SELECT throw to simulate a boot-time DB hiccup.
builder.execute.mockRejectedValueOnce(new Error('db not ready'));
const pageRepo = { removePage: jest.fn() } as any;
const service = new TemporaryNoteCleanupService(builder, pageRepo);
await expect(service.onApplicationBootstrap()).resolves.toBeUndefined();
});
});
@@ -1,13 +1,8 @@
import {
Injectable,
Logger,
OnApplicationBootstrap,
} from '@nestjs/common';
import { Injectable, Logger } from '@nestjs/common';
import { Interval } from '@nestjs/schedule';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { executeTx } from '@docmost/db/utils';
/**
* Background sweeper for temporary notes ("structure or die"). A note whose
@@ -16,7 +11,7 @@ import { executeTx } from '@docmost/db/utils';
* TrashCleanupService; `@nestjs/schedule` is already enabled globally.
*/
@Injectable()
export class TemporaryNoteCleanupService implements OnApplicationBootstrap {
export class TemporaryNoteCleanupService {
private readonly logger = new Logger(TemporaryNoteCleanupService.name);
// Cap a single sweep so a large backlog (e.g. many notes created during
@@ -29,20 +24,6 @@ export class TemporaryNoteCleanupService implements OnApplicationBootstrap {
private readonly pageRepo: PageRepo,
) {}
// Sweep once at startup so notes that expired during downtime are trashed
// right away instead of waiting up to an hour for the first @Interval tick.
// Best-effort: never let a startup-sweep failure block application boot.
async onApplicationBootstrap() {
try {
await this.sweepExpiredTemporaryNotes();
} catch (error) {
this.logger.error(
'Temporary-note startup sweep failed',
error instanceof Error ? error.stack : undefined,
);
}
}
// Hourly granularity: lifetimes are configured in hours, so a sub-hour
// overshoot past the deadline is acceptable.
@Interval('temporary-note-cleanup', 60 * 60 * 1000)
@@ -50,11 +31,9 @@ export class TemporaryNoteCleanupService implements OnApplicationBootstrap {
try {
const now = new Date();
// Candidate ids (non-locking). The authoritative re-check happens per row
// under a row lock below, so this cheap pass just bounds the batch.
const expired = await this.db
.selectFrom('pages')
.select(['id'])
.select(['id', 'creatorId', 'workspaceId'])
.where('temporaryExpiresAt', 'is not', null)
.where('temporaryExpiresAt', '<', now)
.where('deletedAt', 'is', null) // not already in trash
@@ -62,53 +41,50 @@ export class TemporaryNoteCleanupService implements OnApplicationBootstrap {
.execute();
let trashed = 0;
for (const candidate of expired) {
for (const page of expired) {
try {
const didTrash = await executeTx(this.db, async (trx) => {
// Re-check the row UNDER A LOCK inside the transaction. `FOR UPDATE
// SKIP LOCKED`:
// - serialises against a concurrent "Make permanent"
// (toggleTemporary UPDATE takes the same row lock): if it commits
// first, the deadline predicate below no longer matches and we
// skip; if we lock first, it waits until this delete commits.
// - SKIP LOCKED lets a second worker/instance skip a row another
// sweeper already claimed instead of blocking on it (no double
// processing, no thundering herd).
// The predicate re-asserts still-armed AND still-expired AND
// not-already-trashed, so a make-permanent / prior sweep drops the row.
const locked = await trx
.selectFrom('pages')
.select(['id', 'creatorId', 'workspaceId'])
.where('id', '=', candidate.id)
.where('temporaryExpiresAt', 'is not', null)
.where('temporaryExpiresAt', '<', now)
.where('deletedAt', 'is', null)
.forUpdate()
.skipLocked()
.executeTakeFirst();
// Re-check the deadline at deletion time. The SELECT above is not
// transactional, so a user may click "Make permanent"
// (toggleTemporary sets temporary_expires_at = null) in the window
// between the SELECT and this per-row removePage. removePage deletes
// by id with only a `deletedAt IS NULL` filter and never re-reads the
// deadline, so without this guard a concurrently-kept note would
// still be trashed. Re-read the row and skip it unless it is still
// armed AND still expired, so a concurrent make-permanent wins.
const current = await this.db
.selectFrom('pages')
.select(['temporaryExpiresAt', 'deletedAt'])
.where('id', '=', page.id)
.executeTakeFirst();
if (!locked) return false;
if (
!current ||
current.deletedAt !== null ||
current.temporaryExpiresAt === null ||
new Date(current.temporaryExpiresAt) >= now
) {
// Made permanent, already trashed, or no longer expired since the
// SELECT — leave it alone.
continue;
}
// Reuse the exact soft-delete path (recursive children + share
// removal + PAGE_SOFT_DELETED broadcast), running IN this locked
// transaction so the delete is atomic with the re-check and cannot
// deadlock on a nested independent transaction. The broadcast is
// deferred by removePage to this transaction's commit. Attribute the
// automatic deletion to the note's creator (no schema change).
await this.pageRepo.removePage(
locked.id,
// creatorId is set on every created page; a temporary note always
// has one. Cast to satisfy the non-null deletedById parameter.
locked.creatorId as string,
locked.workspaceId,
trx,
);
return true;
});
if (didTrash) trashed++;
// Reuse the exact soft-delete path: recursive over children, removes
// shares in a transaction, and emits PAGE_SOFT_DELETED (tree
// invalidation + watcher notifications). Attribute the automatic
// deletion to the note's creator (no schema change). Both the SELECT
// above and removePage filter `deletedAt IS NULL`, so a double sweep
// is idempotent.
await this.pageRepo.removePage(
page.id,
// creatorId is set on every created page; a temporary note always
// has one. Cast to satisfy the non-null deletedById parameter.
page.creatorId as string,
page.workspaceId,
);
trashed++;
} catch (error) {
this.logger.error(
`Failed to trash expired temporary note ${candidate.id}`,
`Failed to trash expired temporary note ${page.id}`,
error instanceof Error ? error.stack : undefined,
);
}
@@ -133,9 +133,6 @@ describe('ShareAliasController authz gates', () => {
creatorId: 'u-1',
alias: 'promo',
confirmReassign: true,
// The requesting user is forwarded so setAlias can gate the reassign
// 409 title disclosure on target-page view permission (#495).
user,
});
expect(result).toEqual({ id: 'alias-1' });
});
@@ -79,9 +79,6 @@ export class ShareAliasController {
creatorId: user.id,
alias: dto.alias,
confirmReassign: dto.confirmReassign,
// Gates whether the reassign 409 may reveal the current target's title
// (view-permission check on that page) — see setAlias (#495).
user,
});
}
@@ -1,8 +1,4 @@
import {
BadRequestException,
ConflictException,
ForbiddenException,
} from '@nestjs/common';
import { BadRequestException, ConflictException } from '@nestjs/common';
import { NoResultError } from 'kysely';
import { ShareAliasService } from './share-alias.service';
@@ -11,8 +7,6 @@ import { ShareAliasService } from './share-alias.service';
* 409 reassign guard, uniqueness-race handling, availability probe, and the
* request-time readable-target resolution (which re-runs the share boundary).
*/
const USER = { id: 'u-1' } as any;
describe('ShareAliasService', () => {
// Sentinel handed to repo calls so tests can assert they ran inside the tx.
const trx = { __trx: true };
@@ -33,10 +27,6 @@ describe('ShareAliasService', () => {
resolveReadableSharePage: jest.fn(),
isSharingAllowed: jest.fn(),
};
// Default: the requester CAN view the target page (validateCanView resolves),
// so the reassign 409 may disclose its title. Tests override to reject to
// assert the no-leak path.
const pageAccessService = { validateCanView: jest.fn().mockResolvedValue(undefined) };
// Fake kysely db: only .transaction().execute(cb) is used by setAlias.
const db = {
transaction: jest.fn(() => ({
@@ -47,10 +37,9 @@ describe('ShareAliasService', () => {
shareAliasRepo as any,
pageRepo as any,
shareService as any,
pageAccessService as any,
db as any,
);
return { service, shareAliasRepo, pageRepo, shareService, pageAccessService, db };
return { service, shareAliasRepo, pageRepo, shareService, db };
}
describe('setAlias', () => {
@@ -61,7 +50,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'A', // too short + uppercase
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -78,7 +66,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: ' My Page ',
});
@@ -127,7 +114,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'ted',
});
@@ -158,7 +144,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
@@ -194,7 +179,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'new',
});
@@ -206,77 +190,30 @@ describe('ShareAliasService', () => {
);
});
it('throws 409 with the target TITLE (never its id) when the requester CAN view it', async () => {
const { service, shareAliasRepo, pageRepo, pageAccessService } =
makeService();
it('throws 409 with current target when name is taken and not confirmed', async () => {
const { service, shareAliasRepo, pageRepo } = makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
alias: 'foo',
pageId: 'p-other',
});
pageRepo.findById.mockResolvedValue({ id: 'p-other', title: 'Other' });
pageAccessService.validateCanView.mockResolvedValue(undefined); // can view
try {
await service.setAlias({
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
} catch (err) {
expect(err).toBeInstanceOf(ConflictException);
const body = (err as ConflictException).getResponse();
expect(body).toMatchObject({
expect((err as ConflictException).getResponse()).toMatchObject({
code: 'ALIAS_REASSIGN_REQUIRED',
currentPageId: 'p-other',
currentPageTitle: 'Other',
});
// SECURITY (#495): the page id is NEVER disclosed, even to a viewer.
expect(body).not.toHaveProperty('currentPageId');
expect(pageAccessService.validateCanView).toHaveBeenCalledWith(
expect.objectContaining({ id: 'p-other' }),
USER,
);
}
expect(shareAliasRepo.updatePageId).not.toHaveBeenCalled();
});
it('throws 409 WITHOUT the title or id when the requester CANNOT view the target (#495)', async () => {
const { service, shareAliasRepo, pageRepo, pageAccessService } =
makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
alias: 'foo',
pageId: 'p-secret',
});
pageRepo.findById.mockResolvedValue({ id: 'p-secret', title: 'Secret' });
// No view permission on the target page -> validateCanView throws.
pageAccessService.validateCanView.mockRejectedValue(
new ForbiddenException(),
);
try {
await service.setAlias({
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
} catch (err) {
expect(err).toBeInstanceOf(ConflictException);
const body = (err as ConflictException).getResponse() as Record<
string,
unknown
>;
expect(body).toMatchObject({ code: 'ALIAS_REASSIGN_REQUIRED' });
// The enumeration hole: neither the id nor the title of a page the
// requester cannot see may leak.
expect(body).not.toHaveProperty('currentPageId');
expect(body.currentPageTitle ?? null).toBeNull();
}
expect(shareAliasRepo.updatePageId).not.toHaveBeenCalled();
});
@@ -294,7 +231,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
confirmReassign: true,
});
@@ -333,7 +269,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -359,7 +294,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -383,7 +317,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -413,7 +346,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
});
fail('expected ConflictException');
@@ -443,7 +375,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
confirmReassign: true,
});
@@ -475,7 +406,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'ted',
});
fail('expected ConflictException');
@@ -498,7 +428,6 @@ describe('ShareAliasService', () => {
workspaceId: 'ws-1',
pageId: 'p-1',
creatorId: 'u-1',
user: USER,
alias: 'foo',
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -521,22 +450,18 @@ describe('ShareAliasService', () => {
alias: 'free-name',
valid: true,
available: true,
currentPageId: null,
});
// SECURITY (#495): the availability probe must NOT leak any page id.
expect(res).not.toHaveProperty('currentPageId');
});
it('reports taken WITHOUT leaking the current target page id (#495)', async () => {
it('reports taken with the current target page', async () => {
const { service, shareAliasRepo } = makeService();
shareAliasRepo.findByAliasAndWorkspace.mockResolvedValue({
id: 'a-1',
pageId: 'p-9',
});
const res = await service.checkAvailability('taken', 'ws-1');
expect(res).toMatchObject({ available: false });
// The row exists (available:false) but its pageId is never returned — an
// authenticated member cannot map an alias name to a page id it can't view.
expect(res).not.toHaveProperty('currentPageId');
expect(res).toMatchObject({ available: false, currentPageId: 'p-9' });
});
});
@@ -7,8 +7,7 @@ import {
import { ShareAliasRepo } from '@docmost/db/repos/share-alias/share-alias.repo';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { ShareService } from './share.service';
import { PageAccessService } from '../page/page-access/page-access.service';
import { Page, ShareAlias, User } from '@docmost/db/types/entity.types';
import { Page, ShareAlias } from '@docmost/db/types/entity.types';
import { isValidShareAlias, normalizeShareAlias } from './share-alias.util';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
@@ -44,7 +43,6 @@ export class ShareAliasService {
private readonly shareAliasRepo: ShareAliasRepo,
private readonly pageRepo: PageRepo,
private readonly shareService: ShareService,
private readonly pageAccessService: PageAccessService,
@InjectKysely() private readonly db: KyselyDB,
) {}
@@ -57,13 +55,9 @@ export class ShareAliasService {
* `/l/<old>` link survives
* - name already points at pageId -> no-op (idempotent)
* - name points at ANOTHER page -> the "swap". Without confirmReassign
* we throw 409 so the client can confirm. SECURITY (#495): the 409 reveals
* the current target's title ONLY when `user` may VIEW that page, and never
* its id otherwise any member with one editable+shared page could iterate
* alias names with confirmReassign=false and map them to (id, title) of
* pages they cannot see. With confirmReassign we UPDATE the single row's
* page_id (every /l/<alias> link follows the 302 to the new page instantly
* no stale cache).
* we throw 409 carrying the current target so the client can confirm;
* with it we UPDATE the single row's page_id (every /l/<alias> link
* follows the 302 to the new page instantly no stale cache).
*
* To keep the invariant self-healing we DELETE every other alias row still
* pointing at this page (a legacy duplicate, or the target page's own former
@@ -83,12 +77,8 @@ export class ShareAliasService {
creatorId: string;
alias: string;
confirmReassign?: boolean;
// The requesting user — used ONLY to gate whether the reassign 409 may reveal
// the current target page's title (view-permission check). Not an authz gate
// for the write itself (the controller already validated edit on `pageId`).
user: User;
}): Promise<ShareAlias> {
const { workspaceId, pageId, creatorId, confirmReassign, user } = opts;
const { workspaceId, pageId, creatorId, confirmReassign } = opts;
const alias = normalizeShareAlias(opts.alias);
if (!isValidShareAlias(alias)) {
throw new BadRequestException(
@@ -107,30 +97,14 @@ export class ShareAliasService {
// The name is occupied by a DIFFERENT (or dangling) target page.
if (byName && byName.pageId !== pageId) {
if (!confirmReassign) {
// SECURITY (#495): only disclose the current target's TITLE, and only
// when the requester may VIEW that page. Never disclose its id (the
// client's confirm-reassign UX doesn't use it, and it is an enumerable
// identity). A member with one editable+shared page must NOT be able to
// iterate alias names and map them to (id, title) of pages they cannot
// see. When view is denied (or the alias is dangling) the 409 is the
// bare "occupied" fact — the client still shows a generic confirm modal.
const currentPage = byName.pageId
? await this.pageRepo.findById(byName.pageId)
: null;
let currentPageTitle: string | null = null;
if (currentPage) {
try {
await this.pageAccessService.validateCanView(currentPage, user);
currentPageTitle = currentPage.title ?? null;
} catch {
// No view permission on the target -> do not reveal its title.
currentPageTitle = null;
}
}
throw new ConflictException({
message: 'Alias already in use',
code: 'ALIAS_REASSIGN_REQUIRED',
currentPageTitle,
currentPageId: byName.pageId,
currentPageTitle: currentPage?.title ?? null,
});
}
// Confirmed swap. ORDER MATTERS: the partial unique index on
@@ -249,27 +223,21 @@ export class ShareAliasService {
alias: string;
valid: boolean;
available: boolean;
currentPageId: string | null;
}> {
const alias = normalizeShareAlias(rawAlias);
if (!isValidShareAlias(alias)) {
return { alias, valid: false, available: false };
return { alias, valid: false, available: false, currentPageId: null };
}
const existing = await this.shareAliasRepo.findByAliasAndWorkspace(
alias,
workspaceId,
);
// SECURITY (#495): return ONLY the boolean availability. The previous shape
// leaked `currentPageId` — the id of whatever page the alias already targets —
// to ANY authenticated workspace member, with no view-permission check on that
// page. An attacker could enumerate alias names and map them to page ids they
// have no access to. The taken/free bit is all the "is this address free"
// probe needs. The reassign flow (setAlias 409) may surface the target's
// TITLE, but only behind a `validateCanView` on that page (see setAlias); it
// never returns the page id.
return {
alias,
valid: true,
available: !existing,
currentPageId: existing?.pageId ?? null,
};
}
@@ -24,54 +24,6 @@ export const ALLOWED_RATINGS = new Set<string>([
'poor',
]);
// The ONLY route labels accepted. The endpoint is anonymous, so an un-checked
// `route` is a free-text write surface (arbitrary high-cardinality strings /
// injected text into the metrics table). The client only ever sends a label from
// a finite template dictionary (`templateRoute`), so we drop anything not in it.
//
// PARITY: this MUST mirror `KNOWN_ROUTE_TEMPLATES` in the client's
// `apps/client/src/lib/telemetry/route-template.ts` (the canonical source). A
// drift means legit client routes get dropped — keep the two in lockstep; the
// client self-consistency test asserts `templateRoute` only emits these values.
export const ALLOWED_ROUTE_TEMPLATES = new Set<string>([
'/',
'other',
// Static routes.
'/home',
'/spaces',
'/favorites',
'/login',
'/forgot-password',
'/password-reset',
'/setup/register',
'/settings/account/profile',
'/settings/account/preferences',
'/settings/workspace',
'/settings/ai',
'/settings/members',
'/settings/groups',
'/settings/spaces',
'/settings/sharing',
// Dynamic templates (slugs/ids are already collapsed to `:param`).
'/share/:shareId/p/:slug',
'/share/p/:slug',
'/share/:shareId',
'/p/:slug',
'/s/:space/p/:slug',
'/s/:space/trash',
'/s/:space',
'/labels/:label',
'/invites/:invitationId',
'/settings/groups/:groupId',
]);
// `attr` is a web-vitals attribution TARGET: a CSS-selector-ish string (an
// element path like `html>body>div#app>button.cta`), never free prose. Constrain
// it to a conservative CSS-selector charset so the anonymous endpoint cannot be
// used to write arbitrary text / PII / markup into the metrics table. A value
// containing anything outside this set is DROPPED (-> null); the event is kept.
export const ATTR_ALLOWED_CHARSET = /^[A-Za-z0-9#.\-_> :()[\]="'*+~,]+$/;
// Max events accepted per batch; the rest are ignored.
export const MAX_EVENTS_PER_BATCH = 50;
@@ -125,20 +77,14 @@ export function sanitizeVitalEvent(
? e.rating
: null;
// route: accept ONLY a known template label (dictionary check), else drop to
// null. The length cap stays as a cheap pre-guard before the Set lookup.
let route: string | null = null;
if (typeof e.route === 'string' && e.route.length > 0) {
const candidate = e.route.slice(0, MAX_ROUTE_LENGTH);
route = ALLOWED_ROUTE_TEMPLATES.has(candidate) ? candidate : null;
route = e.route.slice(0, MAX_ROUTE_LENGTH);
}
// attr: truncate, then accept ONLY if it is a CSS-selector-shaped string
// (charset whitelist); anything with characters outside the set is dropped.
let attr: string | null = null;
if (typeof e.attr === 'string' && e.attr.length > 0) {
const candidate = e.attr.slice(0, MAX_ATTR_LENGTH);
attr = ATTR_ALLOWED_CHARSET.test(candidate) ? candidate : null;
attr = e.attr.slice(0, MAX_ATTR_LENGTH);
}
let docSize: number | null = null;
@@ -90,44 +90,6 @@ describe('VitalsService.buildRows', () => {
expect(rows[0].attr).toHaveLength(MAX_ATTR_LENGTH);
});
it('keeps a known route template but DROPS an unknown/free-text route (#495)', () => {
const rows = svc.buildRows(
{
events: [
{ name: 'INP', value: 1, route: '/s/:space/p/:slug' }, // known
{ name: 'INP', value: 2, route: '/s/acme-corp/p/secret-slug' }, // raw path (slugs) — not a template
{ name: 'INP', value: 3, route: 'DROP TABLE client_metrics;--' }, // injected free text
{ name: 'INP', value: 4, route: '/home' }, // known static
],
},
WS,
);
expect(rows.map((r) => r.route)).toEqual([
'/s/:space/p/:slug',
null, // raw path dropped
null, // free text dropped
'/home',
]);
});
it('DROPS an attr that is not a CSS-selector-shaped string (#495)', () => {
const rows = svc.buildRows(
{
events: [
{ name: 'INP', value: 1, attr: 'div#app>button.cta' }, // valid selector
{ name: 'INP', value: 2, attr: 'user@example.com wrote a note' }, // free text / PII
{ name: 'INP', value: 3, attr: '<script>alert(1)</script>' }, // markup
],
},
WS,
);
expect(rows.map((r) => r.attr)).toEqual([
'div#app>button.cta',
null,
null,
]);
});
it('caps the batch at 50 events', () => {
const events = Array.from({ length: 200 }, () => ({ name: 'CLS', value: 1 }));
const rows = svc.buildRows({ events }, WS);
@@ -1,114 +0,0 @@
import * as path from 'path';
import { readFileSync } from 'fs';
// Mock ONLY kysely's `sql.raw(...).execute()` so we can observe what
// ensureConcurrentIndexes runs and how it handles failures, without a DB.
const execMock = jest.fn((_db: unknown) => Promise.resolve(undefined));
const rawMock = jest.fn((_stmt: string) => ({ execute: execMock }));
jest.mock('kysely', () => {
const actual = jest.requireActual('kysely');
return { ...actual, sql: { ...actual.sql, raw: rawMock } };
});
import { CONCURRENT_INDEXES, ensureConcurrentIndexes } from './concurrent-indexes';
describe('ensureConcurrentIndexes', () => {
const fakeDb = { __topLevelKysely: true } as never;
beforeEach(() => {
execMock.mockReset().mockResolvedValue(undefined);
rawMock.mockClear();
});
it('redefines the inlinable f_unaccent BEFORE building any index, then builds each CONCURRENTLY outside a transaction', async () => {
const onLog = jest.fn();
await ensureConcurrentIndexes(fakeDb, onLog);
// One f_unaccent redefine + one statement per index.
expect(rawMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
const statements = rawMock.mock.calls.map((c) => c[0] as string);
// ORDER: the f_unaccent redefine MUST be first — otherwise an existing
// tenant's old 2-arg f_unaccent makes every CONCURRENTLY build fail.
expect(statements[0]).toContain('CREATE OR REPLACE FUNCTION f_unaccent');
expect(statements[0]).toContain('SELECT public.unaccent($1)');
expect(statements[0]).not.toContain('CREATE INDEX');
// The rest are the CONCURRENTLY index builds.
for (const stmt of statements.slice(1)) {
expect(stmt).toContain('CREATE INDEX CONCURRENTLY');
expect(stmt).toContain('IF NOT EXISTS');
}
// Executed against the top-level db (a transaction would forbid CONCURRENTLY).
for (const call of execMock.mock.calls) {
expect(call[0]).toBe(fakeDb);
}
expect(onLog).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
});
it('still builds the indexes when the f_unaccent redefine fails (fresh DB, best-effort)', async () => {
// Redefine (first execute) throws; the index loop must still run.
execMock
.mockRejectedValueOnce(new Error('extension "unaccent" does not exist'))
.mockResolvedValue(undefined);
const onLog = jest.fn();
await expect(
ensureConcurrentIndexes(fakeDb, onLog),
).resolves.toBeUndefined();
// 1 redefine attempt + one per index.
expect(execMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
const errored = onLog.mock.calls.filter((c) => c[1] !== undefined);
expect(errored).toHaveLength(1);
expect(String(errored[0][1])).toContain('unaccent');
});
it('is best-effort: a failing index does not abort the rest and is reported', async () => {
// Redefine ok; fail the FIRST index; the remaining ones must still be tried.
execMock
.mockResolvedValueOnce(undefined) // f_unaccent redefine
.mockRejectedValueOnce(new Error('relation "pages" does not exist'))
.mockResolvedValue(undefined);
const onLog = jest.fn();
await expect(
ensureConcurrentIndexes(fakeDb, onLog),
).resolves.toBeUndefined();
expect(execMock).toHaveBeenCalledTimes(CONCURRENT_INDEXES.length + 1);
const errored = onLog.mock.calls.filter((c) => c[1] !== undefined);
expect(errored).toHaveLength(1);
expect(String(errored[0][1])).toContain('does not exist');
});
});
// DRIFT GUARD: each CONCURRENT_INDEXES entry pre-builds an index that a plain
// migration ALSO creates with `IF NOT EXISTS`. If the two expressions diverge,
// Postgres would treat them as different indexes and the pre-build would NOT
// make the migration a no-op. Assert the migration files still contain each
// index's functional expression.
describe('CONCURRENT_INDEXES parity with the migrations', () => {
const migrationsDir = path.join(__dirname, 'migrations');
const files = [
'20260705T120000-perf-indexes.ts',
'20260706T120000-search-lookup-trgm.ts',
].map((f) => readFileSync(path.join(migrationsDir, f), 'utf8'));
const allMigrationSrc = files.join('\n');
it.each(CONCURRENT_INDEXES.map((i) => [i.name, i]))(
'migration source still creates %s with the same expression',
(_name, idx) => {
// Extract the `USING gin ((...expr...) gin_trgm_ops)` tail from the
// canonical create and assert the migration source contains it verbatim.
const m = (idx as { create: string }).create.match(
/ON \w+ (USING gin .+)$/,
);
expect(m).not.toBeNull();
const expr = (m as RegExpMatchArray)[1];
expect(allMigrationSrc).toContain(expr);
// And the migration must build it by the SAME index name.
expect(allMigrationSrc).toContain(
`CREATE INDEX IF NOT EXISTS ${(idx as { name: string }).name}`,
);
},
);
});
@@ -1,142 +0,0 @@
import { Kysely, sql } from 'kysely';
/**
* Indexes that MUST be built with `CREATE INDEX CONCURRENTLY` so an auto-deploy
* migration never takes a `SHARE` lock that blocks writes on a hot table
* (`pages`) for the potentially minutes-long GIN trigram build (#495 item 12).
*
* Kysely runs each migration INSIDE a transaction (Postgres has transactional
* DDL), and `CREATE INDEX CONCURRENTLY` cannot run inside a transaction block, so
* these cannot live in an ordinary migration. Instead {@link ensureConcurrentIndexes}
* builds them out-of-band (no transaction) BEFORE the migrator runs; the matching
* migrations keep a plain `CREATE INDEX IF NOT EXISTS` as a backstop, which then
* no-ops because the index already exists. So:
* - existing prod DB, incremental deploy: the pre-build FIRST redefines
* `f_unaccent` to its index-inlinable 1-arg form (see below), THEN builds each
* index CONCURRENTLY (no write lock); the migration's IF NOT EXISTS no-ops
* the write-blocking build is gone;
* - fresh DB (or a DB whose `pages` / `unaccent` extension does not exist yet):
* the pre-build fails and is swallowed (best-effort), and the migration builds
* the index normally on an empty/small table where the lock is irrelevant.
*
* NOT a strict "worst case = previous behaviour":
* - The f_unaccent form matters. The trigram expressions use `LOWER(f_unaccent(col))`.
* The INDEX-INLINABLE 1-arg `f_unaccent(text)` (`SELECT public.unaccent($1)`)
* is created INSIDE migration 20260705, which runs AFTER this pre-build. On an
* existing tenant the live `f_unaccent` is still the OLD 2-arg form from
* 20250729, which Postgres CANNOT inline into a CONCURRENTLY index expression
* (`function unaccent(unknown, text) does not exist ... during inlining`) the
* build fails outright. So this pre-build redefines `f_unaccent` to the 1-arg
* form FIRST (idempotent, output-identical, in lockstep with 20260705). Without
* that redefine the whole feature is dead-on-arrival for the very case it
* targets.
* - An INTERRUPTED `CREATE INDEX CONCURRENTLY` (killed pod, cancelled query) with
* a working `f_unaccent` leaves an INVALID index behind. A subsequent name-based
* `IF NOT EXISTS` in BOTH this pre-build and the migration backstop sees the
* name and skips, so the invalid index is NEVER repaired automatically and the
* query keeps seq-scanning until an operator `DROP INDEX`es it. The old
* in-transaction build could not leave an invalid index (a failed tx rolled the
* whole index back), so this is a genuinely new failure mode, not "= previous".
* (A future hardening could `DROP` an `indisvalid = false` index before
* rebuilding; not done here.)
*
* The `create` text is the CANONICAL definition it MUST match the migration's
* `IF NOT EXISTS` create expression exactly (same functional expression + opclass)
* or Postgres would treat them as two different indexes.
*/
export const CONCURRENT_INDEXES: ReadonlyArray<{
name: string;
create: string;
}> = [
{
// #348 perf-indexes — pages.title trigram (coalesce-free functional expr).
name: 'idx_pages_title_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_title_trgm ' +
'ON pages USING gin ((LOWER(f_unaccent(title))) gin_trgm_ops)',
},
{
// #348 perf-indexes — users.name trigram (member search-suggest).
name: 'idx_users_name_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_name_trgm ' +
'ON users USING gin ((LOWER(f_unaccent(name))) gin_trgm_ops)',
},
{
// #443 search-lookup-trgm — pages.text_content trigram (the slow, large one).
name: 'idx_pages_text_content_trgm',
create:
'CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_text_content_trgm ' +
'ON pages USING gin ((LOWER(f_unaccent(text_content))) gin_trgm_ops)',
},
];
/**
* Idempotent, output-identical redefinition of `f_unaccent` to the INDEX-INLINABLE
* 1-arg form. MUST stay byte-for-byte in lockstep with migration
* `20260705T120000-perf-indexes.ts` (same signature + body): the trigram indexes
* above only build CONCURRENTLY once `f_unaccent(text)` inlines, and on an existing
* tenant it is still the old 2-arg form until that migration runs which is AFTER
* this pre-build.
*/
const F_UNACCENT_REDEF = `
CREATE OR REPLACE FUNCTION f_unaccent(text)
RETURNS text
LANGUAGE sql
IMMUTABLE PARALLEL SAFE STRICT
AS $func$
SELECT public.unaccent($1);
$func$
`;
/**
* Best-effort, non-transactional pre-build of {@link CONCURRENT_INDEXES}. Run
* BEFORE the migrator so the blocking `CREATE INDEX` in the corresponding
* migration becomes an `IF NOT EXISTS` no-op.
*
* `db` MUST be the top-level Kysely instance (NOT a transaction): each statement
* then executes on its own connection with no surrounding `BEGIN`, which is
* required for `CONCURRENTLY`. Every statement is independent and swallowed on
* error: the migration backstop still builds the index, so a failure here is
* never fatal. `onLog` reports progress/failures for the caller to route to its
* logger.
*
* ORDER MATTERS: redefine `f_unaccent` to the inlinable form BEFORE the index
* loop otherwise an existing tenant's old 2-arg `f_unaccent` makes every
* CONCURRENTLY build fail and the feature is a no-op (see the module docstring).
*/
export async function ensureConcurrentIndexes(
db: Kysely<any>,
onLog?: (message: string, error?: unknown) => void,
): Promise<void> {
// Make f_unaccent inlinable FIRST. On a fresh DB (no `unaccent` extension yet)
// this throws harmlessly and is swallowed — the migration builds everything.
try {
await sql.raw(F_UNACCENT_REDEF).execute(db);
onLog?.('f_unaccent redefined to the index-inlinable 1-arg form');
} catch (error) {
onLog?.(
'f_unaccent redefine skipped (fresh DB / no unaccent extension) — ' +
'the migration will define it and build the indexes',
error,
);
}
for (const idx of CONCURRENT_INDEXES) {
try {
await sql.raw(idx.create).execute(db);
onLog?.(`Concurrent index ensured: ${idx.name}`);
} catch (error) {
// Non-fatal by design — the migration's IF NOT EXISTS create is the
// backstop. Benign on a fresh DB (the table/extension does not exist yet).
// NOT benign, but still swallowed, on an existing tenant whose f_unaccent
// could not be redefined above (then the migration builds the index NON-
// concurrently under the SHARE lock this feature meant to avoid).
onLog?.(
`Concurrent index pre-build skipped for ${idx.name} ` +
`(will fall back to the in-migration build)`,
error,
);
}
}
}
@@ -1,113 +0,0 @@
import { executeTx, registerAfterCommit } from './utils';
import { KyselyDB, KyselyTransaction } from './types/kysely.types';
// Post-commit hook contract (#495 item 13): a side effect registered via
// registerAfterCommit must run ONLY AFTER the owning transaction commits, and a
// hook registered against a passed-through existingTrx must fire at the OUTER
// commit boundary — never inside the inner call. We fake the Kysely transaction
// runner so the ordering is observable without a real DB.
/**
* A minimal db whose `.transaction().execute(cb)` records the commit ORDER: it
* runs `cb(trx)`, pushes 'commit' onto `log` (simulating the real commit that
* happens after the callback resolves), then returns the callback's result.
*/
function fakeDb(log: string[]): { db: KyselyDB; trx: KyselyTransaction } {
const trx = { __fakeTrx: true } as unknown as KyselyTransaction;
const db = {
transaction: () => ({
execute: async (cb: (t: KyselyTransaction) => Promise<unknown>) => {
const result = await cb(trx);
log.push('commit');
return result;
},
}),
} as unknown as KyselyDB;
return { db, trx };
}
describe('executeTx post-commit hooks', () => {
it('runs an afterCommit hook only AFTER the transaction commits', async () => {
const log: string[] = [];
const { db } = fakeDb(log);
await executeTx(db, async (trx) => {
log.push('body');
registerAfterCommit(trx, () => {
log.push('hook');
});
// The hook must NOT have run yet — the tx is still open.
expect(log).toEqual(['body']);
});
// Order proves post-commit: body → commit → hook (never body → hook → commit).
expect(log).toEqual(['body', 'commit', 'hook']);
});
it('drains hooks registered against a passed-through existingTrx at the OUTER commit', async () => {
const log: string[] = [];
const { db, trx: outerTrx } = fakeDb(log);
await executeTx(db, async (outer) => {
// Nested executeTx reuses the outer trx: it must NOT commit or drain now.
await executeTx(
db,
async (inner) => {
registerAfterCommit(inner, () => {
log.push('inner-hook');
});
},
outer,
);
// Still inside the outer tx — the inner hook has not fired.
expect(log).toEqual([]);
});
// The single (outer) commit drains the hook registered on the shared trx.
expect(log).toEqual(['commit', 'inner-hook']);
// Sanity: the trx the hooks were registered against is the outer one.
expect(outerTrx).toBeDefined();
});
it('a hook failure does not reject the already-committed executeTx', async () => {
const log: string[] = [];
const { db } = fakeDb(log);
await expect(
executeTx(db, async (trx) => {
registerAfterCommit(trx, () => {
throw new Error('cache del blew up');
});
registerAfterCommit(trx, () => {
log.push('second-hook-still-runs');
});
return 'ok';
}),
).resolves.toBe('ok');
// The throwing hook is swallowed; a later hook still runs.
expect(log).toEqual(['commit', 'second-hook-still-runs']);
});
it('does NOT run afterCommit hooks when the transaction body throws (rollback)', async () => {
// The body rejects -> the fake transaction never pushes 'commit' and
// db.transaction().execute() rejects, mirroring a real rolled-back tx. The
// drain runs only AFTER the awaited (committed) transaction, so a rollback
// must leave every registered hook UN-run — otherwise a cache-bust / event
// would fire for a write that never landed.
const log: string[] = [];
const { db } = fakeDb(log);
const hook = jest.fn();
await expect(
executeTx(db, async (trx) => {
registerAfterCommit(trx, hook);
throw new Error('write failed -> rollback');
}),
).rejects.toThrow('write failed -> rollback');
// No commit happened, and the post-commit hook never ran.
expect(log).toEqual([]); // no 'commit'
expect(hook).not.toHaveBeenCalled();
});
});
@@ -35,25 +35,4 @@ describe('jsonbBind', () => {
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
// preserveEmpty (#476): opts a column OUT of the empty-to-null collapse so an
// empty container is persisted verbatim (e.g. `[]` = deny-all for
// tool_allowlist). null stays null regardless of the flag.
describe('preserveEmpty', () => {
it('returns a (non-null) bind for an empty array', () => {
const out = jsonbBind([], { preserveEmpty: true });
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
it('returns a (non-null) bind for an empty object', () => {
const out = jsonbBind({}, { preserveEmpty: true });
expect(out).not.toBeNull();
expect(out).toBeDefined();
});
it('still returns null for null (null means null, flag or not)', () => {
expect(jsonbBind(null, { preserveEmpty: true })).toBeNull();
});
});
});
@@ -33,18 +33,16 @@ import { type Kysely, sql } from 'kysely';
* - comments: `findPageComments` does WHERE page_id ORDER BY id ASC, but only
* `(page_id)` exists extra sort.
*
* DEPLOY-TIME LOCK: these are plain (non-CONCURRENT) CREATE INDEX statements
* CONCURRENTLY is impossible HERE because Kysely runs each migration in a
* transaction. The two GIN trigram builds on pages.title / users.name are the
* slow ones and would take a SHARE lock that BLOCKS writes on pages/users for
* minutes on a large tenant. To avoid that, `ensureConcurrentIndexes`
* (database/concurrent-indexes.ts) now pre-builds BOTH trigram indexes with
* CREATE INDEX CONCURRENTLY (no transaction) BEFORE the migrator runs, so on an
* existing DB the two `IF NOT EXISTS` trigram creates below no-op and no write
* lock is taken. On a fresh DB the pre-build is skipped and they build on an
* empty table. Keep the two trigram creates in lockstep with their CANONICAL
* definitions in CONCURRENT_INDEXES. (The plain b-tree indexes further down are
* fast metadata-only builds; they are not pre-built.)
* DEPLOY-TIME LOCK WARNING: these are plain (non-CONCURRENT) CREATE INDEX
* statements CONCURRENTLY is impossible because Kysely runs each migration in a
* transaction. They take a SHARE lock that BLOCKS writes (INSERT/UPDATE/DELETE) on
* pages/users/groups/comments/page_history for the duration of the build. The two
* GIN trigram builds on pages.title / users.name are the slow ones and can take
* minutes on a large tenant a write-outage window during the deploy migration.
* For large installations, run this migration in a maintenance window, or build
* the trigram indexes out-of-band with CREATE INDEX CONCURRENTLY before deploying
* (then this migration's `IF NOT EXISTS` is a no-op). Small/typical tenants are
* unaffected.
*/
export async function up(db: Kysely<any>): Promise<void> {
// Index-compatible, output-identical redefinition of f_unaccent (see header).
@@ -26,16 +26,13 @@ import { type Kysely, sql } from 'kysely';
* to update than b-trees); on the small instances this fork targets that cost
* is acceptable and the read win on agent lookups is the priority.
*
* DEPLOY-TIME LOCK: this is a plain (non-CONCURRENT) CREATE INDEX Kysely runs
* each migration in a transaction, so CONCURRENTLY is impossible HERE, and the
* build would take a SHARE lock that BLOCKS writes on `pages` for its duration
* (the text_content GIN build can take minutes on a large tenant). To avoid that,
* `ensureConcurrentIndexes` (database/concurrent-indexes.ts) now pre-builds this
* index with CREATE INDEX CONCURRENTLY (no transaction) BEFORE the migrator runs,
* so on an existing DB the `IF NOT EXISTS` below no-ops and no write lock is taken.
* On a fresh DB the pre-build is skipped and this builds it on an empty table
* where the lock is irrelevant. Keep this create in lockstep with the CANONICAL
* definition in CONCURRENT_INDEXES (same expression + opclass).
* DEPLOY-TIME LOCK WARNING: plain (non-CONCURRENT) CREATE INDEX Kysely runs
* each migration in a transaction, so CONCURRENTLY is impossible. The build takes
* a SHARE lock that BLOCKS writes on `pages` for its duration. The text_content
* GIN build is the slow one and can take minutes on a large tenant. For large
* installations, run this in a maintenance window or build the index out-of-band
* with CREATE INDEX CONCURRENTLY before deploying (then `IF NOT EXISTS` no-ops
* here). Small/typical tenants are unaffected.
*/
export async function up(db: Kysely<any>): Promise<void> {
// The title predicate is served by #348's idx_pages_title_trgm — see header.
@@ -1,24 +0,0 @@
import { type Kysely, sql } from 'kysely';
export async function up(db: Kysely<any>): Promise<void> {
// Chat-level metadata bag (#490). First use: the deferred-tool ACTIVATION set
// (`activatedTools`) is persisted here so it survives across turns — previously
// the set was reset every turn, forcing the model to re-run loadTools and pay a
// fresh round-trip to re-activate the same tools each turn. On load the stored
// set is intersected with the current valid deferred names, so an allowlist /
// role change can never inject a now-nonexistent tool.
//
// jsonb, defaulted to '{}' so every row (incl. pre-migration ones, backfilled
// by the default) is a readable object — the app never has to null-guard the
// bag itself, only individual keys.
await db.schema
.alterTable('ai_chats')
.addColumn('metadata', 'jsonb', (col) =>
col.notNull().defaultTo(sql`'{}'::jsonb`),
)
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema.alterTable('ai_chats').dropColumn('metadata').execute();
}
@@ -78,9 +78,7 @@ export class AiMcpServerRepo {
headersEnc: values.headersEnc ?? null,
// jsonb column: the postgres driver would otherwise encode a JS array as
// a Postgres array literal. Bind the JSON text and cast it to jsonb.
// preserveEmpty (#476): `[]` is a real value here (deny-all), distinct
// from null ("no restriction") — it must round-trip as `[]`, not null.
toolAllowlist: jsonbBind(values.toolAllowlist, { preserveEmpty: true }),
toolAllowlist: jsonbBind(values.toolAllowlist),
// Plain text column: blank/whitespace-only guidance is stored as null.
instructions: blankToNull(values.instructions),
enabled: values.enabled ?? true,
@@ -113,10 +111,7 @@ export class AiMcpServerRepo {
if (patch.url !== undefined) set.url = patch.url;
if (patch.headersEnc !== undefined) set.headersEnc = patch.headersEnc;
if (patch.toolAllowlist !== undefined) {
// preserveEmpty (#476): see insert — `[]` (deny-all) must not become null.
set.toolAllowlist = jsonbBind(patch.toolAllowlist, {
preserveEmpty: true,
});
set.toolAllowlist = jsonbBind(patch.toolAllowlist);
}
if (patch.instructions !== undefined) {
// Blank/whitespace-only guidance clears the column (stored as null).
@@ -163,9 +158,7 @@ export function blankToNull(value: string | null | undefined): string | null {
* fix), so the driver hands back a string like `'["a","b"]'` rather than an
* array. Be tolerant: normalize a JSON string to its value, then accept it only
* if it is an array of strings; null / a non-array / unparseable value / an
* array with a non-string element all become null. NOTE: null here only means
* "could not parse" the null-vs-deny-all policy decision lives in
* normalizeRow (#476: present-but-corrupt fails CLOSED to `[]`).
* array with a non-string element all become null (unrestricted).
*/
export function parseToolAllowlist(value: unknown): string[] | null {
// Shape guard only; the legacy double-encoding self-heal lives in
@@ -180,20 +173,17 @@ export function parseToolAllowlist(value: unknown): string[] | null {
/**
* Normalize a DB row so `toolAllowlist` is always `string[] | null`.
*
* FAIL-CLOSED (#476): a stored value that is PRESENT but cannot be parsed into
* a string[] (corrupt JSON, a non-array, non-string elements) degrades to `[]`
* = deny-all, so a corrupted allowlist can never silently widen to "the agent
* gets ALL of the server's tools" (the old fail-open null). An error line is
* logged (server id only, never the contents) so the admin can repair the row.
* A column that is truly NULL/absent stays `null` = "no restriction".
* FAIL-OPEN logging: a stored value that is present but cannot be parsed into a
* string[] (corrupt JSON, a non-array, non-string elements) degrades to `null` =
* "no restriction", so the agent silently gets ALL of the server's tools. Log
* one line (server id only, never the contents) so that widening is not silent.
*/
function normalizeRow(row: AiMcpServer): AiMcpServer {
const parsed = parseToolAllowlist(row.toolAllowlist);
if (parsed === null && row.toolAllowlist != null) {
logger.error(
`Corrupt tool_allowlist for MCP server ${row.id}; failing closed (NO tools allowed) — re-save the server's allowlist to repair it`,
logger.warn(
`Corrupt tool_allowlist for MCP server ${row.id}; ignoring it (no tool restriction applied)`,
);
return { ...row, toolAllowlist: [] };
}
return { ...row, toolAllowlist: parsed };
}
@@ -1,7 +1,7 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx, executeTx, registerAfterCommit } from '../../utils';
import { dbOrTx, executeTx } from '../../utils';
import {
InsertablePage,
Page,
@@ -349,23 +349,14 @@ export class PageRepo {
pageId: string,
deletedById: string,
workspaceId: string,
// Optional caller transaction. When passed, the reads + soft-delete run in
// THAT transaction (so a caller holding a `FOR UPDATE` lock on the row — e.g.
// the temporary-note sweeper — can delete under the lock without deadlocking
// on a nested independent transaction) and the PAGE_SOFT_DELETED broadcast is
// deferred to the caller's COMMIT via registerAfterCommit (so a rolled-back
// delete never broadcasts). With no trx the behaviour is unchanged: own
// transaction, broadcast right after it commits.
existingTrx?: KyselyTransaction,
): Promise<void> {
const currentDate = new Date();
const readDb = dbOrTx(this.db, existingTrx);
// Read the root snapshot up front so PAGE_SOFT_DELETED can carry it without
// a post-commit DB read (variant A). Only the root of the deleted subtree is
// needed for the tree broadcast — the client `treeModel.remove` drops all
// descendants, so we don't snapshot/broadcast every descendant.
const rootSnapshot = await readDb
const rootSnapshot = await this.db
.selectFrom('pages')
.select([
'id',
@@ -380,7 +371,7 @@ export class PageRepo {
.where('deletedAt', 'is', null)
.executeTakeFirst();
const descendants = await readDb
const descendants = await this.db
.withRecursive('page_descendants', (db) =>
db
.selectFrom('pages')
@@ -402,60 +393,39 @@ export class PageRepo {
const pageIds = descendants.map((d) => d.id);
if (pageIds.length > 0) {
// Reuse the caller's transaction when given (executeTx passes it straight
// through), else own a fresh one.
await executeTx(
this.db,
async (trx) => {
await trx
.updateTable('pages')
.set({
deletedById: deletedById,
deletedAt: currentDate,
})
.where('id', 'in', pageIds)
.where('deletedAt', 'is', null)
.execute();
await executeTx(this.db, async (trx) => {
await trx
.updateTable('pages')
.set({
deletedById: deletedById,
deletedAt: currentDate,
})
.where('id', 'in', pageIds)
.where('deletedAt', 'is', null)
.execute();
await trx
.deleteFrom('shares')
.where('pageId', 'in', pageIds)
.execute();
},
existingTrx,
);
await trx.deleteFrom('shares').where('pageId', 'in', pageIds).execute();
});
const emitSoftDeleted = () => {
this.eventEmitter.emit(EventName.PAGE_SOFT_DELETED, {
pageIds: pageIds,
workspaceId,
// Root-only snapshot: one `deleteTreeNode` is enough, the client
// removes the whole subtree. Skip if the root vanished between reads.
pages: rootSnapshot
? [
{
id: rootSnapshot.id,
slugId: rootSnapshot.slugId,
title: rootSnapshot.title,
icon: rootSnapshot.icon,
position: rootSnapshot.position,
spaceId: rootSnapshot.spaceId,
parentPageId: rootSnapshot.parentPageId,
},
]
: [],
});
};
if (existingTrx) {
// Inside a caller transaction: the delete above is NOT committed yet.
// Defer the tree broadcast to the caller's commit so a rolled-back delete
// never broadcasts a phantom removal.
registerAfterCommit(existingTrx, emitSoftDeleted);
} else {
// Own transaction already committed above — broadcast now.
emitSoftDeleted();
}
this.eventEmitter.emit(EventName.PAGE_SOFT_DELETED, {
pageIds: pageIds,
workspaceId,
// Root-only snapshot: one `deleteTreeNode` is enough, the client removes
// the whole subtree. Skip if the root vanished between the two reads.
pages: rootSnapshot
? [
{
id: rootSnapshot.id,
slugId: rootSnapshot.slugId,
title: rootSnapshot.title,
icon: rootSnapshot.icon,
position: rootSnapshot.position,
spaceId: rootSnapshot.spaceId,
parentPageId: rootSnapshot.parentPageId,
},
]
: [],
});
}
}
@@ -3,7 +3,7 @@ import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Cache } from 'cache-manager';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx, registerAfterCommit } from '../../utils';
import { dbOrTx } from '../../utils';
import {
InsertableWorkspace,
UpdatableWorkspace,
@@ -80,30 +80,16 @@ export class WorkspaceRepo {
*/
private async bustWorkspaceCache(
workspace?: Pick<Workspace, 'hostname'> | undefined,
trx?: KyselyTransaction,
): Promise<void> {
const del = async () => {
try {
await this.cacheManager.del(CacheKey.WORKSPACE_SELF_HOSTED);
if (workspace?.hostname) {
await this.cacheManager.del(
CacheKey.WORKSPACE_BY_HOST(workspace.hostname),
);
}
} catch {
// cache is best-effort; TTL is the backstop
try {
await this.cacheManager.del(CacheKey.WORKSPACE_SELF_HOSTED);
if (workspace?.hostname) {
await this.cacheManager.del(
CacheKey.WORKSPACE_BY_HOST(workspace.hostname),
);
}
};
if (trx) {
// Inside a caller transaction the write is NOT yet committed: busting now
// opens a repopulation window (a concurrent reader reloads the cache with
// the pre-commit / stale row, which then survives until TTL). Defer the del
// to the transaction's commit (drained by the owning executeTx) (#495).
registerAfterCommit(trx, del);
} else {
// No transaction: the mutation above already auto-committed, so this del is
// already post-commit.
await del();
} catch {
// cache is best-effort; TTL is the backstop
}
}
@@ -194,7 +180,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -209,7 +195,7 @@ export class WorkspaceRepo {
.returning(this.baseFields)
.executeTakeFirst();
// Bust the cached "not found" so a fresh install / new tenant is seen at once.
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -263,7 +249,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -285,7 +271,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -340,7 +326,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -368,7 +354,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -390,7 +376,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -412,7 +398,7 @@ export class WorkspaceRepo {
.where('id', '=', workspaceId)
.returning(this.baseFields)
.executeTakeFirst();
await this.bustWorkspaceCache(workspace, trx);
await this.bustWorkspaceCache(workspace);
return workspace;
}
@@ -4,7 +4,6 @@ import { promises as fs } from 'fs';
import { Migrator, FileMigrationProvider } from 'kysely';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { ensureConcurrentIndexes } from '@docmost/db/concurrent-indexes';
@Injectable()
export class MigrationService {
@@ -13,16 +12,6 @@ export class MigrationService {
constructor(@InjectKysely() private readonly db: KyselyDB) {}
async migrateToLatest(): Promise<void> {
// Build write-blocking trigram indexes CONCURRENTLY (no transaction) BEFORE
// the migrator runs, so the corresponding in-migration `CREATE INDEX IF NOT
// EXISTS` no-ops instead of taking a SHARE lock on `pages` during deploy
// (#495). Best-effort: on a fresh DB (no `pages`/`f_unaccent` yet) this is a
// no-op and the migrations build the index normally.
await ensureConcurrentIndexes(this.db, (message, error) => {
if (error) this.logger.warn(`${message}: ${String(error)}`);
else this.logger.log(message);
});
const migrator = new Migrator({
db: this.db,
provider: new FileMigrationProvider({
-3
View File
@@ -606,9 +606,6 @@ export interface AiChats {
// The document the chat was created in (open page at first message). NULL =>
// started outside any document. ON DELETE SET NULL on the page FK.
pageId: string | null;
// Chat-level metadata bag (#490). jsonb, defaulted to '{}'. First key:
// `activatedTools` — the deferred-tool activation set persisted across turns.
metadata: Generated<Json>;
createdAt: Generated<Timestamp>;
updatedAt: Generated<Timestamp>;
deletedAt: Timestamp | null;
+10 -82
View File
@@ -6,76 +6,16 @@ import { KyselyDB, KyselyTransaction } from './types/kysely.types';
* If an existing transaction is provided, it directly executes the callback with it.
* Otherwise, it starts a new transaction using the provided database instance and executes the callback within that transaction.
*/
/**
* Post-commit side-effect hooks, keyed by the transaction they were registered
* against. A WeakMap so an abandoned/never-drained transaction's entry is GC'd
* with the trx object (no leak). Used by {@link registerAfterCommit} /
* {@link executeTx}.
*/
const afterCommitHooks = new WeakMap<
KyselyTransaction,
Array<() => Promise<void> | void>
>();
/**
* Register a side effect to run ONLY AFTER the transaction that owns `trx`
* commits. THE fix for "bust the cache inside the open transaction" bugs: a
* cache-invalidation (or any read-your-write-visible side effect) done while the
* writing transaction is still open opens a window where a concurrent reader
* repopulates the cache with the PRE-COMMIT (stale) row, so after commit the
* cache holds the old value until its TTL. Deferring the effect to post-commit
* closes that window.
*
* The hook is drained by the OUTERMOST {@link executeTx} that actually owns
* (created) this transaction so registering against a passed-through
* `existingTrx` still fires at the real commit boundary, not at the inner call.
* NOTE: a hook registered against a transaction that was NOT created via
* `executeTx` (untracked) will never be drained always create transactions
* through `executeTx` when you rely on post-commit hooks.
*/
export function registerAfterCommit(
trx: KyselyTransaction,
hook: () => Promise<void> | void,
): void {
const existing = afterCommitHooks.get(trx);
if (existing) existing.push(hook);
else afterCommitHooks.set(trx, [hook]);
}
export async function executeTx<T>(
db: KyselyDB,
callback: (trx: KyselyTransaction) => Promise<T>,
existingTrx?: KyselyTransaction,
): Promise<T> {
if (existingTrx) {
// Reuse the caller's transaction. Any post-commit hooks registered here are
// drained by the OUTER executeTx that created `existingTrx`, at the true
// commit boundary — so we must NOT drain them now.
return await callback(existingTrx);
return await callback(existingTrx); // Execute callback with existing transaction
} else {
return await db.transaction().execute((trx) => callback(trx)); // Start new transaction and execute callback
}
// We OWN this transaction: run the body, then (only once it has COMMITTED)
// drain the post-commit hooks registered against it during the body.
let ownTrx: KyselyTransaction | undefined;
const result = await db.transaction().execute((trx) => {
ownTrx = trx;
return callback(trx);
});
if (ownTrx) {
const hooks = afterCommitHooks.get(ownTrx);
if (hooks) {
afterCommitHooks.delete(ownTrx);
for (const hook of hooks) {
// Best-effort: a failed side effect (e.g. a cache del) must not fail the
// already-committed transaction.
try {
await hook();
} catch {
// swallow — the durable write already committed
}
}
}
}
return result;
}
/*
@@ -138,30 +78,18 @@ export function violatedConstraint(err: unknown): string | undefined {
* verbatim); `::jsonb` then parses it into a real array/object. Read-side
* parsers repair rows written the old buggy way without a migration.
*
* Returns `null` for null/undefined. By default it ALSO returns `null` for
* "empty" values (an empty array, or an object with no own enumerable keys)
* most callers treat empty as "clear/unset", so an empty config never
* round-trips as `[]`/`{}`.
*
* `preserveEmpty` (issue #476) opts a column OUT of that empty-to-null
* normalization so `[]`/`{}` are persisted as real jsonb values. Needed where
* empty and null mean DIFFERENT things: an empty `tool_allowlist` is
* deny-all ("zero tools allowed"), while null is "no restriction" collapsing
* `[]` to null silently widened deny-all to allow-all. Deliberately an opt-in
* flag, NOT a global change: the other jsonb callers (model_config, source)
* keep the empty-means-unset contract.
* Returns `null` for null/undefined and for "empty" values (an empty array, or
* an object with no own enumerable keys) callers treat empty as "clear/unset",
* so an empty allowlist/config never round-trips as `[]`/`{}`.
*/
export function jsonbBind<T>(
value: T | null | undefined,
opts?: { preserveEmpty?: boolean },
): RawBuilder<T> | null {
if (value === null || value === undefined) return null;
if (!opts?.preserveEmpty) {
if (Array.isArray(value)) {
if (value.length === 0) return null;
} else if (typeof value === 'object') {
if (Object.keys(value as object).length === 0) return null;
}
if (Array.isArray(value)) {
if (value.length === 0) return null;
} else if (typeof value === 'object') {
if (Object.keys(value as object).length === 0) return null;
}
return sql<T>`${JSON.stringify(value)}::text::jsonb`;
}
@@ -99,12 +99,10 @@ describe('AiSettingsService.getMasked reindex progress', () => {
// actually pins the progress.total branch rather than coincidentally
// matching the DB fallback. With fix #1 the two sources agree in practice,
// but getMasked must still return progress.total when a record is active.
const startedAt = Date.now();
reindexProgress.get.mockResolvedValue({
total: 500,
done: 120,
startedAt,
runId: 'run-abc',
startedAt: Date.now(),
});
const masked = await service.getMasked(WORKSPACE_ID);
@@ -112,10 +110,6 @@ describe('AiSettingsService.getMasked reindex progress', () => {
expect(masked.indexedPages).toBe(120); // progress.done, not DB 478
expect(masked.totalPages).toBe(500); // progress.total, not DB 478
expect(masked.reindexing).toBe(true);
// The status payload must carry the run identity so the client can key its
// poll on it (a changed runId => a NEW run).
expect(masked.runId).toBe('run-abc');
expect(masked.reindexStartedAt).toBe(startedAt);
});
it('falls back to countIndexedPages when no reindex is active', async () => {
@@ -127,10 +121,6 @@ describe('AiSettingsService.getMasked reindex progress', () => {
expect(masked.indexedPages).toBe(478);
expect(masked.totalPages).toBe(478);
expect(masked.reindexing).toBe(false);
// No active run -> no run identity surfaced (the client keeps its prior
// steady-state behaviour).
expect(masked.runId).toBeUndefined();
expect(masked.reindexStartedAt).toBeUndefined();
});
});
@@ -245,9 +245,6 @@ export class AiSettingsService {
// Max context window for the chat header badge denominator. Stored as
// ::text; 0/unset/invalid = no limit (undefined).
chatContextWindow: parsePositiveInt(provider.chatContextWindow),
// RAW stored value (#490): the replay budgeter reads this to distinguish an
// explicit `0` (off-switch) from unset, which parsePositiveInt cannot.
chatContextWindowRaw: provider.chatContextWindow,
// Plain passthrough; getChatModel defaults unset to 'openai-compatible'.
chatApiStyle: provider.chatApiStyle,
// Cheap model id for the anonymous public-share assistant; reuses the chat
@@ -374,12 +371,6 @@ export class AiSettingsService {
totalPages,
// Optional hint for the client: a reindex run is currently in progress.
reindexing: progress != null,
// Per-run identity so the client can key its poll on a stable run id and
// reset its per-run state when a NEW run starts. Present only while a run
// is active; `runId` may be '' for a legacy/degraded record (the client
// treats that as "no identity").
runId: progress?.runId,
reindexStartedAt: progress?.startedAt,
};
}
@@ -1,86 +0,0 @@
// `.provider` alone cannot prove the gemini/ollama chat factories were built
// with the instrumented streaming fetch — a regression dropping it (which drops
// them back to the global undici fetch: no keep-alive recycle, no reset retries,
// unbounded silence timeout; incident classes #140/#175/#310) would still pass.
// So mock the factories and assert the exact fetch argument. jest.mock is
// module-scoped, hence a dedicated file.
const mockGeminiModel = { provider: 'google.generative-ai', modelId: 'm' };
const mockOllamaModel = { provider: 'ollama.chat', modelId: 'm' };
// jest allows `mock`-prefixed vars inside a jest.mock factory.
const mockCreateGoogle = jest.fn((_settings: unknown) => () => mockGeminiModel);
const mockCreateOllama = jest.fn((_settings: unknown) => () => mockOllamaModel);
jest.mock('@ai-sdk/google', () => ({
createGoogleGenerativeAI: (settings: unknown) => mockCreateGoogle(settings),
}));
jest.mock('ai-sdk-ollama', () => ({
createOllama: (settings: unknown) => mockCreateOllama(settings),
}));
import { AiService } from './ai.service';
describe('AiService.getChatModel provider transport fetch (gemini/ollama)', () => {
function serviceWith(cfg: Record<string, unknown>) {
const aiSettings = {
resolve: jest.fn().mockResolvedValue(cfg),
};
return new AiService(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
aiSettings as any,
{ find: jest.fn() } as never,
{ decryptSecret: jest.fn() } as never,
);
}
beforeEach(() => {
mockCreateGoogle.mockClear();
mockCreateOllama.mockClear();
});
it('builds the gemini chat model with the instrumented streaming fetch', async () => {
await serviceWith({
driver: 'gemini',
chatModel: 'gemini-2.5-pro',
apiKey: 'the-key',
}).getChatModel('ws-1');
expect(mockCreateGoogle).toHaveBeenCalledTimes(1);
expect(mockCreateGoogle).toHaveBeenCalledWith(
expect.objectContaining({
apiKey: 'the-key',
fetch: expect.any(Function),
}),
);
});
it('builds the ollama chat model with the instrumented streaming fetch', async () => {
await serviceWith({
driver: 'ollama',
chatModel: 'llama3',
baseUrl: 'http://localhost:11434/api',
}).getChatModel('ws-1');
expect(mockCreateOllama).toHaveBeenCalledTimes(1);
expect(mockCreateOllama).toHaveBeenCalledWith(
expect.objectContaining({
baseURL: 'http://localhost:11434/api',
fetch: expect.any(Function),
}),
);
});
it('reuses ONE service-lifetime fetch instance across both providers', async () => {
const svc = serviceWith({
driver: 'gemini',
chatModel: 'gemini-2.5-pro',
apiKey: 'k',
});
await svc.getChatModel('ws-1');
const geminiFetch = mockCreateGoogle.mock.calls[0][0] as { fetch: unknown };
// Same instance on a second call — the fetch is held for the service
// lifetime to reuse the streaming dispatcher's connection pool.
await svc.getChatModel('ws-1');
const geminiFetch2 = mockCreateGoogle.mock.calls[1][0] as { fetch: unknown };
expect(geminiFetch.fetch).toBe(geminiFetch2.fetch);
});
});
+3 -15
View File
@@ -190,22 +190,10 @@ export class AiService {
}).chat(chatModel);
}
case 'gemini':
// Route gemini through the same instrumented streaming fetch as openai
// (finite silence timeouts + keep-alive recycling + pre-response
// connection-reset retry). Without it the provider ran on the global
// undici fetch — no keep-alive recycle, no reset retries, default
// (unbounded silence) timeout — so incident classes #140/#175/#310 were
// reproducible for gemini too.
return createGoogleGenerativeAI({
apiKey,
fetch: this.aiProviderFetch,
})(chatModel);
return createGoogleGenerativeAI({ apiKey })(chatModel);
case 'ollama':
// Ollama needs no API key. Same transport hardening as above (#140/#175/#310).
return createOllama({
baseURL: baseUrl,
fetch: this.aiProviderFetch,
})(chatModel);
// Ollama needs no API key.
return createOllama({ baseURL: baseUrl })(chatModel);
default:
throw new AiNotConfiguredException();
}
@@ -105,10 +105,6 @@ export interface ResolvedAiConfig extends Partial<AiProviderSettings> {
// Max context window in tokens; surfaced to the chat header badge as the
// "current / max" denominator. 0/unset = no limit.
chatContextWindow?: number;
// RAW stored context window (::text), BEFORE parsePositiveInt collapses `0` and
// unset to `undefined`. The #490 replay budgeter needs the raw value to honor an
// explicit `0` off-switch distinctly from "unset -> flat default".
chatContextWindowRaw?: string | number;
// Cheap model id for the public-share assistant; reuses the chat creds.
publicShareChatModel?: string;
// Agent-role id whose persona the public-share assistant adopts (empty/unset
@@ -153,14 +149,4 @@ export interface MaskedAiSettings {
// True while a full workspace reindex is actively running (the counts above
// then reflect the live run progress rather than the steady-state DB count).
reindexing?: boolean;
// Identity of the ACTIVE reindex run (present only while `reindexing`). The
// client keys its poll on `runId`: a changed value means a NEW run (reset the
// per-run poll state it latched), the same value means the run it is already
// watching — removing the "same run or a fresh one?" ambiguity a stale
// pre-reindex snapshot otherwise causes. Absent/empty degrades gracefully.
runId?: string;
// Epoch-ms the active run started (present only while `reindexing`). Paired
// with `runId` so a run that restarts with the same (recycled) id is still
// seen as new.
reindexStartedAt?: number;
}
@@ -48,38 +48,19 @@ describe('EmbeddingReindexProgressService', () => {
}
describe('get', () => {
it('maps a valid hash to a ReindexProgress object (incl. the run identity)', async () => {
it('maps a valid hash to a ReindexProgress object', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({
total: '478',
done: '120',
startedAt: '1000',
runId: 'run-xyz',
});
hgetall.mockResolvedValue({ total: '478', done: '120', startedAt: '1000' });
const service = makeService(redis);
await expect(service.get(WORKSPACE_ID)).resolves.toEqual({
total: 478,
done: 120,
startedAt: 1000,
runId: 'run-xyz',
});
expect(hgetall).toHaveBeenCalledWith(KEY);
});
it('degrades a missing runId to an empty string (legacy/partial record)', async () => {
const { redis, hgetall } = makeRedis();
// A record written before runId existed: get() must still succeed and
// report runId='' so the client treats it as "no identity", never breaks.
hgetall.mockResolvedValue({ total: '10', done: '3', startedAt: '5' });
await expect(makeService(redis).get(WORKSPACE_ID)).resolves.toEqual({
total: 10,
done: 3,
startedAt: 5,
runId: '',
});
});
it('returns null for an empty hash (no record)', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({});
@@ -106,17 +87,11 @@ describe('EmbeddingReindexProgressService', () => {
it('coerces a non-finite startedAt to 0', async () => {
const { redis, hgetall } = makeRedis();
hgetall.mockResolvedValue({
total: '10',
done: '2',
startedAt: 'nope',
runId: 'run-1',
});
hgetall.mockResolvedValue({ total: '10', done: '2', startedAt: 'nope' });
await expect(makeService(redis).get(WORKSPACE_ID)).resolves.toEqual({
total: 10,
done: 2,
startedAt: 0,
runId: 'run-1',
});
});
@@ -140,21 +115,6 @@ describe('EmbeddingReindexProgressService', () => {
expect(multiObj.exec).toHaveBeenCalledTimes(1);
});
it('mints a fresh non-empty runId into the record on each start', async () => {
const { redis, multiObj } = makeRedis();
const service = makeService(redis);
await service.start(WORKSPACE_ID, 1);
await service.start(WORKSPACE_ID, 1);
const firstRunId = multiObj.hset.mock.calls[0][1].runId;
const secondRunId = multiObj.hset.mock.calls[1][1].runId;
expect(typeof firstRunId).toBe('string');
expect(firstRunId).not.toBe('');
// Each run gets its OWN identity so the client can tell a re-trigger apart
// from the run it is already watching.
expect(secondRunId).not.toBe(firstRunId);
});
it('defaults the expire TTL to the full 1h record TTL', async () => {
const { redis, multiObj } = makeRedis();
await makeService(redis).start(WORKSPACE_ID, 478);
@@ -1,28 +1,17 @@
import { Injectable, Logger } from '@nestjs/common';
import { RedisService } from '@nestjs-labs/nestjs-ioredis';
import { randomUUID } from 'node:crypto';
import type { Redis } from 'ioredis';
/**
* Live progress of an in-flight workspace embeddings reindex run.
* `total` is the number of pages the run will process, `done` how many it has
* already processed (success OR handled failure), `startedAt` the epoch-ms the
* record was created, and `runId` a per-run identity minted at `start()`.
*
* `runId` gives each reindex run a stable identity so a poller can tell "same
* run I've been watching" from "a NEW run started" WITHOUT guessing from the
* progress counters (the ambiguity that a stale pre-reindex snapshot vs a fresh
* run otherwise causes the bug class fixed twice under #262). It is best-
* effort like the rest of this record: a record written before this field
* existed (or a Redis hiccup) yields an empty `runId`, which the client must
* treat as "no identity available" and degrade to its prior behaviour, never
* break.
* record was created.
*/
export interface ReindexProgress {
total: number;
done: number;
startedAt: number;
runId: string;
}
/** Redis key namespace for the per-workspace reindex-progress record. */
@@ -97,18 +86,12 @@ export class EmbeddingReindexProgressService {
): Promise<void> {
const key = this.key(workspaceId);
try {
// A fresh identity per run so the client poll can key on it: a changed
// runId means a genuinely NEW run (reset any latched per-run poll state),
// the same runId means the run the client is already watching. Best-effort
// like the counters — never surfaced to the user, only used to disambiguate.
const runId = randomUUID();
await this.redis
.multi()
.hset(key, {
total: String(total),
done: '0',
startedAt: String(Date.now()),
runId,
})
.expire(key, ttlSeconds)
.exec();
@@ -167,15 +150,7 @@ export class EmbeddingReindexProgressService {
const done = Number(data.done);
const startedAt = Number(data.startedAt);
if (!Number.isFinite(total) || !Number.isFinite(done)) return null;
// `runId` degrades gracefully: a pre-existing record (written before this
// field) or a stripped value reads as '' — the client treats that as "no
// identity" and keeps its prior behaviour rather than breaking the poll.
return {
total,
done,
startedAt: Number.isFinite(startedAt) ? startedAt : 0,
runId: typeof data.runId === 'string' ? data.runId : '',
};
return { total, done, startedAt: Number.isFinite(startedAt) ? startedAt : 0 };
} catch (err) {
this.logger.warn(
`reindex-progress read failed for workspace ${workspaceId}; ` +
@@ -0,0 +1,70 @@
import { INTERNAL_LINK_REGEX } from './utils';
import { isInternalPagePath } from '@docmost/prosemirror-markdown';
/**
* Cross-package DRIFT GUARD for the internal-link subset invariant (#522).
*
* The client-side `isInternalPagePath`
* (`packages/prosemirror-markdown/src/lib/internal-links.ts`) promotes a markdown
* link to `internal: true` on import. Every link it marks internal MUST be one
* the server would backlink and export-rewrite i.e. the client matcher MUST be
* a STRICT SUBSET of the server's canonical `INTERNAL_LINK_REGEX`
* (`./utils.ts`). If the client ever accepts a path the server rejects, that link
* is stored internal but silently dropped from the backlink graph and broken on
* export the exact bug #522 fixed.
*
* This spec is the load-bearing guard: it imports the LIVE server regex AND the
* LIVE `isInternalPagePath` (no hand-copied regex on either side). A narrowing of
* EITHER most dangerously the server regex reddens here. The in-package
* accept/reject test documents the client's behaviour but cannot see the server
* regex; this top-layer spec is what makes the subset relation mechanical
* (AGENTS.md rule #7: a CI test that fails on drift of the source of truth).
*/
describe('internal-link subset parity (client isInternalPagePath ⊆ server INTERNAL_LINK_REGEX)', () => {
// Every path the CLIENT accepts. Kept deliberately broad across the risky
// dimensions — the slug charset (digits, hyphens, mixed case, leading/trailing
// hyphen), the space charset, and the optional trailing slash — so a narrowing
// of the server regex on any of them reddens the subset assertion below.
const CLIENT_ACCEPTS = [
'/s/eng/p/abc123',
'/s/eng/p/abc123/', // trailing slash
'/s/My-Space/p/A-b-C-9', // hyphens + mixed case in both segments
'/s/x/p/z', // shortest
'/s/eng/p/my-page-abc123', // slug with hyphens (extractPageSlugId shape)
'/s/eng/p/-lead', // leading hyphen in slug
'/s/eng/p/trail-', // trailing hyphen in slug
'/s/eng/p/0123456789', // all-digit slug
'/s/a.b/p/abc', // dot in the SPACE segment (space charset is [^/]+)
'/s/space with space/p/abc', // space char in the SPACE segment
];
it('every corpus path is actually client-accepted (guards the corpus itself)', () => {
// If a path here stopped being client-accepted the subset test would pass
// vacuously; assert acceptance up front so the corpus stays meaningful. The
// filter-to-empty form names the offending paths on failure.
const notAccepted = CLIENT_ACCEPTS.filter((h) => !isInternalPagePath(h));
expect(notAccepted).toEqual([]);
});
it('every client-accepted path also matches the LIVE server regex (subset)', () => {
// The mechanical drift guard: narrow the server INTERNAL_LINK_REGEX and at
// least one hyphen/charset/structure case appears here.
const notInServer = CLIENT_ACCEPTS.filter((h) => !INTERNAL_LINK_REGEX.test(h));
expect(notInServer).toEqual([]);
});
it('the client rejects forbidden-slug-char paths the server also rejects', () => {
// Documents the subset BOUNDARY: correct shape, forbidden slug char. The
// client and the LIVE server regex must agree on rejection.
const forbidden = [
'/s/eng/p/abc.def',
'/s/eng/p/abc_def',
'/s/eng/p/abc%20',
'/s/eng/p/abc~x',
];
const clientAccepts = forbidden.filter((h) => isInternalPagePath(h));
const serverAccepts = forbidden.filter((h) => INTERNAL_LINK_REGEX.test(h));
expect(clientAccepts).toEqual([]);
expect(serverAccepts).toEqual([]);
});
});
@@ -22,12 +22,10 @@ import { v7 } from 'uuid';
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import {
markdownToProseMirror,
normalizeForeignMarkdown,
} from '@docmost/prosemirror-markdown';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
import { formatImportHtml } from '../utils/import-formatter';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import {
buildAttachmentCandidates,
collectMarkdownAndHtmlFiles,
@@ -18,10 +18,8 @@ import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
import { TiptapTransformer } from '@hocuspocus/transformer';
import * as Y from 'yjs';
import { canonicalizeFootnotes } from '@docmost/editor-ext';
import {
markdownToProseMirror,
normalizeForeignMarkdown,
} from '@docmost/prosemirror-markdown';
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
import {
FileTaskStatus,
FileTaskType,
@@ -1,15 +1,12 @@
import { describe, it, expect } from 'vitest';
import { convertProseMirrorToMarkdown } from '../src/lib/markdown-converter.js';
import { markdownToProseMirror } from '../src/lib/markdown-to-prosemirror.js';
import {
normalizeForeignMarkdown,
normalizeAgentMarkdown,
} from '../src/lib/foreign-markdown.js';
convertProseMirrorToMarkdown,
markdownToProseMirror,
} from '@docmost/prosemirror-markdown';
import { normalizeForeignMarkdown } from './foreign-markdown';
/**
* STEP 2 goldens for issue #345 (moved into the package with the normalizer in
* #493): the foreign-markdown normalizer that runs at the import boundary BEFORE
* the strict canonical parser (`markdownToProseMirror`).
* STEP 2 goldens for issue #345: the foreign-markdown normalizer that runs at the
* import boundary BEFORE the strict canonical parser (`markdownToProseMirror`).
*
* Two layers:
* 1. PURE stringstring cases pinning the normalizer's own behavior (GFM
@@ -219,53 +216,3 @@ describe('foreign markdown import acceptance (normalizer + canonical parser)', (
).toHaveLength(1);
});
});
describe('normalizeAgentMarkdown vs normalizeForeignMarkdown — front-matter strip is IMPORT-only (#493 review)', () => {
// A page that OPENS with a horizontalRule and contains a later `---` serializes
// to a `---…---`-shaped body. On a full-body AGENT rewrite this must NOT be
// mistaken for YAML front-matter and stripped — that silently dropped the
// page's leading content.
const rulePage = '---\n\nIntro\n\nMore\n\n---\n\nRest';
it('normalizeAgentMarkdown does NOT strip a leading ---…--- (no content loss)', () => {
expect(normalizeAgentMarkdown(rulePage)).toBe(rulePage);
});
it('normalizeForeignMarkdown (file import) STILL strips a real leading YAML front-matter block', () => {
const withYaml = '---\ntitle: My Page\ntags: [a, b]\n---\n\nBody here.';
const out = normalizeForeignMarkdown(withYaml);
expect(out).toBe('Body here.');
// And the horizontalRule-shaped body IS stripped on the import path (its
// documented file-import behavior) — the two variants differ ONLY here.
expect(normalizeForeignMarkdown(rulePage)).not.toContain('Intro');
});
it('agent-write round-trip keeps a horizontalRule-led doc with a second rule intact', async () => {
// Simulate the serializer output for [horizontalRule, para, para, horizontalRule, para].
const doc = {
type: 'doc',
content: [
{ type: 'horizontalRule' },
{ type: 'paragraph', content: [{ type: 'text', text: 'Intro' }] },
{ type: 'paragraph', content: [{ type: 'text', text: 'More' }] },
{ type: 'horizontalRule' },
{ type: 'paragraph', content: [{ type: 'text', text: 'Rest' }] },
],
};
const body = convertProseMirrorToMarkdown(doc);
// The agent-write normalization must NOT eat the head; re-import keeps every
// paragraph's text.
const back = await markdownToProseMirror(normalizeAgentMarkdown(body));
const texts = JSON.stringify(back);
for (const t of ['Intro', 'More', 'Rest']) expect(texts).toContain(t);
// Both horizontal rules survive.
expect(back.content.filter((n: any) => n.type === 'horizontalRule')).toHaveLength(2);
});
it('agent-write STILL rewrites GFM reference footnotes (the shared drift-fix)', () => {
const gfm = 'See[^1].\n\n[^1]: the note.';
const out = normalizeAgentMarkdown(gfm);
expect(out).toContain('^[the note.]');
expect(out).not.toMatch(/\[\^1\]:/);
});
});
@@ -1,14 +1,7 @@
/**
* Foreign-markdown normalizer an input-liberal / output-canonical adapter that
* runs at the IMPORT boundary, BEFORE the canonical parser
* (`markdownToProseMirror`, this package).
*
* OWNED BY THIS PACKAGE (#493): the normalizer used to live only in
* apps/server's import path, so the MCP page-write path (`updatePageMarkdown` ->
* `markdownToProseMirrorCanonical`) handled the SAME foreign input differently
* (no front-matter strip, no `[^id]` reference-footnote rewrite) than the server
* importer. Moving it here and calling it from `markdownToProseMirrorCanonical`
* makes every canonical import boundary treat foreign markdown identically.
* (`markdownToProseMirror` from `@docmost/prosemirror-markdown`).
*
* The canonical parser is deliberately STRICT: it only understands Docmost's
* canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
@@ -254,18 +247,11 @@ function convertReferenceFootnotes(markdown: string): string {
const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
/**
* Normalize a foreign markdown string from a FILE IMPORT into Docmost's canonical
* markdown surface so the strict canonical parser accepts it losslessly: normalize
* line endings, strip a leading YAML front-matter block, then rewrite GFM reference
* footnotes into inline footnotes. Add further fixture-driven foreign-surface cases
* here as they are found.
*
* FRONT-MATTER STRIP IS IMPORT-ONLY (#493 review): use this ONLY at the server
* file-import boundary, where a `.md` file really can open with an Obsidian/Hugo
* YAML header. Do NOT use it on the canonical AGENT-WRITE path see
* {@link normalizeAgentMarkdown} for why a full-body agent rewrite must NOT strip
* a leading `---…---` (it is normally a horizontalRule the serializer emitted, and
* stripping it would silently drop the page's leading content).
* Normalize a foreign markdown string into Docmost's canonical markdown surface
* so the strict canonical parser accepts it losslessly: normalize line endings,
* strip a leading YAML front-matter block, then rewrite GFM reference footnotes
* into inline footnotes. Add further fixture-driven foreign-surface cases here as
* they are found.
*/
export function normalizeForeignMarkdown(markdown: string): string {
if (!markdown) return markdown;
@@ -278,26 +264,3 @@ export function normalizeForeignMarkdown(markdown: string): string {
const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
return convertReferenceFootnotes(withoutFrontMatter);
}
/**
* Canonical AGENT-WRITE normalization: normalize line endings and rewrite GFM
* `[^id]` reference footnotes to inline `^[body]` but DELIBERATELY NOT strip a
* leading YAML front-matter block.
*
* WHY the split (#493 review): the reference-footnote rewrite is the drift the
* MCP page-write path (`updatePageMarkdown` -> `markdownToProseMirrorCanonical`)
* needed unified with the server import (an agent may paste GFM footnotes). The
* front-matter strip, however, is a FILE-import concern: on a full-body agent
* rewrite a leading `---…---` is (almost) always a `horizontalRule` the
* serializer emitted plus a later rule/heading NOT a foreign YAML header so
* `YAML_FRONT_MATTER_RE` would match it and SILENTLY DELETE the page's leading
* content (a page that starts with a horizontal rule and contains a second `---`
* lost everything up to it). Agent writes must never lose already-stored content,
* so this variant skips the strip. It IS a no-op on canonical serialized content
* (which never emits `[^id]:` reference-definition lines).
*/
export function normalizeAgentMarkdown(markdown: string): string {
if (!markdown) return markdown;
const src = markdown.replace(/\r\n/g, '\n');
return convertReferenceFootnotes(src);
}
@@ -1,6 +1,5 @@
import { Kysely, sql } from 'kysely';
import { randomUUID } from 'node:crypto';
import { Logger } from '@nestjs/common';
import { AiMcpServerRepo } from '@docmost/db/repos/ai-chat/ai-mcp-server.repo';
import { getTestDb, destroyTestDb, createWorkspace } from './db';
@@ -55,11 +54,7 @@ describe('AiMcpServerRepo tool_allowlist jsonb round-trip [integration]', () =>
expect(Array.isArray(found?.toolAllowlist)).toBe(true);
});
// #476 (deliberate behaviour change): an empty allowlist used to be
// normalized to SQL NULL, which downstream means "no restriction" — so an
// admin's deny-all `[]` silently became allow-all. It must now round-trip as
// a real jsonb `[]` (deny-all), distinct from NULL.
it('an empty allowlist round-trips as jsonb [] (deny-all), not null (#476)', async () => {
it('an empty allowlist is normalized to null (no restriction), not []', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
@@ -67,27 +62,7 @@ describe('AiMcpServerRepo tool_allowlist jsonb round-trip [integration]', () =>
url: 'https://example.com/mcp',
toolAllowlist: [],
});
// The column holds a real (empty) jsonb ARRAY, not SQL NULL.
expect(await jsonbTypeof(row.id)).toBe('array');
expect((await repo.findById(row.id, ws))?.toolAllowlist).toEqual([]);
});
it('update to [] persists jsonb [] and update to null clears to SQL NULL (#476)', async () => {
const row = await repo.insert({
workspaceId: ws,
name: `srv-${randomUUID()}`,
transport: 'http',
url: 'https://example.com/mcp',
toolAllowlist: ['search'],
});
// Deny-all via update: [] must survive as a real jsonb array.
await repo.update(row.id, ws, { toolAllowlist: [] });
expect(await jsonbTypeof(row.id)).toBe('array');
expect((await repo.findById(row.id, ws))?.toolAllowlist).toEqual([]);
// Explicit clear (null) still means "no restriction" = SQL NULL.
await repo.update(row.id, ws, { toolAllowlist: null });
// The column is SQL NULL, so jsonb_typeof returns SQL NULL (JS null).
expect(await jsonbTypeof(row.id)).toBeNull();
expect((await repo.findById(row.id, ws))?.toolAllowlist).toBeNull();
});
@@ -117,60 +92,23 @@ describe('AiMcpServerRepo tool_allowlist jsonb round-trip [integration]', () =>
expect(healed?.toolAllowlist).toEqual(['alpha', 'beta']);
});
// #476 (deliberate behaviour change, replaces the old FAIL-OPEN pin): a
// present-but-corrupt tool_allowlist used to degrade to null ("no
// restriction"), silently handing the agent ALL of the server's tools. It
// must now FAIL CLOSED to `[]` (deny-all) and log an error.
it('FAIL-CLOSED: a present-but-corrupt tool_allowlist reads back as [] (deny-all) + error log (#476)', async () => {
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined);
try {
// The column is PRESENT but does not parse into a string[] — a jsonb
// string scalar holding unparseable text (a truncated legacy write).
const id = randomUUID();
await sql`
INSERT INTO ai_mcp_servers (id, workspace_id, name, transport, url, tool_allowlist)
VALUES (
${id}, ${ws}, ${`srv-${id}`}, 'http', 'https://example.com/mcp',
to_jsonb(${'{oops'}::text)
)
`.execute(db);
// Sanity: the column is present (a jsonb string scalar), not SQL NULL.
expect(await jsonbTypeof(id)).toBe('string');
// ...and the read degrades to [] (fail-closed deny-all), not null.
expect((await repo.findById(id, ws))?.toolAllowlist).toEqual([]);
// The narrowing is not silent: an error names the server id (never the
// corrupt contents).
expect(errorSpy).toHaveBeenCalledWith(
expect.stringContaining(`Corrupt tool_allowlist for MCP server ${id}`),
);
expect(
errorSpy.mock.calls.some((c) => String(c[0]).includes('{oops')),
).toBe(false);
} finally {
errorSpy.mockRestore();
}
});
it('FAIL-CLOSED: corrupt non-array JSON (an object) also reads back as [] (#476)', async () => {
const errorSpy = jest
.spyOn(Logger.prototype, 'error')
.mockImplementation(() => undefined);
try {
const id = randomUUID();
await sql`
INSERT INTO ai_mcp_servers (id, workspace_id, name, transport, url, tool_allowlist)
VALUES (
${id}, ${ws}, ${`srv-${id}`}, 'http', 'https://example.com/mcp',
to_jsonb(${'{"not":"an array"}'}::text)
)
`.execute(db);
expect(await jsonbTypeof(id)).toBe('string');
expect((await repo.findById(id, ws))?.toolAllowlist).toEqual([]);
} finally {
errorSpy.mockRestore();
}
it('FAIL-OPEN: a present-but-corrupt tool_allowlist reads back as null (no restriction)', async () => {
// #185 re-review pt 8: normalizeRow's fail-open branch — the column is
// PRESENT but does not parse into a string[] (here a jsonb string scalar
// holding non-array JSON). The read must degrade to `null` ("no restriction"),
// not crash. (A warn is logged with the server id; not asserted here.)
const id = randomUUID();
await sql`
INSERT INTO ai_mcp_servers (id, workspace_id, name, transport, url, tool_allowlist)
VALUES (
${id}, ${ws}, ${`srv-${id}`}, 'http', 'https://example.com/mcp',
to_jsonb(${'{"not":"an array"}'}::text)
)
`.execute(db);
// Sanity: the column is present (a jsonb string scalar), not SQL NULL.
expect(await jsonbTypeof(id)).toBe('string');
// ...yet the read degrades to null (fail-open).
expect((await repo.findById(id, ws))?.toolAllowlist).toBeNull();
});
});
@@ -33,11 +33,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
const pageRepo = {
findById: async (id: string) => ({ id, title: `title-${id}` }),
};
// The requester can view the target page (permissive), so the reassign 409 may
// include its title — these tests exercise the one-per-page invariant, not the
// #495 disclosure gate (that is unit-tested in share-alias.service.spec.ts).
const pageAccessService = { validateCanView: async () => {} };
const USER = { id: 'u-int' } as any;
beforeAll(async () => {
db = getTestDb();
@@ -46,7 +41,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
repo as any,
pageRepo as any,
{} as any, // shareService — unused by setAlias
pageAccessService as any,
db as any,
);
wsId = (await createWorkspace(db)).id;
@@ -194,7 +188,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId,
user: USER,
alias: 'te',
});
expect(first.alias).toBe('te');
@@ -203,7 +196,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId,
user: USER,
alias: 'ted',
});
// Same row id — a RENAME, not a new insert.
@@ -225,14 +217,12 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'hello',
});
const again = await service.setAlias({
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'hello',
});
expect(again.id).toBe(inserted.id);
@@ -254,7 +244,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
flakyRepo as any,
pageRepo as any,
{} as any,
pageAccessService as any,
db as any,
);
@@ -263,7 +252,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId,
creatorId: null as any,
user: USER,
alias: 'rollback-me',
}),
).rejects.toBeInstanceOf(BadRequestException);
@@ -287,7 +275,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageA,
creatorId: null as any,
user: USER,
alias: 'shared',
});
@@ -296,7 +283,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared',
}),
).rejects.toBeInstanceOf(ConflictException);
@@ -305,7 +291,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared',
confirmReassign: true,
});
@@ -332,14 +317,12 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageA,
creatorId: null as any,
user: USER,
alias: 'shared-target',
});
await service.setAlias({
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'bee',
});
@@ -347,7 +330,6 @@ describe('share_aliases one-per-page invariant [integration]', () => {
workspaceId: wsId,
pageId: pageB,
creatorId: null as any,
user: USER,
alias: 'shared-target',
confirmReassign: true,
});
-1
View File
@@ -22,7 +22,6 @@
"^@docmost/db/(.*)$": "<rootDir>/src/database/$1",
"^@docmost/transactional/(.*)$": "<rootDir>/src/integrations/transactional/$1",
"^@docmost/ee/(.*)$": "<rootDir>/src/ee/$1",
"^@docmost/token-estimate$": "<rootDir>/../../packages/token-estimate/src/index.ts",
"^src/(.*)$": "<rootDir>/src/$1"
}
}
+78 -174
View File
@@ -8,35 +8,19 @@ real pain (a "which tools fail most?" analysis that confidently answered
Read the **Gotchas** section before you trust any error count.
> **TWO ERAS — check the marker first.** The `tool_calls` shape changed in **#490
> (trace v2)**. A row written by v2 carries `metadata.toolTraceVersion = 2`; older
> rows have no such key. The two shapes store DIFFERENT things (v2 dropped the tool
> OUTPUT from the trace), so **every query below is dual-shape** — branch on the
> marker. **Never compare an aggregate or trend across the era boundary**: a metric
> jump on the cut-over week is an artifact of the shape change, not a behavior
> change.
## TL;DR
- Agent chats live in Postgres, DB `docmost`, tables `ai_chat_*`.
- **Era marker:** `metadata.toolTraceVersion = 2` ⇒ v2 (#490) row; absent ⇒ legacy row.
- Each tool invocation is stored as **two** consecutive array elements — a
`tool-call` part then an OUTCOME part — so naive counting double-counts.
- **v2 (#490):** outcome is `{toolName, ok: true}` on success, or
`{toolName, error, kind: 'thrown'|'interrupted'}` on failure. The tool **OUTPUT
is NOT in `tool_calls`** any more — it lives once in `metadata.parts` (this
removed a hundreds-of-MB-per-run write duplication). Soft-failure analysis
therefore reads `metadata.parts`, not `tool_calls`.
- **legacy:** outcome is `{toolName, output}` on success; a **thrown** failure is
a `{toolName, error}` element **only on rows after #407**, and is dropped
entirely (silent orphan) on pre-#407 rows.
- **A tool that *throws* writes no result part.** In v2 it is a
`{error, kind:'thrown'}` element; an interrupted/aborted call is a distinct
`{error, kind:'interrupted'}`. `isError`/`success=false` scans read the *output*
and so under-report thrown failures in every era.
- To find where agents fail: (1) soft-failure markers in `metadata.parts` outputs
(v2) / `tool_calls` outputs (legacy), (2) the `error`/`kind` fields for thrown
failures (v2 + post-#407), (3) server logs / the live UI for full stack traces.
- Each tool invocation is stored as **two** array elements (a `tool-call` part and
a `tool-result` part), so naive counting double-counts.
- **A tool that *throws* writes no result part.** Since the #407 fix its error is
persisted as a dedicated `{toolName, error}` element in `tool_calls` (queryable +
replayed to the model). **Rows written before #407 still drop it** — the error is
nowhere in the DB and shows only in the live UI. So `isError` / `success=false`
scans under-report by design, and pre-#407 thrown errors are invisible.
- To find where agents fail: (1) soft-failure markers in `tool_calls`, (2) the new
`error` field for thrown errors (new rows) / the orphan-gap proxy (old rows),
(3) server logs / the live UI for full stack traces beyond the truncated message.
## Where the data lives
@@ -69,67 +53,33 @@ are rows in `workspaces`, not separate deployments.
separate `tool` role), `content` (text), `tool_calls` (jsonb array), `metadata`
(jsonb, holds run `error` + rendered `parts`), `status`, `tsv` (full-text index).
## Era marker — check this before every query
```sql
-- how many rows are in each era?
SELECT COALESCE((metadata->>'toolTraceVersion'), 'legacy') AS era, count(*)
FROM ai_chat_messages
WHERE role = 'assistant' AND jsonb_typeof(tool_calls) = 'array'
GROUP BY 1 ORDER BY 2 DESC;
```
- `toolTraceVersion = '2'`**v2** (#490): outcome flags, **no output in the trace**.
- `NULL` (`'legacy'`) → pre-#490: outcome carries the tool `output` inline.
**Do not trend a metric across the cut-over.** The shape change alone shifts counts
(e.g. "elements with `output`" collapses to zero for v2), so a week that straddles
the boundary shows an artifact, not a behavior change. Segment by era, or restrict to
one era, before comparing.
## How tool calls are stored — READ THIS
Tool calls are **not** one-object-per-call. Each logical invocation is split into
two consecutive elements of the `tool_calls` array — a **call** then an **outcome**.
The outcome shape is era-dependent:
two consecutive elements of the `tool_calls` array:
```text
# v2 (#490) — metadata.toolTraceVersion = 2
index 0: { "toolName":"getPage", "input":{...} } ← call (has input)
index 1: { "toolName":"getPage", "ok":true } ← success (NO output here)
or : { "toolName":"getPage", "error":"…", "kind":"thrown" } ← threw
or : { "toolName":"getPage", "error":"…", "kind":"interrupted" } ← aborted mid-step
# legacy — no toolTraceVersion
index 0: { "toolName":"getPage", "input":{...} } ← call (has input, NO output)
index 1: { "toolName":"getPage", "output":{...} } ← success (has output)
or : { "toolName":"getPage", "error":"…" } ← threw (post-#407 only)
index 0: { "toolName": "getPage", "input": { "pageId": "…" } } ← tool-call (has input, NO output)
index 1: { "toolName": "getPage", "output": { … } } ← tool-result (has output, NO input)
```
The keys that can appear: `toolName`, `input` (call), and on the outcome — **v2:**
`ok` **or** `error`+`kind`; **legacy:** `output` **or** (post-#407) `error`. There is
no `state`, no `errorText`, no `type` in `tool_calls` (those live on `metadata.parts`).
Consequences:
The keys that appear on an element are `toolName`, `input`, `output`, and — for a
**thrown** failure on rows written after the #407 fix — `error` (the tool's error
message; see the "Hard failures" section below). There is no `state`, no `errorText`,
no `type`. On pre-#407 rows a thrown failure has NO paired result element at all
(silent orphan). Consequences:
1. **Real invocation count** — count the OUTCOME elements, not every element (else you
double-count): **v2** = elements with `ok` or `error`; **legacy** = elements with
`output` or `error`.
2. **Pairing:** a call (`input`) is followed by its outcome. `toolName` is on both, so
you can group by tool on either. In v2 the `kind` field separates a real hard-fail
(`thrown`) from an aborted call (`interrupted`) — a distinction legacy rows cannot
make (both are orphans; see below).
3. **The tool OUTPUT is only in `metadata.parts` on v2 rows.** To inspect what a tool
returned (soft-error markers, page bodies) on a v2 row, read the parts
(`part->>'type' LIKE 'tool-%'`, `part->>'state' = 'output-available'`, `part->'output'`),
not `tool_calls`.
1. **Real invocation count = elements that have `output` or `error`.** Counting every
element double-counts (you get ~2× and a spurious "~50% of every tool has no output").
2. **Pairing:** a call = a `tool-call` part followed by its result part. A success
carries `output`; a thrown failure (post-#407) carries `error` instead. Both carry
`toolName`, so you can group by tool on either.
## The two classes of failure (and which the DB can see)
### 1. Soft failures — tool RAN and returned an error-shaped result → PERSISTED ✅
These are visible in the tool `output`**on v2 rows in `metadata.parts`** (the
`output-available` part's `output`), on **legacy rows in the `tool_calls` outcome
element's `output`**. The marker differs per tool:
These are visible in the `tool-result` `output`. The marker differs per tool:
| Tool(s) | Error marker in `output` |
| --- | --- |
@@ -141,32 +91,37 @@ element's `output`**. The marker differs per tool:
Note `editPageText` returns `failed: []` on success — filtering on the *presence*
of the key gives false positives; filter on **non-empty**.
### 2. Hard failures — tool THREW → PERSISTED ✅
### 2. Hard failures — tool THREW → NOW PERSISTED ✅ (since the #407 fix)
When a tool throws (the classic one is `patchNode` / `insertNode` / `tableUpdateCell`
`Failed to encode document to Yjs (fromJSON): Unknown node type: undefined`), the
runtime writes **no `tool-result` part** — the failure is an ai@6 `tool-error` content
part. How that lands in `tool_calls` depends on the era:
runtime still writes **no `tool-result` part** — the failure is an ai@6 `tool-error`
content part instead. **Since the #407 fix, that error is persisted**: `serializeSteps`
appends a dedicated element `{toolName, error: "<message>"}` right after the failed
call, mirroring how a successful `{toolName, output}` element is appended. So a thrown
error now leaves a queryable `error` field carrying its (truncated) reason, and the
same real text is replayed to the model on the next turn (an `output-error` part with
the real `errorText`, no longer the `'Tool call did not complete.'` placeholder).
- **v2 (#490):** a `{toolName, error, kind:'thrown'}` outcome element. An interrupted /
aborted mid-step call is a **distinct** `{toolName, error:'Tool call did not
complete.', kind:'interrupted'}` element — so you can tell a real hard-fail from an
abort **directly, without the orphan heuristic**. Query `kind = 'thrown'`.
- **post-#407 legacy:** a `{toolName, error}` element (no `kind`) right after the call.
- **pre-#407 legacy:** the error is **dropped** — a silent **orphan** (a `call` with no
`output` *and* no `error`).
**Cutover caveat — old rows keep the old blind shape.** Rows written **before** this
change have the two-part shape (`call` + `output` only) and simply **drop** thrown
errors, leaving a silent **orphan** (a `call` with no `output` *and* no `error`). Rows
written **after** the fix additionally carry the `error` element. So:
The same real error text is replayed to the model on the next turn (an `output-error`
part with the real `errorText`, from `metadata.parts`), in every era.
- **New rows:** query the `error` field directly (see the hard-error query below) — no
orphan heuristic needed for thrown failures.
- **Old rows (pre-#407):** the only DB-side proxy is still an **orphan**: a `tool-call`
part with no matching `tool-result` *and* no `error`. Orphans also appear when a run
is **aborted** mid-flight (server restart), so a high-volume tool (`createComment`,
`searchInPage`, `Search_web_search`) shows orphans from aborts, not real errors on
old rows. Treat the orphan gap as an *upper bound*, and cross-check the tool: a gap on
a structural editor (`patchNode`, `insertNode`, `updatePageJson`, `transformPage`) is
almost certainly a thrown Yjs-encode error; a gap on `createComment` is mostly aborts.
**Cutover caveat.** Only pre-#407 legacy rows need the orphan proxy: an orphan is a
`tool-call` with no matching outcome. Orphans there also appear when a run is **aborted**
mid-flight (server restart), so a high-volume tool (`createComment`, `searchInPage`,
`Search_web_search`) shows orphans from aborts, not real errors. Treat the orphan gap as
an *upper bound* and cross-check the tool: a gap on a structural editor (`patchNode`,
`insertNode`, `updatePageJson`, `transformPage`) is almost certainly a thrown Yjs-encode
error; a gap on `createComment` is mostly aborts. **On v2 rows this ambiguity is gone**
`kind` labels each outcome.
A note on the aborted-call fallback: a call with **neither** a result **nor** a
`tool-error` (genuinely interrupted mid-step) still replays with the
`'Tool call did not complete.'` placeholder and persists as an orphan — that path is
unchanged, and is distinct from a real thrown error, which now carries `error`.
### 3. Run-level failures → `ai_chat_runs`
@@ -179,34 +134,22 @@ the wild: `Run interrupted by a server restart.` (aborts) and
Run all of these via `docker exec gitmost-postgresql psql -U docmost -d docmost -P pager=off -c "…"`.
**Real invocation count per tool** (outcome parts only — the correct denominator).
Dual-shape: a v2 outcome has `ok` or `error`; a legacy outcome has `output` or `error`:
**Real invocation count per tool** (result parts only — the correct denominator):
```sql
SELECT elem->>'toolName' AS tool, count(*) AS calls
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array'
AND (elem ? 'ok' OR elem ? 'output' OR elem ? 'error')
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
GROUP BY 1 ORDER BY 2 DESC;
```
**Soft errors per tool.** The soft-error marker lives in the tool OUTPUT — which on
**v2 rows is in `metadata.parts`**, on **legacy rows is in the `tool_calls` outcome
element**. This query UNIONs both eras, projecting each output as `o`:
**Soft errors per tool** (everything the DB can honestly see):
```sql
WITH res AS (
-- v2 (#490): output is in metadata.parts (output-available tool parts)
SELECT part->>'type' AS tool, part->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.metadata->'parts') part
WHERE (m.metadata->>'toolTraceVersion') = '2'
AND part->>'type' LIKE 'tool-%' AND part->>'state' = 'output-available'
UNION ALL
-- legacy: output is inline in the tool_calls outcome element
SELECT elem->>'toolName' AS tool, elem->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE (m.metadata->>'toolTraceVersion') IS NULL
AND jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'output'
)
SELECT tool, count(*) AS calls,
sum(COALESCE(
@@ -224,23 +167,13 @@ FROM res GROUP BY tool HAVING sum(COALESCE(
ORDER BY soft_errors DESC;
```
Note the v2 `tool` label is the part type (`tool-editPageText`); strip the `tool-`
prefix if you join it against the legacy `toolName`.
**`editPageText` failure reasons** (the most common real agent mistake — bad `find`).
Same dual-shape output source:
**`editPageText` failure reasons** (the most common real agent mistake — bad `find`):
```sql
WITH res AS (
SELECT part->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.metadata->'parts') part
WHERE (m.metadata->>'toolTraceVersion') = '2'
AND part->>'type' = 'tool-editPageText' AND part->>'state' = 'output-available'
UNION ALL
SELECT elem->'output' AS o
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE (m.metadata->>'toolTraceVersion') IS NULL
AND jsonb_typeof(m.tool_calls) = 'array'
WHERE jsonb_typeof(m.tool_calls) = 'array'
AND elem->>'toolName' = 'editPageText' AND elem ? 'output'
)
SELECT f->>'reason' AS reason, count(*)
@@ -249,43 +182,30 @@ WHERE jsonb_typeof(o->'failed') = 'array'
GROUP BY 1 ORDER BY 2 DESC;
```
**Hard errors — persisted `error` field per tool (v2 + post-#407 rows)** — thrown tool
failures carry their real reason, so query them directly. On **v2** rows exclude the
`interrupted` kind so an aborted call is not counted as a hard-fail:
**Hard errors — persisted `error` field per tool (NEW rows, since #407)** — thrown
tool failures now carry their real reason, so query them directly:
```sql
SELECT elem->>'toolName' AS tool, count(*) AS thrown_errors,
min(elem->>'error') AS sample_error
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem ? 'error'
-- v2 rows label the kind; a legacy error element has no kind (count it).
AND COALESCE(elem->>'kind', 'thrown') = 'thrown'
GROUP BY 1 ORDER BY 2 DESC;
```
Aborted mid-step calls on v2 rows are a distinct, directly countable population:
```sql
SELECT elem->>'toolName' AS tool, count(*) AS interrupted
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND elem->>'kind' = 'interrupted'
GROUP BY 1 ORDER BY 2 DESC;
```
**Hard-error proxy for OLD rows (pre-#407) — orphan gap per tool, WITH a spread column**
(call parts minus outcome parts, plus how many distinct chats the gap is spread across).
This is needed ONLY for pre-#407 legacy rows (v2 and post-#407 rows carry the error /
`kind` directly — use the queries above). The `WHERE` restricts to the legacy era so v2
rows (where an `ok` outcome is not an `output`) never produce phantom orphans:
(call parts minus result parts, plus how many distinct chats the gap is spread across).
This covers rows written before thrown errors were persisted; on new rows a thrown
failure now has its own `error` element (use the query above) and an orphan means only
a genuinely aborted mid-step call:
```sql
WITH parts AS (
SELECT m.chat_id, elem->>'toolName' AS tool,
(elem ? 'input' AND NOT (elem ? 'output') AND NOT (elem ? 'ok')) AS is_call,
(elem ? 'output' OR elem ? 'error' OR elem ? 'ok') AS is_result
(elem ? 'input' AND NOT (elem ? 'output')) AS is_call,
(elem ? 'output' OR elem ? 'error') AS is_result
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
WHERE jsonb_typeof(m.tool_calls) = 'array' AND m.role = 'assistant'
AND (m.metadata->>'toolTraceVersion') IS NULL
),
per_chat AS (
SELECT tool, chat_id, sum(is_call::int) - sum(is_result::int) AS gap
@@ -341,21 +261,11 @@ WHERE tsv @@ websearch_to_tsquery('english', 'some phrase') LIMIT 20;
## Don't blow up your context
Tool outputs embed full page content and search payloads (hundreds of KB per row).
On **legacy** rows they are in `tool_calls`; on **v2** rows they moved to
`metadata->'parts'` (the `tool_calls` trace itself is now small). Never `SELECT
tool_calls` / `metadata` (or `jsonb_pretty(...)`) raw — project just the keys you need
and truncate:
A single `tool_calls` row can be **300–400 KB** (results embed full page content and
search payloads). Never `SELECT tool_calls` (or `jsonb_pretty(tool_calls)`) raw.
Always project just the keys you need and truncate:
```sql
-- v2: outputs live in metadata.parts
SELECT part->>'type',
left(regexp_replace((part->'output')::text, '\s+', ' ', 'g'), 200)
FROM ai_chat_messages m, jsonb_array_elements(m.metadata->'parts') part
WHERE (m.metadata->>'toolTraceVersion') = '2'
AND part->>'state' = 'output-available' LIMIT 5;
-- legacy: outputs live in tool_calls
SELECT elem->>'toolName',
left(regexp_replace((elem->'output')::text, '\s+', ' ', 'g'), 200)
FROM ai_chat_messages m, jsonb_array_elements(m.tool_calls) elem
@@ -370,32 +280,26 @@ docker compose -p gitmost logs -f --tail=100 # whole stack
```
Logging is `json-file`, `max-size=10m max-file=5` → ~50 MB retained, then rotated,
and **wiped on container recreate**. Thrown-tool error text is **persisted** — in the
`error` field of `tool_calls` (v2 `kind:'thrown'` / post-#407 legacy) — so you no longer
depend on live logs for it. Logs/live UI remain useful for **pre-#407 rows** (whose
thrown errors were dropped) and for full stack traces beyond the truncated stored
message. A per-tool `tool_calls_total{tool,status}` metric to VictoriaMetrics is still a
possible future add for aggregate dashboards.
and **wiped on container recreate**. Since the #407 fix, thrown-tool error text is
**persisted in the `error` field** of `tool_calls` (see the hard-error query above), so
you no longer depend on live logs for it. Logs/live UI remain useful for **pre-#407
rows** (whose thrown errors were dropped) and for full stack traces beyond the
truncated stored message. A per-tool `tool_calls_total{tool,status}` metric to
VictoriaMetrics is still a possible future add for aggregate dashboards.
## Gotchas checklist
- [ ] **Check `metadata.toolTraceVersion` first.** v2 (`= 2`) has no output in `tool_calls`; legacy has it inline. Never trend a metric across the era boundary.
- [ ] Counting every `tool_calls` element → **overcount**. Count OUTCOME elements — v2: `ok` or `error`; legacy: `output` or `error` — never both call+outcome as invocations.
- [ ] `isError` / `success=false` ≈ 0 does **not** mean "no errors" — thrown errors are an `error` element (v2 `kind:'thrown'` / post-#407), not in the output.
- [ ] **v2:** soft-error markers (the tool output) are in `metadata.parts`, NOT `tool_calls`. Legacy: they are in the `tool_calls` outcome `output`.
- [ ] **v2:** `kind` splits a real hard-fail (`thrown`) from an aborted call (`interrupted`) directly — no orphan heuristic needed. The orphan gap is a pre-#407-legacy-only proxy.
- [ ] Counting every `tool_calls` element → **overcount**. Count `output` elements; add `error` elements for thrown failures (new rows), but don't count both as invocations.
- [ ] `isError` / `success=false` ≈ 0 does **not** mean "no errors" — thrown errors are a separate `error` element (new rows) or dropped entirely (pre-#407 rows).
- [ ] Thrown errors persist only on rows written **after the #407 fix** — pre-#407 rows still drop them (orphan only). Mind the cutover when trending over time.
- [ ] `editPageText.failed` is `[]` on success — test for **non-empty**, not presence.
- [ ] Orphan gap on OLD rows mixes thrown errors **and** aborted runs — split by tool. On NEW rows a thrown error is its own `error` element, so a gap ≈ aborted call.
- [ ] `aborted` runs = server restarts, `failed` runs = provider overload — not agent mistakes.
- [ ] Never dump a raw `tool_calls` **or** `metadata.parts` cell — outputs are hundreds of KB.
- [ ] Logs are ephemeral (≤50 MB, wiped on recreate) — grab pre-#407 hard-error text live.
- [ ] Never dump a raw `tool_calls` cell — it can be hundreds of KB.
- [ ] Logs are ephemeral (≤50 MB, wiped on recreate) — grab hard-error text live.
## Snapshot (2026-07-07, illustrative — rerun the queries for current numbers)
> All rows in this snapshot predate #490, so they are **legacy-era** (outputs inline in
> `tool_calls`, orphan proxy for thrown errors). Do not trend these numbers against v2
> rows — segment by `toolTraceVersion` first.
- 226 chats, 732 messages, 46 runs; ~4 400 real tool invocations.
- Soft errors (persisted): `editPageText` 4/79 (bad/non-unique `find`) + 9 markdown-in-`find` warnings; `semanticSearch` 3/4 (`unavailable`); `Habr_update_draft_from_docmost` 1/2 (`doc` sent as object, not string).
- Missing-result proxy, read WITH the spread column:
+2 -8
View File
@@ -72,13 +72,7 @@ export async function stabilizePageFile(
* keeps re-pulls of an unchanged page byte-identical (no churn, loop-guard).
*/
export async function stabilizePageBody(content: unknown): Promise<string> {
// git-sync is the LOSSLESS mirror path, so run the serializer in `strict`
// mode: a node/mark type the converter has no case for (e.g. one added to the
// schema without a matching serializer arm) throws a ConverterLossError here
// rather than silently degrading — surfacing the loss loudly at write time
// instead of committing a lossy file. Valid content (every current schema type
// has a case) is unaffected.
const md1 = convertProseMirrorToMarkdown(content, { strict: true });
const md1 = convertProseMirrorToMarkdown(content);
const doc2 = await markdownToProseMirror(md1);
return convertProseMirrorToMarkdown(doc2, { strict: true });
return convertProseMirrorToMarkdown(doc2);
}

Some files were not shown because too many files have changed in this diff Show More