Compare commits

...

3 Commits

Author SHA1 Message Date
agent_coder 2e6f1c3de5 fix(ci,mcp): REGISTRY_STAMP в билд-артефакте + кросс-пакетный CI — закрыть skew build/ vs src/ (#447)
Два структурных слепых пятна: (1) сервер грузит СКОМПИЛИРОВАННЫЙ
packages/mcp/build/index.js, а серверные guard-тесты читают src/tool-specs.ts —
правка src без пересборки оставляет тесты зелёными, но рантайм расходится со
спеками; (2) спеки добавляют в packages/mcp, а parity-тесты живут в jest-сьюте
apps/server — PR, трогающий только пакет, проходит зелёным, сломанная in-app-
проводка всплывает уже на develop (кейс f46d89ea).

- REGISTRY_STAMP: codegen (scripts/gen-registry-stamp.mjs) считает sha256 от
  нормализованного (CRLF→LF, один хвостовой \n снят) сырого текста
  src/tool-specs.ts, пишет src/registry-stamp.generated.ts (gitignored),
  index.ts реэкспортит → попадает в build/. Вшит в build/pretest/watch ДО tsc.
- Loader (dev/test): computeSrcRegistryStamp пересчитывает стамп из src рядом с
  build/index.js (dev-vs-prod по existsSync, любая ошибка → null), сверяет с
  build-стампом → при рассинхроне бросает «build is stale — rebuild». В prod
  (src нет) и на pre-#447 билдах (нет REGISTRY_STAMP) — чистый no-op.
- CI: job mcp-server-parity собирает shared-deps+mcp (регенерит стамп) и гоняет
  ОБА сьюта вместе (mcp node:test + server guard-спеки) — именованный гейт, его
  нельзя случайно расщепить.
- AGENTS.md: правка спеков требует ребилда @docmost/mcp.

Тесты (20): mcp-сайд (детерминизм, нормализация, desync-гард стамп-vs-билд) +
server-сайд (null при отсутствии src = prod no-op; mismatch → throw точного
сообщения; pre-#447 no-op). Кросс-импл equality-гард: один фиксированный вход →
один хэш на ОБЕИХ сторонах, ловит рассинхрон двух нормализаций. Внутреннее
ревью: APPROVE WITH SUGGESTIONS (обе — покрытие guard'а — закрыты этим тестом).
Мутационно: любой из двух normalize-имплов расходится → equality-тест краснеет.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:58:37 +03:00
vvzvlad f8d37d8956 Merge pull request 'fix(mcp): курсорная пагинация — устранить тихую потерю страниц в list_pages/check_new_comments (#442)' (#451) from fix/442-cursor-pagination into develop
Reviewed-on: #451
2026-07-10 07:27:31 +03:00
agent_coder 90168eb926 fix(mcp): курсорная пагинация вместо офсетной — устранить тихую потерю страниц (#442)
Апстрим 78b1c1a4 перевёл серверные эндпоинты на КУРСОРНУЮ пагинацию, а
ValidationPipe({whitelist:true}) молча вырезает неизвестное поле `page`.
MCP-клиент так и слал офсетный `page` → сервер отдавал ту же первую двадцатку
с hasNextPage:true, цикл выкручивался до MAX_PAGES=50 одинаковых запросов, дети
№21+ не выгружались (с поддеревьями). Дедуп `visited` гасил дубли → «дырявое»
дерево без ошибок. Netmap: 20/299 страниц терялось, 160 запросов вместо 62.

- A: enumerateSpacePages → один POST /pages/tree (весь спейс/поддерево разом);
  fallback на курсорный BFS при 404/405 (stock upstream). Возврат {pages,
  truncated}; truncated честный — true только при реальном упоре fallback-BFS в
  MAX_NODES.
- B: listSidebarPages → курсорный цикл, limit:100, guard на неподвижный курсор
  (!next || next===cursor → break) — если протокол снова разойдётся, не крутит
  дубли молча; warn при упоре в MAX_PAGES.
- C: paginateAll (/spaces, /shares) → та же курсорная миграция + guard.
- D: check_new_comments — /pages/tree поддерева включает корень
  (getPageAndDescendants), убран лишний getPageRaw; в fallback корень
  засевается явно (иначе его комменты терялись — регрессия того же класса).
- listComments: do/while → for с MAX_PAGES + guard неподвижного курсора
  (был безлимитный — тот же сценарий #442 дал бы бесконечный цикл).

Внутренний цикл: 2 прохода. Первый нашёл потерю комментов корня в fallback
поддерева (data-loss) → засев корня; догрёб honest-truncated, warn в
listSidebarPages, guard в listComments. Второй проход — APPROVE, форма возврата
{pages,truncated} распространена на оба вызова без пропусков.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 07:08:56 +03:00
12 changed files with 1131 additions and 100 deletions
+58
View File
@@ -1,5 +1,13 @@
name: Test
# NO `paths:` filter on purpose (issue #447). The tool-spec REGISTRY is split
# across two packages that MUST stay in sync: the specs live in `packages/mcp`
# but the parity/tier guard tests that read them live in the `apps/server` jest
# suite. A PR touching only `packages/mcp/**` must therefore still run the SERVER
# suite (and vice-versa), or an in-app wiring break slips through green and only
# surfaces on develop after merge. The `test` job below runs BOTH suites via
# `pnpm -r test` on every PR; the dedicated `mcp-server-parity` job makes that
# cross-package gate explicit and fast. Do not add a `paths:` filter here.
on:
pull_request:
workflow_call:
@@ -132,3 +140,53 @@ jobs:
# isolated `docmost_test` DB and migrates it to latest.
- name: Run server integration tests
run: pnpm --filter server test:int
# Cross-package tool-spec parity gate (issue #447). The tool-spec registry lives
# in `packages/mcp` but its parity/tier guard tests live in the `apps/server`
# jest suite, so a PR touching ONLY one of the two packages must still run BOTH
# sides — otherwise an in-app wiring break (e.g. PR #434 drawio) passes the mcp
# suite green and only surfaces on develop after merge. The `test` job already
# runs everything via `pnpm -r test`; this job is a fast, explicitly-named guard
# that runs the mcp `node --test` suite AND the server tool-guard jest specs
# together, so the coupling is visible and can never be accidentally split by a
# path filter. No Postgres/Redis needed: these specs mock the DB/loader.
mcp-server-parity:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up pnpm
uses: pnpm/action-setup@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Shared deps first (build/ dirs are gitignored; see test.yml build order).
- name: Build editor-ext
run: pnpm --filter @docmost/editor-ext build
- name: Build prosemirror-markdown
run: pnpm --filter @docmost/prosemirror-markdown build
# Build the mcp package so build/ carries a FRESH REGISTRY_STAMP (#447): the
# build runs gen-registry-stamp.mjs before tsc, so a build/ vs src/ skew
# cannot slip into the tests that exercise the loader's stale-check.
- name: Build mcp (regenerates REGISTRY_STAMP)
run: pnpm --filter @docmost/mcp build
# mcp side: the standalone MCP server's own tool-spec / instructions guards.
- name: Run mcp tool-spec suite
run: pnpm --filter @docmost/mcp test
# server side: the parity + tier guards that read packages/mcp/src/tool-specs
# and assert the in-app AI-chat wiring matches it.
- name: Run server tool-spec guard specs
run: pnpm --filter server exec jest shared-tool-specs.contract tool-tiers ai-chat-tools.service --runInBand
+5
View File
@@ -19,6 +19,11 @@ packages/prosemirror-markdown/build/
# markdown convention; the package is private and rebuilt at deploy.
packages/mcp/build/
# mcp REGISTRY_STAMP codegen output (issue #447). Regenerated into src/ by
# scripts/gen-registry-stamp.mjs on every `build`/`pretest` (before tsc), so it
# is a build artifact like build/ — never committed, always fresh.
packages/mcp/src/registry-stamp.generated.ts
# Logs
logs
*.log
+16
View File
@@ -248,6 +248,22 @@ pnpm collab:dev # run the collaboration server process standalone (
> that order). Reach for it whenever you run a consumer package's checks on their
> own rather than through the full `pnpm build`.
> **Editing an MCP tool spec requires a rebuild (issue #447).** The running
> server loads the **compiled** `packages/mcp/build/` of `@docmost/mcp` (via the
> runtime loader in `apps/server/src/core/ai-chat/tools/docmost-client.loader.ts`),
> but the parity/tier guard tests read `packages/mcp/src/tool-specs.ts`. So if you
> edit `tool-specs.ts` (any tool name, description, tier, catalog line, or input
> schema) **without rebuilding**, `build/` and `src/` silently diverge — the tests
> stay green while the server serves the OLD tools. To close that gap, the build
> emits a `REGISTRY_STAMP` (a deterministic hash of the tool-specs content, via
> `scripts/gen-registry-stamp.mjs` before `tsc`); on dev/test startup the loader
> recomputes it from `src/` and **refuses to start with a "@docmost/mcp build is
> stale …" error** on a mismatch (a pure no-op in prod, where only `build/` ships).
> After editing tool specs, rebuild:
> ```bash
> pnpm --filter @docmost/mcp build # or: pnpm --filter @docmost/mcp watch
> ```
**Lint** (per package — there is no root lint script):
```bash
pnpm --filter server lint # eslint --fix on server .ts
@@ -0,0 +1,173 @@
import { createHash } from 'node:crypto';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { computeSrcRegistryStamp } from './docmost-client.loader';
// The exact message the loader throws on a build/src skew (issue #447). Kept as a
// literal here so a reworded prod message reddens this test (the message is a
// developer-facing contract: it tells them how to fix it).
const STALE_BUILD_MESSAGE =
'@docmost/mcp build is stale (tool-specs changed since last build) — run: pnpm --filter @docmost/mcp build';
// Replica of the loader's inline stale-check predicate + throw from
// `loadDocmostMcp`. That guard is not independently exported (it lives inside the
// dynamic-import IIFE, wired to a fixed `require.resolve('@docmost/mcp')`), so we
// exercise the exact same three-condition logic against a stamp produced by the
// REAL `computeSrcRegistryStamp`. This documents and locks the throw/no-throw
// behaviour; if the prod predicate changes, this replica must change with it.
function assertStaleGuard(
srcStamp: string | null,
registryStamp: string | undefined,
): void {
if (
srcStamp !== null &&
typeof registryStamp === 'string' &&
srcStamp !== registryStamp
) {
throw new Error(STALE_BUILD_MESSAGE);
}
}
// Build a throwaway `<pkg>/build/index.js` + optional `<pkg>/src/tool-specs.ts`
// layout so `computeSrcRegistryStamp(<pkg>/build/index.js)` resolves src the same
// way the loader does (dirname(dirname(entry))/src/tool-specs.ts).
function makeFakePackage(toolSpecsSource: string | null): {
entry: string;
cleanup: () => void;
} {
const root = mkdtempSync(join(tmpdir(), 'mcp-stamp-'));
const buildDir = join(root, 'build');
mkdirSync(buildDir, { recursive: true });
const entry = join(buildDir, 'index.js');
writeFileSync(entry, '// fake @docmost/mcp build entry\n', 'utf8');
if (toolSpecsSource !== null) {
const srcDir = join(root, 'src');
mkdirSync(srcDir, { recursive: true });
writeFileSync(join(srcDir, 'tool-specs.ts'), toolSpecsSource, 'utf8');
}
return { entry, cleanup: () => rmSync(root, { recursive: true, force: true }) };
}
describe('computeSrcRegistryStamp (#447 stale-build guard)', () => {
it('returns null when src/tool-specs.ts is absent (prod no-op path)', () => {
// A prod image ships only build/, no src/ — the guard must be a silent no-op.
const { entry, cleanup } = makeFakePackage(null);
try {
expect(computeSrcRegistryStamp(entry)).toBeNull();
} finally {
cleanup();
}
});
it('returns null for a bogus package entry (swallowed error path)', () => {
// A resolution/read hiccup must NEVER break startup — it resolves to null.
expect(
computeSrcRegistryStamp('/no/such/pkg/build/index.js'),
).toBeNull();
});
it('computes a 64-char sha256 hex when src/tool-specs.ts exists', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const stamp = computeSrcRegistryStamp(entry);
expect(stamp).toMatch(/^[0-9a-f]{64}$/);
} finally {
cleanup();
}
});
it('normalizes CRLF->LF and strips a single trailing newline', () => {
// A CRLF+trailing-newline variant of the same content hashes identically to
// the bare-LF form — the guard must not fire on a checkout-style difference.
const bare = makeFakePackage('alpha\nbeta');
const crlfTrailing = makeFakePackage('alpha\r\nbeta\r\n');
try {
expect(computeSrcRegistryStamp(crlfTrailing.entry)).toBe(
computeSrcRegistryStamp(bare.entry),
);
} finally {
bare.cleanup();
crlfTrailing.cleanup();
}
});
// CROSS-IMPL EQUALITY (covers reviewer suggestion 2). The SAME fixed input and
// EXPECTED hash are asserted in the mcp-side node test
// (packages/mcp/test/unit/registry-stamp.test.mjs) against the codegen's
// `computeRegistryStamp`. Asserting the SAME pair here against the loader's
// `computeSrcRegistryStamp` proves both implementations normalize+hash
// identically; a divergence in EITHER side reddens one of the two tests.
it('matches the documented cross-impl hash for a fixed input', () => {
const FIXED_INPUT = 'line1\r\nline2\n';
const EXPECTED =
'683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83';
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
try {
expect(computeSrcRegistryStamp(entry)).toBe(EXPECTED);
} finally {
cleanup();
}
});
it('the documented EXPECTED is the normalize+sha256 of the fixed input', () => {
// Proves EXPECTED is not a magic constant but the documented computation.
const FIXED_INPUT = 'line1\r\nline2\n';
const normalized = FIXED_INPUT.replace(/\r\n/g, '\n').replace(/\n$/, '');
const expected = createHash('sha256')
.update(normalized, 'utf8')
.digest('hex');
const { entry, cleanup } = makeFakePackage(FIXED_INPUT);
try {
expect(computeSrcRegistryStamp(entry)).toBe(expected);
} finally {
cleanup();
}
});
});
describe('loadDocmostMcp stale-check predicate (#447)', () => {
it('THROWS the exact stale message when src stamp != built REGISTRY_STAMP', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
expect(srcStamp).not.toBeNull();
// Simulate a stale build: build/ carries a DIFFERENT stamp than src.
expect(() => assertStaleGuard(srcStamp, 'a'.repeat(64))).toThrow(
STALE_BUILD_MESSAGE,
);
} finally {
cleanup();
}
});
it('does NOT throw when src stamp equals the built REGISTRY_STAMP', () => {
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
// Fresh build: build/ stamp == src stamp -> guard is a no-op.
expect(() => assertStaleGuard(srcStamp, srcStamp as string)).not.toThrow();
} finally {
cleanup();
}
});
it('does NOT throw when src is absent (prod: srcStamp === null)', () => {
// Even against a present-but-mismatched REGISTRY_STAMP, a null src stamp
// (prod image with build/ only) must skip the check entirely.
expect(() => assertStaleGuard(null, 'a'.repeat(64))).not.toThrow();
});
it('does NOT throw when REGISTRY_STAMP is absent (pre-#447 build)', () => {
// An older @docmost/mcp build has no REGISTRY_STAMP export; the guard must be
// a no-op so an out-of-date build never wrongly blocks startup.
const { entry, cleanup } = makeFakePackage('export const specs = 1;\n');
try {
const srcStamp = computeSrcRegistryStamp(entry);
expect(() => assertStaleGuard(srcStamp, undefined)).not.toThrow();
} finally {
cleanup();
}
});
});
@@ -1,3 +1,6 @@
import { createHash } from 'node:crypto';
import { existsSync, readFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { pathToFileURL } from 'node:url';
/**
@@ -344,6 +347,50 @@ interface DocmostMcpModule {
// loader in unit tests. The in-app layer treats an absent factory as "signal
// disabled" — a pure no-op that leaves tool results byte-identical.
createCommentSignalTracker?: CommentSignalTrackerFactory;
// Optional (#447): a deterministic hash of the tool-specs registry content,
// generated into build/ by the package's build. Absent on a pre-#447 build (or
// the mocked loader in unit tests) — the stale-check below is a NO-OP when it
// is missing, so an older build never wrongly fails startup.
REGISTRY_STAMP?: string;
}
/**
* Recompute the REGISTRY_STAMP (#447) from the @docmost/mcp source tree, if it is
* present. Returns the stamp string, or `null` when the source is absent (a prod
* image ships only build/, no src/). MUST stay byte-for-byte identical to
* packages/mcp/scripts/gen-registry-stamp.mjs's `computeRegistryStamp` so the
* build-time and src-time hashes agree: same input file (src/tool-specs.ts), same
* normalization (CRLF -> LF, strip a single trailing newline), same sha256.
*
* DEV vs PROD detection is by FILE EXISTENCE, not NODE_ENV: we resolve the
* package's own directory from `require.resolve('@docmost/mcp')` (which points at
* build/index.js) and look for ../src/tool-specs.ts next to it. In a dev/test
* worktree that file exists; in a prod image (build/ only, src/ stripped) it does
* not, so this returns null and the caller skips the check. Any error (ENOENT, a
* bad resolve) is swallowed to null — the stale-check must NEVER break startup.
*
* Exported for unit testing (docmost-client.loader.spec.ts): the export keyword
* is behaviourally a no-op — the module-internal caller `loadDocmostMcp` is
* unaffected. The test drives the null (no-src) path and asserts this
* normalize+sha256 stays identical to the codegen's `computeRegistryStamp`.
*/
export function computeSrcRegistryStamp(packageEntry: string): string | null {
try {
// packageEntry is <pkg>/build/index.js; the source lives at <pkg>/src/.
const toolSpecsPath = join(
dirname(dirname(packageEntry)),
'src',
'tool-specs.ts',
);
if (!existsSync(toolSpecsPath)) return null; // prod: no src tree -> skip.
const source = readFileSync(toolSpecsPath, 'utf8');
const normalized = source.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
} catch {
// Never let a resolution/read hiccup break server startup — treat as "no
// src available" and skip the check (identical to the prod no-op path).
return null;
}
}
// TS with module:commonjs downlevels a literal `import()` to `require()`, which
@@ -375,6 +422,23 @@ export async function loadDocmostMcp(): Promise<{
const mod = (await esmImport(
pathToFileURL(entry).href,
)) as DocmostMcpModule;
// #447 stale-build guard (dev/test only). The server loads the COMPILED
// build/ of @docmost/mcp, but the parity/tier guard tests read src/. If a
// tool spec is edited in src without rebuilding the package, build/ and src/
// silently diverge and the running server serves the OLD tools. Here we
// recompute the stamp from src/tool-specs.ts and compare it to the stamp
// baked into build/. In PROD the src tree is absent (image ships build/
// only), so computeSrcRegistryStamp returns null and this is a pure no-op.
const srcStamp = computeSrcRegistryStamp(entry);
if (
srcStamp !== null &&
typeof mod.REGISTRY_STAMP === 'string' &&
srcStamp !== mod.REGISTRY_STAMP
) {
throw new Error(
'@docmost/mcp build is stale (tool-specs changed since last build) — run: pnpm --filter @docmost/mcp build',
);
}
return mod;
})().catch((err) => {
// Do not cache a rejected import — allow the next call to retry.
+4 -3
View File
@@ -13,10 +13,11 @@
"docmost-mcp": "./build/stdio.js"
},
"scripts": {
"build": "tsc",
"gen:stamp": "node scripts/gen-registry-stamp.mjs",
"build": "node scripts/gen-registry-stamp.mjs && tsc",
"start": "node build/stdio.js",
"watch": "tsc --watch",
"pretest": "tsc",
"watch": "node scripts/gen-registry-stamp.mjs && tsc --watch",
"pretest": "node scripts/gen-registry-stamp.mjs && tsc",
"test": "node --test \"test/unit/*.test.mjs\" \"test/mock/*.test.mjs\"",
"test:unit": "node --test \"test/unit/*.test.mjs\"",
"test:mock": "node --test \"test/mock/*.test.mjs\"",
@@ -0,0 +1,67 @@
// Codegen: emit src/registry-stamp.generated.ts with a REGISTRY_STAMP hash of
// the tool-specs REGISTRY CONTENT, so a build/ vs src/ skew (issue #447) is
// detectable at runtime.
//
// WHY hash the raw source text (not extracted structured data):
// SHARED_TOOL_SPECS carries `buildShape` functions (the input SCHEMAS) which are
// NOT serializable. The input schema is exactly one of the things that MUST stay
// in sync between build/ and src/, so we cannot drop it from the hash. Rather
// than probe zod with a fragile shim to reconstruct the schema shape, we hash the
// STABLE, deterministic source TEXT of tool-specs.ts. That text fully captures
// every field that must stay in sync — mcpName, inAppKey, description, tier,
// catalogLine AND the buildShape bodies (input schemas) — with zero probing
// fragility. Any edit to a spec (a renamed tool, a reworded description, a
// changed schema field) changes the text and therefore the stamp.
//
// DETERMINISM: the hash is computed over the file bytes with line endings
// normalized to LF and a single trailing newline stripped, so a CRLF checkout or
// an editor's trailing-newline habit cannot make build/ and src/ disagree. No
// Date.now / randomness. The loader's dev-only stale-check (docmost-client.loader.ts)
// re-runs THIS SAME normalization + sha256 over src/tool-specs.ts and compares to
// the built REGISTRY_STAMP; the two must compute identically.
//
// This script runs from the `build` and `pretest` npm scripts BEFORE tsc, so
// build/ always carries a stamp derived from the tool-specs.ts that was compiled.
import { createHash } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const SRC_DIR = join(__dirname, '..', 'src');
const TOOL_SPECS_PATH = join(SRC_DIR, 'tool-specs.ts');
const OUT_PATH = join(SRC_DIR, 'registry-stamp.generated.ts');
/**
* Deterministic stamp of the tool-specs registry content. Kept as a plain
* function (exported) so the algorithm has a single home; the loader duplicates
* only the tiny normalize+sha256 steps because it lives in the CJS server build
* and cannot import this ESM script. If you change the normalization here, mirror
* it in apps/server/src/core/ai-chat/tools/docmost-client.loader.ts.
*/
export function computeRegistryStamp(toolSpecsSource) {
const normalized = toolSpecsSource.replace(/\r\n/g, '\n').replace(/\n$/, '');
return createHash('sha256').update(normalized, 'utf8').digest('hex');
}
function main() {
const source = readFileSync(TOOL_SPECS_PATH, 'utf8');
const stamp = computeRegistryStamp(source);
const out =
'// AUTO-GENERATED by scripts/gen-registry-stamp.mjs — DO NOT EDIT BY HAND.\n' +
'// A deterministic hash of src/tool-specs.ts content (tool names, descriptions,\n' +
'// tiers, catalog lines and input schemas). Regenerated on every build/pretest\n' +
'// so build/ always matches the compiled src. The in-app loader recomputes this\n' +
'// from src and refuses to run on a mismatch (issue #447). This file is\n' +
'// gitignored and produced by the build — see .gitignore.\n' +
`export const REGISTRY_STAMP = ${JSON.stringify(stamp)};\n`;
writeFileSync(OUT_PATH, out, 'utf8');
// eslint-disable-next-line no-console
console.log(`gen-registry-stamp: wrote ${OUT_PATH} (${stamp.slice(0, 12)}…)`);
}
// Only run when invoked directly (not when imported for computeRegistryStamp).
if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
main();
}
+169 -78
View File
@@ -553,16 +553,18 @@ export class DocmostClient {
// forever and accumulate duplicates).
const MAX_PAGES = 50;
let page = 1;
let cursor: string | undefined;
let allItems: T[] = [];
let hasNextPage = true;
let truncated = false;
while (hasNextPage && page <= MAX_PAGES) {
const response = await this.client.post(endpoint, {
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = {
...basePayload,
limit: clampedLimit,
page,
});
};
if (cursor) payload.cursor = cursor;
const response = await this.client.post(endpoint, payload);
const data = response.data;
const items = data.data?.items || data.items || [];
@@ -570,22 +572,28 @@ export class DocmostClient {
allItems = allItems.concat(items);
// Stop if the page is empty or shorter than the requested size: a full
// page worth of items is the only situation where another page can exist,
// so this defends against a stuck hasNextPage flag in addition to it.
if (items.length === 0 || items.length < clampedLimit) {
// Advance strictly via the server-issued cursor. A missing nextCursor (or
// hasNextPage false) means we reached the end. A cursor identical to the
// one we just sent means the server did not understand our pagination
// param — stop instead of re-fetching page one forever and duplicating.
const next = meta?.hasNextPage ? meta?.nextCursor : null;
if (!next || next === cursor) {
// If the server still reports more pages but stopped issuing a usable
// cursor at the ceiling, flag the result as truncated below.
if (page === MAX_PAGES - 1 && meta?.hasNextPage) truncated = true;
break;
}
cursor = next;
hasNextPage = meta?.hasNextPage || false;
page++;
// Reaching the ceiling with more pages still available means the result
// set is truncated.
if (page === MAX_PAGES - 1) truncated = true;
}
// If the loop stopped because it hit the MAX_PAGES ceiling while the server
// still reported more results (hasNextPage true and the last page was
// full), the result set is truncated — warn so the caller is not silently
// handed an incomplete list.
if (hasNextPage && page > MAX_PAGES) {
// still reported more results, the result set is truncated — warn so the
// caller is not silently handed an incomplete list.
if (truncated) {
console.warn(
`paginateAll: results from "${endpoint}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
@@ -619,9 +627,10 @@ export class DocmostClient {
* Tree (`tree` true): the space's FULL page hierarchy as a nested tree (each
* node has a `children` array). This mode REQUIRES `spaceId` (a page tree is
* scoped to one space) and IGNORES `limit` — the whole hierarchy is returned.
* It walks the sidebar tree via `enumerateSpacePages`, which performs N
* sidebar requests and is bounded by that method's 10000-node cap (and skips
* soft-deleted pages server-side).
* It fetches the tree via `enumerateSpacePages`, which on the fork server
* resolves to a single `/pages/tree` request returning the whole
* permission-filtered flat page set (soft-deleted pages excluded
* server-side).
*/
async listPages(spaceId?: string, limit: number = 50, tree: boolean = false) {
await this.ensureAuthenticated();
@@ -632,8 +641,8 @@ export class DocmostClient {
"list_pages: tree mode requires a spaceId (a page tree is scoped to one space). Pass spaceId, or omit tree to get the recent-pages list.",
);
}
const nodes = await this.enumerateSpacePages(spaceId);
return buildPageTree(nodes);
const { pages } = await this.enumerateSpacePages(spaceId);
return buildPageTree(pages);
}
const clampedLimit = Math.max(1, Math.min(100, limit));
@@ -655,57 +664,123 @@ export class DocmostClient {
async listSidebarPages(spaceId: string, pageId?: string) {
await this.ensureAuthenticated();
// Paginate: the endpoint returns server-paged children, so posting only
// { page: 1 } silently dropped every child beyond the first page. Loop on
// meta.hasNextPage (with a MAX_PAGES ceiling like paginateAll, guarding
// against a stuck hasNextPage flag) and accumulate all children.
// Paginate via the server-issued cursor. The server switched from OFFSET
// (`page`) to CURSOR (`cursor`/`nextCursor`) pagination, and the global
// ValidationPipe(whitelist:true) SILENTLY STRIPS the obsolete `page` field
// — so the old offset loop got the SAME first page every time (with
// hasNextPage stuck true) and dropped every child beyond the first page.
const MAX_PAGES = 50;
let page = 1;
let cursor: string | undefined;
let allItems: any[] = [];
let hasNextPage = true;
let truncated = false;
while (hasNextPage && page <= MAX_PAGES) {
for (let i = 0; i < MAX_PAGES; i++) {
// limit: 100 is the server-side Max; cuts request count 5x vs the default 20.
const payload: Record<string, any> = { spaceId, limit: 100 };
// Only send pageId when scoping to a page's children; omit it for roots.
const payload: Record<string, any> = { spaceId, page };
if (pageId) payload.pageId = pageId;
if (cursor) payload.cursor = cursor;
const response = await this.client.post("/pages/sidebar-pages", payload);
const data = response.data?.data ?? response.data;
const items = data?.items || [];
allItems = allItems.concat(items);
const data = (await this.client.post("/pages/sidebar-pages", payload)).data
?.data;
allItems = allItems.concat(data?.items ?? []);
hasNextPage = data?.meta?.hasNextPage || false;
page++;
// Advance strictly via the server-issued cursor; a missing/repeated cursor
// means the protocol drifted again — stop instead of looping on page one.
const next = data?.meta?.hasNextPage ? data?.meta?.nextCursor : null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with more pages still available means the child
// list is truncated (mirrors paginateAll).
if (i === MAX_PAGES - 1) truncated = true;
}
// Warn on real truncation (ceiling hit while the server still had pages) so
// the caller is not silently handed an incomplete child list.
if (truncated) {
console.warn(
`listSidebarPages: children of "${pageId ?? spaceId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
return allItems;
}
/**
* Enumerate EVERY page in a space (or in a subtree, when rootPageId is given)
* by walking the sidebar-pages tree.
* Enumerate EVERY page in a space (or in a subtree, when rootPageId is given).
*
* Starting set: the children of rootPageId when provided, otherwise the
* space root pages. From there it does an iterative breadth-first walk: each
* node is collected, and when node.hasChildren is true its direct children
* are fetched via listSidebarPages(spaceId, node.id) and enqueued.
* Primary path (fork server): a SINGLE `POST /pages/tree` returns the whole
* space (or a subtree) as a flat, permission-filtered list in one request, in
* the exact node shape buildPageTree consumes. This replaces the old
* per-node BFS, which issued N sidebar requests and — after the server moved
* to cursor pagination — silently lost every child past the first sidebar
* page (the obsolete `page` param was stripped by ValidationPipe).
*
* This replaces the old "/pages/recent" enumeration, which is a bounded
* recent-activity feed (~5000 cap) and therefore misses comments on older
* pages that were never recently touched.
* The subtree variant (rootPageId given) INCLUDES the root node itself
* (getPageAndDescendants seeds with id = rootPageId), unlike the old BFS
* which started from the root's children.
*
* Safeguards: a `visited` Set of page ids prevents re-processing a node
* (cycles / duplicate references), and a hard node cap bounds pathological
* trees so the walk always terminates.
* Fallback path (stdio mode may target STOCK upstream Docmost, which lacks
* `/pages/tree`): on a 404/405 it falls back to the cursor-based BFS below,
* walking direct children via the fixed cursor listSidebarPages. Safeguards:
* a `visited` Set of page ids prevents re-processing a node (cycles /
* duplicate references), and a hard node cap bounds pathological trees so the
* walk always terminates.
*
* Returns `{ pages, truncated }`. `truncated` is true ONLY when the fallback
* BFS stopped at its MAX_NODES cap — the primary /pages/tree path is uncapped
* and always returns the complete set, so it never reports truncation.
*/
private async enumerateSpacePages(
spaceId: string,
rootPageId?: string,
): Promise<any[]> {
): Promise<{ pages: any[]; truncated: boolean }> {
await this.ensureAuthenticated();
// Single request replaces the whole BFS: /pages/tree returns the full
// permission-filtered flat page set of a space (or a subtree) at once. This
// path is uncapped, so it is never truncated.
const payload = rootPageId ? { pageId: rootPageId } : { spaceId };
try {
const response = await this.client.post("/pages/tree", payload);
const pages = (response.data?.data ?? response.data)?.items ?? [];
return { pages, truncated: false };
} catch (e: any) {
// Only fall back when the endpoint is absent (stock upstream Docmost);
// any other error is a genuine failure and must propagate.
if (
!axios.isAxiosError(e) ||
(e.response?.status !== 404 && e.response?.status !== 405)
) {
throw e;
}
}
// Fallback: cursor-based breadth-first walk via listSidebarPages.
const MAX_NODES = 10000;
const result: any[] = [];
const visited = new Set<string>();
// Seed with the root node itself when scoping to a subtree, so its own
// comments aren't dropped: the primary /pages/tree seeds
// getPageAndDescendants with id = rootPageId (root included), but
// listSidebarPages(spaceId, rootPageId) returns only the root's CHILDREN.
// The `visited` set below prevents a double-add if the root also appears
// among the children. getPageRaw returns a page whose id/title/spaceId are
// exactly what buildPageTree and check_new_comments consume.
if (rootPageId) {
try {
const root = await this.getPageRaw(rootPageId);
if (root?.id) {
result.push(root);
visited.add(root.id);
}
} catch {
// Non-fatal: if the root can't be read, fall through to children-only.
}
}
// Seed the queue with the starting level (subtree children or roots).
const queue: any[] = await this.listSidebarPages(spaceId, rootPageId);
@@ -730,7 +805,12 @@ export class DocmostClient {
}
}
return result;
// Truncated only when the cap was hit with the queue still non-empty (real
// truncation, not a natural end at exactly MAX_NODES).
return {
pages: result,
truncated: result.length >= MAX_NODES && queue.length > 0,
};
}
/** Raw page info including the ProseMirror JSON content and slugId. */
@@ -2364,7 +2444,13 @@ export class DocmostClient {
let allComments: any[] = [];
let cursor: string | null = null;
do {
// Hard ceiling + immovable-cursor guard (mirrors paginateAll): if /comments
// ever stops advancing the cursor (the exact #442 drift scenario) this loop
// would otherwise spin forever accumulating duplicates.
const MAX_PAGES = 50;
let truncated = false;
for (let page = 0; page < MAX_PAGES; page++) {
const payload: Record<string, any> = { pageId, limit: 100 };
if (cursor) payload.cursor = cursor;
@@ -2372,8 +2458,23 @@ export class DocmostClient {
const data = response.data.data || response.data;
const items = data.items || [];
allComments = allComments.concat(items);
cursor = data.meta?.nextCursor || null;
} while (cursor);
// Advance strictly via the server-issued cursor. A missing nextCursor or a
// cursor identical to the one we just sent means the end (or a server that
// ignores our pagination param) — stop instead of re-fetching page one.
const next: string | null = data.meta?.nextCursor || null;
if (!next || next === cursor) break;
cursor = next;
// Reaching the ceiling with a still-advancing cursor means truncation.
if (page === MAX_PAGES - 1) truncated = true;
}
if (truncated) {
console.warn(
`listComments: comments for "${pageId}" truncated at the ${MAX_PAGES}-page cap; more pages exist on the server`,
);
}
const mapped = allComments.map((comment: any) => {
const markdown = comment.content
@@ -2846,36 +2947,27 @@ export class DocmostClient {
);
}
// 1. Enumerate the FULL set of pages in scope by walking the sidebar-pages
// tree (a complete page index), NOT the bounded "/pages/recent" feed which
// caps at ~5000 recent items and silently misses comments on older pages.
// 1. Enumerate the FULL set of pages in scope via the page tree (a complete
// page index), NOT the bounded "/pages/recent" feed which caps at ~5000
// recent items and silently misses comments on older pages.
//
// Subtree scope: when parentPageId is given, the scope is that page ITSELF
// plus every descendant (enumerateSpacePages walks its children). Otherwise
// the scope is the whole space (all roots and their descendants).
// plus every descendant. Otherwise the scope is the whole space (all roots
// and their descendants).
//
// NOTE: do NOT pre-filter by page.updatedAt — creating a comment does not
// bump it (verified on a live server), so such a filter silently misses
// comments on pages that were not otherwise edited. The complete tree walk
// already restricts the scope correctly, so no recent-feed allow-list is
// needed any more.
let pagesInScope: any[];
if (parentPageId) {
const subtree = await this.enumerateSpacePages(spaceId, parentPageId);
// Include the parent page node itself alongside its descendants. Fetch it
// so its title/id are available even though it is not returned by its own
// children listing.
let parentNode: any = { id: parentPageId };
try {
parentNode = await this.getPageRaw(parentPageId);
} catch (e: any) {
// Fall back to a minimal node if the parent can't be fetched; its
// comments are still attempted below (the fetch there is non-fatal).
}
pagesInScope = [parentNode, ...subtree];
} else {
pagesInScope = await this.enumerateSpacePages(spaceId);
}
//
// The subtree scope (parentPageId given) already INCLUDES the root node
// itself: /pages/tree seeds getPageAndDescendants with id = parentPageId, so
// no separate getPageRaw fetch for the parent is needed.
const { pages: pagesInScope, truncated } = await this.enumerateSpacePages(
spaceId,
parentPageId,
);
// 2. Fetch comments for each page, keep ones created after since
const results: any[] = [];
@@ -2904,10 +2996,9 @@ export class DocmostClient {
0,
);
// enumerateSpacePages caps traversal at 10000 nodes; flag when that cap was
// hit so the caller knows the scan may be incomplete (some pages skipped).
const truncated = pagesInScope.length >= 10000;
// `truncated` is reported by enumerateSpacePages: it is true ONLY when the
// stdio fallback BFS hit its node cap. The primary /pages/tree path is
// uncapped, so a space with legitimately many pages is not falsely flagged.
return {
since,
scope: parentPageId ? `subtree of ${parentPageId}` : `space ${spaceId}`,
+8
View File
@@ -29,6 +29,14 @@ export { destroyAllSessions } from "./lib/collab-session.js";
export { SHARED_TOOL_SPECS } from "./tool-specs.js";
export type { SharedToolSpec } from "./tool-specs.js";
// Re-export the build-time REGISTRY_STAMP (issue #447): a deterministic hash of
// the tool-specs registry content, generated into src/registry-stamp.generated.ts
// by scripts/gen-registry-stamp.mjs BEFORE tsc, so it lands in build/. The in-app
// loader recomputes the same hash from src/tool-specs.ts (dev/test only) and
// refuses to run on a mismatch, catching a build/ vs src/ skew (a spec edited in
// src without rebuilding the package the server actually loads from build/).
export { REGISTRY_STAMP } from "./registry-stamp.generated.js";
// Re-export the shared "new comments: N" signal helper (#417) so the in-app
// layer reads the SAME watermark/debounce/injection-safe line builder off the
// loaded module (same pattern as SHARED_TOOL_SPECS). Both surfaces then differ
@@ -0,0 +1,440 @@
// Mock-HTTP tests for the cursor-pagination migration in DocmostClient (#442).
//
// The server switched its list endpoints from OFFSET (`page`) to CURSOR
// (`cursor`/`nextCursor`) pagination, and the global ValidationPipe silently
// strips the obsolete `page` field — so the old offset loops re-fetched page
// one forever (hasNextPage stuck true), dropping every item past the first
// page. These tests pin the new cursor behaviour and the immovable-cursor
// guard that prevents a silent spin/duplication if the protocol drifts again.
//
// A local http.createServer stands in for Docmost so everything stays
// deterministic and offline (same harness style as reauth.test.mjs).
import { test, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { DocmostClient } from "../../build/client.js";
function readBody(req) {
return new Promise((resolve) => {
let raw = "";
req.on("data", (chunk) => {
raw += chunk;
});
req.on("end", () => resolve(raw));
});
}
function startServer(handler) {
return new Promise((resolve) => {
const server = http.createServer(handler);
server.listen(0, "127.0.0.1", () => {
const { port } = server.address();
resolve({ server, baseURL: `http://127.0.0.1:${port}/api` });
});
});
}
function closeServer(server) {
return new Promise((resolve) => server.close(resolve));
}
function sendJson(res, status, obj, extraHeaders = {}) {
res.writeHead(status, { "Content-Type": "application/json", ...extraHeaders });
res.end(JSON.stringify(obj));
}
const openServers = [];
async function spawn(handler) {
const { server, baseURL } = await startServer(handler);
openServers.push(server);
return { server, baseURL };
}
after(async () => {
await Promise.all(openServers.map((s) => closeServer(s)));
});
// A login handler shared by every server below.
function handleLogin(req, res) {
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
});
return true;
}
return false;
}
// -----------------------------------------------------------------------------
// 1) listSidebarPages: collects every cursor page; #requests == #pages.
// -----------------------------------------------------------------------------
test("listSidebarPages walks all cursor pages and collects every item", async () => {
// Three pages keyed by the cursor the client sends back.
const PAGES = {
"": { items: [{ id: "a" }, { id: "b" }], nextCursor: "c1" },
c1: { items: [{ id: "c" }, { id: "d" }], nextCursor: "c2" },
c2: { items: [{ id: "e" }], nextCursor: null },
};
let requests = 0;
const sentLimits = [];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/sidebar-pages") {
requests++;
const body = JSON.parse(raw || "{}");
sentLimits.push(body.limit);
const page = PAGES[body.cursor ?? ""] ?? { items: [], nextCursor: null };
sendJson(res, 200, {
success: true,
data: {
items: page.items,
meta: {
hasNextPage: page.nextCursor != null,
nextCursor: page.nextCursor,
},
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.listSidebarPages("space-1");
assert.equal(requests, 3, "one request per cursor page");
assert.deepEqual(
all.map((p) => p.id),
["a", "b", "c", "d", "e"],
"all items across all pages collected in order",
);
assert.ok(
sentLimits.every((l) => l === 100),
"requests limit:100 (server-side max)",
);
});
// -----------------------------------------------------------------------------
// 2) REGRESSION on the bug class: server IGNORES the cursor param and always
// returns page one with hasNextPage:true -> the immovable-cursor guard must
// terminate the loop with no duplicates, NOT spin to MAX_PAGES.
// -----------------------------------------------------------------------------
test("listSidebarPages terminates (no dups) when the server ignores the cursor", async () => {
let requests = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/sidebar-pages") {
requests++;
// Always the SAME first page with hasNextPage:true and the SAME cursor,
// exactly as a server that no longer understands our pagination param.
sendJson(res, 200, {
success: true,
data: {
items: [{ id: "x1" }, { id: "x2" }],
meta: { hasNextPage: true, nextCursor: "stuck" },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.listSidebarPages("space-1");
// Request 1 (no cursor) gets "stuck"; request 2 (cursor "stuck") gets "stuck"
// again -> guard trips. Far below the MAX_PAGES=50 ceiling; no runaway dups.
assert.equal(requests, 2, "stops as soon as the cursor stops moving");
assert.equal(all.length, 4, "no runaway accumulation / duplication");
});
// -----------------------------------------------------------------------------
// 3a) enumerateSpacePages happy path: a SINGLE /pages/tree request.
// -----------------------------------------------------------------------------
test("enumerateSpacePages (via list_pages tree) uses one /pages/tree request", async () => {
let treeRequests = 0;
let sidebarRequests = 0;
let treeBody = null;
const NODES = [
{ id: "r1", slugId: "r1s", title: "Root 1", parentPageId: null, hasChildren: true, spaceId: "space-1", position: "a", icon: null, canEdit: true },
{ id: "c1", slugId: "c1s", title: "Child 1", parentPageId: "r1", hasChildren: false, spaceId: "space-1", position: "a", icon: null, canEdit: true },
{ id: "r2", slugId: "r2s", title: "Root 2", parentPageId: null, hasChildren: false, spaceId: "space-1", position: "b", icon: null, canEdit: true },
];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
treeRequests++;
treeBody = JSON.parse(raw || "{}");
sendJson(res, 200, { success: true, data: { items: NODES } });
return;
}
if (req.url === "/api/pages/sidebar-pages") {
sidebarRequests++;
sendJson(res, 200, { success: true, data: { items: [], meta: {} } });
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// list_pages tree:true -> enumerateSpacePages(spaceId) -> buildPageTree.
const tree = await client.listPages("space-1", 50, true);
assert.equal(treeRequests, 1, "exactly one /pages/tree request for the space");
assert.equal(sidebarRequests, 0, "no per-node sidebar BFS requests");
assert.deepEqual(treeBody, { spaceId: "space-1" }, "space scope posts spaceId only");
// buildPageTree nests c1 under r1; two roots at the top level.
assert.equal(tree.length, 2, "two root nodes");
const r1 = tree.find((n) => n.id === "r1");
assert.equal(r1.children.length, 1, "child nested under its root");
assert.equal(r1.children[0].id, "c1");
});
// -----------------------------------------------------------------------------
// 3b) enumerateSpacePages fallback: /pages/tree 404 -> cursor BFS via sidebar.
// -----------------------------------------------------------------------------
test("enumerateSpacePages falls back to the cursor BFS on /pages/tree 404", async () => {
let treeRequests = 0;
const sidebarCalls = [];
// Root level: one root with children. Child level (pageId=r1): one leaf.
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
treeRequests++;
// Stock upstream Docmost has no /pages/tree.
sendJson(res, 404, { message: "Not Found" });
return;
}
if (req.url === "/api/pages/sidebar-pages") {
const body = JSON.parse(raw || "{}");
sidebarCalls.push(body.pageId ?? "<root>");
if (!body.pageId) {
sendJson(res, 200, {
success: true,
data: {
items: [
{ id: "r1", title: "Root", parentPageId: null, hasChildren: true },
],
meta: { hasNextPage: false, nextCursor: null },
},
});
} else if (body.pageId === "r1") {
sendJson(res, 200, {
success: true,
data: {
items: [
{ id: "c1", title: "Leaf", parentPageId: "r1", hasChildren: false },
],
meta: { hasNextPage: false, nextCursor: null },
},
});
} else {
sendJson(res, 200, {
success: true,
data: { items: [], meta: { hasNextPage: false, nextCursor: null } },
});
}
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const tree = await client.listPages("space-1", 50, true);
assert.ok(treeRequests >= 1, "the tree endpoint was attempted first");
assert.deepEqual(
sidebarCalls,
["<root>", "r1"],
"fell back to the sidebar BFS: roots then the root's children",
);
assert.equal(tree.length, 1, "one root in the built tree");
assert.equal(tree[0].children[0].id, "c1", "leaf nested via the BFS");
});
// -----------------------------------------------------------------------------
// 3c) enumerateSpacePages fallback SUBTREE: /pages/tree 404 + a rootPageId ->
// the ROOT page itself must be seeded (via getPageRaw) so its own comments
// aren't dropped. listSidebarPages(spaceId, root) returns only the root's
// CHILDREN, so without the seed the root would be absent. (Finding 1.)
// -----------------------------------------------------------------------------
test("enumerateSpacePages fallback subtree seeds the ROOT page itself", async () => {
const sidebarCalls = [];
let infoRequests = 0;
const commentedPages = [];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
// Stock upstream Docmost -> fall back to the BFS.
sendJson(res, 404, { message: "Not Found" });
return;
}
if (req.url === "/api/pages/info") {
// getPageRaw for the root seed. Shape mirrors a real page-info response.
infoRequests++;
sendJson(res, 200, {
success: true,
data: { id: "root", title: "Root", spaceId: "space-1", hasChildren: true },
});
return;
}
if (req.url === "/api/pages/sidebar-pages") {
const body = JSON.parse(raw || "{}");
sidebarCalls.push(body.pageId ?? "<root>");
// Children of the root: one leaf. (Root itself is NOT in this list.)
const items =
body.pageId === "root"
? [{ id: "leaf", title: "Leaf", parentPageId: "root", hasChildren: false }]
: [];
sendJson(res, 200, {
success: true,
data: { items, meta: { hasNextPage: false, nextCursor: null } },
});
return;
}
if (req.url === "/api/comments") {
const body = JSON.parse(raw || "{}");
commentedPages.push(body.pageId);
const items =
body.pageId === "root"
? [{ id: "cm1", createdAt: "2030-01-01T00:00:00.000Z", content: null }]
: [];
sendJson(res, 200, {
success: true,
data: { items, meta: { nextCursor: null } },
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
// checkNewComments(space, since, parentPageId) exercises the subtree fallback.
const result = await client.checkNewComments(
"space-1",
"2020-01-01T00:00:00.000Z",
"root",
);
assert.equal(infoRequests, 1, "root was seeded via one getPageRaw");
assert.equal(sidebarCalls[0], "root", "BFS walked the root's children");
assert.ok(
commentedPages.includes("root"),
"the ROOT page is in scope (its comments were fetched) — not dropped",
);
assert.ok(commentedPages.includes("leaf"), "the descendant is in scope too");
assert.equal(result.checkedPages, 2, "root + one descendant scanned");
assert.equal(result.totalNewComments, 1, "the root's fresh comment found");
});
// -----------------------------------------------------------------------------
// 5) listComments immovable-cursor guard: the server IGNORES the cursor and
// keeps returning the same nextCursor -> the loop must terminate (no
// infinite loop, no duplicates), not spin forever. (Finding 4.)
// -----------------------------------------------------------------------------
test("listComments terminates (no dups) when the server ignores the cursor", async () => {
let requests = 0;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/comments") {
requests++;
// Always the SAME page with the SAME nextCursor, as a server that no
// longer advances the cursor would.
sendJson(res, 200, {
success: true,
data: {
items: [{ id: "cm1", createdAt: "2030-01-01T00:00:00.000Z", content: null }],
meta: { nextCursor: "stuck" },
},
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const { items } = await client.listComments("page-1", true);
// Request 1 (no cursor) gets "stuck"; request 2 (cursor "stuck") gets "stuck"
// again -> guard trips. Bounded far below MAX_PAGES=50, no runaway dups.
assert.equal(requests, 2, "stops as soon as the cursor stops moving");
assert.equal(items.length, 2, "no runaway accumulation / duplication");
});
// -----------------------------------------------------------------------------
// 4) check_new_comments subtree: the root is included in scope WITHOUT a
// separate getPageRaw (/pages/info) request for the parent.
// -----------------------------------------------------------------------------
test("checkNewComments subtree includes the root without a separate getPageRaw", async () => {
let pageInfoRequests = 0;
let treeBody = null;
const commentedPages = [];
// /pages/tree (subtree) returns the parent itself plus a descendant, exactly
// as getPageAndDescendants seeds with id = parentPageId.
const NODES = [
{ id: "parent", title: "Parent", parentPageId: null, hasChildren: true },
{ id: "kid", title: "Kid", parentPageId: "parent", hasChildren: false },
];
const { baseURL } = await spawn(async (req, res) => {
const raw = await readBody(req);
if (handleLogin(req, res)) return;
if (req.url === "/api/pages/tree") {
treeBody = JSON.parse(raw || "{}");
sendJson(res, 200, { success: true, data: { items: NODES } });
return;
}
if (req.url === "/api/pages/info") {
// If checkNewComments still fetched the parent separately this would fire.
pageInfoRequests++;
sendJson(res, 200, { success: true, data: { id: "parent" } });
return;
}
if (req.url === "/api/comments") {
const body = JSON.parse(raw || "{}");
commentedPages.push(body.pageId);
// One fresh comment on the parent, none elsewhere.
const items =
body.pageId === "parent"
? [{ id: "cm1", createdAt: "2030-01-01T00:00:00.000Z", content: null }]
: [];
sendJson(res, 200, {
success: true,
data: { items, meta: { nextCursor: null } },
});
return;
}
sendJson(res, 404, {});
});
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const result = await client.checkNewComments(
"space-1",
"2020-01-01T00:00:00.000Z",
"parent",
);
assert.equal(pageInfoRequests, 0, "no separate getPageRaw for the root");
assert.deepEqual(treeBody, { pageId: "parent" }, "subtree scope posts pageId");
assert.ok(
commentedPages.includes("parent"),
"the root itself is in scope (comments fetched for it)",
);
assert.ok(commentedPages.includes("kid"), "descendants are in scope too");
assert.equal(result.checkedPages, 2, "root + one descendant scanned");
assert.equal(result.totalNewComments, 1, "the root's fresh comment found");
});
+26 -19
View File
@@ -297,12 +297,12 @@ test("a response with ONLY authTokenRefresh (no authToken) rejects login", async
// -----------------------------------------------------------------------------
// 5) paginateAll loop guards.
// -----------------------------------------------------------------------------
test("paginateAll stops at the MAX_PAGES cap when hasNextPage is always true", async () => {
test("paginateAll stops at the MAX_PAGES cap when the server always issues a fresh cursor", async () => {
let pageRequests = 0;
const LIMIT = 100;
const { baseURL } = await spawn(async (req, res) => {
await readBody(req);
const body = JSON.parse((await readBody(req)) || "{}");
if (req.url === "/api/auth/login") {
sendJson(res, 200, { success: true }, {
"Set-Cookie": "authToken=t; Path=/; HttpOnly",
@@ -311,15 +311,18 @@ test("paginateAll stops at the MAX_PAGES cap when hasNextPage is always true", a
}
if (req.url === "/api/spaces") {
pageRequests++;
// Always return a FULL page (== requested limit) AND hasNextPage:true.
// Both the page-length check and the hasNextPage flag say "keep going",
// so only the MAX_PAGES ceiling can stop the loop.
// Always return a FULL page AND hasNextPage:true with a FRESH nextCursor
// that differs from the one the client just sent, so the immovable-cursor
// guard never trips — only the MAX_PAGES ceiling can stop the loop.
const items = Array.from({ length: LIMIT }, (_, i) => ({
id: `s-${pageRequests}-${i}`,
}));
sendJson(res, 200, {
success: true,
data: { items, meta: { hasNextPage: true } },
data: {
items,
meta: { hasNextPage: true, nextCursor: `cursor-${pageRequests}` },
},
});
return;
}
@@ -338,7 +341,7 @@ test("paginateAll stops at the MAX_PAGES cap when hasNextPage is always true", a
assert.equal(all.length, 50 * LIMIT, "accumulates one full page per request");
});
test("paginateAll stops early on a short page even if hasNextPage is true", async () => {
test("paginateAll stops on the immovable-cursor guard when the server ignores the cursor param", async () => {
let pageRequests = 0;
const LIMIT = 100;
@@ -352,15 +355,17 @@ test("paginateAll stops early on a short page even if hasNextPage is true", asyn
}
if (req.url === "/api/spaces") {
pageRequests++;
// First page is full; second page is SHORT (fewer than limit). The short
// page must stop the loop immediately even though hasNextPage stays true.
const count = pageRequests === 1 ? LIMIT : 3;
const items = Array.from({ length: count }, (_, i) => ({
id: `s-${pageRequests}-${i}`,
}));
// The bug class: the server IGNORES the pagination param and keeps
// returning page one with hasNextPage:true and the SAME nextCursor. The
// immovable-cursor guard must stop the loop instead of spinning to
// MAX_PAGES and duplicating items.
const items = Array.from({ length: LIMIT }, (_, i) => ({ id: `s-${i}` }));
sendJson(res, 200, {
success: true,
data: { items, meta: { hasNextPage: true } },
data: {
items,
meta: { hasNextPage: true, nextCursor: "stuck" },
},
});
return;
}
@@ -370,8 +375,10 @@ test("paginateAll stops early on a short page even if hasNextPage is true", asyn
const client = new DocmostClient(baseURL, "user@example.com", "pw");
const all = await client.paginateAll("/spaces", {}, LIMIT);
assert.equal(pageRequests, 2, "stops right after the first short page");
assert.equal(all.length, LIMIT + 3, "full page + short page accumulated");
// Request 1 sends no cursor and receives "stuck"; request 2 sends "stuck" and
// receives "stuck" again -> guard trips after exactly two requests, no dups.
assert.equal(pageRequests, 2, "stops once the cursor stops moving");
assert.equal(all.length, 2 * LIMIT, "no runaway accumulation past the guard");
});
test("paginateAll handles both {data:{items,meta}} and {items,meta} envelopes", async () => {
@@ -387,16 +394,16 @@ test("paginateAll handles both {data:{items,meta}} and {items,meta} envelopes",
}
if (req.url === "/api/groups") {
bareRequests.push(1);
// Page 1: full page, hasNextPage true. Page 2: short page -> stop.
// Page 1: hasNextPage true with a next cursor. Page 2: no next -> stop.
if (bareRequests.length === 1) {
sendJson(res, 200, {
items: Array.from({ length: 100 }, (_, i) => ({ id: `g${i}` })),
meta: { hasNextPage: true },
meta: { hasNextPage: true, nextCursor: "c2" },
});
} else {
sendJson(res, 200, {
items: [{ id: "tail" }],
meta: { hasNextPage: false },
meta: { hasNextPage: false, nextCursor: null },
});
}
return;
@@ -0,0 +1,101 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
import { computeRegistryStamp } from "../../scripts/gen-registry-stamp.mjs";
import { REGISTRY_STAMP } from "../../build/index.js";
// Guard tests for the build/src-skew stamp (issue #447). The codegen script
// exports `computeRegistryStamp(sourceText)` — a sha256 over normalized source
// text (CRLF->LF, single trailing newline stripped). The in-app loader
// (apps/server/.../docmost-client.loader.ts) DUPLICATES that normalize+sha256 to
// recompute the stamp from src and refuse a stale build. These tests pin the
// algorithm's behaviour AND assert the built stamp matches the current src, so a
// stale generated file OR a normalize divergence reddens.
const __dirname = dirname(fileURLToPath(import.meta.url));
const TOOL_SPECS_PATH = join(__dirname, "..", "..", "src", "tool-specs.ts");
test("computeRegistryStamp is deterministic: same input -> same hash", () => {
const input = "export const X = 1;\nexport const Y = 2;\n";
assert.equal(computeRegistryStamp(input), computeRegistryStamp(input));
});
test("computeRegistryStamp returns a 64-char lowercase hex sha256", () => {
const stamp = computeRegistryStamp("anything");
assert.match(stamp, /^[0-9a-f]{64}$/);
});
test("normalizes CRLF vs LF: the same content hashes equal", () => {
const lf = "line1\nline2\nline3";
const crlf = "line1\r\nline2\r\nline3";
assert.equal(computeRegistryStamp(crlf), computeRegistryStamp(lf));
});
test("normalizes a trailing newline: with/without a final \\n hashes equal", () => {
const noTrailing = "alpha\nbeta";
const trailing = "alpha\nbeta\n";
assert.equal(computeRegistryStamp(trailing), computeRegistryStamp(noTrailing));
});
test("a CRLF checkout WITH a trailing CRLF still hashes equal to bare LF", () => {
// A worst-case Windows checkout: CRLF line endings + a trailing CRLF. Both the
// \r\n->\n replace and the trailing-newline strip must apply for parity.
const bare = "alpha\nbeta";
const crlfTrailing = "alpha\r\nbeta\r\n";
assert.equal(
computeRegistryStamp(crlfTrailing),
computeRegistryStamp(bare),
);
});
test("a real content change hashes differently", () => {
const before = "export const description = 'search a page';\n";
const after = "export const description = 'search a PAGE';\n";
assert.notEqual(computeRegistryStamp(before), computeRegistryStamp(after));
});
// Only a SINGLE trailing newline is stripped — a second blank line is content and
// must change the hash. This pins the exact `/\n$/` semantics the loader mirrors.
test("only ONE trailing newline is stripped (two differ from one)", () => {
assert.notEqual(
computeRegistryStamp("x\n"),
computeRegistryStamp("x\n\n"),
);
});
// Cross-impl equality against a fixed, documented input. The SAME literal input
// and expected hash are asserted in the server-side jest test
// (docmost-client.loader.spec.ts). If either side's normalize+sha256 ever
// diverges, one of the two tests reddens. Input exercises BOTH normalize steps.
test("fixed-input hash matches the documented cross-impl value", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const EXPECTED =
"683376e290829b482c2655745caffa7a1dccfa10afaa62dac2b42dd6c68d0f83";
assert.equal(computeRegistryStamp(FIXED_INPUT), EXPECTED);
});
// DESYNC GUARD (covers reviewer suggestion 2). Recompute the stamp from the
// actual src/tool-specs.ts and assert it equals the REGISTRY_STAMP baked into the
// freshly-built build/index.js. This reddens if the generated file is stale OR if
// the codegen normalize ever diverges from what produced the built stamp.
test("built REGISTRY_STAMP equals the stamp recomputed from src/tool-specs.ts", () => {
const source = readFileSync(TOOL_SPECS_PATH, "utf8");
assert.equal(computeRegistryStamp(source), REGISTRY_STAMP);
});
// Sanity: the fixed-input helper computes the SAME way the codegen does, proving
// the EXPECTED constant above is not an arbitrary magic value but the documented
// normalize+sha256 of FIXED_INPUT. Belt-and-braces so a bad EXPECTED can't hide a
// real regression.
test("the documented EXPECTED constant is the normalize+sha256 of FIXED_INPUT", () => {
const FIXED_INPUT = "line1\r\nline2\n";
const normalized = FIXED_INPUT.replace(/\r\n/g, "\n").replace(/\n$/, "");
const expected = createHash("sha256")
.update(normalized, "utf8")
.digest("hex");
assert.equal(computeRegistryStamp(FIXED_INPUT), expected);
});