Compare commits
36 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a6ff7623db | |||
| 3085ec1b50 | |||
| 05ec9feaf9 | |||
| 8e12579925 | |||
| 20703d06c2 | |||
| dab2660999 | |||
| 751d55e9db | |||
| 02308012a6 | |||
| 0665fcb630 | |||
| 97bd554cb5 | |||
| f77a6b42de | |||
| 134b627806 | |||
| 3267512ed9 | |||
| 48bd27b83c | |||
| 265b81c93d | |||
| ed808876be | |||
| a72ddbbe86 | |||
| d8fc724d90 | |||
| e4bfbcabaa | |||
| 4c1ee50dc9 | |||
| b8cce4f814 | |||
| c5bff2d84a | |||
| a325ddbabd | |||
| 80fc30633b | |||
| e17d5bc060 | |||
| bfcee6dddc | |||
| 2c2d60a5dc | |||
| 1417209915 | |||
| f555fc87da | |||
| d6d1195abd | |||
| 0050ad7ebb | |||
| 43b11d92ab | |||
| 7b4617db70 | |||
| 9b4b38a611 | |||
| f759084f41 | |||
| 459d636ffb |
@@ -151,6 +151,12 @@ jobs:
|
||||
- name: Build editor-ext
|
||||
run: pnpm --filter @docmost/editor-ext build
|
||||
|
||||
# @docmost/prosemirror-markdown is an ESM workspace package the server
|
||||
# imports at runtime; its build/ is gitignored and test:e2e has no pretest
|
||||
# hook, so build it before the e2e run (mirrors the test.yml job).
|
||||
- name: Build prosemirror-markdown
|
||||
run: pnpm --filter @docmost/prosemirror-markdown build
|
||||
|
||||
- name: Run migrations
|
||||
run: pnpm --filter ./apps/server migration:latest
|
||||
|
||||
|
||||
@@ -13,6 +13,49 @@ permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
# Guard against a long-lived branch adding a migration whose timestamped
|
||||
# filename sorts BEFORE migrations already applied on the target branch (and
|
||||
# thus in prod). The Kysely startup migrator rejects that as "corrupted
|
||||
# migrations" and crash-loops the app on boot (incident #361). This gate fails
|
||||
# the PR so the migration is renamed to a current timestamp before merge. Only
|
||||
# runs for pull_request events (needs a base branch to diff against).
|
||||
migration-order:
|
||||
if: github.event_name == 'pull_request'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Checkout (full history for the base-branch diff)
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: Added migrations must sort after the newest on the base branch
|
||||
env:
|
||||
TARGET_BRANCH: ${{ github.base_ref }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
MIG_DIR="apps/server/src/database/migrations"
|
||||
# checkout above already did fetch-depth:0 (full history). Fetch the base
|
||||
# WITHOUT --depth (a shallow graft would truncate the base history and
|
||||
# break the merge-base when the base has moved ahead of the PR merge —
|
||||
# exactly the long-branch-vs-moving-base case this gate guards, #361).
|
||||
git fetch --no-tags origin "$TARGET_BRANCH"
|
||||
newest_on_target=$(git ls-tree -r --name-only "origin/${TARGET_BRANCH}" "$MIG_DIR" | sort | tail -1)
|
||||
# NO `|| true`: a diff failure (e.g. an unresolved merge-base) must fail
|
||||
# the job CLOSED — a gate whose job is to BLOCK must never pass on error.
|
||||
# `set -e` above already aborts on a non-zero diff exit.
|
||||
added=$(git diff --diff-filter=A --name-only "origin/${TARGET_BRANCH}...HEAD" -- "$MIG_DIR")
|
||||
bad=0
|
||||
for f in $added; do
|
||||
if [[ "$f" < "$newest_on_target" || "$f" == "$newest_on_target" ]]; then
|
||||
echo "::error::Migration $f sorts at or before the newest on ${TARGET_BRANCH} ($newest_on_target) — rename it with a CURRENT timestamp before merge (do not change its contents). See incident #361."
|
||||
bad=1
|
||||
fi
|
||||
done
|
||||
if [ "$bad" -eq 0 ]; then
|
||||
echo "Migration order OK (added migrations all sort after $newest_on_target)."
|
||||
fi
|
||||
exit $bad
|
||||
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
|
||||
@@ -201,7 +201,7 @@ pnpm workspace (`pnpm@10.4.0`) orchestrated by **Nx**. Four workspace packages:
|
||||
| `apps/client` | `client` | React 18 + Vite + Mantine 8 + TanStack Query + Jotai | SPA frontend |
|
||||
| `packages/editor-ext` | `@docmost/editor-ext` | Tiptap/ProseMirror | Shared Tiptap node/mark extensions, imported by both the client and the server |
|
||||
| `packages/mcp` | `@docmost/mcp` | MCP SDK, Tiptap, Yjs | Standalone MCP server, also bundled into the server at `/mcp`. Consumes the shared converter/schema from `@docmost/prosemirror-markdown` (#293) — it no longer carries its own vendored converter/schema copy |
|
||||
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp` and `git-sync`; there is exactly ONE copy of the converter now |
|
||||
| `packages/prosemirror-markdown` | `@docmost/prosemirror-markdown` | Tiptap, marked, jsdom | The single, canonical ProseMirror↔Markdown converter + Docmost schema mirror (#293). Consumed by `mcp`, `git-sync`, AND `apps/server` (server-side markdown import/export, #345); there is exactly ONE copy of the converter now |
|
||||
|
||||
`build` targets are Nx-cached and dependency-ordered (`dependsOn: ["^build"]`), so `editor-ext` builds before the apps. `nx.json` sets `affected.defaultBase: main`.
|
||||
|
||||
@@ -214,6 +214,12 @@ Run from the repo root unless noted. The dev workflow needs **Postgres (with the
|
||||
> server, `APP_SECRET` mismatch between processes, a stale `editor-ext` white-
|
||||
> screening the client, LAN exposure. See **[docs/dev-stand.md](docs/dev-stand.md)**
|
||||
> for the step-by-step and the traps.
|
||||
>
|
||||
> **Testing the app against a stand** (browser E2E + out-of-band verification) has
|
||||
> its own non-obvious traps — the page has two ProseMirror editors (only the body is
|
||||
> collab-bound), a ~10s store debounce, and API-seeding the thing under test is a
|
||||
> silent no-test. See **[docs/how-to-test.md](docs/how-to-test.md)** before writing
|
||||
> UI tests.
|
||||
|
||||
```bash
|
||||
pnpm install # install all workspaces (uses pnpm patches; see package.json `pnpm.patchedDependencies`)
|
||||
@@ -250,7 +256,10 @@ pnpm --filter server migration:codegen # regenerate src/databa
|
||||
```
|
||||
Migration files live in `apps/server/src/database/migrations/` and are named `YYYYMMDDThhmmss-description.ts`. Fork-specific migrations only **add** tables (`page_embeddings`, `ai_chats`, `ai_chat_messages`, `ai_provider_credentials`, `ai_mcp_servers`, `page_template_references`) and columns (e.g. `pages.is_template`, a `NOT NULL DEFAULT false` boolean) — never drop/rewrite Docmost data.
|
||||
|
||||
**Migration ordering — always check when merging branches/features.** Kysely runs migrations in **alphabetical (= timestamp) order** and refuses to start if a *new* migration sorts **before** one already applied to the DB (`corrupted migrations: ... must always have a name that comes alphabetically after the last executed migration`). When you merge a branch or land a feature, verify your migration's timestamp still sorts **after every migration that may already be applied on the target** (`/bin/ls -1 apps/server/src/database/migrations | sort | tail`). Branches developed in parallel routinely break this: a feature branch adds `…T130000-…`, `main` meanwhile ships and deploys `…T150000-…`, and after the merge the older-timestamped file is rejected at boot. **Fix = rename your migration to a timestamp after the latest one already in the target** (content unchanged — the filename is the ordering key), then rebuild so the compiled `dist/database/migrations/` picks up the new name.
|
||||
**Migration ordering — always check when merging branches/features.** Kysely runs migrations in **alphabetical (= timestamp) order**. A *new* migration that sorts **before** one already applied to the DB is a "back-dated" migration, which branches developed in parallel routinely produce: a feature branch adds `…T130000-…`, `develop` meanwhile ships and deploys `…T150000-…`, and after the merge the older-timestamped file has been skipped. Two layers guard this (both added for incident #361, where a back-dated migration crash-looped prod for ~11 min):
|
||||
|
||||
- **CI gate (primary):** the `migration-order` job in `.github/workflows/test.yml` fails a PR whose added migration sorts at/before the newest on the base branch. **So the fix is to rename your migration to a timestamp after the latest one already in the target** (`/bin/ls -1 apps/server/src/database/migrations | sort | tail`; content unchanged — the filename is the ordering key), then rebuild so the compiled `dist/database/migrations/` picks up the new name.
|
||||
- **Runtime safety net:** both Migrators (`migration.service.ts` startup auto-migrate + `migrate.ts` CLI) set `allowUnorderedMigrations: true`, so the app does **not** refuse to start on an out-of-order migration — it applies the skipped older one instead of crash-looping. Kysely's `#ensureNoMissingMigrations` guard is still on (a *removed* applied migration is still an error). Because apply order can then differ from lexicographic across instances, migrations must stay **independent** (each creates its own objects) — the CI gate remains the primary line; this net only covers a gate bypass (manual push / hotfix branch).
|
||||
|
||||
## Architecture — the big picture
|
||||
|
||||
@@ -284,7 +293,7 @@ The API server is a Fastify app with a global `/api` prefix (`main.ts` excludes
|
||||
### Client structure
|
||||
Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirrors the server domains: `page`, `space`, `comment`, `ai-chat`, `editor`, …). Conventions:
|
||||
- **TanStack Query** for server state (one `queries/` file per feature), **Jotai** atoms for local/shared UI state, **Mantine 8** + CSS modules (`*.module.css`) + `postcss-preset-mantine` for UI.
|
||||
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, import/export) — editor schema changes often need to be made in `editor-ext`, not just the client. The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by both `mcp` and `git-sync` — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
|
||||
- The editor is Tiptap; shared node/mark extensions live in `packages/editor-ext` and are imported by **both the client and the server** (collaboration, schema, `canonicalizeFootnotes`) — editor schema changes often need to be made in `editor-ext`, not just the client. Server-side markdown import/export no longer lives in `editor-ext`: it goes through the canonical converter (#345, see below). The ProseMirror↔Markdown converter and its Docmost schema mirror now live in a SINGLE package, `@docmost/prosemirror-markdown` (#293), consumed by `mcp`, `git-sync`, and `apps/server` (#345) — do NOT reintroduce a per-package copy. `editor-ext` is the upstream source of the Tiptap schema; the package's `docmost-schema.ts` mirrors it and a serializer-contract test (`packages/prosemirror-markdown/test/serializer-contract.test.ts`) guards the boundary (every schema node must have a converter case), so a drift surfaces as a failing test rather than silent divergence.
|
||||
- API access goes through `apps/client/src/lib/api-client.ts` (axios). The `@` alias maps to `apps/client/src`.
|
||||
- Runtime config is injected at build time by `vite.config.ts` via `define` (`APP_URL`, `COLLAB_URL`, `APP_VERSION`, …) — these come from the root `.env`, not from `import.meta.env`.
|
||||
|
||||
@@ -294,7 +303,7 @@ Vite SPA. Code is organized by feature under `apps/client/src/features/*` (mirro
|
||||
- **Errors must never be swallowed or shown as generic messages.** Every caught error MUST (1) be logged in full to the console/logger — error name, message, stack, `cause`, and (for HTTP/provider failures) the status code and response body — and (2) be surfaced to the user with a *specific, human-readable explanation of what actually went wrong*, never a bare generic string like "Something went wrong" / "Could not start recording" / "Transcription failed". Include the real reason (the underlying error/provider message) in the user-facing text. On the server, wrap third-party/provider failures with `describeProviderError` (or equivalent) and rethrow as a meaningful HTTP status + message — never let them collapse into an opaque 500. On the client, `console.error(<context>, err)` the raw error AND show the extracted reason (e.g. `err.response?.data?.message`, or the error `name: message`) in the notification.
|
||||
- The version string shown in the UI comes from `APP_VERSION` (CI/Docker) or `git describe --tags --always` (local), resolved in `vite.config.ts` — not from `package.json`.
|
||||
- Server TS config is permissive (`noImplicitAny: false`, `strictNullChecks: false`, `no-explicit-any` lint disabled). Follow the existing relaxed style rather than tightening types broadly.
|
||||
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons.
|
||||
- Dependency versions are heavily pinned via `pnpm.overrides` and `pnpm.patchedDependencies` (`scimmy`, `yjs`, `ai`) in the root `package.json`. Don't bump pinned/patched deps casually; the patches and overrides exist for compatibility/security reasons. The `ai@6.0.134` patch disables the SDK's O(n²) cumulative `partialOutput` accumulation when no output strategy is requested (server heap OOM on long agent runs, #184; tripwire test: `apps/server/src/integrations/ai/ai-sdk-partial-output.patch.spec.ts`) — it MUST be re-created via `pnpm patch` when bumping `ai`.
|
||||
- **Adding/renaming/removing an MCP tool requires updating `SERVER_INSTRUCTIONS`** in `packages/mcp/src/index.ts` — the intent-routing guide MCP clients receive on initialize. This applies both to inline `server.registerTool(...)` calls in `index.ts` and to specs in `packages/mcp/src/tool-specs.ts`. Enforced by `packages/mcp/test/unit/server-instructions.test.mjs`, which fails when a registered tool is not mentioned in the guide (deliberate opt-outs go into its `EXCEPTIONS` list). `packages/mcp/build/` is gitignored and rebuilt in CI/Docker via `pnpm build` (same convention as `git-sync`/`prosemirror-markdown`) — never commit it; rebuild locally after editing to run the tests.
|
||||
|
||||
## CI / release
|
||||
|
||||
@@ -169,6 +169,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The server no longer runs out of heap during long autonomous agent runs.** A
|
||||
new pnpm patch on `ai@6.0.134` stops the SDK from building a cumulative
|
||||
snapshot of the ENTIRE turn text on every streamed text-delta when no output
|
||||
strategy was requested (our server never requests one). Unpatched, those
|
||||
O(n²) `partialOutput` snapshots piled up in a never-consumed internal
|
||||
`tee()` branch of the stream result — a ~20-step, ~28k-chunk agent run
|
||||
retained ~1.7 GB and OOM'd the 2 GB JS heap. Streaming granularity is
|
||||
unchanged; the patch must be re-created if `ai` is ever bumped. (#184)
|
||||
- **Internal links in exported Markdown no longer lose their visible text.** A
|
||||
link whose target page name had no file extension (e.g. a bare title) was
|
||||
collapsed to empty text during export, producing an unclickable, label-less
|
||||
|
||||
@@ -1373,6 +1373,39 @@
|
||||
"The role catalog is unavailable": "The role catalog is unavailable",
|
||||
"Please try again later.": "Please try again later.",
|
||||
"No bundles available": "No bundles available",
|
||||
"Content": "Content",
|
||||
"Content language of the roles": "Content language of the roles",
|
||||
"{{count}} updates available in {{bundles}} bundles": "{{count}} updates available in {{bundles}} bundles",
|
||||
"Update all ({{count}})": "Update all ({{count}})",
|
||||
"Updating {{current}}/{{total}}…": "Updating {{current}}/{{total}}…",
|
||||
"{{count}} roles are installed in another language. A different language installs separately and appears as new.": "{{count}} roles are installed in another language. A different language installs separately and appears as new.",
|
||||
"{{count}} roles": "{{count}} roles",
|
||||
"{{count}} new — none installed": "{{count}} new — none installed",
|
||||
"All installed · up to date": "All installed · up to date",
|
||||
"{{count}} updates · {{installed}} up to date": "{{count}} updates · {{installed}} up to date",
|
||||
"{{count}} new": "{{count}} new",
|
||||
"{{count}} installed": "{{count}} installed",
|
||||
"{{count}} updates": "{{count}} updates",
|
||||
"Install bundle": "Install bundle",
|
||||
"Install {{count}} selected": "Install {{count}} selected",
|
||||
"Install bundle ({{count}})": "Install bundle ({{count}})",
|
||||
"{{selected}} of {{total}} selected": "{{selected}} of {{total}} selected",
|
||||
"Select all": "Select all",
|
||||
"Deselect all": "Deselect all",
|
||||
"Skipped": "Skipped",
|
||||
"v{{version}}": "v{{version}}",
|
||||
"{{count}} roles installed": "{{count}} roles installed",
|
||||
"{{count}} roles installed · {{renamed}} renamed": "{{count}} roles installed · {{renamed}} renamed",
|
||||
"{{count}} roles updated": "{{count}} roles updated",
|
||||
"Installed {{installed}} · {{skipped}} skipped": "Installed {{installed}} · {{skipped}} skipped",
|
||||
"A role named \"{{name}}\" already exists in this workspace.": "A role named \"{{name}}\" already exists in this workspace.",
|
||||
"\"{{name}}\" is already installed.": "\"{{name}}\" is already installed.",
|
||||
"Rename & install": "Rename & install",
|
||||
"Couldn’t load the catalog": "Couldn’t load the catalog",
|
||||
"Check your connection and try again. Installed roles are not affected.": "Check your connection and try again. Installed roles are not affected.",
|
||||
"Retry": "Retry",
|
||||
"The catalog is empty": "The catalog is empty",
|
||||
"No role bundles are published for this language yet. Try switching the content language.": "No role bundles are published for this language yet. Try switching the content language.",
|
||||
"Already up to date": "Already up to date",
|
||||
"Updated to the latest version": "Updated to the latest version",
|
||||
"This role is no longer in the catalog": "This role is no longer in the catalog",
|
||||
|
||||
@@ -1235,6 +1235,39 @@
|
||||
"The role catalog is unavailable": "Каталог ролей недоступен",
|
||||
"Please try again later.": "Попробуйте позже.",
|
||||
"No bundles available": "Наборы недоступны",
|
||||
"Content": "Язык контента",
|
||||
"Content language of the roles": "Язык контента ролей",
|
||||
"{{count}} updates available in {{bundles}} bundles": "Доступно обновлений: {{count}} в наборах: {{bundles}}",
|
||||
"Update all ({{count}})": "Обновить все ({{count}})",
|
||||
"Updating {{current}}/{{total}}…": "Обновление {{current}}/{{total}}…",
|
||||
"{{count}} roles are installed in another language. A different language installs separately and appears as new.": "Ролей установлено на другом языке: {{count}}. Другой язык устанавливается отдельно и отображается как новый.",
|
||||
"{{count}} roles": "ролей: {{count}}",
|
||||
"{{count}} new — none installed": "новых: {{count}} — ничего не установлено",
|
||||
"All installed · up to date": "Все установлены · актуальны",
|
||||
"{{count}} updates · {{installed}} up to date": "обновлений: {{count}} · актуальны: {{installed}}",
|
||||
"{{count}} new": "новых: {{count}}",
|
||||
"{{count}} installed": "установлено: {{count}}",
|
||||
"{{count}} updates": "обновлений: {{count}}",
|
||||
"Install bundle": "Установить набор",
|
||||
"Install {{count}} selected": "Установить выбранные ({{count}})",
|
||||
"Install bundle ({{count}})": "Установить набор ({{count}})",
|
||||
"{{selected}} of {{total}} selected": "выбрано {{selected}} из {{total}}",
|
||||
"Select all": "Выбрать все",
|
||||
"Deselect all": "Снять выбор",
|
||||
"Skipped": "Пропущено",
|
||||
"v{{version}}": "v{{version}}",
|
||||
"{{count}} roles installed": "Установлено ролей: {{count}}",
|
||||
"{{count}} roles installed · {{renamed}} renamed": "Установлено ролей: {{count}} · переименовано: {{renamed}}",
|
||||
"{{count}} roles updated": "Обновлено ролей: {{count}}",
|
||||
"Installed {{installed}} · {{skipped}} skipped": "Установлено: {{installed}} · пропущено: {{skipped}}",
|
||||
"A role named \"{{name}}\" already exists in this workspace.": "Роль с именем «{{name}}» уже существует в этом рабочем пространстве.",
|
||||
"\"{{name}}\" is already installed.": "«{{name}}» уже установлена.",
|
||||
"Rename & install": "Переименовать и установить",
|
||||
"Couldn’t load the catalog": "Не удалось загрузить каталог",
|
||||
"Check your connection and try again. Installed roles are not affected.": "Проверьте подключение и попробуйте снова. Установленные роли не затронуты.",
|
||||
"Retry": "Повторить",
|
||||
"The catalog is empty": "Каталог пуст",
|
||||
"No role bundles are published for this language yet. Try switching the content language.": "Для этого языка ещё не опубликовано ни одного набора ролей. Попробуйте сменить язык контента.",
|
||||
"No roles configured": "Роли не настроены",
|
||||
"Already up to date": "Уже актуальна",
|
||||
"Updated to the latest version": "Обновлено до последней версии",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import {
|
||||
useInfiniteQuery,
|
||||
useMutation,
|
||||
useQueries,
|
||||
useQuery,
|
||||
useQueryClient,
|
||||
} from "@tanstack/react-query";
|
||||
@@ -307,6 +308,29 @@ export function useAiRoleCatalogBundleQuery(
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Eagerly open EVERY listed bundle's content in parallel for one language. The
|
||||
* redesigned catalog shows each bundle's status summary in its COLLAPSED header,
|
||||
* which needs every role's install state up front — so contents can no longer be
|
||||
* lazy-loaded on expand. The catalog is small, so a fan-out of `useQueries` (one
|
||||
* cached read per bundle, sharing the same cache keys as
|
||||
* `useAiRoleCatalogBundleQuery`) is cheap. Gated by `enabled` (modal open + a
|
||||
* resolved language) so nothing fetches while the modal is closed.
|
||||
*/
|
||||
export function useAiRoleCatalogBundlesQueries(
|
||||
bundleIds: string[],
|
||||
language: string,
|
||||
enabled: boolean,
|
||||
) {
|
||||
return useQueries({
|
||||
queries: bundleIds.map((bundleId) => ({
|
||||
queryKey: AI_ROLE_CATALOG_BUNDLE_RQ_KEY(bundleId, language),
|
||||
queryFn: () => getAiRoleCatalogBundle(bundleId, language),
|
||||
enabled: enabled && !!language,
|
||||
})),
|
||||
});
|
||||
}
|
||||
|
||||
export function useImportAiRolesFromCatalogMutation() {
|
||||
const queryClient = useQueryClient();
|
||||
const { t } = useTranslation();
|
||||
|
||||
@@ -77,7 +77,14 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
|
||||
});
|
||||
|
||||
it("errors:[] -> only the summary notification (counts interpolated)", async () => {
|
||||
await runMutation({ created: 3, renamed: 1, skipped: 2, errors: [] });
|
||||
await runMutation({
|
||||
created: 3,
|
||||
renamed: 1,
|
||||
skipped: 2,
|
||||
errors: [],
|
||||
createdRoles: [],
|
||||
skippedRoles: [],
|
||||
});
|
||||
expect(notificationsShowMock).toHaveBeenCalledTimes(1);
|
||||
expect(notificationsShowMock).toHaveBeenCalledWith({
|
||||
message: "Imported 3, renamed 1, skipped 2",
|
||||
@@ -93,6 +100,8 @@ describe("useImportAiRolesFromCatalogMutation — success notifications", () =>
|
||||
{ slug: "a", message: "name taken" },
|
||||
{ slug: "b", message: "name taken" },
|
||||
],
|
||||
createdRoles: [{ slug: "ok", name: "Ok" }],
|
||||
skippedRoles: [],
|
||||
});
|
||||
expect(notificationsShowMock).toHaveBeenCalledTimes(2);
|
||||
expect(notificationsShowMock).toHaveBeenNthCalledWith(1, {
|
||||
|
||||
@@ -108,12 +108,25 @@ export interface IAiRoleImportPayload {
|
||||
conflict: "skip" | "rename";
|
||||
}
|
||||
|
||||
/** Import result counts (mirrors `importFromCatalog()`). */
|
||||
/**
|
||||
* Import result (mirrors `importFromCatalog()`). The counters (`created`,
|
||||
* `skipped`, `renamed`) drive the summary notification; the per-role lists
|
||||
* (`createdRoles`, `skippedRoles`) drive the redesigned catalog modal's inline
|
||||
* result plaque — which roles were installed (and any rename) and which were
|
||||
* skipped and why (so the plaque can name the conflicting role and offer
|
||||
* "Rename & install").
|
||||
*/
|
||||
export interface IAiRoleImportResult {
|
||||
created: number;
|
||||
skipped: number;
|
||||
renamed: number;
|
||||
errors: { slug: string; message: string }[];
|
||||
createdRoles: { slug: string; name: string; renamedTo?: string }[];
|
||||
skippedRoles: {
|
||||
slug: string;
|
||||
name: string;
|
||||
reason: "name-conflict" | "already-installed";
|
||||
}[];
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
bundleCounts,
|
||||
bundlePhase,
|
||||
installedLangForRole,
|
||||
mapBundleRolesToView,
|
||||
mapCatalogRoleToView,
|
||||
nameConflictSlugs,
|
||||
partialOffersRename,
|
||||
type CatalogViewRole,
|
||||
} from "./catalog-bundle-model.ts";
|
||||
import type {
|
||||
IAiRole,
|
||||
IAiRoleCatalogRole,
|
||||
} from "@/features/ai-chat/types/ai-chat.types.ts";
|
||||
|
||||
function installedRole(
|
||||
source: { slug: string; language: string; version: number },
|
||||
overrides: Partial<IAiRole> = {},
|
||||
): IAiRole {
|
||||
return {
|
||||
id: `role-${source.slug}-${source.language}`,
|
||||
name: source.slug,
|
||||
emoji: null,
|
||||
description: null,
|
||||
enabled: true,
|
||||
autoStart: true,
|
||||
launchMessage: null,
|
||||
source,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function catalogRole(
|
||||
overrides: Partial<IAiRoleCatalogRole> = {},
|
||||
): IAiRoleCatalogRole {
|
||||
return {
|
||||
slug: "writer",
|
||||
emoji: "✍️",
|
||||
name: "Writer",
|
||||
description: "Drafts copy.",
|
||||
instructions: "be a writer",
|
||||
autoStart: true,
|
||||
launchMessage: null,
|
||||
version: 3,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// Build a minimal view role for bundlePhase tests.
|
||||
function viewRole(status: CatalogViewRole["status"]): CatalogViewRole {
|
||||
return { slug: `s-${status}`, name: status, description: "", version: 1, status };
|
||||
}
|
||||
|
||||
describe("bundlePhase", () => {
|
||||
it("empty bundle -> empty", () => {
|
||||
expect(bundlePhase([])).toBe("empty");
|
||||
});
|
||||
|
||||
it("all importable, none installed -> allNew", () => {
|
||||
expect(bundlePhase([viewRole("import"), viewRole("import")])).toBe(
|
||||
"allNew",
|
||||
);
|
||||
});
|
||||
|
||||
it("nothing to import or update -> allInstalled", () => {
|
||||
expect(bundlePhase([viewRole("installed"), viewRole("installed")])).toBe(
|
||||
"allInstalled",
|
||||
);
|
||||
});
|
||||
|
||||
it("updates present, nothing to import -> updates", () => {
|
||||
expect(bundlePhase([viewRole("update"), viewRole("installed")])).toBe(
|
||||
"updates",
|
||||
);
|
||||
});
|
||||
|
||||
it("import + installed (no updates) -> mixed", () => {
|
||||
expect(bundlePhase([viewRole("import"), viewRole("installed")])).toBe(
|
||||
"mixed",
|
||||
);
|
||||
});
|
||||
|
||||
it("import + update -> mixed", () => {
|
||||
expect(bundlePhase([viewRole("import"), viewRole("update")])).toBe("mixed");
|
||||
});
|
||||
|
||||
it("a skipped role with nothing installed -> mixed (NOT allInstalled)", () => {
|
||||
// F1: a bundle whose only non-installed role was skipped has 0 installed for
|
||||
// it, so the collapsed 'All installed · up to date' header would contradict
|
||||
// the open 'Installed 0 · 1 skipped' plaque. It must be mixed until resolved.
|
||||
expect(bundlePhase([viewRole("skipped")])).toBe("mixed");
|
||||
});
|
||||
|
||||
it("installed + a skipped role -> mixed (partial success is not allInstalled)", () => {
|
||||
expect(bundlePhase([viewRole("installed"), viewRole("skipped")])).toBe(
|
||||
"mixed",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("bundleCounts", () => {
|
||||
it("tallies each status once", () => {
|
||||
expect(
|
||||
bundleCounts([
|
||||
viewRole("import"),
|
||||
viewRole("import"),
|
||||
viewRole("installed"),
|
||||
viewRole("update"),
|
||||
viewRole("skipped"),
|
||||
]),
|
||||
).toEqual({ importable: 2, installed: 1, update: 1, skipped: 1 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("nameConflictSlugs / partialOffersRename (reason -> action)", () => {
|
||||
it("only name-conflict skips become the transient overlay / offer rename", () => {
|
||||
const skipped = [
|
||||
{ slug: "writer", name: "Writer", reason: "name-conflict" as const },
|
||||
{ slug: "editor", name: "Editor", reason: "already-installed" as const },
|
||||
];
|
||||
expect(nameConflictSlugs(skipped)).toEqual(["writer"]);
|
||||
expect(partialOffersRename(skipped)).toBe(true);
|
||||
});
|
||||
|
||||
it("an already-installed-only skip is informational: no overlay, no rename", () => {
|
||||
const skipped = [
|
||||
{ slug: "editor", name: "Editor", reason: "already-installed" as const },
|
||||
];
|
||||
expect(nameConflictSlugs(skipped)).toEqual([]);
|
||||
expect(partialOffersRename(skipped)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("installedLangForRole", () => {
|
||||
it("returns the other language when the same slug is installed elsewhere", () => {
|
||||
const roles = [installedRole({ slug: "writer", language: "ru", version: 2 })];
|
||||
expect(installedLangForRole("writer", roles, "en")).toBe("ru");
|
||||
});
|
||||
|
||||
it("returns undefined when the same slug is installed in the SAME language", () => {
|
||||
const roles = [installedRole({ slug: "writer", language: "en", version: 2 })];
|
||||
expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("returns undefined when no install of the slug exists", () => {
|
||||
expect(installedLangForRole("writer", [], "en")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores manually-created roles (no source)", () => {
|
||||
const roles = [
|
||||
installedRole({ slug: "writer", language: "ru", version: 2 }, {
|
||||
source: null,
|
||||
}),
|
||||
];
|
||||
expect(installedLangForRole("writer", roles, "en")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("mapCatalogRoleToView", () => {
|
||||
it("no install -> import status, catalog version, emoji preserved", () => {
|
||||
const view = mapCatalogRoleToView(catalogRole(), [], "en");
|
||||
expect(view).toMatchObject({
|
||||
slug: "writer",
|
||||
emoji: "✍️",
|
||||
name: "Writer",
|
||||
description: "Drafts copy.",
|
||||
status: "import",
|
||||
version: 3,
|
||||
});
|
||||
expect(view.installedRoleId).toBeUndefined();
|
||||
expect(view.installedLang).toBeUndefined();
|
||||
});
|
||||
|
||||
it("import with the slug installed in another language -> installedLang set", () => {
|
||||
const roles = [installedRole({ slug: "writer", language: "ru", version: 9 })];
|
||||
const view = mapCatalogRoleToView(catalogRole(), roles, "en");
|
||||
expect(view.status).toBe("import");
|
||||
expect(view.installedLang).toBe("ru");
|
||||
});
|
||||
|
||||
it("installed (up to date) -> installed status, catalog version, installedRoleId", () => {
|
||||
const installed = installedRole({
|
||||
slug: "writer",
|
||||
language: "en",
|
||||
version: 3,
|
||||
});
|
||||
const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
|
||||
expect(view).toMatchObject({
|
||||
status: "installed",
|
||||
version: 3,
|
||||
installedRoleId: installed.id,
|
||||
});
|
||||
});
|
||||
|
||||
it("update -> version=from, newVersion=to, installedRoleId", () => {
|
||||
const installed = installedRole({
|
||||
slug: "writer",
|
||||
language: "en",
|
||||
version: 1,
|
||||
});
|
||||
const view = mapCatalogRoleToView(catalogRole(), [installed], "en");
|
||||
expect(view).toMatchObject({
|
||||
status: "update",
|
||||
version: 1,
|
||||
newVersion: 3,
|
||||
installedRoleId: installed.id,
|
||||
});
|
||||
});
|
||||
|
||||
it("missing emoji -> emoji undefined; null description -> empty string", () => {
|
||||
const view = mapCatalogRoleToView(
|
||||
catalogRole({ emoji: null, description: null }),
|
||||
[],
|
||||
"en",
|
||||
);
|
||||
expect(view.emoji).toBeUndefined();
|
||||
expect(view.description).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("mapBundleRolesToView", () => {
|
||||
it("maps a bundle's roles preserving order", () => {
|
||||
const roles = [
|
||||
catalogRole({ slug: "a", name: "A", version: 1 }),
|
||||
catalogRole({ slug: "b", name: "B", version: 1 }),
|
||||
];
|
||||
const installed = [installedRole({ slug: "a", language: "en", version: 1 })];
|
||||
const view = mapBundleRolesToView(roles, installed, "en");
|
||||
expect(view.map((r) => r.slug)).toEqual(["a", "b"]);
|
||||
expect(view[0].status).toBe("installed");
|
||||
expect(view[1].status).toBe("import");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,206 @@
|
||||
import type {
|
||||
IAiRole,
|
||||
IAiRoleCatalogRole,
|
||||
} from "@/features/ai-chat/types/ai-chat.types.ts";
|
||||
import { catalogRoleInstallState } from "@/features/ai-chat/utils/catalog-role-install-state.ts";
|
||||
|
||||
/**
|
||||
* The redesigned catalog modal renders bundles as cards with a summary status
|
||||
* (readable without expanding) and a single primary action. The per-role and
|
||||
* per-bundle view model that drives that UI is derived here as PURE functions so
|
||||
* the mapping, the "installed in another language" hint, and the bundle-phase
|
||||
* computation are unit-testable without mounting the component (mirrors the
|
||||
* `catalogRoleInstallState` precedent).
|
||||
*/
|
||||
|
||||
/**
|
||||
* A role's status in the catalog view model.
|
||||
* - `import` — not installed in the current content language.
|
||||
* - `installed` — installed and up to date.
|
||||
* - `update` — installed, but the catalog ships a newer version.
|
||||
* - `skipped` — TRANSIENT client-only status set after a conflicted import
|
||||
* (a name collision under `conflict:'skip'`); never from the
|
||||
* backend.
|
||||
*/
|
||||
export type RoleStatus = "import" | "installed" | "update" | "skipped";
|
||||
|
||||
/** A catalog role mapped into the modal's view model. */
|
||||
export interface CatalogViewRole {
|
||||
// Slug is the stable identity within a bundle; used as the row key and as the
|
||||
// `slugs[]` payload for import.
|
||||
slug: string;
|
||||
// Optional in the catalog — the row reserves space and renders nothing when
|
||||
// absent.
|
||||
emoji?: string;
|
||||
name: string;
|
||||
description: string;
|
||||
// For `installed`/`import`: the catalog version. For `update`: the installed
|
||||
// (from) version, with `newVersion` holding the catalog (to) version.
|
||||
version: number;
|
||||
newVersion?: number;
|
||||
status: RoleStatus;
|
||||
// The language a same-slug role is installed under, when it differs from the
|
||||
// current content language (drives the Р5 hint). Only set for `import` roles.
|
||||
installedLang?: string;
|
||||
// The workspace role id, present for `installed`/`update` — needed to call the
|
||||
// update-from-catalog mutation.
|
||||
installedRoleId?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The summary phase of a bundle, derived from its roles' statuses. Determines
|
||||
* the collapsed-header summary and the bundle's single primary action.
|
||||
* - `empty` — the bundle has no roles.
|
||||
* - `allNew` — everything is importable, nothing installed.
|
||||
* - `allInstalled` — everything installed & up to date; nothing else pending.
|
||||
* - `updates` — updates available and nothing left to import.
|
||||
* - `mixed` — any other combination.
|
||||
*/
|
||||
export type BundlePhase =
|
||||
| "empty"
|
||||
| "allNew"
|
||||
| "allInstalled"
|
||||
| "updates"
|
||||
| "mixed";
|
||||
|
||||
/** Per-status tallies for a bundle's roles (the single source of truth). */
|
||||
export interface BundleCounts {
|
||||
importable: number;
|
||||
installed: number;
|
||||
update: number;
|
||||
skipped: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count a bundle's roles by status ONCE. Both `bundlePhase` and the panel derive
|
||||
* from this, so the tally logic lives in exactly one place (no rescans / drift).
|
||||
*/
|
||||
export function bundleCounts(roles: CatalogViewRole[]): BundleCounts {
|
||||
const counts: BundleCounts = {
|
||||
importable: 0,
|
||||
installed: 0,
|
||||
update: 0,
|
||||
skipped: 0,
|
||||
};
|
||||
for (const r of roles) {
|
||||
if (r.status === "import") counts.importable += 1;
|
||||
else if (r.status === "installed") counts.installed += 1;
|
||||
else if (r.status === "update") counts.update += 1;
|
||||
else if (r.status === "skipped") counts.skipped += 1;
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
|
||||
export function bundlePhase(roles: CatalogViewRole[]): BundlePhase {
|
||||
if (roles.length === 0) return "empty";
|
||||
const { importable, installed, update, skipped } = bundleCounts(roles);
|
||||
// A `skipped` role is a pending post-import conflict (0 installed for it), so a
|
||||
// bundle that has ANY skipped role is NOT "all installed & up to date" — that
|
||||
// would make the collapsed green "up to date" header contradict the open
|
||||
// panel's "Installed 0 · 1 skipped" plaque. It is `mixed` until resolved.
|
||||
if (importable === 0 && update === 0 && skipped === 0) return "allInstalled";
|
||||
if (update > 0 && importable === 0 && skipped === 0) return "updates";
|
||||
if (importable > 0 && installed === 0 && update === 0 && skipped === 0)
|
||||
return "allNew";
|
||||
return "mixed";
|
||||
}
|
||||
|
||||
/**
|
||||
* The subset of a skip result that should be shown as a TRANSIENT `skipped`
|
||||
* overlay in the bundle (so the row offers a re-import path). Only NAME-CONFLICT
|
||||
* skips qualify: an `already-installed` skip (a concurrent-import race) has
|
||||
* nothing to act on — re-importing the same slug would just skip again — so it
|
||||
* must NOT be overlaid (else the row shows a misleading "Rename & install" that
|
||||
* self-heals into a false "installed"). Pure so both reason branches are tested.
|
||||
*/
|
||||
export function nameConflictSlugs(
|
||||
skipped: { slug: string; reason: "name-conflict" | "already-installed" }[],
|
||||
): string[] {
|
||||
return skipped
|
||||
.filter((s) => s.reason === "name-conflict")
|
||||
.map((s) => s.slug);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a partial-import result should offer the "Rename & install" action:
|
||||
* only when at least one skip is a name conflict (renameable). An
|
||||
* `already-installed`-only partial is informational.
|
||||
*/
|
||||
export function partialOffersRename(
|
||||
skipped: { reason: "name-conflict" | "already-installed" }[],
|
||||
): boolean {
|
||||
return skipped.some((s) => s.reason === "name-conflict");
|
||||
}
|
||||
|
||||
/**
|
||||
* For a role NOT installed in the current `language`, find a workspace role with
|
||||
* the same catalog `slug` installed under a DIFFERENT language, and return that
|
||||
* language. Drives the "installed in another language" hint (Р5): a different
|
||||
* language of the same slug is a separate install and appears as `import`.
|
||||
*/
|
||||
export function installedLangForRole(
|
||||
slug: string,
|
||||
workspaceRoles: IAiRole[],
|
||||
language: string,
|
||||
): string | undefined {
|
||||
const other = workspaceRoles.find(
|
||||
(r) =>
|
||||
r.source?.slug === slug &&
|
||||
!!r.source?.language &&
|
||||
r.source.language !== language,
|
||||
);
|
||||
return other?.source?.language;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map one catalog role to the view model, computing its install status against
|
||||
* the workspace roles (via `catalogRoleInstallState`) and, for importable roles,
|
||||
* the other-language hint.
|
||||
*/
|
||||
export function mapCatalogRoleToView(
|
||||
role: IAiRoleCatalogRole,
|
||||
workspaceRoles: IAiRole[],
|
||||
language: string,
|
||||
): CatalogViewRole {
|
||||
const state = catalogRoleInstallState(role, workspaceRoles, language);
|
||||
const base = {
|
||||
slug: role.slug,
|
||||
emoji: role.emoji ?? undefined,
|
||||
name: role.name,
|
||||
description: role.description ?? "",
|
||||
};
|
||||
if (state.state === "update") {
|
||||
return {
|
||||
...base,
|
||||
status: "update",
|
||||
version: state.fromVersion,
|
||||
newVersion: state.toVersion,
|
||||
installedRoleId: state.installed.id,
|
||||
};
|
||||
}
|
||||
if (state.state === "installed") {
|
||||
return {
|
||||
...base,
|
||||
status: "installed",
|
||||
version: role.version,
|
||||
installedRoleId: state.installed.id,
|
||||
};
|
||||
}
|
||||
return {
|
||||
...base,
|
||||
status: "import",
|
||||
version: role.version,
|
||||
installedLang: installedLangForRole(role.slug, workspaceRoles, language),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a whole bundle's catalog roles to the view model, preserving order.
|
||||
*/
|
||||
export function mapBundleRolesToView(
|
||||
roles: IAiRoleCatalogRole[],
|
||||
workspaceRoles: IAiRole[],
|
||||
language: string,
|
||||
): CatalogViewRole[] {
|
||||
return roles.map((r) => mapCatalogRoleToView(r, workspaceRoles, language));
|
||||
}
|
||||
@@ -46,6 +46,13 @@ export function AudioMenu({ editor }: EditorMenuProps) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// #343 PART 1: skip getAttributes unless an audio node is active. The menu
|
||||
// only shows for an active audio node (shouldShow), so the null state while
|
||||
// inactive is never rendered — behavior unchanged.
|
||||
if (!ctx.editor.isActive("audio")) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const audioAttrs = ctx.editor.getAttributes("audio");
|
||||
|
||||
return {
|
||||
|
||||
@@ -43,8 +43,15 @@ export function CalloutMenu({ editor }: EditorMenuProps) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// #343 PART 1: skip the per-type isActive() probes unless a callout is
|
||||
// active. The menu only shows for an active callout (shouldShow), so the
|
||||
// null state while inactive is never rendered — behavior unchanged.
|
||||
if (!ctx.editor.isActive("callout")) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return {
|
||||
isCallout: ctx.editor.isActive("callout"),
|
||||
isCallout: true,
|
||||
isInfo: ctx.editor.isActive("callout", { type: "info" }),
|
||||
isNote: ctx.editor.isActive("callout", { type: "note" }),
|
||||
isSuccess: ctx.editor.isActive("callout", { type: "success" }),
|
||||
|
||||
@@ -22,6 +22,12 @@ export default function CodeBlockView(props: NodeViewProps) {
|
||||
const [isSelected, setIsSelected] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
// #343 PART 6: `isSelected` only drives the mermaid source's visibility (the
|
||||
// `hidden` prop below). For every non-mermaid code block it is never read,
|
||||
// so skip the per-block `selectionUpdate` listener entirely — otherwise N
|
||||
// code blocks each add a global listener + a setState on every caret move.
|
||||
if (language !== "mermaid") return;
|
||||
|
||||
const updateSelection = () => {
|
||||
const { state } = editor;
|
||||
const { from, to } = state.selection;
|
||||
@@ -32,11 +38,14 @@ export default function CodeBlockView(props: NodeViewProps) {
|
||||
setIsSelected(isNodeSelected);
|
||||
};
|
||||
|
||||
// Initialize on attach so switching a block's language to "mermaid" reflects
|
||||
// the current selection immediately (the listener was not running before).
|
||||
updateSelection();
|
||||
editor.on("selectionUpdate", updateSelection);
|
||||
return () => {
|
||||
editor.off("selectionUpdate", updateSelection);
|
||||
};
|
||||
}, [editor, getPos(), node.nodeSize]);
|
||||
}, [editor, getPos(), node.nodeSize, language]);
|
||||
|
||||
function changeLanguage(language: string) {
|
||||
setLanguageValue(language);
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import type { Editor } from "@tiptap/react";
|
||||
import { useEditorState } from "@tiptap/react";
|
||||
import { undoDepth, redoDepth } from "@tiptap/pm/history";
|
||||
import { yUndoPluginKey } from "@tiptap/y-tiptap";
|
||||
|
||||
export interface ToolbarState {
|
||||
isBold: boolean;
|
||||
@@ -16,14 +18,45 @@ export interface ToolbarState {
|
||||
canRedo: boolean;
|
||||
}
|
||||
|
||||
// Undo/redo come from either StarterKit's history or the Yjs collaboration
|
||||
// history extension. During the brief moment a page is rendered with the
|
||||
// static editor (mainExtensions only, undoRedo disabled), neither is loaded
|
||||
// and editor.can().undo/redo is undefined.
|
||||
function safeCan(editor: Editor, command: "undo" | "redo"): boolean {
|
||||
const can = editor.can() as Record<string, unknown>;
|
||||
const fn = can[command];
|
||||
return typeof fn === "function" ? (fn as () => boolean)() : false;
|
||||
// Undo/redo availability, computed WITHOUT `editor.can().undo()/.redo()`.
|
||||
//
|
||||
// `editor.can()` runs the command as a dry-run (building a throwaway state +
|
||||
// transaction) — the most expensive work in this selector, and it ran on every
|
||||
// keystroke (and every REMOTE keystroke under collaboration). Instead we read
|
||||
// the history stack depth directly, which is a cheap plugin-state lookup and
|
||||
// mirrors exactly what the undo/redo commands themselves check:
|
||||
//
|
||||
// - Collaboration (Yjs): the yjs UndoManager's undo/redo stack lengths — the
|
||||
// same `undoStack.length === 0` / `redoStack.length === 0` guard the
|
||||
// Collaboration extension's undo/redo commands use.
|
||||
// - Plain history (templates / non-collab): prosemirror-history's undoDepth /
|
||||
// redoDepth, which back the UndoRedo extension.
|
||||
//
|
||||
// When neither history backend is installed (the pre-sync static editor —
|
||||
// mainExtensions only, undoRedo disabled), both fall through to 0 -> false,
|
||||
// matching the previous `safeCan` behavior.
|
||||
function historyAvailability(editor: Editor): {
|
||||
canUndo: boolean;
|
||||
canRedo: boolean;
|
||||
} {
|
||||
const state = editor.state;
|
||||
|
||||
// Collaboration history (Yjs) takes precedence when present.
|
||||
const yState = yUndoPluginKey.getState(state) as
|
||||
| { undoManager?: { undoStack: unknown[]; redoStack: unknown[] } }
|
||||
| undefined;
|
||||
if (yState?.undoManager) {
|
||||
return {
|
||||
canUndo: yState.undoManager.undoStack.length > 0,
|
||||
canRedo: yState.undoManager.redoStack.length > 0,
|
||||
};
|
||||
}
|
||||
|
||||
// Plain prosemirror-history (returns 0 when the history plugin is absent).
|
||||
return {
|
||||
canUndo: undoDepth(state) > 0,
|
||||
canRedo: redoDepth(state) > 0,
|
||||
};
|
||||
}
|
||||
|
||||
export function useToolbarState(editor: Editor | null): ToolbarState | null {
|
||||
@@ -31,6 +64,7 @@ export function useToolbarState(editor: Editor | null): ToolbarState | null {
|
||||
editor,
|
||||
selector: (ctx) => {
|
||||
if (!ctx.editor) return null;
|
||||
const { canUndo, canRedo } = historyAvailability(ctx.editor);
|
||||
return {
|
||||
isBold: ctx.editor.isActive("bold"),
|
||||
isItalic: ctx.editor.isActive("italic"),
|
||||
@@ -42,8 +76,8 @@ export function useToolbarState(editor: Editor | null): ToolbarState | null {
|
||||
isBulletList: ctx.editor.isActive("bulletList"),
|
||||
isOrderedList: ctx.editor.isActive("orderedList"),
|
||||
isTaskList: ctx.editor.isActive("taskList"),
|
||||
canUndo: safeCan(ctx.editor, "undo"),
|
||||
canRedo: safeCan(ctx.editor, "redo"),
|
||||
canUndo,
|
||||
canRedo,
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
@@ -38,6 +38,14 @@ export function ImageMenu({ editor }: EditorMenuProps) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// #343 PART 1: skip the expensive per-keystroke work (getAttributes + the
|
||||
// alignment isActive() probes) unless an image is actually active. The
|
||||
// menu is only shown when an image is active (see shouldShow), so a null
|
||||
// state while inactive is never rendered — behavior is unchanged.
|
||||
if (!ctx.editor.isActive("image")) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const imageAttrs = ctx.editor.getAttributes("image");
|
||||
|
||||
return {
|
||||
|
||||
@@ -25,6 +25,13 @@ export function PdfMenu({ editor }: EditorMenuProps) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// #343 PART 1: skip getAttributes unless a pdf node is active. The menu
|
||||
// only shows for an active pdf node (shouldShow), so the null state while
|
||||
// inactive is never rendered — behavior unchanged.
|
||||
if (!ctx.editor.isActive("pdf")) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const pdfAttrs = ctx.editor.getAttributes("pdf");
|
||||
|
||||
return {
|
||||
|
||||
@@ -70,7 +70,14 @@ export const SubpagesMenu = React.memo(
|
||||
// toggle without re-rendering on every keystroke.
|
||||
const isRecursive = useEditorState({
|
||||
editor,
|
||||
selector: (ctx) => ctx.editor?.getAttributes("subpages")?.recursive ?? false,
|
||||
// #343 PART 1: skip getAttributes unless a subpages node is active. The
|
||||
// menu only shows for an active subpages node (shouldShow), so the value
|
||||
// is only read then; getAttributes on an inactive node returns the default
|
||||
// (recursive === false) anyway, so this is behavior-preserving.
|
||||
selector: (ctx) =>
|
||||
ctx.editor?.isActive("subpages")
|
||||
? (ctx.editor.getAttributes("subpages")?.recursive ?? false)
|
||||
: false,
|
||||
});
|
||||
|
||||
return (
|
||||
|
||||
@@ -4,6 +4,7 @@ import React, { FC, useEffect, useRef, useState } from "react";
|
||||
import classes from "./table-of-contents.module.css";
|
||||
import clsx from "clsx";
|
||||
import { Box, Text, Title } from "@mantine/core";
|
||||
import { useDebouncedCallback } from "@mantine/hooks";
|
||||
import { useTranslation } from "react-i18next";
|
||||
|
||||
type TableOfContentsProps = {
|
||||
@@ -79,13 +80,21 @@ export const TableOfContents: FC<TableOfContentsProps> = (props) => {
|
||||
setHeadingDOMNodes(result.nodes);
|
||||
};
|
||||
|
||||
// Debounce the update-driven rescan: `$nodes("heading")` scans every heading
|
||||
// in the document, and it previously ran on EVERY keystroke while the TOC
|
||||
// panel was open. The panel is derived UI, so recomputing ~300ms after typing
|
||||
// settles keeps it correct without doing an all-headings scan per keystroke
|
||||
// (#343, PART 7). `useDebouncedCallback` returns a stable reference and always
|
||||
// invokes the latest `handleUpdate`.
|
||||
const debouncedHandleUpdate = useDebouncedCallback(handleUpdate, 300);
|
||||
|
||||
useEffect(() => {
|
||||
props.editor?.on("update", handleUpdate);
|
||||
props.editor?.on("update", debouncedHandleUpdate);
|
||||
|
||||
return () => {
|
||||
props.editor?.off("update", handleUpdate);
|
||||
props.editor?.off("update", debouncedHandleUpdate);
|
||||
};
|
||||
}, [props.editor]);
|
||||
}, [props.editor, debouncedHandleUpdate]);
|
||||
|
||||
useEffect(
|
||||
() => {
|
||||
|
||||
@@ -31,6 +31,13 @@ export function VideoMenu({ editor }: EditorMenuProps) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// #343 PART 1: skip getAttributes + alignment isActive() probes unless a
|
||||
// video is active. The menu only shows for an active video (shouldShow),
|
||||
// so the null state while inactive is never rendered — behavior unchanged.
|
||||
if (!ctx.editor.isActive("video")) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const videoAttrs = ctx.editor.getAttributes("video");
|
||||
|
||||
return {
|
||||
|
||||
@@ -6,6 +6,23 @@ import getSuggestionItems from '@/features/editor/components/slash-menu/menu-ite
|
||||
|
||||
export const slashMenuPluginKey = new PluginKey('slash-command');
|
||||
|
||||
// getSuggestionItems fuzzy-matches EVERY command against the query (plus its
|
||||
// wrong-keyboard-layout remaps) and, while the slash menu is open, is invoked
|
||||
// TWICE per keystroke: once by the synchronous `allow` gate below and once by
|
||||
// the popup's `items` builder. A synchronous gating predicate can't be
|
||||
// debounced without breaking the suggestion decoration/activation, so instead we
|
||||
// memoize the LAST query's result: the two same-query calls in one keystroke
|
||||
// build the list only once, and the cache invalidates the moment the query
|
||||
// changes — so there is no stale-state risk (#343, PART 7).
|
||||
let lastQuery: string | null = null;
|
||||
let lastResult: ReturnType<typeof getSuggestionItems> | null = null;
|
||||
function suggestionItemsForQuery(query: string) {
|
||||
if (query === lastQuery && lastResult) return lastResult;
|
||||
lastQuery = query;
|
||||
lastResult = getSuggestionItems({ query });
|
||||
return lastResult;
|
||||
}
|
||||
|
||||
// @ts-ignore
|
||||
const Command = Extension.create({
|
||||
name: 'slash-command',
|
||||
@@ -38,7 +55,7 @@ const Command = Extension.create({
|
||||
// non-matching queries while keeping multi-word matches (e.g.
|
||||
// "/Heading 1") working.
|
||||
const query = state.doc.textBetween(range.from + 1, range.to);
|
||||
const groups = getSuggestionItems({ query });
|
||||
const groups = suggestionItemsForQuery(query);
|
||||
const hasMatches = Object.values(groups).some(
|
||||
(items) => items.length > 0,
|
||||
);
|
||||
@@ -61,7 +78,9 @@ const Command = Extension.create({
|
||||
|
||||
const SlashCommand = Command.configure({
|
||||
suggestion: {
|
||||
items: getSuggestionItems,
|
||||
// Share the per-query memo with `allow` so the pair of same-query calls in a
|
||||
// single keystroke rebuilds the list once (#343, PART 7).
|
||||
items: ({ query }: { query: string }) => suggestionItemsForQuery(query),
|
||||
render: renderItems,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -24,6 +24,7 @@ import {
|
||||
GitmostListPagesResult,
|
||||
GitmostListSpacesResult,
|
||||
gitmostDecodePayloadToFile,
|
||||
gitmostInsertTranscriptIntoEditor,
|
||||
gitmostUploadFileToEditor,
|
||||
} from "@/features/editor/gitmost/gitmost-recording.ts";
|
||||
|
||||
@@ -281,6 +282,18 @@ export default function GitmostGlobalBridge() {
|
||||
pageId: page.id,
|
||||
};
|
||||
}
|
||||
|
||||
// Best-effort: append the transcript (heading + one paragraph per line)
|
||||
// below the just-inserted audio node. The audio insert already
|
||||
// succeeded, so a transcript failure must NOT turn this into an error —
|
||||
// wrap it and, on any throw, log and still return ok. A missing/empty/
|
||||
// non-string transcript is a no-op inside the helper (audio only).
|
||||
try {
|
||||
gitmostInsertTranscriptIntoEditor(editor, payload?.transcript);
|
||||
} catch (err) {
|
||||
console.error("[gitmost] transcript insert failed", err);
|
||||
}
|
||||
|
||||
return { ok: true, pageId: page.id };
|
||||
} catch (err: any) {
|
||||
console.error("[gitmost] createPageWithRecording failed", err);
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { Editor } from "@tiptap/core";
|
||||
import { Document } from "@tiptap/extension-document";
|
||||
import { Paragraph } from "@tiptap/extension-paragraph";
|
||||
import { Text } from "@tiptap/extension-text";
|
||||
import { Heading } from "@tiptap/extension-heading";
|
||||
import { Bold } from "@tiptap/extension-bold";
|
||||
import { Italic } from "@tiptap/extension-italic";
|
||||
import { Link } from "@tiptap/extension-link";
|
||||
import { gitmostInsertTranscriptIntoEditor } from "./gitmost-recording.ts";
|
||||
|
||||
const ZWSP = ""; // U+200B, the helper's block-trigger neutralizer
|
||||
|
||||
/**
|
||||
* #377 — the web-side bridge must append the native host's transcript below the
|
||||
* recording. These exercise the pure insert helper through a REAL Tiptap editor
|
||||
* (Document/Paragraph/Text/Heading + Bold/Italic/Link marks so an HTML-parsing
|
||||
* regression would be caught), asserting the resulting document rather than
|
||||
* mocking the editor: transcript present -> "Transcript" heading + one paragraph
|
||||
* per non-empty line; content is inserted as LITERAL TEXT (no HTML/markdown
|
||||
* parsing); col-0 markdown block triggers are neutralized so git-sync keeps them
|
||||
* paragraphs; absent/empty/non-string -> no-op.
|
||||
*/
|
||||
describe("gitmostInsertTranscriptIntoEditor", () => {
|
||||
const makeEditor = () =>
|
||||
new Editor({
|
||||
// Bold/Italic/Link are registered specifically so that IF the helper ever
|
||||
// regressed to inserting an HTML/markdown string (instead of a text node),
|
||||
// TipTap would parse `<b>`/`*..*`/`[..](..)` into marks and the literal-
|
||||
// text assertions below would fail.
|
||||
extensions: [Document, Paragraph, Text, Heading, Bold, Italic, Link],
|
||||
// Start from a single empty paragraph (a fresh page's baseline). The
|
||||
// helper appends at the end of the doc, i.e. below existing content.
|
||||
content: { type: "doc", content: [{ type: "paragraph" }] },
|
||||
});
|
||||
|
||||
it("inserts a Transcript heading + one paragraph per non-empty line, verbatim", () => {
|
||||
const editor = makeEditor();
|
||||
|
||||
const inserted = gitmostInsertTranscriptIntoEditor(
|
||||
editor,
|
||||
"You: hello there\nSpeaker 1: hi\n\nYou: bye",
|
||||
);
|
||||
|
||||
expect(inserted).toBe(true);
|
||||
|
||||
const nodes = (editor.getJSON().content ?? []) as any[];
|
||||
// A level-2 "Transcript" heading is present.
|
||||
const heading = nodes.find((n) => n.type === "heading");
|
||||
expect(heading?.attrs?.level).toBe(2);
|
||||
expect(heading?.content?.[0]?.text).toBe("Transcript");
|
||||
|
||||
// Every non-empty transcript line becomes a paragraph, in order, verbatim;
|
||||
// the blank line between them is dropped.
|
||||
const texts = nodes
|
||||
.filter((n) => n.type === "paragraph")
|
||||
.map((n) => n.content?.[0]?.text)
|
||||
.filter((t) => typeof t === "string");
|
||||
expect(texts).toEqual(["You: hello there", "Speaker 1: hi", "You: bye"]);
|
||||
|
||||
editor.destroy();
|
||||
});
|
||||
|
||||
it("inserts HTML + markdown metacharacters as LITERAL text (no injection / no mark parsing)", () => {
|
||||
const editor = makeEditor();
|
||||
const line =
|
||||
"You: <b>bold</b> <script>alert(1)</script> and *stars* and [link](x)";
|
||||
|
||||
const inserted = gitmostInsertTranscriptIntoEditor(editor, line);
|
||||
expect(inserted).toBe(true);
|
||||
|
||||
const paras = (editor.getJSON().content ?? []).filter(
|
||||
(n: any) => n.type === "paragraph",
|
||||
) as any[];
|
||||
// The transcript line is exactly ONE paragraph holding a SINGLE text node
|
||||
// whose text is the verbatim string — not split into bold/link/other nodes,
|
||||
// not carrying any marks, not raw HTML. This FAILS if the helper switched to
|
||||
// insertContent(htmlString): TipTap would then parse <b>/[link](x)/*stars*.
|
||||
const content = paras[paras.length - 1].content;
|
||||
expect(content).toHaveLength(1);
|
||||
expect(content[0].type).toBe("text");
|
||||
expect(content[0].marks ?? []).toEqual([]);
|
||||
expect(content[0].text).toBe(line);
|
||||
// And no bold/italic/link mark exists anywhere in the document.
|
||||
const html = editor.getHTML();
|
||||
expect(html).not.toMatch(/<(strong|b|em|i|a)\b/);
|
||||
// The angle brackets survived as escaped entities (literal text), not a live
|
||||
// <script>/<b> element.
|
||||
expect(html).not.toMatch(/<script/i);
|
||||
|
||||
editor.destroy();
|
||||
});
|
||||
|
||||
it("neutralizes col-0 markdown block triggers with a leading ZWSP (git-sync safety)", () => {
|
||||
const editor = makeEditor();
|
||||
// Trigger lines (some with a leaked indent) + a normal prefixed line.
|
||||
const inserted = gitmostInsertTranscriptIntoEditor(
|
||||
editor,
|
||||
[
|
||||
"- dash",
|
||||
" > quote", // leading indent must be trimmed then neutralized
|
||||
"# hash",
|
||||
"1. one",
|
||||
"> [!info] note",
|
||||
"```js",
|
||||
"---", // solid thematic break -> horizontalRule (text-losing) if unneutralized
|
||||
"***",
|
||||
"___",
|
||||
"You: normal line",
|
||||
].join("\n"),
|
||||
);
|
||||
expect(inserted).toBe(true);
|
||||
|
||||
const texts = (editor.getJSON().content ?? [])
|
||||
.filter((n: any) => n.type === "paragraph")
|
||||
.map((n: any) => n.content?.[0]?.text)
|
||||
.filter((t: any) => typeof t === "string") as string[];
|
||||
|
||||
// Every block-trigger line is prefixed with the invisible ZWSP (indent
|
||||
// trimmed first); the normal `You:` line is left byte-exact.
|
||||
expect(texts).toEqual([
|
||||
ZWSP + "- dash",
|
||||
ZWSP + "> quote",
|
||||
ZWSP + "# hash",
|
||||
ZWSP + "1. one",
|
||||
ZWSP + "> [!info] note",
|
||||
ZWSP + "```js",
|
||||
ZWSP + "---",
|
||||
ZWSP + "***",
|
||||
ZWSP + "___",
|
||||
"You: normal line",
|
||||
]);
|
||||
|
||||
editor.destroy();
|
||||
});
|
||||
|
||||
it("is a no-op for undefined / empty / whitespace-only / non-string transcripts", () => {
|
||||
for (const value of [undefined, "", " \n \n", 42, {}, null]) {
|
||||
const editor = makeEditor();
|
||||
const before = JSON.stringify(editor.getJSON());
|
||||
|
||||
const inserted = gitmostInsertTranscriptIntoEditor(editor, value as any);
|
||||
|
||||
expect(inserted).toBe(false);
|
||||
// Document is untouched (audio-only behavior preserved).
|
||||
expect(JSON.stringify(editor.getJSON())).toBe(before);
|
||||
editor.destroy();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -65,6 +65,11 @@ export interface GitmostCreatePagePayload {
|
||||
base64: string;
|
||||
filename: string;
|
||||
mimeType: string;
|
||||
// Optional transcript for the recording: plain text, `\n`-separated, each
|
||||
// line already formatted as `You: ...` / `Speaker N: ...` by the native host
|
||||
// (ready to insert, no parsing needed). Omitted (no speech / no models) ->
|
||||
// audio only.
|
||||
transcript?: string;
|
||||
}
|
||||
|
||||
export interface GitmostCreatePageResult {
|
||||
@@ -235,6 +240,83 @@ export async function gitmostUploadFileToEditor(
|
||||
}
|
||||
}
|
||||
|
||||
// Zero-width space (U+200B). Prepended to a transcript line that begins with a
|
||||
// markdown BLOCK trigger: it is invisible in the rendered doc but shifts the
|
||||
// trigger off column 0, so the git-sync doc->markdown->doc round-trip keeps the
|
||||
// line a plain paragraph (see GITMOST_MD_BLOCK_TRIGGER_RE).
|
||||
const GITMOST_ZWSP = "";
|
||||
|
||||
// A markdown BLOCK-level construct that, sitting at column 0 of a paragraph
|
||||
// line, the git-sync markdown serializer (packages/prosemirror-markdown
|
||||
// markdown-converter.ts, `case "paragraph"`) would re-parse into a NON-paragraph
|
||||
// block on the doc->markdown->doc cycle. That serializer emits paragraph text
|
||||
// verbatim with NO block-escape (the pre-existing root cause), so a leading
|
||||
// `#`/`-`/`*`/`+`/`>`, an ordered-list `N.`/`N)`, a code fence ```/~~~, a table
|
||||
// `|`, or a `> [!info]` callout opener would silently become a heading / list /
|
||||
// quote / code block / table / callout. The final alternative matches a WHOLE-
|
||||
// LINE thematic break — solid `---`/`***`/`___` or spaced `- - -`/`_ _ _` (3+ of
|
||||
// the same `-`/`*`/`_`) — which round-trips into a `horizontalRule`; because
|
||||
// that node carries NO text, an un-neutralized separator line would LOSE its
|
||||
// text entirely (worse than the list/quote case). This matches a TRIMMED line's
|
||||
// start; the transcript's own `You:` / `Speaker N:` prefix begins with a letter
|
||||
// and never matches, so prefixed lines are left byte-exact.
|
||||
const GITMOST_MD_BLOCK_TRIGGER_RE =
|
||||
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
|
||||
|
||||
// Append a transcript block BELOW the recording's audio node in a live editor:
|
||||
// a "Transcript" heading followed by one paragraph per non-empty transcript
|
||||
// line. The transcript is plain text, `\n`-separated, each line already
|
||||
// formatted as `You: ...` / `Speaker N: ...` by the native host — line text is
|
||||
// inserted as a TEXT node (never HTML/markdown), so there is no injection or
|
||||
// mark-parsing surface. Each kept line is trimmed (drops an indent that would
|
||||
// both leak into the display and, at col 0, form a markdown block trigger) and,
|
||||
// if it still begins with a col-0 markdown block trigger, gets an invisible
|
||||
// zero-width space prepended so the git-sync round-trip cannot turn it into a
|
||||
// list/quote/heading/callout/code/table (defensive boundary against the
|
||||
// serializer's missing block-escape). This is best-effort and meant to run
|
||||
// AFTER the audio has already been inserted; the caller must guard against a
|
||||
// throw so a transcript failure never fails the (already successful) recording.
|
||||
// Returns true when a block was inserted, false when there was nothing to
|
||||
// insert (transcript undefined/empty/not-a-string). A non-string value is a
|
||||
// no-op, not an error.
|
||||
export function gitmostInsertTranscriptIntoEditor(
|
||||
editor: Editor,
|
||||
transcript: unknown,
|
||||
): boolean {
|
||||
if (typeof transcript !== "string") return false;
|
||||
const lines = transcript
|
||||
.split("\n")
|
||||
// Trim each line and drop blank (whitespace-only) ones.
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.length > 0)
|
||||
// Neutralize a col-0 markdown block trigger with an invisible ZWSP so the
|
||||
// git-sync round-trip keeps the line a paragraph. Host lines (`You:` /
|
||||
// `Speaker N:`) never match and stay byte-exact.
|
||||
.map((line) =>
|
||||
GITMOST_MD_BLOCK_TRIGGER_RE.test(line) ? GITMOST_ZWSP + line : line,
|
||||
);
|
||||
if (lines.length === 0) return false;
|
||||
|
||||
const content = [
|
||||
{
|
||||
type: "heading",
|
||||
attrs: { level: 2 },
|
||||
content: [{ type: "text", text: "Transcript" }],
|
||||
},
|
||||
...lines.map((line) => ({
|
||||
type: "paragraph",
|
||||
content: [{ type: "text", text: line }],
|
||||
})),
|
||||
];
|
||||
|
||||
// Append at the end of the document. On a freshly-created recording page the
|
||||
// audio node is the last block, so the end position places the transcript
|
||||
// directly below it.
|
||||
const endPos = editor.state.doc.content.size;
|
||||
editor.chain().focus().insertContentAt(endPos, content).run();
|
||||
return true;
|
||||
}
|
||||
|
||||
// Full insert path used by the open-page bridge (insertRecording): guard the
|
||||
// editor, validate/decode the payload, then upload. Never throws — resolves to
|
||||
// a result code.
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { renderHook, act } from "@testing-library/react";
|
||||
import type { MutableRefObject } from "react";
|
||||
import type { Editor } from "@tiptap/react";
|
||||
|
||||
// Mock the app entry so importing the hook doesn't boot the whole app; the hook
|
||||
// only needs queryClient's cache read/write, which we stub here. Declared via
|
||||
// vi.hoisted so the spies exist before the hoisted vi.mock factory runs.
|
||||
const { getQueryData, setQueryData } = vi.hoisted(() => ({
|
||||
getQueryData: vi.fn(() => undefined as unknown),
|
||||
setQueryData: vi.fn(),
|
||||
}));
|
||||
vi.mock("@/main.tsx", () => ({
|
||||
queryClient: { getQueryData, setQueryData },
|
||||
}));
|
||||
|
||||
import { usePageContentCache } from "./use-page-content-cache";
|
||||
|
||||
const SNAPSHOT = { type: "doc", content: [] };
|
||||
|
||||
function makeFakeEditor(overrides: Partial<Editor> = {}): Editor {
|
||||
return {
|
||||
isEmpty: false,
|
||||
isDestroyed: false,
|
||||
getJSON: vi.fn(() => SNAPSHOT),
|
||||
...overrides,
|
||||
} as unknown as Editor;
|
||||
}
|
||||
|
||||
describe("usePageContentCache (#343 PART 3) — getJSON off the keystroke path", () => {
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
vi.clearAllMocks();
|
||||
// A cached page exists so the write path runs.
|
||||
getQueryData.mockReturnValue({ id: "p1", content: {} });
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("onUpdate (calling the debounced fn) does NOT call getJSON synchronously", () => {
|
||||
const editor = makeFakeEditor();
|
||||
const editorRef = { current: editor } as MutableRefObject<Editor | null>;
|
||||
|
||||
const { result } = renderHook(() =>
|
||||
usePageContentCache(editorRef, "slug-1", 3000),
|
||||
);
|
||||
|
||||
// Simulate a keystroke's onUpdate -> only schedules the debounce.
|
||||
act(() => {
|
||||
result.current();
|
||||
result.current();
|
||||
result.current();
|
||||
});
|
||||
|
||||
// The whole-doc serialization must NOT have happened yet.
|
||||
expect(editor.getJSON).not.toHaveBeenCalled();
|
||||
expect(setQueryData).not.toHaveBeenCalled();
|
||||
|
||||
// Once the debounce window elapses, getJSON runs exactly once (not per call).
|
||||
act(() => vi.advanceTimersByTime(3000));
|
||||
expect(editor.getJSON).toHaveBeenCalledTimes(1);
|
||||
expect(setQueryData).toHaveBeenCalledWith(["pages", "slug-1"], {
|
||||
id: "p1",
|
||||
content: SNAPSHOT,
|
||||
});
|
||||
});
|
||||
|
||||
it("flushes the pending snapshot on unmount so the last edit isn't lost", () => {
|
||||
const editor = makeFakeEditor();
|
||||
const editorRef = { current: editor } as MutableRefObject<Editor | null>;
|
||||
|
||||
const { result, unmount } = renderHook(() =>
|
||||
usePageContentCache(editorRef, "slug-1", 3000),
|
||||
);
|
||||
|
||||
act(() => result.current());
|
||||
expect(editor.getJSON).not.toHaveBeenCalled();
|
||||
|
||||
// Navigation/unmount must flush (not drop) the pending write.
|
||||
act(() => unmount());
|
||||
expect(editor.getJSON).toHaveBeenCalledTimes(1);
|
||||
expect(setQueryData).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("skips the write when the editor is destroyed (flush racing teardown)", () => {
|
||||
const editor = makeFakeEditor({ isDestroyed: true });
|
||||
const editorRef = { current: editor } as MutableRefObject<Editor | null>;
|
||||
|
||||
const { result } = renderHook(() =>
|
||||
usePageContentCache(editorRef, "slug-1", 3000),
|
||||
);
|
||||
|
||||
act(() => result.current());
|
||||
act(() => vi.advanceTimersByTime(3000));
|
||||
|
||||
expect(editor.getJSON).not.toHaveBeenCalled();
|
||||
expect(setQueryData).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,50 @@
|
||||
import type { MutableRefObject } from "react";
|
||||
import { useDebouncedCallback } from "@mantine/hooks";
|
||||
import type { Editor } from "@tiptap/react";
|
||||
import { queryClient } from "@/main.tsx";
|
||||
import { IPage } from "@/features/page/types/page.types.ts";
|
||||
|
||||
/**
|
||||
* Off-keystroke local page-cache updater (issue #343, PART 3).
|
||||
*
|
||||
* The editor's `onUpdate` fires on every keystroke — and, under collaboration,
|
||||
* on every REMOTE keystroke too. Serializing the WHOLE document with
|
||||
* `editor.getJSON()` on that hot path is expensive, and the previous 3s debounce
|
||||
* only guarded the cache WRITE, not the serialization: `getJSON()` still ran per
|
||||
* keystroke.
|
||||
*
|
||||
* This hook moves the serialization INSIDE the debounced callback, so the
|
||||
* full-doc traversal happens at most once per `delay`, not per keystroke. Call
|
||||
* the returned function from `onUpdate` (it only schedules the debounce); the
|
||||
* `getJSON()` snapshot is taken when the debounce fires.
|
||||
*
|
||||
* On unmount/navigation the pending snapshot is FLUSHED (via `flushOnUnmount`)
|
||||
* so the last edits within the debounce window aren't lost from the local cache.
|
||||
* The source of truth is collab/Yjs, but the cache must not go stale.
|
||||
*
|
||||
* IMPORTANT: call this hook BEFORE `useEditor`. React runs effect cleanups in
|
||||
* declaration order on unmount, so the debounce's flush cleanup must be declared
|
||||
* before `useEditor`'s teardown to run while the editor is still alive; the
|
||||
* `isDestroyed` guard keeps a flush that still races teardown safe (it skips).
|
||||
*/
|
||||
export function usePageContentCache(
|
||||
editorRef: MutableRefObject<Editor | null>,
|
||||
slugId: string | undefined,
|
||||
delay = 3000,
|
||||
) {
|
||||
return useDebouncedCallback(
|
||||
() => {
|
||||
const e = editorRef.current;
|
||||
if (!e || e.isDestroyed || e.isEmpty) return;
|
||||
const pageData = queryClient.getQueryData<IPage>(["pages", slugId]);
|
||||
if (pageData) {
|
||||
// getJSON() (full-doc serialization) runs HERE, off the keystroke path.
|
||||
queryClient.setQueryData(["pages", slugId], {
|
||||
...pageData,
|
||||
content: e.getJSON(),
|
||||
});
|
||||
}
|
||||
},
|
||||
{ delay, flushOnUnmount: true },
|
||||
);
|
||||
}
|
||||
@@ -62,7 +62,7 @@ import ExcalidrawMenu from "./components/excalidraw/excalidraw-menu-lazy";
|
||||
import DrawioMenu from "./components/drawio/drawio-menu";
|
||||
import { useCollabToken } from "@/features/auth/queries/auth-query.tsx";
|
||||
import SearchAndReplaceDialog from "@/features/editor/components/search-and-replace/search-and-replace-dialog.tsx";
|
||||
import { useDebouncedCallback, useDocumentVisibility } from "@mantine/hooks";
|
||||
import { useDocumentVisibility } from "@mantine/hooks";
|
||||
import { useIdle } from "@/hooks/use-idle.ts";
|
||||
import { queryClient } from "@/main.tsx";
|
||||
import { IPage } from "@/features/page/types/page.types.ts";
|
||||
@@ -79,6 +79,7 @@ import { PageEditMode } from "@/features/user/types/user.types.ts";
|
||||
import { jwtDecode } from "jwt-decode";
|
||||
import { searchSpotlight } from "@/features/search/constants.ts";
|
||||
import { useEditorScroll } from "./hooks/use-editor-scroll";
|
||||
import { usePageContentCache } from "./hooks/use-page-content-cache";
|
||||
import { useScrollRestoreOnSwap } from "./hooks/use-scroll-position";
|
||||
import { useSwapHeightReservation } from "./hooks/use-swap-height-reservation";
|
||||
import { EditorLinkMenu } from "@/features/editor/components/link/link-menu";
|
||||
@@ -272,8 +273,13 @@ export default function PageEditor({
|
||||
}
|
||||
}, [isIdle, documentState, providersReady, resetIdle]);
|
||||
|
||||
// Attach here, to make sure the connection gets properly established
|
||||
providersRef.current?.remote.attach();
|
||||
// Attach the remote provider once it's ready (and again after a pageId swap
|
||||
// recreates it) to make sure the connection gets properly established. This
|
||||
// used to run in the render body — a side effect during render (#343, PART 7).
|
||||
// `attach()` is idempotent, so re-running it on these deps is safe.
|
||||
useEffect(() => {
|
||||
providersRef.current?.remote.attach();
|
||||
}, [providersReady, pageId]);
|
||||
|
||||
const extensions = useMemo(() => {
|
||||
if (!providersReady || !providersRef.current || !currentUser?.user) {
|
||||
@@ -288,6 +294,12 @@ export default function PageEditor({
|
||||
];
|
||||
}, [providersReady, currentUser?.user]);
|
||||
|
||||
// getJSON() serialization + cache write live in the hook, off the keystroke
|
||||
// path, and flush on unmount so the last snapshot survives navigation (#343).
|
||||
// MUST be declared before useEditor: React runs effect cleanups in declaration
|
||||
// order on unmount, so the flush must run before the editor is torn down.
|
||||
const debouncedUpdateContent = usePageContentCache(editorRef, slugId);
|
||||
|
||||
const editor = useEditor(
|
||||
{
|
||||
extensions,
|
||||
@@ -392,11 +404,11 @@ export default function PageEditor({
|
||||
}
|
||||
}
|
||||
},
|
||||
onUpdate({ editor }) {
|
||||
if (editor.isEmpty) return;
|
||||
const editorJson = editor.getJSON();
|
||||
//update local page cache to reduce flickers
|
||||
debouncedUpdateContent(editorJson);
|
||||
onUpdate() {
|
||||
// Only schedule the debounce here — the whole-doc getJSON() serialization
|
||||
// happens INSIDE the debounced callback (see usePageContentCache), so it
|
||||
// no longer runs synchronously on every (local or remote) keystroke.
|
||||
debouncedUpdateContent();
|
||||
},
|
||||
},
|
||||
[pageId, editable, extensions],
|
||||
@@ -442,17 +454,6 @@ export default function PageEditor({
|
||||
};
|
||||
}, [editor, pageId, editorIsEditable]);
|
||||
|
||||
const debouncedUpdateContent = useDebouncedCallback((newContent: any) => {
|
||||
const pageData = queryClient.getQueryData<IPage>(["pages", slugId]);
|
||||
|
||||
if (pageData) {
|
||||
queryClient.setQueryData(["pages", slugId], {
|
||||
...pageData,
|
||||
content: newContent,
|
||||
});
|
||||
}
|
||||
}, 3000);
|
||||
|
||||
const handleActiveCommentEvent = (event) => {
|
||||
const { commentId, resolved } = event.detail;
|
||||
|
||||
|
||||
+976
-275
File diff suppressed because it is too large
Load Diff
@@ -23,7 +23,7 @@
|
||||
"migration:reset": "tsx src/database/migrate.ts down-to NO_MIGRATIONS",
|
||||
"migration:codegen": "kysely-codegen --dialect=postgres --camel-case --env-file=../../.env --out-file=./src/database/types/db.d.ts",
|
||||
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
|
||||
"pretest": "pnpm --filter @docmost/editor-ext build",
|
||||
"pretest": "pnpm --filter @docmost/editor-ext build && pnpm --filter @docmost/prosemirror-markdown build",
|
||||
"test": "jest",
|
||||
"test:int": "jest --config test/jest-integration.json",
|
||||
"test:watch": "jest --watch",
|
||||
@@ -43,6 +43,7 @@
|
||||
"@clickhouse/client": "^1.18.2",
|
||||
"@docmost/mcp": "workspace:*",
|
||||
"@docmost/pdf-inspector": "1.9.6",
|
||||
"@docmost/prosemirror-markdown": "workspace:*",
|
||||
"@fastify/cookie": "^11.0.2",
|
||||
"@fastify/multipart": "^10.0.0",
|
||||
"@fastify/static": "^9.1.3",
|
||||
@@ -175,7 +176,7 @@
|
||||
"/node_modules/"
|
||||
],
|
||||
"transform": {
|
||||
"happy-dom.+\\.js$": [
|
||||
"(happy-dom.+|prosemirror-markdown/build/.+)\\.js$": [
|
||||
"babel-jest",
|
||||
{
|
||||
"presets": [
|
||||
@@ -193,7 +194,7 @@
|
||||
"^.+\\.(t|j)sx?$": "ts-jest"
|
||||
},
|
||||
"transformIgnorePatterns": [
|
||||
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0)(@|/))"
|
||||
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@docmost/prosemirror-markdown)(@|/))"
|
||||
],
|
||||
"collectCoverageFrom": [
|
||||
"**/*.(t|j)s"
|
||||
@@ -204,7 +205,8 @@
|
||||
"^@docmost/db/(.*)$": "<rootDir>/database/$1",
|
||||
"^@docmost/transactional/(.*)$": "<rootDir>/integrations/transactional/$1",
|
||||
"^@docmost/ee/(.*)$": "<rootDir>/ee/$1",
|
||||
"^src/(.*)$": "<rootDir>/$1"
|
||||
"^src/(.*)$": "<rootDir>/$1",
|
||||
"^@tiptap/react$": "<rootDir>/../test/stubs/tiptap-react.js"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,7 +43,6 @@ import {
|
||||
Column,
|
||||
Status,
|
||||
addUniqueIdsToDoc,
|
||||
htmlToMarkdown,
|
||||
TransclusionSource,
|
||||
TransclusionReference,
|
||||
FootnoteReference,
|
||||
@@ -51,6 +50,7 @@ import {
|
||||
FootnoteDefinition,
|
||||
PageEmbed,
|
||||
} from '@docmost/editor-ext';
|
||||
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
|
||||
import { generateText, getSchema, JSONContent } from '@tiptap/core';
|
||||
import { generateHTML, generateJSON } from '../common/helpers/prosemirror/html';
|
||||
// @tiptap/html library works best for generating prosemirror json state but not HTML
|
||||
@@ -239,6 +239,10 @@ export function prosemirrorNodeToYElement(node: any): Y.XmlElement | Y.XmlText {
|
||||
}
|
||||
|
||||
export function jsonToMarkdown(tiptapJson: any): string {
|
||||
const html = jsonToHtml(tiptapJson);
|
||||
return htmlToMarkdown(html);
|
||||
// Direct ProseMirror JSON -> Markdown via the canonical converter
|
||||
// (`@docmost/prosemirror-markdown`) — no HTML intermediate, no second
|
||||
// editor-ext markdown layer. Same serializer as the page/space export and the
|
||||
// git-sync vault writer, so every server PM->MD path emits identical canonical
|
||||
// markdown (issue #345).
|
||||
return convertProseMirrorToMarkdown(tiptapJson);
|
||||
}
|
||||
|
||||
@@ -610,6 +610,63 @@ describe('AiAgentRolesService guards', () => {
|
||||
expect(repo.insert.mock.calls[0][0].name).toBe('Researcher (2)');
|
||||
});
|
||||
|
||||
it('createdRoles lists the installed role (no renamedTo when not renamed)', async () => {
|
||||
const { service } = makeImportService({});
|
||||
const res = await service.importFromCatalog('ws-1', 'u1', dto());
|
||||
expect(res.createdRoles).toEqual([
|
||||
{ slug: 'researcher', name: 'Researcher' },
|
||||
]);
|
||||
expect(res.skippedRoles).toEqual([]);
|
||||
});
|
||||
|
||||
it('createdRoles carries renamedTo on a rename', async () => {
|
||||
const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
|
||||
const { service } = makeImportService({ existing });
|
||||
const res = await service.importFromCatalog(
|
||||
'ws-1',
|
||||
'u1',
|
||||
dto({ conflict: 'rename' }),
|
||||
);
|
||||
expect(res.createdRoles).toEqual([
|
||||
{ slug: 'researcher', name: 'Researcher', renamedTo: 'Researcher (2)' },
|
||||
]);
|
||||
expect(res.skippedRoles).toEqual([]);
|
||||
});
|
||||
|
||||
it('skippedRoles: already-installed slug carries reason "already-installed"', async () => {
|
||||
const existing = [
|
||||
makeRow({
|
||||
id: 'r-existing',
|
||||
name: 'Old researcher',
|
||||
source: { slug: 'researcher', language: 'en', version: 1 } as never,
|
||||
}),
|
||||
];
|
||||
const { service } = makeImportService({ existing });
|
||||
const res = await service.importFromCatalog('ws-1', 'u1', dto());
|
||||
expect(res.skippedRoles).toEqual([
|
||||
{
|
||||
slug: 'researcher',
|
||||
name: 'Researcher',
|
||||
reason: 'already-installed',
|
||||
},
|
||||
]);
|
||||
expect(res.createdRoles).toEqual([]);
|
||||
});
|
||||
|
||||
it('skippedRoles: a name collision under conflict:skip carries reason "name-conflict"', async () => {
|
||||
const existing = [makeRow({ id: 'r-x', name: 'Researcher' })];
|
||||
const { service } = makeImportService({ existing });
|
||||
const res = await service.importFromCatalog(
|
||||
'ws-1',
|
||||
'u1',
|
||||
dto({ conflict: 'skip' }),
|
||||
);
|
||||
expect(res.skippedRoles).toEqual([
|
||||
{ slug: 'researcher', name: 'Researcher', reason: 'name-conflict' },
|
||||
]);
|
||||
expect(res.createdRoles).toEqual([]);
|
||||
});
|
||||
|
||||
it('dto.slugs filters; an unknown slug becomes an error entry', async () => {
|
||||
const { service, repo } = makeImportService({
|
||||
bundleRoles: [catalogRole()],
|
||||
@@ -677,6 +734,15 @@ describe('AiAgentRolesService guards', () => {
|
||||
// 'a' converged on the concurrent install (skip); 'b' imported; no errors.
|
||||
expect(res).toMatchObject({ created: 1, skipped: 1, renamed: 0 });
|
||||
expect(res.errors).toEqual([]);
|
||||
// The per-role list records 'a' as an already-installed skip (the UI reads
|
||||
// skippedRoles, not the counter, to render its plaque — assert the array,
|
||||
// not just the count).
|
||||
expect(res.skippedRoles).toContainEqual({
|
||||
slug: 'a',
|
||||
name: 'A',
|
||||
reason: 'already-installed',
|
||||
});
|
||||
expect(res.createdRoles.map((r) => r.slug)).toEqual(['b']);
|
||||
// Both inserts were attempted (the batch did not abort on the 23505).
|
||||
expect(repo.insert).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
@@ -305,6 +305,16 @@ export class AiAgentRolesService {
|
||||
skipped: number;
|
||||
renamed: number;
|
||||
errors: { slug: string; message: string }[];
|
||||
// Per-role lists alongside the counters (kept for back-compat). The redesigned
|
||||
// catalog UI needs the actual roles — which were created (and any rename) and
|
||||
// which were skipped and why — to render an inline result plaque with the
|
||||
// conflicting role's name and a "Rename & install" affordance.
|
||||
createdRoles: { slug: string; name: string; renamedTo?: string }[];
|
||||
skippedRoles: {
|
||||
slug: string;
|
||||
name: string;
|
||||
reason: 'name-conflict' | 'already-installed';
|
||||
}[];
|
||||
}> {
|
||||
const { file, versions } = await this.loadBundleById(
|
||||
dto.bundleId,
|
||||
@@ -312,6 +322,13 @@ export class AiAgentRolesService {
|
||||
);
|
||||
|
||||
const errors: { slug: string; message: string }[] = [];
|
||||
const createdRoles: { slug: string; name: string; renamedTo?: string }[] =
|
||||
[];
|
||||
const skippedRoles: {
|
||||
slug: string;
|
||||
name: string;
|
||||
reason: 'name-conflict' | 'already-installed';
|
||||
}[] = [];
|
||||
|
||||
// Resolve the selected catalog roles (honor dto.slugs; flag unknown ones).
|
||||
let selected = file.roles;
|
||||
@@ -351,16 +368,27 @@ export class AiAgentRolesService {
|
||||
// Already installed from the catalog in THIS language => skip (use
|
||||
// update-from-catalog). A different language of the same slug still imports.
|
||||
const installKey = `${role.slug}:${dto.language}`;
|
||||
const originalName = role.name.trim();
|
||||
if (installedKeys.has(installKey)) {
|
||||
skipped++;
|
||||
skippedRoles.push({
|
||||
slug: role.slug,
|
||||
name: originalName,
|
||||
reason: 'already-installed',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
let name = role.name.trim();
|
||||
let name = originalName;
|
||||
let didRename = false;
|
||||
if (takenNames.has(name.toLowerCase())) {
|
||||
if (dto.conflict === 'skip') {
|
||||
skipped++;
|
||||
skippedRoles.push({
|
||||
slug: role.slug,
|
||||
name: originalName,
|
||||
reason: 'name-conflict',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
// conflict === 'rename': find a free " (N)" suffix.
|
||||
@@ -380,6 +408,11 @@ export class AiAgentRolesService {
|
||||
});
|
||||
created++;
|
||||
if (didRename) renamed++;
|
||||
createdRoles.push({
|
||||
slug: role.slug,
|
||||
name: originalName,
|
||||
...(didRename ? { renamedTo: name } : {}),
|
||||
});
|
||||
takenNames.add(name.toLowerCase());
|
||||
installedKeys.add(installKey);
|
||||
} catch (err) {
|
||||
@@ -391,6 +424,11 @@ export class AiAgentRolesService {
|
||||
// skipped (already installed) and continue; do NOT abort or error.
|
||||
if (isSourceUniqueViolation(err)) {
|
||||
skipped++;
|
||||
skippedRoles.push({
|
||||
slug: role.slug,
|
||||
name: originalName,
|
||||
reason: 'already-installed',
|
||||
});
|
||||
installedKeys.add(installKey);
|
||||
continue;
|
||||
}
|
||||
@@ -407,7 +445,7 @@ export class AiAgentRolesService {
|
||||
}
|
||||
}
|
||||
|
||||
return { created, skipped, renamed, errors };
|
||||
return { created, skipped, renamed, errors, createdRoles, skippedRoles };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -52,7 +52,9 @@ import {
|
||||
INTERNAL_LINK_REGEX,
|
||||
extractPageSlugId,
|
||||
} from '../../../integrations/export/utils';
|
||||
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
||||
import { normalizeForeignMarkdown } from '../../../integrations/import/utils/foreign-markdown';
|
||||
import { WatcherService } from '../../watcher/watcher.service';
|
||||
import { sql } from 'kysely';
|
||||
import { TransclusionService } from '../transclusion/transclusion.service';
|
||||
@@ -1301,8 +1303,14 @@ export class PageService {
|
||||
|
||||
switch (format) {
|
||||
case 'markdown': {
|
||||
const html = await markdownToHtml(content as string);
|
||||
prosemirrorJson = htmlToJson(html as string);
|
||||
// Canonical markdown -> ProseMirror JSON directly via
|
||||
// `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate,
|
||||
// no editor-ext markdown layer. Foreign markdown surfaces the strict
|
||||
// parser rejects (GFM `[^id]` reference footnotes) are normalized to the
|
||||
// canonical inline form first.
|
||||
prosemirrorJson = await markdownToProseMirror(
|
||||
normalizeForeignMarkdown(content as string),
|
||||
);
|
||||
break;
|
||||
}
|
||||
case 'html': {
|
||||
|
||||
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
|
||||
import { InjectQueue } from '@nestjs/bullmq';
|
||||
import { QueueJob, QueueName } from '../../integrations/queue/constants';
|
||||
import { Queue } from 'bullmq';
|
||||
import { EnvironmentService } from '../../integrations/environment/environment.service';
|
||||
|
||||
/**
|
||||
* Thin snapshot of a page node carried inside domain events so the WebSocket
|
||||
@@ -112,48 +111,24 @@ export class PageListener {
|
||||
private readonly logger = new Logger(PageListener.name);
|
||||
|
||||
constructor(
|
||||
private readonly environmentService: EnvironmentService,
|
||||
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
|
||||
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
|
||||
) {}
|
||||
|
||||
@OnEvent(EventName.PAGE_CREATED)
|
||||
async handlePageCreated(event: PageEvent) {
|
||||
const { pageIds, workspaceId } = event;
|
||||
if (this.isTypesense()) {
|
||||
await this.searchQueue.add(QueueJob.PAGE_CREATED, {
|
||||
pageIds,
|
||||
});
|
||||
}
|
||||
|
||||
await this.aiQueue.add(QueueJob.PAGE_CREATED, { pageIds, workspaceId });
|
||||
}
|
||||
|
||||
@OnEvent(EventName.PAGE_UPDATED)
|
||||
async handlePageUpdated(event: PageEvent) {
|
||||
const { pageIds } = event;
|
||||
|
||||
await this.searchQueue.add(QueueJob.PAGE_UPDATED, { pageIds });
|
||||
}
|
||||
|
||||
@OnEvent(EventName.PAGE_DELETED)
|
||||
async handlePageDeleted(event: PageEvent) {
|
||||
const { pageIds, workspaceId } = event;
|
||||
if (this.isTypesense()) {
|
||||
await this.searchQueue.add(QueueJob.PAGE_DELETED, { pageIds });
|
||||
}
|
||||
|
||||
await this.aiQueue.add(QueueJob.PAGE_DELETED, { pageIds, workspaceId });
|
||||
}
|
||||
|
||||
@OnEvent(EventName.PAGE_SOFT_DELETED)
|
||||
async handlePageSoftDeleted(event: PageEvent) {
|
||||
const { pageIds, workspaceId } = event;
|
||||
|
||||
if (this.isTypesense()) {
|
||||
await this.searchQueue.add(QueueJob.PAGE_SOFT_DELETED, { pageIds });
|
||||
}
|
||||
|
||||
await this.aiQueue.add(QueueJob.PAGE_SOFT_DELETED, {
|
||||
pageIds,
|
||||
workspaceId,
|
||||
@@ -163,14 +138,6 @@ export class PageListener {
|
||||
@OnEvent(EventName.PAGE_RESTORED)
|
||||
async handlePageRestored(event: PageEvent) {
|
||||
const { pageIds, workspaceId } = event;
|
||||
if (this.isTypesense()) {
|
||||
await this.searchQueue.add(QueueJob.PAGE_RESTORED, { pageIds });
|
||||
}
|
||||
|
||||
await this.aiQueue.add(QueueJob.PAGE_RESTORED, { pageIds, workspaceId });
|
||||
}
|
||||
|
||||
isTypesense(): boolean {
|
||||
return this.environmentService.getSearchDriver() === 'typesense';
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
|
||||
import { InjectQueue } from '@nestjs/bullmq';
|
||||
import { QueueJob, QueueName } from '../../integrations/queue/constants';
|
||||
import { Queue } from 'bullmq';
|
||||
import { EnvironmentService } from '../../integrations/environment/environment.service';
|
||||
|
||||
export class SpaceEvent {
|
||||
spaceId: string;
|
||||
@@ -15,22 +14,12 @@ export class SpaceListener {
|
||||
private readonly logger = new Logger(SpaceListener.name);
|
||||
|
||||
constructor(
|
||||
private readonly environmentService: EnvironmentService,
|
||||
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
|
||||
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
|
||||
) {}
|
||||
|
||||
@OnEvent(EventName.SPACE_DELETED)
|
||||
async handleSpaceDeleted(event: SpaceEvent) {
|
||||
const { spaceId } = event;
|
||||
if (this.isTypesense()) {
|
||||
await this.searchQueue.add(QueueJob.SPACE_DELETED, { spaceId });
|
||||
}
|
||||
|
||||
await this.aiQueue.add(QueueJob.SPACE_DELETED, { spaceId });
|
||||
}
|
||||
|
||||
isTypesense(): boolean {
|
||||
return this.environmentService.getSearchDriver() === 'typesense';
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,7 +4,6 @@ import { EventName } from '../../common/events/event.contants';
|
||||
import { InjectQueue } from '@nestjs/bullmq';
|
||||
import { QueueJob, QueueName } from '../../integrations/queue/constants';
|
||||
import { Queue } from 'bullmq';
|
||||
import { EnvironmentService } from '../../integrations/environment/environment.service';
|
||||
|
||||
export class WorkspaceEvent {
|
||||
workspaceId: string;
|
||||
@@ -15,22 +14,12 @@ export class WorkspaceListener {
|
||||
private readonly logger = new Logger(WorkspaceListener.name);
|
||||
|
||||
constructor(
|
||||
private readonly environmentService: EnvironmentService,
|
||||
@InjectQueue(QueueName.SEARCH_QUEUE) private searchQueue: Queue,
|
||||
@InjectQueue(QueueName.AI_QUEUE) private aiQueue: Queue,
|
||||
) {}
|
||||
|
||||
@OnEvent(EventName.WORKSPACE_DELETED)
|
||||
async handlePageDeleted(event: WorkspaceEvent) {
|
||||
const { workspaceId } = event;
|
||||
if (this.isTypesense()) {
|
||||
await this.searchQueue.add(QueueJob.WORKSPACE_DELETED, { workspaceId });
|
||||
}
|
||||
|
||||
await this.aiQueue.add(QueueJob.WORKSPACE_DELETED, { workspaceId });
|
||||
}
|
||||
|
||||
isTypesense(): boolean {
|
||||
return this.environmentService.getSearchDriver() === 'typesense';
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,10 @@ const migrator = new Migrator({
|
||||
path,
|
||||
migrationFolder,
|
||||
}),
|
||||
// Match the startup auto-migrator (migration.service.ts): a back-dated
|
||||
// migration from a long-lived branch must be applied, not rejected as
|
||||
// "corrupted migrations" (incident #361). See that file for the full rationale.
|
||||
allowUnorderedMigrations: true,
|
||||
});
|
||||
|
||||
run(db, migrator, migrationFolder);
|
||||
|
||||
@@ -19,6 +19,16 @@ export class MigrationService {
|
||||
path,
|
||||
migrationFolder: path.join(__dirname, '..', 'migrations'),
|
||||
}),
|
||||
// A long-lived branch can add a migration whose timestamped filename sorts
|
||||
// BEFORE migrations already applied in prod (e.g. #234's 20260627 landing
|
||||
// after 20260704 was live). With the default (ordered) setting the startup
|
||||
// migrator then sees "corrupted migrations" — the applied set is no longer a
|
||||
// prefix of the sorted list — throws, and the app crash-loops on boot
|
||||
// (incident #361: 502s for ~11 min). allowUnorderedMigrations runs any
|
||||
// not-yet-applied migration regardless of filename order, so a back-dated
|
||||
// migration is applied instead of bricking startup. A CI order-gate still
|
||||
// discourages back-dating; this is the runtime safety net.
|
||||
allowUnorderedMigrations: true,
|
||||
});
|
||||
|
||||
const { error, results } = await migrator.migrateToLatest();
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
import { readFileSync } from 'fs';
|
||||
import { streamText, Output } from 'ai';
|
||||
import { MockLanguageModelV3, simulateReadableStream } from 'ai/test';
|
||||
|
||||
/**
|
||||
* Regression tests for patches/ai@6.0.134.patch (server heap OOM on long
|
||||
* autonomous agent runs, #184).
|
||||
*
|
||||
* Unpatched ai@6.0.134 substitutes the default text() output strategy even
|
||||
* when the caller passes NO `output` option. Its createOutputTransformStream
|
||||
* then accumulates the ENTIRE turn text and, on EVERY text-delta, enqueues a
|
||||
* flat snapshot of all text so far as `partialOutput` (O(n^2) memory). Those
|
||||
* snapshots pile up in the never-consumed leftover tee() branch of
|
||||
* DefaultStreamTextResult.baseStream, which is what OOM'd production during a
|
||||
* ~28k-chunk agent turn. The pnpm patch skips partialOutput production
|
||||
* entirely when no output strategy was requested, while keeping per-delta
|
||||
* streaming granularity.
|
||||
*/
|
||||
describe('ai@6.0.134 pnpm patch: no partialOutput accumulation without an output strategy', () => {
|
||||
const makeModel = () =>
|
||||
new MockLanguageModelV3({
|
||||
doStream: async () => ({
|
||||
stream: simulateReadableStream({
|
||||
chunks: [
|
||||
{ type: 'stream-start' as const, warnings: [] },
|
||||
{ type: 'text-start' as const, id: '1' },
|
||||
{ type: 'text-delta' as const, id: '1', delta: 'Hello' },
|
||||
{ type: 'text-delta' as const, id: '1', delta: ', ' },
|
||||
{ type: 'text-delta' as const, id: '1', delta: 'world!' },
|
||||
{ type: 'text-end' as const, id: '1' },
|
||||
{
|
||||
type: 'finish' as const,
|
||||
finishReason: { unified: 'stop' as const, raw: 'stop' },
|
||||
usage: {
|
||||
inputTokens: {
|
||||
total: 1,
|
||||
noCache: undefined,
|
||||
cacheRead: undefined,
|
||||
cacheWrite: undefined,
|
||||
},
|
||||
outputTokens: { total: 1, text: 1, reasoning: undefined },
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
}),
|
||||
});
|
||||
|
||||
it('preserves per-delta streaming granularity in textStream', async () => {
|
||||
const result = streamText({ model: makeModel(), prompt: 'hi' });
|
||||
|
||||
const deltas: string[] = [];
|
||||
for await (const delta of result.textStream) {
|
||||
deltas.push(delta);
|
||||
}
|
||||
|
||||
// The patch must NOT coalesce or drop deltas: three model deltas arrive
|
||||
// as three separate textStream chunks.
|
||||
expect(deltas).toEqual(['Hello', ', ', 'world!']);
|
||||
});
|
||||
|
||||
it('emits NO partialOutput values when the caller did not request an output strategy', async () => {
|
||||
const result = streamText({ model: makeModel(), prompt: 'hi' });
|
||||
|
||||
// Fully consume the primary stream first (mirrors production usage).
|
||||
for await (const _ of result.textStream) {
|
||||
// drain
|
||||
}
|
||||
|
||||
const partials: unknown[] = [];
|
||||
for await (const partial of result.experimental_partialOutputStream) {
|
||||
partials.push(partial);
|
||||
}
|
||||
|
||||
// TRIPWIRE: on unpatched ai@6.0.134 the default text() output strategy
|
||||
// yields one cumulative partial per text-delta here (['Hello', 'Hello, ',
|
||||
// 'Hello, world!']). An empty stream proves the patch is applied and no
|
||||
// cumulative snapshots are being produced (and thus none can pile up in
|
||||
// the leftover internal tee branch).
|
||||
expect(partials).toEqual([]);
|
||||
});
|
||||
|
||||
it('preserves cumulative partialOutput when the caller DOES request an output strategy', async () => {
|
||||
// PRESERVE-BRANCH GUARD: the patch only short-circuits partialOutput when
|
||||
// `output == null`. When an output strategy IS set (here Output.text()),
|
||||
// createOutputTransformStream must fall through to the ORIGINAL code path
|
||||
// and keep publishing cumulative snapshots, so object/text-output consumers
|
||||
// behave byte-identically to unpatched ai. A careless re-port that routed
|
||||
// output-set calls into the skip branch would leave partialOutput empty and
|
||||
// silently break those consumers — this test is the tripwire for that.
|
||||
const result = streamText({
|
||||
model: makeModel(),
|
||||
prompt: 'hi',
|
||||
experimental_output: Output.text(),
|
||||
});
|
||||
|
||||
// Drain the primary stream fully and accumulate the complete output text.
|
||||
let fullText = '';
|
||||
for await (const delta of result.textStream) {
|
||||
fullText += delta;
|
||||
}
|
||||
|
||||
const partials: string[] = [];
|
||||
for await (const partial of result.experimental_partialOutputStream) {
|
||||
partials.push(partial);
|
||||
}
|
||||
|
||||
// With a strategy set, partialOutput must be PRESERVED (non-empty) and
|
||||
// cumulative: the last emitted partial equals the full accumulated text.
|
||||
expect(partials.length).toBeGreaterThan(0);
|
||||
expect(partials[partials.length - 1]).toBe(fullText);
|
||||
expect(fullText).toBe('Hello, world!');
|
||||
});
|
||||
|
||||
it('both installed dist builds (CJS and ESM) carry the patch marker', () => {
|
||||
// Secondary guard: pins the patch to BOTH bundles the SDK ships, since
|
||||
// the NestJS server consumes CJS while other tooling may load ESM.
|
||||
const cjsPath = require.resolve('ai');
|
||||
const mjsPath = cjsPath.replace(/index\.js$/, 'index.mjs');
|
||||
expect(cjsPath).toMatch(/index\.js$/);
|
||||
expect(readFileSync(cjsPath, 'utf8')).toContain('PATCH(docmost');
|
||||
expect(readFileSync(mjsPath, 'utf8')).toContain('PATCH(docmost');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,145 @@
|
||||
// export.service.ts imports the ESM-only @sindresorhus/slugify (not in jest's
|
||||
// transform allowlist). It is irrelevant to the markdown-serialization path under
|
||||
// test (only used for page-mention link slugs on the DB path), so it is mocked
|
||||
// out to keep the module graph loadable under ts-jest (mirrors the import specs).
|
||||
jest.mock('@sindresorhus/slugify', () => ({
|
||||
__esModule: true,
|
||||
default: (input: string) => String(input),
|
||||
}));
|
||||
|
||||
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
|
||||
import { ExportService } from './export.service';
|
||||
import { ExportFormat } from './dto/export-dto';
|
||||
|
||||
/**
|
||||
* STEP 1 golden test for issue #345: server MARKDOWN export runs DIRECTLY through
|
||||
* the canonical converter (`convertProseMirrorToMarkdown`) — no HTML intermediate
|
||||
* and no `@docmost/editor-ext` markdown layer — so the emitted markdown is in the
|
||||
* canonical package forms and is byte-identical to the git-sync vault body.
|
||||
*
|
||||
* These are the goldens the swap has to satisfy: they assert the CANONICAL
|
||||
* surface (callout `> [!type]`, inline footnote `^[…]`, lossless image
|
||||
* `<!--img …-->`) rather than the old editor-ext forms (`:::type`, `[^id]`,
|
||||
* lossy ``).
|
||||
*
|
||||
* `exportPage(..., singlePage=false)` takes no DB path (no mention rewriting), so
|
||||
* the service is constructed with null collaborators and only the pure
|
||||
* PM -> Markdown path is exercised.
|
||||
*/
|
||||
|
||||
function makeService(): ExportService {
|
||||
return new ExportService(
|
||||
null as any, // pageRepo
|
||||
null as any, // pagePermissionRepo
|
||||
null as any, // db
|
||||
null as any, // storageService
|
||||
null as any, // environmentService
|
||||
null as any, // domainService
|
||||
);
|
||||
}
|
||||
|
||||
// A representative page exercising the node types whose canonical markdown form
|
||||
// changed with the move off the editor-ext layer: callout, inline footnote, and a
|
||||
// lossless image carrying width/align attrs that the old layer dropped.
|
||||
const REPRESENTATIVE_DOC = {
|
||||
type: 'doc',
|
||||
content: [
|
||||
{
|
||||
type: 'paragraph',
|
||||
content: [
|
||||
{ type: 'text', text: 'Body ' },
|
||||
{ type: 'footnoteReference', attrs: { id: 'fn-1' } },
|
||||
{ type: 'text', text: ' end.' },
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'callout',
|
||||
attrs: { type: 'info', icon: null },
|
||||
content: [
|
||||
{
|
||||
type: 'paragraph',
|
||||
content: [{ type: 'text', text: 'Heads up' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'image',
|
||||
attrs: {
|
||||
src: '/files/pic.png',
|
||||
alt: 'Pic',
|
||||
width: 320,
|
||||
align: 'left',
|
||||
},
|
||||
},
|
||||
{
|
||||
type: 'footnotesList',
|
||||
content: [
|
||||
{
|
||||
type: 'footnoteDefinition',
|
||||
attrs: { id: 'fn-1' },
|
||||
content: [
|
||||
{
|
||||
type: 'paragraph',
|
||||
content: [{ type: 'text', text: 'the note' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
describe('ExportService — markdown export via the canonical converter (#345)', () => {
|
||||
it('emits canonical callout, inline footnote and lossless image forms', async () => {
|
||||
const service = makeService();
|
||||
const md = (await service.exportPage(ExportFormat.Markdown, {
|
||||
title: '',
|
||||
content: REPRESENTATIVE_DOC,
|
||||
} as any)) as string;
|
||||
|
||||
// Callout: Obsidian `> [!type]`, NOT the legacy `:::type`.
|
||||
expect(md).toContain('> [!info]');
|
||||
expect(md).not.toContain(':::');
|
||||
|
||||
// Inline footnote: `^[…]`, NOT the reference `[^id]` form.
|
||||
expect(md).toContain('^[the note]');
|
||||
expect(md).not.toMatch(/\[\^/);
|
||||
|
||||
// Lossless image: trailing `<!--img …-->` carrying the dropped attrs.
|
||||
expect(md).toContain('');
|
||||
expect(md).toContain('<!--img');
|
||||
expect(md).toContain('"width":"320"');
|
||||
expect(md).toContain('"align":"left"');
|
||||
});
|
||||
|
||||
it('export body is byte-identical to the git-sync vault serializer (export == vault)', async () => {
|
||||
const service = makeService();
|
||||
// A title-less page: exportPage prepends NO heading, so the whole output is
|
||||
// the page BODY — exactly what git-sync serializes (git-sync stores the title
|
||||
// in frontmatter / the filename, never as an in-body H1).
|
||||
const exported = (await service.exportPage(ExportFormat.Markdown, {
|
||||
title: '',
|
||||
content: REPRESENTATIVE_DOC,
|
||||
} as any)) as string;
|
||||
|
||||
// The git-sync vault writer feeds this SAME converter (git-sync
|
||||
// `stabilizePageBody` = convertProseMirrorToMarkdown(content) at the
|
||||
// fixpoint). For an already-stable doc the single pass IS the fixpoint, so
|
||||
// the two are byte-identical by construction — assert it.
|
||||
const vaultBody = convertProseMirrorToMarkdown(REPRESENTATIVE_DOC);
|
||||
expect(exported).toBe(vaultBody);
|
||||
});
|
||||
|
||||
it('prepends the page title as an H1 heading (the one documented export/vault delta)', async () => {
|
||||
const service = makeService();
|
||||
const md = (await service.exportPage(ExportFormat.Markdown, {
|
||||
title: 'My Page',
|
||||
content: { type: 'doc', content: [] },
|
||||
} as any)) as string;
|
||||
|
||||
// Export makes standalone files, so it prepends the title as an H1. This is
|
||||
// the ONE deliberate difference from the vault body (which carries the title
|
||||
// in frontmatter). The body below the heading still serializes canonically.
|
||||
expect(md.startsWith('# My Page')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -37,7 +37,7 @@ import {
|
||||
getAttachmentIds,
|
||||
getProsemirrorContent,
|
||||
} from '../../common/helpers/prosemirror/utils';
|
||||
import { htmlToMarkdown } from '@docmost/editor-ext';
|
||||
import { convertProseMirrorToMarkdown } from '@docmost/prosemirror-markdown';
|
||||
|
||||
type AllowedAttachment = { id: string; fileName: string; filePath: string };
|
||||
|
||||
@@ -79,9 +79,8 @@ export class ExportService {
|
||||
prosemirrorJson.content.unshift(titleNode);
|
||||
}
|
||||
|
||||
const pageHtml = jsonToHtml(prosemirrorJson);
|
||||
|
||||
if (format === ExportFormat.HTML) {
|
||||
const pageHtml = jsonToHtml(prosemirrorJson);
|
||||
return `<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
@@ -92,11 +91,14 @@ export class ExportService {
|
||||
}
|
||||
|
||||
if (format === ExportFormat.Markdown) {
|
||||
const newPageHtml = pageHtml.replace(
|
||||
/<colgroup[^>]*>[\s\S]*?<\/colgroup>/gim,
|
||||
'',
|
||||
);
|
||||
return htmlToMarkdown(newPageHtml);
|
||||
// Direct ProseMirror JSON -> Markdown via the canonical converter
|
||||
// (`@docmost/prosemirror-markdown`). This is the SAME serializer the
|
||||
// git-sync vault writer feeds (see git-sync `stabilizePageBody`), so an
|
||||
// exported page body is byte-identical to its vault representation — no
|
||||
// HTML intermediate, no second markdown layer, no format drift (issue
|
||||
// #345). The old `<colgroup>` scrub is gone with the HTML step: the
|
||||
// converter emits GFM tables directly and never produces `<colgroup>`.
|
||||
return convertProseMirrorToMarkdown(prosemirrorJson);
|
||||
}
|
||||
|
||||
return;
|
||||
|
||||
+144
-77
@@ -17,6 +17,22 @@ jest.mock('image-dimensions', () => ({
|
||||
__esModule: true,
|
||||
imageDimensionsFromData: () => undefined,
|
||||
}));
|
||||
// FileImportTaskService -> PageService -> collaboration.gateway ->
|
||||
// metrics.registry imports `prom-client`, which is not resolvable in this
|
||||
// workspace's node_modules (types-only stub, no runtime entry). Metrics are
|
||||
// disabled on this path, so a virtual no-op mock keeps the module graph loadable.
|
||||
jest.mock(
|
||||
'prom-client',
|
||||
() => ({
|
||||
collectDefaultMetrics: () => undefined,
|
||||
Registry: class {},
|
||||
Histogram: class {},
|
||||
Gauge: class {},
|
||||
Counter: class {},
|
||||
Summary: class {},
|
||||
}),
|
||||
{ virtual: true },
|
||||
);
|
||||
|
||||
import { promises as fs } from 'fs';
|
||||
import * as os from 'os';
|
||||
@@ -26,14 +42,17 @@ import { ImportService } from './import.service';
|
||||
|
||||
/**
|
||||
* Binding test for issue #228 / review #5: FileImportTaskService.processGenericImport
|
||||
* is a NON-editor write path (markdownToHtml -> processHTML -> JSON, never runs
|
||||
* footnoteSyncPlugin), so it canonicalizes footnotes before persisting. This pins
|
||||
* that binding — the same one import.service has a spec for — which previously had
|
||||
* NO spec at all.
|
||||
* is a NON-editor write path, so a zip-imported `.md` page ends up with canonical
|
||||
* footnotes before persisting: ordered by first reference, reused refs deduped,
|
||||
* orphan definitions dropped.
|
||||
*
|
||||
* The markdown -> HTML -> ProseMirror conversion is REAL (a real ImportService,
|
||||
* its createYdoc stubbed); the filesystem is a real temp dir with one .md file;
|
||||
* the DB transaction is stubbed to capture the persisted page content.
|
||||
* Since #345 the `.md` parse runs `normalizeForeignMarkdown` ->
|
||||
* `markdownToProseMirror` -> `jsonToHtml` (feeding the shared HTML attachment /
|
||||
* link pipeline) -> `processHTML` -> `canonicalizeFootnotes`. The parser assigns
|
||||
* fresh `fn-*` ids, so we assert by definition BODY order rather than the source
|
||||
* labels. The conversion is REAL (a real ImportService, its createYdoc stubbed);
|
||||
* the filesystem is a real temp dir with one .md file; the DB transaction is
|
||||
* stubbed to capture the persisted page content.
|
||||
*/
|
||||
|
||||
// Out-of-order references (c, a, b), a REUSED reference ([^a] twice), and an
|
||||
@@ -49,13 +68,14 @@ const MARKDOWN = [
|
||||
'[^z]: orphan note',
|
||||
].join('\n');
|
||||
|
||||
function footnoteListIds(content: any): string[] {
|
||||
/** Definition body texts of the (single) footnotesList, in list order. */
|
||||
function footnoteListBodies(content: any): string[] {
|
||||
const list = (content?.content ?? []).find(
|
||||
(n: any) => n.type === 'footnotesList',
|
||||
);
|
||||
return (list?.content ?? [])
|
||||
.filter((n: any) => n.type === 'footnoteDefinition')
|
||||
.map((n: any) => n.attrs?.id);
|
||||
.map((n: any) => n.content?.[0]?.content?.[0]?.text);
|
||||
}
|
||||
|
||||
// A permissive chainable stub for the spaces lookup (selectFrom(...).select(...)
|
||||
@@ -71,80 +91,127 @@ function chainable(result: any): any {
|
||||
return proxy;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one markdown file through the REAL zip-import pipeline
|
||||
* (`processGenericImport` -> `markdownToProseMirror` -> `jsonToHtml` ->
|
||||
* `processHTML`/`htmlToJson`) and return the persisted page `content`. This is
|
||||
* the server-specific PM->HTML->PM hop that the package's own PM<->MD tests do
|
||||
* NOT cover.
|
||||
*/
|
||||
async function runZipImport(markdown: string): Promise<any> {
|
||||
const extractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fit-canon-'));
|
||||
await fs.writeFile(path.join(extractDir, 'note.md'), markdown, 'utf-8');
|
||||
|
||||
const importService = new ImportService(
|
||||
{} as any,
|
||||
{} as any,
|
||||
{} as any,
|
||||
{} as any,
|
||||
);
|
||||
jest
|
||||
.spyOn(importService as any, 'createYdoc')
|
||||
.mockResolvedValue(Buffer.from([]) as any);
|
||||
|
||||
let captured: any = null;
|
||||
const trx = {
|
||||
insertInto: (table: string) => ({
|
||||
values: (v: any) => {
|
||||
if (table === 'pages') captured = v;
|
||||
return { execute: async () => {} };
|
||||
},
|
||||
}),
|
||||
};
|
||||
const db: any = {
|
||||
selectFrom: () => chainable({ slug: 'space-slug' }),
|
||||
transaction: () => ({ execute: (fn: any) => fn(trx) }),
|
||||
};
|
||||
|
||||
const importAttachmentService = {
|
||||
processAttachments: async ({ html }: any) => html,
|
||||
};
|
||||
const service = new FileImportTaskService(
|
||||
{} as any, // storageService
|
||||
importService as any,
|
||||
{ nextPagePosition: async () => 'a0' } as any,
|
||||
{ insertBacklink: jest.fn() } as any,
|
||||
db,
|
||||
importAttachmentService as any,
|
||||
{ emit: jest.fn() } as any,
|
||||
{ logBatchWithContext: jest.fn() } as any,
|
||||
);
|
||||
|
||||
const fileTask: any = {
|
||||
id: 'task-1',
|
||||
source: 'generic',
|
||||
spaceId: 'space-1',
|
||||
workspaceId: 'ws-1',
|
||||
creatorId: 'user-1',
|
||||
};
|
||||
|
||||
try {
|
||||
await service.processGenericImport({ extractDir, fileTask });
|
||||
expect(captured).toBeTruthy();
|
||||
return captured.content;
|
||||
} finally {
|
||||
await fs.rm(extractDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
/** Find the first node of a given type anywhere in a PM content tree. */
|
||||
function findFirst(node: any, type: string): any {
|
||||
if (!node || typeof node !== 'object') return null;
|
||||
if (node.type === type) return node;
|
||||
for (const child of node.content ?? []) {
|
||||
const hit = findFirst(child, type);
|
||||
if (hit) return hit;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
describe('FileImportTaskService.processGenericImport — footnote canonicalization (#228)', () => {
|
||||
it('orders footnotes by first reference, dedupes reuse, and drops orphans on zip import', async () => {
|
||||
const extractDir = await fs.mkdtemp(path.join(os.tmpdir(), 'fit-canon-'));
|
||||
await fs.writeFile(path.join(extractDir, 'note.md'), MARKDOWN, 'utf-8');
|
||||
|
||||
// Real ImportService for the html -> JSON conversion; stub the yjs encode.
|
||||
const importService = new ImportService(
|
||||
{} as any,
|
||||
{} as any,
|
||||
{} as any,
|
||||
{} as any,
|
||||
const content = await runZipImport(MARKDOWN);
|
||||
// Definitions ordered by FIRST REFERENCE (C, A, B), NOT the markdown
|
||||
// definition order (A, B, C). Ids are the parser's fresh `fn-*`, so pin
|
||||
// the BODIES.
|
||||
expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
|
||||
// Orphan [^z] dropped; reused [^a] collapses to one definition; one list.
|
||||
expect(footnoteListBodies(content)).not.toContain('orphan note');
|
||||
const lists = (content.content ?? []).filter(
|
||||
(n: any) => n.type === 'footnotesList',
|
||||
);
|
||||
jest
|
||||
.spyOn(importService as any, 'createYdoc')
|
||||
.mockResolvedValue(Buffer.from([]) as any);
|
||||
expect(lists).toHaveLength(1);
|
||||
expect(
|
||||
footnoteListBodies(content).filter((b) => b === 'note A'),
|
||||
).toHaveLength(1);
|
||||
});
|
||||
|
||||
let captured: any = null;
|
||||
const trx = {
|
||||
insertInto: (table: string) => ({
|
||||
values: (v: any) => {
|
||||
if (table === 'pages') captured = v;
|
||||
return { execute: async () => {} };
|
||||
},
|
||||
}),
|
||||
};
|
||||
const db: any = {
|
||||
selectFrom: () => chainable({ slug: 'space-slug' }),
|
||||
transaction: () => ({ execute: (fn: any) => fn(trx) }),
|
||||
};
|
||||
// #345 F4: the zip path routes markdown through jsonToHtml -> processHTML ->
|
||||
// htmlToJson (the shared HTML attachment pipeline). #345's headline is LOSSLESS
|
||||
// image width/align via the `<!--img {...}-->` comment; a callout carries its
|
||||
// `type`. This asserts those survive the PM->HTML->PM hop — the one hop the
|
||||
// package's PM<->MD suite does not exercise.
|
||||
it('preserves image width/align and callout type through the PM->HTML->PM hop', async () => {
|
||||
const md = [
|
||||
'# Doc',
|
||||
'',
|
||||
' <!--img {"width":"320","align":"left"}-->',
|
||||
'',
|
||||
':::warning',
|
||||
'Careful now.',
|
||||
':::',
|
||||
].join('\n');
|
||||
|
||||
const importAttachmentService = {
|
||||
processAttachments: async ({ html }: any) => html,
|
||||
};
|
||||
const backlinkRepo = { insertBacklink: jest.fn() };
|
||||
const eventEmitter = { emit: jest.fn() };
|
||||
const auditService = { logBatchWithContext: jest.fn() };
|
||||
const content = await runZipImport(md);
|
||||
|
||||
const pageService = { nextPagePosition: async () => 'a0' };
|
||||
const image = findFirst(content, 'image');
|
||||
expect(image).toBeTruthy();
|
||||
// The lossless sizing/alignment must survive the HTML hop.
|
||||
expect(String(image.attrs?.width)).toBe('320');
|
||||
expect(image.attrs?.align).toBe('left');
|
||||
|
||||
const service = new FileImportTaskService(
|
||||
{} as any, // storageService
|
||||
importService as any,
|
||||
pageService as any,
|
||||
backlinkRepo as any,
|
||||
db,
|
||||
importAttachmentService as any,
|
||||
eventEmitter as any,
|
||||
auditService as any,
|
||||
);
|
||||
|
||||
const fileTask: any = {
|
||||
id: 'task-1',
|
||||
source: 'generic',
|
||||
spaceId: 'space-1',
|
||||
workspaceId: 'ws-1',
|
||||
creatorId: 'user-1',
|
||||
};
|
||||
|
||||
try {
|
||||
await service.processGenericImport({ extractDir, fileTask });
|
||||
|
||||
expect(captured).toBeTruthy();
|
||||
const content = captured.content;
|
||||
// Reference order is c, a, b (NOT the markdown definition order a, b, c).
|
||||
expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']);
|
||||
// Orphan [^z] dropped; reused [^a] collapses to one definition; one list.
|
||||
expect(footnoteListIds(content)).not.toContain('z');
|
||||
const lists = (content.content ?? []).filter(
|
||||
(n: any) => n.type === 'footnotesList',
|
||||
);
|
||||
expect(lists).toHaveLength(1);
|
||||
expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1);
|
||||
} finally {
|
||||
await fs.rm(extractDir, { recursive: true, force: true });
|
||||
}
|
||||
const callout = findFirst(content, 'callout');
|
||||
expect(callout).toBeTruthy();
|
||||
expect(callout.attrs?.type).toBe('warning');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
import { Inject, Injectable, Logger } from '@nestjs/common';
|
||||
import * as path from 'path';
|
||||
import { jsonToText } from '../../../collaboration/collaboration.util';
|
||||
import {
|
||||
jsonToHtml,
|
||||
jsonToText,
|
||||
} from '../../../collaboration/collaboration.util';
|
||||
import { InjectKysely } from 'nestjs-kysely';
|
||||
import { KyselyDB } from '@docmost/db/types/kysely.types';
|
||||
import {
|
||||
@@ -18,9 +21,11 @@ import { generateSlugId } from '../../../common/helpers';
|
||||
import { v7 } from 'uuid';
|
||||
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
|
||||
import { FileTask, InsertablePage } from '@docmost/db/types/entity.types';
|
||||
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
||||
import { getProsemirrorContent } from '../../../common/helpers/prosemirror/utils';
|
||||
import { formatImportHtml } from '../utils/import-formatter';
|
||||
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
|
||||
import {
|
||||
buildAttachmentCandidates,
|
||||
collectMarkdownAndHtmlFiles,
|
||||
@@ -461,7 +466,18 @@ export class FileImportTaskService {
|
||||
content = await fs.readFile(absPath, 'utf-8');
|
||||
|
||||
if (page.fileExtension.toLowerCase() === '.md') {
|
||||
content = await markdownToHtml(content);
|
||||
// Parse markdown with the single canonical converter
|
||||
// (`@docmost/prosemirror-markdown`), after normalizing foreign
|
||||
// reference footnotes, then serialize to HTML so the shared HTML
|
||||
// pipeline below (processAttachments + formatImportHtml +
|
||||
// processHTML) keeps handling `.md` and `.html` imports
|
||||
// uniformly. The markdown PARSE no longer goes through the
|
||||
// editor-ext markdown layer (issue #345) — the drift source is
|
||||
// gone. The PM -> HTML -> PM hop that follows is lossless
|
||||
// plumbing for attachment/link resolution, NOT a second parse.
|
||||
content = jsonToHtml(
|
||||
await markdownToProseMirror(normalizeForeignMarkdown(content)),
|
||||
);
|
||||
}
|
||||
} catch (err: any) {
|
||||
if (err?.code === 'ENOENT') {
|
||||
@@ -500,10 +516,12 @@ export class FileImportTaskService {
|
||||
this.importService.extractTitleAndRemoveHeading(pmState);
|
||||
|
||||
// Canonicalize footnote topology on this non-editor write path
|
||||
// (markdownToHtml/processHTML never runs footnoteSyncPlugin), so a
|
||||
// zip-imported page's footnotes are reference-ordered, deduped, and
|
||||
// (the HTML pipeline's processHTML never runs footnoteSyncPlugin), so
|
||||
// a zip-imported page's footnotes are reference-ordered, deduped, and
|
||||
// orphan-free like the editor's invariant (issue #228). Pure +
|
||||
// idempotent + shape-safe; a footnote-free doc is unchanged.
|
||||
// idempotent + shape-safe; a footnote-free doc is unchanged. (For a
|
||||
// `.md` file the package parser already yields canonical footnotes,
|
||||
// so this is a no-op there.)
|
||||
// (Future consolidation, architecture B: like import.service, this
|
||||
// path persists directly rather than via PageService — a shared
|
||||
// "prepare JSON for persist" helper would centralize this call.)
|
||||
|
||||
+27
-31
@@ -12,13 +12,19 @@ import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
|
||||
/**
|
||||
* Integration-ish test for the USER-FACING markdown import path
|
||||
* (`ImportService.importPage`). It exercises the REAL markdown -> HTML -> JSON
|
||||
* conversion and asserts that the stored page content has its footnotes
|
||||
* canonicalized — the gap that issue #228 fixes: the import path builds
|
||||
* ProseMirror JSON directly (never running the editor's footnoteSyncPlugin), so
|
||||
* before this wiring the stored footnotes kept the markdown's physical
|
||||
* definition order (out of order vs. references), retained orphan definitions,
|
||||
* and did not collapse reused references.
|
||||
* (`ImportService.importPage`). It exercises the REAL markdown -> ProseMirror
|
||||
* conversion and asserts the stored page's footnotes are canonical: ordered by
|
||||
* FIRST REFERENCE (not markdown definition order), reused references deduped to a
|
||||
* single definition, and orphan definitions dropped.
|
||||
*
|
||||
* Since #345 the markdown parse runs through the canonical package
|
||||
* (`normalizeForeignMarkdown` -> `markdownToProseMirror`), which owns this
|
||||
* canonicalization: the input's GFM `[^id]` reference footnotes are normalized to
|
||||
* inline `^[…]`, and the parser assigns fresh sequential ids (`fn-*`) in
|
||||
* reference order while merging identical bodies — so we assert by definition
|
||||
* BODY order, not by the source labels. `canonicalizeFootnotes` remains wired as
|
||||
* an idempotent safety net (issue #228) and is a no-op on this already-canonical
|
||||
* output.
|
||||
*
|
||||
* The DB/ydoc side-effects are stubbed: `getNewPagePosition` (DB query) and
|
||||
* `createYdoc` (Yjs encode) are spied, and `pageRepo.insertPage` captures the
|
||||
@@ -67,24 +73,14 @@ function makeService() {
|
||||
}
|
||||
|
||||
/** List the footnote-definition ids of the (single) footnotesList, in order. */
|
||||
function footnoteListIds(content: any): string[] {
|
||||
/** Definition body texts of the (single) footnotesList, in list order. */
|
||||
function footnoteListBodies(content: any): string[] {
|
||||
const list = (content.content ?? []).find(
|
||||
(n: any) => n.type === 'footnotesList',
|
||||
);
|
||||
if (!list) return [];
|
||||
return (list.content ?? [])
|
||||
return (list?.content ?? [])
|
||||
.filter((n: any) => n.type === 'footnoteDefinition')
|
||||
.map((n: any) => n.attrs?.id);
|
||||
}
|
||||
|
||||
function definitionText(content: any, id: string): string | undefined {
|
||||
const list = (content.content ?? []).find(
|
||||
(n: any) => n.type === 'footnotesList',
|
||||
);
|
||||
const def = (list?.content ?? []).find(
|
||||
(n: any) => n.type === 'footnoteDefinition' && n.attrs?.id === id,
|
||||
);
|
||||
return def?.content?.[0]?.content?.[0]?.text;
|
||||
.map((n: any) => n.content?.[0]?.content?.[0]?.text);
|
||||
}
|
||||
|
||||
describe('ImportService.importPage — footnote canonicalization (#228)', () => {
|
||||
@@ -101,23 +97,23 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
|
||||
const content = getCaptured().content;
|
||||
expect(content).toBeTruthy();
|
||||
|
||||
// Reference order is c, a, b (NOT the markdown definition order a, b, c).
|
||||
expect(footnoteListIds(content)).toEqual(['c', 'a', 'b']);
|
||||
|
||||
// Definitions preserved and attached to the right ids.
|
||||
expect(definitionText(content, 'c')).toBe('note C');
|
||||
expect(definitionText(content, 'a')).toBe('note A');
|
||||
expect(definitionText(content, 'b')).toBe('note B');
|
||||
// Definitions ordered by FIRST REFERENCE (C, A, B) — NOT the markdown
|
||||
// definition order (A, B, C) — with the orphan [^z] dropped and the reused
|
||||
// [^a] collapsed to a single definition. (Ids are the parser's fresh `fn-*`,
|
||||
// so we pin the BODIES.)
|
||||
expect(footnoteListBodies(content)).toEqual(['note C', 'note A', 'note B']);
|
||||
|
||||
// Orphan definition [^z] is dropped.
|
||||
expect(footnoteListIds(content)).not.toContain('z');
|
||||
expect(footnoteListBodies(content)).not.toContain('orphan note');
|
||||
|
||||
// Reused [^a] yields exactly ONE definition, and exactly one list.
|
||||
const lists = (content.content ?? []).filter(
|
||||
(n: any) => n.type === 'footnotesList',
|
||||
);
|
||||
expect(lists).toHaveLength(1);
|
||||
expect(footnoteListIds(content).filter((id) => id === 'a')).toHaveLength(1);
|
||||
expect(
|
||||
footnoteListBodies(content).filter((b) => b === 'note A'),
|
||||
).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('is idempotent: canonicalizing the stored output again is a no-op', async () => {
|
||||
@@ -134,6 +130,6 @@ describe('ImportService.importPage — footnote canonicalization (#228)', () =>
|
||||
// time must not change it (safe to wire into every write path).
|
||||
const second = canonicalizeFootnotes(stored);
|
||||
expect(second).toEqual(stored);
|
||||
expect(footnoteListIds(second)).toEqual(['c', 'a', 'b']);
|
||||
expect(footnoteListBodies(second)).toEqual(['note C', 'note A', 'note B']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -17,7 +17,9 @@ import {
|
||||
import { generateJitteredKeyBetween } from 'fractional-indexing-jittered';
|
||||
import { TiptapTransformer } from '@hocuspocus/transformer';
|
||||
import * as Y from 'yjs';
|
||||
import { markdownToHtml, canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { canonicalizeFootnotes } from '@docmost/editor-ext';
|
||||
import { markdownToProseMirror } from '@docmost/prosemirror-markdown';
|
||||
import { normalizeForeignMarkdown } from '../utils/foreign-markdown';
|
||||
import {
|
||||
FileTaskStatus,
|
||||
FileTaskType,
|
||||
@@ -85,11 +87,13 @@ export class ImportService {
|
||||
|
||||
const extracted = this.extractTitleAndRemoveHeading(prosemirrorState);
|
||||
const title = extracted.title;
|
||||
// Imported markdown/HTML is built via markdownToHtml -> htmlToJson, which
|
||||
// never runs the editor's footnoteSyncPlugin, so the footnote topology keeps
|
||||
// the source's PHYSICAL definition order (out of order vs. references),
|
||||
// retains orphan definitions, and is not deduped. Canonicalize before
|
||||
// persisting so the stored page matches the editor's invariant (issue #228).
|
||||
// The markdown path now canonicalizes footnotes itself (the package parser),
|
||||
// but the HTML path (processHTML -> htmlToJson) does NOT run the editor's
|
||||
// footnoteSyncPlugin, so an imported HTML doc can keep its source's PHYSICAL
|
||||
// definition order (out of order vs. references), retain orphan definitions,
|
||||
// and not be deduped. Canonicalize before persisting so the stored page
|
||||
// matches the editor's invariant (issue #228); it is an idempotent no-op on
|
||||
// the already-canonical markdown output.
|
||||
// Pure + idempotent + shape-safe: a doc with no footnotes is unchanged.
|
||||
// (Future consolidation, architecture B: this import path persists directly
|
||||
// via pageRepo.insertPage rather than through PageService.createPage, so the
|
||||
@@ -133,12 +137,15 @@ export class ImportService {
|
||||
}
|
||||
|
||||
async processMarkdown(markdownInput: string): Promise<any> {
|
||||
try {
|
||||
const html = await markdownToHtml(markdownInput);
|
||||
return this.processHTML(html);
|
||||
} catch (err) {
|
||||
throw err;
|
||||
}
|
||||
// Canonical markdown -> ProseMirror JSON directly via
|
||||
// `@docmost/prosemirror-markdown` (issue #345) — no HTML intermediate and no
|
||||
// second editor-ext markdown layer. Foreign markdown surfaces the strict
|
||||
// canonical parser does not accept (GFM `[^id]` reference footnotes) are
|
||||
// rewritten to the canonical inline form by `normalizeForeignMarkdown` first.
|
||||
// The HTML-cleanup pass (`normalizeImportHtml`) is intentionally skipped here:
|
||||
// it targets foreign *HTML* (Notion/XWiki), which only ever arrives on the
|
||||
// `.html` path (`processHTML`), never as canonical markdown.
|
||||
return markdownToProseMirror(normalizeForeignMarkdown(markdownInput));
|
||||
}
|
||||
|
||||
async processHTML(htmlInput: string): Promise<any> {
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
import {
|
||||
convertProseMirrorToMarkdown,
|
||||
markdownToProseMirror,
|
||||
} from '@docmost/prosemirror-markdown';
|
||||
import { normalizeForeignMarkdown } from './foreign-markdown';
|
||||
|
||||
/**
|
||||
* STEP 2 goldens for issue #345: the foreign-markdown normalizer that runs at the
|
||||
* import boundary BEFORE the strict canonical parser (`markdownToProseMirror`).
|
||||
*
|
||||
* Two layers:
|
||||
* 1. PURE string→string cases pinning the normalizer's own behavior (GFM
|
||||
* reference footnotes → inline `^[…]`).
|
||||
* 2. END-TO-END acceptance: for a foreign corpus, `normalizeForeignMarkdown`
|
||||
* then `markdownToProseMirror` then `convertProseMirrorToMarkdown` must leave
|
||||
* NO literal `[^id]` / `:::` garbage in the document and must re-export in the
|
||||
* canonical forms.
|
||||
*/
|
||||
|
||||
describe('normalizeForeignMarkdown — GFM reference footnotes', () => {
|
||||
it('inlines a single-line reference footnote and drops its definition', () => {
|
||||
const out = normalizeForeignMarkdown(
|
||||
'A note[^1] here.\n\n[^1]: The definition.',
|
||||
);
|
||||
expect(out).toBe('A note^[The definition.] here.\n');
|
||||
});
|
||||
|
||||
it('inlines every reference to a reused id (downstream dedups)', () => {
|
||||
const out = normalizeForeignMarkdown(
|
||||
'X[^a] and Y[^a].\n\n[^a]: shared.',
|
||||
);
|
||||
expect(out).toBe('X^[shared.] and Y^[shared.].\n');
|
||||
});
|
||||
|
||||
it('joins indented continuation lines of a definition with a space', () => {
|
||||
const out = normalizeForeignMarkdown(
|
||||
'See[^n].\n\n[^n]: line one\n line two',
|
||||
);
|
||||
expect(out).toBe('See^[line one line two].\n');
|
||||
});
|
||||
|
||||
it('never rewrites a reference inside a fenced code block', () => {
|
||||
const out = normalizeForeignMarkdown(
|
||||
'```\ncode[^1] here\n```\n\n[^1]: def.',
|
||||
);
|
||||
expect(out).toContain('code[^1] here');
|
||||
// The (now orphaned) definition line is still removed.
|
||||
expect(out).not.toContain('[^1]: def.');
|
||||
});
|
||||
|
||||
it('never rewrites a reference inside an INLINE-code span (backticks)', () => {
|
||||
// The `[^1]` inside backticks is literal code and must survive verbatim;
|
||||
// the one outside is rewritten. (Bug #1: only fenced blocks were protected.)
|
||||
const out = normalizeForeignMarkdown(
|
||||
'Use `arr[^1]` in code but note[^1] in prose.\n\n[^1]: def.',
|
||||
);
|
||||
expect(out).toBe('Use `arr[^1]` in code but note^[def.] in prose.\n');
|
||||
});
|
||||
|
||||
it('escapes brackets in a body so an unbalanced ] cannot truncate the footnote', () => {
|
||||
// A foreign definition body with a stray `]` would, unescaped, close the
|
||||
// canonical `^[...]` early and leak the tail as text (bug #2). The body's
|
||||
// brackets are backslash-escaped so the footnote stays whole.
|
||||
const out = normalizeForeignMarkdown(
|
||||
'Ref[^1] here.\n\n[^1]: see item ] and [more] later',
|
||||
);
|
||||
expect(out).toBe('Ref^[see item \\] and \\[more\\] later] here.\n');
|
||||
// The tokenizer must see exactly one unescaped closing bracket (our own).
|
||||
expect(out.match(/(?<!\\)\]/g)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('leaves a reference with no matching definition literal (no body to inline)', () => {
|
||||
const out = normalizeForeignMarkdown('Dangling[^x] ref.');
|
||||
expect(out).toBe('Dangling[^x] ref.');
|
||||
});
|
||||
|
||||
it('returns the input unchanged when there are no reference footnotes', () => {
|
||||
const md = '# Title\n\nJust text with `inline code` and a [link](/x).';
|
||||
expect(normalizeForeignMarkdown(md)).toBe(md);
|
||||
});
|
||||
|
||||
it('does NOT touch callout surfaces — the canonical parser handles them', () => {
|
||||
const callouts = ':::info\nHi\n:::\n\n> [!warning]\n> Careful';
|
||||
expect(normalizeForeignMarkdown(callouts)).toBe(callouts);
|
||||
});
|
||||
|
||||
it('strips a leading YAML front-matter block (Obsidian/Hugo/git-sync files)', () => {
|
||||
const out = normalizeForeignMarkdown(
|
||||
'---\ntitle: My Page\ntags: [a, b]\n---\n\n# Heading\n\nBody.',
|
||||
);
|
||||
expect(out).toBe('# Heading\n\nBody.');
|
||||
// The front-matter must not leak into the body as a setext heading.
|
||||
expect(out).not.toContain('title: My Page');
|
||||
expect(out).not.toContain('---');
|
||||
});
|
||||
|
||||
it('does not strip a horizontal rule that is not leading front-matter', () => {
|
||||
const md = 'Intro paragraph.\n\n---\n\nAfter the rule.';
|
||||
expect(normalizeForeignMarkdown(md)).toBe(md);
|
||||
});
|
||||
|
||||
it('is linear on a document with thousands of definitions (no quadratic blowup)', () => {
|
||||
// F2(a): the pass-2 rewrite must be O(text), not O(text × defs). Build a
|
||||
// pathological doc (many defs + many plain text lines) and assert it
|
||||
// completes well under a second — a quadratic implementation took ~14s.
|
||||
const N = 4000;
|
||||
const refs = Array.from({ length: N }, (_, i) => `line ${i} plain text`).join('\n');
|
||||
const defs = Array.from({ length: N }, (_, i) => `[^n${i}]: def ${i}`).join('\n');
|
||||
const doc = `start[^n0] and[^n${N - 1}] end\n\n${refs}\n\n${defs}`;
|
||||
const t0 = Date.now();
|
||||
const out = normalizeForeignMarkdown(doc);
|
||||
const elapsed = Date.now() - t0;
|
||||
expect(elapsed).toBeLessThan(2000);
|
||||
// Sanity: the two real references were still inlined.
|
||||
expect(out).toContain('^[def 0]');
|
||||
expect(out).toContain(`^[def ${N - 1}]`);
|
||||
});
|
||||
|
||||
it('is bounded on a long unclosed backtick run (no inline-split ReDoS)', () => {
|
||||
// F2(b): a huge unterminated backtick run must not cause quadratic
|
||||
// backtracking in the inline-code split. Oversized lines skip the split
|
||||
// entirely (left untouched), so this returns promptly.
|
||||
const line = 'x' + '`'.repeat(200000);
|
||||
const doc = `${line}\n\n[^1]: def`;
|
||||
const t0 = Date.now();
|
||||
normalizeForeignMarkdown(doc);
|
||||
expect(Date.now() - t0).toBeLessThan(2000);
|
||||
});
|
||||
|
||||
it('does not crash or slow down on thousands of prefix-chain definition ids', () => {
|
||||
// F7: the rewrite must use a FIXED generic scanner, not an alternation built
|
||||
// from the ids. A `(a|aa|aaa|…)` alternation over prefix-chain ids blows the
|
||||
// V8 regex compiler (FATAL RegExpCompiler Allocation failed — uncatchable,
|
||||
// kills the process). A fixed scanner has no id-dependent compilation cost.
|
||||
const N = 4000;
|
||||
const ids = Array.from({ length: N }, (_, i) => 'a'.repeat(i + 1));
|
||||
const defs = ids.map((id) => `[^${id}]: body ${id.length}`).join('\n');
|
||||
const doc = `ref[^${ids[0]}] and[^${ids[N - 1]}] end\n\n${defs}`;
|
||||
const t0 = Date.now();
|
||||
const out = normalizeForeignMarkdown(doc);
|
||||
expect(Date.now() - t0).toBeLessThan(2000);
|
||||
// Prefix disambiguation is correct: [^a] and [^aaaa...] inline their OWN body.
|
||||
expect(out).toContain('^[body 1]');
|
||||
expect(out).toContain(`^[body ${N}]`);
|
||||
});
|
||||
|
||||
it('strips a CRLF (Windows) front-matter block, not just LF', () => {
|
||||
// F9: the line-anchored regex needs LF after the opening `---`, so a Windows
|
||||
// file (`---\r\n…`) would slip past the strip and leak the front-matter into
|
||||
// the body. normalizeForeignMarkdown normalizes CRLF -> LF first.
|
||||
const out = normalizeForeignMarkdown(
|
||||
'---\r\ntitle: Foo\r\ntags: [a]\r\n---\r\n\r\n# Heading\r\n\r\nBody.',
|
||||
);
|
||||
expect(out).toBe('# Heading\n\nBody.');
|
||||
expect(out).not.toContain('title: Foo');
|
||||
expect(out).not.toContain('---');
|
||||
});
|
||||
|
||||
it('strips front-matter whose value contains a triple-dash (line-anchored)', () => {
|
||||
// F8: the block must close only on a `\n---` LINE, not the first inline
|
||||
// `---`. A value like `title: Q1 --- Q2` must not truncate the front-matter
|
||||
// and leak the rest (author/closing ---) into the body.
|
||||
const out = normalizeForeignMarkdown(
|
||||
'---\ntitle: Q1 --- Q2 results\nauthor: bob\n---\n\nReal body.',
|
||||
);
|
||||
expect(out).toBe('Real body.');
|
||||
expect(out).not.toContain('author: bob');
|
||||
expect(out).not.toContain('Q2 results');
|
||||
});
|
||||
});
|
||||
|
||||
describe('foreign markdown import acceptance (normalizer + canonical parser)', () => {
|
||||
const FOREIGN = [
|
||||
'# Doc',
|
||||
'',
|
||||
'Body refs [^c] and [^a] and [^b] and again [^a].',
|
||||
'',
|
||||
':::info',
|
||||
'A legacy callout.',
|
||||
':::',
|
||||
'',
|
||||
'| h1 | h2 |',
|
||||
'| --- | --- |',
|
||||
'| 1 | 2 |',
|
||||
'',
|
||||
'[^a]: note A',
|
||||
'[^b]: note B',
|
||||
'[^c]: note C',
|
||||
'[^z]: orphan note',
|
||||
].join('\n');
|
||||
|
||||
it('leaves no literal [^id] or ::: in the imported doc and re-exports canonically', async () => {
|
||||
const normalized = normalizeForeignMarkdown(FOREIGN);
|
||||
const doc = await markdownToProseMirror(normalized);
|
||||
const reexport = convertProseMirrorToMarkdown(doc);
|
||||
|
||||
// No foreign garbage leaks into the document.
|
||||
expect(reexport).not.toMatch(/\[\^/); // no reference footnote refs/defs
|
||||
expect(reexport).not.toContain(':::'); // no legacy callout fences
|
||||
|
||||
// Canonical forms are present.
|
||||
expect(reexport).toContain('^[note C]');
|
||||
expect(reexport).toContain('> [!info]');
|
||||
expect(reexport).toContain('| h1 | h2 |');
|
||||
|
||||
// Footnotes: ordered by first reference (C, A, B), reused [^a] deduped to one,
|
||||
// orphan [^z] dropped (it had no reference after normalization).
|
||||
const list = doc.content.find((n: any) => n.type === 'footnotesList');
|
||||
const bodies = list.content.map(
|
||||
(d: any) => d.content[0].content[0].text,
|
||||
);
|
||||
expect(bodies).toEqual(['note C', 'note A', 'note B']);
|
||||
expect(bodies).not.toContain('orphan note');
|
||||
expect(
|
||||
doc.content.filter((n: any) => n.type === 'footnotesList'),
|
||||
).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,265 @@
|
||||
/**
|
||||
* Foreign-markdown normalizer — an input-liberal / output-canonical adapter that
|
||||
* runs at the IMPORT boundary, BEFORE the canonical parser
|
||||
* (`markdownToProseMirror` from `@docmost/prosemirror-markdown`).
|
||||
*
|
||||
* The canonical parser is deliberately STRICT: it only understands Docmost's
|
||||
* canonical markdown surface (Obsidian-style `> [!type]` callouts, Pandoc/Obsidian
|
||||
* inline footnotes `^[body]`, lossless ` <!--img {...}-->` images, …).
|
||||
* Import, however, ingests FOREIGN files (GitHub/GFM, Notion, old Docmost
|
||||
* exports). Those use surfaces the canonical parser does not accept, most notably
|
||||
* GitHub-flavoured *reference* footnotes:
|
||||
*
|
||||
* Text with a note[^1] and another[^long].
|
||||
*
|
||||
* [^1]: The first definition.
|
||||
* [^long]: A second one.
|
||||
*
|
||||
* Left untouched, the parser does NOT recognise `[^id]` (it only parses `^[body]`),
|
||||
* so the reference leaks as literal text — and worse, the trailing `[^id]: def`
|
||||
* line is a valid CommonMark *link-reference definition*, so `[^id]` is silently
|
||||
* rendered as a bogus link. This normalizer rewrites reference footnotes into the
|
||||
* canonical inline form so the parser materialises real footnote nodes.
|
||||
*
|
||||
* This is a TEXT pre-pass, NOT a second parser fork: it does not re-implement any
|
||||
* converter logic. Callout surfaces (`:::type` and `> [!type]`) are intentionally
|
||||
* NOT touched here — the canonical parser already accepts BOTH natively (its
|
||||
* `preprocessCallouts` pass), so normalizing them would be redundant and would
|
||||
* only risk degrading the parser's nesting/code-fence-aware handling.
|
||||
*/
|
||||
|
||||
/** Matches a fenced code block delimiter (``` or ~~~), capturing the marker run. */
|
||||
const CODE_FENCE_RE = /^(\s*)(`{3,}|~{3,})/;
|
||||
|
||||
/**
|
||||
* Matches a GFM footnote DEFINITION line: `[^id]: body`. The id is any run of
|
||||
* non-`]` characters; the body is the remainder of the line (possibly empty).
|
||||
*/
|
||||
const FOOTNOTE_DEF_RE = /^\[\^([^\]]+)\]:[ \t]?(.*)$/;
|
||||
|
||||
/** True when a line is a code-fence delimiter that toggles fenced-code state. */
|
||||
function fenceMarker(line: string): string | null {
|
||||
const m = line.match(CODE_FENCE_RE);
|
||||
return m ? m[2] : null;
|
||||
}
|
||||
|
||||
/** True when a line is indented (leading space/tab) and not blank — a continuation. */
|
||||
function isIndentedContinuation(line: string): boolean {
|
||||
return /^[ \t]+\S/.test(line);
|
||||
}
|
||||
|
||||
function escapeRegExp(value: string): string {
|
||||
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
}
|
||||
|
||||
/**
|
||||
* Backslash-escape any square bracket in a footnote body before it is wrapped in
|
||||
* `^[...]`. The canonical inline-footnote tokenizer scans the body with bracket
|
||||
* balancing and closes on the first UNMATCHED `]`, so an unbalanced bracket in a
|
||||
* foreign definition (e.g. `[^1]: see item ] later`) would otherwise truncate the
|
||||
* footnote and leak the tail as literal text. Escaping every `[`/`]` makes the
|
||||
* body an inert run of characters — the tokenizer then closes only on our own
|
||||
* closing `]`. (A balanced `[link](url)` inside a body still round-trips because
|
||||
* the escaped form renders the literal brackets, which is the safe reading for a
|
||||
* footnote body; the alternative — brittle balance tracking — risks worse.)
|
||||
*/
|
||||
function escapeFootnoteBody(body: string): string {
|
||||
return body.replace(/[[\]]/g, '\\$&');
|
||||
}
|
||||
|
||||
/**
|
||||
* Rewrite every `[^id]` reference on a line to its `^[body]` form, but ONLY in the
|
||||
* text OUTSIDE inline-code spans. A `[^id]` inside backticks is literal code
|
||||
* content and must be preserved verbatim (a footnote ref never lives inside code).
|
||||
* We split the line on inline-code spans (paired backtick runs) and rewrite only
|
||||
* the non-code segments.
|
||||
*/
|
||||
// Above this length a single line is not split into inline-code spans (see
|
||||
// below). A genuine markdown line carrying a footnote reference is never tens of
|
||||
// KB; the cap only bypasses the inline-code protection for pathological lines.
|
||||
const INLINE_SPLIT_MAX_LINE = 8192;
|
||||
|
||||
function rewriteRefsOutsideInlineCode(
|
||||
line: string,
|
||||
replace: (text: string) => string,
|
||||
): string {
|
||||
// The inline-code split alternation `(`+)(?:(?!\1)[\s\S])*\1` backtracks
|
||||
// quadratically on a long UNCLOSED backtick run (its middle can consume the
|
||||
// rest of the line, then fail to find a closing run and retry from each
|
||||
// position). On an untrusted import this is a request-thread ReDoS. A real
|
||||
// footnote line is short, so for an oversized line we skip the inline-code
|
||||
// protection entirely and leave the line UNTOUCHED (rewriting it wholesale
|
||||
// could corrupt a `[^id]` that legitimately lives inside inline code). This is
|
||||
// a conservative bypass: an over-8KB line simply does not get its reference
|
||||
// footnotes inlined — acceptable for a pathological input.
|
||||
if (line.length > INLINE_SPLIT_MAX_LINE) return line;
|
||||
|
||||
// Alternation: an inline-code span (one or more backticks, then anything up to
|
||||
// the SAME run of backticks) OR a run of non-backtick text. Unterminated
|
||||
// backticks fall through as ordinary text (matched by the second branch on the
|
||||
// leftover), so a stray backtick never swallows the rest of the line.
|
||||
const parts = line.match(/(`+)(?:(?!\1)[\s\S])*\1|[^`]+|`+/g);
|
||||
if (!parts) return line;
|
||||
return parts
|
||||
.map((seg) => (seg.startsWith('`') ? seg : replace(seg)))
|
||||
.join('');
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert GFM reference footnotes (`[^id]` + `[^id]: def`) into canonical inline
|
||||
* footnotes (`^[def]`).
|
||||
*
|
||||
* - Definitions are collected first (a leading `[^id]: text` line plus any
|
||||
* immediately-following indented continuation lines, joined with a space) and
|
||||
* removed from the output.
|
||||
* - Each in-text reference `[^id]` for which a definition was found is replaced by
|
||||
* `^[def]`. References with no matching definition are left literal (there is no
|
||||
* body to inline; the parser fails them open the same way).
|
||||
* - Code is respected on both passes: `[^id]` inside a fenced ``` / ~~~ block is
|
||||
* never rewritten and a `[^id]:` line inside a fence is never a definition; and
|
||||
* on the rewrite pass a `[^id]` inside an INLINE-code span (backticks) is left
|
||||
* literal too.
|
||||
* - The inlined body is bracket-escaped so an unbalanced `[`/`]` in a foreign
|
||||
* definition cannot truncate the resulting `^[...]` footnote.
|
||||
*
|
||||
* Deduplication / reference-ordering / orphan-dropping of the resulting footnotes
|
||||
* is handled downstream by the canonical parser (`assembleFootnotes`); this pass
|
||||
* only changes the surface syntax.
|
||||
*/
|
||||
function convertReferenceFootnotes(markdown: string): string {
|
||||
const lines = markdown.split('\n');
|
||||
|
||||
// Pass 1: collect definitions and mark their lines for removal.
|
||||
const defs = new Map<string, string>();
|
||||
const dropped = new Array<boolean>(lines.length).fill(false);
|
||||
let inFence = false;
|
||||
let fence = '';
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
const marker = fenceMarker(line);
|
||||
if (inFence) {
|
||||
if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
|
||||
inFence = false;
|
||||
fence = '';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (marker) {
|
||||
inFence = true;
|
||||
fence = marker;
|
||||
continue;
|
||||
}
|
||||
|
||||
const def = line.match(FOOTNOTE_DEF_RE);
|
||||
if (!def) continue;
|
||||
|
||||
const id = def[1];
|
||||
const body: string[] = [def[2].trim()];
|
||||
dropped[i] = true;
|
||||
|
||||
// Consume immediately-following indented continuation lines (GFM lazy
|
||||
// continuation is not supported by design — keep it simple and predictable).
|
||||
let j = i + 1;
|
||||
while (j < lines.length && isIndentedContinuation(lines[j])) {
|
||||
body.push(lines[j].trim());
|
||||
dropped[j] = true;
|
||||
j++;
|
||||
}
|
||||
i = j - 1;
|
||||
|
||||
// Last definition wins for a duplicated id (matches CommonMark link-ref
|
||||
// semantics closely enough for a foreign-input adapter).
|
||||
defs.set(id, body.filter((s) => s.length > 0).join(' '));
|
||||
}
|
||||
|
||||
if (defs.size === 0) {
|
||||
return markdown;
|
||||
}
|
||||
|
||||
// ONE fixed, generic scanner regex — NOT one built from the definition ids.
|
||||
// It matches ANY `[^id]` shape, and the replacer decides per match via a map
|
||||
// lookup whether that id is a real definition (replace) or not (leave as-is).
|
||||
// This is genuinely O(total text) with no per-document regex compilation.
|
||||
//
|
||||
// Do NOT rebuild this as an alternation over `[...defs.keys()]`: a giant
|
||||
// `(id1|id2|...)` alternation over thousands of ids can blow the V8 regex
|
||||
// compiler's stack — a fatal, UNCATCHABLE "RegExpCompiler Allocation failed"
|
||||
// on prefix-chain ids (`a`, `aa`, `aaa`, ...) that kills the whole process
|
||||
// (worse than the earlier per-def thread-hang). A fixed scanner has no
|
||||
// id-dependent compilation cost and cannot blow up.
|
||||
const refRe = /\[\^([^\]]+)\]/g;
|
||||
const rewriteSegment = (segment: string): string =>
|
||||
segment.replace(refRe, (whole, id: string) => {
|
||||
const body = defs.get(id);
|
||||
// Only real definitions are inlined; an unknown id is left literal (same as
|
||||
// the old per-def loop, which simply never matched it).
|
||||
return body === undefined ? whole : `^[${escapeFootnoteBody(body)}]`;
|
||||
});
|
||||
|
||||
// Pass 2: rewrite in-text references, skipping fenced code and dropped lines.
|
||||
const out: string[] = [];
|
||||
inFence = false;
|
||||
fence = '';
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (dropped[i]) continue;
|
||||
let line = lines[i];
|
||||
|
||||
const marker = fenceMarker(line);
|
||||
if (inFence) {
|
||||
out.push(line);
|
||||
if (marker && marker[0] === fence[0] && marker.length >= fence.length) {
|
||||
inFence = false;
|
||||
fence = '';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (marker) {
|
||||
inFence = true;
|
||||
fence = marker;
|
||||
out.push(line);
|
||||
continue;
|
||||
}
|
||||
|
||||
line = rewriteRefsOutsideInlineCode(line, rewriteSegment);
|
||||
out.push(line);
|
||||
}
|
||||
|
||||
return out.join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip a single leading YAML front-matter block (`---\n…\n---`). Foreign files
|
||||
* from Obsidian / Hugo / Jekyll / Notion — and Docmost's OWN git-sync page files
|
||||
* — open with front-matter that the canonical parser does not consume, so
|
||||
* without this it leaks into the body (and `title: Foo` above the closing `---`
|
||||
* renders as a setext `<h2>` that `extractTitleAndRemoveHeading` can hijack as
|
||||
* the page title). It is a no-op for front-matter-free input.
|
||||
*
|
||||
* LINE-ANCHORED (the same shape the canonical parser uses in
|
||||
* prosemirror-markdown/page-file.ts): the block opens only on `---\n` at the
|
||||
* very start and closes only on a `\n---` line. The retired `markdownToHtml`
|
||||
* strip closed on the FIRST `---` ANYWHERE (an unanchored close), so a value
|
||||
* containing a triple-dash (e.g. `title: Q1 --- Q2`) truncated the front-matter
|
||||
* and leaked the rest into the body. An optional leading BOM is tolerated.
|
||||
*/
|
||||
const YAML_FRONT_MATTER_RE = /^\uFEFF?---\n[\s\S]*?\n---\n?/;
|
||||
|
||||
/**
|
||||
* Normalize a foreign markdown string into Docmost's canonical markdown surface
|
||||
* so the strict canonical parser accepts it losslessly: normalize line endings,
|
||||
* strip a leading YAML front-matter block, then rewrite GFM reference footnotes
|
||||
* into inline footnotes. Add further fixture-driven foreign-surface cases here as
|
||||
* they are found.
|
||||
*/
|
||||
export function normalizeForeignMarkdown(markdown: string): string {
|
||||
if (!markdown) return markdown;
|
||||
// Normalize CRLF -> LF FIRST. The line-anchored front-matter regex requires a
|
||||
// bare `\n` after the opening `---`, and convertReferenceFootnotes splits on
|
||||
// `\n`; a Windows/CRLF foreign file (`---\r\n…`) would otherwise slip past the
|
||||
// front-matter strip and leak into the body. The canonical parser
|
||||
// (page-file.ts parsePageFile) normalizes the same way before its FRONTMATTER_RE.
|
||||
const src = markdown.replace(/\r\n/g, '\n');
|
||||
const withoutFrontMatter = src.replace(YAML_FRONT_MATTER_RE, '').trimStart();
|
||||
return convertReferenceFootnotes(withoutFrontMatter);
|
||||
}
|
||||
@@ -2,15 +2,36 @@ import { FastifyReply, FastifyRequest } from 'fastify';
|
||||
import { isStreamingResponse } from './metrics.constants';
|
||||
import { observeHttp } from './metrics.registry';
|
||||
|
||||
// URL path prefixes served by @fastify/static (client build output under
|
||||
// client/dist). `/assets/` holds the content-hashed bundle (index-*.js,
|
||||
// chunk-*.js) — a NEW set of names every deploy, i.e. an UNBOUNDED label set
|
||||
// (#362); the others (/vad/, /brand/, /locales/, /icons/ — copied verbatim from
|
||||
// public/) have stable names, so they are merely repetitive per-file labels
|
||||
// rather than unbounded. Either way none of these belong in the API-route
|
||||
// histogram: collapse them all to one bounded `static` label. (Edge latency for
|
||||
// static is already measured by Traefik's traefik_router_request_duration_*.)
|
||||
const STATIC_PATH_PREFIXES = [
|
||||
'/assets/',
|
||||
'/vad/',
|
||||
'/brand/',
|
||||
'/locales/',
|
||||
'/icons/',
|
||||
];
|
||||
|
||||
/**
|
||||
* Resolve the BOUNDED route label for an HTTP response.
|
||||
*
|
||||
* HARD REQUIREMENT (#355): use the ROUTE TEMPLATE (`/pages/:id`), NEVER the raw
|
||||
* URL (`/pages/abc-123`), so label cardinality stays finite. Fastify exposes the
|
||||
* matched template on `req.routeOptions.url`. On 404s (no route matched) that is
|
||||
* missing → collapse to the literal `unknown`.
|
||||
* HARD REQUIREMENT (#355): use the ROUTE TEMPLATE (`/pages/:id`), NEVER a raw
|
||||
* URL (`/pages/abc-123` or `/assets/index-CAbxDtto.js`), so label cardinality
|
||||
* stays finite. Fastify exposes the matched template on `req.routeOptions.url`,
|
||||
* BUT @fastify/static serves each file through a route whose matched url is the
|
||||
* raw (hashed) file path — so for static assets that value is itself unbounded.
|
||||
* Detect static requests by their path prefix FIRST and collapse to `static`;
|
||||
* otherwise use the route template; on a 404 (no route matched) → `unknown`.
|
||||
*/
|
||||
export function resolveRouteLabel(req: FastifyRequest): string {
|
||||
const path = (req.url ?? '').split('?', 1)[0];
|
||||
if (STATIC_PATH_PREFIXES.some((p) => path.startsWith(p))) return 'static';
|
||||
const url = req.routeOptions?.url;
|
||||
return typeof url === 'string' && url.length > 0 ? url : 'unknown';
|
||||
}
|
||||
|
||||
@@ -46,7 +46,6 @@ export class MetricsBullService implements OnModuleInit, OnModuleDestroy {
|
||||
@InjectQueue(QueueName.GENERAL_QUEUE) generalQueue: Queue,
|
||||
@InjectQueue(QueueName.BILLING_QUEUE) billingQueue: Queue,
|
||||
@InjectQueue(QueueName.FILE_TASK_QUEUE) fileTaskQueue: Queue,
|
||||
@InjectQueue(QueueName.SEARCH_QUEUE) searchQueue: Queue,
|
||||
@InjectQueue(QueueName.AI_QUEUE) aiQueue: Queue,
|
||||
@InjectQueue(QueueName.HISTORY_QUEUE) historyQueue: Queue,
|
||||
@InjectQueue(QueueName.NOTIFICATION_QUEUE) notificationQueue: Queue,
|
||||
@@ -58,7 +57,6 @@ export class MetricsBullService implements OnModuleInit, OnModuleDestroy {
|
||||
{ label: 'general', queue: generalQueue },
|
||||
{ label: 'billing', queue: billingQueue },
|
||||
{ label: 'file-task', queue: fileTaskQueue },
|
||||
{ label: 'search', queue: searchQueue },
|
||||
{ label: 'ai', queue: aiQueue },
|
||||
{ label: 'history', queue: historyQueue },
|
||||
{ label: 'notification', queue: notificationQueue },
|
||||
|
||||
@@ -25,6 +25,61 @@ describe('resolveRouteLabel (histogram route label)', () => {
|
||||
const req = { url: '/x' } as unknown as FastifyRequest;
|
||||
expect(resolveRouteLabel(req)).toBe('unknown');
|
||||
});
|
||||
|
||||
it.each([
|
||||
'/assets/index-CAbxDtto.js',
|
||||
'/assets/chunk-3OPIFGDE-CJOt9nr5.js',
|
||||
'/assets/excalidraw-menu-DpsI0kFW.js',
|
||||
'/vad/silero_vad_v5.onnx',
|
||||
'/brand/logo.svg',
|
||||
'/locales/en.json',
|
||||
'/icons/app-icon-192x192.png',
|
||||
])('collapses hashed/static asset %p to "static" (#362 cardinality)', (url) => {
|
||||
// @fastify/static serves each file through a route whose matched url is the
|
||||
// raw (hashed) file path, so routeOptions.url is itself unbounded here.
|
||||
const req = {
|
||||
url,
|
||||
routeOptions: { url },
|
||||
} as unknown as FastifyRequest;
|
||||
const label = resolveRouteLabel(req);
|
||||
expect(label).toBe('static');
|
||||
expect(label).not.toContain('.js');
|
||||
expect(label).not.toContain('index-');
|
||||
});
|
||||
|
||||
it('strips the query string before the static-prefix check', () => {
|
||||
const req = {
|
||||
url: '/assets/index-CAbxDtto.js?v=2',
|
||||
routeOptions: { url: '/assets/index-CAbxDtto.js' },
|
||||
} as unknown as FastifyRequest;
|
||||
expect(resolveRouteLabel(req)).toBe('static');
|
||||
});
|
||||
|
||||
it('does NOT collapse a real API route that merely mentions assets', () => {
|
||||
// A templated API route is kept as-is; only the static path PREFIXES collapse.
|
||||
const req = {
|
||||
url: '/api/pages/assets-guide',
|
||||
routeOptions: { url: '/api/pages/:id' },
|
||||
} as unknown as FastifyRequest;
|
||||
expect(resolveRouteLabel(req)).toBe('/api/pages/:id');
|
||||
});
|
||||
|
||||
it.each([
|
||||
// The TRAILING SLASH on the prefix is the anti-false-collapse guard: a path
|
||||
// that is the prefix WITHOUT its slash, or merely shares the prefix as a
|
||||
// substring of a longer segment, must NOT collapse. These would collapse
|
||||
// under a buggy `includes('/assets/')` / slashless-prefix impl.
|
||||
'/assets',
|
||||
'/assetsx/foo.js',
|
||||
'/iconset/x.png',
|
||||
])('does NOT collapse the prefix-boundary case %p', (url) => {
|
||||
const req = {
|
||||
url,
|
||||
routeOptions: { url: '/some/:route' },
|
||||
} as unknown as FastifyRequest;
|
||||
expect(resolveRouteLabel(req)).not.toBe('static');
|
||||
expect(resolveRouteLabel(req)).toBe('/some/:route');
|
||||
});
|
||||
});
|
||||
|
||||
describe('isStreamingResponse (SSE exclusion)', () => {
|
||||
|
||||
@@ -4,7 +4,6 @@ export enum QueueName {
|
||||
GENERAL_QUEUE = '{general-queue}',
|
||||
BILLING_QUEUE = '{billing-queue}',
|
||||
FILE_TASK_QUEUE = '{file-task-queue}',
|
||||
SEARCH_QUEUE = '{search-queue}',
|
||||
AI_QUEUE = '{ai-queue}',
|
||||
HISTORY_QUEUE = '{history-queue}',
|
||||
NOTIFICATION_QUEUE = '{notification-queue}',
|
||||
@@ -32,12 +31,6 @@ export enum QueueJob {
|
||||
IMPORT_TASK = 'import-task',
|
||||
EXPORT_TASK = 'export-task',
|
||||
|
||||
SEARCH_INDEX_PAGE = 'search-index-page',
|
||||
SEARCH_INDEX_PAGES = 'search-index-pages',
|
||||
SEARCH_INDEX_COMMENT = 'search-index-comment',
|
||||
SEARCH_INDEX_COMMENTS = 'search-index-comments',
|
||||
SEARCH_INDEX_ATTACHMENT = 'search-index-attachment',
|
||||
SEARCH_INDEX_ATTACHMENTS = 'search-index-attachments',
|
||||
SEARCH_REMOVE_PAGE = 'search-remove-page',
|
||||
SEARCH_REMOVE_ASSET = 'search-remove-attachment',
|
||||
SEARCH_REMOVE_FACE = 'search-remove-comment',
|
||||
|
||||
@@ -57,14 +57,6 @@ import { GeneralQueueProcessor } from './processors/general-queue.processor';
|
||||
attempts: 1,
|
||||
},
|
||||
}),
|
||||
BullModule.registerQueue({
|
||||
name: QueueName.SEARCH_QUEUE,
|
||||
defaultJobOptions: {
|
||||
removeOnComplete: true,
|
||||
removeOnFail: true,
|
||||
attempts: 2,
|
||||
},
|
||||
}),
|
||||
BullModule.registerQueue({
|
||||
name: QueueName.AI_QUEUE,
|
||||
defaultJobOptions: {
|
||||
|
||||
@@ -38,6 +38,24 @@ export const TEST_DATABASE_URL =
|
||||
process.env.TEST_DATABASE_URL ??
|
||||
'postgresql://docmost:docmost_dev_pw@localhost:5432/docmost_test';
|
||||
|
||||
// Build the raw postgres.js client (mirrors database.module.ts: max pool,
|
||||
// silenced notices, bigint-as-number parsing). Kept separate so the singleton
|
||||
// can hold a reference to bound its shutdown in destroyTestDb.
|
||||
function buildTestSql(url: string = TEST_DATABASE_URL) {
|
||||
return postgres(url, {
|
||||
max: 5,
|
||||
onnotice: () => {},
|
||||
types: {
|
||||
bigint: {
|
||||
to: 20,
|
||||
from: [20, 1700],
|
||||
serialize: (value: number) => value.toString(),
|
||||
parse: (value: string) => Number.parseInt(value),
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a Kysely instance that MIRRORS the app's setup in database.module.ts:
|
||||
* PostgresJSDialect over postgres(), CamelCasePlugin, and the bigint type
|
||||
@@ -47,38 +65,40 @@ export const TEST_DATABASE_URL =
|
||||
*/
|
||||
export function buildTestDb(url: string = TEST_DATABASE_URL): Kysely<any> {
|
||||
return new Kysely<any>({
|
||||
dialect: new PostgresJSDialect({
|
||||
postgres: postgres(url, {
|
||||
max: 5,
|
||||
onnotice: () => {},
|
||||
types: {
|
||||
bigint: {
|
||||
to: 20,
|
||||
from: [20, 1700],
|
||||
serialize: (value: number) => value.toString(),
|
||||
parse: (value: string) => Number.parseInt(value),
|
||||
},
|
||||
},
|
||||
}),
|
||||
}),
|
||||
dialect: new PostgresJSDialect({ postgres: buildTestSql(url) }),
|
||||
plugins: [new CamelCasePlugin()],
|
||||
});
|
||||
}
|
||||
|
||||
let singleton: Kysely<any> | undefined;
|
||||
let singletonSql: ReturnType<typeof buildTestSql> | undefined;
|
||||
|
||||
/** Lazily-built shared Kysely for the test suite (one per worker; maxWorkers=1). */
|
||||
export function getTestDb(): Kysely<any> {
|
||||
if (!singleton) {
|
||||
singleton = buildTestDb();
|
||||
singletonSql = buildTestSql();
|
||||
singleton = new Kysely<any>({
|
||||
dialect: new PostgresJSDialect({ postgres: singletonSql }),
|
||||
plugins: [new CamelCasePlugin()],
|
||||
});
|
||||
}
|
||||
return singleton;
|
||||
}
|
||||
|
||||
export async function destroyTestDb(): Promise<void> {
|
||||
if (singleton) {
|
||||
await singleton.destroy();
|
||||
singleton = undefined;
|
||||
if (!singleton) return;
|
||||
const sql = singletonSql;
|
||||
// Clear the refs first so a hung end() cannot leave a half-closed singleton.
|
||||
singleton = undefined;
|
||||
singletonSql = undefined;
|
||||
// postgres.js .end() waits indefinitely for in-flight queries by default; a
|
||||
// leaked/stuck pooled connection would hang the afterAll hook (a 60s hook
|
||||
// timeout in CI). Bound the shutdown: the { timeout } grace period lets
|
||||
// active queries drain, then force-closes lingering sockets so teardown
|
||||
// always completes. We close the pool directly instead of Kysely.destroy()
|
||||
// (which would call sql.end() again with no timeout).
|
||||
if (sql) {
|
||||
await sql.end({ timeout: 5 });
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -4,10 +4,14 @@
|
||||
"testEnvironment": "node",
|
||||
"testRegex": ".e2e-spec.ts$",
|
||||
"transform": {
|
||||
"prosemirror-markdown/build/.+\\.js$": [
|
||||
"babel-jest",
|
||||
{ "presets": [["@babel/preset-env", { "targets": { "node": "current" } }]] }
|
||||
],
|
||||
"^.+\\.(t|j)sx?$": ["ts-jest", { "tsconfig": { "allowJs": true } }]
|
||||
},
|
||||
"transformIgnorePatterns": [
|
||||
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@sindresorhus[+/][a-z0-9-]+|escape-string-regexp|p-limit|yocto-queue)(@|/))"
|
||||
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@sindresorhus[+/][a-z0-9-]+|escape-string-regexp|p-limit|yocto-queue|@docmost/prosemirror-markdown)(@|/))"
|
||||
],
|
||||
"moduleNameMapper": {
|
||||
"^@docmost/db/(.*)$": "<rootDir>/../src/database/$1",
|
||||
|
||||
@@ -4,14 +4,19 @@
|
||||
"testRegex": ".*\\.int-spec\\.ts$",
|
||||
"testPathIgnorePatterns": ["/node_modules/"],
|
||||
"transform": {
|
||||
"prosemirror-markdown/build/.+\\.js$": [
|
||||
"babel-jest",
|
||||
{ "presets": [["@babel/preset-env", { "targets": { "node": "current" } }]] }
|
||||
],
|
||||
"^.+\\.(t|j)sx?$": "ts-jest"
|
||||
},
|
||||
"transformIgnorePatterns": [
|
||||
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0)(@|/))"
|
||||
"/node_modules/(?!(\\.pnpm/)?(nanoid|uuid|image-dimensions|marked|happy-dom|lib0|@docmost/prosemirror-markdown)(@|/))"
|
||||
],
|
||||
"testEnvironment": "node",
|
||||
"testTimeout": 60000,
|
||||
"maxWorkers": 1,
|
||||
"forceExit": true,
|
||||
"globalSetup": "<rootDir>/test/integration/global-setup.ts",
|
||||
"globalTeardown": "<rootDir>/test/integration/global-teardown.ts",
|
||||
"moduleNameMapper": {
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
// Jest stub for @tiptap/react.
|
||||
//
|
||||
// The server export/import code paths transitively import editor-ext, whose node
|
||||
// extensions import from `@tiptap/react`. The real module re-exports all of
|
||||
// `@tiptap/core` (headless, safe under node) AND adds React view helpers
|
||||
// (`ReactNodeViewRenderer`, …) that eagerly pull in react-dom — which throws
|
||||
// `navigator is not defined` under jest's node environment.
|
||||
//
|
||||
// So this stub DELEGATES to the real `@tiptap/core` (keeping `mergeAttributes`,
|
||||
// `Node`, `Mark`, `nodeInputRule`, … working — they are used by
|
||||
// `jsonToHtml`/`htmlToJson` on the server) and overrides ONLY the React view
|
||||
// helpers with no-ops. Those helpers are referenced solely inside `addNodeView()`
|
||||
// — code that runs only in a live browser editor, never on the server; if any
|
||||
// were actually invoked here it would (correctly) surface as a test failure.
|
||||
const core = require('@tiptap/core');
|
||||
|
||||
module.exports = {
|
||||
...core,
|
||||
ReactNodeViewRenderer: () => () => ({}),
|
||||
NodeViewWrapper: () => null,
|
||||
NodeViewContent: () => null,
|
||||
ReactRenderer: class {},
|
||||
};
|
||||
@@ -131,5 +131,14 @@ const { Client } = require("pg");
|
||||
7. **Migrations don't auto-run in dev** — run `migration:latest` after every pull
|
||||
or branch switch.
|
||||
|
||||
8. **Automation (Playwright): type into the BODY editor, not the title.** A page has
|
||||
two `.ProseMirror` editors — `[aria-label='Page title']` (non-collab) and
|
||||
`[aria-label='Page content']` (the collab body). `document.querySelector('.ProseMirror')`
|
||||
returns the TITLE editor, so typing there never changes body content and `mod+S`
|
||||
versions nothing. Target `[aria-label='Page content']`, confirm it's collab-bound
|
||||
(`el.editor.extensionManager.extensions.some(e=>e.name==='collaboration')`), and
|
||||
wait ~10-12s for the store debounce before asserting `pages.content` changed. Full
|
||||
testing methodology + traps: **[how-to-test.md](how-to-test.md)**.
|
||||
|
||||
See also the **Commands** and **Architecture → Two server processes** sections in
|
||||
[`AGENTS.md`](../AGENTS.md).
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# How to test the application (browser E2E + out-of-band)
|
||||
|
||||
How to actually verify a feature end-to-end against a running stand — driving the
|
||||
**real app in a browser** and confirming results **out-of-band** in the DB/git, not
|
||||
through the same API you're supposed to be testing. Written from real false-positives
|
||||
that wasted hours (see **Traps** — read them before you write a test).
|
||||
|
||||
Prereq: a running stand — see **[dev-stand.md](dev-stand.md)**. Automation uses
|
||||
Playwright (`pip install playwright && python -m playwright install chromium`).
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Drive the behaviour under test through the browser.** The stand exists so you
|
||||
exercise the real UI + realtime-collab + server path. Using `POST /api/pages/*` to
|
||||
perform the action you're validating tests the API, not the app — an e2e suite can
|
||||
do that. API calls are fine ONLY for one-time setup/fixtures, never for the
|
||||
interaction you're asserting on.
|
||||
2. **Evidence before claim.** Nothing "passes" without an artifact: a DB row, a git
|
||||
diff, a screenshot looked at as an image. If you can't show it, you didn't verify it.
|
||||
3. **Verify out-of-band.** Judge results from a source independent of the UI: `psql`
|
||||
against the DB, a fresh `git clone` of a synced repo, a hard reload. Optimistic UI
|
||||
lies about persistence.
|
||||
4. **Disconfirm by default.** For each feature, actively try to prove it's broken
|
||||
before concluding it works. Reload after every create/edit/save.
|
||||
5. **Recon actuatability FIRST.** Before building editor tests, confirm the
|
||||
interaction even works in your harness (does a typed edit reach the DB?). Skipping
|
||||
this is how you ship a pile of tests that all silently exercised the wrong thing.
|
||||
|
||||
## The editor: two ProseMirror instances (READ THIS)
|
||||
|
||||
A page has **two** `.ProseMirror` editors:
|
||||
|
||||
| index | selector | role | collab? |
|
||||
|---|---|---|---|
|
||||
| 0 | `[aria-label='Page title']` | title field | **NO** (16 exts, no `collaboration`) |
|
||||
| 1 | `[aria-label='Page content']` | body | **YES** (95 exts, has `collaboration`) |
|
||||
|
||||
`document.querySelector('.ProseMirror')` returns the **title** editor (first match).
|
||||
Type there and you edit the title only — body page content never changes, so `mod+S`
|
||||
"versions" unchanged content and every content test silently no-ops.
|
||||
|
||||
**Always target the body editor** and confirm it's collab-bound before typing:
|
||||
|
||||
```js
|
||||
const el = document.querySelector("[aria-label='Page content']");
|
||||
el.editor.extensionManager.extensions.some(e => e.name === 'collaboration'); // must be true
|
||||
```
|
||||
|
||||
Body edits emit ~20 `/collab` websocket frames while typing and land in
|
||||
`pages.content` after the **hocuspocus store debounce (~10s)** — so **wait ~12s**
|
||||
before asserting persistence (checking at 6–8s is a false negative). `mod+S` (the
|
||||
`save-version` stateless message) flushes immediately, so a version created right
|
||||
after a settled body edit holds the typed text.
|
||||
|
||||
## A known-good browser flow
|
||||
|
||||
```
|
||||
1. goto /s/<space-slug> # the "Create page" button lives in the space sidebar, not /home
|
||||
2. click button[aria-label='Create page'] # fully UI-driven page creation
|
||||
3. type into [aria-label='Page title'] # optional title
|
||||
4. click [aria-label='Page content'] → type body text
|
||||
5. wait ~12s (store debounce)
|
||||
6. assert pages.content changed (psql) # out-of-band
|
||||
7. mod+S / menu Save → assert page_history row (psql)
|
||||
8. reload / fresh context → re-assert (persistence round-trip)
|
||||
```
|
||||
|
||||
Auth: log in ONCE, save `storage_state.json`, reuse it across pages/agents (re-login
|
||||
per run trips shared rate-limits). Cookie-based session authorizes both REST and the
|
||||
collab websocket.
|
||||
|
||||
## Judging out-of-band
|
||||
|
||||
```bash
|
||||
# page content / history
|
||||
docker exec <db> psql -U docmost -d docmost -tAc \
|
||||
"select coalesce(kind,'null'), content::text from page_history where page_id='<id>' order by created_at;"
|
||||
# git-sync round-trip: clone the space repo and diff against what you pushed
|
||||
git clone http://<user>:<pass>@127.0.0.1:3000/git/<spaceId>.git /tmp/x
|
||||
```
|
||||
|
||||
`page_history.content` is full JSON — parse it, don't truncate the snippet, or a
|
||||
marker check misses. For sync/async features (autosave, git-sync, idle-flush) use an
|
||||
active probe: write a unique marker, wait past the debounce/poll window, re-read
|
||||
out-of-band, ≥2 iterations — never conclude "broken" from a single snapshot.
|
||||
|
||||
## Traps (each of these produced a false result in a real run)
|
||||
|
||||
- **Wrong editor.** Typed into `.ProseMirror` (= title). Edits never touched body
|
||||
content. → target `[aria-label='Page content']`.
|
||||
- **Checked persistence too early.** Store debounce ~10s; a 6–8s check reads stale.
|
||||
- **Truncated the DB snapshot** below where the test marker sits → false "content
|
||||
missing".
|
||||
- **API-seeded the content under test**, then "verified" the feature — that validated
|
||||
the API, not the app.
|
||||
- **Reused a fixed marker on a non-rebooted stand** → title/row collisions inflate
|
||||
counts (`count==2`). Use a unique per-run marker (timestamp).
|
||||
- **Idle/async read once** and called it "permanently broken" — it was mid-debounce.
|
||||
- **Concluded env-limitation without a cross-build control.** If unsure whether a
|
||||
failure is your harness or the product, run the SAME harness against a known-good
|
||||
build; a divergence localizes it.
|
||||
|
||||
## Scope note
|
||||
|
||||
Some paths genuinely need a human in a real browser (rich drag-drop, native file
|
||||
pickers, clipboard, and anything the harness can't actuate). Label those UNTESTED in
|
||||
the report — "handled gracefully" is not "works". Keep four states distinct:
|
||||
verified-working, defect, untested, env-limitation.
|
||||
+2
-1
@@ -96,7 +96,8 @@
|
||||
"pnpm": {
|
||||
"patchedDependencies": {
|
||||
"scimmy@1.3.5": "patches/scimmy@1.3.5.patch",
|
||||
"yjs@13.6.30": "patches/yjs@13.6.30.patch"
|
||||
"yjs@13.6.30": "patches/yjs@13.6.30.patch",
|
||||
"ai@6.0.134": "patches/ai@6.0.134.patch"
|
||||
},
|
||||
"overrides": {
|
||||
"prosemirror-changeset": "2.4.0",
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
import { describe, it, expect, vi, afterEach } from 'vitest';
|
||||
import { getSchema } from '@tiptap/core';
|
||||
import { Document } from '@tiptap/extension-document';
|
||||
import { Paragraph } from '@tiptap/extension-paragraph';
|
||||
import { Text } from '@tiptap/extension-text';
|
||||
import { EditorState } from '@tiptap/pm/state';
|
||||
import { Node as PMNode } from '@tiptap/pm/model';
|
||||
import { FootnoteReference } from './footnote-reference';
|
||||
import { FootnotesList } from './footnotes-list';
|
||||
import { FootnoteDefinition } from './footnote-definition';
|
||||
import {
|
||||
footnoteNumberingPlugin,
|
||||
footnoteNumberingPluginKey,
|
||||
getFootnoteNumber,
|
||||
} from './footnote-numbering';
|
||||
import {
|
||||
FOOTNOTE_REFERENCE_NAME,
|
||||
FOOTNOTES_LIST_NAME,
|
||||
FOOTNOTE_DEFINITION_NAME,
|
||||
} from './footnote-util';
|
||||
|
||||
const extensions = [
|
||||
Document,
|
||||
Paragraph,
|
||||
Text,
|
||||
FootnoteReference,
|
||||
FootnotesList,
|
||||
FootnoteDefinition,
|
||||
];
|
||||
|
||||
const schema = getSchema(extensions);
|
||||
|
||||
function makeState(docJson: any): EditorState {
|
||||
return EditorState.create({
|
||||
doc: PMNode.fromJSON(schema, docJson),
|
||||
plugins: [footnoteNumberingPlugin()],
|
||||
});
|
||||
}
|
||||
|
||||
const withTwoFootnotes = {
|
||||
type: 'doc',
|
||||
content: [
|
||||
{
|
||||
type: 'paragraph',
|
||||
content: [
|
||||
{ type: 'text', text: 'a' },
|
||||
{ type: FOOTNOTE_REFERENCE_NAME, attrs: { id: 'x' } },
|
||||
{ type: 'text', text: 'b' },
|
||||
{ type: FOOTNOTE_REFERENCE_NAME, attrs: { id: 'y' } },
|
||||
],
|
||||
},
|
||||
{
|
||||
type: FOOTNOTES_LIST_NAME,
|
||||
content: [
|
||||
{
|
||||
type: FOOTNOTE_DEFINITION_NAME,
|
||||
attrs: { id: 'x' },
|
||||
content: [{ type: 'paragraph' }],
|
||||
},
|
||||
{
|
||||
type: FOOTNOTE_DEFINITION_NAME,
|
||||
attrs: { id: 'y' },
|
||||
content: [{ type: 'paragraph' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
describe('footnote numbering plugin — short-circuit (#343 PART 5)', () => {
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
it('does ZERO document traversals on a docChanged transaction when the doc has no footnotes', () => {
|
||||
const state = makeState({
|
||||
type: 'doc',
|
||||
content: [{ type: 'paragraph', content: [{ type: 'text', text: 'hi' }] }],
|
||||
});
|
||||
|
||||
// Only count traversals caused by the transaction, not the initial build.
|
||||
const descendantsSpy = vi.spyOn(PMNode.prototype, 'descendants');
|
||||
|
||||
const before = footnoteNumberingPluginKey.getState(state);
|
||||
// A real content edit (docChanged) that introduces no footnote node.
|
||||
const next = state.apply(state.tr.insertText('!', 3));
|
||||
const after = footnoteNumberingPluginKey.getState(next);
|
||||
|
||||
// The plugin never walked the document...
|
||||
expect(descendantsSpy).not.toHaveBeenCalled();
|
||||
// ...and reused the exact same (empty) state object — proof it short-circuited.
|
||||
expect(after).toBe(before);
|
||||
expect(after?.hasFootnotes).toBe(false);
|
||||
});
|
||||
|
||||
it('rebuilds (numbering appears) the first time a footnote is inserted into a footnote-free doc', () => {
|
||||
const state = makeState({
|
||||
type: 'doc',
|
||||
content: [{ type: 'paragraph', content: [{ type: 'text', text: 'hi' }] }],
|
||||
});
|
||||
expect(footnoteNumberingPluginKey.getState(state)?.hasFootnotes).toBe(false);
|
||||
|
||||
const ref = schema.nodes[FOOTNOTE_REFERENCE_NAME].create({ id: 'x' });
|
||||
const next = state.apply(state.tr.insert(3, ref));
|
||||
|
||||
const after = footnoteNumberingPluginKey.getState(next);
|
||||
expect(after?.hasFootnotes).toBe(true);
|
||||
expect(getFootnoteNumber(next, 'x')).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('footnote numbering plugin — numbering unchanged with footnotes (#343 PART 5)', () => {
|
||||
it('numbers references in document order via the single merged walk', () => {
|
||||
const state = makeState(withTwoFootnotes);
|
||||
expect(getFootnoteNumber(state, 'x')).toBe(1);
|
||||
expect(getFootnoteNumber(state, 'y')).toBe(2);
|
||||
});
|
||||
|
||||
it('produces a decoration for every reference and matching definition', () => {
|
||||
const state = makeState(withTwoFootnotes);
|
||||
const decos = footnoteNumberingPluginKey.getState(state)?.decorations;
|
||||
// 2 references + 2 definitions = 4 number decorations.
|
||||
expect(decos?.find().length).toBe(4);
|
||||
});
|
||||
|
||||
it('keeps numbering current after an edit while footnotes exist', () => {
|
||||
const state = makeState(withTwoFootnotes);
|
||||
// Insert a NEW reference (id "z") before the others: it must become #1 and
|
||||
// shift x -> #2, y -> #3 (deterministic document-order numbering).
|
||||
const ref = schema.nodes[FOOTNOTE_REFERENCE_NAME].create({ id: 'z' });
|
||||
const next = state.apply(state.tr.insert(1, ref));
|
||||
expect(getFootnoteNumber(next, 'z')).toBe(1);
|
||||
expect(getFootnoteNumber(next, 'x')).toBe(2);
|
||||
expect(getFootnoteNumber(next, 'y')).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -1,11 +1,9 @@
|
||||
import { EditorState, Plugin, PluginKey } from '@tiptap/pm/state';
|
||||
import { EditorState, Plugin, PluginKey, Transaction } from '@tiptap/pm/state';
|
||||
import { Decoration, DecorationSet } from '@tiptap/pm/view';
|
||||
import { Node as ProseMirrorNode } from '@tiptap/pm/model';
|
||||
import { Node as ProseMirrorNode, Slice } from '@tiptap/pm/model';
|
||||
import {
|
||||
FOOTNOTE_DEFINITION_NAME,
|
||||
FOOTNOTE_REFERENCE_NAME,
|
||||
computeFootnoteNumbers,
|
||||
computeFootnoteRefCounts,
|
||||
} from './footnote-util';
|
||||
|
||||
export const footnoteNumberingPluginKey = new PluginKey<FootnoteNumberingState>(
|
||||
@@ -27,8 +25,22 @@ interface FootnoteNumberingState {
|
||||
refCounts: Map<string, number>;
|
||||
/** Decorations rendering those numbers (refs + definitions). */
|
||||
decorations: DecorationSet;
|
||||
/** Whether the document contains ANY footnote reference/definition node.
|
||||
* Cached so `apply` can skip the whole-doc walk on every keystroke in the
|
||||
* common case (documents with no footnotes), recomputing only once a
|
||||
* transaction actually inserts a footnote node (#343, PART 5). */
|
||||
hasFootnotes: boolean;
|
||||
}
|
||||
|
||||
/** Reusable empty state for footnote-free documents — avoids reallocating an
|
||||
* empty map/decoration set on every keystroke while there are no footnotes. */
|
||||
const EMPTY_STATE: FootnoteNumberingState = {
|
||||
numbers: new Map(),
|
||||
refCounts: new Map(),
|
||||
decorations: DecorationSet.empty,
|
||||
hasFootnotes: false,
|
||||
};
|
||||
|
||||
/**
|
||||
* Build the decoration set for footnote numbers. Pure function of the document:
|
||||
* walk references in document order, assign 1-based numbers, then attach a
|
||||
@@ -41,50 +53,101 @@ export function buildFootnoteDecorations(doc: ProseMirrorNode): DecorationSet {
|
||||
return buildFootnoteNumberingState(doc).decorations;
|
||||
}
|
||||
|
||||
function numberDecoration(pos: number, nodeSize: number, num: number): Decoration {
|
||||
return Decoration.node(pos, pos + nodeSize, {
|
||||
'data-footnote-number': String(num),
|
||||
style: `--footnote-number: "${num}";`,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute both the number map AND the decorations for `doc` in a single walk.
|
||||
* The plugin caches the result so NodeViews can read numbers without
|
||||
* recomputing.
|
||||
* Compute the number map, reference counts AND the decorations for `doc` in a
|
||||
* SINGLE document walk (previously three separate O(n) traversals per
|
||||
* docChanged — computeFootnoteNumbers + computeFootnoteRefCounts + a decoration
|
||||
* pass, #343 PART 5). The plugin caches the result so NodeViews can read numbers
|
||||
* without recomputing.
|
||||
*
|
||||
* References are numbered and decorated as they are encountered (document
|
||||
* order). Definition positions are collected during the same walk and decorated
|
||||
* afterwards from the completed number map — so a definition that appears before
|
||||
* its reference in document order still resolves to the correct number, and the
|
||||
* output is identical to the previous three-pass implementation. (Decoration
|
||||
* insertion order does not matter: DecorationSet.create indexes by position.)
|
||||
*/
|
||||
function buildFootnoteNumberingState(
|
||||
doc: ProseMirrorNode,
|
||||
): FootnoteNumberingState {
|
||||
const numbers = computeFootnoteNumbers(doc);
|
||||
const refCounts = computeFootnoteRefCounts(doc);
|
||||
const numbers = new Map<string, number>();
|
||||
const refCounts = new Map<string, number>();
|
||||
const decorations: Decoration[] = [];
|
||||
const definitions: { id: string; pos: number; nodeSize: number }[] = [];
|
||||
let n = 0;
|
||||
let hasFootnotes = false;
|
||||
|
||||
doc.descendants((node, pos) => {
|
||||
if (node.type.name === FOOTNOTE_REFERENCE_NAME) {
|
||||
const num = numbers.get(node.attrs.id);
|
||||
if (num != null) {
|
||||
decorations.push(
|
||||
Decoration.node(pos, pos + node.nodeSize, {
|
||||
'data-footnote-number': String(num),
|
||||
style: `--footnote-number: "${num}";`,
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (node.type.name === FOOTNOTE_DEFINITION_NAME) {
|
||||
const num = numbers.get(node.attrs.id);
|
||||
if (num != null) {
|
||||
decorations.push(
|
||||
Decoration.node(pos, pos + node.nodeSize, {
|
||||
'data-footnote-number': String(num),
|
||||
style: `--footnote-number: "${num}";`,
|
||||
}),
|
||||
);
|
||||
const typeName = node.type.name;
|
||||
if (typeName === FOOTNOTE_REFERENCE_NAME) {
|
||||
hasFootnotes = true;
|
||||
const id = node.attrs.id;
|
||||
if (id) {
|
||||
if (!numbers.has(id)) numbers.set(id, ++n);
|
||||
refCounts.set(id, (refCounts.get(id) ?? 0) + 1);
|
||||
decorations.push(numberDecoration(pos, node.nodeSize, numbers.get(id)!));
|
||||
}
|
||||
} else if (typeName === FOOTNOTE_DEFINITION_NAME) {
|
||||
hasFootnotes = true;
|
||||
const id = node.attrs.id;
|
||||
if (id != null) definitions.push({ id, pos, nodeSize: node.nodeSize });
|
||||
}
|
||||
});
|
||||
|
||||
if (!hasFootnotes) return EMPTY_STATE;
|
||||
|
||||
for (const def of definitions) {
|
||||
const num = numbers.get(def.id);
|
||||
if (num != null) {
|
||||
decorations.push(numberDecoration(def.pos, def.nodeSize, num));
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
numbers,
|
||||
refCounts,
|
||||
decorations: DecorationSet.create(doc, decorations),
|
||||
hasFootnotes: true,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Cheap check: does any of a transaction's inserted content contain a footnote
|
||||
* reference/definition node? Footnote nodes can only ENTER the document through
|
||||
* replace steps (ReplaceStep / ReplaceAroundStep both expose a `.slice`), so
|
||||
* scanning only the inserted slices — O(change size), not O(doc) — is sufficient
|
||||
* to detect a newly-added footnote. Mark/attr steps never introduce nodes.
|
||||
* Lets `apply` keep skipping the whole-doc walk until a footnote first appears.
|
||||
*/
|
||||
function transactionInsertsFootnote(tr: Transaction): boolean {
|
||||
for (const step of tr.steps) {
|
||||
const slice = (step as unknown as { slice?: Slice }).slice;
|
||||
if (!slice || slice.content.size === 0) continue;
|
||||
let found = false;
|
||||
slice.content.descendants((node) => {
|
||||
if (found) return false;
|
||||
const typeName = node.type.name;
|
||||
if (
|
||||
typeName === FOOTNOTE_REFERENCE_NAME ||
|
||||
typeName === FOOTNOTE_DEFINITION_NAME
|
||||
) {
|
||||
found = true;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
if (found) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the cached footnote number for `id` from the numbering plugin's state.
|
||||
* This is the source NodeViews should use instead of calling
|
||||
@@ -126,6 +189,13 @@ export function footnoteNumberingPlugin(): Plugin {
|
||||
// the number map NodeViews read stays current on every edit while
|
||||
// non-doc transactions (selection, etc.) reuse the cache for free.
|
||||
if (!tr.docChanged) return old;
|
||||
// Short-circuit the whole-doc walk while the document has no footnotes:
|
||||
// if there were none and this transaction did not INSERT one, there is
|
||||
// still nothing to number, so reuse the empty state (#343, PART 5). Once
|
||||
// a footnote exists we always rebuild (covers renumbering/deletion).
|
||||
if (!old.hasFootnotes && !transactionInsertsFootnote(tr)) {
|
||||
return old;
|
||||
}
|
||||
return buildFootnoteNumberingState(tr.doc);
|
||||
},
|
||||
},
|
||||
|
||||
Vendored
+16
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"_bug": "BUG #351: a `column` whose `width` is a percentage string (e.g. \"50%\") is NOT byte-stable across export->import->export (violates P2). The `column` schema's parseHTML does `parseFloat(getAttribute('data-width'))`, which silently drops the '%' unit and returns the NUMBER 50. So the first export emits data-width=\"50%\" but the re-import stores width=50, and the second export emits data-width=\"50\": md2 !== md1, a permanent GS-EDIT-REVERT churn (every git-sync pull rewrites the column width). The editor authors column widths as percentages, so this is a real data/round-trip defect. Fix belongs in src/lib/docmost-schema.ts column.width parseHTML (preserve the unit / keep the string), which is OUT OF SCOPE for this test-only PR and must be a separate, maintainer-approved change. This flat generator therefore keeps `column.width` frozen (never generates a non-default width).",
|
||||
"doc": {
|
||||
"type": "doc",
|
||||
"content": [
|
||||
{
|
||||
"type": "columns",
|
||||
"attrs": { "layout": "two_equal", "widthMode": "normal" },
|
||||
"content": [
|
||||
{ "type": "column", "attrs": { "width": "50%" }, "content": [{ "type": "paragraph", "content": [{ "type": "text", "text": "L" }] }] },
|
||||
{ "type": "column", "attrs": { "width": "50%" }, "content": [{ "type": "paragraph", "content": [{ "type": "text", "text": "R" }] }] }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"doc": {
|
||||
"type": "doc",
|
||||
"content": [
|
||||
{
|
||||
"type": "orderedList",
|
||||
"attrs": { "type": null, "start": 5 },
|
||||
"content": [
|
||||
{
|
||||
"type": "listItem",
|
||||
"content": [
|
||||
{ "type": "paragraph", "content": [{ "type": "text", "text": "alpha" }] }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,325 @@
|
||||
/**
|
||||
* Schema-DERIVED attribute-state fast-check arbitraries (#351, PR 1).
|
||||
*
|
||||
* This GENERALIZES the #350 stability-matrix helper (roundtrip-stability.helper.ts)
|
||||
* to fast-check. Where that helper sweeps a HAND-WRITTEN 2-state matrix for one
|
||||
* node spec, this module reads the attribute list straight from
|
||||
* `schema.nodes[type].spec.attrs` (never a hand list) and, per attribute,
|
||||
* generates over the FOUR states the issue calls for:
|
||||
*
|
||||
* - `absent` : the attribute is OMITTED entirely (the empty-string-vs-
|
||||
* absent churn class the #350 fix targets).
|
||||
* - `default` : the schema default value, authored explicitly.
|
||||
* - `nonDefault` : a representative legal non-default value.
|
||||
* - `degenerate` : `""` for strings, `0`/negative for numbers, the flipped
|
||||
* value for booleans.
|
||||
*
|
||||
* ── Why a per-attribute override table ──────────────────────────────────────
|
||||
* Everything that CAN be derived generically from the default's runtime type is
|
||||
* (booleans flip; the degenerate value follows the runtime type). But two facts
|
||||
* force a small, DOCUMENTED override table:
|
||||
*
|
||||
* 1. CONSTRAINED domains the schema does not encode. `image.align ∈
|
||||
* {left,center,right}`, `heading.level ∈ 1..6`, `callout.type ∈
|
||||
* {info,success,warning,danger}`, `columns.layout`, table-cell `align`,
|
||||
* `status.color`, `orderedList.start ≥ 1`, etc. A generic "default + 1"
|
||||
* would emit an ILLEGAL value, so these get an explicit legal domain.
|
||||
* 2. ROUND-TRIP-safety, established EMPIRICALLY by probing the live converter
|
||||
* (the classification captured in flat-roundtrip.property.test.ts). A frozen
|
||||
* attribute falls into ONE of TWO explicitly-distinguished classes — never a
|
||||
* silent "it just doesn't round-trip":
|
||||
*
|
||||
* (a) ACCEPTED LIMITATION — the attribute has NO markdown representation,
|
||||
* so the loss is inherent to targeting markdown, not a converter
|
||||
* defect. These: `paragraph`/`heading` `indent`, `callout.icon`,
|
||||
* `orderedList.type` (a/A/i markers), table `colwidth` /
|
||||
* `backgroundColor(Name)` (dropped by the raw-<table> fallback). Each is
|
||||
* tagged `// ACCEPTED:` inline. Freezing them is correct — there is
|
||||
* nothing to preserve in the target format.
|
||||
*
|
||||
* (b) PINNED BUG — the attribute IS representable in markdown but the
|
||||
* converter drops it anyway (a real defect). These are NOT silently
|
||||
* frozen: each is captured as a LOUD `it.fails` counterexample in
|
||||
* test/fixtures/counterexamples/ + counterexamples.test.ts, and the
|
||||
* freeze here only keeps the P1/P2 union green until a MAINTAINER rules
|
||||
* on accept-vs-fix (the epic guardrail reserves that call). These:
|
||||
* `column.width` (parseFloat drops `%`), `orderedList.start` (non-1
|
||||
* start renders as `1.`). Tagged `// PINNED-BUG:` inline.
|
||||
*
|
||||
* (c) DEFERRED-BUG — representable AND round-trips, frozen only because the
|
||||
* flat generator can't yet build a valid instance. Table
|
||||
* `colspan`/`rowspan` round-trip via the raw-<table> fallback, but a
|
||||
* geometrically-valid spanned table is PR-2 structural work; the flat
|
||||
* generator hardcodes span = 1. Tagged `// DEFERRED-BUG:` inline so a
|
||||
* maintainer does not read them as an inherent limitation.
|
||||
* - Several non-null-default attrs are MATERIALIZED on import but are not
|
||||
* in canonicalize's KNOWN_DEFAULTS (`callout.type`, `status.color`,
|
||||
* table `colspan`/`rowspan`, `columns.layout`/`widthMode`,
|
||||
* `embed.width`/`height`, `heading.level`, `taskItem.checked`,
|
||||
* `details.open`, `subpages.recursive`, `orderedList.start`). If left
|
||||
* `absent` they re-materialize as a non-canonical default and diverge
|
||||
* under P1. We mark them `always` so they are authored explicitly.
|
||||
* - The documented numeric→string coercion set (`width height size
|
||||
* aspectRatio`) is generated as STRINGS for the media family (a stored
|
||||
* number re-parses as a string), EXCEPT `embed.width/height` which the
|
||||
* embed schema keeps numeric — handled per-attr.
|
||||
*
|
||||
* Both PINNED-BUG attrs (`column.width` P2 churn, `orderedList.start` P1 loss)
|
||||
* are captured as committed `it.fails` counterexamples — NOT hidden here.
|
||||
*/
|
||||
import fc from 'fast-check';
|
||||
import { getSchema } from '@tiptap/core';
|
||||
import { docmostExtensions } from '../../src/lib/index.js';
|
||||
import { phraseArb, letterPhraseArb, urlArb } from './text-arbitraries.js';
|
||||
|
||||
/** The exact ProseMirror schema the converter targets. */
|
||||
export const schema = getSchema(docmostExtensions as any);
|
||||
|
||||
/** Sentinel: this attribute is OMITTED (the `absent` state). */
|
||||
export const ABSENT = Symbol('ABSENT');
|
||||
|
||||
/** The documented numeric→string coercion set (issue + roundtrip-stability.helper). */
|
||||
export const NUMERIC_STRING_ATTRS = ['width', 'height', 'size', 'aspectRatio'];
|
||||
|
||||
/** Read the schema default for every attribute of a node type. */
|
||||
export function schemaAttrDefaults(type: string): Record<string, unknown> {
|
||||
const specAttrs = (schema.nodes[type]?.spec?.attrs ?? {}) as Record<
|
||||
string,
|
||||
{ default: unknown }
|
||||
>;
|
||||
const out: Record<string, unknown> = {};
|
||||
for (const [k, v] of Object.entries(specAttrs)) out[k] = v.default;
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Attribute names for a node type, straight from the schema (never hand-listed). */
|
||||
export function schemaAttrNames(type: string): string[] {
|
||||
return Object.keys((schema.nodes[type]?.spec?.attrs ?? {}) as object);
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-attribute policy. Everything unlisted falls back to a generic policy:
|
||||
* - a BOOLEAN default is fuzzable (its non-default is the flipped value);
|
||||
* - any other default is `frozen` (only `absent`/`default` are generated) so
|
||||
* we never invent an unverified non-default that might not round-trip.
|
||||
* Listed attrs override this with a legal `arb` domain and/or flags.
|
||||
*/
|
||||
interface AttrPolicy {
|
||||
/** Arbitrary for the `nonDefault` state's value. */
|
||||
arb?: fc.Arbitrary<unknown>;
|
||||
/** Value for the `degenerate` state (fuzz mode only). Omit to skip degenerate. */
|
||||
degen?: unknown;
|
||||
/** Never emit `absent` — the attr must be authored (materialized default class). */
|
||||
always?: boolean;
|
||||
/** Never emit the schema default value (required-ish attrs like `src`). Implies always. */
|
||||
noDefault?: boolean;
|
||||
/** Never emit non-default/degenerate — attr has no md representation or churns. */
|
||||
frozen?: boolean;
|
||||
}
|
||||
|
||||
const num = (...xs: number[]) => fc.constantFrom(...xs);
|
||||
const str = (...xs: string[]) => fc.constantFrom(...xs);
|
||||
const widthStr = str('120', '320', '640');
|
||||
|
||||
// The documented override table, keyed `type.attr`. Every entry is grounded in
|
||||
// the empirical converter probe (see flat-roundtrip.property.test.ts header).
|
||||
const OVERRIDES: Record<string, AttrPolicy> = {
|
||||
// ── block text containers ────────────────────────────────────────────────
|
||||
// 'left' is the IMPLICIT default alignment: the converter drops it on export
|
||||
// (empirically confirmed), so it never round-trips. Only center/right/justify
|
||||
// carry through the `<!--attrs {textAlign}-->` comment.
|
||||
'paragraph.textAlign': { arb: str('center', 'right', 'justify') },
|
||||
'paragraph.indent': { frozen: true }, // ACCEPTED: no md representation
|
||||
'heading.level': { always: true, arb: num(2, 3, 4, 5, 6) },
|
||||
'heading.textAlign': { arb: str('center', 'right', 'justify') },
|
||||
'heading.indent': { frozen: true }, // ACCEPTED: no md representation
|
||||
// ── lists ────────────────────────────────────────────────────────────────
|
||||
// PINNED-BUG: markdown CAN express a non-1 start ("5."), but the converter
|
||||
// renders "1." and drops it -> P1 loss. See counterexamples.test.ts
|
||||
// (ordered-list-start.json). Frozen only until the maintainer rules accept-vs-fix.
|
||||
'orderedList.start': { always: true, frozen: true },
|
||||
'orderedList.type': { frozen: true }, // ACCEPTED: a/A/i markers not expressible in GFM
|
||||
'taskItem.checked': { always: true, arb: fc.constant(true) }, // boolean, default false
|
||||
// ── codeBlock ────────────────────────────────────────────────────────────
|
||||
'codeBlock.language': { arb: str('js', 'ts', 'python', 'go', 'rust', 'bash') },
|
||||
// ── image / media (numeric→string width family) ──────────────────────────
|
||||
'image.src': { noDefault: true, arb: urlArb, degen: '' },
|
||||
'image.align': { arb: str('left', 'right') },
|
||||
'image.alt': { arb: letterPhraseArb, degen: '' },
|
||||
'image.title': { arb: letterPhraseArb },
|
||||
'image.width': { arb: widthStr, degen: '' },
|
||||
'image.height': { arb: widthStr, degen: '' },
|
||||
'video.src': { noDefault: true, arb: urlArb, degen: '' },
|
||||
'video.alt': { arb: letterPhraseArb },
|
||||
'video.width': { arb: widthStr },
|
||||
'video.height': { arb: widthStr },
|
||||
'audio.src': { noDefault: true, arb: urlArb, degen: '' },
|
||||
'youtube.src': { noDefault: true, arb: urlArb },
|
||||
'pdf.src': { noDefault: true, arb: urlArb },
|
||||
'pdf.name': { arb: phraseArb },
|
||||
'drawio.src': { noDefault: true, arb: urlArb },
|
||||
'excalidraw.src': { noDefault: true, arb: urlArb },
|
||||
'attachment.url': { noDefault: true, arb: urlArb },
|
||||
'attachment.name': { arb: phraseArb },
|
||||
// ── callout / status ─────────────────────────────────────────────────────
|
||||
'callout.type': { always: true, arb: str('success', 'warning', 'danger') },
|
||||
'callout.icon': { frozen: true }, // ACCEPTED: no md representation (dropped on export)
|
||||
'status.text': { noDefault: true, arb: phraseArb, degen: '' },
|
||||
'status.color': { always: true, arb: str('green', 'orange', 'red', 'blue', 'yellow', 'purple') },
|
||||
// ── table cells ────────────────────────────────────────────────────────────
|
||||
// DEFERRED-BUG (not ACCEPTED): colspan/rowspan ARE representable and round-trip
|
||||
// — a spanned cell makes the converter emit the whole table as a raw <table>
|
||||
// with colspan/rowspan attrs (markdown-converter.ts tableToHtml), which the
|
||||
// tiptap parser reads back. They are frozen only because generating a
|
||||
// geometrically-valid spanned table is deferred STRUCTURAL work (the flat
|
||||
// generator hardcodes colspan/rowspan = 1), NOT a markdown limitation.
|
||||
'tableCell.colspan': { always: true, frozen: true },
|
||||
'tableCell.rowspan': { always: true, frozen: true },
|
||||
// ACCEPTED: colwidth / backgroundColor(Name) have no representation — the
|
||||
// raw-<table> fallback (tableToHtml) drops them, so there is nothing to preserve.
|
||||
'tableCell.colwidth': { frozen: true },
|
||||
'tableCell.backgroundColor': { frozen: true },
|
||||
'tableCell.backgroundColorName': { frozen: true },
|
||||
'tableCell.align': { arb: str('left', 'center', 'right') },
|
||||
'tableHeader.colspan': { always: true, frozen: true }, // DEFERRED-BUG (see tableCell.colspan)
|
||||
'tableHeader.rowspan': { always: true, frozen: true }, // DEFERRED-BUG (see tableCell.rowspan)
|
||||
'tableHeader.colwidth': { frozen: true }, // ACCEPTED: no representation
|
||||
'tableHeader.backgroundColor': { frozen: true }, // ACCEPTED: no representation
|
||||
'tableHeader.backgroundColorName': { frozen: true }, // ACCEPTED: no representation
|
||||
'tableHeader.align': { arb: str('left', 'center', 'right') },
|
||||
// ── details ──────────────────────────────────────────────────────────────
|
||||
'details.open': { always: true, arb: fc.constant(true) }, // boolean, default false
|
||||
// ── columns ──────────────────────────────────────────────────────────────
|
||||
'columns.layout': { always: true, arb: str('three_equal', 'left_sidebar', 'right_sidebar') },
|
||||
// widthMode round-trips via the `data-width-mode` attribute (verified P1+P2),
|
||||
// so it is fuzzed, not frozen.
|
||||
'columns.widthMode': { always: true, arb: str('custom') },
|
||||
// PINNED-BUG: parseFloat import drops the `%` unit -> P2 churn. See
|
||||
// counterexamples.test.ts (columns-column-width-percent.json).
|
||||
'column.width': { frozen: true },
|
||||
// ── embed (schema keeps width/height NUMERIC, not string-coerced) ─────────
|
||||
'embed.src': { noDefault: true, arb: urlArb, degen: '' },
|
||||
'embed.provider': { noDefault: true, arb: str('iframe', 'youtube', 'vimeo') },
|
||||
'embed.width': { always: true, frozen: true },
|
||||
'embed.height': { always: true, frozen: true },
|
||||
// ── subpages / math / htmlEmbed ──────────────────────────────────────────
|
||||
'subpages.recursive': { always: true, arb: fc.constant(true) }, // boolean, default false
|
||||
'mathBlock.text': { noDefault: true, arb: str('x^2', 'a < b', '\\frac{1}{2}'), degen: '' },
|
||||
'mathInline.text': { noDefault: true, arb: str('x^2', 'a < b', '\\frac{1}{2}'), degen: '' },
|
||||
'htmlEmbed.source': { noDefault: true, arb: str('<b>hi</b>', '<i>x</i>', '<span>y</span>'), degen: '' },
|
||||
'htmlEmbed.height': { arb: num(200, 300, 400) },
|
||||
// ── footnotes / transclusion / pageEmbed / mention ───────────────────────
|
||||
'footnoteDefinition.id': { noDefault: true, arb: str('fn1', 'fn2', 'note') },
|
||||
'footnoteReference.id': { noDefault: true, arb: str('fn1', 'fn2', 'note') },
|
||||
'pageEmbed.sourcePageId': { noDefault: true, arb: fc.uuid() },
|
||||
'transclusionSource.id': { noDefault: true, arb: str('src1', 'src2') },
|
||||
'transclusionReference.sourcePageId': { noDefault: true, arb: fc.uuid() },
|
||||
'transclusionReference.transclusionId': { noDefault: true, arb: str('tr1', 'tr2') },
|
||||
'mention.id': { noDefault: true, arb: fc.uuid() },
|
||||
'mention.label': { noDefault: true, arb: phraseArb },
|
||||
'mention.entityType': { noDefault: true, arb: str('user') },
|
||||
'mention.entityId': { noDefault: true, arb: fc.uuid() },
|
||||
};
|
||||
|
||||
/** Resolve the effective policy for one attribute (override merged over generic). */
|
||||
function policyFor(type: string, attr: string, def: unknown): AttrPolicy {
|
||||
const override = OVERRIDES[`${type}.${attr}`];
|
||||
if (override) return override;
|
||||
// Generic: booleans are fuzzable via their flipped value; everything else is
|
||||
// frozen (only absent/default) so no unverified non-default is invented.
|
||||
if (typeof def === 'boolean') return { arb: fc.constant(!def) };
|
||||
return { frozen: true };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether an attribute is actually exercised at a NON-DEFAULT value (i.e. its
|
||||
* policy has an `arb`, which the generic fallback does not). Used by the
|
||||
* attribute-coverage snapshot test to make the generic-frozen space VISIBLE: any
|
||||
* string/number attr not in OVERRIDES is silently only tested at absent/default,
|
||||
* so the snapshot pins exactly which attrs are NOT value-fuzzed and forces a
|
||||
* reviewer to look when a new attr lands in that invisible bucket.
|
||||
*/
|
||||
export function attrIsValueFuzzed(type: string, attr: string): boolean {
|
||||
const def = schemaAttrDefaults(type)[attr];
|
||||
return !!policyFor(type, attr, def).arb;
|
||||
}
|
||||
|
||||
/** Every node `type.attr` in the schema (excluding the auto `id`), sorted. */
|
||||
export function allSchemaAttrKeys(): string[] {
|
||||
const keys: string[] = [];
|
||||
for (const type of Object.keys(schema.nodes)) {
|
||||
for (const attr of schemaAttrNames(type)) {
|
||||
if (attr === 'id') continue;
|
||||
keys.push(`${type}.${attr}`);
|
||||
}
|
||||
}
|
||||
return keys.sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* Every MARK attribute in the schema, keyed `mark:<name>.<attr>`, sorted. Marks
|
||||
* are not driven by the node OVERRIDES table (they are fuzzed by the text
|
||||
* generator, text-arbitraries.ts), so their value-fuzz coverage is tracked with a
|
||||
* separate snapshot (see flat-roundtrip.property.test.ts) — without this the
|
||||
* "no invisible coverage hole" guarantee would hold for node attrs only, letting a
|
||||
* new mark attr slip through unfuzzed and unallowlisted.
|
||||
*/
|
||||
export function allSchemaMarkAttrKeys(): string[] {
|
||||
const keys: string[] = [];
|
||||
for (const [name, mark] of Object.entries(schema.marks)) {
|
||||
const attrs = (mark.spec?.attrs ?? {}) as Record<string, unknown>;
|
||||
for (const attr of Object.keys(attrs)) keys.push(`mark:${name}.${attr}`);
|
||||
}
|
||||
return keys.sort();
|
||||
}
|
||||
|
||||
export type AttrMode = 'p1' | 'fuzz';
|
||||
|
||||
/**
|
||||
* Build an arbitrary for ONE attribute's value (or the ABSENT sentinel) across
|
||||
* the states legal for `mode`:
|
||||
* - p1 : absent / default / nonDefault (the round-trip-safe space).
|
||||
* - fuzz : the above PLUS degenerate (P2 tolerates the one-time
|
||||
* normalization; P3 only needs totality).
|
||||
*/
|
||||
export function attrValueArb(
|
||||
type: string,
|
||||
attr: string,
|
||||
mode: AttrMode,
|
||||
): fc.Arbitrary<unknown | typeof ABSENT> {
|
||||
const def = schemaAttrDefaults(type)[attr];
|
||||
const p = policyFor(type, attr, def);
|
||||
|
||||
const states: fc.Arbitrary<unknown | typeof ABSENT>[] = [];
|
||||
if (!p.always && !p.noDefault) states.push(fc.constant(ABSENT));
|
||||
if (!p.noDefault) states.push(fc.constant(def));
|
||||
if (!p.frozen && p.arb) states.push(p.arb);
|
||||
if (mode === 'fuzz' && !p.frozen && p.degen !== undefined) {
|
||||
states.push(fc.constant(p.degen));
|
||||
}
|
||||
if (states.length === 0) states.push(fc.constant(def));
|
||||
return fc.oneof(...states);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build an arbitrary for a node's full `attrs` object over all schema attrs.
|
||||
* `base` pins caller-required attrs (e.g. a concrete `src`) verbatim; any attr
|
||||
* present in `base` is NOT re-generated. Omitted (ABSENT) attrs are dropped.
|
||||
*/
|
||||
export function nodeAttrsArb(
|
||||
type: string,
|
||||
mode: AttrMode,
|
||||
base: Record<string, unknown> = {},
|
||||
): fc.Arbitrary<Record<string, unknown>> {
|
||||
const names = schemaAttrNames(type).filter((n) => !(n in base) && n !== 'id');
|
||||
if (names.length === 0) return fc.constant({ ...base });
|
||||
return fc
|
||||
.tuple(...names.map((n) => attrValueArb(type, n, mode)))
|
||||
.map((vals) => {
|
||||
const attrs: Record<string, unknown> = { ...base };
|
||||
names.forEach((n, i) => {
|
||||
if (vals[i] !== ABSENT) attrs[n] = vals[i];
|
||||
});
|
||||
return attrs;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import path from 'node:path';
|
||||
import { convertProseMirrorToMarkdown } from '../../src/lib/markdown-converter.js';
|
||||
import { markdownToProseMirror } from '../../src/lib/markdown-to-prosemirror.js';
|
||||
import { docsCanonicallyEqual } from '../../src/lib/canonicalize.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// #351 committed counterexamples — REAL round-trip bugs surfaced by the flat
|
||||
// generative probing (attribute level). Each is pinned here as an `it.fails`
|
||||
// (vitest passes ONLY WHILE the assertion still fails), so that the day the
|
||||
// underlying src/ bug is fixed, the `it.fails` starts PASSING and vitest turns
|
||||
// this test RED — forcing us to delete the counterexample and (per the epic
|
||||
// guardrail) tighten the generator. A bare `it.fails` would ship silent
|
||||
// corruption, so every case below carries a loud `// BUG #351:` explanation.
|
||||
//
|
||||
// These bugs are NOT worked around by weakening any property: the offending
|
||||
// attribute is kept OUT of the P1/P2 generators (documented in
|
||||
// attr-arbitraries.ts), and the exact failing document lives here as the
|
||||
// regression pin. FIXING the bug is a separate, maintainer-approved src/ change.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const here = path.dirname(fileURLToPath(import.meta.url));
|
||||
const fixtureDir = path.resolve(here, '../fixtures/counterexamples');
|
||||
|
||||
function loadDoc(file: string): any {
|
||||
return JSON.parse(readFileSync(path.join(fixtureDir, file), 'utf8')).doc;
|
||||
}
|
||||
|
||||
describe('#351 counterexamples (known round-trip bugs, pinned as it.fails)', () => {
|
||||
// BUG #351: a `column` with a PERCENTAGE width ("50%") is not byte-stable.
|
||||
// The column schema parses `data-width` with parseFloat, dropping the '%':
|
||||
// md1 = '...data-width="50%"...' (first export)
|
||||
// re-import stores width = 50 (number)
|
||||
// md2 = '...data-width="50"...' (second export) => md2 !== md1
|
||||
// A permanent GS-EDIT-REVERT churn on every git-sync pull. The editor stores
|
||||
// column widths as percentages, so this is a genuine defect. The fix is in
|
||||
// src/lib/docmost-schema.ts (column.width parseHTML must preserve the unit)
|
||||
// and is out of scope for this test-only PR.
|
||||
it.fails('column percentage width is byte-stable (P2)', async () => {
|
||||
const doc = loadDoc('columns-column-width-percent.json');
|
||||
const md1 = convertProseMirrorToMarkdown(doc);
|
||||
const doc2 = await markdownToProseMirror(md1);
|
||||
const md2 = convertProseMirrorToMarkdown(doc2);
|
||||
// This assertion currently FAILS (md2 drops the '%'), which is exactly what
|
||||
// `it.fails` expects. When the schema is fixed, it will PASS and flip this
|
||||
// test red — our cue to remove the pin.
|
||||
expect(md2).toBe(md1);
|
||||
});
|
||||
|
||||
// BUG #351: an `orderedList` with a non-1 `start` loses its start number.
|
||||
// CommonMark CAN express this ("5." starts the list at 5), but the converter
|
||||
// always emits "1." and ignores `attrs.start` (markdown-converter.ts renders
|
||||
// `${index + 1}.`; the <ol> HTML path also omits `start`):
|
||||
// doc.start = 5 -> md1 = "1. alpha" (start dropped on export)
|
||||
// re-import stores start = 1 => docsCanonicallyEqual(rt, doc) === false
|
||||
// This is a P1 (semantic round-trip) loss of the SAME class as column.width:
|
||||
// representable in markdown, silently dropped by the converter. It is pinned
|
||||
// here as the LOUD counterexample rather than being masked as an "accepted
|
||||
// normalization" in the generator — per the epic guardrail, deciding
|
||||
// accept-vs-fix for a markdown-representable loss is a MAINTAINER call, so this
|
||||
// stays a visible known-bug until the maintainer rules on it. The fix would be
|
||||
// in src/lib/markdown-converter.ts (emit the start number on the first item)
|
||||
// and is out of scope for this test-only PR.
|
||||
it.fails('ordered list start number is preserved (P1)', async () => {
|
||||
const doc = loadDoc('ordered-list-start.json');
|
||||
const md1 = convertProseMirrorToMarkdown(doc);
|
||||
const doc2 = await markdownToProseMirror(md1);
|
||||
// Currently FAILS: doc2.start === 1 while doc.start === 5. When the converter
|
||||
// preserves `start`, this PASSES and flips the test red — remove the pin then.
|
||||
expect(docsCanonicallyEqual(doc2, doc)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,284 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import fc from 'fast-check';
|
||||
// Real converter, imported the same way the sibling property test does.
|
||||
import { convertProseMirrorToMarkdown } from '../../src/lib/markdown-converter.js';
|
||||
// Importing markdownToProseMirror mutates the global DOM via jsdom at module
|
||||
// load (expected, required for @tiptap/html's generateJSON under Node).
|
||||
import { markdownToProseMirror } from '../../src/lib/markdown-to-prosemirror.js';
|
||||
import { docsCanonicallyEqual, canonicalizeContent } from '../../src/lib/index.js';
|
||||
import { firstDivergence } from '../roundtrip-helpers.js';
|
||||
import {
|
||||
schema,
|
||||
allSchemaAttrKeys,
|
||||
allSchemaMarkAttrKeys,
|
||||
attrIsValueFuzzed,
|
||||
} from './attr-arbitraries.js';
|
||||
import {
|
||||
buildGenerators,
|
||||
coveredTypes,
|
||||
KNOWN_UNCOVERED,
|
||||
} from './node-generators.js';
|
||||
|
||||
// ── Attribute-value coverage allowlist ──────────────────────────────────────
|
||||
// The node/mark completeness contract guarantees every TYPE is generated, but
|
||||
// NOT that every attribute is exercised at a NON-DEFAULT value. An attribute
|
||||
// with no `arb` in attr-arbitraries.ts is only ever tested at absent/default —
|
||||
// an INVISIBLE coverage hole (the reviewer's concern). This allowlist makes that
|
||||
// hole EXPLICIT: it is the exact set of attrs deliberately not value-fuzzed, so
|
||||
// a NEW attribute (or a newly-frozen one) that lands in this bucket flips the
|
||||
// snapshot test red and forces a reviewer to classify it. Each belongs to one of:
|
||||
// - internal/opaque ids & placeholders (attachmentId, slugId, placeholder,
|
||||
// creatorId, anchorId) — no meaningful non-default to assert;
|
||||
// - dimensions/among the media family with no standalone md form here
|
||||
// (aspectRatio, size, caption, drawio/excalidraw/pdf/video/youtube w/h/align)
|
||||
// — round-trip candidates deferred to a later PR, not silently dropped;
|
||||
// - ACCEPTED limitations with no md representation (indent, callout.icon,
|
||||
// orderedList.type, table spans/bg/colwidth);
|
||||
// - PINNED bugs (column.width, orderedList.start) tracked in
|
||||
// counterexamples.test.ts.
|
||||
const ATTR_VALUE_FUZZ_ALLOWLIST = new Set<string>([
|
||||
'attachment.attachmentId', 'attachment.mime', 'attachment.placeholder', 'attachment.size',
|
||||
'audio.attachmentId', 'audio.placeholder', 'audio.size',
|
||||
'callout.icon', 'column.width',
|
||||
'drawio.align', 'drawio.alt', 'drawio.aspectRatio', 'drawio.attachmentId',
|
||||
'drawio.height', 'drawio.size', 'drawio.title', 'drawio.width',
|
||||
'embed.align', 'embed.height', 'embed.width',
|
||||
'excalidraw.align', 'excalidraw.alt', 'excalidraw.aspectRatio', 'excalidraw.attachmentId',
|
||||
'excalidraw.height', 'excalidraw.size', 'excalidraw.title', 'excalidraw.width',
|
||||
'heading.indent',
|
||||
'image.aspectRatio', 'image.attachmentId', 'image.caption', 'image.placeholder', 'image.size',
|
||||
'mention.anchorId', 'mention.creatorId', 'mention.slugId',
|
||||
'orderedList.start', 'orderedList.type', 'paragraph.indent',
|
||||
'pdf.attachmentId', 'pdf.height', 'pdf.placeholder', 'pdf.size', 'pdf.width',
|
||||
'tableCell.backgroundColor', 'tableCell.backgroundColorName', 'tableCell.colspan',
|
||||
'tableCell.colwidth', 'tableCell.rowspan',
|
||||
'tableHeader.backgroundColor', 'tableHeader.backgroundColorName', 'tableHeader.colspan',
|
||||
'tableHeader.colwidth', 'tableHeader.rowspan',
|
||||
'video.align', 'video.aspectRatio', 'video.attachmentId', 'video.placeholder', 'video.size',
|
||||
'youtube.align', 'youtube.height', 'youtube.width',
|
||||
]);
|
||||
|
||||
// ── MARK attribute-value coverage ───────────────────────────────────────────
|
||||
// Marks are fuzzed by the text generator (text-arbitraries.ts markedTextRunArb),
|
||||
// not the node OVERRIDES table, so their value-fuzz coverage is tracked with this
|
||||
// separate registry — otherwise the "no invisible coverage hole" guarantee would
|
||||
// hold for node attrs only, and a new mark attr (or a new attributed mark) would
|
||||
// silently escape the fuzz set. Every schema mark attr must be in exactly one of:
|
||||
// MARK_ATTR_FUZZED — actually driven at a non-default value by the generator;
|
||||
// MARK_ATTR_ALLOWLIST — deliberately not value-fuzzed, with a reason.
|
||||
const MARK_ATTR_FUZZED = new Set<string>([
|
||||
'mark:link.href', // markedTextRunArb sets a random webUrl href
|
||||
'mark:link.title', // ...and an optional letter-bearing title
|
||||
'mark:highlight.color', // highlight mark carries a generated color
|
||||
'mark:textStyle.color', // textStyle mark carries a generated color
|
||||
'mark:comment.commentId', // comment anchor id (alphanumeric token)
|
||||
'mark:comment.resolved', // comment resolved flag (rides only when true)
|
||||
]);
|
||||
const MARK_ATTR_ALLOWLIST = new Set<string>([
|
||||
// link presentational/routing attrs: not part of the markdown link surface the
|
||||
// converter emits (it round-trips href + title only), so there is no
|
||||
// non-default value to assert here — a deferred concern for a link-specific
|
||||
// fixture, not the flat generative pass.
|
||||
'mark:link.internal',
|
||||
'mark:link.target',
|
||||
'mark:link.rel',
|
||||
'mark:link.class',
|
||||
]);
|
||||
|
||||
// Each run does a real convert + marked + jsdom parse (~ms). Give ample headroom
|
||||
// so the suite is deterministic regardless of parallel worker load (like the
|
||||
// sibling property file).
|
||||
vi.setConfig({ testTimeout: 30000 });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// #351 PR 1 — GENERATIVE (property-based) round-trip over FLAT (single-node)
|
||||
// documents at the ATTRIBUTE level.
|
||||
//
|
||||
// We assert three invariants for ANY generated valid flat document `d`
|
||||
// (pmToMd = convertProseMirrorToMarkdown, mdToPm = markdownToProseMirror):
|
||||
//
|
||||
// P1 — semantic round-trip (nothing lost):
|
||||
// docsCanonicallyEqual(await mdToPm(pmToMd(d)), d) === true
|
||||
// P2 — byte fixpoint (anti "GS-EDIT-REVERT" churn):
|
||||
// pmToMd(await mdToPm(pmToMd(d))) === pmToMd(d)
|
||||
// P3 — totality: neither converter throws; bounded.
|
||||
//
|
||||
// The generators are schema-DERIVED (attribute lists come from
|
||||
// schema.nodes[type].spec.attrs) and stay inside the round-trip-supported space
|
||||
// proven empirically by probing the live converter (see attr-arbitraries.ts and
|
||||
// text-arbitraries.ts). P1 runs over the safe attribute space; P2/P3 run over
|
||||
// the wider 'fuzz' space that also injects degenerate attribute states, which
|
||||
// P2 tolerates via a one-time first-pass normalization and P3 via totality only.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Fixed seed so every failure is reproducible; fast-check also prints the
|
||||
// shrunk counterexample. numRuns starts modest to keep CI under budget — the
|
||||
// issue's CI target is ~300-500 per property; the nightly / PR 3 will crank
|
||||
// this up further. Each property runs over the UNION (fc.oneof) of all flat
|
||||
// node generators, so the runs are shared across node types (one test per
|
||||
// property keeps the jsdom import cost and memory bounded — a per-generator ×
|
||||
// per-property matrix is ~200 heavy tests that OOMs the worker).
|
||||
const SEED = 20250705;
|
||||
const NUM_RUNS = 300;
|
||||
|
||||
const P1_GENERATORS = buildGenerators('p1');
|
||||
const FUZZ_GENERATORS = buildGenerators('fuzz');
|
||||
|
||||
// Union arbitraries: a single draw picks one node generator, then a document
|
||||
// from it. On failure fast-check prints the shrunk counterexample doc, which
|
||||
// names the offending node type directly.
|
||||
const p1Union = fc.oneof(...P1_GENERATORS.map((g) => g.arb));
|
||||
const fuzzUnion = fc.oneof(...FUZZ_GENERATORS.map((g) => g.arb));
|
||||
|
||||
async function roundTrip(doc: unknown): Promise<{ md1: string; md2: string; doc2: any }> {
|
||||
const md1 = convertProseMirrorToMarkdown(doc);
|
||||
const doc2 = await markdownToProseMirror(md1);
|
||||
const md2 = convertProseMirrorToMarkdown(doc2);
|
||||
return { md1, md2, doc2 };
|
||||
}
|
||||
|
||||
describe('#351 flat generative round-trip — completeness contract', () => {
|
||||
it('every schema node and mark is covered by a generator or explicitly allowlisted', () => {
|
||||
const covered = coveredTypes();
|
||||
const uncovered: string[] = [];
|
||||
|
||||
for (const nodeType of Object.keys(schema.nodes)) {
|
||||
if (covered.has(nodeType)) continue;
|
||||
if (nodeType in KNOWN_UNCOVERED) continue;
|
||||
uncovered.push(`node:${nodeType}`);
|
||||
}
|
||||
for (const markType of Object.keys(schema.marks)) {
|
||||
if (covered.has(`mark:${markType}`)) continue;
|
||||
if (markType in KNOWN_UNCOVERED) continue;
|
||||
uncovered.push(`mark:${markType}`);
|
||||
}
|
||||
|
||||
// A new node/mark added to the schema with no generator AND no allowlist
|
||||
// entry MUST turn this test red — that is the whole point (no silent blind
|
||||
// spots).
|
||||
expect(
|
||||
uncovered,
|
||||
`these schema types have no generator and no KNOWN_UNCOVERED reason:\n ${uncovered.join(
|
||||
'\n ',
|
||||
)}`,
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it('every KNOWN_UNCOVERED entry is a real schema type (no stale allowlist rows)', () => {
|
||||
const all = new Set([...Object.keys(schema.nodes), ...Object.keys(schema.marks)]);
|
||||
for (const t of Object.keys(KNOWN_UNCOVERED)) {
|
||||
expect(all.has(t), `stale KNOWN_UNCOVERED entry: ${t}`).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('every attribute is value-fuzzed OR explicitly allowlisted (no invisible hole)', () => {
|
||||
// Makes the "generic-frozen" coverage hole VISIBLE: any schema attr not
|
||||
// exercised at a non-default value must be a KNOWN entry in the allowlist.
|
||||
// A new attr (or one that loses its `arb`) that falls into the not-fuzzed
|
||||
// bucket without an allowlist row turns this red — no silent blind spots.
|
||||
const unaccounted: string[] = [];
|
||||
for (const key of allSchemaAttrKeys()) {
|
||||
const i = key.indexOf('.');
|
||||
const fuzzed = attrIsValueFuzzed(key.slice(0, i), key.slice(i + 1));
|
||||
if (!fuzzed && !ATTR_VALUE_FUZZ_ALLOWLIST.has(key)) unaccounted.push(key);
|
||||
}
|
||||
expect(
|
||||
unaccounted,
|
||||
`these attrs are not value-fuzzed and not in ATTR_VALUE_FUZZ_ALLOWLIST:\n ${unaccounted.join(
|
||||
'\n ',
|
||||
)}`,
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it('the attribute allowlist has no stale rows (every entry is really not-fuzzed)', () => {
|
||||
const notFuzzed = new Set(
|
||||
allSchemaAttrKeys().filter((key) => {
|
||||
const i = key.indexOf('.');
|
||||
return !attrIsValueFuzzed(key.slice(0, i), key.slice(i + 1));
|
||||
}),
|
||||
);
|
||||
for (const key of ATTR_VALUE_FUZZ_ALLOWLIST) {
|
||||
expect(
|
||||
notFuzzed.has(key),
|
||||
`stale allowlist row (attr is now value-fuzzed, remove it): ${key}`,
|
||||
).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('every MARK attribute is value-fuzzed OR allowlisted (no invisible hole)', () => {
|
||||
// The node guard above covers node attrs; marks are fuzzed by the text
|
||||
// generator, so their coverage is tracked separately. A new mark attr (or a
|
||||
// newly-attributed mark) that lands in neither set turns this red.
|
||||
const unaccounted: string[] = [];
|
||||
for (const key of allSchemaMarkAttrKeys()) {
|
||||
if (!MARK_ATTR_FUZZED.has(key) && !MARK_ATTR_ALLOWLIST.has(key)) {
|
||||
unaccounted.push(key);
|
||||
}
|
||||
}
|
||||
expect(
|
||||
unaccounted,
|
||||
`these mark attrs are neither in MARK_ATTR_FUZZED nor MARK_ATTR_ALLOWLIST:\n ${unaccounted.join(
|
||||
'\n ',
|
||||
)}`,
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it('the MARK fuzz/allowlist sets have no stale rows (every entry is a real schema mark attr)', () => {
|
||||
const all = new Set(allSchemaMarkAttrKeys());
|
||||
for (const key of [...MARK_ATTR_FUZZED, ...MARK_ATTR_ALLOWLIST]) {
|
||||
expect(all.has(key), `stale mark-attr registry row: ${key}`).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('#351 flat generative round-trip — properties', () => {
|
||||
it('generator validity: every generated doc passes schema.check()', () => {
|
||||
// A generator that emits an invalid ProseMirror document is a GENERATOR bug.
|
||||
fc.assert(
|
||||
fc.property(fuzzUnion, (doc) => {
|
||||
schema.nodeFromJSON(doc).check(); // throws on an invalid doc
|
||||
return true;
|
||||
}),
|
||||
{ numRuns: NUM_RUNS, seed: SEED },
|
||||
);
|
||||
});
|
||||
|
||||
it('P1 — semantic round-trip: docsCanonicallyEqual(mdToPm(pmToMd(d)), d)', async () => {
|
||||
await fc.assert(
|
||||
fc.asyncProperty(p1Union, async (doc) => {
|
||||
const { doc2 } = await roundTrip(doc);
|
||||
if (!docsCanonicallyEqual(doc2, doc)) {
|
||||
// Surface the precise divergence in the failure message.
|
||||
const div = firstDivergence(
|
||||
JSON.parse(JSON.stringify(canonicalizeContent(doc2))),
|
||||
JSON.parse(JSON.stringify(canonicalizeContent(doc))),
|
||||
);
|
||||
throw new Error(
|
||||
`P1 divergence @ ${div?.path}: got=${JSON.stringify(div?.a)} want=${JSON.stringify(div?.b)}`,
|
||||
);
|
||||
}
|
||||
}),
|
||||
{ numRuns: NUM_RUNS, seed: SEED },
|
||||
);
|
||||
});
|
||||
|
||||
it('P2 — byte fixpoint: pmToMd(mdToPm(pmToMd(d))) === pmToMd(d)', async () => {
|
||||
await fc.assert(
|
||||
fc.asyncProperty(fuzzUnion, async (doc) => {
|
||||
const { md1, md2 } = await roundTrip(doc);
|
||||
expect(md2).toBe(md1);
|
||||
}),
|
||||
{ numRuns: NUM_RUNS, seed: SEED },
|
||||
);
|
||||
});
|
||||
|
||||
it('P3 — totality: neither converter throws', async () => {
|
||||
await fc.assert(
|
||||
fc.asyncProperty(fuzzUnion, async (doc) => {
|
||||
// Throwing here fails the property; fast-check shrinks to a minimal doc.
|
||||
await roundTrip(doc);
|
||||
}),
|
||||
{ numRuns: NUM_RUNS, seed: SEED },
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,310 @@
|
||||
/**
|
||||
* Flat single-node document generators (#351, PR 1).
|
||||
*
|
||||
* For every schema node type that can stand alone, a fast-check arbitrary
|
||||
* producing `{ type:'doc', content:[ <the target node> ] }` with generated attrs
|
||||
* (via nodeAttrsArb) and the minimal REQUIRED immediate children the schema
|
||||
* demands (a heading's inline text, a listItem's one paragraph, a table's
|
||||
* minimal rows, details' summary+content, a callout's one paragraph). Kept
|
||||
* FLAT: a single target node, no deep nesting — nested structural generation is
|
||||
* PR 2.
|
||||
*
|
||||
* The `mode` threads through to the attribute arbitraries:
|
||||
* - 'p1' : the round-trip-safe attribute space (P1 semantic round-trip).
|
||||
* - 'fuzz' : adds degenerate attribute states (P2 byte-fixpoint tolerates the
|
||||
* one-time normalization; P3 only needs totality).
|
||||
*
|
||||
* A COMPLETENESS CONTRACT (see flat-roundtrip.property.test.ts) enumerates the
|
||||
* whole schema and asserts every node/mark is EITHER produced by a generator
|
||||
* here OR listed in KNOWN_UNCOVERED with a reason — so a new schema type with no
|
||||
* generator turns the suite RED.
|
||||
*/
|
||||
import fc from 'fast-check';
|
||||
import { type AttrMode, nodeAttrsArb } from './attr-arbitraries.js';
|
||||
import {
|
||||
inlineContentArb,
|
||||
headingInlineContentArb,
|
||||
plainInlineContentArb,
|
||||
phraseArb,
|
||||
markedTextRunArb,
|
||||
} from './text-arbitraries.js';
|
||||
|
||||
const doc = (node: any) => ({ type: 'doc', content: [node] });
|
||||
const para = (content: any[]) => ({ type: 'paragraph', content });
|
||||
|
||||
/** A named flat-document generator. */
|
||||
export interface NamedGen {
|
||||
name: string;
|
||||
arb: fc.Arbitrary<any>;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Per-target generators, each a function of mode.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const gen = {
|
||||
paragraph: (m: AttrMode) =>
|
||||
fc.tuple(nodeAttrsArb('paragraph', m), inlineContentArb).map(([attrs, content]) =>
|
||||
doc({ type: 'paragraph', attrs, content }),
|
||||
),
|
||||
|
||||
heading: (m: AttrMode) =>
|
||||
fc.tuple(nodeAttrsArb('heading', m), headingInlineContentArb).map(([attrs, content]) =>
|
||||
doc({ type: 'heading', attrs, content }),
|
||||
),
|
||||
|
||||
blockquote: (_m: AttrMode) =>
|
||||
inlineContentArb.map((content) => doc({ type: 'blockquote', content: [para(content)] })),
|
||||
|
||||
bulletList: (_m: AttrMode) =>
|
||||
fc
|
||||
.array(inlineContentArb, { minLength: 1, maxLength: 3 })
|
||||
.map((items) =>
|
||||
doc({
|
||||
type: 'bulletList',
|
||||
content: items.map((c) => ({ type: 'listItem', content: [para(c)] })),
|
||||
}),
|
||||
),
|
||||
|
||||
orderedList: (m: AttrMode) =>
|
||||
fc
|
||||
.tuple(nodeAttrsArb('orderedList', m), fc.array(inlineContentArb, { minLength: 1, maxLength: 3 }))
|
||||
.map(([attrs, items]) =>
|
||||
doc({
|
||||
type: 'orderedList',
|
||||
attrs,
|
||||
content: items.map((c) => ({ type: 'listItem', content: [para(c)] })),
|
||||
}),
|
||||
),
|
||||
|
||||
taskList: (m: AttrMode) =>
|
||||
fc
|
||||
.array(fc.tuple(nodeAttrsArb('taskItem', m), inlineContentArb), { minLength: 1, maxLength: 3 })
|
||||
.map((items) =>
|
||||
doc({
|
||||
type: 'taskList',
|
||||
content: items.map(([attrs, c]) => ({ type: 'taskItem', attrs, content: [para(c)] })),
|
||||
}),
|
||||
),
|
||||
|
||||
codeBlock: (m: AttrMode) =>
|
||||
fc
|
||||
.tuple(
|
||||
nodeAttrsArb('codeBlock', m),
|
||||
// A fenced code block always re-imports with a TRAILING NEWLINE in its
|
||||
// text (empirically confirmed). Author the newline so the doc is already
|
||||
// at the round-trip fixpoint (supported-space shaping, not a masked bug).
|
||||
fc.array(phraseArb, { minLength: 1, maxLength: 3 }).map((lines) => lines.join('\n') + '\n'),
|
||||
)
|
||||
.map(([attrs, code]) =>
|
||||
doc({ type: 'codeBlock', attrs, content: [{ type: 'text', text: code }] }),
|
||||
),
|
||||
|
||||
horizontalRule: (_m: AttrMode) => fc.constant(doc({ type: 'horizontalRule' })),
|
||||
|
||||
pageBreak: (_m: AttrMode) => fc.constant(doc({ type: 'pageBreak' })),
|
||||
|
||||
image: (m: AttrMode) => nodeAttrsArb('image', m).map((attrs) => doc({ type: 'image', attrs })),
|
||||
|
||||
callout: (m: AttrMode) =>
|
||||
fc.tuple(nodeAttrsArb('callout', m), inlineContentArb).map(([attrs, content]) =>
|
||||
doc({ type: 'callout', attrs, content: [para(content)] }),
|
||||
),
|
||||
|
||||
mathBlock: (m: AttrMode) =>
|
||||
nodeAttrsArb('mathBlock', m).map((attrs) => doc({ type: 'mathBlock', attrs })),
|
||||
|
||||
details: (m: AttrMode) =>
|
||||
fc
|
||||
.tuple(nodeAttrsArb('details', m), plainInlineContentArb, inlineContentArb)
|
||||
.map(([attrs, summary, body]) =>
|
||||
doc({
|
||||
type: 'details',
|
||||
attrs,
|
||||
content: [
|
||||
{ type: 'detailsSummary', content: summary },
|
||||
{ type: 'detailsContent', content: [para(body)] },
|
||||
],
|
||||
}),
|
||||
),
|
||||
|
||||
table: (_m: AttrMode) =>
|
||||
fc.integer({ min: 1, max: 3 }).chain((cols) => {
|
||||
// GFM alignment is column-wide (encoded in the header separator), so a
|
||||
// column's alignment must be identical on the header and every body cell,
|
||||
// else the second export re-aligns and churns. Pick ONE align per column.
|
||||
const alignsArb = fc.array(fc.constantFrom(undefined, 'left', 'center', 'right'), {
|
||||
minLength: cols,
|
||||
maxLength: cols,
|
||||
});
|
||||
const cell = (header: boolean, align?: string) =>
|
||||
phraseArb.map((t) => ({
|
||||
type: header ? 'tableHeader' : 'tableCell',
|
||||
// colspan/rowspan pinned to 1 (GFM cannot express spans); optional
|
||||
// column-consistent align.
|
||||
attrs: { colspan: 1, rowspan: 1, ...(align ? { align } : {}) },
|
||||
content: [para([{ type: 'text', text: t }])],
|
||||
}));
|
||||
return alignsArb.chain((aligns) => {
|
||||
const headerRow = fc
|
||||
.tuple(...aligns.map((a) => cell(true, a)))
|
||||
.map((cells) => ({ type: 'tableRow', content: cells }));
|
||||
const bodyRow = fc
|
||||
.tuple(...aligns.map((a) => cell(false, a)))
|
||||
.map((cells) => ({ type: 'tableRow', content: cells }));
|
||||
return fc
|
||||
.tuple(headerRow, fc.array(bodyRow, { minLength: 1, maxLength: 2 }))
|
||||
.map(([h, body]) => doc({ type: 'table', content: [h, ...body] }));
|
||||
});
|
||||
}),
|
||||
|
||||
columns: (m: AttrMode) =>
|
||||
// Couple the column count to the layout so the two stay consistent
|
||||
// (two_equal/left_sidebar/right_sidebar -> 2, three_equal -> 3).
|
||||
fc
|
||||
.constantFrom('two_equal', 'three_equal', 'left_sidebar', 'right_sidebar')
|
||||
.chain((layout) => {
|
||||
const count = layout === 'three_equal' ? 3 : 2;
|
||||
return fc
|
||||
.tuple(
|
||||
nodeAttrsArb('columns', m, { layout, widthMode: 'normal' }),
|
||||
fc.array(inlineContentArb, { minLength: count, maxLength: count }),
|
||||
)
|
||||
.map(([attrs, bodies]) =>
|
||||
doc({
|
||||
type: 'columns',
|
||||
attrs,
|
||||
content: bodies.map((c) => ({ type: 'column', content: [para(c)] })),
|
||||
}),
|
||||
);
|
||||
}),
|
||||
|
||||
subpages: (m: AttrMode) =>
|
||||
nodeAttrsArb('subpages', m).map((attrs) => doc({ type: 'subpages', attrs })),
|
||||
|
||||
audio: (m: AttrMode) => nodeAttrsArb('audio', m).map((attrs) => doc({ type: 'audio', attrs })),
|
||||
video: (m: AttrMode) => nodeAttrsArb('video', m).map((attrs) => doc({ type: 'video', attrs })),
|
||||
pdf: (m: AttrMode) => nodeAttrsArb('pdf', m).map((attrs) => doc({ type: 'pdf', attrs })),
|
||||
youtube: (m: AttrMode) => nodeAttrsArb('youtube', m).map((attrs) => doc({ type: 'youtube', attrs })),
|
||||
embed: (m: AttrMode) => nodeAttrsArb('embed', m).map((attrs) => doc({ type: 'embed', attrs })),
|
||||
drawio: (m: AttrMode) => nodeAttrsArb('drawio', m).map((attrs) => doc({ type: 'drawio', attrs })),
|
||||
excalidraw: (m: AttrMode) =>
|
||||
nodeAttrsArb('excalidraw', m).map((attrs) => doc({ type: 'excalidraw', attrs })),
|
||||
attachment: (m: AttrMode) =>
|
||||
nodeAttrsArb('attachment', m).map((attrs) => doc({ type: 'attachment', attrs })),
|
||||
htmlEmbed: (m: AttrMode) =>
|
||||
nodeAttrsArb('htmlEmbed', m).map((attrs) => doc({ type: 'htmlEmbed', attrs })),
|
||||
pageEmbed: (m: AttrMode) =>
|
||||
nodeAttrsArb('pageEmbed', m).map((attrs) => doc({ type: 'pageEmbed', attrs })),
|
||||
transclusionReference: (m: AttrMode) =>
|
||||
nodeAttrsArb('transclusionReference', m).map((attrs) =>
|
||||
doc({ type: 'transclusionReference', attrs }),
|
||||
),
|
||||
|
||||
transclusionSource: (m: AttrMode) =>
|
||||
fc.tuple(nodeAttrsArb('transclusionSource', m), inlineContentArb).map(([attrs, content]) =>
|
||||
doc({ type: 'transclusionSource', attrs, content: [para(content)] }),
|
||||
),
|
||||
|
||||
// A footnote reference PLUS its definition (the reference has no standalone
|
||||
// markdown form without its definition — see KNOWN_UNCOVERED note for the
|
||||
// bare reference). Both carry the same id. The definition body uses
|
||||
// headingInlineContentArb (NO hard breaks): a footnote is serialized inline as
|
||||
// `^[...]`, so a hard break inside it collapses to a single space on re-parse
|
||||
// (empirically confirmed) — that is the container's markdown limitation, not
|
||||
// an attribute-level concern. The reference-bearing paragraph is a NORMAL
|
||||
// paragraph and keeps the full inline corpus.
|
||||
footnotes: (m: AttrMode) =>
|
||||
fc.tuple(fc.constantFrom('fn1', 'fn2', 'note'), inlineContentArb, headingInlineContentArb).map(
|
||||
([id, refText, noteBody]) => ({
|
||||
type: 'doc',
|
||||
content: [
|
||||
para([...refText, { type: 'footnoteReference', attrs: { id } }]),
|
||||
{
|
||||
type: 'footnotesList',
|
||||
content: [{ type: 'footnoteDefinition', attrs: { id }, content: [para(noteBody)] }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
),
|
||||
|
||||
// ── inline targets wrapped in a paragraph ────────────────────────────────
|
||||
mention: (m: AttrMode) =>
|
||||
nodeAttrsArb('mention', m).map((attrs) => doc(para([{ type: 'mention', attrs }]))),
|
||||
|
||||
mathInline: (m: AttrMode) =>
|
||||
fc.tuple(phraseArb, nodeAttrsArb('mathInline', m)).map(([t, attrs]) =>
|
||||
doc(para([{ type: 'text', text: t }, { type: 'mathInline', attrs }])),
|
||||
),
|
||||
|
||||
status: (m: AttrMode) =>
|
||||
nodeAttrsArb('status', m).map((attrs) => doc(para([{ type: 'status', attrs }]))),
|
||||
|
||||
hardBreak: (_m: AttrMode) =>
|
||||
fc.tuple(phraseArb, phraseArb).map(([a, b]) =>
|
||||
doc(para([{ type: 'text', text: a }, { type: 'hardBreak' }, { type: 'text', text: b }])),
|
||||
),
|
||||
|
||||
// ── marks: a paragraph of marked runs (covers every mark type) ───────────
|
||||
marksOnText: (_m: AttrMode) =>
|
||||
fc.array(markedTextRunArb, { minLength: 1, maxLength: 5 }).map((runs) => {
|
||||
// Merge adjacent same-mark runs (see text-arbitraries.normalizeInline).
|
||||
const out: any[] = [];
|
||||
for (const r of runs) {
|
||||
const prev = out[out.length - 1];
|
||||
if (prev && JSON.stringify(prev.marks ?? []) === JSON.stringify(r.marks ?? [])) {
|
||||
prev.text += r.text;
|
||||
} else out.push({ ...r });
|
||||
}
|
||||
return doc(para(out));
|
||||
}),
|
||||
};
|
||||
|
||||
/** Build the full list of named generators for a given mode. */
|
||||
export function buildGenerators(mode: AttrMode): NamedGen[] {
|
||||
return Object.entries(gen).map(([name, f]) => ({ name, arb: f(mode) }));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Completeness contract support.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Schema node/mark types deliberately NOT covered by a P1/P2 generator, each
|
||||
* with a one-line reason. Excluding a type means it is kept OUT of the round-
|
||||
* trip generators — it does NOT weaken any property.
|
||||
*
|
||||
* NOTE (empirical): the candidates the issue flagged for review — pageEmbed,
|
||||
* subpages, transclusionSource/Reference, mention, status — were PROBED against
|
||||
* the live converter and DO round-trip P1/P2 with placeholder ids, so they are
|
||||
* COVERED by real generators rather than allowlisted here. The allowlist below
|
||||
* holds only types with no standalone flat generator by construction.
|
||||
*/
|
||||
export const KNOWN_UNCOVERED: Record<string, string> = {
|
||||
// The root node; it is the wrapper every generated doc already is, never a
|
||||
// "target" content node, so it has no standalone generator of its own.
|
||||
doc: 'the document root wrapper, not a content node with a standalone generator',
|
||||
};
|
||||
|
||||
/** Recursively collect every node type and `mark:<type>` under a tree. */
|
||||
export function collectTypes(node: any, seen = new Set<string>()): Set<string> {
|
||||
if (!node || typeof node !== 'object') return seen;
|
||||
if (node.type) seen.add(node.type);
|
||||
for (const m of node.marks ?? []) if (m?.type) seen.add(`mark:${m.type}`);
|
||||
for (const c of node.content ?? []) collectTypes(c, seen);
|
||||
return seen;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sample every generator and return the union of node/mark types they produce.
|
||||
* Deterministic (fixed seed) so the completeness contract is stable.
|
||||
*/
|
||||
export function coveredTypes(seed = 12345, perGen = 60): Set<string> {
|
||||
const seen = new Set<string>();
|
||||
for (const { arb } of buildGenerators('p1')) {
|
||||
for (const sample of fc.sample(arb, { numRuns: perGen, seed })) {
|
||||
collectTypes(sample, seen);
|
||||
}
|
||||
}
|
||||
return seen;
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
/**
|
||||
* Hostile inline-text corpus for the generative flat-document round-trip suite
|
||||
* (#351, PR 1).
|
||||
*
|
||||
* These arbitraries are a DIRECT PORT of the "supported space" guardrails that
|
||||
* `test/markdown-roundtrip.property.test.ts` proved empirically against the live
|
||||
* converter. That file's long header documents WHY each guardrail exists; rather
|
||||
* than re-derive them, we reuse the exact same shapes here so the attribute-level
|
||||
* generative suite inherits the same byte-stable text space. Each guardrail is
|
||||
* cited back to that file below.
|
||||
*
|
||||
* The corpus deliberately spans the CommonMark / canon hostile alphabet
|
||||
* (`* _ [ ] ( ) { } | < > & # ! ~ = + -`), unicode / emoji / RTL, and the legal
|
||||
* mark combinations on runs (including the `code` mark, which the schema's
|
||||
* `excludes: "_"` makes suppress every co-occurring mark — so it is never
|
||||
* combined with another mark in the byte-stable space).
|
||||
*/
|
||||
import fc from 'fast-check';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Words and the hostile special-character alphabet.
|
||||
// (Ported from markdown-roundtrip.property.test.ts, "Inline text arbitraries".)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Alphanumeric "word" (no markdown-significant characters). Length 1..6. */
|
||||
export const wordArb = fc
|
||||
.stringMatching(/^[A-Za-z0-9]{1,6}$/)
|
||||
.filter((w) => w.length > 0);
|
||||
|
||||
/**
|
||||
* A SINGLE markdown-significant character, emitted only as an isolated,
|
||||
* space-flanked token. Every char the task calls out plus a few more; each was
|
||||
* verified byte-stable in this position by the sibling property test.
|
||||
*
|
||||
* NOTE: the backtick (`) is DELIBERATELY excluded from free-floating plain text
|
||||
* (it is a code-span delimiter that re-pairs globally). It is exercised only via
|
||||
* the `code` mark and code blocks — see markdown-roundtrip.property.test.ts.
|
||||
*/
|
||||
export const specialCharArb = fc.constantFrom(
|
||||
'*', '_', '[', ']', '(', ')', '{', '}', '|', '<', '>', '&', '#', '!', '~', '=', '+', '-',
|
||||
);
|
||||
|
||||
// A pinch of unicode / emoji / RTL, always word-like (no markdown specials) so
|
||||
// it stays inside the space-flanked corpus. Kept letter/emoji-bearing so it is
|
||||
// never coerced to a number (see letterPhraseArb rationale).
|
||||
export const unicodeWordArb = fc.constantFrom(
|
||||
'café', 'naïve', 'Zürich', 'Москва', 'こんにちは', '你好', '😀', '🚀x', 'مرحبا', 'שלום',
|
||||
);
|
||||
|
||||
/**
|
||||
* A "safe special" text string: a space-joined sequence of tokens that always
|
||||
* BEGINS and ENDS with an alphanumeric word, with any isolated special chars (or
|
||||
* unicode words) confined to the MIDDLE, each space-flanked by words.
|
||||
*
|
||||
* Both boundary guarantees matter (verbatim from the sibling test):
|
||||
* * Leading word: the line never opens with a block/inline trigger
|
||||
* (">", "*", "-", "#", "1." ...).
|
||||
* * Trailing word: adjacent text runs CONCATENATE with no separator, so a run
|
||||
* ending in a bare "<" beside a run starting with a letter would form a fake
|
||||
* HTML tag. Ending every run with a word keeps every special internal and
|
||||
* space-flanked even after concatenation.
|
||||
*/
|
||||
export const safeTextArb: fc.Arbitrary<string> = fc
|
||||
.tuple(
|
||||
wordArb,
|
||||
fc.array(fc.oneof(wordArb, specialCharArb, unicodeWordArb), {
|
||||
minLength: 0,
|
||||
maxLength: 3,
|
||||
}),
|
||||
wordArb,
|
||||
)
|
||||
.map(([first, middle, last]) => [first, ...middle, last].join(' '));
|
||||
|
||||
/**
|
||||
* A plain alphanumeric phrase (1..3 words) for places where even isolated
|
||||
* specials are not wanted (e.g. code-block language, mention labels, status
|
||||
* text, table cells rendered on the plain-markdown path).
|
||||
*/
|
||||
export const phraseArb: fc.Arbitrary<string> = fc
|
||||
.array(wordArb, { minLength: 1, maxLength: 3 })
|
||||
.map((ws) => ws.join(' '));
|
||||
|
||||
/**
|
||||
* A phrase guaranteed to contain at least one letter. Used for image/media alt
|
||||
* text and link titles: a PURELY numeric alt/title (e.g. "0") is parsed back as
|
||||
* a NUMBER and then dropped by the converter's `value || ""` coercion — not
|
||||
* byte-stable. A letter anywhere keeps it a string. (Ported verbatim.)
|
||||
*/
|
||||
export const letterPhraseArb: fc.Arbitrary<string> = fc
|
||||
.tuple(
|
||||
fc.stringMatching(/^[A-Za-z]{1,4}$/),
|
||||
fc.array(wordArb, { minLength: 0, maxLength: 2 }),
|
||||
)
|
||||
.map(([head, rest]) => [head, ...rest].join(' '));
|
||||
|
||||
/** A paren/space-free URL — safe inside markdown link/image `(...)` syntax. */
|
||||
export const urlArb: fc.Arbitrary<string> = fc
|
||||
.webUrl()
|
||||
.filter((u) => !/[()\s]/.test(u));
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Marked inline runs.
|
||||
// (Ported from markdown-roundtrip.property.test.ts "markedTextRunArb".)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* A text run with an OPTIONAL single non-code formatting mark (bold/italic/
|
||||
* strike/underline/superscript/subscript/spoiler), or a SOLE `code` mark, or a
|
||||
* link, or an inline comment anchor. `code` is NEVER combined with another mark
|
||||
* in the byte-stable space (that combination is a documented converter
|
||||
* limitation — the schema's `code` mark declares `excludes: "_"`). Marks wrap
|
||||
* `safeTextArb`, which stays stable even when it contains isolated specials.
|
||||
*
|
||||
* The mark set here is broadened past the sibling test's {bold,italic,strike}
|
||||
* to also cover underline / superscript / subscript / spoiler / textStyle /
|
||||
* highlight (all single, non-code marks), so the marks-on-text generator
|
||||
* exercises every mark the schema declares except the deliberately-excluded
|
||||
* `code`+other combination.
|
||||
*/
|
||||
export const markedTextRunArb: fc.Arbitrary<any> = fc.oneof(
|
||||
// Plain text.
|
||||
safeTextArb.map((t) => ({ type: 'text', text: t })),
|
||||
// Single formatting mark (attribute-free marks).
|
||||
fc
|
||||
.tuple(
|
||||
safeTextArb,
|
||||
fc.constantFrom('bold', 'italic', 'strike', 'underline', 'superscript', 'subscript', 'spoiler'),
|
||||
)
|
||||
.map(([t, m]) => ({ type: 'text', text: t, marks: [{ type: m }] })),
|
||||
// highlight with a color attr.
|
||||
fc
|
||||
.tuple(safeTextArb, fc.constantFrom('#ffcc00', '#a0e0ff', 'yellow'))
|
||||
.map(([t, color]) => ({ type: 'text', text: t, marks: [{ type: 'highlight', attrs: { color } }] })),
|
||||
// textStyle with a color attr.
|
||||
fc
|
||||
.tuple(safeTextArb, fc.constantFrom('#123456', '#ff0000', '#00aa88'))
|
||||
.map(([t, color]) => ({ type: 'text', text: t, marks: [{ type: 'textStyle', attrs: { color } }] })),
|
||||
// Sole code mark (backtick span). safeTextArb is backtick-free, so the span
|
||||
// content cannot contain an inner backtick.
|
||||
safeTextArb.map((t) => ({ type: 'text', text: t, marks: [{ type: 'code' }] })),
|
||||
// Link with safe text, a paren/space-free href, optionally a letter-bearing
|
||||
// title (a purely numeric title is coerced to a number and dropped).
|
||||
fc
|
||||
.tuple(phraseArb, urlArb, fc.option(letterPhraseArb, { nil: undefined }))
|
||||
.map(([t, href, title]) => ({
|
||||
type: 'text',
|
||||
text: t,
|
||||
marks: [{ type: 'link', attrs: title ? { href, title } : { href } }],
|
||||
})),
|
||||
// Inline comment anchor: a span[data-comment-id] that must survive byte-for-
|
||||
// byte. commentId is an alphanumeric token; `resolved` rides only when true.
|
||||
fc
|
||||
.tuple(safeTextArb, fc.stringMatching(/^[A-Za-z0-9]{4,10}$/), fc.boolean())
|
||||
.map(([t, commentId, resolved]) => ({
|
||||
type: 'text',
|
||||
text: t,
|
||||
marks: [
|
||||
{ type: 'comment', attrs: resolved ? { commentId, resolved: true } : { commentId } },
|
||||
],
|
||||
})),
|
||||
);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Inline atoms and inline-content assembly.
|
||||
// (Ported from markdown-roundtrip.property.test.ts.)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Inline math node carrying LaTeX that includes the `a < b` the task asks for. */
|
||||
export const mathInlineArb: fc.Arbitrary<any> = fc
|
||||
.constantFrom('a < b', 'x^2 + y^2', 'a < b < c', '\\frac{1}{2}', 'E = mc^2')
|
||||
.map((text) => ({ type: 'mathInline', attrs: { text } }));
|
||||
|
||||
/** Mention node; label/id/entity are plain phrases / uuids. */
|
||||
export const mentionArb: fc.Arbitrary<any> = fc
|
||||
.tuple(phraseArb, fc.uuid(), fc.uuid())
|
||||
.map(([label, id, entityId]) => ({
|
||||
type: 'mention',
|
||||
attrs: { id, label, entityType: 'user', entityId },
|
||||
}));
|
||||
|
||||
export const hardBreakArb: fc.Arbitrary<any> = fc.constant({ type: 'hardBreak' });
|
||||
|
||||
const sameMarks = (a: any[] | undefined, b: any[] | undefined): boolean =>
|
||||
JSON.stringify(a ?? []) === JSON.stringify(b ?? []);
|
||||
|
||||
/**
|
||||
* Canonicalize a generated inline-content array the way ProseMirror stores it,
|
||||
* then trim the markdown-fragile edges. (Ported verbatim from
|
||||
* markdown-roundtrip.property.test.ts "normalizeInline":)
|
||||
* 1) MERGE adjacent text runs with IDENTICAL marks (the editor coalesces
|
||||
* them; split same-mark runs export to ambiguous "**a****b**").
|
||||
* 2) Collapse CONSECUTIVE hard breaks (two render a blank line marked eats).
|
||||
* 3) Drop a TRAILING hard break (removed by the converter's .trim()).
|
||||
*/
|
||||
export function normalizeInline(nodes: any[]): any[] {
|
||||
const out: any[] = [];
|
||||
for (const node of nodes) {
|
||||
const prev = out[out.length - 1];
|
||||
if (node.type === 'hardBreak' && prev && prev.type === 'hardBreak') continue;
|
||||
if (
|
||||
node.type === 'text' &&
|
||||
prev &&
|
||||
prev.type === 'text' &&
|
||||
sameMarks(prev.marks, node.marks)
|
||||
) {
|
||||
prev.text += node.text;
|
||||
continue;
|
||||
}
|
||||
out.push(node.type === 'text' ? { ...node } : node);
|
||||
}
|
||||
while (out.length > 1 && out[out.length - 1].type === 'hardBreak') out.pop();
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Inline content for a paragraph: at least one marked text run, optionally with
|
||||
* inline atoms (math/mention) and hard breaks interspersed. Always starts with a
|
||||
* text run so the paragraph never opens with a block trigger. (Ported.)
|
||||
*/
|
||||
export const inlineContentArb: fc.Arbitrary<any[]> = fc
|
||||
.tuple(
|
||||
markedTextRunArb,
|
||||
fc.array(
|
||||
fc.oneof(
|
||||
{ weight: 5, arbitrary: markedTextRunArb },
|
||||
{ weight: 1, arbitrary: mathInlineArb },
|
||||
{ weight: 1, arbitrary: mentionArb },
|
||||
{ weight: 1, arbitrary: hardBreakArb },
|
||||
),
|
||||
{ minLength: 0, maxLength: 4 },
|
||||
),
|
||||
)
|
||||
.map(([first, rest]) => normalizeInline([first, ...rest]));
|
||||
|
||||
/**
|
||||
* Inline content for a HEADING — identical to a paragraph's, but WITHOUT hard
|
||||
* breaks. A hard break inside an ATX heading is not byte-stable (marked splits
|
||||
* the heading). (Ported.)
|
||||
*/
|
||||
export const headingInlineContentArb: fc.Arbitrary<any[]> = fc
|
||||
.tuple(
|
||||
markedTextRunArb,
|
||||
fc.array(
|
||||
fc.oneof(
|
||||
{ weight: 5, arbitrary: markedTextRunArb },
|
||||
{ weight: 1, arbitrary: mathInlineArb },
|
||||
{ weight: 1, arbitrary: mentionArb },
|
||||
),
|
||||
{ minLength: 0, maxLength: 4 },
|
||||
),
|
||||
)
|
||||
.map(([first, rest]) => normalizeInline([first, ...rest]));
|
||||
|
||||
/** Simple plain-text inline content (single run) for containers rendered on the
|
||||
* raw-HTML path (table cells / column bodies) where fancy inline is undesirable. */
|
||||
export const plainInlineContentArb: fc.Arbitrary<any[]> = phraseArb.map((t) => [
|
||||
{ type: 'text', text: t },
|
||||
]);
|
||||
@@ -0,0 +1,124 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
// Import DIRECTLY from src (NOT the docmost-client barrel, which pulls in
|
||||
// collaboration.ts and mutates global DOM at import time).
|
||||
import { convertProseMirrorToMarkdown } from "../src/lib/markdown-converter.js";
|
||||
import { markdownToProseMirror } from "../src/lib/markdown-to-prosemirror.js";
|
||||
|
||||
/**
|
||||
* gitmost #377 (round-1 review, finding #1) — proof, against the REAL
|
||||
* converter, that the transcript-insert boundary defense survives git-sync.
|
||||
*
|
||||
* The web bridge (apps/client .../gitmost/gitmost-recording.ts,
|
||||
* `gitmostInsertTranscriptIntoEditor`) appends each transcript line as a
|
||||
* PARAGRAPH text node. The paragraph serializer here (`case "paragraph"`) emits
|
||||
* that text VERBATIM with no block-escape, so a line whose text begins with a
|
||||
* col-0 markdown block trigger would, on the doc -> markdown -> doc git-sync
|
||||
* cycle, silently re-parse into a heading / list / quote / callout / code block.
|
||||
* That missing block-escape is the pre-existing root cause; the bridge's
|
||||
* boundary defense prepends an invisible zero-width space (U+200B) to a line
|
||||
* that begins with such a trigger, shifting it off column 0.
|
||||
*
|
||||
* This test keeps a COPY of the bridge's trigger regex (the bridge is in a
|
||||
* different package and can't be imported here) and asserts:
|
||||
* 1. bare trigger lines DO corrupt (documents the root cause), and
|
||||
* 2. the ZWSP-neutralized form round-trips as a single PARAGRAPH with the
|
||||
* text byte-preserved.
|
||||
*/
|
||||
|
||||
const ZWSP = ""; // U+200B
|
||||
|
||||
// MUST stay in sync with GITMOST_MD_BLOCK_TRIGGER_RE in the client bridge.
|
||||
const MD_BLOCK_TRIGGER_RE =
|
||||
/^(?:#{1,6}(?:\s|$)|[-*+](?:\s|$)|>|\d+[.)](?:\s|$)|```|~~~|\||([-*_])(?:\s*\1){2,}\s*$)/;
|
||||
|
||||
const doc = (...nodes: any[]) => ({ type: "doc", content: nodes });
|
||||
const para = (t: string) => ({
|
||||
type: "paragraph",
|
||||
content: [{ type: "text", text: t }],
|
||||
});
|
||||
|
||||
const roundtrip = async (text: string) => {
|
||||
const md = convertProseMirrorToMarkdown(doc(para(text)));
|
||||
const back = await markdownToProseMirror(md);
|
||||
return back.content as any[];
|
||||
};
|
||||
|
||||
describe("gitmost transcript neutralization (git-sync round-trip)", () => {
|
||||
// Lines that, at column 0, the serializer's missing block-escape would let
|
||||
// git-sync re-parse into a non-paragraph block.
|
||||
const triggerLines = [
|
||||
"- dash",
|
||||
"* star",
|
||||
"+ plus",
|
||||
"> quote",
|
||||
"# hash",
|
||||
"1. one",
|
||||
"1) one",
|
||||
"> [!info] note",
|
||||
"```js",
|
||||
"~~~",
|
||||
// Solid + spaced thematic breaks — these re-parse into a `horizontalRule`,
|
||||
// which carries NO text, so a bare separator line LOSES its text entirely
|
||||
// (round-2 finding). `_` also only forms a block via this construct.
|
||||
"---",
|
||||
"***",
|
||||
"___",
|
||||
"- - -", // spaced dash break (solid form is caught by [-*+]\s too, but this is the break)
|
||||
"_ _ _",
|
||||
];
|
||||
|
||||
it("BARE trigger lines corrupt into non-paragraph blocks (root cause)", async () => {
|
||||
for (const line of triggerLines) {
|
||||
const blocks = await roundtrip(line);
|
||||
// At least one produced block is NOT a paragraph — i.e. corruption.
|
||||
const allParagraphs = blocks.every((b) => b.type === "paragraph");
|
||||
expect(
|
||||
allParagraphs,
|
||||
`expected "${line}" to corrupt when inserted bare`,
|
||||
).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("BARE solid thematic breaks corrupt into a text-LOSING horizontalRule", async () => {
|
||||
// The severe case: no text node survives. Documents why neutralization
|
||||
// matters more here than for list/quote (where the text survived).
|
||||
for (const line of ["---", "***", "___"]) {
|
||||
const blocks = await roundtrip(line);
|
||||
expect(blocks.map((b) => b.type)).toContain("horizontalRule");
|
||||
// No block carries the original text anywhere.
|
||||
const flat = JSON.stringify(blocks);
|
||||
expect(flat).not.toContain(line);
|
||||
}
|
||||
});
|
||||
|
||||
it("ZWSP-neutralized trigger lines round-trip as a single paragraph, text preserved", async () => {
|
||||
for (const line of triggerLines) {
|
||||
// The regex must actually classify each as a trigger.
|
||||
expect(MD_BLOCK_TRIGGER_RE.test(line), `regex missed "${line}"`).toBe(
|
||||
true,
|
||||
);
|
||||
const neutralized = ZWSP + line;
|
||||
const blocks = await roundtrip(neutralized);
|
||||
|
||||
expect(blocks).toHaveLength(1);
|
||||
expect(blocks[0].type).toBe("paragraph");
|
||||
// Text is byte-preserved (ZWSP + original line), so the display is the
|
||||
// original line with only an invisible leading character.
|
||||
expect(blocks[0].content[0].text).toBe(neutralized);
|
||||
}
|
||||
});
|
||||
|
||||
it("normal host-prefixed lines never match the trigger regex and round-trip byte-exact", async () => {
|
||||
for (const line of [
|
||||
"You: hello there",
|
||||
"Speaker 1: - and then a dash mid-line",
|
||||
"Speaker 2: 1. not a list",
|
||||
]) {
|
||||
expect(MD_BLOCK_TRIGGER_RE.test(line)).toBe(false);
|
||||
const blocks = await roundtrip(line);
|
||||
expect(blocks).toHaveLength(1);
|
||||
expect(blocks[0].type).toBe("paragraph");
|
||||
expect(blocks[0].content[0].text).toBe(line);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,68 @@
|
||||
diff --git a/dist/index.js b/dist/index.js
|
||||
index ae447a12f7823ec0a00837ee9f0eb809a610d5f8..a3402b2c2d021ef432cfa76e35d370073d525135 100644
|
||||
--- a/dist/index.js
|
||||
+++ b/dist/index.js
|
||||
@@ -6578,9 +6578,19 @@ function createOutputTransformStream(output) {
|
||||
controller.enqueue({ part: chunk, partialOutput: void 0 });
|
||||
return;
|
||||
}
|
||||
- text2 += chunk.text;
|
||||
textChunk += chunk.text;
|
||||
textProviderMetadata = (_a21 = chunk.providerMetadata) != null ? _a21 : textProviderMetadata;
|
||||
+ if (output == null) {
|
||||
+ // PATCH(docmost #OOM): no output strategy requested -> publish each
|
||||
+ // text-delta immediately and do NOT build cumulative partialOutput
|
||||
+ // snapshots. Unpatched, the default text() output snapshots the ENTIRE
|
||||
+ // accumulated turn text on every delta (O(n^2) memory) and those
|
||||
+ // snapshots pile up in the never-consumed leftover tee branch of
|
||||
+ // DefaultStreamTextResult.baseStream -> heap OOM on long agent turns.
|
||||
+ publishTextChunk({ controller });
|
||||
+ return;
|
||||
+ }
|
||||
+ text2 += chunk.text;
|
||||
const result = await output.parsePartialOutput({ text: text2 });
|
||||
if (result !== void 0) {
|
||||
const currentJson = JSON.stringify(result.partial);
|
||||
@@ -6959,7 +6969,7 @@ var DefaultStreamTextResult = class {
|
||||
})
|
||||
);
|
||||
}
|
||||
- this.baseStream = stream.pipeThrough(createOutputTransformStream(output != null ? output : text())).pipeThrough(eventProcessor);
|
||||
+ this.baseStream = stream.pipeThrough(createOutputTransformStream(output)).pipeThrough(eventProcessor);
|
||||
const { maxRetries, retry } = prepareRetries({
|
||||
maxRetries: maxRetriesArg,
|
||||
abortSignal
|
||||
diff --git a/dist/index.mjs b/dist/index.mjs
|
||||
index 663875332e3f9a9bd167c25583c515876f42951b..b840b0502c9894df983e0154805abb80e70e6331 100644
|
||||
--- a/dist/index.mjs
|
||||
+++ b/dist/index.mjs
|
||||
@@ -6501,9 +6501,19 @@ function createOutputTransformStream(output) {
|
||||
controller.enqueue({ part: chunk, partialOutput: void 0 });
|
||||
return;
|
||||
}
|
||||
- text2 += chunk.text;
|
||||
textChunk += chunk.text;
|
||||
textProviderMetadata = (_a21 = chunk.providerMetadata) != null ? _a21 : textProviderMetadata;
|
||||
+ if (output == null) {
|
||||
+ // PATCH(docmost #OOM): no output strategy requested -> publish each
|
||||
+ // text-delta immediately and do NOT build cumulative partialOutput
|
||||
+ // snapshots. Unpatched, the default text() output snapshots the ENTIRE
|
||||
+ // accumulated turn text on every delta (O(n^2) memory) and those
|
||||
+ // snapshots pile up in the never-consumed leftover tee branch of
|
||||
+ // DefaultStreamTextResult.baseStream -> heap OOM on long agent turns.
|
||||
+ publishTextChunk({ controller });
|
||||
+ return;
|
||||
+ }
|
||||
+ text2 += chunk.text;
|
||||
const result = await output.parsePartialOutput({ text: text2 });
|
||||
if (result !== void 0) {
|
||||
const currentJson = JSON.stringify(result.partial);
|
||||
@@ -6882,7 +6892,7 @@ var DefaultStreamTextResult = class {
|
||||
})
|
||||
);
|
||||
}
|
||||
- this.baseStream = stream.pipeThrough(createOutputTransformStream(output != null ? output : text())).pipeThrough(eventProcessor);
|
||||
+ this.baseStream = stream.pipeThrough(createOutputTransformStream(output)).pipeThrough(eventProcessor);
|
||||
const { maxRetries, retry } = prepareRetries({
|
||||
maxRetries: maxRetriesArg,
|
||||
abortSignal
|
||||
Generated
+11
-5
@@ -44,6 +44,9 @@ overrides:
|
||||
ip-address: 10.1.1
|
||||
|
||||
patchedDependencies:
|
||||
ai@6.0.134:
|
||||
hash: f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9
|
||||
path: patches/ai@6.0.134.patch
|
||||
scimmy@1.3.5:
|
||||
hash: 775d80f86830b2c5dd1a250c9802c10f8fc3da3c7898373de5aa0c23993d1673
|
||||
path: patches/scimmy@1.3.5.patch
|
||||
@@ -543,6 +546,9 @@ importers:
|
||||
'@docmost/pdf-inspector':
|
||||
specifier: 1.9.6
|
||||
version: 1.9.6
|
||||
'@docmost/prosemirror-markdown':
|
||||
specifier: workspace:*
|
||||
version: link:../../packages/prosemirror-markdown
|
||||
'@fastify/cookie':
|
||||
specifier: ^11.0.2
|
||||
version: 11.0.2
|
||||
@@ -623,10 +629,10 @@ importers:
|
||||
version: 8.3.0(socket.io-adapter@2.5.4)
|
||||
ai:
|
||||
specifier: ^6.0.134
|
||||
version: 6.0.134(zod@4.3.6)
|
||||
version: 6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6)
|
||||
ai-sdk-ollama:
|
||||
specifier: ^3.8.1
|
||||
version: 3.8.1(ai@6.0.134(zod@4.3.6))(zod@4.3.6)
|
||||
version: 3.8.1(ai@6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6))(zod@4.3.6)
|
||||
bcrypt:
|
||||
specifier: ^6.0.0
|
||||
version: 6.0.0
|
||||
@@ -16355,17 +16361,17 @@ snapshots:
|
||||
|
||||
agent-base@7.1.4: {}
|
||||
|
||||
ai-sdk-ollama@3.8.1(ai@6.0.134(zod@4.3.6))(zod@4.3.6):
|
||||
ai-sdk-ollama@3.8.1(ai@6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6))(zod@4.3.6):
|
||||
dependencies:
|
||||
'@ai-sdk/provider': 3.0.8
|
||||
'@ai-sdk/provider-utils': 4.0.21(zod@4.3.6)
|
||||
ai: 6.0.134(zod@4.3.6)
|
||||
ai: 6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6)
|
||||
jsonrepair: 3.13.3
|
||||
ollama: 0.6.3
|
||||
transitivePeerDependencies:
|
||||
- zod
|
||||
|
||||
ai@6.0.134(zod@4.3.6):
|
||||
ai@6.0.134(patch_hash=f60bfc3357e01e1f3978c6c40fdd65aeb33fefaad7179cde8676465b6c5ff4d9)(zod@4.3.6):
|
||||
dependencies:
|
||||
'@ai-sdk/gateway': 3.0.77(zod@4.3.6)
|
||||
'@ai-sdk/provider': 3.0.8
|
||||
|
||||
Reference in New Issue
Block a user