Compare commits

...

8 Commits

Author SHA1 Message Date
agent_coder 1fc9c25681 feat(search): semantic fusion layer (Phase B PR-1) over lexical RRF (#530)
Fuse a vector-similarity branch into #529's lexical RRF. Query embed via a
global TEI sidecar (or per-workspace provider), degrading transparently to the
byte-identical Phase-A lexical path on any failure; a kill-switch and env knobs
gate it.

- ai.service: resolveEmbeddingProvider (workspace→global TEI fallback) with a
  deterministic config fingerprint; embedQuery (query prefix + short 800ms
  timeout); extract embedWithModel core shared with embedTexts.
- page-embedding.repo: vectorCandidateArm fragment (page-level NN, dim +
  active-fingerprint filtered) + fingerprint on insertChunks.
- search.service: 3-branch RRF over lexical ∪ vector candidates; try/catch wraps
  only the embed (permission filter stays outside, fail-closed); semantic
  degrade emits search.semantic.degraded; response gains semantic{state,available,reason}.
- indexer: resolve provider, prepend doc prefix, stamp fingerprint per row.
- migration 20260712T120000: add nullable page_embeddings.fingerprint + composite index.
- infra: TEI embeddings sidecar in docker-compose + EMBEDDING_*/SEARCH_* in .env.example.
- tests: 7 semantic int cases (vector-only hit, sidecar-down, no-provider,
  permission-over-union fail-closed, hung sidecar, lexical∪vector de-dup,
  fingerprint isolation) + fingerprint/prefix/timeout unit tests.

total now = permission-filtered size of (lexical ∪ vector top-N) — a documented
change from Phase A's exact lexical count (falls back to it on degrade).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 20:21:19 +03:00
vvzvlad e074b101c7 Merge pull request 'feat(#370 Stage A): MCP save_page_version — агентские версии страниц' (#565) from feat/370-agent-save-version into develop
Reviewed-on: #565
2026-07-12 19:56:05 +03:00
vvzvlad 4496afb481 Merge pull request 'recover(page-tree/dnd): вернуть в develop осиротевший фикс drop-раскрытия (#523/#532)' (#573) from fix/523-tree-dnd into develop
Reviewed-on: #573
2026-07-12 19:43:34 +03:00
vvzvlad 78fc7c4842 Merge pull request 'recover(api-key): вернуть в develop осиротевший UI фазы-2 (#506/#533)' (#572) from recover/506-apikey-ui into develop
Reviewed-on: #572
2026-07-12 19:24:36 +03:00
agent_coder be25b31a0e fix(api-key): add missing i18n keys for api-key UI + correct service comment
en-US (source of truth) and ru-RU were missing ~25 strings introduced by the
api-key management UI, so Russian users saw the controls in English and en-US
was incomplete. Add every new t() key to both locales (24 new in en-US, 26 in
ru-RU) with natural Russian translations, placeholders ({{name}}/{{date}})
preserved; existing keys left untouched (insertions only, no reordering).

Also fix the createApiKey() lifecycle comment: the gcTime:0 + invalidation
lives in queries/api-key-query.ts (there is no use-api-key-query file) and the
reset()-after-read of the token lives in components/api-keys-manager.tsx
handleCreate (create-api-key-modal.tsx only resets the form).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:21:27 +03:00
agent_coder d6411424c1 feat(api-key): distinguish expired from expiring-soon + cover service unwrap
Fix 1: an already-expired key rendered the forward-looking "Expiring soon"
badge/tooltip ("expires within 30 days"), which is false for a past expiry.
Add a pure isExpired() helper and, in ApiKeysManager, render a red "Expired"
badge ("This key has expired") for past expiries; keep the orange "Expiring
soon" badge only for keys expiring in the future within 30 days (the two
states are made mutually exclusive at the call site). Extend utils.test.ts
with isExpired coverage (past/future/null/exact-now boundary) and add a
component assertion that an expired key shows "Expired" and NOT "Expiring
soon".

Fix 2: the response-contract unwrap (res.data turning the server envelope
{data:{token,apiKey},success,status} into {token,apiKey}) was exercised by no
test — component tests fully mock the service. Add a service-layer unit test
that mocks the api-client (axios instance) and asserts createApiKey unwraps to
the inner {token,apiKey} payload and getApiKeys resolves to the row array.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:20:17 +03:00
agent_coder 199a9a1750 feat(client): страница управления API-ключами в настройках аккаунта (#506)
Фаза 2 поверх серверных эндпоинтов #501: страница «Настройки → Аккаунт →
API-ключи» (lazy-роут, отдельный чанк) — список, создание, показ токена
один раз и отзыв.

- Список: явная дата истечения (не «через N дней»), подсветка ключей с
  истечением < 30 дней, «использован» с семантикой «в течение последнего
  часа» (last_used_at троттлится на 1 ч серверно).
- Создание: имя + срок (30д/90д/1 год[дефолт]/бессрочный → null). После
  submit — модалка показа токена один раз с копированием и явной датой.
- Токен-материал живёт ТОЛЬКО в state открытой модалки: mutateAsync +
  reset() чистит копию из query-cache, ничего не пишется в localStorage.
  Закрытие модалки — токен исчезает навсегда.
- Отзыв: подтверждение → revoke → строка уходит из списка.
- Admin (CASL Manage на API = owner/admin) видит ключи всего воркспейса с
  колонкой «автор»; обычный член — только свои.

Тесты (vitest): показ-один-раз + отсутствие токена в localStorage/кэше,
явная дата + подсветка < 30 дней, отзыв убирает строку, admin/member вид,
дефолт срока = 1 год и «бессрочный» → null. Стаб ResizeObserver добавлен в
общий vitest.setup для рендера Mantine ScrollArea/Table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 19:20:17 +03:00
agent_coder 27d51303ba fix(page-tree/dnd): drag авто-раскрытие — задержка 2с, не раскрывать при drop внутрь + guard от потери детей и сигнал «уехало сюда»
Пять точечных клиентских правок (дизайн из адверсариального разбора #523):

1. Задержка hover-раскрытия при drag: `AUTO_EXPAND_MS` 500 → 2000 мс, чтобы
   проведение курсора через дерево не раскрывало ряды — только осознанное
   удержание.
2. Не раскрывать цель при drop внутрь (make-child): удалена строка
   `onToggle(target, true)` в `onDrop` (drop оставляет узел свёрнутым; раскрытие
   только по hover-hold). Восстановление раскрытости самого source не тронуто.
3. Guard от потери детей в `handleMove` (обязателен вместе с п.2): раньше эта же
   `onToggle` была ЕДИНСТВЕННЫМ триггером корректирующего lazy-load. `treeModel.move`
   материализует `target.children = [source]`; для НЕзагруженной цели (предикат
   gate `treeModel.isUnloadedBranch`) это скрыло бы остальных серверных детей папки
   (класс #159 #1). Теперь для незагруженной make-child-цели оптимистичное дерево
   строится БЕЗ материализации source: source удаляется из старого родителя, цели
   ставится `hasChildren:true`; gate остаётся взведён и раскрытие догружает полный
   набор. Для загруженной цели поведение прежнее (append сохраняет детей).
4. Инвалидация крошек: `updateCacheOnMovePage` инвалидирует
   `["breadcrumbs", pageId]` — guard делает `remove(source)`, и для текущей
   открытой страницы `findBreadcrumbPath` промахивается → крошки показывали бы
   старого родителя до refocus.
5. Сигнал «уехало сюда»: транзиентный teal-пульс на строке при make-child-drop в
   свёрнутую цель (узел НЕ раскрывается), отдельный `data-landed-child` +
   `DROP_LANDED_HIGHLIGHT_MS`, таймер чистится при анмаунте; уважает
   prefers-reduced-motion.

Тесты: `handleMove` — незагруженная make-child-цель не материализует `[source]`
(мутация guard'а краснит), загруженная цель append'ит и сохраняет детей,
genuinely-empty лист материализует; инвалидация `["breadcrumbs", pageId]`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-12 01:15:02 +03:00
39 changed files with 2879 additions and 66 deletions
+36
View File
@@ -198,6 +198,42 @@ MCP_DOCMOST_PASSWORD=
# A slow/hung embeddings endpoint fails after this and the batch continues.
# AI_EMBEDDING_TIMEOUT_MS=120000
# ─── #530 Semantic search (Phase B) ──────────────────────────────────────────
# The GLOBAL embedding provider used by search (query embed) AND the indexer
# (document embed) when a workspace has NO embedding provider of its own. It is
# an OpenAI-compatible endpoint — the docker-compose `embeddings` (TEI) sidecar.
# A workspace that configures its own embedding provider OVERRIDES all of this.
# When neither resolves, search runs lexical-only (semantic.reason=no-provider).
EMBEDDING_ENDPOINT=http://embeddings:80/v1
EMBEDDING_MODEL=intfloat/multilingual-e5-small
# Dummy — the self-hosted TEI sidecar is keyless.
EMBEDDING_API_KEY=unused
EMBEDDING_DIMENSIONS=384
# PLACEHOLDER — set to a PINNED intfloat/multilingual-e5-small commit sha (used
# both as the TEI --revision and as part of the embedding fingerprint). Keep this
# in lockstep with docker-compose; a bump changes the fingerprint (PR-2 handles
# the generational swap/GC — PR-1 uses a single active fingerprint).
EMBEDDING_REVISION=REPLACE_WITH_PINNED_E5_SMALL_COMMIT_SHA
# e5 models require these input prefixes. Empty them for a non-e5 model.
EMBEDDING_QUERY_PREFIX="query: "
EMBEDDING_DOC_PREFIX="passage: "
# Per-request timeout (ms) for the interactive SEARCH query embed — much shorter
# than the batch indexer timeout: a slow/hung sidecar degrades search fast to the
# lexical-only path instead of blocking the request. Default 800.
# SEARCH_EMBED_TIMEOUT_MS=800
# RRF weight of the vector leg relative to each lexical leg (1.0 = equal). Default 1.0.
# SEARCH_VECTOR_WEIGHT=1.0
# Vector top-N pulled into the fused candidate union per request. Default 50.
# SEARCH_VECTOR_CANDIDATES=50
# Per-statement timeout (ms) bounding ONLY the fused vector query (the brute-force
# KNN seq scan — there is no ANN index). If the vector scan exceeds this it is
# cancelled and search degrades to lexical-only (never hangs the request). Scoped
# per-query (SET LOCAL), so it never affects other queries. Default 2000.
# SEARCH_VECTOR_STATEMENT_TIMEOUT_MS=2000
# Kill-switch: set to `off` to disable the semantic layer entirely (search then
# runs the byte-identical Phase-A lexical path). Default on.
# SEARCH_SEMANTIC=on
# Silence timeout (ms) for streaming chat/agent AI calls AND external-MCP traffic.
# Bounds time-to-first-byte and the gap BETWEEN chunks (NOT the total turn length),
# so an arbitrarily long turn that keeps streaming is never cut. Finite so a hung
@@ -1,4 +1,7 @@
{
"1 year": "1 year",
"30 days": "30 days",
"90 days": "90 days",
"A new version is available": "A new version is available",
"Account": "Account",
"Active": "Active",
@@ -10,11 +13,16 @@
"Add space members": "Add space members",
"Add to favorites": "Add to favorites",
"Admin": "Admin",
"API key created": "API key created",
"API key revoked": "API key revoked",
"API keys across the workspace.": "API keys across the workspace.",
"Are you sure you want to delete this group? Members will lose access to resources this group has access to.": "Are you sure you want to delete this group? Members will lose access to resources this group has access to.",
"Are you sure you want to delete this page?": "Are you sure you want to delete this page?",
"Are you sure you want to remove this user from the group? The user will lose access to resources this group has access to.": "Are you sure you want to remove this user from the group? The user will lose access to resources this group has access to.",
"Are you sure you want to remove this user from the space? The user will lose all access to this space.": "Are you sure you want to remove this user from the space? The user will lose all access to this space.",
"Are you sure you want to restore this version? Any changes not versioned will be lost.": "Are you sure you want to restore this version? Any changes not versioned will be lost.",
"Are you sure you want to revoke \"{{name}}\"? Any client using this key will immediately lose access. This cannot be undone.": "Are you sure you want to revoke \"{{name}}\"? Any client using this key will immediately lose access. This cannot be undone.",
"Author": "Author",
"Can become members of groups and spaces in workspace": "Can become members of groups and spaces in workspace",
"Can create and edit pages in space.": "Can create and edit pages in space.",
"Can edit": "Can edit",
@@ -33,7 +41,9 @@
"Confirm": "Confirm",
"Copy as Markdown": "Copy as Markdown",
"Copy link": "Copy link",
"Copy your API key now and store it somewhere safe. For security reasons it will not be shown again.": "Copy your API key now and store it somewhere safe. For security reasons it will not be shown again.",
"Create": "Create",
"Create API key": "Create API key",
"Create group": "Create group",
"Create page": "Create page",
"Create space": "Create space",
@@ -55,7 +65,14 @@
"e.g Sales": "e.g Sales",
"e.g Space for product team": "e.g Space for product team",
"e.g Space for sales team to collaborate": "e.g Space for sales team to collaborate",
"e.g. CI deploy token": "e.g. CI deploy token",
"Edit": "Edit",
"Expiring soon": "Expiring soon",
"Failed to create API key": "Failed to create API key",
"Failed to load API keys.": "Failed to load API keys.",
"Failed to revoke API key": "Failed to revoke API key",
"Never used": "Never used",
"No API keys yet": "No API keys yet",
"Read": "Read",
"Edit group": "Edit group",
"Email": "Email",
@@ -134,6 +151,8 @@
"page": "page",
"Page deleted successfully": "Page deleted successfully",
"Page history": "Page history",
"Revoke API key": "Revoke API key",
"Revoke {{name}}": "Revoke {{name}}",
"Select version": "Select version",
"Highlight changes": "Highlight changes",
"Page import is in progress. Please do not close this tab.": "Page import is in progress. Please do not close this tab.",
@@ -189,6 +208,9 @@
"Template": "Template",
"Templates": "Templates",
"Theme": "Theme",
"This key expires within 30 days": "This key expires within 30 days",
"This key has expired": "This key has expired",
"This key never expires": "This key never expires",
"To change your email, you have to enter your password and new email.": "To change your email, you have to enter your password and new email.",
"Toggle full page width": "Toggle full page width",
"Unable to import pages. Please try again.": "Unable to import pages. Please try again.",
@@ -196,6 +218,7 @@
"Untitled": "Untitled",
"Updated successfully": "Updated successfully",
"User": "User",
"Within the last hour": "Within the last hour",
"Workspace": "Workspace",
"Workspace Name": "Workspace Name",
"Workspace settings": "Workspace settings",
@@ -426,6 +449,7 @@
"Write anything. Enter \"/\" for commands": "Write anything. Enter \"/\" for commands",
"Write...": "Write...",
"Column count": "Column count",
"Your personal API keys.": "Your personal API keys.",
"{{count}} Columns": "{{count}} Columns",
"{{count}} command available_one": "1 command available",
"{{count}} command available_other": "{{count}} commands available",
@@ -1,4 +1,7 @@
{
"1 year": "1 год",
"30 days": "30 дней",
"90 days": "90 дней",
"A new version is available": "Доступна новая версия",
"Account": "Аккаунт",
"Active": "Активный",
@@ -10,11 +13,16 @@
"Add space members": "Добавить участников пространства",
"Add to favorites": "Добавить в избранное",
"Admin": "Администратор",
"API key created": "API ключ создан",
"API key revoked": "API ключ отозван",
"API keys across the workspace.": "API ключи всего рабочего пространства.",
"Are you sure you want to delete this group? Members will lose access to resources this group has access to.": "Вы уверены, что хотите удалить эту группу? Участники потеряют доступ к материалам, к которым у этой группы есть доступ.",
"Are you sure you want to delete this page?": "Вы уверены, что хотите удалить эту страницу?",
"Are you sure you want to remove this user from the group? The user will lose access to resources this group has access to.": "Вы уверены, что хотите удалить этого пользователя из группы? Пользователь потеряет доступ к материалам, к которым есть доступ у этой группы.",
"Are you sure you want to remove this user from the space? The user will lose all access to this space.": "Вы уверены, что хотите удалить этого пользователя из пространства? Пользователь потеряет весь доступ к этому пространству.",
"Are you sure you want to restore this version? Any changes not versioned will be lost.": "Вы уверены, что хотите восстановить эту версию? Все не зафиксированные изменения будут потеряны.",
"Are you sure you want to revoke \"{{name}}\"? Any client using this key will immediately lose access. This cannot be undone.": "Вы уверены, что хотите отозвать «{{name}}»? Любой клиент, использующий этот ключ, немедленно потеряет доступ. Это действие нельзя отменить.",
"Author": "Автор",
"Can become members of groups and spaces in workspace": "Могут становиться участниками групп и пространств в рабочей области",
"Can create and edit pages in space.": "Может создавать и редактировать страницы в пространстве.",
"Can edit": "Может изменять",
@@ -33,7 +41,9 @@
"Confirm": "Подтвердить",
"Copy as Markdown": "Копировать как Markdown",
"Copy link": "Копировать ссылку",
"Copy your API key now and store it somewhere safe. For security reasons it will not be shown again.": "Скопируйте API ключ сейчас и сохраните его в надёжном месте. В целях безопасности он больше не будет показан.",
"Create": "Создать",
"Create API key": "Создать API ключ",
"Create group": "Создать группу",
"Create page": "Создать страницу",
"Create space": "Создать пространство",
@@ -46,6 +56,7 @@
"Are you sure you want to delete this page? This will delete its children and page history. This action is irreversible.": "Вы уверены, что хотите удалить эту страницу? Это удалит её дочерние страницы, а также историю страницы. Это действие необратимо.",
"Description": "Описание",
"Details": "Подробности",
"Done": "Готово",
"e.g ACME": "например, ACME",
"e.g ACME Inc": "например, ACME Inc",
"e.g Developers": "например, Developers",
@@ -55,7 +66,15 @@
"e.g Sales": "например, Sales",
"e.g Space for product team": "например, Пространство для команды продукта",
"e.g Space for sales team to collaborate": "например, Пространство для совместной работы команды продаж",
"e.g. CI deploy token": "например, токен деплоя CI",
"Edit": "Редактировать",
"Expiring soon": "Скоро истекает",
"Failed to create API key": "Не удалось создать API ключ",
"Failed to load API keys.": "Не удалось загрузить API ключи.",
"Failed to revoke API key": "Не удалось отозвать API ключ",
"Name is required": "Имя обязательно",
"Never used": "Не использовался",
"No API keys yet": "Пока нет API ключей",
"Read": "Чтение",
"Edit group": "Редактировать группу",
"Email": "Электронная почта",
@@ -134,6 +153,8 @@
"page": "страница",
"Page deleted successfully": "Страница успешно удалена",
"Page history": "История страницы",
"Revoke API key": "Отозвать API ключ",
"Revoke {{name}}": "Отозвать {{name}}",
"Select version": "Выбрать версию",
"Highlight changes": "Выделить изменения",
"Page import is in progress. Please do not close this tab.": "Импорт страницы в процессе. Пожалуйста, не закрывайте эту вкладку.",
@@ -189,6 +210,9 @@
"Template": "Шаблон",
"Templates": "Шаблоны",
"Theme": "Тема",
"This key expires within 30 days": "Этот ключ истекает в течение 30 дней",
"This key has expired": "Этот ключ истек",
"This key never expires": "Этот ключ никогда не истекает",
"To change your email, you have to enter your password and new email.": "Чтобы изменить электронную почту, вам нужно ввести пароль и новый адрес.",
"Toggle full page width": "Переключить полную ширину страницы",
"Unable to import pages. Please try again.": "Не удалось импортировать страницы. Пожалуйста, попробуйте ещё раз.",
@@ -196,6 +220,7 @@
"Untitled": "Без названия",
"Updated successfully": "Успешно обновлено",
"User": "Пользователь",
"Within the last hour": "За последний час",
"Workspace": "Рабочее пространство",
"Workspace Name": "Название рабочего пространства",
"Workspace settings": "Настройки рабочего пространства",
@@ -432,6 +457,7 @@
"Write anything. Enter \"/\" for commands": "Пишите что угодно. Введите \"/\" для команд",
"Write...": "Напишите...",
"Column count": "Количество столбцов",
"Your personal API keys.": "Ваши личные API ключи.",
"{{count}} Columns": "{count, plural, one{# столбец} few{# столбца} many{# столбцов} other{# столбца}}",
"{{count}} command available_one": "Доступна 1 команда",
"{{count}} command available_other": "Доступно {{count}} команд",
+7
View File
@@ -44,6 +44,12 @@ const AccountSettings = lazy(
const AccountPreferences = lazy(
() => import("@/pages/settings/account/account-preferences.tsx"),
);
// #506 — lazy leaf (own chunk): the API-keys management page is route-split so
// its code (Mantine table/modals + the create/revoke flow) stays out of the
// entry bundle (post-#342 bundle discipline).
const AccountApiKeys = lazy(
() => import("@/pages/settings/account/account-api-keys.tsx"),
);
const WorkspaceSettings = lazy(
() => import("@/pages/settings/workspace/workspace-settings"),
);
@@ -105,6 +111,7 @@ export default function App() {
path={"account/preferences"}
element={<AccountPreferences />}
/>
<Route path={"account/api-keys"} element={<AccountApiKeys />} />
<Route path={"workspace"} element={<WorkspaceSettings />} />
<Route path={"ai"} element={<AiSettings />} />
<Route path={"members"} element={<WorkspaceMembers />} />
@@ -10,6 +10,7 @@ import {
IconBrush,
IconWorld,
IconSparkles,
IconKey,
} from "@tabler/icons-react";
import { Link, useLocation } from "react-router-dom";
import classes from "./settings.module.css";
@@ -46,6 +47,11 @@ const groupedData: DataGroup[] = [
icon: IconBrush,
path: "/settings/account/preferences",
},
{
label: "API keys",
icon: IconKey,
path: "/settings/account/api-keys",
},
],
},
{
@@ -0,0 +1,234 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import {
render,
screen,
fireEvent,
waitFor,
within,
} from "@testing-library/react";
import { MantineProvider } from "@mantine/core";
import { ModalsProvider } from "@mantine/modals";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { Provider, createStore } from "jotai";
import { UserRole } from "@/lib/types";
import { currentUserAtom } from "@/features/user/atoms/current-user-atom";
import { IApiKey } from "@/features/api-key/types/api-key.types";
// Mock the service layer so no real HTTP is attempted; every test drives the
// component through these three functions.
vi.mock("@/features/api-key/services/api-key-service", () => ({
getApiKeys: vi.fn(),
createApiKey: vi.fn(),
revokeApiKey: vi.fn(),
}));
import {
getApiKeys,
createApiKey,
revokeApiKey,
} from "@/features/api-key/services/api-key-service";
import ApiKeysManager from "./api-keys-manager";
// matchMedia (read by MantineProvider) is stubbed globally in vitest.setup.ts.
const ISO_SOON = new Date(Date.now() + 10 * 864e5).toISOString();
const ISO_FAR = new Date(Date.now() + 200 * 864e5).toISOString();
const ISO_EXPIRED = new Date(Date.now() - 3 * 864e5).toISOString();
// Dump the storage stub via the Web Storage API — its data lives in a closure
// (see vitest.setup.ts), so JSON.stringify(localStorage) would be vacuous.
function storageDump(): string {
let out = "";
for (let i = 0; i < localStorage.length; i++) {
const k = localStorage.key(i) as string;
out += `${k}=${localStorage.getItem(k)};`;
}
return out;
}
function makeKey(overrides: Partial<IApiKey> = {}): IApiKey {
return {
id: "key-1",
name: "CI token",
expiresAt: ISO_FAR,
lastUsedAt: null,
createdAt: "2026-01-01T00:00:00.000Z",
creator: { id: "u1", name: "Alice", email: "a@x.io", avatarUrl: null },
...overrides,
};
}
function renderManager(role: UserRole) {
const store = createStore();
store.set(currentUserAtom, {
user: { id: "me", role } as never,
workspace: {} as never,
});
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: false } },
});
const utils = render(
<Provider store={store}>
<QueryClientProvider client={queryClient}>
<MantineProvider>
<ModalsProvider>
<ApiKeysManager />
</ModalsProvider>
</MantineProvider>
</QueryClientProvider>
</Provider>,
);
return { store, queryClient, ...utils };
}
beforeEach(() => {
vi.clearAllMocks();
localStorage.clear();
});
describe("ApiKeysManager — list rendering", () => {
it("renders an explicit expiry date and highlights a <30-day key (acceptance #3)", async () => {
vi.mocked(getApiKeys).mockResolvedValue([
makeKey({ id: "k-soon", name: "Soon key", expiresAt: ISO_SOON }),
makeKey({ id: "k-far", name: "Far key", expiresAt: ISO_FAR }),
]);
renderManager(UserRole.MEMBER);
await screen.findByText("Soon key");
// Explicit dates, not "in N days": the year is rendered verbatim.
const soonYear = new Date(ISO_SOON).getFullYear().toString();
expect(screen.getAllByText(new RegExp(soonYear)).length).toBeGreaterThan(0);
// Exactly one key is inside the 30-day warning window.
expect(screen.getAllByText("Expiring soon")).toHaveLength(1);
});
it('shows "Expired" (not "Expiring soon") for an already-expired key', async () => {
vi.mocked(getApiKeys).mockResolvedValue([
makeKey({ id: "k-dead", name: "Dead key", expiresAt: ISO_EXPIRED }),
]);
renderManager(UserRole.MEMBER);
await screen.findByText("Dead key");
// A past expiry is labelled "Expired", never the forward-looking badge.
expect(screen.getByText("Expired")).toBeDefined();
expect(screen.queryByText("Expiring soon")).toBeNull();
});
it('shows "Never" for an unlimited key and no highlight', async () => {
vi.mocked(getApiKeys).mockResolvedValue([
makeKey({ id: "k-forever", name: "Forever", expiresAt: null }),
]);
renderManager(UserRole.MEMBER);
await screen.findByText("Forever");
expect(screen.getByText("Never")).toBeDefined();
expect(screen.queryByText("Expiring soon")).toBeNull();
});
it("empty list shows the empty state", async () => {
vi.mocked(getApiKeys).mockResolvedValue([]);
renderManager(UserRole.MEMBER);
expect(await screen.findByText("No API keys yet")).toBeDefined();
});
});
describe("ApiKeysManager — admin vs member view (acceptance #6)", () => {
it("a member does NOT see the author column", async () => {
vi.mocked(getApiKeys).mockResolvedValue([
makeKey({ creator: { id: "me", name: "Me", email: "m@x.io", avatarUrl: null } }),
]);
renderManager(UserRole.MEMBER);
await screen.findByText("CI token");
// Author header absent + creator name not rendered (no author column).
expect(screen.queryByText("Author")).toBeNull();
expect(screen.queryByText("Me")).toBeNull();
});
it("an admin sees the author column with the creator's name", async () => {
vi.mocked(getApiKeys).mockResolvedValue([
makeKey({ creator: { id: "u1", name: "Alice", email: "a@x.io", avatarUrl: null } }),
]);
renderManager(UserRole.ADMIN);
await screen.findByText("CI token");
expect(screen.getByText("Author")).toBeDefined();
expect(screen.getByText("Alice")).toBeDefined();
});
});
describe("ApiKeysManager — revoke (acceptance #4)", () => {
it("revoke removes the row from the list", async () => {
vi.mocked(getApiKeys)
.mockResolvedValueOnce([
makeKey({ id: "k1", name: "Doomed" }),
makeKey({ id: "k2", name: "Survivor" }),
])
// After revoke, invalidation refetches the reduced list.
.mockResolvedValue([makeKey({ id: "k2", name: "Survivor" })]);
vi.mocked(revokeApiKey).mockResolvedValue();
renderManager(UserRole.MEMBER);
await screen.findByText("Doomed");
fireEvent.click(screen.getByLabelText("Revoke Doomed"));
// Confirm modal → click the destructive confirm button.
const confirm = await screen.findByRole("button", { name: "Revoke" });
fireEvent.click(confirm);
await waitFor(() =>
expect(screen.queryByText("Doomed")).toBeNull(),
);
expect(screen.getByText("Survivor")).toBeDefined();
expect(revokeApiKey).toHaveBeenCalledWith("k1");
});
});
describe("ApiKeysManager — show-once token (acceptance #1 & #2)", () => {
it("shows the token once, then discards it from the UI, localStorage and query cache", async () => {
const SECRET = "gm_secret-token-value-xyz";
vi.mocked(getApiKeys).mockResolvedValue([]);
vi.mocked(createApiKey).mockResolvedValue({
token: SECRET,
apiKey: {
id: "new-1",
name: "My key",
expiresAt: ISO_FAR,
createdAt: new Date().toISOString(),
},
});
const { queryClient } = renderManager(UserRole.MEMBER);
await screen.findByText("No API keys yet");
// Open create modal (use the header CTA), fill the name, submit.
fireEvent.click(
screen.getAllByRole("button", { name: "Create API key" })[0],
);
const nameInput = await screen.findByLabelText(/Name/);
fireEvent.change(nameInput, { target: { value: "My key" } });
fireEvent.click(screen.getByRole("button", { name: "Create" }));
// Token is shown exactly once in the show-once modal.
const tokenEl = await screen.findByTestId("api-key-token");
expect(tokenEl.textContent).toBe(SECRET);
// Close the modal → token discarded from the DOM.
fireEvent.click(screen.getByRole("button", { name: "Done" }));
await waitFor(() =>
expect(screen.queryByTestId("api-key-token")).toBeNull(),
);
// Acceptance #2: the secret is nowhere in localStorage or the react-query
// caches (query cache never carried it; the mutation copy was reset()).
// Non-vacuous: currentUser IS in storage, so the dump is exercised.
const dump = storageDump();
expect(dump).toContain("currentUser");
expect(dump).not.toContain(SECRET);
const cacheDump = JSON.stringify(
queryClient.getQueryCache().getAll().map((q) => q.state.data),
);
expect(cacheDump).not.toContain(SECRET);
const mutationDump = JSON.stringify(
queryClient.getMutationCache().getAll().map((m) => m.state.data),
);
expect(mutationDump).not.toContain(SECRET);
});
});
@@ -0,0 +1,264 @@
import { useState } from "react";
import {
ActionIcon,
Badge,
Button,
Center,
Group,
Loader,
Stack,
Table,
Text,
Tooltip,
} from "@mantine/core";
import { modals } from "@mantine/modals";
import { notifications } from "@mantine/notifications";
import { IconAlertTriangle, IconKey, IconTrash } from "@tabler/icons-react";
import { useTranslation } from "react-i18next";
import useUserRole from "@/hooks/use-user-role.tsx";
import { formatLocalized, useDateFnsLocale } from "@/lib/date-locale";
import { timeAgo } from "@/lib/time";
import {
useApiKeysQuery,
useCreateApiKeyMutation,
useRevokeApiKeyMutation,
} from "@/features/api-key/queries/api-key-query";
import {
IApiKey,
ICreateApiKeyResponse,
} from "@/features/api-key/types/api-key.types";
import {
isExpired,
isExpiringSoon,
lastUsedBucket,
} from "@/features/api-key/utils";
import { CreateApiKeyModal } from "./create-api-key-modal";
import { ShowTokenModal } from "./show-token-modal";
export default function ApiKeysManager() {
const { t } = useTranslation();
const locale = useDateFnsLocale();
// Manage-on-API === Owner/Admin server-side, so the admin (workspace-wide)
// list + "author" column mirror exactly the roles the server serves the
// whole-workspace response to.
const { isAdmin } = useUserRole();
const { data: keys, isLoading, isError } = useApiKeysQuery();
const createMutation = useCreateApiKeyMutation();
const revokeMutation = useRevokeApiKeyMutation();
const [createOpened, setCreateOpened] = useState(false);
// SECURITY: the show-once token lives ONLY here, in this component's local
// state. It is never written to localStorage, the query cache or a log. Closing
// the modal sets it back to null (handleCloseToken) — discarded forever.
const [createdKey, setCreatedKey] = useState<ICreateApiKeyResponse | null>(
null,
);
const handleCreate = async (values: {
name: string;
expiresAt: string | null;
}): Promise<boolean> => {
try {
const res = await createMutation.mutateAsync(values);
setCreateOpened(false);
// Move the token into local state, then immediately purge react-query's
// own copy of the mutation result so the secret does not linger in the
// mutation cache.
setCreatedKey(res);
createMutation.reset();
return true;
} catch {
notifications.show({
message: t("Failed to create API key"),
color: "red",
});
return false;
}
};
const handleCloseToken = () => {
// Token discarded — the only copy the UI ever held is dropped here.
setCreatedKey(null);
};
const openRevokeModal = (key: IApiKey) =>
modals.openConfirmModal({
title: t("Revoke API key"),
centered: true,
children: (
<Text size="sm">
{t(
'Are you sure you want to revoke "{{name}}"? Any client using this key will immediately lose access. This cannot be undone.',
{ name: key.name },
)}
</Text>
),
labels: { confirm: t("Revoke"), cancel: t("Cancel") },
confirmProps: { color: "red" },
onConfirm: () => revokeMutation.mutate(key.id),
});
const formatDate = (iso: string) =>
formatLocalized(new Date(iso), "MMM dd, yyyy", "PP", locale);
const renderLastUsed = (lastUsedAt: string | null) => {
switch (lastUsedBucket(lastUsedAt)) {
case "never":
return t("Never used");
case "recent":
return t("Within the last hour");
case "stale":
// Coarse relative time — last_used_at is throttled to ~1h server-side,
// so we don't promise finer precision.
return timeAgo(new Date(lastUsedAt as string));
}
};
if (isLoading) {
return (
<Center py="xl">
<Loader size="sm" />
</Center>
);
}
const rows = (keys ?? []).map((key) => {
// Mutually exclusive: an already-expired key is labelled "Expired" (a past
// expiry) rather than the forward-looking "Expiring soon". isExpiringSoon
// also matches past expiries, so gate "soon" on !expired.
const expired = isExpired(key.expiresAt);
const soon = !expired && isExpiringSoon(key.expiresAt);
return (
<Table.Tr key={key.id}>
<Table.Td>
<Text fw={500}>{key.name}</Text>
</Table.Td>
<Table.Td>{formatDate(key.createdAt)}</Table.Td>
<Table.Td>
{key.expiresAt ? (
<Group gap={6} wrap="nowrap">
<Text size="sm">{formatDate(key.expiresAt)}</Text>
{expired && (
<Tooltip label={t("This key has expired")} withArrow>
<Badge
color="red"
variant="light"
size="sm"
leftSection={<IconAlertTriangle size={12} />}
>
{t("Expired")}
</Badge>
</Tooltip>
)}
{soon && (
<Tooltip
label={t("This key expires within 30 days")}
withArrow
>
<Badge
color="orange"
variant="light"
size="sm"
leftSection={<IconAlertTriangle size={12} />}
>
{t("Expiring soon")}
</Badge>
</Tooltip>
)}
</Group>
) : (
<Text size="sm" c="dimmed">
{t("Never")}
</Text>
)}
</Table.Td>
<Table.Td>
<Text size="sm" c="dimmed">
{renderLastUsed(key.lastUsedAt)}
</Text>
</Table.Td>
{isAdmin && (
<Table.Td>
<Text size="sm">
{key.creator?.name ?? key.creator?.email ?? t("Unknown")}
</Text>
</Table.Td>
)}
<Table.Td style={{ textAlign: "right" }}>
<Tooltip label={t("Revoke")} withArrow>
<ActionIcon
variant="subtle"
color="red"
aria-label={t("Revoke {{name}}", { name: key.name })}
onClick={() => openRevokeModal(key)}
>
<IconTrash size={16} />
</ActionIcon>
</Tooltip>
</Table.Td>
</Table.Tr>
);
});
return (
<>
<Group justify="space-between" mb="md">
<Text size="sm" c="dimmed">
{isAdmin
? t("API keys across the workspace.")
: t("Your personal API keys.")}
</Text>
<Button
leftSection={<IconKey size={16} />}
onClick={() => setCreateOpened(true)}
>
{t("Create API key")}
</Button>
</Group>
{isError ? (
<Text c="red" size="sm">
{t("Failed to load API keys.")}
</Text>
) : rows.length === 0 ? (
<Stack align="center" gap="xs" py="xl">
<IconKey size={32} opacity={0.4} />
<Text c="dimmed">{t("No API keys yet")}</Text>
<Button
variant="light"
leftSection={<IconKey size={16} />}
onClick={() => setCreateOpened(true)}
>
{t("Create API key")}
</Button>
</Stack>
) : (
<Table.ScrollContainer minWidth={600}>
<Table verticalSpacing="sm" highlightOnHover>
<Table.Thead>
<Table.Tr>
<Table.Th>{t("Name")}</Table.Th>
<Table.Th>{t("Created")}</Table.Th>
<Table.Th>{t("Expires")}</Table.Th>
<Table.Th>{t("Last used")}</Table.Th>
{isAdmin && <Table.Th>{t("Author")}</Table.Th>}
<Table.Th />
</Table.Tr>
</Table.Thead>
<Table.Tbody>{rows}</Table.Tbody>
</Table>
</Table.ScrollContainer>
)}
<CreateApiKeyModal
opened={createOpened}
onClose={() => setCreateOpened(false)}
onSubmit={handleCreate}
loading={createMutation.isPending}
/>
<ShowTokenModal created={createdKey} onClose={handleCloseToken} />
</>
);
}
@@ -0,0 +1,104 @@
import { Button, Group, Modal, Select, Stack, TextInput } from "@mantine/core";
import { useForm } from "@mantine/form";
import { useTranslation } from "react-i18next";
import {
ApiKeyLifetime,
DEFAULT_LIFETIME,
lifetimeToExpiresAt,
} from "@/features/api-key/utils";
interface Props {
opened: boolean;
onClose: () => void;
// Resolves the create request, returning true on success. The parent owns the
// mutation (and the token it returns); this modal only collects the name +
// lifetime. On failure (false) the form state is kept so the user can retry.
onSubmit: (values: {
name: string;
expiresAt: string | null;
}) => Promise<boolean>;
loading?: boolean;
}
interface FormValues {
name: string;
lifetime: ApiKeyLifetime;
}
export function CreateApiKeyModal({
opened,
onClose,
onSubmit,
loading,
}: Props) {
const { t } = useTranslation();
const form = useForm<FormValues>({
initialValues: {
name: "",
lifetime: DEFAULT_LIFETIME,
},
validate: {
name: (value) =>
value.trim().length === 0 ? t("Name is required") : null,
},
});
const lifetimeOptions: { value: ApiKeyLifetime; label: string }[] = [
{ value: "30d", label: t("30 days") },
{ value: "90d", label: t("90 days") },
{ value: "1y", label: t("1 year") },
{ value: "never", label: t("No expiration") },
];
const handleSubmit = form.onSubmit(async (values) => {
const ok = await onSubmit({
name: values.name.trim(),
expiresAt: lifetimeToExpiresAt(values.lifetime),
});
// Reset only after a successful submit so a failed create keeps the form
// state (the parent surfaces the error via a notification).
if (ok) form.reset();
});
const handleClose = () => {
form.reset();
onClose();
};
return (
<Modal
opened={opened}
onClose={handleClose}
title={t("Create API key")}
centered
>
<form onSubmit={handleSubmit}>
<Stack gap="sm">
<TextInput
label={t("Name")}
placeholder={t("e.g. CI deploy token")}
data-autofocus
withAsterisk
{...form.getInputProps("name")}
/>
<Select
label={t("Expiration")}
data={lifetimeOptions}
allowDeselect={false}
checkIconPosition="right"
{...form.getInputProps("lifetime")}
/>
<Group justify="flex-end" mt="xs">
<Button variant="default" onClick={handleClose} type="button">
{t("Cancel")}
</Button>
<Button type="submit" loading={loading}>
{t("Create")}
</Button>
</Group>
</Stack>
</form>
</Modal>
);
}
@@ -0,0 +1,107 @@
import {
Alert,
Button,
Code,
Group,
Modal,
Stack,
Text,
} from "@mantine/core";
import { IconAlertTriangle, IconCheck, IconCopy } from "@tabler/icons-react";
import { useTranslation } from "react-i18next";
import { CopyButton } from "@/components/common/copy-button";
import { formatLocalized, useDateFnsLocale } from "@/lib/date-locale";
import { ICreateApiKeyResponse } from "@/features/api-key/types/api-key.types";
interface Props {
// The freshly-created key incl. its token. Owned by the parent; this modal
// only renders it and never copies it into its own persistent state.
created: ICreateApiKeyResponse | null;
// Closing MUST discard the token in the parent (set the `created` prop back to
// null) — the token is shown exactly once.
onClose: () => void;
}
export function ShowTokenModal({ created, onClose }: Props) {
const { t } = useTranslation();
const locale = useDateFnsLocale();
const expiresAt = created?.apiKey.expiresAt ?? null;
return (
<Modal
opened={created !== null}
onClose={onClose}
title={t("API key created")}
centered
// No dismiss-on-outside-click: the token is irretrievable, so closing is a
// deliberate act (the user confirms they have saved it).
closeOnClickOutside={false}
>
{created && (
<Stack gap="sm">
<Alert
color="orange"
icon={<IconAlertTriangle size={18} />}
variant="light"
>
{t(
"Copy your API key now and store it somewhere safe. For security reasons it will not be shown again.",
)}
</Alert>
<div>
<Text size="sm" c="dimmed" mb={4}>
{t("Token")}
</Text>
<Group gap="xs" wrap="nowrap" align="flex-start">
<Code
block
data-testid="api-key-token"
style={{ flex: 1, wordBreak: "break-all" }}
>
{created.token}
</Code>
<CopyButton value={created.token}>
{({ copied, copy }) => (
<Button
variant="light"
size="xs"
color={copied ? "teal" : "blue"}
leftSection={
copied ? (
<IconCheck size={16} />
) : (
<IconCopy size={16} />
)
}
onClick={copy}
>
{copied ? t("Copied") : t("Copy")}
</Button>
)}
</CopyButton>
</Group>
</div>
<Text size="sm" c="dimmed">
{expiresAt
? t("Expires {{date}}", {
date: formatLocalized(
new Date(expiresAt),
"MMM dd, yyyy",
"PP",
locale,
),
})
: t("This key never expires")}
</Text>
<Group justify="flex-end" mt="xs">
<Button onClick={onClose}>{t("Done")}</Button>
</Group>
</Stack>
)}
</Modal>
);
}
@@ -0,0 +1,66 @@
import {
useMutation,
useQuery,
useQueryClient,
UseQueryResult,
} from "@tanstack/react-query";
import { notifications } from "@mantine/notifications";
import { useTranslation } from "react-i18next";
import {
createApiKey,
getApiKeys,
revokeApiKey,
} from "@/features/api-key/services/api-key-service";
import {
IApiKey,
ICreateApiKey,
ICreateApiKeyResponse,
} from "@/features/api-key/types/api-key.types";
export const API_KEYS_QUERY_KEY = ["api-keys"];
export function useApiKeysQuery(): UseQueryResult<IApiKey[], Error> {
return useQuery({
queryKey: API_KEYS_QUERY_KEY,
queryFn: () => getApiKeys(),
});
}
/**
* Create mutation.
*
* SECURITY: the response contains the token exactly once. This hook deliberately
* does NOT stash it anywhere the caller reads it from `mutateAsync`'s resolved
* value, moves it into the show-once modal's local state, then calls
* `mutation.reset()` to purge react-query's own copy immediately. `gcTime: 0`
* is a second belt so nothing lingers in the mutation cache after the observer
* unmounts. The list is invalidated here (the list carries no token).
*/
export function useCreateApiKeyMutation() {
const queryClient = useQueryClient();
return useMutation<ICreateApiKeyResponse, Error, ICreateApiKey>({
mutationFn: (data) => createApiKey(data),
gcTime: 0,
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: API_KEYS_QUERY_KEY });
},
});
}
export function useRevokeApiKeyMutation() {
const { t } = useTranslation();
const queryClient = useQueryClient();
return useMutation<void, Error, string>({
mutationFn: (id) => revokeApiKey(id),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: API_KEYS_QUERY_KEY });
notifications.show({ message: t("API key revoked") });
},
onError: () => {
notifications.show({
message: t("Failed to revoke API key"),
color: "red",
});
},
});
}
@@ -0,0 +1,78 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
// Mock the api-client (axios instance). vi.mock replaces the whole module, so
// the real response interceptor — which returns `response.data`, i.e. the
// server envelope { data, success, status } — is bypassed. Our mocked post/get
// therefore resolve to that post-interceptor envelope shape directly, and the
// service under test reads `.data` off it to unwrap the inner payload. This is
// the one bug-prone line the component tests (which fully mock the service)
// never exercise.
const { post, get } = vi.hoisted(() => ({ post: vi.fn(), get: vi.fn() }));
vi.mock("@/lib/api-client", () => ({
default: { post, get },
}));
import {
createApiKey,
getApiKeys,
} from "@/features/api-key/services/api-key-service";
beforeEach(() => {
vi.clearAllMocks();
});
describe("api-key-service response-contract unwrap", () => {
it("createApiKey unwraps the envelope to { token, apiKey }", async () => {
const payload = {
token: "gm_secret-abc",
apiKey: {
id: "key-1",
name: "CI token",
expiresAt: "2027-07-11T12:00:00.000Z",
createdAt: "2026-07-11T12:00:00.000Z",
},
};
// The server envelope as the interceptor hands it to the service.
post.mockResolvedValue({ data: payload, success: true, status: 200 });
const result = await createApiKey({
name: "CI token",
expiresAt: "2027-07-11T12:00:00.000Z",
});
// The inner payload only — not the { data, success, status } wrapper.
expect(result).toEqual(payload);
expect(result.token).toBe("gm_secret-abc");
expect(result.apiKey).toEqual(payload.apiKey);
expect(post).toHaveBeenCalledWith("/api-keys/create", {
name: "CI token",
expiresAt: "2027-07-11T12:00:00.000Z",
});
});
it("getApiKeys unwraps the envelope to the array of rows", async () => {
const rows = [
{
id: "key-1",
name: "CI token",
expiresAt: null,
lastUsedAt: null,
createdAt: "2026-01-01T00:00:00.000Z",
},
{
id: "key-2",
name: "Deploy token",
expiresAt: "2027-01-01T00:00:00.000Z",
lastUsedAt: null,
createdAt: "2026-02-01T00:00:00.000Z",
},
];
post.mockResolvedValue({ data: rows, success: true, status: 200 });
const result = await getApiKeys();
expect(result).toEqual(rows);
expect(result).toHaveLength(2);
expect(post).toHaveBeenCalledWith("/api-keys/list");
});
});
@@ -0,0 +1,30 @@
import api from "@/lib/api-client";
import {
IApiKey,
ICreateApiKey,
ICreateApiKeyResponse,
} from "@/features/api-key/types/api-key.types";
// Mint a new key. The response carries the token ONCE — the caller must move it
// straight into the show-once modal's local state and never cache it. See
// queries/api-key-query.ts (gcTime: 0 + query invalidation) and
// components/api-keys-manager.tsx `handleCreate` (createMutation.reset() right
// after reading the token) for the reset()-after-read pattern.
export async function createApiKey(
data: ICreateApiKey,
): Promise<ICreateApiKeyResponse> {
const res = await api.post<ICreateApiKeyResponse>("/api-keys/create", data);
return res.data as ICreateApiKeyResponse;
}
// List the caller's keys (or, for an admin, every key in the workspace with
// creator attribution). Never returns token material.
export async function getApiKeys(): Promise<IApiKey[]> {
const res = await api.post<IApiKey[]>("/api-keys/list");
return res.data as IApiKey[];
}
// Revocation is server-side immediate; the caller drops the row on success.
export async function revokeApiKey(id: string): Promise<void> {
await api.post("/api-keys/revoke", { id });
}
@@ -0,0 +1,48 @@
// Compact creator attribution embedded in the admin (workspace-wide) list. A
// normal member's list only ever contains their own keys, so the field is
// present but redundant; the author column is only rendered for admins.
export interface IApiKeyCreator {
id: string;
name: string;
email: string;
avatarUrl: string | null;
}
// A single api-key row as returned by `POST /api/api-keys/list`. Note: the list
// NEVER carries token material — only metadata.
export interface IApiKey {
id: string;
name: string;
// ISO string, or null for an unlimited ("never expires") key.
expiresAt: string | null;
// ISO string, or null if the key was never used. Throttled to ~1h server-side
// (#501), so the UI must not promise sub-hour precision.
lastUsedAt: string | null;
createdAt: string;
creator?: IApiKeyCreator | null;
}
// Payload for `POST /api/api-keys/create`. `expiresAt`: an ISO date string for a
// bounded lifetime, or null for an unlimited key. (undefined would let the
// server apply its 1-year default, but the form always sends an explicit value.)
export interface ICreateApiKey {
name: string;
expiresAt: string | null;
}
// The metadata half of the create response. The token itself is carried
// separately (see ICreateApiKeyResponse) and is shown exactly once.
export interface ICreatedApiKey {
id: string;
name: string;
expiresAt: string | null;
createdAt: string;
}
// Response of `POST /api/api-keys/create`. `token` is the ONLY time the secret
// is ever returned — it must live only in the show-once modal's local state and
// must never be cached, persisted or logged.
export interface ICreateApiKeyResponse {
token: string;
apiKey: ICreatedApiKey;
}
@@ -0,0 +1,100 @@
import { describe, it, expect } from "vitest";
import {
DEFAULT_LIFETIME,
EXPIRY_WARNING_DAYS,
isExpired,
isExpiringSoon,
lastUsedBucket,
lifetimeToExpiresAt,
} from "./utils";
const NOW = new Date("2026-07-11T12:00:00.000Z");
const daysFromNow = (n: number) =>
new Date(NOW.getTime() + n * 24 * 60 * 60 * 1000).toISOString();
describe("lifetimeToExpiresAt", () => {
it("default lifetime is 1 year (acceptance #7)", () => {
expect(DEFAULT_LIFETIME).toBe("1y");
const iso = lifetimeToExpiresAt("1y", NOW);
expect(iso).toBe("2027-07-11T12:00:00.000Z");
});
it('"never" sends null (acceptance #7)', () => {
expect(lifetimeToExpiresAt("never", NOW)).toBeNull();
});
it("30d / 90d map to the exact future instant", () => {
expect(lifetimeToExpiresAt("30d", NOW)).toBe(daysFromNow(30));
expect(lifetimeToExpiresAt("90d", NOW)).toBe(daysFromNow(90));
});
});
describe("isExpiringSoon (acceptance #3 highlight)", () => {
it("an unlimited key is never 'soon'", () => {
expect(isExpiringSoon(null, NOW)).toBe(false);
});
it("highlights a key expiring within the 30-day window", () => {
expect(isExpiringSoon(daysFromNow(EXPIRY_WARNING_DAYS - 1), NOW)).toBe(true);
expect(isExpiringSoon(daysFromNow(10), NOW)).toBe(true);
});
it("does not highlight a key well outside the window", () => {
expect(isExpiringSoon(daysFromNow(EXPIRY_WARNING_DAYS + 1), NOW)).toBe(
false,
);
expect(isExpiringSoon(daysFromNow(200), NOW)).toBe(false);
});
it("an already-expired key is highlighted", () => {
expect(isExpiringSoon(daysFromNow(-3), NOW)).toBe(true);
});
});
describe("isExpired (past vs future expiry)", () => {
it("an unlimited key is never expired", () => {
expect(isExpired(null, NOW)).toBe(false);
});
it("a past expiry is expired", () => {
expect(isExpired(daysFromNow(-3), NOW)).toBe(true);
expect(isExpired(daysFromNow(-1), NOW)).toBe(true);
});
it("a future expiry is not expired (even within the warning window)", () => {
expect(isExpired(daysFromNow(1), NOW)).toBe(false);
expect(isExpired(daysFromNow(EXPIRY_WARNING_DAYS - 1), NOW)).toBe(false);
expect(isExpired(daysFromNow(200), NOW)).toBe(false);
});
it("an expiry exactly at 'now' counts as expired (boundary is inclusive)", () => {
expect(isExpired(NOW.toISOString(), NOW)).toBe(true);
});
it("is mutually distinguishable from isExpiringSoon: expired vs soon-but-future", () => {
// A key 3 days in the past: expired, and (by design) also matches
// isExpiringSoon — the UI resolves this by checking isExpired first.
expect(isExpired(daysFromNow(-3), NOW)).toBe(true);
// A key 10 days in the future: NOT expired, but expiring soon.
expect(isExpired(daysFromNow(10), NOW)).toBe(false);
expect(isExpiringSoon(daysFromNow(10), NOW)).toBe(true);
});
});
describe("lastUsedBucket (within-the-last-hour semantics)", () => {
const minutesAgo = (n: number) =>
new Date(NOW.getTime() - n * 60 * 1000).toISOString();
it("null last-used is 'never'", () => {
expect(lastUsedBucket(null, NOW)).toBe("never");
});
it("under an hour is 'recent' (no sub-hour precision promised)", () => {
expect(lastUsedBucket(minutesAgo(5), NOW)).toBe("recent");
expect(lastUsedBucket(minutesAgo(59), NOW)).toBe("recent");
});
it("over an hour is 'stale'", () => {
expect(lastUsedBucket(minutesAgo(90), NOW)).toBe("stale");
});
});
+76
View File
@@ -0,0 +1,76 @@
import { addDays, addYears, differenceInMinutes } from "date-fns";
// The single early-warning window (#501/#506): keys whose expiry is closer than
// this are visually highlighted in the list. Also used to classify a key as
// already-expired (a negative "days until" is < 30 too).
export const EXPIRY_WARNING_DAYS = 30;
// last_used_at is throttled to ~1h server-side, so anything under an hour is
// shown as the coarse "within the last hour" rather than a false-precise
// "5 minutes ago".
export const LAST_USED_THROTTLE_MINUTES = 60;
export type ApiKeyLifetime = "30d" | "90d" | "1y" | "never";
export const DEFAULT_LIFETIME: ApiKeyLifetime = "1y";
// Maps a lifetime choice to the `expiresAt` payload sent to the server: an ISO
// string for a bounded lifetime, or null for an explicit unlimited key.
export function lifetimeToExpiresAt(
lifetime: ApiKeyLifetime,
now: Date = new Date(),
): string | null {
switch (lifetime) {
case "30d":
return addDays(now, 30).toISOString();
case "90d":
return addDays(now, 90).toISOString();
case "1y":
return addYears(now, 1).toISOString();
case "never":
return null;
}
}
// True when a bounded key's expiry is already in the past (or exactly now). An
// unlimited key (null) is never expired. This is distinct from "expiring soon":
// the two states are mutually exclusive at the call site (see api-keys-manager),
// so an already-expired key is labelled "Expired", not "Expiring soon".
export function isExpired(
expiresAt: string | null,
now: Date = new Date(),
): boolean {
if (!expiresAt) return false;
const expiry = new Date(expiresAt).getTime();
if (Number.isNaN(expiry)) return false;
return expiry <= now.getTime();
}
// True when a bounded key expires within the warning window (or is already
// expired). An unlimited key (null) is never "expiring soon". Callers that need
// to distinguish an already-past expiry should check isExpired() first, as this
// predicate deliberately also covers the already-expired case.
export function isExpiringSoon(
expiresAt: string | null,
now: Date = new Date(),
): boolean {
if (!expiresAt) return false;
const expiry = new Date(expiresAt).getTime();
if (Number.isNaN(expiry)) return false;
const msLeft = expiry - now.getTime();
return msLeft < EXPIRY_WARNING_DAYS * 24 * 60 * 60 * 1000;
}
// Classifies last-used recency so the caller can pick the honest label without
// promising sub-hour precision. Returns "never", "recent" (< 1h) or "stale".
export function lastUsedBucket(
lastUsedAt: string | null,
now: Date = new Date(),
): "never" | "recent" | "stale" {
if (!lastUsedAt) return "never";
const used = new Date(lastUsedAt);
if (Number.isNaN(used.getTime())) return "never";
return differenceInMinutes(now, used) < LAST_USED_THROTTLE_MINUTES
? "recent"
: "stale";
}
@@ -665,6 +665,13 @@ export function updateCacheOnMovePage(
pageData: Partial<IPage>,
) {
invalidatePageTree();
// Invalidate the moved page's breadcrumbs (#523). The tree-side child-loss
// guard removes the moved node from the local tree when its new parent is an
// unloaded branch, so `findBreadcrumbPath` misses it and the breadcrumb bar
// falls back to the server `["breadcrumbs", pageId]` query — which this move
// must invalidate, otherwise the crumbs keep showing the OLD parent until a
// refocus/navigation.
queryClient.invalidateQueries({ queryKey: ["breadcrumbs", pageId] });
// Remove page from old parent's cache
const oldQueryKey =
oldParentId === null
@@ -0,0 +1,40 @@
import { describe, it, expect, beforeEach, vi } from "vitest";
import type { IPage } from "@/features/page/types/page.types";
// A fresh QueryClient stands in for the app singleton (importing the real
// @/main.tsx would run ReactDOM.createRoot, which has no DOM root in jsdom).
vi.mock("@/main.tsx", async () => {
const { QueryClient } = await import("@tanstack/react-query");
return { queryClient: new QueryClient() };
});
import { queryClient } from "@/main.tsx";
import { updateCacheOnMovePage } from "./page-query";
// #523: the tree-side child-loss guard removes the moved node from the local
// tree when its new parent is an unloaded branch, so `findBreadcrumbPath` misses
// it and the breadcrumb bar falls back to the server `["breadcrumbs", pageId]`
// query. That query MUST be invalidated by a move, or the crumbs keep showing
// the OLD parent until a refocus/navigation.
describe("updateCacheOnMovePage — breadcrumbs invalidation (#523)", () => {
beforeEach(() => {
queryClient.clear();
vi.restoreAllMocks();
});
it("invalidates the moved page's ['breadcrumbs', pageId] query", () => {
const spy = vi.spyOn(queryClient, "invalidateQueries");
updateCacheOnMovePage("s1", "moved-page", "old-parent", "new-parent", {
id: "moved-page",
} as Partial<IPage>);
const invalidatedBreadcrumbs = spy.mock.calls.some(
([arg]) =>
Array.isArray((arg as { queryKey?: unknown[] })?.queryKey) &&
(arg as { queryKey: unknown[] }).queryKey[0] === "breadcrumbs" &&
(arg as { queryKey: unknown[] }).queryKey[1] === "moved-page",
);
expect(invalidatedBreadcrumbs).toBe(true);
});
});
@@ -55,7 +55,14 @@ type Props<T extends object> = {
};
const DRAG_TYPE = 'doc-tree-item';
const AUTO_EXPAND_MS = 500;
// Hover-hold before a collapsed row auto-expands during a drag. 2s (not ~0.5s)
// so merely dragging the cursor THROUGH the tree never expands rows — only a
// deliberate hold does (#523).
const AUTO_EXPAND_MS = 2000;
// How long the "a page just moved in here" cue stays on a collapsed target after
// a make-child drop. Long enough to notice at a glance during frequent
// collapsed-drops, short enough not to linger. Code-only UX constant (#523).
const DROP_LANDED_HIGHLIGHT_MS = 1800;
function DocTreeRowInner<T extends object>(props: Props<T>) {
const {
@@ -93,7 +100,11 @@ function DocTreeRowInner<T extends object>(props: Props<T>) {
const rowRef = useRef<HTMLElement>(null);
const [isDragging, setIsDragging] = useState(false);
const [instruction, setInstruction] = useState<Instruction | null>(null);
// Transient "just received a child" cue: a make-child drop no longer expands
// the (collapsed) target, so flash the row instead so the move isn't invisible.
const [landedChild, setLandedChild] = useState(false);
const autoExpandTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const landedChildTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const cancelAutoExpand = useCallback(() => {
if (autoExpandTimerRef.current) {
@@ -249,11 +260,24 @@ function DocTreeRowInner<T extends object>(props: Props<T>) {
? getDragLabel(sourceNode)
: 'item';
liveRegion.announce(`Moved ${sourceLabel} under ${parentName}.`);
// After a make-child drop, expand this row so the user sees the
// just-dropped child — especially important when the row had no
// children before (chevron just appeared) so the drop would
// otherwise be invisible.
if (op.kind === 'make-child') onToggle(node.id, true);
// Do NOT auto-expand the target on drop: a drop must leave the node
// collapsed. Intentional expansion is handled solely by the
// hover-hold timer (AUTO_EXPAND_MS). Feedback that the drop landed is
// given by the post-move flash + landed-cue highlight + live-region
// announce above. When the make-child target is collapsed, flash a
// distinct "child moved in here" cue on the row (it stays collapsed).
if (op.kind === 'make-child' && !isOpen) {
if (landedChildTimerRef.current) {
clearTimeout(landedChildTimerRef.current);
}
setLandedChild(true);
landedChildTimerRef.current = setTimeout(() => {
setLandedChild(false);
landedChildTimerRef.current = null;
}, DROP_LANDED_HIGHLIGHT_MS);
}
// Restore the openness of the MOVED page itself (source) — untouched
// by the above; the target is never expanded here.
if (source.data.isOpenOnDragStart) onToggle(sourceId, true);
},
}),
@@ -281,6 +305,17 @@ function DocTreeRowInner<T extends object>(props: Props<T>) {
useEffect(() => () => cancelAutoExpand(), [cancelAutoExpand]);
// Clear the landed-child cue timer on unmount (mirrors autoExpandTimerRef).
useEffect(
() => () => {
if (landedChildTimerRef.current) {
clearTimeout(landedChildTimerRef.current);
landedChildTimerRef.current = null;
}
},
[],
);
const effectiveInst =
instruction?.type === 'instruction-blocked'
? instruction.desired
@@ -317,6 +352,7 @@ function DocTreeRowInner<T extends object>(props: Props<T>) {
className={styles.node}
data-dragging={isDragging || undefined}
data-selected={isSelected || undefined}
data-landed-child={landedChild || undefined}
data-receiving-drop={
receivingDrop === 'make-child'
? blocked
@@ -0,0 +1,149 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { renderHook, act } from "@testing-library/react";
import { Provider, createStore } from "jotai";
import type { ReactNode } from "react";
import { treeDataAtom } from "@/features/page/tree/atoms/tree-data-atom.ts";
import { treeModel } from "@/features/page/tree/model/tree-model";
import type { SpaceTreeNode } from "@/features/page/tree/types";
// --- Boundary mocks: only the network/query + router/i18n surfaces the hook
// touches. The tree math (treeModel, dropOpToMovePayload) runs for real so the
// child-loss guard is exercised end-to-end.
const moveMutate = vi.fn().mockResolvedValue({});
const updateCacheOnMovePageMock = vi.fn();
vi.mock("@/features/page/queries/page-query.ts", () => ({
useCreatePageMutation: () => ({ mutateAsync: vi.fn() }),
useUpdatePageMutation: () => ({ mutateAsync: vi.fn() }),
useRemovePageMutation: () => ({ mutateAsync: vi.fn() }),
useMovePageMutation: () => ({ mutateAsync: moveMutate }),
updateCacheOnMovePage: (...args: unknown[]) =>
updateCacheOnMovePageMock(...args),
}));
vi.mock("react-router-dom", () => ({
useNavigate: () => vi.fn(),
useParams: () => ({ spaceSlug: "space", pageSlug: undefined }),
}));
vi.mock("react-i18next", () => ({
useTranslation: () => ({ t: (s: string) => s }),
}));
vi.mock("@mantine/notifications", () => ({
notifications: { show: vi.fn() },
}));
// Import AFTER mocks so the hook binds to them.
import { useTreeMutation } from "./use-tree-mutation";
function node(
id: string,
over: Partial<SpaceTreeNode> = {},
): SpaceTreeNode {
return {
id,
slugId: `slug-${id}`,
name: id.toUpperCase(),
position: "a0",
spaceId: "space-1",
parentPageId: null as unknown as string,
hasChildren: false,
children: [],
...over,
};
}
function setup(before: SpaceTreeNode[]) {
const store = createStore();
store.set(treeDataAtom, before);
const wrapper = ({ children }: { children: ReactNode }) => (
<Provider store={store}>{children}</Provider>
);
const { result } = renderHook(() => useTreeMutation("space-1"), { wrapper });
return { store, result };
}
describe("useTreeMutation.handleMove — child-loss guard (#523)", () => {
beforeEach(() => {
vi.clearAllMocks();
moveMutate.mockResolvedValue({});
});
it("make-child into an UNLOADED folder does NOT materialize [source] — keeps it lazy-loadable", async () => {
// F has children on the server but none are loaded here (canonical unloaded
// form: hasChildren + children:[]). X sits at root.
const before = [
node("F", { position: "a0", hasChildren: true, children: [] }),
node("X", { position: "a5" }),
];
const { store, result } = setup(before);
await act(async () => {
await result.current.handleMove("X", {
kind: "make-child",
targetId: "F",
});
});
const tree = store.get(treeDataAtom);
const f = treeModel.find(tree, "F");
// The guard leaves F unloaded (children stay []), so a later expand fetches
// the FULL server set (incl. X) instead of showing a misleading partial [X].
// MUTATION: dropping the guard (using the `move` result) would put children
// === [X] here and redden this.
expect(f?.children).toEqual([]);
expect(f?.hasChildren).toBe(true);
// X is removed from its old (root) slot; it reappears on expand/load of F.
expect(treeModel.find(tree, "X")).toBeNull();
// The server move is still persisted.
expect(moveMutate).toHaveBeenCalledTimes(1);
});
it("make-child into a LOADED folder appends source and KEEPS the existing children", async () => {
// F is loaded with one child c1; moving X in must preserve c1 (no loss) and
// append X — this path does NOT hit the guard.
const before = [
node("F", {
position: "a0",
hasChildren: true,
children: [node("c1", { position: "a1", parentPageId: "F" })],
}),
node("X", { position: "a5" }),
];
const { store, result } = setup(before);
await act(async () => {
await result.current.handleMove("X", {
kind: "make-child",
targetId: "F",
});
});
const tree = store.get(treeDataAtom);
const f = treeModel.find(tree, "F");
expect(f?.children?.map((n) => n.id)).toEqual(["c1", "X"]);
expect(f?.hasChildren).toBe(true);
// X now lives under F.
expect(treeModel.find(tree, "X")?.parentPageId).toBe("F");
});
it("make-child into a genuinely-empty leaf materializes the child (nothing to lose)", async () => {
// Leaf L has no server children (hasChildren:false). Dropping X in should
// show X immediately — the guard must NOT fire here.
const before = [
node("L", { position: "a0", hasChildren: false, children: [] }),
node("X", { position: "a5" }),
];
const { store, result } = setup(before);
await act(async () => {
await result.current.handleMove("X", {
kind: "make-child",
targetId: "L",
});
});
const tree = store.get(treeDataAtom);
const l = treeModel.find(tree, "L");
expect(l?.children?.map((n) => n.id)).toEqual(["X"]);
expect(l?.hasChildren).toBe(true);
});
});
@@ -61,11 +61,48 @@ export function useTreeMutation(spaceId: string): UseTreeMutation {
if (!source) return;
const oldParentId = source.parentPageId ?? null;
// optimistic apply with the new position from the payload
let optimistic = treeModel.update(after, sourceId, {
position: payload.position,
parentPageId: payload.parentPageId,
} as Partial<SpaceTreeNode>);
// Child-loss guard (#523, twin of #525's realtime `insertByPosition` fix).
// We no longer auto-expand the make-child target on drop, so the old
// `onToggle(target, true)` — which was ALSO the only trigger of the
// corrective lazy-load — is gone. `treeModel.move` materialized
// `target.children = [source]` (only the moved node); if the target is an
// UNLOADED branch (server has children but none are loaded here), keeping
// that partial `[source]` list would defeat the lazy-load gate and hide the
// target's OTHER server children (the #159 #1 data-loss class). So for an
// unloaded make-child target, build the optimistic tree WITHOUT
// materializing source under it: just remove source from its old parent and
// flag the target `hasChildren`. The gate stays armed and a later manual
// expand fetches the FULL set (incl. the moved page, which the awaited
// server move persists). Predicate is the gate's (`isUnloadedBranch`), NOT
// `insertByPosition`'s old `=== undefined` (canonical unloaded is `[]`).
const target =
op.kind === "make-child"
? (treeModel.find(before, op.targetId) as SpaceTreeNode | null)
: null;
const unloadedMakeChild =
op.kind === "make-child" && treeModel.isUnloadedBranch(target);
let optimistic: SpaceTreeNode[];
if (unloadedMakeChild) {
// Do NOT materialize [source] into the unloaded target.
optimistic = treeModel.remove(before, sourceId);
optimistic = treeModel.update(optimistic, op.targetId, {
hasChildren: true,
} as Partial<SpaceTreeNode>);
} else {
// optimistic apply with the new position from the payload
optimistic = treeModel.update(after, sourceId, {
position: payload.position,
parentPageId: payload.parentPageId,
} as Partial<SpaceTreeNode>);
// For make-child onto a previously-childless (loaded) target: flip
// hasChildren on so the new parent shows its chevron.
if (op.kind === "make-child") {
optimistic = treeModel.update(optimistic, op.targetId, {
hasChildren: true,
} as Partial<SpaceTreeNode>);
}
}
// If the old parent has no children left, mark hasChildren: false so the
// chevron disappears. Without this, the empty parent keeps rendering an
@@ -79,14 +116,6 @@ export function useTreeMutation(spaceId: string): UseTreeMutation {
}
}
// For make-child onto a previously-childless target: flip hasChildren on
// so the new parent shows its chevron.
if (op.kind === "make-child") {
optimistic = treeModel.update(optimistic, op.targetId, {
hasChildren: true,
} as Partial<SpaceTreeNode>);
}
setData(optimistic);
try {
@@ -150,6 +150,45 @@
);
}
/* "A page just moved in here" cue (#523). A make-child drop no longer expands
the collapsed target, so the moved page is momentarily invisible; pulse the
target row in a distinct teal (NOT the blue make-child highlight, NOT the
neutral post-move flash) so the landing is noticeable while the node stays
collapsed. Two short pulses fit inside DROP_LANDED_HIGHLIGHT_MS (1.8s), after
which the row clears the attribute and the animation stops. */
@keyframes landedChildPulse {
0% {
background-color: light-dark(
var(--mantine-color-teal-2),
rgba(45, 212, 191, 0.30)
);
outline-color: light-dark(
var(--mantine-color-teal-6),
var(--mantine-color-teal-5)
);
}
100% {
background-color: transparent;
outline-color: transparent;
}
}
.node[data-landed-child="true"] {
outline: 2px solid transparent;
outline-offset: -1px;
animation: landedChildPulse 0.9s ease-out 2;
}
@media (prefers-reduced-motion: reduce) {
.node[data-landed-child="true"] {
animation: none;
background-color: light-dark(
var(--mantine-color-teal-1),
rgba(45, 212, 191, 0.18)
);
}
}
.dropLine {
position: absolute;
left: var(--drop-line-indent, 0);
@@ -36,6 +36,7 @@ const STATIC_ROUTES = new Set<string>([
'/setup/register',
'/settings/account/profile',
'/settings/account/preferences',
'/settings/account/api-keys',
'/settings/workspace',
'/settings/ai',
'/settings/members',
@@ -0,0 +1,22 @@
import SettingsTitle from "@/components/settings/settings-title.tsx";
import ApiKeysManager from "@/features/api-key/components/api-keys-manager";
import { getAppName } from "@/lib/config.ts";
import { Helmet } from "react-helmet-async";
import { useTranslation } from "react-i18next";
export default function AccountApiKeys() {
const { t } = useTranslation();
return (
<>
<Helmet>
<title>
{t("API keys")} - {getAppName()}
</title>
</Helmet>
<SettingsTitle title={t("API keys")} />
<ApiKeysManager />
</>
);
}
+12
View File
@@ -63,3 +63,15 @@ vi.stubGlobal("matchMedia", (query: string) => ({
removeEventListener: vi.fn(),
dispatchEvent: vi.fn(),
}));
// Mantine's ScrollArea (used by Table.ScrollContainer, ScrollArea, etc.) reads
// `ResizeObserver` in a layout effect on mount, which jsdom does not implement.
// A no-op stub lets any test rendering those components mount cleanly.
vi.stubGlobal(
"ResizeObserver",
class {
observe() {}
unobserve() {}
disconnect() {}
},
);
@@ -32,6 +32,14 @@ describe('EmbeddingIndexerService.reindexWorkspace fail-fast', () => {
const pageEmbeddingRepo = {};
const aiService = {
getEmbeddingModel: jest.fn().mockResolvedValue('some-model'),
// #530: reindexWorkspace's pre-check now resolves the provider (workspace
// or global). Resolve it so the batch control flow under test proceeds.
resolveEmbeddingProvider: jest.fn().mockResolvedValue({
model: 'some-model',
queryPrefix: '',
docPrefix: '',
fingerprint: 'fp-test',
}),
};
// Progress is a best-effort cosmetic store; mock its async methods so the
// batch control flow can be tested without Redis.
@@ -108,6 +116,14 @@ describe('EmbeddingIndexerService.reindexWorkspace progress', () => {
const pageEmbeddingRepo = {};
const aiService = {
getEmbeddingModel: jest.fn().mockResolvedValue('some-model'),
// #530: reindexWorkspace's pre-check now resolves the provider (workspace
// or global). Resolve it so the batch control flow under test proceeds.
resolveEmbeddingProvider: jest.fn().mockResolvedValue({
model: 'some-model',
queryPrefix: '',
docPrefix: '',
fingerprint: 'fp-test',
}),
};
const reindexProgress = {
start: jest.fn().mockResolvedValue(undefined),
@@ -174,7 +190,7 @@ describe('EmbeddingIndexerService.reindexWorkspace progress', () => {
const { service, aiService, reindexProgress } = makeService();
// Embeddings not configured: reindexWorkspace returns early WITHOUT starting
// a fresh record, but the finally must still clear the enqueue-time seed.
aiService.getEmbeddingModel = jest
aiService.resolveEmbeddingProvider = jest
.fn()
.mockRejectedValue(new AiEmbeddingNotConfiguredException());
@@ -187,3 +203,87 @@ describe('EmbeddingIndexerService.reindexWorkspace progress', () => {
expect(reindexProgress.clear).toHaveBeenCalledWith(WORKSPACE_ID);
});
});
/**
* #530 PR-1: reindexPage must (a) prepend the provider's DOC prefix to each chunk
* BEFORE embedding (so stored vectors live in the same prefixed space as a
* prefixed query), and (b) stamp the ACTIVE fingerprint on every inserted row (so
* search only fuses same-generation vectors). Uses lightweight mocks; the tx is
* stubbed to run its callback inline.
*/
describe('EmbeddingIndexerService.reindexPage doc-prefix + fingerprint (#530)', () => {
const WORKSPACE_ID = 'ws-1';
const SPACE_ID = 'space-1';
const PAGE_ID = 'page-1';
function makeService(docPrefix: string) {
const pageRepo = {
findById: jest.fn().mockResolvedValue({
id: PAGE_ID,
workspaceId: WORKSPACE_ID,
spaceId: SPACE_ID,
title: 'Заголовок',
// No ProseMirror content -> the plain-text fallback path (single chunk).
content: null,
textContent: 'простой текст страницы',
deletedAt: null,
}),
};
const insertChunks = jest.fn().mockResolvedValue(undefined);
const pageEmbeddingRepo = {
deleteByPage: jest.fn().mockResolvedValue(undefined),
insertChunks,
};
const embedWithModel = jest.fn().mockResolvedValue([[0.1, 0.2, 0.3]]);
const aiService = {
resolveEmbeddingProvider: jest.fn().mockResolvedValue({
model: { modelId: 'e5-small' },
queryPrefix: 'query: ',
docPrefix,
fingerprint: 'fp-gen-1',
}),
embedWithModel,
};
const reindexProgress = {};
// Stub the tx so executeTx runs its callback inline against a fake trx.
const db = {
transaction: () => ({ execute: (cb: any) => cb({}) }),
};
const service = new EmbeddingIndexerService(
pageRepo as unknown as PageRepo,
pageEmbeddingRepo as unknown as PageEmbeddingRepo,
aiService as unknown as AiService,
reindexProgress as unknown as EmbeddingReindexProgressService,
db as unknown as KyselyDB,
);
return { service, embedWithModel, insertChunks };
}
it('prepends the doc prefix to each chunk and stamps the fingerprint on rows', async () => {
const { service, embedWithModel, insertChunks } = makeService('passage: ');
await service.reindexPage(PAGE_ID);
// Embedded values are DOC-prefixed; the model is the resolved provider model.
expect(embedWithModel).toHaveBeenCalledTimes(1);
const [modelArg, wsArg, valuesArg] = embedWithModel.mock.calls[0];
expect(modelArg).toEqual({ modelId: 'e5-small' });
expect(wsArg).toBe(WORKSPACE_ID);
expect(valuesArg).toEqual(['passage: простой текст страницы']);
// Inserted rows carry the active fingerprint and the ORIGINAL (un-prefixed)
// content (the prefix is an embedding-space artifact, not stored text).
expect(insertChunks).toHaveBeenCalledTimes(1);
const rows = insertChunks.mock.calls[0][0];
expect(rows).toHaveLength(1);
expect(rows[0].fingerprint).toBe('fp-gen-1');
expect(rows[0].content).toBe('простой текст страницы');
expect(rows[0].modelName).toBe('e5-small');
});
it('does not prefix when the provider has an empty doc prefix', async () => {
const { service, embedWithModel } = makeService('');
await service.reindexPage(PAGE_ID);
const [, , valuesArg] = embedWithModel.mock.calls[0];
expect(valuesArg).toEqual(['простой текст страницы']);
});
});
@@ -108,15 +108,24 @@ export class EmbeddingIndexerService {
return;
}
// Resolve embeddings config WITHOUT crashing the queue when unconfigured.
// Resolve the embeddings provider WITHOUT crashing the queue when
// unconfigured. #530: resolveEmbeddingProvider prefers the workspace provider
// and falls back to the GLOBAL env provider (TEI sidecar), and yields the
// doc-prefix + the active fingerprint stored per row so search filters by the
// active generation.
let modelName = 'unknown';
let provider: Awaited<
ReturnType<AiService['resolveEmbeddingProvider']>
>;
try {
const model = await this.aiService.getEmbeddingModel(workspaceId);
provider = await this.aiService.resolveEmbeddingProvider(workspaceId);
// Record the model id per row so a future migration can detect + re-index
// rows produced by a different model (see the migration header). The SDK
// type is `string | EmbeddingModel{V2,V3}`; model objects carry `modelId`.
modelName =
typeof model === 'string' ? model : (model.modelId ?? 'unknown');
typeof provider.model === 'string'
? provider.model
: (provider.model.modelId ?? 'unknown');
} catch (err) {
if (err instanceof AiEmbeddingNotConfiguredException) {
// No embeddings provider for this workspace: NO-OP (§6.7). The page can
@@ -145,8 +154,18 @@ export class EmbeddingIndexerService {
return;
}
// Embed all chunks in one batch.
const vectors = await this.aiService.embedTexts(workspaceId, chunks);
// #530: prepend the provider's DOC prefix to each chunk (e5-style
// "passage: "; empty for a non-e5 provider) so stored vectors live in the
// same prefixed space as a prefixed query, then embed with the RESOLVED
// provider model (which may be the global TEI sidecar, not a workspace one).
const prefixedChunks = provider.docPrefix
? chunks.map((c) => provider.docPrefix + c)
: chunks;
const vectors = await this.aiService.embedWithModel(
provider.model,
workspaceId,
prefixedChunks,
);
// The column is dimension-agnostic, so ANY model dimension is stored as-is.
// Defensive sanity check only: all chunks of ONE page come from the SAME
@@ -170,6 +189,7 @@ export class EmbeddingIndexerService {
vectors,
{ pageId, workspaceId, spaceId },
modelName,
provider.fingerprint,
);
// HARD replace in one transaction: delete then insert so search never
@@ -216,7 +236,9 @@ export class EmbeddingIndexerService {
// (seeded at enqueue time); the finally cleans that too.
try {
try {
await this.aiService.getEmbeddingModel(workspaceId);
// #530: resolve via the same path reindexPage uses (workspace provider,
// else the global TEI sidecar) so a global-only deployment is NOT skipped.
await this.aiService.resolveEmbeddingProvider(workspaceId);
} catch (err) {
if (err instanceof AiEmbeddingNotConfiguredException) {
this.logger.log(
@@ -348,6 +370,7 @@ export class EmbeddingIndexerService {
vectors: number[][],
ids: { pageId: string; workspaceId: string; spaceId: string },
modelName: string,
fingerprint: string | null,
): PageEmbeddingChunkRow[] {
const rows: PageEmbeddingChunkRow[] = [];
let cursor = 0;
@@ -370,6 +393,8 @@ export class EmbeddingIndexerService {
// Provenance for a future re-index sweep on model change.
modelName,
modelDimensions: embedding.length,
// #530: the active generation fingerprint this row belongs to.
fingerprint,
embedding,
});
}
@@ -32,10 +32,27 @@ export class SearchResultDto {
matchedTerms: string[];
}
// The paginated envelope (A5). `total` is the EXACT permission-filtered count of
// pages matching the positive lexical query (fail-closed). `hasMore` is true when
// more results exist WITHIN the fusion window; `truncatedAtCap` signals the match
// set exceeded CANDIDATE_CAP and the tail is unreachable by pagination.
// #530 Phase B — the semantic (vector) layer's per-request status.
// - `state`: 'full' when a provider resolved and the vector arm ran this
// request; 'off' otherwise (kill-switch, no provider, or a degrade).
// - `available`: whether the vector arm actually contributed this request.
// - `reason`: why the arm did NOT run — 'no-provider' (no embedding provider
// resolved) or 'degraded' (sidecar down / query embed timed out).
// - `indexed`/`total`: reserved for PR-2 coverage reporting (unused in PR-1).
export class SearchSemanticDto {
state: 'full' | 'off';
available: boolean;
reason?: 'no-provider' | 'degraded';
indexed?: number;
total?: number;
}
// The paginated envelope (A5). `total` is the permission-filtered count of the
// candidate UNION (lexical ∪ vector top-N, fail-closed) — a deliberate, #530
// documented change from Phase A's exact-lexical count (on a semantic degrade it
// falls back to exactly that lexical count). `hasMore` is true when more results
// exist WITHIN the fusion window; `truncatedAtCap` signals the match set exceeded
// CANDIDATE_CAP and the tail is unreachable by pagination.
export class SearchResponseDto {
items: SearchResultDto[];
total: number;
@@ -53,4 +70,7 @@ export class SearchResponseDto {
mode: 'or' | 'and';
match: string;
};
// Absent on the early-exit paths (empty/garbage query); present once search
// actually runs. Optional so those short-circuit responses stay valid.
semantic?: SearchSemanticDto;
}
@@ -1,8 +1,15 @@
import { Module } from '@nestjs/common';
import { SearchController } from './search.controller';
import { SearchService } from './search.service';
import { AiModule } from '../../integrations/ai/ai.module';
/**
* #530: AiModule supplies AiService (embedQuery / resolveEmbeddingProvider).
* PageEmbeddingRepo is provided by the @Global DatabaseModule, so it is injected
* into SearchService without an explicit import here.
*/
@Module({
imports: [AiModule],
controllers: [SearchController],
providers: [SearchService],
exports: [SearchService],
@@ -48,6 +48,10 @@ describe('SearchService.searchPage — scope-security early returns', () => {
shareRepo as any,
spaceMemberRepo as any,
pagePermissionRepo as any,
// #530 semantic path off for these query-mode unit tests: a stubbed
// embedQuery that throws makes the service degrade to the lexical path.
{ embedQuery: jest.fn().mockRejectedValue(new Error('no embed')) } as any,
{ vectorCandidateArm: jest.fn() } as any,
);
return { service, pageRepo, shareRepo, spaceMemberRepo, pagePermissionRepo };
}
@@ -11,6 +11,8 @@ describe('SearchService', () => {
{} as any, // shareRepo
{} as any, // spaceMemberRepo
{} as any, // pagePermissionRepo
{} as any, // aiService
{} as any, // pageEmbeddingRepo
);
expect(service).toBeDefined();
});
@@ -61,6 +63,8 @@ describe('SearchService.searchSuggestions — onlyTemplates filter', () => {
shareRepo as any,
spaceMemberRepo as any,
pagePermissionRepo as any,
{} as any, // aiService (searchSuggestions never embeds)
{} as any, // pageEmbeddingRepo
);
return { service, db, pageBuilder };
+247 -28
View File
@@ -1,8 +1,9 @@
import { Injectable } from '@nestjs/common';
import { Injectable, Logger } from '@nestjs/common';
import { SearchDTO, SearchSuggestionDTO } from './dto/search.dto';
import {
SearchResponseDto,
SearchResultDto,
SearchSemanticDto,
} from './dto/search-response.dto';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
@@ -11,6 +12,10 @@ import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { SpaceMemberRepo } from '@docmost/db/repos/space/space-member.repo';
import { ShareRepo } from '@docmost/db/repos/share/share.repo';
import { PagePermissionRepo } from '@docmost/db/repos/page/page-permission.repo';
import { PageEmbeddingRepo } from '@docmost/db/repos/ai-chat/page-embedding.repo';
import { AiService } from '../../integrations/ai/ai.service';
import { AiEmbeddingNotConfiguredException } from '../../integrations/ai/ai-embedding-not-configured.exception';
import { isStatementTimeout } from '@docmost/db/utils';
import {
ParsedQuery,
ParsedTerm,
@@ -59,14 +64,42 @@ function defaultBooleanMode(): SearchBooleanMode {
return process.env.SEARCH_MODE === 'and' ? 'and' : 'or';
}
// #530 semantic env knobs. SEARCH_SEMANTIC=off is the kill-switch (hybrid is
// otherwise transparent — no per-request mode flag). SEARCH_VECTOR_WEIGHT tunes
// the vector RRF leg's contribution (default 1.0, i.e. equal to each lexical
// leg). SEARCH_VECTOR_CANDIDATES caps the vector top-N pulled into the union.
function semanticKillSwitchOff(): boolean {
return process.env.SEARCH_SEMANTIC === 'off';
}
function getVectorWeight(): number {
const raw = Number(process.env.SEARCH_VECTOR_WEIGHT);
return Number.isFinite(raw) && raw >= 0 ? raw : 1.0;
}
function getVectorCandidates(): number {
const raw = Number(process.env.SEARCH_VECTOR_CANDIDATES);
return Number.isFinite(raw) && raw > 0 ? Math.floor(raw) : 50;
}
// Safety net for the brute-force vector scan (no ANN index — see
// vectorCandidateArm): a per-statement timeout bounding ONLY the fused vector
// query, so a pathological seq scan is cancelled (SQLSTATE 57014) and search
// degrades to lexical instead of hanging the request. Default 2000ms.
function getVectorStatementTimeoutMs(): number {
const raw = Number(process.env.SEARCH_VECTOR_STATEMENT_TIMEOUT_MS);
return Number.isFinite(raw) && raw > 0 ? Math.floor(raw) : 2000;
}
@Injectable()
export class SearchService {
private readonly logger = new Logger(SearchService.name);
constructor(
@InjectKysely() private readonly db: KyselyDB,
private pageRepo: PageRepo,
private shareRepo: ShareRepo,
private spaceMemberRepo: SpaceMemberRepo,
private pagePermissionRepo: PagePermissionRepo,
private aiService: AiService,
private pageEmbeddingRepo: PageEmbeddingRepo,
) {}
// === #529 SQL fragment builders (parameterized AST, never string-concat) =====
@@ -292,33 +325,102 @@ export class SearchService {
const scopeSql = sql`(${sql.join(scopePreds, sql` AND `)})`;
const candidateSql = sql`(${scopeSql} AND ${this.candidatePredicate(parsed, titleOnly)})`;
// --- Semantic (vector) arm: embed the query, degrade gracefully. (#530) ----
// The try/catch wraps ONLY embedQuery + vector-arm construction. On ANY throw
// (TEI down / 800ms timeout / no provider) we omit the vector arm entirely and
// run the byte-identical Phase-A lexical path — never a 500 from the sidecar.
// The permission filter below stays OUTSIDE this try (a permission error must
// 500, never fail-open). Hybrid is transparent; SEARCH_SEMANTIC=off disables it.
let vectorArm: RawBuilder<unknown> | null = null;
let semanticAvailable = false;
let semanticReason: 'no-provider' | 'degraded' | undefined;
if (!semanticKillSwitchOff()) {
try {
const { vector, fingerprint } = await this.aiService.embedQuery(
opts.workspaceId,
rawQuery,
);
vectorArm = this.pageEmbeddingRepo.vectorCandidateArm({
queryEmbedding: vector,
dimensions: vector.length,
fingerprint,
scope: scopeSql,
limit: getVectorCandidates(),
// Filter page_embeddings by (immutable) workspace_id directly, so the
// composite index bites on its leading column and candidates are
// workspace-scoped at the embedding level (#530 review WARNING 2). Space
// scoping stays on the pages join only — page_embeddings.space_id can be
// STALE after a cross-space move, so filtering it here would drop a moved
// page's vector hit (re-review regression fix).
workspaceId: scope.workspaceId,
});
semanticAvailable = true;
} catch (err) {
if (err instanceof AiEmbeddingNotConfiguredException) {
// No embedding provider is the DEFAULT state of any deployment without
// the TEI sidecar — normal, not a problem. Log at DEBUG so it never
// floods WARN once per search request.
semanticReason = 'no-provider';
this.logger.debug(
`search.semantic.no-provider workspace=${opts.workspaceId}`,
);
} else {
// A provider IS configured but the embed call failed/timed out — a real
// sidecar problem worth a WARN. Structured, greppable event (fixed code)
// — no query text/secrets.
semanticReason = 'degraded';
this.logger.warn(
`search.semantic.degraded reason=degraded workspace=${opts.workspaceId}`,
);
}
}
}
// --- Ranked candidate ids (ALL matches, RRF order). -----------------------
// Two independent branches ranked SEPARATELY (FTS by ts_rank_cd, substring by
// tier) then fused with RRF: score = Σ 1/(k + rank_branch). Deterministic
// ORDER BY rrf DESC, id so pagination never dupes/skips (A4, acceptance #8).
const rankedRows = await sql<{ id: string }>`
WITH candidates AS (
SELECT pages.id AS id,
${this.ftsScoreExpr(parsed)} AS fts_score,
${this.subTierExpr(parsed, titleOnly)} AS sub_tier
FROM pages
WHERE ${candidateSql}
),
ranked AS (
SELECT id, fts_score, sub_tier,
row_number() OVER (ORDER BY fts_score DESC NULLS LAST, id) AS rn_fts,
row_number() OVER (ORDER BY sub_tier DESC NULLS LAST, id) AS rn_sub
FROM candidates
)
SELECT id
FROM ranked
ORDER BY
(CASE WHEN fts_score IS NOT NULL THEN 1.0/(${RRF_K} + rn_fts) ELSE 0 END)
+ (CASE WHEN sub_tier IS NOT NULL THEN 1.0/(${RRF_K} + rn_sub) ELSE 0 END) DESC,
id ASC
`.execute(this.db);
// Lexical branches ranked SEPARATELY (FTS by ts_rank_cd, substring by tier);
// when a query vector is available a third VECTOR branch is fused over the
// UNION of lexical + vector candidates. RRF: score = Σ w/(k + rank_branch).
// Deterministic ORDER BY rrf DESC, id so pagination never dupes/skips (A4,
// acceptance #8). `total` is thus the union size (lexical ∪ vector top-N),
// post-permission-filter — a documented change from Phase A's exact lexical
// count (#530). On degrade the query is byte-identical to Phase A.
//
// The fused query is bounded by a per-statement timeout (see runRankedQuery):
// if the brute-force vector scan is cancelled (SQLSTATE 57014) we degrade to
// the byte-identical lexical path — search returns lexical results, never
// hangs, never 500s. Only the timeout is caught here; any OTHER SQL error
// propagates. The permission filter below stays outside, unchanged.
let orderedIds: string[];
try {
orderedIds = await this.runRankedQuery(
candidateSql,
parsed,
titleOnly,
vectorArm,
);
} catch (err) {
if (vectorArm && isStatementTimeout(err)) {
semanticAvailable = false;
semanticReason = 'degraded';
this.logger.warn(
`search.semantic.degraded reason=degraded workspace=${opts.workspaceId} (vector statement timeout)`,
);
// Re-run the exact Phase-A lexical path (no vector arm, no timeout).
orderedIds = await this.runRankedQuery(
candidateSql,
parsed,
titleOnly,
null,
);
} else {
throw err;
}
}
let orderedIds = rankedRows.rows.map((r) => r.id);
const semantic: SearchSemanticDto = {
state: semanticAvailable ? 'full' : 'off',
available: semanticAvailable,
...(semanticReason ? { reason: semanticReason } : {}),
};
// --- Permission filter (fail-closed, exact total). ------------------------
// filterAccessiblePageIds runs the #348 hasRestricted pre-check internally:
@@ -355,14 +457,131 @@ export class SearchService {
const hasMore = offset + pageIds.length < window.length;
if (pageIds.length === 0) {
return { items: [], total, hasMore, truncatedAtCap, offset, query: queryMeta };
return {
items: [],
total,
hasMore,
truncatedAtCap,
offset,
query: queryMeta,
semantic,
};
}
// --- Detail fetch for the page slice only (ts_headline/snippet are costly, so
// compute them ONLY for the returned rows), preserving RRF order. -------
const items = await this.fetchDetails(pageIds, parsed, titleOnly);
return { items, total, hasMore, truncatedAtCap, offset, query: queryMeta };
return {
items,
total,
hasMore,
truncatedAtCap,
offset,
query: queryMeta,
semantic,
};
}
/**
* Run the ranked-candidate-ids query and return the ids in RRF order (#530).
*
* When `vectorArm` is null (degrade / semantic off / no query vector) this runs
* the BYTE-IDENTICAL Phase-A 2-branch lexical query the semantic layer must
* be a pure superset that never changes lexical behaviour on degrade.
*
* When a vector arm is supplied it is UNION ALL'd with the lexical arm into one
* `candidates` set; `agg` collapses each page to its best per-branch signal
* (MAX fts_score / MAX sub_tier / MIN vec_distance); `ranked` assigns a
* per-branch row_number; the final ORDER BY fuses the three legs with RRF,
* adding the vector leg ONLY for rows that have a vec_distance (a lexical-only
* hit contributes 0 to the vector leg, and vice-versa). W_VEC weights the
* vector leg.
*/
private async runRankedQuery(
candidateSql: RawBuilder<unknown>,
parsed: ParsedQuery,
titleOnly: boolean,
vectorArm: RawBuilder<unknown> | null,
): Promise<string[]> {
if (!vectorArm) {
// Phase-A lexical path — DO NOT change (byte-identical on degrade, #529).
const rankedRows = await sql<{ id: string }>`
WITH candidates AS (
SELECT pages.id AS id,
${this.ftsScoreExpr(parsed)} AS fts_score,
${this.subTierExpr(parsed, titleOnly)} AS sub_tier
FROM pages
WHERE ${candidateSql}
),
ranked AS (
SELECT id, fts_score, sub_tier,
row_number() OVER (ORDER BY fts_score DESC NULLS LAST, id) AS rn_fts,
row_number() OVER (ORDER BY sub_tier DESC NULLS LAST, id) AS rn_sub
FROM candidates
)
SELECT id
FROM ranked
ORDER BY
(CASE WHEN fts_score IS NOT NULL THEN 1.0/(${RRF_K} + rn_fts) ELSE 0 END)
+ (CASE WHEN sub_tier IS NOT NULL THEN 1.0/(${RRF_K} + rn_sub) ELSE 0 END) DESC,
id ASC
`.execute(this.db);
return rankedRows.rows.map((r) => r.id);
}
// #530 fused 3-branch RRF over lexical ∪ vector candidates. The lexical arm
// adds NULL::float vec_distance to stay UNION-compatible with the vector arm.
const wVec = getVectorWeight();
// Bound the brute-force vector scan with a per-statement timeout, scoped to
// THIS query only. `set_config(..., is_local := true)` = SET LOCAL semantics:
// it applies within this transaction and auto-resets at COMMIT/ROLLBACK, so
// it can NEVER leak onto the next query that reuses this pooled connection.
// On a cancellation the driver raises SQLSTATE 57014, which searchPage catches
// to degrade to lexical-only. Running inside a transaction also guarantees the
// SET and the ranked query share one connection.
const timeoutMs = getVectorStatementTimeoutMs();
return this.db.transaction().execute(async (trx) => {
await sql`SELECT set_config('statement_timeout', ${String(timeoutMs)}, true)`.execute(
trx,
);
const rankedRows = await sql<{ id: string }>`
WITH candidates AS (
(SELECT pages.id AS id,
${this.ftsScoreExpr(parsed)} AS fts_score,
${this.subTierExpr(parsed, titleOnly)} AS sub_tier,
NULL::float AS vec_distance
FROM pages
WHERE ${candidateSql})
UNION ALL
(${vectorArm})
),
agg AS (
SELECT id,
MAX(fts_score) AS fts_score,
MAX(sub_tier) AS sub_tier,
MIN(vec_distance) AS vec_distance
FROM candidates
GROUP BY id
),
ranked AS (
SELECT id, fts_score, sub_tier, vec_distance,
row_number() OVER (ORDER BY fts_score DESC NULLS LAST, id) AS rn_fts,
row_number() OVER (ORDER BY sub_tier DESC NULLS LAST, id) AS rn_sub,
row_number() OVER (ORDER BY vec_distance ASC NULLS LAST, id) AS rn_vec
FROM agg
)
SELECT id
FROM ranked
ORDER BY
(CASE WHEN fts_score IS NOT NULL THEN 1.0/(${RRF_K} + rn_fts) ELSE 0 END)
+ (CASE WHEN sub_tier IS NOT NULL THEN 1.0/(${RRF_K} + rn_sub) ELSE 0 END)
+ (CASE WHEN vec_distance IS NOT NULL THEN ${wVec}::float/(${RRF_K} + rn_vec) ELSE 0 END) DESC,
id ASC
`.execute(trx);
return rankedRows.rows.map((r) => r.id);
});
}
// Resolve the search scope: explicit space, the authed user's member spaces, or
@@ -0,0 +1,49 @@
import { type Kysely, sql } from 'kysely';
/**
* #530 Search Phase B (PR-1): add the embedding FINGERPRINT column.
*
* The fingerprint is a deterministic id of the embedding configuration that
* produced a row model id + revision + query/doc prefix scheme + dimensions
* (see AiService.computeEmbeddingFingerprint). Search filters vector candidates
* by the workspace's ACTIVE fingerprint so a revision bump or a prefix-scheme
* change never fuses incompatible vectors into results.
*
* PR-1 scope: this migration only ADDS the column (nullable existing rows stay
* NULL = legacy) and the composite index the vector-candidate scan uses. The full
* generational swap / GC lifecycle (target-fingerprint reindex, atomic flip,
* old-generation GC) is deliberately deferred to PR-2.
*
* Independent migration (per the #363 crash-loop net): it creates ONLY its own
* objects and never touches another migration's tables/indexes, so a partial
* failure cannot leave a shared object half-built.
*/
export async function up(db: Kysely<any>): Promise<void> {
// Nullable text column. Existing rows keep NULL (legacy generation) and are
// simply not matched by the active-fingerprint filter until re-indexed.
await sql`
ALTER TABLE page_embeddings
ADD COLUMN IF NOT EXISTS fingerprint text
`.execute(db);
// Composite btree supporting the scoped, dimension- + fingerprint-filtered
// vector-candidate scan (workspace_id + space_id + fingerprint + model_dimensions).
await db.schema
.createIndex('idx_page_embeddings_ws_space_fp_dim')
.ifNotExists()
.on('page_embeddings')
.columns(['workspace_id', 'space_id', 'fingerprint', 'model_dimensions'])
.execute();
}
export async function down(db: Kysely<any>): Promise<void> {
await db.schema
.dropIndex('idx_page_embeddings_ws_space_fp_dim')
.ifExists()
.execute();
await sql`
ALTER TABLE page_embeddings
DROP COLUMN IF EXISTS fingerprint
`.execute(db);
}
@@ -1,6 +1,6 @@
import { Injectable } from '@nestjs/common';
import { InjectKysely } from 'nestjs-kysely';
import { sql } from 'kysely';
import { RawBuilder, sql } from 'kysely';
import * as pgvector from 'pgvector';
import { KyselyDB, KyselyTransaction } from '../../types/kysely.types';
import { dbOrTx } from '../../utils';
@@ -35,6 +35,9 @@ export interface PageEmbeddingChunkRow {
content: string;
modelName: string;
modelDimensions: number;
// #530 PR-1: the active embedding fingerprint (see computeEmbeddingFingerprint).
// null only when no provider resolves (the indexer no-ops in that case).
fingerprint: string | null;
embedding: number[];
}
@@ -120,6 +123,7 @@ export class PageEmbeddingRepo {
content: row.content,
modelName: row.modelName,
modelDimensions: row.modelDimensions,
fingerprint: row.fingerprint,
// pgvector.toSql -> '[1,2,3]'; cast the bound literal to vector.
embedding: sql`${pgvector.toSql(row.embedding)}::vector`,
})),
@@ -127,6 +131,82 @@ export class PageEmbeddingRepo {
.execute();
}
/**
* #530: build the VECTOR candidate arm for SearchService's fused RRF union a
* page-level nearest-neighbour sub-select. Returns a `sql` fragment (not an
* executed query) so the caller can UNION ALL it with the lexical arm inside a
* single ranked-ids query.
*
* SCALING BOUNDARY: the column is dimension-agnostic, so it carries NO ANN
* index this is a brute-force O(N) KNN seq scan with `<=>`. Accepted for the
* small-tenant / homelab fork target (ANN + a pinned-dimension column are
* deferred). It is NOT unbounded, though: the caller (SearchService) runs this
* fused query under a per-statement timeout (SEARCH_VECTOR_STATEMENT_TIMEOUT_MS)
* and DEGRADES to lexical-only on a 57014 cancellation, so a pathological scan
* can never hang the interactive search request.
*
* The arm collapses a page's chunks to its best (MIN) cosine distance:
* SELECT pages.id, NULL fts_score, NULL sub_tier,
* MIN(pe.embedding <=> $qvec) AS vec_distance
* FROM page_embeddings pe JOIN pages ON pages.id = pe.page_id
* WHERE <scope> AND pe.model_dimensions = $dim AND pe.fingerprint = $fp
* GROUP BY pages.id ORDER BY vec_distance LIMIT $limit
*
* `scope` is SearchService's shared scope predicate (workspace + space/id set +
* creator + descendants + deleted_at), referencing the `pages` table it must
* be spliced verbatim so the vector candidate set mirrors the lexical scope
* EXACTLY, minus the lexical text predicate (vector candidates need not match
* text). The vector is bound via pgvector's `toSql(...)::vector`, and both the
* dimension and the ACTIVE fingerprint are filtered so `<=>` only ever compares
* compatible, same-generation vectors (pgvector errors on a dimension
* mismatch; a fingerprint mismatch would fuse incomparable vectors).
*
* `workspaceId` is ALSO filtered directly on `page_embeddings` not only via
* the pages join. It is IMMUTABLE (there are no cross-workspace page moves), so
* an embedding row's workspace_id always matches its page's, and this drops NO
* legitimate hit; it lets the composite index idx_page_embeddings_ws_space_fp_dim
* (workspace_id, space_id, fingerprint, model_dimensions) bite from its LEADING
* column, and workspace-scopes candidates at the embedding level (defense in
* depth).
*
* SPACE is deliberately NOT filtered on page_embeddings: `page_embeddings.space_id`
* is stamped at index time and is NOT updated when a page is MOVED between spaces
* (movePageToSpace does not reindex, and PAGE_MOVED_TO_SPACE has no reindex
* consumer), so a moved page's rows carry the OLD space until the next reindex. A
* `page_embeddings.space_id = ANY(scope)` predicate would then wrongly drop a
* legitimate vector hit for a page moved INTO the searched space. Space scoping
* is therefore enforced ONLY by the join to `pages` (pages.space_id scope,
* always current) the same way the lexical arm scopes, so no space leak.
*
* The NULL fts_score / sub_tier columns keep the arm UNION-compatible with the
* lexical arm's column list.
*/
vectorCandidateArm(params: {
queryEmbedding: number[];
dimensions: number;
fingerprint: string;
scope: RawBuilder<unknown>;
limit: number;
workspaceId: string;
}): RawBuilder<unknown> {
const qvec = sql`${pgvector.toSql(params.queryEmbedding)}::vector`;
return sql`
SELECT pages.id AS id,
NULL::float AS fts_score,
NULL::int AS sub_tier,
MIN(page_embeddings.embedding <=> ${qvec}) AS vec_distance
FROM page_embeddings
JOIN pages ON pages.id = page_embeddings.page_id
WHERE ${params.scope}
AND page_embeddings.workspace_id = ${params.workspaceId}
AND page_embeddings.model_dimensions = ${params.dimensions}
AND page_embeddings.fingerprint = ${params.fingerprint}
GROUP BY pages.id
ORDER BY vec_distance ASC
LIMIT ${params.limit}
`;
}
/**
* Cosine search over the embeddings, scoped to a workspace AND a set of
* spaces the caller may read (see semanticSearch access-scoping). Orders by
@@ -7,6 +7,11 @@ export interface PageEmbeddings {
spaceId: string;
modelName: string;
modelDimensions: number;
// #530 PR-1: the active embedding fingerprint (model id + revision + prefix
// scheme + dimensions). NULL on legacy rows written before this column existed.
// Search filters candidates by the ACTIVE fingerprint so a revision/prefix
// change never mixes incompatible vectors.
fingerprint: string | null;
workspaceId: string;
// Nullable: page-body embeddings have no attachment (only attachment chunks set it).
attachmentId: string | null;
+17
View File
@@ -108,6 +108,23 @@ export function isUniqueViolation(err: unknown): boolean {
return (err as { code?: unknown } | null | undefined)?.code === PG_UNIQUE_VIOLATION;
}
/** Postgres `query_canceled` SQLSTATE — raised when statement_timeout fires. */
const PG_QUERY_CANCELED = '57014';
/**
* Whether `err` is a Postgres statement-timeout cancellation (SQLSTATE `57014`).
* Used by #530 semantic search to degrade a slow vector scan to lexical-only
* instead of hanging/500-ing. Matches on the SQLSTATE (the driver surfaces it as
* `.code`), with a message fallback for any wrapper that drops the code.
*/
export function isStatementTimeout(err: unknown): boolean {
if ((err as { code?: unknown } | null | undefined)?.code === PG_QUERY_CANCELED) {
return true;
}
const msg = err instanceof Error ? err.message : '';
return /canceling statement due to statement timeout/i.test(msg);
}
/**
* The name of the UNIQUE index/constraint a `23505` error violated, or
* undefined. The `kysely-postgres-js` / `postgres@3.x` driver surfaces it as
@@ -0,0 +1,143 @@
import { AiService, computeEmbeddingFingerprint } from './ai.service';
/**
* #530 Search Phase B (PR-1) unit coverage for the embedding layer:
* - computeEmbeddingFingerprint: a revision bump / prefix toggle MUST change the
* fingerprint even when the bare model name + dimension are unchanged (it is
* deliberately NOT the bare model name);
* - embedQuery: applies the QUERY prefix and embeds under the short SEARCH embed
* timeout (not the long batch timeout);
* - embedWithModel: a hung model + short timeout degrades with a clear timeout
* error (the search caller then falls back to the lexical path).
*/
describe('computeEmbeddingFingerprint (#530)', () => {
const base = {
modelId: 'intfloat/multilingual-e5-small',
revision: 'sha-aaa',
queryPrefix: 'query: ',
docPrefix: 'passage: ',
dimensions: 384,
};
it('is deterministic for identical inputs', () => {
expect(computeEmbeddingFingerprint(base)).toBe(
computeEmbeddingFingerprint({ ...base }),
);
});
it('CHANGES on a revision bump (same model name + dimension)', () => {
const a = computeEmbeddingFingerprint(base);
const b = computeEmbeddingFingerprint({ ...base, revision: 'sha-bbb' });
expect(b).not.toBe(a);
});
it('CHANGES on a query-prefix toggle (same model name + dimension + revision)', () => {
const a = computeEmbeddingFingerprint(base);
const b = computeEmbeddingFingerprint({ ...base, queryPrefix: '' });
expect(b).not.toBe(a);
});
it('CHANGES on a doc-prefix toggle', () => {
const a = computeEmbeddingFingerprint(base);
const b = computeEmbeddingFingerprint({ ...base, docPrefix: '' });
expect(b).not.toBe(a);
});
it('CHANGES on a dimension change', () => {
const a = computeEmbeddingFingerprint(base);
const b = computeEmbeddingFingerprint({ ...base, dimensions: 768 });
expect(b).not.toBe(a);
});
it('is NOT the bare model name (guards the mutation)', () => {
// If the fingerprint were the bare model name, a revision bump would not
// change it — the tests above would red. Assert it directly too.
expect(computeEmbeddingFingerprint(base)).not.toBe(base.modelId);
});
it('distinguishes swapped prefixes — ("q","") != ("","q")', () => {
const a = computeEmbeddingFingerprint({
...base,
queryPrefix: 'q',
docPrefix: '',
});
const b = computeEmbeddingFingerprint({
...base,
queryPrefix: '',
docPrefix: 'q',
});
expect(a).not.toBe(b);
});
});
describe('AiService.embedQuery (#530)', () => {
function makeService(): AiService {
// The constructor body only stores its deps; embedQuery is fully driven by
// the spied resolveEmbeddingProvider + embedWithModel below.
return new AiService({} as any, {} as any, {} as any);
}
it('prepends the query prefix and returns the provider fingerprint', async () => {
const svc = makeService();
jest.spyOn(svc, 'resolveEmbeddingProvider').mockResolvedValue({
model: { modelId: 'm' } as any,
queryPrefix: 'query: ',
docPrefix: 'passage: ',
fingerprint: 'fp-active',
});
const embedSpy = jest
.spyOn(svc, 'embedWithModel')
.mockResolvedValue([[0.1, 0.2, 0.3]]);
const res = await svc.embedQuery('ws-1', 'кофе');
// The single embedded value carries the query prefix.
expect(embedSpy).toHaveBeenCalledTimes(1);
const [, wsArg, valuesArg, timeoutArg] = embedSpy.mock.calls[0];
expect(wsArg).toBe('ws-1');
expect(valuesArg).toEqual(['query: кофе']);
// Bound by the SHORT search-embed timeout (default 800ms), not the 120000ms
// batch timeout.
expect(timeoutArg).toBe(800);
expect(res).toEqual({ vector: [0.1, 0.2, 0.3], fingerprint: 'fp-active' });
});
it('propagates a no-provider error (drives the no-provider degrade)', async () => {
const svc = makeService();
const { AiEmbeddingNotConfiguredException } = await import(
'./ai-embedding-not-configured.exception'
);
jest
.spyOn(svc, 'resolveEmbeddingProvider')
.mockRejectedValue(new AiEmbeddingNotConfiguredException());
await expect(svc.embedQuery('ws-1', 'x')).rejects.toBeInstanceOf(
AiEmbeddingNotConfiguredException,
);
});
});
describe('AiService.embedWithModel timeout (#530)', () => {
it('degrades a hung model with a timeout error under the short window', async () => {
const svc = new AiService({} as any, {} as any, {} as any);
// A model whose doEmbed never resolves but honours the abort signal, so the
// AbortSignal.timeout fires and embedWithModel raises a clear timeout error.
const hangingModel: any = {
specificationVersion: 'v2',
provider: 'fake',
modelId: 'fake',
maxEmbeddingsPerCall: 100,
supportsParallelCalls: true,
doEmbed: ({ abortSignal }: { abortSignal?: AbortSignal }) =>
new Promise((_resolve, reject) => {
abortSignal?.addEventListener('abort', () => {
const err = new Error('The operation was aborted');
err.name = 'AbortError';
reject(err);
});
}),
};
await expect(
svc.embedWithModel(hangingModel, 'ws-1', ['x'], 30),
).rejects.toThrow(/timed out/i);
});
});
+162 -7
View File
@@ -23,6 +23,50 @@ import {
import { AiProviderCredentialsRepo } from '@docmost/db/repos/ai-chat/ai-provider-credentials.repo';
import { SecretBoxService } from '../crypto/secret-box';
import { AiDriver } from './ai.types';
import { createHash } from 'node:crypto';
/**
* A resolved embedding provider for #530 semantic search. `model` is the AI SDK
* embedding model; `queryPrefix`/`docPrefix` are prepended to a query / a stored
* chunk respectively (e5-style `"query: "` / `"passage: "`, empty for a non-e5
* provider); `fingerprint` is the deterministic id of the whole configuration.
*/
export interface ResolvedEmbeddingProvider {
model: EmbeddingModel;
queryPrefix: string;
docPrefix: string;
fingerprint: string;
}
/**
* Deterministic embedding FINGERPRINT (#530). Encodes model id + revision +
* prefix scheme + dimensions so that ANY of them changing (a revision bump, a
* prefix toggle) yields a DIFFERENT fingerprint even when the bare model name and
* dimension are unchanged. Deliberately NOT the bare model name: two rows from
* the same model but a different revision/prefix must not be fused together.
*
* Exported as a pure function so it can be unit-tested in isolation and reused by
* the indexer without a service instance.
*/
export function computeEmbeddingFingerprint(parts: {
modelId: string;
revision: string;
queryPrefix: string;
docPrefix: string;
dimensions: number | null;
}): string {
// The two prefixes are SEPARATE keys (not a concatenated string): JSON.stringify
// escapes each independently, so the prefix scheme ("a","") is distinct from
// ("","a") with no separator/collision hazard even if a prefix contains spaces.
const canonical = JSON.stringify({
m: parts.modelId,
r: parts.revision,
q: parts.queryPrefix,
d: parts.docPrefix,
dim: parts.dimensions ?? 0,
});
return createHash('sha256').update(canonical).digest('hex').slice(0, 32);
}
/**
* Optional chat-model override carried by an agent role (`ai_agent_roles.
@@ -362,11 +406,112 @@ export class AiService {
async embedTexts(workspaceId: string, texts: string[]): Promise<number[][]> {
if (texts.length === 0) return [];
const model = await this.getEmbeddingModel(workspaceId);
// Bound the embedding call: a slow/hung embeddings endpoint must fail loudly
// (and let the caller move on to the next page) instead of blocking forever.
// The single signal caps the WHOLE call, including the SDK's internal
// retries/backoff (embedMany defaults to maxRetries: 2).
const timeoutMs = AiService.embeddingTimeoutMs();
return this.embedWithModel(model, workspaceId, texts);
}
/**
* #530: resolve the embedding provider for a workspace. Prefers the workspace's
* own configured embedding provider; falls back to the GLOBAL env provider (a
* TEI sidecar via the OpenAI-compatible path) when the workspace has none.
* Returns the model + the query/doc prefixes + the config fingerprint. Throws
* AiEmbeddingNotConfiguredException when NEITHER resolves, so callers can drive
* a `no-provider` degrade path.
*/
async resolveEmbeddingProvider(
workspaceId: string,
): Promise<ResolvedEmbeddingProvider> {
// 1. Per-workspace provider (uses the workspace's own creds/endpoint). When
// it is not configured getEmbeddingModel throws the not-configured
// exception; we swallow ONLY that and fall through to the global provider.
try {
const model = await this.getEmbeddingModel(workspaceId);
const modelId =
typeof model === 'string' ? model : (model.modelId ?? 'unknown');
// A per-workspace (typically non-e5) provider gets no e5-style prefixes.
return {
model,
queryPrefix: '',
docPrefix: '',
fingerprint: computeEmbeddingFingerprint({
modelId,
revision: 'workspace',
queryPrefix: '',
docPrefix: '',
dimensions: null,
}),
};
} catch (err) {
if (!(err instanceof AiEmbeddingNotConfiguredException)) throw err;
}
// 2. Global env provider (TEI sidecar). TEI is OpenAI-compatible, so reuse
// the existing openai path — no new SDK. A dummy key is fine for a
// keyless self-hosted sidecar.
const endpoint = process.env.EMBEDDING_ENDPOINT?.trim();
const globalModel = process.env.EMBEDDING_MODEL?.trim();
if (endpoint && globalModel) {
const model = createOpenAI({
baseURL: endpoint,
apiKey: process.env.EMBEDDING_API_KEY || 'unused',
}).textEmbeddingModel(globalModel);
const queryPrefix = process.env.EMBEDDING_QUERY_PREFIX ?? '';
const docPrefix = process.env.EMBEDDING_DOC_PREFIX ?? '';
const dimRaw = Number(process.env.EMBEDDING_DIMENSIONS);
const dimensions = Number.isFinite(dimRaw) && dimRaw > 0 ? dimRaw : null;
return {
model,
queryPrefix,
docPrefix,
fingerprint: computeEmbeddingFingerprint({
modelId: globalModel,
revision: process.env.EMBEDDING_REVISION ?? '',
queryPrefix,
docPrefix,
dimensions,
}),
};
}
// Neither resolved -> drives semantic.reason=no-provider.
throw new AiEmbeddingNotConfiguredException();
}
/**
* #530: embed a SEARCH QUERY. Resolves the provider, prepends its query prefix,
* and embeds the single value under its OWN short timeout
* (SEARCH_EMBED_TIMEOUT_MS, default 800ms) NOT the long batch-indexing
* timeout so a slow/hung sidecar degrades search fast. Throws on
* timeout/error (the caller degrades to the lexical-only path). Returns the
* vector plus the active fingerprint used to filter candidate rows.
*/
async embedQuery(
workspaceId: string,
text: string,
): Promise<{ vector: number[]; fingerprint: string }> {
const provider = await this.resolveEmbeddingProvider(workspaceId);
const [vector] = await this.embedWithModel(
provider.model,
workspaceId,
[provider.queryPrefix + text],
AiService.searchEmbedTimeoutMs(),
);
return { vector, fingerprint: provider.fingerprint };
}
/**
* Embed values with an EXPLICIT model, bounded by `timeoutMs` (default: the
* batch-indexing timeout). Shared core of embedTexts / embedQuery: a slow/hung
* embeddings endpoint must fail loudly instead of blocking forever. The single
* signal caps the WHOLE call, including the SDK's internal retries/backoff
* (embedMany defaults to maxRetries: 2).
*/
async embedWithModel(
model: EmbeddingModel,
workspaceId: string,
texts: string[],
timeoutMs: number = AiService.embeddingTimeoutMs(),
): Promise<number[][]> {
if (texts.length === 0) return [];
const signal = AbortSignal.timeout(timeoutMs);
try {
const { embeddings } = await embedMany({
@@ -391,8 +536,8 @@ export class AiService {
if (signal.aborted && abortLike) {
throw new Error(
`Embedding request timed out after ${timeoutMs}ms ` +
`(workspace ${workspaceId}, ${texts.length} chunk(s)). ` +
`Increase AI_EMBEDDING_TIMEOUT_MS or check the embeddings endpoint.`,
`(workspace ${workspaceId}, ${texts.length} value(s)). ` +
`Increase the embedding timeout or check the embeddings endpoint.`,
);
}
throw err;
@@ -408,6 +553,16 @@ export class AiService {
return Number.isFinite(raw) && raw > 0 ? raw : 120_000;
}
/**
* #530: per-call timeout for the interactive SEARCH query embed. Much shorter
* than the batch-indexing timeout (a search request cannot wait 2 minutes on
* the sidecar). Configurable via SEARCH_EMBED_TIMEOUT_MS; default 800ms.
*/
private static searchEmbedTimeoutMs(): number {
const raw = Number(process.env.SEARCH_EMBED_TIMEOUT_MS);
return Number.isFinite(raw) && raw > 0 ? raw : 800;
}
// Build a tiny valid WAV (mono, 16-bit PCM, 16 kHz, ~1s of silence), used only
// as a connectivity probe for the STT endpoint in testConnection.
private static silentWavProbe(): Uint8Array {
@@ -1,7 +1,10 @@
import { randomUUID } from 'node:crypto';
import { Kysely, sql } from 'kysely';
import { Logger } from '@nestjs/common';
import { SearchService } from 'src/core/search/search.service';
import { PageRepo } from '@docmost/db/repos/page/page.repo';
import { PageEmbeddingRepo } from '@docmost/db/repos/ai-chat/page-embedding.repo';
import { AiEmbeddingNotConfiguredException } from 'src/integrations/ai/ai-embedding-not-configured.exception';
import {
getTestDb,
destroyTestDb,
@@ -53,10 +56,19 @@ describe('SearchService #529 lexical overhaul [integration]', () => {
// Service wired to the real DB + real PageRepo (recursive descendants) with
// stubbed space-membership + permission repos so a test controls scope and the
// permission filter explicitly. `accessibleIds` (when set) is the KEEP list.
// #530: a FAKE AiService whose embedQuery is fully controlled per test. By
// DEFAULT it throws AiEmbeddingNotConfiguredException, so the semantic arm is
// omitted and every Phase-A case runs the BYTE-IDENTICAL lexical path — the
// semantic layer must never change lexical behaviour on degrade. Semantic cases
// pass `embedVector` (a deterministic 384-dim query vector) or `embedThrows`.
function buildService(opts?: {
userSpaceIds?: string[];
accessibleIds?: string[] | null;
filterThrows?: boolean;
embedVector?: number[];
embedFingerprint?: string;
embedThrows?: 'no-provider' | 'degraded';
slowVectorArm?: boolean;
}): SearchService {
const pageRepo = new PageRepo(db as any, null as any, null as any);
const spaceMemberRepo = {
@@ -71,15 +83,86 @@ describe('SearchService #529 lexical overhaul [integration]', () => {
: pageIds;
},
};
const aiService = {
embedQuery: async () => {
if (opts?.embedVector) {
return {
vector: opts.embedVector,
fingerprint: opts.embedFingerprint ?? ACTIVE_FP,
};
}
if (opts?.embedThrows === 'degraded') {
const err = new Error('embedding request timed out after 1ms');
err.name = 'TimeoutError';
throw err;
}
// Default (and explicit 'no-provider'): no embedding provider resolved.
throw new AiEmbeddingNotConfiguredException();
},
};
// Real repo on the migrated test DB — exercises the actual vector-candidate
// SQL (pgvector <=>, fingerprint + dimension filters). When slowVectorArm is
// set, wrap the arm so it sleeps well past the (tiny, test-set) statement
// timeout, forcing a 57014 cancellation to exercise the degrade-to-lexical net.
const realRepo = new PageEmbeddingRepo(db as any);
const pageEmbeddingRepo = opts?.slowVectorArm
? {
vectorCandidateArm: (p: any) => {
const inner = realRepo.vectorCandidateArm(p);
return sql`SELECT * FROM (${inner}) AS slowarm WHERE (SELECT true FROM pg_sleep(0.5))`;
},
}
: realRepo;
return new SearchService(
db as any,
pageRepo as any,
{} as any,
spaceMemberRepo as any,
pagePermissionRepo as any,
aiService as any,
pageEmbeddingRepo as any,
);
}
// Active embedding fingerprint used by planted rows + the fake query embed.
const ACTIVE_FP = 'fp-active-530';
// A deterministic unit-ish 384-dim vector with a single 1 at `concept`. Two
// vectors of the same concept have cosine distance 0; different concepts are
// orthogonal (distance 1), so nearest-neighbour ordering is fully controlled.
function conceptVec(concept: number): number[] {
const v = new Array(384).fill(0);
v[concept % 384] = 1;
return v;
}
// Plant one chunk embedding for a page via the REAL repo (exercises the #530
// fingerprint insert path). Dimension is fixed at 384 to match conceptVec.
async function plantEmbedding(
pageId: string,
vector: number[],
fingerprint: string = ACTIVE_FP,
planSpaceId: string = spaceId,
): Promise<void> {
const repo = new PageEmbeddingRepo(db as any);
await repo.insertChunks([
{
pageId,
workspaceId,
spaceId: planSpaceId,
attachmentId: null,
chunkIndex: 0,
chunkStart: 0,
chunkLength: 1,
content: 'planted chunk',
modelName: 'test-model',
modelDimensions: 384,
fingerprint,
embedding: vector,
},
]);
}
const search = (service: SearchService, params: any) =>
service.searchPage(params, { userId: 'u-1', workspaceId }) as any;
@@ -609,4 +692,324 @@ describe('SearchService #529 lexical overhaul [integration]', () => {
expect(hit2).toBeDefined();
expect(hit2.path).toEqual([]);
});
// === #530 Phase B (PR-1): semantic fusion ==================================
// Deterministic fake embedder (no live model) + real PageEmbeddingRepo on the
// migrated DB with planted vectors + the active fingerprint, so cosine ordering
// is fully controlled. Each case runs in its OWN space so planted rows never
// leak across tests.
describe('#530 semantic fusion', () => {
// 1. Vector-only hit: nearest to a page that has NO lexical match for the
// query term — it appears via the vector arm, and a pure-lexical run does
// not return it. Mutation (a): drop the vector arm -> this reddens.
it('#530-1 a vector-only hit appears; a pure-lexical run would not return it', async () => {
const s = (await createSpace(db, workspaceId)).id;
const page = await insertPage({
title: 'нейтральный вект заголовок',
textContent: 'содержимое без искомого термина',
spaceId: s,
});
await plantEmbedding(page, conceptVec(1), ACTIVE_FP, s);
const svc = buildService({ embedVector: conceptVec(1), userSpaceIds: [s] });
const res = await search(svc, { query: 'квазонимикс', spaceId: s });
expect(res.items.map((i: any) => i.id)).toContain(page);
expect(res.semantic.available).toBe(true);
expect(res.semantic.state).toBe('full');
// Pure-lexical (default degrade): the vector-only page is absent.
const lex = buildService({ userSpaceIds: [s] });
const res2 = await search(lex, { query: 'квазонимикс', spaceId: s });
expect(res2.items.map((i: any) => i.id)).not.toContain(page);
expect(res2.semantic.available).toBe(false);
});
// 2. Sidecar down: embedQuery throws -> results = the lexical set; semantic
// unavailable; the fixed 'search.semantic.degraded' event is logged.
it('#530-2 sidecar-down degrades to lexical and logs search.semantic.degraded', async () => {
const s = (await createSpace(db, workspaceId)).id;
const lexHit = await insertPage({
title: 'семдвамаркер лексический',
textContent: 'обычный текст',
spaceId: s,
});
const warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as any);
try {
const svc = buildService({ embedThrows: 'degraded', userSpaceIds: [s] });
const res = await search(svc, { query: 'семдвамаркер', spaceId: s });
expect(res.items.map((i: any) => i.id)).toContain(lexHit);
expect(res.semantic.available).toBe(false);
expect(res.semantic.reason).toBe('degraded');
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining('search.semantic.degraded'),
);
} finally {
warnSpy.mockRestore();
}
});
// 3. No provider: resolveEmbeddingProvider yields none -> lexical results,
// reason no-provider, no 500.
it('#530-3 no-provider yields lexical results with reason no-provider', async () => {
const s = (await createSpace(db, workspaceId)).id;
const hit = await insertPage({
title: 'семтримаркер тема',
textContent: 'текст',
spaceId: s,
});
const svc = buildService({ embedThrows: 'no-provider', userSpaceIds: [s] });
const res = await search(svc, { query: 'семтримаркер', spaceId: s });
expect(res.items.map((i: any) => i.id)).toContain(hit);
expect(res.semantic.available).toBe(false);
expect(res.semantic.reason).toBe('no-provider');
});
// 4. Permission over the union: a restricted vector-only hit is dropped from
// items AND total; with the permission filter throwing the call rejects
// (fail-closed, never fail-open). Mutation (b): move filterAccessiblePageIds
// inside the semantic try/catch -> the reject assertion reddens.
it('#530-4 permission filters a vector-only hit from items AND total (fail-closed)', async () => {
const s = (await createSpace(db, workspaceId)).id;
const visibleLex = await insertPage({
title: 'семчетмаркер видимый',
textContent: 'доступный текст',
spaceId: s,
});
const hiddenVec = await insertPage({
title: 'скрытая вект страница',
textContent: 'содержимое без искомого термина',
spaceId: s,
});
await plantEmbedding(hiddenVec, conceptVec(4), ACTIVE_FP, s);
// Query hits visibleLex lexically; hiddenVec only via the vector arm.
const svc = buildService({
embedVector: conceptVec(4),
accessibleIds: [visibleLex],
userSpaceIds: [s],
});
const res = await search(svc, { query: 'семчетмаркер', spaceId: s });
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(visibleLex);
expect(ids).not.toContain(hiddenVec);
// The hidden vector hit does not leak into total either.
expect(res.total).toBe(1);
expect(ids).toHaveLength(res.total);
// A permission-query error PROPAGATES (never a fail-open empty result).
const boom = buildService({
embedVector: conceptVec(4),
filterThrows: true,
userSpaceIds: [s],
});
await expect(
search(boom, { query: 'семчетмаркер', spaceId: s }),
).rejects.toThrow(/permission query failed/);
});
// 5. Hung sidecar under a short embed timeout: graceful degrade (never a 500).
it('#530-5 a hung sidecar under a short timeout degrades gracefully', async () => {
const s = (await createSpace(db, workspaceId)).id;
const hit = await insertPage({
title: 'семпятьмаркер тема',
textContent: 'текст',
spaceId: s,
});
process.env.SEARCH_EMBED_TIMEOUT_MS = '1';
try {
const svc = buildService({ embedThrows: 'degraded', userSpaceIds: [s] });
const res = await search(svc, { query: 'семпятьмаркер', spaceId: s });
expect(res.items.map((i: any) => i.id)).toContain(hit);
expect(res.semantic.available).toBe(false);
expect(res.semantic.reason).toBe('degraded');
} finally {
delete process.env.SEARCH_EMBED_TIMEOUT_MS;
}
});
// 6. A page matched BOTH lexically and by vector is fused (de-duped), not
// returned twice — the agg CTE collapses it to one row.
it('#530-6 a page matched lexically AND by vector appears once', async () => {
const s = (await createSpace(db, workspaceId)).id;
const both = await insertPage({
title: 'семшестьмаркер общий',
textContent: 'и лексика и вектор',
spaceId: s,
});
await plantEmbedding(both, conceptVec(6), ACTIVE_FP, s);
const svc = buildService({ embedVector: conceptVec(6), userSpaceIds: [s] });
const res = await search(svc, { query: 'семшестьмаркер', spaceId: s });
const ids = res.items.map((i: any) => i.id);
expect(ids.filter((id: string) => id === both)).toHaveLength(1);
expect(res.total).toBe(1);
});
// 7. Fingerprint isolation: a planted row under a DIFFERENT fingerprint is not
// a vector candidate for the active-fingerprint query (guards mixing
// generations). It only appears if it also matches lexically (it does not).
it('#530-7 a stale-fingerprint vector row is not fused into results', async () => {
const s = (await createSpace(db, workspaceId)).id;
const stale = await insertPage({
title: 'нейтральный семь заголовок',
textContent: 'без искомого термина совсем',
spaceId: s,
});
// Same concept vector as the query, but an OLD generation fingerprint.
await plantEmbedding(stale, conceptVec(7), 'fp-stale-old', s);
const svc = buildService({ embedVector: conceptVec(7), userSpaceIds: [s] });
const res = await search(svc, { query: 'квазонимиксseven', spaceId: s });
expect(res.items.map((i: any) => i.id)).not.toContain(stale);
});
// 8. Stale embedding space_id (moved page): a page currently in space B whose
// embedding rows still carry space A (movePageToSpace does NOT reindex) is
// STILL a vector hit when searching B — space scoping is enforced only by
// the pages join (pages.space_id, always current), NOT by a stale
// page_embeddings.space_id. Guards against re-adding that predicate (which
// would wrongly drop the moved page's vector-only hit).
it('#530-8 a vector hit with a STALE embedding space_id (moved page) is still returned', async () => {
const newSpace = (await createSpace(db, workspaceId)).id;
const oldSpace = (await createSpace(db, workspaceId)).id;
// The page currently lives in newSpace (as after a move into newSpace).
const page = await insertPage({
title: 'перемещённая вект страница',
textContent: 'содержимое без искомого термина',
spaceId: newSpace,
});
// Its embedding was stamped with the OLD space and never reindexed.
await plantEmbedding(page, conceptVec(8), ACTIVE_FP, oldSpace);
const svc = buildService({
embedVector: conceptVec(8),
userSpaceIds: [newSpace],
});
const res = await search(svc, {
query: 'квазонимиксeight',
spaceId: newSpace,
});
// Survives despite page_embeddings.space_id (oldSpace) != pages.space_id.
expect(res.items.map((i: any) => i.id)).toContain(page);
expect(res.semantic.available).toBe(true);
});
// 9. Statement-timeout safety net: a pathologically slow vector scan is
// cancelled (SQLSTATE 57014) and search degrades to lexical-only — the
// lexical hit survives, the vector-only hit vanishes, no 500/hang, and the
// degraded event is logged. Mutation: make the fallback re-throw instead of
// degrading -> this test reds (the call rejects with 57014).
it('#530-9 a vector-query timeout degrades to lexical (57014), never 500', async () => {
const s = (await createSpace(db, workspaceId)).id;
const lexHit = await insertPage({
title: 'семдевятьмаркер лексический',
textContent: 'обычный текст',
spaceId: s,
});
const vecOnly = await insertPage({
title: 'нейтральный девять заголовок',
textContent: 'без искомого термина',
spaceId: s,
});
await plantEmbedding(vecOnly, conceptVec(9), ACTIVE_FP, s);
const warnSpy = jest
.spyOn(Logger.prototype, 'warn')
.mockImplementation(() => undefined as any);
// Tiny per-statement timeout so the 0.5s sleeping arm is cancelled.
process.env.SEARCH_VECTOR_STATEMENT_TIMEOUT_MS = '50';
try {
const svc = buildService({
embedVector: conceptVec(9),
slowVectorArm: true,
userSpaceIds: [s],
});
const res = await search(svc, { query: 'семдевятьмаркер', spaceId: s });
// Lexical result survives; the vector-only page does NOT (arm cancelled).
const ids = res.items.map((i: any) => i.id);
expect(ids).toContain(lexHit);
expect(ids).not.toContain(vecOnly);
expect(res.semantic.available).toBe(false);
expect(res.semantic.reason).toBe('degraded');
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining('search.semantic.degraded'),
);
} finally {
delete process.env.SEARCH_VECTOR_STATEMENT_TIMEOUT_MS;
warnSpy.mockRestore();
}
});
// 10. SET LOCAL scoping: the search's statement timeout must NOT leak onto a
// later query on the same pooled connection. After a search that set a
// tiny timeout, a deliberately-slow standalone query still completes.
it('#530-10 the per-query statement timeout does not leak to later queries', async () => {
const s = (await createSpace(db, workspaceId)).id;
await insertPage({
title: 'семдесятьмаркер тема',
textContent: 'текст',
spaceId: s,
});
process.env.SEARCH_VECTOR_STATEMENT_TIMEOUT_MS = '50';
try {
const svc = buildService({
embedVector: conceptVec(10),
slowVectorArm: true,
userSpaceIds: [s],
});
// This search trips + resets the local timeout (via SET LOCAL semantics).
await search(svc, { query: 'семдесятьмаркер', spaceId: s });
// A subsequent query that sleeps 200ms must NOT be cancelled — proving the
// 50ms search timeout did not persist on the connection (it would raise
// 57014 if it had leaked). SET LOCAL auto-resets at the search tx end.
await expect(sql`SELECT pg_sleep(0.2)`.execute(db)).resolves.toBeDefined();
} finally {
delete process.env.SEARCH_VECTOR_STATEMENT_TIMEOUT_MS;
}
});
// 11. COMMIT-path no-leak (locks is_local=true). #530-10 only covers the
// ROLLBACK path (a timed-out search rolls back — a plain session SET would
// ALSO be undone on rollback, so it would NOT catch is_local true->false).
// Here a NORMAL, FAST semantic search COMMITs its transaction; a leaked
// session-level statement_timeout would then persist on the pooled
// connection and cancel later queries. We prove it does NOT: after the
// committing search (bounded at a small 100ms), a 300ms query succeeds.
// Because postgres.js pools, we run several sequential slow queries so at
// least one reuses the connection the committed search ran on (which is
// where a leaked timeout would live); ALL must succeed. Mutation: flip
// is_local true->false -> the committed 100ms timeout leaks and the
// follow-up pg_sleep is cancelled (57014), reddening this test.
it('#530-11 a COMMITTED search does not leak its statement timeout (is_local)', async () => {
const s = (await createSpace(db, workspaceId)).id;
const page = await insertPage({
title: 'семодиннадцать вект страница',
textContent: 'содержимое без искомого термина',
spaceId: s,
});
await plantEmbedding(page, conceptVec(11), ACTIVE_FP, s);
// Small bound: the fast vector arm still completes (<100ms) so the search
// COMMITs, but small enough that a LEAKED timeout would cancel the 300ms
// follow-ups below.
process.env.SEARCH_VECTOR_STATEMENT_TIMEOUT_MS = '100';
try {
const svc = buildService({
embedVector: conceptVec(11),
userSpaceIds: [s],
});
const res = await search(svc, { query: 'семодиннадцать', spaceId: s });
// The (committing) vector path actually ran.
expect(res.semantic.available).toBe(true);
// Each 300ms query must SUCCEED — a leaked 100ms timeout would cancel the
// one that reuses the committed search's connection. 8 sweeps cover the
// whole pool (max 5 conns), so a leak on any connection is caught.
for (let i = 0; i < 8; i++) {
await expect(
sql`SELECT pg_sleep(0.3)`.execute(db),
).resolves.toBeDefined();
}
} finally {
delete process.env.SEARCH_VECTOR_STATEMENT_TIMEOUT_MS;
}
});
});
});
+41
View File
@@ -4,11 +4,26 @@ services:
depends_on:
- db
- redis
- embeddings
environment:
APP_URL: 'http://localhost:3000'
APP_SECRET: 'REPLACE_WITH_LONG_SECRET'
DATABASE_URL: 'postgresql://docmost:STRONG_DB_PASSWORD@db:5432/docmost'
REDIS_URL: 'redis://redis:6379'
# #530 semantic search: the GLOBAL embedding provider (the `embeddings` TEI
# sidecar below). A workspace that configures its own embedding provider
# overrides this. TEI is OpenAI-compatible, so the endpoint is /v1.
EMBEDDING_ENDPOINT: 'http://embeddings:80/v1'
EMBEDDING_MODEL: 'intfloat/multilingual-e5-small'
EMBEDDING_API_KEY: 'unused'
EMBEDDING_DIMENSIONS: '384'
# MUST match the --revision the sidecar pins (see EMBEDDING_REVISION in
# .env). A revision bump changes the embedding fingerprint (PR-2 swaps
# generations); keep the two in lockstep.
EMBEDDING_REVISION: '${EMBEDDING_REVISION}'
# e5 models require these input prefixes; empty them for a non-e5 model.
EMBEDDING_QUERY_PREFIX: 'query: '
EMBEDDING_DOC_PREFIX: 'passage: '
ports:
- "3000:3000"
restart: unless-stopped
@@ -39,7 +54,33 @@ services:
volumes:
- redis_data:/data
# #530 Text Embeddings Inference (TEI) sidecar — the GLOBAL embedding provider
# for semantic search. OpenAI-compatible, reached only on the internal network
# (no published port). The model + revision are PINNED so the embedding
# fingerprint is stable; set EMBEDDING_REVISION to a real commit sha in .env.
embeddings:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9
command:
- "--model-id"
- "intfloat/multilingual-e5-small"
- "--revision"
- "${EMBEDDING_REVISION}"
restart: unless-stopped
# Cache downloaded model weights so a restart does not re-download them.
volumes:
- tei-models:/data
# GET /health is TEI's readiness probe. (Drop this block if your TEI image
# variant ships without curl.)
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:80/health || exit 1"]
interval: 30s
timeout: 5s
retries: 5
# Model download on first boot can be slow; don't flap as unhealthy meanwhile.
start_period: 120s
volumes:
docmost:
db_data:
redis_data:
tei-models: