Commit Graph

53 Commits

Author SHA1 Message Date
claude code agent 227
daf6c9ea16 fix(git-sync): propagate remote custom-event handler errors instead of 30s timeout
When a git-sync body write (gitSyncWriteBody) is routed to the collab instance
that owns the doc, the handler runs remotely inside handleRedisMessage and CAN
throw (markdown->ProseMirror transform). Previously the throw was uncaught: the
customEventComplete reply was never published, so the origin's writePageBody
promise only rejected after customEventTTL (~30s) as a generic 'TIMEOUT', and an
unhandledRejection escaped the async messageBuffer listener on the owning
instance.

Now the owner wraps handleEventLocally in try/catch and, on throw, publishes a
customEventComplete carrying an `error` field on the same correlation channel.
The origin's pendingReplies holds {resolve, reject} and rejects promptly with the
real Error. The TTL TIMEOUT remains as the fallback for a genuinely lost reply.
The no-throw and local (same-instance) paths are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 15:10:10 +03:00
claude code agent 227
d5079aa1d8 fix(git-sync): red-team hardening — 12 confirmed sync-breaking bugs + regression tests
A 10-agent red-team pass on the two-way Docmost<->git sync surfaced 16 ranked
findings (9 others triaged out as already-defended). Wrote a reproduction test
per finding (each asserts the CORRECT behavior, so it fails on the bug), then
fixed the production code so every repro goes green. All confirmed bugs:

Round-trip data loss (markdown-converter.ts + docmost-schema.ts mirror):
- #1 editor-ext node types silently dropped on export — ported the 8 missing
  canon nodes (footnoteReference/footnotesList/footnoteDefinition, htmlEmbed,
  status, pageEmbed, transclusionSource/Reference) into the git-sync schema
  mirror and added converter cases that emit their schema-matching HTML instead
  of flattening unknown nodes to '' (this was the critical data-loss flagged in
  review #1679: footnotes/htmlEmbed lost on sync). Snapshot surface updated.
- #2 top-level image lost width/height/align/attachmentId — now emits an HTML
  <img> (like video/diagrams) when it carries layout attrs; bare images stay
  ![](src). Image node parses width/height as strings so they re-import.
- #3 code block containing a ``` fence corrupted on round-trip — outer fence is
  now widened to (longest-inner-backtick-run + 1).
- #16 deep nesting threw RangeError (page never synced) — added a depth guard
  (MAX_NODE_DEPTH=400) so the converter never overflows the stack.

Push/layout/cycle (engine):
- #4 disambiguation ' ~slugId' suffix corrupted Docmost titles + order-dependent
  layout — deterministic, order-independent sibling disambiguation; suffix is
  stripped from a path-derived title ONLY when the new name is exactly the old
  title plus the suffix (never a genuine retitle ending in ' ~token').
- #6 retry-adopt by (parent,title) clobbered the wrong duplicate-title sibling —
  ambiguous (parent,title) is no longer adopted (falls back to fresh create).
- #12 a new child under a new parent was created at ROOT — creates are ordered
  parent-before-child with an in-memory created-id map for parent resolution.
- #13 git conflict markers could reach Docmost — bodies are scanned and the
  marker lines stripped (a '=======' line is only treated as a conflict
  separator inside a <<<<<<< ... >>>>>>> block, so setext headings are safe).
- #15 a divergent `docmost` mirror was escalated by runPush but dropped by
  runCycle — RunCycleResult now forwards divergentDocmost to the orchestrator.

Server (merge / lock / provenance):
- #9 3-way merge lost a human's block edit when git inserted an adjacent block —
  finer-grained diff3 region merge (via lcs) preserves non-overlapping human
  edits; genuine same-block conflicts still resolve git-wins.
- #10 single-writer race — module-static liveLocks closes the same-process TOCTOU
  window, and a heartbeat refresh that cannot confirm the lock now aborts the
  cycle at its next write checkpoint (cooperative AbortSignal threaded through
  runCycle). Cross-process fencing tokens remain a follow-up.
- #14 sticky-agent provenance overrode an explicit actor='git-sync' write,
  blinding the listener loop-guard — resolveSource now lets an explicit actor
  win over the sticky-agent fallback (explicit agent still wins).

Verified: git-sync vitest 617 pass (+1 expected-fail), server unit jest 1541
pass, server tsc clean. A review pass over the fixes caught and corrected a
title-suffix over-strip, an inert abort signal, a document-wide conflict-marker
strip, and two leaf-atom content-holes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 15:10:10 +03:00
claude code agent 227
8fa32e8438 docs(git-sync): document GIT_SYNC_* env vars; fix stale/non-English comments (PR #119 review)
Addresses the documentation/convention warnings from the #119 review:
- .env.example: add the GIT-SYNC block (9 GIT_SYNC_* vars with defaults), noting
  GIT_SYNC_SERVICE_USER_ID is required when sync is enabled.
- yjs-body-merge.ts: translate the Russian review note in the docstring to
  English (comments-only-in-English rule).
- persistence.extension.ts: correct the stale "git-sync writes are full-body
  replaces" rationale — a git-sync write is now a block-level merge into the live
  doc, which is why it is debounced like a human edit rather than snapshotted.
- history-item.tsx: the GitSyncBadge version is created on the PUSH path (writing
  the git body back into the doc), not by the pull — fix the comment.
- edit-space-form.tsx: log the raw error in the git-sync toggle catch instead of
  swallowing it (AGENTS.md).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:10:10 +03:00
claude_code
d9d1d54aaa test(git-sync): add reviewer-requested coverage across engine, server, client
Implements the test cases called out in the PR #119 review threads
(code-review, test-strategy report, red-team) — TESTS ONLY, no production
code changes.

packages/git-sync (vitest):
- lib converter/markdown gaps: pageBreak data-loss (it.fails repro),
  subpages lossy round-trip, nested/fenced callouts, ol->taskList bridge,
  column.width number<->string drift, empty details.
- engine units: parentFolderFile, planReconciliation swap/chained move,
  buildVaultLayout last-resort-by-id, firstDivergence, applyPushActions /
  applyPullActions failure isolation.
- real temp-git integration: diffNameStatus -z rename+add/modify
  alignment, copy-line behavior, per-invocation committer identity (no
  leak into repo/global config).
- ENFORCED type-level GitSyncClient contract via vitest typecheck over a
  *.test-d.ts file (tsconfig.vitest.json; build tsconfig untouched).

apps/server (jest):
- orchestrator: delete-cap neutralization + fail-safe, Redis lock / mutex
  skip ladder + release-on-throw, merge guard, pull/push order, remote
  template substitution, poll lifecycle.
- page-change listener: loop-guard, debounce coalescing, id resolution,
  error swallowing.
- vault registry, controller authz (trigger + status), env
  validation/getters, page.service git-sync provenance stamping,
  persistence precedence (agent > git-sync > user) + no boundary snapshot,
  space.service audit-delta, space.repo jsonb-merge, converter-gate corpus
  extension (mention/math/details/marks).

apps/client (vitest + testing-library):
- history-item git-sync badge: render gating + non-clickable.
- edit-space-form toggle: initial state, optimistic payload, rollback on
  error, disabled states.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:09:57 +03:00
claude code agent 227
5a4d9f84d7 feat(git-sync): native GitmostDataSource + 'git-sync' provenance (Phase A.4a)
Native data plane for git-sync (plan §3, §8.1):
- provenance: widen actor to 'user'|'agent'|'git-sync' (jwt-payload,
  auth-provenance decorator); PersistenceExtension resolves lastUpdatedSource
  with precedence agent > git-sync > user, debounced history (like a human edit,
  not the agent's immediate snapshot).
- GitmostDataSourceService implements @docmost/git-sync's GitSyncClient natively:
  reads via PageRepo/SpaceRepo (listSpaceTree complete:true, getPageJson), writes
  via PageService (create/removePage soft-delete/movePage with computed fractional
  position/update-rename/restore) + the writeBody linchpin through collab
  openDirectConnection('page.'+id, {actor:'git-sync'}) mirroring
  collaboration.handler withYdocConnection 'replace'. bind({workspaceId,userId})
  returns the context-bound client for the orchestrator.
- 10 unit/contract tests (mapping + soft-delete + move-position), tsc clean.

Known gap (closed in A.4b): PageService.create/update/movePage only branch on
actor==='agent'; git-sync provenance is already passed through so the row source
marker propagates once PageService honors 'git-sync'. Module/orchestrator/config
come next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:09:57 +03:00
claude code agent 227
2d36641f28 test(coverage): add regression tests for issues #192, #206, #204
Additive test coverage across server, editor-ext, client and mcp.

#192 — AiChatService.stream integration (Section 3, against real Postgres):
- new apps/server/test/integration/ai-chat-stream.int-spec.ts drives the real
  streamText through a seeded ai/test MockLanguageModelV3 and a real Node
  ServerResponse, covering: onError persists an assistant error record
  (status 'error' + partial answer + provider cause in metadata); external MCP
  client closed exactly once on BOTH onFinish and onError; anti-tamper —
  history is rebuilt from the DB transcript, not from body.messages.

#206 — red-team findings (most already fixed+tested in #212):
- mdrt-2 (UNFIXED, data loss): turndown.dataloss.test.ts documents that
  pageBreak / transclusionReference / mention are silently dropped on Markdown
  export (characterization + it.fails for the desired survive-export contract).
- persist-6 (UNFIXED, data loss): persistence-store.spec.ts adds an it.failing
  documenting that a momentarily-empty live doc overwrites non-empty content
  (left unfixed — a store-side empty-guard is a behaviour change).

#204 — test-strategy plan, highest-priority subset:
- Phase 1: mcp-clients.lease.spec.ts covers the external MCP client
  lease/refcount/eviction lifecycle (leak / premature-close / double-close).
- Phase 2 data-integrity pure functions: editor-ext table-utils
  (transpose/moveRow/convert round-trip) and math tokenizer false-positive
  guard; client emoji-menu (+ it.fails for the unguarded localStorage
  JSON.parse bug), sort-cells, normalizeTableColumnWidths; mcp htmlEmbed/
  pageBreak markdown data-loss + footnote-diff; server export
  getInternalLinkPageName extensionless-path bug — FIXED (small/clear) + tested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 06:15:55 +03:00
claude code agent 227
770ba70541 fix(collab): retry transient store failures so autosave edits aren't lost (#206)
persist-1: onStoreDocument wrapped the page write in a try/catch that only
logged and swallowed the error, then resolved "successfully". hocuspocus
destroys/unloads the in-memory Y.Doc right after the hook resolves (the only
copy of the latest edit), so a transient DB error (deadlock, serialization
failure, dropped connection) silently lost the edit. Worse, the post-store
branch ran on the partially-assigned `page`, broadcasting a phantom
"page.updated" and enqueueing a history snapshot for content never written.

Wrap the write in a small bounded retry (3 attempts) so the save is
re-attempted while we still hold the doc, and clear `page` on failure so the
success-only side effects never report a save that didn't happen.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 06:05:28 +03:00
claude_code
683b9d5de2 fix(provenance): address #143 review — page-stamp tests, confine is_agent, doc fixes
Resolves the open items from the latest PR #143 code review:

- test(page): cover the four agentSourceFields stamp sites (create, update,
  movePage, movePageToSpace) with agent + normal-user payload assertions;
  add findById({ includeIsAgent: true }) wiring guards to the JWT and collab
  auth-seam specs so a future drop of the option is caught.
- fix(privacy): drop `isAgent` from UserRepo.baseFields and gate it behind a
  new opt-in `findById({ includeIsAgent })`, requested only by the two auth
  seams that derive provenance — stops the flag leaking via the workspace
  member list and generic user payloads.
- docs: correct the agentSourceFields JSDoc and the two UPDATE-site comments
  to distinguish INSERT (omitted column → DB default 'user') from UPDATE
  (omitted column → existing value kept, Kysely writes only present keys).
- style(page): collapse three stray double blank lines left by an earlier edit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 02:04:23 +03:00
claude code agent 227
7705d44fc6 fix(provenance): address #143 re-review — shared resolver + decoupled badge
Architecture & design:
- Arch A: introduce resolveProvenance() as the single source of truth for
  deriving a write's actor/aiChatId from the SIGNED identity, and wire it into
  BOTH transport seams — the REST jwt.strategy and the collab
  authentication.extension. Previously the collab seam derived actor from the
  token claim alone and ignored user.isAgent, so a flagged service account's
  page-content edits over the websocket persisted as lastUpdatedSource='user',
  drifting from REST. The seams now share one resolver and can't diverge.
- Arch B: drop AiAgentBadge's page-history coupling. The generic ui/ badge no
  longer imports historyAtoms; it exposes an onActivate callback fired after the
  deep-link, and the history row passes onActivate to close its own modal.

Suggestions/warnings:
- S1: soften the jwt.strategy provenance comment (applies to every REST write).
- S2/suggestion-3: drop the redundant comment-list-item null-aiChatId test
  (covered by ai-agent-badge.test.tsx).
- S3: de-duplicate jwt.strategy.spec test #3 (the no-claim→'user' half
  duplicated test #2); keep only the signed actor='agent' claim assertion.
- W2: add keyboard-activation tests for the badge (Enter/Space, unrelated key).
- W3: flip the design doc status to "реализовано (#143)".

Tests:
- new auth-provenance.decorator.spec.ts unit-tests resolveProvenance +
  agentSourceFields.
- new collab-seam test: is_agent user with no claim → actor='agent'
  (Arch A regression guard).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 00:27:03 +03:00
claude code agent 227
0647faefcd chore(comments): address PR #143 review (operator doc, badge test, dedup, type)
- [warn 1] Document the is_agent operator setup so it survives plan deletion:
  added an AI-agent block to .env.example (use a DEDICATED account, set is_agent
  via SQL, never flag a human/shared account) + a CHANGELOG "Added" entry.
- [warn 2] Test the badge deep-link side effects: ai-agent-badge.test.tsx now
  renders inside an explicit jotai store, clicks the badge, and asserts the
  active chat id, window-open, cleared draft, closed history modal, AND that
  stopPropagation keeps a parent onClick from firing.
- [suggestion 3] Hoist the window.matchMedia stub into vitest.setup.ts and drop
  the duplicated beforeAll block from the three test files (ai-agent-badge,
  comment-list-item, role-cards).
- [suggestion 4] Merge the two near-duplicate "non-clickable" cases via it.each.
- [follow-up 6] Introduce a single ProvenanceSource = 'user' | 'agent' type in
  jwt-payload.ts and reference it from AuthProvenanceData, JwtPayload/
  JwtCollabPayload, and resolveSource() — so a typo can't slip through as a bare
  string. (Server auth chain; client IComment mirroring left as a follow-up.)

Follow-up 5 (shared agentSourceFields write-stamp helper) is deferred as the
review marked it — the 6 REST sites use varied shapes (create-spread vs
resolve-conditional-null vs page move), so it's a separate focused refactor.

Tests: client badge/comment/role-cards suites 11/11 pass; server auth+comment
suites 62 pass; typecheck clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 23:56:26 +03:00
claude_code
3d4ad664b3 test(refactor-tail): extract pure cores + cover collab/share/ai-chat/client gate
Batches 6-9: behaviour-preserving extractions of testable pure cores plus the
tests they unblock, and a fix for the broken client test environment.
Full suites green: server 113 suites / 1117 + 1 todo, client 30 files / 338.

client (R0 infra):
- vitest.setup.ts: in-memory localStorage/sessionStorage Storage stub wired via
  setupFiles. Unblocks menu-items.gating.test.ts (was 9 failing) -> client suite
  fully green. + menu-items.suggestions.test.ts (getSuggestionItems filter/sort).

share:
- extract buildShareMetaHtml (share-seo.util.ts) from the SEO controller; tests
  for reflected-XSS escaping in <title>/og/twitter meta, noindex, truncation;
  extractPageSlugId; updateAttachmentAttr; prepareContentForShare comment-strip
  (anonymous-viewer metadata-leak guard).

ai-chat (security extractions):
- selectAccessibleHits: CASL post-filter for semantic search (restricted page in
  an accessible space must NOT leak to the agent).
- validateResolvedAddresses: SSRF connect-time guard (block if ANY resolved
  address is private).
- resolveAudioFormat: mime whitelist (dead `?? 'webm'` fallback dropped, set
  unchanged). + mcp-servers toView header-leak guard, MCP tool namespacing.

collaboration (data-loss area):
- extract computeHistoryJob (pins the "agent delay MUST stay 0" invariant) and
  resolveSource. Integration: onAuthenticate read-only matrix (collab auth
  bypass), HistoryProcessor (contributor restore on save failure), onStoreDocument
  Approach-A boundary snapshot (human revision pinned before agent overwrite).

Reviewed (APPROVE WITH SUGGESTIONS): extractions behaviour-preserving, security
tests mutation-resistant.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 19:10:27 +03:00
claude_code
81823fce1e feat(html-embed): sandbox the embed block; split trusted trackers into an admin field
Convert the htmlEmbed node from same-origin raw-HTML execution to a sandboxed
iframe (sandbox="allow-scripts allow-popups allow-forms", no allow-same-origin,
srcdoc) with postMessage auto-resize (validated by event.source) and an optional
manual height attr. The block now runs in an opaque origin and cannot reach the
viewer's cookies/session/API, so it is safe for any member.

Because the block is now harmless, remove the entire admin/role gating apparatus:
drop htmlEmbedAllowed/canAuthorHtmlEmbed/stripDisallowedHtmlEmbedNodes/
collectHtmlEmbedSources and every role-based strip on the write paths (collab
REST/MCP + socket, page create/duplicate, import x2, transclusion unsync), along
with the now-unused WorkspaceRepo/UserRepo injections and the PageService.create
callerRole param. Keep one strip: prepareContentForShare still removes htmlEmbed
on the anonymous public-share read path when the workspace master toggle is OFF.

The workspace settings.htmlEmbed toggle is now a plain feature switch (gates the
slash-menu and share rendering); when ON the block is available to all members.

Add settings.trackerHead: an admin-only raw HTML/JS analytics snippet injected
verbatim into the <head> of public share pages only (ShareSeoController), for
trackers that genuinely need same-origin. Admin-gated via the existing CASL
Manage/Settings ability; never injected into the authenticated app shell.

Closes security-review findings #1, #2, #4, #5, #10 (and #3 as a security issue).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 02:48:41 +03:00
claude_code
19ae6a0efa Merge pull request 'feat(editor): page templates — live whole-page embed (MVP)' (#17) from feat/page-templates into develop
Some checks failed
Develop / build (push) Has been cancelled
2026-06-20 20:34:44 +03:00
claude code agent 227
8fcce6a674 feat(html-embed): per-workspace feature toggle, default OFF
The admin-only raw HTML/JS embed is a deliberate stored-XSS surface, so gate the
whole feature behind a workspace toggle that is OFF by default; it only works
when a workspace admin explicitly enables it.

- settings.htmlEmbed (boolean, default false) + workspace-update field htmlEmbed,
  persisted via WorkspaceRepo.updateSetting with an audit diff. Flipping it is
  admin-only (same Manage Settings CASL as other workspace toggles).
- New gate htmlEmbedAllowed(featureEnabled, role) = featureEnabled && admin/owner.
  All 7 server write paths (create, duplicate, collab onStoreDocument, REST/MCP/AI
  updatePageContent, single + zip import, transclusion unsync) now read the
  workspace's settings.htmlEmbed and strip unless (toggle ON AND admin). OFF
  (default, or a failed/empty workspace lookup) strips htmlEmbed for EVERYONE
  including admins -> existing embeds are cleaned up on next save, none persist.
- Client (defense-in-depth): the /html slash item is hidden unless toggle ON +
  admin; the NodeView executes nothing and shows a 'disabled in this workspace'
  placeholder when OFF; an admin Switch in Workspace Settings -> General with a
  description of the behavior.
- docs/html-embed-admin.md documents the toggle + admin-only + fail-closed
  coedit (a non-admin save strips an admin's embed) + execution semantics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 19:28:39 +03:00
claude code agent 227
caac5c7f36 test(html-embed): exercise the REAL admin-gate write paths + import round-trip
Release-cycle test audit: the strip boundary was tested only via a stand-in
helper re-implemented in the spec, so a deleted/misplaced guard kept CI green
(the missing create() guard was proof). Replace it with tests against real code:
- persistence.extension.onStoreDocument: real ydoc from a rich doc (columns/
  table/mention/htmlEmbed) -> non-admin strip removes only htmlEmbed, every other
  node preserved (data-loss guard); admin keeps; empty fragment no-throw.
- collaboration.handler.updatePageContent: real path, user?.role gate, decoded
  ydoc embed-free for non-admin, kept for admin.
- transclusion unsync: member stripped, admin preserved.
- editor-ext gains a vitest setup (was zero tests) + a markdown round-trip:
  the <!--html-embed:BASE64--> marker -> htmlEmbed node with decoded source, and
  hasHtmlEmbedNode matches it — pinning the marked/turndown shape the import
  strip relies on. tsconfig now excludes specs from the shipped dist.
- Fail-closed identity: source-pinned contracts that the gate keys on
  fileTask.creatorId (zip) / request userId (single) / callerRole (create) /
  authUser.role (duplicate), and missing-user -> strip (services can't load under
  jest's ESM graph; helpers replay the exact predicate).
Adds the verified-safe ^src/ jest moduleNameMapper (identical fail set).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 14:52:29 +03:00
claude code agent 227
39ae89264d feat(editor): page templates - live whole-page embed (MVP)
Embed another page's LIVE content into a host page (it updates when the source
changes, not a static copy). A page can be flagged a template for discovery in
the picker; any accessible page can be embedded.

Server:
- migrations: pages.is_template (+ partial index) and page_template_references
  (whole-page back-refs); db.d.ts/entity types hand-merged (db.d.ts is curated).
- POST /pages/toggle-template (CASL Edit) flips is_template; is_template is
  returned by findById + the sidebar tree select so the tree menu label
  reflects state. Search suggestions gain an onlyTemplates filter for the picker.
- POST /pages/template/lookup ({sourcePageIds[]}, <=50): returns each accessible
  source's {title, icon, slugId, content, sourceUpdatedAt} with comment marks
  stripped (same access path as transclusion: filterViewerAccessiblePageIds;
  inaccessible -> no_access, missing -> not_found; error path -> not_found, never
  raw content).
- reference sync (collectPageEmbedsFromPmJson + syncPageTemplateReferences) on
  the Yjs save hook; duplicatePage remaps pageEmbed.sourcePageId + inserts refs.
  Known MVP gap: REST content updates don't resync refs (lookup uses in-doc ids).

Client:
- pageEmbed node (editor-ext, registered in BOTH client + server schemas);
  read-only NodeView with a batching lookup; '/Embed page' slash + template
  picker (self-embed prevented); 'Make/Unset template' in the tree node menu.
- Cycle guard: an ancestry-chain context + depth cap (5) render a 'circular
  embed' placeholder instead of recursing.
- Public shares show a placeholder (no public lookup in MVP).

MVP excludes (follow-ups): public-share lookup, unsync->static copy, server-side
expansion for export/RAG, MCP schema mirror, point-in-time snapshots.

Implements docs/page-templates-plan.md (MVP, variant A).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 10:05:00 +03:00
claude code agent 227
bd28dbfe2b feat(editor): admin-only raw HTML/CSS/JS embed node
Adds an htmlEmbed block node that renders and executes raw HTML/CSS/JS in the
wiki origin (e.g. an analytics tracker) — the owner-chosen variant C. Because
this is stored-XSS by design, only workspace admins/owners may get such a node
persisted; everyone executes it when reading.

- Node (editor-ext): htmlEmbed atom/isolating block; source stored base64 in
  data-source for lossless HTML<->JSON round-trip. renderHTML emits only the
  encoded marker (never inlines raw markup), so generateHTML/export/search are
  not themselves injection vectors. Registered in BOTH client extensions and
  server tiptapExtensions. Markdown round-trip via an <!--html-embed:b64-->
  comment (turndown) + a marked rule.
- Client NodeView: injects source and re-creates <script> elements so they
  actually run; edit modal; renders in read-only/share too. Slash item is
  admin-gated (adminOnly filtered by the user's workspace role).
- SERVER ENFORCEMENT (the real control — UI gating alone is insufficient):
  stripHtmlEmbedNodes() removes htmlEmbed from any document persisted by a
  non-admin, applied at every write path that introduces content from an
  untrusted author: collab onStoreDocument, REST/MCP/AI updatePageContent,
  single-file import, zip/multi-file import, page duplication, and transclusion
  unsync. Page restore introduces no new content. Public share/readonly viewers
  render fetched (already-stripped) content and do NOT open a collab socket, so
  the only residual is a transient broadcast window to concurrent authenticated
  editors (documented).

Implements docs/arbitrary-html-embed-plan.md (variant C).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 08:54:54 +03:00
vvzvlad
551f975886 fix(collab): use '-' instead of ':' in agent page-history jobId
BullMQ rejects custom job IDs containing ':' (Redis key separator),
throwing "Custom Id cannot contain :" inside the onStoreDocument hook
for every agent edit. This broke agent-driven page saves (MCP
create_page runs as actor='agent') with HTTP 400.

Switch the agent dedup suffix from `${page.id}:agent` to
`${page.id}-agent`. The jobId is only used as a BullMQ dedup key and is
never parsed by the history processor; page.id is a UUID, so the
hyphenated id cannot collide with a human job whose id is a bare page.id.
2026-06-17 17:38:32 +03:00
vvzvlad
0a9788e89a feat(collab): separate agent edits from human edits in page history
Page-history snapshots are debounced/coalesced (one per 1–5 min window,
jobId=page.id). A human edit followed by an agent edit in the same window
collapsed into a single snapshot, losing both the pre-agent human state and
a deterministic record of the agent's result.

Two provenance-aware boundaries now bracket an agent intervention:
- Before: on a user->agent transition, onStoreDocument synchronously pins the
  current (pre-agent) human content as its own history version tagged 'user',
  inside the page-write transaction, before the agent overwrites it.
- After: agent stores enqueue an immediate (delay 0), source-keyed history job
  (jobId=`${pageId}:agent`) so the agent's result snapshots deterministically
  as 'agent' and a later human edit (jobId=page.id) cannot coalesce/retag it.

Also add an `id desc` tie-break to findPageLastHistory so "last history" stays
deterministic when two snapshots share a created_at, consistent with
findPageHistoryByPageId.

Known trade-offs (Variant 1): the delay-0 worker re-reads the row, leaving a
millisecond mis-tag window; multiple agent edits in one turn may yield multiple
versions. The reverse agent->human boundary is intentionally out of scope.
2026-06-17 06:40:28 +03:00
vvzvlad
683da7a4c5 feat(ai-chat): per-user AI agent backend — LLM config, read-only agent, provenance schema
WIP checkpoint of the gitmost AI-chat backend (plan stages A + B1 + B3a).
The agent acts under the requesting user's JWT (Docmost CASL enforces page
access); the external service-account /mcp endpoint is untouched.

LLM provider config (A2-A4):
- integrations/crypto: AES-256-GCM SecretBoxService (key derived from APP_SECRET,
  per-record salt/iv; clear error on rotation instead of crashing).
- ai_provider_credentials table/repo/types: encrypted API key stored outside
  workspace settings/baseFields, write-only (never returned by any endpoint).
- integrations/ai: per-workspace AI SDK v6 provider driver (openai/gemini/ollama),
  admin-gated GET(masked)/PATCH(write-only key)/Test endpoints; settings.ai.provider
  holds non-secret config incl. systemPrompt. Removed unused AI_* env getters (DB is
  the single source of truth).

Chat module (A1, A5-A8):
- ai_chats/ai_chat_messages repos (workspace-scoped, soft-delete, tsv never selected).
- core/ai-chat: CRUD + POST /ai-chat/stream (Fastify hijack + AI SDK v6
  pipeUIMessageStreamToResponse, abort on disconnect, persist user/assistant msgs).
- Agent loop: streamText + stepCountIs(8); read tools searchPages/getPage via a
  per-request DocmostClient over loopback REST under the user's minted access token.
- Gate settings.ai.chat (+ 503 when provider unconfigured); buildSystemPrompt with a
  non-removable safety/anti-prompt-injection framework. Per-user rate limit.

Per-user auth (B1):
- @docmost/mcp DocmostClient gains an additive getToken variant (carry a user JWT,
  re-fetch on 401) and exports DocmostClient; the email/password service-account path
  (external /mcp, stdio) is unchanged.

Agent-edit provenance backbone (B3a):
- Migration: pages/page_history (last_updated_source, last_updated_ai_chat_id) and
  comments (created_source, ai_chat_id, resolved_source).
- Signed actor/aiChatId claim in the collab token; onAuthenticate propagates it,
  onStoreDocument writes it with a sticky agent marker, saveHistory copies it.

Migrations auto-run on boot (additive). Write tools, frontend, RAG and external MCP
servers are not in this checkpoint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 01:36:41 +03:00
Philip Okugbe
33895b0607 bug fixes (#2250)
* util

* fix page position collation

* support fixed toolbar in templates editor

* date localization

* fix clipped emoji in templates editor

* fix page updated time object

* fix flickers

* fix: remove redundant breadcrumb from destination modal
2026-05-28 16:20:37 +01:00
Philip Okugbe
de60aa7e61 feat: synced blocks (transclusion) (#2163)
* feat: synced blocks (transclusion)

* fix:remove name

* make placeholders smaller

* feat: enforce strict transclusion schema

* fix: scope synced blocks to workspace, gate unsync on edit permission

* fix collab module error
2026-05-08 13:23:16 +01:00
Philip Okugbe
879aa2c3d8 feat: page update notifications (#2074)
* feat: watchers notification and email preferences

* fix: email copy

* digests

* clean up

* fix

* clean up

* move backlinks queue-up to history processor

* fix

* fix keys

* feat: group notifications

* filter

* adjust email digest window
2026-03-31 16:03:59 +01:00
Philipinho
721651e2e2 feat: user deactivation 2026-03-02 19:05:10 +00:00
Philip Okugbe
59e945562d feat(ee): page-level access/permissions (#1971)
* Add page_hierarchy table

* feat(ee): page-level permissions

* pagination

* rename migration
fixes

* fix

* tabs

* fix theme

* cleanup

* sync

* page permissions notification
* other fixes

* sharing disbled

* fix column nodes

* toggle error handling
2026-02-26 19:49:10 +00:00
Philip Okugbe
05b3c65b0f feat: notifications (#1947)
* feat: notifications
* feat: watchers

* improvements

* handle page move for watchers

* make watchers non-blocking

* more
2026-02-14 20:00:38 -08:00
Philip Okugbe
18b5781522 feat(API): page content update and retrieval (#1937)
* feat: page content update and retrieval output

* import module

* refactor naming
* support prepend

* rename contentOperation -> operation

* dry

* add yjs utils
2026-02-12 11:13:47 -08:00
Philip Okugbe
0f02261ee6 feat: page version history improvements (#1925)
* Refactor: use queue for page history

* feat: save multiple version contributors

* display contributor avatars in history list

* fix interval
2026-02-09 18:25:35 -08:00
Philip Okugbe
60501de992 fix: missing logs on OnApplicationBootstrap hook (#1882)
* - fix: set default Nest logger and bufferLogs to false for pino compatibility
- handle redis error event

* fix collab server logging too
2026-01-29 09:25:23 +00:00
Philip Okugbe
74e915546b feat: collab redis extension with server affinity (#1873)
* feat(collab): better redis extension
* move types to own file
* debug logging
* fix: graceful collab shutdown
* rename default prefix
* pass wsAdapter to gateway
* expose event handler
* unique collab serverId generation
* uninstall @hocuspocus/extension-redis package
* expose more functions
* sync with latest
* cleanup
* fastify router options
* cleanup type
2026-01-27 17:05:05 +00:00
Philip Okugbe
657fdf8cb7 feat: Tiptap V3 migration (#1854)
* Tiptap3 migration - WIP

* fix collaboration

* remove unused code

* fix flicker

* disable duplicate extensions

* update tiptap version

* Switch to useEditorState
- Set shouldRerenderOnTransaction to false

* fix editable state

* add tippyoptions for reference

* merge main

* tiptap 3.6.1

* fix bubble menu

* fix converter

* fix menus

* fix collaboration caret css

* fix: Set `isInitialized` to force immediate react node view rendering

* feat: Migrate tippy.js menus to Floating UI

* feat: Update collaboration connection for HocusPocus v3

* fix: Connect/disconnect websocketProvider

* cleanup

* cleanup

* feat: Improved placeholder and upload handling for images

* feat: Improved placeholder and upload handling for videos

* refactor: Image node and view clean-up

* feat: Improved placeholder and upload handling for attachments

* fix: Video view styles

* fix: Transaction handling on asset upload

* fix: Use imageDimensionsFromStream

* feat: Multiple file upload, improved placeholders, local previews

* fix: Drag & drop, paste upload

* fix: Allow media as attachment

* * add skeleton pulse animation
* add translation strings
* fix attachment view responsiveness

* fix collab connection status display

* Tiptap v3.17.0

* fix suggestion menu exit bug

* fix search shortcut

* fix history editor css

* tiptap 3.17.1

---------

Co-authored-by: Arek Nawo <areknawo@areknawo.com>
2026-01-24 20:41:08 +00:00
Philip Okugbe
9fb16bc842 feat(EE): AI vector search (#1691)
* WIP

* AI module - init

* WIP

* sync

* WIP

* refactor naming

* new columns

* sync

* sync

* fix search bug

* stream response

* WIP

* feat embeddings sync

* refine

* Add workspaceId to page events

* refine

* WIP

* add translation string

* sync

* reset ai answer on query change

* hide AI search in cloud

* capture streaming error

* sync
2025-12-01 11:50:25 +00:00
Philip Okugbe
e51a93221c more checks for collab auth token (#1345) 2025-07-14 10:35:03 +01:00
Philip Okugbe
6d024fc3de feat: bulk page imports (#1219)
* refactor imports - WIP

* Add readstream

* WIP

* fix attachmentId render

* fix attachmentId render

* turndown video tag

* feat: add stream upload support and improve file handling

- Add stream upload functionality to storage drivers\n- Improve ZIP file extraction with better encoding handling\n- Fix attachment ID rendering issues\n- Add AWS S3 upload stream support\n- Update dependencies for better compatibility

* WIP

* notion formatter

* move embed parser to editor-ext package

* import embeds

* utility files

* cleanup

* Switch from happy-dom to cheerio
* Refine code

* WIP

* bug fixes and UI

* sync

* WIP

* sync

* keep import modal mounted

* Show modal during upload

* WIP

* WIP
2025-06-09 04:29:27 +01:00
Philip Okugbe
e8847bd9cd fix: handle unhandled exceptions (#1116)
* Handle unhandled exceptions
* cleanup
2025-04-29 23:29:00 +01:00
Philip Okugbe
64f0531093 feat: keep track of page contributors (#959)
* WIP

* feat: store and retrieve page contributors
2025-04-04 13:03:57 +01:00
Philip Okugbe
fd36076ae7 feat: disconnect collab websocket on idle tabs (#848)
* disconnect real-time collab if user is idle
* log yjs document disconnect and unload in dev mode
* no longer set editor to read-only mode on collab websocket disconnection
* treat delayed collab websocket "connecting" state as disconnected
* increase maxDebounce to 45 seconds
* add reset handle to useIdle hook
2025-03-08 18:16:23 +00:00
Philip Okugbe
b81c9ee10c feat: cloud and ee (#805)
* stripe init
git submodules for enterprise modules

* * Cloud billing UI - WIP
* Proxy websockets in dev mode
* Separate workspace login and creation for cloud
* Other fixes

* feat: billing (cloud)

* * add domain service
* prepare links from workspace hostname

* WIP

* Add exchange token generation
* Validate JWT token type during verification

* domain service

* add SkipTransform decorator

* * updates (server)
* add new packages
* new sso migration file

* WIP

* Fix hostname generation

* WIP

* WIP

* Reduce input error font-size
* set max password length

* jwt package

* license page - WIP

* * License management UI
* Move license key store to db

* add reflector

* SSO enforcement

* * Add default plan
* Add usePlan hook

* * Fix auth container margin in mobile
* Redirect login and home to select page in cloud

* update .gitignore

* Default to yearly

* * Trial messaging
* Handle ended trials

* Don't set to readonly on collab disconnect (Cloud)

* Refine trial (UI)
* Fix bug caused by using jotai optics atom in AppHeader component

* configurable database maximum pool

* Close SSO form on save

* wip

* sync

* Only show sign-in in cloud

* exclude base api part from workspaceId check

* close db connection beforeApplicationShutdown

* Add health/live endpoint

* clear cookie on hostname change

* reset currentUser atom

* Change text

* return 401 if workspace does not match

* feat: show user workspace list in cloud login page

* sync

* Add home path

* Prefetch to speed up queries

* * Add robots.txt
* Disallow login and forgot password routes

* wildcard user-agent

* Fix space query cache

* fix

* fix

* use space uuid for recent pages

* prefetch billing plans

* enhance license page

* sync
2025-03-06 13:38:37 +00:00
Philip Okugbe
e209aaa272 feat: internal page links and mentions (#604)
* Work on mentions

* fix: properly parse page slug

* fix editor suggestion bugs

* mentions must start with whitespace

* add icon to page mention render

* feat: backlinks - WIP

* UI - WIP

* permissions check
* use FTS for page suggestion

* cleanup

* WIP

* page title fallback

* feat: handle internal link paste

* link styling

* WIP

* Switch back to LIKE operator for search suggestion

* WIP
* scope to workspaceId
* still create link for pages not found

* select necessary columns

* cleanups
2025-02-14 15:36:44 +00:00
Philip Okugbe
990612793f refactor: switch to HttpOnly cookie (#660)
* Switch to httpOnly cookie
* create endpoint to retrieve temporary collaboration token

* cleanups
2025-01-22 22:11:11 +00:00
Philipinho
1f4bd129a8 editor improvements
* add callout, youtube embed, image, video, table, detail, math
* fix attachments module
* other fixes
2024-06-20 14:57:00 +01:00
Philipinho
d4eefa48a8 restructure directories
* set log level based on env
2024-06-09 15:57:52 +01:00
Philipinho
2afbede8ce fix page history generation 2024-06-07 16:23:23 +01:00
Philipinho
886d9591fa frontend permissions
* rework backend workspace permissions
2024-06-03 02:54:12 +01:00
Philipinho
9c7c2f1163 updates and fixes
* seo friendly urls
* custom client serve-static module
* database fixes
* fix recent pages
* other fixes
2024-05-18 03:19:42 +01:00
Philipinho
a2768e7d30 collaboration module cleanup x2 2024-05-04 17:21:44 +01:00
Philipinho
6a57e3edb2 Make colloboration module not dependant on page module
* add unloadImmediately to collab
2024-04-22 23:40:50 +01:00
Philipinho
28ec542ed6 server: page permissions 2024-04-22 02:25:03 +01:00
Philipinho
c18c9ae02b Refactoring
* replace TypeORM with Kysely query builder
* refactor migrations
* other changes and fixes
2024-03-29 01:46:11 +00:00
Philipinho
cdcb4e87d0 * Make JwtGuard better
* Fix auth decorators
2024-02-28 03:07:24 +00:00