refactor(git-sync): internalize the engine — first-class ESM, no vendoring bridge (#119 review)

Closes the architecture item from the #119 review: drop the "vendored from
docmost-sync" framing and the CJS↔ESM `Function('import()')` bridge so the engine
is a normal first-class gitmost package.

Part 1 — vendoring markers removed (prose only, zero behavior change): reworded
"VENDORED into gitmost" / "vendored from docmost-sync" / "Engine LOGIC is
byte-identical" / "it's a port" comments across the engine. Behavior-bearing
strings are untouched: BOT_AUTHOR_NAME/EMAIL and the `Docmost-Sync-Source:`
provenance trailers (changing them would break git authorship + the loop-guard).

Part 2 — the package is now ESM (matching the sibling @docmost/mcp): `type: module`,
tsconfig Node16, `.js` extensions on relative imports, and a static
`import { marked }` replacing the `new Function('return import(...)')` /
`loadMarked` hack — the bridge is GONE from the package. The CommonJS NestJS
server loads the now-ESM engine via a new `git-sync.loader.ts` that mirrors the
existing `docmost-client.loader.ts` mcp loader exactly (Function-indirected
dynamic import + cached promise + retry-on-reject). The 4 server consumers
(orchestrator/datasource/vault-registry/git-http-backend) call `await loadGitSync()`
for value exports; types stay `import type` (erased). The converter-gate spec —
which needs the real converter — loads the package's TS source via a jest
moduleNameMapper + isolatedModules (documented in that spec); the other git-sync
specs mock the loader.

Verified: engine builds pure ESM (no Function/require leftover), vitest 614,
editor-ext build, server + client tsc, full server jest 1397/0. Live stand
smoke-test: server starts clean on the ESM engine (no ERR_REQUIRE_ESM), a real
sync cycle runs through the loader, and the basic e2e suite is 12/12 (clone via
git-http-backend, push, pull, delete, 3-way merge — all through the new loader).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
claude code agent 227
2026-06-24 14:23:40 +03:00
parent 9f6478a0ed
commit a093bc8331
29 changed files with 270 additions and 205 deletions

View File

@@ -1,24 +1,21 @@
/**
* The client seam. Upstream `pull.ts`/`push.ts` reached into the
* REST `DocmostClient` from the `docmost-client` package via `Pick<DocmostClient,
* ...>` subsets. That package is NOT vendored here (the gitmost server writes
* NATIVELY — through repositories + collab `openDirectConnection`),
* so the engine must depend on a narrow STRUCTURAL interface instead.
* The client seam. `pull.ts`/`push.ts` depend on a narrow STRUCTURAL interface
* rather than any concrete client, because the gitmost server writes NATIVELY —
* through repositories + collab `openDirectConnection`.
*
* `GitSyncClient` is that interface: the native datasource (server side, a later
* step) implements it, and the vendored engine only ever uses `Pick<GitSyncClient,
* ...>` subsets of it. The signatures below MIRROR exactly the methods the
* vendored `pull.ts`/`push.ts` actually call (arg shapes + the fields the engine
* reads off each result) — verified against the upstream `DocmostClient`
* (packages/docmost-client/src/client.ts) so a real REST client is still
* structurally assignable, and so the native adapter has a precise contract.
* `GitSyncClient` is that interface: the native datasource (server side)
* implements it, and the engine only ever uses `Pick<GitSyncClient, ...>`
* subsets of it. The signatures below MIRROR exactly the methods the engine's
* `pull.ts`/`push.ts` actually call (arg shapes + the fields the engine reads
* off each result), so a REST-style client is still structurally assignable and
* the native adapter has a precise contract.
*/
/**
* A page node as returned by `listSpaceTree` (the sidebar/tree walk, no body).
* The engine layout (`buildVaultLayout`) consumes `PageNode` from `./layout`,
* which only requires `id` (+ optional `title`/`slugId`/`parentPageId`); this
* lite shape documents the fields the tree walk surfaces. Upstream nodes also
* lite shape documents the fields the tree walk surfaces. Real tree nodes also
* carry `position`, `icon`, `hasChildren` — kept open via the index signature.
*/
export interface GitSyncPageNodeLite {
@@ -27,7 +24,7 @@ export interface GitSyncPageNodeLite {
title?: string;
parentPageId?: string | null;
hasChildren?: boolean;
/** Upstream `listSpaceTree` nodes carry extra fields (position, icon, …). */
/** `listSpaceTree` nodes carry extra fields (position, icon, …). */
[key: string]: unknown;
}

View File

@@ -1,8 +1,8 @@
import { VaultGit } from "./git";
import { GitSyncClient } from "./client.types";
import { Settings } from "./settings";
import { readExisting, computePullActions, applyPullActions } from "./pull";
import { runPush } from "./push";
import { VaultGit } from "./git.js";
import { GitSyncClient } from "./client.types.js";
import { Settings } from "./settings.js";
import { readExisting, computePullActions, applyPullActions } from "./pull.js";
import { runPush } from "./push.js";
/**
* Absolute-path filesystem primitives the cycle needs. Injected (not imported)

View File

@@ -3,11 +3,10 @@
*
* IMPORTANT — VAULT-SCOPED: every operation here runs with `cwd = vaultPath`,
* which is the vault's OWN git repository (default `data/vault`), SEPARATE from
* the docmost-sync source repo. This module MUST NEVER run git against the
* source repo. `data/` is gitignored by the source repo, so a nested repo under
* `data/vault` is safe. The pull cycle is READ-ONLY toward Docmost; this module
* only touches the local vault git, never a git remote (push is deferred, see
* SPEC §7).
* the gitmost application repo. This module MUST NEVER run git against the
* application repo. `data/` is gitignored, so a nested repo under `data/vault`
* is safe. The pull cycle is READ-ONLY toward Docmost; this module only touches
* the local vault git, never a git remote (push is deferred, see SPEC §7).
*
* Implementation notes:
* - We shell out via `node:child_process` `execFile` (promisified), passing

View File

@@ -10,7 +10,7 @@
* lives in each file's meta block (pageId / slugId).
*/
import { sanitizeTitle, disambiguate } from "./sanitize";
import { sanitizeTitle, disambiguate } from "./sanitize.js";
/** Flat page node as returned by `listAllSpacePages` (no content). */
export interface PageNode {

View File

@@ -25,30 +25,29 @@
* (read-only: listSpaceTree + getPageJson). All git operations run against
* the vault repo (`cwd = vaultPath`), never the source repo (see ./git.ts).
*
* VENDORED into gitmost: the client seam is the native
* `GitSyncClient` (`Pick<GitSyncClient, ...>`), not the upstream REST
* `DocmostClient`; the upstream CLI `main()` entry point is dropped (the gitmost
* server drives the engine in-process). Engine LOGIC is byte-identical.
* The client seam is the native `GitSyncClient` (`Pick<GitSyncClient, ...>`);
* the gitmost server drives the engine in-process (there is no standalone CLI
* entry point).
*/
import { dirname } from "node:path";
import { sep } from "node:path";
import { parsePageFile, serializePageFile } from "../lib/page-file";
import type { GitSyncClient } from "./client.types";
import { buildVaultLayout, type PageNode } from "./layout";
import { parsePageFile, serializePageFile } from "../lib/page-file.js";
import type { GitSyncClient } from "./client.types.js";
import { buildVaultLayout, type PageNode } from "./layout.js";
import {
VaultGit,
BOT_AUTHOR_NAME,
BOT_AUTHOR_EMAIL,
DEFAULT_BRANCH,
} from "./git";
} from "./git.js";
import {
planReconciliation,
decideAbsenceDeletions,
type LiveEntry,
type MovedEntry,
type DeletionDecision,
} from "./reconcile";
import { stabilizePageBody } from "./stabilize";
} from "./reconcile.js";
import { stabilizePageBody } from "./stabilize.js";
// Engine-only mirror branch (SPEC §5): the engine writes here, humans never do.
const DOCMOST_BRANCH = "docmost";

View File

@@ -22,21 +22,20 @@
* then calls `move_page` / `rename_page` (both for a reparent+retitle), or
* records a NO-OP for a cosmetic local-only file-path rename.
*
* VENDORED into gitmost: the client seam is the native
* `GitSyncClient` (`Pick<GitSyncClient, ...>`), not the upstream REST
* `DocmostClient`; the upstream CLI `main()` entry point is dropped (the gitmost
* server drives the engine in-process). Engine LOGIC is byte-identical.
* The client seam is the native `GitSyncClient` (`Pick<GitSyncClient, ...>`);
* the gitmost server drives the engine in-process (there is no standalone CLI
* entry point).
*/
import { type DocmostMdMeta } from "../lib/index";
import { parsePageFile, serializePageFile } from "../lib/page-file";
import type { GitSyncClient } from "./client.types";
import type { DiffEntry } from "./git";
import { VaultGit, DEFAULT_BRANCH } from "./git";
import { bodyHash } from "./loop-guard";
import { type Settings } from "./settings";
import { type DocmostMdMeta } from "../lib/index.js";
import { parsePageFile, serializePageFile } from "../lib/page-file.js";
import type { GitSyncClient } from "./client.types.js";
import type { DiffEntry } from "./git.js";
import { VaultGit, DEFAULT_BRANCH } from "./git.js";
import { bodyHash } from "./loop-guard.js";
import { type Settings } from "./settings.js";
// Re-export so callers/tests can import the diff row shape from either module.
export type { DiffEntry } from "./git";
export type { DiffEntry } from "./git.js";
/** A page to CREATE in Docmost (new local file, meta has no pageId yet). */
export interface CreateAction {

View File

@@ -1,9 +1,7 @@
/**
* Pure helpers extracted from the docmost-sync Phase-0 idempotency harness
* (`src/roundtrip.ts`). Only the IO-free comparison utilities are vendored —
* the CLI scaffold (`--fixture`/`--page`/`--corpus`, `loadSettings`, the
* `DocmostClient` live path and `process.exit`) is NOT vendored (the roundtrip
* harness moves into the package's tests, not the engine).
* Pure, IO-free comparison helpers for the idempotency round-trip checks. The
* round-trip harness that drives these lives in the package's tests, not in the
* engine.
*/
/**

View File

@@ -1,16 +1,14 @@
/**
* Engine settings (ADAPTED for vendoring).
* Engine settings.
*
* Upstream this module also loaded `.env` (`dotenv`) and bound `parseSettings`
* to `process.env` via a `loadSettings()` entry point. In gitmost the engine is
* driven IN-PROCESS by the NestJS server, which builds the `Settings` object
* from `EnvironmentService` — so the engine must NOT reach into
* `process.env` here. We therefore vendor ONLY:
* The engine is driven IN-PROCESS by the NestJS server, which builds the
* `Settings` object from `EnvironmentService` — so this module must NOT reach
* into `process.env`. It exposes only:
* - the `Settings` type the engine consumes, and
* - `parseSettings(env)` as a PURE function (validate a raw env object -> typed
* `Settings`), kept for unit tests and for the server to reuse if it wants
* to validate an env-shaped object.
* The `loadSettings()` / `loadDotenv()` side-effecting entry point is dropped.
* There is no `.env`-loading side-effecting entry point.
*/
import { z } from 'zod';

View File

@@ -17,7 +17,7 @@ import {
markdownToProseMirror,
serializeDocmostMarkdownBody,
type DocmostMdMeta,
} from "../lib/index";
} from "../lib/index.js";
/**
* Meta object as `exportPageBody` builds it (SPEC §4). Kept byte-for-byte

View File

@@ -1,9 +1,10 @@
/**
* Public surface of `@docmost/git-sync`.
*
* Phase A vendors only the PURE converter + pure engine modules
* from docmost-sync. Server integration (GitmostDataSource, orchestrator,
* VaultGit, pull/push) is added in later steps.
* Exposes the pure converter (markdown <-> ProseMirror, file envelope,
* canonicalization) and the sync engine (reconcile planner, vault layout,
* pull/push, the git wrapper, and the settings parser) that the gitmost server
* drives in-process.
*/
// Pure converter (markdown <-> ProseMirror, file envelope, canonicalization).
@@ -15,8 +16,8 @@ export {
markdownToProseMirror,
canonicalizeContent,
docsCanonicallyEqual,
} from "./lib/index";
export type { DocmostMdMeta } from "./lib/index";
} from "./lib/index.js";
export type { DocmostMdMeta } from "./lib/index.js";
// Pure engine (no IO): reconcile planner, vault layout, sanitize, stabilize,
// loop-guard body hash.
@@ -25,7 +26,7 @@ export {
decideAbsenceDeletions,
MASS_DELETE_MIN_EXISTING,
MASS_DELETE_FRACTION,
} from "./engine/reconcile";
} from "./engine/reconcile.js";
export type {
LiveEntry,
ExistingEntry,
@@ -33,23 +34,23 @@ export type {
MovedEntry,
ReconciliationPlan,
DeletionDecision,
} from "./engine/reconcile";
} from "./engine/reconcile.js";
export { buildVaultLayout } from "./engine/layout";
export type { PageNode, VaultEntry } from "./engine/layout";
export { buildVaultLayout } from "./engine/layout.js";
export type { PageNode, VaultEntry } from "./engine/layout.js";
export { sanitizeTitle, disambiguate } from "./engine/sanitize";
export { sanitizeTitle, disambiguate } from "./engine/sanitize.js";
export { stabilizePageFile } from "./engine/stabilize";
export type { PageMeta } from "./engine/stabilize";
export { stabilizePageFile } from "./engine/stabilize.js";
export type { PageMeta } from "./engine/stabilize.js";
export { bodyHash } from "./engine/loop-guard";
export { bodyHash } from "./engine/loop-guard.js";
// IO engine: the client seam, the VaultGit git wrapper, the
// pull (Docmost->FS) + push (FS->Docmost) planners/appliers, and the (pure)
// settings parser. The engine consumes the native `GitSyncClient` seam (server
// implements it) the upstream REST `DocmostClient` is NOT vendored.
export type { GitSyncClient, GitSyncPageNodeLite } from "./engine/client.types";
// settings parser. The engine consumes the native `GitSyncClient` seam (the
// server implements it) rather than any REST client.
export type { GitSyncClient, GitSyncPageNodeLite } from "./engine/client.types.js";
export {
VaultGit,
@@ -58,21 +59,21 @@ export {
BOT_AUTHOR_NAME,
BOT_AUTHOR_EMAIL,
DEFAULT_BRANCH,
} from "./engine/git";
export type { DiffEntry, MergeResult, CommitOptions } from "./engine/git";
} from "./engine/git.js";
export type { DiffEntry, MergeResult, CommitOptions } from "./engine/git.js";
export {
readExisting,
computePullActions,
applyPullActions,
} from "./engine/pull";
} from "./engine/pull.js";
export type {
ReadExistingDeps,
PullActionsInput,
PullActions,
ApplyPullActionsDeps,
ApplyResult,
} from "./engine/pull";
} from "./engine/pull.js";
export {
classifyRenameMoves,
@@ -86,7 +87,7 @@ export {
LOCAL_AUTHOR_NAME,
LOCAL_AUTHOR_EMAIL,
LOCAL_SOURCE_TRAILER,
} from "./engine/push";
} from "./engine/push.js";
export type {
CreateAction,
UpdateAction,
@@ -106,18 +107,18 @@ export type {
PushDeps,
PushRunResult,
PushParsedArgs,
} from "./engine/push";
} from "./engine/push.js";
export { parseSettings, envSchema } from "./engine/settings";
export type { Settings } from "./engine/settings";
export { parseSettings, envSchema } from "./engine/settings.js";
export type { Settings } from "./engine/settings.js";
export { loadSettingsOrExit } from "./engine/config-errors";
export { loadSettingsOrExit } from "./engine/config-errors.js";
export { runCycle } from "./engine/cycle";
export { runCycle } from "./engine/cycle.js";
export type {
RunCycleDeps,
RunCycleResult,
CycleFs,
} from "./engine/cycle";
} from "./engine/cycle.js";
export { parsePageFile, serializePageFile } from "./lib/page-file";
export { parsePageFile, serializePageFile } from "./lib/page-file.js";

View File

@@ -1,9 +1,7 @@
/**
* docmost-sync ADDITION (not present in docmost-mcp).
*
* Semantic canonicalization of ProseMirror/TipTap documents for the Phase-0
* round-trip idempotency check (SPEC §11, "Задача №0", option (б): compare a
* CANONICALIZED form rather than raw bytes).
* Semantic canonicalization of ProseMirror/TipTap documents for the round-trip
* idempotency check (SPEC §11, "Задача №0", option (б): compare a CANONICALIZED
* form rather than raw bytes).
*
* `markdownToProseMirror` reconstructs schema DEFAULT attributes (e.g.
* `indent: null` where the source omitted it) and regenerates per-block ids on
@@ -12,8 +10,7 @@
* normalizes a document so that two semantically-equal docs compare deep-equal
* regardless of block ids and absent-vs-explicit-default-null attributes.
*
* This file is intentionally a NEW, self-contained module so it is trivial to
* backport into docmost-mcp without touching existing code.
* It is a self-contained module with no external dependencies.
*/
/**

View File

@@ -21,7 +21,7 @@ import { getSchema } from "@tiptap/core";
import { Node } from "@tiptap/pm/model";
import { ChangeSet, simplifyChanges } from "@tiptap/pm/changeset";
import { recreateTransform } from "@fellow/prosemirror-recreate-transform";
import { docmostExtensions } from "./docmost-schema";
import { docmostExtensions } from "./docmost-schema.js";
/** A single inserted/deleted change with its containing-block context. */
export interface DiffChange {

View File

@@ -1,28 +1,26 @@
/**
* Public surface of the vendored pure converter (the `lib/` half of the
* docmost-sync `docmost-client` package). This barrel re-exports only the
* Public surface of the pure converter (`lib/`). This barrel re-exports the
* PURE, IO-free pieces the sync engine needs: the self-contained markdown
* (de)serializers, the lossless ProseMirror <-> Markdown converter, the
* markdown -> ProseMirror import path, and semantic canonicalization for the
* round-trip idempotency check (SPEC §11).
*
* The REST client, websocket/collab write-path, auth-utils and page-lock from
* the upstream package are deliberately NOT vendored (the gitmost server writes
* natively).
* There is no REST client, websocket/collab write-path, auth-utils or page-lock
* here — the gitmost server writes natively.
*/
export {
serializeDocmostMarkdown,
parseDocmostMarkdown,
serializeDocmostMarkdownBody,
} from "./markdown-document";
export type { DocmostMdMeta } from "./markdown-document";
} from "./markdown-document.js";
export type { DocmostMdMeta } from "./markdown-document.js";
export { convertProseMirrorToMarkdown } from "./markdown-converter";
export { convertProseMirrorToMarkdown } from "./markdown-converter.js";
export { markdownToProseMirror } from "./markdown-to-prosemirror";
export { markdownToProseMirror } from "./markdown-to-prosemirror.js";
export {
canonicalizeContent,
docsCanonicallyEqual,
} from "./canonicalize";
export { parsePageFile, serializePageFile } from "./page-file";
} from "./canonicalize.js";
export { parsePageFile, serializePageFile } from "./page-file.js";

View File

@@ -135,11 +135,9 @@ export function parseDocmostMarkdown(full: string): {
return { meta, body, comments };
}
// --- docmost-sync addition (backport target: docmost-mcp/src/lib/markdown-document.ts) ---
/**
* Serialize a self-contained markdown file with the meta block + body ONLY —
* NO trailing `docmost:comments` block. The docmost-sync engine never touches
* NO trailing `docmost:comments` block. The sync engine never touches
* `/comments` (SPEC §3): the synced file carries just page identity (meta) and
* the body, where comment threads survive only as inline `<span
* data-comment-id>` anchor marks inside the body.

View File

@@ -1,67 +1,16 @@
/**
* Pure markdown -> ProseMirror conversion (extracted from docmost-sync's
* `packages/docmost-client/src/lib/collaboration.ts`).
* Pure markdown -> ProseMirror conversion.
*
* Only the PURE converter path is vendored here: `markdownToProseMirror`
* (marked -> HTML -> generateJSON) plus the two pre/post processors it needs
* (`preprocessCallouts`, `bridgeTaskLists`). The collaboration/websocket
* write-path (Hocuspocus, Yjs, `ws`, `withPageLock`, `sanitizeForYjs`) that
* lives in the same upstream file is intentionally NOT vendored — the gitmost
* server writes page bodies natively through the collab gateway.
* The converter path is `markdownToProseMirror` (marked -> HTML ->
* generateJSON) plus the two pre/post processors it needs (`preprocessCallouts`,
* `bridgeTaskLists`). The gitmost server writes the resulting page bodies
* natively through the collab gateway, so no websocket/Yjs write-path lives
* here.
*/
import { generateJSON } from "@tiptap/html";
import { JSDOM } from "jsdom";
import { docmostExtensions } from "./docmost-schema";
/**
* Structural type for the bits of the `marked` ESM module we use: just the
* `marked` named export's `parse` method (markdown -> HTML string).
*/
interface MarkedModule {
marked: { parse(markdown: string): string | Promise<string> };
}
// `marked` is ESM-only. Under this package's CommonJS build TS would otherwise
// downlevel a literal `import()` to `require()`, which cannot load an ESM-only
// module. Indirect through `Function` so the real dynamic `import()` survives
// compilation and loads ESM from CommonJS at runtime in Node (same trick as
// apps/server/src/core/ai-chat/tools/docmost-client.loader.ts).
const esmImport = new Function(
"specifier",
"return import(specifier)",
) as (specifier: string) => Promise<unknown>;
// Memoize the in-flight/loaded module so the dynamic import runs at most once.
let markedPromise: Promise<MarkedModule> | null = null;
/**
* Lazily load the ESM-only `marked` module (cached).
*
* In the built CommonJS package (Node, jest with ts-jest) the `esmImport`
* Function trick performs a real dynamic `import()` of the ESM module. Under
* vitest, however, the transformed module is evaluated without a dynamic-import
* callback, so `new Function('return import(...)')` throws "A dynamic import
* callback was not specified"; there `require('marked')` succeeds because the
* test runner's loader interops ESM. We therefore try the Function import first
* and fall back to `require` so BOTH runtimes resolve `marked` transparently.
*/
async function loadMarked(): Promise<MarkedModule["marked"]> {
if (!markedPromise) {
markedPromise = (esmImport("marked") as Promise<MarkedModule>)
.catch(() => {
// Function-trick import is unavailable (e.g. under vitest's evaluator):
// fall back to require, which the test runner can interop for ESM.
// eslint-disable-next-line @typescript-eslint/no-var-requires
return require("marked") as MarkedModule;
})
.catch((err) => {
// Do not cache a rejected import — allow the next call to retry.
markedPromise = null;
throw err;
});
}
return (await markedPromise).marked;
}
import { marked } from "marked";
import { docmostExtensions } from "./docmost-schema.js";
// Setup DOM environment for Tiptap HTML parsing in Node.js
const dom = new JSDOM("<!DOCTYPE html><html><body></body></html>");
@@ -110,8 +59,6 @@ async function preprocessCallouts(markdown: string): Promise<string> {
return markdown;
}
const marked = await loadMarked();
// Recursively transform a slice of lines, converting top-level callouts in
// that slice into <div> blocks and rendering their inner content (which may
// itself contain nested callouts) through this same function.
@@ -379,7 +326,6 @@ function stripEmptyParagraphs(node: any): any {
export async function markdownToProseMirror(
markdownContent: string,
): Promise<any> {
const marked = await loadMarked();
const withCallouts = await preprocessCallouts(markdownContent);
const html = await marked.parse(withCallouts);
const bridged = bridgeTaskLists(html);