vvzvlad/gitmost
1
0
Fork 0
Code Issues 15 Pull Requests 12 Packages Projects Releases Activity

refactor(git-sync): move the PULL->PUSH cycle into the engine as runCycle (PR #119 review, arch #1)

Browse Source 
The reconcile choreography (ensureRepo -> merge-check -> ensureBranch ->
checkout('docmost') -> pull -> push) was hand-rolled in the app orchestrator's
driveCycle, duplicating an order the vendored engine owns and could drift from on
upgrade — the failure mode is data clobber. Lift it into @docmost/git-sync as a
single entry point, `runCycle(deps)`. The orchestrator now calls runCycle and
keeps only the lock (its caller) and the gitmost-specific delete-cap POLICY,
injected as the `resolveApplyClient` hook (the engine does the dry-run, hands the
hook the planned delete count — Infinity if planning failed — and uses whatever
client it returns for the apply). driveCycle drops from ~150 lines to ~30.

Tests:
- engine test/cycle.test.ts: composition (merge-in-progress short-circuit;
  ensureRepo->ensureBranch->checkout staging order before the pull; the cap hook
  is consulted with the planned count; no dry-run when no hook).
- engine test/cycle-roundtrip.test.ts: runCycle against a REAL VaultGit in a temp
  repo with a faked Docmost client — a git-originated CREATE flows pull->push and
  the assigned pageId is written back; an unresolved merge short-circuits before
  any client call.
- orchestrator spec rewired to mock runCycle and assert the wiring + the
  resolveApplyClient cap policy (the engine-internal cycle-order/merge tests moved
  to the engine).

Validated end to end on a live stand (real Postgres/Redis + server): a git clone
-> edit -> push over the /git remote round-trips the change into the Docmost page
through the refactored cycle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
claude code agent 227
2026-06-24 02:08:38 +03:00
parent 3b334d9624
commit b0fc49cf9d
6 changed files with 580 additions and 249 deletions
Download Patch File Download Diff File Expand all files Collapse all files

198
apps/server/src/integrations/git-sync/services/git-sync.orchestrator.spec.ts
View File

@@ -1,26 +1,19 @@
// Unit tests for the git-sync control plane. The vendored
// engine (@docmost/git-sync) is fully mocked so we exercise ONLY the
// orchestrator's wiring: gating, the Redis leader lock + in-process mutex,
// the pull/push call order, the delete-cap anti-data-loss guard, the remote
// template substitution, and the idempotent interval lifecycle.
// Unit tests for the git-sync control plane. The vendored engine's `runCycle`
// (which owns the PULL->PUSH branch choreography) is mocked so we exercise ONLY
// the orchestrator's wiring: gating, the Redis leader lock + in-process mutex
// (via SpaceLockService), the delete-cap POLICY it injects as `resolveApplyClient`,
// the remote-template substitution in the settings it hands the engine, the
// external-push ingest, and the idempotent interval lifecycle. The cycle
// mechanics themselves are covered by the engine's own cycle round-trip spec.
//
// The engine mock must be declared before importing the orchestrator so the
// module-graph import binds to the mocked functions (same idiom as the
// datasource spec's top-of-file jest.mock stubs that avoid the React graph).
// module-graph import binds to the mocked function.
jest.mock('@docmost/git-sync', () => ({
readExisting: jest.fn(),
computePullActions: jest.fn(),
applyPullActions: jest.fn(),
runPush: jest.fn(),
runCycle: jest.fn(),
}));
import { Logger } from '@nestjs/common';
import {
readExisting,
computePullActions,
applyPullActions,
runPush,
} from '@docmost/git-sync';
import { runCycle } from '@docmost/git-sync';
import {
GitSyncOrchestrator,
GitSyncLockHeldError,
@@ -29,10 +22,14 @@ import { SpaceLockService } from './space-lock.service';
type AnyMock = jest.Mock;
const readExistingMock = readExisting as unknown as AnyMock;
const computePullActionsMock = computePullActions as unknown as AnyMock;
const applyPullActionsMock = applyPullActions as unknown as AnyMock;
const runPushMock = runPush as unknown as AnyMock;
const runCycleMock = runCycle as unknown as AnyMock;
/** The default happy-path cycle result the engine returns. */
const OK_CYCLE = {
ran: true,
pull: { written: 0, deleted: 0, conflict: false },
push: { mode: 'apply', failures: 0 },
};
interface BuildOptions {
/** Env tunables (only the load-bearing ones are surfaced as overrides). */
@@ -150,16 +147,18 @@ function build(opts: BuildOptions = {}): Built {
};
}
/** Reasonable engine defaults so a happy-path driveCycle completes. */
/** The engine runs a clean cycle by default. */
function primeEngineHappyPath(): void {
readExistingMock.mockResolvedValue({});
computePullActionsMock.mockReturnValue({ creates: [], updates: [], deletes: [] });
applyPullActionsMock.mockResolvedValue({
written: 0,
deleted: 0,
merge: { conflict: false },
});
runPushMock.mockResolvedValue({ mode: 'apply', failures: [], planned: { deletes: 0 } });
runCycleMock.mockResolvedValue(OK_CYCLE);
}
/** Pull the `resolveApplyClient` hook out of the (single) runCycle call args. */
function lastResolveApplyClient(): (
plannedDeletes: number,
client: unknown,
) => unknown {
const calls = runCycleMock.mock.calls;
return calls[calls.length - 1][0].resolveApplyClient;
}
beforeEach(() => {
@@ -242,13 +241,10 @@ describe('GitSyncOrchestrator', () => {
});
describe('poisoned-space protection', () => {
it('releases the lock and clears the mutex when driveCycle throws, returning { error }', async () => {
it('releases the lock and clears the mutex when the cycle throws, returning { error }', async () => {
const built = build();
jest.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
// Make the real apply runPush reject; dry-run still resolves first.
runPushMock
.mockResolvedValueOnce({ mode: 'apply', failures: [], planned: { deletes: 0 } })
.mockRejectedValueOnce(new Error('boom'));
runCycleMock.mockRejectedValueOnce(new Error('boom'));
const res = await built.orchestrator.runOnce('space-1', 'ws-1');
expect(res.ran).toBe(false);
@@ -257,16 +253,34 @@ describe('GitSyncOrchestrator', () => {
expect(built.redis.eval).toHaveBeenCalledTimes(1);
// A subsequent call can re-acquire (mutex cleared after the throw).
runPushMock.mockResolvedValue({ mode: 'apply', failures: [], planned: { deletes: 0 } });
runCycleMock.mockResolvedValue(OK_CYCLE);
const res2 = await built.orchestrator.runOnce('space-1', 'ws-1');
expect(res2.ran).toBe(true);
});
});
describe('merge-in-progress guard', () => {
it("returns skipped:'merge-in-progress' and runs no pull/push", async () => {
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
const built = build({ vaultOverrides: { isMergeInProgress: jest.fn(async () => true) } });
describe('cycle wiring', () => {
it('drives runCycle with the space vault, the bound client, and settings', async () => {
const built = build();
await built.orchestrator.runOnce('space-1', 'ws-1');
expect(runCycleMock).toHaveBeenCalledTimes(1);
const [deps] = runCycleMock.mock.calls[0];
expect(deps.spaceId).toBe('space-1');
expect(deps.vault).toBe(built.vault);
expect(deps.client).toBe(built.client);
expect(deps.settings.vaultPath).toBe('/vaults/space-1');
expect(typeof deps.resolveApplyClient).toBe('function');
// The bound datasource identity is the (workspace, service-user) pair.
expect(built.dataSource.bind).toHaveBeenCalledWith({
workspaceId: 'ws-1',
userId: 'svc-user',
});
});
it("surfaces the engine's skipped status (e.g. merge-in-progress) verbatim", async () => {
const built = build();
runCycleMock.mockResolvedValue({ ran: false, skipped: 'merge-in-progress' });
const res = await built.orchestrator.runOnce('space-1', 'ws-1');
expect(res).toEqual({
@@ -274,40 +288,6 @@ describe('GitSyncOrchestrator', () => {
ran: false,
skipped: 'merge-in-progress',
});
expect(applyPullActionsMock).not.toHaveBeenCalled();
expect(runPushMock).not.toHaveBeenCalled();
});
});
describe('cycle order', () => {
it('runs ensureRepo -> ensureBranch(docmost,main) -> checkout(docmost) -> applyPullActions in order', async () => {
const order: string[] = [];
const built = build({
vaultOverrides: {
ensureRepo: jest.fn(async () => {
order.push('ensureRepo');
}),
ensureBranch: jest.fn(async (branch: string, base: string) => {
order.push(`ensureBranch:${branch}:${base}`);
}),
checkout: jest.fn(async (branch: string) => {
order.push(`checkout:${branch}`);
}),
},
});
applyPullActionsMock.mockImplementation(async () => {
order.push('applyPullActions');
return { written: 0, deleted: 0, merge: { conflict: false } };
});
await built.orchestrator.runOnce('space-1', 'ws-1');
expect(order).toEqual([
'ensureRepo',
'ensureBranch:docmost:main',
'checkout:docmost',
'applyPullActions',
]);
});
});
@@ -315,9 +295,9 @@ describe('GitSyncOrchestrator', () => {
it('streams the receive-pack FIRST, then runs the Docmost cycle', async () => {
const order: string[] = [];
const built = build();
applyPullActionsMock.mockImplementation(async () => {
runCycleMock.mockImplementation(async () => {
order.push('cycle');
return { written: 0, deleted: 0, merge: { conflict: false } };
return OK_CYCLE;
});
const runReceivePack = jest.fn(async () => {
order.push('receive-pack');
@@ -341,17 +321,14 @@ describe('GitSyncOrchestrator', () => {
// We must never write to the working tree concurrently with a cycle.
expect(runReceivePack).not.toHaveBeenCalled();
expect(applyPullActionsMock).not.toHaveBeenCalled();
expect(runCycleMock).not.toHaveBeenCalled();
});
it('swallows a post-push cycle error (the push is durable; poll retries)', async () => {
jest.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
const built = build();
// Dry-run resolves, the real apply rejects → driveCycle throws AFTER the
// receive-pack already succeeded.
runPushMock
.mockResolvedValueOnce({ mode: 'apply', failures: [], planned: { deletes: 0 } })
.mockRejectedValueOnce(new Error('cycle boom'));
// The cycle throws AFTER the receive-pack already succeeded.
runCycleMock.mockRejectedValueOnce(new Error('cycle boom'));
const runReceivePack = jest.fn(async () => undefined);
// Does NOT throw — the durable push must not be reported as failed.
@@ -373,7 +350,7 @@ describe('GitSyncOrchestrator', () => {
).resolves.toBeUndefined();
// The push is durable on main; the immediate cycle is skipped, not failed.
expect(runReceivePack).toHaveBeenCalledTimes(1);
expect(applyPullActionsMock).not.toHaveBeenCalled();
expect(runCycleMock).not.toHaveBeenCalled();
});
it('refuses (LockHeldError) and runs nothing when git-sync is globally disabled', async () => {
@@ -389,22 +366,16 @@ describe('GitSyncOrchestrator', () => {
});
describe('delete cap (anti-data-loss)', () => {
it('suppresses deletePage on the apply client (by throwing) when planned deletes exceed the cap', async () => {
// The cap is now a POLICY the orchestrator injects as runCycle's
// `resolveApplyClient` hook; the engine calls it with the dry-run's planned
// delete count. We pull the hook out of the runCycle args and exercise it.
it('suppresses deletePage (by throwing) when planned deletes exceed the cap', async () => {
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
const built = build({ maxDeletes: 5 });
// Dry-run plans 9 deletes (over the cap of 5); apply still runs.
runPushMock
.mockResolvedValueOnce({ mode: 'plan', failures: [], planned: { deletes: 9 } })
.mockResolvedValueOnce({ mode: 'apply', failures: [], planned: { deletes: 0 } });
await built.orchestrator.runOnce('space-1', 'ws-1');
const res = await built.orchestrator.runOnce('space-1', 'ws-1');
expect(res.ran).toBe(true);
expect(runPushMock).toHaveBeenCalledTimes(2);
// The second runPush (real apply, dryRun:false) got a suppressed client.
const [applyDeps, applyOpts] = runPushMock.mock.calls[1];
expect(applyOpts).toEqual({ dryRun: false });
const applyClient = applyDeps.makeClient();
const resolveApplyClient = lastResolveApplyClient();
const applyClient = resolveApplyClient(9, built.client) as any;
// deletePage is still a function (the engine calls it)...
expect(typeof applyClient.deletePage).toBe('function');
// ...but it THROWS, so the engine records a per-page failure and holds
@@ -416,32 +387,28 @@ describe('GitSyncOrchestrator', () => {
expect(applyClient.createPage).toBe(built.client.createPage);
});
it('fails safe: a throwing dry-run still suppresses deletes and does not throw the cycle', async () => {
it('fails safe: an Infinity planned-delete count (dry-run failed) is suppressed', async () => {
jest.spyOn(Logger.prototype, 'warn').mockImplementation(() => undefined);
const built = build({ maxDeletes: 5 });
runPushMock
.mockRejectedValueOnce(new Error('plan failed'))
.mockResolvedValueOnce({ mode: 'apply', failures: [], planned: { deletes: 0 } });
await built.orchestrator.runOnce('space-1', 'ws-1');
const res = await built.orchestrator.runOnce('space-1', 'ws-1');
// The cycle still completes (ran:true), it does NOT throw.
expect(res.ran).toBe(true);
const [applyDeps] = runPushMock.mock.calls[1];
const applyClient = applyDeps.makeClient();
// Suppressed via throw (same fail-safe path as the over-cap case).
// runCycle passes Number.POSITIVE_INFINITY when its dry-run planning threw;
// Infinity > cap → suppress (same throwing path).
const resolveApplyClient = lastResolveApplyClient();
const applyClient = resolveApplyClient(
Number.POSITIVE_INFINITY,
built.client,
) as any;
await expect(applyClient.deletePage('p1')).rejects.toThrow(/suppress/i);
expect(built.client.deletePage).not.toHaveBeenCalled();
});
it('passes through the original client when planned deletes are within the cap', async () => {
const built = build({ maxDeletes: 5 });
runPushMock
.mockResolvedValueOnce({ mode: 'plan', failures: [], planned: { deletes: 3 } })
.mockResolvedValueOnce({ mode: 'apply', failures: [], planned: { deletes: 0 } });
await built.orchestrator.runOnce('space-1', 'ws-1');
const [applyDeps] = runPushMock.mock.calls[1];
const applyClient = applyDeps.makeClient();
const resolveApplyClient = lastResolveApplyClient();
const applyClient = resolveApplyClient(3, built.client) as any;
// The ORIGINAL client is used (deletePage forwards to the real one).
expect(applyClient).toBe(built.client);
await applyClient.deletePage('p1');
@@ -450,12 +417,11 @@ describe('GitSyncOrchestrator', () => {
});
describe('remote template substitution', () => {
it('substitutes {spaceId} into the gitRemote handed to runPush', async () => {
it('substitutes {spaceId} into the gitRemote settings handed to the engine', async () => {
const built = build({ remoteTemplate: 'git@h:vault-{spaceId}.git' });
await built.orchestrator.runOnce('space-42', 'ws-1');
// Inspect the settings on the dry-run call (first runPush).
const [dryDeps] = runPushMock.mock.calls[0];
expect(dryDeps.settings.gitRemote).toBe('git@h:vault-space-42.git');
const [deps] = runCycleMock.mock.calls[0];
expect(deps.settings.gitRemote).toBe('git@h:vault-space-42.git');
});
});

168
apps/server/src/integrations/git-sync/services/git-sync.orchestrator.ts
View File

@@ -10,13 +10,7 @@ import { dirname } from 'node:path';
import { InjectKysely } from 'nestjs-kysely';
import { KyselyDB } from '@docmost/db/types/kysely.types';
import { sql } from 'kysely';
import {
type Settings,
readExisting,
computePullActions,
applyPullActions,
runPush,
} from '@docmost/git-sync';
import { type Settings, runCycle } from '@docmost/git-sync';
import { EnvironmentService } from '../../environment/environment.service';
import { GitmostDataSourceService } from './gitmost-datasource.service';
import { VaultRegistryService } from './vault-registry.service';
@@ -242,10 +236,11 @@ export class GitSyncOrchestrator implements OnModuleInit, OnModuleDestroy {
}
/**
* The actual engine wiring. Mirrors the engine's own `main`:
* PULL — readExisting -> computePullActions -> applyPullActions,
* PUSH — runPush (dry-run disabled: a real apply).
* The dependency-object shapes match pull.ts/push.ts exactly (see comments).
* Drive ONE reconcile cycle for a space. The PULL->PUSH branch choreography
* lives in the engine's `runCycle` (so it can never drift from the engine it
* ships with); the orchestrator owns only the lock (its caller) and the
* gitmost-specific delete-cap POLICY, injected here as the `resolveApplyClient`
* hook.
*/
private async driveCycle(
spaceId: string,
@@ -254,118 +249,41 @@ export class GitSyncOrchestrator implements OnModuleInit, OnModuleDestroy {
): Promise<GitSyncRunStatus> {
const settings = this.buildSettings(spaceId);
const vault = await this.vaultRegistry.getVault(spaceId);
const vaultRoot = settings.vaultPath;
const client = this.dataSource.bind({ workspaceId, userId: serviceUserId });
const maxDeletes = this.environmentService.getGitSyncMaxDeletesPerCycle();
// Engine state store is git: make sure the repo + branches exist before any
// tracked-file listing or diff (the engine's pull/push assume an inited repo).
await vault.assertGitAvailable();
await vault.ensureRepo();
// Refuse to run on top of an unresolved merge (SPEC §9): a prior
// conflicting pull leaves the vault mid-merge; the next checkout would fail.
if (await vault.isMergeInProgress()) {
this.logger.warn(
`git-sync[${spaceId}]: vault has an unresolved merge — resolve it (or ` +
`'git merge --abort') and re-run (SPEC §9); skipping cycle.`,
);
return { spaceId, ran: false, skipped: 'merge-in-progress' };
}
await vault.ensureBranch('docmost', 'main');
// CRITICAL: pull writes happen on the `docmost` branch — applyPullActions
// commits there, then checks out `main` and merges docmost -> main. We MUST be
// on `docmost` BEFORE applying (mirrors the engine's own pull main()), else the
// Docmost content is written straight onto `main`, clobbering local file edits
// before push can diff them.
await vault.checkout('docmost');
// --- PULL --------------------------------------------
// readExisting deps (ReadExistingDeps): list tracked *.md + read by relPath.
const existing = await readExisting({
listTracked: () => vault.listTrackedFiles('*.md'),
readFile: (relPath) => readFile(`${vaultRoot}/${relPath}`, 'utf8'),
});
const tree = await client.listSpaceTree(spaceId);
const pullActions = computePullActions({
pages: tree.pages,
treeComplete: true,
existing,
});
// applyPullActions deps (ApplyPullActionsDeps): the read-side client subset,
// the vault git subset, and ABSOLUTE-path fs ops (mkdir/writeFile/rm).
const pullResult = await applyPullActions(
{
client,
git: vault,
const result = await runCycle({
spaceId,
client,
vault,
settings,
// ABSOLUTE-path fs primitives the engine cycle injects (it stays IO-free).
fs: {
readFile: (absPath) => readFile(absPath, 'utf8'),
writeFile: (absPath, text) => writeFile(absPath, text, 'utf8'),
mkdir: (absDir) => mkdir(absDir, { recursive: true }).then(() => undefined),
rm: (absPath) => rm(absPath, { force: true }),
},
pullActions,
vaultRoot,
);
// --- PUSH --------------------------------------------------
// runPush deps (PushDeps): settings, the full vault git object (method `this`
// binding must be preserved — pass the object, not bound method refs), a
// makeClient factory returning the push client subset, vault-relative fs
// read/write, and a logger. dryRun:false performs the real Docmost writes.
const pushDeps = {
settings,
git: vault,
makeClient: () => client,
readFile: (relPath: string) => readFile(`${vaultRoot}/${relPath}`, 'utf8'),
writeFile: (relPath: string, text: string) =>
writeFile(`${vaultRoot}/${relPath}`, text, 'utf8'),
log: (line: string) => this.logger.log(`git-sync[${spaceId}] ${line}`),
};
// DEFENSE-IN-DEPTH delete cap. A non-convergent vault
// (e.g. empty/duplicate titles -> colliding paths) can compute PHANTOM
// absence-deletions that slip under the engine's mass-delete FRACTION guard
// and soft-delete real pages. So plan the push as a DRY-RUN FIRST to read the
// delete count, and if it exceeds GIT_SYNC_MAX_DELETES_PER_CYCLE, run the real
// apply with a client whose deletePage is NEUTRALIZED — creates/updates/
// moves/renames still apply, deletions are skipped this cycle. Never throws.
const maxDeletes = this.environmentService.getGitSyncMaxDeletesPerCycle();
let suppressDeletes = false;
try {
const dry = await runPush(pushDeps, { dryRun: true });
const plannedDeletes = dry.planned?.deletes ?? 0;
if (plannedDeletes > maxDeletes) {
suppressDeletes = true;
// DEFENSE-IN-DEPTH delete cap (gitmost-specific policy). A non-convergent
// vault (e.g. empty/duplicate titles -> colliding paths) can compute
// PHANTOM absence-deletions. When the push's planned delete count exceeds
// GIT_SYNC_MAX_DELETES_PER_CYCLE (or planning failed -> Infinity), suppress
// deletes by making deletePage THROW: the engine records each as a per-page
// failure, which keeps `refs/docmost/last-pushed` from advancing past the
// dropped-file commit, so the deletion is RETRIED next cycle rather than
// silently dropped (a no-op that resolved would advance the ref and a pull
// would then recreate the user's deleted files). See PR #119 review.
resolveApplyClient: (plannedDeletes, c) => {
if (plannedDeletes <= maxDeletes) return c;
this.logger.warn(
`git-sync[${spaceId}]: push delete count ${plannedDeletes} exceeds ` +
`GIT_SYNC_MAX_DELETES_PER_CYCLE=${maxDeletes}; skipping deletions this ` +
`cycle (possible non-convergence / collision). Investigate vault layout.`,
`GIT_SYNC_MAX_DELETES_PER_CYCLE=${maxDeletes}; suppressing deletions ` +
`this cycle (possible non-convergence / collision). Investigate vault ` +
`layout.`,
);
}
} catch (err) {
// A failed dry-run plan must not block the apply, but we cannot trust a
// delete count we never got — fail SAFE by suppressing deletes this cycle.
suppressDeletes = true;
this.logger.warn(
`git-sync[${spaceId}]: push dry-run planning failed (${
err instanceof Error ? err.message : String(err)
}); skipping deletions this cycle as a precaution.`,
);
}
// When over the cap, suppress deletes by making deletePage THROW (every
// other op is forwarded). A throw is recorded by the engine as a per-page
// `failure`, which (a) keeps `refs/docmost/last-pushed` from advancing past
// the commit that dropped the files, and (b) makes the next cycle re-diff
// from the un-advanced ref and re-plan the same deletes — so a transient
// over-cap is retried rather than silently dropped forever. (A no-op that
// resolved would let the engine count `deleted++` with no failure, advance
// the ref, and never replay the deletions — a pull would then recreate the
// user's deleted files. See PR #119 review.)
const applyClient = suppressDeletes
? {
...client,
return {
...c,
deletePage: async () => {
throw new Error(
'git-sync: delete suppressed this cycle ' +
@@ -373,27 +291,11 @@ export class GitSyncOrchestrator implements OnModuleInit, OnModuleDestroy {
'so the deletion is retried, not dropped',
);
},
}
: client;
const pushResult = await runPush(
{ ...pushDeps, makeClient: () => applyClient },
{ dryRun: false },
);
return {
spaceId,
ran: true,
pull: {
written: pullResult.written,
deleted: pullResult.deleted,
conflict: pullResult.merge.conflict,
};
},
push: {
mode: pushResult.mode,
failures: pushResult.failures?.length ?? 0,
},
};
});
return { spaceId, ...result };
}
// --- poll-safety interval -------------------------------------