Commit Graph

12 Commits

Author SHA1 Message Date
vvzvlad
2f7c0649bb feat(sync): FS->Docmost push #3 — move/rename apply (§5 path-as-truth)
Complete the push action coverage (create/update/delete/move/rename/noop).

- push.ts classifyRenameMoves (pure): the file PATH is the source of truth for
  tree position (§5) — new parent resolved from the enclosing folder's <dir>.md
  page, not the stale meta.parentPageId. Emit move iff parent changed, rename iff
  meta.title changed; a pure path-only rename is a NOOP (no Docmost call — the
  path is local, identity is pageId)
- applyPushActions: move (move_page, reparent) THEN rename (rename_page); noop
  records and calls nothing; per-page isolation + refs-only-on-success preserved
- resolveParentPageId reads <dir>.md meta via readFile (current) /
  git.showFileAtRef(last-pushed) (prev), matching buildVaultLayout
- review fixes: prefetch wrapped in per-page try/catch so a tree-read throw
  isolates one page (§12), not the batch; failures.kind attributes the op that
  actually threw (rename-after-move -> "rename")
- tests (+13): classifier (move/rename/both/noop/to-root), apply (calls/no-calls,
  ordering, isolation); 724 -> 737 green (x2 stable); corpus STABLE

Deferred (final increment): live main() daemon, FS-watcher/debounce (§7.1),
git-remote push (§7.2), pull-side bodyHash/updatedAt consumption, fractional-index
position, escalate-on-divergent-docmost.
2026-06-20 18:52:54 +03:00
vvzvlad
2d13e5ca15 feat(sync): FS->Docmost push #2 — loop-close (§6.3/§10) + fix flaky property timeout
- git.ts: fastForwardBranch(branch, toCommit) — advances ONLY on a true
  fast-forward (merge-base --is-ancestor), refuses a non-ff without clobbering
  divergent docmost history
- push.ts: after a CLEAN push (failures===0) advance both refs/docmost/last-pushed
  AND fast-forward the docmost mirror, so the next pull sees no diff for pushed
  pages (loop-guard, git-native); a partial push advances NEITHER (§12)
- push.ts: per-page error isolation (one bad page doesn't block the batch,
  failures recorded); create requires a non-empty spaceId else skipped (§8 spirit)
- loop-guard.ts: bodyHash() (sha256) + per-page pushed:[{pageId,updatedAt?,bodyHash}]
  record for the §10 self-write suppression (pull-side consumption deferred)
- test: markdown-roundtrip property tests get a 30s per-test timeout (deterministic
  inputs via fixed seed; the only flakiness was wall-clock under parallel load,
  which intermittently failed CI/docker)
- 709 -> 724 green (3x stable); build clean; corpus STABLE

Deferred (next/final increment): move/rename apply, pull-side loop-guard consumption,
FS-watcher/debounce (§7.1), git-remote push (§7.2), runnable live main(),
escalate-on-divergent-docmost.
2026-06-20 17:10:09 +03:00
vvzvlad
9c6283aa8e feat(sync): FS->Docmost push #1 — diff/ref primitives + pure planner + apply (fakes)
First slice of the push direction (SPEC §6), mirroring pull: VaultGit primitives +
pure planner + thin injectable apply, exercised via fakes (no live destructive run).

- git.ts: diffNameStatus (--name-status -M -z, NUL-parsed, rename-aware),
  revParse/readRef/updateRef (refs/docmost/last-pushed), showFileAtRef (recover a
  deleted file's pre-image pageId)
- push.ts computePushActions (pure): A/M/D/R -> create/update/delete/renamesMoves;
  delete only when pageId is recovered from the pre-image, else skipped (§8 guard —
  no spurious Docmost delete)
- push.ts applyPushActions (fakes): update via importPageMarkdown (collab/Yjs path,
  §2 — never a raw jsonb overwrite); create via createPage then write the assigned
  pageId back into the file meta (body preserved); delete via deletePage (soft, §8);
  renamesMoves deferred; advances last-pushed
- tests (+26): diffNameStatus A/M/D/rename, ref round-trip, showFileAtRef; pure
  classification incl. §8 no-pageid skip; apply with fakes (collab-path update,
  pageid write-back, soft-delete, deferred moves)
- 683 -> 709 green; build clean; corpus STABLE

Deferred (next increment): move/rename apply, loop-guard (§10), watcher/debounce,
remote push, live main wiring, empty-spaceId create guard, per-page error isolation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 02:32:15 +03:00
vvzvlad
1750058503 refactor(sync): testability seams for pull + collab; integration tests
Behavior-preserving refactors (R-Collab-1, R-Pull-1, R-Pull-2) to unblock testing,
plus the integration tests they enable.

- collaboration: extract applyTransformToYdoc from onSynced; onSynced stays
  synchronous (NO await between Yjs read and write — SPEC §2 atomicity preserved)
- pull: readExisting(deps) injectable IO; split main into pure computePullActions
  (plan + suppression/mass-delete decisions) + thin applyPullActions(deps) (IO);
  ordering and data-loss guards preserved bit-for-bit
- tests (+35): collaboration-apply (atomicity/null-abort/throw-no-partial),
  read-existing, compute/apply-pull-actions (move-write-fail keeps old path),
  git temp-repo 3-way non-FF merge
- transforms-extra property: constrain the generator to mutually-non-substring
  words (the domain where the renumber property holds) -> deterministic; document
  the inherited commentsToFootnotes substring-overlap comment-drop via it.fails
  (off the sync path, SPEC §3; backport-fix lives in docmost-mcp)
- 695 -> 731 green; build clean; corpus STABLE
2026-06-17 01:29:49 +03:00
vvzvlad
d9d8538846 test(sync): implement test-strategy Phase 1-2 (pure unit/golden/property), +102 tests
Work through test-strategy-report.md, high-ROI no-refactor subset (no regen).

- R-Infra: vitest resolve.alias docmost-client -> packages/docmost-client/src
  (fixes the dist-vs-src coverage artifact: canonicalize 0% -> real)
- R-Cfg-1: export parseArgs + tests
- canonicalize: align family / comment.resolved kept / link non-default +
  fixpoint & docsCanonicallyEqual reflexive/symmetric properties (0 -> 100%)
- markdown-converter golden matrix: columns/embed/audio/pdf, drawio data-align
  rule, inline-mark matrix, textAlign, escaping idempotence, table sanitization
  (61 -> 79%)
- schema parse-closures via generateJSON (TextStyle/comment/mention/Highlight/Column)
- node-ops (immutability, table edge cases, makeFreshId property), transforms
  (setCalloutRange/insertMarkerAfter/commentsToFootnotes + renumber property)
- stabilize normalize-on-write fixpoint (0 -> 100%); diff coarse-fallback;
  client-utils; firstDivergence; corpus fixtures details/columns/mention
- 593 -> 695 green; build clean; corpus STABLE

Deferred (Phase 3-4, refactor-gated): pull/collab/client-REST/git-merge integration.
2026-06-17 01:01:26 +03:00
vvzvlad
f68168e3c1 refactor(sync): unify git exec layer; fix non-ASCII paths + diagnostics (review)
Address a code review of the git-hardening changes.

- single runRaw primitive: every git invocation funnels through it; run() is a
  thin throw+trim wrapper; the two direct execFileAsync bypasses (commitRaw,
  assertGitAvailable) removed; one unified error format
- `-c core.quotepath=false` is now the argv baseline for ALL commands (was only
  listTrackedFiles) — removes the latent quoting asymmetry on ls-files -u /
  diff --name-only; persisted LOCAL config (autocrlf/safecrlf/gpgsign/
  attributesFile) kept as-is in ensureRepo
- preserve spawn-error message (ENOENT): use `||` not `??` (promisified execFile
  sets stderr to "" on spawn failure)
- contextual error when pinning vault git config; module/vaultGitEnv docs corrected
- README: require a system git binary on PATH for local runs
- tests: --no-verify honored (failing pre-commit hook), vaultGitEnv pins,
  core.attributesFile=/dev/null neutralization (593 green)
2026-06-17 00:32:37 +03:00
vvzvlad
ec0a3d47c7 fix(sync): robust git coupling — non-ASCII paths, config neutralization, runtime git
Address git-integration fragility (output is not parsed for control flow; we rely
on exit codes + plumbing — but porcelain BEHAVIOR is config-sensitive, and the
runtime image lacked git).

- listTrackedFiles: `git -c core.quotepath=false ls-files -z` + NUL split — fixes
  Cyrillic/UTF-8 vault filenames being returned octal-escaped/quoted
- Dockerfile: install git (node:22-slim ships none; the daemon shells out at runtime)
- VaultGit env: LC_ALL=C/LANG=C, GIT_PAGER=cat, GIT_TERMINAL_PROMPT=0; keep
  stripping GIT_DIR/GIT_WORK_TREE (cwd-isolation, §12)
- ensureRepo local config: core.autocrlf=false + core.safecrlf=false (protect §11
  byte-stability from a global autocrlf=true), commit.gpgsign=false, and
  core.attributesFile=/dev/null (neutralize a global clean/smudge filter that
  would rewrite the stored blob); commit uses --no-verify (skip injected hooks)
- assertGitAvailable() preflight: clear error if the git binary is missing
- tests: Cyrillic listTrackedFiles, LF byte-preservation of the stored blob,
  local-config neutralization incl. attributesFile (590+ green)
2026-06-17 00:15:17 +03:00
vvzvlad
531b320776 feat(sync): add git vault layer (§5) and the Docmost->vault pull cycle (§6)
Turn the read-only mirror into a git-backed pull cycle. Read-only toward Docmost.

- git.ts (VaultGit): system-git wrapper, all ops cwd=vaultPath (vault is its own
  repo under data/vault, never the source repo); ensureRepo/branches main+docmost,
  commit with provenance (author/committer identity + Docmost-Sync-Source trailer,
  §7.3), merge with conflict surfacing (no auto-resolve, §9), isMergeInProgress;
  GIT_DIR/GIT_WORK_TREE stripped from env (§12 cwd isolation)
- stabilize.ts: normalize-on-write (one export->import->export fixpoint pass, §11)
- reconcile.ts: pure planReconciliation (add/update/move/delete by pageId) +
  decideAbsenceDeletions gate
- pull.ts: write/commit on docmost -> merge into main; listSpaceTree completeness
  signal suppresses absence-deletions on a partial fetch (§8); mass-delete guard;
  merge-in-progress guard makes re-runs converge (§12); move old-path removal only
  on successful write
- docmost-client: listSpaceTree({pages, complete}) without touching the 1:1-copied
  enumerateSpacePages
- tests: reconcile planner + decideAbsenceDeletions, VaultGit incl. real temp-repo
  merge conflict, listSpaceTree completeness (586 green)

Push to a git remote and the FS->Docmost direction are deferred to the next increment.
2026-06-16 23:57:50 +03:00
vvzvlad
4b34f4d30a feat(sync): resolve §11 idempotency via canonical comparison + corpus harness
Close Задача №0 (SPEC §11) with the spec-sanctioned option (b): compare a
canonicalized ProseMirror form instead of raw bytes.

- canonicalize.ts: canonicalizeContent/docsCanonicallyEqual — strip node attrs.id,
  drop null/undefined attrs, and drop attrs equal to their type's known non-null
  schema default (KNOWN_DEFAULTS: link target/rel, comment.resolved, orderedList.start,
  diagram/media align) so "absent" ≡ "default"; comment anchors + meaningful attrs kept
- roundtrip.ts: assert markdown byte-stability AND canonical stability; add --corpus
  mode and mutually-exclusive-flag warning
- synthetic corpus (headings, marks, lists, table, callout, code w/ trailing \n,
  diagrams, textStyle/mention) + canonicalize/corpus tests (558 green)
- known converter asymmetries (block image after paragraph; embed width/height
  coercion) converge to a fixpoint after one export->import pass -> handled by
  normalize-on-write at vault-write time; isolated under it.fails
- SPEC §11: record the resolution and normalize-on-write strategy
2026-06-16 23:23:32 +03:00
vvzvlad
c6edd73324 refactor(pull): extract tested vault-layout module; harden pull; close review findings
Address the Increment-1 code review (3 warnings + suggestions).

- layout: new pure src/layout.ts (buildVaultLayout) — page-tree -> vault paths,
  sibling + full-path collision disambiguation (sanitized ~slugId suffix), parent
  cycle guard; pull.ts is now a thin I/O loop
- layout: resolve orphan/root collisions at the NAME stage so an orphan ancestor
  can't desync its children's folder segments (fixes review Major); covered by test
- pull: per-page try/catch (one bad page no longer aborts the mirror), bounded
  concurrency (6), progress logging, process.exitCode=1 on partial mirror
- security: filename disambiguation suffix now passes through sanitizeTitle
- docs: AGENTS.md -> Increment 1 status/structure/run targets; pull.ts meta-block
  comment; collectRecentSince JSDoc (lexicographic UTC-ISO precondition)
- tests: layout (9), markdown-document round-trip (no comments block, SPEC §3),
  firstDivergence; export firstDivergence. 49 tests green.
2026-06-16 21:09:40 +03:00
vvzvlad
447d2508ae feat(sync): scaffold monorepo, extract docmost-client, add Phase-0 harness + read-only pull
Lock the access-layer decision (REST only) and start implementation per SPEC.

- monorepo (npm workspaces): packages/docmost-client = DocmostClient + lib/*
  copied 1:1 from docmost-mcp/src (backport target), plus bannered sync methods
  (listTrash, restorePage, listAllSpacePages, exportPageBody, listRecentSince /
  collectRecentSince cursor scan)
- engine stays the root app per AGENTS.md (src/, test/, build/, data/, settings.ts);
  add roundtrip.ts (SPEC §11 idempotency harness), pull.ts (SPEC §6 read-only
  Docmost->FS mirror), sanitize.ts (SPEC §12 filenames, path-traversal-safe)
- Dockerfile builds the workspace lib before the app; vitest gates CI
- exportPageBody never touches /comments (SPEC §3); serializeDocmostMarkdownBody
  emits meta + body only
- SPEC: resolve access-layer (REST), reflect root-engine layout + REST pagination
- tests: sanitize (incl. dot-traversal), collectRecentSince (cutoff/dedup/cap),
  stripBlockIds, markdown round-trip byte-stability

Note: raw ProseMirror round-trip is byte-stable in Markdown but not yet attribute-
idempotent (SPEC §11 Задача №0, before Phase 2).
2026-06-16 20:20:20 +03:00
vvzvlad
ef223e13ff chore(scaffold): bootstrap docmost-sync Node/TS project skeleton
Set up the project structure per the new-project guide, adapted from the
Python skeleton to the Node/TS stack fixed in SPEC.md (reuses docmost-mcp).
Scaffold only — the sync engine is not implemented yet.

- src/settings.ts: single config layer on zod, schema keyed by real ENV
  names; credentials and own-service address have no default (fail fast).
- src/config-errors.ts: loadSettingsOrExit — clear startup message naming
  the missing/invalid env var instead of a raw stack trace; exit(1).
- src/index.ts: thin entry point that validates config and logs (stub).
- test/: vitest unit tests for settings parsing and config errors (10 tests).
- Makefile (install/env/build/test/run/dev/clean), strict tsconfig, vitest.
- Dockerfile (single-stage, no EXPOSE, prunes dev deps), docker-compose
  (daemon, volume on /app/data, watchtower), ghcr CI with build needs test.
- .env.example, .gitignore/.dockerignore, AGENTS.md, README.md.
- Pinned deps (dotenv, zod) + committed package-lock.json.
2026-06-16 18:54:29 +03:00