git-workflow — multi-repo-coordination

Detail page of the git-workflow recipe card.

B2 — multi-repo / multi-worktree coordination

One logical change spanning several repos uses the same feat/<slug> in each (main, sdk, plugins, docs, box/<distro>), so the branches correlate. R10 runs against the assembled superproject (submodule pointers at the feat/ commits) — the whole change is verified before any PR is opened. Then land in dependency order, each repo as its OWN two-step PR (author opens; fresh pr-validator merges + tags):

0. the sdk contract repo (github.com/opencharly/sdk — NOT a submodule; this repo consumes it from the module proxy at the pinned require version, so only a touched CONTRACT lands there) — PR → native auto-merge + tag-on-merge → tag v0.<YYYYDDD>.<HHMM leading-zeros-stripped> (its Go-module tag scheme; the superproject vYYYY.DDD.HHMM form is not a valid Go module version — e.g. superproject v2026.185.0751 ⇄ sdk v0.2026185.751) — whenever the cutover touched sdk content;
1. each box/<distro> submodule — PR → auto-merge + tag-on-merge tags (it has charly.yml);
2. plugins — PR → auto-merge + tag-on-merge tags v<YYYY.DDD.HHMM> (no charly.yml, so no schema version: bump — but the tag marks the merge, same as every repo);
3. the superproject — stage the now-MERGED submodule pointers (a touched sdk: adopt the tagged sdk release as the new shared pinned require in every module — task mods:tidy re-syncs go.sum and the canonical-go.mod gate asserts the ONE shared pin; there is no gitlink and no replace) → PR → native auto-merge + tag-on-merge tags main.

A producer PR must be merged (not merely green) before the consumer’s pointer bump — the superproject pointer must reference a commit that is on the submodule’s real main, which only the merge produces.

A valid base is an ASSEMBLED PAIR, not a lone submodule advance. A new consumer cutover branches from a base that is valid only once BOTH halves have merged: an sdk main advance is a valid consumer base ONLY after its superproject adaptation (the tagged sdk release adopted as the new shared pinned require — there is no gitlink) has ALSO merged. Branch a consumer off a bare sdk main advance whose super side is still open and you pin a superproject state no main records — the consumer’s R10 builds against a half-assembled base and its pointer bump references a commit main has never seen. Wait for the pair before treating a producer advance as a base.

Submodule-pointer-bump safety (step 3) — bump AFTER the session worktree exists, then stage AND verify. A git switch / git checkout — and equally the git worktree add that creates a session’s landing branch — re-materializes each submodule at the gitlink the target records, silently discarding an unstaged working-tree pointer bump (it happens even with submodule.recurse unset — an unstaged gitlink is not carried across the switch / worktree-add). So bumping the pointer before creating the session worktree (git worktree add .claude/worktrees/<slug> -b feat/<slug> origin/main, B1 step 0 — the feat branch is created there at session start, never by git switch -c in the shared main tree) — or merely git -C <sub> checkout <new> without git add — drops it from the commit, and a git add <sub>; git commit afterward stages nothing because the working tree was reset to the old pointer. Always, in order: (a) create the session worktree + feat branch FIRST (B1 step 0); (b) THEN git -C <sub> checkout <new-commit> + git add <sub>; (c) VERIFY it is staged — git diff --cached --submodule=short <sub> must print <old>...<new>; (d) after committing, confirm the commit records it — git show --stat lists <sub> and git ls-tree HEAD <sub> shows <new>. A pointer-bump commit whose --stat omits the submodule is the silent-drop failure.

Attribution of the pointer-bump commit — derived from what it points at. When the bumped submodule commit is itself all-documentation (a skill / *.md edit), the superproject pointer-bump commit IS the Documentation-only change class and lands at documentation reviewed: the fresh validator inspects the submodule’s own old..new diff to certify it. A bump that integrates submodule CODE is a code class and takes a runtime tier, the docs riding along. So a docs-only skill cutover lands plugins (the *.md) at documentation reviewed, then the superproject pointer bump at documentation reviewed too — both halves honest.

The derivation stops at the gitlink, and the reason is a general rule: a change class must be decidable FROM THE DIFF. A gitlink is a pointer git itself resolves, so “what it points at” is reachable from the PR — the validator reads the submodule’s own old..new in the repo the PR is against, and the derivation above is certifiable without leaving it. A sha inside a candy var: is not: it is a string, and following it means cloning a remote repository at a named commit. Were “the referenced content is documentation” the test, the class would be a property of a tree the PR does not contain, and no validator could rule on it from the PR — which is why re-pinning DOCS_REF after a docs merge is a CONFIG change carrying its own bed gate (B6a step 4), no matter how documentary the commit it names. Any pin the diff cannot resolve behaves the same way. When the diff cannot decide the class, the class is the heavier one.

For the full multi-worktree end-to-end — the doc-tier git -C literal-path rule, and the mandatory post-landing worktree refresh — see B7.

Per-module verification — verify by MODULE CLASS, and prove the fix is in the BINARY

Two mechanics that bite every cross-repo landing:

Verify each Go module by its CLASS, always with GOWORK=off (the repo is a Go workspace, so module-level checks must disable it or they pull the workspace’s transitive requires):

• sdk is a STANDALONE-CONSUMED contract module (out-of-tree consumers import it directly), so it earns the full standalone battery: GOWORK=off go mod tidy && go mod verify && go build ./... && go test ./.... It MUST tidy and build cleanly on its own.
• charly is a WORKSPACE MEMBER, not a standalone module: verify it with GOWORK=off go mod verify + the WORKSPACE build. A full standalone go mod tidy on charly POLLUTES its go.mod with candy/plugin-* pseudo-requires (they resolve through the workspace, not the module graph), and a standalone go build that fails ONLY on candy/plugin-* imports is an ARCHITECTURAL fact (the plugins are workspace siblings), never a defect to “fix” by hand-editing go.mod.
• plugins candies tidy PER-MODULE — each candy is its own module; tidy/verify them individually, never as one tree.

Prove a fix is in the BUILT BINARY by a content marker, NOT by the version stamp. scripts/calver.sh derives the CalVer from the HEAD commit’s UTC time (TZ=UTC0 git log -1 --format=%cd --date='format-local:%Y %j %H %M' — the TZ=UTC0 is what makes the bare %cd UTC; without it %cd uses the commit’s own TZ offset), so the stamp identifies the SOURCE COMMIT, never the build moment — a task build:binary on a DIRTY working tree reports the IDENTICAL version as the clean commit under it. So charly version matching the expected CalVer does NOT prove your uncommitted fix compiled in. Prove fix-presence by a content marker instead: strings bin/charly | grep '<a string unique to the fix>' (a new error message, flag name, or symbol). The stamp answers “which commit”; the strings marker answers “is my change actually in this binary”.

B3 — agent teams in per-teammate worktrees

When an agent team parallelizes work, each teammate works in its OWN worktree under .claude/worktrees/<slug>/ (B1 step 0) — the same worktree-per-session model every session uses, never a shared checkout. A team is N worktrees, each with its own bin/charly and its own freshness-guard scope; there is no shared-tree alternative. Within a worktree, the check bed is the unit of isolation: each teammate owns a disjoint check bed’s SOURCE files; distinct beds get distinct charly-<bed> container/VM/domain names, and a bed run tags every fixture IMAGE it builds with a per-run <bed-root>-<runCalver> tag (#75) so two beds building the SAME fixture image name never race the store-global tag namespace; host-port disjointness is not statically guaranteed, so every bed uses port auto-allocation — never a hardcoded host port (the loader checks no ports, so a collision surfaces only at deploy; manual port-picking is forbidden), and a bed pins an image → layers → files, so bed-ownership already isolates the source files each teammate edits. Teammates edit; a PERSISTENT owner runs every full charly check run <bed> as a run_in_background task — the lead’s persistent session, a background agent, or (interactive tmux) a split-pane teammate; an in-process teammate CANNOT (its bg dies on yield).

• Teammates edit their bed-scoped files + run short foreground checks (charly check box) — never the full charly check run, and never commit, push, or open a PR. The lead runs the full beds and, on R10 PASS, opens the SINGLE PR for the cutover (B1 step 1); a FRESH pr-validator (never a teammate that authored code) merges it.
• Schedule longest-pole-first. charly check run has no bed-level concurrency and no charly cap — the limit is host CPU/RAM/podman. Run ALL full beds as concurrent background tasks; order by expected DURATION, not bed count: launch the slow VM/desktop beds first and overlap the cheap pod beds, so wall-clock ≈ the slowest single bed, not the sum.
• No shared freeze barrier — per-worktree self-freeze instead. Because each worktree carries its own bin/charly and its own freshness-guard scope, a teammate editing charly/*.go in its own worktree never trips another teammate’s bed run — there is no shared binary to freeze. The only freeze that applies is the WITHIN-worktree self-freeze: freeze your own worktree’s charly/*.go for the duration of your own bed run (the per-tree freshness guard compares the invoked binary against the cwd’s sources at every heavy verb, mid-run), and keep one binary-build owner per worktree at a time. Full discipline: /charly-internals:agents “The charly binary in a multi-teammate / multi-worktree setup” + “Within-worktree self-freeze”.

B6 — cross-repo landing when a change is referenced via @github

The resolver (EnsureRepoDownloaded) fetches a producer repo from the REMOTE at the pinned ref, so a producer change on a local feat/ branch — or an OPEN, unmerged PR — is invisible to a consumer’s R10. The producer PR must be MERGED first. Staged landing:

1. Develop producer (A) + consumer (B) on the same feat/<slug>.
2. Land the producer FIRST: A’s own R10 PASS → open A’s PR → fresh pr-validator validates, merges, and tags A v<CalVer_A> — now an immutable, fetchable remote tag on A’s real main.
3. Repoint the consumer: charly box reconcile rewrites B’s @github.../A:... pins to v<CalVer_A> (see /charly-build:reconcile).
4. Authoritative consumer R10 against the real tag: B’s R10 now fetches A from the pushed v<CalVer_A> — verified against exactly what shipped.
5. Land the consumer: open B’s PR → fresh pr-validator validates, merges, tags.
6. New candy: a new candy has no standalone R10 — its gate is the consuming image’s build. A lands a provisional v<CalVer_A> (layer + go test / charly box generate smoke); step 4 (B’s image R10 against that tag) is the real gate. On failure, fix A, land a new tag (immutable + accumulate — never move the old one), re-reconcile, re-run step 4.

Each repo gets ONE R10 against ITS final code; repos land producer→consumer. Multi-level chains (A→B→C) recurse the same way.

B6a — the documentation landing is the same chain

This project publishes opencharly.ai from GENERATED trees, so a prose edit is a producer→consumer landing exactly like a code one, and the same “producer must be merged first” rule applies. /charly-build:docs owns the projection model; the load-bearing part of it is that the two chains have DIFFERENT lengths — candy and box prose is a ONE-hop projection read straight off each repo’s charly.yml, while skill prose is a TWO-hop projection that charly docs generate reads from the marketplace/ tree, never from candy/*/charly.yml.

The commands below are maintenance THIS REPOSITORY performs on itself. They are not steps an charly user runs, and they never belong on a reader-facing page.

1. Edit the source, never a generated file — the candy’s description: / plan:, or the owning candy’s skill: content.
2. For a skill edit, project and land plugins FIRST (charly marketplace generate, then the plugins PR). Locally the docs generator reads the dirty marketplace/ tree, so a local regeneration picks the edit up immediately; every other reader gets it only once the gitlink advances. Advancing that gitlink WITHOUT regenerating docs in the same landing is precisely what leaves the published site stale, and only the drift gate will say so. Superproject-facing drift is EXPECTED while the projection PR is in flight — the superproject necessarily still pins the old gitlink until the projection merges — so that redness is not the tagging bar for the projection PR itself: its gate is the plugins tree, and the superproject catches up at step 5.
3. Regenerate and land docs — a docs PR that bumps the charly pin in docs/.gitmodules and carries the regenerated pages; the docs repo’s deploy workflow is the gate (it regenerates on every run and fails on any diff). Regeneration rewrites the generated trees WHOLESALE, so a mirror already behind cannot be brought forward selectively: every pending page lands with the next commit or none of them do. Budget for carrying someone else’s backlog when the mirror has drifted.
4. Re-pin the new docs merge in candy/docs-site/charly.yml; task docs:pin is the gate (it compares DOCS_REF against the docs repo’s current main head) and asserts more than one occurrence (/charly-tools:docs-site owns the pin contract). This step edits candy CONFIG rather than prose, so its own gate is the check-docs bed — a documentation-only change class does NOT cover it, and the commit that carries it cannot claim the documentation reviewed tier. Positively: its gate is charly box validate PLUS check-docs — the disposable: true bed that composes the changed entity, through docs-site-app — at fully tested and validated. Watch for it on a landing whose only “extra” is this re-pin: the step silently upgrades the WHOLE cutover’s gate, because DOCS_REF is emitted as an ENV line above the clone step, so re-pinning it changes the emitted Containerfile and therefore the image. The general rule the boundary rests on is in B2, “the derivation stops at the gitlink”.
5. Bump the superproject gitlinks for whichever submodules moved.

Why that order and not any other — the two-directional pin rule. A superproject gitlink and a mirror’s own pin are DIFFERENT relations, and conflating them is what produces an unpinnable mirror:

• Source-covers. The superproject pins a plugins sha that must CONTAIN every source edit producing what the mirror shows. It is a coverage claim: “everything rendered downstream is generated from something at or before this sha.”
• Mirror-reflects. The docs mirror is a PROJECTION of one specific source state. It is an identity claim: “these pages are what that state renders to.”

The two shas are not the same and are not required to be — but the second is only meaningful relative to the first. A mirror leg that accumulates a SECOND merge before its superproject leg pins the first makes the mirror unpinnable by any state the superproject can reach: the mirror becomes a composite of projections from several submodule commits, and no single pin reproduces a composite.

Landing the mirror ahead is not the hazard — it is REQUIRED, because a superproject cannot pin a commit that does not exist yet. The hazard is the GAP, and specifically other legs’ projections landing inside it.

The mechanism, stated because a reader without it reconstructs the unqualified rule and then finds it forbids the procedure below: several legs share ONE submodule pointer. While your mirror merge sits unpinned, every other leg that lands projects into the same tree. The superproject then has no sha that reproduces only your change — the mirror is a composite, and the last leg to pin inherits every projection beneath it. That is not hypothetical: it is how a four-cutover forced union — charly#278 — came to carry four cutovers’ candy sources in one commit: the check-verb resolver, the git-workflow landing lessons, the merge-tree guard, atop the R4a sweep. None of those authors chose to couple their work; no intermediate self-consistent superproject state existed for them to land against.

So: land each mirror leg immediately before its superproject leg, keeping at most one outstanding merge per mirror — not one across all of them. B6 legitimately has plugins and docs both outstanding between steps 2/3 and step 5; that is two mirrors with one merge each, which is fine. What is not fine is two merges in ONE mirror awaiting a single pin. Then nothing intervenes, each mirror contains only your own projection, and the shas coincide naturally with no reconciliation to perform.

Proof that this is a live invariant rather than an aspiration: regenerate at main’s OWN pins and count the pages that move. Zero means every published page is reproducible from the state the superproject currently points at. A non-zero count is the composite above, and it names its own repair — the pages listed are exactly those whose source landed out of order.

B7 — Multi-worktree landing + refresh (the canonical end-to-end)

When this project is driven from multiple git worktrees sharing one .git, only ONE worktree can have main checked out at a time. The main tree stays on main; every feature session works in its OWN worktree under .claude/worktrees/<slug>/ — one worktree per session, created at session start and removed after landing (B1 step 0 + B4 “Worktree hygiene”). Every “land + update all worktrees” follows this EXACT ordered sequence. It composes B1 (branch loop), B2 (per-repo order + pointer-bump safety), B4 (sync/prune).

0. Pre-flight (worktree safety). git worktree list → note which worktree holds main. Pin ONE worktree for the whole edit→commit→push sequence; drive every step with a literal absolute path git -C /abs/path …. NEVER a leading cd+\-continued chain (it scopes every later command into the submodule) and NEVER a shell variable for a path — shell variables do NOT persist between Bash tool calls, so a WT=… set in an earlier call is EMPTY later and git -C "$WT/plugins" silently becomes git -C /plugins (this was a real failure). Verify: git -C /abs rev-parse --show-toplevel == the path you edited AND git -C /abs status --short lists your edits.

1. Sync-before-start. git fetch origin --prune --tags; ff local main to origin/main (B4).

2. Open the PRs in dependency order, same feat/<slug> in every repo (sdk when touched → box submodules → plugins → superproject). Per-repo mechanics = B2 + B1 step 1; pointer-bump safety = B2 step 3. Two proven additions:

• plugins docs commit at documentation reviewed: git -C <LITERAL-abs-plugins> commit …. The literal path keeps repository selection explicit and lets the fresh validator inspect the plugins diff independently. Do NOT use a shell variable that may be unset or an in-command directory change.
• box/ re-stamp (schema-HEAD bump): edit on the submodule’s own feat branch; gate = charly box validate standalone (a version-stamp change has no build behavior — building proves nothing); commit, open PR, native auto-merge + tag-on-merge tags.

3. Land main via the PR — NEVER git push origin main (blocked) and NEVER git switch main in another worktree (git fatals “already used by worktree”). The org-wide native auto-merge performs the server-side squash merge (it advances origin/main remotely); then advance the LOCAL main ref where it lives: git -C <main-wt> merge --ff-only origin/main. A local main now only ever fast-forwards to what was merged remotely.

4. Tags: annotated only (git tag -a v<…> -m "<desc>" <merged-HEAD>), applied by tag-on-merge on the merged main HEAD and pushed as refs/tags/… (allowed by the pre-push-gate; the user token triggers the release-binary workflow). Verify git cat-file -t <tag> == tag AND git ls-remote --tags origin <tag> is non-empty.

5. Reconcile (when box submodules were re-stamped). Bump the superproject GITLINKS +1 to the re-stamped box mains (a separate superproject PR; B2 step-3 safety) — do NOT bump the @github build pins: they lag deliberately, charly box reconcile reports “already reconciled”, and bumping them pulls multi-cutover producer drift (a separate version-adoption cutover, NOT reconciliation).

6. Refresh EVERY worktree — PART of landing, NEVER a follow-up (R2). For each worktree: the one on main → git -C <wt> merge --ff-only origin/main; each other → git -C <wt> checkout --detach origin/main; THEN refresh only already-initialized submodules with git -C <wt> submodule update --recursive (no --init). Initialize only the paths the next task needs: a root R10 worktree needs sdk, so use git -C <wt> submodule update --init --recursive sdk; box-submodule work initializes its own declared path. Never blanket-initialize every submodule merely to refresh a worktree: it creates unnecessary per-worktree clone state. The Skill tool serves skills from the MAIN worktree — a stale main worktree silently serves STALE SKILLS to sessions, so refreshing it is mandatory. (A M <sub> in a worktree used only for the ff-merge is this drift, not lost work.)

A disposable bed that builds from a submodule builds from the submodule’s ON-DISK WORKING-TREE checkout, NOT from the committed gitlink alone. After a CLEAN gitlink auto-merge (no conflict), git ls-tree HEAD <sub> can already show the correct new pin while the submodule’s on-disk HEAD is still the OLD commit — a gitlink merge does not itself check out the new submodule content; that needs its own git -C <wt> submodule update --checkout <path> (or --recursive over the initialized set), same as any other post-merge refresh in this step. Skipping it means the bed silently builds STALE source even though the committed pointer is correct.

Landing gotchas (each cost real time): git merge-base --is-ancestor A B ERRORS if B’s object isn’t fetched (common for a sibling-worktree submodule) → git fetch first; cross-check git ls-tree origin/main <sub> before concluding “DIVERGED”. A git grep -- <submodule-path> from the superproject is a FALSE ZERO (git grep does not cross a gitlink) → git -C <sub> grep for the R5 sweep. The authoritative feat-branch head SHA comes from git ls-remote origin refs/heads/<branch> — gh pr view --json headRefOid LAGS a fresh push and will post the status on a stale SHA. Deriving the merge-time CalVer from a COMMIT’s recorded date (git show --date=format:'<fmt>' <sha>, git log -1 --format=%cd) uses that commit’s own author/committer TZ offset, not UTC, and can mis-stamp the tag/changelog by hours — $VER always comes from the LIVE clock at the moment of merge (date -u +%Y.%j.%H%M; the CalVer contract lives in references/validator-and-calver.md, not in this file), never from a commit’s stored timestamp. A metric or grep verification command (a LOC count, a git grep sweep, a file-count claim) run from an ambient, cd-inherited working directory silently measures the WRONG tree the moment more than one worktree is in play — anchor every such command to an explicit repo root (git -C /abs/path grep …, or a git rev-parse --show-toplevel cross-check first), never a bare relative command trusting the shell’s current directory. A stronger, WRITE-side form of the same footgun: a MUTATING command (anything that writes files or runs git submodule update as a side effect — task cue:gen is the canonical offender) run against a stale, ambient cwd doesn’t just misreport, it MUTATES the wrong tree. Failure mode: a fresh evaluator runs task cue:gen with a persisted shell cwd that had drifted to the main session worktree — the task’s own git submodule update chain rewound 5 submodule checkouts there before the mistake was caught (fully restored, disclosed). So every mutating task/command invocation in an isolated-worktree workflow (a validator run, a teammate’s branch work, a spike) carries an explicit cd <worktree> && anchor IN THE SAME compound command — never a bare task cue:gen (or any command with submodule/file-write side effects) trusting a cwd set by an earlier, unrelated step.