Recovery & Capacity Plan

2026-09-14 · TantraStudio / Coolify fleet · prod host max-vps · builder 173.212.238.95

Status snapshot — measured, not assumed

Thing	State
prod Tantra	running:healthy · pulls `ghcr.io/maxpetrusenko/tantrastudio:2b594ad16f70…` (pinned)
prod build surface	Tantra = `dockerimage` = cannot build. 5 other live apps still can.
builder box	6 vCPU · 4 runner listeners · 3 active workers
builder load	5.41 (1m) / 8.45 (5m) / 9.06 (15m) on 6 cores → 1.5× oversubscribed
stale worktrees	31 registered on the Tantra repo

Builder load is the headline: 15-minute average 9.06 against 6 vCPU means the box is running 50% more work than it has cores, sustained. Your read — capacity-bound, not stalled — is correct and this is the number that proves it.

CORRECTIONS What a review of the working tree found (v2)

Method failure, stated plainly. Three load-bearing facts in v1 were wrong because I planned against a fresh --depth 1 clone of the remote instead of reading the working tree on this box. Two of the three errors come from that single gap.

v1 claim	Reality on disk
"The only chatbox workflow is `stale-check.yml`; author a new one"	`.github/workflows/fast-coolify-deploy.yml` exists (3,293 B), untracked (`??`), on `security/tanstack-supply-chain-2026-05-12` — quality gate → buildx → push → PATCH → deploy. The task is `git add` + push, not authoring.
"Create a new `dockerimage` app"	The draft PATCHes `RESOURCE_UUID: jr9thwkx7rg4jxnmg4wvp0qw` — the existing app, in place. Creating a second app means the next push to main redeploys the old one: two containers, one Traefik router. That is gate 3's failure mode made permanent, and "leave the old app running" as rollback is what arms it.
"Pin the bare sha"	The draft ships `sha-<sha>`. Mandating bare sha and shipping both = the exact drift the plan forbids.

The plan found the custom_labels uuid hazard, then chose the path that has it. PATCHing in place keeps the uuid and dissolves the hazard entirely.

CORRECTED — this conflict was created by MY OWN subagent, not a pre-existing lane. Commit 7575d37 ("ci: build and push chatbox image to GHCR on the builder box") is authored and committed by the local git identity at 14:10:39 — three minutes after I dispatched a cutover agent at 14:07:07, and inside that agent's run window (stopped 14:15:53). It committed on ci/ghcr-build-push and pushed directly to main with no PR, leaving the branch and a .chatbox-main worktree behind. v2 of this plan attributed it to an unknown lane; that was wrong.

Two consequences worth naming. First, a CI workflow landed on chatbox main unreviewed, from a worker that a rate-limited lane had dispatched — the same class of unreviewed-to-main push this whole exercise is trying to stop. Second, the conflict is self-inflicted and therefore cheap to unwind: 7575d37 runs on [self-hosted, contabo-builder] and pushes the bare sha — which is the right tag convention and the wrong runner per P2. Keep the tag, move the runner.

Residue cleaned: branch ci/ghcr-build-push, worktree .chatbox-main, /tmp/chatbox_app_payload.json and my probe clone all removed; the repo is back to M docs/deploy-coolify.md + ?? .github/workflows/fast-coolify-deploy.yml. Nothing was lost — 0 commits on that branch were unreachable from origin/main. The pushed workflow is deliberately left in place because moving it off the builder runner is Max's call, not a cleanup step.

P0 chatbox → pull-only, corrected: PATCH in place, do not create

Chosen first because it is the only clean one: zero env vars, no volumes, no build_only secrets. Every other candidate has the build-time-secrets problem below.

The actual change — one PATCH, uuid preserved

PATCH /api/v1/applications/jr9thwkx7rg4jxnmg4wvp0qw
{
  "build_pack":                 "dockerimage",
  "docker_registry_image_name": "ghcr.io/maxpetrusenko/chatbox",
  "docker_registry_image_tag":  ""
}

Keep the uuid jr9thwkx7rg4jxnmg4wvp0qw. The app converts in place, Traefik labels stay keyed to the same uuid, and the domain never moves. No new app, no custom_labels trap, no domain cutover, no second container.

This supersedes v1's instruction to POST /api/v1/applications/dockerimage. That path was wrong twice over: it was unnecessary, and it armed the two-container failure. The reason it looked necessary was a real constraint — PATCH build_pack 422s with "The selected build pack is invalid" — which means the draft's PATCH only works once the tag convention and workflow are settled. Verify the PATCH returns 200 and build_pack reads back as dockerimage before triggering.

Reconcile the tags FIRST — this blocks the PATCH

A push to main now fires two workflows with incompatible tags:

Workflow	Runner	Tag pushed
`build-and-push-ghcr.yml` (on main, `7575d37`)	`self-hosted, contabo-builder`	bare sha + latest
`fast-coolify-deploy.yml` (untracked, draft)	`ubuntu-latest`	`sha-<sha>` + latest

Point the PATCH at the wrong one and Coolify pulls a tag that does not exist. Pick one workflow, one convention. P2 argues for ubuntu-latest — a free runner takes the compile off a box measured at 1.5× oversubscribed, and the draft's own choice is already right.

v1 said "never copy custom_labels" — correct, and now moot

The rule stands for any future app you do create: the labels are uuid-keyed Traefik router rules keyed to jr9thwkx7rg4jxnmg4wvp0qw, so a verbatim copy onto a new uuid yields a router pointing at the old app and no route at all. But because P0 now PATCHes in place, this hazard does not arise for chatbox.

The workflow: commit what already exists (v1 was wrong here)

v1 claimed chatbox had no build/push pipeline and the cutover must author one. False. .github/workflows/fast-coolify-deploy.yml exists on disk (3,293 B) and is complete and correct: quality gate → buildx → push to ghcr.io/maxpetrusenko/chatbox → PATCH build_pack: dockerimage + tag → POST /deploy. It is untracked (??) on security/tanstack-supply-chain-2026-05-12 (15 behind / 4 ahead of origin/main), which is why a clone of the remote cannot see it.

This is the plan's own lesson sitting in the repo as an artifact instead of a principle: an uncommitted workflow is not a shipped change. The task is git add + push, not authoring. And because main already carries build-and-push-ghcr.yml, committing this one without reconciling gives you two workflows racing on the same push.

Why the PATCH is required here and was noise on the other app: for a dockerimage app the configured tag is authoritative — Coolify pulls exactly what it says. The fleet already does this two ways (tantra ships a bare sha, ai-math-tutor ships sha-<sha>); both work because the tag the workflow pushes is the tag the app is told to pull. Pick the bare sha, match it on both sides, and never let them drift.

Dockerfile is node:20-bookworm-slim → pnpm build:web → caddy:2.8.4-alpine serving release/app/dist/renderer, EXPOSE 8080. packageManager: pnpm@10.15.1 — below the pnpm 11 allowBuilds breakage, but confirm the build log anyway.

How this gets tested — five gates, none of which is "the job was green"

No build happened on prod. Read the new Coolify deployment's log: it must contain neither Importing … to /artifacts/ nor Building docker image. Durations: the working pull-only deployments on this box run 21–101 s; a build is ~240 s. <150 s with no build lines = pass.
The right image is really running. docker inspect the container on max-vps and compare its image id to the image built on the builder — equal ids, not "both exist".
Exactly one chatbox container, and the old app's container is gone. Two containers behind one Traefik router is the failure mode that looks like a 200.
From outside the box: curl -sIL https://chatbox.maxpetrusenko.com → 200, plus a byte-diff of a known static asset against the built artifact.
Logs show it serving, not crash-looping — Caddy up, no repeated restarts in docker logs after 60 s.

Fails if: the app 503s, two containers serve, or the deployment log shows a build. Rollback: the old app is left running and untouched — do not delete it until gates 1–5 pass. Bringing it back is re-pointing the domain, not a rebuild.

Then clean up, in this order

Verify gates pass.
Commit + push the new workflow. Per this repo's own lesson: a direct upload or an uncommitted workflow is not a shipped change.
Delete the old dockerfile chatbox app only after the new one has served for a full deploy cycle.
Remove the agent's scratch: worktree, temp clone, /tmp tars, and the /tmp/chatbox-probe clone I made.

P1 The worktrees — and the finding that outranks them

The real P1 is not disk. It is 187 uncommitted files on a stale main.

main tree:   HEAD ceea7be   187 uncommitted files
             behind / ahead of origin/main:  184 / 2

~20 GB of worktrees is a disk problem; 187 uncommitted files on a tree 184 commits behind is a lost-work problem, and it is the one that can end in tears with no undo. Treat it as P1, worktrees as P2.

v1's counts were wrong — I read a stale clone. Re-measured after a fresh git fetch, using the same mechanical rule (merge-base --is-ancestor × status --porcelain):

Class	v1	measured	Action
merged + clean	17	22	Safe to remove — the disk win
merged + dirty	4	3	Inspect first — uncommitted work on a merged branch is either shipped or abandoned, so look
unmerged	7	8 (6 clean, 2 dirty)	Leave alone

34 registered worktrees total (33 non-main + the main tree). v1's classes summed to 33 because two of them were misfiled.

The "2 outside the repo" class does not exist — it was my invention, and it is how trees get deleted. Both Codex trees at ~/Documents/Codex/2026-09-13/ are registered worktrees of this repo and both are unmerged (codex/task-a-auth-rls at e241e28, detached be5e2ac). Filing them as "another tool's workspace" would have excluded them from scrutiny and put them in the delete path.

Duplicate shas are more than 3: 5fb6722 is checked out four times (ts-couples, ts-i18n, ts-recon, ci-builder-publish) and c4b1375 twice.

RESCUED — done, not pending. ci/name-contabo-hosts lived in a session-scoped scratchpad (/private/tmp/claude-501/-Users-maxpetrusenko/1e429178-…/scratchpad/wt-rename) that gets reaped without anyone running worktree remove. It is now on the remote:

git push origin ci/name-contabo-hosts
 -> 5af998acd1676d55269bf7e82512d5dc32a4d1b6   (NOT merged into main)

Its content is the concurrency fix that is still outstanding:

5af998a ci: coalesce deploys, and stop a new push cancelling one mid-flight
4ad5f18 ci: name the two Contabo hosts in the secrets, and split their key paths
 .github/workflows/build-push-image.yml    | 134 ++++++++++----------
 .github/workflows/fast-coolify-deploy.yml |  26 +++---

Note it touches the same two workflow files that alcohol-label-verifier-style work and the deploy-cancel lane touched — expect conflicts when it lands. It still wants a review and a merge decision; that is a separate call from the cleanup.

How the cleanup gets tested

Before: git worktree list | wc -l = 31, and record git status --porcelain per tree.
Only remove trees where git merge-base --is-ancestor <HEAD> origin/main exits 0 and status --porcelain is empty. Both conditions, mechanically — never by branch name.
Every unmerged or dirty tree: kanban_comment/note only, no deletion, and list it in the report.
After: git worktree list still contains the main tree + every excluded one; git worktree prune then git fsck --no-progress to confirm the object store is intact; re-run git worktree list and compare against the expected remainder count.
Falsifier: if git fsck reports missing objects or any unmerged branch's commit becomes unreachable, the cleanup was wrong — stop and restore from git reflog.

Disk: du -sh each tree before/after so the reclaimed number is real, not estimated.

P2 Capacity — the real lever

Max's framing is right: the queue is capacity-bound, not stalled. Four runner listeners on 6 vCPU, 3 running jobs, 15-min load 9.06. More workflows make this worse — every lane added is another concurrent compile on the same 6 cores.

Option	Effect	Cost / risk
A. Fewer pushes per lane (batch commits, path filters)	Direct, no money	Needs discipline; path filters already exist (`.github/**` is not an image input — a 23 s cache-hit build proved it)
B. Bigger box	Raises the real ceiling	Money, and a migration; but 6 vCPU hosting 4 runners + browser suites is undersized by construction
C. Structured admission control	Stops the 2-builds-pass-before-either-ramps TOCTOU	Design work; necessary regardless — the current preflight is a single sample of `/proc/meminfo`

Sequence: C, then A, then B only if still saturated. C first because the existing memory preflight is genuinely broken — systemctl show docker.service -p MemoryMax returns infinity, and BuildKit for the docker driver runs inside the already-running dockerd, so systemd-run --scope -p MemoryMax=5G wraps the CLI client, not the compile. Prod's containment was never containment.

How capacity gets tested

Baseline: uptime load sampled every 30 s for 10 min while 3 jobs run → record p50/p95.
Claim "path filters cut redundant builds" is only true if proven: count image-input commits vs total commits over a week, and show the builds avoided.
Admission control: fire two builds within 5 s of each other and assert the second queues rather than co-starting. That's the TOCTOU test the current design fails.
Falsifier: if p95 load is unchanged after A+C, the bottleneck isn't admission — it's raw cores, and B becomes the honest answer.

P3 Other findings, carried forward

#	Finding	Test
F1	An in-flight deploy polls 25 minutes against a deleted app (stage 6 has no "does this app exist" check, no failure fast-path)	Point a deploy at a deleted uuid; assert it fails in <60 s with a clear message, not 25 min
F2	Nothing detects staleness. The original incident was 15 h of no deploys with no signal	Freeze deploys deliberately; assert an alert fires before 60 min. This is the finding that would have caught everything else
F3	Every `Application PR Verification` fails — `next build` can't fetch IBM Plex / Newsreader. Cause: `fonts.googleapis.com` → `192.168.1.254` (LAN router)	`dig` from the mini runner must return real IPs, not the router; then a green `next build` on that runner. Router fix, not repo fix
F5	`fast-coolify-deploy.yml` still ships the legacy ssh build design (`ssh -i ~/.ssh/contabo_deploy_key`, lines 113–117) incl. the no-op `MemoryMax`	v1's test was broken: it grepped `BUILD_HOST_DIR`/`MemoryMax` for zero hits — but the Tantra file already has zero hits for both and still ssh-builds. Grep `ssh -i` / `CONTABO_SSH_KEY` / `contabo_deploy_key` instead: a passing test on absent strings proves nothing
F6	Rotate the Coolify API token — it was echoed into a session transcript today	Old token returns 401; new token deploys successfully

Cleanup contract — what "done" includes

Every item above ends with the same three steps, and the plan isn't complete until all three are true:

Deployed — verified live from outside, not from the deploy log.
Committed — the change is in git on main, because an uncommitted workflow is invisible to CI and disappears on the next redeploy.
Cleaned — worktree removed, branch deleted if merged, temp clones/tars/probe dirs gone, and the build cache on the builder box left no larger than it needs to be.

The build-secrets question, reframed (v1 asked the wrong thing)

v1 asked "Doppler or something else" for the remaining four apps' build_only vars. That is the wrong first question. The right one is whether each var is actually a secret at all:

Sort each var into	Where it belongs
Public build-time config — for a Vite/CRA build this is almost always `VITE_` / `REACT_APP_`	Hardcode, or a plain build arg. Public by construction — it ships in the client bundle, so no secret store makes it private.
Runtime secret	Coolify env on the app. Never a build arg — baked into a layer it stays readable in the image history and in GHCR.
Genuinely secret AND needed at build time	GitHub Environments secrets. The build already runs in Actions: one less dependency, one less token to rotate.

Doppler earns its place only if runtime secrets already live there and you want one source for both. This is a sort, not a decision to make up front — and it is per-var, not per-app. chatbox has none of these and is unaffected.