Runbooks

Worker returning 1101 / 500 on every request

Symptom: error code: 1101 from infra.omg.dev, every request, including ones that should be cheap.

Most likely cause: CF KV daily quota exceeded (free tier = 100k reads/day). Every Worker invocation does 1 KV.get; SSE retry storms on deleted deploys can burn it from a single tab.

Confirm:

cd apps/infra/worker && \
  CLOUDFLARE_API_TOKEN=$CF_API_TOKEN CLOUDFLARE_ACCOUNT_ID=$CF_ACCOUNT_ID \
  npx wrangler tail

Look for Error: KV get() limit exceeded for the day.

Fixes (in order of preference):

Already on Workers Paid? Wait — quota resets at 00:00 UTC.
Not on Paid? dash.cloudflare.com → Workers → Plans → Workers Paid.
The runtime is already minimised (isolate cache + 5min cacheTtl + SDK exponential backoff with circuit-break). If reads are still climbing, check for new SSE-using deploys hitting 4xx and dragging the SDK retry into a tight loop.

DELETE /v1/sandboxes/:id returns 500 from the Worker

Cause (history): vibes-router set duplex: "half" on every forwarded request. That is only valid for body-carrying methods. DELETE fetched without a body crashed the Worker.

Fix: shipped as commit 5357661. If you see this regress, the forward block in apps/infra/worker/src/index.ts should still read:

if (bodyCarryingMethod) {
  init.body = request.body
  init.duplex = "half"
}

Stuck Firecracker VMs after a service restart

Symptom: pgrep firecracker | wc -l on box-1 grows over time; DELETE on a sandbox returns "sandbox X not found" even though systemctl list-units 'vibes-fc-*' shows it running.

Cause: pre-migration sandbox rows had scope_unit IS NULL and weren't re-adopted by the boot reconciler. Post-2026-04-20 rows re-adopt cleanly; this only affects rows that survived from before scope_unit was added.

Fix (one-time, manual):

ssh root@178.63.205.231 \
  'for u in $(systemctl list-units --no-pager --no-legend "vibes-fc-*.service" | awk "{print \$1}"); do
     systemctl stop "$u"
   done'

Deploy stuck at status=building

Query sqld directly — bunx drizzle-studio, or select * from deploys where slug=? from the auth box. If status is building for >3 minutes, the async build goroutine probably crashed; the next deploy with the same slug will overwrite the row.

Deploy says `tarball restore unavailable`

This means the project tarball exists but the node could not obtain a usable base rootfs. Check findmnt /var/lib/vibes and lsblk on the assigned node. Production requires /var/lib/vibes to be its dedicated Btrfs mount; the service must not run against the writable mountpoint directory on /.

Current nodes fail closed at startup and runtime when the mount identity, filesystem, write probe, or NVMe health check fails. Readiness also requires a usable local warm rootfs or its durable Tigris object. The edge routes a deploy away from an unhealthy owner, and the healthy peer restores both warm rootfs and project artifact from Tigris. Do not recreate files under the bare mountpoint or increase cold-start timeouts; repair/replace the missing volume after the node has drained.

Service token revocation

See Auth → Rotating a service token.

Killing a runaway deploy

curl -X DELETE \
  -H "Authorization: Bearer $VIBES_INFRA_SERVICE_TOKEN_CI" \
  -H "X-On-Behalf-Of: ops" \
  "https://infra.omg.dev/v1/deploys/${SLUG}"

This stops the sandbox (drains litestream first), removes the deploy row, deletes the KV entry, and async-GCs both the litestream and static prefixes.