-| [ci.md](ci.md) | CI — running `.github/workflows` on push: the dispatch seam, the one-runner-three-substrates model, `act` self-hosted vs. nested, secrets, and the self-deploy loop. |

-the workflow in an isolated container, keep logs, report pass/fail, and surface

-builds, marketplace, service containers, caching, or OIDC.

-The strategic payoff is **inverting the source of truth**: S3 + gitmote become

-the origin, GitHub a mirror — and, once the [self-deploy loop](#not-yet-closed-the-self-deploy-loop)

-closes, a push can redeploy gitmote itself, made safe by the leased writer

-([safety.md §1](safety.md)).

-The engine is **[`act`](https://github.com/nektos/act)**: it runs GitHub-Actions

-YAML, so **one** workflow definition runs both self-hosted *and* on the GitHub

-mirror, and the break-glass redeploy uses the same workflow. Actions-compat is

-load-bearing, not a preference.

-

-## The pipeline, end to end

-

-CI doesn't bolt on — it hangs off primitives already built. The trigger is the

-**ref CAS commit** ([request-flows.md](request-flows.md), the push path): the one

-durable "a ref advanced" moment. After `CASRefs` succeeds, an **after-commit**

-callback dispatches — **fire-and-forget**, so a dispatch failure is a *missed

-deploy, not a failed push* (content-before-pointer, generalized).

- participant S as S3

-

- W->>W: CAS refs (the durable "advanced" moment)

- Note over W: after-commit, fire-and-forget

- W->>W: materialize @ SHA, read .github/workflows

- W->>W: record run + job(s), mint scoped clone token

- J->>W: GET /internal/ci/jobs/{id} (claim: queued→running)

- J->>W: git clone @ SHA (x-access-token: scoped token)

- J->>J: act over .github/workflows

- J->>W: POST /internal/ci/jobs/{id}/complete?status=… (log body)

- W->>S: PUT ci/{repo}/{run}/{job}.log (content)

- W->>W: SetJobResult(status, logKey) (pointer, after the blob)

-The runner ([`cmd/gitmote-runner`](../../cmd/gitmote-runner), [`internal/runner`](../../internal/runner))

-is one linear pass: **claim → clone → engine → complete**. It holds only a scoped

-clone token and injected secrets, reports over the authenticated API, and **never

-touches s3lite or S3 directly** — same discipline as the push-hook channel.

-

-The runner code and its env contract are identical everywhere; only *what starts

-the job* differs. Selected at boot from the environment

-([`cmd/gitmote/main.go`](../../cmd/gitmote/main.go)):

-| Condition | Trigger | Where the job runs |

-| `SCW_CI_JOB_DEFINITION_ID` set | Scaleway Serverless Jobs ([`internal/scaleway`](../../internal/scaleway)) — one `POST …/job-definitions/{id}/start` with per-run env | Cloud — an ephemeral, scale-to-zero Job |

-| `GITMOTE_URL` + `WORKER_SECRET` set | `LocalTrigger` ([`internal/ci`](../../internal/ci)) — execs the runner binary as a detached local process | The dev machine (`make dev`) |

-| neither | `NoopTrigger` | Nowhere — runs still **record** in the UI, but nothing executes |

-

-This is why `make dev` exercises the *entire* CI path locally with no cloud

-account: same runner, same claim/clone/complete API, same `act` — only the

-substrate is a local process instead of a Serverless Job. Cloud requires

-`WORKER_SECRET` and `GITMOTE_URL` too, or the server **refuses to start** (a

-Scaleway trigger with no way for the runner to report back is a misconfiguration,

-not a degraded mode).

-

-### Why `act` runs *self-hosted* on Scaleway — and nested locally

-

-`act` normally runs each job by nesting a per-job container via a local Docker

-daemon. **Scaleway Serverless Jobs have no Docker daemon** — so on the cloud

-runner, `act` runs in **self-hosted mode** (`-P ubuntu-latest=-self-hosted`, set

-by `GITMOTE_ACT_PLATFORMS` in [`Dockerfile.runner`](../../Dockerfile.runner)):

-the workflow steps execute *directly in the Job container*. The Job is a fresh,

-ephemeral, scale-to-zero container, so **the Job itself is the sandbox** — no

-Docker-in-Docker. The corollary: every tool a workflow needs (git, node, bash)

-must live in the runner image.

-

-Locally (`make dev`), where a real daemon exists (Docker or podman),

-`GITMOTE_ACT_PLATFORMS` is unset and `act` keeps its default nested-container

-behavior, unchanged. One binary, two isolation strategies, chosen by one env var.

-

-> This is the one place the shipped design diverges from the CI epic's original

-> sketch, which assumed the runner substrate offered "a real Docker daemon."

-> Serverless Jobs don't; self-hosted `act` is the resolution.

-

-Three s3lite tables in [`internal/meta`](../../internal/meta), written **only by

-the leader** (like refs), litestream-replicated like everything else:

-

-- `ci_runs` — one row per dispatched run (repo, ref, SHA, status).

-- `ci_jobs` — one row per workflow file in the run (name, status, log pointer).

-- `ci_secrets` — the sealed per-repo secret envelopes (see [Secrets](#secrets)).

-

-Logs are append-only blobs in the object store under a top-level `ci/` key space

-(peer of `objects/` and `meta/`): `ci/{repoID}/{runID}/{jobID}.log`. The runner

-POSTs the complete log on completion; the server PUTs the blob, then records the

-pointer in `ci_jobs` — **content before pointer**, unchanged from the git write

-path. Logs aren't sacred (a re-run regenerates them), so they need no

-litestream-grade durability; living in the same durable bucket is enough. A

-per-log size cap truncates a runaway log with an explicit marker, never silently.

-

-### The internal report API

-

-Two routes, mounted **outside** `/ui` (these are runner-facing, not

-browser-facing) in [`internal/ci`](../../internal/ci/report.go):

-| Route | Purpose |

-| --- | --- |

-| `GET /internal/ci/jobs/{id}` | Claim: atomically flip `queued → running`, return the job spec (repo, SHA, ref, workflow dir). Not claimable (absent / already claimed / terminal) → 404. |

-| `POST /internal/ci/jobs/{id}/complete?status=…` | Upload the log body, set the terminal status. **Idempotent** — a double-POST is a no-op. |

-

-Both authenticate with a constant-time compare against `WORKER_SECRET`. Only the

-**leader** may write completions; a follower returns a retryable 503. A

-leader-only **reconcile ticker** sweeps jobs stuck in `running` (a runner that

-died mid-run) to a terminal `error` after a max age.

-

-Per-repo CI secrets, encrypted at rest, decrypted only to inject into the runner

-env at trigger. The crypto and its **narrow** threat model live in

-[safety.md §5](safety.md): AES-256-GCM under a **server-held** master key

-(`GITMOTE_CI_SECRET_KEY_V<n>`), a per-repo subkey via HKDF-SHA256, and AAD

-binding `(repoID, name, version)`. Encryption-at-rest buys exactly one thing — a

-compromise of the S3 replica / DB snapshot doesn't leak values — and is

-explicitly *not* a defense against a compromised running server, which must hold

-the key to decrypt. Values are write-only in the UI (only names are shown) and

-never logged.

-

-Injection is deliberately indirect. The dispatcher puts each secret in the job

-env as `GITMOTE_CI_SECRET_<NAME>`; the runner strips the prefix and forwards it to

-`act` as `-s <NAME>`, which reads the **value from its own environment** — so the

-secret value never touches the `act` argv, and multiline values survive. The

-workflow sees it as `${{ secrets.NAME }}`, exactly as on GitHub.

-

-> This indirection was earned the hard way: `act` does not forward the host env

-> into job containers, so a naive "set it in the environment" leaks nothing *and*

-> delivers nothing. The `-s NAME` hand-off is what actually reaches the workflow.

-

-repo-scoped, short-lived token**, minted by the leader at dispatch

-(`https://x-access-token:<tok>@gitmote…/owner/repo`, then checkout the SHA). It's

-the **same authenticated path as any client** — no CI backdoor in the git

-handler. Rejected alternatives: a `WORKER_SECRET`-derived credential (shared

-across all runs → a leak grants broad standing access) and presigned-S3 /

-git-bundle hand-off (duplicates the git path and needs broad bucket creds). The

-token model gained nullable `expires_at`, a single-repo scope, and a `read_only`

-flag to support this — generally useful beyond CI.

-

-To fix `github.ref` inside the workflow, the runner checks the SHA out *onto its

-branch* (`checkout -B <branch> <sha>` when the ref is `refs/heads/…`) rather than

-leaving a detached HEAD.

-

-- **Untrusted code execution.** Repo workflows are attacker-controlled. The

- **Serverless Job is the isolation boundary** — the writer never executes repo

- code. The runner holds only a scoped clone token + injected secrets, reports

- over the authenticated API, and cannot reach s3lite or S3.

-- **Fire-and-forget everywhere.** Dispatch, trigger, and (eventually) deploy must

- never fail or block a push. A missed run is recoverable; a failed push is not

- acceptable.

-- **Single writer holds.** Only the leader dispatches, writes run/job state, and

- applies completions; runners funnel results back through the API. Nothing here

- adds a second writer — so a self-replacing deploy stays safe by construction.

-- **Concurrency.** Runs are *dispatched* serially by the single writer (in push

- order) but *execute* in parallel on Scaleway. The MVP does **not** cancel

- in-flight runs; a superseded run runs to completion and correctness comes from

- the latest-wins deploy guard, not from racing to cancel.

-

-

-## Not yet closed: the self-deploy loop

-

-Today gitmote **already redeploys itself** on a green `master` push — via GitHub

-Actions ([`.github/workflows/ci.yml`](../../.github/workflows/ci.yml)): build +

-push the amd64 image, `scw container container update … --wait`

-([ops.md](../ops.md)). The leased writer makes the swap safe: the new image boots

-a follower, the old releases the lease on SIGTERM, the new promotes — never two

-writers.

-

-The **ouroboros** — gitmote's *own* runner doing that deploy, with GitHub demoted

-to break-glass — is the remaining capstone (`tasks/24`). Two pieces stand between

-here and there:

-

-1. **Daemonless image builds.** The deploy step is `docker build`, but the

- self-hosted runner has no Docker daemon (that's *why* `act` runs self-hosted).

- Building the image *inside* the Job means nested, rootless `buildah`/`podman`

- (fuse-overlayfs where the sandbox allows it, else the slower `vfs` driver) — a

- spike, not a wall. The fallback is legitimate: keep the deploy on the GitHub

- mirror and let the self-hosted runner do build+test only.

-2. **Latest-wins guard.** Rapid pushes race; the lease keeps that *safe*, but a

- slow run could still ship an older SHA after a newer one. The deploy step must

- no-op unless its target SHA is still `master`'s tip in gitmote — a stale run

- deploys nothing. Implemented as a workflow step, so the mirror enforces it too.

-

-Until then, the GitHub-Actions path *is* the deployer (and the break-glass copy).

-

-

-## Operational notes

-

-- **One job definition serves all repos** — created once out of band

- (`scw jobs definition create`); the app injects per-run env at trigger. See

- [ops.md](../ops.md) for the CI env/secrets on the container (`SCW_SECRET_KEY`,

- `SCW_REGION`, `SCW_CI_JOB_DEFINITION_ID`, `WORKER_SECRET`, and the secrets-store

- master key), distinct from the deploy pipeline's GitHub Actions secrets.

-- **The runner image is pushed by hand today** (`rg.fr-par.scw.cloud/atmin/gitmote-runner:master`);

- nothing in CI rebuilds it, so a stale runner is easy to forget. Automating that

- build/push is a natural follow-up.

-> **Status: evolution note — speculative, not a commitment.** Where gitmote could

-> go, captured so the reasoning survives a fresh session. Nothing here is planned

-> or specced; don't cite it as a spec.

-# CI — run workflows on push (epic index)

-

-> **Epic index — Stage 0 resolved, stages split into their own tasks.** This file

-> holds the locked cross-cutting decisions (Stage 0) and the staged overview; the

-> implementable units are **[tasks 17–24](README.md)**, each self-contained with

-> Spec/Current/Change/Verify. Keep this index (and delete it only once every stage

-> has landed); the stage tasks are deleted as they land, per the frontier rule.

-> The *why* lives in [../docs/evolution/ci-runner.md](../docs/evolution/ci-runner.md);

-> this file is the *how*, not a re-argument of the vision.

-

-## Spec

-

-Run workflows on push, in a limited-but-real form: on a successful ref advance,

-execute the repo's workflow in an isolated container, keep logs, report pass/fail,

-surface status in the UI. The strategic payoff (per the evolution note) is

-**inverting the source of truth** — S3 + gitmote become the origin, GitHub a

-mirror — and closing the loop so a push can **redeploy gitmote itself**, made

-safe by the leased writer ([safety.md §1](../docs/architecture/safety.md),

-[ops.md](../docs/ops.md)).

-

-**Non-goals (MVP):** full GitHub Actions parity — marketplace, matrix builds,

-service containers, remote caching, OIDC. ~80% of personal CI is "on push, run

-steps in a container, pass/fail, keep logs"; build that, grow compat only where

-the lack is felt.

-

-## Current — the seams this reuses

-

-CI does not bolt on; it hangs off primitives already built (the evolution note's

-table, made concrete):

-

-| CI need | Existing seam | File |

-|---|---|---|

-| Trigger (`on: push`) | The ref **CAS commit** — the one durable "a ref advanced" moment | [`Writer.handle`](../internal/githttp/receive.go#L189) — immediately after `CASRefs` succeeds (`receive.go:216`), before `Response{OK:true}`. There is already a `beforeCAS` hook; this adds an **after-commit, fire-and-forget** dispatch. |

-| Read the workflow file at the pushed SHA | The browse reader from tasks 14/15 | [`repo.Tree` / `repo.Blob`](../internal/repo/browse.go) over a `Materialize`d dir — no new object plumbing. |

-| Run / job state | An s3lite table, single-writer | [`internal/meta`](../internal/meta) — new `ci_runs` / `ci_jobs`, written **only by the leader** (like refs). |

-| Logs & artifacts | Append-only blobs in S3, content-before-pointer | [`store.Store`](../internal/store) under a new `ci/` key space. |

-| Runner → server report | An authenticated internal API + shared secret | Same discipline as the [hookrpc](../internal/hookrpc) push channel; runners **never touch s3lite directly**. |

-| Status UI + log view | Admin-gated server-rendered pages + sanitized rendering | [`internal/webui`](../internal/webui) + [`internal/render`](../internal/render). |

-

-**Runner substrate — Scaleway Serverless Jobs.** A finite task that scales to

-zero, with a real Docker daemon + CPU the always-warm Container can't offer. The

-**exact invocation pattern is proven in a sibling project** and directly

-stealable — `~/dev/pan0.com/lib/scaleway/jobs.go`:

-

-- `POST https://api.scaleway.com/serverless-jobs/v1alpha2/regions/{region}/job-definitions/{id}/start`

-- Header `X-Auth-Token: <SCW_SECRET_KEY>`; body `{"environment_variables": { … }}`.

-- The job **definition** is created once, out of band (`scw jobs definition create … cpu-limit … memory-limit … local-storage-capacity …`); the app injects **per-run env at start** (so one definition serves all repos/branches).

-- **Fire-and-forget + webhook callback**, not polling: worker reads `JOB_ID`/`APP_URL`/`WORKER_SECRET` from env, `GET`s the job spec to claim it, runs, then `POST`s completion — all authenticated with a constant-time `WORKER_SECRET` compare. pan0's `handle_internal_jobs.go` (claim + idempotent `ApplyCompletion`) is the template for our internal API.

-- No-op when the job-definition ID env is unset → local dev / tests run the whole flow with a stub, exactly as pan0 does.

-

-## Stage 0 — Decisions to lock first (no code)

-

-These gate the design; resolve before Stage 1.

-

-1. **Engine — ✅ LOCKED: `act`** (not Woodpecker). Decisive reason is the mirror:

- `act` runs `.github/workflows` YAML (~80% Actions-compatible), so **one** CI

- definition runs both self-hosted *and* on the GitHub mirror — the break-glass

- redeploy uses the same workflow. Woodpecker would mean **two** definitions that

- drift, weakening "GitHub as a clean redeploy." Actions-compat is load-bearing

- here, not a preference. The runner image (Stage 4) is built around `act`'s

- engine (cf. Forgejo's `act_runner`).

-2. **Workflow file location & format — ✅ settled by (1):** `.github/workflows/*.yml`,

- GitHub Actions YAML. No native format.

-3. **Secrets model — ✅ LOCKED.** Per-repo CI secrets, encrypted at rest,

- injected into the runner env only at trigger time. Prior art:

- `~/dev/atmin.net` has mature crypto worth stealing *selectively* — but its

- **trust model is inverted from ours and that difference drives the design.**

-

- - **Trust model (the honest part first).** atmin.net is E2E: the server never

- sees plaintext, the master key is *derived from a user password client-side*.

- gitmote is the opposite — the **server must decrypt** to inject secrets into

- the runner, so it **holds the master key** (a container secret env var,

- alongside `GITMOTE_COOKIE_KEY` / AWS keys — see [ops.md](../docs/ops.md)).

- Therefore encryption-at-rest here buys exactly one thing: **a compromise of

- the S3 replica / DB snapshot (the litestream WAL, the object bucket) does not

- leak secrets, because the master key lives only in the container env, not in

- the replicated DB.** It does **not** protect against a compromised *running*

- server. Say this plainly in the safety doc — don't oversell it. No Argon2id,

- no password-KDF: there's no user password in this path.

- - **Primitive — port directly.** AES-256-GCM, **12-byte random IV per

- encryption** from `crypto/rand`, never reused (atmin.net's `backupEncrypt`,

- `web/src/lib/crypto.ts`). In Go: `crypto/aes` + `crypto/cipher` GCM.

- - **Per-repo subkeys — steal the HKDF-per-purpose pattern.** Derive a per-repo

- key from the master with HKDF-SHA256 (`info = "ci-secret:" + repoID`), so a

- single container-held master gives per-repo key isolation for free (mirrors

- atmin.net's `hkdfDerive(secret, info)`).

- - **Versioned envelope + O(1) rotation — steal the key-chain idea.** Store

- `{v, iv, ciphertext}` with a leading key version; rotate by **wrapping the old

- key under the new one** (a `key_chain`) rather than re-encrypting every secret

- — atmin.net's ADR-0012 / `key-chain.ts`. Lazy migration: rotation is O(1),

- old-secret reads walk the chain.

- - **One improvement over the prior art:** atmin.net's envelope does **not**

- bind `v`/name into the GCM tag (an acknowledged AAD gap). We should pass

- **AAD = (repoID, secretName, version)** to GCM, so a stolen ciphertext can't

- be replayed under a different repo/name/version.

- - **Storage:** a `ci_secrets` table in `meta` (s3lite) — single-writer,

- litestream-replicated like everything else; the master key is the one thing

- *not* in there. Never logged, never rendered (Stage 5 test asserts this).

-4. **Runner clone auth — ✅ LOCKED.** A **per-run, read-only,

- repo-scoped, short-lived token**, minted by the leader at dispatch and revoked

- on completion. The runner clones over normal git-HTTPS

- (`https://x-access-token:<tok>@gitmote…/owner/repo`, checkout the SHA) — the

- **same authenticated path as any client**, no CI backdoor in the git handler.

- Rejected: a `WORKER_SECRET`-derived clone credential (shared across all runs →

- a leak grants broad standing access; violates least-privilege). Rejected:

- presigned-S3 / git-bundle hand-off (duplicates the git path, and the runner

- would need broad bucket creds).

- - **Cost:** a contained extension to the token model — the

- [tokens table](../internal/meta) gains nullable `expires_at`, a `repo_scope`

- (single repo), and a `read_only` flag; [`auth.Guard`](../internal/auth)

- honors them (expired / over-scoped → denied). Expiring, scoped tokens are

- generally useful beyond CI. TTL = run max-duration + margin; sweep expired

- tokens on the log-GC cadence. Lands as a small precursor sub-task to Stage 4.

-5. **Log storage & retention — ✅ LOCKED.**

- - **Layout:** a new top-level `ci/` key space in the object store (peer of

- `objects/` and `meta/`): `ci/{repoID}/{runID}/{jobID}.log`. Artifacts, when

- they exist, go under `ci/{repoID}/{runID}/artifacts/` (out of MVP scope).

- - **Write:** MVP uploads the **complete log once on completion** (one PUT, like

- pan0's worker); the pointer is recorded in `meta` **after** the blob lands

- (content-before-pointer, unchanged). Live tailing / chunked append is a later

- want, not MVP.

- - **Caps (no silent truncation):** a per-log size cap (e.g. 10 MiB) so a

- runaway/malicious workflow can't fill S3; on hitting it, truncate with an

- explicit marker (the "more" discipline from task 14), never silently.

- - **Retention:** time-based default (**30 days**) plus keep-last-N per repo,

- swept by the **leader** — deletes old run rows and their `ci/` blobs together.

- Logs aren't sacred (a re-run regenerates them), so they need no

- litestream-grade durability; being in the same durable bucket is enough.

-6. **Concurrency / cancellation — ✅ LOCKED.** MVP does **not** cancel

- in-flight runs. Dispatch is serialized by the single writer (runs enqueue in

- push order); execution is parallel on Scaleway. When a newer push supersedes an

- older run's ref, the older run is **marked `superseded`** in the UI but **runs

- to completion** — correctness comes from Stage 7's **latest-wins deploy guard**

- (a stale run deploys nothing), not from racing to cancel. Actively stopping a

- superseded Scaleway job (to save compute) and a per-repo concurrent-run cap are

- **follow-ups**, added only if compute cost or a push-storm is felt; rely on

- Scaleway's own job-concurrency limits until then.

-

-## Staged plan

-

-Each stage lands non-breaking and tested (golden **and** failure path, per

-[CONTRIBUTING](../CONTRIBUTING.md)). Ordered so value appears early and the risky

-surfaces (untrusted execution, secrets) come after the data model is solid.

-

-### Stage 1 — Data model + dispatch seam (no execution)

-- `ci_runs` / `ci_jobs` tables in `meta`; a `Runs` API mirroring `refs` discipline.

-- Add an **after-commit** callback to `Writer` (sibling to `beforeCAS`), invoked

- after `CASRefs` succeeds. It records a queued run row and returns immediately —

- **never blocks or fails the push** (a failed dispatch is a *missed deploy, not a

- failed push* — the content-before-pointer rule, applied to CI).

-- **Verify:** a ref-advancing push creates exactly one queued run; a rejected /

- non-fast-forward push creates none; a forced dispatch error leaves the push

- green.

-

-### Stage 2 — Config discovery

-- On dispatch, read the workflow file from the pushed commit's tree via the

- browse reader at the SHA. No file → no run. Malformed config → a run recorded as

- **failed with a clear message**, never a crash.

-- **Verify:** repo with / without the file; a malformed file surfaces as a failed

- run, not a 500 or a panic in the writer.

-

-### Stage 3 — Scaleway Jobs client + trigger

-- New `internal/scaleway` `JobsClient.Trigger` — port of pan0's `jobs.go`

- (raw `net/http`, `X-Auth-Token`, `…/start`, env-var body). No-op when the

- job-definition ID is unset.

-- Wire Stage 1's dispatch to call `Trigger` with `RUN_ID`, `APP_URL`,

- `WORKER_SECRET`, repo, SHA. Fire-and-forget.

-- **Verify:** an httptest server asserts URL / `X-Auth-Token` / body shape; unset

- definition ID → no-op; a trigger error marks the run failed but the push stays

- green.

-

-### Stage 4 — Runner image + internal report API

-- `cmd/gitmote-runner` + `Dockerfile.runner`: claim the job

- (`GET /internal/ci/jobs/{id}`, `X-Worker-Secret`), clone at the SHA, run the

- engine over the workflow, stream logs to S3, `POST …/complete`.

-- `internal/webui` (or a dedicated internal handler): claim + **idempotent**

- completion (pan0's `ApplyCompletion` shape), constant-time secret compare,

- runner **never** touches s3lite.

-- **Verify:** end-to-end against a test server with a local/fake engine;

- completion is idempotent (double-POST is a no-op); missing/incorrect secret →

- 401; a run that never reports is reconcilable (timeout → failed).

-

-### Stage 5 — Secrets store

-- Per-repo secrets, AES-256-GCM at rest, decrypted only to inject into the trigger

- env. Never logged, never rendered.

-- **Verify:** encrypt/decrypt round-trip; a secret never appears in logs or the

- UI; injected only into the Scaleway `start` env, not persisted in the run row.

-

-### Stage 6 — Status UI + logs

-- webui: per-repo runs list, run detail with per-job status, a log viewer

- (`render`, ANSI→HTML if needed). A commit-status affordance in the browse views

- (tasks 14/15). Admin-gated like the rest of `/ui`.

-- **Verify:** route tests, golden + failure, still behind `requireAdmin`.

-

-### Stage 7 — Self-deploy loop + latest-wins guard

-- A green run on `master` redeploys gitmote (the ouroboros). **Latest-wins:** the

- deploy step no-ops unless its target SHA is still the branch tip in gitmote — a

- slow, stale run deploys nothing rather than regressing prod. The GitHub mirror

- becomes **break-glass**: an independent clean copy to redeploy from by hand if a

- bad push wedges the loop.

-- **Verify:** a run whose SHA is no longer the tip deploys nothing; the leased

- writer keeps a redeploy from ever running two writers (already proven by

- `make e2e-restore` + the lease); `make e2e-local` stays green.

-

-## Cross-cutting safety (likely a new [safety.md](../docs/architecture/safety.md) section)

-

-- **Untrusted code execution.** Repo workflows are attacker-controlled. The

- **Scaleway Job is the isolation boundary** — the writer never executes repo

- code. The runner holds only a scoped clone token + injected secrets, reports

- over the authenticated API, and cannot reach s3lite.

-- **Fire-and-forget everywhere.** CI dispatch, trigger, and deploy must never fail

- or block a push (content-before-pointer, generalized).

-- **Single writer holds.** Only the leader dispatches and writes run state;

- runners funnel results back through the API. Nothing here adds a second writer.

-- **Secrets** are the sole genuinely new surface — Stage 5's encrypted store, kept

- separate and never logged.

-

-## Ops (deferred to the relevant stages; lands in [ops.md](../docs/ops.md))

-

-- One-time `scw jobs definition create` for the runner image (cpu/memory/storage),

- mirroring pan0's worker definition; **one definition serves all repos**, env

- injected at trigger.

-- New CI secrets / env: `SCW_SECRET_KEY`, `SCW_REGION`, `SCW_CI_JOB_DEFINITION_ID`,

- `WORKER_SECRET`, and the secrets-store master key — on the container, distinct

- from the deploy pipeline's GitHub Actions secrets.

-

-## Open questions

-

-Beyond Stage 0's gating decisions: run retention & GC; per-run resource limits

-and timeouts; how `act`'s Docker-in-Docker needs map onto a Scaleway Job's

-capabilities; whether pull-request-style events exist before there's a PR concept;

-surfacing status back to the GitHub mirror (checks API) or leaving them

-independent.

-# CI stage 4b(cloud) — Scaleway runner image

-

-Part of the CI epic ([16-ci.md](16-ci.md)). The runner and the whole execution

-path have **landed and run locally**: on a push, gitmote records a run, mints a

-scoped clone token, and triggers `gitmote-runner`, which claims the job, clones

-at the SHA, runs `.github/workflows` with `act`, and reports the log + pass/fail.

-What remains is packaging that exact runner to run on **Scaleway Serverless Jobs**

-instead of a local process.

-

-## What has landed (the whole local path)

-

-- **Runner** — `cmd/gitmote-runner` + `internal/runner` (claim → clone@SHA →

- engine → complete). Engine is `act` (`internal/runner/act.go`); clone is the

- normal git-HTTP path with a scoped token (`git.go`). Substrate-agnostic: the

- same binary runs locally and (once this task lands) in a container.

-- **Local trigger** — `ci.LocalTrigger` spawns the runner as a local process,

- the dev analog of Scaleway starting a job. `main.go` picks it when

- `GITMOTE_URL` + `WORKER_SECRET` are set and `SCW_CI_JOB_DEFINITION_ID` is not.

- `make dev` wires this, so **CI is testable locally** (needs `act` + Docker).

-- **Real clone token** — the dispatcher mints a read-only, repo-scoped, expiring

- token under the pusher (`auth.Guard.MintScoped`), injected as

- `GITMOTE_CI_CLONE_TOKEN`. No CI service identity or ACL grant needed.

-- **Report API + reconcile** — `internal/ci/report.go` (claim/complete, idempotent,

- content-before-pointer log store) plus a leader-only `ReconcileStuck` ticker in

- `main.go` that sweeps abandoned `running` jobs to `error`.

-

-## The remaining work — cloud packaging

-

-> **Infra spike to resolve first:** `act` runs each job in a Docker container, so

-> the runner needs a Docker daemon inside the Scaleway Serverless Job. Confirm

-> Jobs expose a usable daemon (privileged / DinD) or pick the fallback: run steps

-> in a single provided container without full `act` nesting. Spike this before

-> building the image. (Local dev sidesteps this — the host daemon is real.)

-

-- **`Dockerfile.runner`** — `act` + git + the `gitmote-runner` binary + the daemon

- strategy from the spike. The runner reads the same env it does locally.

-- **`scw jobs definition create`** — one-time, out of band:

- `name=gitmote-ci-runner image=… cpu-limit=… memory-limit=…

- local-storage-capacity=…`, documented in [ops.md](../docs/ops.md). One

- definition serves all repos (env injected at trigger, per stage 3). Setting

- `SCW_CI_JOB_DEFINITION_ID` flips `main.go` from the local trigger to Scaleway —

- no code change.

-

-## Verify

-

-- **image smoke:** a container built from `Dockerfile.runner`, given a job's env

- against a reachable gitmote, drives a real workflow to `passed`/`failed`; the

- log round-trips (the local path already proves the runner logic).

-- **security:** the runner env holds only the scoped clone token + `WORKER_SECRET`

- — no S3/DB credentials (already true of the env contract; assert it in the job

- definition).

-- `gofmt`/`golangci-lint`/`go test ./...` clean; `make e2e-local` +

- `make e2e-restore` green.

-# CI stage 7 — self-deploy loop + latest-wins guard

-

-Part of the CI epic ([16-ci.md](16-ci.md)). The capstone: a green CI run on

-`master` **redeploys gitmote**, closing the ouroboros. Depends on the full CI

-pipeline (stages 1–6). The leased writer already makes an instance replacing

-itself safe by construction ([ops.md](../docs/ops.md), the single-writer section)

-— this stage adds the trigger and the anti-regression guard.

-

-## Spec

-

-On a green run for `master`, deploy the new image. **Latest-wins:** the deploy

-no-ops unless its target SHA is still `master`'s tip in gitmote, so a slow/stale

-run never regresses prod (pairs with stage 6's `superseded` marking). The GitHub

-mirror runs the **same** `.github/workflows` (engine is `act`, epic §Stage 0 #1),

-so it doubles as **break-glass**: an independent clean copy to redeploy from by

-hand if a bad push ever wedges the self-deploy loop.

-

-## Current

-

-- Today's deploy is GitHub Actions (`.github/workflows/ci.yml`) → build+push the

- amd64 image → `scw container container update … --wait` ([ops.md](../docs/ops.md)

- "Deploying"). No drain step (the lease covers it).

-- With `act`, that **same workflow** now also runs self-hosted on gitmote CI —

- one definition, two runners (self + mirror).

-- gitmote exposes `master`'s current tip via the ref advertisement

- (`info/refs`) / the metadata layer.

-

-## Change

-

-**1. Deploy job in the workflow.** Keep the existing build+push+`scw … update`

-steps (they already work on GitHub Actions and, via `act`, self-hosted). No new

-deploy mechanism — reuse it.

-

-**2. Latest-wins guard.** Before the `scw … update` step, check that the run's

-target SHA still equals `master`'s tip in gitmote (query `info/refs` or a tiny

-read endpoint for the branch SHA); if not, **skip the deploy** and log that a

-newer commit superseded this run — deploy nothing rather than ship an older SHA.

-Prefer implementing the guard as a workflow step (portable to both runners) over

-server-side logic, so the mirror enforces it too.

-

-**3. Docs.** Update [ops.md](../docs/ops.md) "Deploying" for the self-hosted path

-and the latest-wins guard, and add a **break-glass** note: if the self-deploy loop

-is wedged, redeploy by hand from the GitHub mirror (the same workflow, clean

-copy). Add a new [safety.md](../docs/architecture/safety.md) subsection tying the

-loop to the leased-writer invariant (new boots follower, old releases on SIGTERM,

-new promotes — never two writers). Cross-link from

-[evolution/ci-runner.md](../docs/evolution/ci-runner.md) (mark the loop realized).

-

-## Verify

-

-- **latest-wins guard test:** a run whose SHA is still the tip proceeds; a run

- whose SHA is **no longer** the tip deploys **nothing** (unit-test the guard's

- decision against a stubbed tip; integration if a runner harness exists).

-- **safety regression:** `make e2e-restore` (cold-start restore) and

- `make e2e-local` (push/clone + restart durability) stay green — the redeploy

- path relies on the same lease behavior they already prove; no two-writer window

- is introduced.

-- **mirror parity:** the same `.github/workflows` runs on both GitHub Actions and

- the self-hosted runner without divergence (the load-bearing reason `act` was

- chosen).

-- `gofmt`/`golangci-lint`/`go test ./...` clean.

-

-## Not in scope

-

-Deploy environments/approvals, rollbacks beyond "redeploy the previous image

-tag", and multi-service deploys. The loop here is gitmote deploying gitmote.

-Active and upcoming work toward the milestone: **gitmote can host itself.**

-One file per unit of work, `Spec → Current → Change → Verify`, **deleted once it

-lands** (commits are the record — this list never becomes a changelog).

-Ordered by dependency; each task lands in a non-breaking, tested state.

-The **gitmote can host itself** milestone is reached: it runs on Scaleway at

-`gitmote.atmin.net`, deploys itself on every green master push, survives cold

-starts (litestream restore), and enforces the single writer by a lease (s3lite

-`RoleAuto`), so rolling deploys are safe by construction. Speculative directions

-live in [../docs/evolution/](../docs/evolution/).

-**CI — run workflows on push.** Design + locked Stage 0 decisions in the epic

-index [16-ci.md](16-ci.md) (the *why* is in

-[../docs/evolution/ci-runner.md](../docs/evolution/ci-runner.md)). Split into

-dependency-ordered stages, each implementable from fresh context:

-

-- [21-ci-runner.md](21-ci-runner.md) — **4b(cloud):** Scaleway runner image only.

- The runner, local trigger, scoped clone token, and reconcile ticker have landed

- (CI runs locally via `make dev` + `act`); what remains is packaging the runner

- for Scaleway (Dockerfile.runner + `scw jobs definition`), gated on a DinD spike.

-- [24-ci-self-deploy.md](24-ci-self-deploy.md) — green master run redeploys

- gitmote; latest-wins guard; GitHub mirror as break-glass.