gitmote

gitmote / docs / architecture / ci.md  ·  history  ·  branches

12715 bytes · raw

gitmote — CI

Run a repo's .gitmote/workflows on push: on a successful ref advance, execute the workflow in an isolated container, keep logs, report pass/fail, surface status in the UI. Deliberately limited but real — the ~80% of personal CI that is "on push, run steps in a container, pass/fail, keep logs." No matrix builds, marketplace, service containers, or caching.

The engine is act: it runs GitHub-Actions YAML, so a workflow authored for Actions runs unmodified under gitmote's runner. Actions-compat is load-bearing (the GitHub mirror is break-glass — move the same YAML to .github/workflows and GitHub runs it), not a preference.

Why .gitmote/workflows, not .github/workflows. GitHub only ever reads .github, so the directory a workflow lives in declares which forge runs it: a repo mirrored to GitHub puts CI in .gitmote to run only here, in .github to run only on GitHub, and so never double-runs on both. (gitmote's own repo keeps its deploy pipeline in .github and therefore never CI's on gitmote — the concrete case that motivated this.) act reads the directory via --workflows; the YAML is byte-identical to GitHub's, so moving a file between the two directories is the whole switch.

Pipeline

The trigger is the ref CAS commit (request-flows.md) — the one durable "a ref advanced" moment. After it succeeds, an after-commit callback dispatches, fire-and-forget: a dispatch failure is a missed run, not a failed push (content-before-pointer, generalized).

sequenceDiagram
    participant C as git client
    participant W as gitmote (leader)
    participant J as runner (Serverless Job)
    C->>W: git push
    W->>W: CAS refs
    W-->>C: 200 OK (push done — CI never blocks it)
    W->>J: Trigger (env: job id, URL, WORKER_SECRET, clone token, secrets)
    J->>W: claim (queued→running) → clone @ SHA → act → complete (+log)
    W->>W: PUT log blob, then record pointer (content before pointer)

The runner (internal/runner) is one linear pass: claim → clone → act → complete. It holds only a scoped clone token and injected secrets, reports over the authenticated API, and never touches s3lite or S3 directly.

Branch filters gate dispatch, GitHub-style. The dispatcher parses each workflow's on.push and records a run only for workflows whose branch filter selects the pushed branch — so a push that matches nothing produces no run at all, exactly as GitHub shows no workflow run when a push triggers nothing (a green no-op run, its log full of engine start-up chatter, would otherwise be pure noise — the concrete case: gitmote's own deploy.yml gated on self-deploy, which must stay silent on every master push). It models the common filter subset — branches / branches-ignore with * and ** globs, the ~80%; paths, tag filters, and the rarer ?/+/[]/! patterns are not evaluated (a workflow using them is kept, erring toward running). A workflow file that can't be read or parsed is also kept, so a malformed workflow surfaces as a failed run rather than vanishing.

One runner, three substrates

The runner code and env contract are identical everywhere; only what starts a job differs, selected at boot (cmd/gitmote/main.go):

Condition Trigger Runs where
SCW_CI_JOB_DEFINITION_ID set Scaleway Serverless Jobs (internal/scaleway) Cloud (ephemeral, scale-to-zero Job)
GITMOTE_URL + WORKER_SECRET set LocalTrigger — execs the runner binary locally Dev machine (make dev)
neither NoopTrigger Nowhere — runs record in the UI but don't execute

So make dev exercises the whole path locally — same runner, same API, same act. Cloud also requires WORKER_SECRET + GITMOTE_URL or the server refuses to start (a trigger the runner can't report back to is a misconfiguration).

act runs self-hosted on Scaleway, nested locally. Serverless Jobs have no Docker daemon, so on the cloud runner act runs in self-hosted mode (GITMOTE_ACT_PLATFORMS in Dockerfile.runner): steps execute directly in the ephemeral Job container — the Job is the sandbox, no Docker-in-Docker. Every tool a workflow needs (git, node, bash) must live in the runner image. Locally, where a real daemon exists, that var is unset and act keeps its default nested-container behavior.

Data model

Three leader-only, litestream-replicated tables in internal/meta: ci_runs, ci_jobs, ci_secrets. Logs are append-only blobs under a ci/ key space within the storage root (ci/{repoID}/{runID}/{jobID}.log), PUT before the ci_jobs pointer is recorded, with a size cap (explicit truncation marker, never silent).

The runner-facing report API lives under /internal, apart from the browser UI (internal/ci/report.go): GET /internal/ci/jobs/{id} claims a job (queued→running), POST …/log appends a live-log chunk (below), POST …/complete uploads the log and sets a terminal status (idempotent). All do a constant-time WORKER_SECRET compare; only the leader writes completions (a follower returns a retryable 503). A leader-only reconcile ticker sweeps jobs stuck in running to error.

Live logs

The durable log above is shipped once, at completion — so while a job runs there is otherwise nothing to watch. A best-effort live tail fills that gap without touching the durable path. The runner tees act's combined output and, on a ~1 s ticker, POSTs the delta to …/log; the leader appends it to an in-memory, per-job buffer (LiveLogs) capped like the durable blob. The UI's job-log page polls GET /{repo}/runs/{id}/job/{jid}/log/live?offset=N ({bytes, next, done}) and, on done, reloads to the rendered durable log.

It is deliberately never replicated or persisted: a lost chunk or a leader restart only blips the live view — the runner still ships the whole log at completion, so the durable record is unaffected. This is the fire-and-forget discipline again: the tail is a UX nicety, never a correctness path. The buffer is leader-only (the browser polls the same instance the runner streams to) and swept after a short idle TTL on the reconcile tick.

Secrets

Per-repo CI secrets, encrypted at rest, decrypted only to inject at trigger. The crypto and its narrow threat model are in safety.md §5: AES-256-GCM under a server-held master key (GITMOTE_CI_SECRET_KEY_V<n>), a per-repo HKDF subkey, AAD binding (repoID, name, version). It protects against a leaked S3 replica / DB snapshot — not a compromised running server, which must hold the key to decrypt. The dispatcher passes each secret as GITMOTE_CI_SECRET_<NAME>; the runner forwards it to act as -s <NAME>, which reads the value from its env — so the value never touches the act argv and the workflow sees ${{ secrets.NAME }} as on GitHub. Values are write-only in the UI (only names shown) and never logged.

Env-seeded secrets (a second source). Beside the encrypted UI/DB secrets, the dispatcher overlays any per-repo secrets supplied in the server's own environment as GITMOTE_REPO_SECRET_<REPO>__<NAME> (repo uppercased, -→_). These are plain operator config — never stored, never encrypted, and needing no master key — so they're invisible to the UI and win over a same-named DB secret. The host env is already the CI trust boundary (safety.md §7), so this adds no exposure; it exists so the automated self-deploy (make self-deploy) can carry its Scaleway/GHCR secrets in a gitignored .env instead of the UI.

Clone auth

The runner clones over ordinary git-HTTPS with a per-run, read-only, repo-scoped, expiring token minted by the leader at dispatch — the same authenticated path as any client, no CI backdoor. It checks the SHA out onto its branch (checkout -B) so github.ref resolves.

Safety

Container image builds

Image builds are substrate-dependent, and off by default:

Limitations