# Operations Canonical infra doc — resources, env, deploy, and the one operational rule. Keep it current whenever infra or env changes. gitmote is `gitmote.atmin.net` — a subdomain of atmin.net by DNS only. It is otherwise independent; it currently happens to run on Scaleway Serverless Containers like atmin.net, but that is incidental and may change. ## Infrastructure | Component | Provider | Product | |-----------|----------|---------| | Compute | Scaleway | Serverless Containers (`fr-par`, custom domain via CNAME, auto TLS) | | Object storage | Scaleway | Object Storage (S3-compatible), bucket `gitmote` | | Registry | GitHub | GHCR — **public** `ghcr.io/atmin/gitmote` (server) + `ghcr.io/atmin/gitmote-runner` (CI). Scaleway pulls both anonymously | Reachable at **`gitmote.atmin.net`** (CNAME → the container endpoint). EU-resident infrastructure, consistent with atmin.net's stance. Everything lives under one **storage root** — the whole bucket, or a `bucket/base` sub-path when sharing a bucket (see `GITMOTE_S3_BUCKET`). Under the root: `{repo}/objects/` (immutable git objects/packs), `ci/` (run logs), and `meta/` (the litestream WAL of the metadata SQLite). Refs are the source of truth and live in the metadata DB; objects live beside it, all under the same root. ## ⚠️ Single writer is a correctness invariant [safety.md §1](architecture/safety.md) — **never two writer instances.** gitmote replicates its metadata SQLite to S3 with litestream; two instances writing the same WAL can corrupt the replica. This is stronger than a stateless service's in-process-race concern: it is a data-integrity precondition. The invariant is enforced **by a lease, not by procedure.** The server opens its metadata in s3lite `RoleAuto`: it becomes the writer only while it holds a lease (litestream's `s3.Leaser` — an `If-None-Match`/`If-Match` conditional-write CAS on `lock.json` beside the metadata replica, at `{root}/meta/lock.json`), and runs as a read-only follower otherwise. This makes **atomic conditional writes a hard requirement on the S3 provider**: the lease — and therefore the single-writer correctness invariant — holds only if the backend supports the compare-and-swap primitive (`If-None-Match` to acquire, `If-Match` to renew/release). Most do (AWS S3, Scaleway, …); one that doesn't cannot safely back gitmote. gitmote gates every metadata-derived response on the writer lease (`leaderGate`): a follower **promotes on demand** — the request triggers an immediate acquisition (`s3lite.TryPromote`) and, once it restores the latest state, serves — and refuses with a retryable `503` only when the lease is still held elsewhere (liveness probes and static assets are always served). - **Rolling deploys are safe by construction.** A rolling deploy briefly runs a second container alongside the old one — two instances live at once — which does **not** violate the single-writer invariant. The new instance boots as a **follower** (the old still holds the lease); the old releases on its graceful SIGTERM (`Close` flushes, then releases the lease last); the new then **promotes** on demand — the first metadata request to reach it after the old releases acquires the freed lease (`TryPromote` blocks ≤ `promoteTimeout` for the restore, then serves) — or, absent traffic, on its next background lease poll. Never two writers — no drain step, no `POST /admin/quit`. **No handoff `503` on a graceful deploy:** the triggering request blocks briefly for the restore instead of being refused. A `503` remains only when the lease is genuinely held — after a *hard* kill the successor can't acquire until the ≤30 s TTL expires, so requests in that window get the retryable `503`; a graceful exit releases at once. - **`max-scale = 1`** still holds, re-asserted on every deploy. Not for the writer invariant anymore (the lease covers that) but because a follower serves a *restored snapshot* and polls — it is not a continuously-fresh read replica — so scaling out for read throughput needs follower-freshness work first. Raising it is the tracked follow-up ([reader-writer-split](evolution/reader-writer-split.md)); until then keep it at 1. `min-scale = 0` (idle to zero) is orthogonal — cost/latency only. - By the content-before-pointer ordering ([safety.md §3](architecture/safety.md)), even a hypothetical overlap could only leak orphan objects (garbage `gc` reclaims), never a ref pointing at a missing object. The [safety.md §4](architecture/safety.md) escape hatch (refs behind a conditional-PUT CAS on object storage) would rely on the same conditional-write primitive; gitmote's ref CAS remains a SQL transaction in the metadata DB (unchanged — it never needed an S3 precondition). ## Runtime env vars (on the container) These configure the **running server** and are set **on the Scaleway Serverless Container** (via `scw container container create/update`, secrets as `secret-environment-variables`). They are **not** the CI credentials — the deploy pipeline's Scaleway API keys are separate GitHub Actions secrets (next section). Setting these on the container does nothing for CI, and vice versa. > ⚠️ **`update` replaces the entire `secret-environment-variables` map — it does > not merge.** Passing one secret wipes the rest (observed: setting a single > secret dropped the AWS keys, so litestream fell back to EC2 IMDS and the server > crash-looped on restore). **Always pass *all* secret env vars together on every > `update`.** Plain `environment-variables` are a separate map, unaffected. The CI > deploy's `update` sets no env vars, so it preserves both maps. **Required — the whole core.** A storage root plus its credentials is a complete forge. | Variable | Value / meaning | |----------|-----------------| | `GITMOTE_S3_BUCKET` | the **storage root**: `gitmote` (a whole bucket) or `bucket/base` to scope the forge under one sub-path of a shared bucket. Derives the metadata replica (`{root}/meta`) and the single-writer lease | | `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | secret — S3 key pair; covers both the object store and the litestream replica (Scaleway API key on prod) | | `AWS_REGION` | region; defaults to `us-east-1` (ignored by most S3-compatible stores). **Set it for real AWS S3** (e.g. `fr-par` on Scaleway) | | `GITMOTE_S3_ENDPOINT` | `https://s3.fr-par.scw.cloud` — the S3 endpoint; **omit for real AWS S3** | **Auto-managed — leave unset; a sensible default applies and an explicit env still wins.** Not part of the required surface. | Variable | Value / meaning | |----------|-----------------| | `GITMOTE_COOKIE_KEY` | signs management-UI session cookies; auto-generated + persisted in meta (stable across restart / scale-to-zero) | | `WORKER_SECRET` | runner report-back auth; auto-generated + persisted in meta **when a CI trigger is configured** (below) | | `GITMOTE_DATA` | base dir for the db (`meta.sqlite3`), cache, and socket; **preset to `/data` in the image** — mount a volume there to keep the cache across restarts. Ephemeral by design (restored from S3 on cold start) | **CI — optional, off by default** (runs are still recorded; nothing executes until one of these turns a trigger on — see [CI substrate](#ci-substrate)). | Variable | Value / meaning | |----------|-----------------| | `SCW_CI_JOB_DEFINITION_ID` | the Scaleway Serverless Job definition. **Set → cloud trigger** (a Scaleway job per CI job); requires `GITMOTE_URL` | | `GITMOTE_URL` | public base URL (`https://gitmote.atmin.net`) the runner clones + reports back to; required for any CI. **Set alone → local `act` trigger** | | `SCW_SECRET_KEY` | secret — Scaleway API secret key (UUID) to start the job (the same key also deploys the container from CI) | | `SCW_REGION` | region for the job start (falls back to `AWS_REGION`) | | `GITMOTE_CI_SECRET_KEY_V` | secret — base64 of 32 bytes; **env-only** master key for per-repo CI secrets, never persisted ([safety.md §5](architecture/safety.md)). Add `_V2`, … to rotate (highest is current; old envelopes still decrypt). Unset → the secrets UI is off and none are injected | | `GITMOTE_CI_ALLOW_BUILDS` | `1`/`true` → let workflows build container images on a **daemon-backed local/VPS** runner (leaves `act`'s host-Docker-socket mount in place). ⚠️ **Hands untrusted workflow code the host daemon (host root) — trusted repos only** ([safety.md §7](architecture/safety.md)). Unset/false (default) → the socket mount is suppressed. No effect on the Scaleway substrate (can't build regardless) | `GITMOTE_DATA` is ephemeral on purpose: the object store + litestream replica are the durable state, and the local disk is a cache. On a cold start (scale-to-zero → wake, or a redeploy) the DB is restored from the derived replica and repos re-materialize from the bucket. This path is proven locally by `make e2e-restore` (wipes the data volume, confirms the repo still clones). Resources: **1000 mVCPU / 1024 MB / 8 GB ephemeral** (`cpu-limit=1000`, `memory-limit-bytes=1024MB` — the CLI requires a `G`/`GB`/`MB` unit). Observed usage sits well under these; 256 MB was already plenty for `git index-pack` / `receive-pack`. The CPU is set high to shorten cold start — the litestream restore and the first git operation run unthrottled instead of at a fraction of a core — and 1024 MB unlocks the larger ephemeral scratch tier (8 GB `/tmp`), not for RAM headroom. Tune from logs. Scale: **`min-scale = 0`** (idle to zero). A remote pushed to occasionally is idle almost always, and even at 1024 MB always-on exceeds Scaleway's free tier (~2.6M GB-s/mo vs 400k), so scale-to-zero is the cheaper default. The cost is a few seconds of cold start on the first request after idle (restore + materialize, the path above); scale-down is a graceful SIGTERM, so the shutdown `Close` durably flushes the WAL (and releases the writer lease) — no lost writes. Set `min-scale = 1` only if instant first-push latency is worth the always-on cost. `max-scale` stays **1** regardless (see the single-writer section — follower reads aren't fresh enough to scale out yet). ## CI substrate gitmote always records a run per pushed workflow; **what executes it is chosen from the env at startup** (one runner, three substrates — see [ci.md](architecture/ci.md)): 1. `SCW_CI_JOB_DEFINITION_ID` set → **Scaleway Serverless Jobs** (cloud): a job per CI job. Also requires `GITMOTE_URL` (the runner clones + reports back there); `WORKER_SECRET` is auto-provisioned. The server refuses to start if the job definition is set without `GITMOTE_URL`. 2. else `GITMOTE_URL` set → **local `act`**: the server spawns `gitmote-runner` as a local process — the *same* runner code and env contract, a local substrate instead of a cloud job (this is what `make dev` uses). 3. else → **disabled**: runs and jobs are recorded, but nothing executes. **Prerequisite for both executing substrates:** the runner runs `.gitmote/workflows` with [`act`](https://github.com/nektos/act), which needs a reachable **Docker or podman daemon** — in the cloud that's the Serverless Job image; locally it's the daemon on the host (the same one MinIO uses under `make dev`). **No daemon → every CI run fails visibly**: the job is still dispatched and recorded, but act fails at its "Set up job" step with a `check if … the daemon is running` error and the run is marked failed — no workflow step (and no build) executes. A leader-only ticker sweeps jobs stuck `running` past ~1h back to `error`. **Container image builds (local/VPS only).** With `GITMOTE_CI_ALLOW_BUILDS` set, `act`'s default behaviour of mounting the host Docker socket into each job container is kept, so a workflow step (`docker build`, `docker/build-push-action`, …) reaches the daemon and builds images — including gitmote's own. Off by default because it gives untrusted workflow code the daemon, i.e. host root ([safety.md §7](architecture/safety.md)); turn it on only on a host running trusted repos. The runtime follows `DOCKER_HOST`: plain Docker needs nothing extra; **podman** must expose its Docker-API socket (`systemctl --user enable --now podman.socket`, then `DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock`) — a *daemonless* podman cannot back `act`. The Scaleway substrate ignores the flag (self-hosted, no daemon). ## CI secrets (GitHub Actions) Separate from the container env above: these are the Scaleway API credentials the deploy pipeline uses, set in the **GitHub repo** → Settings → Secrets and variables → **Actions** → **Repository secrets**. A missing one doesn't error at reference time — GitHub substitutes an empty string, so a wrong name surfaces later as e.g. `scw … api_key not_found`. The image push itself needs **no secret**: publishing to GHCR uses the built-in `GITHUB_TOKEN` with the workflow's `packages: write` permission. The Scaleway secrets below are only for the `scw container update` deploy (and starting CI jobs). A Scaleway API key has two halves — don't swap them: the **access key** is `SCWXXXXXXXXXXXXXXXXX` (starts with `SCW`), the **secret key** is a UUID (`xxxxxxxx-xxxx-…`); both must come from the gitmote Organization/Project. | Secret | Value | |--------|-------| | `SCW_SECRET_KEY` | Scaleway API **secret key** — the UUID, not the `SCW…` access key (deploy + start CI job) | | `SCW_ACCESS_KEY` | Scaleway **access key** (`SCW…`) | | `SCW_ORGANIZATION_ID` | Scaleway organization ID | | `SCW_PROJECT_ID` | Scaleway project ID | | `SCW_CONTAINER_ID` | the gitmote Serverless Container ID | If a secret reads as empty despite being set, check: it's under **Secrets** not **Variables**; it's a **Repository** secret (a job with no `environment:` can't see Environment secrets); the name matches exactly (case-sensitive); and, if kept as an **Organization** secret, the `gitmote` repo is granted access to it. ## One-time Scaleway setup ```bash # 1. Object Storage bucket (console or CLI), fr-par, name "gitmote". # Generate an API key pair for it. # 2. Publish the initial images to GHCR, then flip both packages to PUBLIC once by # hand (GitHub → your profile/org → Packages → gitmote / gitmote-runner → # Package settings → Change visibility → Public). New GHCR packages default to # private; until public, Scaleway's anonymous pull fails with an error the code # can't diagnose. CI republishes on every push (server) and on runner changes, # so this manual build is only the very first seed. Scaleway requires amd64. echo "" | docker login ghcr.io -u atmin --password-stdin docker build --platform=linux/amd64 -t ghcr.io/atmin/gitmote:master . docker push ghcr.io/atmin/gitmote:master docker build --platform=linux/amd64 -f Dockerfile.runner -t ghcr.io/atmin/gitmote-runner:master . docker push ghcr.io/atmin/gitmote-runner:master # 3. Create the container (single writer: min-scale=0 idle-to-zero, max-scale=1). # Arg names per the current CLI: image= (not registry-image), cpu-limit (mVCPU), # memory-limit-bytes, and key-based (secret-)environment-variables.KEY=value. # Only the credentials are secret; the rest are plain environment-variables. The # cookie key and worker secret are NOT set — they auto-generate and persist. The # ephemeral /tmp scratch tier is coupled to the memory tier — 1024 MB unlocks # 8 GB scratch (set in the console). scw container container create \ namespace-id= \ name=gitmote \ image=ghcr.io/atmin/gitmote:master \ min-scale=0 max-scale=1 \ cpu-limit=1000 \ memory-limit-bytes=1024MB \ port=8080 \ environment-variables.GITMOTE_S3_BUCKET=gitmote \ environment-variables.GITMOTE_S3_ENDPOINT=https://s3.fr-par.scw.cloud \ environment-variables.GITMOTE_DATA=/tmp/gitmote \ environment-variables.AWS_REGION=fr-par \ secret-environment-variables.AWS_ACCESS_KEY_ID= \ secret-environment-variables.AWS_SECRET_ACCESS_KEY= # 4. Custom domain — two parts: # (a) DNS: add a CNAME gitmote.atmin.net → .scw.cloud # (b) Register the domain ON the container so Scaleway issues a managed cert # (the CNAME alone is not enough — until this, HTTPS serves the endpoint's # own cert and browsers report "not secure"): scw container domain create container-id= hostname=gitmote.atmin.net scw container domain list container-id= # status pending → ready (minutes) # Note: "/" is the web dashboard (200); for a plain liveness probe use # /healthz. Git lives under //…. # 5. CI runner Serverless Job — point its definition at the public GHCR runner # image (one image serves all repos; per-job env is injected at trigger). Set # the resulting definition ID as the container's SCW_CI_JOB_DEFINITION_ID to # turn on the cloud CI trigger. Update the image the same way after a runner # change (CI republishes the tag; the Job pulls it on next start): scw jobs definition create name=gitmote-runner \ image-uri=ghcr.io/atmin/gitmote-runner:master \ cpu-limit=1000 memory-limit=2048 scw jobs definition update image-uri=ghcr.io/atmin/gitmote-runner:master # 6. Set the GitHub Actions repository secrets for the deploy pipeline — these are # the CI credentials, NOT container env vars. See "CI secrets (GitHub Actions)". ``` ## Bootstrap (first run is automatic) An empty instance has no admin/token. **On first start the server auto-bootstraps itself**: when it is the writer and no admin exists, it creates the admin (handle `admin`), mints a token, and prints it once to the logs behind an unmissable banner. So a fresh `docker run` against an empty bucket is usable without a second command — just grab the token from the logs. The token **transits the logs**, which are operator-visible (same trust boundary as the env and bucket). Treat it as a one-time credential: after first sign-in, mint your own token in the UI and revoke the bootstrap one (see below). No initial repo is created — make repos in the UI in two clicks. ### Manual bootstrap / token recovery Both run **from your machine against the bucket** so you are transiently the single writer — do this *before* the container serves traffic (or while it is scaled to zero). They open in `RoleWriter`: acquire the writer lease and fail loudly if the server already holds it, so they can never race the live writer. ```bash # Common env for both: export GITMOTE_S3_BUCKET=gitmote \ GITMOTE_S3_ENDPOINT=https://s3.fr-par.scw.cloud \ GITMOTE_DATA=/tmp/bootstrap \ AWS_REGION=fr-par AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= # Bootstrap by hand (optional -repo). Idempotent — refuses to clobber an admin: gitmote bootstrap -handle admin # Lost the token? Mint a fresh one for the existing admin (safe: whoever can run # this against the bucket already has total infra control): gitmote bootstrap -reissue -handle admin ``` Each prints the access token once (save it) and durably flushes the new state to `s3://gitmote/meta` on `Close`; the server restores it on next start. Only the token's hash is ever stored — the raw token is never recoverable, hence `-reissue`. ## Deploying Automatic on every green push to `master` (`.github/workflows/ci.yml`, `deploy` job): `ci` (lint/test/build) → `e2e` (local push/clone + litestream restore) → build+push the public amd64 image to `ghcr.io/atmin/gitmote:{master,}` → `scw container container update image=ghcr.io/atmin/gitmote: … min-scale=0 max-scale=1 --wait` (Scaleway pulls the public image). No drain step: the writer lease makes the rolling deploy safe by construction (new boots as a follower, old releases on SIGTERM, new promotes — see the single-writer section). The **runner** image publishes separately on runner/Dockerfile.runner changes (or a release) via `.github/workflows/publish-runner.yml` → `ghcr.io/atmin/gitmote-runner`; the CI Job definition pulls it on next start. ```bash git push origin master ``` **Break-glass: self-deploy from a local instance (`make self-deploy`).** When GitHub is unavailable (or an out-of-band deploy is wanted), a local gitmote instance builds and deploys itself via [`.gitmote/workflows/deploy.yml`](../.gitmote/workflows/deploy.yml) — the GitHub deploy job gated on the `self-deploy` branch instead of `master`. It runs only on a local instance (Scaleway can't build). `make self-deploy` automates it end to end: copy [`.env.example`](../.env.example) to `.env` (gitignored) and fill in the deploy secrets, then run it. The target starts a throwaway instance on `:8081` with its own MinIO (`:9200`, separate from `make dev`), `GITMOTE_CI_ALLOW_BUILDS=1`, and the `.env` secrets seeded per-repo via `GITMOTE_REPO_SECRET_GITMOTE__` — then pushes `HEAD` to `self-deploy`, which triggers the workflow. These env secrets are **never stored and need no `GITMOTE_CI_SECRET_KEY`** (that key only guards the UI/DB secrets path): the host env is the trust boundary ([safety.md §7](architecture/safety.md)). Requires `act` and a running Docker daemon — the script resolves the socket into `DOCKER_HOST` from the active docker context (so colima and Docker Desktop both work) and fails with setup steps if none is up. On Apple Silicon, `colima start --vm-type vz --vz-rosetta` gives Rosetta-accelerated `linux/amd64` builds. The secrets to put in `.env` are the same Scaleway five the GitHub pipeline uses (below) **plus** `GHCR_USER`/`GHCR_TOKEN` (a GHCR PAT with `write:packages` — there is no built-in `GITHUB_TOKEN` off GitHub). The build is `linux/amd64` only (Scaleway's arch; emulated on an arm64 dev host, so the first build is slow); it tags `:self-deploy` + `:` and deploys the SHA tag — never `:master`, which the GitHub deployer owns. Confined to `self-deploy`, it never double-deploys with the `master` path, and the writer lease keeps its rolling swap safe exactly as the GitHub deploy's. The instance is disposable — `make self-deploy` resets its throwaway state (the `data-selfdeploy/` dir and its own MinIO volume) at the start of every run, so an interrupted prior run can't leave stale litestream state drifted from the bucket; the previous run's state survives only until the next run. `make self-deploy-reset` wipes it on demand. ## Verifying it hosts itself ```bash # Push the gitmote repo over HTTPS to the deployed instance: git push https://admin:@gitmote.atmin.net/gitmote HEAD:refs/heads/master git clone https://admin:@gitmote.atmin.net/gitmote /tmp/gitmote-clone git -C /tmp/gitmote-clone fsck --full # clean ``` Cold-start check: trigger a redeploy (or let it idle to zero and wake it), then clone again — the metadata restores from the replica and the repo serves. The same path is exercised locally by `make e2e-restore`. ## Troubleshooting ```bash scw container container get # status, error messages, instance count scw container container redeploy # re-pull the current image tag # Images live in GHCR — inspect/pull them there: docker manifest inspect ghcr.io/atmin/gitmote:master # server image docker manifest inspect ghcr.io/atmin/gitmote-runner:master # CI runner image ``` If a deploy fails on an image pull, the most likely cause is a **private** GHCR package: flip `gitmote` / `gitmote-runner` to public (see one-time setup step 2). Logs flow to Scaleway Cockpit (console → Observability → Cockpit → Grafana → Explore → Loki). gitmote logs JSON to stderr. **Grab the bootstrap token from the logs.** The auto-bootstrap and `-reissue` print the token once (above); on Scaleway that "print" is a log line. This LogQL pulls just the token out — a gitmote PAT is `gmt_.`, both halves hex ([`internal/auth/token.go`](../internal/auth/token.go)): ```logql {resource_type="serverless_container", resource_name=~"atmina71bb4d5-gitmote"} |= "gmt_" | regexp `(?Pgmt_[0-9a-f]+\.[0-9a-f]+)` | line_format `{{.token}}` ``` `|= "gmt_"` filters raw lines first (no `| json` needed — the token appears literally in the line); `regexp` captures it; `line_format` collapses each match to the bare token. Set the time range to cover the container's last boot (the token is minted once at start) and sort newest-first so the top line is current.