Ga naar inhoud

Architecture — how the Codeberg CI platform works

Read this first if you're new. It ties together the per-component docs (docs/components/*.md) and names the cross-repo dependencies that aren't obvious from this repo alone. For step-by-step enrollment see docs/onboarding.md; for when things break see docs/troubleshooting.md.

What this is

talos runs Codeberg Actions CI on the con-prod cluster (Gardener/Calico, namespaces con-ci and con-ci-oci). Forgejo runners poll Codeberg, pick up jobs from Conduction repos, and run them on-cluster — behind a default-deny egress with an allowlisting forward-proxy. Everything is GitOps: ArgoCD syncs manifests/overlays/con-prod (app talos) and manifests/overlays/con-ci-oci (app talos-oci) from main. Only secrets are applied by hand (SOPS, out-of-band).

Current capability status (2026-07-06)

Works today: - con-ci (host runner): polls Codeberg and runs jobs end-to-end, in-cluster, behind the default-deny egress proxy. Shell steps, run:, checkout, tool downloads through the proxy. - Egress allowlist enforcement (squid), auto-rolling on config change. - con-ci-oci (container runner): container: jobs proven end-to-end — claim, pull through its own egress-proxy, run inside the image, and podman build — rootless on overlay + fuse-overlayfs (the earlier "mount wall" was the containerd AppArmor profile, not mount propagation; resolved 2026-07-04, see docs/troubleshooting.md). - Secrets via SOPS+age; org-scope runner registration; the label guardrail.

Decided (2026-07-04): image builds run on con-ci-oci (podman build), not on the host runner — the stock runner image ships no buildah and host jobs are non-root without a package manager. con-ci stays the plain host class (scripts, lint, tooling, the sign step); the dormant runner-build patch remains until its change is archived.

The mental model of one job

  1. A repo has a workflow with runs-on: con-ci (or con-ci-oci) in .forgejo/workflows/.
  2. A forgejo-runner in con-ci long-polls Codeberg (through the egress-proxy) and claims the job. There is one org-scope registration per runner (uuid+token in a SOPS-encrypted config), not per-repo tokens.
  3. The runner fetches every uses: action. Bare actions/checkout@v4 resolves to data.forgejo.org (Forgejo's mirror), NOT github.com — so that host must be allowlisted in the proxy.
  4. All egress (action clone, image pulls, registry push, API calls) goes through the egress-proxy (squid, FQDN allowlist). Anything not on the list is refused (403).
  5. The job runs; results/artefacts/registry pushes go back through the proxy.

Two runners — pick the right label

Label Backend Use for
con-ci host/local (the runner container itself) plain host jobs: scripts, lint, tooling, the Nextcloud sign step
con-ci-oci rootless podman service (Docker-compatible socket) in the same pod anything involving container images: container:/services: jobs, Hydra personas, and image builds (podman build)

Guardrail: scripts/check-forgejo-runs-on.sh (a pre-commit hook) blocks workflows that use an unserved label or put container: on con-ci. Heuristic (revised 2026-07-04): plain host job → con-ci; anything involving container images — running AND building — → con-ci-oci.

The con-ci-oci runner uses STORAGE_DRIVER=overlay + fuse-overlayfs (vfs is retired — see troubleshooting: the mount denial was AppArmor, not a rootless limitation).

Components in this repo (manifests/)

  • base + components/runner — the con-ci host runner Deployment (v12 connection model: uuid+token config, no register, non-root).
  • components/runner-build — a Kustomize Component that patches the host runner to add rootless buildah (subuid/subgid, BUILDAH_ISOLATION=chroot, vfs). Enabled in the overlay.
  • components/runner-oci — the con-ci-oci runner (2 containers: rootless podman + forgejo-runner). Has its own overlay (overlays/con-ci-oci: own namespace + egress proxy/allowlist), synced by its own ArgoCD Application talos-oci (selfHeal — manual applies get reverted). See docs/components/runner-oci.md.
  • components/egress — default-deny NetworkPolicies + the egress-proxy (squid). The allowlist is squid.conf, delivered via configMapGenerator (content-hashed name → editing it makes ArgoCD roll the squid pod; see troubleshooting for why that matters).
  • components/isolation — east-west NetworkPolicies + ResourceQuota/LimitRange.
  • components/runner-secrets — SOPS+age templates (delivered out-of-band, see below).
  • overlays/con-prod — what ArgoCD syncs; aggregates the buildable components, pins the squid image.

Cross-repo dependencies (easy to miss)

The con-ci-oci runner does not work from this repo alone. It depends on two components in the Conduction/cluster-infra repo (synced by a separate ArgoCD cluster-infra project into kube-system):

  1. fuse-device-plugin — advertises the extended resource squat.ai/fuse on pool worker-0b1p9-1. (Currently unused because both runners are on vfs; kept for the future fuse-overlayfs switch.)
  2. seccomp-profiles — a DaemonSet that installs podman-rootless.json (podman's own default seccomp profile) to /var/lib/kubelet/seccomp/ on pool worker-0b1p9-1. The con-ci-oci podman container references it as a Localhost seccomp profile so it can create user namespaces (the containerd RuntimeDefault profile gates that behind CAP_SYS_ADMIN, and PodSecurity baseline forbids Unconfined).

Because the seccomp profile only exists on worker-0b1p9-1, the con-ci-oci pod pins there with a nodeSelector. If you move CI to another pool, the cluster-infra DaemonSets' node scope must move too.

Security posture (con-ci-oci, non-privileged)

Rootless podman needs four relaxations (never --privileged, no host docker socket): - Localhost seccomp profile (userns clone). - allowPrivilegeEscalation: true (so the setuid newuidmap/newgidmap can run). - capabilities.add: [SETUID, SETGID] (only these two; everything else stays dropped). - appArmorProfile: Unconfined on the podman container (the containerd default AppArmor profile denies mount(2) even inside the pod's own userns). This one is above baseline PSA, so the con-ci-oci namespace enforces privileged (warn/audit stay restricted) — see the trade-off and the Localhost-AppArmor end state in docs/components/runner-oci.md.

The host con-ci runner needs none of these — buildah with chroot isolation avoids user namespaces entirely.

Secrets model

  • Runner connection (forgejo-runner-conn, forgejo-runner-oci-conn): org-scope uuid+token in a config.yml, SOPS+age-encrypted in git, applied out-of-band (sops -d | kubectl apply -f -) until ArgoCD KSOPS exists. Codeberg only holds the runner identity, never a deploy credential.
  • Signing key (Nextcloud): crown jewel, decrypted only at runtime inside the signing job, never a Codeberg Actions secret.
  • Registry-push token / Anthropic key may be ${{ secrets.X }} at Codeberg (non-crown-jewel).

GitOps flow (how to ship a change)

  1. Edit manifests on a branch or main, commit.
  2. Push to Codeberg (origin, SSH — see troubleshooting: HTTPS/IPv4 is broken on the ops workstation).
  3. ArgoCD auto-syncs main. It polls ~every 3 min; force with a hard refresh (kubectl -n argocd annotate application <app> argocd.argoproj.io/refresh=hard --overwrite).
  4. For the egress allowlist specifically: the squid pod rolls automatically thanks to the content-hashed ConfigMap. Other stateful reloads may need a rollout restart.

Cluster-infra changes (fuse, seccomp) follow the same flow but in the Conduction/cluster-infra repo under the cluster-infra ArgoCD project; each needs a one-time kubectl apply -f argo/applications/<app>.yaml bootstrap.