Ga naar inhoud

runner-oci (runner-container-backend)

Purpose: a second runner with a rootless container backend, label con-ci-oci, so jobs that run inside a container image (container:) and Hydra personas can run. Incremental (Option 2): the host con-ci runner stays untouched.

Deploys (manifests/components/runner-oci/): a forgejo-runner-oci Deployment + SA. Two containers in one pod: - podman — a rootless podman API service (Docker-compatible socket), STORAGE_DRIVER=overlay + fuse-overlayfs (forced via the mounted rootless storage.conf; needs /dev/fuse from the squat.ai/fuse device plugin). NEVER --privileged, no host docker socket. - runner — forgejo-runner with the docker backend (DOCKER_HOST → the podman socket), advertising con-ci-oci. Reads its own connection from SOPS secret forgejo-runner-oci-conn.

Key facts: - Needs its own org-scope registration (a 2nd "Create new runner" → 2nd uuid+token) — a daemon can't share the host runner's uuid. - Job containers run in the runner's own namespace con-ci-oci (own overlay: manifests/overlays/con-ci-oci with its own default-deny egress + squid allowlist + quota). The allowlist deliberately excludes apps.nextcloud.com — the signing key never enters this build class (see docs/components/ci-flows.md). - Non-privileged rootless relaxations: Localhost seccomp profile (podman-rootless.json, from the cluster-infra seccomp-profiles DaemonSet) for userns clone + mount; allowPrivilegeEscalation: true + default-set caps for newuidmap / layer extraction; and appArmorProfile: Unconfined on the podman container (see below — the one relaxation that is above baseline PSA). - nodeSelector: worker.gardener.cloud/pool=worker-0b1p9-1 — the pool where the seccomp profile is installed (the podman container can't start without the profile file on the node). - Job-container seccomp (BuildKit enablement) — see the section below: a second, targeted profile for the job containers themselves, delivered via ConfigMap + container.options, inert until the SOPS config.yml opts in.

Status: LIVE, GitOps (since 2026-07-06 via the talos-oci ArgoCD Application). container: jobs proven end-to-end (claim → pull via own egress-proxy → run inside the image) and podman build works rootless — see the RESOLVED section below. The overlay manifests/overlays/con-ci-oci is synced (selfHeal) from main — a manual kubectl apply -k gets reverted by Argo; change the allowlist/manifests via git.

Bring-up: 1. 2nd org token → fill forgejo-runner-oci-conn (SOPS) and apply it. 2. kubectl apply -k manifests/components/runner-oci. 3. kubectl -n con-ci logs deploy/forgejo-runner-oci -c runner -f (and -c podman). 4. Test: a workflow runs-on: con-ci-oci + container: node:20-bookworm.

Cross-repo dependency: the podman container references a Localhost seccomp profile that lives on the nodes, installed by the seccomp-profiles DaemonSet in the Conduction/cluster-infra repo. See docs/architecture.md and docs/troubleshooting.md.

RESOLVED (2026-07-04) — the mount wall was AppArmor, not mount propagation

The long-standing mount … permission denied that forced vfs and deferred fuse-overlayfs was misdiagnosed as a userns/mount-propagation limit. The probe ladder (manifests/components/runner-oci/TEST-standalone-build-pod.yaml → privileged diagnostic, TEST-peelback-sysadmin-pod.yaml → peel-back) proved:

  • privileged + hostUsers:false → run (A) and build (B) work. SYS_ADMIN alone did not — which looked like "only full privilege works", but the real reason privileged worked is that privileged pods run AppArmor-unconfined.
  • the containerd default AppArmor profile denies mount(2) even inside a userns the pod owns. With only appArmorProfile: Unconfined on the podman container — no privileged, no SYS_ADMIN, hostUsers:false intact, baseline caps, Localhost seccomp — A and B are both green on overlay+fuse-overlayfs.

Consequences: - STORAGE_DRIVER=overlay + fuse-overlayfs is the active config (vfs story retired). - Image builds work with podman build — note quay.io/podman/stable ships no buildah binary; buildah is vendored inside podman, so flows must use podman build, not buildah bud. - PSA: AppArmor Unconfined is above baseline (baseline allows only RuntimeDefault/Localhost), so the con-ci-oci namespace enforces privileged (warn/audit stay restricted) — a single relaxation on a dedicated, tainted, own-egress pool; see the rationale comment in overlays/con-ci-oci/namespace.yaml. Cleaner end state (TODO): a custom Localhost AppArmor profile that allows mount, installed on the ci-oci pool by cluster-infra exactly like the seccomp profile — then enforce returns to baseline. - Autoscale-from-zero: squat.ai/* requests block 0→1 scale-up of the ci-oci pool (fresh node doesn't advertise the resources yet → autoscaler "Insufficient …"). Fix via Gardener worker-pool nodeTemplate.capacity, or pool min 1.

Job-container seccomp — BuildKit enablement (staged 2026-07-07)

Rootless BuildKit inside a job container needs the mount/unshare/clone(CLONE_NEWUSER) syscall family, which podman's default job profile gates behind CAP_SYS_ADMIN. Three runner facts (source-verified at forgejo-runner v12.12.0) shape the solution:

  • Workflow-level container.options passes a hardcoded allowlist (volume/tmpfs/ memory/hostname/user/group-add/health only); --security-opt is parsed then silently discarded — no config toggle exists. Jobs can never grant themselves relaxations.
  • Runner-config container.options is fully honored and applies to every job/service/ step container of the process; it cannot vary per label.
  • A seccomp=<path> option is resolved client-side: the runner reads the file from its own container fs and inlines the JSON — hence a plain ConfigMap mount, no node installer.

Mechanism: runner-oci-job-seccomp ConfigMap (buildkit-job.json, a verbatim copy of podman-rootless.json — the same profile family the podman engine itself runs under) is mounted into the runner container at /etc/talos/buildkit-job.json. Activation is the container.options: "--security-opt seccomp=/etc/talos/buildkit-job.json" line in the SOPS config.yml (see the secret example). Without that line the mount is inert. AppArmor needs no extra relaxation at the job layer: rootless podman doesn't apply AppArmor to child containers (they inherit the engine container's profile).

Rollback: remove the options line from the SOPS config.yml + restart → job containers return to podman's default confinement; the ConfigMap/mount can stay (inert) or be reverted with the manifest commit. Never container.privileged.