Ga naar inhoud

Risk acceptance — CI on the production cluster (con-prod)

Status: DRAFT, pending sign-off by Robert & Ruben. Authored 2026-06-29 from the as-built platform. Review at least annually or on any change to the isolation/egress model.

Context

Talos runs CI runners (forgejo-runner) for Codeberg Actions on the production cluster con-prod, in the isolated namespace con-ci. con-prod was chosen over the original homelab because the homelab is personal property and not transferable; con-prod is owned, operated and backed up as Conduction infrastructure. The trade-off accepted here is blast radius: CI executes workflow-authored code (and, with con-ci-oci, rootless container builds) on the same cluster as live municipality workloads.

The risk

A malicious or buggy workflow could attempt to: reach other workloads (east-west), exfiltrate data or credentials, exhaust cluster resources, or escalate privileges on the node.

Mitigations (as built)

  • Namespace isolation. con-ci is separate; default-deny-ingress blocks any connection into it and default-deny-egress blocks all outbound except DNS and the egress-proxy → no east-west traffic to/from production namespaces.
  • Egress allowlist. All outbound goes through a squid forward-proxy with an explicit FQDN allowlist (codeberg.org, apps.nextcloud.com, api.anthropic.com, ghcr.io + GitHub releases, docker.io) — no wildcards. The NetworkPolicy is the L3/L4 backstop.
  • Resource bounds. ResourceQuota + LimitRange cap CPU/memory/pods so a runaway job cannot starve or evict production pods.
  • No privilege. All pods run non-root and drop all capabilities beyond the container default set. Image building and the container backend (podman build / rootless podman, as built 2026-07-04) are rootless — never --privileged, no SYS_ADMIN, no host docker socket, no host PID/network. The podman engine container carries two documented relaxations: a Localhost seccomp profile (podman-rootless.json, installed per-node by cluster-infra) and appArmorProfile: Unconfined — the latter is the single above-baseline relaxation on the platform (the containerd default AppArmor profile denies mount(2) even inside a pod-owned user namespace); its enabler is PSA enforce: privileged on the con-ci-oci namespace only.
  • Secrets. The Nextcloud app-signing key (crown jewel) is SOPS+age-encrypted, decrypted only in-cluster, mounted only at runtime, and never sent to Codeberg. The runner connection token is cluster-side; Codeberg holds only the runner identity, never a deploy/push credential.
  • Two-layer allowlist. A repo must have Actions enabled AND a workflow must opt in with a Conduction label (con-ci/con-ci-oci); generic labels are not served.
  • Auditability. GitOps via ArgoCD (self-heal, one-action rollback by deleting the Application); pre-commit credential gates; explicit --kubeconfig in all ops commands.

Accepted with review date — kaniko builder image (bridge, review by 2026-10-01)

Image-build jobs on con-ci-oci may use the kaniko executor (fully unprivileged Dockerfile builds — no daemon, no user-namespace or seccomp relaxations, runs as a plain container: job). Kaniko's upstream (GoogleContainerTools/kaniko) was archived on 2025-06-03: the version in use (v1.23.2) is frozen and will receive no security patches, while it builds images that are deployed to production. This is accepted as a bridge, not an end state, on these mitigations:

  • The builder image is pinned by digest to a self-hosted mirror (codeberg.org/conduction/kaniko@sha256:c3109d5926a997b100c4343944e06c6b30a6804b2f9abe0994d3de6ef92b028e, mirrored verbatim from gcr.io/kaniko-project/executor:v1.23.2-debug with digest preservation) — no egress to, or trust in, a mutable tag on an archived upstream project.
  • It runs under the full con-ci-oci posture: rootless podman job container, egress allowlist, ResourceQuota, no added privileges.
  • Review date: 2026-10-01, or earlier when the BuildKit enablement (targeted seccomp profile for job containers, design on file) lands — whichever comes first. At review: migrate builds to BuildKit or re-accept explicitly.

Residual risk (accepted)

  • Shared kernel/nodes. Pods share the node kernel with production; a kernel-level escape would cross the boundary. Mitigated by non-privileged + seccomp, not eliminated.
  • Rootless container nesting (con-ci-oci). New, larger runtime surface; being introduced incrementally and security-reviewed, but nested containers are inherently more complex.
  • Allowlisted-host abuse. A compromised workflow can still use its quota and reach allowlisted hosts (e.g. push to Codeberg with a job-scoped token). Bounded by quota + the allowlist + the non-crown-jewel scope of job credentials.
  • Policy correctness depends on Calico behaving as configured (the DNS/8053 + selector quirks are documented and tested).

Decision

Accepted for the MVP scope (host runner con-ci, and the container runner con-ci-oci once its rootless runtime passes security review), on the mitigations above. Sign-off:

  • Robert — pending
  • Ruben — pending