Ga naar inhoud

Troubleshooting & operational gotchas

Symptom → cause → fix. These are the non-obvious traps, most learned the hard way on con-prod. See docs/architecture.md for the model.

CI job fails instantly with CONNECT tunnel failed, response 403

Cause: the egress-proxy (squid, FQDN allowlist) refused the host. All job egress — including cloning uses: actions — goes through it. Bare actions/checkout@v4 resolves to data.forgejo.org (Forgejo's mirror), not github.com, so that host must be allowlisted. The failure happens in the "workflow prepared" phase, before any run: step, and skips all post-steps.

Fix: add the host to manifests/components/egress/squid.conf (acl allowed_domains dstdomain <host>), commit, push, let ArgoCD sync. Common hosts already present: codeberg.org, code.forgejo.org, data.forgejo.org, registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com, production.cloudfront.docker.com, ghcr.io, api.anthropic.com, apps.nextcloud.com. The con-ci-oci overlay has its own allowlist (manifests/overlays/con-ci-oci/squid.conf) — since 2026-07-06 incl. pypi.org + files.pythonhosted.org (handbook docs pipeline), since 2026-07-07 also registry.npmjs.org (npm install inside kaniko image builds) and index.docker.io — gotcha: kaniko (go-containerregistry) normalizes docker.io to index.docker.io, podman/docker to registry-1.docker.io; a base-image pull that works in a container: job can still 403 inside a kaniko build if only registry-1 is allowlisted. Same day, same lesson variant: GitHub release downloads (github.com/<o>/<r>/releases/download/…) now 302 to release-assets.githubusercontent.com, not objects.githubusercontent.com — bit sharp/libvips' postinstall fetch. Since 2026-07-13 also api.cloudflare.com (wrangler deploys to Cloudflare Workers from container jobs; the telemetry host sparrow.cloudflare.com stays denied — set WRANGLER_SEND_METRICS=false in the workflow). When a client reports a proxy 403 despite the obvious host being allowlisted, read the squid access log (kubectl -n con-ci-oci logs deploy/egress-proxy | grep DENIED) — it names the exact denied host; redirects make the real target invisible in client errors.

Verify which hosts a job needs without guessing: exec a /dev/tcp CONNECT probe through the proxy from a throwaway pod (200 = allowed+tunnelled, 403 = denied). squid dstdomain matches the requested host literally (CNAMEs are irrelevant — an earlier hunch that it matched the CNAME target was wrong).

actions/setup-node fails: tunneling socket could not be established, statusCode=403

Cause: setup-node downloads a Node toolchain at job time — the version manifest from api.github.com, the tarball from nodejs.org. Neither is allowlisted, and widening the allowlist for a redundant download is the wrong trade: on con-ci-oci the job already runs in a container image.

Fix (workflow-side, preferred): run the job in a node:<version> image and drop the setup-node step entirely — npm ci then works via the already-allowed registry.npmjs.org (observed 2026-07-13 on a docs pipeline). The general rule: when a toolchain-installer action fails at the proxy, first ask whether the container image can simply provide the toolchain.

Allowlist change committed but the 403 persists

Cause: squid reads its config only at startup (squid -N -f …); a ConfigMap change does not reload it. Worse, a manual rollout restart can race the ConfigMap projection: the new squid can start ~1s before kubelet swaps in the new file, loading the stale allowlist (observed 2026-07-01 — needed two restarts).

Fix (already in place): the squid config is delivered via kustomize configMapGenerator with the name hash on, so any edit yields a new ConfigMap name → new pod template → ArgoCD rolls the pod after the new ConfigMap exists. No manual restart. If you ever do restart manually, confirm the projected file is current: compare kubectl -n con-ci exec <squid-pod> -- readlink -f /etc/squid/squid.conf (its ..<timestamp> dir) against the squid "Starting Squid Cache" log line.

Docker Hub image pull 403s on production.cloudfront.docker.com

Cause: Docker Hub serves blobs from both cloudflare and cloudfront CDNs; only cloudflare was allowlisted, so the manifest pulled but the blob 403'd.

Fix: both production.cloudflare.docker.com and production.cloudfront.docker.com are now in squid.conf. For a persistent registry, Harbor (Change-2) collapses this to one host.

git push to Codeberg fails with TLS connect error / Connection closed

Cause (ops workstation alma, 2026-07-01): TLS to codeberg.org works over IPv6 but fails over IPv4. The ~/.ssh/config block for codeberg had AddressFamily inet (IPv4) with a stale comment claiming v6 was broken — it's inverted.

Fix: push over SSH with AddressFamily inet6. Both repos have an SSH origin (git@codeberg.org:Conduction/<repo>.git); the HTTPS remote (codeberg) is the broken path. Claude cannot push to Codeberg at all (a pre-tool hook blocks it) — a human runs the push.

ArgoCD app stuck Unknown / app path does not exist after a push

Cause: ArgoCD's repo cache is stale, or the push didn't actually land on the branch ArgoCD tracks (main). It caught us that a cluster-infra push went to a different remote / never reached the server.

Fix: (1) confirm the commit is on the remote: git -C <repo> ls-remote origin main vs your local main. (2) force a re-pull: kubectl -n argocd annotate application <app> argocd.argoproj.io/refresh=hard --overwrite. ArgoCD auto-polls ~every 3 min otherwise.

con-ci-oci runner (forgejo-runner-oci) crashlooping

This is rootless podman in a non-privileged pod under PodSecurity baseline. The three blockers we cleared, in order (each has a distinct error):

  1. cannot clone: Operation not permitted — podman can't create a user namespace. The containerd RuntimeDefault seccomp profile gates the namespace clone flags behind CAP_SYS_ADMIN. baseline forbids Unconfined but allows Localhost, so use the podman-rootless Localhost profile (installed by the cluster-infra seccomp-profiles DaemonSet). Reference it on the podman container: seccompProfile: {type: Localhost, localhostProfile: podman-rootless.json}.

  2. newuidmap …: operation not permitted — the setuid-root newuidmap can't run/map the subuid range. allowPrivilegeEscalation: false (no_new_privs) blocks the setuid exec, and drop: ALL removes the caps it then needs. Fix: allowPrivilegeEscalation: true + capabilities.add: [SETUID, SETGID] (only those; baseline permits both). Still not privileged.

  3. mount … permission denied / cannot change root filesystem propagation / overlay is not supported — CORRECTED 2026-07-04: this was AppArmor, not mount propagation. The containerd default AppArmor profile denies mount(2) even inside a userns the pod owns (privileged pods "worked" only because privileged runs AppArmor-unconfined). Fix: appArmorProfile: {type: Unconfined} on the podman container — then STORAGE_DRIVER=overlay + fuse-overlayfs works fully rootless (run and podman build), no privileged, no SYS_ADMIN, hostUsers: false intact. Proven by the probe ladder in manifests/components/runner-oci/TEST-*.yaml. NOTE: AppArmor Unconfined is above baseline PSA — see the runner-oci doc for the enforce-privileged vs. Localhost-AppArmor trade-off. (The old vfs workaround also hit the same wall at layer extraction, so vfs is retired.)

  4. cannot load seccomp profile "…/podman-rootless.json": no such file — the pod landed on a node without the profile. The seccomp-profiles installer only runs on pool worker-0b1p9-1, so runner-oci pins there with a nodeSelector. (When it requested squat.ai/fuse it pinned implicitly; on vfs it needs the explicit selector.)

Working config (since 2026-07-04) = overlay+fuse-overlayfs + Localhost seccomp + allowPrivilegeEscalation + default-set caps + AppArmor Unconfined + in-range subuid (1000:1001:64535) + nodeSelector. Result: the daemon declares [con-ci-oci] and container jobs can pull, run and build images.

Debugging technique: patch the podman container's command to sleep infinity (kubectl -n con-ci patch deploy forgejo-runner-oci --type=strategic -p '{"spec":{"template":{"spec":{"containers":[{"name":"podman","command":["sleep","infinity"]}]}}}}'), scale to 1, then kubectl exec in and run podman --log-level=debug info / unshare to read the real trace. Re-apply the manifest to restore the real command.

PodSecurity baseline — what's allowed in con-ci

con-ci enforces baseline (audit/warn are restricted). Practical implications: - seccomp Unconfined → forbidden; Localhost and RuntimeDefault → allowed. - allowPrivilegeEscalation: true → allowed (restricted would forbid it). - adding capabilities like SETUID/SETGID → allowed (restricted would forbid). - --privileged, host namespaces, host docker socket → forbidden (and we never use them).

fuse device-plugin / seccomp profile (cluster-infra) missing

If con-ci-oci can't schedule or can't load its seccomp profile, check the cluster-infra repo's apps: kubectl -n kube-system get ds fuse-device-plugin seccomp-profile-installer and kubectl -n argocd get app fuse-device-plugin seccomp-profiles. Both are scoped to pool worker-0b1p9-1. The seccomp file must exist at /var/lib/kubelet/seccomp/podman-rootless.json on the node before the pod is admitted.

buildah: command not found on a con-ci job

Cause: the stock code.forgejo.org/forgejo/runner:12.12.0 image has no buildah. The runner-build patch adds subuid/subgid + BUILDAH_ISOLATION=chroot + vfs, but it does not install the binary. So runs-on: con-ci steps that call buildah fail with exit 127.

Status: DECIDED (2026-07-04) — option (c): image builds go to con-ci-oci. (a) is a maintenance burden with a chicken-and-egg (build the builder where?), (b) is impossible (host jobs are non-root, no package manager). Since con-ci-oci proves podman build works rootless, builds belong there; con-ci stays the plain host class (scripts, lint, tooling, the sign step). test-con-ci.yml was rewritten accordingly (host shell + proxy-enforcement smoke, no buildah). The runner-build subuid/vfs patch on the host runner stays dormant — revisit/remove it when the add-codeberg-ci-runners change is archived.

con-ci-oci: image pull works but ApplyLayer … remount /, … permission denied

Cause: the con-ci-oci runner picks up container: jobs and the image pull succeeds (blobs come through the proxy — cloudflare/cloudfront allowlisted). Extraction fails: applying a layer does a remount of /, which the rootless userns cannot do — the container rootfs mount is owned by the host userns (the SAME mount-propagation limit that blocks overlay/fuse-overlayfs, now hit by vfs layer extraction).

Status: RESOLVED (2026-07-04). The "remount /" denial had the same root cause as blocker 3 above: the containerd default AppArmor profile, not a userns/mount-propagation limit. With appArmorProfile: Unconfined on the podman container, layer extraction, podman run and podman build all work rootless on overlay+fuse-overlayfs (probe-proven, A+B green). Two follow-ups recorded in the runner-oci doc: the PSA enforce-level decision (Unconfined is above baseline) and the squat.ai/fuse autoscale-from-zero gap.

con-ci-oci: every job container dies with crun: mountproctoproc: Operation not permitted

Cause (empirically bisected 2026-07-04, diag pods 4–8): an emptyDir volume mounted at /home/podman — any medium, even unused — makes every nested procfs mount by crun fail with EPERM inside this pod shape (hostUsers:false → kubelet uses idmapped mounts for emptyDirs). Probe pods without a home emptyDir never hit it, which made everything else look guilty first. Red herrings ruled out one by one: AppArmor (identical unconfined), seccomp (same Localhost filter), subuid mapping (full range fine), CLI-vs-API path (both), tun device, pod-level seccomp, graphroot location, kubectl-exec context. The podman-sock (/run/podman) and containers (…/.local/share/containers) emptyDirs are harmless.

Fix: no volume at /home/podman itself — the image's home is uid-1000-owned and writable. Related knock-ons fixed in the same session: storage.conf addressed via CONTAINERS_STORAGE_CONF=/etc/talos/storage.conf (a subPath under $HOME/.config makes kubelet create .config as root → podman refuses to start); job networking on container.network: host in the runner config (per-workflow bridge networks need /dev/net/tun = squat.ai/tun, PLUS netavark sysctl writes under the masked /proc/sys, which needs the ProcMountType feature gate + procMount: Unmasked — gate is OFF on this shoot, so the field is silently dropped; enable via Gardener shoot spec if bridge networks are ever needed).