Ga naar inhoud

ROLLOUTS

Controlled rollouts with a Canary Ring (Argo CD)

This document describes how we do safe, controlled upgrades of the Nextcloud platform using Argo CD:

  • Canary ring first (blue/green-like safety)
  • then batched promotion to the rest of the tenants (e.g. 60 tenants)

We keep this in-repo because it is platform governance: it defines how the platform is operated and upgraded.

For CI guardrails and operational gates, see docs/CHECKS-AND-BALANCES.md.

Concepts

  • Tenant: one Nextcloud instance generated by the nextcloud-tenants ApplicationSet from values/tenants/tenant-*.yaml.
  • Canary ring: a small, always-present set of tenants that receives upgrades first.
  • Stable ring: all other tenants.
  • Promotion: explicitly widening the change from canary → batches → all tenants.
  • GitOps rule: desired state lives in Git; Argo CD applies it. No manual occ installs or in-cluster hotfixes.

Change classification (checks and balances)

Every change must be classified in one of two types before merge:

  1. Platform change (high blast radius, must pass canary ring first)
  2. Tenant additive change (low blast radius, can be applied directly)

Platform change (must pass canary)

Treat as platform change when a PR modifies any shared behavior or rollout mechanics, including:

  • values/common.yaml
  • values/env/*.yaml
  • values/db/*.yaml
  • values/templates/*
  • argo/applicationsets/*
  • argo/projects/*
  • chart/version rollout mechanics (including tenant.chartVersion policy)
  • platform-level components under platform/**

Required flow:

  1. Canary PR (only canary tenants / ring)
  2. Canary validation (health + smoke)
  3. Batched promotion PRs for stable tenants

Tenant additive change (direct apply allowed)

Treat as tenant additive when a PR only updates tenant-specific desired state, for example:

  • add a new tenant file
  • change one/few tenant app declarations (tenant.apps.*)
  • tenant-specific hostname/namespace fields

These changes can be merged directly without full canary promotion, as long as:

  • the change is isolated to tenant files (values/tenants/tenant-*.yaml)
  • validation scripts pass
  • blast radius stays small (recommended: small batches)

Safety defaults for PR reviews

  • If in doubt, classify as platform change.
  • Large multi-tenant updates should still be batched, even when additive.
  • Rollback must be possible by reverting a single PR batch.

Why this is “blue/green-like” (and what it is not)

Nextcloud upgrades often touch schema/data. Running two versions in parallel on the same DB/PVC is not safe.

Instead, we use a ringed canary model:

  • Canary tenants upgrade first (real workloads / realistic config).
  • If healthy, we promote the same change to stable tenants in batches.

This gives most benefits of blue/green (early validation + controlled blast radius) without duplicating infra per tenant.

Repo building blocks

  • ApplicationSet: argo/applicationsets/nextcloud-tenants.yaml
  • Common defaults: values/common.yaml
  • Environment overrides: values/env/{accept,prod}.yaml
  • DB profiles: values/db/{mariadb,postgres,external}.yaml
  • Tenant input: values/tenants/tenant-<org>-<env>.yaml
  • Validation: scripts/validate-values.sh
  • Local render/sanity: scripts/smoke-checks.sh

Defining the Canary ring

We recommend a dedicated organization name canary:

  • canary-accept → canary.accept.commonground.nu
  • canary-prod → canary.commonground.nu

This keeps the ring obvious, permanent, and convention-compliant.

Operational rule: every change must pass canary before stable promotion.

How we control “who upgrades when”

We use two levers:

  1. Per-tenant chart pin (preferred for canary→stable promotion)
  2. Add tenant.chartVersion only to canary tenants first.

  3. After validation, roll it out to stable tenants in batches by adding the same tenant.chartVersion to those tenant files.

  4. Argo sync windows (safety net)

  5. Stable tenants can be restricted to business hours.
  6. Canary tenants can be allowed to sync anytime.

Upgrade workflow (chart minor bump example)

0) Prep

  • Ensure tenant values pass validation:
./nextcloud-platform/scripts/validate-values.sh
  • Render locally (sanity check):
./nextcloud-platform/scripts/smoke-checks.sh --tenant canary-prod

1) Canary upgrade PR

Goal: upgrade only the canary ring.

  • Update only the canary tenant files with the new chart version:
tenant:
  chartVersion: "8.10.0"   # example

Commit + open PR. After merge, Argo upgrades canary only.

2) Validate canary

Minimum validation:

  • Argo app is Healthy
  • Nextcloud reachable (/status.php)
  • php occ status shows expected version
  • no error spikes in logs
  • optional: run scripts/smoke-checks.sh + a small WebDAV upload/download test

3) Promote to stable in batches

Create a batch PR that updates a subset of tenants (e.g. 10) by adding the same tenant.chartVersion.

Repeat for batch 2, batch 3, etc.

This keeps changes reviewable and rollbackable:

  • Rollback a batch: revert that batch PR (only those tenants roll back).
  • Rollback canary: revert only the canary PR.

Rollback notes

  • Chart rollback: revert tenant.chartVersion in the affected tenants and let Argo sync back.
  • Data/schema rollback: may require DB restore depending on the Nextcloud version jump. Treat major upgrades as requiring a formal rollback plan.

FAQ

“Do we need to specify apps in tenants?”

Yes (industry-standard “desired state”): the tenant declares its app set via tenant.apps.enabled, and optional pinned versions via tenant.apps.versions.*.

The platform maps that declarative input to runtime env vars and runs an idempotent hook to ensure apps are installed.