Operations

Runbooks for a cluster that already exists.

Runbook	Use when
Kubernetes upgrades	Moving the cluster to a new Kubernetes minor or patch
Adding a worker node	Growing the cluster
Oracle free tier worker	Adding or removing a cloud node created by Stage 0
Bitwarden secrets	Adding, rotating or debugging a secret
Troubleshooting	Something is broken

Symptom index

Symptom	Start at
task stage1:ansible:ping fails	Inventory and groups
A secret is empty inside the container	Bitwarden secrets
LoadBalancer service stuck <pending>	localhost_post_setup: MetalLB
A certificate never becomes ready	cert-manager
PVC stuck Pending	Longhorn
A worker will not join	kubeadm_agent
A worker will not drain during an upgrade	Kubernetes upgrades
kubeadm init fails on a cgroup preflight	Handlers: the reboot did not happen
Locked out of a host after Stage 1	host_setup: UFW and the SSH port
Terraform apply times out	Stage 2: deploy the platform

Safety properties worth knowing before you touch anything

The control plane is never drained

Draining a single-node control plane evicts every workload in the cluster. The upgrade automation deliberately does not drain it. Do not add a drain to kubeadm_server/tasks/upgrade-control-plane.yml.

serial: 1 on the worker play is a correctness constraint

It guarantees exactly one worker is drained at a time. Raising it to go faster can evict every replica of a workload simultaneously.

Only consecutive minor upgrades are tested

For both Kubernetes and Cilium. Always update to the latest patch of the current minor before moving to the next minor.