February 22, 2026Your cluster will fail. The question is whether you can rebuild it in hours or weeks. Kubernetes DR is not a single tool – it is a layered strategy combining etcd snapshots, resource-level backups, GitOps state, and tested recovery procedures.
The three layers of Kubernetes DR: etcd gives you raw cluster state, Velero gives you portable resource and volume backups, and GitOps gives you declarative rebuild capability. You need at least two of these.
February 22, 2026Running Kubernetes in a single region is a single point of failure at the infrastructure level. Region outages are rare but real – AWS us-east-1 has gone down multiple times, taking entire companies offline. Multi-region Kubernetes addresses this, but it introduces complexity in networking, state management, and deployment coordination that you must handle deliberately.
The simplest multi-region pattern: run completely independent clusters in each region, deploy the same applications to all of them using GitOps, and route traffic with DNS or a global load balancer.
February 22, 2026Stateless workloads are easy to recover – redeploy from Git and they are running. Stateful workloads carry data that cannot be regenerated. Databases, message queues, object stores, and anything with a PersistentVolume needs a deliberate DR strategy that goes beyond “we have Velero.”
The fundamental challenge: you must capture data at a point in time where the application state is consistent, replicate that data to a recovery site, and restore it in the correct order. Get any of these wrong and you recover corrupted data or a broken dependency chain.
May 27, 2026A common, maddening symptom: your Jenkins organization folder (or multibranch pipeline) backed by Gitea builds the default branch fine, but pull request commits never build — the commit status stays pending forever (or never appears), so a branch-protection gate that requires a CI status can never be satisfied and the PR can never merge.
The pipeline is fine. The problem is branch/PR discovery configuration, and there are several layered traps. Here is how to diagnose and fix each.
May 7, 2026A Helm operator runs an upgrade with --reuse-values -f new-values.yaml. Helm reports success, increments the revision counter, and returns STATUS: deployed. The cluster behavior does not change. The new values file might as well not exist. This is a silent no-op upgrade — the load-bearing failure mode of --reuse-values — and it is one of several Day-2 Helm operations where the verbs look correct but the semantics are not what most operators assume. This article covers the flag combinations that bite, how to inspect any past revision, how rollback actually works, and the snapshot-before-upgrade discipline that turns Helm’s revision storage into a real disaster-recovery backstop.
May 7, 2026A receiver YAML passes static review and the helm release reports deployed. The alertmanager pod is Running 1/1. A real critical alert fires and goes nowhere. The alertmanager pod logs are clean. The receiver works fine for a hand-rolled curl to the webhook URL. The trap is that the prometheus-operator generated a Secret containing the rendered config but flagged a sync error in its own logs — and the alertmanager pod kept serving the previous-good rendering, silently. This article assumes familiarity with the basic alertmanager routing tree, receivers, inhibition rules, and templating covered in alertmanager-configuration. It extends that material with the Day-2 operations of the kube-prometheus-stack chart specifically: where errors actually surface, what the native receiver schemas allow (and don’t), and the silence discipline that keeps the alert pipeline trustworthy.
May 7, 2026A single-node Kubernetes cluster running seven Helm-managed services concurrently — Gitea, Mattermost, PostgreSQL, kube-prometheus-stack, Jenkins, Temporal, and NATS — looks tractable on paper. The charts are all upstream-maintained. The hardware is modest but adequate. The operational reality is that zero of the seven ran cleanly on out-of-the-box values. Every chart needed at least one customization to coexist with the others, and several needed substantial rewrites of the helm-values surface. This survey catalogs what those customizations are, why each was necessary, and what the common failure modes look like across the fleet.
May 7, 2026A minikube cluster on Apple Silicon looks like a pure Kubernetes problem until the first Docker Desktop crash. The failure modes that bite hardest on M-series Macs live one layer below the cluster: in Docker Desktop’s memory allocator, in QEMU’s address-space layout, and in the destructive default of minikube delete. None of these are mentioned in the standard minikube setup guide, and all three will eat real workload state when they fire. This is the operational layer on top of minikube setup and drivers and ARM64 K8s images — the host-side discipline that keeps the cluster alive.
May 7, 2026A self-hosted Gitea forge running on Kubernetes covers four operational concerns that the upstream chart leaves to the operator: identity hygiene for bots and humans, branch protection rendered from code rather than clickops, webhook wiring to CI, and a backup story that survives a cluster wipe. The companion article Gitea Collaborator Grants and Review Officiality covers the narrow operational gotcha of official=false reviews; this article is the broader runbook for running the forge well.
February 22, 2026An agent that says “you should probably add a readiness probe to your deployment” is giving advice. An agent that hands you a tested manifest with the readiness probe configured, verified against a real cluster, with rollback steps if the probe misconfigures – that agent is producing a deliverable. The difference matters.
The core thesis of infrastructure agent work is that the output is always a deliverable – a runbook, playbook, tested manifest, or validated configuration – never a direct action on someone else’s systems. This article covers the complete workflow for generating those deliverables: understanding requirements, planning steps, executing in a sandbox, capturing what worked, and packaging the result.