February 22, 2026etcd is the single most critical component in a Kubernetes cluster. It stores everything: pod specs, secrets, configmaps, RBAC rules, service account tokens, and all cluster state. By default, Kubernetes secrets are stored in etcd as base64-encoded plaintext. Anyone with read access to etcd has read access to every secret in the cluster. Securing etcd is not optional.
Run this against an unencrypted etcd and you will see why:
February 22, 2026The ingress controller is the front door to your cluster. Every request from the internet passes through it, making it both the most exposed component and the best place to enforce security controls. Most teams deploy an ingress controller and stop at basic routing. That leaves the door wide open.
Every ingress should terminate TLS. Never serve production traffic over plain HTTP. With nginx-ingress, force HTTPS redirects and add HSTS headers:
February 22, 2026This operational sequence takes a default Kubernetes cluster and locks it down. Phases are ordered by impact and dependency: assessment first, then RBAC, pod security, networking, images, auditing, and finally data protection. Each phase includes the commands, policy YAML, and verification steps.
Do not skip the assessment phase. You need to know what you are fixing before you start fixing it.
Before changing anything, establish a baseline. This phase produces a prioritized list of findings that drives the order of remediation in later phases.
February 22, 2026Security incidents in infrastructure environments follow a predictable lifecycle. The difference between a contained incident and a catastrophic breach is usually preparation and speed of response. This playbook covers the six phases of incident response with specific commands and procedures for Kubernetes and containerized infrastructure.
The phases are sequential but overlap in practice: you may be containing one aspect of an incident while still detecting the full scope.
February 22, 2026Every pod in Kubernetes runs as a service account. By default, that is the default service account in the pod’s namespace, with an auto-mounted API token that never expires. This default configuration is overly permissive for most workloads. Hardening service accounts is one of the highest-impact security improvements you can make in a Kubernetes cluster.
When a pod starts without specifying a service account, Kubernetes does three things:
February 22, 2026In a microservice architecture, services communicate over the network. Without authentication, any process that can reach a service can call it. Without authorization, any authenticated caller can do anything. Zero-trust networking assumes the internal network is hostile and requires every service-to-service call to be authenticated, authorized, and encrypted.
Standard TLS has the client verify the server’s identity. Mutual TLS adds the reverse – the server also verifies the client’s identity. Both sides present certificates. This provides three things: encryption in transit, server authentication, and client authentication.
February 22, 2026This operational sequence deploys a complete observability stack on Kubernetes: metrics (Prometheus + Grafana), logs (Loki + Promtail), traces (Tempo + OpenTelemetry), and alerting (Alertmanager). Each phase is self-contained with verification steps. Complete them in order – later phases depend on earlier infrastructure.
Prerequisite: a running Kubernetes cluster with Helm installed and a monitoring namespace created.
kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo updateMetrics are the foundation. Logging and tracing integrations all route through Grafana, so this phase must be solid before continuing.
February 22, 2026Running a single environment is straightforward. Running three that drift apart silently is where teams lose weeks debugging “it works in dev.” This operational sequence walks through setting up dev, staging, and production environments that stay consistent where it matters and diverge only where intended.
Each environment serves a distinct purpose:
Decision point: Separate clusters per environment versus namespaces in a shared cluster.
February 22, 2026Deployments treat pods as interchangeable. StatefulSets do not – each pod gets a stable hostname, a persistent volume, and an ordered startup sequence. This is what you need for databases, message queues, and any workload where identity matters.
| Feature | Deployment | StatefulSet |
|---|---|---|
| Pod names | Random suffix (web-api-6d4f8) | Ordinal index (postgres-0, postgres-1) |
| Startup order | All at once | Sequential (0, then 1, then 2) |
| Stable network identity | No | Yes, via headless Service |
| Persistent storage | Shared or none | Per-pod via volumeClaimTemplates |
| Scaling down | Removes random pods | Removes highest ordinal first |
Use StatefulSets when your application needs any of: stable hostnames, ordered deployment/scaling, or per-pod persistent storage. Common examples: PostgreSQL, MySQL, Redis Sentinel, Kafka, ZooKeeper, Elasticsearch.
February 22, 2026An on-call engineer paged at 3 AM has limited cognitive capacity. They may not be familiar with the specific service that is failing. They may have joined the team two weeks ago. A runbook bridges the gap between the alert firing and the correct human response. Without runbooks, incident response depends on tribal knowledge – the engineer who built the service and knows its failure modes. That engineer is on vacation when the incident hits.