Securing etcd: Encryption at Rest, TLS, and Access Control

Securing etcd#

etcd is the single most critical component in a Kubernetes cluster. It stores everything: pod specs, secrets, configmaps, RBAC rules, service account tokens, and all cluster state. By default, Kubernetes secrets are stored in etcd as base64-encoded plaintext. Anyone with read access to etcd has read access to every secret in the cluster. Securing etcd is not optional.

Why etcd Is the Crown Jewel#

Run this against an unencrypted etcd and you will see why:

Securing Kubernetes Ingress: TLS, Rate Limiting, WAF, and Access Control

Securing Kubernetes Ingress#

The ingress controller is the front door to your cluster. Every request from the internet passes through it, making it both the most exposed component and the best place to enforce security controls. Most teams deploy an ingress controller and stop at basic routing. That leaves the door wide open.

TLS Termination and HTTPS Enforcement#

Every ingress should terminate TLS. Never serve production traffic over plain HTTP. With nginx-ingress, force HTTPS redirects and add HSTS headers:

Security Compliance and Benchmarks

Why Benchmarks Matter#

Security benchmarks translate “harden the cluster” into specific, testable checks. Run a scan, get a pass/fail report, fix what failed. CIS publishes the most widely adopted benchmarks for Kubernetes and Docker. NSA/CISA provide additional Kubernetes-specific threat guidance.

CIS Kubernetes Benchmark with kube-bench#

kube-bench runs CIS Kubernetes Benchmark checks against cluster nodes, testing API server flags, etcd configuration, kubelet settings, and control plane security:

# Run on a master node
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-master.yaml

# Run on worker nodes
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-node.yaml

# Read results
kubectl logs job/kube-bench

Or run directly on a node:

Security Hardening a Kubernetes Cluster: End-to-End Operational Sequence

Security Hardening a Kubernetes Cluster#

This operational sequence takes a default Kubernetes cluster and locks it down. Phases are ordered by impact and dependency: assessment first, then RBAC, pod security, networking, images, auditing, and finally data protection. Each phase includes the commands, policy YAML, and verification steps.

Do not skip the assessment phase. You need to know what you are fixing before you start fixing it.


Phase 1 – Assessment#

Before changing anything, establish a baseline. This phase produces a prioritized list of findings that drives the order of remediation in later phases.

Security Incident Response for Infrastructure

Incident Response Overview#

Security incidents in infrastructure environments follow a predictable lifecycle. The difference between a contained incident and a catastrophic breach is usually preparation and speed of response. This playbook covers the six phases of incident response with specific commands and procedures for Kubernetes and containerized infrastructure.

The phases are sequential but overlap in practice: you may be containing one aspect of an incident while still detecting the full scope.

Self-Service Infrastructure Patterns

The Problem Self-Service Solves#

Developers need infrastructure: databases, caches, message queues, storage buckets, DNS records. In most organizations, getting these means filing a ticket, waiting days for someone to provision, and receiving credentials in a Slack DM. This bottleneck incentivizes workarounds — manual console provisioning, skipped security configs, everything crammed into shared databases.

Self-service infrastructure lets developers provision what they need directly, within guardrails the platform team defines. Choose a resource from a catalog, fill in parameters, and the system provisions it and returns connection details. No tickets, no waiting.

Service Account Security: Tokens, RBAC Binding, and Workload Identity

Service Account Security#

Every pod in Kubernetes runs as a service account. By default, that is the default service account in the pod’s namespace, with an auto-mounted API token that never expires. This default configuration is overly permissive for most workloads. Hardening service accounts is one of the highest-impact security improvements you can make in a Kubernetes cluster.

The Default Problem#

When a pod starts without specifying a service account, Kubernetes does three things:

Service-to-Service Authentication and Authorization

Service-to-Service Authentication and Authorization#

In a microservice architecture, services communicate over the network. Without authentication, any process that can reach a service can call it. Without authorization, any authenticated caller can do anything. Zero-trust networking assumes the internal network is hostile and requires every service-to-service call to be authenticated, authorized, and encrypted.

Mutual TLS (mTLS)#

Standard TLS has the client verify the server’s identity. Mutual TLS adds the reverse – the server also verifies the client’s identity. Both sides present certificates. This provides three things: encryption in transit, server authentication, and client authentication.

Setting Up and Configuring Backstage

What Backstage Provides#

Backstage is an open-source developer portal originally built by Spotify, now a CNCF Incubating project. It serves as the single UI layer for an internal developer platform, unifying the service catalog, documentation, scaffolding templates, and plugin-based integrations behind one interface. It does not replace your tools — it provides a consistent frontend for discovering and interacting with them.

The core components:

  • Software Catalog: A registry of all services, libraries, APIs, and infrastructure components, populated from YAML descriptor files in your repositories.
  • TechDocs: Documentation-as-code powered by MkDocs, rendered directly in the Backstage UI alongside the service it describes.
  • Scaffolder: A template engine that creates new projects from predefined templates — repositories, CI pipelines, Kubernetes manifests, and all.
  • Plugins: Backstage’s extension mechanism. The community provides plugins for Kubernetes, ArgoCD, PagerDuty, GitHub Actions, Terraform, and hundreds of other tools.

Installation#

Backstage requires Node.js 18+ and Yarn. Create a new Backstage app:

Setting Up Full Observability from Scratch: Metrics, Logs, Traces, and Alerting

Setting Up Full Observability from Scratch#

This operational sequence deploys a complete observability stack on Kubernetes: metrics (Prometheus + Grafana), logs (Loki + Promtail), traces (Tempo + OpenTelemetry), and alerting (Alertmanager). Each phase is self-contained with verification steps. Complete them in order – later phases depend on earlier infrastructure.

Prerequisite: a running Kubernetes cluster with Helm installed and a monitoring namespace created.

kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Phase 1 – Metrics (Prometheus + Grafana)#

Metrics are the foundation. Logging and tracing integrations all route through Grafana, so this phase must be solid before continuing.