Securing etcd: Encryption at Rest, TLS, and Access Control

Securing etcd#

etcd is the single most critical component in a Kubernetes cluster. It stores everything: pod specs, secrets, configmaps, RBAC rules, service account tokens, and all cluster state. By default, Kubernetes secrets are stored in etcd as base64-encoded plaintext. Anyone with read access to etcd has read access to every secret in the cluster. Securing etcd is not optional.

Why etcd Is the Crown Jewel#

Run this against an unencrypted etcd and you will see why:

TLS and mTLS Fundamentals: Certificates, Chains of Trust, Mutual Authentication, and Troubleshooting

TLS and mTLS Fundamentals#

TLS (Transport Layer Security) encrypts traffic between two endpoints. Mutual TLS (mTLS) adds a second layer: both sides prove their identity with certificates. Understanding these is not optional for anyone building distributed systems — nearly every production failure involving “connection refused” or “certificate verify failed” traces back to a TLS misconfiguration.

How TLS Works#

A TLS handshake establishes an encrypted channel before any application data is sent. The simplified flow:

Admission Controllers and Webhooks: Intercepting and Enforcing Kubernetes API Requests

Admission Controllers and Webhooks#

Every request to the Kubernetes API server passes through a chain: authentication, authorization, and then admission control. Admission controllers are plugins that intercept requests after a user is authenticated and authorized but before the object is persisted to etcd. They can validate requests, reject them, or mutate objects on the fly. This is where you enforce organizational policy, inject sidecar containers, set defaults, and block dangerous configurations.

TLS Deep Dive: Certificate Chains, Handshake, Cipher Suites, and Debugging Connection Issues

The TLS Handshake#

Every HTTPS connection starts with a TLS handshake that establishes encryption parameters and verifies the server’s identity. The simplified flow for TLS 1.2:

Client                              Server
  |── ClientHello ──────────────────>|   (supported versions, cipher suites, random)
  |<────────────────── ServerHello ──|   (chosen version, cipher suite, random)
  |<──────────────── Certificate  ──|   (server's certificate chain)
  |<───────────── ServerKeyExchange ─|   (key exchange parameters)
  |<───────────── ServerHelloDone  ──|
  |── ClientKeyExchange ───────────>|   (client's key exchange contribution)
  |── ChangeCipherSpec ────────────>|   (switching to encrypted communication)
  |── Finished ────────────────────>|   (encrypted verification)
  |<──────────── ChangeCipherSpec ──|
  |<──────────────────── Finished ──|
  |<═══════ Encrypted traffic ═════>|

TLS 1.3 simplifies this significantly. The client sends its key share in the ClientHello, allowing the handshake to complete in a single round trip. TLS 1.3 also removed insecure cipher suites and compression, eliminating entire classes of vulnerabilities (BEAST, CRIME, POODLE).