From Empty Cluster to Production-Ready: The Complete Setup Sequence

From Empty Cluster to Production-Ready#

This is the definitive operational plan for taking a fresh Kubernetes cluster and making it production-ready. Each phase builds on the previous one, with verification steps between phases and rollback notes where applicable. An agent should be able to follow this sequence end-to-end.

Estimated timeline: 5 days for a single operator. Phases 1-2 are blocking prerequisites. Phases 3-6 can partially overlap.


Phase 1 – Foundation (Day 1)#

Everything else depends on a healthy cluster with proper namespacing and storage. Do not proceed until every verification step passes.

gRPC Security: TLS, mTLS, Authentication Interceptors, and Token-Based Access Control

gRPC Security#

gRPC uses HTTP/2 as its transport, which means TLS is not just a security feature — it is a practical necessity. Many load balancers, proxies, and clients expect HTTP/2 over TLS (h2) rather than plaintext HTTP/2 (h2c). Securing gRPC means configuring TLS correctly, authenticating clients, authorizing RPCs, and handling the gRPC-specific gotchas that do not exist with REST APIs.

gRPC Over TLS#

Server-Side TLS in Go#

import (
    "crypto/tls"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
)

func main() {
    cert, err := tls.LoadX509KeyPair("server-cert.pem", "server-key.pem")
    if err != nil {
        log.Fatal(err)
    }

    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{cert},
        MinVersion:   tls.VersionTLS13,
    }

    creds := credentials.NewTLS(tlsConfig)
    server := grpc.NewServer(grpc.Creds(creds))

    pb.RegisterMyServiceServer(server, &myService{})

    lis, _ := net.Listen("tcp", ":50051")
    server.Serve(lis)
}

Client-Side TLS in Go#

import (
    "crypto/x509"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
)

func main() {
    // For public CAs (Let's Encrypt, etc.), use system cert pool
    creds := credentials.NewTLS(&tls.Config{
        MinVersion: tls.VersionTLS13,
    })

    // For internal CAs, load the CA cert explicitly
    caCert, _ := os.ReadFile("ca-cert.pem")
    certPool := x509.NewCertPool()
    certPool.AppendCertsFromPEM(caCert)
    creds = credentials.NewTLS(&tls.Config{
        RootCAs:    certPool,
        MinVersion: tls.VersionTLS13,
    })

    conn, err := grpc.NewClient("api.internal:50051",
        grpc.WithTransportCredentials(creds),
    )
    defer conn.Close()

    client := pb.NewMyServiceClient(conn)
}

TLS in Python#

import grpc

# Server
server_credentials = grpc.ssl_server_credentials(
    [(open("server-key.pem", "rb").read(), open("server-cert.pem", "rb").read())]
)
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
pb_grpc.add_MyServiceServicer_to_server(MyService(), server)
server.add_secure_port("[::]:50051", server_credentials)
server.start()

# Client
ca_cert = open("ca-cert.pem", "rb").read()
channel_credentials = grpc.ssl_channel_credentials(root_certificates=ca_cert)
channel = grpc.secure_channel("api.internal:50051", channel_credentials)
client = pb_grpc.MyServiceStub(channel)

Mutual TLS for gRPC#

mTLS is the strongest authentication model for service-to-service gRPC. Each service has a certificate, and both sides verify each other.

Securing Kubernetes Ingress: TLS, Rate Limiting, WAF, and Access Control

Securing Kubernetes Ingress#

The ingress controller is the front door to your cluster. Every request from the internet passes through it, making it both the most exposed component and the best place to enforce security controls. Most teams deploy an ingress controller and stop at basic routing. That leaves the door wide open.

TLS Termination and HTTPS Enforcement#

Every ingress should terminate TLS. Never serve production traffic over plain HTTP. With nginx-ingress, force HTTPS redirects and add HSTS headers:

TLS and mTLS Fundamentals: Certificates, Chains of Trust, Mutual Authentication, and Troubleshooting

TLS and mTLS Fundamentals#

TLS (Transport Layer Security) encrypts traffic between two endpoints. Mutual TLS (mTLS) adds a second layer: both sides prove their identity with certificates. Understanding these is not optional for anyone building distributed systems — nearly every production failure involving “connection refused” or “certificate verify failed” traces back to a TLS misconfiguration.

How TLS Works#

A TLS handshake establishes an encrypted channel before any application data is sent. The simplified flow:

TLS Certificate Lifecycle Management

Certificate Basics#

A TLS certificate binds a public key to a domain name. The certificate is signed by a Certificate Authority (CA) that browsers and operating systems trust. The chain goes: your certificate, signed by an intermediate CA, signed by a root CA. All three must be present and valid for a client to trust the connection.

Self-Signed Certificates for Development#

For local development and testing, generate a self-signed certificate. Clients will not trust it by default, but you can add it to your local trust store.

Admission Controllers and Webhooks: Intercepting and Enforcing Kubernetes API Requests

Admission Controllers and Webhooks#

Every request to the Kubernetes API server passes through a chain: authentication, authorization, and then admission control. Admission controllers are plugins that intercept requests after a user is authenticated and authorized but before the object is persisted to etcd. They can validate requests, reject them, or mutate objects on the fly. This is where you enforce organizational policy, inject sidecar containers, set defaults, and block dangerous configurations.

Certificate Management Deep Dive

PKI Fundamentals#

A Public Key Infrastructure (PKI) is a hierarchy of trust. At the top sits the Root CA, a certificate authority that signs its own certificate and is explicitly trusted by all participants. Below it are Intermediate CAs, signed by the root, which handle day-to-day certificate issuance. At the bottom are leaf certificates, the actual certificates used by servers, clients, and workloads.

Root CA (self-signed, offline, 10-20 year validity)
  |
  +-- Intermediate CA (signed by root, online, 3-5 year validity)
        |
        +-- Leaf Certificate (signed by intermediate, 90 days or less)
        +-- Leaf Certificate
        +-- Leaf Certificate

Never use the root CA directly to sign leaf certificates. If the root CA’s private key is compromised, the entire PKI must be rebuilt from scratch. Keeping it offline and behind an intermediate CA limits the blast radius. If an intermediate CA is compromised, you revoke it and issue a new one from the root – leaf certificates from other intermediates remain valid.

Secrets Rotation Patterns

Why Rotation Matters#

A credential that never changes is a credential waiting to be exploited. Leaked credentials appear in git history, log files, CI build outputs, developer laptops, and third-party SaaS tools. If a database password has been the same for two years, every person who has ever had access to it still has access – former employees, former contractors, compromised CI systems.

Regular rotation limits the blast radius. A credential that rotates every 24 hours is only useful for 24 hours after compromise. Compliance frameworks (PCI-DSS, SOC2, HIPAA) mandate rotation schedules. But compliance aside, rotation is a pragmatic defense: assume credentials will leak and make the leak time-limited.

TLS Deep Dive: Certificate Chains, Handshake, Cipher Suites, and Debugging Connection Issues

The TLS Handshake#

Every HTTPS connection starts with a TLS handshake that establishes encryption parameters and verifies the server’s identity. The simplified flow for TLS 1.2:

Client                              Server
  |── ClientHello ──────────────────>|   (supported versions, cipher suites, random)
  |<────────────────── ServerHello ──|   (chosen version, cipher suite, random)
  |<──────────────── Certificate  ──|   (server's certificate chain)
  |<───────────── ServerKeyExchange ─|   (key exchange parameters)
  |<───────────── ServerHelloDone  ──|
  |── ClientKeyExchange ───────────>|   (client's key exchange contribution)
  |── ChangeCipherSpec ────────────>|   (switching to encrypted communication)
  |── Finished ────────────────────>|   (encrypted verification)
  |<──────────── ChangeCipherSpec ──|
  |<──────────────────── Finished ──|
  |<═══════ Encrypted traffic ═════>|

TLS 1.3 simplifies this significantly. The client sends its key share in the ClientHello, allowing the handshake to complete in a single round trip. TLS 1.3 also removed insecure cipher suites and compression, eliminating entire classes of vulnerabilities (BEAST, CRIME, POODLE).