Choosing Kubernetes Storage#

Storage decisions in Kubernetes are harder to change than almost any other architectural choice. Migrating data between storage backends in production involves downtime, risk, and careful planning. Understand the tradeoffs before provisioning your first PersistentVolumeClaim.

The decision comes down to five criteria: performance (IOPS and latency), durability (can you survive node failure), portability (can you move the workload), cost, and access mode (single pod or shared).

Storage Categories#

Block Storage (ReadWriteOnce)#

Block storage provides a raw disk attached to a single node. Only one pod on that node can mount it at a time (ReadWriteOnce). This is the most common storage type for databases, caches, and any workload that needs fast, consistent disk I/O.

Cilium Deep Dive#

Cilium replaces the traditional Kubernetes networking stack with eBPF programs that run directly in the Linux kernel. Instead of kube-proxy translating Service definitions into iptables rules and a traditional CNI plugin managing pod networking through bridge interfaces and routing tables, Cilium attaches eBPF programs to kernel hooks that process packets at wire speed. The result is a networking layer that is faster at scale, capable of Layer 7 policy enforcement, and provides built-in observability without application instrumentation.

Custom Resource Definitions (CRDs)#

CRDs extend the Kubernetes API with your own resource types. Once you create a CRD, you can kubectl get, kubectl apply, and kubectl delete instances of your custom type just like built-in resources. The custom resources are stored in etcd alongside native Kubernetes objects, benefit from the same RBAC, and participate in the same API machinery.

When to Use CRDs#

CRDs make sense when you need to represent application-specific concepts inside Kubernetes:

DaemonSets#

A DaemonSet ensures that a copy of a pod runs on every node in the cluster – or on a selected subset of nodes. When a new node joins the cluster, the DaemonSet controller automatically schedules a pod on it. When a node is removed, the pod is garbage collected.

This is the right abstraction for infrastructure that needs to run everywhere: log collectors, monitoring agents, network plugins, storage drivers, and security tooling.

EKS vs AKS vs GKE: Choosing a Managed Kubernetes Provider#

All three major managed Kubernetes services run certified, conformant Kubernetes. The differences lie in networking models, identity integration, node management, upgrade experience, cost, and ecosystem strengths. Your choice should be driven by where the rest of your infrastructure lives, your team’s existing expertise, and specific feature requirements.

Feature Comparison#

Control Plane#

GKE has the most polished upgrade experience. Release channels (Rapid, Regular, Stable) provide automatic upgrades with configurable maintenance windows. Surge upgrades handle node pools with minimal disruption. Google invented Kubernetes, and GKE reflects that pedigree in control plane operations.

Emulating Production Namespace Organization in Minikube#

Setting up namespaces locally the same way you organize them in production builds muscle memory for real operations. When your local cluster mirrors production namespace structure, you catch RBAC misconfigurations, resource limit issues, and network policy gaps before they reach staging. It also means your Helm values files, Kustomize overlays, and deployment scripts work identically across environments.

Why Bother Locally#

The default minikube experience is everything deployed into default. This teaches bad habits. Developers forget -n flags, RBAC issues are never caught, resource contention is never simulated, and the first time anyone encounters namespace isolation is in production – where the consequences are real.

Gateway API: The Modern Replacement for Ingress#

The Ingress resource has been the standard way to expose HTTP services in Kubernetes since the early days. It works, but it has fundamental limitations: it only supports HTTP, its routing capabilities are minimal (host and path matching only), and every controller extends it through non-standard annotations that are not portable. Gateway API is the official successor – a set of purpose-built resources that provide richer routing, protocol support beyond HTTP, and a role-oriented design that cleanly separates infrastructure concerns from application concerns.

GitOps for Kubernetes#

GitOps is a deployment model where git is the source of truth for your cluster’s desired state. A controller running inside the cluster watches a git repository and continuously reconciles the live state to match what is declared in git. When you want to change something, you commit to git. The controller detects the change and applies it.

This replaces kubectl apply from laptops and CI pipelines with a pull-based model where the cluster pulls its own configuration. The benefits are an audit trail in git history, easy rollback via git revert, and drift detection when someone makes manual changes.

Helm Release Naming Gotchas#

Helm charts derive Kubernetes resource names from the release name, but every chart does it differently. If you assume a consistent pattern, you will get bitten by DNS resolution failures, broken connection strings, and mysterious “service not found” errors.

Bitnami PostgreSQL: Names Are Not What You Expect#

The Bitnami PostgreSQL chart names resources using the release name directly, not {release-name}-postgresql. This catches nearly everyone.

# You deploy like this:
helm upgrade --install dt-postgresql bitnami/postgresql \
  --namespace dream-team \
  --set auth.database=mattermost \
  --set auth.username=mmuser

# You expect these resource names:
#   Pod:     dt-postgresql-postgresql-0   <-- WRONG
#   Service: dt-postgresql-postgresql     <-- WRONG

# Actual names:
#   Pod:     dt-postgresql-0
#   Service: dt-postgresql

This means your application connection string should reference dt-postgresql, not dt-postgresql-postgresql. If you chose release name postgresql, your service is just postgresql – which might collide with other things in your namespace.

Init Containers and Sidecar Patterns#

A pod can contain more than one container. Init containers run sequentially before the main application starts. Sidecars run alongside the main container for the lifetime of the pod. Together, they enable patterns where setup logic and cross-cutting concerns are separated from application code.

Init Containers#

Init containers are defined in spec.initContainers[] and run in order. Each must exit 0 before the next one starts. If any init container fails, Kubernetes restarts the pod (subject to restartPolicy). The main application containers do not start until every init container has completed successfully.