What OpenTelemetry Is#

OpenTelemetry (OTel) is a vendor-neutral framework for generating, collecting, and exporting telemetry data: traces, metrics, and logs. It provides APIs, SDKs, and the Collector – a standalone binary that receives, processes, and exports telemetry. OTel replaces the fragmented landscape of Jaeger client libraries, Zipkin instrumentation, Prometheus client libraries, and proprietary agents with a single standard.

The three signal types:

• Traces: Record the path of a request through distributed services as a tree of spans. Each span has a name, duration, attributes, and parent reference.
• Metrics: Numeric measurements (counters, gauges, histograms) emitted by applications and infrastructure. OTel metrics can be exported to Prometheus.
• Logs: Structured log records correlated with trace context. OTel log support bridges existing logging libraries with trace correlation.

The OTel Collector Pipeline#

The Collector is the central hub. It has three pipeline stages:

Pod Lifecycle and Probes#

Understanding how Kubernetes starts, monitors, and stops pods is essential for running reliable services. Misconfigurations here cause cascading failures, dropped requests, and restart loops that are difficult to diagnose.

Pod Startup Sequence#

When a pod is scheduled, this is the exact order of operations:

1. Init containers run sequentially. Each must exit 0 before the next starts.
2. All regular containers start simultaneously.
3. postStart hooks fire (in parallel with the container’s main process).
4. Startup probe begins checking (if defined).
5. Once the startup probe passes, liveness and readiness probes begin.

Init Containers#

Init containers run before your application containers and are used for setup tasks: waiting for a dependency, running database migrations, cloning config from a remote source.

Pod Security Standards#

Kubernetes Pod Security Standards define three security profiles that control what pods are allowed to do. Pod Security Admission (PSA) enforces these standards at the namespace level. This is the replacement for PodSecurityPolicy, which was removed in Kubernetes 1.25.

The Three Levels#

Privileged – Unrestricted. No security controls applied. Used for system-level workloads like CNI plugins, storage drivers, and logging agents that genuinely need host access.

Baseline – Prevents known privilege escalations. Blocks hostNetwork, hostPID, hostIPC, privileged containers, and most host path mounts. Allows most workloads to run without modification.

PodDisruptionBudgets Deep Dive#

A PodDisruptionBudget (PDB) limits how many pods from a set can be simultaneously down during voluntary disruptions – node drains, cluster upgrades, autoscaler scale-down. PDBs do not protect against involuntary disruptions like node crashes or OOM kills. They are the mechanism by which you tell Kubernetes “this service needs at least N healthy pods at all times during maintenance.”

minAvailable vs maxUnavailable#

PDBs support two fields. Use one or the other, not both.

Prometheus Architecture#

Prometheus pulls metrics from targets at regular intervals (scraping). Each target exposes an HTTP endpoint (typically /metrics) that returns metrics in a text format. Prometheus stores the scraped data in a local time-series database and evaluates alerting rules against it. Grafana connects to Prometheus as a data source and renders dashboards.

Scrape Configuration#

The core of Prometheus configuration is the scrape config. Each scrape_config block defines a set of targets and how to scrape them.

RBAC Patterns#

Kubernetes RBAC controls who can do what to which resources. It is built on four objects: Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings. Getting RBAC right means understanding how these four pieces compose and knowing the common patterns that cover 90% of real-world needs.

The Four RBAC Objects#

Role – Defines permissions within a single namespace. Lists API groups, resources, and allowed verbs.

ClusterRole – Defines permissions cluster-wide or for non-namespaced resources (nodes, persistent volumes, namespaces themselves).

Release Management Patterns#

Releasing software is more than merging to main and deploying. A disciplined release process ensures that every version is identifiable, every change is documented, every deployment is reversible, and failures are contained before they reach all users. This operational sequence walks through each phase of a production release workflow.

Phase 1 – Semantic Versioning#

Step 1: Adopt Semantic Versioning#

Semantic versioning (semver) communicates the impact of changes through the version number itself: MAJOR.MINOR.PATCH.

Resource Requests and Limits#

Requests and limits control how Kubernetes schedules pods and enforces resource usage. Getting them wrong leads to pods that get evicted, throttled to a crawl, or that starve other workloads on the same node.

Requests vs Limits#

Requests are what the scheduler uses for placement. When you request 500m CPU and 256Mi memory, Kubernetes finds a node with that much allocatable capacity. The request is a guarantee – the kubelet reserves those resources for your container.

Running Kafka on Kubernetes with Strimzi#

Running Kafka on Kubernetes without an operator is painful. You need StatefulSets, headless Services, init containers for broker ID assignment, and careful handling of storage and networking. Strimzi eliminates most of this by managing the entire Kafka lifecycle through Custom Resource Definitions.

Installing Strimzi#

# Option 1: Helm
helm repo add strimzi https://strimzi.io/charts
helm install strimzi strimzi/strimzi-kafka-operator \
  --namespace kafka \
  --create-namespace

# Option 2: Direct YAML install
kubectl create namespace kafka
kubectl apply -f https://strimzi.io/install/latest?namespace=kafka -n kafka

Verify the operator is running:

Running Redis on Kubernetes#

Redis on Kubernetes ranges from dead simple (single pod for caching) to operationally complex (Redis Cluster with persistence). The right choice depends on whether you need data durability, high availability, or just a fast throwaway cache.

Single-Instance Redis with Persistence#

For development or small workloads, a single Redis Deployment with a PVC is enough:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        command: ["redis-server", "--appendonly", "yes", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-data
          mountPath: /data
        resources:
          requests:
            cpu: 100m
            memory: 300Mi
          limits:
            cpu: 500m
            memory: 350Mi
      volumes:
      - name: redis-data
        persistentVolumeClaim:
          claimName: redis-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379

Set the memory limit in Redis (--maxmemory) lower than the container memory limit. If Redis uses 350Mi and the container limit is 350Mi, the kernel OOM-kills the process during background save operations when Redis forks and temporarily doubles its memory usage. A safe ratio: set maxmemory to 60-75% of the container memory limit.