Decision-first: One model per GPU (cloud-main + local-wake-filter for multi-model); unload-and-verify before every load; never lower the Docker Desktop VM cap; tunnel to loopback to dodge macOS Local Network Privacy; serialize loads and don’t download during inference.

Scope & freshness: Apple-Silicon Mac + minikube/Docker Desktop and a single-GPU LLM host (GB10), as of 2026-05-25. Incident patterns are durable; specific recovery commands assume kubectl/minikube/Docker Desktop.

A field runbook of failure modes seen running local LLMs next to development Kubernetes clusters. Each is a real incident pattern, not a hypothetical. (This whole doc is effectively a “what didn’t work” catalog — that’s the point.)

A Helm operator runs an upgrade with --reuse-values -f new-values.yaml. Helm reports success, increments the revision counter, and returns STATUS: deployed. The cluster behavior does not change. The new values file might as well not exist. This is a silent no-op upgrade — the load-bearing failure mode of --reuse-values — and it is one of several Day-2 Helm operations where the verbs look correct but the semantics are not what most operators assume. This article covers the flag combinations that bite, how to inspect any past revision, how rollback actually works, and the snapshot-before-upgrade discipline that turns Helm’s revision storage into a real disaster-recovery backstop.

eval $(minikube docker-env) repoints the shell’s Docker client at the daemon running inside the minikube VM. A docker build afterwards lands the image directly in the cluster’s container store, so pods can pull it without a registry. The pattern is correct but unforgiving: every failure mode looks like a different problem (image pull error, runtime crash, stale pod) and only a handful of them actually point back to the env-var setup.

A receiver YAML passes static review and the helm release reports deployed. The alertmanager pod is Running 1/1. A real critical alert fires and goes nowhere. The alertmanager pod logs are clean. The receiver works fine for a hand-rolled curl to the webhook URL. The trap is that the prometheus-operator generated a Secret containing the rendered config but flagged a sync error in its own logs — and the alertmanager pod kept serving the previous-good rendering, silently. This article assumes familiarity with the basic alertmanager routing tree, receivers, inhibition rules, and templating covered in alertmanager-configuration. It extends that material with the Day-2 operations of the kube-prometheus-stack chart specifically: where errors actually surface, what the native receiver schemas allow (and don’t), and the silence discipline that keeps the alert pipeline trustworthy.

A single-node Kubernetes cluster running seven Helm-managed services concurrently — Gitea, Mattermost, PostgreSQL, kube-prometheus-stack, Jenkins, Temporal, and NATS — looks tractable on paper. The charts are all upstream-maintained. The hardware is modest but adequate. The operational reality is that zero of the seven ran cleanly on out-of-the-box values. Every chart needed at least one customization to coexist with the others, and several needed substantial rewrites of the helm-values surface. This survey catalogs what those customizations are, why each was necessary, and what the common failure modes look like across the fleet.

A minikube cluster on Apple Silicon looks like a pure Kubernetes problem until the first Docker Desktop crash. The failure modes that bite hardest on M-series Macs live one layer below the cluster: in Docker Desktop’s memory allocator, in QEMU’s address-space layout, and in the destructive default of minikube delete. None of these are mentioned in the standard minikube setup guide, and all three will eat real workload state when they fire. This is the operational layer on top of minikube setup and drivers and ARM64 K8s images — the host-side discipline that keeps the cluster alive.

A self-hosted Gitea forge running on Kubernetes covers four operational concerns that the upstream chart leaves to the operator: identity hygiene for bots and humans, branch protection rendered from code rather than clickops, webhook wiring to CI, and a backup story that survives a cluster wipe. The companion article Gitea Collaborator Grants and Review Officiality covers the narrow operational gotcha of official=false reviews; this article is the broader runbook for running the forge well.

A single-node minikube cluster on Docker Desktop runs the entire control plane, kubelet, every PVC, every Secret, and the container image cache inside one VM whose disk is one file: ~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw on macOS. When that file is lost or corrupted, every piece of cluster state goes with it in a single event. There is no “node failure vs storage failure” distinction to design around. Every backup strategy that assumes those are separable does not apply.

Agent Runbook Generation#

An agent that says “you should probably add a readiness probe to your deployment” is giving advice. An agent that hands you a tested manifest with the readiness probe configured, verified against a real cluster, with rollback steps if the probe misconfigures – that agent is producing a deliverable. The difference matters.

The core thesis of infrastructure agent work is that the output is always a deliverable – a runbook, playbook, tested manifest, or validated configuration – never a direct action on someone else’s systems. This article covers the complete workflow for generating those deliverables: understanding requirements, planning steps, executing in a sandbox, capturing what worked, and packaging the result.

AKS Identity and Security#

AKS identity operates at three levels: who can access the cluster API (authentication), what they can do inside it (authorization), and how pods authenticate to Azure services (workload identity). Each level has Azure-specific mechanisms that replace or extend vanilla Kubernetes patterns.

Entra ID Integration (Azure AD)#

AKS supports two Entra ID integration modes.

AKS-managed Azure AD: Enable with --enable-aad at cluster creation. AKS handles the app registrations and token validation. This is the recommended approach.