Builder Pool Naming: The (role, tier, replica) Coordinate#

Naming agent pools after the model they run today (kimi-N, deepseek-N, flash-N, lite-N) felt natural when each pool ran one model. It stopped feeling natural the third time a pool’s model churned — when the lite-tier swapped through qwen → gemma → gemini in six weeks and every rename cascaded through K8s manifests, secret names, MM bot accounts, Gitea identities, and helm values. The fix was to make pool names model-independent: builder-lite-0 runs whatever model the pool config says it runs today.

Docker-in-Docker on Jenkins: Postgres Tests Can’t Reach localhost#

A Jenkins job runs docker run -d -p 5432:5432 postgres:17-alpine and gets back a container ID. The next step is psql -h localhost -p 5432 -U postgres and it returns Connection refused. The retry loop tries 30 times and gives up. The test job fails with “could not connect to server”.

If you’ve added longer waits, switched to --network host, or rewritten the test script to launch its own postgres container, none of that will help. The problem is the network model: Jenkins running in a Kubernetes pod uses the host’s docker socket to launch SIBLING containers. Those siblings live on the host’s docker bridge network, not in Jenkins’s pod network namespace. localhost from inside Jenkins is the pod’s loopback; the published port is on the host’s interface.

Running Temporal Server on Minikube#

This guide deploys Temporal Server on a local Minikube cluster with PostgreSQL persistence. By the end, you will have the Temporal frontend, Web UI, and CLI all working against a real Kubernetes deployment.

If you need background on what Temporal is, start with Introduction to Temporal.

Prerequisites#

Your machine needs at least 4 CPU cores and 8 GB RAM available to Docker. For minikube driver details, see Minikube Setup and Drivers and Minikube Docker Driver.

Temporal High Availability#

A single-replica Temporal deployment works for development, but any pod going down takes the workflow engine offline. This guide configures a multi-replica cluster with proper resource allocation, Elasticsearch visibility, and health monitoring.

For the single-replica setup this builds on, see Running Temporal Server on Minikube.

Why HA Matters#

With multiple replicas, losing a pod triggers a brief rebalance (seconds), not an outage.

Multiple Temporal Servers on Minikube#

Running two independent Temporal Server instances locally lets you develop and test cross-cluster patterns – worker bridges, namespace replication, and multi-region failover – without cloud infrastructure. This article walks through deploying two Temporal clusters on minikube using profiles and connecting them over Docker networking.

All configuration files and Makefile targets reference the companion repository at github.com/statherm/temporal-examples in the multi-cluster/ directory.

Why Multiple Clusters?#

A single Temporal cluster handles most use cases. You need multiple clusters when:

What Crossplane Does#

Crossplane extends Kubernetes to provision and manage cloud infrastructure using the Kubernetes API. Instead of writing Terraform and running apply, you write Kubernetes manifests and kubectl apply them. Crossplane controllers reconcile the desired state with the actual cloud resources.

The real value is not replacing Terraform — it is building abstractions. Platform teams define custom resource types (like DatabaseClaim) that developers consume without knowing whether they are getting RDS, CloudSQL, or Azure Database. The composition layer maps the simple claim to the actual cloud resources.

Blue-Green Deployments#

A blue-green deployment runs two identical production environments. One (blue) serves live traffic. The other (green) is idle or running the new version. When the green environment passes validation, you switch traffic from blue to green. If something goes wrong, you switch back. The old environment stays running until you are confident the new version is stable.

The fundamental advantage over rolling updates is atomicity. Traffic switches from 100% old to 100% new in a single operation. There is no period where some users see the old version and others see the new one.

Self-Hosted CI Runners at Scale#

GitHub-hosted and GitLab SaaS runners work until they do not. You hit limits when you need private network access to deploy to internal infrastructure, specific hardware like GPUs or ARM64 machines, compliance requirements that prohibit running code on shared infrastructure, or cost control when you are burning thousands of dollars per month on hosted runner minutes.

Self-hosted runners solve these problems but introduce operational complexity: you now own runner provisioning, scaling, security, image updates, and cost management.

A single-node Kubernetes cluster running seven Helm-managed services concurrently — Gitea, Mattermost, PostgreSQL, kube-prometheus-stack, Jenkins, Temporal, and NATS — looks tractable on paper. The charts are all upstream-maintained. The hardware is modest but adequate. The operational reality is that zero of the seven ran cleanly on out-of-the-box values. Every chart needed at least one customization to coexist with the others, and several needed substantial rewrites of the helm-values surface. This survey catalogs what those customizations are, why each was necessary, and what the common failure modes look like across the fleet.

A self-hosted Gitea forge running on Kubernetes covers four operational concerns that the upstream chart leaves to the operator: identity hygiene for bots and humans, branch protection rendered from code rather than clickops, webhook wiring to CI, and a backup story that survives a cluster wipe. The companion article Gitea Collaborator Grants and Review Officiality covers the narrow operational gotcha of official=false reviews; this article is the broader runbook for running the forge well.