Multiple Temporal Servers on Minikube#

Running two independent Temporal Server instances locally lets you develop and test cross-cluster patterns – worker bridges, namespace replication, and multi-region failover – without cloud infrastructure. This article walks through deploying two Temporal clusters on minikube using profiles and connecting them over Docker networking.

All configuration files and Makefile targets reference the companion repository at github.com/statherm/temporal-examples in the multi-cluster/ directory.

Why Multiple Clusters?#

A single Temporal cluster handles most use cases. You need multiple clusters when:

What Crossplane Does#

Crossplane extends Kubernetes to provision and manage cloud infrastructure using the Kubernetes API. Instead of writing Terraform and running apply, you write Kubernetes manifests and kubectl apply them. Crossplane controllers reconcile the desired state with the actual cloud resources.

The real value is not replacing Terraform — it is building abstractions. Platform teams define custom resource types (like DatabaseClaim) that developers consume without knowing whether they are getting RDS, CloudSQL, or Azure Database. The composition layer maps the simple claim to the actual cloud resources.

Blue-Green Deployments#

A blue-green deployment runs two identical production environments. One (blue) serves live traffic. The other (green) is idle or running the new version. When the green environment passes validation, you switch traffic from blue to green. If something goes wrong, you switch back. The old environment stays running until you are confident the new version is stable.

The fundamental advantage over rolling updates is atomicity. Traffic switches from 100% old to 100% new in a single operation. There is no period where some users see the old version and others see the new one.

DR Runbook Design: Failover Procedures, Communication Plans, and Decision Trees#

A DR runbook is used during the worst moments of an engineer’s career: systems are down, customers are impacted, leadership is asking for updates, and decisions carry real consequences. The runbook must be clear enough that someone running on adrenaline and three hours of sleep can execute it correctly.

This means: short sentences, numbered steps, explicit commands (copy-paste ready), no ambiguity about who does what, and timing estimates for each phase so the incident commander knows if things are taking too long.

Kubernetes Cluster Disaster Recovery#

Your cluster will fail. The question is whether you can rebuild it in hours or weeks. Kubernetes DR is not a single tool – it is a layered strategy combining etcd snapshots, resource-level backups, GitOps state, and tested recovery procedures.

The three layers of Kubernetes DR: etcd gives you raw cluster state, Velero gives you portable resource and volume backups, and GitOps gives you declarative rebuild capability. You need at least two of these.

Multi-Region Kubernetes#

Running Kubernetes in a single region is a single point of failure at the infrastructure level. Region outages are rare but real – AWS us-east-1 has gone down multiple times, taking entire companies offline. Multi-region Kubernetes addresses this, but it introduces complexity in networking, state management, and deployment coordination that you must handle deliberately.

Independent Clusters with Shared GitOps#

The simplest multi-region pattern: run completely independent clusters in each region, deploy the same applications to all of them using GitOps, and route traffic with DNS or a global load balancer.

Cloud Multi-Region Architecture Patterns#

Multi-region is not just running clusters in two places. It is the networking between them, the data replication strategy, the traffic routing, and the cost of keeping it all running. Each cloud provider has different primitives and different pricing models. Here is how to build it on each.

The three pillars: a Kubernetes cluster per region for compute, a global traffic routing layer to direct users to the nearest healthy region, and a multi-region database for state. Get any one wrong and multi-region gives you complexity without resilience.

Stateful Workload Disaster Recovery#

Stateless workloads are easy to recover – redeploy from Git and they are running. Stateful workloads carry data that cannot be regenerated. Databases, message queues, object stores, and anything with a PersistentVolume needs a deliberate DR strategy that goes beyond “we have Velero.”

The fundamental challenge: you must capture data at a point in time where the application state is consistent, replicate that data to a recovery site, and restore it in the correct order. Get any of these wrong and you recover corrupted data or a broken dependency chain.

An Autonomous PR-to-Deploy Loop#

The goal: a contributor (human or agent) opens a PR; if it passes CI and gets the required approvals, it merges and deploys itself with no human clicking buttons. The loop:

PR → CI gate (required status) → N approvals → auto-merge → auto-tag → build image:<tag> → deploy (pin tag)

This is buildable on plain Jenkins/Gitea/Kubernetes (or GitHub/Actions/Argo equivalents). The pieces are independent; wire them in order.

Jenkins Multibranch with Gitea: Why Pull Request Builds Never Run#

A common, maddening symptom: your Jenkins organization folder (or multibranch pipeline) backed by Gitea builds the default branch fine, but pull request commits never build — the commit status stays pending forever (or never appears), so a branch-protection gate that requires a CI status can never be satisfied and the PR can never merge.

The pipeline is fine. The problem is branch/PR discovery configuration, and there are several layered traps. Here is how to diagnose and fix each.