{"page":{"agent_metadata":{"content_type":"guide","outputs":["build-temporal-worker-bridge","execute-cross-cluster-workflows","implement-cross-cluster-idempotency","deploy-bridge-to-kubernetes"],"prerequisites":["temporal-container-lifecycle-workflow","temporal-cross-cluster-communication"]},"categories":["workflow-orchestration"],"content_plain":"Building a Temporal Worker Bridge# The architecture article evaluated three cross-cluster communication patterns and identified the worker bridge as the best fit for most open-source Temporal deployments. This article builds the bridge.\nThe worker bridge is a single binary that holds connections to two Temporal clusters. It polls Cluster A for tasks on a dedicated queue and executes those tasks using Cluster B\u0026rsquo;s resources \u0026ndash; its Temporal client, databases, APIs, and services. From Cluster A\u0026rsquo;s perspective, the bridge is just another worker. From Cluster B\u0026rsquo;s perspective, the bridge is just another client starting workflows.\nAll code is in the companion repository at github.com/statherm/temporal-examples under cross-cluster/bridge/.\nBridge Architecture Recap# Cluster A (Source) Cluster B (Execution) ┌─────────────────────┐ ┌─────────────────────┐ │ Temporal Server A │ │ Temporal Server B │ │ │ │ │ │ CrossClusterWF │ ┌────────┐ │ FulfillmentWF │ │ ───────────────── │ │ Bridge │ │ ───────────────── │ │ Task Queue: │◄─│ Worker │─►│ Task Queue: │ │ bridge-queue │ │ │ │ fulfillment-queue │ │ │ │ Polls │ │ │ │ Waits for result │ │ A │ │ Executes locally │ │ from bridge │ │ Starts │ │ Returns result │ │ │ │ on B │ │ │ └─────────────────────┘ └────────┘ └─────────────────────┘The bridge worker process:\nConnects to Cluster A (source) and registers on the bridge-queue task queue Connects to Cluster B (execution target) for starting and monitoring remote workflows When a task arrives on bridge-queue, the bridge activity starts a corresponding workflow on Cluster B The bridge activity polls the Cluster B workflow until it completes, then returns the result to Cluster A The Bridge Worker Binary# The bridge binary establishes two independent Temporal client connections at startup. Configuration comes from environment variables, making it easy to deploy in different environments.\npackage main import ( \u0026#34;log\u0026#34; \u0026#34;os\u0026#34; \u0026#34;go.temporal.io/sdk/client\u0026#34; \u0026#34;go.temporal.io/sdk/worker\u0026#34; \u0026#34;github.com/statherm/temporal-examples/cross-cluster/bridge\u0026#34; ) func main() { // Connect to Cluster A — the source of bridge tasks clientA, err := client.Dial(client.Options{ HostPort: getEnv(\u0026#34;CLUSTER_A_HOST\u0026#34;, \u0026#34;localhost:7233\u0026#34;), Namespace: getEnv(\u0026#34;CLUSTER_A_NAMESPACE\u0026#34;, \u0026#34;default\u0026#34;), }) if err != nil { log.Fatalf(\u0026#34;Failed to connect to Cluster A: %v\u0026#34;, err) } defer clientA.Close() // Connect to Cluster B — where work actually executes clientB, err := client.Dial(client.Options{ HostPort: getEnv(\u0026#34;CLUSTER_B_HOST\u0026#34;, \u0026#34;localhost:7234\u0026#34;), Namespace: getEnv(\u0026#34;CLUSTER_B_NAMESPACE\u0026#34;, \u0026#34;default\u0026#34;), }) if err != nil { log.Fatalf(\u0026#34;Failed to connect to Cluster B: %v\u0026#34;, err) } defer clientB.Close() // Create bridge activities with Cluster B client bridgeActivities := bridge.NewBridgeActivities(clientB, bridge.BridgeConfig{ SourceCluster: getEnv(\u0026#34;CLUSTER_A_NAME\u0026#34;, \u0026#34;cluster-a\u0026#34;), DestinationCluster: getEnv(\u0026#34;CLUSTER_B_NAME\u0026#34;, \u0026#34;cluster-b\u0026#34;), PollInterval: 5 * time.Second, MaxPollAttempts: 720, // 1 hour at 5s intervals }) // Register worker on Cluster A\u0026#39;s bridge queue w := worker.New(clientA, \u0026#34;bridge-queue\u0026#34;, worker.Options{ MaxConcurrentActivityExecutionSize: 10, }) w.RegisterWorkflow(bridge.CrossClusterWorkflow) w.RegisterActivity(bridgeActivities) log.Println(\u0026#34;Bridge worker starting\u0026#34;, \u0026#34;clusterA\u0026#34;, getEnv(\u0026#34;CLUSTER_A_HOST\u0026#34;, \u0026#34;localhost:7233\u0026#34;), \u0026#34;clusterB\u0026#34;, getEnv(\u0026#34;CLUSTER_B_HOST\u0026#34;, \u0026#34;localhost:7234\u0026#34;)) if err := w.Run(worker.InterruptCh()); err != nil { log.Fatalf(\u0026#34;Bridge worker failed: %v\u0026#34;, err) } } func getEnv(key, fallback string) string { if v := os.Getenv(key); v != \u0026#34;\u0026#34; { return v } return fallback }Key points:\nclientA is used only for polling tasks. The worker registers on Cluster A\u0026rsquo;s bridge-queue. clientB is used for executing work. Bridge activities use it to start workflows, query status, and retrieve results from Cluster B. The two clients are completely independent. They can point to different Temporal versions, different namespaces, and different network segments. Bridge Activities# Bridge activities wrap the Cluster B client and provide three operations: start a remote workflow, query its status, and wait for completion.\npackage bridge import ( \u0026#34;context\u0026#34; \u0026#34;errors\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;time\u0026#34; \u0026#34;go.temporal.io/api/enums/v1\u0026#34; \u0026#34;go.temporal.io/api/serviceerror\u0026#34; \u0026#34;go.temporal.io/sdk/activity\u0026#34; \u0026#34;go.temporal.io/sdk/client\u0026#34; ) type BridgeConfig struct { SourceCluster string DestinationCluster string PollInterval time.Duration MaxPollAttempts int } type BridgeActivities struct { remoteClient client.Client config BridgeConfig } func NewBridgeActivities(remoteClient client.Client, config BridgeConfig) *BridgeActivities { return \u0026amp;BridgeActivities{ remoteClient: remoteClient, config: config, } } type RemoteWorkflowRequest struct { SourceWorkflowID string WorkflowType string TaskQueue string Input []byte // Serialized workflow input Timeout time.Duration } type RemoteWorkflowResult struct { WorkflowID string RunID string Status string Output []byte // Serialized workflow output }StartWorkflowInRemote# This activity starts a workflow on Cluster B with an idempotent workflow ID derived from the source workflow:\nfunc (b *BridgeActivities) StartWorkflowInRemote( ctx context.Context, req RemoteWorkflowRequest, ) (*RemoteWorkflowResult, error) { logger := activity.GetLogger(ctx) // Generate deterministic workflow ID for idempotency remoteWorkflowID := GenerateCrossClusterKey( b.config.SourceCluster, req.SourceWorkflowID, req.WorkflowType, ) timeout := req.Timeout if timeout == 0 { timeout = 1 * time.Hour } opts := client.StartWorkflowOptions{ ID: remoteWorkflowID, TaskQueue: req.TaskQueue, WorkflowExecutionTimeout: timeout, } we, err := b.remoteClient.ExecuteWorkflow(ctx, opts, req.WorkflowType, req.Input) if err != nil { // Handle already-started as success (idempotent retry) var alreadyStarted *serviceerror.WorkflowExecutionAlreadyStarted if errors.As(err, \u0026amp;alreadyStarted) { logger.Info(\u0026#34;Remote workflow already exists, attaching\u0026#34;, \u0026#34;workflowID\u0026#34;, remoteWorkflowID) return \u0026amp;RemoteWorkflowResult{ WorkflowID: remoteWorkflowID, Status: \u0026#34;already_running\u0026#34;, }, nil } return nil, fmt.Errorf(\u0026#34;start remote workflow failed: %w\u0026#34;, err) } logger.Info(\u0026#34;Started remote workflow\u0026#34;, \u0026#34;workflowID\u0026#34;, we.GetID(), \u0026#34;runID\u0026#34;, we.GetRunID(), \u0026#34;cluster\u0026#34;, b.config.DestinationCluster) return \u0026amp;RemoteWorkflowResult{ WorkflowID: we.GetID(), RunID: we.GetRunID(), Status: \u0026#34;started\u0026#34;, }, nil }WaitForRemoteCompletion# This activity polls the remote workflow until it completes, using activity heartbeating to prevent Temporal from timing out the long-running poll:\nfunc (b *BridgeActivities) WaitForRemoteCompletion( ctx context.Context, workflowID string, runID string, ) (*RemoteWorkflowResult, error) { logger := activity.GetLogger(ctx) for attempt := 0; attempt \u0026lt; b.config.MaxPollAttempts; attempt++ { // Heartbeat to prevent activity timeout activity.RecordHeartbeat(ctx, fmt.Sprintf(\u0026#34;poll attempt %d\u0026#34;, attempt)) // Check if activity context is cancelled if ctx.Err() != nil { return nil, ctx.Err() } // Describe the remote workflow execution resp, err := b.remoteClient.DescribeWorkflowExecution(ctx, workflowID, runID) if err != nil { logger.Warn(\u0026#34;Failed to describe remote workflow, retrying\u0026#34;, \u0026#34;workflowID\u0026#34;, workflowID, \u0026#34;error\u0026#34;, err, \u0026#34;attempt\u0026#34;, attempt) time.Sleep(b.config.PollInterval) continue } status := resp.WorkflowExecutionInfo.Status switch status { case enums.WORKFLOW_EXECUTION_STATUS_COMPLETED: logger.Info(\u0026#34;Remote workflow completed\u0026#34;, \u0026#34;workflowID\u0026#34;, workflowID) // Get the result run := b.remoteClient.GetWorkflow(ctx, workflowID, runID) var output []byte if err := run.Get(ctx, \u0026amp;output); err != nil { return nil, fmt.Errorf(\u0026#34;get remote workflow result: %w\u0026#34;, err) } return \u0026amp;RemoteWorkflowResult{ WorkflowID: workflowID, RunID: runID, Status: \u0026#34;completed\u0026#34;, Output: output, }, nil case enums.WORKFLOW_EXECUTION_STATUS_FAILED: return \u0026amp;RemoteWorkflowResult{ WorkflowID: workflowID, RunID: runID, Status: \u0026#34;failed\u0026#34;, }, fmt.Errorf(\u0026#34;remote workflow failed: %s\u0026#34;, workflowID) case enums.WORKFLOW_EXECUTION_STATUS_CANCELED: return \u0026amp;RemoteWorkflowResult{ WorkflowID: workflowID, RunID: runID, Status: \u0026#34;canceled\u0026#34;, }, fmt.Errorf(\u0026#34;remote workflow was canceled: %s\u0026#34;, workflowID) case enums.WORKFLOW_EXECUTION_STATUS_TERMINATED: return \u0026amp;RemoteWorkflowResult{ WorkflowID: workflowID, RunID: runID, Status: \u0026#34;terminated\u0026#34;, }, fmt.Errorf(\u0026#34;remote workflow was terminated: %s\u0026#34;, workflowID) default: // Still running, wait and poll again logger.Debug(\u0026#34;Remote workflow still running\u0026#34;, \u0026#34;workflowID\u0026#34;, workflowID, \u0026#34;status\u0026#34;, status, \u0026#34;attempt\u0026#34;, attempt) } time.Sleep(b.config.PollInterval) } return nil, fmt.Errorf(\u0026#34;timed out waiting for remote workflow %s after %d attempts\u0026#34;, workflowID, b.config.MaxPollAttempts) }Cross-Cluster Idempotency# Idempotency is critical in bridge scenarios. The bridge activity may be retried (worker crash, network timeout, Temporal retry policy), and each retry must not create a duplicate workflow on the remote cluster.\nThe key strategy is deterministic workflow IDs:\nfunc GenerateCrossClusterKey(sourceCluster, workflowID, activityName string) string { return fmt.Sprintf(\u0026#34;bridge-%s-%s-%s\u0026#34;, sourceCluster, workflowID, activityName) } func GenerateKey(workflowID, activityName, attempt string) string { return fmt.Sprintf(\u0026#34;%s:%s:%s\u0026#34;, workflowID, activityName, attempt) }By constructing the remote workflow ID from the source cluster name, source workflow ID, and activity type, we guarantee that retries of the same bridge activity always target the same remote workflow. Temporal\u0026rsquo;s \u0026ldquo;workflow already exists\u0026rdquo; semantics handle the deduplication.\nThis means:\nFirst attempt: starts remote workflow bridge-cluster-a-order-123-fulfillment Bridge worker crashes and retries: tries to start bridge-cluster-a-order-123-fulfillment again, gets WorkflowExecutionAlreadyStarted, attaches to existing execution Result: exactly one fulfillment workflow runs, regardless of how many times the bridge retries The CrossCluster Workflow# The workflow that runs on Cluster A orchestrates the bridge activities:\npackage bridge import ( \u0026#34;fmt\u0026#34; \u0026#34;time\u0026#34; \u0026#34;go.temporal.io/sdk/temporal\u0026#34; \u0026#34;go.temporal.io/sdk/workflow\u0026#34; ) type CrossClusterRequest struct { WorkflowType string TaskQueue string Input []byte Timeout time.Duration } func CrossClusterWorkflow(ctx workflow.Context, req CrossClusterRequest) (*RemoteWorkflowResult, error) { logger := workflow.GetLogger(ctx) // Activity options with heartbeat for long-running poll activityOpts := workflow.ActivityOptions{ StartToCloseTimeout: 2 * time.Hour, HeartbeatTimeout: 30 * time.Second, RetryPolicy: \u0026amp;temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Minute, MaximumAttempts: 5, }, } ctx = workflow.WithActivityOptions(ctx, activityOpts) workflowInfo := workflow.GetInfo(ctx) // Step 1: Start the remote workflow remoteReq := RemoteWorkflowRequest{ SourceWorkflowID: workflowInfo.WorkflowExecution.ID, WorkflowType: req.WorkflowType, TaskQueue: req.TaskQueue, Input: req.Input, Timeout: req.Timeout, } var startResult *RemoteWorkflowResult err := workflow.ExecuteActivity(ctx, (*BridgeActivities).StartWorkflowInRemote, remoteReq). Get(ctx, \u0026amp;startResult) if err != nil { return nil, fmt.Errorf(\u0026#34;failed to start remote workflow: %w\u0026#34;, err) } logger.Info(\u0026#34;Remote workflow started\u0026#34;, \u0026#34;remoteWorkflowID\u0026#34;, startResult.WorkflowID, \u0026#34;status\u0026#34;, startResult.Status) // Step 2: Wait for the remote workflow to complete var finalResult *RemoteWorkflowResult err = workflow.ExecuteActivity(ctx, (*BridgeActivities).WaitForRemoteCompletion, startResult.WorkflowID, startResult.RunID). Get(ctx, \u0026amp;finalResult) if err != nil { return nil, fmt.Errorf(\u0026#34;remote workflow execution failed: %w\u0026#34;, err) } logger.Info(\u0026#34;Remote workflow completed\u0026#34;, \u0026#34;remoteWorkflowID\u0026#34;, finalResult.WorkflowID, \u0026#34;status\u0026#34;, finalResult.Status) return finalResult, nil }Error Handling# Cross-cluster operations have more failure modes than single-cluster workflows. The bridge must handle:\nCluster B Unreachable# If the bridge cannot connect to Cluster B, StartWorkflowInRemote fails. Temporal\u0026rsquo;s retry policy on the activity retries with exponential backoff. If Cluster B is down for an extended period, the activity eventually exhausts its retries and the calling workflow receives an error.\nConfigure retry policies based on your SLA:\nretryPolicy := \u0026amp;temporal.RetryPolicy{ InitialInterval: 5 * time.Second, BackoffCoefficient: 2.0, MaximumInterval: 5 * time.Minute, MaximumAttempts: 0, // Retry indefinitely NonRetryableErrorTypes: []string{\u0026#34;INVALID_REQUEST\u0026#34;}, // Don\u0026#39;t retry bad inputs }Remote Workflow Failure# If the workflow on Cluster B fails, WaitForRemoteCompletion detects it via the WORKFLOW_EXECUTION_STATUS_FAILED status and returns an error to the calling workflow. The calling workflow can then decide to retry, compensate, or escalate.\nBridge Worker Crash During Poll# If the bridge worker crashes while polling for remote completion, Temporal detects the missed heartbeat, times out the activity, and schedules it on another bridge worker. The new worker resumes polling with the same remote workflow ID. Because the remote workflow ID is deterministic, it attaches to the same execution.\nDeploying the Bridge# The bridge runs as a standard Kubernetes Deployment. It needs network access to both clusters.\napiVersion: apps/v1 kind: Deployment metadata: name: temporal-bridge-worker labels: app: temporal-bridge spec: replicas: 2 # HA: at least 2 replicas selector: matchLabels: app: temporal-bridge template: metadata: labels: app: temporal-bridge spec: containers: - name: bridge image: ghcr.io/statherm/temporal-bridge:latest env: - name: CLUSTER_A_HOST valueFrom: configMapKeyRef: name: bridge-config key: cluster-a-host - name: CLUSTER_A_NAMESPACE value: \u0026#34;default\u0026#34; - name: CLUSTER_B_HOST valueFrom: configMapKeyRef: name: bridge-config key: cluster-b-host - name: CLUSTER_B_NAMESPACE value: \u0026#34;default\u0026#34; - name: CLUSTER_A_NAME value: \u0026#34;cluster-a\u0026#34; - name: CLUSTER_B_NAME value: \u0026#34;cluster-b\u0026#34; resources: requests: cpu: 250m memory: 256Mi limits: cpu: 500m memory: 512Mi readinessProbe: exec: command: - /bin/sh - -c - \u0026#34;pgrep -f bridge-worker\u0026#34; initialDelaySeconds: 5 periodSeconds: 10 --- apiVersion: v1 kind: ConfigMap metadata: name: bridge-config data: cluster-a-host: \u0026#34;temporal-frontend.temporal-cluster-a.svc.cluster.local:7233\u0026#34; cluster-b-host: \u0026#34;temporal-frontend.default.svc.cluster.local:7233\u0026#34;Deploy the bridge to Cluster B (where it has local access to Cluster B\u0026rsquo;s resources):\nminikube profile temporal-cluster-b kubectl apply -f deploy/bridge-worker.yamlEnd-to-End Test# With both clusters running (see Multi-Cluster Minikube Setup) and the bridge deployed, test the full flow.\nStart a cross-cluster workflow on Cluster A:\n# Ensure you are pointing at Cluster A export TEMPORAL_ADDRESS=localhost:7233 # Start the cross-cluster workflow temporal workflow start \\ --task-queue bridge-queue \\ --type CrossClusterWorkflow \\ --input \u0026#39;{\u0026#34;WorkflowType\u0026#34;:\u0026#34;FulfillmentWorkflow\u0026#34;,\u0026#34;TaskQueue\u0026#34;:\u0026#34;fulfillment-queue\u0026#34;,\u0026#34;Input\u0026#34;:\u0026#34;eyJvcmRlcklEIjoiMTIzIn0=\u0026#34;}\u0026#39;Monitor the workflow in both Web UIs:\nCluster A (http://localhost:8080): Shows the CrossClusterWorkflow running, with StartWorkflowInRemote and WaitForRemoteCompletion activities Cluster B (http://localhost:8081): Shows the FulfillmentWorkflow started by the bridge The companion repository provides a Makefile target for the full flow:\nmake clusters-up deploy-bridge start-cross-cluster-workflowThis starts both clusters, deploys the bridge, starts a test workflow on Cluster A, and tails the bridge worker logs.\nMonitoring and Observability# Bridge workers need additional monitoring beyond standard Temporal worker metrics.\nKey Metrics# Track these bridge-specific metrics:\nMetric Description Alert Threshold bridge_cross_cluster_latency_ms Time from activity start to remote workflow completion p99 \u0026gt; 60s bridge_queue_depth Number of pending tasks on bridge-queue \u0026gt; 100 bridge_remote_start_errors Failed remote workflow starts per minute \u0026gt; 5/min bridge_remote_poll_errors Failed status polls per minute \u0026gt; 10/min bridge_heartbeat_age_ms Time since last successful heartbeat \u0026gt; 25s Structured Logging# Every bridge log line should include both cluster contexts:\nlogger.Info(\u0026#34;Bridge activity executing\u0026#34;, \u0026#34;sourceCluster\u0026#34;, b.config.SourceCluster, \u0026#34;destinationCluster\u0026#34;, b.config.DestinationCluster, \u0026#34;sourceWorkflowID\u0026#34;, req.SourceWorkflowID, \u0026#34;remoteWorkflowID\u0026#34;, remoteWorkflowID, \u0026#34;remoteTaskQueue\u0026#34;, req.TaskQueue)This makes it possible to correlate events across clusters when debugging. Use a shared request ID or trace ID that propagates through both clusters.\nProduction Considerations# Bridge Replicas# Run at least two bridge replicas. Temporal distributes tasks across all workers polling the same task queue, so multiple bridges provide both throughput and availability. If one bridge crashes, the other continues processing tasks, and Temporal reschedules the failed bridge\u0026rsquo;s in-progress activities after the heartbeat timeout.\nGraceful Shutdown# The bridge binary uses worker.InterruptCh() to handle SIGTERM gracefully. On shutdown, it stops polling for new tasks and waits for in-progress activities to complete (up to a configured drain timeout). In Kubernetes, set terminationGracePeriodSeconds to match your longest expected activity duration.\nVersion Compatibility# The bridge connects to two Temporal servers that may run different versions. The Temporal Go SDK is backward-compatible with older server versions, but not forward-compatible. Run the bridge with the SDK version matching your newer cluster, and verify compatibility with the older cluster in staging first.\nCapacity Planning# Each bridge activity holds two open connections (one to each cluster) and one polling goroutine for the remote workflow. With MaxConcurrentActivityExecutionSize: 10, each bridge replica handles 10 concurrent cross-cluster operations. Scale replicas based on your expected cross-cluster throughput.\nNext Steps# You now have a working cross-cluster Temporal setup with a bridge pattern. This forms the foundation for multi-region deployment strategies. For the underlying signal-based coordination patterns, see Temporal Signals for Automated Coordination. For the infrastructure setup, see Multiple Temporal Servers on Minikube. For the container lifecycle workflows that the bridge can orchestrate, see Container Lifecycle Workflow.\n","date":"2026-02-22","description":"Implement a Temporal worker bridge that polls one cluster for tasks and executes them using a second cluster's resources, with cross-cluster idempotency and Kubernetes deployment.","lastmod":"2026-02-22","levels":["advanced"],"reading_time_minutes":11,"section":"knowledge","skills":["temporal-bridge-implementation","cross-cluster-workflow-execution","multi-client-management","cross-cluster-idempotency"],"tags":["temporal","cross-cluster","worker-bridge","multi-cluster","distributed-execution","bridge-pattern","idempotency"],"title":"Building a Temporal Worker Bridge: Cluster A Jobs Executed in Cluster B","tools":["temporal","go","docker","minikube"],"url":"https://agent-zone.ai/knowledge/workflow-orchestration/temporal-cross-cluster-worker-bridge/","word_count":2157}}