---
title: "ARM64 Kubernetes: The QEMU Problem with Go Binaries"
description: "QEMU user-mode emulation cannot reliably run Go binaries on ARM64. Learn how to diagnose the lfstack crash, build native ARM64 images, and work around missing multi-arch support."
url: https://agent-zone.ai/knowledge/kubernetes/arm64-k8s-images/
section: knowledge
date: 2026-02-21
categories: ["kubernetes"]
tags: ["arm64","qemu","go","images","minikube"]
skills: ["arm64-container-builds","image-debugging"]
tools: ["docker","minikube","kubectl"]
levels: ["intermediate"]
word_count: 615
formats:
  json: https://agent-zone.ai/knowledge/kubernetes/arm64-k8s-images/index.json
  html: https://agent-zone.ai/knowledge/kubernetes/arm64-k8s-images/?format=html
  api: https://api.agent-zone.ai/api/v1/knowledge/search?q=ARM64+Kubernetes%3A+The+QEMU+Problem+with+Go+Binaries
---


# ARM64 Kubernetes: The QEMU Problem with Go Binaries

If you run Kubernetes on Apple Silicon (M1/M2/M3/M4) via minikube with the Docker driver, you will eventually try to run an amd64-only container image. For most software this works through QEMU emulation. For Go binaries, it crashes hard.

## The Problem

Go's garbage collector uses a lock-free stack (`lfstack`) that packs pointers with counter bits in the high bits of a 64-bit integer. QEMU's user-mode address translation changes the effective address space layout, which breaks this packing assumption. The result:

```
runtime: lfstack.push invalid packing: node 0xffff8b410100 cnt 0x1 packed 0x8b41010000000001 -> node 0xffff00008b410100
fatal error: lfstack.push

goroutine 1 [running]:
runtime.throw({0x2bfaa07, 0x14})
```

This is not a random flake. It is a fundamental incompatibility between QEMU user-mode emulation and Go's runtime memory model on ARM64 hosts. Every Go binary running under QEMU on ARM64 is affected.

## Why Rosetta Does Not Help in Minikube

Docker Desktop on macOS can use Apple's Rosetta 2 for amd64 emulation, which handles Go binaries correctly. However, minikube with the Docker driver runs its own containerd inside the minikube VM. That containerd does not use Rosetta -- it falls back to QEMU. So even if Docker Desktop has Rosetta enabled, images pulled inside minikube still crash.

## Diagnosing the Issue

When a pod is crash-looping with no obvious application error, check the logs for the lfstack signature:

```bash
kubectl logs <pod-name> -n <namespace> --previous
# Look for: "runtime: lfstack.push invalid packing"
```

Confirm the image architecture:

```bash
# Check what architecture an image was built for
docker manifest inspect mattermost/mattermost-team-edition:latest | jq '.manifests[].platform'
# If you only see "amd64", you have the problem
```

## Solution 1: Build a Native ARM64 Image

Many projects publish ARM64 binaries even when they do not publish ARM64 Docker images. Mattermost is a prime example -- there is no official ARM64 image, but ARM64 binary tarballs are available:

```
https://releases.mattermost.com/10.4.2/mattermost-team-10.4.2-linux-arm64.tar.gz
```

Create a minimal Dockerfile that uses the native binary:

```dockerfile
FROM ubuntu:22.04

RUN apt-get update && apt-get install -y ca-certificates curl && rm -rf /var/lib/apt/lists/*

ARG MM_VERSION=10.4.2
RUN curl -L https://releases.mattermost.com/${MM_VERSION}/mattermost-team-${MM_VERSION}-linux-arm64.tar.gz \
    | tar xz -C /opt

RUN mkdir -p /opt/mattermost/data /opt/mattermost/logs /opt/mattermost/config \
    && adduser --system --group mattermost \
    && chown -R mattermost:mattermost /opt/mattermost

USER mattermost
WORKDIR /opt/mattermost

EXPOSE 8065
ENTRYPOINT ["/opt/mattermost/bin/mattermost"]
```

Build it directly inside minikube's Docker daemon so you do not need a registry:

```bash
# Point your shell's Docker client at minikube's Docker
eval $(minikube docker-env)

# Build the image -- it is now available inside minikube
docker build -t mattermost-arm64:10.4.2 ./docker/mattermost-arm64/
```

Then reference the local image in your Helm values with `imagePullPolicy: Never`:

```yaml
image:
  repository: mattermost-arm64
  tag: "10.4.2"
  pullPolicy: Never
```

## Solution 2: QEMU for Non-Go amd64 Images

If you need to run an amd64 image that is not built with Go (Python, Node.js, C/C++ applications), QEMU works fine. Install the binfmt handlers:

```bash
docker run --privileged --rm tonistiigi/binfmt --install amd64
```

This registers QEMU as the handler for amd64 binaries. It persists across Docker restarts but not across host reboots.

## Decision Flowchart

When you encounter an amd64-only image on ARM64 Kubernetes:

1. **Is the application written in Go?** Check the Dockerfile or project repository.
2. **If yes:** Do NOT use QEMU. Find or build a native ARM64 image.
3. **If no:** Install QEMU binfmt support and run the amd64 image directly.
4. **If unsure:** Try running it. If you see the `lfstack.push` crash, it is Go -- switch to a native build.

## Key Takeaways

- The `lfstack.push invalid packing` error is always QEMU + Go on ARM64. There is no workaround short of native binaries.
- `eval $(minikube docker-env)` lets you build images directly into minikube without pushing to a registry.
- Check for ARM64 binary releases before assuming you need to cross-compile from source.
- QEMU is perfectly fine for non-Go workloads -- do not avoid it entirely, just for Go.