---
title: "kubectl debug and Ephemeral Containers: Non-Invasive Production Debugging"
description: "Use kubectl debug and ephemeral containers to troubleshoot distroless images, inspect running processes, and debug node-level issues without restarting pods."
url: https://agent-zone.ai/knowledge/kubernetes/kubectl-debug-and-ephemeral-containers/
section: knowledge
date: 2026-02-21
categories: ["kubernetes"]
tags: ["kubectl-debug","ephemeral-containers","distroless","production-debugging","troubleshooting"]
skills: ["ephemeral-container-debugging","process-inspection","node-debugging","network-troubleshooting"]
tools: ["kubectl"]
levels: ["intermediate"]
word_count: 1353
formats:
  json: https://agent-zone.ai/knowledge/kubernetes/kubectl-debug-and-ephemeral-containers/index.json
  html: https://agent-zone.ai/knowledge/kubernetes/kubectl-debug-and-ephemeral-containers/?format=html
  api: https://api.agent-zone.ai/api/v1/knowledge/search?q=kubectl+debug+and+Ephemeral+Containers%3A+Non-Invasive+Production+Debugging
---


# kubectl debug and Ephemeral Containers

Production containers should be minimal. Distroless images, scratch-based Go binaries, and hardened base images strip out shells, package managers, and debugging tools. This is good for security and image size, but it means `kubectl exec` gives you nothing to work with. Ephemeral containers solve this problem.

## The Problem

A typical distroless container has no shell:

```bash
$ kubectl exec -it payments-api-7f8b9c6d4-x2k9m -- /bin/sh
OCI runtime exec failed: exec failed: unable to start container process:
exec: "/bin/sh": stat /bin/sh: no such file or directory
```

You cannot install tools, you cannot inspect files, and you cannot run any diagnostic commands. The application is returning 500 errors and you have nothing but logs.

## Ephemeral Containers

Ephemeral containers (GA since Kubernetes v1.25) are temporary containers that you add to a running pod for debugging. They are not defined in the pod spec. They do not restart if they exit. They exist purely for investigation.

The `kubectl debug` command is the primary interface for creating ephemeral containers. It operates in three distinct modes.

## Mode 1: Attach to an Existing Pod

This is the most common mode. You add a debug container directly to the running pod without restarting it.

```bash
kubectl debug -it payments-api-7f8b9c6d4-x2k9m \
  --image=nicolaka/netshoot \
  --target=payments-api \
  -n payments-prod
```

What happens:
- An ephemeral container running `nicolaka/netshoot` is added to the existing pod
- The `--target=payments-api` flag shares the process namespace with the `payments-api` container
- You get a shell inside the debug container with full networking tools
- The original pod keeps running -- no restart, no downtime

### Process namespace sharing

With `--target`, the debug container shares the PID namespace of the target container. This means `ps aux` in the debug container shows the target's processes:

```bash
# Inside the debug container
$ ps aux
PID   USER     COMMAND
1     root     /app/payments-api --config=/etc/config/app.yaml
12    root     sh   # this is you, in the debug container
```

### Filesystem access

You can reach the target container's filesystem through `/proc/1/root/`:

```bash
# Read the target container's config file
cat /proc/1/root/etc/config/app.yaml

# Check mounted secrets
ls /proc/1/root/var/run/secrets/kubernetes.io/serviceaccount/

# Inspect the application binary
file /proc/1/root/app/payments-api
```

This works because PID 1 in the shared namespace is the target container's main process, and `/proc/<pid>/root/` is a symlink to that process's root filesystem.

### Network debugging

The debug container shares the pod's network namespace, so all network tests reflect the pod's actual connectivity:

```bash
# DNS resolution
nslookup postgres-primary.database.svc.cluster.local

# HTTP connectivity to another service
curl -v http://order-service.orders.svc.cluster.local:8080/health

# Check what ports the application is listening on
ss -tlnp

# Packet capture (netshoot has tcpdump)
tcpdump -i eth0 -n port 5432
```

## Mode 2: Copy Pod for Debugging

Sometimes you need to change the pod's configuration to debug it -- different image, different command, different environment variables. Copy mode creates a clone of the pod that you can modify freely.

```bash
# Copy the pod, replacing its image with busybox
kubectl debug payments-api-7f8b9c6d4-x2k9m -it \
  --copy-to=debug-payments \
  --set-image=payments-api=busybox \
  --share-processes \
  -n payments-prod
```

What happens:
- A new pod named `debug-payments` is created as a copy of the original
- The `payments-api` container image is replaced with `busybox`
- `--share-processes` enables PID namespace sharing between all containers in the copy
- The original pod is completely untouched

This is useful when:
- You want to run the application with a different entrypoint to test something
- You need to add environment variables or change the command
- The application crashes too fast to attach a debug container

```bash
# Copy the pod but override the command to keep it alive
kubectl debug payments-api-7f8b9c6d4-x2k9m -it \
  --copy-to=debug-payments \
  --container=payments-api \
  -- sh -c "sleep infinity"
```

Now you can exec into `debug-payments` and manually run the application to see what happens:

```bash
kubectl exec -it debug-payments -c payments-api -n payments-prod -- /bin/sh
# Try running the app manually
$ /app/payments-api --config=/etc/config/app.yaml
# See the actual error output in real-time
```

Clean up the debug copy when you are done:

```bash
kubectl delete pod debug-payments -n payments-prod
```

## Mode 3: Debug a Node

Node-level debugging creates a privileged pod on a specific node with the host filesystem mounted.

```bash
kubectl debug node/ip-10-0-1-42.ec2.internal -it --image=busybox
```

What happens:
- A privileged pod is created on the target node
- The host filesystem is mounted at `/host`
- You have root access to the node

```bash
# Get full host access
chroot /host

# Check kubelet status
systemctl status kubelet
journalctl -u kubelet --since "10 minutes ago"

# Check container runtime
crictl ps
crictl logs <container-id>

# Check disk usage (common cause of evictions)
df -h

# Check system memory
free -m

# Check what pods are actually running on this node
crictl pods
```

This is invaluable when kubelet is misbehaving, the container runtime has issues, or you need to inspect host-level networking or storage.

## Choosing a Debug Image

The debug image determines what tools you have available. Pick based on what you need to investigate.

| Image | Size | Best For |
|-------|------|----------|
| `busybox` | ~1 MB | Basic inspection: sh, ls, cat, wget, vi |
| `alpine` | ~7 MB | When you need a package manager: `apk add strace curl` |
| `nicolaka/netshoot` | ~300 MB | Full networking toolkit: tcpdump, nslookup, curl, iperf, netstat, dig |
| `ubuntu` | ~78 MB | When you need apt and a familiar environment |
| Custom | Varies | Build your own with exactly the tools your team uses |

For most debugging sessions, `nicolaka/netshoot` is the best choice. It has every network tool you are likely to need, plus general-purpose utilities. If you are only checking files or processes, `busybox` keeps things fast.

### Building a custom debug image

If your team frequently debugs the same type of applications, build a dedicated debug image:

```dockerfile
FROM nicolaka/netshoot
RUN apk add --no-cache \
    strace \
    gdb \
    postgresql-client \
    redis
COPY custom-scripts/ /usr/local/bin/
```

## Practical Example: Debugging a Distroless Go Service

Scenario: `payments-api` is a distroless Go binary returning HTTP 500 errors. Logs show `connection refused` when reaching the database, but the database pod is running and other services connect to it successfully.

**Step 1: Attach a network debug container.**

```bash
kubectl debug -it payments-api-7f8b9c6d4-x2k9m \
  --image=nicolaka/netshoot \
  --target=payments-api \
  -n payments-prod
```

**Step 2: Test DNS resolution from inside the pod.**

```bash
$ nslookup postgres-primary.database.svc.cluster.local
Server:    10.96.0.10
Address:   10.96.0.10#53

Name:   postgres-primary.database.svc.cluster.local
Address: 10.108.42.15
```

DNS works. The service resolves.

**Step 3: Test TCP connectivity to the database port.**

```bash
$ curl -v telnet://postgres-primary.database.svc.cluster.local:5432
* Trying 10.108.42.15:5432...
* connect to 10.108.42.15 port 5432 failed: Connection refused
```

Connection refused. The service IP resolves but nothing is listening.

**Step 4: Check if the service has endpoints.**

```bash
# From a different terminal
kubectl get endpointslices -n database -l kubernetes.io/service-name=postgres-primary
```

Empty endpoint slices. The service has no backing pods. The database pod is running, but its readiness probe is failing, so it is not added to the endpoint slice. Fix the readiness probe or the underlying database issue, and the payments service will recover.

**Step 5: Check the app's configuration.**

```bash
# Still inside the debug container
cat /proc/1/root/etc/config/app.yaml
```

Confirm the connection string matches the service name you tested.

## Limitations

Ephemeral containers have restrictions you should understand before relying on them:

- **Cannot be removed.** Once added, an ephemeral container stays in the pod spec until the pod terminates. It does not consume resources after it exits, but it shows up in `kubectl describe pod`.
- **Cannot mount new volumes.** The debug container can only access volumes already mounted in the pod. You cannot add a new volume mount.
- **Cannot change resource limits.** The debug container inherits the pod's cgroup but cannot modify resource allocations.
- **RBAC required.** Creating ephemeral containers requires the `pods/ephemeralcontainers` subresource permission. Add this to your debug RBAC role:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-debugger
rules:
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
```

## Security Considerations

Debug containers with `--target` can see the target container's process memory, filesystem, and environment variables. This means a debug session can read database passwords from environment variables, TLS private keys from mounted secrets, and application memory containing sensitive data.

Restrict who can create ephemeral containers. In production, this should be limited to on-call engineers and require audit logging. Some organizations require a break-glass process to get debug access to production namespaces.

Node debugging is even more sensitive -- `chroot /host` gives full root access to the node, including access to every container's filesystem and the kubelet's credentials. Treat `kubectl debug node/` as equivalent to SSH root access.