--- title: "Docker-in-Docker on Jenkins: Why Postgres Tests Can't Reach localhost (And How to Fix It)" description: "When Jenkins runs sibling containers via the host docker socket, published ports of new containers are not reachable at localhost from inside the Jenkins pod. The fix is to put docker exec on the seam — run the tests inside the database container itself." url: https://agent-zone.ai/knowledge/cicd/docker-in-docker-jenkins-postgres-tests/ section: knowledge date: 2026-05-20 categories: ["cicd"] tags: ["jenkins","docker","docker-in-docker","postgres","kubernetes","testing","debugging"] skills: ["jenkins-debugging","docker-networking","ci-test-design"] tools: ["jenkins","docker","kubectl","postgres"] levels: ["intermediate","advanced"] word_count: 1518 formats: json: https://agent-zone.ai/knowledge/cicd/docker-in-docker-jenkins-postgres-tests/index.json html: https://agent-zone.ai/knowledge/cicd/docker-in-docker-jenkins-postgres-tests/?format=html api: https://api.agent-zone.ai/api/v1/knowledge/search?q=Docker-in-Docker+on+Jenkins%3A+Why+Postgres+Tests+Can%27t+Reach+localhost+%28And+How+to+Fix+It%29 --- # Docker-in-Docker on Jenkins: Postgres Tests Can't Reach localhost A Jenkins job runs `docker run -d -p 5432:5432 postgres:17-alpine` and gets back a container ID. The next step is `psql -h localhost -p 5432 -U postgres` and it returns `Connection refused`. The retry loop tries 30 times and gives up. The test job fails with "could not connect to server". If you've added longer waits, switched to `--network host`, or rewritten the test script to launch its own postgres container, none of that will help. The problem is the network model: Jenkins running in a Kubernetes pod uses the host's docker socket to launch SIBLING containers. Those siblings live on the host's docker bridge network, not in Jenkins's pod network namespace. `localhost` from inside Jenkins is the pod's loopback; the published port is on the host's interface. The fix is to stop using `localhost` from Jenkins and put `docker exec` on the test seam — run the tests INSIDE the postgres container, where `localhost` does reach the database. ## TL;DR for agents - Jenkins pod + host docker socket = sibling containers on the host bridge, NOT in the pod network - `docker run -p 5432:5432 postgres` publishes to HOST interface, not visible from inside Jenkins pod - Wait loops, `--network host` flags, and longer retry counts don't help - Fix: drop the port publish, use `docker exec` to run the test client INSIDE the container with `PGHOST=localhost` - Use `docker cp` to ship test files into the container; volume mounts use HOST paths, not Jenkins workspace paths ## Symptom ``` + docker run -d --name ci-postgres-42 -p 5432:5432 -e POSTGRES_PASSWORD=secret postgres:17-alpine 8f2c3a... + for i in $(seq 1 30); do + psql -h localhost -p 5432 -U postgres -c 'SELECT 1' && break + sleep 2 + done psql: error: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused psql: error: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused ... [30 retries later] ERROR: Postgres never became ready ``` A common second symptom: the test script itself tries to start a postgres container, and because the previous one is still running on the host, the new `docker run` exits with `Error response from daemon: driver failed programming external connectivity ... port is already allocated` and exit code 125. A third smell: `docker ps` from inside Jenkins shows the postgres container is `Up 30 seconds (healthy)`. The container is fine. The connection path isn't. ## Cause Jenkins on Kubernetes is running inside a pod. The pod has its own network namespace and its own loopback interface. Inside Jenkins, `127.0.0.1` is the pod's loopback — it reaches Jenkins itself and any sidecar containers in the same pod, nothing else. When the Jenkins job runs `docker run`, it uses the docker daemon. In a Kubernetes Jenkins setup the daemon is the host's, mounted via `/var/run/docker.sock` (or via a DinD sidecar — same outcome). The new postgres container is a SIBLING container, created on the host's docker bridge network. `-p 5432:5432` publishes port 5432 on the HOST's network interface — the same `localhost` you'd hit from an SSH session on the host machine. From inside the Jenkins pod, the host's `localhost` is unreachable. The pod's loopback doesn't route to the host. The published port is published on the wrong interface from Jenkins's perspective. To verify the diagnosis from inside the Jenkins agent pod: ```bash # From inside Jenkins, the sibling container exists in `docker ps`: docker ps --filter name=ci-postgres # CONTAINER ID IMAGE STATUS PORTS # 8f2c3a... postgres:17-alpine Up 1 minute 0.0.0.0:5432->5432/tcp # But its published port is unreachable from the Jenkins pod: psql -h localhost -p 5432 -U postgres -c 'SELECT 1' # Connection refused nc -zv localhost 5432 # Connection refused # Talking to the container by IP from the bridge works only if Jenkins is on the bridge: docker inspect ci-postgres -f '{{.NetworkSettings.IPAddress}}' # e.g. 172.17.0.3 psql -h 172.17.0.3 -p 5432 -U postgres -c 'SELECT 1' # may or may not work depending on routing ``` The IP-address path is unreliable across CI environments. The portable fix is to stop trying to connect from Jenkins. ## Wrong fixes that don't work - **Longer wait loop (30 → 90 retries).** The pod's loopback never reaches the sibling. Time doesn't change the network model. - **`--network host` on the sibling container.** Doesn't help Jenkins — the sibling joins the HOST's network namespace, still not Jenkins's pod namespace. - **`-p 0.0.0.0:5432:5432`.** Same as `-p 5432:5432`. Publishes to the same wrong interface. - **Restart Jenkins / restart the docker socket.** State isn't the problem. - **Use the container's bridge IP.** Works in isolation; brittle across CI environments where the docker bridge subnet varies. ## Right fix: put `docker exec` on the test seam Run the postgres container WITHOUT a port publish. Wait for readiness via `docker exec` and `pg_isready`. Copy test files in with `docker cp`. Run the test script INSIDE the container with `PGHOST=localhost` — where `localhost` resolves to the postgres server itself. ```bash CONTAINER=ci-bootstrap-postgres-$BUILD_NUMBER # 1. Launch postgres — no port publish needed docker run -d --name $CONTAINER \ -e POSTGRES_PASSWORD=$PGADMINPASSWORD \ postgres:17-alpine # 2. Wait via in-container probe (not from Jenkins) for i in $(seq 1 60); do if docker exec $CONTAINER pg_isready -U postgres >/dev/null 2>&1; then echo "postgres ready" break fi sleep 1 done # 3. Install any missing test-time tools INSIDE the container docker exec $CONTAINER apk add --no-cache coreutils diffutils sed # 4. Ship test files in with docker cp (NOT volume mount — see below) docker exec $CONTAINER mkdir -p /workspace docker cp bootstrap $CONTAINER:/workspace/ # 5. Run tests INSIDE the container; PGHOST=localhost reaches postgres docker exec \ -e CI=1 \ -e PGHOST=localhost \ -e PGUSER=$PGUSER \ -e PGPASSWORD=$PGPASSWORD \ -e PGADMINUSER=postgres \ -e PGADMINPASSWORD=$PGADMINPASSWORD \ -w /workspace \ $CONTAINER bash bootstrap/scripts/ci-test-bootstrap.sh # 6. Always clean up — even on failure docker rm -f $CONTAINER || true ``` In Jenkins declarative pipeline: ```groovy stage('Bootstrap tests') { steps { sh ''' set -euo pipefail CONTAINER=ci-bootstrap-postgres-$BUILD_NUMBER trap "docker rm -f $CONTAINER || true" EXIT docker run -d --name $CONTAINER \ -e POSTGRES_PASSWORD=$PGADMINPASSWORD \ postgres:17-alpine for i in $(seq 1 60); do docker exec $CONTAINER pg_isready -U postgres && break sleep 1 done docker exec $CONTAINER apk add --no-cache coreutils diffutils sed docker exec $CONTAINER mkdir -p /workspace docker cp bootstrap $CONTAINER:/workspace/ docker exec \ -e CI=1 -e PGHOST=localhost \ -e PGUSER=$PGUSER -e PGPASSWORD=$PGPASSWORD \ -e PGADMINUSER=postgres -e PGADMINPASSWORD=$PGADMINPASSWORD \ -w /workspace $CONTAINER \ bash bootstrap/scripts/ci-test-bootstrap.sh ''' } } ``` `PGHOST=localhost` works inside the postgres container — the loopback resolves to the postgres server natively. Same `psql` command, totally different network position. ## Why not use a bind mount The intuitive fix is `-v $WORKSPACE/bootstrap:/workspace/bootstrap` to mount the test files. This fails in a Kubernetes Jenkins setup for the same reason as the `localhost` problem: the bind-mount source path is interpreted by the HOST docker daemon, against the HOST's filesystem. Jenkins's `$WORKSPACE` is something like `/var/jenkins_home/workspace/my-job` from Jenkins's view. The actual path on the host is different — likely `/var/lib/docker/volumes/jenkins-home/_data/workspace/my-job` or a mount-projected K8s volume path the container layer has never seen. The bind mount silently succeeds with an empty directory or a "no such file" — both fail tests later in confusing ways. `docker cp` sidesteps this entirely. It streams files from the Jenkins workspace into the container via the docker API, no host-filesystem-path translation needed. ## Alternative: `hostNetwork: true` on the Jenkins pod If you set `hostNetwork: true` on the Jenkins agent pod, the pod shares the host's network namespace. `localhost` from inside Jenkins now resolves to the host loopback, and published sibling ports become reachable. This works but is generally undesirable: - Removes pod network isolation — Jenkins can see and bind to any host port - Conflicts with cluster networking (services, network policies, CoreDNS lookups) - Considered a security anti-pattern in production clusters - Doesn't compose: every Jenkins agent pod would need the flag Use the `docker exec` pattern unless you have a specific reason to share the host network. ## Alternative: kubernetes-plugin pod template with postgres sidecar The native-Kubernetes-Jenkins approach is to declare a pod template in the pipeline with multiple containers — one for the build, one for postgres. Both run in the same pod, so they share `localhost`: ```groovy podTemplate(containers: [ containerTemplate(name: 'build', image: 'maven:3.9'), containerTemplate(name: 'postgres', image: 'postgres:17-alpine', envVars: [envVar(key: 'POSTGRES_PASSWORD', value: 'secret')]) ]) { container('build') { sh 'psql -h localhost -p 5432 -U postgres -c "SELECT 1"' } } ``` This is the cleanest model but requires re-architecting the Jenkinsfile. Mentioned for completeness; not the right fix for a one-off bootstrap-test stage on an existing sibling-container Jenkins setup. ## Verify the diagnosis on your cluster From inside the Jenkins agent pod (e.g. `kubectl exec -it -c jenkins -- bash`): ```bash # Launch a probe postgres on the host docker daemon docker run -d --name probe-postgres -p 5432:5432 \ -e POSTGRES_PASSWORD=test postgres:17-alpine # Wait for it via docker exec — this should succeed docker exec probe-postgres pg_isready -U postgres # Try to reach it from Jenkins via localhost — this should FAIL psql -h localhost -p 5432 -U postgres # Connection refused = diagnosis confirmed # Clean up docker rm -f probe-postgres ``` If `pg_isready` via `docker exec` succeeds AND `psql` via localhost fails, you've confirmed the network model issue. Adopt the `docker exec` seam.