Docker-in-Docker on Jenkins: Postgres Tests Can’t Reach localhost#

A Jenkins job runs docker run -d -p 5432:5432 postgres:17-alpine and gets back a container ID. The next step is psql -h localhost -p 5432 -U postgres and it returns Connection refused. The retry loop tries 30 times and gives up. The test job fails with “could not connect to server”.

If you’ve added longer waits, switched to --network host, or rewritten the test script to launch its own postgres container, none of that will help. The problem is the network model: Jenkins running in a Kubernetes pod uses the host’s docker socket to launch SIBLING containers. Those siblings live on the host’s docker bridge network, not in Jenkins’s pod network namespace. localhost from inside Jenkins is the pod’s loopback; the published port is on the host’s interface.

Jenkins Multibranch Silent Skip After Branch Recreate#

Push a branch named fix/foo. Trigger a multibranch scan. The scan log shows Checking branch fix/foo and immediately moves to the next branch with no verdict line. No job appears under the multibranch. No build fires. Other branches scan and build normally.

This is Jenkins’s branch source plugin silently skipping a branch because its internal cache treats the name as a duplicate of a previously-deleted entry. The cache survives plugin restarts, multibranch rescans, and kubectl rollout restart jenkins. The reliable recovery is to push the same commits under a different branch name — the cache has no entry for the new name and processes it cleanly.

Pipeline Observability#

You cannot improve what you do not measure. Most teams have detailed monitoring for their production applications but treat their CI/CD pipelines as black boxes. When builds are slow, flaky, or failing, the response is anecdotal – “builds feel slow lately” – rather than data-driven. Pipeline observability turns CI/CD from a cost center you tolerate into infrastructure you actively manage.

Core CI/CD Metrics#

Build Duration#

Total time from pipeline trigger to completion. Track this as a histogram, not an average, because averages hide bimodal distributions. A pipeline that takes 5 minutes for code-only changes and 25 minutes for dependency updates averages 15 minutes, which describes neither case accurately.

CI/CD Anti-Patterns and Migration Strategies#

CI/CD pipelines accumulate technical debt faster than application code. Nobody refactors a Jenkinsfile. Nobody reviews pipeline YAML with the same rigor as production code. Over time, pipelines become slow, fragile, inconsistent, and actively hostile to developer productivity. Recognizing the anti-patterns is the first step. Migrating to better tooling is often the second.

Anti-Pattern: Snowflake Pipelines#

Every repository has a unique pipeline that someone wrote three years ago and nobody fully understands. Repository A uses Makefile targets, B uses bash scripts, C calls Python, and D has inline shell commands across 40 pipeline steps. There is no shared structure, no reusable components, and no way to make organization-wide changes.

An Autonomous PR-to-Deploy Loop#

The goal: a contributor (human or agent) opens a PR; if it passes CI and gets the required approvals, it merges and deploys itself with no human clicking buttons. The loop:

PR → CI gate (required status) → N approvals → auto-merge → auto-tag → build image:<tag> → deploy (pin tag)

This is buildable on plain Jenkins/Gitea/Kubernetes (or GitHub/Actions/Argo equivalents). The pieces are independent; wire them in order.

Jenkins Multibranch with Gitea: Why Pull Request Builds Never Run#

A common, maddening symptom: your Jenkins organization folder (or multibranch pipeline) backed by Gitea builds the default branch fine, but pull request commits never build — the commit status stays pending forever (or never appears), so a branch-protection gate that requires a CI status can never be satisfied and the PR can never merge.

The pipeline is fine. The problem is branch/PR discovery configuration, and there are several layered traps. Here is how to diagnose and fix each.

A backlog item flips to status='completed' in the database. The dashboard ticks up. The agent posts “PR ready for review” and walks away. Three hours later, a different agent notices the fleet is running yesterday’s binary. The PR was never reviewed. CI was red on main. No image got built. Nothing actually shipped.

This is the closed-loop problem. When an autonomous agent declares work complete, what does “complete” mean? In most agent fleets, it means the agent called the last tool in its own workflow — typically open_pr or its equivalent. That is not the same as “the change is live for users”, and the gap between the two is where state-of-record systematically lies.

Jenkins Debugging#

Jenkins failures fall into a few categories: builds stuck waiting, cryptic pipeline errors, performance degradation, and Kubernetes agent pods that refuse to launch.

Builds Stuck in Queue#

When a build sits in the queue and never starts, check the queue tooltip in the UI – it tells you why. Common causes:

No agents with matching labels. The pipeline requests agent { label 'docker-arm64' } but no agent has that label. Check Manage Jenkins > Nodes to see available labels.

Jenkins Kubernetes Integration#

The kubernetes plugin gives Jenkins elastic build capacity. Each build spins up a pod, runs its work, and the pod is deleted. No idle agents, no capacity planning, no snowflake build servers.

The Kubernetes Plugin#

The plugin creates agent pods on demand. When a pipeline requests an agent, a pod is created from a template, its JNLP container connects back to Jenkins, the build runs, and the pod is deleted.

Jenkins Pipeline Patterns#

Jenkins pipelines define your build, test, and deploy process as code in a Jenkinsfile stored alongside your application source. This eliminates configuration drift and makes CI/CD reproducible across branches.

Declarative vs Scripted#

Declarative is the standard choice. It has a fixed structure, better error reporting, and supports the Blue Ocean visual editor. Scripted is raw Groovy – more flexible, but harder to read and maintain. Use declarative unless you need control flow that declarative cannot express.