Docker-in-Docker on Jenkins: Postgres Tests Can’t Reach localhost#

A Jenkins job runs docker run -d -p 5432:5432 postgres:17-alpine and gets back a container ID. The next step is psql -h localhost -p 5432 -U postgres and it returns Connection refused. The retry loop tries 30 times and gives up. The test job fails with “could not connect to server”.

If you’ve added longer waits, switched to --network host, or rewritten the test script to launch its own postgres container, none of that will help. The problem is the network model: Jenkins running in a Kubernetes pod uses the host’s docker socket to launch SIBLING containers. Those siblings live on the host’s docker bridge network, not in Jenkins’s pod network namespace. localhost from inside Jenkins is the pod’s loopback; the published port is on the host’s interface.

CircleCI Pipeline Patterns#

CircleCI pipelines are defined in .circleci/config.yml. The configuration model uses workflows to orchestrate jobs, jobs to define execution units, and steps to define commands within a job. Every job runs inside an executor – a Docker container, Linux VM, macOS VM, or Windows VM.

Config Structure and Executors#

A minimal config defines a job and a workflow:

version: 2.1

executors:
  go-executor:
    docker:
      - image: cimg/go:1.22
    resource_class: medium
    working_directory: ~/project

jobs:
  build:
    executor: go-executor
    steps:
      - checkout
      - run:
          name: Build application
          command: go build -o myapp ./cmd/myapp

workflows:
  main:
    jobs:
      - build

Named executors let you reuse environment definitions across jobs. The resource_class controls CPU and memory – small (1 vCPU/2GB), medium (2 vCPU/4GB), large (4 vCPU/8GB), xlarge (8 vCPU/16GB). Choose the smallest class that avoids OOM kills to keep costs down.

Jenkins Multibranch Silent Skip After Branch Recreate#

Push a branch named fix/foo. Trigger a multibranch scan. The scan log shows Checking branch fix/foo and immediately moves to the next branch with no verdict line. No job appears under the multibranch. No build fires. Other branches scan and build normally.

This is Jenkins’s branch source plugin silently skipping a branch because its internal cache treats the name as a duplicate of a previously-deleted entry. The cache survives plugin restarts, multibranch rescans, and kubectl rollout restart jenkins. The reliable recovery is to push the same commits under a different branch name — the cache has no entry for the new name and processes it cleanly.

Buildkite Pipeline Patterns#

Buildkite splits CI/CD into two parts: a hosted web service that manages pipelines, builds, and the UI, and self-hosted agents that execute the actual work. This architecture means your code, secrets, and build artifacts never touch Buildkite’s infrastructure. The agents run on your machines – EC2 instances, Kubernetes pods, bare metal, laptops.

Why Teams Choose Buildkite#

The question usually comes up against Jenkins and GitHub Actions.

Over Jenkins: Buildkite eliminates the Jenkins controller as a single point of failure. There is no plugin compatibility hell, no Groovy DSL, no Java memory tuning. Agents are stateless binaries that poll for work. Scaling is adding more agents. Jenkins requires careful capacity planning of the controller itself.

Azure DevOps Pipelines#

Azure DevOps Pipelines uses YAML files stored in your repository to define build and deployment workflows. The pipeline model has three levels: stages contain jobs, jobs contain steps. This hierarchy maps directly to how you think about CI/CD – build stage, test stage, deploy-to-staging stage, deploy-to-production stage – with each stage containing one or more parallel jobs.

Pipeline Structure#

A complete pipeline in azure-pipelines.yml:

trigger:
  branches:
    include:
      - main
      - release/*
  paths:
    exclude:
      - docs/**
      - README.md

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: common-vars
  - name: buildConfiguration
    value: 'Release'

stages:
  - stage: Build
    jobs:
      - job: BuildApp
        steps:
          - task: GoTool@0
            inputs:
              version: '1.22'
          - script: |
              go build -o $(Build.ArtifactStagingDirectory)/myapp ./cmd/myapp
            displayName: 'Build binary'
          - publish: $(Build.ArtifactStagingDirectory)
            artifact: drop

  - stage: Test
    dependsOn: Build
    jobs:
      - job: UnitTests
        steps:
          - task: GoTool@0
            inputs:
              version: '1.22'
          - script: go test ./... -v -coverprofile=coverage.out
            displayName: 'Run tests'
          - task: PublishCodeCoverageResults@2
            inputs:
              summaryFileLocation: coverage.out
              codecoverageTool: 'Cobertura'

  - stage: DeployStaging
    dependsOn: Test
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployToStaging
        environment: staging
        strategy:
          runOnce:
            deploy:
              steps:
                - download: current
                  artifact: drop
                - script: echo "Deploying to staging"

trigger controls which branches and paths trigger the pipeline. dependsOn creates stage ordering. condition adds logic – succeeded() checks the previous stage passed, and you can combine it with variable checks to restrict certain stages to specific branches.

AWS CodePipeline and CodeBuild#

AWS CodePipeline orchestrates CI/CD workflows as a series of stages. CodeBuild executes the actual build and test commands. Together they provide a fully managed pipeline that integrates natively with S3, ECR, ECS, EKS, Lambda, and CloudFormation. No servers to manage, no agents to maintain – but the trade-off is less flexibility than self-hosted systems and tighter coupling to the AWS ecosystem.

Pipeline Structure#

A CodePipeline has stages, and each stage has actions. Actions can run in parallel or sequentially within a stage. The most common pattern is Source -> Build -> Deploy:

Pipeline Security Hardening with SLSA#

Software supply chain attacks exploit the gap between source code and deployed artifact. The SLSA framework (Supply-chain Levels for Software Artifacts) defines concrete requirements for closing that gap. It is not a tool you install – it is a set of verifiable properties your build process must satisfy.

SLSA Levels#

SLSA defines four levels of increasing assurance:

Level 0: No guarantees. Most pipelines start here.

Secrets Management in CI/CD Pipelines#

Every CI/CD pipeline needs credentials: registry tokens, cloud provider keys, database passwords, API keys for third-party services. How you store, deliver, and scope those credentials determines whether a single compromised pipeline job can escalate into a full infrastructure breach. The difference between a mature and an immature pipeline is rarely in the build steps – it is in the secrets management.

The Problem with Static Secrets#

The default approach on every CI platform is storing secrets as encrypted variables: GitHub Actions secrets, GitLab CI variables, Jenkins credentials store. These work but create compounding risks:

CI/CD Cost Optimization#

CI/CD costs grow quietly. A team of ten pushing five times a day, running a 15-minute pipeline on 4-core runners, burns through 2,500 build minutes per week. On GitHub Actions at $0.008/minute for Linux runners, that is $20/week. Scale to fifty developers with integration tests, matrix builds, and nightly jobs, and you are looking at $500-$2,000/month before anyone notices.

The fix is not running fewer tests or skipping builds. It is eliminating waste: jobs that use more compute than they need, caches that are never restored, full builds triggered by README changes, and runners sitting idle between jobs.

Database Schema Migrations in CI/CD#

Schema migrations are the riskiest step in most deployment pipelines. Application code can be rolled back in seconds by deploying the previous container image. A database migration that drops a column, changes a data type, or restructures a table cannot be undone by pressing a button. Yet many teams run migrations manually, or tack them onto deployment scripts without testing, rollback plans, or zero-downtime considerations.