CircleCI Pipeline Patterns#

CircleCI pipelines are defined in .circleci/config.yml. The configuration model uses workflows to orchestrate jobs, jobs to define execution units, and steps to define commands within a job. Every job runs inside an executor – a Docker container, Linux VM, macOS VM, or Windows VM.

Config Structure and Executors#

A minimal config defines a job and a workflow:

version: 2.1

executors:
  go-executor:
    docker:
      - image: cimg/go:1.22
    resource_class: medium
    working_directory: ~/project

jobs:
  build:
    executor: go-executor
    steps:
      - checkout
      - run:
          name: Build application
          command: go build -o myapp ./cmd/myapp

workflows:
  main:
    jobs:
      - build

Named executors let you reuse environment definitions across jobs. The resource_class controls CPU and memory – small (1 vCPU/2GB), medium (2 vCPU/4GB), large (4 vCPU/8GB), xlarge (8 vCPU/16GB). Choose the smallest class that avoids OOM kills to keep costs down.

Buildkite Pipeline Patterns#

Buildkite splits CI/CD into two parts: a hosted web service that manages pipelines, builds, and the UI, and self-hosted agents that execute the actual work. This architecture means your code, secrets, and build artifacts never touch Buildkite’s infrastructure. The agents run on your machines – EC2 instances, Kubernetes pods, bare metal, laptops.

Why Teams Choose Buildkite#

The question usually comes up against Jenkins and GitHub Actions.

Over Jenkins: Buildkite eliminates the Jenkins controller as a single point of failure. There is no plugin compatibility hell, no Groovy DSL, no Java memory tuning. Agents are stateless binaries that poll for work. Scaling is adding more agents. Jenkins requires careful capacity planning of the controller itself.

Building Machine Images with Packer#

Machine images (AMIs, Azure Managed Images, GCP Images) are the foundation of immutable infrastructure. Instead of provisioning a base OS and configuring it at boot, you build a pre-configured image and launch instances from it. Packer automates this process: it launches a temporary instance, runs provisioners to configure it, creates an image from the result, and destroys the temporary instance.

This operational sequence walks through building, testing, and managing machine images with Packer from template creation through CI/CD integration.

Multi-Service Stack Structure#

A typical local development stack has an application, a database, and maybe a cache or message broker. The compose file should read top-to-bottom like a description of your system.

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    env_file:
      - .env
    volumes:
      - ./src:/app/src
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: myapp
      POSTGRES_PASSWORD: localdev
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp"]
      interval: 5s
      timeout: 3s
      retries: 5

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  pgdata:

depends_on and Healthchecks#

The depends_on field controls startup order, but without a condition it only waits for the container to start, not for the service inside to be ready. A Postgres container starts in under a second, but the database process takes several seconds to accept connections. Use condition: service_healthy paired with a healthcheck to block until the dependency is actually ready.

Jenkins Setup and Configuration#

Jenkins is a self-hosted automation server. Unlike managed CI services, you own the infrastructure, which means you control everything from plugin versions to executor capacity. This guide covers the three main installation methods and the configuration patterns that make Jenkins manageable at scale.

Installation with Docker#

The fastest way to run Jenkins locally or in a VM:

docker run -d \
  --name jenkins \
  -p 8080:8080 \
  -p 50000:50000 \
  -v jenkins_home:/var/jenkins_home \
  jenkins/jenkins:lts-jdk17

Port 8080 is the web UI. Port 50000 is the JNLP agent port for inbound agent connections. The volume mount is critical – without it, all configuration and build history is lost when the container restarts.

kind Validation Templates#

kind (Kubernetes IN Docker) runs Kubernetes clusters using Docker containers as nodes. It was designed for testing Kubernetes itself, which makes it an excellent tool for validating infrastructure changes. It starts fast, uses fewer resources than minikube, and is disposable by design.

This article provides copy-paste cluster configurations and complete lifecycle scripts for common validation scenarios.

Cluster Configuration Templates#

Basic Single-Node#

The simplest configuration. One container acts as both control plane and worker. Sufficient for validating that deployments, services, ConfigMaps, and Secrets work correctly.

PostgreSQL Setup and Configuration#

Every PostgreSQL deployment boils down to three things: get the binary running, configure who can connect, and tune the memory settings.

Installation Methods#

Package Managers#

On Debian/Ubuntu, use the official PostgreSQL APT repository:

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install -y postgresql-16

On macOS: brew install postgresql@16 && brew services start postgresql@16

On RHEL/Fedora:

sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo dnf install -y postgresql16-server
sudo /usr/pgsql-16/bin/postgresql-16-setup initdb
sudo systemctl enable --now postgresql-16

Config files live at /etc/postgresql/16/main/ (Debian) or /var/lib/pgsql/16/data/ (RHEL).

Securing Docker-Based Validation Templates#

Validation templates define the environment agents use to test infrastructure changes. If a template runs containers as root, mounts the Docker socket, or skips resource limits, every agent that copies it inherits those risks. This reference covers the security patterns every docker-compose validation template must follow.

1. Non-Root Execution#

Containers run as root by default. A vulnerability in a root-running process gives an attacker full control inside the container and a much larger attack surface for container escapes.

Validation Path Selection#

Not every infrastructure change needs a full Kubernetes cluster to validate. Some changes can be verified with a linter in under a second. Others genuinely need a multi-node cluster with ingress, persistent volumes, and network policies. The cost of choosing wrong is real in both directions: too little validation lets broken configs reach production, while too much wastes minutes or hours on environments you did not need.

Container Build Optimization#

A container build that takes eight minutes in CI is not just slow – it compounds across every push, every developer, every day. The difference between a naive Dockerfile and an optimized one is often the difference between a two-minute build and a twelve-minute build. The techniques here are not theoretical. They are the specific changes that eliminate wasted time.

BuildKit Over Legacy Builder#

BuildKit is the modern Docker build engine and the default since Docker 23.0. If you are running an older version, enable it explicitly with DOCKER_BUILDKIT=1. BuildKit provides several capabilities the legacy builder lacks.