The Secret Zero Problem#

Every secrets management system has the same fundamental challenge: you need a secret to access your secrets. Your Vault token is itself a secret. Your AWS credentials for SSM Parameter Store are themselves secrets. This is the “secret zero” problem – there is always one secret that must be bootstrapped outside the system.

Understanding this helps you make pragmatic choices. No tool eliminates all risk. The goal is to reduce the blast radius and make rotation possible.

Secrets Management in CI/CD Pipelines#

Every CI/CD pipeline needs credentials: registry tokens, cloud provider keys, database passwords, API keys for third-party services. How you store, deliver, and scope those credentials determines whether a single compromised pipeline job can escalate into a full infrastructure breach. The difference between a mature and an immature pipeline is rarely in the build steps – it is in the secrets management.

The Problem with Static Secrets#

The default approach on every CI platform is storing secrets as encrypted variables: GitHub Actions secrets, GitLab CI variables, Jenkins credentials store. These work but create compounding risks:

Choosing a Secret Management Strategy#

Secrets – database credentials, API keys, TLS certificates, encryption keys – must be available to pods at runtime. At the same time, they must not be stored in plain text in git, should be rotatable without downtime, and should produce an audit trail showing who accessed what and when. No single tool satisfies every requirement, and the right choice depends on your security maturity, operational capacity, and compliance obligations.

HashiCorp Vault on Kubernetes#

Vault centralizes secret management with dynamic credentials, encryption as a service, and fine-grained access control. On Kubernetes, workloads authenticate using service accounts and pull secrets without hardcoding anything.

Installation with Helm#

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Dev Mode (Single Pod, In-Memory)#

Automatically initialized and unsealed, stores everything in memory, loses all data on restart. Root token is root. Never use this in production.

helm upgrade --install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set server.dev.enabled=true \
  --set injector.enabled=true

Production Mode (HA with Integrated Raft Storage)#

Run Vault in HA mode with Raft consensus – a 3-node StatefulSet with persistent storage.

The Problem with Environment Variables#

Environment variables are the most common way to pass secrets to applications. Every framework supports them and they require zero dependencies. They are also the least secure option. Any process running as the same user can read them via /proc/<pid>/environ on Linux. Crash dumps include the full environment. Child processes inherit all variables by default.

# Anyone with host access can read another process's environment
cat /proc/$(pgrep myapp)/environ | tr '\0' '\n' | grep DB_PASSWORD

Environment variables are acceptable for local development. For production secrets, use one of the patterns below.

PKI Fundamentals#

A Public Key Infrastructure (PKI) is a hierarchy of trust. At the top sits the Root CA, a certificate authority that signs its own certificate and is explicitly trusted by all participants. Below it are Intermediate CAs, signed by the root, which handle day-to-day certificate issuance. At the bottom are leaf certificates, the actual certificates used by servers, clients, and workloads.

Root CA (self-signed, offline, 10-20 year validity)
  |
  +-- Intermediate CA (signed by root, online, 3-5 year validity)
        |
        +-- Leaf Certificate (signed by intermediate, 90 days or less)
        +-- Leaf Certificate
        +-- Leaf Certificate

Never use the root CA directly to sign leaf certificates. If the root CA’s private key is compromised, the entire PKI must be rebuilt from scratch. Keeping it offline and behind an intermediate CA limits the blast radius. If an intermediate CA is compromised, you revoke it and issue a new one from the root – leaf certificates from other intermediates remain valid.

Why Rotation Matters#

A credential that never changes is a credential waiting to be exploited. Leaked credentials appear in git history, log files, CI build outputs, developer laptops, and third-party SaaS tools. If a database password has been the same for two years, every person who has ever had access to it still has access – former employees, former contractors, compromised CI systems.

Regular rotation limits the blast radius. A credential that rotates every 24 hours is only useful for 24 hours after compromise. Compliance frameworks (PCI-DSS, SOC2, HIPAA) mandate rotation schedules. But compliance aside, rotation is a pragmatic defense: assume credentials will leak and make the leak time-limited.