The Secret Zero Problem#

Every secrets management system has the same fundamental challenge: you need a secret to access your secrets. Your Vault token is itself a secret. Your AWS credentials for SSM Parameter Store are themselves secrets. This is the “secret zero” problem – there is always one secret that must be bootstrapped outside the system.

Understanding this helps you make pragmatic choices. No tool eliminates all risk. The goal is to reduce the blast radius and make rotation possible.

ArgoCD Secrets Management#

GitOps says everything should be in Git. Kubernetes Secrets are base64-encoded, not encrypted. Committing base64 secrets to Git is equivalent to committing plaintext – anyone with repo access can decode them. This is the fundamental tension of GitOps secrets management.

Three approaches solve this, each with different tradeoffs.

Approach 1: Sealed Secrets#

Sealed Secrets encrypts secrets client-side so the encrypted form can be safely committed to Git. Only the Sealed Secrets controller running in-cluster can decrypt them.

Choosing a Secret Management Strategy#

Secrets – database credentials, API keys, TLS certificates, encryption keys – must be available to pods at runtime. At the same time, they must not be stored in plain text in git, should be rotatable without downtime, and should produce an audit trail showing who accessed what and when. No single tool satisfies every requirement, and the right choice depends on your security maturity, operational capacity, and compliance obligations.

The Problem with Environment Variables#

Environment variables are the most common way to pass secrets to applications. Every framework supports them and they require zero dependencies. They are also the least secure option. Any process running as the same user can read them via /proc/<pid>/environ on Linux. Crash dumps include the full environment. Child processes inherit all variables by default.

# Anyone with host access can read another process's environment
cat /proc/$(pgrep myapp)/environ | tr '\0' '\n' | grep DB_PASSWORD

Environment variables are acceptable for local development. For production secrets, use one of the patterns below.

Secrets Management in Minikube: From Basic to Production Patterns#

Secrets in Kubernetes are simultaneously simple (just base64-encoded data in etcd) and complex (getting the workflow right for rotation, RBAC, and git-safe storage requires multiple tools). Setting up proper secrets management locally means you can validate the entire workflow – from creation through mounting through rotation – before touching production credentials.

Kubernetes Secret Types#

Kubernetes has several built-in secret types, each with its own structure and validation:

Why Rotation Matters#

A credential that never changes is a credential waiting to be exploited. Leaked credentials appear in git history, log files, CI build outputs, developer laptops, and third-party SaaS tools. If a database password has been the same for two years, every person who has ever had access to it still has access – former employees, former contractors, compromised CI systems.

Regular rotation limits the blast radius. A credential that rotates every 24 hours is only useful for 24 hours after compromise. Compliance frameworks (PCI-DSS, SOC2, HIPAA) mandate rotation schedules. But compliance aside, rotation is a pragmatic defense: assume credentials will leak and make the leak time-limited.