The Secret Zero Problem#

Every secrets management system has the same fundamental challenge: you need a secret to access your secrets. Your Vault token is itself a secret. Your AWS credentials for SSM Parameter Store are themselves secrets. This is the “secret zero” problem – there is always one secret that must be bootstrapped outside the system.

Understanding this helps you make pragmatic choices. No tool eliminates all risk. The goal is to reduce the blast radius and make rotation possible.

Agent Sandboxing#

An AI agent that can execute code, run shell commands, or call APIs needs a sandbox. Without one, a single bad tool call – whether from a bug, a hallucination, or a prompt injection attack – can read secrets, modify production data, or pivot to other systems. This article is a decision framework for choosing the right sandboxing strategy based on your trust level, threat model, and performance requirements.

Choosing a Kubernetes Policy Engine#

Kubernetes does not enforce security best practices by default. A freshly deployed cluster allows containers to run as root, pull images from any registry, mount the host filesystem, and use the host network. Policy engines close this gap by intercepting API requests through admission webhooks and rejecting or modifying resources that violate your rules.

The three main options – Pod Security Admission (built-in), OPA Gatekeeper, and Kyverno – serve different needs. Choosing the wrong one leads to either insufficient enforcement or unnecessary operational burden.

Choosing a Secret Management Strategy#

Secrets – database credentials, API keys, TLS certificates, encryption keys – must be available to pods at runtime. At the same time, they must not be stored in plain text in git, should be rotatable without downtime, and should produce an audit trail showing who accessed what and when. No single tool satisfies every requirement, and the right choice depends on your security maturity, operational capacity, and compliance obligations.

Threat Modeling for Developers#

Threat modeling is the practice of systematically identifying what can go wrong in a system before it goes wrong. It is not a security team activity that happens once. It is a design activity that happens every time the architecture changes.

The output of threat modeling is not a report that sits in a wiki. It is a prioritized list of threats that becomes security requirements in the backlog.