--- title: "Kubernetes Audit Logging: Tracking API Activity for Security and Compliance" description: "Configuring and operating Kubernetes audit logging including audit policies, backends, managed service integration, security event detection, and compliance requirements." url: https://agent-zone.ai/knowledge/kubernetes/audit-logging-and-compliance/ section: knowledge date: 2026-02-22 categories: ["kubernetes"] tags: ["audit-logging","security","compliance","soc2","cis-benchmark","falco","siem","api-server"] skills: ["audit-policy-design","audit-log-analysis","security-event-detection","compliance-implementation","siem-integration"] tools: ["kubectl","jq","falco"] levels: ["advanced"] word_count: 1589 formats: json: https://agent-zone.ai/knowledge/kubernetes/audit-logging-and-compliance/index.json html: https://agent-zone.ai/knowledge/kubernetes/audit-logging-and-compliance/?format=html api: https://api.agent-zone.ai/api/v1/knowledge/search?q=Kubernetes+Audit+Logging%3A+Tracking+API+Activity+for+Security+and+Compliance --- # Kubernetes Audit Logging: Tracking API Activity for Security and Compliance Audit logging records every request to the Kubernetes API server. Every `kubectl` command, every controller reconciliation, every kubelet heartbeat, every admission webhook call -- all of it can be captured with the requester's identity, the target resource, the timestamp, and optionally the full request and response bodies. Without audit logging, you have no record of who did what in your cluster. With it, you can trace security incidents, satisfy compliance requirements, and debug access control issues. ## Audit Stages Each API request passes through four stages. Audit events can be generated at any stage: | Stage | When | |-------|------| | `RequestReceived` | Immediately after the request is received, before any processing | | `ResponseStarted` | After response headers are sent but before the response body (long-running requests like watch) | | `ResponseComplete` | After the response body is complete | | `Panic` | When a panic occurs during request handling | Most audit configurations generate events at `ResponseComplete` since this captures the outcome (response code) along with the request details. ## Audit Levels The audit level controls how much detail is captured for each event: | Level | Captures | |-------|----------| | `None` | Nothing (skip this request) | | `Metadata` | User, timestamp, resource, verb, response code -- no body content | | `Request` | Metadata plus the full request body | | `RequestResponse` | Metadata plus request body plus response body | The trade-off is clear: `RequestResponse` gives you complete forensic detail but generates enormous log volume. `Metadata` is compact and sufficient for most operational and compliance needs. `Request` captures what was submitted (useful for seeing exactly what RBAC change was made) without the response bloat. ## Audit Policy The audit policy is a YAML file that defines which requests are logged at which level. Rules are evaluated in order -- the first match wins. Requests that match no rule are logged at the `Metadata` level by default (or not at all if there is a catch-all `None` rule). ```yaml apiVersion: audit.k8s.io/v1 kind: Policy # Do not log requests to these endpoints (high volume, low value) omitStages: - "RequestReceived" rules: # Do not log health checks, readiness probes, or API discovery - level: None nonResourceURLs: - /healthz* - /livez* - /readyz* - /metrics - /openapi/v2* # Do not log events from the system:nodes group (kubelet heartbeats) - level: None users: ["system:apiserver"] resources: - group: "" resources: ["endpoints", "services", "services/status"] # Log Secret access at Metadata level only (do NOT log Secret bodies) - level: Metadata resources: - group: "" resources: ["secrets"] # Log RBAC changes with full request body - level: Request resources: - group: "rbac.authorization.k8s.io" resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"] # Log pod exec and attach with full request - level: Request resources: - group: "" resources: ["pods/exec", "pods/attach", "pods/portforward"] # Log token creation - level: Request resources: - group: "" resources: ["serviceaccounts/token"] # Log all other write operations at Metadata level - level: Metadata verbs: ["create", "update", "patch", "delete", "deletecollection"] # Log everything else at Metadata level - level: Metadata ``` The policy above represents a practical starting point: verbose logging for security-sensitive operations (RBAC, exec, tokens), metadata-only for routine operations, and silence for high-volume noise (health checks, system heartbeats). Never log Secret bodies -- `Request` or `RequestResponse` level on Secrets means Secret values appear in your audit logs, creating a security hole in your audit infrastructure. ## Audit Backends ### Log Backend Writes audit events as JSON lines to a file on disk. The API server handles file rotation via built-in flags. Configure on the API server: ``` --audit-policy-file=/etc/kubernetes/audit/policy.yaml --audit-log-path=/var/log/kubernetes/audit/audit.log --audit-log-maxage=30 # days to retain old log files --audit-log-maxbackup=10 # number of old log files to keep --audit-log-maxsize=100 # max size in MB before rotation --audit-log-format=json # json or legacy ``` For kubeadm clusters, add these to the API server static pod manifest: ```yaml # /etc/kubernetes/manifests/kube-apiserver.yaml (relevant excerpts) spec: containers: - command: - kube-apiserver - --audit-policy-file=/etc/kubernetes/audit/policy.yaml - --audit-log-path=/var/log/kubernetes/audit/audit.log - --audit-log-maxage=30 - --audit-log-maxbackup=10 - --audit-log-maxsize=100 volumeMounts: - mountPath: /etc/kubernetes/audit name: audit-policy readOnly: true - mountPath: /var/log/kubernetes/audit name: audit-log volumes: - hostPath: path: /etc/kubernetes/audit type: DirectoryOrCreate name: audit-policy - hostPath: path: /var/log/kubernetes/audit type: DirectoryOrCreate name: audit-log ``` ### Webhook Backend Sends audit events to an external HTTP endpoint in real-time. This integrates with SIEM systems (Splunk, Elasticsearch, Datadog) or dedicated audit analysis tools. ```yaml # /etc/kubernetes/audit/webhook-config.yaml apiVersion: v1 kind: Config clusters: - name: audit-webhook cluster: server: https://audit-collector.monitoring.svc:8443/audit certificate-authority: /etc/kubernetes/audit/ca.crt contexts: - context: cluster: audit-webhook name: default current-context: default ``` ``` --audit-webhook-config-file=/etc/kubernetes/audit/webhook-config.yaml --audit-webhook-batch-max-wait=5s --audit-webhook-batch-max-size=100 ``` The webhook backend batches events for efficiency. Tune `batch-max-wait` and `batch-max-size` based on your latency and throughput requirements. If the webhook endpoint is unreachable, events are buffered in memory and eventually dropped -- monitor the `apiserver_audit_event_total` and `apiserver_audit_error_total` metrics. ## Managed Kubernetes Audit Logging Managed services handle audit logging differently since you do not have access to the API server flags. **Amazon EKS.** Enable audit logging through the cluster logging configuration. Audit logs go to CloudWatch Logs under the `/aws/eks//cluster` log group. ```bash aws eks update-cluster-config --name my-cluster \ --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}' ``` **Azure AKS.** Configure diagnostic settings to send the `kube-audit` log category to a Log Analytics workspace, storage account, or Event Hub. ```bash az monitor diagnostic-settings create \ --name audit-logs \ --resource $(az aks show -g my-rg -n my-cluster --query id -o tsv) \ --workspace $(az monitor log-analytics workspace show -g my-rg -n my-workspace --query id -o tsv) \ --logs '[{"category":"kube-audit","enabled":true}]' ``` **Google GKE.** Admin Activity audit logs are enabled by default at no charge and cannot be disabled. Data Access audit logs (which include read operations) must be enabled separately and incur logging charges. ## Analyzing Audit Logs ### Key Security Events to Monitor ```bash # Find unauthorized access attempts (403 responses) jq 'select(.responseStatus.code == 403)' audit.log | \ jq '{user: .user.username, verb: .verb, resource: .objectRef.resource, ns: .objectRef.namespace}' # Track who accessed Secrets jq 'select(.objectRef.resource == "secrets" and .verb == "get")' audit.log | \ jq '{user: .user.username, secret: .objectRef.name, ns: .objectRef.namespace, time: .requestReceivedTimestamp}' # Find RBAC modifications jq 'select(.objectRef.apiGroup == "rbac.authorization.k8s.io" and (.verb == "create" or .verb == "update" or .verb == "patch" or .verb == "delete"))' audit.log | \ jq '{user: .user.username, verb: .verb, kind: .objectRef.resource, name: .objectRef.name}' # Track exec into pods (potential lateral movement) jq 'select(.objectRef.subresource == "exec")' audit.log | \ jq '{user: .user.username, pod: .objectRef.name, ns: .objectRef.namespace, time: .requestReceivedTimestamp}' # Find privilege escalation attempts (creating pods with hostPath or privileged) jq 'select(.objectRef.resource == "pods" and .verb == "create" and .responseStatus.code == 201)' audit.log | \ jq 'select(.requestObject.spec.containers[].securityContext.privileged == true) | {user: .user.username, pod: .objectRef.name, ns: .objectRef.namespace}' ``` ### Audit Event Structure A single audit event in JSON format: ```json { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/production/secrets/db-credentials", "verb": "get", "user": { "username": "alice@example.com", "groups": ["dev-team", "system:authenticated"] }, "sourceIPs": ["10.0.1.50"], "objectRef": { "resource": "secrets", "namespace": "production", "name": "db-credentials", "apiVersion": "v1" }, "responseStatus": { "code": 200 }, "requestReceivedTimestamp": "2026-02-22T10:30:45.123456Z", "stageTimestamp": "2026-02-22T10:30:45.125678Z" } ``` ## Compliance Frameworks Audit logging is not optional in regulated environments: **CIS Kubernetes Benchmark.** Section 3.2 requires audit logging to be enabled with an appropriate policy. The benchmark recommends logging at `Metadata` level minimum for all requests, with `Request` level for sensitive resources. **SOC 2 (Trust Services Criteria).** CC6.1 requires logging of user activities and CC7.2 requires monitoring for anomalies. Kubernetes audit logs satisfy these when combined with alerting on suspicious patterns. **PCI-DSS.** Requirement 10 mandates tracking access to cardholder data environments. If your Kubernetes cluster processes card data, you must log and monitor all API access, especially to namespaces containing payment workloads. **HIPAA.** Requires audit controls for information systems containing protected health information. Access to PHI-related Kubernetes resources must be logged and reviewable. ## Falco Integration Falco complements audit logging by monitoring at the syscall level inside containers. While audit logs tell you who called `kubectl exec`, Falco tells you what commands were run inside the container after exec. ```yaml # Falco can consume Kubernetes audit events directly # In falco.yaml: webserver: enabled: true listen_port: 8765 # Configure the API server to send audit events to Falco # --audit-webhook-config-file pointing to Falco's endpoint ``` Falco rules can trigger alerts on both audit events and runtime behavior: - A user execs into a production pod (audit event) - A process inside a container reads `/etc/shadow` (syscall event) - A container spawns an unexpected shell process (syscall event) ## Storage Considerations Audit logs grow rapidly. A moderately active cluster with 50 nodes and common controller activity can generate 1-5 GB of audit logs per day at `Metadata` level. At `RequestResponse` level, expect 10-50x more. Strategies for managing volume: 1. **Filter aggressively.** Use `None` level for high-frequency, low-value events (health checks, leader election, node heartbeats). 2. **Use Metadata level as the default.** Reserve `Request` and `RequestResponse` for security-critical resources only. 3. **Set retention policies.** Keep detailed logs for 30-90 days, archive summarized data longer for compliance. 4. **Stream to external storage.** Use the webhook backend to send events to a scalable log store (Elasticsearch, S3, cloud-native logging) rather than relying on local disk. ## Common Gotchas **RequestResponse for everything causes API server performance degradation.** Serializing every response body (especially large list operations) adds latency and memory pressure to the API server. A `kubectl get pods -A` response body can be several megabytes. Multiply this by every controller's periodic list operation, and the API server spends significant resources just generating audit data. Start with `Metadata` and selectively increase. **No audit policy means no audit logs.** Many clusters, especially development and self-managed installations, ship with no audit policy configured. The API server does not generate audit events unless `--audit-policy-file` is explicitly set. Check your API server flags -- if audit is not configured, you have zero visibility into API activity. **Audit logs containing Secrets.** If you set `Request` or `RequestResponse` level for Secrets, the actual Secret values (base64-encoded, trivially decoded) appear in your audit logs. Anyone with access to the audit log storage can read every Secret in the cluster. Always use `Metadata` level for Secrets.