Choosing a Log Aggregation Stack#

Logs are the most fundamental observability signal. Every application produces them, every incident investigation starts with them, and every compliance framework requires retaining them. The challenge is not collecting logs – it is storing, indexing, querying, and retaining them at scale without spending a fortune.

The choice of log aggregation stack determines your query speed, operational burden, storage costs, and how effectively you can correlate logs with metrics and traces during incident response.

Kubernetes Audit Logging#

Kubernetes audit logging records every request to the API server: who made the request, what they asked for, and what happened. Without audit logging, you have no visibility into who accessed secrets, who changed RBAC roles, or who exec’d into a production pod. It is the foundation of security monitoring in Kubernetes.

Audit Policy#

The audit policy defines which events to record and at what detail level. There are four levels:

The Decision Landscape#

Log management is deceptively simple on the surface – applications write text, you store it, you search it later. In practice, every decision in the log pipeline involves tradeoffs between cost, query speed, retention depth, operational complexity, and correlation with other observability signals. This guide provides a framework for making those decisions based on your actual requirements rather than defaults or trends.

Structured Logging: The Foundation#

Before choosing any aggregation tool, standardize on structured logging. Unstructured logs are human-readable but machine-hostile. Structured logs are both.

How Kubernetes Captures Logs#

Containers write to stdout and stderr. The container runtime (containerd, CRI-O) captures these streams and writes them to files on the node. The kubelet manages these files at /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/ with symlinks from /var/log/containers/.

The format depends on the runtime. Containerd writes logs in a format with timestamp, stream tag, and the log line:

2026-02-22T10:15:32.123456789Z stdout F {"level":"info","msg":"request handled","status":200}
2026-02-22T10:15:32.456789012Z stderr F error: connection refused to database

kubectl logs reads these files. It only works while the pod exists – once a pod is deleted, its log files are eventually cleaned up. This is why centralized log collection is essential.

SIEM and Security Log Correlation#

A SIEM collects logs from across your infrastructure, normalizes them, and applies correlation rules to detect threats that no single log source would reveal. A brute force attempt is visible in auth logs. Lateral movement after successful brute force requires correlating auth logs with network flow data and process execution logs. The SIEM makes that correlation possible.

Log Sources#

The value of a SIEM depends entirely on the logs you feed it. Missing a log source means missing the attacks that source would reveal.

Elasticsearch and OpenSearch: Indexing, Queries, Cluster Management, and Performance#

Elasticsearch and OpenSearch are distributed search and analytics engines built on Apache Lucene. They excel at full-text search, log aggregation, metrics storage, and any workload that benefits from inverted indices. Understanding index design, mappings, query mechanics, and cluster management separates a working setup from one that collapses under production load.

Elasticsearch vs OpenSearch#

OpenSearch is the AWS-maintained fork of Elasticsearch, created after Elastic changed its license from Apache 2.0 to the Server Side Public License (SSPL) in early 2021. For the vast majority of use cases, the two are interchangeable. APIs are compatible, concepts are identical, and most tooling works with both. OpenSearch Dashboards replaces Kibana. This guide applies to both unless explicitly noted.