Prometheus Architecture#

Prometheus pulls metrics from targets at regular intervals (scraping). Each target exposes an HTTP endpoint (typically /metrics) that returns metrics in a text format. Prometheus stores the scraped data in a local time-series database and evaluates alerting rules against it. Grafana connects to Prometheus as a data source and renders dashboards.

Scrape Configuration#

The core of Prometheus configuration is the scrape config. Each scrape_config block defines a set of targets and how to scrape them.

The Problem with Environment Variables#

Environment variables are the most common way to pass secrets to applications. Every framework supports them and they require zero dependencies. They are also the least secure option. Any process running as the same user can read them via /proc/<pid>/environ on Linux. Crash dumps include the full environment. Child processes inherit all variables by default.

# Anyone with host access can read another process's environment
cat /proc/$(pgrep myapp)/environ | tr '\0' '\n' | grep DB_PASSWORD

Environment variables are acceptable for local development. For production secrets, use one of the patterns below.

Tekton Pipelines#

Tekton is a Kubernetes-native CI/CD framework. Every pipeline concept – tasks, runs, triggers – is a Kubernetes Custom Resource. Pipelines execute as pods. There is no central server, no UI-driven configuration, no special runtime. If you know Kubernetes, you know how to operate Tekton.

Core Concepts#

Tekton has four primary resources:

• Task: A sequence of steps that run in a single pod. Each step is a container.
• TaskRun: An instantiation of a Task with specific inputs. Creating a TaskRun executes the Task.
• Pipeline: An ordered collection of Tasks with dependencies, parameter passing, and conditional execution.
• PipelineRun: An instantiation of a Pipeline. Creating a PipelineRun executes the entire pipeline.

The separation between definition (Task/Pipeline) and execution (TaskRun/PipelineRun) means you define your CI/CD process once and trigger it many times with different inputs.

Choosing a Database Strategy#

Every Kubernetes-based platform eventually faces two questions: should the database run inside the cluster or as a managed service, and which database engine fits the workload? These decisions are difficult to reverse. A database migration is one of the highest-risk operations in production. Getting the initial decision roughly right saves months of future pain.

Where to Run: Kubernetes vs Managed Service#

This is not a technology question. It is an organizational question about who owns database operations and what tradeoffs the team will accept.

Choosing a GitOps Tool#

The term “GitOps” is applied to everything from a simple kubectl apply in a GitHub Actions workflow to a fully reconciled, pull-based deployment architecture with drift detection. These are fundamentally different approaches. Choosing between them depends on your team’s operational maturity, cluster count, and tolerance for running controllers in your cluster.

What GitOps Actually Means#

GitOps, as defined by the OpenGitOps principles (a CNCF sandbox project), has four requirements: declarative desired state, state versioned in git, changes applied automatically, and continuous reconciliation with drift detection. The last two are what separate true GitOps from “CI/CD that uses git.”

Choosing a Kubernetes Backup Strategy#

Kubernetes clusters contain two fundamentally different types of state: cluster state (the Kubernetes objects themselves – Deployments, Services, ConfigMaps, Secrets, CRDs) and application data (the contents of Persistent Volumes). A complete backup strategy must address both. Most backup failures happen because teams back up one but not the other, or because they never test the restore process.

What Needs Backing Up#

Before choosing tools, inventory what your cluster contains:

Database Connection Pooling: PgBouncer, ProxySQL, and Application-Level Patterns#

Database connections are expensive resources. PostgreSQL forks a new OS process for every connection. MySQL creates a thread. Both allocate memory for session state, query buffers, and sort areas. When your application scales horizontally in Kubernetes – 10 pods, then 20, then 50 – the connection count multiplies, and most databases buckle long before your application pods do.

Connection pooling solves this by maintaining a smaller set of persistent connections to the database and sharing them across many application clients. Understanding pooling options, deployment patterns, and sizing is essential for any production database workload on Kubernetes.

How DNS Resolution Works#

When a client requests api.example.com, the resolution follows a chain of queries. The client asks its configured recursive resolver (often the ISP’s, or a public one like 8.8.8.8). The recursive resolver does the heavy lifting: it asks a root name server for .com, the .com TLD server for example.com, and the authoritative name server for example.com returns the answer for api.example.com. Each level caches the result according to the record’s TTL, so subsequent requests short-circuit the chain.

sysctl: Kernel Parameter Tuning#

The sysctl interface exposes kernel parameters that control how Linux manages memory, networking, file systems, and processes. Changes take effect immediately but are lost on reboot unless persisted.

Memory Parameters#

# Reduce swap aggressiveness (default is 60, range 0-100)
# Lower values make the kernel prefer reclaiming page cache over swapping
# Set to 10 for database servers -- swapping destroys database performance
sysctl -w vm.swappiness=10

# Overcommit behavior
# 0 = heuristic overcommit (default, kernel estimates if there is enough memory)
# 1 = always overcommit (never refuse malloc -- dangerous but used by Redis)
# 2 = strict overcommit (never allocate more than swap + ratio*physical)
sysctl -w vm.overcommit_memory=0

The vm.swappiness parameter is one of the most impactful settings for database servers. The default of 60 means the kernel will fairly aggressively swap application memory to disk in favor of filesystem cache. For databases that manage their own caching (PostgreSQL shared_buffers, MySQL innodb_buffer_pool), this is counterproductive – the database’s carefully managed cache gets swapped out to make room for OS-level cache the database does not use.

OAuth2 vs OIDC: What Actually Matters#

OAuth2 is an authorization framework. It answers the question “what is this client allowed to do?” by issuing access tokens. It does not tell you who the user is. OIDC (OpenID Connect) is a layer on top of OAuth2 that adds authentication. It answers “who is this user?” by adding an ID token – a signed JWT containing user identity claims like email, name, and group memberships.