Why Monitor a POC Cluster#

Monitoring on minikube serves two purposes. First, it catches resource problems early – your app might work in tests but OOM-kill under load, and you will not know without metrics. Second, it validates that your monitoring configuration works before you deploy it to production. If your ServiceMonitors, dashboards, and alert rules work on minikube, they will work on EKS or GKE.

The Right Chart: kube-prometheus-stack#

There are multiple Prometheus-related Helm charts. Use the right one:

Why GitOps on a POC Cluster#

Setting up ArgoCD on minikube is not about automating deployments for a local cluster – you could just run kubectl apply. The point is to prove the deployment workflow before production. If your Git repo structure, Helm values, and sync policies work on minikube, they will work on EKS or GKE. If you skip this and bolt on GitOps later, you will spend days restructuring your repo and debugging sync failures under production pressure.

Choosing the Right Workload Type#

Every application fits one of four deployment patterns. Choosing the wrong one creates problems that are hard to fix later – a database deployed as a Deployment loses data on reschedule, a batch job deployed as a Deployment wastes resources running 24/7.

Pattern 1: Stateless Web App#

A web API that can be scaled horizontally with no persistent state. Any pod can handle any request.

The Secret Zero Problem#

Every secrets management system has the same fundamental challenge: you need a secret to access your secrets. Your Vault token is itself a secret. Your AWS credentials for SSM Parameter Store are themselves secrets. This is the “secret zero” problem – there is always one secret that must be bootstrapped outside the system.

Understanding this helps you make pragmatic choices. No tool eliminates all risk. The goal is to reduce the blast radius and make rotation possible.

What Toil Actually Is#

Toil is work tied to running a production service that is manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with service growth. Not all operational work is toil. Capacity planning requires judgment. Postmortem analysis produces lasting improvements. Writing automation code is engineering. Toil is the opposite: it is the work that a machine could do but currently a human is doing, over and over, without making the system any better.

Builder Pool Naming: The (role, tier, replica) Coordinate#

Naming agent pools after the model they run today (kimi-N, deepseek-N, flash-N, lite-N) felt natural when each pool ran one model. It stopped feeling natural the third time a pool’s model churned — when the lite-tier swapped through qwen → gemma → gemini in six weeks and every rename cascaded through K8s manifests, secret names, MM bot accounts, Gitea identities, and helm values. The fix was to make pool names model-independent: builder-lite-0 runs whatever model the pool config says it runs today.

Claude Code /loop Daemon Hygiene#

A claude /loop 5m /role-daemon daemon is the easiest way to run an autonomous agent on a Max subscription: tmux session, one command, comes back every five minutes forever. It works perfectly for the first hour. By hour six it has accumulated 50,000+ tokens of stale “in cycle 47 I posted to MM” history that ships to Anthropic on every prompt. By day two it has three overlapping cron entries firing the same daemon every two minutes instead of every five. By day three it has auto-compact-exited and the tmux session is bare.

Cloudflare GraphQL Analytics: A Field-Discovery Cookbook When Introspection Is Locked#

Cloudflare’s GraphQL Analytics API at https://api.cloudflare.com/client/v4/graphql is the richest source of metrics about your CF account — Workers invocations, D1 reads/writes, KV ops, Workers AI neurons, Vectorize queries. The dashboard’s charts are powered by it. The CLI is not: wrangler exposes a fraction of what GraphQL does.

But the schema is hostile to discovery:

• __type(name: "WorkersInvocationsAdaptive") returns null for almost every node.
• The official schema docs at developers.cloudflare.com/analytics/graphql-api are partial and stale by months.
• Nodes like vectorizeQueriesAdaptiveGroups exist, but their sum/dimensions field names are nowhere on the public internet.

You can still derive the schema. The trick is deliberate-error probing: send a query with a guessed field name; the error message tells you whether the parent node exists. This page is the recipe.

Cloudflare KV Cache-Warming Doesn’t Work the Way You Think#

A common “obvious” optimization for Cloudflare KV: at the end of your deploy, write the top-N popular cache entries (search results, config blobs, computed views) so the cache is “warm” when production traffic arrives. This doesn’t do what you think.

KV writes go to central data stores only. Regional edges populate on first read in that region — and replication propagation adds up to 60 seconds. Writing from one Worker doesn’t push the value globally; subsequent first-reads in each region still pay the central-store fetch.

Cloudflare Search Optimization: A Tiered Methodology#

A CF Workers + D1 + KV search endpoint has three classes of work you can ship to make it faster. They differ by cost-to-ship, not by impact. Order them right and you ship ~50% latency reduction in a day; order them wrong and you burn a week on Vectorize when the real win was a SELECT * you forgot to trim.

This page is the methodology, observed end-to-end on api.agent-zone.ai/api/v1/knowledge/search going from a 677ms baseline to 355ms then unlocking platform-level scale. Each tier is scope -> moves -> measured impact -> shipped commit.