--- title: "Operating prometheus-stack Alertmanager: Operator Validation, Native Receivers, and Silence Discipline" description: "Day-2 operations for kube-prometheus-stack alertmanager: where validation errors actually surface, native receiver schema asymmetries (including Mattermost), secret-mount patterns, two-tier silence discipline, and synthetic alert testing." url: https://agent-zone.ai/knowledge/observability/prometheus-stack-alertmanager-operations/ section: knowledge date: 2026-05-07 categories: ["observability"] tags: ["alertmanager","prometheus-operator","kube-prometheus-stack","mattermost","silences","amtool","helm","observability"] skills: ["alertmanager-debugging","operator-config-validation","silence-discipline","synthetic-alert-testing"] tools: ["alertmanager","prometheus-operator","amtool","helm","kubectl"] levels: ["intermediate"] word_count: 1937 formats: json: https://agent-zone.ai/knowledge/observability/prometheus-stack-alertmanager-operations/index.json html: https://agent-zone.ai/knowledge/observability/prometheus-stack-alertmanager-operations/?format=html api: https://api.agent-zone.ai/api/v1/knowledge/search?q=Operating+prometheus-stack+Alertmanager%3A+Operator+Validation%2C+Native+Receivers%2C+and+Silence+Discipline --- A receiver YAML passes static review and the helm release reports `deployed`. The alertmanager pod is `Running 1/1`. A real critical alert fires and goes nowhere. The alertmanager pod logs are clean. The receiver works fine for a hand-rolled `curl` to the webhook URL. The trap is that the prometheus-operator generated a Secret containing the rendered config but flagged a sync error in *its own* logs — and the alertmanager pod kept serving the previous-good rendering, silently. This article assumes familiarity with the basic alertmanager routing tree, receivers, inhibition rules, and templating covered in [alertmanager-configuration](../alertmanager-configuration). It extends that material with the Day-2 operations of the kube-prometheus-stack chart specifically: where errors actually surface, what the native receiver schemas allow (and don't), and the silence discipline that keeps the alert pipeline trustworthy. ## Operator validation hierarchy When the prometheus-operator manages alertmanager via the `Alertmanager` and `AlertmanagerConfig` CRDs (or kube-prometheus-stack's `alertmanager.config` helm-values block), the validation pipeline has three layers — and errors at each layer surface in a different place. | Layer | Validates | Errors surface in | |---|---|---| | Helm template | YAML syntax, helm template functions | `helm upgrade` output | | prometheus-operator reconcile | alertmanager config schema (unknown fields, type mismatches, regex compile) | **operator pod logs** | | Alertmanager pod | Final `coordinator.go` load of the rendered file | alertmanager pod logs | The trap: layers 1 and 3 commonly look healthy when layer 2 has rejected the new config. The operator generates the Secret, attaches a sync-error condition, and stops promoting it; the alertmanager pod continues serving the last successfully-loaded config. There is no `CrashLoopBackOff`, no `Pending`, no helm-release failure — only a quiet reconcile loop in operator logs. ``` level=error ts=2026-05-07T14:22:11.493Z caller=operator.go:1421 component=alertmanageroperator msg="provisioning alertmanager configuration failed" err="failed to unmarshal alertmanager config: yaml: unmarshal errors: line 47: field title not found in type config.plain" ``` The reconcile retry is silent and indefinite (~30s by default). Nothing in the alertmanager pod, the helm release, or `kubectl get alertmanager` makes this visible without targeted log inspection. ### The 7-step diagnostic ladder When an alert isn't reaching its receiver, run these in order. Each step rules out one layer. ```bash # (1) Did the operator successfully reconcile the new config? kubectl logs -n monitoring deploy/-kube-prometheus-operator --tail=200 \ | grep -iE 'alertmanager|provision|reconcil|error' # (2) What did the operator render into the alertmanager Secret? kubectl get secret -n monitoring \ alertmanager--kube-prometh-alertmanager-generated \ -o jsonpath='{.data.alertmanager\.yaml\.gz}' | base64 -d | gunzip | head -200 # (3) Did the alertmanager pod actually load it? kubectl port-forward -n monitoring svc/-kube-prometh-alertmanager 9093:9093 amtool config show --alertmanager.url=http://localhost:9093 curl -s http://localhost:9093/api/v2/status | jq .config # (4) Does the alert route to the expected receiver? amtool config routes test --alertmanager.url=http://localhost:9093 \ severity=critical alertname=PodCrashLoopBackOff namespace=production # (5) Did the alert reach alertmanager at all? amtool alert query --alertmanager.url=http://localhost:9093 # (6) Is it silenced or inhibited? amtool silence query --alertmanager.url=http://localhost:9093 # (7) Did the receiver call succeed? kubectl logs -n monitoring alertmanager--kube-prometh-alertmanager-0 \ | grep -iE 'notify|error|retry' ``` A green step (1) with a stale config in step (2) and matching `config show` in step (3) confirms the silent-rejection trap. A green step (3) with empty step (5) points at Prometheus-side rule problems (see PrometheusRule discipline below). A green step (5) with empty step (7) points at receiver connectivity (DNS, secret mount, network policy). ## The Mattermost native receiver Alertmanager v0.27 introduced `mattermost_configs` as a first-class receiver, removing the need for a webhook translator sidecar. The schema is real but **not symmetric with `slack_configs` or `pagerduty_configs`**. The full set of allowed fields: - `webhook_url` *or* `webhook_url_file` - `send_resolved` (bool) - `channel` (string — channel name, no leading `#`) - `username` (string — overrides webhook default) - `icon_emoji` *or* `icon_url` - `priority` (string — `important`, `urgent`, or empty) - `text` (template-rendered body) - `http_config` **There is no `title:` field.** Slack, PagerDuty, OpsGenie, and Email receivers all have one. Mattermost does not. A config that copies the Slack pattern and sets `title:` will pass static review, render, and produce the silent-rejection symptom from the previous section. Fold the title prefix into the first line of `text:`: ```yaml receivers: - name: 'ops-critical' mattermost_configs: - webhook_url_file: '/etc/alertmanager/secrets/monitoring-secrets/alertmanager-mm-webhook' send_resolved: true channel: 'your-alerts-channel' # no leading # text: | [CRITICAL] {{ .GroupLabels.alertname }} @oncall @platform-lead {{ range .Alerts }}**{{ .Annotations.summary }}** {{ .Annotations.description }} --- Pod: `{{ .Labels.namespace }}/{{ .Labels.pod }}` · Status: `{{ .Status }}` {{ end }} ``` The wider lesson: **native receiver schemas are not symmetric**. Read the alertmanager config-schema reference for each integration before assuming a field exists. Static YAML review will not catch unknown-field errors; only an operator dry-run or `amtool check-config` against the rendered Secret will. ### Mounting the webhook URL via prometheus-operator `secrets:` Webhook URLs contain bearer-token-equivalent path segments. Embedding them in a helm values file (or in the operator-generated alertmanager Secret committed via GitOps) defeats rotation hygiene. The `webhook_url_file:` form combined with the operator's `secrets:` mount field reads from an out-of-band Secret that can be rotated without a `helm upgrade`. In the `Alertmanager` CRD spec — or under `alertmanager.alertmanagerSpec` in kube-prometheus-stack values: ```yaml alertmanager: alertmanagerSpec: secrets: - monitoring-secrets # k8s Secret in the same namespace ``` Mount path inside the alertmanager pod follows a fixed pattern: ``` /etc/alertmanager/secrets// # e.g. /etc/alertmanager/secrets/monitoring-secrets/alertmanager-mm-webhook ``` Create or rotate the Secret, then trigger the alertmanager pod to re-read: ```bash kubectl create secret generic monitoring-secrets \ -n monitoring \ --from-literal=alertmanager-mm-webhook='http://mattermost.svc.cluster.local:8065/hooks/REDACTED' \ --dry-run=client -o yaml | kubectl apply -f - kubectl rollout restart statefulset -n monitoring \ alertmanager--kube-prometh-alertmanager ``` ### Render path quirk: `props.attachments[0].text` When the receiver is Mattermost, alertmanager v0.32 sends notifications using MM's *attachment* format, not as plain post text. Tools that grep the Mattermost REST API response for keywords in `posts[*].message` will see nothing — the rendered alert body lives in `posts[*].props.attachments[0].text`: ```json { "id": "abc123", "message": "", "props": { "attachments": [{ "fallback": "[CRITICAL] PodCrashLoopBackOff", "text": "[CRITICAL] PodCrashLoopBackOff\n@oncall ..." }] } } ``` Visual rendering in the UI is correct. Test harnesses that assert against `message` will report false delivery failures. ## Silencing: amtool vs helm-values disable Chart-default rules generate false positives on minikube and other single-node dev clusters where `kube-etcd`, `kube-controller-manager`, `kube-scheduler`, and `kube-proxy` are not exposed as `Service` resources. The `TargetDown`, `etcdMembersDown`, and `NodeClockNotSynchronising` rules fire indefinitely. Silencing them buys quiet but doesn't fix the underlying noise generator. **Silences are the bridge; helm-values rule-disablement is the destination.** The two-tier strategy: ```bash # Tier 1 — immediate silence at runtime via amtool (or the UI) amtool silence add \ alertname=TargetDown job=kube-etcd \ --alertmanager.url=http://localhost:9093 \ --duration=720h \ --comment='single-node cluster, no etcd Service' amtool silence add \ 'alertname=~Target.*' job=kube-controller-manager \ --duration=720h \ --comment='single-node cluster' ``` ```yaml # Tier 2 — disable the rule generators in helm values defaultRules: disabled: NodeClockNotSynchronising: true kubeEtcd: enabled: false kubeControllerManager: enabled: false kubeScheduler: enabled: false kubeProxy: enabled: false ``` Once the Tier 2 PR ships, the silences become redundant and expire naturally. Editing the rule directly is not durable: chart-managed `PrometheusRule` resources regenerate on the next `helm upgrade`. Use `defaultRules.disabled.: true` for chart rules; use `kubeEtcd.enabled: false` and siblings to stop the upstream `ServiceMonitor`/`Endpoint` generators. ### Silence retention semantics - Silences are **deleted** ~5 days after they expire (default `--data.retention=120h` on alertmanager). - The `expired` state is queryable for that retention window via `amtool silence query --expired`. - Silences survive alertmanager pod restarts (persisted in alertmanager's `nflog` file under `--storage.path`, default `/alertmanager`). - Silences are **runtime state**, not config. They do not sync from prometheus-operator; recreate after a fresh cluster bootstrap. ## Verifying routing with synthetic alerts A synthetic alert via `POST /api/v2/alerts` is the only test that exercises the full receiver path: routing tree match, template render, secret read, webhook resolve, downstream channel post. Static `amtool check-config` catches schema. The chart's always-firing `Watchdog` alert proves the pipeline runs end-to-end on its baseline route — it does not prove that a *new* severity branch added in this `helm upgrade` routes correctly. Treat synthetic-alert delivery as a **mandatory** post-change verification step. ```bash curl -X POST http://localhost:9093/api/v2/alerts \ -H 'Content-Type: application/json' \ -d '[{ "labels": { "alertname": "TestAlert", "severity": "warning", "namespace": "production" }, "annotations": { "summary": "Synthetic test from operator", "description": "Verifying receiver routing" }, "startsAt": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'", "endsAt": "'"$(date -u -v+5M +%Y-%m-%dT%H:%M:%SZ)"'" }]' ``` Equivalent via amtool: ```bash amtool alert add alertname=TestAlert severity=warning namespace=production \ --alertmanager.url=http://localhost:9093 \ --annotation=summary='Synthetic test' \ --annotation=description='Verifying delivery' ``` Run one synthetic alert per severity branch on every config change. Confirm receipt in the downstream channel before declaring the change complete. ## Authoring custom PrometheusRule Two discipline points trip up most `PrometheusRule` authors writing rules outside the chart defaults. **The `release:` selector label.** The operator selects `PrometheusRule` resources via `Prometheus.spec.ruleSelector`. The kube-prometheus-stack default is: ```yaml ruleSelector: matchLabels: release: ``` A `PrometheusRule` without this label is dropped silently. Symptom: `kubectl get prometheusrules` shows the resource, but `/rules` in the Prometheus UI doesn't list it, and the alert never fires regardless of expression value. **`for:` thresholds calibrated to rollout windows.** `for:` controls how long an expression must be true before the alert transitions from `pending` to `firing`. Setting it too short produces false positives during normal `helm upgrade` rollouts; setting it too long hides slow rollouts past the point of usefulness. ```yaml - alert: DeploymentNotAvailable expr: | kube_deployment_status_replicas_available{namespace="production"} / kube_deployment_spec_replicas{namespace="production"} == 0 for: 4m labels: severity: critical ``` **4m is the empirical sweet spot for healthcheck-gated pods with ~30s readiness.** During a rollout, replicas can briefly hit 0 available (old terminating, new not-yet-Ready). 2m fires false; 5m hides slow rollouts; 4m is long enough to span typical rollouts and short enough to catch real outages within the operator's first reconcile cycle. Calibrate to your own readiness SLAs — but pick a number deliberately, not a round one. **Companion rules for missing-limit cases.** A common pattern is alerting on a ratio whose denominator can be absent. The main rule fails to fire for the most dangerous case (no limit set at all) because the ratio is undefined: ```yaml - alert: PodCPUSustained_NoLimit expr: | rate(container_cpu_usage_seconds_total{namespace="production",container!=""}[5m]) > 1.0 AND on (pod, container) absent(kube_pod_container_resource_limits{namespace="production",resource="cpu"}) for: 10m labels: { severity: warning } ``` Pair every ratio-based rule with an `absent()` companion that catches the denominator-missing case explicitly. ## The `--reuse-values` interaction `helm upgrade --reuse-values -f new-values.yaml` **silently ignores `-f`** for any path already present in stored values. The operator continues serving the old `alertmanager.config` block. There is no error and the helm release shows `deployed`. This is the most common reason a "deployed" alertmanager config update has no effect. ```bash # Wrong — appears to succeed, no-ops the -f helm upgrade prometheus-community/kube-prometheus-stack \ --reuse-values -f bootstrap/helm-values/prometheus-stack.yaml # Right — full re-render with -f helm upgrade prometheus-community/kube-prometheus-stack \ -n monitoring -f bootstrap/helm-values/prometheus-stack.yaml \ --version # Helm 3.14+: reset stored values, then layer new -f on top helm upgrade prometheus-community/kube-prometheus-stack \ --reset-then-reuse-values -f bootstrap/helm-values/prometheus-stack.yaml ``` Verify the new values landed: ```bash helm get values -n monitoring --revision helm history -n monitoring ``` If `helm get values` doesn't show your `-f` paths, the upgrade silently dropped them. Full mechanic and recovery patterns in [helm-gotchas-reuse-values-revisions-rollback](../../kubernetes/helm-gotchas-reuse-values-revisions-rollback). ## Operational lessons - **When the prometheus-operator manages alertmanager, alertmanager config errors surface in operator logs — the alertmanager pod itself stays quiet on bad config and serves the previous-good rendering.** Always include a step-(1) check on operator logs before chasing alertmanager-pod symptoms. - **Native receiver schemas are not symmetric.** `mattermost_configs` has no `title:` field despite slack/pagerduty/opsgenie all having one. Don't assume; read the schema for the specific integration. - **Static review can't catch unknown-field errors.** An `amtool check-config` (or operator dry-run) belongs in CI as a gate, not on the code reviewer's checklist. - **`helm upgrade --reuse-values -f new.yaml` silently ignores `-f` for paths already in stored values.** If the alertmanager config doesn't seem to be updating after a "successful" upgrade, suspect this first. - **Silence chart-default false-positives, but layer that under a helm-values rule-disablement PR.** Silences are the bridge, not the destination. - **A synthetic `POST /api/v2/alerts` is the only test that exercises the full receiver path.** Watchdog proves the pipeline runs; it doesn't prove a new severity branch routes correctly. - **When the receiver is Mattermost, debugging tools that grep the post `message` field will see nothing.** Alertmanager writes to `props.attachments[0].text`.