Agent Failure Alerting

Idea In 30 Seconds

Failure alerting for AI agents gives an early signal when the system enters degradation.

Its goal is not just to "notify about an error", but to show in time what is breaking: tools, LLM steps, latency, or health checks.

Without alerts, teams usually learn about issues from users, not from the system.

Core Problem

Logs and tracing explain incidents well after they happen.

But without alerts, it is hard to notice the moment when the problem only starts: timeout rate grows, synthetic success drops, p95 latency rises. Because of this, it is easy to miss the transition from local degradation to cascading failure.

Next, we break down how to build alerts so they are useful, not noisy.

In production this often looks like:

• signal appears too late, when SLO is already violated;
• alerts are noisy because of temporary spikes and start being ignored;
• one problem generates dozens of duplicates across channels;
• there is no clear route: who should react and what should be done first.

That is why alert layer should be designed as a separate observability element, not as an "extra webhook".

How It Works

Failure alerting usually has three levels:

• signals (error_rate, timeout_rate, latency_p95, health_score);
• rules (threshold, window, severity, cooldown);
• routing (on-call, team, playbook, escalation).

These levels answer: when to react, who reacts, and how to act. Logs and tracing are needed to quickly move from alert to root cause. In production, an alert usually contains not only severity, but also owner/team or playbook_link. Alert rules should reflect SLO violations, not arbitrary thresholds.

Alert noise != reliability. If alerts fire often and without priority, the team starts ignoring them.

Alerts appear where degradation is already visible in metrics, latency, or health checks. Synthetic alerts show that the system is "alive", but user cannot complete the task.

Typical Production Metrics For Alerting

mtta and mttr are usually calculated in incident platform (PagerDuty/Opsgenie/custom incident log), not directly in agent runtime code.

To keep alerts useful, metrics are usually segmented by severity, workflow, release, and component.

Important: do not add high-cardinality fields (run_id, request_id, user_id) to labels, or alert metrics quickly become unmanageable.

How To Read The Alert Layer

What fired -> why it fired -> who should do what. These are three levels you should always read together.

Focus on trends and signal correlation, not one isolated alert.

Now look at signal combinations:

• timeout_rate up + latency_p95 up -> service degradation already impacts users;
• health_score down + synthetic_run_success_rate down -> critical workflow stops working end-to-end;
• tool_error_rate up + alert_fire_rate up -> unstable tool creates an alert cascade;
• false_positive_rate up + mtta up -> team trust in alerts drops;
• missed_incident_rate up + error_rate up -> there are gaps in alerting rules.

When To Use

Full failure alerting is not always required.

For a simple prototype, a basic alert on service-down can be enough.

But system-level alerting becomes critical when:

• agent system is already in production;
• there are availability/latency/workflow-success SLO/SLA;
• system depends on multiple tools and external APIs;
• on-call response is needed without manual dashboard watching.

Implementation Example

Below is a simplified alert-evaluator loop. It shows baseline approach: threshold + window + cooldown + event deduplication.

import time
from collections import defaultdict, deque

ALERT_RULES = {
    "high_timeout_rate": {
        "threshold": 0.05,
        "window_sec": 300,
        "severity": "high",
        "cooldown_sec": 600,
    },
    "latency_p95_regression": {
        "threshold": 2500,  # ms
        "window_sec": 300,
        "severity": "medium",
        "cooldown_sec": 600,
    },
    "synthetic_run_failed": {
        "threshold": 1,
        "window_sec": 120,
        "severity": "critical",
        "cooldown_sec": 300,
    },
}


class AlertEngine:
    def __init__(self):
        self.series = defaultdict(deque)  # metric_name -> [(ts, value), ...]
        self.last_fired_at = {}  # rule_name -> ts

    def ingest(self, metric_name, value, ts=None):
        ts = ts or time.time()
        self.series[metric_name].append((ts, value))

    def evaluate(self, ts=None):
        ts = ts or time.time()
        fired = []

        for rule_name, rule in ALERT_RULES.items():
            if self._in_cooldown(rule_name, ts, rule["cooldown_sec"]):
                continue

            if rule_name == "high_timeout_rate":
                value = self._latest_in_window("timeout_rate", ts, rule["window_sec"])
                if value is not None and value >= rule["threshold"]:
                    fired.append(self._build_alert(rule_name, value, rule, ts))

            if rule_name == "latency_p95_regression":
                value = self._latest_in_window("run_latency_p95_ms", ts, rule["window_sec"])
                if value is not None and value >= rule["threshold"]:
                    fired.append(self._build_alert(rule_name, value, rule, ts))

            if rule_name == "synthetic_run_failed":
                value = self._latest_in_window("synthetic_run_failed", ts, rule["window_sec"])
                if value is not None and value >= rule["threshold"]:
                    fired.append(self._build_alert(rule_name, value, rule, ts))

        return fired

    def _latest_in_window(self, metric_name, now_ts, window_sec):
        # NOTE:
        # This example checks only the latest point (spikes may fire alerts).
        # In production (Prometheus/Datadog), teams usually require
        # sustained anomaly duration (for example, "for: 5m")
        # to avoid alerts on short spikes.
        # Alternative: check sustained breach across whole window,
        # not only latest point.
        points = self.series[metric_name]
        while points and now_ts - points[0][0] > window_sec:
            points.popleft()
        return points[-1][1] if points else None

    def _sustained_breach(self, metric_name, now_ts, window_sec, threshold):
        points = self.series[metric_name]
        while points and now_ts - points[0][0] > window_sec:
            points.popleft()
        return points and all(v >= threshold for _, v in points)

    def _in_cooldown(self, rule_name, now_ts, cooldown_sec):
        last_ts = self.last_fired_at.get(rule_name)
        return last_ts is not None and now_ts - last_ts < cooldown_sec

    def _build_alert(self, rule_name, value, rule, now_ts):
        self.last_fired_at[rule_name] = now_ts
        return {
            "rule": rule_name,
            "severity": rule["severity"],
            "value": value,
            "timestamp": now_ts,
        }

In production, alerts usually fire not on one spike, but when threshold holds for the full window.

This is how alert metrics can look on a real dashboard:

Investigation

When an alert fires:

1. check severity and whether this is duplicate in cooldown window;
2. find correlated metrics signals (latency, timeout, health);
3. open problematic runs in tracing;
4. confirm root cause in logs and run playbook.

Common Mistakes

Even when alerts exist, they often fail because of common mistakes below.

Too many alerts without prioritization

If all alerts are equally critical, team quickly loses trust.

No cooldown and deduplication

One problem creates dozens of identical notifications and makes on-call response harder.

No synthetic-based alerts

Infrastructure-only alerts do not guarantee workflow actually works. Because of this, teams can miss early multi-agent chaos.

Alerts are not linked to playbook

Notifications exist, but team does not know what to do next. This increases MTTR during incident.

High-cardinality labels in alert metrics

Adding run_id or request_id to labels quickly overloads metrics system and complicates analysis.

Self-Check

Below is a short checklist for baseline failure alerting before release.

FAQ

Q: How is failure alerting different from health checks?
A: Health checks show current system state, while failure alerting decides when and whom to notify for timely response.

Q: What is the minimum alert set to start with?
A: Start with timeout_rate, error_rate, latency_p95, and synthetic_run_success_rate.

Q: How to reduce alert noise?
A: Add severity levels, cooldown, deduplication, and remove rules with frequent false positives.

Q: How to know alerts cover real risks?
A: Review missed_incident_rate after incidents and update rules where system degraded without notification.

Related Pages

Next on this topic:

• Agent Health Checks — early degradation signals before incident.
• Agent Metrics — system signals for alert rules.
• Agent Latency Monitoring — how to build latency-based alerts.
• Agent Tracing — how to move from alert to problematic step.
• Agent Logging — data for fast root-cause analysis.