Alert logic
Why a failing job sometimes pages you instantly and sometimes says nothing at all.
Four settings on the monitor decide whether a failure reaches a human:
{
"failure_tolerance": 1,
"schedule_tolerance": 1,
"alert_surge_threshold": 10,
"alert_enabled": true
}{
"failure_tolerance": 1,
"schedule_tolerance": 1,
"alert_surge_threshold": 10,
"alert_enabled": true
}Those are the defaults. As shipped, the first failure pages, and the eleventh consecutive alert stops paging for an hour.
Incidents are recorded regardless. Every setting on this page controls notifications, never whether the outage goes on the record.
What opens an incident#
| Condition | Incident event | Applies to |
|---|---|---|
No ping by the expected time plus grace_period_seconds | MISSED | Cron, Heartbeat |
A failure ping, or a non-zero exit code | FAILED | All job types |
start with no finish inside max_duration_seconds | FAILED | All job types |
Run finishes slower than max_duration_seconds | DURATION_HIGH | Cron, Heartbeat, Event-driven |
Run finishes faster than min_duration_seconds | DURATION_LOW | Cron, Heartbeat, Event-driven |
| Three failed HTTP checks in a row | FAILED | Uptime |
| Certificate crosses 30, 14, 7, or 1 day remaining | SSL_EXPIRING | Uptime |
DURATION_HIGH, DURATION_LOW, and SSL_EXPIRING open an incident but leave the monitor UP. They are warnings, not outages.
Absorb transient failures with tolerance#
failure_tolerance is how many consecutive failure pings it takes before the monitor flips DOWN and the group is paged.
| Value | Behaviour |
|---|---|
1 | The first failure pages. The default |
2 | One transient failure is absorbed. The second consecutive one pages |
3 | Useful for jobs with known flakiness, such as a scraper hitting a rate limit |
schedule_tolerance does the same for MISSED events on Cron and Heartbeat monitors: how many windows in a row can close empty before anyone hears about it.
Both counters reset to zero on the next success. They count consecutive failures, not failures in a period, so a job that fails every other run at failure_tolerance: 2 never pages at all.
Stop an alert storm#
A job stuck in a failure loop can produce an alert every minute. alert_surge_threshold caps it, at 10 consecutive alerts by default.
When the counter reaches the threshold, Drumbeats pauses paging for that monitor for one hour. Incidents keep being recorded and the timeline keeps filling in. Only the channel deliveries stop. After the hour, paging resumes if the monitor is still failing.
The counter resets to zero when the monitor recovers, so a fresh outage starts with a full budget.
Silence a monitor without deleting it#
Set alert_enabled to false and the monitor keeps evaluating, keeps recording incidents, and keeps its history, but never pages. Use it for a monitor you are still tuning, or one whose failures you are tracking without wanting to act on them yet.
To stop evaluation as well, pause the monitor instead. Paused monitors accept pings without judging them, which is what you want during planned maintenance.
Acknowledge to stop the nudging#
| State | What happens next |
|---|---|
OPEN | New failure events in the same incident can page again, subject to surge protection |
ACKNOWLEDGED | Paging pauses. Someone is on it. New failures still extend the timeline and still count toward the surge threshold |
RESOLVED | Closed. The next outage opens a fresh incident with fresh counters |
Acknowledge from the incident's Acknowledge button, or through the REST API. The states are OPEN, ACKNOWLEDGED, and RESOLVED, with no separate recovered state. Recovery is an INCIDENT_RECOVERY event carrying status: "RESOLVED".
What recovery does#
A success ping, or a passing check on an Uptime monitor, triggers all of this at once:
- The incident moves to
RESOLVED. - A recovery message goes to every channel that was originally paged.
- The monitor flips back to UP.
- The consecutive failure, miss, and alert counters reset to zero.
Keep recovery messages on. A team that only ever hears about failures has no signal to stand down, so someone keeps watching a dashboard for a job that recovered forty minutes ago.
What each message carries#
| Field | In the alert | In the recovery message |
|---|---|---|
| Monitor name | yes | yes |
| Incident event | yes | yes, the event that opened it |
| Timestamp | started at | started at and resolved at |
| Outage duration | no | yes |
| Link to the incident | yes | yes |
| Payload preview | first 200 characters on FAILED | no |
Set the grace period wider than feels right#
A short grace_period_seconds is the leading cause of false MISSED alerts. A cron job scheduled at 02:00 that starts at 02:04 because the host was busy is not a failure, but a 60-second grace period says it is.
The default is 300 seconds. Start there. A late alert costs a few minutes. A false alert costs the team's trust in the whole system, and the first thing people do with an alert they do not trust is mute it.
Next#
Channels and groups for the delivery wiring this page assumes exists. Incidents for the lifecycle these rules act on. Monitor types for which tolerances apply to which type.