Alert Logic
When Drumbeats actually pages — tolerances, surge protection, recovery messages, and the OPEN/ACKNOWLEDGED/RESOLVED cooldown rules that drive re-paging.
Alert logic is the set of per-monitor rules Drumbeats uses to decide when a notification group actually fires — and when it stops firing. A single failed ping does not always page the team; how often it pages on repeated failures depends on failure_tolerance, schedule_tolerance, and alert_surge_threshold. Acknowledging an incident silences the re-page until recovery; a success ping closes the incident and triggers a recovery message.
When an alert fires
Drumbeats opens an incident — and pages the assigned notification group — when one of these conditions is satisfied:
| Condition | Incident event | Monitor types it can fire on |
|---|---|---|
No ping arrives by expected_time + grace_period_seconds | MISSED | Cron, Heartbeat |
failure ping (or non-zero exit-code ping) arrives | FAILED | All |
start ping arrives but no success / failure before max_duration_seconds | Hung-run, counted as FAILED | All |
Run finishes faster than min_duration_seconds | DURATION_LOW | Cron, Heartbeat, Event-driven |
Run finishes slower than max_duration_seconds | DURATION_HIGH | Cron, Heartbeat, Event-driven |
DURATION_LOW and DURATION_HIGH open an incident but do not flip the monitor to DOWN by default — they are warnings.
Failure tolerance
A single failure ping does not always page the team. The monitor's failure_tolerance (default 1) is the number of consecutive failure pings required before the monitor flips DOWN and the group is paged.
failure_tolerance | Behaviour |
|---|---|
1 | Page on the very first failure ping (default). |
2 | Page on two consecutive failure pings — one transient failure is absorbed. |
3 | Page on three consecutive failure pings — useful for flaky jobs with known transient errors. |
schedule_tolerance works the same way for MISSED events on Cron and Heartbeat monitors — how many missed windows in a row before paging.
Bump tolerance for jobs you know have transient errors (network blips, race conditions); leave it at 1 for critical jobs where every failure should fire.
Alert surge protection
When a job enters a failure loop — failing every run, every minute, every second — repeated paging quickly becomes useless. alert_surge_threshold (default 10) is the cap on consecutive alerts for the same monitor. Once reached, Drumbeats pauses re-paging until the monitor recovers or the incident is RESOLVED. The incident still records every event in the timeline; only the channel deliveries are suppressed.
The acknowledgement cooldown
Acknowledging an incident is the team's signal that someone is on it. The Drumbeats Ping API state machine tracks this explicitly:
| State | Re-page behaviour |
|---|---|
OPEN | Each new failure event in the same incident re-pages the group, subject to surge protection. |
ACKNOWLEDGED | Re-page is paused — the on-call rotation stops getting nudged while the team is actively fixing it. New failures still extend the incident timeline and contribute to surge counting. |
RESOLVED | Closed. The next outage starts a fresh incident with a fresh surge counter. |
Acknowledging happens in the dashboard (per-incident Acknowledge button) or via REST API — Incidents. The incident enum is OPEN | ACKNOWLEDGED | RESOLVED everywhere — there is no separate RECOVERED state; recovery sends an INCIDENT_RECOVERY event with status: "RESOLVED".
Recovery messages
When an open incident's monitor receives a success ping (or, for Uptime monitors, a passing scheduled check), Drumbeats:
- Transitions the incident to
RESOLVED. - Sends a recovery message to every channel in every group that was originally paged.
- Flips the monitor back to UP.
- Resets the consecutive-failure and consecutive-miss counters to zero — the next outage starts fresh.
Recovery messages are valuable confirmation that the job is healthy again; keep them enabled so the on-call rotation knows when to stand down.
What an alert message contains
| Field | In incident alert | In recovery message |
|---|---|---|
| Monitor name | Yes | Yes |
Incident event (MISSED / FAILED / DURATION_HIGH / DURATION_LOW) | Yes | Yes (the event that originally triggered it) |
| Timestamp | Started-at | Started-at + resolved-at |
| Outage duration | — | Yes |
| Deep link into the dashboard | Yes | Yes |
| Payload preview (when present) | First 200 chars on FAILED | — |
[SCREENSHOT: notifications/slack-pair — a Slack pair showing the MISSED incident message followed by the recovery message in the same channel]
Tips
Related guides
- Notifications hub — channels, groups, alert logic together.
- Channels and Groups — the delivery wiring this page assumes is already set up.
- Incidents — the state machine alert logic operates on.
- Monitor types — which
failure_tolerance/schedule_toleranceapply to each type. - REST API — Incidents — programmatic acknowledge / resolve.