Incidents

When a monitor goes DOWN, Drumbeats opens an incident, notifies the assigned notification group, and auto-resolves the incident when the job recovers. The incident is the unit of "something needs attention" — a single one rolls up every alert, acknowledgement, recovery notification, and timeline event for one outage.

Incident lifecycle

The terminal state is always RESOLVED. There is no manual "closed without resolving" — if the underlying job stops pinging entirely, the incident stays open until a success arrives or the monitor is paused.

Incident types

The IncidentEvent field on the API tells you why the incident opened. Four values:

Duration events are warnings — they open an incident but do not flip the monitor to DOWN by default. Tune the thresholds from the monitor's settings page.

Incident states

What happens when a monitor goes DOWN

1. Drumbeats records the failure event (MISSED, FAILED, DURATION_HIGH, DURATION_LOW).
2. If failure_tolerance is exceeded, the monitor flips to DOWN.
3. Drumbeats opens an incident with the matching IncidentEvent.
4. Every channel in the assigned notification group receives the alert in parallel. See Alert logic for the cooldown rules that prevent repeat alerts during the same outage.
5. The incident timeline records every subsequent event — additional failures, log events, manual acknowledgements, and finally the recovery.

Recovery

A success ping (or, for Uptime monitors, a passing scheduled check) closes the incident:

1. Incident state transitions to RESOLVED.
2. The notification group receives a recovery message.
3. The monitor flips back to UP.
4. The cooldown counter resets — the next outage starts a fresh incident, not a continuation.

Tuning incident sensitivity

• failure_tolerance — consecutive failure pings before opening an incident. Set higher for flaky jobs to avoid alerts on transient errors.
• grace_period_seconds — additional time after the scheduled run before declaring MISSED. Increase if your scheduler is loose about timing.
• max_duration_seconds — used by Event-driven monitors to detect hung runs and by Cron / Heartbeat monitors to flag DURATION_HIGH.
• Surge protection. When the same monitor would open repeated incidents back-to-back, Drumbeats coalesces them. The notification group is paged once per outage, not per failure ping.

Acknowledge an incident

Acknowledging signals to your team that someone is on it. It does not resolve the incident — the monitor still needs a success ping. Acknowledging silences the per-channel re-page (per Alert logic) so the on-call rotation stops getting nudged while the team is actively fixing the issue.

From the dashboard, click Acknowledge on the incident card. From the REST API:

curl -X POST https://api.drumbeats.io/v1/incidents/<incident-id>/acknowledge \
  -H "X-API-Key: $DRUMBEATS_API_KEY"

curl -X POST https://api.drumbeats.io/v1/incidents/<incident-id>/acknowledge \
  -H "X-API-Key: $DRUMBEATS_API_KEY"

See REST API: incidents for the full surface.

Reading the timeline

The incident detail page shows every event for the outage in chronological order:

• Every ping received with its event type, source IP, and payload preview.
• Status transitions (UP → DOWN → UP).
• Notification deliveries with per-channel result.
• Manual acknowledgement / resolution actions.

If the failing job sent stdout / stderr as a payload (-d '{"payload": "..."}'), it shows up inline — no need to dig through logs to find the cause.

Related guides

• Notifications: alert logic — cooldown rules and re-page behaviour.
• Notifications: channels — where the incident messages are delivered.
• REST API: incidents — programmatic acknowledge / resolve and pagination.
• Cron monitors and Event-driven monitors — what opens incidents on each monitor type.