Notification groups

Bundle channels into routes so changing your on-call rotation is one edit, not fifty.

A group is a named bundle of channels:

plaintext
"Backend on-call"  =  Slack #backend-alerts
                   +  email backend@company.com
                   +  browser push to the duty engineer
"Backend on-call"  =  Slack #backend-alerts
                   +  email backend@company.com
                   +  browser push to the duty engineer

Point fifty monitors at that group and changing the on-call rotation is one edit. Point them at channels individually and it is fifty.

When a group is paged, every channel in it fires in parallel.

Create one#

  1. Open Settings → Notification Groups → New Group.
  2. Name it after the audience or the severity: Backend on-call, Critical infra, Low priority.
  3. Tick the channels it should contain.
  4. Save.

Then open any monitor, go to Notifications, and assign the group. A monitor can hold several groups. Every channel in every assigned group fires, and Drumbeats removes duplicates when a channel appears in more than one.

Name groups after who gets woken up#

A group called Critical infra tells you something. A group called Cron monitors does not, because the monitor type has nothing to do with who should respond.

MonitorGroupFans out to
Nightly backupDBA on-callSlack #dba, email dba@company.com
Invoice generatorFinance on-callSlack #finance, email finance@company.com
Queue workerBackend on-callSlack #backend-alerts, push to the duty engineer
Health checkLow prioritySlack #monitoring

Layer groups instead of duplicating them#

Two groups on one monitor compose. Both fire, so you can keep a base group for everyday visibility and add a severity group on top without rebuilding the channel list:

plaintext
Payment processor  →  "Backend on-call"   (Slack + email)
                   →  "Critical infra"    (PagerDuty)
Payment processor  →  "Backend on-call"   (Slack + email)
                   →  "Critical infra"    (PagerDuty)

Skip the group for a one-off destination#

A monitor can also point at channels directly, without a group. Both lists fire.

Use this sparingly. A direct channel is invisible to anyone reading the group list later, so it is the routing that gets forgotten during a handover. Reach for it only when exactly one monitor needs exactly one destination and a group would be ceremony.

What happens when it breaks#

SituationWhat you get
Monitor has no group and no direct channelNothing. The incident is recorded and nobody is paged
Group exists but holds no channelsNothing. Same outcome
One channel in the group fails, others succeedThe working channels get the alert. The failed one is not retried
Every channel in the group failsDrumbeats retries the delivery for about a minute, then stops

How you get alerted during setup#

Add a quiet channel to a new group while you are wiring it up, such as a personal Telegram chat or a #dev-alerts Slack channel. Trigger a real failure with a manual failure ping and confirm the whole path works before the group is attached to anything that pages a human. Remove the quiet channel once you trust it.

Next#

Channels for setting up the endpoints a group bundles. Alert logic for what decides a group is paged at all. REST API notification groups for managing these from code.