Heartbeat monitors

A worker loop that reports it is alive:

import time, requests

API = "https://api.drumbeats.io/v1/ping/<monitor-id>"

while True:
    poll_queue()
    requests.get(f"{API}/success", timeout=5)
    time.sleep(300)

import time, requests

API = "https://api.drumbeats.io/v1/ping/<monitor-id>"

while True:
    poll_queue()
    requests.get(f"{API}/success", timeout=5)
    time.sleep(300)

Drumbeats expects one success inside each interval plus the grace period. Miss the window and it opens an incident.

Use a Heartbeat monitor for long-lived workers, polling loops, and any process that should stay alive rather than run once and exit.

Configure the monitor#

curl -X POST https://api.drumbeats.io/v1/monitors \
  -H "X-API-Key: $DRUMBEATS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "<project-id>",
    "name": "Queue poller",
    "type": "JOB_HEARTBEAT",
    "schedule": "5m",
    "grace_period_seconds": 60,
    "max_duration_seconds": 300
  }'

curl -X POST https://api.drumbeats.io/v1/monitors \
  -H "X-API-Key: $DRUMBEATS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "<project-id>",
    "name": "Queue poller",
    "type": "JOB_HEARTBEAT",
    "schedule": "5m",
    "grace_period_seconds": 60,
    "max_duration_seconds": 300
  }'

Free accounts cannot go below a 60 second interval. Paid plans cannot go below 30 seconds.

Understand the interval grid#

Expected beat times sit on a fixed grid. Drumbeats anchors the grid when you create the monitor, then expects a ping at anchor, anchor plus one interval, anchor plus two intervals, and so on.

Monitor created 14:02:30, schedule 5m

Grid:      14:02:30   14:07:30   14:12:30   14:17:30
Due by:    +5m grace  +5m grace  +5m grace  +5m grace

Monitor created 14:02:30, schedule 5m

Grid:      14:02:30   14:07:30   14:12:30   14:17:30
Due by:    +5m grace  +5m grace  +5m grace  +5m grace

The grid does not slide forward to match your last ping. A ping at 14:09 does not move the next deadline to 14:14. It stays at 14:12:30 plus the grace period. The anchor is recalculated when a monitor recovers from an outage, so a worker that comes back on a new cadence re-aligns rather than fighting the old grid.

Wire the ping into your worker#

Branch on the work when the loop body can fail:

import time, requests

API = "https://api.drumbeats.io/v1/ping/<monitor-id>"

while True:
    try:
        process_batch()
        requests.get(f"{API}/success", timeout=5)
    except Exception as exc:
        requests.post(f"{API}/failure", json={"payload": str(exc)}, timeout=5)
    time.sleep(300)

import time, requests

API = "https://api.drumbeats.io/v1/ping/<monitor-id>"

while True:
    try:
        process_batch()
        requests.get(f"{API}/success", timeout=5)
    except Exception as exc:
        requests.post(f"{API}/failure", json={"payload": str(exc)}, timeout=5)
    time.sleep(300)

Add start when an iteration can hang, so max_duration_seconds has something to measure against:

import time, uuid, requests

API = "https://api.drumbeats.io/v1/ping/<monitor-id>"

while True:
    run_id = str(uuid.uuid4())
    requests.get(f"{API}/start", params={"run_id": run_id}, timeout=5)
    try:
        process_batch()
        requests.get(f"{API}/success", params={"run_id": run_id}, timeout=5)
    except Exception as exc:
        requests.post(
            f"{API}/failure",
            params={"run_id": run_id},
            json={"payload": str(exc)},
            timeout=5,
        )
    time.sleep(300)

import time, uuid, requests

API = "https://api.drumbeats.io/v1/ping/<monitor-id>"

while True:
    run_id = str(uuid.uuid4())
    requests.get(f"{API}/start", params={"run_id": run_id}, timeout=5)
    try:
        process_batch()
        requests.get(f"{API}/success", params={"run_id": run_id}, timeout=5)
    except Exception as exc:
        requests.post(
            f"{API}/failure",
            params={"run_id": run_id},
            json={"payload": str(exc)},
            timeout=5,
        )
    time.sleep(300)

What happens when it breaks#

How you get alerted#

The incident pages every notification group on the monitor, and all their channels fire in parallel. The next success ping resolves the incident, flips the monitor to UP, sends a recovery message with the outage duration, and re-anchors the interval grid to the recovery time.

Handle the awkward cases#

One ping per batch, not per item#

Ping once after the batch completes and pass a count so the dashboard shows throughput:

batch = fetch_batch()
for item in batch:
    process(item)
requests.get(f"{API}/success", params={"items": len(batch)}, timeout=5)

batch = fetch_batch()
for item in batch:
    process(item)
requests.get(f"{API}/success", params={"items": len(batch)}, timeout=5)

Planned maintenance#

Pause the monitor before the window and resume after. Paused monitors record pings but do not evaluate them, so nothing pages. Pause from the dashboard or with the REST API.

Several replicas of the same worker#

Point all replicas at one monitor. Drumbeats only needs one ping per interval to call it healthy, so a rolling restart does not page anyone. Create separate monitors only when you genuinely need per-replica visibility.

Choose Heartbeat or Cron#

A job that an external scheduler starts every hour belongs on Cron, even though the cadence looks like an interval. You get exact-time alerting and a timeline that lines up with the schedule.

Next#

Cron monitors for schedule-bound jobs. Scheduled pings for the endpoint reference. Alert logic for tuning the tolerance fields.