Heartbeat

How heartbeat monitoring works

You pick an interval that matches how often the process does its work, for example every 5 minutes. Inside the work loop, after each healthy pass, the process sends a heartbeat, usually an HTTP request to a monitoring service. The monitor keeps track of when the next one is due. If an interval plus a grace period passes without a heartbeat, it raises an alert.

A heartbeat is a kind of dead man's switch: the alert is caused by silence, not by an error message. That is what lets it catch a worker that is still running but stuck, such as a consumer waiting forever on a lock, or a loop that swallows every exception and tries again.

Heartbeat or uptime check?

An uptime check works from the outside: a monitor requests your health endpoint and expects a good answer. A heartbeat works from the inside: the process reports in on its own. A worker can pass a health check, with the process up and the port answering, while doing no useful work. A heartbeat sent from inside the work loop only arrives when the work actually happens. Many teams use both.

Other meanings

In clusters and networks, a heartbeat is a message that machines send each other to find out when a peer has failed, for example between database replicas or between the two nodes of a high-availability pair. The idea is the same, a missing heartbeat means trouble, but sender and receiver are parts of one system rather than a job and an outside monitor.

Example

A worker loop that sends a heartbeat

import time
import requests

PING = "https://api.drumbeats.io/v1/ping/<monitor-id>"

def heartbeat():
    try:
        requests.get(f"{PING}/success", timeout=5)
    except requests.RequestException:
        pass  # a lost heartbeat must not stop the worker

while True:
    process_next_batch()
    heartbeat()
    time.sleep(300)  # about every 5 minutes

The heartbeat is sent after the batch, not before it, so it means 'a batch was processed', not just 'the process is up'. If process_next_batch() hangs or the process dies, the heartbeats stop and the monitor raises an alert once the interval and the grace period have passed. A heartbeat request that fails is caught, so a network problem never stops the worker itself.

Common mistakes

• Beating from a separate thread or timer. A background timer keeps sending heartbeats while the main loop is stuck. Send the heartbeat from the code that does the work.
• An interval with no headroom. A loop that sleeps 5 minutes and then works for 20 seconds really repeats every 5 minutes and 20 seconds. Set the interval to the real cadence, or leave room for the difference in the grace period.
• One heartbeat per item. Sending a heartbeat for every message in a busy queue adds thousands of requests and no information. One per batch or per pass of the loop is enough.
• Replicas that hide each other. Several replicas sending to one monitor is fine when any healthy replica means the service is healthy. If you need to know which replica stopped, give each one its own monitor.

Related terms

• Dead man's switchA mechanism that acts when an expected signal stops arriving. In monitoring, silence is the alert.
• Check-inThe request a job sends to its monitor to say that it ran. Many tools call it a ping.
• Grace periodThe extra time a monitor waits after a job was due before it calls the run missed.
• Missed runA run that was expected but never reported in, so it left no error behind.

All glossary terms