Dead man's switch

Three meanings of the term

People who search for 'dead man switch' are after different things, so it helps to separate them.

• Machines and vehicles. The original meaning. A train driver holds a handle or pedal, a walk-behind lawn mower has a bar you squeeze, and a boat or jet ski has a kill cord clipped to the rider. If the operator lets go or falls off, the machine stops. Many trains also have a vigilance control that asks the driver to press a button at intervals, which is close to the software idea.
• Check in or it fires. Some personal and security tools use the name for a timer you must keep resetting. If you stop checking in, for example because you cannot be reached, the tool sends a stored message or releases or deletes data.
• Software monitoring. A job or service sends a signal on an expected schedule, and a separate system raises an alert when the signal stops. This is what monitoring tools, Drumbeats included, mean by the term. It is also called heartbeat monitoring.

How it works in monitoring

A normal check asks a system whether it is healthy. A dead man's switch turns that around. The thing being watched sends a check-in each time it does its work, and the monitor only has to notice when a check-in is late.

The monitor knows the expected schedule, such as every 5 minutes or 02:00 every night, and adds a grace period for normal delays. If that window closes with no check-in, the run counts as a missed run and the alert goes out.

This is why the pattern catches failures that logs and error trackers miss. A job that never started writes no logs and throws no errors. A server that is switched off, a crontab line lost in a deploy and an expired credential all look the same from the outside: nothing happens. A dead man's switch is built to notice nothing happening.

Prometheus users know a variant that watches the alerting system itself. An alert that always fires (called Watchdog in kube-prometheus, and DeadMansSwitch until early 2019) is sent to an outside service, which pages someone if that alert ever stops arriving.

Example

A nightly backup with a dead man's switch

# crontab: back up at 02:00, then check in
0 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 https://api.drumbeats.io/v1/ping/<monitor-id>/success

# crontab: back up at 02:00, then check in
0 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 https://api.drumbeats.io/v1/ping/<monitor-id>/success

The monitor expects a check-in shortly after 02:00 every night. If the server is down, cron is not running or backup.sh exits with an error, no check-in arrives, and once the grace period runs out the monitor sends an alert. Nothing on the server has to notice the problem.

Common mistakes

• Checking in before the work. A check-in at the start of a script proves the script started, not that it finished. Send it at the end, or send a start check-in and a separate one for success.
• Checking in no matter what. backup.sh; curl ... sends the check-in even when the backup failed. Chain it with &&, or report the failure explicitly so you hear about it straight away.
• A grace period that is too tight. Jobs start late when a host is busy. If the grace period does not allow for that, the switch fires on healthy runs and people learn to ignore it.
• Watching from the same machine. A watcher that runs on the server it watches goes down with it. The deadline has to be kept somewhere else.

Related terms

• HeartbeatA signal a running process sends at a regular interval to show it is still alive and working.
• Check-inThe request a job sends to its monitor to say that it ran. Many tools call it a ping.
• Missed runA run that was expected but never reported in, so it left no error behind.
• Grace periodThe extra time a monitor waits after a job was due before it calls the run missed.

All glossary terms