Missed run

Why runs get missed

• The job never started. The server was off or rebooting, the cron daemon was stopped, the crontab line was removed in a deploy, or the job was paused and never resumed.
• The job started but never finished. It hung on a network call, waited on a lock, or was killed for using too much memory.
• The check-in was lost. The script exited before it reached the check-in, or the server could not reach the monitoring service.
• The timing is off. The expected schedule in the monitor does not match the real one, often because of a timezone or daylight saving difference.

Missed run or failed run?

A failed run is the job telling you it went wrong, with a non-zero exit code or an explicit failure signal. You can alert on it the moment it happens. A missed run is the absence of any signal, so a monitor can only call it once the deadline, the due time plus the grace period, has passed.

Good monitoring catches both. Report failures explicitly for fast alerts, and keep a deadline as the backstop for everything that cannot report at all.

Example

How a missed run is detected

Expected schedule:  0 * * * *   (every hour, on the hour)
Grace period:       5 minutes

13:00  due   check-in at 13:02     on time
14:00  due   check-in at 14:01     on time
15:00  due   nothing by 15:05      deadline passed: missed run, alert sent
16:00  due   check-in at 16:03     on time, incident resolved

Expected schedule:  0 * * * *   (every hour, on the hour)
Grace period:       5 minutes

13:00  due   check-in at 13:02     on time
14:00  due   check-in at 14:01     on time
15:00  due   nothing by 15:05      deadline passed: missed run, alert sent
16:00  due   check-in at 16:03     on time, incident resolved

The 15:00 run left no trace on the server, but the monitor knew it was due. The alert goes out at the 15:05 deadline, not the next morning when someone notices a report is missing.

Common mistakes

• Only alerting on errors. Error trackers and log alerts need the job to run and say something. A run that never started says nothing, so it never triggers them.
• Skipping the check-in on purpose. If a job finds nothing to do and exits early without checking in, the monitor sees a missed run. Send the success check-in anyway, with a note that the run was skipped.
• Several jobs on one monitor. If two jobs check in to the same monitor, a check-in from one can hide a missed run of the other. Give each job its own monitor.
• A timezone mismatch. A job that runs at 02:00 local time, watched by a monitor that expects 02:00 UTC, looks missed every day, or, in a zone that matches UTC only in winter such as London, all summer. Set the monitor to the timezone the job runs in, and avoid times that the clock change skips or repeats.

Related terms

• Grace periodThe extra time a monitor waits after a job was due before it calls the run missed.
• Silent failureA failure that raises no error and no alert, so it is only found when someone misses the result.
• Dead man's switchA mechanism that acts when an expected signal stops arriving. In monitoring, silence is the alert.
• Check-inThe request a job sends to its monitor to say that it ran. Many tools call it a ping.

All glossary terms