Silent failure

Why failures go unnoticed

A failure is only loud if something is listening for it. Cron starts commands but does not check how they end, and many job runners record a failure in their own interface and wait for someone to look. Typical cases:

• The job never ran. A job that does not start cannot report anything.
• The errors went nowhere. Output went to /dev/null or to a log file nobody reads.
• The job succeeded at nothing. It exited with code 0 but did no useful work.

For the full list of causes in cron jobs, with a checklist and fixes, read Why your cron job failed silently.

How to catch them

You cannot alert on an error that is never raised, so alert on a missing success instead. Have the job send a check-in when it finishes, and let an outside monitor expect it on the job's expected schedule. That turns most silent failures into missed runs, which a monitor can see.

For jobs that can finish without doing anything useful, also check what they did: a row count, a file size or a minimum run time.

Example

Turning a silent failure into an alert

# Before: output is thrown away, nobody hears about a failure
0 3 * * * /opt/jobs/export.sh > /dev/null 2>&1

# After: keep the output and report the exit code either way
0 3 * * * /opt/jobs/export.sh >> /var/log/export.log 2>&1; curl -fsS -m 10 "https://api.drumbeats.io/v1/ping/<monitor-id>/$?"

# Before: output is thrown away, nobody hears about a failure
0 3 * * * /opt/jobs/export.sh > /dev/null 2>&1

# After: keep the output and report the exit code either way
0 3 * * * /opt/jobs/export.sh >> /var/log/export.log 2>&1; curl -fsS -m 10 "https://api.drumbeats.io/v1/ping/<monitor-id>/$?"

The first line hides every error. The second keeps the output and sends the exit code to the monitor, where 0 counts as success and anything else as a failure. If the server is down and the line never runs, the monitor still notices, because the check-in it expects never arrives.

Common mistakes

• Reading no news as good news. A dashboard with nothing red on it can mean every job worked, or that nothing ran.
• Checking in at the start. A check-in before the work proves the job started. It says nothing about whether it finished.
• Watching only the exit code. A job can exit with 0 after doing nothing. If an empty run is a problem, check the result as well.

Related terms

• Missed runA run that was expected but never reported in, so it left no error behind.
• Dead man's switchA mechanism that acts when an expected signal stops arriving. In monitoring, silence is the alert.
• Check-inThe request a job sends to its monitor to say that it ran. Many tools call it a ping.

All glossary terms