Monitoring glossary

Grace period

Also called: grace time, grace window

A grace period is the extra time a monitor waits after a job was due before it treats the run as missed and sends an alert. It covers normal delays: a busy host that starts the job a minute late, a run that takes longer than usual, a slow network between the job and the monitor. Without one, every small delay would look like an outage.

How it works

A cron or heartbeat monitor works out from its expected schedule when the next check-in is due. The grace period is added on top of that time. The result is the deadline: only when the due time plus the grace period has passed with no check-in does the run count as a missed run.

Pay attention to what the due time means. Many monitors, Drumbeats included, measure a cron job from the time it is due to start. Most jobs check in when they finish. So the grace period has to cover how long the job normally runs, plus how late it may start.

Picking a value

Start from the longest normal run time you have seen, not the average.
Add the delay you see before jobs start: a few seconds on a quiet server, minutes on a shared CI runner or an autoscaled cluster.
Round up. A grace period that is a few minutes too long delays one alert by a few minutes. One that is too short sends false alerts every day.

Example

A grace period sized to the job

Expected schedule:  0 2 * * *   (02:00 every night)
Normal run time:    8 to 12 minutes
Grace period:       20 minutes

02:00  job is due to start
02:11  job finishes and checks in      -> on time
02:20  deadline (02:00 plus 20 minutes)
       no check-in by now               -> missed run, alert

Expected schedule:  0 2 * * *   (02:00 every night)
Normal run time:    8 to 12 minutes
Grace period:       20 minutes

02:00  job is due to start
02:11  job finishes and checks in      -> on time
02:20  deadline (02:00 plus 20 minutes)
       no check-in by now               -> missed run, alert

With a 5 minute grace period this job would be called missed at 02:05 every night, while it was still running. With 20 minutes the healthy runs pass quietly, and a run that never happens still raises an alert at 02:20.

Common mistakes

Shorter than the run time. If the check-in is sent at the end and the grace period is shorter than the job, every healthy run looks missed. Teams then mute the monitor, and it stops protecting anything.
Far longer than needed. A grace period of several hours on a job that runs every 15 minutes means you hear about an outage long after it started.
Tuned on one good day. Run times grow with data. Check the run times you actually see every few months, and widen the grace period before it starts to bite.
Confused with alert tolerance. The grace period delays the decision on one run. Allowing several missed runs in a row before alerting is a different setting, and it delays the alert by whole cycles.

How Drumbeats handles it

Every Drumbeats cron and heartbeat monitor has a grace period. The default is 5 minutes, and you can set it in seconds, minutes, hours or days. On a cron monitor, the deadline is the time the job is due to start, in the monitor's timezone, plus the grace period. If your job sends success when it finishes, make the grace period longer than its normal run time. On a heartbeat monitor, the next ping is due one interval after the last one, and the deadline is that due time plus the grace period.

The Missed check-ins setting is separate. It decides how many missed runs in a row it takes before the monitor goes down and alerts, and it defaults to 1.

All glossary terms

Monitoring for scheduled work

Hear about the run that did not happen.

Your job pings Drumbeats when it finishes. If the ping does not arrive on time, you get an alert. No agent to install, free for up to 50 monitors.

Start monitoring free Set up with AI agent

Free for up to 50 monitors · Setup in 60 seconds · No credit card required

Definition