Status signals

Decide between named events and exit codes, and see exactly what each signal does to a monitor.

Two ways to tell Drumbeats how a run went. They are equivalent for success and failure:

Named event
your_job && curl -sf "$API/success" || curl -sf "$API/failure"
your_job && curl -sf "$API/success" || curl -sf "$API/failure"
Exit code
your_job
curl -sf "$API/$?"
your_job
curl -sf "$API/$?"

Named events cost the same and do more. Exit codes cost the same and take one line. Pick by what the job needs.

Choose one#

Pick named events whenPick exit codes when
You need start, so you get duration tracking and hung-run detectionThe whole job is one command and $? is right there
You want log pings between phasesYou want the smallest possible change to the script
The job is event-driven and needs run_id correlationThe job has one clear success and failure boundary
You want to attach stdout or an error message to the finishYou do not need progress, duration, or correlation

Nothing stops you mixing them. A common shape is start as a named event and the finish as an exit code, which gives you duration tracking with one-line finish handling:

bash
RUN_ID="job-$(uuidgen)"
curl -sf "$API/start?run_id=$RUN_ID"
python worker.py
EXIT=$?
curl -sf "$API/${EXIT}?run_id=$RUN_ID"
RUN_ID="job-$(uuidgen)"
curl -sf "$API/start?run_id=$RUN_ID"
python worker.py
EXIT=$?
curl -sf "$API/${EXIT}?run_id=$RUN_ID"

What each signal does#

SignalOpens a runCloses a runChanges monitor statusCan open an incident
startyesnonoonly by timing out later
successnoyesflips UPno, it resolves them
failurenoyesflips DOWN at failure_toleranceyes, FAILED
lognononono
Exit code 0noyesflips UPno, it resolves them
Exit code 1 to 255noyesflips DOWN at failure_toleranceyes, FAILED

Two things follow from this table that surprise people:

A start on its own is not harmless. If no finish arrives inside max_duration_seconds, the run is recorded as hung and counted as a failure. Adding start to a job you do not always finish reporting will create alerts you did not have before.

A log ping is genuinely inert. It cannot resolve an incident, cannot keep a run alive, and cannot flip a monitor. It only costs a beat and leaves a note.

What happens when it breaks#

Incidents come from the finish signal, or the absence of one:

TriggerIncident event
failure, or a non-zero exit code, reaching failure_toleranceFAILED
start with no finish inside max_duration_secondsFAILED
No ping at all by the expected time plus the grace period, on Cron and Heartbeat monitorsMISSED
Finish slower than max_duration_secondsDURATION_HIGH, a warning
Finish faster than min_duration_secondsDURATION_LOW, a warning

How you get alerted#

Any of the above pages every notification group attached to the monitor, subject to the tolerances in alert logic. Recovery is automatic on the next success or 0.

The URLs#

By monitor UUID
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/start
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/success
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/failure
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/<exit-code>
POST https://api.drumbeats.io/v1/ping/<monitor-id>/log
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/start
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/success
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/failure
GET  https://api.drumbeats.io/v1/ping/<monitor-id>/<exit-code>
POST https://api.drumbeats.io/v1/ping/<monitor-id>/log
By project and monitor slug
GET  https://api.drumbeats.io/v1/s-ping/<project-id>/<monitor-slug>/<event>
GET  https://api.drumbeats.io/v1/s-ping/<project-id>/<monitor-slug>/<exit-code>
GET  https://api.drumbeats.io/v1/s-ping/<project-id>/<monitor-slug>/<event>
GET  https://api.drumbeats.io/v1/s-ping/<project-id>/<monitor-slug>/<exit-code>

Next#

Exit codes for the mapping table and the $? pitfalls. Scheduled pings and event-driven pings for complete patterns. Payloads for attaching context to any of these.