Background Job Monitoring

Background job monitoring: know when an async task fails or hangs

A background job is an async task with no user waiting on it, so nobody sees it fail. Drumbeats gives each job a URL to call when a run starts and when it ends. A failure ping, or a start with no finish inside the time limit, opens an incident and sends an alert. For a job on a schedule, so does a run that never starts.

50 monitors freeNo SDK or agentAny language that can send HTTPS

Which monitor fits which background job

How to monitor a background job depends on what starts it. A job on a clock can be late. An async task that runs when something triggers it cannot, so it needs a different check.

Background jobs by trigger, with the Drumbeats monitor type for each, what it alerts on and what it cannot tell you
The job runsExamplesMonitor to useAlerts onCannot tell you
On a clockA nightly export, an hourly sync, an invoice run in crontab, a Kubernetes CronJob, a scheduled GitHub Actions workflowA cron monitor with the same cron expression and time zone. See cron job monitoring.A run that has not reported by its due time plus the grace period, a failure ping, a run slower or faster than its limitsAnything about the job between runs
In a loop that never exitsA queue consumer, a poller, an outbox relay, a sync daemonA heartbeat monitor on an interval such as 1m. See heartbeat monitoring, and queue worker monitoring for Sidekiq, Celery, BullMQ, RQ and Laravel.An interval that passes with no ping, a failure pingWhether each item the loop handled succeeded
When something triggers itA webhook handler, upload processing, a job enqueued by a user action, a CI step, a serverless functionAn event-driven monitor, with a run_id on every ping and max_duration_seconds setA failure ping, a start with no finish inside the limit, a run slower or faster than its limitsThat the trigger stopped. With no schedule, silence is normal

The rest of this page is about the third row: one run of a triggered async task, from its start ping to its finish ping. All three monitor types take the same pings, described in the ping API reference.

How it works

One monitor per job, two pings per run.

1

Create an event-driven monitor

It has no schedule: Drumbeats records each run as the job reports it. Set max_duration_seconds, and min_duration_seconds if a run can be suspiciously fast.

2

Send /start, then /success or /failure

One outbound HTTPS request when the run begins and one when it ends. Both carry the same run_id: your job id, an idempotency key or a UUID.

3

A run that never finishes is failed for you

When a start ping has no success or failure ping inside max_duration_seconds, Drumbeats records the run as failed and alerts through any of 8 channels, from email and Slack to PagerDuty and webhooks. The task does not have to survive to report its own death.

Why background jobs are different

Async tasks fail in ways an APM does not report

An APM agent and an error tracker report what running code does: a slow span, an exception, a log line. Three failures of an async task produce none of those. A task that never started has no trace. A task killed by the kernel for running out of memory cannot send its last error. A task that exits cleanly after processing zero rows looks like every other success.

Drumbeats works from the other direction. The job says what it is about to do by calling a URL when a run starts and again when it ends, and the alert comes from Drumbeats when the expected call does not arrive or reports a failure. The check runs outside your infrastructure, so it still works when the host, the scheduler or the worker is the part that broke.

Five background job failures and what logs, an APM or error tracker, and job monitoring each show
FailureLogsAPM or error trackerJob monitoring with Drumbeats
The run threw an exceptionAn error line, found when someone searches or when an alert rule matches it.Yes. An error event with a stack trace.Yes. A failure ping, with the message you send as its payload.
The run never startedNothing is written, so there is nothing to find.Nothing ran, so there is no trace and no error. It takes a rule on missing data to notice.Yes for a job on a schedule: a cron or heartbeat monitor opens a MISSED incident once the due time and the grace period pass. A triggered job has no due time, so its event-driven monitor stays quiet.
The run was killed part-way (out of memory, SIGKILL, a drained node)The log stops. There is no error line, because the process did not get to write one.Usually no error event either, for the same reason.Yes. The start ping has no finish ping, and the run is failed at max_duration_seconds. That check runs every 60 seconds.
The run succeeded and did nothingLooks like every other success.Looks like every other success.A DURATION_LOW warning when the run is faster than min_duration_seconds, or the row count sent as the payload of the success ping.
The run is slow or stuckTimestamps, if the job logs them.Yes, and traces show where the time went.A run still open at max_duration_seconds is failed and alerts. If it finishes later it is also flagged DURATION_HIGH. You get the duration of every run, not the cause.

The two are complements: keep the APM for the cause of a failure and the job monitor for the fact of one. The longer comparison, with what Datadog, New Relic and Sentry do well and what each path costs for a 50-job workload, is in background job monitoring vs APM.

Start and finish pings in your language

A ping is one HTTPS request, GET or POST, to https://api.drumbeats.io/v1/ping/<monitor-id>/<event>. There is no SDK, no agent and no runtime to install, so the same requests work from a shell script, a Lambda function or a container that has nothing but curl. The snippets read the URL up to the monitor id from DRUMBEATS_MONITOR_URL.

Node.js

A wrapper for any async function. The example gives every delivery of a Stripe webhook event its own run.

const base = process.env.DRUMBEATS_MONITOR_URL!;

// A failed or slow ping must not fail the job.
const ping = (event: string, runId: string, error?: string) =>
  fetch(`${base}/${event}?run_id=${encodeURIComponent(runId)}`, {
    method: error ? 'POST' : 'GET',
    headers: error ? { 'Content-Type': 'application/json' } : {},
    body: error ? JSON.stringify({ payload: error }) : undefined,
    signal: AbortSignal.timeout(3000),
  }).catch(() => {});

export async function monitored<T>(
  runId: string,
  job: () => Promise<T>,
): Promise<T> {
  await ping('start', runId);
  try {
    const result = await job();
    await ping('success', runId);
    return result;
  } catch (err) {
    const message = err instanceof Error ? err.message : String(err);
    await ping('failure', runId, message);
    throw err;
  }
}

app.post('/webhooks/stripe', async (req, res) => {
  // Stripe redelivers an event under the same id, so the id alone
  // is not a run id: the timestamp keeps each delivery apart.
  const runId = `${req.body.id}-${Date.now()}`;
  try {
    await monitored(runId, () => handleStripeEvent(req.body));
    res.sendStatus(200);
  } catch {
    // monitored() has sent the failure ping and rethrown. Answer,
    // or the request hangs until Stripe times out.
    res.sendStatus(500);
  }
});
const base = process.env.DRUMBEATS_MONITOR_URL!;

// A failed or slow ping must not fail the job.
const ping = (event: string, runId: string, error?: string) =>
  fetch(`${base}/${event}?run_id=${encodeURIComponent(runId)}`, {
    method: error ? 'POST' : 'GET',
    headers: error ? { 'Content-Type': 'application/json' } : {},
    body: error ? JSON.stringify({ payload: error }) : undefined,
    signal: AbortSignal.timeout(3000),
  }).catch(() => {});

export async function monitored<T>(
  runId: string,
  job: () => Promise<T>,
): Promise<T> {
  await ping('start', runId);
  try {
    const result = await job();
    await ping('success', runId);
    return result;
  } catch (err) {
    const message = err instanceof Error ? err.message : String(err);
    await ping('failure', runId, message);
    throw err;
  }
}

app.post('/webhooks/stripe', async (req, res) => {
  // Stripe redelivers an event under the same id, so the id alone
  // is not a run id: the timestamp keeps each delivery apart.
  const runId = `${req.body.id}-${Date.now()}`;
  try {
    await monitored(runId, () => handleStripeEvent(req.body));
    res.sendStatus(200);
  } catch {
    // monitored() has sent the failure ping and rethrown. Answer,
    // or the request hangs until Stripe times out.
    res.sendStatus(500);
  }
});

Python

A decorator. Every call is one run with a fresh UUID, and the exception text travels as the payload of the failure ping.

import functools
import os
import uuid

import requests

BASE = os.environ["DRUMBEATS_MONITOR_URL"]

def ping(event, run_id, error=None):
    url, params = f"{BASE}/{event}", {"run_id": run_id}
    try:
        if error is None:
            requests.get(url, params=params, timeout=3)
        else:
            requests.post(url, params=params,
                          json={"payload": error}, timeout=3)
    except requests.RequestException:
        pass  # a failed ping must not fail the job

def monitored(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        run_id = str(uuid.uuid4())
        ping("start", run_id)
        try:
            result = fn(*args, **kwargs)
        except Exception as exc:
            ping("failure", run_id, str(exc))
            raise
        ping("success", run_id)
        return result
    return wrapper

@monitored
def render_thumbnails(image_id):
    ...
import functools
import os
import uuid

import requests

BASE = os.environ["DRUMBEATS_MONITOR_URL"]

def ping(event, run_id, error=None):
    url, params = f"{BASE}/{event}", {"run_id": run_id}
    try:
        if error is None:
            requests.get(url, params=params, timeout=3)
        else:
            requests.post(url, params=params,
                          json={"payload": error}, timeout=3)
    except requests.RequestException:
        pass  # a failed ping must not fail the job

def monitored(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        run_id = str(uuid.uuid4())
        ping("start", run_id)
        try:
            result = fn(*args, **kwargs)
        except Exception as exc:
            ping("failure", run_id, str(exc))
            raise
        ping("success", run_id)
        return result
    return wrapper

@monitored
def render_thumbnails(image_id):
    ...

Shell

For a script, a CI step or a container entrypoint. The exit code is sent as the event: 0 records a success, any other number a failure.

#!/usr/bin/env bash
BASE="$DRUMBEATS_MONITOR_URL"
RUN_ID="export-$(uuidgen)"

curl -fsS -m 5 --retry 2 "$BASE/start?run_id=$RUN_ID" > /dev/null

/opt/scripts/export-customer-data.sh
status=$?

# The exit code is the event: 0 is a success, anything else a failure.
curl -fsS -m 5 --retry 2 "$BASE/$status?run_id=$RUN_ID" > /dev/null
#!/usr/bin/env bash
BASE="$DRUMBEATS_MONITOR_URL"
RUN_ID="export-$(uuidgen)"

curl -fsS -m 5 --retry 2 "$BASE/start?run_id=$RUN_ID" > /dev/null

/opt/scripts/export-customer-data.sh
status=$?

# The exit code is the event: 0 is a success, anything else a failure.
curl -fsS -m 5 --retry 2 "$BASE/$status?run_id=$RUN_ID" > /dev/null

Three rules keep the pings honest. Send the success ping after the work is committed, not before. Send the finish ping on every exit path, or an early return looks like a hung run. Give every ping a timeout, so a slow connection to Drumbeats cannot hold the job.

Step-by-step guides: Node.js, Python, PHP, Go, Shell, Kubernetes, GitHub Actions. For jobs on Sidekiq, Celery, BullMQ, RQ or Laravel queues, the hook that wraps every job is on the queue worker monitoring page.

Choosing a run id and time limits

Four settings and the payload decide what a background job monitor can tell you. The two time limits are off until you set them.

Per-run settings of a background job monitor, what each controls and how to choose it
SettingWhat it controlsHow to choose it
run_idOne run: it pairs a start ping with its finish ping. Required on an event-driven monitor, at most 255 characters.Unique per run and the same on every ping of that run. A job id, an order number or a UUID works. If a retry reuses the job id, append the attempt number so each attempt is its own run.
max_duration_secondsHow long a run may go from its start ping without a finish ping before it is recorded as failed. Not set by default, and without it a run that never finishes is never flagged.The slowest run you would still call healthy, plus headroom. The average and p95 duration on the monitor's page show what normal looks like. log pings do not extend the limit.
min_duration_secondsA floor: a run that finishes faster opens a DURATION_LOW warning and the monitor stays up. Not set by default.Below the fastest run that does real work. An import that needs 40 seconds for a normal file and finishes in 1 has usually read an empty one.
failure_toleranceFailure pings in a row before the monitor goes down and the alert is sent. The default is 1.1 for anything that moves money or data. 2 or 3 for a job that retries by itself through known flakiness. Any success on the monitor resets the count, so give each job its own monitor.
PayloadA POST body stored with the ping: an error message, a row count, a few lines of output. Up to 100 KB on Free, 1 MB on Pro and 2 MB above.Send one on failures. A ping with a body costs at least 2 beats, one without costs 1. Leave secrets and customer data out: every project member can read payloads.

What background job monitoring does not cover

A run that reports is all a ping can describe. Six things fall outside that.

A triggered job that stopped being triggered
An event-driven monitor has no schedule, so it stays quiet when the webhooks stop coming or the consumer dies. If silence is the risk, add a heartbeat monitor for the consumer or an uptime monitor for its health endpoint.
What happened inside a run
You get the start, the finish, the duration and whatever the job sent as payload or log pings. Traces, query timings and memory profiles come from an APM.
Whether the work was correct
A success ping means the code reached that line. Send the result count as the payload, or set min_duration_seconds, to catch a run that did nothing.
Queue depth
Pings describe runs, not the jobs still waiting. Queue worker monitoring covers what to pair with them.
Retrying or restarting the job
Drumbeats alerts; it does not run your code. A webhook channel can call an endpoint of yours when an incident opens, and what that endpoint does is up to you.
A ping that never arrived
A finish ping lost on the network leaves the run open, and it is recorded as failed at the time limit. Give pings a timeout and a retry: see production hardening.

Four views into a webhook handler

Four dashboard components fed with sample data: an event-driven monitor on a Stripe webhook handler.

Ping Analytics

Volume and success rate per 30-minute bucket

The chart from the monitor's page: pings per 30-minute bucket, split into successes and failures. Hovering a bucket shows its counts, success rate and average duration.

Stats & thresholds

Four ways a triggered run raises an alert

A triggered job has no schedule, so it cannot be late. What can go wrong is per run: it failed, it never finished, or it finished in a time that makes no sense for the work.

  • FAILED: a /failure ping or a non-zero exit code
  • FAILED: a /start ping with no finish inside max_duration_seconds. The run is failed at the limit and the alert goes out
  • DURATION_HIGH: added when a run finishes after max_duration_seconds, whether or not it was failed first
  • DURATION_LOW: the run finished faster than min_duration_seconds. A warning: the monitor stays up

Workflow monitoring

Track multi-step jobs with /log

Between /start and the finish ping, send a /log ping at each step. Drumbeats lists them under the same run_id, so a run that hangs shows the last step it reached. A log ping never changes the status of the monitor and does not extend the time limit.

# Multi-step workflow: the same run_id on every ping
RUN_ID="invoice-$(uuidgen)"
BASE="https://api.drumbeats.io/v1/ping/MONITOR_ID"
# -m 5 is the timeout. A failed ping must not stop the job.
ping() { curl -fsS -m 5 "$BASE/$1" > /dev/null || true; }

ping "start?run_id=$RUN_ID"
pull_line_items &&
  ping "log?run_id=$RUN_ID&message=Pulled+48+entries" &&
  render_pdf &&
  ping "log?run_id=$RUN_ID&message=PDF+rendered" &&
  upload_to_s3 &&
  ping "log?run_id=$RUN_ID&message=Uploaded" &&
  send_receipt
# The exit code is the event: 0 is a success, anything else a failure.
ping "$??run_id=$RUN_ID"
# Multi-step workflow: the same run_id on every ping
RUN_ID="invoice-$(uuidgen)"
BASE="https://api.drumbeats.io/v1/ping/MONITOR_ID"
# -m 5 is the timeout. A failed ping must not stop the job.
ping() { curl -fsS -m 5 "$BASE/$1" > /dev/null || true; }

ping "start?run_id=$RUN_ID"
pull_line_items &&
  ping "log?run_id=$RUN_ID&message=Pulled+48+entries" &&
  render_pdf &&
  ping "log?run_id=$RUN_ID&message=PDF+rendered" &&
  upload_to_s3 &&
  ping "log?run_id=$RUN_ID&message=Uploaded" &&
  send_receipt
# The exit code is the event: 0 is a success, anything else a failure.
ping "$??run_id=$RUN_ID"
  • Each /log ping costs one beat, so send them at real milestones
  • The message query parameter attaches a short label to the step
  • Longer context goes in a POST body, stored as the payload of that ping
  • The timestamps of the log pings show how long each step took

Ping history

Every ping, with the payload the job sent

Start, success and failure pings, paired by run_id, each with its payload. The history downloads as CSV.

What does it cost?

What background job monitoring costs

A start ping and a finish ping are 2 beats per run, and a failure ping that carries an error message costs at least one beat more. Free includes 200,000 beats a month, about 100,000 runs. A webhook handler that runs 1,000 times a day uses 60,000 beats in a 30-day month.

200,000 beats a month for $0

Pro is $20 a month for 1,000,000 beats, about 500,000 runs. Cron, heartbeat and uptime monitors draw on the same beats, and team seats are unlimited on every plan.

FAQ

Common questions

What is background job monitoring?

Checking, from outside the job, that each run of an async task started, finished and succeeded. An async task is work with no user waiting on it: a webhook handler, an export, a queued job. In practice the task calls a monitoring URL at the start and at the end of every run, and the monitor alerts when a call reports a failure or an expected call does not arrive.

What is a run_id, and do I need one?

A run_id ties the pings of one run together, so that ten concurrent runs of the same task do not close each other. Without it Drumbeats cannot compute the duration of a run or notice one that never finished, and it does not generate one for you. The settings table above says what to use.

How do I monitor a background job that runs once a week or once a month?

With a cron monitor that has the same cron expression as the job, for example 0 6 * * 1 for Mondays at 06:00. The monitor waits for the due time plus the grace period and opens a MISSED incident if no ping came, so a weekly job that silently stopped is reported that week and not found a month later.

Can it detect slow jobs, not only failed ones?

Yes. Send a start ping and a finish ping with the same run_id and Drumbeats computes the duration of each run. A run that is still open at max_duration_seconds is failed and alerts, and a run that finishes after the limit is also flagged DURATION_HIGH. For a job that may be slow without paging anyone, keep the limit generous and read the average and p95 duration on the monitor page.

Does it work with AWS Lambda, Cloud Functions or CI jobs?

Yes, anywhere that can make an outbound HTTPS request. Send the start ping at the top of the handler and the finish ping on every return path, before the function returns, because a serverless runtime may freeze the process afterwards. In CI, use the pipeline run id as the run_id; the GitHub Actions guide has a full example.

Did your last background job finish?

Create an event-driven monitor, add a start and a finish ping to one job, and kill a run half-way to watch the incident open at the time limit.

No credit card required · 50 monitors and 200,000 beats a month free

Keep exploring

What to read next

Related monitoring

  • Queue worker monitoring

    For jobs on Sidekiq, Celery, BullMQ, RQ or Laravel queues: worker heartbeats and the hook that wraps every job.

  • Cron job monitoring

    For jobs on a schedule: an alert when a run does not report by its due time.

  • Heartbeat monitoring

    For processes that should never stop: an alert when the pings do.

  • Incidents & alerts

    Where a failed run goes next: notification groups, acknowledgement and recovery messages.

Free tools

Resources