Production hardening

Two rules cover most of this page:

// 1. Every ping has a timeout.  2. No ping can ever throw.
await fetch(url, { signal: AbortSignal.timeout(3_000) }).catch(() => {});

// 1. Every ping has a timeout.  2. No ping can ever throw.
await fetch(url, { signal: AbortSignal.timeout(3_000) }).catch(() => {});

Your job is the thing that matters. Drumbeats is a side channel watching it. A side channel that can hang or crash the thing it watches is worse than no monitoring at all.

The examples below are Node.js and shell for brevity. The same shapes appear in Python, Go, PHP, and Shell.

Give every ping a timeout#

Three seconds is a reasonable cap. The exact number matters far less than having one. Without a timeout, a hung TCP connection can hold your worker for minutes while the operating system works through its retries.

curl -sf --max-time 3 "$API/success" >/dev/null 2>&1 || true

curl -sf --max-time 3 "$API/success" >/dev/null 2>&1 || true

requests.get(url, timeout=3)

requests.get(url, timeout=3)

var client = &http.Client{Timeout: 3 * time.Second}

var client = &http.Client{Timeout: 3 * time.Second}

Never let a ping throw#

Swallow the error. Log it if you like, but do not propagate it.

A failed ping costs you one data point. A ping that raises inside a try block your job did not expect costs you the run. If Drumbeats is unreachable, the next ping recovers the state, and if it stays unreachable the missed window will tell you something is wrong anyway.

Do not call response.raise_for_status() in Python, or check response.ok in Node, on a ping. Those turn a monitoring hiccup into an application error.

Cap the payload before you send it#

const MAX = 20_000;
const raw = err instanceof Error ? (err.stack ?? err.message) : String(err);
const payload = raw.length > MAX ? `${raw.slice(0, MAX)}…` : raw;

const MAX = 20_000;
const raw = err instanceof Error ? (err.stack ?? err.message) : String(err);
const payload = raw.length > MAX ? `${raw.slice(0, MAX)}…` : raw;

Two reasons to truncate yourself rather than let Drumbeats do it.

Drumbeats truncates at your plan's limit silently and keeps the leading bytes. On a long stack trace the leading bytes are the least useful part. Truncating in your own code lets you keep the tail, which is where the actual error usually is.

And beats are metered on what you sent, not what was stored. The formula is 1 + ceil(payload_bytes / 25000), so every payload costs at least one extra beat and a 5 MB dump costs 201 whether or not the plan kept it all. See beats and usage.

Correlate concurrent runs with run_id#

Without run_id, two parallel runs of the same job produce interleaved pings and Drumbeats cannot tell which finish belongs to which start. Duration becomes noise and hung-run detection stops working.

It is required on Event-driven monitors and worth sending everywhere else. Use a UUID, or a domain identifier such as order-${order.id} when you want the dashboard to line up with your own logs.

Plan for the process that never comes back#

The failure ping only fires if your code is still running to send it. These situations skip it entirely:

Set max_duration_seconds on the monitor and send a start ping. Drumbeats then detects the hang from its own side, with no cooperation from the dead process. Without both of those, a job that dies mid-run is invisible until the next scheduled window, and on an Event-driven monitor it is invisible forever.

Send progress on long jobs#

A 30-minute backup that sends one start and one success gives you two points on a timeline. The same backup sending a log every five minutes tells you which phase it died in.

curl -sf --max-time 3 -X POST "$API/log?run_id=$RUN_ID" \
  -H 'Content-Type: application/json' \
  -d '{"payload": "Compressed dump (45MB)"}'

curl -sf --max-time 3 -X POST "$API/log?run_id=$RUN_ID" \
  -H 'Content-Type: application/json' \
  -d '{"payload": "Compressed dump (45MB)"}'

Each log costs a beat plus its payload chunk. A nightly job with five progress logs is around 300 beats a month, which buys a lot of clarity during an incident.

Run one code path across environments#

Three workable approaches:

1. A monitor per environment, ID from an env var. Simple, and the IDs are easy to mix up.
2. The slug endpoint. /v1/s-ping/<project-id>/<slug>/<event> routes by project and slug, so the code is identical everywhere and only the project ID changes. This is the one to reach for on anything non-trivial.
3. No staging monitors at all. Often the right answer. Staging fails constantly and by design, and alerts nobody acts on train people to ignore alerts.

Watch your own wiring#

Add the ping wrapper to the observability you already have. Count pings sent by event type, so a spike in failures shows up in your metrics before anyone reads the Drumbeats alert. Log the ping's response status at debug level, so "why did Drumbeats not alert?" is answerable later. Put the ping inside the same trace span as the work, so you can see where in the run it fired.

Choose what actually pages#

Route one monitor to a group holding a fast channel for active incidents, a durable channel for escalation, and a status page for customers.

Next#

Monitor types for picking the right type before hardening it. Payloads for limits and truncation. Alert logic for the tolerance settings that decide whether any of this pages a human. Alternatives if you are still choosing a vendor.