Drumbeats logo
Drumbeats
Guide

Production hardening

Battle-tested patterns for production Drumbeats integrations. Retries, timeouts, payload limits, observability hooks, and failure isolation.

This page collects the patterns that turn a basic Drumbeats integration into a production-grade one. The language guides show the happy-path wiring; this page covers what happens when the network is slow, the job is flaky, or the payload is huge.

The advice is language-agnostic. Wherever a code sample appears it uses Node.js or shell for clarity — the equivalents in Python, Go, PHP, and Shell are straightforward.

Time-bound every ping

A slow Drumbeats request should never block your job. Cap the HTTP timeout at 3 seconds. The exact number matters less than having one — a worker that hangs because a side-channel observability call is slow is the worst kind of incident.

typescript
await fetch(url, { signal: AbortSignal.timeout(3_000) }).catch(() => {});
await fetch(url, { signal: AbortSignal.timeout(3_000) }).catch(() => {});

The trailing .catch(() => {}) is intentional. A failed ping should never propagate — your job is the priority. If the ping fails, the next one will surface the state. Treat Drumbeats as eventually consistent observability, not as part of the critical path.

Cap payload size

Each ping payload is stored against the run. Drumbeats includes up to 25KB of payload at no extra beat cost; everything beyond costs additional beats in 25KB chunks (see Beats and usage).

Production jobs that error rarely send small payloads but sometimes send 10MB of stack trace. Truncate before sending:

typescript
const MAX = 20_000;
const truncated = raw.length > MAX ? `${raw.slice(0, MAX)}…` : raw;
const MAX = 20_000;
const truncated = raw.length > MAX ? `${raw.slice(0, MAX)}…` : raw;

A 20KB cap leaves headroom for JSON envelope overhead while staying inside the free tier.

Use run_id correlation

Drumbeats correlates start and success / failure pings by the run_id query parameter. Without it, parallel runs of the same job race — success from run A can match start from run B.

For batch jobs that are not parallel, run_id is optional but recommended. The dashboard surfaces each run as a separate record so you can replay history.

For Event-driven monitors handling concurrent messages, run_id is mandatory. Suggested patterns:

  • Time-based: "job-${Date.now()}-${random}"
  • UUID: "job-${randomUUID()}"
  • Domain ID: "order-${order.id}" — natural correlation with your existing logs

Send progress logs for long jobs

A 30-minute backup that sends one start and one success shows up on the timeline as two events. A backup that sends log events every five minutes shows a heartbeat of progress — when the incident timeline becomes a root-cause investigation, the log entries are the most useful artifact you have.

bash
curl -sf --max-time 3 -X POST "$API/log?run_id=$RUN_ID" \
  -H 'Content-Type: application/json' \
  -d '{"payload": "Compressed dump (45MB)"}'
curl -sf --max-time 3 -X POST "$API/log?run_id=$RUN_ID" \
  -H 'Content-Type: application/json' \
  -d '{"payload": "Compressed dump (45MB)"}'

Each log event costs one beat. For a job that runs nightly with five progress logs, that is 150 beats/month — cheap insurance.

Isolate ping failures from job logic

Failure modes to design against:

  • Drumbeats unreachable. Network partition between your worker and api.drumbeats.io. Pings fail; the job should still run. The .catch(() => {}) pattern handles this.
  • Drumbeats slow. The 3-second timeout above handles this. Without a timeout a hung TCP connection can take minutes to fail.
  • Drumbeats returns a 4xx. Likely a stale monitor ID or a malformed run_id. Treat as a logged warning, not a failure of the wrapped function.

The wrappers in the language guides already do this. If you write your own, do not call response.raise_for_status() (Python) or check response.ok (Node.js) on the ping — log the response, do not throw.

Multi-environment via slug

The same code path runs in staging and production. Three options:

  1. One monitor per environment, separate IDs. Pass the ID via env var: DRUMBEATS_MONITOR=.... Simple but ID juggling is error-prone.
  2. Slug-based ping endpoint. /v1/s-ping/<project>/<slug>/<event> routes by project + slug. Code is identical in both environments — the project ID changes. Recommended for non-trivial deployments.
  3. No staging monitors. Sometimes the right answer. Staging is noisy; you do not need Drumbeats to alert on every transient staging failure.

See Ping API: scheduled pings for the slug endpoint shape.

Observe the observer

Add the ping wrapper to your existing observability:

  • Metrics. Count pings_sent_total{event=start|success|failure} per job. A spike in failure pings is a leading indicator before the on-call sees them on Drumbeats.
  • Tracing. Wrap the ping in the same span as the work. The trace shows when in the request lifecycle the ping fired.
  • Logs. Log the ping's response status code at debug level. Use it when investigating "why didn't Drumbeats alert?".

Pair max_duration_seconds with start

If your monitor type supports max_duration_seconds (Cron, Heartbeat, Event-driven), set it. A worker that crashes mid-run never sends failure — only Drumbeats' server-side hang detection catches the outage. Without max_duration_seconds, a hung worker is invisible until the next scheduled window.

What to put on call

Pair Drumbeats with three notification surfaces:

  1. Slack / Telegram for active incidents — interactive, fast acknowledgement.
  2. Email or PagerDuty for escalation — for incidents that stay open past business hours.
  3. Status page — for customers, so they know before they file a ticket. See Status pages.

The same monitor can route to all three via a single notification group.