Guide

Kubernetes integration

Monitor CronJob and Job resources, including images whose command you cannot change.

Wrap the container command. This works for any image that ships curl:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-backup
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: ghcr.io/myorg/backup:latest
              env:
                - name: DRUMBEATS_MONITOR
                  value: "11111111-2222-3333-4444-555555555555"
                - name: JOB_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -uo pipefail
                  API="https://api.drumbeats.io/v1/ping/${DRUMBEATS_MONITOR}"
                  RUN_ID="${JOB_NAME}"

                  curl -sf --max-time 3 "${API}/start?run_id=${RUN_ID}" || true

                  if /usr/local/bin/backup.sh; then
                    curl -sf --max-time 3 "${API}/success?run_id=${RUN_ID}" || true
                  else
                    curl -sf --max-time 3 "${API}/failure?run_id=${RUN_ID}" || true
                    exit 1
                  fi

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-backup
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: ghcr.io/myorg/backup:latest
              env:
                - name: DRUMBEATS_MONITOR
                  value: "11111111-2222-3333-4444-555555555555"
                - name: JOB_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -uo pipefail
                  API="https://api.drumbeats.io/v1/ping/${DRUMBEATS_MONITOR}"
                  RUN_ID="${JOB_NAME}"

                  curl -sf --max-time 3 "${API}/start?run_id=${RUN_ID}" || true

                  if /usr/local/bin/backup.sh; then
                    curl -sf --max-time 3 "${API}/success?run_id=${RUN_ID}" || true
                  else
                    curl -sf --max-time 3 "${API}/failure?run_id=${RUN_ID}" || true
                    exit 1
                  fi

Three details matter here.

The exit 1 after the failure ping preserves the pod's failure state, so Kubernetes' own backoffLimit and restart policy still behave normally. Drop it and Kubernetes thinks the job succeeded.

|| true on each curl keeps a ping outage from failing the job. Note that set -e is deliberately absent, because it would abort before the failure ping.

Using metadata.name as the run ID ties the Drumbeats run to the Kubernetes Job. Job names embed the CronJob's scheduled timestamp, so you can find the pod from the incident.

Handle images you cannot modify#

For a vendor image whose command you cannot change, send start from an init container:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vendor-export
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          initContainers:
            - name: drumbeats-start
              image: curlimages/curl:8.10.1
              env:
                - name: DRUMBEATS_MONITOR
                  value: "66666666-7777-8888-9999-aaaaaaaaaaaa"
                - name: RUN_ID
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              command:
                - sh
                - -c
                - |
                  curl -sf --max-time 3 \
                    "https://api.drumbeats.io/v1/ping/${DRUMBEATS_MONITOR}/start?run_id=${RUN_ID}" || true
          containers:
            - name: workload
              image: vendor/export:1.4.0
              # vendor spec, command unchanged

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vendor-export
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          initContainers:
            - name: drumbeats-start
              image: curlimages/curl:8.10.1
              env:
                - name: DRUMBEATS_MONITOR
                  value: "66666666-7777-8888-9999-aaaaaaaaaaaa"
                - name: RUN_ID
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              command:
                - sh
                - -c
                - |
                  curl -sf --max-time 3 \
                    "https://api.drumbeats.io/v1/ping/${DRUMBEATS_MONITOR}/start?run_id=${RUN_ID}" || true
          containers:
            - name: workload
              image: vendor/export:1.4.0
              # vendor spec, command unchanged

The init container reports the start. Nothing reports the finish, so the run is closed by max_duration_seconds on the Drumbeats side, which means a successful run still shows as hung.

Match the monitor type to the resource#

CronJob pairs with a Cron monitor using the same expression and timezone. Kubernetes CronJobs use the cluster's timezone unless you set spec.timeZone, so set the same value on both sides or the schedules drift apart.

Job, created on demand, pairs with an Event-driven monitor. There is no schedule to enforce, only whether the run finished.

What happens when it breaks#

Situation	What Drumbeats sees	What to do
The command exits non-zero	`failure`, then the pod fails	Nothing. This is the designed path
`activeDeadlineSeconds` fires	A `start` with no finish. The pod is terminated outright	Set `max_duration_seconds` to the same value so the hang surfaces
The pod is evicted or preempted	A `start` with no finish	Same. Server-side detection is the only route
A `backoffLimit` retry runs	A second `start` with a new Job name	Expected. Each attempt is its own run
The image has no `curl`	No pings at all. The container fails to start	Bake `curl` in, or use the init-container pattern
The CronJob is suspended	Repeated `MISSED` incidents	Real signal. Pause the Drumbeats monitor while it is suspended

A distroless or scratch image will not have curl. Check before you ship the manifest, because the failure mode is a container that will not start at all.

How you get alerted#

A failure ping, or a hang detected by max_duration_seconds, opens a FAILED incident once failure_tolerance is reached and pages every notification group on the monitor.

Next#

Monitor types for Cron against Event-driven. Ping API for the endpoints these manifests call. Production hardening for the language-agnostic rules. Alternatives if you are still choosing a vendor.