1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Prometheus Operator / kube-prometheus-stack
kubectl -n monitoring patch prometheus kube-prometheus-stack-prometheus --type merge \
  -p '{"spec":{"enableAdminAPI":true}}'
kubectl -n monitoring port-forward svc/prometheus-operated 9090 &

# create the snapshot (the path is /api/v1, not /api/v2)
curl -s -XPOST http://localhost:9090/api/v1/admin/tsdb/snapshot
# {"status":"success","data":{"name":"20260926T101500Z-3f2b1c9a7d0e4b11"}}

# get it out of the pod with tar over stdout, much faster than kubectl cp
SNAP=20260926T101500Z-3f2b1c9a7d0e4b11
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  tar cf - -C /prometheus/snapshots $SNAP | tar xf - -C ./backup/

Outside Kubernetes it is the same with the --web.enable-admin-api flag on the Prometheus command line and the snapshot under <storage.tsdb.path>/snapshots/.

Why copying the directory is not enough

The TSDB writes to the head (WAL plus the in-memory block) while it compacts two-hour blocks into larger ones and deletes the old ones. A cp -r or an rsync of a running /prometheus catches half-compacted blocks and a WAL that no longer matches its checkpoints; with luck it starts, with less luck you get opening storage failed: … corrupted.

The snapshot is made by Prometheus from the inside: it creates snapshots/<date>-<id>/ with hardlinks to the already compacted blocks (it takes almost no space and a few seconds) and dumps the head as one more block, so the copy is consistent up to the instant of the request. With ?skip_head=true it skips the head and only includes what is compacted, which is enough if you snapshot daily and do not care about the last two hours.

Restoring

A snapshot is a directory of blocks (01HXYZ…/, each with chunks/, index, meta.json, tombstones). Restoring means placing them in the root of the data directory of the target instance:

1
2
3
4

# new instance (StatefulSet or binary) stopped, or at least with an empty data dir
kubectl -n newprom exec prometheus-new-0 -c prometheus -- rm -rf /prometheus/wal /prometheus/01*
tar cf - -C ./backup/$SNAP . | kubectl -n newprom exec -i prometheus-new-0 -c prometheus -- tar xf - -C /prometheus/
kubectl -n newprom delete pod prometheus-new-0     # restart so it reads the blocks

Prometheus loads the blocks at startup and compacts them together with new ones. Blocks with different external labels (another cluster, another replica) mix without trouble; what you cannot do is restore blocks that overlap in time with data that already exists and expect them to merge: they stay duplicated until the next compaction, and queries see both series.

promtool tsdb list /prometheus and promtool tsdb analyze /prometheus <block> tell you which range every block covers and which series weigh the most.

Deleting series and freeing space

The same API removes metrics that should not be there (an exporter with runaway cardinality, data from a test):

1
2

curl -s -XPOST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="noisy-exporter"}&start=2026-09-01T00:00:00Z&end=2026-09-20T00:00:00Z'
curl -s -XPOST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

delete_series only marks (tombstones); the space is not freed until clean_tombstones or the next compaction. Both return 204 with no body.

Traps

• The admin API has no authentication: anyone who reaches port 9090 can wipe the whole database. Enable it, take the snapshot over port-forward, and set it back to false. Never behind an Ingress without auth.
• kubectl cp with thousands of files takes forever and sometimes breaks; the tar over stdout above is a single stream. For snapshots of tens of GB, mount a backup PVC or use a sidecar with mc/rclone to S3.
• Delete old snapshots from /prometheus/snapshots/: they are hardlinks, but they keep alive blocks that retention would already have removed, and the disk grows in a way retention does not explain.
• Thanos or Mimir with a sidecar already upload every block to S3: there the snapshot only matters for the head. The backup is the buckets.
• kube-prometheus-stack: prometheus.prometheusSpec.enableAdminAPI: true in the values if you want the patch to survive the next helm upgrade.

References: TSDB Admin APIs in the Prometheus documentation.

See also

• Grafana with SSO — the other half of the stack.
• ZFS: protect snapshots with hold — if the Prometheus data dir lives on ZFS, filesystem snapshots are another consistent option.