Consistent Prometheus backups: snapshot through the admin API (enableAdminAPI on the operator), copy it out, restore it and delete series
kubectl patch prometheus … enableAdminAPI true; POST http://localhost:9090/api/v1/admin/tsdb/snapshot; kubectl exec … tar cf - -C /prometheus/snapshots NAME | tar xf -. Restore = copy the blocks into the root of the new instance’s data dir. The API has no authentication: port-forward and turn it off when done.