Monitoring

Consistent Prometheus backups: snapshot through the admin API (enableAdminAPI on the operator), copy it out, restore it and delete series

kubectl patch prometheus … enableAdminAPI true; POST http://localhost:9090/api/v1/admin/tsdb/snapshot; kubectl exec … tar cf - -C /prometheus/snapshots NAME | tar xf -. Restore = copy the blocks into the root of the new instance’s data dir. The API has no authentication: port-forward and turn it off when done.

Grafana with SSO: logging in with the local user (?disableAutoLogin), roles overwritten on every login (skip_org_role_sync) and provisioned data sources with secrets

/login?disableAutoLogin=true to reach the local form; grafana cli admin reset-admin-password if you also forgot the password. skip_org_role_sync: true per provider so Grafana stops overwriting roles; or role_attribute_path with role_attribute_strict. Data sources in provisioning/datasources with editable: false, secrets in secureJsonData, removal with deleteDatasources or prune: true.