Find out current retention period for monitoring servers and ensure that data is kept for 5 years
- Looking at monitoring.place6 I see data for less than 90 days.
- My expectation is to be able to zoom out to 5 years so that we can see changes we did over years
- My assumption is that prometheus is configured with some storage size and it auto drops/rotates
- So probably the solution is just to add/extend the hdd space on monitoring.placeX by a factor of Y
Updated by Dominique Roux over 2 years ago
it's configured in __dcl_monitoring_server type
... __prometheus_server \ --config "$prometheus_config" \ --retention-days 30 \ --rule-files "$__type/files/*.rules" \ --storage-path /data/prometheus ...
on the __prometheus_server you see:
... FLAGS="--storage.tsdb.path $storage_path --storage.tsdb.retention $((retention_days*24))h --web.listen-address [::]:9090" ...
See here for the configuration
I suggest to use the flag --storage.tsdb.retention.size 90GB (not sure if 90GB works) and get rid of the --storage.tsdb.retention flag
Here some interesting information about prometheus storage handling https://prometheus.io/docs/prometheus/1.8/storage/