Hy Stuart, Julien and Ben, Hope you don't mind that I answer all three replies in one... don't wanna spam the list ;-)
On Tue, 2023-02-21 at 07:31 +0000, Stuart Clark wrote: > Prometheus itself cannot do downsampling, but other related projects > such as Cortex & Thanos have such features. Uhm, I see. Unfortunately neither is packaged for Debian. Plus it seems to make the overall system even more complex. I want to Prometheus merely or monitoring a few hundred nodes (thus it seems a bit overkill to have something like Cortex, which sounds like a system for really large number of nodes) at the university, though as indicated before, we'd need both: - details data for a like the last week or perhaps two - far less detailed data for much longer terms (like several years) Right now my Prometheus server runs in a medium sized VM, but when I visualise via Grafana and select a time span of a month, it already takes considerable time (like 10-15s) to render the graph. Is this expected? On Tue, 2023-02-21 at 11:45 +0100, Julien Pivotto wrote: > We would love to have this in the future but it would require careful > planning and design document. So native support is nothing on the near horizon? And I guess it's really not possible to "simply" ( ;-) ) have different retention times for different metrics? On Tue, 2023-02-21 at 15:52 +0100, Ben Kochie wrote: > This is mostly unnecessary in Prometheus because it uses compression > in the TSDB samples. What would take up a lot of space in an RRD file > takes up very little space in Prometheus. Well right now I scrape only the node-exporter data from 40 hosts at a 15s interval plus the metrics from prometheus itself. I'm doing this on test install since the 21st of February. Retention time is still at it's default. That gives me: # du --apparent-size -l -c -s --si /var/lib/prometheus/metrics2/* 68M /var/lib/prometheus/metrics2/01GSST2X0KDHZ0VM2WEX0FPS2H 481M /var/lib/prometheus/metrics2/01GSVQWH7BB6TDCEWXV4QFC9V2 501M /var/lib/prometheus/metrics2/01GSXNP1T77WCEM44CGD7E95QH 485M /var/lib/prometheus/metrics2/01GSZKFK53BQRXFAJ7RK9EDHQX 490M /var/lib/prometheus/metrics2/01GT1H90WKAHYGSFED5W2BW49Q 487M /var/lib/prometheus/metrics2/01GT3F2SJ6X22HFFPFKMV6DB3B 498M /var/lib/prometheus/metrics2/01GT5CW8HNJSGFJH2D3ADGC9HH 490M /var/lib/prometheus/metrics2/01GT7ANS5KDVHVQZJ7RTVNQQGH 501M /var/lib/prometheus/metrics2/01GT98FETDR3PN34ZP59Y0KNXT 172M /var/lib/prometheus/metrics2/01GT9X2BPN51JGB6QVK2X8R3BR 60M /var/lib/prometheus/metrics2/01GTAASP91FSFGBBH8BBN2SQDJ 60M /var/lib/prometheus/metrics2/01GTAHNDG070WXY8WGDVS22D2Y 171M /var/lib/prometheus/metrics2/01GTAHNHQ587CQVGWVDAN26V8S 102M /var/lib/prometheus/metrics2/chunks_head 21k /var/lib/prometheus/metrics2/queries.active 427M /var/lib/prometheus/metrics2/wal 5,0G total Not sure whether I understood meta.json correctly (haven't found a documentation for minTime/maxTime) but I guess that the big ones correspond to 64800s? Seem at least quite big to me... that would - assuming all days can be compressed roughly to that (which isn't sure of course) - mean for one year one needs ~ 250 GB for that 40 nodes or about 6,25 GB per node (just for the data for node exporter with a 15s interval). Does that sound reasonable/expected? > What's actually more > difficult is doing all the index loads for this long period of time. > But Prometheus uses mmap to opportunistically access the data on > disk. And is there anything that can be done to improve that? Other than simply using some fast NVMe or so? Thanks, Chris. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/45f21aedf2412705809fc69522055ca82b2f95f2.camel%40gmail.com.
