On Tue, Feb 28, 2023 at 1:45 AM Christoph Anton Mitterer <cales...@gmail.com> wrote:
> Hy Stuart, Julien and Ben, > > Hope you don't mind that I answer all three replies in one... don't > wanna spam the list ;-) > Thanks! > > > > On Tue, 2023-02-21 at 07:31 +0000, Stuart Clark wrote: > > Prometheus itself cannot do downsampling, but other related projects > > such as Cortex & Thanos have such features. > > Uhm, I see. Unfortunately neither is packaged for Debian. Plus it seems > to make the overall system even more complex. > I do not recommend using Debian packages for Prometheus. Debian release cycles are too slow for the pace of Prometheus development. Every release brings new improvements. In the last year we've made improvements to memory use and query performance. Debian also ignores the Go source vendor versions we provide which leads to bugs. You'd be better off running Prometheus using podman, or deploying official binaries with Ansible[0]. [0]: https://github.com/prometheus-community/ansible > > I want to Prometheus merely or monitoring a few hundred nodes (thus it > seems a bit overkill to have something like Cortex, which sounds like a > system for really large number of nodes) at the university, though as > indicated before, we'd need both: > - details data for a like the last week or perhaps two > - far less detailed data for much longer terms (like several years) > > Right now my Prometheus server runs in a medium sized VM, but when I > visualise via Grafana and select a time span of a month, it already > takes considerable time (like 10-15s) to render the graph. > > Is this expected? > No, but It depends on your queries. Without seeing what you're graphing there's no way to tell. Your queries could be complex or inefficient. Kinda like writing slow SQL queries. There are ways to speed up graphs for specific things, for example you can use recording rules to pre-render parts of the queries. For example, if you want to graph node CPU utilization you can have a recording rule like this: groups: - name: node_exporter interval: 60s rules: - record: instance:node_cpu_utilization:ratio_rate1m expr: > avg without (cpu) ( sum without (mode) ( rate(node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"}[1m]) ) ) This will give you a single metric per node that will be faster to render over longer periods of time. It also effectively down-samples by only recording one point per minute. Also "Medium sized VM" doesn't give us any indication of how much CPU or memory you have. Prometheus uses page cache for database access. So maybe your system is lacking enough memory to effectively cache the data you're accessing. > > > > > On Tue, 2023-02-21 at 11:45 +0100, Julien Pivotto wrote: > > We would love to have this in the future but it would require careful > > planning and design document. > > So native support is nothing on the near horizon? > > And I guess it's really not possible to "simply" ( ;-) ) have different > retention times for different metrics? > No, we've talked about having variable retention times, but nobody has implemented this. It's possible to script this via the DELETE endpoint[1]. It would be easy enough to write a cron job that deletes specific metrics older than X, but I haven't seen this packaged into a simple tool. I would love to see something like this created. [1]: https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series > > > > > On Tue, 2023-02-21 at 15:52 +0100, Ben Kochie wrote: > > This is mostly unnecessary in Prometheus because it uses compression > > in the TSDB samples. What would take up a lot of space in an RRD file > > takes up very little space in Prometheus. > > Well right now I scrape only the node-exporter data from 40 hosts at a > 15s interval plus the metrics from prometheus itself. > I'm doing this on test install since the 21st of February. > Retention time is still at it's default. > > That gives me: > # du --apparent-size -l -c -s --si /var/lib/prometheus/metrics2/* > 68M /var/lib/prometheus/metrics2/01GSST2X0KDHZ0VM2WEX0FPS2H > 481M /var/lib/prometheus/metrics2/01GSVQWH7BB6TDCEWXV4QFC9V2 > 501M /var/lib/prometheus/metrics2/01GSXNP1T77WCEM44CGD7E95QH > 485M /var/lib/prometheus/metrics2/01GSZKFK53BQRXFAJ7RK9EDHQX > 490M /var/lib/prometheus/metrics2/01GT1H90WKAHYGSFED5W2BW49Q > 487M /var/lib/prometheus/metrics2/01GT3F2SJ6X22HFFPFKMV6DB3B > 498M /var/lib/prometheus/metrics2/01GT5CW8HNJSGFJH2D3ADGC9HH > 490M /var/lib/prometheus/metrics2/01GT7ANS5KDVHVQZJ7RTVNQQGH > 501M /var/lib/prometheus/metrics2/01GT98FETDR3PN34ZP59Y0KNXT > 172M /var/lib/prometheus/metrics2/01GT9X2BPN51JGB6QVK2X8R3BR > 60M /var/lib/prometheus/metrics2/01GTAASP91FSFGBBH8BBN2SQDJ > 60M /var/lib/prometheus/metrics2/01GTAHNDG070WXY8WGDVS22D2Y > 171M /var/lib/prometheus/metrics2/01GTAHNHQ587CQVGWVDAN26V8S > 102M /var/lib/prometheus/metrics2/chunks_head > 21k /var/lib/prometheus/metrics2/queries.active > 427M /var/lib/prometheus/metrics2/wal > 5,0G total > > Not sure whether I understood meta.json correctly (haven't found a > documentation for minTime/maxTime) but I guess that the big ones > correspond to 64800s? > Yes, Prometheus writes blocks every 2 hours, and then compacts them. It does this 3 blocks at a time so 2h -> 6h -> 18h (64800s). With a longer retention time, the compactions will increase up to 21 days, which improves index size efficiency. So the efficiency should improve if you update your retention to a longer time. > > Seem at least quite big to me... that would - assuming all days can be > compressed roughly to that (which isn't sure of course) - mean for one > year one needs ~ 250 GB for that 40 nodes or about 6,25 GB per node > (just for the data for node exporter with a 15s interval). > Without seeing a full meta.json and the size of the files in one dir, it's hard to say exactly if this is good or bad. It depends a bit on how many series/samples are in each block. Just guessing, it seems like you have about 2000 metrics per node. Seems reasonable, we're only talking about 2TiB per year for all 300 of your servers. Seems perfectly reasonable to me. > Does that sound reasonable/expected? > > > > > What's actually more > > difficult is doing all the index loads for this long period of time. > > But Prometheus uses mmap to opportunistically access the data on > > disk. > > And is there anything that can be done to improve that? Other than > simply using some fast NVMe or so? > NVMe isn't necessary. As I said above, memory for computing results is far more important. Prometheus needs page cache space. You want to avoid reading and swapping out the data needed for a graph. > > > > Thanks, > Chris. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoSYBNZ2gZnXdy9nq39MLGeG9WfUBKdefcyFZoxYLFguQ%40mail.gmail.com.
