hai-ben commented on PR #2402: URL: https://github.com/apache/activemq/pull/2402#issuecomment-5346410772
@graben on the lazy cache I did some reading and some tests. [Prometheus recommends](https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling): > Metrics should only be pulled from the application when Prometheus scrapes them, exporters should not perform scrapes based on their own timers. That is, all scrapes should be synchronous. The logic being that it's whoever's scraping Prometheus's responsibility to manage time-stamps. I also did some perf testing to see what affect this would actually have on a real broker using worst-practices. I spun up an m5.xlarge broker and put some queues on it (1_000, 5_000, and 25_0000). Callers would scrape Prometheus every 15s. I tried with 1, 5, and 20 callers. I also ran this under no-load and a load of ~2k msg/s. The best practice is to only touch this endpoint from one caller once every 30-60s and only touch the per-objects when you really need them. Here are the results: <img width="1180" height="1050" alt="PrometheusTest" src="https://github.com/user-attachments/assets/83d11c6e-0df0-48ce-979f-5b773365c602"; /> The CPU load is noticeable, but even with 20 concurrent scrapers on 25k queues with load, scrape times were well under 10s. Importantly, the CPU spikes on 20 concurrent scrapers weren't much worse than 1 or 5 scrapers. There are also a lot of edge cases to handle with a lazy cache (deleted queues, race conditions with concurrent scrapes, etc) and it would not provide much benefit against worst practices. And a scheduled cache also puts this non-zero load on the broker, even if it's not being used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] For further information, visit: https://activemq.apache.org/contact
