The http_sd_config refresh is going to be a very tiny part of the resource utilisation of Prometheus, although 15 seconds is quite aggressive.
As for the exporters, it depends very much on the scrape interval and the duration of each probe, the type of probe, and number of cores you have. For example: let's say you have a 15 second scrape interval and 10K targets = a new scrape every 1.5ms on average (it spreads them out over the time period) If each blackbox or snmp probe takes 150ms to complete, then you are processing 100 probes concurrently on average. If you have 4 cores, then each core is handling 25 probe goroutines. Most of the time each goroutine will be waiting for network response from the target system. But some probes may be more computationally expensive, e.g. those which involve setting up TLS connections, or SNMP privacy/authentication modes. In short, it sounds to me like it should be fine, but monitor it to be sure. Before doing any sort of sharding, I'd first put blackbox/snmp exporters into separate VMs (i.e. separate from Prometheus itself). That's very simple to implement, and gives you a clearer picture of the resource utilisation of each. On Friday 12 January 2024 at 01:32:17 UTC Elliott Balsley wrote: > I'm curious if anyone has experimented to find out how many targets can > reasonably be scraped by a single instance of blackbox and snmp exporters. > I know Prometheus itself can handle tens of thousands of targets, but I'm > wondering at what point it becomes necessary to split up the scraping. > I'll find out for myself soon enough, I just wanted to check and see if > anyone has tested this already. I'm thinking I would have around 10K > targets for blackbox, and 1K for snmp. > > I'm using http_sd_config with a 15 second refresh interval, so that's > another potential bottleneck I'll have to test. > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3e614743-0e9a-44b2-bb78-5f0d8747d7bfn%40googlegroups.com.
