Re: [prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-29 Thread Brian Candler
> Given the limitations around ingestion of historical data maybe remote > write to Victoria Metrics would work better for us. > Yes, VictoriaMetrics works well for this application. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To un

Re: [prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-27 Thread James Peverill
Is data that is kept in the WAL considered "historical" in this context? I was reading that the WAL drops data after ~2 hours. The way our system works, hosts may be disconnected for a while during logging. Typically it is only 30-60 minutes, but it can be longer. Is the 2 hour age limit chan

Re: [prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-23 Thread Brian Candler
No, you definitely *cannot* use pushgateway for this. Pushgateway is only for recording the *last* metric written. It cannot record a history of measurements; neither can prometheus import historical measurements (by scraping or any other means). -- You received this message because you are s

Re: [prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-22 Thread Laurent Dumont
I wonder if you could use a Push Gateway to which all the spotty hosts would write to. They could write locally the metrics to a file and keep trying to push the metrics once they have the connectivity to the central Prometheus. On Sat, Aug 22, 2020 at 1:43 PM Brian Candler wrote: > Another opti

Re: [prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-22 Thread Brian Candler
Another option to look at for the remote write target is Victoria Metrics. I like this because it's a trivial single-binary install. You can point multiple prometheus instances at it with remote_write protocol (as well as various other write protocols and formats), and you can query it directl

Re: [prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-21 Thread Ben Kochie
Yes, the only good option to retain data in remote nodes like this is to have Prometheus running in the remote location. If it's one-off nodes, you'll need a localhost Prometheus. The good news is, this is relatively efficient, as the minimum footprint of Prometheus is pretty small. As for how to

[prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-21 Thread James Peverill
I have an application where I am trying to aggregate metrics from hosts that have spotty connectivity. I want to be logging locally when the hosts are periodically offline, then sync all the metrics to a central server once they connect again. Another complication is that these remote hosts som