[prometheus-users] Re: Prometheus Talk with other web UI

Reading this I think there is a simple solution. Lets say you go with the aggressive 5 sec file_sd interval, but you're still worried about a false alert, why not just add a "for" clause in your alerting rules? https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ groups:

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you very much Brian. On Monday, December 13, 2021 at 7:04:33 PM UTC+8 Brian Candler wrote: > I think there's not much point continuing this discussion. Prometheus > does not have exactly what you keep asking for, which is "to check" > something externally before sending an alert to alert

[prometheus-users] Re: Prometheus Talk with other web UI

I think there's not much point continuing this discussion. Prometheus does not have exactly what you keep asking for, which is "to check" something externally before sending an alert to alertmanager. All it can check is what's in its timeseries database at that time. However, you can achieve

[prometheus-users] Re: Prometheus Talk with other web UI

if some one changes the status to in maintenance and performs a reboot ... prometheus will notice this status update after 5 secs and during this time it could generates fake alerts On Monday, December 13, 2021 at 5:10:51 PM UTC+8 nina guo wrote: > Thank you Brian. > > Let's assume the refresh

[prometheus-users] Re: Prometheus Talk with other web UI

if some one changes the status to in maintenance and performs a reboot ... prometheus will notice this status update after 3 min and during this time it could generates fake alerts On Monday, December 13, 2021 at 5:10:51 PM UTC+8 nina guo wrote: > Thank you Brian. > > Let's assume the refresh i

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you Brian. Let's assume the refresh interval is only 5 secs, but during 5 secs the state might be changed. For exmaple, the target is changed to maintence(which is not the state we want to monitor) and start a reboot, but at this time point, prometheus doesn't know the state change, a fa

[prometheus-users] Re: Prometheus Talk with other web UI

You can always read the inventory as often as you like. Read it every 5 seconds if you like. Write it to a new file like "targets.new", compare it with the old "targets" file, and if they are different rename targets.new to targets. Then prometheus will pick it up the new file immediately. T

[prometheus-users] Re: Prometheus Talk with other web UI

Correct the words * there is a target is removed from the inventory On Monday, December 13, 2021 at 4:31:09 PM UTC+8 nina guo wrote: > Thank you Brian. > > Your suggestion is really good. May I have one more question? > > For example, if you read the inventory via Netbox every 5mins, let's > as

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you Brian. Your suggestion is really good. May I have one more question? For example, if you read the inventory via Netbox every 5mins, let's assume, if during the 5mins, there is a state of the target is removed from the inventory, at this time point Prometheus still triggers an fake a

[prometheus-users] Re: Prometheus Talk with other web UI

I've given you three different solutions to this. I'm afraid if you don't like any of them, then maybe a different tool will suit your needs better. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop re

[prometheus-users] Re: Prometheus Talk with other web UI

Sorry missed several words: So we want to have this implement: before triggerring an alert, Prometheus sends a request to the inventory management system to get the current state of the target, if the target is with the state we don't want to monitor, then no alert generated. Can Prometheus hav

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you very much Brian. Our file discovery is happened every 5mins to get the latest target list. We are worry that during this 5 mins if there is one of the target state changes from monitored state to the state which we don't want to monitor, actually we don't want to receive the alert fro

[prometheus-users] Re: Prometheus Talk with other web UI

I forgot to say: if you don't want to code directly to the alertmanager API, there is amtool which gives you a simple command-line frontend: amtool silence add instance=device1 amtool silence expire instance=device1 You should be able to proto

[prometheus-users] Re: Prometheus Talk with other web UI

When using file discovery: as soon as you change the contents, prometheus will pick it up and apply the new settings immediately - there's no need to signal prometheus that it has changed. Or there are other service discovery mechanisms you can use, that your inventory could update. Now, the f

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you for your detailed reply. Yes, currently we use file discovery. But one important thing maybe I misssed is that the status of the target might be changed. That is to say, now the status of the target is the status which we want to monitor, but after several minis, the target is down, w

[prometheus-users] Re: Prometheus Talk with other web UI

You could do that - for example, in your inventory system, you expose the status of the targets as a prometheus exporter endpoint: monitoring_active{instance="foo"} 1 monitoring_active{instance="bar"} 0 Then you scrape this to create a new set of timeseries, and you write your alerting expressi

[prometheus-users] Re: Prometheus Talk with other web UI

The web application is the application which manages all the targets. We want to check with the web application to get the real status of the target. Then if the target is with a "monitored" status, we expect there will be an alert triggered. So can Prometheus send this kind of check/command t

[prometheus-users] Re: Prometheus Talk with other web UI

If you are *testing* a remote web UI: https://github.com/prometheus/blackbox_exporter/ If you want to *collect data* from a remote web UI, then you'll need to scrape the HTML to extract the metrics you want, and convert them into OpenMetrics format. Usually it's better to look for an API that y

[prometheus-users] Re: Prometheus Talk with other web UI

Any one can give us some help on this? On Tuesday, December 7, 2021 at 1:28:38 PM UTC+8 nina guo wrote: > Hi, > > Is there a way for Prometheus to talk with other web UI to get the > real-time status of targets? > -- You received this message because you are subscribed to the Google Groups "Pr