if some one changes the status to in maintenance and performs a reboot ... prometheus will notice this status update after 3 min and during this time it could generates fake alerts
On Monday, December 13, 2021 at 5:10:51 PM UTC+8 nina guo wrote: > Thank you Brian. > > Let's assume the refresh interval is only 5 secs, but during 5 secs the > state might be changed. For exmaple, the target is changed to > maintence(which is not the state we want to monitor) and start a reboot, > but at this time point, prometheus doesn't know the state change, a fake > alert will also be triggered for the reboot action. But the maintenance > state is not the state we are going to monitor, we don't want to receive > such kind of alerts. So we would like to check if there is a way for > Prometheus to know the target state instantly before triggering an alert. > > On Monday, December 13, 2021 at 4:50:55 PM UTC+8 Brian Candler wrote: > >> You can always read the inventory as often as you like. Read it every 5 >> seconds if you like. Write it to a new file like "targets.new", compare it >> with the old "targets" file, and if they are different rename targets.new >> to targets. Then prometheus will pick it up the new file immediately. >> >> The third solution I gave you (creating silences) might work better for >> you. You can create silences whenever you like, and they take effect >> immediately. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/95de0d51-f0a6-4f13-b444-fd1a85c61567n%40googlegroups.com.
