Re: [prometheus-users] Load metrics from file (or anything else)

Thanks for your answers both! To clarify, I am looking on the Prometheus client side. The metrics are not counters or anything that can be exported straight away, rather based on some "events" (e.g., a file has been changed, etc. -- even in an example like that, I would need the app, after restart,

[prometheus-users] Re: WAL size estimate

In most cases WAL growing graphic is pretty linear, if you're not adding dynamically exporters. So you can check increase per hour and it will be your number. Also this article maybe not very actual on current versions, but giving some highlights on what causing WAL to grow: https://www.robustp

Re: [prometheus-users] WAL size estimate

Awesome! Thank you, got it, will upgrade. On Friday, February 21, 2020 at 1:59:50 PM UTC-8, Dhiman Barman wrote: > > > In 2.15.0 > [ENHANCEMENT] TSDB: WAL size is now used for size based retention > calculation. > https://github.co

Re: [prometheus-users] WAL size estimate

In 2.15.0 [ENHANCEMENT] TSDB: WAL size is now used for size based retention calculation. https://github.com/prometheus/prometheus/pull/5886 On Fri, Feb 21, 2020 at 1:45 PM Florin Andrei wrote: > To store Prometheus data, I have a

[prometheus-users] WAL size estimate

To store Prometheus data, I have a volume with a given, fixed size. I need to make sure usage level stays below 100%. I've already set storage.tsdb.retention.time, but this is not always very effective. Sometimes large data spikes will fill up the volume. I cannot reduce the retention time too

Re: [prometheus-users] Load metrics from file (or anything else)

Why are you trying to persist your metrics? If it's to preserve counters, this is unnecessary. PromQL functions that operate on counters (such as rate) will gracefully handle cases where the counter restarts. On Friday, February 21, 2020 at 9:21:47 AM UTC-8, Stuart Clark wrote: > > Your metrics

[prometheus-users] Re: Relabel Problems - Not Working

Your query looks fine. You should be able to just paste that into the `expr` field of the recording rule. Beyond that, I'd have to see your rule group and recording rule to understand why it isn't working. Preferably with a full error as well. On Friday, February 21, 2020 at 1:10:03 PM UTC-8,

[prometheus-users] Re: Relabel Problems - Not Working

Thanks again Jake. I got that working as long as I paste it into Prometheus UI GRAPH tab or pasted into Grafana - (label_replace(label_replace(saas_health:memory_avail_pct, "prd_pod", "$1", "POD_NAME", "(.*)"), "POD_NAME", "", "POD_NAME", "(.*)")) What I haven't figured out is how to get this

Re: [prometheus-users] Highly available PushGateway

On Fri, 21 Feb 2020 at 18:29, Jake Utley wrote: > We are exploring using the PushGateway to handle metric collection for > batch jobs. I am wondering how we should operating the PushGateways in a > reliable way. > > What do we do if the PushGateway is temporarily unavailable? Should we > assume t

[prometheus-users] Highly available PushGateway

We are exploring using the PushGateway to handle metric collection for batch jobs. I am wondering how we should operating the PushGateways in a reliable way. What do we do if the PushGateway is temporarily unavailable? Should we assume the client needs to retry? What happens if we push metrics

[prometheus-users] Re: Relabel Problems - Not Working

The /metrics endpoint on Prometheus provides metrics about the Prometheus instance itself. It will not include the metrics that Prometheus has scraped. If you want to see those metrics on a Prometheus endpoint, you can look into using the /federate endpoint. https://prometheus.io/docs/prometheus

Re: [prometheus-users] Load metrics from file (or anything else)

Your metrics are being stored in Prometheus so this shouldn't be necessary. On 21 February 2020 17:15:18 GMT, Mircea wrote: > > >Hello, > > >This might be a silly question, but here I am: I would need to enable >persistence for some metrics (looks like the push gateway might be a >no-go >for n

[prometheus-users] Load metrics from file (or anything else)

Hello, This might be a silly question, but here I am: I would need to enable persistence for some metrics (looks like the push gateway might be a no-go for now) and I was thinking to have my app dump the metrics, say into a file (e.g., using the write_to_textfile function) - then when the ap

[prometheus-users] Re: Relabel Problems - Not Working

Thank you Jake! I got the two cases I mentioned working. Now I am having trouble changing labels for recording rules. I noticed you do not see the rule/metric name in the results on a /metrics with all the other metrics.You do see 3 stats for the execution of the rule. The rule/metric

[prometheus-users] Re: Relabel Problems - Not Working

On Wednesday, February 19, 2020 at 2:24:32 PM UTC-6, Kevin Kruppa wrote: > > I have been trying to get metric_relabel_configs to work but have not been > successful so far. > > I have tried to break it down by going very basic to what I need. I > cannot even get the basic relabel to work. > >

Re: [prometheus-users] Alertmanager Slack .Labels.SortedPairs issue

Those I have under labels in rules. On Friday, 21 February 2020 14:34:54 UTC, Simon Pasquier wrote: > > Do you in advance which labels/annotations you want to keep? > > On Fri, Feb 21, 2020 at 10:20 AM Kamil Sebastian > > wrote: > > > > Hi, > > I have Alertmanager configured to send alerts on

Re: [prometheus-users] Alertmanager sent multiple empty mails when template rendering failed

See https://github.com/prometheus/alertmanager/issues/1944 A group_interval parameter of 1sec isn't really recommended and explains why Alertmanager sends emails so frequently. On Tue, Feb 18, 2020 at 1:25 PM Theofanis Pispirigkos wrote: > > Hi , in my case a rendering error caused alertmanager

Re: [prometheus-users] Alertmanager Pagerduty description

are you sure that you've shared the correct configuration? can you check again in the Alertmanager UI status page? On Wed, Feb 19, 2020 at 2:45 AM Kenneth Lim wrote: > I am trying to use following for Pagerduty description: > > > - name: 'pagerduty-notify-techops' > pagerduty_configs: > - se

Re: [prometheus-users] Re: sometimes I just received a resolved email but not firing email

What Chris says: Prometheus will send a resolved alert to Alertmanager. If Alertmanager didn't receive any firing alert for the group, it won't send the resolved notification. You may get more insights by running Alertmanager with --log.level=debug. On Tue, Feb 18, 2020 at 8:07 PM Chris Siebenmann

Re: [prometheus-users] alertmanager instance failure

You need to run Alertmanager instances on different machines and setup HA as described in the README.md [1]. This way your setup will be resilient to (N-1) instances going down. If you want to detect a failure in your monitoring pipeline, you need to setup something like a dead man's snitch integra

Re: [prometheus-users] Is there a way to hide labels in received notifications?

You would need to customize the email notification template and drop the labels you're not interested in. On Thu, Feb 20, 2020 at 6:51 PM Shlomi wrote: > > Hi, > > Currently my Prometheus+Alertmanager deployment is notifying me on Alerts via > emails. > The thing is, within each email i receive

Re: [prometheus-users] Alertmanager Slack .Labels.SortedPairs issue

Do you in advance which labels/annotations you want to keep? On Fri, Feb 21, 2020 at 10:20 AM Kamil Sebastian wrote: > > Hi, > I have Alertmanager configured to send alerts on Slack. However each alert > includes too many information taken from Labels. > > My Alertmanager config: > > tit

[prometheus-users] Alertmanager Slack .Labels.SortedPairs issue

Hi, I have Alertmanager configured to send alerts on Slack. However each alert includes too many information taken from Labels. My Alertmanager config: title: |- [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts. Firing | len }}{{ end }}] {{ .CommonLabels.alertnam