[prometheus-users] Getting old label

2021-01-29 Thread Anup Tripathi
Hi Team, I had given label-"ABC" to node x.x.x.x now I have changed label - "KLM" to node x.x.x.x but in Prometheus I am getting both the label associated with node x.x.x.x. Is there a way to remove label "ABC" from Prometheus. -- You received this message because you are subscribed to the Googl

Re: [prometheus-users] How to specify Tags for timeSeries that represent url with parameters

2021-01-29 Thread Debashish Ghosh
Yes the values will actually have real values. But the number of timeseries will be limited since there will be only few configured values that will be used to create the url.. Sent from my iPhone > On Jan 29, 2021, at 2:42 PM, Julius Volz wrote: > >  > Are you saying that the time series it

Re: [prometheus-users] Re: missing recorded metrics

2021-01-29 Thread Julien Pivotto
Johny, Do you have new lines in your label value? Can you try: label4=~"(?s:.+)" Are you using the latest release? Regards, On 29 Jan 22:54, Julius Volz wrote: > Hi Johny, > > If this is indeed happening the way you describe it, that would be a bug. > The 'label4=~".+"' selector definitely

Re: [prometheus-users] Re: missing recorded metrics

2021-01-29 Thread Julius Volz
Hi Johny, If this is indeed happening the way you describe it, that would be a bug. The 'label4=~".+"' selector definitely should match any series that have a value for label4. What Prometheus version are you using, and could you file a bug report with more details (maybe a minimal reproducible t

[prometheus-users] Re: missing recorded metrics

2021-01-29 Thread Johny
It appears the issue is tied to the use of regex wild card filter in experession expr: my_sample_label{label1="..",label4=~".+"} When I remove the 2nd label, issue goes away. expr: my_sample_label{label1=".."} THis is strange as all underlying series clearly have label4 values. Why is this fa

[prometheus-users] missing recorded metrics

2021-01-29 Thread Johny
I'm having a consistent problem wherein some recorded metrics are missing randomly. The underlying expression evaluates to around 50-100 time series. When I filter the expression to produce fewer number of time series (5-10), then the recorded metrics are available and I don't see the issue. Is

Re: [prometheus-users] How to specify Tags for timeSeries that represent url with parameters

2021-01-29 Thread Julius Volz
Are you saying that the time series itself only has those placeholders in the "uri" label value? Or does it actually have all the real values for the placeholders in there (which may end up being a lot of time series then)? On Fri, Jan 29, 2021 at 6:05 PM Debashish Ghosh wrote: > I have a timese

Re: [prometheus-users] looks like rate/irate/delta/increase are not calculating consistent

2021-01-29 Thread Julius Volz
Yup, thanks! Goes to show just how much to say there is just about this topic :) On Fri, Jan 29, 2021 at 6:29 PM Julien Pivotto wrote: > There are other resources: > > https://www.youtube.com/watch?v=67Ulrq6DxwA ( > https://slideshare.net/brianbrazil/counting-with-prometheus-cloudnativeconkubeco

Re: [prometheus-users] looks like rate/irate/delta/increase are not calculating consistent

2021-01-29 Thread Julien Pivotto
There are other resources: https://www.youtube.com/watch?v=67Ulrq6DxwA ( https://slideshare.net/brianbrazil/counting-with-prometheus-cloudnativeconkubecon-europe-2017 ) https://www.robustperception.io/what-range-should-i-use-with-rate https://grafana.com/go/grafanaconline/prometheus-rate-queries

Re: [prometheus-users] looks like rate/irate/delta/increase are not calculating consistent

2021-01-29 Thread Julius Volz
This actually triggered me to write a blog post about exactly this :) https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates On Fri, Jan 29, 2021 at 2:12 PM Julius Volz wrote: > On Fri, Jan 29, 2021 at 1:56 PM Alex > wrote: > >> First of all thank you very much for your de

[prometheus-users] How to specify Tags for timeSeries that represent url with parameters

2021-01-29 Thread Debashish Ghosh
I have a timeseries (http requests) that has uri as tag and the uri has parameters. How to specify that in prometheus query. I am using the tag as follows : http_server_requests_seconds_bucket{le="1.0",uri="/interopapi/v1/org/{orgId}/facility/{facility}/api/FHIR/{fhirVersion}/{resourceName} }

Re: [prometheus-users] looks like rate/irate/delta/increase are not calculating consistent

2021-01-29 Thread Julius Volz
On Fri, Jan 29, 2021 at 1:56 PM Alex wrote: > First of all thank you very much for your detailed answer. Some things > however are still not clear to me. > > Your drawn rate window is 75s in total (not 60s as I expect it to be) > since it contains 4x 15s samples plus 2x 1/2 step to the next sampl

Re: [prometheus-users] looks like rate/irate/delta/increase are not calculating consistent

2021-01-29 Thread Alex
First of all thank you very much for your detailed answer. Some things however are still not clear to me. Your drawn rate window is 75s in total (not 60s as I expect it to be) since it contains 4x 15s samples plus 2x 1/2 step to the next sample. I would expect the window to be exactly 60s includ

Re: [prometheus-users] max latency value (golang client)

2021-01-29 Thread Rafal Skolasinski
Hi Julius, I think having max latency over the max-age window is good. That’s what I believe we had before in the Java app. Is possible to easily get the value from summary that corresponds to quantile=1 and expose it as separate gauge? I would like to avoid trouble of reinventing the wheel

Re: [prometheus-users] max latency value (golang client)

2021-01-29 Thread Julius Volz
Hi Rafal, That would still be the way to do it. Note that in the Go client library at least, the summaries are computed over a max-age window within which the observations are relevant or the summary output: https://github.com/prometheus/client_golang/blob/b8fcd28885584356c562e9ecad42cbec45f4ce96/

Re: [prometheus-users] systemctl start prometheus failed

2021-01-29 Thread Julius Volz
Ah yes, that's what I meant of course :) "journalctl -u prometheus" On Fri, Jan 29, 2021 at 11:32 AM Антон Зубков wrote: > systemctl doesn't have -u flag, but journalctl have it :) > > пятница, 29 января 2021 г. в 11:24:31 UTC+2, juliu...@promlabs.com: > >> "systemctl -u prometheus" should give

Re: [prometheus-users] systemctl start prometheus failed

2021-01-29 Thread Антон Зубков
systemctl doesn't have -u flag, but journalctl have it :) пятница, 29 января 2021 г. в 11:24:31 UTC+2, juliu...@promlabs.com: > "systemctl -u prometheus" should give you the full log output that > hopefully shows why the startup failed. > > On Fri, Jan 29, 2021 at 10:16 AM guo galen wrote: > >

[prometheus-users] max latency value (golang client)

2021-01-29 Thread Rafal Skolasinski
Hi Everyone, Some time ago we moved one of our applications from Java (instrumented with micrometer library) to Go and used prometheus/golang_client to instrument it. Our old application was in addition to summary quantiles also reporting max latency that was registered. Recently users have

Re: [prometheus-users] systemctl start prometheus failed

2021-01-29 Thread Julius Volz
"systemctl -u prometheus" should give you the full log output that hopefully shows why the startup failed. On Fri, Jan 29, 2021 at 10:16 AM guo galen wrote: > hi, I can't start prometheus with systemctl , thanks for any help. > > 1. vim /etc/systemd/system/prometheus.service > [Unit] > Descript

Re: [prometheus-users] systemctl start prometheus failed

2021-01-29 Thread sayf eddine Hammemi
Can you try running the command directly to see what is the issue ? /opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --web.listen-address="0.0.0.0:9696" --storage.tsdb.path=/opt/prometheus/data --web.console.templates=/opt/prometheus/consoles --web.console.libraries=/opt/prom

[prometheus-users] systemctl start prometheus failed

2021-01-29 Thread guo galen
hi, I can't start prometheus with systemctl , thanks for any help. 1. vim /etc/systemd/system/prometheus.service [Unit] Description=prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple Restart=on-failure ExecStart=/opt/promethe