Re: [prometheus-users] Alerts Description and Summary

[sayf.eddi...@gmail.com]

well yeah but I cant guarantee they understand the alerting format. also 
product operator can access the documentation too.

On Tuesday, March 28, 2023 at 2:14:29 PM UTC+2 Brian Candler wrote:

> If it's only for consumption by developers reading the rules files, then 
> would YAML comments be sufficient?
>
> On Monday, 27 March 2023 at 15:00:47 UTC+1 sayf.eddi...@gmail.com wrote:
>
>> Thanks for the response
>>
>> By generating automatic documentation I meant automatically creating 
>> developer documentation of the existing alerts and their descriptions from 
>> the yml files rather than runtime information.
>> which means the labels will not be replaced, so the less labels the more 
>> readable the result.
>>
>> I am considering the addition of an extra annotation field for that 
>> purpose I think it is better to separate the concerns here.
>>
>> On Monday, March 27, 2023 at 3:46:11 PM UTC+2 Stuart Clark wrote:
>>
>>> On 2023-03-27 14:43, sayf.eddi...@gmail.com wrote: 
>>> > Hello, I have looked online and I cant find any best practices for 
>>> > filling up the description and the summary. from the examples I see 
>>> > that Summary should be the shortest (plus the minimum usage of 
>>> > labels). But maybe it is an observation bias. 
>>> > 
>>> > I am trying to generate some automatic documentation around alerting 
>>> > and having a lot of labels makes it as user friendly as reading the 
>>> > yaml file directly 
>>> > 
>>>
>>> It really depends how you are wanting to use those. If you are wanting 
>>> to use the summary in an email's subject line then you probably want it 
>>> to be fairly short for example. You can have as many labels/annotations 
>>> as you like, so you don't even have to have one called "summary" if you 
>>> don't want to, and there's nothing stopping you from having much more 
>>> specific labels (e.g. severity, service, environment) which you can then 
>>> include in email/ticket subjects. 
>>>
>>> -- 
>>> Stuart Clark 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/916cea26-f30a-4a18-95b1-0fd6f59bdecan%40googlegroups.com.

Re: [prometheus-users] Alerts Description and Summary

[sayf.eddi...@gmail.com]

Thanks for the response

By generating automatic documentation I meant automatically creating 
developer documentation of the existing alerts and their descriptions from 
the yml files rather than runtime information.
which means the labels will not be replaced, so the less labels the more 
readable the result.

I am considering the addition of an extra annotation field for that purpose 
I think it is better to separate the concerns here.

On Monday, March 27, 2023 at 3:46:11 PM UTC+2 Stuart Clark wrote:

> On 2023-03-27 14:43, sayf.eddi...@gmail.com wrote:
> > Hello, I have looked online and I cant find any best practices for
> > filling up the description and the summary. from the examples I see
> > that Summary should be the shortest (plus the minimum usage of
> > labels). But maybe it is an observation bias.
> > 
> > I am trying to generate some automatic documentation around alerting
> > and having a lot of labels makes it as user friendly as reading the
> > yaml file directly
> > 
>
> It really depends how you are wanting to use those. If you are wanting 
> to use the summary in an email's subject line then you probably want it 
> to be fairly short for example. You can have as many labels/annotations 
> as you like, so you don't even have to have one called "summary" if you 
> don't want to, and there's nothing stopping you from having much more 
> specific labels (e.g. severity, service, environment) which you can then 
> include in email/ticket subjects.
>
> -- 
> Stuart Clark
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8e2f11b4-547f-4647-9cf0-ad6e39088dccn%40googlegroups.com.

[prometheus-users] Alerts Description and Summary

[sayf.eddi...@gmail.com]

Hello, I have looked online and I cant find any best practices for filling 
up the description and the summary. from the examples I see that Summary 
should be the shortest (plus the minimum usage of labels). But maybe it is 
an observation bias. 

I am trying to generate some automatic documentation around alerting and 
having a lot of labels makes it as user friendly as reading the yaml file 
directly

Regards

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/14525699-58bb-4ece-b76a-43ea92878400n%40googlegroups.com.

[prometheus-users] Re: Add parameters in rules files expression

[sayf.eddi...@gmail.com]

FWIW, we use saltstack based jinja2 templating with a configured systemd 
reload (ExecReload=/bin/kill -HUP $MAINPID) to update the thresholds, and 
it works fine


On Tuesday, February 28, 2023 at 9:31:12 AM UTC+1 Brian Candler wrote:

> After changing the rules file, you'll need to trigger a prometheus reload, 
> either by hitting a special API endpoint or sending it a SIGHUP. See 
> https://prometheus.io/docs/prometheus/latest/management_api/#reload
>
> (Note: *don't* restart the prometheus process! That's a much slower 
> operation as it has to dump out and reload its WAL and can take several 
> minutes on a busy server)
>
> On Tuesday, 28 February 2023 at 06:55:24 UTC inmar tzemach wrote:
>
>> Thanks!
>>
>> How can I change the threshold from outside for giving expression while 
>> Prometheus running? 
>>
>> On Monday, February 27, 2023 at 5:06:27 PM UTC+2 Brian Candler wrote:
>>
>>> No, but you can expand the rules files using a preprocessor of your 
>>> choice.
>>>
>>> If you want the threshold to be different for different time series, 
>>> then see 
>>> https://www.robustperception.io/using-time-series-as-alert-thresholds
>>>
>>> On Monday, 27 February 2023 at 13:58:47 UTC inmar tzemach wrote:
>>>
 Is it possible to add parameters in rules files expression?
 Something like:
 expr: (100 - (avg by (instance) 
 (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > {{ $treshhold}}

 Thanks

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e3c57e07-094c-4d71-af0e-c9b0e0b34378n%40googlegroups.com.

Re: [prometheus-users] Document recording rules

[sayf.eddi...@gmail.com]

Done https://github.com/prometheus/prometheus/issues/11969

Thanks

On Monday, February 13, 2023 at 3:37:12 PM UTC+1 Ben Kochie wrote:

> That's a very interesting idea. I don't think that's possible right now. 
> But maybe file a feature request on github?
>
> On Mon, Feb 13, 2023 at 3:27 PM sayf.eddi...@gmail.com <
> sayf.eddi...@gmail.com> wrote:
>
>> Hello,
>>
>> I am working on adding recording rules. But I dont see how to document 
>> them in the code.
>> We have `HELP` entry for metrics, `description` and `summary` for alerts. 
>> what about recording rules?
>>
>> Best regards
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/9cb12679-6585-4340-bd28-b045f5237fcdn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/prometheus-users/9cb12679-6585-4340-bd28-b045f5237fcdn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/bd5701da-750e-4ddb-9559-c241b549201bn%40googlegroups.com.

[prometheus-users] Document recording rules

[sayf.eddi...@gmail.com]

Hello,

I am working on adding recording rules. But I dont see how to document them 
in the code.
We have `HELP` entry for metrics, `description` and `summary` for alerts. 
what about recording rules?

Best regards

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9cb12679-6585-4340-bd28-b045f5237fcdn%40googlegroups.com.

[prometheus-users] Re: passwords in config file

[sayf.eddi...@gmail.com]

If you have control on the version of systemd you can update and use the 
credentials module https://systemd.io/CREDENTIALS/

On Friday, January 6, 2023 at 7:06:34 PM UTC+1 natach...@gmail.com wrote:

> Aaah, that is lovely. Thank you so much for pointing me in the right 
> direction.
>
> On Sunday, December 18, 2022 at 4:35:02 PM UTC-3 Brian Candler wrote:
>
>> It's pretty simple. You point password_file at a file containing the 
>> password; and you use Unix permissions to ensure that this file is readable 
>> only by the prometheus process (i.e. the userid that prometheus runs as).
>>
>> If you are using Kubernetes, it has the ability to expose "secrets" at a 
>> specific path in the filesystem, so you could point to one of those.
>>
>> Certainly, if someone breaks into the system as 'root' or the prometheus 
>> user, they'll be able to read the secret. But that's pretty much a 
>> requirement, since the prometheus process itself needs to know the secret.
>>
>> On Sunday, 18 December 2022 at 13:56:12 UTC natach...@gmail.com wrote:
>>
>>> Hi Brian,
>>>
>>> Yes, that's what I meant. But I also have some concerns about 
>>> password_file, can you recommend some strategies I can study to use it 
>>> securely? 
>>> I've been trying to find it online for a few days before asking here, 
>>> but without success.
>>>
>>> On Saturday, December 17, 2022 at 6:53:03 AM UTC-3 Brian Candler wrote:
>>>
 If you're talking about basic_auth in scrape jobs 
 ,
  
 then use password_file instead of password.

 Otherwise, please clarify, or give an example of the embedded 
 username+password config you're talking about.

 On Saturday, 17 December 2022 at 08:49:30 UTC natach...@gmail.com 
 wrote:

> Hey guys,
>
> I'm looking for some best practices advice for securing my prometheus 
> stack, because I don't wanna have username+password for my targets in my 
> prometheus.yml file
>
> I've looked for environment variables because this is one way that I 
> know of, and that turned out to be a huge discussion and a dead end. 
>
> So what is you recommendation? What should I study/do ?
>
> Regards,
> Nat
>


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/bcdd0884-7820-486e-8fc3-12bb727ce053n%40googlegroups.com.

Re: [prometheus-users] Prometheus agent mode HA pushing to remote write

[sayf.eddi...@gmail.com]

I faced this issue in the design phase too, found this 
https://github.com/beorn-/highlander but it looks a bit dead now...
else you can push duplicated metrics to thanos (while setting a unique 
label for every prometheus instance) and then configure thanos querier to 
deduplicate when serving metrics
On Friday, November 4, 2022 at 3:40:29 PM UTC+1 Julien Pivotto wrote:

> You would need additional tools like corosync or keepalived to achieve
> this. In prometheus terms, all the prometheis servers are active and
> deduplication is done at the remote write end if needed.
>
> On 04 Nov 07:24, Navid Shakibapour wrote:
> > Is there a way to set up multiple Prometheus servers in an 
> active/passive 
> > way? Meaning only 1 server scrapes targets at a time and other ones only 
> > scrape if the active one doesn't "work" after x seconds/minutes.
> > 
> > On Friday, November 4, 2022 at 3:00:15 AM UTC-4 sup...@gmail.com wrote:
> > 
> > > There is no option without using a single instance. If your 3rd party 
> > > solution does not handle HA, you can not use HA.
> > >
> > > On Fri, Nov 4, 2022 at 7:57 AM Navid Shakibapour <
> navid.sh...@gmail.com> 
> > > wrote:
> > >
> > >> We are evaluating using Prometheus agent mode in a highly available 
> > >> manner to forward metrics to a 3rd party remote storage using remote 
> write 
> > >> API.
> > >> We need to have multiple shards and multiple replicas per shard. Our 
> 3rd 
> > >> party remote storage does not handle deduplication on their side.
> > >>
> > >> Could someone point me to different options for metrics deduplication 
> on 
> > >> ingestion?
> > >>
> > >>
> > >> -- 
> > >> You received this message because you are subscribed to the Google 
> Groups 
> > >> "Prometheus Users" group.
> > >> To unsubscribe from this group and stop receiving emails from it, 
> send an 
> > >> email to prometheus-use...@googlegroups.com.
> > >> To view this discussion on the web visit 
> > >> 
> https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com
>  
> > >> <
> https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> > >> .
> > >>
> > >
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/91c8b0d4-67e2-40d3-8314-bd58409cf937n%40googlegroups.com
> .
>
>
> -- 
> Julien Pivotto
> @roidelapluie
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/5cfbf874-6449-4215-bcdd-3c856fa68e7dn%40googlegroups.com.

Re: [prometheus-users] PostgreSQL To Prometheus

[sayf.eddi...@gmail.com]

As a rule of thumb always start by looking in this page
https://prometheus.io/docs/instrumenting/exporters/
On Sunday, October 23, 2022 at 11:03:33 AM UTC+2 olivier...@outscale.com 
wrote:

> Hello Giorgi ! 
>
> You have multiple SQL / query exporters that works with Psql, MySQL and 
> even SQLite and can import data the proper way in Prometheus. 
>
> As an example : 
> https://github.com/free/sql_exporter
>
> Have a good day ! 
>
> Olivier 
>
> Le dim. 23 oct. 2022 à 11:00, giorgi jalabadze  a 
> écrit :
>
>> Hello, is there way to export postgreSQL data to prometheus?
>>
>> Like I have data in postgres for example Bitrates and start time, how can 
>> I monitor it with Prometheus+Grafana? 
>>
>> Is There any solution?
>>
>>
>> Thanks 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/54e2816c-aca3-447d-9e72-f6e2bf5b5b9fn%40googlegroups.com
>>  
>> 
>> .
>>
> -- 
>  
>
> Olivier Harand
>
> Chief Monitoring Officer
>
> +33012028
> +33783296234
> olivier...@outscale.com
> 319 Bureaux de la Colline - 92210 Saint-Cloud
>
> 
> 
> 
>
> IMPORTANT : The information contained in this message may be privileged 
> and confidential and
> protected from disclosure. If the reader of this message is not the 
> intended recipient, or
> an employee or agent responsible for delivering this message to the 
> intended recipient, you
> are hereby notified that any dissemination, distribution or copying of 
> this communication is
> strictly prohibited. If you have received this communication in error, 
> please notify us immediately
> by replying to the message and deleting it from your computer.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/75c75936-c57c-4639-acf0-7408f05f3f6fn%40googlegroups.com.

Re: [prometheus-users] Prometheus and Alertmanager behind Apache Proxy

[sayf.eddi...@gmail.com]

I use the following jinja template to expose alertmanagers on 
`/alertmanager`


























* 
Alertmanager###Header 
add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" 
env=BALANCER_ROUTE_CHANGED {%- for member 
in members %} BalancerMember http://{{ member }}:{{ port }} route={{ 
loop.index }} timeout=5 {%- endfor %} ProxySet lbmethod=bybusyness ProxySet 
stickysession=ROUTEIDProxyPass /alertmanager 
balancer://alertmgrclusterProxyPassReverse /alertmanager 
balancer://alertmgrcluster*
On Saturday, October 22, 2022 at 2:54:41 PM UTC+2 wpjo...@gmail.com wrote:

> Wow thanks for the fast response!
>
>
> It looks like his now:
> ProxyPass   / http://localhost:9090/
> ProxyPassReverse/ http://localhost:9090/
> ProxyPass   / http://localhost:9093/
> ProxyPassReverse/ http://localhost:9093/
>
> but does not work either.
>
> michael@azuleng.com schrieb am Samstag, 22. Oktober 2022 um 14:48:04 
> UTC+2:
>
>> Get rid of the /alertmanager in the proxypass statements and you should 
>> be good. 
>>
>> Get Outlook for iOS 
>> --
>> *From:* promethe...@googlegroups.com  on 
>> behalf of wpjo...@gmail.com 
>> *Sent:* Saturday, October 22, 2022 8:46:12 AM
>> *To:* Prometheus Users 
>> *Subject:* [prometheus-users] Prometheus and Alertmanager behind Apache 
>> Proxy 
>>  
>> Hi, I have Prometheus and Alertmanager running behind an Apache Proxy.
>>
>> My problem is that i cannot make Alertmanager accessible through
>> "https://my.url.com/alertmanager";
>> With my current config i only get an 404 for that site.
>>
>> I start Alertmanager with following flags:
>>
>> ExecStart=/etc/alertmanager/alertmanager 
>> --config.file=/etc/alertmanager/alertmanager.yml 
>> --storage.path=/etc/alertmanager/data --cluster.listen-address= 
>> --web.config.file=/etc/prometheus/web.yml --web.external-url=
>> https://my.url.com/alertmanager --web.route-prefix=/alertmanager
>>
>> and my Proxy looks like this:
>> 
>> ...
>> ProxyPass   / http://localhost:9090/
>> ProxyPassReverse/ http://localhost:9090/
>> ProxyPass   /alertmanager http://localhost:9093/alertmanager/
>> ProxyPassReverse/alertmanager http://localhost:9093/alertmanager/
>> 
>>
>> please send some help!!!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/98a2def6-78ee-41b4-a192-10ba117888bcn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7b855de4-3a50-4573-bd5b-cadb5e56f269n%40googlegroups.com.

Re: [prometheus-users] Use Case Assessment

[sayf.eddi...@gmail.com]

Hello,
Monitoring the health of the system with Prometheus is fine. but I think 
you are trying to include it as a functional brick in the application, 
which I am not very keen with. imo The monitoring system should not coupled 
with the functionning of your system (as in your system should continue to 
work fine if Prometheus is down for exp).
You need sth else like issuing events and alert on them (there you are free 
to focus on the payementID info)

On Monday, October 17, 2022 at 9:35:17 AM UTC+2 Rishabh Arora wrote:

> Thank you for the clarification, Stuart.
>
> On Monday, 17 October 2022 at 12:50:57 UTC+5:30 Stuart Clark wrote:
>
>> On 17/10/2022 07:26, Rishabh Arora wrote:
>>
>> Hello!
>>
>> I'm currently in the process of implementing Prometheus along with 
>> Alertmanager as our de facto solution for node health monitoring. We have a 
>> kubernetes, kafka, mqtt setup and for monitoring our infrastructure, 
>> prometheus is an obvious good fit.
>>
>> We have an application / business case, where I'm wondering whether 
>> Prometheus may be a reasonable solution. Our application needs to meet 
>> certain SLAs. In case those SLAs are not being, some alerts need to be 
>> firing. For example, consider the following case which bears close 
>> resemblance to our real business case:
>>
>> An *Order* schema in our system has a *payment* field which can be one 
>> of ['COMPLETED','FAILED','PENDING']. In our HA real time system, we need to 
>> fire alerts for Orders which are in a PENDING state. Rows in our *Orders* 
>> collection 
>> will be in the order of potentially millions. An order also has a 
>> *paymentEngine* field, which represents the entity responsible for 
>> processing the payment for the order.
>>
>> Now, with Prometheus, finding the total count of PENDING Orders would be 
>> a simple metric, but what we're interested in is also the Order IDs. For 
>> instance, is there a way I could capture the PENDING order IDs in the 
>> "metadata"(???) or "payload" of the metric? Downstream in the alertmanager, 
>> I'd also like to group by *paymentEngine* so I could potentially inhibit 
>> alerts for an unstable engine.
>>
>> Can anyone please help me out? Apologies in advance for my naivety :)
>>
>> What you are asking for isn't really the job of Prometheus.
>>
>> Having a metric detailing the number of pending orders & alerting on that 
>> is completely within the normal area for Prometheus & Alertmanager - 
>> observing the system and alerting if there are issues that need 
>> investigation. However the next step of dealing with the individual 
>> events/orders is the job for a different system. If paymentEngine could be 
>> a small number of options (e.g. PayPal, Swipe, Cash) then it would be 
>> reasonable to have that as a label to the pending orders metric (which then 
>> would allow you to alert if one method stops working), but order ID isn't 
>> something you should ever put in the metrics. Instead once you were alerted 
>> about a potential issue you might query your order database directly or 
>> look at log files to dig into the detail and figure out what is happening.
>>
>> -- 
>> Stuart Clark
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/91f5d8c2-d1f4-42df-bf9e-ccfae3f6d9b0n%40googlegroups.com.

[prometheus-users] Documentation standards?

[sayf.eddi...@gmail.com]

Hello
We have a large product with multiple exporters within different teams.

I was wondering if there is a standard on how to document each 
application/exporter metrics and alerts. If not, Did you already encounter 
such issue and what direction did you choose.

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8235179d-e1e1-47ae-8914-3ae222407387n%40googlegroups.com.

[prometheus-users] Re: Simulate alerting rules on historical data

[sayf.eddi...@gmail.com]

Hello,
No I am not aware of such tool, but it shouldnt be hard to write a simple 
exporter (maybe using python prometheus_client lib) to replay historical 
data and expose it to a Prometheus/Alertmanager setup.
Or, given the alerts are also stored in the TSDB, you can build sth that 
navigate the data on time basis and detect when the state of the alert 
changed to "pending" or "firing" and check the thresholds

On Tuesday, September 28, 2021 at 10:20:31 AM UTC+2 wills...@monzo.com 
wrote:

> I'm thinking about ways we can reduce noisy alerts. One of the problems is 
> it's tricky to tweak alert thresholds without any data on the precision and 
> recall of the alert. It's a non-trivial problem to get this data because a 
> human is typically required to classify an alert as a true positive or a 
> false negative. This makes it hard to fully automate gathering this data. I 
> am considering whether there is a way of obtaining this data using a hybrid 
> approach: a human is able to classify an alert as a true positive of false 
> positive - for example via a button in the alert body (e.g. in Slack or 
> PagerDuty) and this gets sent to an analytics database which we can later 
> prioritise which alert thresholds that need tweaking.
>
> My question is, is there any precedent for this kind of system in 
> the Prometheus/Alertmanager ecosystem? i.e. open source software that does 
> this out of the box, or experience report blog posts?
>
> Many thanks,
> Will
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/95d8ce9b-4354-4a5e-b1c2-9abb051ffc66n%40googlegroups.com.

Re: [prometheus-users] Large cluster simulation

[sayf.eddi...@gmail.com]

thanks I ll take a look

On Thursday, September 23, 2021 at 9:47:08 AM UTC+2 sup...@gmail.com wrote:

> Take a look at https://github.com/prometheus/test-infra
>
> This is what we use to benchmark changes and each release.
>
> On Thu, Sep 23, 2021 at 9:28 AM sayf.eddi...@gmail.com <
> sayf.eddi...@gmail.com> wrote:
>
>> Hello,
>> I want to test the behavior of my Prometheus setup (HA, and/or 
>> federation) on large setup (resource consumption, possible crashes, 
>> latencies etc).
>> Are there any tool available for that?
>> I am thinking about using a small number of servers but configure them 
>> multiple times in Prometheus with different "job" label to achieve the 
>> desired cluster size.
>> Do you see any problems in this idea?
>> Thanks
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/fe58c5d7-e5b7-49cc-a5bd-b91a2cd72f9bn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/prometheus-users/fe58c5d7-e5b7-49cc-a5bd-b91a2cd72f9bn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7adbd343-e203-4b96-b425-bcf9f6dd52cfn%40googlegroups.com.

[prometheus-users] Large cluster simulation

[sayf.eddi...@gmail.com]

Hello,
I want to test the behavior of my Prometheus setup (HA, and/or federation) 
on large setup (resource consumption, possible crashes, latencies etc).
Are there any tool available for that?
I am thinking about using a small number of servers but configure them 
multiple times in Prometheus with different "job" label to achieve the 
desired cluster size.
Do you see any problems in this idea?
Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/fe58c5d7-e5b7-49cc-a5bd-b91a2cd72f9bn%40googlegroups.com.

[prometheus-users] Re: Horizontal Pod Autoscaling using Nvidia GPU Metrics

[sayf.eddi...@gmail.com]

Hi,
It depends on how the pods from the same node are sharing the GPU, but I 
think it is doable if you configure the hpa to spawn new pods and the pods 
to `request` GPU resources, this will force the GKE cluster autoscaler into 
creating new nodes to locate the new pods.
Are you using KubeFlow on top of GKE or a homemade platform?

On Saturday, May 1, 2021 at 3:36:37 PM UTC+2 a.gui...@instadeep.com wrote:

> Hi all.
>
> I'm trying to implement HPA based on GPU utilization metrics. 
>
> My initial approach is to use DCGM Exporter which is a daemonset that 
> runs a pod on every GPU node and exports GPU metrics. 
>
> By setting an additional scrape config when installing 
> kube-prometheus-community and a custom rule when installing 
> prometheus-adapter, I'm able to query the prometheus API and get the 
> dcgm_gpu_utilization for each node:
> dcgm_gpu_utilization{Hostname="dcgm-exporter-dmrff", 
> UUID="GPU-e26f8adc-c4aa-4a46-b3d3-ff4599da50a3", device="nvidia0", gpu="0", 
> instance="10.28.0.50:9400", job="gpu-metrics", 
> kubernetes_node="gke-test-hpa-gpu-nodes-0f879509-qth8"} 3
> dcgm_gpu_utilization{Hostname="dcgm-exporter-rxjfm", 
> UUID="GPU-0446c63e-3843-62fa-56db-423958021f5c", device="nvidia0", gpu="0", 
> instance="10.28.1.27:9400", job="gpu-metrics", 
> kubernetes_node="gke-test-hpa-gpu-nodes-0f879509-8bgb"} 0
>
> What I'd like to ask is this: Is it possible to configure HPA for a 
> deployment based on this metric (even though it's being exported for each 
> node through dcgm-exporter pods and not the pods corresponding to the 
> deployment we want to autoscale)?
>
> Perhaps there's a way to generate a metric like mydeploy_gpu_avg which is 
> equal to avg(dcgm_gpu_utilization) over all nodes that have a replica of 
> the deployment mydeploy? That would make it possible to configure HPA 
> with a custom object that targets this mydeploy_gpu_avg metric of mydeploy
> .
>
>
> I hope I'm making sense so far. To my surprise, this is a very rare 
> scenario it seems. Our use-case is autoscaling GPU-based machine learning 
> inference servers, in case it helps to know.
>
>
> I would really appreciate any advice regarding this. I tried to document 
> my current progress in a Github repo: 
> https://github.com/ashrafgt/k8s-gpu-hpa
>
> Thank you.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6f769f7b-b62b-4262-a563-eba82b11ef31n%40googlegroups.com.

[prometheus-users] Re: How to make Prometheus setup via docker-compose reads a certain prometheus.yml file

[sayf.eddi...@gmail.com]

If you have an error in the config file, prometheus will just ignore the 
new config and stays with the old one, check the logs of the container and 
make sure there are not errors

On Friday, April 9, 2021 at 12:22:18 PM UTC+2 shutima.p...@geotwin.io wrote:

> Hello all,
>
> I am new to Prometheus and docker-compose. I have a project structure with 
> docker-compose.yml file that setup Prometheus and Grafana:
> "*/prometheus-grafana/prometheus/docker-compose.yml*"
>
> version: '3'
>
> services:
> prometheus:
> image: prom/prometheus:v2.21.0
> ports:
> - 9000:9090
> volumes:
> - ./prometheus:/etc/prometheus
> - prometheus-data:/prometheus
> command: --web.enable-lifecycle 
> --config.file=/etc/prometheus/prometheus.yml
>
> grafana:
> image: grafana/grafana:$GRAFANA_VERSION
> environment:
> GF_SECURITY_ADMIN_USER: $GRAFANA_ADMIN_USER
> GF_SECURITY_ADMIN_PASSWORD: $GRAFANA_ADMIN_PASSWORD
> ports:
> - 3000:3000
> volumes:
> - grafana-storage:/var/lib/grafana
> depends_on:
> - prometheus
> networks:
> - internal
>
> networks:
> internal:
>
> volumes:
> prometheus-data:
> grafana-storage:
>
> I just added more config in "
> */prometheus-grafana/prometheus/prometheus/prometheus.yml*" like this, 
> the part that I added is yellow highlighted:
>
> global:
> scrape_interval: 30s
> scrape_timeout: 10s
>
> rule_files:
> - alert.yml
>
> scrape_configs:
> - job_name: services
> metrics_path: /metrics
> static_configs:
> - targets:
> - 'prometheus:9090'
> - 'idonotexists:564'
> - job_name: myapp
> scrape_interval: 10s
> static_configs:
> - targets:
> - localhost:2112
>
> I started Prometheus by running: "docker-compose up -d", I got to 
> Prometheus on "http://localhost:9000/graph"; but I don't see the new 
> config I added to "*prometheus.yml*", in docker-compose.yml file, there 
> is a line "*command: --web.enable-lifecycle 
> --config.file=/etc/prometheus/prometheus.yml*" Am I supposed to change 
> this to refers to the path of my another Prometheus.yml in my project?
>
> Thank you in advance.
>
> Rgds,
> Shutima Potivorakun
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/52bcdfe1-52fd-491b-af8d-74a7dffba933n%40googlegroups.com.

[prometheus-users] Re: Couldn't start prometheus server

[sayf.eddi...@gmail.com]

http://www.catb.org/~esr/faqs/smart-questions.html

On Monday, December 14, 2020 at 8:23:45 AM UTC+1 minhlu...@gmail.com wrote:

>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a802044e-da09-4a2a-a9d4-443eb2ac6fc5n%40googlegroups.com.

[prometheus-users] Re: New in this world

[sayf.eddi...@gmail.com]

Hello,
Can you explain what you tried and how did you setup your monitoring system?

On Thursday, October 29, 2020 at 9:06:01 AM UTC+1 jorge.fer...@gmail.com 
wrote:

> Hello, I am a Spanish guy trying to understand this tool! (Sorry for my 
> English) 
>
> My question is... why I can’t see in the api (of alert manager or 
> Prometheus) the alerts that I receive in the alert mail? I just want create 
> an script that can read the api and print the alerts of the day, like just 
> a summary or something like that.
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d7ce7e97-b7fb-4d4e-b36b-8430240c2d56n%40googlegroups.com.

[prometheus-users] Re: Keeping Config Files in sync for HA Prometheus.

[sayf.eddi...@gmail.com]

Hello
1- you can use a CMS tool like ansible to salt to manage prometheus config 
and update it in //
2- alertmanager will take care of deduplicating the alert

On Wednesday, October 14, 2020 at 9:32:20 PM UTC+2 yagyans...@gmail.com 
wrote:

> Hi. I am moving from vanilla Prometheus setup to HA Prometheus setup with 
> 2 instances scraping the same metrics. I'll be using either Thanos or 
> VictoriaMetrics on top of that.
> I have a couple of doubts:
>
> 1. How to manage the config files and keep all the config files in sync in 
> both the Prometheus instance? Will I have to manually update the config 
> files in both the instances every time I change something?
>
> 1. What happens to alerting? Will both the instances have the same 
> alerting rules and keep communicating with the Alertmanager and 
> de-duplication of alerts will be taken care by Alertmanager itself? Or 
> something else has to be done?
>
> Thanks in advance. Sorry if the queries seem repeated. Didn't find the 
> answer for these yet.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6d361d2d-37cd-4ddb-a1cd-bfdd4cb73c55n%40googlegroups.com.

[prometheus-users] Re: Prometheus as long time storage

[sayf.eddi...@gmail.com]

The initial release of Prometheus was in 2012 so no one have tried it yet 
:D. I am not super expert but I think 50 metrics is manageable on such 
retention period (in your formula you forgot to take the labels into 
consideration). But keep in mind prometheus has no data protection schemes 
given it is a very long period I suggest using another storage system 
coupled with prometheus like M3db or victoria metrics

On Tuesday, September 22, 2020 at 2:31:56 PM UTC+2 rodolp...@gmail.com 
wrote:

> Hi everyone ! 
> I'm going to use Prometheus as long time TS storage,
> here's all specifications I would like to have :
> - scrape : every minute
> - retention : 10 years
> - number of metrics : 50
> - Number of point : 262 800 000 (  number of metric *  retentio(min)) 
> I've read that Prometheus is using 1/2 bytes per sample so if we calcul 
> the disk space I need, I'm between 262,8 mo and 525,6  mo.
>
> So what are the problems I may face in my use case ?
>
> Cheers,
> Rodolphe G.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/625c1d93-2769-4890-8eba-58e1135b445en%40googlegroups.com.

[prometheus-users] Re: Multiple exporters installation

[sayf.eddi...@gmail.com]

Hello, the proper way imho is to use a configuration management system like 
saltstack, ansible etc. I would also package those exporters in OS packages 
(rpm, deb) but that is not required given most of them do not have 
dependencies

On Wednesday, September 23, 2020 at 2:05:15 AM UTC+2 kira...@gmail.com 
wrote:

> Hello all
>
> In my use case I want to install multiple exporters at once on each Linux 
> target. What is the best way to bundle the needed exporters and install as 
> a package. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/330a2fca-6ef1-4ac5-a11a-8df5cab664ben%40googlegroups.com.

[prometheus-users] Re: How to write/develop you own custom exporters

[sayf.eddi...@gmail.com]

Hello,
1- start by reading the guidelines and best practices from Prometheus 
website https://prometheus.io/docs/instrumenting/writing_exporters/
Then check the Python prometheus_client (used to write exporters) which is 
the best entry-level tool you can use
2- Before you start writing an exporter always check if there is one doing 
the same (don't reinvent the wheel), for example, your b) point, if you 
want to export metrics from a file there are mtail and grok_exporter.
3- ephemeral jobs (like running an SQL script) cannot monitored directly by 
Prometheus, you need to check https://prometheus.io/docs/practices/pushing/ 
and the tool https://github.com/prometheus/pushgateway

On Monday, September 21, 2020 at 6:58:42 AM UTC+2 Satya Prakash wrote:

> Dear All,
>
> Request you to help me for developing custom exporters.
>  1. Any reference link?
>  2. Any tutorial (freely available on web)?
>  3. Step by step verified documents, will also be fine.
>
> I want to write custom  exporters like 
> a. node_exporter (where I can update whenever team need some different 
> monitoring values),
>  b. publish application monitoring results, it can be like an output file 
> reading and storing values in tsdb.
> c. monitoring any custom sql script results.
>
> I look forward to hearing from you as soon as possible.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4824302d-8cec-4fac-b467-318441457f59n%40googlegroups.com.

[prometheus-users] Re: multiple Prometheus.yml file in single machine

[sayf.eddi...@gmail.com]

Hello, You can simply run 5 instances of Prometheus each with its own 
configuration file, just make sure every instance is listening on a 
different port

On Friday, September 18, 2020 at 11:58:07 AM UTC+2 karthiga...@gmail.com 
wrote:

> Is it possible to create multiple target file(prometheus.yml) file for a 
> single prometheus machine?
>
> My use case is I have one prometheus machne and configuring 5 projects in 
> a single machine. Is it possible to split up their configurations into 5 
> prometheus.yml files?
>
> Or any othey way to tackle this to split a file?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9a9a8019-c880-46c6-8a45-045860e028cbn%40googlegroups.com.

Re: [prometheus-users] Re: monitor DNS availability

[sayf.eddi...@gmail.com]

https://www.robustperception.io/probing-dns-servers-with-the-blackbox-exporter#:~:text=Among%20the%20Blackbox%20exporter's%20probe%20types%20is%20DNS.&text=If%20you%20visit%20http%3A%2F%2F,Public%20DNS%2C%20and%20should%20succeed.

On Thursday, September 17, 2020 at 10:31:14 AM UTC+2 Amari CH wrote:

> sayf.eddi...@gmail.com wrote:
> > Hi, you need a proper exporter
> > You can use https://github.com/prometheus/blackbox_exporter, it can 
> > monitor DNS
>
> Do you have a sample configuration?
> Thanks.
>
> -- 
> Amari CH
> https://maddoghost.com/
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/401c00b0-7ffb-4af6-ab6e-8116bd235840n%40googlegroups.com.

[prometheus-users] Re: monitor DNS availability

[sayf.eddi...@gmail.com]

Hi, you need a proper exporter
You can use https://github.com/prometheus/blackbox_exporter, it can monitor 
DNS

On Thursday, September 17, 2020 at 10:09:08 AM UTC+2 Amari CH wrote:

> Hello
>
> How can I setup prometheus to monitor the availability of name servers?
>
> regards.
>
> -- 
> Amari CH
> https://maddoghost.com/
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8fee0679-bf79-4259-a66e-2fbc088d92bfn%40googlegroups.com.

[prometheus-users] Re: how to make prometheus as system service

[sayf.eddi...@gmail.com]

Nice tip, thanks !

On Thursday, September 17, 2020 at 9:06:51 AM UTC+2 b.ca...@pobox.com wrote:

> Under [Service] I'd also recommend
>
> TimeoutStopSec=300
>
> This is to give prometheus more time to finish writing out its WAL if you 
> do "systemctl stop" or "systemctl restart".
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1b3af5c1-b909-4fea-bb8a-ffded6d2abc4n%40googlegroups.com.

[prometheus-users] Re: how to make prometheus as system service

[sayf.eddi...@gmail.com]

this is my systemd file 

```
[Unit]
After=network.target
Description=prometheus server
Documentation=http://prometheus.io

[Service]
User=prometheus
Type=simple
EnvironmentFile=/etc/sysconfig/prometheus
ExecStart=/usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml 
$EXTRA_ARGS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10
SuccessExitStatus=0
LimitNOFILE=300

[Install]
WantedBy=multi-user.target
```

Notice this is for centos/rhel, by the standard the environment files are 
under `/etc/default/prometheus` for debian based OSs, the file should 
contain the rest of the config (tsdb path web address and port etc)
Drop this as /usr/lib/systemd/system/prometheus.service and run `systemctl 
enable prometheus`
On Thursday, September 17, 2020 at 8:12:05 AM UTC+2 Amari CH wrote:

> Currently I just nohup run the processes in backend, this is ugly.
> I am using ubuntu 18.04 system, how to make those processes to 
> start/stop as system daemons?
>
> Thank you.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e7060fed-ef99-45e2-8d30-d2baeff75bd4n%40googlegroups.com.

Re: [prometheus-users] Re: Prometheus.service status failed

[sayf.eddi...@gmail.com]

Did you try having only one entry of `targets` where you put both addresses 
in one list ?

On Monday, September 14, 2020 at 3:42:41 PM UTC+2 suryapr...@gmail.com 
wrote:

> Yes,  /var/lib/prometheus is owned by the "prometheus" user. 
>
> And /etc/prometheus/prometheus.yml content is below
>
> global:
>   scrape_interval: 10s
>
> scrape_configs:
>   - job_name: 'prometheus'
> scrape_interval: 5s
> static_configs:
>   - targets: ['localhost:9090']
>   - targets: ['localhost:9100']
>
>  
> Thanks & Regards,
> Surya Prakash K
> Cloud & Devops Engineer
> M:+91-91544 63598 <+91%2091544%2063598>.
>
> On Mon, Sep 14, 2020 at 6:56 PM Brian Candler  wrote:
>
>> That's much easier to read, thank you.
>>
>> I can't see any problem with the unit file - as long as 
>> /var/lib/prometheus is owned by the "prometheus" user.
>>
>> What about the contents of the configuration file, 
>> /etc/prometheus/prometheus.yml ?
>>
>> -- 
>>
> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/09c005c0-0251-4440-999d-8067869ce894o%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/02567b80-d5a3-4d2d-b6e8-86e4b4f4a343n%40googlegroups.com.