Re: [prometheus-users] Prometheus monitoring Fortigate --- Error

2022-05-04 Thread Đỗ Ngọc Thiện
Could you please tell me where is wrong?
Vào lúc 17:09:19 UTC+7 ngày Thứ Tư, 4 tháng 5, 2022, sup...@gmail.com đã 
viết:

> I would highly recommend looking at the fortigate_exporter.
>
> https://github.com/bluecmd/fortigate_exporter
>
> On Wed, May 4, 2022 at 12:08 PM Đỗ Ngọc Thiện  wrote:
>
>> Hi all,
>> i am a new member,  I recently configure Prometheus/snmp_exporter to 
>> monitors Fortigate.
>> i am  using the default smnp.yml which I generate form snmp generator. 
>> my  generator.yml  setting look like this:
>>
>> cat go/src/github.com/prometheus/snmp_exporter/generator/generator.yml
>> modules:
>> ## Fortigate
>>   fortigate_snmp:  # The module name. You can have as many modules as you 
>> want.
>> walk:   # List of OIDs to walk. Can also be SNMP object names or 
>> specific instances.
>>   - ifXTable
>>   - fgVpn
>>   - fgSystem
>>   - fgIntf
>> version: 2  # SNMP version to use. Defaults to 2.
>> # 1 will use GETNEXT, 2 and 3 use GETBULK.
>> max_repetitions: 25  # How many objects to request with GET/GETBULK, 
>> defaults to 25.
>>  # May need to be reduced for buggy devices.
>> retries: 3   # How many times to retry a failed request, defaults to 
>> 3.
>> timeout: 5s  # Timeout for each individual SNMP request, defaults to 
>> 5s.
>>
>> auth:
>>   # Community string is used with SNMP v1 and v2. Defaults to 
>> "public".
>>   community: itpgroup
>>
>> and my  Prometheus.yml   setting look like this:
>>  FORTIGATE
>>   - job_name: 'fortigate'
>> static_configs:
>>   - targets:
>> - 10.1.11.1 # fortigate device.
>> labels:   
>>  hostname: FW_ITP
>>  device: fortigate
>>  company: ITP
>> scrape_interval: 3m
>> scrape_timeout : 3m
>> metrics_path: /snmp
>> params:
>>   module: [fortigate_snmp]
>> relabel_configs:
>>   - source_labels: [__address__]
>> target_label: __param_target
>>   - source_labels: [__param_target]
>> target_label: instance
>>   - target_label: __address__
>> replacement: 172.144.101.26:9116  # SNMP exporter.
>>
>> then i check curl: http://172.144.101.26:9116/
>> [image: Untitled1.png]
>> [image: Untitled1.png]
>>
>>
>> please help.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/0aaf073a-d23a-4e2a-b6b9-c172adb8ea56n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4a275d41-1921-491d-a61b-f918a248601fn%40googlegroups.com.


Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

2022-05-04 Thread tejaswini vadlamudi
Thanks Stuart, now I understand support in Prometheus project.
Though renowned storage vendors claim full compliance and seem to work with 
Prometheus data, it is better to stay away from it.

Cheers, Teja
On Tuesday, May 3, 2022 at 9:53:52 PM UTC+2 Stuart Clark wrote:

> On 03/05/2022 01:34, tejaswini vadlamudi wrote:
> > Prometheus documentation recommends to avoid NFS storage backends. But 
> > a few users over internet claim support for NFS based storage from 
> > Netapp in Prometheus. Just checking for opinions and feedback if this 
> > storage backend is compatible enough for regular Prometheus operations.
> >
> NFS isn't a supported option for Prometheus. Prometheus can do a lot of 
> I/O and therefore network storage often doesn't work all that well. You 
> can also have issues if the filesystem isn't fully POSIX compliant.
>
> Just because it isn't supported doesn't mean it won't work. However 
> there are issues that people find, and if you do find those the 
> recommendation would always be to switch the filesystem to something 
> local. So basically you'd be fairly on your own if you did have problems.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c436add2-e58b-43bb-99bd-bba0f11a4c23n%40googlegroups.com.


Re: [prometheus-users] WAL fsync

2022-05-04 Thread Volker Dormeyer

Thank you for your answer!

On 5/4/22 13:33, Ben Kochie wrote:
That's not correct, the author seems to be unaware of the difference 
between fsync and fdatasync[0].


The TSDB's WAL calls an internal sync(), which is implemented as 
fdatasync() by default every minute on a timer.


https://github.com/prometheus/prometheus/blob/2381d7be57314df63c17291f22e5e770b44a70f3/tsdb/wal.go#L699-L702

It also calls the sync() when closing out individual WAL segments.

[0]: https://linux.die.net/man/2/fdatasync

On Wed, May 4, 2022 at 12:23 PM Volker Dormeyer  
wrote:


Hello all,

I have serious question regarding the following article:


https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704

It is about the statement as follows from the author:

“Prometheus calls fsync



only after big chunk of data (aka |segment|) is written into WAL,
so all
the |segment| data may be lost / corrupted on power loss before
|fsync|.
The data may be corrupted if the OS flushes a few pages with the
written
data to disk, but doesn’t flush the remaining pages. Prometheus
|fscync|’s segments every 2 hours by default, so a lot of data may be
corrupted on hardware reset.”

Is it still true? Can we shorten the interval between fsync calls?
Does
this make sense?

Thanks!

Best Regards,
Volker


-- 
You received this message because you are subscribed to the Google

Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to prometheus-users+unsubscr...@googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/msgid/prometheus-users/4a8ac708-57c7-ea33-8973-ac2ce0ab1ca7%40ixolution.de.



--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1166bca5-7535-4d4a-2f7b-0779cb9b80c2%40ixolution.de.


Re: [prometheus-users] Getting no Jetty Metrics

2022-05-04 Thread 'Fabian Stäber' via Prometheus Users
Hi Gagan,

Sorry for the late reply, your mail ended up in my Spam folder for some
reason. Could you let me know which Jetty version you are using?

Fabian

On Wed, Apr 27, 2022 at 3:43 PM Gagan Singhal 
wrote:

> I have configured the Jetty metrics in my code as mentioned in the
> prometheus client_java documentation.
> I can see the jetty metrics on the mentioned endpoint (/metrics) but they
> all show the value as 0. I don't know how to proceed further. Any help is
> appreciated.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/a499c069-4c5d-475c-bb46-80813d7a07abn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPX310g1PZux48D2DP4EaQ%3Dueo2Rc0kT5DkUHAPEwjU-uM_LyA%40mail.gmail.com.


Re: [prometheus-users] WAL fsync

2022-05-04 Thread Ben Kochie
That's not correct, the author seems to be unaware of the difference
between fsync and fdatasync[0].

The TSDB's WAL calls an internal sync(), which is implemented as
fdatasync() by default every minute on a timer.

https://github.com/prometheus/prometheus/blob/2381d7be57314df63c17291f22e5e770b44a70f3/tsdb/wal.go#L699-L702

It also calls the sync() when closing out individual WAL segments.

[0]: https://linux.die.net/man/2/fdatasync

On Wed, May 4, 2022 at 12:23 PM Volker Dormeyer  wrote:

> Hello all,
>
> I have serious question regarding the following article:
>
>
> https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704
>
> It is about the statement as follows from the author:
>
> “Prometheus calls fsync
> <
> https://github.com/prometheus/tsdb/blob/bc3b0bd429153ab54662a930df3817e4f29d169e/wal/wal.go#L390>
>
> only after big chunk of data (aka |segment|) is written into WAL, so all
> the |segment| data may be lost / corrupted on power loss before |fsync|.
> The data may be corrupted if the OS flushes a few pages with the written
> data to disk, but doesn’t flush the remaining pages. Prometheus
> |fscync|’s segments every 2 hours by default, so a lot of data may be
> corrupted on hardware reset.”
>
> Is it still true? Can we shorten the interval between fsync calls? Does
> this make sense?
>
> Thanks!
>
> Best Regards,
> Volker
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/4a8ac708-57c7-ea33-8973-ac2ce0ab1ca7%40ixolution.de
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmqPcWKu-KxYouqZ7f41CXUTUdoNKdy8KRWwXx%3DKsXw_SQ%40mail.gmail.com.


Re: [prometheus-users] timestamps in email messagefrom Alertmanager.

2022-05-04 Thread Julius Volz
Hi Sebastian,

Ah yes, the issue is that different data is available on the Prometheus
side in alerting rules, than in notification templates on the Alertmanager
side.

* For alerting rules, see:
https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#templating.
So you only get: $labels, $externalLabels, and $value as variables.

* For Alertmanager notification templates, you get a whole list of alerts
with some more fields:
https://prometheus.io/docs/alerting/latest/notifications/#data

So if you want to access the ".StartsAt" field of an alert, you would have
to access that from an Alertmanager notification template.

As a very simplistic example, you could set the "html" field of an
"email_config" in alertmanager.yml to something like this to range over all
alerts in the notification and output the ".StartsAt" field for each of
them:

  html: "{{ range .Alerts }}{{ .StartsAt }}{{ end }}"

Of course in practice you probably also want to put in other alert info
(labels, annotations, etc.) in that loop. See the default for the "html"
field in
https://github.com/prometheus/alertmanager/blob/main/template/email.tmpl as
inspiration.

Cheers,
Julius



On Wed, May 4, 2022 at 9:42 AM Sebastian Osztovits <
sebastian.osztov...@gmail.com> wrote:

> Hi,
> As you have guessed I want to know the time when the alert first started
> firing.
> I simply tried adding the alerting rules as in example :
>
>
>   - alert: InstanceDown
> expr: up == 0
> for: 5m
> labels:
>   severity: critical
> annotations:
>   timestamp: >
> time:  {{ .StartsAt }}
>
> Which leaves me with the error:
> =  __alert_InstanceDown: template: __alert_InstanceDown:1:90: executing
> "__alert_InstanceDown" at <.StartsAt>: can't evaluate field StartsAt in
> type struct { Labels map[string]string; ExternalLabels map[string]string;
> Value float64 }>
>
> Regards,
> Sebastian
>
>
>
> On Tuesday, May 3, 2022 at 9:47:25 PM UTC+2 juliu...@promlabs.com wrote:
>
>> Hi Sebastian,
>>
>> What error are you getting and what's your config? The first answer puts
>> the current time of the evaluation of the alerting rule into the "time"
>> annotation (which wouldn't necessarily be the time the alert started
>> firing, but the last time it was still evaluated as firing). The second
>> Stack Overflow answer is maybe more what you want, accessing the
>> ".StartsAt" field of an alert directly from an Alertmanager notification
>> template, and thus telling you when the alert first started firing. If you
>> share what you already have (and which kind of timestamp you expect), maybe
>> we can help you make it work.
>>
>> Regards,
>> Julius
>>
>> On Tue, May 3, 2022 at 11:49 AM Sebastian Osztovits <
>> sebastian...@gmail.com> wrote:
>>
>>>
>>> Hello,
>>> I'm currently using Alert manager 0.22.2 and Prometheus 2.18.1.
>>> I have tried to alter my alert as shown in stack overflow  post:
>>>
>>>
>>> https://stackoverflow.com/questions/41069912/populating-a-date-time-in-prometheus-alert-manager-e-mail
>>> yet i am met with an error message when doing so.
>>> Does anyone have and email Template where they successfully added a
>>> Timestamp?
>>> Thanks in advance.
>>> Best Regards,
>>> Sebastian
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/e03a03be-4d88-4ded-b969-f36a06c469f2n%40googlegroups.com
>>> 
>>> .
>>>
>>
>>
>> --
>> Julius Volz
>> PromLabs - promlabs.com
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/a2631779-dd7c-40fe-897e-7de27d8c7c0dn%40googlegroups.com
> 
> .
>


-- 
Julius Volz
PromLabs - promlabs.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAObpH5ypV_MYjQStHAQ3GW4ReKoDm3GKf8EM5fEGEWpw8PBt_A%40mail.gmail.com.


[prometheus-users] WAL fsync

2022-05-04 Thread Volker Dormeyer

Hello all,

I have serious question regarding the following article:

https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704

It is about the statement as follows from the author:

“Prometheus calls fsync 
 
only after big chunk of data (aka |segment|) is written into WAL, so all 
the |segment| data may be lost / corrupted on power loss before |fsync|. 
The data may be corrupted if the OS flushes a few pages with the written 
data to disk, but doesn’t flush the remaining pages. Prometheus 
|fscync|’s segments every 2 hours by default, so a lot of data may be 
corrupted on hardware reset.”


Is it still true? Can we shorten the interval between fsync calls? Does 
this make sense?


Thanks!

Best Regards,
Volker


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4a8ac708-57c7-ea33-8973-ac2ce0ab1ca7%40ixolution.de.


Re: [prometheus-users] Prometheus monitoring Fortigate --- Error

2022-05-04 Thread Ben Kochie
I would highly recommend looking at the fortigate_exporter.

https://github.com/bluecmd/fortigate_exporter

On Wed, May 4, 2022 at 12:08 PM Đỗ Ngọc Thiện  wrote:

> Hi all,
> i am a new member,  I recently configure Prometheus/snmp_exporter to
> monitors Fortigate.
> i am  using the default smnp.yml which I generate form snmp generator.
> my  generator.yml  setting look like this:
>
> cat go/src/github.com/prometheus/snmp_exporter/generator/generator.yml
> modules:
> ## Fortigate
>   fortigate_snmp:  # The module name. You can have as many modules as you
> want.
> walk:   # List of OIDs to walk. Can also be SNMP object names or
> specific instances.
>   - ifXTable
>   - fgVpn
>   - fgSystem
>   - fgIntf
> version: 2  # SNMP version to use. Defaults to 2.
> # 1 will use GETNEXT, 2 and 3 use GETBULK.
> max_repetitions: 25  # How many objects to request with GET/GETBULK,
> defaults to 25.
>  # May need to be reduced for buggy devices.
> retries: 3   # How many times to retry a failed request, defaults to 3.
> timeout: 5s  # Timeout for each individual SNMP request, defaults to
> 5s.
>
> auth:
>   # Community string is used with SNMP v1 and v2. Defaults to "public".
>   community: itpgroup
>
> and my  Prometheus.yml   setting look like this:
>  FORTIGATE
>   - job_name: 'fortigate'
> static_configs:
>   - targets:
> - 10.1.11.1 # fortigate device.
> labels:
>  hostname: FW_ITP
>  device: fortigate
>  company: ITP
> scrape_interval: 3m
> scrape_timeout : 3m
> metrics_path: /snmp
> params:
>   module: [fortigate_snmp]
> relabel_configs:
>   - source_labels: [__address__]
> target_label: __param_target
>   - source_labels: [__param_target]
> target_label: instance
>   - target_label: __address__
> replacement: 172.144.101.26:9116  # SNMP exporter.
>
> then i check curl: http://172.144.101.26:9116/
> [image: Untitled1.png]
> [image: Untitled1.png]
>
>
> please help.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/0aaf073a-d23a-4e2a-b6b9-c172adb8ea56n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmoCh1hpfXmuunrJdb%3DLhHqANEf0-ud9aG%2Bz%3Df_WyHRJtQ%40mail.gmail.com.


[prometheus-users] Prometheus monitoring Fortigate --- Error

2022-05-04 Thread Đỗ Ngọc Thiện
Hi all,
i am a new member,  I recently configure Prometheus/snmp_exporter to 
monitors Fortigate.
i am  using the default smnp.yml which I generate form snmp generator. 
my  generator.yml  setting look like this:

cat go/src/github.com/prometheus/snmp_exporter/generator/generator.yml
modules:
## Fortigate
  fortigate_snmp:  # The module name. You can have as many modules as you 
want.
walk:   # List of OIDs to walk. Can also be SNMP object names or 
specific instances.
  - ifXTable
  - fgVpn
  - fgSystem
  - fgIntf
version: 2  # SNMP version to use. Defaults to 2.
# 1 will use GETNEXT, 2 and 3 use GETBULK.
max_repetitions: 25  # How many objects to request with GET/GETBULK, 
defaults to 25.
 # May need to be reduced for buggy devices.
retries: 3   # How many times to retry a failed request, defaults to 3.
timeout: 5s  # Timeout for each individual SNMP request, defaults to 5s.

auth:
  # Community string is used with SNMP v1 and v2. Defaults to "public".
  community: itpgroup

and my  Prometheus.yml   setting look like this:
 FORTIGATE
  - job_name: 'fortigate'
static_configs:
  - targets:
- 10.1.11.1 # fortigate device.
labels:   
 hostname: FW_ITP
 device: fortigate
 company: ITP
scrape_interval: 3m
scrape_timeout : 3m
metrics_path: /snmp
params:
  module: [fortigate_snmp]
relabel_configs:
  - source_labels: [__address__]
target_label: __param_target
  - source_labels: [__param_target]
target_label: instance
  - target_label: __address__
replacement: 172.144.101.26:9116  # SNMP exporter.

then i check curl: http://172.144.101.26:9116/
[image: Untitled1.png]
[image: Untitled1.png]


please help.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/0aaf073a-d23a-4e2a-b6b9-c172adb8ea56n%40googlegroups.com.


Re: [prometheus-users] timestamps in email messagefrom Alertmanager.

2022-05-04 Thread Sebastian Osztovits
Hi, 
As you have guessed I want to know the time when the alert first started 
firing. 
I simply tried adding the alerting rules as in example :


  - alert: InstanceDown
expr: up == 0
for: 5m
labels:
  severity: critical
annotations:
  timestamp: >
time:  {{ .StartsAt }}

Which leaves me with the error:
= : can't evaluate field StartsAt in type struct { Labels 
map[string]string; ExternalLabels map[string]string; Value float64 }>

Regards,
Sebastian 



On Tuesday, May 3, 2022 at 9:47:25 PM UTC+2 juliu...@promlabs.com wrote:

> Hi Sebastian,
>
> What error are you getting and what's your config? The first answer puts 
> the current time of the evaluation of the alerting rule into the "time" 
> annotation (which wouldn't necessarily be the time the alert started 
> firing, but the last time it was still evaluated as firing). The second 
> Stack Overflow answer is maybe more what you want, accessing the 
> ".StartsAt" field of an alert directly from an Alertmanager notification 
> template, and thus telling you when the alert first started firing. If you 
> share what you already have (and which kind of timestamp you expect), maybe 
> we can help you make it work.
>
> Regards,
> Julius
>
> On Tue, May 3, 2022 at 11:49 AM Sebastian Osztovits <
> sebastian...@gmail.com> wrote:
>
>>
>> Hello, 
>> I'm currently using Alert manager 0.22.2 and Prometheus 2.18.1. 
>> I have tried to alter my alert as shown in stack overflow  post:  
>>
>>
>> https://stackoverflow.com/questions/41069912/populating-a-date-time-in-prometheus-alert-manager-e-mail
>> yet i am met with an error message when doing so. 
>> Does anyone have and email Template where they successfully added a 
>> Timestamp?
>> Thanks in advance. 
>> Best Regards, 
>> Sebastian 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/e03a03be-4d88-4ded-b969-f36a06c469f2n%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Julius Volz
> PromLabs - promlabs.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a2631779-dd7c-40fe-897e-7de27d8c7c0dn%40googlegroups.com.